如果能調高或禁止 LLM 選用特定 token，會怎樣？

Hello, I'm Rijul, and I'm building LiveReview — a blast-radius aware AI code review built for your business-critical systems. Star us to help devs discover the project, give it a try, and share your feedback to help improve the product.

Sometimes, you may want a specific word to appear more often in an LLM's output. In other cases, you may want to prevent the model from generating certain words altogether.

Prompting is one way to influence the model's choices. But there is another approach: logit bias.

Logit bias lets you adjust the likelihood of specific tokens being selected during text generation. Let's understand how it works and how to use it.

1. How a Language Model Generates Text

When you type on your phone, it suggests the next word. Language models work similarly, but on a much larger scale.

A language model generates text one token at a time. A token can be a complete word, part of a word, punctuation, or even a space combined with a word.

Suppose you ask a model to complete this sentence:

The cat sat on the ...

The model might consider several possible next tokens:

• floor
• mat
• sofa

It calculates a score for each possible next token. These scores help determine which token the model selects.

2. How the Model Scores Its Options

Before choosing the next token, the model assigns a score called a logit to each candidate token.

For example, imagine the model produces these logits:

A higher logit means the token is more favored relative to the other candidates.

However, logits are not probabilities. A logit of 4.1 does not mean the token has a 410% probability or a 4.1% probability.

The model converts these scores into probabilities using a function called softmax.

3. From Logits to Probabilities

Softmax converts logits into probabilities that add up to 100%.

Using the example above, the model's probabilities are approximately:

The model now has a probability distribution over these three candidates. Depending on its decoding settings, it can select a token from that distribution.

Two settings can influence this process: temperature and logit bias.

How temperature changes the probabilities

Temperature controls how concentrated or spread out the probability distribution becomes.

Using the same logits from the previous example, here's how different temperatures affect the probabilities:

At a lower temperature, the model favors the highest-scoring token more strongly. At a higher temperature, the probabilities become more evenly distributed, giving lower-scoring tokens a better chance of being selected.

Notice that the logits stay the same. Temperature changes how the model converts those logits into probabilities.

Logit bias works differently. Instead of changing the temperature, it modifies the logits of selected tokens before the probabilities are calculated.

4. What Is Logit Bias?

Logit bias lets you add a positive or negative adjustment to the logit of a specific token.

• Positive bias: Increases the token's logit, making it more likely to be selected.
• Negative bias: Decreases the token's logit, making it less likely to be selected.
• A sufficiently strong negative bias: Can effectively prevent a token from being selected.

Let's return to our example.

Initially, the logits are:

Suppose we apply a logit bias of +2 to dog.

Its new logit becomes:

(3.4 + 2 = 5.4)

The adjusted logits are now:

After softmax, the approximate probabilities become:

The model now strongly favors dog, which previously had a lower probability than cat.

This illustrates the core idea: logit bias changes the odds of selecting particular tokens without changing the model's underlying parameters.

5. Think of Logit Bias as a Dial

Imagine each token has a dial controlling how strongly the model favors it.

• Turning the dial up makes the token more likely to appear.
• Turning the dial down makes it less likely to appear.
• Turning it all the way down can prevent the token from being selected.

The adjustment applies to the token's logit, not directly to its final probability. Therefore, the resulting probability change depends on the logits of the other candidates too.

A positive bias does not guarantee that a token will appear. It only changes its relative likelihood.

6. Your First Logit Bias Request

The OpenAI Chat Completions API supports a logit_bias parameter that maps token IDs to bias values.

For example, in Python:

from openai import OpenAI

client = OpenAI()

response = client.chat.completions.create(
    model="YOUR_MODEL",
    messages=[
        {
            "role": "user",
            "content": "Write a short sentence about animals."
        }
    ],
    logit_bias={
        "3797": -100
    }
)

print(response.choices[0].message.content)

In this example, "3797" is an illustrative token ID, not a guaranteed ID for a particular word. You must find the correct token ID for your chosen model and tokenizer.

A bias of -100 strongly suppresses the specified token. The API accepts bias values from -100 to 100.

Before using this parameter, check the documentation for your chosen model to confirm that it supports logit_bias. Support can vary by model.

What does the token ID mean?

The API does not accept a word directly as the key in the logit_bias dictionary. It expects a token ID represented as a string.

To find token IDs, you need to tokenize the text using a tokenizer compatible with your model.

7. How to Find Token IDs

For models supported by the tiktoken library, you can inspect token IDs with Python:

import tiktoken

enc = tiktoken.encoding_for_model("YOUR_SUPPORTED_MODEL")

print(enc.encode(" cat"))

The output contains the token IDs corresponding to the input text.

Replace the example model name with a model supported by tiktoken. Do not assume that every model uses the same tokenizer or that a token ID from one model will work for another.

Also, pay attention to the exact text you encode. For example, these strings may tokenize differently:

• "cat"
• " cat"
• "Cat"
• "cats"

A word may consist of multiple tokens, and different capitalization or spacing may produce different token IDs.

8. Where Can You Use Logit Bias?

Logit bias is useful when you want more control over the vocabulary a model uses.

Constraining possible answers

Suppose you're building a classifier that should respond with positive, negative, or neutral.

You can use prompting to request those labels and potentially use logit bias to discourage unwanted alternatives.

However, logit bias alone does not guarantee valid output. For strict structured responses, use an appropriate constrained-output or structured-output feature when available.

Banning specific tokens

Suppose you don't want a model to use a particular token. A strong negative bias can suppress it.

But there is an important limitation: suppressing one token does not necessarily prevent the model from generating the same word through a different tokenization.

Reducing common clichés

Suppose a model repeatedly uses words such as "delve" or "tapestry" in its answers.

You could apply a negative bias to the relevant token IDs to make those choices less likely.

This is a form of vocabulary steering. It is not a perfect word filter, because a word may have several tokenization variants or appear as part of a larger token.

Steering vocabulary

You can also use positive bias to encourage certain vocabulary. For example, a writing assistant could make selected technical terms more likely to appear.

Still, positive bias does not guarantee that a word will appear. The model must also consider the surrounding context and its other candidate tokens.

For important requirements, combine logit bias with clear prompting and appropriate validation.

9. The Tricky Part: Words Aren't Always Single Tokens

One of the most important details about logit bias is that it operates on tokens, not complete words.

Consider the word cat. Depending on the tokenizer and context, it might be represented differently from cat or Cat. Other words may be split into multiple tokens.

This creates a limitation when you want to ban a word completely.

If you suppress one token ID, you may only suppress one tokenization variant. The model might still generate the word using another sequence of tokens.

Similarly, applying a positive bias to one token does not guarantee that the complete word will appear.

Logit bias provides token-level control, not reliable word-level enforcement.

When you need to enforce a strict rule, use a mechanism designed for that purpose rather than relying on logit bias alone.

10. Choosing the Right Bias Value

The API accepts bias values between -100 and 100, but the right value depends on your use case and the model's behavior.

A practical approach is to start with a small adjustment and test the results:

• Small positive or negative values: Gently influence token selection.
• Larger adjustments: Apply stronger pressure toward or away from a token.
• Extreme negative values, such as -100: Strongly suppress the selected token.

These are general guidelines, not universal thresholds. A bias value does not correspond to a fixed percentage-point change in probability.

Test your settings with representative prompts. Check whether the model follows the intended behavior and whether the adjustment introduces unwanted side effects.

Conclusion

Prompting lets you tell a language model what you want. Logit bias gives you another way to influence its choices by adjusting the scores of specific tokens.

It can help steer vocabulary, discourage certain tokens, or make selected outputs more likely. But because it operates at the token level, it cannot reliably enforce every word-level or content-level rule.

Understanding that distinction helps you use logit bias where it is useful, while choosing more suitable controls when you need stronger guarantees.

Your team's attention is limited, and the deluge of AI-generated code is making it harder to keep production reliable and secure without slowing you down.

I'm building LiveReview, a blast-radius aware AI code review built for your business-critical systems.

Instead of presenting every diff with equal emphasis, LiveReview scores each change by blast radius — how far its impact reaches through your call graph — so you can focus attention where it actually matters.

Spend code review effort where business risk is highest — not spread evenly across every diff.

LiveReview is built for engineering teams that ship often, but also care about keeping production stable for customers. It approves some 40-50% of low-risk changes automatically, and it gives your team the tools to handle the rest:

• Understand risky changes faster.
• Fix problems in those changes, with human attention directed to the work that needs it.
• Enforce your engineering standards.
• Learn from production outcomes, such as incidents, rollbacks, and queue wait times.
• Improve the review process over time and make it more efficient.
• Connect to the major engineering tools that your team already uses.

Auto-Approve: Ship the Safe 50% Without a Human

Auto-Approve Upto 50% of Your Pull Requests, Safely. Risk scoring decides which MRs/PRs are safe enough to ship without a human, so your team only spends review time on risky and critical changes.

demo.mp4

LiveReview auto-approving a low-risk change while…

Click below to try LiveReview with your codebase: