大語言模型（LLM）的核心元素：模型 token、上下文窗口、Temperature 與抽樣

Large Language Models (LLMs) have become a fundamental part of modern AI applications, powering everything from conversational chatbots to code generation and intelligent assistants. While their capabilities may seem complex, understanding the core elements behind how they process and generate text provides a clearer picture of how they work.

In this article, we'll explore some of the fundamental concepts behind LLMs, including tokens, context windows, temperature, Top-K sampling, Top-P sampling, and repetition penalties. Understanding these elements will help you build a stronger foundation in LLMs and understand how different parameters influence their outputs.

What are Tokens?

Tokens are individual units of data that are fed into a model during training. They can be words, phrases, or even entire sentences depending on the type of model being trained.

For example, in NLP, tokens are commonly used to represent words in a text. Consider the sentence "Hello, world!" - it might be tokenized into ["Hello", ",", "world", "!"]. These tokens are then used as inputs for the model to learn patterns and relationships between them.

Tokens can also represent other types of data such as numerical values or images. For instance, in image recognition tasks, each pixel in an image is a token that the model uses to identify and classify objects.

Types of Tokens

Tokens can take many forms depending on the type of data and the task at hand. Let’s look at a brief overview of the common types of tokens:

• Word Tokens: Each word is treated as a separate token.
• Subword Tokens: Words are broken down into smaller meaningful units to handle out-of-vocabulary words better. E.g. "cats" can be broken down into "cat" and "s".
• Phrase Tokens: These consist of multiple words that are grouped together, such as "New York City" or "machine learning".
• Character Tokens: These represent individual characters within a word.
• Image Tokens: These can include pixels, image segments, or other visual features used in computer vision tasks.
• Byte-Pair Encoding (BPE): This is a type of tokenization that uses an algorithm to merge the most frequently occurring character pairs in a given text corpus. BPE is commonly used in speech recognition and natural language processing tasks.

Context Window

The context window (or “context length”) of a large language model (LLM) is the amount of text, in tokens, that the model can consider or “remember” at any one time. A larger context window enables an AI model to process longer inputs and incorporate a greater amount of information into each output.

An LLM’s context window can be thought of as the equivalent of its working memory. It determines how long of a conversation it can carry out without forgetting details from earlier in the exchange. It also determines the maximum size of documents or code samples that it can process at once. When a prompt, conversation, document or code base exceeds an artificial intelligence model’s context window, it must be truncated or summarized for the model to proceed.

Generally speaking, increasing an LLM’s context window size translates to increased accuracy, fewer hallucinations, more coherent model responses, longer conversations and an improved ability to analyze longer sequences of data. However, increasing context length is not without tradeoffs: it often entails increased computational power requirements and therefore increased costs.

Equipping a model with a large context window comes at a cost, both figuratively and literally. Compute requirements scale quadratically with the length of a sequence: for instance, if the number of input tokens doubles, the model needs 4 times as much processing power to handle it.

Temperature

Temperature is a crucial feature for controlling randomness in model performance. It allows users to adjust the LLM output to better suit different real-world applications of text generation. More specifically, this LLM setting allows users to balance coherence and creativity when generating output for a specific use case. For instance, a low temperature might be preferable for tasks requiring precision and factual accuracy, such as technical documentation or conversational replies with chatbots. The lower temperature value helps the LLM to produce more coherent and consistent text and avoid irrelevant responses. By contrast, a high temperature is preferable for creative outputs or creative tasks such as creative writing or concept brainstorming. The temperature setting effectively allows users to fine-tune LLMs and adjust a model's output to their own desired outcome.

Top-K Sampling

Top-K sampling limits the model to choosing from the K most likely tokens at each step. For instance:

• If top-K = 20, the model only considers the 20 most probable tokens, ignoring less likely ones. This makes responses more focused and factual.
• If top-K = 40, the model has more options, which adds variety but can introduce less predictable results.

Think of top-K as narrowing the model’s “vocabulary” to a shortlist of top contenders. A smaller K value keeps things tight and controlled, while a larger K allows for more creativity.

Top-P (Nucleus Sampling)

Top-P sampling, also called nucleus sampling, selects a group of tokens whose cumulative probability is at least P. For example:

• If top-P = 0.9, the model picks the smallest set of tokens whose combined probability reaches 90%. This filters out low-probability tokens, balancing accuracy and creativity.
• A lower top-P (e.g., 0.8) makes the output more conservative, while a higher top-P (e.g., 0.99) allows more diversity.

Unlike top-K, which fixes the number of tokens, top-P dynamically adjusts based on probability, making it more flexible for tasks requiring varied outputs.

How They Work Together

Top-K and top-P are often used alongside temperature, which controls the randomness of the model’s choices:

• Low temperature (e.g., 0.1) makes the model stick closely to high-probability tokens, ideal for precise tasks.
• High temperature (e.g., 0.9) increases randomness, boosting creativity but risking incoherent outputs.

Repetition Penalty

A repetition penalty is an inference-time decoding parameter in Large Language Models (LLMs) that reduces the probability of generating tokens that have already appeared in the text. It helps prevent repetitive phrases and endless loops during text generation.

The penalty is applied to previously encountered tokens by modifying their logits before the next token is selected.

• 1.0: No penalty is applied.
• Greater than 1.0: Repeated tokens are discouraged.
• Too high: Can make the output unnatural or reduce clarity.

Repetition penalty is useful in chatbots, content generation, and long-form text generation. However, excessive penalties can discourage necessary word repetition and affect the quality of responses.

Conclusion

Understanding the core elements of LLMs is an important step toward understanding how modern language models work. From breaking text into tokens and processing information within a context window to controlling output generation through temperature, sampling strategies, and repetition penalties, each element plays a role in shaping the model's behavior.

These concepts provide a foundation for exploring more advanced topics, such as Transformer architecture, attention mechanisms, model inference, and LLM optimization. As you continue learning about LLMs, understanding these fundamentals will help you move beyond simply using AI models to understanding how to configure, optimize, and build applications around them.