大語言模型推理的一次請求流程：tokenize、prefill、decode 與串流

You type "Why is the sky blue?" and press Enter.

There's a short pause. Then words start appearing, one after another, like someone is typing to you very fast.

What happened during that pause? And why do the words come out one at a time instead of all at once?

Let's take a walk through the Word Factory.

Welcome to the Word Factory

Picture a small factory with a conveyor belt. Your question goes in at one end, and an answer comes out on your phone at the other. Along the way there are four machines:

1. The Chopper cuts your question into small pieces.
2. The Reader has lots of eyes and reads every piece at the same time.
3. The Typewriter writes the answer, but only one piece at a time.
4. The Delivery Tape carries each piece to your phone the moment it's typed.

There's also a notebook hanging above the Reader. It's the factory's secret weapon, and it matters a lot later.

Engineers have proper names for these four steps: tokenize, prefill, decode and stream. That's all an "inference engine" is: the factory that runs a language model and turns your question into an answer.

Let's visit each machine.

Machine 1: The Chopper (tokenize)

A language model can't read letters the way you do. It only understands numbers. So the first job is to cut your text into small pieces called tokens and give each piece a number.

A token is usually a word or part of a word. Think of LEGO bricks: common words are one brick, and long or rare words get built from a few smaller bricks.

"Why is the sky blue?"
   ↓ chop
[ "Why" ][ " is" ][ " the" ][ " sky" ][ " blue" ][ "?" ]
   ↓ give each a number
[  512,     87,     33,    2041,     978,     11  ]

(These numbers are made up for the example. Each model has its own dictionary that decides which number goes with which piece.)

How long does it take? Almost no time at all: a tiny fraction of a millisecond. The Chopper is never the slow machine.

Machine 2: The Reader (prefill)

Now the Reader looks at your whole question.

Here's the clever part: the Reader has lots of eyes, and it reads every token at the same time. It doesn't go word by word. Whether your question is 6 tokens or 600, it reads them all in one big gulp.

While it reads, it writes notes in the notebook: for every token, a short note about what that token means and how it connects to the others. Engineers call this notebook the KV cache. (KV stands for "keys and values", which is just what the notes are called inside the model.)

When the Reader finishes, it has a guess for the very first word of the answer.

This step is called prefill because it fills the notebook with notes about your whole question before any answer is written.

The important bit: a longer question means more to read. So the longer your question, the longer you wait for the first word.

Machine 3: The Typewriter (decode)

Now the answer gets written, and this is where things slow down.

The Typewriter can only type one token per step. Why? Because each new word depends on the words before it. You can't decide the 5th word of a sentence until you know the first 4. Try finishing "The sky is blue because..." without knowing the word "because" was there.

So for every single token, the factory does a full trip:

1. Look at the notebook (all the notes so far)
2. Run through the whole model to pick the next token
3. Add one new line to the notebook
4. Type the token
5. Go back to step 1 for the next token

That's the pink "again & again" arrow in the picture. Engineers call this the decode loop.

Why the notebook is so useful: without it, the Typewriter would have to re-read your entire question and everything it has typed so far, every single step. That would be like re-reading a whole book from page 1 every time you want to write the next word. With the notebook, it just glances at its notes.

Machine 4: The Delivery Tape (stream)

The moment a token is typed, it's turned back into text and sent straight to your screen. The factory doesn't wait to finish the whole answer first.

That's why you see words appear one by one. It's not a fancy animation. You're watching the Typewriter work in real time.

This is called streaming, and it makes chatbots feel much faster. You start reading while the rest is still being written.

Three stopwatches

So how do we say if a factory is "fast"? There are three different stopwatches, and they measure different things.

⏱ Stopwatch 1: Time to First Token (TTFT)
How long from pressing Enter until the first word shows up. This is the "awkward pause." It includes chopping, the whole reading step, and picking the first token.

TTFT = time first token arrives − time you pressed Enter

⏱ Stopwatch 2: Inter-Token Latency (ITL)
The gap between one word and the next. If it's small and steady, the text flows smoothly. If one gap is suddenly huge, it feels like the chatbot froze, even if the average is fine.

ITL = time token 2 arrives − time token 1 arrived   (and so on, for every gap)

⏱ Stopwatch 3: Throughput (tokens per second)
How many tokens come out per second. Careful, there are two kinds:

• For one person: how fast your answer is typed.
• For the whole factory: how many tokens it produces for everyone at once.

A factory serving 100 people at 5,000 tokens per second in total gives each person about 50 tokens per second. Both numbers are true. They just answer different questions.

And the total wait for your full answer is simply:

total time ≈ TTFT + (number of tokens − 1) × ITL

Let's build a tiny factory and time it

Talking is fine, but let's measure. Below is a complete Word Factory in about 120 lines of Python, using only NumPy.

It has the same shape as a small real model (GPT-2 small: 12 layers, 85 million numbers inside). One honest warning: its "brain" is filled with random numbers, so the words it types are gibberish. But it does exactly the same amount of work as a trained model of that size, so the stopwatch readings are real.

The Chopper

To keep it simple, our Chopper cuts text into single bytes (roughly, one letter = one token).

import codecs
import numpy as np

class ByteTokenizer:
    vocab_size = 256

    def encode(self, text: str) -> list[int]:
        # Chop: every byte of the text becomes one token number.
        return list(text.encode("utf-8"))

    def stream_decoder(self):
        # Turn token numbers back into text, one at a time.
        # (It politely waits if a letter like "é" is split across two tokens.)
        dec = codecs.getincrementaldecoder("utf-8")(errors="replace")
        return lambda tok: dec.decode(bytes([tok]))

The Reader and the Typewriter are the same machine

Here's a surprise: reading and typing use the same machine. The only difference is how many tokens you feed it. Reading (prefill) feeds it the whole question at once. Typing (decode) feeds it one token at a time.

You don't need to follow every line. Look for the two comments marked 📓: that's where the notebook gets written and read.

class TinyDecoder:
    def __init__(self, n_layers=12, d=768, n_heads=12, d_ff=3072,
                 vocab=256, max_seq=4096, seed=0):
        rng = np.random.default_rng(seed)
        w = lambda *s: (rng.standard_normal(s) * 0.02).astype(np.float32)
        self.L, self.d, self.H, self.hd = n_layers, d, n_heads, d // n_heads
        self.tok_emb, self.pos_emb = w(vocab, d), w(max_seq, d)
        self.layers = [dict(qkv=w(d, 3 * d), o=w(d, d),
                            up=w(d, d_ff), down=w(d_ff, d))
                       for _ in range(n_layers)]
        self.lm_head = w(d, vocab)
        # The notebook (KV cache): room for notes on up to 4096 tokens.
        self.k = np.zeros((n_layers, n_heads, max_seq, self.hd), np.float32)
        self.v = np.zeros_like(self.k)
        self.n_cached = 0

    @staticmethod
    def _norm(x):
        return (x - x.mean(-1, keepdims=True)) / np.sqrt(x.var(-1, keepdims=True) + 1e-5)

    def forward(self, ids: np.ndarray) -> np.ndarray:
        """Feed in NEW tokens (the whole question, or just one).
        Returns the scores for what the next token should be."""
        T, start = len(ids), self.n_cached
        x = self.tok_emb[ids] + self.pos_emb[start:start + T]
        for i, p in enumerate(self.layers):
            q, k, v = np.split(self._norm(x) @ p["qkv"], 3, axis=-1)
            q, k, v = (t.reshape(T, self.H, self.hd).transpose(1, 0, 2) for t in (q, k, v))
            # 📓 write notes for the new tokens into the notebook
            self.k[i, :, start:start + T], self.v[i, :, start:start + T] = k, v
            # 📓 read ALL notes so far (old + new)
            K, V = self.k[i, :, :start + T], self.v[i, :, :start + T]
            att = q @ K.transpose(0, 2, 1) / np.sqrt(self.hd)
            mask = np.triu(np.ones((T, start + T), bool), k=start + 1)  # no peeking ahead
            att = np.where(mask, -1e9, att)
            att = np.exp(att - att.max(-1, keepdims=True))
            att /= att.sum(-1, keepdims=True)
            y = (att @ V).transpose(1, 0, 2).reshape(T, self.d)
            x = x + y @ p["o"]
            h = self._norm(x) @ p["up"]
            x = x + (0.5 * h * (1 + np.tanh(0.79788456 * (h + 0.044715 * h**3)))) @ p["down"]
        self.n_cached += T
        return self._norm(x[-1]) @ self.lm_head

The whole factory, start to finish

This little function is the conveyor belt. Read it top to bottom and you'll spot all four machines:

import time

def generate(model, tok, prompt: str, max_new_tokens: int):
    """Yield (event, text, timestamp) for one request, end to end."""
    model.n_cached = 0
    ids = np.array(tok.encode(prompt))                     # 1. CHOP
    yield "tokenized", "", time.perf_counter()
    logits = model.forward(ids)                            # 2. READ it all at once
    yield "prefilled", "", time.perf_counter()
    detok = tok.stream_decoder()
    for _ in range(max_new_tokens):                        # 3. TYPE, one at a time...
        nxt = int(np.argmax(logits))                       #    pick the best next token
        yield "token", detok(nxt), time.perf_counter()     # 4. SEND it right away
        logits = model.forward(np.array([nxt]))            #    ...and again

The full file, including a small measure() helper that reads the stopwatches, is tiny_engine.py in the code for this post.

The results

I ran the factory with three question sizes (64, 256 and 1,024 tokens), asked for 64 tokens of answer each time, and did one warm-up run first. Plain CPU, 2 cores, no GPU:

 question  chop ms  read ms   TTFT ms  gap p50  gap p99  total s  tokens/s
     64      0.01     452.2    452.2    101.8    120.9     6.83      9.9
    256      0.03    1656.2   1656.4    106.9    126.4     8.33      9.4
   1024      0.04    9444.8   9445.0    134.4    175.0    18.00      7.4

("gap p50" is the typical gap between words. "gap p99" is the gap that only 1 in 100 steps is slower than: the "worst normal" gap.)

Here's the 256-token run drawn on a timeline:

 press Enter                                                               done
 │                                                                          │
 ├─ chop: 0.03 ms (too small to see)
 │
 ├────────── READ the whole question: 1,656 ms ──────────┤
 │                                                        │
 │                                  first word appears ──▶ █ █ █ █ █ █ █ ... █
 │                                                         └─┘
 │                                                       gap ≈ 107 ms, 63 times
 │◀──────────────── TTFT = 1.66 s ───────────────────────▶│◀── typing: 6.7 s ──▶│

Four things we just discovered

1. Reading is about 16 times cheaper per token than typing

For the 256-token question, reading took 1,656 ms. That's about 6.5 ms per token.

Typing took 107 ms per token.

Same machine, same computer. Why the huge difference?

Every trip through the model means hauling all 85 million numbers (about 340 MB) through the processor. When the Reader does that trip, it handles 256 tokens at once, so the cost is shared 256 ways. When the Typewriter does it, the whole trip is for one token.

It's like a school bus. Driving the bus across town costs about the same whether it carries 1 kid or 50. The Reader fills the bus. The Typewriter drives it across town with one passenger, over and over again.

This is the single most important idea in making chatbots fast, and it's why real servers try to type for many people at once (filling the bus). Later posts in this series dig into it.

2. The awkward pause is almost all reading

Chopping took 0.01 to 0.04 ms: basically nothing. TTFT was almost exactly the reading time.

And reading time grew with the question: 0.45 s for 64 tokens, 9.4 s for 1,024 tokens. A question 16 times longer made the wait about 21 times longer. (It grows a bit faster than the question, because every token has to "look at" every other token.)

Lesson: if the first word takes too long, the first suspect is a long question.

3. The notebook gets fatter, so typing slowly gets slower

The typical gap between words grew from 102 ms (short question) to 134 ms (long question).

That's the notebook. Every step, the Typewriter reads all the notes. Every token adds a new line. For our little model, each token adds about 72 KB of notes:

notes per token = 2 (K and V) × 12 layers × 768 numbers × 4 bytes = 73,728 bytes ≈ 72 KB

After about 1,100 tokens, that's roughly 80 MB of notes, read again on every single step. Bigger models have much bigger notebooks, and that's what decides how many people one GPU can serve at a time.

4. Most of the total time is typing

For 256 tokens in and 64 tokens out: 1.66 s of reading + 63 gaps × 0.107 s of typing ≈ 8.4 s. The real measurement was 8.33 s.

Typing was 80% of the total. So the length of the answer matters more than anything else for total waiting time. That's why chatbots that "think" for thousands of tokens before answering are so expensive to run.

Bonus: time a real chatbot server

Measuring inside the factory flatters it. It doesn't count time spent waiting in line or travelling over the network. The fair way is to measure from your side, like a real user.

This script works with any server that speaks the OpenAI API format, including popular engines like vLLM, SGLang and TensorRT-LLM:

import argparse, statistics, time
from openai import OpenAI

ap = argparse.ArgumentParser()
ap.add_argument("--base-url", default="http://localhost:8000/v1")
ap.add_argument("--model", default="tiny")
ap.add_argument("--max-tokens", type=int, default=64)
ap.add_argument("--prompt", default="Explain how a large language model answers "
                                    "a question, step by step. " * 4)
args = ap.parse_args()
client = OpenAI(base_url=args.base_url, api_key="unused")

t_send = time.perf_counter()                       # press Enter
stream = client.completions.create(model=args.model, prompt=args.prompt,
                                   max_tokens=args.max_tokens, temperature=0,
                                   stream=True,
                                   stream_options={"include_usage": True})
arrivals, usage = [], None
for chunk in stream:
    if chunk.choices:                              # a token arrived!
        arrivals.append(time.perf_counter())
    if chunk.usage:
        usage = chunk.usage
t_done = time.perf_counter()

ttft = arrivals[0] - t_send
gaps = [b - a for a, b in zip(arrivals, arrivals[1:])]
out_tokens = usage.completion_tokens if usage else len(arrivals)
print(f"prompt tokens      : {usage.prompt_tokens if usage else '?'}")
print(f"output tokens      : {out_tokens}")
print(f"TTFT               : {ttft*1e3:8.1f} ms")
print(f"ITL p50 / p99      : {statistics.median(gaps)*1e3:8.1f} / "
      f"{sorted(gaps)[int(0.99*(len(gaps)-1))]*1e3:.1f} ms")
print(f"end-to-end latency : {t_done - t_send:8.2f} s")
print(f"decode tokens/s    : {(out_tokens - 1) / (arrivals[-1] - arrivals[0]):8.1f}")

To test it without a GPU, I wrapped our tiny factory in a small web server (tiny_server.py, 42 lines, in the code for this post) and pointed the script at it:

prompt tokens      : 276
output tokens      : 64
TTFT               :   1713.4 ms
ITL p50 / p99      :    107.5 / 129.0 ms
end-to-end latency :     8.53 s
decode tokens/s    :      9.4

It matches the factory's own numbers, plus a little extra for the trip over the network. Point it at a real GPU server and the numbers get much smaller, but the pattern stays the same: the first word waits for reading, and then the words arrive at a steady beat.

Five easy mistakes when timing a chatbot

1. Starting the stopwatch at the wrong place. Start it when the user presses Enter, not when the server starts working. Waiting in line is part of the wait.
2. Counting chunks instead of tokens. Some servers bundle a few tokens into one message. Check how many messages you got against how many tokens the server says it made.
3. Only looking at the average gap. One 2-second freeze in a long answer barely changes the average, but everybody notices it. Always look at the slow end too (p99).
4. Forgetting to warm up. The very first request is slow because the engine is still setting up. Don't count it.
5. Mixing up "fast for me" and "fast for everyone." Per-person speed and whole-factory speed are different numbers. Keep them in separate columns.

Cheat sheet

What's next

Next time, we open up the Reader-Typewriter machine and look inside. F2: A Forward Pass for One Token: Where the Work Goes. We'll follow one token through a single layer and count the work done by each part, and find out why almost all of it is just multiplying big tables of numbers.

Every code block in this post was actually run, and every number above comes from those runs. The tiny factory used random weights on a 2-core CPU, so it shows the pattern, not the speed of any real GPU.