Tokenization 為何令智能體費用看起來異常

Models don't read text. They read integers.

Between your string and the model there's a tokenizer, and it quietly decides how much you pay, how much fits in the context window, and why your model is weirdly bad at counting letters.

I always treated it as a preprocessing detail. Then i started counting.

What a tokenizer actually does

It chops text into pieces from a fixed vocabulary, usually 30k to 200k entries, and maps each piece to an id. Common words become one token. Rare stuff gets split into smaller bits.

You can see it directly with tiktoken:

import tiktoken
enc = tiktoken.get_encoding("cl100k_base")
for w in ("tokenization", " tokenization", "Pneumonoultramicroscopicsilicovolcanoconiosis"):
    print(repr(w), [enc.decode([t]) for t in enc.encode(w)])

'tokenization'  ['token', 'ization']
' tokenization' [' token', 'ization']
'Pneumonoultramicroscopicsilicovolcanoconiosis'
  ['P', 'ne', 'um', 'on', 'oul', 'tram', 'icro', 'sc', 'op', 'ics', 'il', 'ic', 'ov', 'ol', 'cano', 'con', 'iosis']

A common word: two pieces. A rare one: seventeen. Also notice that a leading space changes the token. Small thing, but it's why prompts with different whitespace can behave differently.

How the vocabulary gets built

Most modern ones are some flavour of BPE (byte pair encoding). The idea is almost boringly simple: start with single characters, find the most frequent adjacent pair, merge it into a new symbol, repeat.

I ran it on a tiny made-up corpus (low x5, lower x2, newest x6, widest x3):

1  merge s+t   (9x)   n e w e st
2  merge e+st  (9x)   n e w est
3  merge o+w   (7x)   l ow
4  merge l+ow  (7x)   low
5  merge w+est (6x)   n e west
6  merge n+e   (6x)   ne west

Do that for tens of thousands of merges over a huge corpus and you get a vocabulary. Whatever was common in the training text gets short, cheap tokens. Whatever wasn't, doesn't.

That last sentence is the whole post.

The families, quickly

• BPE: merge frequent pairs. GPT-2 style.
• Byte-level BPE: BPE over raw bytes, so there's never an "unknown" character. Anything can be encoded, it just might take many tokens. GPT-2 and the OpenAI tiktoken encodings work this way.
• WordPiece: similar, but picks merges by likelihood instead of raw frequency. BERT uses it.
• SentencePiece: works on raw text including spaces (no pre-splitting on whitespace), so it handles languages without spaces. T5 and many other models use it.

So I counted

Same strings, four tokenizers: GPT-2, cl100k_base (GPT-4 era), o200k_base (GPT-4o era) and T5's SentencePiece. Normalised to tokens per 100 characters so short and long samples compare.

Raw counts:

(These are single samples I picked, not a benchmark. Your text will differ. T5 and the others also have different vocabularies, so it isn't a pure "which is better" ranking.)

A few things jumped out.

English is the happy path. Everyone's about the same, because every vocabulary was built mostly on English.

Code and JSON halved between GPT-2 and the newer encodings. My guess is that newer vocabularies were trained on a lot more code, so runs of whitespace and common JSON punctuation get merged into single tokens. I didn't dig into the vocab files to confirm that.

Hindi is where it hurts. 111 tokens in GPT-2, 78 in cl100k, and 24 in o200k. Same sentence. With the older encoding it's more than 4x the cost and 4x the context used, and I'd guess quality is worse too, though I didn't measure that. Dropping to individual characters is what a byte-level tokenizer does when it has never seen your script much:

'मेरा' -> ['म', 'े', 'र', 'ा']   # one token per character/mark

Emoji are all over the place.

Why this matters for agents

• Tool schemas are paid for on every call. My little 610 character JSON schema is ~160 tokens. Twenty tools like that is 3,000+ tokens of overhead on every single step of the loop. Shorter descriptions, fewer tools per call, and a stable ordering (so prompt caching works, see the KV cache post) all help.
• Pretty-printed JSON costs more than compact JSON. Indentation is tokens. Try json.dumps(obj, separators=(",", ":")) on your tool results.
• Count before you guess. Use the actual tokenizer for your model. Characters / 4 is a fine rule of thumb for English and a terrible one for Hindi, code, or emoji.
• Non-English users pay more, in money and in context. If your product serves them, measure with their text, not yours.
• Weird model failures like counting letters in "strawberry" or arithmetic on long numbers are partly tokenization: the model never saw the letters, only the chunks.

Honest limits

I tested one sample per category, with four tokenizers. It's an illustration of the effect, not a study. Also I'm only showing token counts; whether a model does well in a language depends on training data, not only on how its text is split.

If you want to try it: swap in your own strings, your own language, your own tool schemas. Ten minutes, and you will probably find something that surprises you. I did.