LLM token 計數器的兩項錯誤與修正方法

My token counter had two bugs. The first one made it dead. The second one made it plausible.

Only the second one was dangerous.

Bug one: the page looked perfect and the tool never ran

The counter is a single inline <script> inside a WordPress wp:html block. I saved it through the REST API. Everything worked locally.

In production, the input box rendered, the styling rendered, the FAQ rendered, and typing did absolutely nothing.

Here is why. WordPress runs wptexturize over post content before it hits the browser. The database held the literal characters &&. The rendered HTML held && — and then, because of a second pass, &&. Inside <script>, HTML entities are not decoded. So this:

if (e && e.message) { show(e.message); }

was served to the browser as this:

if (e && e.message) { show(e.message); }

which is a syntax error. The whole IIFE failed to parse, so nothing initialized. No console check, no smoke test, no screenshot of mine caught it, because the screenshot looked identical to a working page.

The fix was boring and permanent: no bare & in inline JavaScript on this site. Nested if instead of &&, aliases instead of || where it matters, String.fromCharCode() if I ever truly need the byte. And one deploy check that costs four seconds:

curl -s https://aisubtools.xyz/llm-token-counter/ | grep -c '&'

If that prints anything other than 0, the deploy is broken. I also grep for a sentinel string that only exists after the tool boots, because "the HTML is valid" and "the script ran" are two different claims.

Bug two: the fallback number

The real tokenizer is a lazy import(). If that fails — offline, blocked CDN, a browser that refuses the module — the old code did what every token estimator does:

var guess = Math.ceil(chars / 4);
$("tk-tokens").textContent = "~" + fmt(guess);

A tilde. One character of honesty in front of a number that is wrong often enough to matter.

I only found out how wrong by measuring it. I took 10,000 real prompts from a corpus of dev.to article bodies and tokenized every one:

• The batch looked fine. 2,715,542 characters, 646,092 tokens, 4.203 characters per token. Against the assumed 4.0, the whole batch was off by 5.08%. If you are paying one invoice for ten thousand calls, chars / 4 is a perfectly good number.
• The rows were not fine. Median per prompt: 4.469 characters per token, median bias +11.72%. And 40.5% of individual prompts were off by more than 20%, spanning -23.9% at p5 to +46.8% at p95.
• Short inputs were the worst. Under 200 characters, 43.6% were off by more than 20%. Over 1500 characters, only 4.8% were.
• Content type moved more than any other single variable. Prose 4.58, code 4.16, structured docs (think OpenAPI, JSON configs) 3.78. That is an 18% swing between two things people happily paste into the same box.

So the same estimator that is accurate to 5% on the monthly bill is wrong by a fifth on four out of ten individual requests. A single user does not pay the batch. A single user pays their row.

The diff

Three changes, in order of how much they matter.

1. Never print a number you did not compute. The fallback still runs, but it now says what it is:

$("tk-heur").textContent =
  "Tokenizer unavailable offline-like fallback is active: showing the four-characters-per-token estimate only. Reload once you are back online for the exact count.";

An estimate that announces itself is a tool. An estimate wearing a count's clothes is a bug.

2. Show the disagreement. When the real tokenizer loads, the tool computes the naive guess alongside it and prints the delta:

var guess = Math.ceil(chars / 4);
var diff  = guess ? Math.round((n - guess) / guess * 100) : 0;

Now the heuristic is not hidden behind the answer — it is sitting next to it, and on a structured payload you can watch it miss by 20% in the browser.

3. If you must estimate, stop using one constant. 4.58 for prose, 4.16 for code, 3.78 for structured text. Three numbers instead of one, and the error band on the thing you actually paste drops immediately. This is also the cheapest possible check on anyone else's cost calculator: paste a paragraph, paste a JSON config, and if both come back at the same chars-per-token, the calculator is guessing.

One more thing that bit me and will bite you: if your pipeline base64-encodes payloads before they reach the model, that step inflates the byte count by about 33% before tokenization ever happens. The estimator is then wrong on top of being wrong. Decode and measure the real thing.

The part that generalizes

Silent failure has two shapes.

The first shape is dead: script fails to parse, tool never boots, page looks finished. A smoke test catches it. One curl | grep catches it. Mine didn't exist, which is why the first bug shipped.

The second shape is plausible: the number is computed, formatted, and wrong by 20% on four out of ten rows. No smoke test catches this. No type check catches it. The only thing that catches it is a ground truth you went and got — 10,000 real prompts in my case — and a habit of grading per row instead of per batch.

Every estimator I have shipped since gets the same two questions: what does it print when it fails, and what does it print when it is merely unsure. If the answer to either is "a number," it is bug two waiting to happen.

The counter is at aisubtools.xyz/llm-token-counter — it runs the real vocabularies in your browser, and it shows you the naive estimate beside the exact count so you can see the gap yourself.

If you have a cost estimator in production, run this on it: take 100 of your real prompts, tokenize them properly, and count how many are off by more than 20%. My guess is you will land near 40, and you will not have known until you looked.