LLM 的重複率可能量錯了對象
Your LLM's Repetition Rate Is Probably Measuring the Wrong Thing
作者比較 Ollama 與 llama.cpp 的本地模型輸出後指出,若把推理和答案合併計算,重複率未必反映用戶實際讀到的答案。Ollama 合併輸出的句子重複率由未調校時的 76% 降至 6%;但 llama-server 分開測量時,答案在所有條件下均為 0%,重複主要出現在推理。
I spent an evening measuring a repetition loop in a local model. The headline number said 76% of the output was repeated sentences. I tuned settings until that number fell to 6% — and only then discovered that the number had mostly been measuring the model's reasoning, not its answers.
If you benchmark repetition in local LLMs, this trap is easy to fall into. This post is about the trap itself: what a mixed output stream does to your measurements, and how to measure repetition so the number means what you think it means.
This is a companion to my earlier measured report, How I Fixed Repetition Loops in an Abliterated Local LLM, which has the full settings progression. Same experiment, different lesson.
The setup, briefly
One model (qwen3.5-abliterated:9b), one prompt (a ~1,000-character essay on Japan's four seasons), run on a Mac mini in two stacks: Ollama and llama.cpp's llama-server. I measured repetition as a sentence-level duplicate rate: split the output into sentences, and count what share of them are repeats of a sentence that already appeared.
With no anti-repetition settings, the Ollama run produced 16,291 characters and a 76% duplicate rate. After tuning (repeat_penalty 1.3 with repeat_last_n 1024), the same measurement read 6%. Case closed — except for what llama-server revealed next.
Two stacks, two different objects
llama-server returns reasoning and final content as separate streams. Ollama's ollama run output, as I used it, prints them as one combined stream. When I measured the llama.cpp runs stream by stream, the picture changed completely:
| Condition | Reasoning repetition | Answer repetition |
|---|---|---|
| No DRY | 64% (max 9×) | 0% |
| DRY multiplier 0.8 | 0% | 0% |
| DRY multiplier 0.4 | 5% | 0% |
The answer text had 0% repetition in every condition — including no protection at all. The loop I had been chasing lived in the reasoning. Which means the 76% → 6% improvement in the Ollama runs was, in large part, a measurement of a reasoning loop getting quieter — real progress, but not the progress the headline number seemed to describe.
Neither number is fake. They are numbers about different objects. Comparing them directly — "stack A repeats 6%, stack B repeats 0%" — would have been a false conclusion drawn from true measurements.
A short checklist for measuring repetition
What I now do before trusting any repetition number, mine included:
- Find out what your client actually returns. Does your stack expose reasoning and content separately, or does it merge them into one stream? If it merges them, your repetition rate is a property of the mixture, not of the answer your users read.
- Measure the streams separately when you can. If the answer is clean and the reasoning loops, your problem is budget and latency (reasoning ate 16k characters of context in my first run), not answer quality. Those need different fixes.
- State what you measured. "76% sentence duplication in the combined stream" is an honest claim. "The model repeats 76% of the time" is not.
- Watch the side effects on the stream you care about. In my runs, strong DRY settings cleaned up reasoning but made the answer less natural — overused quotation marks went from 1.3 to 52.3 per 1,000 characters at multiplier 0.8. A single repetition number would have hidden that cost completely.
- Mind the context budget interaction. At ctx 8192, one run's reasoning consumed the entire budget and the answer came back empty. A "0% repetition, empty answer" run is a failure your repetition metric will happily score as perfect.
The honest limits
Everything above is one run per condition, one model, one prompt — sampling randomness is not separated from the effects, and I say so in the full report too. But the methodological point doesn't depend on sample size: a merged stream and a separated stream are different measurements, and no amount of re-running fixes comparing them as if they were the same.
If you measure repetition in your own stack, I'd be curious what your client returns — merged or separated — and whether your loops also live in the reasoning.
The full measured note from these experiments (complete settings progression, exact configurations, side-effect counts): on Gumroad. The settings-focused companion article is linked at the top.
來源:DEV Community · dev.to