評測分數不等於測量結果：六個可重現性陷阱

Your eval score moved from 0.81 to 0.84. You shipped the change. Next week the old baseline won again.

Was that improvement real? Or was it noise wearing a decimal point?

I keep meeting the same pattern in agent work. A single number gets treated as a measurement. It is usually an anecdote with a decimal.

Here are six myths I keep running into, plus a small harness that reports the number people actually need: how often an item flips.

Myth 1: Same prompt, same model, same output

Hosted inference is not a function. It is a service with batching, sampling, and version drift.

Temperature zero reduces variance. It does not delete variance. Batching and backend changes still move tokens around.

The corrected mental model: every call is a sample. One call is a sample of size one.

So what does that make your score? A sample mean of an unknown distribution.

Myth 2: The mean score summarizes the run

Means hide flips. Two runs can both score 0.80 and disagree on completely different items.

Ask a sharper question instead. How many items changed verdict between runs?

That number is a flip rate. It tells you whether your delta is signal or jitter.

A mean answers "how did we do?" A flip rate answers "can I trust the diff?" You need the second one to ship.

Myth 3: The grader is a constant

If an LLM grades your outputs, the grader is also a sampled system. Rubric wording moves it. Item order moves it. Context length moves it.

So measure the judge too. Feed the same saved outputs through it twice. If the judge flips, your score is a coin toss with extra steps.

Prefer exact checks wherever possible:

1. String or normalized string match for short answers.
2. Schema validation with jsonschema for structured output.
3. Unit tests around a parsed artifact, not around raw prose.
4. Diff assertions against a known-good file.

Keep the LLM judge for the cases nothing else can grade. Then report judge agreement next to the score.

Myth 4: Caching and dedupe are invisible

Prompt caching speeds things up. It also changes what the model actually sees.

Deduplication shrinks your item set. Your denominator changes. Your score moves for free.

Record the item count and a hash of the input file in every report. If the hash changed, the comparison is not a comparison.

Myth 5: The dataset is frozen because it is a file

Files get edited. Items get added. Ordering changes. Someone quietly fixes a gold label.

Pin the dataset by hash, not by path. items.jsonl is a location. A SHA-256 is an identity.

When a score jumps, check the dataset hash first. Then check the prompt hash. Then argue about the model.

Myth 6: Crossing a threshold means improvement

A 0.03 delta inside a 0.12 noise band is decoration, not progress.

Set the gate from observed variance, not from hope. Run the harness a few times before you trust any threshold.

And be honest about what you are comparing. A new prompt against an old score is a diff of two different weeks, not two different prompts.

The artifact: a flip-rate harness

The script below runs each item several times and reports per-item flip rate. It is illustrative, not a benchmark. Point it at your own items and endpoint.

# flip_rate.py - illustrative harness. Not a benchmark. Bring your own items.
import hashlib, json, os, statistics, urllib.request

BASE  = os.environ['MODEL_BASE_URL'].rstrip('/')
KEY   = os.environ['MODEL_API_KEY']
MODEL = os.environ['MODEL_NAME']
RUNS  = int(os.environ.get('RUNS', '5'))
TEMP  = float(os.environ.get('TEMPERATURE', '0'))

def sha(text: str) -> str:
    return hashlib.sha256(text.encode()).hexdigest()[:12]

def ask(item, seed):
    body = json.dumps({
        'model': MODEL,
        'messages': item['messages'],
        'temperature': TEMP,
        'seed': seed,
    }).encode()
    req = urllib.request.Request(
        BASE + '/chat/completions',
        data=body,
        headers={'Authorization': 'Bearer ' + KEY,
                 'Content-Type': 'application/json'})
    with urllib.request.urlopen(req, timeout=120) as r:
        return json.load(r)['choices'][0]['message']['content']

def normalize(text: str) -> str:
    # Keep the verdict, drop whitespace and case noise.
    return ' '.join(text.strip().lower().split())

def flip_rate(outputs):
    counts = {}
    for o in outputs:
        key = normalize(o)
        counts[key] = counts.get(key, 0) + 1
    return 1 - max(counts.values()) / len(outputs)

items = [json.loads(line) for line in open('items.jsonl')]
report, worst = [], 0.0

for item in items:
    outs = [ask(item, seed=1000 + i) for i in range(RUNS)]
    fr = flip_rate(outs)
    worst = max(worst, fr)
    report.append({
        'id': item['id'],
        'flip_rate': round(fr, 2),
        'distinct_outputs': len({normalize(o) for o in outs}),
        'sample_hash': sha(outs[0]),
    })

manifest = {
    'model': MODEL,
    'temperature': TEMP,
    'runs': RUNS,
    'items_hash': sha(open('items.jsonl').read()),
    'mean_flip_rate': round(
        statistics.mean(r['flip_rate'] for r in report), 3),
}

print(json.dumps({'manifest': manifest, 'items': report}, indent=2))
raise SystemExit(1 if worst > float(os.environ.get('MAX_FLIP', '0.2')) else 0)

Run it like this in CI:

pip install --quiet jsonschema
RUNS=5 MAX_FLIP=0.2 python flip_rate.py > eval_report.json

Two notes on the code. seed is a hint, not a contract; providers treat it differently, so verify in your own docs. And the script exits non-zero on a bad item, which makes it a gate instead of a dashboard nobody reads.

Reading the output

These bands are a starting heuristic, not measured constants. Calibrate them on your own set before you enforce anything.

Where a free endpoint and a free server fit

This harness is repetitive by design. It wants several calls per item and a place to run them.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The operator describes free model access and a free server option. Those two things are genuinely useful here, because flip-rate measurement burns calls and wall-clock time for no user-visible feature.

A free endpoint lets you take the first pass across more items. A free server lets the loop run somewhere other than your laptop lid. I cannot speak to quotas, limits, or longevity, and you should check current terms before you build a pipeline on it.

Point the harness at whatever endpoint you have. The script does not care who answers.

Limitations, and who should skip this

Be clear about what this does not do.

• Flip rate is not correctness. A stably wrong answer has a flip rate of zero.
• Small item sets give noisy flip rates. Ten items cannot support four decimal places.
• Repeated runs cost money, time, and rate-limit budget on paid endpoints.
• A judge that agrees with itself twice is not necessarily right.
• On a 429, stop and back off. Do not wrap the harness in a hot retry loop.

Skip this approach if you have fewer than about ten items. Skip it for smoke tests where any answer is acceptable. Skip it for one-shot creative tasks with no ground truth to compare.

Use it when a prompt change needs a decision, and the only evidence is a number that keeps wobbling.

The question to ask next time

Before you cite the next score, ask one thing. If I re-ran this today, how many items would change?

If you cannot answer that, you have a decimal point, not a measurement.