基準評測要固定數據集，再檢驗評分者一致性

Most benchmark scores arrive as a single number, yet they rest on two artifacts almost nobody publishes: the exact bytes of the task set and the agreement between the graders assigning labels. When either artifact moves, the score moves, and the movement looks exactly like progress. The remedy is procedural rather than statistical: pin the dataset and rubric by hash before the first run, pre-register a minimum agreement threshold, and treat any drift as a failed run instead of a footnote.

The first artifact is a hash, not a score

A benchmark is a repository of decisions, and like any repository it needs a stable commit identifier before comparison becomes meaningful. Hashing the task file, the rubric, and the judge prompt costs one command and removes a whole class of accidental retreat. Line endings, JSON key order, and a quietly reworded rubric sentence all shift the input distribution, and none of them announce themselves in a results table.

sha256sum tasks.jsonl rubric.md prompts/system.txt | tee prereg.hashes
wc -l tasks.jsonl

That transcript belongs in the artifact bundle beside the scores, because a reproducing reader must be able to prove the run started from the same bytes. A number without a hash is a claim about a file that may no longer exist in that form.

Write the pre-registration file first

Pre-registration gets dismissed as academic ceremony, but its practical job here is narrow and useful: it makes the re-run decision someone else's to audit. The file below freezes the inputs, names the two graders, sets the agreement floor, and lists the slices that will be reported separately. Fill the placeholder hashes from the command above, then never edit the file mid-experiment.

{
  "experiment": "agent-pass-rate-v3",
  "frozen_at": "2026-10-26T00:00:00Z",
  "dataset": { "path": "tasks.jsonl", "sha256": "REPLACE_ME" },
  "rubric": { "path": "rubric.md", "sha256": "REPLACE_ME" },
  "graders": ["human-a", "model-b"],
  "min_kappa": 0.6,
  "slices": ["refusal", "multi_step", "tool_error"],
  "exclusions": "none after freeze; ambiguity is recorded, not deleted",
  "metric": "pass = grader label 'pass'; reported per slice, never pooled alone"
}

The exclusions field is where honest and motivated analysts diverge, since declaring a task ambiguous after seeing it fail is a re-run wearing a lab coat. Writing "none after freeze" in advance forces ambiguity into a recorded note rather than a silent deletion, which keeps the denominator stable across iterations.

A harness that refuses to run on drift

The following script uses only the Python standard library, makes zero network calls, and exits non-zero the moment a frozen input changes. Exit code 2 means hash drift and exit code 3 means the graders disagreed more than the pre-registered floor allowed. Both failures are informative, and both are cheaper to read than a suspiciously tidy score.

#!/usr/bin/env python3
"""Refuse to score when the frozen inputs moved. stdlib only, Python 3.10+."""
import argparse, hashlib, json, platform, sys, time
from pathlib import Path


def sha256(path: Path) -> str:
    digest = hashlib.sha256()
    with path.open('rb') as handle:
        for block in iter(lambda: handle.read(65536), b''):
            digest.update(block)
    return digest.hexdigest()


def load_labels(path: Path) -> dict:
    labels = {}
    for line in path.read_text(encoding='utf-8').splitlines():
        if line.strip():
            row = json.loads(line)
            labels[row['id']] = row['label']
    return labels


def cohen_kappa(first: dict, second: dict):
    shared = sorted(set(first) & set(second))
    if not shared:
        raise SystemExit('no overlapping task ids between graders')
    n = len(shared)
    observed = sum(first[k] == second[k] for k in shared) / n
    labels = sorted({first[k] for k in shared} | {second[k] for k in shared})
    chance = 0.0
    for label in labels:
        p_first = sum(first[k] == label for k in shared) / n
        p_second = sum(second[k] == label for k in shared) / n
        chance += p_first * p_second
    kappa = 1.0 if chance == 1.0 else (observed - chance) / (1.0 - chance)
    return kappa, n


def main() -> int:
    parser = argparse.ArgumentParser()
    parser.add_argument('--prereg', type=Path, required=True)
    parser.add_argument('--labels', type=Path, nargs=2, required=True)
    args = parser.parse_args()

    prereg = json.loads(args.prereg.read_text(encoding='utf-8'))
    drift = []
    for key in ('dataset', 'rubric'):
        expected = prereg[key]['sha256']
        found = sha256(Path(prereg[key]['path']))
        if found != expected:
            drift.append(key + ': expected ' + expected[:12] + ', found ' + found[:12])
    if drift:
        print('FAIL: frozen inputs changed\n' + '\n'.join(drift), file=sys.stderr)
        return 2

    first, second = (load_labels(path) for path in args.labels)
    kappa, n = cohen_kappa(first, second)
    manifest = {
        'experiment': prereg['experiment'],
        'generated_at': time.strftime('%Y-%m-%dT%H:%M:%SZ', time.gmtime()),
        'python': platform.python_version(),
        'hashes': {key: prereg[key]['sha256'] for key in ('dataset', 'rubric')},
        'items_compared': n,
        'kappa': round(kappa, 4),
        'min_kappa': prereg['min_kappa'],
        'verdict': 'pass' if kappa >= prereg['min_kappa'] else 'fail',
    }
    print(json.dumps(manifest, indent=2))
    return 0 if manifest['verdict'] == 'pass' else 3


if __name__ == '__main__':
    sys.exit(main())

Label files are plain JSON Lines, one decision per task, which keeps diffs readable in review:

{"id": "t-014", "label": "pass"}
{"id": "t-015", "label": "refuse"}

A full pass looks like this, and the printed manifest is the thing worth archiving:

python3 harness.py --prereg prereg.json --labels labels/human_a.jsonl labels/model_b.jsonl
echo $?   # 0 pass, 2 hash drift, 3 agreement below floor

Agreement is the control, not a headline

Raw percent agreement flatters benchmark graders because most task sets are skewed toward success. If both graders label ninety percent of items as passing, their chance agreement is already 0.82, so ninety percent observed agreement collapses to a kappa near 0.44. That gap between a reassuring percentage and a mediocre kappa is the entire reason the metric belongs in the harness rather than in a spreadsheet note.

The conventional reading treats values above roughly 0.6 as substantial and above roughly 0.8 as strong, though the thresholds are conventions rather than laws of nature. Below thirty items the estimate itself is noisy, so a small pilot set should be described as a pilot rather than promoted into evidence. When kappa lands low, the productive responses are tightening rubric wording, adding anchor examples for borderline cases, and reporting slices separately so a stable category is not averaged away by a contested one.

Where a free tier helps, and where it cannot

The economics of this design are the two independent grading passes, because a single model grading its own family of outputs measures style consistency more than correctness. MonkeyCode exposes free model access and a free server option, which lowers the cost of running a second grader pass and of hosting the harness plus its manifests at a stable URL. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The operator currently advertises a free token allotment alongside the free server, and readers should confirm the present size, expiry, and rate limits on the project's own page, since free-tier terms change faster than articles do.

What the free tier cannot do is fix a method that rewards re-running until the interval looks narrow. Pre-registration blocks that path by making the threshold older than the result, and the hash check blocks the quieter path of trimming awkward tasks. If a team wants a place to run the two grader passes without a card, the free model access and free server are a reasonable on-ramp; bring the harness and the frozen hashes, not a hope for a better number.

Limitations and who should skip this

This workflow assumes a task set that can be frozen and a rubric that two graders can read without a meeting, and it degrades when either assumption fails. Open-ended creative evaluation, single-annotator shops with no second grader, and teams whose datasets change daily should use a lighter scheme and say so plainly rather than borrow the vocabulary of rigor. Anyone reporting a score publicly without the manifest is better served by publishing the limitations paragraph alone, because a number with an unknown denominator invites the wrong kind of comparison.

The harness above is short enough to run before lunch, and a published manifest is a more durable credibility signal than any headline figure. Pin the bytes, grade the grader, then report the score with its exit code attached.