如何用配對案例而非兩組平均分測試模型切換

A model swap looks safe right up until you compare the two runs case by case, because aggregate scores hide the flips that actually break your product. Two backends can both land at 71 percent on the same frozen set while nine cases move in each direction, and the nine that moved down are the only thing your users will notice. The fix is not a bigger benchmark; it is a paired test over a frozen golden set, with graders that are audited before they are trusted. This article builds that harness end to end, then shows how to read its output without fooling yourself.

Freeze the cases before you touch the model

A golden set is only useful if it is immutable during the experiment, so write the cases to disk once, hash the file, and record that hash in every report you produce. Each case needs three fields: a stable identifier, the exact prompt you will send, and a deterministic checker that returns a boolean. Keep the checkers boring on purpose, because JSON-schema validation, numeric comparison with a tolerance, and required-substring checks give you a verdict that does not drift when the judge model changes underneath you. Tag every case with a capability label such as extraction, refusal, or format, so a later drop can be localized instead of argued about.

Eighty cases is a reasonable floor for a first pass, and it is not arbitrary: paired tests only gain power from discordant pairs, so a set where both models pass everything tells you nothing no matter how large it is. Your job when curating is to include cases that sit near the decision boundary, the ones you suspect a cheaper or differently tuned model will fumble. If a case has never failed against any backend you have tried, it is a smoke test, not a regression test, and it belongs in a different file.

Audit the graders with canaries first

Before any comparison runs, feed your graders a handful of outputs you know are wrong and require them to reject every one. This is cheap insurance against the most humiliating failure mode in eval work, which is a grader that silently started passing everything after someone relaxed a regex. The canary file lives next to the golden set and is versioned with it, and a canary that survives a grader change is a stop-the-line event rather than a warning.

# graders.py — deterministic checkers plus a canary audit
import json, re

def check_json_keys(keys):
    def _check(output: str) -> bool:
        try:
            obj = json.loads(output)
        except json.JSONDecodeError:
            return False
        return all(k in obj for k in keys)
    return _check

def check_number(expected: float, tol: float = 1e-6):
    def _check(output: str) -> bool:
        m = re.search(r"-?\d+(?:\.\d+)?", output)
        return bool(m) and abs(float(m.group()) - expected) <= tol
    return _check

CANARIES = [
    ("empty_json",  check_json_keys(["id", "total"]), "{}"),
    ("prose_only",  check_json_keys(["id", "total"]), "Sure! Here is the answer."),
    ("wrong_total", check_number(41.50), "The total is 14.50"),
    ("near_miss",   check_number(41.50), "41.5 apples"),  # must PASS: tolerance case
]

def audit(canaries):
    must_fail = [c for c in canaries if c[0] != "near_miss"]
    survived = [name for name, check, out in must_fail if check(out)]
    if survived:
        raise SystemExit(f"grader audit failed, canaries passed: {survived}")
    for name, check, out in canaries:
        if name == "near_miss" and not check(out):
            raise SystemExit("grader too strict: tolerance case rejected")

That last canary matters as much as the first three, because a grader that rejects everything will happily manufacture a regression story for you. Auditing in both directions keeps the check honest, and it takes about ten minutes to write once per project.

Run both legs and record a manifest

The harness needs to send the same frozen prompts to two backends and persist raw outputs, because re-running a flaky endpoint in the middle of an analysis will quietly invalidate your paired design. Cache each response keyed by a hash of the case identifier, the prompt text, and the backend tag, and write a manifest containing the backend identifier string, the sampling parameters, and the golden-set hash. If you cannot name the exact backend version that served a request, you do not have a reproducible comparison; you have an anecdote with timestamps.

# swap_test.py — paired comparison with exact McNemar test
import json, hashlib, math, os, urllib.request
from collections import Counter

def sha(*parts: str) -> str:
    return hashlib.sha256("|".join(parts).encode()).hexdigest()[:16]

def call(endpoint: str, model_tag: str, prompt: str) -> str:
    body = json.dumps({"model": model_tag, "temperature": 0,
                       "messages": [{"role": "user", "content": prompt}]}).encode()
    req = urllib.request.Request(endpoint + "/chat/completions", data=body,
        headers={"Content-Type": "application/json",
                 "Authorization": f"Bearer {os.environ['API_KEY']}"})
    with urllib.request.urlopen(req, timeout=60) as r:
        return json.loads(r.read())["choices"][0]["message"]["content"]

def exact_mcnemar(b: int, c: int) -> float:
    n = b + c
    if n == 0:
        return 1.0
    tail = sum(math.comb(n, i) for i in range(0, min(b, c) + 1)) / 2 ** n
    return min(1.0, 2 * tail)

def run(cases, backends, cache_path="cache.jsonl"):
    cache = {}
    if os.path.exists(cache_path):
        for line in open(cache_path):
            row = json.loads(line)
            cache[row["key"]] = row
    for case in cases:
        for name, (endpoint, tag) in backends.items():
            key = sha(name, case["id"], case["prompt"], tag)
            if key not in cache:
                out = call(endpoint, tag, case["prompt"])
                cache[key] = {"key": key, "case": case["id"], "backend": name,
                              "tag": tag, "output": out,
                              "passed": bool(case<a href="out">"check"</a>)}
                with open(cache_path, "a") as f:
                    f.write(json.dumps(cache[key]) + "\n")
    return cache

def report(cases, cache, old, new):
    pairs = []
    for case in cases:
        key = lambda be: sha(be, case["id"], case["prompt"], cache_backend_tag(cache, be))
        a = next(r for r in cache.values() if r["case"] == case["id"] and r["backend"] == old)
        b = next(r for r in cache.values() if r["case"] == case["id"] and r["backend"] == new)
        pairs.append((case["id"], case.get("tag", "untagged"), a["passed"], b["passed"]))
    b_only = sum(1 for _, _, p, q in pairs if q and not p)   # new passes, old fails
    c_only = sum(1 for _, _, p, q in pairs if p and not q)   # old passes, new fails
    delta = (b_only - c_only) / len(pairs)
    flips = [(cid, tag) for cid, tag, p, q in pairs if p and not q]
    return {"n": len(pairs), "old_pass": sum(p for _, _, p, _ in pairs),
            "new_pass": sum(q for _, _, _, q in pairs), "gain": b_only,
            "loss": c_only, "delta": round(delta, 4),
            "p_value": round(exact_mcnemar(b_only, c_only), 4),
            "flips_by_tag": Counter(t for _, t in flips),
            "flip_list": [cid for cid, _ in flips]}

The cache_backend_tag helper is a two-line lookup in the cache; keep it separate so the report function stays readable. Note that the delta is computed from discordant counts only, which is why an unchanged average across the two runs can coexist with a real regression.

Read the report as a decision, not a score

The output is a small dictionary, and every field maps to an action. An illustrative shape of it looks like {"n": 96, "gain": 7, "loss": 11, "delta": -0.0417, "p_value": 0.48, "flips_by_tag": {"format": 6, "extraction": 5}}, and the numbers below are a template rather than results from anyone's system.

The concentration column is where most of the practical value sits, since eleven losses spread evenly across five capabilities look like noise while eleven losses inside format usually point to a single missing instruction. Read the flip list before you read the p-value, and read the raw outputs before you read the flip list.

Where this harness stops working

The paired design assumes each case is a Bernoulli trial under a pinned backend, so a backend that routes consecutive requests to different model versions will inflate variance and quietly break the pairing. That risk is highest with shared free tiers, which is exactly why the manifest records the returned identifier string verbatim rather than trusting the name you asked for. If you sample at any temperature above zero, run k repetitions per case and decide the case-level verdict by majority, then re-run the canary audit against the majority verdicts as well.

This is also not a load test; it says nothing about tokens per second, queueing behavior, or cost at volume, and those deserve a separate measurement with a different method. Teams that need regulatory-grade evaluation evidence, or products whose outputs have no deterministic anchor at all, should route this work to human review instead of stretching a Boolean checker past what it can support. Nor is it a replacement for product-level tracing, because a golden set only knows the failures you already imagined.

If you want to run the reference leg without paying for two full backend bills, the free model access and free server option from MonkeyCode is enough to execute the harness above on a small set. Disclosure: This article was prepared as part of MonkeyCode's product outreach. Start with twenty boundary cases, one canary file, and one pinned backend tag per leg, and let the flip list tell you whether the swap deserves the rest of your afternoon.