評測集黃金案例逐漸失效：如何安排區分能力審計

A case that can no longer fail is no longer a test. Eval suites grow by addition and almost never shrink, which is why a mature suite can sit at a 0.97 pass rate and still catch nothing you did not already know. The audit below gives every golden case a measurable ability to reject wrong answers, then schedules that measurement so rot surfaces in weeks instead of quarters. It needs no paid judge model, which matters for the scoring panel further down.

Two failure modes hide behind a green dashboard. Saturation is the gradual one: a case written when the model was weaker still passes, and now every model passes it on every run, so it burns latency and returns no signal. Reference drift is the abrupt one: the prompt contract changed, the stored reference still describes the old contract, and the case quietly penalizes correct behavior. Neither appears in a failure count, because one is permanently green and the other reads as a single stubborn row nobody re-reads.

The fix is to treat a golden case as a test of your grader, not only of your model. That means storing the case together with its own audit history, so a rate can be computed later without archaeology through CI logs. A small dataclass is enough to start.

# cases.py - every golden case carries its own audit history
import re
from dataclasses import dataclass, field
from datetime import date

@dataclass
class Case:
    id: str
    prompt: str
    reference: str
    grader: str = "contains_all"        # contains_all | regex | rubric
    added: str = field(default_factory=lambda: date.today().isoformat())
    runs: int = 0                       # live model attempts recorded
    passes: int = 0
    mutant_total: int = 0               # mutants fed to the grader
    mutant_killed: int = 0              # mutants the grader rejected
    last_audit: str | None = None

    @property
    def pass_rate(self) -> float:
        return 0.0 if self.runs == 0 else self.passes / self.runs

    @property
    def kill_rate(self) -> float:
        return 0.0 if self.mutant_total == 0 else self.mutant_killed / self.mutant_total

Each field answers one question during triage. pass_rate over the last window says whether the case still discriminates between current models. kill_rate says whether the case can discriminate at all, independently of which model happens to be in front of it. Tracking last_audit separately from added keeps age and freshness as different signals, since a three-year-old case that still kills mutants is worth more than last week's saturated one.

The core measurement is mutation testing applied to the case rather than to the code. Take the reference answer, break it in ways a plausible wrong answer would break it, and check whether the grader notices the break. A case that accepts a mutated reference never had a real requirement behind it, because it cannot reject a wrong model answer either. The generator here is deliberately crude and deterministic; label it as a helper, not an oracle.

# mutants.py - plausible wrong answers derived from a reference
def mutants(reference: str, n: int = 6) -> list[str]:
    out: list[str] = []
    for num in re.findall(r"\b\d+(?:\.\d+)?\b", reference):
        bumped = str(float(num) * 10) if "." in num else str(int(num) * 10)
        out.append(reference.replace(num, bumped, 1))        # wrong magnitude
    clauses = [c for c in re.split(r"(?<=[.;])\s+", reference) if len(c.split()) >= 4]
    for clause in clauses:
        out.append(reference.replace(clause, "", 1))         # dropped requirement
        negated = "not " + clause[0].lower() + clause[1:]
        out.append(reference.replace(clause, negated, 1))     # inverted constraint
    return out[:n]

def audit_case(case, grader) -> Case:
    for mutant in mutants(case.reference):
        case.mutant_total += 1
        if not grader(case.prompt, mutant):   # grader rejects it -> mutant killed
            case.mutant_killed += 1
    case.last_audit = date.today().isoformat()
    return case

The kill rate is more informative than the pass rate because it is model-independent. A case that every current model passes may still be sharp, as long as three of six mutants die against its grader; that case is simply solved and belongs on a watchlist rather than in the bin. A case whose mutants all survive is the dangerous one, since it will keep reporting green through every future regression while asserting almost nothing about the output. Combining the two rates into a verdict keeps retirement decisions boring and repeatable.

# verdict.py - thresholds are starting defaults to tune per suite, not measured truths
def verdict(case, min_kill: float = 0.6, window: int = 20) -> str:
    if case.runs >= window and case.pass_rate >= 0.99:
        return "saturated"      # retire or harden: solved by every model in the panel
    if case.mutant_total and case.kill_rate < min_kill:
        return "weak"           # grader accepts wrong answers: fix the assertion first
    return "healthy"

Saturation needs a panel to be meaningful, because one model's 100% is another model's 80%. This is the point where widening the panel helps and where cost usually stops people: three or four models times a few hundred cases times weekly runs adds up quickly on metered APIs. MonkeyCode's free model access is what makes a wider panel affordable for this specific job, since the audit only needs cheap batch completions and never needs your best model. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The panel result feeds straight back into runs and passes, so the saturation window stays honest across model swaps.

The second half is scheduling, and it is where a spare machine usually goes missing. A weekly job is enough for case rot, since the underlying drift moves on the order of prompt changes rather than minutes. The free server option covers exactly this shape of workload: a short cron-driven batch that idles for six days and wakes up to score mutants and replay a sample of live cases. Keep the job small, keep the output diffable, and do not let it become the suite's only execution path.

# audit a suite and keep a dated report
python audit_cases.py --cases 'cases/*.json' --mutants 6 --window 20 --out audit-2026-11-05.json

# weekly: run, then diff against the previous report
# 17 3 * * 1  cd /srv/eval && python audit_cases.py --cases 'cases/*.json' \
#   --diff latest.json --out "audit-$(date +\%F).json" && cp "audit-$(date +\%F).json" latest.json

The diff, not the report, is the deliverable. A case moving from healthy to weak means someone edited the reference without tightening the assertion, which is a five-minute fix if you catch it the same week. A case moving to saturated is a retirement candidate or a prompt for a harder variant, and both decisions are cheap when the evidence is a single line in a diff. Reviewing fifty verdict changes takes less time than reading one flaky CI log.

Three limits deserve stating plainly. Mutation kill rate is not accuracy: mutants inherit the blind spots of the reference they came from, so a cleverly wrong answer that the reference never anticipated will still slip through both. Small suites produce noisy rates, and a case with four mutants has a kill rate quantized to 0.25 steps, so require a minimum window before acting on any verdict. Judge-model graders need their own audit first, because a lenient rubric will happily kill every mutant and look healthy while doing it.

Do not adopt this if your suite is under roughly thirty cases, where individual rates carry too little information to act on. Skip it if your prompts are not versioned, because you cannot separate reference drift from prompt drift without that history. And if a single wrong answer carries legal, medical, or financial consequence, per-case human review is the control you need; a kill-rate heuristic is a triage tool, not a gate. If you want to run the loop without metering every batch call, the free model access and free server option are the part that makes a weekly cadence realistic.