Favur Evals 基準測試不再只聽智能體一面之詞

Sorry this week's post is a little late. We have been heads-down shipping a new evidence feature for Favur Evals, and this post is about it.

Ask a coding agent if it is done and it says yes. Since October 19, Favur Evals no longer takes its word for it. A run now earns its score from what the harness saw happen, so a model only gets credit for work it can prove.

The agent's answer is still worth reading. It will list the tests it ran, the files it changed and the criteria it met, and most of the time that list is honest. The trouble is that the list is the agent's own account of its work, written by the same model that did the work, and nothing in it tells you which lines were seen to happen and which were only said.

Favur Evals is our public benchmark of coding-agent models, and every run on it is built inside Favur, the harness (the program that runs a team of agents) we make. Until this month a quarter of each score came from one fact, whether the run finished. We kept the measured part of the score exactly as it was and moved most of that quarter onto evidence.

What the score already measured

Three quarters of the score have always come from the work itself. Eight subjects are read from logs and analyzers. Code quality comes from lint, complexity and coverage tools. Speed comes from request latency, and cost discipline comes from cache behavior. Tool discipline comes from tool errors and failed requests. None of that depends on what an agent says about itself.

The other quarter came from the run's final status, read from two files the run leaves behind. That status answers one question well, whether the run ended in success. It cannot say what the run was asked to do at the start, which steps were seen to happen on the way, or whether each claim along the way held. Those are the questions the new 20 points answer.

A claim and an observation are different records

The idea under the whole upgrade is a distinction most agent tooling never draws. When an agent says "all six documents created", that sentence is an assertion, a claim with an author. When the harness sees a process exit with code 0, or reads a file and records its digest, the fingerprint that changes if a single byte changes, that is an observation, a fact the infrastructure recorded itself.

Both are worth keeping. An assertion tells you what the agent believed, and that is useful when you want to know why it did what it did next. An observation tells you what happened. The harness now stores the two as different kinds of record, so nothing downstream can mistake one for the other.

The board scores the observations. An agent can report anything it likes, and its report becomes one more record, labeled as an assertion and kept beside the facts. When the board decides whether a requirement was met, it reads the receipts the harness captured, such as the process that ran, the output it produced and the bytes it left on disk.

Requirements are frozen before the work starts

A receipt only means something if you know what it was supposed to prove, so every task's requirements are now written down and frozen before any agent is dispatched to do it. Favur calls the frozen set a contract, which holds the task's criteria, its required deliverables and the checks that will judge them.

A frozen contract cannot drift. If the plan changes partway through, the change creates a new revision of the contract and the old one stays on record, so a run is always judged against what was actually asked at the time it was asked. Nobody, human or model, can make a requirement easier after seeing the result.

Every criterion names its judge

A frozen criterion still needs something to judge it, so the contract also names an evaluator for it, a small versioned program that reads specific observations and returns a verdict for that one criterion.

Each evaluator does one narrow job. A process evaluator reads every final attempt of a command, with its exit, its environment and the source it ran against. A report-agreement evaluator takes the counts an agent claimed, such as tests run and tests passed, and compares them exactly against what the test parser read from the real output. A requirement-mapping evaluator follows each original requirement through design, plan, code and test.

An evaluator never writes a score and never moves the workflow forward. It writes a verdict. If it crashes or times out, the verdict is recorded as unknown, never as a quiet pass, and a failing test is recorded as measured evidence of a failure. The record also keeps who judged what, so the same model asked twice under a different prompt does not count as an independent reviewer.

Every attempt gets its own identity

The harness now gives every participant, every delegation and every tool or process attempt an identity before the work is released. The identity comes from the infrastructure, never from the agent, and a retry gets a new one rather than overwriting the first.

That sounds like bookkeeping, and it is the bookkeeping that makes everything else trustworthy. Each record names the attempt that produced it and the contract it answers to, so the chain from "this was asked" to "this was observed" to "this was accepted" can be walked link by link. In a run where different agents use different models, it also means every piece of work is tied to the model that actually did it.

How the 20 points are computed

A run's evidence score multiplies three factors.

• Capture is the share of expected records that exist and are valid. Half of it counts the run's criteria and deliverables, and half counts the activity around them, such as dispatches, steps, tool calls, processes and agents opening and closing.
• Closure is 1.0 when the run ended cleanly with nothing left open, 0.7 when it ended with open items, and 0.4 when the record stops mid-run.
• Integrity confirms that every reference points at something real and every digest matches its bytes.

Multiply the three and scale to 20. A run that meets every criterion with a recorded check, closes with nothing left open and keeps its record intact earns all 20.

The harness does most of the recording, so most of capture and all of integrity reflect the harness. The part a model controls is the part that matters to a benchmark. It decides how many of the frozen criteria reach a passing, recorded check, and whether the job finishes with nothing left open.

The record checks its own denominator

Capture is a fraction, and a fraction is only as honest as its denominator. If the list of expected records were built from what the producers announced, a record that went missing together with its announcement would shrink the list with it, and the score would look fine.

So the list of what the run should record is derived independently and frozen before any producer runs, and every arrival is checked against it. A producer cannot shrink its own denominator, which means a high capture figure says the run recorded everything that was expected of it.

The first run with a full record

We scored our first fully recorded run under the new rubric. It was a full build of a small drawing game, with 87 agents across 14 roles.

The harness wrote 9,934 records from 13 producers, and every one of them passed integrity. Every expected operation arrived, 2,438 of 2,438 operational records, from 39 dispatches and 293 workflow steps through 1,468 tool calls. All 87 agents that started were closed and tied to the model they ran on. Six contracts were frozen before any agent started, covering 29 criteria.

The operational half of capture was perfect. The other half follows each of the 29 criteria and the deliverable to a recorded verdict, and this canary build closed with some of those checks still open, so the run earned 10.7 of the 20 evidence points, points that did not exist under the old formula. The remaining points sit exactly where the record says they do, in named criteria a stronger run can close. It also lifted the score's coverage, the share of the score that rests on measured data, to 0.82. A run without a record tops out at 0.80, so a recorded run is one the board can back up line by line.

What it changes going forward

The rule applies to every run on the board. All 262 runs listed today were built before the record existed, and each now shows "no evidence recorded" in its evidence column. We chose not to fill those columns in after the fact. A record has to be written while the work happens, and reconstructing one later would be exactly the kind of claim the upgrade exists to replace.

That leaves the top 20 points of the scale open. The first recorded runs to post will be the first to claim them, and from then on two runs that finish with similar code can separate on how much of their work they can prove. A model that carries more criteria to a passing check, and closes with nothing open, will show it on the board.

It also changes what a score means. A number on Favur Evals now shows its working. It carries the subjects measured from the code, the evidence captured while the code was written, and a coverage figure that says how much of the total rests on measurement. A reader comparing two models can see not just which scored higher but how much of each score was observed.

Favur is invite-only and closed-source, while the repositories its runs produce are open. The scoring method is on Favur Evals, the product has a site of its own, and new results post on @favurdev.