同一審查器、三個模型:差異來自抽樣還是模型選擇?
Same reviewer, three models: is the noise sampling or model choice?
研究者以相同流程,在 10 個 CVE 案例上測試 agnes-3.0-flash、agnes-2.5-pro 和 sensenova-ds,發現三個模型分別命中 10/10、9/10 和 9/10。兩個漏報案例沒有重疊,但每格只測試一次,命中率仍不能視為穩定比率。跨模型的風險分數中位差為 30,同一模型跨次測試則為 21。
The multi-pass experiment ended with a confession: nineteen commits on one model cannot tell you whether the ±21-point cross-pass spread we measured is a property of that model, of the provider, or of the task. The next cell we could afford was the 10-CVE suite — the same ten revert-diffs from the original benchmark — run on three models with everything else held constant.
The design, including the flag we had to build
Three models, same pipeline (two agents, same prompts, same reconciliation code), one pass per cell:
- `agnes-3.0-flash` — the model behind every number we have published so far
- `agnes-2.5-pro` — same provider, a reasoning model
- `sensenova-ds` (DeepSeek V4.1 via a free gateway) — a different provider entirely
One design problem surfaced immediately. Our CLI silently rotates to a smaller model when the primary hits a rate limit — the right behavior for production uptime, and exactly wrong for a benchmark, where a substituted model's answer pollutes the cell it lands in. (It polluted our *first* benchmark that way, before we knew to look.) So we built `--fallback none`: the named model's answer, or a recorded failure. Never a substitution.
Results: every model missed a different case
Hit = the report names the core vulnerability the revert reintroduced (judged on finding title, description, and anchored file path; the three disagreement cases were verified manually). Raw JSON for all thirty runs is in the repository under `benchmark-runs/2026-11-21-3model/`.
| case | agnes-3.0-flash | agnes-2.5-pro | sensenova-ds |
|---|---|---|---|
| c01 SSRF / local file read | hit | hit | hit |
| c02 password hash disclosure | hit | hit | hit |
| c03 empty vs missing headers | hit | hit | hit |
| c04 unescaped log messages | hit | hit | hit |
| c05 unbounded multipart memory | hit | hit | hit |
| c06 forged formset actions | hit | hit | hit |
| c07 translation parser memory | hit | hit | miss |
| c08 unvalidated URL rendering | hit | hit | hit |
| c09 axios SSRF | hit | miss | hit |
| c10 template command injection | hit | hit | hit |
| total | 10/10 | 9/10 | 9/10 |
The headline is not that one model won. It is that **the two misses do not overlap**. `agnes-2.5-pro` returned risk 0 and zero findings on c09 — silent on a reintroduced SSRF. `sensenova-ds` missed c07 entirely. No single model is strictly better; an ensemble voting 2-of-3 would have caught all ten. We are not claiming that as a feature — it is a hypothesis the next cell has to earn.
The case that flipped
The original benchmark reported agnes-3.0-flash at 8/10, missing c02 (password hash disclosure). Today, the same model on the same revert returns a CRITICAL anchored at `django/contrib/admin/utils.py:435` describing exactly that vulnerability.
Attribution honesty: we cannot credit a single change. Since that first run the security prompt gained the evidence-cap rubric, findings gained real file anchoring, and severity reconciliation moved from prose to code. Anchoring is post-processing, so it did not make the model *see* the bug — but prompt changes and plain sampling noise are both live candidates, and one run cannot separate them. What we can say: the published 8/10 is no longer the current number, and the published number will be re-baselined multi-pass before anyone treats either as a rate.
The fail-closed gate met real data
On `sensenova-ds`, the free tier's TPM limit kept killing the LogicAgent — in 6 of 10 runs. Under the old pipeline, those runs would have shipped a verdict computed from half the checks. Instead, every one of them carries a withheld risk score: findings listed, verdict refused, reason in the summary. Six withheld reports are the clearest demonstration we have that the gate does its job on real provider failures — a partial review cannot masquerade as a clean one.
Practical read: the free tier is workable for *evaluation* (the SecurityAgent completed all ten cases and found 9/10 vulnerabilities), but production-shaped reviews need a provider that finishes what it starts. That is why our BYOK table still puts paid DeepSeek in the recommended slot.
Which noise is bigger?
Risk-score spread, median (raw JSON behind every number):
- Across three models, same case, same pass: **median 30** (n=3 cases where all three models scored; the withheld runs are excluded, not zeroed)
- Across passes, same model, same case: **median 21** (n=19 commits, the multi-pass cell)
- Across two models of the same provider, same case: **median 14** (n=9)
Reading it: crossing providers moves the verdict about as much as re-rolling the same model. Swapping models inside one provider moves it less than either. Neither number is a substitute for the other — **model choice does not buy you stability, and multi-pass does not buy you a model that sees c09**. If your CI gates on a single-run score from a single model, you are stacking both noise sources and calling the result a verdict.
One correction in our own tooling
Our first automated pass of the hit matrix scored one case wrong: the matching script compared keywords against finding titles and descriptions but not anchored file paths, so a correct critical on `django/utils/translation/trans_real.py` read as a miss. We caught it while verifying the disagreement cases, fixed the script, and re-ran the matrix before publishing anything. We mention it because it is the same failure mode we criticize in vendor benchmarks — an eval script that quietly mislabels results — and it happened to us on a Saturday afternoon with the publish button in reach.
Honest limits
- One pass per cell. The multi-pass post showed single-run numbers carry ±21 points of sampling noise; these hit rates are one sample, not a rate. The 10/10 will get a multi-pass re-baseline before we print it on the benchmark page.
- Hit judgment is ours, on named findings, verified manually only for the disagreement cases.
- CVE-diff shapes: these are reverts of real security fixes — well-formed, security-flavored diffs. Ordinary merged PRs behave differently, as the noise post showed.
- `agnes-2.5-pro` is a reasoning model: slower per call, and its behavior under the pipeline differs in ways a single cell does not fully expose.
Open items from the comments, in order: the 100-commit double-labelled run, and a multi-pass re-baseline of this cell. The harness, all thirty raw runs, and the corrected analysis script are public. If you can break the matrix, we want the counterexample.
來源:DEV Community · dev.to