LSCBench 提供 2,700 條智能體執行軌跡，作者重算排行榜並發現一項差異
A benchmark shipped 2,700 agent trajectories. I recomputed its leaderboard.
作者重算 LSCBench 的 2,700 條執行軌跡後，發現結果與論文大致相符，只有 Kimi-K3 的功能通過率不同：資料目錄為 92.2%（415/450），論文為 91.3%，相差四項任務。
Most eval papers give you a table and a PDF. LSCBench gives you the runs.
LSCBench is a secure-code benchmark out of NUS and the Qwen team: 150 real vulnerabilities cut into 450 tasks across seven languages, at three granularities (fill in a function, write a whole file, rebuild a repo's source tree). The task spec never mentions security. Hidden functional and security suites grade the result. Six frontier agents run it under Claude Code or Codex, with a 500-turn cap, interactive tools off, and no network access.
The headline number is that the best model, Claude Fable 5, produces code that is both functional and secure on 25.1% of tasks. Functional completion alone runs 73.6% to 91.3%. In other words, the code mostly works, and mostly keeps the vulnerability.
What got me to actually open the thing: the repo ships the traces. alibaba/last-secure-code-benchmark has a traces/ directory with 2,700 trace JSONs, 450 per model, plus a catalog.json that carries the verdict for every single run. So instead of trusting the table, I recomputed it.
The recompute
I pulled catalog.json and counted, per model over the 450 tasks: functional, secure, and both. Here is my number next to the paper's Table 10.
| Model | FuncPass (mine) | Func ∧ Sec (mine) | Func ∧ Sec (paper) |
|---|---|---|---|
| Claude Fable 5 | 91.1% | 25.1% | 25.1% |
| GPT-5.6-Sol | 89.1% | 24.7% | 24.7% |
| Qwen3.8-Max | 89.3% | 22.0% | 22.0% |
| GLM-5.3 | 87.1% | 21.3% | 21.3% |
| DeepSeek-V4-Pro | 73.6% | 18.0% | 18.0% |
| Kimi-K3 | 92.2% | 16.4% | 16.4% |
Five rows match to the decimal. The scenario-level column matches too: I get 12.0% for Claude Fable 5 on scenarios secured at all three granularities, which is exactly what the paper reports. The pooled union matches as well, 198 of 450 tasks secured by at least one model (44.0%), 19 tasks secured by all six, 62 secured by exactly one.
One row does not match. Kimi-K3's FuncPass is 92.2% in the catalog (415 of 450) and 91.3% in the paper. That is a four-task gap. It changes no ranking and no conclusion. It is also the only difference I found across 2,700 runs.
That is a good result for the benchmark, and it is the reason the shipped artifact matters at all.
Why the traces matter more than the table
A number you cannot recompute is a claim. A number you can recompute is a measurement. The difference shows up the moment someone else's run disagrees with yours, which is exactly when you need raw outcomes instead of a summary.
I have written before about why offline benchmark scores and online review performance diverge, and the mechanism is always the same: the summary drops the conditions. Shipping per-run outcomes is the cheap fix. It costs a tarball and a catalog file. Compare that to benchmarks where the headline claim rots in months because nobody can re-derive it.
The four-task gap is small enough to be version skew, or a difference in how a missing verdict gets counted. I cannot tell which from outside. That is the point. With the traces shipped, I can at least see that there is a disagreement. With only the table, I would have no way to know one existed.
The reward field is one bit, and it hides two different failures
Every trace carries reward, and it is 1 only when the run is both functional and secure. So a run that rebuilt the module and kept the vulnerability intact scores the same 0 as a run that never produced a compiling build.
Those are different events. One is a security miss, the other is a capability miss, and they call for different responses from anyone reading the results. The paper is careful about this and reports FuncPass, SecPass and Func ∧ Sec separately. The shipped reward column collapses all three back into one bit.
Concretely: 574 of 2,700 runs scored reward 1. If you filter traces by reward to find "the agent gave up", you get the wrong set, because giving up and shipping the bug are indistinguishable there. Sort by reward and you cannot rank a security regression against a build regression. You have to go back to the two booleans.
120 runs ended on a harness error, and 15 of them still passed
The catalog also records exit_reason, which is the field I did not expect and the one I ended up caring about most.
| exit_reason | runs | of which reward 1 |
|---|---|---|
| (none) | 2,580 | 559 |
| NonZeroAgentExitCodeError | 41 | 1 |
| AgentMaxTurnsError | 32 | 6 |
| ApiRateLimitError | 25 | 5 |
| AgentTimeoutError | 16 | 3 |
| VerifierTimeoutError | 5 | 0 |
| AgentProcessSignalError | 1 | 0 |
120 of 2,700 runs, 4.4%, ended on something other than the agent deciding it was done. Six hit the 500-turn cap. Twenty-five died on an API rate limit. Sixteen timed out. Five failed inside the verifier itself, which is not the agent's fault at all.
Now the part that matters. Fifteen of those errored runs still scored reward 1. A run that hit the turn cap or died on a rate limit can still have left a correct, secure implementation behind, so the pass is real. But 2.6% of all passes in the dataset come from a run that never cleanly finished. If you are ranking models on a difference of a couple of points, that is the same order of magnitude as your margin.
The five verifier timeouts are the cleanest example. They are recorded as functional: false, secure: false, which is the same treatment the paper gives a missing verdict by design ("counted as failures when missing"). That is a defensible convention. It is also a convention that turns an infrastructure failure into a model failure, and you can only see it because the traces are there.
What I would check before quoting any of it
Four things, and each takes minutes when the traces ship:
- Count the rows before you count the scores. 450 tasks times six models is 2,700. If the catalog is short, your denominators are wrong.
- Recompute
functionalandsecureseparately rather than using the collapsed reward. The reward bit merges a security miss with a build miss. - Read
exit_reasonon every run and see how many passes came from a run that errored out. If a model's lead is smaller than its count of errored-but-passed runs, you do not have a ranking yet. - Check the scenario-level column, not just the task-level one. Securing a vulnerability at one granularity is no evidence the model understood it. Claude Fable 5 goes from 25.1% on tasks to 12.0% on scenarios, and that gap is the honest measure of consistency.
The takeaway
When a benchmark ships its traces, recompute it before you cite it. You will usually find that the paper is right, and occasionally you will find the one row that is not. Both are more useful than trusting a table, and the second one is only findable if the artifact exists.
If you maintain an eval, the cheapest credibility you can buy is publishing the per-run outcomes next to the summary, including the failure reasons. If you consume one, the cheapest sanity check is counting the rows, splitting the reward bit, and reading the exit reasons. That is three filters between you and a number that sounds better than it is.
來源：DEV Community · dev.to