cap-evolve 在 SpreadsheetBench 超越已發表最佳結果，Gemma-4-31B 由 43.5% 升至 76.8%，模型權重不變

cap-evolve is an open-source framework that makes an AI agent better by rewriting what the agent reads, not its weights. On SpreadsheetBench it lifts Gemma-4-31B-It from 43.5% to **76.8%* of held-out tasks, above the best published result of 68.0%. It lifts GPT-5.5 from 38.6% to 87.6%, above SkillOpt's 80.7%. This post shows how, and how we made sure the comparison is fair.*

cap-evolve · website · GitHub · getting started · results

You bring your agent and your evaluation. A coding agent such as Claude Code reads the failed traces, proposes an edit to the prompts, tools or skills, and the edit is kept only if it beats a held-out split by a significant margin. The test split is sealed and scored once.

The short version

We gave Gemma-4-31B-It, an open-weight model, spreadsheet tasks from SpreadsheetBench (verified 400). With no help it solves 43.5% of the 280 held-out test tasks. After cap-evolve rewrote the instructions the model receives, it solves 76.8%. That is the mean of three independent runs, and the three runs are within 2.5 points of each other.

The WikiSkill paper from Google Research (arXiv 2608.27454) reports 68.0% for the same model, benchmark, split and metric. That is the best of the five methods in its Table 1, so cap-evolve beats the state of the art on this benchmark by 8.8 points, from a lower starting point.

The gain is the test score minus the no-skill score, the same way WikiSkill computes its +19.7. We start lower than WikiSkill (43.5 against 48.3) and end higher. Even in WikiSkill's own setting, where Gemma writes its own skill, cap-evolve scores 71.2 against 68.0, and its gain is +27.7 against +19.7.

WikiSkill's skills are self-evolved: Gemma writes its own skill. So we also ran that setting, with Gemma as both the model being taught and the optimizer. At first the repeats spread from 55.8 to 73.0. The optimizer was deleting the rules the runtime depends on. After we sealed those rules away from the optimizer, three repeats scored 72.5, 71.2 and 69.9. The section "Gemma as its own optimizer" tells that story.

We also ran GPT-5.5, where the reference is the SkillOpt paper (Microsoft and others, arXiv 2605.23904). SkillOpt reports 41.8 → 80.7 in its "direct chat" setting. We got 38.6 → 87.6, the mean of three runs. This comparison is weaker than the Gemma one, for reasons we explain in the details part. The main reason is that most of GPT-5.5's gain comes from a single lesson: write values, not formulas.

What "skill evolution" means

A skill here is plain text: instructions the model receives with every task. Think of it as the onboarding document a new colleague reads before starting the job.

Skill evolution is a loop that writes and improves that document automatically:

The model's weights never change. Only the text changes. The final number is measured once, on test tasks the loop never saw.

WikiSkill, SkillOpt, EvoSkill, Trace2Skill and cap-evolve all follow this loop. They differ in who proposes the new text, what the proposer is shown, and what it may edit.

What we ran

Each configuration ran three times, independently. Every run started from the same empty skill.

• Agent: Gemma-4-31B-It, the model being taught. It works in a Python sandbox with up to 30 turns per task.
• Optimizer: Claude Opus 5, running as a coding agent (Claude Code). It can read files, search the agent's trajectories and edit the skill.
• Algorithm: hill-climbing for 5 rounds. Each round proposes one new skill and scores it on the 40 validation tasks. The new skill is kept only if its task-by-task improvement clears a noise threshold.
• Final score: the kept skill (the "champion") is scored once on the 280 sealed test tasks.

The three Opus 5 runs scored 75.4, 77.1 and 77.9 on test. Each run picked a different champion, from round 3, round 5 and round 1. On the 40 validation tasks the champions scored 95 to 97.5. With only 40 tasks, the best of several candidates always looks better than it is, so we only report the test number.

One scoring detail changes the result by 15 points

Before any of this was comparable, we had to fix our scorer. This is the part of the work we would most like other people to know about.

SpreadsheetBench checks the cell values in the output workbook. There are two ways to read them:

• As saved. Open the file with openpyxl.load_workbook(path, data_only=True) and read the stored values. When Python writes a formula such as =SUM(A1:A9) into a cell, the cell has no stored value. It reads back as empty, and the task fails.
• After recalculation. First open the file in LibreOffice, which computes every formula, then read the values. The same formula now counts as a correct answer.

SkillOpt's evaluator reads the workbook as saved. WikiSkill says its setup is "strictly matched" with SkillOpt, so we score the same way. Our harness originally recalculated. The difference for no-skill Gemma is large:

So on 15% of the test tasks, the model wrote a correct formula that the paper's scorer counts as wrong. A SkillOpt user noticed a gap of a similar size for gpt-5.4: SkillOpt reports 41.4 with no skill, and another paper reports 62.5. The maintainers traced it to four protocol differences, and recalculation is one of them (SkillOpt#35).

Every cap-evolve number in this post is scored as saved, the paper's way. With recalculation, our three champions score 75.7, 77.8 and 78.9. That is almost the same as their as-saved scores, which tells us they learned to write values instead of formulas.

For GPT-5.5 the effect is much larger. No-skill GPT-5.5 scores 38.6 as saved and 77.5 after recalculation. So if you care about the recalculated score, skill evolution adds about 10 points to GPT-5.5, not 49.

Design choices we think matter

We think the six choices below account for most of the gap to WikiSkill. We have not tested them one at a time on SpreadsheetBench, so treat them as our best explanation, not a measured breakdown. None of them is specific to cap-evolve, and any skill-evolution method can use them.

A strong optimizer, separate from the model being taught

In WikiSkill's main results the skills are "self-evolved", which the paper defines as "the inference model and skill source are the same". As we read it, the same 31B model that struggles with the tasks also diagnoses its own failures and writes the fixes.

cap-evolve splits the two roles:

A student can follow good instructions long before it could have written them. The model being taught only has to follow the skill. The optimizer has to understand why tasks fail.

This argument applies to Gemma, not to GPT-5.5. SkillOpt optimizes GPT-5.5 with GPT-5.5, which is already a frontier model.

The optimizer is told who will read the skill

A strong optimizer tends to write for a reader like itself: short, dense and implicit. That is the wrong style for a smaller model. So cap-evolve names the model under test in the optimizer's configuration and maps it to a capability tier: frontier, strong, mid or weak. The optimizer's instructions then open with a "reader" section. In our runs it read (model id shortened):

THE READER (who consumes what you edit). At runtime these capabilities are read by gemma-4-31B-it — capability tier: strong. The reader is a capable general model, but less robust than frontier on long or ambiguous context. Keep instructions clear and reasonably explicit; add a worked example for tricky formats; prefer code enforcement for behavioral rules over trusting inference.

The skills it produced use explicit rules, numbered checks and worked examples. Gemma is not yet in cap-evolve's table of known models, so it got the default tier. We have not measured whether a different tier changes the result.

Feedback that says why a task failed

After each rollout, cap-evolve's scorer writes feedback for the optimizer. The feedback never contains the expected values, but it does say:

• which test case failed, and which cells in the answer range were wrong;
• whether the output file was missing;
• whether the answer was correct but written as a formula, which the as-saved scorer reads as empty.

The last point is the biggest lever on this benchmark. In round 1 the optimizer sees this pattern across many failures and writes a rule for it. Round 1 produced most of the gain in every run: it fixed 18 to 21 validation tasks and broke at most one.

The skill describes the grader's contract, not only the task

Here is the opening of one champion skill. The optimizer wrote all of it:

How your work is graded — read this before you write any code.
Your output file is opened with openpyxl.load_workbook(path, data_only=True) and the values stored in the cells of answer_position are compared with the expected workbook. Nothing is recalculated…

(a) A formula is not an answer. Write the computed literal value — even when the instruction says "formula".
(b) Types must match. Numbers as int/float, never "1,234"; dates as datetime, never an Excel serial number…
(c) Precision. Both sides are rounded to 2 decimal places…

The rest of the skill explains what answer_position means for cell-level and sheet-level tasks. It says to leave cells the instruction does not mention unchanged. It also gives a routine: write the solution, run it, then check the output. A 31B model is good at following explicit rules and bad at guessing implicit ones, and this skill turns the benchmark's implicit rules into explicit ones.

A larger editable surface

cap-evolve lets the optimizer edit both the system prompt and the per-task message template. The template controls how the task fields are explained and how the agent works through its turns. WikiSkill inserts the skill into a fixed prompt, and SkillOpt edits a single skill document. Rewriting how each task is framed gives the optimizer more room to fix misunderstandings. For example, for sheet-level tasks answer_position is the largest range the agent may write to, not the exact range it must fill.

A larger surface also lets the optimizer delete text the runtime needs. The next section shows when that happens.

A statistical gate, and a single look at test

Each candidate is compared with the current skill task by task on the same 40 validation tasks. It is kept only if the improvement is larger than a threshold based on its standard error. SkillOpt keeps an edit only if it strictly improves the validation score. In both methods, the final number is measured once, on 280 test tasks the loop never sees.

Gemma as its own optimizer: seal the harness, edit the skill

A strong optimizer helps, so the obvious next question is WikiSkill's own setting: what if the model being taught also writes its skill? We ran Gemma-4-31B-It as both agent and optimizer, with the same seed, split, scorer and 5 rounds.

The first runs failed on mechanics. Gemma read large tool outputs again and again, and one round stalled for 88 minutes. An optimizer profile fixed that: a short rule for reading large files, and no sub-agents. But three repeats with the profile still spread over 17.2 points. One run beat WikiSkill and one scored 55.8.

What caused the spread

The agent's first message comes from a task template, and the optimizer could edit all of it. The template holds two kinds of text:

• The skill: how to inspect a workbook, type rules, pitfalls. Editing this is the point of the exercise.
• The runtime contract: "reply with exactly ONE python code block per round", the round structure and the stop rule. The harness only runs code in fenced Python blocks, so it depends on this text.

In 2 of the 3 runs, the optimizer deleted the contract in round 1. The gate accepted those candidates anyway, because they still beat the empty seed, and every later round built on them. In the worst run, 93% of test tasks had a wasted round ("No python code block was found"). The agent used 3.6 turns per task, against about 22 in the good run. Most of its replies used Gemma's native tool-call syntax, which this harness does not run.

We checked this directly. Putting back only the 4-line contract, with nothing else changed, raised the two weak skills by +6.4 and +4.7 points.

The fix: an allow-list of editable files

• task_frame.md is sealed. It holds the task fields, the round structure and the stop rule, plus one {guidance} slot.
• The optimizer may edit only guidance.md, which fills the slot, and prompt.md, the system prompt.
• The spec sets editable_paths: [prompt.md, guidance.md]. A candidate that changes any other file is refused before a single rollout runs.
• With an empty guidance file, the sealed frame renders the same message as before, byte for byte, so the seed did not change.

Result: three repeats within 2.6 points

The sealed frame stayed unchanged in all 15 rounds, and the agent kept the round structure in every run. Pooled per task, the sealed runs are level with the best old run (−1.6 points, 95% CI [−5.3, +1.8]). So the seal does not cost score. What it does is make the best case the normal case.

The 95% CI of the 3-run mean is [68.0, 74.4]. That puts us just above WikiSkill's 68.0 on the absolute score. On the gain the gap is larger: +27.7 against +19.7, from a lower start.

This is the seventh design choice, and it applies to any optimizer: put the text the runtime depends on in sealed files, and let the optimizer own the rest. A strong optimizer rarely deletes the contract. A weak one does, and then one bad edit in round 1 decides the whole run.

The Opus 5 optimizer still leads by 5.6 points (76.8 against 71.2). That gap measures the value of a stronger teacher, with the same loop and the same seal.

How we kept the comparison fair

A benchmark number is only comparable if it was produced the same way. Before the main runs, we aligned five things with the WikiSkill paper:

Some things are still different or unknown:

• The WikiSkill and SkillOpt numbers are reported by their papers, not reproduced by us.
• WikiSkill gives the agent a bash tool and states no turn budget. Ours is a Python sandbox with up to 30 turns.
• WikiSkill does not state its decoding settings. We use greedy decoding (temperature 0).
• WikiSkill runs 8 iterations. We run 5.
• Our no-skill score is 4.8 points lower than theirs, so our harness does not give the model an easier start.

Try cap-evolve on your own agent

SpreadsheetBench is one benchmark. cap-evolve is a general framework, and the same loop runs on any agent that has an evaluation:

• It optimizes what the agent reads: system prompts, task templates, tool code, MCP tool surfaces and skill packages, one at a time or together.
• The optimizer is a coding agent. Claude Code, Codex and other coding-agent CLIs plug in as the optimizer.
• Every number is checked. Each accepted edit passes a paired significance test, the test split is scored once, and every published number comes with its run artifact. See how cap-evolve evaluates.
• Every step is a git commit, so you can read, diff and revert what the optimizer did.
• It works on other benchmarks too: τ²-bench, SWE-bench, SkillsBench and more. See the benchmark runs.

To start, follow getting started, which includes a two-minute demo that needs no API key. Then use optimize your own agent to connect your own agent and evaluation.

The code, the benchmark adapters and the run records are in skillberry-ai/cap-evolve. The SpreadsheetBench work is tracked in issue #538. The Gemma self-optimizer work is in #651, and the sealed harness is in #871 and PR #873.

The items we want to do next are at the end of this post. Pick one, open an issue, or bring a benchmark or model you care about.

Details

Per-run results, Gemma-4-31B-It with Claude Opus 5

"Gain, paired" is the mean per-task difference on the tasks both sides scored. The CIs come from a paired bootstrap with 2,000 resamples. WikiSkill uses the same kind of test, with 1,000 resamples. Every run's lower CI bound is above WikiSkill's +19.7.

A few rollouts failed for infrastructure reasons: 4 on the no-skill side, and at most 1 per champion run. We exclude them on both sides of every paired comparison.

The chart above is for the sealed Gemma-only runs. A best validation score of 80 maps to about 71 on test. The best of 5 noisy 40-task scores is always optimistic.

Per-run results, Gemma as its own optimizer, sealed harness

For comparison, the old layout's best run had a paired gain of +29.6 [+23.4, +36.1] and its worst +11.8 [+6.2, +17.6].

GPT-5.5, compared with SkillOpt

We ran the same pipeline with GPT-5.5 as the agent, with the same dataset, split, as-saved scoring, 30-turn sandbox, Claude Opus 5 optimizer and 5 rounds. GPT-5.5 used reasoning effort medium, which is SkillOpt's default, and up to 16,384 output tokens per call. GPT-5.x models do not accept a temperature setting, so we sent none.

We measured the no-skill baseline twice, on two different API endpoints, and got 38.6 on test both times.

Run #3 had one infrastructure error on the champion side, so its paired numbers use 279 tasks.

Where this comparison is weaker than the Gemma one:

1. Most of the gain is the "values, not formulas" lesson. After recalculation, no-skill GPT-5.5 scores 77.5 and the champions 87.4 on average. SkillOpt's own GPT-5.5 skill contains the same rule ("write evaluated static values … instead of relying on Excel recalculation"), so its gain has the same source.
2. The optimizer is different. SkillOpt uses GPT-5.5 as both agent and optimizer, and we use Claude Opus 5. We cannot say how much of the 6.9-point gap (87.6 against 80.7) comes from the optimizer and how much from the loop.
3. SkillOpt reports one number per setting, with no repeated runs or error bars. Our three runs have a standard deviation of 0.6 points.
4. We are not sure which SkillOpt setting is closest to ours. The paper describes SpreadsheetBench as "multi-round codegen with up to 30 turns", which is like our setup. It also describes direct chat as "a single chat completion call with the skill prepended". The Codex and Claude Code rows start much lower (27.5 and 22.1), so we show them only for context.
5. Validation saturates. Two of the three champions score 100 on the 40 validation tasks and 87 on test. Above about 95, a 40-task set cannot rank candidates, so the choice between the top candidates is partly noise.

A fair summary: cap-evolve with Claude Opus 5 as optimizer reaches 87.6 on GPT-5.5, against 80.7 for SkillOpt with GPT-5.5 as optimizer. This does not show that cap-evolve beats SkillOpt with the same optimizer.

Time, cost and tokens

WikiSkill and SkillOpt do not report wall time, dollars or tokens. Here is one cap-evolve optimization run with Gemma:

• The 560 rollouts are 5 rounds × 40 validation tasks, plus 280 test and 80 training tasks for the final report.
• The no-skill measurement ran once and was shared by all three runs: 8 h 4 m and 79.4 M agent tokens. With an empty skill the model wanders. It used about 165 k tokens per task, against about 45 k with an evolved skill.
• Gemma was served on a vLLM endpoint that is not metered, so agent tokens have no dollar figure.
• Everything ran on one 32-core machine, with 8 tasks in parallel.

GPT-5.5 is metered, so for GPT-5.5 we also have the agent's cost:

The three GPT-5.5 champions score within 1.1 points of each other on test, but their test evaluations cost $18.86, $38.48 and $44.19. Selection only looks at the score today, so it does not prefer the cheaper skill.

WikiSkill counts optimizer calls instead: 1 + T_ReAct per iteration, with T_ReAct between 10 and 20 (its Appendix D.2). Over 8 iterations that is roughly 90 to 170 calls. cap-evolve runs one optimizer session per round, 5 in total, and each session is a multi-step agent with up to 80 tool turns. So the number of LLM calls is similar. The difference is which model makes them.

Same measurement for the baseline and the champions

We measured each no-skill baseline once and reused it in all three runs. We checked that both sides were measured the same way:

• Code: the scoring function, agent loop, prompts, workbook preview, sandbox, model settings, dataset and split are identical in the commits that measured the baseline and the champions.
• Data: in every run, the reused baseline matches the no-skill run's own per-task results on all 276 scored tasks. All runs used the same 280 test tasks.
• GPT-5.5: the same holds. All three runs reused the no-skill result exactly (val 35.0, train 37.5, test 38.6).

The full comparison setup

Our no-skill GPT-5.5 score is 3.2 points below SkillOpt's (38.6 against 41.8). That is inside our 95% CI of [32.9, 44.3].

What comes next

• Stronger optimizers. Re-run with newer optimizer models such as Claude Opus 5.5 and Claude Fable 5.1, and plot score against optimizer strength. The weak end is measured: Gemma as its own optimizer reaches 71.2.
• GPT-5.5 as its own optimizer. That is SkillOpt's setting, so it gives a same-optimizer comparison with its 80.7.
• Measure the reader profile. Hide the reader from the optimizer, or give it another tier, and see what changes. Each such run reuses the saved no-skill baseline.
• Test the design choices one at a time, so the list above becomes a measured breakdown.
• More models. The rest of WikiSkill's model list: Qwen-3.5-4B and 9B, Qwen-3.6-27B and Gemini-3.5-Flash. We especially want to know how far a skill can lift small models, and whether a skill evolved for one model transfers to another.
• More benchmarks. WikiSkill's other four: LiveMathematicianBench, SealQA, OfficeQA and ALFWorld. cap-evolve already runs τ²-bench, SWE-bench, SkillsBench and others, so each new benchmark mostly needs an adapter.
• Better selection. Larger or cross-validated selection sets, so validation scores are less optimistic. Agent cost as a tie-breaker, so the cheaper of two equal skills wins. A population of candidates instead of one champion. Memory across rounds, the core idea of WikiSkill.
• Shared baselines. Re-run SkillOpt, EvoSkill and Trace2Skill in the same harness, and publish each run's scorer, decoding and turn budget next to its number.

Run records (GitHub Actions run ids in skillberry-ai/cap-evolve). Gemma-4-31B-It: no-skill baseline 36387149344; optimization runs 36466624841, 36523096755 and 36622615059. GPT-5.5: no-skill baseline 36836354465, repeated in 36706313097; optimization runs 36841799223, 36862654281 and 36887243479. An earlier GPT-5.5 run (36711181652) stopped when its API budget ran out and is not used here. Gemma as its own optimizer: old layout 37204406089, 37345961613 (no profile), 37372504934, 38446925230 and 38481415411 (profile); sealed harness 38723057487, 38862804840 (test from 39118077989) and 39118080899.

References: WikiSkill, "Compiling Agent Experience into Persistent Knowledge for Skill Evolution", Tang et al., arXiv 2608.27454, Table 1. SkillOpt, "Executive Strategy for Self-Evolving Agent Skills", Yang et al., arXiv 2605.23904, Table 1; code and split at microsoft/SkillOpt.