surviving-lines 在六組程式碼所有權基準測試中得分最高

I wrote surviving-lines to answer one question: who wrote the code that is still in this repository? It is a command-line tool that counts each author's surviving lines with git blame and prints them next to the author's share of commits. It also ships as a Claude Code plugin, so an agent can run it instead of estimating. For version 0.4.0 I wanted to know whether the plugin changes how well an agent answers ownership questions. So I measured it against three other tools and against Claude with no plugin.

The short result: on 5 sealed tasks the plugin scored 9.67 of 25 and the best other arm scored 7.33. The 95% bootstrap interval of that difference is [+0.33, +4.33], so it stays above zero but only just. A score of 9.67 of 25 also means that the plugin passed fewer than 4 criteria in 10, so these tasks are hard for every arm. On 11 public tasks, which I had read while developing the tool, it scored 47.67 of 51 against 41.00 for the best other arm. It did not win every task, and the details are below.

What was measured

Each task gives an agent a git repository and a question that a person taking over a project would ask, for example "Who should I ask to review changes to time.go, version7.go and the SQL code, and are the people who actually wrote those parts still around?" A task has five criteria or fewer, such as "names this person as the top owner with a share between 51% and 59%" or "does not credit the person who only ran the formatter". The criteria were written with a ground truth computed from the repository, and a model grader scored each answer against them. The grader gave its reason before its verdict, with five votes per criterion, and the verdict is the majority.

There are two sets of tasks:

• The sealed set has 5 tasks. An agent that had not seen the surviving-lines repository wrote them. It was given a plain description of the user's job and the file layout. Each task uses a real open-source repository in a different ecosystem (Python, Go, Zig, Ruby, PHP) with commits added by the writer that create hazards: a formatter run, the same person under several addresses, squash-merged work, mechanical file moves, an author who differs from the committer.
• The public set has 11 tasks. It uses one small JavaScript repository where the main author has two addresses, asked in several ways (who owns the code, the bus factor of a directory, who should review a file, code owners per directory and so on). The tasks were read while the tool was being developed, so I label the set development-seen. The claim in this article rests on the sealed set.

The six arms are surviving-lines at commit 8af57f4, project-pilot 1.0.0 with its natural prompt and again with the prompt "Use the /project-pilot:bus-factor command for this.", mcp-git-historian, busfactor by k41n, and a Claude Code run with an empty plugin. The other tools were run from copies pinned at the commits listed in the evidence folder. project-pilot 1.0.0, invoked only has runs on the 8 public tasks that define an invoked prompt, so its public total counts the other 3 tasks as 0. Each arm ran each task three times on Sonnet 5.5 through Claude Code 2.1.308. The protocol was committed before the first run, and no result from an earlier round was reused. Version 0.4.0 on npm ships the same plugin code as that commit: the bin, hooks, agents and schema folders are byte-identical, and only the skill description text and version number differ.

Sealed set

A total is the sum over the tasks of the mean criteria passed over the three runs. The difference to the best other arm, project-pilot 1.0.0, invoked, is +2.33 with a 95% interval of [+0.33, +4.33]. Against the other four arms the differences are between +6.33 and +7.67, and every interval is above zero. With 5 tasks and 3 runs, the lower bound of 0.33 is close to zero, and a different set of tasks could move it either way.

The protocol also defines stricter checks, and the plugin fails them on this set. On HZ3 it scored 0.67 where project-pilot 1.0.0, invoked scored 1.33, and on HZ5 it scored 1.33 where the empty-plugin run scored 1.67. The per-task rule (rule B), under which the plugin may not trail the best rival on any task, therefore fails, and so does the rule that the plugin must lose at most 80% of the points the best rival loses: it lost 15.33 of 25 and the rival lost 17.67, where the limit was 14.13. I do not claim it is first on every task.

Public set

On the public set the plugin scored 47.67 of 51. The best other arm was the empty-plugin run at 41.00, and the difference is +6.67 with an interval of [+5.00, +8.33]. Against project-pilot taking the better of its two arms on each task, the difference is +4.33 with an interval of [+2.00, +6.00]. The totals of the other arms are 40.33 (project-pilot 1.0.0), 40.33 (busfactor), 38.33 (mcp-git-historian) and 27.33 (project-pilot 1.0.0, invoked, with the 3 missing tasks).

It lost three tasks: T1 and T1i, where it scored 3.00 and busfactor and the empty-plugin run scored 4.00, and E5, where it scored 4.00 and mcp-git-historian scored 5.00. The 80% rule passes on this set. The per-task rule fails on those three tasks. Because these tasks were read during development, a high score here is weaker evidence than the sealed result.

Other models

I ran Haiku 5.5 and Opus 5.5 once per task, with only surviving-lines, the empty-plugin run and the best rival of each task. These totals are descriptive. One run per task is too few to compute an interval, and I make no claim of being first from them.

The plugin is ahead in total in all four rows. The per-task rule fails on several tasks for both models, except on Opus with the public set.

How much to trust the grading

I regraded a random sample of 28 of the 279 graded cells with fresh grader sessions. The new grade agreed with the first on 129 of 131 criteria (98.5%), and no cell moved by more than one point. I also ran the same 28 cells again once. 25 of those 28 re-runs scored within the range of the three original runs, so a single run of a cell can differ from the earlier ones. On the sealed set the five votes were unanimous on 91.6% of verdicts and on the public set on 97.0%.

What it costs and what went wrong

At list price the plugin costs 1.67 times the empty-plugin run per cell on the sealed set (0.117 US dollars against 0.070) and 1.56 times on the public set (0.076 against 0.049), because it uses more turns and tokens. The runs were charged to a subscription. The whole round, including the development gates that decided whether to run the final measurement, would cost 185.46 US dollars at list price.

The plugin has a Stop hook that checks the numbers in an answer once before the agent finishes. It blocked the first answer in 5 of 48 plugin cells (10.4%). Those cells scored 3.40 on average and the unblocked ones 3.60 criteria, so this data does not show whether the hook helps.

These points limit the result:

• The tasks are ours. The sealed writer did not read the surviving-lines repository, but the agent that ran the measurement saw the writer's closing summary (the five repositories and a line of hazards per task) before I wrote the protocol. The hazards are the ones the tool was built to handle. A benchmark by someone who builds a rival could rank the arms differently.
• Five sealed tasks is a small set, and the intervals reflect only the three runs per task, not a different choice of tasks.
• One model produced the sealed and public runs, one CLI version, one login.
• The session limit stopped the drivers twice, both times during the audit re-runs. Cells that failed on a limit were set aside, never graded, and run again, and no scored run was set aside. I renewed the login after the second stop, so the audit re-runs after that point and the other-model runs may have gone through a different account.
• The sealed scripts are published with contributor addresses replaced by their SHA-256. The published scripts rebuild the fixtures with the same commit ids and reproduce the ground truth byte for byte.
• The interval of each arm's own total in the figure is not printed in the evidence summary, which only has intervals of differences. I computed them from the same grades with the same method (intervals.py, next to the figure script), and it checks its totals against the summary.

An earlier measurement: commit share against surviving lines

The benchmark above asks whether an agent uses the tool well. A separate question is why the tool counts both numbers. In September I ran it on twelve public repositories, with the design and the meaning of a null result written down first. For each repository I measured every author's share of non-merge commits and share of surviving lines at HEAD, from a one-in-five deterministic file sample, with three seeds per repository. That is 36 runs and 2,748 author rows, measured on 5 September 2026 with version 0.1.6.

The author with the most commits also held the most surviving lines in 25 of the 36 runs, and on all three seeds in 8 of the 12 repositories. Commit counts therefore usually point at the right person. The gap between the two shares is larger than that suggests: among authors holding at least 5% of either measure (122 author rows), the median gap was 7.5 percentage points and the mean 12.2. 43 of the 122 (35%) differed by 10 points or more, and the largest single gap was 63.8 points. In cloudflare/capnweb the top committer, holding 46% of commits, held between 3% and 16% of the sampled surviving lines depending on the seed, while a contributor with 2% of the commits held 45% to 58%. I did not look into why.

This has limits. Twelve repositories describe twelve repositories and are not a statement about open source. The sample moves the magnitudes: the top author's line share had a median spread of 5.4 points across seeds and a maximum of 17.4, so I read a direction from one run and a magnitude from several. The tool counted bots as authors in that run, and since 0.3.0 it leaves them out by default, so that table has not been re-run with the current behaviour. Surviving code is not merit, since code can stay because nobody dares to touch it. HEAD has moved on in all twelve repositories and the data file records no commit ids, so a re-run will give different numbers.

Try it and check it

npx surviving-lines --sort gap

--sort gap puts the author with the largest difference between line share and commit share first. The tool needs Node 22 or newer and git 2.15 or newer. To use it from Claude Code:

/plugin install surviving-lines --marketplace Bubblegunn/surviving-lines

Every cell, the departures from the protocol and a recompute step are in the round 13 evidence folder. REPRODUCE.md in that folder gives the commands for a re-run, and the earlier rounds and rulings are in docs/HEAD-TO-HEAD.md. The repository is Bubblegunn/surviving-lines, and version 0.4.0 is on npm. I wrote about the first version of the tool in September: Who really owns this code? A git blame sampler you can run in one command.

Three people outside my own commits have shipped code in the tool: @shivam-070208 wrote --markdown, @Aaqibhafeezkhan wrote --sort gap, and @Enard52 sent fixes that make empty runs say which filter emptied them. If you run it on something and the numbers look wrong to you, tell me which repository and I will look.

Disclosure: I drafted this article with AI assistance (Claude) and reviewed it myself. Every number comes from the evidence folder linked above and the research folder in the repository, and the benchmark can be re-run from it.