作者抽查 317 個已合併 AI 編碼智能體 PR，檢視其工單要求完成情況

Coding agents now open a lot of pull requests that say Fixes #123. I wanted to know how often the ticket is actually done when that PR merges.

So I took 317 public pull requests from 317 different organisations. Each one was written by a coding agent (Claude Code, Devin, Copilot's coding agent or Cursor), was merged, and closed at least one issue. I checked every requirement in each issue against the diff.

The numbers

1 in 7 PRs left at least one requirement missed or only partly done (15%).
6% skipped a requirement entirely.
1 in 4 when the ticket was edited after the agent started (6 of 23). That's a small sample, but it's the case I care about most.

Most agent PRs do what the ticket asks. The ones that don't are hard to spot in review, because the reviewer reads the diff, not the ticket.

What gets missed

Of the 70 requirements that were missed or only partly done:

Core behaviour: 49%
Tests: 21%
Error and edge cases: 10%
Docs and changelog: 10%
UI: 6%
CI and release: 4%

What I kept seeing

The last line of the ticket. Acceptance criteria at the bottom, such as "check the other files for the same pattern" or "add a test for the empty case", drop out more often than the headline change.
The description says it, the diff doesn't. "Tests pass", "kept the manifest in step". The claim is in the PR body, and nothing in the diff backs it up.
Yesterday's spec. When a requirement changes after the agent starts, the PR often builds the version it started from.

How I checked, including what I got wrong

Each PR was checked by Lians check. Claude reads the issue the PR closes and its discussion, splits it into requirements, and marks each one met, partial, missed or "can't tell" from the diff.

My first pass was wrong. It graded every issue a PR so much as mentioned, including "#123 stays separate", and it ignored scope changes in the issue's comments. That run said 44% of PRs missed something. Hand-checking a sample showed most of those were false alarms. I fixed both problems and re-ran everything. On the hand-checked findings, about 3 in 4 of the new run's "missed" or "partial" calls held up. So read 15% as an upper bound, and the true rate as closer to 1 in 9.

Another 323 requirements came back "can't tell", usually because the evidence was a CI run, a sibling PR or code outside the diff. They aren't counted as gaps. The sample is public repositories found through GitHub search, so it leans toward smaller teams and open-source companies. I'm not naming any repository.

What to do about it

When you review an agent's PR, read the ticket as well as the diff, and look hardest at PRs whose ticket changed after the work started.

If you'd rather not do that by hand, Lians check does it on every PR. It runs in your own GitHub Actions, posts one comment listing each requirement, and re-checks the open PRs linked to a ticket when someone edits it. Here's what a check looks like on a real PR: https://github.com/ebeirne/lians-check-demo/pull/2

I'm looking for a few teams using coding agents on real repos to try it and tell me what's useful and what's noise. Full write-up and method: https://www.lians.ai/agent-prs