AI 輔助提交是否更常被撤銷?663 組配對研究的誠實答案是「無法判斷」
Are AI-assisted commits reverted more often? I checked 663 matched pairs. The honest answer is 'can't tell'
一項觀察性研究比較 663 組 AI 輔助提交與人工作業提交,未能判斷 AI 輔助提交是否更常被撤銷或後續修復。30 日內,AI 輔助組有 1 次撤銷,對照組有 2 次;信賴區間很寬,研究樣本不足以得出明確結論。研究者指出,修復判定方法雜訊較大,可靠解答需要約 10 倍的配對數及經驗證的修復追蹤方法。
Disclosure: I ran this study with an AI assistant (Claude), which wrote the code, did the analysis and drafted the paper. It also did the manual check of my fix heuristic, so that check is not an independent human review. Results are aggregate; no developer, repo or commit is named. This is observational: "associated with", never "causes".
Question: are commits that disclose AI help (e.g. a Co-Authored-By: Claude trailer) reverted or followed by a bug-fix more often than comparable human-only commits?
I wrote down the hypotheses and definitions before pulling any data. The short answer: no clear difference, but the study is too small to say much.
Method in brief
- Public GitHub APIs only, through my own authenticated
gh. No repo was cloned and no code was run. - Repos with 50+ stars, not forks or archived, active in the last year.
- Each AI-disclosed commit was matched 1:1 to a human-only commit from the same repo, within 30 days, with similar diff size, and from the same author when possible (82.5% of pairs).
- 663 matched pairs, 378 repos, commits from 2026-03-12 to 2026-05-04.
- Primary model: conditional logistic regression on the matched pairs.
Matching on size worked: median diff was 102 changed lines for AI-disclosed commits and 103 for controls.
Result 1: reverts are too rare to learn anything
Within 30 days, 1 of 663 AI-disclosed commits was reverted, versus 2 of 663 controls. Odds ratio 0.51, 95% CI 0.05 to 5.57. That interval spans "much safer" to "5x worse". I checked my revert detector against the 264k commits in the downloaded histories: about 0.2% of commits get reverted, so reverts are just rare. Detecting a 2x effect at that rate would take on the order of 10,000 pairs (rough calculation).
Result 2: the fix heuristic is mostly noise
My pre-registered "fix follow-up" rule: a later commit within 14 days, touching a same file, with fix|bug|hotfix|regression in the message. On a feasible 220-pair subset: OR 1.15 (0.75-1.77), consistent with no difference.
But I then read 100 flagged follow-ups and judged only about 30% to be plausible fixes (95% CI 22-40%). Common errors: shared package.json/README/lockfiles, a keyword in the body of a non-fix commit, merges of the same fix. So that "null" mostly reflects a noisy measurement. A stricter, exploratory variant gave the same picture (OR 1.16, 0.72-1.86).
What went wrong, and what I changed
- Sample shortfall: target was 2,000 treatment commits; I got 663 pairs. The 50-star filter removed about 93% of repos, and I stopped collection early for time.
- Bug I caught: my first fix analysis silently dropped 39 pairs where a commit had no follow-up candidates. Fixed; the paper uses the corrected numbers.
- Feasibility cut: the fix outcome only covers pairs with at most 20 candidate follow-ups, which favours quieter periods.
Threats to validity (the important ones)
- Only disclosed AI use is visible, so this studies developers who disclose.
- Controls may contain undisclosed AI code, which biases toward "no difference".
- A revert is a weak quality signal.
- The search sample is capped at 100 results per query-day and favours the latest commits of each day; it covers only 39 days.
- One unblinded AI reviewer judged the heuristic; recall is unknown.
Takeaway
For this sample: no evidence of a difference, and little ability to detect one. A real answer needs about 10x more pairs and a validated way to link fixes to the commits they fix (issue links or SZZ-style tracing), plus some idea of how reliably people disclose.
Related reading (arXiv): On the Use of Agentic Coding, Agentic Much? Adoption of Coding Agents on GitHub.
Preprint-style write-up, not peer reviewed.
By Vyankatesh Rohokale
來源:DEV Community · dev.to

