哪些 AI 智能體程式碼變更容易引發正式環境故障？缺陷研究指向分散度、規模與歷史

• Book: Decoupled PHP — Clean and Hexagonal Architecture for Applications That Outlive the Framework
• Also by me: Thinking in Go (2-book series) — Complete Guide to Go Programming + Hexagonal Architecture in Go
• My project: Hermes IDE | GitHub — an IDE for developers who ship with Claude Code and other AI coding tools
• Me: xgabriel.com | GitHub

In 2000, a web-based tool at Bell Labs could look at a finished change to the 5ESS telephone switch software and mark it high risk, with the reasons attached. The paper that describes it gives an example: a change flagged with two of them, "many subsystems touched" and "is a field fault fix" (Mockus and Weiss, Bell Labs Technical Journal, 2000).

Five of the model's six inputs described the change. The sixth was the author's experience.

Today the author of a change is often a coding agent, and the change arrives as one long diff. The default way to read it is file by file, in path order, which puts api/ before worker/ whether or not api/ is where the danger is. Two studies, 13 years apart, point to a better order, and most of the signals they used can be pulled out of git in a few lines of shell.

What Bell Labs measured in 2000

Audris Mockus and David Weiss studied about 15,000 change requests to the 5ESS switch software over ten years. Each one either went out in a software update without trouble or was later found, by root-cause analysis, to have caused an update to fail (paper, p. 172). They measured each change on five axes: size, spread (they call it diffusion), how long it took, the author's experience, and whether it fixed a defect found in the field.

After model selection, six predictors survived (Table III): the number of subsystems touched, the number of separate edits (deltas), whether it was a field-fault fix, the time between first and last edit, developer experience, and lines of code added. More subsystems, more edits, more added lines, a longer interval and being a fix all raised the odds of failure. More experience lowered them. Lines deleted did not matter (p = 0.87 in the full model, Table II).

They called diffusion "one of the most important factors" in predicting failure, and explained why (p. 173): "A large diffusion indicates that the modularity of the code is not compatible with the change, because several modules have to be touched to implement the change."

Diffusion: one logical change, many places touched.

They were careful about what this proves. In their words, the model "does not prove causal relationships." It finds which properties of a change go together with failure, and the tool used that to decide which changes got extra inspection and testing.

The 2013 replication across eleven projects

Thirteen years later Yasutaka Kamei and colleagues, with Mockus among the co-authors, tested the same idea at scale in IEEE Transactions on Software Engineering. They used more than 250,000 changes from six open source projects (Bugzilla, Columba, Eclipse JDT, Eclipse Platform, Mozilla, PostgreSQL) and five commercial ones. They called the approach "Just-In-Time Quality Assurance": flag a risky change at check-in, while the people who wrote it still remember why.

They measured 14 properties in five groups: spread (subsystems, directories, files, how evenly the edit is scattered), size (lines added, deleted, file size before), purpose (is it a fix), history (how many developers and prior changes touched those files, and how long since the last change) and author experience.

The results that held up (Table 6):

• Number of files touched raised risk in all 11 projects.
• Lines added relative to the file's size before the change raised risk in 10 of 11. They divided by the file's size because earlier work found relative churn predicts defects better than raw line counts. In practice, 200 new lines in a 100-line file is a rewrite, while 200 lines in a 5,000-line file may be one new function.
• Being a fix raised risk in four of the six open source projects.
• How recently the files last changed mattered in every open source project: the more recent the last change to a touched file, the more likely the new change introduced a defect. In two commercial projects the effect ran the other way.

Author experience was the least portable signal. It raised risk in three projects and lowered it in three others. Agent work adds a second problem: if the agent commits under your git identity, git would score your experience, not the agent's.

The model was far from perfect. On average it caught 64% of defect-inducing changes, with 68% accuracy and 34% precision (Section 5.1), so most changes it flagged were fine. As a pointer to where to look first, that's enough.

Four signals, four git commands

Run these on the agent's branch. Every command compares against the point where the branch left main, which is what git diff main...HEAD does (git docs).

Spread. Kamei's study used the top-level directory as the subsystem name, so counting top-level directories gets you close to their measure:

git diff --shortstat main...HEAD
git diff --dirstat=files,0 main...HEAD
git diff --name-only main...HEAD \
  | cut -d/ -f1 | sort | uniq -c | sort -rn

--shortstat gives files changed, insertions and deletions in one line. --dirstat=files,0 gives each directory's share of the changed files, with the cut-off at 0% so nothing is hidden. The last command counts files per top-level directory. A one-feature task that lands in five of them is the kind of change the 2000 tool flagged as "many subsystems touched". It raised that flag when a change touched more subsystems than 95% of past changes (p. 177).

Size relative to the file. --numstat prints added and deleted lines per file, tab-separated, with - for binaries:

base=$(git merge-base main HEAD)
git diff --numstat --no-renames "$base" HEAD |
while read -r added deleted path; do
  before=$(git show "$base:$path" 2>/dev/null | wc -l)
  echo "$added $before $path"
done

The second column is the file's length before the agent touched it. A 0 there is a new file. A row where the first number is close to or above the second is a file the agent mostly rewrote.

History. Reusing $base from above, for each touched file: commits in the last six months, past commits whose message mentions a fix or bug (the same keyword trick Kamei's team used to find fixes), and how long ago it last changed:

git diff --name-only "$base" HEAD |
while read -r f; do
  n=$(git log --oneline --since="6 months ago" \
      "$base" -- "$f" | wc -l)
  fx=$(git log --oneline -i -E --grep='fix|bug' \
      "$base" -- "$f" | wc -l)
  last=$(git log -1 --format=%cr "$base" -- "$f")
  echo "$n $fx $f (last: ${last:-new file})"
done | sort -rn

The list is sorted by the six-month commit count. Read the fix count and the last: value next to each file by eye.

Purpose. You don't need git for this one. You wrote the prompt. If the task was "fix the bug where…", note that Bell Labs' model scored a field-fault fix as riskier, and so did four of Kamei's six open source projects (none of the five commercial ones).

Read the diff in risk order

Kamei's team also asked a practical question: with limited review time, how much do you catch if you look at the riskiest changes first? Ranking whole changes by predicted risk per line changed, reviewing 20% of the changed lines found on average 35% of the defect-inducing changes (Section 5.2, paper). That ranking favours small, dense changes, so it says nothing about the order of files inside one diff.

Same diff, different order: the risky files first.

For the order inside one diff, go back to the risk factors in Table 6. Applied to one agent branch, they suggest a reading order. The order is my interpretation of those factors, not something either paper tested:

1. Check the spread before any line. If the task was one feature and the diff crosses four top-level directories, find out why before reading what. Sometimes the answer is a shared type. Sometimes the agent wandered.
2. Read the hot files next. Top of the history list: files that change often, get fixed often, or changed last week.
3. Then the rewrites. Rows from the size script where the added lines rival the file's old length.
4. New files and small edits last. Read them anyway. Kamei's team found the defect-inducing changes their model missed were mostly ones that touched few files or weren't fixes (Section 6.5).

If the agent's work is split into turns or commits, run the same checks per turn. A turn is a smaller change, and each measure above is about one change.

Where Hermes IDE fits

I build Hermes IDE, so weigh this with that in mind. Everything above works in any terminal. Hermes keeps the turn as the unit: every agent turn is saved, so you can diff or restore any single turn, with undo. The Review Desk (⌘G, or Ctrl+Shift+G on Windows and Linux) shows an agent's changes by turn or by file, with risk flags, and a line comment you leave there goes back to the agent that wrote that line (README).

The Review Desk in By turn view: two Claude Code turns in orders-api (T1: 3 files +69 −1, T2: 1 file +1 −0), turn 1's diff of src/db.ts, and a commit message drafted from the turns.

The per-turn counts in that view are a turn's file count and added lines, two signals both papers measured. I'm not claiming Hermes's risk flags reproduce either paper's model. Timing is where it lines up with the research: Kamei's case for checking at check-in was that the author still has the context, and a line comment goes to the agent session that wrote the change.

Hermes is free, source-available under BSL 1.1, and runs on macOS, Windows and Linux. It isn't affiliated with Anthropic, OpenAI, Nous Research or any agent vendor. Download it or read the source on GitHub.

What these studies can't tell you

Both papers are about human-written changes in C, C++ and Java codebases from the CVS and SCCS era. Neither studied agent-written code. Using their signals on an agent's branch is my extrapolation, and the precision numbers say even the original models flagged many safe changes. In the open source projects, fixes and the changes that caused them were found with commit-message keywords and the SZZ algorithm, heuristics that can mislabel a change, and the authors discuss that in their threats-to-validity section.

None of these signals cares who or what typed the code. On the next agent branch, run the spread commands before you open a single file.

If this was useful

Mockus and Weiss read a change that sprawls across modules as a sign that the code's boundaries don't fit the change. When an agent's one-feature diff touches your controllers, your Eloquent models and three services at once, that is often a boundary problem before it is an agent problem. Decoupled PHP is my book on drawing those boundaries in PHP apps, so a change to the domain stays in the domain and the framework stays at the edge.

Available on Kindle, Paperback, and Hardcover. English, German, and Japanese editions out now — Portuguese and Spanish coming soon.