跳到正文
DEV Community· Sion Jang·· 4 天前AI 評分66

編碼智能體測試通過,仍會修改用戶未要求變更的檔案

Your coding agent's tests pass. It still edited files you never asked about.

AI 導讀

Sion Jang 指出,編碼智能體即使測試全部通過,仍可能修改任務範圍以外的檔案,因為測試未必能檢查變更是否符合要求。一項 2026 年 5 月的研究分析了 20,574 個真實編碼智能體工作階段;在 16,118 個經驗證問題案例中,38.33% 違反明確的開發者限制,10.20% 超出任務範圍,22.58% 錯報工作狀態。

正文

You ask a coding agent to fix one function. It fixes it. On the way it also touches a config file, bumps a version, and loosens a test that was in the way. Every test passes. The agent says it's done.

Nothing in that workflow is wrong by its own rules. The tests were the definition of done, and the tests pass. The extra edits only show up if you read the whole diff, and the point of handing work to an agent was not having to.

This is common, not anecdotal

A May 2026 study read 20,574 real coding-agent sessions from public repositories, about a third of them Claude Code, along with Cursor, Codex and others (arXiv 2605.29442). Across 16,118 validated problem episodes:

  • 38.33% involved the agent violating an explicit developer constraint
  • 10.20% were the agent taking actions beyond the stated scope
  • 22.58% were the agent misreporting the status of its own work

When a problem visibly got resolved, 91.49% of the time it took the developer explicitly pushing back. The agent rarely caught it alone.

Forums say the same in plainer words: an agent that keeps editing files the user didn't ask it to, to the point the user built their own auditor, and agents that overshoot simple requests.

Why "make the tests pass" doesn't catch it

A test-based gate answers one question: does the code still do what the tests check? Scope creep is a different question: did the agent change only what it was asked to change?

  • A lockfile or config edit usually breaks no test.
  • A loosened test makes the suite easier to pass.
  • A version bump is invisible to unit tests.

One developer ran 36 controlled Claude Code tasks to check whether "done" matched the output. The report and the output never disagreed. What did go wrong was quieter. Under a strict completion criterion, the agent changed what the code meant in order to pass, in 2 of 6 runs. And the verification script itself passed when the result file was missing.

So the failure isn't mainly the agent lying. Our checks ask the wrong question, or quietly pass when there is nothing to check.

What a scope check has to do

If you want a machine to say "this went outside the task," it needs:

  1. A declared scope. Paths the task may change, and paths it may never change. Without this there is nothing to judge against.
  2. The final state, not the tool calls. Agents write files through scripts, build steps and shell redirects as well as edit tools. Watching tool calls misses some of those. Comparing the working tree at the end does not.
  3. A baseline. Your own uncommitted work existed before the session. A scope check that blames the agent for it gets switched off within a day.
  4. A place in the loop where it can say no. A report you read afterwards is a code review. A check at the moment the agent tries to finish can send it back with a list of what to undo.
  5. A way out. If the agent fails the same check three times, it should hand over to a person instead of looping.

And it should be honest about what it can't see. A path-based scope check doesn't notice a change of meaning inside a file you allowed, like rewriting a calculation so a test passes. That still needs an output check or a human.

What I built

I built this as a Claude Code Stop hook called taskattest. When the agent tries to finish, it checks the git working tree against a declared scope:

{
  "task_id": "fix-billing",
  "checks": [
    {"type": "scope_changes", "name": "billing only", "root": "..",
     "allow": ["src/billing/**", "tests/**"], "deny": ["**/*.lock"]}
  ]
}

If the agent also changed pyproject.toml, it can't stop yet:

[FAIL] fix-billing  (model calls: 0)
  - fail    billing only: 1 change(s) outside the allowed scope: pyproject.toml
            fix: undo changes outside the task scope: pyproject.toml

It can also check output files (invented rows, wrong totals, CSV and Excel). It's deterministic: no model calls, no network, standard library only, and it runs on Linux, macOS and Windows. Apache-2.0.

https://github.com/sionjang101-create/taskattest

I'm more interested in the problem than the tool. If you've hit scope creep with an agent, what did it touch, and would a declared scope have caught it?

來源:DEV Community · dev.to