跳到正文
DEV Community· sazerfenor·· 1 小時前AI 評分54

AI 編碼智能體的規則檔也是程式碼,應測試:20 點清單中的三項檢查

Your agent's rules file is code, so test it: three checks from a 20-point list

AI 導讀

作者從 20 點清單中選出三項管理編碼智能體規則檔的做法:將提示詞、規則、工具設定及執行政策納入版本控制,並在設定變更合併時加入驗證檢查。另須預先訂明重試停止及交由人工或更高層工作流程處理的條件,並提供分類、位置和建議恢復路徑等結構化錯誤資訊。同時追蹤 token 和執行時間成本,設定門檻及自動提醒。

正文

Most teams that use a coding agent end up with a rules file. Few of them treat it like the code it effectively is. When the agent starts behaving differently on Tuesday, nobody can say which edit to which file did it.

I run a small reference site, Harness Engineering, about the environment around AI coding agents (the AI kind, not wire harnesses in manufacturing, which the site's FAQ rules out in so many words). Its centrepiece is a 20-point checklist, five items each for feedforward, feedback, infrastructure and team readiness. Each item comes with a failure mode, which is the part I find most useful, because it tells you what you'd see if the item is missing. Here are the three I'd start with.

Put the config in version control, then test it

IN-1 is "version-control harness config": prompts, rules, tool settings and execution policies live in source control with traceable history. The failure mode it lists is that untracked changes make behavior differences impossible to audit or reproduce.

IN-2 goes one step further: attach validation checks whenever a configuration change is merged. The checklist's stated failure mode is that harness regressions ship undetected because the configuration is treated as non-testable text. The framing is the useful part: a rules file isn't prose, it's an input to a system, and inputs can be checked. Even a tiny check, such as "does the agent still run the test command before finishing", turns a vague worry about drift into something you can see fail.

Decide in advance when retries stop

FB-5 is "design an escalation mechanism": define when retries stop and when control goes to a human or a higher-level workflow. Without it, the listed failure is infinite retry behaviour with rising cost and no resolution.

It pairs with FB-3, structured error feedback. Instead of raw logs, return the error with a category, a location and a suggested recovery path. The checklist's reasoning is that vague errors trigger blind retries. Put the two together and you get a loop that either gets better information each time or hands the problem to a person, which is all an escalation rule really is.

Put a number on the bill

IN-4 asks you to track token and runtime cost with thresholds and automated alerts. It's the least glamorous item, and it's the one that turns the previous two from advice into something you'd notice when it fails.

Why these three

They come from the same lens the site's guide uses: separate what you inject before the agent runs (project rules, task framing, tool contracts) from what you measure after it runs (checks, tests, reviews, escalation). Config tests and cost alerts are feedback on your feedforward, which is easy to forget.

Disclosure: it's my site.

Do you test the file that tells your agent how to behave, and if so, what does the test actually assert?

來源:DEV Community · dev.to