BenchSmoke：把 AI 編碼智能體的「完成」視為一項主張

When we test AI coding tools at BenchSmoke, we do not treat the agent's final message as proof. We run our own checks. That habit has saved us from shipping a false "it works" more than once.

What the agent said vs what we checked

In one recent run, an agent wrote a small to-do CLI, told us the start command, and stopped. It never ran its own code. The app mostly worked when we checked it ourselves: add, complete, delete, and a restart that kept state. Our separate script still found two edge bugs. Empty input was accepted as a task. A non-numeric complete command printed the wrong error. Both would have been invisible if we had stopped at the agent's "done" line.

Write the verify steps first

For our standard coding task we always check the same things: add two tasks, mark one complete, delete one, restart the process, and confirm state survived. Same starting prompt for every tool in the category. Same verify steps every time. That is how comparisons stay fair.

Log the boring facts while you work

We write down date, version, plan and price we checked, exact prompts, time to a working result, retry count, what broke, and limits we hit. If it is not in the notes, it does not go in a review. We score six questions from 1 to 5 (does it work, time saved, output quality, ease of use, value, reliability), then pick Use it, Watch it, or Skip it. When two verdicts feel close, we take the lower one and explain why.

Say what you did not test

A feature we could not finish on our machine gets marked "not tested," not guessed from the docs. Guessing is how reviews turn into marketing copy.

None of this needs a fancy lab. A short shell script and a consistent task beat vibes. If you only click around until the happy path works, you will miss the edges the agent never tried.

Our full method is public: How we test.