AI 生成的測試為何會因照搬程式碼假設而通過

An agent writes the function, writes the covering test in the same pass, CI turns green, and the reviewer merges on a clean stack. Months later an edge case breaks in production, you open the test that was supposed to cover it, and it stays green. It stayed green because it never knew the requirement; it copied the implementation's own assumption back at it.

A mirror test repeats the code instead of judging it

A real test derives the expected value from somewhere the code does not: a written rule, a known worked example, an invariant from the domain, or a bug report that already happened. A mirror test derives the expected value by running the new function, reading its output, and hardcoding that output back into the assertion. The test then verifies that the function returns what the function returns. When the implementation changes its own behavior, the agent tends to update the expected value in the same pass, and the coupling closes permanently.

The shallow version is an assertion that only checks the response is non-null, or that the call does not throw. It proves the function ran, nothing about whether the answer is correct.

Why a single pass makes the blind spot shared

When a human writes a test after thinking about the requirement, the test writer can disagree with the implementation and catches a wrong branch. When the same model generates code and test from one prompt, both inherit the same unstated assumption: how empty input behaves, what counts as a valid identifier, which timezone the date means, whether an out-of-range value should clamp or reject. The model has no independent oracle, so the test cannot contradict the code it just wrote. Green in that case is a syntax and plumbing check, not evidence of correctness.

Make the expected value come from outside the code

• Derive expectations from the requirement. Use a worked example from the spec, a known customer case, or a domain invariant, and hardcode that answer independently of the function. If the test author must open the implementation to compute the expected value, it has already become a mirror.
• Pin the boundaries the model tends to skip. Empty input, maximum size, malformed shape, negative or zero, timezone shift, and a value that falls outside the accepted set. Write those cases as explicit expected failures or explicit clamps rather than leaving them to the happy path.
• Assert the result type and error, not just presence. A non-null object or a caught exception proves almost nothing; assert the specific value, the thrown error class, or the rejected status the requirement names.

The cheap mutation probe

On functions that sit on a trust boundary, flip one boundary condition in the implementation deliberately: clamp instead of reject, drop the empty-input guard, shift a rounding rule. Run the suite. If every test still passes, the test is not touching that branch and it provides no safety net. This is a lightweight version of mutation testing; you do not need a full framework for a handful of risky functions, just one local flip per probe and a clean revert after. A test that survives a mutation it should kill is advertising coverage it does not have.

Let real bugs pin the behavior

When a bug does arrive, write the failing characterization test for the exact reported input first, before touching the fix. That test carries the behavior from outside the codebase, so it cannot mirror the repaired implementation. It also becomes the permanent guard that prevents a future agent from smoothly reverting the fix while keeping the suite green.

Tests generated in the same pass as the code are a smoke check. They confirm the wiring runs; they do not confirm the answer is right. Separate the oracle from the implementation, and green starts to mean something again.