two-engines-gate 如何處理被弱化或刪除的測試

If Claude Code weakens a test in the same commit that breaks the code, my test gate passes that commit. Codex is told to look for weakened and deleted tests when it reviews, and the only hard check in the scripts is for a run where zero tests ran.

Kartik pointed out this weak spot under my Hashnode post on how Claude Code and Codex review each other's work. I gave him the short answer in the comments. This post is the full answer, with the commands I ran to check it.

Why the tests in the same commit are the weak spot

two-engines-gate is the free, MIT licensed version of how I make Claude Code and Codex check each other. gate.sh runs your tests and writes a pass record named after the commit, review.sh gets the other engine to review it, and merge-check.sh refuses the merge unless the commit has both. I wrote about the review loop and the commit hash records before.

gate.sh runs whatever tests are in the commit. When a commit changes the code and the tests together, the same commit also decides what a pass means.

I checked this on 23 October with the public scripts and a small Python project. It had an add function and two assertEqual tests. First I broke add so it returned a + b + 1, and the gate failed as it should:

FAILED (failures=2)
gate: FAIL (exit 1) on 8390173080fb; no pass record; full log .ai/tests/8390173080fb.log

Then, in the next commit, I changed the two tests to assertTrue(add(2, 3)) and assertIsNotNone(add(-1, 1)). Both hold for the broken function, so the gate wrote a pass record:

gate: running 'python3 -m unittest -q' on 266c4d979b71
gate: PASS on 266c4d979b71, record .ai/tests/266c4d979b7102820377e64ab3f20b069e41bf01.pass

The test command exited 0 and two tests still ran, so gate.sh had nothing to refuse. Catching this falls to the reviewer.

What the reviewer is told

The free review-prompt.md asks for a static review. Codex reads the commits, the diff and any file in the checkout, and runs no tests or builds. This is the line that covers tests:

Check in particular: behaviour the commits promise but the diff does not deliver, error paths and edge cases, tests that would still pass on the old code, and anything that weakens an existing check.

The same prompt says a blocker or should-fix needs "a concrete, reproducible failure". For a weakened assertion that failure is easy to write down. Call add(2, 3) on the new code, get 6, and watch the new test pass anyway. A reviewer that reads the test diff next to the code diff has everything it needs to flag it.

In my own projects the prompt Codex gets is stricter. Every test change has to fail on the old behaviour, nothing can be weakened without a stated reason, and the reviewer has to check the diff for any weakened or deleted test. A loosened assertion with no reason comes back as CHANGES REQUIRED. The README tells you to edit review-prompt.md to add project rules, so you can put the same three rules in your copy.

That is still the reviewer's judgement. The README says so directly: "A review can miss things, and tests only check what they test." If Codex misses the weakened test and answers ACCEPT, merge-check.sh finds a pass record and an ACCEPT for the commit and lets it through.

The hard check: a run with zero tests

One case is checked by the script itself. gate.sh refuses a pass when the test command prints nothing, or when the test summaries it knows add up to zero tests that ran. It reads the summary lines of pytest, unittest, node --test, mocha, Jest and Vitest. That covers deleting every test, a filter or file list that selects nothing, and, for every runner except unittest, a run where every test was skipped.

In my Python project, deleting the test file made unittest exit 5 with "NO TESTS RAN", so the gate already failed on the exit code. To see the zero-test check on its own, I ran the same command with || true so it always exited 0:

gate: the test summaries in the output show that no test ran; set GATE_COUNT_RE or GATE_MIN_TESTS (README)
gate: FAIL (exit 1) on f93de2d8e8a6; no pass record; full log .ai/tests/f93de2d8e8a6.log

Deleting one of the two tests is a different case. One test still ran, so the default check passed the commit. If you want the gate to notice a drop in the count, set GATE_COUNT_RE to a pattern for the count line your runner prints, and GATE_MIN_TESTS to the number of tests you have now. With GATE_COUNT_RE='^Ran ([0-9]+) tests? in ' and GATE_MIN_TESTS=2, the same commit failed:

gate: the output reports 1 tests, fewer than GATE_MIN_TESTS=2; set GATE_COUNT_RE or GATE_MIN_TESTS (README)
gate: FAIL (exit 1) on 1ea182ad01af; no pass record; full log .ai/tests/1ea182ad01af.log

The minimum is set where you run the gate, outside the commit, so you raise it by hand when you add tests. go test and cargo test need GATE_COUNT_RE before any count check works for them, and the README has an example for each. GATE_MIN_TESTS=0 turns the check off.

A habit that covers new tests

New tests get one more step, which I described in the post on how I make Codex review Claude Code's commits. The engine that writes the code often writes the tests too, and a test that already passes on the old code proves nothing about the change. So the writer runs each new test once against the base commit and pastes the failing output into the commit body. That does nothing for an old test that got weaker, which is why the prompt rule matters.

What isn't built

A script that reads the diff and flags removed or loosened assertions does not exist. I told Kartik it would be a good addition and that I'd look at it. Today neither the free gate nor the kit has one, so a single weakened assertion gets caught only if the reviewer notices it.

What to set up in your project

1. Add the test rules to your review-prompt.md: new tests must fail on the old behaviour, and no test gets weakened or deleted without a stated reason.
2. Leave the zero-test check on, and set GATE_COUNT_RE if you use go test or cargo test.
3. Set GATE_MIN_TESTS to your current count if a deleted test should fail the gate.
4. Have the writing engine run each new test against the base commit and paste the output into the commit body.

The same rule in the kit

The paid Two Engines Kit (currently v1.0.2) ships a review prompt template whose project rules include "tests must fail on the old behaviour; nothing weakened without reason", and its list of risks to check includes "any weakened test". The Two Engines Kit is built on information gathered since the first Codex and Claude CLIs.

Try it

Clone two-engines-gate and run bash demo.sh to watch one full loop with a stand-in reviewer, so no model or account is used. tests/run.sh has the zero-test cases for all six test runners, plus the GATE_COUNT_RE and GATE_MIN_TESTS cases.