agentwatch v0.2.0 現場測試盤點：94 個案例中約 8 個斷言能在主張為假時失敗

The v0.2.0 field test for agentwatch ran 94 cases across 15 suites. 92 passed. 0 failed.

It was the least trustworthy green I'd ever shipped. Not because anything was failing — because I couldn't say why it was passing.

So I did a census. Of the 94 case assertions, how many could fail if the claim they were checking turned out to be false?

The answer was about 8.

The census

Every case had a step script. I read all 94 and sorted them into shapes:

That table is the whole finding. A suite where 8 of 94 assertions can fail isn't a test suite — it's a deployment script with extra steps. Every one of those 86 other cases would have stayed green with the feature broken, in many cases completely broken.

Two of the 92 greens were hiding exactly that: a Tempo backend that didn't exist, and a version banner that printed the wrong number. Both passed. Both were "plumbing green."

The method: four patterns, not 84 one-offs

Reading 94 cases sounds like a week of work. It was one evening, because the cases collapse into four patterns — and each pattern has one fix:

Pattern A — the shipped test already covers the plan condition. The single biggest surprise of the census: for something like 44 of the weak cases, the repo already had a canonical pytest module that asserts exactly what the field case was groping at. test_browser_verifier.py for the offline verifier, test_conformance_runner.py for the plugin API, test_access.py for the role matrix. The field case was doing test -f on a file while a real test sat two directories over.

The fix was one generic driver — shipped_tests.py, ~40 lines, which runs a named pytest module under the right PYTHONPATH and fails loudly if it doesn't pass. One driver, 44 cases grounded, zero re-implementation. If you take one thing from this post: before you write an assertion, grep your own test suite. The cheapest sufficient check usually already exists, and a hand-rolled one will drift from it.

Pattern B — the driver runs but its ok() is unconditional. The driver calls the real CLI, gets real JSON back, prints ok: ..., exits 0. Nothing ever looks inside the JSON. The fix is boring: parse it and assert the contract — the derivation version is present, the join summary carries the classified form, the statement is backed by a config key. The driver already had the data in its hand.

Pattern C — existence standing in for function. test -f verifier.html. json.tool bundle.json. import agentsec_agentwatch.conformance. Each of these has a real condition one step deeper, and each one got rewritten to it.

Pattern D — needs tooling that isn't in the room. Two cases need a real LangGraph SDK driver and two independent OSS parsers. No local fix exists. These stayed run-only, flagged, never faked — the report says "blocked" and names exactly what's missing. That's the fourth pattern's whole fix: honest labeling.

What grounding did to the failure count

Here's the uncomfortable, excellent part. Hardening didn't just make checks stronger — it made them fail. Three hardened cases went red on their first honest run:

• A backend check that now queried Tempo — which was never started.
• A matrix check that now asserted "no modeled Tier-1 rows" — one row was modeled.
• A console accessibility check that now ran real axe rules — the page had no <main> landmark.

All three were product defects the old assertions couldn't see. Zero of them were test bugs. The green had been hiding them the entire time, and the only way any of them surfaced was by making an assertion able to fail.

That's the sentence I want to keep: you don't find bugs by running your suite, you find them by making your suite able to fail.

The bookkeeping that made it stick

Two small disciplines carried the campaign:

Every fix, re-run, before the report changes. Not "fixed in batch, verified later." One case, one run, one verdict on disk, then the report row updates. The report's totals are derived from verdict.json files, never from the summary JSON — stale summaries were exactly how earlier false greens hid.

A grounded/run-only tally in the report, not just PASS/FAIL. The final state reads 92 PASS / 0 FAIL / 1 declared / 1 N/A, and 90 of the 92 grounded. That second number is the one that took weeks. It's also the only number in the report I'd defend in an audit, because it's the only one with receipts per case.

Batches of 5, then 7 — same-shape cases together, one shared stack per batch, one commit per batch. Pattern-classification isn't just a reporting convenience; it's the difference between 84 one-off fixes and 4 fixes applied 84 times.

What I keep from this

• A green suite answers "did it run." It takes deliberate work to make it answer "is it true."
• Your repo's own test suite is the cheapest, least-drifty source of assertions. Grep it before writing any.
• If a check can't fail, it will pass through anything — including the outage you wrote it to catch.
• Blocked cases get labeled, not faked. The tally 90 grounded / 2 blocked is trustworthy because the 2 are named.

All of it is in the v0.2.0 field test report, shipped with agentwatch v0.2.1 — on PyPI.

The question worth taking to your own CI tonight: of your green suite, how many assertions could fail if their claim were false? If you don't know — that number is your suite's real size. Everything else is rehearsal.

References

• Release: agentwatch v0.2.1 on GitHub · on PyPI
• v0.2.0 field test report — the assertion-shape census, the four hardening patterns, and the per-case grounding tally
• The generic driver that grounded 44 cases
• The three defects hardening surfaced: FT-BACKEND-2 (Tempo), FT-MATRIX-1 (modeled Tier-1 row), FT-LUI-1 (console <main> landmark)
• Suite: agentic-suite — the full set of agent-security tools