如何用 Stryker 對 AI 智能體修改的差異執行變異測試，檢查測試是否有效

• Book: AI Agents Pocket Guide
• Also by me: Thinking in Go (2-book series) — Complete Guide to Go Programming + Hexagonal Architecture in Go
• My project: Hermes IDE | GitHub — an IDE for developers who ship with Claude Code and other AI coding tools
• Me: xgabriel.com | GitHub

Coverage on the agent's new file is 100% and both of its tests are green. It added a discount rule and wrote two tests for it. You are about to merge. The rule and its tests:

// src/discount.ts
export function discount(total: number): number {
  if (total >= 100) {
    return total * 0.1;
  }
  return 0;
}

// src/discount.test.ts
import { expect, test } from "vitest";
import { discount } from "./discount";

test("big orders get a discount", () => {
  expect(discount(150)).toBeGreaterThan(0);
});

test("small orders get none", () => {
  expect(discount(50)).toBe(0);
});

Every line runs. Every branch runs. Now change >= to >. Both tests still pass, because nothing checks an order of exactly 100. Change * to /. Still green: 1,500 is greater than 0 too. Two real bugs, zero red tests.

Coverage tells you a line executed. It does not tell you a test would fail if that line were wrong. Mutation testing measures the second thing, and you can point it at only the code the agent touched.

What a mutant is, and why "survived" is the useful word

A mutation tester makes small, deliberate changes to your source, one at a time, and runs your tests against each changed version. Each changed version is a mutant. StrykerJS is the best-known one for JavaScript and TypeScript. Its mutator list includes both edits above, and more: a >= b becomes a > b or a < b, arithmetic operators get swapped, if conditions get replaced with true or false, and whole blocks get emptied.

Stryker's definitions are blunt. A mutant is Killed when "at least one test failed while this mutant was active." It Survived when "all tests passed while this mutant was active, the mutant survived. You're missing a test for it." No coverage means no test even ran that code. The mutation score is detected mutants (killed plus timed out) divided by valid mutants.

So the survivors are a list of lines where a plausible bug would ship and your tests would stay green. That is a better review checklist than a coverage number, and it is the right question to ask about tests an agent wrote to make its own change pass.

Killed mutants get caught by a test. Survivors walk out the door.

This is a different check from diff coverage. diff-cover will "automatically find diff lines that need test coverage," which catches lines no test runs at all. The discount tests above would pass diff-cover with a perfect score. Mutation testing catches the lines that run but are never really asserted on.

Step 1: install Stryker with the Vitest runner

StrykerJS 10 is the current major (v10.1.1, October 23, 2026), and v10.0.0 dropped Node.js 20: you need Node 22 or newer. The getting-started guide runs an interactive setup with npm init stryker@latest. If you already know you use Vitest, install the pieces directly:

npm i -D @stryker-mutator/core \
  @stryker-mutator/vitest-runner

Then a small stryker.config.json at the repo root:

{
  "testRunner": "vitest",
  "reporters": ["clear-text", "progress", "html", "json"],
  "thresholds": { "high": 80, "low": 60, "break": 60 }
}

A few things from the configuration reference matter here. The json reporter writes reports/mutation/mutation.json, which you will read in step 3. The html reporter writes reports/mutation.html. With break set, Stryker exits with code 1 when the score falls below it, so a script or CI job can fail on it. The Vitest runner always uses per-test coverage analysis, so each mutant only runs the tests that cover it, and it does not support Vitest browser mode.

Leave mutate out of the config. The script in step 2 supplies it.

Step 2: mutate only the files the agent's branch changed

The agent's work lives on its own branch. Three-dot diff gives you what that branch changed since it split from main. The git docs put it this way: git diff A...B "is equivalent to git diff $(git merge-base A B) B," so commits that landed on main in the meantime don't leak in.

#!/usr/bin/env bash
# scripts/mutate-diff.sh: mutate only this branch's changes
set -euo pipefail
base="${1:-main}"

files=$(git diff --name-only --diff-filter=d \
  "$base"...HEAD -- 'src/*.ts' \
  ':!*.test.ts' ':!*.spec.ts' | paste -sd, -)

if [ -z "$files" ]; then
  echo "No changed source files since $base."
  exit 0
fi

npx stryker run --mutate "$files"

Why each piece is there:

• --diff-filter=d drops deleted files. In git, a lowercase filter letter excludes that status, so there is no point asking Stryker to mutate a file that no longer exists.
• The :! pathspecs drop test files. This matters because a --mutate value on the command line "will completely replace" the one in the config, per the Stryker docs, so Stryker's default of skipping test files no longer applies.
• paste -sd, - joins the paths with commas, which is the list form --mutate takes.

Three-dot diff compares commits, so it misses edits the agent hasn't committed yet. If you want those too, diff the merge base against the working tree instead: git diff --name-only $(git merge-base main HEAD). New untracked files still won't show up until they are added.

If a branch touches one function in a 600-line file, you can go narrower. mutate accepts line ranges like src/app.js:1-11, so you can turn the hunks of git diff -U0 into ranges if a file-level run is too slow. File level is a good default anyway: survivors in the untouched code around the change tell you how much the agent's edit leans on code nobody really tests.

For repeat runs while the agent iterates, add --incremental. Stryker stores results in reports/stryker-incremental.json and reuses them on the next run, so only mutants affected by new edits get tested again.

Step 3: turn survivors into a list someone can act on

The HTML report is good for browsing. For handing work back to an agent, you want plain lines. The JSON report follows the mutation-testing report schema: files is keyed by path, and each mutant has a mutatorName, a location with a 1-based start.line, an optional replacement, and a status.

# scripts/survivors.jq
.files
| to_entries[]
| .key as $file
| .value.mutants[]
| select(.status == "Survived"
    or .status == "NoCoverage")
| "\($file):\(.location.start.line)"
  + "  \(.mutatorName)"
  + "  -> \(.replacement // "")"

jq -r -f scripts/survivors.jq \
  reports/mutation/mutation.json

Each line names a file, a line number, the kind of mutation, and the code that replaced the original. For the discount example, working it through by hand, you'd expect two survivors, on the if line and the return line inside it: the >= boundary and the * to / swap. The negated comparison, the true/false conditions and the emptied block all break one of the two tests, so those get killed.

Each survivor tells you which assertion is missing. A test at exactly 100 kills the boundary mutant. Asserting the actual amount instead of "greater than zero" kills the arithmetic one:

test("discount starts at exactly 100", () => {
  expect(discount(100)).toBeCloseTo(10);
  expect(discount(99)).toBe(0);
});

test("discount is ten percent", () => {
  expect(discount(150)).toBeCloseTo(15);
});

toBeCloseTo is there because multiplying by 0.1 isn't always exact in floating point (3 * 0.1 is 0.30000000000000004), and the rate may change later.

Not every survivor is a missing test. Some mutants are equivalent: the change produces code that behaves the same, and no test could kill it. Logging lines and defensive defaults often land here. Read the list as review questions, not as a to-do list the agent must clear to zero.

Step 4: run it in the agent's worktree, before you land

This part is where Hermes IDE fits, and I build Hermes, so weigh it with that in mind. In Hermes, ⌘N (Ctrl+Shift+N on Windows and Linux) starts each task on its own hermes/<name> branch in its own git worktree. That is exactly the setup mutate-diff.sh wants: the branch has only the agent's changes, and running Stryker there doesn't touch your main checkout or another agent's folder.

The Review Desk's Worktrees tab: one Hermes worktree each for orders-api, web-app and orders-types under ~/code, with each worktree's hermes/ branch, task, age and disk size.

Open a plain terminal in the task's worktree and run the two scripts. If you want the run to happen every time, Hermes reads a done_when list from .hermes/worktree.toml as the task's default checks (recipe docs), so your own checks decide when a task counts as done:

# .hermes/worktree.toml
setup = ["npm ci"]
done_when = ["npm test", "bash scripts/mutate-diff.sh"]

Think about cost before you do that. Even with only the diff mutated and per-test coverage, a mutation run executes tests many times. On a big change it can take minutes. Running it by hand before Land is a reasonable middle ground.

Then hand the survivors back. The Review Desk (⌘G, Ctrl+Shift+G on Windows and Linux) shows the task's changes by turn or by file, and a line comment there goes back to the agent that wrote the line. Put a comment on the if line of discount.ts saying "a mutant changing >= to > survives; add a test at exactly 100," and the agent gets a concrete, checkable job instead of "write better tests." The other route is pasting the survivors.jq output into the agent's terminal as its next prompt.

The Review Desk (⌘G) in By turn view for a Claude Code task in orders-api: two turns listed, turn 1's diff of src/db.ts, the changed-files list, and a commit message drafted from the turns.

When the rerun comes back with the survivors you agreed to fix gone, Land commits, opens a pull request or merges locally into the branch the task started from, and it can be undone.

None of steps 1 to 3 needs Hermes. git worktree add -b agent/discount ../discount gives you the same isolated branch, and the scripts run the same way there.

What this does and doesn't tell you

A high mutation score on the diff says the agent's tests react when its code changes in the ways Stryker knows how to change it. It does not say the code does what the ticket asked. A test suite can kill every mutant and still encode the wrong business rule, so you still read the diff.

It also doesn't replace property-based tests, which search for inputs that break an invariant. Mutation testing grades the tests you have. Property tests generate new ones: use them to find inputs you didn't think of, and Stryker to check the assertions on the ones you did.

What it does give you, cheaply, is a way to stop taking "all tests pass" at face value when the same agent wrote the code and the tests.

If this was useful

Agents grading their own work is one of the core guardrail problems: the check has to come from outside the loop that produced the change. My AI Agents Pocket Guide covers that and the other patterns for keeping autonomous agents honest, in a short read.

If you want each agent's work on its own branch with a review step before it lands, Hermes IDE is source-available under BSL 1.1 and runs on macOS, Windows and Linux. It is on GitHub if you want to look under the hood.