TypeScript 變異測試：AI 編碼智能體時代如何驗證測試有效性

Your tests pass. But would they catch a bug?

Imagine asking an AI coding agent to implement a feature. A few minutes later, it has generated the implementation, written unit tests, and opened a pull request.

The CI pipeline is green. All tests pass. Code coverage looks good.

Can you trust the change?

Not necessarily.

The implementation and its tests may share the same misunderstanding of the requirements. The tests might execute every line without checking the behavior that matters. And a test suite that passes today might not detect a regression tomorrow.

As AI makes code generation faster, we need to think more carefully about how we verify the code we produce.

In this tutorial, we'll explore one technique that can help: mutation testing.

We'll build a small TypeScript project, introduce a bug deliberately, install StrykerJS, and use mutation testing to discover weaknesses in our tests. Then we'll explore how the same technique can help us verify AI-generated pull requests.

By the end, you'll understand not just how mutation testing works, but why it can become an important part of a multi-layer verification strategy.

1. Let's start with a simple problem

Imagine we're building an online shop. One of our business rules is that orders worth £100 or more qualify for free shipping.

Let's implement that rule in TypeScript.

Create a file named src/shipping.ts:

export function calculateShipping(
  orderTotal: number,
): number {
  return orderTotal >= 100 ? 0 : 10;
}

The function is straightforward:

• Orders below £100 pay £10 shipping.
• Orders worth £100 or more get free shipping.

Now let's write some unit tests.

Create src/shipping.test.ts:

import { describe, expect, it } from 'vitest';
import { calculateShipping } from './shipping';

describe('calculateShipping', () => {
  it('charges shipping for small orders', () => {
    expect(calculateShipping(50)).toBe(10);
  });

  it('offers free shipping for large orders', () => {
    expect(calculateShipping(150)).toBe(0);
  });
});

Run the tests:

npx vitest run

Both tests pass.

Great! We have an implementation, some unit tests, and a green test suite.

But let's ask a slightly different question.

What would happen if somebody accidentally changed >= to >?

Our implementation would become:

export function calculateShipping(
  orderTotal: number,
): number {
  return orderTotal > 100 ? 0 : 10;
}

Now an order worth exactly £100 is charged shipping, even though the requirement says it should qualify for free shipping.

We've introduced a bug.

Run the existing tests again.

They still pass!

Why? Because we tested orders worth £50 and £150, but we never tested the boundary at £100.

Our tests execute the function, but they don't distinguish the correct implementation from this incorrect one.

This is the problem mutation testing helps us investigate.

2. What is mutation testing?

Mutation testing is a technique for evaluating how effectively your tests detect certain defects.

Instead of simply running your tests against your implementation, a mutation-testing tool deliberately makes small changes to your production code and runs the tests against each modified version.

These changes are called mutations. Each modified version is called a mutant.

For example, a mutation-testing tool might make changes such as:

The tool then observes whether your tests detect the change.

There are two particularly important outcomes.

Killed mutant: At least one test fails because of the mutation. Your tests detected the change.

Survived mutant: The tests still pass. Your tests didn't detect the change.

Let's apply this to our shipping example.

The original implementation is:

return orderTotal >= 100 ? 0 : 10;

A mutant changes it to:

return orderTotal > 100 ? 0 : 10;

Our existing tests still pass, so the mutant survives.

That's a useful finding. It tells us that the test suite doesn't distinguish these two implementations.

Notice what we're evaluating: we're not simply asking whether the implementation works for our current examples. We're asking whether the tests can detect selected changes that might introduce defects.

This is a different question from ordinary unit testing and code coverage.

3. Code coverage isn't the same as test effectiveness

You may already measure code coverage in your projects.

Coverage tells you which parts of your code were executed during testing. It's useful, but it doesn't necessarily tell you whether your assertions are strong enough.

Consider this test:

it('calculates shipping', () => {
  calculateShipping(100);
});

The function executes. Depending on the coverage measurement, the test may contribute to branch or line coverage.

But the test doesn't assert the expected result.

The implementation could return £10 instead of £0, and the test would still pass.

Now consider:

it('offers free shipping at exactly £100', () => {
  expect(calculateShipping(100)).toBe(0);
});

This test checks the actual business rule.

It also kills our mutant because the mutated implementation returns £10.

We can summarize the difference:

• Code coverage: Did our tests execute this code?
• Assertions: Did our tests verify the expected behavior?
• Mutation testing: Would our tests detect selected changes to the implementation?

These techniques complement each other. Mutation testing doesn't replace coverage, and a mutation score doesn't prove correctness. Together, however, they can provide more useful evidence about the effectiveness of a test suite.

4. Let's set up StrykerJS

We'll use StrykerJS, a mutation-testing framework for JavaScript and TypeScript.

We'll assume you already have Node.js and npm installed.

If you're following along from scratch, create a project:

mkdir mutation-testing-demo
cd mutation-testing-demo
npm init -y

Install TypeScript and Vitest:

npm install --save-dev typescript vitest

Create the src directory and add the two files from the previous sections.

Now run:

npx vitest run

Make sure your tests pass before continuing.

Initialize StrykerJS

Run:

npx stryker init

Follow the prompts to configure Stryker for your project. Select TypeScript and the Vitest test runner where applicable.

Stryker's setup options can vary with the version and project configuration, so follow the current prompts rather than assuming every project requires an identical configuration.

Official documentation: StrykerJS — Getting Started.

Once initialization is complete, inspect the generated configuration and make sure the mutation scope includes our production file:

src/shipping.ts

Our test file, src/shipping.test.ts, should not be treated as production code to mutate.

You can configure the mutation scope explicitly if necessary. For example, in a TypeScript Stryker configuration, the relevant pattern might be:

mutate: ['src/**/*.ts', '!src/**/*.test.ts'],

Keep the configuration generated by your installed version as the starting point, and adapt it to your repository's conventions.

Run your first mutation test

Execute:

npx stryker run

Stryker will run your tests, generate mutations in the configured production code, and evaluate the results.

The first run may take longer than your ordinary unit tests because Stryker needs to evaluate multiple mutants.

Once it finishes, inspect the terminal output and generated HTML report.

Look for surviving mutants associated with the comparison in calculateShipping.

If the comparison changes from >= to > and all our tests still pass, we've found the weakness we deliberately introduced.

Congratulations: you've just used mutation testing to discover a gap in your test suite.

5. Understand the mutation report

A mutation report contains more than just a score.

The most useful outcomes to understand are:

Killed

A test detected the mutation and failed. This is generally what we want for a meaningful behavioral change.

Survived

The tests passed despite the mutation. Investigate whether this reveals missing test coverage or an assertion that doesn't adequately verify the requirement.

No coverage

The mutated code wasn't exercised by the relevant tests. You may need to investigate whether the code is tested at all.

Timeout

The tool couldn't obtain a normal result within its configured time limit. This doesn't necessarily mean your tests detected the mutation.

Equivalent mutant

The mutation doesn't change observable behavior for the relevant inputs. No meaningful test can distinguish it from the original implementation.

Not every surviving mutant is a defect, and not every mutant needs to be killed. Some mutations are equivalent, some aren't meaningful for your requirements, and some may reveal behavior you don't actually need to constrain.

The purpose of the report is to help you make better decisions about your tests.

What about the mutation score?

Stryker calculates a mutation score based on the results of mutation testing.

A simplified formula is:

Mutation score = killed mutants / valid mutants × 100%

For example, suppose a run produces 100 valid mutants and your tests kill 80 of them. The mutation score is 80%.

That can be a useful signal, but be careful about treating it as a universal quality metric.

Two projects can have the same mutation score while having very different levels of risk. The score depends on the mutations generated, the code being analyzed, and which results are excluded.

A surviving mutation in payment authorization may matter much more than a surviving mutation in an unimportant formatting helper.

Focus on understanding meaningful surviving mutants, not maximizing a percentage at any cost.

6. Improve the tests and run mutation testing again

Let's fix the weakness we discovered.

Our business rule is explicit: orders worth £100 or more qualify for free shipping.

We should test the boundary directly.

Add this test to src/shipping.test.ts:

it('offers free shipping at exactly £100', () => {
  expect(calculateShipping(100)).toBe(0);
});

Run the ordinary test suite:

npx vitest run

The test passes against the correct implementation.

Now run mutation testing again:

npx stryker run

The mutation that changes >= to > should now be killed by the boundary test.

We've improved the test suite because it now detects a meaningful defect.

Notice that we didn't add a test just to improve a score. We added one because the requirement explicitly defines what should happen at the boundary.

This distinction is important.

A mutation-testing report is a source of questions, not an instruction to write more assertions blindly.

For every surviving mutant, ask:

1. Does this change affect behavior that matters?
2. What does the requirement say the correct behavior should be?
3. Should the test suite detect this change?
4. If so, what is the smallest meaningful test that would detect it?

This approach fits naturally into Test-Driven Development.

TDD helps us implement behavior from requirements. Mutation testing challenges whether our tests can detect selected deviations from that behavior.

7. From local experiments to pull request verification

So far, we've run mutation testing manually.

The next step is to integrate it into CI.

Imagine an AI coding agent creates a PR that modifies calculateShipping and adds several unit tests.

The ordinary CI pipeline runs:

1. Type-checking.
2. Linting.
3. Unit and integration tests.
4. Other project-specific checks.

Everything passes.

Now Stryker runs against the configured production code. It discovers that the tests don't detect a meaningful mutation in the shipping calculation.

The PR report can expose that weakness before the change is merged.

This is particularly useful for AI-generated code because the implementation and its tests may have been produced by the same agent, based on the same interpretation of the requirements.

The tests passing is evidence that the implementation satisfies the assertions the agent wrote. It isn't independent evidence that those assertions capture every important requirement.

Mutation testing adds another challenge: it deliberately changes the implementation and checks whether the test suite notices.

Should mutation testing block every PR?

I'd start with a gradual approach.

Stage 1: Report findings without blocking merges.

Run mutation testing on a small set of important modules. Learn how long it takes, inspect surviving mutants, and identify noisy or irrelevant results.

Stage 2: Gate critical code.

Introduce stronger requirements for areas where correctness is particularly important, such as permissions, pricing, validation, or state transitions.

Stage 3: Track regressions over time.

Use scheduled runs and historical reports to identify weaknesses across the broader codebase.

For larger projects, Stryker's incremental mode can reuse eligible previous results to reduce the work needed on subsequent runs. See the incremental testing documentation.

One important distinction: mutating changed production files is not the same as testing only the new tests in a PR. You want relevant existing tests to participate in verifying the changed behavior as well.

You should also avoid making a repository-wide mutation score an immediate merge requirement. A small PR shouldn't necessarily be responsible for fixing every weakness in a legacy test suite.

8. Let the AI agent investigate surviving mutants

We can take the workflow a step further.

Instead of using mutation testing only as a CI gate, we can feed its findings back to the AI agent.

Imagine this sequence:

1. An agent implements a feature and writes unit tests.
2. CI runs the ordinary checks.
3. Stryker finds a surviving mutation.
4. The agent receives the mutation report and relevant requirements.
5. The agent determines whether the mutation exposes a genuine testing gap.
6. The agent proposes an additional test or explains why the mutant isn't meaningful.
7. CI reruns the checks against the revised PR.

For our shipping example, the agent might receive a report showing that changing >= to > doesn't cause any tests to fail.

The agent can then inspect the business rule and recognize that the boundary at £100 is important.

It adds the boundary test, reruns the suite, and verifies that the mutation is now killed.

This creates a feedback loop in which the agent can improve not just the implementation, but also the evidence we have about that implementation.

However, there's a critical caveat.

Don't let the agent blindly optimize the mutation score.

An agent could make a mutant fail by adding an assertion that encodes an incorrect assumption. It could also change production code simply to make the existing tests pass.

The correct behavior must come from explicit requirements, domain rules, or other trustworthy sources—not from the mutation report alone.

The agent can propose a fix, but the final verification should run independently in CI against the revised commit.

9. Mutation testing is one layer in a Swiss cheese model

This brings us to the broader reason mutation testing matters.

No single verification technique catches every defect.

Type checking can detect certain classes of errors, but it doesn't establish that a business rule is correct.

Unit tests can verify specified behavior, but they only check the cases and assertions we've written.

Mutation testing can challenge the effectiveness of those tests, but it can't guarantee that the original requirements were understood correctly.

Integration tests can expose problems between components, but they won't necessarily identify every defect in an individual component.

Code review can identify design and requirement problems, but human reviewers also have limited time and attention.

Each layer has gaps.

This is the idea behind a Swiss cheese model of verification: arrange multiple imperfect defenses so that a defect that passes through one layer has other opportunities to be detected.

A practical model might include:

Layer 1: Types and static analysis

Catch incompatible interfaces, invalid types, and selected classes of programming mistakes.

Layer 2: Unit and integration tests

Check specified behavior, interactions, and component integration.

Layer 3: Mutation testing

Challenge whether tests detect selected changes to the implementation.

Layer 4: Contract and architecture checks

Verify boundaries, invariants, and integration assumptions.

Layer 5: Independent review and production feedback

Identify misunderstood requirements, broader design risks, and failures that automated checks miss.

The layers are complementary. They aren't completely independent: mutation testing relies on the test suite, and both can be based on the same misunderstood requirement.

But combining different forms of verification can provide stronger evidence than relying on any one of them.

This connects directly to the principles discussed in Revisiting the Toyota Production System in the Age of Coding Agents. In particular, the idea of building quality into the production process is relevant here: detect abnormalities as early as possible, rather than allowing them to move downstream.

Mutation testing can be one of the automated checks that helps us detect weaknesses before a pull request is merged.

10. Why this matters when code generation scales

AI changes the economics of software development.

An agent can produce a substantial implementation and its accompanying tests much faster than a human might write them manually.

But generating code is not the same as establishing that the code is correct.

If the volume of changes increases dramatically, manually reviewing every implementation and every test may become increasingly difficult.

We need verification mechanisms that scale with the rate of change.

This is the connection to When Code Gets Cheap, Verification Becomes Expensive.

Mutation testing is not a complete solution to that problem. It has a computational cost, and it only evaluates the mutations it generates. It cannot establish that the implementation satisfies every requirement or that every important defect is represented by a mutation.

What it can do is make one important question cheaper to investigate:

Can our tests detect plausible defects in the code we're producing?

And because the process can be automated, we can apply that challenge repeatedly as code changes.

The objective isn't to maximize code generation or mutation scores in isolation. It's to increase our ability to produce and verify valuable changes without letting the volume of generated code overwhelm our verification capacity.

Conclusion

Mutation testing is a relatively simple idea with important consequences.

We deliberately modify production code, run our tests, and observe whether those tests detect the changes. When meaningful mutants survive, we investigate whether our tests are missing important behavioral checks.

With TypeScript and StrykerJS, it's possible to introduce this technique into an existing project without replacing your test runner or abandoning your current TDD workflow.

And when AI agents generate code and tests, mutation testing provides a useful additional challenge: rather than accepting a green test suite at face value, we can investigate whether it detects selected mistakes.

It doesn't prove that our code is correct. No single testing technique does.

But that's precisely why it belongs in a multi-layer verification strategy.

When code generation scales, verification must scale with it. Mutation testing helps us challenge one of the most important defenses we already have: our tests.

Further reading

• From Rigidity to Explicitness: How AI Changes the Role of Constraints in Software
• When Code Gets Cheap, Verification Becomes Expensive
• Revisiting the Toyota Production System in the Age of Coding Agents
• StrykerJS documentation