Mutation Testing
Mutation testing systematically breaks your code and checks whether your tests catch the breaks. A high mutation score means your tests are actually testing the logic, not just executing code. Code coverage is a vanity metric; mutation score is real quality.
1 The Hook
A Wellington fintech is rightly proud of its dashboards: 100% code coverage on the module that calculates KiwiSaver employer contributions. Green across the board. Management treats that number as proof the maths is safe.
Then a payroll run goes out with employer contributions calculated at the wrong rate for hundreds of staff. The investigation finds the bug had been there for months — and so had a test that “covered” the calculation. The test called the function and checked it did not throw an exception. It never asserted the actual dollar figure. The line was executed, so coverage counted it as tested, but nothing checked the answer was right.
That is the gap mutation testing exposes. If you had deliberately changed a + to a - in that calculation and re-run the suite, every test would still have passed — a screaming signal that the tests have no teeth. Coverage told a comforting lie. The only honest question is: if I break the code on purpose, do the tests notice?
Every team I have introduced mutation testing to runs it once, sees 1,400 surviving mutants, and quietly shelves it. The bottleneck is never the tool — it is the blank-canvas problem. The unlock is to scope your first run to a single method: the one method where a silent bug would cost the most. On a recent NZ health-sector project we scoped PIT to a single eligibility decision function — eight lines of code. Six survivors in ten minutes, two of which were real gaps in the boundary logic. The team fixed them the same afternoon. That story travels further inside an organisation than any presentation about mutation score thresholds ever will. Start absurdly small. Win visibly. Then expand.
2 The Rule
Coverage proves a line ran; it does not prove a test checked the result. Mutation testing makes small, deliberate changes to the code and re-runs the suite — a test that does not fail when the logic is broken is not testing anything, and the mutation score, not coverage, is the honest measure of test quality.
3 The Analogy
A smoke alarm you never test with actual smoke.
A smoke alarm on the ceiling with a glowing green light looks reassuring. But the green light only tells you it has power — it does not tell you it will scream when the kitchen catches fire. The only way to know is to hold a smoking match under it and check it actually goes off. Plenty of houses have a powered, “working” alarm that stays silent in a real fire because nobody ever tested it with smoke.
Code coverage is the green light: it confirms the test ran. Mutation testing is holding the match under the alarm: it introduces real “smoke” (a broken operator, a flipped return value) and checks the test actually screams. A test that stays silent when the code is broken is a dead alarm with a green light on.
What it is
Mutation testing is a technique for measuring test quality. A mutation tool (like PIT or Stryker) automatically modifies your code in small ways — changing operators, boundary conditions, return values — and re-runs your test suite. If a test fails after the mutation, the test “kills” the mutant and is doing its job. If no test fails, the mutant “survives,” meaning your tests didn’t catch the change.
The key insight: 100% code coverage is meaningless if your tests don’t actually assert anything. A test that just calls a function without checking the result “covers” that function but doesn’t test it. Mutation testing forces you to write tests that have teeth.
Coverage is not quality. A test suite with 100% code coverage but 40% mutation score is barely testing. The coverage number is useless; the mutation score tells you the truth.
Code coverage vs. mutation score
| Coverage claim | Example | Mutation truth |
|---|---|---|
| 100% line coverage | int x = calculatePrice(item); — test runs this line but never asserts the value of x |
The test doesn’t actually verify calculatePrice works. Mutation score: near 0%. |
| 100% branch coverage | if (age >= 18) { ... } else { ... } — test hits both branches but never asserts correctness in each |
Changing >= to > is not caught. Mutation score: weak. |
| All methods called | Test calls calculateTotal() but doesn’t assert the return value is correct |
Changing the calculation (+ to -, * to /) is not caught. Mutation score: low. |
How mutation testing works
- Mutation engine scans your code and identifies places to mutate: operators, conditionals, return values, etc.
- For each mutation location, the tool makes a single small change (e.g., change
>=to>). - The tool runs your entire test suite against the mutated code.
- If a test fails, the mutant is killed (your test caught the change — good).
- If all tests pass, the mutant survives (your test suite did not catch the change — gap in your testing).
- The tool reports a mutation score: (killed mutants / total mutants) * 100%.
Common mutations
| Mutation | Example | Catches |
|---|---|---|
| Conditional boundary | >= becomes > |
Off-by-one errors; tests that don’t check boundary values |
| Arithmetic operator | + becomes - |
Tests that don’t verify the actual calculation |
| Logical operator | && becomes || |
Tests that don’t cover both branches of complex conditions |
| Return value | return true; becomes return false; |
Tests that don’t assert the return value |
| Constant | return 10; becomes return 0; |
Tests that don’t verify specific values |
| Statement deletion | Remove an entire line of code | Tests that don’t verify all side effects |
Interpreting mutation results
Mutation score
The mutation score is a percentage: (killed mutants / total mutants) * 100%. What does each range mean?
- 0–30%: Critical gaps. Your tests are barely testing anything. You need more assertions.
- 30–60%: Weak tests. You cover happy paths but miss edge cases, boundary conditions, error handling.
- 60–80%: Decent. Most logic is tested, but there are still gaps (usually error paths, rare branches).
- 80%+: Strong. Your tests have teeth. Nearly all mutations are caught.
- 90%+: Excellent. Your tests are thorough and rigorous.
Survivors (surviving mutants)
The mutation tool reports which mutants survived (which mutations your tests did not catch). For each survivor, ask: is it legitimate, or do I need to write a better test?
- Legitimate survivor: The mutation produces code that is functionally equivalent. E.g., changing
if (x) { y(); return 1; } else { return 1; }toif (x) { y(); } return 1;— both branches do the same thing, so it’s a false positive. Ignore it or configure the tool to skip. - Illegitimate survivor: The mutation breaks logic, but your test didn’t catch it. You need a better test or an additional test case.
When to use mutation testing
- Code quality audit — when you inherit a codebase or suspect test quality is weak, run mutation testing to get an honest assessment.
- Critical code — for business logic, payment calculations, security code, run mutation testing to ensure tests are thorough.
- After refactoring — if you refactored code, run mutation tests to ensure your tests still catch the old bugs (i.e., the tests, not the code structure, are what matters).
- Not for continuous CI — mutation testing is slow (often 10–50x slower than running tests normally). Run it locally, before commit, or nightly, not on every CI run.
Worked example: Testing a discount calculator
Consider a simple method:
public BigDecimal applyDiscount(BigDecimal price, int discountPercent) {
if (discountPercent > 100) {
throw new IllegalArgumentException("Discount cannot exceed 100%");
}
if (discountPercent < 0) {
throw new IllegalArgumentException("Discount cannot be negative");
}
BigDecimal discountAmount = price.multiply(
BigDecimal.valueOf(discountPercent).divide(BigDecimal.valueOf(100))
);
return price.subtract(discountAmount);
}
Weak test suite (50% coverage, but likely 20% mutation):
@Test
public void testDiscount() {
BigDecimal result = applyDiscount(BigDecimal.valueOf(100), 10);
// Oops: test runs the code but never asserts the result
// Mutation: if the calculation changed to +discountAmount instead of -discountAmount,
// this test would pass (because it doesn't assert anything).
}
@Test
public void testDiscountOverHundred() {
applyDiscount(BigDecimal.valueOf(100), 101); // Just calls the method, never checks if exception thrown
}
Strong test suite (50% coverage, 85% mutation):
@Test
public void testValidDiscount() {
BigDecimal result = applyDiscount(BigDecimal.valueOf(100), 10);
assertEquals(BigDecimal.valueOf(90), result); // Assert the calculation
}
@Test
public void testZeroDiscount() {
BigDecimal result = applyDiscount(BigDecimal.valueOf(100), 0);
assertEquals(BigDecimal.valueOf(100), result); // Boundary: no discount
}
@Test
public void testMaxDiscount() {
BigDecimal result = applyDiscount(BigDecimal.valueOf(100), 100);
assertEquals(BigDecimal.valueOf(0), result); // Boundary: full discount
}
@Test
public void testDiscountOverHundred() {
assertThrows(IllegalArgumentException.class,
() -> applyDiscount(BigDecimal.valueOf(100), 101));
}
@Test
public void testNegativeDiscount() {
assertThrows(IllegalArgumentException.class,
() -> applyDiscount(BigDecimal.valueOf(100), -1));
}
Mutation results: The strong test suite kills mutations like:
- Changing
> 100to>= 100(caught by testMaxDiscount). - Changing
< 0to<= 0(would fail testZeroDiscount). - Changing
subtracttoadd(caught by testValidDiscount and boundary tests). - Removing the exception throws (caught by testDiscountOverHundred and testNegativeDiscount).
Tools
- PIT (Pitest) — Java; the gold standard. Integrates with JUnit, produces detailed HTML reports.
- Stryker — JavaScript/TypeScript; very good HTML reports and dashboard integration.
- Mutiny — Python; simpler, good for getting started.
- cargo-mutants — Rust; built into the Rust ecosystem.
- Mutagen — .NET; mutation testing for C# and F#.
Common pitfalls
- Running mutation tests in CI for every commit — it’s too slow. Run locally or nightly.
- Ignoring equivalent mutants — some mutations produce equivalent code. The tool will report them as survivors; verify they’re truly equivalent before ignoring.
- Chasing 100% mutation score — some code (e.g., getters, logging) produces lots of equivalent mutants. Aim for 80%+ on critical code; 60%+ on non-critical.
- Testing too broadly — mutation test only the code you want to improve. Testing third-party libraries or generated code wastes time.
- Not fixing weak tests — once you see survivors, don’t ignore them. Write tests to kill them or document why they’re equivalent.
4 Industry Reality
- Most teams have never run mutation testing. Even teams with mature CI pipelines and 80%+ code coverage often have zero experience with mutation testing. You will regularly be the first person to introduce it — expect curiosity mixed with scepticism from developers who have always treated coverage as the gold standard.
- Legacy codebases produce thousands of survivors and nobody knows where to start. Running Pitest or Stryker on a 5-year-old Java or TypeScript codebase routinely surfaces 2,000+ surviving mutants. Senior testers triage rather than boil the ocean: identify the highest-risk modules (payment processing, authorisation, core business rules) and focus mutation effort there first. Covering a legacy e-commerce platform's checkout logic before touching logging utilities is the pragmatic call.
- Slow test suites make mutation testing genuinely painful. If your test suite already takes 8 minutes, mutation testing may push total time to over an hour. In practice this means teams run it on a nightly schedule or as a pre-release gate, not on every pull request. Configuring Pitest's incremental mode or Stryker's dry-run to only mutate recently changed code is the real-world workaround — not a textbook option, an operational necessity.
- Developers sometimes push back on mutation-driven test changes. When you surface surviving mutants and ask for better tests, some developers will argue the survivors are "equivalent mutants" or that the code path is "not worth testing." Senior testers learn to differentiate — an equivalent mutant genuinely cannot be caught by any test, while a surviving boundary mutation on a GST calculation is a real gap. Being able to walk a developer through why a specific survivor matters (and which real-world bug it maps to) is what gets the fix written.
- In NZ financial and health contexts, mutation testing is a compliance conversation. Organisations subject to FMA oversight or the Privacy Act 2020 increasingly need evidence that their test suites verify correctness, not just coverage. Mutation score reports (especially for KiwiSaver calculators, loan repayment logic, or patient data processing) are becoming part of audit packs — having a 30% mutation score on your payment module is a very different conversation with an auditor than a 90% score backed by Pitest HTML reports.
5 When to Use It — and When Not To
✓ Use it when
- You're working on business-critical logic — payment calculations, tax/GST functions, KiwiSaver eligibility rules, authorisation decisions — where a wrong answer has real consequences
- You inherit a codebase and need an honest audit of test quality before making changes; coverage numbers are already high but you don't trust them
- You've just completed a refactor and want to verify the test suite would still catch the bugs the original code had — not just that coverage is maintained
- You're trying to justify upgrading test quality to a developer or team lead; a Pitest report showing 40% mutation score is far more persuasive than "I think our tests are weak"
- You're doing a pre-release quality gate on a critical module and need evidence for a release sign-off or audit pack
✗ Skip it when
- You're testing UI rendering logic — visual snapshot tools like Percy or Chromatic catch pixel regressions far better than mutation operators can
- The code is auto-generated or third-party (ORM models, protobuf output, vendor SDKs) — you don't own that logic and the mutant count will inflate your survivor list with noise
- You have no existing test suite or coverage is under 40% — mutation testing requires tests to kill mutants; fix the test gap first or you'll just see 95% survivors with no signal
- Your test suite runs for more than 20 minutes already and you have no plan for incremental mutation runs — mutation testing will be too slow to be actionable without tooling investment first
- You're in a rapid prototype phase or building a spike — the code will be thrown away; spend the time writing exploratory tests for the business logic instead
Context guide
How the right level of mutation testing effort changes based on project context.
| Context | Priority | Why |
|---|---|---|
| Revenue NZ or FMA-regulated financial calculations (KiwiSaver contributions, PIE tax, PAYE withholding logic) | Essential | A wrong dollar figure triggers regulatory non-compliance. A Pitest HTML report scoped to the calculation module is the artefact that answers an FMA audit, not a coverage badge. |
| Benefits NZ or CoverNZ benefit-eligibility and entitlement logic before a legislative rule change | Essential | Eligibility functions are riddled with boundary conditions (income thresholds, age cutoffs, duration requirements). Surviving boundary mutants before the rule change goes live are the highest-risk undetected bugs you can find. |
| HealthNZ (HealthNZ) or Pharmac clinical-decision or dosage-calculation code | Essential | Patient safety and Medsafe obligations mean weak assertions are not a quality debt item — they are a safety risk. An 85%+ mutation score on dosage or eligibility logic should be a release gate. |
| Harbour Bank or Pacific Bank payment processing, interest calculations, or loan repayment logic | High | RBNZ prudential standards and CCCFA obligations mean calculation errors carry both financial and legal exposure. Mutation testing scoped to the core arithmetic is proportionate effort for the risk. |
| TransitNZ or TransitNZ RUC, tolling, or licensing fee calculation logic | High | Public-facing fee calculations are subject to OIA scrutiny and public trust expectations. A surviving boundary mutant on a RUC rate band is the kind of bug that surfaces in media rather than in a test report. |
| Pacific Air or Spark internal tooling, reporting dashboards, or CRM workflows | Medium | Business impact is real but consequences are recoverable. Scope mutation testing to the most business-critical functions (pricing, loyalty point accrual) rather than the full application. Run nightly, not on every commit. |
| Rapid prototype, internal spike, or throw-away tooling with no compliance obligations | Low | The code will be replaced before it matters. Write exploratory tests to validate the concept; invest mutation effort on the production version once the approach is proven. |
Trade-offs
What you gain and what you give up when you choose mutation testing.
| Advantage | Disadvantage | Use instead when… |
|---|---|---|
| Exposes tests that assert nothing — surfaces the gap between “code ran” and “logic was verified.” The most honest measure of test quality currently available. | Slow. On a suite that already takes 8 minutes, mutation testing may take over an hour. Without incremental mode configured, it blocks pipelines and gets removed. | You have no existing tests (coverage below 40%) — mutation testing produces noise, not signal. Fix the test baseline first. |
| Generates a concrete, prioritised list of gaps — surviving mutants point at specific lines and operators, not vague advice to “add more tests.” | Produces equivalent mutants — a proportion of survivors are false positives that no test could ever catch. Triaging them costs time and confuses less-experienced testers. | The problem is untested input ranges, not weak assertions. Property-based testing (e.g. Hypothesis, fast-check) generates far more input combinations and scales better for that problem. |
| Produces audit-quality artefacts. Pitest HTML reports and Stryker dashboards give an FMA or OAG auditor evidence that test quality was measured, not assumed. | Cannot detect semantic bugs. A 90% mutation score says nothing about whether the Revenue NZ levy rate, the rounding rule, or the HealthNZ eligibility criterion is correct — only that the logic was consistently applied. | The defect risk is in requirements correctness, not code logic — exploratory testing, specification walkthroughs with domain experts, and BDD scenario reviews are the right tools there. |
| Forces teams to write precise assertions. The discipline of killing survivors permanently improves how developers write tests — the benefit outlasts any single run. | High initial investment on legacy codebases. Running Pitest on a 5-year-old Java monolith may produce 2,000+ survivors with no clear starting point, demoralising the team if not scoped deliberately. | The codebase is auto-generated (ORM models, protobuf, generated address validators) or third-party — mutants inflate the survivor list with noise unrelated to your own logic. |
Enterprise reality
How mutation testing changes at 200–300-developer scale in NZ enterprise — where manual triage becomes automation, compliance evidence is non-negotiable, and tool choices are locked in for years.
- At 10+ squad scale, nobody triages survivors by hand. Organisations like CloudBooks and Revenue NZ configure Pitest or Stryker with a curated exclusion list (equivalent mutants, generated code, logging paths) maintained as a shared artefact in version control — any squad adding a new exclusion must peer-review it, because an unreviewed exclusion is the same as a deleted test.
- The Privacy Act 2020 and NZISM (New Zealand Information Security Manual) require demonstrable assurance on systems handling personal data. At TechServNZ or Benefits NZ scale, mutation score reports on authentication, authorisation, and data-masking logic are included in audit packs alongside penetration test results — a green coverage badge alone will not satisfy an OAG or GCDO reviewer.
- Tool standardisation matters more than tool quality. A large NZ bank (Harbour Bank, Coastal Bank) typically mandates one mutation framework per language stack — Pitest for Java, Stryker for TypeScript — so mutation reports from every squad share the same format and thresholds. Teams do not get to pick their own tool; they inherit a configured baseline and are expected to keep the score above the floor set by the Platform Engineering chapter.
- Cross-squad boundary logic is the hardest problem. When 15 squads share a core eligibility or pricing library (common at TeleNZ or HealthNZ), a mutation survivor in the shared module affects every squad's release train. Enterprise practice is to gate shared-library releases on an 85%+ mutation score and require the releasing squad to attach the Pitest HTML report to the change request — not to the PR, to the ITSM ticket, so it survives the audit trail even after the branch is deleted.
◆ What I would do
Professional judgment — when to reach for mutation testing, when to skip it, and what to watch for.
>= flipped to > on an income threshold is one character, but it miscategorises clients earning exactly the threshold amount. I would run Stryker or Pitest scoped to that eligibility function before the PR is merged, look specifically for surviving conditional-boundary mutants, and add tests at the new threshold value, one cent below it, and one cent above it. This takes 20 minutes. The alternative is a silent off-by-one that affects real benefit recipients and is not discovered until a client complaint months later.withHistory or Stryker’s incremental flag) so only changed code is mutated on each run, and move the full-suite mutation report to a nightly schedule rather than every PR.The bottom line: Mutation testing is not a metric to manage — it is a tool for finding the specific tests your suite is missing. Scope it to the riskiest module, present survivors as concrete questions about correctness, and let the score improve as a side effect of better assertions.
6 Best Practices
- Start with a single critical module, not the whole codebase. Run Pitest or Stryker scoped to one package — the one with the highest business risk. Getting a first mutation report in 5 minutes beats a 2-hour full-codebase run that nobody acts on.
- Always pair mutation testing with boundary value analysis. Surviving boundary mutants (
>=to>) are almost always solved by boundary tests. If you run mutation testing and see conditional boundary survivors, your first move is to check your equivalence partitions and add tests at the exact boundary. - Triage survivors before writing new tests. For each surviving mutant, decide: real gap, equivalent mutant, or dead code. Annotate equivalent mutants in the tool config so they stop appearing. Only then write tests for real gaps — otherwise you're writing tests that can never fail, which is worse than no test.
- Set realistic targets by code criticality, not a blanket number. Aim for 85%+ on payment and authorisation logic, 70%+ on general business rules, 50%+ on infrastructure/config code. A blanket "80% everywhere" target causes teams to write pointless tests on getters and logging to hit the number.
- Use mutation testing to review test quality in PRs for critical modules. Add Pitest or Stryker to the pipeline specifically for the
src/payments,src/auth, or equivalent packages, and fail the build if mutation score drops below your threshold. Keep it out of the general CI pipeline where slowness kills adoption. - When a mutation survivor points at a gap, write the test to describe the bug, not just kill the mutant. "Test that the discount cannot reduce the price below zero" is a better test name than "killMutant47." The test should be readable on its own and survive even if the tool changes.
- Document surviving equivalent mutants with a comment near the code. A comment like
// mutation-ignore: equivalent — both branches return early with the same resultkeeps future testers from re-investigating the same false positive every time the tool runs. - Run mutation testing after writing new tests, not before. Write your test suite normally, run mutation testing to find gaps, fill the gaps, then re-run to confirm your score improved. Using mutation testing as a TDD loop (write the mutant, write the test to kill it) is a valid advanced pattern but confusing for teams new to the technique.
- Treat a drop in mutation score the same way you'd treat a drop in coverage. If a PR reduces mutation score from 82% to 71%, that's a signal that new code was added without adequate tests — worth a conversation before merging, especially on financial logic.
- In NZ compliance contexts, save the HTML reports. Pitest's HTML output and Stryker's dashboard JSON are artefacts you can attach to a release pack or audit response. A mutation score report from the day of release is evidence that test quality was measured, not assumed.
7 Common Misconceptions
❌ Myth: If I have 100% code coverage, mutation testing will just confirm everything is fine.
Reality: Code coverage and mutation score measure completely different things. Coverage records that a line was executed; mutation testing checks that a test actually verified what that line does. Teams routinely have 100% coverage and a 25–40% mutation score because their tests call functions without asserting the return values. The Wellington KiwiSaver example at the top of this page is not hypothetical — this pattern appears in virtually every codebase when mutation testing is run for the first time.
❌ Myth: A 100% mutation score means my tests are perfect.
Reality: A 100% mutation score means every mutation the tool could generate was killed — but mutation tools only generate a finite, predefined set of mutations (arithmetic operators, boundary conditions, return values, etc.). They cannot generate semantic mutations that represent real bugs in business logic, like using the wrong tax rate for the wrong tax year, or failing to apply a NZ-specific rounding rule. Mutation score is a strong signal, not a guarantee. Use it alongside exploratory testing, specification review, and domain-expert walkthrough for critical systems.
❌ Myth: Every surviving mutant needs a new test to kill it.
Reality: A significant proportion of survivors in any real codebase are equivalent mutants — changes that produce code that is functionally identical to the original. For example, removing a redundant null check on a value that can never be null, or swapping two branches that both return the same constant. Writing a test to "kill" an equivalent mutant is impossible by definition — you'd be writing a test that can never fail, which pollutes the test suite with meaningless assertions. The correct approach is to review each survivor, classify it as a real gap or equivalent mutant, and configure the tool to exclude confirmed equivalents rather than chasing 100%.
Senior engineer insight
Mutation testing changed how I think about test assertions the moment I realised that asserting something is returned is almost never the same as asserting the right thing is returned. The shift is subtle but it rewires how you review PRs: I now look at every new test and ask "if I deleted this assertion, would the test still pass?" — if yes, the assertion is theatre. On a Wellington payments platform we ran Stryker for the first time and found 60% of our tests were pure theatre: they called the methods, timed out nothing, asserted nothing that mattered. We had 94% coverage and felt safe.
The most common mistake: treating mutation score like a coverage percentage and aiming to hit a number. Teams write trivial tests to boost the score without asking what behaviour those tests actually verify. Run mutation testing to find the gaps, fix the gaps, then let the score improve as a side effect — not the other way around.
From the field
A NZ government agency had a CI pipeline with mandatory 80% coverage before merge. The team was proud of it — it had taken months to enforce. When we ran PIT scoped to their benefit-calculation module as part of a pre-release review, the mutation score came back at 31%. The existing tests were structured as integration smoke tests: call the endpoint, check the HTTP 200 comes back, move on. Nobody had failed a build in eighteen months, but nobody had caught a wrong dollar figure either.
The generalising lesson: coverage gates and mutation gates measure completely different properties. A coverage gate tells you code was executed. A mutation gate tells you code was verified. For anything with financial or compliance consequences — and in the NZ public sector, that is a lot of things — you need both, and you need to communicate clearly to stakeholders that a green coverage badge is not a correctness certificate.
8 Now You Try
Three graded exercises — spot, fix, then build. Write your answer, run it for AI feedback, then compare to the model answer.
A GST calculator has function gst(amount) { return amount * 0.15; } (NZ GST is 15%). The only test is:assertNotNull(gst(100)); — it just checks the result is not null. Mutation tool applies these mutants: (a) * 0.15 → * 0.10; (b) * 0.15 → + 0.15; (c) return amount * 0.15 → return 0. For each mutant, say whether the test KILLS it or it SURVIVES, and why. Then state what coverage this test reports.
Show model answer
All three mutants SURVIVE. (a) * 0.10 — Survives. gst(100) returns 10 instead of 15. Both are non-null, so assertNotNull still passes. The test never checks the value. (b) + 0.15 — Survives. gst(100) returns 100.15 instead of 15. Still non-null, test still passes. (c) return 0 — Survives. gst(100) returns 0. In most languages 0 is non-null, so assertNotNull passes. (Even if 0 were treated as falsy, the point stands: the test asserts existence, not correctness.) Code coverage reported: 100% line coverage. The single line of gst() is executed by the test, so coverage looks perfect — while the mutation score is 0% (0 of 3 killed). This is the whole lesson: 100% coverage, 0% mutation score, a calculator nobody is actually testing. The fix is assertEquals(15, gst(100)).
A function decides KiwiSaver first-home-withdrawal eligibility: boolean canWithdraw(int yearsMember) { return yearsMember >= 3; } (you must have been a member at least 3 years). The current test is assertTrue(canWithdraw(5));. A mutation run shows two survivors: >= 3 → > 3, and >= 3 → >= 1. Write the additional test cases needed to kill both survivors, and explain why the existing single test misses them.
Show model answer
Kill ">=3 → >3": test the boundary itself. assertTrue(canWithdraw(3)); // exactly 3 years must be eligible Under the mutant >3, canWithdraw(3) returns false, so this assertion fails — mutant killed. Kill ">=3 → >=1": test a value below the boundary that must be ineligible. assertFalse(canWithdraw(2)); // 2 years must NOT be eligible Under the mutant >=1, canWithdraw(2) returns true, so this assertion fails — mutant killed. Why the existing test misses both: assertTrue(canWithdraw(5)) only checks a value comfortably inside the eligible range. 5 is >= 3, > 3, and >= 1, so it returns true under the original code AND under both mutants — it can never distinguish them. You have to test at and just outside the boundary (3 and 2) to catch off-by-one and loosened-condition mutants. This is exactly why mutation survivors so often point at missing boundary tests.
A road-user-charges (RUC) calculator: cost(km) charges $0 for the first 0 km, a flat rate per 1,000 km, and throws if km is negative. Design a test suite aimed at a high mutation score: list the test cases, what each one asserts, and which kinds of mutants each is designed to kill (arithmetic, boundary, return value, exception). Use concrete NZ-flavoured values.
Show model answer
A mutation-killing RUC test suite: Test 1 — assertEquals(0, cost(0)). Asserts the zero-km case returns exactly $0. Kills constant mutants (return 0 → return 1) and confirms the base case. Boundary at the bottom of the range. Test 2 — assertEquals(76, cost(1000)) (using a concrete flat rate, e.g. $76 per 1,000 km). Asserts the actual dollar figure. Kills arithmetic mutants (* → /, the rate constant changed) — this is the one that catches the wrong-multiplier bug coverage would miss. Test 3 — assertEquals(152, cost(2000)). A second multiple checks the calculation scales correctly, killing mutants that happen to pass at a single value (e.g. an off-by-a-constant that only matches at 1,000). Test 4 — assertThrows(IllegalArgumentException.class, () -> cost(-100)). Asserts the negative input throws. Kills statement-deletion / removed-guard mutants (delete the throw) and return-value mutants on the error path. Hardest to kill: equivalent mutants (changes that produce functionally identical behaviour, e.g. a redundant branch) — these legitimately survive and should be reviewed and excluded rather than chased. Also constant mutants in rarely-meaningful spots (logging) inflate the survivor list without indicating a real gap. Aim for high mutation score on the calculation and guard logic, not 100% everywhere.
Why teams fail here
- Running mutation testing on the entire codebase first time — 3,000 survivors with no triage plan is demoralising and the initiative dies within a week.
- Treating every survivor as something that needs a new test, including equivalent mutants that can never be caught — pollutes the test suite with assertions that can never fail.
- Adding mutation testing to the CI pipeline without configuring incremental mode — turns a 6-minute pipeline into a 90-minute one and guarantees it gets removed inside a sprint.
- Conflating a high mutation score with a correct test suite — a 90% mutation score on a GST calculator still says nothing about whether the GST rate, the rounding rule, or the zero-rated supply logic is right.
Key takeaway
Coverage proves a line ran; mutation testing proves a test would scream if the logic were wrong — and only the second one gives you the right to feel safe.
How this has changed
The field moved. Here is how Mutation Testing evolved from its origins to current practice.
Richard Lipton proposes mutation testing in a class project at Yale. Richard DeMillo, Richard Lipton, and Fred Sayward publish the formal theory in 1978. The idea: introduce small bugs (mutations) into the code and measure whether existing tests detect them. Tests that miss mutations are weak.
Mutation testing is academically promising but computationally expensive — generating and testing thousands of mutants on 1980s hardware is impractical at scale. Used only in research contexts.
PIT (Pitest) for Java makes mutation testing feasible with incremental analysis — only mutating changed code and code covered by changed tests. Mutant survival rates below 20% become an achievable target for teams with fast test suites.
Stryker (JavaScript), mutmut (Python), and cross-language mutation testing tools emerge. Mutation scores complement branch coverage metrics — revealing tests that execute code without asserting on outcomes. The concept of "survived mutants as test design debt" enters QA thinking.
Higher-order mutation testing — introducing multiple simultaneous mutations — remains computationally expensive but increasingly feasible with distributed execution. AI tools can prioritise which mutations to generate based on fault history and code complexity. Mutation testing is the most reliable measure of test suite quality currently available.
Self-Check
Click each question to reveal the answer.
Q1: Why can a test suite have 100% code coverage but a near-zero mutation score?
Coverage only records that a line was executed, not that anything checked the result. A test that calls a function but never asserts its return value “covers” that code while verifying nothing. Mutation testing breaks the code on purpose; if no test fails, the tests were executing the line without checking it — high coverage, zero real testing.
Q2: What does it mean when a mutant is “killed” versus when it “survives”?
A mutant is killed when at least one test fails after the code is mutated — the tests caught the change, which is good. A mutant survives when every test still passes despite the broken code — a gap, meaning no test checks that behaviour. Mutation score is killed mutants divided by total mutants.
Q3: Why do surviving mutants so often point at missing boundary tests?
A common mutation flips a boundary operator, e.g. >= to >. A test using a value comfortably inside the range behaves identically under both versions, so it cannot kill that mutant. Only a test at and just outside the exact boundary distinguishes them — so survivors of boundary mutations reveal that you tested the middle, not the edge.
Q4: What is an equivalent mutant, and how should you treat it?
An equivalent mutant is a change that produces functionally identical behaviour, so no test could ever distinguish it — it survives legitimately and is a false positive. You should verify it really is equivalent, then exclude it (or configure the tool to skip it) rather than writing a pointless test or treating it as a gap.
Q5: Why is mutation testing usually run locally or nightly rather than on every CI commit?
It is slow — often 10 to 50 times slower than a normal test run — because it re-runs the suite once per mutant. Running it on every commit would clog the pipeline. Run it locally before committing, on critical modules, or on a nightly schedule, and treat the score as a guide to write better tests rather than a per-commit gate.
Q: Your team is reviewing the test suite for Benefits NZ’s benefit-eligibility calculator before a major rule change. Coverage is 91% but you suspect the tests are weak. Would you reach for mutation testing or property-based testing first, and why?
Mutation testing first. The specific concern is whether existing tests verify correctness, not whether rules hold across all input combinations — that is exactly what mutation score measures. Run Pitest or Stryker scoped to the eligibility module: if you see surviving arithmetic or boundary mutants (e.g. >= to > on income thresholds), you have concrete evidence of weak assertions to fix before the rule change goes live. Property-based testing is the better follow-up once you have confirmed the tests have teeth, because it will generate edge-case inputs the mutation tool cannot imagine — but you cannot property-test your way to a high mutation score, and you cannot mutation-test your way to exhaustive input coverage. Do both; start with mutation testing because the symptom is untrusted assertions, not untested input space.
Q: What is the key difference between mutation testing and code coverage, and why does that difference matter when signing off a KiwiSaver contribution calculator for a compliance audit?
Code coverage records whether a line was executed during a test; mutation testing checks whether a test would fail if that line’s logic were wrong. A test can execute the contribution calculation on every line while asserting only that the result is non-null — 100% coverage, but the maths is untested. For a compliance audit under FMA oversight, a mutation score report is the honest artefact: it shows the test suite actually verified the dollar figures, not just that the code ran. Attaching a Pitest HTML report with an 85%+ score on the calculation module gives an auditor evidence of test quality that a coverage badge simply cannot provide.
Q: When should you NOT run mutation testing, even on safety-critical code?
Skip it when your existing test suite is too thin to produce a useful signal — if coverage is below roughly 40%, nearly every mutant will survive, and the report tells you nothing you did not already know. Also skip it on auto-generated code (ORM models, protobuf output, generated NZ address validators), third-party libraries, and UI rendering logic, where mutation operators produce noise rather than gaps in your own logic. And avoid running it in CI on every pull request if your suite already takes more than about 20 minutes: without incremental mutation support configured, it will block pipelines and kill adoption. Fix the test baseline first, then introduce mutation testing where it has signal.
Q: A developer on your team says, “Our Stryker report shows 78% mutation score, so we have a strong test suite.” What is wrong with this claim and how do you respond?
The score is a useful signal but not a guarantee of correctness. Mutation tools generate a predefined set of syntactic mutations — flipped operators, changed constants, deleted statements — but they cannot generate semantic bugs: using the wrong CoverNZ levy rate for the wrong employment category, applying NZ GST at 15% when the correct rate for a specific supply is 0%, or misreading a HealthNZ eligibility rule. A 78% mutation score means 78% of those syntactic changes were caught, which is good evidence the tests have assertions — but it says nothing about whether the tests are testing the right behaviour. The correct response is to treat the score as a floor, not a ceiling: pair it with exploratory testing, specification walkthroughs with domain experts, and boundary value analysis to cover the semantic gaps mutation tools cannot reach.
Best practice: Use mutation testing during development to drive better tests, not as a metric to hit. If your mutation score is 60%, the goal is not “get to 80%” — the goal is “write tests that actually verify the logic.” The score is a side effect of better testing.
The Mutation Testing (Specialised) track goes further: multi-lesson deep-dive with NZ-specific compliance context, advanced tooling, and practice exercises. Recommended once you have the fundamentals on this page.