Right after feeling reassured by receiving test results reporting 100% coverage, a bug crops up in production—a scene we have witnessed countless times during handovers and acceptance testing in custom development. The coverage figure is not lying. The problem is that the metric indicates only that code was executed, without guaranteeing that the tests properly validated what happened inside. Even if a test runs through the target code, bugs slip through if the crucial assert checks (verifying expected values) are weak.
As a technique for exposing this coverage trap, mutation testing has been re-evaluated in 2026. Its presence in articles like Zenn's Even 100% Coverage Cannot Stop Bugs: Shifting from Test Quantity to Test Quality and its entry into the Trial ring of Thoughtworks' Technology Radar Vol. 34 (April 2026) highlight this trend. A major driver is that as LLMs mass-generate tests, coverage has become even less reliable as an indicator of quality. When an AI-written test passes without actually validating anything, it is difficult for a human to spot the issue through visual review alone. When responsible for guaranteeing deliverable quality in custom development, we believe mutation testing serves as a test of the tests and represents one of the most practical solutions available today.
Why Coverage Does Not Guarantee Quality
Coverage is merely a metric that counts which lines and branches were traversed during test execution. A test like the following can still achieve 100% coverage:
// 対象: 割引額を計算する関数
function calcDiscount(price: number, rate: number): number {
if (rate > 0.5) return price * 0.5; // 上限50%
return price * rate;
}
// "通るだけ" のテスト（カバレッジ100%だが検証が弱い）
test('calcDiscount runs', () => {
calcDiscount(1000, 0.3); // 呼ぶだけ
calcDiscount(1000, 0.8); // 呼ぶだけ
expect(true).toBe(true); // 何も検証していない
});This test hits both branches and achieves 100% coverage, but because it does not validate the return value at all, rewriting return price * 0.5 to return price * 0.4 still results in a passing test. In other words, it cannot stop a single bug. Coverage is a classic example of Goodhart's Law—when a measure becomes a target, it ceases to be a good measure. The moment it becomes the target, pressure mounts to satisfy the number using hollow tests like this.
What Mutation Testing Actually Does
Mutation testing checks whether intentionally injecting small bugs (mutants) into the target code causes existing tests to detect them and fail. If detected, the mutant is marked as killed; if undetected and the test passes, it is marked as survived. This visualizes surviving mutants as test blind spots.
| Dimension | Code Coverage | Mutation Testing |
|---|---|---|
| What it measures | Whether code was executed | Whether tests can detect bugs |
| Weak assertions | Cannot detect | Detected as survivors |
| Empty tests | Can reach 100% | Exposed by lower scores |
| LLM-generated tests | Cannot differentiate quality | Can quantify quality |
| Primary use case | Understanding execution scope | Guaranteeing validation capability |
For example, mutations such as changing > to >=, swapping return a + b for return a - b, or inverting conditions are injected automatically in high volume, and the test suite's efficacy is reported as the mutation score (percentage killed). As seen in KAKEHASHI's case study, Validating the Quality of LLM-Generated Tests with Mutation Testing, an increasing number of organizations are adopting it to measure the effectiveness of AI-written tests. If custom development relies on AI to mass-produce tests, a mechanism to ensure their quality must accompany it. This perspective of ensuring AI-generated deliverable quality connects directly to Adopting QA Automation in Custom Development (GH Media).
Minimal Implementation Example (Stryker)
For JavaScript and TypeScript, Stryker Mutator is the standard choice. It runs directly on top of existing Jest or Vitest suites.
# 1) 導入
npm i -D @stryker-mutator/core @stryker-mutator/vitest-runner
# 2) 設定（stryker.conf.json の最小例）
# - 重要モジュールだけを対象に、まず小さく始める
# - しきい値で CI を制御（high=合格 / break=失敗）
cat > stryker.conf.json <<'JSON'
{
"testRunner": "vitest",
"mutate": ["src/domain/**/*.ts"],
"thresholds": { "high": 80, "low": 60, "break": 60 }
}
JSON
# 3) 実行 → Mutation Score を出力
npx stryker runThe key is not applying it across all code from day one. Because mutation testing requires heavier execution than standard test suites, begin by narrowing focus to critical business logic where mistakes cause direct damage—such as pricing calculations, inventory management, and authorization checks—and enforce quality thresholds mechanically in CI using break. For setups in Vitest environments, Adopting Vitest 4.1 in Custom Development (GH Media) provides useful guidance.
How We Apply It in Custom Development: Turning "Tests Exist" into "Tests Stop Bugs"
Reporting during acceptance or review that "tests were written" or "coverage is at 90%" does not guarantee quality on its own. To make a higher-tier commitment for deliverables in custom development, we apply it in the following ways:
For core logic, we build a minimum mutation score threshold into contractual terms and CI pass criteria. CI fails unless code meets not only criteria like 80%+ coverage, but also 70%+ mutation score on critical modules. This establishes an environment where tests that merely run without verifying cannot pass. For one backend development client, inherited code had high coverage, but the pricing logic's mutation score was extremely low, uncovering numerous tests where assert never validated calculated amounts. Reinforcing these areas ensured subsequent releases were virtually free of billing-related bugs.
On projects where LLMs mass-produce tests, we standardize a step to run mutation testing immediately after generation and eliminate surviving mutants. This serves as a safeguard to prevent AI-generated tests from stopping at merely passing.
Common pitfalls to avoid
First, running it across the entire codebase at once causes execution times to explode. Because it is resource-intensive, limit the scope to core logic and offload CI execution to pull-request diffs or nightly runs. Second, aiming for a 100% mutation score. Equivalent mutants (mutations that are semantically identical and cannot be killed) inevitably appear, so operate with realistic thresholds (such as 70–80% for core logic). Third, leaving survivors unaddressed. Visualizing flaws is meaningless unless they are resolved, so survivors must always be triaged during reviews as either requiring added tests or being classified as equivalent. For broader quality assurance including E2E, please also read Automating QA with Playwright and AI in Custom Development (GH Media).
Conclusion: Shifting Metrics from Execution Rate to Detection Power
100% coverage does not mean everything was tested; it simply means every line was traversed. What stops bugs in production is not running code, but resilient tests capable of detecting changes. Mutation testing quantifies that detection power and makes the quality of LLM-generated tests transparent. When responsible for guaranteeing quality in custom development, establishing minimum mutation score thresholds on core logic, incorporating them into CI pass conditions, and eliminating surviving mutants offers a practical strategy to turn "tests exist" into "tests stop bugs."
If you are facing situations where coverage is high yet production bugs persist, or have concerns regarding the quality of AI-written tests, please reach out via our contact form. We can start by introducing mutation testing to your core logic.
Sources
- Even 100% Coverage Cannot Stop Bugs: Shifting from Test Quantity to Test Quality (Zenn)
- Validating the Quality of LLM-Generated Tests with Mutation Testing (KAKEHASHI Tech Blog)
- Why 100% Test Coverage Cannot Guarantee Meaningful Tests (logmi Business)
- Stryker Mutator (Official)
- Adopting QA Automation in Custom Development (GH Media)
- Adopting Vitest 4.1 in Custom Development (GH Media)
- Automating QA with Playwright and AI in Custom Development (GH Media)
