Mutation Testing with Pest: Why 100% Code Coverage Is an Illusion in the AI Era

It has never been easier to achieve 100% line coverage in a Laravel project. You point an AI coding agent at a service class, prompt it to write tests using Pest, and within thirty seconds you receive a pull request with fifteen green tests and every single line marked green in your coverage report.

And yet, that same service class can silently deploy an off-by-one error or an inverted permission check straight into production.

The reason is simple: code coverage measures execution, not verification. Coverage tells you that an execution thread passed through a line of code during a test run. It says nothing about whether the test made an assertion that would actually fail if that line were wrong.

When humans write tests slowly, we usually write assertions tied to our assumptions. When AI writes tests at machine speed, it naturally optimises for the path of least resistance: execute the code without throwing an unhandled exception, assert status(200) or $response->toBeTrue(), and declare victory.

If we want to trust automated tests in an era of AI-generated diffs, we need a tool that tests the tests themselves. In the PHP ecosystem, that tool is mutation testing with Pest.

The anatomy of a fake 100% test

Consider a simple order discount calculator in a Laravel application:

namespace App\Services;

class DiscountCalculator
{
    public function calculate(float $subtotal, int $loyaltyYears, bool $isVip): float
    {
        if ($isVip || $loyaltyYears >= 5) {
            return $subtotal * 0.80; // 20% discount
        }

        if ($subtotal > 100.00) {
            return $subtotal * 0.90; // 10% discount
        }

        return $subtotal;
    }
}

An AI assistant asked to “write comprehensive unit tests for DiscountCalculator” will often hand you something like this:

it('calculates discounts correctly', function () {
    $calculator = new App\Services\DiscountCalculator();

    $vipDiscount = $calculator->calculate(100.0, 2, true);
    expect($vipDiscount)->toBeGreaterThan(0.0);

    $loyaltyDiscount = $calculator->calculate(100.0, 6, false);
    expect($loyaltyDiscount)->toBeFloat();

    $bulkDiscount = $calculator->calculate(150.0, 1, false);
    expect($bulkDiscount)->toBeLessThan(150.0);

    $regular = $calculator->calculate(50.0, 1, false);
    expect($regular)->toEqual(50.0);
});

Run pest --coverage:

• Every branch was visited.
• Every line executed.
• Coverage report: 100%.
• Confidence score: completely misplaced.

Look at those assertions:

• expect($vipDiscount)->toBeGreaterThan(0.0) does not check if the discount was 20%. If a developer changes 0.80 to 0.99, or deletes the discount entirely and returns $subtotal, the test still passes.
• expect($loyaltyDiscount)->toBeFloat() checks the return type, which PHP’s type system already guarantees.
• expect($bulkDiscount)->toBeLessThan(150.0) accepts an order total of $1.00 or $149.99 without complaint.

The suite is green, the coverage badge is bright green, and the business logic is entirely unguarded.

What mutation testing does differently

Instead of measuring which lines your tests touch, mutation testing actively tampers with your code in memory and runs your test suite against each mutation:

1. Mutation: The engine modifies an operator or value. For example, it changes $loyaltyYears >= 5 to $loyaltyYears > 5, or replaces * 0.80 with * 0.0.
2. Execution: It runs the relevant tests against the mutated code.
3. Evaluation:
   • If your tests fail, the mutation was caught. The mutant is killed (good).
   • If your tests pass, your assertions did not notice the broken logic. The mutant escaped / survived (bad).

A mutation score is the percentage of killed mutants over total generated mutants:

$$\text{Mutation Score} = \frac{\text{Killed Mutants}}{\text{Total Mutants}} \times 100$$

If your test suite has 100% line coverage but only a 40% mutation score, more than half of your logic can be broken or deleted without a single test complaining.

Running mutation testing in Pest

Pest provides native mutation testing out of the box. You don’t need to configure complex XML schemas or separate runners.

To run mutation testing on your project:

./vendor/bin/pest --mutate

When run against our sloppy discount test, Pest’s mutation engine outputs something like this:

  FAIL  Mutants survived!

  • [Escaped] LogicalAnd / GreaterThanOrEqualTo
    App\Services\DiscountCalculator: Line 7
    - if ($isVip || $loyaltyYears >= 5) {
    + if ($isVip || $loyaltyYears > 5) {

  • [Escaped] Multiplication / FloatNegation
    App\Services\DiscountCalculator: Line 8
    - return $subtotal * 0.80;
    + return $subtotal * 0.00;

  • [Escaped] GreaterThan / GreaterThanOrEqualTo
    App\Services\DiscountCalculator: Line 11
    - if ($subtotal > 100.00) {
    + if ($subtotal >= 100.00) {

  Tests:    1 passed
  Mutants:  6 generated, 3 killed, 3 survived
  Score:    50.00%

The output gives you the exact line number, the exact mutation that survived, and why your assertions let it slip through.

Writing tests that kill mutants

Killing mutants requires testing boundary conditions and exact outputs, not broad ranges:

it('applies 20% discount for VIP customers regardless of tenure', function () {
    $calculator = new App\Services\DiscountCalculator();

    expect($calculator->calculate(100.00, 0, true))->toBe(80.00);
});

it('applies 20% discount when loyalty tenure hits exactly five years', function () {
    $calculator = new App\Services\DiscountCalculator();

    expect($calculator->calculate(100.00, 4, false))->toBe(100.00)
        ->and($calculator->calculate(100.00, 5, false))->toBe(80.00);
});

it('applies 10% discount strictly above 100 currency units', function () {
    $calculator = new App\Services\DiscountCalculator();

    expect($calculator->calculate(100.00, 1, false))->toBe(100.00)
        ->and($calculator->calculate(100.01, 1, false))->toBe(90.009);
});

With these tests:

• If >= 5 is changed to > 5, the second test fails because 5 years yields 100.00 instead of 80.00. Mutant killed.
• If * 0.80 is changed to * 0.90 or 0.0, the exact equality check fails immediately. Mutant killed.
• If > 100.00 becomes >= 100.00, the boundary check at 100.00 fails. Mutant killed.

Now the mutation score is 100%, and that number actually means something.

The practical reality: mutation testing is slow

The obvious trade-off is computational cost. If you have 500 unit tests and Pest generates 1,200 mutants, running your test suite 1,200 times locally will bring your fan to full blast and destroy your feedback loop.

You do not run --mutate across your entire repository on every keystroke. You run it strategically.

1. Scope to the git diff during local development

When working on a feature or reviewing an AI-generated branch, run mutation testing only against unstaged or committed branch changes:

# Only mutate files modified in the working tree
./vendor/bin/pest --mutate --dirty

# Or target a specific domain service directly
./vendor/bin/pest --mutate --path=app/Services/Billing

This keeps the run time under 10 seconds while giving you immediate feedback on the tests you just added or changed.

2. Set realistic thresholds in CI

Do not demand a 100% mutation score across an entire legacy codebase on day one. Start by enforcing thresholds on critical domains:

./vendor/bin/pest --mutate --path=app/Domain/Payments --min=80

The --min=80 flag causes Pest to exit with code 1 if the mutation score drops below 80%. This prevents regressions in your financial, authorization, and billing logic without blocking teams over minor boilerplate.

3. Exclude pure boilerplate

Not all code warrants mutation testing. DTOs, simple Eloquent relationships, and configuration wrappers waste CPU cycles mutating trivial getters. Configure your pest.php configuration to focus on business logic:

// tests/Pest.php
pest()->extend(Tests\TestCase::class)
    ->in('Feature', 'Unit');

And in your CLI options or configuration, target your domain services and actions where conditional decisions actually happen.

A rule of thumb for reviewing AI-written tests

When an AI copilot or autonomous agent submits a pull request with unit tests, use this checklist before hitting approve:

1. Beware of soft assertions: Flag any test that relies primarily on toBeGreaterThan(), toBeFloat(), toBeArray(), or assertNotNull() where a deterministic value was expected.
2. Search for boundary tests: If the code has > or <, is there a test for the exact boundary value, one unit below, and one unit above?
3. Run --mutate --dirty: Before you spend ten minutes manually reading diffs, let Pest run mutations on the PR’s modified files. If half the mutants survive, the tests are decorative. Send the PR back with the mutation report.

Code coverage was invented in an era where writing tests was tedious, so simply proving you had written a test was an achievement. In the era of autonomous generation, tests are cheap. High-leverage software craft is no longer about how many lines of test code you have — it is about how many ways your tests refuse to let a bug survive.