git.stella-ops.org/docs/testing/mutation-testing-guide.md

Mutation Testing Guide

This guide documents the integration and usage of Stryker.NET mutation testing in StellaOps.

Overview

Mutation testing measures test suite effectiveness by introducing small code changes (mutants) and verifying that tests detect them. Unlike line coverage, mutation testing answers: "Would my tests catch this bug?"

Installation

Stryker.NET is configured as a local dotnet tool:

# Restore tools (includes Stryker.NET)
dotnet tool restore

# Verify installation
dotnet stryker --version

Configuration

Solution-Level Configuration

Base configuration is at stryker-config.json in the solution root. Module-specific configs override these settings.

Module Configurations

Module	Config Path	Mutation Break Threshold
Scanner.Core	src/Scanner/__Libraries/StellaOps.Scanner.Core/stryker-config.json	60%
Policy.Engine	src/Policy/StellaOps.Policy.Engine/stryker-config.json	60%
Authority	src/Authority/StellaOps.Authority/stryker-config.json	65%

Running Mutation Tests

Single Module

# Navigate to module directory
cd src/Scanner/__Libraries/StellaOps.Scanner.Core

# Run mutation testing
dotnet stryker

# With specific config
dotnet stryker --config-file stryker-config.json

All Configured Modules

# From solution root
dotnet stryker --solution StellaOps.Router.slnx

CI Mode (Threshold Enforcement)

# Fails if mutation score below threshold
dotnet stryker --break-at-score 60

Understanding Results

Mutation Score

Mutation Score = (Killed Mutants / Total Mutants) × 100

• Killed: Test failed when mutant was introduced (good!)
• Survived: Test passed with mutant present (test gap!)
• No Coverage: No test covered the mutated code
• Timeout: Test timed out (usually treated as killed)

Thresholds

Level	Score	Meaning
High	≥80%	Excellent test effectiveness
Low	≥60%	Acceptable, improvements needed
Break	<50%	Build fails, critical gaps

Example Output

All mutants have been tested, and your mutation score has been calculated
╔═══════════════════════════════════════════════════════════════════════╗
║ Mutation Testing Report                                                ║
╠═══════════════════════════════════════════════════════════════════════╣
║ Mutants tested: 156                                                    ║
║ Mutants killed: 134                                                    ║
║ Mutants survived: 18                                                   ║
║ Mutants no coverage: 4                                                 ║
║ Mutation score: 85.90%                                                 ║
╚═══════════════════════════════════════════════════════════════════════╝

Common Mutators

Mutator	Original	Mutant
Comparison	>=	>
Equality	==	!=
Boolean	true	false
Logical	&&	||
Arithmetic	+	-
NullCoalescing	??	(remove)

Fixing Survived Mutants

1. Analyze the Report

Open the HTML report in .stryker/output/<module>/mutation-report.html.

2. Identify the Gap

Look at the survived mutant:

// Original
if (score >= threshold) { return "PASS"; }

// Mutant (survived!)
if (score > threshold) { return "PASS"; }

3. Add Missing Test

[Fact]
public void Should_Pass_When_Score_Equals_Threshold()
{
    var score = 60;
    var threshold = 60;
    
    var result = EvaluateScore(score, threshold);
    
    result.Should().Be("PASS"); // Now kills the >= to > mutant
}

Best Practices

1. Focus on Critical Modules First

Prioritize mutation testing for:

• Security-critical code (Authority, Signer)
• Business logic (Policy decisions, Scanner matching)
• Boundary conditions

2. Don't Chase 100%

Some mutants are false positives or equivalent mutants. Aim for 80%+ on critical modules.

3. Use Baseline Mode

Enable baseline to only test changed files:

dotnet stryker --with-baseline:main

4. Exclude Non-Critical Code

Exclude from mutation testing:

• DTOs and models
• Generated code
• Migrations
• UI components

CI Integration

Mutation testing runs in CI:

mutation-test:
  runs-on: ubuntu-latest
  steps:
    - uses: actions/checkout@v4
    - name: Run Stryker
      run: |
        dotnet tool restore
        dotnet stryker --break-at-score 60

Troubleshooting

Slow Execution

• Use --concurrency to control parallelism
• Enable coverage-analysis: perTest for smarter mutant selection
• Use --since:main to only test changed code

Out of Memory

• Reduce --concurrency value
• Exclude large test projects

Timeout Issues

• Adjust --timeout setting
• Some infinite loop mutants may timeout (this is expected)

References

• Stryker.NET Documentation
• Mutation Testing Theory
• StellaOps Test Suite Overview: docs/19_TEST_SUITE_OVERVIEW.md