git.stella-ops.org/docs/technical/testing/cross-cutting-testing-guide.md

Cross-Cutting Testing Standards Guide

This guide documents the cross-cutting testing standards implemented for StellaOps, including blast-radius annotations, schema evolution testing, dead-path detection, and config-diff testing.

Sprint Reference: SPRINT_20260105_002_005_TEST_cross_cutting

---

Table of Contents

1. Overview
2. Blast-Radius Annotations
3. Schema Evolution Testing
4. Dead-Path Detection
5. Config-Diff Testing
6. CI Workflows
7. Best Practices

---

Overview

Cross-cutting testing standards ensure consistent test quality across all modules:

Standard	Purpose	Enforcement
Blast-Radius	Categorize tests by operational surface	CI validation on PRs
Schema Evolution	Verify backward compatibility	CI on schema changes
Dead-Path Detection	Identify uncovered code	CI with baseline comparison
Config-Diff	Validate config behavioral isolation	Integration tests

---

Blast-Radius Annotations

Purpose

Blast-radius annotations categorize tests by the operational surfaces they affect. During incidents, this enables targeted test runs for specific areas (e.g., run only Auth-related tests when investigating an authentication issue).

Categories

Category	Description	Examples
Auth	Authentication, authorization, tokens	Login, OAuth, DPoP
Scanning	SBOM generation, vulnerability scanning	Scanner, analyzers
Evidence	Attestation, evidence storage	EvidenceLocker, Attestor
Compliance	Audit, regulatory, GDPR	Compliance reports
Advisories	Advisory ingestion, VEX processing	Concelier, VexLens
RiskPolicy	Risk scoring, policy evaluation	RiskEngine, Policy
Crypto	Cryptographic operations	Signing, verification
Integrations	External systems, webhooks	Notifications, webhooks
Persistence	Database operations	Repositories, migrations
Api	API surface, contracts	Controllers, endpoints

Usage

using StellaOps.TestKit;
using Xunit;

// Single blast-radius
[Trait("Category", TestCategories.Integration)]
[Trait("BlastRadius", TestCategories.BlastRadius.Auth)]
public class TokenValidationTests
{
    [Fact]
    public async Task ValidToken_ReturnsSuccess()
    {
        // Test implementation
    }
}

// Multiple blast-radii (affects multiple surfaces)
[Trait("Category", TestCategories.Integration)]
[Trait("BlastRadius", TestCategories.BlastRadius.Auth)]
[Trait("BlastRadius", TestCategories.BlastRadius.Api)]
public class AuthenticatedApiTests
{
    // Tests that affect both Auth and Api surfaces
}

Requirements

• Integration tests: Must have at least one BlastRadius annotation
• Contract tests: Must have at least one BlastRadius annotation
• Security tests: Must have at least one BlastRadius annotation
• Unit tests: BlastRadius optional but recommended

Running Tests by Blast-Radius

# Run all Auth-related tests
dotnet test --filter "BlastRadius=Auth"

# Run tests for multiple surfaces
dotnet test --filter "BlastRadius=Auth|BlastRadius=Api"

# Run incident response test suite
dotnet run --project src/__Libraries/StellaOps.TestKit \
  -- run-blast-radius Auth,Api --fail-fast

---

Schema Evolution Testing

Purpose

Schema evolution tests verify that code remains compatible with previous database schema versions. This prevents breaking changes during:

• Rolling deployments (new code, old schema)
• Rollbacks (old code, new schema)
• Migration windows

Schema Versions

Version	Description
N	Current schema (HEAD)
N-1	Previous schema version
N-2	Two versions back

Using SchemaEvolutionTestBase

using StellaOps.Testing.SchemaEvolution;
using Testcontainers.PostgreSql;
using Xunit;

[Trait("Category", TestCategories.SchemaEvolution)]
public class ScannerSchemaEvolutionTests : PostgresSchemaEvolutionTestBase
{
    public ScannerSchemaEvolutionTests()
        : base(new SchemaEvolutionConfig
        {
            ModuleName = "Scanner",
            CurrentVersion = new SchemaVersion("v2.1.0",
                DateTimeOffset.Parse("2026-01-01")),
            PreviousVersions =
            [
                new SchemaVersion("v2.0.0",
                    DateTimeOffset.Parse("2025-10-01")),
                new SchemaVersion("v1.9.0",
                    DateTimeOffset.Parse("2025-07-01"))
            ],
            ConnectionStringTemplate =
                "Host={0};Port={1};Database={2};Username={3};Password={4}"
        })
    {
    }

    [Fact]
    public async Task ReadOperations_CompatibleWithPreviousSchema()
    {
        var result = await TestReadBackwardCompatibilityAsync(
            async (connection, version) =>
            {
                // Test read operations against old schema
                var repository = new ScanRepository(connection);
                var scans = await repository.GetRecentScansAsync(10);
                return scans.Count >= 0;
            });

        Assert.True(result.IsSuccess);
    }

    [Fact]
    public async Task WriteOperations_CompatibleWithPreviousSchema()
    {
        var result = await TestWriteForwardCompatibilityAsync(
            async (connection, version) =>
            {
                // Test write operations
                var repository = new ScanRepository(connection);
                await repository.CreateScanAsync(new ScanRequest { /* ... */ });
                return true;
            });

        Assert.True(result.IsSuccess);
    }
}

Versioned Container Images

Build versioned PostgreSQL images for testing:

# Build all versions for a module
./devops/docker/schema-versions/build-schema-images.sh scanner

# Build specific version
./devops/docker/schema-versions/build-schema-images.sh scanner v2.0.0

# Use in tests
docker run -d -p 5432:5432 ghcr.io/stellaops/schema-test:scanner-v2.0.0

---

Dead-Path Detection

Purpose

Dead-path detection identifies uncovered code branches. This helps:

• Find untested edge cases
• Identify potentially dead code
• Prevent coverage regression

How It Works

1. Tests run with branch coverage collection (Coverlet)
2. Cobertura XML report is parsed
3. Uncovered branches are identified
4. New dead paths are compared against baseline
5. CI fails if new dead paths are introduced

Baseline Management

The baseline file (dead-paths-baseline.json) tracks known dead paths:

{
  "version": "1.0.0",
  "activeDeadPaths": 42,
  "totalDeadPaths": 50,
  "exemptedPaths": 8,
  "entries": [
    {
      "file": "src/Scanner/Services/AnalyzerService.cs",
      "line": 128,
      "coverage": "1/2",
      "isExempt": false
    }
  ]
}

Exemptions

Add exemptions for intentionally untested code in coverage-exemptions.yaml:

exemptions:
  - path: "src/Authority/Emergency/BreakGlassHandler.cs:42"
    category: emergency
    justification: "Emergency access bypass - tested in incident drills"
    added: "2026-01-06"
    owner: "security-team"

  - path: "src/Scanner/Platform/WindowsRegistryScanner.cs:*"
    category: platform
    justification: "Windows-only code - CI runs on Linux"
    added: "2026-01-06"
    owner: "scanner-team"

ignore_patterns:
  - "*.Generated.cs"
  - "**/Migrations/*.cs"

Using BranchCoverageEnforcer

using StellaOps.Testing.Coverage;

var enforcer = new BranchCoverageEnforcer(new BranchCoverageConfig
{
    MinimumBranchCoverage = 80,
    FailOnNewDeadPaths = true,
    ExemptionFiles = ["coverage-exemptions.yaml"]
});

// Parse coverage report
var parser = new CoberturaParser();
var coverage = await parser.ParseFileAsync("coverage.cobertura.xml");

// Validate
var result = enforcer.Validate(coverage);
if (!result.IsValid)
{
    foreach (var violation in result.Violations)
    {
        Console.WriteLine($"Violation: {violation.File}:{violation.Line}");
    }
}

// Generate dead-path report
var report = enforcer.GenerateDeadPathReport(coverage);
Console.WriteLine($"Active dead paths: {report.ActiveDeadPaths}");

---

Config-Diff Testing

Purpose

Config-diff tests verify that configuration changes produce only expected behavioral deltas. This prevents:

• Unintended side effects from config changes
• Config options affecting unrelated behaviors
• Regressions in config handling

Using ConfigDiffTestBase

using StellaOps.Testing.ConfigDiff;
using Xunit;

[Trait("Category", TestCategories.ConfigDiff)]
public class ConcelierConfigDiffTests : ConfigDiffTestBase
{
    [Fact]
    public async Task ChangingCacheTimeout_OnlyAffectsCacheBehavior()
    {
        var baselineConfig = new ConcelierOptions
        {
            CacheTimeoutMinutes = 30,
            MaxConcurrentDownloads = 10
        };

        var changedConfig = baselineConfig with
        {
            CacheTimeoutMinutes = 60
        };

        var result = await TestConfigIsolationAsync(
            baselineConfig,
            changedConfig,
            changedSetting: "CacheTimeoutMinutes",
            unrelatedBehaviors:
            [
                async config => await GetDownloadBehavior(config),
                async config => await GetParseBehavior(config),
                async config => await GetMergeBehavior(config)
            ]);

        Assert.True(result.IsSuccess,
            $"Unexpected changes: {string.Join(", ", result.UnexpectedChanges)}");
    }

    [Fact]
    public async Task ChangingRetryPolicy_ProducesExpectedDelta()
    {
        var baseline = new ConcelierOptions { MaxRetries = 3 };
        var changed = new ConcelierOptions { MaxRetries = 5 };

        var expectedDelta = new ConfigDelta(
            ChangedBehaviors: ["RetryCount", "TotalRequestTime"],
            BehaviorDeltas:
            [
                new BehaviorDelta("RetryCount", "3", "5", null),
                new BehaviorDelta("TotalRequestTime", "increase", null,
                    "More retries = longer total time")
            ]);

        var result = await TestConfigBehavioralDeltaAsync(
            baseline,
            changed,
            getBehavior: async config => await CaptureRetryBehavior(config),
            computeDelta: ComputeBehaviorSnapshotDelta,
            expectedDelta: expectedDelta);

        Assert.True(result.IsSuccess);
    }
}

Behavior Snapshots

Capture behavior at specific configuration states:

var snapshot = CreateSnapshotBuilder("baseline-config")
    .AddBehavior("CacheHitRate", cacheMetrics.HitRate)
    .AddBehavior("ResponseTime", responseMetrics.P99)
    .AddBehavior("ErrorRate", errorMetrics.Rate)
    .WithCapturedAt(DateTimeOffset.UtcNow)
    .Build();

---

CI Workflows

Available Workflows

Workflow	File	Trigger
Blast-Radius Validation	test-blast-radius.yml	PRs with test changes
Dead-Path Detection	dead-path-detection.yml	Push to main, PRs
Schema Evolution	schema-evolution.yml	Schema/migration changes
Rollback Lag	rollback-lag.yml	Manual trigger, weekly
Test Infrastructure	test-infrastructure.yml	All changes, nightly

Workflow Outputs

Each workflow posts results as PR comments:

## Test Infrastructure :white_check_mark: All checks passed

| Check | Status | Details |
|-------|--------|---------|
| Blast-Radius | :white_check_mark: | 0 violations |
| Dead-Path Detection | :white_check_mark: | Coverage: 82.5% |
| Schema Evolution | :white_check_mark: | Compatible: N-1,N-2 |
| Config-Diff | :white_check_mark: | Tested: Concelier,Authority,Scanner |

Running Locally

# Blast-radius validation
dotnet test --filter "Category=Integration" | grep BlastRadius

# Dead-path detection
dotnet test /p:CollectCoverage=true /p:CoverletOutputFormat=cobertura

# Schema evolution (requires Docker)
docker-compose -f devops/compose/schema-test.yml up -d
dotnet test --filter "Category=SchemaEvolution"

# Config-diff
dotnet test --filter "Category=ConfigDiff"

---

Best Practices

General Guidelines

1. Test categories: Always categorize tests correctly

   • Unit tests: Pure logic, no I/O
   • Integration tests: Database, network, external systems
   • Contract tests: API contracts, schemas
   • Security tests: Authentication, authorization, injection

2. Blast-radius: Choose the narrowest applicable category

   • If a test affects Auth only, use BlastRadius.Auth
   • If it affects Auth and Api, use both

3. Schema evolution: Test both read and write paths

   • Read compatibility: Old data readable by new code
   • Write compatibility: New code writes valid old-schema data

4. Dead-path exemptions: Document thoroughly

   • Include justification
   • Set owner and review date
   • Remove when no longer applicable

5. Config-diff: Focus on high-impact options

   • Security-related configs
   • Performance-related configs
   • Feature flags

Code Review Checklist

• Integration/Contract/Security tests have BlastRadius annotations
• Schema changes include evolution tests
• New branches have test coverage
• Config option tests verify isolation
• Exemptions have justifications

Troubleshooting

Blast-radius validation fails:

# Find tests missing BlastRadius
dotnet test --filter "Category=Integration" --list-tests | \
  xargs -I {} grep -L "BlastRadius" {}

Dead-path baseline drift:

# Regenerate baseline
dotnet test /p:CollectCoverage=true
python extract-dead-paths.py coverage.cobertura.xml
cp dead-paths-report.json dead-paths-baseline.json

Schema evolution test fails:

# Check schema version compatibility
docker run -it ghcr.io/stellaops/schema-test:scanner-v2.0.0 \
  psql -U stellaops_test -d stellaops_schema_test \
  -c "SELECT * FROM _schema_metadata;"

---

Related Documentation

• Test Infrastructure Overview
• Database Schema Specification
• CI/CD Workflows
• Module Testing Agents