# Cross-Cutting Testing Standards Guide This guide documents the cross-cutting testing standards implemented for StellaOps, including blast-radius annotations, schema evolution testing, dead-path detection, and config-diff testing. **Sprint Reference:** SPRINT_20260105_002_005_TEST_cross_cutting --- ## Table of Contents 1. [Overview](#overview) 2. [Blast-Radius Annotations](#blast-radius-annotations) 3. [Schema Evolution Testing](#schema-evolution-testing) 4. [Dead-Path Detection](#dead-path-detection) 5. [Config-Diff Testing](#config-diff-testing) 6. [CI Workflows](#ci-workflows) 7. [Best Practices](#best-practices) --- ## Overview Cross-cutting testing standards ensure consistent test quality across all modules: | Standard | Purpose | Enforcement | |----------|---------|-------------| | **Blast-Radius** | Categorize tests by operational surface | CI validation on PRs | | **Schema Evolution** | Verify backward compatibility | CI on schema changes | | **Dead-Path Detection** | Identify uncovered code | CI with baseline comparison | | **Config-Diff** | Validate config behavioral isolation | Integration tests | --- ## Blast-Radius Annotations ### Purpose Blast-radius annotations categorize tests by the operational surfaces they affect. During incidents, this enables targeted test runs for specific areas (e.g., run only Auth-related tests when investigating an authentication issue). ### Categories | Category | Description | Examples | |----------|-------------|----------| | `Auth` | Authentication, authorization, tokens | Login, OAuth, DPoP | | `Scanning` | SBOM generation, vulnerability scanning | Scanner, analyzers | | `Evidence` | Attestation, evidence storage | EvidenceLocker, Attestor | | `Compliance` | Audit, regulatory, GDPR | Compliance reports | | `Advisories` | Advisory ingestion, VEX processing | Concelier, VexLens | | `RiskPolicy` | Risk scoring, policy evaluation | RiskEngine, Policy | | `Crypto` | Cryptographic operations | Signing, verification | | `Integrations` | External systems, webhooks | Notifications, webhooks | | `Persistence` | Database operations | Repositories, migrations | | `Api` | API surface, contracts | Controllers, endpoints | ### Usage ```csharp using StellaOps.TestKit; using Xunit; // Single blast-radius [Trait("Category", TestCategories.Integration)] [Trait("BlastRadius", TestCategories.BlastRadius.Auth)] public class TokenValidationTests { [Fact] public async Task ValidToken_ReturnsSuccess() { // Test implementation } } // Multiple blast-radii (affects multiple surfaces) [Trait("Category", TestCategories.Integration)] [Trait("BlastRadius", TestCategories.BlastRadius.Auth)] [Trait("BlastRadius", TestCategories.BlastRadius.Api)] public class AuthenticatedApiTests { // Tests that affect both Auth and Api surfaces } ``` ### Requirements - **Integration tests**: Must have at least one BlastRadius annotation - **Contract tests**: Must have at least one BlastRadius annotation - **Security tests**: Must have at least one BlastRadius annotation - **Unit tests**: BlastRadius optional but recommended ### Running Tests by Blast-Radius ```bash # Run all Auth-related tests dotnet test --filter "BlastRadius=Auth" # Run tests for multiple surfaces dotnet test --filter "BlastRadius=Auth|BlastRadius=Api" # Run incident response test suite dotnet run --project src/__Libraries/StellaOps.TestKit \ -- run-blast-radius Auth,Api --fail-fast ``` --- ## Schema Evolution Testing ### Purpose Schema evolution tests verify that code remains compatible with previous database schema versions. This prevents breaking changes during: - Rolling deployments (new code, old schema) - Rollbacks (old code, new schema) - Migration windows ### Schema Versions | Version | Description | |---------|-------------| | `N` | Current schema (HEAD) | | `N-1` | Previous schema version | | `N-2` | Two versions back | ### Using SchemaEvolutionTestBase ```csharp using StellaOps.Testing.SchemaEvolution; using Testcontainers.PostgreSql; using Xunit; [Trait("Category", TestCategories.SchemaEvolution)] public class ScannerSchemaEvolutionTests : PostgresSchemaEvolutionTestBase { public ScannerSchemaEvolutionTests() : base(new SchemaEvolutionConfig { ModuleName = "Scanner", CurrentVersion = new SchemaVersion("v2.1.0", DateTimeOffset.Parse("2026-01-01")), PreviousVersions = [ new SchemaVersion("v2.0.0", DateTimeOffset.Parse("2025-10-01")), new SchemaVersion("v1.9.0", DateTimeOffset.Parse("2025-07-01")) ], ConnectionStringTemplate = "Host={0};Port={1};Database={2};Username={3};Password={4}" }) { } [Fact] public async Task ReadOperations_CompatibleWithPreviousSchema() { var result = await TestReadBackwardCompatibilityAsync( async (connection, version) => { // Test read operations against old schema var repository = new ScanRepository(connection); var scans = await repository.GetRecentScansAsync(10); return scans.Count >= 0; }); Assert.True(result.IsSuccess); } [Fact] public async Task WriteOperations_CompatibleWithPreviousSchema() { var result = await TestWriteForwardCompatibilityAsync( async (connection, version) => { // Test write operations var repository = new ScanRepository(connection); await repository.CreateScanAsync(new ScanRequest { /* ... */ }); return true; }); Assert.True(result.IsSuccess); } } ``` ### Versioned Container Images Build versioned PostgreSQL images for testing: ```bash # Build all versions for a module ./devops/docker/schema-versions/build-schema-images.sh scanner # Build specific version ./devops/docker/schema-versions/build-schema-images.sh scanner v2.0.0 # Use in tests docker run -d -p 5432:5432 ghcr.io/stellaops/schema-test:scanner-v2.0.0 ``` --- ## Dead-Path Detection ### Purpose Dead-path detection identifies uncovered code branches. This helps: - Find untested edge cases - Identify potentially dead code - Prevent coverage regression ### How It Works 1. Tests run with branch coverage collection (Coverlet) 2. Cobertura XML report is parsed 3. Uncovered branches are identified 4. New dead paths are compared against baseline 5. CI fails if new dead paths are introduced ### Baseline Management The baseline file (`dead-paths-baseline.json`) tracks known dead paths: ```json { "version": "1.0.0", "activeDeadPaths": 42, "totalDeadPaths": 50, "exemptedPaths": 8, "entries": [ { "file": "src/Scanner/Services/AnalyzerService.cs", "line": 128, "coverage": "1/2", "isExempt": false } ] } ``` ### Exemptions Add exemptions for intentionally untested code in `coverage-exemptions.yaml`: ```yaml exemptions: - path: "src/Authority/Emergency/BreakGlassHandler.cs:42" category: emergency justification: "Emergency access bypass - tested in incident drills" added: "2026-01-06" owner: "security-team" - path: "src/Scanner/Platform/WindowsRegistryScanner.cs:*" category: platform justification: "Windows-only code - CI runs on Linux" added: "2026-01-06" owner: "scanner-team" ignore_patterns: - "*.Generated.cs" - "**/Migrations/*.cs" ``` ### Using BranchCoverageEnforcer ```csharp using StellaOps.Testing.Coverage; var enforcer = new BranchCoverageEnforcer(new BranchCoverageConfig { MinimumBranchCoverage = 80, FailOnNewDeadPaths = true, ExemptionFiles = ["coverage-exemptions.yaml"] }); // Parse coverage report var parser = new CoberturaParser(); var coverage = await parser.ParseFileAsync("coverage.cobertura.xml"); // Validate var result = enforcer.Validate(coverage); if (!result.IsValid) { foreach (var violation in result.Violations) { Console.WriteLine($"Violation: {violation.File}:{violation.Line}"); } } // Generate dead-path report var report = enforcer.GenerateDeadPathReport(coverage); Console.WriteLine($"Active dead paths: {report.ActiveDeadPaths}"); ``` --- ## Config-Diff Testing ### Purpose Config-diff tests verify that configuration changes produce only expected behavioral deltas. This prevents: - Unintended side effects from config changes - Config options affecting unrelated behaviors - Regressions in config handling ### Using ConfigDiffTestBase ```csharp using StellaOps.Testing.ConfigDiff; using Xunit; [Trait("Category", TestCategories.ConfigDiff)] public class ConcelierConfigDiffTests : ConfigDiffTestBase { [Fact] public async Task ChangingCacheTimeout_OnlyAffectsCacheBehavior() { var baselineConfig = new ConcelierOptions { CacheTimeoutMinutes = 30, MaxConcurrentDownloads = 10 }; var changedConfig = baselineConfig with { CacheTimeoutMinutes = 60 }; var result = await TestConfigIsolationAsync( baselineConfig, changedConfig, changedSetting: "CacheTimeoutMinutes", unrelatedBehaviors: [ async config => await GetDownloadBehavior(config), async config => await GetParseBehavior(config), async config => await GetMergeBehavior(config) ]); Assert.True(result.IsSuccess, $"Unexpected changes: {string.Join(", ", result.UnexpectedChanges)}"); } [Fact] public async Task ChangingRetryPolicy_ProducesExpectedDelta() { var baseline = new ConcelierOptions { MaxRetries = 3 }; var changed = new ConcelierOptions { MaxRetries = 5 }; var expectedDelta = new ConfigDelta( ChangedBehaviors: ["RetryCount", "TotalRequestTime"], BehaviorDeltas: [ new BehaviorDelta("RetryCount", "3", "5", null), new BehaviorDelta("TotalRequestTime", "increase", null, "More retries = longer total time") ]); var result = await TestConfigBehavioralDeltaAsync( baseline, changed, getBehavior: async config => await CaptureRetryBehavior(config), computeDelta: ComputeBehaviorSnapshotDelta, expectedDelta: expectedDelta); Assert.True(result.IsSuccess); } } ``` ### Behavior Snapshots Capture behavior at specific configuration states: ```csharp var snapshot = CreateSnapshotBuilder("baseline-config") .AddBehavior("CacheHitRate", cacheMetrics.HitRate) .AddBehavior("ResponseTime", responseMetrics.P99) .AddBehavior("ErrorRate", errorMetrics.Rate) .WithCapturedAt(DateTimeOffset.UtcNow) .Build(); ``` --- ## CI Workflows ### Available Workflows | Workflow | File | Trigger | |----------|------|---------| | Blast-Radius Validation | `test-blast-radius.yml` | PRs with test changes | | Dead-Path Detection | `dead-path-detection.yml` | Push to main, PRs | | Schema Evolution | `schema-evolution.yml` | Schema/migration changes | | Rollback Lag | `rollback-lag.yml` | Manual trigger, weekly | | Test Infrastructure | `test-infrastructure.yml` | All changes, nightly | ### Workflow Outputs Each workflow posts results as PR comments: ```markdown ## Test Infrastructure :white_check_mark: All checks passed | Check | Status | Details | |-------|--------|---------| | Blast-Radius | :white_check_mark: | 0 violations | | Dead-Path Detection | :white_check_mark: | Coverage: 82.5% | | Schema Evolution | :white_check_mark: | Compatible: N-1,N-2 | | Config-Diff | :white_check_mark: | Tested: Concelier,Authority,Scanner | ``` ### Running Locally ```bash # Blast-radius validation dotnet test --filter "Category=Integration" | grep BlastRadius # Dead-path detection dotnet test /p:CollectCoverage=true /p:CoverletOutputFormat=cobertura # Schema evolution (requires Docker) docker-compose -f devops/compose/schema-test.yml up -d dotnet test --filter "Category=SchemaEvolution" # Config-diff dotnet test --filter "Category=ConfigDiff" ``` --- ## Best Practices ### General Guidelines 1. **Test categories**: Always categorize tests correctly - Unit tests: Pure logic, no I/O - Integration tests: Database, network, external systems - Contract tests: API contracts, schemas - Security tests: Authentication, authorization, injection 2. **Blast-radius**: Choose the narrowest applicable category - If a test affects Auth only, use `BlastRadius.Auth` - If it affects Auth and Api, use both 3. **Schema evolution**: Test both read and write paths - Read compatibility: Old data readable by new code - Write compatibility: New code writes valid old-schema data 4. **Dead-path exemptions**: Document thoroughly - Include justification - Set owner and review date - Remove when no longer applicable 5. **Config-diff**: Focus on high-impact options - Security-related configs - Performance-related configs - Feature flags ### Code Review Checklist - [ ] Integration/Contract/Security tests have BlastRadius annotations - [ ] Schema changes include evolution tests - [ ] New branches have test coverage - [ ] Config option tests verify isolation - [ ] Exemptions have justifications ### Troubleshooting **Blast-radius validation fails:** ```bash # Find tests missing BlastRadius dotnet test --filter "Category=Integration" --list-tests | \ xargs -I {} grep -L "BlastRadius" {} ``` **Dead-path baseline drift:** ```bash # Regenerate baseline dotnet test /p:CollectCoverage=true python extract-dead-paths.py coverage.cobertura.xml cp dead-paths-report.json dead-paths-baseline.json ``` **Schema evolution test fails:** ```bash # Check schema version compatibility docker run -it ghcr.io/stellaops/schema-test:scanner-v2.0.0 \ psql -U stellaops_test -d stellaops_schema_test \ -c "SELECT * FROM _schema_metadata;" ``` --- ## Related Documentation - [Test Infrastructure Overview](../testing/README.md) - [Database Schema Specification](../db/SPECIFICATION.md) - [CI/CD Workflows](../../.gitea/workflows/README.md) - [Module Testing Agents](../../src/__Tests/AGENTS.md)