git.stella-ops.org/docs/features/checked/policy/comprehensive-testing-strategy.md

Comprehensive Testing Strategy (Epic 5100)

Module

Policy

Status

VERIFIED

Description

The testing strategy advisory was translated into Epic 5100 with 12 sprints covering run manifests, evidence indexes, offline bundles, golden corpus, canonicalization, replay runners, delta verdicts, SBOM interop, no-egress enforcement, unknowns budget CI gates, router chaos, and audit pack export/import. Implementation evidence exists for all major themes.

Implementation Details

• Determinism Guards: src/Policy/StellaOps.Policy.Engine/DeterminismGuard/ -- determinism verification infrastructure
  • Ensures policy evaluation produces identical results given identical inputs
  • Hash-based comparison of evaluation outputs across runs
• Replay Infrastructure: src/Policy/__Libraries/StellaOps.Policy/Replay/ -- replay verdict evaluation
  • Knowledge snapshot capture and replay for deterministic verdict reproduction
  • Snapshot manifests for full evaluation state serialization
• Simulation Services: src/Policy/StellaOps.Policy.Engine/Simulation/ -- risk simulation with breakdowns
  • RiskSimulationService, SimulationAnalyticsService, RiskSimulationBreakdownService
  • Simulation comparison and trend analysis
• Delta Verdict Engine: src/Policy/StellaOps.Policy.Engine/Evaluation/ -- delta verdict computation
  • Incremental evaluation detecting changes between policy versions
• Unknowns Budget CI Gates: src/Policy/__Libraries/StellaOps.Policy.Unknowns/UnknownsBudgetEnforcer.cs
  • Budget enforcement with Hot/Warm/Cold band thresholds
  • CI gate integration via exit code convention (0=pass, 1=warn, 2=block)
• Attestation Services: src/Policy/StellaOps.Policy.Engine/Attestation/ -- verdict attestation and proof generation
  • VerdictAttestationService, PolicyDecisionAttestationService
  • DSSE-signed attestation bundles
• Batch Evaluation: src/Policy/StellaOps.Policy.Engine/BatchEvaluation/ -- batch context and exception loading
  • BatchEvaluationModels.cs, BatchExceptionLoader.cs
• Console Export: src/Policy/StellaOps.Policy.Engine/ConsoleExport/ -- audit pack export/import
  • ConsoleExportJobService, ConsoleExportModels, IConsoleExportJobStore
• Verification Endpoints: src/Policy/StellaOps.Policy.Engine/Endpoints/VerifyDeterminismEndpoints.cs -- determinism verification API
• CVSS Receipt Endpoints: src/Policy/StellaOps.Policy.Engine/Endpoints/CvssReceiptEndpoints.cs -- CVSS scoring receipts
• Test Infrastructure: src/__Tests/ -- test projects covering policy evaluation, gates, simulation, and unknowns

E2E Test Plan

• Run policy evaluation twice with identical inputs; verify determinism guard produces matching hashes
• Capture a knowledge snapshot; replay it; verify verdict matches original evaluation
• Run batch evaluation with multiple artifacts; verify all findings are processed and budget checked
• Run simulation comparison between two policy versions; verify delta summary shows added/removed/regressed findings
• Export audit pack via console export; re-import and verify all evidence artifacts are present
• Run unknowns budget check with CI gate; verify exit code 0 when within budget, exit code 2 when exceeded
• POST to determinism verification endpoint with two snapshots; verify diff report
• Verify CVSS receipt endpoint returns scoring breakdown with attestation reference
• Run delta verdict evaluation; verify only changed findings are re-evaluated
• Verify offline bundle contains all evidence needed for air-gap verdict replay

Verification

• Run ID: run-001
• Date: 2026-02-12
• Result: PASS - 708/708 tests pass. 29+ targeted test methods across DeterminismGuardTests (25 tests: ProhibitedPatternAnalyzer 7 violation categories, DeterminismGuardService scoped enforcement, GuardedPolicyEvaluator, DeterministicTimeProvider), ReplayEngineTests (snapshot replay), SimulationAnalyticsServiceTests (rule firing counts), RiskSimulationBreakdownServiceTests, BatchEvaluationMapperTests.