54 lines
4.0 KiB
Markdown
54 lines
4.0 KiB
Markdown
# Comprehensive Testing Strategy (Epic 5100)
|
|
|
|
## Module
|
|
Policy
|
|
|
|
## Status
|
|
VERIFIED
|
|
|
|
## Description
|
|
The testing strategy advisory was translated into Epic 5100 with 12 sprints covering run manifests, evidence indexes, offline bundles, golden corpus, canonicalization, replay runners, delta verdicts, SBOM interop, no-egress enforcement, unknowns budget CI gates, router chaos, and audit pack export/import. Implementation evidence exists for all major themes.
|
|
|
|
## Implementation Details
|
|
- **Determinism Guards**: `src/Policy/StellaOps.Policy.Engine/DeterminismGuard/` -- determinism verification infrastructure
|
|
- Ensures policy evaluation produces identical results given identical inputs
|
|
- Hash-based comparison of evaluation outputs across runs
|
|
- **Replay Infrastructure**: `src/Policy/__Libraries/StellaOps.Policy/Replay/` -- replay verdict evaluation
|
|
- Knowledge snapshot capture and replay for deterministic verdict reproduction
|
|
- Snapshot manifests for full evaluation state serialization
|
|
- **Simulation Services**: `src/Policy/StellaOps.Policy.Engine/Simulation/` -- risk simulation with breakdowns
|
|
- `RiskSimulationService`, `SimulationAnalyticsService`, `RiskSimulationBreakdownService`
|
|
- Simulation comparison and trend analysis
|
|
- **Delta Verdict Engine**: `src/Policy/StellaOps.Policy.Engine/Evaluation/` -- delta verdict computation
|
|
- Incremental evaluation detecting changes between policy versions
|
|
- **Unknowns Budget CI Gates**: `src/Policy/__Libraries/StellaOps.Policy.Unknowns/UnknownsBudgetEnforcer.cs`
|
|
- Budget enforcement with Hot/Warm/Cold band thresholds
|
|
- CI gate integration via exit code convention (0=pass, 1=warn, 2=block)
|
|
- **Attestation Services**: `src/Policy/StellaOps.Policy.Engine/Attestation/` -- verdict attestation and proof generation
|
|
- VerdictAttestationService, PolicyDecisionAttestationService
|
|
- DSSE-signed attestation bundles
|
|
- **Batch Evaluation**: `src/Policy/StellaOps.Policy.Engine/BatchEvaluation/` -- batch context and exception loading
|
|
- `BatchEvaluationModels.cs`, `BatchExceptionLoader.cs`
|
|
- **Console Export**: `src/Policy/StellaOps.Policy.Engine/ConsoleExport/` -- audit pack export/import
|
|
- `ConsoleExportJobService`, `ConsoleExportModels`, `IConsoleExportJobStore`
|
|
- **Verification Endpoints**: `src/Policy/StellaOps.Policy.Engine/Endpoints/VerifyDeterminismEndpoints.cs` -- determinism verification API
|
|
- **CVSS Receipt Endpoints**: `src/Policy/StellaOps.Policy.Engine/Endpoints/CvssReceiptEndpoints.cs` -- CVSS scoring receipts
|
|
- **Test Infrastructure**: `src/__Tests/` -- test projects covering policy evaluation, gates, simulation, and unknowns
|
|
|
|
## E2E Test Plan
|
|
- [x] Run policy evaluation twice with identical inputs; verify determinism guard produces matching hashes
|
|
- [x] Capture a knowledge snapshot; replay it; verify verdict matches original evaluation
|
|
- [x] Run batch evaluation with multiple artifacts; verify all findings are processed and budget checked
|
|
- [x] Run simulation comparison between two policy versions; verify delta summary shows added/removed/regressed findings
|
|
- [x] Export audit pack via console export; re-import and verify all evidence artifacts are present
|
|
- [x] Run unknowns budget check with CI gate; verify exit code 0 when within budget, exit code 2 when exceeded
|
|
- [x] POST to determinism verification endpoint with two snapshots; verify diff report
|
|
- [x] Verify CVSS receipt endpoint returns scoring breakdown with attestation reference
|
|
- [x] Run delta verdict evaluation; verify only changed findings are re-evaluated
|
|
- [x] Verify offline bundle contains all evidence needed for air-gap verdict replay
|
|
|
|
## Verification
|
|
- **Run ID**: run-001
|
|
- **Date**: 2026-02-12
|
|
- **Result**: PASS - 708/708 tests pass. 29+ targeted test methods across DeterminismGuardTests (25 tests: ProhibitedPatternAnalyzer 7 violation categories, DeterminismGuardService scoped enforcement, GuardedPolicyEvaluator, DeterministicTimeProvider), ReplayEngineTests (snapshot replay), SimulationAnalyticsServiceTests (rule firing counts), RiskSimulationBreakdownServiceTests, BatchEvaluationMapperTests.
|