46 lines
3.6 KiB
Markdown
46 lines
3.6 KiB
Markdown
# Deterministic Evaluation with Knowledge Snapshots
|
|
|
|
## Module
|
|
Policy
|
|
|
|
## Status
|
|
IMPLEMENTED
|
|
|
|
## Description
|
|
Deterministic evaluation engine that pins all inputs via knowledge snapshot digests and can replay evaluations offline with identical results.
|
|
|
|
## Implementation Details
|
|
- **Knowledge Snapshot Manifest**: `src/Policy/__Libraries/StellaOps.Policy/Snapshots/KnowledgeSnapshotManifest.cs` -- manifest containing all input digests
|
|
- Captures: SBOM digest, advisory feed digest, policy bundle digest, VEX document digests, reachability graph digest
|
|
- Content-addressed snapshot ID via `SnapshotIdGenerator.cs`
|
|
- **SnapshotBuilder**: `src/Policy/__Libraries/StellaOps.Policy/Snapshots/SnapshotBuilder.cs` -- fluent builder for constructing knowledge snapshots
|
|
- **SnapshotAwarePolicyEvaluator**: `src/Policy/__Libraries/StellaOps.Policy/Snapshots/SnapshotAwarePolicyEvaluator.cs` -- evaluator that pins inputs to snapshot
|
|
- Evaluation uses frozen state from snapshot (no live data fetching)
|
|
- Results are reproducible: same snapshot always produces same verdicts
|
|
- **SnapshotIdGenerator**: `src/Policy/__Libraries/StellaOps.Policy/Snapshots/SnapshotIdGenerator.cs` -- deterministic ID from snapshot content
|
|
- **KnowledgeSourceDescriptor**: `src/Policy/__Libraries/StellaOps.Policy/Snapshots/KnowledgeSourceDescriptor.cs` -- describes a knowledge source (type, URI, digest, timestamp)
|
|
- **SnapshotService (Library)**: `src/Policy/__Libraries/StellaOps.Policy/Snapshots/SnapshotService.cs` -- snapshot lifecycle management
|
|
- **SnapshotService (Engine)**: `src/Policy/StellaOps.Policy.Engine/Snapshots/SnapshotService.cs` -- engine-level snapshot operations
|
|
- **SnapshotStore**: `src/Policy/StellaOps.Policy.Engine/Snapshots/SnapshotStore.cs` -- snapshot persistence
|
|
- **SnapshotModels**: `src/Policy/StellaOps.Policy.Engine/Snapshots/SnapshotModels.cs` -- snapshot DTOs
|
|
- **Replay Engine**: `src/Policy/__Libraries/StellaOps.Policy/Replay/ReplayEngine.cs` -- replays evaluation from snapshot
|
|
- `ReplayRequest.cs` -- replay parameters including snapshot reference
|
|
- `ReplayResult.cs` -- replay outcome with verdict comparison
|
|
- `VerdictComparer.cs` -- compares original and replayed verdicts for drift detection
|
|
- `ReplayReport.cs` -- detailed replay report with match/mismatch analysis
|
|
- `KnowledgeSourceResolver.cs` -- resolves snapshot references to evaluation inputs
|
|
- **Snapshot Endpoints**: `src/Policy/StellaOps.Policy.Engine/Endpoints/SnapshotEndpoint.cs`, `SnapshotEndpoints.cs`, `PolicySnapshotEndpoints.cs` -- REST API for snapshot CRUD
|
|
- **Determinism Guards Integration**: `src/Policy/StellaOps.Policy.Engine/DeterminismGuard/` -- ensures no wall-clock or RNG leaks into snapshot-pinned evaluation
|
|
|
|
## E2E Test Plan
|
|
- [ ] Build a knowledge snapshot with SBOM, advisory feed, and policy bundle digests; verify snapshot ID is content-addressed
|
|
- [ ] Evaluate finding using SnapshotAwarePolicyEvaluator with pinned snapshot; verify deterministic verdict
|
|
- [ ] Re-evaluate same snapshot; verify identical verdict (byte-for-byte match)
|
|
- [ ] Replay evaluation from snapshot using ReplayEngine; verify VerdictComparer shows no drift
|
|
- [ ] Modify advisory feed and replay with original snapshot; verify replay uses original feed (not modified)
|
|
- [ ] POST snapshot to snapshot endpoint; verify snapshot is persisted and retrievable by ID
|
|
- [ ] Verify KnowledgeSourceDescriptor contains type, URI, digest, and timestamp for each source
|
|
- [ ] Build snapshot with SnapshotBuilder; verify manifest contains all expected source descriptors
|
|
- [ ] Replay evaluation with intentionally modified policy; verify VerdictComparer detects mismatch
|
|
- [ ] Verify snapshot ID changes when any input digest changes
|