stella-ops.org/git.stella-ops.org
1
0
Fork 0
Code Issues Pull Requests Actions 2 Releases Wiki Activity
Files
cf5b72974f923126944f4dcd47e85864ebb31a3b
git.stella-ops.org/docs/features/unchecked/policy/deterministic-evaluation-with-knowledge-snapshots.md

3.6 KiB
Raw Blame History

Deterministic Evaluation with Knowledge Snapshots

Module

Policy

Status

IMPLEMENTED

Description

Deterministic evaluation engine that pins all inputs via knowledge snapshot digests and can replay evaluations offline with identical results.

Implementation Details

  • Knowledge Snapshot Manifest: src/Policy/__Libraries/StellaOps.Policy/Snapshots/KnowledgeSnapshotManifest.cs -- manifest containing all input digests
    • Captures: SBOM digest, advisory feed digest, policy bundle digest, VEX document digests, reachability graph digest
    • Content-addressed snapshot ID via SnapshotIdGenerator.cs
  • SnapshotBuilder: src/Policy/__Libraries/StellaOps.Policy/Snapshots/SnapshotBuilder.cs -- fluent builder for constructing knowledge snapshots
  • SnapshotAwarePolicyEvaluator: src/Policy/__Libraries/StellaOps.Policy/Snapshots/SnapshotAwarePolicyEvaluator.cs -- evaluator that pins inputs to snapshot
    • Evaluation uses frozen state from snapshot (no live data fetching)
    • Results are reproducible: same snapshot always produces same verdicts
  • SnapshotIdGenerator: src/Policy/__Libraries/StellaOps.Policy/Snapshots/SnapshotIdGenerator.cs -- deterministic ID from snapshot content
  • KnowledgeSourceDescriptor: src/Policy/__Libraries/StellaOps.Policy/Snapshots/KnowledgeSourceDescriptor.cs -- describes a knowledge source (type, URI, digest, timestamp)
  • SnapshotService (Library): src/Policy/__Libraries/StellaOps.Policy/Snapshots/SnapshotService.cs -- snapshot lifecycle management
  • SnapshotService (Engine): src/Policy/StellaOps.Policy.Engine/Snapshots/SnapshotService.cs -- engine-level snapshot operations
  • SnapshotStore: src/Policy/StellaOps.Policy.Engine/Snapshots/SnapshotStore.cs -- snapshot persistence
  • SnapshotModels: src/Policy/StellaOps.Policy.Engine/Snapshots/SnapshotModels.cs -- snapshot DTOs
  • Replay Engine: src/Policy/__Libraries/StellaOps.Policy/Replay/ReplayEngine.cs -- replays evaluation from snapshot
    • ReplayRequest.cs -- replay parameters including snapshot reference
    • ReplayResult.cs -- replay outcome with verdict comparison
    • VerdictComparer.cs -- compares original and replayed verdicts for drift detection
    • ReplayReport.cs -- detailed replay report with match/mismatch analysis
    • KnowledgeSourceResolver.cs -- resolves snapshot references to evaluation inputs
  • Snapshot Endpoints: src/Policy/StellaOps.Policy.Engine/Endpoints/SnapshotEndpoint.cs, SnapshotEndpoints.cs, PolicySnapshotEndpoints.cs -- REST API for snapshot CRUD
  • Determinism Guards Integration: src/Policy/StellaOps.Policy.Engine/DeterminismGuard/ -- ensures no wall-clock or RNG leaks into snapshot-pinned evaluation

E2E Test Plan

  • Build a knowledge snapshot with SBOM, advisory feed, and policy bundle digests; verify snapshot ID is content-addressed
  • Evaluate finding using SnapshotAwarePolicyEvaluator with pinned snapshot; verify deterministic verdict
  • Re-evaluate same snapshot; verify identical verdict (byte-for-byte match)
  • Replay evaluation from snapshot using ReplayEngine; verify VerdictComparer shows no drift
  • Modify advisory feed and replay with original snapshot; verify replay uses original feed (not modified)
  • POST snapshot to snapshot endpoint; verify snapshot is persisted and retrievable by ID
  • Verify KnowledgeSourceDescriptor contains type, URI, digest, and timestamp for each source
  • Build snapshot with SnapshotBuilder; verify manifest contains all expected source descriptors
  • Replay evaluation with intentionally modified policy; verify VerdictComparer detects mismatch
  • Verify snapshot ID changes when any input digest changes