stella-ops.org/git.stella-ops.org
1
0
Fork 0
Code Issues Pull Requests Actions 2 Releases Wiki Activity
Files
cf5b72974f923126944f4dcd47e85864ebb31a3b
git.stella-ops.org/docs/features/unchecked/policy/batch-simulation-orchestration.md

4.6 KiB
Raw Blame History

Batch Simulation Orchestration

Module

Policy

Status

IMPLEMENTED

Description

Batch simulation orchestration for running multiple policy simulations in parallel with a dedicated simulation service in the policy registry.

Implementation Details

  • RiskSimulationService: src/Policy/StellaOps.Policy.Engine/Simulation/RiskSimulationService.cs -- RiskSimulationService (sealed class)
    • Simulate(RiskSimulationRequest) runs a single risk simulation with finding scores, distributions, top movers, and aggregate metrics
    • SimulateWithBreakdown(RiskSimulationRequest, RiskSimulationBreakdownOptions?) runs with detailed breakdown analytics per POLICY-RISK-67-003
    • CompareProfilesWithBreakdown(baseProfileId, compareProfileId, findings) runs comparison simulation between two risk profiles with trend analysis
    • GenerateBreakdown(result, findings) generates standalone breakdown for existing simulation results
    • Score formula: signal values * weights -> normalized to 0-100 range
    • Signal types: Boolean (0/1), Numeric (direct), Categorical (mapped: none=0.0, low=0.3, medium=0.6, critical=1.0)
    • Severity thresholds: Critical >= 90, High >= 70, Medium >= 40, Low >= 10, else Informational
    • Actions: Critical/High -> Deny, Medium -> Review, Low/Info -> Allow
    • Supports severity and decision overrides via predicate matching on signal values
    • Produces distribution with 10 buckets and percentiles (p25, p50, p75, p90, p95, p99)
    • Top 10 movers identified by score with primary driver signals
  • RiskSimulationBreakdownService: src/Policy/StellaOps.Policy.Engine/Simulation/RiskSimulationBreakdownService.cs -- generates detailed breakdown analytics and comparison breakdowns
  • SimulationAnalyticsService: src/Policy/StellaOps.Policy.Engine/Simulation/SimulationAnalyticsService.cs -- simulation analytics with delta summary computation
  • WhatIfSimulationService: src/Policy/StellaOps.Policy.Engine/WhatIfSimulation/WhatIfSimulationService.cs -- WhatIfSimulationService (internal sealed class)
    • SimulateAsync(WhatIfSimulationRequest) executes what-if simulation without persisting results
    • Supports hypothetical SBOM diffs (add/remove/upgrade/downgrade operations) and draft policy evaluation
    • Computes decision changes between baseline and simulated: status_changed, severity_changed, new, removed
    • Impact analysis: risk delta (increased/decreased/unchanged), blocked/warning deltas, recommendations
    • VEX override handling: not_affected overrides deny to allow; unreachable downgrades deny to warn
    • Uses EffectiveDecisionMap, PolicyCompilationService, PolicyPackRepository
  • Simulation Endpoints: src/Policy/StellaOps.Policy.Engine/Endpoints/RiskSimulationEndpoints.cs -- REST API for simulation runs
  • Overlay Simulation: src/Policy/StellaOps.Policy.Engine/Endpoints/OverlaySimulationEndpoint.cs -- overlay-based simulation
  • Console Simulation: src/Policy/StellaOps.Policy.Engine/Endpoints/ConsoleSimulationEndpoint.cs -- console surface simulation
  • Path Scope Simulation: src/Policy/StellaOps.Policy.Engine/Endpoints/PathScopeSimulationEndpoint.cs -- path-scoped simulation
  • Batch Evaluation: src/Policy/StellaOps.Policy.Engine/BatchEvaluation/BatchEvaluationModels.cs -- models for batch evaluation runs
  • Telemetry: RiskSimulationsRun counter via PolicyEngineTelemetry, activity tracing with risk_simulation.run and policy.whatif.simulate

E2E Test Plan

  • POST to risk simulation endpoint with a profile ID and list of findings; verify response contains simulation ID, finding scores, distributions, and aggregate metrics
  • POST with IncludeContributions=true and verify signal contribution percentages sum to ~100% and override chain is returned
  • POST with IncludeDistribution=true and verify 10 buckets, 6 percentile levels, severity breakdown covering all 5 levels
  • Run SimulateWithBreakdown and verify breakdown analytics are present alongside the base simulation result
  • Run CompareProfilesWithBreakdown with two profile IDs and verify baseline vs compare results with trend analysis
  • POST what-if simulation with SBOM diff (add component with advisory) and verify new deny decision appears
  • POST what-if simulation with SBOM diff (remove component) and verify allow decision and removed change type
  • POST what-if simulation with draft policy YAML and verify simulated policy reference includes computed digest
  • Verify simulation IDs are deterministic (same inputs produce same rsim-* prefix ID)
  • Verify simulation with 0 findings returns zeroed aggregate metrics