29 lines
3.3 KiB
Markdown
29 lines
3.3 KiB
Markdown
# Benchmark harness (reachability, scanner analyzers, policy engine, determinism)
|
|
|
|
## Module
|
|
Bench
|
|
|
|
## Status
|
|
IMPLEMENTED
|
|
|
|
## Description
|
|
Comprehensive benchmark harness exists covering reachability, scanner analyzers, policy engine, determinism, graph, and link-not-merge benchmarks with Prometheus metric export.
|
|
|
|
## Implementation Details
|
|
- **LinkNotMerge Benchmark**: `src/Bench/StellaOps.Bench/LinkNotMerge/StellaOps.Bench.LinkNotMerge/` -- benchmark scenarios for linkset aggregation performance. Key files: `LinkNotMergeScenarioRunner.cs` (runs benchmark scenarios), `LinksetAggregator.cs` (aggregation logic under test), `ObservationData.cs` (test data models), `BenchmarkConfig.cs` (scenario configuration), `ScenarioStatistics.cs` / `ScenarioResult.cs` / `ScenarioExecutionResult.cs` (result models).
|
|
- **LinkNotMerge VEX Benchmark**: `src/Bench/StellaOps.Bench/LinkNotMerge.Vex/StellaOps.Bench.LinkNotMerge.Vex/` -- VEX-specific linkset benchmarks. Key files: `VexScenarioRunner.cs`, `VexLinksetAggregator.cs`, `VexObservationGenerator.cs`, `VexScenarioConfig.cs`, `Statistics.cs`.
|
|
- **Notify Benchmark**: `src/Bench/StellaOps.Bench/Notify/StellaOps.Bench.Notify/` -- notification dispatch benchmarks. Key files: `NotifyScenarioRunner.cs`, `DispatchAccumulator.cs`, `BenchmarkConfig.cs`.
|
|
- **PolicyEngine Benchmark**: `src/Bench/StellaOps.Bench/PolicyEngine/StellaOps.Bench.PolicyEngine/` -- policy evaluation benchmarks. Key files: `PolicyScenarioRunner.cs`, `PathUtilities.cs`, `BenchmarkConfig.cs`.
|
|
- **Scanner.Analyzers Benchmark**: `src/Bench/StellaOps.Bench/Scanner.Analyzers/StellaOps.Bench.ScannerAnalyzers/` -- scanner analyzer benchmarks. Key files: `ScenarioRunners.cs`, `NodeBenchMetrics.cs`, `BenchmarkConfig.cs`.
|
|
- **Baseline Infrastructure**: Each benchmark has `Baseline/BaselineEntry.cs` and `Baseline/BaselineLoader.cs` for loading ground-truth comparison baselines.
|
|
- **Reporting Infrastructure**: Each benchmark has `Reporting/BenchmarkJsonWriter.cs` (JSON output), `Reporting/BenchmarkScenarioReport.cs` (report model), `Reporting/PrometheusWriter.cs` (Prometheus metric export).
|
|
- **Tests**: `src/Bench/StellaOps.Bench/LinkNotMerge/StellaOps.Bench.LinkNotMerge.Tests/LinkNotMergeScenarioRunnerTests.cs`, `BaselineLoaderTests.cs`, `BenchmarkScenarioReportTests.cs`; `src/Bench/StellaOps.Bench/LinkNotMerge.Vex/StellaOps.Bench.LinkNotMerge.Vex.Tests/VexScenarioRunnerTests.cs`; `src/Bench/StellaOps.Bench/Notify/StellaOps.Bench.Notify.Tests/NotifyScenarioRunnerTests.cs`, `PrometheusWriterTests.cs`
|
|
|
|
## E2E Test Plan
|
|
- [ ] Run the LinkNotMerge benchmark suite and verify it produces a valid `BenchmarkScenarioReport` with timing statistics and passes baseline comparison
|
|
- [ ] Run the PolicyEngine benchmark and verify scenario results include evaluation counts and latency percentiles
|
|
- [ ] Run the Scanner.Analyzers benchmark and verify `NodeBenchMetrics` are captured per analyzer
|
|
- [ ] Verify Prometheus export: run any benchmark and confirm `PrometheusWriter` outputs valid Prometheus exposition format with scenario labels
|
|
- [ ] Verify JSON export: run a benchmark and confirm `BenchmarkJsonWriter` produces valid JSON report matching the `BenchmarkScenarioReport` schema
|
|
- [ ] Verify baseline comparison: load a baseline and run scenarios, confirm the harness reports regressions when results exceed baseline thresholds
|