stella-ops.org/git.stella-ops.org
1
0
Fork 0
Code Issues Pull Requests Actions 2 Releases Wiki Activity
Files
cf5b72974f923126944f4dcd47e85864ebb31a3b
git.stella-ops.org/docs/features/unchecked/bench/benchmark-harness.md

3.3 KiB
Raw Blame History

Benchmark harness (reachability, scanner analyzers, policy engine, determinism)

Module

Bench

Status

IMPLEMENTED

Description

Comprehensive benchmark harness exists covering reachability, scanner analyzers, policy engine, determinism, graph, and link-not-merge benchmarks with Prometheus metric export.

Implementation Details

  • LinkNotMerge Benchmark: src/Bench/StellaOps.Bench/LinkNotMerge/StellaOps.Bench.LinkNotMerge/ -- benchmark scenarios for linkset aggregation performance. Key files: LinkNotMergeScenarioRunner.cs (runs benchmark scenarios), LinksetAggregator.cs (aggregation logic under test), ObservationData.cs (test data models), BenchmarkConfig.cs (scenario configuration), ScenarioStatistics.cs / ScenarioResult.cs / ScenarioExecutionResult.cs (result models).
  • LinkNotMerge VEX Benchmark: src/Bench/StellaOps.Bench/LinkNotMerge.Vex/StellaOps.Bench.LinkNotMerge.Vex/ -- VEX-specific linkset benchmarks. Key files: VexScenarioRunner.cs, VexLinksetAggregator.cs, VexObservationGenerator.cs, VexScenarioConfig.cs, Statistics.cs.
  • Notify Benchmark: src/Bench/StellaOps.Bench/Notify/StellaOps.Bench.Notify/ -- notification dispatch benchmarks. Key files: NotifyScenarioRunner.cs, DispatchAccumulator.cs, BenchmarkConfig.cs.
  • PolicyEngine Benchmark: src/Bench/StellaOps.Bench/PolicyEngine/StellaOps.Bench.PolicyEngine/ -- policy evaluation benchmarks. Key files: PolicyScenarioRunner.cs, PathUtilities.cs, BenchmarkConfig.cs.
  • Scanner.Analyzers Benchmark: src/Bench/StellaOps.Bench/Scanner.Analyzers/StellaOps.Bench.ScannerAnalyzers/ -- scanner analyzer benchmarks. Key files: ScenarioRunners.cs, NodeBenchMetrics.cs, BenchmarkConfig.cs.
  • Baseline Infrastructure: Each benchmark has Baseline/BaselineEntry.cs and Baseline/BaselineLoader.cs for loading ground-truth comparison baselines.
  • Reporting Infrastructure: Each benchmark has Reporting/BenchmarkJsonWriter.cs (JSON output), Reporting/BenchmarkScenarioReport.cs (report model), Reporting/PrometheusWriter.cs (Prometheus metric export).
  • Tests: src/Bench/StellaOps.Bench/LinkNotMerge/StellaOps.Bench.LinkNotMerge.Tests/LinkNotMergeScenarioRunnerTests.cs, BaselineLoaderTests.cs, BenchmarkScenarioReportTests.cs; src/Bench/StellaOps.Bench/LinkNotMerge.Vex/StellaOps.Bench.LinkNotMerge.Vex.Tests/VexScenarioRunnerTests.cs; src/Bench/StellaOps.Bench/Notify/StellaOps.Bench.Notify.Tests/NotifyScenarioRunnerTests.cs, PrometheusWriterTests.cs

E2E Test Plan

  • Run the LinkNotMerge benchmark suite and verify it produces a valid BenchmarkScenarioReport with timing statistics and passes baseline comparison
  • Run the PolicyEngine benchmark and verify scenario results include evaluation counts and latency percentiles
  • Run the Scanner.Analyzers benchmark and verify NodeBenchMetrics are captured per analyzer
  • Verify Prometheus export: run any benchmark and confirm PrometheusWriter outputs valid Prometheus exposition format with scenario labels
  • Verify JSON export: run a benchmark and confirm BenchmarkJsonWriter produces valid JSON report matching the BenchmarkScenarioReport schema
  • Verify baseline comparison: load a baseline and run scenarios, confirm the harness reports regressions when results exceed baseline thresholds