stella-ops.org/git.stella-ops.org
1
0
Fork 0
Code Issues Pull Requests Actions 2 Releases Wiki Activity
Files
4bdc298ec1fc821ee0d9dda6d2d4643dcedf6b35
git.stella-ops.org/docs/features/unchecked/bench/benchmark-harness.md

3.3 KiB
Raw Blame History

Benchmark harness (reachability, scanner analyzers, policy engine, determinism)

Module

Bench

Status

IMPLEMENTED

Description

Comprehensive benchmark harness exists covering reachability, scanner analyzers, policy engine, determinism, graph, and link-not-merge benchmarks with Prometheus metric export.

Implementation Details

  • LinkNotMerge Benchmark: src/Bench/StellaOps.Bench/LinkNotMerge/StellaOps.Bench.LinkNotMerge/ -- benchmark scenarios for linkset aggregation performance. Key files: LinkNotMergeScenarioRunner.cs (runs benchmark scenarios), LinksetAggregator.cs (aggregation logic under test), ObservationData.cs (test data models), BenchmarkConfig.cs (scenario configuration), ScenarioStatistics.cs / ScenarioResult.cs / ScenarioExecutionResult.cs (result models).
  • LinkNotMerge VEX Benchmark: src/Bench/StellaOps.Bench/LinkNotMerge.Vex/StellaOps.Bench.LinkNotMerge.Vex/ -- VEX-specific linkset benchmarks. Key files: VexScenarioRunner.cs, VexLinksetAggregator.cs, VexObservationGenerator.cs, VexScenarioConfig.cs, Statistics.cs.
  • Notify Benchmark: src/Bench/StellaOps.Bench/Notify/StellaOps.Bench.Notify/ -- notification dispatch benchmarks. Key files: NotifyScenarioRunner.cs, DispatchAccumulator.cs, BenchmarkConfig.cs.
  • PolicyEngine Benchmark: src/Bench/StellaOps.Bench/PolicyEngine/StellaOps.Bench.PolicyEngine/ -- policy evaluation benchmarks. Key files: PolicyScenarioRunner.cs, PathUtilities.cs, BenchmarkConfig.cs.
  • Scanner.Analyzers Benchmark: src/Bench/StellaOps.Bench/Scanner.Analyzers/StellaOps.Bench.ScannerAnalyzers/ -- scanner analyzer benchmarks. Key files: ScenarioRunners.cs, NodeBenchMetrics.cs, BenchmarkConfig.cs.
  • Baseline Infrastructure: Each benchmark has Baseline/BaselineEntry.cs and Baseline/BaselineLoader.cs for loading ground-truth comparison baselines.
  • Reporting Infrastructure: Each benchmark has Reporting/BenchmarkJsonWriter.cs (JSON output), Reporting/BenchmarkScenarioReport.cs (report model), Reporting/PrometheusWriter.cs (Prometheus metric export).
  • Tests: src/Bench/StellaOps.Bench/LinkNotMerge/StellaOps.Bench.LinkNotMerge.Tests/LinkNotMergeScenarioRunnerTests.cs, BaselineLoaderTests.cs, BenchmarkScenarioReportTests.cs; src/Bench/StellaOps.Bench/LinkNotMerge.Vex/StellaOps.Bench.LinkNotMerge.Vex.Tests/VexScenarioRunnerTests.cs; src/Bench/StellaOps.Bench/Notify/StellaOps.Bench.Notify.Tests/NotifyScenarioRunnerTests.cs, PrometheusWriterTests.cs

E2E Test Plan

  • Run the LinkNotMerge benchmark suite and verify it produces a valid BenchmarkScenarioReport with timing statistics and passes baseline comparison
  • Run the PolicyEngine benchmark and verify scenario results include evaluation counts and latency percentiles
  • Run the Scanner.Analyzers benchmark and verify NodeBenchMetrics are captured per analyzer
  • Verify Prometheus export: run any benchmark and confirm PrometheusWriter outputs valid Prometheus exposition format with scenario labels
  • Verify JSON export: run a benchmark and confirm BenchmarkJsonWriter produces valid JSON report matching the BenchmarkScenarioReport schema
  • Verify baseline comparison: load a baseline and run scenarios, confirm the harness reports regressions when results exceed baseline thresholds