36 lines
2.4 KiB
Markdown
36 lines
2.4 KiB
Markdown
# Benchmark harness (reachability, scanner analyzers, policy engine, determinism)
|
|
|
|
## Module
|
|
Bench
|
|
|
|
## Status
|
|
VERIFIED
|
|
|
|
## Description
|
|
Comprehensive benchmark harness code exists across LinkNotMerge, LinkNotMerge.Vex, Notify, PolicyEngine, and Scanner.Analyzers modules with deterministic benchmark/reporting support.
|
|
|
|
## Implementation Details
|
|
- **LinkNotMerge Benchmark**: `src/Bench/StellaOps.Bench/LinkNotMerge/StellaOps.Bench.LinkNotMerge/` -- benchmark scenarios for linkset aggregation performance.
|
|
- **LinkNotMerge VEX Benchmark**: `src/Bench/StellaOps.Bench/LinkNotMerge.Vex/StellaOps.Bench.LinkNotMerge.Vex/` -- VEX-specific linkset benchmarks.
|
|
- **Notify Benchmark**: `src/Bench/StellaOps.Bench/Notify/StellaOps.Bench.Notify/` -- notification dispatch benchmarks.
|
|
- **PolicyEngine Benchmark**: `src/Bench/StellaOps.Bench/PolicyEngine/StellaOps.Bench.PolicyEngine/` -- policy evaluation benchmarks.
|
|
- **PolicyEngine Benchmark Policy**: `src/Bench/StellaOps.Bench/PolicyEngine/policies/benchmark-default.yaml` -- benchmark policy fixture compatible with current `StellaOps.Policy` binder schema.
|
|
- **Scanner.Analyzers Benchmark**: `src/Bench/StellaOps.Bench/Scanner.Analyzers/StellaOps.Bench.ScannerAnalyzers/` -- scanner analyzer benchmarks.
|
|
- **Baseline Infrastructure**: benchmark modules include `Baseline/BaselineEntry.cs` and `Baseline/BaselineLoader.cs` for ground-truth comparison.
|
|
- **Reporting Infrastructure**: benchmark modules include JSON and Prometheus writers for machine-consumable artifacts.
|
|
- **Tests**: link-not-merge, vex, notify, and scanner-analyzer benchmark test suites under `src/Bench/StellaOps.Bench/**.Tests/`.
|
|
|
|
## E2E Test Plan
|
|
- [x] Run LinkNotMerge benchmark harness and verify scenario table output is generated.
|
|
- [x] Verify JSON report output is produced and non-empty.
|
|
- [x] Verify Prometheus metrics output is produced and non-empty.
|
|
- [x] Verify CSV result output is produced and non-empty.
|
|
- [x] Verify negative-path CLI behavior (`--config` missing path) exits non-zero.
|
|
|
|
## Verification
|
|
- Verified on 2026-02-11 via FLOW Tier 0/1/2 replay in `run-005`.
|
|
- Tier 0: `docs/qa/feature-checks/runs/bench/benchmark-harness/run-005/tier0-source-check.json`
|
|
- Tier 1: `docs/qa/feature-checks/runs/bench/benchmark-harness/run-005/tier1-build-check.json`
|
|
- Tier 2: `docs/qa/feature-checks/runs/bench/benchmark-harness/run-005/tier2-integration-check.json`
|
|
- Tier 2 evidence: `docs/qa/feature-checks/runs/bench/benchmark-harness/run-005/evidence/`
|