stella-ops.org/git.stella-ops.org
1
0
Fork 0
Code Issues Pull Requests Actions 2 Releases Wiki Activity
Files
9ca2de05dff9c7db11eb49df6d4e47bbfe7a99f6
git.stella-ops.org/docs/features/checked/bench/vendor-comparison-scanner-parity-tracking.md
master 5bca406787 save checkpoint: save features
2026-02-12 10:27:23 +02:00

2.4 KiB
Raw Blame History

Vendor comparison / scanner parity tracking

Module

Bench

Status

VERIFIED

Description

Scanner analyzer benchmark parity tracking capabilities are present through benchmark reports and metric exports. Fresh behavioral verification confirmed parity-report fields are emitted in benchmark JSON output and CLI error semantics are enforced for invalid configuration.

What's Implemented

  • Scanner Analyzers Benchmark: src/Bench/StellaOps.Bench/Scanner.Analyzers/StellaOps.Bench.ScannerAnalyzers/ -- benchmark harness evaluating analyzer scenarios and recording metrics.
  • Baseline Loader: src/Bench/StellaOps.Bench/Scanner.Analyzers/StellaOps.Bench.ScannerAnalyzers/Baseline/BaselineLoader.cs -- loads baseline data for benchmark comparisons.
  • Baseline Entry: src/Bench/StellaOps.Bench/Scanner.Analyzers/StellaOps.Bench.ScannerAnalyzers/Baseline/BaselineEntry.cs -- baseline model.
  • Benchmark Scenario Report: src/Bench/StellaOps.Bench/Scanner.Analyzers/StellaOps.Bench.ScannerAnalyzers/Reporting/BenchmarkScenarioReport.cs -- per-scenario report model including regression and parity fields.
  • Benchmark JSON Writer: src/Bench/StellaOps.Bench/Scanner.Analyzers/StellaOps.Bench.ScannerAnalyzers/Reporting/BenchmarkJsonWriter.cs -- JSON report writer.
  • Prometheus Writer: src/Bench/StellaOps.Bench/Scanner.Analyzers/StellaOps.Bench.ScannerAnalyzers/Reporting/PrometheusWriter.cs -- Prometheus metrics exporter.
  • Vendor Parity Analyzer: src/Bench/StellaOps.Bench/Scanner.Analyzers/StellaOps.Bench.ScannerAnalyzers/Reporting/VendorParityAnalyzer.cs -- computes vendor parity projections where vendor fixtures are available.

E2E Test Plan

  • Run scanner-analyzers benchmark harness and verify JSON/Prometheus/CSV outputs are generated.
  • Validate benchmark JSON output contains vendorParity fields in scenario reports.
  • Verify baseline/regression metadata is emitted in benchmark JSON.
  • Verify negative-path behavior with missing config returns non-zero exit code.

Verification

  • Verified on 2026-02-11 via FLOW Tier 0/1/2 replay in run-001.
  • Tier 0: docs/qa/feature-checks/runs/bench/vendor-comparison-scanner-parity-tracking/run-001/tier0-source-check.json
  • Tier 1: docs/qa/feature-checks/runs/bench/vendor-comparison-scanner-parity-tracking/run-001/tier1-build-check.json
  • Tier 2: docs/qa/feature-checks/runs/bench/vendor-comparison-scanner-parity-tracking/run-001/tier2-integration-check.json