up

docs/benchmarks/signals/bench-determinism.md

@@ -42,8 +42,9 @@ for sbom, vex in zip(SBOMS, VEXES):
 - CVSS delta σ vs reference; VEX stability (σ_after ≤ σ_before).
 
 ## Deliverables
-- `bench/determinism/` with harness, hashed inputs, and `results.csv`.
-- `bench/determinism/inputs.sha256` listing SBOM, VEX, feed bundle hashes (deterministic ordering).
+- Harness at `src/Bench/StellaOps.Bench/Determinism` (offline-friendly mock scanner included).
+- `results/*.csv` with per-run hashes plus `summary.json` determinism rate.
+- `results/inputs.sha256` listing SBOM, VEX, and config hashes (deterministic ordering).
 - `bench/reachability/dataset.sha256` listing reachability corpus inputs (graphs, runtime traces) when running combined bench.
 - CI target `bench:determinism` producing determinism% and σ per scanner; optional `bench:reachability` to recompute graph hash and runtime hit stability.
 
@@ -56,16 +57,11 @@ for sbom, vex in zip(SBOMS, VEXES):
 ## How to run (local)
 
 ```sh
-cd bench/determinism
-python3 -m venv .venv && source .venv/bin/activate
-pip install -r requirements.txt
+cd src/Bench/StellaOps.Bench/Determinism
 
-# Freeze feeds and policy hashes
-./freeze_feeds.sh ../feeds/bundle.tar.gz > inputs.sha256
-
-# Run determinism bench
+# Run determinism bench (uses built-in mock scanner by default; defaults to 10 runs)
 python run_bench.py --sboms inputs/sboms/*.json --vex inputs/vex/*.json \
-  --scanners configs/scanners.yaml --runs 20 --shuffle
+  --config configs/scanners.json --shuffle --output results
 
 # Reachability dataset (optional)
 python run_reachability.py --graphs ../reachability/graphs/*.json \
@@ -76,9 +72,9 @@ Outputs are written to `results.csv` (determinism) and `results-reach.csv` (reac
 
 ## How to run (CI)
 
-- Target `bench:determinism` in CI (see `.gitea/workflows/bench-determinism.yml`) runs the harness with frozen feeds and uploads `results.csv` + `inputs.sha256` as artifacts.
-- Optional `bench:reachability` target replays reachability corpus, recomputes graph hashes, and compares against expected `dataset.sha256`.
-- CI must fail if determinism rate < 0.95 or any graph hash mismatch.
+- Workflow `.gitea/workflows/bench-determinism.yml` calls `scripts/bench/determinism-run.sh`, which runs the harness with the bundled mock scanner and uploads `out/bench-determinism/**` (results, manifests, summary). Set `DET_EXTRA_INPUTS` to include frozen feed bundles in `inputs.sha256`.
+- Optional `bench:reachability` target (future) will replay reachability corpus, recompute graph hashes, and compare against expected `dataset.sha256`.
+- CI fails when `determinism_rate` < `BENCH_DETERMINISM_THRESHOLD` (defaults to 0.95; set via env in the workflow).
 
 ## Offline/air-gap workflow