git.stella-ops.org/docs/implplan/SPRINT_0512_0001_0001_bench.md

Sprint 0512 · Ops & Offline · Bench (190.G)

Topic & Scope

• Build and capture performance benchmarks for graph, UI interactions, impact index, policy deltas, and reachability scoring to support offline/ops readiness.
• Target harnesses under src/Bench/StellaOps.Bench with reproducible datasets.
• Working directory: src/Bench/StellaOps.Bench.

Dependencies & Concurrency

• Upstream data: graph fixtures (SAMPLES-GRAPH-24-003), reachability schema (Sprint 0400/0401), policy delta inputs.
• UI bench depends on BENCH-GRAPH-21-001/002 harness foundation.

Documentation Prerequisites

• docs/07_HIGH_LEVEL_ARCHITECTURE.md
• docs/modules/platform/architecture-overview.md
• docs/modules/graph/architecture.md (for graph bench scenarios)
• docs/modules/signals/architecture.md (for reachability benches)
• docs/modules/policy/architecture.md

Delivery Tracker

#	Task ID	Status	Key dependency / next step	Owners	Task Definition
P1	PREP-BENCH-GRAPH-21-001-NEED-GRAPH-BENCH-HARN	DONE (2025-11-20)	Prep doc at docs/benchmarks/graph/bench-graph-21-001-prep.md; awaits fixtures (SAMPLES-GRAPH-24-003).	Bench Guild · Graph Platform Guild	Need graph bench harness scaffolding (50k/100k nodes). Document artefact/deliverable for BENCH-GRAPH-21-001 and publish location so downstream tasks can proceed.
P2	PREP-BENCH-GRAPH-21-002-BLOCKED-ON-21-001-HAR	DONE (2025-11-20)	Due 2025-11-26 · Accountable: Bench Guild · UI Guild	Bench Guild · UI Guild	Prep artefact published at docs/benchmarks/graph/bench-graph-21-002-prep.md (Playwright UI bench plan leveraging 50k/100k fixtures; scenarios, metrics, determinism).
P3	PREP-BENCH-IMPACT-16-001-IMPACT-INDEX-DATASET	DONE (2025-11-20)	Due 2025-11-26 · Accountable: Bench Guild · Scheduler Team	Bench Guild · Scheduler Team	Prep artefact published at docs/benchmarks/impact/bench-impact-16-001-prep.md (dataset shape, replay plan, deterministic metrics).
P4	PREP-BENCH-POLICY-20-002-POLICY-DELTA-SAMPLE	DONE (2025-11-20)	Due 2025-11-26 · Accountable: Bench Guild · Policy Guild · Scheduler Guild	Bench Guild · Policy Guild · Scheduler Guild	Prep artefact published at docs/benchmarks/policy/bench-policy-20-002-prep.md (baseline + delta datasets, deterministic harness plan, metrics).
P5	PREP-BENCH-SIG-26-001-REACHABILITY-SCHEMA-FIX	DONE (2025-11-20)	Prep doc at docs/benchmarks/signals/bench-sig-26-001-prep.md; awaits reachability schema hash.	Bench Guild · Signals Guild	Reachability schema/fixtures pending Sprint 0400/0401. Document artefact/deliverable for BENCH-SIG-26-001 and publish location so downstream tasks can proceed.
P6	PREP-BENCH-SIG-26-002-BLOCKED-ON-26-001-OUTPU	DONE (2025-11-20)	Prep doc at docs/benchmarks/signals/bench-sig-26-002-prep.md; depends on 26-001 datasets.	Bench Guild · Policy Guild	Blocked on 26-001 outputs. Document artefact/deliverable for BENCH-SIG-26-002 and publish location so downstream tasks can proceed.
1	BENCH-GRAPH-21-001	BLOCKED	PREP-BENCH-GRAPH-21-001-NEED-GRAPH-BENCH-HARN	Bench Guild · Graph Platform Guild	Build graph viewport/path benchmark harness (50k/100k nodes) measuring Graph API/Indexer latency, memory, and tile cache hit rates.
2	BENCH-GRAPH-21-002	BLOCKED	PREP-BENCH-GRAPH-21-002-BLOCKED-ON-21-001-HAR	Bench Guild · UI Guild	Add headless UI load benchmark (Playwright) for graph canvas interactions to track render times and FPS budgets.
3	BENCH-GRAPH-24-002	BLOCKED	Waiting for 50k/100k graph fixture (SAMPLES-GRAPH-24-003)	Bench Guild · UI Guild	Implement UI interaction benchmarks (filter/zoom/table operations) citing p95 latency; integrate with perf dashboards.
4	BENCH-IMPACT-16-001	BLOCKED	PREP-BENCH-IMPACT-16-001-IMPACT-INDEX-DATASET	Bench Guild · Scheduler Team	ImpactIndex throughput bench (resolve 10k productKeys) + RAM profile.
5	BENCH-POLICY-20-002	BLOCKED	PREP-BENCH-POLICY-20-002-POLICY-DELTA-SAMPLE	Bench Guild · Policy Guild · Scheduler Guild	Add incremental run benchmark measuring delta evaluation vs full; capture SLA compliance.
6	BENCH-SIG-26-001	BLOCKED	PREP-BENCH-SIG-26-001-REACHABILITY-SCHEMA-FIX	Bench Guild · Signals Guild	Develop benchmark for reachability scoring pipeline (facts/sec, latency, memory) using synthetic callgraphs/runtime batches.
7	BENCH-SIG-26-002	BLOCKED	PREP-BENCH-SIG-26-002-BLOCKED-ON-26-001-OUTPU	Bench Guild · Policy Guild	Measure policy evaluation overhead with reachability cache hot/cold; ensure ≤8 ms p95 added latency.
8	BENCH-DETERMINISM-401-057	DONE (2025-11-26)	Feed-freeze hash + SBOM/VEX bundle list from Sprint 0401.	Bench Guild · Signals Guild · Policy Guild (bench/determinism, docs/benchmarks/signals/bench-determinism.md)	Run cross-scanner determinism bench from 23-Nov advisory; publish determinism% and CVSS delta σ; CI workflow bench-determinism runs harness and uploads manifests/results.

Wave Coordination

• Single wave; benches sequenced by dataset availability. No parallel wave gating beyond Delivery Tracker dependencies.

Wave Detail Snapshots

• N/A (single wave). Add per-wave snapshots if additional waves are introduced.

Interlocks

• Graph fixtures SAMPLES-GRAPH-24-003 delivery (Bench Guild ↔ Graph Platform Guild).
• Reachability schema alignment from Sprints 0400/0401 (Signals Guild ↔ Policy Guild).
• Policy delta dataset delivery (Policy Guild ↔ Scheduler Guild).

Upcoming Checkpoints

• 2025-11-22 · Confirm availability of graph fixtures for BENCH-GRAPH-21-001/002/24-002. Owner: Bench Guild.
• 2025-11-23 · Escalate to Graph Platform Guild if SAMPLES-GRAPH-24-003 location still missing; confirm interim synthetic path (ACT-0512-04). Owner: Bench Guild.
• 2025-11-24 · Reachability schema alignment outcome to unblock BENCH-SIG-26-001. Owner: Signals Guild.
• 2025-11-26 · Decide impact index dataset for BENCH-IMPACT-16-001. Owner: Scheduler Team.

Action Tracker

Action ID	Status	Owner	Due (UTC)	Details
ACT-0512-01	PENDING	Bench Guild	2025-11-22	Confirm SAMPLES-GRAPH-24-003 fixtures availability and publish location for BENCH-GRAPH-21-001/002/24-002.
ACT-0512-02	PENDING	Signals Guild	2025-11-24	Provide reachability schema hash/output to unblock BENCH-SIG-26-001/002.
ACT-0512-03	PENDING	Scheduler Team	2025-11-26	Finalize impact index dataset selection and share deterministic replay bundle.
ACT-0512-04	PENDING	Bench Guild	2025-11-24	Prepare interim synthetic 50k/100k graph fixture (documented in samples/graph/fixtures-plan.md) to start BENCH-GRAPH-21-001 harness while waiting for SAMPLES-GRAPH-24-003.
ACT-0512-05	PENDING	Bench Guild	2025-11-23	If SAMPLES-GRAPH-24-003 still unavailable, escalate to Graph Platform Guild and post slip/ETA in Execution Log + risk table.
ACT-0512-06	PENDING	Signals Guild	2025-11-24	If reachability schema hash slips past 2025-11-24, publish synthetic schema + sample batches in docs/benchmarks/signals/bench-sig-26-001-prep.md to unblock BENCH-SIG-26-001/002 harness scaffolding.
ACT-0512-07	PENDING	Bench Guild · UI Guild	2025-11-25	Draft Playwright bench harness skeleton (headless, deterministic seeds, no network) reusing bench-graph-21-002-prep scenarios; commit once fixture source (real or synthetic) is bound.

Decisions & Risks

Risk	Impact	Mitigation	Status	Owner	Due (UTC)
Graph fixtures SAMPLES-GRAPH-24-003 not delivered	Blocks BENCH-GRAPH-21-001/002/24-002; benches unstartable	Track via ACT-0512-01; ACT-0512-05 escalation if missed	At risk	Bench Guild	2025-11-22
Reachability schema hash pending from Sprint 0400/0401	BENCH-SIG-26-001/002 remain blocked	ACT-0512-02 to deliver hash; ACT-0512-06 fallback synthetic set if delayed	Open	Signals Guild	2025-11-24
Impact index dataset undecided	BENCH-IMPACT-16-001 stalled; no reproducibility	ACT-0512-03 to finalize dataset; require deterministic replay bundle	Open	Scheduler Team	2025-11-26
UI harness blocked waiting for fixture binding	BENCH-GRAPH-21-002/24-002 cannot start scripting	ACT-0512-07 to draft harness skeleton with deterministic seeds; binds once fixture path set	Open	Bench Guild · UI Guild	2025-11-25

• Graph fixture still blocked per docs/implplan/SPRINT_0509_0001_0001_samples.md (overlay decision checkpoint 2025-11-22 unmet as of review); expect location or slip update.
• Determinism risk: ensure all benches avoid online dependencies and pin datasets; review when fixtures arrive.

Execution Log

Date (UTC)	Update	Owner
2025-11-26	Default runs raised to 10 per scanner/SBOM pair in harness and determinism-run wrapper to match 10x2 matrix requirement.	Bench Guild
2025-11-26	Added DET_EXTRA_INPUTS/DET_RUN_EXTRA_ARGS support to determinism run script to include frozen feeds in manifests; documented in scripts/bench/README.md.	Bench Guild
2025-11-26	Added scripts/bench/README.md documenting determinism-run wrapper and threshold env.	Bench Guild
2025-11-26	Bench CI workflow added (.gitea/workflows/bench-determinism.yml) with threshold gating via BENCH_DETERMINISM_THRESHOLD; run wrapper scripts/bench/determinism-run.sh uploads artifacts.	Bench Guild
2025-11-26	Added scripts/bench/determinism-run.sh and CI workflow .gitea/workflows/bench-determinism.yml to run/upload determinism artifacts.	Bench Guild
2025-11-26	Built determinism bench harness with mock scanner at src/Bench/StellaOps.Bench/Determinism, added sample SBOM/VEX inputs, generated results/inputs.sha256 + results.csv, updated bench doc, and marked BENCH-DETERMINISM-401-057 DONE. Tests: python -m unittest discover -s src/Bench/StellaOps.Bench/Determinism/tests -t src/Bench/StellaOps.Bench/Determinism.	Bench Guild
2025-11-22	Added ACT-0512-07 and corresponding risk entry to have UI bench harness skeleton ready once fixtures bind; no status changes.	Project Mgmt
2025-11-22	Added ACT-0512-04 to build interim synthetic graph fixture so BENCH-GRAPH-21-001 can start while awaiting SAMPLES-GRAPH-24-003; no status changes.	Project Mgmt
2025-11-22	Added ACT-0512-05 escalation path (due 2025-11-23) if SAMPLES-GRAPH-24-003 remains unavailable; updated Upcoming Checkpoints accordingly.	Project Mgmt
2025-11-22	Added ACT-0512-06 fallback synthetic reachability schema/fixtures (due 2025-11-24) in case hash delivery slips; no status changes.	Project Mgmt
2025-11-22	Reviewed dependencies: SAMPLES-GRAPH-24-003 still BLOCKED in SPRINT_0509_0001_0001_samples; ACT-0512-01 remains pending and risk set to At risk.	Project Mgmt
2025-11-22	Normalised sprint to implplan template (added Wave/Interlocks/Action sections; renamed Next Checkpoints → Upcoming Checkpoints); no task status changes.	Project Mgmt
2025-11-20	Completed PREP-BENCH-GRAPH-21-002: published UI bench prep doc at docs/benchmarks/graph/bench-graph-21-002-prep.md; status set to DONE.	Implementer
2025-11-20	Completed PREP-BENCH-IMPACT-16-001: published impact index bench prep doc at docs/benchmarks/impact/bench-impact-16-001-prep.md; status set to DONE.	Implementer
2025-11-20	Completed PREP-BENCH-POLICY-20-002: published policy delta bench prep doc at docs/benchmarks/policy/bench-policy-20-002-prep.md; status set to DONE.	Implementer
2025-11-20	Published prep artefacts for PREP-BENCH-GRAPH-21-001, PREP-BENCH-SIG-26-001, and PREP-BENCH-SIG-26-002 under docs/benchmarks/; marked P1, P5, P6 DONE.	Implementer
2025-11-19	Trimmed trailing hyphen from PREP-BENCH-POLICY-20-002 Task ID to keep BENCH-POLICY-20-002 blocker resolvable.	Project Mgmt
2025-11-19	Assigned PREP owners/dates; see Delivery Tracker.	Planning
2025-11-18	Marked BENCH-GRAPH-24-002, BENCH-IMPACT-16-001, BENCH-POLICY-20-002, BENCH-SIG-26-001/002 as BLOCKED pending fixtures/datasets and reachability schema.	Bench
2025-11-18	Normalised sprint to standard template; renamed from SPRINT_512_bench.md.	Ops/Docs