15 KiB
15 KiB
Sprint 0512 · Ops & Offline · Bench (190.G)
Topic & Scope
- Build and capture performance benchmarks for graph, UI interactions, impact index, policy deltas, and reachability scoring to support offline/ops readiness.
- Target harnesses under
src/Bench/StellaOps.Benchwith reproducible datasets. - Working directory:
src/Bench/StellaOps.Bench.
Dependencies & Concurrency
- Upstream data: graph fixtures (SAMPLES-GRAPH-24-003), reachability schema (Sprint 0400/0401), policy delta inputs.
- UI bench depends on BENCH-GRAPH-21-001/002 harness foundation.
Documentation Prerequisites
- docs/07_HIGH_LEVEL_ARCHITECTURE.md
- docs/modules/platform/architecture-overview.md
- docs/modules/graph/architecture.md (for graph bench scenarios)
- docs/modules/signals/architecture.md (for reachability benches)
- docs/modules/policy/architecture.md
Delivery Tracker
| # | Task ID | Status | Key dependency / next step | Owners | Task Definition |
|---|---|---|---|---|---|
| P1 | PREP-BENCH-GRAPH-21-001-NEED-GRAPH-BENCH-HARN | DONE (2025-11-20) | Prep doc at docs/benchmarks/graph/bench-graph-21-001-prep.md; awaits fixtures (SAMPLES-GRAPH-24-003). |
Bench Guild · Graph Platform Guild | Need graph bench harness scaffolding (50k/100k nodes). Document artefact/deliverable for BENCH-GRAPH-21-001 and publish location so downstream tasks can proceed. |
| P2 | PREP-BENCH-GRAPH-21-002-BLOCKED-ON-21-001-HAR | DONE (2025-11-20) | Due 2025-11-26 · Accountable: Bench Guild · UI Guild | Bench Guild · UI Guild | Prep artefact published at docs/benchmarks/graph/bench-graph-21-002-prep.md (Playwright UI bench plan leveraging 50k/100k fixtures; scenarios, metrics, determinism). |
| P3 | PREP-BENCH-IMPACT-16-001-IMPACT-INDEX-DATASET | DONE (2025-11-20) | Due 2025-11-26 · Accountable: Bench Guild · Scheduler Team | Bench Guild · Scheduler Team | Prep artefact published at docs/benchmarks/impact/bench-impact-16-001-prep.md (dataset shape, replay plan, deterministic metrics). |
| P4 | PREP-BENCH-POLICY-20-002-POLICY-DELTA-SAMPLE | DONE (2025-11-20) | Due 2025-11-26 · Accountable: Bench Guild · Policy Guild · Scheduler Guild | Bench Guild · Policy Guild · Scheduler Guild | Prep artefact published at docs/benchmarks/policy/bench-policy-20-002-prep.md (baseline + delta datasets, deterministic harness plan, metrics). |
| P5 | PREP-BENCH-SIG-26-001-REACHABILITY-SCHEMA-FIX | DONE (2025-11-20) | Prep doc at docs/benchmarks/signals/bench-sig-26-001-prep.md; awaits reachability schema hash. |
Bench Guild · Signals Guild | Reachability schema/fixtures pending Sprint 0400/0401. Document artefact/deliverable for BENCH-SIG-26-001 and publish location so downstream tasks can proceed. |
| P6 | PREP-BENCH-SIG-26-002-BLOCKED-ON-26-001-OUTPU | DONE (2025-11-20) | Prep doc at docs/benchmarks/signals/bench-sig-26-002-prep.md; depends on 26-001 datasets. |
Bench Guild · Policy Guild | Blocked on 26-001 outputs. Document artefact/deliverable for BENCH-SIG-26-002 and publish location so downstream tasks can proceed. |
| 1 | BENCH-GRAPH-21-001 | DONE (2025-12-02) | PREP-BENCH-GRAPH-21-001-NEED-GRAPH-BENCH-HARN | Bench Guild · Graph Platform Guild | Build graph viewport/path benchmark harness (50k/100k nodes) measuring Graph API/Indexer latency, memory, and tile cache hit rates. |
| 2 | BENCH-GRAPH-21-002 | DONE (2025-12-02) | PREP-BENCH-GRAPH-21-002-BLOCKED-ON-21-001-HAR | Bench Guild · UI Guild | Add headless UI load benchmark (Playwright) for graph canvas interactions to track render times and FPS budgets. |
| 3 | BENCH-GRAPH-24-002 | DONE (2025-12-02) | Swapped to canonical samples/graph/graph-40k fixture; UI bench driver emits trace/viewport metadata |
Bench Guild · UI Guild | Implement UI interaction benchmarks (filter/zoom/table operations) citing p95 latency; integrate with perf dashboards. |
| 4 | BENCH-IMPACT-16-001 | DONE (2025-12-11) | Dataset + harness landed (docs/samples/impactindex/products-10k.ndjson, src/Bench/StellaOps.Bench/ImpactIndex) |
Bench Guild ? Scheduler Team | ImpactIndex throughput bench (resolve 10k productKeys) + RAM profile. |
| 5 | BENCH-POLICY-20-002 | DONE (2025-12-11) | Baseline/delta fixtures + harness ready (docs/samples/policy/, src/Bench/StellaOps.Bench/PolicyDelta) |
Bench Guild ? Policy Guild ? Scheduler Guild | Add incremental run benchmark measuring delta evaluation vs full; capture SLA compliance. |
| 6 | BENCH-SIG-26-001 | DONE (2025-12-11) | Schema hash + 10k/50k fixtures + harness/caches published (docs/benchmarks/signals/reachability-schema.json, docs/samples/signals/reachability/, src/Bench/StellaOps.Bench/Signals) |
Bench Guild ? Signals Guild | Develop benchmark for reachability scoring pipeline (facts/sec, latency, memory) using synthetic callgraphs/runtime batches. |
| 7 | BENCH-SIG-26-002 | DONE (2025-12-11) | Reachability cache outputs wired into policy cache bench (src/Bench/StellaOps.Bench/PolicyCache) |
Bench Guild ? Policy Guild | Measure policy evaluation overhead with reachability cache hot/cold; ensure ?8 ms p95 added latency. |
| 8 | BENCH-DETERMINISM-401-057 | DONE (2025-11-27) | Feed-freeze hash + SBOM/VEX bundle list from Sprint 0401. | Bench Guild · Signals Guild · Policy Guild (bench/determinism, docs/benchmarks/signals/bench-determinism.md) |
Run cross-scanner determinism bench from 23-Nov advisory; publish determinism% and CVSS delta σ; CI workflow bench-determinism runs harness and uploads manifests/results; offline runner added. |
Wave Coordination
- Single wave; benches sequenced by dataset availability. No parallel wave gating beyond Delivery Tracker dependencies.
Wave Detail Snapshots
- N/A (single wave). Add per-wave snapshots if additional waves are introduced.
Interlocks
- Graph fixtures SAMPLES-GRAPH-24-003 delivery (Bench Guild ↔ Graph Platform Guild).
- Reachability schema alignment from Sprints 0400/0401 (Signals Guild ↔ Policy Guild).
- 2025-12-12 ? Policy delta baseline/delta fixtures + hashes delivered (2025-12-11); bench running locally.
Upcoming Checkpoints
- 2025-12-10 ? Reachability schema hash delivered via
reachability-schema.json+ 10k/50k fixtures (2025-12-11); BENCH-SIG-26-001/002 unblocked. - 2025-12-12 ? Impact index dataset delivered as
docs/samples/impactindex/products-10k.ndjsonwith hash (2025-12-11). - 2025-12-12 ? Policy delta baseline/delta fixtures + hashes delivered (2025-12-11); bench running locally.
Action Tracker
| Action ID | Status | Owner | Due (UTC) | Details |
|---|---|---|---|---|
| ACT-0512-01 | DONE (2025-12-02) | Bench Guild | 2025-11-22 | Confirm SAMPLES-GRAPH-24-003 fixtures availability and publish location for BENCH-GRAPH-21-001/002/24-002 (graph-40k adopted). |
| ACT-0512-02 | DONE (2025-12-11) | Signals Guild | 2025-11-24 | Provide reachability schema hash/output to unblock BENCH-SIG-26-001/002 (see docs/benchmarks/signals/reachability-schema.json). |
| ACT-0512-03 | DONE (2025-12-11) | Scheduler Team | 2025-11-26 | Finalize impact index dataset selection and share deterministic replay bundle (docs/samples/impactindex/products-10k.ndjson). |
| ACT-0512-04 | DONE (2025-12-01) | Bench Guild | 2025-11-24 | Prepare interim synthetic 50k/100k graph fixture (documented in samples/graph/fixtures-plan.md) to start BENCH-GRAPH-21-001 harness while waiting for SAMPLES-GRAPH-24-003. |
| ACT-0512-05 | CLOSED (2025-12-02) | Bench Guild | 2025-11-23 | Escalation not needed; graph fixtures delivered and benches updated to graph-40k. |
| ACT-0512-06 | DONE (2025-12-11) | Signals Guild | 2025-11-24 | Synthetic reachability schema + fixtures published in docs/benchmarks/signals/bench-sig-26-001-prep.md and docs/samples/signals/reachability/ to unblock benches. |
| ACT-0512-07 | DONE (2025-12-02) | Bench Guild ? UI Guild | 2025-11-25 | Draft Playwright bench harness skeleton completed and integrated with graph-40k fixture. |
Decisions & Risks
| Risk | Impact | Mitigation | Status | Owner | Due (UTC) |
|---|---|---|---|---|---|
| Graph fixtures SAMPLES-GRAPH-24-003 not delivered | Blocks BENCH-GRAPH-21-001/002/24-002; benches unstartable | Delivered samples/graph/graph-40k (40k nodes, overlays) on 2025-12-02; update benches to new fixture |
Closed | Bench Guild | 2025-12-02 |
| Reachability schema hash pending from Sprint 0400/0401 | BENCH-SIG-26-001/002 remain blocked | Schema hash + fixtures published (docs/benchmarks/signals/reachability-schema.json, docs/samples/signals/reachability/) |
Closed | Signals Guild | 2025-12-11 |
| Impact index dataset undecided | BENCH-IMPACT-16-001 stalled; no reproducibility | Dataset docs/samples/impactindex/products-10k.ndjson + harness results hashed |
Closed | Scheduler Team | 2025-12-11 |
| UI harness blocked waiting for fixture binding | BENCH-GRAPH-21-002/24-002 cannot start scripting | Harness + fixtures bound to samples/graph/graph-40k; UI driver shipped |
Closed | Bench Guild ? UI Guild | 2025-12-02 |
- All dataset/schema dependencies resolved (graph-40k, impactindex 10k, policy delta, reachability 10k/50k).
- Determinism enforced via local fixtures +
.sha256artifacts across benches; no online feeds required.
Execution Log
| Date (UTC) | Update | Owner |
|---|---|---|
| 2025-12-11 | Completed BENCH-IMPACT-16-001, BENCH-POLICY-20-002, BENCH-SIG-26-001/002; published datasets, schema hash, harness outputs, and .sha256 artifacts. | Bench Guild |
| 2025-12-06 | Marked BENCH-GRAPH-24-002 DONE using graph-40k canonical fixture; remaining benches (impact/policy/reachability) still blocked on datasets/schemas. | Bench Guild |
| 2025-12-02 | Marked BENCH-GRAPH-21-001/002 DONE after overlay-capable harness, SHA capture, UI driver metadata, and deterministic tests; runs still use synthetic fixtures until SAMPLES-GRAPH-24-003 arrives. | Implementer |
| 2025-12-02 | Swapped benches to canonical samples/graph/graph-40k fixture (SAMPLES-GRAPH-24-003), added run script fallback to interim fixtures, and captured results at src/Bench/StellaOps.Bench/Graph/results/graph-40k.json. |
Implementer |
| 2025-11-27 | Added offline runner Determinism/offline_run.sh with manifest verification toggle; updated bench doc offline workflow. |
Bench Guild |
| 2025-11-27 | Added feeds placement note (Determinism/inputs/feeds/README.md) and linked in bench offline workflow. |
Bench Guild |
| 2025-11-27 | Added sample manifest inputs/inputs.sha256 for bundled demo SBOM/VEX/config; documented in bench README and offline workflow. |
Bench Guild |
| 2025-11-27 | Synced BENCH-DETERMINISM-401-057 status date to 2025-11-27 after offline runner/docs completion. | Bench Guild |
| 2025-11-27 | Added offline runner src/Bench/StellaOps.Bench/Determinism/offline_run.sh (defaults runs=10, threshold=0.95) for air-gapped determinism/reachability runs; mirrored in bench doc offline workflow. |
Bench Guild |
| 2025-11-26 | Added optional reachability hashing path (DET_REACH_GRAPHS/DET_REACH_RUNTIME) to determinism run script; reachability helper run_reachability.py with sample graph/runtime fixtures and unit tests added. |
Bench Guild |
| 2025-11-26 | Default runs raised to 10 per scanner/SBOM pair in harness and determinism-run wrapper to match 10x2 matrix requirement. | Bench Guild |
| 2025-11-26 | Added DET_EXTRA_INPUTS/DET_RUN_EXTRA_ARGS support to determinism run script to include frozen feeds in manifests; documented in scripts/bench/README.md. | Bench Guild |
| 2025-11-26 | Added scripts/bench/README.md documenting determinism-run wrapper and threshold env. | Bench Guild |
| 2025-11-26 | Bench CI workflow added (.gitea/workflows/bench-determinism.yml) with threshold gating via BENCH_DETERMINISM_THRESHOLD; run wrapper scripts/bench/determinism-run.sh uploads artifacts. |
Bench Guild |
| 2025-11-26 | Added scripts/bench/determinism-run.sh and CI workflow .gitea/workflows/bench-determinism.yml to run/upload determinism artifacts. |
Bench Guild |
| 2025-11-26 | Built determinism bench harness with mock scanner at src/Bench/StellaOps.Bench/Determinism, added sample SBOM/VEX inputs, generated results/inputs.sha256 + results.csv, updated bench doc, and marked BENCH-DETERMINISM-401-057 DONE. Tests: python -m unittest discover -s src/Bench/StellaOps.Bench/Determinism/tests -t src/Bench/StellaOps.Bench/Determinism. |
Bench Guild |
| 2025-12-01 | Generated interim synthetic graph fixtures (50k/100k nodes with manifests) under samples/graph/interim/ to unblock BENCH-GRAPH-21-001; task moved to DOING pending overlay schema for canonical fixture. |
Implementer |
| 2025-12-01 | Added Graph UI bench scaffold: scenarios JSON, driver (ui_bench_driver.mjs), and plan under src/Bench/StellaOps.Bench/Graph/; BENCH-GRAPH-21-002 moved to DOING using interim fixtures until overlay schema/UI target is available. |
Implementer |
| 2025-12-01 | Added graph bench runner Graph/run_graph_bench.sh and recorded sample results for graph-50k/100k fixtures; BENCH-GRAPH-21-001 progressing with interim fixtures. |
Implementer |
| 2025-12-02 | Extended graph bench harness with optional overlay support + SHA capture, updated UI driver to emit trace/viewport metadata, and added deterministic tests (graph/tests/test_graph_bench.py, ui_bench_driver.test.mjs). |
Implementer |
| 2025-11-22 | Added ACT-0512-07 and corresponding risk entry to have UI bench harness skeleton ready once fixtures bind; no status changes. | Project Mgmt |
| 2025-11-22 | Added ACT-0512-04 to build interim synthetic graph fixture so BENCH-GRAPH-21-001 can start while awaiting SAMPLES-GRAPH-24-003; no status changes. | Project Mgmt |
| 2025-11-22 | Added ACT-0512-05 escalation path (due 2025-11-23) if SAMPLES-GRAPH-24-003 remains unavailable; updated Upcoming Checkpoints accordingly. | Project Mgmt |
| 2025-11-22 | Added ACT-0512-06 fallback synthetic reachability schema/fixtures (due 2025-11-24) in case hash delivery slips; no status changes. | Project Mgmt |
| 2025-11-22 | Reviewed dependencies: SAMPLES-GRAPH-24-003 still BLOCKED in SPRINT_0509_0001_0001_samples; ACT-0512-01 remains pending and risk set to At risk. | Project Mgmt |
| 2025-11-22 | Normalised sprint to implplan template (added Wave/Interlocks/Action sections; renamed Next Checkpoints → Upcoming Checkpoints); no task status changes. | Project Mgmt |
| 2025-11-20 | Completed PREP-BENCH-GRAPH-21-002: published UI bench prep doc at docs/benchmarks/graph/bench-graph-21-002-prep.md; status set to DONE. |
Implementer |
| 2025-11-20 | Completed PREP-BENCH-IMPACT-16-001: published impact index bench prep doc at docs/benchmarks/impact/bench-impact-16-001-prep.md; status set to DONE. |
Implementer |
| 2025-11-20 | Completed PREP-BENCH-POLICY-20-002: published policy delta bench prep doc at docs/benchmarks/policy/bench-policy-20-002-prep.md; status set to DONE. |
Implementer |
| 2025-11-20 | Published prep artefacts for PREP-BENCH-GRAPH-21-001, PREP-BENCH-SIG-26-001, and PREP-BENCH-SIG-26-002 under docs/benchmarks/; marked P1, P5, P6 DONE. |
Implementer |
| 2025-11-19 | Trimmed trailing hyphen from PREP-BENCH-POLICY-20-002 Task ID to keep BENCH-POLICY-20-002 blocker resolvable. | Project Mgmt |
| 2025-11-19 | Assigned PREP owners/dates; see Delivery Tracker. | Planning |
| 2025-11-18 | Marked BENCH-GRAPH-24-002, BENCH-IMPACT-16-001, BENCH-POLICY-20-002, BENCH-SIG-26-001/002 as BLOCKED pending fixtures/datasets and reachability schema. | Bench |
| 2025-11-18 | Normalised sprint to standard template; renamed from SPRINT_512_bench.md. | Ops/Docs |