stella-ops.org/git.stella-ops.org
1
0
Fork 0
Code Issues Pull Requests Actions 2 Releases Wiki Activity

up
Some checks failed
Signals CI & Image / signals-ci (push) Has been cancelled
Details
Signals Reachability Scoring & Events / reachability-smoke (push) Has been cancelled
Details
Signals Reachability Scoring & Events / sign-and-upload (push) Has been cancelled
Details
Manifest Integrity / Validate Schema Integrity (push) Has been cancelled
Details
Manifest Integrity / Validate Contract Documents (push) Has been cancelled
Details
Manifest Integrity / Validate Pack Fixtures (push) Has been cancelled
Details
Manifest Integrity / Audit SHA256SUMS Files (push) Has been cancelled
Details
Manifest Integrity / Verify Merkle Roots (push) Has been cancelled
Details
Docs CI / lint-and-preview (push) Has been cancelled
Details

Browse Source
This commit is contained in:
StellaOps Bot
2025-12-12 09:35:37 +02:00
parent ce5ec9c158
commit efaf3cb789
238 changed files with 146274 additions and 5767 deletions
Download Patch File Download Diff File Expand all files Collapse all files

19
docs/benchmarks/impact/bench-impact-16-001-prep.md
View File

@@ -1,16 +1,16 @@
# Bench Prep — PREP-BENCH-IMPACT-16-001 (ImpactIndex dataset/replay)
Status: **Ready for implementation** (2025-11-20)
Status: **Ready for execution** (2025-12-11)
Owners: Bench Guild · Scheduler Team
Scope: Provide deterministic dataset + replay plan for ImpactIndex throughput benchmark (resolve 10k productKeys; measure latency/throughput/memory).
## Inputs/dataset
- Snapshot file: `bench/impactindex/products-10k.ndjson` (10,000 productKeys, shuffled once with seed `2025-01-01T00:00:00Z`).
- Snapshot file: `docs/samples/impactindex/products-10k.ndjson` (10,000 productKeys, shuffled once with seed `2025-01-01T00:00:00Z`).
- SHA256: `caa79c83b5a9affc3b9cc4e54a516281ddceff4804ce853fee3b62d7afb7ab69` (`products-10k.ndjson.sha256` included).
- Each line: `{ "productKey": "pkg:<ecosystem>/<name>@<version>", "tenant": "bench" }`.
- Include checksum file `products-10k.ndjson.sha256` and drop into repo under `docs/samples/impactindex/`.
## Benchmark procedure
- Harness location: `src/Bench/StellaOps.Bench.ImpactIndex`.
- Harness location: `src/Bench/StellaOps.Bench/ImpactIndex/impact_index_bench.py`.
- Warmup: 1k lookups (excluded from metrics) to trigger caches.
- Run: process all 10k productKeys twice (cold, warm). Record per-pass statistics.
- Metrics to capture (per pass):
@@ -21,11 +21,10 @@ Scope: Provide deterministic dataset + replay plan for ImpactIndex throughput be
- Determinism: fixed seed, single-threaded option flag `--threads 1` for reproducibility; timestamps in UTC ISO-8601.
## Acceptance criteria
- Dataset and checksum published; harness reads from local sample path (no network).
- Benchmark run produces deterministic NDJSON for given seed and hardware profile; differences limited to RSS variability but within ±5%.
- Cold vs warm pass metrics logged; throughput target ≥ 2k items/sec on reference hardware, p95 ≤ 25 ms.
- Dataset and checksum published; harness reads from local sample path (no network). ?
- Benchmark run produces deterministic NDJSON for given seed and hardware profile; differences limited to ?5%.
- Cold vs warm pass metrics logged; throughput target ? 2k items/sec on reference hardware, p95 ? 25 ms.
## Next steps
- Commit dataset + checksum under `docs/samples/impactindex/`.
- Wire harness CLI (`dotnet run -- impactindex --input docs/samples/impactindex/products-10k.ndjson --threads 1 --seed 20250101`).
- Surface metrics to perf dashboard once harness lands; otherwise store under `out/bench/impactindex/` with hashes.
- Harness command: `python src/Bench/StellaOps.Bench/ImpactIndex/impact_index_bench.py --input docs/samples/impactindex/products-10k.ndjson --output src/Bench/StellaOps.Bench/ImpactIndex/results/impactindex.ndjson --threads 1 --seed 20250101`.
- Surface metrics to perf dashboard once harness lands; otherwise store under `out/bench/impactindex/` with hashes (`results/impactindex.ndjson.sha256` present).