efaf3cb789
Some checks failed
Signals CI & Image / signals-ci (push) Has been cancelled
Signals Reachability Scoring & Events / reachability-smoke (push) Has been cancelled
Signals Reachability Scoring & Events / sign-and-upload (push) Has been cancelled
Manifest Integrity / Validate Schema Integrity (push) Has been cancelled
Manifest Integrity / Validate Contract Documents (push) Has been cancelled
Manifest Integrity / Validate Pack Fixtures (push) Has been cancelled
Manifest Integrity / Audit SHA256SUMS Files (push) Has been cancelled
Manifest Integrity / Verify Merkle Roots (push) Has been cancelled
Docs CI / lint-and-preview (push) Has been cancelled
31 lines
2.1 KiB
Markdown
31 lines
2.1 KiB
Markdown
# Bench Prep — PREP-BENCH-IMPACT-16-001 (ImpactIndex dataset/replay)
|
|
|
|
Status: **Ready for execution** (2025-12-11)
|
|
Owners: Bench Guild · Scheduler Team
|
|
Scope: Provide deterministic dataset + replay plan for ImpactIndex throughput benchmark (resolve 10k productKeys; measure latency/throughput/memory).
|
|
|
|
## Inputs/dataset
|
|
- Snapshot file: `docs/samples/impactindex/products-10k.ndjson` (10,000 productKeys, shuffled once with seed `2025-01-01T00:00:00Z`).
|
|
- SHA256: `caa79c83b5a9affc3b9cc4e54a516281ddceff4804ce853fee3b62d7afb7ab69` (`products-10k.ndjson.sha256` included).
|
|
- Each line: `{ "productKey": "pkg:<ecosystem>/<name>@<version>", "tenant": "bench" }`.
|
|
|
|
## Benchmark procedure
|
|
- Harness location: `src/Bench/StellaOps.Bench/ImpactIndex/impact_index_bench.py`.
|
|
- Warmup: 1k lookups (excluded from metrics) to trigger caches.
|
|
- Run: process all 10k productKeys twice (cold, warm). Record per-pass statistics.
|
|
- Metrics to capture (per pass):
|
|
- `throughput_items_per_sec`, `p95_ms`, `p99_ms`, `max_ms` for lookups.
|
|
- `rss_mb`, `managed_mb`, `gc_gen2_count` from .NET counters.
|
|
- `cache_hit_rate` if cache present.
|
|
- Output format: NDJSON; one object per pass with fields `{ pass: "cold"|"warm", startedAtUtc, durationMs, throughput, p95Ms, p99Ms, maxMs, rssMb, managedMb, gcGen2, cacheHitRate }`.
|
|
- Determinism: fixed seed, single-threaded option flag `--threads 1` for reproducibility; timestamps in UTC ISO-8601.
|
|
|
|
## Acceptance criteria
|
|
- Dataset and checksum published; harness reads from local sample path (no network). ?
|
|
- Benchmark run produces deterministic NDJSON for given seed and hardware profile; differences limited to ?5%.
|
|
- Cold vs warm pass metrics logged; throughput target ? 2k items/sec on reference hardware, p95 ? 25 ms.
|
|
|
|
## Next steps
|
|
- Harness command: `python src/Bench/StellaOps.Bench/ImpactIndex/impact_index_bench.py --input docs/samples/impactindex/products-10k.ndjson --output src/Bench/StellaOps.Bench/ImpactIndex/results/impactindex.ndjson --threads 1 --seed 20250101`.
|
|
- Surface metrics to perf dashboard once harness lands; otherwise store under `out/bench/impactindex/` with hashes (`results/impactindex.ndjson.sha256` present).
|