32 lines
2.0 KiB
Markdown
32 lines
2.0 KiB
Markdown
# Bench Prep — PREP-BENCH-IMPACT-16-001 (ImpactIndex dataset/replay)
|
|
|
|
Status: **Ready for implementation** (2025-11-20)
|
|
Owners: Bench Guild · Scheduler Team
|
|
Scope: Provide deterministic dataset + replay plan for ImpactIndex throughput benchmark (resolve 10k productKeys; measure latency/throughput/memory).
|
|
|
|
## Inputs/dataset
|
|
- Snapshot file: `bench/impactindex/products-10k.ndjson` (10,000 productKeys, shuffled once with seed `2025-01-01T00:00:00Z`).
|
|
- Each line: `{ "productKey": "pkg:<ecosystem>/<name>@<version>", "tenant": "bench" }`.
|
|
- Include checksum file `products-10k.ndjson.sha256` and drop into repo under `docs/samples/impactindex/`.
|
|
|
|
## Benchmark procedure
|
|
- Harness location: `src/Bench/StellaOps.Bench.ImpactIndex`.
|
|
- Warmup: 1k lookups (excluded from metrics) to trigger caches.
|
|
- Run: process all 10k productKeys twice (cold, warm). Record per-pass statistics.
|
|
- Metrics to capture (per pass):
|
|
- `throughput_items_per_sec`, `p95_ms`, `p99_ms`, `max_ms` for lookups.
|
|
- `rss_mb`, `managed_mb`, `gc_gen2_count` from .NET counters.
|
|
- `cache_hit_rate` if cache present.
|
|
- Output format: NDJSON; one object per pass with fields `{ pass: "cold"|"warm", startedAtUtc, durationMs, throughput, p95Ms, p99Ms, maxMs, rssMb, managedMb, gcGen2, cacheHitRate }`.
|
|
- Determinism: fixed seed, single-threaded option flag `--threads 1` for reproducibility; timestamps in UTC ISO-8601.
|
|
|
|
## Acceptance criteria
|
|
- Dataset and checksum published; harness reads from local sample path (no network).
|
|
- Benchmark run produces deterministic NDJSON for given seed and hardware profile; differences limited to RSS variability but within ±5%.
|
|
- Cold vs warm pass metrics logged; throughput target ≥ 2k items/sec on reference hardware, p95 ≤ 25 ms.
|
|
|
|
## Next steps
|
|
- Commit dataset + checksum under `docs/samples/impactindex/`.
|
|
- Wire harness CLI (`dotnet run -- impactindex --input docs/samples/impactindex/products-10k.ndjson --threads 1 --seed 20250101`).
|
|
- Surface metrics to perf dashboard once harness lands; otherwise store under `out/bench/impactindex/` with hashes.
|