stella-ops.org/git.stella-ops.org
1
0
Fork 0
Code Issues Pull Requests Actions 16 Releases Wiki Activity
Files
d519782a8f0b30f425c9b6ae0f316b19259972a2
git.stella-ops.org/docs/benchmarks/impact/bench-impact-16-001-prep.md
master d519782a8f
Some checks failed
Docs CI / lint-and-preview (push) Has been cancelled
Details
prep docs and service updates
2025-11-21 06:56:36 +00:00

2.0 KiB
Raw Blame History

Bench Prep — PREP-BENCH-IMPACT-16-001 (ImpactIndex dataset/replay)

Status: Ready for implementation (2025-11-20) Owners: Bench Guild · Scheduler Team Scope: Provide deterministic dataset + replay plan for ImpactIndex throughput benchmark (resolve 10k productKeys; measure latency/throughput/memory).

Inputs/dataset

  • Snapshot file: bench/impactindex/products-10k.ndjson (10,000 productKeys, shuffled once with seed 2025-01-01T00:00:00Z).
  • Each line: { "productKey": "pkg:<ecosystem>/<name>@<version>", "tenant": "bench" }.
  • Include checksum file products-10k.ndjson.sha256 and drop into repo under docs/samples/impactindex/.

Benchmark procedure

  • Harness location: src/Bench/StellaOps.Bench.ImpactIndex.
  • Warmup: 1k lookups (excluded from metrics) to trigger caches.
  • Run: process all 10k productKeys twice (cold, warm). Record per-pass statistics.
  • Metrics to capture (per pass):
    • throughput_items_per_sec, p95_ms, p99_ms, max_ms for lookups.
    • rss_mb, managed_mb, gc_gen2_count from .NET counters.
    • cache_hit_rate if cache present.
  • Output format: NDJSON; one object per pass with fields { pass: "cold"|"warm", startedAtUtc, durationMs, throughput, p95Ms, p99Ms, maxMs, rssMb, managedMb, gcGen2, cacheHitRate }.
  • Determinism: fixed seed, single-threaded option flag --threads 1 for reproducibility; timestamps in UTC ISO-8601.

Acceptance criteria

  • Dataset and checksum published; harness reads from local sample path (no network).
  • Benchmark run produces deterministic NDJSON for given seed and hardware profile; differences limited to RSS variability but within ±5%.
  • Cold vs warm pass metrics logged; throughput target ≥ 2k items/sec on reference hardware, p95 ≤ 25 ms.

Next steps

  • Commit dataset + checksum under docs/samples/impactindex/.
  • Wire harness CLI (dotnet run -- impactindex --input docs/samples/impactindex/products-10k.ndjson --threads 1 --seed 20250101).
  • Surface metrics to perf dashboard once harness lands; otherwise store under out/bench/impactindex/ with hashes.