stella-ops.org/git.stella-ops.org
1
0
Fork 0
Code Issues Pull Requests Actions 12 Releases Wiki Activity
Files
efaf3cb7899907a8d18840fa6af146795f3c3bf9
git.stella-ops.org/docs/benchmarks/impact/bench-impact-16-001-prep.md
StellaOps Bot efaf3cb789
Some checks failed
Signals CI & Image / signals-ci (push) Has been cancelled
Details
Signals Reachability Scoring & Events / reachability-smoke (push) Has been cancelled
Details
Signals Reachability Scoring & Events / sign-and-upload (push) Has been cancelled
Details
Manifest Integrity / Validate Schema Integrity (push) Has been cancelled
Details
Manifest Integrity / Validate Contract Documents (push) Has been cancelled
Details
Manifest Integrity / Validate Pack Fixtures (push) Has been cancelled
Details
Manifest Integrity / Audit SHA256SUMS Files (push) Has been cancelled
Details
Manifest Integrity / Verify Merkle Roots (push) Has been cancelled
Details
Docs CI / lint-and-preview (push) Has been cancelled
Details
up
2025-12-12 09:35:37 +02:00

2.1 KiB
Raw Blame History

Bench Prep — PREP-BENCH-IMPACT-16-001 (ImpactIndex dataset/replay)

Status: Ready for execution (2025-12-11) Owners: Bench Guild · Scheduler Team Scope: Provide deterministic dataset + replay plan for ImpactIndex throughput benchmark (resolve 10k productKeys; measure latency/throughput/memory).

Inputs/dataset

  • Snapshot file: docs/samples/impactindex/products-10k.ndjson (10,000 productKeys, shuffled once with seed 2025-01-01T00:00:00Z).
  • SHA256: caa79c83b5a9affc3b9cc4e54a516281ddceff4804ce853fee3b62d7afb7ab69 (products-10k.ndjson.sha256 included).
  • Each line: { "productKey": "pkg:<ecosystem>/<name>@<version>", "tenant": "bench" }.

Benchmark procedure

  • Harness location: src/Bench/StellaOps.Bench/ImpactIndex/impact_index_bench.py.
  • Warmup: 1k lookups (excluded from metrics) to trigger caches.
  • Run: process all 10k productKeys twice (cold, warm). Record per-pass statistics.
  • Metrics to capture (per pass):
    • throughput_items_per_sec, p95_ms, p99_ms, max_ms for lookups.
    • rss_mb, managed_mb, gc_gen2_count from .NET counters.
    • cache_hit_rate if cache present.
  • Output format: NDJSON; one object per pass with fields { pass: "cold"|"warm", startedAtUtc, durationMs, throughput, p95Ms, p99Ms, maxMs, rssMb, managedMb, gcGen2, cacheHitRate }.
  • Determinism: fixed seed, single-threaded option flag --threads 1 for reproducibility; timestamps in UTC ISO-8601.

Acceptance criteria

  • Dataset and checksum published; harness reads from local sample path (no network). ?
  • Benchmark run produces deterministic NDJSON for given seed and hardware profile; differences limited to ?5%.
  • Cold vs warm pass metrics logged; throughput target ? 2k items/sec on reference hardware, p95 ? 25 ms.

Next steps

  • Harness command: python src/Bench/StellaOps.Bench/ImpactIndex/impact_index_bench.py --input docs/samples/impactindex/products-10k.ndjson --output src/Bench/StellaOps.Bench/ImpactIndex/results/impactindex.ndjson --threads 1 --seed 20250101.
  • Surface metrics to perf dashboard once harness lands; otherwise store under out/bench/impactindex/ with hashes (results/impactindex.ndjson.sha256 present).