32 lines
2.1 KiB
Markdown
32 lines
2.1 KiB
Markdown
# Bench Prep — PREP-BENCH-GRAPH-21-001 (Graph API/Indexer harness)
|
||
|
||
Status: **Ready for implementation** (2025-11-20)
|
||
Owners: Bench Guild · Graph Platform Guild
|
||
Scope: Build deterministic Graph benchmark harness for 50k/100k node fixtures measuring API/Indexer latency, memory, and tile cache hit rates.
|
||
|
||
## Fixtures
|
||
- Use SAMPLES-GRAPH-24-003 (40–50k) and extend to 100k via duplication with new ids; store under `docs/samples/graph/50k.ndjson` and `100k.ndjson` with `.sha256` hashes.
|
||
- Node ordering deterministic; timestamps fixed to `2025-01-01T00:00:00Z`.
|
||
|
||
## Harness plan (project: `src/Bench/StellaOps.Bench.GraphApi`)
|
||
- Scenarios (repeat 5x; report median/p95):
|
||
1. **Viewport fetch**: `/v1/graph/tiles?bbox=<seed>` — measure server latency + tile count.
|
||
2. **Path query**: `/v1/graph/path?from=...&to=...` — latency + hops + cache hits.
|
||
3. **Overlay apply**: apply policy overlay to 1k nodes; measure apply time and index rebuild cost.
|
||
4. **Cold vs warm cache**: run viewport + path with cache cold then warm; capture hit rate.
|
||
- Metrics captured as NDJSON per run: `{ scenario, fixture, pass: cold|warm, medianMs, p95Ms, maxMs, rssMb, managedMb, cacheHitRate }` plus start/end UTC timestamps.
|
||
- Determinism: fixed seed (`GRAPH_BENCH_SEED=2025-01-01T00:00:00Z`); single-thread option `--threads 1` for reproducibility; clear caches between cold/warm phases.
|
||
|
||
## Outputs
|
||
- Store under `out/bench/graph/api/{runId}/results.ndjson` with `.sha256`.
|
||
- Summary CSV optional derived from NDJSON; no dynamic wall-clock in filenames beyond runId.
|
||
|
||
## Acceptance criteria
|
||
- Harness runs offline against local fixtures; no external calls.
|
||
- Median/p95 for each scenario produced for both 50k and 100k fixtures; cache hit rate recorded where applicable.
|
||
- Re-running with same seed/fixtures yields identical NDJSON (apart from RSS variance).
|
||
|
||
## Next steps
|
||
- Generate fixtures + hashes; wire CLI entry `dotnet run -- graph-api --fixture docs/samples/graph/50k.ndjson --seed 20250101`.
|
||
- Add perf dashboard hook if available; otherwise publish artifacts under `out/bench/graph/api/latest/`.
|