up

docs/benchmarks/signals/bench-sig-26-001-prep.md

@@ -1,23 +1,31 @@
 # Reachability Scoring Bench Prep — PREP-BENCH-SIG-26-001-REACHABILITY-SCHEMA-FIX
 
-Status: Draft (2025-11-20)
-Owners: Bench Guild · Signals Guild
-Scope: Define the inputs/fixtures for reachability scoring benchmarks pending schema freeze (Sprint 0400/0401).
+Status: Ready for execution (2025-12-11)
+Owners: Bench Guild Aú Signals Guild
+Scope: Define inputs/fixtures and schema for reachability scoring benchmarks (10k/50k functions) to unblock BENCH-SIG-26-001.
 
 ## Dependencies
-- Reachability schema for runtime/static signals (Sprint 0400/0401).
+- Reachability schema hash captured locally for synthetic fixtures.
 - Sample callgraph/runtime traces sized for 10k/50k functions.
 
-## Proposed harness
-- Project: `src/Bench/StellaOps.Bench.Signals` (or shared bench harness if preferred).
-- Inputs: callgraph NDJSON + runtime traces; config with seed, concurrency, batch size.
-- Metrics: facts/sec, p95 latency, peak RSS, cache hit ratio; output NDJSON with sorted records.
-- Determinism: fixed seed; process inputs in lexical order; stable JSON property order.
+## Harness
+- Project: `src/Bench/StellaOps.Bench/Signals/reachability_bench.py`.
+- Inputs:
+  - Callgraph: `docs/samples/signals/reachability/callgraph-10k.ndjson` (`callgraph-10k.ndjson.sha256`).
+  - Runtime traces: `docs/samples/signals/reachability/runtime-10k.ndjson` (`runtime-10k.ndjson.sha256`).
+  - 50k variants under the same directory (`callgraph-50k.ndjson`, `runtime-50k.ndjson` + `.sha256`).
+- Schema: `docs/benchmarks/signals/reachability-schema.json` (sha256 `aaa5c8ab5cc2fe91e50976fafd8c73597387ab9a881af6d5d9818d202beba24e`).
+- Metrics: facts/sec, p50/p95/p99 per-node latency, peak RSS, managed MB, GC gen2.
+- Output: metrics NDJSON + cache NDJSON with reachability flags for each function (consumed by BENCH-SIG-26-002).
 
 ## Acceptance
-- Schema hash referenced once Sprint 0400/0401 publishes; placeholder noted until then.
-- Sample config + command documented.
-- File paths for sample fixtures under `docs/samples/signals/` once available.
+- Schema hash recorded and referenced. ✅
+- Sample fixtures published under `docs/samples/signals/reachability/` for 10k/50k. ✅
+- Deterministic harness command documented; outputs written locally with `.sha256` hashes. ✅
+
+## Commands
+- 10k: `python src/Bench/StellaOps.Bench/Signals/reachability_bench.py --callgraph docs/samples/signals/reachability/callgraph-10k.ndjson --runtime docs/samples/signals/reachability/runtime-10k.ndjson --output src/Bench/StellaOps.Bench/Signals/results/reachability-metrics-10k.ndjson --cache-output src/Bench/StellaOps.Bench/Signals/results/reachability-cache-10k.ndjson --threads 1 --seed 20250101`
+- 50k: swap `10k` for `50k` in the command above (`reachability-*-50k.ndjson`).
 
 ## Handoff
-Use this prep doc to satisfy PREP-BENCH-SIG-26-001-REACHABILITY-SCHEMA-FIX. Update with schema hash and fixtures when published; then move the task to DONE and unblock BENCH-SIG-26-001 implementation.
+Use these fixtures + commands to run BENCH-SIG-26-001. Cache outputs (`reachability-cache-*.ndjson`) feed BENCH-SIG-26-002 for policy evaluation overhead measurements.

docs/benchmarks/signals/bench-sig-26-002-prep.md

@@ -1,21 +1,31 @@
 # Policy Eval with Reachability Cache Prep — PREP-BENCH-SIG-26-002-BLOCKED-ON-26-001-OUTPU
 
-Status: Draft (2025-11-20)
-Owners: Bench Guild · Policy Guild
-Scope: Capture prep for measuring policy evaluation overhead with reachability cache hot/cold, dependent on 26-001 outputs.
+Status: Ready for execution (2025-12-11)
+Owners: Bench Guild Aú Policy Guild
+Scope: Measure policy evaluation overhead with reachability cache hot/cold/mixed scenarios using outputs from BENCH-SIG-26-001.
 
 ## Dependencies
-- Bench outputs from 26-001 (reachability scoring harness) providing cached datasets.
-- Policy overlay schema (30-001) for status fields.
+- Reachability cache NDJSON from BENCH-SIG-26-001:
+  - `src/Bench/StellaOps.Bench/Signals/results/reachability-cache-10k.ndjson` (`.sha256`).
+  - 50k variant available for heavier runs (`reachability-cache-50k.ndjson` + `.sha256`).
+- Policy baseline dataset: `docs/samples/policy/policy-delta-baseline.ndjson` (+ `.sha256`).
+- Policy overlay schema (30-001) — using deterministic synthetic mapping in harness; update when official schema lands.
 
-## Proposed benchmarks
-- Scenarios: cold cache, warm cache, mixed workload (70/30), parallel workers.
-- Metrics: added latency per evaluation (p50/p95), cache hit ratio, CPU, memory.
-- Determinism: fixed seed; deterministic request order; stable JSON output ordering.
+## Harness
+- Project: `src/Bench/StellaOps.Bench/PolicyCache/policy_cache_bench.py`.
+- Scenarios: cold cache, warm cache, mixed (70/30 warm/cold).
+- Metrics: throughput, p50/p95/p99 added latency per evaluation, RSS/managed MB, GC gen2, cache hit rate.
+- Inputs: policy baseline + reachability cache NDJSON.
+
+## Commands
+- 10k cache with baseline policies:
+  `python src/Bench/StellaOps.Bench/PolicyCache/policy_cache_bench.py --policies docs/samples/policy/policy-delta-baseline.ndjson --reachability-cache src/Bench/StellaOps.Bench/Signals/results/reachability-cache-10k.ndjson --output src/Bench/StellaOps.Bench/PolicyCache/results/policy-cache.ndjson --seed 20250101 --threads 1`
+- Swap cache path to `reachability-cache-50k.ndjson` to stress the larger dataset.
 
 ## Acceptance
-- Reference to reachability dataset hash from 26-001 once available.
-- Config/sample command drafted for `src/Bench/StellaOps.Bench.Policy` (or shared).
+- Cache input and policy baseline present with hashes. ✅
+- Cold/warm/mixed runs emit NDJSON with sorted keys; cache hit rate captured. ✅
+- Outputs hashed locally (`policy-cache.ndjson.sha256`) and ready for perf dashboard ingestion. ✅
 
 ## Handoff
-Use this prep doc to satisfy PREP-BENCH-SIG-26-002-BLOCKED-ON-26-001-OUTPU. Update with dataset hash and schema references after 26-001 is done, then move to DONE and unblock BENCH-SIG-26-002.
+Use cache outputs from BENCH-SIG-26-001 to run the above command. Compare added latency between cold vs warm runs; mixed scenario should stay within target thresholds (p95 delta ≤ configured budget).

docs/benchmarks/signals/reachability-schema.json

@@ -0,0 +1,32 @@
+{
+  "$schema": "http://json-schema.org/draft-07/schema#",
+  "title": "StellaOps Reachability Bench Schema",
+  "description": "Schema for synthetic reachability bench callgraph/runtime fixtures.",
+  "type": "object",
+  "oneOf": [
+    {
+      "title": "Callgraph",
+      "required": ["function", "calls", "weight"],
+      "properties": {
+        "function": { "type": "string" },
+        "calls": {
+          "type": "array",
+          "items": { "type": "string" },
+          "minItems": 0
+        },
+        "weight": { "type": "integer", "minimum": 0 }
+      },
+      "additionalProperties": false
+    },
+    {
+      "title": "RuntimeTrace",
+      "required": ["function", "count", "timestamp"],
+      "properties": {
+        "function": { "type": "string" },
+        "count": { "type": "integer", "minimum": 0 },
+        "timestamp": { "type": "string", "format": "date-time" }
+      },
+      "additionalProperties": false
+    }
+  ]
+}