docs consolidation and others

docs/operations/runbooks/reachability-runtime.md

@@ -0,0 +1,63 @@
+# Reachability Runtime Ingestion Runbook
+
+> **Imposed rule:** Runtime traces must never bypass CAS/DSSE verification; ingest only CAS-addressed NDJSON with hashes logged to Timeline and Evidence Locker.
+
+This runbook guides operators through ingesting runtime reachability evidence (EntryTrace, probes, Signals ingestion) and wiring it into the reachability evidence chain.
+
+## 1. Prerequisites
+- Services: `Signals` API, `Zastava Observer` (or other probes), `Evidence Locker`, optional `Attestor` for DSSE.
+- Reachability schema: `docs/modules/reach-graph/guides/function-level-evidence.md`, `docs/modules/reach-graph/guides/evidence-schema.md`.
+- CAS: configured bucket/path for `cas://reachability/runtime/*` and `.../graphs/*`.
+- Time sync: AirGap Time anchor if sealed; otherwise NTP with drift <200ms.
+
+## 2. Ingestion workflow (online)
+1) **Capture traces** from Observer/probes → NDJSON (`runtime-trace.ndjson.gz`) with `symbol_id`, `purl`, `timestamp`, `pid`, `container`, `count`.
+2) **Stage to CAS**: upload file, record `sha256`, store at `cas://reachability/runtime/<sha256>`.
+3) **Optionally sign**: wrap CAS digest in DSSE (`stella attest runtime --bundle runtime.dsse.json`).
+4) **Ingest** via Signals API:
+   ```sh
+   curl -H "X-Stella-Tenant: acme" \
+        -H "Content-Type: application/x-ndjson" \
+        --data-binary @runtime-trace.ndjson.gz \
+        "https://signals.example/api/v1/runtime-facts?graph_hash=<graph>"
+   ```
+   Headers returned: `Content-SHA256`, `X-Graph-Hash`, `X-Ingest-Id`.
+5) **Emit timeline**: ensure Timeline event `reach.runtime.ingested` with CAS digest and ingest id.
+6) **Verify**: run `stella graph verify --runtime runtime-trace.ndjson.gz --graph <graph_hash>` to confirm edges mapped.
+
+## 3. Ingestion workflow (air-gap)
+1) Receive runtime bundle containing `runtime-trace.ndjson.gz`, `manifest.json` (hashes), optional DSSE.
+2) Validate hashes against manifest; if present, verify DSSE bundle.
+3) Import into CAS path `cas://reachability/runtime/<sha256>` using offline loader.
+4) Run Signals offline ingest tool:
+   ```sh
+   signals-offline ingest-runtime \
+     --tenant acme \
+     --graph-hash <graph_hash> \
+     --runtime runtime-trace.ndjson.gz \
+     --manifest manifest.json
+   ```
+5) Export ingest receipt and add to Evidence Locker; update Timeline when reconnected.
+
+## 4. Checks & alerts
+- **Drift**: block ingest if time anchor age > configured budget; surface `staleness_seconds`.
+- **Hash mismatch**: fail ingest; write `runtime.ingest.failed` event with reason.
+- **Orphan traces**: if no matching `graph_hash`, queue for retry and alert `reachability.orphan_traces` counter.
+
+## 5. Troubleshooting
+- **400 Bad Request**: validate NDJSON schema; run `scripts/reachability/validate_runtime_trace.py`.
+- **Hash mismatch**: recompute `sha256sum runtime-trace.ndjson.gz`; compare to manifest.
+- **Missing symbols**: ensure symbol manifest ingested (see `docs/specs/symbols/SYMBOL_MANIFEST_v1.md`); rerun `stella graph verify`.
+- **High drift**: refresh time anchor (AirGap Time service) or resync NTP; retry ingest.
+
+## 6. Artefact checklist
+- `runtime-trace.ndjson.gz` (or `.json`), `sha256` recorded.
+- Optional `runtime.dsse.json` DSSE bundle.
+- Ingest receipt (ingest id, graph hash, CAS digest, tenant).
+- Timeline event `reach.runtime.ingested` and Evidence Locker record (bundle + receipt).
+
+## 7. References
+- `docs/modules/reach-graph/guides/DELIVERY_GUIDE.md`
+- `docs/modules/reach-graph/guides/function-level-evidence.md`
+- `docs/modules/reach-graph/guides/evidence-schema.md`
+- `docs/specs/symbols/SYMBOL_MANIFEST_v1.md`