docs(ops): Complete operations runbooks for Epic 3500

Sprint 3500.0004.0004 (Documentation & Handoff) - T2 DONE Operations Runbooks Added: - score-replay-runbook.md: Deterministic replay procedures - proof-verification-runbook.md: DSSE/Merkle verification ops - airgap-operations-runbook.md: Offline kit management CLI Reference Docs: - reachability-cli-reference.md - score-proofs-cli-reference.md - unknowns-cli-reference.md Air-Gap Guides: - score-proofs-reachability-airgap-runbook.md Training Materials: - score-proofs-concept-guide.md UI API Clients: - proof.client.ts - reachability.client.ts - unknowns.client.ts All 5 operations runbooks now complete (reachability, unknowns-queue, score-replay, proof-verification, airgap-operations).
2025-12-20 22:30:02 +02:00
parent 09c7155f1b
commit 4b3db9ca85
13 changed files with 5630 additions and 12 deletions
--- a/docs/airgap/score-proofs-reachability-airgap-runbook.md
+++ b/docs/airgap/score-proofs-reachability-airgap-runbook.md
@@ -0,0 +1,616 @@
+# Air-Gap Operations Runbook for Score Proofs & Reachability
+
+> **Version**: 1.0.0  
+> **Sprint**: 3500.0004.0004  
+> **Last Updated**: 2025-12-20
+
+This runbook covers air-gapped operations for Score Proofs and Reachability features, including offline kit deployment, proof verification, and bundle management.
+
+---
+
+## Table of Contents
+
+1. [Overview](#1-overview)
+2. [Offline Kit Deployment](#2-offline-kit-deployment)
+3. [Score Proofs in Air-Gap Mode](#3-score-proofs-in-air-gap-mode)
+4. [Reachability in Air-Gap Mode](#4-reachability-in-air-gap-mode)
+5. [Bundle Import Operations](#5-bundle-import-operations)
+6. [Proof Verification Offline](#6-proof-verification-offline)
+7. [Troubleshooting](#7-troubleshooting)
+8. [Monitoring & Alerting](#8-monitoring--alerting)
+
+---
+
+## 1. Overview
+
+### Air-Gap Modes
+
+| Mode | Network | Use Case |
+|------|---------|----------|
+| **Sealed** | No external connectivity | Classified environments |
+| **Constrained** | Limited egress (allowlist) | Regulated networks |
+| **Hybrid** | Selective connectivity | Standard enterprise |
+
+### Score Proofs Air-Gap Capabilities
+
+| Feature | Sealed | Constrained | Hybrid |
+|---------|--------|-------------|--------|
+| Score computation | ✅ | ✅ | ✅ |
+| Score replay | ✅ | ✅ | ✅ |
+| Proof generation | ✅ | ✅ | ✅ |
+| Proof verification | ✅ | ✅ | ✅ |
+| Rekor logging | ❌ | 🔶 (optional) | ✅ |
+| Feed updates | Bundle | Bundle | Online |
+
+### Reachability Air-Gap Capabilities
+
+| Feature | Sealed | Constrained | Hybrid |
+|---------|--------|-------------|--------|
+| Call graph upload | ✅ | ✅ | ✅ |
+| Reachability compute | ✅ | ✅ | ✅ |
+| Explain queries | ✅ | ✅ | ✅ |
+| Symbol resolution | Bundle | Bundle | Online |
+
+---
+
+## 2. Offline Kit Deployment
+
+### 2.1 Offline Kit Contents
+
+The offline kit contains everything needed for air-gapped Score Proofs and Reachability:
+
+```
+offline-kit/
+├── manifests/
+│   ├── kit-manifest.json          # Kit metadata and versions
+│   ├── feed-manifest.json         # Advisory feed snapshot
+│   └── vex-manifest.json          # VEX data snapshot
+├── feeds/
+│   ├── concelier/                 # Advisory feed data
+│   │   ├── advisories.ndjson
+│   │   └── snapshot.dsse.json
+│   └── excititor/                 # VEX data
+│       ├── vex-statements.ndjson
+│       └── snapshot.dsse.json
+├── policies/
+│   ├── scoring-policy.yaml
+│   └── policy.dsse.json
+├── trust/
+│   ├── trust-anchors.json         # Public keys for verification
+│   └── time-anchor.json           # Time attestation
+├── symbols/
+│   └── symbol-index.db            # Symbol resolution database
+└── tools/
+    ├── stella-cli                 # CLI binary
+    └── verify-kit.sh              # Kit verification script
+```
+
+### 2.2 Verify Kit Integrity
+
+Before deployment, always verify the offline kit:
+
+```bash
+# Verify kit signature
+stella airgap verify-kit --kit /path/to/offline-kit
+
+# Output:
+# Kit manifest: VALID
+# Feed snapshot: VALID (sha256:feed123...)
+# VEX snapshot: VALID (sha256:vex456...)
+# Policy: VALID (sha256:policy789...)
+# Trust anchors: VALID (3 anchors)
+# Time anchor: VALID (expires: 2025-12-31T00:00:00Z)
+# 
+# Kit verification: PASSED
+
+# Verify individual components
+stella airgap verify --file feeds/concelier/snapshot.dsse.json
+stella airgap verify --file feeds/excititor/snapshot.dsse.json
+stella airgap verify --file policies/policy.dsse.json
+```
+
+### 2.3 Deploy Offline Kit
+
+```bash
+# Deploy kit (sealed mode)
+stella airgap deploy --kit /path/to/offline-kit --mode sealed
+
+# Deploy kit (constrained mode with limited egress)
+stella airgap deploy --kit /path/to/offline-kit \
+  --mode constrained \
+  --egress-allowlist https://rekor.sigstore.dev
+
+# Verify deployment
+stella airgap status
+
+# Output:
+# Mode: sealed
+# Kit version: 2025.12.20
+# Feed snapshot: sha256:feed123... (2025-12-20)
+# VEX snapshot: sha256:vex456... (2025-12-20)
+# Policy: sha256:policy789... (v1.2.3)
+# Trust anchors: 3 active
+# Time anchor: Valid until 2025-12-31
+# Staleness: OK (0 days)
+```
+
+### 2.4 Kit Updates
+
+```bash
+# Check for kit updates (requires external access or new media)
+stella airgap check-update --current-kit /path/to/current-kit
+
+# Import new kit
+stella airgap import-kit --kit /path/to/new-kit --validate
+
+# Rollback to previous kit
+stella airgap rollback --generation previous
+```
+
+---
+
+## 3. Score Proofs in Air-Gap Mode
+
+### 3.1 Create Scan (Air-Gap)
+
+```bash
+# Create scan referencing offline kit snapshots
+stella scan create --artifact sha256:abc123... \
+  --airgap \
+  --feed-snapshot sha256:feed123... \
+  --vex-snapshot sha256:vex456... \
+  --policy-snapshot sha256:policy789...
+
+# Or auto-detect from deployed kit
+stella scan create --artifact sha256:abc123... --use-offline-kit
+```
+
+### 3.2 Score Replay (Air-Gap)
+
+```bash
+# Replay with offline kit snapshots
+stella score replay --scan-id $SCAN_ID --offline
+
+# Replay with specific bundle
+stella score replay --scan-id $SCAN_ID \
+  --offline \
+  --bundle /path/to/proof-bundle.zip
+
+# Compare with different kit versions
+stella score replay --scan-id $SCAN_ID \
+  --offline \
+  --feed-snapshot sha256:newfeed... \
+  --diff
+```
+
+### 3.3 Generate Proof Bundle (Air-Gap)
+
+```bash
+# Generate proof bundle for export
+stella proof export --scan-id $SCAN_ID \
+  --include-manifest \
+  --include-chain \
+  --output proof-bundle.zip
+
+# Proof bundle contents (air-gap safe):
+# - manifest.json (canonical)
+# - manifest.dsse.json
+# - score_proof.json
+# - proof_root.dsse.json
+# - meta.json
+# - NO external references
+
+# Generate portable bundle
+stella proof export --scan-id $SCAN_ID \
+  --portable \
+  --include-trust-anchors \
+  --output portable-proof.zip
+```
+
+---
+
+## 4. Reachability in Air-Gap Mode
+
+### 4.1 Call Graph Operations (Air-Gap)
+
+```bash
+# Upload call graph (works identically)
+stella scan graph upload --scan-id $SCAN_ID --file callgraph.json
+
+# Call graph processing is fully local
+# No external network required
+```
+
+### 4.2 Compute Reachability (Air-Gap)
+
+```bash
+# Compute reachability (fully offline)
+stella reachability compute --scan-id $SCAN_ID --offline
+
+# Symbol resolution uses offline database
+stella reachability compute --scan-id $SCAN_ID \
+  --offline \
+  --symbol-db /path/to/offline-kit/symbols/symbol-index.db
+```
+
+### 4.3 Explain Queries (Air-Gap)
+
+```bash
+# Explain queries work offline
+stella reachability explain --scan-id $SCAN_ID \
+  --cve CVE-2024-1234 \
+  --purl "pkg:npm/lodash@4.17.20" \
+  --offline
+
+# Export explanations for external review
+stella reachability explain-all --scan-id $SCAN_ID \
+  --output explanations.json \
+  --offline
+```
+
+---
+
+## 5. Bundle Import Operations
+
+### 5.1 Import Feed Updates
+
+```bash
+# Verify feed bundle before import
+stella airgap verify --bundle feed-update.zip
+
+# Dry-run import
+stella airgap import --bundle feed-update.zip \
+  --type feed \
+  --dry-run
+
+# Import feed bundle
+stella airgap import --bundle feed-update.zip \
+  --type feed \
+  --generation 2025.12.21
+
+# Verify import
+stella airgap verify-import --generation 2025.12.21
+```
+
+### 5.2 Import VEX Updates
+
+```bash
+# Import VEX bundle
+stella airgap import --bundle vex-update.zip \
+  --type vex \
+  --generation 2025.12.21
+
+# Verify VEX statements
+stella airgap vex-status
+
+# Output:
+# VEX statements: 15,432
+# Last update: 2025-12-21
+# Generation: 2025.12.21
+# Signature: VALID
+```
+
+### 5.3 Import Trust Anchors
+
+```bash
+# Import new trust anchor (requires approval)
+stella airgap import-anchor --file new-anchor.json \
+  --reason "Key rotation Q4 2025" \
+  --approver admin@example.com
+
+# Verify anchor chain
+stella airgap verify-anchors
+
+# List active anchors
+stella airgap anchors list
+```
+
+### 5.4 Import Checklist
+
+**Pre-Import**:
+- [ ] Verify bundle signature (DSSE)
+- [ ] Verify bundle hash matches manifest
+- [ ] Confirm sealed/constrained mode is set
+- [ ] Backup current generation
+
+**Import**:
+- [ ] Run dry-run import
+- [ ] Apply import
+- [ ] Verify import succeeded
+
+**Post-Import**:
+- [ ] Verify timeline event emitted
+- [ ] Update staleness dashboard
+- [ ] Archive import manifest
+- [ ] Update audit log
+
+---
+
+## 6. Proof Verification Offline
+
+### 6.1 Verify Proof Bundle (Full Offline)
+
+```bash
+# Verify proof bundle without any network access
+stella proof verify --bundle proof-bundle.zip \
+  --offline \
+  --trust-anchor /path/to/trust-anchors.json
+
+# Verification checks (offline):
+# ✅ Signature valid (DSSE)
+# ✅ Content-addressed ID matches
+# ✅ Merkle path valid
+# ⏭️ Rekor inclusion (SKIPPED - offline mode)
+# ✅ Time anchor valid
+```
+
+### 6.2 Verify with Portable Bundle
+
+```bash
+# Portable bundles include trust anchors
+stella proof verify --bundle portable-proof.zip \
+  --offline \
+  --self-contained
+
+# Output:
+# Using embedded trust anchors
+# Signature verification: PASS
+# ID recomputation: PASS
+# Merkle path: PASS
+# Time anchor: VALID
+# Overall: VERIFIED
+```
+
+### 6.3 Batch Verification
+
+```bash
+# Verify multiple bundles
+stella proof verify-batch --dir /path/to/bundles/ \
+  --offline \
+  --trust-anchor /path/to/trust-anchors.json \
+  --output verification-report.json
+
+# Generate verification report
+cat verification-report.json | jq '.summary'
+# Output:
+# {
+#   "total": 100,
+#   "verified": 98,
+#   "failed": 2,
+#   "skipped": 0
+# }
+```
+
+### 6.4 Verification Without CLI
+
+For environments without the CLI, manual verification is possible:
+
+```bash
+# 1. Extract bundle
+unzip proof-bundle.zip -d ./verify/
+
+# 2. Verify DSSE signature (using openssl)
+# Extract payload from DSSE envelope
+cat ./verify/manifest.dsse.json | jq -r '.payload' | base64 -d > payload.json
+
+# Verify signature
+cat ./verify/manifest.dsse.json | jq -r '.signatures[0].sig' | base64 -d > signature.bin
+openssl dgst -sha256 -verify trust-anchor-pubkey.pem -signature signature.bin payload.json
+
+# 3. Verify content-addressed ID
+# Compute canonical hash
+cat ./verify/manifest.json | jq -cS . | sha256sum
+# Compare with manifestHash in bundle
+
+# 4. Verify merkle path
+# (See docs/airgap/proof-chain-verification.md for algorithm)
+```
+
+---
+
+## 7. Troubleshooting
+
+### 7.1 Kit Verification Failed
+
+**Symptom**: `stella airgap verify-kit` fails.
+
+**Diagnosis**:
+
+```bash
+# Check specific component
+stella airgap verify-kit --verbose --component feeds
+
+# Common errors:
+# - "Signature verification failed": Key mismatch
+# - "Hash mismatch": Bundle corrupted during transfer
+# - "Time anchor expired": Anchor needs refresh
+```
+
+**Resolution**:
+
+| Error | Cause | Resolution |
+|-------|-------|------------|
+| Signature failed | Wrong trust anchor | Import correct anchor |
+| Hash mismatch | Corruption | Re-transfer bundle |
+| Time anchor expired | Clock drift or expired | Import new time anchor |
+
+### 7.2 Staleness Alert
+
+**Symptom**: Staleness warning/alert.
+
+**Diagnosis**:
+
+```bash
+# Check staleness status
+stella airgap staleness-status
+
+# Output:
+# Feed age: 5 days (threshold: 7 days)
+# VEX age: 3 days (threshold: 7 days)
+# Policy age: 30 days (threshold: 90 days)
+# Status: AMBER (approaching threshold)
+```
+
+**Resolution**:
+
+```bash
+# Import updated bundles
+stella airgap import --bundle /path/to/latest-feed.zip --type feed
+
+# If bundles unavailable and breach imminent:
+# - Raise amber alert (5-7 days)
+# - If >7 days, raise red alert and halt new ingests
+# - Request emergency bundle via secure channel
+```
+
+### 7.3 Proof Verification Fails Offline
+
+**Symptom**: Proof verification fails in sealed mode.
+
+**Diagnosis**:
+
+```bash
+# Check verification error
+stella proof verify --bundle proof.zip --offline --verbose
+
+# Common errors:
+# - "Trust anchor not found": Missing anchor in offline kit
+# - "Time anchor expired": Time validation failed
+# - "Unsupported algorithm": Key algorithm not supported
+```
+
+**Resolution**:
+
+```bash
+# For missing trust anchor:
+# Import the required anchor
+stella airgap import-anchor --file required-anchor.json
+
+# For expired time anchor:
+# Import new time anchor
+stella airgap import-time-anchor --file new-time-anchor.json
+
+# For algorithm issues:
+# Regenerate proof with supported algorithm
+stella proof regenerate --scan-id $SCAN_ID --algorithm ECDSA-P256
+```
+
+### 7.4 Symbol Resolution Fails
+
+**Symptom**: Reachability shows "symbol not found" errors.
+
+**Diagnosis**:
+
+```bash
+# Check symbol database status
+stella airgap symbols-status
+
+# Output:
+# Symbol DB: /path/to/symbols/symbol-index.db
+# Version: 2025.12.15
+# Entries: 5,234,567
+# Coverage: Java, .NET, Python
+```
+
+**Resolution**:
+
+```bash
+# Import updated symbol database
+stella airgap import --bundle symbol-update.zip --type symbols
+
+# Recompute reachability with new symbols
+stella reachability compute --scan-id $SCAN_ID --offline --force
+```
+
+---
+
+## 8. Monitoring & Alerting
+
+### 8.1 Key Metrics (Air-Gap)
+
+| Metric | Description | Alert Threshold |
+|--------|-------------|-----------------|
+| `airgap_staleness_days` | Days since last bundle import | > 5 (amber), > 7 (red) |
+| `airgap_time_anchor_validity_days` | Days until time anchor expires | < 7 |
+| `airgap_verification_failures` | Offline verification failures | > 0 |
+| `airgap_import_failures` | Bundle import failures | > 0 |
+
+### 8.2 Alerting Rules
+
+```yaml
+groups:
+  - name: airgap-operations
+    rules:
+      - alert: AirgapStalenessAmber
+        expr: airgap_staleness_days > 5
+        for: 1h
+        labels:
+          severity: warning
+        annotations:
+          summary: "Air-gap feed staleness approaching threshold"
+          
+      - alert: AirgapStalenessRed
+        expr: airgap_staleness_days > 7
+        for: 1h
+        labels:
+          severity: critical
+        annotations:
+          summary: "Air-gap feed staleness breach - halt new ingests"
+          
+      - alert: AirgapTimeAnchorExpiring
+        expr: airgap_time_anchor_validity_days < 7
+        for: 1h
+        labels:
+          severity: warning
+        annotations:
+          summary: "Time anchor expiring in {{ $value }} days"
+          
+      - alert: AirgapVerificationFailure
+        expr: increase(airgap_verification_failures_total[1h]) > 0
+        for: 5m
+        labels:
+          severity: critical
+        annotations:
+          summary: "Air-gap verification failures detected"
+```
+
+### 8.3 Audit Requirements
+
+For air-gapped environments, maintain strict audit trails:
+
+```bash
+# Record every import
+{
+  "timestamp": "2025-12-20T10:00:00Z",
+  "action": "import",
+  "bundleType": "feed",
+  "bundleHash": "sha256:feed123...",
+  "generation": "2025.12.20",
+  "actor": "operator@example.com",
+  "mode": "sealed",
+  "verification": "PASS"
+}
+
+# Daily audit log export
+stella airgap audit-export --date today --output audit-$(date +%Y%m%d).json
+
+# Verify audit log integrity
+stella airgap audit-verify --log audit-20251220.json
+```
+
+---
+
+## Related Documentation
+
+- [Air-Gap Overview](./overview.md)
+- [Offline Bundle Format](./offline-bundle-format.md)
+- [Proof Chain Verification](./proof-chain-verification.md)
+- [Time Anchor Schema](./time-anchor-schema.md)
+- [Score Proofs Runbook](../operations/score-proofs-runbook.md)
+- [Reachability Runbook](../operations/reachability-runbook.md)
+
+---
+
+**Last Updated**: 2025-12-20  
+**Version**: 1.0.0  
+**Sprint**: 3500.0004.0004