synergy moats product advisory implementations

docs/operations/runbooks/orchestrator-evidence-missing.md

@@ -0,0 +1,183 @@
+# Runbook: Release Orchestrator - Required Evidence Not Found
+
+> **Sprint:** SPRINT_20260117_029_DOCS_runbook_coverage
+> **Task:** RUN-004 - Release Orchestrator Runbooks
+
+## Metadata
+
+| Field | Value |
+|-------|-------|
+| **Component** | Release Orchestrator |
+| **Severity** | High |
+| **On-call scope** | Platform team, Security team |
+| **Last updated** | 2026-01-17 |
+| **Doctor check** | `check.orchestrator.evidence-availability` |
+
+---
+
+## Symptoms
+
+- [ ] Promotion failing with "required evidence not found"
+- [ ] Alert `OrchestratorEvidenceMissing` firing
+- [ ] Gate evaluation blocked waiting for evidence
+- [ ] Error: "SBOM not found" or "attestation missing"
+- [ ] Evidence chain incomplete for artifact
+
+---
+
+## Impact
+
+| Impact Type | Description |
+|-------------|-------------|
+| **User-facing** | Promotion blocked until evidence is generated |
+| **Data integrity** | Indicates missing security artifact - must be resolved |
+| **SLA impact** | Release blocked; compliance requirements not met |
+
+---
+
+## Diagnosis
+
+### Quick checks
+
+1. **Check Doctor diagnostics:**
+   ```bash
+   stella doctor --check check.orchestrator.evidence-availability
+   ```
+
+2. **List missing evidence for promotion:**
+   ```bash
+   stella promotion evidence <promotion-id> --missing
+   ```
+
+3. **Check what evidence exists for artifact:**
+   ```bash
+   stella evidence list --artifact <digest>
+   ```
+
+### Deep diagnosis
+
+1. **Check evidence chain completeness:**
+   ```bash
+   stella evidence chain --artifact <digest> --verbose
+   ```
+   Look for: Missing nodes in the chain
+
+2. **Check if scan completed:**
+   ```bash
+   stella scanner jobs list --artifact <digest>
+   ```
+   Problem if: No completed scan or scan failed
+
+3. **Check if attestation was created:**
+   ```bash
+   stella attest list --subject <digest>
+   ```
+   Problem if: No attestation or attestation failed
+
+4. **Check evidence store health:**
+   ```bash
+   stella evidence store health
+   ```
+
+---
+
+## Resolution
+
+### Immediate mitigation
+
+1. **Generate missing SBOM:**
+   ```bash
+   stella scan image --image <image-ref> --sbom-only
+   ```
+
+2. **Generate missing attestation:**
+   ```bash
+   stella attest create --subject <digest> --type slsa-provenance
+   ```
+
+3. **Re-scan artifact to regenerate all evidence:**
+   ```bash
+   stella scan image --image <image-ref> --force
+   ```
+
+### Root cause fix
+
+**If scan never ran:**
+
+1. Check why artifact wasn't scanned:
+   ```bash
+   stella scanner queue list --artifact <digest>
+   ```
+
+2. Configure automatic scanning on push:
+   ```bash
+   stella scanner config set auto_scan.enabled true
+   stella scanner config set auto_scan.triggers "push,promote"
+   ```
+
+**If evidence was generated but not stored:**
+
+1. Check evidence store connectivity:
+   ```bash
+   stella evidence store health
+   ```
+
+2. Retry evidence storage:
+   ```bash
+   stella evidence retry-store --artifact <digest>
+   ```
+
+**If attestation signing failed:**
+
+1. Check attestor status:
+   ```bash
+   stella attest status
+   ```
+
+2. See `attestor-signing-failed.md` runbook
+
+**If evidence expired or was deleted:**
+
+1. Check evidence retention policy:
+   ```bash
+   stella evidence policy show
+   ```
+
+2. Regenerate evidence:
+   ```bash
+   stella scan image --image <image-ref> --force
+   stella attest create --subject <digest> --type slsa-provenance
+   ```
+
+### Verification
+
+```bash
+# Check all evidence now exists
+stella evidence list --artifact <digest>
+
+# Verify evidence chain is complete
+stella evidence chain --artifact <digest>
+
+# Retry promotion
+stella promotion retry <promotion-id>
+
+# Verify promotion proceeds
+stella promotion status <promotion-id>
+```
+
+---
+
+## Prevention
+
+- [ ] **Auto-scan:** Enable automatic scanning for all pushed images
+- [ ] **Gates:** Configure evidence requirements clearly in promotion policy
+- [ ] **Monitoring:** Alert on evidence generation failures
+- [ ] **Retention:** Set appropriate evidence retention periods
+
+---
+
+## Related Resources
+
+- **Architecture:** `docs/modules/evidence-locker/architecture.md`
+- **Related runbooks:** `orchestrator-promotion-stuck.md`, `attestor-signing-failed.md`
+- **Evidence requirements:** `docs/operations/evidence-requirements.md`