# Runbook: Release Orchestrator - Required Evidence Not Found

> **Sprint:** SPRINT_20260117_029_DOCS_runbook_coverage
> **Task:** RUN-004 - Release Orchestrator Runbooks

## Metadata

| Field | Value |
|-------|-------|
| **Component** | Release Orchestrator |
| **Severity** | High |
| **On-call scope** | Platform team, Security team |
| **Last updated** | 2026-01-17 |
| **Doctor check** | `check.orchestrator.evidence-availability` |

---

## Symptoms

- [ ] Promotion failing with "required evidence not found"
- [ ] Alert `OrchestratorEvidenceMissing` firing
- [ ] Gate evaluation blocked waiting for evidence
- [ ] Error: "SBOM not found" or "attestation missing"
- [ ] Evidence chain incomplete for artifact

---

## Impact

| Impact Type | Description |
|-------------|-------------|
| **User-facing** | Promotion blocked until evidence is generated |
| **Data integrity** | Indicates missing security artifact - must be resolved |
| **SLA impact** | Release blocked; compliance requirements not met |

---

## Diagnosis

### Quick checks

1. **Check Doctor diagnostics:**
   ```bash
   stella doctor --check check.orchestrator.evidence-availability
   ```

2. **List missing evidence for promotion:**
   ```bash
   stella promotion evidence <promotion-id> --missing
   ```

3. **Check what evidence exists for artifact:**
   ```bash
   stella evidence list --artifact <digest>
   ```

### Deep diagnosis

1. **Check evidence chain completeness:**
   ```bash
   stella evidence chain --artifact <digest> --verbose
   ```
   Look for: Missing nodes in the chain

2. **Check if scan completed:**
   ```bash
   stella scanner jobs list --artifact <digest>
   ```
   Problem if: No completed scan or scan failed

3. **Check if attestation was created:**
   ```bash
   stella attest list --subject <digest>
   ```
   Problem if: No attestation or attestation failed

4. **Check evidence store health:**
   ```bash
   stella evidence store health
   ```

---

## Resolution

### Immediate mitigation

1. **Generate missing SBOM:**
   ```bash
   stella scan image --image <image-ref> --sbom-only
   ```

2. **Generate missing attestation:**
   ```bash
   stella attest create --subject <digest> --type slsa-provenance
   ```

3. **Re-scan artifact to regenerate all evidence:**
   ```bash
   stella scan image --image <image-ref> --force
   ```

### Root cause fix

**If scan never ran:**

1. Check why artifact wasn't scanned:
   ```bash
   stella scanner queue list --artifact <digest>
   ```

2. Configure automatic scanning on push:
   ```bash
   stella scanner config set auto_scan.enabled true
   stella scanner config set auto_scan.triggers "push,promote"
   ```

**If evidence was generated but not stored:**

1. Check evidence store connectivity:
   ```bash
   stella evidence store health
   ```

2. Retry evidence storage:
   ```bash
   stella evidence retry-store --artifact <digest>
   ```

**If attestation signing failed:**

1. Check attestor status:
   ```bash
   stella attest status
   ```

2. See `attestor-signing-failed.md` runbook

**If evidence expired or was deleted:**

1. Check evidence retention policy:
   ```bash
   stella evidence policy show
   ```

2. Regenerate evidence:
   ```bash
   stella scan image --image <image-ref> --force
   stella attest create --subject <digest> --type slsa-provenance
   ```

### Verification

```bash
# Check all evidence now exists
stella evidence list --artifact <digest>

# Verify evidence chain is complete
stella evidence chain --artifact <digest>

# Retry promotion
stella promotion retry <promotion-id>

# Verify promotion proceeds
stella promotion status <promotion-id>
```

---

## Prevention

- [ ] **Auto-scan:** Enable automatic scanning for all pushed images
- [ ] **Gates:** Configure evidence requirements clearly in promotion policy
- [ ] **Monitoring:** Alert on evidence generation failures
- [ ] **Retention:** Set appropriate evidence retention periods

---

## Related Resources

- **Architecture:** `docs/modules/evidence-locker/architecture.md`
- **Related runbooks:** `orchestrator-promotion-stuck.md`, `attestor-signing-failed.md`
- **Evidence requirements:** `docs/operations/evidence-requirements.md`