# Runbook: [Component] - [Failure Scenario]

> **Sprint:** SPRINT_20260117_029_DOCS_runbook_coverage
> **Task:** RUN-001 - Runbook Template

## Metadata

| Field | Value |
|-------|-------|
| **Component** | [Module name: Scanner, Policy, Orchestrator, Attestor, etc.] |
| **Severity** | Critical / High / Medium / Low |
| **On-call scope** | [Who should be paged: Platform team, Security team, etc.] |
| **Last updated** | [YYYY-MM-DD] |
| **Doctor check** | [Check ID if applicable, e.g., `check.scanner.worker-health`] |

---

## Symptoms

Observable indicators that this failure is occurring:

- [ ] [Symptom 1: e.g., "Scan jobs stuck in pending state for >5 minutes"]
- [ ] [Symptom 2: e.g., "Error logs contain 'worker timeout exceeded'"]
- [ ] [Metric/alert that fires: e.g., "Alert `ScannerWorkerStuck` firing"]

---

## Impact

| Impact Type | Description |
|-------------|-------------|
| **User-facing** | [e.g., "New scans cannot complete, blocking CI/CD pipelines"] |
| **Data integrity** | [e.g., "No data loss, but stale scan results may be served"] |
| **SLA impact** | [e.g., "Scan latency SLO violated if not resolved within 15 minutes"] |

---

## Diagnosis

### Quick checks (< 2 minutes)

Run these first to confirm the failure:

1. **Check Doctor diagnostics:**
   ```bash
   stella doctor --check [relevant-check-id]
   ```

2. **Check service status:**
   ```bash
   stella [component] status
   ```

3. **Check recent logs:**
   ```bash
   stella [component] logs --tail 50 --level error
   ```

### Deep diagnosis (if quick checks inconclusive)

1. **[Investigation step 1]:**
   ```bash
   [command]
   ```
   Expected output: [description]
   If unexpected: [what it means]

2. **[Investigation step 2]:**
   ```bash
   [command]
   ```

3. **Check related services:**
   - Postgres connectivity: `stella doctor --check check.storage.postgres`
   - Valkey connectivity: `stella doctor --check check.storage.valkey`
   - Network connectivity: `stella doctor --check check.network.[target]`

---

## Resolution

### Immediate mitigation (restore service quickly)

Use these steps to restore service, even if root cause isn't fixed yet:

1. **[Mitigation step 1]:**
   ```bash
   [command]
   ```
   This will: [explanation]

2. **[Mitigation step 2]:**
   ```bash
   [command]
   ```

### Root cause fix

Once service is restored, address the underlying issue:

1. **[Fix step 1]:**
   ```bash
   [command]
   ```

2. **[Fix step 2]:**
   ```bash
   [command]
   ```

3. **Verify fix is complete:**
   ```bash
   stella doctor --check [relevant-check-id]
   ```

### Verification

Confirm the issue is fully resolved:

```bash
# Re-run the failing operation
stella [component] [test-command]

# Verify metrics are healthy
stella obs metrics --filter [component] --last 5m

# Verify no new errors in logs
stella [component] logs --tail 20 --level error
```

---

## Prevention

How to prevent this failure from recurring:

- [ ] **Monitoring:** [e.g., "Add alert for queue depth > 100"]
- [ ] **Configuration:** [e.g., "Increase worker count in high-volume environments"]
- [ ] **Code change:** [e.g., "Implement circuit breaker for external service calls"]
- [ ] **Documentation:** [e.g., "Update capacity planning guide"]

---

## Related Resources

- **Architecture doc:** [Link to relevant architecture documentation]
- **Related runbooks:** [Links to related failure scenarios]
- **Doctor check source:** [Link to Doctor check implementation]
- **Grafana dashboard:** [Link to relevant dashboard]

---

## Revision History

| Date | Author | Changes |
|------|--------|---------|
| YYYY-MM-DD | [Name] | Initial version |