206 lines
4.5 KiB
Markdown
206 lines
4.5 KiB
Markdown
# Runbook: Policy Engine - OPA Process Crashed
|
|
|
|
> **Sprint:** SPRINT_20260117_029_DOCS_runbook_coverage
|
|
> **Task:** RUN-003 - Policy Engine Runbooks
|
|
|
|
## Metadata
|
|
|
|
| Field | Value |
|
|
|-------|-------|
|
|
| **Component** | Policy Engine |
|
|
| **Severity** | Critical |
|
|
| **On-call scope** | Platform team |
|
|
| **Last updated** | 2026-01-17 |
|
|
| **Doctor check** | `check.policy.opa-health` |
|
|
|
|
---
|
|
|
|
## Symptoms
|
|
|
|
- [ ] Policy evaluations failing with "OPA unavailable" error
|
|
- [ ] Alert `PolicyOPACrashed` firing
|
|
- [ ] OPA process exited unexpectedly
|
|
- [ ] Error: "connection refused" when connecting to OPA
|
|
- [ ] Metric `policy_opa_restarts_total` increasing
|
|
|
|
---
|
|
|
|
## Impact
|
|
|
|
| Impact Type | Description |
|
|
|-------------|-------------|
|
|
| **User-facing** | All policy evaluations fail; gate decisions blocked |
|
|
| **Data integrity** | No data loss; decisions delayed until OPA recovers |
|
|
| **SLA impact** | Gate latency SLO violated; release pipeline blocked |
|
|
|
|
---
|
|
|
|
## Diagnosis
|
|
|
|
### Quick checks
|
|
|
|
1. **Check Doctor diagnostics:**
|
|
```bash
|
|
stella doctor --check check.policy.opa-health
|
|
```
|
|
|
|
2. **Check OPA process status:**
|
|
```bash
|
|
stella policy status
|
|
```
|
|
Look for: OPA process state, restart count
|
|
|
|
3. **Check OPA logs for crash reason:**
|
|
```bash
|
|
stella policy opa logs --last 30m --level error
|
|
```
|
|
|
|
### Deep diagnosis
|
|
|
|
1. **Check OPA memory usage before crash:**
|
|
```bash
|
|
stella policy stats --opa-metrics
|
|
```
|
|
Problem if: Memory usage near limit before crash
|
|
|
|
2. **Check for problematic policy:**
|
|
```bash
|
|
stella policy list --last-error
|
|
```
|
|
Look for: Policies that caused evaluation errors
|
|
|
|
3. **Check OPA configuration:**
|
|
```bash
|
|
stella policy opa config show
|
|
```
|
|
Look for: Invalid configuration, missing bundles
|
|
|
|
4. **Check for infinite loops in Rego:**
|
|
```bash
|
|
stella policy analyze --detect-loops
|
|
```
|
|
|
|
---
|
|
|
|
## Resolution
|
|
|
|
### Immediate mitigation
|
|
|
|
1. **Restart OPA process:**
|
|
```bash
|
|
stella policy opa restart
|
|
```
|
|
|
|
2. **If OPA keeps crashing, start in safe mode:**
|
|
```bash
|
|
stella policy opa start --safe-mode
|
|
```
|
|
Note: Safe mode disables custom policies
|
|
|
|
3. **Enable failopen temporarily (if allowed by policy):**
|
|
```bash
|
|
stella policy config set failopen true
|
|
stella policy reload
|
|
```
|
|
**Warning:** Only use if compliance allows fail-open mode
|
|
|
|
### Root cause fix
|
|
|
|
**If OOM killed:**
|
|
|
|
1. Increase OPA memory limit:
|
|
```bash
|
|
stella policy opa config set memory_limit 2Gi
|
|
stella policy opa restart
|
|
```
|
|
|
|
2. Enable garbage collection tuning:
|
|
```bash
|
|
stella policy opa config set gc_min_heap_size 256Mi
|
|
stella policy opa config set gc_max_heap_size 1Gi
|
|
```
|
|
|
|
**If policy caused crash:**
|
|
|
|
1. Identify problematic policy:
|
|
```bash
|
|
stella policy list --status error
|
|
```
|
|
|
|
2. Disable the problematic policy:
|
|
```bash
|
|
stella policy disable <policy-id>
|
|
stella policy reload
|
|
```
|
|
|
|
3. Fix and re-enable:
|
|
```bash
|
|
stella policy validate --file <fixed-policy.rego>
|
|
stella policy update <policy-id> --file <fixed-policy.rego>
|
|
stella policy enable <policy-id>
|
|
```
|
|
|
|
**If bundle loading failed:**
|
|
|
|
1. Check bundle integrity:
|
|
```bash
|
|
stella policy bundle verify
|
|
```
|
|
|
|
2. Rebuild bundle:
|
|
```bash
|
|
stella policy bundle build --output bundle.tar.gz
|
|
stella policy bundle load bundle.tar.gz
|
|
```
|
|
|
|
**If configuration issue:**
|
|
|
|
1. Reset to default configuration:
|
|
```bash
|
|
stella policy opa config reset
|
|
```
|
|
|
|
2. Reconfigure with validated settings:
|
|
```bash
|
|
stella policy opa config set workers 4
|
|
stella policy opa config set decision_log true
|
|
stella policy opa restart
|
|
```
|
|
|
|
### Verification
|
|
|
|
```bash
|
|
# Check OPA is running
|
|
stella policy status
|
|
|
|
# Check OPA health
|
|
stella policy opa health
|
|
|
|
# Test policy evaluation
|
|
stella policy evaluate --test
|
|
|
|
# Check no crashes in recent logs
|
|
stella policy opa logs --level error --last 30m
|
|
|
|
# Monitor stability
|
|
stella policy stats --watch
|
|
```
|
|
|
|
---
|
|
|
|
## Prevention
|
|
|
|
- [ ] **Resources:** Set appropriate memory limits based on policy complexity
|
|
- [ ] **Validation:** Validate all policies before deployment
|
|
- [ ] **Monitoring:** Alert on OPA restart count > 2 in 10 minutes
|
|
- [ ] **Testing:** Load test policies before production deployment
|
|
|
|
---
|
|
|
|
## Related Resources
|
|
|
|
- **Architecture:** `docs/modules/policy/architecture.md`
|
|
- **Related runbooks:** `policy-evaluation-slow.md`, `policy-compilation-failed.md`
|
|
- **Doctor check:** `src/Doctor/__Plugins/StellaOps.Doctor.Plugin.Policy/`
|
|
- **OPA documentation:** https://www.openpolicyagent.org/docs/latest/
|