up

ops/devops/observability/policy-playbook.md

@@ -0,0 +1,39 @@
+# Policy Pipeline Playbook
+
+Scope: policy compile → simulation → approval → promotion path.
+
+## Dashboards
+- Grafana: import `ops/devops/observability/grafana/policy-pipeline.json` (datasource `Prometheus`).
+- Key tiles: Compile p99, Simulation Queue Depth, Approval p95, Promotion Success Rate, Promotion Outcomes.
+
+## Alerts (Prometheus)
+- Rules: `ops/devops/observability/policy-alerts.yaml`
+  - `PolicyCompileLatencyP99High` (p99 > 5s for 10m)
+  - `PolicySimulationQueueBacklog` (queue depth > 100 for 10m)
+  - `PolicyApprovalLatencyHigh` (p95 > 30s for 15m)
+  - `PolicyPromotionFailureRate` (failures >20% over 15m)
+  - `PolicyPromotionStall` (no successes while queue non-empty for 10m)
+
+## Runbook
+1. **Compile latency alert**
+   - Check build nodes for CPU cap; verify cache hits for policy engine.
+   - Roll restart single runner; if persists, scale policy compile workers (+1) or purge stale cache.
+2. **Simulation backlog**
+   - Inspect queue per stage (panel "Queue Depth by Stage").
+   - If queue limited to one stage, increase concurrency for that stage or drain stuck items; otherwise, add workers.
+3. **Approval latency high**
+   - Look for blocked approvals (UI/API outages). Re-run approval service health check; fail over to standby.
+4. **Promotion failure rate/stall**
+   - Pull recent logs for promotion job; compare failure reasons (policy validation vs. target registry).
+   - If registry errors, pause promotions and file incident with registry owner; if policy validation, revert latest policy change or apply override to unblock critical tenants.
+5. **Verification**
+   - After mitigation, ensure promotion success rate gauge recovers >95% and queues drain to baseline (<10).
+
+## Escalation
+- Primary: Policy On-Call (week N roster).
+- Secondary: DevOps Guild (release).
+- Page if two critical alerts fire concurrently or any critical alert lasts >30m.
+
+## Notes
+- Metrics assumed available: `policy_compile_duration_seconds_bucket`, `policy_simulation_queue_depth`, `policy_approval_latency_seconds_bucket`, `policy_promotion_outcomes_total{outcome=*}`.
+- Keep alert thresholds stable unless load profile changes; adjust in Git with approval from Policy + DevOps leads.