# Notifications runbook

Purpose
- Deploy and operate the Notifications WebService and Worker.

Pre-flight
- Secrets stored in Authority (SMTP, Slack, webhook HMAC).
- Outbound allowlist configured for channels.
- PostgreSQL and Valkey reachable; health checks pass.
- Offline kit loaded with templates and rule seeds.

Deploy
- Deploy images with digests pinned.
- Set Notify Postgres, Redis, Authority, and allowlist settings.
- Warm caches via /api/v1/notify/admin/warm when needed.

Monitor
- notify_delivery_attempts_total by status and channel.
- notify_escalation_stage_total and notify_rule_eval_seconds.
- Logs include tenant, ruleId, deliveryId, channel, status.

Common operations
- List failed deliveries and replay.
- Pause a tenant without dropping audit events.
- Rotate channel secrets via refresh endpoints.

Failure recovery
- Validate templates and Redis connectivity for worker crashes.
- Replay deliveries after database recovery.
- Disable channels during upstream outages.

Determinism safeguards
- Rule snapshots versioned per tenant.
- Template rendering uses deterministic helpers.
- UTC time sources for quiet hours.

Related references
- notifications/overview.md
- notifications/rules.md
- docs/operations/notifier-runbook.md