--- checkId: check.notify.queue.health plugin: stellaops.doctor.notify severity: fail tags: [notify, queue, redis, nats, infrastructure] --- # Notification Queue Health ## What It Checks Verifies that the notification event and delivery queues are healthy. The check: - Reads the `Notify:Queue:Transport` (or `Kind`) setting to determine the queue transport type (Redis/Valkey or NATS). - Resolves `NotifyQueueHealthCheck` and `NotifyDeliveryQueueHealthCheck` from the DI container. - Invokes each registered health check and aggregates the results. - Fails if any queue reports an `Unhealthy` status; warns if degraded; passes if all are healthy. The check only runs when a queue transport is configured in `Notify:Queue:Transport`. ## Why It Matters The notification queue is the backbone of the notification pipeline. If the event queue is unhealthy, new notification events are lost. If the delivery queue is unhealthy, pending notifications to email, Slack, Teams, and webhook channels will not be delivered. This is a severity-fail check because queue failure means complete notification blackout. ## Common Causes - Queue server (Redis/Valkey/NATS) not running - Network connectivity issues between the Notify service and the queue server - Authentication failure (wrong password or credentials) - Incorrect connection string in configuration ## How to Fix ### Docker Compose For Redis/Valkey transport: ```bash # Check Redis health docker exec redis-cli ping # Check connection string docker exec env | grep Notify__Queue # Restart Redis if needed docker restart ``` For NATS transport: ```bash # Check NATS server status docker exec nats server ping # Check NATS logs docker logs --tail 50 ``` ### Bare Metal / systemd ```bash # Redis/Valkey redis-cli ping redis-cli info server # NATS nats server ping systemctl status nats ``` Verify the connection string in `appsettings.json`: ```json { "Notify": { "Queue": { "Transport": "redis", "Redis": { "ConnectionString": "127.1.1.2:6379" } } } } ``` ### Kubernetes / Helm ```bash kubectl exec -it -- redis-cli ping kubectl logs --tail 50 | grep -i queue ``` ## Verification ``` stella doctor run --check check.notify.queue.health ``` ## Related Checks - `check.notify.email.configured` — verifies email channel configuration - `check.notify.slack.configured` — verifies Slack channel configuration - `check.notify.webhook.configured` — verifies webhook channel configuration