Files
git.stella-ops.org/docs/doctor/articles/observability/observability-alerting.md
2026-03-31 23:26:24 +03:00

1.8 KiB

checkId, plugin, severity, tags
checkId plugin severity tags
check.observability.alerting stellaops.doctor.observability info
observability
alerting
notifications

Alerting Configuration

What It Checks

Looks for configured alert destinations such as Alertmanager, Slack, email recipients, or PagerDuty routing keys.

The check reports info when alerting is explicitly disabled or when no destination is configured. It warns only when a destination is present but obviously malformed, such as invalid email addresses.

Why It Matters

Metrics and logs are not actionable if nobody is notified when thresholds are crossed. Production installs should route alerts somewhere outside the application process.

Common Causes

  • Alerting was never configured after initial compose bring-up
  • Notification secrets were omitted from environment variables
  • Recipient lists contain placeholders or invalid values

How to Fix

Docker Compose

docker compose -f devops/compose/docker-compose.stella-ops.yml exec doctor-web printenv | grep -E 'ALERT|SLACK|PAGERDUTY|SMTP'

Example compose-style configuration:

services:
  doctor-web:
    environment:
      Alerting__Enabled: "true"
      Alerting__AlertManagerUrl: http://alertmanager:9093
      Alerting__Email__Recipients__0: ops@example.com

Bare Metal / systemd

Configure Alerting:* settings in the service configuration and ensure secrets come from the platform secrets provider rather than clear text files.

Kubernetes / Helm

Store webhook URLs and routing keys in Secrets, then mount them into Alerting:* values.

Verification

stella doctor --check check.observability.alerting
  • check.observability.metrics - alerting is usually driven by metrics
  • check.observability.logging - logs are the fallback when alerts are missing