git.stella-ops.org/docs/notifications/operations/retries.md

Notification Retries

This document describes the retry mechanism for failed notification deliveries.

Overview

The retry system ensures reliable notification delivery even when temporary failures occur.

Retry Strategy

Exponential Backoff

• Initial delay: 5 seconds
• Maximum delay: 1 hour
• Backoff multiplier: 2x

Retry Limits

• Maximum attempts: 10
• Maximum retry duration: 24 hours

Retry Conditions

• Network errors: Always retry
• HTTP 5xx errors: Always retry
• HTTP 429 (rate limit): Retry with Retry-After header
• HTTP 4xx errors: Do not retry (permanent failure)

Dead Letter Queue

Notifications that exceed retry limits are moved to the dead letter queue for:

• Manual inspection
• Automatic alerting
• Scheduled reprocessing

Monitoring

Retry metrics are exposed for:

• Retry count per notification
• Success rate after retries
• Average retry duration