# Drift Remediation Automation

## Overview

Drift Remediation Automation extends the existing drift detection system with intelligent, policy-driven automatic remediation. While drift detection identifies divergence between expected and actual state, remediation automation closes the loop by taking corrective action without manual intervention.

This is a best-in-class implementation that balances automation with safety, providing configurable remediation strategies, severity-based prioritization, and comprehensive audit trails.

---

## Design Principles

1. **Safety First**: Auto-remediation never executes without explicit policy authorization
2. **Gradual Escalation**: Start with notifications, escalate to remediation based on drift age/severity
3. **Deterministic Actions**: Remediation produces identical outcomes for identical drift states
4. **Full Auditability**: Every remediation action generates signed evidence packets
5. **Blast Radius Control**: Limit concurrent remediations; prevent cascading failures
6. **Human Override**: Operators can pause, cancel, or override any remediation

---

## Architecture

### Component Overview

```
┌─────────────────────────────────────────────────────────────────────┐
│                     Drift Remediation System                        │
├─────────────────────────────────────────────────────────────────────┤
│                                                                     │
│  ┌─────────────────┐    ┌──────────────────┐    ┌───────────────┐  │
│  │ DriftDetector   │───▶│ RemediationEngine│───▶│ ActionExecutor│  │
│  │ (existing)      │    │                  │    │               │  │
│  └─────────────────┘    └──────────────────┘    └───────────────┘  │
│           │                      │                      │          │
│           ▼                      ▼                      ▼          │
│  ┌─────────────────┐    ┌──────────────────┐    ┌───────────────┐  │
│  │ SeverityScorer  │    │ PolicyEvaluator  │    │ EvidenceWriter│  │
│  │                 │    │                  │    │               │  │
│  └─────────────────┘    └──────────────────┘    └───────────────┘  │
│           │                      │                      │          │
│           ▼                      ▼                      ▼          │
│  ┌─────────────────┐    ┌──────────────────┐    ┌───────────────┐  │
│  │ AlertRouter     │    │ ReconcileScheduler│   │ MetricsEmitter│  │
│  │                 │    │                  │    │               │  │
│  └─────────────────┘    └──────────────────┘    └───────────────┘  │
│                                                                     │
└─────────────────────────────────────────────────────────────────────┘
```

### Key Components

#### 1. SeverityScorer

Calculates drift severity based on multiple weighted factors:

```csharp
public sealed record DriftSeverity
{
    public DriftSeverityLevel Level { get; init; }      // Critical, High, Medium, Low, Info
    public int Score { get; init; }                     // 0-100 numeric score
    public ImmutableArray<SeverityFactor> Factors { get; init; }
    public TimeSpan DriftAge { get; init; }
    public bool RequiresImmediate { get; init; }
}

public enum DriftSeverityLevel
{
    Info = 0,       // Cosmetic differences (labels, annotations)
    Low = 25,       // Non-critical drift (resource limits changed)
    Medium = 50,    // Functional drift (ports, volumes)
    High = 75,      // Security drift (image digest mismatch)
    Critical = 100  // Severe drift (container missing, wrong image)
}
```

**Severity Factors:**

| Factor | Weight | Description |
|--------|--------|-------------|
| Drift Type | 30% | Missing > Digest Mismatch > Status Mismatch > Unexpected |
| Drift Age | 25% | Older drift = higher severity |
| Environment Criticality | 20% | Production > Staging > Development |
| Component Criticality | 15% | Core services weighted higher |
| Blast Radius | 10% | Number of dependent services affected |

#### 2. RemediationPolicy

Defines when and how to remediate drift:

```csharp
public sealed record RemediationPolicy
{
    public Guid Id { get; init; }
    public string Name { get; init; }
    public Guid EnvironmentId { get; init; }

    // Triggers
    public RemediationTrigger Trigger { get; init; }
    public DriftSeverityLevel MinimumSeverity { get; init; }
    public TimeSpan MinimumDriftAge { get; init; }
    public TimeSpan MaximumDriftAge { get; init; }  // Escalate to manual if exceeded

    // Actions
    public RemediationAction Action { get; init; }
    public RemediationStrategy Strategy { get; init; }

    // Safety limits
    public int MaxConcurrentRemediations { get; init; }
    public int MaxRemediationsPerHour { get; init; }
    public TimeSpan CooldownPeriod { get; init; }

    // Schedule
    public RemediationWindow? MaintenanceWindow { get; init; }
    public ImmutableArray<DayOfWeek> AllowedDays { get; init; }
    public TimeOnly AllowedStartTime { get; init; }
    public TimeOnly AllowedEndTime { get; init; }

    // Notifications
    public NotificationConfig Notifications { get; init; }
}

public enum RemediationTrigger
{
    Immediate,          // Remediate as soon as detected
    Scheduled,          // Wait for maintenance window
    AgeThreshold,       // Remediate after drift exceeds age
    SeverityEscalation, // Remediate when severity increases
    Manual              // Notification only, human initiates
}

public enum RemediationAction
{
    NotifyOnly,         // Alert but don't act
    Reconcile,          // Restore to expected state
    Rollback,           // Rollback to previous known-good release
    Scale,              // Adjust replica count
    Restart,            // Restart containers
    Quarantine          // Isolate drifted targets from traffic
}

public enum RemediationStrategy
{
    AllAtOnce,          // Remediate all drifted targets simultaneously
    Rolling,            // Remediate one at a time with health checks
    Canary,             // Remediate one, verify, then proceed
    BlueGreen           // Deploy to standby, switch traffic
}
```

#### 3. RemediationEngine

Orchestrates the remediation process:

```csharp
public sealed class RemediationEngine
{
    public async Task<RemediationPlan> CreatePlanAsync(
        DriftReport driftReport,
        RemediationPolicy policy,
        CancellationToken ct)
    {
        // 1. Score severity for each drift item
        var scoredDrifts = await _severityScorer.ScoreAsync(driftReport.Items, ct);

        // 2. Filter by policy thresholds
        var actionable = scoredDrifts
            .Where(d => d.Severity.Level >= policy.MinimumSeverity)
            .Where(d => d.Severity.DriftAge >= policy.MinimumDriftAge)
            .ToImmutableArray();

        // 3. Check maintenance window
        if (!IsWithinMaintenanceWindow(policy))
            return RemediationPlan.Deferred(actionable, policy.MaintenanceWindow);

        // 4. Check rate limits
        var allowed = await CheckRateLimitsAsync(actionable, policy, ct);

        // 5. Build execution plan
        return BuildExecutionPlan(allowed, policy);
    }

    public async Task<RemediationResult> ExecuteAsync(
        RemediationPlan plan,
        CancellationToken ct)
    {
        // Execute with blast radius control
        var semaphore = new SemaphoreSlim(plan.Policy.MaxConcurrentRemediations);
        var results = new ConcurrentBag<TargetRemediationResult>();

        foreach (var batch in plan.Batches)
        {
            var tasks = batch.Targets.Select(async target =>
            {
                await semaphore.WaitAsync(ct);
                try
                {
                    return await RemediateTargetAsync(target, plan, ct);
                }
                finally
                {
                    semaphore.Release();
                }
            });

            var batchResults = await Task.WhenAll(tasks);
            results.AddRange(batchResults);

            // Health check between batches for rolling strategy
            if (plan.Policy.Strategy == RemediationStrategy.Rolling)
            {
                await VerifyBatchHealthAsync(batchResults, ct);
            }
        }

        // Generate evidence
        var evidence = await _evidenceWriter.WriteAsync(plan, results, ct);

        return new RemediationResult(plan.Id, results.ToImmutableArray(), evidence);
    }
}
```

#### 4. ReconcileScheduler

Manages scheduled reconciliation runs:

```csharp
public sealed class ReconcileScheduler
{
    private readonly TimeProvider _timeProvider;
    private readonly IRemediationPolicyStore _policyStore;
    private readonly IDriftDetector _driftDetector;
    private readonly RemediationEngine _engine;

    public async Task RunScheduledReconciliationAsync(CancellationToken ct)
    {
        var policies = await _policyStore.GetScheduledPoliciesAsync(ct);

        foreach (var policy in policies)
        {
            if (!IsWithinWindow(policy))
                continue;

            // Detect drift
            var inventory = await _inventoryService.GetCurrentAsync(policy.EnvironmentId, ct);
            var expected = await _releaseService.GetExpectedStateAsync(policy.EnvironmentId, ct);
            var drift = _driftDetector.Detect(inventory, expected);

            if (drift.HasDrift)
            {
                var plan = await _engine.CreatePlanAsync(drift, policy, ct);
                await _engine.ExecuteAsync(plan, ct);
            }
        }
    }
}
```

---

## Data Models

### RemediationPlan

```csharp
public sealed record RemediationPlan
{
    public Guid Id { get; init; }
    public Guid DriftReportId { get; init; }
    public RemediationPolicy Policy { get; init; }
    public RemediationPlanStatus Status { get; init; }
    public ImmutableArray<RemediationBatch> Batches { get; init; }
    public DateTimeOffset CreatedAt { get; init; }
    public DateTimeOffset? ScheduledFor { get; init; }
    public DateTimeOffset? StartedAt { get; init; }
    public DateTimeOffset? CompletedAt { get; init; }
    public string? DeferralReason { get; init; }
}

public enum RemediationPlanStatus
{
    Created,
    Scheduled,
    Deferred,       // Waiting for maintenance window
    Running,
    Paused,         // Human intervention requested
    Succeeded,
    PartialSuccess, // Some targets remediated, some failed
    Failed,
    Cancelled
}

public sealed record RemediationBatch
{
    public int Order { get; init; }
    public ImmutableArray<RemediationTarget> Targets { get; init; }
    public TimeSpan? DelayAfter { get; init; }
    public bool RequiresHealthCheck { get; init; }
}

public sealed record RemediationTarget
{
    public Guid TargetId { get; init; }
    public string TargetName { get; init; }
    public DriftItem Drift { get; init; }
    public DriftSeverity Severity { get; init; }
    public RemediationAction Action { get; init; }
    public string? ActionPayload { get; init; }  // Compose file, rollback digest, etc.
}
```

### RemediationResult

```csharp
public sealed record RemediationResult
{
    public Guid PlanId { get; init; }
    public RemediationResultStatus Status { get; init; }
    public ImmutableArray<TargetRemediationResult> TargetResults { get; init; }
    public Guid EvidencePacketId { get; init; }
    public TimeSpan Duration { get; init; }
    public RemediationMetrics Metrics { get; init; }
}

public sealed record TargetRemediationResult
{
    public Guid TargetId { get; init; }
    public RemediationTargetStatus Status { get; init; }
    public string? Error { get; init; }
    public TimeSpan Duration { get; init; }
    public string? PreviousDigest { get; init; }
    public string? CurrentDigest { get; init; }
    public ImmutableArray<string> Logs { get; init; }
}

public sealed record RemediationMetrics
{
    public int TotalTargets { get; init; }
    public int Succeeded { get; init; }
    public int Failed { get; init; }
    public int Skipped { get; init; }
    public TimeSpan TotalDuration { get; init; }
    public TimeSpan AverageTargetDuration { get; init; }
}
```

---

## API Design

### REST Endpoints

```
# Policies
POST   /api/v1/remediation/policies                    # Create policy
GET    /api/v1/remediation/policies                    # List policies
GET    /api/v1/remediation/policies/{id}               # Get policy
PUT    /api/v1/remediation/policies/{id}               # Update policy
DELETE /api/v1/remediation/policies/{id}               # Delete policy
POST   /api/v1/remediation/policies/{id}/activate      # Activate policy
POST   /api/v1/remediation/policies/{id}/deactivate    # Deactivate policy

# Plans
GET    /api/v1/remediation/plans                       # List plans
GET    /api/v1/remediation/plans/{id}                  # Get plan details
POST   /api/v1/remediation/plans/{id}/execute          # Execute deferred plan
POST   /api/v1/remediation/plans/{id}/pause            # Pause running plan
POST   /api/v1/remediation/plans/{id}/resume           # Resume paused plan
POST   /api/v1/remediation/plans/{id}/cancel           # Cancel plan

# On-demand
POST   /api/v1/remediation/preview                     # Preview remediation (dry-run)
POST   /api/v1/remediation/execute                     # Execute immediate remediation

# History
GET    /api/v1/remediation/history                     # List remediation history
GET    /api/v1/remediation/history/{id}                # Get remediation result
GET    /api/v1/remediation/history/{id}/evidence       # Get evidence packet
```

### WebSocket Events

```typescript
// Real-time remediation updates
interface RemediationEvent {
  type: 'plan.created' | 'plan.started' | 'plan.completed' |
        'target.started' | 'target.completed' | 'target.failed';
  planId: string;
  targetId?: string;
  status: string;
  progress?: number;
  message?: string;
  timestamp: string;
}
```

---

## Severity Scoring Algorithm

```csharp
public sealed class SeverityScorer
{
    private readonly SeverityScoringConfig _config;

    public DriftSeverity Score(DriftItem drift, ScoringContext context)
    {
        var factors = new List<SeverityFactor>();
        var score = 0.0;

        // Factor 1: Drift Type (30%)
        var typeScore = drift.Type switch
        {
            DriftType.Missing => 100,
            DriftType.DigestMismatch => 80,
            DriftType.StatusMismatch => 50,
            DriftType.Unexpected => 30,
            _ => 10
        };
        factors.Add(new SeverityFactor("DriftType", typeScore, 0.30));
        score += typeScore * 0.30;

        // Factor 2: Drift Age (25%)
        var ageScore = CalculateAgeScore(drift.DetectedAt, context.Now);
        factors.Add(new SeverityFactor("DriftAge", ageScore, 0.25));
        score += ageScore * 0.25;

        // Factor 3: Environment Criticality (20%)
        var envScore = context.Environment.Criticality switch
        {
            EnvironmentCriticality.Production => 100,
            EnvironmentCriticality.Staging => 60,
            EnvironmentCriticality.Development => 20,
            _ => 10
        };
        factors.Add(new SeverityFactor("EnvironmentCriticality", envScore, 0.20));
        score += envScore * 0.20;

        // Factor 4: Component Criticality (15%)
        var componentScore = context.ComponentCriticality.GetValueOrDefault(drift.ComponentId, 50);
        factors.Add(new SeverityFactor("ComponentCriticality", componentScore, 0.15));
        score += componentScore * 0.15;

        // Factor 5: Blast Radius (10%)
        var blastScore = CalculateBlastRadius(drift, context.DependencyGraph);
        factors.Add(new SeverityFactor("BlastRadius", blastScore, 0.10));
        score += blastScore * 0.10;

        return new DriftSeverity
        {
            Level = ScoreToLevel((int)score),
            Score = (int)score,
            Factors = factors.ToImmutableArray(),
            DriftAge = context.Now - drift.DetectedAt,
            RequiresImmediate = score >= 90
        };
    }

    private int CalculateAgeScore(DateTimeOffset detectedAt, DateTimeOffset now)
    {
        var age = now - detectedAt;
        return age.TotalMinutes switch
        {
            < 5 => 10,      // Very fresh - low urgency
            < 30 => 30,     // Recent
            < 60 => 50,     // 1 hour
            < 240 => 70,    // 4 hours
            < 1440 => 85,   // 24 hours
            _ => 100        // > 24 hours - critical
        };
    }

    private int CalculateBlastRadius(DriftItem drift, DependencyGraph graph)
    {
        var dependents = graph.GetDependents(drift.ComponentId);
        return dependents.Count switch
        {
            0 => 10,
            < 3 => 30,
            < 10 => 60,
            < 25 => 80,
            _ => 100
        };
    }
}
```

---

## Safety Mechanisms

### 1. Rate Limiting

```csharp
public sealed class RemediationRateLimiter
{
    public async Task<RateLimitResult> CheckAsync(
        RemediationPolicy policy,
        int requestedCount,
        CancellationToken ct)
    {
        var hourlyCount = await GetHourlyRemediationCountAsync(policy.Id, ct);
        var dailyCount = await GetDailyRemediationCountAsync(policy.Id, ct);

        if (hourlyCount + requestedCount > policy.MaxRemediationsPerHour)
        {
            return RateLimitResult.Exceeded(
                $"Hourly limit exceeded: {hourlyCount}/{policy.MaxRemediationsPerHour}");
        }

        var lastRemediation = await GetLastRemediationAsync(policy.Id, ct);
        if (lastRemediation != null)
        {
            var timeSinceLast = _timeProvider.GetUtcNow() - lastRemediation.CompletedAt;
            if (timeSinceLast < policy.CooldownPeriod)
            {
                return RateLimitResult.Cooldown(policy.CooldownPeriod - timeSinceLast);
            }
        }

        return RateLimitResult.Allowed(requestedCount);
    }
}
```

### 2. Blast Radius Control

```csharp
// Maximum percentage of targets that can be remediated in one operation
public const int MaxTargetPercentage = 25;

// Never remediate more than this many targets at once
public const int AbsoluteMaxTargets = 10;

// Minimum healthy targets required before remediation
public const double MinHealthyPercentage = 0.75;
```

### 3. Circuit Breaker

```csharp
public sealed class RemediationCircuitBreaker
{
    private int _consecutiveFailures;
    private DateTimeOffset? _openedAt;

    public bool IsOpen => _openedAt != null &&
        (_timeProvider.GetUtcNow() - _openedAt.Value) < _config.OpenDuration;

    public void RecordSuccess()
    {
        _consecutiveFailures = 0;
        _openedAt = null;
    }

    public void RecordFailure()
    {
        _consecutiveFailures++;
        if (_consecutiveFailures >= _config.FailureThreshold)
        {
            _openedAt = _timeProvider.GetUtcNow();
            _logger.LogWarning("Remediation circuit breaker opened after {Failures} failures",
                _consecutiveFailures);
        }
    }
}
```

---

## Metrics & Observability

### Prometheus Metrics

```
# Counters
stella_remediation_plans_total{environment, policy, status}
stella_remediation_targets_total{environment, action, status}
stella_remediation_rate_limit_hits_total{policy}

# Histograms
stella_remediation_plan_duration_seconds{environment, strategy}
stella_remediation_target_duration_seconds{environment, action}
stella_remediation_detection_to_action_seconds{environment, severity}

# Gauges
stella_drift_items_pending_remediation{environment, severity}
stella_remediation_circuit_breaker_open{policy}
```

### Structured Logging

```json
{
  "event": "remediation.target.completed",
  "plan_id": "abc-123",
  "target_id": "target-456",
  "environment": "production",
  "action": "reconcile",
  "drift_type": "digest_mismatch",
  "severity": "high",
  "duration_ms": 4532,
  "status": "succeeded",
  "previous_digest": "sha256:abc...",
  "current_digest": "sha256:def...",
  "correlation_id": "xyz-789"
}
```

---

## Evidence Generation

Every remediation produces a sealed evidence packet:

```csharp
public sealed record RemediationEvidence
{
    // What drifted
    public ImmutableArray<DriftItem> DetectedDrift { get; init; }
    public ImmutableArray<DriftSeverity> Severities { get; init; }

    // Policy applied
    public RemediationPolicy Policy { get; init; }

    // Plan executed
    public RemediationPlan Plan { get; init; }

    // Results
    public ImmutableArray<TargetRemediationResult> Results { get; init; }

    // Who/when
    public string InitiatedBy { get; init; }  // "system:auto" or user ID
    public DateTimeOffset InitiatedAt { get; init; }
    public DateTimeOffset CompletedAt { get; init; }

    // Artifacts
    public ImmutableArray<string> GeneratedArtifacts { get; init; }  // Compose files, scripts
}
```

---

## Configuration

### Default Policy Template

```yaml
name: "production-auto-remediation"
environment_id: "prod-001"

trigger: age_threshold
minimum_severity: high
minimum_drift_age: "00:15:00"  # 15 minutes
maximum_drift_age: "24:00:00"  # 24 hours, then escalate to manual

action: reconcile
strategy: rolling

safety:
  max_concurrent_remediations: 2
  max_remediations_per_hour: 10
  cooldown_period: "00:05:00"  # 5 minutes between remediations

schedule:
  maintenance_window:
    enabled: true
    start: "02:00"
    end: "06:00"
    timezone: "UTC"
  allowed_days: [monday, tuesday, wednesday, thursday, friday]

notifications:
  on_plan_created: true
  on_remediation_started: true
  on_remediation_completed: true
  on_remediation_failed: true
  channels:
    - type: slack
      channel: "#ops-alerts"
    - type: email
      recipients: ["ops-team@example.com"]
```

---

## Test Strategy

### Unit Tests

- Severity scoring with various drift combinations
- Rate limiting logic
- Circuit breaker state transitions
- Policy evaluation with edge cases

### Integration Tests

- Full remediation flow: detect → plan → execute → verify
- Maintenance window enforcement
- Rate limit enforcement across multiple requests
- Evidence packet generation and signing

### Chaos Tests

- Agent failure during remediation
- Database unavailability during plan execution
- Concurrent remediation requests
- Clock skew handling

### Golden Tests

- Deterministic severity scores for fixed inputs
- Deterministic plan generation for fixed drift reports
- Evidence packet structure validation

---

## Migration Path

### Phase 1: Foundation (Week 1-2)
- Severity scoring service
- Remediation policy model and store
- Basic API endpoints

### Phase 2: Engine (Week 3-4)
- Remediation engine implementation
- Plan creation and execution
- Target remediation logic

### Phase 3: Safety (Week 5)
- Rate limiting
- Circuit breaker
- Blast radius controls

### Phase 4: Scheduling (Week 6)
- Maintenance window support
- Scheduled reconciliation
- Age-based escalation

### Phase 5: Observability (Week 7)
- Metrics emission
- Evidence generation
- Alert integration

### Phase 6: UI & Polish (Week 8)
- Web console integration
- Real-time updates
- Policy management UI