38 KiB
38 KiB
Progressive Delivery Enhancements
Overview
Progressive Delivery Enhancements transforms the existing progressive delivery system into a fully automated, metrics-driven deployment platform. This enhancement provides metric-driven canary automation, feature flag integration, automatic traffic percentage calculation based on error rates, and sophisticated rollout strategies.
This is a best-in-class implementation inspired by Argo Rollouts, Flagger, and modern GitOps practices, tailored for non-Kubernetes environments.
Design Principles
- Metrics-Driven Decisions: All traffic shifts based on objective data
- Fail-Fast, Recover-Faster: Detect issues early, rollback automatically
- Gradual Risk Exposure: Minimize blast radius through incremental rollouts
- Feature-Aware Deployments: Coordinate releases with feature flags
- Traffic Engineering: Fine-grained control over request routing
- Full Observability: Every decision traceable and auditable
Architecture
Component Overview
┌────────────────────────────────────────────────────────────────────────┐
│ Progressive Delivery System │
├────────────────────────────────────────────────────────────────────────┤
│ │
│ ┌──────────────────┐ ┌───────────────────┐ ┌─────────────────┐ │
│ │ RolloutController│───▶│ MetricsAnalyzer │───▶│ TrafficManager │ │
│ │ │ │ │ │ │ │
│ └──────────────────┘ └───────────────────┘ └─────────────────┘ │
│ │ │ │ │
│ ▼ ▼ ▼ │
│ ┌──────────────────┐ ┌───────────────────┐ ┌─────────────────┐ │
│ │ CanaryController │ │ FeatureFlagBridge │ │ LoadBalancer │ │
│ │ │ │ │ │ Integrations │ │
│ └──────────────────┘ └───────────────────┘ └─────────────────┘ │
│ │ │ │ │
│ ▼ ▼ ▼ │
│ ┌──────────────────┐ ┌───────────────────┐ ┌─────────────────┐ │
│ │ BlueGreenManager │ │ ExperimentEngine │ │ RollbackTrigger │ │
│ │ │ │ │ │ │ │
│ └──────────────────┘ └───────────────────┘ └─────────────────┘ │
│ │
└────────────────────────────────────────────────────────────────────────┘
Key Components
1. RolloutController
Orchestrates progressive rollout execution:
public sealed class RolloutController
{
public async Task<RolloutSession> StartRolloutAsync(
RolloutConfig config,
CancellationToken ct)
{
var session = new RolloutSession
{
Id = Guid.NewGuid(),
ReleaseId = config.ReleaseId,
EnvironmentId = config.EnvironmentId,
Strategy = config.Strategy,
StartedAt = _timeProvider.GetUtcNow(),
Status = RolloutStatus.Initializing
};
await _sessionStore.SaveAsync(session, ct);
// Initialize based on strategy
session = config.Strategy.Type switch
{
RolloutStrategyType.Canary => await InitializeCanaryAsync(session, config, ct),
RolloutStrategyType.BlueGreen => await InitializeBlueGreenAsync(session, config, ct),
RolloutStrategyType.Linear => await InitializeLinearAsync(session, config, ct),
RolloutStrategyType.Exponential => await InitializeExponentialAsync(session, config, ct),
_ => throw new UnsupportedStrategyException(config.Strategy.Type)
};
// Start the rollout loop
_ = RunRolloutLoopAsync(session, ct);
return session;
}
private async Task RunRolloutLoopAsync(
RolloutSession session,
CancellationToken ct)
{
try
{
while (!ct.IsCancellationRequested && !session.IsTerminal)
{
session = await _sessionStore.GetAsync(session.Id, ct);
// Check for manual pause
if (session.Status == RolloutStatus.Paused)
{
await Task.Delay(TimeSpan.FromSeconds(5), ct);
continue;
}
// Analyze current metrics
var analysis = await _metricsAnalyzer.AnalyzeAsync(session, ct);
// Make advancement decision
var decision = await DecideNextActionAsync(session, analysis, ct);
// Execute decision
session = await ExecuteDecisionAsync(session, decision, ct);
// Wait for observation period
if (decision.Action == RolloutAction.Advance)
{
await Task.Delay(session.CurrentStage.ObservationPeriod, ct);
}
}
}
catch (Exception ex)
{
_logger.LogError(ex, "Rollout loop failed for session {SessionId}", session.Id);
await FailRolloutAsync(session, ex.Message, ct);
}
}
private async Task<RolloutDecision> DecideNextActionAsync(
RolloutSession session,
MetricsAnalysis analysis,
CancellationToken ct)
{
var decision = new RolloutDecision
{
SessionId = session.Id,
DecidedAt = _timeProvider.GetUtcNow(),
Analysis = analysis
};
// Check for failures
if (analysis.HealthStatus == HealthStatus.Critical)
{
decision.Action = RolloutAction.Rollback;
decision.Reason = "Critical health degradation detected";
decision.TriggeringMetrics = analysis.CriticalMetrics;
return decision;
}
// Check if current stage requirements met
if (!IsStageRequirementsMet(session.CurrentStage, analysis))
{
if (analysis.StageDuration > session.CurrentStage.MaxDuration)
{
decision.Action = RolloutAction.Rollback;
decision.Reason = $"Stage {session.CurrentStage.Name} exceeded max duration";
}
else
{
decision.Action = RolloutAction.Wait;
decision.Reason = "Waiting for stage requirements";
}
return decision;
}
// Check if we're at final stage
if (session.IsAtFinalStage)
{
decision.Action = RolloutAction.Complete;
decision.Reason = "All stages completed successfully";
return decision;
}
// Ready to advance
decision.Action = RolloutAction.Advance;
decision.NextStage = session.GetNextStage();
decision.Reason = $"Stage {session.CurrentStage.Name} requirements met, advancing";
return decision;
}
}
public sealed record RolloutSession
{
public Guid Id { get; init; }
public Guid ReleaseId { get; init; }
public Guid EnvironmentId { get; init; }
public RolloutStrategy Strategy { get; init; }
public RolloutStatus Status { get; init; }
// Progress
public int CurrentStageIndex { get; init; }
public RolloutStage CurrentStage => Strategy.Stages[CurrentStageIndex];
public bool IsAtFinalStage => CurrentStageIndex >= Strategy.Stages.Length - 1;
public double CurrentTrafficPercent { get; init; }
// Timing
public DateTimeOffset StartedAt { get; init; }
public DateTimeOffset? CompletedAt { get; init; }
public DateTimeOffset StageStartedAt { get; init; }
// History
public ImmutableArray<RolloutDecision> DecisionHistory { get; init; }
// Terminal check
public bool IsTerminal => Status is RolloutStatus.Completed
or RolloutStatus.RolledBack or RolloutStatus.Failed;
}
2. MetricsAnalyzer
Analyzes metrics for rollout decisions:
public sealed class MetricsAnalyzer
{
private readonly ImmutableArray<IMetricsProvider> _providers;
public async Task<MetricsAnalysis> AnalyzeAsync(
RolloutSession session,
CancellationToken ct)
{
var analysis = new MetricsAnalysis
{
SessionId = session.Id,
AnalyzedAt = _timeProvider.GetUtcNow(),
StageDuration = _timeProvider.GetUtcNow() - session.StageStartedAt
};
// Collect metrics from all providers
var metrics = new Dictionary<string, MetricValue>();
foreach (var provider in _providers)
{
var providerMetrics = await provider.CollectAsync(session, ct);
foreach (var (name, value) in providerMetrics)
{
metrics[$"{provider.Name}:{name}"] = value;
}
}
// Get baseline for comparison
var baseline = await _baselineStore.GetAsync(session.EnvironmentId, ct);
// Analyze each metric against thresholds
foreach (var threshold in session.Strategy.SuccessThresholds)
{
var metricValue = metrics.GetValueOrDefault(threshold.MetricName);
if (metricValue == null)
{
analysis.MissingMetrics.Add(threshold.MetricName);
continue;
}
var evaluation = EvaluateMetric(metricValue, threshold, baseline);
analysis.MetricEvaluations.Add(evaluation);
if (evaluation.Status == MetricStatus.Critical)
{
analysis.CriticalMetrics.Add(evaluation);
}
}
// Calculate overall health
analysis.HealthStatus = CalculateOverallHealth(analysis.MetricEvaluations);
// Calculate recommended traffic percentage
analysis.RecommendedTrafficPercent = CalculateRecommendedTraffic(
session, analysis.MetricEvaluations);
return analysis;
}
private MetricEvaluation EvaluateMetric(
MetricValue value,
MetricThreshold threshold,
Baseline? baseline)
{
var evaluation = new MetricEvaluation
{
MetricName = threshold.MetricName,
CurrentValue = value.Value,
Threshold = threshold,
BaselineValue = baseline?.GetMetric(threshold.MetricName)
};
// Compare against threshold
var meetsThreshold = threshold.Comparison switch
{
ComparisonOperator.LessThan => value.Value < threshold.Value,
ComparisonOperator.LessThanOrEqual => value.Value <= threshold.Value,
ComparisonOperator.GreaterThan => value.Value > threshold.Value,
ComparisonOperator.GreaterThanOrEqual => value.Value >= threshold.Value,
ComparisonOperator.Equal => Math.Abs(value.Value - threshold.Value) < 0.001,
_ => false
};
// Compare against baseline if available
double? baselineDeviation = null;
if (evaluation.BaselineValue.HasValue)
{
baselineDeviation = (value.Value - evaluation.BaselineValue.Value)
/ Math.Max(evaluation.BaselineValue.Value, 0.001);
}
evaluation.MeetsThreshold = meetsThreshold;
evaluation.BaselineDeviation = baselineDeviation;
evaluation.Status = DetermineStatus(meetsThreshold, baselineDeviation, threshold);
return evaluation;
}
private double CalculateRecommendedTraffic(
RolloutSession session,
IReadOnlyList<MetricEvaluation> evaluations)
{
// All metrics healthy -> advance to next stage's target
if (evaluations.All(e => e.Status == MetricStatus.Healthy))
{
return session.IsAtFinalStage
? 100.0
: session.GetNextStage().TrafficPercent;
}
// Some degradation -> hold current or reduce
var worstStatus = evaluations.Max(e => e.Status);
return worstStatus switch
{
MetricStatus.Warning =>
session.CurrentTrafficPercent, // Hold
MetricStatus.Degraded =>
Math.Max(session.CurrentTrafficPercent * 0.5, 5), // Reduce 50%
MetricStatus.Critical =>
0, // Rollback
_ => session.CurrentTrafficPercent
};
}
}
public sealed record MetricsAnalysis
{
public Guid SessionId { get; init; }
public DateTimeOffset AnalyzedAt { get; init; }
public TimeSpan StageDuration { get; init; }
public HealthStatus HealthStatus { get; init; }
public double RecommendedTrafficPercent { get; init; }
public List<MetricEvaluation> MetricEvaluations { get; init; } = new();
public List<MetricEvaluation> CriticalMetrics { get; init; } = new();
public List<string> MissingMetrics { get; init; } = new();
}
3. CanaryController
Manages canary deployments with automated progression:
public sealed class CanaryController
{
public async Task<CanaryDeployment> CreateCanaryAsync(
CanaryConfig config,
CancellationToken ct)
{
var canary = new CanaryDeployment
{
Id = Guid.NewGuid(),
ReleaseId = config.ReleaseId,
EnvironmentId = config.EnvironmentId,
BaselineReleaseId = config.BaselineReleaseId,
Stages = config.Stages,
SuccessThresholds = config.SuccessThresholds,
CreatedAt = _timeProvider.GetUtcNow(),
Status = CanaryStatus.Initializing
};
// Deploy canary version
await DeployCanaryVersionAsync(canary, ct);
// Initialize traffic at first stage
canary.CurrentStageIndex = 0;
canary.CurrentTrafficPercent = canary.Stages[0].TrafficPercent;
await _trafficManager.SetCanaryTrafficAsync(canary, ct);
canary.Status = CanaryStatus.Running;
canary.StageStartedAt = _timeProvider.GetUtcNow();
await _canaryStore.SaveAsync(canary, ct);
return canary;
}
public async Task<CanaryAnalysis> AnalyzeCanaryAsync(
Guid canaryId,
CancellationToken ct)
{
var canary = await _canaryStore.GetAsync(canaryId, ct);
// Collect metrics for both versions
var canaryMetrics = await CollectVersionMetricsAsync(
canary.ReleaseId, canary.EnvironmentId, ct);
var baselineMetrics = await CollectVersionMetricsAsync(
canary.BaselineReleaseId, canary.EnvironmentId, ct);
var analysis = new CanaryAnalysis
{
CanaryId = canaryId,
AnalyzedAt = _timeProvider.GetUtcNow(),
CanaryMetrics = canaryMetrics,
BaselineMetrics = baselineMetrics
};
// Compare each threshold
foreach (var threshold in canary.SuccessThresholds)
{
var canaryValue = canaryMetrics.GetValueOrDefault(threshold.MetricName);
var baselineValue = baselineMetrics.GetValueOrDefault(threshold.MetricName);
if (canaryValue == null || baselineValue == null)
{
analysis.InsufficientData = true;
continue;
}
var comparison = new MetricComparison
{
MetricName = threshold.MetricName,
CanaryValue = canaryValue.Value,
BaselineValue = baselineValue.Value,
Threshold = threshold
};
// Calculate statistical significance
comparison.Difference = canaryValue.Value - baselineValue.Value;
comparison.DifferencePercent = comparison.Difference / Math.Max(baselineValue.Value, 0.001);
comparison.IsStatisticallySignificant = CalculateSignificance(
canaryValue, baselineValue, threshold.MinSampleSize);
// Determine if canary is better/worse/same
comparison.Verdict = DetermineVerdict(comparison, threshold);
analysis.Comparisons.Add(comparison);
}
// Overall verdict
analysis.OverallVerdict = DetermineOverallVerdict(analysis.Comparisons);
return analysis;
}
private CanaryVerdict DetermineVerdict(
MetricComparison comparison,
MetricThreshold threshold)
{
if (!comparison.IsStatisticallySignificant)
return CanaryVerdict.Inconclusive;
var isBetter = threshold.DesiredDirection switch
{
MetricDirection.Lower => comparison.Difference < 0,
MetricDirection.Higher => comparison.Difference > 0,
_ => false
};
if (isBetter)
return CanaryVerdict.Better;
// Check if within acceptable margin
var margin = Math.Abs(comparison.DifferencePercent);
if (margin <= threshold.AcceptableMargin)
return CanaryVerdict.Same;
return CanaryVerdict.Worse;
}
}
public sealed record CanaryDeployment
{
public Guid Id { get; init; }
public Guid ReleaseId { get; init; }
public Guid BaselineReleaseId { get; init; }
public Guid EnvironmentId { get; init; }
public CanaryStatus Status { get; init; }
// Configuration
public ImmutableArray<CanaryStage> Stages { get; init; }
public ImmutableArray<MetricThreshold> SuccessThresholds { get; init; }
// Progress
public int CurrentStageIndex { get; init; }
public double CurrentTrafficPercent { get; init; }
public DateTimeOffset StageStartedAt { get; init; }
// Analysis history
public ImmutableArray<CanaryAnalysis> AnalysisHistory { get; init; }
// Timing
public DateTimeOffset CreatedAt { get; init; }
public DateTimeOffset? CompletedAt { get; init; }
}
4. FeatureFlagBridge
Coordinates deployments with feature flags:
public sealed class FeatureFlagBridge
{
private readonly ImmutableArray<IFeatureFlagProvider> _providers;
public async Task<FeatureFlagSync> SyncFlagsForReleaseAsync(
Guid releaseId,
FeatureFlagSyncConfig config,
CancellationToken ct)
{
var release = await _releaseStore.GetAsync(releaseId, ct);
var flags = await GetAssociatedFlagsAsync(releaseId, ct);
var sync = new FeatureFlagSync
{
Id = Guid.NewGuid(),
ReleaseId = releaseId,
SyncedAt = _timeProvider.GetUtcNow()
};
foreach (var flag in flags)
{
var provider = _providers.First(p => p.Name == flag.Provider);
switch (config.Action)
{
case FlagSyncAction.EnableForTraffic:
// Enable flag for canary traffic percentage
await provider.SetRolloutPercentageAsync(
flag.Key, config.TrafficPercent, ct);
break;
case FlagSyncAction.EnableForUsers:
// Enable for specific user segment
await provider.EnableForSegmentAsync(
flag.Key, config.UserSegment, ct);
break;
case FlagSyncAction.EnableFully:
// Enable 100%
await provider.EnableAsync(flag.Key, ct);
break;
case FlagSyncAction.Disable:
// Disable flag (rollback scenario)
await provider.DisableAsync(flag.Key, ct);
break;
}
sync.FlagsUpdated.Add(new FlagUpdate
{
FlagKey = flag.Key,
Provider = flag.Provider,
Action = config.Action,
NewState = await provider.GetStateAsync(flag.Key, ct)
});
}
return sync;
}
public async Task<IReadOnlyList<FeatureFlag>> GetAssociatedFlagsAsync(
Guid releaseId,
CancellationToken ct)
{
var release = await _releaseStore.GetAsync(releaseId, ct);
// Get flags from release metadata
var flagKeys = release.Metadata.GetValueOrDefault("feature_flags", "")
.Split(',', StringSplitOptions.RemoveEmptyEntries);
var flags = new List<FeatureFlag>();
foreach (var key in flagKeys)
{
foreach (var provider in _providers)
{
var flag = await provider.GetFlagAsync(key, ct);
if (flag != null)
{
flags.Add(flag);
break;
}
}
}
return flags;
}
public async Task CoordinateRolloutWithFlagsAsync(
RolloutSession session,
CancellationToken ct)
{
var flags = await GetAssociatedFlagsAsync(session.ReleaseId, ct);
if (!flags.Any())
return;
// Sync flag rollout percentage with traffic percentage
await SyncFlagsForReleaseAsync(session.ReleaseId, new FeatureFlagSyncConfig
{
Action = FlagSyncAction.EnableForTraffic,
TrafficPercent = session.CurrentTrafficPercent
}, ct);
_logger.LogInformation(
"Synced {FlagCount} feature flags to {TrafficPercent}%",
flags.Count, session.CurrentTrafficPercent);
}
}
public interface IFeatureFlagProvider
{
string Name { get; }
Task<FeatureFlag?> GetFlagAsync(string key, CancellationToken ct);
Task EnableAsync(string key, CancellationToken ct);
Task DisableAsync(string key, CancellationToken ct);
Task SetRolloutPercentageAsync(string key, double percent, CancellationToken ct);
Task EnableForSegmentAsync(string key, string segment, CancellationToken ct);
Task<FlagState> GetStateAsync(string key, CancellationToken ct);
}
// Implementations for popular providers
public sealed class LaunchDarklyProvider : IFeatureFlagProvider { }
public sealed class SplitProvider : IFeatureFlagProvider { }
public sealed class UnleashProvider : IFeatureFlagProvider { }
public sealed class FlagsmithProvider : IFeatureFlagProvider { }
public sealed class ConfigCatProvider : IFeatureFlagProvider { }
5. TrafficManager
Manages traffic routing across load balancers:
public sealed class TrafficManager
{
private readonly ImmutableArray<ILoadBalancerAdapter> _adapters;
public async Task<TrafficConfiguration> SetTrafficSplitAsync(
TrafficSplitRequest request,
CancellationToken ct)
{
var targets = await _targetStore.GetByEnvironmentAsync(request.EnvironmentId, ct);
// Group targets by load balancer type
var targetsByLB = targets.GroupBy(t => t.LoadBalancerType);
var config = new TrafficConfiguration
{
Id = Guid.NewGuid(),
EnvironmentId = request.EnvironmentId,
AppliedAt = _timeProvider.GetUtcNow()
};
foreach (var group in targetsByLB)
{
var adapter = _adapters.FirstOrDefault(a => a.Type == group.Key);
if (adapter == null)
{
_logger.LogWarning("No adapter for load balancer type {Type}", group.Key);
continue;
}
foreach (var target in group)
{
await adapter.SetWeightsAsync(target, request.Weights, ct);
config.AppliedTargets.Add(target.Id);
}
}
await _configStore.SaveAsync(config, ct);
return config;
}
public async Task SetCanaryTrafficAsync(
CanaryDeployment canary,
CancellationToken ct)
{
var weights = new TrafficWeights
{
Weights = new Dictionary<string, double>
{
[canary.BaselineReleaseId.ToString()] = 100 - canary.CurrentTrafficPercent,
[canary.ReleaseId.ToString()] = canary.CurrentTrafficPercent
}.ToImmutableDictionary()
};
await SetTrafficSplitAsync(new TrafficSplitRequest
{
EnvironmentId = canary.EnvironmentId,
Weights = weights
}, ct);
}
public async Task<TrafficMetrics> GetTrafficMetricsAsync(
Guid environmentId,
CancellationToken ct)
{
var targets = await _targetStore.GetByEnvironmentAsync(environmentId, ct);
var metrics = new TrafficMetrics
{
EnvironmentId = environmentId,
CollectedAt = _timeProvider.GetUtcNow()
};
foreach (var target in targets)
{
var adapter = _adapters.FirstOrDefault(a => a.Type == target.LoadBalancerType);
if (adapter == null)
continue;
var targetMetrics = await adapter.GetMetricsAsync(target, ct);
metrics.TargetMetrics[target.Id] = targetMetrics;
}
return metrics;
}
}
public interface ILoadBalancerAdapter
{
LoadBalancerType Type { get; }
Task SetWeightsAsync(Target target, TrafficWeights weights, CancellationToken ct);
Task<TargetTrafficMetrics> GetMetricsAsync(Target target, CancellationToken ct);
Task<bool> HealthCheckAsync(Target target, CancellationToken ct);
}
// Adapters for various load balancers
public sealed class NginxAdapter : ILoadBalancerAdapter
{
public LoadBalancerType Type => LoadBalancerType.Nginx;
public async Task SetWeightsAsync(Target target, TrafficWeights weights, CancellationToken ct)
{
// Generate nginx upstream config with weights
var config = GenerateUpstreamConfig(target, weights);
// Write config and reload
await WriteConfigAsync(target, config, ct);
await ReloadNginxAsync(target, ct);
}
private string GenerateUpstreamConfig(Target target, TrafficWeights weights)
{
var sb = new StringBuilder();
sb.AppendLine($"upstream {target.Name} {{");
foreach (var (version, weight) in weights.Weights)
{
var servers = GetServersForVersion(target, version);
foreach (var server in servers)
{
sb.AppendLine($" server {server} weight={weight};");
}
}
sb.AppendLine("}");
return sb.ToString();
}
}
public sealed class HAProxyAdapter : ILoadBalancerAdapter { }
public sealed class TraefikAdapter : ILoadBalancerAdapter { }
public sealed class AWSALBAdapter : ILoadBalancerAdapter { }
public sealed class EnvoyAdapter : ILoadBalancerAdapter { }
6. ExperimentEngine
Manages A/B experiments:
public sealed class ExperimentEngine
{
public async Task<Experiment> CreateExperimentAsync(
ExperimentConfig config,
CancellationToken ct)
{
var experiment = new Experiment
{
Id = Guid.NewGuid(),
Name = config.Name,
EnvironmentId = config.EnvironmentId,
Hypothesis = config.Hypothesis,
Variants = config.Variants,
TrafficAllocation = config.TrafficAllocation,
SuccessMetrics = config.SuccessMetrics,
GuardrailMetrics = config.GuardrailMetrics,
MinSampleSize = config.MinSampleSize,
MaxDuration = config.MaxDuration,
CreatedAt = _timeProvider.GetUtcNow(),
Status = ExperimentStatus.Draft
};
await _experimentStore.SaveAsync(experiment, ct);
return experiment;
}
public async Task<Experiment> StartExperimentAsync(
Guid experimentId,
CancellationToken ct)
{
var experiment = await _experimentStore.GetAsync(experimentId, ct);
// Validate prerequisites
await ValidateExperimentAsync(experiment, ct);
// Deploy all variants
foreach (var variant in experiment.Variants)
{
await DeployVariantAsync(experiment, variant, ct);
}
// Set up traffic split
var weights = new TrafficWeights
{
Weights = experiment.Variants
.ToImmutableDictionary(
v => v.Id.ToString(),
v => experiment.TrafficAllocation.GetValueOrDefault(v.Id, 0))
};
await _trafficManager.SetTrafficSplitAsync(new TrafficSplitRequest
{
EnvironmentId = experiment.EnvironmentId,
Weights = weights
}, ct);
experiment = experiment with
{
Status = ExperimentStatus.Running,
StartedAt = _timeProvider.GetUtcNow()
};
await _experimentStore.SaveAsync(experiment, ct);
return experiment;
}
public async Task<ExperimentResults> AnalyzeExperimentAsync(
Guid experimentId,
CancellationToken ct)
{
var experiment = await _experimentStore.GetAsync(experimentId, ct);
var results = new ExperimentResults
{
ExperimentId = experimentId,
AnalyzedAt = _timeProvider.GetUtcNow()
};
// Collect metrics for each variant
foreach (var variant in experiment.Variants)
{
var variantMetrics = await CollectVariantMetricsAsync(experiment, variant, ct);
results.VariantMetrics[variant.Id] = variantMetrics;
}
// Statistical analysis
var control = experiment.Variants.First(v => v.IsControl);
foreach (var variant in experiment.Variants.Where(v => !v.IsControl))
{
var analysis = PerformStatisticalAnalysis(
results.VariantMetrics[control.Id],
results.VariantMetrics[variant.Id],
experiment.SuccessMetrics);
results.VariantAnalyses[variant.Id] = analysis;
}
// Check guardrail metrics
results.GuardrailViolations = await CheckGuardrailsAsync(experiment, results, ct);
// Determine winner
results.Winner = DetermineWinner(experiment, results);
results.Confidence = CalculateOverallConfidence(results);
results.Recommendation = GenerateRecommendation(experiment, results);
return results;
}
private VariantAnalysis PerformStatisticalAnalysis(
VariantMetrics control,
VariantMetrics treatment,
ImmutableArray<MetricDefinition> successMetrics)
{
var analysis = new VariantAnalysis();
foreach (var metric in successMetrics)
{
var controlValues = control.GetMetricValues(metric.Name);
var treatmentValues = treatment.GetMetricValues(metric.Name);
// Calculate effect size
var effectSize = (treatmentValues.Mean - controlValues.Mean) / controlValues.Mean;
// Perform t-test
var tTest = PerformTTest(controlValues, treatmentValues);
// Calculate confidence interval
var ci = CalculateConfidenceInterval(controlValues, treatmentValues, 0.95);
analysis.MetricResults[metric.Name] = new MetricAnalysisResult
{
ControlMean = controlValues.Mean,
TreatmentMean = treatmentValues.Mean,
EffectSize = effectSize,
PValue = tTest.PValue,
IsSignificant = tTest.PValue < 0.05,
ConfidenceInterval = ci,
SampleSize = controlValues.Count + treatmentValues.Count
};
}
return analysis;
}
}
public sealed record Experiment
{
public Guid Id { get; init; }
public string Name { get; init; }
public Guid EnvironmentId { get; init; }
public string Hypothesis { get; init; }
public ExperimentStatus Status { get; init; }
// Variants
public ImmutableArray<ExperimentVariant> Variants { get; init; }
public ImmutableDictionary<Guid, double> TrafficAllocation { get; init; }
// Metrics
public ImmutableArray<MetricDefinition> SuccessMetrics { get; init; }
public ImmutableArray<MetricDefinition> GuardrailMetrics { get; init; }
// Configuration
public int MinSampleSize { get; init; }
public TimeSpan MaxDuration { get; init; }
// Timing
public DateTimeOffset CreatedAt { get; init; }
public DateTimeOffset? StartedAt { get; init; }
public DateTimeOffset? EndedAt { get; init; }
}
Rollout Strategies
Canary Strategy
strategy:
type: canary
stages:
- name: "canary-5"
traffic_percent: 5
duration: "00:15:00"
observation_period: "00:05:00"
- name: "canary-25"
traffic_percent: 25
duration: "00:30:00"
observation_period: "00:10:00"
- name: "canary-50"
traffic_percent: 50
duration: "01:00:00"
observation_period: "00:15:00"
- name: "full-rollout"
traffic_percent: 100
duration: "00:00:00"
success_thresholds:
- metric: error_rate
comparison: less_than
value: 0.01
desired_direction: lower
- metric: latency_p99
comparison: less_than
value: 1000
desired_direction: lower
auto_rollback:
enabled: true
on_metric_failure: true
on_analysis_failure: true
Linear Strategy
strategy:
type: linear
increment_percent: 10
increment_interval: "00:10:00"
max_traffic_percent: 100
success_thresholds:
- metric: success_rate
comparison: greater_than
value: 0.99
Exponential Strategy
strategy:
type: exponential
initial_percent: 1
multiplier: 2.0
max_traffic_percent: 100
stage_duration: "00:10:00"
# Results in: 1% → 2% → 4% → 8% → 16% → 32% → 64% → 100%
Blue-Green Strategy
strategy:
type: blue_green
stages:
- name: "deploy-green"
action: deploy_standby
- name: "smoke-test"
action: run_tests
test_suite: smoke
- name: "switch-traffic"
action: switch_traffic
switch_mode: instant # or 'gradual'
- name: "verify"
action: verify
duration: "00:30:00"
- name: "cleanup"
action: terminate_blue
API Design
REST Endpoints
# Rollouts
POST /api/v1/rollouts # Start rollout
GET /api/v1/rollouts # List rollouts
GET /api/v1/rollouts/{id} # Get rollout
POST /api/v1/rollouts/{id}/pause # Pause rollout
POST /api/v1/rollouts/{id}/resume # Resume rollout
POST /api/v1/rollouts/{id}/advance # Manual advance
POST /api/v1/rollouts/{id}/rollback # Manual rollback
POST /api/v1/rollouts/{id}/complete # Force complete
# Canary
POST /api/v1/canary # Start canary
GET /api/v1/canary/{id} # Get canary
GET /api/v1/canary/{id}/analysis # Get analysis
# Experiments
POST /api/v1/experiments # Create experiment
GET /api/v1/experiments # List experiments
GET /api/v1/experiments/{id} # Get experiment
POST /api/v1/experiments/{id}/start # Start experiment
POST /api/v1/experiments/{id}/stop # Stop experiment
GET /api/v1/experiments/{id}/results # Get results
# Traffic
GET /api/v1/traffic/{environmentId} # Get traffic config
POST /api/v1/traffic/{environmentId} # Set traffic split
GET /api/v1/traffic/{environmentId}/metrics # Get traffic metrics
# Feature Flags
GET /api/v1/releases/{id}/flags # Get release flags
POST /api/v1/releases/{id}/flags/sync # Sync flags
Metrics & Observability
Prometheus Metrics
# Rollout Progress
stella_rollout_traffic_percent{session_id, stage}
stella_rollout_stage_duration_seconds{session_id, stage}
stella_rollout_decisions_total{session_id, action}
# Canary Analysis
stella_canary_health_score{canary_id}
stella_canary_metric_comparison{canary_id, metric, verdict}
stella_canary_sample_size{canary_id, variant}
# Experiments
stella_experiment_variant_traffic{experiment_id, variant_id}
stella_experiment_metric_value{experiment_id, variant_id, metric}
stella_experiment_statistical_significance{experiment_id, variant_id}
# Traffic
stella_traffic_split_percent{environment_id, version}
stella_traffic_requests_total{environment_id, version}
stella_traffic_errors_total{environment_id, version}
Test Strategy
Unit Tests
- Metric threshold evaluation
- Statistical significance calculation
- Traffic weight calculation
- Strategy stage progression
Integration Tests
- Full canary lifecycle
- Experiment creation and analysis
- Traffic manager with mock LB
- Feature flag synchronization
Chaos Tests
- Metrics provider failures
- Load balancer unavailability
- Rapid traffic shifts
Golden Tests
- Deterministic analysis results
- Consistent winner selection
- Reproducible rollout decisions
Migration Path
Phase 1: Metrics Integration (Week 1-2)
- Metrics analyzer
- Baseline management
- Provider adapters
Phase 2: Rollout Controller (Week 3-4)
- Session management
- Stage progression
- Decision engine
Phase 3: Canary (Week 5-6)
- Canary controller
- Statistical analysis
- Auto-progression
Phase 4: Traffic Management (Week 7-8)
- Load balancer adapters
- Weight synchronization
- Health monitoring
Phase 5: Feature Flags (Week 9-10)
- Provider integrations
- Rollout coordination
- Flag lifecycle
Phase 6: Experiments (Week 11-12)
- Experiment engine
- Statistical analysis
- Results visualization