stella-ops.org/git.stella-ops.org
1
0
Fork 0
Code Issues Pull Requests Actions 3 Releases Wiki Activity
Files
7bafcc3eef9ae15d9441641a2ba3ef7ca4e67907
git.stella-ops.org/docs/features/checked/orchestrator/quota-governance-and-circuit-breakers.md
master e9aeadc040 save checkpoint
2026-02-14 09:11:48 +02:00

3.7 KiB
Raw Blame History

Quota Governance and Circuit Breakers

Module

Orchestrator

Status

VERIFIED

Description

Quota governance services with cross-tenant allocation policies and circuit breaker automation for downstream service failure protection, integrated with rate limiting and load shedding.

Implementation Details

  • Modules: src/Orchestrator/StellaOps.Orchestrator/StellaOps.Orchestrator.Core/Domain/, src/Orchestrator/StellaOps.Orchestrator/StellaOps.Orchestrator.Core/RateLimiting/, src/Orchestrator/StellaOps.Orchestrator/StellaOps.Orchestrator.Core/Scale/, src/Orchestrator/StellaOps.Orchestrator/StellaOps.Orchestrator.Infrastructure/
  • Key Classes:
    • QuotaGovernanceService (src/Orchestrator/StellaOps.Orchestrator/StellaOps.Orchestrator.Infrastructure/Services/QuotaGovernanceService.cs) - cross-tenant quota allocation with 5 strategies (unlimited, proportional, priority, reserved, max-limit)
    • CircuitBreakerService (src/Orchestrator/StellaOps.Orchestrator/StellaOps.Orchestrator.Infrastructure/Services/CircuitBreakerService.cs) - circuit breaker with Closed/Open/HalfOpen state transitions
    • Quota (src/Orchestrator/StellaOps.Orchestrator/StellaOps.Orchestrator.Core/Domain/Quota.cs) - quota entity with limits and allocation
    • QuotaEndpoints (src/Orchestrator/StellaOps.Orchestrator/StellaOps.Orchestrator.WebService/Endpoints/QuotaEndpoints.cs) - REST API for quota queries and adjustments
    • QuotaContracts (src/Orchestrator/StellaOps.Orchestrator/StellaOps.Orchestrator.WebService/Contracts/QuotaContracts.cs) - API contracts for quota operations
    • Throttle (src/Orchestrator/StellaOps.Orchestrator/StellaOps.Orchestrator.Core/Domain/Throttle.cs) - throttle configuration for rate limiting
    • AdaptiveRateLimiter (src/Orchestrator/StellaOps.Orchestrator/StellaOps.Orchestrator.Core/RateLimiting/AdaptiveRateLimiter.cs) - adaptive rate limiting based on system load
    • ConcurrencyLimiter (src/Orchestrator/StellaOps.Orchestrator/StellaOps.Orchestrator.Core/RateLimiting/ConcurrencyLimiter.cs) - limits concurrent job execution
    • BackpressureHandler (src/Orchestrator/StellaOps.Orchestrator/StellaOps.Orchestrator.Core/RateLimiting/BackpressureHandler.cs) - backpressure signaling
    • LoadShedder (src/Orchestrator/StellaOps.Orchestrator/StellaOps.Orchestrator.Core/Scale/LoadShedder.cs) - load shedding under saturation
    • PostgresQuotaRepository (src/Orchestrator/StellaOps.Orchestrator/StellaOps.Orchestrator.Infrastructure/Postgres/PostgresQuotaRepository.cs) - Postgres-backed quota storage
    • PostgresThrottleRepository (src/Orchestrator/StellaOps.Orchestrator/StellaOps.Orchestrator.Infrastructure/Postgres/PostgresThrottleRepository.cs) - Postgres-backed throttle storage
  • Source: Feature matrix scan

E2E Test Plan

  • Configure a quota policy with proportional allocation and verify QuotaGovernanceService distributes capacity across tenants
  • Request quota above max limit and verify the request is capped
  • Pause a tenant and verify quota requests are denied
  • Trigger circuit breaker by exceeding failure threshold and verify downstream requests are blocked
  • Verify circuit breaker recovery: wait for timeout, verify HalfOpen state, send success to close
  • Force-open and force-close the circuit breaker and verify state changes
  • Test concurrent access to circuit breaker and verify thread safety
  • Verify all 5 allocation strategies produce correct quota distributions

Verification

  • Verified on 2026-02-13 via run-002.
  • Tier 0: Source files confirmed present on disk.
  • Tier 1: dotnet build passed (0 errors); 1292/1292 tests passed.
  • Tier 2d: docs/qa/feature-checks/runs/orchestrator/quota-governance-and-circuit-breakers/run-002/tier2-integration-check.json