stella-ops.org/git.stella-ops.org
1
0
Fork 0
Code Issues Pull Requests Actions 2 Releases Wiki Activity
Files
8e1cb9448d752df07141efe450cf4b9b67fe55d1
git.stella-ops.org/docs/features/checked/jobengine/quota-governance-and-circuit-breakers.md

3.6 KiB
Raw Blame History

Quota Governance and Circuit Breakers

Module

Orchestrator

Status

VERIFIED

Description

Quota governance services with cross-tenant allocation policies and circuit breaker automation for downstream service failure protection, integrated with rate limiting and load shedding.

Implementation Details

  • Modules: src/JobEngine/StellaOps.JobEngine/StellaOps.JobEngine.Core/Domain/, src/JobEngine/StellaOps.JobEngine/StellaOps.JobEngine.Core/RateLimiting/, src/JobEngine/StellaOps.JobEngine/StellaOps.JobEngine.Core/Scale/, src/JobEngine/StellaOps.JobEngine/StellaOps.JobEngine.Infrastructure/
  • Key Classes:
    • QuotaGovernanceService (src/JobEngine/StellaOps.JobEngine/StellaOps.JobEngine.Infrastructure/Services/QuotaGovernanceService.cs) - cross-tenant quota allocation with 5 strategies (unlimited, proportional, priority, reserved, max-limit)
    • CircuitBreakerService (src/JobEngine/StellaOps.JobEngine/StellaOps.JobEngine.Infrastructure/Services/CircuitBreakerService.cs) - circuit breaker with Closed/Open/HalfOpen state transitions
    • Quota (src/JobEngine/StellaOps.JobEngine/StellaOps.JobEngine.Core/Domain/Quota.cs) - quota entity with limits and allocation
    • QuotaEndpoints (src/JobEngine/StellaOps.JobEngine/StellaOps.JobEngine.WebService/Endpoints/QuotaEndpoints.cs) - REST API for quota queries and adjustments
    • QuotaContracts (src/JobEngine/StellaOps.JobEngine/StellaOps.JobEngine.WebService/Contracts/QuotaContracts.cs) - API contracts for quota operations
    • Throttle (src/JobEngine/StellaOps.JobEngine/StellaOps.JobEngine.Core/Domain/Throttle.cs) - throttle configuration for rate limiting
    • AdaptiveRateLimiter (src/JobEngine/StellaOps.JobEngine/StellaOps.JobEngine.Core/RateLimiting/AdaptiveRateLimiter.cs) - adaptive rate limiting based on system load
    • ConcurrencyLimiter (src/JobEngine/StellaOps.JobEngine/StellaOps.JobEngine.Core/RateLimiting/ConcurrencyLimiter.cs) - limits concurrent job execution
    • BackpressureHandler (src/JobEngine/StellaOps.JobEngine/StellaOps.JobEngine.Core/RateLimiting/BackpressureHandler.cs) - backpressure signaling
    • LoadShedder (src/JobEngine/StellaOps.JobEngine/StellaOps.JobEngine.Core/Scale/LoadShedder.cs) - load shedding under saturation
    • PostgresQuotaRepository (src/JobEngine/StellaOps.JobEngine/StellaOps.JobEngine.Infrastructure/Postgres/PostgresQuotaRepository.cs) - Postgres-backed quota storage
    • PostgresThrottleRepository (src/JobEngine/StellaOps.JobEngine/StellaOps.JobEngine.Infrastructure/Postgres/PostgresThrottleRepository.cs) - Postgres-backed throttle storage
  • Source: Feature matrix scan

E2E Test Plan

  • Configure a quota policy with proportional allocation and verify QuotaGovernanceService distributes capacity across tenants
  • Request quota above max limit and verify the request is capped
  • Pause a tenant and verify quota requests are denied
  • Trigger circuit breaker by exceeding failure threshold and verify downstream requests are blocked
  • Verify circuit breaker recovery: wait for timeout, verify HalfOpen state, send success to close
  • Force-open and force-close the circuit breaker and verify state changes
  • Test concurrent access to circuit breaker and verify thread safety
  • Verify all 5 allocation strategies produce correct quota distributions

Verification

  • Verified on 2026-02-13 via run-002.
  • Tier 0: Source files confirmed present on disk.
  • Tier 1: dotnet build passed (0 errors); 1292/1292 tests passed.
  • Tier 2d: docs/qa/feature-checks/runs/jobengine/quota-governance-and-circuit-breakers/run-002/tier2-integration-check.json