44 lines
3.6 KiB
Markdown
44 lines
3.6 KiB
Markdown
# Quota Governance and Circuit Breakers
|
|
|
|
## Module
|
|
Orchestrator
|
|
|
|
## Status
|
|
VERIFIED
|
|
|
|
## Description
|
|
Quota governance services with cross-tenant allocation policies and circuit breaker automation for downstream service failure protection, integrated with rate limiting and load shedding.
|
|
|
|
## Implementation Details
|
|
- **Modules**: `src/JobEngine/StellaOps.JobEngine/StellaOps.JobEngine.Core/Domain/`, `src/JobEngine/StellaOps.JobEngine/StellaOps.JobEngine.Core/RateLimiting/`, `src/JobEngine/StellaOps.JobEngine/StellaOps.JobEngine.Core/Scale/`, `src/JobEngine/StellaOps.JobEngine/StellaOps.JobEngine.Infrastructure/`
|
|
- **Key Classes**:
|
|
- `QuotaGovernanceService` (`src/JobEngine/StellaOps.JobEngine/StellaOps.JobEngine.Infrastructure/Services/QuotaGovernanceService.cs`) - cross-tenant quota allocation with 5 strategies (unlimited, proportional, priority, reserved, max-limit)
|
|
- `CircuitBreakerService` (`src/JobEngine/StellaOps.JobEngine/StellaOps.JobEngine.Infrastructure/Services/CircuitBreakerService.cs`) - circuit breaker with Closed/Open/HalfOpen state transitions
|
|
- `Quota` (`src/JobEngine/StellaOps.JobEngine/StellaOps.JobEngine.Core/Domain/Quota.cs`) - quota entity with limits and allocation
|
|
- `QuotaEndpoints` (`src/JobEngine/StellaOps.JobEngine/StellaOps.JobEngine.WebService/Endpoints/QuotaEndpoints.cs`) - REST API for quota queries and adjustments
|
|
- `QuotaContracts` (`src/JobEngine/StellaOps.JobEngine/StellaOps.JobEngine.WebService/Contracts/QuotaContracts.cs`) - API contracts for quota operations
|
|
- `Throttle` (`src/JobEngine/StellaOps.JobEngine/StellaOps.JobEngine.Core/Domain/Throttle.cs`) - throttle configuration for rate limiting
|
|
- `AdaptiveRateLimiter` (`src/JobEngine/StellaOps.JobEngine/StellaOps.JobEngine.Core/RateLimiting/AdaptiveRateLimiter.cs`) - adaptive rate limiting based on system load
|
|
- `ConcurrencyLimiter` (`src/JobEngine/StellaOps.JobEngine/StellaOps.JobEngine.Core/RateLimiting/ConcurrencyLimiter.cs`) - limits concurrent job execution
|
|
- `BackpressureHandler` (`src/JobEngine/StellaOps.JobEngine/StellaOps.JobEngine.Core/RateLimiting/BackpressureHandler.cs`) - backpressure signaling
|
|
- `LoadShedder` (`src/JobEngine/StellaOps.JobEngine/StellaOps.JobEngine.Core/Scale/LoadShedder.cs`) - load shedding under saturation
|
|
- `PostgresQuotaRepository` (`src/JobEngine/StellaOps.JobEngine/StellaOps.JobEngine.Infrastructure/Postgres/PostgresQuotaRepository.cs`) - Postgres-backed quota storage
|
|
- `PostgresThrottleRepository` (`src/JobEngine/StellaOps.JobEngine/StellaOps.JobEngine.Infrastructure/Postgres/PostgresThrottleRepository.cs`) - Postgres-backed throttle storage
|
|
- **Source**: Feature matrix scan
|
|
|
|
## E2E Test Plan
|
|
- [ ] Configure a quota policy with proportional allocation and verify QuotaGovernanceService distributes capacity across tenants
|
|
- [ ] Request quota above max limit and verify the request is capped
|
|
- [ ] Pause a tenant and verify quota requests are denied
|
|
- [ ] Trigger circuit breaker by exceeding failure threshold and verify downstream requests are blocked
|
|
- [ ] Verify circuit breaker recovery: wait for timeout, verify HalfOpen state, send success to close
|
|
- [ ] Force-open and force-close the circuit breaker and verify state changes
|
|
- [ ] Test concurrent access to circuit breaker and verify thread safety
|
|
- [ ] Verify all 5 allocation strategies produce correct quota distributions
|
|
|
|
## Verification
|
|
- Verified on 2026-02-13 via `run-002`.
|
|
- Tier 0: Source files confirmed present on disk.
|
|
- Tier 1: `dotnet build` passed (0 errors); 1292/1292 tests passed.
|
|
- Tier 2d: `docs/qa/feature-checks/runs/jobengine/quota-governance-and-circuit-breakers/run-002/tier2-integration-check.json`
|