# Quota Governance and Circuit Breakers

## Module
Orchestrator

## Status
VERIFIED

## Description
Quota governance services with cross-tenant allocation policies and circuit breaker automation for downstream service failure protection, integrated with rate limiting and load shedding.

## Implementation Details
- **Modules**: `src/JobEngine/StellaOps.JobEngine/StellaOps.JobEngine.Core/Domain/`, `src/JobEngine/StellaOps.JobEngine/StellaOps.JobEngine.Core/RateLimiting/`, `src/JobEngine/StellaOps.JobEngine/StellaOps.JobEngine.Core/Scale/`, `src/JobEngine/StellaOps.JobEngine/StellaOps.JobEngine.Infrastructure/`
- **Key Classes**:
  - `QuotaGovernanceService` (`src/JobEngine/StellaOps.JobEngine/StellaOps.JobEngine.Infrastructure/Services/QuotaGovernanceService.cs`) - cross-tenant quota allocation with 5 strategies (unlimited, proportional, priority, reserved, max-limit)
  - `CircuitBreakerService` (`src/JobEngine/StellaOps.JobEngine/StellaOps.JobEngine.Infrastructure/Services/CircuitBreakerService.cs`) - circuit breaker with Closed/Open/HalfOpen state transitions
  - `Quota` (`src/JobEngine/StellaOps.JobEngine/StellaOps.JobEngine.Core/Domain/Quota.cs`) - quota entity with limits and allocation
  - `QuotaEndpoints` (`src/JobEngine/StellaOps.JobEngine/StellaOps.JobEngine.WebService/Endpoints/QuotaEndpoints.cs`) - REST API for quota queries and adjustments
  - `QuotaContracts` (`src/JobEngine/StellaOps.JobEngine/StellaOps.JobEngine.WebService/Contracts/QuotaContracts.cs`) - API contracts for quota operations
  - `Throttle` (`src/JobEngine/StellaOps.JobEngine/StellaOps.JobEngine.Core/Domain/Throttle.cs`) - throttle configuration for rate limiting
  - `AdaptiveRateLimiter` (`src/JobEngine/StellaOps.JobEngine/StellaOps.JobEngine.Core/RateLimiting/AdaptiveRateLimiter.cs`) - adaptive rate limiting based on system load
  - `ConcurrencyLimiter` (`src/JobEngine/StellaOps.JobEngine/StellaOps.JobEngine.Core/RateLimiting/ConcurrencyLimiter.cs`) - limits concurrent job execution
  - `BackpressureHandler` (`src/JobEngine/StellaOps.JobEngine/StellaOps.JobEngine.Core/RateLimiting/BackpressureHandler.cs`) - backpressure signaling
  - `LoadShedder` (`src/JobEngine/StellaOps.JobEngine/StellaOps.JobEngine.Core/Scale/LoadShedder.cs`) - load shedding under saturation
  - `PostgresQuotaRepository` (`src/JobEngine/StellaOps.JobEngine/StellaOps.JobEngine.Infrastructure/Postgres/PostgresQuotaRepository.cs`) - Postgres-backed quota storage
  - `PostgresThrottleRepository` (`src/JobEngine/StellaOps.JobEngine/StellaOps.JobEngine.Infrastructure/Postgres/PostgresThrottleRepository.cs`) - Postgres-backed throttle storage
- **Source**: Feature matrix scan

## E2E Test Plan
- [ ] Configure a quota policy with proportional allocation and verify QuotaGovernanceService distributes capacity across tenants
- [ ] Request quota above max limit and verify the request is capped
- [ ] Pause a tenant and verify quota requests are denied
- [ ] Trigger circuit breaker by exceeding failure threshold and verify downstream requests are blocked
- [ ] Verify circuit breaker recovery: wait for timeout, verify HalfOpen state, send success to close
- [ ] Force-open and force-close the circuit breaker and verify state changes
- [ ] Test concurrent access to circuit breaker and verify thread safety
- [ ] Verify all 5 allocation strategies produce correct quota distributions

## Verification
- Verified on 2026-02-13 via `run-002`.
- Tier 0: Source files confirmed present on disk.
- Tier 1: `dotnet build` passed (0 errors); 1292/1292 tests passed.
- Tier 2d: `docs/qa/feature-checks/runs/jobengine/quota-governance-and-circuit-breakers/run-002/tier2-integration-check.json`