save checkpoint

docs/features/checked/orchestrator/quota-governance-and-circuit-breakers.md

@@ -0,0 +1,43 @@
+# Quota Governance and Circuit Breakers
+
+## Module
+Orchestrator
+
+## Status
+VERIFIED
+
+## Description
+Quota governance services with cross-tenant allocation policies and circuit breaker automation for downstream service failure protection, integrated with rate limiting and load shedding.
+
+## Implementation Details
+- **Modules**: `src/Orchestrator/StellaOps.Orchestrator/StellaOps.Orchestrator.Core/Domain/`, `src/Orchestrator/StellaOps.Orchestrator/StellaOps.Orchestrator.Core/RateLimiting/`, `src/Orchestrator/StellaOps.Orchestrator/StellaOps.Orchestrator.Core/Scale/`, `src/Orchestrator/StellaOps.Orchestrator/StellaOps.Orchestrator.Infrastructure/`
+- **Key Classes**:
+  - `QuotaGovernanceService` (`src/Orchestrator/StellaOps.Orchestrator/StellaOps.Orchestrator.Infrastructure/Services/QuotaGovernanceService.cs`) - cross-tenant quota allocation with 5 strategies (unlimited, proportional, priority, reserved, max-limit)
+  - `CircuitBreakerService` (`src/Orchestrator/StellaOps.Orchestrator/StellaOps.Orchestrator.Infrastructure/Services/CircuitBreakerService.cs`) - circuit breaker with Closed/Open/HalfOpen state transitions
+  - `Quota` (`src/Orchestrator/StellaOps.Orchestrator/StellaOps.Orchestrator.Core/Domain/Quota.cs`) - quota entity with limits and allocation
+  - `QuotaEndpoints` (`src/Orchestrator/StellaOps.Orchestrator/StellaOps.Orchestrator.WebService/Endpoints/QuotaEndpoints.cs`) - REST API for quota queries and adjustments
+  - `QuotaContracts` (`src/Orchestrator/StellaOps.Orchestrator/StellaOps.Orchestrator.WebService/Contracts/QuotaContracts.cs`) - API contracts for quota operations
+  - `Throttle` (`src/Orchestrator/StellaOps.Orchestrator/StellaOps.Orchestrator.Core/Domain/Throttle.cs`) - throttle configuration for rate limiting
+  - `AdaptiveRateLimiter` (`src/Orchestrator/StellaOps.Orchestrator/StellaOps.Orchestrator.Core/RateLimiting/AdaptiveRateLimiter.cs`) - adaptive rate limiting based on system load
+  - `ConcurrencyLimiter` (`src/Orchestrator/StellaOps.Orchestrator/StellaOps.Orchestrator.Core/RateLimiting/ConcurrencyLimiter.cs`) - limits concurrent job execution
+  - `BackpressureHandler` (`src/Orchestrator/StellaOps.Orchestrator/StellaOps.Orchestrator.Core/RateLimiting/BackpressureHandler.cs`) - backpressure signaling
+  - `LoadShedder` (`src/Orchestrator/StellaOps.Orchestrator/StellaOps.Orchestrator.Core/Scale/LoadShedder.cs`) - load shedding under saturation
+  - `PostgresQuotaRepository` (`src/Orchestrator/StellaOps.Orchestrator/StellaOps.Orchestrator.Infrastructure/Postgres/PostgresQuotaRepository.cs`) - Postgres-backed quota storage
+  - `PostgresThrottleRepository` (`src/Orchestrator/StellaOps.Orchestrator/StellaOps.Orchestrator.Infrastructure/Postgres/PostgresThrottleRepository.cs`) - Postgres-backed throttle storage
+- **Source**: Feature matrix scan
+
+## E2E Test Plan
+- [ ] Configure a quota policy with proportional allocation and verify QuotaGovernanceService distributes capacity across tenants
+- [ ] Request quota above max limit and verify the request is capped
+- [ ] Pause a tenant and verify quota requests are denied
+- [ ] Trigger circuit breaker by exceeding failure threshold and verify downstream requests are blocked
+- [ ] Verify circuit breaker recovery: wait for timeout, verify HalfOpen state, send success to close
+- [ ] Force-open and force-close the circuit breaker and verify state changes
+- [ ] Test concurrent access to circuit breaker and verify thread safety
+- [ ] Verify all 5 allocation strategies produce correct quota distributions
+
+## Verification
+- Verified on 2026-02-13 via `run-002`.
+- Tier 0: Source files confirmed present on disk.
+- Tier 1: `dotnet build` passed (0 errors); 1292/1292 tests passed.
+- Tier 2d: `docs/qa/feature-checks/runs/orchestrator/quota-governance-and-circuit-breakers/run-002/tier2-integration-check.json`