consolidation of some of the modules, localization fixes, product advisories work, qa work

This commit is contained in:
master
2026-03-05 03:54:22 +02:00
parent 7bafcc3eef
commit 8e1cb9448d
3878 changed files with 72600 additions and 46861 deletions

View File

@@ -0,0 +1,42 @@
# SKIP LOCKED Queue Pattern
## Module
Orchestrator
## Status
VERIFIED
## Description
SKIP LOCKED queue pattern is used in Scheduler and Orchestrator job repositories for reliable work distribution.
## Implementation Details
- **Modules**: `src/JobEngine/StellaOps.JobEngine/StellaOps.JobEngine.Core/Scheduling/`, `src/JobEngine/StellaOps.JobEngine/StellaOps.JobEngine.Core/RateLimiting/`, `src/JobEngine/StellaOps.JobEngine/StellaOps.JobEngine.Core/Scale/`
- **Key Classes**:
- `JobScheduler` (`src/JobEngine/StellaOps.JobEngine/StellaOps.JobEngine.Core/Scheduling/JobScheduler.cs`) - job scheduler using PostgreSQL `SELECT ... FOR UPDATE SKIP LOCKED` for concurrent job dequeuing without contention
- `Job` (`src/JobEngine/StellaOps.JobEngine/StellaOps.JobEngine.Core/Domain/Job.cs`) - job entity with status field used for queue filtering
- `JobStatus` (`src/JobEngine/StellaOps.JobEngine/StellaOps.JobEngine.Core/Domain/JobStatus.cs`) - job states used in queue queries (Pending jobs are available for dequeuing)
- `Watermark` (`src/JobEngine/StellaOps.JobEngine/StellaOps.JobEngine.Core/Domain/Watermark.cs`) - watermark tracking for ordered processing
- `AdaptiveRateLimiter` (`src/JobEngine/StellaOps.JobEngine/StellaOps.JobEngine.Core/RateLimiting/AdaptiveRateLimiter.cs`) - rate limiter that adjusts based on queue depth and processing speed
- `ConcurrencyLimiter` (`src/JobEngine/StellaOps.JobEngine/StellaOps.JobEngine.Core/RateLimiting/ConcurrencyLimiter.cs`) - limits concurrent job processing
- `TokenBucket` (`src/JobEngine/StellaOps.JobEngine/StellaOps.JobEngine.Core/RateLimiting/TokenBucket.cs`) - token bucket rate limiter for smooth job distribution
- `BackpressureHandler` (`src/JobEngine/StellaOps.JobEngine/StellaOps.JobEngine.Core/RateLimiting/BackpressureHandler.cs`) - applies backpressure when queue depth exceeds thresholds
- `LoadShedder` (`src/JobEngine/StellaOps.JobEngine/StellaOps.JobEngine.Core/Scale/LoadShedder.cs`) - sheds load when system is saturated
- `ScaleMetrics` (`src/JobEngine/StellaOps.JobEngine/StellaOps.JobEngine.Core/Scale/ScaleMetrics.cs`) - metrics for monitoring queue depth and throughput
- **Interfaces**: `IJobRepository` (`src/JobEngine/StellaOps.JobEngine/StellaOps.JobEngine.Infrastructure/Repositories/IJobRepository.cs`), `IWatermarkRepository` (`src/JobEngine/StellaOps.JobEngine/StellaOps.JobEngine.Infrastructure/Repositories/IWatermarkRepository.cs`)
- **Source**: Feature matrix scan
## E2E Test Plan
- [ ] Enqueue 10 jobs and dequeue from 3 concurrent workers using SKIP LOCKED via `JobScheduler`; verify each job is assigned to exactly one worker
- [ ] Verify no contention: dequeue rapidly from 5 workers and verify no blocking or deadlocks occur
- [ ] Verify job visibility: a job locked by worker A is not visible to worker B during dequeue
- [ ] Complete a locked job and verify it is no longer in the queue
- [ ] Verify `AdaptiveRateLimiter`: increase queue depth and verify the rate limiter increases throughput
- [ ] Verify `BackpressureHandler`: fill the queue beyond the threshold and verify backpressure is signaled to producers
- [ ] Verify `LoadShedder`: saturate the system and verify new jobs are rejected with a 503 response
- [ ] Test `TokenBucket`: configure a rate of 10 jobs/second and verify the bucket enforces the limit
## Verification
- Verified on 2026-02-13 via `run-002`.
- Tier 0: Source files confirmed present on disk.
- Tier 1: `dotnet build` passed (0 errors); 1292/1292 tests passed.
- Tier 2d: `docs/qa/feature-checks/runs/jobengine/skip-locked-queue-pattern/run-002/tier2-integration-check.json`