43 lines
3.5 KiB
Markdown
43 lines
3.5 KiB
Markdown
# SKIP LOCKED Queue Pattern
|
|
|
|
## Module
|
|
Orchestrator
|
|
|
|
## Status
|
|
VERIFIED
|
|
|
|
## Description
|
|
SKIP LOCKED queue pattern is used in Scheduler and Orchestrator job repositories for reliable work distribution.
|
|
|
|
## Implementation Details
|
|
- **Modules**: `src/JobEngine/StellaOps.JobEngine/StellaOps.JobEngine.Core/Scheduling/`, `src/JobEngine/StellaOps.JobEngine/StellaOps.JobEngine.Core/RateLimiting/`, `src/JobEngine/StellaOps.JobEngine/StellaOps.JobEngine.Core/Scale/`
|
|
- **Key Classes**:
|
|
- `JobScheduler` (`src/JobEngine/StellaOps.JobEngine/StellaOps.JobEngine.Core/Scheduling/JobScheduler.cs`) - job scheduler using PostgreSQL `SELECT ... FOR UPDATE SKIP LOCKED` for concurrent job dequeuing without contention
|
|
- `Job` (`src/JobEngine/StellaOps.JobEngine/StellaOps.JobEngine.Core/Domain/Job.cs`) - job entity with status field used for queue filtering
|
|
- `JobStatus` (`src/JobEngine/StellaOps.JobEngine/StellaOps.JobEngine.Core/Domain/JobStatus.cs`) - job states used in queue queries (Pending jobs are available for dequeuing)
|
|
- `Watermark` (`src/JobEngine/StellaOps.JobEngine/StellaOps.JobEngine.Core/Domain/Watermark.cs`) - watermark tracking for ordered processing
|
|
- `AdaptiveRateLimiter` (`src/JobEngine/StellaOps.JobEngine/StellaOps.JobEngine.Core/RateLimiting/AdaptiveRateLimiter.cs`) - rate limiter that adjusts based on queue depth and processing speed
|
|
- `ConcurrencyLimiter` (`src/JobEngine/StellaOps.JobEngine/StellaOps.JobEngine.Core/RateLimiting/ConcurrencyLimiter.cs`) - limits concurrent job processing
|
|
- `TokenBucket` (`src/JobEngine/StellaOps.JobEngine/StellaOps.JobEngine.Core/RateLimiting/TokenBucket.cs`) - token bucket rate limiter for smooth job distribution
|
|
- `BackpressureHandler` (`src/JobEngine/StellaOps.JobEngine/StellaOps.JobEngine.Core/RateLimiting/BackpressureHandler.cs`) - applies backpressure when queue depth exceeds thresholds
|
|
- `LoadShedder` (`src/JobEngine/StellaOps.JobEngine/StellaOps.JobEngine.Core/Scale/LoadShedder.cs`) - sheds load when system is saturated
|
|
- `ScaleMetrics` (`src/JobEngine/StellaOps.JobEngine/StellaOps.JobEngine.Core/Scale/ScaleMetrics.cs`) - metrics for monitoring queue depth and throughput
|
|
- **Interfaces**: `IJobRepository` (`src/JobEngine/StellaOps.JobEngine/StellaOps.JobEngine.Infrastructure/Repositories/IJobRepository.cs`), `IWatermarkRepository` (`src/JobEngine/StellaOps.JobEngine/StellaOps.JobEngine.Infrastructure/Repositories/IWatermarkRepository.cs`)
|
|
- **Source**: Feature matrix scan
|
|
|
|
## E2E Test Plan
|
|
- [ ] Enqueue 10 jobs and dequeue from 3 concurrent workers using SKIP LOCKED via `JobScheduler`; verify each job is assigned to exactly one worker
|
|
- [ ] Verify no contention: dequeue rapidly from 5 workers and verify no blocking or deadlocks occur
|
|
- [ ] Verify job visibility: a job locked by worker A is not visible to worker B during dequeue
|
|
- [ ] Complete a locked job and verify it is no longer in the queue
|
|
- [ ] Verify `AdaptiveRateLimiter`: increase queue depth and verify the rate limiter increases throughput
|
|
- [ ] Verify `BackpressureHandler`: fill the queue beyond the threshold and verify backpressure is signaled to producers
|
|
- [ ] Verify `LoadShedder`: saturate the system and verify new jobs are rejected with a 503 response
|
|
- [ ] Test `TokenBucket`: configure a rate of 10 jobs/second and verify the bucket enforces the limit
|
|
|
|
## Verification
|
|
- Verified on 2026-02-13 via `run-002`.
|
|
- Tier 0: Source files confirmed present on disk.
|
|
- Tier 1: `dotnet build` passed (0 errors); 1292/1292 tests passed.
|
|
- Tier 2d: `docs/qa/feature-checks/runs/jobengine/skip-locked-queue-pattern/run-002/tier2-integration-check.json`
|