# SKIP LOCKED Queue Pattern

## Module
Orchestrator

## Status
VERIFIED

## Description
SKIP LOCKED queue pattern is used in Scheduler and Orchestrator job repositories for reliable work distribution.

## Implementation Details
- **Modules**: `src/JobEngine/StellaOps.JobEngine/StellaOps.JobEngine.Core/Scheduling/`, `src/JobEngine/StellaOps.JobEngine/StellaOps.JobEngine.Core/RateLimiting/`, `src/JobEngine/StellaOps.JobEngine/StellaOps.JobEngine.Core/Scale/`
- **Key Classes**:
  - `JobScheduler` (`src/JobEngine/StellaOps.JobEngine/StellaOps.JobEngine.Core/Scheduling/JobScheduler.cs`) - job scheduler using PostgreSQL `SELECT ... FOR UPDATE SKIP LOCKED` for concurrent job dequeuing without contention
  - `Job` (`src/JobEngine/StellaOps.JobEngine/StellaOps.JobEngine.Core/Domain/Job.cs`) - job entity with status field used for queue filtering
  - `JobStatus` (`src/JobEngine/StellaOps.JobEngine/StellaOps.JobEngine.Core/Domain/JobStatus.cs`) - job states used in queue queries (Pending jobs are available for dequeuing)
  - `Watermark` (`src/JobEngine/StellaOps.JobEngine/StellaOps.JobEngine.Core/Domain/Watermark.cs`) - watermark tracking for ordered processing
  - `AdaptiveRateLimiter` (`src/JobEngine/StellaOps.JobEngine/StellaOps.JobEngine.Core/RateLimiting/AdaptiveRateLimiter.cs`) - rate limiter that adjusts based on queue depth and processing speed
  - `ConcurrencyLimiter` (`src/JobEngine/StellaOps.JobEngine/StellaOps.JobEngine.Core/RateLimiting/ConcurrencyLimiter.cs`) - limits concurrent job processing
  - `TokenBucket` (`src/JobEngine/StellaOps.JobEngine/StellaOps.JobEngine.Core/RateLimiting/TokenBucket.cs`) - token bucket rate limiter for smooth job distribution
  - `BackpressureHandler` (`src/JobEngine/StellaOps.JobEngine/StellaOps.JobEngine.Core/RateLimiting/BackpressureHandler.cs`) - applies backpressure when queue depth exceeds thresholds
  - `LoadShedder` (`src/JobEngine/StellaOps.JobEngine/StellaOps.JobEngine.Core/Scale/LoadShedder.cs`) - sheds load when system is saturated
  - `ScaleMetrics` (`src/JobEngine/StellaOps.JobEngine/StellaOps.JobEngine.Core/Scale/ScaleMetrics.cs`) - metrics for monitoring queue depth and throughput
- **Interfaces**: `IJobRepository` (`src/JobEngine/StellaOps.JobEngine/StellaOps.JobEngine.Infrastructure/Repositories/IJobRepository.cs`), `IWatermarkRepository` (`src/JobEngine/StellaOps.JobEngine/StellaOps.JobEngine.Infrastructure/Repositories/IWatermarkRepository.cs`)
- **Source**: Feature matrix scan

## E2E Test Plan
- [ ] Enqueue 10 jobs and dequeue from 3 concurrent workers using SKIP LOCKED via `JobScheduler`; verify each job is assigned to exactly one worker
- [ ] Verify no contention: dequeue rapidly from 5 workers and verify no blocking or deadlocks occur
- [ ] Verify job visibility: a job locked by worker A is not visible to worker B during dequeue
- [ ] Complete a locked job and verify it is no longer in the queue
- [ ] Verify `AdaptiveRateLimiter`: increase queue depth and verify the rate limiter increases throughput
- [ ] Verify `BackpressureHandler`: fill the queue beyond the threshold and verify backpressure is signaled to producers
- [ ] Verify `LoadShedder`: saturate the system and verify new jobs are rejected with a 503 response
- [ ] Test `TokenBucket`: configure a rate of 10 jobs/second and verify the bucket enforces the limit

## Verification
- Verified on 2026-02-13 via `run-002`.
- Tier 0: Source files confirmed present on disk.
- Tier 1: `dotnet build` passed (0 errors); 1292/1292 tests passed.
- Tier 2d: `docs/qa/feature-checks/runs/jobengine/skip-locked-queue-pattern/run-002/tier2-integration-check.json`