stella-ops.org/git.stella-ops.org
1
0
Fork 0
Code Issues Pull Requests Actions 2 Releases Wiki Activity
Files
ab794e167c2759162758096872c5941d308e57b8
git.stella-ops.org/docs/features/checked/orchestrator/skip-locked-queue-pattern.md
master e9aeadc040 save checkpoint
2026-02-14 09:11:48 +02:00

3.7 KiB
Raw Blame History

SKIP LOCKED Queue Pattern

Module

Orchestrator

Status

VERIFIED

Description

SKIP LOCKED queue pattern is used in Scheduler and Orchestrator job repositories for reliable work distribution.

Implementation Details

  • Modules: src/Orchestrator/StellaOps.Orchestrator/StellaOps.Orchestrator.Core/Scheduling/, src/Orchestrator/StellaOps.Orchestrator/StellaOps.Orchestrator.Core/RateLimiting/, src/Orchestrator/StellaOps.Orchestrator/StellaOps.Orchestrator.Core/Scale/
  • Key Classes:
    • JobScheduler (src/Orchestrator/StellaOps.Orchestrator/StellaOps.Orchestrator.Core/Scheduling/JobScheduler.cs) - job scheduler using PostgreSQL SELECT ... FOR UPDATE SKIP LOCKED for concurrent job dequeuing without contention
    • Job (src/Orchestrator/StellaOps.Orchestrator/StellaOps.Orchestrator.Core/Domain/Job.cs) - job entity with status field used for queue filtering
    • JobStatus (src/Orchestrator/StellaOps.Orchestrator/StellaOps.Orchestrator.Core/Domain/JobStatus.cs) - job states used in queue queries (Pending jobs are available for dequeuing)
    • Watermark (src/Orchestrator/StellaOps.Orchestrator/StellaOps.Orchestrator.Core/Domain/Watermark.cs) - watermark tracking for ordered processing
    • AdaptiveRateLimiter (src/Orchestrator/StellaOps.Orchestrator/StellaOps.Orchestrator.Core/RateLimiting/AdaptiveRateLimiter.cs) - rate limiter that adjusts based on queue depth and processing speed
    • ConcurrencyLimiter (src/Orchestrator/StellaOps.Orchestrator/StellaOps.Orchestrator.Core/RateLimiting/ConcurrencyLimiter.cs) - limits concurrent job processing
    • TokenBucket (src/Orchestrator/StellaOps.Orchestrator/StellaOps.Orchestrator.Core/RateLimiting/TokenBucket.cs) - token bucket rate limiter for smooth job distribution
    • BackpressureHandler (src/Orchestrator/StellaOps.Orchestrator/StellaOps.Orchestrator.Core/RateLimiting/BackpressureHandler.cs) - applies backpressure when queue depth exceeds thresholds
    • LoadShedder (src/Orchestrator/StellaOps.Orchestrator/StellaOps.Orchestrator.Core/Scale/LoadShedder.cs) - sheds load when system is saturated
    • ScaleMetrics (src/Orchestrator/StellaOps.Orchestrator/StellaOps.Orchestrator.Core/Scale/ScaleMetrics.cs) - metrics for monitoring queue depth and throughput
  • Interfaces: IJobRepository (src/Orchestrator/StellaOps.Orchestrator/StellaOps.Orchestrator.Infrastructure/Repositories/IJobRepository.cs), IWatermarkRepository (src/Orchestrator/StellaOps.Orchestrator/StellaOps.Orchestrator.Infrastructure/Repositories/IWatermarkRepository.cs)
  • Source: Feature matrix scan

E2E Test Plan

  • Enqueue 10 jobs and dequeue from 3 concurrent workers using SKIP LOCKED via JobScheduler; verify each job is assigned to exactly one worker
  • Verify no contention: dequeue rapidly from 5 workers and verify no blocking or deadlocks occur
  • Verify job visibility: a job locked by worker A is not visible to worker B during dequeue
  • Complete a locked job and verify it is no longer in the queue
  • Verify AdaptiveRateLimiter: increase queue depth and verify the rate limiter increases throughput
  • Verify BackpressureHandler: fill the queue beyond the threshold and verify backpressure is signaled to producers
  • Verify LoadShedder: saturate the system and verify new jobs are rejected with a 503 response
  • Test TokenBucket: configure a rate of 10 jobs/second and verify the bucket enforces the limit

Verification

  • Verified on 2026-02-13 via run-002.
  • Tier 0: Source files confirmed present on disk.
  • Tier 1: dotnet build passed (0 errors); 1292/1292 tests passed.
  • Tier 2d: docs/qa/feature-checks/runs/orchestrator/skip-locked-queue-pattern/run-002/tier2-integration-check.json