semi implemented and features implemented save checkpoint

docs/features/unchecked/orchestrator/skip-locked-queue-pattern.md

@@ -0,0 +1,36 @@
+# SKIP LOCKED Queue Pattern
+
+## Module
+Orchestrator
+
+## Status
+IMPLEMENTED
+
+## Description
+SKIP LOCKED queue pattern is used in Scheduler and Orchestrator job repositories for reliable work distribution.
+
+## Implementation Details
+- **Modules**: `src/Orchestrator/StellaOps.Orchestrator/StellaOps.Orchestrator.Core/Scheduling/`, `src/Orchestrator/StellaOps.Orchestrator/StellaOps.Orchestrator.Core/RateLimiting/`, `src/Orchestrator/StellaOps.Orchestrator/StellaOps.Orchestrator.Core/Scale/`
+- **Key Classes**:
+  - `JobScheduler` (`src/Orchestrator/StellaOps.Orchestrator/StellaOps.Orchestrator.Core/Scheduling/JobScheduler.cs`) - job scheduler using PostgreSQL `SELECT ... FOR UPDATE SKIP LOCKED` for concurrent job dequeuing without contention
+  - `Job` (`src/Orchestrator/StellaOps.Orchestrator/StellaOps.Orchestrator.Core/Domain/Job.cs`) - job entity with status field used for queue filtering
+  - `JobStatus` (`src/Orchestrator/StellaOps.Orchestrator/StellaOps.Orchestrator.Core/Domain/JobStatus.cs`) - job states used in queue queries (Pending jobs are available for dequeuing)
+  - `Watermark` (`src/Orchestrator/StellaOps.Orchestrator/StellaOps.Orchestrator.Core/Domain/Watermark.cs`) - watermark tracking for ordered processing
+  - `AdaptiveRateLimiter` (`src/Orchestrator/StellaOps.Orchestrator/StellaOps.Orchestrator.Core/RateLimiting/AdaptiveRateLimiter.cs`) - rate limiter that adjusts based on queue depth and processing speed
+  - `ConcurrencyLimiter` (`src/Orchestrator/StellaOps.Orchestrator/StellaOps.Orchestrator.Core/RateLimiting/ConcurrencyLimiter.cs`) - limits concurrent job processing
+  - `TokenBucket` (`src/Orchestrator/StellaOps.Orchestrator/StellaOps.Orchestrator.Core/RateLimiting/TokenBucket.cs`) - token bucket rate limiter for smooth job distribution
+  - `BackpressureHandler` (`src/Orchestrator/StellaOps.Orchestrator/StellaOps.Orchestrator.Core/RateLimiting/BackpressureHandler.cs`) - applies backpressure when queue depth exceeds thresholds
+  - `LoadShedder` (`src/Orchestrator/StellaOps.Orchestrator/StellaOps.Orchestrator.Core/Scale/LoadShedder.cs`) - sheds load when system is saturated
+  - `ScaleMetrics` (`src/Orchestrator/StellaOps.Orchestrator/StellaOps.Orchestrator.Core/Scale/ScaleMetrics.cs`) - metrics for monitoring queue depth and throughput
+- **Interfaces**: `IJobRepository` (`src/Orchestrator/StellaOps.Orchestrator/StellaOps.Orchestrator.Infrastructure/Repositories/IJobRepository.cs`), `IWatermarkRepository` (`src/Orchestrator/StellaOps.Orchestrator/StellaOps.Orchestrator.Infrastructure/Repositories/IWatermarkRepository.cs`)
+- **Source**: Feature matrix scan
+
+## E2E Test Plan
+- [ ] Enqueue 10 jobs and dequeue from 3 concurrent workers using SKIP LOCKED via `JobScheduler`; verify each job is assigned to exactly one worker
+- [ ] Verify no contention: dequeue rapidly from 5 workers and verify no blocking or deadlocks occur
+- [ ] Verify job visibility: a job locked by worker A is not visible to worker B during dequeue
+- [ ] Complete a locked job and verify it is no longer in the queue
+- [ ] Verify `AdaptiveRateLimiter`: increase queue depth and verify the rate limiter increases throughput
+- [ ] Verify `BackpressureHandler`: fill the queue beyond the threshold and verify backpressure is signaled to producers
+- [ ] Verify `LoadShedder`: saturate the system and verify new jobs are rejected with a 503 response
+- [ ] Test `TokenBucket`: configure a rate of 10 jobs/second and verify the bucket enforces the limit