Implementation plan — Source & Job Orchestrator

Delivery phases

Phase 1 – Core service & job ledger
Implement source registry, run/job tables, queue abstraction, lease management, token-bucket rate limiting, watchdogs, and API primitives (/sources, /runs, /jobs).
Phase 2 – Worker SDK & artifact registry
Embed worker SDK in Conseiller, Excititor, SBOM, Policy Engine; capture artifact metadata + hashes, enforce idempotency, publish progress/metrics.
Phase 3 – Observability & dashboard
Ship metrics, traces, incident logging, SSE/WebSocket feeds, and Console dashboard (DAG/timeline, heatmaps, error clustering, SLO burn rate).
Phase 4 – Controls & resilience
Deliver pause/resume/throttle/retry/backfill tooling, dead-letter review, circuit breakers, blackouts, backpressure handling, and automation hooks.
Phase 5 – Offline & compliance
Generate deterministic audit bundles (jobs.jsonl, history.jsonl, throttles.jsonl), provenance manifests, and offline replay scripts.

Service & persistence
- Postgres schema (sources, runs, jobs, artifacts, dag_edges, quotas, schedules, incidents).
- Lease manager with heartbeats, retries, and dead-letter queues.
- Token-bucket rate limiter per {tenant, source.host}; adaptive refill on upstream throttles.
- Watermark/backfill orchestration for event-time windows.
Worker SDK
- Claim/heartbeat/report contract, deterministic artifact hashing, idempotency enforcement.
- Library release for .NET workers plus language bindings for Rust/Go ingestion agents.
Control plane APIs
- CRUD for sources, runs, jobs, quotas, schedules; control actions (retry, cancel, prioritize, pause/resume, backfill).
- SSE/WebSocket stream for Console updates.
Observability
- Metrics: queue depth, job latency, failure classes, rate-limit hits, burn rate.
- Error clustering (HTTP 429/5xx, schema mismatch, parse errors), incident logging with reason codes.
- Gantt timeline and DAG JSON for Console visualisation.
Console & CLI
- Console app sections: overview, sources, runs, job detail, incidents, throttles.
- CLI commands: stella orchestrator sources|runs|jobs|throttle|backfill.
Compliance & offline
- Immutable audit bundles with signatures; exports via Export Center; Offline Kit instructions.
- Tenant isolation validation and secret redaction for logs/UI.

Orchestrator schedules all advisory/VEX/SBOM/policy jobs with quotas, rate limits, and idempotency; retries and replay preserve provenance.
Console dashboard reflects real-time DAG status, queue depth, SLO burn rate, and allows pause/resume/throttle/backfill with audit trail.
Worker SDK integrated across producer services, emitting progress and artifact metadata.
Observability stack exposes metrics, logs, traces, incidents, and alerts for stuck jobs, throttling, and failure spikes.
Offline audit bundles reproduce job history deterministically and verify signatures.

Backpressure/queue overload: adaptive token buckets, circuit breakers, dynamic concurrency; degrade gracefully.
Upstream vendor throttles: throttle management with user-visible state, automatic jitter and retry.
Tenant leakage: enforce tenant filters at API/queue/storage, fuzz tests, redaction.
Complex DAG errors: built-in diagnostics, error clustering, partial replay tooling.
Operator error: confirmation prompts, RBAC, runbook guidance, reason codes logged.

Unit: scheduling, quota enforcement, lease renewals, token bucket, watermark arithmetic.
Integration: worker SDK with Conseiller/Excititor/SBOM pipelines, pause/resume/backfill flows, failure recovery.
Performance: high-volume job workloads, queue backpressure, concurrency caps, dashboard SSE load tests.
Chaos: simulate upstream outages, stuck workers, clock skew, Postgres failover.
Compliance: audit bundle generation, signature verification, offline replay.

All phases delivered with telemetry, dashboards, and runbooks published.
Console + CLI parity validated; Offline Kit instructions complete.
./TASKS.md and ../../TASKS.md updated with status; documentation (README/architecture/this plan) reflects latest behaviour.