git.stella-ops.org/docs/operations/orchestrator-runbook.md

Orchestrator Runbook (DOCS-ORCH-34-003)

Last updated: 2025-11-25

Pre-flight

• Ensure Mongo and queue backend reachable; health at /api/v1/orchestrator/admin/health green.
• Verify tenant allowlist and scopes (orchestrator:*) configured in Authority.
• Plugin bundles present and signatures verified.

Common operations

• Start a run: POST /api/v1/orchestrator/runs or stella orch run start ....
• Cancel a run: POST /runs/{runId}:cancel; best-effort, idempotent.
• Stream status: WebSocket /runs/stream or CLI stella orch run stream.
• Export ledger: NDJSON export by time window for audits.

Incident response

• Queue backlog: Check queue depth; scale workers or pause schedulers; drain oldest first. Verify no stuck plugin.
• Repeated failures: Inspect run ledger for error.code; compare inputsHash and plugin version; roll back DAG version if regression.
• Plugin auth errors: rotate secretRef in Authority; warm worker cache; re-run impacted DAGs.
• Scheduler runaway: disable offending DAG version; clear scheduled triggers; confirm queue drains.

Health checks

• GET /admin/health — liveness/readiness + queue depth.
• Metrics: orchestrator_runs_total, orchestrator_queue_depth, orchestrator_step_retries_total, orchestrator_run_duration_seconds.
• Logs: structured JSON with tenant, dagId, runId, status; check for redaction markers.

Determinism/immutability

• Runs are append-only; do not mutate ledger entries. Use new DAG versions for fixes.
• Idempotency via runToken; reruns should reuse the same token when repeating intended work.

Offline/air-gap

• Keep plugin bundles and DAG specs in sealed storage; no remote fetch.
• Export logs/metrics/traces as NDJSON for offline analysis; include manifest/hash.

Quick checks

• Health green, queue depth normal.
• Latest plugin bundle signatures valid.
• No secrets in logs (spot-check redaction).
• Error budget within SLO (see docs/observability/metrics-and-slos.md).