git.stella-ops.org/docs/orchestrator/architecture.md

Orchestrator Architecture (DOCS-ORCH-32-002)

Last updated: 2025-11-25

Runtime components

• WebService: REST + WebSocket API for DAG definitions, run status, and admin actions; issues idempotency tokens and enforces tenant isolation.
• Scheduler: timer/cron runner that instantiates DAG runs from schedules; publishes run intents into per-tenant queues.
• Worker: executes DAG steps; pulls from tenant queues, applies resource limits, and reports spans/metrics/logs.
• Plugin host: task plugins (HTTP call, queue dispatch, CLI tool, script) loaded from signed bundles; execution is sandboxed with deny-by-default network.

Data model

• DAG: directed acyclic graph with topological order; tie-break lexicographically by step id for determinism.
• Run: immutable record with runId, dagVersion, tenant, inputsHash, status, traceId, startedUtc, endedUtc.
• Step execution: each step captures inputsHash, outputsHash, status, attempt, durationMs, logsRef, metricsRef.

Execution flow

1. Client or scheduler creates a run (idempotent on runToken, dagId, inputsHash).
2. Scheduler enqueues run intent into tenant queue.
3. Worker dequeues, reconstructs DAG ordering, and executes steps:
   • skip disabled steps;
   • apply per-step concurrency, retries, and backoff;
   • emit spans/metrics/logs with propagated traceparent.
4. Results are persisted append-only; WebSocket pushes status to clients.

Storage & queues

• Mongo stores DAG specs, versions, and run history (per-tenant collections or tenant key prefix).
• Queues: Redis/Mongo-backed FIFO per tenant; message includes traceparent, runToken, dagVersion, inputsHash.
• Artifacts (logs, outputs) referenced by content hash; stored in object storage or Mongo GridFS; hashes recorded in run record.

Security & AOC alignment

• Mandatory X-Stella-Tenant; cross-tenant DAGs prohibited.
• Scopes: orchestrator:read|write|admin; admin needed for DAG publish/delete.
• AOC: Orchestrator only schedules/executes; no policy/severity decisions. Inputs/outputs immutable; runs replayable.
• Sandboxing: per-step CPU/memory limits; network egress blocked unless step declares allowlist entry.

Determinism

• Step ordering: topological + lexical tie-breaks.
• Idempotency: runToken + inputsHash; retries reuse same traceId; outputs hashed (lowercase hex).
• Timestamps UTC; NDJSON exports sorted by (startedUtc, dagId, runId).

Offline posture

• DAG specs and plugins shipped in signed offline bundles; no remote fetch.
• Transparency: export runs/logs/metrics/traces as NDJSON for air-gapped audit.

Observability

• Traces: spans named orchestrator.run, orchestrator.step with attributes tenant, dagId, runId, stepId, status.
• Metrics: orchestrator_runs_total{tenant,status}, orchestrator_run_duration_seconds, orchestrator_queue_depth, orchestrator_step_retries_total.
• Logs: structured JSON, redacted, carrying trace_id, tenant, dagId, runId, stepId.

Governance & rollout

• DAG publishing requires signature/owner metadata; versions immutable after publish.
• Rollback: schedule new version and disable old; runs stay immutable.
• Upgrade path: workers hot-reload plugins from bundle catalog; scheduler is stateless.