bc4318ef97
- Created `StellaOps.TestKit.Tests` project for unit tests related to determinism. - Implemented `DeterminismManifestTests` to validate deterministic output for canonical bytes and strings, file read/write operations, and error handling for invalid schema versions. - Added `SbomDeterminismTests` to ensure identical inputs produce consistent SBOMs across SPDX 3.0.1 and CycloneDX 1.6/1.7 formats, including parallel execution tests. - Updated project references in `StellaOps.Integration.Determinism` to include the new determinism testing library.
37 lines
1.2 KiB
Markdown
37 lines
1.2 KiB
Markdown
# Orchestrator runbook
|
|
|
|
Pre-flight
|
|
- Verify database and queue backends are healthy.
|
|
- Confirm tenant allowlist and orchestrator scopes in Authority.
|
|
- Ensure plugin bundles are present and signatures verified.
|
|
|
|
Common operations
|
|
- Start a run via API or CLI.
|
|
- Cancel runs with idempotent requests.
|
|
- Stream status via WebSocket or CLI.
|
|
- Export run ledger as NDJSON for audit.
|
|
|
|
Incident response
|
|
- Queue backlog: scale workers and drain oldest first.
|
|
- Repeated failures: inspect error codes and inputsHash; roll back DAG version.
|
|
- Plugin auth errors: rotate secrets and warm caches.
|
|
|
|
Health checks
|
|
- /admin/health for liveness and queue depth.
|
|
- Metrics: orchestrator_runs_total, orchestrator_queue_depth,
|
|
orchestrator_step_retries_total, orchestrator_run_duration_seconds.
|
|
- Logs include tenant, dagId, runId, status with redaction.
|
|
|
|
Determinism and immutability
|
|
- Runs are append-only; never mutate ledger entries.
|
|
- Use runToken for idempotent retries.
|
|
|
|
Offline posture
|
|
- Keep DAG specs and plugins in sealed storage.
|
|
- Export logs, metrics, and traces as NDJSON.
|
|
|
|
Related references
|
|
- orchestrator/overview.md
|
|
- orchestrator/architecture.md
|
|
- docs/operations/orchestrator-runbook.md
|