bc4318ef97
- Created `StellaOps.TestKit.Tests` project for unit tests related to determinism. - Implemented `DeterminismManifestTests` to validate deterministic output for canonical bytes and strings, file read/write operations, and error handling for invalid schema versions. - Added `SbomDeterminismTests` to ensure identical inputs produce consistent SBOMs across SPDX 3.0.1 and CycloneDX 1.6/1.7 formats, including parallel execution tests. - Updated project references in `StellaOps.Integration.Determinism` to include the new determinism testing library.
1.2 KiB
1.2 KiB
Orchestrator runbook
Pre-flight
- Verify database and queue backends are healthy.
- Confirm tenant allowlist and orchestrator scopes in Authority.
- Ensure plugin bundles are present and signatures verified.
Common operations
- Start a run via API or CLI.
- Cancel runs with idempotent requests.
- Stream status via WebSocket or CLI.
- Export run ledger as NDJSON for audit.
Incident response
- Queue backlog: scale workers and drain oldest first.
- Repeated failures: inspect error codes and inputsHash; roll back DAG version.
- Plugin auth errors: rotate secrets and warm caches.
Health checks
- /admin/health for liveness and queue depth.
- Metrics: orchestrator_runs_total, orchestrator_queue_depth, orchestrator_step_retries_total, orchestrator_run_duration_seconds.
- Logs include tenant, dagId, runId, status with redaction.
Determinism and immutability
- Runs are append-only; never mutate ledger entries.
- Use runToken for idempotent retries.
Offline posture
- Keep DAG specs and plugins in sealed storage.
- Export logs, metrics, and traces as NDJSON.
Related references
- orchestrator/overview.md
- orchestrator/architecture.md
- docs/operations/orchestrator-runbook.md