feat: Implement Filesystem and MongoDB provenance writers for PackRun execution context
- Added `FilesystemPackRunProvenanceWriter` to write provenance manifests to the filesystem. - Introduced `MongoPackRunArtifactReader` to read artifacts from MongoDB. - Created `MongoPackRunProvenanceWriter` to store provenance manifests in MongoDB. - Developed unit tests for filesystem and MongoDB provenance writers. - Established `ITimelineEventStore` and `ITimelineIngestionService` interfaces for timeline event handling. - Implemented `TimelineIngestionService` to validate and persist timeline events with hashing. - Created PostgreSQL schema and migration scripts for timeline indexing. - Added dependency injection support for timeline indexer services. - Developed tests for timeline ingestion and schema validation.
This commit is contained in:
StellaOps Bot
87
docs/modules/taskrunner/architecture.md
Normal file
87
docs/modules/taskrunner/architecture.md
Normal file
@@ -0,0 +1,87 @@
|
||||
# TaskRunner Architecture (v1)
|
||||
|
||||
> Canonical contract for TaskRunner delivery scoped by SPRINT_0157_0001_0002 (TaskRunner Blockers) and SPRINT_0157_0001_0001 (TaskRunner I). Anchored in product advisory **"29-Nov-2025 - Task Pack Orchestration and Automation"** and the Task Pack runbook/spec (`docs/task-packs/*.md`).
|
||||
|
||||
## 1. Purpose and Scope
|
||||
- Execute Task Packs deterministically with approvals, sealed-mode enforcement, and evidence capture.
|
||||
- Provide API/CLI surface for pack submission, status, logs, approvals, artifacts, and cancellation.
|
||||
- Produce provenance: DSSE attestation + evidence bundle for every completed run.
|
||||
- Operate offline/air-gapped with plan-hash binding and sealed-mode network allowlists.
|
||||
|
||||
## 2. Components
|
||||
- **WebService** (`StellaOps.TaskRunner.WebService`) - HTTP API, plan hash validation, SSE log streaming, approval endpoints.
|
||||
- **Worker** (`StellaOps.TaskRunner.Worker`) - run orchestration, retries/backoff, artifact capture, attestation generation.
|
||||
- **Core** (`StellaOps.TaskRunner.Core`) - execution graph builder, simulation engine, step state machine, policy/approval gate abstractions.
|
||||
- **Infrastructure** (`StellaOps.TaskRunner.Infrastructure`) - storage adapters (Mongo, file), artifact/object store clients, evidence bundle writer.
|
||||
|
||||
## 3. Execution Phases
|
||||
1. **Plan** - parse manifest, validate schema, resolve inputs/secrets, build execution graph, compute canonical `planHash` (SHA-256 over normalised graph).
|
||||
2. **Simulation (optional)** - dry-run graph; emit determinstic preview with approvals/policy gates highlighted.
|
||||
3. **Execution** - verify runtime graph matches `planHash`; execute steps honoring `maxParallel`, `continueOnError`, `map`/`parallel` semantics; stream logs/events.
|
||||
4. **Evidence** - capture artifacts + transcripts, emit DSSE attestation binding `planHash`, inputs/outputs, steps, and timestamps; expose artifact listings via API for post-run retrieval.
|
||||
|
||||
## 4. API Surface (v1)
|
||||
- `POST /api/runs` (`packs.run`) - submit pack run; requires manifest/version, inputs, tenant context; returns `runId` + `planHash`.
|
||||
- `GET /api/runs/{runId}` (`packs.read`) - run status (graph, attempts, pending gates).
|
||||
- `GET /api/runs/{runId}/logs` (`packs.read`) - SSE stream of ordered log events.
|
||||
- `GET /api/runs/{runId}/artifacts` (`packs.read`) - list captured artifacts with digests/paths.
|
||||
- `POST /api/runs/{runId}/approve` (`packs.approve`) - record approval gate decision (requires Authority token claims `pack_run_id`, `pack_gate_id`, `pack_plan_hash`).
|
||||
- `POST /api/runs/{runId}/cancel` (`packs.run`) - cancel active run.
|
||||
- TODO (Phase II): `GET /.well-known/openapi` (TASKRUN-OAS-61-002) after OAS publication.
|
||||
|
||||
## 5. Data Model (Mongo, mirrors migration doc)
|
||||
- **pack_runs**: `_id`, `planHash`, `plan`, `failurePolicy`, `requestedAt`, `createdAt`, `updatedAt`, `steps[]`, `tenantId`.
|
||||
- **pack_run_logs**: `_id`, `runId`, `sequence` (monotonic), `timestamp` (UTC), `level`, `eventType`, `message`, `stepId?`, `metadata`.
|
||||
- **pack_artifacts**: `_id`, `runId`, `name`, `type`, `sourcePath?`, `storedPath?`, `status`, `notes?`, `capturedAt`.
|
||||
- Indexes as defined in `docs/modules/taskrunner/migrations/pack-run-collections.md`.
|
||||
|
||||
## 6. Step Types and Semantics
|
||||
- `run` - module invocation; declares `inputs`/`outputs`.
|
||||
- `parallel` - executes nested `steps[]`; honors `maxParallel`.
|
||||
- `map` - expands items into child steps (`stepId[index]::templateId`).
|
||||
- `gate.approval` - human approval checkpoint; enforces TTL/required count; pauses run until satisfied or expired.
|
||||
- `gate.policy` - Policy Engine evaluation; `failAction` decides halt vs. continue.
|
||||
|
||||
## 7. Determinism, Air-Gap, and Security
|
||||
- Plan hash binding: runtime graph must equal planned graph; mismatch aborts run.
|
||||
- All timestamps UTC ISO-8601; ordered logs via `(runId, sequence)` unique index.
|
||||
- Secrets never logged; evidence bundles store only redacted metadata.
|
||||
- Sealed mode: reject non-allowlisted network calls; approvals can be processed offline via request/response bundles.
|
||||
- RBAC scopes: `packs.read`, `packs.write`, `packs.run`, `packs.approve`.
|
||||
- Approval enforcement: service rejects approval decisions when provided `planHash` does not match stored run state (protects against stale/forged tokens).
|
||||
|
||||
## 8. Evidence & Attestation
|
||||
- DSSE attestation payload (`payloadType`: `application/vnd.stellaops.pack-run+json`) includes `runId`, `packName/version`, `planHash`, input/output digests, step statuses, `completedAt`.
|
||||
- Evidence bundle contents: signed manifest, inputs (redacted), outputs, transcripts, DSSE attestation; optional Rekor anchoring when online.
|
||||
|
||||
## 9. Observability (Phase I delivered)
|
||||
- Metrics: step latency, retries, queue depth, resource usage (`TASKRUN-OBS-50/51-001` DONE).
|
||||
- Pending: timeline events (`TASKRUN-OBS-52-001`), evidence snapshots (`TASKRUN-OBS-53-001`), attestations (`TASKRUN-OBS-54-001`), incident mode (`TASKRUN-OBS-55-001`).
|
||||
|
||||
## 10. Integration Points
|
||||
- **Authority** - approval tokens, scope validation, sealed-vault secrets.
|
||||
- **Policy Engine** - `gate.policy` decisions, policy context in evidence.
|
||||
- **Export Center** - evidence bundles and manifests for offline/air-gapped export.
|
||||
- **Orchestrator/CLI** - submission + resume flows; SSE log consumption.
|
||||
|
||||
## 11. Configuration (Mongo example)
|
||||
```json
|
||||
\"TaskRunner\": {
|
||||
\"Storage\": {
|
||||
\"Mode\": \"mongo\",
|
||||
\"Mongo\": {
|
||||
\"ConnectionString\": \"mongodb://127.0.0.1:27017/taskrunner\",
|
||||
\"Database\": \"taskrunner\",
|
||||
\"RunsCollection\": \"pack_runs\",
|
||||
\"LogsCollection\": \"pack_run_logs\",
|
||||
\"ArtifactsCollection\": \"pack_artifacts\",
|
||||
\"ApprovalsCollection\": \"pack_run_approvals\"
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## 12. References
|
||||
- Product advisory: `docs/product-advisories/29-Nov-2025 - Task Pack Orchestration and Automation.md`.
|
||||
- Task Pack spec + authoring + runbook: `docs/task-packs/spec.md`, `docs/task-packs/authoring-guide.md`, `docs/task-packs/runbook.md`.
|
||||
- Migration detail: `docs/modules/taskrunner/migrations/pack-run-collections.md`.
|
||||
Reference in New Issue
Block a user