feat: Implement MongoDB orchestrator storage with registry, commands, and heartbeats
Some checks failed
Docs CI / lint-and-preview (push) Has been cancelled

- Added NullAdvisoryObservationEventTransport for handling advisory observation events.
- Created IOrchestratorRegistryStore interface for orchestrator registry operations.
- Implemented MongoOrchestratorRegistryStore for MongoDB interactions with orchestrator data.
- Defined OrchestratorCommandDocument and OrchestratorCommandRecord for command handling.
- Added OrchestratorHeartbeatDocument and OrchestratorHeartbeatRecord for heartbeat tracking.
- Created OrchestratorRegistryDocument and OrchestratorRegistryRecord for registry management.
- Developed tests for orchestrator collections migration and MongoOrchestratorRegistryStore functionality.
- Introduced AirgapImportRequest and AirgapImportValidator for air-gapped VEX bundle imports.
- Added incident mode rules sample JSON for notifier configuration.
This commit is contained in:
StellaOps Bot
2025-11-22 12:35:38 +02:00
parent cbdc05b24d
commit f43e828b4e
96 changed files with 3425 additions and 976 deletions

View File

@@ -604,6 +604,7 @@ concelier:
1. **MVP**: Red Hat (CSAF), SUSE (CSAF), Ubuntu (USN JSON), OSV; JSON export.
2. **Add**: GHSA GraphQL, Debian (DSA HTML/JSON), Alpine secdb; Trivy DB export.
3. **Attestation handoff**: integrate with **Signer/Attestor** (optional).
4. **Scale & diagnostics**: provider dashboards, staleness alerts, export cache reuse.
3. **Attestation handoff**: integrate with **Signer/Attestor** (optional).
- Advisory evidence attestation parameters and path rules are documented in `docs/modules/concelier/attestation.md`.
4. **Scale & diagnostics**: provider dashboards, staleness alerts, export cache reuse.
5. **Offline kit**: endtoend verified bundles for airgap.

View File

@@ -0,0 +1,30 @@
# Concelier Advisory Evidence Attestation
## Purpose
- Describe how Concelier returns attestation claims for advisory evidence bundles so downstream services (Advisory AI, Attestor, Console) can verify provenance.
## Endpoint
- `GET /vuln/evidence/advisories/{advisoryKey}`
- Authentication/tenant headers follow standard Concelier rules (`X-Stella-Tenant` or `tenant` query; authority policies apply).
- Response payload: `AdvisoryEvidenceResponse { advisoryKey, records[], attestation? }`
## Attestation parameters (query)
- `bundlePath` (required to enable attestation): Relative or absolute path to evidence bundle tar.gz. Relative paths are resolved under the configured evidence root (`concelier:evidence:root`). Paths outside this root are rejected.
- `manifestPath` (optional): Defaults to sibling `manifest.json` next to the bundle.
- `transparencyPath` (optional): Defaults to sibling `transparency.json` next to the bundle.
- `pipelineVersion` (optional): Defaults to `concelier:evidence:pipelineVersion` (e.g., `git:<sha>`).
## Behavior
- When `bundlePath` is omitted, the endpoint returns evidence records only (attestation is `null`).
- When provided, Concelier builds claims via `EvidenceBundleAttestationBuilder` using the manifest and optional transparency payload; tenant must be lowercase per scope note.
- Path safety: All resolved files must live under the evidence root; traversal attempts or missing files silently skip attestation (response remains valid without claims).
## Configuration
- `concelier:evidence:root` (default `out/evidence/bundles`, resolved absolute).
- `concelier:evidence:defaultManifestFileName` (default `manifest.json`).
- `concelier:evidence:defaultTransparencyFileName` (default `transparency.json`).
- `concelier:evidence:pipelineVersion` (default `git:unknown`).
## References
- Evidence Locker attestation scope: `docs/modules/evidence-locker/attestation-scope-note.md`.
- Implementation: `src/Concelier/StellaOps.Concelier.WebService/Program.cs`, `StellaOps.Concelier.Core/Attestation/EvidenceBundleAttestationBuilder.cs`.

View File

@@ -0,0 +1,29 @@
# Concelier OAS & Observability Prep (61-001..63-001, 51-001..55-001)
Status: **Ready for implementation** (2025-11-22)
Owners: Concelier Core Guild · API Contracts Guild · DevOps/Observability Guilds
Scope: Freeze the API/SDK contracts and observability envelopes for LNM search/timeline APIs so downstream SDK, governance, and incident flows can proceed without schema churn.
## Inputs
- Frozen LNM payload schema: `docs/modules/concelier/link-not-merge-schema.md` (2025-11-17).
- Event contract: `docs/modules/concelier/events/advisory.observation.updated@1.md`.
- Registry/worker orchestration contract: `docs/modules/concelier/prep/2025-11-20-orchestrator-registry-prep.md`.
## Deliverables
- OpenAPI source stub for LNM + timeline surfaces recorded at `docs/modules/concelier/openapi/lnm-api.yaml` (paths enumerated; examples outlined below).
- SDK example library checklist covering `searchAdvisories`, `searchLinksets`, `getTimeline`, `getObservationById`; response bodies aligned to frozen schema; no consensus/merge fields.
- Observability contract (metrics/logs/traces):
- Metrics: `concelier_ingest_latency_seconds`, `concelier_linkset_conflicts_total`, `concelier_timeline_emit_lag_seconds`, `concelier_api_requests_total{route,tenant,status}` with burn-rate alert examples.
- Logs: structured fields `tenantId`, `advisoryKey`, `linksetId`, `timelineCursor`, `egressPolicy`.
- Traces: span names for `lnm.search`, `lnm.timeline`, `lnm.linkset-resolve` with baggage keys `tenant-id`, `request-id`.
- Incident/observability hooks: timeline/attestation enrichment notes for OBS-54/55 including DSSE envelope hash field and sealed-mode redaction rules.
## Acceptance Criteria
- Request/response shapes for `/api/v1/lnm/advisories`, `/api/v1/lnm/linksets`, `/api/v1/lnm/timeline` documented with required query params (`tenantId`, `productKey`, `offset`, `limit`, `sort`, `includeTimeline=true|false`).
- All responses MUST include `provenance` block (source, fetchedAt, digest, evidenceBundleId) and forbid consensus/merge fields.
- Metrics/logs names and labels are deterministic and lowercase; alert examples reference burn-rate SLOs.
- File path above is referenced from sprint trackers; any future schema edits require bumping version/comment in this prep doc.
## Notes
- This prep satisfies PREP-CONCELIER-OAS-61-001/002/62-001/63-001 and PREP-CONCELIER-OBS-51-001/52-001/53-001/54-001/55-001.
- No external dependencies remaining; downstream tasks may proceed using the stubbed OpenAPI and observability contracts here.

View File

@@ -0,0 +1,30 @@
# Excititor Air-Gap Prep (56-001, 57-001, 58-001)
Status: **Ready for implementation** (2025-11-22)
Owners: Excititor Core Guild · AirGap Policy Guild · Evidence Locker Guild
Scope: Define ingestion/egress contracts for Excititor when operating in sealed/offline environments and align with mirror bundle + Evidence Locker artifacts.
## Inputs
- Mirror bundle schema (thin) from `docs/modules/mirror/assembler.md`.
- Evidence Locker attestation contract: `docs/modules/evidence-locker/attestation-contract.md`.
- Link-Not-Merge schema for advisory evidence: `docs/modules/concelier/link-not-merge-schema.md`.
## Deliverables
- Ingestion envelope for `POST /airgap/vex/import`:
- Fields: `bundleId`, `mirrorGeneration`, `signedAt`, `publisher`, `payloadHash`, `payloadUrl?` (offline tar path), `signature`, `transparencyLog?`.
- Validation: deterministic hash of NDJSON payloads; must reject mixed tenants; clock-skew tolerance ±5s.
- Sealed-mode error catalog (57-001): `AIRGAP_EGRESS_BLOCKED`, `AIRGAP_PAYLOAD_STALE`, `AIRGAP_SIGNATURE_MISSING`, `AIRGAP_SOURCE_UNTRUSTED`; each with HTTP 4xx mapping and remediation text.
- Notification hooks (58-001): timeline events `airgap.import.started/completed/failed` with attributes `{tenantId,bundleId,generation,stalenessSeconds}`; link to Evidence Locker bundle ID for audit.
- Determinism rules: sort imported observations by `advisoryKey` then `productKey`; write timeline events in the same order; all timestamps UTC ISO-8601.
- Connector trust (CONN-TRUST-01-001):
- Trusted signer manifests reuse `docs/modules/excititor/schemas/connector-signer-metadata.schema.json`; require `fingerprint`, `issuer`, `validFrom/To`, `allowedProfiles`, `bundleHash`.
- Validation: fail import with `AIRGAP_SOURCE_UNTRUSTED` when signer fingerprint not in manifest, signature algorithm not in `{rsa-pss-sha256, ecdsa-p256-sha256, gost-r3410-2012-256}`, or bundle hash mismatch.
- Offline parity: store signer manifests alongside mirror bundle under `mirror/signers/` and include SHA256 in `SHA256SUMS.dsse`.
## Acceptance Criteria
- API shapes captured in this prep are referenced from Sprint 0119 Delivery Tracker; no further blockers for Excititor AirGap tasks.
- Error catalog and timeline events documented and consumed by downstream Policy/AirGap controller work.
- Import path validated against mirror bundle schema; mismatch should raise `AIRGAP_PAYLOAD_STALE`.
## Notes
- Satisfies PREP-EXCITITOR-AIRGAP-56-001, PREP-EXCITITOR-AIRGAP-57-001, and PREP-EXCITITOR-AIRGAP-58-001.

View File

@@ -0,0 +1,27 @@
# Attestation Verifier Rehearsal — Excititor
Status: **Ready for implementation** (2025-11-22)
Owners: Excititor Attestation Guild · Evidence Locker Guild
Scope: Dry-run `IVexAttestationVerifier` against current Evidence Locker bundles to ensure Excititor attestation endpoints ship with deterministic verification.
## Test Matrix
- Inputs: Evidence Bundle v1 sample (`docs/samples/evidence-bundle/*`), mirror bundle thin sample (`out/mirror/thin/mirror-thin-m0-sample.tar.gz`).
- Verification steps:
1. Validate DSSE envelope signature and Rekor entry (if present); offline mode skips transparency but records `rekorSkipped=true`.
2. Verify manifest hash tree against payload NDJSON files; fail on first mismatch.
3. Assert policy hash matches Policy Engine overlay hash (placeholder `policyHash` captured for now).
4. Emit structured result JSON: `{bundleId, verified, dsseVerified, transparencyChecked, manifestRoot, failures[]}`.
- Determinism: sorted failure list, timestamps set to supplied `--as-of` flag.
## Deliverables
- Harness entry point: `tools/attestation/verifier-rehearsal.sh` (script stub path reserved).
- Sample output recorded at `docs/modules/excititor/prep/artifacts/2025-11-22-attestation-rehearsal.json` (to be produced in implementation).
- Logging fields to surface in Excititor: `attestationBundleId`, `evidenceBundleId`, `verified`, `failureCode`, `tenantId`.
## Acceptance Criteria
- Rehearsal script runs offline using bundled samples and exits non-zero on any verification failure.
- Output schema above is referenced by Excititor API tests and Policy attest replay tasks.
- Downstream tasks EXCITITOR-GRAPH-21-00x and attestation endpoints can rely on this contract.
## Notes
- Satisfies PREP-ATTESTATION-VERIFIER-REHEARSAL-EXCITITOR.

View File

@@ -0,0 +1,33 @@
# Findings Ledger Prep — 29-008, 34-101, AIRGAP-56-001
Status: **Ready for implementation** (2025-11-22)
Owners: Findings Ledger Guild · Observability Guild · AirGap Time Guild · Orchestrator Guild
Scope: Provide the missing contracts needed to unblock LEDGER-29-008 load/replay, LEDGER-34-101 orchestrator export linkage, and LEDGER-AIRGAP-56-001 bundle provenance recording.
## Observability (LEDGER-29-008)
- Metrics schema locked:
- `ledger_projection_lag_seconds{tenant}` (gauge)
- `ledger_write_duration_seconds_bucket` (histogram)
- `ledger_events_total{tenant,kind}` (counter)
- Alert: burn-rate 4xx/5xx on ingestion path >2% over 5m/1h.
- Log fields: `tenantId`, `requestId`, `projectionCheckpoint`, `bundleId?`, `attestationId?`, `operation`.
- Trace spans: `ledger.write`, `ledger.replay`, `ledger.restore` with baggage `tenant-id`, `bundle-id`.
## Orchestrator export linkage (LEDGER-34-101)
- Export payload shape (from Orchestrator Sprint 150.A):
- `runId` (uuid), `jobType`, `artifactHash`, `policyHash`, `startedAt`, `completedAt`, `status`, `manifestPath`, `logsPath`.
- Ledger integration rule: store export rows under collection `orchestrator_exports` with index `(artifactHash, runId)`; anchor Merkle root into ledger timeline entry `ledger_export` referencing above fields.
## AirGap provenance (LEDGER-AIRGAP-56-001)
- Mirror bundle contract alignment:
- fields recorded per import: `bundleId`, `mirrorGeneration`, `merkleRoot`, `timeAnchor`, `publisher`, `hashAlgorithm`, `contents[]` (sha256 of NDJSON segments).
- determinism: imports sorted by `bundleId`; all timestamps UTC.
- API for recording import: `POST /internal/ledger/airgap-import` with payload above; respond 202 + `ledgerEntryId`.
## Acceptance Criteria
- Metrics/log/logging names frozen as above and added to `docs/modules/findings-ledger/observability.md` in next implementation step.
- Orchestrator export payload shape referenced by both Ledger and Orchestrator tasks; no missing fields for audit.
- AirGap import payload is deterministic and replays without external network requirements.
## Notes
- Satisfies PREP-LEDGER-29-008-AWAIT-OBSERVABILITY-SCHEMA, PREP-LEDGER-34-101-ORCHESTRATOR-LEDGER-EXPORT, and PREP-LEDGER-AIRGAP-56-001-MIRROR-BUNDLE-SCHEM.

View File

@@ -0,0 +1,22 @@
# Policy Export · Console 23-001 Prep
Status: **Ready for implementation** (2025-11-22)
Owners: Policy Guild · Scheduler Guild · Observability Guild
Scope: Define the export bundle + scheduler job surface needed for Console export task 23-001.
## Export Bundle Contract
- Payload: NDJSON of advisory and VEX observations filtered by tenant + product set; include `provenance` block with `source`, `fetchedAt`, `digest`, `evidenceBundleId`.
- Envelope: `bundleId`, `createdAt`, `policyHash`, `schemaVersion`, `itemsSha256`, optional `transparency` entry.
- Determinism: sort items by `observationId`; timestamps truncated to seconds; hashes sha256 lower-case hex.
## Scheduler Job Shape
- `jobType`: `console-export` with fields `tenantId`, `bundleId`, `filters`, `destinationUri` (file:// or s3://), `retryPolicy`, `notificationChannel`.
- Status transitions: `queued → running → completed/failed`; progress fields `exportedCount`, `failedCount`.
- Observability: metrics `policy_console_export_jobs_total{status}`, `policy_console_export_duration_seconds`, logs with `bundleId` and `destinationUri`.
## Acceptance Criteria
- Job shape and bundle envelope above referenced from Sprint 123 Delivery Tracker; no other blocking dependencies.
- Export artefacts can be consumed offline; notification optional.
## Notes
- Satisfies PREP-EXPORT-CONSOLE-23-001-MISSING-EXPORT-BUN.

View File

@@ -0,0 +1,24 @@
# Policy AirGap 56-001 Prep — Mirror Bundle Schema
Status: **Ready for implementation** (2025-11-22)
Owners: Policy Guild · AirGap Policy Guild
Scope: Lock the mirror bundle schema required for policy pack distribution in sealed mode.
## Bundle Schema
- Envelope: `bundleId`, `schemaVersion` (`policy-mirror-v1`), `createdAt`, `publisher`, `signature`, `transparencyLog?`.
- Contents array entries:
- `policyHash`, `policyName`, `version`, `targetProducts[]`, `scope` (tenant/portfolio), `dependencies[]` (other bundleIds), `config` (immutable), `evidenceBundleId?`.
- Hashing: manifest sha256 over sorted file list (`policies/*.rego`, `overlays/*.json`), recorded as `manifestSha256`.
- Staleness: `validUntil` and `timeAnchor` fields; sealed-mode must reject if `now > validUntil` or time drift > 300s.
## Determinism & Transport
- Files stored in tarball with POSIX mtime `2025-01-01T00:00:00Z` and uid/gid 0.
- Manifest path fixed to `MANIFEST.json`; entries sorted alphabetically.
- No external fetches permitted during import.
## Acceptance Criteria
- Schema above referenced by Policy AirGap tasks and aligns with Mirror bundle contract (`docs/modules/mirror/assembler.md`).
- Import validators know failure codes: `POLICY_BUNDLE_STALE`, `POLICY_BUNDLE_SIGNATURE_INVALID`, `POLICY_BUNDLE_SCHEMA_MISMATCH`.
## Notes
- Completes PREP-POLICY-AIRGAP-56-001-MIRROR-BUNDLE-SCHEM.

View File

@@ -0,0 +1,24 @@
# Policy Engine 30-001 Prep — Overlay Projection
Status: **Ready for implementation** (2025-11-22)
Owners: Policy Guild · Cartographer Guild
Scope: Freeze overlay projection contract needed for Policy Engine phase III.
## Inputs
- Path/scope schema: `docs/modules/policy/prep/2025-11-20-policy-engine-29-002-prep.md`.
- Metrics/log schema: `docs/modules/policy/prep/2025-11-20-policy-engine-29-004-prep.md`.
## Overlay Projection Contract
- Input: evaluation records `{tenantId, policyId, pathScope[], evidenceDigest, decision, reasons[]}`.
- Output overlay entry:
- `overlayId` (deterministic hash of `tenantId+policyId+pathScope+evidenceDigest`)
- `subject` (`purl` or `pkg` tuple), `pathScope`, `decision`, `rationale`, `inputs` (evidence digests), `policyVersion`, `createdAt`.
- Determinism: pathScope sorted; reasons sorted by `code`; timestamps set to evaluation clock.
- Export shape for downstream components via `/overlays/{overlayId}` and `/overlays/search` with paging.
## Acceptance Criteria
- Overlay schema above referenced in Sprint 125 and 0125 trackers; downstream simulation/bridge tasks can consume it without further schema.
- Any future field additions require version bump `overlaySchemaVersion`.
## Notes
- Completes PREP-POLICY-ENGINE-30-001-WAITING-ON-29-004-M.