feat: Implement MongoDB orchestrator storage with registry, commands, and heartbeats
Some checks failed
Docs CI / lint-and-preview (push) Has been cancelled
Some checks failed
Docs CI / lint-and-preview (push) Has been cancelled
- Added NullAdvisoryObservationEventTransport for handling advisory observation events. - Created IOrchestratorRegistryStore interface for orchestrator registry operations. - Implemented MongoOrchestratorRegistryStore for MongoDB interactions with orchestrator data. - Defined OrchestratorCommandDocument and OrchestratorCommandRecord for command handling. - Added OrchestratorHeartbeatDocument and OrchestratorHeartbeatRecord for heartbeat tracking. - Created OrchestratorRegistryDocument and OrchestratorRegistryRecord for registry management. - Developed tests for orchestrator collections migration and MongoOrchestratorRegistryStore functionality. - Introduced AirgapImportRequest and AirgapImportValidator for air-gapped VEX bundle imports. - Added incident mode rules sample JSON for notifier configuration.
This commit is contained in:
@@ -604,6 +604,7 @@ concelier:
|
||||
|
||||
1. **MVP**: Red Hat (CSAF), SUSE (CSAF), Ubuntu (USN JSON), OSV; JSON export.
|
||||
2. **Add**: GHSA GraphQL, Debian (DSA HTML/JSON), Alpine secdb; Trivy DB export.
|
||||
3. **Attestation hand‑off**: integrate with **Signer/Attestor** (optional).
|
||||
4. **Scale & diagnostics**: provider dashboards, staleness alerts, export cache reuse.
|
||||
3. **Attestation hand‑off**: integrate with **Signer/Attestor** (optional).
|
||||
- Advisory evidence attestation parameters and path rules are documented in `docs/modules/concelier/attestation.md`.
|
||||
4. **Scale & diagnostics**: provider dashboards, staleness alerts, export cache reuse.
|
||||
5. **Offline kit**: end‑to‑end verified bundles for air‑gap.
|
||||
|
||||
30
docs/modules/concelier/attestation.md
Normal file
30
docs/modules/concelier/attestation.md
Normal file
@@ -0,0 +1,30 @@
|
||||
# Concelier Advisory Evidence Attestation
|
||||
|
||||
## Purpose
|
||||
- Describe how Concelier returns attestation claims for advisory evidence bundles so downstream services (Advisory AI, Attestor, Console) can verify provenance.
|
||||
|
||||
## Endpoint
|
||||
- `GET /vuln/evidence/advisories/{advisoryKey}`
|
||||
- Authentication/tenant headers follow standard Concelier rules (`X-Stella-Tenant` or `tenant` query; authority policies apply).
|
||||
- Response payload: `AdvisoryEvidenceResponse { advisoryKey, records[], attestation? }`
|
||||
|
||||
## Attestation parameters (query)
|
||||
- `bundlePath` (required to enable attestation): Relative or absolute path to evidence bundle tar.gz. Relative paths are resolved under the configured evidence root (`concelier:evidence:root`). Paths outside this root are rejected.
|
||||
- `manifestPath` (optional): Defaults to sibling `manifest.json` next to the bundle.
|
||||
- `transparencyPath` (optional): Defaults to sibling `transparency.json` next to the bundle.
|
||||
- `pipelineVersion` (optional): Defaults to `concelier:evidence:pipelineVersion` (e.g., `git:<sha>`).
|
||||
|
||||
## Behavior
|
||||
- When `bundlePath` is omitted, the endpoint returns evidence records only (attestation is `null`).
|
||||
- When provided, Concelier builds claims via `EvidenceBundleAttestationBuilder` using the manifest and optional transparency payload; tenant must be lowercase per scope note.
|
||||
- Path safety: All resolved files must live under the evidence root; traversal attempts or missing files silently skip attestation (response remains valid without claims).
|
||||
|
||||
## Configuration
|
||||
- `concelier:evidence:root` (default `out/evidence/bundles`, resolved absolute).
|
||||
- `concelier:evidence:defaultManifestFileName` (default `manifest.json`).
|
||||
- `concelier:evidence:defaultTransparencyFileName` (default `transparency.json`).
|
||||
- `concelier:evidence:pipelineVersion` (default `git:unknown`).
|
||||
|
||||
## References
|
||||
- Evidence Locker attestation scope: `docs/modules/evidence-locker/attestation-scope-note.md`.
|
||||
- Implementation: `src/Concelier/StellaOps.Concelier.WebService/Program.cs`, `StellaOps.Concelier.Core/Attestation/EvidenceBundleAttestationBuilder.cs`.
|
||||
29
docs/modules/concelier/prep/2025-11-22-oas-obs-prep.md
Normal file
29
docs/modules/concelier/prep/2025-11-22-oas-obs-prep.md
Normal file
@@ -0,0 +1,29 @@
|
||||
# Concelier OAS & Observability Prep (61-001..63-001, 51-001..55-001)
|
||||
|
||||
Status: **Ready for implementation** (2025-11-22)
|
||||
Owners: Concelier Core Guild · API Contracts Guild · DevOps/Observability Guilds
|
||||
Scope: Freeze the API/SDK contracts and observability envelopes for LNM search/timeline APIs so downstream SDK, governance, and incident flows can proceed without schema churn.
|
||||
|
||||
## Inputs
|
||||
- Frozen LNM payload schema: `docs/modules/concelier/link-not-merge-schema.md` (2025-11-17).
|
||||
- Event contract: `docs/modules/concelier/events/advisory.observation.updated@1.md`.
|
||||
- Registry/worker orchestration contract: `docs/modules/concelier/prep/2025-11-20-orchestrator-registry-prep.md`.
|
||||
|
||||
## Deliverables
|
||||
- OpenAPI source stub for LNM + timeline surfaces recorded at `docs/modules/concelier/openapi/lnm-api.yaml` (paths enumerated; examples outlined below).
|
||||
- SDK example library checklist covering `searchAdvisories`, `searchLinksets`, `getTimeline`, `getObservationById`; response bodies aligned to frozen schema; no consensus/merge fields.
|
||||
- Observability contract (metrics/logs/traces):
|
||||
- Metrics: `concelier_ingest_latency_seconds`, `concelier_linkset_conflicts_total`, `concelier_timeline_emit_lag_seconds`, `concelier_api_requests_total{route,tenant,status}` with burn-rate alert examples.
|
||||
- Logs: structured fields `tenantId`, `advisoryKey`, `linksetId`, `timelineCursor`, `egressPolicy`.
|
||||
- Traces: span names for `lnm.search`, `lnm.timeline`, `lnm.linkset-resolve` with baggage keys `tenant-id`, `request-id`.
|
||||
- Incident/observability hooks: timeline/attestation enrichment notes for OBS-54/55 including DSSE envelope hash field and sealed-mode redaction rules.
|
||||
|
||||
## Acceptance Criteria
|
||||
- Request/response shapes for `/api/v1/lnm/advisories`, `/api/v1/lnm/linksets`, `/api/v1/lnm/timeline` documented with required query params (`tenantId`, `productKey`, `offset`, `limit`, `sort`, `includeTimeline=true|false`).
|
||||
- All responses MUST include `provenance` block (source, fetchedAt, digest, evidenceBundleId) and forbid consensus/merge fields.
|
||||
- Metrics/logs names and labels are deterministic and lowercase; alert examples reference burn-rate SLOs.
|
||||
- File path above is referenced from sprint trackers; any future schema edits require bumping version/comment in this prep doc.
|
||||
|
||||
## Notes
|
||||
- This prep satisfies PREP-CONCELIER-OAS-61-001/002/62-001/63-001 and PREP-CONCELIER-OBS-51-001/52-001/53-001/54-001/55-001.
|
||||
- No external dependencies remaining; downstream tasks may proceed using the stubbed OpenAPI and observability contracts here.
|
||||
30
docs/modules/excititor/prep/2025-11-22-airgap-56-58-prep.md
Normal file
30
docs/modules/excititor/prep/2025-11-22-airgap-56-58-prep.md
Normal file
@@ -0,0 +1,30 @@
|
||||
# Excititor Air-Gap Prep (56-001, 57-001, 58-001)
|
||||
|
||||
Status: **Ready for implementation** (2025-11-22)
|
||||
Owners: Excititor Core Guild · AirGap Policy Guild · Evidence Locker Guild
|
||||
Scope: Define ingestion/egress contracts for Excititor when operating in sealed/offline environments and align with mirror bundle + Evidence Locker artifacts.
|
||||
|
||||
## Inputs
|
||||
- Mirror bundle schema (thin) from `docs/modules/mirror/assembler.md`.
|
||||
- Evidence Locker attestation contract: `docs/modules/evidence-locker/attestation-contract.md`.
|
||||
- Link-Not-Merge schema for advisory evidence: `docs/modules/concelier/link-not-merge-schema.md`.
|
||||
|
||||
## Deliverables
|
||||
- Ingestion envelope for `POST /airgap/vex/import`:
|
||||
- Fields: `bundleId`, `mirrorGeneration`, `signedAt`, `publisher`, `payloadHash`, `payloadUrl?` (offline tar path), `signature`, `transparencyLog?`.
|
||||
- Validation: deterministic hash of NDJSON payloads; must reject mixed tenants; clock-skew tolerance ±5s.
|
||||
- Sealed-mode error catalog (57-001): `AIRGAP_EGRESS_BLOCKED`, `AIRGAP_PAYLOAD_STALE`, `AIRGAP_SIGNATURE_MISSING`, `AIRGAP_SOURCE_UNTRUSTED`; each with HTTP 4xx mapping and remediation text.
|
||||
- Notification hooks (58-001): timeline events `airgap.import.started/completed/failed` with attributes `{tenantId,bundleId,generation,stalenessSeconds}`; link to Evidence Locker bundle ID for audit.
|
||||
- Determinism rules: sort imported observations by `advisoryKey` then `productKey`; write timeline events in the same order; all timestamps UTC ISO-8601.
|
||||
- Connector trust (CONN-TRUST-01-001):
|
||||
- Trusted signer manifests reuse `docs/modules/excititor/schemas/connector-signer-metadata.schema.json`; require `fingerprint`, `issuer`, `validFrom/To`, `allowedProfiles`, `bundleHash`.
|
||||
- Validation: fail import with `AIRGAP_SOURCE_UNTRUSTED` when signer fingerprint not in manifest, signature algorithm not in `{rsa-pss-sha256, ecdsa-p256-sha256, gost-r3410-2012-256}`, or bundle hash mismatch.
|
||||
- Offline parity: store signer manifests alongside mirror bundle under `mirror/signers/` and include SHA256 in `SHA256SUMS.dsse`.
|
||||
|
||||
## Acceptance Criteria
|
||||
- API shapes captured in this prep are referenced from Sprint 0119 Delivery Tracker; no further blockers for Excititor AirGap tasks.
|
||||
- Error catalog and timeline events documented and consumed by downstream Policy/AirGap controller work.
|
||||
- Import path validated against mirror bundle schema; mismatch should raise `AIRGAP_PAYLOAD_STALE`.
|
||||
|
||||
## Notes
|
||||
- Satisfies PREP-EXCITITOR-AIRGAP-56-001, PREP-EXCITITOR-AIRGAP-57-001, and PREP-EXCITITOR-AIRGAP-58-001.
|
||||
@@ -0,0 +1,27 @@
|
||||
# Attestation Verifier Rehearsal — Excititor
|
||||
|
||||
Status: **Ready for implementation** (2025-11-22)
|
||||
Owners: Excititor Attestation Guild · Evidence Locker Guild
|
||||
Scope: Dry-run `IVexAttestationVerifier` against current Evidence Locker bundles to ensure Excititor attestation endpoints ship with deterministic verification.
|
||||
|
||||
## Test Matrix
|
||||
- Inputs: Evidence Bundle v1 sample (`docs/samples/evidence-bundle/*`), mirror bundle thin sample (`out/mirror/thin/mirror-thin-m0-sample.tar.gz`).
|
||||
- Verification steps:
|
||||
1. Validate DSSE envelope signature and Rekor entry (if present); offline mode skips transparency but records `rekorSkipped=true`.
|
||||
2. Verify manifest hash tree against payload NDJSON files; fail on first mismatch.
|
||||
3. Assert policy hash matches Policy Engine overlay hash (placeholder `policyHash` captured for now).
|
||||
4. Emit structured result JSON: `{bundleId, verified, dsseVerified, transparencyChecked, manifestRoot, failures[]}`.
|
||||
- Determinism: sorted failure list, timestamps set to supplied `--as-of` flag.
|
||||
|
||||
## Deliverables
|
||||
- Harness entry point: `tools/attestation/verifier-rehearsal.sh` (script stub path reserved).
|
||||
- Sample output recorded at `docs/modules/excititor/prep/artifacts/2025-11-22-attestation-rehearsal.json` (to be produced in implementation).
|
||||
- Logging fields to surface in Excititor: `attestationBundleId`, `evidenceBundleId`, `verified`, `failureCode`, `tenantId`.
|
||||
|
||||
## Acceptance Criteria
|
||||
- Rehearsal script runs offline using bundled samples and exits non-zero on any verification failure.
|
||||
- Output schema above is referenced by Excititor API tests and Policy attest replay tasks.
|
||||
- Downstream tasks EXCITITOR-GRAPH-21-00x and attestation endpoints can rely on this contract.
|
||||
|
||||
## Notes
|
||||
- Satisfies PREP-ATTESTATION-VERIFIER-REHEARSAL-EXCITITOR.
|
||||
@@ -0,0 +1,33 @@
|
||||
# Findings Ledger Prep — 29-008, 34-101, AIRGAP-56-001
|
||||
|
||||
Status: **Ready for implementation** (2025-11-22)
|
||||
Owners: Findings Ledger Guild · Observability Guild · AirGap Time Guild · Orchestrator Guild
|
||||
Scope: Provide the missing contracts needed to unblock LEDGER-29-008 load/replay, LEDGER-34-101 orchestrator export linkage, and LEDGER-AIRGAP-56-001 bundle provenance recording.
|
||||
|
||||
## Observability (LEDGER-29-008)
|
||||
- Metrics schema locked:
|
||||
- `ledger_projection_lag_seconds{tenant}` (gauge)
|
||||
- `ledger_write_duration_seconds_bucket` (histogram)
|
||||
- `ledger_events_total{tenant,kind}` (counter)
|
||||
- Alert: burn-rate 4xx/5xx on ingestion path >2% over 5m/1h.
|
||||
- Log fields: `tenantId`, `requestId`, `projectionCheckpoint`, `bundleId?`, `attestationId?`, `operation`.
|
||||
- Trace spans: `ledger.write`, `ledger.replay`, `ledger.restore` with baggage `tenant-id`, `bundle-id`.
|
||||
|
||||
## Orchestrator export linkage (LEDGER-34-101)
|
||||
- Export payload shape (from Orchestrator Sprint 150.A):
|
||||
- `runId` (uuid), `jobType`, `artifactHash`, `policyHash`, `startedAt`, `completedAt`, `status`, `manifestPath`, `logsPath`.
|
||||
- Ledger integration rule: store export rows under collection `orchestrator_exports` with index `(artifactHash, runId)`; anchor Merkle root into ledger timeline entry `ledger_export` referencing above fields.
|
||||
|
||||
## AirGap provenance (LEDGER-AIRGAP-56-001)
|
||||
- Mirror bundle contract alignment:
|
||||
- fields recorded per import: `bundleId`, `mirrorGeneration`, `merkleRoot`, `timeAnchor`, `publisher`, `hashAlgorithm`, `contents[]` (sha256 of NDJSON segments).
|
||||
- determinism: imports sorted by `bundleId`; all timestamps UTC.
|
||||
- API for recording import: `POST /internal/ledger/airgap-import` with payload above; respond 202 + `ledgerEntryId`.
|
||||
|
||||
## Acceptance Criteria
|
||||
- Metrics/log/logging names frozen as above and added to `docs/modules/findings-ledger/observability.md` in next implementation step.
|
||||
- Orchestrator export payload shape referenced by both Ledger and Orchestrator tasks; no missing fields for audit.
|
||||
- AirGap import payload is deterministic and replays without external network requirements.
|
||||
|
||||
## Notes
|
||||
- Satisfies PREP-LEDGER-29-008-AWAIT-OBSERVABILITY-SCHEMA, PREP-LEDGER-34-101-ORCHESTRATOR-LEDGER-EXPORT, and PREP-LEDGER-AIRGAP-56-001-MIRROR-BUNDLE-SCHEM.
|
||||
@@ -0,0 +1,22 @@
|
||||
# Policy Export · Console 23-001 Prep
|
||||
|
||||
Status: **Ready for implementation** (2025-11-22)
|
||||
Owners: Policy Guild · Scheduler Guild · Observability Guild
|
||||
Scope: Define the export bundle + scheduler job surface needed for Console export task 23-001.
|
||||
|
||||
## Export Bundle Contract
|
||||
- Payload: NDJSON of advisory and VEX observations filtered by tenant + product set; include `provenance` block with `source`, `fetchedAt`, `digest`, `evidenceBundleId`.
|
||||
- Envelope: `bundleId`, `createdAt`, `policyHash`, `schemaVersion`, `itemsSha256`, optional `transparency` entry.
|
||||
- Determinism: sort items by `observationId`; timestamps truncated to seconds; hashes sha256 lower-case hex.
|
||||
|
||||
## Scheduler Job Shape
|
||||
- `jobType`: `console-export` with fields `tenantId`, `bundleId`, `filters`, `destinationUri` (file:// or s3://), `retryPolicy`, `notificationChannel`.
|
||||
- Status transitions: `queued → running → completed/failed`; progress fields `exportedCount`, `failedCount`.
|
||||
- Observability: metrics `policy_console_export_jobs_total{status}`, `policy_console_export_duration_seconds`, logs with `bundleId` and `destinationUri`.
|
||||
|
||||
## Acceptance Criteria
|
||||
- Job shape and bundle envelope above referenced from Sprint 123 Delivery Tracker; no other blocking dependencies.
|
||||
- Export artefacts can be consumed offline; notification optional.
|
||||
|
||||
## Notes
|
||||
- Satisfies PREP-EXPORT-CONSOLE-23-001-MISSING-EXPORT-BUN.
|
||||
@@ -0,0 +1,24 @@
|
||||
# Policy AirGap 56-001 Prep — Mirror Bundle Schema
|
||||
|
||||
Status: **Ready for implementation** (2025-11-22)
|
||||
Owners: Policy Guild · AirGap Policy Guild
|
||||
Scope: Lock the mirror bundle schema required for policy pack distribution in sealed mode.
|
||||
|
||||
## Bundle Schema
|
||||
- Envelope: `bundleId`, `schemaVersion` (`policy-mirror-v1`), `createdAt`, `publisher`, `signature`, `transparencyLog?`.
|
||||
- Contents array entries:
|
||||
- `policyHash`, `policyName`, `version`, `targetProducts[]`, `scope` (tenant/portfolio), `dependencies[]` (other bundleIds), `config` (immutable), `evidenceBundleId?`.
|
||||
- Hashing: manifest sha256 over sorted file list (`policies/*.rego`, `overlays/*.json`), recorded as `manifestSha256`.
|
||||
- Staleness: `validUntil` and `timeAnchor` fields; sealed-mode must reject if `now > validUntil` or time drift > 300s.
|
||||
|
||||
## Determinism & Transport
|
||||
- Files stored in tarball with POSIX mtime `2025-01-01T00:00:00Z` and uid/gid 0.
|
||||
- Manifest path fixed to `MANIFEST.json`; entries sorted alphabetically.
|
||||
- No external fetches permitted during import.
|
||||
|
||||
## Acceptance Criteria
|
||||
- Schema above referenced by Policy AirGap tasks and aligns with Mirror bundle contract (`docs/modules/mirror/assembler.md`).
|
||||
- Import validators know failure codes: `POLICY_BUNDLE_STALE`, `POLICY_BUNDLE_SIGNATURE_INVALID`, `POLICY_BUNDLE_SCHEMA_MISMATCH`.
|
||||
|
||||
## Notes
|
||||
- Completes PREP-POLICY-AIRGAP-56-001-MIRROR-BUNDLE-SCHEM.
|
||||
@@ -0,0 +1,24 @@
|
||||
# Policy Engine 30-001 Prep — Overlay Projection
|
||||
|
||||
Status: **Ready for implementation** (2025-11-22)
|
||||
Owners: Policy Guild · Cartographer Guild
|
||||
Scope: Freeze overlay projection contract needed for Policy Engine phase III.
|
||||
|
||||
## Inputs
|
||||
- Path/scope schema: `docs/modules/policy/prep/2025-11-20-policy-engine-29-002-prep.md`.
|
||||
- Metrics/log schema: `docs/modules/policy/prep/2025-11-20-policy-engine-29-004-prep.md`.
|
||||
|
||||
## Overlay Projection Contract
|
||||
- Input: evaluation records `{tenantId, policyId, pathScope[], evidenceDigest, decision, reasons[]}`.
|
||||
- Output overlay entry:
|
||||
- `overlayId` (deterministic hash of `tenantId+policyId+pathScope+evidenceDigest`)
|
||||
- `subject` (`purl` or `pkg` tuple), `pathScope`, `decision`, `rationale`, `inputs` (evidence digests), `policyVersion`, `createdAt`.
|
||||
- Determinism: pathScope sorted; reasons sorted by `code`; timestamps set to evaluation clock.
|
||||
- Export shape for downstream components via `/overlays/{overlayId}` and `/overlays/search` with paging.
|
||||
|
||||
## Acceptance Criteria
|
||||
- Overlay schema above referenced in Sprint 125 and 0125 trackers; downstream simulation/bridge tasks can consume it without further schema.
|
||||
- Any future field additions require version bump `overlaySchemaVersion`.
|
||||
|
||||
## Notes
|
||||
- Completes PREP-POLICY-ENGINE-30-001-WAITING-ON-29-004-M.
|
||||
Reference in New Issue
Block a user