docs consolidation work
This commit is contained in:
StellaOps Bot
@@ -1,6 +1,15 @@
|
||||
# StellaOps Concelier
|
||||
|
||||
Concelier ingests signed advisories from dozens of sources and converts them into immutable observations plus linksets under the Aggregation-Only Contract (AOC).
|
||||
Concelier ingests signed advisories from **32 advisory connectors** and converts them into immutable observations plus linksets under the Aggregation-Only Contract (AOC).
|
||||
|
||||
**Advisory Sources (32 connectors):**
|
||||
- **National CERTs (8):** ACSC (Australia), CCCS (Canada), CERT-Bund (Germany), CERT-CC (US), CERT-FR (France), CERT-IN (India), JVN (Japan), KISA (Korea)
|
||||
- **OS Distros (5):** Alpine SecDB, Debian Security Tracker, RedHat OVAL, SUSE OVAL, Ubuntu USN
|
||||
- **Vendors (7):** Apple, Adobe, Chromium, Cisco PSIRT, Microsoft MSRC, Oracle, VMware
|
||||
- **Standards (5):** CVE, NVD, GHSA (GitHub), OSV, EPSS v4
|
||||
- **Threat Intel (3):** KEV (CISA Exploited Vulns), CISA ICS, Kaspersky ICS
|
||||
- **Regional (3):** Russia BDU, Russia NKCKI, Plus regional mirrors
|
||||
- **Internal (1):** StellaOps internal mirror
|
||||
|
||||
## Responsibilities
|
||||
- Fetch and normalise vulnerability advisories via restart-time connectors.
|
||||
@@ -41,3 +50,28 @@ Concelier ingests signed advisories from dozens of sources and converts them int
|
||||
## Epic alignment
|
||||
- **Epic 1 – AOC enforcement:** uphold raw observation invariants, provenance requirements, linkset-only enrichment, and AOC verifier guardrails across every connector.
|
||||
- **Epic 10 – Export Center:** expose deterministic advisory exports and metadata required by JSON/Trivy/mirror bundles.
|
||||
|
||||
## Implementation Status
|
||||
|
||||
**Delivery Phases:**
|
||||
- Phase 1 (Guardrails & schema) – PostgreSQL validators, AOCWriteGuard interceptor, deterministic linkset builders operational
|
||||
- Phase 2 (API & observability) – Ingestion/verification endpoints with Authority scopes, telemetry, Offline Kit packaging
|
||||
- Phase 3 (Experience polish) – CLI/Console affordances, Export Center hand-off metadata, CI enforcement
|
||||
|
||||
**Acceptance Criteria:**
|
||||
- PostgreSQL validators and runtime guards reject forbidden fields and missing provenance with ERR_AOC_00x codes
|
||||
- Linksets and supersedes chains deterministic; identical payloads yield byte-identical documents
|
||||
- CLI `stella aoc verify` exits non-zero on violations, zero on clean datasets
|
||||
- Export Center consumes advisory datasets without legacy normalized fields
|
||||
- CI fails on lint violations or guard test regressions
|
||||
|
||||
**Key Risks & Mitigations:**
|
||||
- Collector drift: guard middleware + CI lint + schema validation; RFC required for linkset changes
|
||||
- Migration complexity: staged cutover with backup copies, temporary views for Policy Engine parity
|
||||
- Performance overhead: guard remains O(number of keys), index review for insert latency targets
|
||||
- Tenancy leakage: tenant required in schema, Authority claims enforced, observability alerts
|
||||
|
||||
**Recent Milestones:**
|
||||
- Sprint 110 attestation chain validated, evidence bundle tests green
|
||||
- Link-Not-Merge cache and console consumption docs frozen
|
||||
- Observation events transport reviewed, NATS/air-gap guidance updated
|
||||
|
||||
@@ -1,73 +0,0 @@
|
||||
# Implementation plan — Concelier
|
||||
|
||||
## Delivery timeline
|
||||
- **Phase 1 — Guardrails & schema**
|
||||
Stand up PostgreSQL JSON schema validators for `advisory_raw` and `vex_raw`, wire the `AOCWriteGuard` repository interceptor, and seed deterministic linkset builders. Freeze legacy normalisation paths and migrate callers to the new raw schema.
|
||||
- **Phase 2 — API & observability**
|
||||
Publish ingestion and verification endpoints (`POST /ingest/*`, `GET /advisories.raw`, `POST /aoc/verify`) with Authority scopes, expose telemetry (`aoc_violation_total`, guard spans, structured logs), and ensure Offline Kit packaging captures validator deployment steps.
|
||||
- **Phase 3 — Experience polish**
|
||||
Ship CLI/Console affordances (`stella sources ingest --dry-run`, dashboard tiles, violation drill-downs), finish Export Center hand-off metadata, and close out CI enforcement (`stella aoc verify` preflight, AST lint, seeded fixtures).
|
||||
|
||||
## Work breakdown by component
|
||||
- **Concelier WebService & worker**
|
||||
- Add PostgreSQL validators and unique indexes over `(tenant, source.vendor, upstream.upstream_id, upstream.content_hash)`.
|
||||
- Implement write interceptors rejecting forbidden fields, missing provenance, or merge attempts.
|
||||
- Deterministically compute linksets and persist canonical JSON payloads.
|
||||
- Introduce `/ingest/advisory`, `/advisories/raw*`, and `/aoc/verify` surfaces guarded by `advisory:*` and `aoc:verify` scopes.
|
||||
- Emit guard metrics/traces and surface supersedes/violation audit logs.
|
||||
- **Excititor (shared ingestion contract)**
|
||||
- Mirror Concelier guard and schema changes for `vex_raw`.
|
||||
- Maintain restart-time plug-in determinism and linkset extraction parity.
|
||||
- **Shared libraries**
|
||||
- Publish `StellaOps.Ingestion.AOC` (forbidden key catalog, guard middleware, provenance helpers, signature verification).
|
||||
- Share error codes (`ERR_AOC_00x`) and deterministic hashing utilities.
|
||||
- **Policy Engine integration**
|
||||
- Enforce `effective_finding_*` write exclusivity.
|
||||
- Consume only raw documents + linksets, removing any implicit normalisation.
|
||||
- **Authority scopes**
|
||||
- Provision `advisory:ingest|read`, `vex:ingest|read`, `aoc:verify`; propagate tenant claims to ingestion services.
|
||||
- **CLI & Console**
|
||||
- Implement `stella sources ingest --dry-run` and `stella aoc verify` (with exit codes mapped to `ERR_AOC_00x`).
|
||||
- Surface AOC dashboards, violation drill-down, and verification shortcuts in the Console.
|
||||
- **CI/CD**
|
||||
- Add Roslyn analyzer / AST lint to block forbidden writes.
|
||||
- Seed fixtures and run `stella aoc verify` against snapshots in pipeline gating.
|
||||
|
||||
## Documentation deliverables
|
||||
- Update `docs/aoc/aggregation-only-contract.md` with guard invariants, schemas, error codes, and migration guidance.
|
||||
- Refresh `docs/modules/concelier/operations/*.md` (mirror, conflict-resolution, authority audit) with validator rollouts and observability dashboards.
|
||||
- Cross-link Authority scope definitions, CLI reference, Console sources guide, and observability runbooks to the AOC guard changes.
|
||||
- Ensure Offline Kit documentation captures validator bootstrap and verify workflows.
|
||||
|
||||
## Acceptance criteria
|
||||
- PostgreSQL validators and runtime guards reject forbidden fields and missing provenance with the documented `ERR_AOC_00x` codes.
|
||||
- Linksets and supersedes chains are deterministic; rerunning ingestion over identical payloads yields byte-identical documents.
|
||||
- CLI `stella aoc verify` exits non-zero on seeded violations and zero on clean datasets; Console dashboards show real-time guard status.
|
||||
- Export Center consumes advisory datasets without relying on legacy normalised fields.
|
||||
- CI fails if lint rules detect forbidden writes or if seeded guard tests regress.
|
||||
|
||||
## Risks & mitigations
|
||||
- **Collector drift introduces new forbidden keys.** Mitigated by guard middleware + CI lint + schema validation; RFC required for linkset changes.
|
||||
- **Migration complexity from legacy normalisation.** Staged cutover with `_backup_*` copies and temporary views to keep Policy Engine parity.
|
||||
- **Performance overhead during ingest.** Guard remains O(number of keys); index review ensures insert latency stays within warm (<5 s) / cold (<30 s) targets.
|
||||
- **Tenancy leakage.** `tenant` required in schema, Authority-supplied claims enforced per request, observability alerts fire on missing tenant identifiers.
|
||||
|
||||
## Test strategy
|
||||
- **Unit**: guard rejection paths, provenance enforcement, idempotent insertions, linkset determinism.
|
||||
- **Property**: fuzz upstream payloads to guarantee no forbidden fields emerge.
|
||||
- **Integration**: batch ingest (50k advisories, mixed VEX fixtures), verifying zero guard violations and consistent supersedes.
|
||||
- **Contract**: Policy Engine consumers verify raw-only reads; Export Center consumes canonical datasets.
|
||||
- **End-to-end**: ingest/verify flow with CLI + Console actions to confirm observability and guard reporting.
|
||||
|
||||
## Definition of done
|
||||
- Validators deployed and verified in staging/offline environments.
|
||||
- Runtime guards, CLI/Console workflows, and CI linting all active.
|
||||
- Observability dashboards and runbooks updated; metrics visible.
|
||||
- Documentation updates merged; Offline Kit instructions published.
|
||||
- ./TASKS.md reflects status transitions; cross-module dependencies acknowledged in ../../TASKS.md.
|
||||
|
||||
## Readiness checkpoints (2025-11-25)
|
||||
- Sprint 110 attestation chain validated: `/internal/attestations/verify` endpoint and evidence bundle tests green (`TestResults/concelier-attestation/web.trx`, `core.trx`).
|
||||
- Link-Not-Merge cache + console consumption docs frozen (see `operations/lnm-cache-plan.md`, `operations/console-lnm-consumption.md`); cache headers remain deterministic.
|
||||
- Observation events transport reviewed; backlog guardrails and NATS/air-gap guidance updated in `operations/observation-events.md`.
|
||||
- Next gating dependency: TaskRunner contract drop (sprint 0157 blockers) before wiring approvals/pack ingest flows into Concelier.
|
||||
Reference in New Issue
Block a user