git.stella-ops.org/docs/modules/concelier/implementation_plan.md

Implementation plan — Concelier

Delivery timeline

• Phase 1 — Guardrails & schema
  Stand up Mongo JSON validators for advisory_raw and vex_raw, wire the AOCWriteGuard repository interceptor, and seed deterministic linkset builders. Freeze legacy normalisation paths and migrate callers to the new raw schema.
• Phase 2 — API & observability
  Publish ingestion and verification endpoints (POST /ingest/*, GET /advisories.raw, POST /aoc/verify) with Authority scopes, expose telemetry (aoc_violation_total, guard spans, structured logs), and ensure Offline Kit packaging captures validator deployment steps.
• Phase 3 — Experience polish
  Ship CLI/Console affordances (stella sources ingest --dry-run, dashboard tiles, violation drill-downs), finish Export Center hand-off metadata, and close out CI enforcement (stella aoc verify preflight, AST lint, seeded fixtures).

Work breakdown by component

• Concelier WebService & worker
  • Add Mongo validators and unique indexes over (tenant, source.vendor, upstream.upstream_id, upstream.content_hash).
  • Implement write interceptors rejecting forbidden fields, missing provenance, or merge attempts.
  • Deterministically compute linksets and persist canonical JSON payloads.
  • Introduce /ingest/advisory, /advisories/raw*, and /aoc/verify surfaces guarded by advisory:* and aoc:verify scopes.
  • Emit guard metrics/traces and surface supersedes/violation audit logs.
• Excititor (shared ingestion contract)
  • Mirror Concelier guard and schema changes for vex_raw.
  • Maintain restart-time plug-in determinism and linkset extraction parity.
• Shared libraries
  • Publish StellaOps.Ingestion.AOC (forbidden key catalog, guard middleware, provenance helpers, signature verification).
  • Share error codes (ERR_AOC_00x) and deterministic hashing utilities.
• Policy Engine integration
  • Enforce effective_finding_* write exclusivity.
  • Consume only raw documents + linksets, removing any implicit normalisation.
• Authority scopes
  • Provision advisory:ingest|read, vex:ingest|read, aoc:verify; propagate tenant claims to ingestion services.
• CLI & Console
  • Implement stella sources ingest --dry-run and stella aoc verify (with exit codes mapped to ERR_AOC_00x).
  • Surface AOC dashboards, violation drill-down, and verification shortcuts in the Console.
• CI/CD
  • Add Roslyn analyzer / AST lint to block forbidden writes.
  • Seed fixtures and run stella aoc verify against snapshots in pipeline gating.

Documentation deliverables

• Update docs/ingestion/aggregation-only-contract.md with guard invariants, schemas, error codes, and migration guidance.
• Refresh docs/modules/concelier/operations/*.md (mirror, conflict-resolution, authority audit) with validator rollouts and observability dashboards.
• Cross-link Authority scope definitions, CLI reference, Console sources guide, and observability runbooks to the AOC guard changes.
• Ensure Offline Kit documentation captures validator bootstrap and verify workflows.

Acceptance criteria

• Mongo validators and runtime guards reject forbidden fields and missing provenance with the documented ERR_AOC_00x codes.
• Linksets and supersedes chains are deterministic; rerunning ingestion over identical payloads yields byte-identical documents.
• CLI stella aoc verify exits non-zero on seeded violations and zero on clean datasets; Console dashboards show real-time guard status.
• Export Center consumes advisory datasets without relying on legacy normalised fields.
• CI fails if lint rules detect forbidden writes or if seeded guard tests regress.

Risks & mitigations

• Collector drift introduces new forbidden keys. Mitigated by guard middleware + CI lint + schema validation; RFC required for linkset changes.
• Migration complexity from legacy normalisation. Staged cutover with _backup_* copies and temporary views to keep Policy Engine parity.
• Performance overhead during ingest. Guard remains O(number of keys); index review ensures insert latency stays within warm (<5 s) / cold (<30 s) targets.
• Tenancy leakage. tenant required in schema, Authority-supplied claims enforced per request, observability alerts fire on missing tenant identifiers.

Test strategy

• Unit: guard rejection paths, provenance enforcement, idempotent insertions, linkset determinism.
• Property: fuzz upstream payloads to guarantee no forbidden fields emerge.
• Integration: batch ingest (50k advisories, mixed VEX fixtures), verifying zero guard violations and consistent supersedes.
• Contract: Policy Engine consumers verify raw-only reads; Export Center consumes canonical datasets.
• End-to-end: ingest/verify flow with CLI + Console actions to confirm observability and guard reporting.

Definition of done

• Validators deployed and verified in staging/offline environments.
• Runtime guards, CLI/Console workflows, and CI linting all active.
• Observability dashboards and runbooks updated; metrics visible.
• Documentation updates merged; Offline Kit instructions published.
• ./TASKS.md reflects status transitions; cross-module dependencies acknowledged in ../../TASKS.md.