master
7b5bdcf4d3
- Introduced AGENTS.md, README.md, TASKS.md, and implementation_plan.md for Vexer, detailing mission, responsibilities, key components, and operational notes. - Established similar documentation structure for Vulnerability Explorer and Zastava modules, including their respective workflows, integrations, and observability notes. - Created risk scoring profiles documentation outlining the core workflow, factor model, governance, and deliverables. - Ensured all modules adhere to the Aggregation-Only Contract and maintain determinism and provenance in outputs.
68 lines
5.2 KiB
Markdown
68 lines
5.2 KiB
Markdown
# Implementation plan — Concelier
|
||
|
||
## Delivery timeline
|
||
- **Phase 1 — Guardrails & schema**
|
||
Stand up Mongo JSON validators for `advisory_raw` and `vex_raw`, wire the `AOCWriteGuard` repository interceptor, and seed deterministic linkset builders. Freeze legacy normalisation paths and migrate callers to the new raw schema.
|
||
- **Phase 2 — API & observability**
|
||
Publish ingestion and verification endpoints (`POST /ingest/*`, `GET /advisories.raw`, `POST /aoc/verify`) with Authority scopes, expose telemetry (`aoc_violation_total`, guard spans, structured logs), and ensure Offline Kit packaging captures validator deployment steps.
|
||
- **Phase 3 — Experience polish**
|
||
Ship CLI/Console affordances (`stella sources ingest --dry-run`, dashboard tiles, violation drill-downs), finish Export Center hand-off metadata, and close out CI enforcement (`stella aoc verify` preflight, AST lint, seeded fixtures).
|
||
|
||
## Work breakdown by component
|
||
- **Concelier WebService & worker**
|
||
- Add Mongo validators and unique indexes over `(tenant, source.vendor, upstream.upstream_id, upstream.content_hash)`.
|
||
- Implement write interceptors rejecting forbidden fields, missing provenance, or merge attempts.
|
||
- Deterministically compute linksets and persist canonical JSON payloads.
|
||
- Introduce `/ingest/advisory`, `/advisories/raw*`, and `/aoc/verify` surfaces guarded by `advisory:*` and `aoc:verify` scopes.
|
||
- Emit guard metrics/traces and surface supersedes/violation audit logs.
|
||
- **Excititor (shared ingestion contract)**
|
||
- Mirror Concelier guard and schema changes for `vex_raw`.
|
||
- Maintain restart-time plug-in determinism and linkset extraction parity.
|
||
- **Shared libraries**
|
||
- Publish `StellaOps.Ingestion.AOC` (forbidden key catalog, guard middleware, provenance helpers, signature verification).
|
||
- Share error codes (`ERR_AOC_00x`) and deterministic hashing utilities.
|
||
- **Policy Engine integration**
|
||
- Enforce `effective_finding_*` write exclusivity.
|
||
- Consume only raw documents + linksets, removing any implicit normalisation.
|
||
- **Authority scopes**
|
||
- Provision `advisory:ingest|read`, `vex:ingest|read`, `aoc:verify`; propagate tenant claims to ingestion services.
|
||
- **CLI & Console**
|
||
- Implement `stella sources ingest --dry-run` and `stella aoc verify` (with exit codes mapped to `ERR_AOC_00x`).
|
||
- Surface AOC dashboards, violation drill-down, and verification shortcuts in the Console.
|
||
- **CI/CD**
|
||
- Add Roslyn analyzer / AST lint to block forbidden writes.
|
||
- Seed fixtures and run `stella aoc verify` against snapshots in pipeline gating.
|
||
|
||
## Documentation deliverables
|
||
- Update `docs/ingestion/aggregation-only-contract.md` with guard invariants, schemas, error codes, and migration guidance.
|
||
- Refresh `docs/modules/concelier/operations/*.md` (mirror, conflict-resolution, authority audit) with validator rollouts and observability dashboards.
|
||
- Cross-link Authority scope definitions, CLI reference, Console sources guide, and observability runbooks to the AOC guard changes.
|
||
- Ensure Offline Kit documentation captures validator bootstrap and verify workflows.
|
||
|
||
## Acceptance criteria
|
||
- Mongo validators and runtime guards reject forbidden fields and missing provenance with the documented `ERR_AOC_00x` codes.
|
||
- Linksets and supersedes chains are deterministic; rerunning ingestion over identical payloads yields byte-identical documents.
|
||
- CLI `stella aoc verify` exits non-zero on seeded violations and zero on clean datasets; Console dashboards show real-time guard status.
|
||
- Export Center consumes advisory datasets without relying on legacy normalised fields.
|
||
- CI fails if lint rules detect forbidden writes or if seeded guard tests regress.
|
||
|
||
## Risks & mitigations
|
||
- **Collector drift introduces new forbidden keys.** Mitigated by guard middleware + CI lint + schema validation; RFC required for linkset changes.
|
||
- **Migration complexity from legacy normalisation.** Staged cutover with `_backup_*` copies and temporary views to keep Policy Engine parity.
|
||
- **Performance overhead during ingest.** Guard remains O(number of keys); index review ensures insert latency stays within warm (<5 s) / cold (<30 s) targets.
|
||
- **Tenancy leakage.** `tenant` required in schema, Authority-supplied claims enforced per request, observability alerts fire on missing tenant identifiers.
|
||
|
||
## Test strategy
|
||
- **Unit**: guard rejection paths, provenance enforcement, idempotent insertions, linkset determinism.
|
||
- **Property**: fuzz upstream payloads to guarantee no forbidden fields emerge.
|
||
- **Integration**: batch ingest (50k advisories, mixed VEX fixtures), verifying zero guard violations and consistent supersedes.
|
||
- **Contract**: Policy Engine consumers verify raw-only reads; Export Center consumes canonical datasets.
|
||
- **End-to-end**: ingest/verify flow with CLI + Console actions to confirm observability and guard reporting.
|
||
|
||
## Definition of done
|
||
- Validators deployed and verified in staging/offline environments.
|
||
- Runtime guards, CLI/Console workflows, and CI linting all active.
|
||
- Observability dashboards and runbooks updated; metrics visible.
|
||
- Documentation updates merged; Offline Kit instructions published.
|
||
- ./TASKS.md reflects status transitions; cross-module dependencies acknowledged in ../../TASKS.md.
|