feat(docs): Add comprehensive documentation for Vexer, Vulnerability Explorer, and Zastava modules

- Introduced AGENTS.md, README.md, TASKS.md, and implementation_plan.md for Vexer, detailing mission, responsibilities, key components, and operational notes. - Established similar documentation structure for Vulnerability Explorer and Zastava modules, including their respective workflows, integrations, and observability notes. - Created risk scoring profiles documentation outlining the core workflow, factor model, governance, and deliverables. - Ensured all modules adhere to the Aggregation-Only Contract and maintain determinism and provenance in outputs.
2025-10-30 00:09:39 +02:00
parent 86f606a115
commit e8537460a3
503 changed files with 16136 additions and 54638 deletions
--- a/docs/modules/vex-lens/AGENTS.md
+++ b/docs/modules/vex-lens/AGENTS.md
@@ -0,0 +1,22 @@
+# VEX Consensus Lens agent guide
+
+## Mission
+VEX Lens computes deterministic consensus across conflicting VEX statements while preserving raw provenance.
+
+## Key docs
+- [Module README](./README.md)
+- [Architecture](./architecture.md)
+- [Implementation plan](./implementation_plan.md)
+- [Task board](./TASKS.md)
+
+## How to get started
+1. Review ./architecture.md for consensus algorithm, trust model, and export contracts.
+2. Open ../../implplan/SPRINTS.md and locate stories for this component.
+3. Check ./TASKS.md and update status before/after work.
+4. Read README/architecture for design context and update as the implementation evolves.
+
+## Guardrails
+- Uphold Aggregation-Only Contract boundaries when consuming ingestion data.
+- Preserve determinism and provenance in all derived outputs.
+- Document offline/air-gap pathways for any new feature.
+- Update telemetry/observability assets alongside feature work.
--- a/docs/modules/vex-lens/README.md
+++ b/docs/modules/vex-lens/README.md
@@ -0,0 +1,28 @@
+# StellaOps VEX Consensus Lens
+
+VEX Lens computes deterministic consensus across conflicting VEX statements while preserving raw provenance.
+
+## Responsibilities
+- Ingest VEX evidence from Excititor and align it to SBOM inventory.
+- Apply issuer trust weights, freshness rules, and policy-defined tie breakers.
+- Publish consensus snapshots and disagreement metadata for Policy Engine and Explorer surfaces.
+- Expose APIs for explainability and offline bundle exports.
+
+## Key components
+- Consensus computation service and job pipeline.
+- Consensus store with versioned snapshots.
+- Explain trace generator for disagreements.
+
+## Integrations & dependencies
+- Excititor for raw VEX ingestion.
+- Policy Engine for applying consensus in suppression flows.
+- Vulnerability Explorer and Advisory AI for evidence overlays.
+
+## Operational notes
+- Trust model configuration and issuer scoring dashboards.
+- Offline kit packaging of consensus snapshots.
+- Telemetry on issuer coverage and disagreement counts.
+
+## Epic alignment
+- Epic 7: VEX Consensus Lens.
+- Lens implementation stories tracked in ../../TASKS.md.
--- a/docs/modules/vex-lens/TASKS.md
+++ b/docs/modules/vex-lens/TASKS.md
@@ -0,0 +1,9 @@
+# Task board — VEX Consensus Lens
+
+> Local tasks should link back to ./AGENTS.md and mirror status updates into ../../TASKS.md when applicable.
+
+| ID | Status | Owner(s) | Description | Notes |
+|----|--------|----------|-------------|-------|
+| VEX-CONSENSUS-LENS-DOCS-0001 | DOING (2025-10-29) | Docs Guild | Ensure ./README.md reflects the latest epic deliverables. | Align with ./AGENTS.md |
+| VEX-CONSENSUS-LENS-ENG-0001 | TODO | Module Team | Break down epic milestones into actionable stories. | Sync into ../../TASKS.md |
+| VEX-CONSENSUS-LENS-OPS-0001 | TODO | Ops Guild | Prepare runbooks/observability assets once MVP lands. | Document outputs in ./README.md |
--- a/docs/modules/vex-lens/architecture.md
+++ b/docs/modules/vex-lens/architecture.md
@@ -0,0 +1,69 @@
+# VEX Consensus Lens architecture
+
+> Based on Epic 7 – VEX Consensus Lens; consolidates trust modelling, consensus computation, and API contracts.
+
+## 1) Purpose
+
+Compute a deterministic, reproducible consensus view over multiple VEX statements for each `(artifact, advisory)` tuple while preserving conflicts and provenance. Output is consumed by Policy Engine, Vulnerability Explorer, Advisory AI, and Console overlays.
+
+## 2) Inputs
+
+- `vex_normalized` tuples emitted by Excititor (status, justification, scope, timestamp, content hash).
+- Issuer trust registry (`vex_issuer_registry`) providing trust tier, confidence, authority scope.
+- Optional runtime context (Zastava exposure) and policy precedence rules.
+
+## 3) Core algorithm
+
+1. Group tuples by `(artifactId, advisoryKey)`.
+2. Sort statements by `timestamp` then trust tier (critical → low) then confidence.
+3. Apply lattice join:
+   - States: `unknown < under_investigation < not_affected | affected < fixed`.
+   - Conflicts triggered when competing issuers disagree at the same precedence level.
+4. Generate consensus record with fields:
+   ```json
+   {
+     "artifact": "pkg:rpm/redhat/openssl@3.0.9",
+     "advisory": "CVE-2025-13579",
+     "status": "not_affected",
+     "confidence": 0.92,
+     "derived_from": ["vex_raw:openvex:VEX-2025-00001:v2", "vex_raw:csaf:RHSA-2025:1234:v1"],
+     "conflicts": [{"issuer":"vendor:upstream","status":"affected"}],
+     "issued_at": "2025-08-30T12:05:00Z",
+     "consensus_digest": "sha256:..."
+   }
+   ```
+5. Persist consensus and emit `vex.consensus.updated` DSSE event.
+
+Conflicts remain visible through `conflicts` array; Policy Engine can decide suppression strategy based on trust weights.
+
+## 4) APIs
+
+- `GET /v1/vex/consensus?artifact=...&advisory=...` — returns consensus record and conflict list.
+- `GET /v1/vex/conflicts` — list outstanding conflicts with severity, trust delta, and time since disagreement.
+- `POST /v1/vex/trust/weights` — (admin) update trust tiers/confidence (requires Authority scopes); updates propagate to recomputation jobs.
+- `GET /v1/vex/consensus/export` — stream deterministic JSONL for Offline Kit / Export Center mirror bundles.
+
+All responses include provenance fields (`consensus_digest`, `derived_from`, DSSE signature references) for audit.
+
+## 5) Storage
+
+- `vex_consensus` collection keyed by `(tenant, artifactId, advisoryKey)` with current consensus, metadata, conflict summary, and digests.
+- `vex_consensus_history` append-only history to support replay and audit.
+- `vex_conflict_queue` for unresolved conflicts requiring manual review.
+
+## 6) Recompute strategy
+
+- Incremental updates triggered by Excititor deltas, trust registry changes, or policy precedence updates.
+- Recompute jobs run via Orchestrator; deterministic ordering ensures identical results for the same input set.
+- Jobs produce SRM-style manifests for recomputation verification.
+
+## 7) Observability
+
+- Metrics: `vex_consensus_conflicts_total`, `vex_consensus_latency_seconds`, `vex_consensus_recompute_seconds{reason}`.
+- Logs: include `artifactId`, `advisoryKey`, `issuer`, `status`, `trustTier`.
+- Traces: `consensus.group`, `consensus.join`, `consensus.persist` spans.
+
+## 8) Offline & export
+
+- Bundle format: `consensus.jsonl`, `conflicts.jsonl`, `manifest.json`, `signatures/`. Each record references raw statement digests and trust metadata.
+- Export Center uses the bundle for mirror profiles; CLI supports `stella vex consensus export` mirroring the API.
--- a/docs/modules/vex-lens/implementation_plan.md
+++ b/docs/modules/vex-lens/implementation_plan.md
@@ -0,0 +1,63 @@
+# Implementation plan — VEX Consensus Lens
+
+## Delivery phases
+- **Phase 1 – Core lens service**  
+  Build normalisation pipeline (CSAF/OpenVEX/CycloneDX), product mapping library, trust weighting functions, consensus algorithm, and persistence (`vex_consensus`, history, conflicts).
+- **Phase 2 – API & integrations**  
+  Expose `/vex/consensus` query/detail/simulate/export endpoints, integrate Policy Engine thresholds, Vuln Explorer UI chips, and VEX Lens change events.
+- **Phase 3 – Issuer Directory & signatures**  
+  Deliver issuer registry, key management, signature verification, RBAC, audit logs, and tenant overrides.
+- **Phase 4 – Console & CLI experiences**  
+  Ship Console module (lists, evidence table, quorum bar, conflicts, simulation drawer) and CLI commands (`stella vex consensus ...`) with export support.
+- **Phase 5 – Recompute & performance**  
+  Implement recompute scheduling (policy activation, Excitator deltas), caching, load tests (10M records/tenant), observability dashboards, and Offline Kit exports.
+
+## Work breakdown
+- **VEX Lens service**
+  - Normalise VEX payloads, maintain scope scores, compute consensus digest.
+  - Trust weighting functions (issuer tier, freshness decay, scope quality).
+  - Idempotent workers for consensus projection and history tracking.
+  - Conflict handling queue for manual review and notifications.
+- **Integrations**
+  - Excitator: enrich VEX events with issuer hints, signatures, product trees.
+  - Policy Engine: trust knobs, simulation endpoints, policy-driven recompute.
+  - Vuln Explorer & Advisory AI: consensus badges, conflict surfacing.
+- **Issuer Directory**
+  - CRUD for issuers/keys, audit logs, import CSAF publishers, tenant overrides.
+  - Signature verification endpoints consumed by Lens.
+- **APIs & UX**
+  - REST endpoints for query/detail/conflict export, trust weight updates.
+  - Console module with filters, saved views, evidence table, simulation drawer.
+  - CLI commands for list/show/simulate/export with JSON/CSV output.
+- **Observability & Ops**
+  - Metrics (consensus latency, conflict rate, signature failures, cache hit rate), logs, traces.
+  - Dashboards + runbooks for recompute storms, mapping failures, signature errors, quota breaches.
+  - Offline exports for Export Center/Offline Kit.
+
+## Acceptance criteria
+- Consensus results reproducible across supported VEX formats with deterministic digests and provenance.
+- Signature verification influences trust weights; unverifiable evidence is down-weighted without pipeline failure.
+- Policy simulations show quorum shifts without persisting state; Vuln Explorer consumes consensus signals.
+- Issuer Directory enforces RBAC, audit logs, and key rotation; CLI & Console parity achieved.
+- Recompute pipeline handles Excitator deltas and policy activations with backpressure and incident surfacing.
+- Observability dashboards/alerts cover ingestion lag, conflict spikes, signature failures, performance budgets (P95 < 500 ms for 100-row pages at 10M records/tenant).
+
+## Risks & mitigations
+- **Product mapping ambiguity:** conservative scope scoring, manual overrides, surfaced warnings, policy review hooks.
+- **Issuer compromise:** signature verification, trust weighting, tenant overrides, revocation runbooks.
+- **Evidence storms:** batching, worker sharding, orchestrator rate limiting, priority queues.
+- **Performance degradation:** caching, indexing, load tests, quota enforcement.
+- **Offline gaps:** deterministic exports, manifest hashes, Offline Kit tests.
+
+## Test strategy
+- **Unit:** normalisers, mapping, trust weights, consensus lattice, signature verification.
+- **Property:** randomised evidence sets verifying lattice commutativity and determinism.
+- **Integration:** Excitator → Lens → Policy/Vuln Explorer flow, issuer overrides, simulation.
+- **Performance:** large tenant datasets, cache behaviour, concurrency tests.
+- **Security:** RBAC, tenant scoping, signature tampering, issuer revocation.
+- **Offline:** export/import verification, CLI parity.
+
+## Definition of done
+- Lens service, issuer directory, API/CLI/Console components deployed with telemetry and runbooks.
+- Documentation set (overview, algorithm, issuer directory, API, console, policy trust) updated with imposed rule statements.
+- ./TASKS.md and ../../TASKS.md reflect current status; Offline Kit parity confirmed.