git.stella-ops.org/docs/modules/vex-lens/architecture.md

VEX Consensus Lens architecture

Based on Epic 7 – VEX Consensus Lens; consolidates trust modelling, consensus computation, and API contracts.

1) Purpose

Compute a deterministic, reproducible consensus view over multiple VEX statements for each (artifact, advisory) tuple while preserving conflicts and provenance. Output is consumed by Policy Engine, Vulnerability Explorer, Advisory AI, and Console overlays.

2) Inputs

• vex_normalized tuples emitted by Excititor (status, justification, scope, timestamp, content hash).
• Issuer trust registry (vex_issuer_registry) providing trust tier, confidence, authority scope.
• Optional runtime context (Zastava exposure) and policy precedence rules.

Provenance field mapping (new input contract)

Excititor connectors now stamp every raw VEX document with vex.provenance.* metadata. Lens ingests these keys alongside the normalized tuples:

Field	Description	Lens usage
vex.provenance.provider / providerName / providerKind	Logical issuer identity and type supplied by the connector (e.g., excititor:ubuntu, distro).	Seed issuer lookup, short-circuit Issuer Directory calls when we already trust the connector’s profile.
vex.provenance.trust.weight	Connector-provided base weight (0–1).	Multiplied by freshness decay & justification multipliers; overrides registry default.
vex.provenance.trust.tier & trust.note	Human/ops tier labels (vendor, distro-trusted, etc.) plus descriptive note.	Drives secondary sort (after timestamp) and Console labels; conflicts report per-tier deltas.
vex.provenance.cosign.*	Cosign issuer/identity pattern (+ optional Fulcio/Rekor URIs).	When present, Lens marks the statement as “cryptographically attested” and applies the higher confidence bucket immediately.
vex.provenance.pgp.fingerprints	Ordered list of PGP fingerprints used by the feed.	Enables Lens to validate deterministic fingerprint sets against Issuer Directory entries and flag mismatches in conflict summaries.

The trust engine preserves the raw metadata so downstream components can audit decisions or remap tiers without replaying ingestion.

3) Core algorithm

1. Group tuples by (artifactId, advisoryKey).
2. Sort statements by timestamp then trust tier (critical → low) then confidence.
3. Apply lattice join:
   • States: unknown < under_investigation < not_affected | affected < fixed.
   • Conflicts triggered when competing issuers disagree at the same precedence level.
4. Generate consensus record with fields:

   {
     "artifact": "pkg:rpm/redhat/openssl@3.0.9",
     "advisory": "CVE-2025-13579",
     "status": "not_affected",
     "confidence": 0.92,
     "derived_from": ["vex_raw:openvex:VEX-2025-00001:v2", "vex_raw:csaf:RHSA-2025:1234:v1"],
     "conflicts": [{"issuer":"vendor:upstream","status":"affected"}],
     "issued_at": "2025-08-30T12:05:00Z",
     "consensus_digest": "sha256:..."
   }
   

5. Persist consensus and emit vex.consensus.updated DSSE event.

Conflicts remain visible through conflicts array; Policy Engine can decide suppression strategy based on trust weights.

4) APIs

• GET /v1/vex/consensus?artifact=...&advisory=... — returns consensus record and conflict list.
• GET /v1/vex/conflicts — list outstanding conflicts with severity, trust delta, and time since disagreement.
• POST /v1/vex/trust/weights — (admin) update trust tiers/confidence (requires Authority scopes); updates propagate to recomputation jobs.
• GET /v1/vex/consensus/export — stream deterministic JSONL for Offline Kit / Export Center mirror bundles.

All responses include provenance fields (consensus_digest, derived_from, DSSE signature references) for audit.

5) Storage

• vex_consensus collection keyed by (tenant, artifactId, advisoryKey) with current consensus, metadata, conflict summary, and digests.
• vex_consensus_history append-only history to support replay and audit.
• vex_conflict_queue for unresolved conflicts requiring manual review.
• vexlens.noise_gate_raw_snapshots, vexlens.noise_gate_gated_snapshots, and vexlens.noise_gate_statistics store the live noise-gating raw inputs, persisted gated outputs, and aggregated statistics for the /api/v1/vexlens/gating/* surface.

Noise-gating runtime contract (2026-04-14)

• The owning VexLens web runtime resolves ISnapshotStore and IGatingStatisticsStore to PostgreSQL-backed implementations, not process-local in-memory stores.
• Startup schema convergence for the noise-gating tables is owned by embedded startup migration 002_noise_gating_state.sql in StellaOps.VexLens.Persistence; fresh local volumes must converge without manual SQL.
• POST /api/v1/vexlens/gating/snapshots/{snapshotId}/gate now persists the gated snapshot and statistics as part of the live request path, so later delta/statistics reads reflect durable backend state.
• The Angular production app binds the real VexLens noise-gating HTTP client through app.config.ts, so triage surfaces call /api/v1/vexlens/gating/* instead of relying on optional providers or mock helpers.

6) Recompute strategy

• Incremental updates triggered by Excititor deltas, trust registry changes, or policy precedence updates.
• Recompute jobs run via Orchestrator; deterministic ordering ensures identical results for the same input set.
• Jobs produce SRM-style manifests for recomputation verification.

7) Observability

• Metrics: vex_consensus_conflicts_total, vex_consensus_latency_seconds, vex_consensus_recompute_seconds{reason}.
• Logs: include artifactId, advisoryKey, issuer, status, trustTier.
• Traces: consensus.group, consensus.join, consensus.persist spans.
• Runbook + dashboard stub (offline import): runbooks/observability.md, runbooks/dashboards/vex-lens-observability.json.

8) Offline & export

• Bundle format: consensus.jsonl, conflicts.jsonl, manifest.json, signatures/. Each record references raw statement digests and trust metadata.
• Export Center uses the bundle for mirror profiles; CLI supports stella vex consensus export mirroring the API.

9) Advisory Gap Status (2026-03-04 Batch)

Status: implementation delivered in Sprint 305.

• Normalized status contract now exposes explicit unknown (VexStatus.Unknown) in active model paths.
• Normalizers preserve unknown semantics instead of collapsing unrecognized statuses to under_investigation:
  • OpenVEX unknown values map to unknown.
  • CycloneDX unknown analysis.state maps to unknown with warning WARN_CDX_008.
  • CSAF explicit unknown product status categories (known_unknown, unknown) map to unknown.
• Consensus merge precedence is deterministic with explicit tie-breaks:
  • trust weight desc
  • statement timestamp desc
  • lexical source id asc
  • statement id asc
• Unresolvable ties now remain explicit unknown with indeterminate outcome and zero confidence.
• Projection storage/list/history ordering includes deterministic secondary keys for equal timestamps in both in-memory and Postgres paths.
• Projection API contracts include unknown audit fields (unknownRationale, unknownProvenanceTrace) for summary/detail responses.

Closure sprint:

• docs/implplan/SPRINT_20260304_305_VexLens_unknown_lifecycle_and_merge_determinism.md