17d45a6d30
- Added `FilesystemPackRunProvenanceWriter` to write provenance manifests to the filesystem. - Introduced `MongoPackRunArtifactReader` to read artifacts from MongoDB. - Created `MongoPackRunProvenanceWriter` to store provenance manifests in MongoDB. - Developed unit tests for filesystem and MongoDB provenance writers. - Established `ITimelineEventStore` and `ITimelineIngestionService` interfaces for timeline event handling. - Implemented `TimelineIngestionService` to validate and persist timeline events with hashing. - Created PostgreSQL schema and migration scripts for timeline indexing. - Added dependency injection support for timeline indexer services. - Developed tests for timeline ingestion and schema validation.
85 lines
5.1 KiB
Markdown
85 lines
5.1 KiB
Markdown
# VEX Consensus Lens architecture
|
||
|
||
> Based on Epic 7 – VEX Consensus Lens; consolidates trust modelling, consensus computation, and API contracts.
|
||
|
||
## 1) Purpose
|
||
|
||
Compute a deterministic, reproducible consensus view over multiple VEX statements for each `(artifact, advisory)` tuple while preserving conflicts and provenance. Output is consumed by Policy Engine, Vulnerability Explorer, Advisory AI, and Console overlays.
|
||
|
||
## 2) Inputs
|
||
|
||
- `vex_normalized` tuples emitted by Excititor (status, justification, scope, timestamp, content hash).
|
||
- Issuer trust registry (`vex_issuer_registry`) providing trust tier, confidence, authority scope.
|
||
- Optional runtime context (Zastava exposure) and policy precedence rules.
|
||
|
||
### Provenance field mapping (new input contract)
|
||
|
||
Excititor connectors now stamp every raw VEX document with `vex.provenance.*` metadata. Lens ingests these keys alongside the normalized tuples:
|
||
|
||
| Field | Description | Lens usage |
|
||
| --- | --- | --- |
|
||
| `vex.provenance.provider` / `providerName` / `providerKind` | Logical issuer identity and type supplied by the connector (e.g., `excititor:ubuntu`, `distro`). | Seed issuer lookup, short-circuit Issuer Directory calls when we already trust the connector’s profile. |
|
||
| `vex.provenance.trust.weight` | Connector-provided base weight (0–1). | Multiplied by freshness decay & justification multipliers; overrides registry default. |
|
||
| `vex.provenance.trust.tier` & `trust.note` | Human/ops tier labels (`vendor`, `distro-trusted`, etc.) plus descriptive note. | Drives secondary sort (after timestamp) and Console labels; conflicts report per-tier deltas. |
|
||
| `vex.provenance.cosign.*` | Cosign issuer/identity pattern (+ optional Fulcio/Rekor URIs). | When present, Lens marks the statement as “cryptographically attested” and applies the higher confidence bucket immediately. |
|
||
| `vex.provenance.pgp.fingerprints` | Ordered list of PGP fingerprints used by the feed. | Enables Lens to validate deterministic fingerprint sets against Issuer Directory entries and flag mismatches in conflict summaries. |
|
||
|
||
The trust engine preserves the raw metadata so downstream components can audit decisions or remap tiers without replaying ingestion.
|
||
|
||
## 3) Core algorithm
|
||
|
||
1. Group tuples by `(artifactId, advisoryKey)`.
|
||
2. Sort statements by `timestamp` then trust tier (critical → low) then confidence.
|
||
3. Apply lattice join:
|
||
- States: `unknown < under_investigation < not_affected | affected < fixed`.
|
||
- Conflicts triggered when competing issuers disagree at the same precedence level.
|
||
4. Generate consensus record with fields:
|
||
```json
|
||
{
|
||
"artifact": "pkg:rpm/redhat/openssl@3.0.9",
|
||
"advisory": "CVE-2025-13579",
|
||
"status": "not_affected",
|
||
"confidence": 0.92,
|
||
"derived_from": ["vex_raw:openvex:VEX-2025-00001:v2", "vex_raw:csaf:RHSA-2025:1234:v1"],
|
||
"conflicts": [{"issuer":"vendor:upstream","status":"affected"}],
|
||
"issued_at": "2025-08-30T12:05:00Z",
|
||
"consensus_digest": "sha256:..."
|
||
}
|
||
```
|
||
5. Persist consensus and emit `vex.consensus.updated` DSSE event.
|
||
|
||
Conflicts remain visible through `conflicts` array; Policy Engine can decide suppression strategy based on trust weights.
|
||
|
||
## 4) APIs
|
||
|
||
- `GET /v1/vex/consensus?artifact=...&advisory=...` — returns consensus record and conflict list.
|
||
- `GET /v1/vex/conflicts` — list outstanding conflicts with severity, trust delta, and time since disagreement.
|
||
- `POST /v1/vex/trust/weights` — (admin) update trust tiers/confidence (requires Authority scopes); updates propagate to recomputation jobs.
|
||
- `GET /v1/vex/consensus/export` — stream deterministic JSONL for Offline Kit / Export Center mirror bundles.
|
||
|
||
All responses include provenance fields (`consensus_digest`, `derived_from`, DSSE signature references) for audit.
|
||
|
||
## 5) Storage
|
||
|
||
- `vex_consensus` collection keyed by `(tenant, artifactId, advisoryKey)` with current consensus, metadata, conflict summary, and digests.
|
||
- `vex_consensus_history` append-only history to support replay and audit.
|
||
- `vex_conflict_queue` for unresolved conflicts requiring manual review.
|
||
|
||
## 6) Recompute strategy
|
||
|
||
- Incremental updates triggered by Excititor deltas, trust registry changes, or policy precedence updates.
|
||
- Recompute jobs run via Orchestrator; deterministic ordering ensures identical results for the same input set.
|
||
- Jobs produce SRM-style manifests for recomputation verification.
|
||
|
||
## 7) Observability
|
||
|
||
- Metrics: `vex_consensus_conflicts_total`, `vex_consensus_latency_seconds`, `vex_consensus_recompute_seconds{reason}`.
|
||
- Logs: include `artifactId`, `advisoryKey`, `issuer`, `status`, `trustTier`.
|
||
- Traces: `consensus.group`, `consensus.join`, `consensus.persist` spans.
|
||
- Runbook + dashboard stub (offline import): `runbooks/observability.md`, `runbooks/dashboards/vex-lens-observability.json`.
|
||
|
||
## 8) Offline & export
|
||
|
||
- Bundle format: `consensus.jsonl`, `conflicts.jsonl`, `manifest.json`, `signatures/`. Each record references raw statement digests and trust metadata.
|
||
- Export Center uses the bundle for mirror profiles; CLI supports `stella vex consensus export` mirroring the API.
|