Implement ledger metrics for observability and add tests for Ruby packages endpoints

- Added `LedgerMetrics` class to record write latency and total events for ledger operations. - Created comprehensive tests for Ruby packages endpoints, covering scenarios for missing inventory, successful retrieval, and identifier handling. - Introduced `TestSurfaceSecretsScope` for managing environment variables during tests. - Developed `ProvenanceMongoExtensions` for attaching DSSE provenance and trust information to event documents. - Implemented `EventProvenanceWriter` and `EventWriter` classes for managing event provenance in MongoDB. - Established MongoDB indexes for efficient querying of events based on provenance and trust. - Added models and JSON parsing logic for DSSE provenance and trust information.
2025-11-13 09:29:09 +02:00
parent 151f6b35cc
commit 61f963fd52
101 changed files with 5881 additions and 1776 deletions
--- a/docs/modules/excititor/vex_observations.md
+++ b/docs/modules/excititor/vex_observations.md
@@ -0,0 +1,131 @@
+# VEX Observation Model (`vex_observations`)
+
+> Authored 2025-11-14 for Sprint 120 (`EXCITITOR-LNM-21-001`). This document is the canonical schema description for Excititor’s immutable observation records. It unblocks downstream documentation tasks (`DOCS-LNM-22-002`) and aligns the WebService/Worker data structures with Mongo persistence.
+
+Excititor ingests heterogeneous VEX statements, normalizes them under the Aggregation-Only Contract (AOC), and persists each normalized statement as a **VEX observation**. These observations are the source of truth for:
+
+- Advisory AI citation APIs (`/v1/vex/observations/{vulnerabilityId}/{productKey}`)
+- Graph/Vuln Explorer overlays (batch observation APIs)
+- Evidence Locker + portable bundle manifests
+- Policy Engine materialization and audit trails
+
+All observation documents are immutable. New information creates a new observation record linked by `observationId`; supersedence happens through Graph/Lens layers, not by mutating this collection.
+
+## Storage & routing
+
+| Aspect | Value |
+| --- | --- |
+| Collection | `vex_observations` (Mongo) |
+| Upstream generator | `VexObservationProjectionService` (WebService) and Worker normalization pipeline |
+| Primary key | `{tenant, observationId}` |
+| Required indexes | `{tenant, vulnerabilityId}`, `{tenant, productKey}`, `{tenant, document.digest}`, `{tenant, providerId, status}` |
+| Source of truth for | `/v1/vex/observations`, Graph batch APIs, Excititor → Evidence Locker replication |
+
+## Canonical document shape
+
+```jsonc
+{
+  "tenant": "default",
+  "observationId": "vex:obs:sha256:...",
+  "vulnerabilityId": "CVE-2024-12345",
+  "productKey": "pkg:maven/org.example/app@1.2.3",
+  "providerId": "ubuntu-csaf",
+  "status": "affected",                // matches VexClaimStatus enum
+  "justification": {
+    "type": "component_not_present",
+    "reason": "Package not shipped in this profile",
+    "detail": "Binary not in base image"
+  },
+  "detail": "Free-form vendor detail",
+  "confidence": {
+    "score": 0.9,
+    "level": "high",
+    "method": "vendor"
+  },
+  "signals": {
+    "severity": {
+      "scheme": "cvss3.1",
+      "score": 7.8,
+      "label": "High",
+      "vector": "CVSS:3.1/..."
+    },
+    "kev": true,
+    "epss": 0.77
+  },
+  "scope": {
+    "key": "pkg:deb/ubuntu/apache2@2.4.58-1",
+    "purls": [
+      "pkg:deb/ubuntu/apache2@2.4.58-1",
+      "pkg:docker/example/app@sha256:..."
+    ],
+    "cpes": ["cpe:2.3:a:apache:http_server:2.4.58:*:*:*:*:*:*:*"]
+  },
+  "anchors": [
+    "#/statements/0/justification",
+    "#/statements/0/detail"
+  ],
+  "document": {
+    "format": "csaf",
+    "digest": "sha256:abc123...",
+    "revision": "2024-10-22T09:00:00Z",
+    "sourceUri": "https://ubuntu.com/security/notices/USN-0000-1",
+    "signature": {
+      "type": "cosign",
+      "issuer": "https://token.actions.githubusercontent.com",
+      "keyId": "ubuntu-vex-prod",
+      "verifiedAt": "2024-10-22T09:01:00Z",
+      "transparencyLogReference": "rekor://UUID",
+      "trust": {
+        "tenantId": "default",
+        "issuerId": "ubuntu",
+        "effectiveWeight": 0.9,
+        "tenantOverrideApplied": false,
+        "retrievedAtUtc": "2024-10-22T09:00:30Z"
+      }
+    }
+  },
+  "aoc": {
+    "guardVersion": "2024.10.0",
+    "violations": [],                    // non-empty -> stored + surfaced
+    "ingestedAt": "2024-10-22T09:00:05Z",
+    "retrievedAt": "2024-10-22T08:59:59Z"
+  },
+  "metadata": {
+    "provider-hint": "Mainline feed",
+    "source-channel": "mirror"
+  }
+}
+```
+
+### Field notes
+
+- **`tenant`** – logical tenant resolved by WebService based on headers or default configuration.
+- **`observationId`** – deterministic hash (sha256) over `{tenant, vulnerabilityId, productKey, providerId, statementDigest}`. Never reused.
+- **`status` + `justification`** – follow the OpenVEX semantics enforced by `StellaOps.Excititor.Core.VexClaim`.
+- **`scope`** – includes canonical `key` plus normalized PURLs/CPES; deterministic ordering.
+- **`anchors`** – optional JSON-pointer hints pointing to the source document sections; stored as trimmed strings.
+- **`document.signature`** – mirrors `VexSignatureMetadata`; empty if upstream feed lacks signatures.
+- **`aoc.violations`** – stored if the guard detected non-fatal issues; fatal issues never create an observation.
+- **`metadata`** – reserved for deterministic provider hints; keys follow `vex.*` prefix guidance.
+
+## Determinism & AOC guarantees
+
+1. **Write-once** – once inserted, observation documents never change. New evidence creates a new `observationId`.
+2. **Sorted collections** – arrays (`anchors`, `purls`, `cpes`) are sorted lexicographically before persistence.
+3. **Guard metadata** – `aoc.guardVersion` records the guard library version (`docs/aoc/guard-library.md`), enabling audits.
+4. **Signatures** – only verification metadata proven by the Worker is stored; WebService never recomputes trust.
+5. **Time normalization** – all timestamps stored as UTC ISO-8601 strings (Mongo `DateTime`).
+
+## API mapping
+
+| API | Source fields | Notes |
+| --- | --- | --- |
+| `/v1/vex/observations/{vuln}/{product}` | `tenant`, `vulnerabilityId`, `productKey`, `scope`, `statements[]` | Response uses `VexObservationProjectionService` to render `statements`, `document`, and `signature` fields. |
+| `/vex/aoc/verify` | `document.digest`, `providerId`, `aoc` | Replays guard validation for recent digests; guard violations here align with `aoc.violations`. |
+| Evidence batch API (Graph) | `statements[]`, `scope`, `signals`, `anchors` | Format optimized for overlays; resuces `document` to digest/URI. |
+
+## Related work
+
+- `EXCITITOR-GRAPH-24-*` relies on this schema to build overlays.
+- `DOCS-LNM-22-002` (Link-Not-Merge documentation) references this file.
+- `EXCITITOR-ATTEST-73-*` uses `document.digest` + `signature` to embed provenance in attestation payloads.