# VEX Observation Model (`vex_observations`)

> Authored 2025-11-14 for Sprint 120 (`EXCITITOR-LNM-21-001`). This document is the canonical schema description for Excititor’s immutable observation records. It unblocks downstream documentation tasks (`DOCS-LNM-22-002`) and aligns the WebService/Worker data structures with Mongo persistence.

Excititor ingests heterogeneous VEX statements, normalizes them under the Aggregation-Only Contract (AOC), and persists each normalized statement as a **VEX observation**. These observations are the source of truth for:

- Advisory AI citation APIs (`/v1/vex/observations/{vulnerabilityId}/{productKey}`)
- Graph/Vuln Explorer overlays (batch observation APIs)
- Evidence Locker + portable bundle manifests
- Policy Engine materialization and audit trails

All observation documents are immutable. New information creates a new observation record linked by `observationId`; supersedence happens through Graph/Lens layers, not by mutating this collection.

## Storage & routing

| Aspect | Value |
| --- | --- |
| Collection | `vex_observations` (Mongo) |
| Upstream generator | `VexObservationProjectionService` (WebService) and Worker normalization pipeline |
| Primary key | `{tenant, observationId}` |
| Required indexes | `{tenant, vulnerabilityId}`, `{tenant, productKey}`, `{tenant, document.digest}`, `{tenant, providerId, status}` |
| Source of truth for | `/v1/vex/observations`, Graph batch APIs, Excititor → Evidence Locker replication |

## Canonical document shape

```jsonc
{
  "tenant": "default",
  "observationId": "vex:obs:sha256:...",
  "vulnerabilityId": "CVE-2024-12345",
  "productKey": "pkg:maven/org.example/app@1.2.3",
  "providerId": "ubuntu-csaf",
  "status": "affected",                // matches VexClaimStatus enum
  "justification": {
    "type": "component_not_present",
    "reason": "Package not shipped in this profile",
    "detail": "Binary not in base image"
  },
  "detail": "Free-form vendor detail",
  "confidence": {
    "score": 0.9,
    "level": "high",
    "method": "vendor"
  },
  "signals": {
    "severity": {
      "scheme": "cvss3.1",
      "score": 7.8,
      "label": "High",
      "vector": "CVSS:3.1/..."
    },
    "kev": true,
    "epss": 0.77
  },
  "scope": {
    "key": "pkg:deb/ubuntu/apache2@2.4.58-1",
    "purls": [
      "pkg:deb/ubuntu/apache2@2.4.58-1",
      "pkg:docker/example/app@sha256:..."
    ],
    "cpes": ["cpe:2.3:a:apache:http_server:2.4.58:*:*:*:*:*:*:*"]
  },
  "anchors": [
    "#/statements/0/justification",
    "#/statements/0/detail"
  ],
  "document": {
    "format": "csaf",
    "digest": "sha256:abc123...",
    "revision": "2024-10-22T09:00:00Z",
    "sourceUri": "https://ubuntu.com/security/notices/USN-0000-1",
    "signature": {
      "type": "cosign",
      "issuer": "https://token.actions.githubusercontent.com",
      "keyId": "ubuntu-vex-prod",
      "verifiedAt": "2024-10-22T09:01:00Z",
      "transparencyLogReference": "rekor://UUID",
      "trust": {
        "tenantId": "default",
        "issuerId": "ubuntu",
        "effectiveWeight": 0.9,
        "tenantOverrideApplied": false,
        "retrievedAtUtc": "2024-10-22T09:00:30Z"
      }
    }
  },
  "aoc": {
    "guardVersion": "2024.10.0",
    "violations": [],                    // non-empty -> stored + surfaced
    "ingestedAt": "2024-10-22T09:00:05Z",
    "retrievedAt": "2024-10-22T08:59:59Z"
  },
  "metadata": {
    "provider-hint": "Mainline feed",
    "source-channel": "mirror"
  }
}
```

### Field notes

- **`tenant`** – logical tenant resolved by WebService based on headers or default configuration.
- **`observationId`** – deterministic hash (sha256) over `{tenant, vulnerabilityId, productKey, providerId, statementDigest}`. Never reused.
- **`status` + `justification`** – follow the OpenVEX semantics enforced by `StellaOps.Excititor.Core.VexClaim`.
- **`scope`** – includes canonical `key` plus normalized PURLs/CPES; deterministic ordering.
- **`anchors`** – optional JSON-pointer hints pointing to the source document sections; stored as trimmed strings.
- **`document.signature`** – mirrors `VexSignatureMetadata`; empty if upstream feed lacks signatures.
- **`aoc.violations`** – stored if the guard detected non-fatal issues; fatal issues never create an observation.
- **`metadata`** – reserved for deterministic provider hints; keys follow `vex.*` prefix guidance.

## Determinism & AOC guarantees

1. **Write-once** – once inserted, observation documents never change. New evidence creates a new `observationId`.
2. **Sorted collections** – arrays (`anchors`, `purls`, `cpes`) are sorted lexicographically before persistence.
3. **Guard metadata** – `aoc.guardVersion` records the guard library version (`docs/aoc/guard-library.md`), enabling audits.
4. **Signatures** – only verification metadata proven by the Worker is stored; WebService never recomputes trust.
5. **Time normalization** – all timestamps stored as UTC ISO-8601 strings (Mongo `DateTime`).

## API mapping

| API | Source fields | Notes |
| --- | --- | --- |
| `GET /vex/observations` | `tenant`, `vulnerabilityId`, `productKey`, `providerId` | List observations with filters. Implemented in `ObservationEndpoints.cs`. |
| `GET /vex/observations/{observationId}` | `tenant`, `observationId` | Get single observation by ID with full detail. |
| `GET /vex/observations/count` | `tenant` | Count all observations for tenant. |
| `/v1/vex/observations/{vuln}/{product}` | `tenant`, `vulnerabilityId`, `productKey`, `scope`, `statements[]` | Response uses `VexObservationProjectionService` to render `statements`, `document`, and `signature` fields. |
| `/vex/aoc/verify` | `document.digest`, `providerId`, `aoc` | Replays guard validation for recent digests; guard violations here align with `aoc.violations`. |
| Evidence batch API (Graph) | `statements[]`, `scope`, `signals`, `anchors` | Format optimized for overlays; reduces `document` to digest/URI. |

## Related work

- `EXCITITOR-GRAPH-24-*` relies on this schema to build overlays.
- `DOCS-LNM-22-002` (Link-Not-Merge documentation) references this file.
- `EXCITITOR-ATTEST-73-*` uses `document.digest` + `signature` to embed provenance in attestation payloads.