VEX Observation Model (`vex_observations`)

Authored 2025-11-14 for Sprint 120 (EXCITITOR-LNM-21-001). This document is the canonical schema description for Excititor's immutable observation records. It unblocks downstream documentation tasks (DOCS-LNM-22-002) and aligns the WebService/Worker data structures with PostgreSQL persistence.

Excititor ingests heterogeneous VEX statements, normalizes them under the Aggregation-Only Contract (AOC), and persists each normalized statement as a VEX observation. These observations are the source of truth for:

Advisory AI citation APIs (/v1/vex/observations/{vulnerabilityId}/{productKey})
Graph/Vuln Explorer overlays (batch observation APIs)
Evidence Locker + portable bundle manifests
Policy Engine materialization and audit trails

All observation documents are immutable. New information creates a new observation record linked by observationId; supersedence happens through Graph/Lens layers, not by mutating this collection.

Storage & routing

Aspect	Value
Table	`vex_observations` (PostgreSQL)
Upstream generator	`VexObservationProjectionService` (WebService) and Worker normalization pipeline
Primary key	`{tenant, observationId}`
Required indexes	`{tenant, vulnerabilityId}`, `{tenant, productKey}`, `{tenant, document.digest}`, `{tenant, providerId, status}`
Source of truth for	`/v1/vex/observations`, Graph batch APIs, Excititor → Evidence Locker replication

Canonical document shape

{
  "tenant": "default",
  "observationId": "vex:obs:sha256:...",
  "vulnerabilityId": "CVE-2024-12345",
  "productKey": "pkg:maven/org.example/app@1.2.3",
  "providerId": "ubuntu-csaf",
  "status": "affected",                // matches VexClaimStatus enum
  "justification": {
    "type": "component_not_present",
    "reason": "Package not shipped in this profile",
    "detail": "Binary not in base image"
  },
  "detail": "Free-form vendor detail",
  "confidence": {
    "score": 0.9,
    "level": "high",
    "method": "vendor"
  },
  "signals": {
    "severity": {
      "scheme": "cvss3.1",
      "score": 7.8,
      "label": "High",
      "vector": "CVSS:3.1/..."
    },
    "kev": true,
    "epss": 0.77
  },
  "scope": {
    "key": "pkg:deb/ubuntu/apache2@2.4.58-1",
    "purls": [
      "pkg:deb/ubuntu/apache2@2.4.58-1",
      "pkg:docker/example/app@sha256:..."
    ],
    "cpes": ["cpe:2.3:a:apache:http_server:2.4.58:*:*:*:*:*:*:*"]
  },
  "anchors": [
    "#/statements/0/justification",
    "#/statements/0/detail"
  ],
  "document": {
    "format": "csaf",
    "digest": "sha256:abc123...",
    "revision": "2024-10-22T09:00:00Z",
    "sourceUri": "https://ubuntu.com/security/notices/USN-0000-1",
    "signature": {
      "type": "cosign",
      "issuer": "https://token.actions.githubusercontent.com",
      "keyId": "ubuntu-vex-prod",
      "verifiedAt": "2024-10-22T09:01:00Z",
      "transparencyLogReference": "rekor://UUID",
      "trust": {
        "tenantId": "default",
        "issuerId": "ubuntu",
        "effectiveWeight": 0.9,
        "tenantOverrideApplied": false,
        "retrievedAtUtc": "2024-10-22T09:00:30Z"
      }
    }
  },
  "aoc": {
    "guardVersion": "2024.10.0",
    "violations": [],                    // non-empty -> stored + surfaced
    "ingestedAt": "2024-10-22T09:00:05Z",
    "retrievedAt": "2024-10-22T08:59:59Z"
  },
  "metadata": {
    "provider-hint": "Mainline feed",
    "source-channel": "mirror"
  }
}

Field notes

tenant – logical tenant resolved by WebService based on headers or default configuration.
observationId – deterministic hash (sha256) over {tenant, vulnerabilityId, productKey, providerId, statementDigest}. Never reused.
status + justification – follow the OpenVEX semantics enforced by StellaOps.Excititor.Core.VexClaim.
scope – includes canonical key plus normalized PURLs/CPES; deterministic ordering.
anchors – optional JSON-pointer hints pointing to the source document sections; stored as trimmed strings.
document.signature – mirrors VexSignatureMetadata; empty if upstream feed lacks signatures.
aoc.violations – stored if the guard detected non-fatal issues; fatal issues never create an observation.
metadata – reserved for deterministic provider hints; keys follow vex.* prefix guidance.

Determinism & AOC guarantees

Write-once – once inserted, observation documents never change. New evidence creates a new observationId.
Sorted collections – arrays (anchors, purls, cpes) are sorted lexicographically before persistence.
Guard metadata – aoc.guardVersion records the guard library version (docs/aoc/guard-library.md), enabling audits.
Signatures – only verification metadata proven by the Worker is stored; WebService never recomputes trust.
Time normalization – all timestamps stored as UTC ISO-8601 strings (PostgreSQL timestamptz).

API mapping

API	Source fields	Notes
`GET /vex/observations`	`tenant`, `vulnerabilityId`, `productKey`, `providerId`	List observations with filters. Implemented in `ObservationEndpoints.cs`.
`GET /vex/observations/{observationId}`	`tenant`, `observationId`	Get single observation by ID with full detail.
`GET /vex/observations/count`	`tenant`	Count all observations for tenant.
`/v1/vex/observations/{vuln}/{product}`	`tenant`, `vulnerabilityId`, `productKey`, `scope`, `statements[]`	Response uses `VexObservationProjectionService` to render `statements`, `document`, and `signature` fields.
`/vex/aoc/verify`	`document.digest`, `providerId`, `aoc`	Replays guard validation for recent digests; guard violations here align with `aoc.violations`.
Evidence batch API (Graph)	`statements[]`, `scope`, `signals`, `anchors`	Format optimized for overlays; reduces `document` to digest/URI.

EXCITITOR-GRAPH-24-* relies on this schema to build overlays.
DOCS-LNM-22-002 (Link-Not-Merge documentation) references this file.
EXCITITOR-ATTEST-73-* uses document.digest + signature to embed provenance in attestation payloads.

Rekor Transparency Log Linkage

Sprint Reference: SPRINT_20260117_002_EXCITITOR_vex_rekor_linkage

VEX observations can be attested to the Sigstore Rekor transparency log, providing an immutable, publicly verifiable record of when each observation was recorded. This supports:

Auditability: Independent verification that an observation existed at a specific time
Non-repudiation: Cryptographic proof of observation provenance
Supply chain compliance: Evidence for regulatory and security requirements
Offline verification: Stored inclusion proofs enable air-gapped verification

Rekor Linkage Fields

The following fields are added to vex_observations when an observation is attested:

Field	Type	Description
`rekor_uuid`	TEXT	Rekor entry UUID (64-char hex)
`rekor_log_index`	BIGINT	Monotonically increasing log position
`rekor_integrated_time`	TIMESTAMPTZ	When entry was integrated into log
`rekor_log_url`	TEXT	Rekor server URL where submitted
`rekor_inclusion_proof`	JSONB	RFC 6962 inclusion proof for offline verification
`rekor_linked_at`	TIMESTAMPTZ	When linkage was recorded locally

Schema Extension

-- V20260117__vex_rekor_linkage.sql
ALTER TABLE excititor.vex_observations
ADD COLUMN IF NOT EXISTS rekor_uuid TEXT,
ADD COLUMN IF NOT EXISTS rekor_log_index BIGINT,
ADD COLUMN IF NOT EXISTS rekor_integrated_time TIMESTAMPTZ,
ADD COLUMN IF NOT EXISTS rekor_log_url TEXT,
ADD COLUMN IF NOT EXISTS rekor_inclusion_proof JSONB,
ADD COLUMN IF NOT EXISTS rekor_linked_at TIMESTAMPTZ;

-- Indexes for Rekor queries
CREATE INDEX idx_vex_observations_rekor_uuid
ON excititor.vex_observations(rekor_uuid)
WHERE rekor_uuid IS NOT NULL;

CREATE INDEX idx_vex_observations_pending_rekor
ON excititor.vex_observations(created_at)
WHERE rekor_uuid IS NULL;

API Endpoints

Endpoint	Method	Description
`/attestations/rekor/observations/{id}`	POST	Attest observation to Rekor
`/attestations/rekor/observations/batch`	POST	Batch attestation
`/attestations/rekor/observations/{id}/verify`	GET	Verify Rekor linkage
`/attestations/rekor/pending`	GET	List observations pending attestation

CLI Commands

# Show observation with Rekor details
stella vex observation show <id> --show-rekor

# Attest an observation to Rekor
stella vex observation attest <id> [--rekor-url URL]

# Verify Rekor linkage
stella vex observation verify-rekor <id> [--offline]

# List pending attestations
stella vex observation list-pending

Inclusion Proof Structure

{
  "treeSize": 1234567,
  "rootHash": "base64-encoded-root-hash",
  "logIndex": 12345,
  "hashes": [
    "base64-hash-1",
    "base64-hash-2",
    "base64-hash-3"
  ]
}

Verification Modes

Mode	Network	Use Case
Online	Required	Full verification against live Rekor
Offline	Not required	Verify using stored inclusion proof

Offline mode uses the stored rekor_inclusion_proof to verify the Merkle path locally. This is essential for air-gapped environments.

9.3 KiB Raw Blame History Unescape Escape

VEX Observation Model (vex_observations)