git.stella-ops.org/docs/dev/EXCITITOR_STATEMENT_BACKFILL.md

Excititor Statement Backfill Runbook

Last updated: 2025-10-19

Overview

Use this runbook when you need to rebuild the vex.statements collection from historical raw documents. Typical scenarios:

• Upgrading the statement schema (e.g., adding severity/KEV/EPSS signals).
• Recovering from a partial ingest outage where statements were never persisted.
• Seeding a freshly provisioned Excititor deployment from an existing raw archive.

Backfill operates server-side via the Excititor WebService and reuses the same pipeline that powers the /excititor/statements ingestion endpoint. Each raw document is normalized, signed metadata is preserved, and duplicate statements are skipped unless the run is forced.

Prerequisites

1. Connectivity to Excititor WebService – the CLI uses the backend URL configured in stellaops.yml or the --backend-url argument.
2. Authority credentials – the CLI honours the existing Authority client configuration; ensure the caller has permission to invoke admin endpoints.
3. Mongo replica set (recommended) – causal consistency guarantees rely on majority read/write concerns. Standalone deployment works but skips cross-document transactions.

CLI command

stellaops excititor backfill-statements \
  [--retrieved-since <ISO8601>] \
  [--force] \
  [--batch-size <int>] \
  [--max-documents <int>]

Option	Description
--retrieved-since	Only process raw documents fetched on or after the specified timestamp (UTC by default).
--force	Reprocess documents even if matching statements already exist (useful after schema upgrades).
--batch-size	Number of raw documents pulled per batch (default 100).
--max-documents	Optional hard limit on the number of raw documents to evaluate.

Example – replay the last 48 hours of Red Hat ingest while keeping existing statements:

stellaops excititor backfill-statements \
  --retrieved-since "$(date -u -d '48 hours ago' +%Y-%m-%dT%H:%M:%SZ)"

Example – full replay with forced overwrites, capped at 2,000 documents:

stellaops excititor backfill-statements --force --max-documents 2000

The command returns a summary similar to:

Backfill completed: evaluated 450, backfilled 180, claims written 320, skipped 270, failures 0.

Behaviour

• Raw documents are streamed in ascending retrievedAt order.
• Each document is normalized using the registered VEX normalizers (CSAF, CycloneDX, OpenVEX).
• Statements are appended through the same IVexClaimStore.AppendAsync path that powers /excititor/statements.
• Duplicate detection compares Document.Digest; duplicates are skipped unless --force is specified.
• Failures are logged with the offending digest and continue with the next document.

Observability

• CLI logs aggregate counts and the backend logs per-digest warnings or errors.
• Mongo writes carry majority write concern; expect backfill throughput to match ingest baselines (≈5 seconds warm, 30 seconds cold).
• Monitor the excititor.storage.backfill log scope for detailed telemetry.

Post-run verification

1. Inspect the vex.statements collection for the targeted window (check InsertedAt).
2. Re-run the Excititor storage test suite if possible:

   dotnet test src/StellaOps.Excititor.Storage.Mongo.Tests/StellaOps.Excititor.Storage.Mongo.Tests.csproj
   

3. Optionally, call /excititor/statements/{vulnerabilityId}/{productKey} to confirm the expected statements exist.

Rollback

If a forced run produced incorrect statements, use the standard Mongo rollback procedure:

1. Identify the InsertedAt window for the backfill run.
2. Delete affected records from vex.statements (and any downstream exports if applicable).
3. Rerun the backfill command with corrected parameters.