--- checkId: check.evidencelocker.index plugin: stellaops.doctor.evidencelocker severity: warn tags: [evidence, index, consistency] --- # Evidence Index Consistency ## What It Checks Verifies that the evidence index is consistent with the artifacts stored on disk. The check operates on the local evidence locker path (`EvidenceLocker:Path`) and performs: 1. **Index existence**: looks for `index.json` or an `index/` directory at the locker root. 2. **Artifact counting**: counts `.json` files across five artifact directories: `attestations/`, `sboms/`, `vex/`, `verdicts/`, `provenance/`. 3. **Cross-reference validation**: for each entry in `index.json`, verifies the referenced artifact file exists on disk. Records any artifacts that are indexed but missing from disk. 4. **Drift detection**: compares the total indexed count against the total disk artifact count. Flags a warning if drift exceeds 10% of total artifacts. | Condition | Result | |---|---| | Evidence locker path not configured or missing | Skip | | Index file and index directory both missing | Warn | | Artifacts indexed but missing from disk | Fail | | Index count drifts > 10% from disk count | Warn | | Index consistent with disk artifacts | Pass | Evidence collected: `IndexedCount`, `DiskArtifactCount`, `MissingFromDisk`, `MissingSamples`, `Drift`, per-directory counts (`attestationsCount`, `sbomsCount`, `vexCount`, `verdictsCount`, `provenanceCount`). The check only runs when `EvidenceLocker:Path` is configured and the directory exists. ## Why It Matters The evidence index provides fast lookup for attestations, SBOMs, VEX documents, and provenance records. An inconsistent index means queries may return stale references to deleted artifacts (causing retrieval errors) or miss artifacts that exist on disk (causing incomplete audit reports). Index drift accumulates over time and degrades the reliability of evidence searches, compliance exports, and release verification lookups. ## Common Causes - Index never created (evidence locker not initialized) - Index file was deleted or corrupted - Artifacts deleted without updating the index (manual cleanup) - Disk corruption causing artifact loss - Background indexer not running or crashed - Race condition during concurrent writes - Incomplete cleanup operations removing files but not index entries ## How to Fix ### Docker Compose ```bash # Check index status docker compose exec evidence-locker ls -la /data/evidence/index.json # Rebuild evidence index docker compose exec evidence-locker stella evidence index rebuild # Fix orphaned index entries docker compose exec evidence-locker stella evidence index rebuild --fix-orphans # Verify evidence integrity after rebuild docker compose exec evidence-locker stella evidence verify --all # Refresh index (less aggressive than rebuild) docker compose exec evidence-locker stella evidence index refresh # Check disk health docker compose exec evidence-locker df -h /data/evidence ``` ### Bare Metal / systemd ```bash # Check index file ls -la /var/lib/stellaops/evidence/index.json # Rebuild evidence index stella evidence index rebuild # Fix orphaned entries stella evidence index rebuild --fix-orphans # Refresh index stella evidence index refresh # Check for disk errors sudo fsck -n /dev/sda1 # Verify evidence integrity stella evidence verify --all sudo systemctl restart stellaops-evidence-locker ``` ### Kubernetes / Helm ```bash # Check index in pod kubectl exec deploy/stellaops-evidence-locker -- ls -la /data/evidence/index.json # Rebuild index kubectl exec deploy/stellaops-evidence-locker -- stella evidence index rebuild --fix-orphans # Verify evidence kubectl exec deploy/stellaops-evidence-locker -- stella evidence verify --all # Check persistent volume health kubectl describe pvc stellaops-evidence-data ``` ```yaml # values.yaml - enable background indexer evidenceLocker: indexer: enabled: true intervalMinutes: 15 repairOnDrift: true ``` ## Verification ``` stella doctor run --check check.evidencelocker.index ``` ## Related Checks - `check.evidencelocker.retrieval` — retrieval depends on index accuracy for lookups - `check.evidencelocker.provenance` — provenance records are one of the indexed artifact types - `check.evidencelocker.merkle` — Merkle anchors reference indexed artifacts - `check.compliance.evidence-integrity` — evidence integrity includes index consistency