git.stella-ops.org/docs/doctor/articles/evidence-locker/provenance.md

checkId, plugin, severity, tags

checkId	plugin	severity	tags
check.evidencelocker.provenance	stellaops.doctor.evidencelocker	fail	evidence provenance integrity chain

Provenance Chain Integrity

What It Checks

Validates provenance chain integrity using random sample verification. The check operates on provenance records stored in the provenance/ subdirectory of the evidence locker path. It performs:

1. Random sampling: selects up to 5 random provenance records from the pool for validation (configurable via SampleSize constant).
2. Hash verification: for each sampled record, reads the contentHash and payload fields, recomputes the SHA-256 hash of the payload, and compares it against the declared hash. Supports sha256: prefixed hash values.
3. Structural validation: verifies that each record contains required contentHash and payload fields.

Condition	Result
Evidence locker path not configured	Skip
No provenance directory or no records	Pass (nothing to verify)
Any sampled records fail hash verification	Fail
All sampled records pass hash verification	Pass

Evidence collected: TotalRecords, SamplesChecked, ValidCount, InvalidCount, InvalidRecords.

The check only runs when EvidenceLocker:Path is configured and the directory exists.

Why It Matters

Provenance records link each software artifact to its build source, build system, and build steps. The content hash ensures that the provenance payload has not been modified since it was created. A broken hash indicates the provenance record was corrupted or tampered with, which invalidates the supply-chain integrity guarantee for the associated release. Even a single invalid provenance record undermines trust in the entire provenance chain and should be investigated as a potential security incident.

Common Causes

• Provenance record corrupted on disk (storage errors, incomplete writes)
• Hash verification failure after accidental file modification
• Chain link broken due to missing predecessor records
• Data tampered or modified by unauthorized access
• Hash format mismatch (missing or extra sha256: prefix)
• Character encoding differences during payload serialization

How to Fix

Docker Compose

# Run full provenance audit
docker compose exec evidence-locker stella evidence audit --type provenance --full

# Check specific invalid records
docker compose exec evidence-locker stella evidence verify --id <record-id>

# Review evidence locker integrity
docker compose exec evidence-locker stella evidence integrity-check

# Check for storage errors
docker compose exec evidence-locker dmesg | grep -i error

# Check disk health
docker compose exec evidence-locker df -h /data/evidence/provenance/

Bare Metal / systemd

# Full provenance audit
stella evidence audit --type provenance --full

# Verify specific records
stella evidence verify --id <record-id>

# Full integrity check
stella evidence integrity-check

# Check filesystem health
sudo fsck -n /dev/sda1

# Check for disk I/O errors
dmesg | grep -i "i/o error"

# List provenance records
ls -la /var/lib/stellaops/evidence/provenance/

Kubernetes / Helm

# Full provenance audit
kubectl exec deploy/stellaops-evidence-locker -- stella evidence audit --type provenance --full

# Verify specific record
kubectl exec deploy/stellaops-evidence-locker -- stella evidence verify --id <record-id>

# Integrity check
kubectl exec deploy/stellaops-evidence-locker -- stella evidence integrity-check

# Check persistent volume health
kubectl describe pvc stellaops-evidence-data

# Check for pod restarts that might indicate storage issues
kubectl get events --field-selector involvedObject.name=stellaops-evidence-locker -n stellaops

# values.yaml - schedule periodic integrity checks
evidenceLocker:
  integrityCheck:
    enabled: true
    schedule: "0 4 * * *"  # Daily at 4am
    sampleSize: 10

Verification

stella doctor run --check check.evidencelocker.provenance

Related Checks

• check.evidencelocker.merkle — Merkle anchoring provides checkpoint-level integrity on top of per-record verification
• check.evidencelocker.index — index consistency ensures provenance records are discoverable
• check.evidencelocker.retrieval — retrieval health is required to access provenance records
• check.compliance.provenance-completeness — verifies provenance exists for all releases (completeness vs. integrity)
• check.compliance.evidence-integrity — broader evidence integrity check including provenance