---
checkId: check.evidencelocker.provenance
plugin: stellaops.doctor.evidencelocker
severity: fail
tags: [evidence, provenance, integrity, chain]
---
# Provenance Chain Integrity

## What It Checks
Validates provenance chain integrity using random sample verification. The check operates on provenance records stored in the `provenance/` subdirectory of the evidence locker path. It performs:

1. **Random sampling**: selects up to 5 random provenance records from the pool for validation (configurable via `SampleSize` constant).
2. **Hash verification**: for each sampled record, reads the `contentHash` and `payload` fields, recomputes the SHA-256 hash of the payload, and compares it against the declared hash. Supports `sha256:` prefixed hash values.
3. **Structural validation**: verifies that each record contains required `contentHash` and `payload` fields.

| Condition | Result |
|---|---|
| Evidence locker path not configured | Skip |
| No provenance directory or no records | Pass (nothing to verify) |
| Any sampled records fail hash verification | Fail |
| All sampled records pass hash verification | Pass |

Evidence collected: `TotalRecords`, `SamplesChecked`, `ValidCount`, `InvalidCount`, `InvalidRecords`.

The check only runs when `EvidenceLocker:Path` is configured and the directory exists.

## Why It Matters
Provenance records link each software artifact to its build source, build system, and build steps. The content hash ensures that the provenance payload has not been modified since it was created. A broken hash indicates the provenance record was corrupted or tampered with, which invalidates the supply-chain integrity guarantee for the associated release. Even a single invalid provenance record undermines trust in the entire provenance chain and should be investigated as a potential security incident.

## Common Causes
- Provenance record corrupted on disk (storage errors, incomplete writes)
- Hash verification failure after accidental file modification
- Chain link broken due to missing predecessor records
- Data tampered or modified by unauthorized access
- Hash format mismatch (missing or extra `sha256:` prefix)
- Character encoding differences during payload serialization

## How to Fix

### Docker Compose
```bash
# Run full provenance audit
docker compose exec evidence-locker stella evidence audit --type provenance --full

# Check specific invalid records
docker compose exec evidence-locker stella evidence verify --id <record-id>

# Review evidence locker integrity
docker compose exec evidence-locker stella evidence integrity-check

# Check for storage errors
docker compose exec evidence-locker dmesg | grep -i error

# Check disk health
docker compose exec evidence-locker df -h /data/evidence/provenance/
```

### Bare Metal / systemd
```bash
# Full provenance audit
stella evidence audit --type provenance --full

# Verify specific records
stella evidence verify --id <record-id>

# Full integrity check
stella evidence integrity-check

# Check filesystem health
sudo fsck -n /dev/sda1

# Check for disk I/O errors
dmesg | grep -i "i/o error"

# List provenance records
ls -la /var/lib/stellaops/evidence/provenance/
```

### Kubernetes / Helm
```bash
# Full provenance audit
kubectl exec deploy/stellaops-evidence-locker -- stella evidence audit --type provenance --full

# Verify specific record
kubectl exec deploy/stellaops-evidence-locker -- stella evidence verify --id <record-id>

# Integrity check
kubectl exec deploy/stellaops-evidence-locker -- stella evidence integrity-check

# Check persistent volume health
kubectl describe pvc stellaops-evidence-data

# Check for pod restarts that might indicate storage issues
kubectl get events --field-selector involvedObject.name=stellaops-evidence-locker -n stellaops
```

```yaml
# values.yaml - schedule periodic integrity checks
evidenceLocker:
  integrityCheck:
    enabled: true
    schedule: "0 4 * * *"  # Daily at 4am
    sampleSize: 10
```

## Verification
```
stella doctor run --check check.evidencelocker.provenance
```

## Related Checks
- `check.evidencelocker.merkle` — Merkle anchoring provides checkpoint-level integrity on top of per-record verification
- `check.evidencelocker.index` — index consistency ensures provenance records are discoverable
- `check.evidencelocker.retrieval` — retrieval health is required to access provenance records
- `check.compliance.provenance-completeness` — verifies provenance exists for all releases (completeness vs. integrity)
- `check.compliance.evidence-integrity` — broader evidence integrity check including provenance