git.stella-ops.org/docs/doctor/articles/evidence-locker/retrieval.md

checkId, plugin, severity, tags

checkId	plugin	severity	tags
check.evidencelocker.retrieval	stellaops.doctor.evidencelocker	fail	evidence attestation retrieval core

Attestation Retrieval

What It Checks

Verifies that attestation artifacts can be retrieved from the evidence locker. The check supports two modes depending on the deployment:

HTTP mode (when IHttpClientFactory is available): Sends a GET request to {endpoint}/v1/attestations/sample with a 5-second timeout and measures response latency.

Local file mode (fallback): Checks the local evidence locker path at EvidenceLocker:Path, verifies the attestations/ subdirectory exists, and attempts to read a sample attestation JSON file.

Condition	Result
Endpoint not configured	Skip
HTTP request times out (> 5000ms)	Fail
HTTP error status code	Fail
Connection error	Fail
HTTP success but latency > 500ms	Warn
Local attestations directory missing	Warn
HTTP success with latency <= 500ms	Pass
Local file read successful	Pass

Evidence collected: Endpoint, StatusCode, LatencyMs, Threshold, Path, SampleAttestation, ContentLength.

The check only runs when EvidenceLocker:Endpoint or Services:EvidenceLocker is configured.

Why It Matters

Attestation retrieval is a core operation used throughout the release pipeline. Release approvals, audit queries, compliance reports, and evidence exports all depend on being able to retrieve attestation artifacts from the evidence locker. If retrieval is slow or failing, release approvals may time out, audit queries will fail, and compliance reports cannot be generated. Latency above 500ms indicates performance degradation that will compound when retrieving multiple attestations during a release or audit.

Common Causes

• Evidence locker service unavailable or not running
• Authentication failure when accessing the evidence locker API
• Artifact not found (empty or uninitialized evidence locker)
• Evidence locker under heavy load causing elevated latency
• Network latency between services
• Storage backend slow (disk I/O bottleneck)
• Local evidence locker path not configured or directory missing
• File permission issues on local attestation files

How to Fix

Docker Compose

# Check evidence locker service status
docker compose ps evidence-locker

# Test evidence retrieval
docker compose exec evidence-locker stella evidence status

# Test authentication
docker compose exec evidence-locker stella evidence auth-test

# Check service logs for errors
docker compose logs evidence-locker --since 5m

# If local mode, verify the evidence path and permissions
docker compose exec evidence-locker ls -la /data/evidence/attestations/

# Initialize evidence locker if needed
docker compose exec evidence-locker stella evidence init

# Set endpoint configuration
# EvidenceLocker__Endpoint=http://evidence-locker:5080

Bare Metal / systemd

# Check service status
sudo systemctl status stellaops-evidence-locker

# Test evidence retrieval
stella evidence status

# Test connectivity
stella evidence ping

# Check attestations directory
ls -la /var/lib/stellaops/evidence/attestations/

# Initialize if empty
stella evidence init

# Check disk I/O
iostat -x 1 5

# In appsettings.json:
# "EvidenceLocker": { "Endpoint": "http://localhost:5080" }

sudo systemctl restart stellaops-evidence-locker

Kubernetes / Helm

# Check evidence locker pod status
kubectl get pods -l app=stellaops-evidence-locker

# Check pod logs
kubectl logs deploy/stellaops-evidence-locker --since=5m

# Test retrieval from within cluster
kubectl exec deploy/stellaops-evidence-locker -- stella evidence status

# Check persistent volume
kubectl describe pvc stellaops-evidence-data

# Check for resource constraints
kubectl top pod -l app=stellaops-evidence-locker

# values.yaml
evidenceLocker:
  endpoint: http://stellaops-evidence-locker:5080
  resources:
    requests:
      memory: 256Mi
      cpu: 100m
    limits:
      memory: 512Mi
      cpu: 500m

Verification

stella doctor run --check check.evidencelocker.retrieval

Related Checks

• check.evidencelocker.index — evidence index consistency affects retrieval accuracy
• check.evidencelocker.provenance — provenance chain integrity depends on reliable retrieval
• check.evidencelocker.merkle — Merkle anchor verification requires attestation access
• check.compliance.evidence-rate — evidence generation feeds the retrieval pipeline
• check.compliance.evidence-integrity — integrity verification requires successful retrieval