git.stella-ops.org/docs/operations/evidence-migration.md

Evidence Migration Guide

This guide covers evidence-specific migration procedures during upgrades, schema changes, or disaster recovery scenarios.

Overview

Evidence bundles are cryptographically linked data structures that must maintain integrity across upgrades. This guide ensures chain-of-custody is preserved during migrations.

Quick Reference

Scenario	CLI Command	Risk Level	Downtime
Schema upgrade	stella evidence migrate	Medium	Minutes
Reindex after algorithm change	stella evidence reindex	Low	None
Cross-version continuity check	stella evidence verify-continuity	None	None
Full evidence export	stella evidence export --all	None	None

Pre-Migration Checklist

1. Capture Current State

# Record current evidence statistics
stella evidence stats --detailed > pre-migration-stats.json

# Export Merkle roots for all tenants
stella evidence roots-export --all > pre-migration-roots.json

# Verify existing evidence integrity
stella evidence verify-all --output pre-migration-verify.json
if [ $? -ne 0 ]; then
  echo "ABORT: Evidence integrity check failed"
  exit 1
fi

2. Create Evidence Backup

# Full evidence bundle export
stella evidence export \
  --all \
  --include-attestations \
  --include-proofs \
  --output /backup/evidence-$(date +%Y%m%d)/

# Verify export integrity
stella evidence verify-bundle /backup/evidence-*/

3. Document Chain-of-Custody

# Record the current root hashes
OLD_MERKLE_ROOT=$(stella evidence roots-export --format json | jq -r '.globalRoot')
echo "Pre-migration Merkle root: ${OLD_MERKLE_ROOT}" > custody-log.txt
date >> custody-log.txt

Migration Procedures

Schema Migration (Version Upgrade)

When upgrading between versions with schema changes:

# Step 1: Assess migration impact (dry-run)
stella evidence migrate \
  --from-version 1.0 \
  --to-version 2.0 \
  --dry-run

# Step 2: Review migration plan output
# Ensure all changes are expected

# Step 3: Execute migration
stella evidence migrate \
  --from-version 1.0 \
  --to-version 2.0

# Step 4: Verify migration
stella evidence verify-all

Evidence Reindex (Algorithm Change)

When the hashing algorithm or Merkle tree structure changes:

# Step 1: Assess reindex impact
stella evidence reindex \
  --dry-run \
  --output reindex-plan.json

# Review reindex-plan.json for:
# - Total records affected
# - Estimated duration
# - New schema version

# Step 2: Execute reindex with batching
stella evidence reindex \
  --batch-size 100 \
  --since 2026-01-01

# Step 3: Capture new root
NEW_MERKLE_ROOT=$(stella evidence roots-export --format json | jq -r '.globalRoot')
echo "Post-migration Merkle root: ${NEW_MERKLE_ROOT}" >> custody-log.txt
date >> custody-log.txt

Chain-of-Custody Verification

After any evidence migration, verify continuity:

# Verify that old proofs remain valid
stella evidence verify-continuity \
  --old-root "${OLD_MERKLE_ROOT}" \
  --new-root "${NEW_MERKLE_ROOT}" \
  --output continuity-report.html \
  --format html

# Check verification results
if grep -q "FAIL" continuity-report.html; then
  echo "ERROR: Chain-of-custody verification failed!"
  echo "Review continuity-report.html for details"
  exit 1
fi

Rollback Procedures

Immediate Rollback (Within Migration Window)

# If migration fails mid-way, rollback is automatic
# Check current migration state
stella evidence migrate --status

# Force rollback if needed
stella evidence migrate \
  --rollback \
  --from-version 2.0

Restore from Backup

# Step 1: Stop evidence-related services
kubectl scale deployment evidence-locker --replicas=0

# Step 2: Restore PostgreSQL evidence tables
pg_restore -d stellaops \
  --table='evidence.*' \
  /backup/stellaops-backup.dump

# Step 3: Restore evidence files
stella evidence import /backup/evidence-*/

# Step 4: Verify restoration
stella evidence verify-all

# Step 5: Restart services
kubectl scale deployment evidence-locker --replicas=3

Air-Gap Migration

For air-gapped environments without network access:

Export Phase (Online Environment)

# Create portable evidence bundle
stella evidence export \
  --all \
  --portable \
  --include-schemas \
  --output /media/airgap-evidence.tar.gz

# Generate checksums
sha256sum /media/airgap-evidence.tar.gz > /media/checksums.txt

Transfer Phase

1. Copy to removable media
2. Verify checksums at destination
3. Scan media for security

Import Phase (Air-Gap Environment)

# Verify transfer integrity
sha256sum -c /media/checksums.txt

# Import evidence bundle
stella evidence import \
  --portable \
  /media/airgap-evidence.tar.gz

# Verify import
stella evidence verify-all

Troubleshooting

Migration Stuck or Timeout

# Check migration status
stella evidence migrate --status

# View migration logs
stella evidence migrate --logs

# Resume from last checkpoint
stella evidence migrate --resume

Root Hash Mismatch

If verification reports root hash mismatch:

1. Do not proceed with upgrade
2. Check for data corruption:

   stella evidence integrity-check --deep
   

3. Review recent changes to evidence store
4. Contact support with integrity report

Missing Evidence Records

# Count records by type
stella evidence stats --by-type

# Find orphaned records
stella evidence orphans --list

# Reconcile with source systems
stella evidence reconcile --source attestor

Performance Issues

For large evidence stores (>1M records):

# Run reindex in parallel batches
stella evidence reindex \
  --parallel 4 \
  --batch-size 500 \
  --since 2026-01-01

# Monitor progress
stella evidence reindex --progress

Audit Trail Requirements

All evidence migrations must maintain audit trail:

Event	Required Data	Retention
Migration Start	Timestamp, version, operator	Permanent
Schema Change	Before/after schema versions	Permanent
Root Hash Change	Old root, new root, cross-reference	Permanent
Verification	Pass/fail, anomalies, timestamps	7 years
Rollback	Reason, restored version	Permanent

Related Documents

• Upgrade Runbook - Overall upgrade procedures
• Blue-Green Deployment - Zero-downtime deployment
• Evidence Locker Architecture - Technical design
• Air-Gap Operations - Offline deployment guide