git.stella-ops.org/docs/operations/evidence-migration.md

# Evidence Migration Guide

This guide covers evidence-specific migration procedures during upgrades, schema changes, or disaster recovery scenarios.

## Overview

Evidence bundles are cryptographically linked data structures that must maintain integrity across upgrades. This guide ensures chain-of-custody is preserved during migrations.

## Quick Reference

| Scenario | CLI Command | Risk Level | Downtime |
|----------|-------------|------------|----------|
| Schema upgrade | `stella evidence migrate` | Medium | Minutes |
| Reindex after algorithm change | `stella evidence reindex` | Low | None |
| Cross-version continuity check | `stella evidence verify-continuity` | None | None |
| Full evidence export | `stella evidence export --all` | None | None |

## Pre-Migration Checklist

### 1. Capture Current State

```bash
# Record current evidence statistics
stella evidence stats --detailed > pre-migration-stats.json

# Export Merkle roots for all tenants
stella evidence roots-export --all > pre-migration-roots.json

# Verify existing evidence integrity
stella evidence verify-all --output pre-migration-verify.json
if [ $? -ne 0 ]; then
  echo "ABORT: Evidence integrity check failed"
  exit 1
fi
```

### 2. Create Evidence Backup

```bash
# Full evidence bundle export
stella evidence export \
  --all \
  --include-attestations \
  --include-proofs \
  --output /backup/evidence-$(date +%Y%m%d)/

# Verify export integrity
stella evidence verify-bundle /backup/evidence-*/
```

### 3. Document Chain-of-Custody

```bash
# Record the current root hashes
OLD_MERKLE_ROOT=$(stella evidence roots-export --format json | jq -r '.globalRoot')
echo "Pre-migration Merkle root: ${OLD_MERKLE_ROOT}" > custody-log.txt
date >> custody-log.txt
```

## Migration Procedures

### Schema Migration (Version Upgrade)

When upgrading between versions with schema changes:

```bash
# Step 1: Assess migration impact (dry-run)
stella evidence migrate \
  --from-version 1.0 \
  --to-version 2.0 \
  --dry-run

# Step 2: Review migration plan output
# Ensure all changes are expected

# Step 3: Execute migration
stella evidence migrate \
  --from-version 1.0 \
  --to-version 2.0

# Step 4: Verify migration
stella evidence verify-all
```

### Evidence Reindex (Algorithm Change)

When the hashing algorithm or Merkle tree structure changes:

```bash
# Step 1: Assess reindex impact
stella evidence reindex \
  --dry-run \
  --output reindex-plan.json

# Review reindex-plan.json for:
# - Total records affected
# - Estimated duration
# - New schema version

# Step 2: Execute reindex with batching
stella evidence reindex \
  --batch-size 100 \
  --since 2026-01-01

# Step 3: Capture new root
NEW_MERKLE_ROOT=$(stella evidence roots-export --format json | jq -r '.globalRoot')
echo "Post-migration Merkle root: ${NEW_MERKLE_ROOT}" >> custody-log.txt
date >> custody-log.txt
```

### Chain-of-Custody Verification

After any evidence migration, verify continuity:

```bash
# Verify that old proofs remain valid
stella evidence verify-continuity \
  --old-root "${OLD_MERKLE_ROOT}" \
  --new-root "${NEW_MERKLE_ROOT}" \
  --output continuity-report.html \
  --format html

# Check verification results
if grep -q "FAIL" continuity-report.html; then
  echo "ERROR: Chain-of-custody verification failed!"
  echo "Review continuity-report.html for details"
  exit 1
fi
```

## Rollback Procedures

### Immediate Rollback (Within Migration Window)

```bash
# If migration fails mid-way, rollback is automatic
# Check current migration state
stella evidence migrate --status

# Force rollback if needed
stella evidence migrate \
  --rollback \
  --from-version 2.0
```

### Restore from Backup

```bash
# Step 1: Stop evidence-related services
kubectl scale deployment evidence-locker --replicas=0

# Step 2: Restore PostgreSQL evidence tables
pg_restore -d stellaops \
  --table='evidence.*' \
  /backup/stellaops-backup.dump

# Step 3: Restore evidence files
stella evidence import /backup/evidence-*/

# Step 4: Verify restoration
stella evidence verify-all

# Step 5: Restart services
kubectl scale deployment evidence-locker --replicas=3
```

## Air-Gap Migration

For air-gapped environments without network access:

### Export Phase (Online Environment)

```bash
# Create portable evidence bundle
stella evidence export \
  --all \
  --portable \
  --include-schemas \
  --output /media/airgap-evidence.tar.gz

# Generate checksums
sha256sum /media/airgap-evidence.tar.gz > /media/checksums.txt
```

### Transfer Phase

1. Copy to removable media
2. Verify checksums at destination
3. Scan media for security

### Import Phase (Air-Gap Environment)

```bash
# Verify transfer integrity
sha256sum -c /media/checksums.txt

# Import evidence bundle
stella evidence import \
  --portable \
  /media/airgap-evidence.tar.gz

# Verify import
stella evidence verify-all
```

## Troubleshooting

### Migration Stuck or Timeout

```bash
# Check migration status
stella evidence migrate --status

# View migration logs
stella evidence migrate --logs

# Resume from last checkpoint
stella evidence migrate --resume
```

### Root Hash Mismatch

If verification reports root hash mismatch:

1. **Do not proceed** with upgrade
2. Check for data corruption:
   ```bash
   stella evidence integrity-check --deep
   ```
3. Review recent changes to evidence store
4. Contact support with integrity report

### Missing Evidence Records

```bash
# Count records by type
stella evidence stats --by-type

# Find orphaned records
stella evidence orphans --list

# Reconcile with source systems
stella evidence reconcile --source attestor
```

### Performance Issues

For large evidence stores (>1M records):

```bash
# Run reindex in parallel batches
stella evidence reindex \
  --parallel 4 \
  --batch-size 500 \
  --since 2026-01-01

# Monitor progress
stella evidence reindex --progress
```

## Audit Trail Requirements

All evidence migrations must maintain audit trail:

| Event | Required Data | Retention |
|-------|---------------|-----------|
| Migration Start | Timestamp, version, operator | Permanent |
| Schema Change | Before/after schema versions | Permanent |
| Root Hash Change | Old root, new root, cross-reference | Permanent |
| Verification | Pass/fail, anomalies, timestamps | 7 years |
| Rollback | Reason, restored version | Permanent |

## Related Documents

- [Upgrade Runbook](upgrade-runbook.md) - Overall upgrade procedures
- [Blue-Green Deployment](blue-green-deployment.md) - Zero-downtime deployment
- [Evidence Locker Architecture](../modules/evidencelocker/architecture.md) - Technical design
- [Air-Gap Operations](airgap-operations-runbook.md) - Offline deployment guide