git.stella-ops.org/docs/modules/issuer-directory/operations/backup-restore.md

Issuer Directory Backup & Restore

Scope

• Applies to: Issuer Directory when deployed via Docker Compose (deploy/compose/docker-compose.*.yaml) or the Helm chart (deploy/helm/stellaops).
• Artifacts covered: PostgreSQL database issuer_directory, service configuration (etc/issuer-directory.yaml), CSAF seed file (data/csaf-publishers.json), and secret material for the PostgreSQL connection string.
• Frequency: Take a hot backup before every upgrade and at least daily in production. Keep encrypted copies off-site/air-gapped according to your compliance program.

Inventory checklist

Component	Location (Compose default)	Notes
PostgreSQL data	postgres-data volume (/var/lib/docker/volumes/.../postgres-data)	Contains issuers, issuer_keys, issuer_trust_overrides, and issuer_audit tables in the issuer_directory schema.
Configuration	etc/issuer-directory.yaml	Mounted read-only at /etc/issuer-directory.yaml inside the container.
CSAF seed file	src/IssuerDirectory/StellaOps.IssuerDirectory/data/csaf-publishers.json	Ensure customised seeds are part of the backup; regenerate if you ship regional overrides.
PostgreSQL secret	.env entry ISSUER_DIRECTORY_POSTGRES_CONNECTION_STRING or secret store export	Required to restore connectivity; treat as sensitive.

Tip: Export the secret via kubectl get secret issuer-directory-secrets -o yaml (sanitize before storage) or copy the Compose .env file into an encrypted vault. For PostgreSQL credentials, consider using pg_dump with connection info from environment variables.

Hot backup (no downtime)

1. Create output directory

   BACKUP_DIR=backup/issuer-directory/$(date +%Y-%m-%dT%H%M%S)
   mkdir -p "$BACKUP_DIR"
   

2. Dump PostgreSQL tables

   docker compose -f deploy/compose/docker-compose.prod.yaml exec postgres \
     pg_dump --format=custom --compress=9 \
     --file=/dump/issuer-directory-$(date +%Y%m%dT%H%M%SZ).dump \
     --schema=issuer_directory issuer_directory
   
   docker compose -f deploy/compose/docker-compose.prod.yaml cp \
     postgres:/dump/issuer-directory-$(date +%Y%m%dT%H%M%SZ).dump "$BACKUP_DIR/"
   

   For Kubernetes, run the same pg_dump command inside the stellaops-postgres pod and copy the archive via kubectl cp.
3. Capture configuration and seeds

   cp etc/issuer-directory.yaml "$BACKUP_DIR/"
   cp src/IssuerDirectory/StellaOps.IssuerDirectory/data/csaf-publishers.json "$BACKUP_DIR/"
   

4. Capture secrets

   grep '^ISSUER_DIRECTORY_POSTGRES_CONNECTION_STRING=' dev.env > "$BACKUP_DIR/issuer-directory.postgres.secret"
   chmod 600 "$BACKUP_DIR/issuer-directory.postgres.secret"
   

5. Generate checksums and encrypt

   (cd "$BACKUP_DIR" && sha256sum * > SHA256SUMS)
   tar czf "$BACKUP_DIR.tar.gz" -C "$BACKUP_DIR" .
   age -r you@example.org "$BACKUP_DIR.tar.gz" > "$BACKUP_DIR.tar.gz.age"
   

Cold backup (planned downtime)

1. Notify stakeholders and pause automation calling the API.
2. Stop services:

   docker compose -f deploy/compose/docker-compose.prod.yaml down issuer-directory
   

   (For Helm: kubectl scale deploy stellaops-issuer-directory --replicas=0.)
3. Snapshot volumes:

   docker run --rm -v postgres-data:/data \
     -v "$(pwd)":/backup busybox tar czf /backup/postgres-data-$(date +%Y%m%d).tar.gz -C /data .
   

4. Copy configuration, seeds, and secrets as in the hot backup.
5. Restart services and confirm /health/live returns 200 OK.

Restore procedure

1. Provision clean volumes
   • Compose: docker volume rm postgres-data (optional) then docker compose up -d postgres.
   • Helm: delete the PostgreSQL PVC or attach a fresh volume snapshot.
2. Restore PostgreSQL

   docker compose exec -T postgres \
     pg_restore --format=custom --clean --if-exists \
     --dbname=issuer_directory < issuer-directory-YYYYMMDDTHHMMSSZ.dump
   

3. Restore configuration/secrets
   • Copy issuer-directory.yaml into etc/.
   • Reapply the secret: kubectl apply -f issuer-directory-secret.yaml or repopulate .env.
4. Restore CSAF seeds (optional)
   • If you maintain a customised seed file, copy it back before starting the container. Otherwise the bundled file will be used.
5. Start services

   docker compose up -d issuer-directory
   # or
   kubectl scale deploy stellaops-issuer-directory --replicas=1
   

6. Validate
   • curl -fsSL https://localhost:8447/health/live
   • Issue an access token and list issuers to confirm results.
   • Check PostgreSQL counts match expectations (SELECT COUNT(*) FROM issuer_directory.issuers;, etc.).
   • Confirm Prometheus scrapes issuer_directory_changes_total and issuer_directory_key_operations_total for the tenants you restored.

Disaster recovery notes

• Retention: Maintain 30 daily + 12 monthly archives. Store copies in geographically separate, access-controlled vaults.
• Audit reconciliation: Ensure issuer_audit entries cover the restore window; export them for compliance.
• Seed replay: If the CSAF seed file was lost, set ISSUER_DIRECTORY_SEED_CSAF=true for the first restart to rehydrate the global tenant.
• Testing: Run quarterly restore drills in a staging environment to validate procedure drift.

Verification checklist

• /health/live returns 200 OK.
• PostgreSQL tables (issuers, issuer_keys, issuer_trust_overrides) have expected counts.
• issuer_directory_changes_total, issuer_directory_key_operations_total, and issuer_directory_key_validation_failures_total metrics resume within 1 minute.
• Audit entries exist for post-restore CRUD activity.
• Client integrations (VEX Lens, Excititor) resolve issuers successfully.