# Issuer Directory Backup & Restore

## Scope
- **Applies to:** Issuer Directory when deployed via Docker Compose (`deploy/compose/docker-compose.*.yaml`) or the Helm chart (`deploy/helm/stellaops`).
- **Artifacts covered:** MongoDB database `issuer-directory`, service configuration (`etc/issuer-directory.yaml`), CSAF seed file (`data/csaf-publishers.json`), and secret material for the Mongo connection string.
- **Frequency:** Take a hot backup before every upgrade and at least daily in production. Keep encrypted copies off-site/air-gapped according to your compliance program.

## Inventory checklist
| Component | Location (Compose default) | Notes |
| --- | --- | --- |
| Mongo data | `mongo-data` volume (`/var/lib/docker/volumes/.../mongo-data`) | Contains `issuers`, `issuer_keys`, `issuer_trust_overrides`, and `issuer_audit` collections. |
| Configuration | `etc/issuer-directory.yaml` | Mounted read-only at `/etc/issuer-directory.yaml` inside the container. |
| CSAF seed file | `src/IssuerDirectory/StellaOps.IssuerDirectory/data/csaf-publishers.json` | Ensure customised seeds are part of the backup; regenerate if you ship regional overrides. |
| Mongo secret | `.env` entry `ISSUER_DIRECTORY_MONGO_CONNECTION_STRING` or secret store export | Required to restore connectivity; treat as sensitive. |

> **Tip:** Export the secret via `kubectl get secret issuer-directory-secrets -o yaml` (sanitize before storage) or copy the Compose `.env` file into an encrypted vault.

## Hot backup (no downtime)
1. **Create output directory**
   ```bash
   BACKUP_DIR=backup/issuer-directory/$(date +%Y-%m-%dT%H%M%S)
   mkdir -p "$BACKUP_DIR"
   ```
2. **Dump Mongo collections**
   ```bash
   docker compose -f deploy/compose/docker-compose.prod.yaml exec mongo \
     mongodump --archive=/dump/issuer-directory-$(date +%Y%m%dT%H%M%SZ).gz \
     --gzip --db issuer-directory

   docker compose -f deploy/compose/docker-compose.prod.yaml cp \
     mongo:/dump/issuer-directory-$(date +%Y%m%dT%H%M%SZ).gz "$BACKUP_DIR/"
   ```
   For Kubernetes, run the same `mongodump` command inside the `stellaops-mongo` pod and copy the archive via `kubectl cp`.
3. **Capture configuration and seeds**
   ```bash
   cp etc/issuer-directory.yaml "$BACKUP_DIR/"
   cp src/IssuerDirectory/StellaOps.IssuerDirectory/data/csaf-publishers.json "$BACKUP_DIR/"
   ```
4. **Capture secrets**
   ```bash
   grep '^ISSUER_DIRECTORY_MONGO_CONNECTION_STRING=' dev.env > "$BACKUP_DIR/issuer-directory.mongo.secret"
   chmod 600 "$BACKUP_DIR/issuer-directory.mongo.secret"
   ```
5. **Generate checksums and encrypt**
   ```bash
   (cd "$BACKUP_DIR" && sha256sum * > SHA256SUMS)
   tar czf "$BACKUP_DIR.tar.gz" -C "$BACKUP_DIR" .
   age -r you@example.org "$BACKUP_DIR.tar.gz" > "$BACKUP_DIR.tar.gz.age"
   ```

## Cold backup (planned downtime)
1. Notify stakeholders and pause automation calling the API.
2. Stop services:
   ```bash
   docker compose -f deploy/compose/docker-compose.prod.yaml down issuer-directory
   ```
   (For Helm: `kubectl scale deploy stellaops-issuer-directory --replicas=0`.)
3. Snapshot volumes:
   ```bash
   docker run --rm -v mongo-data:/data \
     -v "$(pwd)":/backup busybox tar czf /backup/mongo-data-$(date +%Y%m%d).tar.gz -C /data .
   ```
4. Copy configuration, seeds, and secrets as in the hot backup.
5. Restart services and confirm `/health/live` returns `200 OK`.

## Restore procedure
1. **Provision clean volumes**
   - Compose: `docker volume rm mongo-data` (optional) then `docker compose up -d mongo`.
   - Helm: delete the Mongo PVC or attach a fresh volume snapshot.
2. **Restore Mongo**
   ```bash
   docker compose exec -T mongo \
     mongorestore --archive \
     --gzip --drop < issuer-directory-YYYYMMDDTHHMMSSZ.gz
   ```
3. **Restore configuration/secrets**
   - Copy `issuer-directory.yaml` into `etc/`.
   - Reapply the secret: `kubectl apply -f issuer-directory-secret.yaml` or repopulate `.env`.
4. **Restore CSAF seeds** (optional)
   - If you maintain a customised seed file, copy it back before starting the container. Otherwise the bundled file will be used.
5. **Start services**
   ```bash
   docker compose up -d issuer-directory
   # or
   kubectl scale deploy stellaops-issuer-directory --replicas=1
   ```
6. **Validate**
   - `curl -fsSL https://localhost:8447/health/live`
   - Issue an access token and list issuers to confirm results.
   - Check Mongo counts match expectations (`db.issuers.countDocuments()`, etc.).

## Disaster recovery notes
- **Retention:** Maintain 30 daily + 12 monthly archives. Store copies in geographically separate, access-controlled vaults.
- **Audit reconciliation:** Ensure `issuer_audit` entries cover the restore window; export them for compliance.
- **Seed replay:** If the CSAF seed file was lost, set `ISSUER_DIRECTORY_SEED_CSAF=true` for the first restart to rehydrate the global tenant.
- **Testing:** Run quarterly restore drills in a staging environment to validate procedure drift.

## Verification checklist
- [ ] `/health/live` returns `200 OK`.
- [ ] Mongo collections (`issuers`, `issuer_keys`, `issuer_trust_overrides`) have expected counts.
- [ ] `issuer_directory_changes_total` and `issuer_directory_key_operations_total` metrics resume within 1 minute.
- [ ] Audit entries exist for post-restore CRUD activity.
- [ ] Client integrations (VEX Lens, Excititor) resolve issuers successfully.