consolidation of some of the modules, localization fixes, product advisories work, qa work
This commit is contained in:
master
@@ -0,0 +1,105 @@
|
||||
# Issuer Directory Backup & Restore
|
||||
|
||||
## Scope
|
||||
- **Applies to:** Issuer Directory when deployed via Docker Compose (`devops/compose/docker-compose.*.yaml`) or the Helm chart (`devops/helm/stellaops`).
|
||||
- **Artifacts covered:** PostgreSQL database `issuer_directory`, service configuration (`etc/issuer-directory.yaml`), CSAF seed file (`data/csaf-publishers.json`), and secret material for the PostgreSQL connection string.
|
||||
- **Frequency:** Take a hot backup before every upgrade and at least daily in production. Keep encrypted copies off-site/air-gapped according to your compliance program.
|
||||
|
||||
## Inventory checklist
|
||||
| Component | Location (Compose default) | Notes |
|
||||
| --- | --- | --- |
|
||||
| PostgreSQL data | `postgres-data` volume (`/var/lib/docker/volumes/.../postgres-data`) | Contains `issuers`, `issuer_keys`, `issuer_trust_overrides`, and `issuer_audit` tables in the `issuer_directory` schema. |
|
||||
| Configuration | `etc/issuer-directory.yaml` | Mounted read-only at `/etc/issuer-directory.yaml` inside the container. |
|
||||
| CSAF seed file | `src/IssuerDirectory/StellaOps.IssuerDirectory/data/csaf-publishers.json` | Ensure customised seeds are part of the backup; regenerate if you ship regional overrides. |
|
||||
| PostgreSQL secret | `.env` entry `ISSUER_DIRECTORY_POSTGRES_CONNECTION_STRING` or secret store export | Required to restore connectivity; treat as sensitive. |
|
||||
|
||||
> **Tip:** Export the secret via `kubectl get secret issuer-directory-secrets -o yaml` (sanitize before storage) or copy the Compose `.env` file into an encrypted vault. For PostgreSQL credentials, consider using `pg_dump` with connection info from environment variables.
|
||||
|
||||
## Hot backup (no downtime)
|
||||
1. **Create output directory**
|
||||
```bash
|
||||
BACKUP_DIR=backup/issuer-directory/$(date +%Y-%m-%dT%H%M%S)
|
||||
mkdir -p "$BACKUP_DIR"
|
||||
```
|
||||
2. **Dump PostgreSQL tables**
|
||||
```bash
|
||||
docker compose -f devops/compose/docker-compose.prod.yaml exec postgres \
|
||||
pg_dump --format=custom --compress=9 \
|
||||
--file=/dump/issuer-directory-$(date +%Y%m%dT%H%M%SZ).dump \
|
||||
--schema=issuer_directory issuer_directory
|
||||
|
||||
docker compose -f devops/compose/docker-compose.prod.yaml cp \
|
||||
postgres:/dump/issuer-directory-$(date +%Y%m%dT%H%M%SZ).dump "$BACKUP_DIR/"
|
||||
```
|
||||
For Kubernetes, run the same `pg_dump` command inside the `stellaops-postgres` pod and copy the archive via `kubectl cp`.
|
||||
3. **Capture configuration and seeds**
|
||||
```bash
|
||||
cp etc/issuer-directory.yaml "$BACKUP_DIR/"
|
||||
cp src/IssuerDirectory/StellaOps.IssuerDirectory/data/csaf-publishers.json "$BACKUP_DIR/"
|
||||
```
|
||||
4. **Capture secrets**
|
||||
```bash
|
||||
grep '^ISSUER_DIRECTORY_POSTGRES_CONNECTION_STRING=' dev.env > "$BACKUP_DIR/issuer-directory.postgres.secret"
|
||||
chmod 600 "$BACKUP_DIR/issuer-directory.postgres.secret"
|
||||
```
|
||||
5. **Generate checksums and encrypt**
|
||||
```bash
|
||||
(cd "$BACKUP_DIR" && sha256sum * > SHA256SUMS)
|
||||
tar czf "$BACKUP_DIR.tar.gz" -C "$BACKUP_DIR" .
|
||||
age -r you@example.org "$BACKUP_DIR.tar.gz" > "$BACKUP_DIR.tar.gz.age"
|
||||
```
|
||||
|
||||
## Cold backup (planned downtime)
|
||||
1. Notify stakeholders and pause automation calling the API.
|
||||
2. Stop services:
|
||||
```bash
|
||||
docker compose -f devops/compose/docker-compose.prod.yaml down issuer-directory
|
||||
```
|
||||
(For Helm: `kubectl scale deploy stellaops-issuer-directory --replicas=0`.)
|
||||
3. Snapshot volumes:
|
||||
```bash
|
||||
docker run --rm -v postgres-data:/data \
|
||||
-v "$(pwd)":/backup busybox tar czf /backup/postgres-data-$(date +%Y%m%d).tar.gz -C /data .
|
||||
```
|
||||
4. Copy configuration, seeds, and secrets as in the hot backup.
|
||||
5. Restart services and confirm `/health/live` returns `200 OK`.
|
||||
|
||||
## Restore procedure
|
||||
1. **Provision clean volumes**
|
||||
- Compose: `docker volume rm postgres-data` (optional) then `docker compose up -d postgres`.
|
||||
- Helm: delete the PostgreSQL PVC or attach a fresh volume snapshot.
|
||||
2. **Restore PostgreSQL**
|
||||
```bash
|
||||
docker compose exec -T postgres \
|
||||
pg_restore --format=custom --clean --if-exists \
|
||||
--dbname=issuer_directory < issuer-directory-YYYYMMDDTHHMMSSZ.dump
|
||||
```
|
||||
3. **Restore configuration/secrets**
|
||||
- Copy `issuer-directory.yaml` into `etc/`.
|
||||
- Reapply the secret: `kubectl apply -f issuer-directory-secret.yaml` or repopulate `.env`.
|
||||
4. **Restore CSAF seeds** (optional)
|
||||
- If you maintain a customised seed file, copy it back before starting the container. Otherwise the bundled file will be used.
|
||||
5. **Start services**
|
||||
```bash
|
||||
docker compose up -d issuer-directory
|
||||
# or
|
||||
kubectl scale deploy stellaops-issuer-directory --replicas=1
|
||||
```
|
||||
6. **Validate**
|
||||
- `curl -fsSL https://localhost:8447/health/live`
|
||||
- Issue an access token and list issuers to confirm results.
|
||||
- Check PostgreSQL counts match expectations (`SELECT COUNT(*) FROM issuer_directory.issuers;`, etc.).
|
||||
- Confirm Prometheus scrapes `issuer_directory_changes_total` and `issuer_directory_key_operations_total` for the tenants you restored.
|
||||
|
||||
## Disaster recovery notes
|
||||
- **Retention:** Maintain 30 daily + 12 monthly archives. Store copies in geographically separate, access-controlled vaults.
|
||||
- **Audit reconciliation:** Ensure `issuer_audit` entries cover the restore window; export them for compliance.
|
||||
- **Seed replay:** If the CSAF seed file was lost, set `ISSUER_DIRECTORY_SEED_CSAF=true` for the first restart to rehydrate the global tenant.
|
||||
- **Testing:** Run quarterly restore drills in a staging environment to validate procedure drift.
|
||||
|
||||
## Verification checklist
|
||||
- [ ] `/health/live` returns `200 OK`.
|
||||
- [ ] PostgreSQL tables (`issuers`, `issuer_keys`, `issuer_trust_overrides`) have expected counts.
|
||||
- [ ] `issuer_directory_changes_total`, `issuer_directory_key_operations_total`, and `issuer_directory_key_validation_failures_total` metrics resume within 1 minute.
|
||||
- [ ] Audit entries exist for post-restore CRUD activity.
|
||||
- [ ] Client integrations (VEX Lens, Excititor) resolve issuers successfully.
|
||||
Reference in New Issue
Block a user