git.stella-ops.org/docs/modules/findings-ledger/deployment.md

# Findings Ledger Deployment & Operations Guide

> **Applies to** `StellaOps.Findings.Ledger` writer + projector services (Sprint 120).  
> **Audience** Platform/DevOps engineers bringing up Findings Ledger across dev/stage/prod and air-gapped sites.

## 1. Prerequisites

| Component | Requirement |
| --- | --- |
| Database | PostgreSQL 14+ with `citext`, `uuid-ossp`, `pgcrypto`, and `pg_partman`. Provision dedicated database/user per environment. |
| Storage | Minimum 200 GB SSD per production environment (ledger + projection + Merkle tables). |
| TLS & identity | Authority reachable for service-to-service JWTs; mTLS optional but recommended. |
| Secrets | Store DB connection string, encryption keys (`LEDGER__ATTACHMENTS__ENCRYPTIONKEY`), signing credentials for Merkle anchoring in secrets manager. |
| Observability | OTLP collector endpoint (or Loki/Prometheus endpoints) configured; see `docs/modules/findings-ledger/observability.md`. |

## 2. Docker Compose deployment

1. **Create env files**
   ```bash
   cp deploy/compose/env/ledger.env.example ledger.env
   cp etc/secrets/ledger.postgres.secret.example ledger.postgres.env
   # Populate LEDGER__DB__CONNECTIONSTRING, LEDGER__ATTACHMENTS__ENCRYPTIONKEY, etc.
   ```
2. **Add ledger service overlay** (append to the Compose file in use, e.g. `docker-compose.prod.yaml`):
   ```yaml
   services:
     findings-ledger:
       image: stellaops/findings-ledger:${STELLA_VERSION:-2025.11.0}
       restart: unless-stopped
       env_file:
         - ledger.env
         - ledger.postgres.env
       environment:
         ASPNETCORE_URLS: http://0.0.0.0:8080
         LEDGER__DB__CONNECTIONSTRING: ${LEDGER__DB__CONNECTIONSTRING}
         LEDGER__OBSERVABILITY__ENABLED: "true"
         LEDGER__MERKLE__ANCHORINTERVAL: "00:05:00"
       ports:
         - "8188:8080"
       depends_on:
         - postgres
       volumes:
         - ./etc/ledger/appsettings.json:/app/appsettings.json:ro
   ```
3. **Run migrations then start services**
   ```bash
   dotnet run --project src/Findings/StellaOps.Findings.Ledger.Migrations \
     -- --connection "$LEDGER__DB__CONNECTIONSTRING"

   docker compose --env-file ledger.env --env-file ledger.postgres.env \
     -f deploy/compose/docker-compose.prod.yaml up -d findings-ledger
   ```
4. **Smoke test**
   ```bash
   curl -sf http://localhost:8188/health/ready
   curl -sf http://localhost:8188/metrics | grep ledger_write_latency_seconds
   ```

## 3. Helm deployment

1. **Create secret**
   ```bash
   kubectl create secret generic findings-ledger-secrets \
     --from-literal=LEDGER__DB__CONNECTIONSTRING="$CONN_STRING" \
     --from-literal=LEDGER__ATTACHMENTS__ENCRYPTIONKEY="$ENC_KEY" \
     --dry-run=client -o yaml | kubectl apply -f -
   ```
2. **Helm values excerpt**
   ```yaml
   services:
     findingsLedger:
       enabled: true
       image:
         repository: stellaops/findings-ledger
         tag: 2025.11.0
       envFromSecrets:
         - name: findings-ledger-secrets
       env:
         LEDGER__OBSERVABILITY__ENABLED: "true"
         LEDGER__MERKLE__ANCHORINTERVAL: "00:05:00"
       resources:
         requests: { cpu: "500m", memory: "1Gi" }
         limits:   { cpu: "2",    memory: "4Gi" }
       probes:
         readinessPath: /health/ready
         livenessPath: /health/live
   ```
3. **Install/upgrade**
   ```bash
   helm upgrade --install stellaops deploy/helm/stellaops \
     -f deploy/helm/stellaops/values-prod.yaml
   ```
4. **Verify**
   ```bash
   kubectl logs deploy/stellaops-findings-ledger | grep "Ledger started"
   kubectl port-forward svc/stellaops-findings-ledger 8080 &
   curl -sf http://127.0.0.1:8080/metrics | head
   ```

## 4. Backups & restores

### 4.1 Backup quickstart (PostgreSQL)

| Task | Command / guidance |
| --- | --- |
| Full dump | `pg_dump -Fc --dbname="$LEDGER_DB" --file ledger-$(date -u +%Y%m%d).dump` (run daily). |
| WAL archiving | `archive_mode=on`, `archive_command='test ! -f /wal/%f && cp %p /wal/%f'`; retain ≥7 days or per policy. |
| Integrity check | `pg_restore -l ledger-YYYYMMDD.dump | head` (validate readable) + verify `ledger_merkle_roots` count matches production before pruning. |

### 4.2 Restore + replay

1. Restore database (full + WAL).  
   ```bash
   pg_restore -C -d postgres ledger-YYYYMMDD.dump
   ```
2. Run projection replay/determinism harness to repopulate projections and validate hashes:
   ```bash
   dotnet run --project src/Findings/tools/LedgerReplayHarness \
     -- --connection "$LEDGER_DB" \
     --tenant all \
     --maxParallel 8 \
     --report out/harness/restore-report.json
   ```
3. Compare Merkle roots: query `select root_hash from ledger_merkle_roots order by anchored_at desc limit 5;` and ensure harness report `merkleRoot` matches latest root.
4. Recreate indexes/materialized views if disabled during restore (see `migrations/` for schema reference).

### 4.3 Evidence & audit artefacts

- Store the following with each backup set:
  - `ledger_merkle_roots` CSV export.
  - Replay harness signed report (`harness-report.json` + `.sig`).
  - Latest Grafana dashboard JSON and alert rules used during the period.
- Keep anchor references (`anchor_reference`) when external anchoring is enabled; include external proof bundle if present.

## 5. Offline / air-gapped workflow

- Use `stella ledger observability snapshot --out offline/ledger/metrics.tar.gz` before exporting Offline Kits. Include:
  - `ledger_write_latency_seconds` summaries
  - `ledger_merkle_anchor_duration_seconds` histogram
  - Latest `ledger_merkle_roots` rows (export via `psql \copy`)
- Package ledger service binaries + migrations using `ops/offline-kit/build_offline_kit.py --include ledger`.
- Document sealed-mode restrictions: disable outbound attachments unless egress policy allows Evidence Locker endpoints; set `LEDGER__ATTACHMENTS__ALLOWEGRESS=false`.

**Approved asset locations (dev/stage/prod + offline kit):**
- Compose overlays: `ops/devops/findings-ledger/compose/` (per-env files e.g., `docker-compose.prod.yaml`, `env/ledger.prod.env`).
- Helm chart overrides: `ops/devops/findings-ledger/helm/` (values per env, secrets templates).
- Offline kit bundle: `ops/devops/findings-ledger/offline-kit/` (binaries, migrations, dashboards, replay harness artefacts).
- Keep module-local examples in this doc; commit deploy artefacts only under the approved `ops/devops/findings-ledger/**` paths.

## 6. Post-deploy checklist

- [ ] Health + metrics endpoints respond.
- [ ] Merkle anchors writing to `ledger_merkle_roots`.
- [ ] Projection lag < 30 s (`ledger_projection_lag_seconds`).
- [ ] Grafana dashboards imported under “Findings Ledger”.
- [ ] Backups scheduled + restore playbook tested.
- [ ] Offline snapshot taken (air-gapped sites).

---

*Draft updated 2025-12-07 for LEDGER-29-009: asset paths approved under `ops/devops/findings-ledger/**`; Compose/Helm/offline-kit overlays should land there.*