git.stella-ops.org/docs/modules/findings-ledger/deployment.md

Findings Ledger Deployment & Operations Guide

Applies to StellaOps.Findings.Ledger writer + projector services (Sprint 120).
Audience Platform/DevOps engineers bringing up Findings Ledger across dev/stage/prod and air-gapped sites.

1. Prerequisites

Component	Requirement
Database	PostgreSQL 14+ with citext, uuid-ossp, pgcrypto, and pg_partman. Provision dedicated database/user per environment.
Storage	Minimum 200 GB SSD per production environment (ledger + projection + Merkle tables).
TLS & identity	Authority reachable for service-to-service JWTs; mTLS optional but recommended.
Secrets	Store DB connection string, encryption keys (LEDGER__ATTACHMENTS__ENCRYPTIONKEY), signing credentials for Merkle anchoring in secrets manager.
Observability	OTLP collector endpoint (or Loki/Prometheus endpoints) configured; see docs/modules/findings-ledger/observability.md.

2. Docker Compose deployment

1. Create env files

   cp deploy/compose/env/ledger.env.example ledger.env
   cp etc/secrets/ledger.postgres.secret.example ledger.postgres.env
   # Populate LEDGER__DB__CONNECTIONSTRING, LEDGER__ATTACHMENTS__ENCRYPTIONKEY, etc.
   

2. Add ledger service overlay (append to the Compose file in use, e.g. docker-compose.prod.yaml):

   services:
     findings-ledger:
       image: stellaops/findings-ledger:${STELLA_VERSION:-2025.11.0}
       restart: unless-stopped
       env_file:
         - ledger.env
         - ledger.postgres.env
       environment:
         ASPNETCORE_URLS: http://0.0.0.0:8080
         LEDGER__DB__CONNECTIONSTRING: ${LEDGER__DB__CONNECTIONSTRING}
         LEDGER__OBSERVABILITY__ENABLED: "true"
         LEDGER__MERKLE__ANCHORINTERVAL: "00:05:00"
       ports:
         - "8188:8080"
       depends_on:
         - postgres
       volumes:
         - ./etc/ledger/appsettings.json:/app/appsettings.json:ro
   

3. Run migrations then start services

   dotnet run --project src/Findings/StellaOps.Findings.Ledger.Migrations \
     -- --connection "$LEDGER__DB__CONNECTIONSTRING"
   
   docker compose --env-file ledger.env --env-file ledger.postgres.env \
     -f deploy/compose/docker-compose.prod.yaml up -d findings-ledger
   

4. Smoke test

   curl -sf http://localhost:8188/health/ready
   curl -sf http://localhost:8188/metrics | grep ledger_write_latency_seconds
   

3. Helm deployment

1. Create secret

   kubectl create secret generic findings-ledger-secrets \
     --from-literal=LEDGER__DB__CONNECTIONSTRING="$CONN_STRING" \
     --from-literal=LEDGER__ATTACHMENTS__ENCRYPTIONKEY="$ENC_KEY" \
     --dry-run=client -o yaml | kubectl apply -f -
   

2. Helm values excerpt

   services:
     findingsLedger:
       enabled: true
       image:
         repository: stellaops/findings-ledger
         tag: 2025.11.0
       envFromSecrets:
         - name: findings-ledger-secrets
       env:
         LEDGER__OBSERVABILITY__ENABLED: "true"
         LEDGER__MERKLE__ANCHORINTERVAL: "00:05:00"
       resources:
         requests: { cpu: "500m", memory: "1Gi" }
         limits:   { cpu: "2",    memory: "4Gi" }
       probes:
         readinessPath: /health/ready
         livenessPath: /health/live
   

3. Install/upgrade

   helm upgrade --install stellaops deploy/helm/stellaops \
     -f deploy/helm/stellaops/values-prod.yaml
   

4. Verify

   kubectl logs deploy/stellaops-findings-ledger | grep "Ledger started"
   kubectl port-forward svc/stellaops-findings-ledger 8080 &
   curl -sf http://127.0.0.1:8080/metrics | head
   

4. Backups & restores

4.1 Backup quickstart (PostgreSQL)

Task	Command / guidance
Full dump	pg_dump -Fc --dbname="$LEDGER_DB" --file ledger-$(date -u +%Y%m%d).dump (run daily).
WAL archiving	archive_mode=on, archive_command='test ! -f /wal/%f && cp %p /wal/%f'; retain ≥7 days or per policy.
Integrity check	`pg_restore -l ledger-YYYYMMDD.dump

4.2 Restore + replay

1. Restore database (full + WAL).

   pg_restore -C -d postgres ledger-YYYYMMDD.dump
   

2. Run projection replay/determinism harness to repopulate projections and validate hashes:

   dotnet run --project src/Findings/tools/LedgerReplayHarness \
     -- --connection "$LEDGER_DB" \
     --tenant all \
     --maxParallel 8 \
     --report out/harness/restore-report.json
   

3. Compare Merkle roots: query select root_hash from ledger_merkle_roots order by anchored_at desc limit 5; and ensure harness report merkleRoot matches latest root.
4. Recreate indexes/materialized views if disabled during restore (see migrations/ for schema reference).

4.3 Evidence & audit artefacts

• Store the following with each backup set:
  • ledger_merkle_roots CSV export.
  • Replay harness signed report (harness-report.json + .sig).
  • Latest Grafana dashboard JSON and alert rules used during the period.
• Keep anchor references (anchor_reference) when external anchoring is enabled; include external proof bundle if present.

5. Offline / air-gapped workflow

• Use stella ledger observability snapshot --out offline/ledger/metrics.tar.gz before exporting Offline Kits. Include:
  • ledger_write_latency_seconds summaries
  • ledger_merkle_anchor_duration_seconds histogram
  • Latest ledger_merkle_roots rows (export via psql \copy)
• Package ledger service binaries + migrations using ops/offline-kit/build_offline_kit.py --include ledger.
• Document sealed-mode restrictions: disable outbound attachments unless egress policy allows Evidence Locker endpoints; set LEDGER__ATTACHMENTS__ALLOWEGRESS=false.

Approved asset locations (dev/stage/prod + offline kit):

• Compose overlays: ops/devops/findings-ledger/compose/ (per-env files e.g., docker-compose.prod.yaml, env/ledger.prod.env).
• Helm chart overrides: ops/devops/findings-ledger/helm/ (values per env, secrets templates).
• Offline kit bundle: ops/devops/findings-ledger/offline-kit/ (binaries, migrations, dashboards, replay harness artefacts).
• Keep module-local examples in this doc; commit deploy artefacts only under the approved ops/devops/findings-ledger/** paths.

6. Post-deploy checklist

• Health + metrics endpoints respond.
• Merkle anchors writing to ledger_merkle_roots.
• Projection lag < 30 s (ledger_projection_lag_seconds).
• Grafana dashboards imported under “Findings Ledger”.
• Backups scheduled + restore playbook tested.
• Offline snapshot taken (air-gapped sites).

---

Draft updated 2025-12-07 for LEDGER-29-009: asset paths approved under ops/devops/findings-ledger/**; Compose/Helm/offline-kit overlays should land there.