up

ops/devops/postgres/README.md

@@ -0,0 +1,66 @@
+# PostgreSQL 16 Cluster (staging / production)
+
+This directory provisions StellaOps PostgreSQL clusters with **CloudNativePG (CNPG)**. It is pinned to Postgres 16.x, includes connection pooling (PgBouncer), Prometheus scraping, and S3-compatible backups. Everything is air-gap friendly: fetch the operator and images once, then render/apply manifests offline.
+
+## Targets
+- **Staging:** `stellaops-pg-stg` (2 instances, 200 Gi data, WAL 64 Gi, PgBouncer x2)
+- **Production:** `stellaops-pg-prod` (3 instances, 500 Gi data, WAL 128 Gi, PgBouncer x3)
+- **Namespace:** `platform-postgres`
+
+## Prerequisites
+- Kubernetes ≥ 1.27 with CSI storage classes `fast-ssd` (data) and `fast-wal` (WAL) available.
+- CloudNativePG operator 1.23.x mirrored or downloaded to `artifacts/cloudnative-pg-1.23.0.yaml`.
+- Images mirrored to your registry (example tags):
+  - `ghcr.io/cloudnative-pg/postgresql:16.4`
+  - `ghcr.io/cloudnative-pg/postgresql-operator:1.23.0`
+  - `ghcr.io/cloudnative-pg/pgbouncer:1.23.0`
+- Secrets created from the templates under `ops/devops/postgres/secrets/` (superuser, app user, backup credentials).
+
+## Render & Apply (deterministic)
+```bash
+# 1) Create namespace
+kubectl apply -f ops/devops/postgres/namespace.yaml
+
+# 2) Install operator (offline-friendly: use the pinned manifest you mirrored)
+kubectl apply -f artifacts/cloudnative-pg-1.23.0.yaml
+
+# 3) Create secrets (replace passwords/keys first)
+kubectl apply -f ops/devops/postgres/secrets/example-superuser.yaml
+kubectl apply -f ops/devops/postgres/secrets/example-app.yaml
+kubectl apply -f ops/devops/postgres/secrets/example-backup-credentials.yaml
+
+# 4) Apply the cluster and pooler for the target environment
+kubectl apply -f ops/devops/postgres/cluster-staging.yaml
+kubectl apply -f ops/devops/postgres/pooler-staging.yaml
+# or
+kubectl apply -f ops/devops/postgres/cluster-production.yaml
+kubectl apply -f ops/devops/postgres/pooler-production.yaml
+```
+
+## Connection Endpoints
+- RW service: `<cluster>-rw` (e.g., `stellaops-pg-stg-rw:5432`)
+- RO service: `<cluster>-ro`
+- PgBouncer pooler: `<pooler-name>` (e.g., `stellaops-pg-stg-pooler:6432`)
+
+**Application connection string (matches library defaults):**
+`Host=stellaops-pg-stg-pooler;Port=6432;Username=stellaops_app;Password=<app-password>;Database=stellaops;Pooling=true;Timeout=15;CommandTimeout=30;Ssl Mode=Require;`
+
+## Monitoring & Backups
+- `monitoring.enablePodMonitor: true` exposes PodMonitor for Prometheus Operator.
+- Barman/S3 backups are enabled by default; set `backup.barmanObjectStore.destinationPath` per env and populate `stellaops-pg-backup` credentials.
+- WAL compression is `gzip`; retention is operator-managed (configure via Barman bucket policies).
+
+## Alignment with code defaults
+- Session settings: UTC timezone, 30s `statement_timeout`, tenant context via `set_config('app.current_tenant', ...)`.
+- Connection pooler uses **transaction** mode with a `server_reset_query` that clears session state, keeping RepositoryBase deterministic.
+
+## Verification checklist
+- `kubectl get cluster -n platform-postgres` shows `Ready` replicas matching `instances`.
+- `kubectl logs deploy/cnpg-controller-manager -n cnpg-system` has no failing webhooks.
+- `kubectl get podmonitor -n platform-postgres` returns entries for the cluster and pooler.
+- `psql "<rw-connection-string>" -c 'select 1'` works from CI runner subnet.
+- `cnpg` `barman-cloud-backup-list` shows successful full + WAL backups.
+
+## Offline notes
+- Mirror the operator manifest and container images to the approved registry first; no live downloads occur at runtime.
+- If Prometheus is not present, leave PodMonitor applied; it is inert without the CRD.

ops/devops/postgres/cluster-production.yaml

@@ -0,0 +1,57 @@
+apiVersion: postgresql.cnpg.io/v1
+kind: Cluster
+metadata:
+  name: stellaops-pg-prod
+  namespace: platform-postgres
+spec:
+  instances: 3
+  imageName: ghcr.io/cloudnative-pg/postgresql:16.4
+  primaryUpdateStrategy: unsupervised
+  storage:
+    size: 500Gi
+    storageClass: fast-ssd
+  walStorage:
+    size: 128Gi
+    storageClass: fast-wal
+  superuserSecret:
+    name: stellaops-pg-superuser
+  bootstrap:
+    initdb:
+      database: stellaops
+      owner: stellaops_app
+      secret:
+        name: stellaops-pg-app
+  monitoring:
+    enablePodMonitor: true
+  postgresql:
+    parameters:
+      max_connections: "900"
+      shared_buffers: "4096MB"
+      work_mem: "96MB"
+      maintenance_work_mem: "768MB"
+      wal_level: "replica"
+      max_wal_size: "4GB"
+      timezone: "UTC"
+      log_min_duration_statement: "250"
+      statement_timeout: "30000"
+  resources:
+    requests:
+      cpu: "4"
+      memory: "16Gi"
+    limits:
+      cpu: "8"
+      memory: "24Gi"
+  backup:
+    barmanObjectStore:
+      destinationPath: s3://stellaops-backups/production
+      s3Credentials:
+        accessKeyId:
+          name: stellaops-pg-backup
+          key: ACCESS_KEY_ID
+        secretAccessKey:
+          name: stellaops-pg-backup
+          key: SECRET_ACCESS_KEY
+      wal:
+        compression: gzip
+        maxParallel: 4
+  logLevel: info

ops/devops/postgres/cluster-staging.yaml

@@ -0,0 +1,57 @@
+apiVersion: postgresql.cnpg.io/v1
+kind: Cluster
+metadata:
+  name: stellaops-pg-stg
+  namespace: platform-postgres
+spec:
+  instances: 2
+  imageName: ghcr.io/cloudnative-pg/postgresql:16.4
+  primaryUpdateStrategy: unsupervised
+  storage:
+    size: 200Gi
+    storageClass: fast-ssd
+  walStorage:
+    size: 64Gi
+    storageClass: fast-wal
+  superuserSecret:
+    name: stellaops-pg-superuser
+  bootstrap:
+    initdb:
+      database: stellaops
+      owner: stellaops_app
+      secret:
+        name: stellaops-pg-app
+  monitoring:
+    enablePodMonitor: true
+  postgresql:
+    parameters:
+      max_connections: "600"
+      shared_buffers: "2048MB"
+      work_mem: "64MB"
+      maintenance_work_mem: "512MB"
+      wal_level: "replica"
+      max_wal_size: "2GB"
+      timezone: "UTC"
+      log_min_duration_statement: "500"
+      statement_timeout: "30000"
+  resources:
+    requests:
+      cpu: "2"
+      memory: "8Gi"
+    limits:
+      cpu: "4"
+      memory: "12Gi"
+  backup:
+    barmanObjectStore:
+      destinationPath: s3://stellaops-backups/staging
+      s3Credentials:
+        accessKeyId:
+          name: stellaops-pg-backup
+          key: ACCESS_KEY_ID
+        secretAccessKey:
+          name: stellaops-pg-backup
+          key: SECRET_ACCESS_KEY
+      wal:
+        compression: gzip
+        maxParallel: 2
+  logLevel: info

ops/devops/postgres/namespace.yaml

@@ -0,0 +1,4 @@
+apiVersion: v1
+kind: Namespace
+metadata:
+  name: platform-postgres

ops/devops/postgres/pooler-production.yaml

@@ -0,0 +1,29 @@
+apiVersion: postgresql.cnpg.io/v1
+kind: Pooler
+metadata:
+  name: stellaops-pg-prod-pooler
+  namespace: platform-postgres
+spec:
+  cluster:
+    name: stellaops-pg-prod
+  instances: 3
+  type: rw
+  pgbouncer:
+    parameters:
+      pool_mode: transaction
+      max_client_conn: "1500"
+      default_pool_size: "80"
+      server_reset_query: "RESET ALL; SET SESSION AUTHORIZATION DEFAULT; SET TIME ZONE 'UTC';"
+    authQuerySecret:
+      name: stellaops-pg-app
+  template:
+    spec:
+      containers:
+        - name: pgbouncer
+          resources:
+            requests:
+              cpu: "150m"
+              memory: "192Mi"
+            limits:
+              cpu: "750m"
+              memory: "384Mi"

ops/devops/postgres/pooler-staging.yaml

@@ -0,0 +1,29 @@
+apiVersion: postgresql.cnpg.io/v1
+kind: Pooler
+metadata:
+  name: stellaops-pg-stg-pooler
+  namespace: platform-postgres
+spec:
+  cluster:
+    name: stellaops-pg-stg
+  instances: 2
+  type: rw
+  pgbouncer:
+    parameters:
+      pool_mode: transaction
+      max_client_conn: "800"
+      default_pool_size: "50"
+      server_reset_query: "RESET ALL; SET SESSION AUTHORIZATION DEFAULT; SET TIME ZONE 'UTC';"
+    authQuerySecret:
+      name: stellaops-pg-app
+  template:
+    spec:
+      containers:
+        - name: pgbouncer
+          resources:
+            requests:
+              cpu: "100m"
+              memory: "128Mi"
+            limits:
+              cpu: "500m"
+              memory: "256Mi"

ops/devops/postgres/secrets/example-app.yaml

@@ -0,0 +1,9 @@
+apiVersion: v1
+kind: Secret
+metadata:
+  name: stellaops-pg-app
+  namespace: platform-postgres
+type: kubernetes.io/basic-auth
+stringData:
+  username: stellaops_app
+  password: CHANGEME_APP_PASSWORD

ops/devops/postgres/secrets/example-backup-credentials.yaml

@@ -0,0 +1,9 @@
+apiVersion: v1
+kind: Secret
+metadata:
+  name: stellaops-pg-backup
+  namespace: platform-postgres
+type: Opaque
+stringData:
+  ACCESS_KEY_ID: CHANGEME_ACCESS_KEY
+  SECRET_ACCESS_KEY: CHANGEME_SECRET_KEY

ops/devops/postgres/secrets/example-superuser.yaml

@@ -0,0 +1,9 @@
+apiVersion: v1
+kind: Secret
+metadata:
+  name: stellaops-pg-superuser
+  namespace: platform-postgres
+type: kubernetes.io/basic-auth
+stringData:
+  username: postgres
+  password: CHANGEME_SUPERUSER_PASSWORD