CD/CD consolidation

2025-12-26 17:32:23 +02:00
parent a866eb6277
commit c786faae84
638 changed files with 3821 additions and 181 deletions
--- a/devops/deployment/AGENTS.md
+++ b/devops/deployment/AGENTS.md
@@ -0,0 +1,15 @@
+# Deployment & Operations — Agent Charter
+
+## Mission
+Maintain deployment/upgrade/rollback workflows (Helm/Compose) per `docs/modules/devops/ARCHITECTURE.md` including environment-specific configs.
+
+## Required Reading
+- `docs/modules/platform/architecture-overview.md`
+- `docs/modules/airgap/airgap-mode.md`
+
+## Working Agreement
+- 1. Update task status to `DOING`/`DONE` inside the corresponding `docs/implplan/SPRINT_*.md` entry when you start or finish work.
+- 2. Review this charter and the Required Reading documents before coding; confirm prerequisites are met.
+- 3. Keep changes deterministic (stable ordering, timestamps, hashes) and align with offline/air-gap expectations.
+- 4. Coordinate doc updates, tests, and cross-guild communication whenever contracts or workflows change.
+- 5. Revert to `TODO` if you pause the task without shipping changes; leave notes in commit/PR descriptions for context.
--- a/devops/deployment/TASKS.completed.md
+++ b/devops/deployment/TASKS.completed.md
@@ -0,0 +1,5 @@
+# Completed Tasks
+
+| ID | Status | Owner(s) | Depends on | Description | Exit Criteria |
+|----|--------|----------|------------|-------------|---------------|
+| DEVOPS-OPS-14-003 | DONE (2025-10-26) | Deployment Guild | DEVOPS-REL-14-001 | Document and script upgrade/rollback flows, channel management, and compatibility matrices per architecture. | Helm/Compose guides updated with digest pinning, automated checks committed, rollback drill recorded. |
--- a/devops/deployment/advisory-ai/README.md
+++ b/devops/deployment/advisory-ai/README.md
@@ -0,0 +1,91 @@
+# Advisory AI Deployment Runbook
+
+## Scope
+- Helm and Compose packaging for `advisory-ai-web` (API/plan cache) and `advisory-ai-worker` (inference/queue).
+- GPU toggle (NVIDIA) for on-prem inference; defaults remain CPU-safe.
+- Offline kit pickup instructions for including advisory AI artefacts.
+
+## Helm
+Values already ship in `deploy/helm/stellaops/values-*.yaml` under `services.advisory-ai-web` and `advisory-ai-worker`.
+
+GPU enablement (example):
+```yaml
+services:
+  advisory-ai-worker:
+    runtimeClassName: nvidia
+    nodeSelector:
+      nvidia.com/gpu.present: "true"
+    tolerations:
+      - key: nvidia.com/gpu
+        operator: Exists
+        effect: NoSchedule
+    resources:
+      limits:
+        nvidia.com/gpu: 1
+  advisory-ai-web:
+    runtimeClassName: nvidia
+    resources:
+      limits:
+        nvidia.com/gpu: 1
+```
+Apply:
+```bash
+helm upgrade --install stellaops ./deploy/helm/stellaops \
+  -f deploy/helm/stellaops/values-prod.yaml \
+  -f deploy/helm/stellaops/values-mirror.yaml \
+  --set services.advisory-ai-worker.resources.limits.nvidia\.com/gpu=1 \
+  --set services.advisory-ai-worker.runtimeClassName=nvidia
+```
+
+## Compose
+- Base profiles: `docker-compose.dev.yaml`, `stage`, `prod`, `airgap` already include advisory AI services and shared volumes.
+- GPU overlay: `docker-compose.gpu.yaml` (adds NVIDIA device reservations and `ADVISORY_AI_INFERENCE_GPU=true`). Use:
+```bash
+docker compose --env-file prod.env \
+  -f docker-compose.prod.yaml \
+  -f docker-compose.gpu.yaml up -d
+```
+
+## Offline kit pickup
+- Ensure advisory AI images are mirrored to your registry (or baked into airgap tar) before running the offline kit build.
+- Copy the following into `out/offline-kit/metadata/` before invoking the offline kit script:
+  - `advisory-ai-web` image tar
+  - `advisory-ai-worker` image tar
+  - SBOM/provenance generated by the release pipeline
+- Verify `docs/24_OFFLINE_KIT.md` includes the advisory AI entries and rerun `tests/offline/test_build_offline_kit.py` if it changes.
+
+## Runbook (prod quickstart)
+1) Prepare secrets in ExternalSecret or Kubernetes secret named `stellaops-prod-core` (see helm values).
+2) Run Helm install with prod values and GPU overrides as needed.
+3) For Compose, use `prod.env` and optionally `docker-compose.gpu.yaml` overlay.
+4) Validate health:
+   - `GET /healthz` on `advisory-ai-web`
+   - Check queue directories under `advisory-ai-*` volumes remain writable
+   - Confirm inference path logs when GPU is detected (log key `advisory.ai.inference.gpu=true`).
+
+## Advisory Feed Packaging (DEVOPS-AIAI-31-002)
+
+Package advisory feeds (SBOM pointers + provenance) for release/offline kit:
+
+```bash
+# Production (CI with COSIGN_PRIVATE_KEY_B64 secret)
+./ops/deployment/advisory-ai/package-advisory-feeds.sh
+
+# Development (uses tools/cosign/cosign.dev.key)
+COSIGN_ALLOW_DEV_KEY=1 COSIGN_PASSWORD=stellaops-dev \
+  ./ops/deployment/advisory-ai/package-advisory-feeds.sh
+```
+
+Outputs:
+- `out/advisory-ai/feeds/advisory-feeds.tar.gz` - Feed bundle
+- `out/advisory-ai/feeds/advisory-feeds.manifest.json` - Manifest with SBOM pointers
+- `out/advisory-ai/feeds/advisory-feeds.manifest.dsse.json` - DSSE signed manifest
+- `out/advisory-ai/feeds/provenance.json` - Build provenance
+
+CI workflow: `.gitea/workflows/advisory-ai-release.yml`
+
+## Evidence to attach (sprint)
+- Helm release output (rendered templates for advisory AI)
+- `docker-compose config` with/without GPU overlay
+- Offline kit metadata listing advisory AI images + SBOMs
+- Advisory feed package manifest with SBOM pointers
--- a/devops/deployment/advisory-ai/package-advisory-feeds.sh
+++ b/devops/deployment/advisory-ai/package-advisory-feeds.sh
@@ -0,0 +1,165 @@
+#!/usr/bin/env bash
+# Package advisory feeds (SBOM pointers + provenance) for release/offline kit
+# Usage: ./package-advisory-feeds.sh
+# Dev mode: COSIGN_ALLOW_DEV_KEY=1 COSIGN_PASSWORD=stellaops-dev ./package-advisory-feeds.sh
+
+set -euo pipefail
+
+ROOT=$(cd "$(dirname "$0")/../../.." && pwd)
+OUT_DIR="${OUT_DIR:-$ROOT/out/advisory-ai/feeds}"
+CREATED="${CREATED:-$(date -u +%Y-%m-%dT%H:%M:%SZ)}"
+
+mkdir -p "$OUT_DIR"
+
+# Key resolution (same pattern as tools/cosign/sign-signals.sh)
+resolve_key() {
+  if [[ -n "${COSIGN_KEY_FILE:-}" && -f "$COSIGN_KEY_FILE" ]]; then
+    echo "$COSIGN_KEY_FILE"
+  elif [[ -n "${COSIGN_PRIVATE_KEY_B64:-}" ]]; then
+    local tmp_key="$OUT_DIR/.cosign.key"
+    echo "$COSIGN_PRIVATE_KEY_B64" | base64 -d > "$tmp_key"
+    chmod 600 "$tmp_key"
+    echo "$tmp_key"
+  elif [[ -f "$ROOT/tools/cosign/cosign.key" ]]; then
+    echo "$ROOT/tools/cosign/cosign.key"
+  elif [[ "${COSIGN_ALLOW_DEV_KEY:-0}" == "1" && -f "$ROOT/tools/cosign/cosign.dev.key" ]]; then
+    echo "[info] Using development key (non-production)" >&2
+    echo "$ROOT/tools/cosign/cosign.dev.key"
+  else
+    echo "[error] No signing key available. Set COSIGN_PRIVATE_KEY_B64 or COSIGN_ALLOW_DEV_KEY=1" >&2
+    return 1
+  fi
+}
+
+KEY_FILE=$(resolve_key)
+
+# Collect advisory feed sources
+FEED_SOURCES=(
+  "$ROOT/docs/samples/advisory-feeds"
+  "$ROOT/src/AdvisoryAI/feeds"
+  "$ROOT/out/feeds"
+)
+
+echo "==> Collecting advisory feeds..."
+STAGE_DIR="$OUT_DIR/stage"
+mkdir -p "$STAGE_DIR"
+
+for src in "${FEED_SOURCES[@]}"; do
+  if [[ -d "$src" ]]; then
+    echo "    Adding feeds from $src"
+    cp -r "$src"/* "$STAGE_DIR/" 2>/dev/null || true
+  fi
+done
+
+# Create placeholder if no feeds found (dev mode)
+if [[ -z "$(ls -A "$STAGE_DIR" 2>/dev/null)" ]]; then
+  echo "[info] No feed sources found; creating placeholder for dev mode"
+  cat > "$STAGE_DIR/placeholder.json" <<EOF
+{
+  "type": "advisory-feed-placeholder",
+  "created": "$CREATED",
+  "note": "Placeholder for development; replace with real feeds in production"
+}
+EOF
+fi
+
+# Create feed bundle
+echo "==> Creating feed bundle..."
+BUNDLE_TAR="$OUT_DIR/advisory-feeds.tar.gz"
+tar -czf "$BUNDLE_TAR" -C "$STAGE_DIR" .
+
+# Compute hashes
+sha256() {
+  sha256sum "$1" | awk '{print $1}'
+}
+
+BUNDLE_HASH=$(sha256 "$BUNDLE_TAR")
+
+# Generate manifest with SBOM pointers
+echo "==> Generating manifest..."
+MANIFEST="$OUT_DIR/advisory-feeds.manifest.json"
+cat > "$MANIFEST" <<EOF
+{
+  "schemaVersion": "1.0.0",
+  "created": "$CREATED",
+  "bundle": {
+    "path": "advisory-feeds.tar.gz",
+    "sha256": "$BUNDLE_HASH",
+    "size": $(stat -c%s "$BUNDLE_TAR" 2>/dev/null || stat -f%z "$BUNDLE_TAR")
+  },
+  "sbom": {
+    "format": "spdx-json",
+    "path": "advisory-feeds.sbom.json",
+    "note": "SBOM generated during CI; pointer only in manifest"
+  },
+  "provenance": {
+    "path": "provenance.json",
+    "builder": "stellaops-advisory-ai-release"
+  }
+}
+EOF
+
+# Sign manifest with DSSE
+echo "==> Signing manifest..."
+DSSE_OUT="$OUT_DIR/advisory-feeds.manifest.dsse.json"
+
+# Check for cosign
+COSIGN="${COSIGN:-$ROOT/tools/cosign/cosign}"
+if ! command -v cosign &>/dev/null && [[ ! -x "$COSIGN" ]]; then
+  echo "[warn] cosign not found; skipping DSSE signing" >&2
+else
+  COSIGN_CMD="${COSIGN:-cosign}"
+  if command -v cosign &>/dev/null; then
+    COSIGN_CMD="cosign"
+  fi
+
+  COSIGN_PASSWORD="${COSIGN_PASSWORD:-}" "$COSIGN_CMD" sign-blob \
+    --key "$KEY_FILE" \
+    --bundle "$DSSE_OUT" \
+    --tlog-upload=false \
+    --yes \
+    "$MANIFEST" 2>/dev/null || echo "[warn] DSSE signing skipped (cosign error)"
+fi
+
+# Generate provenance
+echo "==> Generating provenance..."
+PROVENANCE="$OUT_DIR/provenance.json"
+cat > "$PROVENANCE" <<EOF
+{
+  "_type": "https://in-toto.io/Statement/v1",
+  "subject": [
+    {
+      "name": "advisory-feeds.tar.gz",
+      "digest": {"sha256": "$BUNDLE_HASH"}
+    }
+  ],
+  "predicateType": "https://slsa.dev/provenance/v1",
+  "predicate": {
+    "buildDefinition": {
+      "buildType": "https://stella-ops.org/advisory-ai-release/v1",
+      "externalParameters": {},
+      "internalParameters": {
+        "created": "$CREATED"
+      }
+    },
+    "runDetails": {
+      "builder": {
+        "id": "https://stella-ops.org/advisory-ai-release"
+      },
+      "metadata": {
+        "invocationId": "$(uuidgen 2>/dev/null || echo "dev-$(date +%s)")",
+        "startedOn": "$CREATED"
+      }
+    }
+  }
+}
+EOF
+
+# Cleanup temp key
+[[ -f "$OUT_DIR/.cosign.key" ]] && rm -f "$OUT_DIR/.cosign.key"
+
+echo "==> Advisory feed packaging complete"
+echo "    Bundle: $BUNDLE_TAR"
+echo "    Manifest: $MANIFEST"
+echo "    DSSE: $DSSE_OUT"
+echo "    Provenance: $PROVENANCE"
--- a/devops/deployment/cli/README.md
+++ b/devops/deployment/cli/README.md
@@ -0,0 +1,107 @@
+# StellaOps CLI Release Packaging
+
+## Scope
+- Package and publish StellaOps CLI binaries for all supported OS/arch targets with checksums, signatures, completions, and a container image.
+- Outputs feed three lanes: (1) public release mirrors, (2) air-gapped/offline kit, (3) internal regression runners.
+- Source artefacts come from DevOps pipelines (`.gitea/workflows/cli-build.yml`, `.gitea/workflows/cli-chaos-parity.yml`).
+
+## Inputs (expected layout)
+```
+out/cli/<version>/
+  stella-cli-linux-amd64.tar.gz
+  stella-cli-linux-arm64.tar.gz
+  stella-cli-darwin-arm64.tar.gz
+  stella-cli-windows-amd64.zip
+  completions/
+    bash/stella
+    zsh/_stella
+    fish/stella.fish
+  parity/
+    parity-report.json
+  sbom/
+    stella-cli.spdx.json
+```
+`<version>` must match the git tag and container tag (e.g., `2025.12.0`).
+
+## Packaging steps (deterministic)
+1) Set version and workdir
+```bash
+export CLI_VERSION=2025.12.0
+export CLI_OUT=out/cli/$CLI_VERSION
+```
+
+2) Generate checksums (sorted, LF endings)
+```bash
+cd "$CLI_OUT"
+find . -maxdepth 1 -type f \( -name 'stella-cli-*' -o -name '*.zip' \) \
+  -print0 | sort -z | xargs -0 sha256sum > SHA256SUMS
+```
+
+3) Sign checksum file (cosign keyless or key)
+```bash
+COSIGN_YES=true cosign sign-blob \
+  --key env://MIRROR_SIGN_KEY_B64 \
+  --output-signature SHA256SUMS.sig \
+  --output-certificate SHA256SUMS.pem \
+  SHA256SUMS
+```
+
+4) Build/push container image (optional if pipeline already produced)
+```bash
+docker build -t registry.local/stella/cli:$CLI_VERSION -f deploy/compose/cli/Dockerfile .
+docker push registry.local/stella/cli:$CLI_VERSION
+```
+
+5) Produce offline image tar (for airgap kit)
+```bash
+docker pull registry.local/stella/cli:$CLI_VERSION
+docker save registry.local/stella/cli:$CLI_VERSION \
+  | gzip -9 > stella-cli-image-$CLI_VERSION.tar.gz
+```
+
+6) Bundle completions
+```bash
+tar -C "$CLI_OUT/completions" -czf stella-cli-completions-$CLI_VERSION.tar.gz .
+```
+
+7) Publish artefact manifest (for mirrors/offline kit)
+```bash
+cat > release-manifest-$CLI_VERSION.json <<'EOF'
+{
+  "version": "REPLACE_VERSION",
+  "binaries": [
+    "stella-cli-linux-amd64.tar.gz",
+    "stella-cli-linux-arm64.tar.gz",
+    "stella-cli-darwin-arm64.tar.gz",
+    "stella-cli-windows-amd64.zip"
+  ],
+  "completions": "stella-cli-completions-REPLACE_VERSION.tar.gz",
+  "checksums": "SHA256SUMS",
+  "signatures": ["SHA256SUMS.sig", "SHA256SUMS.pem"],
+  "container": {
+    "image": "registry.local/stella/cli:REPLACE_VERSION",
+    "offline_tar": "stella-cli-image-REPLACE_VERSION.tar.gz"
+  }
+}
+EOF
+sed -i "s/REPLACE_VERSION/$CLI_VERSION/g" release-manifest-$CLI_VERSION.json
+```
+
+## Distribution lanes
+- **Mirror / public:** upload binaries, completions, SBOM, `SHA256SUMS*`, and `release-manifest-<version>.json` to the mirror bucket; expose via CDN.
+- **Offline kit:** copy the same files plus `stella-cli-image-<version>.tar.gz` into `out/offline-kit/cli/` before running `ops/offline-kit/scripts/build_offline_kit.sh`.
+- **Internal runners:** sync `SHA256SUMS` and `SHA256SUMS.sig` to the runner cache; store container tar in the runner image cache path.
+
+## Verification
+```bash
+cd "$CLI_OUT"
+sha256sum --check SHA256SUMS
+cosign verify-blob --key env://MIRROR_SIGN_KEY_B64 --signature SHA256SUMS.sig --certificate SHA256SUMS.pem SHA256SUMS
+```
+
+## Rollback / re-spin
+- To revoke a bad drop, delete the mirror path for that version and reissue `release-manifest-<version>.json` with `"revoked": true` field; keep signatures for audit.
+- Re-spin by rerunning steps with a new version tag; never overwrite artefacts in-place.
+
+## Evidence to attach in sprint
+- `SHA256SUMS`, `SHA256SUMS.sig`, `release-manifest-<version>.json`, and offline image tar path uploaded to sprint evidence locker.
--- a/devops/deployment/export/helm-overlays.md
+++ b/devops/deployment/export/helm-overlays.md
@@ -0,0 +1,35 @@
+# Export Center Helm Overlays (DEPLOY-EXPORT-35-001)
+
+## Values files (download-only)
+- `deploy/helm/stellaops/values-export.yaml` (add) with:
+  - `exportcenter:`
+    - `image.repository`: `registry.stella-ops.org/export-center`
+    - `image.tag`: set via pipeline
+    - `objectStorage.endpoint`: `http://minio:9000`
+    - `objectStorage.bucket`: `export-prod`
+    - `objectStorage.accessKeySecret`: `exportcenter-minio`
+    - `objectStorage.secretKeySecret`: `exportcenter-minio`
+    - `signing.kmsKey`: `exportcenter-kms`
+    - `signing.kmsRegion`: `us-east-1`
+    - `dsse.enabled`: true
+
+## Secrets
+- KMS signing: create secret `exportcenter-kms` with JSON key material (KMS provider specific). Example: `ops/deployment/export/secrets-example.yaml`.
+- MinIO creds: `exportcenter-minio` with `accesskey`, `secretkey` keys (see example manifest).
+
+## Rollout
+- `helm upgrade --install export-center deploy/helm/stellaops -f deploy/helm/stellaops/values-export.yaml --set image.tag=$TAG`
+- Pre-flight: `helm template ...` and `helm lint`.
+- Post: verify readiness `kubectl rollout status deploy/export-center` and run `curl /healthz`.
+
+## Rollback
+- `helm rollback export-center <rev>`; ensure previous tag exists.
+
+## Required artefacts
+- Signed images + provenance (from release pipeline).
+- SBOM attached via registry (cosign attestations acceptable).
+
+## Acceptance
+- Overlay renders without missing values.
+- Secrets documented and referenced in template.
+- Rollout/rollback steps documented.
--- a/devops/deployment/export/secrets-example.yaml
+++ b/devops/deployment/export/secrets-example.yaml
@@ -0,0 +1,15 @@
+apiVersion: v1
+kind: Secret
+metadata:
+  name: exportcenter-minio
+stringData:
+  accesskey: REPLACE_ME
+  secretkey: REPLACE_ME
+---
+apiVersion: v1
+kind: Secret
+metadata:
+  name: exportcenter-kms
+stringData:
+  key.json: |
+    {"kmsProvider":"awskms","keyId":"arn:aws:kms:...","region":"us-east-1"}
--- a/devops/deployment/notify/helm-overlays.md
+++ b/devops/deployment/notify/helm-overlays.md
@@ -0,0 +1,28 @@
+# Notifier Helm Overlays (DEPLOY-NOTIFY-38-001)
+
+## Values file
+- `deploy/helm/stellaops/values-notify.yaml` (added) with:
+  - `notify:`
+    - `image.repository`: `registry.stella-ops.org/notify`
+    - `image.tag`: set by pipeline
+    - `smtp.host`, `smtp.port`, `smtp.usernameSecret`, `smtp.passwordSecret`
+    - `webhook.allowedHosts`: list
+    - `chat.webhookSecret`: secret name for chat tokens
+    - `tls.secretName`: optional ingress cert
+
+## Secrets
+- SMTP creds secret `notify-smtp` with keys `username`, `password` (see `ops/deployment/notify/secrets-example.yaml`).
+- Chat/webhook secret `notify-chat` with key `token` (see example manifest).
+
+## Rollout
+- `helm upgrade --install notify deploy/helm/stellaops -f deploy/helm/stellaops/values-notify.yaml --set image.tag=$TAG`
+- Pre-flight: `helm lint`, `helm template`.
+- Post: `kubectl rollout status deploy/notify` and `curl /healthz`.
+
+## Rollback
+- `helm rollback notify <rev>`; confirm previous image tag exists.
+
+## Acceptance
+- Overlay renders without missing values.
+- Secrets documented and referenced.
+- Rollout/rollback steps documented.
--- a/devops/deployment/notify/secrets-example.yaml
+++ b/devops/deployment/notify/secrets-example.yaml
@@ -0,0 +1,14 @@
+apiVersion: v1
+kind: Secret
+metadata:
+  name: notify-smtp
+stringData:
+  username: REPLACE_ME
+  password: REPLACE_ME
+---
+apiVersion: v1
+kind: Secret
+metadata:
+  name: notify-chat
+stringData:
+  token: REPLACE_ME