CD/CD consolidation

2025-12-26 17:32:23 +02:00
parent a866eb6277
commit c786faae84
638 changed files with 3821 additions and 181 deletions
--- a/devops/deployment/advisory-ai/README.md
+++ b/devops/deployment/advisory-ai/README.md
@@ -0,0 +1,91 @@
+# Advisory AI Deployment Runbook
+
+## Scope
+- Helm and Compose packaging for `advisory-ai-web` (API/plan cache) and `advisory-ai-worker` (inference/queue).
+- GPU toggle (NVIDIA) for on-prem inference; defaults remain CPU-safe.
+- Offline kit pickup instructions for including advisory AI artefacts.
+
+## Helm
+Values already ship in `deploy/helm/stellaops/values-*.yaml` under `services.advisory-ai-web` and `advisory-ai-worker`.
+
+GPU enablement (example):
+```yaml
+services:
+  advisory-ai-worker:
+    runtimeClassName: nvidia
+    nodeSelector:
+      nvidia.com/gpu.present: "true"
+    tolerations:
+      - key: nvidia.com/gpu
+        operator: Exists
+        effect: NoSchedule
+    resources:
+      limits:
+        nvidia.com/gpu: 1
+  advisory-ai-web:
+    runtimeClassName: nvidia
+    resources:
+      limits:
+        nvidia.com/gpu: 1
+```
+Apply:
+```bash
+helm upgrade --install stellaops ./deploy/helm/stellaops \
+  -f deploy/helm/stellaops/values-prod.yaml \
+  -f deploy/helm/stellaops/values-mirror.yaml \
+  --set services.advisory-ai-worker.resources.limits.nvidia\.com/gpu=1 \
+  --set services.advisory-ai-worker.runtimeClassName=nvidia
+```
+
+## Compose
+- Base profiles: `docker-compose.dev.yaml`, `stage`, `prod`, `airgap` already include advisory AI services and shared volumes.
+- GPU overlay: `docker-compose.gpu.yaml` (adds NVIDIA device reservations and `ADVISORY_AI_INFERENCE_GPU=true`). Use:
+```bash
+docker compose --env-file prod.env \
+  -f docker-compose.prod.yaml \
+  -f docker-compose.gpu.yaml up -d
+```
+
+## Offline kit pickup
+- Ensure advisory AI images are mirrored to your registry (or baked into airgap tar) before running the offline kit build.
+- Copy the following into `out/offline-kit/metadata/` before invoking the offline kit script:
+  - `advisory-ai-web` image tar
+  - `advisory-ai-worker` image tar
+  - SBOM/provenance generated by the release pipeline
+- Verify `docs/24_OFFLINE_KIT.md` includes the advisory AI entries and rerun `tests/offline/test_build_offline_kit.py` if it changes.
+
+## Runbook (prod quickstart)
+1) Prepare secrets in ExternalSecret or Kubernetes secret named `stellaops-prod-core` (see helm values).
+2) Run Helm install with prod values and GPU overrides as needed.
+3) For Compose, use `prod.env` and optionally `docker-compose.gpu.yaml` overlay.
+4) Validate health:
+   - `GET /healthz` on `advisory-ai-web`
+   - Check queue directories under `advisory-ai-*` volumes remain writable
+   - Confirm inference path logs when GPU is detected (log key `advisory.ai.inference.gpu=true`).
+
+## Advisory Feed Packaging (DEVOPS-AIAI-31-002)
+
+Package advisory feeds (SBOM pointers + provenance) for release/offline kit:
+
+```bash
+# Production (CI with COSIGN_PRIVATE_KEY_B64 secret)
+./ops/deployment/advisory-ai/package-advisory-feeds.sh
+
+# Development (uses tools/cosign/cosign.dev.key)
+COSIGN_ALLOW_DEV_KEY=1 COSIGN_PASSWORD=stellaops-dev \
+  ./ops/deployment/advisory-ai/package-advisory-feeds.sh
+```
+
+Outputs:
+- `out/advisory-ai/feeds/advisory-feeds.tar.gz` - Feed bundle
+- `out/advisory-ai/feeds/advisory-feeds.manifest.json` - Manifest with SBOM pointers
+- `out/advisory-ai/feeds/advisory-feeds.manifest.dsse.json` - DSSE signed manifest
+- `out/advisory-ai/feeds/provenance.json` - Build provenance
+
+CI workflow: `.gitea/workflows/advisory-ai-release.yml`
+
+## Evidence to attach (sprint)
+- Helm release output (rendered templates for advisory AI)
+- `docker-compose config` with/without GPU overlay
+- Offline kit metadata listing advisory AI images + SBOMs
+- Advisory feed package manifest with SBOM pointers
--- a/devops/deployment/advisory-ai/package-advisory-feeds.sh
+++ b/devops/deployment/advisory-ai/package-advisory-feeds.sh
@@ -0,0 +1,165 @@
+#!/usr/bin/env bash
+# Package advisory feeds (SBOM pointers + provenance) for release/offline kit
+# Usage: ./package-advisory-feeds.sh
+# Dev mode: COSIGN_ALLOW_DEV_KEY=1 COSIGN_PASSWORD=stellaops-dev ./package-advisory-feeds.sh
+
+set -euo pipefail
+
+ROOT=$(cd "$(dirname "$0")/../../.." && pwd)
+OUT_DIR="${OUT_DIR:-$ROOT/out/advisory-ai/feeds}"
+CREATED="${CREATED:-$(date -u +%Y-%m-%dT%H:%M:%SZ)}"
+
+mkdir -p "$OUT_DIR"
+
+# Key resolution (same pattern as tools/cosign/sign-signals.sh)
+resolve_key() {
+  if [[ -n "${COSIGN_KEY_FILE:-}" && -f "$COSIGN_KEY_FILE" ]]; then
+    echo "$COSIGN_KEY_FILE"
+  elif [[ -n "${COSIGN_PRIVATE_KEY_B64:-}" ]]; then
+    local tmp_key="$OUT_DIR/.cosign.key"
+    echo "$COSIGN_PRIVATE_KEY_B64" | base64 -d > "$tmp_key"
+    chmod 600 "$tmp_key"
+    echo "$tmp_key"
+  elif [[ -f "$ROOT/tools/cosign/cosign.key" ]]; then
+    echo "$ROOT/tools/cosign/cosign.key"
+  elif [[ "${COSIGN_ALLOW_DEV_KEY:-0}" == "1" && -f "$ROOT/tools/cosign/cosign.dev.key" ]]; then
+    echo "[info] Using development key (non-production)" >&2
+    echo "$ROOT/tools/cosign/cosign.dev.key"
+  else
+    echo "[error] No signing key available. Set COSIGN_PRIVATE_KEY_B64 or COSIGN_ALLOW_DEV_KEY=1" >&2
+    return 1
+  fi
+}
+
+KEY_FILE=$(resolve_key)
+
+# Collect advisory feed sources
+FEED_SOURCES=(
+  "$ROOT/docs/samples/advisory-feeds"
+  "$ROOT/src/AdvisoryAI/feeds"
+  "$ROOT/out/feeds"
+)
+
+echo "==> Collecting advisory feeds..."
+STAGE_DIR="$OUT_DIR/stage"
+mkdir -p "$STAGE_DIR"
+
+for src in "${FEED_SOURCES[@]}"; do
+  if [[ -d "$src" ]]; then
+    echo "    Adding feeds from $src"
+    cp -r "$src"/* "$STAGE_DIR/" 2>/dev/null || true
+  fi
+done
+
+# Create placeholder if no feeds found (dev mode)
+if [[ -z "$(ls -A "$STAGE_DIR" 2>/dev/null)" ]]; then
+  echo "[info] No feed sources found; creating placeholder for dev mode"
+  cat > "$STAGE_DIR/placeholder.json" <<EOF
+{
+  "type": "advisory-feed-placeholder",
+  "created": "$CREATED",
+  "note": "Placeholder for development; replace with real feeds in production"
+}
+EOF
+fi
+
+# Create feed bundle
+echo "==> Creating feed bundle..."
+BUNDLE_TAR="$OUT_DIR/advisory-feeds.tar.gz"
+tar -czf "$BUNDLE_TAR" -C "$STAGE_DIR" .
+
+# Compute hashes
+sha256() {
+  sha256sum "$1" | awk '{print $1}'
+}
+
+BUNDLE_HASH=$(sha256 "$BUNDLE_TAR")
+
+# Generate manifest with SBOM pointers
+echo "==> Generating manifest..."
+MANIFEST="$OUT_DIR/advisory-feeds.manifest.json"
+cat > "$MANIFEST" <<EOF
+{
+  "schemaVersion": "1.0.0",
+  "created": "$CREATED",
+  "bundle": {
+    "path": "advisory-feeds.tar.gz",
+    "sha256": "$BUNDLE_HASH",
+    "size": $(stat -c%s "$BUNDLE_TAR" 2>/dev/null || stat -f%z "$BUNDLE_TAR")
+  },
+  "sbom": {
+    "format": "spdx-json",
+    "path": "advisory-feeds.sbom.json",
+    "note": "SBOM generated during CI; pointer only in manifest"
+  },
+  "provenance": {
+    "path": "provenance.json",
+    "builder": "stellaops-advisory-ai-release"
+  }
+}
+EOF
+
+# Sign manifest with DSSE
+echo "==> Signing manifest..."
+DSSE_OUT="$OUT_DIR/advisory-feeds.manifest.dsse.json"
+
+# Check for cosign
+COSIGN="${COSIGN:-$ROOT/tools/cosign/cosign}"
+if ! command -v cosign &>/dev/null && [[ ! -x "$COSIGN" ]]; then
+  echo "[warn] cosign not found; skipping DSSE signing" >&2
+else
+  COSIGN_CMD="${COSIGN:-cosign}"
+  if command -v cosign &>/dev/null; then
+    COSIGN_CMD="cosign"
+  fi
+
+  COSIGN_PASSWORD="${COSIGN_PASSWORD:-}" "$COSIGN_CMD" sign-blob \
+    --key "$KEY_FILE" \
+    --bundle "$DSSE_OUT" \
+    --tlog-upload=false \
+    --yes \
+    "$MANIFEST" 2>/dev/null || echo "[warn] DSSE signing skipped (cosign error)"
+fi
+
+# Generate provenance
+echo "==> Generating provenance..."
+PROVENANCE="$OUT_DIR/provenance.json"
+cat > "$PROVENANCE" <<EOF
+{
+  "_type": "https://in-toto.io/Statement/v1",
+  "subject": [
+    {
+      "name": "advisory-feeds.tar.gz",
+      "digest": {"sha256": "$BUNDLE_HASH"}
+    }
+  ],
+  "predicateType": "https://slsa.dev/provenance/v1",
+  "predicate": {
+    "buildDefinition": {
+      "buildType": "https://stella-ops.org/advisory-ai-release/v1",
+      "externalParameters": {},
+      "internalParameters": {
+        "created": "$CREATED"
+      }
+    },
+    "runDetails": {
+      "builder": {
+        "id": "https://stella-ops.org/advisory-ai-release"
+      },
+      "metadata": {
+        "invocationId": "$(uuidgen 2>/dev/null || echo "dev-$(date +%s)")",
+        "startedOn": "$CREATED"
+      }
+    }
+  }
+}
+EOF
+
+# Cleanup temp key
+[[ -f "$OUT_DIR/.cosign.key" ]] && rm -f "$OUT_DIR/.cosign.key"
+
+echo "==> Advisory feed packaging complete"
+echo "    Bundle: $BUNDLE_TAR"
+echo "    Manifest: $MANIFEST"
+echo "    DSSE: $DSSE_OUT"
+echo "    Provenance: $PROVENANCE"