2e70c9fdb6
Some checks failed
LNM Migration CI / build-runner (push) Has been cancelled
Ledger OpenAPI CI / deprecation-check (push) Has been cancelled
Docs CI / lint-and-preview (push) Has been cancelled
Airgap Sealed CI Smoke / sealed-smoke (push) Has been cancelled
Ledger Packs CI / build-pack (push) Has been cancelled
Export Center CI / export-ci (push) Has been cancelled
Ledger OpenAPI CI / validate-oas (push) Has been cancelled
Ledger OpenAPI CI / check-wellknown (push) Has been cancelled
Ledger Packs CI / verify-pack (push) Has been cancelled
LNM Migration CI / validate-metrics (push) Has been cancelled
AOC Guard CI / aoc-guard (push) Has been cancelled
AOC Guard CI / aoc-verify (push) Has been cancelled
92 lines
3.5 KiB
Markdown
92 lines
3.5 KiB
Markdown
# Advisory AI Deployment Runbook
|
|
|
|
## Scope
|
|
- Helm and Compose packaging for `advisory-ai-web` (API/plan cache) and `advisory-ai-worker` (inference/queue).
|
|
- GPU toggle (NVIDIA) for on-prem inference; defaults remain CPU-safe.
|
|
- Offline kit pickup instructions for including advisory AI artefacts.
|
|
|
|
## Helm
|
|
Values already ship in `deploy/helm/stellaops/values-*.yaml` under `services.advisory-ai-web` and `advisory-ai-worker`.
|
|
|
|
GPU enablement (example):
|
|
```yaml
|
|
services:
|
|
advisory-ai-worker:
|
|
runtimeClassName: nvidia
|
|
nodeSelector:
|
|
nvidia.com/gpu.present: "true"
|
|
tolerations:
|
|
- key: nvidia.com/gpu
|
|
operator: Exists
|
|
effect: NoSchedule
|
|
resources:
|
|
limits:
|
|
nvidia.com/gpu: 1
|
|
advisory-ai-web:
|
|
runtimeClassName: nvidia
|
|
resources:
|
|
limits:
|
|
nvidia.com/gpu: 1
|
|
```
|
|
Apply:
|
|
```bash
|
|
helm upgrade --install stellaops ./deploy/helm/stellaops \
|
|
-f deploy/helm/stellaops/values-prod.yaml \
|
|
-f deploy/helm/stellaops/values-mirror.yaml \
|
|
--set services.advisory-ai-worker.resources.limits.nvidia\.com/gpu=1 \
|
|
--set services.advisory-ai-worker.runtimeClassName=nvidia
|
|
```
|
|
|
|
## Compose
|
|
- Base profiles: `docker-compose.dev.yaml`, `stage`, `prod`, `airgap` already include advisory AI services and shared volumes.
|
|
- GPU overlay: `docker-compose.gpu.yaml` (adds NVIDIA device reservations and `ADVISORY_AI_INFERENCE_GPU=true`). Use:
|
|
```bash
|
|
docker compose --env-file prod.env \
|
|
-f docker-compose.prod.yaml \
|
|
-f docker-compose.gpu.yaml up -d
|
|
```
|
|
|
|
## Offline kit pickup
|
|
- Ensure advisory AI images are mirrored to your registry (or baked into airgap tar) before running the offline kit build.
|
|
- Copy the following into `out/offline-kit/metadata/` before invoking the offline kit script:
|
|
- `advisory-ai-web` image tar
|
|
- `advisory-ai-worker` image tar
|
|
- SBOM/provenance generated by the release pipeline
|
|
- Verify `docs/24_OFFLINE_KIT.md` includes the advisory AI entries and rerun `tests/offline/test_build_offline_kit.py` if it changes.
|
|
|
|
## Runbook (prod quickstart)
|
|
1) Prepare secrets in ExternalSecret or Kubernetes secret named `stellaops-prod-core` (see helm values).
|
|
2) Run Helm install with prod values and GPU overrides as needed.
|
|
3) For Compose, use `prod.env` and optionally `docker-compose.gpu.yaml` overlay.
|
|
4) Validate health:
|
|
- `GET /healthz` on `advisory-ai-web`
|
|
- Check queue directories under `advisory-ai-*` volumes remain writable
|
|
- Confirm inference path logs when GPU is detected (log key `advisory.ai.inference.gpu=true`).
|
|
|
|
## Advisory Feed Packaging (DEVOPS-AIAI-31-002)
|
|
|
|
Package advisory feeds (SBOM pointers + provenance) for release/offline kit:
|
|
|
|
```bash
|
|
# Production (CI with COSIGN_PRIVATE_KEY_B64 secret)
|
|
./ops/deployment/advisory-ai/package-advisory-feeds.sh
|
|
|
|
# Development (uses tools/cosign/cosign.dev.key)
|
|
COSIGN_ALLOW_DEV_KEY=1 COSIGN_PASSWORD=stellaops-dev \
|
|
./ops/deployment/advisory-ai/package-advisory-feeds.sh
|
|
```
|
|
|
|
Outputs:
|
|
- `out/advisory-ai/feeds/advisory-feeds.tar.gz` - Feed bundle
|
|
- `out/advisory-ai/feeds/advisory-feeds.manifest.json` - Manifest with SBOM pointers
|
|
- `out/advisory-ai/feeds/advisory-feeds.manifest.dsse.json` - DSSE signed manifest
|
|
- `out/advisory-ai/feeds/provenance.json` - Build provenance
|
|
|
|
CI workflow: `.gitea/workflows/advisory-ai-release.yml`
|
|
|
|
## Evidence to attach (sprint)
|
|
- Helm release output (rendered templates for advisory AI)
|
|
- `docker-compose config` with/without GPU overlay
|
|
- Offline kit metadata listing advisory AI images + SBOMs
|
|
- Advisory feed package manifest with SBOM pointers
|