git.stella-ops.org/EPIC_13.md

No file to print
Fine. Shipping containers, but for software. Here’s the serious version you can paste into your docs without the sarcasm.

> **Imposed rule:** Work of this type or tasks of this type on this component must also be applied everywhere else it should be applied.

---

# Epic 13: Containerized Distribution & Quickstart

**Short name:** `Containerized Distribution & Quickstart`
**Primary components:** OCI images for all services, Compose Quickstart, Helm chart for production, Air‑gap bundles
**Surfaces:** Container registry, `/deploy/*`, `/docs/install/*`, Console onboarding screen
**Touches:** Authority (authN/Z), Web Services API, Orchestrator, Task Runner, Policy Engine, Conseiller (Feedser), Excitator (Vexer), Findings Ledger, Export Center, Notifications Studio, Advisory AI Assistant, Object Storage/KMS, Telemetry

**AOC ground rule reminder:** Conseiller and Excitator aggregate and link advisories/VEX. They never merge or mutate source records. Containerized deployments must preserve this behavior and expose links to originals.

---

## 1) What it is

A complete, reproducible containerized distribution of StellaOps with three delivery modes:

1. **Quickstart (single host)** using Docker Compose: one command to run a full stack suitable for evaluation and local development. Ships with seed data and sane defaults.

2. **Production Helm chart** for Kubernetes: modular, scalable, secure‑by‑default deployment with optional HA and external dependencies.

3. **Air‑gapped bundles**: signed offline packages containing images, seed configs, and installation scripts for disconnected environments.

All images are multi‑arch (amd64/arm64), signed, SBOM‑attached, and versioned with consistent tags. A “Download & Install” doc set guides users from zero to a working system in minutes and to a production‑ready posture in hours.

---

## 2) Why (brief)

People don’t adopt tools they can’t run quickly or securely. Containers make our deployment reproducible; Quickstart removes friction; Helm unlocks real ops. Air‑gap bundles acknowledge reality in regulated environments.

---

## 3) How it should work (maximum detail)

### 3.1 Image catalog

Build and publish OCI images for the following:

* `stella-api` (Web Services API)
* `stella-console` (Web UI)
* `stella-orchestrator` (source/job scheduler)
* `stella-task-runner` (executes Task Packs remotely)
* `stella-conseiller` (Feedser; advisory aggregator)
* `stella-excitator` (Vexer; VEX aggregator)
* `stella-policy` (Policy Engine)
* `stella-ledger` (Findings Ledger worker; if separated from API)
* `stella-export` (Export Center worker; optional if part of API)
* `stella-notify` (Notifications Studio worker)
* `stella-ai` (Advisory AI Assistant; lightweight service calling configured LLM backends or local models)
* Support services (optionally bundled for Quickstart): `postgres`, `redis`, `object-store` (S3‑compatible), `queue` (NATS or RabbitMQ), `otel-collector`.

**Image standards**

* **Base:** distroless or minimal; non‑root user; read‑only filesystem; writable `/tmp` only if needed.
* **Ports:** declare via labels; expose health endpoints `/health/liveness`, `/health/readiness`.
* **Env:** explicit, documented, with safe defaults; secrets via env or file mounts only.
* **Config:** `STELLA_*` envs or mounted config directory `/etc/stella/`.
* **SBOM:** attach SPDX JSON as OCI artifact and include in `/app/sbom.spdx.json` baked at build time.
* **Signing:** cosign attestations for image, SBOM, and provenance.
* **Labels:** org.opencontainers.image.* (title, version, revision, source, licenses).
* **Entrypoint:** PID 1 with reap; graceful shutdown on SIGTERM; configurable termination grace period.
* **Logs:** structured JSON by default; stdout/stderr only.

**Tagging scheme**

* `:vX.Y.Z` (immutable release)
* `:vX.Y.Z-rc.N` (release candidate)
* `:edge` (latest main)
* `:nightly-YYYYMMDD` (optional)
* Multi‑arch manifest lists for linux/amd64 and linux/arm64.

### 3.2 Quickstart (Compose)

**Goal:** `curl | sh` equivalent that yields a working stack on a laptop/server with defaults and demo data. No internet beyond pulling images, unless configured.

**Compose file `deploy/compose/docker-compose.yml`**

* Services:

  * `api`, `console`, `orchestrator`, `task-runner`, `conseiller`, `excitator`, `policy`, `notify`, `export`, `ai`
  * `postgres`, `redis`, `minio` (S3), `nats` or `rabbitmq`, `otel-collector`
* Volumes:

  * `pgdata`, `minio-data`, `redis-data`, `stella-state` (for local cache, packs registry)
* Networks:

  * `stella-net` bridge
* Ports (defaults):

  * Console `8080`, API `8081`, MinIO `9000`, NATS/RabbitMQ default ports
* Env files:

  * `.env.example` with safe defaults; users copy to `.env`.

**Seed data**

* Seed admin account and tenant on first run via `stella-api` migration/seed job.
* Seed demo SBOMs, advisories, VEX samples, baseline policy, and a task pack.
* On first login, Console shows “Welcome” wizard: confirm endpoints, generate API token, run sample scan import, open Vulnerability Explorer.

**Security posture**

* Default credentials only for Quickstart; randomize secrets on first `up` and store in `.secrets/` file.
* All services run as non‑root; bind to localhost by default unless `EXPOSE_PUBLIC=1` set.
* TLS optional via `CADDY` or `nginx` sidecar disabled by default.

**One‑liner**

* `./deploy/compose/quickstart.sh` does: preflight checks, pulls images, writes `.env`, runs `docker compose up -d`, polls readiness, prints URLs and credentials.

**Backups & reset**

* `./deploy/compose/backup.sh` creates a tarball of volumes and config.
* `./deploy/compose/reset.sh` nukes persistent volumes with a big scary prompt unless `--yes`.

### 3.3 Production Helm chart

**Chart location:** `deploy/helm/stella/` with subcharts or toggles.

**Chart features**

* Components enabled via values: `api`, `console`, `orchestrator`, `taskRunner`, `conseiller`, `excitator`, `policy`, `notify`, `export`, `ai`.
* External dependencies by default:

  * PostgreSQL, Redis, S3 bucket, Message queue, OTel endpoint provided via values.
  * Optional “bundled” mode for lab clusters using StatefulSets.
* Security:

  * PodSecurityContext: runAsNonRoot, readOnlyRootFilesystem, fsGroup when needed.
  * NetworkPolicy for east‑west traffic; deny‑all then allow specific ports.
  * Secrets as `Secret` from External Secrets operator or sealed secrets.
  * HPA per component; PDBs; liveness/readiness probes.
* Ingress:

  * One hostname for Console, one for API; TLS required in production values.
  * Option to serve Console as static behind CDN while API behind private ingress gateway.
* Config:

  * Values for Authority provider, token TTLs, policy cache TTL, pack registry endpoint, notifications sinks, export locations.
  * Feature flags per epic enablement.
* Migrations:

  * `stella-migrator` Job runs before rollouts; idempotent migrations.
  * Optional “break glass” manual job.
* Observability:

  * `/metrics` endpoints scraped by Prometheus; exemplars via OTel; logs structured.
  * OpenTelemetry auto‑config via env if collector provided.
* Upgrades:

  * Blue/green or rolling; readiness gates based on background indexers catching up.
  * Chart hooks to block until Conseiller/Excitator catch up to feed watermarks.

### 3.4 Air‑gapped distribution

**Bundle format**

* `stella-bundle-vX.Y.Z.tar.zst` containing:

  * All images as OCI layout (multi‑arch), cosign signatures, SBOMs, SLSA provenance.
  * `load.sh` to import into a local registry.
  * `compose/` and `helm/` directories with pinned image digests.
  * `checksums.txt` and `bundle.sig`.
* **Process**

  * Online build job crafts bundle; signatures produced by CI keys.
  * Offline install:

    * Verify `bundle.sig`
    * `./load.sh --to registry.local:5000`
    * `helm install stella ./helm -f values-airgap.yaml --set image.registry=registry.local:5000`

### 3.5 Configuration matrix

Document every config knob in a single table:

* Auth: Authority issuer, JWKS, RBAC cache TTL.
* Storage: DB URL, pool sizes, migration flags.
* Object store: S3 endpoint, buckets, SSE, IAM.
* Queue: URL, prefetch, retention.
* Policy engine: rule cache TTL, default policy version.
* Conseiller/Excitator: polling intervals, feed sources, retry backoff, max in‑flight; **merge disabled** enforced.
* Orchestrator/Task Runner: concurrency, sandbox, network egress policy, artifact retention.
* Notifications: sinks, templates path, batch windows.
* Export Center: formats enabled, rate limits.
* AI Assistant: model endpoint, token limits, guardrails, disable by default.

### 3.6 Health, readiness, and upgrades

* **Health endpoints:** `GET /health/liveness` returns 200 if process responsive; `GET /health/readiness` checks dependencies with timeout.
* **Graceful shutdown:** SIGTERM starts drain; HTTP returns 503; background workers flush; exit on deadline.
* **Upgrade choreography:** migrations run, API becomes ready, workers rolling restart, indexes catch up, AOC evaluation warms caches, then flip traffic.
* **Version skew policy:** define supported skew between components; chart validates.

### 3.7 Security & compliance

* **Image signing & verification:** cosign attestations; optional admission policy to verify signatures by key.
* **SBOM provenance:** attach SPDX and provenance attestations; publish via registry referrers.
* **Non‑root & least privilege:** capabilities dropped; only NET_BIND for proxies if needed.
* **Secrets handling:** mount from files; avoid putting secrets in args; redacted logs by default.
* **Audit:** container labels propagate release metadata to all logs and spans.
* **AOC enforcement:** images for Conseiller/Excitator hard‑disable merge code paths via env/defaults.

### 3.8 Quickstart UX polish

* Console shows “Connected to Quickstart” banner with a button “View install docs” and “Export pack to production.”
* One click to generate a `Task Pack` that exports seed data from Quickstart to a production tenant via Export Center.

---

## 4) Architecture

### 4.1 Repos & layout

```
/deploy
  /compose
    docker-compose.yml
    .env.example
    quickstart.sh
    backup.sh
    reset.sh
  /helm
    /stella
      Chart.yaml
      values.yaml
      values-prod.yaml
      values-airgap.yaml
      templates/*.yaml
  /docker
    stella-api.Dockerfile
    stella-console.Dockerfile
    stella-orchestrator.Dockerfile
    stella-task-runner.Dockerfile
    stella-conseiller.Dockerfile
    stella-excitator.Dockerfile
    stella-policy.Dockerfile
    stella-notify.Dockerfile
    stella-export.Dockerfile
    stella-ai.Dockerfile
```

### 4.2 CI/CD flow

* Build multi‑arch with buildx; run unit/integration tests; embed version metadata and SBOM.
* Sign images; push to registry; publish Helm chart with pinned digests.
* Generate Air‑gap bundle and signatures.
* Smoke test Quickstart on fresh VM; e2e tests exercise Console and CLI parity (Epic 12).

---

## 5) APIs and contracts

No new external APIs, but every service must expose:

* `GET /health/liveness` and `GET /health/readiness`.
* `GET /version` returning `{ version, gitCommit, buildDate }`.
* `GET /metrics` when enabled.
* Config discovery endpoint for Console with trimmed, safe values (no secrets).
* Conseiller/Excitator must expose `GET /capabilities` returning `{"merge": false}` to prove merge is disabled.

---

## 6) Documentation changes

Create/update:

1. `/docs/install/overview.md`
   Supported deployment modes, hardware requirements, network ports, quickstart vs production.

2. `/docs/install/compose-quickstart.md`
   Preconditions, one‑liner, first‑login wizard, seed data, reset/backup, common pitfalls.

3. `/docs/install/helm-prod.md`
   Prereqs, external dependencies, values reference, TLS/ingress, HPA, PDB, upgrades, rollbacks.

4. `/docs/install/airgap.md`
   Bundle verification, loading into private registry, running without internet, patching images.

5. `/docs/install/configuration-reference.md`
   The full configuration matrix with examples.

6. `/docs/security/supply-chain.md`
   Image signing, SBOMs, provenance, admission controls, non‑root posture.

7. `/docs/operations/health-and-readiness.md`
   Endpoints, probes, troubleshooting, expected states during upgrades.

8. `/docs/release/image-catalog.md`
   All image names, tags, architectures, checksums; mapping between chart version and image digests.

9. `/docs/console/onboarding.md`
   Quickstart banner, links to install docs, exporting data to production.

Add at the top of each page:

> **Imposed rule:** Work of this type or tasks of this type on this component must also be applied everywhere else it should be applied.

---

## 7) Implementation plan

### New modules/artifacts

* Dockerfiles per service under `/deploy/docker/` with common builder stages.
* Helm chart under `/deploy/helm/stella`.
* Compose quickstart under `/deploy/compose/`.
* Air‑gap bundle generator in CI, script `tools/make-airgap-bundle.sh`.
* Seed dataset packaged as container image layer or mounted config.

### Changes to services

* Add health/version/metrics endpoints where missing.
* Ensure all services read config from env/files with defaults suitable for Quickstart.
* Conseiller/Excitator: add hard config flag `DISABLE_MERGE=true` defaulted in images and values.
* API: seed job and migration runner; serve `/welcome` state for Console wizard.
* Console: onboarding wizard and Quickstart banner.
* Task Runner: respect offline mode by failing gracefully if egress blocked.

### Packaging & signing

* Embed SBOM in all images; publish as OCI referrers.
* Cosign sign images and attest provenance; verify in CI.
* Publish checksums and signatures on release page.

---

## 8) Engineering tasks

**Images**

* [ ] Author multi‑stage Dockerfiles with cache‑efficient builds.
* [ ] Add non‑root user, drop capabilities, read‑only FS, healthcheck scripts.
* [ ] Generate and attach SBOM for each image.
* [ ] Implement `/health/*`, `/version`, optional `/metrics`.

**Compose**

* [ ] Write `docker-compose.yml` with all core services and deps.
* [ ] Create `.env.example`, `quickstart.sh`, `backup.sh`, `reset.sh`.
* [ ] Seed job container and sample data ingestion on first run.

**Helm**

* [ ] Scaffold chart; values for each component; pinned digests.
* [ ] Ingress, TLS, HPA, PDB, NetworkPolicy, ServiceAccount/RBAC.
* [ ] Migration Job and upgrade hooks; readiness gates for indexers.
* [ ] Documentation of values with `helm-docs` generator.

**Air‑gap**

* [ ] Build script to save images to OCI layout; compress, sign, and checksum.
* [ ] `load.sh` to import into private registry and rewrite manifests.
* [ ] `values-airgap.yaml` with image registry overrides.

**Console & API**

* [ ] Onboarding wizard, Quickstart banner, links to docs.
* [ ] Seed data endpoints guarded behind `QUICKSTART_MODE`.
* [ ] Config discovery endpoint for console.

**Security**

* [ ] Cosign integration; key management; CI verification step.
* [ ] Admission policy example in docs to enforce signatures.
* [ ] Secret redaction in logs; env var audit.

**Observability**

* [ ] OTel config sample; `/metrics` endpoints; compose prom scrape.
* [ ] Helm values for tracing and metrics.

**Validation**

* [ ] Fresh VM smoke test for Compose quickstart.
* [ ] Kind cluster e2e for Helm path.
* [ ] Air‑gap install test in CI with a local registry.

**Docs**

* [ ] Write all pages listed in §6 with copy‑pasteable commands and screenshots.
* [ ] Include a troubleshooting matrix: symptom → probable cause → fix.
* [ ] Add “Imposed rule” header line to each page.

---

## 9) Feature changes required

* **Console:** Onboarding wizard, Quickstart banner, and deep links to install docs; “Copy CLI” buttons should prefer the `stella` container image in quickstart if local binary missing.
* **API:** Seed job and health endpoints; version reporting; feature flag `QUICKSTART_MODE`.
* **Registry/Release tooling:** Publish image catalog and checksums; maintain compatibility matrix per chart version.
* **Task Runner:** Offline mode awareness and explicit error when attempting egress in air‑gap.
* **Conseiller/Excitator:** enforce non‑merge at runtime and show capability endpoint.

---

## 10) Acceptance criteria

* Quickstart: from clean host to working Console in under 5 minutes on a typical laptop; seed data visible; AOC rules active.
* Helm: install succeeds with external dependencies; roll forward and roll back with zero data loss; probes green.
* Air‑gap: bundle verifies, loads to a private registry, and installs without external network.
* All images: signed, SBOM‑attached, non‑root, read‑only FS, health endpoints exposed.
* Docs: a new user can complete Quickstart without assistance; a platform team can deploy the chart with only values editing.
* Conseiller/Excitator: capability endpoint confirms `merge=false`; tests prove aggregation‑only behavior.

---

## 11) Risks & mitigations

* **Config sprawl.** Centralize in `/docs/install/configuration-reference.md` and ship sane defaults.
* **Drift between Compose and Helm.** Pin digests; generate manifests from a common values source where possible; CI diff.
* **Resource contention in Quickstart.** Limit concurrency; ship low default worker counts; document overrides.
* **Air‑gap surprises.** Remove implicit egress; provide offline doc copies in bundle; deterministic artifact paths.
* **Security regressions.** Enforce non‑root/read‑only in CI; signature verification gates release.

---

## 12) Philosophy

* **First run matters.** Quickstart must be boring, predictable, and immediately useful.
* **Prod isn’t a flag.** Helm defaults are safe; “convenience” belongs in Quickstart, not production.
* **Prove your supply chain.** Signed images, SBOMs, and provenance are table stakes, not an upsell.

> **Imposed rule:** Work of this type or tasks of this type on this component must also be applied everywhere else it should be applied.