# 13 · Release Engineering Playbook — Stella Ops A concise, automation‑first guide describing **how source code on `main` becomes a verifiably signed, air‑gap‑friendly release**. It is opinionated for offline use‑cases and supply‑chain security (SLSA ≥ level 2 today, aiming for level 3). --- ## 0 Release Philosophy * **Fast but fearless** – every commit on `main` must be releasable; broken builds break the build, not the team. * **Reproducible** – anyone can rebuild byte‑identical artefacts with a single `make release` offline. * **Secure by default** – every artefact ships with a SBOM, Cosign signature and (future) Rekor log entry. * **Offline‑first** – all dependencies are vendored or mirrored into the internal registry; no Internet required at runtime. --- ## 1 Versioning & Branching | Branch | Purpose | Auto‑publish? | | ------------- | ------------------------------ | --------------------------------------- | | `main` | Always‑green development trunk | `nightly-*` images | | `release/X.Y` | Stabilise a minor line | `stella:X.Y-rcN` | | Tags | `X.Y.Z` = SemVer | `stella:X.Y.Z`, OUK tarball, Helm chart | * **SemVer** – MAJOR for breaking API/CLI changes, MINOR for features, PATCH for fixes. * Release tags are **signed** (`git tag -s`) with the Stella Ops GPG key (`0x90C4…`). --- ## 2 CI/CD Overview (GitLab CI + GitLab Runner) ```mermaid graph LR A[push / MR] --> Lint Lint --> Unit Unit --> Build Build --> Test-Container Test-Container --> SBOM SBOM --> Sign Sign --> Publish Publish --> E2E Publish --> Notify ``` ### Pipeline Stages | Stage | Key tasks | | ------------------ | ------------------------------------------------------------------------------------------------ | | **Lint** | ESLint, golangci‑lint, hadolint, markdown‑lint. | | **Unit** | `dotnet test`, `go test`, Jest UI tests. | | **Quota unit‑tests 🏷** | Validate QuotaService logic: reset at UTC, 5 s vs 60 s waits, header correctness. | | **Build** | Multi‑arch container build (`linux/amd64`, `linux/arm64`) using **BuildKit** + `--provenance` 📌. | | **Test‑Container** | Spin up compose file, run smoke APIs. | | **SBOM** 📌 | Invoke **StellaOps.SBOMBuilder** to generate SPDX JSON + attach `.sbom` label to image. | | **Sign** | Sign image with **Cosign** (`cosign sign --key cosign.key`). | | **Publish** | Push to `registry.git.stella-ops.ru`. | | **E2E** | Kind‑based Kubernetes test incl. Zastava DaemonSet; verify sub‑5 s scan SLA. | | **Notify** | Report to Mattermost & GitLab Slack app. | | **OfflineToken** | Call `JwtIssuer.Generate(exp=30d)` → store `client.jwt` artefact → attach to OUK build context | *All stages run in parallel where possible; max wall‑time < 15 min.* --- ## 3 Container Image Strategy | Image | Registry Tag | Contents | | ------------------------------ | --------------------------- | ---------------------------------------------------------------------- | | **backend** | `stella/backend:{ver}` | ASP.NET API, plugin loader. | | **ui** | `stella/ui:{ver}` | Pre‑built Angular SPA. | | **runner-trivy** | `stella/runner-trivy:{ver}` | Trivy CLI + SPDX/CycloneDX 🛠. | | **runner-grype** | `stella/runner-grype:{ver}` | Optional plug‑in scanner. | | **🏷️ StellaOps.Registry** 📌 | `stella/registry:{ver}` | Scratch image embedding Docker Registry v2 + Cosign policy controller. | | **🏷️ StellaOps.MutePolicies** 📌 | `stella/policies:{ver}` | Sidecar serving policy bundles. | | **🏷️ StellaOps.Attestor** 📌 | `stella/attestor:{ver}` | SLSA provenance & Rekor signer (future). | *Images are **`--label org.opencontainers.image.source=git.stella-ops.ru`** and include SBOMs generated at build time.* --- ## 4 📌 Offline Update Kit (OUK) Build & Distribution **Purpose** – deliver updated CVE feeds & Trivy DB to air‑gapped clusters. ### 4.1 CLI Tool *Go binary `ouk` lives in `tools/ouk/`.* ```sh ouk fetch \ --nvd --osv \ --trivy-db --date $(date -I) \ --output ouk-$(date +%Y%m%d).tar.gz \ --sign cosign.key ``` ### 4.2 Pipeline Hook * Runs on **first Friday** each month (cron). * Generates tarball, signs it, uploads to **GitLab Release asset**. * SHA‑256 + signature published alongside. ### 4.3 Activation Flow (runtime) 1. Admin uploads `.tar.gz` via **UI → Settings → Offline Updates (OUK)**. 2. Backend verifies Cosign signature & digest. 3. Files extracted into `var/lib/stella/db`. 4. Redis caches invalidated; Dashboard “Feed Age” ticks green. 5. Audit event `ouk_update` stored. ### 4.4 Token Detail client.jwt placed under /root/ inside the tarball. CI job fails if token expiry < 29 days (guard against stale caches). --- ## 5 Artifact Signing & Transparency | Artefact | Signer | Tool | | ------------ | --------------- | --------------------- | | Git tags | GPG (`0x90C4…`) | `git tag -s` | | Containers | Cosign key pair | `cosign sign` | | Helm Charts | prov file | `helm package --sign` | | OUK tarballs | Cosign | `cosign sign-blob` | **Rekor** integration is **TODO** – once the internal Rekor mirror is online (`StellaOpsAttestor`) a post‑publish job will submit transparency log entries. --- ## 6 Release Checklist 1. CI pipeline green. 2. Bump `VERSION` file. 3. Tag `git tag -s X.Y.Z -m "Release X.Y.Z"` & push. 4. GitLab CI auto‑publishes images & charts. 5. Draft GitLab **Release Notes** using `tools/release-notes-gen`. 6. Verify SBOM attachment with `stella sbom verify stella/backend:X.Y.Z`. 7. Smoke‑test OUK tarball in offline lab. 8. Announce in `#stella-release` Mattermost channel. --- ## 7 Hot‑fix Procedure * Branch from latest tag → `hotfix/X.Y.Z+1-hf1`. * Apply minimal patch, add regression test. * CI pipeline (with reduced stages) must pass. * Tag `X.Y.Z+1`. * Publish only container + Helm chart; OUK not rebuilt. * Cherry‑pick back to `main`. --- ## 8 Deprecation & End‑of‑Life Policy | Feature | Deprecation notice | Removal earliest | | ------------------------ | ------------------ | ---------------- | | Legacy CSV policy import | 2025‑10‑01 | 2026‑04‑01 | | Docker v1 Registry auth | 2025‑12‑01 | 2026‑06‑01 | | In‑image Trivy DB | 2025‑12‑15 | 2026‑03‑15 | *At least 6 months notice; removal requires major version bump.* --- ## 9 📌 Non‑Commercial Usage Rules (English canonical) 1. **Free for internal security assessments** (company or personal). 2. **SaaS resale / re‑hosting prohibited** without prior written consent (AGPL §13). 3. If you distribute a fork with UI or backend modifications **you must**: * Publish the complete modified source code. * Retain the original Stella Ops attribution in UI footer and CLI `--version`. 4. All third‑party dependencies remain under their respective licences (MIT, Apache‑2.0, ISC, BSD). 5. Deployments in state‑regulated or classified environments must obey **ФЗ‑187** export rules. --- ## 10 Best Practices Snapshot 📌 * **SBOM‑per‑image** → attach at build time; store as OCI artifact for supply‑chain introspection. * **Provenance flag** (`--provenance=true`) in BuildKit fulfils SLSA 2 requirement. * Use **multi‑arch, reproducible builds** (`SOURCE_DATE_EPOCH` pins timestamps). * All pipelines enforce **Signed‑off‑by (DCO)**; CI fails if trailer missing. * `cosign policy` ensures only images signed by the project key run in production. --- ## 11 Contributing to Release Engineering * Fork & create MR to `infra/release-*`. * All infra changes require green **`integration-e2e-offline`** job. * Discuss larger infra migrations in `#sig-release` Mattermost; decisions recorded in `ADR/` folder. --- ## 12 Change Log (high‑level) | Version | Date | Note | | ------- | ---------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | | v2.1 | 2025‑07‑15 | Added OUK build/publish pipeline, internal registry image (`StellaOps.Registry`), non‑commercial usage rules extraction, SBOM stage, BuildKit provenance. | | v2.0 | 2025‑07‑12 | Initial open‑sourcing of Release Engineering guide. | | v1.1 | 2025‑07‑09 | Fixed inner fencing; added retention policy | | v1.0 | 2025‑07‑09 | Initial playbook | --- *(End of Release Engineering Playbook v1.1)*