git.stella-ops.org/docs/13_RELEASE_ENGINEERING_PLAYBOOK.md

# 13 · Release Engineering Playbook — Stella Ops

A concise, automation‑first guide describing how source code on main becomes a verifiably signed, air‑gap‑friendly release.
It is opinionated for offline use‑cases and supply‑chain security (SLSA ≥ level 2 today, aiming for level 3).

---

## 0 Release Philosophy

• Fast but fearless – every commit on main must be releasable; broken builds break the build, not the team.
• Reproducible – anyone can rebuild byte‑identical artefacts with a single make release offline.
• Secure by default – every artefact ships with a SBOM, Cosign signature and (future) Rekor log entry.
• Offline‑first – all dependencies are vendored or mirrored into the internal registry; no Internet required at runtime.

---

## 1 Versioning & Branching

Branch	Purpose	Auto‑publish?
main	Always‑green development trunk	nightly-* images
release/X.Y	Stabilise a minor line	stella:X.Y-rcN
Tags	X.Y.Z = SemVer	stella:X.Y.Z, OUK tarball, Helm chart

• SemVer – MAJOR for breaking API/CLI changes, MINOR for features, PATCH for fixes.
• Release tags are signed (git tag -s) with the Stella Ops GPG key (0x90C4…).

---

## 2 CI/CD Overview (GitLab CI + GitLab Runner)

graph LR
  A[push / MR] --> Lint
  Lint --> Unit
  Unit --> Build
  Build --> Test-Container
  Test-Container --> SBOM
  SBOM --> Sign
  Sign --> Publish
  Publish --> E2E
  Publish --> Notify

### Pipeline Stages

Stage	Key tasks
Lint	ESLint, golangci‑lint, hadolint, markdown‑lint.
Unit	dotnet test, go test, Jest UI tests.
Quota unit‑tests 🏷	Validate QuotaService logic: reset at UTC, 5 s vs 60 s waits, header correctness.
Build	Multi‑arch container build (linux/amd64, linux/arm64) using BuildKit + --provenance 📌.
Test‑Container	Spin up compose file, run smoke APIs.
SBOM 📌	Invoke StellaOps.SBOMBuilder to generate SPDX JSON + attach .sbom label to image.
Sign	Sign image with Cosign (cosign sign --key cosign.key).
Publish	Push to registry.git.stella-ops.org.
E2E	Kind‑based Kubernetes test incl. Zastava DaemonSet; verify sub‑5 s scan SLA.
Notify	Report to Mattermost & GitLab Slack app.
OfflineToken	Call JwtIssuer.Generate(exp=30d) → store client.jwt artefact → attach to OUK build context

All stages run in parallel where possible; max wall‑time < 15 min.

---

## 3 Container Image Strategy

Image	Registry Tag	Contents
backend	stella/backend:{ver}	ASP.NET API, plugin loader.
ui	stella/ui:{ver}	Pre‑built Angular SPA.
runner-trivy	stella/runner-trivy:{ver}	Trivy CLI + SPDX/CycloneDX 🛠.
runner-grype	stella/runner-grype:{ver}	Optional plug‑in scanner.
🏷️ StellaOps.Registry 📌	stella/registry:{ver}	Scratch image embedding Docker Registry v2 + Cosign policy controller.
🏷️ StellaOps.MutePolicies 📌	stella/policies:{ver}	Sidecar serving policy bundles.
🏷️ StellaOps.Attestor 📌	stella/attestor:{ver}	SLSA provenance & Rekor signer (future).

Images are --label org.opencontainers.image.source=git.stella-ops.ru and include SBOMs generated at build time.

---

## 4 📌 Offline Update Kit (OUK) Build & Distribution

Purpose – deliver updated CVE feeds & Trivy DB to air‑gapped clusters.

### 4.1 CLI Tool

Go binary ouk lives in tools/ouk/.

ouk fetch \
  --nvd --osv \
  --trivy-db --date $(date -I) \
  --output ouk-$(date +%Y%m%d).tar.gz \
  --sign cosign.key

### 4.2 Pipeline Hook

• Runs on first Friday each month (cron).
• Generates tarball, signs it, uploads to GitLab Release asset.
• SHA‑256 + signature published alongside.

### 4.3 Activation Flow (runtime)

1. Admin uploads .tar.gz via UI → Settings → Offline Updates (OUK).
2. Backend verifies Cosign signature & digest.
3. Files extracted into var/lib/stella/db.
4. Redis caches invalidated; Dashboard “Feed Age” ticks green.
5. Audit event ouk_update stored.

4.4 Token Detail

client.jwt placed under /root/ inside the tarball. CI job fails if token expiry < 29 days (guard against stale caches).

---

## 5 Artifact Signing & Transparency

Artefact	Signer	Tool
Git tags	GPG (0x90C4…)	git tag -s
Containers	Cosign key pair	cosign sign
Helm Charts	prov file	helm package --sign
OUK tarballs	Cosign	cosign sign-blob

Rekor integration is TODO – once the internal Rekor mirror is online (StellaOpsAttestor) a post‑publish job will submit transparency log entries.

---

## 6 Release Checklist

1. CI pipeline green.
2. Bump VERSION file.
3. Tag git tag -s X.Y.Z -m "Release X.Y.Z" & push.
4. GitLab CI auto‑publishes images & charts.
5. Draft GitLab Release Notes using tools/release-notes-gen.
6. Verify SBOM attachment with stella sbom verify stella/backend:X.Y.Z.
7. Smoke‑test OUK tarball in offline lab.
8. Announce in #stella-release Mattermost channel.

---

## 7 Hot‑fix Procedure

• Branch from latest tag → hotfix/X.Y.Z+1-hf1.
• Apply minimal patch, add regression test.
• CI pipeline (with reduced stages) must pass.
• Tag X.Y.Z+1.
• Publish only container + Helm chart; OUK not rebuilt.
• Cherry‑pick back to main.

---

## 8 Deprecation & End‑of‑Life Policy

Feature	Deprecation notice	Removal earliest
Legacy CSV policy import	2025‑10‑01	2026‑04‑01
Docker v1 Registry auth	2025‑12‑01	2026‑06‑01
In‑image Trivy DB	2025‑12‑15	2026‑03‑15

At least 6 months notice; removal requires major version bump.

---

## 9 📌 Non‑Commercial Usage Rules (English canonical)

1. Free for internal security assessments (company or personal).
2. SaaS resale / re‑hosting prohibited without prior written consent (AGPL §13).
3. If you distribute a fork with UI or backend modifications you must:
   • Publish the complete modified source code.
   • Retain the original Stella Ops attribution in UI footer and CLI --version.
4. All third‑party dependencies remain under their respective licences (MIT, Apache‑2.0, ISC, BSD).
5. Deployments in state‑regulated or classified environments must obeyapplicable local regulations governing cryptography and software distribution.

---

## 10 Best Practices Snapshot 📌

• SBOM‑per‑image → attach at build time; store as OCI artifact for supply‑chain introspection.
• Provenance flag (--provenance=true) in BuildKit fulfils SLSA 2 requirement.
• Use multi‑arch, reproducible builds (SOURCE_DATE_EPOCH pins timestamps).
• All pipelines enforce Signed‑off‑by (DCO); CI fails if trailer missing.
• cosign policy ensures only images signed by the project key run in production.

---

## 11 Contributing to Release Engineering

• Fork & create MR to infra/release-*.
• All infra changes require green integration-e2e-offline job.
• Discuss larger infra migrations in #sig-release Mattermost; decisions recorded in ADR/ folder.

---

## 12 Change Log (high‑level)

Version	Date	Note
v2.1	2025‑07‑15	Added OUK build/publish pipeline, internal registry image (StellaOps.Registry), non‑commercial usage rules extraction, SBOM stage, BuildKit provenance.
v2.0	2025‑07‑12	Initial open‑sourcing of Release Engineering guide.
v1.1	2025‑07‑09	Fixed inner fencing; added retention policy
v1.0	2025‑07‑09	Initial playbook

---

(End of Release Engineering Playbook v1.1)