git.stella-ops.org/docs/13_RELEASE_ENGINEERING_PLAYBOOK.md

# 13 · Release Engineering Playbook — Stella Ops

A concise, automation‑first guide describing how source code on main becomes a verifiably signed, air‑gap‑friendly release.
It is opinionated for offline use‑cases and supply‑chain security (SLSA ≥ level 2 today, aiming for level 3).

---

## 0 Release Philosophy

• Fast but fearless – every commit on main must be releasable; broken builds break the build, not the team.
• Reproducible – anyone can rebuild byte‑identical artefacts with a single make release offline.
• Secure by default – every artefact ships with a SBOM, Cosign signature and (future) Rekor log entry.
• Offline‑first – all dependencies are vendored or mirrored into the internal registry; no Internet required at runtime.

---

## 1 Versioning & Branching

Branch	Purpose	Auto‑publish?
main	Always‑green development trunk	nightly-* images
release/X.Y	Stabilise a minor line	stella:X.Y-rcN
Tags	X.Y.Z = SemVer	stella:X.Y.Z, OUK tarball, Helm chart

• SemVer – MAJOR for breaking API/CLI changes, MINOR for features, PATCH for fixes.
• Release tags are signed (git tag -s) with the Stella Ops GPG key (0x90C4…).

---

## 2 CI/CD Overview (GitLab CI + GitLab Runner)

graph LR
  A[push / MR] --> Lint
  Lint --> Unit
  Unit --> Build
  Build --> Test-Container
  Test-Container --> SBOM
  SBOM --> Sign
  Sign --> Publish
  Publish --> E2E
  Publish --> Notify

### Pipeline Stages

Stage	Key tasks
Lint	ESLint, golangci‑lint, hadolint, markdown‑lint.
Unit	dotnet test, go test, Jest UI tests.
Quota unit‑tests 🏷	Validate QuotaService logic: reset at UTC, 5 s vs 60 s waits, header correctness.
Build	Multi‑arch container build (linux/amd64, linux/arm64) using BuildKit + --provenance 📌.
Test‑Container	Spin up compose file, run smoke APIs.
SBOM 📌	Invoke StellaOps.SBOMBuilder to generate SPDX JSON + attach .sbom label to image.
Sign	Sign image with Cosign (cosign sign --key cosign.key).
Publish	Push to registry.git.stella-ops.org.
E2E	Kind‑based Kubernetes test incl. Zastava DaemonSet; verify sub‑5 s scan SLA.
Notify	Report to Mattermost & GitLab Slack app.
OfflineToken	Call JwtIssuer.Generate(exp=30d) → store client.jwt artefact → attach to OUK build context

All stages run in parallel where possible; max wall‑time < 15 min.

Implementation note. .gitea/workflows/release.yml executes ops/devops/release/build_release.py to build multi-arch images, attach CycloneDX SBOMs and SLSA provenance with Cosign, and emit out/release/release.yaml for downstream packaging (Helm, Compose, Offline Kit). The build-test-deploy workflow also runs python ops/devops/release/test_verify_release.py so release verifier regressions fail fast during every CI pass.

---

## 3 Container Image Strategy

Image	Registry Tag	Contents
backend	stella/backend:{ver}	ASP.NET API, plugin loader.
ui	stella/ui:{ver}	Pre‑built Angular SPA.
runner-trivy	stella/runner-trivy:{ver}	Trivy CLI + SPDX/CycloneDX 🛠.
runner-grype	stella/runner-grype:{ver}	Optional plug‑in scanner.
🏷️ StellaOps.Registry 📌	stella/registry:{ver}	Scratch image embedding Docker Registry v2 + Cosign policy controller.
🏷️ StellaOps.MutePolicies 📌	stella/policies:{ver}	Sidecar serving policy bundles.
🏷️ StellaOps.Attestor 📌	stella/attestor:{ver}	SLSA provenance & Rekor signer (future).

Images are --label org.opencontainers.image.source=git.stella-ops.ru and include SBOMs generated at build time.

---

## 4 📌 Offline Update Kit (OUK) Build & Distribution

Purpose – deliver updated CVE feeds & Trivy DB to air‑gapped clusters.

### 4.1 CLI Tool

Go binary ouk lives in tools/ouk/.

ouk fetch \
  --nvd --osv \
  --trivy-db --date $(date -I) \
  --output ouk-$(date +%Y%m%d).tar.gz \
  --sign cosign.key

### 4.2 Pipeline Hook

• Runs on first Friday each month (cron).
• Generates tarball, signs it, uploads to GitLab Release asset.
• SHA‑256 + signature published alongside.
• Release job must emit out/release/debug/ with debug-manifest.json and .sha256 so ops/offline-kit/mirror_debug_store.py can mirror symbols into the Offline Kit (see DEVOPS-REL-17-004).

### 4.3 Activation Flow (runtime)

1. Admin uploads .tar.gz via UI → Settings → Offline Updates (OUK).
2. Backend verifies Cosign signature & digest.
3. Files extracted into var/lib/stella/db.
4. Redis caches invalidated; Dashboard “Feed Age” ticks green.
5. Audit event ouk_update stored.

4.4 Token Detail

client.jwt placed under /root/ inside the tarball. CI job fails if token expiry < 29 days (guard against stale caches).

---

## 5 Artifact Signing & Transparency

Artefact	Signer	Tool/Notes
Git tags	GPG (0x90C4…)	git tag -s
Containers	Cosign key pair	cosign sign
Helm Charts	prov file	helm package --sign
OUK tarballs	Cosign	cosign sign-blob
Debug store	—	debug/debug-manifest.json hashed

Rekor integration is TODO – once the internal Rekor mirror is online (StellaOpsAttestor) a post‑publish job will submit transparency log entries.

---

## 6 Release Checklist

1. CI pipeline green.
2. Bump VERSION file.
3. Tag git tag -s X.Y.Z -m "Release X.Y.Z" & push.
4. GitLab CI auto‑publishes images & charts.
5. Draft GitLab Release Notes using tools/release-notes-gen.
6. Verify SBOM attachment with stella sbom verify stella/backend:X.Y.Z.
7. Run the release verifier locally if CI isn’t available (mirrors the workflow step):
   python ops/devops/release/test_verify_release.py
8. Mirror the release debug store into the Offline Kit staging tree and re-check the manifest:

   ./ops/offline-kit/mirror_debug_store.py \
     --release-dir out/release \
     --offline-kit-dir out/offline-kit
   jq '.artifacts | length' out/offline-kit/debug/debug-manifest.json
   readelf -n /app/... | grep -i 'Build ID'
   

   Validate that the hash from readelf matches the .build-id/<aa>/<rest>.debug path created by the script.
9. Smoke-test OUK tarball in offline lab.
10. Announce in #stella-release Mattermost channel.

---

## 7 Hot‑fix Procedure

• Branch from latest tag → hotfix/X.Y.Z+1-hf1.
• Apply minimal patch, add regression test.
• CI pipeline (with reduced stages) must pass.
• Tag X.Y.Z+1.
• Publish only container + Helm chart; OUK not rebuilt.
• Cherry‑pick back to main.

---

## 8 Deprecation & End‑of‑Life Policy

Feature	Deprecation notice	Removal earliest
Legacy CSV policy import	2025‑10‑01	2026‑04‑01
Docker v1 Registry auth	2025‑12‑01	2026‑06‑01
In‑image Trivy DB	2025‑12‑15	2026‑03‑15

At least 6 months notice; removal requires major version bump.

---

## 9 📌 Non‑Commercial Usage Rules (English canonical)

1. Free for internal security assessments (company or personal).
2. SaaS resale / re‑hosting prohibited without prior written consent (AGPL §13).
3. If you distribute a fork with UI or backend modifications you must:
   • Publish the complete modified source code.
   • Retain the original Stella Ops attribution in UI footer and CLI --version.
4. All third‑party dependencies remain under their respective licences (MIT, Apache‑2.0, ISC, BSD).
5. Deployments in state‑regulated or classified environments must obeyapplicable local regulations governing cryptography and software distribution.

---

## 10 Best Practices Snapshot 📌

• SBOM‑per‑image → attach at build time; store as OCI artifact for supply‑chain introspection.
• Provenance flag (--provenance=true) in BuildKit fulfils SLSA 2 requirement.
• Use multi‑arch, reproducible builds (SOURCE_DATE_EPOCH pins timestamps).
• All pipelines enforce Signed‑off‑by (DCO); CI fails if trailer missing.
• cosign policy ensures only images signed by the project key run in production.

---

## 11 Contributing to Release Engineering

• Fork & create MR to infra/release-*.
• All infra changes require green integration-e2e-offline job.
• Discuss larger infra migrations in #sig-release Mattermost; decisions recorded in ADR/ folder.

---

## 12 Change Log (high‑level)

Version	Date	Note
v2.1	2025‑07‑15	Added OUK build/publish pipeline, internal registry image (StellaOps.Registry), non‑commercial usage rules extraction, SBOM stage, BuildKit provenance.
v2.0	2025‑07‑12	Initial open‑sourcing of Release Engineering guide.
v1.1	2025‑07‑09	Fixed inner fencing; added retention policy
v1.0	2025‑07‑09	Initial playbook

---

(End of Release Engineering Playbook v1.1)