# 13 · Release Engineering Playbook — Stella Ops  


A concise, automation‑first guide describing **how source code on `main` becomes a verifiably signed, air‑gap‑friendly release**.  
It is opinionated for offline use‑cases and supply‑chain security (SLSA ≥ level 2 today, aiming for level 3).

---

## 0 Release Philosophy

* **Fast but fearless** – every commit on `main` must be releasable; broken builds break the build, not the team.  
* **Reproducible** – anyone can rebuild byte‑identical artefacts with a single `make release` offline.  
* **Secure by default** – every artefact ships with a SBOM, Cosign signature and (future) Rekor log entry.  
* **Offline‑first** – all dependencies are vendored or mirrored into the internal registry; no Internet required at runtime.

---

## 1 Versioning & Branching

| Branch        | Purpose                        | Auto‑publish?                           |
| ------------- | ------------------------------ | --------------------------------------- |
| `main`        | Always‑green development trunk | `nightly-*` images                      |
| `release/X.Y` | Stabilise a minor line         | `stella:X.Y-rcN`                        |
| Tags          | `X.Y.Z` = SemVer               | `stella:X.Y.Z`, OUK tarball, Helm chart |

* **SemVer** – MAJOR for breaking API/CLI changes, MINOR for features, PATCH for fixes.  
* Release tags are **signed** (`git tag -s`) with the Stella Ops GPG key (`0x90C4…`).

---

## 2 CI/CD Overview (GitLab CI + GitLab Runner)

```mermaid
graph LR
  A[push / MR] --> Lint
  Lint --> Unit
  Unit --> Build
  Build --> Test-Container
  Test-Container --> SBOM
  SBOM --> Sign
  Sign --> Publish
  Publish --> E2E
  Publish --> Notify
```

### Pipeline Stages

| Stage              | Key tasks                                                                                        |
| ------------------ | ------------------------------------------------------------------------------------------------ |
| **Lint**           | ESLint, golangci‑lint, hadolint, markdown‑lint.                                                  |
| **Unit**           | `dotnet test`, `go test`, Jest UI tests.                                                         |
| **Quota unit‑tests 🏷**  | Validate QuotaService logic: reset at UTC, 5 s vs 60 s waits, header correctness.              |
| **Build**          | Multi‑arch container build (`linux/amd64`, `linux/arm64`) using **BuildKit** + `--provenance` 📌. |
| **Test‑Container** | Spin up compose file, run smoke APIs.                                                            |
| **SBOM** 📌         | Invoke **StellaOps.SBOMBuilder** to generate SPDX JSON + attach `.sbom` label to image.          |
| **Sign**           | Sign image with **Cosign** (`cosign sign --key cosign.key`).                                     |
| **Publish**        | Push to `registry.git.stella-ops.org`.                                                            |
| **E2E**            | Kind‑based Kubernetes test incl. Zastava DaemonSet; verify sub‑5 s scan SLA.                     |
| **Notify**         | Report to Mattermost & GitLab Slack app.                                                         |
| **OfflineToken**         | Call `JwtIssuer.Generate(exp=30d)` → store `client.jwt` artefact → attach to OUK build context  |

*All stages run in parallel where possible; max wall‑time < 15 min.*

---

## 3 Container Image Strategy

| Image                          | Registry Tag                | Contents                                                               |
| ------------------------------ | --------------------------- | ---------------------------------------------------------------------- |
| **backend**                    | `stella/backend:{ver}`      | ASP.NET API, plugin loader.                                            |
| **ui**                         | `stella/ui:{ver}`           | Pre‑built Angular SPA.                                                 |
| **runner-trivy**               | `stella/runner-trivy:{ver}` | Trivy CLI + SPDX/CycloneDX 🛠.                                          |
| **runner-grype**               | `stella/runner-grype:{ver}` | Optional plug‑in scanner.                                              |
| **🏷️ StellaOps.Registry** 📌     | `stella/registry:{ver}`     | Scratch image embedding Docker Registry v2 + Cosign policy controller. |
| **🏷️ StellaOps.MutePolicies** 📌 | `stella/policies:{ver}`     | Sidecar serving policy bundles.                                        |
| **🏷️ StellaOps.Attestor** 📌     | `stella/attestor:{ver}`     | SLSA provenance & Rekor signer (future).                               |

*Images are **`--label org.opencontainers.image.source=git.stella-ops.ru`** and include SBOMs generated at build time.*

---

## 4 📌 Offline Update Kit (OUK) Build & Distribution

**Purpose** – deliver updated CVE feeds & Trivy DB to air‑gapped clusters.

### 4.1 CLI Tool

*Go binary `ouk` lives in `tools/ouk/`.*

```sh
ouk fetch \
  --nvd --osv \
  --trivy-db --date $(date -I) \
  --output ouk-$(date +%Y%m%d).tar.gz \
  --sign cosign.key
```

### 4.2 Pipeline Hook

* Runs on **first Friday** each month (cron).  
* Generates tarball, signs it, uploads to **GitLab Release asset**.  
* SHA‑256 + signature published alongside.

### 4.3 Activation Flow (runtime)

1. Admin uploads `.tar.gz` via **UI → Settings → Offline Updates (OUK)**.  
2. Backend verifies Cosign signature & digest.  
3. Files extracted into `var/lib/stella/db`.  
4. Redis caches invalidated; Dashboard “Feed Age” ticks green.  
5. Audit event `ouk_update` stored.

### 4.4 Token Detail

client.jwt placed under /root/ inside the tarball.
CI job fails if token expiry < 29 days (guard against stale caches).

---

## 5 Artifact Signing & Transparency

| Artefact     | Signer          | Tool                  |
| ------------ | --------------- | --------------------- |
| Git tags     | GPG (`0x90C4…`) | `git tag -s`          |
| Containers   | Cosign key pair | `cosign sign`         |
| Helm Charts  | prov file       | `helm package --sign` |
| OUK tarballs | Cosign          | `cosign sign-blob`    |

**Rekor** integration is **TODO** – once the internal Rekor mirror is online (`StellaOpsAttestor`) a post‑publish job will submit transparency log entries.

---

## 6 Release Checklist

1. CI pipeline green.  
2. Bump `VERSION` file.  
3. Tag `git tag -s X.Y.Z -m "Release X.Y.Z"` & push.  
4. GitLab CI auto‑publishes images & charts.  
5. Draft GitLab **Release Notes** using `tools/release-notes-gen`.  
6. Verify SBOM attachment with `stella sbom verify stella/backend:X.Y.Z`.  
7. Smoke‑test OUK tarball in offline lab.  
8. Announce in `#stella-release` Mattermost channel.

---

## 7 Hot‑fix Procedure

* Branch from latest tag → `hotfix/X.Y.Z+1-hf1`.  
* Apply minimal patch, add regression test.  
* CI pipeline (with reduced stages) must pass.  
* Tag `X.Y.Z+1`.  
* Publish only container + Helm chart; OUK not rebuilt.  
* Cherry‑pick back to `main`.

---

## 8 Deprecation & End‑of‑Life Policy

| Feature                  | Deprecation notice | Removal earliest |
| ------------------------ | ------------------ | ---------------- |
| Legacy CSV policy import | 2025‑10‑01         | 2026‑04‑01       |
| Docker v1 Registry auth  | 2025‑12‑01         | 2026‑06‑01       |
| In‑image Trivy DB        | 2025‑12‑15         | 2026‑03‑15       |

*At least 6 months notice; removal requires major version bump.*

---

## 9 📌 Non‑Commercial Usage Rules (English canonical)

1. **Free for internal security assessments** (company or personal).  
2. **SaaS resale / re‑hosting prohibited** without prior written consent (AGPL §13).  
3. If you distribute a fork with UI or backend modifications **you must**:  
   * Publish the complete modified source code.  
   * Retain the original Stella Ops attribution in UI footer and CLI `--version`.  
4. All third‑party dependencies remain under their respective licences (MIT, Apache‑2.0, ISC, BSD).  
5. Deployments in state‑regulated or classified environments must obey**applicable local regulations** governing cryptography and software distribution.

---

## 10 Best Practices Snapshot 📌

* **SBOM‑per‑image** → attach at build time; store as OCI artifact for supply‑chain introspection.  
* **Provenance flag** (`--provenance=true`) in BuildKit fulfils SLSA 2 requirement.  
* Use **multi‑arch, reproducible builds** (`SOURCE_DATE_EPOCH` pins timestamps).  
* All pipelines enforce **Signed‑off‑by (DCO)**; CI fails if trailer missing.  
* `cosign policy` ensures only images signed by the project key run in production.

---

## 11 Contributing to Release Engineering

* Fork & create MR to `infra/release-*`.  
* All infra changes require green **`integration-e2e-offline`** job.  
* Discuss larger infra migrations in `#sig-release` Mattermost; decisions recorded in `ADR/` folder.

---

## 12 Change Log (high‑level)

| Version | Date       | Note                                                                                                                                                      |
| ------- | ---------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- |
| v2.1    | 2025‑07‑15 | Added OUK build/publish pipeline, internal registry image (`StellaOps.Registry`), non‑commercial usage rules extraction, SBOM stage, BuildKit provenance. |
| v2.0    | 2025‑07‑12 | Initial open‑sourcing of Release Engineering guide.                                                                                                       |
| v1.1    | 2025‑07‑09 | Fixed inner fencing; added retention policy                                                                                                               |
| v1.0    | 2025‑07‑09 | Initial playbook                                                                                                                                          |

---

*(End of Release Engineering Playbook v1.1)*
