stella-ops.org/git.stella-ops.org
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
main
git.stella-ops.org/docs/13_RELEASE_ENGINEERING_PLAYBOOK.md

10 KiB
Raw Permalink Blame History

# 13 · Release Engineering Playbook — Stella Ops

A concise, automation‑first guide describing how source code on main becomes a verifiably signed, air‑gap‑friendly release.
It is opinionated for offline use‑cases and supply‑chain security (SLSA ≥ level 2 today, aiming for level 3).

## 0 Release Philosophy

  • Fast but fearless – every commit on main must be releasable; broken builds break the build, not the team.
  • Reproducible – anyone can rebuild byte‑identical artefacts with a single make release offline.
  • Secure by default – every artefact ships with a SBOM, Cosign signature and (future) Rekor log entry.
  • Offline‑first – all dependencies are vendored or mirrored into the internal registry; no Internet required at runtime.

## 1 Versioning & Branching

Branch Purpose Auto‑publish?
main Always‑green development trunk nightly-* images
release/X.Y Stabilise a minor line stella:X.Y-rcN
Tags X.Y.Z = SemVer stella:X.Y.Z, OUK tarball, Helm chart
  • SemVer – MAJOR for breaking API/CLI changes, MINOR for features, PATCH for fixes.
  • Release tags are signed (git tag -s) with the Stella Ops GPG key (0x90C4…).

## 2 CI/CD Overview (GitLab CI + GitLab Runner)

graph LR
  A[push / MR] --> Lint
  Lint --> Unit
  Unit --> Build
  Build --> Test-Container
  Test-Container --> SBOM
  SBOM --> Sign
  Sign --> Publish
  Publish --> E2E
  Publish --> Notify

### Pipeline Stages

Stage Key tasks
Lint ESLint, golangci‑lint, hadolint, markdown‑lint.
Unit dotnet test, go test, Jest UI tests.
Quota unit‑tests 🏷 Validate QuotaService logic: reset at UTC, 5 s vs 60 s waits, header correctness.
Build Multi‑arch container build (linux/amd64, linux/arm64) using BuildKit + --provenance 📌.
Test‑Container Spin up compose file, run smoke APIs.
SBOM 📌 Invoke StellaOps.SBOMBuilder to generate SPDX JSON + attach .sbom label to image.
Sign Sign image with Cosign (cosign sign --key cosign.key).
Publish Push to registry.git.stella-ops.ru.
E2E Kind‑based Kubernetes test incl. Zastava DaemonSet; verify sub‑5 s scan SLA.
Notify Report to Mattermost & GitLab Slack app.
OfflineToken Call JwtIssuer.Generate(exp=30d) → store client.jwt artefact → attach to OUK build context

All stages run in parallel where possible; max wall‑time < 15 min.

## 3 Container Image Strategy

Image Registry Tag Contents
backend stella/backend:{ver} ASP.NET API, plugin loader.
ui stella/ui:{ver} Pre‑built Angular SPA.
runner-trivy stella/runner-trivy:{ver} Trivy CLI + SPDX/CycloneDX 🛠.
runner-grype stella/runner-grype:{ver} Optional plug‑in scanner.
🏷️ StellaOps.Registry 📌 stella/registry:{ver} Scratch image embedding Docker Registry v2 + Cosign policy controller.
🏷️ StellaOps.MutePolicies 📌 stella/policies:{ver} Sidecar serving policy bundles.
🏷️ StellaOps.Attestor 📌 stella/attestor:{ver} SLSA provenance & Rekor signer (future).

Images are --label org.opencontainers.image.source=git.stella-ops.ru and include SBOMs generated at build time.

## 4 📌 Offline Update Kit (OUK) Build & Distribution

Purpose – deliver updated CVE feeds & Trivy DB to air‑gapped clusters.

### 4.1 CLI Tool

Go binary ouk lives in tools/ouk/.

ouk fetch \
  --nvd --osv \
  --trivy-db --date $(date -I) \
  --output ouk-$(date +%Y%m%d).tar.gz \
  --sign cosign.key

### 4.2 Pipeline Hook

  • Runs on first Friday each month (cron).
  • Generates tarball, signs it, uploads to GitLab Release asset.
  • SHA‑256 + signature published alongside.

### 4.3 Activation Flow (runtime)

  1. Admin uploads .tar.gz via UI → Settings → Offline Updates (OUK).
  2. Backend verifies Cosign signature & digest.
  3. Files extracted into var/lib/stella/db.
  4. Redis caches invalidated; Dashboard “Feed Age” ticks green.
  5. Audit event ouk_update stored.

4.4 Token Detail

client.jwt placed under /root/ inside the tarball. CI job fails if token expiry < 29 days (guard against stale caches).

## 5 Artifact Signing & Transparency

Artefact Signer Tool
Git tags GPG (0x90C4…) git tag -s
Containers Cosign key pair cosign sign
Helm Charts prov file helm package --sign
OUK tarballs Cosign cosign sign-blob

Rekor integration is TODO – once the internal Rekor mirror is online (StellaOpsAttestor) a post‑publish job will submit transparency log entries.

## 6 Release Checklist

  1. CI pipeline green.
  2. Bump VERSION file.
  3. Tag git tag -s X.Y.Z -m "Release X.Y.Z" & push.
  4. GitLab CI auto‑publishes images & charts.
  5. Draft GitLab Release Notes using tools/release-notes-gen.
  6. Verify SBOM attachment with stella sbom verify stella/backend:X.Y.Z.
  7. Smoke‑test OUK tarball in offline lab.
  8. Announce in #stella-release Mattermost channel.

## 7 Hot‑fix Procedure

  • Branch from latest tag → hotfix/X.Y.Z+1-hf1.
  • Apply minimal patch, add regression test.
  • CI pipeline (with reduced stages) must pass.
  • Tag X.Y.Z+1.
  • Publish only container + Helm chart; OUK not rebuilt.
  • Cherry‑pick back to main.

## 8 Deprecation & End‑of‑Life Policy

Feature Deprecation notice Removal earliest
Legacy CSV policy import 2025‑10‑01 2026‑04‑01
Docker v1 Registry auth 2025‑12‑01 2026‑06‑01
In‑image Trivy DB 2025‑12‑15 2026‑03‑15

At least 6 months notice; removal requires major version bump.

## 9 📌 Non‑Commercial Usage Rules (English canonical)

  1. Free for internal security assessments (company or personal).
  2. SaaS resale / re‑hosting prohibited without prior written consent (AGPL §13).
  3. If you distribute a fork with UI or backend modifications you must:
    • Publish the complete modified source code.
    • Retain the original Stella Ops attribution in UI footer and CLI --version.
  4. All third‑party dependencies remain under their respective licences (MIT, Apache‑2.0, ISC, BSD).
  5. Deployments in state‑regulated or classified environments must obey ФЗ‑187 export rules.

## 10 Best Practices Snapshot 📌

  • SBOM‑per‑image → attach at build time; store as OCI artifact for supply‑chain introspection.
  • Provenance flag (--provenance=true) in BuildKit fulfils SLSA 2 requirement.
  • Use multi‑arch, reproducible builds (SOURCE_DATE_EPOCH pins timestamps).
  • All pipelines enforce Signed‑off‑by (DCO); CI fails if trailer missing.
  • cosign policy ensures only images signed by the project key run in production.

## 11 Contributing to Release Engineering

  • Fork & create MR to infra/release-*.
  • All infra changes require green integration-e2e-offline job.
  • Discuss larger infra migrations in #sig-release Mattermost; decisions recorded in ADR/ folder.

## 12 Change Log (high‑level)

Version Date Note
v2.1 2025‑07‑15 Added OUK build/publish pipeline, internal registry image (StellaOps.Registry), non‑commercial usage rules extraction, SBOM stage, BuildKit provenance.
v2.0 2025‑07‑12 Initial open‑sourcing of Release Engineering guide.
v1.1 2025‑07‑09 Fixed inner fencing; added retention policy
v1.0 2025‑07‑09 Initial playbook

(End of Release Engineering Playbook v1.1)