master
7b5bdcf4d3
- Introduced AGENTS.md, README.md, TASKS.md, and implementation_plan.md for Vexer, detailing mission, responsibilities, key components, and operational notes. - Established similar documentation structure for Vulnerability Explorer and Zastava modules, including their respective workflows, integrations, and observability notes. - Created risk scoring profiles documentation outlining the core workflow, factor model, governance, and deliverables. - Ensured all modules adhere to the Aggregation-Only Contract and maintain determinism and provenance in outputs.
231 lines
12 KiB
Markdown
Executable File
231 lines
12 KiB
Markdown
Executable File
# 13 · Release Engineering Playbook — Stella Ops
|
||
|
||
|
||
A concise, automation‑first guide describing **how source code on `main` becomes a verifiably signed, air‑gap‑friendly release**.
|
||
It is opinionated for offline use‑cases and supply‑chain security (SLSA ≥ level 2 today, aiming for level 3).
|
||
|
||
---
|
||
|
||
## 0 Release Philosophy
|
||
|
||
* **Fast but fearless** – every commit on `main` must be releasable; broken builds break the build, not the team.
|
||
* **Reproducible** – anyone can rebuild byte‑identical artefacts with a single `make release` offline.
|
||
* **Secure by default** – every artefact ships with a SBOM, Cosign signature and (future) Rekor log entry.
|
||
* **Offline‑first** – all dependencies are vendored or mirrored into the internal registry; no Internet required at runtime.
|
||
|
||
---
|
||
|
||
## 1 Versioning & Branching
|
||
|
||
| Branch | Purpose | Auto‑publish? |
|
||
| ------------- | ------------------------------ | --------------------------------------- |
|
||
| `main` | Always‑green development trunk | `nightly-*` images |
|
||
| `release/X.Y` | Stabilise a minor line | `stella:X.Y-rcN` |
|
||
| Tags | `X.Y.Z` = SemVer | `stella:X.Y.Z`, OUK tarball, Helm chart |
|
||
|
||
* **SemVer** – MAJOR for breaking API/CLI changes, MINOR for features, PATCH for fixes.
|
||
* Release tags are **signed** (`git tag -s`) with the Stella Ops GPG key (`0x90C4…`).
|
||
|
||
---
|
||
|
||
## 2 CI/CD Overview (GitLab CI + GitLab Runner)
|
||
|
||
```mermaid
|
||
graph LR
|
||
A[push / MR] --> Lint
|
||
Lint --> Unit
|
||
Unit --> Build
|
||
Build --> Test-Container
|
||
Test-Container --> SBOM
|
||
SBOM --> Sign
|
||
Sign --> Publish
|
||
Publish --> E2E
|
||
Publish --> Notify
|
||
```
|
||
|
||
### Pipeline Stages
|
||
|
||
| Stage | Key tasks |
|
||
| ------------------ | ------------------------------------------------------------------------------------------------ |
|
||
| **Lint** | ESLint, golangci‑lint, hadolint, markdown‑lint. |
|
||
| **Unit** | `dotnet test`, `go test`, Jest UI tests. |
|
||
| **Quota unit‑tests 🏷** | Validate QuotaService logic: reset at UTC, 5 s vs 60 s waits, header correctness. |
|
||
| **Build** | Multi‑arch container build (`linux/amd64`, `linux/arm64`) using **BuildKit** + `--provenance` 📌. |
|
||
| **Test‑Container** | Spin up compose file, run smoke APIs. |
|
||
| **SBOM** 📌 | Invoke **StellaOps.SBOMBuilder** to generate SPDX JSON + attach `.sbom` label to image. |
|
||
| **Sign** | Sign image with **Cosign** (`cosign sign --key cosign.key`). |
|
||
| **Publish** | Push to `registry.git.stella-ops.org`. |
|
||
| **E2E** | Kind‑based Kubernetes test incl. Zastava DaemonSet; verify sub‑5 s scan SLA. |
|
||
| **Notify** | Report to Mattermost & GitLab Slack app. |
|
||
| **OfflineToken** | Call `JwtIssuer.Generate(exp=30d)` → store `client.jwt` artefact → attach to OUK build context |
|
||
|
||
*All stages run in parallel where possible; max wall‑time < 15 min.*
|
||
|
||
**Implementation note.** `.gitea/workflows/release.yml` executes
|
||
`ops/devops/release/build_release.py` to build multi-arch images, attach
|
||
CycloneDX SBOMs and SLSA provenance with Cosign, and emit
|
||
`out/release/release.yaml` for downstream packaging (Helm, Compose, Offline Kit).
|
||
The `build-test-deploy` workflow also runs
|
||
`python ops/devops/release/test_verify_release.py` so release verifier
|
||
regressions fail fast during every CI pass.
|
||
|
||
---
|
||
|
||
## 3 Container Image Strategy
|
||
|
||
| Image | Registry Tag | Contents |
|
||
| ------------------------------ | --------------------------- | ---------------------------------------------------------------------- |
|
||
| **backend** | `stella/backend:{ver}` | ASP.NET API, plugin loader. |
|
||
| **ui** | `stella/ui:{ver}` | Pre‑built Angular SPA. |
|
||
| **runner-trivy** | `stella/runner-trivy:{ver}` | Trivy CLI + SPDX/CycloneDX 🛠. |
|
||
| **runner-grype** | `stella/runner-grype:{ver}` | Optional plug‑in scanner. |
|
||
| **🏷️ StellaOps.Registry** 📌 | `stella/registry:{ver}` | Scratch image embedding Docker Registry v2 + Cosign policy controller. |
|
||
| **🏷️ StellaOps.MutePolicies** 📌 | `stella/policies:{ver}` | Sidecar serving policy bundles. |
|
||
| **🏷️ StellaOps.Attestor** 📌 | `stella/attestor:{ver}` | SLSA provenance & Rekor signer (future). |
|
||
|
||
*Images are **`--label org.opencontainers.image.source=git.stella-ops.ru`** and include SBOMs generated at build time.*
|
||
|
||
---
|
||
|
||
## 4 📌 Offline Update Kit (OUK) Build & Distribution
|
||
|
||
**Purpose** – deliver updated CVE feeds & Trivy DB to air‑gapped clusters.
|
||
|
||
### 4.1 CLI Tool
|
||
|
||
*Go binary `ouk` lives in `src/Tools/ouk/`.*
|
||
|
||
```sh
|
||
ouk fetch \
|
||
--nvd --osv \
|
||
--trivy-db --date $(date -I) \
|
||
--output ouk-$(date +%Y%m%d).tar.gz \
|
||
--sign cosign.key
|
||
```
|
||
|
||
### 4.2 Pipeline Hook
|
||
|
||
* Runs on **first Friday** each month (cron).
|
||
* Generates tarball, signs it, uploads to **GitLab Release asset**.
|
||
* SHA‑256 + signature published alongside.
|
||
* Release job must emit `out/release/debug/` with `debug-manifest.json` and `.sha256` so `ops/offline-kit/mirror_debug_store.py` can mirror symbols into the Offline Kit (see `DEVOPS-REL-17-004`).
|
||
|
||
### 4.3 Activation Flow (runtime)
|
||
|
||
1. Admin uploads `.tar.gz` via **UI → Settings → Offline Updates (OUK)**.
|
||
2. Backend verifies Cosign signature & digest.
|
||
3. Files extracted into `var/lib/stella/db`.
|
||
4. Redis caches invalidated; Dashboard “Feed Age” ticks green.
|
||
5. Audit event `ouk_update` stored.
|
||
|
||
### 4.4 Token Detail
|
||
|
||
client.jwt placed under /root/ inside the tarball.
|
||
CI job fails if token expiry < 29 days (guard against stale caches).
|
||
|
||
---
|
||
|
||
## 5 Artifact Signing & Transparency
|
||
|
||
| Artefact | Signer | Tool/Notes |
|
||
| ------------ | --------------- | ---------------------------------- |
|
||
| Git tags | GPG (`0x90C4…`) | `git tag -s` |
|
||
| Containers | Cosign key pair | `cosign sign` |
|
||
| Helm Charts | prov file | `helm package --sign` |
|
||
| OUK tarballs | Cosign | `cosign sign-blob` |
|
||
| Debug store | — | `debug/debug-manifest.json` hashed |
|
||
|
||
**Rekor** integration is **TODO** – once the internal Rekor mirror is online (`StellaOpsAttestor`) a post‑publish job will submit transparency log entries.
|
||
|
||
---
|
||
|
||
## 6 Release Checklist
|
||
|
||
1. CI pipeline green.
|
||
2. Bump `VERSION` file.
|
||
3. Tag `git tag -s X.Y.Z -m "Release X.Y.Z"` & push.
|
||
4. GitLab CI auto‑publishes images & charts.
|
||
5. Draft GitLab **Release Notes** using `src/Tools/release-notes-gen`.
|
||
6. Verify SBOM attachment with `stella sbom verify stella/backend:X.Y.Z`.
|
||
7. Run the release verifier locally if CI isn’t available (mirrors the workflow step):
|
||
`python ops/devops/release/test_verify_release.py`
|
||
8. Mirror the release debug store into the Offline Kit staging tree and re-check the manifest:
|
||
```bash
|
||
./ops/offline-kit/mirror_debug_store.py \
|
||
--release-dir out/release \
|
||
--offline-kit-dir out/offline-kit
|
||
jq '.artifacts | length' out/offline-kit/debug/debug-manifest.json
|
||
readelf -n /app/... | grep -i 'Build ID'
|
||
```
|
||
Validate that the hash from `readelf` matches the `.build-id/<aa>/<rest>.debug` path created by the script.
|
||
9. Smoke-test OUK tarball in offline lab.
|
||
10. Announce in `#stella-release` Mattermost channel.
|
||
|
||
---
|
||
|
||
## 7 Hot‑fix Procedure
|
||
|
||
* Branch from latest tag → `hotfix/X.Y.Z+1-hf1`.
|
||
* Apply minimal patch, add regression test.
|
||
* CI pipeline (with reduced stages) must pass.
|
||
* Tag `X.Y.Z+1`.
|
||
* Publish only container + Helm chart; OUK not rebuilt.
|
||
* Cherry‑pick back to `main`.
|
||
|
||
---
|
||
|
||
## 8 Deprecation & End‑of‑Life Policy
|
||
|
||
| Feature | Deprecation notice | Removal earliest |
|
||
| ------------------------ | ------------------ | ---------------- |
|
||
| Legacy CSV policy import | 2025‑10‑01 | 2026‑04‑01 |
|
||
| Docker v1 Registry auth | 2025‑12‑01 | 2026‑06‑01 |
|
||
| In‑image Trivy DB | 2025‑12‑15 | 2026‑03‑15 |
|
||
|
||
*At least 6 months notice; removal requires major version bump.*
|
||
|
||
---
|
||
|
||
## 9 📌 Non‑Commercial Usage Rules (English canonical)
|
||
|
||
1. **Free for internal security assessments** (company or personal).
|
||
2. **SaaS resale / re‑hosting prohibited** without prior written consent (AGPL §13).
|
||
3. If you distribute a fork with UI or backend modifications **you must**:
|
||
* Publish the complete modified source code.
|
||
* Retain the original Stella Ops attribution in UI footer and CLI `--version`.
|
||
4. All third‑party dependencies remain under their respective licences (MIT, Apache‑2.0, ISC, BSD).
|
||
5. Deployments in state‑regulated or classified environments must obey**applicable local regulations** governing cryptography and software distribution.
|
||
|
||
---
|
||
|
||
## 10 Best Practices Snapshot 📌
|
||
|
||
* **SBOM‑per‑image** → attach at build time; store as OCI artifact for supply‑chain introspection.
|
||
* **Provenance flag** (`--provenance=true`) in BuildKit fulfils SLSA 2 requirement.
|
||
* Use **multi‑arch, reproducible builds** (`SOURCE_DATE_EPOCH` pins timestamps).
|
||
* All pipelines enforce **Signed‑off‑by (DCO)**; CI fails if trailer missing.
|
||
* `cosign policy` ensures only images signed by the project key run in production.
|
||
|
||
---
|
||
|
||
## 11 Contributing to Release Engineering
|
||
|
||
* Fork & create MR to `infra/release-*`.
|
||
* All infra changes require green **`integration-e2e-offline`** job.
|
||
* Discuss larger infra migrations in `#sig-release` Mattermost; decisions recorded in `ADR/` folder.
|
||
|
||
---
|
||
|
||
## 12 Change Log (high‑level)
|
||
|
||
| Version | Date | Note |
|
||
| ------- | ---------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||
| v2.1 | 2025‑07‑15 | Added OUK build/publish pipeline, internal registry image (`StellaOps.Registry`), non‑commercial usage rules extraction, SBOM stage, BuildKit provenance. |
|
||
| v2.0 | 2025‑07‑12 | Initial open‑sourcing of Release Engineering guide. |
|
||
| v1.1 | 2025‑07‑09 | Fixed inner fencing; added retention policy |
|
||
| v1.0 | 2025‑07‑09 | Initial playbook |
|
||
|
||
---
|
||
|
||
*(End of Release Engineering Playbook v1.1)*
|