feat(docs): Add comprehensive documentation for Vexer, Vulnerability Explorer, and Zastava modules
- Introduced AGENTS.md, README.md, TASKS.md, and implementation_plan.md for Vexer, detailing mission, responsibilities, key components, and operational notes. - Established similar documentation structure for Vulnerability Explorer and Zastava modules, including their respective workflows, integrations, and observability notes. - Created risk scoring profiles documentation outlining the core workflow, factor model, governance, and deliverables. - Ensured all modules adhere to the Aggregation-Only Contract and maintain determinism and provenance in outputs.
This commit is contained in:
@@ -1,62 +1,62 @@
|
||||
# 13 · Release Engineering Playbook — Stella Ops
|
||||
|
||||
|
||||
A concise, automation‑first guide describing **how source code on `main` becomes a verifiably signed, air‑gap‑friendly release**.
|
||||
It is opinionated for offline use‑cases and supply‑chain security (SLSA ≥ level 2 today, aiming for level 3).
|
||||
|
||||
---
|
||||
|
||||
## 0 Release Philosophy
|
||||
|
||||
* **Fast but fearless** – every commit on `main` must be releasable; broken builds break the build, not the team.
|
||||
* **Reproducible** – anyone can rebuild byte‑identical artefacts with a single `make release` offline.
|
||||
* **Secure by default** – every artefact ships with a SBOM, Cosign signature and (future) Rekor log entry.
|
||||
* **Offline‑first** – all dependencies are vendored or mirrored into the internal registry; no Internet required at runtime.
|
||||
|
||||
---
|
||||
|
||||
## 1 Versioning & Branching
|
||||
|
||||
| Branch | Purpose | Auto‑publish? |
|
||||
| ------------- | ------------------------------ | --------------------------------------- |
|
||||
| `main` | Always‑green development trunk | `nightly-*` images |
|
||||
| `release/X.Y` | Stabilise a minor line | `stella:X.Y-rcN` |
|
||||
| Tags | `X.Y.Z` = SemVer | `stella:X.Y.Z`, OUK tarball, Helm chart |
|
||||
|
||||
* **SemVer** – MAJOR for breaking API/CLI changes, MINOR for features, PATCH for fixes.
|
||||
* Release tags are **signed** (`git tag -s`) with the Stella Ops GPG key (`0x90C4…`).
|
||||
|
||||
---
|
||||
|
||||
## 2 CI/CD Overview (GitLab CI + GitLab Runner)
|
||||
|
||||
```mermaid
|
||||
graph LR
|
||||
A[push / MR] --> Lint
|
||||
Lint --> Unit
|
||||
Unit --> Build
|
||||
Build --> Test-Container
|
||||
Test-Container --> SBOM
|
||||
SBOM --> Sign
|
||||
Sign --> Publish
|
||||
Publish --> E2E
|
||||
Publish --> Notify
|
||||
```
|
||||
|
||||
### Pipeline Stages
|
||||
|
||||
| Stage | Key tasks |
|
||||
| ------------------ | ------------------------------------------------------------------------------------------------ |
|
||||
| **Lint** | ESLint, golangci‑lint, hadolint, markdown‑lint. |
|
||||
| **Unit** | `dotnet test`, `go test`, Jest UI tests. |
|
||||
| **Quota unit‑tests 🏷** | Validate QuotaService logic: reset at UTC, 5 s vs 60 s waits, header correctness. |
|
||||
| **Build** | Multi‑arch container build (`linux/amd64`, `linux/arm64`) using **BuildKit** + `--provenance` 📌. |
|
||||
| **Test‑Container** | Spin up compose file, run smoke APIs. |
|
||||
| **SBOM** 📌 | Invoke **StellaOps.SBOMBuilder** to generate SPDX JSON + attach `.sbom` label to image. |
|
||||
| **Sign** | Sign image with **Cosign** (`cosign sign --key cosign.key`). |
|
||||
| **Publish** | Push to `registry.git.stella-ops.org`. |
|
||||
| **E2E** | Kind‑based Kubernetes test incl. Zastava DaemonSet; verify sub‑5 s scan SLA. |
|
||||
| **Notify** | Report to Mattermost & GitLab Slack app. |
|
||||
# 13 · Release Engineering Playbook — Stella Ops
|
||||
|
||||
|
||||
A concise, automation‑first guide describing **how source code on `main` becomes a verifiably signed, air‑gap‑friendly release**.
|
||||
It is opinionated for offline use‑cases and supply‑chain security (SLSA ≥ level 2 today, aiming for level 3).
|
||||
|
||||
---
|
||||
|
||||
## 0 Release Philosophy
|
||||
|
||||
* **Fast but fearless** – every commit on `main` must be releasable; broken builds break the build, not the team.
|
||||
* **Reproducible** – anyone can rebuild byte‑identical artefacts with a single `make release` offline.
|
||||
* **Secure by default** – every artefact ships with a SBOM, Cosign signature and (future) Rekor log entry.
|
||||
* **Offline‑first** – all dependencies are vendored or mirrored into the internal registry; no Internet required at runtime.
|
||||
|
||||
---
|
||||
|
||||
## 1 Versioning & Branching
|
||||
|
||||
| Branch | Purpose | Auto‑publish? |
|
||||
| ------------- | ------------------------------ | --------------------------------------- |
|
||||
| `main` | Always‑green development trunk | `nightly-*` images |
|
||||
| `release/X.Y` | Stabilise a minor line | `stella:X.Y-rcN` |
|
||||
| Tags | `X.Y.Z` = SemVer | `stella:X.Y.Z`, OUK tarball, Helm chart |
|
||||
|
||||
* **SemVer** – MAJOR for breaking API/CLI changes, MINOR for features, PATCH for fixes.
|
||||
* Release tags are **signed** (`git tag -s`) with the Stella Ops GPG key (`0x90C4…`).
|
||||
|
||||
---
|
||||
|
||||
## 2 CI/CD Overview (GitLab CI + GitLab Runner)
|
||||
|
||||
```mermaid
|
||||
graph LR
|
||||
A[push / MR] --> Lint
|
||||
Lint --> Unit
|
||||
Unit --> Build
|
||||
Build --> Test-Container
|
||||
Test-Container --> SBOM
|
||||
SBOM --> Sign
|
||||
Sign --> Publish
|
||||
Publish --> E2E
|
||||
Publish --> Notify
|
||||
```
|
||||
|
||||
### Pipeline Stages
|
||||
|
||||
| Stage | Key tasks |
|
||||
| ------------------ | ------------------------------------------------------------------------------------------------ |
|
||||
| **Lint** | ESLint, golangci‑lint, hadolint, markdown‑lint. |
|
||||
| **Unit** | `dotnet test`, `go test`, Jest UI tests. |
|
||||
| **Quota unit‑tests 🏷** | Validate QuotaService logic: reset at UTC, 5 s vs 60 s waits, header correctness. |
|
||||
| **Build** | Multi‑arch container build (`linux/amd64`, `linux/arm64`) using **BuildKit** + `--provenance` 📌. |
|
||||
| **Test‑Container** | Spin up compose file, run smoke APIs. |
|
||||
| **SBOM** 📌 | Invoke **StellaOps.SBOMBuilder** to generate SPDX JSON + attach `.sbom` label to image. |
|
||||
| **Sign** | Sign image with **Cosign** (`cosign sign --key cosign.key`). |
|
||||
| **Publish** | Push to `registry.git.stella-ops.org`. |
|
||||
| **E2E** | Kind‑based Kubernetes test incl. Zastava DaemonSet; verify sub‑5 s scan SLA. |
|
||||
| **Notify** | Report to Mattermost & GitLab Slack app. |
|
||||
| **OfflineToken** | Call `JwtIssuer.Generate(exp=30d)` → store `client.jwt` artefact → attach to OUK build context |
|
||||
|
||||
*All stages run in parallel where possible; max wall‑time < 15 min.*
|
||||
@@ -70,63 +70,63 @@ The `build-test-deploy` workflow also runs
|
||||
regressions fail fast during every CI pass.
|
||||
|
||||
---
|
||||
|
||||
## 3 Container Image Strategy
|
||||
|
||||
| Image | Registry Tag | Contents |
|
||||
| ------------------------------ | --------------------------- | ---------------------------------------------------------------------- |
|
||||
| **backend** | `stella/backend:{ver}` | ASP.NET API, plugin loader. |
|
||||
| **ui** | `stella/ui:{ver}` | Pre‑built Angular SPA. |
|
||||
| **runner-trivy** | `stella/runner-trivy:{ver}` | Trivy CLI + SPDX/CycloneDX 🛠. |
|
||||
| **runner-grype** | `stella/runner-grype:{ver}` | Optional plug‑in scanner. |
|
||||
| **🏷️ StellaOps.Registry** 📌 | `stella/registry:{ver}` | Scratch image embedding Docker Registry v2 + Cosign policy controller. |
|
||||
| **🏷️ StellaOps.MutePolicies** 📌 | `stella/policies:{ver}` | Sidecar serving policy bundles. |
|
||||
| **🏷️ StellaOps.Attestor** 📌 | `stella/attestor:{ver}` | SLSA provenance & Rekor signer (future). |
|
||||
|
||||
*Images are **`--label org.opencontainers.image.source=git.stella-ops.ru`** and include SBOMs generated at build time.*
|
||||
|
||||
---
|
||||
|
||||
## 4 📌 Offline Update Kit (OUK) Build & Distribution
|
||||
|
||||
**Purpose** – deliver updated CVE feeds & Trivy DB to air‑gapped clusters.
|
||||
|
||||
### 4.1 CLI Tool
|
||||
|
||||
*Go binary `ouk` lives in `tools/ouk/`.*
|
||||
|
||||
```sh
|
||||
ouk fetch \
|
||||
--nvd --osv \
|
||||
--trivy-db --date $(date -I) \
|
||||
--output ouk-$(date +%Y%m%d).tar.gz \
|
||||
--sign cosign.key
|
||||
```
|
||||
|
||||
|
||||
## 3 Container Image Strategy
|
||||
|
||||
| Image | Registry Tag | Contents |
|
||||
| ------------------------------ | --------------------------- | ---------------------------------------------------------------------- |
|
||||
| **backend** | `stella/backend:{ver}` | ASP.NET API, plugin loader. |
|
||||
| **ui** | `stella/ui:{ver}` | Pre‑built Angular SPA. |
|
||||
| **runner-trivy** | `stella/runner-trivy:{ver}` | Trivy CLI + SPDX/CycloneDX 🛠. |
|
||||
| **runner-grype** | `stella/runner-grype:{ver}` | Optional plug‑in scanner. |
|
||||
| **🏷️ StellaOps.Registry** 📌 | `stella/registry:{ver}` | Scratch image embedding Docker Registry v2 + Cosign policy controller. |
|
||||
| **🏷️ StellaOps.MutePolicies** 📌 | `stella/policies:{ver}` | Sidecar serving policy bundles. |
|
||||
| **🏷️ StellaOps.Attestor** 📌 | `stella/attestor:{ver}` | SLSA provenance & Rekor signer (future). |
|
||||
|
||||
*Images are **`--label org.opencontainers.image.source=git.stella-ops.ru`** and include SBOMs generated at build time.*
|
||||
|
||||
---
|
||||
|
||||
## 4 📌 Offline Update Kit (OUK) Build & Distribution
|
||||
|
||||
**Purpose** – deliver updated CVE feeds & Trivy DB to air‑gapped clusters.
|
||||
|
||||
### 4.1 CLI Tool
|
||||
|
||||
*Go binary `ouk` lives in `src/Tools/ouk/`.*
|
||||
|
||||
```sh
|
||||
ouk fetch \
|
||||
--nvd --osv \
|
||||
--trivy-db --date $(date -I) \
|
||||
--output ouk-$(date +%Y%m%d).tar.gz \
|
||||
--sign cosign.key
|
||||
```
|
||||
|
||||
### 4.2 Pipeline Hook
|
||||
|
||||
* Runs on **first Friday** each month (cron).
|
||||
* Generates tarball, signs it, uploads to **GitLab Release asset**.
|
||||
* SHA‑256 + signature published alongside.
|
||||
* Release job must emit `out/release/debug/` with `debug-manifest.json` and `.sha256` so `ops/offline-kit/mirror_debug_store.py` can mirror symbols into the Offline Kit (see `DEVOPS-REL-17-004`).
|
||||
|
||||
### 4.3 Activation Flow (runtime)
|
||||
|
||||
1. Admin uploads `.tar.gz` via **UI → Settings → Offline Updates (OUK)**.
|
||||
2. Backend verifies Cosign signature & digest.
|
||||
3. Files extracted into `var/lib/stella/db`.
|
||||
4. Redis caches invalidated; Dashboard “Feed Age” ticks green.
|
||||
5. Audit event `ouk_update` stored.
|
||||
|
||||
### 4.4 Token Detail
|
||||
|
||||
client.jwt placed under /root/ inside the tarball.
|
||||
CI job fails if token expiry < 29 days (guard against stale caches).
|
||||
|
||||
---
|
||||
|
||||
## 5 Artifact Signing & Transparency
|
||||
|
||||
|
||||
### 4.3 Activation Flow (runtime)
|
||||
|
||||
1. Admin uploads `.tar.gz` via **UI → Settings → Offline Updates (OUK)**.
|
||||
2. Backend verifies Cosign signature & digest.
|
||||
3. Files extracted into `var/lib/stella/db`.
|
||||
4. Redis caches invalidated; Dashboard “Feed Age” ticks green.
|
||||
5. Audit event `ouk_update` stored.
|
||||
|
||||
### 4.4 Token Detail
|
||||
|
||||
client.jwt placed under /root/ inside the tarball.
|
||||
CI job fails if token expiry < 29 days (guard against stale caches).
|
||||
|
||||
---
|
||||
|
||||
## 5 Artifact Signing & Transparency
|
||||
|
||||
| Artefact | Signer | Tool/Notes |
|
||||
| ------------ | --------------- | ---------------------------------- |
|
||||
| Git tags | GPG (`0x90C4…`) | `git tag -s` |
|
||||
@@ -134,18 +134,18 @@ CI job fails if token expiry < 29 days (guard against stale caches).
|
||||
| Helm Charts | prov file | `helm package --sign` |
|
||||
| OUK tarballs | Cosign | `cosign sign-blob` |
|
||||
| Debug store | — | `debug/debug-manifest.json` hashed |
|
||||
|
||||
**Rekor** integration is **TODO** – once the internal Rekor mirror is online (`StellaOpsAttestor`) a post‑publish job will submit transparency log entries.
|
||||
|
||||
---
|
||||
|
||||
## 6 Release Checklist
|
||||
|
||||
1. CI pipeline green.
|
||||
2. Bump `VERSION` file.
|
||||
3. Tag `git tag -s X.Y.Z -m "Release X.Y.Z"` & push.
|
||||
4. GitLab CI auto‑publishes images & charts.
|
||||
5. Draft GitLab **Release Notes** using `tools/release-notes-gen`.
|
||||
|
||||
**Rekor** integration is **TODO** – once the internal Rekor mirror is online (`StellaOpsAttestor`) a post‑publish job will submit transparency log entries.
|
||||
|
||||
---
|
||||
|
||||
## 6 Release Checklist
|
||||
|
||||
1. CI pipeline green.
|
||||
2. Bump `VERSION` file.
|
||||
3. Tag `git tag -s X.Y.Z -m "Release X.Y.Z"` & push.
|
||||
4. GitLab CI auto‑publishes images & charts.
|
||||
5. Draft GitLab **Release Notes** using `src/Tools/release-notes-gen`.
|
||||
6. Verify SBOM attachment with `stella sbom verify stella/backend:X.Y.Z`.
|
||||
7. Run the release verifier locally if CI isn’t available (mirrors the workflow step):
|
||||
`python ops/devops/release/test_verify_release.py`
|
||||
@@ -160,71 +160,71 @@ CI job fails if token expiry < 29 days (guard against stale caches).
|
||||
Validate that the hash from `readelf` matches the `.build-id/<aa>/<rest>.debug` path created by the script.
|
||||
9. Smoke-test OUK tarball in offline lab.
|
||||
10. Announce in `#stella-release` Mattermost channel.
|
||||
|
||||
---
|
||||
|
||||
## 7 Hot‑fix Procedure
|
||||
|
||||
* Branch from latest tag → `hotfix/X.Y.Z+1-hf1`.
|
||||
* Apply minimal patch, add regression test.
|
||||
* CI pipeline (with reduced stages) must pass.
|
||||
* Tag `X.Y.Z+1`.
|
||||
* Publish only container + Helm chart; OUK not rebuilt.
|
||||
* Cherry‑pick back to `main`.
|
||||
|
||||
---
|
||||
|
||||
## 8 Deprecation & End‑of‑Life Policy
|
||||
|
||||
| Feature | Deprecation notice | Removal earliest |
|
||||
| ------------------------ | ------------------ | ---------------- |
|
||||
| Legacy CSV policy import | 2025‑10‑01 | 2026‑04‑01 |
|
||||
| Docker v1 Registry auth | 2025‑12‑01 | 2026‑06‑01 |
|
||||
| In‑image Trivy DB | 2025‑12‑15 | 2026‑03‑15 |
|
||||
|
||||
*At least 6 months notice; removal requires major version bump.*
|
||||
|
||||
---
|
||||
|
||||
## 9 📌 Non‑Commercial Usage Rules (English canonical)
|
||||
|
||||
1. **Free for internal security assessments** (company or personal).
|
||||
2. **SaaS resale / re‑hosting prohibited** without prior written consent (AGPL §13).
|
||||
3. If you distribute a fork with UI or backend modifications **you must**:
|
||||
* Publish the complete modified source code.
|
||||
* Retain the original Stella Ops attribution in UI footer and CLI `--version`.
|
||||
4. All third‑party dependencies remain under their respective licences (MIT, Apache‑2.0, ISC, BSD).
|
||||
5. Deployments in state‑regulated or classified environments must obey**applicable local regulations** governing cryptography and software distribution.
|
||||
|
||||
---
|
||||
|
||||
## 10 Best Practices Snapshot 📌
|
||||
|
||||
* **SBOM‑per‑image** → attach at build time; store as OCI artifact for supply‑chain introspection.
|
||||
* **Provenance flag** (`--provenance=true`) in BuildKit fulfils SLSA 2 requirement.
|
||||
* Use **multi‑arch, reproducible builds** (`SOURCE_DATE_EPOCH` pins timestamps).
|
||||
* All pipelines enforce **Signed‑off‑by (DCO)**; CI fails if trailer missing.
|
||||
* `cosign policy` ensures only images signed by the project key run in production.
|
||||
|
||||
---
|
||||
|
||||
## 11 Contributing to Release Engineering
|
||||
|
||||
* Fork & create MR to `infra/release-*`.
|
||||
* All infra changes require green **`integration-e2e-offline`** job.
|
||||
* Discuss larger infra migrations in `#sig-release` Mattermost; decisions recorded in `ADR/` folder.
|
||||
|
||||
---
|
||||
|
||||
## 12 Change Log (high‑level)
|
||||
|
||||
| Version | Date | Note |
|
||||
| ------- | ---------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| v2.1 | 2025‑07‑15 | Added OUK build/publish pipeline, internal registry image (`StellaOps.Registry`), non‑commercial usage rules extraction, SBOM stage, BuildKit provenance. |
|
||||
| v2.0 | 2025‑07‑12 | Initial open‑sourcing of Release Engineering guide. |
|
||||
| v1.1 | 2025‑07‑09 | Fixed inner fencing; added retention policy |
|
||||
| v1.0 | 2025‑07‑09 | Initial playbook |
|
||||
|
||||
---
|
||||
|
||||
*(End of Release Engineering Playbook v1.1)*
|
||||
|
||||
---
|
||||
|
||||
## 7 Hot‑fix Procedure
|
||||
|
||||
* Branch from latest tag → `hotfix/X.Y.Z+1-hf1`.
|
||||
* Apply minimal patch, add regression test.
|
||||
* CI pipeline (with reduced stages) must pass.
|
||||
* Tag `X.Y.Z+1`.
|
||||
* Publish only container + Helm chart; OUK not rebuilt.
|
||||
* Cherry‑pick back to `main`.
|
||||
|
||||
---
|
||||
|
||||
## 8 Deprecation & End‑of‑Life Policy
|
||||
|
||||
| Feature | Deprecation notice | Removal earliest |
|
||||
| ------------------------ | ------------------ | ---------------- |
|
||||
| Legacy CSV policy import | 2025‑10‑01 | 2026‑04‑01 |
|
||||
| Docker v1 Registry auth | 2025‑12‑01 | 2026‑06‑01 |
|
||||
| In‑image Trivy DB | 2025‑12‑15 | 2026‑03‑15 |
|
||||
|
||||
*At least 6 months notice; removal requires major version bump.*
|
||||
|
||||
---
|
||||
|
||||
## 9 📌 Non‑Commercial Usage Rules (English canonical)
|
||||
|
||||
1. **Free for internal security assessments** (company or personal).
|
||||
2. **SaaS resale / re‑hosting prohibited** without prior written consent (AGPL §13).
|
||||
3. If you distribute a fork with UI or backend modifications **you must**:
|
||||
* Publish the complete modified source code.
|
||||
* Retain the original Stella Ops attribution in UI footer and CLI `--version`.
|
||||
4. All third‑party dependencies remain under their respective licences (MIT, Apache‑2.0, ISC, BSD).
|
||||
5. Deployments in state‑regulated or classified environments must obey**applicable local regulations** governing cryptography and software distribution.
|
||||
|
||||
---
|
||||
|
||||
## 10 Best Practices Snapshot 📌
|
||||
|
||||
* **SBOM‑per‑image** → attach at build time; store as OCI artifact for supply‑chain introspection.
|
||||
* **Provenance flag** (`--provenance=true`) in BuildKit fulfils SLSA 2 requirement.
|
||||
* Use **multi‑arch, reproducible builds** (`SOURCE_DATE_EPOCH` pins timestamps).
|
||||
* All pipelines enforce **Signed‑off‑by (DCO)**; CI fails if trailer missing.
|
||||
* `cosign policy` ensures only images signed by the project key run in production.
|
||||
|
||||
---
|
||||
|
||||
## 11 Contributing to Release Engineering
|
||||
|
||||
* Fork & create MR to `infra/release-*`.
|
||||
* All infra changes require green **`integration-e2e-offline`** job.
|
||||
* Discuss larger infra migrations in `#sig-release` Mattermost; decisions recorded in `ADR/` folder.
|
||||
|
||||
---
|
||||
|
||||
## 12 Change Log (high‑level)
|
||||
|
||||
| Version | Date | Note |
|
||||
| ------- | ---------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| v2.1 | 2025‑07‑15 | Added OUK build/publish pipeline, internal registry image (`StellaOps.Registry`), non‑commercial usage rules extraction, SBOM stage, BuildKit provenance. |
|
||||
| v2.0 | 2025‑07‑12 | Initial open‑sourcing of Release Engineering guide. |
|
||||
| v1.1 | 2025‑07‑09 | Fixed inner fencing; added retention policy |
|
||||
| v1.0 | 2025‑07‑09 | Initial playbook |
|
||||
|
||||
---
|
||||
|
||||
*(End of Release Engineering Playbook v1.1)*
|
||||
|
||||
Reference in New Issue
Block a user