510 lines
22 KiB
Markdown
510 lines
22 KiB
Markdown
# Release Pipelines
|
|
|
|
> Complete guide to StellaOps release automation including suite releases, module publishing, and promotion workflows.
|
|
|
|
---
|
|
|
|
## Release Strategy Overview
|
|
|
|
StellaOps uses a **dual-versioning strategy**:
|
|
|
|
1. **Suite Releases** - Ubuntu-style `YYYY.MM` versioning with codenames
|
|
2. **Module Releases** - Semantic versioning `MAJOR.MINOR.PATCH` per module
|
|
|
|
### Release Channels
|
|
|
|
| Channel | Purpose | Stability | Update Frequency |
|
|
|---------|---------|-----------|------------------|
|
|
| **Edge** | Latest features, early adopters | Beta | Every merge to main |
|
|
| **Stable** | Production-ready, tested | Production | Bi-weekly |
|
|
| **LTS** | Long-term support, enterprise | Enterprise | Quarterly |
|
|
|
|
---
|
|
|
|
## Suite Release Pipeline
|
|
|
|
### Trigger
|
|
|
|
```bash
|
|
# Tag-based trigger
|
|
git tag suite-2026.04
|
|
git push origin suite-2026.04
|
|
|
|
# Or manual trigger via Gitea Actions UI
|
|
# Workflow: release-suite.yml
|
|
# Inputs: version, codename, channel, skip_tests, dry_run
|
|
```
|
|
|
|
### Workflow: `release-suite.yml`
|
|
|
|
```
|
|
┌─────────────────────────────────────────────────────────────────┐
|
|
│ SUITE RELEASE PIPELINE │
|
|
│ │
|
|
│ ┌──────────────┐ │
|
|
│ │ parse-tag │ (if triggered by tag push) │
|
|
│ │ or validate │ (if triggered manually) │
|
|
│ └──────┬───────┘ │
|
|
│ │ │
|
|
│ ▼ │
|
|
│ ┌──────────────┐ │
|
|
│ │ test-gate │ (optional, skipped with skip_tests=true) │
|
|
│ └──────┬───────┘ │
|
|
│ │ │
|
|
│ ┌────┴────────────────────────────────────────┐ │
|
|
│ │ BUILD PHASE │ │
|
|
│ │ │ │
|
|
│ │ ┌─────────────────┐ ┌─────────────────┐ │ │
|
|
│ │ │ build-modules │ │ build-containers│ │ │
|
|
│ │ │ (9 in parallel)│ │ (9 in parallel)│ │ │
|
|
│ │ └─────────────────┘ └─────────────────┘ │ │
|
|
│ │ │ │
|
|
│ │ ┌─────────────────┐ ┌─────────────────┐ │ │
|
|
│ │ │ build-cli │ │ build-helm │ │ │
|
|
│ │ │ (5 platforms) │ │ │ │ │
|
|
│ │ └─────────────────┘ └─────────────────┘ │ │
|
|
│ │ │ │
|
|
│ └─────────────────────┬────────────────────────┘ │
|
|
│ │ │
|
|
│ ▼ │
|
|
│ ┌───────────────────────────────────────────────┐ │
|
|
│ │ release-manifest │ │
|
|
│ │ - Binary manifest with SHA256 checksums │ │
|
|
│ │ - SBOM generation (CycloneDX, SPDX) │ │
|
|
│ │ - Provenance attestation (in-toto/DSSE) │ │
|
|
│ └───────────────────────┬────────────────────────┘ │
|
|
│ │ │
|
|
│ ┌─────────────────────┴─────────────────────────┐ │
|
|
│ │ │ │
|
|
│ ▼ ▼ ▼ │
|
|
│ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │
|
|
│ │ changelog │ │ suite-docs │ │ compose │ │
|
|
│ │ generation │ │ generation │ │ generation │ │
|
|
│ └──────────────┘ └──────────────┘ └──────────────┘ │
|
|
│ │ │
|
|
│ ▼ │
|
|
│ ┌───────────────────────────────────────────────┐ │
|
|
│ │ create-release │ │
|
|
│ │ - Upload artifacts to Gitea Releases │ │
|
|
│ │ - Sign with Cosign (keyless Sigstore) │ │
|
|
│ │ - Publish to container registry │ │
|
|
│ └───────────────────────┬────────────────────────┘ │
|
|
│ │ │
|
|
│ ▼ │
|
|
│ ┌───────────────────────────────────────────────┐ │
|
|
│ │ commit-docs │ │
|
|
│ │ - Update docs/releases/ │ │
|
|
│ │ - Update devops/compose/ │ │
|
|
│ └───────────────────────────────────────────────┘ │
|
|
│ │
|
|
└─────────────────────────────────────────────────────────────────┘
|
|
```
|
|
|
|
### Suite Versioning
|
|
|
|
| Component | Format | Example |
|
|
|-----------|--------|---------|
|
|
| Suite Version | `YYYY.MM` | `2026.04` |
|
|
| Codename | Alpha name | `Nova`, `Orion`, `Phoenix` |
|
|
| Full Tag | `suite-YYYY.MM` | `suite-2026.04` |
|
|
| Docker Tag | `YYYY.MM-channel` | `2026.04-stable` |
|
|
|
|
### Modules Built
|
|
|
|
| Module | NuGet Package | Container Image |
|
|
|--------|---------------|-----------------|
|
|
| Authority | `StellaOps.Authority` | `stellaops/authority` |
|
|
| Scanner | `StellaOps.Scanner` | `stellaops/scanner` |
|
|
| Concelier | `StellaOps.Concelier` | `stellaops/concelier` |
|
|
| Excititor | `StellaOps.Excititor` | `stellaops/excititor` |
|
|
| SbomService | `StellaOps.SbomService` | `stellaops/sbom-service` |
|
|
| EvidenceLocker | `StellaOps.EvidenceLocker` | `stellaops/evidence-locker` |
|
|
| Policy | `StellaOps.Policy` | `stellaops/policy` |
|
|
| Attestor | `StellaOps.Attestor` | `stellaops/attestor` |
|
|
| VexLens | `StellaOps.VexLens` | `stellaops/vexlens` |
|
|
|
|
### CLI Platforms
|
|
|
|
| Runtime ID | OS | Architecture | Binary Name |
|
|
|------------|-----|--------------|-------------|
|
|
| `linux-x64` | Linux | x86_64 | `stellaops-linux-x64` |
|
|
| `linux-arm64` | Linux | ARM64 | `stellaops-linux-arm64` |
|
|
| `win-x64` | Windows | x86_64 | `stellaops-win-x64.exe` |
|
|
| `osx-x64` | macOS | Intel | `stellaops-osx-x64` |
|
|
| `osx-arm64` | macOS | Apple Silicon | `stellaops-osx-arm64` |
|
|
|
|
---
|
|
|
|
## Module Release Pipeline
|
|
|
|
### Trigger
|
|
|
|
```bash
|
|
# Tag-based trigger
|
|
git tag module-authority-v1.2.3
|
|
git push origin module-authority-v1.2.3
|
|
|
|
# Or manual trigger via Gitea Actions UI
|
|
# Workflow: module-publish.yml
|
|
# Inputs: module, version, publish_nuget, publish_container, prerelease
|
|
```
|
|
|
|
### Workflow: `module-publish.yml`
|
|
|
|
```
|
|
┌─────────────────────────────────────────────────────────────────┐
|
|
│ MODULE PUBLISH PIPELINE │
|
|
│ │
|
|
│ ┌──────────────┐ │
|
|
│ │ parse-tag │ Extract module name and version from tag │
|
|
│ │ or validate │ Normalize manual inputs │
|
|
│ └──────┬───────┘ │
|
|
│ │ │
|
|
│ ┌────┴────────────────────────────────────────┐ │
|
|
│ │ │ │
|
|
│ ▼ ▼ │
|
|
│ ┌──────────────┐ ┌──────────────┐ │
|
|
│ │publish-nuget │ (if flag set) │publish-cont. │ │
|
|
│ │ │ │ (if flag set)│ │
|
|
│ │ - Pack │ │ - Build │ │
|
|
│ │ - Sign │ │ - Scan │ │
|
|
│ │ - Push │ │ - Sign │ │
|
|
│ └──────────────┘ │ - Push │ │
|
|
│ └──────────────┘ │
|
|
│ │ │
|
|
│ OR (if module=CLI) │ │
|
|
│ ▼ │
|
|
│ ┌──────────────────────────────────────────────────┐ │
|
|
│ │ publish-cli │ │
|
|
│ │ - Build for 5 platforms │ │
|
|
│ │ - Native AOT compilation │ │
|
|
│ │ - Code sign binaries │ │
|
|
│ │ - Generate checksums │ │
|
|
│ │ - Upload to release │ │
|
|
│ └──────────────────────────────────────────────────┘ │
|
|
│ │ │
|
|
│ ▼ │
|
|
│ ┌──────────────────────────────────────────────────┐ │
|
|
│ │ summary │ │
|
|
│ │ - Release notes │ │
|
|
│ │ - Artifact links │ │
|
|
│ │ - SBOM references │ │
|
|
│ └──────────────────────────────────────────────────┘ │
|
|
│ │
|
|
└─────────────────────────────────────────────────────────────────┘
|
|
```
|
|
|
|
### Module Tag Format
|
|
|
|
```
|
|
module-<name>-v<semver>
|
|
|
|
Examples:
|
|
module-authority-v1.2.3
|
|
module-scanner-v2.0.0
|
|
module-cli-v3.1.0-beta.1
|
|
```
|
|
|
|
### Available Modules
|
|
|
|
| Module Name | NuGet | Container | CLI |
|
|
|-------------|-------|-----------|-----|
|
|
| `authority` | Yes | Yes | No |
|
|
| `scanner` | Yes | Yes | No |
|
|
| `concelier` | Yes | Yes | No |
|
|
| `excititor` | Yes | Yes | No |
|
|
| `sbomservice` | Yes | Yes | No |
|
|
| `evidencelocker` | Yes | Yes | No |
|
|
| `policy` | Yes | Yes | No |
|
|
| `attestor` | Yes | Yes | No |
|
|
| `vexlens` | Yes | Yes | No |
|
|
| `cli` | No | No | Yes (multi-platform) |
|
|
|
|
---
|
|
|
|
## Bundle Release Pipeline
|
|
|
|
### Trigger
|
|
|
|
```bash
|
|
# Tag-based trigger
|
|
git tag v2025.12.1
|
|
git push origin v2025.12.1
|
|
|
|
# Channel-specific tags
|
|
git tag v2025.12.0-edge
|
|
git tag v2025.12.0-stable
|
|
git tag v2025.12.0-lts
|
|
```
|
|
|
|
### Workflow: `release.yml`
|
|
|
|
Creates deterministic release bundles with:
|
|
- Signed container images
|
|
- SBOM generation
|
|
- Provenance attestations
|
|
- CLI parity verification
|
|
|
|
---
|
|
|
|
## Rollback Pipeline
|
|
|
|
### Trigger
|
|
|
|
```bash
|
|
# Manual trigger only via Gitea Actions UI
|
|
# Workflow: rollback.yml
|
|
# Inputs: environment, service, version, reason
|
|
```
|
|
|
|
### Workflow: `rollback.yml`
|
|
|
|
```
|
|
┌─────────────────────────────────────────────────────────────────┐
|
|
│ ROLLBACK PIPELINE │
|
|
│ (SLA Target: < 5 min) │
|
|
│ │
|
|
│ ┌──────────────┐ │
|
|
│ │ validate │ Verify inputs and permissions │
|
|
│ └──────┬───────┘ │
|
|
│ │ │
|
|
│ ▼ │
|
|
│ ┌──────────────┐ │
|
|
│ │ fetch-prev │ Download previous version artifacts │
|
|
│ │ version │ │
|
|
│ └──────┬───────┘ │
|
|
│ │ │
|
|
│ ▼ │
|
|
│ ┌──────────────┐ │
|
|
│ │ execute │ Run rollback via Helm/kubectl │
|
|
│ │ rollback │ │
|
|
│ └──────┬───────┘ │
|
|
│ │ │
|
|
│ ▼ │
|
|
│ ┌──────────────┐ │
|
|
│ │health-check │ Verify service health post-rollback │
|
|
│ └──────┬───────┘ │
|
|
│ │ │
|
|
│ ▼ │
|
|
│ ┌──────────────┐ │
|
|
│ │ notify │ Send notification (Slack/Teams/Webhook) │
|
|
│ └──────────────┘ │
|
|
│ │
|
|
└─────────────────────────────────────────────────────────────────┘
|
|
```
|
|
|
|
### Rollback Parameters
|
|
|
|
| Parameter | Type | Description |
|
|
|-----------|------|-------------|
|
|
| `environment` | choice | `staging`, `production` |
|
|
| `service` | choice | Service to rollback (or `all`) |
|
|
| `version` | string | Target version to rollback to |
|
|
| `reason` | string | Reason for rollback (audit log) |
|
|
| `dry_run` | boolean | Simulate without executing |
|
|
|
|
---
|
|
|
|
## Promotion Pipeline
|
|
|
|
### Trigger
|
|
|
|
```bash
|
|
# Manual trigger only via Gitea Actions UI
|
|
# Workflow: promote.yml
|
|
# Inputs: from_environment, to_environment, version
|
|
```
|
|
|
|
### Promotion Flow
|
|
|
|
```
|
|
┌─────────────┐
|
|
│ Edge │ (Automatic on main merge)
|
|
└──────┬──────┘
|
|
│
|
|
│ promote.yml (manual)
|
|
▼
|
|
┌─────────────┐
|
|
│ Stable │ (After testing period)
|
|
└──────┬──────┘
|
|
│
|
|
│ promote.yml (manual)
|
|
▼
|
|
┌─────────────┐
|
|
│ LTS │ (After extended validation)
|
|
└─────────────┘
|
|
```
|
|
|
|
### Promotion Checklist (Automated)
|
|
|
|
1. **Pre-Flight Checks**
|
|
- All tests passing in source environment
|
|
- No critical vulnerabilities
|
|
- Performance SLOs met
|
|
- Documentation complete
|
|
|
|
2. **Promotion Steps**
|
|
- Re-tag containers with new channel
|
|
- Update Helm chart values
|
|
- Deploy to target environment
|
|
- Run smoke tests
|
|
|
|
3. **Post-Promotion**
|
|
- Health check verification
|
|
- Update release documentation
|
|
- Notify stakeholders
|
|
|
|
---
|
|
|
|
## Artifact Signing
|
|
|
|
### Cosign Integration
|
|
|
|
All release artifacts are signed using Cosign with Sigstore keyless signing:
|
|
|
|
```bash
|
|
# Verify container signature
|
|
cosign verify \
|
|
--certificate-identity-regexp=".*github.com/stellaops.*" \
|
|
--certificate-oidc-issuer="https://token.actions.githubusercontent.com" \
|
|
ghcr.io/stellaops/scanner:2026.04
|
|
|
|
# Verify SBOM
|
|
cosign verify-attestation \
|
|
--type spdxjson \
|
|
--certificate-identity-regexp=".*github.com/stellaops.*" \
|
|
ghcr.io/stellaops/scanner:2026.04
|
|
```
|
|
|
|
### Signature Artifacts
|
|
|
|
| Artifact Type | Signature Location |
|
|
|---------------|-------------------|
|
|
| Container Image | OCI registry (same repo) |
|
|
| CLI Binary | `.sig` file alongside binary |
|
|
| SBOM | Attestation on OCI image |
|
|
| Provenance | Attestation on OCI image |
|
|
|
|
---
|
|
|
|
## Release Artifacts
|
|
|
|
### Per-Release Artifacts
|
|
|
|
| Artifact | Format | Location |
|
|
|----------|--------|----------|
|
|
| Release Notes | Markdown | Gitea Release |
|
|
| Changelog | `CHANGELOG.md` | Gitea Release, `docs/releases/` |
|
|
| Binary Checksums | `SHA256SUMS.txt` | Gitea Release |
|
|
| SBOM (CycloneDX) | JSON | Gitea Release, OCI attestation |
|
|
| SBOM (SPDX) | JSON | Gitea Release |
|
|
| Provenance | in-toto/DSSE | OCI attestation |
|
|
| Docker Compose | YAML | `devops/compose/` |
|
|
| Helm Chart | TGZ | OCI registry |
|
|
|
|
### Artifact Retention
|
|
|
|
| Environment | Retention Period |
|
|
|-------------|------------------|
|
|
| PR/Preview | 7 days |
|
|
| Edge | 30 days |
|
|
| Stable | 1 year |
|
|
| LTS | 3 years |
|
|
|
|
---
|
|
|
|
## Creating a Release
|
|
|
|
### Suite Release
|
|
|
|
```bash
|
|
# 1. Ensure main is stable
|
|
git checkout main
|
|
git pull
|
|
|
|
# 2. Create and push tag
|
|
git tag suite-2026.04
|
|
git push origin suite-2026.04
|
|
|
|
# 3. Monitor release pipeline
|
|
# Gitea Actions → release-suite.yml
|
|
|
|
# 4. Verify artifacts
|
|
# - Check Gitea Releases page
|
|
# - Verify container images pushed
|
|
# - Validate SBOM and signatures
|
|
```
|
|
|
|
### Module Release
|
|
|
|
```bash
|
|
# 1. Update module version
|
|
# Edit src/<Module>/version.txt or .csproj
|
|
|
|
# 2. Create and push tag
|
|
git tag module-authority-v1.2.3
|
|
git push origin module-authority-v1.2.3
|
|
|
|
# 3. Monitor release pipeline
|
|
# Gitea Actions → module-publish.yml
|
|
```
|
|
|
|
### Hotfix Release
|
|
|
|
```bash
|
|
# 1. Create hotfix branch from release tag
|
|
git checkout -b hotfix/v2025.12.1 v2025.12.0
|
|
|
|
# 2. Apply fix
|
|
# ... make changes ...
|
|
git commit -m "Fix: critical security issue"
|
|
|
|
# 3. Create hotfix tag
|
|
git tag v2025.12.1
|
|
git push origin hotfix/v2025.12.1 v2025.12.1
|
|
|
|
# 4. Fast-track through pipeline
|
|
# Workflow will run with reduced test scope
|
|
```
|
|
|
|
---
|
|
|
|
## Troubleshooting Releases
|
|
|
|
### Release Pipeline Failed
|
|
|
|
1. **Check build logs** - Gitea Actions → failed job
|
|
2. **Verify tag format** - Must match expected pattern
|
|
3. **Check secrets** - Registry credentials, signing keys
|
|
4. **Review test failures** - May need to skip with `skip_tests=true`
|
|
|
|
### Container Not Published
|
|
|
|
1. **Check registry authentication** - `REGISTRY_TOKEN` secret
|
|
2. **Verify image name** - Check for typos in workflow
|
|
3. **Check rate limits** - May need to wait and retry
|
|
4. **Review scan results** - Image may be blocked by vulnerability scan
|
|
|
|
### Signature Verification Failed
|
|
|
|
1. **Check Sigstore availability** - May have temporary outage
|
|
2. **Verify certificate identity** - Workflow must match expected pattern
|
|
3. **Check OIDC issuer** - Must be GitHub/Gitea Actions
|
|
|
|
### Rollback Failed
|
|
|
|
1. **Verify target version exists** - Check artifact storage
|
|
2. **Check Helm/kubectl access** - Cluster credentials
|
|
3. **Review health check** - Service may need manual intervention
|
|
4. **Check resource constraints** - May need to scale down first
|
|
|
|
---
|
|
|
|
## Related Documentation
|
|
|
|
- [README - CI/CD Overview](./README.md)
|
|
- [Workflow Triggers](./workflow-triggers.md)
|
|
- [Versioning Guide](../releases/VERSIONING.md)
|
|
- [Container Registry Guide](../operations/container-registry.md)
|
|
- [Helm Deployment Guide](../operations/helm-deployment.md)
|