git.stella-ops.org/docs/releases/RELEASE_PROCESS.md

StellaOps Release Process

This document describes the release process for StellaOps suite and module releases.

Overview

StellaOps uses automated CI/CD pipelines for releases:

Release Type	Workflow	Trigger
Module	.gitea/workflows/module-publish.yml	Tag or manual dispatch
Suite	.gitea/workflows/release-suite.yml	Tag or manual dispatch

---

Module Release Process

Prerequisites

• All tests passing on main branch
• CHANGELOG.md updated with changes
• Version bumped in module's version.txt (if applicable)
• Breaking changes documented

Steps

Option A: Tag-based Release

# Create and push tag
git tag module-authority-v1.2.3
git push origin module-authority-v1.2.3

The pipeline will automatically:

1. Parse module name and version from tag
2. Build the module
3. Publish NuGet package to Gitea registry
4. Build and push container image (if applicable)

Option B: Manual Dispatch

1. Navigate to Actions > Module Publish
2. Click Run workflow
3. Select:
   • Module: e.g., Authority
   • Version: e.g., 1.2.3
   • Publish NuGet: true
   • Publish Container: true
4. Click Run

Artifacts Published

Artifact	Location
NuGet	git.stella-ops.org/api/packages/stella-ops.org/nuget/index.json
Container	git.stella-ops.org/stella-ops.org/{module}:{version}

---

Suite Release Process

Prerequisites

• All module versions finalized
• Integration tests passing
• Security scan completed
• CHANGELOG.md updated
• Compatibility matrix documented
• Codename selected (see codenames.md)

Pre-Release Checklist

- [ ] All P1 issues resolved
- [ ] Performance benchmarks meet SLOs
- [ ] Documentation updated
- [ ] Migration guide prepared
- [ ] Release notes drafted
- [ ] Security advisory review complete
- [ ] Air-gap bundle tested
- [ ] Helm chart validated

Steps

Option A: Tag-based Release

# Create and push tag
git tag suite-2026.04-nova
git push origin suite-2026.04-nova

Option B: Manual Dispatch

1. Navigate to Actions > Suite Release
2. Click Run workflow
3. Fill in:
   • Version: e.g., 2026.04
   • Codename: e.g., Nova
   • Channel: edge, stable, or lts
   • Skip tests: false (default)
   • Dry run: false for actual release
4. Click Run

Pipeline Stages

validate → test-gate → build-modules → build-containers
                    ↘              ↓
                     build-cli → build-helm → release-manifest → create-release → summary

1. Validate - Check version format, resolve inputs
2. Test Gate - Run unit, architecture, and contract tests
3. Build Modules - Build all 9 modules (matrix)
4. Build Containers - Push container images (9 modules)
5. Build CLI - Build for 5 platforms
6. Build Helm - Package Helm chart
7. Release Manifest - Generate suite-{version}.yaml
8. Create Release - Create Gitea release with artifacts
9. Summary - Generate summary report

Artifacts Published

Artifact	Files
Container images	9 modules × 3 tags (version, channel, latest)
CLI binaries	5 platforms (linux-x64, linux-arm64, win-x64, osx-x64, osx-arm64)
Helm chart	stellaops-{version}.tgz
Release manifest	suite-{version}.yaml
Checksums	SHA256SUMS-{version}.txt

---

Release Channels

Edge

• Pre-release builds
• May contain experimental features
• Not recommended for production
• Triggered by: channel: edge or tag without -stable/-lts

Stable

• Production-ready releases
• Thoroughly tested
• 9 months of support (feature releases)
• Triggered by: channel: stable

LTS (Long Term Support)

• April releases only (XX.04)
• 5 years of security updates
• 3 years of standard support
• Triggered by: channel: lts

---

Rollback Procedures

Container Rollback

# Pull previous version
docker pull git.stella-ops.org/stella-ops.org/authority:2025.10

# Update deployment
kubectl set image deployment/authority authority=git.stella-ops.org/stella-ops.org/authority:2025.10

Helm Rollback

# List releases
helm history stellaops

# Rollback to previous revision
helm rollback stellaops 1

Database Rollback

1. Stop all services
2. Restore database from backup
3. Deploy previous version
4. Verify data integrity

Important: Always test rollback procedures in staging before production.

---

Hotfix Process

For critical security fixes:

1. Create hotfix branch from release tag

   git checkout -b hotfix/2026.04.1 suite-2026.04
   

2. Apply fix and test

3. Tag hotfix release

   git tag suite-2026.04.1
   git push origin suite-2026.04.1
   

4. Cherry-pick fix to main branch

---

Release Evidence Pack

Every release includes a Release Evidence Pack for customer verification and compliance.

Evidence Pack Contents

Component	Description
Artifacts	Release binaries and container references
Checksums	SHA-256 and SHA-512 checksum files
Signatures	Cosign signatures for all artifacts
SBOMs	CycloneDX Software Bill of Materials
Provenance	SLSA v1.0 provenance statements
Rekor Proofs	Transparency log inclusion proofs
Verification Scripts	verify.sh and verify.ps1

Generation Workflow

The evidence pack is generated by .gitea/workflows/release-evidence-pack.yml:

1. Verify Test Gates - Ensures all test workflows passed
2. Generate Checksums - Create SHA256SUMS and SHA512SUMS
3. Sign Artifacts - Sign with cosign (keyless or key-based)
4. Generate SBOMs - Create CycloneDX SBOMs per artifact
5. Generate Provenance - Create SLSA v1.0 statements
6. Collect Rekor Proofs - Fetch inclusion proofs from Rekor
7. Build Pack - Assemble final evidence pack bundle
8. Self-Verify - Run verify.sh to validate the pack

Manual Trigger

# Trigger evidence pack generation for a release
gh workflow run release-evidence-pack.yml \
  -f version=2.5.0 \
  -f release_tag=v2.5.0

Verification

Customers can verify releases offline:

tar -xzf stella-release-2.5.0-evidence-pack.tgz
cd stella-release-2.5.0-evidence-pack
./verify.sh --verbose

See Release Evidence Pack for detailed documentation.

---

Reproducible Builds

All release builds are reproducible using SOURCE_DATE_EPOCH:

# Set from git commit timestamp
export SOURCE_DATE_EPOCH=$(git show -s --format=%ct HEAD)

# Build with deterministic settings
dotnet build -c Release /p:Deterministic=true /p:ContinuousIntegrationBuild=true

The CI verifies reproducibility by building twice and comparing checksums.

See Reproducible Builds for details.

---

Post-Release Tasks

• Verify artifacts in registry
• Generate and publish Release Evidence Pack
• Verify evidence pack passes self-verification
• Update documentation site
• Send release announcement
• Update compatibility matrix
• Monitor for issues (24-48 hours)
• Update roadmap

---

Troubleshooting

Build Failures

1. Check test results in artifacts
2. Review workflow logs
3. Verify secrets are configured (GITEA_TOKEN)

Push Failures

1. Verify registry authentication
2. Check network connectivity
3. Ensure no conflicting tags exist

Common Issues

Issue	Resolution
Tag already exists	Delete tag and recreate, or use next version
NuGet push fails	Check package already exists, use --skip-duplicate
Container push fails	Verify registry login, check image size limits

---

Related Documentation

• Versioning Strategy
• Codename Registry
• CI/CD Workflows