Fix build and code structure improvements. New but essential UI functionality. CI improvements. Documentation improvements. AI module improvements.
This commit is contained in:
StellaOps Bot
329
docs/cicd/README.md
Normal file
329
docs/cicd/README.md
Normal file
@@ -0,0 +1,329 @@
|
||||
# CI/CD Infrastructure Overview
|
||||
|
||||
> **Sprint:** CI/CD Enhancement - Documentation
|
||||
> **Last Updated:** 2025-12-28
|
||||
> **Workflow Count:** 100 workflows
|
||||
|
||||
## Quick Links
|
||||
|
||||
- [Workflow Triggers & Dependencies](./workflow-triggers.md)
|
||||
- [Release Pipelines](./release-pipelines.md)
|
||||
- [Security Scanning](./security-scanning.md)
|
||||
- [Test Strategy](./test-strategy.md)
|
||||
- [Troubleshooting Guide](../.gitea/docs/troubleshooting.md)
|
||||
|
||||
---
|
||||
|
||||
## Architecture Overview
|
||||
|
||||
The StellaOps CI/CD infrastructure uses **Gitea Actions** (GitHub Actions compatible) with a sophisticated multi-tier triggering strategy designed for:
|
||||
|
||||
- **Determinism & Reproducibility** - Identical builds across runs
|
||||
- **Offline-First Operation** - Air-gap compatible pipelines
|
||||
- **Supply Chain Security** - SLSA Level 2-3 compliance
|
||||
- **Developer Velocity** - Fast PR feedback with comprehensive nightly testing
|
||||
|
||||
### Pipeline Tiers
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────────────────┐
|
||||
│ TRIGGER HIERARCHY │
|
||||
├─────────────────────────────────────────────────────────────────────────┤
|
||||
│ │
|
||||
│ TIER 1: PR GATING (Every Pull Request) │
|
||||
│ ├── test-matrix.yml (Unit, Architecture, Contract, Integration, │
|
||||
│ │ Security, Golden) │
|
||||
│ ├── build-test-deploy.yml (Build verification) │
|
||||
│ ├── policy-lint.yml (Policy file validation) │
|
||||
│ ├── sast-scan.yml (Static security analysis) │
|
||||
│ └── docs.yml (Documentation validation) │
|
||||
│ │
|
||||
│ TIER 2: MAIN BRANCH (Post-Merge) │
|
||||
│ ├── All Tier 1 workflows │
|
||||
│ ├── build-test-deploy.yml → Deploy stage (staging environment) │
|
||||
│ ├── integration-tests-gate.yml → Extended coverage │
|
||||
│ └── coverage-report (Full coverage analysis) │
|
||||
│ │
|
||||
│ TIER 3: SCHEDULED (Nightly/Weekly) │
|
||||
│ ├── nightly-regression.yml (2:00 AM UTC daily) │
|
||||
│ ├── test-matrix.yml → Extended tests (5:00 AM UTC daily) │
|
||||
│ ├── dependency-security-scan.yml (2:00 AM UTC Sunday) │
|
||||
│ ├── renovate.yml (3:00 AM & 3:00 PM UTC daily) │
|
||||
│ ├── sast-scan.yml (3:30 AM UTC Monday) │
|
||||
│ └── migration-test.yml (4:30 AM UTC daily) │
|
||||
│ │
|
||||
│ TIER 4: RELEASE (Tag-Triggered) │
|
||||
│ ├── release-suite.yml (suite-YYYY.MM tags) │
|
||||
│ ├── release.yml (v* tags) │
|
||||
│ └── module-publish.yml (module-*-v* tags) │
|
||||
│ │
|
||||
│ TIER 5: MANUAL (On-Demand) │
|
||||
│ ├── cli-build.yml, scanner-determinism.yml │
|
||||
│ ├── rollback.yml, promote.yml │
|
||||
│ └── 20+ specialized test/debug workflows │
|
||||
│ │
|
||||
└─────────────────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Workflow Categories
|
||||
|
||||
### 1. Core Build & Test (12 workflows)
|
||||
|
||||
| Workflow | Purpose | Triggers |
|
||||
|----------|---------|----------|
|
||||
| `build-test-deploy.yml` | Main build pipeline | PR, main push, daily, manual |
|
||||
| `test-matrix.yml` | Unified test execution | PR, main push, daily, manual |
|
||||
| `integration-tests-gate.yml` | Extended integration testing | PR, main push, manual |
|
||||
| `nightly-regression.yml` | Comprehensive nightly suite | Daily 2 AM UTC |
|
||||
| `migration-test.yml` | Database migration validation | PR (migrations), daily |
|
||||
|
||||
### 2. Release Automation (8 workflows)
|
||||
|
||||
| Workflow | Purpose | Triggers |
|
||||
|----------|---------|----------|
|
||||
| `release-suite.yml` | Ubuntu-style suite releases | `suite-*` tags, manual |
|
||||
| `release.yml` | Version bundle releases | `v*` tags, manual |
|
||||
| `module-publish.yml` | Per-module publishing | `module-*-v*` tags, manual |
|
||||
| `cli-build.yml` | Multi-platform CLI builds | Manual only |
|
||||
| `promote.yml` | Environment promotion | Manual only |
|
||||
| `rollback.yml` | Emergency rollback | Manual only |
|
||||
|
||||
### 3. Security Scanning (6 workflows)
|
||||
|
||||
| Workflow | Purpose | Triggers |
|
||||
|----------|---------|----------|
|
||||
| `sast-scan.yml` | Static code analysis | PR, main push, weekly |
|
||||
| `secrets-scan.yml` | Credential detection | PR, main push |
|
||||
| `container-scan.yml` | Image vulnerability scanning | Dockerfile changes, daily |
|
||||
| `dependency-security-scan.yml` | NuGet/npm vulnerability audit | Weekly, PR (deps) |
|
||||
| `dependency-license-gate.yml` | License compliance | PR (deps) |
|
||||
|
||||
### 4. Quality Assurance (15 workflows)
|
||||
|
||||
| Workflow | Purpose | Triggers |
|
||||
|----------|---------|----------|
|
||||
| `policy-lint.yml` | Policy file validation | PR, main push |
|
||||
| `docs.yml` | Documentation linting | docs/** changes |
|
||||
| `scanner-determinism.yml` | Output reproducibility | Manual only |
|
||||
| `determinism-gate.yml` | Build determinism | Manual only |
|
||||
| `cross-platform-determinism.yml` | Multi-OS verification | Manual only |
|
||||
|
||||
### 5. Module-Specific (30+ workflows)
|
||||
|
||||
Specialized workflows for individual modules (Scanner, Concelier, Authority, etc.)
|
||||
|
||||
---
|
||||
|
||||
## Trigger Quick Reference
|
||||
|
||||
### Branch Patterns
|
||||
|
||||
| Pattern | Example | Workflows Triggered |
|
||||
|---------|---------|---------------------|
|
||||
| Push to `main` | Direct commit or merge | All Tier 1 + Tier 2 |
|
||||
| Push to `develop` | Feature integration | Selected gating workflows |
|
||||
| Pull Request | Any PR to main/develop | All Tier 1 (gating) |
|
||||
| Push to `feature/*` | Feature branches | None (PR required) |
|
||||
| Push to `release/*` | Release prep branches | Selected validation |
|
||||
|
||||
### Tag Patterns
|
||||
|
||||
| Pattern | Example | Workflow |
|
||||
|---------|---------|----------|
|
||||
| `v*` | `v2025.12.1` | `release.yml` |
|
||||
| `suite-*` | `suite-2026.04` | `release-suite.yml` |
|
||||
| `module-*-v*` | `module-authority-v1.2.3` | `module-publish.yml` |
|
||||
|
||||
### Schedule Summary
|
||||
|
||||
| Time (UTC) | Frequency | Workflow |
|
||||
|------------|-----------|----------|
|
||||
| 2:00 AM | Daily | `nightly-regression.yml` |
|
||||
| 2:00 AM | Sunday | `dependency-security-scan.yml` |
|
||||
| 3:00 AM | Daily | `renovate.yml` |
|
||||
| 3:30 AM | Monday | `sast-scan.yml` |
|
||||
| 4:30 AM | Daily | `migration-test.yml` |
|
||||
| 5:00 AM | Daily | `build-test-deploy.yml`, `test-matrix.yml` |
|
||||
| 3:00 PM | Daily | `renovate.yml` |
|
||||
|
||||
---
|
||||
|
||||
## Environment Flow
|
||||
|
||||
```
|
||||
┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐
|
||||
│ PR │───▶│ Staging │───▶│ Stable │───▶│ LTS │
|
||||
│ (Preview)│ │ (Edge) │ │ (Tested) │ │(Long-Term)│
|
||||
└──────────┘ └──────────┘ └──────────┘ └──────────┘
|
||||
│ │ │ │
|
||||
│ │ │ │
|
||||
▼ ▼ ▼ ▼
|
||||
PR tests Auto-deploy promote.yml promote.yml
|
||||
(gating) on main merge (manual) (manual)
|
||||
```
|
||||
|
||||
### Environment Matrix
|
||||
|
||||
| Environment | Branch/Tag | Auto-Deploy | Rollback |
|
||||
|-------------|------------|-------------|----------|
|
||||
| Preview | PR | Yes (ephemeral) | N/A |
|
||||
| Staging (Edge) | `main` | Yes | `rollback.yml` |
|
||||
| Stable | `v*` tags | Manual | `rollback.yml` |
|
||||
| LTS | `suite-*` tags | Manual | `rollback.yml` |
|
||||
|
||||
---
|
||||
|
||||
## Key Features
|
||||
|
||||
### 1. PR-Gating Tests
|
||||
|
||||
Required tests that must pass before merge:
|
||||
|
||||
- **Unit Tests** - Fast, isolated tests
|
||||
- **Architecture Tests** - Dependency rule enforcement
|
||||
- **Contract Tests** - API compatibility
|
||||
- **Integration Tests** - PostgreSQL integration
|
||||
- **Security Tests** - Security-focused assertions
|
||||
- **Golden Tests** - Corpus-based validation
|
||||
|
||||
### 2. Determinism Verification
|
||||
|
||||
All builds produce identical outputs:
|
||||
|
||||
- Binary checksums compared across runs
|
||||
- UTC timezone enforcement (`TZ: UTC`)
|
||||
- Stable JSON serialization
|
||||
- Reproducible SBOM generation
|
||||
|
||||
### 3. Supply Chain Security
|
||||
|
||||
- **SBOM Generation** - Syft for CycloneDX/SPDX
|
||||
- **Artifact Signing** - Cosign/Sigstore integration
|
||||
- **Provenance** - in-toto/DSSE attestations
|
||||
- **Dependency Scanning** - Automated vulnerability detection
|
||||
|
||||
### 4. Rollback Automation
|
||||
|
||||
Emergency rollback via `rollback.yml`:
|
||||
- Target: < 5 minute SLA
|
||||
- Helm-based deployment rollback
|
||||
- Health check verification
|
||||
- Notification integration
|
||||
|
||||
---
|
||||
|
||||
## Directory Structure
|
||||
|
||||
```
|
||||
.gitea/
|
||||
├── workflows/ # 100 workflow files
|
||||
│ ├── build-test-deploy.yml
|
||||
│ ├── test-matrix.yml
|
||||
│ ├── release-suite.yml
|
||||
│ └── ...
|
||||
├── scripts/ # CI/CD scripts
|
||||
│ ├── build/ # Build orchestration
|
||||
│ ├── test/ # Test execution
|
||||
│ ├── release/ # Release automation
|
||||
│ ├── sign/ # Signing operations
|
||||
│ └── validate/ # Validation scripts
|
||||
└── docs/ # CI-specific docs
|
||||
├── architecture.md
|
||||
├── scripts.md
|
||||
└── troubleshooting.md
|
||||
|
||||
devops/
|
||||
├── scripts/
|
||||
│ └── lib/ # Shared bash libraries
|
||||
│ ├── logging.sh
|
||||
│ ├── exit-codes.sh
|
||||
│ ├── git-utils.sh
|
||||
│ ├── path-utils.sh
|
||||
│ └── hash-utils.sh
|
||||
├── compose/ # Docker Compose profiles
|
||||
├── helm/ # Helm charts
|
||||
└── docker/ # Dockerfiles
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Getting Started
|
||||
|
||||
### Running Workflows Locally
|
||||
|
||||
```bash
|
||||
# Run test matrix locally
|
||||
./devops/scripts/test-local.sh
|
||||
|
||||
# Validate compose files
|
||||
./devops/scripts/validate-compose.sh
|
||||
|
||||
# Run a specific test category
|
||||
./.gitea/scripts/test/run-test-category.sh Unit
|
||||
```
|
||||
|
||||
### Triggering Manual Workflows
|
||||
|
||||
```bash
|
||||
# Via Gitea UI: Actions → Workflow → Run workflow
|
||||
|
||||
# Or via API:
|
||||
curl -X POST \
|
||||
-H "Authorization: token $GITEA_TOKEN" \
|
||||
"$GITEA_URL/api/v1/repos/owner/repo/actions/workflows/rollback.yml/dispatches" \
|
||||
-d '{"ref":"main","inputs":{"environment":"staging","version":"v2025.12.0"}}'
|
||||
```
|
||||
|
||||
### Creating a Release
|
||||
|
||||
1. **Module Release:**
|
||||
```bash
|
||||
git tag module-authority-v1.2.3
|
||||
git push origin module-authority-v1.2.3
|
||||
```
|
||||
|
||||
2. **Suite Release:**
|
||||
```bash
|
||||
git tag suite-2026.04
|
||||
git push origin suite-2026.04
|
||||
```
|
||||
|
||||
3. **Bundle Release:**
|
||||
```bash
|
||||
git tag v2025.12.1
|
||||
git push origin v2025.12.1
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Related Documentation
|
||||
|
||||
- [Workflow Triggers Deep Dive](./workflow-triggers.md)
|
||||
- [Release Pipeline Details](./release-pipelines.md)
|
||||
- [Security Scanning Guide](./security-scanning.md)
|
||||
- [Test Strategy](./test-strategy.md)
|
||||
- [CI Quality Gates](../testing/ci-quality-gates.md)
|
||||
- [Troubleshooting](../.gitea/docs/troubleshooting.md)
|
||||
- [Script Reference](../.gitea/docs/scripts.md)
|
||||
|
||||
---
|
||||
|
||||
## Metrics & Monitoring
|
||||
|
||||
### Key Metrics Tracked
|
||||
|
||||
| Metric | Target | Measurement |
|
||||
|--------|--------|-------------|
|
||||
| PR Build Time | < 15 min | Workflow duration |
|
||||
| Main Build Time | < 20 min | Workflow duration |
|
||||
| Test Flakiness | < 1% | Flaky test detection |
|
||||
| Security Scan Coverage | 100% | SAST/DAST coverage |
|
||||
| Rollback SLA | < 5 min | Rollback workflow duration |
|
||||
|
||||
### Dashboard Links
|
||||
|
||||
- [Workflow Runs](../../.gitea/workflows/) (Gitea Actions UI)
|
||||
- [Test Results](./test-results/) (TRX/JUnit artifacts)
|
||||
- [Coverage Reports](./coverage/) (Generated nightly)
|
||||
414
docs/cicd/path-filters.md
Normal file
414
docs/cicd/path-filters.md
Normal file
@@ -0,0 +1,414 @@
|
||||
# Path Filters Reference
|
||||
|
||||
> Complete reference for path filter patterns used in CI/CD workflows.
|
||||
|
||||
---
|
||||
|
||||
## Overview
|
||||
|
||||
Path filters determine which workflows run based on changed files. This ensures:
|
||||
- **Efficiency**: Only relevant tests run for each change
|
||||
- **Speed**: Module-specific changes don't trigger full builds
|
||||
- **Cascading**: Shared library changes trigger dependent module tests
|
||||
|
||||
---
|
||||
|
||||
## Configuration Location
|
||||
|
||||
Centralized path filter definitions are maintained in:
|
||||
|
||||
```
|
||||
.gitea/config/path-filters.yml
|
||||
```
|
||||
|
||||
This file serves as the source of truth for all path filter patterns.
|
||||
|
||||
---
|
||||
|
||||
## Path Filter Categories
|
||||
|
||||
### 1. Infrastructure Files (Trigger FULL CI)
|
||||
|
||||
Changes to these files trigger all tests and full build validation:
|
||||
|
||||
```yaml
|
||||
infrastructure:
|
||||
- 'Directory.Build.props' # Root MSBuild properties
|
||||
- 'Directory.Build.rsp' # MSBuild response file
|
||||
- 'Directory.Packages.props' # Central package versions
|
||||
- 'src/Directory.Build.props' # Source directory properties
|
||||
- 'src/Directory.Packages.props'
|
||||
- 'nuget.config' # NuGet feed configuration
|
||||
- 'StellaOps.sln' # Solution file
|
||||
- '.gitea/workflows/**' # CI/CD workflow changes
|
||||
```
|
||||
|
||||
**When to use:** All PR-gating and integration workflows should include these paths.
|
||||
|
||||
### 2. Documentation Paths (Skip CI)
|
||||
|
||||
These paths should use `paths-ignore` to skip builds:
|
||||
|
||||
```yaml
|
||||
docs_ignore:
|
||||
- 'docs/**' # All documentation
|
||||
- '*.md' # Root markdown files
|
||||
- 'etc/**' # Configuration samples
|
||||
- 'LICENSE' # License file
|
||||
- '.gitignore' # Git ignore
|
||||
- '.editorconfig' # Editor configuration
|
||||
```
|
||||
|
||||
**Exceptions:** These markdown files SHOULD trigger CI:
|
||||
- `CLAUDE.md` - Agent instructions (affects behavior)
|
||||
- `AGENTS.md` - Module-specific guidance
|
||||
|
||||
### 3. Shared Library Paths (Trigger Cascading)
|
||||
|
||||
Changes to shared libraries trigger tests in dependent modules:
|
||||
|
||||
#### Cryptography (CRITICAL - affects security)
|
||||
|
||||
```yaml
|
||||
cryptography:
|
||||
paths:
|
||||
- 'src/__Libraries/StellaOps.Cryptography*/**'
|
||||
- 'src/Cryptography/**'
|
||||
cascades_to:
|
||||
- Scanner tests
|
||||
- Attestor tests
|
||||
- Authority tests
|
||||
- EvidenceLocker tests
|
||||
- Signer tests
|
||||
- AirGap tests
|
||||
- Security test suite
|
||||
- Offline E2E tests
|
||||
```
|
||||
|
||||
#### Evidence & Provenance
|
||||
|
||||
```yaml
|
||||
evidence:
|
||||
paths:
|
||||
- 'src/__Libraries/StellaOps.Evidence*/**'
|
||||
- 'src/__Libraries/StellaOps.Provenance/**'
|
||||
cascades_to:
|
||||
- Scanner tests
|
||||
- Attestor tests
|
||||
- EvidenceLocker tests
|
||||
- ExportCenter tests
|
||||
- SbomService tests
|
||||
```
|
||||
|
||||
#### Infrastructure & Database
|
||||
|
||||
```yaml
|
||||
infrastructure:
|
||||
paths:
|
||||
- 'src/__Libraries/StellaOps.Infrastructure*/**'
|
||||
- 'src/__Libraries/StellaOps.DependencyInjection/**'
|
||||
cascades_to:
|
||||
- ALL integration tests
|
||||
```
|
||||
|
||||
#### Replay & Determinism
|
||||
|
||||
```yaml
|
||||
replay:
|
||||
paths:
|
||||
- 'src/__Libraries/StellaOps.Replay*/**'
|
||||
- 'src/__Libraries/StellaOps.Testing.Determinism/**'
|
||||
cascades_to:
|
||||
- Scanner determinism tests
|
||||
- Determinism gate
|
||||
- Replay module tests
|
||||
```
|
||||
|
||||
#### Verdict & Policy Primitives
|
||||
|
||||
```yaml
|
||||
verdict:
|
||||
paths:
|
||||
- 'src/__Libraries/StellaOps.Verdict/**'
|
||||
- 'src/__Libraries/StellaOps.DeltaVerdict/**'
|
||||
cascades_to:
|
||||
- Policy engine tests
|
||||
- RiskEngine tests
|
||||
- ReachGraph tests
|
||||
```
|
||||
|
||||
#### Plugin Framework
|
||||
|
||||
```yaml
|
||||
plugin:
|
||||
paths:
|
||||
- 'src/__Libraries/StellaOps.Plugin/**'
|
||||
cascades_to:
|
||||
- Authority tests (plugin-based auth)
|
||||
- Scanner tests (analyzer plugins)
|
||||
- Concelier tests (connector plugins)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Module-Specific Paths
|
||||
|
||||
Each module has defined source and test paths:
|
||||
|
||||
### Core Platform
|
||||
|
||||
| Module | Source Paths | Test Paths |
|
||||
|--------|--------------|------------|
|
||||
| Authority | `src/Authority/**` | `src/Authority/__Tests/**` |
|
||||
| Gateway | `src/Gateway/**` | `src/Gateway/__Tests/**` |
|
||||
| Router | `src/Router/**` | `src/Router/__Tests/**` |
|
||||
|
||||
### Scanning & Analysis
|
||||
|
||||
| Module | Source Paths | Test Paths |
|
||||
|--------|--------------|------------|
|
||||
| Scanner | `src/Scanner/**`, `src/BinaryIndex/**` | `src/Scanner/__Tests/**`, `src/BinaryIndex/__Tests/**` |
|
||||
| AdvisoryAI | `src/AdvisoryAI/**` | `src/AdvisoryAI/__Tests/**` |
|
||||
| ReachGraph | `src/ReachGraph/**` | `src/ReachGraph/__Tests/**` |
|
||||
|
||||
### Data Ingestion
|
||||
|
||||
| Module | Source Paths | Test Paths |
|
||||
|--------|--------------|------------|
|
||||
| Concelier | `src/Concelier/**` | `src/Concelier/__Tests/**` |
|
||||
| Excititor | `src/Excititor/**` | `src/Excititor/__Tests/**` |
|
||||
| VexLens | `src/VexLens/**` | `src/VexLens/__Tests/**` |
|
||||
| VexHub | `src/VexHub/**` | `src/VexHub/__Tests/**` |
|
||||
|
||||
### Artifacts & Evidence
|
||||
|
||||
| Module | Source Paths | Test Paths |
|
||||
|--------|--------------|------------|
|
||||
| Attestor | `src/Attestor/**` | `src/Attestor/__Tests/**` |
|
||||
| SbomService | `src/SbomService/**` | `src/SbomService/__Tests/**` |
|
||||
| EvidenceLocker | `src/EvidenceLocker/**` | `src/EvidenceLocker/__Tests/**` |
|
||||
| ExportCenter | `src/ExportCenter/**` | `src/ExportCenter/__Tests/**` |
|
||||
| Findings | `src/Findings/**` | `src/Findings/__Tests/**` |
|
||||
|
||||
### Policy & Risk
|
||||
|
||||
| Module | Source Paths | Test Paths |
|
||||
|--------|--------------|------------|
|
||||
| Policy | `src/Policy/**` | `src/Policy/__Tests/**` |
|
||||
| RiskEngine | `src/RiskEngine/**` | `src/RiskEngine/__Tests/**` |
|
||||
|
||||
### Operations
|
||||
|
||||
| Module | Source Paths | Test Paths |
|
||||
|--------|--------------|------------|
|
||||
| Notify | `src/Notify/**`, `src/Notifier/**` | `src/Notify/__Tests/**` |
|
||||
| Orchestrator | `src/Orchestrator/**` | `src/Orchestrator/__Tests/**` |
|
||||
| Scheduler | `src/Scheduler/**` | `src/Scheduler/__Tests/**` |
|
||||
| PacksRegistry | `src/PacksRegistry/**` | `src/PacksRegistry/__Tests/**` |
|
||||
| Replay | `src/Replay/**` | `src/Replay/__Tests/**` |
|
||||
|
||||
### Infrastructure
|
||||
|
||||
| Module | Source Paths | Test Paths |
|
||||
|--------|--------------|------------|
|
||||
| Cryptography | `src/Cryptography/**` | `src/__Libraries/__Tests/StellaOps.Cryptography*/**` |
|
||||
| Telemetry | `src/Telemetry/**` | `src/Telemetry/__Tests/**` |
|
||||
| Signals | `src/Signals/**` | `src/Signals/__Tests/**` |
|
||||
| AirGap | `src/AirGap/**` | `src/AirGap/__Tests/**` |
|
||||
| AOC | `src/Aoc/**` | `src/Aoc/__Tests/**` |
|
||||
|
||||
### Integration
|
||||
|
||||
| Module | Source Paths | Test Paths |
|
||||
|--------|--------------|------------|
|
||||
| CLI | `src/Cli/**` | `src/Cli/__Tests/**` |
|
||||
| Web | `src/Web/**` | `src/Web/**/*.spec.ts` |
|
||||
|
||||
---
|
||||
|
||||
## DevOps & CI/CD Paths
|
||||
|
||||
### Docker & Containers
|
||||
|
||||
```yaml
|
||||
docker:
|
||||
- 'devops/docker/**'
|
||||
- '**/Dockerfile'
|
||||
- '**/Dockerfile.*'
|
||||
```
|
||||
|
||||
### Compose Profiles
|
||||
|
||||
```yaml
|
||||
compose:
|
||||
- 'devops/compose/**'
|
||||
- 'docker-compose*.yml'
|
||||
```
|
||||
|
||||
### Helm Charts
|
||||
|
||||
```yaml
|
||||
helm:
|
||||
- 'devops/helm/**'
|
||||
- 'devops/helm/stellaops/**'
|
||||
```
|
||||
|
||||
### Database
|
||||
|
||||
```yaml
|
||||
database:
|
||||
- 'devops/database/**'
|
||||
- 'devops/database/postgres/**'
|
||||
```
|
||||
|
||||
### CI/CD Scripts
|
||||
|
||||
```yaml
|
||||
scripts:
|
||||
- '.gitea/scripts/**'
|
||||
- 'devops/scripts/**'
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Test Infrastructure Paths
|
||||
|
||||
### Global Test Suites
|
||||
|
||||
```yaml
|
||||
global_tests:
|
||||
- 'src/__Tests/**'
|
||||
- 'src/__Tests/Integration/**'
|
||||
- 'src/__Tests/architecture/**'
|
||||
- 'src/__Tests/security/**'
|
||||
- 'src/__Tests/chaos/**'
|
||||
- 'src/__Tests/e2e/**'
|
||||
```
|
||||
|
||||
### Shared Test Libraries
|
||||
|
||||
```yaml
|
||||
test_libraries:
|
||||
- 'src/__Tests/__Libraries/**'
|
||||
- 'src/__Tests/__Libraries/StellaOps.TestKit/**'
|
||||
- 'src/__Tests/__Libraries/StellaOps.Infrastructure.Postgres.Testing/**'
|
||||
```
|
||||
|
||||
### Test Datasets
|
||||
|
||||
```yaml
|
||||
datasets:
|
||||
- 'src/__Tests/__Datasets/**'
|
||||
- 'src/__Tests/__Benchmarks/**'
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Example Workflow Configurations
|
||||
|
||||
### PR-Gating Workflow (Skip Docs)
|
||||
|
||||
```yaml
|
||||
on:
|
||||
push:
|
||||
branches: [main]
|
||||
paths-ignore:
|
||||
- 'docs/**'
|
||||
- '*.md'
|
||||
- 'etc/**'
|
||||
pull_request:
|
||||
paths-ignore:
|
||||
- 'docs/**'
|
||||
- '*.md'
|
||||
- 'etc/**'
|
||||
```
|
||||
|
||||
### Module-Specific Workflow (With Cascading)
|
||||
|
||||
```yaml
|
||||
on:
|
||||
push:
|
||||
branches: [main]
|
||||
paths:
|
||||
# Direct module paths
|
||||
- 'src/Scanner/**'
|
||||
- 'src/BinaryIndex/**'
|
||||
# Shared library cascades
|
||||
- 'src/__Libraries/StellaOps.Evidence*/**'
|
||||
- 'src/__Libraries/StellaOps.Cryptography*/**'
|
||||
- 'src/__Libraries/StellaOps.Replay*/**'
|
||||
- 'src/__Libraries/StellaOps.Provenance/**'
|
||||
# Infrastructure cascades
|
||||
- 'Directory.Build.props'
|
||||
- 'Directory.Packages.props'
|
||||
# Self-reference
|
||||
- '.gitea/workflows/scanner-*.yml'
|
||||
pull_request:
|
||||
paths:
|
||||
- 'src/Scanner/**'
|
||||
- 'src/BinaryIndex/**'
|
||||
- 'src/__Libraries/StellaOps.Evidence*/**'
|
||||
- 'src/__Libraries/StellaOps.Cryptography*/**'
|
||||
```
|
||||
|
||||
### Documentation-Only Workflow
|
||||
|
||||
```yaml
|
||||
on:
|
||||
push:
|
||||
paths:
|
||||
- 'docs/**'
|
||||
- '*.md'
|
||||
- 'scripts/render_docs.py'
|
||||
pull_request:
|
||||
paths:
|
||||
- 'docs/**'
|
||||
- '*.md'
|
||||
```
|
||||
|
||||
### Docker/Container Workflow
|
||||
|
||||
```yaml
|
||||
on:
|
||||
push:
|
||||
paths:
|
||||
- '**/Dockerfile'
|
||||
- '**/Dockerfile.*'
|
||||
- 'devops/docker/**'
|
||||
schedule:
|
||||
- cron: '0 4 * * *' # Also run daily for vulnerability updates
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Validation Checklist
|
||||
|
||||
When adding or modifying path filters:
|
||||
|
||||
- [ ] Does the workflow skip docs-only changes? (Use `paths-ignore`)
|
||||
- [ ] Does the workflow include dependent shared library paths? (Cascading)
|
||||
- [ ] Does the workflow include infrastructure files for full builds?
|
||||
- [ ] Are glob patterns correct? (`**` for recursive, `*` for single level)
|
||||
- [ ] Is the workflow self-referenced? (e.g., `.gitea/workflows/module-*.yml`)
|
||||
|
||||
---
|
||||
|
||||
## Glob Pattern Reference
|
||||
|
||||
| Pattern | Matches |
|
||||
|---------|---------|
|
||||
| `src/**` | All files under src/ recursively |
|
||||
| `src/*` | Direct children of src/ only |
|
||||
| `**/*.cs` | All .cs files anywhere |
|
||||
| `*.md` | Markdown files in root only |
|
||||
| `src/**/*.csproj` | All .csproj files under src/ |
|
||||
| `!src/**/*.md` | Exclude markdown in src/ |
|
||||
| `**/Dockerfile*` | Dockerfile, Dockerfile.prod, etc. |
|
||||
|
||||
---
|
||||
|
||||
## Related Documentation
|
||||
|
||||
- [Workflow Triggers](./workflow-triggers.md) - Complete trigger reference
|
||||
- [Test Strategy](./test-strategy.md) - Test categories and execution
|
||||
- [CI/CD Overview](./README.md) - Architecture overview
|
||||
509
docs/cicd/release-pipelines.md
Normal file
509
docs/cicd/release-pipelines.md
Normal file
@@ -0,0 +1,509 @@
|
||||
# Release Pipelines
|
||||
|
||||
> Complete guide to StellaOps release automation including suite releases, module publishing, and promotion workflows.
|
||||
|
||||
---
|
||||
|
||||
## Release Strategy Overview
|
||||
|
||||
StellaOps uses a **dual-versioning strategy**:
|
||||
|
||||
1. **Suite Releases** - Ubuntu-style `YYYY.MM` versioning with codenames
|
||||
2. **Module Releases** - Semantic versioning `MAJOR.MINOR.PATCH` per module
|
||||
|
||||
### Release Channels
|
||||
|
||||
| Channel | Purpose | Stability | Update Frequency |
|
||||
|---------|---------|-----------|------------------|
|
||||
| **Edge** | Latest features, early adopters | Beta | Every merge to main |
|
||||
| **Stable** | Production-ready, tested | Production | Bi-weekly |
|
||||
| **LTS** | Long-term support, enterprise | Enterprise | Quarterly |
|
||||
|
||||
---
|
||||
|
||||
## Suite Release Pipeline
|
||||
|
||||
### Trigger
|
||||
|
||||
```bash
|
||||
# Tag-based trigger
|
||||
git tag suite-2026.04
|
||||
git push origin suite-2026.04
|
||||
|
||||
# Or manual trigger via Gitea Actions UI
|
||||
# Workflow: release-suite.yml
|
||||
# Inputs: version, codename, channel, skip_tests, dry_run
|
||||
```
|
||||
|
||||
### Workflow: `release-suite.yml`
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────────┐
|
||||
│ SUITE RELEASE PIPELINE │
|
||||
│ │
|
||||
│ ┌──────────────┐ │
|
||||
│ │ parse-tag │ (if triggered by tag push) │
|
||||
│ │ or validate │ (if triggered manually) │
|
||||
│ └──────┬───────┘ │
|
||||
│ │ │
|
||||
│ ▼ │
|
||||
│ ┌──────────────┐ │
|
||||
│ │ test-gate │ (optional, skipped with skip_tests=true) │
|
||||
│ └──────┬───────┘ │
|
||||
│ │ │
|
||||
│ ┌────┴────────────────────────────────────────┐ │
|
||||
│ │ BUILD PHASE │ │
|
||||
│ │ │ │
|
||||
│ │ ┌─────────────────┐ ┌─────────────────┐ │ │
|
||||
│ │ │ build-modules │ │ build-containers│ │ │
|
||||
│ │ │ (9 in parallel)│ │ (9 in parallel)│ │ │
|
||||
│ │ └─────────────────┘ └─────────────────┘ │ │
|
||||
│ │ │ │
|
||||
│ │ ┌─────────────────┐ ┌─────────────────┐ │ │
|
||||
│ │ │ build-cli │ │ build-helm │ │ │
|
||||
│ │ │ (5 platforms) │ │ │ │ │
|
||||
│ │ └─────────────────┘ └─────────────────┘ │ │
|
||||
│ │ │ │
|
||||
│ └─────────────────────┬────────────────────────┘ │
|
||||
│ │ │
|
||||
│ ▼ │
|
||||
│ ┌───────────────────────────────────────────────┐ │
|
||||
│ │ release-manifest │ │
|
||||
│ │ - Binary manifest with SHA256 checksums │ │
|
||||
│ │ - SBOM generation (CycloneDX, SPDX) │ │
|
||||
│ │ - Provenance attestation (in-toto/DSSE) │ │
|
||||
│ └───────────────────────┬────────────────────────┘ │
|
||||
│ │ │
|
||||
│ ┌─────────────────────┴─────────────────────────┐ │
|
||||
│ │ │ │
|
||||
│ ▼ ▼ ▼ │
|
||||
│ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │
|
||||
│ │ changelog │ │ suite-docs │ │ compose │ │
|
||||
│ │ generation │ │ generation │ │ generation │ │
|
||||
│ └──────────────┘ └──────────────┘ └──────────────┘ │
|
||||
│ │ │
|
||||
│ ▼ │
|
||||
│ ┌───────────────────────────────────────────────┐ │
|
||||
│ │ create-release │ │
|
||||
│ │ - Upload artifacts to Gitea Releases │ │
|
||||
│ │ - Sign with Cosign (keyless Sigstore) │ │
|
||||
│ │ - Publish to container registry │ │
|
||||
│ └───────────────────────┬────────────────────────┘ │
|
||||
│ │ │
|
||||
│ ▼ │
|
||||
│ ┌───────────────────────────────────────────────┐ │
|
||||
│ │ commit-docs │ │
|
||||
│ │ - Update docs/releases/ │ │
|
||||
│ │ - Update devops/compose/ │ │
|
||||
│ └───────────────────────────────────────────────┘ │
|
||||
│ │
|
||||
└─────────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
### Suite Versioning
|
||||
|
||||
| Component | Format | Example |
|
||||
|-----------|--------|---------|
|
||||
| Suite Version | `YYYY.MM` | `2026.04` |
|
||||
| Codename | Alpha name | `Nova`, `Orion`, `Phoenix` |
|
||||
| Full Tag | `suite-YYYY.MM` | `suite-2026.04` |
|
||||
| Docker Tag | `YYYY.MM-channel` | `2026.04-stable` |
|
||||
|
||||
### Modules Built
|
||||
|
||||
| Module | NuGet Package | Container Image |
|
||||
|--------|---------------|-----------------|
|
||||
| Authority | `StellaOps.Authority` | `stellaops/authority` |
|
||||
| Scanner | `StellaOps.Scanner` | `stellaops/scanner` |
|
||||
| Concelier | `StellaOps.Concelier` | `stellaops/concelier` |
|
||||
| Excititor | `StellaOps.Excititor` | `stellaops/excititor` |
|
||||
| SbomService | `StellaOps.SbomService` | `stellaops/sbom-service` |
|
||||
| EvidenceLocker | `StellaOps.EvidenceLocker` | `stellaops/evidence-locker` |
|
||||
| Policy | `StellaOps.Policy` | `stellaops/policy` |
|
||||
| Attestor | `StellaOps.Attestor` | `stellaops/attestor` |
|
||||
| VexLens | `StellaOps.VexLens` | `stellaops/vexlens` |
|
||||
|
||||
### CLI Platforms
|
||||
|
||||
| Runtime ID | OS | Architecture | Binary Name |
|
||||
|------------|-----|--------------|-------------|
|
||||
| `linux-x64` | Linux | x86_64 | `stellaops-linux-x64` |
|
||||
| `linux-arm64` | Linux | ARM64 | `stellaops-linux-arm64` |
|
||||
| `win-x64` | Windows | x86_64 | `stellaops-win-x64.exe` |
|
||||
| `osx-x64` | macOS | Intel | `stellaops-osx-x64` |
|
||||
| `osx-arm64` | macOS | Apple Silicon | `stellaops-osx-arm64` |
|
||||
|
||||
---
|
||||
|
||||
## Module Release Pipeline
|
||||
|
||||
### Trigger
|
||||
|
||||
```bash
|
||||
# Tag-based trigger
|
||||
git tag module-authority-v1.2.3
|
||||
git push origin module-authority-v1.2.3
|
||||
|
||||
# Or manual trigger via Gitea Actions UI
|
||||
# Workflow: module-publish.yml
|
||||
# Inputs: module, version, publish_nuget, publish_container, prerelease
|
||||
```
|
||||
|
||||
### Workflow: `module-publish.yml`
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────────┐
|
||||
│ MODULE PUBLISH PIPELINE │
|
||||
│ │
|
||||
│ ┌──────────────┐ │
|
||||
│ │ parse-tag │ Extract module name and version from tag │
|
||||
│ │ or validate │ Normalize manual inputs │
|
||||
│ └──────┬───────┘ │
|
||||
│ │ │
|
||||
│ ┌────┴────────────────────────────────────────┐ │
|
||||
│ │ │ │
|
||||
│ ▼ ▼ │
|
||||
│ ┌──────────────┐ ┌──────────────┐ │
|
||||
│ │publish-nuget │ (if flag set) │publish-cont. │ │
|
||||
│ │ │ │ (if flag set)│ │
|
||||
│ │ - Pack │ │ - Build │ │
|
||||
│ │ - Sign │ │ - Scan │ │
|
||||
│ │ - Push │ │ - Sign │ │
|
||||
│ └──────────────┘ │ - Push │ │
|
||||
│ └──────────────┘ │
|
||||
│ │ │
|
||||
│ OR (if module=CLI) │ │
|
||||
│ ▼ │
|
||||
│ ┌──────────────────────────────────────────────────┐ │
|
||||
│ │ publish-cli │ │
|
||||
│ │ - Build for 5 platforms │ │
|
||||
│ │ - Native AOT compilation │ │
|
||||
│ │ - Code sign binaries │ │
|
||||
│ │ - Generate checksums │ │
|
||||
│ │ - Upload to release │ │
|
||||
│ └──────────────────────────────────────────────────┘ │
|
||||
│ │ │
|
||||
│ ▼ │
|
||||
│ ┌──────────────────────────────────────────────────┐ │
|
||||
│ │ summary │ │
|
||||
│ │ - Release notes │ │
|
||||
│ │ - Artifact links │ │
|
||||
│ │ - SBOM references │ │
|
||||
│ └──────────────────────────────────────────────────┘ │
|
||||
│ │
|
||||
└─────────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
### Module Tag Format
|
||||
|
||||
```
|
||||
module-<name>-v<semver>
|
||||
|
||||
Examples:
|
||||
module-authority-v1.2.3
|
||||
module-scanner-v2.0.0
|
||||
module-cli-v3.1.0-beta.1
|
||||
```
|
||||
|
||||
### Available Modules
|
||||
|
||||
| Module Name | NuGet | Container | CLI |
|
||||
|-------------|-------|-----------|-----|
|
||||
| `authority` | Yes | Yes | No |
|
||||
| `scanner` | Yes | Yes | No |
|
||||
| `concelier` | Yes | Yes | No |
|
||||
| `excititor` | Yes | Yes | No |
|
||||
| `sbomservice` | Yes | Yes | No |
|
||||
| `evidencelocker` | Yes | Yes | No |
|
||||
| `policy` | Yes | Yes | No |
|
||||
| `attestor` | Yes | Yes | No |
|
||||
| `vexlens` | Yes | Yes | No |
|
||||
| `cli` | No | No | Yes (multi-platform) |
|
||||
|
||||
---
|
||||
|
||||
## Bundle Release Pipeline
|
||||
|
||||
### Trigger
|
||||
|
||||
```bash
|
||||
# Tag-based trigger
|
||||
git tag v2025.12.1
|
||||
git push origin v2025.12.1
|
||||
|
||||
# Channel-specific tags
|
||||
git tag v2025.12.0-edge
|
||||
git tag v2025.12.0-stable
|
||||
git tag v2025.12.0-lts
|
||||
```
|
||||
|
||||
### Workflow: `release.yml`
|
||||
|
||||
Creates deterministic release bundles with:
|
||||
- Signed container images
|
||||
- SBOM generation
|
||||
- Provenance attestations
|
||||
- CLI parity verification
|
||||
|
||||
---
|
||||
|
||||
## Rollback Pipeline
|
||||
|
||||
### Trigger
|
||||
|
||||
```bash
|
||||
# Manual trigger only via Gitea Actions UI
|
||||
# Workflow: rollback.yml
|
||||
# Inputs: environment, service, version, reason
|
||||
```
|
||||
|
||||
### Workflow: `rollback.yml`
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────────┐
|
||||
│ ROLLBACK PIPELINE │
|
||||
│ (SLA Target: < 5 min) │
|
||||
│ │
|
||||
│ ┌──────────────┐ │
|
||||
│ │ validate │ Verify inputs and permissions │
|
||||
│ └──────┬───────┘ │
|
||||
│ │ │
|
||||
│ ▼ │
|
||||
│ ┌──────────────┐ │
|
||||
│ │ fetch-prev │ Download previous version artifacts │
|
||||
│ │ version │ │
|
||||
│ └──────┬───────┘ │
|
||||
│ │ │
|
||||
│ ▼ │
|
||||
│ ┌──────────────┐ │
|
||||
│ │ execute │ Run rollback via Helm/kubectl │
|
||||
│ │ rollback │ │
|
||||
│ └──────┬───────┘ │
|
||||
│ │ │
|
||||
│ ▼ │
|
||||
│ ┌──────────────┐ │
|
||||
│ │health-check │ Verify service health post-rollback │
|
||||
│ └──────┬───────┘ │
|
||||
│ │ │
|
||||
│ ▼ │
|
||||
│ ┌──────────────┐ │
|
||||
│ │ notify │ Send notification (Slack/Teams/Webhook) │
|
||||
│ └──────────────┘ │
|
||||
│ │
|
||||
└─────────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
### Rollback Parameters
|
||||
|
||||
| Parameter | Type | Description |
|
||||
|-----------|------|-------------|
|
||||
| `environment` | choice | `staging`, `production` |
|
||||
| `service` | choice | Service to rollback (or `all`) |
|
||||
| `version` | string | Target version to rollback to |
|
||||
| `reason` | string | Reason for rollback (audit log) |
|
||||
| `dry_run` | boolean | Simulate without executing |
|
||||
|
||||
---
|
||||
|
||||
## Promotion Pipeline
|
||||
|
||||
### Trigger
|
||||
|
||||
```bash
|
||||
# Manual trigger only via Gitea Actions UI
|
||||
# Workflow: promote.yml
|
||||
# Inputs: from_environment, to_environment, version
|
||||
```
|
||||
|
||||
### Promotion Flow
|
||||
|
||||
```
|
||||
┌─────────────┐
|
||||
│ Edge │ (Automatic on main merge)
|
||||
└──────┬──────┘
|
||||
│
|
||||
│ promote.yml (manual)
|
||||
▼
|
||||
┌─────────────┐
|
||||
│ Stable │ (After testing period)
|
||||
└──────┬──────┘
|
||||
│
|
||||
│ promote.yml (manual)
|
||||
▼
|
||||
┌─────────────┐
|
||||
│ LTS │ (After extended validation)
|
||||
└─────────────┘
|
||||
```
|
||||
|
||||
### Promotion Checklist (Automated)
|
||||
|
||||
1. **Pre-Flight Checks**
|
||||
- All tests passing in source environment
|
||||
- No critical vulnerabilities
|
||||
- Performance SLOs met
|
||||
- Documentation complete
|
||||
|
||||
2. **Promotion Steps**
|
||||
- Re-tag containers with new channel
|
||||
- Update Helm chart values
|
||||
- Deploy to target environment
|
||||
- Run smoke tests
|
||||
|
||||
3. **Post-Promotion**
|
||||
- Health check verification
|
||||
- Update release documentation
|
||||
- Notify stakeholders
|
||||
|
||||
---
|
||||
|
||||
## Artifact Signing
|
||||
|
||||
### Cosign Integration
|
||||
|
||||
All release artifacts are signed using Cosign with Sigstore keyless signing:
|
||||
|
||||
```bash
|
||||
# Verify container signature
|
||||
cosign verify \
|
||||
--certificate-identity-regexp=".*github.com/stellaops.*" \
|
||||
--certificate-oidc-issuer="https://token.actions.githubusercontent.com" \
|
||||
ghcr.io/stellaops/scanner:2026.04
|
||||
|
||||
# Verify SBOM
|
||||
cosign verify-attestation \
|
||||
--type spdxjson \
|
||||
--certificate-identity-regexp=".*github.com/stellaops.*" \
|
||||
ghcr.io/stellaops/scanner:2026.04
|
||||
```
|
||||
|
||||
### Signature Artifacts
|
||||
|
||||
| Artifact Type | Signature Location |
|
||||
|---------------|-------------------|
|
||||
| Container Image | OCI registry (same repo) |
|
||||
| CLI Binary | `.sig` file alongside binary |
|
||||
| SBOM | Attestation on OCI image |
|
||||
| Provenance | Attestation on OCI image |
|
||||
|
||||
---
|
||||
|
||||
## Release Artifacts
|
||||
|
||||
### Per-Release Artifacts
|
||||
|
||||
| Artifact | Format | Location |
|
||||
|----------|--------|----------|
|
||||
| Release Notes | Markdown | Gitea Release |
|
||||
| Changelog | `CHANGELOG.md` | Gitea Release, `docs/releases/` |
|
||||
| Binary Checksums | `SHA256SUMS.txt` | Gitea Release |
|
||||
| SBOM (CycloneDX) | JSON | Gitea Release, OCI attestation |
|
||||
| SBOM (SPDX) | JSON | Gitea Release |
|
||||
| Provenance | in-toto/DSSE | OCI attestation |
|
||||
| Docker Compose | YAML | `devops/compose/` |
|
||||
| Helm Chart | TGZ | OCI registry |
|
||||
|
||||
### Artifact Retention
|
||||
|
||||
| Environment | Retention Period |
|
||||
|-------------|------------------|
|
||||
| PR/Preview | 7 days |
|
||||
| Edge | 30 days |
|
||||
| Stable | 1 year |
|
||||
| LTS | 3 years |
|
||||
|
||||
---
|
||||
|
||||
## Creating a Release
|
||||
|
||||
### Suite Release
|
||||
|
||||
```bash
|
||||
# 1. Ensure main is stable
|
||||
git checkout main
|
||||
git pull
|
||||
|
||||
# 2. Create and push tag
|
||||
git tag suite-2026.04
|
||||
git push origin suite-2026.04
|
||||
|
||||
# 3. Monitor release pipeline
|
||||
# Gitea Actions → release-suite.yml
|
||||
|
||||
# 4. Verify artifacts
|
||||
# - Check Gitea Releases page
|
||||
# - Verify container images pushed
|
||||
# - Validate SBOM and signatures
|
||||
```
|
||||
|
||||
### Module Release
|
||||
|
||||
```bash
|
||||
# 1. Update module version
|
||||
# Edit src/<Module>/version.txt or .csproj
|
||||
|
||||
# 2. Create and push tag
|
||||
git tag module-authority-v1.2.3
|
||||
git push origin module-authority-v1.2.3
|
||||
|
||||
# 3. Monitor release pipeline
|
||||
# Gitea Actions → module-publish.yml
|
||||
```
|
||||
|
||||
### Hotfix Release
|
||||
|
||||
```bash
|
||||
# 1. Create hotfix branch from release tag
|
||||
git checkout -b hotfix/v2025.12.1 v2025.12.0
|
||||
|
||||
# 2. Apply fix
|
||||
# ... make changes ...
|
||||
git commit -m "Fix: critical security issue"
|
||||
|
||||
# 3. Create hotfix tag
|
||||
git tag v2025.12.1
|
||||
git push origin hotfix/v2025.12.1 v2025.12.1
|
||||
|
||||
# 4. Fast-track through pipeline
|
||||
# Workflow will run with reduced test scope
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Troubleshooting Releases
|
||||
|
||||
### Release Pipeline Failed
|
||||
|
||||
1. **Check build logs** - Gitea Actions → failed job
|
||||
2. **Verify tag format** - Must match expected pattern
|
||||
3. **Check secrets** - Registry credentials, signing keys
|
||||
4. **Review test failures** - May need to skip with `skip_tests=true`
|
||||
|
||||
### Container Not Published
|
||||
|
||||
1. **Check registry authentication** - `REGISTRY_TOKEN` secret
|
||||
2. **Verify image name** - Check for typos in workflow
|
||||
3. **Check rate limits** - May need to wait and retry
|
||||
4. **Review scan results** - Image may be blocked by vulnerability scan
|
||||
|
||||
### Signature Verification Failed
|
||||
|
||||
1. **Check Sigstore availability** - May have temporary outage
|
||||
2. **Verify certificate identity** - Workflow must match expected pattern
|
||||
3. **Check OIDC issuer** - Must be GitHub/Gitea Actions
|
||||
|
||||
### Rollback Failed
|
||||
|
||||
1. **Verify target version exists** - Check artifact storage
|
||||
2. **Check Helm/kubectl access** - Cluster credentials
|
||||
3. **Review health check** - Service may need manual intervention
|
||||
4. **Check resource constraints** - May need to scale down first
|
||||
|
||||
---
|
||||
|
||||
## Related Documentation
|
||||
|
||||
- [README - CI/CD Overview](./README.md)
|
||||
- [Workflow Triggers](./workflow-triggers.md)
|
||||
- [Versioning Guide](../releases/VERSIONING.md)
|
||||
- [Container Registry Guide](../operations/container-registry.md)
|
||||
- [Helm Deployment Guide](../operations/helm-deployment.md)
|
||||
508
docs/cicd/security-scanning.md
Normal file
508
docs/cicd/security-scanning.md
Normal file
@@ -0,0 +1,508 @@
|
||||
# Security Scanning
|
||||
|
||||
> Complete guide to security scanning workflows in the StellaOps CI/CD pipeline.
|
||||
|
||||
---
|
||||
|
||||
## Security Scanning Overview
|
||||
|
||||
StellaOps implements a **defense-in-depth** security scanning strategy:
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────────┐
|
||||
│ SECURITY SCANNING LAYERS │
|
||||
├─────────────────────────────────────────────────────────────────┤
|
||||
│ │
|
||||
│ Layer 1: PRE-COMMIT │
|
||||
│ └── Secrets scanning (pre-commit hook) │
|
||||
│ │
|
||||
│ Layer 2: PULL REQUEST │
|
||||
│ ├── SAST (Static Application Security Testing) │
|
||||
│ ├── Secrets scanning │
|
||||
│ ├── Dependency vulnerability audit │
|
||||
│ └── License compliance check │
|
||||
│ │
|
||||
│ Layer 3: MAIN BRANCH │
|
||||
│ ├── All Layer 2 scans │
|
||||
│ ├── Container image scanning │
|
||||
│ └── Extended SAST analysis │
|
||||
│ │
|
||||
│ Layer 4: SCHEDULED │
|
||||
│ ├── Weekly deep SAST scan (Monday) │
|
||||
│ ├── Weekly dependency audit (Sunday) │
|
||||
│ ├── Daily container scanning │
|
||||
│ └── Nightly regression security tests │
|
||||
│ │
|
||||
│ Layer 5: RELEASE │
|
||||
│ ├── Final vulnerability gate │
|
||||
│ ├── SBOM generation and signing │
|
||||
│ ├── Provenance attestation │
|
||||
│ └── Container signing │
|
||||
│ │
|
||||
└─────────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Scanning Workflows
|
||||
|
||||
### 1. SAST Scanning (`sast-scan.yml`)
|
||||
|
||||
**Purpose:** Detect security vulnerabilities in source code through static analysis.
|
||||
|
||||
**Triggers:**
|
||||
- Pull requests (source code changes)
|
||||
- Push to main/develop
|
||||
- Weekly Monday 3:30 AM UTC
|
||||
- Manual dispatch
|
||||
|
||||
**Scanned Languages:**
|
||||
- C# / .NET
|
||||
- JavaScript / TypeScript
|
||||
- Python
|
||||
- YAML
|
||||
- Dockerfile
|
||||
|
||||
**Checks Performed:**
|
||||
|
||||
| Check | Tool | Scope |
|
||||
|-------|------|-------|
|
||||
| Code vulnerabilities | Semgrep/CodeQL (placeholder) | All source |
|
||||
| .NET security analyzers | Built-in Roslyn | C# code |
|
||||
| Dependency vulnerabilities | `dotnet list --vulnerable` | NuGet packages |
|
||||
| Dockerfile best practices | Hadolint | Dockerfiles |
|
||||
|
||||
**Configuration:**
|
||||
|
||||
```yaml
|
||||
# sast-scan.yml inputs
|
||||
workflow_dispatch:
|
||||
inputs:
|
||||
scan_level:
|
||||
type: choice
|
||||
options:
|
||||
- quick # Fast scan, critical issues only
|
||||
- standard # Default, balanced coverage
|
||||
- comprehensive # Full scan, all rules
|
||||
fail_on_findings:
|
||||
type: boolean
|
||||
default: true # Block on findings
|
||||
```
|
||||
|
||||
**.NET Security Analyzer Rules:**
|
||||
|
||||
The workflow enforces these security-critical CA rules as errors:
|
||||
|
||||
| Category | Rules | Description |
|
||||
|----------|-------|-------------|
|
||||
| SQL Injection | CA2100 | Review SQL queries for vulnerabilities |
|
||||
| Cryptography | CA5350-5403 | Weak crypto, insecure algorithms |
|
||||
| Deserialization | CA2300-2362 | Unsafe deserialization |
|
||||
| XML Security | CA3001-3012 | XXE, XPath injection |
|
||||
| Web Security | CA3061, CA5358-5398 | XSS, CSRF, CORS |
|
||||
|
||||
---
|
||||
|
||||
### 2. Secrets Scanning (`secrets-scan.yml`)
|
||||
|
||||
**Purpose:** Detect hardcoded credentials, API keys, and secrets in code.
|
||||
|
||||
**Triggers:**
|
||||
- Pull requests
|
||||
- Push to main/develop
|
||||
- Manual dispatch
|
||||
|
||||
**Detection Patterns:**
|
||||
|
||||
| Secret Type | Example Pattern |
|
||||
|-------------|-----------------|
|
||||
| API Keys | `sk_live_[a-zA-Z0-9]+` |
|
||||
| AWS Keys | `AKIA[0-9A-Z]{16}` |
|
||||
| Private Keys | `-----BEGIN RSA PRIVATE KEY-----` |
|
||||
| Connection Strings | `Password=.*;User ID=.*` |
|
||||
| JWT Tokens | `eyJ[A-Za-z0-9-_]+\.[A-Za-z0-9-_]+` |
|
||||
| GitHub Tokens | `gh[ps]_[A-Za-z0-9]{36}` |
|
||||
|
||||
**Tool Options (Placeholder):**
|
||||
|
||||
```yaml
|
||||
# Choose one by uncommenting in sast-scan.yml:
|
||||
|
||||
# Option 1: TruffleHog (recommended for open source)
|
||||
# - name: TruffleHog Scan
|
||||
# uses: trufflesecurity/trufflehog@main
|
||||
# with:
|
||||
# extra_args: --only-verified
|
||||
|
||||
# Option 2: Gitleaks
|
||||
# - name: Gitleaks Scan
|
||||
# uses: gitleaks/gitleaks-action@v2
|
||||
# env:
|
||||
# GITLEAKS_LICENSE: ${{ secrets.GITLEAKS_LICENSE }}
|
||||
|
||||
# Option 3: Semgrep
|
||||
# - name: Semgrep Secrets
|
||||
# uses: returntocorp/semgrep-action@v1
|
||||
# with:
|
||||
# config: p/secrets
|
||||
```
|
||||
|
||||
**Allowlist Configuration:**
|
||||
|
||||
Create `.gitleaksignore` or `.secretsignore` for false positives:
|
||||
|
||||
```
|
||||
# Ignore test fixtures
|
||||
src/__Tests/**/*
|
||||
docs/examples/**/*
|
||||
|
||||
# Ignore specific files
|
||||
path/to/test-credentials.json
|
||||
|
||||
# Ignore by rule ID
|
||||
[allowlist]
|
||||
regexes = ["test_api_key_[a-z]+"]
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### 3. Container Scanning (`container-scan.yml`)
|
||||
|
||||
**Purpose:** Scan container images for OS and application vulnerabilities.
|
||||
|
||||
**Triggers:**
|
||||
- Dockerfile changes
|
||||
- Daily schedule (4 AM UTC)
|
||||
- Manual dispatch
|
||||
|
||||
**Scan Targets:**
|
||||
|
||||
| Image | Built From | Scanned Components |
|
||||
|-------|------------|-------------------|
|
||||
| `stellaops/authority` | `src/Authority/Dockerfile` | OS packages, .NET runtime |
|
||||
| `stellaops/scanner` | `src/Scanner/Dockerfile` | OS packages, .NET runtime, analyzers |
|
||||
| `stellaops/concelier` | `src/Concelier/Dockerfile` | OS packages, .NET runtime |
|
||||
| (9 total images) | ... | ... |
|
||||
|
||||
**Vulnerability Severity Levels:**
|
||||
|
||||
| Severity | Action | Example |
|
||||
|----------|--------|---------|
|
||||
| CRITICAL | Block release | Remote code execution |
|
||||
| HIGH | Block release (configurable) | Privilege escalation |
|
||||
| MEDIUM | Warning | Information disclosure |
|
||||
| LOW | Log only | Minor issues |
|
||||
| UNKNOWN | Log only | Unclassified |
|
||||
|
||||
**Tool Options (Placeholder):**
|
||||
|
||||
```yaml
|
||||
# Choose one by uncommenting in container-scan.yml:
|
||||
|
||||
# Option 1: Trivy (recommended)
|
||||
# - name: Trivy Scan
|
||||
# uses: aquasecurity/trivy-action@master
|
||||
# with:
|
||||
# image-ref: ${{ steps.build.outputs.image }}
|
||||
# format: sarif
|
||||
# output: trivy-results.sarif
|
||||
# severity: CRITICAL,HIGH
|
||||
|
||||
# Option 2: Grype
|
||||
# - name: Grype Scan
|
||||
# uses: anchore/scan-action@v3
|
||||
# with:
|
||||
# image: ${{ steps.build.outputs.image }}
|
||||
# fail-build: true
|
||||
# severity-cutoff: high
|
||||
|
||||
# Option 3: Snyk Container
|
||||
# - name: Snyk Container
|
||||
# uses: snyk/actions/docker@master
|
||||
# env:
|
||||
# SNYK_TOKEN: ${{ secrets.SNYK_TOKEN }}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### 4. Dependency Security Scanning (`dependency-security-scan.yml`)
|
||||
|
||||
**Purpose:** Audit NuGet and npm packages for known vulnerabilities.
|
||||
|
||||
**Triggers:**
|
||||
- Weekly Sunday 2 AM UTC
|
||||
- Pull requests (dependency file changes)
|
||||
- Manual dispatch
|
||||
|
||||
**Scanned Files:**
|
||||
|
||||
| Ecosystem | Files |
|
||||
|-----------|-------|
|
||||
| NuGet | `src/Directory.Packages.props`, `**/*.csproj` |
|
||||
| npm | `**/package.json`, `**/package-lock.json` |
|
||||
|
||||
**Vulnerability Sources:**
|
||||
|
||||
- GitHub Advisory Database
|
||||
- NVD (National Vulnerability Database)
|
||||
- OSV (Open Source Vulnerabilities)
|
||||
- Vendor security advisories
|
||||
|
||||
**Scan Process:**
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────────┐
|
||||
│ DEPENDENCY SECURITY SCAN │
|
||||
├─────────────────────────────────────────────────────────────────┤
|
||||
│ │
|
||||
│ ┌──────────────┐ │
|
||||
│ │ scan-nuget │ dotnet list package --vulnerable │
|
||||
│ └──────┬───────┘ │
|
||||
│ │ │
|
||||
│ ▼ │
|
||||
│ ┌──────────────┐ │
|
||||
│ │ scan-npm │ npm audit --json │
|
||||
│ └──────┬───────┘ │
|
||||
│ │ │
|
||||
│ ▼ │
|
||||
│ ┌──────────────┐ │
|
||||
│ │ summary │ Aggregate results, generate report │
|
||||
│ └──────────────┘ │
|
||||
│ │
|
||||
└─────────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
**Example Output:**
|
||||
|
||||
```
|
||||
## Dependency Vulnerability Audit
|
||||
|
||||
### NuGet Packages
|
||||
| Package | Installed | Vulnerable | Severity | Advisory |
|
||||
|---------|-----------|------------|----------|----------|
|
||||
| Newtonsoft.Json | 12.0.1 | < 13.0.1 | HIGH | GHSA-xxxx |
|
||||
|
||||
### npm Packages
|
||||
| Package | Installed | Vulnerable | Severity | Advisory |
|
||||
|---------|-----------|------------|----------|----------|
|
||||
| lodash | 4.17.15 | < 4.17.21 | CRITICAL | npm:lodash:1 |
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### 5. License Compliance (`dependency-license-gate.yml`)
|
||||
|
||||
**Purpose:** Ensure all dependencies use approved licenses.
|
||||
|
||||
**Approved Licenses:**
|
||||
|
||||
| License | SPDX ID | Status |
|
||||
|---------|---------|--------|
|
||||
| MIT | MIT | Approved |
|
||||
| Apache 2.0 | Apache-2.0 | Approved |
|
||||
| BSD 2-Clause | BSD-2-Clause | Approved |
|
||||
| BSD 3-Clause | BSD-3-Clause | Approved |
|
||||
| ISC | ISC | Approved |
|
||||
| MPL 2.0 | MPL-2.0 | Review Required |
|
||||
| LGPL 2.1+ | LGPL-2.1-or-later | Review Required |
|
||||
| GPL 2.0+ | GPL-2.0-or-later | Blocked (copyleft) |
|
||||
| AGPL 3.0 | AGPL-3.0 | Blocked (copyleft) |
|
||||
|
||||
**Blocked on Violation:**
|
||||
- GPL-licensed runtime dependencies
|
||||
- Unknown/proprietary licenses without explicit approval
|
||||
|
||||
---
|
||||
|
||||
## Scan Results & Reporting
|
||||
|
||||
### GitHub Step Summary
|
||||
|
||||
All security scans generate GitHub Step Summary reports:
|
||||
|
||||
```markdown
|
||||
## SAST Scan Summary
|
||||
|
||||
| Check | Status |
|
||||
|-------|--------|
|
||||
| SAST Analysis | ✅ Pass |
|
||||
| .NET Security | ⚠️ 3 warnings |
|
||||
| Dependency Check | ✅ Pass |
|
||||
| Dockerfile Lint | ✅ Pass |
|
||||
|
||||
### .NET Security Warnings
|
||||
- CA5350: Weak cryptographic algorithm (src/Crypto/Legacy.cs:42)
|
||||
- CA2100: SQL injection risk (src/Data/Query.cs:78)
|
||||
```
|
||||
|
||||
### SARIF Integration
|
||||
|
||||
Scan results are uploaded in SARIF format for IDE integration:
|
||||
|
||||
```yaml
|
||||
- name: Upload SARIF
|
||||
uses: github/codeql-action/upload-sarif@v3
|
||||
with:
|
||||
sarif_file: scan-results.sarif
|
||||
```
|
||||
|
||||
### Artifact Retention
|
||||
|
||||
| Artifact | Retention |
|
||||
|----------|-----------|
|
||||
| SARIF files | 30 days |
|
||||
| Vulnerability reports | 90 days |
|
||||
| License audit logs | 1 year |
|
||||
|
||||
---
|
||||
|
||||
## Security Gates
|
||||
|
||||
### PR Merge Requirements
|
||||
|
||||
| Gate | Threshold | Block Merge? |
|
||||
|------|-----------|--------------|
|
||||
| SAST Critical | 0 | Yes |
|
||||
| SAST High | 0 | Configurable |
|
||||
| Secrets Found | 0 | Yes |
|
||||
| Vulnerable Dependencies (Critical) | 0 | Yes |
|
||||
| Vulnerable Dependencies (High) | 5 | Warning |
|
||||
| License Violations | 0 | Yes |
|
||||
|
||||
### Release Requirements
|
||||
|
||||
| Gate | Threshold | Block Release? |
|
||||
|------|-----------|----------------|
|
||||
| Container Scan (Critical) | 0 | Yes |
|
||||
| Container Scan (High) | 0 | Yes |
|
||||
| SBOM Generation | Success | Yes |
|
||||
| Signature Verification | Valid | Yes |
|
||||
|
||||
---
|
||||
|
||||
## Remediation Workflows
|
||||
|
||||
### Dependency Vulnerability Fix
|
||||
|
||||
1. **Renovate Auto-Fix:**
|
||||
```yaml
|
||||
# renovate.json
|
||||
{
|
||||
"vulnerabilityAlerts": {
|
||||
"enabled": true,
|
||||
"labels": ["security"],
|
||||
"automerge": false
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
2. **Manual Override:**
|
||||
```bash
|
||||
# Update specific package
|
||||
dotnet add package Newtonsoft.Json --version 13.0.3
|
||||
|
||||
# Audit and fix npm
|
||||
npm audit fix
|
||||
```
|
||||
|
||||
### False Positive Suppression
|
||||
|
||||
**.NET Analyzer Suppression:**
|
||||
|
||||
```csharp
|
||||
// Suppress specific instance
|
||||
#pragma warning disable CA2100 // Review SQL queries for vulnerability
|
||||
var query = $"SELECT * FROM {tableName}";
|
||||
#pragma warning restore CA2100
|
||||
|
||||
// Or in .editorconfig
|
||||
[*.cs]
|
||||
dotnet_diagnostic.CA2100.severity = none # NOT RECOMMENDED
|
||||
```
|
||||
|
||||
**Semgrep/SAST Suppression:**
|
||||
|
||||
```csharp
|
||||
// nosemgrep: sql-injection
|
||||
var query = $"SELECT * FROM {tableName}";
|
||||
```
|
||||
|
||||
**Container Scan Ignore:**
|
||||
|
||||
```yaml
|
||||
# .trivyignore
|
||||
CVE-2021-44228 # Log4j - not applicable (no Java)
|
||||
CVE-2022-12345 # Accepted risk with mitigation
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Configuration Files
|
||||
|
||||
### Location
|
||||
|
||||
| File | Purpose | Location |
|
||||
|------|---------|----------|
|
||||
| `.gitleaksignore` | Secrets scan allowlist | Repository root |
|
||||
| `.trivyignore` | Container scan ignore list | Repository root |
|
||||
| `.semgrepignore` | SAST ignore patterns | Repository root |
|
||||
| `renovate.json` | Dependency update config | Repository root |
|
||||
| `.editorconfig` | Analyzer severity | Repository root |
|
||||
|
||||
### Example `.trivyignore`
|
||||
|
||||
```
|
||||
# Ignore by CVE ID
|
||||
CVE-2021-44228
|
||||
|
||||
# Ignore by package
|
||||
pkg:npm/lodash@4.17.15
|
||||
|
||||
# Ignore with expiration
|
||||
CVE-2022-12345 exp:2025-06-01
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Scheduled Scan Summary
|
||||
|
||||
| Day | Time (UTC) | Workflow | Focus |
|
||||
|-----|------------|----------|-------|
|
||||
| Daily | 2:00 AM | `nightly-regression.yml` | Security tests |
|
||||
| Daily | 4:00 AM | `container-scan.yml` | Image vulnerabilities |
|
||||
| Sunday | 2:00 AM | `dependency-security-scan.yml` | Package audit |
|
||||
| Monday | 3:30 AM | `sast-scan.yml` | Deep code analysis |
|
||||
|
||||
---
|
||||
|
||||
## Monitoring & Alerts
|
||||
|
||||
### Notification Channels
|
||||
|
||||
Configure notifications for security findings:
|
||||
|
||||
```yaml
|
||||
# In workflow
|
||||
- name: Notify on Critical
|
||||
if: steps.scan.outputs.critical_count > 0
|
||||
run: |
|
||||
curl -X POST "${{ secrets.SLACK_WEBHOOK }}" \
|
||||
-d '{"text":"🚨 Critical security finding in '${{ github.repository }}'"}'
|
||||
```
|
||||
|
||||
### Dashboard Integration
|
||||
|
||||
Security scan results can be exported to:
|
||||
- Grafana dashboards (via OTLP metrics)
|
||||
- Security Information and Event Management (SIEM)
|
||||
- Vulnerability management platforms
|
||||
|
||||
---
|
||||
|
||||
## Related Documentation
|
||||
|
||||
- [README - CI/CD Overview](./README.md)
|
||||
- [Workflow Triggers](./workflow-triggers.md)
|
||||
- [Release Pipelines](./release-pipelines.md)
|
||||
- [Dependency Management](../operations/dependency-management.md)
|
||||
- [SBOM Guide](../sbom/guide.md)
|
||||
461
docs/cicd/test-strategy.md
Normal file
461
docs/cicd/test-strategy.md
Normal file
@@ -0,0 +1,461 @@
|
||||
# Test Strategy
|
||||
|
||||
> Complete guide to the StellaOps testing strategy and CI/CD integration.
|
||||
|
||||
---
|
||||
|
||||
## Test Category Overview
|
||||
|
||||
StellaOps uses a **tiered testing strategy** with 13 test categories:
|
||||
|
||||
### PR-Gating Tests (Required for Merge)
|
||||
|
||||
| Category | Purpose | Timeout | Parallelism |
|
||||
|----------|---------|---------|-------------|
|
||||
| **Unit** | Isolated component tests | 20 min | High |
|
||||
| **Architecture** | Dependency rule enforcement | 15 min | High |
|
||||
| **Contract** | API compatibility | 15 min | High |
|
||||
| **Integration** | Database/service integration | 45 min | Medium |
|
||||
| **Security** | Security-focused assertions | 25 min | High |
|
||||
| **Golden** | Corpus-based validation | 25 min | High |
|
||||
|
||||
### Extended Tests (Scheduled/On-Demand)
|
||||
|
||||
| Category | Purpose | Timeout | Trigger |
|
||||
|----------|---------|---------|---------|
|
||||
| **Performance** | Latency/throughput benchmarks | 45 min | Daily, Manual |
|
||||
| **Benchmark** | BenchmarkDotNet profiling | 60 min | Daily, Manual |
|
||||
| **AirGap** | Offline operation validation | 45 min | Manual |
|
||||
| **Chaos** | Resilience testing | 45 min | Manual |
|
||||
| **Determinism** | Reproducibility verification | 45 min | Manual |
|
||||
| **Resilience** | Failure recovery testing | 45 min | Manual |
|
||||
| **Observability** | Telemetry validation | 30 min | Manual |
|
||||
|
||||
---
|
||||
|
||||
## Test Discovery
|
||||
|
||||
### Automatic Discovery
|
||||
|
||||
The `test-matrix.yml` workflow automatically discovers all test projects:
|
||||
|
||||
```bash
|
||||
# Discovery pattern
|
||||
find src \( \
|
||||
-name "*.Tests.csproj" \
|
||||
-o -name "*UnitTests.csproj" \
|
||||
-o -name "*SmokeTests.csproj" \
|
||||
-o -name "*FixtureTests.csproj" \
|
||||
-o -name "*IntegrationTests.csproj" \
|
||||
\) -type f \
|
||||
! -path "*/node_modules/*" \
|
||||
! -path "*/bin/*" \
|
||||
! -path "*/obj/*" \
|
||||
! -name "StellaOps.TestKit.csproj" \
|
||||
! -name "*Testing.csproj"
|
||||
```
|
||||
|
||||
### Test Category Trait
|
||||
|
||||
Tests are categorized using xUnit traits:
|
||||
|
||||
```csharp
|
||||
[Trait("Category", "Unit")]
|
||||
public class MyUnitTests
|
||||
{
|
||||
[Fact]
|
||||
public void Should_Do_Something()
|
||||
{
|
||||
// ...
|
||||
}
|
||||
}
|
||||
|
||||
[Trait("Category", "Integration")]
|
||||
public class MyIntegrationTests
|
||||
{
|
||||
[Fact]
|
||||
public void Should_Connect_To_Database()
|
||||
{
|
||||
// ...
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Running Specific Categories
|
||||
|
||||
```bash
|
||||
# Run Unit tests only
|
||||
dotnet test --filter "Category=Unit"
|
||||
|
||||
# Run multiple categories
|
||||
dotnet test --filter "Category=Unit|Category=Integration"
|
||||
|
||||
# Run excluding a category
|
||||
dotnet test --filter "Category!=Performance"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Test Infrastructure
|
||||
|
||||
### Shared Libraries
|
||||
|
||||
| Library | Purpose | Location |
|
||||
|---------|---------|----------|
|
||||
| `StellaOps.TestKit` | Common test utilities | `src/__Tests/__Libraries/` |
|
||||
| `StellaOps.Infrastructure.Postgres.Testing` | PostgreSQL fixtures | `src/__Tests/__Libraries/` |
|
||||
| `StellaOps.Concelier.Testing` | Concelier test fixtures | `src/Concelier/__Tests/` |
|
||||
|
||||
### Testcontainers
|
||||
|
||||
Integration tests use Testcontainers for isolated dependencies:
|
||||
|
||||
```csharp
|
||||
public class PostgresFixture : IAsyncLifetime
|
||||
{
|
||||
private readonly PostgreSqlContainer _container = new PostgreSqlBuilder()
|
||||
.WithImage("postgres:16")
|
||||
.WithDatabase("test_db")
|
||||
.Build();
|
||||
|
||||
public string ConnectionString => _container.GetConnectionString();
|
||||
|
||||
public Task InitializeAsync() => _container.StartAsync();
|
||||
public Task DisposeAsync() => _container.DisposeAsync().AsTask();
|
||||
}
|
||||
```
|
||||
|
||||
### Ground Truth Corpus
|
||||
|
||||
Golden tests use a corpus of known-good outputs:
|
||||
|
||||
```
|
||||
src/__Tests/__Datasets/
|
||||
├── scanner/
|
||||
│ ├── golden/
|
||||
│ │ ├── npm-package.expected.json
|
||||
│ │ ├── dotnet-project.expected.json
|
||||
│ │ └── container-image.expected.json
|
||||
│ └── fixtures/
|
||||
│ ├── npm-package/
|
||||
│ ├── dotnet-project/
|
||||
│ └── container-image/
|
||||
└── concelier/
|
||||
├── golden/
|
||||
└── fixtures/
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## CI/CD Integration
|
||||
|
||||
### Test Matrix Workflow
|
||||
|
||||
```yaml
|
||||
# Simplified test-matrix.yml structure
|
||||
jobs:
|
||||
discover:
|
||||
# Find all test projects
|
||||
outputs:
|
||||
test-projects: ${{ steps.find.outputs.projects }}
|
||||
|
||||
pr-gating-tests:
|
||||
strategy:
|
||||
matrix:
|
||||
include:
|
||||
- category: Unit
|
||||
timeout: 20
|
||||
- category: Architecture
|
||||
timeout: 15
|
||||
- category: Contract
|
||||
timeout: 15
|
||||
- category: Security
|
||||
timeout: 25
|
||||
- category: Golden
|
||||
timeout: 25
|
||||
steps:
|
||||
- run: .gitea/scripts/test/run-test-category.sh "${{ matrix.category }}"
|
||||
|
||||
integration:
|
||||
services:
|
||||
postgres:
|
||||
image: postgres:16
|
||||
steps:
|
||||
- run: .gitea/scripts/test/run-test-category.sh Integration
|
||||
|
||||
summary:
|
||||
needs: [discover, pr-gating-tests, integration]
|
||||
```
|
||||
|
||||
### Test Results
|
||||
|
||||
All tests produce TRX (Visual Studio Test Results) files:
|
||||
|
||||
```bash
|
||||
# Output structure
|
||||
TestResults/
|
||||
├── Unit/
|
||||
│ ├── src_Scanner___Tests_StellaOps.Scanner.Tests-unit.trx
|
||||
│ └── src_Authority___Tests_StellaOps.Authority.Tests-unit.trx
|
||||
├── Integration/
|
||||
│ └── ...
|
||||
└── Combined/
|
||||
└── test-results-combined.trx
|
||||
```
|
||||
|
||||
### Coverage Collection
|
||||
|
||||
```yaml
|
||||
# Collect coverage for Unit tests
|
||||
- run: |
|
||||
.gitea/scripts/test/run-test-category.sh Unit --collect-coverage
|
||||
```
|
||||
|
||||
Coverage reports are generated in Cobertura format and converted to HTML.
|
||||
|
||||
---
|
||||
|
||||
## Test Categories Deep Dive
|
||||
|
||||
### Unit Tests
|
||||
|
||||
**Purpose:** Test isolated components without external dependencies.
|
||||
|
||||
**Characteristics:**
|
||||
- No I/O (database, network, file system)
|
||||
- No async waits or delays
|
||||
- Fast execution (< 100ms per test)
|
||||
- High parallelism
|
||||
|
||||
**Example:**
|
||||
|
||||
```csharp
|
||||
[Trait("Category", "Unit")]
|
||||
public class VexPolicyBinderTests
|
||||
{
|
||||
[Fact]
|
||||
public void Bind_WithValidPolicy_ReturnsSuccess()
|
||||
{
|
||||
var binder = new VexPolicyBinder();
|
||||
var policy = new VexPolicy { /* ... */ };
|
||||
|
||||
var result = binder.Bind(policy);
|
||||
|
||||
Assert.True(result.IsSuccess);
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Architecture Tests
|
||||
|
||||
**Purpose:** Enforce architectural rules and dependency constraints.
|
||||
|
||||
**Rules Enforced:**
|
||||
- Layer dependencies (UI → Application → Domain → Infrastructure)
|
||||
- Namespace conventions
|
||||
- Circular dependency prevention
|
||||
- Interface segregation
|
||||
|
||||
**Example:**
|
||||
|
||||
```csharp
|
||||
[Trait("Category", "Architecture")]
|
||||
public class DependencyTests
|
||||
{
|
||||
[Fact]
|
||||
public void Domain_Should_Not_Depend_On_Infrastructure()
|
||||
{
|
||||
var result = Types.InAssembly(typeof(DomainMarker).Assembly)
|
||||
.That().ResideInNamespace("StellaOps.Domain")
|
||||
.ShouldNot().HaveDependencyOn("StellaOps.Infrastructure")
|
||||
.GetResult();
|
||||
|
||||
Assert.True(result.IsSuccessful);
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Contract Tests
|
||||
|
||||
**Purpose:** Validate API contracts are maintained.
|
||||
|
||||
**Checks:**
|
||||
- Request/response schemas
|
||||
- OpenAPI specification compliance
|
||||
- Backward compatibility
|
||||
|
||||
**Example:**
|
||||
|
||||
```csharp
|
||||
[Trait("Category", "Contract")]
|
||||
public class VulnerabilityApiContractTests
|
||||
{
|
||||
[Fact]
|
||||
public async Task GetVulnerability_ReturnsExpectedSchema()
|
||||
{
|
||||
var response = await _client.GetAsync("/api/v1/vulnerabilities/CVE-2024-1234");
|
||||
|
||||
await Verify(response)
|
||||
.UseDirectory("Snapshots")
|
||||
.UseMethodName("GetVulnerability");
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Integration Tests
|
||||
|
||||
**Purpose:** Test component integration with real dependencies.
|
||||
|
||||
**Dependencies:**
|
||||
- PostgreSQL (via Testcontainers)
|
||||
- Valkey/Redis (via Testcontainers)
|
||||
- File system
|
||||
|
||||
**Example:**
|
||||
|
||||
```csharp
|
||||
[Trait("Category", "Integration")]
|
||||
public class VulnerabilityRepositoryTests : IClassFixture<PostgresFixture>
|
||||
{
|
||||
private readonly PostgresFixture _fixture;
|
||||
|
||||
[Fact]
|
||||
public async Task Save_AndRetrieve_Vulnerability()
|
||||
{
|
||||
var repo = new VulnerabilityRepository(_fixture.ConnectionString);
|
||||
var vuln = new Vulnerability { Id = "CVE-2024-1234" };
|
||||
|
||||
await repo.SaveAsync(vuln);
|
||||
var retrieved = await repo.GetAsync("CVE-2024-1234");
|
||||
|
||||
Assert.Equal(vuln.Id, retrieved.Id);
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Security Tests
|
||||
|
||||
**Purpose:** Validate security controls and assertions.
|
||||
|
||||
**Checks:**
|
||||
- Input validation
|
||||
- Authorization enforcement
|
||||
- Cryptographic operations
|
||||
- Secrets handling
|
||||
|
||||
**Example:**
|
||||
|
||||
```csharp
|
||||
[Trait("Category", "Security")]
|
||||
public class AuthorizationTests
|
||||
{
|
||||
[Fact]
|
||||
public async Task Unauthorized_User_Cannot_Access_Admin_Endpoint()
|
||||
{
|
||||
var client = _factory.CreateClient();
|
||||
client.DefaultRequestHeaders.Authorization = null;
|
||||
|
||||
var response = await client.GetAsync("/api/admin/settings");
|
||||
|
||||
Assert.Equal(HttpStatusCode.Unauthorized, response.StatusCode);
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Golden Tests
|
||||
|
||||
**Purpose:** Verify output matches known-good corpus.
|
||||
|
||||
**Example:**
|
||||
|
||||
```csharp
|
||||
[Trait("Category", "Golden")]
|
||||
public class ScannerGoldenTests
|
||||
{
|
||||
[Theory]
|
||||
[InlineData("npm-package")]
|
||||
[InlineData("dotnet-project")]
|
||||
public async Task Scan_MatchesGoldenOutput(string fixture)
|
||||
{
|
||||
var scanner = new ContainerScanner();
|
||||
var result = await scanner.ScanAsync($"fixtures/{fixture}");
|
||||
|
||||
await Verify(result)
|
||||
.UseDirectory("golden")
|
||||
.UseFileName(fixture);
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Performance Testing
|
||||
|
||||
### BenchmarkDotNet
|
||||
|
||||
```csharp
|
||||
[Trait("Category", "Benchmark")]
|
||||
[MemoryDiagnoser]
|
||||
public class ScannerBenchmarks
|
||||
{
|
||||
[Benchmark]
|
||||
public async Task ScanSmallImage()
|
||||
{
|
||||
await _scanner.ScanAsync(_smallImage);
|
||||
}
|
||||
|
||||
[Benchmark]
|
||||
public async Task ScanLargeImage()
|
||||
{
|
||||
await _scanner.ScanAsync(_largeImage);
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Performance SLOs
|
||||
|
||||
| Metric | Target | Action on Breach |
|
||||
|--------|--------|------------------|
|
||||
| Unit test P95 | < 100ms | Warning |
|
||||
| Integration test P95 | < 5s | Warning |
|
||||
| Scan time P95 | < 5 min | Block |
|
||||
| Memory peak | < 2GB | Block |
|
||||
|
||||
---
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Tests Fail in CI but Pass Locally
|
||||
|
||||
1. **Check timezone** - CI uses `TZ=UTC`
|
||||
2. **Check parallelism** - CI runs tests in parallel
|
||||
3. **Check container availability** - Testcontainers requires Docker
|
||||
4. **Check file paths** - Case sensitivity on Linux
|
||||
|
||||
### Flaky Tests
|
||||
|
||||
1. **Add retry logic** for network operations
|
||||
2. **Use proper async/await** - no `Task.Run` for async operations
|
||||
3. **Isolate shared state** - use fresh fixtures per test
|
||||
4. **Increase timeouts** for integration tests
|
||||
|
||||
### Missing Test Category
|
||||
|
||||
Ensure your test class has the correct trait:
|
||||
|
||||
```csharp
|
||||
[Trait("Category", "Unit")] // Add this!
|
||||
public class MyTests
|
||||
{
|
||||
// ...
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Related Documentation
|
||||
|
||||
- [README - CI/CD Overview](./README.md)
|
||||
- [Workflow Triggers](./workflow-triggers.md)
|
||||
- [CI Quality Gates](../testing/ci-quality-gates.md)
|
||||
- [Test Catalog](../testing/TEST_CATALOG.yml)
|
||||
719
docs/cicd/workflow-triggers.md
Normal file
719
docs/cicd/workflow-triggers.md
Normal file
@@ -0,0 +1,719 @@
|
||||
# Workflow Triggers & Dependencies
|
||||
|
||||
> Complete reference for CI/CD workflow triggering rules and dependency chains.
|
||||
|
||||
---
|
||||
|
||||
## Trigger Types Overview
|
||||
|
||||
### 1. Push Triggers
|
||||
|
||||
Workflows triggered by commits pushed to branches.
|
||||
|
||||
```yaml
|
||||
on:
|
||||
push:
|
||||
branches: [main, develop] # Branch filter
|
||||
paths: # Path filter (optional)
|
||||
- 'src/**'
|
||||
paths-ignore: # Exclude paths (optional)
|
||||
- 'docs/**'
|
||||
tags: # Tag filter (for releases)
|
||||
- 'v*'
|
||||
```
|
||||
|
||||
### 2. Pull Request Triggers
|
||||
|
||||
Workflows triggered by PR events.
|
||||
|
||||
```yaml
|
||||
on:
|
||||
pull_request:
|
||||
branches: [main, develop] # Target branch filter
|
||||
types: [opened, synchronize, reopened] # Event types
|
||||
paths:
|
||||
- 'src/**'
|
||||
```
|
||||
|
||||
### 3. Schedule Triggers
|
||||
|
||||
Cron-based scheduled execution.
|
||||
|
||||
```yaml
|
||||
on:
|
||||
schedule:
|
||||
- cron: '0 5 * * *' # Daily at 5 AM UTC
|
||||
- cron: '0 2 * * 0' # Weekly Sunday at 2 AM UTC
|
||||
```
|
||||
|
||||
### 4. Manual Triggers
|
||||
|
||||
On-demand workflow execution with inputs.
|
||||
|
||||
```yaml
|
||||
on:
|
||||
workflow_dispatch:
|
||||
inputs:
|
||||
environment:
|
||||
type: choice
|
||||
options: [staging, production]
|
||||
dry_run:
|
||||
type: boolean
|
||||
default: false
|
||||
```
|
||||
|
||||
### 5. Workflow Call (Reusable)
|
||||
|
||||
Called by other workflows.
|
||||
|
||||
```yaml
|
||||
on:
|
||||
workflow_call:
|
||||
inputs:
|
||||
category:
|
||||
type: string
|
||||
required: true
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Complete Trigger Matrix
|
||||
|
||||
### PR-Gating Workflows (Always Run on PR)
|
||||
|
||||
| Workflow | Branches | Path Filters | Purpose |
|
||||
|----------|----------|--------------|---------|
|
||||
| `test-matrix.yml` | main | `!docs/**`, `!*.md` | Unit, Architecture, Contract, Integration, Security, Golden tests |
|
||||
| `build-test-deploy.yml` | main, develop | `src/**`, `docs/**`, `scripts/**` | Build verification |
|
||||
| `policy-lint.yml` | main | `docs/policy/**`, `src/Cli/**` | Policy file validation |
|
||||
| `sast-scan.yml` | main, develop | `src/**`, `*.cs`, `*.ts`, `Dockerfile*` | Static security analysis |
|
||||
| `docs.yml` | - | `docs/**`, `scripts/render_docs.py` | Documentation validation |
|
||||
| `integration-tests-gate.yml` | main, develop | `src/**`, `src/__Tests/**` | Extended integration |
|
||||
|
||||
### Main Branch Only Workflows
|
||||
|
||||
| Workflow | Trigger Condition | Purpose |
|
||||
|----------|-------------------|---------|
|
||||
| `build-test-deploy.yml` → deploy | `github.ref == 'refs/heads/main'` | Deploy to staging |
|
||||
| `integration-tests-gate.yml` → corpus-validation | `github.ref == 'refs/heads/main'` | Ground truth validation |
|
||||
| `coverage-report` | After integration tests on main | Full coverage analysis |
|
||||
|
||||
### Tag-Triggered Workflows
|
||||
|
||||
| Workflow | Tag Pattern | Example | Purpose |
|
||||
|----------|-------------|---------|---------|
|
||||
| `release-suite.yml` | `suite-*` | `suite-2026.04` | Ubuntu-style suite release |
|
||||
| `release.yml` | `v*` | `v2025.12.1`, `v2025.12.0-edge` | Version bundle release |
|
||||
| `module-publish.yml` | `module-*-v*` | `module-authority-v1.2.3` | Per-module publishing |
|
||||
|
||||
### Scheduled Workflows
|
||||
|
||||
| Workflow | Schedule (UTC) | Frequency | Purpose |
|
||||
|----------|----------------|-----------|---------|
|
||||
| `nightly-regression.yml` | `0 2 * * *` | Daily 2 AM | Full regression suite |
|
||||
| `dependency-security-scan.yml` | `0 2 * * 0` | Sunday 2 AM | Vulnerability audit |
|
||||
| `renovate.yml` | `0 3,15 * * *` | Daily 3 AM & 3 PM | Dependency updates |
|
||||
| `sast-scan.yml` | `30 3 * * 1` | Monday 3:30 AM | Weekly deep scan |
|
||||
| `migration-test.yml` | `30 4 * * *` | Daily 4:30 AM | Migration validation |
|
||||
| `build-test-deploy.yml` | `0 5 * * *` | Daily 5 AM | Extended build tests |
|
||||
| `test-matrix.yml` | `0 5 * * *` | Daily 5 AM | Extended test categories |
|
||||
|
||||
### Manual-Only Workflows
|
||||
|
||||
| Workflow | Inputs | Purpose |
|
||||
|----------|--------|---------|
|
||||
| `cli-build.yml` | rids, config, sign | Multi-platform CLI builds |
|
||||
| `scanner-determinism.yml` | - | Verify scanner reproducibility |
|
||||
| `cross-platform-determinism.yml` | - | Cross-OS build verification |
|
||||
| `rollback.yml` | environment, service, version | Emergency rollback |
|
||||
| `promote.yml` | from_env, to_env, version | Environment promotion |
|
||||
|
||||
---
|
||||
|
||||
## Dependency Chains
|
||||
|
||||
### Build → Test → Deploy Pipeline
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────────┐
|
||||
│ build-test-deploy.yml │
|
||||
├─────────────────────────────────────────────────────────────────┤
|
||||
│ │
|
||||
│ profile-validation ─────────────────────────────────┐ │
|
||||
│ │ │ │
|
||||
│ ▼ │ │
|
||||
│ build-test ─────────────────────────────────────────┤ │
|
||||
│ │ (CLI, Concelier, Authority, Scanner, etc.) │ │
|
||||
│ │ │ │
|
||||
│ ▼ │ │
|
||||
│ quality-gates ──────────────────────────────────────┤ │
|
||||
│ │ (Reachability, TTFS, Performance SLOs) │ │
|
||||
│ │ │ │
|
||||
│ ▼ │ │
|
||||
│ security-testing (PR label or schedule) ────────────┤ │
|
||||
│ │ │ │
|
||||
│ ▼ │ │
|
||||
│ sealed-mode-ci ─────────────────────────────────────┤ │
|
||||
│ │ │ │
|
||||
│ ▼ │ │
|
||||
│ docs ───────────────────────────────────────────────┤ │
|
||||
│ │ │ │
|
||||
│ ▼ │ │
|
||||
│ scanner-perf ───────────────────────────────────────┘ │
|
||||
│ │ │
|
||||
│ ▼ │
|
||||
│ deploy (main branch only OR workflow_dispatch) │
|
||||
│ │ │
|
||||
│ ▼ │
|
||||
│ summary │
|
||||
│ │
|
||||
└─────────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
### Test Matrix Pipeline
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────────┐
|
||||
│ test-matrix.yml │
|
||||
├─────────────────────────────────────────────────────────────────┤
|
||||
│ │
|
||||
│ discover ─────────────────────────────────────────────────┐ │
|
||||
│ │ (Find all *.Tests.csproj files) │ │
|
||||
│ │ │ │
|
||||
│ ├───▶ pr-gating-tests (Matrix: 5 categories) │ │
|
||||
│ │ ├── Unit │ │
|
||||
│ │ ├── Architecture │ │
|
||||
│ │ ├── Contract │ │
|
||||
│ │ ├── Security │ │
|
||||
│ │ └── Golden │ │
|
||||
│ │ │ │
|
||||
│ ├───▶ integration (PostgreSQL service) │ │
|
||||
│ │ │ │
|
||||
│ └───▶ extended-tests (schedule or manual) │ │
|
||||
│ ├── Performance │ │
|
||||
│ ├── Benchmark │ │
|
||||
│ ├── AirGap │ │
|
||||
│ ├── Chaos │ │
|
||||
│ ├── Determinism │ │
|
||||
│ ├── Resilience │ │
|
||||
│ └── Observability │ │
|
||||
│ │ │ │
|
||||
│ ◀────────────────────────────────────────────┘ │ │
|
||||
│ │ │ │
|
||||
│ ▼ │ │
|
||||
│ summary ◀─────────────────────────────────────────────────┘ │
|
||||
│ │
|
||||
└─────────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
### Suite Release Pipeline
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────────┐
|
||||
│ release-suite.yml │
|
||||
│ (suite-* tag OR manual) │
|
||||
├─────────────────────────────────────────────────────────────────┤
|
||||
│ │
|
||||
│ parse-tag (if push event) ────────────────────────────────┐ │
|
||||
│ │ │ │
|
||||
│ ▼ │ │
|
||||
│ validate ─────────────────────────────────────────────────┤ │
|
||||
│ │ │ │
|
||||
│ ▼ │ │
|
||||
│ test-gate (optional, skip with skip_tests=true) ──────────┤ │
|
||||
│ │ │ │
|
||||
│ ├───▶ build-modules (Matrix: 9 modules) │ │
|
||||
│ │ ├── Authority │ │
|
||||
│ │ ├── Scanner │ │
|
||||
│ │ ├── Concelier │ │
|
||||
│ │ ├── Excititor │ │
|
||||
│ │ ├── SbomService │ │
|
||||
│ │ ├── EvidenceLocker │ │
|
||||
│ │ ├── Policy │ │
|
||||
│ │ ├── Attestor │ │
|
||||
│ │ └── VexLens │ │
|
||||
│ │ │ │
|
||||
│ ├───▶ build-containers (Matrix: 9 images) │ │
|
||||
│ │ │ │
|
||||
│ ├───▶ build-cli (Matrix: 5 runtimes) │ │
|
||||
│ │ ├── linux-x64 │ │
|
||||
│ │ ├── linux-arm64 │ │
|
||||
│ │ ├── win-x64 │ │
|
||||
│ │ ├── osx-x64 │ │
|
||||
│ │ └── osx-arm64 │ │
|
||||
│ │ │ │
|
||||
│ └───▶ build-helm │ │
|
||||
│ │ │
|
||||
│ ◀────────────────────────────────────────────────────┘ │
|
||||
│ │ │
|
||||
│ ▼ │
|
||||
│ release-manifest ─────────────────────────────────────────┐ │
|
||||
│ │ │ │
|
||||
│ ├───▶ generate-changelog │ │
|
||||
│ ├───▶ generate-suite-docs │ │
|
||||
│ └───▶ generate-compose │ │
|
||||
│ │ │
|
||||
│ ◀────────────────────────────────────────────────────┘ │
|
||||
│ │ │
|
||||
│ ▼ │
|
||||
│ create-release ───────────────────────────────────────────┐ │
|
||||
│ │ │ │
|
||||
│ ▼ │ │
|
||||
│ commit-docs ──────────────────────────────────────────────┤ │
|
||||
│ │ │ │
|
||||
│ ▼ │ │
|
||||
│ summary ◀─────────────────────────────────────────────────┘ │
|
||||
│ │
|
||||
└─────────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Conditional Execution Patterns
|
||||
|
||||
### Branch-Based Conditions
|
||||
|
||||
```yaml
|
||||
# Deploy only on main branch
|
||||
deploy:
|
||||
if: github.ref == 'refs/heads/main'
|
||||
|
||||
# Run on main or develop
|
||||
if: github.ref == 'refs/heads/main' || github.ref == 'refs/heads/develop'
|
||||
|
||||
# Skip on release branches
|
||||
if: "!startsWith(github.ref, 'refs/heads/release/')"
|
||||
```
|
||||
|
||||
### Event-Based Conditions
|
||||
|
||||
```yaml
|
||||
# Different behavior based on trigger
|
||||
steps:
|
||||
- name: Full scan (schedule)
|
||||
if: github.event_name == 'schedule'
|
||||
run: ./scan.sh --full
|
||||
|
||||
- name: Quick scan (PR)
|
||||
if: github.event_name == 'pull_request'
|
||||
run: ./scan.sh --quick
|
||||
```
|
||||
|
||||
### Input-Based Conditions
|
||||
|
||||
```yaml
|
||||
# Run extended tests if requested
|
||||
extended-tests:
|
||||
if: >-
|
||||
github.event_name == 'schedule' ||
|
||||
github.event.inputs.include_performance == 'true' ||
|
||||
github.event.inputs.include_benchmark == 'true'
|
||||
```
|
||||
|
||||
### Failure-Based Conditions
|
||||
|
||||
```yaml
|
||||
# Run cleanup on failure
|
||||
cleanup:
|
||||
if: failure()
|
||||
|
||||
# Run notification always
|
||||
notify:
|
||||
if: always()
|
||||
|
||||
# Run only on success
|
||||
deploy:
|
||||
if: success()
|
||||
```
|
||||
|
||||
### Complex Conditions
|
||||
|
||||
```yaml
|
||||
# Deploy gate: multiple conditions
|
||||
deploy:
|
||||
if: >-
|
||||
needs.build-test.result == 'success' &&
|
||||
needs.docs.result == 'success' &&
|
||||
needs.scanner-perf.result == 'success' &&
|
||||
((github.event_name == 'push' && github.ref == 'refs/heads/main') ||
|
||||
github.event_name == 'workflow_dispatch')
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Path Filters Reference
|
||||
|
||||
### Common Path Patterns
|
||||
|
||||
| Pattern | Matches | Example Files |
|
||||
|---------|---------|---------------|
|
||||
| `src/**` | All source code | `src/Scanner/Program.cs` |
|
||||
| `docs/**` | All documentation | `docs/api/openapi.yaml` |
|
||||
| `*.md` | Root markdown | `README.md`, `CHANGELOG.md` |
|
||||
| `**/*.csproj` | All project files | `src/Cli/StellaOps.Cli.csproj` |
|
||||
| `devops/**` | DevOps config | `devops/helm/values.yaml` |
|
||||
| `.gitea/workflows/**` | Workflow files | `.gitea/workflows/test-matrix.yml` |
|
||||
|
||||
### Path Filter Examples
|
||||
|
||||
```yaml
|
||||
# Source code changes only
|
||||
paths:
|
||||
- 'src/**'
|
||||
- '!src/**/*.md' # Exclude markdown in src
|
||||
|
||||
# Documentation only
|
||||
paths:
|
||||
- 'docs/**'
|
||||
- '*.md'
|
||||
- 'scripts/render_docs.py'
|
||||
|
||||
# Security-relevant files
|
||||
paths:
|
||||
- 'src/**/*.cs'
|
||||
- 'src/**/*.csproj'
|
||||
- '**/Dockerfile*'
|
||||
- '.gitea/workflows/sast-scan.yml'
|
||||
|
||||
# Dependency files
|
||||
paths:
|
||||
- 'src/Directory.Packages.props'
|
||||
- '**/package.json'
|
||||
- '**/package-lock.json'
|
||||
- '**/*.csproj'
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Tag Patterns Reference
|
||||
|
||||
### Semantic Version Tags
|
||||
|
||||
```yaml
|
||||
# Standard releases
|
||||
tags:
|
||||
- 'v*' # v1.0.0, v2025.12.1, v2025.12.0-edge
|
||||
|
||||
# Channel-specific
|
||||
tags:
|
||||
- 'v*-edge' # v2025.12.0-edge
|
||||
- 'v*-stable' # v2025.12.0-stable
|
||||
- 'v*-lts' # v2025.12.0-lts
|
||||
```
|
||||
|
||||
### Suite Tags
|
||||
|
||||
```yaml
|
||||
tags:
|
||||
- 'suite-*' # suite-2026.04, suite-2026.10
|
||||
```
|
||||
|
||||
### Module Tags
|
||||
|
||||
```yaml
|
||||
tags:
|
||||
- 'module-*-v*' # module-authority-v1.2.3
|
||||
# module-scanner-v2.0.0
|
||||
# module-cli-v3.1.0
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Workflow Inputs Reference
|
||||
|
||||
### Common Input Types
|
||||
|
||||
```yaml
|
||||
workflow_dispatch:
|
||||
inputs:
|
||||
# String input
|
||||
version:
|
||||
description: 'Version to release'
|
||||
required: true
|
||||
type: string
|
||||
|
||||
# Choice input
|
||||
environment:
|
||||
description: 'Target environment'
|
||||
type: choice
|
||||
options:
|
||||
- staging
|
||||
- production
|
||||
default: staging
|
||||
|
||||
# Boolean input
|
||||
dry_run:
|
||||
description: 'Run without making changes'
|
||||
type: boolean
|
||||
default: false
|
||||
|
||||
# Multi-select (via string)
|
||||
rids:
|
||||
description: 'Runtime identifiers (comma-separated)'
|
||||
type: string
|
||||
default: 'linux-x64,linux-arm64,win-x64'
|
||||
```
|
||||
|
||||
### Accessing Inputs
|
||||
|
||||
```yaml
|
||||
steps:
|
||||
- name: Use input
|
||||
run: |
|
||||
echo "Version: ${{ github.event.inputs.version }}"
|
||||
echo "Environment: ${{ inputs.environment }}"
|
||||
|
||||
if [[ "${{ inputs.dry_run }}" == "true" ]]; then
|
||||
echo "Dry run mode"
|
||||
fi
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Best Practices
|
||||
|
||||
### 1. Minimize PR Workflow Duration
|
||||
|
||||
```yaml
|
||||
# Use path filters to skip irrelevant runs
|
||||
paths-ignore:
|
||||
- 'docs/**'
|
||||
- '*.md'
|
||||
- 'LICENSE'
|
||||
|
||||
# Use concurrency to cancel outdated runs
|
||||
concurrency:
|
||||
group: ${{ github.workflow }}-${{ github.ref }}
|
||||
cancel-in-progress: true
|
||||
```
|
||||
|
||||
### 2. Fail Fast for Critical Issues
|
||||
|
||||
```yaml
|
||||
strategy:
|
||||
fail-fast: true # Stop all jobs if one fails
|
||||
matrix:
|
||||
category: [Unit, Integration, Security]
|
||||
```
|
||||
|
||||
### 3. Use Matrix for Parallel Execution
|
||||
|
||||
```yaml
|
||||
strategy:
|
||||
matrix:
|
||||
include:
|
||||
- category: Unit
|
||||
timeout: 20
|
||||
- category: Integration
|
||||
timeout: 45
|
||||
```
|
||||
|
||||
### 4. Preserve Artifacts
|
||||
|
||||
```yaml
|
||||
- uses: actions/upload-artifact@v4
|
||||
with:
|
||||
name: test-results-${{ matrix.category }}
|
||||
path: ./TestResults
|
||||
retention-days: 14 # PR artifacts
|
||||
# retention-days: 90 # Release artifacts
|
||||
```
|
||||
|
||||
### 5. Use Conditional Steps
|
||||
|
||||
```yaml
|
||||
- name: Deploy (main only)
|
||||
if: github.ref == 'refs/heads/main'
|
||||
run: ./deploy.sh
|
||||
|
||||
- name: Notify on failure
|
||||
if: failure()
|
||||
run: ./notify-slack.sh
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Troubleshooting Triggers
|
||||
|
||||
### Workflow Not Running
|
||||
|
||||
1. **Check branch protection rules** - Ensure workflow runs are allowed
|
||||
2. **Verify path filters** - File changes must match `paths` patterns
|
||||
3. **Check `if` conditions** - Job may be skipped by condition
|
||||
4. **Review concurrency settings** - May be cancelled by concurrent run
|
||||
|
||||
### Workflow Running Unexpectedly
|
||||
|
||||
1. **Check `paths-ignore`** - May need to exclude more paths
|
||||
2. **Review schedule** - Cron schedule may overlap with events
|
||||
3. **Check tag patterns** - Tag may match unexpected pattern
|
||||
|
||||
### Schedule Not Triggering
|
||||
|
||||
1. **Verify cron syntax** - Use [crontab.guru](https://crontab.guru/)
|
||||
2. **Check workflow file location** - Must be on default branch
|
||||
3. **Review repository activity** - Inactive repos may have schedules disabled
|
||||
|
||||
---
|
||||
|
||||
## Trigger Decision Tree
|
||||
|
||||
Use this decision tree to determine which workflows run for each event:
|
||||
|
||||
```
|
||||
On PUSH to branch:
|
||||
│
|
||||
├── Is branch main/develop?
|
||||
│ ├── YES → Run Category A (PR-Gating) + B (Main-Only) + affected C (Module)
|
||||
│ └── NO (feature branch) → Skip CI (rely on PR workflow)
|
||||
│
|
||||
On PULL REQUEST:
|
||||
│
|
||||
├── Check changed paths
|
||||
│ ├── docs/** only → Skip all (or run docs.yml only)
|
||||
│ ├── src/** changed → Run Category A + affected C modules
|
||||
│ └── *.csproj or *.props changed → Run Category A + B (full infrastructure)
|
||||
│
|
||||
On TAG push:
|
||||
│
|
||||
├── Match tag pattern
|
||||
│ ├── suite-* → release-suite.yml
|
||||
│ ├── module-*-v* → module-publish.yml
|
||||
│ ├── service-*-v* → service-release.yml
|
||||
│ ├── v*.*.* → containers-multiarch.yml + cli-build.yml
|
||||
│ └── Other → Ignore
|
||||
│
|
||||
On SCHEDULE:
|
||||
│
|
||||
└── Run Category E pipelines per cron schedule
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Smart Dependency Cascading
|
||||
|
||||
When shared libraries change, dependent module tests must also run:
|
||||
|
||||
### Dependency Graph
|
||||
|
||||
```
|
||||
SHARED LIBRARY TRIGGERS TESTS FOR
|
||||
─────────────────────────────────────────────────────────────────
|
||||
StellaOps.Cryptography* → ALL modules (security-critical)
|
||||
- Scanner, Attestor, Authority
|
||||
- EvidenceLocker, Signer
|
||||
- AirGap, Offline tests
|
||||
- Security test suite
|
||||
|
||||
StellaOps.Evidence* → Scanner, Attestor, EvidenceLocker
|
||||
StellaOps.Provenance → ExportCenter, SbomService
|
||||
|
||||
StellaOps.Infrastructure.* → ALL integration tests
|
||||
StellaOps.Postgres* (database-dependent modules)
|
||||
|
||||
StellaOps.Replay* → Scanner, Determinism tests
|
||||
StellaOps.Determinism Replay module tests
|
||||
|
||||
StellaOps.Verdict → Policy, RiskEngine, ReachGraph
|
||||
StellaOps.DeltaVerdict
|
||||
|
||||
StellaOps.Plugin → Authority, Scanner, Concelier
|
||||
(plugin-based modules)
|
||||
|
||||
Directory.Build.props → ALL modules (build config)
|
||||
Directory.Packages.props ALL tests
|
||||
nuget.config
|
||||
```
|
||||
|
||||
### Cascade Implementation
|
||||
|
||||
Each workflow includes paths from its dependencies:
|
||||
|
||||
```yaml
|
||||
# Example: scanner-ci.yml with cascading
|
||||
name: Scanner CI
|
||||
on:
|
||||
push:
|
||||
branches: [main]
|
||||
paths:
|
||||
# Direct module paths
|
||||
- 'src/Scanner/**'
|
||||
- 'src/BinaryIndex/**'
|
||||
# Shared library dependencies (cascading)
|
||||
- 'src/__Libraries/StellaOps.Evidence*/**'
|
||||
- 'src/__Libraries/StellaOps.Cryptography*/**'
|
||||
- 'src/__Libraries/StellaOps.Replay*/**'
|
||||
- 'src/__Libraries/StellaOps.Provenance/**'
|
||||
# Infrastructure (triggers full test)
|
||||
- 'Directory.Build.props'
|
||||
- 'Directory.Packages.props'
|
||||
```
|
||||
|
||||
### Cascade Matrix Quick Reference
|
||||
|
||||
| When This Changes | Run These Tests |
|
||||
|-------------------|-----------------|
|
||||
| `src/__Libraries/StellaOps.Cryptography*/**` | Scanner, Attestor, Authority, Evidence, Signer, AirGap, Security |
|
||||
| `src/__Libraries/StellaOps.Evidence*/**` | Scanner, Attestor, EvidenceLocker, Export |
|
||||
| `src/__Libraries/StellaOps.Infrastructure*/**` | ALL integration tests |
|
||||
| `src/__Libraries/StellaOps.Replay*/**` | Scanner, Determinism, Replay |
|
||||
| `src/__Libraries/StellaOps.Verdict/**` | Policy, RiskEngine, ReachGraph |
|
||||
| `src/__Libraries/StellaOps.Plugin/**` | Authority, Scanner, Concelier |
|
||||
| `Directory.Build.props` | ALL modules |
|
||||
|
||||
---
|
||||
|
||||
## Master Trigger Configuration
|
||||
|
||||
### Complete Workflow Trigger Table
|
||||
|
||||
| Workflow | Feature Branch | PR | Main Push | Tag | Schedule |
|
||||
|----------|:--------------:|:--:|:---------:|:---:|:--------:|
|
||||
| **Category A: PR-Gating** |||||
|
||||
| build-test-deploy.yml | ❌ | ✅ | ✅ | ❌ | ✅ Daily |
|
||||
| test-matrix.yml | ❌ | ✅ | ✅ | ❌ | ✅ Daily |
|
||||
| determinism-gate.yml | ❌ | ✅ | ✅ | ❌ | ❌ |
|
||||
| policy-lint.yml | ❌ | ✅* | ✅* | ❌ | ❌ |
|
||||
| sast-scan.yml | ❌ | ✅ | ✅ | ❌ | ✅ Weekly |
|
||||
| secrets-scan.yml | ❌ | ✅ | ✅ | ❌ | ❌ |
|
||||
| dependency-license-gate.yml | ❌ | ✅* | ❌ | ❌ | ❌ |
|
||||
| **Category B: Main-Only** |||||
|
||||
| container-scan.yml | ❌ | ❌ | ✅* | ❌ | ✅ Daily |
|
||||
| integration-tests-gate.yml | ❌ | ❌ | ✅ | ❌ | ❌ |
|
||||
| api-governance.yml | ❌ | ✅* | ✅* | ❌ | ❌ |
|
||||
| aoc-guard.yml | ❌ | ✅* | ✅* | ❌ | ❌ |
|
||||
| provenance-check.yml | ❌ | ❌ | ✅ | ❌ | ❌ |
|
||||
| **Category C: Module-Specific** |||||
|
||||
| scanner-*.yml | ❌ | ✅* | ✅* | ❌ | ❌ |
|
||||
| concelier-*.yml | ❌ | ✅* | ✅* | ❌ | ❌ |
|
||||
| authority-*.yml | ❌ | ✅* | ✅* | ❌ | ❌ |
|
||||
| findings-ledger-ci.yml | ❌ | ✅* | ✅* | ❌ | ❌ |
|
||||
| evidence-locker.yml | ❌ | ❌ | ❌ | ❌ | ❌ (manual) |
|
||||
| [all module-*.yml] | ❌ | ✅* | ✅* | ❌ | ❌ |
|
||||
| **Category D: Release** |||||
|
||||
| release-suite.yml | ❌ | ❌ | ❌ | ✅ suite-* | ❌ |
|
||||
| module-publish.yml | ❌ | ❌ | ❌ | ✅ module-*-v* | ❌ |
|
||||
| service-release.yml | ❌ | ❌ | ❌ | ✅ service-*-v* | ❌ |
|
||||
| release.yml | ❌ | ❌ | ❌ | ✅ v* | ❌ |
|
||||
| cli-build.yml | ❌ | ❌ | ❌ | ❌ | ❌ (manual) |
|
||||
| containers-multiarch.yml | ❌ | ❌ | ❌ | ❌ | ❌ (manual) |
|
||||
| rollback.yml | ❌ | ❌ | ❌ | ❌ | ❌ (manual) |
|
||||
| promote.yml | ❌ | ❌ | ❌ | ❌ | ❌ (manual) |
|
||||
| **Category E: Scheduled** |||||
|
||||
| nightly-regression.yml | ❌ | ❌ | ❌ | ❌ | ✅ 2AM |
|
||||
| dependency-security-scan.yml | ❌ | ✅* | ❌ | ❌ | ✅ Sun 2AM |
|
||||
| container-scan.yml | ❌ | ❌ | ✅* | ❌ | ✅ 4AM |
|
||||
| renovate.yml | ❌ | ❌ | ❌ | ❌ | ✅ 3AM/3PM |
|
||||
| migration-test.yml | ❌ | ❌ | ❌ | ❌ | ✅ 4:30AM |
|
||||
|
||||
*Legend: ✅* = with path filter, ✅ = always, ❌ = never*
|
||||
|
||||
---
|
||||
|
||||
## Related Documentation
|
||||
|
||||
- [README - CI/CD Overview](./README.md)
|
||||
- [Release Pipelines](./release-pipelines.md)
|
||||
- [Test Strategy](./test-strategy.md)
|
||||
- [Path Filters Reference](./path-filters.md)
|
||||
- [Troubleshooting](../../.gitea/docs/troubleshooting.md)
|
||||
Reference in New Issue
Block a user