git.stella-ops.org/.gitea/docs/architecture.md

# CI/CD Architecture

> **Extended Documentation:** See [docs/cicd/](../../docs/cicd/) for comprehensive CI/CD guides.

## Overview

StellaOps CI/CD infrastructure is built on Gitea Actions with a modular, layered architecture designed for:
- **Determinism**: Reproducible builds and tests across environments
- **Offline-first**: Support for air-gapped deployments
- **Security**: Cryptographic signing and attestation at every stage
- **Scalability**: Parallel execution with intelligent caching

## Quick Links

| Document | Purpose |
|----------|---------|
| [CI/CD Overview](../../docs/cicd/README.md) | High-level architecture and getting started |
| [Workflow Triggers](../../docs/cicd/workflow-triggers.md) | Complete trigger matrix and dependency chains |
| [Release Pipelines](../../docs/cicd/release-pipelines.md) | Suite, module, and bundle release flows |
| [Security Scanning](../../docs/cicd/security-scanning.md) | SAST, secrets, container, and dependency scanning |
| [Troubleshooting](./troubleshooting.md) | Common issues and solutions |
| [Script Reference](./scripts.md) | CI/CD script documentation |

## Workflow Trigger Summary

### Trigger Matrix (100 Workflows)

| Trigger Type | Count | Examples |
|--------------|-------|----------|
| PR + Main Push | 15 | `test-matrix.yml`, `build-test-deploy.yml` |
| Tag-Based | 3 | `release-suite.yml`, `release.yml`, `module-publish.yml` |
| Scheduled | 8 | `nightly-regression.yml`, `renovate.yml` |
| Manual Only | 25+ | `rollback.yml`, `cli-build.yml` |
| Module-Specific | 50+ | Scanner, Concelier, Authority workflows |

### Tag Patterns

| Pattern | Workflow | Example |
|---------|----------|---------|
| `suite-*` | Suite release | `suite-2026.04` |
| `v*` | Bundle release | `v2025.12.1` |
| `module-*-v*` | Module publish | `module-authority-v1.2.3` |

### Schedule Overview

| Time (UTC) | Workflow | Purpose |
|------------|----------|---------|
| 2:00 AM Daily | `nightly-regression.yml` | Full regression |
| 3:00 AM/PM Daily | `renovate.yml` | Dependency updates |
| 3:30 AM Monday | `sast-scan.yml` | Weekly security scan |
| 5:00 AM Daily | `test-matrix.yml` | Extended tests |

> **Full Details:** See [Workflow Triggers](../../docs/cicd/workflow-triggers.md)

## Pipeline Architecture

### Release Pipeline Flow

```mermaid
graph TD
    subgraph "Trigger Layer"
        TAG[Git Tag] --> PARSE[Parse Tag]
        DISPATCH[Manual Dispatch] --> PARSE
        SCHEDULE[Scheduled] --> PARSE
    end

    subgraph "Validation Layer"
        PARSE --> VALIDATE[Validate Inputs]
        VALIDATE --> RESOLVE[Resolve Versions]
    end

    subgraph "Build Layer"
        RESOLVE --> BUILD[Build Modules]
        BUILD --> TEST[Run Tests]
        TEST --> DETERMINISM[Determinism Check]
    end

    subgraph "Artifact Layer"
        DETERMINISM --> CONTAINER[Build Container]
        CONTAINER --> SBOM[Generate SBOM]
        SBOM --> SIGN[Sign Artifacts]
    end

    subgraph "Release Layer"
        SIGN --> MANIFEST[Update Manifest]
        MANIFEST --> CHANGELOG[Generate Changelog]
        CHANGELOG --> DOCS[Generate Docs]
        DOCS --> PUBLISH[Publish Release]
    end

    subgraph "Post-Release"
        PUBLISH --> VERIFY[Verify Release]
        VERIFY --> NOTIFY[Notify Stakeholders]
    end
```

### Service Release Pipeline

```mermaid
graph LR
    subgraph "Trigger"
        A[service-{name}-v{semver}] --> B[Parse Service & Version]
    end

    subgraph "Build"
        B --> C[Read Directory.Versions.props]
        C --> D[Bump Version]
        D --> E[Build Service]
        E --> F[Run Tests]
    end

    subgraph "Package"
        F --> G[Build Container]
        G --> H[Generate Docker Tag]
        H --> I[Push to Registry]
    end

    subgraph "Attestation"
        I --> J[Generate SBOM]
        J --> K[Sign with Cosign]
        K --> L[Create Attestation]
    end

    subgraph "Finalize"
        L --> M[Update Manifest]
        M --> N[Commit Changes]
    end
```

### Test Matrix Execution

```mermaid
graph TD
    subgraph "Matrix Strategy"
        TRIGGER[PR/Push] --> FILTER[Path Filter]
        FILTER --> MATRIX[Generate Matrix]
    end

    subgraph "Parallel Execution"
        MATRIX --> UNIT[Unit Tests]
        MATRIX --> INT[Integration Tests]
        MATRIX --> DET[Determinism Tests]
    end

    subgraph "Test Types"
        UNIT --> UNIT_FAST[Fast Unit]
        UNIT --> UNIT_SLOW[Slow Unit]
        INT --> INT_PG[PostgreSQL]
        INT --> INT_VALKEY[Valkey]
        DET --> DET_SCANNER[Scanner]
        DET --> DET_BUILD[Build Output]
    end

    subgraph "Reporting"
        UNIT_FAST --> TRX[TRX Reports]
        UNIT_SLOW --> TRX
        INT_PG --> TRX
        INT_VALKEY --> TRX
        DET_SCANNER --> TRX
        DET_BUILD --> TRX
        TRX --> SUMMARY[Job Summary]
    end
```

## Workflow Dependencies

### Core Dependencies

```mermaid
graph TD
    BTD[build-test-deploy.yml] --> TM[test-matrix.yml]
    BTD --> DG[determinism-gate.yml]

    TM --> TL[test-lanes.yml]
    TM --> ITG[integration-tests-gate.yml]

    RS[release-suite.yml] --> BTD
    RS --> MP[module-publish.yml]
    RS --> AS[artifact-signing.yml]

    SR[service-release.yml] --> BTD
    SR --> AS

    MP --> AS
    MP --> AB[attestation-bundle.yml]
```

### Security Chain

```mermaid
graph LR
    BUILD[Build] --> SBOM[SBOM Generation]
    SBOM --> SIGN[Cosign Signing]
    SIGN --> ATTEST[Attestation]
    ATTEST --> VERIFY[Verification]
    VERIFY --> PUBLISH[Publish]
```

## Execution Stages

### Stage 1: Validation

| Step | Purpose | Tools |
|------|---------|-------|
| Parse trigger | Extract tag/input parameters | bash |
| Validate config | Check required files exist | bash |
| Resolve versions | Read from Directory.Versions.props | Python |
| Check permissions | Verify secrets available | Gitea Actions |

### Stage 2: Build

| Step | Purpose | Tools |
|------|---------|-------|
| Restore packages | NuGet/npm dependencies | dotnet restore, npm ci |
| Build solution | Compile all projects | dotnet build |
| Run analyzers | Code analysis | dotnet analyzers |

### Stage 3: Test

| Step | Purpose | Tools |
|------|---------|-------|
| Unit tests | Component testing | xUnit |
| Integration tests | Service integration | Testcontainers |
| Determinism tests | Output reproducibility | Custom scripts |

### Stage 4: Package

| Step | Purpose | Tools |
|------|---------|-------|
| Build container | Docker image | docker build |
| Generate SBOM | Software bill of materials | Syft |
| Sign artifacts | Cryptographic signing | Cosign |
| Create attestation | in-toto/DSSE envelope | Custom tools |

### Stage 5: Publish

| Step | Purpose | Tools |
|------|---------|-------|
| Push container | Registry upload | docker push |
| Upload attestation | Rekor transparency | Cosign |
| Update manifest | Version tracking | Python |
| Generate docs | Release documentation | Python |

## Concurrency Control

### Strategy

```yaml
concurrency:
  group: ${{ github.workflow }}-${{ github.ref }}
  cancel-in-progress: true
```

### Workflow Groups

| Group | Behavior | Workflows |
|-------|----------|-----------|
| Build | Cancel in-progress | `build-test-deploy.yml` |
| Release | No cancel (sequential) | `release-suite.yml` |
| Deploy | Environment-locked | `promote.yml` |
| Scheduled | Allow concurrent | `renovate.yml` |

## Caching Strategy

### Cache Layers

```mermaid
graph TD
    subgraph "Package Cache"
        NUGET[NuGet Cache<br>~/.nuget/packages]
        NPM[npm Cache<br>~/.npm]
    end

    subgraph "Build Cache"
        OBJ[Object Files<br>**/obj]
        BIN[Binaries<br>**/bin]
    end

    subgraph "Test Cache"
        TC[Testcontainers<br>Images]
        FIX[Test Fixtures]
    end

    subgraph "Keys"
        K1[runner.os-nuget-hash] --> NUGET
        K2[runner.os-npm-hash] --> NPM
        K3[runner.os-dotnet-hash] --> OBJ
        K3 --> BIN
    end
```

### Cache Configuration

| Cache | Key Pattern | Restore Keys |
|-------|-------------|--------------|
| NuGet | `${{ runner.os }}-nuget-${{ hashFiles('**/*.csproj') }}` | `${{ runner.os }}-nuget-` |
| npm | `${{ runner.os }}-npm-${{ hashFiles('**/package-lock.json') }}` | `${{ runner.os }}-npm-` |
| .NET Build | `${{ runner.os }}-dotnet-${{ github.sha }}` | `${{ runner.os }}-dotnet-` |

## Runner Requirements

### Self-Hosted Runners

| Label | Purpose | Requirements |
|-------|---------|--------------|
| `ubuntu-latest` | General builds | 4 CPU, 16GB RAM, 100GB disk |
| `linux-arm64` | ARM builds | ARM64 host |
| `windows-latest` | Windows builds | Windows Server 2022 |
| `macos-latest` | macOS builds | macOS 13+ |

### Docker-in-Docker

Required for:
- Testcontainers integration tests
- Multi-architecture builds
- Container scanning

### Network Requirements

| Endpoint | Purpose | Required |
|----------|---------|----------|
| `git.stella-ops.org` | Source, Registry | Always |
| `nuget.org` | NuGet packages | Online mode |
| `registry.npmjs.org` | npm packages | Online mode |
| `ghcr.io` | GitHub Container Registry | Optional |

## Artifact Flow

### Build Artifacts

```
artifacts/
├── binaries/
│   ├── StellaOps.Cli-linux-x64
│   ├── StellaOps.Cli-linux-arm64
│   ├── StellaOps.Cli-win-x64
│   └── StellaOps.Cli-osx-arm64
├── containers/
│   ├── scanner:1.2.3+20250128143022
│   └── authority:1.0.0+20250128143022
├── sbom/
│   ├── scanner.cyclonedx.json
│   └── authority.cyclonedx.json
└── attestations/
    ├── scanner.intoto.jsonl
    └── authority.intoto.jsonl
```

### Release Artifacts

```
docs/releases/2026.04/
├── README.md
├── CHANGELOG.md
├── services.md
├── docker-compose.yml
├── docker-compose.airgap.yml
├── upgrade-guide.md
├── checksums.txt
└── manifest.yaml
```

## Error Handling

### Retry Strategy

| Step Type | Retries | Backoff |
|-----------|---------|---------|
| Network calls | 3 | Exponential |
| Docker push | 3 | Linear (30s) |
| Tests | 0 | N/A |
| Signing | 2 | Linear (10s) |

### Failure Actions

| Failure Type | Action |
|--------------|--------|
| Build failure | Fail fast, notify |
| Test failure | Continue, report |
| Signing failure | Fail, alert security |
| Deploy failure | Rollback, notify |

## Security Architecture

### Secret Management

```mermaid
graph TD
    subgraph "Gitea Secrets"
        GS[Organization Secrets]
        RS[Repository Secrets]
        ES[Environment Secrets]
    end

    subgraph "Usage"
        GS --> BUILD[Build Workflows]
        RS --> SIGN[Signing Workflows]
        ES --> DEPLOY[Deploy Workflows]
    end

    subgraph "Rotation"
        ROTATE[Key Rotation] --> RS
        ROTATE --> ES
    end
```

### Signing Chain

1. **Build outputs**: SHA-256 checksums
2. **Container images**: Cosign keyless/keyed signing
3. **SBOMs**: in-toto attestation
4. **Releases**: GPG-signed tags

## Monitoring & Observability

### Workflow Metrics

| Metric | Source | Dashboard |
|--------|--------|-----------|
| Build duration | Gitea Actions | Grafana |
| Test pass rate | TRX reports | Grafana |
| Cache hit rate | Actions cache | Prometheus |
| Artifact size | Upload artifact | Prometheus |

### Alerts

| Alert | Condition | Action |
|-------|-----------|--------|
| Build time > 30m | Duration threshold | Investigate |
| Test failures > 5% | Rate threshold | Review |
| Cache miss streak | 3 consecutive | Clear cache |
| Security scan critical | Any critical CVE | Block merge |