# CI/CD Infrastructure Overview

> **Sprint:** CI/CD Enhancement - Documentation
> **Last Updated:** 2025-12-28
> **Workflow Count:** 100 workflows

## Quick Links

- [Workflow Triggers & Dependencies](./workflow-triggers.md)
- [Release Pipelines](./release-pipelines.md)
- [Security Scanning](./security-scanning.md)
- [Test Strategy](./test-strategy.md)
- [Troubleshooting Guide](../.gitea/docs/troubleshooting.md)

---

## Architecture Overview

The StellaOps CI/CD infrastructure uses **Gitea Actions** (GitHub Actions compatible) with a sophisticated multi-tier triggering strategy designed for:

- **Determinism & Reproducibility** - Identical builds across runs
- **Offline-First Operation** - Air-gap compatible pipelines
- **Supply Chain Security** - SLSA Level 2-3 compliance
- **Developer Velocity** - Fast PR feedback with comprehensive nightly testing

### Pipeline Tiers

```
┌─────────────────────────────────────────────────────────────────────────┐
│                        TRIGGER HIERARCHY                                 │
├─────────────────────────────────────────────────────────────────────────┤
│                                                                         │
│  TIER 1: PR GATING (Every Pull Request)                                │
│  ├── test-matrix.yml (Unit, Architecture, Contract, Integration,       │
│  │                    Security, Golden)                                 │
│  ├── build-test-deploy.yml (Build verification)                        │
│  ├── policy-lint.yml (Policy file validation)                          │
│  ├── sast-scan.yml (Static security analysis)                          │
│  └── docs.yml (Documentation validation)                               │
│                                                                         │
│  TIER 2: MAIN BRANCH (Post-Merge)                                      │
│  ├── All Tier 1 workflows                                              │
│  ├── build-test-deploy.yml → Deploy stage (staging environment)        │
│  ├── integration-tests-gate.yml → Extended coverage                    │
│  └── coverage-report (Full coverage analysis)                          │
│                                                                         │
│  TIER 3: SCHEDULED (Nightly/Weekly)                                    │
│  ├── nightly-regression.yml (2:00 AM UTC daily)                        │
│  ├── test-matrix.yml → Extended tests (5:00 AM UTC daily)              │
│  ├── dependency-security-scan.yml (2:00 AM UTC Sunday)                 │
│  ├── renovate.yml (3:00 AM & 3:00 PM UTC daily)                        │
│  ├── sast-scan.yml (3:30 AM UTC Monday)                                │
│  └── migration-test.yml (4:30 AM UTC daily)                            │
│                                                                         │
│  TIER 4: RELEASE (Tag-Triggered)                                       │
│  ├── release-suite.yml (suite-YYYY.MM tags)                            │
│  ├── release.yml (v* tags)                                             │
│  └── module-publish.yml (module-*-v* tags)                             │
│                                                                         │
│  TIER 5: MANUAL (On-Demand)                                            │
│  ├── cli-build.yml, scanner-determinism.yml                            │
│  ├── rollback.yml, promote.yml                                         │
│  └── 20+ specialized test/debug workflows                              │
│                                                                         │
└─────────────────────────────────────────────────────────────────────────┘
```

---

## Workflow Categories

### 1. Core Build & Test (12 workflows)

| Workflow | Purpose | Triggers |
|----------|---------|----------|
| `build-test-deploy.yml` | Main build pipeline | PR, main push, daily, manual |
| `test-matrix.yml` | Unified test execution | PR, main push, daily, manual |
| `integration-tests-gate.yml` | Extended integration testing | PR, main push, manual |
| `nightly-regression.yml` | Comprehensive nightly suite | Daily 2 AM UTC |
| `migration-test.yml` | Database migration validation | PR (migrations), daily |

### 2. Release Automation (8 workflows)

| Workflow | Purpose | Triggers |
|----------|---------|----------|
| `release-suite.yml` | Ubuntu-style suite releases | `suite-*` tags, manual |
| `release.yml` | Version bundle releases | `v*` tags, manual |
| `module-publish.yml` | Per-module publishing | `module-*-v*` tags, manual |
| `cli-build.yml` | Multi-platform CLI builds | Manual only |
| `promote.yml` | Environment promotion | Manual only |
| `rollback.yml` | Emergency rollback | Manual only |

### 3. Security Scanning (6 workflows)

| Workflow | Purpose | Triggers |
|----------|---------|----------|
| `sast-scan.yml` | Static code analysis | PR, main push, weekly |
| `secrets-scan.yml` | Credential detection | PR, main push |
| `container-scan.yml` | Image vulnerability scanning | Dockerfile changes, daily |
| `dependency-security-scan.yml` | NuGet/npm vulnerability audit | Weekly, PR (deps) |
| `dependency-license-gate.yml` | License compliance | PR (deps) |

### 4. Quality Assurance (15 workflows)

| Workflow | Purpose | Triggers |
|----------|---------|----------|
| `policy-lint.yml` | Policy file validation | PR, main push |
| `docs.yml` | Documentation linting | docs/** changes |
| `scanner-determinism.yml` | Output reproducibility | Manual only |
| `determinism-gate.yml` | Build determinism | Manual only |
| `cross-platform-determinism.yml` | Multi-OS verification | Manual only |

### 5. Module-Specific (30+ workflows)

Specialized workflows for individual modules (Scanner, Concelier, Authority, etc.)

---

## Trigger Quick Reference

### Branch Patterns

| Pattern | Example | Workflows Triggered |
|---------|---------|---------------------|
| Push to `main` | Direct commit or merge | All Tier 1 + Tier 2 |
| Push to `develop` | Feature integration | Selected gating workflows |
| Pull Request | Any PR to main/develop | All Tier 1 (gating) |
| Push to `feature/*` | Feature branches | None (PR required) |
| Push to `release/*` | Release prep branches | Selected validation |

### Tag Patterns

| Pattern | Example | Workflow |
|---------|---------|----------|
| `v*` | `v2025.12.1` | `release.yml` |
| `suite-*` | `suite-2026.04` | `release-suite.yml` |
| `module-*-v*` | `module-authority-v1.2.3` | `module-publish.yml` |

### Schedule Summary

| Time (UTC) | Frequency | Workflow |
|------------|-----------|----------|
| 2:00 AM | Daily | `nightly-regression.yml` |
| 2:00 AM | Sunday | `dependency-security-scan.yml` |
| 3:00 AM | Daily | `renovate.yml` |
| 3:30 AM | Monday | `sast-scan.yml` |
| 4:30 AM | Daily | `migration-test.yml` |
| 5:00 AM | Daily | `build-test-deploy.yml`, `test-matrix.yml` |
| 3:00 PM | Daily | `renovate.yml` |

---

## Environment Flow

```
┌──────────┐    ┌──────────┐    ┌──────────┐    ┌──────────┐
│    PR    │───▶│  Staging │───▶│  Stable  │───▶│   LTS    │
│ (Preview)│    │  (Edge)  │    │ (Tested) │    │(Long-Term)│
└──────────┘    └──────────┘    └──────────┘    └──────────┘
     │               │               │               │
     │               │               │               │
     ▼               ▼               ▼               ▼
  PR tests      Auto-deploy      promote.yml    promote.yml
  (gating)      on main merge    (manual)       (manual)
```

### Environment Matrix

| Environment | Branch/Tag | Auto-Deploy | Rollback |
|-------------|------------|-------------|----------|
| Preview | PR | Yes (ephemeral) | N/A |
| Staging (Edge) | `main` | Yes | `rollback.yml` |
| Stable | `v*` tags | Manual | `rollback.yml` |
| LTS | `suite-*` tags | Manual | `rollback.yml` |

---

## Key Features

### 1. PR-Gating Tests

Required tests that must pass before merge:

- **Unit Tests** - Fast, isolated tests
- **Architecture Tests** - Dependency rule enforcement
- **Contract Tests** - API compatibility
- **Integration Tests** - PostgreSQL integration
- **Security Tests** - Security-focused assertions
- **Golden Tests** - Corpus-based validation

### 2. Determinism Verification

All builds produce identical outputs:

- Binary checksums compared across runs
- UTC timezone enforcement (`TZ: UTC`)
- Stable JSON serialization
- Reproducible SBOM generation

### 3. Supply Chain Security

- **SBOM Generation** - Syft for CycloneDX/SPDX
- **Artifact Signing** - Cosign/Sigstore integration
- **Provenance** - in-toto/DSSE attestations
- **Dependency Scanning** - Automated vulnerability detection

### 4. Rollback Automation

Emergency rollback via `rollback.yml`:
- Target: < 5 minute SLA
- Helm-based deployment rollback
- Health check verification
- Notification integration

---

## Directory Structure

```
.gitea/
├── workflows/           # 100 workflow files
│   ├── build-test-deploy.yml
│   ├── test-matrix.yml
│   ├── release-suite.yml
│   └── ...
├── scripts/             # CI/CD scripts
│   ├── build/           # Build orchestration
│   ├── test/            # Test execution
│   ├── release/         # Release automation
│   ├── sign/            # Signing operations
│   └── validate/        # Validation scripts
└── docs/                # CI-specific docs
    ├── architecture.md
    ├── scripts.md
    └── troubleshooting.md

devops/
├── scripts/
│   └── lib/             # Shared bash libraries
│       ├── logging.sh
│       ├── exit-codes.sh
│       ├── git-utils.sh
│       ├── path-utils.sh
│       └── hash-utils.sh
├── compose/             # Docker Compose profiles
├── helm/                # Helm charts
└── docker/              # Dockerfiles
```

---

## Getting Started

### Running Workflows Locally

```bash
# Run test matrix locally
./devops/scripts/test-local.sh

# Validate compose files
./devops/scripts/validate-compose.sh

# Run a specific test category
./.gitea/scripts/test/run-test-category.sh Unit
```

### Triggering Manual Workflows

```bash
# Via Gitea UI: Actions → Workflow → Run workflow

# Or via API:
curl -X POST \
  -H "Authorization: token $GITEA_TOKEN" \
  "$GITEA_URL/api/v1/repos/owner/repo/actions/workflows/rollback.yml/dispatches" \
  -d '{"ref":"main","inputs":{"environment":"staging","version":"v2025.12.0"}}'
```

### Creating a Release

1. **Module Release:**
   ```bash
   git tag module-authority-v1.2.3
   git push origin module-authority-v1.2.3
   ```

2. **Suite Release:**
   ```bash
   git tag suite-2026.04
   git push origin suite-2026.04
   ```

3. **Bundle Release:**
   ```bash
   git tag v2025.12.1
   git push origin v2025.12.1
   ```

---

## Related Documentation

- [Workflow Triggers Deep Dive](./workflow-triggers.md)
- [Release Pipeline Details](./release-pipelines.md)
- [Security Scanning Guide](./security-scanning.md)
- [Test Strategy](./test-strategy.md)
- [CI Quality Gates](../testing/ci-quality-gates.md)
- [Troubleshooting](../.gitea/docs/troubleshooting.md)
- [Script Reference](../.gitea/docs/scripts.md)

---

## Metrics & Monitoring

### Key Metrics Tracked

| Metric | Target | Measurement |
|--------|--------|-------------|
| PR Build Time | < 15 min | Workflow duration |
| Main Build Time | < 20 min | Workflow duration |
| Test Flakiness | < 1% | Flaky test detection |
| Security Scan Coverage | 100% | SAST/DAST coverage |
| Rollback SLA | < 5 min | Rollback workflow duration |

### Dashboard Links

- [Workflow Runs](../../.gitea/workflows/) (Gitea Actions UI)
- [Test Results](./test-results/) (TRX/JUnit artifacts)
- [Coverage Reports](./coverage/) (Generated nightly)