Fix build and code structure improvements. New but essential UI functionality. CI improvements. Documentation improvements. AI module improvements.
This commit is contained in:
StellaOps Bot
432
.gitea/docs/architecture.md
Normal file
432
.gitea/docs/architecture.md
Normal file
@@ -0,0 +1,432 @@
|
||||
# CI/CD Architecture
|
||||
|
||||
> **Extended Documentation:** See [docs/cicd/](../../docs/cicd/) for comprehensive CI/CD guides.
|
||||
|
||||
## Overview
|
||||
|
||||
StellaOps CI/CD infrastructure is built on Gitea Actions with a modular, layered architecture designed for:
|
||||
- **Determinism**: Reproducible builds and tests across environments
|
||||
- **Offline-first**: Support for air-gapped deployments
|
||||
- **Security**: Cryptographic signing and attestation at every stage
|
||||
- **Scalability**: Parallel execution with intelligent caching
|
||||
|
||||
## Quick Links
|
||||
|
||||
| Document | Purpose |
|
||||
|----------|---------|
|
||||
| [CI/CD Overview](../../docs/cicd/README.md) | High-level architecture and getting started |
|
||||
| [Workflow Triggers](../../docs/cicd/workflow-triggers.md) | Complete trigger matrix and dependency chains |
|
||||
| [Release Pipelines](../../docs/cicd/release-pipelines.md) | Suite, module, and bundle release flows |
|
||||
| [Security Scanning](../../docs/cicd/security-scanning.md) | SAST, secrets, container, and dependency scanning |
|
||||
| [Troubleshooting](./troubleshooting.md) | Common issues and solutions |
|
||||
| [Script Reference](./scripts.md) | CI/CD script documentation |
|
||||
|
||||
## Workflow Trigger Summary
|
||||
|
||||
### Trigger Matrix (100 Workflows)
|
||||
|
||||
| Trigger Type | Count | Examples |
|
||||
|--------------|-------|----------|
|
||||
| PR + Main Push | 15 | `test-matrix.yml`, `build-test-deploy.yml` |
|
||||
| Tag-Based | 3 | `release-suite.yml`, `release.yml`, `module-publish.yml` |
|
||||
| Scheduled | 8 | `nightly-regression.yml`, `renovate.yml` |
|
||||
| Manual Only | 25+ | `rollback.yml`, `cli-build.yml` |
|
||||
| Module-Specific | 50+ | Scanner, Concelier, Authority workflows |
|
||||
|
||||
### Tag Patterns
|
||||
|
||||
| Pattern | Workflow | Example |
|
||||
|---------|----------|---------|
|
||||
| `suite-*` | Suite release | `suite-2026.04` |
|
||||
| `v*` | Bundle release | `v2025.12.1` |
|
||||
| `module-*-v*` | Module publish | `module-authority-v1.2.3` |
|
||||
|
||||
### Schedule Overview
|
||||
|
||||
| Time (UTC) | Workflow | Purpose |
|
||||
|------------|----------|---------|
|
||||
| 2:00 AM Daily | `nightly-regression.yml` | Full regression |
|
||||
| 3:00 AM/PM Daily | `renovate.yml` | Dependency updates |
|
||||
| 3:30 AM Monday | `sast-scan.yml` | Weekly security scan |
|
||||
| 5:00 AM Daily | `test-matrix.yml` | Extended tests |
|
||||
|
||||
> **Full Details:** See [Workflow Triggers](../../docs/cicd/workflow-triggers.md)
|
||||
|
||||
## Pipeline Architecture
|
||||
|
||||
### Release Pipeline Flow
|
||||
|
||||
```mermaid
|
||||
graph TD
|
||||
subgraph "Trigger Layer"
|
||||
TAG[Git Tag] --> PARSE[Parse Tag]
|
||||
DISPATCH[Manual Dispatch] --> PARSE
|
||||
SCHEDULE[Scheduled] --> PARSE
|
||||
end
|
||||
|
||||
subgraph "Validation Layer"
|
||||
PARSE --> VALIDATE[Validate Inputs]
|
||||
VALIDATE --> RESOLVE[Resolve Versions]
|
||||
end
|
||||
|
||||
subgraph "Build Layer"
|
||||
RESOLVE --> BUILD[Build Modules]
|
||||
BUILD --> TEST[Run Tests]
|
||||
TEST --> DETERMINISM[Determinism Check]
|
||||
end
|
||||
|
||||
subgraph "Artifact Layer"
|
||||
DETERMINISM --> CONTAINER[Build Container]
|
||||
CONTAINER --> SBOM[Generate SBOM]
|
||||
SBOM --> SIGN[Sign Artifacts]
|
||||
end
|
||||
|
||||
subgraph "Release Layer"
|
||||
SIGN --> MANIFEST[Update Manifest]
|
||||
MANIFEST --> CHANGELOG[Generate Changelog]
|
||||
CHANGELOG --> DOCS[Generate Docs]
|
||||
DOCS --> PUBLISH[Publish Release]
|
||||
end
|
||||
|
||||
subgraph "Post-Release"
|
||||
PUBLISH --> VERIFY[Verify Release]
|
||||
VERIFY --> NOTIFY[Notify Stakeholders]
|
||||
end
|
||||
```
|
||||
|
||||
### Service Release Pipeline
|
||||
|
||||
```mermaid
|
||||
graph LR
|
||||
subgraph "Trigger"
|
||||
A[service-{name}-v{semver}] --> B[Parse Service & Version]
|
||||
end
|
||||
|
||||
subgraph "Build"
|
||||
B --> C[Read Directory.Versions.props]
|
||||
C --> D[Bump Version]
|
||||
D --> E[Build Service]
|
||||
E --> F[Run Tests]
|
||||
end
|
||||
|
||||
subgraph "Package"
|
||||
F --> G[Build Container]
|
||||
G --> H[Generate Docker Tag]
|
||||
H --> I[Push to Registry]
|
||||
end
|
||||
|
||||
subgraph "Attestation"
|
||||
I --> J[Generate SBOM]
|
||||
J --> K[Sign with Cosign]
|
||||
K --> L[Create Attestation]
|
||||
end
|
||||
|
||||
subgraph "Finalize"
|
||||
L --> M[Update Manifest]
|
||||
M --> N[Commit Changes]
|
||||
end
|
||||
```
|
||||
|
||||
### Test Matrix Execution
|
||||
|
||||
```mermaid
|
||||
graph TD
|
||||
subgraph "Matrix Strategy"
|
||||
TRIGGER[PR/Push] --> FILTER[Path Filter]
|
||||
FILTER --> MATRIX[Generate Matrix]
|
||||
end
|
||||
|
||||
subgraph "Parallel Execution"
|
||||
MATRIX --> UNIT[Unit Tests]
|
||||
MATRIX --> INT[Integration Tests]
|
||||
MATRIX --> DET[Determinism Tests]
|
||||
end
|
||||
|
||||
subgraph "Test Types"
|
||||
UNIT --> UNIT_FAST[Fast Unit]
|
||||
UNIT --> UNIT_SLOW[Slow Unit]
|
||||
INT --> INT_PG[PostgreSQL]
|
||||
INT --> INT_VALKEY[Valkey]
|
||||
DET --> DET_SCANNER[Scanner]
|
||||
DET --> DET_BUILD[Build Output]
|
||||
end
|
||||
|
||||
subgraph "Reporting"
|
||||
UNIT_FAST --> TRX[TRX Reports]
|
||||
UNIT_SLOW --> TRX
|
||||
INT_PG --> TRX
|
||||
INT_VALKEY --> TRX
|
||||
DET_SCANNER --> TRX
|
||||
DET_BUILD --> TRX
|
||||
TRX --> SUMMARY[Job Summary]
|
||||
end
|
||||
```
|
||||
|
||||
## Workflow Dependencies
|
||||
|
||||
### Core Dependencies
|
||||
|
||||
```mermaid
|
||||
graph TD
|
||||
BTD[build-test-deploy.yml] --> TM[test-matrix.yml]
|
||||
BTD --> DG[determinism-gate.yml]
|
||||
|
||||
TM --> TL[test-lanes.yml]
|
||||
TM --> ITG[integration-tests-gate.yml]
|
||||
|
||||
RS[release-suite.yml] --> BTD
|
||||
RS --> MP[module-publish.yml]
|
||||
RS --> AS[artifact-signing.yml]
|
||||
|
||||
SR[service-release.yml] --> BTD
|
||||
SR --> AS
|
||||
|
||||
MP --> AS
|
||||
MP --> AB[attestation-bundle.yml]
|
||||
```
|
||||
|
||||
### Security Chain
|
||||
|
||||
```mermaid
|
||||
graph LR
|
||||
BUILD[Build] --> SBOM[SBOM Generation]
|
||||
SBOM --> SIGN[Cosign Signing]
|
||||
SIGN --> ATTEST[Attestation]
|
||||
ATTEST --> VERIFY[Verification]
|
||||
VERIFY --> PUBLISH[Publish]
|
||||
```
|
||||
|
||||
## Execution Stages
|
||||
|
||||
### Stage 1: Validation
|
||||
|
||||
| Step | Purpose | Tools |
|
||||
|------|---------|-------|
|
||||
| Parse trigger | Extract tag/input parameters | bash |
|
||||
| Validate config | Check required files exist | bash |
|
||||
| Resolve versions | Read from Directory.Versions.props | Python |
|
||||
| Check permissions | Verify secrets available | Gitea Actions |
|
||||
|
||||
### Stage 2: Build
|
||||
|
||||
| Step | Purpose | Tools |
|
||||
|------|---------|-------|
|
||||
| Restore packages | NuGet/npm dependencies | dotnet restore, npm ci |
|
||||
| Build solution | Compile all projects | dotnet build |
|
||||
| Run analyzers | Code analysis | dotnet analyzers |
|
||||
|
||||
### Stage 3: Test
|
||||
|
||||
| Step | Purpose | Tools |
|
||||
|------|---------|-------|
|
||||
| Unit tests | Component testing | xUnit |
|
||||
| Integration tests | Service integration | Testcontainers |
|
||||
| Determinism tests | Output reproducibility | Custom scripts |
|
||||
|
||||
### Stage 4: Package
|
||||
|
||||
| Step | Purpose | Tools |
|
||||
|------|---------|-------|
|
||||
| Build container | Docker image | docker build |
|
||||
| Generate SBOM | Software bill of materials | Syft |
|
||||
| Sign artifacts | Cryptographic signing | Cosign |
|
||||
| Create attestation | in-toto/DSSE envelope | Custom tools |
|
||||
|
||||
### Stage 5: Publish
|
||||
|
||||
| Step | Purpose | Tools |
|
||||
|------|---------|-------|
|
||||
| Push container | Registry upload | docker push |
|
||||
| Upload attestation | Rekor transparency | Cosign |
|
||||
| Update manifest | Version tracking | Python |
|
||||
| Generate docs | Release documentation | Python |
|
||||
|
||||
## Concurrency Control
|
||||
|
||||
### Strategy
|
||||
|
||||
```yaml
|
||||
concurrency:
|
||||
group: ${{ github.workflow }}-${{ github.ref }}
|
||||
cancel-in-progress: true
|
||||
```
|
||||
|
||||
### Workflow Groups
|
||||
|
||||
| Group | Behavior | Workflows |
|
||||
|-------|----------|-----------|
|
||||
| Build | Cancel in-progress | `build-test-deploy.yml` |
|
||||
| Release | No cancel (sequential) | `release-suite.yml` |
|
||||
| Deploy | Environment-locked | `promote.yml` |
|
||||
| Scheduled | Allow concurrent | `renovate.yml` |
|
||||
|
||||
## Caching Strategy
|
||||
|
||||
### Cache Layers
|
||||
|
||||
```mermaid
|
||||
graph TD
|
||||
subgraph "Package Cache"
|
||||
NUGET[NuGet Cache<br>~/.nuget/packages]
|
||||
NPM[npm Cache<br>~/.npm]
|
||||
end
|
||||
|
||||
subgraph "Build Cache"
|
||||
OBJ[Object Files<br>**/obj]
|
||||
BIN[Binaries<br>**/bin]
|
||||
end
|
||||
|
||||
subgraph "Test Cache"
|
||||
TC[Testcontainers<br>Images]
|
||||
FIX[Test Fixtures]
|
||||
end
|
||||
|
||||
subgraph "Keys"
|
||||
K1[runner.os-nuget-hash] --> NUGET
|
||||
K2[runner.os-npm-hash] --> NPM
|
||||
K3[runner.os-dotnet-hash] --> OBJ
|
||||
K3 --> BIN
|
||||
end
|
||||
```
|
||||
|
||||
### Cache Configuration
|
||||
|
||||
| Cache | Key Pattern | Restore Keys |
|
||||
|-------|-------------|--------------|
|
||||
| NuGet | `${{ runner.os }}-nuget-${{ hashFiles('**/*.csproj') }}` | `${{ runner.os }}-nuget-` |
|
||||
| npm | `${{ runner.os }}-npm-${{ hashFiles('**/package-lock.json') }}` | `${{ runner.os }}-npm-` |
|
||||
| .NET Build | `${{ runner.os }}-dotnet-${{ github.sha }}` | `${{ runner.os }}-dotnet-` |
|
||||
|
||||
## Runner Requirements
|
||||
|
||||
### Self-Hosted Runners
|
||||
|
||||
| Label | Purpose | Requirements |
|
||||
|-------|---------|--------------|
|
||||
| `ubuntu-latest` | General builds | 4 CPU, 16GB RAM, 100GB disk |
|
||||
| `linux-arm64` | ARM builds | ARM64 host |
|
||||
| `windows-latest` | Windows builds | Windows Server 2022 |
|
||||
| `macos-latest` | macOS builds | macOS 13+ |
|
||||
|
||||
### Docker-in-Docker
|
||||
|
||||
Required for:
|
||||
- Testcontainers integration tests
|
||||
- Multi-architecture builds
|
||||
- Container scanning
|
||||
|
||||
### Network Requirements
|
||||
|
||||
| Endpoint | Purpose | Required |
|
||||
|----------|---------|----------|
|
||||
| `git.stella-ops.org` | Source, Registry | Always |
|
||||
| `nuget.org` | NuGet packages | Online mode |
|
||||
| `registry.npmjs.org` | npm packages | Online mode |
|
||||
| `ghcr.io` | GitHub Container Registry | Optional |
|
||||
|
||||
## Artifact Flow
|
||||
|
||||
### Build Artifacts
|
||||
|
||||
```
|
||||
artifacts/
|
||||
├── binaries/
|
||||
│ ├── StellaOps.Cli-linux-x64
|
||||
│ ├── StellaOps.Cli-linux-arm64
|
||||
│ ├── StellaOps.Cli-win-x64
|
||||
│ └── StellaOps.Cli-osx-arm64
|
||||
├── containers/
|
||||
│ ├── scanner:1.2.3+20250128143022
|
||||
│ └── authority:1.0.0+20250128143022
|
||||
├── sbom/
|
||||
│ ├── scanner.cyclonedx.json
|
||||
│ └── authority.cyclonedx.json
|
||||
└── attestations/
|
||||
├── scanner.intoto.jsonl
|
||||
└── authority.intoto.jsonl
|
||||
```
|
||||
|
||||
### Release Artifacts
|
||||
|
||||
```
|
||||
docs/releases/2026.04/
|
||||
├── README.md
|
||||
├── CHANGELOG.md
|
||||
├── services.md
|
||||
├── docker-compose.yml
|
||||
├── docker-compose.airgap.yml
|
||||
├── upgrade-guide.md
|
||||
├── checksums.txt
|
||||
└── manifest.yaml
|
||||
```
|
||||
|
||||
## Error Handling
|
||||
|
||||
### Retry Strategy
|
||||
|
||||
| Step Type | Retries | Backoff |
|
||||
|-----------|---------|---------|
|
||||
| Network calls | 3 | Exponential |
|
||||
| Docker push | 3 | Linear (30s) |
|
||||
| Tests | 0 | N/A |
|
||||
| Signing | 2 | Linear (10s) |
|
||||
|
||||
### Failure Actions
|
||||
|
||||
| Failure Type | Action |
|
||||
|--------------|--------|
|
||||
| Build failure | Fail fast, notify |
|
||||
| Test failure | Continue, report |
|
||||
| Signing failure | Fail, alert security |
|
||||
| Deploy failure | Rollback, notify |
|
||||
|
||||
## Security Architecture
|
||||
|
||||
### Secret Management
|
||||
|
||||
```mermaid
|
||||
graph TD
|
||||
subgraph "Gitea Secrets"
|
||||
GS[Organization Secrets]
|
||||
RS[Repository Secrets]
|
||||
ES[Environment Secrets]
|
||||
end
|
||||
|
||||
subgraph "Usage"
|
||||
GS --> BUILD[Build Workflows]
|
||||
RS --> SIGN[Signing Workflows]
|
||||
ES --> DEPLOY[Deploy Workflows]
|
||||
end
|
||||
|
||||
subgraph "Rotation"
|
||||
ROTATE[Key Rotation] --> RS
|
||||
ROTATE --> ES
|
||||
end
|
||||
```
|
||||
|
||||
### Signing Chain
|
||||
|
||||
1. **Build outputs**: SHA-256 checksums
|
||||
2. **Container images**: Cosign keyless/keyed signing
|
||||
3. **SBOMs**: in-toto attestation
|
||||
4. **Releases**: GPG-signed tags
|
||||
|
||||
## Monitoring & Observability
|
||||
|
||||
### Workflow Metrics
|
||||
|
||||
| Metric | Source | Dashboard |
|
||||
|--------|--------|-----------|
|
||||
| Build duration | Gitea Actions | Grafana |
|
||||
| Test pass rate | TRX reports | Grafana |
|
||||
| Cache hit rate | Actions cache | Prometheus |
|
||||
| Artifact size | Upload artifact | Prometheus |
|
||||
|
||||
### Alerts
|
||||
|
||||
| Alert | Condition | Action |
|
||||
|-------|-----------|--------|
|
||||
| Build time > 30m | Duration threshold | Investigate |
|
||||
| Test failures > 5% | Rate threshold | Review |
|
||||
| Cache miss streak | 3 consecutive | Clear cache |
|
||||
| Security scan critical | Any critical CVE | Block merge |
|
||||
Reference in New Issue
Block a user