docs consolidation and others
This commit is contained in:
master
553
docs/technical/testing/LOCAL_CI_GUIDE.md
Normal file
553
docs/technical/testing/LOCAL_CI_GUIDE.md
Normal file
@@ -0,0 +1,553 @@
|
||||
# Local CI Testing Guide
|
||||
|
||||
> Run CI/CD pipelines locally before pushing to verify your changes.
|
||||
|
||||
---
|
||||
|
||||
## Quick Start
|
||||
|
||||
### Prerequisites
|
||||
|
||||
- **Docker Desktop** (Windows/macOS) or Docker Engine (Linux)
|
||||
- **.NET 10 SDK** ([download](https://dot.net/download))
|
||||
- **Git**
|
||||
- **Bash** (Linux/macOS native, Windows via WSL2 or Git Bash)
|
||||
- **act** (optional, for workflow simulation) - [install](https://github.com/nektos/act)
|
||||
|
||||
### Installation
|
||||
|
||||
1. **Copy environment template:**
|
||||
```bash
|
||||
cp devops/ci-local/.env.local.sample devops/ci-local/.env.local
|
||||
```
|
||||
|
||||
2. **(Optional) Install act for workflow simulation:**
|
||||
```bash
|
||||
# macOS
|
||||
brew install act
|
||||
|
||||
# Linux
|
||||
curl -sSL https://raw.githubusercontent.com/nektos/act/master/install.sh | sudo bash
|
||||
|
||||
# Windows (via Chocolatey)
|
||||
choco install act-cli
|
||||
```
|
||||
|
||||
3. **(Optional) Build CI Docker image:**
|
||||
```bash
|
||||
docker build -t stellaops-ci:local -f devops/docker/Dockerfile.ci .
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Usage
|
||||
|
||||
### Basic Commands
|
||||
|
||||
```bash
|
||||
# Quick smoke test (~2 min)
|
||||
./devops/scripts/local-ci.sh smoke
|
||||
|
||||
# Smoke steps (isolate build vs unit tests)
|
||||
./devops/scripts/local-ci.sh smoke --smoke-step build
|
||||
./devops/scripts/local-ci.sh smoke --smoke-step unit
|
||||
./devops/scripts/local-ci.sh smoke --smoke-step unit-split
|
||||
./devops/scripts/local-ci.sh smoke --smoke-step unit-split --test-timeout 5m --progress-interval 60
|
||||
./devops/scripts/local-ci.sh smoke --smoke-step unit-split --project-start 1 --project-count 50
|
||||
|
||||
# Full PR-gating suite (~15 min)
|
||||
./devops/scripts/local-ci.sh pr
|
||||
|
||||
# Test specific module
|
||||
./devops/scripts/local-ci.sh module --module Scanner
|
||||
|
||||
# Auto-detect changed modules
|
||||
./devops/scripts/local-ci.sh module
|
||||
|
||||
# Simulate workflow
|
||||
./devops/scripts/local-ci.sh workflow --workflow test-matrix
|
||||
|
||||
# Release dry-run
|
||||
./devops/scripts/local-ci.sh release --dry-run
|
||||
|
||||
# Full test suite (~45 min)
|
||||
./devops/scripts/local-ci.sh full
|
||||
```
|
||||
|
||||
### Windows (PowerShell)
|
||||
|
||||
```powershell
|
||||
# Quick smoke test
|
||||
.\devops\scripts\local-ci.ps1 smoke
|
||||
|
||||
# Smoke steps (isolate build vs unit tests)
|
||||
.\devops\scripts\local-ci.ps1 smoke -SmokeStep build
|
||||
.\devops\scripts\local-ci.ps1 smoke -SmokeStep unit
|
||||
.\devops\scripts\local-ci.ps1 smoke -SmokeStep unit-split
|
||||
.\devops\scripts\local-ci.ps1 smoke -SmokeStep unit-split -TestTimeout 5m -ProgressInterval 60
|
||||
.\devops\scripts\local-ci.ps1 smoke -SmokeStep unit-split -ProjectStart 1 -ProjectCount 50
|
||||
|
||||
# Full PR check
|
||||
.\devops\scripts\local-ci.ps1 pr
|
||||
|
||||
# With options
|
||||
.\devops\scripts\local-ci.ps1 pr -Verbose -Docker
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Modes
|
||||
|
||||
### smoke (~2 minutes)
|
||||
Quick validation before pushing. Runs only Unit tests.
|
||||
|
||||
```bash
|
||||
./devops/scripts/local-ci.sh smoke
|
||||
```
|
||||
|
||||
Optional stepwise smoke (to isolate hangs):
|
||||
|
||||
```bash
|
||||
./devops/scripts/local-ci.sh smoke --smoke-step build
|
||||
./devops/scripts/local-ci.sh smoke --smoke-step unit
|
||||
./devops/scripts/local-ci.sh smoke --smoke-step unit-split
|
||||
```
|
||||
|
||||
**What it does:**
|
||||
1. Builds the solution
|
||||
2. Runs Unit tests
|
||||
3. Reports pass/fail
|
||||
|
||||
### pr (~15 minutes)
|
||||
Full PR-gating suite matching CI exactly. Run this before opening a PR.
|
||||
|
||||
```bash
|
||||
./devops/scripts/local-ci.sh pr
|
||||
```
|
||||
|
||||
**What it does:**
|
||||
1. Starts CI services (PostgreSQL, Valkey)
|
||||
2. Builds with warnings-as-errors
|
||||
3. Runs all PR-gating categories:
|
||||
- Unit
|
||||
- Architecture
|
||||
- Contract
|
||||
- Integration
|
||||
- Security
|
||||
- Golden
|
||||
4. Generates TRX reports
|
||||
5. Stops CI services
|
||||
|
||||
### module
|
||||
Test only the modules you've changed. Detects changes via git diff.
|
||||
|
||||
```bash
|
||||
# Auto-detect changed modules
|
||||
./devops/scripts/local-ci.sh module
|
||||
|
||||
# Test specific module
|
||||
./devops/scripts/local-ci.sh module --module Scanner
|
||||
./devops/scripts/local-ci.sh module --module Concelier
|
||||
./devops/scripts/local-ci.sh module --module Authority
|
||||
```
|
||||
|
||||
**Available modules:**
|
||||
Scanner, Concelier, Authority, Policy, Attestor, EvidenceLocker, ExportCenter,
|
||||
Findings, SbomService, Notify, Router, Cryptography, AirGap, Cli, AdvisoryAI,
|
||||
ReachGraph, Orchestrator, PacksRegistry, Replay, Aoc, IssuerDirectory, Telemetry, Signals
|
||||
|
||||
### workflow
|
||||
Simulate a specific Gitea Actions workflow using `act`.
|
||||
|
||||
```bash
|
||||
# Simulate test-matrix workflow
|
||||
./devops/scripts/local-ci.sh workflow --workflow test-matrix
|
||||
|
||||
# Simulate build-test-deploy
|
||||
./devops/scripts/local-ci.sh workflow --workflow build-test-deploy
|
||||
```
|
||||
|
||||
**Requirements:**
|
||||
- `act` must be installed
|
||||
- CI Docker image built (`stellaops-ci:local`)
|
||||
|
||||
### release
|
||||
Dry-run release pipeline to verify release artifacts.
|
||||
|
||||
```bash
|
||||
./devops/scripts/local-ci.sh release --dry-run
|
||||
```
|
||||
|
||||
**What it does:**
|
||||
1. Builds all modules
|
||||
2. Validates CLI packaging
|
||||
3. Validates Helm chart
|
||||
4. Shows what would be released
|
||||
5. **Does NOT publish anything**
|
||||
|
||||
### full (~45 minutes)
|
||||
Complete test suite including extended categories.
|
||||
|
||||
```bash
|
||||
./devops/scripts/local-ci.sh full
|
||||
```
|
||||
|
||||
**Categories run:**
|
||||
- PR-Gating: Unit, Architecture, Contract, Integration, Security, Golden
|
||||
- Extended: Performance, Benchmark, AirGap, Chaos, Determinism, Resilience, Observability
|
||||
|
||||
---
|
||||
|
||||
## Options
|
||||
|
||||
| Option | Description |
|
||||
|--------|-------------|
|
||||
| `--category <cat>` | Run specific test category |
|
||||
| `--module <name>` | Test specific module |
|
||||
| `--workflow <name>` | Workflow to simulate |
|
||||
| `--smoke-step <step>` | Smoke step: build, unit, unit-split |
|
||||
| `--test-timeout <t>` | Per-test timeout (e.g., 5m) using --blame-hang |
|
||||
| `--progress-interval <s>` | Progress heartbeat in seconds |
|
||||
| `--project-start <n>` | Start index (1-based) for unit-split slicing |
|
||||
| `--project-count <n>` | Limit number of projects for unit-split slicing |
|
||||
| `--docker` | Force Docker execution |
|
||||
| `--native` | Force native execution |
|
||||
| `--act` | Force act execution |
|
||||
| `--parallel <n>` | Parallel test runners |
|
||||
| `--verbose` | Verbose output |
|
||||
| `--dry-run` | Show what would run |
|
||||
| `--rebuild` | Force rebuild CI image |
|
||||
| `--no-services` | Skip CI services |
|
||||
| `--keep-services` | Keep services running after tests |
|
||||
|
||||
---
|
||||
|
||||
## Test Categories
|
||||
|
||||
### PR-Gating (Required for merge)
|
||||
|
||||
| Category | Purpose | Time |
|
||||
|----------|---------|------|
|
||||
| **Unit** | Component isolation tests | ~3 min |
|
||||
| **Architecture** | Dependency rules | ~2 min |
|
||||
| **Contract** | API compatibility | ~2 min |
|
||||
| **Integration** | Database/service tests | ~8 min |
|
||||
| **Security** | Security assertions | ~3 min |
|
||||
| **Golden** | Corpus-based validation | ~3 min |
|
||||
|
||||
### Extended (On-demand)
|
||||
|
||||
| Category | Purpose | Time |
|
||||
|----------|---------|------|
|
||||
| **Performance** | Latency/throughput | ~20 min |
|
||||
| **Benchmark** | BenchmarkDotNet | ~30 min |
|
||||
| **Determinism** | Reproducibility | ~15 min |
|
||||
| **AirGap** | Offline operation | ~15 min |
|
||||
| **Chaos** | Resilience testing | ~20 min |
|
||||
|
||||
---
|
||||
|
||||
## CI Services
|
||||
|
||||
The local CI uses Docker Compose to run required services.
|
||||
|
||||
### Services Started
|
||||
|
||||
| Service | Port | Purpose |
|
||||
|---------|------|---------|
|
||||
| postgres-ci | 5433 | PostgreSQL 16 for tests |
|
||||
| valkey-ci | 6380 | Cache/messaging tests |
|
||||
| nats-ci | 4223 | Message queue tests |
|
||||
| mock-registry | 5001 | Container registry |
|
||||
| minio-ci | 9100 | S3-compatible storage |
|
||||
|
||||
### Manual Service Management
|
||||
|
||||
```bash
|
||||
# Start services
|
||||
docker compose -f devops/compose/docker-compose.ci.yaml up -d
|
||||
|
||||
# Check status
|
||||
docker compose -f devops/compose/docker-compose.ci.yaml ps
|
||||
|
||||
# View logs
|
||||
docker compose -f devops/compose/docker-compose.ci.yaml logs postgres-ci
|
||||
|
||||
# Stop services
|
||||
docker compose -f devops/compose/docker-compose.ci.yaml down
|
||||
|
||||
# Stop and remove volumes
|
||||
docker compose -f devops/compose/docker-compose.ci.yaml down -v
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Configuration
|
||||
|
||||
### Environment Variables
|
||||
|
||||
Copy the template and customize:
|
||||
|
||||
```bash
|
||||
cp devops/ci-local/.env.local.sample devops/ci-local/.env.local
|
||||
```
|
||||
|
||||
Key variables:
|
||||
|
||||
```bash
|
||||
# Database
|
||||
STELLAOPS_TEST_POSTGRES_CONNECTION="Host=localhost;Port=5433;..."
|
||||
|
||||
# Cache
|
||||
VALKEY_CONNECTION_STRING="localhost:6380"
|
||||
|
||||
# Execution
|
||||
LOCAL_CI_MODE=docker # docker, native, act
|
||||
LOCAL_CI_PARALLEL=4 # Parallel jobs
|
||||
LOCAL_CI_VERBOSE=false # Verbose output
|
||||
```
|
||||
|
||||
### Act Configuration
|
||||
|
||||
The `.actrc` file configures `act` for workflow simulation:
|
||||
|
||||
```ini
|
||||
--platform ubuntu-22.04=stellaops-ci:local
|
||||
--env-file devops/ci-local/.env.local
|
||||
--bind
|
||||
--reuse
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Offline & Cache
|
||||
|
||||
Local CI can run in offline or rate-limited environments. These steps help ensure reliable builds.
|
||||
|
||||
### NuGet Cache Warmup
|
||||
|
||||
Before running tests offline or during rate-limited periods, warm the NuGet cache:
|
||||
|
||||
```bash
|
||||
# Warm cache with throttled requests to avoid 429 errors
|
||||
export NUGET_MAX_HTTP_REQUESTS=4
|
||||
dotnet restore src/StellaOps.sln --disable-parallel
|
||||
|
||||
# Verify cache is populated
|
||||
ls ~/.nuget/packages | wc -l
|
||||
```
|
||||
|
||||
```powershell
|
||||
# PowerShell equivalent
|
||||
$env:NUGET_MAX_HTTP_REQUESTS = "4"
|
||||
dotnet restore src\StellaOps.sln --disable-parallel
|
||||
|
||||
# Verify cache
|
||||
(Get-ChildItem "$env:USERPROFILE\.nuget\packages").Count
|
||||
```
|
||||
|
||||
### Rate Limiting Mitigation
|
||||
|
||||
If encountering NuGet 429 (Too Many Requests) errors from package sources:
|
||||
|
||||
1. **Reduce concurrency:**
|
||||
```bash
|
||||
export NUGET_MAX_HTTP_REQUESTS=2
|
||||
dotnet restore --disable-parallel
|
||||
```
|
||||
|
||||
2. **Retry off-peak hours** (avoid UTC 09:00-17:00 on weekdays)
|
||||
|
||||
3. **Use local cache fallback:**
|
||||
```bash
|
||||
# Configure fallback to local cache only (offline mode)
|
||||
dotnet restore --source ~/.nuget/packages
|
||||
```
|
||||
|
||||
### Docker Image Caching
|
||||
|
||||
Pre-pull required CI images to avoid network dependency during tests:
|
||||
|
||||
```bash
|
||||
# Pull CI services
|
||||
docker compose -f devops/compose/docker-compose.ci.yaml pull
|
||||
|
||||
# Build local CI image
|
||||
docker build -t stellaops-ci:local -f devops/docker/Dockerfile.ci .
|
||||
|
||||
# Verify images are cached
|
||||
docker images | grep -E "stellaops|postgres|valkey|nats"
|
||||
```
|
||||
|
||||
### Offline-Safe Test Execution
|
||||
|
||||
For fully offline validation:
|
||||
|
||||
```bash
|
||||
# 1. Ensure NuGet cache is warm (see above)
|
||||
# 2. Start local CI services (pre-pulled)
|
||||
docker compose -f devops/compose/docker-compose.ci.yaml up -d
|
||||
|
||||
# 3. Run smoke with no network dependency
|
||||
./devops/scripts/local-ci.sh smoke --no-restore
|
||||
|
||||
# 4. Or run specific category offline
|
||||
dotnet test src/StellaOps.sln \
|
||||
--filter "Category=Unit" \
|
||||
--no-restore \
|
||||
--no-build
|
||||
```
|
||||
|
||||
### Test Fixtures for Air-Gap
|
||||
|
||||
Some tests require fixture data that must be present locally:
|
||||
|
||||
| Fixture | Location | Purpose |
|
||||
|---------|----------|---------|
|
||||
| Golden bundles | `src/__Tests/__Datasets/EvidenceLocker/` | Evidence locker tests |
|
||||
| Reachability fixtures | `src/tests/reachability/` | Reachability drift tests |
|
||||
| Connector snapshots | `src/<Module>/__Tests/**/Fixtures/` | Connector replay tests |
|
||||
|
||||
To verify fixtures are present:
|
||||
```bash
|
||||
find src -type d -name "Fixtures" | head -20
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Docker Issues
|
||||
|
||||
```bash
|
||||
# Reset CI services
|
||||
docker compose -f devops/compose/docker-compose.ci.yaml down -v
|
||||
|
||||
# Rebuild CI image
|
||||
docker build --no-cache -t stellaops-ci:local -f devops/docker/Dockerfile.ci .
|
||||
|
||||
# Check Docker is running
|
||||
docker info
|
||||
```
|
||||
|
||||
### Test Failures
|
||||
|
||||
```bash
|
||||
# Run with verbose output
|
||||
./devops/scripts/local-ci.sh pr --verbose
|
||||
|
||||
# Run specific category
|
||||
./devops/scripts/local-ci.sh pr --category Unit
|
||||
|
||||
# Check logs
|
||||
cat out/local-ci/logs/Unit-*.log
|
||||
|
||||
# Check current test project during unit-split
|
||||
cat out/local-ci/active-test.txt
|
||||
```
|
||||
|
||||
### Act Issues
|
||||
|
||||
```bash
|
||||
# List available workflows
|
||||
act -l
|
||||
|
||||
# Dry run workflow
|
||||
act -n pull_request -W .gitea/workflows/test-matrix.yml
|
||||
|
||||
# Debug mode
|
||||
act --verbose pull_request
|
||||
```
|
||||
|
||||
### Windows Issues
|
||||
|
||||
```powershell
|
||||
# Check WSL is installed
|
||||
wsl --status
|
||||
|
||||
# Install WSL if needed
|
||||
wsl --install
|
||||
|
||||
# Check Git Bash
|
||||
& "C:\Program Files\Git\bin\bash.exe" --version
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Results
|
||||
|
||||
Test results are saved to `out/local-ci/`:
|
||||
|
||||
```
|
||||
out/local-ci/
|
||||
├── trx/ # TRX test reports
|
||||
│ ├── Unit-20250128_120000.trx
|
||||
│ ├── Integration-20250128_120000.trx
|
||||
│ └── ...
|
||||
└── logs/ # Execution logs
|
||||
├── Unit-20250128_120000.log
|
||||
└── ...
|
||||
```
|
||||
|
||||
### Viewing Results
|
||||
|
||||
```bash
|
||||
# List TRX files
|
||||
ls -la out/local-ci/trx/
|
||||
|
||||
# View test log
|
||||
cat out/local-ci/logs/Unit-*.log | less
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Examples
|
||||
|
||||
### Pre-Push Workflow
|
||||
|
||||
```bash
|
||||
# 1. Quick validation
|
||||
./devops/scripts/local-ci.sh smoke
|
||||
|
||||
# 2. If smoke passes, run full PR check
|
||||
./devops/scripts/local-ci.sh pr
|
||||
|
||||
# 3. Push your changes
|
||||
git push origin feature/my-branch
|
||||
```
|
||||
|
||||
### Module Development
|
||||
|
||||
```bash
|
||||
# 1. Make changes to Scanner module
|
||||
|
||||
# 2. Run module-specific tests
|
||||
./devops/scripts/local-ci.sh module --module Scanner
|
||||
|
||||
# 3. If passes, run full PR check before PR
|
||||
./devops/scripts/local-ci.sh pr
|
||||
```
|
||||
|
||||
### Debugging Test Failures
|
||||
|
||||
```bash
|
||||
# 1. Run with verbose output
|
||||
./devops/scripts/local-ci.sh pr --verbose --category Unit
|
||||
|
||||
# 2. Check the log
|
||||
cat out/local-ci/logs/Unit-*.log
|
||||
|
||||
# 3. Run specific test directly
|
||||
dotnet test src/Scanner/__Tests/StellaOps.Scanner.Tests/StellaOps.Scanner.Tests.csproj \
|
||||
--filter "Category=Unit" \
|
||||
--verbosity detailed
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Related Documentation
|
||||
|
||||
- [CI/CD Overview](../cicd/README.md)
|
||||
- [Test Strategy](../cicd/test-strategy.md)
|
||||
- [Workflow Triggers](../cicd/workflow-triggers.md)
|
||||
- [Path Filters](../cicd/path-filters.md)
|
||||
Reference in New Issue
Block a user