# Local CI Testing Guide

> Run CI/CD pipelines locally before pushing to verify your changes.

---

## Quick Start

### Prerequisites

- **Docker Desktop** (Windows/macOS) or Docker Engine (Linux)
- **.NET 10 SDK** ([download](https://dot.net/download))
- **Git**
- **Bash** (Linux/macOS native, Windows via WSL2 or Git Bash)
- **act** (optional, for workflow simulation) - [install](https://github.com/nektos/act)

### Installation

1. **Copy environment template:**
   ```bash
   cp devops/ci-local/.env.local.sample devops/ci-local/.env.local
   ```

2. **(Optional) Install act for workflow simulation:**
   ```bash
   # macOS
   brew install act

   # Linux
   curl -sSL https://raw.githubusercontent.com/nektos/act/master/install.sh | sudo bash

   # Windows (via Chocolatey)
   choco install act-cli
   ```

3. **(Optional) Build CI Docker image:**
   ```bash
   docker build -t stellaops-ci:local -f devops/docker/Dockerfile.ci .
   ```

---

## Usage

### Basic Commands

```bash
# Quick smoke test (~2 min)
./devops/scripts/local-ci.sh smoke

# Smoke steps (isolate build vs unit tests)
./devops/scripts/local-ci.sh smoke --smoke-step build
./devops/scripts/local-ci.sh smoke --smoke-step unit
./devops/scripts/local-ci.sh smoke --smoke-step unit-split
./devops/scripts/local-ci.sh smoke --smoke-step unit-split --test-timeout 5m --progress-interval 60
./devops/scripts/local-ci.sh smoke --smoke-step unit-split --project-start 1 --project-count 50

# Full PR-gating suite (~15 min)
./devops/scripts/local-ci.sh pr

# Test specific module
./devops/scripts/local-ci.sh module --module Scanner

# Auto-detect changed modules
./devops/scripts/local-ci.sh module

# Simulate workflow
./devops/scripts/local-ci.sh workflow --workflow test-matrix

# Release dry-run
./devops/scripts/local-ci.sh release --dry-run

# Full test suite (~45 min)
./devops/scripts/local-ci.sh full
```

### Windows (PowerShell)

```powershell
# Quick smoke test
.\devops\scripts\local-ci.ps1 smoke

# Smoke steps (isolate build vs unit tests)
.\devops\scripts\local-ci.ps1 smoke -SmokeStep build
.\devops\scripts\local-ci.ps1 smoke -SmokeStep unit
.\devops\scripts\local-ci.ps1 smoke -SmokeStep unit-split
.\devops\scripts\local-ci.ps1 smoke -SmokeStep unit-split -TestTimeout 5m -ProgressInterval 60
.\devops\scripts\local-ci.ps1 smoke -SmokeStep unit-split -ProjectStart 1 -ProjectCount 50

# Full PR check
.\devops\scripts\local-ci.ps1 pr

# With options
.\devops\scripts\local-ci.ps1 pr -Verbose -Docker
```

---

## Modes

### smoke (~2 minutes)
Quick validation before pushing. Runs only Unit tests.

```bash
./devops/scripts/local-ci.sh smoke
```

Optional stepwise smoke (to isolate hangs):

```bash
./devops/scripts/local-ci.sh smoke --smoke-step build
./devops/scripts/local-ci.sh smoke --smoke-step unit
./devops/scripts/local-ci.sh smoke --smoke-step unit-split
```

**What it does:**
1. Builds the solution
2. Runs Unit tests
3. Reports pass/fail

### pr (~15 minutes)
Full PR-gating suite matching CI exactly. Run this before opening a PR.

```bash
./devops/scripts/local-ci.sh pr
```

**What it does:**
1. Starts CI services (PostgreSQL, Valkey)
2. Builds with warnings-as-errors
3. Runs all PR-gating categories:
   - Unit
   - Architecture
   - Contract
   - Integration
   - Security
   - Golden
4. Generates TRX reports
5. Stops CI services

### module
Test only the modules you've changed. Detects changes via git diff.

```bash
# Auto-detect changed modules
./devops/scripts/local-ci.sh module

# Test specific module
./devops/scripts/local-ci.sh module --module Scanner
./devops/scripts/local-ci.sh module --module Concelier
./devops/scripts/local-ci.sh module --module Authority
```

**Available modules:**
Scanner, Concelier, Authority, Policy, Attestor, EvidenceLocker, ExportCenter,
Findings, SbomService, Notify, Router, Cryptography, AirGap, Cli, AdvisoryAI,
ReachGraph, Orchestrator, PacksRegistry, Replay, Aoc, IssuerDirectory, Telemetry, Signals

### workflow
Simulate a specific Gitea Actions workflow using `act`.

```bash
# Simulate test-matrix workflow
./devops/scripts/local-ci.sh workflow --workflow test-matrix

# Simulate build-test-deploy
./devops/scripts/local-ci.sh workflow --workflow build-test-deploy
```

**Requirements:**
- `act` must be installed
- CI Docker image built (`stellaops-ci:local`)

### release
Dry-run release pipeline to verify release artifacts.

```bash
./devops/scripts/local-ci.sh release --dry-run
```

**What it does:**
1. Builds all modules
2. Validates CLI packaging
3. Validates Helm chart
4. Shows what would be released
5. **Does NOT publish anything**

### full (~45 minutes)
Complete test suite including extended categories.

```bash
./devops/scripts/local-ci.sh full
```

**Categories run:**
- PR-Gating: Unit, Architecture, Contract, Integration, Security, Golden
- Extended: Performance, Benchmark, AirGap, Chaos, Determinism, Resilience, Observability

---

## Options

| Option | Description |
|--------|-------------|
| `--category <cat>` | Run specific test category |
| `--module <name>` | Test specific module |
| `--workflow <name>` | Workflow to simulate |
| `--smoke-step <step>` | Smoke step: build, unit, unit-split |
| `--test-timeout <t>` | Per-test timeout (e.g., 5m) using --blame-hang |
| `--progress-interval <s>` | Progress heartbeat in seconds |
| `--project-start <n>` | Start index (1-based) for unit-split slicing |
| `--project-count <n>` | Limit number of projects for unit-split slicing |
| `--docker` | Force Docker execution |
| `--native` | Force native execution |
| `--act` | Force act execution |
| `--parallel <n>` | Parallel test runners |
| `--verbose` | Verbose output |
| `--dry-run` | Show what would run |
| `--rebuild` | Force rebuild CI image |
| `--no-services` | Skip CI services |
| `--keep-services` | Keep services running after tests |

---

## Test Categories

### PR-Gating (Required for merge)

| Category | Purpose | Time |
|----------|---------|------|
| **Unit** | Component isolation tests | ~3 min |
| **Architecture** | Dependency rules | ~2 min |
| **Contract** | API compatibility | ~2 min |
| **Integration** | Database/service tests | ~8 min |
| **Security** | Security assertions | ~3 min |
| **Golden** | Corpus-based validation | ~3 min |

### Extended (On-demand)

| Category | Purpose | Time |
|----------|---------|------|
| **Performance** | Latency/throughput | ~20 min |
| **Benchmark** | BenchmarkDotNet | ~30 min |
| **Determinism** | Reproducibility | ~15 min |
| **AirGap** | Offline operation | ~15 min |
| **Chaos** | Resilience testing | ~20 min |

---

## CI Services

The local CI uses Docker Compose to run required services.

### Services Started

| Service | Port | Purpose |
|---------|------|---------|
| postgres-test | 5433 | PostgreSQL 18 for tests |
| valkey-test | 6380 | Cache/messaging tests |
| rustfs-test | 8180 | S3-compatible storage |
| mock-registry | 5001 | Container registry |

### Manual Service Management

```bash
# Start services
docker compose -f devops/compose/docker-compose.testing.yml --profile ci up -d

# Check status
docker compose -f devops/compose/docker-compose.testing.yml --profile ci ps

# View logs
docker compose -f devops/compose/docker-compose.testing.yml logs postgres-test

# Stop services
docker compose -f devops/compose/docker-compose.testing.yml --profile ci down

# Stop and remove volumes
docker compose -f devops/compose/docker-compose.testing.yml --profile ci down -v
```

---

## Configuration

### Environment Variables

Copy the template and customize:

```bash
cp devops/ci-local/.env.local.sample devops/ci-local/.env.local
```

Key variables:

```bash
# Database
STELLAOPS_TEST_POSTGRES_CONNECTION="Host=localhost;Port=5433;..."

# Cache
VALKEY_CONNECTION_STRING="localhost:6380"

# Execution
LOCAL_CI_MODE=docker     # docker, native, act
LOCAL_CI_PARALLEL=4      # Parallel jobs
LOCAL_CI_VERBOSE=false   # Verbose output
```

### Act Configuration

The `.actrc` file configures `act` for workflow simulation:

```ini
--platform ubuntu-22.04=stellaops-ci:local
--env-file devops/ci-local/.env.local
--bind
--reuse
```

---

## Offline & Cache

Local CI can run in offline or rate-limited environments. These steps help ensure reliable builds.

### NuGet Cache Warmup

Before running tests offline or during rate-limited periods, warm the NuGet cache:

```bash
# Warm cache with throttled requests to avoid 429 errors
export NUGET_MAX_HTTP_REQUESTS=4
dotnet restore src/<Module>/StellaOps.<Module>.sln --disable-parallel

# Verify cache is populated
ls ~/.nuget/packages | wc -l
```

```powershell
# PowerShell equivalent
$env:NUGET_MAX_HTTP_REQUESTS = "4"
dotnet restore src\<Module>\StellaOps.<Module>.sln --disable-parallel

# Verify cache
(Get-ChildItem "$env:USERPROFILE\.nuget\packages").Count
```

See docs/dev/SOLUTION_BUILD_GUIDE.md for the module solution list.

### Rate Limiting Mitigation

If encountering NuGet 429 (Too Many Requests) errors from package sources:

1. **Reduce concurrency:**
   ```bash
   export NUGET_MAX_HTTP_REQUESTS=2
   dotnet restore --disable-parallel
   ```

2. **Retry off-peak hours** (avoid UTC 09:00-17:00 on weekdays)

3. **Use local cache fallback:**
   ```bash
   # Configure fallback to local cache only (offline mode)
   dotnet restore --source ~/.nuget/packages
   ```

### Docker Image Caching

Pre-pull required CI images to avoid network dependency during tests:

```bash
# Pull CI services
docker compose -f devops/compose/docker-compose.testing.yml --profile ci pull

# Build local CI image
docker build -t stellaops-ci:local -f devops/docker/Dockerfile.ci .

# Verify images are cached
docker images | grep -E "stellaops|postgres|valkey|rustfs"
```

### Offline-Safe Test Execution

For fully offline validation:

```bash
# 1. Ensure NuGet cache is warm (see above)
# 2. Start local CI services (pre-pulled)
docker compose -f devops/compose/docker-compose.testing.yml --profile ci up -d

# 3. Run smoke with no network dependency
./devops/scripts/local-ci.sh smoke --no-restore

# 4. Or run specific category offline
dotnet test src/<Module>/StellaOps.<Module>.sln \
  --filter "Category=Unit" \
  --no-restore \
  --no-build
```

### Test Fixtures for Air-Gap

Some tests require fixture data that must be present locally:

| Fixture | Location | Purpose |
|---------|----------|---------|
| Golden bundles | `src/__Tests/__Datasets/EvidenceLocker/` | Evidence locker tests |
| Reachability fixtures | `src/tests/reachability/` | Reachability drift tests |
| Connector snapshots | `src/<Module>/__Tests/**/Fixtures/` | Connector replay tests |

To verify fixtures are present:
```bash
find src -type d -name "Fixtures" | head -20
```

---

## Troubleshooting

### Docker Issues

```bash
# Reset CI services
docker compose -f devops/compose/docker-compose.testing.yml --profile ci down -v

# Rebuild CI image
docker build --no-cache -t stellaops-ci:local -f devops/docker/Dockerfile.ci .

# Check Docker is running
docker info
```

### Test Failures

```bash
# Run with verbose output
./devops/scripts/local-ci.sh pr --verbose

# Run specific category
./devops/scripts/local-ci.sh pr --category Unit

# Check logs
cat out/local-ci/logs/Unit-*.log

# Check current test project during unit-split
cat out/local-ci/active-test.txt
```

### Act Issues

```bash
# List available workflows
act -l

# Dry run workflow
act -n pull_request -W .gitea/workflows/test-matrix.yml

# Debug mode
act --verbose pull_request
```

### Windows Issues

```powershell
# Check WSL is installed
wsl --status

# Install WSL if needed
wsl --install

# Check Git Bash
& "C:\Program Files\Git\bin\bash.exe" --version
```

---

## Results

Test results are saved to `out/local-ci/`:

```
out/local-ci/
├── trx/                    # TRX test reports
│   ├── Unit-20250128_120000.trx
│   ├── Integration-20250128_120000.trx
│   └── ...
└── logs/                   # Execution logs
    ├── Unit-20250128_120000.log
    └── ...
```

### Viewing Results

```bash
# List TRX files
ls -la out/local-ci/trx/

# View test log
cat out/local-ci/logs/Unit-*.log | less
```

---

## Examples

### Pre-Push Workflow

```bash
# 1. Quick validation
./devops/scripts/local-ci.sh smoke

# 2. If smoke passes, run full PR check
./devops/scripts/local-ci.sh pr

# 3. Push your changes
git push origin feature/my-branch
```

### Module Development

```bash
# 1. Make changes to Scanner module

# 2. Run module-specific tests
./devops/scripts/local-ci.sh module --module Scanner

# 3. If passes, run full PR check before PR
./devops/scripts/local-ci.sh pr
```

### Debugging Test Failures

```bash
# 1. Run with verbose output
./devops/scripts/local-ci.sh pr --verbose --category Unit

# 2. Check the log
cat out/local-ci/logs/Unit-*.log

# 3. Run specific test directly
dotnet test src/Scanner/__Tests/StellaOps.Scanner.Tests/StellaOps.Scanner.Tests.csproj \
  --filter "Category=Unit" \
  --verbosity detailed
```

---

## Related Documentation

- [CI/CD Overview](../cicd/README.md)
- [Test Strategy](../cicd/test-strategy.md)
- [Workflow Triggers](../cicd/workflow-triggers.md)
- [Path Filters](../cicd/path-filters.md)