# Test Strategy

> Complete guide to the StellaOps testing strategy and CI/CD integration.

---

## Test Category Overview

StellaOps uses a **tiered testing strategy** with 13 test categories:

### PR-Gating Tests (Required for Merge)

| Category | Purpose | Timeout | Parallelism |
|----------|---------|---------|-------------|
| **Unit** | Isolated component tests | 20 min | High |
| **Architecture** | Dependency rule enforcement | 15 min | High |
| **Contract** | API compatibility | 15 min | High |
| **Integration** | Database/service integration | 45 min | Medium |
| **Security** | Security-focused assertions | 25 min | High |
| **Golden** | Corpus-based validation | 25 min | High |

### Extended Tests (Scheduled/On-Demand)

| Category | Purpose | Timeout | Trigger |
|----------|---------|---------|---------|
| **Performance** | Latency/throughput benchmarks | 45 min | Daily, Manual |
| **Benchmark** | BenchmarkDotNet profiling | 60 min | Daily, Manual |
| **AirGap** | Offline operation validation | 45 min | Manual |
| **Chaos** | Resilience testing | 45 min | Manual |
| **Determinism** | Reproducibility verification | 45 min | Manual |
| **Resilience** | Failure recovery testing | 45 min | Manual |
| **Observability** | Telemetry validation | 30 min | Manual |

---

## Test Discovery

### Automatic Discovery

The `test-matrix.yml` workflow automatically discovers all test projects:

```bash
# Discovery pattern
find src \( \
  -name "*.Tests.csproj" \
  -o -name "*UnitTests.csproj" \
  -o -name "*SmokeTests.csproj" \
  -o -name "*FixtureTests.csproj" \
  -o -name "*IntegrationTests.csproj" \
\) -type f \
  ! -path "*/node_modules/*" \
  ! -path "*/bin/*" \
  ! -path "*/obj/*" \
  ! -name "StellaOps.TestKit.csproj" \
  ! -name "*Testing.csproj"
```

### Test Category Trait

Tests are categorized using xUnit traits:

```csharp
[Trait("Category", "Unit")]
public class MyUnitTests
{
    [Fact]
    public void Should_Do_Something()
    {
        // ...
    }
}

[Trait("Category", "Integration")]
public class MyIntegrationTests
{
    [Fact]
    public void Should_Connect_To_Database()
    {
        // ...
    }
}
```

### Running Specific Categories

```bash
# Run Unit tests only
dotnet test --filter "Category=Unit"

# Run multiple categories
dotnet test --filter "Category=Unit|Category=Integration"

# Run excluding a category
dotnet test --filter "Category!=Performance"
```

---

## Test Infrastructure

### Shared Libraries

| Library | Purpose | Location |
|---------|---------|----------|
| `StellaOps.TestKit` | Common test utilities | `src/__Tests/__Libraries/` |
| `StellaOps.Infrastructure.Postgres.Testing` | PostgreSQL fixtures | `src/__Tests/__Libraries/` |
| `StellaOps.Concelier.Testing` | Concelier test fixtures | `src/Concelier/__Tests/` |

### Testcontainers

Integration tests use Testcontainers for isolated dependencies:

```csharp
public class PostgresFixture : IAsyncLifetime
{
    private readonly PostgreSqlContainer _container = new PostgreSqlBuilder()
        .WithImage("postgres:16")
        .WithDatabase("test_db")
        .Build();

    public string ConnectionString => _container.GetConnectionString();

    public Task InitializeAsync() => _container.StartAsync();
    public Task DisposeAsync() => _container.DisposeAsync().AsTask();
}
```

### Ground Truth Corpus

Golden tests use a corpus of known-good outputs:

```
src/__Tests/__Datasets/
├── scanner/
│   ├── golden/
│   │   ├── npm-package.expected.json
│   │   ├── dotnet-project.expected.json
│   │   └── container-image.expected.json
│   └── fixtures/
│       ├── npm-package/
│       ├── dotnet-project/
│       └── container-image/
└── concelier/
    ├── golden/
    └── fixtures/
```

---

## CI/CD Integration

### Test Matrix Workflow

```yaml
# Simplified test-matrix.yml structure
jobs:
  discover:
    # Find all test projects
    outputs:
      test-projects: ${{ steps.find.outputs.projects }}

  pr-gating-tests:
    strategy:
      matrix:
        include:
          - category: Unit
            timeout: 20
          - category: Architecture
            timeout: 15
          - category: Contract
            timeout: 15
          - category: Security
            timeout: 25
          - category: Golden
            timeout: 25
    steps:
      - run: .gitea/scripts/test/run-test-category.sh "${{ matrix.category }}"

  integration:
    services:
      postgres:
        image: postgres:16
    steps:
      - run: .gitea/scripts/test/run-test-category.sh Integration

  summary:
    needs: [discover, pr-gating-tests, integration]
```

### Test Results

All tests produce TRX (Visual Studio Test Results) files:

```bash
# Output structure
TestResults/
├── Unit/
│   ├── src_Scanner___Tests_StellaOps.Scanner.Tests-unit.trx
│   └── src_Authority___Tests_StellaOps.Authority.Tests-unit.trx
├── Integration/
│   └── ...
└── Combined/
    └── test-results-combined.trx
```

### Coverage Collection

```yaml
# Collect coverage for Unit tests
- run: |
    .gitea/scripts/test/run-test-category.sh Unit --collect-coverage
```

Coverage reports are generated in Cobertura format and converted to HTML.

---

## Test Categories Deep Dive

### Unit Tests

**Purpose:** Test isolated components without external dependencies.

**Characteristics:**
- No I/O (database, network, file system)
- No async waits or delays
- Fast execution (< 100ms per test)
- High parallelism

**Example:**

```csharp
[Trait("Category", "Unit")]
public class VexPolicyBinderTests
{
    [Fact]
    public void Bind_WithValidPolicy_ReturnsSuccess()
    {
        var binder = new VexPolicyBinder();
        var policy = new VexPolicy { /* ... */ };

        var result = binder.Bind(policy);

        Assert.True(result.IsSuccess);
    }
}
```

### Architecture Tests

**Purpose:** Enforce architectural rules and dependency constraints.

**Rules Enforced:**
- Layer dependencies (UI → Application → Domain → Infrastructure)
- Namespace conventions
- Circular dependency prevention
- Interface segregation

**Example:**

```csharp
[Trait("Category", "Architecture")]
public class DependencyTests
{
    [Fact]
    public void Domain_Should_Not_Depend_On_Infrastructure()
    {
        var result = Types.InAssembly(typeof(DomainMarker).Assembly)
            .That().ResideInNamespace("StellaOps.Domain")
            .ShouldNot().HaveDependencyOn("StellaOps.Infrastructure")
            .GetResult();

        Assert.True(result.IsSuccessful);
    }
}
```

### Contract Tests

**Purpose:** Validate API contracts are maintained.

**Checks:**
- Request/response schemas
- OpenAPI specification compliance
- Backward compatibility

**Example:**

```csharp
[Trait("Category", "Contract")]
public class VulnerabilityApiContractTests
{
    [Fact]
    public async Task GetVulnerability_ReturnsExpectedSchema()
    {
        var response = await _client.GetAsync("/api/v1/vulnerabilities/CVE-2024-1234");

        await Verify(response)
            .UseDirectory("Snapshots")
            .UseMethodName("GetVulnerability");
    }
}
```

### Integration Tests

**Purpose:** Test component integration with real dependencies.

**Dependencies:**
- PostgreSQL (via Testcontainers)
- Valkey/Redis (via Testcontainers)
- File system

**Example:**

```csharp
[Trait("Category", "Integration")]
public class VulnerabilityRepositoryTests : IClassFixture<PostgresFixture>
{
    private readonly PostgresFixture _fixture;

    [Fact]
    public async Task Save_AndRetrieve_Vulnerability()
    {
        var repo = new VulnerabilityRepository(_fixture.ConnectionString);
        var vuln = new Vulnerability { Id = "CVE-2024-1234" };

        await repo.SaveAsync(vuln);
        var retrieved = await repo.GetAsync("CVE-2024-1234");

        Assert.Equal(vuln.Id, retrieved.Id);
    }
}
```

### Security Tests

**Purpose:** Validate security controls and assertions.

**Checks:**
- Input validation
- Authorization enforcement
- Cryptographic operations
- Secrets handling

**Example:**

```csharp
[Trait("Category", "Security")]
public class AuthorizationTests
{
    [Fact]
    public async Task Unauthorized_User_Cannot_Access_Admin_Endpoint()
    {
        var client = _factory.CreateClient();
        client.DefaultRequestHeaders.Authorization = null;

        var response = await client.GetAsync("/api/admin/settings");

        Assert.Equal(HttpStatusCode.Unauthorized, response.StatusCode);
    }
}
```

### Golden Tests

**Purpose:** Verify output matches known-good corpus.

**Example:**

```csharp
[Trait("Category", "Golden")]
public class ScannerGoldenTests
{
    [Theory]
    [InlineData("npm-package")]
    [InlineData("dotnet-project")]
    public async Task Scan_MatchesGoldenOutput(string fixture)
    {
        var scanner = new ContainerScanner();
        var result = await scanner.ScanAsync($"fixtures/{fixture}");

        await Verify(result)
            .UseDirectory("golden")
            .UseFileName(fixture);
    }
}
```

---

## Performance Testing

### BenchmarkDotNet

```csharp
[Trait("Category", "Benchmark")]
[MemoryDiagnoser]
public class ScannerBenchmarks
{
    [Benchmark]
    public async Task ScanSmallImage()
    {
        await _scanner.ScanAsync(_smallImage);
    }

    [Benchmark]
    public async Task ScanLargeImage()
    {
        await _scanner.ScanAsync(_largeImage);
    }
}
```

### Performance SLOs

| Metric | Target | Action on Breach |
|--------|--------|------------------|
| Unit test P95 | < 100ms | Warning |
| Integration test P95 | < 5s | Warning |
| Scan time P95 | < 5 min | Block |
| Memory peak | < 2GB | Block |

---

## Troubleshooting

### Tests Fail in CI but Pass Locally

1. **Check timezone** - CI uses `TZ=UTC`
2. **Check parallelism** - CI runs tests in parallel
3. **Check container availability** - Testcontainers requires Docker
4. **Check file paths** - Case sensitivity on Linux

### Flaky Tests

1. **Add retry logic** for network operations
2. **Use proper async/await** - no `Task.Run` for async operations
3. **Isolate shared state** - use fresh fixtures per test
4. **Increase timeouts** for integration tests

### Missing Test Category

Ensure your test class has the correct trait:

```csharp
[Trait("Category", "Unit")]  // Add this!
public class MyTests
{
    // ...
}
```

---

## Related Documentation

- [README - CI/CD Overview](./README.md)
- [Workflow Triggers](./workflow-triggers.md)
- [CI Quality Gates](../testing/ci-quality-gates.md)
- [Test Catalog](../testing/TEST_CATALOG.yml)