git.stella-ops.org/docs/cicd/test-strategy.md

Test Strategy

Complete guide to the StellaOps testing strategy and CI/CD integration.

---

Test Category Overview

StellaOps uses a tiered testing strategy with 13 test categories:

PR-Gating Tests (Required for Merge)

Category	Purpose	Timeout	Parallelism
Unit	Isolated component tests	20 min	High
Architecture	Dependency rule enforcement	15 min	High
Contract	API compatibility	15 min	High
Integration	Database/service integration	45 min	Medium
Security	Security-focused assertions	25 min	High
Golden	Corpus-based validation	25 min	High

Extended Tests (Scheduled/On-Demand)

Category	Purpose	Timeout	Trigger
Performance	Latency/throughput benchmarks	45 min	Daily, Manual
Benchmark	BenchmarkDotNet profiling	60 min	Daily, Manual
AirGap	Offline operation validation	45 min	Manual
Chaos	Resilience testing	45 min	Manual
Determinism	Reproducibility verification	45 min	Manual
Resilience	Failure recovery testing	45 min	Manual
Observability	Telemetry validation	30 min	Manual

---

Test Discovery

Automatic Discovery

The test-matrix.yml workflow automatically discovers all test projects:

# Discovery pattern
find src \( \
  -name "*.Tests.csproj" \
  -o -name "*UnitTests.csproj" \
  -o -name "*SmokeTests.csproj" \
  -o -name "*FixtureTests.csproj" \
  -o -name "*IntegrationTests.csproj" \
\) -type f \
  ! -path "*/node_modules/*" \
  ! -path "*/bin/*" \
  ! -path "*/obj/*" \
  ! -name "StellaOps.TestKit.csproj" \
  ! -name "*Testing.csproj"

Test Category Trait

Tests are categorized using xUnit traits:

[Trait("Category", "Unit")]
public class MyUnitTests
{
    [Fact]
    public void Should_Do_Something()
    {
        // ...
    }
}

[Trait("Category", "Integration")]
public class MyIntegrationTests
{
    [Fact]
    public void Should_Connect_To_Database()
    {
        // ...
    }
}

Running Specific Categories

# Run Unit tests only
dotnet test --filter "Category=Unit"

# Run multiple categories
dotnet test --filter "Category=Unit|Category=Integration"

# Run excluding a category
dotnet test --filter "Category!=Performance"

---

Test Infrastructure

Shared Libraries

Library	Purpose	Location
StellaOps.TestKit	Common test utilities	src/__Tests/__Libraries/
StellaOps.Infrastructure.Postgres.Testing	PostgreSQL fixtures	src/__Tests/__Libraries/
StellaOps.Concelier.Testing	Concelier test fixtures	src/Concelier/__Tests/

Testcontainers

Integration tests use Testcontainers for isolated dependencies:

public class PostgresFixture : IAsyncLifetime
{
    private readonly PostgreSqlContainer _container = new PostgreSqlBuilder()
        .WithImage("postgres:16")
        .WithDatabase("test_db")
        .Build();

    public string ConnectionString => _container.GetConnectionString();

    public Task InitializeAsync() => _container.StartAsync();
    public Task DisposeAsync() => _container.DisposeAsync().AsTask();
}

Ground Truth Corpus

Golden tests use a corpus of known-good outputs:

src/__Tests/__Datasets/
├── scanner/
│   ├── golden/
│   │   ├── npm-package.expected.json
│   │   ├── dotnet-project.expected.json
│   │   └── container-image.expected.json
│   └── fixtures/
│       ├── npm-package/
│       ├── dotnet-project/
│       └── container-image/
└── concelier/
    ├── golden/
    └── fixtures/

---

CI/CD Integration

Test Matrix Workflow

# Simplified test-matrix.yml structure
jobs:
  discover:
    # Find all test projects
    outputs:
      test-projects: ${{ steps.find.outputs.projects }}

  pr-gating-tests:
    strategy:
      matrix:
        include:
          - category: Unit
            timeout: 20
          - category: Architecture
            timeout: 15
          - category: Contract
            timeout: 15
          - category: Security
            timeout: 25
          - category: Golden
            timeout: 25
    steps:
      - run: .gitea/scripts/test/run-test-category.sh "${{ matrix.category }}"

  integration:
    services:
      postgres:
        image: postgres:16
    steps:
      - run: .gitea/scripts/test/run-test-category.sh Integration

  summary:
    needs: [discover, pr-gating-tests, integration]

Test Results

All tests produce TRX (Visual Studio Test Results) files:

# Output structure
TestResults/
├── Unit/
│   ├── src_Scanner___Tests_StellaOps.Scanner.Tests-unit.trx
│   └── src_Authority___Tests_StellaOps.Authority.Tests-unit.trx
├── Integration/
│   └── ...
└── Combined/
    └── test-results-combined.trx

Coverage Collection

# Collect coverage for Unit tests
- run: |
    .gitea/scripts/test/run-test-category.sh Unit --collect-coverage

Coverage reports are generated in Cobertura format and converted to HTML.

---

Test Categories Deep Dive

Unit Tests

Purpose: Test isolated components without external dependencies.

Characteristics:

• No I/O (database, network, file system)
• No async waits or delays
• Fast execution (< 100ms per test)
• High parallelism

Example:

[Trait("Category", "Unit")]
public class VexPolicyBinderTests
{
    [Fact]
    public void Bind_WithValidPolicy_ReturnsSuccess()
    {
        var binder = new VexPolicyBinder();
        var policy = new VexPolicy { /* ... */ };

        var result = binder.Bind(policy);

        Assert.True(result.IsSuccess);
    }
}

Architecture Tests

Purpose: Enforce architectural rules and dependency constraints.

Rules Enforced:

• Layer dependencies (UI → Application → Domain → Infrastructure)
• Namespace conventions
• Circular dependency prevention
• Interface segregation

Example:

[Trait("Category", "Architecture")]
public class DependencyTests
{
    [Fact]
    public void Domain_Should_Not_Depend_On_Infrastructure()
    {
        var result = Types.InAssembly(typeof(DomainMarker).Assembly)
            .That().ResideInNamespace("StellaOps.Domain")
            .ShouldNot().HaveDependencyOn("StellaOps.Infrastructure")
            .GetResult();

        Assert.True(result.IsSuccessful);
    }
}

Contract Tests

Purpose: Validate API contracts are maintained.

Checks:

• Request/response schemas
• OpenAPI specification compliance
• Backward compatibility

Example:

[Trait("Category", "Contract")]
public class VulnerabilityApiContractTests
{
    [Fact]
    public async Task GetVulnerability_ReturnsExpectedSchema()
    {
        var response = await _client.GetAsync("/api/v1/vulnerabilities/CVE-2024-1234");

        await Verify(response)
            .UseDirectory("Snapshots")
            .UseMethodName("GetVulnerability");
    }
}

Integration Tests

Purpose: Test component integration with real dependencies.

Dependencies:

• PostgreSQL (via Testcontainers)
• Valkey/Redis (via Testcontainers)
• File system

Example:

[Trait("Category", "Integration")]
public class VulnerabilityRepositoryTests : IClassFixture<PostgresFixture>
{
    private readonly PostgresFixture _fixture;

    [Fact]
    public async Task Save_AndRetrieve_Vulnerability()
    {
        var repo = new VulnerabilityRepository(_fixture.ConnectionString);
        var vuln = new Vulnerability { Id = "CVE-2024-1234" };

        await repo.SaveAsync(vuln);
        var retrieved = await repo.GetAsync("CVE-2024-1234");

        Assert.Equal(vuln.Id, retrieved.Id);
    }
}

Security Tests

Purpose: Validate security controls and assertions.

Checks:

• Input validation
• Authorization enforcement
• Cryptographic operations
• Secrets handling

Example:

[Trait("Category", "Security")]
public class AuthorizationTests
{
    [Fact]
    public async Task Unauthorized_User_Cannot_Access_Admin_Endpoint()
    {
        var client = _factory.CreateClient();
        client.DefaultRequestHeaders.Authorization = null;

        var response = await client.GetAsync("/api/admin/settings");

        Assert.Equal(HttpStatusCode.Unauthorized, response.StatusCode);
    }
}

Golden Tests

Purpose: Verify output matches known-good corpus.

Example:

[Trait("Category", "Golden")]
public class ScannerGoldenTests
{
    [Theory]
    [InlineData("npm-package")]
    [InlineData("dotnet-project")]
    public async Task Scan_MatchesGoldenOutput(string fixture)
    {
        var scanner = new ContainerScanner();
        var result = await scanner.ScanAsync($"fixtures/{fixture}");

        await Verify(result)
            .UseDirectory("golden")
            .UseFileName(fixture);
    }
}

---

Performance Testing

BenchmarkDotNet

[Trait("Category", "Benchmark")]
[MemoryDiagnoser]
public class ScannerBenchmarks
{
    [Benchmark]
    public async Task ScanSmallImage()
    {
        await _scanner.ScanAsync(_smallImage);
    }

    [Benchmark]
    public async Task ScanLargeImage()
    {
        await _scanner.ScanAsync(_largeImage);
    }
}

Performance SLOs

Metric	Target	Action on Breach
Unit test P95	< 100ms	Warning
Integration test P95	< 5s	Warning
Scan time P95	< 5 min	Block
Memory peak	< 2GB	Block

---

Troubleshooting

Tests Fail in CI but Pass Locally

1. Check timezone - CI uses TZ=UTC
2. Check parallelism - CI runs tests in parallel
3. Check container availability - Testcontainers requires Docker
4. Check file paths - Case sensitivity on Linux

Flaky Tests

1. Add retry logic for network operations
2. Use proper async/await - no Task.Run for async operations
3. Isolate shared state - use fresh fixtures per test
4. Increase timeouts for integration tests

Missing Test Category

Ensure your test class has the correct trait:

[Trait("Category", "Unit")]  // Add this!
public class MyTests
{
    // ...
}

---

Related Documentation

• README - CI/CD Overview
• Workflow Triggers
• CI Quality Gates
• Test Catalog