462 lines
10 KiB
Markdown
462 lines
10 KiB
Markdown
# Test Strategy
|
|
|
|
> Complete guide to the StellaOps testing strategy and CI/CD integration.
|
|
|
|
---
|
|
|
|
## Test Category Overview
|
|
|
|
StellaOps uses a **tiered testing strategy** with 13 test categories:
|
|
|
|
### PR-Gating Tests (Required for Merge)
|
|
|
|
| Category | Purpose | Timeout | Parallelism |
|
|
|----------|---------|---------|-------------|
|
|
| **Unit** | Isolated component tests | 20 min | High |
|
|
| **Architecture** | Dependency rule enforcement | 15 min | High |
|
|
| **Contract** | API compatibility | 15 min | High |
|
|
| **Integration** | Database/service integration | 45 min | Medium |
|
|
| **Security** | Security-focused assertions | 25 min | High |
|
|
| **Golden** | Corpus-based validation | 25 min | High |
|
|
|
|
### Extended Tests (Scheduled/On-Demand)
|
|
|
|
| Category | Purpose | Timeout | Trigger |
|
|
|----------|---------|---------|---------|
|
|
| **Performance** | Latency/throughput benchmarks | 45 min | Daily, Manual |
|
|
| **Benchmark** | BenchmarkDotNet profiling | 60 min | Daily, Manual |
|
|
| **AirGap** | Offline operation validation | 45 min | Manual |
|
|
| **Chaos** | Resilience testing | 45 min | Manual |
|
|
| **Determinism** | Reproducibility verification | 45 min | Manual |
|
|
| **Resilience** | Failure recovery testing | 45 min | Manual |
|
|
| **Observability** | Telemetry validation | 30 min | Manual |
|
|
|
|
---
|
|
|
|
## Test Discovery
|
|
|
|
### Automatic Discovery
|
|
|
|
The `test-matrix.yml` workflow automatically discovers all test projects:
|
|
|
|
```bash
|
|
# Discovery pattern
|
|
find src \( \
|
|
-name "*.Tests.csproj" \
|
|
-o -name "*UnitTests.csproj" \
|
|
-o -name "*SmokeTests.csproj" \
|
|
-o -name "*FixtureTests.csproj" \
|
|
-o -name "*IntegrationTests.csproj" \
|
|
\) -type f \
|
|
! -path "*/node_modules/*" \
|
|
! -path "*/bin/*" \
|
|
! -path "*/obj/*" \
|
|
! -name "StellaOps.TestKit.csproj" \
|
|
! -name "*Testing.csproj"
|
|
```
|
|
|
|
### Test Category Trait
|
|
|
|
Tests are categorized using xUnit traits:
|
|
|
|
```csharp
|
|
[Trait("Category", "Unit")]
|
|
public class MyUnitTests
|
|
{
|
|
[Fact]
|
|
public void Should_Do_Something()
|
|
{
|
|
// ...
|
|
}
|
|
}
|
|
|
|
[Trait("Category", "Integration")]
|
|
public class MyIntegrationTests
|
|
{
|
|
[Fact]
|
|
public void Should_Connect_To_Database()
|
|
{
|
|
// ...
|
|
}
|
|
}
|
|
```
|
|
|
|
### Running Specific Categories
|
|
|
|
```bash
|
|
# Run Unit tests only
|
|
dotnet test --filter "Category=Unit"
|
|
|
|
# Run multiple categories
|
|
dotnet test --filter "Category=Unit|Category=Integration"
|
|
|
|
# Run excluding a category
|
|
dotnet test --filter "Category!=Performance"
|
|
```
|
|
|
|
---
|
|
|
|
## Test Infrastructure
|
|
|
|
### Shared Libraries
|
|
|
|
| Library | Purpose | Location |
|
|
|---------|---------|----------|
|
|
| `StellaOps.TestKit` | Common test utilities | `src/__Tests/__Libraries/` |
|
|
| `StellaOps.Infrastructure.Postgres.Testing` | PostgreSQL fixtures | `src/__Tests/__Libraries/` |
|
|
| `StellaOps.Concelier.Testing` | Concelier test fixtures | `src/Concelier/__Tests/` |
|
|
|
|
### Testcontainers
|
|
|
|
Integration tests use Testcontainers for isolated dependencies:
|
|
|
|
```csharp
|
|
public class PostgresFixture : IAsyncLifetime
|
|
{
|
|
private readonly PostgreSqlContainer _container = new PostgreSqlBuilder()
|
|
.WithImage("postgres:16")
|
|
.WithDatabase("test_db")
|
|
.Build();
|
|
|
|
public string ConnectionString => _container.GetConnectionString();
|
|
|
|
public Task InitializeAsync() => _container.StartAsync();
|
|
public Task DisposeAsync() => _container.DisposeAsync().AsTask();
|
|
}
|
|
```
|
|
|
|
### Ground Truth Corpus
|
|
|
|
Golden tests use a corpus of known-good outputs:
|
|
|
|
```
|
|
src/__Tests/__Datasets/
|
|
├── scanner/
|
|
│ ├── golden/
|
|
│ │ ├── npm-package.expected.json
|
|
│ │ ├── dotnet-project.expected.json
|
|
│ │ └── container-image.expected.json
|
|
│ └── fixtures/
|
|
│ ├── npm-package/
|
|
│ ├── dotnet-project/
|
|
│ └── container-image/
|
|
└── concelier/
|
|
├── golden/
|
|
└── fixtures/
|
|
```
|
|
|
|
---
|
|
|
|
## CI/CD Integration
|
|
|
|
### Test Matrix Workflow
|
|
|
|
```yaml
|
|
# Simplified test-matrix.yml structure
|
|
jobs:
|
|
discover:
|
|
# Find all test projects
|
|
outputs:
|
|
test-projects: ${{ steps.find.outputs.projects }}
|
|
|
|
pr-gating-tests:
|
|
strategy:
|
|
matrix:
|
|
include:
|
|
- category: Unit
|
|
timeout: 20
|
|
- category: Architecture
|
|
timeout: 15
|
|
- category: Contract
|
|
timeout: 15
|
|
- category: Security
|
|
timeout: 25
|
|
- category: Golden
|
|
timeout: 25
|
|
steps:
|
|
- run: .gitea/scripts/test/run-test-category.sh "${{ matrix.category }}"
|
|
|
|
integration:
|
|
services:
|
|
postgres:
|
|
image: postgres:16
|
|
steps:
|
|
- run: .gitea/scripts/test/run-test-category.sh Integration
|
|
|
|
summary:
|
|
needs: [discover, pr-gating-tests, integration]
|
|
```
|
|
|
|
### Test Results
|
|
|
|
All tests produce TRX (Visual Studio Test Results) files:
|
|
|
|
```bash
|
|
# Output structure
|
|
TestResults/
|
|
├── Unit/
|
|
│ ├── src_Scanner___Tests_StellaOps.Scanner.Tests-unit.trx
|
|
│ └── src_Authority___Tests_StellaOps.Authority.Tests-unit.trx
|
|
├── Integration/
|
|
│ └── ...
|
|
└── Combined/
|
|
└── test-results-combined.trx
|
|
```
|
|
|
|
### Coverage Collection
|
|
|
|
```yaml
|
|
# Collect coverage for Unit tests
|
|
- run: |
|
|
.gitea/scripts/test/run-test-category.sh Unit --collect-coverage
|
|
```
|
|
|
|
Coverage reports are generated in Cobertura format and converted to HTML.
|
|
|
|
---
|
|
|
|
## Test Categories Deep Dive
|
|
|
|
### Unit Tests
|
|
|
|
**Purpose:** Test isolated components without external dependencies.
|
|
|
|
**Characteristics:**
|
|
- No I/O (database, network, file system)
|
|
- No async waits or delays
|
|
- Fast execution (< 100ms per test)
|
|
- High parallelism
|
|
|
|
**Example:**
|
|
|
|
```csharp
|
|
[Trait("Category", "Unit")]
|
|
public class VexPolicyBinderTests
|
|
{
|
|
[Fact]
|
|
public void Bind_WithValidPolicy_ReturnsSuccess()
|
|
{
|
|
var binder = new VexPolicyBinder();
|
|
var policy = new VexPolicy { /* ... */ };
|
|
|
|
var result = binder.Bind(policy);
|
|
|
|
Assert.True(result.IsSuccess);
|
|
}
|
|
}
|
|
```
|
|
|
|
### Architecture Tests
|
|
|
|
**Purpose:** Enforce architectural rules and dependency constraints.
|
|
|
|
**Rules Enforced:**
|
|
- Layer dependencies (UI → Application → Domain → Infrastructure)
|
|
- Namespace conventions
|
|
- Circular dependency prevention
|
|
- Interface segregation
|
|
|
|
**Example:**
|
|
|
|
```csharp
|
|
[Trait("Category", "Architecture")]
|
|
public class DependencyTests
|
|
{
|
|
[Fact]
|
|
public void Domain_Should_Not_Depend_On_Infrastructure()
|
|
{
|
|
var result = Types.InAssembly(typeof(DomainMarker).Assembly)
|
|
.That().ResideInNamespace("StellaOps.Domain")
|
|
.ShouldNot().HaveDependencyOn("StellaOps.Infrastructure")
|
|
.GetResult();
|
|
|
|
Assert.True(result.IsSuccessful);
|
|
}
|
|
}
|
|
```
|
|
|
|
### Contract Tests
|
|
|
|
**Purpose:** Validate API contracts are maintained.
|
|
|
|
**Checks:**
|
|
- Request/response schemas
|
|
- OpenAPI specification compliance
|
|
- Backward compatibility
|
|
|
|
**Example:**
|
|
|
|
```csharp
|
|
[Trait("Category", "Contract")]
|
|
public class VulnerabilityApiContractTests
|
|
{
|
|
[Fact]
|
|
public async Task GetVulnerability_ReturnsExpectedSchema()
|
|
{
|
|
var response = await _client.GetAsync("/api/v1/vulnerabilities/CVE-2024-1234");
|
|
|
|
await Verify(response)
|
|
.UseDirectory("Snapshots")
|
|
.UseMethodName("GetVulnerability");
|
|
}
|
|
}
|
|
```
|
|
|
|
### Integration Tests
|
|
|
|
**Purpose:** Test component integration with real dependencies.
|
|
|
|
**Dependencies:**
|
|
- PostgreSQL (via Testcontainers)
|
|
- Valkey/Redis (via Testcontainers)
|
|
- File system
|
|
|
|
**Example:**
|
|
|
|
```csharp
|
|
[Trait("Category", "Integration")]
|
|
public class VulnerabilityRepositoryTests : IClassFixture<PostgresFixture>
|
|
{
|
|
private readonly PostgresFixture _fixture;
|
|
|
|
[Fact]
|
|
public async Task Save_AndRetrieve_Vulnerability()
|
|
{
|
|
var repo = new VulnerabilityRepository(_fixture.ConnectionString);
|
|
var vuln = new Vulnerability { Id = "CVE-2024-1234" };
|
|
|
|
await repo.SaveAsync(vuln);
|
|
var retrieved = await repo.GetAsync("CVE-2024-1234");
|
|
|
|
Assert.Equal(vuln.Id, retrieved.Id);
|
|
}
|
|
}
|
|
```
|
|
|
|
### Security Tests
|
|
|
|
**Purpose:** Validate security controls and assertions.
|
|
|
|
**Checks:**
|
|
- Input validation
|
|
- Authorization enforcement
|
|
- Cryptographic operations
|
|
- Secrets handling
|
|
|
|
**Example:**
|
|
|
|
```csharp
|
|
[Trait("Category", "Security")]
|
|
public class AuthorizationTests
|
|
{
|
|
[Fact]
|
|
public async Task Unauthorized_User_Cannot_Access_Admin_Endpoint()
|
|
{
|
|
var client = _factory.CreateClient();
|
|
client.DefaultRequestHeaders.Authorization = null;
|
|
|
|
var response = await client.GetAsync("/api/admin/settings");
|
|
|
|
Assert.Equal(HttpStatusCode.Unauthorized, response.StatusCode);
|
|
}
|
|
}
|
|
```
|
|
|
|
### Golden Tests
|
|
|
|
**Purpose:** Verify output matches known-good corpus.
|
|
|
|
**Example:**
|
|
|
|
```csharp
|
|
[Trait("Category", "Golden")]
|
|
public class ScannerGoldenTests
|
|
{
|
|
[Theory]
|
|
[InlineData("npm-package")]
|
|
[InlineData("dotnet-project")]
|
|
public async Task Scan_MatchesGoldenOutput(string fixture)
|
|
{
|
|
var scanner = new ContainerScanner();
|
|
var result = await scanner.ScanAsync($"fixtures/{fixture}");
|
|
|
|
await Verify(result)
|
|
.UseDirectory("golden")
|
|
.UseFileName(fixture);
|
|
}
|
|
}
|
|
```
|
|
|
|
---
|
|
|
|
## Performance Testing
|
|
|
|
### BenchmarkDotNet
|
|
|
|
```csharp
|
|
[Trait("Category", "Benchmark")]
|
|
[MemoryDiagnoser]
|
|
public class ScannerBenchmarks
|
|
{
|
|
[Benchmark]
|
|
public async Task ScanSmallImage()
|
|
{
|
|
await _scanner.ScanAsync(_smallImage);
|
|
}
|
|
|
|
[Benchmark]
|
|
public async Task ScanLargeImage()
|
|
{
|
|
await _scanner.ScanAsync(_largeImage);
|
|
}
|
|
}
|
|
```
|
|
|
|
### Performance SLOs
|
|
|
|
| Metric | Target | Action on Breach |
|
|
|--------|--------|------------------|
|
|
| Unit test P95 | < 100ms | Warning |
|
|
| Integration test P95 | < 5s | Warning |
|
|
| Scan time P95 | < 5 min | Block |
|
|
| Memory peak | < 2GB | Block |
|
|
|
|
---
|
|
|
|
## Troubleshooting
|
|
|
|
### Tests Fail in CI but Pass Locally
|
|
|
|
1. **Check timezone** - CI uses `TZ=UTC`
|
|
2. **Check parallelism** - CI runs tests in parallel
|
|
3. **Check container availability** - Testcontainers requires Docker
|
|
4. **Check file paths** - Case sensitivity on Linux
|
|
|
|
### Flaky Tests
|
|
|
|
1. **Add retry logic** for network operations
|
|
2. **Use proper async/await** - no `Task.Run` for async operations
|
|
3. **Isolate shared state** - use fresh fixtures per test
|
|
4. **Increase timeouts** for integration tests
|
|
|
|
### Missing Test Category
|
|
|
|
Ensure your test class has the correct trait:
|
|
|
|
```csharp
|
|
[Trait("Category", "Unit")] // Add this!
|
|
public class MyTests
|
|
{
|
|
// ...
|
|
}
|
|
```
|
|
|
|
---
|
|
|
|
## Related Documentation
|
|
|
|
- [README - CI/CD Overview](./README.md)
|
|
- [Workflow Triggers](./workflow-triggers.md)
|
|
- [CI Quality Gates](../testing/ci-quality-gates.md)
|
|
- [Test Catalog](../testing/TEST_CATALOG.yml)
|