Fix build and code structure improvements. New but essential UI functionality. CI improvements. Documentation improvements. AI module improvements.
This commit is contained in:
StellaOps Bot
461
docs/cicd/test-strategy.md
Normal file
461
docs/cicd/test-strategy.md
Normal file
@@ -0,0 +1,461 @@
|
||||
# Test Strategy
|
||||
|
||||
> Complete guide to the StellaOps testing strategy and CI/CD integration.
|
||||
|
||||
---
|
||||
|
||||
## Test Category Overview
|
||||
|
||||
StellaOps uses a **tiered testing strategy** with 13 test categories:
|
||||
|
||||
### PR-Gating Tests (Required for Merge)
|
||||
|
||||
| Category | Purpose | Timeout | Parallelism |
|
||||
|----------|---------|---------|-------------|
|
||||
| **Unit** | Isolated component tests | 20 min | High |
|
||||
| **Architecture** | Dependency rule enforcement | 15 min | High |
|
||||
| **Contract** | API compatibility | 15 min | High |
|
||||
| **Integration** | Database/service integration | 45 min | Medium |
|
||||
| **Security** | Security-focused assertions | 25 min | High |
|
||||
| **Golden** | Corpus-based validation | 25 min | High |
|
||||
|
||||
### Extended Tests (Scheduled/On-Demand)
|
||||
|
||||
| Category | Purpose | Timeout | Trigger |
|
||||
|----------|---------|---------|---------|
|
||||
| **Performance** | Latency/throughput benchmarks | 45 min | Daily, Manual |
|
||||
| **Benchmark** | BenchmarkDotNet profiling | 60 min | Daily, Manual |
|
||||
| **AirGap** | Offline operation validation | 45 min | Manual |
|
||||
| **Chaos** | Resilience testing | 45 min | Manual |
|
||||
| **Determinism** | Reproducibility verification | 45 min | Manual |
|
||||
| **Resilience** | Failure recovery testing | 45 min | Manual |
|
||||
| **Observability** | Telemetry validation | 30 min | Manual |
|
||||
|
||||
---
|
||||
|
||||
## Test Discovery
|
||||
|
||||
### Automatic Discovery
|
||||
|
||||
The `test-matrix.yml` workflow automatically discovers all test projects:
|
||||
|
||||
```bash
|
||||
# Discovery pattern
|
||||
find src \( \
|
||||
-name "*.Tests.csproj" \
|
||||
-o -name "*UnitTests.csproj" \
|
||||
-o -name "*SmokeTests.csproj" \
|
||||
-o -name "*FixtureTests.csproj" \
|
||||
-o -name "*IntegrationTests.csproj" \
|
||||
\) -type f \
|
||||
! -path "*/node_modules/*" \
|
||||
! -path "*/bin/*" \
|
||||
! -path "*/obj/*" \
|
||||
! -name "StellaOps.TestKit.csproj" \
|
||||
! -name "*Testing.csproj"
|
||||
```
|
||||
|
||||
### Test Category Trait
|
||||
|
||||
Tests are categorized using xUnit traits:
|
||||
|
||||
```csharp
|
||||
[Trait("Category", "Unit")]
|
||||
public class MyUnitTests
|
||||
{
|
||||
[Fact]
|
||||
public void Should_Do_Something()
|
||||
{
|
||||
// ...
|
||||
}
|
||||
}
|
||||
|
||||
[Trait("Category", "Integration")]
|
||||
public class MyIntegrationTests
|
||||
{
|
||||
[Fact]
|
||||
public void Should_Connect_To_Database()
|
||||
{
|
||||
// ...
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Running Specific Categories
|
||||
|
||||
```bash
|
||||
# Run Unit tests only
|
||||
dotnet test --filter "Category=Unit"
|
||||
|
||||
# Run multiple categories
|
||||
dotnet test --filter "Category=Unit|Category=Integration"
|
||||
|
||||
# Run excluding a category
|
||||
dotnet test --filter "Category!=Performance"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Test Infrastructure
|
||||
|
||||
### Shared Libraries
|
||||
|
||||
| Library | Purpose | Location |
|
||||
|---------|---------|----------|
|
||||
| `StellaOps.TestKit` | Common test utilities | `src/__Tests/__Libraries/` |
|
||||
| `StellaOps.Infrastructure.Postgres.Testing` | PostgreSQL fixtures | `src/__Tests/__Libraries/` |
|
||||
| `StellaOps.Concelier.Testing` | Concelier test fixtures | `src/Concelier/__Tests/` |
|
||||
|
||||
### Testcontainers
|
||||
|
||||
Integration tests use Testcontainers for isolated dependencies:
|
||||
|
||||
```csharp
|
||||
public class PostgresFixture : IAsyncLifetime
|
||||
{
|
||||
private readonly PostgreSqlContainer _container = new PostgreSqlBuilder()
|
||||
.WithImage("postgres:16")
|
||||
.WithDatabase("test_db")
|
||||
.Build();
|
||||
|
||||
public string ConnectionString => _container.GetConnectionString();
|
||||
|
||||
public Task InitializeAsync() => _container.StartAsync();
|
||||
public Task DisposeAsync() => _container.DisposeAsync().AsTask();
|
||||
}
|
||||
```
|
||||
|
||||
### Ground Truth Corpus
|
||||
|
||||
Golden tests use a corpus of known-good outputs:
|
||||
|
||||
```
|
||||
src/__Tests/__Datasets/
|
||||
├── scanner/
|
||||
│ ├── golden/
|
||||
│ │ ├── npm-package.expected.json
|
||||
│ │ ├── dotnet-project.expected.json
|
||||
│ │ └── container-image.expected.json
|
||||
│ └── fixtures/
|
||||
│ ├── npm-package/
|
||||
│ ├── dotnet-project/
|
||||
│ └── container-image/
|
||||
└── concelier/
|
||||
├── golden/
|
||||
└── fixtures/
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## CI/CD Integration
|
||||
|
||||
### Test Matrix Workflow
|
||||
|
||||
```yaml
|
||||
# Simplified test-matrix.yml structure
|
||||
jobs:
|
||||
discover:
|
||||
# Find all test projects
|
||||
outputs:
|
||||
test-projects: ${{ steps.find.outputs.projects }}
|
||||
|
||||
pr-gating-tests:
|
||||
strategy:
|
||||
matrix:
|
||||
include:
|
||||
- category: Unit
|
||||
timeout: 20
|
||||
- category: Architecture
|
||||
timeout: 15
|
||||
- category: Contract
|
||||
timeout: 15
|
||||
- category: Security
|
||||
timeout: 25
|
||||
- category: Golden
|
||||
timeout: 25
|
||||
steps:
|
||||
- run: .gitea/scripts/test/run-test-category.sh "${{ matrix.category }}"
|
||||
|
||||
integration:
|
||||
services:
|
||||
postgres:
|
||||
image: postgres:16
|
||||
steps:
|
||||
- run: .gitea/scripts/test/run-test-category.sh Integration
|
||||
|
||||
summary:
|
||||
needs: [discover, pr-gating-tests, integration]
|
||||
```
|
||||
|
||||
### Test Results
|
||||
|
||||
All tests produce TRX (Visual Studio Test Results) files:
|
||||
|
||||
```bash
|
||||
# Output structure
|
||||
TestResults/
|
||||
├── Unit/
|
||||
│ ├── src_Scanner___Tests_StellaOps.Scanner.Tests-unit.trx
|
||||
│ └── src_Authority___Tests_StellaOps.Authority.Tests-unit.trx
|
||||
├── Integration/
|
||||
│ └── ...
|
||||
└── Combined/
|
||||
└── test-results-combined.trx
|
||||
```
|
||||
|
||||
### Coverage Collection
|
||||
|
||||
```yaml
|
||||
# Collect coverage for Unit tests
|
||||
- run: |
|
||||
.gitea/scripts/test/run-test-category.sh Unit --collect-coverage
|
||||
```
|
||||
|
||||
Coverage reports are generated in Cobertura format and converted to HTML.
|
||||
|
||||
---
|
||||
|
||||
## Test Categories Deep Dive
|
||||
|
||||
### Unit Tests
|
||||
|
||||
**Purpose:** Test isolated components without external dependencies.
|
||||
|
||||
**Characteristics:**
|
||||
- No I/O (database, network, file system)
|
||||
- No async waits or delays
|
||||
- Fast execution (< 100ms per test)
|
||||
- High parallelism
|
||||
|
||||
**Example:**
|
||||
|
||||
```csharp
|
||||
[Trait("Category", "Unit")]
|
||||
public class VexPolicyBinderTests
|
||||
{
|
||||
[Fact]
|
||||
public void Bind_WithValidPolicy_ReturnsSuccess()
|
||||
{
|
||||
var binder = new VexPolicyBinder();
|
||||
var policy = new VexPolicy { /* ... */ };
|
||||
|
||||
var result = binder.Bind(policy);
|
||||
|
||||
Assert.True(result.IsSuccess);
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Architecture Tests
|
||||
|
||||
**Purpose:** Enforce architectural rules and dependency constraints.
|
||||
|
||||
**Rules Enforced:**
|
||||
- Layer dependencies (UI → Application → Domain → Infrastructure)
|
||||
- Namespace conventions
|
||||
- Circular dependency prevention
|
||||
- Interface segregation
|
||||
|
||||
**Example:**
|
||||
|
||||
```csharp
|
||||
[Trait("Category", "Architecture")]
|
||||
public class DependencyTests
|
||||
{
|
||||
[Fact]
|
||||
public void Domain_Should_Not_Depend_On_Infrastructure()
|
||||
{
|
||||
var result = Types.InAssembly(typeof(DomainMarker).Assembly)
|
||||
.That().ResideInNamespace("StellaOps.Domain")
|
||||
.ShouldNot().HaveDependencyOn("StellaOps.Infrastructure")
|
||||
.GetResult();
|
||||
|
||||
Assert.True(result.IsSuccessful);
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Contract Tests
|
||||
|
||||
**Purpose:** Validate API contracts are maintained.
|
||||
|
||||
**Checks:**
|
||||
- Request/response schemas
|
||||
- OpenAPI specification compliance
|
||||
- Backward compatibility
|
||||
|
||||
**Example:**
|
||||
|
||||
```csharp
|
||||
[Trait("Category", "Contract")]
|
||||
public class VulnerabilityApiContractTests
|
||||
{
|
||||
[Fact]
|
||||
public async Task GetVulnerability_ReturnsExpectedSchema()
|
||||
{
|
||||
var response = await _client.GetAsync("/api/v1/vulnerabilities/CVE-2024-1234");
|
||||
|
||||
await Verify(response)
|
||||
.UseDirectory("Snapshots")
|
||||
.UseMethodName("GetVulnerability");
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Integration Tests
|
||||
|
||||
**Purpose:** Test component integration with real dependencies.
|
||||
|
||||
**Dependencies:**
|
||||
- PostgreSQL (via Testcontainers)
|
||||
- Valkey/Redis (via Testcontainers)
|
||||
- File system
|
||||
|
||||
**Example:**
|
||||
|
||||
```csharp
|
||||
[Trait("Category", "Integration")]
|
||||
public class VulnerabilityRepositoryTests : IClassFixture<PostgresFixture>
|
||||
{
|
||||
private readonly PostgresFixture _fixture;
|
||||
|
||||
[Fact]
|
||||
public async Task Save_AndRetrieve_Vulnerability()
|
||||
{
|
||||
var repo = new VulnerabilityRepository(_fixture.ConnectionString);
|
||||
var vuln = new Vulnerability { Id = "CVE-2024-1234" };
|
||||
|
||||
await repo.SaveAsync(vuln);
|
||||
var retrieved = await repo.GetAsync("CVE-2024-1234");
|
||||
|
||||
Assert.Equal(vuln.Id, retrieved.Id);
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Security Tests
|
||||
|
||||
**Purpose:** Validate security controls and assertions.
|
||||
|
||||
**Checks:**
|
||||
- Input validation
|
||||
- Authorization enforcement
|
||||
- Cryptographic operations
|
||||
- Secrets handling
|
||||
|
||||
**Example:**
|
||||
|
||||
```csharp
|
||||
[Trait("Category", "Security")]
|
||||
public class AuthorizationTests
|
||||
{
|
||||
[Fact]
|
||||
public async Task Unauthorized_User_Cannot_Access_Admin_Endpoint()
|
||||
{
|
||||
var client = _factory.CreateClient();
|
||||
client.DefaultRequestHeaders.Authorization = null;
|
||||
|
||||
var response = await client.GetAsync("/api/admin/settings");
|
||||
|
||||
Assert.Equal(HttpStatusCode.Unauthorized, response.StatusCode);
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Golden Tests
|
||||
|
||||
**Purpose:** Verify output matches known-good corpus.
|
||||
|
||||
**Example:**
|
||||
|
||||
```csharp
|
||||
[Trait("Category", "Golden")]
|
||||
public class ScannerGoldenTests
|
||||
{
|
||||
[Theory]
|
||||
[InlineData("npm-package")]
|
||||
[InlineData("dotnet-project")]
|
||||
public async Task Scan_MatchesGoldenOutput(string fixture)
|
||||
{
|
||||
var scanner = new ContainerScanner();
|
||||
var result = await scanner.ScanAsync($"fixtures/{fixture}");
|
||||
|
||||
await Verify(result)
|
||||
.UseDirectory("golden")
|
||||
.UseFileName(fixture);
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Performance Testing
|
||||
|
||||
### BenchmarkDotNet
|
||||
|
||||
```csharp
|
||||
[Trait("Category", "Benchmark")]
|
||||
[MemoryDiagnoser]
|
||||
public class ScannerBenchmarks
|
||||
{
|
||||
[Benchmark]
|
||||
public async Task ScanSmallImage()
|
||||
{
|
||||
await _scanner.ScanAsync(_smallImage);
|
||||
}
|
||||
|
||||
[Benchmark]
|
||||
public async Task ScanLargeImage()
|
||||
{
|
||||
await _scanner.ScanAsync(_largeImage);
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Performance SLOs
|
||||
|
||||
| Metric | Target | Action on Breach |
|
||||
|--------|--------|------------------|
|
||||
| Unit test P95 | < 100ms | Warning |
|
||||
| Integration test P95 | < 5s | Warning |
|
||||
| Scan time P95 | < 5 min | Block |
|
||||
| Memory peak | < 2GB | Block |
|
||||
|
||||
---
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Tests Fail in CI but Pass Locally
|
||||
|
||||
1. **Check timezone** - CI uses `TZ=UTC`
|
||||
2. **Check parallelism** - CI runs tests in parallel
|
||||
3. **Check container availability** - Testcontainers requires Docker
|
||||
4. **Check file paths** - Case sensitivity on Linux
|
||||
|
||||
### Flaky Tests
|
||||
|
||||
1. **Add retry logic** for network operations
|
||||
2. **Use proper async/await** - no `Task.Run` for async operations
|
||||
3. **Isolate shared state** - use fresh fixtures per test
|
||||
4. **Increase timeouts** for integration tests
|
||||
|
||||
### Missing Test Category
|
||||
|
||||
Ensure your test class has the correct trait:
|
||||
|
||||
```csharp
|
||||
[Trait("Category", "Unit")] // Add this!
|
||||
public class MyTests
|
||||
{
|
||||
// ...
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Related Documentation
|
||||
|
||||
- [README - CI/CD Overview](./README.md)
|
||||
- [Workflow Triggers](./workflow-triggers.md)
|
||||
- [CI Quality Gates](../testing/ci-quality-gates.md)
|
||||
- [Test Catalog](../testing/TEST_CATALOG.yml)
|
||||
Reference in New Issue
Block a user