Add reference architecture and testing strategy documentation
- Created a new document for the Stella Ops Reference Architecture outlining the system's topology, trust boundaries, artifact association, and interfaces. - Developed a comprehensive Testing Strategy document detailing the importance of offline readiness, interoperability, determinism, and operational guardrails. - Introduced a README for the Testing Strategy, summarizing processing details and key concepts implemented. - Added guidance for AI agents and developers in the tests directory, including directory structure, test categories, key patterns, and rules for test development.
This commit is contained in:
189
tests/AGENTS.md
Normal file
189
tests/AGENTS.md
Normal file
@@ -0,0 +1,189 @@
|
||||
# tests/AGENTS.md
|
||||
|
||||
## Overview
|
||||
|
||||
This document provides guidance for AI agents and developers working in the `tests/` directory of the StellaOps codebase.
|
||||
|
||||
## Directory Structure
|
||||
|
||||
```
|
||||
tests/
|
||||
├── acceptance/ # Acceptance test suites
|
||||
├── AirGap/ # Air-gap specific tests
|
||||
├── authority/ # Authority module tests
|
||||
├── chaos/ # Chaos engineering tests
|
||||
├── e2e/ # End-to-end test suites
|
||||
├── EvidenceLocker/ # Evidence storage tests
|
||||
├── fixtures/ # Shared test fixtures
|
||||
│ ├── offline-bundle/ # Offline bundle for air-gap tests
|
||||
│ ├── images/ # Container image tarballs
|
||||
│ └── sboms/ # Sample SBOM documents
|
||||
├── Graph/ # Graph module tests
|
||||
├── integration/ # Integration test suites
|
||||
├── interop/ # Interoperability tests
|
||||
├── load/ # Load testing scripts
|
||||
├── native/ # Native code tests
|
||||
├── offline/ # Offline operation tests
|
||||
├── plugins/ # Plugin tests
|
||||
├── Policy/ # Policy module tests
|
||||
├── Provenance/ # Provenance/attestation tests
|
||||
├── reachability/ # Reachability analysis tests
|
||||
├── Replay/ # Replay functionality tests
|
||||
├── security/ # Security tests (OWASP)
|
||||
├── shared/ # Shared test utilities
|
||||
└── Vex/ # VEX processing tests
|
||||
```
|
||||
|
||||
## Test Categories
|
||||
|
||||
### When writing tests, use appropriate category traits:
|
||||
|
||||
```csharp
|
||||
[Trait("Category", "Unit")] // Fast, isolated unit tests
|
||||
[Trait("Category", "Integration")] // Tests requiring infrastructure
|
||||
[Trait("Category", "E2E")] // Full end-to-end workflows
|
||||
[Trait("Category", "AirGap")] // Must work without network
|
||||
[Trait("Category", "Interop")] // Third-party tool compatibility
|
||||
[Trait("Category", "Performance")] // Performance benchmarks
|
||||
[Trait("Category", "Chaos")] // Failure injection tests
|
||||
[Trait("Category", "Security")] // Security-focused tests
|
||||
```
|
||||
|
||||
## Key Patterns
|
||||
|
||||
### 1. PostgreSQL Integration Tests
|
||||
|
||||
Use the shared fixture from `StellaOps.Infrastructure.Postgres.Testing`:
|
||||
|
||||
```csharp
|
||||
public class MyIntegrationTests : IClassFixture<MyPostgresFixture>
|
||||
{
|
||||
private readonly MyPostgresFixture _fixture;
|
||||
|
||||
public MyIntegrationTests(MyPostgresFixture fixture)
|
||||
{
|
||||
_fixture = fixture;
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task MyTest()
|
||||
{
|
||||
// _fixture.ConnectionString is available
|
||||
// _fixture.TruncateAllTablesAsync() for cleanup
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### 2. Air-Gap Tests
|
||||
|
||||
Inherit from `NetworkIsolatedTestBase` for network-free tests:
|
||||
|
||||
```csharp
|
||||
[Trait("Category", "AirGap")]
|
||||
public class OfflineTests : NetworkIsolatedTestBase
|
||||
{
|
||||
[Fact]
|
||||
public async Task Test_WorksOffline()
|
||||
{
|
||||
// Test implementation
|
||||
AssertNoNetworkCalls(); // Fails if network accessed
|
||||
}
|
||||
|
||||
protected string GetOfflineBundlePath() =>
|
||||
Path.Combine(AppContext.BaseDirectory, "fixtures", "offline-bundle");
|
||||
}
|
||||
```
|
||||
|
||||
### 3. Determinism Tests
|
||||
|
||||
Use `DeterminismVerifier` to ensure reproducibility:
|
||||
|
||||
```csharp
|
||||
[Fact]
|
||||
public void Output_IsDeterministic()
|
||||
{
|
||||
var verifier = new DeterminismVerifier();
|
||||
var result = verifier.Verify(myObject, iterations: 10);
|
||||
|
||||
result.IsDeterministic.Should().BeTrue();
|
||||
}
|
||||
```
|
||||
|
||||
### 4. Golden Corpus Tests
|
||||
|
||||
Reference cases from `bench/golden-corpus/`:
|
||||
|
||||
```csharp
|
||||
[Theory]
|
||||
[MemberData(nameof(GetCorpusCases))]
|
||||
public async Task Corpus_Case_Passes(string caseId)
|
||||
{
|
||||
var testCase = CorpusLoader.Load(caseId);
|
||||
var result = await ProcessAsync(testCase.Input);
|
||||
result.Should().BeEquivalentTo(testCase.Expected);
|
||||
}
|
||||
```
|
||||
|
||||
## Rules for Test Development
|
||||
|
||||
### DO:
|
||||
|
||||
1. **Tag tests with appropriate categories** for filtering
|
||||
2. **Use Testcontainers** for infrastructure dependencies
|
||||
3. **Inherit from shared fixtures** to avoid duplication
|
||||
4. **Assert no network calls** in air-gap tests
|
||||
5. **Verify determinism** for any serialization output
|
||||
6. **Use property-based tests** (FsCheck) for invariants
|
||||
7. **Document test purpose** in method names
|
||||
|
||||
### DON'T:
|
||||
|
||||
1. **Don't skip tests** without documenting why
|
||||
2. **Don't use Thread.Sleep** - use proper async waits
|
||||
3. **Don't hardcode paths** - use `AppContext.BaseDirectory`
|
||||
4. **Don't make network calls** in non-interop tests
|
||||
5. **Don't depend on test execution order**
|
||||
6. **Don't leave test data in shared databases**
|
||||
|
||||
## Test Infrastructure
|
||||
|
||||
### Required Services (CI)
|
||||
|
||||
```yaml
|
||||
services:
|
||||
postgres:
|
||||
image: postgres:16-alpine
|
||||
env:
|
||||
POSTGRES_PASSWORD: test
|
||||
valkey:
|
||||
image: valkey/valkey:7-alpine
|
||||
```
|
||||
|
||||
### Environment Variables
|
||||
|
||||
| Variable | Purpose | Default |
|
||||
|----------|---------|---------|
|
||||
| `STELLAOPS_OFFLINE_MODE` | Enable offline mode | `false` |
|
||||
| `STELLAOPS_OFFLINE_BUNDLE` | Path to offline bundle | - |
|
||||
| `STELLAOPS_TEST_POSTGRES` | PostgreSQL connection | Testcontainers |
|
||||
| `STELLAOPS_TEST_VALKEY` | Valkey connection | Testcontainers |
|
||||
|
||||
## Related Sprints
|
||||
|
||||
| Sprint | Topic |
|
||||
|--------|-------|
|
||||
| 5100.0001.0001 | Run Manifest Schema |
|
||||
| 5100.0001.0002 | Evidence Index Schema |
|
||||
| 5100.0001.0004 | Golden Corpus Expansion |
|
||||
| 5100.0002.0001 | Canonicalization Utilities |
|
||||
| 5100.0002.0002 | Replay Runner Service |
|
||||
| 5100.0003.0001 | SBOM Interop Round-Trip |
|
||||
| 5100.0003.0002 | No-Egress Enforcement |
|
||||
| 5100.0005.0001 | Router Chaos Suite |
|
||||
|
||||
## Contact
|
||||
|
||||
For test infrastructure questions, see:
|
||||
- `docs/19_TEST_SUITE_OVERVIEW.md`
|
||||
- `docs/implplan/SPRINT_5100_SUMMARY.md`
|
||||
- Sprint files in `docs/implplan/SPRINT_5100_*.md`
|
||||
Reference in New Issue
Block a user