217 lines
8.0 KiB
Markdown
217 lines
8.0 KiB
Markdown
# src/__Tests/AGENTS.md
|
|
|
|
## Purpose & Scope
|
|
|
|
This directory contains all global test infrastructure, benchmarks, datasets, and shared testing libraries for the StellaOps platform.
|
|
|
|
- **Working directory:** `src/__Tests/`
|
|
- **Roles:** QA engineer, performance/bench engineer, integration test developer, docs contributor
|
|
|
|
## Directory Structure
|
|
|
|
```
|
|
src/__Tests/
|
|
????????? __Libraries/ # Shared testing libraries
|
|
??? ????????? StellaOps.Infrastructure.Postgres.Testing/
|
|
??? ????????? StellaOps.Messaging.Testing/
|
|
??? ????????? StellaOps.Testing.AirGap/
|
|
??? ????????? StellaOps.Testing.Determinism/
|
|
??? ????????? StellaOps.Testing.Manifests/
|
|
??? ????????? StellaOps.Concelier.Testing/
|
|
??? ????????? StellaOps.Router.Testing/
|
|
????????? __Benchmarks/ # Golden corpus, CVE findings, determinism fixtures
|
|
??? ????????? golden-corpus/ # Canonical test cases (severity, VEX, reachability)
|
|
??? ????????? findings/ # CVE bundles with reachability evidence
|
|
??? ????????? reachability-benchmark/ # Public multi-language benchmark
|
|
??? ????????? determinism/ # Determinism test fixtures
|
|
??? ????????? tools/ # Verification utilities
|
|
????????? __Datasets/ # Ground truth samples, schemas
|
|
??? ????????? reachability/ # Reachability ground truth
|
|
????????? Integration/ # Cross-module integration tests
|
|
????????? acceptance/ # Acceptance test packs
|
|
????????? load/ # k6 load tests
|
|
????????? security/ # OWASP security tests
|
|
????????? chaos/ # Chaos engineering tests
|
|
????????? AirGap/ # Offline operation tests
|
|
????????? reachability/ # Reachability analysis tests
|
|
????????? fixtures/ # Shared test fixtures (offline-bundle, images, sboms)
|
|
????????? ... # Other test categories
|
|
```
|
|
|
|
## Required Reading
|
|
|
|
Before working in this directory:
|
|
- `docs/README.md`
|
|
- `docs/technical/testing/TEST_SUITE_OVERVIEW.md`
|
|
- `src/__Tests/__Benchmarks/README.md`
|
|
- Sprint-specific guidance for corpus/bench artifacts
|
|
|
|
## Test Categories
|
|
|
|
Use the standardized test categories from `StellaOps.TestKit.TestCategories`:
|
|
|
|
```csharp
|
|
using StellaOps.TestKit;
|
|
|
|
// PR-GATING TESTS (run on every push/PR)
|
|
[Trait("Category", TestCategories.Unit)] // Fast, in-memory, no external dependencies
|
|
[Trait("Category", TestCategories.Architecture)] // Module dependency rules, naming conventions
|
|
[Trait("Category", TestCategories.Contract)] // API/WebService contract verification
|
|
[Trait("Category", TestCategories.Integration)] // Testcontainers, PostgreSQL, Valkey
|
|
[Trait("Category", TestCategories.Security)] // Cryptographic validation, vulnerability scanning
|
|
[Trait("Category", TestCategories.Golden)] // Output comparison against known-good references
|
|
|
|
// SCHEDULED/ON-DEMAND TESTS
|
|
[Trait("Category", TestCategories.Performance)] // Performance measurements, SLO enforcement
|
|
[Trait("Category", TestCategories.Benchmark)] // BenchmarkDotNet measurements
|
|
[Trait("Category", TestCategories.AirGap)] // Offline/air-gapped environment validation
|
|
[Trait("Category", TestCategories.Chaos)] // Fault injection, failure recovery
|
|
[Trait("Category", TestCategories.Determinism)] // Reproducibility, stable ordering, idempotency
|
|
[Trait("Category", TestCategories.Resilience)] // Retry policies, circuit breakers, timeouts
|
|
[Trait("Category", TestCategories.Observability)] // OpenTelemetry traces, metrics, logging
|
|
|
|
// OTHER CATEGORIES
|
|
[Trait("Category", TestCategories.Property)] // FsCheck/generative testing for invariants
|
|
[Trait("Category", TestCategories.Snapshot)] // Golden master regression testing
|
|
[Trait("Category", TestCategories.Live)] // Require external services (Rekor, feeds)
|
|
```
|
|
|
|
### CI/CD Integration
|
|
|
|
Tests are discovered dynamically by `.gitea/workflows/test-matrix.yml` which runs all `*.Tests.csproj` files with Category filtering:
|
|
- **PR-Gating:** Unit, Architecture, Contract, Integration, Security, Golden
|
|
- **Scheduled:** Performance, Benchmark (daily)
|
|
- **On-Demand:** AirGap, Chaos, Determinism, Resilience, Observability
|
|
|
|
### Validation
|
|
|
|
Run the validation script to ensure all tests have Category traits:
|
|
```bash
|
|
python devops/scripts/validate-test-traits.py # Report coverage
|
|
python devops/scripts/validate-test-traits.py --fix # Add default Unit trait
|
|
python devops/scripts/validate-test-traits.py --json # JSON output for CI
|
|
```
|
|
|
|
## Key Patterns
|
|
|
|
### 1. PostgreSQL Integration Tests
|
|
|
|
Use the shared fixture from `__Libraries/StellaOps.Infrastructure.Postgres.Testing`:
|
|
|
|
```csharp
|
|
public class MyIntegrationTests : IClassFixture<MyPostgresFixture>
|
|
{
|
|
private readonly MyPostgresFixture _fixture;
|
|
|
|
public MyIntegrationTests(MyPostgresFixture fixture)
|
|
{
|
|
_fixture = fixture;
|
|
}
|
|
|
|
[Fact]
|
|
public async Task MyTest()
|
|
{
|
|
// _fixture.ConnectionString is available
|
|
// _fixture.TruncateAllTablesAsync() for cleanup
|
|
}
|
|
}
|
|
```
|
|
|
|
### 2. Air-Gap Tests
|
|
|
|
Inherit from `NetworkIsolatedTestBase` for network-free tests:
|
|
|
|
```csharp
|
|
[Trait("Category", "AirGap")]
|
|
public class OfflineTests : NetworkIsolatedTestBase
|
|
{
|
|
[Fact]
|
|
public async Task Test_WorksOffline()
|
|
{
|
|
AssertNoNetworkCalls(); // Fails if network accessed
|
|
}
|
|
|
|
protected string GetOfflineBundlePath() =>
|
|
Path.Combine(AppContext.BaseDirectory, "fixtures", "offline-bundle");
|
|
}
|
|
```
|
|
|
|
### 3. Determinism Tests
|
|
|
|
Use `DeterminismVerifier` to ensure reproducibility:
|
|
|
|
```csharp
|
|
[Fact]
|
|
public void Output_IsDeterministic()
|
|
{
|
|
var verifier = new DeterminismVerifier();
|
|
var result = verifier.Verify(myObject, iterations: 10);
|
|
result.IsDeterministic.Should().BeTrue();
|
|
}
|
|
```
|
|
|
|
### 4. Golden Corpus Tests
|
|
|
|
Reference cases from `__Benchmarks/golden-corpus/`:
|
|
|
|
```csharp
|
|
[Theory]
|
|
[MemberData(nameof(GetCorpusCases))]
|
|
public async Task Corpus_Case_Passes(string caseId)
|
|
{
|
|
var testCase = CorpusLoader.Load(caseId);
|
|
var result = await ProcessAsync(testCase.Input);
|
|
result.Should().BeEquivalentTo(testCase.Expected);
|
|
}
|
|
```
|
|
|
|
## Working Agreements
|
|
|
|
1. **Determinism:** Stable ordering, fixed seeds, UTC timestamps
|
|
2. **Offline-first:** No network dependencies unless explicitly required
|
|
3. **Testcontainers:** Use PostgreSQL fixtures from `__Libraries/`
|
|
4. **Air-gap validation:** Inherit from `NetworkIsolatedTestBase`
|
|
5. **Golden corpus:** Reference cases from `__Benchmarks/golden-corpus/`
|
|
6. **Fixtures:** Keep ASCII and reproducible; avoid oversized binaries
|
|
|
|
## Module Tests vs Global Tests
|
|
|
|
- **Module tests:** Stay in `src/<Module>/__Tests/` - component-specific testing
|
|
- **Global tests:** Go in `src/__Tests/` - cross-cutting, infrastructure, benchmarks, integration
|
|
|
|
## Rules for Test Development
|
|
|
|
### DO:
|
|
1. Tag tests with appropriate categories for filtering
|
|
2. Use Testcontainers for infrastructure dependencies
|
|
3. Inherit from shared fixtures to avoid duplication
|
|
4. Assert no network calls in air-gap tests
|
|
5. Verify determinism for any serialization output
|
|
6. Use property-based tests (FsCheck) for invariants
|
|
7. Document test purpose in method names
|
|
|
|
### DON'T:
|
|
1. Don't skip tests without documenting why
|
|
2. Don't use `Thread.Sleep` - use proper async waits
|
|
3. Don't hardcode paths - use `AppContext.BaseDirectory`
|
|
4. Don't make network calls in non-interop tests
|
|
5. Don't depend on test execution order
|
|
6. Don't leave test data in shared databases
|
|
|
|
## Environment Variables
|
|
|
|
| Variable | Purpose | Default |
|
|
|----------|---------|---------|
|
|
| `STELLAOPS_OFFLINE_MODE` | Enable offline mode | `false` |
|
|
| `STELLAOPS_OFFLINE_BUNDLE` | Path to offline bundle | - |
|
|
| `STELLAOPS_TEST_POSTGRES` | PostgreSQL connection | Testcontainers |
|
|
| `STELLAOPS_TEST_VALKEY` | Valkey connection | Testcontainers |
|
|
|
|
## Related Documentation
|
|
|
|
- `docs/technical/testing/TEST_SUITE_OVERVIEW.md` - Comprehensive test taxonomy
|
|
- `docs/technical/testing/webservice-test-discipline.md` - WebService test patterns
|
|
- `docs/technical/testing/SPRINT_EXECUTION_PLAYBOOK.md` - Sprint execution guide
|
|
- `docs/dev/fixtures.md` - Fixture maintenance patterns
|
|
|