git.stella-ops.org/src/__Tests/AGENTS.md

src/__Tests/AGENTS.md

Purpose & Scope

This directory contains all global test infrastructure, benchmarks, datasets, and shared testing libraries for the StellaOps platform.

• Working directory: src/__Tests/
• Roles: QA engineer, performance/bench engineer, integration test developer, docs contributor

Directory Structure

src/__Tests/
├── __Libraries/           # Shared testing libraries
│   ├── StellaOps.Infrastructure.Postgres.Testing/
│   ├── StellaOps.Messaging.Testing/
│   ├── StellaOps.Testing.AirGap/
│   ├── StellaOps.Testing.Determinism/
│   ├── StellaOps.Testing.Manifests/
│   ├── StellaOps.Concelier.Testing/
│   └── StellaOps.Router.Testing/
├── __Benchmarks/          # Golden corpus, CVE findings, determinism fixtures
│   ├── golden-corpus/     # Canonical test cases (severity, VEX, reachability)
│   ├── findings/          # CVE bundles with reachability evidence
│   ├── reachability-benchmark/  # Public multi-language benchmark
│   ├── determinism/       # Determinism test fixtures
│   └── tools/             # Verification utilities
├── __Datasets/            # Ground truth samples, schemas
│   └── reachability/      # Reachability ground truth
├── Integration/           # Cross-module integration tests
├── acceptance/            # Acceptance test packs
├── load/                  # k6 load tests
├── security/              # OWASP security tests
├── chaos/                 # Chaos engineering tests
├── AirGap/                # Offline operation tests
├── reachability/          # Reachability analysis tests
├── fixtures/              # Shared test fixtures (offline-bundle, images, sboms)
└── ...                    # Other test categories

Required Reading

Before working in this directory:

• docs/README.md
• docs/19_TEST_SUITE_OVERVIEW.md
• src/__Tests/__Benchmarks/README.md
• Sprint-specific guidance for corpus/bench artifacts

Test Categories

Use the standardized test categories from StellaOps.TestKit.TestCategories:

using StellaOps.TestKit;

// PR-GATING TESTS (run on every push/PR)
[Trait("Category", TestCategories.Unit)]           // Fast, in-memory, no external dependencies
[Trait("Category", TestCategories.Architecture)]   // Module dependency rules, naming conventions
[Trait("Category", TestCategories.Contract)]       // API/WebService contract verification
[Trait("Category", TestCategories.Integration)]    // Testcontainers, PostgreSQL, Valkey
[Trait("Category", TestCategories.Security)]       // Cryptographic validation, vulnerability scanning
[Trait("Category", TestCategories.Golden)]         // Output comparison against known-good references

// SCHEDULED/ON-DEMAND TESTS
[Trait("Category", TestCategories.Performance)]    // Performance measurements, SLO enforcement
[Trait("Category", TestCategories.Benchmark)]      // BenchmarkDotNet measurements
[Trait("Category", TestCategories.AirGap)]         // Offline/air-gapped environment validation
[Trait("Category", TestCategories.Chaos)]          // Fault injection, failure recovery
[Trait("Category", TestCategories.Determinism)]    // Reproducibility, stable ordering, idempotency
[Trait("Category", TestCategories.Resilience)]     // Retry policies, circuit breakers, timeouts
[Trait("Category", TestCategories.Observability)]  // OpenTelemetry traces, metrics, logging

// OTHER CATEGORIES
[Trait("Category", TestCategories.Property)]       // FsCheck/generative testing for invariants
[Trait("Category", TestCategories.Snapshot)]       // Golden master regression testing
[Trait("Category", TestCategories.Live)]           // Require external services (Rekor, feeds)

CI/CD Integration

Tests are discovered dynamically by .gitea/workflows/test-matrix.yml which runs all *.Tests.csproj files with Category filtering:

• PR-Gating: Unit, Architecture, Contract, Integration, Security, Golden
• Scheduled: Performance, Benchmark (daily)
• On-Demand: AirGap, Chaos, Determinism, Resilience, Observability

Validation

Run the validation script to ensure all tests have Category traits:

python devops/scripts/validate-test-traits.py           # Report coverage
python devops/scripts/validate-test-traits.py --fix     # Add default Unit trait
python devops/scripts/validate-test-traits.py --json    # JSON output for CI

Key Patterns

1. PostgreSQL Integration Tests

Use the shared fixture from __Libraries/StellaOps.Infrastructure.Postgres.Testing:

public class MyIntegrationTests : IClassFixture<MyPostgresFixture>
{
    private readonly MyPostgresFixture _fixture;

    public MyIntegrationTests(MyPostgresFixture fixture)
    {
        _fixture = fixture;
    }

    [Fact]
    public async Task MyTest()
    {
        // _fixture.ConnectionString is available
        // _fixture.TruncateAllTablesAsync() for cleanup
    }
}

2. Air-Gap Tests

Inherit from NetworkIsolatedTestBase for network-free tests:

[Trait("Category", "AirGap")]
public class OfflineTests : NetworkIsolatedTestBase
{
    [Fact]
    public async Task Test_WorksOffline()
    {
        AssertNoNetworkCalls();  // Fails if network accessed
    }

    protected string GetOfflineBundlePath() =>
        Path.Combine(AppContext.BaseDirectory, "fixtures", "offline-bundle");
}

3. Determinism Tests

Use DeterminismVerifier to ensure reproducibility:

[Fact]
public void Output_IsDeterministic()
{
    var verifier = new DeterminismVerifier();
    var result = verifier.Verify(myObject, iterations: 10);
    result.IsDeterministic.Should().BeTrue();
}

4. Golden Corpus Tests

Reference cases from __Benchmarks/golden-corpus/:

[Theory]
[MemberData(nameof(GetCorpusCases))]
public async Task Corpus_Case_Passes(string caseId)
{
    var testCase = CorpusLoader.Load(caseId);
    var result = await ProcessAsync(testCase.Input);
    result.Should().BeEquivalentTo(testCase.Expected);
}

Working Agreements

1. Determinism: Stable ordering, fixed seeds, UTC timestamps
2. Offline-first: No network dependencies unless explicitly required
3. Testcontainers: Use PostgreSQL fixtures from __Libraries/
4. Air-gap validation: Inherit from NetworkIsolatedTestBase
5. Golden corpus: Reference cases from __Benchmarks/golden-corpus/
6. Fixtures: Keep ASCII and reproducible; avoid oversized binaries

Module Tests vs Global Tests

• Module tests: Stay in src/<Module>/__Tests/ - component-specific testing
• Global tests: Go in src/__Tests/ - cross-cutting, infrastructure, benchmarks, integration

Rules for Test Development

DO:

1. Tag tests with appropriate categories for filtering
2. Use Testcontainers for infrastructure dependencies
3. Inherit from shared fixtures to avoid duplication
4. Assert no network calls in air-gap tests
5. Verify determinism for any serialization output
6. Use property-based tests (FsCheck) for invariants
7. Document test purpose in method names

DON'T:

1. Don't skip tests without documenting why
2. Don't use Thread.Sleep - use proper async waits
3. Don't hardcode paths - use AppContext.BaseDirectory
4. Don't make network calls in non-interop tests
5. Don't depend on test execution order
6. Don't leave test data in shared databases

Environment Variables

Variable	Purpose	Default
STELLAOPS_OFFLINE_MODE	Enable offline mode	false
STELLAOPS_OFFLINE_BUNDLE	Path to offline bundle	-
STELLAOPS_TEST_POSTGRES	PostgreSQL connection	Testcontainers
STELLAOPS_TEST_VALKEY	Valkey connection	Testcontainers

Related Documentation

• docs/19_TEST_SUITE_OVERVIEW.md - Comprehensive test taxonomy
• docs/testing/webservice-test-discipline.md - WebService test patterns
• docs/testing/SPRINT_EXECUTION_PLAYBOOK.md - Sprint execution guide
• docs/dev/fixtures.md - Fixture maintenance patterns