165 lines
7.3 KiB
Markdown
165 lines
7.3 KiB
Markdown
# Testing Strategy Models and Lanes (2026)
|
|
|
|
Source advisory: `docs/product/advisories/22-Dec-2026 - Better testing strategy.md`
|
|
Supersedes/extends: `docs/product/advisories/archived/2025-12-21-testing-strategy/20-Dec-2025 - Testing strategy.md`
|
|
|
|
## Purpose
|
|
- Define a single testing taxonomy for all StellaOps project types.
|
|
- Make determinism, offline readiness, and evidence integrity testable by default.
|
|
- Align CI lanes with a shared catalog so coverage is visible and enforceable.
|
|
|
|
## Strategy in brief
|
|
- Use test models (L0, S1, C1, W1, WK1, T1, AN1, CLI1, PERF) to encode required test types.
|
|
- Map every module to one or more models in `docs/technical/testing/TEST_CATALOG.yml`.
|
|
- Run tests through standardized CI lanes (Unit, Contract, Integration, Security, Performance, Live).
|
|
|
|
## Test models (requirements)
|
|
- L0 (Library/Core): unit + property + snapshot + determinism.
|
|
- S1 (Storage/Postgres): migrations + idempotency + concurrency + query ordering.
|
|
- T1 (Transport/Queue): protocol roundtrip + fuzz invalid + delivery semantics.
|
|
- C1 (Connector/External): fixtures + snapshot + resilience + security; optional Live smoke.
|
|
- W1 (WebService/API): contract + authz + OTel trace assertions + negative cases.
|
|
- WK1 (Worker/Indexer): end-to-end job flow + retries + idempotency + telemetry.
|
|
- AN1 (Analyzer/SourceGen): Roslyn harness + diagnostics + golden generated output.
|
|
- CLI1 (Tool/CLI): exit codes + golden output + deterministic formatting.
|
|
- PERF (Benchmark): perf smoke subset + regression thresholds (relative).
|
|
|
|
## Repository foundations
|
|
- TestKit primitives: deterministic time/random, canonical JSON asserts, snapshot helpers, Postgres/Valkey fixtures, OTel capture.
|
|
- Determinism gate: canonical bytes and stable hashes for SBOM/VEX/verdict artifacts.
|
|
- Hybrid reachability posture: graph DSSE mandatory; edge-bundle DSSE optional/targeted with deterministic ordering.
|
|
- Architecture guards: enforce cross-module dependency boundaries (no lattice in Concelier/Excititor).
|
|
- Offline defaults: no network access unless explicitly tagged `Live`.
|
|
|
|
## CI lanes (standard filters)
|
|
- Unit: fast, offline; includes property and snapshot sub-traits.
|
|
- Contract: schema/OpenAPI stability and response envelopes.
|
|
- Integration: Testcontainers-backed service and storage tests.
|
|
- Security: authz/negative tests and security regressions.
|
|
- Performance: perf smoke and benchmark guards.
|
|
- Live: opt-in upstream connector checks (never PR gating by default).
|
|
|
|
## Documentation moments (when to update)
|
|
- New model or required test type: update `docs/technical/testing/TEST_CATALOG.yml`.
|
|
- New lane or gate: update `docs/technical/testing/TEST_SUITE_OVERVIEW.md` and `docs/technical/testing/ci-quality-gates.md`.
|
|
- Module-specific test policy change: update the module dossier under `docs/modules/<module>/`.
|
|
- New fixtures or runnable harnesses: place under `docs/benchmarks/**` or `tests/**` and link here.
|
|
|
|
## Related artifacts
|
|
- Test catalog (source of truth): `docs/technical/testing/TEST_CATALOG.yml`
|
|
- Test suite overview: `docs/technical/testing/TEST_SUITE_OVERVIEW.md`
|
|
- Quality guardrails: `docs/technical/testing/testing-quality-guardrails-implementation.md`
|
|
- Code samples from the advisory: `docs/benchmarks/testing/better-testing-strategy-samples.md`
|
|
|
|
---
|
|
|
|
## Turn #6 Enhancements (2026-01-27)
|
|
|
|
Source advisory: Testing Enhancements (Automation Turn #6)
|
|
Sprint: `docs/implplan/SPRINT_0127_002_DOCS_testing_enhancements_turn6.md`
|
|
|
|
### New test intent categories
|
|
|
|
Every non-trivial test must declare intent. Intent clarifies *why* the behavior exists.
|
|
|
|
```csharp
|
|
public static class TestIntents
|
|
{
|
|
public const string Regulatory = "Regulatory"; // Compliance, audit, legal
|
|
public const string Safety = "Safety"; // Security, fail-secure, crypto
|
|
public const string Performance = "Performance"; // Latency, throughput, resources
|
|
public const string Competitive = "Competitive"; // Parity with competitor tools
|
|
public const string Operational = "Operational"; // Observability, operability
|
|
}
|
|
|
|
// Usage
|
|
[Trait("Intent", TestIntents.Safety)]
|
|
[Trait("Category", "Unit")]
|
|
public void Signer_RejectsExpiredCertificate() { /* ... */ }
|
|
```
|
|
|
|
### New test trait categories
|
|
|
|
| Category | Purpose | Example Usage |
|
|
|----------|---------|---------------|
|
|
| `Intent` | Test intent classification | `[Trait("Intent", "Safety")]` |
|
|
| `Evidence` | Evidence chain validation | `[Trait("Category", "Evidence")]` |
|
|
| `Observability` | OTel/log/metrics contracts | `[Trait("Category", "Observability")]` |
|
|
| `Longevity` | Time-extended stability tests | `[Trait("Category", "Longevity")]` |
|
|
| `Interop` | Cross-version/environment skew | `[Trait("Category", "Interop")]` |
|
|
| `PostIncident` | Tests from production incidents | `[Trait("Category", "PostIncident")]` |
|
|
|
|
### Updated test model requirements
|
|
|
|
| Model | Turn #6 Additions |
|
|
|-------|-------------------|
|
|
| L0 (Library/Core) | + Intent trait required for non-trivial tests |
|
|
| S1 (Storage/Postgres) | + Interop tests for schema version migrations |
|
|
| W1 (WebService/API) | + Observability contract tests (OTel spans, log fields, metrics) |
|
|
| WK1 (Worker/Indexer) | + Longevity tests for memory/connection stability |
|
|
| CLI1 (Tool/CLI) | + PostIncident regression tests |
|
|
|
|
### New CI lanes
|
|
|
|
| Lane | Purpose | Cadence | Gating |
|
|
|------|---------|---------|--------|
|
|
| Evidence | Evidence chain validation, traceability | Per PR | PR-gating for regulatory modules |
|
|
| Longevity | Time-extended stability tests | Nightly | Release-gating |
|
|
| Interop | Cross-version compatibility | Weekly + pre-release | Release-gating |
|
|
|
|
### Observability contract requirements (W1 model)
|
|
|
|
WebService tests must validate:
|
|
- **OTel spans**: required spans exist, attributes present, cardinality bounded.
|
|
- **Structured logs**: required fields present, no PII, appropriate log levels.
|
|
- **Metrics**: required metrics exist, label cardinality bounded, counters monotonic.
|
|
|
|
```csharp
|
|
[Trait("Category", "Observability")]
|
|
[Trait("Intent", "Operational")]
|
|
public async Task Scanner_EmitsRequiredTelemetry()
|
|
{
|
|
using var otel = new OtelCapture();
|
|
await sut.ScanAsync(request);
|
|
|
|
OTelContractAssert.HasRequiredSpans(otel, "ScanImage", "ExtractLayers", "AnalyzeSBOM");
|
|
OTelContractAssert.NoHighCardinalityAttributes(otel, threshold: 100);
|
|
}
|
|
```
|
|
|
|
### Evidence traceability requirements
|
|
|
|
Regulatory tests must link to requirements:
|
|
```csharp
|
|
[Requirement("REQ-EVIDENCE-001")]
|
|
[Trait("Intent", "Regulatory")]
|
|
public void EvidenceBundle_IsImmutableAfterSigning() { /* ... */ }
|
|
```
|
|
|
|
CI generates traceability matrix: requirement -> test -> artifact.
|
|
|
|
### Cross-version testing requirements (Interop)
|
|
|
|
For modules with schema or API versioning:
|
|
- Test N-1 compatibility (current server, previous client).
|
|
- Test N+1 compatibility (previous server, current client).
|
|
- Document compatibility matrix.
|
|
|
|
### Time-extended testing requirements (Longevity)
|
|
|
|
For worker modules (WK1 model):
|
|
- Memory stability: verify no growth under sustained load.
|
|
- Connection pool stability: verify no leaks.
|
|
- Counter drift: verify values remain bounded.
|
|
|
|
Run duration: 1+ hours for nightly, 4+ hours for release validation.
|
|
|
|
### Post-incident testing requirements
|
|
|
|
For P1/P2 production incidents:
|
|
1. Capture event sequence via replay infrastructure.
|
|
2. Generate test scaffold from replay manifest.
|
|
3. Include incident metadata (ID, root cause, severity).
|
|
4. Tag with `[Trait("Category", "PostIncident")]`.
|
|
5. Test failures block releases.
|