Testing Strategy Models and Lanes (2026)

Source advisory: docs/product/advisories/22-Dec-2026 - Better testing strategy.md
Supersedes/extends: docs/product/advisories/archived/2025-12-21-testing-strategy/20-Dec-2025 - Testing strategy.md

Purpose

Define a single testing taxonomy for all StellaOps project types.
Make determinism, offline readiness, and evidence integrity testable by default.
Align CI lanes with a shared catalog so coverage is visible and enforceable.

Strategy in brief

Use test models (L0, S1, C1, W1, WK1, T1, AN1, CLI1, PERF) to encode required test types.
Map every module to one or more models in docs/technical/testing/TEST_CATALOG.yml.
Run tests through standardized CI lanes (Unit, Contract, Integration, Security, Performance, Live).

Test models (requirements)

L0 (Library/Core): unit + property + snapshot + determinism.
S1 (Storage/Postgres): migrations + idempotency + concurrency + query ordering.
T1 (Transport/Queue): protocol roundtrip + fuzz invalid + delivery semantics.
C1 (Connector/External): fixtures + snapshot + resilience + security; optional Live smoke.
W1 (WebService/API): contract + authz + OTel trace assertions + negative cases.
WK1 (Worker/Indexer): end-to-end job flow + retries + idempotency + telemetry.
AN1 (Analyzer/SourceGen): Roslyn harness + diagnostics + golden generated output.
CLI1 (Tool/CLI): exit codes + golden output + deterministic formatting.
PERF (Benchmark): perf smoke subset + regression thresholds (relative).

Repository foundations

TestKit primitives: deterministic time/random, canonical JSON asserts, snapshot helpers, Postgres/Valkey fixtures, OTel capture.
Determinism gate: canonical bytes and stable hashes for SBOM/VEX/verdict artifacts.
Hybrid reachability posture: graph DSSE mandatory; edge-bundle DSSE optional/targeted with deterministic ordering.
Architecture guards: enforce cross-module dependency boundaries (no lattice in Concelier/Excititor).
Offline defaults: no network access unless explicitly tagged Live.

CI lanes (standard filters)

Unit: fast, offline; includes property and snapshot sub-traits.
Contract: schema/OpenAPI stability and response envelopes.
Integration: Testcontainers-backed service and storage tests.
Security: authz/negative tests and security regressions.
Performance: perf smoke and benchmark guards.
Live: opt-in upstream connector checks (never PR gating by default).

Documentation moments (when to update)

New model or required test type: update docs/technical/testing/TEST_CATALOG.yml.
New lane or gate: update docs/technical/testing/TEST_SUITE_OVERVIEW.md and docs/technical/testing/ci-quality-gates.md.
Module-specific test policy change: update the module dossier under docs/modules/<module>/.
New fixtures or runnable harnesses: place under docs/benchmarks/** or tests/** and link here.

Test catalog (source of truth): docs/technical/testing/TEST_CATALOG.yml
Test suite overview: docs/technical/testing/TEST_SUITE_OVERVIEW.md
Quality guardrails: docs/technical/testing/testing-quality-guardrails-implementation.md
Code samples from the advisory: docs/benchmarks/testing/better-testing-strategy-samples.md

3.2 KiB Raw Blame History