stella-ops.org/git.stella-ops.org
1
0
Fork 0
Code Issues Pull Requests Actions 2 Releases Wiki Activity
Files
644887997c334d23495db2c4e61092f1f57ca027
git.stella-ops.org/docs/technical/testing/testing-strategy-models.md
master 644887997c test fixes and new product advisories work
2026-01-28 02:30:48 +02:00

7.3 KiB
Raw Blame History

Testing Strategy Models and Lanes (2026)

Source advisory: docs/product/advisories/22-Dec-2026 - Better testing strategy.md
Supersedes/extends: docs/product/advisories/archived/2025-12-21-testing-strategy/20-Dec-2025 - Testing strategy.md

Purpose

  • Define a single testing taxonomy for all StellaOps project types.
  • Make determinism, offline readiness, and evidence integrity testable by default.
  • Align CI lanes with a shared catalog so coverage is visible and enforceable.

Strategy in brief

  • Use test models (L0, S1, C1, W1, WK1, T1, AN1, CLI1, PERF) to encode required test types.
  • Map every module to one or more models in docs/technical/testing/TEST_CATALOG.yml.
  • Run tests through standardized CI lanes (Unit, Contract, Integration, Security, Performance, Live).

Test models (requirements)

  • L0 (Library/Core): unit + property + snapshot + determinism.
  • S1 (Storage/Postgres): migrations + idempotency + concurrency + query ordering.
  • T1 (Transport/Queue): protocol roundtrip + fuzz invalid + delivery semantics.
  • C1 (Connector/External): fixtures + snapshot + resilience + security; optional Live smoke.
  • W1 (WebService/API): contract + authz + OTel trace assertions + negative cases.
  • WK1 (Worker/Indexer): end-to-end job flow + retries + idempotency + telemetry.
  • AN1 (Analyzer/SourceGen): Roslyn harness + diagnostics + golden generated output.
  • CLI1 (Tool/CLI): exit codes + golden output + deterministic formatting.
  • PERF (Benchmark): perf smoke subset + regression thresholds (relative).

Repository foundations

  • TestKit primitives: deterministic time/random, canonical JSON asserts, snapshot helpers, Postgres/Valkey fixtures, OTel capture.
  • Determinism gate: canonical bytes and stable hashes for SBOM/VEX/verdict artifacts.
  • Hybrid reachability posture: graph DSSE mandatory; edge-bundle DSSE optional/targeted with deterministic ordering.
  • Architecture guards: enforce cross-module dependency boundaries (no lattice in Concelier/Excititor).
  • Offline defaults: no network access unless explicitly tagged Live.

CI lanes (standard filters)

  • Unit: fast, offline; includes property and snapshot sub-traits.
  • Contract: schema/OpenAPI stability and response envelopes.
  • Integration: Testcontainers-backed service and storage tests.
  • Security: authz/negative tests and security regressions.
  • Performance: perf smoke and benchmark guards.
  • Live: opt-in upstream connector checks (never PR gating by default).

Documentation moments (when to update)

  • New model or required test type: update docs/technical/testing/TEST_CATALOG.yml.
  • New lane or gate: update docs/technical/testing/TEST_SUITE_OVERVIEW.md and docs/technical/testing/ci-quality-gates.md.
  • Module-specific test policy change: update the module dossier under docs/modules/<module>/.
  • New fixtures or runnable harnesses: place under docs/benchmarks/** or tests/** and link here.

Related artifacts

  • Test catalog (source of truth): docs/technical/testing/TEST_CATALOG.yml
  • Test suite overview: docs/technical/testing/TEST_SUITE_OVERVIEW.md
  • Quality guardrails: docs/technical/testing/testing-quality-guardrails-implementation.md
  • Code samples from the advisory: docs/benchmarks/testing/better-testing-strategy-samples.md

Turn #6 Enhancements (2026-01-27)

Source advisory: Testing Enhancements (Automation Turn #6) Sprint: docs/implplan/SPRINT_0127_002_DOCS_testing_enhancements_turn6.md

New test intent categories

Every non-trivial test must declare intent. Intent clarifies why the behavior exists.

public static class TestIntents
{
    public const string Regulatory = "Regulatory";   // Compliance, audit, legal
    public const string Safety = "Safety";           // Security, fail-secure, crypto
    public const string Performance = "Performance"; // Latency, throughput, resources
    public const string Competitive = "Competitive"; // Parity with competitor tools
    public const string Operational = "Operational"; // Observability, operability
}

// Usage
[Trait("Intent", TestIntents.Safety)]
[Trait("Category", "Unit")]
public void Signer_RejectsExpiredCertificate() { /* ... */ }

New test trait categories

Category Purpose Example Usage
Intent Test intent classification [Trait("Intent", "Safety")]
Evidence Evidence chain validation [Trait("Category", "Evidence")]
Observability OTel/log/metrics contracts [Trait("Category", "Observability")]
Longevity Time-extended stability tests [Trait("Category", "Longevity")]
Interop Cross-version/environment skew [Trait("Category", "Interop")]
PostIncident Tests from production incidents [Trait("Category", "PostIncident")]

Updated test model requirements

Model Turn #6 Additions
L0 (Library/Core) + Intent trait required for non-trivial tests
S1 (Storage/Postgres) + Interop tests for schema version migrations
W1 (WebService/API) + Observability contract tests (OTel spans, log fields, metrics)
WK1 (Worker/Indexer) + Longevity tests for memory/connection stability
CLI1 (Tool/CLI) + PostIncident regression tests

New CI lanes

Lane Purpose Cadence Gating
Evidence Evidence chain validation, traceability Per PR PR-gating for regulatory modules
Longevity Time-extended stability tests Nightly Release-gating
Interop Cross-version compatibility Weekly + pre-release Release-gating

Observability contract requirements (W1 model)

WebService tests must validate:

  • OTel spans: required spans exist, attributes present, cardinality bounded.
  • Structured logs: required fields present, no PII, appropriate log levels.
  • Metrics: required metrics exist, label cardinality bounded, counters monotonic.
[Trait("Category", "Observability")]
[Trait("Intent", "Operational")]
public async Task Scanner_EmitsRequiredTelemetry()
{
    using var otel = new OtelCapture();
    await sut.ScanAsync(request);

    OTelContractAssert.HasRequiredSpans(otel, "ScanImage", "ExtractLayers", "AnalyzeSBOM");
    OTelContractAssert.NoHighCardinalityAttributes(otel, threshold: 100);
}

Evidence traceability requirements

Regulatory tests must link to requirements:

[Requirement("REQ-EVIDENCE-001")]
[Trait("Intent", "Regulatory")]
public void EvidenceBundle_IsImmutableAfterSigning() { /* ... */ }

CI generates traceability matrix: requirement -> test -> artifact.

Cross-version testing requirements (Interop)

For modules with schema or API versioning:

  • Test N-1 compatibility (current server, previous client).
  • Test N+1 compatibility (previous server, current client).
  • Document compatibility matrix.

Time-extended testing requirements (Longevity)

For worker modules (WK1 model):

  • Memory stability: verify no growth under sustained load.
  • Connection pool stability: verify no leaks.
  • Counter drift: verify values remain bounded.

Run duration: 1+ hours for nightly, 4+ hours for release validation.

Post-incident testing requirements

For P1/P2 production incidents:

  1. Capture event sequence via replay infrastructure.
  2. Generate test scaffold from replay manifest.
  3. Include incident metadata (ID, root cause, severity).
  4. Tag with [Trait("Category", "PostIncident")].
  5. Test failures block releases.