stella-ops.org/git.stella-ops.org
1
0
Fork 0
Code Issues Pull Requests Actions 2 Releases Wiki Activity
Files
main
git.stella-ops.org/docs/technical/testing/TEST_COVERAGE_MATRIX.md
master 644887997c test fixes and new product advisories work
2026-01-28 02:30:48 +02:00

16 KiB
Raw Permalink Blame History

Testing Strategy Coverage Matrix

Purpose: Visual map of test model requirements per module, quality gates, and sprint-to-model relationships.

Module-to-Model Coverage Map

Legend

  • ✅ Required (from TEST_CATALOG.yml)
  • 🟡 Optional (recommended but not mandatory)
  • ⬜ Not Applicable

Model Definitions (Quick Reference)

Model Description Key Tests
L0 Library/Core Unit, property, snapshot, determinism
S1 Storage/Postgres Integration, migrations, idempotency, query ordering
T1 Transport/Queue Protocol roundtrip, fuzz invalid, delivery semantics, backpressure
C1 Connector/External Fixtures, snapshot, resilience, security
W1 WebService/API Contract, authz, OTel, negative
WK1 Worker/Indexer End-to-end, retries, idempotency, OTel
AN1 Analyzer/SourceGen Diagnostics, codefixes, golden generated
CLI1 Tool/CLI Exit codes, golden output, determinism
PERF Benchmarks Benchmark, perf smoke, regression thresholds

Coverage Matrix

Core Modules

Module L0 S1 T1 C1 W1 WK1 AN1 CLI1 PERF Sprint Tasks
Scanner ✅ ✅ ✅ ⬜ ✅ ✅ ✅ ⬜ ✅ 5100.0009.0001 25
Concelier ✅ ✅ ⬜ ✅ ✅ ⬜ ✅ ⬜ ⬜ 5100.0009.0002 18
Excititor ✅ ✅ ⬜ ✅ ✅ ✅ ⬜ ⬜ ⬜ 5100.0009.0003 21
Policy ✅ ✅ ⬜ ⬜ ✅ ⬜ ⬜ ⬜ ⬜ 5100.0009.0004 15

Security & Compliance Modules

Module L0 S1 T1 C1 W1 WK1 AN1 CLI1 PERF Sprint Tasks
Authority ✅ ⬜ ⬜ ✅ ✅ ⬜ ⬜ ⬜ ⬜ 5100.0009.0005 17
Signer ✅ ⬜ ⬜ ✅ ✅ ⬜ ⬜ ⬜ ⬜ 5100.0009.0006 17
Attestor ✅ ⬜ ⬜ ⬜ ✅ ⬜ ⬜ ⬜ ⬜ 5100.0009.0007 14

Platform Services

Module L0 S1 T1 C1 W1 WK1 AN1 CLI1 PERF Sprint Tasks
Scheduler ✅ ✅ ⬜ ⬜ ✅ ✅ ⬜ ⬜ ⬜ 5100.0009.0008 14
Notify ✅ ✅ ⬜ ✅ ✅ ✅ ⬜ ⬜ ⬜ 5100.0009.0009 18

Client Interfaces

Module L0 S1 T1 C1 W1 WK1 AN1 CLI1 PERF Sprint Tasks
CLI ⬜ ⬜ ⬜ ⬜ ⬜ ⬜ ⬜ ✅ ⬜ 5100.0009.0010 13
UI ⬜ ⬜ ⬜ ⬜ ✅ ⬜ ⬜ ⬜ ⬜ 5100.0009.0011 13

Infrastructure & Platform

Module L0 S1 T1 C1 W1 WK1 AN1 CLI1 PERF Sprint Tasks
EvidenceLocker ✅ ✅ ⬜ ⬜ ✅ ⬜ ⬜ ⬜ ⬜ 5100.0010.0001 16
Graph/Timeline ✅ ✅ ⬜ ⬜ ✅ ✅ ⬜ ⬜ ⬜ 5100.0010.0002 15
Router/Messaging ✅ ✅ ✅ ⬜ ✅ ⬜ ⬜ ⬜ ⬜ 5100.0010.0003 14
AirGap ✅ ✅ ⬜ ⬜ ✅ ⬜ ✅ ✅ ⬜ 5100.0010.0004 17

Model Distribution Analysis

Models by Usage Frequency

Model Modules Using Percentage Complexity
L0 (Library/Core) 13/15 modules 87% HIGH (property tests, snapshots)
W1 (WebService) 13/15 modules 87% MEDIUM (contract tests, auth)
S1 (Storage) 10/15 modules 67% HIGH (migrations, idempotency)
C1 (Connectors) 5/15 modules 33% MEDIUM (fixtures, resilience)
WK1 (Workers) 5/15 modules 33% MEDIUM (end-to-end, retries)
AN1 (Analyzers) 3/15 modules 20% HIGH (Roslyn, diagnostics)
T1 (Transport) 2/15 modules 13% HIGH (protocol compliance)
CLI1 (CLI Tools) 2/15 modules 13% LOW (exit codes, snapshots)
PERF (Performance) 1/15 modules 7% MEDIUM (benchmarks, regression)

Complexity Heatmap

High Complexity (>15 tasks per sprint):

  • Scanner (25 tasks: L0+AN1+S1+T1+W1+WK1+PERF)
  • Excititor (21 tasks: C1+L0+S1+W1+WK1)
  • Concelier (18 tasks: C1+L0+S1+W1+AN1)
  • Notify (18 tasks: L0+C1+S1+W1+WK1)
  • Authority (17 tasks: L0+W1+C1)
  • Signer (17 tasks: L0+W1+C1)
  • AirGap (17 tasks: L0+AN1+S1+W1+CLI1)

Medium Complexity (10-15 tasks):

  • Policy (15 tasks: L0+S1+W1)
  • EvidenceLocker (16 tasks: L0+S1+W1)
  • Graph/Timeline (15 tasks: L0+S1+W1+WK1)
  • Scheduler (14 tasks: L0+S1+W1+WK1)
  • Attestor (14 tasks: L0+W1)
  • Router/Messaging (14 tasks: L0+T1+W1+S1)
  • CLI (13 tasks: CLI1)
  • UI (13 tasks: W1)

Quality Gate Coverage

Module-Specific Quality Gates (from TEST_CATALOG.yml)

Module Quality Gates Enforced By
Scanner determinism, reachability_evidence, proof_spine Sprint 5100.0009.0001 Tasks 7-10, 23-25
Concelier fixture_coverage, normalization_determinism, no_lattice_dependency Sprint 5100.0009.0002 Tasks 1-7, 8-10, 18
Excititor preserve_prune_source, format_snapshots, no_lattice_dependency Sprint 5100.0009.0003 Tasks 6-11, 21
Policy unknown_budget, verdict_snapshot Sprint 5100.0009.0004 Tasks 2, 4, 14-15
Authority scope_enforcement, sign_verify Sprint 5100.0009.0005 Tasks 3-5, 16-17
Signer canonical_payloads, sign_verify Sprint 5100.0009.0006 Tasks 1-3, 15-17
Attestor rekor_receipts, dsse_verify Sprint 5100.0009.0007 Tasks 6-8, 2
Scheduler idempotent_jobs, retry_backoff Sprint 5100.0009.0008 Tasks 4, 3, 12
Notify connector_snapshots, retry_semantics Sprint 5100.0009.0009 Tasks 1-6, 16
CLI exit_codes, stdout_snapshots Sprint 5100.0009.0010 Tasks 1-4, 5-8
UI contract_snapshots, e2e_smoke Sprint 5100.0009.0011 Tasks 1-2, 7-10

Cross-Cutting Quality Gates

Gate Applies To Enforced By
Determinism Contract Scanner, Excititor, Signer, CLI, AirGap, Concelier Sprint 5100.0007.0003 (Determinism Gate)
Architecture Boundaries Concelier, Excititor (must NOT reference Scanner lattice) Sprint 5100.0007.0007 (Architecture Tests)
Contract Stability All WebServices (13 modules) Sprint 5100.0007.0006 (WebService Contract)
Storage Idempotency All S1 modules (10 modules) Sprint 5100.0007.0004 (Storage Harness)
Connector Resilience All C1 modules (5 modules) Sprint 5100.0007.0005 (Connector Fixtures)

CI Lane Coverage

Test Distribution Across CI Lanes

CI Lane Models Modules Sprint Tasks Est. Runtime
Unit L0, AN1, CLI1 All 15 modules ~120 tasks <5 min
Contract W1 13 modules ~50 tasks <2 min
Integration S1, WK1, T1 12 modules ~100 tasks 10-15 min
Security C1 (security tests), W1 (auth tests) 5 connectors + 13 WebServices ~60 tasks 5-10 min
Performance PERF Scanner only ~3 tasks 3-5 min
Live C1 (live smoke tests) Concelier, Excititor, Notify, Authority, Signer ~5 tasks (opt-in) 5-10 min (nightly)

CI Lane Dependencies

PR Gate (Must Pass):
├─ Unit Lane (L0, AN1, CLI1) ← Fast feedback
├─ Contract Lane (W1) ← API stability
├─ Architecture Lane (Sprint 5100.0007.0007) ← Boundary enforcement
└─ Integration Lane (S1, WK1, T1) ← Testcontainers

Merge Gate (Must Pass):
├─ All PR Gate lanes
├─ Security Lane (C1 security, W1 auth)
└─ Determinism Lane (Sprint 5100.0007.0003)

Nightly (Optional):
├─ Performance Lane (PERF)
└─ Live Lane (C1 live smoke)

Weekly (Optional):
└─ Competitor Parity (Sprint 5100.0008.0001)

Epic-to-Model Coverage

Epic Sprints Support Multiple Models

Epic Sprint Models Enabled Consuming Modules Tasks
5100.0007.0002 (TestKit) ALL (L0, S1, T1, C1, W1, WK1, AN1, CLI1, PERF) ALL 15 modules 13
5100.0007.0003 (Determinism) L0 (determinism), CLI1 (determinism) Scanner, Excititor, Signer, CLI, AirGap, Concelier 12
5100.0007.0004 (Storage) S1 10 modules 12
5100.0007.0005 (Connectors) C1 Concelier, Excititor, Authority, Signer, Notify 12
5100.0007.0006 (WebService) W1 13 modules 12
5100.0007.0007 (Architecture) (Cross-cutting) Concelier, Excititor 17

Test Type Distribution

By Test Category (Trait)

Test Category Model Coverage Estimated Test Count CI Lane
Unit L0, AN1 ~150 tests across 13 modules Unit
Property L0 (subset) ~40 tests (Scanner, Policy, Scheduler, Router) Unit
Snapshot L0, C1, CLI1 ~80 tests (all modules with canonical outputs) Unit/Contract
Integration S1, WK1, T1 ~120 tests across 12 modules Integration
Contract W1 ~50 tests (13 WebServices × avg 4 endpoints) Contract
Security C1 (security), W1 (auth) ~60 tests Security
Performance PERF ~3 tests (Scanner only) Performance
Live C1 (live smoke) ~5 tests (opt-in, nightly) Live

Coverage Gaps & Recommendations

Current Gaps

  1. Performance Testing: Only Scanner has PERF model

    • Recommendation: Add PERF to Policy (policy evaluation latency), Concelier (merge performance), Scheduler (scheduling overhead)

  2. Transport Testing: Only Router/Messaging has T1 model

    • Recommendation: Scanner has T1 in TEST_CATALOG.yml but should validate Valkey transport for job queues

  3. Live Connector Tests: Only 5 modules have C1 live smoke tests (opt-in)

    • Recommendation: Run weekly, not nightly; treat as early warning system for schema drift

Recommended Additions (Future Sprints)

Module Missing Model Justification Priority
Policy PERF Policy evaluation latency critical for real-time decisioning HIGH
Concelier PERF Merge performance affects ingestion throughput MEDIUM
Scheduler PERF Scheduling overhead affects job execution latency MEDIUM
Scanner T1 (validate) Job queue transport (Valkey) should have compliance tests HIGH
Authority S1 Token storage/revocation should have migration tests MEDIUM

Summary Statistics

Total Test Models: 9 Total Modules Covered: 15 Total Module Test Sprints: 15 (11 module + 4 infrastructure) Total Epic Sprints: 6 Total Quality Gate Sprints: 1 (Competitor Parity)

Model Usage:

  • L0: 13 modules (87%)
  • W1: 13 modules (87%)
  • S1: 10 modules (67%)
  • C1: 5 modules (33%)
  • WK1: 5 modules (33%)
  • AN1: 3 modules (20%)
  • T1: 2 modules (13%)
  • CLI1: 2 modules (13%)
  • PERF: 1 module (7%)

Estimated Total Tests: ~500 tests across all modules and models

Turn #6 Testing Enhancements Coverage

New Coverage Dimensions (Sprint 0127.002)

The following dimensions track adoption of Turn #6 testing practices across modules:

Dimension Description Target Coverage
Intent Tags Tests with [Intent] attribute declaring regulatory/safety/performance/competitive/operational 100% non-trivial tests in Policy, Authority, Signer, Attestor
Observability Contracts W1 tests with OTel schema validation, log field contracts 100% of W1 tests
Evidence Traceability Tests with [Requirement] attribute linking to requirements 100% of regulatory-tagged tests
Longevity Tests Memory stability, counter drift, connection pool tests Scanner, Scheduler, Notify workers
Interop Tests N-1/N+1 version compatibility tests EvidenceLocker, Policy (schema-dependent)
Environment Skew Tests across infrastructure profiles (network latency, resource limits) Integration tests

Turn #6 Coverage Matrix

Module Intent Tags Observability Evidence Longevity Interop Skew
Policy ✅ Pilot 🟡 ✅ Pilot 🟡 🟡 ⬜
EvidenceLocker 🟡 🟡 ✅ Pilot 🟡 ✅ 🟡
Scanner 🟡 ✅ Pilot 🟡 ✅ 🟡 🟡
Authority 🟡 🟡 🟡 ⬜ 🟡 ⬜
Signer 🟡 🟡 🟡 ⬜ 🟡 ⬜
Attestor 🟡 🟡 🟡 ⬜ 🟡 ⬜
Scheduler 🟡 🟡 🟡 ✅ ⬜ 🟡
Notify 🟡 🟡 🟡 ✅ ⬜ ⬜

Legend:

  • ✅ Pilot implementation complete
  • 🟡 Recommended, not yet implemented
  • ⬜ Not applicable

Turn #6 TestKit Components

Component Location Purpose Status
IntentAttribute TestKit/Traits/IntentAttribute.cs Tag tests with intent ✅ Complete
IntentAnalyzer TestKit.Analyzers/IntentAnalyzer.cs Detect missing intent tags ✅ Complete
OTelContractAssert TestKit/Observability/OTelContractAssert.cs Span/attribute validation ✅ Complete
LogContractAssert TestKit/Observability/LogContractAssert.cs Log field validation ✅ Complete
MetricsContractAssert TestKit/Observability/MetricsContractAssert.cs Cardinality bounds ✅ Complete
RequirementAttribute TestKit/Evidence/RequirementAttribute.cs Link tests to requirements ✅ Complete
EvidenceChainAssert TestKit/Evidence/EvidenceChainAssert.cs Hash/immutability validation ✅ Complete
EvidenceChainReporter TestKit/Evidence/EvidenceChainReporter.cs Traceability matrix ✅ Complete
IncidentTestGenerator TestKit/Incident/IncidentTestGenerator.cs Post-incident test scaffolds ✅ Complete
SchemaVersionMatrix TestKit/Interop/SchemaVersionMatrix.cs Version compatibility ✅ Complete
VersionCompatibilityFixture TestKit/Interop/VersionCompatibilityFixture.cs N-1/N+1 testing ✅ Complete
StabilityMetrics TestKit/Longevity/StabilityMetrics.cs Memory/counter tracking ✅ Complete
StabilityTestRunner TestKit/Longevity/StabilityTestRunner.cs Time-extended tests ✅ Complete
EnvironmentProfile TestKit/Environment/EnvironmentProfile.cs Infrastructure profiles ✅ Complete
SkewTestRunner TestKit/Environment/SkewTestRunner.cs Cross-profile testing ✅ Complete

Turn #6 Test Categories

New categories added to TestCategories.cs:

Category Filter CI Lane Gating
PostIncident Category=PostIncident Release P1/P2 block
EvidenceChain Category=EvidenceChain Merge Block
Longevity Category=Longevity Nightly Warning
Interop Category=Interop Release Block
EnvironmentSkew Category=EnvironmentSkew Nightly Warning

Coverage Targets (End of Q1 2026)

Dimension Current Baseline Target Tracking
Intent Tags (Policy, Authority, Signer, Attestor) 5 tests 100% non-trivial IntentCoverageReport
Observability Contracts (W1 tests) 5 tests 100% OTelContractAssert usage
Evidence Traceability (Regulatory tests) 3 tests 100% EvidenceChainReporter
Longevity Tests (Worker modules) 0 tests 1 per worker StabilityTestRunner usage
Interop Tests (Schema modules) 0 tests 1 per schema SchemaVersionMatrix usage

Prepared by: Project Management Date: 2026-01-27 Next Review: 2026-02-03 (Turn #6 adoption review) Source: docs/technical/testing/TEST_CATALOG.yml, Sprint files 5100.0009.* and 5100.0010.*, SPRINT_0127_002_DOCS_testing_enhancements_turn6.md