git.stella-ops.org/docs/testing/TEST_COVERAGE_MATRIX.md

Testing Strategy Coverage Matrix

Purpose: Visual map of test model requirements per module, quality gates, and sprint-to-model relationships.

---

Module-to-Model Coverage Map

Legend

• ✅ Required (from TEST_CATALOG.yml)
• 🟡 Optional (recommended but not mandatory)
• ⬜ Not Applicable

Model Definitions (Quick Reference)

Model	Description	Key Tests
L0	Library/Core	Unit, property, snapshot, determinism
S1	Storage/Postgres	Integration, migrations, idempotency, query ordering
T1	Transport/Queue	Protocol roundtrip, fuzz invalid, delivery semantics, backpressure
C1	Connector/External	Fixtures, snapshot, resilience, security
W1	WebService/API	Contract, authz, OTel, negative
WK1	Worker/Indexer	End-to-end, retries, idempotency, OTel
AN1	Analyzer/SourceGen	Diagnostics, codefixes, golden generated
CLI1	Tool/CLI	Exit codes, golden output, determinism
PERF	Benchmarks	Benchmark, perf smoke, regression thresholds

---

Coverage Matrix

Core Modules

Module	L0	S1	T1	C1	W1	WK1	AN1	CLI1	PERF	Sprint	Tasks
Scanner	✅	✅	✅	⬜	✅	✅	✅	⬜	✅	5100.0009.0001	25
Concelier	✅	✅	⬜	✅	✅	⬜	✅	⬜	⬜	5100.0009.0002	18
Excititor	✅	✅	⬜	✅	✅	✅	⬜	⬜	⬜	5100.0009.0003	21
Policy	✅	✅	⬜	⬜	✅	⬜	⬜	⬜	⬜	5100.0009.0004	15

Security & Compliance Modules

Module	L0	S1	T1	C1	W1	WK1	AN1	CLI1	PERF	Sprint	Tasks
Authority	✅	⬜	⬜	✅	✅	⬜	⬜	⬜	⬜	5100.0009.0005	17
Signer	✅	⬜	⬜	✅	✅	⬜	⬜	⬜	⬜	5100.0009.0006	17
Attestor	✅	⬜	⬜	⬜	✅	⬜	⬜	⬜	⬜	5100.0009.0007	14

Platform Services

Module	L0	S1	T1	C1	W1	WK1	AN1	CLI1	PERF	Sprint	Tasks
Scheduler	✅	✅	⬜	⬜	✅	✅	⬜	⬜	⬜	5100.0009.0008	14
Notify	✅	✅	⬜	✅	✅	✅	⬜	⬜	⬜	5100.0009.0009	18

Client Interfaces

Module	L0	S1	T1	C1	W1	WK1	AN1	CLI1	PERF	Sprint	Tasks
CLI	⬜	⬜	⬜	⬜	⬜	⬜	⬜	✅	⬜	5100.0009.0010	13
UI	⬜	⬜	⬜	⬜	✅	⬜	⬜	⬜	⬜	5100.0009.0011	13

Infrastructure & Platform

Module	L0	S1	T1	C1	W1	WK1	AN1	CLI1	PERF	Sprint	Tasks
EvidenceLocker	✅	✅	⬜	⬜	✅	⬜	⬜	⬜	⬜	5100.0010.0001	16
Graph/Timeline	✅	✅	⬜	⬜	✅	✅	⬜	⬜	⬜	5100.0010.0002	15
Router/Messaging	✅	✅	✅	⬜	✅	⬜	⬜	⬜	⬜	5100.0010.0003	14
AirGap	✅	✅	⬜	⬜	✅	⬜	✅	✅	⬜	5100.0010.0004	17

---

Model Distribution Analysis

Models by Usage Frequency

Model	Modules Using	Percentage	Complexity
L0 (Library/Core)	13/15 modules	87%	HIGH (property tests, snapshots)
W1 (WebService)	13/15 modules	87%	MEDIUM (contract tests, auth)
S1 (Storage)	10/15 modules	67%	HIGH (migrations, idempotency)
C1 (Connectors)	5/15 modules	33%	MEDIUM (fixtures, resilience)
WK1 (Workers)	5/15 modules	33%	MEDIUM (end-to-end, retries)
AN1 (Analyzers)	3/15 modules	20%	HIGH (Roslyn, diagnostics)
T1 (Transport)	2/15 modules	13%	HIGH (protocol compliance)
CLI1 (CLI Tools)	2/15 modules	13%	LOW (exit codes, snapshots)
PERF (Performance)	1/15 modules	7%	MEDIUM (benchmarks, regression)

Complexity Heatmap

High Complexity (>15 tasks per sprint):

• Scanner (25 tasks: L0+AN1+S1+T1+W1+WK1+PERF)
• Excititor (21 tasks: C1+L0+S1+W1+WK1)
• Concelier (18 tasks: C1+L0+S1+W1+AN1)
• Notify (18 tasks: L0+C1+S1+W1+WK1)
• Authority (17 tasks: L0+W1+C1)
• Signer (17 tasks: L0+W1+C1)
• AirGap (17 tasks: L0+AN1+S1+W1+CLI1)

Medium Complexity (10-15 tasks):

• Policy (15 tasks: L0+S1+W1)
• EvidenceLocker (16 tasks: L0+S1+W1)
• Graph/Timeline (15 tasks: L0+S1+W1+WK1)
• Scheduler (14 tasks: L0+S1+W1+WK1)
• Attestor (14 tasks: L0+W1)
• Router/Messaging (14 tasks: L0+T1+W1+S1)
• CLI (13 tasks: CLI1)
• UI (13 tasks: W1)

---

Quality Gate Coverage

Module-Specific Quality Gates (from TEST_CATALOG.yml)

Module	Quality Gates	Enforced By
Scanner	determinism, reachability_evidence, proof_spine	Sprint 5100.0009.0001 Tasks 7-10, 23-25
Concelier	fixture_coverage, normalization_determinism, no_lattice_dependency	Sprint 5100.0009.0002 Tasks 1-7, 8-10, 18
Excititor	preserve_prune_source, format_snapshots, no_lattice_dependency	Sprint 5100.0009.0003 Tasks 6-11, 21
Policy	unknown_budget, verdict_snapshot	Sprint 5100.0009.0004 Tasks 2, 4, 14-15
Authority	scope_enforcement, sign_verify	Sprint 5100.0009.0005 Tasks 3-5, 16-17
Signer	canonical_payloads, sign_verify	Sprint 5100.0009.0006 Tasks 1-3, 15-17
Attestor	rekor_receipts, dsse_verify	Sprint 5100.0009.0007 Tasks 6-8, 2
Scheduler	idempotent_jobs, retry_backoff	Sprint 5100.0009.0008 Tasks 4, 3, 12
Notify	connector_snapshots, retry_semantics	Sprint 5100.0009.0009 Tasks 1-6, 16
CLI	exit_codes, stdout_snapshots	Sprint 5100.0009.0010 Tasks 1-4, 5-8
UI	contract_snapshots, e2e_smoke	Sprint 5100.0009.0011 Tasks 1-2, 7-10

Cross-Cutting Quality Gates

Gate	Applies To	Enforced By
Determinism Contract	Scanner, Excititor, Signer, CLI, AirGap, Concelier	Sprint 5100.0007.0003 (Determinism Gate)
Architecture Boundaries	Concelier, Excititor (must NOT reference Scanner lattice)	Sprint 5100.0007.0007 (Architecture Tests)
Contract Stability	All WebServices (13 modules)	Sprint 5100.0007.0006 (WebService Contract)
Storage Idempotency	All S1 modules (10 modules)	Sprint 5100.0007.0004 (Storage Harness)
Connector Resilience	All C1 modules (5 modules)	Sprint 5100.0007.0005 (Connector Fixtures)

---

CI Lane Coverage

Test Distribution Across CI Lanes

CI Lane	Models	Modules	Sprint Tasks	Est. Runtime
Unit	L0, AN1, CLI1	All 15 modules	~120 tasks	<5 min
Contract	W1	13 modules	~50 tasks	<2 min
Integration	S1, WK1, T1	12 modules	~100 tasks	10-15 min
Security	C1 (security tests), W1 (auth tests)	5 connectors + 13 WebServices	~60 tasks	5-10 min
Performance	PERF	Scanner only	~3 tasks	3-5 min
Live	C1 (live smoke tests)	Concelier, Excititor, Notify, Authority, Signer	~5 tasks (opt-in)	5-10 min (nightly)

CI Lane Dependencies

PR Gate (Must Pass):
├─ Unit Lane (L0, AN1, CLI1) ← Fast feedback
├─ Contract Lane (W1) ← API stability
├─ Architecture Lane (Sprint 5100.0007.0007) ← Boundary enforcement
└─ Integration Lane (S1, WK1, T1) ← Testcontainers

Merge Gate (Must Pass):
├─ All PR Gate lanes
├─ Security Lane (C1 security, W1 auth)
└─ Determinism Lane (Sprint 5100.0007.0003)

Nightly (Optional):
├─ Performance Lane (PERF)
└─ Live Lane (C1 live smoke)

Weekly (Optional):
└─ Competitor Parity (Sprint 5100.0008.0001)

---

Epic-to-Model Coverage

Epic Sprints Support Multiple Models

Epic Sprint	Models Enabled	Consuming Modules	Tasks
5100.0007.0002 (TestKit)	ALL (L0, S1, T1, C1, W1, WK1, AN1, CLI1, PERF)	ALL 15 modules	13
5100.0007.0003 (Determinism)	L0 (determinism), CLI1 (determinism)	Scanner, Excititor, Signer, CLI, AirGap, Concelier	12
5100.0007.0004 (Storage)	S1	10 modules	12
5100.0007.0005 (Connectors)	C1	Concelier, Excititor, Authority, Signer, Notify	12
5100.0007.0006 (WebService)	W1	13 modules	12
5100.0007.0007 (Architecture)	(Cross-cutting)	Concelier, Excititor	17

---

Test Type Distribution

By Test Category (Trait)

Test Category	Model Coverage	Estimated Test Count	CI Lane
Unit	L0, AN1	~150 tests across 13 modules	Unit
Property	L0 (subset)	~40 tests (Scanner, Policy, Scheduler, Router)	Unit
Snapshot	L0, C1, CLI1	~80 tests (all modules with canonical outputs)	Unit/Contract
Integration	S1, WK1, T1	~120 tests across 12 modules	Integration
Contract	W1	~50 tests (13 WebServices × avg 4 endpoints)	Contract
Security	C1 (security), W1 (auth)	~60 tests	Security
Performance	PERF	~3 tests (Scanner only)	Performance
Live	C1 (live smoke)	~5 tests (opt-in, nightly)	Live

---

Coverage Gaps & Recommendations

Current Gaps

1. Performance Testing: Only Scanner has PERF model

   • Recommendation: Add PERF to Policy (policy evaluation latency), Concelier (merge performance), Scheduler (scheduling overhead)

2. Transport Testing: Only Router/Messaging has T1 model

   • Recommendation: Scanner has T1 in TEST_CATALOG.yml but should validate Valkey transport for job queues

3. Live Connector Tests: Only 5 modules have C1 live smoke tests (opt-in)

   • Recommendation: Run weekly, not nightly; treat as early warning system for schema drift

Recommended Additions (Future Sprints)

Module	Missing Model	Justification	Priority
Policy	PERF	Policy evaluation latency critical for real-time decisioning	HIGH
Concelier	PERF	Merge performance affects ingestion throughput	MEDIUM
Scheduler	PERF	Scheduling overhead affects job execution latency	MEDIUM
Scanner	T1 (validate)	Job queue transport (Valkey) should have compliance tests	HIGH
Authority	S1	Token storage/revocation should have migration tests	MEDIUM

---

Summary Statistics

Total Test Models: 9 Total Modules Covered: 15 Total Module Test Sprints: 15 (11 module + 4 infrastructure) Total Epic Sprints: 6 Total Quality Gate Sprints: 1 (Competitor Parity)

Model Usage:

• L0: 13 modules (87%)
• W1: 13 modules (87%)
• S1: 10 modules (67%)
• C1: 5 modules (33%)
• WK1: 5 modules (33%)
• AN1: 3 modules (20%)
• T1: 2 modules (13%)
• CLI1: 2 modules (13%)
• PERF: 1 module (7%)

Estimated Total Tests: ~500 tests across all modules and models

---

Prepared by: Project Management Date: 2025-12-23 Next Review: 2026-01-06 (Week 1 kickoff) Source: docs/testing/TEST_CATALOG.yml, Sprint files 5100.0009.* and 5100.0010.*