dac8e10e36
## Summary
This commit completes Phase 2 of the configuration-driven crypto architecture, achieving
100% crypto compliance by eliminating all hardcoded cryptographic implementations.
## Key Changes
### Phase 1: Plugin Loader Infrastructure
- **Plugin Discovery System**: Created StellaOps.Cryptography.PluginLoader with manifest-based loading
- **Configuration Model**: Added CryptoPluginConfiguration with regional profiles support
- **Dependency Injection**: Extended DI to support plugin-based crypto provider registration
- **Regional Configs**: Created appsettings.crypto.{international,russia,eu,china}.yaml
- **CI Workflow**: Added .gitea/workflows/crypto-compliance.yml for audit enforcement
### Phase 2: Code Refactoring
- **API Extension**: Added ICryptoProvider.CreateEphemeralVerifier for verification-only scenarios
- **Plugin Implementation**: Created OfflineVerificationCryptoProvider with ephemeral verifier support
- Supports ES256/384/512, RS256/384/512, PS256/384/512
- SubjectPublicKeyInfo (SPKI) public key format
- **100% Compliance**: Refactored DsseVerifier to remove all BouncyCastle cryptographic usage
- **Unit Tests**: Created OfflineVerificationProviderTests with 39 passing tests
- **Documentation**: Created comprehensive security guide at docs/security/offline-verification-crypto-provider.md
- **Audit Infrastructure**: Created scripts/audit-crypto-usage.ps1 for static analysis
### Testing Infrastructure (TestKit)
- **Determinism Gate**: Created DeterminismGate for reproducibility validation
- **Test Fixtures**: Added PostgresFixture and ValkeyFixture using Testcontainers
- **Traits System**: Implemented test lane attributes for parallel CI execution
- **JSON Assertions**: Added CanonicalJsonAssert for deterministic JSON comparisons
- **Test Lanes**: Created test-lanes.yml workflow for parallel test execution
### Documentation
- **Architecture**: Created CRYPTO_CONFIGURATION_DRIVEN_ARCHITECTURE.md master plan
- **Sprint Tracking**: Created SPRINT_1000_0007_0002_crypto_refactoring.md (COMPLETE)
- **API Documentation**: Updated docs2/cli/crypto-plugins.md and crypto.md
- **Testing Strategy**: Created testing strategy documents in docs/implplan/SPRINT_5100_0007_*
## Compliance & Testing
- ✅ Zero direct System.Security.Cryptography usage in production code
- ✅ All crypto operations go through ICryptoProvider abstraction
- ✅ 39/39 unit tests passing for OfflineVerificationCryptoProvider
- ✅ Build successful (AirGap, Crypto plugin, DI infrastructure)
- ✅ Audit script validates crypto boundaries
## Files Modified
**Core Crypto Infrastructure:**
- src/__Libraries/StellaOps.Cryptography/CryptoProvider.cs (API extension)
- src/__Libraries/StellaOps.Cryptography/CryptoSigningKey.cs (verification-only constructor)
- src/__Libraries/StellaOps.Cryptography/EcdsaSigner.cs (fixed ephemeral verifier)
**Plugin Implementation:**
- src/__Libraries/StellaOps.Cryptography.Plugin.OfflineVerification/ (new)
- src/__Libraries/StellaOps.Cryptography.PluginLoader/ (new)
**Production Code Refactoring:**
- src/AirGap/StellaOps.AirGap.Importer/Validation/DsseVerifier.cs (100% compliant)
**Tests:**
- src/__Libraries/__Tests/StellaOps.Cryptography.Plugin.OfflineVerification.Tests/ (new, 39 tests)
- src/__Libraries/__Tests/StellaOps.Cryptography.PluginLoader.Tests/ (new)
**Configuration:**
- etc/crypto-plugins-manifest.json (plugin registry)
- etc/appsettings.crypto.*.yaml (regional profiles)
**Documentation:**
- docs/security/offline-verification-crypto-provider.md (600+ lines)
- docs/implplan/CRYPTO_CONFIGURATION_DRIVEN_ARCHITECTURE.md (master plan)
- docs/implplan/SPRINT_1000_0007_0002_crypto_refactoring.md (Phase 2 complete)
## Next Steps
Phase 3: Docker & CI/CD Integration
- Create multi-stage Dockerfiles with all plugins
- Build regional Docker Compose files
- Implement runtime configuration selection
- Add deployment validation scripts
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
263 lines
11 KiB
Markdown
263 lines
11 KiB
Markdown
# Testing Strategy Coverage Matrix
|
||
|
||
> **Purpose:** Visual map of test model requirements per module, quality gates, and sprint-to-model relationships.
|
||
|
||
---
|
||
|
||
## Module-to-Model Coverage Map
|
||
|
||
### Legend
|
||
- ✅ **Required** (from TEST_CATALOG.yml)
|
||
- 🟡 **Optional** (recommended but not mandatory)
|
||
- ⬜ **Not Applicable**
|
||
|
||
### Model Definitions (Quick Reference)
|
||
| Model | Description | Key Tests |
|
||
|-------|-------------|-----------|
|
||
| **L0** | Library/Core | Unit, property, snapshot, determinism |
|
||
| **S1** | Storage/Postgres | Integration, migrations, idempotency, query ordering |
|
||
| **T1** | Transport/Queue | Protocol roundtrip, fuzz invalid, delivery semantics, backpressure |
|
||
| **C1** | Connector/External | Fixtures, snapshot, resilience, security |
|
||
| **W1** | WebService/API | Contract, authz, OTel, negative |
|
||
| **WK1** | Worker/Indexer | End-to-end, retries, idempotency, OTel |
|
||
| **AN1** | Analyzer/SourceGen | Diagnostics, codefixes, golden generated |
|
||
| **CLI1** | Tool/CLI | Exit codes, golden output, determinism |
|
||
| **PERF** | Benchmarks | Benchmark, perf smoke, regression thresholds |
|
||
|
||
---
|
||
|
||
## Coverage Matrix
|
||
|
||
### Core Modules
|
||
|
||
| Module | L0 | S1 | T1 | C1 | W1 | WK1 | AN1 | CLI1 | PERF | Sprint | Tasks |
|
||
|--------|----|----|----|----|----|----|-----|------|------|--------|-------|
|
||
| **Scanner** | ✅ | ✅ | ✅ | ⬜ | ✅ | ✅ | ✅ | ⬜ | ✅ | 5100.0009.0001 | 25 |
|
||
| **Concelier** | ✅ | ✅ | ⬜ | ✅ | ✅ | ⬜ | ✅ | ⬜ | ⬜ | 5100.0009.0002 | 18 |
|
||
| **Excititor** | ✅ | ✅ | ⬜ | ✅ | ✅ | ✅ | ⬜ | ⬜ | ⬜ | 5100.0009.0003 | 21 |
|
||
| **Policy** | ✅ | ✅ | ⬜ | ⬜ | ✅ | ⬜ | ⬜ | ⬜ | ⬜ | 5100.0009.0004 | 15 |
|
||
|
||
### Security & Compliance Modules
|
||
|
||
| Module | L0 | S1 | T1 | C1 | W1 | WK1 | AN1 | CLI1 | PERF | Sprint | Tasks |
|
||
|--------|----|----|----|----|----|----|-----|------|------|--------|-------|
|
||
| **Authority** | ✅ | ⬜ | ⬜ | ✅ | ✅ | ⬜ | ⬜ | ⬜ | ⬜ | 5100.0009.0005 | 17 |
|
||
| **Signer** | ✅ | ⬜ | ⬜ | ✅ | ✅ | ⬜ | ⬜ | ⬜ | ⬜ | 5100.0009.0006 | 17 |
|
||
| **Attestor** | ✅ | ⬜ | ⬜ | ⬜ | ✅ | ⬜ | ⬜ | ⬜ | ⬜ | 5100.0009.0007 | 14 |
|
||
|
||
### Platform Services
|
||
|
||
| Module | L0 | S1 | T1 | C1 | W1 | WK1 | AN1 | CLI1 | PERF | Sprint | Tasks |
|
||
|--------|----|----|----|----|----|----|-----|------|------|--------|-------|
|
||
| **Scheduler** | ✅ | ✅ | ⬜ | ⬜ | ✅ | ✅ | ⬜ | ⬜ | ⬜ | 5100.0009.0008 | 14 |
|
||
| **Notify** | ✅ | ✅ | ⬜ | ✅ | ✅ | ✅ | ⬜ | ⬜ | ⬜ | 5100.0009.0009 | 18 |
|
||
|
||
### Client Interfaces
|
||
|
||
| Module | L0 | S1 | T1 | C1 | W1 | WK1 | AN1 | CLI1 | PERF | Sprint | Tasks |
|
||
|--------|----|----|----|----|----|----|-----|------|------|--------|-------|
|
||
| **CLI** | ⬜ | ⬜ | ⬜ | ⬜ | ⬜ | ⬜ | ⬜ | ✅ | ⬜ | 5100.0009.0010 | 13 |
|
||
| **UI** | ⬜ | ⬜ | ⬜ | ⬜ | ✅ | ⬜ | ⬜ | ⬜ | ⬜ | 5100.0009.0011 | 13 |
|
||
|
||
### Infrastructure & Platform
|
||
|
||
| Module | L0 | S1 | T1 | C1 | W1 | WK1 | AN1 | CLI1 | PERF | Sprint | Tasks |
|
||
|--------|----|----|----|----|----|----|-----|------|------|--------|-------|
|
||
| **EvidenceLocker** | ✅ | ✅ | ⬜ | ⬜ | ✅ | ⬜ | ⬜ | ⬜ | ⬜ | 5100.0010.0001 | 16 |
|
||
| **Graph/Timeline** | ✅ | ✅ | ⬜ | ⬜ | ✅ | ✅ | ⬜ | ⬜ | ⬜ | 5100.0010.0002 | 15 |
|
||
| **Router/Messaging** | ✅ | ✅ | ✅ | ⬜ | ✅ | ⬜ | ⬜ | ⬜ | ⬜ | 5100.0010.0003 | 14 |
|
||
| **AirGap** | ✅ | ✅ | ⬜ | ⬜ | ✅ | ⬜ | ✅ | ✅ | ⬜ | 5100.0010.0004 | 17 |
|
||
|
||
---
|
||
|
||
## Model Distribution Analysis
|
||
|
||
### Models by Usage Frequency
|
||
|
||
| Model | Modules Using | Percentage | Complexity |
|
||
|-------|---------------|------------|------------|
|
||
| **L0** (Library/Core) | 13/15 modules | 87% | HIGH (property tests, snapshots) |
|
||
| **W1** (WebService) | 13/15 modules | 87% | MEDIUM (contract tests, auth) |
|
||
| **S1** (Storage) | 10/15 modules | 67% | HIGH (migrations, idempotency) |
|
||
| **C1** (Connectors) | 5/15 modules | 33% | MEDIUM (fixtures, resilience) |
|
||
| **WK1** (Workers) | 5/15 modules | 33% | MEDIUM (end-to-end, retries) |
|
||
| **AN1** (Analyzers) | 3/15 modules | 20% | HIGH (Roslyn, diagnostics) |
|
||
| **T1** (Transport) | 2/15 modules | 13% | HIGH (protocol compliance) |
|
||
| **CLI1** (CLI Tools) | 2/15 modules | 13% | LOW (exit codes, snapshots) |
|
||
| **PERF** (Performance) | 1/15 modules | 7% | MEDIUM (benchmarks, regression) |
|
||
|
||
### Complexity Heatmap
|
||
|
||
**High Complexity (>15 tasks per sprint):**
|
||
- Scanner (25 tasks: L0+AN1+S1+T1+W1+WK1+PERF)
|
||
- Excititor (21 tasks: C1+L0+S1+W1+WK1)
|
||
- Concelier (18 tasks: C1+L0+S1+W1+AN1)
|
||
- Notify (18 tasks: L0+C1+S1+W1+WK1)
|
||
- Authority (17 tasks: L0+W1+C1)
|
||
- Signer (17 tasks: L0+W1+C1)
|
||
- AirGap (17 tasks: L0+AN1+S1+W1+CLI1)
|
||
|
||
**Medium Complexity (10-15 tasks):**
|
||
- Policy (15 tasks: L0+S1+W1)
|
||
- EvidenceLocker (16 tasks: L0+S1+W1)
|
||
- Graph/Timeline (15 tasks: L0+S1+W1+WK1)
|
||
- Scheduler (14 tasks: L0+S1+W1+WK1)
|
||
- Attestor (14 tasks: L0+W1)
|
||
- Router/Messaging (14 tasks: L0+T1+W1+S1)
|
||
- CLI (13 tasks: CLI1)
|
||
- UI (13 tasks: W1)
|
||
|
||
---
|
||
|
||
## Quality Gate Coverage
|
||
|
||
### Module-Specific Quality Gates (from TEST_CATALOG.yml)
|
||
|
||
| Module | Quality Gates | Enforced By |
|
||
|--------|---------------|-------------|
|
||
| **Scanner** | determinism, reachability_evidence, proof_spine | Sprint 5100.0009.0001 Tasks 7-10, 23-25 |
|
||
| **Concelier** | fixture_coverage, normalization_determinism, no_lattice_dependency | Sprint 5100.0009.0002 Tasks 1-7, 8-10, 18 |
|
||
| **Excititor** | preserve_prune_source, format_snapshots, no_lattice_dependency | Sprint 5100.0009.0003 Tasks 6-11, 21 |
|
||
| **Policy** | unknown_budget, verdict_snapshot | Sprint 5100.0009.0004 Tasks 2, 4, 14-15 |
|
||
| **Authority** | scope_enforcement, sign_verify | Sprint 5100.0009.0005 Tasks 3-5, 16-17 |
|
||
| **Signer** | canonical_payloads, sign_verify | Sprint 5100.0009.0006 Tasks 1-3, 15-17 |
|
||
| **Attestor** | rekor_receipts, dsse_verify | Sprint 5100.0009.0007 Tasks 6-8, 2 |
|
||
| **Scheduler** | idempotent_jobs, retry_backoff | Sprint 5100.0009.0008 Tasks 4, 3, 12 |
|
||
| **Notify** | connector_snapshots, retry_semantics | Sprint 5100.0009.0009 Tasks 1-6, 16 |
|
||
| **CLI** | exit_codes, stdout_snapshots | Sprint 5100.0009.0010 Tasks 1-4, 5-8 |
|
||
| **UI** | contract_snapshots, e2e_smoke | Sprint 5100.0009.0011 Tasks 1-2, 7-10 |
|
||
|
||
### Cross-Cutting Quality Gates
|
||
|
||
| Gate | Applies To | Enforced By |
|
||
|------|-----------|-------------|
|
||
| **Determinism Contract** | Scanner, Excititor, Signer, CLI, AirGap, Concelier | Sprint 5100.0007.0003 (Determinism Gate) |
|
||
| **Architecture Boundaries** | Concelier, Excititor (must NOT reference Scanner lattice) | Sprint 5100.0007.0007 (Architecture Tests) |
|
||
| **Contract Stability** | All WebServices (13 modules) | Sprint 5100.0007.0006 (WebService Contract) |
|
||
| **Storage Idempotency** | All S1 modules (10 modules) | Sprint 5100.0007.0004 (Storage Harness) |
|
||
| **Connector Resilience** | All C1 modules (5 modules) | Sprint 5100.0007.0005 (Connector Fixtures) |
|
||
|
||
---
|
||
|
||
## CI Lane Coverage
|
||
|
||
### Test Distribution Across CI Lanes
|
||
|
||
| CI Lane | Models | Modules | Sprint Tasks | Est. Runtime |
|
||
|---------|--------|---------|--------------|--------------|
|
||
| **Unit** | L0, AN1, CLI1 | All 15 modules | ~120 tasks | <5 min |
|
||
| **Contract** | W1 | 13 modules | ~50 tasks | <2 min |
|
||
| **Integration** | S1, WK1, T1 | 12 modules | ~100 tasks | 10-15 min |
|
||
| **Security** | C1 (security tests), W1 (auth tests) | 5 connectors + 13 WebServices | ~60 tasks | 5-10 min |
|
||
| **Performance** | PERF | Scanner only | ~3 tasks | 3-5 min |
|
||
| **Live** | C1 (live smoke tests) | Concelier, Excititor, Notify, Authority, Signer | ~5 tasks (opt-in) | 5-10 min (nightly) |
|
||
|
||
### CI Lane Dependencies
|
||
|
||
```
|
||
PR Gate (Must Pass):
|
||
├─ Unit Lane (L0, AN1, CLI1) ← Fast feedback
|
||
├─ Contract Lane (W1) ← API stability
|
||
├─ Architecture Lane (Sprint 5100.0007.0007) ← Boundary enforcement
|
||
└─ Integration Lane (S1, WK1, T1) ← Testcontainers
|
||
|
||
Merge Gate (Must Pass):
|
||
├─ All PR Gate lanes
|
||
├─ Security Lane (C1 security, W1 auth)
|
||
└─ Determinism Lane (Sprint 5100.0007.0003)
|
||
|
||
Nightly (Optional):
|
||
├─ Performance Lane (PERF)
|
||
└─ Live Lane (C1 live smoke)
|
||
|
||
Weekly (Optional):
|
||
└─ Competitor Parity (Sprint 5100.0008.0001)
|
||
```
|
||
|
||
---
|
||
|
||
## Epic-to-Model Coverage
|
||
|
||
### Epic Sprints Support Multiple Models
|
||
|
||
| Epic Sprint | Models Enabled | Consuming Modules | Tasks |
|
||
|-------------|----------------|-------------------|-------|
|
||
| **5100.0007.0002 (TestKit)** | ALL (L0, S1, T1, C1, W1, WK1, AN1, CLI1, PERF) | ALL 15 modules | 13 |
|
||
| **5100.0007.0003 (Determinism)** | L0 (determinism), CLI1 (determinism) | Scanner, Excititor, Signer, CLI, AirGap, Concelier | 12 |
|
||
| **5100.0007.0004 (Storage)** | S1 | 10 modules | 12 |
|
||
| **5100.0007.0005 (Connectors)** | C1 | Concelier, Excititor, Authority, Signer, Notify | 12 |
|
||
| **5100.0007.0006 (WebService)** | W1 | 13 modules | 12 |
|
||
| **5100.0007.0007 (Architecture)** | (Cross-cutting) | Concelier, Excititor | 17 |
|
||
|
||
---
|
||
|
||
## Test Type Distribution
|
||
|
||
### By Test Category (Trait)
|
||
|
||
| Test Category | Model Coverage | Estimated Test Count | CI Lane |
|
||
|---------------|----------------|----------------------|---------|
|
||
| **Unit** | L0, AN1 | ~150 tests across 13 modules | Unit |
|
||
| **Property** | L0 (subset) | ~40 tests (Scanner, Policy, Scheduler, Router) | Unit |
|
||
| **Snapshot** | L0, C1, CLI1 | ~80 tests (all modules with canonical outputs) | Unit/Contract |
|
||
| **Integration** | S1, WK1, T1 | ~120 tests across 12 modules | Integration |
|
||
| **Contract** | W1 | ~50 tests (13 WebServices × avg 4 endpoints) | Contract |
|
||
| **Security** | C1 (security), W1 (auth) | ~60 tests | Security |
|
||
| **Performance** | PERF | ~3 tests (Scanner only) | Performance |
|
||
| **Live** | C1 (live smoke) | ~5 tests (opt-in, nightly) | Live |
|
||
|
||
---
|
||
|
||
## Coverage Gaps & Recommendations
|
||
|
||
### Current Gaps
|
||
|
||
1. **Performance Testing:** Only Scanner has PERF model
|
||
- **Recommendation:** Add PERF to Policy (policy evaluation latency), Concelier (merge performance), Scheduler (scheduling overhead)
|
||
|
||
2. **Transport Testing:** Only Router/Messaging has T1 model
|
||
- **Recommendation:** Scanner has T1 in TEST_CATALOG.yml but should validate Valkey transport for job queues
|
||
|
||
3. **Live Connector Tests:** Only 5 modules have C1 live smoke tests (opt-in)
|
||
- **Recommendation:** Run weekly, not nightly; treat as early warning system for schema drift
|
||
|
||
### Recommended Additions (Future Sprints)
|
||
|
||
| Module | Missing Model | Justification | Priority |
|
||
|--------|---------------|---------------|----------|
|
||
| Policy | PERF | Policy evaluation latency critical for real-time decisioning | HIGH |
|
||
| Concelier | PERF | Merge performance affects ingestion throughput | MEDIUM |
|
||
| Scheduler | PERF | Scheduling overhead affects job execution latency | MEDIUM |
|
||
| Scanner | T1 (validate) | Job queue transport (Valkey) should have compliance tests | HIGH |
|
||
| Authority | S1 | Token storage/revocation should have migration tests | MEDIUM |
|
||
|
||
---
|
||
|
||
## Summary Statistics
|
||
|
||
**Total Test Models:** 9
|
||
**Total Modules Covered:** 15
|
||
**Total Module Test Sprints:** 15 (11 module + 4 infrastructure)
|
||
**Total Epic Sprints:** 6
|
||
**Total Quality Gate Sprints:** 1 (Competitor Parity)
|
||
|
||
**Model Usage:**
|
||
- L0: 13 modules (87%)
|
||
- W1: 13 modules (87%)
|
||
- S1: 10 modules (67%)
|
||
- C1: 5 modules (33%)
|
||
- WK1: 5 modules (33%)
|
||
- AN1: 3 modules (20%)
|
||
- T1: 2 modules (13%)
|
||
- CLI1: 2 modules (13%)
|
||
- PERF: 1 module (7%)
|
||
|
||
**Estimated Total Tests:** ~500 tests across all modules and models
|
||
|
||
---
|
||
|
||
**Prepared by:** Project Management
|
||
**Date:** 2025-12-23
|
||
**Next Review:** 2026-01-06 (Week 1 kickoff)
|
||
**Source:** `docs/testing/TEST_CATALOG.yml`, Sprint files 5100.0009.* and 5100.0010.*
|