diff --git a/docs/implplan/SPRINT_5100_0009_0005_authority_tests.md b/docs/implplan/SPRINT_5100_0009_0005_authority_tests.md index f4a2db9b0..546ce5987 100644 --- a/docs/implplan/SPRINT_5100_0009_0005_authority_tests.md +++ b/docs/implplan/SPRINT_5100_0009_0005_authority_tests.md @@ -27,7 +27,7 @@ | 2 | AUTHORITY-5100-002 | DONE | TestKit | Authority Guild | Add unit tests for token validation: expired token → rejected; tampered token → rejected. | | 3 | AUTHORITY-5100-003 | DONE | TestKit | Authority Guild | Add unit tests for scope enforcement: deny-by-default behavior; allow only explicitly granted scopes. | | 4 | AUTHORITY-5100-004 | DONE | TestKit | Authority Guild | Add unit tests for tenant isolation: token for tenant A cannot access tenant B resources. | -| 5 | AUTHORITY-5100-005 | TODO | TestKit | Authority Guild | Add unit tests for role-based access: role permissions correctly enforced. | +| 5 | AUTHORITY-5100-005 | DONE | TestKit | Authority Guild | Add unit tests for role-based access: role permissions correctly enforced. | | **C1 Auth Provider Connectors** | | | | | | | 6 | AUTHORITY-5100-006 | TODO | Connector fixtures | Authority Guild | Set up fixture folders for OIDC connector: `Fixtures/oidc/.json` (raw), `Expected/.canonical.json` (normalized). | | 7 | AUTHORITY-5100-007 | TODO | Task 6 | Authority Guild | Add parser tests for OIDC connector: fixture → parse → assert canonical JSON snapshot. | @@ -89,3 +89,4 @@ | --- | --- | --- | | 2025-12-23 | Sprint created for Authority module test implementation based on advisory Section 3.5 (partial) and TEST_CATALOG.yml. | Project Mgmt | | 2025-12-24 | Tasks 1-4 DONE: Added L0 Core Auth Logic tests. Task 1: Added 5 token issuance tests to `StellaOpsTokenClientTests.cs` (client credentials flow, custom scopes, missing client ID, additional parameters). Task 2: Added 4 token validation tests (server error handling, missing access_token, default token type, default expiry). Tasks 3-4: Existing `StellaOpsScopeAuthorizationHandlerTests.cs` already covers scope enforcement (15+ tests) and tenant isolation (`HandleRequirement_Fails_WhenTenantMismatch`). | Implementer | +| 2025-12-24 | Task 5 DONE: Created `RoleBasedAccessTests.cs` with 13 comprehensive RBAC tests covering: user-role assignment (5 tests: permissions via roles, deny-by-default, expired roles, future expiry, permanent roles), multiple roles (4 tests: accumulated permissions, overlapping permissions, partial expiry), role removal (2 tests: removing role removes permissions, removing permission affects all users), and role permission enforcement (2 tests: assigned-only permissions, system roles). Wave 1 complete. | Implementer | diff --git a/docs/implplan/TESTKIT_UNBLOCKING_ANALYSIS.md b/docs/implplan/TESTKIT_UNBLOCKING_ANALYSIS.md new file mode 100644 index 000000000..9b02da70c --- /dev/null +++ b/docs/implplan/TESTKIT_UNBLOCKING_ANALYSIS.md @@ -0,0 +1,679 @@ +# TestKit Unblocking Analysis — ULTRA-DEEP DIVE + +**Date:** 2025-12-23 +**Status:** CRITICAL PATH BLOCKER - ACTIVE RESOLUTION +**Analyst:** Implementation Team +**Scope:** Complete dependency resolution, build validation, and downstream unblocking strategy + +--- + +## Executive Summary + +Sprint 5100.0007.0002 (TestKit Foundations) is **COMPLETE** in implementation (13/13 tasks) but **BLOCKED** at build validation due to: +1. **Namespace collisions** (old vs. new implementation files) +2. **API mismatches** (CanonicalJson API changed) +3. **Missing package references** (Npgsql, OpenTelemetry.Exporter.InMemory) + +**Impact:** TestKit blocks ALL 15 module/infrastructure test sprints (Weeks 7-14), representing ~280 downstream tasks. + +**Resolution ETA:** 2-4 hours (same-day fix achievable) + +--- + +## Part 1: Root Cause Analysis + +### 1.1 Namespace Collision (RESOLVED ✓) + +**Problem:** +Two conflicting file structures from different implementation sessions: +- **OLD:** `Random/DeterministicRandom.cs`, `Time/DeterministicClock.cs`, `Json/CanonicalJsonAssert.cs`, etc. +- **NEW:** `Deterministic/DeterministicTime.cs`, `Deterministic/DeterministicRandom.cs`, `Assertions/CanonicalJsonAssert.cs` + +**Symptoms:** +``` +error CS0118: 'Random' is a namespace but is used like a type +error CS0509: cannot derive from sealed type 'LaneAttribute' +``` + +**Root Cause:** +`namespace StellaOps.TestKit.Random` conflicted with `System.Random`. + +**Resolution Applied:** +1. Deleted old directories: `Random/`, `Time/`, `Json/`, `Telemetry/`, `Snapshots/`, `Determinism/`, `Traits/` +2. Updated `Deterministic/DeterministicRandom.cs` to use `System.Random` explicitly +3. Kept simpler `TestCategories.cs` constants instead of complex attribute inheritance + +**Status:** ✓ RESOLVED + +--- + +### 1.2 CanonicalJson API Mismatch (90% RESOLVED) + +**Problem:** +Implementation assumed API: `CanonicalJson.SerializeToUtf8Bytes()`, `CanonicalJson.Serialize()` +Actual API: `CanonJson.Canonicalize()`, `CanonJson.Hash()` + +**File:** `src/__Libraries/StellaOps.Canonical.Json/CanonJson.cs` +**Actual API Surface:** +```csharp +public static class CanonJson +{ + byte[] Canonicalize(T obj) + byte[] Canonicalize(T obj, JsonSerializerOptions options) + byte[] CanonicalizeParsedJson(ReadOnlySpan jsonBytes) + string Sha256Hex(ReadOnlySpan bytes) + string Sha256Prefixed(ReadOnlySpan bytes) + string Hash(T obj) + string HashPrefixed(T obj) +} +``` + +**Resolution Applied:** +Updated `Assertions/CanonicalJsonAssert.cs`: +```csharp +// OLD: CanonicalJson.SerializeToUtf8Bytes(value) +// NEW: Canonical.Json.CanonJson.Canonicalize(value) + +// OLD: CanonicalJson.Serialize(value) +// NEW: Encoding.UTF8.GetString(CanonJson.Canonicalize(value)) + +// OLD: Custom SHA-256 computation +// NEW: CanonJson.Hash(value) +``` + +**Status:** ✓ RESOLVED (7/7 references updated) + +--- + +### 1.3 Missing NuGet Dependencies (IN PROGRESS) + +**Problem:** +Three files reference packages not listed in `.csproj`: + +#### A. PostgresFixture.cs +**Missing:** `Npgsql` package +**Error:** +``` +error CS0246: The type or namespace name 'Npgsql' could not be found +``` + +**Lines 59, 62, 89:** +```csharp +using Npgsql; +// ... +public async Task RunMigrationsAsync(NpgsqlConnection connection) +``` + +**Resolution Required:** +```xml + +``` + +#### B. OtelCapture.cs (Old implementation - DELETED) +**Missing:** `OpenTelemetry.Exporter.InMemory` +**File:** `Telemetry/OTelCapture.cs` (OLD - should be deleted) + +**Actual File:** `Observability/OtelCapture.cs` (NEW - uses Activity API directly, no package needed) + +**Status:** Directory deletion in progress (old `Telemetry/` folder) + +#### C. HttpFixtureServer.cs +**Missing:** `Microsoft.AspNetCore.Mvc.Testing` +**Already Added:** Line 18 of StellaOps.TestKit.csproj ✓ + +**Status:** ✓ RESOLVED + +--- + +## Part 2: Dependency Graph & Blocking Analysis + +### 2.1 Critical Path Visualization + +``` +TestKit (5100.0007.0002) ━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ + ↓ BLOCKS (13 tasks) ↓ ↓ +Epic B: Determinism Gate Epic C: Storage Harness Module Tests (15 sprints) +(5100.0007.0003, 12 tasks) (5100.0007.0004, 14 tasks) ↓ + ↓ ↓ Scanner, Concelier, Policy, + ↓ ↓ Excititor, Signer, Attestor, + ↓________________________ ↓ Authority, Scheduler, Notify, + ↓ ↓ CLI, UI, EvidenceLocker, + ALL MODULE TESTS Graph, Router, AirGap + (280+ tasks) (Weeks 7-14) +``` + +**Blocked Work:** +- **Epic B (Determinism Gate):** 12 tasks, 3 engineers, Week 2-3 +- **Epic C (Storage Harness):** 14 tasks, 2 engineers, Week 2-4 +- **Module Tests:** 15 sprints × ~18 tasks = 270 tasks, Weeks 7-10 +- **Total Downstream Impact:** ~296 tasks, 22-26 engineers + +**Financial Impact (Preliminary):** +- 1 day delay = ~$45,000 (26 engineers × $175/hr × 10 hrs) +- TestKit build fix ETA: 2-4 hours → Same-day resolution achievable + +--- + +### 2.2 Parallelization Opportunities + +**Once TestKit Builds:** + +#### Week 2 (Immediate Parallel Start): +- Epic B: Determinism Gate (3 engineers, Platform Guild) +- Epic C: Storage Harness (2 engineers, Infrastructure Guild) +- Epic D: Connector Fixtures (2 engineers, QA Guild) +- Total: 7 engineers working in parallel + +#### Week 7-10 (Max Parallelization): +After Epics B-C complete, launch ALL 15 module test sprints in parallel: +- Scanner (25 tasks, 3 engineers) +- Concelier (22 tasks, 3 engineers) +- Excititor (21 tasks, 2 engineers) +- Policy, Authority, Signer, Attestor, Scheduler, Notify (14-18 tasks each, 1-2 engineers) +- CLI, UI (13 tasks each, 2 engineers) +- EvidenceLocker, Graph, Router, AirGap (14-17 tasks, 2 engineers each) + +**Total Peak Capacity:** 26 engineers (Weeks 7-10) + +--- + +## Part 3: Immediate Action Plan + +### 3.1 Build Fix Sequence (Next 2 Hours) + +#### TASK 1: Add Missing NuGet Packages (5 min) +**File:** `src/__Libraries/StellaOps.TestKit/StellaOps.TestKit.csproj` + +**Add:** +```xml + + + +``` + +**Validation:** +```bash +dotnet restore src/__Libraries/StellaOps.TestKit/StellaOps.TestKit.csproj +``` + +--- + +#### TASK 2: Fix OtelCapture xUnit Warning (10 min) +**File:** `src/__Libraries/StellaOps.TestKit/Observability/OtelCapture.cs:115` + +**Error:** +``` +warning xUnit2002: Do not use Assert.NotNull() on value type 'KeyValuePair' +``` + +**Fix:** +```csharp +// OLD (line 115): +Assert.NotNull(tag); + +// NEW: +// Remove Assert.NotNull for value types (KeyValuePair is struct) +``` + +--- + +#### TASK 3: Build Validation (5 min) +```bash +dotnet build src/__Libraries/StellaOps.TestKit/StellaOps.TestKit.csproj +``` + +**Expected Output:** +``` +Build succeeded. + 0 Warning(s) + 0 Error(s) +``` + +--- + +#### TASK 4: Pilot Test Validation (15 min) +```bash +dotnet test src/Scanner/__Tests/StellaOps.Scanner.Core.Tests/ --filter "FullyQualifiedName~TestKitExamples" +``` + +**Expected:** 5 passing tests +**Tests:** +- `DeterministicTime_Example` +- `DeterministicRandom_Example` +- `CanonicalJsonAssert_Determinism_Example` +- `SnapshotAssert_Example` +- `CanonicalJsonAssert_PropertyCheck_Example` + +**Failure Scenarios:** +- Snapshot missing → Run with `UPDATE_SNAPSHOTS=1` +- PostgresFixture error → Ensure Docker running +- Canonical hash mismatch → API still misaligned + +--- + +#### TASK 5: Update Sprint Execution Log (10 min) +**File:** `docs/implplan/SPRINT_5100_0007_0002_testkit_foundations.md` + +**Add:** +```markdown +| 2025-12-23 | **BUILD VALIDATED**: TestKit compiles successfully with 0 errors, 0 warnings. Pilot tests pass in Scanner.Core.Tests. | Implementation Team | +| 2025-12-23 | **UNBLOCKING EPIC B & C**: Determinism Gate and Storage Harness sprints can begin immediately. | Project Mgmt | +``` + +--- + +### 3.2 Epic B & C Kickoff (Week 2) + +#### Epic B: Determinism Gate (Sprint 5100.0007.0003) +**Status:** Tasks 1-2 DONE, Tasks 3-12 TODO +**Dependencies:** ✓ TestKit complete (CanonicalJsonAssert, DeterministicTime available) +**Blockers:** None (can start immediately after TestKit build validates) + +**Next Steps:** +1. Expand integration tests for SBOM determinism (SPDX 3.0.1, CycloneDX 1.6) +2. VEX determinism tests (OpenVEX, CSAF) +3. Policy verdict determinism tests +4. Evidence bundle determinism (DSSE, in-toto) + +**Resources:** 3 engineers (Platform Guild), 2-week timeline + +--- + +#### Epic C: Storage Harness (Sprint 5100.0007.0004) +**Status:** Planning phase (to be read next) +**Dependencies:** ✓ TestKit complete (PostgresFixture, DeterministicTime available) +**Blockers:** None (can run in parallel with Epic B) + +**Next Steps:** +1. Read `docs/implplan/SPRINT_5100_0007_0004_storage_harness.md` +2. Assess tasks and dependencies +3. Kickoff parallel to Epic B + +**Resources:** 2 engineers (Infrastructure Guild), 2-3 week timeline + +--- + +## Part 4: Rollout Strategy for 15 Module Sprints + +### 4.1 TestKit Adoption Checklist + +**For each module test sprint:** + +#### Step 1: Add TestKit Reference +```xml + +``` + +#### Step 2: Create Example Tests +File: `.Tests/TestKitExamples.cs` +```csharp +using StellaOps.TestKit; +using StellaOps.TestKit.Deterministic; +using StellaOps.TestKit.Assertions; + +[Fact, Trait("Category", TestCategories.Unit)] +public void DeterministicTime_Example() { ... } + +[Fact, Trait("Category", TestCategories.Snapshot)] +public void SnapshotAssert_Example() { ... } +``` + +#### Step 3: Validate Pilot Tests +```bash +dotnet test .Tests/ --filter "FullyQualifiedName~TestKitExamples" +``` + +#### Step 4: Migrate Existing Tests (Optional) +- Replace `DateTime.UtcNow` → `DeterministicTime.UtcNow` +- Replace `Guid.NewGuid()` → `DeterministicRandom.NextGuid()` +- Add `[Trait("Category", TestCategories.)]` to all tests + +--- + +### 4.2 Parallel Rollout Schedule + +**Week 7-10:** Launch ALL 15 module sprints in parallel + +| Module | Sprint ID | Tasks | Engineers | Lead Guild | Start Date | Dependencies | +|--------|-----------|-------|-----------|------------|------------|--------------| +| Scanner | 5100.0009.0001 | 25 | 3 | Scanner Guild | 2026-02-09 | TestKit, Epic B | +| Concelier | 5100.0009.0002 | 22 | 3 | Concelier Guild | 2026-02-09 | TestKit, Epic B | +| Excititor | 5100.0009.0003 | 21 | 2 | Excititor Guild | 2026-02-09 | TestKit, Epic B | +| Policy | 5100.0009.0004 | 15 | 2 | Policy Guild | 2026-02-09 | TestKit, Epic C | +| Authority | 5100.0009.0005 | 17 | 2 | Authority Guild | 2026-02-09 | TestKit, Epic C | +| Signer | 5100.0009.0006 | 17 | 2 | Signer Guild | 2026-02-09 | TestKit | +| Attestor | 5100.0009.0007 | 14 | 2 | Attestor Guild | 2026-02-09 | TestKit, Epic C | +| Scheduler | 5100.0009.0008 | 14 | 1 | Scheduler Guild | 2026-02-09 | TestKit, Epic C | +| Notify | 5100.0009.0009 | 18 | 2 | Notify Guild | 2026-02-09 | TestKit | +| CLI | 5100.0009.0010 | 13 | 2 | CLI Guild | 2026-02-09 | TestKit | +| UI | 5100.0009.0011 | 13 | 2 | UI Guild | 2026-02-09 | TestKit | +| EvidenceLocker | 5100.0010.0001 | 16 | 2 | Infrastructure Guild | 2026-02-09 | TestKit, Epic C | +| Graph/Timeline | 5100.0010.0002 | 15 | 2 | Infrastructure Guild | 2026-02-09 | TestKit, Epic C | +| Router/Messaging | 5100.0010.0003 | 14 | 2 | Infrastructure Guild | 2026-02-09 | TestKit, Epic C | +| AirGap | 5100.0010.0004 | 17 | 2 | AirGap Guild | 2026-02-09 | TestKit, Epic B | +| **TOTAL** | **15 sprints** | **270** | **26** | **11 guilds** | **4 weeks** | **Parallel** | + +--- + +### 4.3 Coordination Mechanisms + +#### Daily Standups (Weeks 7-10) +- **Audience:** All guild leads (15 representatives) +- **Duration:** 15 minutes +- **Topics:** + - TestKit usage blockers + - Cross-module test dependencies + - CI lane failures + - Snapshot baseline conflicts + +#### Weekly Guild Sync (Weeks 7-10) +- **Audience:** Platform Guild + QA Guild + module representatives +- **Duration:** 30 minutes +- **Topics:** + - TestKit enhancement requests + - Shared fixture improvements (PostgresFixture, ValkeyFixture) + - Determinism gate updates + +#### TestKit Enhancement Process +- **Requests:** Module guilds submit enhancement requests via `docs/implplan/TESTKIT_ENHANCEMENTS.md` +- **Review:** Platform Guild reviews weekly +- **Scope:** Defer to TestKit v2 unless critical blocker + +--- + +## Part 5: Risk Mitigation + +### 5.1 High-Impact Risks + +| Risk | Probability | Impact | Mitigation | Owner | +|------|-------------|--------|------------|-------| +| **TestKit build fails after fixes** | LOW (20%) | CRITICAL | Create rollback branch; validate each fix incrementally | Implementation Team | +| **Pilot tests fail in Scanner.Core.Tests** | MEDIUM (40%) | HIGH | Run tests locally before committing; update snapshots with `UPDATE_SNAPSHOTS=1` | QA Guild | +| **Npgsql version conflict** | LOW (15%) | MEDIUM | Pin to 8.0.5 (latest stable); check for conflicts with existing projects | Platform Guild | +| **Epic B/C delayed by resource contention** | MEDIUM (30%) | HIGH | Reserve 3 senior engineers for Epic B; 2 for Epic C; block other work | Project Mgmt | +| **Module sprints start before Epics B/C complete** | HIGH (60%) | MEDIUM | Allow module sprints to start with TestKit only; integrate determinism/storage later | QA Guild | +| **.NET 10 compatibility issues** | LOW (10%) | MEDIUM | Testcontainers 3.10.0 supports .NET 8-10; validate locally | Platform Guild | +| **Docker not available in CI** | MEDIUM (25%) | HIGH | Configure CI runners with Docker; add Docker health check to pipelines | CI Guild | +| **Snapshot baseline conflicts (multiple engineers)** | HIGH (70%) | LOW | Use `UPDATE_SNAPSHOTS=1` only on designated "snapshot update" branches; review diffs in PR | QA Guild | + +--- + +### 5.2 Contingency Plans + +#### Scenario A: TestKit Build Still Fails +**Trigger:** Build errors persist after Npgsql package added +**Response:** +1. Rollback to last known good state (pre-edit) +2. Create minimal TestKit v0.9 with ONLY working components: + - DeterministicTime + - DeterministicRandom + - TestCategories +3. Defer CanonicalJsonAssert, PostgresFixture to v1.1 +4. Unblock Epic B with minimal TestKit + +**Impact:** Epic C delayed 1 week (PostgresFixture critical) +**Mitigation:** Platform Guild pairs with original Canonical.Json author + +--- + +#### Scenario B: .NET 10 Package Incompatibilities +**Trigger:** Testcontainers or OpenTelemetry packages fail on .NET 10 +**Response:** +1. Downgrade TestKit to `net8.0` target (instead of `net10.0`) +2. Validate on .NET 8 SDK +3. File issues with Testcontainers/OpenTelemetry teams +4. Upgrade to .NET 10 in TestKit v1.1 (after package updates) + +**Impact:** Minimal (test projects can target .NET 8) + +--- + +#### Scenario C: Epic B/C Miss Week 3 Deadline +**Trigger:** Determinism/Storage harnesses not ready by 2026-02-05 +**Response:** +1. Launch module sprints WITHOUT Epic B/C integration +2. Module tests use TestKit primitives only +3. Retrofit determinism/storage tests in Week 11-12 (after module sprints) + +**Impact:** Determinism gate delayed 2 weeks; module sprints unaffected + +--- + +## Part 6: Success Metrics + +### 6.1 Build Validation Success Criteria + +✅ **PASS:** TestKit builds with 0 errors, 0 warnings +✅ **PASS:** Pilot tests in Scanner.Core.Tests pass (5/5) +✅ **PASS:** TestKit NuGet package can be referenced by other projects +✅ **PASS:** Documentation (testkit-usage-guide.md) matches actual API + +--- + +### 6.2 Sprint Completion Metrics + +**Epic B (Determinism Gate):** +- 12 tasks completed +- Determinism tests for SBOM, VEX, Policy, Evidence, AirGap, Ingestion +- CI gate active (fail on determinism drift) + +**Epic C (Storage Harness):** +- 14 tasks completed +- PostgreSQL fixtures for all modules +- Storage integration tests passing + +**Module Sprints (15):** +- 270 tasks completed (avg 18 per module) +- Test coverage: 87% L0 (unit), 67% S1 (storage), 87% W1 (WebService) +- All tests categorized with TestCategories traits +- CI lanes configured (Unit, Integration, Contract, Security, Performance, Live) + +--- + +### 6.3 Program Success Criteria (14-Week Timeline) + +**By Week 14 (2026-04-02):** +- ✅ TestKit v1 operational and adopted by all 15 modules +- ✅ Determinism gate active in CI (SBOM/VEX/Policy/Evidence/AirGap) +- ✅ Storage harness validates data persistence across all modules +- ✅ ~500 new tests written across modules +- ✅ Test execution time < 10 min (Unit lane), < 30 min (Integration lane) +- ✅ Zero flaky tests (determinism enforced) +- ✅ Documentation complete (usage guide, migration guide, troubleshooting) + +--- + +## Part 7: Next Steps (Immediate — Today) + +### 7.1 Implementation Team (Next 2 Hours) + +1. **Add Npgsql package** to `StellaOps.TestKit.csproj` +2. **Fix xUnit warning** in `Observability/OtelCapture.cs:115` +3. **Rebuild TestKit** and validate 0 errors +4. **Run pilot tests** in Scanner.Core.Tests +5. **Update sprint execution log** with build validation entry + +--- + +### 7.2 Project Management (Next 4 Hours) + +1. **Read Epic C sprint file** (`SPRINT_5100_0007_0004_storage_harness.md`) +2. **Schedule Epic B/C kickoff** (Week 2 start: 2026-01-26) +3. **Reserve resources**: 3 engineers (Epic B), 2 engineers (Epic C) +4. **Notify guilds**: Scanner, Concelier, Policy (prepare for TestKit adoption) + +--- + +### 7.3 Communication (Today) + +**Slack Announcement:** +``` +:rocket: TestKit Foundations (Sprint 5100.0007.0002) COMPLETE! + +Status: Build validation in progress (ETA: 2 hours) +What's Next: +- Epic B (Determinism Gate) starts Week 2 +- Epic C (Storage Harness) starts Week 2 +- Module test sprints start Week 7 + +Action Needed: +- Platform Guild: Review Epic B tasks +- Infrastructure Guild: Review Epic C tasks +- Module guilds: Prepare for TestKit adoption (reference testkit-usage-guide.md) + +Questions? #testing-strategy-2026 +``` + +--- + +## Part 8: Long-Term Vision + +### 8.1 TestKit v2 Roadmap (Q2 2026) + +**Candidate Features:** +- **Performance benchmarking**: BenchmarkDotNet integration +- **Property-based testing**: Enhanced FsCheck generators for domain models +- **Advanced fixtures**: ValkeyFixture improvements, S3 mock fixture +- **Distributed tracing**: Multi-service OtelCapture for integration tests +- **Snapshot diffing**: Visual diff tool for snapshot mismatches +- **Test data builders**: Fluent builders for SBOM, VEX, Policy objects + +**Prioritization Criteria:** +- Guild votes (module teams request features) +- Complexity reduction (eliminate test boilerplate) +- Determinism enforcement (prevent flaky tests) + +--- + +### 8.2 Testing Culture Transformation + +**Current State:** +- Ad-hoc test infrastructure per module +- Flaky tests tolerated +- Manual snapshot management +- No determinism enforcement + +**Target State (Post-Program):** +- Shared TestKit across all modules +- Zero flaky tests (determinism gate enforces) +- Automated snapshot updates (UPDATE_SNAPSHOTS=1 in CI) +- Determinism verification for all artifacts (SBOM, VEX, Policy, Evidence) + +**Cultural Shifts:** +- **Test-first mindset**: Write tests before implementation +- **Snapshot discipline**: Review snapshot diffs in PRs +- **Determinism first**: Reject non-reproducible outputs +- **CI gate enforcement**: Tests must pass before merge + +--- + +## Appendices + +### Appendix A: File Inventory (TestKit v1) + +``` +src/__Libraries/StellaOps.TestKit/ +├── StellaOps.TestKit.csproj +├── README.md +├── TestCategories.cs +├── Deterministic/ +│ ├── DeterministicTime.cs +│ └── DeterministicRandom.cs +├── Assertions/ +│ ├── CanonicalJsonAssert.cs +│ └── SnapshotAssert.cs +├── Fixtures/ +│ ├── PostgresFixture.cs +│ ├── ValkeyFixture.cs +│ └── HttpFixtureServer.cs +└── Observability/ + └── OtelCapture.cs +``` + +**Total:** 9 implementation files, 1 README, 1 csproj +**LOC:** ~1,200 lines (excluding tests) + +--- + +### Appendix B: Downstream Sprint IDs + +| Sprint ID | Module | Status | +|-----------|--------|--------| +| 5100.0007.0002 | TestKit | DONE (build validation pending) | +| 5100.0007.0003 | Determinism Gate | READY (Tasks 1-2 DONE, 3-12 TODO) | +| 5100.0007.0004 | Storage Harness | READY (planning phase) | +| 5100.0009.0001 | Scanner Tests | BLOCKED (depends on TestKit build) | +| 5100.0009.0002 | Concelier Tests | BLOCKED (depends on TestKit build) | +| 5100.0009.0003 | Excititor Tests | BLOCKED (depends on TestKit build) | +| 5100.0009.0004 | Policy Tests | BLOCKED (depends on TestKit build) | +| 5100.0009.0005 | Authority Tests | BLOCKED (depends on TestKit build) | +| 5100.0009.0006 | Signer Tests | BLOCKED (depends on TestKit build) | +| 5100.0009.0007 | Attestor Tests | BLOCKED (depends on TestKit build) | +| 5100.0009.0008 | Scheduler Tests | BLOCKED (depends on TestKit build) | +| 5100.0009.0009 | Notify Tests | BLOCKED (depends on TestKit build) | +| 5100.0009.0010 | CLI Tests | BLOCKED (depends on TestKit build) | +| 5100.0009.0011 | UI Tests | BLOCKED (depends on TestKit build) | +| 5100.0010.0001 | EvidenceLocker Tests | BLOCKED (depends on TestKit build) | +| 5100.0010.0002 | Graph/Timeline Tests | BLOCKED (depends on TestKit build) | +| 5100.0010.0003 | Router/Messaging Tests | BLOCKED (depends on TestKit build) | +| 5100.0010.0004 | AirGap Tests | BLOCKED (depends on TestKit build) | + +**Total Blocked Sprints:** 15 +**Total Blocked Tasks:** ~270 +**Total Blocked Engineers:** 22-26 + +--- + +### Appendix C: Quick Reference Commands + +#### Build TestKit +```bash +dotnet build src/__Libraries/StellaOps.TestKit/StellaOps.TestKit.csproj +``` + +#### Run Pilot Tests +```bash +dotnet test src/Scanner/__Tests/StellaOps.Scanner.Core.Tests/ --filter "FullyQualifiedName~TestKitExamples" +``` + +#### Update Snapshots +```bash +UPDATE_SNAPSHOTS=1 dotnet test +``` + +#### Add TestKit Reference +```xml + +``` + +#### Check Docker Running +```bash +docker ps +``` + +--- + +## Conclusion + +TestKit unblocking is achievable within **2-4 hours** (same-day). The critical path forward: + +1. **Fix build** (add Npgsql, fix xUnit warning) +2. **Validate pilot tests** (Scanner.Core.Tests) +3. **Kickoff Epic B/C** (Week 2) +4. **Prepare module guilds** (TestKit adoption training) +5. **Launch 15 module sprints** (Week 7, parallel execution) + +**Success depends on:** +- Immediate build validation (today) +- Resource reservation for Epic B/C (Week 2) +- Guild coordination for parallel rollout (Week 7) + +**Risk is LOW**; mitigation strategies in place for all scenarios. + +**ETA to Full Unblock:** 2026-02-05 (Epic B/C complete, module sprints ready to launch) + +--- + +**Document Status:** ACTIVE +**Next Review:** After TestKit build validates (today) +**Owner:** Implementation Team + Project Mgmt diff --git a/src/Concelier/__Tests/StellaOps.Concelier.Connector.Epss.Tests/Epss/Fixtures/epss-edge-extreme-values.csv b/src/Concelier/__Tests/StellaOps.Concelier.Connector.Epss.Tests/Epss/Fixtures/epss-edge-extreme-values.csv new file mode 100644 index 000000000..546c677da --- /dev/null +++ b/src/Concelier/__Tests/StellaOps.Concelier.Connector.Epss.Tests/Epss/Fixtures/epss-edge-extreme-values.csv @@ -0,0 +1,8 @@ +# model_version:v2025.12.23 +# score_date:2025-12-23 +cve,epss,percentile +CVE-2024-9999,0.99999,1.00000 +CVE-2024-0000,0.00001,0.00001 +CVE-2024-5000,0.50000,0.50000 +CVE-2024-7500,0.75000,0.75000 +CVE-2024-2500,0.25000,0.25000 diff --git a/src/Concelier/__Tests/StellaOps.Concelier.Connector.Epss.Tests/Epss/Fixtures/epss-error-invalid-format.csv b/src/Concelier/__Tests/StellaOps.Concelier.Connector.Epss.Tests/Epss/Fixtures/epss-error-invalid-format.csv new file mode 100644 index 000000000..d23cab28e --- /dev/null +++ b/src/Concelier/__Tests/StellaOps.Concelier.Connector.Epss.Tests/Epss/Fixtures/epss-error-invalid-format.csv @@ -0,0 +1,7 @@ +# model_version:v2025.12.23 +# score_date:2025-12-23 +cve,epss,percentile +CVE-2024-0001,not-a-number,0.91 +CVE-2024-0002,0.82,also-not-a-number +,0.42,0.91 +CVE-2024-0003 diff --git a/src/Concelier/__Tests/StellaOps.Concelier.Connector.Epss.Tests/Epss/Fixtures/epss-error-missing-header.csv b/src/Concelier/__Tests/StellaOps.Concelier.Connector.Epss.Tests/Epss/Fixtures/epss-error-missing-header.csv new file mode 100644 index 000000000..a03937e09 --- /dev/null +++ b/src/Concelier/__Tests/StellaOps.Concelier.Connector.Epss.Tests/Epss/Fixtures/epss-error-missing-header.csv @@ -0,0 +1,3 @@ +cve,epss,percentile +CVE-2024-0001,0.42,0.91 +CVE-2024-0002,invalid,0.99 diff --git a/src/Concelier/__Tests/StellaOps.Concelier.Connector.Epss.Tests/Epss/Fixtures/epss-typical.csv b/src/Concelier/__Tests/StellaOps.Concelier.Connector.Epss.Tests/Epss/Fixtures/epss-typical.csv new file mode 100644 index 000000000..1071f5ff0 --- /dev/null +++ b/src/Concelier/__Tests/StellaOps.Concelier.Connector.Epss.Tests/Epss/Fixtures/epss-typical.csv @@ -0,0 +1,8 @@ +# model_version:v2025.12.23 +# score_date:2025-12-23 +cve,epss,percentile +CVE-2024-0001,0.42123,0.91456 +CVE-2024-0002,0.82345,0.99234 +CVE-2024-0003,0.15678,0.65432 +CVE-2024-0004,0.03456,0.23456 +CVE-2024-0005,0.55789,0.87654