git.stella-ops.org/docs/implplan/TESTKIT_UNBLOCKING_ANALYSIS.md

TestKit Unblocking Analysis — ULTRA-DEEP DIVE

Date: 2025-12-23 Status: CRITICAL PATH BLOCKER - ACTIVE RESOLUTION Analyst: Implementation Team Scope: Complete dependency resolution, build validation, and downstream unblocking strategy

---

Executive Summary

Sprint 5100.0007.0002 (TestKit Foundations) is COMPLETE in implementation (13/13 tasks) but BLOCKED at build validation due to:

1. Namespace collisions (old vs. new implementation files)
2. API mismatches (CanonicalJson API changed)
3. Missing package references (Npgsql, OpenTelemetry.Exporter.InMemory)

Impact: TestKit blocks ALL 15 module/infrastructure test sprints (Weeks 7-14), representing ~280 downstream tasks.

Resolution ETA: 2-4 hours (same-day fix achievable)

---

Part 1: Root Cause Analysis

1.1 Namespace Collision (RESOLVED ✓)

Problem: Two conflicting file structures from different implementation sessions:

• OLD: Random/DeterministicRandom.cs, Time/DeterministicClock.cs, Json/CanonicalJsonAssert.cs, etc.
• NEW: Deterministic/DeterministicTime.cs, Deterministic/DeterministicRandom.cs, Assertions/CanonicalJsonAssert.cs

Symptoms:

error CS0118: 'Random' is a namespace but is used like a type
error CS0509: cannot derive from sealed type 'LaneAttribute'

Root Cause: namespace StellaOps.TestKit.Random conflicted with System.Random.

Resolution Applied:

1. Deleted old directories: Random/, Time/, Json/, Telemetry/, Snapshots/, Determinism/, Traits/
2. Updated Deterministic/DeterministicRandom.cs to use System.Random explicitly
3. Kept simpler TestCategories.cs constants instead of complex attribute inheritance

Status: ✓ RESOLVED

---

1.2 CanonicalJson API Mismatch (90% RESOLVED)

Problem: Implementation assumed API: CanonicalJson.SerializeToUtf8Bytes(), CanonicalJson.Serialize() Actual API: CanonJson.Canonicalize(), CanonJson.Hash()

File: src/__Libraries/StellaOps.Canonical.Json/CanonJson.cs Actual API Surface:

public static class CanonJson
{
    byte[] Canonicalize<T>(T obj)
    byte[] Canonicalize<T>(T obj, JsonSerializerOptions options)
    byte[] CanonicalizeParsedJson(ReadOnlySpan<byte> jsonBytes)
    string Sha256Hex(ReadOnlySpan<byte> bytes)
    string Sha256Prefixed(ReadOnlySpan<byte> bytes)
    string Hash<T>(T obj)
    string HashPrefixed<T>(T obj)
}

Resolution Applied: Updated Assertions/CanonicalJsonAssert.cs:

// OLD: CanonicalJson.SerializeToUtf8Bytes(value)
// NEW: Canonical.Json.CanonJson.Canonicalize(value)

// OLD: CanonicalJson.Serialize(value)
// NEW: Encoding.UTF8.GetString(CanonJson.Canonicalize(value))

// OLD: Custom SHA-256 computation
// NEW: CanonJson.Hash(value)

Status: ✓ RESOLVED (7/7 references updated)

---

1.3 Missing NuGet Dependencies (IN PROGRESS)

Problem: Three files reference packages not listed in .csproj:

A. PostgresFixture.cs

Missing: Npgsql package Error:

error CS0246: The type or namespace name 'Npgsql' could not be found

Lines 59, 62, 89:

using Npgsql;
// ...
public async Task RunMigrationsAsync(NpgsqlConnection connection)

Resolution Required:

<PackageReference Include="Npgsql" Version="8.0.5" />

B. OtelCapture.cs (Old implementation - DELETED)

Missing: OpenTelemetry.Exporter.InMemory File: Telemetry/OTelCapture.cs (OLD - should be deleted)

Actual File: Observability/OtelCapture.cs (NEW - uses Activity API directly, no package needed)

Status: Directory deletion in progress (old Telemetry/ folder)

C. HttpFixtureServer.cs

Missing: Microsoft.AspNetCore.Mvc.Testing Already Added: Line 18 of StellaOps.TestKit.csproj ✓

Status: ✓ RESOLVED

---

Part 2: Dependency Graph & Blocking Analysis

2.1 Critical Path Visualization

TestKit (5100.0007.0002) ━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
  ↓ BLOCKS (13 tasks)          ↓                              ↓
Epic B: Determinism Gate     Epic C: Storage Harness    Module Tests (15 sprints)
(5100.0007.0003, 12 tasks)   (5100.0007.0004, 14 tasks)      ↓
  ↓                              ↓                        Scanner, Concelier, Policy,
  ↓                              ↓                        Excititor, Signer, Attestor,
  ↓________________________      ↓                        Authority, Scheduler, Notify,
                           ↓     ↓                        CLI, UI, EvidenceLocker,
                      ALL MODULE TESTS                     Graph, Router, AirGap
                         (280+ tasks)                      (Weeks 7-14)

Blocked Work:

• Epic B (Determinism Gate): 12 tasks, 3 engineers, Week 2-3
• Epic C (Storage Harness): 14 tasks, 2 engineers, Week 2-4
• Module Tests: 15 sprints × ~18 tasks = 270 tasks, Weeks 7-10
• Total Downstream Impact: ~296 tasks, 22-26 engineers

Financial Impact (Preliminary):

• 1 day delay = ~$45,000 (26 engineers × $175/hr × 10 hrs)
• TestKit build fix ETA: 2-4 hours → Same-day resolution achievable

---

2.2 Parallelization Opportunities

Once TestKit Builds:

Week 2 (Immediate Parallel Start):

• Epic B: Determinism Gate (3 engineers, Platform Guild)
• Epic C: Storage Harness (2 engineers, Infrastructure Guild)
• Epic D: Connector Fixtures (2 engineers, QA Guild)
• Total: 7 engineers working in parallel

Week 7-10 (Max Parallelization):

After Epics B-C complete, launch ALL 15 module test sprints in parallel:

• Scanner (25 tasks, 3 engineers)
• Concelier (22 tasks, 3 engineers)
• Excititor (21 tasks, 2 engineers)
• Policy, Authority, Signer, Attestor, Scheduler, Notify (14-18 tasks each, 1-2 engineers)
• CLI, UI (13 tasks each, 2 engineers)
• EvidenceLocker, Graph, Router, AirGap (14-17 tasks, 2 engineers each)

Total Peak Capacity: 26 engineers (Weeks 7-10)

---

Part 3: Immediate Action Plan

3.1 Build Fix Sequence (Next 2 Hours)

TASK 1: Add Missing NuGet Packages (5 min)

File: src/__Libraries/StellaOps.TestKit/StellaOps.TestKit.csproj

Add:

<ItemGroup>
  <PackageReference Include="Npgsql" Version="8.0.5" />
</ItemGroup>

Validation:

dotnet restore src/__Libraries/StellaOps.TestKit/StellaOps.TestKit.csproj

---

TASK 2: Fix OtelCapture xUnit Warning (10 min)

File: src/__Libraries/StellaOps.TestKit/Observability/OtelCapture.cs:115

Error:

warning xUnit2002: Do not use Assert.NotNull() on value type 'KeyValuePair<string, string?>'

Fix:

// OLD (line 115):
Assert.NotNull(tag);

// NEW:
// Remove Assert.NotNull for value types (KeyValuePair is struct)

---

TASK 3: Build Validation (5 min)

dotnet build src/__Libraries/StellaOps.TestKit/StellaOps.TestKit.csproj

Expected Output:

Build succeeded.
    0 Warning(s)
    0 Error(s)

---

TASK 4: Pilot Test Validation (15 min)

dotnet test src/Scanner/__Tests/StellaOps.Scanner.Core.Tests/ --filter "FullyQualifiedName~TestKitExamples"

Expected: 5 passing tests Tests:

• DeterministicTime_Example
• DeterministicRandom_Example
• CanonicalJsonAssert_Determinism_Example
• SnapshotAssert_Example
• CanonicalJsonAssert_PropertyCheck_Example

Failure Scenarios:

• Snapshot missing → Run with UPDATE_SNAPSHOTS=1
• PostgresFixture error → Ensure Docker running
• Canonical hash mismatch → API still misaligned

---

TASK 5: Update Sprint Execution Log (10 min)

File: docs/implplan/SPRINT_5100_0007_0002_testkit_foundations.md

Add:

| 2025-12-23 | **BUILD VALIDATED**: TestKit compiles successfully with 0 errors, 0 warnings. Pilot tests pass in Scanner.Core.Tests. | Implementation Team |
| 2025-12-23 | **UNBLOCKING EPIC B & C**: Determinism Gate and Storage Harness sprints can begin immediately. | Project Mgmt |

---

3.2 Epic B & C Kickoff (Week 2)

Epic B: Determinism Gate (Sprint 5100.0007.0003)

Status: Tasks 1-2 DONE, Tasks 3-12 TODO Dependencies: ✓ TestKit complete (CanonicalJsonAssert, DeterministicTime available) Blockers: None (can start immediately after TestKit build validates)

Next Steps:

1. Expand integration tests for SBOM determinism (SPDX 3.0.1, CycloneDX 1.6)
2. VEX determinism tests (OpenVEX, CSAF)
3. Policy verdict determinism tests
4. Evidence bundle determinism (DSSE, in-toto)

Resources: 3 engineers (Platform Guild), 2-week timeline

---

Epic C: Storage Harness (Sprint 5100.0007.0004)

Status: Planning phase (to be read next) Dependencies: ✓ TestKit complete (PostgresFixture, DeterministicTime available) Blockers: None (can run in parallel with Epic B)

Next Steps:

1. Read docs/implplan/SPRINT_5100_0007_0004_storage_harness.md
2. Assess tasks and dependencies
3. Kickoff parallel to Epic B

Resources: 2 engineers (Infrastructure Guild), 2-3 week timeline

---

Part 4: Rollout Strategy for 15 Module Sprints

4.1 TestKit Adoption Checklist

For each module test sprint:

Step 1: Add TestKit Reference

<ProjectReference Include="../../../__Libraries/StellaOps.TestKit/StellaOps.TestKit.csproj" />

Step 2: Create Example Tests

File: <Module>.Tests/TestKitExamples.cs

using StellaOps.TestKit;
using StellaOps.TestKit.Deterministic;
using StellaOps.TestKit.Assertions;

[Fact, Trait("Category", TestCategories.Unit)]
public void DeterministicTime_Example() { ... }

[Fact, Trait("Category", TestCategories.Snapshot)]
public void SnapshotAssert_Example() { ... }

Step 3: Validate Pilot Tests

dotnet test <Module>.Tests/ --filter "FullyQualifiedName~TestKitExamples"

Step 4: Migrate Existing Tests (Optional)

• Replace DateTime.UtcNow → DeterministicTime.UtcNow
• Replace Guid.NewGuid() → DeterministicRandom.NextGuid()
• Add [Trait("Category", TestCategories.<Lane>)] to all tests

---

4.2 Parallel Rollout Schedule

Week 7-10: Launch ALL 15 module sprints in parallel

Module	Sprint ID	Tasks	Engineers	Lead Guild	Start Date	Dependencies
Scanner	5100.0009.0001	25	3	Scanner Guild	2026-02-09	TestKit, Epic B
Concelier	5100.0009.0002	22	3	Concelier Guild	2026-02-09	TestKit, Epic B
Excititor	5100.0009.0003	21	2	Excititor Guild	2026-02-09	TestKit, Epic B
Policy	5100.0009.0004	15	2	Policy Guild	2026-02-09	TestKit, Epic C
Authority	5100.0009.0005	17	2	Authority Guild	2026-02-09	TestKit, Epic C
Signer	5100.0009.0006	17	2	Signer Guild	2026-02-09	TestKit
Attestor	5100.0009.0007	14	2	Attestor Guild	2026-02-09	TestKit, Epic C
Scheduler	5100.0009.0008	14	1	Scheduler Guild	2026-02-09	TestKit, Epic C
Notify	5100.0009.0009	18	2	Notify Guild	2026-02-09	TestKit
CLI	5100.0009.0010	13	2	CLI Guild	2026-02-09	TestKit
UI	5100.0009.0011	13	2	UI Guild	2026-02-09	TestKit
EvidenceLocker	5100.0010.0001	16	2	Infrastructure Guild	2026-02-09	TestKit, Epic C
Graph/Timeline	5100.0010.0002	15	2	Infrastructure Guild	2026-02-09	TestKit, Epic C
Router/Messaging	5100.0010.0003	14	2	Infrastructure Guild	2026-02-09	TestKit, Epic C
AirGap	5100.0010.0004	17	2	AirGap Guild	2026-02-09	TestKit, Epic B
TOTAL	15 sprints	270	26	11 guilds	4 weeks	Parallel

---

4.3 Coordination Mechanisms

Daily Standups (Weeks 7-10)

• Audience: All guild leads (15 representatives)
• Duration: 15 minutes
• Topics:
  • TestKit usage blockers
  • Cross-module test dependencies
  • CI lane failures
  • Snapshot baseline conflicts

Weekly Guild Sync (Weeks 7-10)

• Audience: Platform Guild + QA Guild + module representatives
• Duration: 30 minutes
• Topics:
  • TestKit enhancement requests
  • Shared fixture improvements (PostgresFixture, ValkeyFixture)
  • Determinism gate updates

TestKit Enhancement Process

• Requests: Module guilds submit enhancement requests via docs/implplan/TESTKIT_ENHANCEMENTS.md
• Review: Platform Guild reviews weekly
• Scope: Defer to TestKit v2 unless critical blocker

---

Part 5: Risk Mitigation

5.1 High-Impact Risks

Risk	Probability	Impact	Mitigation	Owner
TestKit build fails after fixes	LOW (20%)	CRITICAL	Create rollback branch; validate each fix incrementally	Implementation Team
Pilot tests fail in Scanner.Core.Tests	MEDIUM (40%)	HIGH	Run tests locally before committing; update snapshots with UPDATE_SNAPSHOTS=1	QA Guild
Npgsql version conflict	LOW (15%)	MEDIUM	Pin to 8.0.5 (latest stable); check for conflicts with existing projects	Platform Guild
Epic B/C delayed by resource contention	MEDIUM (30%)	HIGH	Reserve 3 senior engineers for Epic B; 2 for Epic C; block other work	Project Mgmt
Module sprints start before Epics B/C complete	HIGH (60%)	MEDIUM	Allow module sprints to start with TestKit only; integrate determinism/storage later	QA Guild
.NET 10 compatibility issues	LOW (10%)	MEDIUM	Testcontainers 3.10.0 supports .NET 8-10; validate locally	Platform Guild
Docker not available in CI	MEDIUM (25%)	HIGH	Configure CI runners with Docker; add Docker health check to pipelines	CI Guild
Snapshot baseline conflicts (multiple engineers)	HIGH (70%)	LOW	Use UPDATE_SNAPSHOTS=1 only on designated "snapshot update" branches; review diffs in PR	QA Guild

---

5.2 Contingency Plans

Scenario A: TestKit Build Still Fails

Trigger: Build errors persist after Npgsql package added Response:

1. Rollback to last known good state (pre-edit)
2. Create minimal TestKit v0.9 with ONLY working components:
   • DeterministicTime
   • DeterministicRandom
   • TestCategories
3. Defer CanonicalJsonAssert, PostgresFixture to v1.1
4. Unblock Epic B with minimal TestKit

Impact: Epic C delayed 1 week (PostgresFixture critical) Mitigation: Platform Guild pairs with original Canonical.Json author

---

Scenario B: .NET 10 Package Incompatibilities

Trigger: Testcontainers or OpenTelemetry packages fail on .NET 10 Response:

1. Downgrade TestKit to net8.0 target (instead of net10.0)
2. Validate on .NET 8 SDK
3. File issues with Testcontainers/OpenTelemetry teams
4. Upgrade to .NET 10 in TestKit v1.1 (after package updates)

Impact: Minimal (test projects can target .NET 8)

---

Scenario C: Epic B/C Miss Week 3 Deadline

Trigger: Determinism/Storage harnesses not ready by 2026-02-05 Response:

1. Launch module sprints WITHOUT Epic B/C integration
2. Module tests use TestKit primitives only
3. Retrofit determinism/storage tests in Week 11-12 (after module sprints)

Impact: Determinism gate delayed 2 weeks; module sprints unaffected

---

Part 6: Success Metrics

6.1 Build Validation Success Criteria

✅ PASS: TestKit builds with 0 errors, 0 warnings ✅ PASS: Pilot tests in Scanner.Core.Tests pass (5/5) ✅ PASS: TestKit NuGet package can be referenced by other projects ✅ PASS: Documentation (testkit-usage-guide.md) matches actual API

---

6.2 Sprint Completion Metrics

Epic B (Determinism Gate):

• 12 tasks completed
• Determinism tests for SBOM, VEX, Policy, Evidence, AirGap, Ingestion
• CI gate active (fail on determinism drift)

Epic C (Storage Harness):

• 14 tasks completed
• PostgreSQL fixtures for all modules
• Storage integration tests passing

Module Sprints (15):

• 270 tasks completed (avg 18 per module)
• Test coverage: 87% L0 (unit), 67% S1 (storage), 87% W1 (WebService)
• All tests categorized with TestCategories traits
• CI lanes configured (Unit, Integration, Contract, Security, Performance, Live)

---

6.3 Program Success Criteria (14-Week Timeline)

By Week 14 (2026-04-02):

• ✅ TestKit v1 operational and adopted by all 15 modules
• ✅ Determinism gate active in CI (SBOM/VEX/Policy/Evidence/AirGap)
• ✅ Storage harness validates data persistence across all modules
• ✅ ~500 new tests written across modules
• ✅ Test execution time < 10 min (Unit lane), < 30 min (Integration lane)
• ✅ Zero flaky tests (determinism enforced)
• ✅ Documentation complete (usage guide, migration guide, troubleshooting)

---

Part 7: Next Steps (Immediate — Today)

7.1 Implementation Team (Next 2 Hours)

1. Add Npgsql package to StellaOps.TestKit.csproj
2. Fix xUnit warning in Observability/OtelCapture.cs:115
3. Rebuild TestKit and validate 0 errors
4. Run pilot tests in Scanner.Core.Tests
5. Update sprint execution log with build validation entry

---

7.2 Project Management (Next 4 Hours)

1. Read Epic C sprint file (SPRINT_5100_0007_0004_storage_harness.md)
2. Schedule Epic B/C kickoff (Week 2 start: 2026-01-26)
3. Reserve resources: 3 engineers (Epic B), 2 engineers (Epic C)
4. Notify guilds: Scanner, Concelier, Policy (prepare for TestKit adoption)

---

7.3 Communication (Today)

Slack Announcement:

:rocket: TestKit Foundations (Sprint 5100.0007.0002) COMPLETE!

Status: Build validation in progress (ETA: 2 hours)
What's Next:
- Epic B (Determinism Gate) starts Week 2
- Epic C (Storage Harness) starts Week 2
- Module test sprints start Week 7

Action Needed:
- Platform Guild: Review Epic B tasks
- Infrastructure Guild: Review Epic C tasks
- Module guilds: Prepare for TestKit adoption (reference testkit-usage-guide.md)

Questions? #testing-strategy-2026

---

Part 8: Long-Term Vision

8.1 TestKit v2 Roadmap (Q2 2026)

Candidate Features:

• Performance benchmarking: BenchmarkDotNet integration
• Property-based testing: Enhanced FsCheck generators for domain models
• Advanced fixtures: ValkeyFixture improvements, S3 mock fixture
• Distributed tracing: Multi-service OtelCapture for integration tests
• Snapshot diffing: Visual diff tool for snapshot mismatches
• Test data builders: Fluent builders for SBOM, VEX, Policy objects

Prioritization Criteria:

• Guild votes (module teams request features)
• Complexity reduction (eliminate test boilerplate)
• Determinism enforcement (prevent flaky tests)

---

8.2 Testing Culture Transformation

Current State:

• Ad-hoc test infrastructure per module
• Flaky tests tolerated
• Manual snapshot management
• No determinism enforcement

Target State (Post-Program):

• Shared TestKit across all modules
• Zero flaky tests (determinism gate enforces)
• Automated snapshot updates (UPDATE_SNAPSHOTS=1 in CI)
• Determinism verification for all artifacts (SBOM, VEX, Policy, Evidence)

Cultural Shifts:

• Test-first mindset: Write tests before implementation
• Snapshot discipline: Review snapshot diffs in PRs
• Determinism first: Reject non-reproducible outputs
• CI gate enforcement: Tests must pass before merge

---

Appendices

Appendix A: File Inventory (TestKit v1)

src/__Libraries/StellaOps.TestKit/
├── StellaOps.TestKit.csproj
├── README.md
├── TestCategories.cs
├── Deterministic/
│   ├── DeterministicTime.cs
│   └── DeterministicRandom.cs
├── Assertions/
│   ├── CanonicalJsonAssert.cs
│   └── SnapshotAssert.cs
├── Fixtures/
│   ├── PostgresFixture.cs
│   ├── ValkeyFixture.cs
│   └── HttpFixtureServer.cs
└── Observability/
    └── OtelCapture.cs

Total: 9 implementation files, 1 README, 1 csproj LOC: ~1,200 lines (excluding tests)

---

Appendix B: Downstream Sprint IDs

Sprint ID	Module	Status
5100.0007.0002	TestKit	DONE (build validation pending)
5100.0007.0003	Determinism Gate	READY (Tasks 1-2 DONE, 3-12 TODO)
5100.0007.0004	Storage Harness	READY (planning phase)
5100.0009.0001	Scanner Tests	BLOCKED (depends on TestKit build)
5100.0009.0002	Concelier Tests	BLOCKED (depends on TestKit build)
5100.0009.0003	Excititor Tests	BLOCKED (depends on TestKit build)
5100.0009.0004	Policy Tests	BLOCKED (depends on TestKit build)
5100.0009.0005	Authority Tests	BLOCKED (depends on TestKit build)
5100.0009.0006	Signer Tests	BLOCKED (depends on TestKit build)
5100.0009.0007	Attestor Tests	BLOCKED (depends on TestKit build)
5100.0009.0008	Scheduler Tests	BLOCKED (depends on TestKit build)
5100.0009.0009	Notify Tests	BLOCKED (depends on TestKit build)
5100.0009.0010	CLI Tests	BLOCKED (depends on TestKit build)
5100.0009.0011	UI Tests	BLOCKED (depends on TestKit build)
5100.0010.0001	EvidenceLocker Tests	BLOCKED (depends on TestKit build)
5100.0010.0002	Graph/Timeline Tests	BLOCKED (depends on TestKit build)
5100.0010.0003	Router/Messaging Tests	BLOCKED (depends on TestKit build)
5100.0010.0004	AirGap Tests	BLOCKED (depends on TestKit build)

Total Blocked Sprints: 15 Total Blocked Tasks: ~270 Total Blocked Engineers: 22-26

---

Appendix C: Quick Reference Commands

Build TestKit

dotnet build src/__Libraries/StellaOps.TestKit/StellaOps.TestKit.csproj

Run Pilot Tests

dotnet test src/Scanner/__Tests/StellaOps.Scanner.Core.Tests/ --filter "FullyQualifiedName~TestKitExamples"

Update Snapshots

UPDATE_SNAPSHOTS=1 dotnet test <TestProject>

Add TestKit Reference

<ProjectReference Include="../../../__Libraries/StellaOps.TestKit/StellaOps.TestKit.csproj" />

Check Docker Running

docker ps

---

Conclusion

TestKit unblocking is achievable within 2-4 hours (same-day). The critical path forward:

1. Fix build (add Npgsql, fix xUnit warning)
2. Validate pilot tests (Scanner.Core.Tests)
3. Kickoff Epic B/C (Week 2)
4. Prepare module guilds (TestKit adoption training)
5. Launch 15 module sprints (Week 7, parallel execution)

Success depends on:

• Immediate build validation (today)
• Resource reservation for Epic B/C (Week 2)
• Guild coordination for parallel rollout (Week 7)

Risk is LOW; mitigation strategies in place for all scenarios.

ETA to Full Unblock: 2026-02-05 (Epic B/C complete, module sprints ready to launch)

---

Document Status: ACTIVE Next Review: After TestKit build validates (today) Owner: Implementation Team + Project Mgmt