Add reference architecture and testing strategy documentation
- Created a new document for the Stella Ops Reference Architecture outlining the system's topology, trust boundaries, artifact association, and interfaces. - Developed a comprehensive Testing Strategy document detailing the importance of offline readiness, interoperability, determinism, and operational guardrails. - Introduced a README for the Testing Strategy, summarizing processing details and key concepts implemented. - Added guidance for AI agents and developers in the tests directory, including directory structure, test categories, key patterns, and rules for test development.
This commit is contained in:
243
docs/implplan/SPRINT_5100_SUMMARY.md
Normal file
243
docs/implplan/SPRINT_5100_SUMMARY.md
Normal file
@@ -0,0 +1,243 @@
|
||||
# Sprint Epic 5100 · Comprehensive Testing Strategy
|
||||
|
||||
## Overview
|
||||
|
||||
Epic 5100 implements the comprehensive testing strategy defined in the Testing Strategy advisory (20-Dec-2025). This epic transforms Stella Ops' testing moats into continuously verified guarantees through deterministic replay, offline compliance, interoperability contracts, and chaos resilience testing.
|
||||
|
||||
**IMPLID**: 5100 (Test Infrastructure)
|
||||
**Total Sprints**: 12
|
||||
**Total Tasks**: ~75
|
||||
|
||||
---
|
||||
|
||||
## Epic Structure
|
||||
|
||||
### Phase 0: Harness & Corpus Foundation
|
||||
**Objective**: Standardize test artifacts and expand the golden corpus.
|
||||
|
||||
| Sprint | Name | Tasks | Priority |
|
||||
|--------|------|-------|----------|
|
||||
| 5100.0001.0001 | [Run Manifest Schema](SPRINT_5100_0001_0001_run_manifest_schema.md) | 7 | HIGH |
|
||||
| 5100.0001.0002 | [Evidence Index Schema](SPRINT_5100_0001_0002_evidence_index_schema.md) | 7 | HIGH |
|
||||
| 5100.0001.0003 | [Offline Bundle Manifest](SPRINT_5100_0001_0003_offline_bundle_manifest.md) | 7 | HIGH |
|
||||
| 5100.0001.0004 | [Golden Corpus Expansion](SPRINT_5100_0001_0004_golden_corpus_expansion.md) | 10 | MEDIUM |
|
||||
|
||||
**Key Deliverables**:
|
||||
- `RunManifest` schema capturing all replay inputs
|
||||
- `EvidenceIndex` schema linking verdict to evidence chain
|
||||
- `BundleManifest` for offline operation
|
||||
- 50+ golden test corpus cases
|
||||
|
||||
---
|
||||
|
||||
### Phase 1: Determinism & Replay
|
||||
**Objective**: Ensure byte-identical verdicts across time and machines.
|
||||
|
||||
| Sprint | Name | Tasks | Priority |
|
||||
|--------|------|-------|----------|
|
||||
| 5100.0002.0001 | [Canonicalization Utilities](SPRINT_5100_0002_0001_canonicalization_utilities.md) | 7 | HIGH |
|
||||
| 5100.0002.0002 | [Replay Runner Service](SPRINT_5100_0002_0002_replay_runner_service.md) | 7 | HIGH |
|
||||
| 5100.0002.0003 | [Delta-Verdict Generator](SPRINT_5100_0002_0003_delta_verdict_generator.md) | 7 | MEDIUM |
|
||||
|
||||
**Key Deliverables**:
|
||||
- Canonical JSON serialization (RFC 8785 principles)
|
||||
- Stable ordering for all collections
|
||||
- Replay engine with frozen time/PRNG
|
||||
- Delta-verdict for diff-aware release gates
|
||||
- Property-based tests with FsCheck
|
||||
|
||||
---
|
||||
|
||||
### Phase 2: Offline E2E & Interop
|
||||
**Objective**: Prove air-gap compliance and tool interoperability.
|
||||
|
||||
| Sprint | Name | Tasks | Priority |
|
||||
|--------|------|-------|----------|
|
||||
| 5100.0003.0001 | [SBOM Interop Round-Trip](SPRINT_5100_0003_0001_sbom_interop_roundtrip.md) | 7 | HIGH |
|
||||
| 5100.0003.0002 | [No-Egress Enforcement](SPRINT_5100_0003_0002_no_egress_enforcement.md) | 6 | HIGH |
|
||||
|
||||
**Key Deliverables**:
|
||||
- Syft → cosign → Grype round-trip tests
|
||||
- CycloneDX 1.6 and SPDX 3.0.1 validation
|
||||
- 95%+ findings parity with consumer tools
|
||||
- Network-isolated test infrastructure
|
||||
- `--network none` CI enforcement
|
||||
|
||||
---
|
||||
|
||||
### Phase 3: Unknowns Budgets CI Gates
|
||||
**Objective**: Enforce unknowns-budget policy gates in CI/CD.
|
||||
|
||||
| Sprint | Name | Tasks | Priority |
|
||||
|--------|------|-------|----------|
|
||||
| 5100.0004.0001 | [Unknowns Budget CI Gates](SPRINT_5100_0004_0001_unknowns_budget_ci_gates.md) | 6 | HIGH |
|
||||
|
||||
**Key Deliverables**:
|
||||
- `stella budget check` CLI command
|
||||
- CI workflow with environment-based budgets
|
||||
- PR comments with budget status
|
||||
- UI budget visualization
|
||||
- Attestation integration
|
||||
|
||||
---
|
||||
|
||||
### Phase 4: Backpressure & Chaos
|
||||
**Objective**: Validate router resilience under load.
|
||||
|
||||
| Sprint | Name | Tasks | Priority |
|
||||
|--------|------|-------|----------|
|
||||
| 5100.0005.0001 | [Router Chaos Suite](SPRINT_5100_0005_0001_router_chaos_suite.md) | 6 | MEDIUM |
|
||||
|
||||
**Key Deliverables**:
|
||||
- k6 load test harness
|
||||
- 429/503 response verification
|
||||
- Retry-After header compliance
|
||||
- Recovery within 30 seconds
|
||||
- Valkey failure injection tests
|
||||
|
||||
---
|
||||
|
||||
### Phase 5: Audit Packs & Time-Travel
|
||||
**Objective**: Enable sealed export/import for auditors.
|
||||
|
||||
| Sprint | Name | Tasks | Priority |
|
||||
|--------|------|-------|----------|
|
||||
| 5100.0006.0001 | [Audit Pack Export/Import](SPRINT_5100_0006_0001_audit_pack_export_import.md) | 6 | MEDIUM |
|
||||
|
||||
**Key Deliverables**:
|
||||
- Sealed audit pack format
|
||||
- One-command replay verification
|
||||
- Signature verification with included trust roots
|
||||
- CLI commands for auditor workflow
|
||||
|
||||
---
|
||||
|
||||
## Dependency Graph
|
||||
|
||||
```
|
||||
Phase 0 (Foundation)
|
||||
├── 5100.0001.0001 (Run Manifest)
|
||||
│ └── Phase 1 depends
|
||||
├── 5100.0001.0002 (Evidence Index)
|
||||
│ └── Phase 2, 5 depend
|
||||
├── 5100.0001.0003 (Offline Bundle)
|
||||
│ └── Phase 2 depends
|
||||
└── 5100.0001.0004 (Golden Corpus)
|
||||
└── All phases use
|
||||
|
||||
Phase 1 (Determinism)
|
||||
├── 5100.0002.0001 (Canonicalization)
|
||||
│ └── 5100.0002.0002, 5100.0002.0003 depend
|
||||
├── 5100.0002.0002 (Replay Runner)
|
||||
│ └── Phase 5 depends
|
||||
└── 5100.0002.0003 (Delta-Verdict)
|
||||
|
||||
Phase 2 (Offline & Interop)
|
||||
├── 5100.0003.0001 (SBOM Interop)
|
||||
└── 5100.0003.0002 (No-Egress)
|
||||
|
||||
Phase 3 (Unknowns Gates)
|
||||
└── 5100.0004.0001 (CI Gates)
|
||||
└── Depends on 4100.0001.0002
|
||||
|
||||
Phase 4 (Chaos)
|
||||
└── 5100.0005.0001 (Router Chaos)
|
||||
|
||||
Phase 5 (Audit Packs)
|
||||
└── 5100.0006.0001 (Export/Import)
|
||||
└── Depends on Phase 0, Phase 1
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## CI/CD Integration
|
||||
|
||||
### New Workflows
|
||||
|
||||
| Workflow | Trigger | Purpose |
|
||||
|----------|---------|---------|
|
||||
| `replay-verification.yml` | PR (scanner changes) | Verify deterministic replay |
|
||||
| `interop-e2e.yml` | PR + Nightly | SBOM interoperability |
|
||||
| `offline-e2e.yml` | PR + Nightly | Air-gap compliance |
|
||||
| `unknowns-gate.yml` | PR + Push | Budget enforcement |
|
||||
| `router-chaos.yml` | Nightly | Resilience testing |
|
||||
|
||||
### Release Blocking Gates
|
||||
|
||||
A release candidate is blocked if any of these fail:
|
||||
|
||||
1. **Replay Verification**: Zero non-deterministic diffs
|
||||
2. **Interop Suite**: 95%+ findings parity
|
||||
3. **Offline E2E**: All tests pass with no network
|
||||
4. **Unknowns Budget**: Within budget for prod environment
|
||||
5. **Performance**: No breach of p95/memory budgets
|
||||
|
||||
---
|
||||
|
||||
## Success Criteria
|
||||
|
||||
| Criteria | Metric | Gate |
|
||||
|----------|--------|------|
|
||||
| Full scan + attest + verify with no network | `offline-e2e` passes | Release |
|
||||
| Re-running fixed input = identical verdict | 0 byte diff | Release |
|
||||
| Grype from SBOM matches image scan | 95%+ parity | Release |
|
||||
| Builds fail when unknowns > budget | Exit code 2 | PR |
|
||||
| Router under burst emits correct Retry-After | 100% compliance | Nightly |
|
||||
| Evidence index links complete | Validation passes | Release |
|
||||
|
||||
---
|
||||
|
||||
## Artifacts Standardized
|
||||
|
||||
| Artifact | Schema Location | Purpose |
|
||||
|----------|-----------------|---------|
|
||||
| Run Manifest | `StellaOps.Testing.Manifests` | Replay key |
|
||||
| Evidence Index | `StellaOps.Evidence` | Verdict → evidence chain |
|
||||
| Offline Bundle | `StellaOps.AirGap.Bundle` | Air-gap operation |
|
||||
| Delta Verdict | `StellaOps.DeltaVerdict` | Diff-aware gates |
|
||||
| Audit Pack | `StellaOps.AuditPack` | Compliance verification |
|
||||
|
||||
---
|
||||
|
||||
## Implementation Order
|
||||
|
||||
### Immediate (This Week)
|
||||
1. **5100.0001.0001** - Run Manifest Schema
|
||||
2. **5100.0002.0001** - Canonicalization Utilities
|
||||
3. **5100.0004.0001** - Unknowns Budget CI Gates
|
||||
|
||||
### Short Term (Next 2 Sprints)
|
||||
4. **5100.0001.0002** - Evidence Index Schema
|
||||
5. **5100.0002.0002** - Replay Runner Service
|
||||
6. **5100.0003.0001** - SBOM Interop Round-Trip
|
||||
|
||||
### Medium Term (Following Sprints)
|
||||
7. **5100.0001.0003** - Offline Bundle Manifest
|
||||
8. **5100.0003.0002** - No-Egress Enforcement
|
||||
9. **5100.0002.0003** - Delta-Verdict Generator
|
||||
|
||||
### Later
|
||||
10. **5100.0001.0004** - Golden Corpus Expansion
|
||||
11. **5100.0005.0001** - Router Chaos Suite
|
||||
12. **5100.0006.0001** - Audit Pack Export/Import
|
||||
|
||||
---
|
||||
|
||||
## Related Documentation
|
||||
|
||||
- [Test Suite Overview](../19_TEST_SUITE_OVERVIEW.md)
|
||||
- [Testing Strategy Advisory](../product-advisories/20-Dec-2025%20-%20Testing%20strategy.md)
|
||||
- [Offline Operation Guide](../24_OFFLINE_KIT.md)
|
||||
- [tests/AGENTS.md](../../tests/AGENTS.md)
|
||||
|
||||
---
|
||||
|
||||
## Execution Log
|
||||
|
||||
| Date (UTC) | Update | Owner |
|
||||
|------------|--------|-------|
|
||||
| 2025-12-21 | Epic created from Testing Strategy advisory analysis. 12 sprints defined across 6 phases. | Agent |
|
||||
|
||||
---
|
||||
|
||||
**Epic Status**: PLANNING (0/12 sprints complete)
|
||||
Reference in New Issue
Block a user