stella-ops.org/git.stella-ops.org
1
0
Fork 0
Code Issues Pull Requests Actions 3 Releases Wiki Activity
Files
aff0ceb2feb9e5b2e3f38bb7296d3f7e255718a2
git.stella-ops.org/docs/implplan/SPRINT_5100_SUMMARY.md
master 53503cb407 Add reference architecture and testing strategy documentation 
- Created a new document for the Stella Ops Reference Architecture outlining the system's topology, trust boundaries, artifact association, and interfaces.
- Developed a comprehensive Testing Strategy document detailing the importance of offline readiness, interoperability, determinism, and operational guardrails.
- Introduced a README for the Testing Strategy, summarizing processing details and key concepts implemented.
- Added guidance for AI agents and developers in the tests directory, including directory structure, test categories, key patterns, and rules for test development.
2025-12-22 07:59:30 +02:00

244 lines
7.8 KiB
Markdown
Raw Blame History

# Sprint Epic 5100 · Comprehensive Testing Strategy
## Overview
Epic 5100 implements the comprehensive testing strategy defined in the Testing Strategy advisory (20-Dec-2025). This epic transforms Stella Ops' testing moats into continuously verified guarantees through deterministic replay, offline compliance, interoperability contracts, and chaos resilience testing.
**IMPLID**: 5100 (Test Infrastructure)
**Total Sprints**: 12
**Total Tasks**: ~75
---
## Epic Structure
### Phase 0: Harness & Corpus Foundation
**Objective**: Standardize test artifacts and expand the golden corpus.
| Sprint | Name | Tasks | Priority |
|--------|------|-------|----------|
| 5100.0001.0001 | [Run Manifest Schema](SPRINT_5100_0001_0001_run_manifest_schema.md) | 7 | HIGH |
| 5100.0001.0002 | [Evidence Index Schema](SPRINT_5100_0001_0002_evidence_index_schema.md) | 7 | HIGH |
| 5100.0001.0003 | [Offline Bundle Manifest](SPRINT_5100_0001_0003_offline_bundle_manifest.md) | 7 | HIGH |
| 5100.0001.0004 | [Golden Corpus Expansion](SPRINT_5100_0001_0004_golden_corpus_expansion.md) | 10 | MEDIUM |
**Key Deliverables**:
- `RunManifest` schema capturing all replay inputs
- `EvidenceIndex` schema linking verdict to evidence chain
- `BundleManifest` for offline operation
- 50+ golden test corpus cases
---
### Phase 1: Determinism & Replay
**Objective**: Ensure byte-identical verdicts across time and machines.
| Sprint | Name | Tasks | Priority |
|--------|------|-------|----------|
| 5100.0002.0001 | [Canonicalization Utilities](SPRINT_5100_0002_0001_canonicalization_utilities.md) | 7 | HIGH |
| 5100.0002.0002 | [Replay Runner Service](SPRINT_5100_0002_0002_replay_runner_service.md) | 7 | HIGH |
| 5100.0002.0003 | [Delta-Verdict Generator](SPRINT_5100_0002_0003_delta_verdict_generator.md) | 7 | MEDIUM |
**Key Deliverables**:
- Canonical JSON serialization (RFC 8785 principles)
- Stable ordering for all collections
- Replay engine with frozen time/PRNG
- Delta-verdict for diff-aware release gates
- Property-based tests with FsCheck
---
### Phase 2: Offline E2E & Interop
**Objective**: Prove air-gap compliance and tool interoperability.
| Sprint | Name | Tasks | Priority |
|--------|------|-------|----------|
| 5100.0003.0001 | [SBOM Interop Round-Trip](SPRINT_5100_0003_0001_sbom_interop_roundtrip.md) | 7 | HIGH |
| 5100.0003.0002 | [No-Egress Enforcement](SPRINT_5100_0003_0002_no_egress_enforcement.md) | 6 | HIGH |
**Key Deliverables**:
- Syft → cosign → Grype round-trip tests
- CycloneDX 1.6 and SPDX 3.0.1 validation
- 95%+ findings parity with consumer tools
- Network-isolated test infrastructure
- `--network none` CI enforcement
---
### Phase 3: Unknowns Budgets CI Gates
**Objective**: Enforce unknowns-budget policy gates in CI/CD.
| Sprint | Name | Tasks | Priority |
|--------|------|-------|----------|
| 5100.0004.0001 | [Unknowns Budget CI Gates](SPRINT_5100_0004_0001_unknowns_budget_ci_gates.md) | 6 | HIGH |
**Key Deliverables**:
- `stella budget check` CLI command
- CI workflow with environment-based budgets
- PR comments with budget status
- UI budget visualization
- Attestation integration
---
### Phase 4: Backpressure & Chaos
**Objective**: Validate router resilience under load.
| Sprint | Name | Tasks | Priority |
|--------|------|-------|----------|
| 5100.0005.0001 | [Router Chaos Suite](SPRINT_5100_0005_0001_router_chaos_suite.md) | 6 | MEDIUM |
**Key Deliverables**:
- k6 load test harness
- 429/503 response verification
- Retry-After header compliance
- Recovery within 30 seconds
- Valkey failure injection tests
---
### Phase 5: Audit Packs & Time-Travel
**Objective**: Enable sealed export/import for auditors.
| Sprint | Name | Tasks | Priority |
|--------|------|-------|----------|
| 5100.0006.0001 | [Audit Pack Export/Import](SPRINT_5100_0006_0001_audit_pack_export_import.md) | 6 | MEDIUM |
**Key Deliverables**:
- Sealed audit pack format
- One-command replay verification
- Signature verification with included trust roots
- CLI commands for auditor workflow
---
## Dependency Graph
```
Phase 0 (Foundation)
├── 5100.0001.0001 (Run Manifest)
│ └── Phase 1 depends
├── 5100.0001.0002 (Evidence Index)
│ └── Phase 2, 5 depend
├── 5100.0001.0003 (Offline Bundle)
│ └── Phase 2 depends
└── 5100.0001.0004 (Golden Corpus)
└── All phases use
Phase 1 (Determinism)
├── 5100.0002.0001 (Canonicalization)
│ └── 5100.0002.0002, 5100.0002.0003 depend
├── 5100.0002.0002 (Replay Runner)
│ └── Phase 5 depends
└── 5100.0002.0003 (Delta-Verdict)
Phase 2 (Offline & Interop)
├── 5100.0003.0001 (SBOM Interop)
└── 5100.0003.0002 (No-Egress)
Phase 3 (Unknowns Gates)
└── 5100.0004.0001 (CI Gates)
└── Depends on 4100.0001.0002
Phase 4 (Chaos)
└── 5100.0005.0001 (Router Chaos)
Phase 5 (Audit Packs)
└── 5100.0006.0001 (Export/Import)
└── Depends on Phase 0, Phase 1
```
---
## CI/CD Integration
### New Workflows
| Workflow | Trigger | Purpose |
|----------|---------|---------|
| `replay-verification.yml` | PR (scanner changes) | Verify deterministic replay |
| `interop-e2e.yml` | PR + Nightly | SBOM interoperability |
| `offline-e2e.yml` | PR + Nightly | Air-gap compliance |
| `unknowns-gate.yml` | PR + Push | Budget enforcement |
| `router-chaos.yml` | Nightly | Resilience testing |
### Release Blocking Gates
A release candidate is blocked if any of these fail:
1. **Replay Verification**: Zero non-deterministic diffs
2. **Interop Suite**: 95%+ findings parity
3. **Offline E2E**: All tests pass with no network
4. **Unknowns Budget**: Within budget for prod environment
5. **Performance**: No breach of p95/memory budgets
---
## Success Criteria
| Criteria | Metric | Gate |
|----------|--------|------|
| Full scan + attest + verify with no network | `offline-e2e` passes | Release |
| Re-running fixed input = identical verdict | 0 byte diff | Release |
| Grype from SBOM matches image scan | 95%+ parity | Release |
| Builds fail when unknowns > budget | Exit code 2 | PR |
| Router under burst emits correct Retry-After | 100% compliance | Nightly |
| Evidence index links complete | Validation passes | Release |
---
## Artifacts Standardized
| Artifact | Schema Location | Purpose |
|----------|-----------------|---------|
| Run Manifest | `StellaOps.Testing.Manifests` | Replay key |
| Evidence Index | `StellaOps.Evidence` | Verdict → evidence chain |
| Offline Bundle | `StellaOps.AirGap.Bundle` | Air-gap operation |
| Delta Verdict | `StellaOps.DeltaVerdict` | Diff-aware gates |
| Audit Pack | `StellaOps.AuditPack` | Compliance verification |
---
## Implementation Order
### Immediate (This Week)
1. **5100.0001.0001** - Run Manifest Schema
2. **5100.0002.0001** - Canonicalization Utilities
3. **5100.0004.0001** - Unknowns Budget CI Gates
### Short Term (Next 2 Sprints)
4. **5100.0001.0002** - Evidence Index Schema
5. **5100.0002.0002** - Replay Runner Service
6. **5100.0003.0001** - SBOM Interop Round-Trip
### Medium Term (Following Sprints)
7. **5100.0001.0003** - Offline Bundle Manifest
8. **5100.0003.0002** - No-Egress Enforcement
9. **5100.0002.0003** - Delta-Verdict Generator
### Later
10. **5100.0001.0004** - Golden Corpus Expansion
11. **5100.0005.0001** - Router Chaos Suite
12. **5100.0006.0001** - Audit Pack Export/Import
---
## Related Documentation
- [Test Suite Overview](../19_TEST_SUITE_OVERVIEW.md)
- [Testing Strategy Advisory](../product-advisories/20-Dec-2025%20-%20Testing%20strategy.md)
- [Offline Operation Guide](../24_OFFLINE_KIT.md)
- [tests/AGENTS.md](../../tests/AGENTS.md)
---
## Execution Log
| Date (UTC) | Update | Owner |
|------------|--------|-------|
| 2025-12-21 | Epic created from Testing Strategy advisory analysis. 12 sprints defined across 6 phases. | Agent |
---
**Epic Status**: PLANNING (0/12 sprints complete)
Reference in New Issue View Git Blame Copy Permalink