31 lines
2.2 KiB
Markdown
31 lines
2.2 KiB
Markdown
# Chaos/Failure Testing Infrastructure
|
|
|
|
## Module
|
|
__Tests
|
|
|
|
## Status
|
|
VERIFIED
|
|
|
|
## Description
|
|
A chaos testing library exists for failure choreography and integration testing scenarios.
|
|
|
|
## Implementation Details
|
|
- **Failure Choreographer**: `src/__Tests/__Libraries/StellaOps.Testing.Chaos/FailureChoreographer.cs` -- orchestrates failure injection sequences across distributed services, coordinating timed failures for integration tests.
|
|
- **Failure Injector Interface**: `src/__Tests/__Libraries/StellaOps.Testing.Chaos/IFailureInjector.cs` -- contract for failure injection strategies (network partition, service crash, resource exhaustion).
|
|
- **Convergence Tracker Interface**: `src/__Tests/__Libraries/StellaOps.Testing.Chaos/IConvergenceTracker.cs` -- tracks system convergence after failure injection, verifying the system recovers to a consistent state within a timeout.
|
|
- **Chaos Models**: `src/__Tests/__Libraries/StellaOps.Testing.Chaos/Models.cs` -- data models for failure scenarios, injection points, and convergence results.
|
|
- **Tests**: `src/__Tests/__Libraries/StellaOps.Testing.Chaos.Tests/FailureChoreographerTests.cs`, `FailureInjectorTests.cs`, `ConvergenceTrackerTests.cs`
|
|
|
|
## E2E Test Plan
|
|
- [ ] Configure a `FailureChoreographer` with a network partition scenario and verify it injects the failure at the specified time and restores connectivity after the duration
|
|
- [ ] Inject a service crash via `IFailureInjector` and verify `IConvergenceTracker` detects the system has not converged within the expected timeout
|
|
- [ ] Inject a failure and verify the system eventually converges to a consistent state after the failure is removed
|
|
- [ ] Run a choreographed sequence of 3 failures (network delay, service restart, resource exhaustion) and verify each failure is applied in order with correct timing
|
|
- [ ] Verify the chaos tests are isolated and do not affect other test suites running in parallel
|
|
|
|
## Verification
|
|
- Verified on 2026-02-13 via `run-001`.
|
|
- Tier 0: Source files confirmed present on disk.
|
|
- Tier 1: `dotnet build` passed (0 errors); 266/266 tests passed across Chaos.Tests, Evidence.Tests, Replay.Tests, FixtureTests.
|
|
- Tier 2d: `docs/qa/feature-checks/runs/tests/chaos-failure-testing-infrastructure/run-001/tier2-integration-check.json`
|