25 lines
1.9 KiB
Markdown
25 lines
1.9 KiB
Markdown
# Chaos/Failure Testing Infrastructure
|
|
|
|
## Module
|
|
__Tests
|
|
|
|
## Status
|
|
IMPLEMENTED
|
|
|
|
## Description
|
|
A chaos testing library exists for failure choreography and integration testing scenarios.
|
|
|
|
## Implementation Details
|
|
- **Failure Choreographer**: `src/__Tests/__Libraries/StellaOps.Testing.Chaos/FailureChoreographer.cs` -- orchestrates failure injection sequences across distributed services, coordinating timed failures for integration tests.
|
|
- **Failure Injector Interface**: `src/__Tests/__Libraries/StellaOps.Testing.Chaos/IFailureInjector.cs` -- contract for failure injection strategies (network partition, service crash, resource exhaustion).
|
|
- **Convergence Tracker Interface**: `src/__Tests/__Libraries/StellaOps.Testing.Chaos/IConvergenceTracker.cs` -- tracks system convergence after failure injection, verifying the system recovers to a consistent state within a timeout.
|
|
- **Chaos Models**: `src/__Tests/__Libraries/StellaOps.Testing.Chaos/Models.cs` -- data models for failure scenarios, injection points, and convergence results.
|
|
- **Tests**: `src/__Tests/__Libraries/StellaOps.Testing.Chaos.Tests/FailureChoreographerTests.cs`, `FailureInjectorTests.cs`, `ConvergenceTrackerTests.cs`
|
|
|
|
## E2E Test Plan
|
|
- [ ] Configure a `FailureChoreographer` with a network partition scenario and verify it injects the failure at the specified time and restores connectivity after the duration
|
|
- [ ] Inject a service crash via `IFailureInjector` and verify `IConvergenceTracker` detects the system has not converged within the expected timeout
|
|
- [ ] Inject a failure and verify the system eventually converges to a consistent state after the failure is removed
|
|
- [ ] Run a choreographed sequence of 3 failures (network delay, service restart, resource exhaustion) and verify each failure is applied in order with correct timing
|
|
- [ ] Verify the chaos tests are isolated and do not affect other test suites running in parallel
|