Files
git.stella-ops.org/docs/features/unchecked/tests/chaos-failure-testing-infrastructure.md

1.9 KiB

Chaos/Failure Testing Infrastructure

Module

__Tests

Status

IMPLEMENTED

Description

A chaos testing library exists for failure choreography and integration testing scenarios.

Implementation Details

  • Failure Choreographer: src/__Tests/__Libraries/StellaOps.Testing.Chaos/FailureChoreographer.cs -- orchestrates failure injection sequences across distributed services, coordinating timed failures for integration tests.
  • Failure Injector Interface: src/__Tests/__Libraries/StellaOps.Testing.Chaos/IFailureInjector.cs -- contract for failure injection strategies (network partition, service crash, resource exhaustion).
  • Convergence Tracker Interface: src/__Tests/__Libraries/StellaOps.Testing.Chaos/IConvergenceTracker.cs -- tracks system convergence after failure injection, verifying the system recovers to a consistent state within a timeout.
  • Chaos Models: src/__Tests/__Libraries/StellaOps.Testing.Chaos/Models.cs -- data models for failure scenarios, injection points, and convergence results.
  • Tests: src/__Tests/__Libraries/StellaOps.Testing.Chaos.Tests/FailureChoreographerTests.cs, FailureInjectorTests.cs, ConvergenceTrackerTests.cs

E2E Test Plan

  • Configure a FailureChoreographer with a network partition scenario and verify it injects the failure at the specified time and restores connectivity after the duration
  • Inject a service crash via IFailureInjector and verify IConvergenceTracker detects the system has not converged within the expected timeout
  • Inject a failure and verify the system eventually converges to a consistent state after the failure is removed
  • Run a choreographed sequence of 3 failures (network delay, service restart, resource exhaustion) and verify each failure is applied in order with correct timing
  • Verify the chaos tests are isolated and do not affect other test suites running in parallel