33 lines
2.6 KiB
Markdown
33 lines
2.6 KiB
Markdown
# Acceptance Test Packs with Guardrails
|
|
|
|
## Module
|
|
__Tests
|
|
|
|
## Status
|
|
VERIFIED
|
|
|
|
## Description
|
|
Acceptance test packs with guardrail definitions exist under the test fixtures with expected output validation.
|
|
|
|
## Implementation Details
|
|
- **Acceptance Test Directory**: `src/__Tests/acceptance/` -- acceptance test suite containing end-to-end scenarios with expected output validation and guardrail definitions.
|
|
- **Test Evidence Service**: `src/__Tests/__Libraries/StellaOps.Testing.Evidence/TestEvidenceService.cs` (implements `ITestEvidenceService`) -- captures test execution evidence (inputs, outputs, assertions) for acceptance validation.
|
|
- **Explainability Assertions**: `src/__Tests/__Libraries/StellaOps.Testing.Explainability/ExplainabilityAssertions.cs` -- assertion library for verifying decision explainability in acceptance tests, ensuring verdicts include human-readable rationale.
|
|
- **Explainable Decision Interface**: `src/__Tests/__Libraries/StellaOps.Testing.Explainability/IExplainableDecision.cs` -- contract for decisions that must provide explanations as part of guardrail verification.
|
|
- **Policy Regression Test Base**: `src/__Tests/__Libraries/StellaOps.Testing.Policy/PolicyRegressionTestBase.cs` -- base class for policy acceptance tests with guardrail enforcement.
|
|
- **Policy Diff Engine**: `src/__Tests/__Libraries/StellaOps.Testing.Policy/PolicyDiffEngine.cs` -- diffs policy evaluation outcomes between test runs to detect regressions.
|
|
- **Tests**: `src/__Tests/__Libraries/StellaOps.Testing.Evidence.Tests/TestEvidenceServiceTests.cs`
|
|
|
|
## E2E Test Plan
|
|
- [ ] Run the acceptance test suite and verify all test packs pass with expected outputs matching guardrail definitions
|
|
- [ ] Verify explainability guardrail: run an acceptance test that produces a verdict and confirm the decision includes a human-readable explanation via `ExplainabilityAssertions`
|
|
- [ ] Verify regression detection: modify a policy rule, re-run acceptance tests, and confirm `PolicyDiffEngine` detects the outcome change
|
|
- [ ] Verify evidence capture: run an acceptance test and confirm `TestEvidenceService` captures the full input/output evidence for audit review
|
|
- [ ] Verify guardrail enforcement: introduce a test that violates a guardrail (e.g., missing explanation) and confirm the test fails with a descriptive error
|
|
|
|
## Verification
|
|
- Verified on 2026-02-13 via `run-001`.
|
|
- Tier 0: Source files confirmed present on disk.
|
|
- Tier 1: `dotnet build` passed (0 errors); 266/266 tests passed across Chaos.Tests, Evidence.Tests, Replay.Tests, FixtureTests.
|
|
- Tier 2d: `docs/qa/feature-checks/runs/tests/acceptance-test-packs-with-guardrails/run-001/tier2-integration-check.json`
|