stella-ops.org/git.stella-ops.org
1
0
Fork 0
Code Issues Pull Requests Actions 2 Releases Wiki Activity
Files
44248482836423b62d07e1e00d78de38a7e3bcad
git.stella-ops.org/docs/features/unchecked/tests/acceptance-test-packs-with-guardrails.md

2.3 KiB
Raw Blame History

Acceptance Test Packs with Guardrails

Module

__Tests

Status

IMPLEMENTED

Description

Acceptance test packs with guardrail definitions exist under the test fixtures with expected output validation.

Implementation Details

  • Acceptance Test Directory: src/__Tests/acceptance/ -- acceptance test suite containing end-to-end scenarios with expected output validation and guardrail definitions.
  • Test Evidence Service: src/__Tests/__Libraries/StellaOps.Testing.Evidence/TestEvidenceService.cs (implements ITestEvidenceService) -- captures test execution evidence (inputs, outputs, assertions) for acceptance validation.
  • Explainability Assertions: src/__Tests/__Libraries/StellaOps.Testing.Explainability/ExplainabilityAssertions.cs -- assertion library for verifying decision explainability in acceptance tests, ensuring verdicts include human-readable rationale.
  • Explainable Decision Interface: src/__Tests/__Libraries/StellaOps.Testing.Explainability/IExplainableDecision.cs -- contract for decisions that must provide explanations as part of guardrail verification.
  • Policy Regression Test Base: src/__Tests/__Libraries/StellaOps.Testing.Policy/PolicyRegressionTestBase.cs -- base class for policy acceptance tests with guardrail enforcement.
  • Policy Diff Engine: src/__Tests/__Libraries/StellaOps.Testing.Policy/PolicyDiffEngine.cs -- diffs policy evaluation outcomes between test runs to detect regressions.
  • Tests: src/__Tests/__Libraries/StellaOps.Testing.Evidence.Tests/TestEvidenceServiceTests.cs

E2E Test Plan

  • Run the acceptance test suite and verify all test packs pass with expected outputs matching guardrail definitions
  • Verify explainability guardrail: run an acceptance test that produces a verdict and confirm the decision includes a human-readable explanation via ExplainabilityAssertions
  • Verify regression detection: modify a policy rule, re-run acceptance tests, and confirm PolicyDiffEngine detects the outcome change
  • Verify evidence capture: run an acceptance test and confirm TestEvidenceService captures the full input/output evidence for audit review
  • Verify guardrail enforcement: introduce a test that violates a guardrail (e.g., missing explanation) and confirm the test fails with a descriptive error