git.stella-ops.org/docs/features/checked/policy/explainability-testing-framework.md

Explainability Testing Framework

Module

Policy

Status

IMPLEMENTED

Description

Explainability testing framework with assertion helpers and verdict rationale rendering, ensuring decisions can be traced back to evidence and assumptions.

Implementation Details

• VerdictRationaleRenderer: src/Policy/__Libraries/StellaOps.Policy.Explainability/VerdictRationaleRenderer.cs (sealed class implements IVerdictRationaleRenderer)
  • Render(VerdictRationaleInput) produces structured 4-line rationale with content-addressed RationaleId
  • RenderPlainText(rationale) produces plain text output (4 lines)
  • RenderMarkdown(rationale) produces Markdown with headers (Evidence, Policy Clause, Attestations, Decision)
  • RenderJson(rationale) produces canonical JSON (RFC 8785) via CanonJson.Serialize
  • Content-addressed ID: rat:sha256:{hash} computed from SHA256 of canonical JSON (with empty RationaleId for self-referential hashing)
• VerdictRationale model: src/Policy/__Libraries/StellaOps.Policy.Explainability/VerdictRationale.cs
  • 4-line template structure:
    • Line 1 (RationaleEvidence): CVE, Component (PURL, name, version, ecosystem), Reachability (vulnerable function, entry point, path summary)
    • Line 2 (RationalePolicyClause): ClauseId, RuleDescription, Conditions
    • Line 3 (RationaleAttestations): PathWitness, VexStatements, Provenance (each as AttestationReference with Id, Type, Digest, Summary)
    • Line 4 (RationaleDecision): Verdict, Score, Recommendation, Mitigation (Action, Details)
  • RationaleInputDigests for reproducibility: VerdictDigest, PolicyDigest, EvidenceDigest
  • SchemaVersion: "1.0"
• VerdictRationaleInput: src/Policy/__Libraries/StellaOps.Policy.Explainability/IVerdictRationaleRenderer.cs
  • Full input record with VerdictRef, Cve, Component, Reachability, PolicyClauseId, PolicyRuleDescription, PolicyConditions, attestation references, Verdict, Score, Recommendation, Mitigation, GeneratedAt, digest fields
• Property-based tests: src/Policy/__Tests/StellaOps.Policy.Determinization.Tests/PropertyTests/DeterminismPropertyTests.cs -- tests determinism of rationale rendering
• Explainability integration tests: verify that Render followed by RenderPlainText, RenderMarkdown, and RenderJson all produce deterministic output for identical inputs

E2E Test Plan

• Render rationale with CVE, component, reachability; verify Evidence.FormattedText contains CVE ID, component name, vulnerable function, entry point
• Render rationale with policy clause; verify PolicyClause.FormattedText contains ClauseId and conditions
• Render rationale with path witness and VEX statements; verify Attestations.FormattedText includes both references
• Render rationale without attestations; verify FormattedText is "No attestations available."
• Render rationale with score and mitigation; verify Decision.FormattedText includes score value and mitigation action
• Render same input twice; verify RationaleId is identical (content-addressed determinism)
• Render different inputs; verify RationaleIds differ
• RenderMarkdown produces valid Markdown with ## headers for each section
• RenderJson produces valid JSON parseable by standard JSON parser
• Verify RationaleId format matches rat:sha256:{64 hex chars}