stella-ops.org/git.stella-ops.org
1
0
Fork 0
Code Issues Pull Requests Actions 2 Releases Wiki Activity
Files
cf5b72974f923126944f4dcd47e85864ebb31a3b
git.stella-ops.org/docs/features/unchecked/attestor/ai-authority-classification-engine.md

2.8 KiB
Raw Blame History

AI Authority Classification Engine

Module

Attestor

Status

IMPLEMENTED

Description

Authority classification engine that determines whether AI outputs are evidence-backed (authoritative) or suggestion-only, with configurable thresholds and scoring across multiple artifact types.

Implementation Details

  • Core Classifier: src/Attestor/__Libraries/StellaOps.Attestor.ProofChain/Predicates/AI/AIAuthorityClassifier.cs -- partial class with DetermineAuthority() method that evaluates citationRate, verifiedRate, confidenceScore, and qualityScore against configurable thresholds.
  • Artifact-Specific Scoring: Partial files implement scoring for each artifact type:
    • AIAuthorityClassifier.Explanation.cs / AIAuthorityClassifier.ExplanationScore.cs -- explanation classification and scoring
    • AIAuthorityClassifier.PolicyDraft.cs / AIAuthorityClassifier.PolicyDraftScore.cs -- policy draft classification
    • AIAuthorityClassifier.Remediation.cs / AIAuthorityClassifier.RemediationScore.cs -- remediation plan classification
    • AIAuthorityClassifier.VexDraft.cs / AIAuthorityClassifier.VexDraftScore.cs -- VEX statement draft classification
  • Authority Levels: AIArtifactAuthority.cs -- Suggestion (no evidence backing), EvidenceBacked (citations verified, evidence resolvable), AuthorityThreshold (auto-processing eligible)
  • Thresholds Config: AIAuthorityThresholds.cs -- MinCitationRate (0.8), MinConfidenceScore (0.7), MinVerifiedCitationRate (0.9), AuthorityThresholdScore (0.95), RequireResolvableEvidence (true)
  • Classification Result: AIAuthorityClassificationResult.cs -- captures authority level, reasons, and individual scores
  • Evidence Resolution: Constructor accepts optional Func<string, bool> evidence resolver to verify that cited evidence IDs are resolvable
  • Tests: __Libraries/StellaOps.Attestor.ProofChain.Tests/AI/AIAuthorityClassifierTests.cs

E2E Test Plan

  • Classify an explanation with all metrics above thresholds and verify EvidenceBacked result with three reason entries (citation rate, verified rate, confidence)
  • Classify a policy draft with qualityScore >= 0.95 and verify AuthorityThreshold result regardless of other metrics
  • Classify a remediation plan with citationRate = 0.5 and verify Suggestion result with reason mentioning citation rate below threshold
  • Classify a VEX draft with an evidence resolver that returns false for some evidence IDs and verify the verified rate drops below threshold
  • Override AIAuthorityThresholds with stricter values (e.g., MinCitationRate = 0.95) and verify classification changes accordingly
  • Verify all four artifact-type classifiers (Explanation, PolicyDraft, Remediation, VexDraft) produce correct AIAuthorityClassificationResult with type-specific scoring