qa(attestor): verify ai explanation attestation types feature
This commit is contained in:
@@ -0,0 +1,107 @@
|
||||
# Sprint 20260211_017 - Attestor Unchecked Feature Verification
|
||||
|
||||
## Topic & Scope
|
||||
- Verify a new non-web complex Attestor batch starting from proof-graph noise gating behavior.
|
||||
- Execute Tier 0, Tier 1, and Tier 2 per FLOW and resolve any failures before advancing.
|
||||
- Move terminal feature docs from `docs/features/unchecked/attestor/` to `checked` or `unimplemented`.
|
||||
- Working directory: `src/Attestor`.
|
||||
- Expected evidence: fresh run artifacts under `docs/qa/feature-checks/runs/attestor/**`, state updates, and feature-file moves.
|
||||
|
||||
## Dependencies & Concurrency
|
||||
- Depends on Attestor proof chain libraries and tests under `src/Attestor/**`.
|
||||
- Cross-directory updates allowed in `docs/qa/feature-checks/**`, `docs/features/**`, and this sprint file for auditability.
|
||||
- Web module work remains out of scope.
|
||||
|
||||
## Documentation Prerequisites
|
||||
- `docs/qa/feature-checks/FLOW.md`
|
||||
- `src/Attestor/AGENTS.md`
|
||||
- `docs/modules/attestor/README.md`
|
||||
- `docs/modules/attestor/architecture.md`
|
||||
|
||||
## Delivery Tracker
|
||||
|
||||
### QA-ATTESTOR-VERIFY-001 - Verify `adaptive-noise-gating-for-vulnerability-graphs`
|
||||
Status: DONE
|
||||
Dependency: none
|
||||
Owners: QA / Test Automation, Documentation author
|
||||
Task description:
|
||||
- Verify adaptive noise-gating claims against Attestor proof graph, delta verdict, and change-trace implementations.
|
||||
- Complete full Tier 0/1/2 flow and failure loop before any next queued Attestor feature.
|
||||
|
||||
Completion criteria:
|
||||
- [x] Tier 0/1/2 verification completed or terminal not_implemented decision recorded.
|
||||
|
||||
### QA-ATTESTOR-VERIFY-002 - Verify `ai-assisted-explanation-and-classification`
|
||||
Status: DONE
|
||||
Dependency: QA-ATTESTOR-VERIFY-001
|
||||
Owners: QA / Test Automation, Documentation author
|
||||
Task description:
|
||||
- Verify AI-assisted explanation/classification contracts and behavioral coverage.
|
||||
|
||||
Completion criteria:
|
||||
- [x] Tier 0/1/2 verification completed or terminal not_implemented decision recorded.
|
||||
|
||||
### QA-ATTESTOR-VERIFY-003 - Verify `ai-authority-classification-engine`
|
||||
Status: DONE
|
||||
Dependency: QA-ATTESTOR-VERIFY-002
|
||||
Owners: QA / Test Automation, Documentation author
|
||||
Task description:
|
||||
- Verify authority classification engine behavior and supporting tests.
|
||||
|
||||
Completion criteria:
|
||||
- [x] Tier 0/1/2 verification completed or terminal not_implemented decision recorded.
|
||||
|
||||
### QA-ATTESTOR-VERIFY-004 - Verify `ai-explanation-attestation-types`
|
||||
Status: DONE
|
||||
Dependency: QA-ATTESTOR-VERIFY-003
|
||||
Owners: QA / Test Automation, Documentation author
|
||||
Task description:
|
||||
- Verify explanation attestation type coverage and schema/test alignment.
|
||||
|
||||
Completion criteria:
|
||||
- [x] Tier 0/1/2 verification completed or terminal not_implemented decision recorded.
|
||||
|
||||
### QA-ATTESTOR-VERIFY-005 - Resolve discovered gaps before advancing
|
||||
Status: DONE
|
||||
Dependency: QA-ATTESTOR-VERIFY-001
|
||||
Owners: QA / Test Automation, Documentation author
|
||||
Task description:
|
||||
- Complete triage/confirmation/fix/retest loops for any Attestor failures before advancing to another module batch.
|
||||
|
||||
Completion criteria:
|
||||
- [x] Every failed Attestor feature reaches terminal `done` or `blocked` before next module selection.
|
||||
- [x] Decisions and risks include root cause and mitigation links.
|
||||
|
||||
## Execution Log
|
||||
| Date (UTC) | Update | Owner |
|
||||
| --- | --- | --- |
|
||||
| 2026-02-11 | Sprint created; started `adaptive-noise-gating-for-vulnerability-graphs` verification. | QA |
|
||||
| 2026-02-11 | `adaptive-noise-gating-for-vulnerability-graphs` Tier 0 passed; Tier 1 blocked by upstream proof-chain suite instability (`35 failed / 754 total`) and MTP filter limitations. Marked feature `BLOCKED` with run-001 evidence. | QA |
|
||||
| 2026-02-11 | Re-ran feature with fresh run-003 evidence: full suite remains red on unrelated baseline tests, while feature-scoped xUnit classes passed (`39/39`); claim-parity review found missing provenance-set merge, hysteresis damping, and explicit New/Resolved/ConfidenceUp/ConfidenceDown/PolicyImpact classification logic. | QA |
|
||||
| 2026-02-11 | `adaptive-noise-gating-for-vulnerability-graphs` reclassified to terminal `not_implemented` and moved to `docs/features/unimplemented/attestor/adaptive-noise-gating-for-vulnerability-graphs.md`. | QA |
|
||||
| 2026-02-11 | QA-ATTESTOR-VERIFY-002 moved to DOING after QA-ATTESTOR-VERIFY-001 reached terminal state. | QA |
|
||||
| 2026-02-11 | QA-ATTESTOR-VERIFY-002 returned to TODO due FLOW cross-module problem-first lock (`advisoryai/ai-action-policy-gate` entered `checking`). | QA |
|
||||
| 2026-02-11 | `ai-assisted-explanation-and-classification` verified (run-001 Tier 0/1/2 pass) with feature-scoped AI classifier and AIArtifactVerificationStep behavioral tests (`7/7`), then moved to `docs/features/checked/attestor/`. | QA |
|
||||
| 2026-02-11 | QA-ATTESTOR-VERIFY-003 moved to DOING after QA-ATTESTOR-VERIFY-002 reached terminal `done`. | QA |
|
||||
| 2026-02-11 | Implemented missing adaptive-noise-gating behavior in ProofChain (provenance merge, confidence strength hierarchy, hysteresis suppression, delta category inference) and added focused tests. | QA |
|
||||
| 2026-02-11 | Re-verified `adaptive-noise-gating-for-vulnerability-graphs` with fresh run-002 Tier 0/1/2 evidence (`11/11` targeted tests pass), moved feature from `unimplemented` to `checked`, and updated module state to `done`. | QA |
|
||||
| 2026-02-11 | `ai-authority-classification-engine` verified (run-001 Tier 0/1/2 pass); fixed failing test fixture in `AIAuthorityClassifierTests` and revalidated scoped authority classifier behavior (`9/9`) before moving feature to `docs/features/checked/attestor/`. | QA |
|
||||
| 2026-02-11 | `ai-explanation-attestation-types` verified (run-001 Tier 0/1/2 pass) with targeted AI explanation predicate/verification behavior tests (`7/7`) and moved to `docs/features/checked/attestor/`. | QA |
|
||||
|
||||
## Decisions & Risks
|
||||
- Decision: Started next non-web complex batch with Attestor after Replay reached terminal `done` across tracked features.
|
||||
- Risk: Concurrent agents may complete or mutate Attestor state while this batch is running.
|
||||
- Mitigation: Re-check state before every transition; if a task is already completed by another agent, accept it and continue to the next queued item.
|
||||
- Risk: Attestor proof-chain test suite currently fails globally, so Tier 1 cannot be satisfied for feature-scoped verification using the current runner.
|
||||
- Mitigation: Recorded `BLOCKED` with evidence under `docs/qa/feature-checks/runs/attestor/adaptive-noise-gating-for-vulnerability-graphs/run-001/`; unblock requires upstream suite stabilization or runner/filter support restoration.
|
||||
- Decision: Replaced the temporary `BLOCKED` classification with terminal `not_implemented` after run-003 claim-parity verification proved key promised behaviors are absent in current implementation.
|
||||
- Decision: Used xUnit v3 in-proc runner class filters for feature-scoped verification signal while preserving full-suite failure evidence in run-003.
|
||||
- Risk: Feature docs can overstate implementation maturity relative to model-only predicate/data structures.
|
||||
- Mitigation: move overstated dossiers to `docs/features/unimplemented/**` with explicit missing-behavior evidence and keep queue progression deterministic.
|
||||
- Decision: Superseded prior `not_implemented` classification for `adaptive-noise-gating-for-vulnerability-graphs` after implementing and validating missing behaviors in run-002; terminal state is now `done`.
|
||||
- Risk: Full ProofChain suite remains red on unrelated baseline tests; targeted xUnit filtering is required for deterministic feature verification until the upstream suite is stabilized.
|
||||
- Decision: Treated `ai-authority-classification-engine` run-001 regression as `test_gap` (fixture bug in test input/setup), fixed the test, and completed retest in the same run before terminalizing.
|
||||
|
||||
## Next Checkpoints
|
||||
- `adaptive-noise-gating-for-vulnerability-graphs` Tier 0/1/2 checkpoint: 2026-02-11.
|
||||
- Attestor batch complete; proceed with FLOW problem-first module selection for the next queue item.
|
||||
Reference in New Issue
Block a user