stella-ops.org/git.stella-ops.org
1
0
Fork 0
Code Issues Pull Requests Actions 2 Releases Wiki Activity

Add self-serve search telemetry gap surfacing

Browse Source
This commit is contained in:
master
2026-03-07 17:15:38 +02:00
parent 5fac47f99f
commit 14d7612cc2
13 changed files with 677 additions and 28 deletions
Download Patch File Download Diff File Expand all files Collapse all files

12
docs/implplan/SPRINT_20260307_005_AdvisoryAI_grounded_search_answer_orchestration.md
View File

@@ -65,7 +65,7 @@ Completion criteria:
- [ ] Ambiguous queries return clarifying prompts instead of a blank answer slot.
### AI-SELF-004 - Self-serve telemetry and gap surfacing
Status: TODO
Status: DONE
Dependency: AI-SELF-002
Owners: Developer (AdvisoryAI), Test Automation
Task description:
@@ -73,9 +73,9 @@ Task description:
- Expose enough structured data to drive a gap-closure backlog.
Completion criteria:
- [ ] Telemetry captures unanswered and reformulated journeys without persisting raw sensitive prompts unnecessarily.
- [ ] Operational docs explain how to review self-serve gaps.
- [ ] Tests cover telemetry emission for fallback paths.
- [x] Telemetry captures unanswered and reformulated journeys without persisting raw sensitive prompts unnecessarily.
- [x] Operational docs explain how to review self-serve gaps.
- [x] Tests cover telemetry emission for fallback paths.
### AI-SELF-005 - Targeted behavioral verification
Status: DONE
@@ -110,6 +110,8 @@ Completion criteria:
| 2026-03-07 | Implemented `contextAnswer` in unified search/backend API mapping, added deterministic `grounded` / `clarify` / `insufficient` rules plus follow-up question generation, and extended telemetry fields for answer-state visibility. | Developer |
| 2026-03-07 | Verified the AdvisoryAI test project after the contract change with `dotnet test "src/AdvisoryAI/__Tests/StellaOps.AdvisoryAI.Tests/StellaOps.AdvisoryAI.Tests.csproj" --no-restore -v normal` (`877/877` passing). | Test Automation |
| 2026-03-07 | Exercised the live rebuilt-corpus lane against `http://127.0.0.1:10451`: `POST /v1/advisory-ai/index/rebuild`, `POST /v1/search/index/rebuild`, then `POST /v1/search/query` for `database connectivity`, which returned `contextAnswer.status = grounded`, 3 citations, and 10 cards over ingested data. | Test Automation |
| 2026-03-07 | Implemented optional self-serve analytics columns (`session_id`, `answer_status`, `answer_code`), metrics (`fallbackAnswerRate`, `clarifyRate`, `insufficientRate`, `reformulationCount`, `rescueActionCount`, `abandonedFallbackCount`), and alerting (`fallback_loop`, `abandoned_fallback`) with privacy-preserving session hashing. | Developer |
| 2026-03-07 | Verified targeted telemetry coverage with xUnit v3 runner commands against `UnifiedSearchSprintIntegrationTests` and added a recovered-session regression test so rescue actions clear abandonment as expected. | Test Automation |
## Decisions & Risks
- Decision: the backend contract must return explicit answer states instead of leaving the UI to infer confidence from cards alone.
@@ -122,6 +124,8 @@ Completion criteria:
- Risk: mocked endpoint tests can overstate confidence if ingestion adapters or corpus rebuild order drift.
- Mitigation: keep rebuild order documented, execute it during verification, and record which routes have live-ingested parity.
- Decision: `stella advisoryai sources prepare` is optional for local verification when checked-in Doctor seed/control files are already sufficient, but it requires `STELLAOPS_BACKEND_URL` whenever live Doctor discovery is expected.
- Decision: self-serve analytics is optional and additive; search behavior must not depend on telemetry event emission.
- Decision: targeted AdvisoryAI verification uses xUnit v3 / Microsoft.Testing.Platform-compatible filters (`-- --filter-class` or the built test executable), not legacy VSTest `--filter`.
## Next Checkpoints
- 2026-03-10: Freeze answer payload shape and fallback taxonomy.