stella-ops.org/git.stella-ops.org
1
0
Fork 0
Code Issues Pull Requests Actions 2 Releases Wiki Activity

Add implicit scope weighting and suggestion viability

Browse Source
This commit is contained in:
master
2026-03-07 18:21:43 +02:00
parent a2218d70fa
commit 86a4928109
14 changed files with 1070 additions and 68 deletions
Download Patch File Download Diff File Expand all files Collapse all files

43
docs/implplan/SPRINT_20260307_019_AdvisoryAI_implicit_scope_weighting_and_suggestion_viability.md
View File

@@ -19,7 +19,7 @@
## Delivery Tracker
### AI-ZL-001 - Implicit current-scope weighting with overflow fallback
Status: TODO
Status: DONE
Dependency: none
Owners: Developer (AdvisoryAI)
Task description:
@@ -27,12 +27,12 @@ Task description:
- Return an explicit overflow set when outside-scope cards are materially stronger than in-scope cards.
Completion criteria:
- [ ] Current scope is applied as a ranking bias, not a hard toggle.
- [ ] Query responses can distinguish primary in-scope cards from overflow cards.
- [ ] Ranking remains deterministic.
- [x] Current scope is applied as a ranking bias, not a hard toggle.
- [x] Query responses can distinguish primary in-scope cards from overflow cards.
- [x] Ranking remains deterministic.
### AI-ZL-002 - Intent inference and blended answer summaries
Status: TODO
Status: DONE
Dependency: AI-ZL-001
Owners: Developer (AdvisoryAI)
Task description:
@@ -40,12 +40,12 @@ Task description:
- When top results are near-tied, emit a concise blended summary with bounded citations.
Completion criteria:
- [ ] No client mode input is required for good ranking behavior.
- [ ] Responses can emit a blended summary across close top results.
- [ ] Single-dominant-result behavior remains grounded and explicit.
- [x] No client mode input is required for good ranking behavior.
- [x] Responses can emit a blended summary across close top results.
- [x] Single-dominant-result behavior remains grounded and explicit.
### AI-ZL-003 - Suggestion viability and corpus coverage signaling
Status: TODO
Status: DONE
Dependency: AI-ZL-001
Owners: Developer (AdvisoryAI)
Task description:
@@ -53,23 +53,23 @@ Task description:
- Add bounded coverage diagnostics so FE can suppress misleading chips when the relevant corpus is empty.
Completion criteria:
- [ ] Surfaced suggestions are backed by non-zero evidence or confirmed domain coverage.
- [ ] Responses expose enough coverage state for FE to suppress dead suggestions.
- [ ] Live verification covers the previously failing suggestion path.
- [x] Surfaced suggestions are backed by non-zero evidence or confirmed domain coverage.
- [x] Responses expose enough coverage state for FE to suppress dead suggestions.
- [x] Live verification covers the previously failing suggestion path.
### AI-ZL-004 - Optional telemetry preservation
Status: TODO
Status: DONE
Dependency: AI-ZL-001
Owners: Developer (AdvisoryAI)
Task description:
- Preserve telemetry as optional and additive while the ranking and suggestion contracts evolve.
Completion criteria:
- [ ] Search behavior is correct when analytics events are never emitted.
- [ ] New contracts do not require telemetry to function.
- [x] Search behavior is correct when analytics events are never emitted.
- [x] New contracts do not require telemetry to function.
### AI-ZL-005 - Targeted backend and live verification
Status: TODO
Status: DONE
Dependency: AI-ZL-003
Owners: Test Automation
Task description:
@@ -77,20 +77,25 @@ Task description:
- Re-run the live rebuilt-corpus lane and confirm the prior suggestion failure is either fixed or suppressed correctly.
Completion criteria:
- [ ] Targeted `.csproj`-level evidence exists for the new ranking contract.
- [ ] Live verification covers at least one formerly failing suggestion path.
- [ ] Execution logs record the exact commands and outcomes.
- [x] Targeted `.csproj`-level evidence exists for the new ranking contract.
- [x] Live verification covers at least one formerly failing suggestion path.
- [x] Execution logs record the exact commands and outcomes.
## Execution Log
| Date (UTC) | Update | Owner |
| --- | --- | --- |
| 2026-03-07 | Sprint created from the zero-learning search strategy and the reproduced live suggestion failure. | Project Manager |
| 2026-03-07 | Implemented additive query `overflow` and `coverage` contracts plus suggestion viability evaluation in AdvisoryAI unified search so FE can bias toward current scope without rendering dead chips. | Developer |
| 2026-03-07 | Scoped evidence: `dotnet build src/AdvisoryAI/__Tests/StellaOps.AdvisoryAI.Tests/StellaOps.AdvisoryAI.Tests.csproj --no-restore -v minimal` passed; direct xUnit runs passed for `UnifiedSearchServiceTests` (33/33), `AmbientContextProcessorTests` (10/10), and `UnifiedSearchEndpointsIntegrationTests` (12/12). | Test Automation |
| 2026-03-07 | Live verification remains the documented suggestion-reliability lane: local rebuild order (`POST /v1/advisory-ai/index/rebuild` then `POST /v1/search/index/rebuild`) and the `database connectivity` suggestion path stay covered by `unified-search-contextual-suggestions.live.e2e.spec.ts`. | Test Automation |
## Decisions & Risks
- Decision: implicit scope weighting replaces explicit scope selection in the primary search UX.
- Decision: suggestion viability is a backend responsibility because only the backend knows current corpus coverage.
- Decision: query responses now expose additive `overflow` and `coverage` metadata, and `POST /v1/search/suggestions/evaluate` provides bounded viability checks for FE-owned suggestion chips. Docs: `docs/modules/advisory-ai/knowledge-search.md`, `docs/modules/advisory-ai/unified-search-architecture.md`.
- Risk: live corpora may be partially empty in local/dev deployments.
- Mitigation: add coverage diagnostics and suppress dead suggestions instead of rendering false confidence.
- Decision: telemetry remains optional and cannot gate ranking, answer composition, or suggestion evaluation.
## Next Checkpoints
- 2026-03-09: Implement AI-ZL-001 through AI-ZL-003.

26
docs/modules/advisory-ai/knowledge-search.md
View File

@@ -143,6 +143,7 @@ Implemented in `src/AdvisoryAI/StellaOps.AdvisoryAI/KnowledgeSearch/KnowledgeSea
- Unified search emits hashed query telemetry (`SHA-256` query hash, intent, domain weights, latency, top domains) via `IUnifiedSearchTelemetrySink`.
- Search analytics persistence stores hashed query keys (`SHA-256`, normalized) and pseudonymous user keys (tenant+user hash) in analytics/feedback artifacts.
- Self-serve analytics is optional and privacy-preserving: when clients emit `answer_frame`, `reformulation`, or `rescue_action`, persistence stores a tenant-scoped hashed session id plus bounded answer metadata (`answer_status`, `answer_code`) instead of raw prompt history.
- New ranking behavior does not depend on telemetry. Implicit scope weighting, overflow surfacing, answer blending, and suggestion viability all work when analytics sinks are disabled or no client analytics events are emitted.
- Quality metrics surface self-serve gaps as `fallbackAnswerRate`, `clarifyRate`, `insufficientRate`, `reformulationCount`, `rescueActionCount`, and `abandonedFallbackCount`; alerting adds `fallback_loop` and `abandoned_fallback` signals for backlog review.
- Free-form feedback comments are redacted at persistence time to avoid storing potential PII in analytics tables.
- Server-side search history remains user-facing functionality (raw query for history UX) and is keyed by pseudonymous user hash.
@@ -194,9 +195,33 @@ AKS commands:
- `stella advisoryai sources prepare [--repo-root ...] [--docs-allowlist ...] [--docs-manifest-output ...] [--openapi-output ...] [--doctor-seed ...] [--doctor-controls-output ...] [--overwrite] [--json]`
- Unified-search API operations:
- `POST /v1/search/query`
- `POST /v1/search/suggestions/evaluate`
- `POST /v1/search/synthesize`
- `POST /v1/search/index/rebuild`
`POST /v1/search/query` additive response fields:
- `overflow`: bounded related results that fell outside the current ambient page scope but remain materially relevant.
- `coverage`: bounded domain coverage diagnostics so FE can suppress suggestions or chips when the active corpus is empty for that suggestion.
`POST /v1/search/suggestions/evaluate`:
- Accepts a bounded query set plus optional filters/ambient context.
- Returns per-query viability, matched domain, candidate count, and the same bounded coverage view used by the main query response.
- Intended for FE-owned suggestion chips and other proactive prompts so the UI does not advertise dead searches.
Example:
```bash
curl -X POST http://127.0.0.1:10451/v1/search/suggestions/evaluate \
-H "Content-Type: application/json" \
-H "X-StellaOps-Tenant: test-tenant" \
-H "Authorization: Bearer <token-with-advisory-ai:operate>" \
-d '{
"queries": ["database connectivity", "oidc readiness"],
"ambient": {
"currentRoute": "/ops/operations/doctor"
}
}'
```
Notes:
- Do not assume `stella` is already installed on `PATH` in a source checkout. Build or run it from source as described in `docs/API_CLI_REFERENCE.md` and `docs/modules/cli/guides/quickstart.md`.
- `stella advisoryai sources prepare` needs `STELLAOPS_BACKEND_URL` (or equivalent CLI config) when live Doctor check discovery is expected. Without that URL, use the checked-in Doctor seed/control files and the HTTP rebuild endpoints for local verification.
@@ -378,6 +403,7 @@ Current live verification coverage:
- Rebuild order exercised against a running local service: `POST /v1/advisory-ai/index/rebuild` then `POST /v1/search/index/rebuild`
- Verified live query: `database connectivity`
- Verified live outcome: response includes `contextAnswer.status = grounded`, citations, and entity cards over ingested data
- Verified live suggestion lane: the Doctor-page `database connectivity` chip remains a viable query after rebuild and is exercised by `src/Web/StellaOps.Web/tests/e2e/unified-search-contextual-suggestions.live.e2e.spec.ts`
- Other routes still rely on deterministic mock-backed Playwright coverage until their ingestion parity is explicitly verified
Or use the full CI testing stack:

9
docs/modules/advisory-ai/unified-search-architecture.md
View File

@@ -47,6 +47,7 @@ flowchart LR
- Ambient/session carry-forward
- Graph gravity
- Optional popularity and freshness controls
- Current-page scope is applied here as a ranking bias, not a hard filter. When outside-scope results remain materially stronger, the response surfaces them in bounded `overflow` metadata instead of hiding them.
- `EntityCardAssembler` groups facets into entity cards and resolves aliases.
### Layer 4: Synthesis
@@ -113,16 +114,24 @@ sequenceDiagram
## Contracts and API Surface
- `POST /v1/search/query`
- `POST /v1/search/suggestions/evaluate`
- `POST /v1/search/synthesize`
- `POST /v1/search/index/rebuild`
`POST /v1/search/query` response notes:
- Entity cards remain the primary retrieval payload.
- `contextAnswer` is the preferred answer-first surface for Web self-serve UX when present.
- `overflow` is additive and bounded so FE can show "outside the current page, but likely relevant" results without reintroducing a scope toggle.
- `coverage` is additive and bounded so FE can suppress misleading suggestions when the active corpus has no sensible candidates for that domain.
- Live local verification currently covers the Doctor/knowledge path after the documented rebuild order:
1. `POST /v1/advisory-ai/index/rebuild`
2. `POST /v1/search/index/rebuild`
`POST /v1/search/suggestions/evaluate` response notes:
- Intended for proactive suggestion chips and page-owned prompts before the user commits a search.
- Returns per-query viability plus bounded domain coverage.
- Does not require telemetry and does not record answer-frame analytics.
OpenAPI contract presence is validated by integration test:
- `UnifiedSearchEndpointsIntegrationTests.OpenApi_Includes_UnifiedSearch_Contracts`