stela ops usage fixes roles propagation and timoeut, one account to support multi tenants, migrations consolidation, search to support documentation, doctor and open api vector db search

2026-02-22 19:27:54 +02:00
parent a29f438f53
commit bd8fee6ed8
373 changed files with 832097 additions and 3369 deletions
--- a/docs/modules/advisory-ai/README.md
+++ b/docs/modules/advisory-ai/README.md
@@ -17,6 +17,7 @@ Advisory AI is the retrieval-augmented assistant that synthesizes advisory and V
 - RAG pipeline drawing from Conseiller, Excititor, VEX Lens, Policy Engine, and SBOM Service data.
 - Prompt templates and guard models enforcing provenance and redaction policies.
 - Vercel/offline inference workers with deterministic caching of generated artefacts.
+- AdvisoryAI Knowledge Search (AKS) for deterministic docs/API/Doctor retrieval: `docs/modules/advisory-ai/knowledge-search.md`.

 ## Integrations & dependencies
 - Authority for tenant-aware access control.
--- a/docs/modules/advisory-ai/guides/api.md
+++ b/docs/modules/advisory-ai/guides/api.md
@@ -202,6 +202,79 @@ Fetch cached artefact (same envelope as §6). Requires `advisory-ai:view`.

 When plan preview is enabled (feature flag `advisoryAi.planPreview.enabled`), this endpoint returns the orchestration plan using `AdvisoryPipelinePlanResponse` (task metadata, chunk/vector counts). Requires `advisory-ai:operate`.

+### 7.8 `LLM Adapter Exposure (RVM-08)`
+
+AdvisoryAI now exposes a unified adapter surface for provider discovery and OpenAI-compatible completion calls.
+
+- `GET /v1/advisory-ai/adapters/llm/providers`
+- `POST /v1/advisory-ai/adapters/llm/{providerId}/chat/completions`
+- `POST /v1/advisory-ai/adapters/openai/v1/chat/completions` (alias for `providerId=openai`)
+
+Scope requirements:
+
+- Read/discovery: `advisory:adapter:read` or `advisory:run`
+- Invoke/completion: `advisory:adapter:invoke` or `advisory:openai:invoke` or `advisory:{providerId}:invoke` or `advisory:run`
+
+Operational notes:
+
+- Adapter endpoints are disabled by default and must be explicitly enabled by config.
+- `stream=true` is currently rejected on adapter completion endpoints.
+- Provider discovery returns configuration/validation/exposure state per provider so operators can verify whether `openai` is configured before traffic cutover.
+
+Runtime config keys:
+
+- `AdvisoryAI:Adapters:Llm:Enabled` (env `ADVISORYAI__AdvisoryAI__Adapters__Llm__Enabled`)
+- `AdvisoryAI:LlmProviders:ConfigDirectory` (env `ADVISORYAI__AdvisoryAI__LlmProviders__ConfigDirectory`)
+
+Gateway exposure paths:
+
+- `/v1/advisory-ai/adapters`
+- `/api/v1/advisory-ai/adapters`
+
+### 7.9 `Knowledge Search (AKS)`
+
+Deterministic retrieval API for docs, OpenAPI operations, and Doctor checks.
+
+- `POST /v1/advisory-ai/search`
+- `POST /v1/advisory-ai/index/rebuild`
+
+Scope notes:
+- Search: one of `advisory:run`, `advisory:search`, `advisory:read`.
+- Rebuild: one of `advisory:run`, `advisory:admin`, `advisory:index:write`.
+
+Search request:
+```json
+{
+  "q": "docker login fails with x509 unknown authority",
+  "k": 10,
+  "filters": {
+    "type": ["docs", "doctor"],
+    "product": "stella-ops",
+    "version": "2026.02",
+    "service": "gateway",
+    "tags": ["troubleshooting", "tls"]
+  },
+  "includeDebug": false
+}
+```
+
+Search response contains only grounded results with source actions:
+- docs: `{ path, anchor, spanStart, spanEnd }`
+- api: `{ service, method, path, operationId }`
+- doctor: `{ checkCode, severity, canRun, runCommand }`
+
+Rebuild response:
+```json
+{
+  "documentCount": 12034,
+  "chunkCount": 68291,
+  "apiSpecCount": 41,
+  "apiOperationCount": 2132,
+  "doctorProjectionCount": 84,
+  "durationMs": 8245
+}
+```
+
 ## 8. Error model

 Errors follow a standard problem+JSON envelope:
--- a/docs/modules/advisory-ai/knowledge-search.md
+++ b/docs/modules/advisory-ai/knowledge-search.md
@@ -0,0 +1,137 @@
+# AdvisoryAI Knowledge Search (AKS)
+
+## Why retrieval-first
+AKS is a deterministic retrieval system for operational problem solving across Stella Ops docs, OpenAPI contracts, and Doctor checks. It is designed to work offline and does not require GPU-backed or hosted LLM inference for correctness.
+
+LLMs can still be used as optional formatters later, but AKS correctness is grounded in source retrieval and explicit references.
+
+## Scope
+- Module owner: `src/AdvisoryAI/**`.
+- Search surfaces consuming AKS:
+  - Web global search in `src/Web/StellaOps.Web/**`.
+  - CLI commands in `src/Cli/**`.
+- Doctor execution remains authoritative in Doctor module. AKS only indexes metadata and remediation references.
+
+## Architecture
+1. Ingestion/indexing:
+   - Markdown (`docs/**`) -> section chunks.
+   - OpenAPI (`openapi.json`) -> per-operation chunks + normalized operation tables.
+   - Doctor seed/metadata -> doctor projection chunks.
+2. Storage:
+   - PostgreSQL tables in schema `advisoryai` via migration `src/AdvisoryAI/StellaOps.AdvisoryAI/Storage/Migrations/002_knowledge_search.sql`.
+3. Retrieval:
+   - FTS (`tsvector` + `websearch_to_tsquery`) + optional vector stage.
+   - Deterministic fusion and tie-breaking in `KnowledgeSearchService`.
+4. Delivery:
+   - API endpoint: `POST /v1/advisory-ai/search`.
+   - Index rebuild endpoint: `POST /v1/advisory-ai/index/rebuild`.
+
+## Data model
+AKS schema tables:
+- `advisoryai.kb_doc`: canonical source docs with product/version/content hash metadata.
+- `advisoryai.kb_chunk`: searchable units (`md_section`, `api_operation`, `doctor_check`) with anchors, spans, `tsvector`, and embeddings.
+- `advisoryai.api_spec`: raw OpenAPI snapshot (`jsonb`) by service.
+- `advisoryai.api_operation`: normalized operation records (`method`, `path`, `operation_id`, tags, request/response/security json).
+- `advisoryai.doctor_search_projection`: searchable doctor metadata and remediation.
+
+Vector support:
+- Tries `CREATE EXTENSION vector`.
+- If unavailable, AKS remains fully functional via FTS and deterministic array embeddings fallback.
+
+## Deterministic ingestion rules
+### Markdown
+- Chunk by H2/H3 headings.
+- Stable anchors using slug + duplicate suffix.
+- Stable chunk IDs from source path + anchor + span.
+- Metadata includes path, anchor, section path, tags.
+
+### OpenAPI
+- Parse `openapi.json` only for deterministic MVP.
+- Emit one searchable chunk per HTTP operation.
+- Preserve structured operation payloads (`request_json`, `responses_json`, `security_json`).
+
+### Doctor
+- Source order:
+  1. Seed file `src/AdvisoryAI/StellaOps.AdvisoryAI/KnowledgeSearch/doctor-search-seed.json`.
+  2. Optional Doctor endpoint metadata (`DoctorChecksEndpoint`) when configured.
+- Emit doctor chunk + projection record including:
+  - `checkCode`, `title`, `severity`, `runCommand`, remediation, symptoms.
+
+## Ranking strategy
+Implemented in `src/AdvisoryAI/StellaOps.AdvisoryAI/KnowledgeSearch/KnowledgeSearchService.cs`:
+- Candidate retrieval:
+  - lexical set from FTS.
+  - optional vector set from embedding candidates.
+- Fusion:
+  - reciprocal rank fusion style scoring.
+- Deterministic boosts:
+  - exact `checkCode` match.
+  - exact `operationId` match.
+  - `METHOD /path` match.
+  - filter-aligned service/tag boosts.
+- Deterministic ordering:
+  - score desc -> kind asc -> chunk id asc.
+
+## API contract
+### Search
+- `POST /v1/advisory-ai/search`
+- Request:
+  - `q` (required), `k`, `filters.type|product|version|service|tags`, `includeDebug`.
+- Response:
+  - typed results (`docs|api|doctor`) with snippet, score, and open action.
+
+### Rebuild
+- `POST /v1/advisory-ai/index/rebuild`
+- Rebuilds AKS deterministically from local docs/specs/doctor metadata.
+
+## Web behavior
+Global search now consumes AKS and supports:
+- Mixed grouped results (`Docs`, `API Endpoints`, `Doctor Checks`).
+- Type filter chips.
+- Result actions:
+  - Docs: `Open`.
+  - API: `Curl` (copy command).
+  - Doctor: `Run` (navigate to doctor and copy run command).
+- `More` action for "show more like this" local query expansion.
+
+## CLI behavior
+AKS commands:
+- `stella search "<query>" [--type docs|api|doctor] [--product ...] [--version ...] [--service ...] [--tag ...] [--k N] [--json]`
+- `stella doctor suggest "<symptom>" [--product ...] [--version ...] [--k N] [--json]`
+- `stella advisoryai index rebuild [--json]`
+
+Output:
+- Human mode: grouped actionable references.
+- JSON mode: stable machine-readable payload.
+
+## Test/benchmark strategy
+Implemented benchmark framework:
+- Generator: `KnowledgeSearchBenchmarkDatasetGenerator` (deterministic synthetic set with explicit ground truth).
+- Runner: `KnowledgeSearchBenchmarkRunner` (recall@k, p50/p95 latency, stability pass).
+- Models/serialization:
+  - `KnowledgeSearchBenchmarkModels.cs`
+  - `KnowledgeSearchBenchmarkJson.cs`
+
+Tests:
+- `src/AdvisoryAI/__Tests/StellaOps.AdvisoryAI.Tests/KnowledgeSearch/KnowledgeSearchBenchmarkTests.cs`
+  - verifies deterministic dataset generation with >= 1000 queries.
+  - verifies recall/latency metrics and top-k match behavior.
+
+## Dedicated AKS test DB
+Compose profile:
+- `devops/compose/docker-compose.advisoryai-knowledge-test.yml`
+
+Init script:
+- `devops/compose/postgres-init/advisoryai-knowledge-test/01_extensions.sql`
+
+Example workflow:
+```bash
+docker compose -f devops/compose/docker-compose.advisoryai-knowledge-test.yml up -d
+stella advisoryai index rebuild --json
+dotnet test src/AdvisoryAI/__Tests/StellaOps.AdvisoryAI.Tests/StellaOps.AdvisoryAI.Tests.csproj
+```
+
+## Known limitations and follow-ups
+- YAML OpenAPI ingestion is not included in MVP.
+- End-to-end benchmark against live Postgres-backed AKS service is planned as a follow-up CI lane.
+- Optional external embedding providers can be added later without changing API contracts.