# AdvisoryAI Knowledge Search (AKS) ## Why retrieval-first AKS is a deterministic retrieval system for operational problem solving across Stella Ops docs, OpenAPI contracts, and Doctor checks. It is designed to work offline and does not require GPU-backed or hosted LLM inference for correctness. LLMs can still be used as optional formatters later, but AKS correctness is grounded in source retrieval and explicit references. ## Scope - Module owner: `src/AdvisoryAI/**`. - Search surfaces consuming AKS: - Web global search in `src/Web/StellaOps.Web/**`. - CLI commands in `src/Cli/**`. - Doctor execution remains authoritative in Doctor module. AKS only indexes metadata and remediation references. ## Architecture 1. Ingestion/indexing: - Markdown allow-list/manifest -> section chunks. - OpenAPI aggregate (`openapi_current.json` style artifact) -> per-operation chunks + normalized operation tables. - Doctor seed + controls metadata (including CLI-discovered Doctor check catalog projection) -> doctor projection chunks. 2. Storage: - PostgreSQL tables in schema `advisoryai` via migration `src/AdvisoryAI/StellaOps.AdvisoryAI/Storage/Migrations/002_knowledge_search.sql`. 3. Retrieval: - FTS (`tsvector` + `websearch_to_tsquery`) + optional vector stage. - Deterministic fusion and tie-breaking in `KnowledgeSearchService`. 4. Delivery: - API endpoint: `POST /v1/advisory-ai/search`. - Index rebuild endpoint: `POST /v1/advisory-ai/index/rebuild`. Unified-search architecture reference: - `docs/modules/advisory-ai/unified-search-architecture.md` ## Data model AKS schema tables: - `advisoryai.kb_doc`: canonical source docs with product/version/content hash metadata. - `advisoryai.kb_chunk`: searchable units (`md_section`, `api_operation`, `doctor_check`) with anchors, spans, `tsvector`, and embeddings. - `advisoryai.api_spec`: raw OpenAPI snapshot (`jsonb`) by service. - `advisoryai.api_operation`: normalized operation records (`method`, `path`, `operation_id`, tags, request/response/security json). - `advisoryai.doctor_search_projection`: searchable doctor metadata and remediation. Vector support: - Tries `CREATE EXTENSION vector`. - If unavailable, AKS remains fully functional via FTS and deterministic array embeddings fallback. ## Deterministic ingestion rules ### Markdown - Source order: 1. Allow-list file: `src/AdvisoryAI/StellaOps.AdvisoryAI/KnowledgeSearch/knowledge-docs-allowlist.json`. 2. Generated manifest (optional, from CLI tool): `knowledge-docs-manifest.json`. 3. Fallback scan roots (`docs/**`) only if allow-list resolves no markdown files. - Chunk by H2/H3 headings. - Stable anchors using slug + duplicate suffix. - Stable chunk IDs from source path + anchor + span. - Metadata includes path, anchor, section path, tags. ### OpenAPI - Source order: 1. Aggregated OpenAPI file path (default `devops/compose/openapi_current.json`). 2. Fallback repository scan for `openapi.json` when aggregate is missing. - Parse deterministic JSON aggregate for MVP. - Emit one searchable chunk per HTTP operation. - Preserve structured operation payloads (`request_json`, `responses_json`, `security_json`). ### Doctor - Source order: 1. Seed file `src/AdvisoryAI/StellaOps.AdvisoryAI/KnowledgeSearch/doctor-search-seed.json`. 2. Controls file `src/AdvisoryAI/StellaOps.AdvisoryAI/KnowledgeSearch/doctor-search-controls.json` (contains control fields plus fallback metadata from `stella advisoryai sources prepare`). 3. Optional Doctor endpoint metadata (`DoctorChecksEndpoint`) when configured. - `stella advisoryai sources prepare` merges configured seed entries with `DoctorEngine.ListChecks()` (when available in CLI runtime) and writes enriched control projection metadata (`title`, `severity`, `description`, `remediation`, `runCommand`, `symptoms`, `tags`, `references`). - Emit doctor chunk + projection record including: - `checkCode`, `title`, `severity`, `runCommand`, remediation, symptoms. - control metadata (`control`, `requiresConfirmation`, `isDestructive`, `inspectCommand`, `verificationCommand`). ## Ranking strategy Implemented in `src/AdvisoryAI/StellaOps.AdvisoryAI/KnowledgeSearch/KnowledgeSearchService.cs`: - Candidate retrieval: - lexical set from FTS. - optional vector set from embedding candidates. - Fusion: - reciprocal rank fusion style scoring. - Deterministic boosts: - exact `checkCode` match. - exact `operationId` match. - `METHOD /path` match. - filter-aligned service/tag boosts. - Deterministic ordering: - score desc -> kind asc -> chunk id asc. ## API contract ### Search - `POST /v1/advisory-ai/search` - Legacy notice: endpoint emits deprecation metadata and points to unified replacement `POST /v1/search/query`. - Authorization: `advisory-ai:operate` (or `advisory-ai:admin`). - Filter validation: `filters.type` allowlist is strictly enforced (`docs`, `api`, `doctor`); unsupported values return HTTP 400. - Request: - `q` (required), `k`, `filters.type|product|version|service|tags`, `includeDebug`. - Response: - typed results (`docs|api|doctor`) with snippet, score, and open action. ### Rebuild - `POST /v1/advisory-ai/index/rebuild` - Rebuilds AKS deterministically from local docs/specs/doctor metadata. - Authorization: `advisory-ai:admin`. ## Localization runtime contract - AdvisoryAI WebService localization is enabled through `AddStellaOpsLocalization(...)`, embedded service bundles (`Translations/*.advisoryai.json`), and `AddRemoteTranslationBundles()`. - Locale behavior follows backend contract: `X-Locale` -> `Accept-Language` -> default locale. - Supported service locales for this rollout slice: `en-US`, `de-DE`. - Remote translation bundles are enabled when Platform base URL is configured via `STELLAOPS_PLATFORM_URL`, `Platform:BaseUrl`, or `StellaOps:Platform:BaseUrl`. - Localized validation keys used by both `POST /v1/advisory-ai/search` and `POST /v1/search/query`: - `advisoryai.validation.q_required` - `advisoryai.validation.q_max_512` - `advisoryai.validation.tenant_required` ## Unified search interoperability - Unified endpoint: `POST /v1/search/query`. - Query validation: `q` is required and capped at 512 characters. - Tenant validation: unified and AKS search endpoints now require tenant context (`X-StellaOps-Tenant` or `X-Tenant-Id`) and bind tenant into backend search filters. - Unified filter allowlists are enforced server-side: - Supported `filters.domains`: `knowledge`, `findings`, `vex`, `policy`, `platform`. - Supported `filters.entityTypes`: `docs`, `api`, `doctor`, `finding`, `vex_statement`, `policy_rule`, `platform_entity`. - Unsupported domain/entity filter values are rejected with HTTP 400; they are not silently broadened to an unfiltered query. - Unified index lifecycle: - Manual rebuild endpoint: `POST /v1/search/index/rebuild`. - Optional background refresh loop is available via `KnowledgeSearchOptions` (`UnifiedAutoIndexEnabled`, `UnifiedAutoIndexOnStartup`, `UnifiedIndexRefreshIntervalSeconds`). - Unified ingestion adapters for findings/vex/policy now use live upstream service payloads as primary source, with deterministic snapshot fallback only when upstream endpoints are unavailable or unconfigured. - Live adapters: `FindingsSearchAdapter`, `VexSearchAdapter`, `PolicySearchAdapter`. - Platform catalog remains a deterministic snapshot projection via `PlatformCatalogIngestionAdapter`. - Default snapshot fallback paths: - `src/AdvisoryAI/StellaOps.AdvisoryAI/UnifiedSearch/Snapshots/findings.snapshot.json` - `src/AdvisoryAI/StellaOps.AdvisoryAI/UnifiedSearch/Snapshots/vex.snapshot.json` - `src/AdvisoryAI/StellaOps.AdvisoryAI/UnifiedSearch/Snapshots/policy.snapshot.json` - Ranking determinism: - Freshness boost is disabled by default and only applies when `UnifiedFreshnessBoostEnabled` is explicitly enabled. - Ranking no longer depends on ambient wall-clock time unless that option is enabled. - Query telemetry: - Unified search emits hashed query telemetry (`SHA-256` query hash, intent, domain weights, latency, top domains) via `IUnifiedSearchTelemetrySink`. - Search analytics persistence stores hashed query keys (`SHA-256`, normalized) and pseudonymous user keys (tenant+user hash) in analytics/feedback artifacts. - Free-form feedback comments are redacted at persistence time to avoid storing potential PII in analytics tables. - Server-side search history remains user-facing functionality (raw query for history UX) and is keyed by pseudonymous user hash. - Web fallback behavior: when unified search fails, `UnifiedSearchClient` falls back to legacy AKS (`/v1/advisory-ai/search`) and maps grouped legacy results into unified cards (`diagnostics.mode = legacy-fallback`). - UI now shows an explicit degraded-mode banner for `legacy-fallback` / `fallback-empty` modes and clears it automatically on recovery. - Degraded-mode enter/exit transitions emit analytics markers (`__degraded_mode_enter__`, `__degraded_mode_exit__`); server-side search history intentionally ignores `__*` synthetic markers. - Deprecation timeline and migration milestones are tracked in `docs/modules/advisory-ai/CHANGELOG.md`. ## Unified search threat model (USRCH-POL-005) Primary attack vectors and implemented mitigations: - Cross-tenant data leakage: - Risk: chunks from tenant A becoming visible in tenant B through weak filtering or identity collisions. - Mitigations: mandatory tenant context on AKS/unified endpoints; tenant-aware store filters (`metadata.tenant` + `global` allowance); tenant-scoped chunk/doc identity for findings/vex/policy live adapters to prevent cross-tenant upsert collisions. - Prompt/content injection from indexed sources: - Risk: untrusted indexed text influencing synthesis or downstream operators. - Mitigations: deterministic retrieval-first pipeline; synthesis grounding enforcement; analytics stores hashed query identifiers only; prompt payloads are not persisted in raw form. - UI/script injection via snippets: - Risk: malicious `