Search/AdvisoryAI and DAL conversion to EF finishes up. Preparation for microservices consolidation.

docs/modules/advisory-ai/knowledge-search.md

@@ -26,6 +26,9 @@ LLMs can still be used as optional formatters later, but AKS correctness is grou
    - API endpoint: `POST /v1/advisory-ai/search`.
    - Index rebuild endpoint: `POST /v1/advisory-ai/index/rebuild`.
 
+Unified-search architecture reference:
+- `docs/modules/advisory-ai/unified-search-architecture.md`
+
 ## Data model
 AKS schema tables:
 - `advisoryai.kb_doc`: canonical source docs with product/version/content hash metadata.
@@ -119,8 +122,10 @@ Implemented in `src/AdvisoryAI/StellaOps.AdvisoryAI/KnowledgeSearch/KnowledgeSea
 - Unified index lifecycle:
   - Manual rebuild endpoint: `POST /v1/search/index/rebuild`.
   - Optional background refresh loop is available via `KnowledgeSearchOptions` (`UnifiedAutoIndexEnabled`, `UnifiedAutoIndexOnStartup`, `UnifiedIndexRefreshIntervalSeconds`).
-- Unified ingestion adapters now ingest from deterministic snapshot files (findings/vex/policy) plus platform catalog projection, replacing hardcoded sample chunks.
-  - Default snapshot paths:
+- Unified ingestion adapters for findings/vex/policy now use live upstream service payloads as primary source, with deterministic snapshot fallback only when upstream endpoints are unavailable or unconfigured.
+  - Live adapters: `FindingsSearchAdapter`, `VexSearchAdapter`, `PolicySearchAdapter`.
+  - Platform catalog remains a deterministic snapshot projection via `PlatformCatalogIngestionAdapter`.
+  - Default snapshot fallback paths:
     - `src/AdvisoryAI/StellaOps.AdvisoryAI/UnifiedSearch/Snapshots/findings.snapshot.json`
     - `src/AdvisoryAI/StellaOps.AdvisoryAI/UnifiedSearch/Snapshots/vex.snapshot.json`
     - `src/AdvisoryAI/StellaOps.AdvisoryAI/UnifiedSearch/Snapshots/policy.snapshot.json`
@@ -129,9 +134,28 @@ Implemented in `src/AdvisoryAI/StellaOps.AdvisoryAI/KnowledgeSearch/KnowledgeSea
   - Ranking no longer depends on ambient wall-clock time unless that option is enabled.
 - Query telemetry:
   - Unified search emits hashed query telemetry (`SHA-256` query hash, intent, domain weights, latency, top domains) via `IUnifiedSearchTelemetrySink`.
+  - Search analytics persistence stores hashed query keys (`SHA-256`, normalized) and pseudonymous user keys (tenant+user hash) in analytics/feedback artifacts.
+  - Free-form feedback comments are redacted at persistence time to avoid storing potential PII in analytics tables.
+  - Server-side search history remains user-facing functionality (raw query for history UX) and is keyed by pseudonymous user hash.
 - Web fallback behavior: when unified search fails, `UnifiedSearchClient` falls back to legacy AKS (`/v1/advisory-ai/search`) and maps grouped legacy results into unified cards (`diagnostics.mode = legacy-fallback`).
   - UI now shows an explicit degraded-mode banner for `legacy-fallback` / `fallback-empty` modes and clears it automatically on recovery.
   - Degraded-mode enter/exit transitions emit analytics markers (`__degraded_mode_enter__`, `__degraded_mode_exit__`); server-side search history intentionally ignores `__*` synthetic markers.
+- Deprecation timeline and migration milestones are tracked in `docs/modules/advisory-ai/CHANGELOG.md`.
+
+## Unified search threat model (USRCH-POL-005)
+Primary attack vectors and implemented mitigations:
+- Cross-tenant data leakage:
+  - Risk: chunks from tenant A becoming visible in tenant B through weak filtering or identity collisions.
+  - Mitigations: mandatory tenant context on AKS/unified endpoints; tenant-aware store filters (`metadata.tenant` + `global` allowance); tenant-scoped chunk/doc identity for findings/vex/policy live adapters to prevent cross-tenant upsert collisions.
+- Prompt/content injection from indexed sources:
+  - Risk: untrusted indexed text influencing synthesis or downstream operators.
+  - Mitigations: deterministic retrieval-first pipeline; synthesis grounding enforcement; analytics stores hashed query identifiers only; prompt payloads are not persisted in raw form.
+- UI/script injection via snippets:
+  - Risk: malicious `<script>`/HTML in indexed body or highlighted snippets leading to XSS in search result cards.
+  - Mitigations: backend snippet sanitization strips script and HTML tags before response mapping; web client normalizes and strips tags again as defense-in-depth.
+- Query-amplification and expensive-query DoS:
+  - Risk: oversized/invalid filters and high-rate query floods increasing DB and fusion cost.
+  - Mitigations: `q` length cap (512), strict allowlist validation for domains/entity types, per-tenant rate limiting, bounded candidate limits/timeouts in retrieval stages.
 
 ## Web behavior
 Global search now consumes AKS and supports:
@@ -143,13 +167,19 @@ Global search now consumes AKS and supports:
   - Doctor: `Run` (navigate to doctor and copy run command).
 - `More` action for "show more like this" local query expansion.
 - Search-quality metrics taxonomy is standardized on `query`, `click`, and `zero_result` event types (no legacy `search` event dependency in quality SQL).
+- Synthesis usage is tracked via dedicated `synthesis` analytics events, while quality aggregates continue to compute totals from `query` + `zero_result`.
+- Quality dashboard query dimensions are exposed as query hashes (not raw query text) for privacy-preserving analytics.
 
 ## CLI behavior
 AKS commands:
-- `stella search "<query>" [--type docs|api|doctor] [--product ...] [--version ...] [--service ...] [--tag ...] [--k N] [--json]`
+- `stella search "<query>" [--type docs|api|doctor] [--product ...] [--version ...] [--service ...] [--tag ...] [--k N] [--synthesize] [--json]`
 - `stella doctor suggest "<symptom>" [--product ...] [--version ...] [--k N] [--json]`
 - `stella advisoryai index rebuild [--json]`
 - `stella advisoryai sources prepare [--repo-root ...] [--docs-allowlist ...] [--docs-manifest-output ...] [--openapi-output ...] [--doctor-seed ...] [--doctor-controls-output ...] [--overwrite] [--json]`
+- Unified-search API operations:
+  - `POST /v1/search/query`
+  - `POST /v1/search/synthesize`
+  - `POST /v1/search/index/rebuild`
 
 Output:
 - Human mode: grouped actionable references.
@@ -168,6 +198,16 @@ Tests:
   - verifies deterministic dataset generation with >= 1000 queries.
   - verifies recall/latency metrics and top-k match behavior.
 
+Unified-search quality benchmarks:
+- Corpus: `src/AdvisoryAI/__Tests/StellaOps.AdvisoryAI.Tests/TestData/unified-search-quality-corpus.json` (250 graded queries).
+- Runner: `UnifiedSearchQualityBenchmarkRunner`.
+- Fast PR gate: `UnifiedSearchQualityBenchmarkFastSubsetTests` (50 queries).
+- Full suite: `UnifiedSearchQualityBenchmarkTests` and `UnifiedSearchPerformanceEnvelopeTests`.
+- Reports:
+  - `docs/modules/advisory-ai/unified-search-ranking-benchmark.md`
+  - `docs/modules/advisory-ai/unified-search-release-readiness.md`
+  - `docs/operations/unified-search-operations.md`
+
 ## Dedicated AKS test DB
 Compose profile:
 - `devops/compose/docker-compose.advisoryai-knowledge-test.yml`
@@ -387,10 +427,28 @@ All in `KnowledgeSearchOptions` (`src/AdvisoryAI/StellaOps.AdvisoryAI/KnowledgeS
 | `LlmAdapterBaseUrl` | `null` | G3 | LLM adapter service URL |
 | `LlmProviderId` | `null` | G3 | LLM provider selection |
 | `PopularityBoostEnabled` | `false` | G6 | Enable click-weighted ranking |
-| `PopularityBoostWeight` | `0.1` | G6 | Popularity boost factor |
-| `RoleBasedBiasEnabled` | `false` | G6 | Enable scope-based domain weighting |
+| `PopularityBoostWeight` | `0.05` | G6 | Popularity boost factor |
+| `RoleBasedBiasEnabled` | `true` | G6 | Enable scope-based domain weighting |
+| `SearchQualityMonitorEnabled` | `true` | G10 | Enable periodic quality-alert refresh |
+| `SearchQualityMonitorIntervalSeconds` | `300` | G10 | Quality-alert refresh cadence |
+| `SearchAnalyticsRetentionEnabled` | `true` | G10 | Enable automatic analytics/feedback/history pruning |
+| `SearchAnalyticsRetentionDays` | `90` | G10 | Retention window for search analytics artifacts |
+| `SearchAnalyticsRetentionIntervalSeconds` | `3600` | G10 | Retention pruning cadence |
 | `FtsLanguageConfigs` | `{}` | G9 | Per-locale FTS config map |
 
+Unified-search options (`UnifiedSearchOptions`, `src/AdvisoryAI/StellaOps.AdvisoryAI/UnifiedSearch/UnifiedSearchOptions.cs`):
+- `Enabled`
+- `BaseDomainWeights`
+- `Weighting.*`
+- `Federation.*`
+- `GravityBoost.*`
+- `Synthesis.*`
+- `Ingestion.*`
+- `Session.*`
+- `TenantFeatureFlags.<tenant>.Enabled`
+- `TenantFeatureFlags.<tenant>.FederationEnabled`
+- `TenantFeatureFlags.<tenant>.SynthesisEnabled`
+
 ## Known limitations and follow-ups
 - YAML OpenAPI ingestion is not included in MVP.
 - End-to-end benchmark against live Postgres-backed AKS service is planned as a follow-up CI lane.
@@ -398,4 +456,3 @@ All in `KnowledgeSearchOptions` (`src/AdvisoryAI/StellaOps.AdvisoryAI/KnowledgeS
 - ONNX model file (`all-MiniLM-L6-v2.onnx`, ~80MB) must be provisioned separately for deployments opting into `VectorEncoderType=onnx`. Air-gap bundles must include the model.
 - Doctor seed localization covers de-DE and fr-FR only. Other locales (es-ES, ru-RU, bg-BG, etc.) use English fallback.
 - Search quality dashboard deferred items: low-quality results table, top queries table, 30-day trend chart (require additional backend aggregation queries).
-- Periodic `SearchQualityMonitor` background job not yet wired (zero-result alerting runs on-demand via metrics endpoint).