stella-ops.org/git.stella-ops.org
1
0
Fork 0
Code Issues Pull Requests Actions 2 Releases Wiki Activity
Files
5e850d056ba9d9d6348a3660ffd2e2a17d9d2536
git.stella-ops.org/docs/modules/advisory-ai/knowledge-search.md

38 KiB
Raw Blame History

AdvisoryAI Knowledge Search (AKS)

Why retrieval-first

AKS is a deterministic retrieval system for operational problem solving across Stella Ops docs, OpenAPI contracts, and Doctor checks. It is designed to work offline and does not require GPU-backed or hosted LLM inference for correctness.

LLMs can still be used as optional formatters later, but AKS correctness is grounded in source retrieval and explicit references.

Scope

  • Module owner: src/AdvisoryAI/**.
  • Search surfaces consuming AKS:
    • Web global search in src/Web/StellaOps.Web/**.
    • CLI commands in src/Cli/**.
  • Doctor execution remains authoritative in Doctor module. AKS only indexes metadata and remediation references.

Architecture

  1. Ingestion/indexing:
    • Markdown allow-list/manifest -> section chunks.
    • OpenAPI aggregate (openapi_current.json style artifact) -> per-operation chunks + normalized operation tables.
    • Doctor seed + controls metadata (including CLI-discovered Doctor check catalog projection) -> doctor projection chunks.
  2. Storage:
    • PostgreSQL tables in schema advisoryai via migration src/AdvisoryAI/StellaOps.AdvisoryAI/Storage/Migrations/002_knowledge_search.sql.
  3. Retrieval:
    • FTS (tsvector + websearch_to_tsquery) + optional vector stage.
    • Deterministic fusion and tie-breaking in KnowledgeSearchService.
  4. Delivery:
    • API endpoint: POST /v1/advisory-ai/search.
    • Index rebuild endpoint: POST /v1/advisory-ai/index/rebuild.

Unified-search architecture reference:

  • docs/modules/advisory-ai/unified-search-architecture.md

Data model

AKS schema tables:

  • advisoryai.kb_doc: canonical source docs with product/version/content hash metadata.
  • advisoryai.kb_chunk: searchable units (md_section, api_operation, doctor_check) with anchors, spans, tsvector, and embeddings.
  • advisoryai.api_spec: raw OpenAPI snapshot (jsonb) by service.
  • advisoryai.api_operation: normalized operation records (method, path, operation_id, tags, request/response/security json).
  • advisoryai.doctor_search_projection: searchable doctor metadata and remediation.

Vector support:

  • Tries CREATE EXTENSION vector.
  • If unavailable, AKS remains fully functional via FTS and deterministic array embeddings fallback.

Deterministic ingestion rules

Markdown

  • Source order:
    1. Allow-list file: src/AdvisoryAI/StellaOps.AdvisoryAI/KnowledgeSearch/knowledge-docs-allowlist.json.
    2. Generated manifest (optional, from CLI tool): knowledge-docs-manifest.json.
    3. Fallback scan roots (docs/**) only if allow-list resolves no markdown files.
  • Chunk by H2/H3 headings.
  • Stable anchors using slug + duplicate suffix.
  • Stable chunk IDs from source path + anchor + span.
  • Metadata includes path, anchor, section path, tags.

OpenAPI

  • Source order:
    1. Aggregated OpenAPI file path (default devops/compose/openapi_current.json).
    2. Fallback repository scan for openapi.json when aggregate is missing.
  • Parse deterministic JSON aggregate for MVP.
  • Emit one searchable chunk per HTTP operation.
  • Preserve structured operation payloads (request_json, responses_json, security_json).

Doctor

  • Source order:
    1. Seed file src/AdvisoryAI/StellaOps.AdvisoryAI/KnowledgeSearch/doctor-search-seed.json.
    2. Controls file src/AdvisoryAI/StellaOps.AdvisoryAI/KnowledgeSearch/doctor-search-controls.json (contains control fields plus fallback metadata from stella advisoryai sources prepare).
    3. Optional Doctor endpoint metadata (DoctorChecksEndpoint) when configured.
  • stella advisoryai sources prepare merges configured seed entries with DoctorEngine.ListChecks() (when available in CLI runtime) and writes enriched control projection metadata (title, severity, description, remediation, runCommand, symptoms, tags, references).
  • Emit doctor chunk + projection record including:
    • checkCode, title, severity, runCommand, remediation, symptoms.
    • control metadata (control, requiresConfirmation, isDestructive, inspectCommand, verificationCommand).

Ranking strategy

Implemented in src/AdvisoryAI/StellaOps.AdvisoryAI/KnowledgeSearch/KnowledgeSearchService.cs:

  • Candidate retrieval:
    • lexical set from FTS.
    • optional vector set from embedding candidates.
  • Fusion:
    • reciprocal rank fusion style scoring.
  • Deterministic boosts:
    • exact checkCode match.
    • exact operationId match.
    • METHOD /path match.
    • filter-aligned service/tag boosts.
  • Deterministic ordering:
    • score desc -> kind asc -> chunk id asc.

API contract

Search

  • POST /v1/advisory-ai/search
  • Legacy notice: endpoint emits deprecation metadata and points to unified replacement POST /v1/search/query.
  • Authorization: advisory-ai:operate (or advisory-ai:admin).
  • Filter validation: filters.type allowlist is strictly enforced (docs, api, doctor); unsupported values return HTTP 400.
  • Request:
    • q (required), k, filters.type|product|version|service|tags, includeDebug.
  • Response:
    • typed results (docs|api|doctor) with snippet, score, and open action.

Rebuild

  • POST /v1/advisory-ai/index/rebuild
  • Rebuilds AKS deterministically from local docs/specs/doctor metadata.
  • Authorization: advisory-ai:admin.

Localization runtime contract

  • AdvisoryAI WebService localization is enabled through AddStellaOpsLocalization(...), embedded service bundles (Translations/*.advisoryai.json), and AddRemoteTranslationBundles().
  • Locale behavior follows backend contract: X-Locale -> Accept-Language -> default locale.
  • Supported service locales for this rollout slice: en-US, de-DE.
  • Remote translation bundles are enabled when Platform base URL is configured via STELLAOPS_PLATFORM_URL, Platform:BaseUrl, or StellaOps:Platform:BaseUrl.
  • Localized validation keys used by both POST /v1/advisory-ai/search and POST /v1/search/query:
    • advisoryai.validation.q_required
    • advisoryai.validation.q_max_512
    • advisoryai.validation.tenant_required

Unified search interoperability

  • Unified endpoint: POST /v1/search/query.
  • Query validation: q is required and capped at 512 characters.
  • Tenant validation: unified and AKS search endpoints now require tenant context (X-StellaOps-Tenant or X-Tenant-Id) and bind tenant into backend search filters.
  • Unified filter allowlists are enforced server-side:
    • Supported filters.domains: knowledge, findings, vex, policy, platform.
    • Supported filters.entityTypes: docs, api, doctor, finding, vex_statement, policy_rule, platform_entity.
  • Unsupported domain/entity filter values are rejected with HTTP 400; they are not silently broadened to an unfiltered query.
  • Web ambient contract:
    • Global search emits ambient context with each unified query: currentRoute, visibleEntityKeys, recentSearches, sessionId, and optional lastAction (action, source, queryHint, domain, entityKey, route, occurredAt).
    • Contract remains backward-compatible: if an API deployment does not yet consume lastAction, unknown ambient fields are ignored and base search behavior remains unchanged.
    • UI suggestion behavior now combines obvious route defaults with one strategic non-obvious suggestion and action-aware variants (for example, policy/VEX impact and incident timeline pivots).
    • Unified search now also returns optional contextAnswer metadata with status, code, summary, reason, evidence, bounded citations, and bounded follow-up questions.
    • contextAnswer.status is deterministic and must be one of grounded, clarify, or insufficient.
    • Suggestion viability now returns additive readiness detail: viabilityState (grounded, needs_clarification, no_match, scope_unready, corpus_unready) plus scopeReady.
    • Starter chips and page-owned questions must only be treated as executable when viability is grounded; broad clarify-only suggestions are intentionally hidden.
  • Unified index lifecycle:
    • Manual rebuild endpoint: POST /v1/search/index/rebuild.
    • Optional background refresh loop is available via KnowledgeSearchOptions (UnifiedAutoIndexEnabled, UnifiedAutoIndexOnStartup, UnifiedIndexRefreshIntervalSeconds).
  • Unified ingestion adapters for findings/vex/policy now use live upstream service payloads as primary source, with deterministic snapshot fallback only when upstream endpoints are unavailable or unconfigured.
    • Live adapters: FindingsSearchAdapter, VexSearchAdapter, PolicySearchAdapter.
    • Platform catalog remains a deterministic snapshot projection via PlatformCatalogIngestionAdapter.
    • Live compose/runtime must wire all three adapter bases together: AdvisoryAI:KnowledgeSearch:FindingsAdapterBaseUrl, ...:VexAdapterBaseUrl, and ...:PolicyAdapterBaseUrl. Leaving only findings configured produces grounded findings cards while VEX/policy answers degrade to corpus_unready/zero-chunk responses.
    • Default snapshot fallback paths:
      • src/AdvisoryAI/StellaOps.AdvisoryAI/UnifiedSearch/Snapshots/findings.snapshot.json
      • src/AdvisoryAI/StellaOps.AdvisoryAI/UnifiedSearch/Snapshots/vex.snapshot.json
      • src/AdvisoryAI/StellaOps.AdvisoryAI/UnifiedSearch/Snapshots/policy.snapshot.json
      • src/AdvisoryAI/StellaOps.AdvisoryAI/UnifiedSearch/Snapshots/graph.snapshot.json
      • src/AdvisoryAI/StellaOps.AdvisoryAI/UnifiedSearch/Snapshots/opsmemory.snapshot.json
      • src/AdvisoryAI/StellaOps.AdvisoryAI/UnifiedSearch/Snapshots/timeline.snapshot.json
      • src/AdvisoryAI/StellaOps.AdvisoryAI/UnifiedSearch/Snapshots/scanner.snapshot.json
  • Ranking determinism:
    • Freshness boost is disabled by default and only applies when UnifiedFreshnessBoostEnabled is explicitly enabled.
    • Ranking no longer depends on ambient wall-clock time unless that option is enabled.
    • Generic advisory/security-id intent (cve, ghsa, vulnerability/advisory wording) boosts findings and VEX evidence ahead of API-operation knowledge cards unless the query explicitly asks for API/endpoint material.
  • Query telemetry:
    • Unified search emits hashed query telemetry (SHA-256 query hash, intent, domain weights, latency, top domains) via IUnifiedSearchTelemetrySink.
    • Search analytics persistence stores hashed query keys (SHA-256, normalized) and pseudonymous user keys (tenant+user hash) in analytics/feedback artifacts.
    • Self-serve analytics is optional and privacy-preserving: when clients emit answer_frame, reformulation, or rescue_action, persistence stores a tenant-scoped hashed session id plus bounded answer metadata (answer_status, answer_code) instead of raw prompt history.
    • AdvisoryAI:KnowledgeSearch:SearchTelemetryEnabled=false disables analytics persistence, feedback persistence, popularity-map reads, and unified-search telemetry sink emission. Retrieval, scope weighting, suggestions, and search history remain functional.
    • New ranking behavior does not depend on telemetry. Implicit scope weighting, overflow surfacing, answer blending, and suggestion viability all work when analytics sinks are disabled or no client analytics events are emitted.
    • Quality metrics surface self-serve gaps as fallbackAnswerRate, clarifyRate, insufficientRate, reformulationCount, rescueActionCount, and abandonedFallbackCount; alerting adds fallback_loop and abandoned_fallback signals for backlog review.
    • Free-form feedback comments are redacted at persistence time to avoid storing potential PII in analytics tables.
    • Server-side search history remains user-facing functionality (raw query for history UX) and is keyed by pseudonymous user hash.
  • Web fallback behavior: when unified search fails, UnifiedSearchClient falls back to legacy AKS (/v1/advisory-ai/search) and maps grouped legacy results into unified cards (diagnostics.mode = legacy-fallback).
    • UI now shows an explicit degraded-mode banner for legacy-fallback / fallback-empty modes and clears it automatically on recovery.
    • Degraded-mode enter/exit transitions emit analytics markers (__degraded_mode_enter__, __degraded_mode_exit__); server-side search history intentionally ignores __* synthetic markers.
  • Deprecation timeline and migration milestones are tracked in docs/modules/advisory-ai/CHANGELOG.md.

Unified search threat model (USRCH-POL-005)

Primary attack vectors and implemented mitigations:

  • Cross-tenant data leakage:
    • Risk: chunks from tenant A becoming visible in tenant B through weak filtering or identity collisions.
    • Mitigations: mandatory tenant context on AKS/unified endpoints; tenant-aware store filters (metadata.tenant + global allowance); tenant-scoped chunk/doc identity for findings/vex/policy live adapters to prevent cross-tenant upsert collisions.
  • Prompt/content injection from indexed sources:
    • Risk: untrusted indexed text influencing synthesis or downstream operators.
    • Mitigations: deterministic retrieval-first pipeline; synthesis grounding enforcement; analytics stores hashed query identifiers only; prompt payloads are not persisted in raw form.
  • UI/script injection via snippets:
    • Risk: malicious <script>/HTML in indexed body or highlighted snippets leading to XSS in search result cards.
    • Mitigations: backend snippet sanitization strips script and HTML tags before response mapping; web client normalizes and strips tags again as defense-in-depth.
  • Query-amplification and expensive-query DoS:
    • Risk: oversized/invalid filters and high-rate query floods increasing DB and fusion cost.
    • Mitigations: q length cap (512), strict allowlist validation for domains/entity types, per-tenant rate limiting, bounded candidate limits/timeouts in retrieval stages.

Web behavior

Global search now consumes AKS and supports:

  • Mixed grouped results (Docs, API Endpoints, Doctor Checks).
  • Type filter chips.
  • Result actions:
    • Docs: Open.
    • API: Copy Curl plus Copy Operation ID; API cards must not navigate to dead-end placeholder routes.
    • Doctor: Run (navigate to doctor and copy run command).
  • More action for "show more like this" local query expansion.
  • A shared mode switch (Find, Explain, Act) across search and AdvisoryAI with mode-aware chip ranking and handoff prompts.
  • An answer-first search experience: every non-empty search renders a visible answer state (grounded, clarify, insufficient) before raw cards.
    • Preferred source is backend contextAnswer.
    • FE shell composition remains only as a backward-compatible fallback for older API deployments that do not emit contextAnswer.
  • Answer shaping is query-driven:
    • compare-like queries widen the blended-answer score band so close evidence clusters are summarized together
    • scoped troubleshoot-like queries narrow the band so the dominant current-scope answer stays decisive
    • empty or unready corpora return explicit insufficient codes instead of misleading clarify states
  • Page-owned self-serve questions and clarifiers, defined in docs/modules/ui/search-self-serve-contract.md, so search can offer "Common questions" and recovery prompts without per-page conditionals in the component.
  • Zero-result rescue actions that keep the current query visible while broadening scope, trying a related pivot, retrying with page context, or opening AdvisoryAI reformulation.
  • AdvisoryAI evidence-first next-step cards that can return search pivots (chat_next_step_search, chat_next_step_policy) back into global search or open cited evidence/context directly.
  • Search-quality metrics taxonomy is standardized on query, click, and zero_result event types (no legacy search event dependency in quality SQL).
  • Synthesis usage is tracked via dedicated synthesis analytics events, while quality aggregates continue to compute totals from query + zero_result.
  • Quality dashboard query dimensions are exposed as query hashes (not raw query text) for privacy-preserving analytics.

CLI behavior

AKS commands:

  • stella search "<query>" [--type docs|api|doctor] [--product ...] [--version ...] [--service ...] [--tag ...] [--k N] [--synthesize] [--json]
  • stella doctor suggest "<symptom>" [--product ...] [--version ...] [--k N] [--json]
  • stella advisoryai index rebuild [--json]
  • stella advisoryai sources prepare [--repo-root ...] [--docs-allowlist ...] [--docs-manifest-output ...] [--openapi-output ...] [--doctor-seed ...] [--doctor-controls-output ...] [--overwrite] [--json]
  • Unified-search API operations:
    • POST /v1/search/query
    • POST /v1/search/suggestions/evaluate
    • POST /v1/search/synthesize
    • POST /v1/search/index/rebuild

POST /v1/search/query additive response fields:

  • overflow: bounded related results that fell outside the current ambient page scope but remain materially relevant. The backend only emits this section when the outside-scope candidate outranks the scoped winner or stays inside a narrow relative score band.
  • coverage: bounded domain coverage diagnostics so FE can suppress suggestions or chips when the active corpus is empty for that suggestion.

POST /v1/search/suggestions/evaluate:

  • Accepts a bounded query set plus optional filters/ambient context.
  • Returns per-query viability, matched domain, candidate count, and the same bounded coverage view used by the main query response.
  • Intended for FE-owned suggestion chips and other proactive prompts so the UI does not advertise dead searches.

Example:

curl -X POST http://127.0.0.1:10451/v1/search/suggestions/evaluate \
  -H "Content-Type: application/json" \
  -H "X-StellaOps-Tenant: test-tenant" \
  -H "Authorization: Bearer <token-with-advisory-ai:operate>" \
  -d '{
    "queries": ["database connectivity", "oidc readiness"],
    "ambient": {
      "currentRoute": "/ops/operations/doctor"
    }
  }'

Notes:

  • Do not assume stella is already installed on PATH in a source checkout. Build or run it from source as described in docs/API_CLI_REFERENCE.md and docs/modules/cli/guides/quickstart.md.
  • stella advisoryai sources prepare needs STELLAOPS_BACKEND_URL (or equivalent CLI config) when live Doctor check discovery is expected. Without that URL, use the checked-in Doctor seed/control files and the HTTP rebuild endpoints for local verification.

Output:

  • Human mode: grouped actionable references.
  • JSON mode: stable machine-readable payload.

Test/benchmark strategy

Implemented benchmark framework:

  • Generator: KnowledgeSearchBenchmarkDatasetGenerator (deterministic synthetic set with explicit ground truth).
  • Runner: KnowledgeSearchBenchmarkRunner (recall@k, p50/p95 latency, stability pass).
  • Models/serialization:
    • KnowledgeSearchBenchmarkModels.cs
    • KnowledgeSearchBenchmarkJson.cs

Tests:

  • src/AdvisoryAI/__Tests/StellaOps.AdvisoryAI.Tests/KnowledgeSearch/KnowledgeSearchBenchmarkTests.cs
    • verifies deterministic dataset generation with >= 1000 queries.
    • verifies recall/latency metrics and top-k match behavior.

Unified-search quality benchmarks:

  • Corpus: src/AdvisoryAI/__Tests/StellaOps.AdvisoryAI.Tests/TestData/unified-search-quality-corpus.json (250 graded queries).
  • Runner: UnifiedSearchQualityBenchmarkRunner.
  • Fast PR gate: UnifiedSearchQualityBenchmarkFastSubsetTests (50 queries).
  • Full suite: UnifiedSearchQualityBenchmarkTests and UnifiedSearchPerformanceEnvelopeTests.
  • Reports:
    • docs/modules/advisory-ai/unified-search-ranking-benchmark.md
    • docs/modules/advisory-ai/unified-search-release-readiness.md
    • docs/operations/unified-search-operations.md

Dedicated AKS test DB

Compose profile:

  • devops/compose/docker-compose.advisoryai-knowledge-test.yml

Init script:

  • devops/compose/postgres-init/advisoryai-knowledge-test/01_extensions.sql

Example workflow:

docker compose -f devops/compose/docker-compose.advisoryai-knowledge-test.yml up -d
stella advisoryai sources prepare --json
stella advisoryai index rebuild --json
dotnet test src/AdvisoryAI/__Tests/StellaOps.AdvisoryAI.Tests/StellaOps.AdvisoryAI.Tests.csproj

Search improvement sprints (G1–G10) — testing infrastructure guide

Ten search improvement sprints (SPRINT_20260224_101 through SPRINT_20260224_110) were implemented as a batch. This section documents how to set up infrastructure and run the full test suite.

Sprint inventory

Sprint Gap Topic Module(s)
101 G5 FTS English stemming + trigram fuzzy AdvisoryAI (backend)
102 G1 ONNX semantic vector encoder AdvisoryAI (backend)
103 G2 Cross-domain live-data adapters AdvisoryAI (backend)
104 G3 LLM-grounded synthesis engine AdvisoryAI (backend)
105 G4 Search onboarding + guided discovery + "Did you mean?" FE + AdvisoryAI
106 G6 Search personalization (popularity boost, role-based bias, history) AdvisoryAI + FE
107 G7 Search → Chat bridge ("Ask AI" button) FE
108 G8 Inline result previews (expandable entity cards) AdvisoryAI + FE
109 G9 Multilingual search (de/fr/es/ru FTS, language detection, localized doctor seeds) AdvisoryAI + FE
110 G10 Search feedback loop (thumbs up/down, quality dashboard, query refinements) AdvisoryAI + FE

Test projects and files

All backend tests live in a single test project:

src/AdvisoryAI/__Tests/StellaOps.AdvisoryAI.Tests/StellaOps.AdvisoryAI.Tests.csproj

Key test files added by the search sprints:

File Coverage Type
Integration/UnifiedSearchSprintIntegrationTests.cs All 10 sprints (87 tests) — endpoint auth, domain filtering, synthesis, suggestions, role-based bias, multilingual detection, feedback validation Integration (WebApplicationFactory)
Integration/KnowledgeSearchEndpointsIntegrationTests.cs AKS endpoints: auth, search, localization, rebuild Integration (WebApplicationFactory)
KnowledgeSearch/FtsRecallBenchmarkTests.cs G5-005: FTS recall benchmark (12 tests, 34-query fixture) Benchmark
KnowledgeSearch/FtsRecallBenchmarkStore.cs In-memory FTS store simulating Simple vs English modes Test harness
KnowledgeSearch/SemanticRecallBenchmarkTests.cs G1-004: Semantic recall benchmark (13 tests, 48-query fixture) Benchmark
KnowledgeSearch/SemanticRecallBenchmarkStore.cs In-memory vector store with cosine similarity search Test harness
UnifiedSearch/UnifiedSearchServiceTests.cs G8: Preview generation (7 tests) Unit

Test data fixtures (auto-copied to output via TestData/*.json glob in .csproj):

  • TestData/fts-recall-benchmark.json — 34 queries across exact/stemming/typos/short/natural categories
  • TestData/semantic-recall-benchmark.json — 48 queries across synonym/paraphrase/conceptual/acronym/exact categories

Prerequisites to run

Detailed infrastructure setup guide: src/AdvisoryAI/__Tests/INFRASTRUCTURE.md — covers 4 tiers (in-process, live database, ONNX model, frontend E2E) with exact Docker commands, connection strings, extension requirements, and config examples.

No external infrastructure needed for the in-process test suite. All integration tests use WebApplicationFactory<Program> with stubbed services. Benchmarks use in-memory stores. No PostgreSQL, no Docker, no network access required.

Run the full suite:

dotnet test "src/AdvisoryAI/__Tests/StellaOps.AdvisoryAI.Tests/StellaOps.AdvisoryAI.Tests.csproj" -v normal

Targeted runs in this project use xUnit v3 / Microsoft.Testing.Platform. Do not use VSTest dotnet test --filter ... syntax here; use xUnit pass-through or the built test executable instead.

Run only the search sprint integration tests:

dotnet test "src/AdvisoryAI/__Tests/StellaOps.AdvisoryAI.Tests/StellaOps.AdvisoryAI.Tests.csproj" \
  -- --filter-class StellaOps.AdvisoryAI.Tests.Integration.UnifiedSearchSprintIntegrationTests

Run only the FTS recall benchmark:

dotnet test "src/AdvisoryAI/__Tests/StellaOps.AdvisoryAI.Tests/StellaOps.AdvisoryAI.Tests.csproj" \
  -- --filter-class StellaOps.AdvisoryAI.Tests.KnowledgeSearch.FtsRecallBenchmarkTests

Run only the semantic recall benchmark:

dotnet test "src/AdvisoryAI/__Tests/StellaOps.AdvisoryAI.Tests/StellaOps.AdvisoryAI.Tests.csproj" \
  -- --filter-class StellaOps.AdvisoryAI.Tests.KnowledgeSearch.SemanticRecallBenchmarkTests

For live database tests (e.g., full AKS rebuild + query against real Postgres with pg_trgm/pgvector):

# Start the dedicated AKS test database
docker compose -f devops/compose/docker-compose.advisoryai-knowledge-test.yml up -d

# Wait for health check
docker compose -f devops/compose/docker-compose.advisoryai-knowledge-test.yml ps

# Start the local AdvisoryAI service against that database
export AdvisoryAI__KnowledgeSearch__ConnectionString="Host=localhost;Port=55432;Database=advisoryai_knowledge_test;Username=stellaops_knowledge;Password=stellaops_knowledge"
dotnet run --project "src/AdvisoryAI/StellaOps.AdvisoryAI.WebService/StellaOps.AdvisoryAI.WebService.csproj" --no-launch-profile

# In a second shell, rebuild the live corpus in the required order
export STELLAOPS_BACKEND_URL="http://127.0.0.1:10451"
dotnet run --project "src/Cli/StellaOps.Cli/StellaOps.Cli.csproj" -- advisoryai sources prepare --json
curl -X POST http://127.0.0.1:10451/v1/advisory-ai/index/rebuild \
  -H "X-StellaOps-Scopes: advisory-ai:admin" \
  -H "X-StellaOps-Tenant: test-tenant" \
  -H "X-StellaOps-Actor: local-search-test"
curl -X POST http://127.0.0.1:10451/v1/search/index/rebuild \
  -H "X-StellaOps-Scopes: advisory-ai:admin" \
  -H "X-StellaOps-Tenant: test-tenant" \
  -H "X-StellaOps-Actor: local-search-test"

# Run tests with the Live category (requires database)
dotnet build "src/AdvisoryAI/__Tests/StellaOps.AdvisoryAI.Tests/StellaOps.AdvisoryAI.Tests.csproj" -v minimal
src/AdvisoryAI/__Tests/StellaOps.AdvisoryAI.Tests/bin/Debug/net10.0/StellaOps.AdvisoryAI.Tests.exe \
  -trait "Category=Live" -reporter verbose -noColor

Notes:

  • AdvisoryAI knowledge ingestion now auto-detects the repository root from the current working directory and AppContext.BaseDirectory when it is running inside a normal Stella Ops source checkout.
  • Set AdvisoryAI__KnowledgeSearch__RepositoryRoot only when you are running the service from a non-standard layout or a packaged binary tree that is not inside the repository.
  • stella advisoryai index rebuild and stella search index rebuild invoke authenticated backend endpoints. For a local source-checkout verification lane without a signed-in CLI session, use sources prepare via CLI and the direct HTTP rebuild calls above with explicit X-StellaOps-* headers.
  • Compose/runtime requirement: the published AdvisoryAI service image must carry a repo-shaped local corpus under its app content root so POST /v1/advisory-ai/index/rebuild can resolve docs/**, devops/compose/openapi_current.json, and src/AdvisoryAI/StellaOps.AdvisoryAI/KnowledgeSearch/*.json even when the source checkout is not mounted into the container. If those assets are absent, live search on stella-ops.local degrades to partial unified rows only and documentation/Doctor/API answers disappear.
  • Fresh service startup now auto-runs the knowledge rebuild by default (AdvisoryAI__KnowledgeSearch__KnowledgeAutoIndexOnStartup=true). This is the scratch-setup convergence path for stella-ops.local: a wiped deployment must populate the documentation/API/Doctor corpus without requiring operators to call POST /v1/advisory-ai/index/rebuild manually. Keep the manual endpoint for explicit refreshes and local live-search lanes, but do not depend on it for first-run correctness.
  • Startup schema bootstrap is protected by a PostgreSQL advisory transaction lock. AdvisoryAI cold start can trigger both the knowledge rebuild host and unified-search refresh paths against the same store, so EnsureSchemaAsync() must serialize CREATE SCHEMA and migration application instead of relying on IF NOT EXISTS alone.
  • The published app content root must also carry the full unified snapshot corpus under src/AdvisoryAI/StellaOps.AdvisoryAI/UnifiedSearch/Snapshots/*.json; packaging only findings/VEX/policy snapshots leaves graph, OpsMemory, timeline, and scanner answer lanes permanently corpus-unready in the live shell.

CLI setup in a source checkout

Do not assume stella is already installed on the machine running local AdvisoryAI work. From a repository checkout, either run the CLI through dotnet run or publish a local binary first.

Quick references:

  • Build/install note and command reference: docs/API_CLI_REFERENCE.md
  • CLI quickstart: docs/modules/cli/guides/quickstart.md

Local examples:

# Run directly from source without installing to PATH
export STELLAOPS_BACKEND_URL="http://127.0.0.1:10451"
dotnet run --project "src/Cli/StellaOps.Cli/StellaOps.Cli.csproj" -- advisoryai sources prepare --json

# Publish a reusable local binary
dotnet publish "src/Cli/StellaOps.Cli/StellaOps.Cli.csproj" -c Release -o ".artifacts/stella-cli"

# Windows
.artifacts/stella-cli/StellaOps.Cli.exe advisoryai index rebuild --json

# Linux/macOS
./.artifacts/stella-cli/StellaOps.Cli advisoryai index rebuild --json

If the CLI is not built yet, the equivalent HTTP endpoints are:

  • POST /v1/advisory-ai/index/rebuild for the docs/OpenAPI/Doctor corpus
  • POST /v1/search/index/rebuild for unified overlay domains

Standard browser acceptance lane for live search:

cd src/Web/StellaOps.Web
npm run test:e2e:search:live

This command is now the standard local search-browser setup because it:

  • starts or reuses the dedicated AdvisoryAI knowledge-test Postgres
  • starts or reuses the source-run AdvisoryAI WebService on http://127.0.0.1:10451
  • publishes the local CLI when .artifacts/stella-cli/ is missing
  • runs advisoryai sources prepare --json
  • rebuilds the knowledge and unified search indexes in the required order
  • proves a grounded smoke query before the browser tests run

Current live verification coverage:

  • Rebuild order exercised against a running local service: POST /v1/advisory-ai/index/rebuild then POST /v1/search/index/rebuild, both with explicit X-StellaOps-Scopes, X-StellaOps-Tenant, and X-StellaOps-Actor headers
  • Verified live query: database connectivity
  • Verified live outcome: response includes contextAnswer.status = grounded, citations, and entity cards over ingested data
  • Verified live suggestion lane: src/Web/StellaOps.Web/tests/e2e/unified-search-contextual-suggestions.live.e2e.spec.ts now runs through the standard Playwright setup lane, preflights corpus readiness, validates suggestion viability, executes every surfaced Doctor suggestion, asserts grounded answer states for surfaced live suggestions, verifies follow-up chips after result open, verifies grounded-search history/clear behavior, and verifies Ask-AdvisoryAI inherits the live query context
  • Standard Playwright setup/config entrypoints: src/Web/StellaOps.Web/scripts/run-live-search-e2e.mjs, src/Web/StellaOps.Web/playwright.live-search.config.ts, and src/Web/StellaOps.Web/tests/e2e/live-search.setup.ts
  • Verified combined browser gate on 2026-03-08: 24/24 executed tests passed with 3 explicit route-unready skips across deterministic UX, telemetry-off search flows, self-serve answer panel, and the supported-route live suggestion lane against the ingested local corpus
  • Verified local corpus baseline on 2026-03-07 after advisoryai sources prepare: documentCount = 470, chunkCount = 9050, apiOperationCount = 2190, doctorProjectionCount = 8
  • Other routes still rely on deterministic mock-backed Playwright coverage until their ingestion parity is explicitly verified

Or use the full CI testing stack:

docker compose -f devops/compose/docker-compose.testing.yml --profile ci up -d

Database extensions required for live tests

The AKS knowledge test database init script (devops/compose/postgres-init/advisoryai-knowledge-test/01_extensions.sql) must enable:

  • vector (pgvector) — for embedding_vec vector(384) columns and cosine similarity
  • pg_trgm — for trigram fuzzy matching (similarity(), GIN trigram indexes)

These are already configured in the compose init scripts. If setting up a custom test database:

CREATE EXTENSION IF NOT EXISTS vector;
CREATE EXTENSION IF NOT EXISTS pg_trgm;

Migrations required for search sprints

The search sprints added several migrations under src/AdvisoryAI/StellaOps.AdvisoryAI/Storage/Migrations/:

Migration Sprint Content
004_fts_english_config.sql G5 (101) body_tsv_en tsvector column + GIN index, pg_trgm extension + trigram indexes
005_search_feedback.sql G10 (110) search_feedback + search_quality_alerts tables
005_search_analytics.sql G6 (106) search_events + search_history tables
007_multilingual_fts.sql G9 (109) body_tsv_de, body_tsv_fr, body_tsv_es, body_tsv_ru tsvector columns + GIN indexes
008_search_self_serve_analytics.sql AI-SELF-004 session_id, answer_status, answer_code analytics columns plus self-serve indexes

All migrations are idempotent (IF NOT EXISTS guards). They run automatically via EnsureSchemaAsync() at service startup.

Frontend tests

Frontend changes span src/Web/StellaOps.Web/. To run Angular unit tests:

cd src/Web/StellaOps.Web
npm install
npm run test:ci

For E2E tests (requires the full stack running):

cd src/Web/StellaOps.Web
npx playwright install
npm run test:e2e

Relevant E2E config: src/Web/StellaOps.Web/playwright.e2e.config.ts.

InternalsVisibleTo

The production assembly StellaOps.AdvisoryAI grants InternalsVisibleTo to StellaOps.AdvisoryAI.Tests (see src/AdvisoryAI/StellaOps.AdvisoryAI/Properties/AssemblyInfo.cs). This allows tests to access internal types including:

  • IVectorEncoder, DeterministicHashVectorEncoder, OnnxVectorEncoder
  • ISynthesisEngine, SynthesisTemplateEngine, CompositeSynthesisEngine, LlmSynthesisEngine
  • IntentClassifier, QueryLanguageDetector, MultilingualIntentKeywords, DomainWeightCalculator
  • SearchAnalyticsService, SearchQualityMonitor
  • WeightedRrfFusion, UnifiedSearchService
  • IKnowledgeSearchStore, KnowledgeChunkRow

Key interfaces to stub in integration tests

Interface Purpose Typical stub behavior
IKnowledgeSearchService AKS search Return hardcoded results per query
IKnowledgeIndexer AKS index rebuild Return fixed summary counts
IUnifiedSearchService Unified search Return entity cards with domain filtering
IUnifiedSearchIndexer Unified index rebuild Return fixed summary
ISynthesisEngine AI synthesis Return template-based synthesis
IVectorEncoder Embedding generation Use DeterministicHashVectorEncoder or EmptyVectorEncoder
IKnowledgeSearchStore FTS/vector storage Use DeterministicBenchmarkStore or FtsRecallBenchmarkStore

Test categories and filtering

Use [Trait("Category", TestCategories.XXX)] to categorize tests. Key categories:

  • Unit — fast, in-memory, no external deps (default for most tests)
  • Integration — uses WebApplicationFactory or test containers
  • Performance — benchmarks (FTS recall, semantic recall)
  • Live — requires running database (skip in standard CI)

Filter examples:

# All except Live
dotnet test ... --filter "Category!=Live"

# Only integration
dotnet test ... --filter "Category=Integration"

# Specific test class
dotnet test ... --filter "FullyQualifiedName~FtsRecallBenchmarkTests"

Localized doctor seeds

Doctor check content is available in 3 locales:

  • doctor-search-seed.json — English (base, 8 checks)
  • doctor-search-seed.de.json — German (de-DE)
  • doctor-search-seed.fr.json — French (fr-FR)

The KnowledgeIndexer.IngestDoctorAsync() method auto-discovers locale files via DoctorSearchSeedLoader.LoadLocalized() and ingests locale-tagged chunks alongside English. This enables German/French FTS queries to match doctor check content.

Configuration options added by search sprints

All in KnowledgeSearchOptions (src/AdvisoryAI/StellaOps.AdvisoryAI/KnowledgeSearch/KnowledgeSearchOptions.cs):

Option Default Sprint Purpose
FtsLanguageConfig "english" G5 Primary FTS text search config
FuzzyFallbackEnabled true G5 Enable pg_trgm fuzzy fallback
MinFtsResultsForFuzzyFallback 3 G5 Threshold for fuzzy activation
FuzzySimilarityThreshold 0.3 G5 pg_trgm similarity cutoff
VectorEncoderType "hash" G1 "hash" or "onnx"
OnnxModelPath "models/all-MiniLM-L6-v2.onnx" G1 Path to ONNX model file
LlmSynthesisEnabled false G3 Enable LLM-grounded synthesis
SynthesisTimeoutMs 5000 G3 LLM synthesis timeout
LlmAdapterBaseUrl null G3 LLM adapter service URL
LlmProviderId null G3 LLM provider selection
PopularityBoostEnabled false G6 Enable click-weighted ranking
PopularityBoostWeight 0.05 G6 Popularity boost factor
RoleBasedBiasEnabled true G6 Enable scope-based domain weighting
SearchQualityMonitorEnabled true G10 Enable periodic quality-alert refresh
SearchQualityMonitorIntervalSeconds 300 G10 Quality-alert refresh cadence
SearchAnalyticsRetentionEnabled true G10 Enable automatic analytics/feedback/history pruning
SearchAnalyticsRetentionDays 90 G10 Retention window for search analytics artifacts
SearchAnalyticsRetentionIntervalSeconds 3600 G10 Retention pruning cadence
FtsLanguageConfigs {} G9 Per-locale FTS config map

Unified-search options (UnifiedSearchOptions, src/AdvisoryAI/StellaOps.AdvisoryAI/UnifiedSearch/UnifiedSearchOptions.cs):

  • Enabled
  • BaseDomainWeights
  • Weighting.*
  • Federation.*
  • GravityBoost.*
  • Synthesis.*
  • Ingestion.*
  • Session.*
  • TenantFeatureFlags.<tenant>.Enabled
  • TenantFeatureFlags.<tenant>.FederationEnabled
  • TenantFeatureFlags.<tenant>.SynthesisEnabled

Known limitations and follow-ups

  • YAML OpenAPI ingestion is not included in MVP.
  • End-to-end benchmark against live Postgres-backed AKS service is planned as a follow-up CI lane.
  • Optional external embedding providers can be added later without changing API contracts.
  • ONNX model file (all-MiniLM-L6-v2.onnx, ~80MB) must be provisioned separately for deployments opting into VectorEncoderType=onnx. Air-gap bundles must include the model.
  • Doctor seed localization covers de-DE and fr-FR only. Other locales (es-ES, ru-RU, bg-BG, etc.) use English fallback.
  • Search quality dashboard deferred items: low-quality results table, top queries table, 30-day trend chart (require additional backend aggregation queries).