Standardize live search Playwright setup lane
This commit is contained in:
@@ -0,0 +1,82 @@
|
||||
# Sprint 20260308-009 - FE Standard Live Search Playwright Setup
|
||||
|
||||
## Topic & Scope
|
||||
- Turn the ad hoc live-search ingestion/browser flow into a standard one-command Playwright acceptance lane for search.
|
||||
- Make the lane self-preparing where feasible: start the dedicated AdvisoryAI knowledge test database, start the local AdvisoryAI service, prepare sources, rebuild indexes, and gate the browser suite on a grounded smoke query.
|
||||
- Keep the implementation centered on `src/Web/StellaOps.Web` while allowing the minimum supporting doc updates in AdvisoryAI test/setup guidance.
|
||||
- Working directory: `src/Web/StellaOps.Web`.
|
||||
- Expected evidence: setup script/config, live setup artifact, Playwright live browser results, and updated operator docs.
|
||||
|
||||
## Dependencies & Concurrency
|
||||
- Depends on archived search rollout/correction sprints plus archived `SPRINT_20260308_008_FE_live_search_ingestion_browser_validation.md`.
|
||||
- Safe parallelism: do not edit unrelated Router, platform-setup, or shell-cutover files while executing this lane.
|
||||
- Cross-module allowance: documentation updates are allowed in `docs/modules/advisory-ai/**` and `src/AdvisoryAI/__Tests/INFRASTRUCTURE.md` because they define the live-search setup contract consumed by this FE lane.
|
||||
|
||||
## Documentation Prerequisites
|
||||
- `docs/qa/feature-checks/FLOW.md`
|
||||
- `docs/code-of-conduct/TESTING_PRACTICES.md`
|
||||
- `src/Web/StellaOps.Web/AGENTS.md`
|
||||
- `src/AdvisoryAI/__Tests/INFRASTRUCTURE.md`
|
||||
- `docs/modules/advisory-ai/knowledge-search.md`
|
||||
|
||||
## Delivery Tracker
|
||||
|
||||
### FE-LIVESETUP-001 - Standardize the Playwright live-search setup lane
|
||||
Status: DONE
|
||||
Dependency: none
|
||||
Owners: QA, Test Automation, Developer (FE)
|
||||
Task description:
|
||||
- Add a standard Playwright entrypoint for live search acceptance that boots the dedicated knowledge-test database, starts the source-run AdvisoryAI service, and runs a setup project before browser tests execute.
|
||||
- The setup project must prepare the ingestion sources, rebuild both search indexes, and fail fast when the live service is unhealthy or the grounded smoke query does not work.
|
||||
|
||||
Completion criteria:
|
||||
- [x] A documented one-command live-search Playwright lane exists under `src/Web/StellaOps.Web`.
|
||||
- [x] The lane does not assume `stella` is already installed; it compiles or publishes the CLI when needed.
|
||||
- [x] Live setup artifacts record rebuild/query evidence for operator diagnosis.
|
||||
|
||||
### FE-LIVESETUP-002 - Expand live browser verification on the standardized lane
|
||||
Status: DONE
|
||||
Dependency: FE-LIVESETUP-001
|
||||
Owners: QA, Test Automation
|
||||
Task description:
|
||||
- Run the live browser suite through the new standard lane and add at least one operator-relevant assertion beyond bare suggestion execution.
|
||||
- The added assertion should cover a real UX contract from the shipped search experience, not just service availability.
|
||||
|
||||
Completion criteria:
|
||||
- [x] The standardized live lane executes the existing live search browser suite successfully.
|
||||
- [x] The live suite verifies at least one user-facing persistence/continuity behavior on real ingested data.
|
||||
- [x] Test evidence is captured with exact command and result counts.
|
||||
|
||||
### FE-LIVESETUP-003 - Update docs and close the lane
|
||||
Status: DONE
|
||||
Dependency: FE-LIVESETUP-002
|
||||
Owners: Documentation, Project Manager
|
||||
Task description:
|
||||
- Update Web and AdvisoryAI setup docs to point at the new standard live-search acceptance lane.
|
||||
- Archive this sprint when the setup lane, verification, and docs are complete.
|
||||
|
||||
Completion criteria:
|
||||
- [x] Web README documents the standard live-search Playwright command.
|
||||
- [x] AdvisoryAI infrastructure/search docs document the setup contract and prerequisites.
|
||||
- [x] Sprint execution log captures implementation and validation evidence before archival.
|
||||
|
||||
## Execution Log
|
||||
| Date (UTC) | Update | Owner |
|
||||
| --- | --- | --- |
|
||||
| 2026-03-08 | Sprint created to replace manual live-search ingestion/browser steps with a standard Playwright setup lane. | Developer |
|
||||
| 2026-03-08 | Added the standard lane under `src/Web/StellaOps.Web` with `scripts/run-live-search-e2e.mjs`, `playwright.live-search.config.ts`, and a `live-search-setup` Playwright project that compiles the CLI if needed, runs `sources prepare`, rebuilds both indexes, and writes `output/playwright/live-search-setup.json`. | Developer |
|
||||
| 2026-03-08 | Extended the live browser suite so the standardized lane also verifies user-facing continuity: a grounded live search persists in Recent and can be removed through the clear-history icon. | QA |
|
||||
| 2026-03-08 | Validation: `npm run test:e2e:search:live` passed with `10 passed`, `3 skipped` in ~2.6 minutes; skips were the explicit corpus-unready branches for routes that are now ready enough to bypass those tests. | QA |
|
||||
|
||||
## Decisions & Risks
|
||||
- Decision: standardize the search live lane as a dedicated acceptance path instead of forcing ingestion into every default FE test run.
|
||||
- Decision: use a wrapper script plus a dedicated Playwright config so the live lane can bring up the dedicated knowledge-test database before Playwright starts the source-run AdvisoryAI service and browser projects.
|
||||
- Decision: the setup project writes a JSON artifact under `output/playwright/live-search-setup.json` so failures can be diagnosed from corpus preparation and rebuild evidence, not only browser traces.
|
||||
- Risk: the setup lane depends on Docker, dotnet, Node/npm, and the dedicated AdvisoryAI knowledge test database compose file.
|
||||
- Mitigation: fail early with explicit setup diagnostics and keep the lane isolated from the fast mocked/offline FE loop.
|
||||
- Risk: CLI `--json` output can be prefixed by infrastructure logs on some hosts.
|
||||
- Mitigation: the setup parser now tolerates log-prefixed JSON and still fails hard when no valid payload is present.
|
||||
|
||||
## Next Checkpoints
|
||||
- 2026-03-08: setup script/config implemented and locally executable.
|
||||
- 2026-03-08: live browser suite rerun through the standard lane with recorded evidence.
|
||||
@@ -413,11 +413,27 @@ If the CLI is not built yet, the equivalent HTTP endpoints are:
|
||||
- `POST /v1/advisory-ai/index/rebuild` for the docs/OpenAPI/Doctor corpus
|
||||
- `POST /v1/search/index/rebuild` for unified overlay domains
|
||||
|
||||
Standard browser acceptance lane for live search:
|
||||
|
||||
```bash
|
||||
cd src/Web/StellaOps.Web
|
||||
npm run test:e2e:search:live
|
||||
```
|
||||
|
||||
This command is now the standard local search-browser setup because it:
|
||||
- starts or reuses the dedicated AdvisoryAI knowledge-test Postgres
|
||||
- starts or reuses the source-run AdvisoryAI WebService on `http://127.0.0.1:10451`
|
||||
- publishes the local CLI when `.artifacts/stella-cli/` is missing
|
||||
- runs `advisoryai sources prepare --json`
|
||||
- rebuilds the knowledge and unified search indexes in the required order
|
||||
- proves a grounded smoke query before the browser tests run
|
||||
|
||||
Current live verification coverage:
|
||||
- Rebuild order exercised against a running local service: `POST /v1/advisory-ai/index/rebuild` then `POST /v1/search/index/rebuild`, both with explicit `X-StellaOps-Scopes`, `X-StellaOps-Tenant`, and `X-StellaOps-Actor` headers
|
||||
- Verified live query: `database connectivity`
|
||||
- Verified live outcome: response includes `contextAnswer.status = grounded`, citations, and entity cards over ingested data
|
||||
- Verified live suggestion lane: `src/Web/StellaOps.Web/tests/e2e/unified-search-contextual-suggestions.live.e2e.spec.ts` now preflights corpus readiness, validates suggestion viability, executes every surfaced Doctor suggestion, asserts grounded answer states for surfaced live suggestions, verifies follow-up chips after result open, and verifies Ask-AdvisoryAI inherits the live query context
|
||||
- Verified live suggestion lane: `src/Web/StellaOps.Web/tests/e2e/unified-search-contextual-suggestions.live.e2e.spec.ts` now runs through the standard Playwright setup lane, preflights corpus readiness, validates suggestion viability, executes every surfaced Doctor suggestion, asserts grounded answer states for surfaced live suggestions, verifies follow-up chips after result open, verifies grounded-search history/clear behavior, and verifies Ask-AdvisoryAI inherits the live query context
|
||||
- Standard Playwright setup/config entrypoints: `src/Web/StellaOps.Web/scripts/run-live-search-e2e.mjs`, `src/Web/StellaOps.Web/playwright.live-search.config.ts`, and `src/Web/StellaOps.Web/tests/e2e/live-search.setup.ts`
|
||||
- Verified combined browser gate on 2026-03-08: `24/24` executed tests passed with `3` explicit route-unready skips across deterministic UX, telemetry-off search flows, self-serve answer panel, and the supported-route live suggestion lane against the ingested local corpus
|
||||
- Verified local corpus baseline on 2026-03-07 after `advisoryai sources prepare`: `documentCount = 470`, `chunkCount = 9050`, `apiOperationCount = 2190`, `doctorProjectionCount = 8`
|
||||
- Other routes still rely on deterministic mock-backed Playwright coverage until their ingestion parity is explicitly verified
|
||||
|
||||
@@ -334,6 +334,29 @@ E2E config: `src/Web/StellaOps.Web/playwright.e2e.config.ts`
|
||||
- Timeout: 60s per test
|
||||
- Workers: 1 (sequential)
|
||||
|
||||
### Standard live-search browser lane
|
||||
|
||||
For search verification against a freshly ingested local AdvisoryAI corpus, use the dedicated Web acceptance lane instead of manually sequencing database startup, CLI prepare, and rebuild calls:
|
||||
|
||||
```bash
|
||||
cd src/Web/StellaOps.Web
|
||||
npm run test:e2e:search:live
|
||||
```
|
||||
|
||||
This lane:
|
||||
- starts or reuses `devops/compose/docker-compose.advisoryai-knowledge-test.yml`
|
||||
- starts or reuses the source-run AdvisoryAI WebService on `http://127.0.0.1:10451`
|
||||
- publishes the local CLI to `.artifacts/stella-cli/` when needed
|
||||
- runs `advisoryai sources prepare --json`
|
||||
- rebuilds `POST /v1/advisory-ai/index/rebuild` then `POST /v1/search/index/rebuild`
|
||||
- proves the grounded smoke query `database connectivity`
|
||||
- runs the live browser suite through `src/Web/StellaOps.Web/playwright.live-search.config.ts`
|
||||
|
||||
Useful overrides:
|
||||
- `PLAYWRIGHT_LIVE_SEARCH_SKIP_DOCKER=1`
|
||||
- `LIVE_ADVISORYAI_SEARCH_BASE_URL=http://127.0.0.1:10451`
|
||||
- `AdvisoryAI__KnowledgeSearch__ConnectionString=...`
|
||||
|
||||
---
|
||||
|
||||
## Configuration reference
|
||||
|
||||
@@ -70,6 +70,33 @@ npm run test:e2e
|
||||
```
|
||||
|
||||
The Playwright config auto-starts `npm run serve:test` and intercepts Authority redirects, so no live IdP is required. For CI/offline nodes, pre-install the required browsers via `npx playwright install --with-deps` and cache the results alongside your npm cache.
|
||||
|
||||
### Standard live-search acceptance lane
|
||||
|
||||
Use this lane when you need browser verification against a freshly ingested local AdvisoryAI corpus instead of mocked search responses:
|
||||
|
||||
```bash
|
||||
npm run test:e2e:search:live
|
||||
```
|
||||
|
||||
What it does:
|
||||
- starts or reuses the dedicated AdvisoryAI knowledge-test Postgres via `devops/compose/docker-compose.advisoryai-knowledge-test.yml`
|
||||
- starts or reuses the local AdvisoryAI WebService on `http://127.0.0.1:10451`
|
||||
- compiles the local CLI if `.artifacts/stella-cli/` is missing
|
||||
- runs `advisoryai sources prepare --json`
|
||||
- rebuilds `POST /v1/advisory-ai/index/rebuild` and `POST /v1/search/index/rebuild`
|
||||
- proves a grounded smoke query before the browser suite runs
|
||||
- executes the live search Playwright spec through `playwright.live-search.config.ts`
|
||||
|
||||
Prerequisites:
|
||||
- `docker compose`
|
||||
- `dotnet`
|
||||
- Playwright browsers installed locally
|
||||
|
||||
Optional environment overrides:
|
||||
- `PLAYWRIGHT_LIVE_SEARCH_SKIP_DOCKER=1` if the dedicated AdvisoryAI test database is already running
|
||||
- `LIVE_ADVISORYAI_SEARCH_BASE_URL` to target a non-default local AdvisoryAI host
|
||||
- `AdvisoryAI__KnowledgeSearch__ConnectionString` to override the dedicated test database connection string
|
||||
|
||||
## Running end-to-end tests
|
||||
|
||||
|
||||
@@ -13,6 +13,7 @@
|
||||
"test:watch": "ng test",
|
||||
"test:ci": "npm run test",
|
||||
"test:e2e": "playwright test",
|
||||
"test:e2e:search:live": "node ./scripts/run-live-search-e2e.mjs",
|
||||
"serve:test": "ng serve --configuration development --port 4400 --host 127.0.0.1 --ssl",
|
||||
"verify:chromium": "node ./scripts/verify-chromium.js",
|
||||
"ci:install": "npm ci --prefer-offline --no-audit --no-fund",
|
||||
|
||||
81
src/Web/StellaOps.Web/playwright.live-search.config.ts
Normal file
81
src/Web/StellaOps.Web/playwright.live-search.config.ts
Normal file
@@ -0,0 +1,81 @@
|
||||
import path from 'node:path';
|
||||
|
||||
import { defineConfig, devices } from '@playwright/test';
|
||||
|
||||
// eslint-disable-next-line @typescript-eslint/no-var-requires
|
||||
const { resolveChromeBinary } = require('./scripts/chrome-path');
|
||||
|
||||
const webRoot = __dirname;
|
||||
const repoRoot = path.resolve(__dirname, '..', '..', '..');
|
||||
const appPort = process.env.PLAYWRIGHT_PORT
|
||||
? Number.parseInt(process.env.PLAYWRIGHT_PORT, 10)
|
||||
: 4400;
|
||||
const appBaseUrl = process.env.PLAYWRIGHT_BASE_URL ?? `https://127.0.0.1:${appPort}`;
|
||||
const liveSearchBaseUrl = process.env.LIVE_ADVISORYAI_SEARCH_BASE_URL?.trim()
|
||||
|| 'http://127.0.0.1:10451';
|
||||
const advisoryConnectionString = process.env.AdvisoryAI__KnowledgeSearch__ConnectionString?.trim()
|
||||
|| 'Host=localhost;Port=55432;Database=advisoryai_knowledge_test;Username=stellaops_knowledge;Password=stellaops_knowledge';
|
||||
|
||||
process.env.LIVE_ADVISORYAI_SEARCH_BASE_URL = liveSearchBaseUrl;
|
||||
process.env.LIVE_ADVISORYAI_TENANT = process.env.LIVE_ADVISORYAI_TENANT?.trim() || 'test-tenant';
|
||||
process.env.LIVE_ADVISORYAI_SCOPES =
|
||||
process.env.LIVE_ADVISORYAI_SCOPES?.trim()
|
||||
|| 'advisory-ai:view advisory-ai:operate advisory-ai:admin';
|
||||
|
||||
const chromiumExecutable = resolveChromeBinary(__dirname) as string | null;
|
||||
|
||||
export default defineConfig({
|
||||
testDir: 'tests/e2e',
|
||||
timeout: 120_000,
|
||||
expect: { timeout: 10_000 },
|
||||
fullyParallel: false,
|
||||
retries: process.env.CI ? 1 : 0,
|
||||
workers: 1,
|
||||
reporter: [['list'], ['html', { open: 'never' }]],
|
||||
use: {
|
||||
baseURL: appBaseUrl,
|
||||
ignoreHTTPSErrors: true,
|
||||
trace: 'retain-on-failure',
|
||||
screenshot: 'only-on-failure',
|
||||
video: 'retain-on-failure',
|
||||
...(chromiumExecutable ? { launchOptions: { executablePath: chromiumExecutable } } : {}),
|
||||
},
|
||||
webServer: [
|
||||
{
|
||||
command: 'npm run serve:test',
|
||||
cwd: webRoot,
|
||||
reuseExistingServer: !process.env.CI,
|
||||
url: appBaseUrl,
|
||||
ignoreHTTPSErrors: true,
|
||||
stdout: 'ignore',
|
||||
stderr: 'ignore',
|
||||
timeout: 120_000,
|
||||
},
|
||||
{
|
||||
command: 'dotnet run --project src/AdvisoryAI/StellaOps.AdvisoryAI.WebService/StellaOps.AdvisoryAI.WebService.csproj --no-launch-profile',
|
||||
cwd: repoRoot,
|
||||
reuseExistingServer: !process.env.CI,
|
||||
url: `${liveSearchBaseUrl}/health`,
|
||||
stdout: 'ignore',
|
||||
stderr: 'ignore',
|
||||
timeout: 180_000,
|
||||
env: {
|
||||
...process.env,
|
||||
AdvisoryAI__KnowledgeSearch__ConnectionString: advisoryConnectionString,
|
||||
ASPNETCORE_URLS: liveSearchBaseUrl,
|
||||
},
|
||||
},
|
||||
],
|
||||
projects: [
|
||||
{
|
||||
name: 'live-search-setup',
|
||||
testMatch: /live-search\.setup\.ts/,
|
||||
},
|
||||
{
|
||||
name: 'live-search-chromium',
|
||||
testMatch: /unified-search-contextual-suggestions\.live\.e2e\.spec\.ts/,
|
||||
use: { ...devices['Desktop Chrome'] },
|
||||
dependencies: ['live-search-setup'],
|
||||
},
|
||||
],
|
||||
});
|
||||
108
src/Web/StellaOps.Web/scripts/run-live-search-e2e.mjs
Normal file
108
src/Web/StellaOps.Web/scripts/run-live-search-e2e.mjs
Normal file
@@ -0,0 +1,108 @@
|
||||
#!/usr/bin/env node
|
||||
|
||||
import net from 'node:net';
|
||||
import path from 'node:path';
|
||||
import { fileURLToPath } from 'node:url';
|
||||
import { spawnSync } from 'node:child_process';
|
||||
|
||||
const __filename = fileURLToPath(import.meta.url);
|
||||
const __dirname = path.dirname(__filename);
|
||||
const webRoot = path.resolve(__dirname, '..');
|
||||
const repoRoot = path.resolve(webRoot, '..', '..', '..');
|
||||
const composeFile = path.join(
|
||||
repoRoot,
|
||||
'devops',
|
||||
'compose',
|
||||
'docker-compose.advisoryai-knowledge-test.yml',
|
||||
);
|
||||
|
||||
async function main() {
|
||||
const forwardedArgs = process.argv.slice(2);
|
||||
const env = {
|
||||
...process.env,
|
||||
LIVE_ADVISORYAI_SEARCH_BASE_URL:
|
||||
process.env.LIVE_ADVISORYAI_SEARCH_BASE_URL?.trim() || 'http://127.0.0.1:10451',
|
||||
PLAYWRIGHT_LIVE_SEARCH_SKIP_REBUILD: '1',
|
||||
};
|
||||
|
||||
if (process.env.PLAYWRIGHT_LIVE_SEARCH_SKIP_DOCKER !== '1') {
|
||||
runCommand('docker', ['compose', '-f', composeFile, 'up', '-d'], repoRoot);
|
||||
await waitForTcpPort('127.0.0.1', 55432, 60_000);
|
||||
}
|
||||
|
||||
const playwrightArgs = ['playwright', 'test', '--config', 'playwright.live-search.config.ts', ...forwardedArgs];
|
||||
const result = process.platform === 'win32'
|
||||
? spawnSync('cmd.exe', ['/d', '/s', '/c', 'npx', ...playwrightArgs], {
|
||||
cwd: webRoot,
|
||||
env,
|
||||
stdio: 'inherit',
|
||||
})
|
||||
: spawnSync('npx', playwrightArgs, {
|
||||
cwd: webRoot,
|
||||
env,
|
||||
stdio: 'inherit',
|
||||
});
|
||||
|
||||
if (result.error) {
|
||||
throw result.error;
|
||||
}
|
||||
|
||||
process.exit(result.status ?? 1);
|
||||
}
|
||||
|
||||
function runCommand(command, args, cwd) {
|
||||
const result = spawnSync(command, args, {
|
||||
cwd,
|
||||
stdio: 'inherit',
|
||||
env: process.env,
|
||||
});
|
||||
|
||||
if (result.error) {
|
||||
throw result.error;
|
||||
}
|
||||
|
||||
if ((result.status ?? 1) !== 0) {
|
||||
throw new Error(`${command} ${args.join(' ')} exited with status ${result.status ?? 1}.`);
|
||||
}
|
||||
}
|
||||
|
||||
function waitForTcpPort(host, port, timeoutMs) {
|
||||
const startedAt = Date.now();
|
||||
|
||||
return new Promise((resolve, reject) => {
|
||||
const attempt = () => {
|
||||
const socket = new net.Socket();
|
||||
|
||||
socket.setTimeout(2_000);
|
||||
socket.once('connect', () => {
|
||||
socket.destroy();
|
||||
resolve();
|
||||
});
|
||||
socket.once('timeout', () => {
|
||||
socket.destroy();
|
||||
retry(new Error(`Timed out while connecting to ${host}:${port}.`));
|
||||
});
|
||||
socket.once('error', (error) => {
|
||||
socket.destroy();
|
||||
retry(error);
|
||||
});
|
||||
socket.connect(port, host);
|
||||
};
|
||||
|
||||
const retry = (lastError) => {
|
||||
if (Date.now() - startedAt >= timeoutMs) {
|
||||
reject(new Error(`Port ${host}:${port} did not become ready within ${timeoutMs}ms: ${lastError.message}`));
|
||||
return;
|
||||
}
|
||||
|
||||
setTimeout(attempt, 1_000);
|
||||
};
|
||||
|
||||
attempt();
|
||||
});
|
||||
}
|
||||
|
||||
main().catch((error) => {
|
||||
console.error(`[live-search-e2e] ${error instanceof Error ? error.message : String(error)}`);
|
||||
process.exit(1);
|
||||
});
|
||||
40
src/Web/StellaOps.Web/tests/e2e/live-search.setup.ts
Normal file
40
src/Web/StellaOps.Web/tests/e2e/live-search.setup.ts
Normal file
@@ -0,0 +1,40 @@
|
||||
import { expect, test } from '@playwright/test';
|
||||
|
||||
import {
|
||||
ensureLiveServiceHealthy,
|
||||
rebuildLiveIndexes,
|
||||
resolveLiveSearchRuntimeConfig,
|
||||
runGroundedSmokeQuery,
|
||||
runSourcesPrepare,
|
||||
writeLiveSearchSetupArtifact,
|
||||
} from './support/live-search-support';
|
||||
|
||||
test.describe('Live search setup', () => {
|
||||
test('prepares the ingested corpus and proves grounded retrieval before browser execution', async ({}, testInfo) => {
|
||||
testInfo.setTimeout(300_000);
|
||||
|
||||
const config = resolveLiveSearchRuntimeConfig();
|
||||
await ensureLiveServiceHealthy(config, 180_000);
|
||||
|
||||
const sourcesPrepare = runSourcesPrepare(config);
|
||||
const { knowledge, unified } = await rebuildLiveIndexes(config);
|
||||
const smokeQuery = await runGroundedSmokeQuery(config);
|
||||
const artifactPath = writeLiveSearchSetupArtifact(config, {
|
||||
preparedAtUtc: new Date().toISOString(),
|
||||
baseUrl: config.baseUrl,
|
||||
tenant: config.tenant,
|
||||
sourcesPrepare,
|
||||
knowledgeRebuild: knowledge,
|
||||
unifiedRebuild: unified,
|
||||
smokeQuery,
|
||||
});
|
||||
|
||||
expect((smokeQuery['contextAnswer'] as Record<string, unknown> | undefined)?.['status']).toBe('grounded');
|
||||
expect(Array.isArray(smokeQuery['cards']) ? smokeQuery['cards'].length : 0).toBeGreaterThan(0);
|
||||
|
||||
await testInfo.attach('live-search-setup.json', {
|
||||
path: artifactPath,
|
||||
contentType: 'application/json',
|
||||
});
|
||||
});
|
||||
});
|
||||
267
src/Web/StellaOps.Web/tests/e2e/support/live-search-support.ts
Normal file
267
src/Web/StellaOps.Web/tests/e2e/support/live-search-support.ts
Normal file
@@ -0,0 +1,267 @@
|
||||
import { execFileSync } from 'node:child_process';
|
||||
import { existsSync, mkdirSync, writeFileSync } from 'node:fs';
|
||||
import path from 'node:path';
|
||||
|
||||
export type LiveSearchRuntimeConfig = {
|
||||
baseUrl: string;
|
||||
tenant: string;
|
||||
scopes: string;
|
||||
actor: string;
|
||||
advisoryConnectionString: string;
|
||||
repoRoot: string;
|
||||
webRoot: string;
|
||||
};
|
||||
|
||||
export type LiveSearchSetupReport = {
|
||||
preparedAtUtc: string;
|
||||
baseUrl: string;
|
||||
tenant: string;
|
||||
sourcesPrepare: Record<string, unknown>;
|
||||
knowledgeRebuild: Record<string, unknown>;
|
||||
unifiedRebuild: Record<string, unknown>;
|
||||
smokeQuery: Record<string, unknown>;
|
||||
};
|
||||
|
||||
const DEFAULT_LIVE_SEARCH_BASE_URL = 'http://127.0.0.1:10451';
|
||||
const DEFAULT_LIVE_TENANT = 'test-tenant';
|
||||
const DEFAULT_LIVE_SCOPES = 'advisory-ai:view advisory-ai:operate advisory-ai:admin';
|
||||
const DEFAULT_LIVE_ACTOR = 'playwright-live-setup';
|
||||
const DEFAULT_ADVISORY_CONNSTRING =
|
||||
'Host=localhost;Port=55432;Database=advisoryai_knowledge_test;Username=stellaops_knowledge;Password=stellaops_knowledge';
|
||||
|
||||
export function resolveLiveSearchRuntimeConfig(): LiveSearchRuntimeConfig {
|
||||
const repoRoot = findRepoRoot(process.cwd());
|
||||
const webRoot = path.join(repoRoot, 'src', 'Web', 'StellaOps.Web');
|
||||
|
||||
return {
|
||||
baseUrl: process.env['LIVE_ADVISORYAI_SEARCH_BASE_URL']?.trim() || DEFAULT_LIVE_SEARCH_BASE_URL,
|
||||
tenant: process.env['LIVE_ADVISORYAI_TENANT']?.trim() || DEFAULT_LIVE_TENANT,
|
||||
scopes: process.env['LIVE_ADVISORYAI_SCOPES']?.trim() || DEFAULT_LIVE_SCOPES,
|
||||
actor: process.env['LIVE_ADVISORYAI_ACTOR']?.trim() || DEFAULT_LIVE_ACTOR,
|
||||
advisoryConnectionString:
|
||||
process.env['AdvisoryAI__KnowledgeSearch__ConnectionString']?.trim() || DEFAULT_ADVISORY_CONNSTRING,
|
||||
repoRoot,
|
||||
webRoot,
|
||||
};
|
||||
}
|
||||
|
||||
export async function ensureLiveServiceHealthy(
|
||||
config: LiveSearchRuntimeConfig,
|
||||
timeoutMs = 120_000,
|
||||
): Promise<void> {
|
||||
const start = Date.now();
|
||||
let lastError = 'service did not respond';
|
||||
|
||||
while (Date.now() - start < timeoutMs) {
|
||||
try {
|
||||
const response = await fetch(`${config.baseUrl}/health`);
|
||||
if (response.ok) {
|
||||
return;
|
||||
}
|
||||
|
||||
lastError = `status ${response.status}`;
|
||||
} catch (error) {
|
||||
lastError = error instanceof Error ? error.message : String(error);
|
||||
}
|
||||
|
||||
await delay(1_000);
|
||||
}
|
||||
|
||||
throw new Error(`Live AdvisoryAI health check failed at ${config.baseUrl}/health: ${lastError}`);
|
||||
}
|
||||
|
||||
export function ensureLocalCliPublished(config: LiveSearchRuntimeConfig): string {
|
||||
const publishDirectory = path.join(config.repoRoot, '.artifacts', 'stella-cli');
|
||||
const cliBinaryName = process.platform === 'win32' ? 'StellaOps.Cli.exe' : 'StellaOps.Cli';
|
||||
const cliBinaryPath = path.join(publishDirectory, cliBinaryName);
|
||||
|
||||
if (existsSync(cliBinaryPath)) {
|
||||
return cliBinaryPath;
|
||||
}
|
||||
|
||||
execFileSync(
|
||||
'dotnet',
|
||||
[
|
||||
'publish',
|
||||
path.join(config.repoRoot, 'src', 'Cli', 'StellaOps.Cli', 'StellaOps.Cli.csproj'),
|
||||
'-c',
|
||||
'Release',
|
||||
'-o',
|
||||
publishDirectory,
|
||||
],
|
||||
{
|
||||
cwd: config.repoRoot,
|
||||
env: process.env,
|
||||
stdio: 'pipe',
|
||||
},
|
||||
);
|
||||
|
||||
if (!existsSync(cliBinaryPath)) {
|
||||
throw new Error(`CLI publish completed without producing ${cliBinaryPath}.`);
|
||||
}
|
||||
|
||||
return cliBinaryPath;
|
||||
}
|
||||
|
||||
export function runSourcesPrepare(
|
||||
config: LiveSearchRuntimeConfig,
|
||||
cliBinaryPath = ensureLocalCliPublished(config),
|
||||
): Record<string, unknown> {
|
||||
const stdout = execFileSync(
|
||||
cliBinaryPath,
|
||||
['advisoryai', 'sources', 'prepare', '--json'],
|
||||
{
|
||||
cwd: config.repoRoot,
|
||||
env: {
|
||||
...process.env,
|
||||
STELLAOPS_BACKEND_URL: config.baseUrl,
|
||||
},
|
||||
stdio: 'pipe',
|
||||
encoding: 'utf8',
|
||||
},
|
||||
);
|
||||
|
||||
return parseJsonObject(stdout, 'sources prepare');
|
||||
}
|
||||
|
||||
export async function rebuildLiveIndexes(
|
||||
config: LiveSearchRuntimeConfig,
|
||||
): Promise<{ knowledge: Record<string, unknown>; unified: Record<string, unknown> }> {
|
||||
const adminHeaders = buildLiveHeaders(config, 'advisory-ai:admin');
|
||||
|
||||
const knowledge = await postJson(`${config.baseUrl}/v1/advisory-ai/index/rebuild`, adminHeaders);
|
||||
const unified = await postJson(`${config.baseUrl}/v1/search/index/rebuild`, adminHeaders);
|
||||
|
||||
return { knowledge, unified };
|
||||
}
|
||||
|
||||
export async function runGroundedSmokeQuery(
|
||||
config: LiveSearchRuntimeConfig,
|
||||
): Promise<Record<string, unknown>> {
|
||||
return postJson(
|
||||
`${config.baseUrl}/v1/search/query`,
|
||||
buildLiveHeaders(config),
|
||||
{
|
||||
q: 'database connectivity',
|
||||
k: 5,
|
||||
includeSynthesis: false,
|
||||
ambient: {
|
||||
currentRoute: '/ops/operations/doctor',
|
||||
},
|
||||
},
|
||||
);
|
||||
}
|
||||
|
||||
export function writeLiveSearchSetupArtifact(
|
||||
config: LiveSearchRuntimeConfig,
|
||||
report: LiveSearchSetupReport,
|
||||
): string {
|
||||
const outputDirectory = path.join(config.webRoot, 'output', 'playwright');
|
||||
const outputPath = path.join(outputDirectory, 'live-search-setup.json');
|
||||
mkdirSync(outputDirectory, { recursive: true });
|
||||
writeFileSync(outputPath, `${JSON.stringify(report, null, 2)}\n`, 'utf8');
|
||||
return outputPath;
|
||||
}
|
||||
|
||||
export function buildLiveHeaders(
|
||||
config: LiveSearchRuntimeConfig,
|
||||
scopes = config.scopes,
|
||||
): Record<string, string> {
|
||||
return {
|
||||
'content-type': 'application/json',
|
||||
'x-stellaops-scopes': scopes,
|
||||
'x-stellaops-tenant': config.tenant,
|
||||
'x-stellaops-actor': config.actor,
|
||||
};
|
||||
}
|
||||
|
||||
async function postJson(
|
||||
url: string,
|
||||
headers: Record<string, string>,
|
||||
body?: Record<string, unknown>,
|
||||
): Promise<Record<string, unknown>> {
|
||||
const response = await fetch(url, {
|
||||
method: 'POST',
|
||||
headers,
|
||||
body: body ? JSON.stringify(body) : undefined,
|
||||
});
|
||||
|
||||
const raw = await response.text();
|
||||
if (!response.ok) {
|
||||
throw new Error(`Request to ${url} failed with status ${response.status}: ${raw}`);
|
||||
}
|
||||
|
||||
return parseJsonObject(raw, url);
|
||||
}
|
||||
|
||||
function parseJsonObject(raw: string, source: string): Record<string, unknown> {
|
||||
const trimmed = raw.trim();
|
||||
if (!trimmed) {
|
||||
return {};
|
||||
}
|
||||
|
||||
try {
|
||||
const parsed = JSON.parse(trimmed) as Record<string, unknown>;
|
||||
return parsed && typeof parsed === 'object' ? parsed : {};
|
||||
} catch {
|
||||
const extracted = tryExtractEmbeddedJson(trimmed);
|
||||
if (extracted) {
|
||||
return extracted;
|
||||
}
|
||||
|
||||
throw new Error(`Expected JSON output from ${source}, received: ${trimmed.slice(0, 400)}`);
|
||||
}
|
||||
}
|
||||
|
||||
function tryExtractEmbeddedJson(raw: string): Record<string, unknown> | null {
|
||||
const firstBrace = raw.indexOf('{');
|
||||
const lastBrace = raw.lastIndexOf('}');
|
||||
if (firstBrace >= 0 && lastBrace > firstBrace) {
|
||||
const candidate = raw.slice(firstBrace, lastBrace + 1);
|
||||
try {
|
||||
const parsed = JSON.parse(candidate) as Record<string, unknown>;
|
||||
return parsed && typeof parsed === 'object' ? parsed : {};
|
||||
} catch {
|
||||
// Fall through to narrower candidates.
|
||||
}
|
||||
}
|
||||
|
||||
for (let index = raw.indexOf('{'); index >= 0; index = raw.indexOf('{', index + 1)) {
|
||||
try {
|
||||
const parsed = JSON.parse(raw.slice(index)) as Record<string, unknown>;
|
||||
return parsed && typeof parsed === 'object' ? parsed : {};
|
||||
} catch {
|
||||
// Keep scanning for the start of the JSON payload.
|
||||
}
|
||||
}
|
||||
|
||||
return null;
|
||||
}
|
||||
|
||||
function findRepoRoot(startDirectory: string): string {
|
||||
let current = path.resolve(startDirectory);
|
||||
|
||||
while (true) {
|
||||
if (isRepoRoot(current)) {
|
||||
return current;
|
||||
}
|
||||
|
||||
const parent = path.dirname(current);
|
||||
if (parent === current) {
|
||||
throw new Error(`Unable to locate the Stella Ops repository root from ${startDirectory}.`);
|
||||
}
|
||||
|
||||
current = parent;
|
||||
}
|
||||
}
|
||||
|
||||
function isRepoRoot(candidate: string): boolean {
|
||||
return existsSync(path.join(candidate, 'global.json'))
|
||||
&& existsSync(path.join(candidate, 'src'))
|
||||
&& existsSync(path.join(candidate, 'docs'))
|
||||
&& existsSync(path.join(candidate, 'devops'));
|
||||
}
|
||||
|
||||
function delay(ms: number): Promise<void> {
|
||||
return new Promise((resolve) => setTimeout(resolve, ms));
|
||||
}
|
||||
@@ -1,12 +1,18 @@
|
||||
import { expect, test, type Page, type Route } from '@playwright/test';
|
||||
|
||||
import { policyAuthorSession } from '../../src/app/testing';
|
||||
import {
|
||||
buildLiveHeaders,
|
||||
ensureLiveServiceHealthy,
|
||||
rebuildLiveIndexes,
|
||||
resolveLiveSearchRuntimeConfig,
|
||||
} from './support/live-search-support';
|
||||
import { waitForEntityCards, waitForResults } from './unified-search-fixtures';
|
||||
|
||||
const liveSearchBaseUrl = process.env['LIVE_ADVISORYAI_SEARCH_BASE_URL']?.trim() ?? '';
|
||||
const liveTenant = process.env['LIVE_ADVISORYAI_TENANT']?.trim() || 'test-tenant';
|
||||
const liveScopes = process.env['LIVE_ADVISORYAI_SCOPES']?.trim()
|
||||
|| 'advisory-ai:view advisory-ai:operate advisory-ai:admin';
|
||||
const liveRuntime = resolveLiveSearchRuntimeConfig();
|
||||
const liveSearchBaseUrl = liveRuntime.baseUrl;
|
||||
const liveTenant = liveRuntime.tenant;
|
||||
const liveScopes = liveRuntime.scopes;
|
||||
|
||||
const mockConfig = {
|
||||
authority: {
|
||||
@@ -162,8 +168,10 @@ test.describe('Unified Search - Live contextual suggestions', () => {
|
||||
|
||||
test.beforeAll(async ({}, testInfo) => {
|
||||
testInfo.setTimeout(120_000);
|
||||
await ensureLiveServiceHealthy(liveSearchBaseUrl);
|
||||
await rebuildLiveIndexes(liveSearchBaseUrl);
|
||||
await ensureLiveServiceHealthy(liveRuntime);
|
||||
if (process.env['PLAYWRIGHT_LIVE_SEARCH_SKIP_REBUILD'] !== '1') {
|
||||
await rebuildLiveIndexes(liveRuntime);
|
||||
}
|
||||
liveRouteStates.clear();
|
||||
|
||||
for (const routeConfig of liveRouteConfigs) {
|
||||
@@ -319,6 +327,34 @@ test.describe('Unified Search - Live contextual suggestions', () => {
|
||||
expect(String(capturedTurnBodies.at(-1)?.['content'] ?? '')).toMatch(/grounded answer|best next step/i);
|
||||
});
|
||||
|
||||
test('persists a grounded live search in recent history and clears it with the icon action', async ({ page }) => {
|
||||
await routeLiveUnifiedSearch(page);
|
||||
await openDoctor(page);
|
||||
|
||||
const searchInput = page.locator('app-global-search input[type="text"]');
|
||||
await searchInput.focus();
|
||||
await waitForResults(page);
|
||||
|
||||
await page.locator('.search__suggestions .search__chip', {
|
||||
hasText: /database connectivity/i,
|
||||
}).first().click();
|
||||
|
||||
await expect(searchInput).toHaveValue('database connectivity');
|
||||
await waitForResults(page);
|
||||
await expect(page.locator('[data-answer-status="grounded"]')).toBeVisible();
|
||||
|
||||
await searchInput.fill('');
|
||||
await waitForResults(page);
|
||||
|
||||
const recentGroup = page.locator('.search__group').filter({ hasText: 'Recent' });
|
||||
await expect(recentGroup).toContainText('database connectivity');
|
||||
|
||||
const clearButton = page.locator('.search__clear-history');
|
||||
await expect(clearButton).toBeVisible();
|
||||
await clearButton.click();
|
||||
await expect(recentGroup).toHaveCount(0);
|
||||
});
|
||||
|
||||
for (const routeConfig of liveRouteConfigs.filter((route) => route.key !== 'doctor')) {
|
||||
test(`${routeConfig.label} suppresses surfaced starter chips when the live route corpus is unready`, async ({ page }) => {
|
||||
const state = requireLiveRouteState(routeConfig.key);
|
||||
@@ -534,6 +570,8 @@ async function routeLiveUnifiedSearch(
|
||||
page: Page,
|
||||
capturedRequests?: Array<Record<string, unknown>>,
|
||||
): Promise<void> {
|
||||
const liveHeaders = buildLiveHeaders(liveRuntime);
|
||||
|
||||
await page.route('**/api/v1/search/query', async (route) => {
|
||||
const rawBody = route.request().postData() ?? '{}';
|
||||
const parsedBody = safeParseRequest(rawBody);
|
||||
@@ -543,12 +581,7 @@ async function routeLiveUnifiedSearch(
|
||||
|
||||
const response = await fetch(`${liveSearchBaseUrl}/v1/search/query`, {
|
||||
method: 'POST',
|
||||
headers: {
|
||||
'content-type': 'application/json',
|
||||
'x-stellaops-scopes': liveScopes,
|
||||
'x-stellaops-tenant': liveTenant,
|
||||
'x-stellaops-actor': 'playwright-live',
|
||||
},
|
||||
headers: liveHeaders,
|
||||
body: rawBody,
|
||||
});
|
||||
|
||||
@@ -571,38 +604,6 @@ async function routeLiveUnifiedSearch(
|
||||
});
|
||||
}
|
||||
|
||||
async function ensureLiveServiceHealthy(baseUrl: string): Promise<void> {
|
||||
const response = await fetch(`${baseUrl}/health`);
|
||||
if (!response.ok) {
|
||||
throw new Error(`Live AdvisoryAI health check failed with status ${response.status}.`);
|
||||
}
|
||||
}
|
||||
|
||||
async function rebuildLiveIndexes(baseUrl: string): Promise<void> {
|
||||
const headers = {
|
||||
'content-type': 'application/json',
|
||||
'x-stellaops-scopes': 'advisory-ai:admin',
|
||||
'x-stellaops-tenant': liveTenant,
|
||||
'x-stellaops-actor': 'playwright-live',
|
||||
};
|
||||
|
||||
const knowledgeResponse = await fetch(`${baseUrl}/v1/advisory-ai/index/rebuild`, {
|
||||
method: 'POST',
|
||||
headers,
|
||||
});
|
||||
if (!knowledgeResponse.ok) {
|
||||
throw new Error(`Knowledge rebuild failed with status ${knowledgeResponse.status}.`);
|
||||
}
|
||||
|
||||
const unifiedResponse = await fetch(`${baseUrl}/v1/search/index/rebuild`, {
|
||||
method: 'POST',
|
||||
headers,
|
||||
});
|
||||
if (!unifiedResponse.ok) {
|
||||
throw new Error(`Unified rebuild failed with status ${unifiedResponse.status}.`);
|
||||
}
|
||||
}
|
||||
|
||||
function safeParseRequest(rawBody: string): Record<string, unknown> {
|
||||
try {
|
||||
const parsed = JSON.parse(rawBody) as Record<string, unknown>;
|
||||
@@ -613,12 +614,7 @@ function safeParseRequest(rawBody: string): Record<string, unknown> {
|
||||
}
|
||||
|
||||
async function fetchLiveSuggestionViability(rawBody: string): Promise<Record<string, unknown>> {
|
||||
const headers = {
|
||||
'content-type': 'application/json',
|
||||
'x-stellaops-scopes': liveScopes,
|
||||
'x-stellaops-tenant': liveTenant,
|
||||
'x-stellaops-actor': 'playwright-live',
|
||||
};
|
||||
const headers = buildLiveHeaders(liveRuntime);
|
||||
|
||||
const directResponse = await fetch(`${liveSearchBaseUrl}/v1/search/suggestions/evaluate`, {
|
||||
method: 'POST',
|
||||
|
||||
Reference in New Issue
Block a user