From 4dc5db4efbe0d42b09226065bbe6609340db5906 Mon Sep 17 00:00:00 2001 From: master <> Date: Wed, 11 Mar 2026 12:07:00 +0200 Subject: [PATCH] Harden scratch-stack live QA sweeps --- ...0_005_FE_integrations_live_action_sweep.md | 3 + ...e_frontdoor_unified_search_route_matrix.md | 6 +- ...-frontdoor-unified-search-route-matrix.mjs | 91 +++++++++- .../live-integrations-action-sweep.mjs | 157 ++++++++++++++++-- 4 files changed, 238 insertions(+), 19 deletions(-) diff --git a/docs/implplan/SPRINT_20260310_005_FE_integrations_live_action_sweep.md b/docs/implplan/SPRINT_20260310_005_FE_integrations_live_action_sweep.md index 69f381e00..0f48c2af5 100644 --- a/docs/implplan/SPRINT_20260310_005_FE_integrations_live_action_sweep.md +++ b/docs/implplan/SPRINT_20260310_005_FE_integrations_live_action_sweep.md @@ -49,11 +49,14 @@ Completion criteria: | 2026-03-10 | Added `live-integrations-action-sweep.mjs` to cover shell tabs, typed onboarding CTAs, generic onboarding fallbacks, and activity handoffs across the live Integrations family. | QA | | 2026-03-10 | First live sweep reported one false failure: `Advisory & VEX` was present in the Integrations shell but exposed as a tabbed-nav item rather than a plain link, so the harness selector missed it. No runtime issues were present. | Developer | | 2026-03-10 | Corrected the harness to search both `link` and `tab` roles for shell navigation and reran the exact live sweep. Final evidence is clean at `failedActionCount=0` and `runtimeIssueCount=0` in `src/Web.StellaOps.Web/output/playwright/live-integrations-action-sweep.json`. Commit hash pending local commit. | QA | +| 2026-03-11 | Replayed the sweep after a full scratch teardown and rebuild. Fresh cold loads surfaced a second harness-only failure mode: the Integrations shell and list pages lazy-hydrated after navigation, so one-shot selector checks mislabeled tabs, add buttons, and empty-state CTAs as missing. Hardened the sweep with bounded control polling, support for the split `Advisory Sources` / `VEX Sources` hub tiles, and state-aware empty-state-or-detail checks. Final rerun is clean again at `failedActionCount=0` and `runtimeIssueCount=0`. | QA / Developer | ## Decisions & Risks - Decision: typed onboarding is part of the live integrations contract. If a type-specific list page routes to the wrong onboarding target, that is a product defect, not a cosmetic issue. - Risk: the Integrations hub mixes typed and generic onboarding flows. The harness must assert the intended destination per integration type instead of treating all add buttons as equivalent. - Decision: the Integrations shell uses tabbed navigation semantics. The live harness must consider both `link` and `tab` roles when verifying shell actions so accessibility-correct tabs do not register as false defects. +- Decision: scratch-rebuild verification must wait for lazy route hydration before judging the Integrations shell. Fresh chunk loads can present the route URL before the tab bar and list CTAs are interactable, so the sweep now uses bounded polling instead of a single immediate DOM sample. +- Decision: list-page coverage is state-aware. When a list is empty, the harness verifies the empty-state CTA; when rows exist, it verifies the first detail handoff instead of demanding an empty-state button that should not be present. ## Next Checkpoints - Capture the first integrations action sweep against the live frontdoor. diff --git a/docs/implplan/SPRINT_20260310_033_FE_live_frontdoor_unified_search_route_matrix.md b/docs/implplan/SPRINT_20260310_033_FE_live_frontdoor_unified_search_route_matrix.md index 5f2da074a..633317c46 100644 --- a/docs/implplan/SPRINT_20260310_033_FE_live_frontdoor_unified_search_route_matrix.md +++ b/docs/implplan/SPRINT_20260310_033_FE_live_frontdoor_unified_search_route_matrix.md @@ -27,7 +27,7 @@ Owners: QA, 3rd line support, Product Manager, Architect, Developer Task description: - The repo already has live search verification for the standalone local shell plus AdvisoryAI runtime, but this scratch iteration needs the same route-by-route proof against the real authenticated Stella Ops frontdoor. - Add a script that authenticates against `https://stella-ops.local`, opens the supported route-local search surfaces, captures surfaced starter chips, executes each chip, and fails on missing context, missing starters, degraded banners, dead-end query execution, or runtime/network errors. -- Live proof now shows a deeper backend/setup failure: Doctor context renders, but `POST /api/v1/search/suggestions/evaluate` returns `current_scope_corpus_unready` for the knowledge scope after a full scratch rebuild. The fix must make AdvisoryAI converge the knowledge corpus on startup instead of relying on manual rebuild commands. +- The matrix must distinguish real search-runtime defects from cold-load convergence: starter chips are only trustworthy after the route-local `suggestions/evaluate` call settles, and backend/search response errors must be captured as first-class evidence instead of being flattened into a generic "no chips" failure. Completion criteria: - [x] A live frontdoor search matrix script exists under `src/Web/StellaOps.Web/scripts/`. @@ -42,6 +42,7 @@ Completion criteria: | 2026-03-10 | Added `scripts/live-frontdoor-unified-search-route-matrix.mjs` and ran it against the rebuilt stack. Doctor search reproduces a real setup/runtime defect: the frontdoor returns `current_scope_corpus_unready` for all knowledge-scope starter queries even though the shell context is correct. Root-cause work is now moving into AdvisoryAI startup convergence. | QA / 3rd line support | | 2026-03-10 | Implemented AdvisoryAI startup convergence so the knowledge corpus rebuilds automatically on fresh service startup, rebuilt and redeployed `advisory-ai-web`, and confirmed the live container reports `documents=470`, `chunks=9051`, `api_operations=2190`, `doctor_projections=8` during startup rebuild. | Developer / 3rd line support | | 2026-03-10 | Reverified the live authenticated shell with a Playwright all-chip probe and wrote `src/Web/StellaOps.Web/output/playwright/live-frontdoor-unified-search-route-matrix-manual.json`. Doctor, Security Triage, Policy, and Advisories & VEX all render context-aware starter chips and their visible chip actions now resolve to grounded answers with cards. | QA | +| 2026-03-11 | Replayed the live matrix after a full scratch teardown and rebuild. The first rerun reported "no starter chips" on Doctor, Security Triage, Policy, and Advisories & VEX, but direct browser/network inspection showed the live product was healthy: `POST /api/v1/search/suggestions/evaluate` returned `200` with viable suggestions after about 9 seconds on cold load, and the chips rendered immediately after that response. Hardened the matrix with bounded starter-panel polling plus `/api/v1/search*` response/request error capture, then reran it cleanly with `runtimeIssueCount=0`. | QA / 3rd line support / Developer | ## Decisions & Risks - Decision: frontdoor search verification must not rely on the standalone Angular/AdvisoryAI harness alone; the authenticated shell is the product surface the client sees. @@ -49,6 +50,9 @@ Completion criteria: - Decision: only the AdvisoryAI web host owns startup knowledge-index convergence. The shared library must not register that hosted service globally because the worker shares the same core registrations and would otherwise perform a duplicate rebuild on startup. - Risk: live search starters depend on current route context and runtime corpus readiness, so the sweep must distinguish product regressions from transient auth/runtime setup failures with structured evidence. +- Decision: cold-loaded unified search routes now wait for the starter panel to settle before judging chip availability. On fresh scratch installs the suggestion-viability request can take about 9 seconds even when the product is healthy, so one-shot 4-second waits are not reliable evidence. +- Decision: the matrix now captures `/api/v1/search*` response and request failures directly. Missing starter chips without transport/runtime evidence are treated as a UI-settling problem until the bounded wait expires. + ## Next Checkpoints - Implement the live frontdoor search sweep harness. - Run it against the rebuilt stack and triage any failures before widening to the next untouched page family. diff --git a/src/Web/StellaOps.Web/scripts/live-frontdoor-unified-search-route-matrix.mjs b/src/Web/StellaOps.Web/scripts/live-frontdoor-unified-search-route-matrix.mjs index b87c81442..293b9ec7a 100644 --- a/src/Web/StellaOps.Web/scripts/live-frontdoor-unified-search-route-matrix.mjs +++ b/src/Web/StellaOps.Web/scripts/live-frontdoor-unified-search-route-matrix.mjs @@ -48,6 +48,8 @@ function createRuntime() { return { consoleErrors: [], pageErrors: [], + requestFailures: [], + responseErrors: [], }; } @@ -69,6 +71,43 @@ function attachRuntimeListeners(page, runtime) { message: error.message, }); }); + + page.on('requestfailed', (request) => { + const url = request.url(); + if (/\.(?:css|js|map|png|jpg|jpeg|svg|woff2?)(?:$|\?)/i.test(url)) { + return; + } + + const errorText = request.failure()?.errorText ?? 'unknown'; + if (errorText === 'net::ERR_ABORTED') { + return; + } + + runtime.requestFailures.push({ + timestamp: Date.now(), + page: page.url(), + method: request.method(), + url, + error: errorText, + }); + }); + + page.on('response', (response) => { + const url = response.url(); + if (!url.includes('/api/v1/search')) { + return; + } + + if (response.status() >= 400) { + runtime.responseErrors.push({ + timestamp: Date.now(), + page: page.url(), + method: response.request().method(), + status: response.status(), + url, + }); + } + }); } async function readVisibleTexts(locator) { @@ -88,7 +127,45 @@ async function openSearch(page, route) { const input = page.locator('app-global-search input[type="text"]').first(); await input.click({ timeout: 10_000 }); await page.waitForSelector('.search__results', { state: 'visible', timeout: 10_000 }); - await page.waitForTimeout(4_000); + await waitForStarterPanel(page); +} + +async function waitForStarterPanel(page, timeoutMs = 20_000) { + const startedAt = Date.now(); + + while (Date.now() - startedAt < timeoutMs) { + const state = await page.evaluate(() => { + const starterChips = document.querySelectorAll('.search__suggestions .search__chip').length; + const degradedBanners = document.querySelectorAll('.search__degraded-banner').length; + const emptyTexts = Array.from(document.querySelectorAll('.search__empty, .search__empty-state-copy')) + .map((node) => (node.textContent || '').trim().replace(/\s+/g, ' ')) + .filter((text) => text.length > 0); + + return { + starterChips, + degradedBanners, + emptyTexts, + }; + }).catch(() => ({ + starterChips: 0, + degradedBanners: 0, + emptyTexts: [], + })); + + if (state.starterChips > 0 || state.degradedBanners > 0) { + await page.waitForTimeout(750); + return; + } + + const hasDefaultPromptOnly = state.emptyTexts.length > 0 + && state.emptyTexts.every((text) => /ask about what is on this page/i.test(text)); + if (!hasDefaultPromptOnly && state.emptyTexts.length > 0) { + await page.waitForTimeout(750); + return; + } + + await page.waitForTimeout(500); + } } async function captureSnapshot(page, routeConfig, runtime, routeStartedAt) { @@ -107,6 +184,8 @@ async function captureSnapshot(page, routeConfig, runtime, routeStartedAt) { cardTitles: await readVisibleTexts(page.locator('.search__cards .entity-card__title')), consoleErrors: runtime.consoleErrors.filter((entry) => entry.timestamp >= routeStartedAt), pageErrors: runtime.pageErrors.filter((entry) => entry.timestamp >= routeStartedAt), + requestFailures: runtime.requestFailures.filter((entry) => entry.timestamp >= routeStartedAt), + responseErrors: runtime.responseErrors.filter((entry) => entry.timestamp >= routeStartedAt), }; } @@ -154,6 +233,16 @@ function buildIssues(routeResult) { } issues.push(...routeResult.snapshot.consoleErrors.map((entry) => `console:${entry.text}`)); issues.push(...routeResult.snapshot.pageErrors.map((entry) => `pageerror:${entry.message}`)); + issues.push( + ...routeResult.snapshot.requestFailures.map( + (entry) => `requestfailed:${entry.method} ${entry.url} ${entry.error}`, + ), + ); + issues.push( + ...routeResult.snapshot.responseErrors.map( + (entry) => `response:${entry.status} ${entry.method} ${entry.url}`, + ), + ); for (const starter of routeResult.executedStarters) { if (!starter.ok) { issues.push(`Starter index ${starter.starterIndex} "${starter.starterText}" did not resolve to grounded results on ${routeResult.route}`); diff --git a/src/Web/StellaOps.Web/scripts/live-integrations-action-sweep.mjs b/src/Web/StellaOps.Web/scripts/live-integrations-action-sweep.mjs index 9a0f776f5..1beb7c392 100644 --- a/src/Web/StellaOps.Web/scripts/live-integrations-action-sweep.mjs +++ b/src/Web/StellaOps.Web/scripts/live-integrations-action-sweep.mjs @@ -16,7 +16,7 @@ const STEP_TIMEOUT_MS = 30_000; async function settle(page) { await page.waitForLoadState('domcontentloaded', { timeout: 15_000 }).catch(() => {}); - await page.waitForTimeout(1_500); + await page.waitForTimeout(2_500); } async function headingText(page) { @@ -65,6 +65,51 @@ async function navigate(page, route) { return url; } +function escapeRegExp(value) { + return value.replace(/[.*+?^${}()|[\]\\]/g, '\\$&'); +} + +async function resolveInteractiveLocator(page, name, role = 'link', timeoutMs = 8_000) { + const matcher = name instanceof RegExp ? name : new RegExp(escapeRegExp(name), 'i'); + const startedAt = Date.now(); + + while (Date.now() - startedAt < timeoutMs) { + const candidates = [ + page.getByRole(role, { name: matcher }), + page.locator(role === 'tab' ? '[role="tab"]' : 'a, [role="link"]').filter({ hasText: matcher }), + ]; + + for (const candidate of candidates) { + if ((await candidate.count().catch(() => 0)) > 0) { + const locator = candidate.first(); + if (await locator.isVisible().catch(() => true)) { + return locator; + } + } + } + + await page.waitForTimeout(250); + } + + return null; +} + +async function resolveButtonLocator(page, name, timeoutMs = 8_000) { + const matcher = name instanceof RegExp ? name : new RegExp(escapeRegExp(name), 'i'); + const startedAt = Date.now(); + + while (Date.now() - startedAt < timeoutMs) { + const locator = page.getByRole('button', { name: matcher }).first(); + if ((await locator.count().catch(() => 0)) > 0 && (await locator.isVisible().catch(() => true))) { + return locator; + } + + await page.waitForTimeout(250); + } + + return null; +} + async function runAction(page, route, label, runner) { const startedAtUtc = new Date().toISOString(); const startedAt = Date.now(); @@ -117,14 +162,14 @@ async function runAction(page, route, label, runner) { async function clickLinkVerify(page, route, name, expectedPath) { await navigate(page, route); const candidates = [ - page.getByRole('link', { name }), - page.getByRole('tab', { name }), + await resolveInteractiveLocator(page, name, 'link'), + await resolveInteractiveLocator(page, name, 'tab'), ]; let locator = null; for (const candidate of candidates) { - if ((await candidate.count()) > 0) { - locator = candidate.first(); + if (candidate) { + locator = candidate; break; } } @@ -150,8 +195,8 @@ async function clickLinkVerify(page, route, name, expectedPath) { async function clickButtonVerify(page, route, name, expectedPath) { await navigate(page, route); - const locator = page.getByRole('button', { name }).first(); - if ((await locator.count()) === 0) { + const locator = await resolveButtonLocator(page, name); + if (!locator) { return { ok: false, reason: 'missing-button', @@ -170,6 +215,62 @@ async function clickButtonVerify(page, route, name, expectedPath) { }; } +async function clickEmptyStateOrFirstDetailVerify(page, route, buttonName, expectedPath) { + await navigate(page, route); + + const emptyStateButton = await resolveButtonLocator(page, buttonName, 4_000); + if (emptyStateButton) { + await emptyStateButton.click({ timeout: 10_000 }); + await page.waitForURL((url) => url.pathname.includes(expectedPath), { timeout: 15_000 }); + await settle(page); + + return { + ok: page.url().includes(expectedPath), + mode: 'empty-state-button', + expectedPath, + snapshot: await captureSnapshot(page, `after-empty-state-button:${buttonName}`), + }; + } + + const startedAt = Date.now(); + while (Date.now() - startedAt < 8_000) { + const detailLink = page.locator('tbody tr td a').first(); + if ((await detailLink.count().catch(() => 0)) > 0 && (await detailLink.isVisible().catch(() => true))) { + const originPath = new URL(page.url()).pathname; + await detailLink.click({ timeout: 10_000 }); + await page.waitForURL( + (url) => url.pathname !== originPath && url.pathname.startsWith('/ops/integrations/'), + { timeout: 15_000 }, + ); + await settle(page); + + return { + ok: true, + mode: 'detail-link', + expectedPath: '/ops/integrations/:integrationId', + snapshot: await captureSnapshot(page, `after-detail-link:${route}`), + }; + } + + const loadError = page.locator('.error-state').first(); + if ((await loadError.count().catch(() => 0)) > 0 && (await loadError.isVisible().catch(() => true))) { + return { + ok: false, + reason: 'load-error', + snapshot: await captureSnapshot(page, `load-error:${route}`), + }; + } + + await page.waitForTimeout(250); + } + + return { + ok: false, + reason: 'missing-empty-state-and-detail', + snapshot: await captureSnapshot(page, `missing-empty-state-and-detail:${route}`), + }; +} + async function main() { await mkdir(outputDir, { recursive: true }); @@ -255,8 +356,10 @@ async function main() { clickLinkVerify(page, '/ops/integrations', 'CI/CD', '/ops/integrations/ci')), await runAction(page, '/ops/integrations', 'link:Runtimes / Hosts', () => clickLinkVerify(page, '/ops/integrations', 'Runtimes / Hosts', '/ops/integrations/runtime-hosts')), - await runAction(page, '/ops/integrations', 'link:Advisory & VEX', () => - clickLinkVerify(page, '/ops/integrations', 'Advisory & VEX', '/ops/integrations/advisory-vex-sources')), + await runAction(page, '/ops/integrations', 'link:Advisory Sources', () => + clickLinkVerify(page, '/ops/integrations', 'Advisory Sources', '/ops/integrations/advisory-vex-sources')), + await runAction(page, '/ops/integrations', 'link:VEX Sources', () => + clickLinkVerify(page, '/ops/integrations', 'VEX Sources', '/ops/integrations/advisory-vex-sources')), await runAction(page, '/ops/integrations', 'link:Secrets', () => clickLinkVerify(page, '/ops/integrations', 'Secrets', '/ops/integrations/secrets')), await runAction(page, '/ops/integrations', 'button:+ Add Integration', () => @@ -272,8 +375,13 @@ async function main() { actions: [ await runAction(page, '/ops/integrations/registries', 'button:+ Add Registry', () => clickButtonVerify(page, '/ops/integrations/registries', '+ Add Registry', '/ops/integrations/onboarding/registry')), - await runAction(page, '/ops/integrations/registries', 'button:Add your first registry', () => - clickButtonVerify(page, '/ops/integrations/registries', 'Add your first registry', '/ops/integrations/onboarding/registry')), + await runAction(page, '/ops/integrations/registries', 'empty-or-detail:Registries', () => + clickEmptyStateOrFirstDetailVerify( + page, + '/ops/integrations/registries', + 'Add your first registry', + '/ops/integrations/onboarding/registry', + )), ], }); await persistSummary(summary); @@ -283,8 +391,13 @@ async function main() { actions: [ await runAction(page, '/ops/integrations/runtime-hosts', 'button:+ Add RuntimeHost', () => clickButtonVerify(page, '/ops/integrations/runtime-hosts', '+ Add RuntimeHost', '/ops/integrations/onboarding/host')), - await runAction(page, '/ops/integrations/runtime-hosts', 'button:Add your first runtimehost', () => - clickButtonVerify(page, '/ops/integrations/runtime-hosts', 'Add your first runtimehost', '/ops/integrations/onboarding/host')), + await runAction(page, '/ops/integrations/runtime-hosts', 'empty-or-detail:Runtimes / Hosts', () => + clickEmptyStateOrFirstDetailVerify( + page, + '/ops/integrations/runtime-hosts', + 'Add your first runtimehost', + '/ops/integrations/onboarding/host', + )), ], }); await persistSummary(summary); @@ -294,8 +407,13 @@ async function main() { actions: [ await runAction(page, '/ops/integrations/secrets', 'button:+ Add Integration', () => clickButtonVerify(page, '/ops/integrations/secrets', '+ Add Integration', '/ops/integrations/onboarding')), - await runAction(page, '/ops/integrations/secrets', 'button:Open add integration hub', () => - clickButtonVerify(page, '/ops/integrations/secrets', 'Open add integration hub', '/ops/integrations/onboarding')), + await runAction(page, '/ops/integrations/secrets', 'empty-or-detail:Secrets', () => + clickEmptyStateOrFirstDetailVerify( + page, + '/ops/integrations/secrets', + 'Open add integration hub', + '/ops/integrations/onboarding', + )), ], }); await persistSummary(summary); @@ -305,8 +423,13 @@ async function main() { actions: [ await runAction(page, '/ops/integrations/advisory-vex-sources', 'button:+ Add Integration', () => clickButtonVerify(page, '/ops/integrations/advisory-vex-sources', '+ Add Integration', '/ops/integrations/onboarding')), - await runAction(page, '/ops/integrations/advisory-vex-sources', 'button:Open add integration hub', () => - clickButtonVerify(page, '/ops/integrations/advisory-vex-sources', 'Open add integration hub', '/ops/integrations/onboarding')), + await runAction(page, '/ops/integrations/advisory-vex-sources', 'empty-or-detail:Advisory & VEX Sources', () => + clickEmptyStateOrFirstDetailVerify( + page, + '/ops/integrations/advisory-vex-sources', + 'Open add integration hub', + '/ops/integrations/onboarding', + )), ], }); await persistSummary(summary);