Harden scratch-stack live QA sweeps
This commit is contained in:
@@ -49,11 +49,14 @@ Completion criteria:
|
||||
| 2026-03-10 | Added `live-integrations-action-sweep.mjs` to cover shell tabs, typed onboarding CTAs, generic onboarding fallbacks, and activity handoffs across the live Integrations family. | QA |
|
||||
| 2026-03-10 | First live sweep reported one false failure: `Advisory & VEX` was present in the Integrations shell but exposed as a tabbed-nav item rather than a plain link, so the harness selector missed it. No runtime issues were present. | Developer |
|
||||
| 2026-03-10 | Corrected the harness to search both `link` and `tab` roles for shell navigation and reran the exact live sweep. Final evidence is clean at `failedActionCount=0` and `runtimeIssueCount=0` in `src/Web.StellaOps.Web/output/playwright/live-integrations-action-sweep.json`. Commit hash pending local commit. | QA |
|
||||
| 2026-03-11 | Replayed the sweep after a full scratch teardown and rebuild. Fresh cold loads surfaced a second harness-only failure mode: the Integrations shell and list pages lazy-hydrated after navigation, so one-shot selector checks mislabeled tabs, add buttons, and empty-state CTAs as missing. Hardened the sweep with bounded control polling, support for the split `Advisory Sources` / `VEX Sources` hub tiles, and state-aware empty-state-or-detail checks. Final rerun is clean again at `failedActionCount=0` and `runtimeIssueCount=0`. | QA / Developer |
|
||||
|
||||
## Decisions & Risks
|
||||
- Decision: typed onboarding is part of the live integrations contract. If a type-specific list page routes to the wrong onboarding target, that is a product defect, not a cosmetic issue.
|
||||
- Risk: the Integrations hub mixes typed and generic onboarding flows. The harness must assert the intended destination per integration type instead of treating all add buttons as equivalent.
|
||||
- Decision: the Integrations shell uses tabbed navigation semantics. The live harness must consider both `link` and `tab` roles when verifying shell actions so accessibility-correct tabs do not register as false defects.
|
||||
- Decision: scratch-rebuild verification must wait for lazy route hydration before judging the Integrations shell. Fresh chunk loads can present the route URL before the tab bar and list CTAs are interactable, so the sweep now uses bounded polling instead of a single immediate DOM sample.
|
||||
- Decision: list-page coverage is state-aware. When a list is empty, the harness verifies the empty-state CTA; when rows exist, it verifies the first detail handoff instead of demanding an empty-state button that should not be present.
|
||||
|
||||
## Next Checkpoints
|
||||
- Capture the first integrations action sweep against the live frontdoor.
|
||||
|
||||
@@ -27,7 +27,7 @@ Owners: QA, 3rd line support, Product Manager, Architect, Developer
|
||||
Task description:
|
||||
- The repo already has live search verification for the standalone local shell plus AdvisoryAI runtime, but this scratch iteration needs the same route-by-route proof against the real authenticated Stella Ops frontdoor.
|
||||
- Add a script that authenticates against `https://stella-ops.local`, opens the supported route-local search surfaces, captures surfaced starter chips, executes each chip, and fails on missing context, missing starters, degraded banners, dead-end query execution, or runtime/network errors.
|
||||
- Live proof now shows a deeper backend/setup failure: Doctor context renders, but `POST /api/v1/search/suggestions/evaluate` returns `current_scope_corpus_unready` for the knowledge scope after a full scratch rebuild. The fix must make AdvisoryAI converge the knowledge corpus on startup instead of relying on manual rebuild commands.
|
||||
- The matrix must distinguish real search-runtime defects from cold-load convergence: starter chips are only trustworthy after the route-local `suggestions/evaluate` call settles, and backend/search response errors must be captured as first-class evidence instead of being flattened into a generic "no chips" failure.
|
||||
|
||||
Completion criteria:
|
||||
- [x] A live frontdoor search matrix script exists under `src/Web/StellaOps.Web/scripts/`.
|
||||
@@ -42,6 +42,7 @@ Completion criteria:
|
||||
| 2026-03-10 | Added `scripts/live-frontdoor-unified-search-route-matrix.mjs` and ran it against the rebuilt stack. Doctor search reproduces a real setup/runtime defect: the frontdoor returns `current_scope_corpus_unready` for all knowledge-scope starter queries even though the shell context is correct. Root-cause work is now moving into AdvisoryAI startup convergence. | QA / 3rd line support |
|
||||
| 2026-03-10 | Implemented AdvisoryAI startup convergence so the knowledge corpus rebuilds automatically on fresh service startup, rebuilt and redeployed `advisory-ai-web`, and confirmed the live container reports `documents=470`, `chunks=9051`, `api_operations=2190`, `doctor_projections=8` during startup rebuild. | Developer / 3rd line support |
|
||||
| 2026-03-10 | Reverified the live authenticated shell with a Playwright all-chip probe and wrote `src/Web/StellaOps.Web/output/playwright/live-frontdoor-unified-search-route-matrix-manual.json`. Doctor, Security Triage, Policy, and Advisories & VEX all render context-aware starter chips and their visible chip actions now resolve to grounded answers with cards. | QA |
|
||||
| 2026-03-11 | Replayed the live matrix after a full scratch teardown and rebuild. The first rerun reported "no starter chips" on Doctor, Security Triage, Policy, and Advisories & VEX, but direct browser/network inspection showed the live product was healthy: `POST /api/v1/search/suggestions/evaluate` returned `200` with viable suggestions after about 9 seconds on cold load, and the chips rendered immediately after that response. Hardened the matrix with bounded starter-panel polling plus `/api/v1/search*` response/request error capture, then reran it cleanly with `runtimeIssueCount=0`. | QA / 3rd line support / Developer |
|
||||
|
||||
## Decisions & Risks
|
||||
- Decision: frontdoor search verification must not rely on the standalone Angular/AdvisoryAI harness alone; the authenticated shell is the product surface the client sees.
|
||||
@@ -49,6 +50,9 @@ Completion criteria:
|
||||
- Decision: only the AdvisoryAI web host owns startup knowledge-index convergence. The shared library must not register that hosted service globally because the worker shares the same core registrations and would otherwise perform a duplicate rebuild on startup.
|
||||
- Risk: live search starters depend on current route context and runtime corpus readiness, so the sweep must distinguish product regressions from transient auth/runtime setup failures with structured evidence.
|
||||
|
||||
- Decision: cold-loaded unified search routes now wait for the starter panel to settle before judging chip availability. On fresh scratch installs the suggestion-viability request can take about 9 seconds even when the product is healthy, so one-shot 4-second waits are not reliable evidence.
|
||||
- Decision: the matrix now captures `/api/v1/search*` response and request failures directly. Missing starter chips without transport/runtime evidence are treated as a UI-settling problem until the bounded wait expires.
|
||||
|
||||
## Next Checkpoints
|
||||
- Implement the live frontdoor search sweep harness.
|
||||
- Run it against the rebuilt stack and triage any failures before widening to the next untouched page family.
|
||||
|
||||
Reference in New Issue
Block a user