feat: Add UI benchmark driver and scenarios for graph interactions
Some checks failed
AOC Guard CI / aoc-guard (push) Has been cancelled
AOC Guard CI / aoc-verify (push) Has been cancelled
Docs CI / lint-and-preview (push) Has been cancelled
Policy Lint & Smoke / policy-lint (push) Has been cancelled
devportal-offline / build-offline (push) Has been cancelled
Some checks failed
AOC Guard CI / aoc-guard (push) Has been cancelled
AOC Guard CI / aoc-verify (push) Has been cancelled
Docs CI / lint-and-preview (push) Has been cancelled
Policy Lint & Smoke / policy-lint (push) Has been cancelled
devportal-offline / build-offline (push) Has been cancelled
- Introduced `ui_bench_driver.mjs` to read scenarios and fixture manifest, generating a deterministic run plan. - Created `ui_bench_plan.md` outlining the purpose, scope, and next steps for the benchmark. - Added `ui_bench_scenarios.json` containing various scenarios for graph UI interactions. - Implemented tests for CLI commands, ensuring bundle verification and telemetry defaults. - Developed schemas for orchestrator components, including replay manifests and event envelopes. - Added mock API for risk management, including listing and statistics functionalities. - Implemented models for risk profiles and query options to support the new API.
This commit is contained in:
24
src/Bench/StellaOps.Bench/Graph/run_graph_bench.sh
Normal file
24
src/Bench/StellaOps.Bench/Graph/run_graph_bench.sh
Normal file
@@ -0,0 +1,24 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
|
||||
ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
# Repo root is four levels up from Graph/
|
||||
REPO_ROOT="$(cd "${ROOT}/../../../.." && pwd)"
|
||||
FIXTURES_ROOT="${FIXTURES_ROOT:-${REPO_ROOT}/samples/graph/interim}"
|
||||
OUT_DIR="${OUT_DIR:-$ROOT/results}"
|
||||
SAMPLES="${SAMPLES:-100}"
|
||||
|
||||
mkdir -p "${OUT_DIR}"
|
||||
|
||||
run_one() {
|
||||
local fixture="$1"
|
||||
local name
|
||||
name="$(basename "${fixture}")"
|
||||
local out_file="${OUT_DIR}/${name}.json"
|
||||
python "${ROOT}/graph_bench.py" --fixture "${fixture}" --output "${out_file}" --samples "${SAMPLES}"
|
||||
}
|
||||
|
||||
run_one "${FIXTURES_ROOT}/graph-50k"
|
||||
run_one "${FIXTURES_ROOT}/graph-100k"
|
||||
|
||||
echo "Graph bench complete. Results in ${OUT_DIR}"
|
||||
50
src/Bench/StellaOps.Bench/Graph/ui_bench_driver.mjs
Normal file
50
src/Bench/StellaOps.Bench/Graph/ui_bench_driver.mjs
Normal file
@@ -0,0 +1,50 @@
|
||||
#!/usr/bin/env node
|
||||
/**
|
||||
* ui_bench_driver.mjs
|
||||
*
|
||||
* Reads scenarios and fixture manifest, and emits a deterministic run plan.
|
||||
* This is browser-free; intended to be wrapped by Playwright later.
|
||||
*/
|
||||
import fs from "fs";
|
||||
import path from "path";
|
||||
|
||||
function readJson(p) {
|
||||
return JSON.parse(fs.readFileSync(p, "utf-8"));
|
||||
}
|
||||
|
||||
function buildPlan(scenarios, manifest, fixtureName) {
|
||||
const now = new Date().toISOString();
|
||||
return {
|
||||
version: "1.0.0",
|
||||
fixture: fixtureName,
|
||||
manifestHash: manifest?.hashes || {},
|
||||
timestamp: now,
|
||||
steps: scenarios.map((s, idx) => ({
|
||||
order: idx + 1,
|
||||
id: s.id,
|
||||
name: s.name,
|
||||
actions: s.steps,
|
||||
})),
|
||||
};
|
||||
}
|
||||
|
||||
function main() {
|
||||
const fixtureDir = process.argv[2];
|
||||
const scenariosPath = process.argv[3];
|
||||
const outputPath = process.argv[4];
|
||||
if (!fixtureDir || !scenariosPath || !outputPath) {
|
||||
console.error("usage: ui_bench_driver.mjs <fixture_dir> <scenarios.json> <output.json>");
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
const manifestPath = path.join(fixtureDir, "manifest.json");
|
||||
const manifest = fs.existsSync(manifestPath) ? readJson(manifestPath) : {};
|
||||
const scenarios = readJson(scenariosPath).scenarios || [];
|
||||
|
||||
const plan = buildPlan(scenarios, manifest, path.basename(fixtureDir));
|
||||
fs.mkdirSync(path.dirname(outputPath), { recursive: true });
|
||||
fs.writeFileSync(outputPath, JSON.stringify(plan, null, 2));
|
||||
console.log(`Wrote plan to ${outputPath}`);
|
||||
}
|
||||
|
||||
main();
|
||||
30
src/Bench/StellaOps.Bench/Graph/ui_bench_plan.md
Normal file
30
src/Bench/StellaOps.Bench/Graph/ui_bench_plan.md
Normal file
@@ -0,0 +1,30 @@
|
||||
# Graph UI Bench Plan (BENCH-GRAPH-21-002)
|
||||
|
||||
Purpose: provide a deterministic, headless flow for measuring graph UI interactions over large fixtures (50k/100k nodes).
|
||||
|
||||
## Scope
|
||||
- Use synthetic fixtures under `samples/graph/interim/` until canonical SAMPLES-GRAPH-24-003 lands.
|
||||
- Drive a deterministic sequence of interactions:
|
||||
1) Load graph canvas with specified fixture.
|
||||
2) Pan to node `pkg-000001`.
|
||||
3) Zoom in 2×, zoom out 1×.
|
||||
4) Apply filter `name contains "package-0001"`.
|
||||
5) Select node, expand neighbors (depth 1), collapse.
|
||||
6) Toggle overlay layer (once available).
|
||||
- Capture timings: initial render, filter apply, expand/collapse, overlay toggle.
|
||||
|
||||
## Determinism rules
|
||||
- Fixed seed for any randomized layouts (seed `424242`).
|
||||
- Disable animations/transitions where possible; otherwise measure after `requestAnimationFrame` settle.
|
||||
- No network calls; fixtures loaded from local file served by test harness.
|
||||
- Stable viewport (width=1280, height=720), device scale factor 1.
|
||||
|
||||
## Artifacts
|
||||
- `ui_bench_scenarios.json` — canonical scenario list with step ids and notes.
|
||||
- `ui_bench_driver.mjs` — helper that reads scenario + fixture manifest and emits a run plan (no browser dependency). Intended to be wrapped by a Playwright script later.
|
||||
- Results format (proposed): NDJSON with `{stepId, name, durationMs, fixture, timestamp}`.
|
||||
|
||||
## Next steps
|
||||
- Bind `ui_bench_driver.mjs` into a Playwright harness when Graph UI build/serve target is available.
|
||||
- Swap fixtures to SAMPLES-GRAPH-24-003 + overlay once schema finalized; keep scenario ids stable.
|
||||
- Add CI slice to run the driver and validate scenario/fixture bindings (no browser) to keep determinism checked in commits.
|
||||
35
src/Bench/StellaOps.Bench/Graph/ui_bench_scenarios.json
Normal file
35
src/Bench/StellaOps.Bench/Graph/ui_bench_scenarios.json
Normal file
@@ -0,0 +1,35 @@
|
||||
{
|
||||
"version": "1.0.0",
|
||||
"scenarios": [
|
||||
{
|
||||
"id": "load",
|
||||
"name": "Load graph canvas",
|
||||
"steps": ["navigate", "waitForRender"]
|
||||
},
|
||||
{
|
||||
"id": "pan-start-node",
|
||||
"name": "Pan to pkg-000001",
|
||||
"steps": ["panTo:pkg-000001"]
|
||||
},
|
||||
{
|
||||
"id": "zoom-in-out",
|
||||
"name": "Zoom in twice, out once",
|
||||
"steps": ["zoomIn", "zoomIn", "zoomOut"]
|
||||
},
|
||||
{
|
||||
"id": "filter-name",
|
||||
"name": "Filter name contains package-0001",
|
||||
"steps": ["setFilter:name=package-0001", "waitForRender"]
|
||||
},
|
||||
{
|
||||
"id": "expand-collapse",
|
||||
"name": "Expand neighbors then collapse",
|
||||
"steps": ["select:pkg-000001", "expandDepth:1", "collapseSelection"]
|
||||
},
|
||||
{
|
||||
"id": "overlay-toggle",
|
||||
"name": "Toggle overlay layer",
|
||||
"steps": ["toggleOverlay:on", "toggleOverlay:off"]
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -4,3 +4,4 @@
|
||||
| --- | --- | --- | --- | --- |
|
||||
| BENCH-DETERMINISM-401-057 | DONE (2025-11-26) | SPRINT_0512_0001_0001_bench | Determinism harness and mock scanner added under `src/Bench/StellaOps.Bench/Determinism`; manifests + sample inputs included. | `src/Bench/StellaOps.Bench/Determinism/results` (generated) |
|
||||
| BENCH-GRAPH-21-001 | DOING (2025-12-01) | SPRINT_0512_0001_0001_bench | Added interim graph bench harness (`Graph/graph_bench.py`) using synthetic 50k/100k fixtures; measures adjacency build + depth-3 reach; pending overlay schema for final fixture integration. | `src/Bench/StellaOps.Bench/Graph` |
|
||||
| BENCH-GRAPH-21-002 | DOING (2025-12-01) | SPRINT_0512_0001_0001_bench | Added Graph UI bench scaffold (scenarios JSON + driver + plan) using interim fixtures; awaits overlay schema/UI target for Playwright binding and timing collection. | `src/Bench/StellaOps.Bench/Graph` |
|
||||
|
||||
Reference in New Issue
Block a user