feat: Add UI benchmark driver and scenarios for graph interactions
Some checks failed
AOC Guard CI / aoc-guard (push) Has been cancelled
AOC Guard CI / aoc-verify (push) Has been cancelled
Docs CI / lint-and-preview (push) Has been cancelled
Policy Lint & Smoke / policy-lint (push) Has been cancelled
devportal-offline / build-offline (push) Has been cancelled

- Introduced `ui_bench_driver.mjs` to read scenarios and fixture manifest, generating a deterministic run plan.
- Created `ui_bench_plan.md` outlining the purpose, scope, and next steps for the benchmark.
- Added `ui_bench_scenarios.json` containing various scenarios for graph UI interactions.
- Implemented tests for CLI commands, ensuring bundle verification and telemetry defaults.
- Developed schemas for orchestrator components, including replay manifests and event envelopes.
- Added mock API for risk management, including listing and statistics functionalities.
- Implemented models for risk profiles and query options to support the new API.
This commit is contained in:
StellaOps Bot
2025-12-02 01:28:17 +02:00
parent 909d9b6220
commit 44171930ff
94 changed files with 3606 additions and 271 deletions

View File

@@ -0,0 +1,66 @@
# Signals Heuristic Catalog · Deterministic Scoring
**Compiled:** 2025-12-01 (UTC)
**Scope:** Close UT1UT10 gaps from `docs/product-advisories/31-Nov-2025 FINDINGS.md` by publishing a signed heuristic catalog and golden outputs.
**Status:** Draft; target publish 2025-12-05 with DSSE signature.
## Decisions (UT1UT10)
- **Signed catalog/schema (UT1):** Catalog lives at `heuristics.catalog.json` with schema versioned `heuristics.schema.json`; DSSE predicate `stella.ops/heuristicCatalog@v1` required.
- **Deterministic scoring formula (UT2):** Each heuristic defines `inputs`, `weights`, and `normalization`; scoring outputs canonicalized (sorted keys, fixed 3dp). Engine must be pure/deterministic; randomization forbidden.
- **Quality bands (UT3):** Bands: `gold` (precision≥0.9, recall≥0.8), `silver` (≥0.8/0.7), `bronze` (≥0.7/0.6). Bands recorded in catalog and enforced in admission checks.
- **Waiver policy with DSSE (UT4):** Waivers require DSSE envelope `stella.ops/heuristicWaiver@v1`, include reason, scope, expiry; dual approval (Signals+Policy).
- **SLA coupling (UT5):** SLA-tagged items cannot use heuristics below `silver`; SLA enforcement checks band before accepting results.
- **Offline kit packaging (UT6):** Catalog, schema, golden fixtures, and DSSE envelopes bundled in offline kits with `SHA256SUMS`.
- **Observability/alerts (UT7):** Metrics: `heuristics_eval_latency`, `heuristics_band_usage`, `heuristics_waivers_total`, `heuristics_score_drift`. Alerts when drift >1 band vs golden fixtures or when waivers exceed threshold.
- **Backfill plan (UT8):** Backfill job recomputes heuristic scores with current catalog; outputs `heuristics_backfill.ndjson` + checksum; mismatches raise alerts.
- **Explainability fields/exports (UT9):** Outputs must include `explanation` block: contributing signals with weights, normalized scores, and rule IDs. CLI/API export supports `--explain` and deterministic ordering.
- **Fixtures with golden outputs (UT10):** Golden set per heuristic under `fixtures/<heuristic>/` containing `input.json`, `expected.json`, and `README`; used in CI for determinism.
## Catalog Structure (draft)
```json
{
"version": "1.0.0",
"updatedAt": "2025-12-01T00:00:00Z",
"heuristics": [
{
"id": "heur.callgraph.hotpath",
"band": "gold",
"inputs": ["callgraph.depth", "callgraph.betweenness"],
"formula": "0.6*depth_norm + 0.4*betweenness_norm",
"normalization": "minmax",
"evidence": ["signals/callgraph"]
},
{
"id": "heur.pkg.sbom_age",
"band": "silver",
"inputs": ["sbom.age_days", "release_channel"],
"formula": "if release_channel=='stable' then age_norm else 0.8*age_norm",
"normalization": "log1p"
}
],
"signing": {
"predicate": "stella.ops/heuristicCatalog@v1",
"dsse_required": true
}
}
```
## Golden Fixtures (layout)
- `docs/modules/signals/heuristics/fixtures/heur.callgraph.hotpath/{input.json,expected.json}`
- `docs/modules/signals/heuristics/fixtures/heur.pkg.sbom_age/{input.json,expected.json}`
- `expected.json` must be canonicalized (sorted keys, fixed 3dp) and include explanation block.
## CI / Determinism Checks
- Lint: reject heuristics without band or DSSE signature.
- Determinism test: run golden fixtures; fail if output hash differs.
- Drift alert: compare live scores vs golden baselines; trigger if >1 band difference for same input hash.
## Publish Steps (12-05)
1) Finalize catalog + schema; canonicalize via JCS; sign DSSE envelope.
2) Populate fixtures and compute `SHA256SUMS` for all files.
3) Update sprint doc status and Evidence Locker with catalog + fixtures + signatures.
4) Enable observability dashboards and waiver policy checks.
## Open Items
- Confirm minimum band allowed for non-SLA items (proposal: bronze acceptable, but not for SLA).
- Decide on additional heuristics for runtime traces vs SBOM freshness.