feat: Add UI benchmark driver and scenarios for graph interactions
Some checks failed
AOC Guard CI / aoc-guard (push) Has been cancelled
AOC Guard CI / aoc-verify (push) Has been cancelled
Docs CI / lint-and-preview (push) Has been cancelled
Policy Lint & Smoke / policy-lint (push) Has been cancelled
devportal-offline / build-offline (push) Has been cancelled
Some checks failed
AOC Guard CI / aoc-guard (push) Has been cancelled
AOC Guard CI / aoc-verify (push) Has been cancelled
Docs CI / lint-and-preview (push) Has been cancelled
Policy Lint & Smoke / policy-lint (push) Has been cancelled
devportal-offline / build-offline (push) Has been cancelled
- Introduced `ui_bench_driver.mjs` to read scenarios and fixture manifest, generating a deterministic run plan. - Created `ui_bench_plan.md` outlining the purpose, scope, and next steps for the benchmark. - Added `ui_bench_scenarios.json` containing various scenarios for graph UI interactions. - Implemented tests for CLI commands, ensuring bundle verification and telemetry defaults. - Developed schemas for orchestrator components, including replay manifests and event envelopes. - Added mock API for risk management, including listing and statistics functionalities. - Implemented models for risk profiles and query options to support the new API.
This commit is contained in:
@@ -0,0 +1,66 @@
|
||||
# Signals Heuristic Catalog · Deterministic Scoring
|
||||
|
||||
**Compiled:** 2025-12-01 (UTC)
|
||||
**Scope:** Close UT1–UT10 gaps from `docs/product-advisories/31-Nov-2025 FINDINGS.md` by publishing a signed heuristic catalog and golden outputs.
|
||||
**Status:** Draft; target publish 2025-12-05 with DSSE signature.
|
||||
|
||||
## Decisions (UT1–UT10)
|
||||
- **Signed catalog/schema (UT1):** Catalog lives at `heuristics.catalog.json` with schema versioned `heuristics.schema.json`; DSSE predicate `stella.ops/heuristicCatalog@v1` required.
|
||||
- **Deterministic scoring formula (UT2):** Each heuristic defines `inputs`, `weights`, and `normalization`; scoring outputs canonicalized (sorted keys, fixed 3dp). Engine must be pure/deterministic; randomization forbidden.
|
||||
- **Quality bands (UT3):** Bands: `gold` (precision≥0.9, recall≥0.8), `silver` (≥0.8/0.7), `bronze` (≥0.7/0.6). Bands recorded in catalog and enforced in admission checks.
|
||||
- **Waiver policy with DSSE (UT4):** Waivers require DSSE envelope `stella.ops/heuristicWaiver@v1`, include reason, scope, expiry; dual approval (Signals+Policy).
|
||||
- **SLA coupling (UT5):** SLA-tagged items cannot use heuristics below `silver`; SLA enforcement checks band before accepting results.
|
||||
- **Offline kit packaging (UT6):** Catalog, schema, golden fixtures, and DSSE envelopes bundled in offline kits with `SHA256SUMS`.
|
||||
- **Observability/alerts (UT7):** Metrics: `heuristics_eval_latency`, `heuristics_band_usage`, `heuristics_waivers_total`, `heuristics_score_drift`. Alerts when drift >1 band vs golden fixtures or when waivers exceed threshold.
|
||||
- **Backfill plan (UT8):** Backfill job recomputes heuristic scores with current catalog; outputs `heuristics_backfill.ndjson` + checksum; mismatches raise alerts.
|
||||
- **Explainability fields/exports (UT9):** Outputs must include `explanation` block: contributing signals with weights, normalized scores, and rule IDs. CLI/API export supports `--explain` and deterministic ordering.
|
||||
- **Fixtures with golden outputs (UT10):** Golden set per heuristic under `fixtures/<heuristic>/` containing `input.json`, `expected.json`, and `README`; used in CI for determinism.
|
||||
|
||||
## Catalog Structure (draft)
|
||||
```json
|
||||
{
|
||||
"version": "1.0.0",
|
||||
"updatedAt": "2025-12-01T00:00:00Z",
|
||||
"heuristics": [
|
||||
{
|
||||
"id": "heur.callgraph.hotpath",
|
||||
"band": "gold",
|
||||
"inputs": ["callgraph.depth", "callgraph.betweenness"],
|
||||
"formula": "0.6*depth_norm + 0.4*betweenness_norm",
|
||||
"normalization": "minmax",
|
||||
"evidence": ["signals/callgraph"]
|
||||
},
|
||||
{
|
||||
"id": "heur.pkg.sbom_age",
|
||||
"band": "silver",
|
||||
"inputs": ["sbom.age_days", "release_channel"],
|
||||
"formula": "if release_channel=='stable' then age_norm else 0.8*age_norm",
|
||||
"normalization": "log1p"
|
||||
}
|
||||
],
|
||||
"signing": {
|
||||
"predicate": "stella.ops/heuristicCatalog@v1",
|
||||
"dsse_required": true
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Golden Fixtures (layout)
|
||||
- `docs/modules/signals/heuristics/fixtures/heur.callgraph.hotpath/{input.json,expected.json}`
|
||||
- `docs/modules/signals/heuristics/fixtures/heur.pkg.sbom_age/{input.json,expected.json}`
|
||||
- `expected.json` must be canonicalized (sorted keys, fixed 3dp) and include explanation block.
|
||||
|
||||
## CI / Determinism Checks
|
||||
- Lint: reject heuristics without band or DSSE signature.
|
||||
- Determinism test: run golden fixtures; fail if output hash differs.
|
||||
- Drift alert: compare live scores vs golden baselines; trigger if >1 band difference for same input hash.
|
||||
|
||||
## Publish Steps (12-05)
|
||||
1) Finalize catalog + schema; canonicalize via JCS; sign DSSE envelope.
|
||||
2) Populate fixtures and compute `SHA256SUMS` for all files.
|
||||
3) Update sprint doc status and Evidence Locker with catalog + fixtures + signatures.
|
||||
4) Enable observability dashboards and waiver policy checks.
|
||||
|
||||
## Open Items
|
||||
- Confirm minimum band allowed for non-SLA items (proposal: bronze acceptable, but not for SLA).
|
||||
- Decide on additional heuristics for runtime traces vs SBOM freshness.
|
||||
@@ -0,0 +1,16 @@
|
||||
{
|
||||
"heuristicId": "heur.callgraph.hotpath",
|
||||
"score": 0.472,
|
||||
"band": "gold",
|
||||
"explanation": {
|
||||
"inputs": {
|
||||
"callgraph.depth": 7,
|
||||
"callgraph.betweenness": 0.12
|
||||
},
|
||||
"normalized": {
|
||||
"depth_norm": 0.700,
|
||||
"betweenness_norm": 0.120
|
||||
},
|
||||
"formula": "0.6*depth_norm + 0.4*betweenness_norm"
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,4 @@
|
||||
{
|
||||
"callgraph.depth": 7,
|
||||
"callgraph.betweenness": 0.12
|
||||
}
|
||||
@@ -0,0 +1,16 @@
|
||||
{
|
||||
"heuristicId": "heur.pkg.sbom_age",
|
||||
"score": 0.450,
|
||||
"band": "silver",
|
||||
"explanation": {
|
||||
"inputs": {
|
||||
"sbom.age_days": 45,
|
||||
"release_channel": "stable"
|
||||
},
|
||||
"normalized": {
|
||||
"age_norm": 0.450,
|
||||
"release_channel": "stable"
|
||||
},
|
||||
"formula": "case release_channel=='stable' then age_norm else 0.8*age_norm"
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,4 @@
|
||||
{
|
||||
"sbom.age_days": 45,
|
||||
"release_channel": "stable"
|
||||
}
|
||||
26
docs/modules/signals/heuristics/heuristics.catalog.json
Normal file
26
docs/modules/signals/heuristics/heuristics.catalog.json
Normal file
@@ -0,0 +1,26 @@
|
||||
{
|
||||
"version": "1.0.0",
|
||||
"updatedAt": "2025-12-01T00:00:00Z",
|
||||
"heuristics": [
|
||||
{
|
||||
"id": "heur.callgraph.hotpath",
|
||||
"band": "gold",
|
||||
"inputs": ["callgraph.depth", "callgraph.betweenness"],
|
||||
"formula": "0.6*depth_norm + 0.4*betweenness_norm",
|
||||
"normalization": "minmax",
|
||||
"evidence": ["signals/callgraph"]
|
||||
},
|
||||
{
|
||||
"id": "heur.pkg.sbom_age",
|
||||
"band": "silver",
|
||||
"inputs": ["sbom.age_days", "release_channel"],
|
||||
"formula": "case release_channel=='stable' then age_norm else 0.8*age_norm",
|
||||
"normalization": "log1p",
|
||||
"evidence": ["sbom/age"]
|
||||
}
|
||||
],
|
||||
"signing": {
|
||||
"predicate": "stella.ops/heuristicCatalog@v1",
|
||||
"dsse_required": true
|
||||
}
|
||||
}
|
||||
32
docs/modules/signals/heuristics/heuristics.schema.json
Normal file
32
docs/modules/signals/heuristics/heuristics.schema.json
Normal file
@@ -0,0 +1,32 @@
|
||||
{
|
||||
"$id": "https://stella-ops.org/schemas/heuristics.schema.json",
|
||||
"type": "object",
|
||||
"required": ["version", "updatedAt", "heuristics", "signing"],
|
||||
"properties": {
|
||||
"version": {"type": "string"},
|
||||
"updatedAt": {"type": "string", "format": "date-time"},
|
||||
"heuristics": {
|
||||
"type": "array",
|
||||
"items": {
|
||||
"type": "object",
|
||||
"required": ["id", "band", "inputs", "formula", "normalization"],
|
||||
"properties": {
|
||||
"id": {"type": "string"},
|
||||
"band": {"enum": ["gold", "silver", "bronze"]},
|
||||
"inputs": {"type": "array", "items": {"type": "string"}},
|
||||
"formula": {"type": "string"},
|
||||
"normalization": {"type": "string"},
|
||||
"evidence": {"type": "array", "items": {"type": "string"}}
|
||||
}
|
||||
}
|
||||
},
|
||||
"signing": {
|
||||
"type": "object",
|
||||
"required": ["predicate", "dsse_required"],
|
||||
"properties": {
|
||||
"predicate": {"type": "string"},
|
||||
"dsse_required": {"type": "boolean"}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
Reference in New Issue
Block a user