feat: Add UI benchmark driver and scenarios for graph interactions
Some checks failed
AOC Guard CI / aoc-guard (push) Has been cancelled
AOC Guard CI / aoc-verify (push) Has been cancelled
Docs CI / lint-and-preview (push) Has been cancelled
Policy Lint & Smoke / policy-lint (push) Has been cancelled
devportal-offline / build-offline (push) Has been cancelled

- Introduced `ui_bench_driver.mjs` to read scenarios and fixture manifest, generating a deterministic run plan.
- Created `ui_bench_plan.md` outlining the purpose, scope, and next steps for the benchmark.
- Added `ui_bench_scenarios.json` containing various scenarios for graph UI interactions.
- Implemented tests for CLI commands, ensuring bundle verification and telemetry defaults.
- Developed schemas for orchestrator components, including replay manifests and event envelopes.
- Added mock API for risk management, including listing and statistics functionalities.
- Implemented models for risk profiles and query options to support the new API.
This commit is contained in:
StellaOps Bot
2025-12-02 01:28:17 +02:00
parent 909d9b6220
commit 44171930ff
94 changed files with 3606 additions and 271 deletions

View File

@@ -0,0 +1,66 @@
# Signals Heuristic Catalog · Deterministic Scoring
**Compiled:** 2025-12-01 (UTC)
**Scope:** Close UT1UT10 gaps from `docs/product-advisories/31-Nov-2025 FINDINGS.md` by publishing a signed heuristic catalog and golden outputs.
**Status:** Draft; target publish 2025-12-05 with DSSE signature.
## Decisions (UT1UT10)
- **Signed catalog/schema (UT1):** Catalog lives at `heuristics.catalog.json` with schema versioned `heuristics.schema.json`; DSSE predicate `stella.ops/heuristicCatalog@v1` required.
- **Deterministic scoring formula (UT2):** Each heuristic defines `inputs`, `weights`, and `normalization`; scoring outputs canonicalized (sorted keys, fixed 3dp). Engine must be pure/deterministic; randomization forbidden.
- **Quality bands (UT3):** Bands: `gold` (precision≥0.9, recall≥0.8), `silver` (≥0.8/0.7), `bronze` (≥0.7/0.6). Bands recorded in catalog and enforced in admission checks.
- **Waiver policy with DSSE (UT4):** Waivers require DSSE envelope `stella.ops/heuristicWaiver@v1`, include reason, scope, expiry; dual approval (Signals+Policy).
- **SLA coupling (UT5):** SLA-tagged items cannot use heuristics below `silver`; SLA enforcement checks band before accepting results.
- **Offline kit packaging (UT6):** Catalog, schema, golden fixtures, and DSSE envelopes bundled in offline kits with `SHA256SUMS`.
- **Observability/alerts (UT7):** Metrics: `heuristics_eval_latency`, `heuristics_band_usage`, `heuristics_waivers_total`, `heuristics_score_drift`. Alerts when drift >1 band vs golden fixtures or when waivers exceed threshold.
- **Backfill plan (UT8):** Backfill job recomputes heuristic scores with current catalog; outputs `heuristics_backfill.ndjson` + checksum; mismatches raise alerts.
- **Explainability fields/exports (UT9):** Outputs must include `explanation` block: contributing signals with weights, normalized scores, and rule IDs. CLI/API export supports `--explain` and deterministic ordering.
- **Fixtures with golden outputs (UT10):** Golden set per heuristic under `fixtures/<heuristic>/` containing `input.json`, `expected.json`, and `README`; used in CI for determinism.
## Catalog Structure (draft)
```json
{
"version": "1.0.0",
"updatedAt": "2025-12-01T00:00:00Z",
"heuristics": [
{
"id": "heur.callgraph.hotpath",
"band": "gold",
"inputs": ["callgraph.depth", "callgraph.betweenness"],
"formula": "0.6*depth_norm + 0.4*betweenness_norm",
"normalization": "minmax",
"evidence": ["signals/callgraph"]
},
{
"id": "heur.pkg.sbom_age",
"band": "silver",
"inputs": ["sbom.age_days", "release_channel"],
"formula": "if release_channel=='stable' then age_norm else 0.8*age_norm",
"normalization": "log1p"
}
],
"signing": {
"predicate": "stella.ops/heuristicCatalog@v1",
"dsse_required": true
}
}
```
## Golden Fixtures (layout)
- `docs/modules/signals/heuristics/fixtures/heur.callgraph.hotpath/{input.json,expected.json}`
- `docs/modules/signals/heuristics/fixtures/heur.pkg.sbom_age/{input.json,expected.json}`
- `expected.json` must be canonicalized (sorted keys, fixed 3dp) and include explanation block.
## CI / Determinism Checks
- Lint: reject heuristics without band or DSSE signature.
- Determinism test: run golden fixtures; fail if output hash differs.
- Drift alert: compare live scores vs golden baselines; trigger if >1 band difference for same input hash.
## Publish Steps (12-05)
1) Finalize catalog + schema; canonicalize via JCS; sign DSSE envelope.
2) Populate fixtures and compute `SHA256SUMS` for all files.
3) Update sprint doc status and Evidence Locker with catalog + fixtures + signatures.
4) Enable observability dashboards and waiver policy checks.
## Open Items
- Confirm minimum band allowed for non-SLA items (proposal: bronze acceptable, but not for SLA).
- Decide on additional heuristics for runtime traces vs SBOM freshness.

View File

@@ -0,0 +1,16 @@
{
"heuristicId": "heur.callgraph.hotpath",
"score": 0.472,
"band": "gold",
"explanation": {
"inputs": {
"callgraph.depth": 7,
"callgraph.betweenness": 0.12
},
"normalized": {
"depth_norm": 0.700,
"betweenness_norm": 0.120
},
"formula": "0.6*depth_norm + 0.4*betweenness_norm"
}
}

View File

@@ -0,0 +1,4 @@
{
"callgraph.depth": 7,
"callgraph.betweenness": 0.12
}

View File

@@ -0,0 +1,16 @@
{
"heuristicId": "heur.pkg.sbom_age",
"score": 0.450,
"band": "silver",
"explanation": {
"inputs": {
"sbom.age_days": 45,
"release_channel": "stable"
},
"normalized": {
"age_norm": 0.450,
"release_channel": "stable"
},
"formula": "case release_channel=='stable' then age_norm else 0.8*age_norm"
}
}

View File

@@ -0,0 +1,4 @@
{
"sbom.age_days": 45,
"release_channel": "stable"
}

View File

@@ -0,0 +1,26 @@
{
"version": "1.0.0",
"updatedAt": "2025-12-01T00:00:00Z",
"heuristics": [
{
"id": "heur.callgraph.hotpath",
"band": "gold",
"inputs": ["callgraph.depth", "callgraph.betweenness"],
"formula": "0.6*depth_norm + 0.4*betweenness_norm",
"normalization": "minmax",
"evidence": ["signals/callgraph"]
},
{
"id": "heur.pkg.sbom_age",
"band": "silver",
"inputs": ["sbom.age_days", "release_channel"],
"formula": "case release_channel=='stable' then age_norm else 0.8*age_norm",
"normalization": "log1p",
"evidence": ["sbom/age"]
}
],
"signing": {
"predicate": "stella.ops/heuristicCatalog@v1",
"dsse_required": true
}
}

View File

@@ -0,0 +1,32 @@
{
"$id": "https://stella-ops.org/schemas/heuristics.schema.json",
"type": "object",
"required": ["version", "updatedAt", "heuristics", "signing"],
"properties": {
"version": {"type": "string"},
"updatedAt": {"type": "string", "format": "date-time"},
"heuristics": {
"type": "array",
"items": {
"type": "object",
"required": ["id", "band", "inputs", "formula", "normalization"],
"properties": {
"id": {"type": "string"},
"band": {"enum": ["gold", "silver", "bronze"]},
"inputs": {"type": "array", "items": {"type": "string"}},
"formula": {"type": "string"},
"normalization": {"type": "string"},
"evidence": {"type": "array", "items": {"type": "string"}}
}
}
},
"signing": {
"type": "object",
"required": ["predicate", "dsse_required"],
"properties": {
"predicate": {"type": "string"},
"dsse_required": {"type": "boolean"}
}
}
}
}