feat: Add UI benchmark driver and scenarios for graph interactions
Some checks failed
AOC Guard CI / aoc-guard (push) Has been cancelled
AOC Guard CI / aoc-verify (push) Has been cancelled
Docs CI / lint-and-preview (push) Has been cancelled
Policy Lint & Smoke / policy-lint (push) Has been cancelled
devportal-offline / build-offline (push) Has been cancelled

- Introduced `ui_bench_driver.mjs` to read scenarios and fixture manifest, generating a deterministic run plan.
- Created `ui_bench_plan.md` outlining the purpose, scope, and next steps for the benchmark.
- Added `ui_bench_scenarios.json` containing various scenarios for graph UI interactions.
- Implemented tests for CLI commands, ensuring bundle verification and telemetry defaults.
- Developed schemas for orchestrator components, including replay manifests and event envelopes.
- Added mock API for risk management, including listing and statistics functionalities.
- Implemented models for risk profiles and query options to support the new API.
This commit is contained in:
StellaOps Bot
2025-12-02 01:28:17 +02:00
parent 909d9b6220
commit 44171930ff
94 changed files with 3606 additions and 271 deletions

View File

@@ -0,0 +1,8 @@
170892f6a48b0aef6f426ea97a86f6cd4420bc52634f12a92f72e20f0fa12e29 decay/confidence_decay_config.yaml
450675035928e4771cca1b9e5f9e42035dbe10b3de7b66a4077a7b729b2c5b13 unknowns/unknowns_scoring_manifest.json
e33fa0963493252a5ac379a12f820f6b356ea94310afd1db9ad7394e8307000e heuristics/heuristics.catalog.json
6e2e6dfeeb4ae016b7ae881d4653ab79a3babba28a4c6d072266e81c61366e2c heuristics/heuristics.schema.json
99b1a4abc941bea5d4ee6b1dbd6faea37de9c67474bb1bac5f23570a44beff17 heuristics/fixtures/heur.callgraph.hotpath/input.json
851452aeac775e71d7507b70e9b7eb05bc619b37ba656a31b53fb3a504fb3b4a heuristics/fixtures/heur.callgraph.hotpath/expected.json
4d0f69b6064df2014f20673feda7881f815b489ab22e0a219301514b766b29d1 heuristics/fixtures/heur.pkg.sbom_age/input.json
bd9e5b2081e7fe9d947e7a44b0c493efd4d8e97d4c13ec4efd577869559fb1c2 heuristics/fixtures/heur.pkg.sbom_age/expected.json

View File

@@ -0,0 +1,79 @@
# Confidence Decay Controls · Signals Runtime
**Compiled:** 2025-12-01 (UTC)
**Scope:** Close U1U10 gaps from `docs/product-advisories/31-Nov-2025 FINDINGS.md` for confidence decay of unknowns/signals.
**Status:** Draft for review on 2025-12-03; to be signed (DSSE) after sign-off.
## Decisions (U1U10)
- **τ governance (U1):** All τ values live in `confidence_decay_config.yaml`, change-controlled via DSSE-signed PRs; allowable τ range 190 days. Changes require dual approval (Signals + Policy), recorded in history.
- **Floor / freeze (U2):** `confidence_floor` per severity; `is_confidence_frozen=true` when SLA-bound or manual pin. Floors: Critical 0.60, High 0.45, Medium 0.30, Low 0.20. Freeze auto-expires at `freeze_until`.
- **Weighted signals (U3):** Signal taxonomy with weights: exploit=1.0, customer_incident=0.9, threat_intel=0.7, code_change=0.4, artifact_refresh=0.3, metadata_touch=0.1. `last_signal_weighted_at` uses max(weighted timestamp).
- **Time source / drift (U4):** All timestamps in UTC; decay uses monotonic clock fallback; reject events >5 minutes in the future or >30 days backdated, log corrections.
- **Deterministic recompute (U5):** Nightly job at 03:00 UTC recomputes decay for all items; emits `decay_snapshot_YYYY-MM-DD.ndjson` with SHA256 and checksum record. On-read recompute only if snapshot is older than 24h.
- **SLA coupling (U6):** Items with active SLA clamp to `sla_floor` (0.60 Critical, 0.50 High) until SLA met. SLA flag and floor are emitted in API.
- **Uncertainty linkage (U7):** Confidence is capped by `(1 - uncertainty_score)`; if uncertainty_score ≥0.4, band forced to "under_review" and alerts fire.
- **Backfill & migration (U8):** Initial migration seeds `last_signal_at` from latest activity; default τ from entity profile; dry-run impact report required; backfill script outputs before/after bands.
- **API/UX surfacing (U9):** New fields: `confidence`, `confidence_band` (critical/high/medium/low/under_review), `tau_days`, `is_frozen`, `confidence_floor`, `uncertainty_score`, `last_signal_weighted_at`. Sort default: `priority * confidence`.
- **Observability & alerts (U10):** Counters/gauges: `confidence_recalc_latency`, `items_below_floor`, `signals_weighted_by_type{type}`, `decay_snapshots_age_hours`, `uncertainty_forced_under_review`. Alerts on missing nightly snapshot, decay drift >1 band, or SLA items below floor.
## Reference Config (draft)
```yaml
version: 1
updated_at: 2025-12-01T00:00:00Z
entities:
vulnerability:
tau_days: 21
tau_min: 7
tau_max: 90
confidence_floor: {critical: 0.60, high: 0.45, medium: 0.30, low: 0.20}
sla_floor: {critical: 0.60, high: 0.50}
freeze_default_days: 30
incident:
tau_days: 14
tau_min: 3
tau_max: 60
signals_taxonomy:
exploit: 1.0
customer_incident: 0.9
threat_intel: 0.7
code_change: 0.4
artifact_refresh: 0.3
metadata_touch: 0.1
time:
reject_future_minutes: 5
reject_backdated_days: 30
recompute:
schedule_utc: "03:00"
snapshot_retention_days: 30
observability:
alerts:
missing_snapshot_hours: 26
sla_floor_breach: true
uncertainty_band_force: 0.4
signing:
predicate: stella.ops/confidenceDecayConfig@v1
dsse_required: true
```
## Operational Rules
- Config changes must produce a new DSSE envelope and update the checksum in the nightly snapshot header.
- Nightly job writes `decay_snapshot_<date>.ndjson` (sorted by `item_id`) plus `SHA256SUMS`; both stored in Evidence Locker.
- Any on-read recompute must emit an audit log with reasons (stale snapshot or forced recalculation).
## Migration Playbook
1) Run dry-run backfill: compute bands with proposed config; write `decay_backfill_diff.ndjson` (before/after bands, delta) and checksum.
2) Get dual approval; sign `confidence_decay_config.yaml` with DSSE predicate above.
3) Apply config, execute full recompute, publish snapshot + checksums, update observability dashboard baselines.
## API Notes
- Add fields to Signals API and CLI responses; ensure canonical serialization (sorted keys, UTC timestamps, fixed decimals 3dp) to avoid hash drift.
- Bands map: `>=0.75 critical`, `>=0.55 high`, `>=0.35 medium`, `>=0.20 low`, else `under_review`.
## Evidence & Storage
- Store config DSSE, snapshots, and backfill reports in Evidence Locker with retention class `signals-decay-config`.
- For offline kits, include latest config DSSE + last 3 snapshots and checksums.
## Open Items for Review (12-03)
- Confirm weights for threat_intel vs exploit; adjust if customer data suggests different ordering.
- Confirm `under_review` threshold (currently uncertainty ≥0.4).
- Align with Policy on SLA floors for High severity (0.50 proposed).

View File

@@ -0,0 +1,41 @@
version: 1
updated_at: 2025-12-01T00:00:00Z
entities:
vulnerability:
tau_days: 21
tau_min: 7
tau_max: 90
confidence_floor:
critical: 0.60
high: 0.45
medium: 0.30
low: 0.20
sla_floor:
critical: 0.60
high: 0.50
freeze_default_days: 30
incident:
tau_days: 14
tau_min: 3
tau_max: 60
signals_taxonomy:
exploit: 1.0
customer_incident: 0.9
threat_intel: 0.7
code_change: 0.4
artifact_refresh: 0.3
metadata_touch: 0.1
time:
reject_future_minutes: 5
reject_backdated_days: 30
recompute:
schedule_utc: "03:00"
snapshot_retention_days: 30
observability:
alerts:
missing_snapshot_hours: 26
sla_floor_breach: true
uncertainty_band_force: 0.4
signing:
predicate: stella.ops/confidenceDecayConfig@v1
dsse_required: true

View File

@@ -0,0 +1,66 @@
# Signals Heuristic Catalog · Deterministic Scoring
**Compiled:** 2025-12-01 (UTC)
**Scope:** Close UT1UT10 gaps from `docs/product-advisories/31-Nov-2025 FINDINGS.md` by publishing a signed heuristic catalog and golden outputs.
**Status:** Draft; target publish 2025-12-05 with DSSE signature.
## Decisions (UT1UT10)
- **Signed catalog/schema (UT1):** Catalog lives at `heuristics.catalog.json` with schema versioned `heuristics.schema.json`; DSSE predicate `stella.ops/heuristicCatalog@v1` required.
- **Deterministic scoring formula (UT2):** Each heuristic defines `inputs`, `weights`, and `normalization`; scoring outputs canonicalized (sorted keys, fixed 3dp). Engine must be pure/deterministic; randomization forbidden.
- **Quality bands (UT3):** Bands: `gold` (precision≥0.9, recall≥0.8), `silver` (≥0.8/0.7), `bronze` (≥0.7/0.6). Bands recorded in catalog and enforced in admission checks.
- **Waiver policy with DSSE (UT4):** Waivers require DSSE envelope `stella.ops/heuristicWaiver@v1`, include reason, scope, expiry; dual approval (Signals+Policy).
- **SLA coupling (UT5):** SLA-tagged items cannot use heuristics below `silver`; SLA enforcement checks band before accepting results.
- **Offline kit packaging (UT6):** Catalog, schema, golden fixtures, and DSSE envelopes bundled in offline kits with `SHA256SUMS`.
- **Observability/alerts (UT7):** Metrics: `heuristics_eval_latency`, `heuristics_band_usage`, `heuristics_waivers_total`, `heuristics_score_drift`. Alerts when drift >1 band vs golden fixtures or when waivers exceed threshold.
- **Backfill plan (UT8):** Backfill job recomputes heuristic scores with current catalog; outputs `heuristics_backfill.ndjson` + checksum; mismatches raise alerts.
- **Explainability fields/exports (UT9):** Outputs must include `explanation` block: contributing signals with weights, normalized scores, and rule IDs. CLI/API export supports `--explain` and deterministic ordering.
- **Fixtures with golden outputs (UT10):** Golden set per heuristic under `fixtures/<heuristic>/` containing `input.json`, `expected.json`, and `README`; used in CI for determinism.
## Catalog Structure (draft)
```json
{
"version": "1.0.0",
"updatedAt": "2025-12-01T00:00:00Z",
"heuristics": [
{
"id": "heur.callgraph.hotpath",
"band": "gold",
"inputs": ["callgraph.depth", "callgraph.betweenness"],
"formula": "0.6*depth_norm + 0.4*betweenness_norm",
"normalization": "minmax",
"evidence": ["signals/callgraph"]
},
{
"id": "heur.pkg.sbom_age",
"band": "silver",
"inputs": ["sbom.age_days", "release_channel"],
"formula": "if release_channel=='stable' then age_norm else 0.8*age_norm",
"normalization": "log1p"
}
],
"signing": {
"predicate": "stella.ops/heuristicCatalog@v1",
"dsse_required": true
}
}
```
## Golden Fixtures (layout)
- `docs/modules/signals/heuristics/fixtures/heur.callgraph.hotpath/{input.json,expected.json}`
- `docs/modules/signals/heuristics/fixtures/heur.pkg.sbom_age/{input.json,expected.json}`
- `expected.json` must be canonicalized (sorted keys, fixed 3dp) and include explanation block.
## CI / Determinism Checks
- Lint: reject heuristics without band or DSSE signature.
- Determinism test: run golden fixtures; fail if output hash differs.
- Drift alert: compare live scores vs golden baselines; trigger if >1 band difference for same input hash.
## Publish Steps (12-05)
1) Finalize catalog + schema; canonicalize via JCS; sign DSSE envelope.
2) Populate fixtures and compute `SHA256SUMS` for all files.
3) Update sprint doc status and Evidence Locker with catalog + fixtures + signatures.
4) Enable observability dashboards and waiver policy checks.
## Open Items
- Confirm minimum band allowed for non-SLA items (proposal: bronze acceptable, but not for SLA).
- Decide on additional heuristics for runtime traces vs SBOM freshness.

View File

@@ -0,0 +1,16 @@
{
"heuristicId": "heur.callgraph.hotpath",
"score": 0.472,
"band": "gold",
"explanation": {
"inputs": {
"callgraph.depth": 7,
"callgraph.betweenness": 0.12
},
"normalized": {
"depth_norm": 0.700,
"betweenness_norm": 0.120
},
"formula": "0.6*depth_norm + 0.4*betweenness_norm"
}
}

View File

@@ -0,0 +1,4 @@
{
"callgraph.depth": 7,
"callgraph.betweenness": 0.12
}

View File

@@ -0,0 +1,16 @@
{
"heuristicId": "heur.pkg.sbom_age",
"score": 0.450,
"band": "silver",
"explanation": {
"inputs": {
"sbom.age_days": 45,
"release_channel": "stable"
},
"normalized": {
"age_norm": 0.450,
"release_channel": "stable"
},
"formula": "case release_channel=='stable' then age_norm else 0.8*age_norm"
}
}

View File

@@ -0,0 +1,4 @@
{
"sbom.age_days": 45,
"release_channel": "stable"
}

View File

@@ -0,0 +1,26 @@
{
"version": "1.0.0",
"updatedAt": "2025-12-01T00:00:00Z",
"heuristics": [
{
"id": "heur.callgraph.hotpath",
"band": "gold",
"inputs": ["callgraph.depth", "callgraph.betweenness"],
"formula": "0.6*depth_norm + 0.4*betweenness_norm",
"normalization": "minmax",
"evidence": ["signals/callgraph"]
},
{
"id": "heur.pkg.sbom_age",
"band": "silver",
"inputs": ["sbom.age_days", "release_channel"],
"formula": "case release_channel=='stable' then age_norm else 0.8*age_norm",
"normalization": "log1p",
"evidence": ["sbom/age"]
}
],
"signing": {
"predicate": "stella.ops/heuristicCatalog@v1",
"dsse_required": true
}
}

View File

@@ -0,0 +1,32 @@
{
"$id": "https://stella-ops.org/schemas/heuristics.schema.json",
"type": "object",
"required": ["version", "updatedAt", "heuristics", "signing"],
"properties": {
"version": {"type": "string"},
"updatedAt": {"type": "string", "format": "date-time"},
"heuristics": {
"type": "array",
"items": {
"type": "object",
"required": ["id", "band", "inputs", "formula", "normalization"],
"properties": {
"id": {"type": "string"},
"band": {"enum": ["gold", "silver", "bronze"]},
"inputs": {"type": "array", "items": {"type": "string"}},
"formula": {"type": "string"},
"normalization": {"type": "string"},
"evidence": {"type": "array", "items": {"type": "string"}}
}
}
},
"signing": {
"type": "object",
"required": ["predicate", "dsse_required"],
"properties": {
"predicate": {"type": "string"},
"dsse_required": {"type": "boolean"}
}
}
}
}

View File

@@ -0,0 +1,86 @@
# Unknowns Registry & Scoring Manifest
**Compiled:** 2025-12-01 (UTC)
**Scope:** Close UN1UN10 gaps from `docs/product-advisories/31-Nov-2025 FINDINGS.md` for Unknowns Registry.
**Status:** Draft; review 2025-12-04; DSSE signing required before adoption.
## Decisions (UN1UN10)
- **Canonical schema/enums (UN1):** Unknown types: `vulnerability`, `asset`, `signal`, `evidence-gap`, `policy-gap`. Status enums: `new`, `triaging`, `under_review`, `validated`, `dismissed`. Severity: `critical/high/medium/low/none`.
- **Deterministic scoring manifest (UN2):** Manifest `unknowns_scoring_manifest.json` defines inputs, weights, and canonical serialization (JCS, sorted keys, UTC timestamps, fixed 3dp). Hash used as `scoringManifestHash` in API/DSSE.
- **Decay policy catalog (UN3):** Unknowns reuse `confidence_decay_config` but may override τ by type (see table). Overrides stored in manifest; DSSE-signed.
- **Evidence/provenance capture (UN4):** Each unknown must reference Evidence Locker URIs with DSSE envelopes; minimal evidence: `{source, observedAt, evidenceType, hash}`. Provenance includes tool identity and policy hash.
- **SBOM/VEX linkage (UN5):** Unknown links: `sbomDigest`, `vexDecisionId` (if present), `reachabilityGraphHash`. If absent, status forced to `under_review`.
- **SLA / suppression rules (UN6):** SLA timers mirror severity; suppression requires dual sign-off and DSSE note with expiry. Suppressed items emit `suppression_reason`, `expiresAt`.
- **API/CLI contracts (UN7):** New endpoints `/unknowns` support filter by `status`, `type`, `confidence_band`, `uncertainty_score`, `suppressed`. CLI mirrors with `--format ndjson` and `--include-provenance` flags. Output sorted deterministically by `createdAt, id`.
- **Observability/reporting (UN8):** Metrics: `unknowns_total{type,status}`, `unknowns_suppressed_total`, `unknowns_without_sbom`, `unknowns_without_vex`, `unknowns_confidence_band`, `unknowns_manifest_hash_mismatch`. Alerts on manifest hash mismatch, >1% unknowns missing SBOM/VEX, or suppression expiry.
- **Offline bundle inclusion (UN9):** Include latest manifest, schema, and NDJSON export in offline kit; bundle hashes recorded in kit manifest; verify against DSSE signatures.
- **Migration/backfill (UN10):** Backfill script `backfill_unknowns_v1` seeds `scoringManifestHash`, `sbomDigest`, and `vexDecisionId` from existing records; produces `unknowns_backfill_report.ndjson` with before/after status/bands and checksum.
## Schema (draft)
```json
{
"$id": "https://stella-ops.org/schemas/unknown.json",
"type": "object",
"required": ["id", "type", "status", "severity", "createdAt", "confidence", "confidenceBand"],
"properties": {
"id": {"type": "string"},
"type": {"enum": ["vulnerability", "asset", "signal", "evidence-gap", "policy-gap"]},
"status": {"enum": ["new", "triaging", "under_review", "validated", "dismissed"]},
"severity": {"enum": ["critical", "high", "medium", "low", "none"]},
"confidence": {"type": "number"},
"confidenceBand": {"enum": ["critical", "high", "medium", "low", "under_review"]},
"uncertaintyScore": {"type": "number", "minimum": 0, "maximum": 1},
"tauDays": {"type": "integer"},
"sbomDigest": {"type": "string"},
"vexDecisionId": {"type": "string"},
"reachabilityGraphHash": {"type": "string"},
"scoringManifestHash": {"type": "string"},
"suppression": {
"type": "object",
"properties": {
"isSuppressed": {"type": "boolean"},
"reason": {"type": "string"},
"expiresAt": {"type": "string", "format": "date-time"},
"signedBy": {"type": "string"}
}
},
"evidence": {"type": "array", "items": {"$ref": "#/definitions/evidenceRef"}},
"createdAt": {"type": "string", "format": "date-time"},
"updatedAt": {"type": "string", "format": "date-time"}
},
"definitions": {
"evidenceRef": {
"type": "object",
"required": ["uri", "hash", "observedAt", "evidenceType"],
"properties": {
"uri": {"type": "string"},
"hash": {"type": "string"},
"observedAt": {"type": "string", "format": "date-time"},
"evidenceType": {"type": "string"},
"provenance": {"type": "string"}
}
}
}
}
```
## Scoring Manifest (summary)
- Inputs: severity weight, decay factor (τ), uncertainty cap, SLA floor, suppression flag, weighted signals timestamp.
- Formula (deterministic): `confidence = max(floor, min((exp(-Δt/τ) * weight_signal), 1 - uncertainty))`, then clamp by SLA floor if SLA active.
- Canonicalization: JSON Canonicalization Scheme (JCS); decimals fixed 3dp; UTC ISO-8601 timestamps.
- Hash: SHA256 over canonical manifest; published as `scoringManifestHash` and signed via DSSE `stella.ops/unknownsScoringManifest@v1`.
## Offline & Evidence
- Bundle schema, manifest, and latest NDJSON export with `SHA256SUMS` and DSSE envelope for each artifact.
- Evidence Locker class: `signals-unknowns-manifest` (30d retention minimum).
## Migration Checklist (UN10)
1) Generate `unknowns_scoring_manifest.json` and sign (DSSE).
2) Run `backfill_unknowns_v1 --manifest <hash>`; produce report and checksums.
3) Update API/CLI serializers to include new fields and canonical ordering.
4) Enable observability dashboards and alerts; verify thresholds.
## Review Questions (12-04)
- Confirm suppression expiry default (proposal: 30 days).
- Validate `under_review` trigger when SBOM/VEX missing—keep or allow grace period?
- Align SLA floors with decay config (Critical 0.60, High 0.50).

View File

@@ -0,0 +1,48 @@
{
"version": "1.0.0",
"updatedAt": "2025-12-01T00:00:00Z",
"canonicalization": {
"scheme": "JCS",
"decimals": 3,
"timezone": "UTC"
},
"inputs": {
"severityWeight": {
"critical": 1.0,
"high": 0.8,
"medium": 0.6,
"low": 0.4,
"none": 0.2
},
"uncertaintyCap": 1.0,
"slaFloor": {
"critical": 0.60,
"high": 0.50
},
"tauOverrides": {
"vulnerability": 21,
"asset": 28,
"signal": 14,
"evidence-gap": 21,
"policy-gap": 30
}
},
"formula": "confidence = max(floor, min(exp(-deltaDays/tau) * severityWeight, 1 - uncertainty))",
"floor": 0.20,
"uncertaintyThreshold": 0.4,
"bands": {
"critical": 0.75,
"high": 0.55,
"medium": 0.35,
"low": 0.20,
"under_review": 0.0
},
"hash": {
"algorithm": "SHA-256",
"value": "TO_BE_SIGNED"
},
"signing": {
"predicate": "stella.ops/unknownsScoringManifest@v1",
"dsse_required": true
}
}