feat: Add UI benchmark driver and scenarios for graph interactions

- Introduced `ui_bench_driver.mjs` to read scenarios and fixture manifest, generating a deterministic run plan. - Created `ui_bench_plan.md` outlining the purpose, scope, and next steps for the benchmark. - Added `ui_bench_scenarios.json` containing various scenarios for graph UI interactions. - Implemented tests for CLI commands, ensuring bundle verification and telemetry defaults. - Developed schemas for orchestrator components, including replay manifests and event envelopes. - Added mock API for risk management, including listing and statistics functionalities. - Implemented models for risk profiles and query options to support the new API.
2025-12-02 01:28:17 +02:00
parent 909d9b6220
commit 44171930ff
94 changed files with 3606 additions and 271 deletions
--- a/docs/modules/signals/unknowns/2025-12-01-unknowns-registry.md
+++ b/docs/modules/signals/unknowns/2025-12-01-unknowns-registry.md
@@ -0,0 +1,86 @@
+# Unknowns Registry & Scoring Manifest
+
+**Compiled:** 2025-12-01 (UTC)
+**Scope:** Close UN1–UN10 gaps from `docs/product-advisories/31-Nov-2025 FINDINGS.md` for Unknowns Registry.
+**Status:** Draft; review 2025-12-04; DSSE signing required before adoption.
+
+## Decisions (UN1–UN10)
+- **Canonical schema/enums (UN1):** Unknown types: `vulnerability`, `asset`, `signal`, `evidence-gap`, `policy-gap`. Status enums: `new`, `triaging`, `under_review`, `validated`, `dismissed`. Severity: `critical/high/medium/low/none`.
+- **Deterministic scoring manifest (UN2):** Manifest `unknowns_scoring_manifest.json` defines inputs, weights, and canonical serialization (JCS, sorted keys, UTC timestamps, fixed 3dp). Hash used as `scoringManifestHash` in API/DSSE.
+- **Decay policy catalog (UN3):** Unknowns reuse `confidence_decay_config` but may override τ by type (see table). Overrides stored in manifest; DSSE-signed.
+- **Evidence/provenance capture (UN4):** Each unknown must reference Evidence Locker URIs with DSSE envelopes; minimal evidence: `{source, observedAt, evidenceType, hash}`. Provenance includes tool identity and policy hash.
+- **SBOM/VEX linkage (UN5):** Unknown links: `sbomDigest`, `vexDecisionId` (if present), `reachabilityGraphHash`. If absent, status forced to `under_review`.
+- **SLA / suppression rules (UN6):** SLA timers mirror severity; suppression requires dual sign-off and DSSE note with expiry. Suppressed items emit `suppression_reason`, `expiresAt`.
+- **API/CLI contracts (UN7):** New endpoints `/unknowns` support filter by `status`, `type`, `confidence_band`, `uncertainty_score`, `suppressed`. CLI mirrors with `--format ndjson` and `--include-provenance` flags. Output sorted deterministically by `createdAt, id`.
+- **Observability/reporting (UN8):** Metrics: `unknowns_total{type,status}`, `unknowns_suppressed_total`, `unknowns_without_sbom`, `unknowns_without_vex`, `unknowns_confidence_band`, `unknowns_manifest_hash_mismatch`. Alerts on manifest hash mismatch, >1% unknowns missing SBOM/VEX, or suppression expiry.
+- **Offline bundle inclusion (UN9):** Include latest manifest, schema, and NDJSON export in offline kit; bundle hashes recorded in kit manifest; verify against DSSE signatures.
+- **Migration/backfill (UN10):** Backfill script `backfill_unknowns_v1` seeds `scoringManifestHash`, `sbomDigest`, and `vexDecisionId` from existing records; produces `unknowns_backfill_report.ndjson` with before/after status/bands and checksum.
+
+## Schema (draft)
+```json
+{
+  "$id": "https://stella-ops.org/schemas/unknown.json",
+  "type": "object",
+  "required": ["id", "type", "status", "severity", "createdAt", "confidence", "confidenceBand"],
+  "properties": {
+    "id": {"type": "string"},
+    "type": {"enum": ["vulnerability", "asset", "signal", "evidence-gap", "policy-gap"]},
+    "status": {"enum": ["new", "triaging", "under_review", "validated", "dismissed"]},
+    "severity": {"enum": ["critical", "high", "medium", "low", "none"]},
+    "confidence": {"type": "number"},
+    "confidenceBand": {"enum": ["critical", "high", "medium", "low", "under_review"]},
+    "uncertaintyScore": {"type": "number", "minimum": 0, "maximum": 1},
+    "tauDays": {"type": "integer"},
+    "sbomDigest": {"type": "string"},
+    "vexDecisionId": {"type": "string"},
+    "reachabilityGraphHash": {"type": "string"},
+    "scoringManifestHash": {"type": "string"},
+    "suppression": {
+      "type": "object",
+      "properties": {
+        "isSuppressed": {"type": "boolean"},
+        "reason": {"type": "string"},
+        "expiresAt": {"type": "string", "format": "date-time"},
+        "signedBy": {"type": "string"}
+      }
+    },
+    "evidence": {"type": "array", "items": {"$ref": "#/definitions/evidenceRef"}},
+    "createdAt": {"type": "string", "format": "date-time"},
+    "updatedAt": {"type": "string", "format": "date-time"}
+  },
+  "definitions": {
+    "evidenceRef": {
+      "type": "object",
+      "required": ["uri", "hash", "observedAt", "evidenceType"],
+      "properties": {
+        "uri": {"type": "string"},
+        "hash": {"type": "string"},
+        "observedAt": {"type": "string", "format": "date-time"},
+        "evidenceType": {"type": "string"},
+        "provenance": {"type": "string"}
+      }
+    }
+  }
+}
+```
+
+## Scoring Manifest (summary)
+- Inputs: severity weight, decay factor (τ), uncertainty cap, SLA floor, suppression flag, weighted signals timestamp.
+- Formula (deterministic): `confidence = max(floor, min((exp(-Δt/τ) * weight_signal), 1 - uncertainty))`, then clamp by SLA floor if SLA active.
+- Canonicalization: JSON Canonicalization Scheme (JCS); decimals fixed 3dp; UTC ISO-8601 timestamps.
+- Hash: SHA256 over canonical manifest; published as `scoringManifestHash` and signed via DSSE `stella.ops/unknownsScoringManifest@v1`.
+
+## Offline & Evidence
+- Bundle schema, manifest, and latest NDJSON export with `SHA256SUMS` and DSSE envelope for each artifact.
+- Evidence Locker class: `signals-unknowns-manifest` (30d retention minimum).
+
+## Migration Checklist (UN10)
+1) Generate `unknowns_scoring_manifest.json` and sign (DSSE).
+2) Run `backfill_unknowns_v1 --manifest <hash>`; produce report and checksums.
+3) Update API/CLI serializers to include new fields and canonical ordering.
+4) Enable observability dashboards and alerts; verify thresholds.
+
+## Review Questions (12-04)
+- Confirm suppression expiry default (proposal: 30 days).
+- Validate `under_review` trigger when SBOM/VEX missing—keep or allow grace period?
+- Align SLA floors with decay config (Critical 0.60, High 0.50).
--- a/docs/modules/signals/unknowns/unknowns_scoring_manifest.json
+++ b/docs/modules/signals/unknowns/unknowns_scoring_manifest.json
@@ -0,0 +1,48 @@
+{
+  "version": "1.0.0",
+  "updatedAt": "2025-12-01T00:00:00Z",
+  "canonicalization": {
+    "scheme": "JCS",
+    "decimals": 3,
+    "timezone": "UTC"
+  },
+  "inputs": {
+    "severityWeight": {
+      "critical": 1.0,
+      "high": 0.8,
+      "medium": 0.6,
+      "low": 0.4,
+      "none": 0.2
+    },
+    "uncertaintyCap": 1.0,
+    "slaFloor": {
+      "critical": 0.60,
+      "high": 0.50
+    },
+    "tauOverrides": {
+      "vulnerability": 21,
+      "asset": 28,
+      "signal": 14,
+      "evidence-gap": 21,
+      "policy-gap": 30
+    }
+  },
+  "formula": "confidence = max(floor, min(exp(-deltaDays/tau) * severityWeight, 1 - uncertainty))",
+  "floor": 0.20,
+  "uncertaintyThreshold": 0.4,
+  "bands": {
+    "critical": 0.75,
+    "high": 0.55,
+    "medium": 0.35,
+    "low": 0.20,
+    "under_review": 0.0
+  },
+  "hash": {
+    "algorithm": "SHA-256",
+    "value": "TO_BE_SIGNED"
+  },
+  "signing": {
+    "predicate": "stella.ops/unknownsScoringManifest@v1",
+    "dsse_required": true
+  }
+}