Signals Heuristic Catalog · Deterministic Scoring

Compiled: 2025-12-01 (UTC) Scope: Close UT1–UT10 gaps from docs/product/advisories/31-Nov-2025 FINDINGS.md by publishing a signed heuristic catalog and golden outputs. Status: Draft; target publish 2025-12-05 with DSSE signature.

Decisions (UT1–UT10)

Signed catalog/schema (UT1): Catalog lives at heuristics.catalog.json with schema versioned heuristics.schema.json; DSSE predicate stella.ops/heuristicCatalog@v1 required.
Deterministic scoring formula (UT2): Each heuristic defines inputs, weights, and normalization; scoring outputs canonicalized (sorted keys, fixed 3dp). Engine must be pure/deterministic; randomization forbidden.
Quality bands (UT3): Bands: gold (precision≥0.9, recall≥0.8), silver (≥0.8/0.7), bronze (≥0.7/0.6). Bands recorded in catalog and enforced in admission checks.
Waiver policy with DSSE (UT4): Waivers require DSSE envelope stella.ops/heuristicWaiver@v1, include reason, scope, expiry; dual approval (Signals+Policy).
SLA coupling (UT5): SLA-tagged items cannot use heuristics below silver; SLA enforcement checks band before accepting results.
Offline kit packaging (UT6): Catalog, schema, golden fixtures, and DSSE envelopes bundled in offline kits with SHA256SUMS.
Observability/alerts (UT7): Metrics: heuristics_eval_latency, heuristics_band_usage, heuristics_waivers_total, heuristics_score_drift. Alerts when drift >1 band vs golden fixtures or when waivers exceed threshold.
Backfill plan (UT8): Backfill job recomputes heuristic scores with current catalog; outputs heuristics_backfill.ndjson + checksum; mismatches raise alerts.
Explainability fields/exports (UT9): Outputs must include explanation block: contributing signals with weights, normalized scores, and rule IDs. CLI/API export supports --explain and deterministic ordering.
Fixtures with golden outputs (UT10): Golden set per heuristic under fixtures/<heuristic>/ containing input.json, expected.json, and README; used in CI for determinism.

Catalog Structure (draft)

{
  "version": "1.0.0",
  "updatedAt": "2025-12-01T00:00:00Z",
  "heuristics": [
    {
      "id": "heur.callgraph.hotpath",
      "band": "gold",
      "inputs": ["callgraph.depth", "callgraph.betweenness"],
      "formula": "0.6*depth_norm + 0.4*betweenness_norm",
      "normalization": "minmax",
      "evidence": ["signals/callgraph"]
    },
    {
      "id": "heur.pkg.sbom_age",
      "band": "silver",
      "inputs": ["sbom.age_days", "release_channel"],
      "formula": "if release_channel=='stable' then age_norm else 0.8*age_norm",
      "normalization": "log1p"
    }
  ],
  "signing": {
    "predicate": "stella.ops/heuristicCatalog@v1",
    "dsse_required": true
  }
}

Golden Fixtures (layout)

docs/modules/signals/heuristics/fixtures/heur.callgraph.hotpath/{input.json,expected.json}
docs/modules/signals/heuristics/fixtures/heur.pkg.sbom_age/{input.json,expected.json}
expected.json must be canonicalized (sorted keys, fixed 3dp) and include explanation block.

CI / Determinism Checks

Lint: reject heuristics without band or DSSE signature.
Determinism test: run golden fixtures; fail if output hash differs.
Drift alert: compare live scores vs golden baselines; trigger if >1 band difference for same input hash.

Publish Steps (12-05)

Finalize catalog + schema; canonicalize via JCS; sign DSSE envelope.
Populate fixtures and compute SHA256SUMS for all files.
Update sprint doc status and Evidence Locker with catalog + fixtures + signatures.
Enable observability dashboards and waiver policy checks.

Open Items

Confirm minimum band allowed for non-SLA items (proposal: bronze acceptable, but not for SLA).
Decide on additional heuristics for runtime traces vs SBOM freshness.

3.8 KiB Raw Blame History Unescape Escape