Files

master c58a236d70 Doctor plugin checks: implement health check classes and documentation

Implement remediation-aware health checks across all Doctor plugin modules
(Agent, Attestor, Auth, BinaryAnalysis, Compliance, Crypto, Environment,
EvidenceLocker, Notify, Observability, Operations, Policy, Postgres, Release,
Scanner, Storage, Vex) and their backing library counterparts (AI, Attestation,
Authority, Core, Cryptography, Database, Docker, Integration, Notify,
Observability, Security, ServiceGraph, Sources, Verification).

Each check now emits structured remediation metadata (severity, category,
runbook links, and fix suggestions) consumed by the Doctor dashboard
remediation panel.

Also adds:
- docs/doctor/articles/ knowledge base for check explanations
- Advisory AI search seed and allowlist updates for doctor content
- Sprint plan for doctor checks documentation

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

2026-03-27 12:28:00 +02:00

2.8 KiB

Raw Blame History

checkId, plugin, severity, tags

checkId

plugin

severity

KPI Baseline Configuration

What It Checks

Verifies that a KPI baseline file exists for regression detection in CI gates. The check:

Looks for a baseline file at the configured directory (default /var/lib/stella/baselines) and filename (default current.json), configurable via BinaryAnalysis:Corpus:BaselineDirectory and BinaryAnalysis:Corpus:BaselineFilename.
If the directory does not exist, warns.
If the default baseline file is missing but other .json files exist in the directory, warns and identifies the latest one.
Validates the baseline file as JSON and checks for expected KPI fields: precision, recall, falseNegativeRate, deterministicReplayRate, ttfrpP95Ms.
Fails if the file exists but is invalid JSON or has no recognized KPI fields.
Warns if some recommended fields are missing.

Why It Matters

Without a KPI baseline, CI gates cannot detect regressions in binary matching accuracy. A regression in precision or recall means vulnerability detection quality has degraded without anyone being alerted. The baseline enables automated quality gates that block releases when binary analysis accuracy drops.

Common Causes

KPI baseline has never been established (first run of corpus validation not yet completed)
Baseline directory path misconfigured
Baseline file was deleted or corrupted
Baseline created with an older tool version missing newer KPI fields

How to Fix

Docker Compose

# Create baseline directory
docker exec <binaryindex-container> mkdir -p /var/lib/stella/baselines

# Run corpus validation to establish baseline
docker exec <binaryindex-container> stella groundtruth validate run \
  --corpus datasets/golden-corpus/seed/ --output-baseline

Bare Metal / systemd

sudo mkdir -p /var/lib/stella/baselines

# Run validation and save baseline
stella groundtruth validate run \
  --corpus datasets/golden-corpus/seed/ \
  --output /var/lib/stella/baselines/current.json

# Or promote latest results
stella groundtruth baseline update --from-latest \
  --output /var/lib/stella/baselines/current.json

Kubernetes / Helm

binaryAnalysis:
  corpus:
    baselineDirectory: "/var/lib/stella/baselines"
    persistence:
      enabled: true

Run a one-time job to establish the baseline:

kubectl exec -it <binaryindex-pod> -- stella groundtruth validate run --output-baseline

Verification

stella doctor run --check check.binaryanalysis.corpus.kpi.baseline

check.binaryanalysis.corpus.mirror.freshness — verifies corpus mirror data is not stale
check.binaryanalysis.symbol.recovery.fallback — meta-check for symbol recovery availability

2.8 KiB Raw Blame History