git.stella-ops.org/docs/modules/scanner/determinism-score.md

Scanner Determinism Score Guide

Status: Stable (2025-11)
Owners: Scanner Guild · QA Guild · DevEx/CLI Guild · DevOps Guild

1. Goal

Quantify how repeatable a scanner release is by re-running scans under frozen conditions and reporting the ratio of bit-for-bit identical outputs. The determinism score lets customers and auditors confirm that Stella Ops scans are replayable and trustworthy.

2. Test harness overview (SCAN-DETER-186-009)

1. Inputs: image digests, policy bundle SHA, feed snapshot SHA, scanner container digest, platform (linux/amd64 by default).
2. Execution loop: run the scanner N times (default 10) with:
   • --fixed-clock <timestamp>
   • RNG_SEED=1337
   • SCANNER_MAX_CONCURRENCY=1
   • feeds/policy tarballs mounted read-only
   • --network=none, --cpuset-cpus=0, --memory=2G
3. Canonicalisation: normalise JSON outputs (SBOM, VEX, findings, logs) using the same serializer as production (StellaOps.Scanner.Replay helpers).
4. Hashing: compute SHA-256 for each canonical artefact per run.
5. Score calculation: identical_runs / total_runs (per image and overall). A run is “identical” if all artefact hashes match the baseline (run 1).

The harness persists the full run set under CAS, allowing regression tests and Offline kit inclusion.

3. determinism.json schema (SCAN-DETER-186-010)

Required fields:

• version (string, default 1)
• release (string; e.g., scanner-0.14.3)
• platform (os/arch)
• policy_sha, feeds_sha, scanner_sha (content-addressed digests)
• images[]:
  • digest (image digest)
  • runs (int), identical (int), score (float 0–1)
  • artifact_hashes (map filename → sha256)
  • non_deterministic (array of artefact names when diverged)
  • notes (optional string)
• overall_score (float 0–1)
• thresholds { image_min:0.90, overall_min:0.95 }
• generated_at (UTC ISO-8601)

Output bundle layout:

• determinism.json – schema above, includes per-run artefact hashes and determinism pins (feed/policy/tool) plus runtime toggles.
• run_i/*.json – canonicalised artefacts per run
• diffs/ – minimal diffs when divergence occurs
• surface/determinism.json – copy of the worker-emitted determinism manifest from the surface bundle (pins + payload hashes) for cross-checking.

4. CI integration (DEVOPS-SCAN-90-004)

• GitHub/Gitea pipeline stages run the determinism harness for the release matrix.
• Fail the job when overall_score < threshold (default 0.95) or any image falls below 0.90.
• Upload determinism.json and artefacts as build outputs; attach to release notes and Offline kits.

5. CLI support (CLI-DETER-70-003/004)

• stella detscore run – executes the harness locally, honoring the same frozen-clock and seed settings; exits non-zero when score falls below the configured threshold.
• stella detscore report – summarises one or more determinism.json files for release notes, showing per-image scores and detection of non-deterministic artefacts.

6. Policy & UI consumption

• Policy Engine can enforce determinism thresholds (e.g., block promotion if score < 0.95) using the determinism.json evidence.
• UI surfaces the score alongside scans (e.g., badge in scan detail view) referencing task UI-SBOM-DET-01.

7. Evidence & replay

Replay instructions (deterministic, offline-friendly):

1. Fetch a published determinism bundle (determinism.json + run_*) and the corresponding scanner image.
2. Run scripts/scanner/determinism-run.sh --replay --bundle path/to/bundle --image sha256:... (uses fixed clock/seed).
3. Verify hashes match the recorded artifact_hashes; tool exits non-zero on mismatch.
4. Attach the bundle to Evidence Locker with DSSE signature before export.

8. Implementation checklist

Area	Task ID	Notes
Harness	SCAN-DETER-186-009	Deterministic execution + hashing
Artefacts	SCAN-DETER-186-010	Publish JSON, CAS storage
CLI	CLI-DETER-70-003/004	Local runs + reporting
DevOps	DEVOPS-SCAN-90-004	CI enforcement
Docs	DOCS-DETER-70-002	(this document)

Update this guide with links to code once tasks move to DONE.