stella-ops.org/git.stella-ops.org
1
0
Fork 0
Code Issues Pull Requests Actions 16 Releases Wiki Activity
Files
7b01c7d6acd233c1fdfac71cad09fd266be6bfc9
git.stella-ops.org/docs/product-advisories/16-Nov-2026 - validation plan for quiet scans provenance diff-ci.md
master 7b01c7d6ac
Some checks failed
Docs CI / lint-and-preview (push) Has been cancelled
Details
feat: Add comprehensive product advisories for improved scanner functionality 
- Introduced a blueprint for explainable quiet alerts, detailing phases for SBOM, VEX readiness, and attestations.
- Developed a roadmap for deterministic diff-aware rescans, enhancing scanner speed and efficiency.
- Implemented a hash-based SBOM layer cache to optimize container scans by reusing previous results.
- Created a multi-runtime reachability corpus to validate function-level reachability across various programming languages.
- Proposed a stable SBOM model using SPDX 3.0.1 for persistence and CycloneDX 1.6 for interchange.
- Established a validation plan for quiet scans, focusing on provenance and CI integration.
- Documented guidelines for the Findings Ledger module, outlining roles, execution rules, and testing protocols.
2025-11-17 00:09:26 +02:00

4.8 KiB
Raw Blame History

Here’s a practical, plain‑English game plan to validate three big Stella Ops claims—quiet scans, provenance, and diff‑native CI—so you (and auditors/customers) can reproduce the results end‑to‑end.

1) “Explainably quiet by design”

Goal: Fewer false‑alarms, with every suppression justified (reachability/VEX), and every alert deduplicated and actionable.

What to measure

  • Noise rate: total findings vs. actionable (has fix/KB/CWE + reachable or policy‑relevant).
  • Dedup: identical CVE across layers/repos counted once.
  • Explainability: % of findings with a clear path (package → symbol/function → evidence).
  • Suppression justifications: % of suppressed items with VEX reason (not affected, configuration, environment, reachability).

A/B test setup

  • Repos (representative mix): .NET (aspnet app & library), JVM (Spring), Node/TS (Nest), Python (FastAPI), Go (CLI), container base images (Alpine, Debian, Ubuntu), and a known‑noisy mono‑repo.
  • Modes: baseline=no VEX/reach, quiet=reach+VEX+dedup.
  • Metrics capture: emit JSONL per repo with counts and examples.

Minimal harness (pseudo)

# baseline
stella scan repo --out baseline.jsonl --no-reach --no-vex --no-dedup
# quiet
stella scan repo --out quiet.jsonl --reach --vex openvex.json --dedup
stella explain --in quiet.jsonl --evidence callgraph,eventpipe --why > explain.md
stella metrics compare baseline.jsonl quiet.jsonl > ab_summary.md

Pass criteria (suggested)

  • ≥50% reduction in non‑actionable alerts.
  • 100% of suppressions carry VEX+reason.
  • ≥90% of actionable findings link to evidence (reachable symbol or policy gate).

2) “Provenance‑first DevSecOps”

Goal: Ship a verifiable bundle anyone can check offline: SBOM + attestations + transparency‑log proof.

What to export

  • SBOM: CycloneDX 1.6 or SPDX 3.0.1.
  • Provenance attestation: in‑toto/DSSE (builder, materials, recipe, digest).
  • Signatures: Sigstore (cosign) or regional crypto (pluggable).
  • Transparency log receipt: Rekor (or mirror) inclusion proof.
  • Policy snapshot: the exact policy/lattice and feed hashes used.
  • Repro manifest: declarative inputs so scans are replayable.

One‑shot exporter

stella bundle export \
  --sbom cyclonedx.json \
  --attest provenance.intoto.jsonl \
  --sig cosign.sig \
  --rekor-inclusion rekor.json \
  --policy policy.yml \
  --replay manifest.lock.json \
  --out stella-proof-bundle.tgz

Independent verification (clean machine)

stella bundle verify stella-proof-bundle.tgz \
  --check-sig --check-rekor --check-sbom --check-policy --replay
# Output should show digest matches, valid DSSE, Rekor inclusion, and replay parity.

Pass criteria

  • All cryptographic checks pass offline.
  • Replay produces byte‑identical findings set (or a diff limited to time‑varying feeds pinned by hash).

3) “Diff‑native CI for containers”

Goal: Rescan only what changed (layers/deps/policies) with equal detection parity and lower wall‑time.

Test matrix

  • Images: multistage app (runtime+deps), language runtimes (dotnet, jre, node, python), and a “fat” base (ubuntu:XX).
  • Changes: Dockerfile ENV only, add/remove package, patch app DLL/JAR/JS, policy toggle.

Runs

# Full scan
time stella image scan myimg:old > full_old.json
time stella image scan myimg:new > full_new.json

# Diff-aware
time stella image scan myimg:new --diff-from myimg:old --cache .stella-cache > diff_new.json

stella parity check full_new.json diff_new.json > parity.md

Metrics

  • Parity: same actionable findings IDs (allowing dedup).
  • Speedup: (full time) / (diff time).
  • Cache hit ratio: reused layers/components.

Pass criteria

  • 100% actionable parity on modified images.
  • ≥3× faster on typical “small change” commits; no worse than full scan when cache misses.

What you’ll publish (deliverables)

  • VALIDATION_PLAN.md — steps above with fixed seeds (image digests, repo SHAs).
  • harness/ — scripts to run A/B and diff tests, export bundles, and verify.
  • results/YYYY‑MM/ — raw JSONL, parity reports, timing tables, and a 1‑page summary.
  • policy/ — locked policy + feed hashes used in the runs.

Nice‑to‑have extras

  • Reachability/VEX gallery: a few “before/after” call graphs and suppression cards.
  • Auditor mode: stella audit open stella-proof-bundle.tgz → read‑only UI that renders SBOM, VEX, signatures, Rekor proof, and replay log.
  • CI examples: GitLab/GitHub YAML snippets for full vs. diff jobs with caching.

If you want, I can spit out the repo‑ready scaffold (folders, stub scripts, sample policies) tailored to your .NET 10 + Docker setup so you can run this tonight.