Add receipt input JSON and SHA256 hash for CVSS policy scoring tests
- Introduced a new JSON fixture `receipt-input.json` containing base, environmental, and threat metrics for CVSS scoring. - Added corresponding SHA256 hash file `receipt-input.sha256` to ensure integrity of the JSON fixture.
This commit is contained in:
StellaOps Bot
@@ -1,140 +1,42 @@
|
||||
# VEX Evidence Playbook (Bench Repo Blueprint)
|
||||
|
||||
> **Status:** Draft – aligns with the “provable vulnerability decisions” advisory (Nov 2025).
|
||||
> **Owners:** Policy Guild · VEX Lens Guild · CLI Guild · Docs Guild.
|
||||
|
||||
This playbook defines the public benchmark repository layout, artifact shapes, verification tooling, and metrics that prove Stella Ops VEX decisions are reproducible, portable, and superior to baseline scanners. Treat it as the contract for every guild contributing artifacts to `bench/`.
|
||||
|
||||
---
|
||||
|
||||
## 1. Repository layout
|
||||
|
||||
```
|
||||
bench/
|
||||
README.md # repo overview + quickstart
|
||||
findings/
|
||||
CVE-YYYY-NNNNN/ # one folder per advisory/product tuple
|
||||
evidence/
|
||||
reachability.json # static+runtime call graph for the finding
|
||||
sbom.cdx.json # CycloneDX slice containing the involved components
|
||||
decision.openvex.json # OpenVEX statement (status + justification)
|
||||
decision.dsse.json # DSSE envelope wrapping the OpenVEX payload
|
||||
rekor.txt # optional Rekor UUID/index/checkpoint
|
||||
metadata.json # producer info (policy rev, analyzer digests, CAS URIs)
|
||||
tools/
|
||||
verify.sh # shell helper: dsse verify + optional rekor verification
|
||||
verify.py # python verifier (offline) that recomputes digests
|
||||
compare.py # baseline diff against Trivy/Syft/Grype/Snyk/Xray outputs
|
||||
replay.sh # reruns reachability graphs via `stella replay`
|
||||
results/
|
||||
summary.csv # FP reduction, MTTD, reproducibility metrics
|
||||
runs/2025-11-10/ # pinned scanner/policy versions + raw outputs
|
||||
stella/
|
||||
findings.json
|
||||
runtime-facts.ndjson
|
||||
reachability.manifest.json
|
||||
trivy/
|
||||
findings.json
|
||||
...
|
||||
```
|
||||
|
||||
### File contracts
|
||||
|
||||
- `reachability.json` is the canonical export from `cas://reachability/graphs/...` with symbol IDs, call edges, runtime hits, analyzer fingerprints, and CAS references.
|
||||
- `decision.openvex.json` follows OpenVEX v1 with Stella Ops-specific `status_notes`, `justification`, `impact_statement`, and `action_statement` text.
|
||||
- `decision.dsse.json` is the DSSE envelope returned by Signer (see §3). Always include the PEM cert chain (keyless) or KMS key id.
|
||||
- `rekor.txt` captures `{uuid, logIndex, checkpoint}` from Attestor when the decision is logged to Rekor.
|
||||
- `metadata.json` binds the DSSE payload back to internal evidence: `{policy_revision, reachability_graph_sha256, runtime_trace_sha256, evidence_cas_uri[], analyzer_versions[], createdBy, createdAt}`.
|
||||
|
||||
---
|
||||
|
||||
## 2. Evidence production flow
|
||||
|
||||
1. **Scanner Worker**
|
||||
- Generate `reachability.json` + `sbom.cdx.json` per prioritized CVE.
|
||||
- Store artifacts under CAS and surface URIs via `ReachabilityReplayWriter`.
|
||||
2. **Policy Engine / VEXer**
|
||||
- Evaluate reachability states + policy lattice to produce an OpenVEX statement.
|
||||
- Persist `decision.openvex.json` and forward it to Signer.
|
||||
3. **Signer & Attestor**
|
||||
- Sign the OpenVEX payload via DSSE (`payloadType: application/vnd.in-toto+json`) and return `decision.dsse.json`.
|
||||
- Optionally call Attestor to log the DSSE bundle to Rekor; write `{uuid, logIndex, checkpoint}` to `rekor.txt`.
|
||||
4. **Bench harness**
|
||||
- Collect SBOM slice, reachability proof, OpenVEX, DSSE, Rekor metadata, and companion metrics into `bench/findings/CVE-...`.
|
||||
- Record tool versions + CAS digests under `metadata.json`.
|
||||
|
||||
All steps must be deterministic: repeated scans with the same inputs produce identical artifacts and digests.
|
||||
|
||||
---
|
||||
|
||||
## 3. Signing & transparency requirements
|
||||
|
||||
| Artifact | Producer | Format | Notes |
|
||||
|-------------------------|---------------|----------------------------------------|-------|
|
||||
| Reachability evidence | Scanner | Canonical JSON (sorted keys) | CAS URI recorded in metadata. |
|
||||
| SBOM slice | Scanner | CycloneDX 1.6 JSON | Keep only components relevant to the finding. |
|
||||
| OpenVEX decision | Policy/VEXer | OpenVEX v1 | One statement per `(CVE, product)` tuple. |
|
||||
| DSSE bundle | Signer | DSSE envelope over OpenVEX payload | Include Fulcio cert or KMS key id. |
|
||||
| Rekor record (optional) | Attestor | Rekor UUID/index/checkpoint | Store alongside DSSE for offline verification. |
|
||||
|
||||
Signer must expose a predicate alias `stella.ops/vexDecision@v1` (see Sprint task `SIGN-VEX-401-018`). Payload = OpenVEX JSON. Rekor logging reuses the existing Attestor `/rekor/entries` pipeline.
|
||||
|
||||
---
|
||||
|
||||
## 4. Verification tooling
|
||||
|
||||
The repo ships two verifiers:
|
||||
|
||||
1. `tools/verify.sh` (bash) — wraps `cosign verify-attestation`/`in-toto verify`, Rekor inclusion checks (`rekor-cli logproof`), and digest comparison.
|
||||
2. `tools/verify.py` — pure-Python offline verifier for air-gapped environments:
|
||||
- Validates DSSE signature using the embedded Fulcio cert or configured root.
|
||||
- Recomputes `sha256` over `reachability.json`, `sbom.cdx.json`, and `decision.openvex.json` to ensure the DSSE payload matches.
|
||||
- Optionally replays reachability by invoking `stella replay --manifest ... --finding CVE-...`.
|
||||
|
||||
CLI addition (`stella decision verify`) should shell out to these helpers when `--from bench` is provided.
|
||||
|
||||
---
|
||||
|
||||
## 5. Metrics & comparison harness
|
||||
|
||||
`tools/compare.py` ingests raw outputs from Stella Ops and baseline scanners (Trivy, Syft, Grype, Snyk, Xray) stored under `results/runs/<date>/<scanner>/findings.json`. For each target:
|
||||
|
||||
- **False-positive reduction (FPR)** = `1 - (# of findings confirmed true positives / # of baseline findings)`.
|
||||
- **Mean time to decision (MTTD)** = average wall-clock time between scan start and DSSE-signed OpenVEX emission.
|
||||
- **Reproducibility score** = `1` if re-running reachability produces identical digests for all artifacts, else `0`; aggregated per run.
|
||||
|
||||
`results/summary.csv` columns:
|
||||
|
||||
```
|
||||
target,cve,baseline_scanner,baseline_hits,stella_hits,fp_reduction,mttd_seconds,reproducible,rekor_uuid
|
||||
```
|
||||
|
||||
Automate collection via `Makefile` or `bench/run.sh` pipeline (task `BENCH-AUTO-401-019`).
|
||||
|
||||
---
|
||||
|
||||
## 6. Publication & README checklist
|
||||
|
||||
`bench/README.md` must include:
|
||||
|
||||
- High-level workflow diagram (scan → reachability → OpenVEX → DSSE → Rekor → bench).
|
||||
- Prerequisites (`cosign`, `rekor-cli`, `stella` CLI).
|
||||
- Quickstart commands:
|
||||
```bash
|
||||
./tools/verify.sh CVE-2023-12345 pkg:purl/example@1.2.3
|
||||
./tools/compare.py --target sample/nginx --baseline trivy --run 2025-11-10
|
||||
```
|
||||
- How to recreate a finding: `stella replay --manifest results/runs/.../replay.yaml --finding CVE-...`.
|
||||
- Contribution guide (where to place new findings, how to update metrics, required metadata).
|
||||
|
||||
---
|
||||
|
||||
## 7. Implementation tasks (see Sprint 401+)
|
||||
|
||||
- `POLICY-VEX-401-010` — emit OpenVEX per finding and publish to bench repo.
|
||||
- `SIGN-VEX-401-018` — add DSSE predicate + Rekor logging for decision payloads.
|
||||
- `CLI-VEX-401-011` — new `stella decision` verbs (`export`, `verify`, `compare`).
|
||||
- `BENCH-AUTO-401-019` — automation to populate `bench/findings/**`, run baseline scanners, and update `results/summary.csv`.
|
||||
- `DOCS-VEX-401-012` — maintain this playbook + README templates, document verification workflow.
|
||||
|
||||
Update `docs/implplan/SPRINT_0401_0001_0001_reachability_evidence_chain.md` whenever these tasks move state.
|
||||
# VEX Evidence Playbook (VEX1–VEX10)
|
||||
|
||||
Status: Draft · Date: 2025-12-03
|
||||
Scope: Define deterministic VEX evidence bundles, justification catalog, and verification workflow for reachability evidence chain (Sprint 0401 tasks 62, 64, 65).
|
||||
|
||||
## Goals
|
||||
- Publish signed VEX justification catalog with DSSE predicates and canonical hashes.
|
||||
- Require entry-point coverage %, negative tests, and config/flag hash enforcement for each justification.
|
||||
- Provide offline-friendly proof bundle format with CAS paths and DSSE envelopes.
|
||||
|
||||
## Bundle schema (draft)
|
||||
- `proofBundle.schema.json` (to be placed under `docs/benchmarks/vex-evidence-playbook.schema.json`):
|
||||
- `id`, `version`, `createdAt`, `createdBy`.
|
||||
- `graph_hash` (BLAKE3-256), `graph_dsse` (sha256 of envelope).
|
||||
- `entrypoints[]` with `id`, `coverage_percent`, `negative_tests` (bool), `config_hash` (sha256), `flags_hash` (sha256).
|
||||
- `justification_id` (must exist in justification catalog), `justification_dsse` (optional hash).
|
||||
- `evidence[]` items with `type` (`trace`, `coverage`, `sbom`, `scan`, `policy`), `cas_uri`, `hash`, `dsse` (optional), `expiresAt`.
|
||||
- `signatures[]` DSSE/JWS entries with `keyId`, `sig`, `envelopeDigest`.
|
||||
|
||||
## Justification catalog
|
||||
- Canonical JSON, JCS normalized; BLAKE3 + SHA-256 recorded.
|
||||
- Fields: `id`, `title`, `description`, `applicability` (list of predicates), `required_evidence` (array of types/hashes), `expiry`, `policy_links`.
|
||||
- Signed with DSSE predicate `stella.ops/vexJustification@v1`.
|
||||
|
||||
## Determinism rules
|
||||
- Canonical JSON with sorted keys; arrays sorted by `id`.
|
||||
- Timestamps UTC `Z`; strip milliseconds unless non-zero.
|
||||
- Hashes: primary BLAKE3-256, secondary SHA-256 for interoperability.
|
||||
- DSSE subject = canonical JSON hash; verifier must check hash + signature.
|
||||
|
||||
## Offline verification
|
||||
- `scripts/vex/verify_proof_bundle.py` (to be authored) validates schema, hashes, DSSE, and CAS availability.
|
||||
- No network calls; CAS paths resolved locally via `--cas-root`.
|
||||
|
||||
## Fixtures (to add)
|
||||
- `tests/Vex/ProofBundles/sample-proof-bundle.json` with matching DSSE.
|
||||
- `docs/benchmarks/vex-justifications.catalog.json` with 5 sample justifications (VEX1–VEX5).
|
||||
|
||||
## Next steps
|
||||
- Freeze `proofBundle.schema.json` and justification catalog hashes.
|
||||
- Implement `verify_proof_bundle.py` and add CI job to run it over fixtures.
|
||||
- Wire sprint 0401 tasks 62/64/65 to these artifacts; update Decisions & Risks once frozen.
|
||||
|
||||
Reference in New Issue
Block a user