# VEX Evidence Playbook (Bench Repo Blueprint)

> **Status:** Draft – aligns with the “provable vulnerability decisions” advisory (Nov 2025).  
> **Owners:** Policy Guild · VEX Lens Guild · CLI Guild · Docs Guild.

This playbook defines the public benchmark repository layout, artifact shapes, verification tooling, and metrics that prove Stella Ops VEX decisions are reproducible, portable, and superior to baseline scanners. Treat it as the contract for every guild contributing artifacts to `bench/`.

---

## 1. Repository layout

```
bench/
  README.md                  # repo overview + quickstart
  findings/
    CVE-YYYY-NNNNN/          # one folder per advisory/product tuple
      evidence/
        reachability.json    # static+runtime call graph for the finding
        sbom.cdx.json        # CycloneDX slice containing the involved components
      decision.openvex.json  # OpenVEX statement (status + justification)
      decision.dsse.json     # DSSE envelope wrapping the OpenVEX payload
      rekor.txt              # optional Rekor UUID/index/checkpoint
      metadata.json          # producer info (policy rev, analyzer digests, CAS URIs)
  tools/
    verify.sh                # shell helper: dsse verify + optional rekor verification
    verify.py                # python verifier (offline) that recomputes digests
    compare.py               # baseline diff against Trivy/Syft/Grype/Snyk/Xray outputs
    replay.sh                # reruns reachability graphs via `stella replay`
  results/
    summary.csv              # FP reduction, MTTD, reproducibility metrics
    runs/2025-11-10/         # pinned scanner/policy versions + raw outputs
      stella/
        findings.json
        runtime-facts.ndjson
        reachability.manifest.json
      trivy/
        findings.json
      ...
```

### File contracts

- `reachability.json` is the canonical export from `cas://reachability/graphs/...` with symbol IDs, call edges, runtime hits, analyzer fingerprints, and CAS references.
- `decision.openvex.json` follows OpenVEX v1 with Stella Ops-specific `status_notes`, `justification`, `impact_statement`, and `action_statement` text.
- `decision.dsse.json` is the DSSE envelope returned by Signer (see §3). Always include the PEM cert chain (keyless) or KMS key id.
- `rekor.txt` captures `{uuid, logIndex, checkpoint}` from Attestor when the decision is logged to Rekor.
- `metadata.json` binds the DSSE payload back to internal evidence: `{policy_revision, reachability_graph_sha256, runtime_trace_sha256, evidence_cas_uri[], analyzer_versions[], createdBy, createdAt}`.

---

## 2. Evidence production flow

1. **Scanner Worker**
   - Generate `reachability.json` + `sbom.cdx.json` per prioritized CVE.
   - Store artifacts under CAS and surface URIs via `ReachabilityReplayWriter`.
2. **Policy Engine / VEXer**
   - Evaluate reachability states + policy lattice to produce an OpenVEX statement.
   - Persist `decision.openvex.json` and forward it to Signer.
3. **Signer & Attestor**
   - Sign the OpenVEX payload via DSSE (`payloadType: application/vnd.in-toto+json`) and return `decision.dsse.json`.
   - Optionally call Attestor to log the DSSE bundle to Rekor; write `{uuid, logIndex, checkpoint}` to `rekor.txt`.
4. **Bench harness**
   - Collect SBOM slice, reachability proof, OpenVEX, DSSE, Rekor metadata, and companion metrics into `bench/findings/CVE-...`.
   - Record tool versions + CAS digests under `metadata.json`.

All steps must be deterministic: repeated scans with the same inputs produce identical artifacts and digests.

---

## 3. Signing & transparency requirements

| Artifact                | Producer      | Format                                | Notes |
|-------------------------|---------------|----------------------------------------|-------|
| Reachability evidence   | Scanner       | Canonical JSON (sorted keys)           | CAS URI recorded in metadata. |
| SBOM slice              | Scanner       | CycloneDX 1.6 JSON                     | Keep only components relevant to the finding. |
| OpenVEX decision        | Policy/VEXer  | OpenVEX v1                             | One statement per `(CVE, product)` tuple. |
| DSSE bundle             | Signer        | DSSE envelope over OpenVEX payload     | Include Fulcio cert or KMS key id. |
| Rekor record (optional) | Attestor      | Rekor UUID/index/checkpoint            | Store alongside DSSE for offline verification. |

Signer must expose a predicate alias `stella.ops/vexDecision@v1` (see Sprint task `SIGN-VEX-401-018`). Payload = OpenVEX JSON. Rekor logging reuses the existing Attestor `/rekor/entries` pipeline.

---

## 4. Verification tooling

The repo ships two verifiers:

1. `tools/verify.sh` (bash) — wraps `cosign verify-attestation`/`in-toto verify`, Rekor inclusion checks (`rekor-cli logproof`), and digest comparison.
2. `tools/verify.py` — pure-Python offline verifier for air-gapped environments:
   - Validates DSSE signature using the embedded Fulcio cert or configured root.
   - Recomputes `sha256` over `reachability.json`, `sbom.cdx.json`, and `decision.openvex.json` to ensure the DSSE payload matches.
   - Optionally replays reachability by invoking `stella replay --manifest ... --finding CVE-...`.

CLI addition (`stella decision verify`) should shell out to these helpers when `--from bench` is provided.

---

## 5. Metrics & comparison harness

`tools/compare.py` ingests raw outputs from Stella Ops and baseline scanners (Trivy, Syft, Grype, Snyk, Xray) stored under `results/runs/<date>/<scanner>/findings.json`. For each target:

- **False-positive reduction (FPR)** = `1 - (# of findings confirmed true positives / # of baseline findings)`.
- **Mean time to decision (MTTD)** = average wall-clock time between scan start and DSSE-signed OpenVEX emission.
- **Reproducibility score** = `1` if re-running reachability produces identical digests for all artifacts, else `0`; aggregated per run.

`results/summary.csv` columns:

```
target,cve,baseline_scanner,baseline_hits,stella_hits,fp_reduction,mttd_seconds,reproducible,rekor_uuid
```

Automate collection via `Makefile` or `bench/run.sh` pipeline (task `BENCH-AUTO-401-019`).

---

## 6. Publication & README checklist

`bench/README.md` must include:

- High-level workflow diagram (scan → reachability → OpenVEX → DSSE → Rekor → bench).
- Prerequisites (`cosign`, `rekor-cli`, `stella` CLI).
- Quickstart commands:
  ```bash
  ./tools/verify.sh CVE-2023-12345 pkg:purl/example@1.2.3
  ./tools/compare.py --target sample/nginx --baseline trivy --run 2025-11-10
  ```
- How to recreate a finding: `stella replay --manifest results/runs/.../replay.yaml --finding CVE-...`.
- Contribution guide (where to place new findings, how to update metrics, required metadata).

---

## 7. Implementation tasks (see Sprint 401+)

- `POLICY-VEX-401-010` — emit OpenVEX per finding and publish to bench repo.
- `SIGN-VEX-401-018` — add DSSE predicate + Rekor logging for decision payloads.
- `CLI-VEX-401-011` — new `stella decision` verbs (`export`, `verify`, `compare`).
- `BENCH-AUTO-401-019` — automation to populate `bench/findings/**`, run baseline scanners, and update `results/summary.csv`.
- `DOCS-VEX-401-012` — maintain this playbook + README templates, document verification workflow.

Update `docs/implplan/SPRINT_0401_0001_0001_reachability_evidence_chain.md` whenever these tasks move state.