dc7c75b496
Some checks failed
Docs CI / lint-and-preview (push) Has been cancelled
feat: Implement BsonJsonConverter for converting BsonDocument and BsonArray to JSON fix: Update project file to include MongoDB.Bson package test: Add GraphOverlayExporterTests to validate NDJSON export functionality refactor: Refactor Program.cs in Attestation Tool for improved argument parsing and error handling docs: Update README for stella-forensic-verify with usage instructions and exit codes feat: Enhance HmacVerifier with clock skew and not-after checks feat: Add MerkleRootVerifier and ChainOfCustodyVerifier for additional verification methods fix: Update DenoRuntimeShim to correctly handle file paths feat: Introduce ComposerAutoloadData and related parsing in ComposerLockReader test: Add tests for Deno runtime execution and verification test: Enhance PHP package tests to include autoload data verification test: Add unit tests for HmacVerifier and verification logic
141 lines
7.3 KiB
Markdown
141 lines
7.3 KiB
Markdown
# VEX Evidence Playbook (Bench Repo Blueprint)
|
||
|
||
> **Status:** Draft – aligns with the “provable vulnerability decisions” advisory (Nov 2025).
|
||
> **Owners:** Policy Guild · VEX Lens Guild · CLI Guild · Docs Guild.
|
||
|
||
This playbook defines the public benchmark repository layout, artifact shapes, verification tooling, and metrics that prove Stella Ops VEX decisions are reproducible, portable, and superior to baseline scanners. Treat it as the contract for every guild contributing artifacts to `bench/`.
|
||
|
||
---
|
||
|
||
## 1. Repository layout
|
||
|
||
```
|
||
bench/
|
||
README.md # repo overview + quickstart
|
||
findings/
|
||
CVE-YYYY-NNNNN/ # one folder per advisory/product tuple
|
||
evidence/
|
||
reachability.json # static+runtime call graph for the finding
|
||
sbom.cdx.json # CycloneDX slice containing the involved components
|
||
decision.openvex.json # OpenVEX statement (status + justification)
|
||
decision.dsse.json # DSSE envelope wrapping the OpenVEX payload
|
||
rekor.txt # optional Rekor UUID/index/checkpoint
|
||
metadata.json # producer info (policy rev, analyzer digests, CAS URIs)
|
||
tools/
|
||
verify.sh # shell helper: dsse verify + optional rekor verification
|
||
verify.py # python verifier (offline) that recomputes digests
|
||
compare.py # baseline diff against Trivy/Syft/Grype/Snyk/Xray outputs
|
||
replay.sh # reruns reachability graphs via `stella replay`
|
||
results/
|
||
summary.csv # FP reduction, MTTD, reproducibility metrics
|
||
runs/2025-11-10/ # pinned scanner/policy versions + raw outputs
|
||
stella/
|
||
findings.json
|
||
runtime-facts.ndjson
|
||
reachability.manifest.json
|
||
trivy/
|
||
findings.json
|
||
...
|
||
```
|
||
|
||
### File contracts
|
||
|
||
- `reachability.json` is the canonical export from `cas://reachability/graphs/...` with symbol IDs, call edges, runtime hits, analyzer fingerprints, and CAS references.
|
||
- `decision.openvex.json` follows OpenVEX v1 with Stella Ops-specific `status_notes`, `justification`, `impact_statement`, and `action_statement` text.
|
||
- `decision.dsse.json` is the DSSE envelope returned by Signer (see §3). Always include the PEM cert chain (keyless) or KMS key id.
|
||
- `rekor.txt` captures `{uuid, logIndex, checkpoint}` from Attestor when the decision is logged to Rekor.
|
||
- `metadata.json` binds the DSSE payload back to internal evidence: `{policy_revision, reachability_graph_sha256, runtime_trace_sha256, evidence_cas_uri[], analyzer_versions[], createdBy, createdAt}`.
|
||
|
||
---
|
||
|
||
## 2. Evidence production flow
|
||
|
||
1. **Scanner Worker**
|
||
- Generate `reachability.json` + `sbom.cdx.json` per prioritized CVE.
|
||
- Store artifacts under CAS and surface URIs via `ReachabilityReplayWriter`.
|
||
2. **Policy Engine / VEXer**
|
||
- Evaluate reachability states + policy lattice to produce an OpenVEX statement.
|
||
- Persist `decision.openvex.json` and forward it to Signer.
|
||
3. **Signer & Attestor**
|
||
- Sign the OpenVEX payload via DSSE (`payloadType: application/vnd.in-toto+json`) and return `decision.dsse.json`.
|
||
- Optionally call Attestor to log the DSSE bundle to Rekor; write `{uuid, logIndex, checkpoint}` to `rekor.txt`.
|
||
4. **Bench harness**
|
||
- Collect SBOM slice, reachability proof, OpenVEX, DSSE, Rekor metadata, and companion metrics into `bench/findings/CVE-...`.
|
||
- Record tool versions + CAS digests under `metadata.json`.
|
||
|
||
All steps must be deterministic: repeated scans with the same inputs produce identical artifacts and digests.
|
||
|
||
---
|
||
|
||
## 3. Signing & transparency requirements
|
||
|
||
| Artifact | Producer | Format | Notes |
|
||
|-------------------------|---------------|----------------------------------------|-------|
|
||
| Reachability evidence | Scanner | Canonical JSON (sorted keys) | CAS URI recorded in metadata. |
|
||
| SBOM slice | Scanner | CycloneDX 1.6 JSON | Keep only components relevant to the finding. |
|
||
| OpenVEX decision | Policy/VEXer | OpenVEX v1 | One statement per `(CVE, product)` tuple. |
|
||
| DSSE bundle | Signer | DSSE envelope over OpenVEX payload | Include Fulcio cert or KMS key id. |
|
||
| Rekor record (optional) | Attestor | Rekor UUID/index/checkpoint | Store alongside DSSE for offline verification. |
|
||
|
||
Signer must expose a predicate alias `stella.ops/vexDecision@v1` (see Sprint task `SIGN-VEX-401-018`). Payload = OpenVEX JSON. Rekor logging reuses the existing Attestor `/rekor/entries` pipeline.
|
||
|
||
---
|
||
|
||
## 4. Verification tooling
|
||
|
||
The repo ships two verifiers:
|
||
|
||
1. `tools/verify.sh` (bash) — wraps `cosign verify-attestation`/`in-toto verify`, Rekor inclusion checks (`rekor-cli logproof`), and digest comparison.
|
||
2. `tools/verify.py` — pure-Python offline verifier for air-gapped environments:
|
||
- Validates DSSE signature using the embedded Fulcio cert or configured root.
|
||
- Recomputes `sha256` over `reachability.json`, `sbom.cdx.json`, and `decision.openvex.json` to ensure the DSSE payload matches.
|
||
- Optionally replays reachability by invoking `stella replay --manifest ... --finding CVE-...`.
|
||
|
||
CLI addition (`stella decision verify`) should shell out to these helpers when `--from bench` is provided.
|
||
|
||
---
|
||
|
||
## 5. Metrics & comparison harness
|
||
|
||
`tools/compare.py` ingests raw outputs from Stella Ops and baseline scanners (Trivy, Syft, Grype, Snyk, Xray) stored under `results/runs/<date>/<scanner>/findings.json`. For each target:
|
||
|
||
- **False-positive reduction (FPR)** = `1 - (# of findings confirmed true positives / # of baseline findings)`.
|
||
- **Mean time to decision (MTTD)** = average wall-clock time between scan start and DSSE-signed OpenVEX emission.
|
||
- **Reproducibility score** = `1` if re-running reachability produces identical digests for all artifacts, else `0`; aggregated per run.
|
||
|
||
`results/summary.csv` columns:
|
||
|
||
```
|
||
target,cve,baseline_scanner,baseline_hits,stella_hits,fp_reduction,mttd_seconds,reproducible,rekor_uuid
|
||
```
|
||
|
||
Automate collection via `Makefile` or `bench/run.sh` pipeline (task `BENCH-AUTO-401-019`).
|
||
|
||
---
|
||
|
||
## 6. Publication & README checklist
|
||
|
||
`bench/README.md` must include:
|
||
|
||
- High-level workflow diagram (scan → reachability → OpenVEX → DSSE → Rekor → bench).
|
||
- Prerequisites (`cosign`, `rekor-cli`, `stella` CLI).
|
||
- Quickstart commands:
|
||
```bash
|
||
./tools/verify.sh CVE-2023-12345 pkg:purl/example@1.2.3
|
||
./tools/compare.py --target sample/nginx --baseline trivy --run 2025-11-10
|
||
```
|
||
- How to recreate a finding: `stella replay --manifest results/runs/.../replay.yaml --finding CVE-...`.
|
||
- Contribution guide (where to place new findings, how to update metrics, required metadata).
|
||
|
||
---
|
||
|
||
## 7. Implementation tasks (see Sprint 401+)
|
||
|
||
- `POLICY-VEX-401-010` — emit OpenVEX per finding and publish to bench repo.
|
||
- `SIGN-VEX-401-018` — add DSSE predicate + Rekor logging for decision payloads.
|
||
- `CLI-VEX-401-011` — new `stella decision` verbs (`export`, `verify`, `compare`).
|
||
- `BENCH-AUTO-401-019` — automation to populate `bench/findings/**`, run baseline scanners, and update `results/summary.csv`.
|
||
- `DOCS-VEX-401-012` — maintain this playbook + README templates, document verification workflow.
|
||
|
||
Update `docs/implplan/SPRINT_0401_0001_0001_reachability_evidence_chain.md` whenever these tasks move state.
|