git.stella-ops.org/bench

Stella Ops Bench Repository

Status: Active · Last updated: 2025-12-13 Purpose: Host reproducible VEX decisions, reachability evidence, and comparison data proving Stella Ops' signal quality vs. baseline scanners.

Layout

bench/
  README.md                 # this file
  findings/                 # per CVE/product bundles
    CVE-YYYY-NNNNN/
      evidence/
        reachability.json   # richgraph-v1 excerpt
        sbom.cdx.json       # CycloneDX SBOM
      decision.openvex.json # OpenVEX decision
      decision.dsse.json    # DSSE envelope
      rekor.txt             # Rekor log index + inclusion proof
      metadata.json         # finding metadata (purl, CVE, version)
  tools/
    verify.sh               # DSSE + Rekor verifier (online)
    verify.py               # offline verifier
    compare.py              # baseline comparison script
    replay.sh               # runs reachability replay manifests
  results/
    summary.csv             # aggregated metrics
    runs/<date>/...         # raw outputs + replay manifests
  reachability-benchmark/   # reachability benchmark with JDK fixtures

Related Documentation

Document	Purpose
VEX Evidence Playbook	Proof bundle schema, justification catalog, verification workflow
Hybrid Attestation	Graph-level and edge-bundle DSSE decisions
Function-Level Evidence	Cross-module evidence chain guide
Deterministic Replay	Replay manifest specification

Verification Workflows

Quick Verification (Online)

# Verify a VEX proof bundle with DSSE and Rekor
./tools/verify.sh findings/CVE-2021-44228/decision.dsse.json

# Output:
# ✓ DSSE signature valid
# ✓ Rekor inclusion verified (log index: 12345678)
# ✓ Evidence hashes match
# ✓ Justification catalog membership confirmed

Offline Verification

# Verify without network access
python tools/verify.py \
  --bundle findings/CVE-2021-44228/decision.dsse.json \
  --cas-root ./findings/CVE-2021-44228/evidence/ \
  --catalog ../docs/benchmarks/vex-justifications.catalog.json

# Or use the VEX proof bundle verifier
python ../scripts/vex/verify_proof_bundle.py \
  --bundle ../tests/Vex/ProofBundles/sample-proof-bundle.json \
  --cas-root ../tests/Vex/ProofBundles/cas/

Reachability Graph Verification

# Verify graph DSSE
stella graph verify --hash blake3:a1b2c3d4...

# Verify with edge bundles
stella graph verify --hash blake3:a1b2c3d4... --include-bundles

# Offline with local CAS
stella graph verify --hash blake3:a1b2c3d4... --cas-root ./offline-cas/

Baseline Comparison

# Compare Stella Ops findings against baseline scanners
python tools/compare.py \
  --stellaops results/runs/2025-12-13/findings.json \
  --baseline results/baselines/trivy-latest.json \
  --output results/comparison-2025-12-13.csv

# Metrics generated:
# - True positives (reachability-confirmed)
# - False positives (unreachable code paths)
# - MTTD (mean time to detect)
# - Reproducibility score

Artifact Contracts

All bench artifacts must comply with:

1. VEX Proof Bundle Schema (docs/benchmarks/vex-evidence-playbook.schema.json)

   • BLAKE3-256 primary hash, SHA-256 secondary
   • Canonical JSON with sorted keys
   • DSSE envelope with Rekor-ready digest

2. Justification Catalog (docs/benchmarks/vex-justifications.catalog.json)

   • VEX1-VEX10 justification codes
   • Required evidence types per justification
   • Expiry and re-evaluation rules

3. Reachability Graph (docs/contracts/richgraph-v1.md)

   • BLAKE3 graph_hash for content addressing
   • Deterministic node/edge ordering
   • SymbolID/EdgeID format compliance

CI Integration

The bench directory is validated by:

• .gitea/workflows/vex-proof-bundles.yml - Verifies all proof bundles
• .gitea/workflows/bench-determinism.yml - Runs determinism benchmarks
• .gitea/workflows/hybrid-attestation.yml - Verifies graph/edge-bundle fixtures

Contributing

1. Add new findings under findings/CVE-YYYY-NNNNN/
2. Include all required evidence artifacts
3. Generate DSSE envelope and Rekor proof
4. Update results/summary.csv
5. Run verification: ./tools/verify.sh findings/CVE-YYYY-NNNNN/decision.dsse.json