stella-ops.org/git.stella-ops.org
1
0
Fork 0
Code Issues Pull Requests Actions 16 Releases Wiki Activity
Files
8f54ffa2033dcd9f9558dc7dc15f66fa70615c3c
git.stella-ops.org/bench/reachability-benchmark/README.md
StellaOps Bot 8f54ffa203
Some checks failed
Docs CI / lint-and-preview (push) Has been cancelled
Details
devportal-offline / build-offline (push) Has been cancelled
Details
Mirror Thin Bundle Sign & Verify / mirror-sign (push) Has been cancelled
Details
up
2025-11-29 11:37:00 +02:00

1.7 KiB
Raw Blame History

StellaOps Reachability Benchmark (Public)

Deterministic, reproducible benchmark for reachability analysis tools.

Goals

  • Provide open cases with ground truth for reachable/unreachable sinks.
  • Enforce determinism (hash-stable builds, fixed seeds, pinned deps).
  • Enable fair scoring via the rb-score CLI and published schemas.

Layout

  • cases/<lang>/<project>/ — benchmark cases with deterministic Dockerfiles, pinned deps, oracle tests.
  • schemas/ — JSON/YAML schemas for cases, entrypoints, truth, submissions.
  • benchmark/truth/ — ground-truth labels (hidden/internal split optional).
  • benchmark/submissions/ — sample submissions and format reference.
  • tools/scorer/ — rb-score CLI and tests.
  • baselines/ — reference runners (Semgrep, CodeQL, Stella) with normalized outputs.
  • ci/ — deterministic CI workflows and scripts.
  • website/ — static site (leaderboard/docs/downloads).

Determinism & Offline Rules

  • No network during build/test; pin images/deps; set SOURCE_DATE_EPOCH.
  • Sort file lists; stable JSON/YAML emitters; fixed RNG seeds.
  • All scripts must succeed on a clean machine with cached toolchain tarballs only.

Licensing

  • Apache-2.0 for all benchmark assets. Third-party snippets must be license-compatible and attributed.

Quick Start (once populated)

# schema sanity checks (offline)
python tools/validate.py all schemas/examples

# score a submission (coming in task 513-008)
cd tools/scorer
./rb-score --cases ../cases --truth ../benchmark/truth --submission ../benchmark/submissions/sample.json

Contributing

See CONTRIBUTING.md. Open issues/PRs welcome; please provide hashes and logs for reproducibility.