up

bench/reachability-benchmark/docs/submission-guide.md

@@ -0,0 +1,59 @@
+# Reachability Benchmark · Submission Guide
+
+This guide explains how to produce a compliant submission for the Stella Ops reachability benchmark. It is fully offline-friendly.
+
+## Prerequisites
+- Python 3.11+
+- Your analyzer toolchain (no network calls during analysis)
+- Schemas from `schemas/` and truth from `benchmark/truth/`
+
+## Steps
+1) **Build cases deterministically**
+   ```bash
+   python tools/build/build_all.py --cases cases
+   ```
+   - Sets `SOURCE_DATE_EPOCH`.
+   - Skips Java by default if JDK is unavailable (pass `--skip-lang` as needed).
+
+2) **Run your analyzer**
+   - For each case, produce sink predictions in memory-safe JSON.
+   - Do not reach out to the internet, package registries, or remote APIs.
+
+3) **Emit `submission.json`**
+   - Must conform to `schemas/submission.schema.json` (`version: 1.0.0`).
+   - Sort cases and sinks alphabetically to ensure determinism.
+   - Include optional runtime stats under `run` (time_s, peak_mb) if available.
+
+4) **Validate**
+   ```bash
+   python tools/validate.py --submission submission.json --schema schemas/submission.schema.json
+   ```
+
+5) **Score locally**
+   ```bash
+   tools/scorer/rb_score.py --truth benchmark/truth/<aggregate>.json --submission submission.json --format json
+   ```
+
+6) **Compare (optional)**
+   ```bash
+   tools/scorer/rb_compare.py --truth benchmark/truth/<aggregate>.json \
+     --submissions submission.json baselines/*/submission.json \
+     --output leaderboard.json --text
+   ```
+
+## Determinism checklist
+- Set `SOURCE_DATE_EPOCH` for all builds.
+- Disable telemetry/version checks in your analyzer.
+- Avoid nondeterministic ordering (sort file and sink lists).
+- No network access; use vendored toolchains only.
+- Use fixed seeds for any sampling.
+
+## Packaging
+- Submit a zip/tar with:
+  - `submission.json`
+  - Tool version & configuration (README)
+  - Optional logs and runtime metrics
+- Do **not** include binaries that require network access or licenses we cannot redistribute.
+
+## Support
+- Open issues in the public repo (once live) or provide a reproducible script that runs fully offline.