42 lines
1.9 KiB
Markdown
42 lines
1.9 KiB
Markdown
# Reachability Benchmark · Governance & Maintenance
|
|
|
|
## Roles
|
|
- **TAC (Technical Advisory Committee):** approves material changes to schemas, truth sets, and scoring rules; rotates quarterly.
|
|
- **Maintainers:** curate cases, review submissions, run determinism checks, and publish baselines.
|
|
- **Observers:** may propose cases and review reports; no merge rights.
|
|
|
|
## Release cadence
|
|
- **Quarterly update window:** publish new/updated cases and hidden test set refreshes once per quarter.
|
|
- **Patch releases:** critical fixes to schemas or scorer may be shipped off-cycle; must remain backward compatible within `version: 1.x`.
|
|
|
|
## Hidden test set
|
|
- A reserved set of cases is held back to prevent overfitting.
|
|
- Rotation policy: replace at least 25% of hidden cases each quarter; keep prior versions for audit.
|
|
- Hidden cases follow the same determinism rules; hashes and schema versions are documented internally.
|
|
|
|
## Change control
|
|
- All changes require:
|
|
- Schema validation (`tools/validate.py`).
|
|
- Deterministic rebuild (`tools/build/build_all.py` with `SOURCE_DATE_EPOCH`).
|
|
- Updated truth files and baselines.
|
|
- Execution log entry in `docs/implplan/SPRINT_0513_...` with date/owner.
|
|
- Breaking changes to schemas or scoring rules require TAC approval and a new major schema version.
|
|
|
|
## Determinism rules (global)
|
|
- No network access during build, analysis, or scoring.
|
|
- Fixed seeds and sorted outputs.
|
|
- Stable timestamps via `SOURCE_DATE_EPOCH`.
|
|
- Telemetry disabled for all tools.
|
|
|
|
## Licensing & provenance
|
|
- All public artifacts are Apache-2.0.
|
|
- Third-party snippets must retain attribution and be license-compatible.
|
|
- Each release captures toolchain hashes (compilers, runners) in the release notes.
|
|
|
|
## Incident handling
|
|
- If a nondeterminism or licensing issue is found:
|
|
1) Freeze new submissions.
|
|
2) Reproduce with `ci/run-ci.sh`.
|
|
3) Issue a hotfix release of truth/baselines; bump patch version.
|
|
4) Announce in release notes and mark superseded artifacts.
|