stella-ops.org/git.stella-ops.org
1
0
Fork 0
Code Issues Pull Requests Actions 3 Releases Wiki Activity

feat: Add JSON schema definitions for coverage and trace artifacts in reachability benchmark

Browse Source
This commit is contained in:
StellaOps Bot
2025-12-03 09:49:59 +02:00
parent de53785176
commit e0b585c799
3 changed files with 98 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

5
bench/reachability-benchmark/schemas/README.md
View File

@@ -3,8 +3,11 @@
- `case.schema.yaml` — case descriptor (language, sinks, deterministic build/test, environment, optional inline ground truth summary).
- `entrypoints.schema.yaml` — declared entrypoints grouped by type (`http`, `cli`, `scheduled`, etc.).
- `truth.schema.json` — ground-truth labels + evidence per sink (`reachable`/`unreachable`/`unknown`).
- `coverage.schema.json` — coverage artifact emitted by oracle tests (lines covered or function-level coverage).
- `trace.schema.json` — dynamic trace artifact (structured path or event log) proving reachability/unreachability.
- `submission.schema.json` — submission format (tool/run metadata, per-sink predictions, explanations).
- `examples/` — minimal valid examples for each schema.
- `benchmark/schemas/benchmark-manifest.schema.json` — kit manifest with hashed artifacts, resource limits, and optional DSSE signatures.
## Validate quickly
```bash
@@ -15,6 +18,8 @@ python -m pip install -r ../tools/requirements.txt
python ../tools/validate.py case examples/case.sample.yaml
python ../tools/validate.py entrypoints examples/entrypoints.sample.yaml
python ../tools/validate.py truth examples/truth.sample.json
python ../tools/validate.py coverage ../cases/js/unsafe-eval/outputs/coverage.json
python ../tools/validate.py trace ../cases/js/unsafe-eval/outputs/traces/traces.json
python ../tools/validate.py submission examples/submission.sample.json
# or validate everything in one shot