28 lines
1.5 KiB
Markdown
28 lines
1.5 KiB
Markdown
# Schemas
|
|
|
|
- `case.schema.yaml` — case descriptor (language, sinks, deterministic build/test, environment, optional inline ground truth summary).
|
|
- `entrypoints.schema.yaml` — declared entrypoints grouped by type (`http`, `cli`, `scheduled`, etc.).
|
|
- `truth.schema.json` — ground-truth labels + evidence per sink (`reachable`/`unreachable`/`unknown`).
|
|
- `coverage.schema.json` — coverage artifact emitted by oracle tests (lines covered or function-level coverage).
|
|
- `trace.schema.json` — dynamic trace artifact (structured path or event log) proving reachability/unreachability.
|
|
- `submission.schema.json` — submission format (tool/run metadata, per-sink predictions, explanations).
|
|
- `examples/` — minimal valid examples for each schema.
|
|
- `benchmark/schemas/benchmark-manifest.schema.json` — kit manifest with hashed artifacts, resource limits, and optional DSSE signatures.
|
|
|
|
## Validate quickly
|
|
```bash
|
|
# install minimal deps (offline-friendly, pinned)
|
|
python -m pip install -r ../tools/requirements.txt
|
|
|
|
# validate individual files
|
|
python ../tools/validate.py case examples/case.sample.yaml
|
|
python ../tools/validate.py entrypoints examples/entrypoints.sample.yaml
|
|
python ../tools/validate.py truth examples/truth.sample.json
|
|
python ../tools/validate.py coverage ../cases/js/unsafe-eval/outputs/coverage.json
|
|
python ../tools/validate.py trace ../cases/js/unsafe-eval/outputs/traces/traces.json
|
|
python ../tools/validate.py submission examples/submission.sample.json
|
|
|
|
# or validate everything in one shot
|
|
python ../tools/validate.py all examples
|
|
```
|