stella-ops.org/git.stella-ops.org
1
0
Fork 0
Code Issues Pull Requests Actions 2 Releases Wiki Activity

up
Some checks failed
Docs CI / lint-and-preview (push) Has been cancelled
Details
Findings Ledger CI / build-test (push) Has been cancelled
Details
Findings Ledger CI / migration-validation (push) Has been cancelled
Details
Scanner Analyzers / Discover Analyzers (push) Has been cancelled
Details
Signals Reachability Scoring & Events / reachability-smoke (push) Has been cancelled
Details
AOC Guard CI / aoc-guard (push) Has been cancelled
Details
Concelier Attestation Tests / attestation-tests (push) Has been cancelled
Details
cryptopro-linux-csp / build-and-test (push) Has been cancelled
Details
Scanner Analyzers / Validate Test Fixtures (push) Has been cancelled
Details
Signals CI & Image / signals-ci (push) Has been cancelled
Details
sm-remote-ci / build-and-test (push) Has been cancelled
Details
Findings Ledger CI / generate-manifest (push) Has been cancelled
Details
AOC Guard CI / aoc-verify (push) Has been cancelled
Details
Scanner Analyzers / Build Analyzers (push) Has been cancelled
Details
Scanner Analyzers / Test Language Analyzers (push) Has been cancelled
Details
Scanner Analyzers / Verify Deterministic Output (push) Has been cancelled
Details
Signals Reachability Scoring & Events / sign-and-upload (push) Has been cancelled
Details

Browse Source
This commit is contained in:
StellaOps Bot
2025-12-09 09:38:09 +02:00
parent bc0762e97d
commit 108d1c64b3
193 changed files with 7265 additions and 13029 deletions
Download Patch File Download Diff File Expand all files Collapse all files

11
bench/reachability-benchmark/benchmark/CHANGELOG.md
View File

@@ -1,11 +1,16 @@
# Reachability Benchmark Changelog
## 1.0.1 · 2025-12-03
## 1.0.2 Aú 2025-12-05
- Unblocked Java track with vendored Temurin 21 (`tools/java/ensure_jdk.sh`) and deterministic build artifacts (coverage + traces).
- Added three more Java cases (`micronaut-deserialize`, `micronaut-guarded`, `spring-reflection`) to reach 5/5 required cases.
- `tools/build/build_all.py` now supports `--skip-lang` and runs under WSL-aware bash; CI builds Java cases by default.
## 1.0.1 Aú 2025-12-03
- Added manifest schema + sample manifest with hashes, SBOM/attestation entries, and sandbox/redaction metadata.
- Added coverage/trace schemas and extended validator to cover them.
- Introduced `tools/verify_manifest.py` and deterministic offline kit packaging script.
- Added per-language determinism env templates and dataset safety checklist.
- Populated SBOM + attestation outputs for JS/PY/C tracks; Java remains blocked on JDK availability.
- Populated SBOM + attestation outputs for JS/PY/C tracks.
## 1.0.0 · 2025-12-01
## 1.0.0 Aú 2025-12-01
- Initial public dataset, scorer, baselines, and website.

2
bench/reachability-benchmark/benchmark/checklists/dataset-safety.md
View File

@@ -8,7 +8,7 @@ Version: 1.0.1 · Date: 2025-12-03
- [x] Published schemas/validators: truth/submission/coverage/trace + manifest schemas; validated via `tools/validate.py` and `tools/verify_manifest.py`.
- [x] Evidence bundles: coverage + traces + attestation + sbom recorded per case (sample manifest).
- [x] Binary case recipe: `cases/**/build/build.sh` pinned `SOURCE_DATE_EPOCH` and env templates under `benchmark/templates/determinism/`.
- [x] Determinism CI: `ci/run-ci.sh` + `tools/verify_manifest.py` run twice to compare hashes; Java track still blocked on JDK availability.
- [x] Determinism CI: `ci/run-ci.sh` + `tools/verify_manifest.py` run twice to compare hashes; Java track uses vendored Temurin 21 via `tools/java/ensure_jdk.sh`.
- [x] Signed baselines: baseline submissions may include DSSE path in manifest (not required for sample kit); rulepack hashes recorded separately.
- [x] Submission policy: CLA/DSSE optional in sample; production kits require DSSE envelope recorded in `signatures`.
- [x] Semantic versioning & changelog: see `benchmark/CHANGELOG.md`; manifest `version` mirrors dataset release.

279
bench/reachability-benchmark/benchmark/manifest.sample.json
View File

@@ -1,92 +1,203 @@
{
"schemaVersion": "1.0.0",
"kitId": "reachability-benchmark:public-v1",
"version": "1.0.1",
"artifacts": {
"baselineSubmissions": [],
"scorer": {
"path": "tools/scorer/rb_score.py",
"sha256": "32d4f69f5d1d4b87902d6c4f020efde703487d526bf7d42b4438cb2499813f7f"
},
"submissionSchema": {
"path": "schemas/submission.schema.json",
"sha256": "de5bebb2dbcd085d7896f47a16b9d3837a65fb7f816dcf7e587967d5848c50a7"
}
},
"cases": [
{
"hashes": {
"attestation": {
"path": "cases/js/unsafe-eval/outputs/attestation.json",
"sha256": "be3b0971d805f68730a1c4c0f7a4c3c40dfc7a73099a5524c68759fcc1729d7c"
},
"binary": {
"path": "cases/js/unsafe-eval/outputs/binary.tar.gz",
"sha256": "72da19f28c2c36b6666afcc304514b387de20a5de881d5341067481e8418e23e"
},
"case": {
"path": "cases/js/unsafe-eval/case.yaml",
"sha256": "a858ff509fda65d69df476e870d9646c6a84744010c812f3d23a88576f20cb6b"
},
"coverage": {
"path": "cases/js/unsafe-eval/outputs/coverage.json",
"sha256": "c2cf5af508d33f6ecdc7c0f10200a02a4c0ddeb8e1fc08b55d9bd4a2d6cb926b"
},
"entrypoints": {
"path": "cases/js/unsafe-eval/entrypoints.yaml",
"sha256": "77829e728d34c9dc5f56c04784c97f619830ad43bd8410acb3d7134f372a49b3"
},
"sbom": {
"path": "cases/js/unsafe-eval/outputs/sbom.cdx.json",
"sha256": "c00ee1e12b1b6a6237e42174b2fe1393bcf575f6605205a2b84366e867b36d5f"
},
"source": {
"path": "cases/js/unsafe-eval",
"sha256": "69b0d1cbae1e2c9ddc0f4dba8c6db507e1d3a1c5ea0a0a545c6f3e785529c91c"
},
"traces": {
"path": "cases/js/unsafe-eval/outputs/traces/traces.json",
"sha256": "6e63c78e091cc9d06acdc5966dd9e54593ca6b0b97f502928de278b3f80adbd8"
},
"truth": {
"path": "benchmark/truth/js-unsafe-eval.json",
"sha256": "ab42f28ed229eb657ffcb36c3a99287436e1822a4c7d395a94de784457a08f62"
}
},
"id": "js-unsafe-eval:001",
"language": "js",
"redaction": {
"pii": false,
"policy": "benchmark-default/v1"
},
"sandbox": {
"network": "loopback",
"privileges": "rootless"
},
"size": "small",
"truth": {
"confidence": "high",
"label": "reachable",
"rationale": "Unit test hits eval sink via POST /api/exec"
}
},
{
"hashes": {
"attestation": {
"path": "cases/py/fastapi-guarded/outputs/attestation.json",
"sha256": "257aa5408a5c6ffe0e193a75a2a54597f8c6f61babfe8aaf26bd47340c3086c3"
},
"binary": {
"path": "cases/py/fastapi-guarded/outputs/binary.tar.gz",
"sha256": "ca964fef352dc535b63d35b8f8846cc051e10e54cfd8aceef7566f3c94178b76"
},
"case": {
"path": "cases/py/fastapi-guarded/case.yaml",
"sha256": "0add8a5f487ebd21ee20ab88b7c6436fe8471f0a54ab8da0e08c8416aa181346"
},
"coverage": {
"path": "cases/py/fastapi-guarded/outputs/coverage.json",
"sha256": "07b1f6dccaa02bd4e1c3e2771064fa3c6e06d02843a724151721ea694762c750"
},
"entrypoints": {
"path": "cases/py/fastapi-guarded/entrypoints.yaml",
"sha256": "47c9dd15bf7c5bb8641893a92791d3f7675ed6adba17b251f609335400d29d41"
},
"sbom": {
"path": "cases/py/fastapi-guarded/outputs/sbom.cdx.json",
"sha256": "13999d8f3d4c9bdb70ea54ad1de613be3f893d79bdd1a53f7c9401e6add88cf0"
},
"source": {
"path": "cases/py/fastapi-guarded",
"sha256": "0869cab10767ac7e7b33c9bbd634f811d98ce5cdeb244769f1a81949438460fb"
},
"traces": {
"path": "cases/py/fastapi-guarded/outputs/traces/traces.json",
"sha256": "4633748b8b428b45e3702f2f8f5b3f4270728078e26bce1e08900ed1d5bb3046"
},
"truth": {
"path": "benchmark/truth/py-fastapi-guarded.json",
"sha256": "f8c62abeb00006621feeb010d0e47d248918dffd6d6e20e0f47d74e1b3642760"
}
},
"id": "py-fastapi-guarded:104",
"language": "py",
"redaction": {
"pii": false,
"policy": "benchmark-default/v1"
},
"sandbox": {
"network": "loopback",
"privileges": "rootless"
},
"size": "small",
"truth": {
"confidence": "high",
"label": "unreachable",
"rationale": "Feature flag ALLOW_EXEC must be true before sink executes"
}
},
{
"hashes": {
"attestation": {
"path": "cases/c/unsafe-system/outputs/attestation.json",
"sha256": "c3755088182359a45492170fa8a57d826b605176333d109f4f113bc7ccf85f97"
},
"binary": {
"path": "cases/c/unsafe-system/outputs/binary.tar.gz",
"sha256": "62200167bd660bad6d131b21f941acdfebe00e949e353a53c97b6691ac8f0e49"
},
"case": {
"path": "cases/c/unsafe-system/case.yaml",
"sha256": "7799a3a629c22ad47197309f44e32aabbc4e6711ef78d606ba57a7a4974787ce"
},
"coverage": {
"path": "cases/c/unsafe-system/outputs/coverage.json",
"sha256": "03ba8cf09e7e0ed82e9fa8abb48f92355e894fd56e0c0160a504193a6f6ec48a"
},
"entrypoints": {
"path": "cases/c/unsafe-system/entrypoints.yaml",
"sha256": "06afee8350460c9d15b26ea9d4ea293e8eb3f4b86b3179e19401fa99947e4490"
},
"sbom": {
"path": "cases/c/unsafe-system/outputs/sbom.cdx.json",
"sha256": "4c72a213fc4c646f44b4d0be3c23711b120b2a386374ebaa4897e5058980e0f5"
},
"source": {
"path": "cases/c/unsafe-system",
"sha256": "bc39ab3a3e5cb3944a205912ecad8c1ac4b7d15c64b453c9d34a9a5df7fbbbf4"
},
"traces": {
"path": "cases/c/unsafe-system/outputs/traces/traces.json",
"sha256": "f6469e46a57b8a6e8e17c9b8e78168edd6657ea8a5e1e96fe6ab4a0fc88a734e"
},
"truth": {
"path": "benchmark/truth/c-unsafe-system.json",
"sha256": "9a8200c2cf549b3ac8b19b170e9d34df063351879f19f401d8492e280ad08c13"
}
},
"id": "c-unsafe-system:001",
"language": "c",
"redaction": {
"pii": false,
"policy": "benchmark-default/v1"
},
"sandbox": {
"network": "loopback",
"privileges": "rootless"
},
"size": "small",
"truth": {
"confidence": "high",
"label": "reachable",
"rationale": "Command injection sink reachable via argv -> system()"
}
}
],
"createdAt": "2025-12-03T00:00:00Z",
"sourceDateEpoch": 1730000000,
"kitId": "reachability-benchmark:public-v1",
"resourceLimits": {
"cpu": "4",
"memory": "8Gi"
},
"cases": [
{
"id": "js-unsafe-eval:001",
"language": "js",
"size": "small",
"hashes": {
"source": { "path": "cases/js/unsafe-eval", "sha256": "69b0d1cbae1e2c9ddc0f4dba8c6db507e1d3a1c5ea0a0a545c6f3e785529c91c" },
"case": { "path": "cases/js/unsafe-eval/case.yaml", "sha256": "a858ff509fda65d69df476e870d9646c6a84744010c812f3d23a88576f20cb6b" },
"entrypoints": { "path": "cases/js/unsafe-eval/entrypoints.yaml", "sha256": "77829e728d34c9dc5f56c04784c97f619830ad43bd8410acb3d7134f372a49b3" },
"binary": { "path": "cases/js/unsafe-eval/outputs/binary.tar.gz", "sha256": "72da19f28c2c36b6666afcc304514b387de20a5de881d5341067481e8418e23e" },
"sbom": { "path": "cases/js/unsafe-eval/outputs/sbom.cdx.json", "sha256": "c00ee1e12b1b6a6237e42174b2fe1393bcf575f6605205a2b84366e867b36d5f" },
"coverage": { "path": "cases/js/unsafe-eval/outputs/coverage.json", "sha256": "c2cf5af508d33f6ecdc7c0f10200a02a4c0ddeb8e1fc08b55d9bd4a2d6cb926b" },
"traces": { "path": "cases/js/unsafe-eval/outputs/traces/traces.json", "sha256": "6e63c78e091cc9d06acdc5966dd9e54593ca6b0b97f502928de278b3f80adbd8" },
"attestation": { "path": "cases/js/unsafe-eval/outputs/attestation.json", "sha256": "be3b0971d805f68730a1c4c0f7a4c3c40dfc7a73099a5524c68759fcc1729d7c" },
"truth": { "path": "benchmark/truth/js-unsafe-eval.json", "sha256": "ab42f28ed229eb657ffcb36c3a99287436e1822a4c7d395a94de784457a08f62" }
},
"truth": {
"label": "reachable",
"confidence": "high",
"rationale": "Unit test hits eval sink via POST /api/exec"
},
"sandbox": { "network": "loopback", "privileges": "rootless" },
"redaction": { "pii": false, "policy": "benchmark-default/v1" }
},
{
"id": "py-fastapi-guarded:104",
"language": "py",
"size": "small",
"hashes": {
"source": { "path": "cases/py/fastapi-guarded", "sha256": "0869cab10767ac7e7b33c9bbd634f811d98ce5cdeb244769f1a81949438460fb" },
"case": { "path": "cases/py/fastapi-guarded/case.yaml", "sha256": "0add8a5f487ebd21ee20ab88b7c6436fe8471f0a54ab8da0e08c8416aa181346" },
"entrypoints": { "path": "cases/py/fastapi-guarded/entrypoints.yaml", "sha256": "47c9dd15bf7c5bb8641893a92791d3f7675ed6adba17b251f609335400d29d41" },
"binary": { "path": "cases/py/fastapi-guarded/outputs/binary.tar.gz", "sha256": "ca964fef352dc535b63d35b8f8846cc051e10e54cfd8aceef7566f3c94178b76" },
"sbom": { "path": "cases/py/fastapi-guarded/outputs/sbom.cdx.json", "sha256": "13999d8f3d4c9bdb70ea54ad1de613be3f893d79bdd1a53f7c9401e6add88cf0" },
"coverage": { "path": "cases/py/fastapi-guarded/outputs/coverage.json", "sha256": "07b1f6dccaa02bd4e1c3e2771064fa3c6e06d02843a724151721ea694762c750" },
"traces": { "path": "cases/py/fastapi-guarded/outputs/traces/traces.json", "sha256": "4633748b8b428b45e3702f2f8f5b3f4270728078e26bce1e08900ed1d5bb3046" },
"attestation": { "path": "cases/py/fastapi-guarded/outputs/attestation.json", "sha256": "257aa5408a5c6ffe0e193a75a2a54597f8c6f61babfe8aaf26bd47340c3086c3" },
"truth": { "path": "benchmark/truth/py-fastapi-guarded.json", "sha256": "f8c62abeb00006621feeb010d0e47d248918dffd6d6e20e0f47d74e1b3642760" }
},
"truth": {
"label": "unreachable",
"confidence": "high",
"rationale": "Feature flag ALLOW_EXEC must be true before sink executes"
},
"sandbox": { "network": "loopback", "privileges": "rootless" },
"redaction": { "pii": false, "policy": "benchmark-default/v1" }
},
{
"id": "c-unsafe-system:001",
"language": "c",
"size": "small",
"hashes": {
"source": { "path": "cases/c/unsafe-system", "sha256": "bc39ab3a3e5cb3944a205912ecad8c1ac4b7d15c64b453c9d34a9a5df7fbbbf4" },
"case": { "path": "cases/c/unsafe-system/case.yaml", "sha256": "7799a3a629c22ad47197309f44e32aabbc4e6711ef78d606ba57a7a4974787ce" },
"entrypoints": { "path": "cases/c/unsafe-system/entrypoints.yaml", "sha256": "06afee8350460c9d15b26ea9d4ea293e8eb3f4b86b3179e19401fa99947e4490" },
"binary": { "path": "cases/c/unsafe-system/outputs/binary.tar.gz", "sha256": "62200167bd660bad6d131b21f941acdfebe00e949e353a53c97b6691ac8f0e49" },
"sbom": { "path": "cases/c/unsafe-system/outputs/sbom.cdx.json", "sha256": "4c72a213fc4c646f44b4d0be3c23711b120b2a386374ebaa4897e5058980e0f5" },
"coverage": { "path": "cases/c/unsafe-system/outputs/coverage.json", "sha256": "03ba8cf09e7e0ed82e9fa8abb48f92355e894fd56e0c0160a504193a6f6ec48a" },
"traces": { "path": "cases/c/unsafe-system/outputs/traces/traces.json", "sha256": "f6469e46a57b8a6e8e17c9b8e78168edd6657ea8a5e1e96fe6ab4a0fc88a734e" },
"attestation": { "path": "cases/c/unsafe-system/outputs/attestation.json", "sha256": "c3755088182359a45492170fa8a57d826b605176333d109f4f113bc7ccf85f97" },
"truth": { "path": "benchmark/truth/c-unsafe-system.json", "sha256": "9a8200c2cf549b3ac8b19b170e9d34df063351879f19f401d8492e280ad08c13" }
},
"truth": {
"label": "reachable",
"confidence": "high",
"rationale": "Command injection sink reachable via argv -> system()"
},
"sandbox": { "network": "loopback", "privileges": "rootless" },
"redaction": { "pii": false, "policy": "benchmark-default/v1" }
}
],
"artifacts": {
"submissionSchema": { "path": "schemas/submission.schema.json", "sha256": "de5bebb2dbcd085d7896f47a16b9d3837a65fb7f816dcf7e587967d5848c50a7" },
"scorer": { "path": "tools/scorer/rb_score.py", "sha256": "32d4f69f5d1d4b87902d6c4f020efde703487d526bf7d42b4438cb2499813f7f" },
"baselineSubmissions": []
},
"schemaVersion": "1.0.0",
"signatures": [],
"sourceDateEpoch": 1730000000,
"tools": {
"builder": { "path": "tools/build/build_all.py", "sha256": "64a73f3df9b6f2cdaf5cbb33852b8e9bf443f67cf9dff1573fb635a0252bda9a" },
"validator": { "path": "tools/validate.py", "sha256": "776009ef0f3691e60cc87df3f0468181ee7a827be1bd0f73c77fdb68d3ed31c0" }
"builder": {
"path": "tools/build/build_all.py",
"sha256": "64a73f3df9b6f2cdaf5cbb33852b8e9bf443f67cf9dff1573fb635a0252bda9a"
},
"validator": {
"path": "tools/validate.py",
"sha256": "776009ef0f3691e60cc87df3f0468181ee7a827be1bd0f73c77fdb68d3ed31c0"
}
},
"signatures": []
}
"version": "1.0.2"
}

34
bench/reachability-benchmark/benchmark/truth/java-micronaut-deserialize.json Normal file
View File

@@ -0,0 +1,34 @@
{
"version": "1.0.0",
"cases": [
{
"case_id": "java-micronaut-deserialize:203",
"case_version": "1.0.0",
"notes": "Micronaut-style controller deserializes base64 payload",
"sinks": [
{
"sink_id": "MicronautDeserialize::handleUpload",
"label": "reachable",
"confidence": "high",
"dynamic_evidence": {
"covered_by_tests": [
"src/ControllerTest.java"
],
"coverage_files": [
"outputs/coverage.json"
]
},
"static_evidence": {
"call_path": [
"POST /mn/upload",
"Controller.handleUpload",
"ObjectInputStream.readObject"
]
},
"config_conditions": [],
"notes": "No guard; ObjectInputStream invoked on user-controlled bytes"
}
]
}
]
}

35
bench/reachability-benchmark/benchmark/truth/java-micronaut-guarded.json Normal file
View File

@@ -0,0 +1,35 @@
{
"version": "1.0.0",
"cases": [
{
"case_id": "java-micronaut-guarded:204",
"case_version": "1.0.0",
"notes": "Deserialization guarded by ALLOW_MN_DESER flag (unreachable by default)",
"sinks": [
{
"sink_id": "MicronautDeserializeGuarded::handleUpload",
"label": "unreachable",
"confidence": "high",
"dynamic_evidence": {
"covered_by_tests": [
"src/ControllerTest.java"
],
"coverage_files": [
"outputs/coverage.json"
]
},
"static_evidence": {
"call_path": [
"POST /mn/upload",
"Controller.handleUpload"
]
},
"config_conditions": [
"ALLOW_MN_DESER=true"
],
"notes": "Feature flag defaults to false; sink not executed without ALLOW_MN_DESER"
}
]
}
]
}

4
bench/reachability-benchmark/benchmark/truth/java-spring-deserialize.json
View File

@@ -14,7 +14,9 @@
"covered_by_tests": [
"src/AppTest.java"
],
"coverage_files": []
"coverage_files": [
"outputs/coverage.json"
]
},
"static_evidence": {
"call_path": [

2
bench/reachability-benchmark/benchmark/truth/java-spring-guarded.json
View File

@@ -12,7 +12,7 @@
"confidence": "high",
"dynamic_evidence": {
"covered_by_tests": ["src/AppTest.java"],
"coverage_files": []
"coverage_files": ["outputs/coverage.json"]
},
"static_evidence": {
"call_path": [

34
bench/reachability-benchmark/benchmark/truth/java-spring-reflection.json Normal file
View File

@@ -0,0 +1,34 @@
{
"version": "1.0.0",
"cases": [
{
"case_id": "java-spring-reflection:205",
"case_version": "1.0.0",
"notes": "Reflection endpoint loads arbitrary classes supplied by caller",
"sinks": [
{
"sink_id": "SpringReflection::run",
"label": "reachable",
"confidence": "high",
"dynamic_evidence": {
"covered_by_tests": [
"src/ReflectControllerTest.java"
],
"coverage_files": [
"outputs/coverage.json"
]
},
"static_evidence": {
"call_path": [
"POST /api/reflect",
"ReflectController.run",
"Class.forName"
]
},
"config_conditions": [],
"notes": "User-controlled class name flows into Class.forName and reflection instantiation"
}
]
}
]
}