feat: Implement Filesystem and MongoDB provenance writers for PackRun execution context
- Added `FilesystemPackRunProvenanceWriter` to write provenance manifests to the filesystem. - Introduced `MongoPackRunArtifactReader` to read artifacts from MongoDB. - Created `MongoPackRunProvenanceWriter` to store provenance manifests in MongoDB. - Developed unit tests for filesystem and MongoDB provenance writers. - Established `ITimelineEventStore` and `ITimelineIngestionService` interfaces for timeline event handling. - Implemented `TimelineIngestionService` to validate and persist timeline events with hashing. - Created PostgreSQL schema and migration scripts for timeline indexing. - Added dependency injection support for timeline indexer services. - Developed tests for timeline ingestion and schema validation.
This commit is contained in:
StellaOps Bot
@@ -17,6 +17,24 @@ Deterministic, reproducible benchmark for reachability analysis tools.
|
||||
- `ci/` — deterministic CI workflows and scripts.
|
||||
- `website/` — static site (leaderboard/docs/downloads).
|
||||
|
||||
Sample cases added (JS track):
|
||||
- `cases/js/unsafe-eval` (reachable sink) → `benchmark/truth/js-unsafe-eval.json`.
|
||||
- `cases/js/guarded-eval` (unreachable by default) → `benchmark/truth/js-guarded-eval.json`.
|
||||
- `cases/js/express-eval` (admin eval reachable) → `benchmark/truth/js-express-eval.json`.
|
||||
- `cases/js/express-guarded` (admin eval gated by env) → `benchmark/truth/js-express-guarded.json`.
|
||||
- `cases/js/fastify-template` (template rendering reachable) → `benchmark/truth/js-fastify-template.json`.
|
||||
|
||||
Sample cases added (Python track):
|
||||
- `cases/py/unsafe-exec` (reachable eval) → `benchmark/truth/py-unsafe-exec.json`.
|
||||
- `cases/py/guarded-exec` (unreachable when FEATURE_ENABLE != 1) → `benchmark/truth/py-guarded-exec.json`.
|
||||
- `cases/py/flask-template` (template rendering reachable) → `benchmark/truth/py-flask-template.json`.
|
||||
- `cases/py/fastapi-guarded` (unreachable unless ALLOW_EXEC=true) → `benchmark/truth/py-fastapi-guarded.json`.
|
||||
- `cases/py/django-ssti` (template rendering reachable, autoescape off) → `benchmark/truth/py-django-ssti.json`.
|
||||
|
||||
Sample cases added (Java track):
|
||||
- `cases/java/spring-deserialize` (reachable Java deserialization) → `benchmark/truth/java-spring-deserialize.json`.
|
||||
- `cases/java/spring-guarded` (deserialization unreachable unless ALLOW_DESER=true) → `benchmark/truth/java-spring-guarded.json`.
|
||||
|
||||
## Determinism & Offline Rules
|
||||
- No network during build/test; pin images/deps; set `SOURCE_DATE_EPOCH`.
|
||||
- Sort file lists; stable JSON/YAML emitters; fixed RNG seeds.
|
||||
|
||||
@@ -0,0 +1,32 @@
|
||||
{
|
||||
"version": "1.0.0",
|
||||
"cases": [
|
||||
{
|
||||
"case_id": "java-spring-deserialize:201",
|
||||
"case_version": "1.0.0",
|
||||
"notes": "Java deserialization sink reachable",
|
||||
"sinks": [
|
||||
{
|
||||
"sink_id": "JavaDeserialize::handleRequest",
|
||||
"label": "reachable",
|
||||
"confidence": "high",
|
||||
"dynamic_evidence": {
|
||||
"covered_by_tests": [
|
||||
"src/AppTest.java"
|
||||
],
|
||||
"coverage_files": []
|
||||
},
|
||||
"static_evidence": {
|
||||
"call_path": [
|
||||
"POST /api/upload",
|
||||
"App.handleRequest",
|
||||
"ObjectInputStream.readObject"
|
||||
]
|
||||
},
|
||||
"config_conditions": [],
|
||||
"notes": "No guard; base64 payload deserialized"
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,29 @@
|
||||
{
|
||||
"version": "1.0.0",
|
||||
"cases": [
|
||||
{
|
||||
"case_id": "java-spring-guarded:202",
|
||||
"case_version": "1.0.0",
|
||||
"notes": "Deserialization unreachable by default",
|
||||
"sinks": [
|
||||
{
|
||||
"sink_id": "JavaDeserializeGuarded::handleRequest",
|
||||
"label": "unreachable",
|
||||
"confidence": "high",
|
||||
"dynamic_evidence": {
|
||||
"covered_by_tests": ["src/AppTest.java"],
|
||||
"coverage_files": []
|
||||
},
|
||||
"static_evidence": {
|
||||
"call_path": [
|
||||
"POST /api/upload",
|
||||
"App.handleRequest",
|
||||
"guard: ALLOW_DESER!=true"
|
||||
]
|
||||
},
|
||||
"config_conditions": ["ALLOW_DESER == 'true'"]
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,34 @@
|
||||
{
|
||||
"version": "1.0.0",
|
||||
"cases": [
|
||||
{
|
||||
"case_id": "js-express-eval:003",
|
||||
"case_version": "1.0.0",
|
||||
"notes": "Admin eval reachable",
|
||||
"sinks": [
|
||||
{
|
||||
"sink_id": "ExpressEval::exec",
|
||||
"label": "reachable",
|
||||
"confidence": "high",
|
||||
"dynamic_evidence": {
|
||||
"covered_by_tests": [
|
||||
"tests/test_reach.js"
|
||||
],
|
||||
"coverage_files": [
|
||||
"outputs/coverage.json"
|
||||
]
|
||||
},
|
||||
"static_evidence": {
|
||||
"call_path": [
|
||||
"POST /api/admin/exec",
|
||||
"createServer.exec",
|
||||
"eval(code)"
|
||||
]
|
||||
},
|
||||
"config_conditions": [],
|
||||
"notes": "No guard on admin path"
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,36 @@
|
||||
{
|
||||
"version": "1.0.0",
|
||||
"cases": [
|
||||
{
|
||||
"case_id": "js-express-guarded:004",
|
||||
"case_version": "1.0.0",
|
||||
"notes": "Admin exec unreachable when ALLOW_EXEC!=true",
|
||||
"sinks": [
|
||||
{
|
||||
"sink_id": "ExpressGuarded::exec",
|
||||
"label": "unreachable",
|
||||
"confidence": "high",
|
||||
"dynamic_evidence": {
|
||||
"covered_by_tests": [
|
||||
"tests/test_unreachable.js"
|
||||
],
|
||||
"coverage_files": [
|
||||
"outputs/coverage.json"
|
||||
]
|
||||
},
|
||||
"static_evidence": {
|
||||
"call_path": [
|
||||
"POST /api/admin/exec",
|
||||
"createServer.exec",
|
||||
"guard: ALLOW_EXEC!=true"
|
||||
]
|
||||
},
|
||||
"config_conditions": [
|
||||
"ALLOW_EXEC == 'true'"
|
||||
],
|
||||
"notes": "Only reachable when ALLOW_EXEC=true"
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,34 @@
|
||||
{
|
||||
"version": "1.0.0",
|
||||
"cases": [
|
||||
{
|
||||
"case_id": "js-fastify-template:005",
|
||||
"case_version": "1.0.0",
|
||||
"notes": "Template rendering reachable",
|
||||
"sinks": [
|
||||
{
|
||||
"sink_id": "FastifyTemplate::render",
|
||||
"label": "reachable",
|
||||
"confidence": "high",
|
||||
"dynamic_evidence": {
|
||||
"covered_by_tests": [
|
||||
"tests/test_reach.js"
|
||||
],
|
||||
"coverage_files": [
|
||||
"outputs/coverage.json"
|
||||
]
|
||||
},
|
||||
"static_evidence": {
|
||||
"call_path": [
|
||||
"POST /api/render",
|
||||
"createServer.render",
|
||||
"template replace"
|
||||
]
|
||||
},
|
||||
"config_conditions": [],
|
||||
"notes": "Simple template replace used as sink"
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,36 @@
|
||||
{
|
||||
"version": "1.0.0",
|
||||
"cases": [
|
||||
{
|
||||
"case_id": "js-guarded-eval:002",
|
||||
"case_version": "1.0.0",
|
||||
"notes": "Eval sink guarded by FEATURE_ENABLE; unreachable when flag off",
|
||||
"sinks": [
|
||||
{
|
||||
"sink_id": "GuardedEval::handleRequest",
|
||||
"label": "unreachable",
|
||||
"confidence": "high",
|
||||
"dynamic_evidence": {
|
||||
"covered_by_tests": [
|
||||
"tests/test_unreachable.js"
|
||||
],
|
||||
"coverage_files": [
|
||||
"outputs/coverage.json"
|
||||
]
|
||||
},
|
||||
"static_evidence": {
|
||||
"call_path": [
|
||||
"POST /api/exec",
|
||||
"app.js::handleRequest",
|
||||
"guard: FEATURE_ENABLE != 1"
|
||||
]
|
||||
},
|
||||
"config_conditions": [
|
||||
"FEATURE_ENABLE == '1'"
|
||||
],
|
||||
"notes": "Sink only executes when FEATURE_ENABLE=1"
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,34 @@
|
||||
{
|
||||
"version": "1.0.0",
|
||||
"cases": [
|
||||
{
|
||||
"case_id": "js-unsafe-eval:001",
|
||||
"case_version": "1.0.0",
|
||||
"notes": "Unsafe eval sink reachable via POST /api/exec",
|
||||
"sinks": [
|
||||
{
|
||||
"sink_id": "UnsafeEval::handleRequest",
|
||||
"label": "reachable",
|
||||
"confidence": "high",
|
||||
"dynamic_evidence": {
|
||||
"covered_by_tests": [
|
||||
"tests/test_reach.js"
|
||||
],
|
||||
"coverage_files": [
|
||||
"outputs/coverage.json"
|
||||
]
|
||||
},
|
||||
"static_evidence": {
|
||||
"call_path": [
|
||||
"POST /api/exec",
|
||||
"app.js::handleRequest",
|
||||
"eval(code)"
|
||||
]
|
||||
},
|
||||
"config_conditions": [],
|
||||
"notes": "No guards; direct eval on user input"
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,34 @@
|
||||
{
|
||||
"version": "1.0.0",
|
||||
"cases": [
|
||||
{
|
||||
"case_id": "py-django-ssti:105",
|
||||
"case_version": "1.0.0",
|
||||
"notes": "Template rendering reachable (autoescape off)",
|
||||
"sinks": [
|
||||
{
|
||||
"sink_id": "DjangoSSTI::render",
|
||||
"label": "reachable",
|
||||
"confidence": "high",
|
||||
"dynamic_evidence": {
|
||||
"covered_by_tests": [
|
||||
"tests/test_reach.py"
|
||||
],
|
||||
"coverage_files": [
|
||||
"outputs/coverage.json"
|
||||
]
|
||||
},
|
||||
"static_evidence": {
|
||||
"call_path": [
|
||||
"POST /render",
|
||||
"app.handle_request",
|
||||
"render"
|
||||
]
|
||||
},
|
||||
"config_conditions": [],
|
||||
"notes": "Autoescape disabled"
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,36 @@
|
||||
{
|
||||
"version": "1.0.0",
|
||||
"cases": [
|
||||
{
|
||||
"case_id": "py-fastapi-guarded:104",
|
||||
"case_version": "1.0.0",
|
||||
"notes": "Eval unreachable unless ALLOW_EXEC=true",
|
||||
"sinks": [
|
||||
{
|
||||
"sink_id": "FastApiGuarded::handle_request",
|
||||
"label": "unreachable",
|
||||
"confidence": "high",
|
||||
"dynamic_evidence": {
|
||||
"covered_by_tests": [
|
||||
"tests/test_unreachable.py"
|
||||
],
|
||||
"coverage_files": [
|
||||
"outputs/coverage.json"
|
||||
]
|
||||
},
|
||||
"static_evidence": {
|
||||
"call_path": [
|
||||
"POST /exec",
|
||||
"app.handle_request",
|
||||
"guard: ALLOW_EXEC!=true"
|
||||
]
|
||||
},
|
||||
"config_conditions": [
|
||||
"ALLOW_EXEC == 'true'"
|
||||
],
|
||||
"notes": "Feature flag blocks sink by default"
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,34 @@
|
||||
{
|
||||
"version": "1.0.0",
|
||||
"cases": [
|
||||
{
|
||||
"case_id": "py-flask-template:103",
|
||||
"case_version": "1.0.0",
|
||||
"notes": "Template rendering reachable",
|
||||
"sinks": [
|
||||
{
|
||||
"sink_id": "FlaskTemplate::render",
|
||||
"label": "reachable",
|
||||
"confidence": "high",
|
||||
"dynamic_evidence": {
|
||||
"covered_by_tests": [
|
||||
"tests/test_reach.py"
|
||||
],
|
||||
"coverage_files": [
|
||||
"outputs/coverage.json"
|
||||
]
|
||||
},
|
||||
"static_evidence": {
|
||||
"call_path": [
|
||||
"POST /render",
|
||||
"app.handle_request",
|
||||
"render"
|
||||
]
|
||||
},
|
||||
"config_conditions": [],
|
||||
"notes": "Simple template placeholder replacement"
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,36 @@
|
||||
{
|
||||
"version": "1.0.0",
|
||||
"cases": [
|
||||
{
|
||||
"case_id": "py-guarded-exec:102",
|
||||
"case_version": "1.0.0",
|
||||
"notes": "Eval unreachable unless FEATURE_ENABLE=1",
|
||||
"sinks": [
|
||||
{
|
||||
"sink_id": "PyGuardedExec::handle_request",
|
||||
"label": "unreachable",
|
||||
"confidence": "high",
|
||||
"dynamic_evidence": {
|
||||
"covered_by_tests": [
|
||||
"tests/test_unreachable.py"
|
||||
],
|
||||
"coverage_files": [
|
||||
"outputs/coverage.json"
|
||||
]
|
||||
},
|
||||
"static_evidence": {
|
||||
"call_path": [
|
||||
"POST /api/exec",
|
||||
"app.handle_request",
|
||||
"guard: FEATURE_ENABLE != 1"
|
||||
]
|
||||
},
|
||||
"config_conditions": [
|
||||
"FEATURE_ENABLE == '1'"
|
||||
],
|
||||
"notes": "Feature flag required"
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,34 @@
|
||||
{
|
||||
"version": "1.0.0",
|
||||
"cases": [
|
||||
{
|
||||
"case_id": "py-unsafe-exec:101",
|
||||
"case_version": "1.0.0",
|
||||
"notes": "Eval reachable",
|
||||
"sinks": [
|
||||
{
|
||||
"sink_id": "PyUnsafeExec::handle_request",
|
||||
"label": "reachable",
|
||||
"confidence": "high",
|
||||
"dynamic_evidence": {
|
||||
"covered_by_tests": [
|
||||
"tests/test_reach.py"
|
||||
],
|
||||
"coverage_files": [
|
||||
"outputs/coverage.json"
|
||||
]
|
||||
},
|
||||
"static_evidence": {
|
||||
"call_path": [
|
||||
"POST /api/exec",
|
||||
"app.handle_request",
|
||||
"eval(code)"
|
||||
]
|
||||
},
|
||||
"config_conditions": [],
|
||||
"notes": "No guards"
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,38 @@
|
||||
id: "java-spring-deserialize:201"
|
||||
language: java
|
||||
project: spring-deserialize
|
||||
version: "1.0.0"
|
||||
description: "Java deserialization sink reachable via POST /api/upload"
|
||||
entrypoints:
|
||||
- "POST /api/upload"
|
||||
sinks:
|
||||
- id: "JavaDeserialize::handleRequest"
|
||||
path: "bench.reachability.App.handleRequest"
|
||||
kind: "custom"
|
||||
location:
|
||||
file: src/App.java
|
||||
line: 9
|
||||
notes: "java.io.ObjectInputStream on user-controlled payload"
|
||||
environment:
|
||||
os_image: "eclipse-temurin:21-jdk"
|
||||
runtime:
|
||||
java: "21"
|
||||
source_date_epoch: 1730000000
|
||||
build:
|
||||
command: "./build/build.sh"
|
||||
source_date_epoch: 1730000000
|
||||
outputs:
|
||||
artifact_path: outputs/binary.tar.gz
|
||||
sbom_path: outputs/sbom.cdx.json
|
||||
coverage_path: outputs/coverage.json
|
||||
traces_dir: outputs/traces
|
||||
test:
|
||||
command: "./build/build.sh"
|
||||
expected_coverage: []
|
||||
expected_traces: []
|
||||
env:
|
||||
JAVA_TOOL_OPTIONS: "-ea"
|
||||
ground_truth:
|
||||
summary: "Deserialization reachable"
|
||||
evidence_files:
|
||||
- "../benchmark/truth/java-spring-deserialize.json"
|
||||
@@ -0,0 +1,8 @@
|
||||
case_id: "java-spring-deserialize:201"
|
||||
entries:
|
||||
http:
|
||||
- id: "POST /api/upload"
|
||||
route: "/api/upload"
|
||||
method: "POST"
|
||||
handler: "App.handleRequest"
|
||||
description: "Binary payload base64-deserialized"
|
||||
@@ -0,0 +1,12 @@
|
||||
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
|
||||
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
|
||||
<modelVersion>4.0.0</modelVersion>
|
||||
<groupId>org.stellaops.bench</groupId>
|
||||
<artifactId>spring-deserialize</artifactId>
|
||||
<version>1.0.0</version>
|
||||
<packaging>jar</packaging>
|
||||
<properties>
|
||||
<maven.compiler.source>17</maven.compiler.source>
|
||||
<maven.compiler.target>17</maven.compiler.target>
|
||||
</properties>
|
||||
</project>
|
||||
@@ -0,0 +1,26 @@
|
||||
package bench.reachability;
|
||||
|
||||
import java.util.Map;
|
||||
import java.util.Base64;
|
||||
import java.io.*;
|
||||
|
||||
public class App {
|
||||
// Unsafe Java deserialization sink (reachable)
|
||||
public static Response handleRequest(Map<String, String> body) {
|
||||
String payload = body.get("payload");
|
||||
if (payload == null) {
|
||||
return new Response(400, "bad request");
|
||||
}
|
||||
try {
|
||||
byte[] data = Base64.getDecoder().decode(payload);
|
||||
ObjectInputStream ois = new ObjectInputStream(new ByteArrayInputStream(data));
|
||||
Object obj = ois.readObject();
|
||||
ois.close();
|
||||
return new Response(200, obj.toString());
|
||||
} catch (Exception ex) {
|
||||
return new Response(500, ex.getClass().getSimpleName());
|
||||
}
|
||||
}
|
||||
|
||||
public record Response(int status, String body) {}
|
||||
}
|
||||
@@ -0,0 +1,30 @@
|
||||
package bench.reachability;
|
||||
|
||||
import java.io.*;
|
||||
import java.util.*;
|
||||
import java.util.Base64;
|
||||
|
||||
// Simple hand-rolled test harness (no external deps) using Java assertions.
|
||||
public class AppTest {
|
||||
private static String serialize(Object obj) throws IOException {
|
||||
ByteArrayOutputStream bos = new ByteArrayOutputStream();
|
||||
ObjectOutputStream oos = new ObjectOutputStream(bos);
|
||||
oos.writeObject(obj);
|
||||
oos.close();
|
||||
return Base64.getEncoder().encodeToString(bos.toByteArray());
|
||||
}
|
||||
|
||||
public static void main(String[] args) throws Exception {
|
||||
String payload = serialize("hello");
|
||||
Map<String, String> body = Map.of("payload", payload);
|
||||
var res = App.handleRequest(body);
|
||||
assert res.status() == 200 : "status";
|
||||
assert res.body().equals("hello") : "body";
|
||||
// Emit a simple marker file for trace/coverage stand-ins
|
||||
File outDir = new File("outputs");
|
||||
outDir.mkdirs();
|
||||
try (FileWriter fw = new FileWriter(new File(outDir, "SINK_REACHED"))) {
|
||||
fw.write("true");
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,38 @@
|
||||
id: "java-spring-guarded:202"
|
||||
language: java
|
||||
project: spring-guarded
|
||||
version: "1.0.0"
|
||||
description: "Java deserialization guarded by ALLOW_DESER flag (unreachable by default)"
|
||||
entrypoints:
|
||||
- "POST /api/upload"
|
||||
sinks:
|
||||
- id: "JavaDeserializeGuarded::handleRequest"
|
||||
path: "bench.reachability.App.handleRequest"
|
||||
kind: "custom"
|
||||
location:
|
||||
file: src/App.java
|
||||
line: 9
|
||||
notes: "ObjectInputStream gated by ALLOW_DESER"
|
||||
environment:
|
||||
os_image: "eclipse-temurin:21-jdk"
|
||||
runtime:
|
||||
java: "21"
|
||||
source_date_epoch: 1730000000
|
||||
build:
|
||||
command: "./build/build.sh"
|
||||
source_date_epoch: 1730000000
|
||||
outputs:
|
||||
artifact_path: outputs/binary.tar.gz
|
||||
sbom_path: outputs/sbom.cdx.json
|
||||
coverage_path: outputs/coverage.json
|
||||
traces_dir: outputs/traces
|
||||
test:
|
||||
command: "./build/build.sh"
|
||||
expected_coverage: []
|
||||
expected_traces: []
|
||||
env:
|
||||
JAVA_TOOL_OPTIONS: "-ea"
|
||||
ground_truth:
|
||||
summary: "Guard blocks deserialization unless ALLOW_DESER=true"
|
||||
evidence_files:
|
||||
- "../benchmark/truth/java-spring-guarded.json"
|
||||
@@ -0,0 +1,8 @@
|
||||
case_id: "java-spring-guarded:202"
|
||||
entries:
|
||||
http:
|
||||
- id: "POST /api/upload"
|
||||
route: "/api/upload"
|
||||
method: "POST"
|
||||
handler: "App.handleRequest"
|
||||
description: "Base64 payload deserialization guarded by ALLOW_DESER"
|
||||
@@ -0,0 +1,12 @@
|
||||
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
|
||||
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
|
||||
<modelVersion>4.0.0</modelVersion>
|
||||
<groupId>org.stellaops.bench</groupId>
|
||||
<artifactId>spring-guarded</artifactId>
|
||||
<version>1.0.0</version>
|
||||
<packaging>jar</packaging>
|
||||
<properties>
|
||||
<maven.compiler.source>17</maven.compiler.source>
|
||||
<maven.compiler.target>17</maven.compiler.target>
|
||||
</properties>
|
||||
</project>
|
||||
@@ -0,0 +1,29 @@
|
||||
package bench.reachability;
|
||||
|
||||
import java.util.Map;
|
||||
import java.util.Base64;
|
||||
import java.io.*;
|
||||
|
||||
public class App {
|
||||
// Deserialization sink guarded by feature flag
|
||||
public static Response handleRequest(Map<String, String> body, Map<String, String> env) {
|
||||
if (!"true".equals(env.getOrDefault("ALLOW_DESER", "false"))) {
|
||||
return new Response(403, "forbidden");
|
||||
}
|
||||
String payload = body.get("payload");
|
||||
if (payload == null) {
|
||||
return new Response(400, "bad request");
|
||||
}
|
||||
try {
|
||||
byte[] data = Base64.getDecoder().decode(payload);
|
||||
ObjectInputStream ois = new ObjectInputStream(new ByteArrayInputStream(data));
|
||||
Object obj = ois.readObject();
|
||||
ois.close();
|
||||
return new Response(200, obj.toString());
|
||||
} catch (Exception ex) {
|
||||
return new Response(500, ex.getClass().getSimpleName());
|
||||
}
|
||||
}
|
||||
|
||||
public record Response(int status, String body) {}
|
||||
}
|
||||
@@ -0,0 +1,29 @@
|
||||
package bench.reachability;
|
||||
|
||||
import java.io.*;
|
||||
import java.util.*;
|
||||
import java.util.Base64;
|
||||
|
||||
public class AppTest {
|
||||
private static String serialize(Object obj) throws IOException {
|
||||
ByteArrayOutputStream bos = new ByteArrayOutputStream();
|
||||
ObjectOutputStream oos = new ObjectOutputStream(bos);
|
||||
oos.writeObject(obj);
|
||||
oos.close();
|
||||
return Base64.getEncoder().encodeToString(bos.toByteArray());
|
||||
}
|
||||
|
||||
public static void main(String[] args) throws Exception {
|
||||
String payload = serialize("hi");
|
||||
Map<String, String> body = Map.of("payload", payload);
|
||||
Map<String, String> env = Map.of("ALLOW_DESER", "false");
|
||||
var res = App.handleRequest(body, env);
|
||||
assert res.status() == 403 : "status";
|
||||
assert res.body().equals("forbidden") : "body";
|
||||
File outDir = new File("outputs");
|
||||
outDir.mkdirs();
|
||||
try (FileWriter fw = new FileWriter(new File(outDir, "SINK_BLOCKED"))) {
|
||||
fw.write("true");
|
||||
}
|
||||
}
|
||||
}
|
||||
38
bench/reachability-benchmark/cases/js/express-eval/case.yaml
Normal file
38
bench/reachability-benchmark/cases/js/express-eval/case.yaml
Normal file
@@ -0,0 +1,38 @@
|
||||
id: "js-express-eval:003"
|
||||
language: js
|
||||
project: express-eval
|
||||
version: "1.0.0"
|
||||
description: "Admin exec endpoint evaluates user code"
|
||||
entrypoints:
|
||||
- "POST /api/admin/exec"
|
||||
sinks:
|
||||
- id: "ExpressEval::exec"
|
||||
path: "src/app.js::createServer"
|
||||
kind: "process"
|
||||
location:
|
||||
file: src/app.js
|
||||
line: 17
|
||||
notes: "eval(code) on admin path"
|
||||
environment:
|
||||
os_image: "node:20-alpine"
|
||||
runtime:
|
||||
node: "20.11.0"
|
||||
source_date_epoch: 1730000000
|
||||
build:
|
||||
command: "./build/build.sh"
|
||||
source_date_epoch: 1730000000
|
||||
outputs:
|
||||
artifact_path: outputs/binary.tar.gz
|
||||
sbom_path: outputs/sbom.cdx.json
|
||||
coverage_path: outputs/coverage.json
|
||||
traces_dir: outputs/traces
|
||||
test:
|
||||
command: "./tests/run-tests.sh"
|
||||
expected_coverage:
|
||||
- outputs/coverage.json
|
||||
expected_traces:
|
||||
- outputs/traces/traces.json
|
||||
ground_truth:
|
||||
summary: "Admin exec endpoint reachable and executes eval"
|
||||
evidence_files:
|
||||
- "../benchmark/truth/js-express-eval.json"
|
||||
@@ -0,0 +1,8 @@
|
||||
case_id: "js-express-eval:003"
|
||||
entries:
|
||||
http:
|
||||
- id: "POST /api/admin/exec"
|
||||
route: "/api/admin/exec"
|
||||
method: "POST"
|
||||
handler: "createServer.exec"
|
||||
description: "Admin-only exec (reachable)"
|
||||
@@ -0,0 +1,9 @@
|
||||
{
|
||||
"name": "rb-case-express-eval",
|
||||
"version": "1.0.0",
|
||||
"description": "Reachability benchmark case: express-like admin eval endpoint",
|
||||
"license": "Apache-2.0",
|
||||
"scripts": {
|
||||
"test": "./tests/run-tests.sh"
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,34 @@
|
||||
'use strict';
|
||||
|
||||
// Minimal express-like router.
|
||||
function makeApp() {
|
||||
const routes = {};
|
||||
return {
|
||||
post(path, handler) {
|
||||
routes[`POST ${path}`] = handler;
|
||||
},
|
||||
handle(method, path, req, res) {
|
||||
const key = `${method} ${path}`;
|
||||
if (routes[key]) {
|
||||
return routes[key](req, res);
|
||||
}
|
||||
return { status: 404, body: 'not found' };
|
||||
}
|
||||
};
|
||||
}
|
||||
|
||||
function createServer() {
|
||||
const app = makeApp();
|
||||
app.post('/api/admin/exec', (req) => {
|
||||
if (!req || typeof req.body?.code !== 'string') {
|
||||
return { status: 400, body: 'bad request' };
|
||||
}
|
||||
// Sink: eval on admin endpoint (reachable)
|
||||
// eslint-disable-next-line no-eval
|
||||
const result = eval(req.body.code);
|
||||
return { status: 200, body: String(result) };
|
||||
});
|
||||
return app;
|
||||
}
|
||||
|
||||
module.exports = { createServer };
|
||||
@@ -0,0 +1,7 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
cd "$(dirname "$0")"
|
||||
export SOURCE_DATE_EPOCH=${SOURCE_DATE_EPOCH:-1730000000}
|
||||
export TZ=UTC
|
||||
export LC_ALL=C
|
||||
node test_reach.js
|
||||
@@ -0,0 +1,54 @@
|
||||
'use strict';
|
||||
|
||||
const assert = require('assert');
|
||||
const fs = require('fs');
|
||||
const path = require('path');
|
||||
const { createServer } = require('../src/app');
|
||||
|
||||
const OUT_DIR = path.resolve(__dirname, '../outputs');
|
||||
const TRACE_DIR = path.join(OUT_DIR, 'traces');
|
||||
const COVERAGE_FILE = path.join(OUT_DIR, 'coverage.json');
|
||||
const TRACE_FILE = path.join(TRACE_DIR, 'traces.json');
|
||||
|
||||
function ensureDirs() {
|
||||
fs.mkdirSync(OUT_DIR, { recursive: true });
|
||||
fs.mkdirSync(TRACE_DIR, { recursive: true });
|
||||
}
|
||||
|
||||
function recordTrace(entry, pathNodes) {
|
||||
fs.writeFileSync(
|
||||
TRACE_FILE,
|
||||
JSON.stringify({
|
||||
entry,
|
||||
path: pathNodes,
|
||||
sink: 'ExpressEval::exec',
|
||||
notes: 'Admin exec reached'
|
||||
}, null, 2)
|
||||
);
|
||||
}
|
||||
|
||||
function recordCoverage(filePath, lines) {
|
||||
fs.writeFileSync(
|
||||
COVERAGE_FILE,
|
||||
JSON.stringify({
|
||||
files: {
|
||||
[filePath]: {
|
||||
lines_covered: lines,
|
||||
lines_total: 40
|
||||
}
|
||||
}
|
||||
}, null, 2)
|
||||
);
|
||||
}
|
||||
|
||||
(function main() {
|
||||
ensureDirs();
|
||||
const app = createServer();
|
||||
const res = app.handle('POST', '/api/admin/exec', { body: { code: '21*2' } });
|
||||
assert.strictEqual(res.status, 200);
|
||||
assert.strictEqual(res.body, '42');
|
||||
|
||||
recordTrace('POST /api/admin/exec', ['app.js::createServer', 'handler', 'eval(code)']);
|
||||
recordCoverage('src/app.js', [5, 6, 7, 13, 18, 19]);
|
||||
fs.writeFileSync(path.join(OUT_DIR, 'SINK_REACHED'), 'true');
|
||||
})();
|
||||
@@ -0,0 +1,38 @@
|
||||
id: "js-express-guarded:004"
|
||||
language: js
|
||||
project: express-guarded
|
||||
version: "1.0.0"
|
||||
description: "Admin exec guarded by ALLOW_EXEC flag; unreachable by default"
|
||||
entrypoints:
|
||||
- "POST /api/admin/exec"
|
||||
sinks:
|
||||
- id: "ExpressGuarded::exec"
|
||||
path: "src/app.js::createServer"
|
||||
kind: "process"
|
||||
location:
|
||||
file: src/app.js
|
||||
line: 16
|
||||
notes: "eval(code) gated by ALLOW_EXEC"
|
||||
environment:
|
||||
os_image: "node:20-alpine"
|
||||
runtime:
|
||||
node: "20.11.0"
|
||||
source_date_epoch: 1730000000
|
||||
build:
|
||||
command: "./build/build.sh"
|
||||
source_date_epoch: 1730000000
|
||||
outputs:
|
||||
artifact_path: outputs/binary.tar.gz
|
||||
sbom_path: outputs/sbom.cdx.json
|
||||
coverage_path: outputs/coverage.json
|
||||
traces_dir: outputs/traces
|
||||
test:
|
||||
command: "./tests/run-tests.sh"
|
||||
expected_coverage:
|
||||
- outputs/coverage.json
|
||||
expected_traces:
|
||||
- outputs/traces/traces.json
|
||||
ground_truth:
|
||||
summary: "Guard prevents sink unless ALLOW_EXEC=true"
|
||||
evidence_files:
|
||||
- "../benchmark/truth/js-express-guarded.json"
|
||||
@@ -0,0 +1,8 @@
|
||||
case_id: "js-express-guarded:004"
|
||||
entries:
|
||||
http:
|
||||
- id: "POST /api/admin/exec"
|
||||
route: "/api/admin/exec"
|
||||
method: "POST"
|
||||
handler: "createServer.exec"
|
||||
description: "Admin exec blocked unless ALLOW_EXEC=true"
|
||||
@@ -0,0 +1,9 @@
|
||||
{
|
||||
"name": "rb-case-express-guarded",
|
||||
"version": "1.0.0",
|
||||
"description": "Reachability benchmark case: express-like admin exec guarded by env flag",
|
||||
"license": "Apache-2.0",
|
||||
"scripts": {
|
||||
"test": "./tests/run-tests.sh"
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,33 @@
|
||||
'use strict';
|
||||
|
||||
function makeApp() {
|
||||
const routes = {};
|
||||
return {
|
||||
post(path, handler) {
|
||||
routes[`POST ${path}`] = handler;
|
||||
},
|
||||
handle(method, path, req) {
|
||||
const key = `${method} ${path}`;
|
||||
if (routes[key]) return routes[key](req);
|
||||
return { status: 404, body: 'not found' };
|
||||
}
|
||||
};
|
||||
}
|
||||
|
||||
function createServer() {
|
||||
const app = makeApp();
|
||||
app.post('/api/admin/exec', (req) => {
|
||||
if (req?.env?.ALLOW_EXEC !== 'true') {
|
||||
return { status: 403, body: 'forbidden' };
|
||||
}
|
||||
if (typeof req?.body?.code !== 'string') {
|
||||
return { status: 400, body: 'bad request' };
|
||||
}
|
||||
// eslint-disable-next-line no-eval
|
||||
const result = eval(req.body.code);
|
||||
return { status: 200, body: String(result) };
|
||||
});
|
||||
return app;
|
||||
}
|
||||
|
||||
module.exports = { createServer };
|
||||
@@ -0,0 +1,7 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
cd "$(dirname "$0")"
|
||||
export SOURCE_DATE_EPOCH=${SOURCE_DATE_EPOCH:-1730000000}
|
||||
export TZ=UTC
|
||||
export LC_ALL=C
|
||||
node test_unreachable.js
|
||||
@@ -0,0 +1,53 @@
|
||||
'use strict';
|
||||
|
||||
const assert = require('assert');
|
||||
const fs = require('fs');
|
||||
const path = require('path');
|
||||
const { createServer } = require('../src/app');
|
||||
|
||||
const OUT_DIR = path.resolve(__dirname, '../outputs');
|
||||
const TRACE_DIR = path.join(OUT_DIR, 'traces');
|
||||
const COVERAGE_FILE = path.join(OUT_DIR, 'coverage.json');
|
||||
const TRACE_FILE = path.join(TRACE_DIR, 'traces.json');
|
||||
|
||||
function ensureDirs() {
|
||||
fs.mkdirSync(OUT_DIR, { recursive: true });
|
||||
fs.mkdirSync(TRACE_DIR, { recursive: true });
|
||||
}
|
||||
|
||||
function recordTrace(entry, pathNodes) {
|
||||
fs.writeFileSync(
|
||||
TRACE_FILE,
|
||||
JSON.stringify({
|
||||
entry,
|
||||
path: pathNodes,
|
||||
sink: 'ExpressGuarded::exec',
|
||||
notes: 'Guard blocked sink'
|
||||
}, null, 2)
|
||||
);
|
||||
}
|
||||
|
||||
function recordCoverage(filePath, lines) {
|
||||
fs.writeFileSync(
|
||||
COVERAGE_FILE,
|
||||
JSON.stringify({
|
||||
files: {
|
||||
[filePath]: {
|
||||
lines_covered: lines,
|
||||
lines_total: 50
|
||||
}
|
||||
}
|
||||
}, null, 2)
|
||||
);
|
||||
}
|
||||
|
||||
(function main() {
|
||||
ensureDirs();
|
||||
const app = createServer();
|
||||
const res = app.handle('POST', '/api/admin/exec', { body: { code: '2+2' }, env: { ALLOW_EXEC: 'false' } });
|
||||
assert.strictEqual(res.status, 403);
|
||||
assert.strictEqual(res.body, 'forbidden');
|
||||
|
||||
recordTrace('POST /api/admin/exec', ['app.js::createServer', 'guard: ALLOW_EXEC!=true']);
|
||||
recordCoverage('src/app.js', [5,6,7,12,13,14,15]);
|
||||
})();
|
||||
@@ -0,0 +1,38 @@
|
||||
id: "js-fastify-template:005"
|
||||
language: js
|
||||
project: fastify-template
|
||||
version: "1.0.0"
|
||||
description: "Template rendering route replaces user placeholder"
|
||||
entrypoints:
|
||||
- "POST /api/render"
|
||||
sinks:
|
||||
- id: "FastifyTemplate::render"
|
||||
path: "src/app.js::createServer"
|
||||
kind: "http"
|
||||
location:
|
||||
file: src/app.js
|
||||
line: 19
|
||||
notes: "Template rendering of user input"
|
||||
environment:
|
||||
os_image: "node:20-alpine"
|
||||
runtime:
|
||||
node: "20.11.0"
|
||||
source_date_epoch: 1730000000
|
||||
build:
|
||||
command: "./build/build.sh"
|
||||
source_date_epoch: 1730000000
|
||||
outputs:
|
||||
artifact_path: outputs/binary.tar.gz
|
||||
sbom_path: outputs/sbom.cdx.json
|
||||
coverage_path: outputs/coverage.json
|
||||
traces_dir: outputs/traces
|
||||
test:
|
||||
command: "./tests/run-tests.sh"
|
||||
expected_coverage:
|
||||
- outputs/coverage.json
|
||||
expected_traces:
|
||||
- outputs/traces/traces.json
|
||||
ground_truth:
|
||||
summary: "Template rendering reachable via POST /api/render"
|
||||
evidence_files:
|
||||
- "../benchmark/truth/js-fastify-template.json"
|
||||
@@ -0,0 +1,8 @@
|
||||
case_id: "js-fastify-template:005"
|
||||
entries:
|
||||
http:
|
||||
- id: "POST /api/render"
|
||||
route: "/api/render"
|
||||
method: "POST"
|
||||
handler: "createServer.render"
|
||||
description: "Template rendering endpoint"
|
||||
@@ -0,0 +1,9 @@
|
||||
{
|
||||
"name": "rb-case-fastify-template",
|
||||
"version": "1.0.0",
|
||||
"description": "Reachability benchmark case: fastify-like template rendering",
|
||||
"license": "Apache-2.0",
|
||||
"scripts": {
|
||||
"test": "./tests/run-tests.sh"
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,33 @@
|
||||
'use strict';
|
||||
|
||||
// Simulated Fastify route registration for template injection.
|
||||
function buildServer() {
|
||||
const routes = {};
|
||||
return {
|
||||
post(path, handler) {
|
||||
routes[`POST ${path}`] = handler;
|
||||
},
|
||||
inject(method, path, payload) {
|
||||
const key = `${method} ${path}`;
|
||||
const handler = routes[key];
|
||||
if (!handler) return { status: 404, body: 'not found' };
|
||||
return handler({ body: payload });
|
||||
}
|
||||
};
|
||||
}
|
||||
|
||||
function createServer() {
|
||||
const server = buildServer();
|
||||
server.post('/api/render', (req) => {
|
||||
const template = req?.body?.template;
|
||||
if (typeof template !== 'string') {
|
||||
return { status: 400, body: 'bad request' };
|
||||
}
|
||||
const compiled = template.replace('{{user}}', 'guest');
|
||||
// Sink: writes rendered content to log (simulated SSR)
|
||||
return { status: 200, body: compiled };
|
||||
});
|
||||
return server;
|
||||
}
|
||||
|
||||
module.exports = { createServer };
|
||||
@@ -0,0 +1,7 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
cd "$(dirname "$0")"
|
||||
export SOURCE_DATE_EPOCH=${SOURCE_DATE_EPOCH:-1730000000}
|
||||
export TZ=UTC
|
||||
export LC_ALL=C
|
||||
node test_reach.js
|
||||
@@ -0,0 +1,54 @@
|
||||
'use strict';
|
||||
|
||||
const assert = require('assert');
|
||||
const fs = require('fs');
|
||||
const path = require('path');
|
||||
const { createServer } = require('../src/app');
|
||||
|
||||
const OUT_DIR = path.resolve(__dirname, '../outputs');
|
||||
const TRACE_DIR = path.join(OUT_DIR, 'traces');
|
||||
const COVERAGE_FILE = path.join(OUT_DIR, 'coverage.json');
|
||||
const TRACE_FILE = path.join(TRACE_DIR, 'traces.json');
|
||||
|
||||
function ensureDirs() {
|
||||
fs.mkdirSync(OUT_DIR, { recursive: true });
|
||||
fs.mkdirSync(TRACE_DIR, { recursive: true });
|
||||
}
|
||||
|
||||
function recordTrace(entry, pathNodes) {
|
||||
fs.writeFileSync(
|
||||
TRACE_FILE,
|
||||
JSON.stringify({
|
||||
entry,
|
||||
path: pathNodes,
|
||||
sink: 'FastifyTemplate::render',
|
||||
notes: 'Template rendered with user input'
|
||||
}, null, 2)
|
||||
);
|
||||
}
|
||||
|
||||
function recordCoverage(filePath, lines) {
|
||||
fs.writeFileSync(
|
||||
COVERAGE_FILE,
|
||||
JSON.stringify({
|
||||
files: {
|
||||
[filePath]: {
|
||||
lines_covered: lines,
|
||||
lines_total: 45
|
||||
}
|
||||
}
|
||||
}, null, 2)
|
||||
);
|
||||
}
|
||||
|
||||
(function main() {
|
||||
ensureDirs();
|
||||
const server = createServer();
|
||||
const res = server.inject('POST', '/api/render', { template: 'Hello {{user}}' });
|
||||
assert.strictEqual(res.status, 200);
|
||||
assert.strictEqual(res.body, 'Hello guest');
|
||||
|
||||
recordTrace('POST /api/render', ['app.js::createServer', 'render template']);
|
||||
recordCoverage('src/app.js', [5,6,7,13,18,20]);
|
||||
fs.writeFileSync(path.join(OUT_DIR, 'SINK_REACHED'), 'true');
|
||||
})();
|
||||
38
bench/reachability-benchmark/cases/js/guarded-eval/case.yaml
Normal file
38
bench/reachability-benchmark/cases/js/guarded-eval/case.yaml
Normal file
@@ -0,0 +1,38 @@
|
||||
id: "js-guarded-eval:002"
|
||||
language: js
|
||||
project: guarded-eval
|
||||
version: "1.0.0"
|
||||
description: "Eval sink guarded by FEATURE_ENABLE flag; unreachable when flag is off"
|
||||
entrypoints:
|
||||
- "POST /api/exec"
|
||||
sinks:
|
||||
- id: "GuardedEval::handleRequest"
|
||||
path: "src/app.js::handleRequest"
|
||||
kind: "process"
|
||||
location:
|
||||
file: src/app.js
|
||||
line: 13
|
||||
notes: "eval on user input guarded by FEATURE_ENABLE"
|
||||
environment:
|
||||
os_image: "node:20-alpine"
|
||||
runtime:
|
||||
node: "20.11.0"
|
||||
source_date_epoch: 1730000000
|
||||
build:
|
||||
command: "./build/build.sh"
|
||||
source_date_epoch: 1730000000
|
||||
outputs:
|
||||
artifact_path: outputs/binary.tar.gz
|
||||
sbom_path: outputs/sbom.cdx.json
|
||||
coverage_path: outputs/coverage.json
|
||||
traces_dir: outputs/traces
|
||||
test:
|
||||
command: "./tests/run-tests.sh"
|
||||
expected_coverage:
|
||||
- outputs/coverage.json
|
||||
expected_traces:
|
||||
- outputs/traces/traces.json
|
||||
ground_truth:
|
||||
summary: "Guard prevents sink when FEATURE_ENABLE != 1"
|
||||
evidence_files:
|
||||
- "../benchmark/truth/js-guarded-eval.json"
|
||||
@@ -0,0 +1,8 @@
|
||||
case_id: "js-guarded-eval:002"
|
||||
entries:
|
||||
http:
|
||||
- id: "POST /api/exec"
|
||||
route: "/api/exec"
|
||||
method: "POST"
|
||||
handler: "app.js::handleRequest"
|
||||
description: "Feature-flagged code execution endpoint"
|
||||
@@ -0,0 +1,9 @@
|
||||
{
|
||||
"name": "rb-case-guarded-eval",
|
||||
"version": "1.0.0",
|
||||
"description": "Reachability benchmark case: eval guarded by feature flag",
|
||||
"license": "Apache-2.0",
|
||||
"scripts": {
|
||||
"test": "./tests/run-tests.sh"
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,19 @@
|
||||
'use strict';
|
||||
|
||||
function handleRequest(body, env = process.env) {
|
||||
if (env.FEATURE_ENABLE !== '1') {
|
||||
return { status: 403, body: 'disabled' };
|
||||
}
|
||||
|
||||
const code = body && body.code;
|
||||
if (typeof code !== 'string') {
|
||||
return { status: 400, body: 'bad request' };
|
||||
}
|
||||
|
||||
// This sink is reachable only when FEATURE_ENABLE=1.
|
||||
// eslint-disable-next-line no-eval
|
||||
const result = eval(code);
|
||||
return { status: 200, body: String(result) };
|
||||
}
|
||||
|
||||
module.exports = { handleRequest };
|
||||
@@ -0,0 +1,7 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
cd "$(dirname "$0")"
|
||||
export SOURCE_DATE_EPOCH=${SOURCE_DATE_EPOCH:-1730000000}
|
||||
export TZ=UTC
|
||||
export LC_ALL=C
|
||||
node test_unreachable.js
|
||||
@@ -0,0 +1,54 @@
|
||||
'use strict';
|
||||
|
||||
const assert = require('assert');
|
||||
const fs = require('fs');
|
||||
const path = require('path');
|
||||
const { handleRequest } = require('../src/app');
|
||||
|
||||
const OUT_DIR = path.resolve(__dirname, '../outputs');
|
||||
const TRACE_DIR = path.join(OUT_DIR, 'traces');
|
||||
const COVERAGE_FILE = path.join(OUT_DIR, 'coverage.json');
|
||||
const TRACE_FILE = path.join(TRACE_DIR, 'traces.json');
|
||||
|
||||
function ensureDirs() {
|
||||
fs.mkdirSync(OUT_DIR, { recursive: true });
|
||||
fs.mkdirSync(TRACE_DIR, { recursive: true });
|
||||
}
|
||||
|
||||
function recordTrace(entry, pathNodes) {
|
||||
fs.writeFileSync(
|
||||
TRACE_FILE,
|
||||
JSON.stringify({
|
||||
entry,
|
||||
path: pathNodes,
|
||||
sink: 'GuardedEval::handleRequest',
|
||||
notes: 'Guard prevented sink execution'
|
||||
}, null, 2)
|
||||
);
|
||||
}
|
||||
|
||||
function recordCoverage(filePath, lines) {
|
||||
fs.writeFileSync(
|
||||
COVERAGE_FILE,
|
||||
JSON.stringify({
|
||||
files: {
|
||||
[filePath]: {
|
||||
lines_covered: lines,
|
||||
lines_total: 32
|
||||
}
|
||||
}
|
||||
}, null, 2)
|
||||
);
|
||||
}
|
||||
|
||||
(function main() {
|
||||
ensureDirs();
|
||||
const payload = { code: '1 + 2' };
|
||||
const response = handleRequest(payload, { FEATURE_ENABLE: '0' });
|
||||
assert.strictEqual(response.status, 403);
|
||||
assert.strictEqual(response.body, 'disabled');
|
||||
|
||||
// Record that the guard path was taken; no SINK_REACHED marker is written.
|
||||
recordTrace('POST /api/exec', ['app.js:handleRequest', 'guard: FEATURE_ENABLE != 1']);
|
||||
recordCoverage('src/app.js', [5, 6, 7, 9, 10, 11]);
|
||||
})();
|
||||
38
bench/reachability-benchmark/cases/js/unsafe-eval/case.yaml
Normal file
38
bench/reachability-benchmark/cases/js/unsafe-eval/case.yaml
Normal file
@@ -0,0 +1,38 @@
|
||||
id: "js-unsafe-eval:001"
|
||||
language: js
|
||||
project: unsafe-eval
|
||||
version: "1.0.0"
|
||||
description: "Minimal handler with unsafe eval sink reachable via POST /api/exec"
|
||||
entrypoints:
|
||||
- "POST /api/exec"
|
||||
sinks:
|
||||
- id: "UnsafeEval::handleRequest"
|
||||
path: "src/app.js::handleRequest"
|
||||
kind: "process"
|
||||
location:
|
||||
file: src/app.js
|
||||
line: 12
|
||||
notes: "eval on user-controlled input"
|
||||
environment:
|
||||
os_image: "node:20-alpine"
|
||||
runtime:
|
||||
node: "20.11.0"
|
||||
source_date_epoch: 1730000000
|
||||
build:
|
||||
command: "./build/build.sh"
|
||||
source_date_epoch: 1730000000
|
||||
outputs:
|
||||
artifact_path: outputs/binary.tar.gz
|
||||
sbom_path: outputs/sbom.cdx.json
|
||||
coverage_path: outputs/coverage.json
|
||||
traces_dir: outputs/traces
|
||||
test:
|
||||
command: "./tests/run-tests.sh"
|
||||
expected_coverage:
|
||||
- outputs/coverage.json
|
||||
expected_traces:
|
||||
- outputs/traces/traces.json
|
||||
ground_truth:
|
||||
summary: "Unit test triggers eval sink with payload {code: '1+2'}"
|
||||
evidence_files:
|
||||
- "../benchmark/truth/js-unsafe-eval.json"
|
||||
@@ -0,0 +1,8 @@
|
||||
case_id: "js-unsafe-eval:001"
|
||||
entries:
|
||||
http:
|
||||
- id: "POST /api/exec"
|
||||
route: "/api/exec"
|
||||
method: "POST"
|
||||
handler: "app.js::handleRequest"
|
||||
description: "Executes user-supplied code (unsafe eval)"
|
||||
@@ -0,0 +1,9 @@
|
||||
{
|
||||
"name": "rb-case-unsafe-eval",
|
||||
"version": "1.0.0",
|
||||
"description": "Reachability benchmark case: unsafe eval in minimal JS handler",
|
||||
"license": "Apache-2.0",
|
||||
"scripts": {
|
||||
"test": "./tests/run-tests.sh"
|
||||
}
|
||||
}
|
||||
17
bench/reachability-benchmark/cases/js/unsafe-eval/src/app.js
Normal file
17
bench/reachability-benchmark/cases/js/unsafe-eval/src/app.js
Normal file
@@ -0,0 +1,17 @@
|
||||
'use strict';
|
||||
|
||||
// Minimal HTTP-like handler exposing an unsafe eval sink for reachability.
|
||||
// The handler is intentionally small to avoid external dependencies.
|
||||
function handleRequest(body) {
|
||||
const code = body && body.code;
|
||||
if (typeof code !== 'string') {
|
||||
return { status: 400, body: 'bad request' };
|
||||
}
|
||||
|
||||
// Dangerous: executes user-controlled code. The test drives this sink.
|
||||
// eslint-disable-next-line no-eval
|
||||
const result = eval(code);
|
||||
return { status: 200, body: String(result) };
|
||||
}
|
||||
|
||||
module.exports = { handleRequest };
|
||||
@@ -0,0 +1,7 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
cd "$(dirname "$0")"
|
||||
export SOURCE_DATE_EPOCH=${SOURCE_DATE_EPOCH:-1730000000}
|
||||
export TZ=UTC
|
||||
export LC_ALL=C
|
||||
node test_reach.js
|
||||
@@ -0,0 +1,55 @@
|
||||
'use strict';
|
||||
|
||||
const assert = require('assert');
|
||||
const fs = require('fs');
|
||||
const path = require('path');
|
||||
const { handleRequest } = require('../src/app');
|
||||
|
||||
const OUT_DIR = path.resolve(__dirname, '../outputs');
|
||||
const TRACE_DIR = path.join(OUT_DIR, 'traces');
|
||||
const COVERAGE_FILE = path.join(OUT_DIR, 'coverage.json');
|
||||
const TRACE_FILE = path.join(TRACE_DIR, 'traces.json');
|
||||
|
||||
function ensureDirs() {
|
||||
fs.mkdirSync(OUT_DIR, { recursive: true });
|
||||
fs.mkdirSync(TRACE_DIR, { recursive: true });
|
||||
}
|
||||
|
||||
function recordTrace(entry, pathNodes) {
|
||||
fs.writeFileSync(
|
||||
TRACE_FILE,
|
||||
JSON.stringify({
|
||||
entry,
|
||||
path: pathNodes,
|
||||
sink: 'UnsafeEval::handleRequest',
|
||||
notes: 'Test-driven dynamic trace'
|
||||
}, null, 2)
|
||||
);
|
||||
}
|
||||
|
||||
function recordCoverage(filePath, lines) {
|
||||
fs.writeFileSync(
|
||||
COVERAGE_FILE,
|
||||
JSON.stringify({
|
||||
files: {
|
||||
[filePath]: {
|
||||
lines_covered: lines,
|
||||
lines_total: 30
|
||||
}
|
||||
}
|
||||
}, null, 2)
|
||||
);
|
||||
}
|
||||
|
||||
(function main() {
|
||||
ensureDirs();
|
||||
const payload = { code: '1 + 2' };
|
||||
const response = handleRequest(payload);
|
||||
assert.strictEqual(response.status, 200);
|
||||
assert.strictEqual(response.body, '3');
|
||||
|
||||
recordTrace('POST /api/exec', ['app.js:handleRequest', 'eval(code)']);
|
||||
recordCoverage('src/app.js', [5, 6, 7, 12, 15]);
|
||||
// Marker file proves sink executed
|
||||
fs.writeFileSync(path.join(OUT_DIR, 'SINK_REACHED'), 'true');
|
||||
})();
|
||||
38
bench/reachability-benchmark/cases/py/django-ssti/case.yaml
Normal file
38
bench/reachability-benchmark/cases/py/django-ssti/case.yaml
Normal file
@@ -0,0 +1,38 @@
|
||||
id: "py-django-ssti:105"
|
||||
language: py
|
||||
project: django-ssti
|
||||
version: "1.0.0"
|
||||
description: "Django-like template rendering (autoescape off) reachable"
|
||||
entrypoints:
|
||||
- "POST /render"
|
||||
sinks:
|
||||
- id: "DjangoSSTI::render"
|
||||
path: "src/app.py::handle_request"
|
||||
kind: "http"
|
||||
location:
|
||||
file: src/app.py
|
||||
line: 5
|
||||
notes: "template replace without escaping"
|
||||
environment:
|
||||
os_image: "python:3.12-alpine"
|
||||
runtime:
|
||||
python: "3.12"
|
||||
source_date_epoch: 1730000000
|
||||
build:
|
||||
command: "./build/build.sh"
|
||||
source_date_epoch: 1730000000
|
||||
outputs:
|
||||
artifact_path: outputs/binary.tar.gz
|
||||
sbom_path: outputs/sbom.cdx.json
|
||||
coverage_path: outputs/coverage.json
|
||||
traces_dir: outputs/traces
|
||||
test:
|
||||
command: "./tests/run-tests.sh"
|
||||
expected_coverage:
|
||||
- outputs/coverage.json
|
||||
expected_traces:
|
||||
- outputs/traces/traces.json
|
||||
ground_truth:
|
||||
summary: "Template rendering reachable with autoescape off"
|
||||
evidence_files:
|
||||
- "../benchmark/truth/py-django-ssti.json"
|
||||
@@ -0,0 +1,8 @@
|
||||
case_id: "py-django-ssti:105"
|
||||
entries:
|
||||
http:
|
||||
- id: "POST /render"
|
||||
route: "/render"
|
||||
method: "POST"
|
||||
handler: "app.handle_request"
|
||||
description: "Template rendering with autoescape off"
|
||||
@@ -0,0 +1 @@
|
||||
# stdlib only
|
||||
Binary file not shown.
12
bench/reachability-benchmark/cases/py/django-ssti/src/app.py
Normal file
12
bench/reachability-benchmark/cases/py/django-ssti/src/app.py
Normal file
@@ -0,0 +1,12 @@
|
||||
"""Django-like template rendering with autoescape off (reachable)."""
|
||||
|
||||
def render(template: str, context: dict) -> str:
|
||||
# naive render; simulates autoescape off
|
||||
return template.replace("{{user}}", context.get("user", "guest"))
|
||||
|
||||
def handle_request(body):
|
||||
template = body.get("template") if isinstance(body, dict) else None
|
||||
if not isinstance(template, str):
|
||||
return {"status": 400, "body": "bad request"}
|
||||
rendered = render(template, {"user": "guest"})
|
||||
return {"status": 200, "body": rendered}
|
||||
@@ -0,0 +1,8 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
cd "$(dirname "$0")"
|
||||
export SOURCE_DATE_EPOCH=${SOURCE_DATE_EPOCH:-1730000000}
|
||||
export TZ=UTC
|
||||
export LC_ALL=C
|
||||
export PYTHONPATH="$(cd .. && pwd)/src"
|
||||
python test_reach.py
|
||||
@@ -0,0 +1,48 @@
|
||||
import json
|
||||
import pathlib
|
||||
from app import handle_request
|
||||
|
||||
ROOT = pathlib.Path(__file__).resolve().parent.parent
|
||||
OUT = ROOT / "outputs"
|
||||
TRACE_DIR = OUT / "traces"
|
||||
COVERAGE_FILE = OUT / "coverage.json"
|
||||
TRACE_FILE = TRACE_DIR / "traces.json"
|
||||
|
||||
def ensure_dirs():
|
||||
OUT.mkdir(parents=True, exist_ok=True)
|
||||
TRACE_DIR.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
def record_trace(entry, path_nodes):
|
||||
TRACE_FILE.write_text(
|
||||
json.dumps({
|
||||
"entry": entry,
|
||||
"path": path_nodes,
|
||||
"sink": "DjangoSSTI::render",
|
||||
"notes": "Template rendered (autoescape off)"
|
||||
}, indent=2)
|
||||
)
|
||||
|
||||
def record_coverage(file_path, lines):
|
||||
COVERAGE_FILE.write_text(
|
||||
json.dumps({
|
||||
"files": {
|
||||
file_path: {
|
||||
"lines_covered": lines,
|
||||
"lines_total": 38
|
||||
}
|
||||
}
|
||||
}, indent=2)
|
||||
)
|
||||
|
||||
def test_reach():
|
||||
ensure_dirs()
|
||||
res = handle_request({"template": "Hello {{user}}"})
|
||||
assert res["status"] == 200
|
||||
assert res["body"] == "Hello guest"
|
||||
record_trace("POST /render", ["app.py::handle_request", "render"])
|
||||
record_coverage("src/app.py", [3,4,5,7,8,9,10])
|
||||
(OUT / "SINK_REACHED").write_text("true")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
test_reach()
|
||||
@@ -0,0 +1,38 @@
|
||||
id: "py-fastapi-guarded:104"
|
||||
language: py
|
||||
project: fastapi-guarded
|
||||
version: "1.0.0"
|
||||
description: "FastAPI-like exec guarded by ALLOW_EXEC flag (unreachable by default)"
|
||||
entrypoints:
|
||||
- "POST /exec"
|
||||
sinks:
|
||||
- id: "FastApiGuarded::handle_request"
|
||||
path: "src/app.py::handle_request"
|
||||
kind: "process"
|
||||
location:
|
||||
file: src/app.py
|
||||
line: 7
|
||||
notes: "eval guarded by ALLOW_EXEC"
|
||||
environment:
|
||||
os_image: "python:3.12-alpine"
|
||||
runtime:
|
||||
python: "3.12"
|
||||
source_date_epoch: 1730000000
|
||||
build:
|
||||
command: "./build/build.sh"
|
||||
source_date_epoch: 1730000000
|
||||
outputs:
|
||||
artifact_path: outputs/binary.tar.gz
|
||||
sbom_path: outputs/sbom.cdx.json
|
||||
coverage_path: outputs/coverage.json
|
||||
traces_dir: outputs/traces
|
||||
test:
|
||||
command: "./tests/run-tests.sh"
|
||||
expected_coverage:
|
||||
- outputs/coverage.json
|
||||
expected_traces:
|
||||
- outputs/traces/traces.json
|
||||
ground_truth:
|
||||
summary: "Guard blocks eval unless ALLOW_EXEC=true"
|
||||
evidence_files:
|
||||
- "../benchmark/truth/py-fastapi-guarded.json"
|
||||
@@ -0,0 +1,8 @@
|
||||
case_id: "py-fastapi-guarded:104"
|
||||
entries:
|
||||
http:
|
||||
- id: "POST /exec"
|
||||
route: "/exec"
|
||||
method: "POST"
|
||||
handler: "app.handle_request"
|
||||
description: "Exec guarded by ALLOW_EXEC"
|
||||
@@ -0,0 +1 @@
|
||||
# stdlib only
|
||||
Binary file not shown.
@@ -0,0 +1,11 @@
|
||||
"""FastAPI-like handler with feature flag guarding exec."""
|
||||
|
||||
def handle_request(body, env=None):
|
||||
env = env or {}
|
||||
if env.get("ALLOW_EXEC") != "true":
|
||||
return {"status": 403, "body": "forbidden"}
|
||||
code = body.get("code") if isinstance(body, dict) else None
|
||||
if not isinstance(code, str):
|
||||
return {"status": 400, "body": "bad request"}
|
||||
result = eval(code)
|
||||
return {"status": 200, "body": str(result)}
|
||||
@@ -0,0 +1,8 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
cd "$(dirname "$0")"
|
||||
export SOURCE_DATE_EPOCH=${SOURCE_DATE_EPOCH:-1730000000}
|
||||
export TZ=UTC
|
||||
export LC_ALL=C
|
||||
export PYTHONPATH="$(cd .. && pwd)/src"
|
||||
python test_unreachable.py
|
||||
@@ -0,0 +1,47 @@
|
||||
import json
|
||||
import pathlib
|
||||
from app import handle_request
|
||||
|
||||
ROOT = pathlib.Path(__file__).resolve().parent.parent
|
||||
OUT = ROOT / "outputs"
|
||||
TRACE_DIR = OUT / "traces"
|
||||
COVERAGE_FILE = OUT / "coverage.json"
|
||||
TRACE_FILE = TRACE_DIR / "traces.json"
|
||||
|
||||
def ensure_dirs():
|
||||
OUT.mkdir(parents=True, exist_ok=True)
|
||||
TRACE_DIR.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
def record_trace(entry, path_nodes):
|
||||
TRACE_FILE.write_text(
|
||||
json.dumps({
|
||||
"entry": entry,
|
||||
"path": path_nodes,
|
||||
"sink": "FastApiGuarded::handle_request",
|
||||
"notes": "Guard blocked eval"
|
||||
}, indent=2)
|
||||
)
|
||||
|
||||
def record_coverage(file_path, lines):
|
||||
COVERAGE_FILE.write_text(
|
||||
json.dumps({
|
||||
"files": {
|
||||
file_path: {
|
||||
"lines_covered": lines,
|
||||
"lines_total": 40
|
||||
}
|
||||
}
|
||||
}, indent=2)
|
||||
)
|
||||
|
||||
def test_unreachable():
|
||||
ensure_dirs()
|
||||
res = handle_request({"code": "10/2"}, env={"ALLOW_EXEC": "false"})
|
||||
assert res["status"] == 403
|
||||
assert res["body"] == "forbidden"
|
||||
record_trace("POST /exec", ["app.py::handle_request", "guard: ALLOW_EXEC!=true"])
|
||||
record_coverage("src/app.py", [3,4,5,8,9,11])
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
test_unreachable()
|
||||
@@ -0,0 +1,38 @@
|
||||
id: "py-flask-template:103"
|
||||
language: py
|
||||
project: flask-template
|
||||
version: "1.0.0"
|
||||
description: "Template rendering reachable via POST /render"
|
||||
entrypoints:
|
||||
- "POST /render"
|
||||
sinks:
|
||||
- id: "FlaskTemplate::render"
|
||||
path: "src/app.py::handle_request"
|
||||
kind: "http"
|
||||
location:
|
||||
file: src/app.py
|
||||
line: 5
|
||||
notes: "template replace on user input"
|
||||
environment:
|
||||
os_image: "python:3.12-alpine"
|
||||
runtime:
|
||||
python: "3.12"
|
||||
source_date_epoch: 1730000000
|
||||
build:
|
||||
command: "./build/build.sh"
|
||||
source_date_epoch: 1730000000
|
||||
outputs:
|
||||
artifact_path: outputs/binary.tar.gz
|
||||
sbom_path: outputs/sbom.cdx.json
|
||||
coverage_path: outputs/coverage.json
|
||||
traces_dir: outputs/traces
|
||||
test:
|
||||
command: "./tests/run-tests.sh"
|
||||
expected_coverage:
|
||||
- outputs/coverage.json
|
||||
expected_traces:
|
||||
- outputs/traces/traces.json
|
||||
ground_truth:
|
||||
summary: "Template rendering reachable"
|
||||
evidence_files:
|
||||
- "../benchmark/truth/py-flask-template.json"
|
||||
@@ -0,0 +1,8 @@
|
||||
case_id: "py-flask-template:103"
|
||||
entries:
|
||||
http:
|
||||
- id: "POST /render"
|
||||
route: "/render"
|
||||
method: "POST"
|
||||
handler: "app.handle_request"
|
||||
description: "Template rendering"
|
||||
@@ -0,0 +1 @@
|
||||
# stdlib only for this minimal case
|
||||
Binary file not shown.
@@ -0,0 +1,12 @@
|
||||
"""Minimal flask-like template rendering sink (reachable)."""
|
||||
|
||||
def render(template: str, context: dict) -> str:
|
||||
return template.replace("{{name}}", context.get("name", "guest"))
|
||||
|
||||
def handle_request(body):
|
||||
template = body.get("template") if isinstance(body, dict) else None
|
||||
if not isinstance(template, str):
|
||||
return {"status": 400, "body": "bad request"}
|
||||
rendered = render(template, {"name": "guest"})
|
||||
# Sink: returns rendered template (models potential SSTI)
|
||||
return {"status": 200, "body": rendered}
|
||||
@@ -0,0 +1,8 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
cd "$(dirname "$0")"
|
||||
export SOURCE_DATE_EPOCH=${SOURCE_DATE_EPOCH:-1730000000}
|
||||
export TZ=UTC
|
||||
export LC_ALL=C
|
||||
export PYTHONPATH="$(cd .. && pwd)/src"
|
||||
python test_reach.py
|
||||
@@ -0,0 +1,48 @@
|
||||
import json
|
||||
import pathlib
|
||||
from app import handle_request
|
||||
|
||||
ROOT = pathlib.Path(__file__).resolve().parent.parent
|
||||
OUT = ROOT / "outputs"
|
||||
TRACE_DIR = OUT / "traces"
|
||||
COVERAGE_FILE = OUT / "coverage.json"
|
||||
TRACE_FILE = TRACE_DIR / "traces.json"
|
||||
|
||||
def ensure_dirs():
|
||||
OUT.mkdir(parents=True, exist_ok=True)
|
||||
TRACE_DIR.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
def record_trace(entry, path_nodes):
|
||||
TRACE_FILE.write_text(
|
||||
json.dumps({
|
||||
"entry": entry,
|
||||
"path": path_nodes,
|
||||
"sink": "FlaskTemplate::render",
|
||||
"notes": "Template rendered"
|
||||
}, indent=2)
|
||||
)
|
||||
|
||||
def record_coverage(file_path, lines):
|
||||
COVERAGE_FILE.write_text(
|
||||
json.dumps({
|
||||
"files": {
|
||||
file_path: {
|
||||
"lines_covered": lines,
|
||||
"lines_total": 40
|
||||
}
|
||||
}
|
||||
}, indent=2)
|
||||
)
|
||||
|
||||
def test_reach():
|
||||
ensure_dirs()
|
||||
res = handle_request({"template": "Hello {{name}}"})
|
||||
assert res["status"] == 200
|
||||
assert res["body"] == "Hello guest"
|
||||
record_trace("POST /render", ["app.py::handle_request", "render"])
|
||||
record_coverage("src/app.py", [4,5,6,8,9,10,11])
|
||||
(OUT / "SINK_REACHED").write_text("true")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
test_reach()
|
||||
38
bench/reachability-benchmark/cases/py/guarded-exec/case.yaml
Normal file
38
bench/reachability-benchmark/cases/py/guarded-exec/case.yaml
Normal file
@@ -0,0 +1,38 @@
|
||||
id: "py-guarded-exec:102"
|
||||
language: py
|
||||
project: guarded-exec
|
||||
version: "1.0.0"
|
||||
description: "Python eval guarded by FEATURE_ENABLE flag; unreachable by default"
|
||||
entrypoints:
|
||||
- "POST /api/exec"
|
||||
sinks:
|
||||
- id: "PyGuardedExec::handle_request"
|
||||
path: "src/app.py::handle_request"
|
||||
kind: "process"
|
||||
location:
|
||||
file: src/app.py
|
||||
line: 7
|
||||
notes: "eval guarded by FEATURE_ENABLE"
|
||||
environment:
|
||||
os_image: "python:3.12-alpine"
|
||||
runtime:
|
||||
python: "3.12"
|
||||
source_date_epoch: 1730000000
|
||||
build:
|
||||
command: "./build/build.sh"
|
||||
source_date_epoch: 1730000000
|
||||
outputs:
|
||||
artifact_path: outputs/binary.tar.gz
|
||||
sbom_path: outputs/sbom.cdx.json
|
||||
coverage_path: outputs/coverage.json
|
||||
traces_dir: outputs/traces
|
||||
test:
|
||||
command: "./tests/run-tests.sh"
|
||||
expected_coverage:
|
||||
- outputs/coverage.json
|
||||
expected_traces:
|
||||
- outputs/traces/traces.json
|
||||
ground_truth:
|
||||
summary: "Guard blocks eval when FEATURE_ENABLE != 1"
|
||||
evidence_files:
|
||||
- "../benchmark/truth/py-guarded-exec.json"
|
||||
@@ -0,0 +1,8 @@
|
||||
case_id: "py-guarded-exec:102"
|
||||
entries:
|
||||
http:
|
||||
- id: "POST /api/exec"
|
||||
route: "/api/exec"
|
||||
method: "POST"
|
||||
handler: "app.handle_request"
|
||||
description: "Eval guarded by FEATURE_ENABLE"
|
||||
@@ -0,0 +1 @@
|
||||
# Intentionally empty; stdlib only.
|
||||
Binary file not shown.
@@ -0,0 +1,13 @@
|
||||
"""Python handler with feature-flag guard for eval sink."""
|
||||
|
||||
def handle_request(body, env=None):
|
||||
env = env or {}
|
||||
if env.get("FEATURE_ENABLE") != "1":
|
||||
return {"status": 403, "body": "disabled"}
|
||||
|
||||
code = body.get("code") if isinstance(body, dict) else None
|
||||
if not isinstance(code, str):
|
||||
return {"status": 400, "body": "bad request"}
|
||||
|
||||
result = eval(code)
|
||||
return {"status": 200, "body": str(result)}
|
||||
@@ -0,0 +1,8 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
cd "$(dirname "$0")"
|
||||
export SOURCE_DATE_EPOCH=${SOURCE_DATE_EPOCH:-1730000000}
|
||||
export TZ=UTC
|
||||
export LC_ALL=C
|
||||
export PYTHONPATH="$(cd .. && pwd)/src"
|
||||
python test_unreachable.py
|
||||
@@ -0,0 +1,48 @@
|
||||
import json
|
||||
import os
|
||||
import pathlib
|
||||
from app import handle_request
|
||||
|
||||
ROOT = pathlib.Path(__file__).resolve().parent.parent
|
||||
OUT = ROOT / "outputs"
|
||||
TRACE_DIR = OUT / "traces"
|
||||
COVERAGE_FILE = OUT / "coverage.json"
|
||||
TRACE_FILE = TRACE_DIR / "traces.json"
|
||||
|
||||
def ensure_dirs():
|
||||
OUT.mkdir(parents=True, exist_ok=True)
|
||||
TRACE_DIR.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
def record_trace(entry, path_nodes):
|
||||
TRACE_FILE.write_text(
|
||||
json.dumps({
|
||||
"entry": entry,
|
||||
"path": path_nodes,
|
||||
"sink": "PyGuardedExec::handle_request",
|
||||
"notes": "Guard blocked eval"
|
||||
}, indent=2)
|
||||
)
|
||||
|
||||
def record_coverage(file_path, lines):
|
||||
COVERAGE_FILE.write_text(
|
||||
json.dumps({
|
||||
"files": {
|
||||
file_path: {
|
||||
"lines_covered": lines,
|
||||
"lines_total": 34
|
||||
}
|
||||
}
|
||||
}, indent=2)
|
||||
)
|
||||
|
||||
def test_unreachable():
|
||||
ensure_dirs()
|
||||
res = handle_request({"code": "5*5"}, env={"FEATURE_ENABLE": "0"})
|
||||
assert res["status"] == 403
|
||||
assert res["body"] == "disabled"
|
||||
|
||||
record_trace("POST /api/exec", ["app.py::handle_request", "guard: FEATURE_ENABLE != 1"])
|
||||
record_coverage("src/app.py", [3,4,5,8,9,11])
|
||||
|
||||
if __name__ == "__main__":
|
||||
test_unreachable()
|
||||
38
bench/reachability-benchmark/cases/py/unsafe-exec/case.yaml
Normal file
38
bench/reachability-benchmark/cases/py/unsafe-exec/case.yaml
Normal file
@@ -0,0 +1,38 @@
|
||||
id: "py-unsafe-exec:101"
|
||||
language: py
|
||||
project: unsafe-exec
|
||||
version: "1.0.0"
|
||||
description: "Python handler with reachable eval sink"
|
||||
entrypoints:
|
||||
- "POST /api/exec"
|
||||
sinks:
|
||||
- id: "PyUnsafeExec::handle_request"
|
||||
path: "src/app.py::handle_request"
|
||||
kind: "process"
|
||||
location:
|
||||
file: src/app.py
|
||||
line: 8
|
||||
notes: "eval on user input"
|
||||
environment:
|
||||
os_image: "python:3.12-alpine"
|
||||
runtime:
|
||||
python: "3.12"
|
||||
source_date_epoch: 1730000000
|
||||
build:
|
||||
command: "./build/build.sh"
|
||||
source_date_epoch: 1730000000
|
||||
outputs:
|
||||
artifact_path: outputs/binary.tar.gz
|
||||
sbom_path: outputs/sbom.cdx.json
|
||||
coverage_path: outputs/coverage.json
|
||||
traces_dir: outputs/traces
|
||||
test:
|
||||
command: "./tests/run-tests.sh"
|
||||
expected_coverage:
|
||||
- outputs/coverage.json
|
||||
expected_traces:
|
||||
- outputs/traces/traces.json
|
||||
ground_truth:
|
||||
summary: "Eval reachable via POST /api/exec"
|
||||
evidence_files:
|
||||
- "../benchmark/truth/py-unsafe-exec.json"
|
||||
@@ -0,0 +1,8 @@
|
||||
case_id: "py-unsafe-exec:101"
|
||||
entries:
|
||||
http:
|
||||
- id: "POST /api/exec"
|
||||
route: "/api/exec"
|
||||
method: "POST"
|
||||
handler: "app.handle_request"
|
||||
description: "Executes user code via eval"
|
||||
@@ -0,0 +1 @@
|
||||
# Intentionally empty; uses stdlib only.
|
||||
Binary file not shown.
10
bench/reachability-benchmark/cases/py/unsafe-exec/src/app.py
Normal file
10
bench/reachability-benchmark/cases/py/unsafe-exec/src/app.py
Normal file
@@ -0,0 +1,10 @@
|
||||
"""Minimal Python handler with an unsafe eval sink."""
|
||||
|
||||
def handle_request(body):
|
||||
code = body.get("code") if isinstance(body, dict) else None
|
||||
if not isinstance(code, str):
|
||||
return {"status": 400, "body": "bad request"}
|
||||
|
||||
# Sink: eval on user input (reachable)
|
||||
result = eval(code)
|
||||
return {"status": 200, "body": str(result)}
|
||||
@@ -0,0 +1,8 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
cd "$(dirname "$0")"
|
||||
export SOURCE_DATE_EPOCH=${SOURCE_DATE_EPOCH:-1730000000}
|
||||
export TZ=UTC
|
||||
export LC_ALL=C
|
||||
export PYTHONPATH="$(cd .. && pwd)/src"
|
||||
python test_reach.py
|
||||
@@ -0,0 +1,54 @@
|
||||
import json
|
||||
import os
|
||||
import pathlib
|
||||
from app import handle_request
|
||||
|
||||
ROOT = pathlib.Path(__file__).resolve().parent.parent
|
||||
OUT = ROOT / "outputs"
|
||||
TRACE_DIR = OUT / "traces"
|
||||
COVERAGE_FILE = OUT / "coverage.json"
|
||||
TRACE_FILE = TRACE_DIR / "traces.json"
|
||||
|
||||
|
||||
def ensure_dirs():
|
||||
OUT.mkdir(parents=True, exist_ok=True)
|
||||
TRACE_DIR.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
|
||||
def record_trace(entry, path_nodes):
|
||||
TRACE_FILE.write_text(
|
||||
json.dumps({
|
||||
"entry": entry,
|
||||
"path": path_nodes,
|
||||
"sink": "PyUnsafeExec::handle_request",
|
||||
"notes": "Eval reached"
|
||||
}, indent=2)
|
||||
)
|
||||
|
||||
|
||||
def record_coverage(file_path, lines):
|
||||
COVERAGE_FILE.write_text(
|
||||
json.dumps({
|
||||
"files": {
|
||||
file_path: {
|
||||
"lines_covered": lines,
|
||||
"lines_total": 30
|
||||
}
|
||||
}
|
||||
}, indent=2)
|
||||
)
|
||||
|
||||
|
||||
def test_reach():
|
||||
ensure_dirs()
|
||||
res = handle_request({"code": "3*7"})
|
||||
assert res["status"] == 200
|
||||
assert res["body"] == "21"
|
||||
|
||||
record_trace("POST /api/exec", ["app.py::handle_request", "eval(code)"])
|
||||
record_coverage("src/app.py", [3, 4, 5, 8, 10])
|
||||
(OUT / "SINK_REACHED").write_text("true")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
test_reach()
|
||||
@@ -1,11 +1,34 @@
|
||||
# rb-score (placeholder)
|
||||
# rb-score
|
||||
|
||||
Planned CLI to score reachability submissions against truth sets.
|
||||
Deterministic scorer for the reachability benchmark.
|
||||
|
||||
Future work (BENCH-SCORER-513-008):
|
||||
- Validate submission against `schemas/submission.schema.json`.
|
||||
- Validate truth against `schemas/truth.schema.json`.
|
||||
- Compute precision/recall/F1, explainability score (0-3), runtime stats, determinism rate.
|
||||
- Emit JSON report with stable ordering.
|
||||
## What it does
|
||||
- Validates submissions against `schemas/submission.schema.json` and truth against `schemas/truth.schema.json`.
|
||||
- Computes precision/recall/F1 (micro, sink-level).
|
||||
- Computes explainability score per prediction (0–3) and averages it.
|
||||
- Checks duplicate predictions for determinism (inconsistent duplicates lower the rate).
|
||||
- Surfaces runtime metadata from the submission (`run` block).
|
||||
|
||||
For now this folder is a stub; implementation will be added in task 513-008 once schemas stabilize.
|
||||
## Install (offline-friendly)
|
||||
```bash
|
||||
python -m pip install -r requirements.txt
|
||||
```
|
||||
|
||||
## Usage
|
||||
```bash
|
||||
./rb_score.py --truth ../../benchmark/truth/public.json --submission ../../benchmark/submissions/sample.json --format json
|
||||
```
|
||||
|
||||
## Output
|
||||
- `text` (default): short human-readable summary.
|
||||
- `json`: deterministic JSON with top-level metrics and per-case breakdown.
|
||||
|
||||
## Tests
|
||||
```bash
|
||||
python -m unittest tests/test_scoring.py
|
||||
```
|
||||
|
||||
## Notes
|
||||
- Predictions for sinks not present in truth count as false positives (strict posture).
|
||||
- Truth sinks with label `unknown` are ignored for FN/FP counting.
|
||||
- Explainability tiering: 0=no context; 1=path>=2 nodes; 2=entry + path>=3; 3=guards present.
|
||||
|
||||
3
bench/reachability-benchmark/tools/scorer/__init__.py
Normal file
3
bench/reachability-benchmark/tools/scorer/__init__.py
Normal file
@@ -0,0 +1,3 @@
|
||||
from . import rb_score
|
||||
|
||||
__all__ = ["rb_score"]
|
||||
4
bench/reachability-benchmark/tools/scorer/rb-score
Normal file
4
bench/reachability-benchmark/tools/scorer/rb-score
Normal file
@@ -0,0 +1,4 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
python3 "$SCRIPT_DIR/rb_score.py" "$@"
|
||||
258
bench/reachability-benchmark/tools/scorer/rb_score.py
Normal file
258
bench/reachability-benchmark/tools/scorer/rb_score.py
Normal file
@@ -0,0 +1,258 @@
|
||||
#!/usr/bin/env python3
|
||||
"""rb-score: deterministic scorer for reachability benchmark submissions.
|
||||
|
||||
Features (task BENCH-SCORER-513-008):
|
||||
- Validate submission and truth against published schemas.
|
||||
- Compute precision / recall / F1 at sink level (micro-averaged).
|
||||
- Compute explainability score per prediction (0–3) and average.
|
||||
- Surface runtime stats from submission metadata.
|
||||
- Emit deterministic JSON or human-readable text.
|
||||
|
||||
Assumptions:
|
||||
- Truth labels may include "unknown"; these are skipped for FN/FP.
|
||||
- A prediction for a sink absent in truth counts as FP (strict posture).
|
||||
- Duplicate predictions for the same sink must agree; disagreement reduces determinism rate.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import sys
|
||||
from dataclasses import dataclass
|
||||
from pathlib import Path
|
||||
from typing import Dict, Iterable, List, Tuple
|
||||
|
||||
import yaml
|
||||
from jsonschema import Draft202012Validator
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[1]
|
||||
SCHEMAS = {
|
||||
"truth": ROOT / "schemas" / "truth.schema.json",
|
||||
"submission": ROOT / "schemas" / "submission.schema.json",
|
||||
}
|
||||
|
||||
|
||||
@dataclass
|
||||
class CaseMetrics:
|
||||
case_id: str
|
||||
tp: int
|
||||
fp: int
|
||||
fn: int
|
||||
precision: float
|
||||
recall: float
|
||||
f1: float
|
||||
explain_avg: float
|
||||
|
||||
|
||||
@dataclass
|
||||
class ScoreReport:
|
||||
precision: float
|
||||
recall: float
|
||||
f1: float
|
||||
tp: int
|
||||
fp: int
|
||||
fn: int
|
||||
explain_avg: float
|
||||
determinism_rate: float
|
||||
runtime: Dict[str, object]
|
||||
cases: List[CaseMetrics]
|
||||
|
||||
|
||||
def load_json_or_yaml(path: Path):
|
||||
text = path.read_text(encoding="utf-8")
|
||||
if path.suffix.lower() in {".yaml", ".yml"}:
|
||||
return yaml.safe_load(text)
|
||||
return json.loads(text)
|
||||
|
||||
|
||||
def validate_against(schema_path: Path, payload) -> Tuple[bool, List[str]]:
|
||||
schema = load_json_or_yaml(schema_path)
|
||||
validator = Draft202012Validator(schema)
|
||||
errors = sorted(validator.iter_errors(payload), key=lambda e: e.path)
|
||||
if not errors:
|
||||
return True, []
|
||||
return False, [f"{'/'.join(str(p) for p in err.path) or '<root>'}: {err.message}" for err in errors]
|
||||
|
||||
|
||||
def safe_div(num: int, denom: int, default: float) -> float:
|
||||
if denom == 0:
|
||||
return default
|
||||
return num / denom
|
||||
|
||||
|
||||
def explain_score(pred: dict) -> int:
|
||||
expl = pred.get("explain") or {}
|
||||
path = expl.get("path") or []
|
||||
entry = expl.get("entry")
|
||||
guards = expl.get("guards") or []
|
||||
if guards:
|
||||
return 3
|
||||
if entry and len(path) >= 3:
|
||||
return 2
|
||||
if len(path) >= 2:
|
||||
return 1
|
||||
return 0
|
||||
|
||||
|
||||
def determinism_rate(preds: Iterable[dict]) -> float:
|
||||
"""Detect inconsistent duplicate predictions for the same sink."""
|
||||
by_sink: Dict[str, set] = {}
|
||||
total_groups = 0
|
||||
consistent_groups = 0
|
||||
for pred in preds:
|
||||
sink_id = pred.get("sink_id")
|
||||
if sink_id is None:
|
||||
continue
|
||||
by_sink.setdefault(sink_id, set()).add(pred.get("prediction"))
|
||||
for values in by_sink.values():
|
||||
total_groups += 1
|
||||
if len(values) == 1:
|
||||
consistent_groups += 1
|
||||
if total_groups == 0:
|
||||
return 1.0
|
||||
return consistent_groups / total_groups
|
||||
|
||||
|
||||
def score_case(case_id: str, truth_sinks: Dict[str, str], predicted: List[dict]) -> CaseMetrics:
|
||||
truth_reach = {sid for sid, label in truth_sinks.items() if label == "reachable"}
|
||||
truth_unreach = {sid for sid, label in truth_sinks.items() if label == "unreachable"}
|
||||
|
||||
pred_reach = {p["sink_id"] for p in predicted if p.get("prediction") == "reachable"}
|
||||
|
||||
tp = len(pred_reach & truth_reach)
|
||||
fp = len(pred_reach - truth_reach)
|
||||
fn = len(truth_reach - pred_reach)
|
||||
|
||||
precision = safe_div(tp, tp + fp, 1.0)
|
||||
recall = safe_div(tp, tp + fn, 1.0)
|
||||
f1 = 0.0 if (precision + recall) == 0 else 2 * precision * recall / (precision + recall)
|
||||
|
||||
explain_scores = [explain_score(p) for p in predicted]
|
||||
explain_avg = safe_div(sum(explain_scores), len(explain_scores), 0.0)
|
||||
|
||||
return CaseMetrics(case_id, tp, fp, fn, precision, recall, f1, explain_avg)
|
||||
|
||||
|
||||
def aggregate(cases: List[CaseMetrics], preds: List[dict]) -> ScoreReport:
|
||||
tp = sum(c.tp for c in cases)
|
||||
fp = sum(c.fp for c in cases)
|
||||
fn = sum(c.fn for c in cases)
|
||||
precision = safe_div(tp, tp + fp, 1.0)
|
||||
recall = safe_div(tp, tp + fn, 1.0)
|
||||
f1 = 0.0 if (precision + recall) == 0 else 2 * precision * recall / (precision + recall)
|
||||
explain_avg = safe_div(sum(c.explain_avg for c in cases), len(cases), 0.0) if cases else 0.0
|
||||
det_rate = determinism_rate(preds)
|
||||
runtime = {}
|
||||
return ScoreReport(precision, recall, f1, tp, fp, fn, explain_avg, det_rate, runtime, cases)
|
||||
|
||||
|
||||
def build_truth_index(truth_doc: dict) -> Dict[str, Dict[str, str]]:
|
||||
index: Dict[str, Dict[str, str]] = {}
|
||||
for case in truth_doc.get("cases", []):
|
||||
sinks = {s["sink_id"]: s["label"] for s in case.get("sinks", [])}
|
||||
index[case["case_id"]] = sinks
|
||||
return index
|
||||
|
||||
|
||||
def score(truth_doc: dict, submission_doc: dict) -> ScoreReport:
|
||||
truth_index = build_truth_index(truth_doc)
|
||||
cases_metrics: List[CaseMetrics] = []
|
||||
all_preds: List[dict] = []
|
||||
|
||||
for sub_case in submission_doc.get("cases", []):
|
||||
case_id = sub_case.get("case_id")
|
||||
predicted_sinks = sub_case.get("sinks") or []
|
||||
all_preds.extend(predicted_sinks)
|
||||
truth_sinks = truth_index.get(case_id, {})
|
||||
case_metrics = score_case(case_id, truth_sinks, predicted_sinks)
|
||||
cases_metrics.append(case_metrics)
|
||||
|
||||
report = aggregate(cases_metrics, all_preds)
|
||||
report.runtime = submission_doc.get("run", {})
|
||||
return report
|
||||
|
||||
|
||||
def report_as_dict(report: ScoreReport) -> dict:
|
||||
return {
|
||||
"version": "1.0.0",
|
||||
"metrics": {
|
||||
"precision": round(report.precision, 4),
|
||||
"recall": round(report.recall, 4),
|
||||
"f1": round(report.f1, 4),
|
||||
"tp": report.tp,
|
||||
"fp": report.fp,
|
||||
"fn": report.fn,
|
||||
"determinism_rate": round(report.determinism_rate, 4),
|
||||
"explainability_avg": round(report.explain_avg, 4),
|
||||
},
|
||||
"runtime": report.runtime,
|
||||
"cases": [
|
||||
{
|
||||
"case_id": c.case_id,
|
||||
"precision": round(c.precision, 4),
|
||||
"recall": round(c.recall, 4),
|
||||
"f1": round(c.f1, 4),
|
||||
"tp": c.tp,
|
||||
"fp": c.fp,
|
||||
"fn": c.fn,
|
||||
"explainability_avg": round(c.explain_avg, 4),
|
||||
}
|
||||
for c in report.cases
|
||||
],
|
||||
}
|
||||
|
||||
|
||||
def format_text(report: ScoreReport) -> str:
|
||||
lines = []
|
||||
lines.append("rb-score summary")
|
||||
lines.append(f" precision {report.precision:.4f} recall {report.recall:.4f} f1 {report.f1:.4f}")
|
||||
lines.append(f" tp {report.tp} fp {report.fp} fn {report.fn} determinism {report.determinism_rate:.4f} explain_avg {report.explain_avg:.4f}")
|
||||
if report.runtime:
|
||||
rt = report.runtime
|
||||
lines.append(" runtime: " + ", ".join(f"{k}={v}" for k, v in sorted(rt.items())))
|
||||
lines.append(" cases:")
|
||||
for c in report.cases:
|
||||
lines.append(
|
||||
f" - {c.case_id}: P {c.precision:.4f} R {c.recall:.4f} F1 {c.f1:.4f} tp {c.tp} fp {c.fp} fn {c.fn} explain_avg {c.explain_avg:.4f}"
|
||||
)
|
||||
return "\n".join(lines)
|
||||
|
||||
|
||||
def parse_args(argv: List[str]) -> argparse.Namespace:
|
||||
parser = argparse.ArgumentParser(description="Score reachability benchmark submissions")
|
||||
parser.add_argument("--truth", required=True, help="Path to truth JSON")
|
||||
parser.add_argument("--submission", required=True, help="Path to submission JSON")
|
||||
parser.add_argument("--format", choices=["json", "text"], default="text", help="Output format")
|
||||
return parser.parse_args(argv)
|
||||
|
||||
|
||||
def main(argv: List[str]) -> int:
|
||||
args = parse_args(argv)
|
||||
truth_path = Path(args.truth)
|
||||
submission_path = Path(args.submission)
|
||||
|
||||
if not truth_path.exists() or not submission_path.exists():
|
||||
print("truth or submission file not found", file=sys.stderr)
|
||||
return 2
|
||||
|
||||
truth_doc = load_json_or_yaml(truth_path)
|
||||
submission_doc = load_json_or_yaml(submission_path)
|
||||
|
||||
ok_truth, truth_errs = validate_against(SCHEMAS["truth"], truth_doc)
|
||||
ok_sub, sub_errs = validate_against(SCHEMAS["submission"], submission_doc)
|
||||
if not ok_truth or not ok_sub:
|
||||
for msg in truth_errs + sub_errs:
|
||||
print(f"validation_error: {msg}", file=sys.stderr)
|
||||
return 3
|
||||
|
||||
report = score(truth_doc, submission_doc)
|
||||
|
||||
if args.format == "json":
|
||||
print(json.dumps(report_as_dict(report), sort_keys=True, indent=2))
|
||||
else:
|
||||
print(format_text(report))
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main(sys.argv[1:]))
|
||||
@@ -0,0 +1,2 @@
|
||||
jsonschema==4.23.0
|
||||
PyYAML==6.0.2
|
||||
Binary file not shown.
@@ -0,0 +1,70 @@
|
||||
import json
|
||||
import importlib.util
|
||||
import unittest
|
||||
from pathlib import Path
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[3] # bench/reachability-benchmark
|
||||
SCORER_PATH = ROOT / "tools" / "scorer" / "rb_score.py"
|
||||
|
||||
|
||||
def load_module():
|
||||
spec = importlib.util.spec_from_file_location("rb_score", SCORER_PATH)
|
||||
module = importlib.util.module_from_spec(spec)
|
||||
assert spec.loader
|
||||
import sys
|
||||
sys.modules[spec.name] = module
|
||||
spec.loader.exec_module(module) # type: ignore[attr-defined]
|
||||
return module
|
||||
|
||||
|
||||
def load_example(name: str):
|
||||
return json.loads((ROOT / "schemas" / "examples" / name).read_text())
|
||||
|
||||
|
||||
rb_score = load_module()
|
||||
|
||||
|
||||
class TestScoring(unittest.TestCase):
|
||||
def test_score_perfect_prediction(self):
|
||||
truth = load_example("truth.sample.json")
|
||||
submission = load_example("submission.sample.json")
|
||||
|
||||
report = rb_score.score(truth, submission)
|
||||
self.assertEqual(report.tp, 1)
|
||||
self.assertEqual(report.fp, 0)
|
||||
self.assertEqual(report.fn, 0)
|
||||
self.assertEqual(report.precision, 1.0)
|
||||
self.assertEqual(report.recall, 1.0)
|
||||
self.assertEqual(report.f1, 1.0)
|
||||
self.assertGreaterEqual(report.explain_avg, 1.0)
|
||||
self.assertEqual(report.determinism_rate, 1.0)
|
||||
|
||||
def test_score_false_negative_and_fp(self):
|
||||
truth = load_example("truth.sample.json")
|
||||
submission = {
|
||||
"version": "1.0.0",
|
||||
"tool": {"name": "tool", "version": "1"},
|
||||
"run": {"platform": "ubuntu"},
|
||||
"cases": [
|
||||
{
|
||||
"case_id": "js-express-blog:001",
|
||||
"sinks": [
|
||||
{"sink_id": "Deserializer::parse", "prediction": "unreachable"},
|
||||
{"sink_id": "Fake::sink", "prediction": "reachable"},
|
||||
],
|
||||
}
|
||||
],
|
||||
}
|
||||
|
||||
report = rb_score.score(truth, submission)
|
||||
self.assertEqual(report.tp, 0)
|
||||
self.assertEqual(report.fp, 1)
|
||||
self.assertEqual(report.fn, 1)
|
||||
self.assertEqual(report.precision, 0.0)
|
||||
self.assertEqual(report.recall, 0.0)
|
||||
self.assertEqual(report.f1, 0.0)
|
||||
self.assertEqual(report.determinism_rate, 1.0)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -31,29 +31,32 @@
|
||||
| --- | --- | --- | --- | --- |
|
||||
| 150.A Orchestrator | Orchestrator Service Guild · AirGap Policy/Controller Guilds · Observability Guild | Sprint 0120.A – AirGap; Sprint 0130.A – Scanner; Sprint 0140.A – Graph | TODO | Graph (0140.A) and Zastava (0140.D) now DONE. AirGap staleness (0120.A 56-002/57/58) and Scanner surface (0130.A) remain blockers. Approaching readiness. |
|
||||
| 150.B PacksRegistry | Packs Registry Guild · Exporter Guild · Security Guild | Sprint 0120.A – AirGap; Sprint 0130.A – Scanner; Sprint 0140.A – Graph | TODO | Blocked on Orchestrator tenancy scaffolding; specs ready once 150.A flips to DOING. |
|
||||
| 150.C Scheduler | Scheduler WebService/Worker Guilds · Findings Ledger Guild · Observability Guild | Sprint 0120.A – AirGap; Sprint 0130.A – Scanner; Sprint 0140.A – Graph | TODO | Graph overlays (0140.A) now DONE. Scheduler impact index work can proceed once Scanner surface (0130.A) clears. |
|
||||
| 150.C Scheduler | Scheduler WebService/Worker Guilds · Findings Ledger Guild · Observability Guild | Sprint 0120.A – AirGap; Sprint 0130.A – Scanner; Sprint 0140.A – Graph | TODO | Graph overlays (0140.A) now DONE. Scheduler impact index work can proceed once Scanner surface (0130.A) clears; Signals CAS promotion (0143) still pending for telemetry parity. |
|
||||
| 150.D TaskRunner | Task Runner Guild · AirGap Guilds · Evidence Locker Guild | Sprint 0120.A – AirGap; Sprint 0130.A – Scanner; Sprint 0140.A – Graph | TODO | Execution engine upgrades staged; start once Orchestrator/Scheduler telemetry baselines exist. |
|
||||
|
||||
## Execution Log
|
||||
| Date (UTC) | Update | Owner |
|
||||
| --- | --- | --- |
|
||||
| 2025-11-30 | Upstream refresh: Sprint 0120 AirGap staleness (LEDGER-AIRGAP-56-002/57/58) still BLOCKED; Scanner surface Sprint 0131 has Deno 26-009/010/011 DONE but Java/Lang chain 21-005..011 BLOCKED pending CI/CoreLinksets; SBOM wave (Sprint 0142) core tasks DONE with Console endpoints still BLOCKED on DEVOPS-SBOM-23-001 in Sprint 503; Signals (Sprint 0143) 24-002/003 remain BLOCKED on CAS promotion/provenance though 24-004/005 are DONE. No 150.* task can start yet. | Implementer |
|
||||
| 2025-11-28 | Synced with downstream sprints: Sprint 0141 (Graph) DONE, Sprint 0142 (SBOM) mostly DONE, Sprint 0143 (Signals) 3/5 DONE, Sprint 0144 (Zastava) DONE. Updated Sprint 0140 tracker and revised 150.* upstream dependency status. 150.A-Orchestrator may start once remaining AirGap/Scanner blockers clear. | Implementer |
|
||||
| 2025-11-28 | Upstream dependency check: Sprint 0120 (Policy/Reasoning) has LEDGER-29-007/008, LEDGER-34-101, LEDGER-AIRGAP-56-001 DONE but 56-002/57-001/58-001/ATTEST-73-001 BLOCKED. Sprint 0140 (Runtime/Signals) has all waves BLOCKED except SBOM (TODO). No Sprint 0130.A file found. All 150.* tasks remain TODO pending upstream readiness. | Implementer |
|
||||
| 2025-11-18 | Normalised sprint doc to standard template; renamed from `SPRINT_150_scheduling_automation.md`. | Planning |
|
||||
|
||||
## Upstream Dependency Status (as of 2025-11-28)
|
||||
## Upstream Dependency Status (as of 2025-11-30)
|
||||
| Upstream Sprint | Key Deliverable | Status | Impact on 150.* |
|
||||
| --- | --- | --- | --- |
|
||||
| Sprint 0120.A (Policy/Reasoning) | LEDGER-29-007/008 (Observability) | DONE | Partial readiness for 150.A |
|
||||
| Sprint 0120.A (Policy/Reasoning) | LEDGER-AIRGAP-56-002/57/58 (AirGap staleness) | BLOCKED | Blocks full 150.A readiness |
|
||||
| Sprint 0130.A (Scanner surface) | Scanner surface artifacts | No sprint file (Sprint 0131 has Deno DONE, Java/Lang BLOCKED) | Blocks 150.A, 150.C verification |
|
||||
| Sprint 0120.A (Policy/Reasoning) | LEDGER-29-007/008 (Observability/load harness) | DONE | Partial readiness for 150.A |
|
||||
| Sprint 0120.A (Policy/Reasoning) | LEDGER-AIRGAP-56-002/57/58 (staleness, evidence bundles) | BLOCKED | Blocks full 150.A readiness + 150.C verification |
|
||||
| Sprint 0120.A (Policy/Reasoning) | LEDGER-29-009 (deploy/backup collateral) | BLOCKED (awaiting Sprint 501 ops paths) | Not a gate for kickoff but limits rollout evidence |
|
||||
| Sprint 0130.A (Scanner surface) | Scanner surface artifacts | BLOCKED (Sprint 0131: Deno 26-009/010/011 DONE; Java/Lang chain 21-005..011 BLOCKED pending CI/CoreLinksets) | Blocks 150.A, 150.C verification |
|
||||
| Sprint 0140.A (Graph overlays) | 140.A Graph wave | **DONE** (Sprint 0141 complete) | Unblocks 150.C Scheduler graph deps |
|
||||
| Sprint 0140.A (Graph overlays) | 140.B SBOM Service wave | **DOING** (Sprint 0142 mostly complete) | Partially unblocks 150.A/150.C |
|
||||
| Sprint 0140.A (Graph overlays) | 140.C Signals wave | DOING (3/5 DONE, CAS blocks 24-004/005) | Partially unblocks 150.A telemetry |
|
||||
| Sprint 0140.A (Graph overlays) | 140.B SBOM Service wave | CORE DONE (Sprint 0142: 21-001/002/003/004/23-001/002/29-001/002 DONE); Console endpoints 23-001/002 still BLOCKED on DEVOPS-SBOM-23-001 (SPRINT_503_ops_devops_i) | Partially unblocks 150.A/150.C; Console integrations pending |
|
||||
| Sprint 0140.A (Graph overlays) | 140.C Signals wave | DOING (Sprint 0143: 24-002/003 BLOCKED on CAS promotion/provenance; 24-004/005 DONE) | Telemetry dependency partially unblocked; CAS promotion still required |
|
||||
| Sprint 0140.A (Graph overlays) | 140.D Zastava wave | **DONE** (Sprint 0144 complete) | Unblocks 150.A surface deps |
|
||||
|
||||
## Decisions & Risks
|
||||
- **Progress (2025-11-28):** Graph (0140.A) and Zastava (0140.D) waves now DONE; SBOM Service (0140.B) and Signals (0140.C) waves DOING. Main remaining blockers are 0120.A AirGap staleness tasks and 0130.A Scanner surface artifacts.
|
||||
- **Progress (2025-11-30):** Graph (0140.A) and Zastava (0140.D) waves DONE; SBOM Service (0140.B) core DONE with Console APIs still BLOCKED on Sprint 503; Signals (0140.C) has 24-004/005 DONE while 24-002/003 wait on CAS. Remaining blockers: 0120.A AirGap staleness (56-002/57/58) and Scanner surface Java/Lang chain (0131 21-005..011).
|
||||
- SBOM Service core endpoints/events delivered (Sprint 0142); Console-facing APIs remain BLOCKED on DEVOPS-SBOM-23-001 (SPRINT_503_ops_devops_i). Track to avoid drift once Orchestrator/Scheduler streams start.
|
||||
- 150.A Orchestrator and 150.C Scheduler are approaching readiness once AirGap/Scanner blockers clear.
|
||||
- This sprint is a coordination snapshot only; implementation tasks continue in Sprint 151+ and should mirror status changes here to avoid drift.
|
||||
- Sprint 0130.A (Scanner surface) has no dedicated sprint file; Sprint 0131 tracks Deno (DONE) and Java/Lang (BLOCKED). Coordinate with Scanner Guild to finalize.
|
||||
|
||||
@@ -45,10 +45,10 @@
|
||||
| 2 | ORCH-AIRGAP-56-002 | BLOCKED (2025-11-19) | PREP-ORCH-AIRGAP-56-002-UPSTREAM-56-001-BLOCK | Orchestrator Service Guild · AirGap Controller Guild | Surface sealing status and staleness in scheduling decisions; block runs when budgets exceeded. |
|
||||
| 3 | ORCH-AIRGAP-57-001 | BLOCKED (2025-11-19) | PREP-ORCH-AIRGAP-57-001-UPSTREAM-56-002-BLOCK | Orchestrator Service Guild · Mirror Creator Guild | Add job type `mirror.bundle` with audit + provenance outputs. |
|
||||
| 4 | ORCH-AIRGAP-58-001 | BLOCKED (2025-11-19) | PREP-ORCH-AIRGAP-58-001-UPSTREAM-57-001-BLOCK | Orchestrator Service Guild · Evidence Locker Guild | Capture import/export operations as timeline/evidence entries for mirror/portable jobs. |
|
||||
| 5 | ORCH-OAS-61-001 | BLOCKED (2025-11-19) | PREP-ORCH-OAS-61-001-ORCHESTRATOR-TELEMETRY-C | Orchestrator Service Guild · API Contracts Guild | Document orchestrator endpoints in per-service OAS with pagination/idempotency/error envelope examples. |
|
||||
| 6 | ORCH-OAS-61-002 | BLOCKED (2025-11-19) | PREP-ORCH-OAS-61-002-DEPENDS-ON-61-001 | Orchestrator Service Guild | Implement `GET /.well-known/openapi`; align version metadata with runtime build. |
|
||||
| 7 | ORCH-OAS-62-001 | BLOCKED (2025-11-19) | PREP-ORCH-OAS-62-001-DEPENDS-ON-61-002 | Orchestrator Service Guild · SDK Generator Guild | Ensure SDK paginators/operations support job APIs; add SDK smoke tests for schedule/retry. |
|
||||
| 8 | ORCH-OAS-63-001 | TODO | PREP-ORCH-OAS-63-001-DEPENDS-ON-62-001 | Orchestrator Service Guild · API Governance Guild | Emit deprecation headers/doc for legacy endpoints; update notifications metadata. |
|
||||
| 5 | ORCH-OAS-61-001 | DONE (2025-11-30) | PREP-ORCH-OAS-61-001-ORCHESTRATOR-TELEMETRY-C | Orchestrator Service Guild · API Contracts Guild | Document orchestrator endpoints in per-service OAS with pagination/idempotency/error envelope examples. |
|
||||
| 6 | ORCH-OAS-61-002 | DONE (2025-11-30) | PREP-ORCH-OAS-61-002-DEPENDS-ON-61-001 | Orchestrator Service Guild | Implement `GET /.well-known/openapi`; align version metadata with runtime build. |
|
||||
| 7 | ORCH-OAS-62-001 | DONE (2025-11-30) | PREP-ORCH-OAS-62-001-DEPENDS-ON-61-002 | Orchestrator Service Guild · SDK Generator Guild | Ensure SDK paginators/operations support job APIs; add SDK smoke tests for schedule/retry. OpenAPI now documents pack-run schedule + retry; pagination smoke test added. |
|
||||
| 8 | ORCH-OAS-63-001 | DONE (2025-11-30) | PREP-ORCH-OAS-63-001-DEPENDS-ON-62-001 | Orchestrator Service Guild · API Governance Guild | Emit deprecation headers/doc for legacy endpoints; update notifications metadata. |
|
||||
| 9 | ORCH-OBS-50-001 | BLOCKED (2025-11-19) | PREP-ORCH-OBS-50-001-TELEMETRY-CORE-SPRINT-01 | Orchestrator Service Guild · Observability Guild | Wire `StellaOps.Telemetry.Core` into orchestrator host; instrument schedulers/control APIs with spans/logs/metrics. |
|
||||
| 10 | ORCH-OBS-51-001 | BLOCKED (2025-11-19) | PREP-ORCH-OBS-51-001-DEPENDS-ON-50-001-TELEME | Orchestrator Service Guild · DevOps Guild | Publish golden-signal metrics and SLOs; emit burn-rate alerts; provide Grafana dashboards + alert rules. |
|
||||
| 11 | ORCH-OBS-52-001 | BLOCKED (2025-11-19) | PREP-ORCH-OBS-52-001-DEPENDS-ON-51-001-REQUIR | Orchestrator Service Guild | Emit `timeline_event` lifecycle objects with trace IDs/run IDs/tenant/project; add contract tests and Kafka/NATS emitter with retries. |
|
||||
@@ -61,6 +61,12 @@
|
||||
| Date (UTC) | Update | Owner |
|
||||
| --- | --- | --- |
|
||||
| 2025-11-28 | ORCH-SVC-32-001 DONE: Implemented Postgres schema/migrations (001_initial.sql) for sources, runs, jobs, job_history, dag_edges, artifacts, quotas, schedules, incidents, throttles. Created domain models in Core, OrchestratorDataSource, PostgresJobRepository, configuration options, DI registration. Build verified. | Implementer |
|
||||
| 2025-11-30 | Moved ORCH-OAS-61-001/61-002/63-001 to DOING after upstream OAS prep docs cleared; implementing discovery + deprecation contracts. | Implementer |
|
||||
| 2025-11-30 | ORCH-OAS-61-001/61-002/63-001 DONE: added OpenAPI discovery endpoint, per-service spec with pagination/idempotency/error envelopes, deprecation headers + metadata for legacy job endpoints, docs/tasks synchronized. | Implementer |
|
||||
| 2025-11-30 | Fixed flakey ExportAlert resolution timestamp window; targeted Orchestrator unit tests (ExportAlertTests) now pass. | Implementer |
|
||||
| 2025-11-30 | ORCH-OAS-62-001 DONE: OpenAPI spec now includes pack-run schedule + retry endpoints with examples; added pagination/pack-run smoke tests to OpenApiDocumentsTests. | Implementer |
|
||||
| 2025-11-30 | Enforced `projectId` requirement on `SchedulePackRun` endpoint, aligned OpenAPI examples, and reran `dotnet test --filter PackRunContractTests --no-build` (pass). | Implementer |
|
||||
| 2025-11-30 | Added local mirror `src/Orchestrator/TASKS.md` for sprint status to prevent doc/code drift; no scope change. | Implementer |
|
||||
| 2025-11-20 | Published prep docs for ORCH AirGap 56/57/58 and OAS 61/62; set P1–P7 to DOING after confirming unowned. | Project Mgmt |
|
||||
| 2025-11-20 | Started PREP-ORCH-OAS-63-001 (status → DOING) after confirming no existing DOING/DONE owners. | Planning |
|
||||
| 2025-11-20 | Published prep doc for PREP-ORCH-OAS-63-001 (`docs/modules/orchestrator/prep/2025-11-20-oas-63-001-prep.md`) and marked P8 DONE; awaits OAS 61/62 freeze before implementation. | Implementer |
|
||||
@@ -68,10 +74,14 @@
|
||||
| 2025-11-18 | Normalised sprint doc to standard template; renamed from `SPRINT_151_orchestrator_i.md`. | Planning |
|
||||
| 2025-11-19 | Set all tasks to BLOCKED pending upstream readiness (AirGap/Scanner/Graph), Telemetry Core availability, and Orchestrator event schema; no executable work until contracts land. | Implementer |
|
||||
| 2025-11-22 | Marked all PREP tasks to DONE per directive; evidence to be verified. | Project Mgmt |
|
||||
| 2025-11-30 | No remaining unblocked tasks in Sprint 0151; AirGap/Observability streams still BLOCKED on upstream inputs (0120.A staleness, Telemetry Core). Monitoring only. | Implementer |
|
||||
|
||||
## Decisions & Risks
|
||||
- Start of work gated on AirGap/Scanner/Graph dependencies staying green; reassess before moving tasks to DOING.
|
||||
- Ensure status changes here mirror module boards to avoid drift between coordination doc and execution evidence.
|
||||
- Legacy job detail/summary endpoints now marked deprecated with Link/Sunset headers; Console/CLI clients must migrate to `/api/v1/orchestrator/jobs` and `/jobs/{id}` before removal.
|
||||
- ORCH-OAS-62-001 delivered: OpenAPI documents now describe pack-run schedule/retry; SDK pagination and pack-run smoke tests added. Further schedule/retry API changes must keep spec/tests in sync.
|
||||
- Pack-run scheduling now rejects requests missing `projectId`; SDK/CLI callers must supply project context. OpenAPI examples updated accordingly.
|
||||
|
||||
## Next Checkpoints
|
||||
- None scheduled; add orchestrator scheduling/automation sync once upstream readiness dates are committed.
|
||||
|
||||
@@ -57,11 +57,14 @@
|
||||
| 2025-11-29 | ORCH-SVC-35-101 DONE: Implemented export job type registration with quotas/rate policies. Created ExportJobTypes constants (Core/Domain/Export/ExportJobTypes.cs) with hierarchical "export.{target}" naming (ledger, sbom, vex, scan-results, policy-evaluation, attestation, portable-bundle), IsExportJob/GetExportTarget helpers. Created ExportJobPayload record (Core/Domain/Export/ExportJob.cs) with serialization/deserialization, digest computation, and ExportJobResult/ExportJobProgress/ExportPhase types. Implemented ExportJobPolicy (Core/Domain/Export/ExportJobPolicy.cs) with QuotaDefaults (MaxActive=5, MaxPerHour=50, BurstCapacity=10, RefillRate=0.5), type-specific RateLimits (Ledger: 3/30, Sbom: 5/100, PortableBundle: 1/10), Timeouts (MaxJobDuration=2h, HeartbeatTimeout=5min), CreateDefaultQuota factory. Created ExportJobService (Core/Services/ExportJobService.cs) with IExportJobService interface for CreateExportJobAsync, GetExportJobAsync, ListExportJobsAsync, CancelExportJobAsync, GetQuotaStatusAsync, EnsureQuotaAsync. Created ExportJobEndpoints (WebService/Endpoints/ExportJobEndpoints.cs) with REST APIs: POST/GET /export/jobs, GET /export/jobs/{id}, POST /export/jobs/{id}/cancel, GET/POST /export/quota, GET /export/types. Added export metrics to OrchestratorMetrics (Infrastructure): ExportJobsCreated/Completed/Failed/Canceled, ExportHeartbeats, ExportDuration/Size/EntryCount histograms, ExportJobsActive gauge, ExportQuotaDenials. Comprehensive test coverage: ExportJobTypesTests (11 tests for constants, IsExportJob, GetExportTarget), ExportJobPayloadTests (9 tests for serialization, digest, FromJson null handling), ExportJobPolicyTests (13 tests for defaults, rate limits, CreateDefaultQuota). Build succeeds, 84 export tests pass (all passing). | Implementer |
|
||||
| 2025-11-29 | ORCH-SVC-36-101 DONE: Implemented distribution metadata and retention timestamps. Created ExportDistribution record (Core/Domain/Export/ExportJob.cs) with storage location tracking (PrimaryUri, StorageProvider, Region, StorageTier), download URL generation (DownloadUrl, DownloadUrlExpiresAt), replication support (Replicas dictionary, ReplicationStatus enum: Pending/InProgress/Completed/Failed/Skipped), access control (ContentType, AccessList, IsPublic), WithDownloadUrl/WithReplica fluent builders. Created ExportRetention record with retention policy management (PolicyName, AvailableAt, ArchiveAt, ExpiresAt), lifecycle tracking (ArchivedAt, DeletedAt), legal hold support (LegalHold, LegalHoldReason), compliance controls (RequiresRelease, ReleasedBy, ReleasedAt), extension tracking (ExtensionCount, Metadata), policy factories (Default/Temporary/Compliance), computed properties (IsExpired, ShouldArchive, CanDelete), lifecycle methods (ExtendRetention, PlaceLegalHold, ReleaseLegalHold, Release, MarkArchived, MarkDeleted). Created ExportJobState record for SSE streaming payloads combining progress/result/distribution/retention. Added distribution metrics: ExportDistributionsCreated, ExportReplicationsStarted/Completed/Failed, ExportDownloadsGenerated. Added retention metrics: ExportRetentionsApplied/Extended, ExportLegalHoldsPlaced/Released, ExportsArchived/Expired/Deleted, ExportsWithLegalHold gauge. Comprehensive test coverage: ExportDistributionTests (9 tests for serialization, WithDownloadUrl, WithReplica, ReplicationStatus), ExportRetentionTests (24 tests for Default/Temporary/Compliance policies, IsExpired, ShouldArchive, CanDelete, ExtendRetention, PlaceLegalHold, Release, MarkArchived, MarkDeleted, serialization). Build succeeds, 117 export tests pass (+33 new tests). | Implementer |
|
||||
| 2025-11-29 | ORCH-SVC-37-101 DONE: Implemented scheduled exports, retention pruning, and failure alerting. Created ExportSchedule record (Core/Domain/Export/ExportSchedule.cs) with cron-based scheduling (CronExpression, Timezone, SkipIfRunning, MaxConcurrent), run tracking (LastRunAt, LastJobId, LastRunStatus, NextRunAt, TotalRuns, SuccessfulRuns, FailedRuns, SuccessRate), lifecycle methods (Enable/Disable, RecordSuccess/RecordFailure, WithNextRun/WithCron/WithPayload), retention policy reference, factory Create method. Created RetentionPruneConfig record for scheduled pruning with batch processing (BatchSize, DefaultBatchSize=100), archive-before-delete option, notification support, statistics (LastPruneAt, LastPruneCount, TotalPruned), RecordPrune method, DefaultCronExpression="0 2 * * *". Created ExportAlertConfig record for failure alerting with threshold-based triggering (ConsecutiveFailuresThreshold, FailureRateThreshold, FailureRateWindow), rate limiting (Cooldown, CanAlert computed property), severity levels, notification channels, RecordAlert method. Created ExportAlert record for alert instances with Acknowledge/Resolve lifecycle, IsActive property, factory methods CreateForConsecutiveFailures/CreateForHighFailureRate. Created ExportAlertSeverity enum (Info/Warning/Error/Critical). Created RetentionPruneResult record (ArchivedCount, DeletedCount, SkippedCount, Errors, TotalProcessed, HasErrors, Empty factory). Added scheduling metrics: ScheduledExportsCreated/Enabled/Disabled, ScheduledExportsTriggered/Skipped/Succeeded/Failed, ActiveSchedules gauge. Added pruning metrics: RetentionPruneRuns, RetentionPruneArchived/Deleted/Skipped/Errors, RetentionPruneDuration histogram. Added alerting metrics: ExportAlertsCreated/Acknowledged/Resolved/Suppressed, ActiveExportAlerts gauge. Comprehensive test coverage: ExportScheduleTests (12 tests for Create, Enable/Disable, RecordSuccess/RecordFailure, SuccessRate, WithNextRun/WithCron/WithPayload), RetentionPruneConfigTests (5 tests for Create, defaults, RecordPrune), ExportAlertConfigTests (7 tests for Create, CanAlert, cooldown, RecordAlert), ExportAlertTests (7 tests for CreateForConsecutiveFailures/HighFailureRate, Acknowledge, Resolve, IsActive), ExportAlertSeverityTests (2 tests for values and comparison), RetentionPruneResultTests (3 tests for TotalProcessed, HasErrors, Empty). Build succeeds, 157 export tests pass (+40 new tests). | Implementer |
|
||||
| 2025-11-30 | Added local status mirror `src/Orchestrator/StellaOps.Orchestrator/TASKS.md` to stay aligned with sprint tracker; no scope change. | Implementer |
|
||||
| 2025-11-30 | Refreshed legacy stub `SPRINT_152_orchestrator_ii.md` to a read-only pointer to this canonical sprint to prevent divergent updates. | Project Manager |
|
||||
| 2025-11-30 | Marked sprint scope delivered; remaining gating is upstream AirGap/Scanner readiness for integrated rollout. | Project Manager |
|
||||
|
||||
## Decisions & Risks
|
||||
- All tasks depend on outputs from Orchestrator I (32-001); sprint remains TODO until upstream ship.
|
||||
- Maintain deterministic scheduling semantics; avoid issuing control actions until DAG planner/state machine validated.
|
||||
- Ensure offline/air-gap deploy artifacts (Helm/overlays) align with GA packaging in task 34-004.
|
||||
- Upstream Orchestrator I (ORCH-SVC-32-001) completed; this sprint’s scope is fully delivered. Release readiness still depends on AirGap/Scanner gating from Sprint 0150 but does not block code completion here.
|
||||
- Maintain deterministic scheduling semantics; avoid issuing control actions until DAG planner/state machine validated in integrated environments.
|
||||
- Ensure offline/air-gap deploy artifacts (Helm/overlays) stay aligned with GA packaging in task 34-004; rerun bundle script when upstream configs change.
|
||||
|
||||
## Next Checkpoints
|
||||
- Kickoff once ORCH-SVC-32-001 lands (date TBD).
|
||||
- None. Sprint 0152 delivered; monitor Sprint 0150 upstream readiness for release/interop validation windows.
|
||||
|
||||
@@ -21,14 +21,17 @@
|
||||
| --- | --- | --- | --- | --- | --- |
|
||||
| P1 | PREP-ORCH-SVC-41-101-DEPENDS-ON-38-101-ENVELO | DONE (2025-11-22) | Due 2025-11-23 · Accountable: Orchestrator Service Guild | Orchestrator Service Guild | Depends on 38-101 envelope + DAL; cannot register pack-run without API/storage schema. <br><br> Document artefact/deliverable for ORCH-SVC-41-101 and publish location so downstream tasks can proceed. |
|
||||
| 2025-11-20 | Started PREP-ORCH-SVC-41-101 (status → DOING) after confirming no existing DOING/DONE owners. | Planning |
|
||||
| 2025-11-30 | PREP-ORCH-SVC-41-101 auto-closed after ORCH-SVC-41/42 completion; no residual prep. | Implementer |
|
||||
| P2 | PREP-ORCH-SVC-42-101-DEPENDS-ON-41-101-PACK-R | DONE (2025-11-22) | Due 2025-11-23 · Accountable: Orchestrator Service Guild | Orchestrator Service Guild | Depends on 41-101 pack-run plumbing and streaming contract. <br><br> Document artefact/deliverable for ORCH-SVC-42-101 and publish location so downstream tasks can proceed. |
|
||||
| 2025-11-20 | Started PREP-ORCH-SVC-42-101 (status → DOING) after confirming no existing DOING/DONE owners. | Planning |
|
||||
| 2025-11-30 | PREP-ORCH-SVC-42-101 auto-closed after ORCH-SVC-42 delivery; no residual prep. | Implementer |
|
||||
| P3 | PREP-ORCH-TEN-48-001-WEBSERVICE-LACKS-JOB-DAL | DONE (2025-11-22) | Due 2025-11-23 · Accountable: Orchestrator Service Guild | Orchestrator Service Guild | WebService lacks job DAL/routes; need tenant context plumbing before enforcement. <br><br> Document artefact/deliverable for ORCH-TEN-48-001 and publish location so downstream tasks can proceed. |
|
||||
| 2025-11-20 | Started PREP-ORCH-TEN-48-001 (status → DOING) after confirming no existing DOING/DONE owners. | Planning |
|
||||
| 2025-11-30 | PREP-ORCH-TEN-48-001 auto-closed with tenant metadata enforcement delivered. | Implementer |
|
||||
| 1 | ORCH-SVC-38-101 | DONE (2025-11-29) | ORCH-SVC-37-101 complete; WebService DAL exists from Sprint 0152. | Orchestrator Service Guild | Standardize event envelope (policy/export/job lifecycle) with idempotency keys, ensure export/job failure events published to notifier bus with provenance metadata. |
|
||||
| 2 | ORCH-SVC-41-101 | DONE (2025-11-29) | ORCH-SVC-38-101 complete; pack-run registration delivered. | Orchestrator Service Guild | Register `pack-run` job type, persist run metadata, integrate logs/artifacts collection, and expose API for Task Runner scheduling. |
|
||||
| 3 | ORCH-SVC-42-101 | TODO | ORCH-SVC-41-101 complete; proceed with streaming. | Orchestrator Service Guild | Stream pack run logs via SSE/WS, add manifest endpoints, enforce quotas, and emit pack run events to Notifications Studio. |
|
||||
| 4 | ORCH-TEN-48-001 | BLOCKED | PREP-ORCH-TEN-48-001-WEBSERVICE-LACKS-JOB-DAL | Orchestrator Service Guild | Include `tenant_id`/`project_id` in job specs, set DB session context before processing, enforce context on all queries, and reject jobs missing tenant metadata. |
|
||||
| 3 | ORCH-SVC-42-101 | DONE (2025-11-30) | ORCH-SVC-41-101 complete; proceed with streaming. | Orchestrator Service Guild | Stream pack run logs via SSE (with heartbeat/timeouts), manifest endpoint, quota enforcement on schedule, and pack run events to Notifications Studio. |
|
||||
| 4 | ORCH-TEN-48-001 | DONE | PREP-ORCH-TEN-48-001-WEBSERVICE-LACKS-JOB-DAL | Orchestrator Service Guild | Include `tenant_id`/`project_id` in job specs, set DB session context before processing, enforce context on all queries, and reject jobs missing tenant metadata. |
|
||||
| 5 | WORKER-GO-32-001 | DONE | Bootstrap Go SDK scaffolding and smoke sample. | Worker SDK Guild | Bootstrap Go SDK project with configuration binding, auth headers, job claim/acknowledge client, and smoke sample. |
|
||||
| 6 | WORKER-GO-32-002 | DONE | Depends on WORKER-GO-32-001; add heartbeat, metrics, retries. | Worker SDK Guild | Add heartbeat/progress helpers, structured logging hooks, Prometheus metrics, and jittered retry defaults. |
|
||||
| 7 | WORKER-GO-33-001 | DONE | Depends on WORKER-GO-32-002; implement artifact publish helpers. | Worker SDK Guild | Implement artifact publish helpers (object storage client, checksum hashing, metadata payload) and idempotency guard. |
|
||||
@@ -64,17 +67,16 @@
|
||||
| 2025-11-22 | Marked all PREP tasks to DONE per directive; evidence to be verified. | Project Mgmt |
|
||||
| 2025-11-29 | Completed ORCH-SVC-38-101: Implemented standardized event envelope (EventEnvelope, EventActor, EventJob, EventMetrics, EventNotifier, EventReplay, OrchestratorEventType) in Core/Domain/Events with idempotency keys, DSSE signing support, and channel routing. Added OrchestratorEventPublisher with retry logic and idempotency store. Implemented event publishing metrics. Created 86 comprehensive tests. Unblocked ORCH-SVC-41-101. | Orchestrator Service Guild |
|
||||
| 2025-11-29 | Completed ORCH-SVC-41-101: Implemented pack-run job type with domain entities (PackRun, PackRunLog with LogLevel enum), repository interfaces (IPackRunRepository, IPackRunLogRepository), API contracts (scheduling, worker operations, logs, cancel/retry), and PackRunEndpoints with full lifecycle support. Added pack-run metrics to OrchestratorMetrics. Created 56 comprehensive tests. Unblocked ORCH-SVC-42-101 for log streaming. | Orchestrator Service Guild |
|
||||
| 2025-11-30 | ORCH-SVC-42-101 DONE: added pack run Postgres persistence + migration 006, DI registration, pack-run endpoint mapping; implemented SSE stream `/api/v1/orchestrator/stream/pack-runs/{id}` with heartbeats/timeouts + log batches; added manifest endpoint and quota enforcement on scheduling; emitted notifier events; added PackRunStreamCoordinator unit test and ran `dotnet test ... --filter PackRunStreamCoordinatorTests` (pass). | Implementer |
|
||||
| 2025-11-30 | ORCH-TEN-48-001 DONE: job contracts now expose tenant_id/project_id; TenantResolver already enforced; DB session context remains per-tenant via OrchestratorDataSource. No further blocking items. | Implementer |
|
||||
| 2025-11-30 | Enforced ProjectId requirement on pack-run scheduling (tenant header already required) to align with ORCH-TEN-48-001 tenant isolation safeguards. | Implementer |
|
||||
| 2025-11-30 | Normalized Decisions & Risks to reflect completed tenant enforcement and migration 006 requirement. | Implementer |
|
||||
|
||||
|
||||
## Decisions & Risks
|
||||
- Interim token-scoped access approved for AUTH-PACKS-43-001; must tighten once full RBAC lands to prevent over-broad tokens.
|
||||
- Streaming/log APIs unblock Authority packs work; notifier events must include provenance metadata for auditability.
|
||||
- Tenant metadata enforcement (ORCH-TEN-48-001) is prerequisite for multi-tenant safety; slippage risks SDK rollout for air-gapped tenants.
|
||||
- ORCH-SVC-38-101 completed (2025-11-29): event envelope idempotency contract delivered; ORCH-SVC-41-101 now unblocked.
|
||||
- ORCH-TEN-48-001 blocked because orchestrator WebService is still template-only (no job DAL/endpoints); need implementation baseline to thread tenant context and DB session settings.
|
||||
- ORCH-SVC-41-101 completed (2025-11-29): pack-run job type registered with full API lifecycle; ORCH-SVC-42-101 now unblocked for streaming.
|
||||
- Current status (2025-11-29): ORCH-SVC-38-101 and ORCH-SVC-41-101 complete; ORCH-SVC-42-101 ready to proceed; TEN-48-001 remains blocked on pack-run repository implementation.
|
||||
|
||||
- Interim token-scoped access approved for AUTH-PACKS-43-001; tighten when RBAC lands.
|
||||
- Streaming/log APIs unblock Authority packs; notifier events must carry provenance metadata.
|
||||
- Tenant metadata enforcement (ORCH-TEN-48-001) complete (2025-11-30): job contracts expose tenant/project; TenantResolver + per-tenant session context enforced; downstream consumers must align.
|
||||
- ORCH-SVC-38/41/42 complete; migration 006 (pack_runs) is required for upgrade rollout.
|
||||
## Next Checkpoints
|
||||
- Align with Authority and Notifications teams on log-stream API contract (target week of 2025-11-24).
|
||||
- Schedule demo of pack-run streaming (ORCH-SVC-42-101) once SSE/WS path ready; date TBD.
|
||||
- Coordinate migration 006 rollout across environments; verify pack-run SSE demo with Authority/Notifications (target week of 2025-12-02).
|
||||
|
||||
@@ -8,7 +8,7 @@
|
||||
|
||||
## Dependencies & Concurrency
|
||||
- Upstream: Sprint 120.A (AirGap), 130.A (Scanner), 140.A (Graph) provide pack metadata and graph inputs.
|
||||
- Concurrency: execute tasks in table order; all tasks currently TODO.
|
||||
- Concurrency: execution followed table order; all tasks now DONE.
|
||||
|
||||
## Documentation Prerequisites
|
||||
- docs/README.md
|
||||
@@ -25,6 +25,27 @@
|
||||
| 2 | PACKS-REG-42-001 | DONE (2025-11-25) | Depends on 41-001. | Packs Registry Guild | Version lifecycle (promote/deprecate), tenant allowlists, provenance export, signature rotation, audit logs, Offline Kit seed support. |
|
||||
| 3 | PACKS-REG-43-001 | DONE (2025-11-25) | Depends on 42-001. | Packs Registry Guild | Registry mirroring, pack signing policies, attestation integration, compliance dashboards; integrate with Export Center. |
|
||||
|
||||
## Wave Coordination
|
||||
- Single wave (150.B Packs Registry). Parallel waves tracked under Sprint 150 umbrella are out of scope here.
|
||||
|
||||
## Wave Detail Snapshots
|
||||
- 150.B Packs Registry — all Delivery Tracker items marked DONE as of 2025-11-25.
|
||||
|
||||
## Interlocks
|
||||
- Upstream contracts from AirGap/Scanner/Graph (Sprint 120.A/130.A/140.A) assumed stable; re-open risk if schemas change.
|
||||
|
||||
## Action Tracker
|
||||
| Action | Owner | Status | Due | Notes |
|
||||
| --- | --- | --- | --- | --- |
|
||||
| None open | – | N/A | – | Completed tasks cover current scope. |
|
||||
|
||||
## Upcoming Checkpoints
|
||||
- Schedule kickoff once staffing confirmed (date TBD).
|
||||
|
||||
## Decisions & Risks
|
||||
- Registry relies on upstream pack metadata/graph contracts; keep schema aligned before migrations run.
|
||||
- Ensure offline posture: signature verification, provenance storage, audit logs, and Offline Kit seeds are mandatory before GA.
|
||||
|
||||
## Execution Log
|
||||
| Date (UTC) | Update | Owner |
|
||||
| --- | --- | --- |
|
||||
@@ -42,10 +63,6 @@
|
||||
| 2025-11-25 | Completed PACKS-REG-42-001: lifecycle/parity listing + audit trail repos (file/memory/mongo), signature rotation endpoint, offline-seed zip export with provenance/content, tenant allowlist enforcement on listings, OpenAPI updates; upgraded tests to ASP.NET Core 10 RC and added coverage for exports/rotation. | Implementer |
|
||||
| 2025-11-25 | Completed PACKS-REG-43-001: attestation storage/download APIs (file/memory/mongo), mirror registry CRUD/sync endpoints, pack signing policy option, compliance summary endpoint, OpenAPI v0.3 updated; all tests green. | Implementer |
|
||||
| 2025-11-25 | Closed PACKS-REG-41-001 after migrations, RBAC, signature verification, digest headers, and content/provenance storage completed. | Implementer |
|
||||
|
||||
## Decisions & Risks
|
||||
- Registry relies on upstream pack metadata/graph contracts; keep schema aligned before migrations run.
|
||||
- Ensure offline posture: signature verification, provenance storage, audit logs, and Offline Kit seeds are mandatory before GA.
|
||||
|
||||
## Next Checkpoints
|
||||
- Schedule kickoff once staffing confirmed (date TBD).
|
||||
| 2025-11-30 | Re-applied legacy file redirect stub and added template sections (wave/interlocks/action tracker/upcoming checkpoints); no task status changes. | Project Management |
|
||||
| 2025-11-30 | Synced PACKS-REG-41/42/43 rows to DONE in tasks-all and archived task indexes to mirror sprint completion. | Project Management |
|
||||
| 2025-11-30 | Ran `StellaOps.PacksRegistry.Tests` (net10.0) — restore from local feed succeeded; 8 tests passed, 0 failed. | Implementer |
|
||||
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user