Reachability Test Datasets

This directory contains ground truth samples for validating reachability analysis accuracy.

Directory Structure

datasets/reachability/
├── README.md                     # This file
├── samples/                      # Test samples by language
│   ├── csharp/
│   │   ├── simple-reachable/     # Positive: direct call path
│   │   └── dead-code/            # Negative: unreachable code
│   ├── java/
│   │   └── vulnerable-log4j/     # Positive: Log4Shell CVE
│   └── native/
│       └── stripped-elf/         # Positive: stripped binary
└── schema/
    ├── manifest.schema.json      # Sample manifest schema
    └── ground-truth.schema.json  # Ground truth schema

Sample Categories

Positive (Reachable)

Samples where vulnerable code has a confirmed path from entry points:

csharp/simple-reachable - Direct call to vulnerable API
java/vulnerable-log4j - Log4Shell with runtime confirmation
native/stripped-elf - Stripped ELF with heuristic analysis

Negative (Unreachable)

Samples where vulnerable code exists but is never called:

csharp/dead-code - Deprecated API replaced by safe implementation

Schema Reference

manifest.json

Sample metadata including:

sampleId - Unique identifier
language - Primary language (java, csharp, native, etc.)
category - positive, negative, or contested
vulnerabilities - CVEs and affected symbols
artifacts - Binary/SBOM file references

ground-truth.json

Expected outcomes including:

targets - Symbols with expected lattice states
entryPoints - Program entry points
expectedUncertainty - Expected uncertainty tier
expectedGateDecisions - Expected policy gate outcomes

Lattice States

Code	Name	Description
U	Unknown	No analysis performed
SR	StaticallyReachable	Static analysis finds path
SU	StaticallyUnreachable	Static analysis finds no path
RO	RuntimeObserved	Runtime probe observed execution
RU	RuntimeUnobserved	Runtime probe did not observe
CR	ConfirmedReachable	Both static and runtime confirm
CU	ConfirmedUnreachable	Both static and runtime confirm unreachable
X	Contested	Static and runtime evidence conflict

Running Tests

# Validate schemas
npx ajv validate -s schema/ground-truth.schema.json -d samples/**/ground-truth.json

# Run benchmark tests
dotnet test --filter "GroundTruth" src/Scanner/__Tests/StellaOps.Scanner.Reachability.Benchmarks/

Adding New Samples

Create directory: samples/{language}/{sample-name}/
Add manifest.json with sample metadata
Add ground-truth.json with expected outcomes
Include reasoning for each target explaining the expected state
Validate against schema before committing

3.0 KiB Raw Blame History