70 lines
2.7 KiB
Markdown
70 lines
2.7 KiB
Markdown
# Patch-Oracles QA Pattern (Nov 2026)
|
|
|
|
Patch oracles are paired vulnerable/fixed binaries that prove our analyzers can see the function and call-edge deltas introduced by real CVE fixes. This file replaces earlier advisory text; use it directly when adding tests.
|
|
|
|
## 1. Workflow (per CVE)
|
|
|
|
1) Pick a CVE with a small, clean fix (e.g., OpenSSL, zlib, BusyBox). Identify vulnerable commit `A` and fixed commit `B`.
|
|
2) Build two stripped binaries (`vuln`, `fixed`) with identical toolchains/flags; keep a tiny harness that exercises the affected path.
|
|
3) Run Scanner binary analyzers to emit `richgraph-v1` for each binary.
|
|
4) Diff graphs: expect new/removed functions and edges to match the patch (e.g., `foo_parse -> validate_len` added; `foo_parse -> memcpy` removed).
|
|
5) Fail the test if expected functions/edges are absent or unchanged.
|
|
|
|
## 2. Oracle manifest (YAML)
|
|
|
|
```yaml
|
|
cve: CVE-YYYY-XXXX
|
|
target: libfoo 1.2.3
|
|
build:
|
|
cc: clang
|
|
cflags: [-O2, -fno-omit-frame-pointer]
|
|
ldflags: []
|
|
strip: true
|
|
expect:
|
|
functions_added: [validate_len]
|
|
functions_removed: [unsafe_copy]
|
|
edges_added:
|
|
- { caller: foo_parse, callee: validate_len }
|
|
edges_removed:
|
|
- { caller: foo_parse, callee: memcpy }
|
|
tolerances:
|
|
allow_unresolved_symbols: 0
|
|
allow_extra_funcs: 2
|
|
```
|
|
|
|
Place manifests under `tests/reachability/patch-oracles/<cve>/oracle.yml` next to the sources/build scripts.
|
|
|
|
## 3. Repository layout
|
|
|
|
```
|
|
tests/reachability/patch-oracles/
|
|
CVE-YYYY-XXXX-foo/
|
|
src/ # vuln + fixed sources + harness
|
|
build.sh # produces ./out/vuln ./out/fixed
|
|
oracle.yml
|
|
```
|
|
|
|
## 4. Harness rules
|
|
|
|
- Output binaries to `out/vuln` and `out/fixed` with deterministic flags and stripped symbols.
|
|
- Record toolchain version in a sidecar `build-meta.json` so Replay captures provenance.
|
|
- Never download from the internet during CI; vendor tiny sources into the fixture folder.
|
|
|
|
## 5. Test runner expectations
|
|
|
|
- Runs Scanner binary analyzers on both binaries; emits `richgraph-v1` CAS entries.
|
|
- Compares graphs against `oracle.yml` expectations (functions/edges added/removed, tolerances).
|
|
- Fails when deltas are missing; succeeds when expected guards/edges are present.
|
|
|
|
## 6. Integration points
|
|
|
|
- **Scanner**: add fixture runner under `tests/reachability/StellaOps.Scanner.Binary.PatchOracleTests`.
|
|
- **CI**: wire into reachbench/patch-oracles job; ensure artifacts are small and deterministic.
|
|
- **Docs**: link this file from reachability delivery guide once tests are live.
|
|
|
|
## 7. Acceptance criteria
|
|
|
|
- At least three seed oracles (e.g., zlib overflow, OpenSSL length guard, BusyBox ash fix) committed with passing expectations.
|
|
- CI job proves deterministic hashes across reruns.
|
|
- Failures emit clear diffs (`expected edge foo->validate_len missing`).
|