Patch-Oracles QA Pattern (Nov 2026)

Patch oracles are paired vulnerable/fixed binaries that prove our analyzers can see the function and call-edge deltas introduced by real CVE fixes. This file replaces earlier advisory text; use it directly when adding tests.

1. Workflow (per CVE)

Pick a CVE with a small, clean fix (e.g., OpenSSL, zlib, BusyBox). Identify vulnerable commit A and fixed commit B.
Build two stripped binaries (vuln, fixed) with identical toolchains/flags; keep a tiny harness that exercises the affected path.
Run Scanner binary analyzers to emit richgraph-v1 for each binary.
Diff graphs: expect new/removed functions and edges to match the patch (e.g., foo_parse -> validate_len added; foo_parse -> memcpy removed).
Fail the test if expected functions/edges are absent or unchanged.

2. Oracle manifest (YAML)

cve: CVE-YYYY-XXXX
target: libfoo 1.2.3
build:
  cc: clang
  cflags: [-O2, -fno-omit-frame-pointer]
  ldflags: []
  strip: true
expect:
  functions_added: [validate_len]
  functions_removed: [unsafe_copy]
  edges_added:
    - { caller: foo_parse, callee: validate_len }
  edges_removed:
    - { caller: foo_parse, callee: memcpy }
tolerances:
  allow_unresolved_symbols: 0
  allow_extra_funcs: 2

Place manifests under tests/reachability/patch-oracles/<cve>/oracle.yml next to the sources/build scripts.

3. Repository layout

tests/reachability/patch-oracles/
  CVE-YYYY-XXXX-foo/
    src/            # vuln + fixed sources + harness
    build.sh        # produces ./out/vuln ./out/fixed
    oracle.yml

4. Harness rules

Output binaries to out/vuln and out/fixed with deterministic flags and stripped symbols.
Record toolchain version in a sidecar build-meta.json so Replay captures provenance.
Never download from the internet during CI; vendor tiny sources into the fixture folder.

5. Test runner expectations

Runs Scanner binary analyzers on both binaries; emits richgraph-v1 CAS entries.
Compares graphs against oracle.yml expectations (functions/edges added/removed, tolerances).
Fails when deltas are missing; succeeds when expected guards/edges are present.

6. Integration points

Scanner: add fixture runner under tests/reachability/StellaOps.Scanner.Binary.PatchOracleTests.
CI: wire into reachbench/patch-oracles job; ensure artifacts are small and deterministic.
Docs: link this file from reachability delivery guide once tests are live.

7. Acceptance criteria

At least three seed oracles (e.g., zlib overflow, OpenSSL length guard, BusyBox ash fix) committed with passing expectations.
CI job proves deterministic hashes across reruns.
Failures emit clear diffs (expected edge foo->validate_len missing).

2.7 KiB Raw Blame History