git.stella-ops.org/docs/reachability/patch-oracles.md

Patch-Oracles QA Pattern

Patch oracles define expected functions and edges that must be present (or absent) in generated reachability graphs. The CI pipeline uses these oracles to ensure that:

1. Critical vulnerability paths are correctly identified as reachable
2. Mitigated paths are correctly identified as unreachable
3. Graph generation remains deterministic and complete

This document covers both the JSON-based harness (for reachbench integration) and the YAML-based format (for binary patch testing).

---

Part A: JSON Patch-Oracle Harness (v1)

The JSON-based patch-oracle harness integrates with the reachbench fixture system for CI graph validation.

A.1 Schema Overview

Patch-oracle fixtures follow the patch-oracle/v1 schema:

{
  "schema_version": "patch-oracle/v1",
  "id": "curl-CVE-2023-38545-socks5-heap-reachable",
  "case_ref": "curl-CVE-2023-38545-socks5-heap",
  "variant": "reachable",
  "description": "Validates SOCKS5 heap overflow path is reachable",
  "expected_functions": [...],
  "expected_edges": [...],
  "expected_roots": [...],
  "forbidden_functions": [...],
  "forbidden_edges": [...],
  "min_confidence": 0.5,
  "strict_mode": false
}

A.2 Expected Functions

Define functions that MUST be present in the graph:

{
  "symbol_id": "sym://curl:curl.c#sink",
  "lang": "c",
  "kind": "function",
  "purl_pattern": "pkg:github/curl/*",
  "required": true,
  "reason": "Vulnerable buffer handling function"
}

A.3 Expected Edges

Define edges that MUST be present in the graph:

{
  "from": "sym://net:handler#read",
  "to": "sym://curl:curl.c#entry",
  "kind": "call",
  "min_confidence": 0.8,
  "required": true,
  "reason": "Data flows from network to SOCKS5 handler"
}

A.4 Forbidden Elements (for unreachable variants)

{
  "forbidden_functions": [
    {
      "symbol_id": "sym://dangerous#sink",
      "reason": "Should not be reachable when feature disabled"
    }
  ],
  "forbidden_edges": [
    {
      "from": "sym://entry",
      "to": "sym://sink",
      "reason": "Path should be blocked by feature flag"
    }
  ]
}

A.5 Wildcard Patterns

Symbol IDs support * wildcards:

• sym://test#func1 - exact match
• sym://test#* - matches any symbol starting with sym://test#
• * - matches anything

A.6 Directory Structure

tests/reachability/fixtures/patch-oracles/
├── INDEX.json                    # Oracle index
├── schema/
│   └── patch-oracle-v1.json      # JSON Schema
└── cases/
    ├── curl-CVE-2023-38545-socks5-heap/
    │   ├── reachable.oracle.json
    │   └── unreachable.oracle.json
    └── java-log4j-CVE-2021-44228-log4shell/
        └── reachable.oracle.json

A.7 Usage in Tests

var loader = new PatchOracleLoader(fixtureRoot);
var oracle = loader.LoadOracle("curl-CVE-2023-38545-socks5-heap-reachable");

var comparer = new PatchOracleComparer(oracle);
var result = comparer.Compare(richGraph);

if (!result.Success)
{
    foreach (var violation in result.Violations)
    {
        Console.WriteLine($"[{violation.Type}] {violation.From} -> {violation.To}");
    }
}

A.8 Violation Types

Type	Description
MissingFunction	Required function not found
MissingEdge	Required edge not found
MissingRoot	Required root not found
ForbiddenFunctionPresent	Forbidden function found
ForbiddenEdgePresent	Forbidden edge found
UnexpectedFunction	Unexpected function in strict mode
UnexpectedEdge	Unexpected edge in strict mode

---

Part B: YAML Binary Patch-Oracles

The YAML-based format is used for paired vulnerable/fixed binary testing.

B.1 Workflow (per CVE)

1. Pick a CVE with a small, clean fix (e.g., OpenSSL, zlib, BusyBox). Identify vulnerable commit A and fixed commit B.
2. Build two stripped binaries (vuln, fixed) with identical toolchains/flags; keep a tiny harness that exercises the affected path.
3. Run Scanner binary analyzers to emit richgraph-v1 for each binary.
4. Diff graphs: expect new/removed functions and edges to match the patch (e.g., foo_parse -> validate_len added; foo_parse -> memcpy removed).
5. Fail the test if expected functions/edges are absent or unchanged.

B.2 Oracle manifest (YAML)

cve: CVE-YYYY-XXXX
target: libfoo 1.2.3
build:
  cc: clang
  cflags: [-O2, -fno-omit-frame-pointer]
  ldflags: []
  strip: true
expect:
  functions_added: [validate_len]
  functions_removed: [unsafe_copy]
  edges_added:
    - { caller: foo_parse, callee: validate_len }
  edges_removed:
    - { caller: foo_parse, callee: memcpy }
tolerances:
  allow_unresolved_symbols: 0
  allow_extra_funcs: 2

Place manifests under tests/reachability/patch-oracles/<cve>/oracle.yml next to the sources/build scripts.

3. Repository layout

tests/reachability/patch-oracles/
  CVE-YYYY-XXXX-foo/
    src/            # vuln + fixed sources + harness
    build.sh        # produces ./out/vuln ./out/fixed
    oracle.yml

4. Harness rules

• Output binaries to out/vuln and out/fixed with deterministic flags and stripped symbols.
• Record toolchain version in a sidecar build-meta.json so Replay captures provenance.
• Never download from the internet during CI; vendor tiny sources into the fixture folder.

5. Test runner expectations

• Runs Scanner binary analyzers on both binaries; emits richgraph-v1 CAS entries.
• Compares graphs against oracle.yml expectations (functions/edges added/removed, tolerances).
• Fails when deltas are missing; succeeds when expected guards/edges are present.

6. Integration points

• Scanner: add fixture runner under tests/reachability/StellaOps.Scanner.Binary.PatchOracleTests.
• CI: wire into reachbench/patch-oracles job; ensure artifacts are small and deterministic.
• Docs: link this file from reachability delivery guide once tests are live.

B.7 Acceptance criteria

• At least three seed oracles (e.g., zlib overflow, OpenSSL length guard, BusyBox ash fix) committed with passing expectations.
• CI job proves deterministic hashes across reruns.
• Failures emit clear diffs (expected edge foo->validate_len missing).

---

Related Documentation

• Reachability Evidence Chain
• RichGraph Schema
• Ground Truth Schema
• Lattice States
• Reachability Delivery Guide