Files

master 4789027317 docs consolidation and others

2026-01-06 19:07:48 +02:00

6.4 KiB

Raw Permalink Blame History

Patch-Oracles QA Pattern

Patch oracles define expected functions and edges that must be present (or absent) in generated reachability graphs. The CI pipeline uses these oracles to ensure that:

Critical vulnerability paths are correctly identified as reachable
Mitigated paths are correctly identified as unreachable
Graph generation remains deterministic and complete

This document covers both the JSON-based harness (for reachbench integration) and the YAML-based format (for binary patch testing).

Part A: JSON Patch-Oracle Harness (v1)

The JSON-based patch-oracle harness integrates with the reachbench fixture system for CI graph validation.

A.1 Schema Overview

Patch-oracle fixtures follow the patch-oracle/v1 schema:

{
  "schema_version": "patch-oracle/v1",
  "id": "curl-CVE-2023-38545-socks5-heap-reachable",
  "case_ref": "curl-CVE-2023-38545-socks5-heap",
  "variant": "reachable",
  "description": "Validates SOCKS5 heap overflow path is reachable",
  "expected_functions": [...],
  "expected_edges": [...],
  "expected_roots": [...],
  "forbidden_functions": [...],
  "forbidden_edges": [...],
  "min_confidence": 0.5,
  "strict_mode": false
}

A.2 Expected Functions

Define functions that MUST be present in the graph:

{
  "symbol_id": "sym://curl:curl.c#sink",
  "lang": "c",
  "kind": "function",
  "purl_pattern": "pkg:github/curl/*",
  "required": true,
  "reason": "Vulnerable buffer handling function"
}

A.3 Expected Edges

Define edges that MUST be present in the graph:

{
  "from": "sym://net:handler#read",
  "to": "sym://curl:curl.c#entry",
  "kind": "call",
  "min_confidence": 0.8,
  "required": true,
  "reason": "Data flows from network to SOCKS5 handler"
}

A.4 Forbidden Elements (for unreachable variants)

{
  "forbidden_functions": [
    {
      "symbol_id": "sym://dangerous#sink",
      "reason": "Should not be reachable when feature disabled"
    }
  ],
  "forbidden_edges": [
    {
      "from": "sym://entry",
      "to": "sym://sink",
      "reason": "Path should be blocked by feature flag"
    }
  ]
}

A.5 Wildcard Patterns

Symbol IDs support * wildcards:

sym://test#func1 - exact match
sym://test#* - matches any symbol starting with sym://test#
* - matches anything

A.6 Directory Structure

tests/reachability/fixtures/patch-oracles/
├── INDEX.json                    # Oracle index
├── schema/
│   └── patch-oracle-v1.json      # JSON Schema
└── cases/
    ├── curl-CVE-2023-38545-socks5-heap/
    │   ├── reachable.oracle.json
    │   └── unreachable.oracle.json
    └── java-log4j-CVE-2021-44228-log4shell/
        └── reachable.oracle.json

A.7 Usage in Tests

var loader = new PatchOracleLoader(fixtureRoot);
var oracle = loader.LoadOracle("curl-CVE-2023-38545-socks5-heap-reachable");

var comparer = new PatchOracleComparer(oracle);
var result = comparer.Compare(richGraph);

if (!result.Success)
{
    foreach (var violation in result.Violations)
    {
        Console.WriteLine($"[{violation.Type}] {violation.From} -> {violation.To}");
    }
}

A.8 Violation Types

Type	Description
`MissingFunction`	Required function not found
`MissingEdge`	Required edge not found
`MissingRoot`	Required root not found
`ForbiddenFunctionPresent`	Forbidden function found
`ForbiddenEdgePresent`	Forbidden edge found
`UnexpectedFunction`	Unexpected function in strict mode
`UnexpectedEdge`	Unexpected edge in strict mode

Part B: YAML Binary Patch-Oracles

The YAML-based format is used for paired vulnerable/fixed binary testing.

B.1 Workflow (per CVE)

Pick a CVE with a small, clean fix (e.g., OpenSSL, zlib, BusyBox). Identify vulnerable commit A and fixed commit B.
Build two stripped binaries (vuln, fixed) with identical toolchains/flags; keep a tiny harness that exercises the affected path.
Run Scanner binary analyzers to emit richgraph-v1 for each binary.
Diff graphs: expect new/removed functions and edges to match the patch (e.g., foo_parse -> validate_len added; foo_parse -> memcpy removed).
Fail the test if expected functions/edges are absent or unchanged.

B.2 Oracle manifest (YAML)

cve: CVE-YYYY-XXXX
target: libfoo 1.2.3
build:
  cc: clang
  cflags: [-O2, -fno-omit-frame-pointer]
  ldflags: []
  strip: true
expect:
  functions_added: [validate_len]
  functions_removed: [unsafe_copy]
  edges_added:
    - { caller: foo_parse, callee: validate_len }
  edges_removed:
    - { caller: foo_parse, callee: memcpy }
tolerances:
  allow_unresolved_symbols: 0
  allow_extra_funcs: 2

Place manifests under tests/reachability/patch-oracles/<cve>/oracle.yml next to the sources/build scripts.

3. Repository layout

tests/reachability/patch-oracles/
  CVE-YYYY-XXXX-foo/
    src/            # vuln + fixed sources + harness
    build.sh        # produces ./out/vuln ./out/fixed
    oracle.yml

4. Harness rules

Output binaries to out/vuln and out/fixed with deterministic flags and stripped symbols.
Record toolchain version in a sidecar build-meta.json so Replay captures provenance.
Never download from the internet during CI; vendor tiny sources into the fixture folder.

5. Test runner expectations

Runs Scanner binary analyzers on both binaries; emits richgraph-v1 CAS entries.
Compares graphs against oracle.yml expectations (functions/edges added/removed, tolerances).
Fails when deltas are missing; succeeds when expected guards/edges are present.

6. Integration points

Scanner: add fixture runner under tests/reachability/StellaOps.Scanner.Binary.PatchOracleTests.
CI: wire into reachbench/patch-oracles job; ensure artifacts are small and deterministic.
Docs: link this file from reachability delivery guide once tests are live.

B.7 Acceptance criteria

At least three seed oracles (e.g., zlib overflow, OpenSSL length guard, BusyBox ash fix) committed with passing expectations.
CI job proves deterministic hashes across reruns.
Failures emit clear diffs (expected edge foo->validate_len missing).

6.4 KiB Raw Permalink Blame History