git.stella-ops.org/docs/modules/scanner/guides/runtime-linkage.md

# Runtime Linkage Verification Guide

> **Ownership:** Scanner Guild / Signals Guild
> **Services:** `StellaOps.Scanner.Reachability.FunctionMap`
> **API:** `POST /api/v1/function-maps`, `POST /api/v1/function-maps/{id}/verify`
> **CLI:** `stella function-map generate|verify`, `stella observations query`

## What is Runtime Linkage Verification?

Runtime linkage verification bridges the gap between **static analysis** (what code _could_ run) and **runtime observation** (what code _actually_ runs). It works by:

1. **Generating a function map** from static analysis (SBOM + call graph) that declares expected call paths
2. **Deploying probes** (eBPF uprobes/kprobes) to observe actual function invocations at runtime
3. **Verifying** that observed call patterns match the expected static model

This produces a confidence metric (observation rate) quantifying how much of the declared attack surface has been confirmed by runtime evidence.

---

## When to Use Function Maps

| Scenario | Benefit |
|----------|---------|
| **High-risk vulnerabilities** | Confirm whether vulnerable code paths are actually exercised |
| **Reachability disputes** | Resolve static "maybe reachable" findings with runtime evidence |
| **Compliance audits** | Provide cryptographic proof of runtime behavior |
| **Air-gapped environments** | Bundle function maps and observations for offline verification |
| **Continuous monitoring** | Track coverage drift over deployment lifecycle |

---

## Step-by-Step Guide

### 1. Generate a Function Map

Create a function map predicate from your SBOM and optional static analysis:

```bash
stella function-map generate \
  --sbom ./app.cdx.json \
  --service my-backend \
  --hot-functions "crypto/*" --hot-functions "auth/*" \
  --min-rate 0.95 \
  --window 1800 \
  --output function-map.json
```

**Key options:**
- `--hot-functions`: Glob patterns for functions of interest (crypto, auth, network are common)
- `--min-rate`: Minimum observation rate to consider "verified" (default 0.95 = 95%)
- `--window`: Observation window in seconds (default 1800 = 30 minutes)
- `--static-analysis`: Path to static analysis results for richer call paths

The output is a JSON predicate conforming to `https://stella.ops/predicates/function-map/v1`.

### 2. Deploy Probes

Configure the Stella runtime agent to attach probes for the functions declared in your map. The agent uses eBPF to observe function calls without modifying application code.

Supported probe types:
- `uprobe` / `uretprobe` — User-space function entry/exit
- `kprobe` / `kretprobe` — Kernel function entry/exit
- `tracepoint` — Kernel tracepoints
- `usdt` — User-space statically defined tracing

The runtime agent writes observations in NDJSON format with fields:
- `node_hash` — SHA-256(PURL + normalized symbol)
- `function_name` — Observed function symbol
- `probe_type` — How it was observed
- `observed_at` — Timestamp
- `container_id`, `pod_name`, `namespace` — Context

### 3. Verify Observations Against the Map

After accumulating observations, verify coverage:

```bash
stella function-map verify \
  --function-map function-map.json \
  --from "2026-01-23T00:00:00Z" \
  --to "2026-01-23T01:00:00Z" \
  --format table
```

For offline verification with a bundled observations file:

```bash
stella function-map verify \
  --function-map function-map.json \
  --offline \
  --observations observations.ndjson \
  --format json
```

**Output includes:**
- `verified`: Whether observation rate meets the threshold
- `observation_rate`: Fraction of expected paths confirmed (0.0-1.0)
- `target_rate`: Required rate from the function map
- `per_path_breakdown`: Status of each declared call path
- `unexpected_symbols`: Functions observed but not in the map
- `missing_symbols`: Expected functions not yet observed

### 4. Upload to Platform (Optional)

Store function maps in the Platform for centralized management:

```bash
# Create via API
curl -X POST /api/v1/function-maps \
  -H "Content-Type: application/json" \
  -d @function-map.json

# Verify via API
curl -X POST /api/v1/function-maps/{id}/verify \
  -H "Content-Type: application/json" \
  -d '{"observations": [...]}'

# Check coverage dashboard
curl GET /api/v1/function-maps/{id}/coverage
```

---

## Predicate Schema

Function maps use the in-toto attestation framework with predicate type:

```
https://stella.ops/predicates/function-map/v1
```

See [Function Map V1 Contract](../../../contracts/function-map-v1.md) for the full schema specification.

---

## Integration with Air-Gap Bundles

Function maps, observations, and verification reports can be included in offline bundles:

```
bundle.stella.bundle.tgz
├── function-maps/
│   ├── {service}-function-map.json
│   └── {service}-function-map.dsse.json
├── observations/
│   └── {date-label}-observations.ndjson
└── verification/
    ├── verification-report.json
    └── verification-report.dsse.json
```

See [Offline Bundle Format](../../airgap/guides/offline-bundle-format.md) for artifact type details.

---

## Troubleshooting

### Low Observation Rate

**Symptom:** Verification reports `observation_rate < target_rate`.

**Causes:**
- Observation window too short — increase `--window` or widen `--from`/`--to`
- Probes not attached — check runtime agent logs for attachment failures
- Application hasn't exercised the code paths — generate representative load
- Binary stripped or ASLR — provide `--binary-path` hints in the function map

**Resolution:**
1. Use `stella observations query --summary` to see what's been collected
2. Check per-path breakdown for which specific paths are unobserved
3. Extend the observation window or trigger relevant application behavior

### Unexpected Symbols

**Symptom:** Verification reports unexpected function calls not in the map.

**Causes:**
- Dynamic dispatch or reflection invoking functions not in static analysis
- Shared libraries loaded at runtime that weren't in the SBOM
- Hot functions pattern too narrow

**Resolution:**
1. Regenerate the function map with broader `--hot-functions` patterns
2. Add the unexpected symbols as optional paths if they're benign
3. Set `--fail-on-unexpected false` if unexpected calls should be informational only

### Node Hash Mismatch

**Symptom:** Observations exist but don't match expected node hashes.

**Causes:**
- PURL mismatch between SBOM and runtime (version drift)
- Symbol name normalization differences (C++ mangling, etc.)

**Resolution:**
1. Verify the PURL in observations matches the function map subject
2. Check that symbol names are normalized consistently (same demangling rules)
3. Regenerate the function map with the current deployed SBOM version