git.stella-ops.org/docs-archived/product/advisories/2026-03-04 - Trace‑to‑source lineage and reproducible replay harness.md

Here’s a compact, practical pattern for making runtime traces auditable end‑to‑end—so every stack frame ties back to a signed build and can be replayed deterministically.

Why this matters (in plain terms)

When something crashes or behaves oddly, you want to prove which code actually ran, who built it, with what flags, and replay it. The pattern below links: trace → symbol bundle → build artifact → signed provenance, and stores small “replay harness” contracts so auditors (or future you) can reproduce the run.

---

1) Join model: trace → symbols → artifacts → provenance

Use content‑addressed keys that already exist in your toolchain:

• Frames: instruction pointer (IP), build‑id
• Symbols: symbol bundle hash (e.g., sha256 of PDB/dSYM/ELF DWARF bundle)
• Artifacts: release image/object sha256, compiler, flags, commit
• Provenance: DSSE envelope + Rekor inclusion proof (tile ref)

SQL pattern (drop‑in for Postgres):

SELECT
  f.trace_id,
  f.frame_index,
  f.ip,
  f.resolved_symbol,
  s.sha256 AS symbol_bundle,
  a.artifact_id,
  a.builder_commit,
  a.compiler,
  a.compiler_flags,
  a.provenance_dsse
FROM frames f
JOIN symbol_bundles s
  ON f.symbol_bundle_sha256 = s.sha256
JOIN artifacts a
  ON s.origin_artifact_sha256 = a.sha256
WHERE f.trace_id = $1
ORDER BY f.frame_index;

This yields a per‑frame audit trail from IP → symbol → artifact → signed provenance.

Content‑addressed keys you can leverage:

• Linux: function blob sha256, build-id note
• Windows: PDB GUID+Age
• macOS: dSYM UUID
• OCI: layer/config sha256

---

2) Minimal “replay harness” contract (store per trace/run)

Keep a tiny JSON alongside the trace row (e.g., replays.replay_manifest JSONB). It pins environment, symbols, and evidence pointers:

{
  "harness_version": "1",
  "os": "linux|windows|macos",
  "kernel_version": "5.x|10.x|..",
  "libc_version": "glibc 2.3.4",
  "compiler": "gcc 12.1",
  "compiler_flags": "-g -O2 -fno-omit-frame-pointer",
  "build_id": "<id>",
  "symbol_bundle_sha256": "sha256:...",
  "dsse_envelope": "dsse:...",
  "rekor_tile_ref": "rekor:...",
  "sandbox_image_sha256": "sha256:...",
  "seed": 123456,
  "run_instructions": "deterministic-run.sh --seed $seed",
  "verifier_version": "v1.2.3"
}

---

3) Acceptance & auditor checks (automatable)

1. Evidence integrity

   • Verify DSSE (signature + subject)
   • Verify Rekor inclusion proof matches rekor_tile_ref

2. Provenance join completeness

   • ≥95% of top‑N frames resolve to symbol bundles and linked artifacts

3. Reproducible replay

   • Harness run achieves replay_success_ratio ≥ 95%
   • For “forensic” policy: bit‑identical final state

4. Chain‑of‑custody

   • Each join includes signer identity, timestamp, and insertion_rekor_tile_ref

---

4) Operational recommendations (Stella Ops‑ready)

• Gate symbol intake: require SLSA/in‑toto/DSSE attestation before accepting symbol bundles.
• Persist replay contracts: store the JSON above next to each trace (Postgres JSONB).
• One‑click “Audit bundle” export: deliver {trace, symbol_bundles, DSSE envelopes, Rekor tile fragments, replay harness} as a content‑addressed archive for offline/legal review.
• Policies: make “join completeness” and “replay ratio” first‑class pass/fail gates in EvidenceLocker.

---

5) Where this plugs into Stella Ops

• EvidenceLocker: stores DSSE, Rekor fragments, and replay manifests.
• Attestor: validates DSSE + Rekor, stamps chain‑of‑custody.
• ReleaseOrchestrator: enforces “no symbols without attestation”.
• Doctor: offers a “Reproduce this crash” action that pulls the harness and runs it in a pinned sandbox.
• AdvisoryAI: can surface “provenance gaps” and recommend remediation (e.g., missing dSYM, mismatched PDB Age).

---

6) Quick backlog (bite‑sized tasks)

• Tables: frames, symbol_bundles, artifacts, provenance_evidence, replays (JSONB).
• Ingestors: symbol bundle hasher; artifact provenance fetcher; Rekor proof cache.
• Verifiers: DSSE verify, Rekor inclusion verify, join‑completeness scorer, replay runner.
• UI: Trace view with “Audit bundle” download + policy badges (join %, replay %, signer).

If you want, I can draft the Postgres DDL + a tiny Go/TS service that:

1. ingests a trace,
2. resolves frames against symbols,
3. joins to artifacts via sha256/build-id/PDB GUID+Age/dSYM UUID,
4. verifies DSSE/Rekor,
5. emits the replay manifest and an exportable audit bundle.