# Replay Test Strategy (Draft)

> **Ownership:** Docs Guild · Scanner Guild · Evidence Locker Guild · QA Guild  
> **Related:** `docs/replay/DETERMINISTIC_REPLAY.md`, `docs/replay/DEVS_GUIDE_REPLAY.md`, `docs/modules/platform/architecture-overview.md`, `docs/implplan/SPRINT_186_scanner_record_mode.md`, `docs/implplan/SPRINT_187_evidence_cli_replay.md`

This playbook enumerates the deterministic replay validation suite. It guides the work tracked under Sprints 186–187 so every guild ships the same baseline before enabling `scan --record`.

---

## 1 · Test matrix

| ID | Scenario | Purpose | Modules | Required Artifacts |
|----|----------|---------|---------|--------------------|
| T-STRICT-001 | **Golden Replay** | Re-run a recorded scan and expect byte-identical outputs. | Scanner.WebService, Scanner.Worker, CLI | `manifest.json`, input/output bundles, DSSE signatures |
| T-FEED-002 | **Feed Drift What-If** | Re-run with updated feeds (`--what-if feeds`) to ensure only feed hashes change. | Scanner.Worker, Concelier, CLI | Feed snapshot bundles, policy bundle, diff report |
| T-TOOL-003 | **Toolchain Upgrade Guard** | Attempt replay with newer scanner binary; expect rejection with `ToolHashMismatch`. | Scanner.Worker, Replay.Core | Tool hash catalog, error log |
| T-POLICY-004 | **Policy Variation Diff** | Re-run with alternate lattice bundle; expect deterministic diff, not failure. | Policy Engine, CLI | Policy bundle(s), diff output |
| T-LEDGER-005 | **Ledger Verification** | Verify Rekor inclusion proof and DSSE signatures offline. | Attestor, Signer, Authority, CLI | DSSE envelopes, Rekor proof, RootPack |
| T-RETENTION-006 | **Retention Sweep** | Ensure Evidence Locker prunes hot CAS after SLA while preserving cold storage copies. | Evidence Locker, Ops | Replay retention config, audit logs |
| T-OFFLINE-007 | **Offline Kit Replay** | Execute `stella replay` using only Offline Kit artifacts. | CLI, Evidence Locker | Offline kit bundle, local RootPack |
| T-OPA-008 | **Runbook Drill** | Simulate replay-driven incident response per `docs/runbooks/replay_ops.md`. | Ops Guild, Scanner, Authority | Runbook checklist, incident notes |

---

## 2 · Execution guidelines

1. **Deterministic environment** — Freeze clock, locale, timezone, and random seed per manifest. See `docs/replay/DETERMINISTIC_REPLAY.md` §4.  
2. **Canonical verification** — Use `StellaOps.Replay.Core` JSON serializer; reject non-canonical payloads before diffing.  
3. **Data sources** — Replay always consumes `replay_runs` + CAS bundles, never live feeds/policies.  
4. **CI integration** —  
   - Scanner repo: add pipeline stage `ReplayStrict` running T-STRICT-001 on fixture images (x64 + arm64).  
   - CLI repo: smoke test `scan --record`, `verify`, `replay`, `diff` using generated fixtures.  
   - Evidence Locker repo: nightly retention test (T-RETENTION-006) with dry-run mode.  
5. **Observability** — Emit metrics `replay_verify_total{result}`, `replay_diff_total{mode}`, `replay_bundle_size_bytes`. Structured logs require `replay.scan_id`, `subject.digest`, `manifest.hash`.

---

## 3 · Fixtures and tooling

- **Fixture catalog** lives under `tools/replay-fixtures/`. Include `README.md` describing update workflow and deterministic compression command.  
- **Generation script** (`./tools/replay-fixtures/build.sh`) orchestrates recording, verifying, and packaging fixtures.  
- **Checksum manifest** (`fixtures/checksums.json`) lists CAS digests and DSSE hashes for quick sanity checks.  
- **CI secrets** must provide offline RootPack and replay signing keys; use sealed secrets in air-gapped pipelines.

---

## 4 · Acceptance checklist

- [ ] All test scenarios executed on x64 and arm64 runners.  
- [ ] Replay verification metrics ingested into Telemetry Stack dashboards.  
- [ ] Evidence Locker retention job validated against hot/cold tiers.  
- [ ] CLI documentation updated with troubleshooting steps observed during tests.  
- [ ] Runbook drill logged with timestamp and owners in `docs/runbooks/replay_ops.md`.

---

*Drafted: 2025-11-03. Update statuses in Sprint 186/187 boards when this checklist is satisfied.*