ae69b1a8a1
- Created detailed markdown files for Sprints 508 (Ops Offline Kit), 509 (Samples), 510 (AirGap), 511 (Api), 512 (Bench), 513 (Provenance), and 514 (Sovereign Crypto Enablement) outlining tasks, dependencies, and owners. - Introduced a comprehensive Reachability Evidence Delivery Guide to streamline the reachability signal process. - Implemented unit tests for Advisory AI to block known injection patterns and redact secrets. - Added AuthoritySenderConstraintHelper to manage sender constraints in OpenIddict transactions.
60 lines
4.4 KiB
Markdown
60 lines
4.4 KiB
Markdown
# Replay Test Strategy (Draft)
|
||
|
||
> **Ownership:** Docs Guild · Scanner Guild · Evidence Locker Guild · QA Guild
|
||
> **Related:** `docs/replay/DETERMINISTIC_REPLAY.md`, `docs/replay/DEVS_GUIDE_REPLAY.md`, `docs/modules/platform/architecture-overview.md`, `docs/implplan/SPRINT_186_record_deterministic_execution.md`, `docs/implplan/SPRINT_187_evidence_locker_cli_integration.md`
|
||
|
||
This playbook enumerates the deterministic replay validation suite. It guides the work tracked under Sprints 186–187 so every guild ships the same baseline before enabling `scan --record`.
|
||
|
||
---
|
||
|
||
## 1 · Test matrix
|
||
|
||
| ID | Scenario | Purpose | Modules | Required Artifacts |
|
||
|----|----------|---------|---------|--------------------|
|
||
| T-STRICT-001 | **Golden Replay** | Re-run a recorded scan and expect byte-identical outputs. | Scanner.WebService, Scanner.Worker, CLI | `manifest.json`, input/output bundles, DSSE signatures |
|
||
| T-FEED-002 | **Feed Drift What-If** | Re-run with updated feeds (`--what-if feeds`) to ensure only feed hashes change. | Scanner.Worker, Concelier, CLI | Feed snapshot bundles, policy bundle, diff report |
|
||
| T-TOOL-003 | **Toolchain Upgrade Guard** | Attempt replay with newer scanner binary; expect rejection with `ToolHashMismatch`. | Scanner.Worker, Replay.Core | Tool hash catalog, error log |
|
||
| T-POLICY-004 | **Policy Variation Diff** | Re-run with alternate lattice bundle; expect deterministic diff, not failure. | Policy Engine, CLI | Policy bundle(s), diff output |
|
||
| T-LEDGER-005 | **Ledger Verification** | Verify Rekor inclusion proof and DSSE signatures offline. | Attestor, Signer, Authority, CLI | DSSE envelopes, Rekor proof, RootPack |
|
||
| T-RETENTION-006 | **Retention Sweep** | Ensure Evidence Locker prunes hot CAS after SLA while preserving cold storage copies. | Evidence Locker, Ops | Replay retention config, audit logs |
|
||
| T-OFFLINE-007 | **Offline Kit Replay** | Execute `stella replay` using only Offline Kit artifacts. | CLI, Evidence Locker | Offline kit bundle, local RootPack |
|
||
| T-OPA-008 | **Runbook Drill** | Simulate replay-driven incident response per `docs/runbooks/replay_ops.md`. | Ops Guild, Scanner, Authority | Runbook checklist, incident notes |
|
||
| T-REACH-009 | **Reachability Replay** | Rehydrate reachability graphs/traces from replay bundles and compare against reachbench fixtures. | Scanner, Signals, Replay | `reachbench-2025-expanded`, reachability CAS references |
|
||
|
||
---
|
||
|
||
## 2 · Execution guidelines
|
||
|
||
1. **Deterministic environment** — Freeze clock, locale, timezone, and random seed per manifest. See `docs/replay/DETERMINISTIC_REPLAY.md` §4.
|
||
2. **Canonical verification** — Use `StellaOps.Replay.Core` JSON serializer; reject non-canonical payloads before diffing.
|
||
3. **Data sources** — Replay always consumes `replay_runs` + CAS bundles, never live feeds/policies.
|
||
4. **CI integration** —
|
||
- Scanner repo: add pipeline stage `ReplayStrict` running T-STRICT-001 on fixture images (x64 + arm64).
|
||
- CLI repo: smoke test `scan --record`, `verify`, `replay`, `diff` using generated fixtures.
|
||
- Evidence Locker repo: nightly retention test (T-RETENTION-006) with dry-run mode.
|
||
5. **Observability** — Emit metrics `replay_verify_total{result}`, `replay_diff_total{mode}`, `replay_bundle_size_bytes`. Structured logs require `replay.scan_id`, `subject.digest`, `manifest.hash`.
|
||
|
||
---
|
||
|
||
## 3 · Fixtures and tooling
|
||
|
||
- **Fixture catalog** lives under `tools/replay-fixtures/`. Include `README.md` describing update workflow and deterministic compression command.
|
||
- **Generation script** (`./tools/replay-fixtures/build.sh`) orchestrates recording, verifying, and packaging fixtures.
|
||
- **Checksum manifest** (`fixtures/checksums.json`) lists CAS digests and DSSE hashes for quick sanity checks.
|
||
- **CI secrets** must provide offline RootPack and replay signing keys; use sealed secrets in air-gapped pipelines.
|
||
|
||
---
|
||
|
||
## 4 · Acceptance checklist
|
||
|
||
- [ ] All test scenarios executed on x64 and arm64 runners.
|
||
- [ ] Replay verification metrics ingested into Telemetry Stack dashboards.
|
||
- [ ] Evidence Locker retention job validated against hot/cold tiers.
|
||
- [ ] CLI documentation updated with troubleshooting steps observed during tests.
|
||
- [ ] Runbook drill logged with timestamp and owners in `docs/runbooks/replay_ops.md`.
|
||
- [ ] Reachability replay drill captured (`T-REACH-009`) with fixture references and Signals verification logs.
|
||
|
||
---
|
||
|
||
*Drafted: 2025-11-03. Update statuses in Sprint 186/187 boards when this checklist is satisfied.*
|