stella-ops.org/git.stella-ops.org
1
0
Fork 0
Code Issues Pull Requests Actions 16 Releases Wiki Activity
Files
1c782897f762f04b9ed69aed534a03dbe2893039
git.stella-ops.org/docs/replay/TEST_STRATEGY.md
master ae69b1a8a1 feat: Add documentation and task tracking for Sprints 508 to 514 in Ops & Offline 
- Created detailed markdown files for Sprints 508 (Ops Offline Kit), 509 (Samples), 510 (AirGap), 511 (Api), 512 (Bench), 513 (Provenance), and 514 (Sovereign Crypto Enablement) outlining tasks, dependencies, and owners.
- Introduced a comprehensive Reachability Evidence Delivery Guide to streamline the reachability signal process.
- Implemented unit tests for Advisory AI to block known injection patterns and redact secrets.
- Added AuthoritySenderConstraintHelper to manage sender constraints in OpenIddict transactions.
2025-11-08 23:18:28 +02:00

4.4 KiB
Raw Blame History

Replay Test Strategy (Draft)

Ownership: Docs Guild · Scanner Guild · Evidence Locker Guild · QA Guild
Related: docs/replay/DETERMINISTIC_REPLAY.md, docs/replay/DEVS_GUIDE_REPLAY.md, docs/modules/platform/architecture-overview.md, docs/implplan/SPRINT_186_record_deterministic_execution.md, docs/implplan/SPRINT_187_evidence_locker_cli_integration.md

This playbook enumerates the deterministic replay validation suite. It guides the work tracked under Sprints 186–187 so every guild ships the same baseline before enabling scan --record.

1 · Test matrix

ID Scenario Purpose Modules Required Artifacts
T-STRICT-001 Golden Replay Re-run a recorded scan and expect byte-identical outputs. Scanner.WebService, Scanner.Worker, CLI manifest.json, input/output bundles, DSSE signatures
T-FEED-002 Feed Drift What-If Re-run with updated feeds (--what-if feeds) to ensure only feed hashes change. Scanner.Worker, Concelier, CLI Feed snapshot bundles, policy bundle, diff report
T-TOOL-003 Toolchain Upgrade Guard Attempt replay with newer scanner binary; expect rejection with ToolHashMismatch. Scanner.Worker, Replay.Core Tool hash catalog, error log
T-POLICY-004 Policy Variation Diff Re-run with alternate lattice bundle; expect deterministic diff, not failure. Policy Engine, CLI Policy bundle(s), diff output
T-LEDGER-005 Ledger Verification Verify Rekor inclusion proof and DSSE signatures offline. Attestor, Signer, Authority, CLI DSSE envelopes, Rekor proof, RootPack
T-RETENTION-006 Retention Sweep Ensure Evidence Locker prunes hot CAS after SLA while preserving cold storage copies. Evidence Locker, Ops Replay retention config, audit logs
T-OFFLINE-007 Offline Kit Replay Execute stella replay using only Offline Kit artifacts. CLI, Evidence Locker Offline kit bundle, local RootPack
T-OPA-008 Runbook Drill Simulate replay-driven incident response per docs/runbooks/replay_ops.md. Ops Guild, Scanner, Authority Runbook checklist, incident notes
T-REACH-009 Reachability Replay Rehydrate reachability graphs/traces from replay bundles and compare against reachbench fixtures. Scanner, Signals, Replay reachbench-2025-expanded, reachability CAS references

2 · Execution guidelines

  1. Deterministic environment — Freeze clock, locale, timezone, and random seed per manifest. See docs/replay/DETERMINISTIC_REPLAY.md §4.
  2. Canonical verification — Use StellaOps.Replay.Core JSON serializer; reject non-canonical payloads before diffing.
  3. Data sources — Replay always consumes replay_runs + CAS bundles, never live feeds/policies.
  4. CI integration —
    • Scanner repo: add pipeline stage ReplayStrict running T-STRICT-001 on fixture images (x64 + arm64).
    • CLI repo: smoke test scan --record, verify, replay, diff using generated fixtures.
    • Evidence Locker repo: nightly retention test (T-RETENTION-006) with dry-run mode.
  5. Observability — Emit metrics replay_verify_total{result}, replay_diff_total{mode}, replay_bundle_size_bytes. Structured logs require replay.scan_id, subject.digest, manifest.hash.

3 · Fixtures and tooling

  • Fixture catalog lives under tools/replay-fixtures/. Include README.md describing update workflow and deterministic compression command.
  • Generation script (./tools/replay-fixtures/build.sh) orchestrates recording, verifying, and packaging fixtures.
  • Checksum manifest (fixtures/checksums.json) lists CAS digests and DSSE hashes for quick sanity checks.
  • CI secrets must provide offline RootPack and replay signing keys; use sealed secrets in air-gapped pipelines.

4 · Acceptance checklist

  • All test scenarios executed on x64 and arm64 runners.
  • Replay verification metrics ingested into Telemetry Stack dashboards.
  • Evidence Locker retention job validated against hot/cold tiers.
  • CLI documentation updated with troubleshooting steps observed during tests.
  • Runbook drill logged with timestamp and owners in docs/runbooks/replay_ops.md.
  • Reachability replay drill captured (T-REACH-009) with fixture references and Signals verification logs.

Drafted: 2025-11-03. Update statuses in Sprint 186/187 boards when this checklist is satisfied.