Files

Docs CI / lint-and-preview (push) Has been cancelled

Details

Add unit tests for SBOM ingestion and transformation

- Implement `SbomIngestServiceCollectionExtensionsTests` to verify the SBOM ingestion pipeline exports snapshots correctly.
- Create `SbomIngestTransformerTests` to ensure the transformation produces expected nodes and edges, including deduplication of license nodes and normalization of timestamps.
- Add `SbomSnapshotExporterTests` to test the export functionality for manifest, adjacency, nodes, and edges.
- Introduce `VexOverlayTransformerTests` to validate the transformation of VEX nodes and edges.
- Set up project file for the test project with necessary dependencies and configurations.
- Include JSON fixture files for testing purposes.

2025-11-04 07:49:39 +02:00

4.1 KiB

Raw Blame History

Replay Test Strategy (Draft)

Ownership: Docs Guild · Scanner Guild · Evidence Locker Guild · QA Guild
Related: docs/replay/DETERMINISTIC_REPLAY.md, docs/replay/DEVS_GUIDE_REPLAY.md, docs/modules/platform/architecture-overview.md, docs/implplan/SPRINT_186_scanner_record_mode.md, docs/implplan/SPRINT_187_evidence_cli_replay.md

This playbook enumerates the deterministic replay validation suite. It guides the work tracked under Sprints 186–187 so every guild ships the same baseline before enabling scan --record.

1 · Test matrix

ID	Scenario	Purpose	Modules	Required Artifacts
T-STRICT-001	Golden Replay	Re-run a recorded scan and expect byte-identical outputs.	Scanner.WebService, Scanner.Worker, CLI	`manifest.json`, input/output bundles, DSSE signatures
T-FEED-002	Feed Drift What-If	Re-run with updated feeds (`--what-if feeds`) to ensure only feed hashes change.	Scanner.Worker, Concelier, CLI	Feed snapshot bundles, policy bundle, diff report
T-TOOL-003	Toolchain Upgrade Guard	Attempt replay with newer scanner binary; expect rejection with `ToolHashMismatch`.	Scanner.Worker, Replay.Core	Tool hash catalog, error log
T-POLICY-004	Policy Variation Diff	Re-run with alternate lattice bundle; expect deterministic diff, not failure.	Policy Engine, CLI	Policy bundle(s), diff output
T-LEDGER-005	Ledger Verification	Verify Rekor inclusion proof and DSSE signatures offline.	Attestor, Signer, Authority, CLI	DSSE envelopes, Rekor proof, RootPack
T-RETENTION-006	Retention Sweep	Ensure Evidence Locker prunes hot CAS after SLA while preserving cold storage copies.	Evidence Locker, Ops	Replay retention config, audit logs
T-OFFLINE-007	Offline Kit Replay	Execute `stella replay` using only Offline Kit artifacts.	CLI, Evidence Locker	Offline kit bundle, local RootPack
T-OPA-008	Runbook Drill	Simulate replay-driven incident response per `docs/runbooks/replay_ops.md`.	Ops Guild, Scanner, Authority	Runbook checklist, incident notes

2 · Execution guidelines

Deterministic environment — Freeze clock, locale, timezone, and random seed per manifest. See docs/replay/DETERMINISTIC_REPLAY.md §4.
Canonical verification — Use StellaOps.Replay.Core JSON serializer; reject non-canonical payloads before diffing.
Data sources — Replay always consumes replay_runs + CAS bundles, never live feeds/policies.
CI integration —
- Scanner repo: add pipeline stage ReplayStrict running T-STRICT-001 on fixture images (x64 + arm64).
- CLI repo: smoke test scan --record, verify, replay, diff using generated fixtures.
- Evidence Locker repo: nightly retention test (T-RETENTION-006) with dry-run mode.
Observability — Emit metrics replay_verify_total{result}, replay_diff_total{mode}, replay_bundle_size_bytes. Structured logs require replay.scan_id, subject.digest, manifest.hash.

3 · Fixtures and tooling

Fixture catalog lives under tools/replay-fixtures/. Include README.md describing update workflow and deterministic compression command.
Generation script (./tools/replay-fixtures/build.sh) orchestrates recording, verifying, and packaging fixtures.
Checksum manifest (fixtures/checksums.json) lists CAS digests and DSSE hashes for quick sanity checks.
CI secrets must provide offline RootPack and replay signing keys; use sealed secrets in air-gapped pipelines.

4 · Acceptance checklist

All test scenarios executed on x64 and arm64 runners.
Replay verification metrics ingested into Telemetry Stack dashboards.
Evidence Locker retention job validated against hot/cold tiers.
CLI documentation updated with troubleshooting steps observed during tests.
Runbook drill logged with timestamp and owners in docs/runbooks/replay_ops.md.

Drafted: 2025-11-03. Update statuses in Sprint 186/187 boards when this checklist is satisfied.

4.1 KiB Raw Blame History Unescape Escape