git.stella-ops.org/src/__Tests/e2e/ReplayableVerdict

E2E Replayable Verdict Tests

Sprint: SPRINT_20251229_004_005_E2E

Overview

End-to-end tests validating the complete reproducible verdict pipeline:

Image → Scanner → Feedser → VexLens → Verdict Builder → DSSE Signing → UI Delta View

With capture of artifacts bundle enabling byte-for-byte replay.

Test Structure

Golden Bundles

Located in src/__Tests/fixtures/e2e/bundle-XXXX/:

bundle-0001/
├── manifest.json              # ReplayManifest v2
├── inputs/
│   ├── sbom.cdx.json          # Canonical SBOM
│   ├── feeds/
│   │   └── osv-snapshot.json  # Pinned feed subset
│   ├── vex/
│   │   └── vendor.openvex.json
│   └── policy/
│       └── rules.yaml
├── outputs/
│   ├── verdict.json           # Expected verdict
│   └── verdict.dsse.json      # DSSE envelope (when signing enabled)
├── attestation/
│   ├── test-keypair.pem       # Test signing key
│   └── public-key.pem
└── meta.json                  # Bundle metadata

Test Categories

Test	Status	Purpose
E2E-001	✅ DONE	Golden bundle creation and loading
E2E-002	⏳ SKIPPED	Full pipeline test (requires service integration)
E2E-003	⏳ SKIPPED	Replay verification test
E2E-004	⏳ SKIPPED	Delta verdict test
E2E-005	⏳ SKIPPED	DSSE signature verification
E2E-006	⏳ SKIPPED	Offline/air-gap replay test
E2E-007	✅ DONE	CLI stella verify --bundle command
E2E-008	⏳ SKIPPED	Cross-platform replay test

Running Tests

All E2E Tests

dotnet test src/__Tests/E2E/ReplayableVerdict/ \
  --filter "Category=E2E"

Determinism Tests Only

dotnet test src/__Tests/E2E/ReplayableVerdict/ \
  --filter "Category=Determinism"

Individual Test

dotnet test src/__Tests/E2E/ReplayableVerdict/ \
  --filter "FullyQualifiedName~Bundle_LoadsSuccessfully"

Current Status

✅ Completed

• Golden bundle structure created (bundle-0001)
• Minimal test inputs:
  • SBOM: Alpine 3.19 with 5 packages
  • Feeds: 2 synthetic OSV advisories
  • VEX: 1 OpenVEX statement
  • Policy: Basic rules with scoring
• Manifest schema (ReplayManifest v2)
• Bundle loader implementation
• 8 test cases defined (3 passing, 5 skipped pending integration)

⏳ Pending Integration

The following tests are skipped pending service integration:

1. Full Pipeline Test (E2E-002)

   • Requires: Scanner, VexLens, VerdictBuilder services
   • Blocks: End-to-end verdict generation

2. Replay Verification (E2E-003)

   • Requires: VerdictBuilder.ReplayAsync()
   • Blocks: Determinism validation

3. Delta Verdict (E2E-004)

   • Requires: VerdictBuilder.DiffAsync() + bundle-0002
   • Blocks: Version comparison testing

4. DSSE Signing (E2E-005)

   • Requires: Signer service integration
   • Blocks: Attestation verification

5. Offline Replay (E2E-006)

   • Requires: Network isolation test infrastructure
   • Blocks: Air-gap validation

6. Cross-Platform (E2E-008)

   • Requires: Multi-platform CI runners
   • Blocks: Platform-independent determinism

✅ Completed: E2E-007 CLI Verify Command

Implementation:

The stella verify bundle command has been implemented in:

• src/Cli/StellaOps.Cli/Commands/CommandHandlers.VerifyBundle.cs (handler)
• src/Cli/StellaOps.Cli/Commands/VerifyCommandGroup.cs (command registration)
• src/Cli/__Tests/StellaOps.Cli.Tests/Commands/VerifyBundleCommandTests.cs (tests)

Usage:

stella verify bundle --bundle <path-to-bundle-directory>
stella verify bundle --bundle <path-to-bundle.tar.gz>  # Not yet supported
stella verify bundle --bundle ./bundle-0001 --skip-replay  # Skip verdict replay
stella verify bundle --bundle ./bundle-0001 --output json  # JSON output

Features:

1. ✅ Loads bundle manifest
2. ✅ Validates input hashes (SBOM, feeds, VEX, policy)
3. ⏳ Replays verdict (stubbed - requires VerdictBuilder integration)
4. ✅ Compares to expected hash
5. ⏳ Verifies DSSE signature (stubbed - requires Signer integration)
6. ✅ Outputs PASS/FAIL with violations

Exit Codes:

• 0 - PASS: All validations passed
• 7 - File not found (bundle or manifest)
• 8 - FAIL: Validation violations detected
• 9 - Not implemented (tar.gz extraction)

Integration Roadmap

Phase 1: Service Integration (Week 1-2)

1. Integrate Scanner service
2. Integrate VexLens consensus
3. Integrate VerdictBuilder
4. Enable E2E-002 (Full Pipeline)

Phase 2: Replay Functionality (Week 3)

1. Implement VerdictBuilder.ReplayAsync()
2. Enable E2E-003 (Replay Verification)
3. Create bundle-0002 for delta testing
4. Enable E2E-004 (Delta Verdict)

Phase 3: Signing & Attestation (Week 4)

1. Integrate Signer service
2. Generate test keypair
3. Enable E2E-005 (DSSE Signing)
4. Implement CLI verify command (E2E-007)

Phase 4: Advanced Validation (Week 5)

1. Setup network isolation for E2E-006
2. Configure multi-platform CI for E2E-008
3. Add performance benchmarks
4. Add chaos testing variants

Bundle Management

Creating a New Bundle

# 1. Create bundle directory
mkdir -p src/__Tests/fixtures/e2e/bundle-XXXX

# 2. Use Fixture Harvester
cd src/__Tests/Tools/FixtureHarvester
dotnet run harvest --type e2e --id bundle-XXXX

# 3. Add inputs (SBOM, feeds, VEX, policy)
# Place files in bundle-XXXX/inputs/

# 4. Run pipeline to generate outputs
stella scan --record --bundle bundle-XXXX

# 5. Compute hashes and update manifest
dotnet run validate --bundle bundle-XXXX

# 6. Freeze bundle (commit to git)
git add src/__Tests/fixtures/e2e/bundle-XXXX
git commit -m "Add E2E bundle: bundle-XXXX"

Validating Bundles

# Validate all bundles
dotnet run --project src/__Tests/Tools/FixtureHarvester \
  validate --path src/__Tests/fixtures/e2e

# CI validation
.gitea/workflows/e2e-replay.yml

Determinism Guarantees

Input Stability

• All inputs pinned with SHA-256 hashes
• Feed snapshots frozen at capture time
• Policy files versioned
• SBOM canonical format (sorted, normalized)

Output Reproducibility

• Verdict hash computed from canonical JSON
• UTC timestamps in ISO-8601 format
• Stable sorting (CVEs, packages, findings)
• No system-specific paths or UUIDs

Cross-Platform Compatibility

• Tests run on: Ubuntu 22.04, Alpine 3.19, Debian Bookworm
• Verdict hash must match across all platforms
• File path normalization (forward slashes)
• Line ending normalization (LF only)

Troubleshooting

Bundle Load Failures

# Check manifest syntax
cat src/__Tests/fixtures/e2e/bundle-0001/manifest.json | jq .

# Verify file paths
ls src/__Tests/fixtures/e2e/bundle-0001/inputs/

# Validate hashes
sha256sum src/__Tests/fixtures/e2e/bundle-0001/inputs/sbom.cdx.json

Hash Mismatches

# Recompute hashes
dotnet run --project src/__Tests/Tools/FixtureHarvester \
  validate --path src/__Tests/fixtures/e2e

# Compare expected vs actual
diff -u expected.json actual.json | jq .

Skipped Tests

Skipped tests indicate missing service integration. Follow integration roadmap to enable.

See Also

• Fixture Harvester
• Determinism Guide
• Replay Architecture
• Verdict API