git.stella-ops.org/docs/features/dropped/ecosystem-reality-acceptance-test-fixtures.md

Ecosystem Reality Acceptance Test Fixtures

Module

Scanner

Status

PARTIALLY_IMPLEMENTED

Description

Maps five specific real-world production incidents (credential leak, offline DB schema mismatch, SBOM parity drift, scanner instability, ecosystem-specific SCA failure) into deterministic acceptance test fixtures. Each fixture reproduces the incident scenario end-to-end with frozen inputs and expected outputs, ensuring the scanner does not regress on previously observed production failures.

What's Implemented

• SCA Failure Catalogue:
  • src/Scanner/__Tests/StellaOps.Scanner.Analyzers.Lang.Node.Tests/ - Node.js SCA test fixtures covering package resolution edge cases
  • src/Scanner/__Tests/StellaOps.Scanner.Analyzers.Lang.Ruby.Tests/ - Ruby SCA test fixtures covering Gemfile.lock parsing
  • src/Scanner/__Tests/StellaOps.Scanner.Analyzers.Lang.Java.Tests/ - Java SCA test fixtures covering Maven/Gradle dependency resolution
• SmartDiff Golden Fixtures:
  • src/Scanner/__Tests/StellaOps.Scanner.SmartDiff.Tests/ - Deterministic golden fixture tests for diff-aware rescanning, covering expected SBOM delta outputs
• Determinism Infrastructure:
  • Existing test projects use frozen fixture files (JSON SBOMs, lock files, manifest files) as inputs with expected output assertions

What's Missing

• Incident-to-Test Mapping: No structured mapping from five specific real-world incidents to dedicated test fixtures:
  1. Credential Leak Incident: No fixture reproducing a secret detection false-negative that led to a credential leak in production
  2. Offline DB Schema Mismatch: No fixture reproducing scanner behavior when the offline vulnerability database schema version mismatches the scanner version
  3. SBOM Parity Drift: No fixture reproducing divergence between container-scanned SBOM and source-scanned SBOM for the same artifact
  4. Scanner Instability: No fixture reproducing non-deterministic scanner output across repeated scans of the same image layer
  5. Ecosystem SCA Failure: No fixture reproducing ecosystem-specific SCA resolution failures (e.g., npm optional dependency with missing platform binary)
• Incident Metadata: No incident.metadata.json files linking each fixture to its originating production incident (date, severity, root cause, fix)
• Acceptance Test Runner: No dedicated CI job or test category for running ecosystem reality acceptance tests separately from unit tests

Implementation Plan

1. Create src/Scanner/__Tests/StellaOps.Scanner.EcosystemReality.Tests/ project with five incident fixture directories
2. For each incident, create: incident.metadata.json (date, severity, root cause), frozen input fixtures, expected output assertions
3. Implement credential leak fixture using a container layer with an embedded secret that was previously missed
4. Implement offline DB schema mismatch fixture with mismatched vuln-db schema version headers
5. Implement SBOM parity drift fixture with container vs. source scan inputs producing divergent SBOMs
6. Implement scanner instability fixture verifying byte-identical output across 10 repeated scans
7. Implement ecosystem SCA failure fixture with npm optional dependency edge case
8. Add CI job category ecosystem-reality for running these acceptance tests

E2E Test Plan

• Run the credential leak incident fixture and verify the scanner now detects the previously-missed embedded secret in the container layer
• Run the offline DB schema mismatch fixture and verify the scanner produces a clear error or graceful degradation when vuln-db schema version does not match
• Run the SBOM parity drift fixture and verify the scanner flags divergence between container-scanned and source-scanned SBOMs for the same artifact
• Run the scanner instability fixture and verify byte-identical SBOM output across 10 repeated scans of the same frozen image layer
• Run the ecosystem SCA failure fixture and verify correct handling of npm optional dependencies with missing platform binaries
• Verify each fixture includes incident.metadata.json with date, severity, root cause, and link to the originating production incident

Related Documentation

• Source: See feature catalog
• Architecture: docs/modules/scanner/architecture.md