# Ecosystem Reality Acceptance Test Fixtures

## Module
Scanner

## Status
PARTIALLY_IMPLEMENTED

## Description
Maps five specific real-world production incidents (credential leak, offline DB schema mismatch, SBOM parity drift, scanner instability, ecosystem-specific SCA failure) into deterministic acceptance test fixtures. Each fixture reproduces the incident scenario end-to-end with frozen inputs and expected outputs, ensuring the scanner does not regress on previously observed production failures.

## What's Implemented
- **SCA Failure Catalogue**:
  - `src/Scanner/__Tests/StellaOps.Scanner.Analyzers.Lang.Node.Tests/` - Node.js SCA test fixtures covering package resolution edge cases
  - `src/Scanner/__Tests/StellaOps.Scanner.Analyzers.Lang.Ruby.Tests/` - Ruby SCA test fixtures covering Gemfile.lock parsing
  - `src/Scanner/__Tests/StellaOps.Scanner.Analyzers.Lang.Java.Tests/` - Java SCA test fixtures covering Maven/Gradle dependency resolution
- **SmartDiff Golden Fixtures**:
  - `src/Scanner/__Tests/StellaOps.Scanner.SmartDiff.Tests/` - Deterministic golden fixture tests for diff-aware rescanning, covering expected SBOM delta outputs
- **Determinism Infrastructure**:
  - Existing test projects use frozen fixture files (JSON SBOMs, lock files, manifest files) as inputs with expected output assertions

## What's Missing
- **Incident-to-Test Mapping**: No structured mapping from five specific real-world incidents to dedicated test fixtures:
  1. **Credential Leak Incident**: No fixture reproducing a secret detection false-negative that led to a credential leak in production
  2. **Offline DB Schema Mismatch**: No fixture reproducing scanner behavior when the offline vulnerability database schema version mismatches the scanner version
  3. **SBOM Parity Drift**: No fixture reproducing divergence between container-scanned SBOM and source-scanned SBOM for the same artifact
  4. **Scanner Instability**: No fixture reproducing non-deterministic scanner output across repeated scans of the same image layer
  5. **Ecosystem SCA Failure**: No fixture reproducing ecosystem-specific SCA resolution failures (e.g., npm optional dependency with missing platform binary)
- **Incident Metadata**: No `incident.metadata.json` files linking each fixture to its originating production incident (date, severity, root cause, fix)
- **Acceptance Test Runner**: No dedicated CI job or test category for running ecosystem reality acceptance tests separately from unit tests

## Implementation Plan
1. Create `src/Scanner/__Tests/StellaOps.Scanner.EcosystemReality.Tests/` project with five incident fixture directories
2. For each incident, create: `incident.metadata.json` (date, severity, root cause), frozen input fixtures, expected output assertions
3. Implement credential leak fixture using a container layer with an embedded secret that was previously missed
4. Implement offline DB schema mismatch fixture with mismatched vuln-db schema version headers
5. Implement SBOM parity drift fixture with container vs. source scan inputs producing divergent SBOMs
6. Implement scanner instability fixture verifying byte-identical output across 10 repeated scans
7. Implement ecosystem SCA failure fixture with npm optional dependency edge case
8. Add CI job category `ecosystem-reality` for running these acceptance tests

## E2E Test Plan
- [ ] Run the credential leak incident fixture and verify the scanner now detects the previously-missed embedded secret in the container layer
- [ ] Run the offline DB schema mismatch fixture and verify the scanner produces a clear error or graceful degradation when vuln-db schema version does not match
- [ ] Run the SBOM parity drift fixture and verify the scanner flags divergence between container-scanned and source-scanned SBOMs for the same artifact
- [ ] Run the scanner instability fixture and verify byte-identical SBOM output across 10 repeated scans of the same frozen image layer
- [ ] Run the ecosystem SCA failure fixture and verify correct handling of npm optional dependencies with missing platform binaries
- [ ] Verify each fixture includes incident.metadata.json with date, severity, root cause, and link to the originating production incident

## Related Documentation
- Source: See feature catalog
- Architecture: `docs/modules/scanner/architecture.md`