53 lines
4.2 KiB
Markdown
53 lines
4.2 KiB
Markdown
# Ecosystem Reality Acceptance Test Fixtures
|
|
|
|
## Module
|
|
Scanner
|
|
|
|
## Status
|
|
PARTIALLY_IMPLEMENTED
|
|
|
|
## Description
|
|
Maps five specific real-world production incidents (credential leak, offline DB schema mismatch, SBOM parity drift, scanner instability, ecosystem-specific SCA failure) into deterministic acceptance test fixtures. Each fixture reproduces the incident scenario end-to-end with frozen inputs and expected outputs, ensuring the scanner does not regress on previously observed production failures.
|
|
|
|
## What's Implemented
|
|
- **SCA Failure Catalogue**:
|
|
- `src/Scanner/__Tests/StellaOps.Scanner.Analyzers.Lang.Node.Tests/` - Node.js SCA test fixtures covering package resolution edge cases
|
|
- `src/Scanner/__Tests/StellaOps.Scanner.Analyzers.Lang.Ruby.Tests/` - Ruby SCA test fixtures covering Gemfile.lock parsing
|
|
- `src/Scanner/__Tests/StellaOps.Scanner.Analyzers.Lang.Java.Tests/` - Java SCA test fixtures covering Maven/Gradle dependency resolution
|
|
- **SmartDiff Golden Fixtures**:
|
|
- `src/Scanner/__Tests/StellaOps.Scanner.SmartDiff.Tests/` - Deterministic golden fixture tests for diff-aware rescanning, covering expected SBOM delta outputs
|
|
- **Determinism Infrastructure**:
|
|
- Existing test projects use frozen fixture files (JSON SBOMs, lock files, manifest files) as inputs with expected output assertions
|
|
|
|
## What's Missing
|
|
- **Incident-to-Test Mapping**: No structured mapping from five specific real-world incidents to dedicated test fixtures:
|
|
1. **Credential Leak Incident**: No fixture reproducing a secret detection false-negative that led to a credential leak in production
|
|
2. **Offline DB Schema Mismatch**: No fixture reproducing scanner behavior when the offline vulnerability database schema version mismatches the scanner version
|
|
3. **SBOM Parity Drift**: No fixture reproducing divergence between container-scanned SBOM and source-scanned SBOM for the same artifact
|
|
4. **Scanner Instability**: No fixture reproducing non-deterministic scanner output across repeated scans of the same image layer
|
|
5. **Ecosystem SCA Failure**: No fixture reproducing ecosystem-specific SCA resolution failures (e.g., npm optional dependency with missing platform binary)
|
|
- **Incident Metadata**: No `incident.metadata.json` files linking each fixture to its originating production incident (date, severity, root cause, fix)
|
|
- **Acceptance Test Runner**: No dedicated CI job or test category for running ecosystem reality acceptance tests separately from unit tests
|
|
|
|
## Implementation Plan
|
|
1. Create `src/Scanner/__Tests/StellaOps.Scanner.EcosystemReality.Tests/` project with five incident fixture directories
|
|
2. For each incident, create: `incident.metadata.json` (date, severity, root cause), frozen input fixtures, expected output assertions
|
|
3. Implement credential leak fixture using a container layer with an embedded secret that was previously missed
|
|
4. Implement offline DB schema mismatch fixture with mismatched vuln-db schema version headers
|
|
5. Implement SBOM parity drift fixture with container vs. source scan inputs producing divergent SBOMs
|
|
6. Implement scanner instability fixture verifying byte-identical output across 10 repeated scans
|
|
7. Implement ecosystem SCA failure fixture with npm optional dependency edge case
|
|
8. Add CI job category `ecosystem-reality` for running these acceptance tests
|
|
|
|
## E2E Test Plan
|
|
- [ ] Run the credential leak incident fixture and verify the scanner now detects the previously-missed embedded secret in the container layer
|
|
- [ ] Run the offline DB schema mismatch fixture and verify the scanner produces a clear error or graceful degradation when vuln-db schema version does not match
|
|
- [ ] Run the SBOM parity drift fixture and verify the scanner flags divergence between container-scanned and source-scanned SBOMs for the same artifact
|
|
- [ ] Run the scanner instability fixture and verify byte-identical SBOM output across 10 repeated scans of the same frozen image layer
|
|
- [ ] Run the ecosystem SCA failure fixture and verify correct handling of npm optional dependencies with missing platform binaries
|
|
- [ ] Verify each fixture includes incident.metadata.json with date, severity, root cause, and link to the originating production incident
|
|
|
|
## Related Documentation
|
|
- Source: See feature catalog
|
|
- Architecture: `docs/modules/scanner/architecture.md`
|