# Scanner Deterministic Regression Test Framework

## Module
Scanner

## Status
PARTIALLY_IMPLEMENTED

## Description
A structured regression test framework with standardized case layout, golden fixture comparison, and dedicated CI job. Each regression case is identified by `SCN-XXXX-slug`, contains frozen inputs and expected outputs, and uses byte-level comparison to detect scanner output drift.

## What's Implemented
- **Existing Determinism Tests**:
  - `src/Scanner/__Tests/StellaOps.Scanner.SmartDiff.Tests/` - Golden fixture tests for SmartDiff comparing actual vs. expected SBOM deltas with frozen inputs
  - `src/Scanner/__Tests/StellaOps.Scanner.Analyzers.Lang.Node.Tests/` - Deterministic language analyzer tests with frozen package.json/lock files
  - `src/Scanner/__Tests/StellaOps.Scanner.Analyzers.Lang.Ruby.Tests/` - Deterministic Ruby analyzer tests with frozen Gemfile.lock fixtures
  - `src/Scanner/__Tests/StellaOps.Scanner.Analyzers.Lang.Java.Tests/` - Deterministic Java analyzer tests with frozen pom.xml/build.gradle fixtures
- **Reachability Tests**:
  - `src/Scanner/__Tests/StellaOps.Scanner.Reachability.Tests/` - Reachability analysis tests with frozen call-graph fixtures and expected classification outputs
- **Test Infrastructure**:
  - Existing test projects demonstrate the golden fixture pattern (frozen input -> run analyzer -> compare against expected output) but each project uses its own ad-hoc fixture layout

## What's Missing
- **Standardized Case Layout**: No `Regression/` directory with `SCN-XXXX-slug/` subdirectories containing:
  - `case.metadata.json` (case ID, description, scanner version that introduced the regression, severity)
  - `case.md` (human-readable regression description with root cause analysis)
  - `input/` (frozen input fixtures: container layers, SBOMs, lock files)
  - `expected/` (expected output fixtures: SBOMs, reachability results, verdict payloads)
- **Regression Test Runner**: No unified test runner that discovers all `SCN-XXXX-slug/` cases, runs each through the scanner pipeline, and performs byte-level output comparison
- **Dedicated CI Job**: No `scanner-regression` CI job that runs regression tests separately from unit tests with clear pass/fail reporting per case
- **Regression Case Generator**: No tooling to capture a failing scanner scenario and automatically generate a new `SCN-XXXX-slug/` case from it
- **Drift Detection**: No tooling to detect when scanner output changes (intentionally or unintentionally) and prompt for expected-output updates with review

## Implementation Plan
1. Create `src/Scanner/__Tests/StellaOps.Scanner.Regression.Tests/` project with case discovery infrastructure
2. Define `case.metadata.json` schema with fields: caseId, slug, description, introducedInVersion, severity, tags
3. Create initial regression cases from existing golden fixture tests (migrate 5-10 representative cases)
4. Implement `RegressionTestRunner` that discovers cases, runs scanner pipeline on inputs, compares outputs byte-by-byte
5. Add `case-capture` CLI tool that takes a scanner invocation and generates a new case directory with frozen inputs and current outputs
6. Add `scanner-regression` CI job in `.gitea/workflows/` that runs regression tests and reports per-case pass/fail
7. Add drift detection that generates a diff report when expected output changes

## E2E Test Plan
- [ ] Run the regression test runner and verify all `SCN-XXXX-slug/` cases produce output that byte-matches their `expected/` fixtures
- [ ] Add a new regression case using the case-capture tool and verify it is automatically discovered by the test runner on the next run
- [ ] Introduce an intentional scanner change that modifies output for one case and verify the regression test runner detects the drift and fails the case
- [ ] Update the expected output for the changed case and verify the test runner passes again
- [ ] Verify `case.metadata.json` is validated on test startup (missing required fields cause a clear error)
- [ ] Verify the CI job produces a per-case pass/fail report with case ID, slug, and failure diff for any failing cases
- [ ] Verify regression tests run in under 5 minutes for the initial 10-case corpus

## Related Documentation
- Source: See feature catalog
- Architecture: `docs/modules/scanner/architecture.md`