git.stella-ops.org/bench/golden-corpus

Golden Test Corpus

This directory contains the golden test corpus for StellaOps scoring validation. Each test case is a complete, reproducible scenario with known-good inputs and expected outputs.

Schema Version

Corpus Version: 1.0.0 Scoring Algorithm: v2.0 (See docs/modules/scanner/scoring-algorithm.md) OpenVEX Schema: 0.2.0 SPDX Version: 3.0.1 CycloneDX Version: 1.6

Directory Structure

golden-corpus/
├── README.md                    # This file
├── corpus-manifest.json         # Index of all test cases with hashes
├── corpus-version.json          # Versioning metadata
│
├── severity-levels/             # CVE severity coverage
│   ├── critical/
│   ├── high/
│   ├── medium/
│   └── low/
│
├── vex-scenarios/               # VEX override scenarios
│   ├── not-affected/
│   ├── affected/
│   ├── fixed/
│   └── under-investigation/
│
├── reachability/                # Reachability analysis scenarios
│   ├── reachable/
│   ├── unreachable/
│   └── unknown/
│
└── composite/                   # Complex multi-factor scenarios
    ├── reachable-with-vex/
    └── unreachable-high-severity/

Test Case Format

Each test case directory contains:

File	Description
case.json	Scenario metadata and description
sbom.spdx.json	SPDX 3.0.1 SBOM
sbom.cdx.json	CycloneDX 1.6 SBOM (optional)
manifest.json	Scan manifest with digest bindings
vex.openvex.json	OpenVEX document (if applicable)
callgraph.json	Static call graph (if reachability applies)
proof-bundle.json	Expected proof bundle structure
expected-score.json	Expected scoring output

Expected Score Format

{
  "schema_version": "stellaops.golden.expected/v1",
  "score_hash": "sha256:...",
  "stella_score": 7.5,
  "base_cvss": 9.8,
  "temporal_cvss": 8.5,
  "environmental_cvss": 7.5,
  "vex_impact": -1.0,
  "reachability_impact": -1.3,
  "kev_flag": false,
  "exploit_maturity": "proof-of-concept",
  "determinism_salt": "frozen-2025-01-15T00:00:00Z"
}

Running Golden Tests

# Run all golden tests
dotnet test tests/integration/StellaOps.Integration.Determinism \
  --filter "Category=GoldenCorpus"

# Regenerate expected outputs (after algorithm changes)
dotnet run --project bench/tools/corpus-regenerate -- \
  --corpus-path bench/golden-corpus \
  --algorithm-version v2.0

Adding New Cases

1. Create directory under appropriate category
2. Add all required files (see Test Case Format)
3. Run corpus validation: dotnet run --project bench/tools/corpus-validate
4. Update corpus-manifest.json hash entries
5. Commit with message: corpus: add <case-id> for <scenario>

Versioning Policy

• Patch (1.0.x): Add new cases, fix existing case data
• Minor (1.x.0): Algorithm tuning that preserves relative ordering
• Major (x.0.0): Algorithm changes that alter expected scores

When scoring algorithm changes:

1. Increment corpus version
2. Regenerate all expected scores
3. Document changes in CHANGELOG.md