git.stella-ops.org/docs/modules/vex-lens/testing-strategy.md

VexLens Testing Strategy

Overview

This document describes the testing strategy for VexLens module, covering determinism verification, regression testing, and golden corpus validation.

Test Categories

1. Unit Tests (Category=Unit)

Standard unit tests for individual components:

• Location: src/VexLens/StellaOps.VexLens/__Tests/StellaOps.VexLens.Tests/
• Coverage: Models, builders, serializers, individual consensus components
• Run frequency: Every PR, every commit

2. Determinism Tests (Category=Determinism)

Tests that verify VexLens produces identical outputs for identical inputs:

• Location: src/VexLens/StellaOps.VexLens/__Tests/StellaOps.VexLens.Tests/Determinism/
• Key tests:
  • VexProofShuffleDeterminismTests - Verifies statement order doesn't affect consensus
  • VexLensPipelineDeterminismTests - E2E pipeline structural determinism
• Run frequency: Nightly regression suite
• Requirements: Fixed TimeProvider injection for reproducible timestamps

Known Limitations

The VexProofBuilder.GenerateProofId() method currently uses Guid.NewGuid() which introduces non-determinism in the ProofId (and consequently the Digest). This is tracked as risk R-003 and violates AGENTS.md Rule 8.2 (Inject TimeProvider / ID generators). Until IGuidGenerator injection is added, determinism tests validate structural determinism rather than byte-identical outputs.

3. Regression Tests (Category=Regression)

Tests that validate known scenarios produce expected verdicts:

• Location: src/VexLens/StellaOps.VexLens/__Tests/StellaOps.VexLens.Tests/Regression/
• Key tests:
  • Fixed package verdict validation
  • Not_affected with justification
  • Conflict resolution via lattice precedence
  • Backport scenario handling
  • Under_investigation status preservation
  • Signature verification confidence bonus
  • Same-issuer temporal precedence
• Run frequency: Nightly regression suite

4. Golden Corpus Tests (Category=Golden)

Tests using curated real-world backport scenarios:

• Location: src/VexLens/StellaOps.VexLens/__Tests/StellaOps.VexLens.Tests/GoldenCorpus/
• Data location: src/__Tests/__Datasets/GoldenBackports/
• Coverage: 20 backport cases across 8 distributions (Debian, Ubuntu, RHEL, SUSE, CentOS, Alpine, Fedora, Rocky/Alma, Amazon Linux, Oracle Linux)
• Run frequency: Nightly regression suite

Test Infrastructure

Golden Corpus Structure

src/__Tests/__Datasets/GoldenBackports/
├── index.json              # Corpus index with case metadata
├── CVE-YYYY-XXXXX-distro-package/
│   └── case.json           # Individual test case
└── ...

Case Schema

{
  "id": "case identifier",
  "cve": "CVE-YYYY-XXXXX",
  "distro": {
    "name": "debian|rhel|...",
    "version": "7|8|...",
    "family": "debian|rpm|..."
  },
  "package": {
    "name": "package-name",
    "binary": "binary-name",
    "vulnerableEvr": "vulnerable EVR",
    "patchedEvr": "patched EVR"
  },
  "upstream": {
    "vulnerableRange": "SemVer constraint",
    "fixedVersion": "upstream fix version"
  },
  "expectedVerdict": {
    "status": "fixed|not_affected|affected",
    "reason": "backport_detected|...",
    "upstreamWouldSay": "affected|..."
  },
  "evidence": {
    "advisoryUrl": "URL to advisory",
    "patchCommit": "optional commit hash",
    "notes": "explanation"
  }
}

Test Runner Components

1. GoldenCorpusLoader - Loads and filters corpus cases
2. GoldenCorpusTestRunner - Executes evaluator against corpus
3. GoldenCorpusTests - xUnit test class for corpus validation

CI Integration

Nightly Regression (.gitea/workflows/nightly-regression.yml)

Test categories run nightly:

• Unit
• Architecture
• Contract
• Integration
• Security
• Golden
• Determinism
• Regression

PR Gating

Quick tests run on every PR:

• Unit tests only
• Build determinism check

Test Best Practices

1. Use FakeTimeProvider

Always inject Microsoft.Extensions.Time.Testing.FakeTimeProvider to ensure deterministic timestamps:

private readonly FakeTimeProvider _timeProvider;
private readonly DateTimeOffset _fixedTime = new(2026, 1, 3, 12, 0, 0, TimeSpan.Zero);

public MyTests()
{
    _timeProvider = new FakeTimeProvider(_fixedTime);
}

2. Set Confidence Explicitly

The VexProofBuilder calculates confidence from explicit weight values. For predictable test assertions, set all confidence-affecting properties:

builder
    .WithWeightSpread(0.95m)
    .WithConditionCoverage(1.0m)
    .WithSignatureBonus(0.10m)
    .WithFreshnessBonus(0.05m);

3. Set Justification with Status

When testing not_affected status, pass justification to WithFinalStatus:

.WithFinalStatus(VexStatus.NotAffected, VexJustification.VulnerableCodeNotPresent)

4. Use Correct Types

The AddStatement method requires proper types:

• VexProofIssuer - Not a string
• VexProofWeight with VexProofWeightFactors - Not a decimal

builder.AddStatement(
    id: "stmt-001",
    source: "vendor-csaf",
    issuer: new VexProofIssuer("Vendor", IssuerCategory.Vendor, TrustTier.Authoritative),
    status: VexStatus.Fixed,
    justification: null,
    weight: new VexProofWeight(0.95m, new VexProofWeightFactors(0.95m, 1.0m, 0.9m, 1.0m, 0.8m)),
    timestamp: _fixedTime,
    signatureVerified: true);

5. IssuerCategory and TrustTier Values

Valid IssuerCategory values:

• Unknown, Vendor, Distributor, Community, Internal, Aggregator

Valid TrustTier values:

• Authoritative, Trusted, Untrusted, Unknown

Future Improvements

1. Inject IGuidGenerator - Enable full digest determinism (R-003)
2. Expand golden corpus - Add more edge cases and distributions
3. Property-based testing - Use FsCheck for fuzzing VEX inputs
4. Mutation testing - Use Stryker.NET to validate test coverage quality