git.stella-ops.org/docs/modules/vex-lens/testing-strategy.md

# VexLens Testing Strategy

## Overview

This document describes the testing strategy for VexLens module, covering determinism verification, regression testing, and golden corpus validation.

## Test Categories

### 1. Unit Tests (`Category=Unit`)

Standard unit tests for individual components:

- **Location:** `src/VexLens/StellaOps.VexLens/__Tests/StellaOps.VexLens.Tests/`
- **Coverage:** Models, builders, serializers, individual consensus components
- **Run frequency:** Every PR, every commit

### 2. Determinism Tests (`Category=Determinism`)

Tests that verify VexLens produces identical outputs for identical inputs:

- **Location:** `src/VexLens/StellaOps.VexLens/__Tests/StellaOps.VexLens.Tests/Determinism/`
- **Key tests:**
  - `VexProofShuffleDeterminismTests` - Verifies statement order doesn't affect consensus
  - `VexLensPipelineDeterminismTests` - E2E pipeline structural determinism
- **Run frequency:** Nightly regression suite
- **Requirements:** Fixed `TimeProvider` injection for reproducible timestamps

#### Known Limitations

The `VexProofBuilder.GenerateProofId()` method currently uses `Guid.NewGuid()` which introduces non-determinism in the ProofId (and consequently the Digest). This is tracked as risk R-003 and violates AGENTS.md Rule 8.2 (Inject TimeProvider / ID generators). Until `IGuidGenerator` injection is added, determinism tests validate structural determinism rather than byte-identical outputs.

### 3. Regression Tests (`Category=Regression`)

Tests that validate known scenarios produce expected verdicts:

- **Location:** `src/VexLens/StellaOps.VexLens/__Tests/StellaOps.VexLens.Tests/Regression/`
- **Key tests:**
  - Fixed package verdict validation
  - Not_affected with justification
  - Conflict resolution via lattice precedence
  - Backport scenario handling
  - Under_investigation status preservation
  - Signature verification confidence bonus
  - Same-issuer temporal precedence
- **Run frequency:** Nightly regression suite

### 4. Golden Corpus Tests (`Category=Golden`)

Tests using curated real-world backport scenarios:

- **Location:** `src/VexLens/StellaOps.VexLens/__Tests/StellaOps.VexLens.Tests/GoldenCorpus/`
- **Data location:** `src/__Tests/__Datasets/GoldenBackports/`
- **Coverage:** 20 backport cases across 8 distributions (Debian, Ubuntu, RHEL, SUSE, CentOS, Alpine, Fedora, Rocky/Alma, Amazon Linux, Oracle Linux)
- **Run frequency:** Nightly regression suite

## Test Infrastructure

### Golden Corpus Structure

```
src/__Tests/__Datasets/GoldenBackports/
├── index.json              # Corpus index with case metadata
├── CVE-YYYY-XXXXX-distro-package/
│   └── case.json           # Individual test case
└── ...
```

### Case Schema

```json
{
  "id": "case identifier",
  "cve": "CVE-YYYY-XXXXX",
  "distro": {
    "name": "debian|rhel|...",
    "version": "7|8|...",
    "family": "debian|rpm|..."
  },
  "package": {
    "name": "package-name",
    "binary": "binary-name",
    "vulnerableEvr": "vulnerable EVR",
    "patchedEvr": "patched EVR"
  },
  "upstream": {
    "vulnerableRange": "SemVer constraint",
    "fixedVersion": "upstream fix version"
  },
  "expectedVerdict": {
    "status": "fixed|not_affected|affected",
    "reason": "backport_detected|...",
    "upstreamWouldSay": "affected|..."
  },
  "evidence": {
    "advisoryUrl": "URL to advisory",
    "patchCommit": "optional commit hash",
    "notes": "explanation"
  }
}
```

### Test Runner Components

1. **GoldenCorpusLoader** - Loads and filters corpus cases
2. **GoldenCorpusTestRunner** - Executes evaluator against corpus
3. **GoldenCorpusTests** - xUnit test class for corpus validation

## CI Integration

### Nightly Regression (`.gitea/workflows/nightly-regression.yml`)

Test categories run nightly:
- Unit
- Architecture  
- Contract
- Integration
- Security
- Golden
- **Determinism**
- **Regression**

### PR Gating

Quick tests run on every PR:
- Unit tests only
- Build determinism check

## Test Best Practices

### 1. Use FakeTimeProvider

Always inject `Microsoft.Extensions.Time.Testing.FakeTimeProvider` to ensure deterministic timestamps:

```csharp
private readonly FakeTimeProvider _timeProvider;
private readonly DateTimeOffset _fixedTime = new(2026, 1, 3, 12, 0, 0, TimeSpan.Zero);

public MyTests()
{
    _timeProvider = new FakeTimeProvider(_fixedTime);
}
```

### 2. Set Confidence Explicitly

The `VexProofBuilder` calculates confidence from explicit weight values. For predictable test assertions, set all confidence-affecting properties:

```csharp
builder
    .WithWeightSpread(0.95m)
    .WithConditionCoverage(1.0m)
    .WithSignatureBonus(0.10m)
    .WithFreshnessBonus(0.05m);
```

### 3. Set Justification with Status

When testing `not_affected` status, pass justification to `WithFinalStatus`:

```csharp
.WithFinalStatus(VexStatus.NotAffected, VexJustification.VulnerableCodeNotPresent)
```

### 4. Use Correct Types

The `AddStatement` method requires proper types:
- `VexProofIssuer` - Not a string
- `VexProofWeight` with `VexProofWeightFactors` - Not a decimal

```csharp
builder.AddStatement(
    id: "stmt-001",
    source: "vendor-csaf",
    issuer: new VexProofIssuer("Vendor", IssuerCategory.Vendor, TrustTier.Authoritative),
    status: VexStatus.Fixed,
    justification: null,
    weight: new VexProofWeight(0.95m, new VexProofWeightFactors(0.95m, 1.0m, 0.9m, 1.0m, 0.8m)),
    timestamp: _fixedTime,
    signatureVerified: true);
```

### 5. IssuerCategory and TrustTier Values

Valid `IssuerCategory` values:
- `Unknown`, `Vendor`, `Distributor`, `Community`, `Internal`, `Aggregator`

Valid `TrustTier` values:
- `Authoritative`, `Trusted`, `Untrusted`, `Unknown`

## Future Improvements

1. **Inject IGuidGenerator** - Enable full digest determinism (R-003)
2. **Expand golden corpus** - Add more edge cases and distributions
3. **Property-based testing** - Use FsCheck for fuzzing VEX inputs
4. **Mutation testing** - Use Stryker.NET to validate test coverage quality