save progress
This commit is contained in:
StellaOps Bot
195
docs/modules/vex-lens/testing-strategy.md
Normal file
195
docs/modules/vex-lens/testing-strategy.md
Normal file
@@ -0,0 +1,195 @@
|
||||
# VexLens Testing Strategy
|
||||
|
||||
## Overview
|
||||
|
||||
This document describes the testing strategy for VexLens module, covering determinism verification, regression testing, and golden corpus validation.
|
||||
|
||||
## Test Categories
|
||||
|
||||
### 1. Unit Tests (`Category=Unit`)
|
||||
|
||||
Standard unit tests for individual components:
|
||||
|
||||
- **Location:** `src/VexLens/StellaOps.VexLens/__Tests/StellaOps.VexLens.Tests/`
|
||||
- **Coverage:** Models, builders, serializers, individual consensus components
|
||||
- **Run frequency:** Every PR, every commit
|
||||
|
||||
### 2. Determinism Tests (`Category=Determinism`)
|
||||
|
||||
Tests that verify VexLens produces identical outputs for identical inputs:
|
||||
|
||||
- **Location:** `src/VexLens/StellaOps.VexLens/__Tests/StellaOps.VexLens.Tests/Determinism/`
|
||||
- **Key tests:**
|
||||
- `VexProofShuffleDeterminismTests` - Verifies statement order doesn't affect consensus
|
||||
- `VexLensPipelineDeterminismTests` - E2E pipeline structural determinism
|
||||
- **Run frequency:** Nightly regression suite
|
||||
- **Requirements:** Fixed `TimeProvider` injection for reproducible timestamps
|
||||
|
||||
#### Known Limitations
|
||||
|
||||
The `VexProofBuilder.GenerateProofId()` method currently uses `Guid.NewGuid()` which introduces non-determinism in the ProofId (and consequently the Digest). This is tracked as risk R-003 and violates AGENTS.md Rule 8.2 (Inject TimeProvider / ID generators). Until `IGuidGenerator` injection is added, determinism tests validate structural determinism rather than byte-identical outputs.
|
||||
|
||||
### 3. Regression Tests (`Category=Regression`)
|
||||
|
||||
Tests that validate known scenarios produce expected verdicts:
|
||||
|
||||
- **Location:** `src/VexLens/StellaOps.VexLens/__Tests/StellaOps.VexLens.Tests/Regression/`
|
||||
- **Key tests:**
|
||||
- Fixed package verdict validation
|
||||
- Not_affected with justification
|
||||
- Conflict resolution via lattice precedence
|
||||
- Backport scenario handling
|
||||
- Under_investigation status preservation
|
||||
- Signature verification confidence bonus
|
||||
- Same-issuer temporal precedence
|
||||
- **Run frequency:** Nightly regression suite
|
||||
|
||||
### 4. Golden Corpus Tests (`Category=Golden`)
|
||||
|
||||
Tests using curated real-world backport scenarios:
|
||||
|
||||
- **Location:** `src/VexLens/StellaOps.VexLens/__Tests/StellaOps.VexLens.Tests/GoldenCorpus/`
|
||||
- **Data location:** `src/__Tests/__Datasets/GoldenBackports/`
|
||||
- **Coverage:** 20 backport cases across 8 distributions (Debian, Ubuntu, RHEL, SUSE, CentOS, Alpine, Fedora, Rocky/Alma, Amazon Linux, Oracle Linux)
|
||||
- **Run frequency:** Nightly regression suite
|
||||
|
||||
## Test Infrastructure
|
||||
|
||||
### Golden Corpus Structure
|
||||
|
||||
```
|
||||
src/__Tests/__Datasets/GoldenBackports/
|
||||
├── index.json # Corpus index with case metadata
|
||||
├── CVE-YYYY-XXXXX-distro-package/
|
||||
│ └── case.json # Individual test case
|
||||
└── ...
|
||||
```
|
||||
|
||||
### Case Schema
|
||||
|
||||
```json
|
||||
{
|
||||
"id": "case identifier",
|
||||
"cve": "CVE-YYYY-XXXXX",
|
||||
"distro": {
|
||||
"name": "debian|rhel|...",
|
||||
"version": "7|8|...",
|
||||
"family": "debian|rpm|..."
|
||||
},
|
||||
"package": {
|
||||
"name": "package-name",
|
||||
"binary": "binary-name",
|
||||
"vulnerableEvr": "vulnerable EVR",
|
||||
"patchedEvr": "patched EVR"
|
||||
},
|
||||
"upstream": {
|
||||
"vulnerableRange": "SemVer constraint",
|
||||
"fixedVersion": "upstream fix version"
|
||||
},
|
||||
"expectedVerdict": {
|
||||
"status": "fixed|not_affected|affected",
|
||||
"reason": "backport_detected|...",
|
||||
"upstreamWouldSay": "affected|..."
|
||||
},
|
||||
"evidence": {
|
||||
"advisoryUrl": "URL to advisory",
|
||||
"patchCommit": "optional commit hash",
|
||||
"notes": "explanation"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Test Runner Components
|
||||
|
||||
1. **GoldenCorpusLoader** - Loads and filters corpus cases
|
||||
2. **GoldenCorpusTestRunner** - Executes evaluator against corpus
|
||||
3. **GoldenCorpusTests** - xUnit test class for corpus validation
|
||||
|
||||
## CI Integration
|
||||
|
||||
### Nightly Regression (`.gitea/workflows/nightly-regression.yml`)
|
||||
|
||||
Test categories run nightly:
|
||||
- Unit
|
||||
- Architecture
|
||||
- Contract
|
||||
- Integration
|
||||
- Security
|
||||
- Golden
|
||||
- **Determinism**
|
||||
- **Regression**
|
||||
|
||||
### PR Gating
|
||||
|
||||
Quick tests run on every PR:
|
||||
- Unit tests only
|
||||
- Build determinism check
|
||||
|
||||
## Test Best Practices
|
||||
|
||||
### 1. Use FakeTimeProvider
|
||||
|
||||
Always inject `Microsoft.Extensions.Time.Testing.FakeTimeProvider` to ensure deterministic timestamps:
|
||||
|
||||
```csharp
|
||||
private readonly FakeTimeProvider _timeProvider;
|
||||
private readonly DateTimeOffset _fixedTime = new(2026, 1, 3, 12, 0, 0, TimeSpan.Zero);
|
||||
|
||||
public MyTests()
|
||||
{
|
||||
_timeProvider = new FakeTimeProvider(_fixedTime);
|
||||
}
|
||||
```
|
||||
|
||||
### 2. Set Confidence Explicitly
|
||||
|
||||
The `VexProofBuilder` calculates confidence from explicit weight values. For predictable test assertions, set all confidence-affecting properties:
|
||||
|
||||
```csharp
|
||||
builder
|
||||
.WithWeightSpread(0.95m)
|
||||
.WithConditionCoverage(1.0m)
|
||||
.WithSignatureBonus(0.10m)
|
||||
.WithFreshnessBonus(0.05m);
|
||||
```
|
||||
|
||||
### 3. Set Justification with Status
|
||||
|
||||
When testing `not_affected` status, pass justification to `WithFinalStatus`:
|
||||
|
||||
```csharp
|
||||
.WithFinalStatus(VexStatus.NotAffected, VexJustification.VulnerableCodeNotPresent)
|
||||
```
|
||||
|
||||
### 4. Use Correct Types
|
||||
|
||||
The `AddStatement` method requires proper types:
|
||||
- `VexProofIssuer` - Not a string
|
||||
- `VexProofWeight` with `VexProofWeightFactors` - Not a decimal
|
||||
|
||||
```csharp
|
||||
builder.AddStatement(
|
||||
id: "stmt-001",
|
||||
source: "vendor-csaf",
|
||||
issuer: new VexProofIssuer("Vendor", IssuerCategory.Vendor, TrustTier.Authoritative),
|
||||
status: VexStatus.Fixed,
|
||||
justification: null,
|
||||
weight: new VexProofWeight(0.95m, new VexProofWeightFactors(0.95m, 1.0m, 0.9m, 1.0m, 0.8m)),
|
||||
timestamp: _fixedTime,
|
||||
signatureVerified: true);
|
||||
```
|
||||
|
||||
### 5. IssuerCategory and TrustTier Values
|
||||
|
||||
Valid `IssuerCategory` values:
|
||||
- `Unknown`, `Vendor`, `Distributor`, `Community`, `Internal`, `Aggregator`
|
||||
|
||||
Valid `TrustTier` values:
|
||||
- `Authoritative`, `Trusted`, `Untrusted`, `Unknown`
|
||||
|
||||
## Future Improvements
|
||||
|
||||
1. **Inject IGuidGenerator** - Enable full digest determinism (R-003)
|
||||
2. **Expand golden corpus** - Add more edge cases and distributions
|
||||
3. **Property-based testing** - Use FsCheck for fuzzing VEX inputs
|
||||
4. **Mutation testing** - Use Stryker.NET to validate test coverage quality
|
||||
Reference in New Issue
Block a user