Implement VEX document verification system with issuer management and signature verification
- Added IIssuerDirectory interface for managing VEX document issuers, including methods for registration, revocation, and trust validation. - Created InMemoryIssuerDirectory class as an in-memory implementation of IIssuerDirectory for testing and single-instance deployments. - Introduced ISignatureVerifier interface for verifying signatures on VEX documents, with support for multiple signature formats. - Developed SignatureVerifier class as the default implementation of ISignatureVerifier, allowing extensibility for different signature formats. - Implemented handlers for DSSE and JWS signature formats, including methods for verification and signature extraction. - Defined various records and enums for issuer and signature metadata, enhancing the structure and clarity of the verification process.
This commit is contained in:
203
docs/modules/policy/design/policy-determinism-tests.md
Normal file
203
docs/modules/policy/design/policy-determinism-tests.md
Normal file
@@ -0,0 +1,203 @@
|
||||
# Policy Determinism Test Design
|
||||
|
||||
**Document ID:** `DESIGN-POLICY-DETERMINISM-TESTS-001`
|
||||
**Version:** 1.0
|
||||
**Status:** Published
|
||||
**Last Updated:** 2025-12-06
|
||||
|
||||
## Overview
|
||||
|
||||
This document defines the test expectations for ensuring deterministic output from Policy Engine scoring and decision APIs. Determinism is critical for reproducible policy evaluation across environments.
|
||||
|
||||
## Determinism Requirements
|
||||
|
||||
### Output Ordering
|
||||
|
||||
All collections in API responses must have stable, deterministic ordering:
|
||||
|
||||
| Collection | Ordering Rule |
|
||||
|------------|---------------|
|
||||
| Findings | By `finding_id` alphabetically |
|
||||
| Decisions | By `decision_id` (timestamp prefix) |
|
||||
| Signals | By signal name alphabetically |
|
||||
| Severity counts | By canonical severity order: critical → high → medium → low → info |
|
||||
| Contributions | By signal name alphabetically |
|
||||
|
||||
### JSON Serialization
|
||||
|
||||
1. **Property Order**: Use `[JsonPropertyOrder]` or declare properties in stable order
|
||||
2. **No Random Elements**: No GUIDs, random IDs, or timestamps unless from context
|
||||
3. **Stable Key Order**: Dictionary keys must serialize in consistent order
|
||||
|
||||
### Deprecated Field Absence
|
||||
|
||||
After v2.0, responses must NOT include:
|
||||
- `normalized_score`
|
||||
- `top_severity_sources`
|
||||
- `source_rank`
|
||||
|
||||
See [Normalized Field Removal](./policy-normalized-field-removal.md).
|
||||
|
||||
## Test Categories
|
||||
|
||||
### 1. Snapshot Equality Tests
|
||||
|
||||
Verify that identical inputs produce byte-for-byte identical JSON outputs.
|
||||
|
||||
```csharp
|
||||
[Theory]
|
||||
[MemberData(nameof(DeterminismFixtures))]
|
||||
public void Scoring_ShouldProduceDeterministicOutput(string inputFile)
|
||||
{
|
||||
// Arrange
|
||||
var input = LoadFixture(inputFile);
|
||||
|
||||
// Act - Run twice with same input
|
||||
var result1 = _scoringService.Score(input);
|
||||
var result2 = _scoringService.Score(input);
|
||||
|
||||
// Assert - Byte-for-byte equality
|
||||
var json1 = JsonSerializer.Serialize(result1);
|
||||
var json2 = JsonSerializer.Serialize(result2);
|
||||
Assert.Equal(json1, json2);
|
||||
}
|
||||
```
|
||||
|
||||
### 2. Cross-Environment Tests
|
||||
|
||||
Verify output is identical across different environments (CI, local, prod).
|
||||
|
||||
```csharp
|
||||
[Theory]
|
||||
[InlineData("fixture-001")]
|
||||
public void Scoring_ShouldMatchGoldenFile(string fixtureId)
|
||||
{
|
||||
// Arrange
|
||||
var input = LoadFixture($"{fixtureId}-input.json");
|
||||
var expected = LoadFixture($"{fixtureId}-expected.json");
|
||||
|
||||
// Act
|
||||
var result = _scoringService.Score(input);
|
||||
|
||||
// Assert
|
||||
AssertJsonEqual(expected, result);
|
||||
}
|
||||
```
|
||||
|
||||
### 3. Ordering Verification Tests
|
||||
|
||||
Verify collections are always in expected order.
|
||||
|
||||
```csharp
|
||||
[Fact]
|
||||
public void Decisions_ShouldBeOrderedByDecisionId()
|
||||
{
|
||||
// Arrange
|
||||
var input = CreateTestInput();
|
||||
|
||||
// Act
|
||||
var result = _decisionService.Evaluate(input);
|
||||
|
||||
// Assert
|
||||
Assert.True(result.Decisions.SequenceEqual(
|
||||
result.Decisions.OrderBy(d => d.DecisionId)));
|
||||
}
|
||||
```
|
||||
|
||||
### 4. Deprecated Field Absence Tests
|
||||
|
||||
Verify deprecated fields are not serialized (v2.0+).
|
||||
|
||||
```csharp
|
||||
[Fact]
|
||||
public void ScoringResult_ShouldNotIncludeNormalizedScore_InV2()
|
||||
{
|
||||
// Arrange
|
||||
var result = new RiskScoringResult(...);
|
||||
var options = new PolicyScoringOptions { IncludeLegacyNormalizedScore = false };
|
||||
|
||||
// Act
|
||||
var json = JsonSerializer.Serialize(result, CreateJsonOptions(options));
|
||||
var doc = JsonDocument.Parse(json);
|
||||
|
||||
// Assert
|
||||
Assert.False(doc.RootElement.TryGetProperty("normalized_score", out _));
|
||||
}
|
||||
```
|
||||
|
||||
## Fixture Structure
|
||||
|
||||
### Input Fixtures
|
||||
|
||||
Located at `docs/modules/policy/samples/policy-determinism-fixtures*.json`:
|
||||
|
||||
```json
|
||||
{
|
||||
"$schema": "https://stellaops.org/schemas/policy/determinism-fixture-v1.json",
|
||||
"fixture_id": "DET-001",
|
||||
"description": "Basic scoring determinism test",
|
||||
"input": {
|
||||
"finding_id": "CVE-2024-1234",
|
||||
"signals": {
|
||||
"cvss_base": 7.5,
|
||||
"exploitability": 2.8
|
||||
}
|
||||
},
|
||||
"expected_output": {
|
||||
"severity": "high",
|
||||
"signal_order": ["cvss_base", "exploitability"]
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Golden Files
|
||||
|
||||
Pre-computed expected outputs stored alongside inputs:
|
||||
- `policy-determinism-fixtures-input.json`
|
||||
- `policy-determinism-fixtures-expected.json`
|
||||
|
||||
## CI Integration
|
||||
|
||||
### Pipeline Steps
|
||||
|
||||
1. **Build**: Compile with analyzers
|
||||
2. **Unit Tests**: Run determinism unit tests
|
||||
3. **Snapshot Tests**: Compare against golden files
|
||||
4. **Diff Check**: Fail if any unexpected changes
|
||||
|
||||
### GitHub Action
|
||||
|
||||
```yaml
|
||||
- name: Run Determinism Tests
|
||||
run: |
|
||||
dotnet test --filter "Category=Determinism"
|
||||
|
||||
- name: Verify Snapshots
|
||||
run: |
|
||||
dotnet run --project tools/SnapshotVerifier -- \
|
||||
--fixtures docs/modules/policy/samples/policy-determinism-*.json
|
||||
```
|
||||
|
||||
## Maintenance
|
||||
|
||||
### Updating Golden Files
|
||||
|
||||
When intentionally changing output format:
|
||||
|
||||
1. Update design docs (this file, normalized-field-removal.md)
|
||||
2. Re-run tests with `UPDATE_GOLDEN=true`
|
||||
3. Review diffs
|
||||
4. Commit new golden files with explanation
|
||||
|
||||
### Adding New Fixtures
|
||||
|
||||
1. Create input fixture in `samples/`
|
||||
2. Run scoring to generate expected output
|
||||
3. Review for correctness
|
||||
4. Add to test data provider
|
||||
|
||||
## Related Documents
|
||||
|
||||
- [Policy AOC Linting Rules](./policy-aoc-linting-rules.md)
|
||||
- [Normalized Field Removal](./policy-normalized-field-removal.md)
|
||||
- [Deterministic Evaluator Design](./deterministic-evaluator.md)
|
||||
Reference in New Issue
Block a user