git.stella-ops.org/docs/modules/policy/design/policy-determinism-tests.md

Policy Determinism Test Design

Document ID: DESIGN-POLICY-DETERMINISM-TESTS-001 Version: 1.0 Status: Published Last Updated: 2025-12-06

Overview

This document defines the test expectations for ensuring deterministic output from Policy Engine scoring and decision APIs. Determinism is critical for reproducible policy evaluation across environments.

Determinism Requirements

Output Ordering

All collections in API responses must have stable, deterministic ordering:

Collection	Ordering Rule
Findings	By finding_id alphabetically
Decisions	By decision_id (timestamp prefix)
Signals	By signal name alphabetically
Severity counts	By canonical severity order: critical → high → medium → low → info
Contributions	By signal name alphabetically

JSON Serialization

1. Property Order: Use [JsonPropertyOrder] or declare properties in stable order
2. No Random Elements: No GUIDs, random IDs, or timestamps unless from context
3. Stable Key Order: Dictionary keys must serialize in consistent order

Deprecated Field Absence

After v2.0, responses must NOT include:

• normalized_score
• top_severity_sources
• source_rank

See Normalized Field Removal.

Test Categories

1. Snapshot Equality Tests

Verify that identical inputs produce byte-for-byte identical JSON outputs.

[Theory]
[MemberData(nameof(DeterminismFixtures))]
public void Scoring_ShouldProduceDeterministicOutput(string inputFile)
{
    // Arrange
    var input = LoadFixture(inputFile);

    // Act - Run twice with same input
    var result1 = _scoringService.Score(input);
    var result2 = _scoringService.Score(input);

    // Assert - Byte-for-byte equality
    var json1 = JsonSerializer.Serialize(result1);
    var json2 = JsonSerializer.Serialize(result2);
    Assert.Equal(json1, json2);
}

2. Cross-Environment Tests

Verify output is identical across different environments (CI, local, prod).

[Theory]
[InlineData("fixture-001")]
public void Scoring_ShouldMatchGoldenFile(string fixtureId)
{
    // Arrange
    var input = LoadFixture($"{fixtureId}-input.json");
    var expected = LoadFixture($"{fixtureId}-expected.json");

    // Act
    var result = _scoringService.Score(input);

    // Assert
    AssertJsonEqual(expected, result);
}

3. Ordering Verification Tests

Verify collections are always in expected order.

[Fact]
public void Decisions_ShouldBeOrderedByDecisionId()
{
    // Arrange
    var input = CreateTestInput();

    // Act
    var result = _decisionService.Evaluate(input);

    // Assert
    Assert.True(result.Decisions.SequenceEqual(
        result.Decisions.OrderBy(d => d.DecisionId)));
}

4. Deprecated Field Absence Tests

Verify deprecated fields are not serialized (v2.0+).

[Fact]
public void ScoringResult_ShouldNotIncludeNormalizedScore_InV2()
{
    // Arrange
    var result = new RiskScoringResult(...);
    var options = new PolicyScoringOptions { IncludeLegacyNormalizedScore = false };

    // Act
    var json = JsonSerializer.Serialize(result, CreateJsonOptions(options));
    var doc = JsonDocument.Parse(json);

    // Assert
    Assert.False(doc.RootElement.TryGetProperty("normalized_score", out _));
}

Fixture Structure

Input Fixtures

Located at docs/modules/policy/samples/policy-determinism-fixtures*.json:

{
  "$schema": "https://stellaops.org/schemas/policy/determinism-fixture-v1.json",
  "fixture_id": "DET-001",
  "description": "Basic scoring determinism test",
  "input": {
    "finding_id": "CVE-2024-1234",
    "signals": {
      "cvss_base": 7.5,
      "exploitability": 2.8
    }
  },
  "expected_output": {
    "severity": "high",
    "signal_order": ["cvss_base", "exploitability"]
  }
}

Golden Files

Pre-computed expected outputs stored alongside inputs:

• policy-determinism-fixtures-input.json
• policy-determinism-fixtures-expected.json

CI Integration

Pipeline Steps

1. Build: Compile with analyzers
2. Unit Tests: Run determinism unit tests
3. Snapshot Tests: Compare against golden files
4. Diff Check: Fail if any unexpected changes

GitHub Action

- name: Run Determinism Tests
  run: |
    dotnet test --filter "Category=Determinism"

- name: Verify Snapshots
  run: |
    dotnet run --project tools/SnapshotVerifier -- \
      --fixtures docs/modules/policy/samples/policy-determinism-*.json

Maintenance

Updating Golden Files

When intentionally changing output format:

1. Update design docs (this file, normalized-field-removal.md)
2. Re-run tests with UPDATE_GOLDEN=true
3. Review diffs
4. Commit new golden files with explanation

Adding New Fixtures

1. Create input fixture in samples/
2. Run scoring to generate expected output
3. Review for correctness
4. Add to test data provider

Related Documents

• Policy AOC Linting Rules
• Normalized Field Removal
• Deterministic Evaluator Design