git.stella-ops.org/docs/modules/policy/design/policy-determinism-tests.md

# Policy Determinism Test Design

**Document ID:** `DESIGN-POLICY-DETERMINISM-TESTS-001`
**Version:** 1.0
**Status:** Published
**Last Updated:** 2025-12-06

## Overview

This document defines the test expectations for ensuring deterministic output from Policy Engine scoring and decision APIs. Determinism is critical for reproducible policy evaluation across environments.

## Determinism Requirements

### Output Ordering

All collections in API responses must have stable, deterministic ordering:

| Collection | Ordering Rule |
|------------|---------------|
| Findings | By `finding_id` alphabetically |
| Decisions | By `decision_id` (timestamp prefix) |
| Signals | By signal name alphabetically |
| Severity counts | By canonical severity order: critical → high → medium → low → info |
| Contributions | By signal name alphabetically |

### JSON Serialization

1. **Property Order**: Use `[JsonPropertyOrder]` or declare properties in stable order
2. **No Random Elements**: No GUIDs, random IDs, or timestamps unless from context
3. **Stable Key Order**: Dictionary keys must serialize in consistent order

### Deprecated Field Absence

After v2.0, responses must NOT include:
- `normalized_score`
- `top_severity_sources`
- `source_rank`

See [Normalized Field Removal](./policy-normalized-field-removal.md).

## Test Categories

### 1. Snapshot Equality Tests

Verify that identical inputs produce byte-for-byte identical JSON outputs.

```csharp
[Theory]
[MemberData(nameof(DeterminismFixtures))]
public void Scoring_ShouldProduceDeterministicOutput(string inputFile)
{
    // Arrange
    var input = LoadFixture(inputFile);

    // Act - Run twice with same input
    var result1 = _scoringService.Score(input);
    var result2 = _scoringService.Score(input);

    // Assert - Byte-for-byte equality
    var json1 = JsonSerializer.Serialize(result1);
    var json2 = JsonSerializer.Serialize(result2);
    Assert.Equal(json1, json2);
}
```

### 2. Cross-Environment Tests

Verify output is identical across different environments (CI, local, prod).

```csharp
[Theory]
[InlineData("fixture-001")]
public void Scoring_ShouldMatchGoldenFile(string fixtureId)
{
    // Arrange
    var input = LoadFixture($"{fixtureId}-input.json");
    var expected = LoadFixture($"{fixtureId}-expected.json");

    // Act
    var result = _scoringService.Score(input);

    // Assert
    AssertJsonEqual(expected, result);
}
```

### 3. Ordering Verification Tests

Verify collections are always in expected order.

```csharp
[Fact]
public void Decisions_ShouldBeOrderedByDecisionId()
{
    // Arrange
    var input = CreateTestInput();

    // Act
    var result = _decisionService.Evaluate(input);

    // Assert
    Assert.True(result.Decisions.SequenceEqual(
        result.Decisions.OrderBy(d => d.DecisionId)));
}
```

### 4. Deprecated Field Absence Tests

Verify deprecated fields are not serialized (v2.0+).

```csharp
[Fact]
public void ScoringResult_ShouldNotIncludeNormalizedScore_InV2()
{
    // Arrange
    var result = new RiskScoringResult(...);
    var options = new PolicyScoringOptions { IncludeLegacyNormalizedScore = false };

    // Act
    var json = JsonSerializer.Serialize(result, CreateJsonOptions(options));
    var doc = JsonDocument.Parse(json);

    // Assert
    Assert.False(doc.RootElement.TryGetProperty("normalized_score", out _));
}
```

## Fixture Structure

### Input Fixtures

Located at `docs/modules/policy/samples/policy-determinism-fixtures*.json`:

```json
{
  "$schema": "https://stellaops.org/schemas/policy/determinism-fixture-v1.json",
  "fixture_id": "DET-001",
  "description": "Basic scoring determinism test",
  "input": {
    "finding_id": "CVE-2024-1234",
    "signals": {
      "cvss_base": 7.5,
      "exploitability": 2.8
    }
  },
  "expected_output": {
    "severity": "high",
    "signal_order": ["cvss_base", "exploitability"]
  }
}
```

### Golden Files

Pre-computed expected outputs stored alongside inputs:
- `policy-determinism-fixtures-input.json`
- `policy-determinism-fixtures-expected.json`

## CI Integration

### Pipeline Steps

1. **Build**: Compile with analyzers
2. **Unit Tests**: Run determinism unit tests
3. **Snapshot Tests**: Compare against golden files
4. **Diff Check**: Fail if any unexpected changes

### GitHub Action

```yaml
- name: Run Determinism Tests
  run: |
    dotnet test --filter "Category=Determinism"

- name: Verify Snapshots
  run: |
    dotnet run --project tools/SnapshotVerifier -- \
      --fixtures docs/modules/policy/samples/policy-determinism-*.json
```

## Maintenance

### Updating Golden Files

When intentionally changing output format:

1. Update design docs (this file, normalized-field-removal.md)
2. Re-run tests with `UPDATE_GOLDEN=true`
3. Review diffs
4. Commit new golden files with explanation

### Adding New Fixtures

1. Create input fixture in `samples/`
2. Run scoring to generate expected output
3. Review for correctness
4. Add to test data provider

## Related Documents

- [Policy AOC Linting Rules](./policy-aoc-linting-rules.md)
- [Normalized Field Removal](./policy-normalized-field-removal.md)
- [Deterministic Evaluator Design](./deterministic-evaluator.md)