git.stella-ops.org/docs/testing/determinism-gates.md

# Determinism Gates

Determinism is a core principle of StellaOps - all artifact generation (SBOM, VEX, attestations) must be reproducible. This document describes how to test for determinism.

## Why Determinism Matters

- **Reproducible builds**: Same input → same output, always
- **Cryptographic verification**: Hash-based integrity depends on byte-for-byte reproducibility
- **Audit trails**: Deterministic timestamps and ordering for compliance
- **Offline operation**: No reliance on external randomness or timestamps

## Using Determinism Gates

Add `StellaOps.TestKit` to your test project and use the `DeterminismGate` class:

```csharp
using StellaOps.TestKit.Determinism;
using StellaOps.TestKit.Traits;
using Xunit;

public class SbomGeneratorTests
{
    [Fact]
    [UnitTest]
    [DeterminismTest]
    public void SbomGenerationIsDeterministic()
    {
        // Verify that calling the function 3 times produces identical output
        DeterminismGate.AssertDeterministic(() =>
        {
            return GenerateSbom();
        }, iterations: 3);
    }

    [Fact]
    [UnitTest]
    [DeterminismTest]
    public void SbomBinaryIsDeterministic()
    {
        // Verify binary reproducibility
        DeterminismGate.AssertDeterministic(() =>
        {
            return GenerateSbomBytes();
        }, iterations: 3);
    }
}
```

## JSON Determinism

JSON output must have:
- Stable property ordering (alphabetical or schema-defined)
- Consistent whitespace/formatting
- No random IDs or timestamps (unless explicitly from deterministic clock)

```csharp
[Fact]
[UnitTest]
[DeterminismTest]
public void VexDocumentJsonIsDeterministic()
{
    // Verifies JSON canonicalization and property ordering
    DeterminismGate.AssertJsonDeterministic(() =>
    {
        var vex = GenerateVexDocument();
        return JsonSerializer.Serialize(vex);
    });
}

[Fact]
[UnitTest]
[DeterminismTest]
public void VerdictObjectIsDeterministic()
{
    // Verifies object serialization is deterministic
    DeterminismGate.AssertJsonDeterministic(() =>
    {
        return GenerateVerdict();
    });
}
```

## Canonical Equality

Compare two objects for canonical equivalence:

```csharp
[Fact]
[UnitTest]
public void VerdictFromDifferentPathsAreCanonicallyEqual()
{
    var verdict1 = GenerateVerdictFromSbom();
    var verdict2 = GenerateVerdictFromCache();

    // Asserts that both produce identical canonical JSON
    DeterminismGate.AssertCanonicallyEqual(verdict1, verdict2);
}
```

## Hash-Based Regression Testing

Compute stable hashes for regression detection:

```csharp
[Fact]
[UnitTest]
[DeterminismTest]
public void SbomHashMatchesBaseline()
{
    var sbom = GenerateSbom();
    var hash = DeterminismGate.ComputeHash(sbom);

    // This hash should NEVER change unless SBOM format changes intentionally
    const string expectedHash = "abc123...";
    Assert.Equal(expectedHash, hash);
}
```

## Path Ordering

File paths in manifests must be sorted:

```csharp
[Fact]
[UnitTest]
[DeterminismTest]
public void SbomFilePathsAreSorted()
{
    var sbom = GenerateSbom();
    var filePaths = ExtractFilePaths(sbom);

    // Asserts paths are in deterministic (lexicographic) order
    DeterminismGate.AssertSortedPaths(filePaths);
}
```

## Timestamp Validation

All timestamps must be UTC ISO 8601:

```csharp
[Fact]
[UnitTest]
[DeterminismTest]
public void AttestationTimestampIsUtcIso8601()
{
    var attestation = GenerateAttestation();

    // Asserts timestamp is UTC with 'Z' suffix
    DeterminismGate.AssertUtcIso8601(attestation.Timestamp);
}
```

## Determin

istic Time in Tests

Use `DeterministicClock` for reproducible timestamps:

```csharp
using StellaOps.TestKit.Time;

[Fact]
[UnitTest]
[DeterminismTest]
public void AttestationWithDeterministicTime()
{
    var clock = new DeterministicClock();

    // All operations using this clock will get the same time
    var attestation1 = GenerateAttestation(clock);
    var attestation2 = GenerateAttestation(clock);

    Assert.Equal(attestation1.Timestamp, attestation2.Timestamp);
}
```

## Deterministic Random in Tests

Use `DeterministicRandom` for reproducible randomness:

```csharp
using StellaOps.TestKit.Random;

[Fact]
[UnitTest]
[DeterminismTest]
public void GeneratedIdsAreReproducible()
{
    var rng1 = DeterministicRandomExtensions.WithTestSeed();
    var id1 = GenerateId(rng1);

    var rng2 = DeterministicRandomExtensions.WithTestSeed();
    var id2 = GenerateId(rng2);

    // Same seed → same output
    Assert.Equal(id1, id2);
}
```

## Module-Specific Gates

### Scanner Determinism
- SBOM file path ordering
- Component hash stability
- Dependency graph ordering

### Concelier Determinism
- Advisory normalization (same advisory → same canonical form)
- Vulnerability merge determinism
- No lattice ordering dependencies

### Excititor Determinism
- VEX document format stability
- Preserve/prune decision ordering
- No lattice dependencies

### Policy Determinism
- Verdict reproducibility (same inputs → same verdict)
- Policy evaluation ordering
- Unknown budget calculations

### Attestor Determinism
- DSSE envelope canonical bytes
- Signature ordering (multiple signers)
- Rekor receipt stability

## Common Determinism Violations

❌ **Timestamps from system clock**
```csharp
// Bad: Uses system time
var timestamp = DateTimeOffset.UtcNow.ToString("o");

// Good: Uses injected clock
var timestamp = clock.UtcNow.ToString("o");
```

❌ **Random GUIDs**
```csharp
// Bad: Non-deterministic
var id = Guid.NewGuid().ToString();

// Good: Deterministic or content-addressed
var id = ComputeContentHash(data);
```

❌ **Unordered collections**
```csharp
// Bad: Dictionary iteration order is undefined
foreach (var (key, value) in dict) { ... }

// Good: Explicit ordering
foreach (var (key, value) in dict.OrderBy(x => x.Key)) { ... }
```

❌ **Floating-point comparisons**
```csharp
// Bad: Floating-point can differ across platforms
var score = 0.1 + 0.2;  // Might not equal 0.3 exactly

// Good: Use fixed-point or integers
var scoreInt = (int)((0.1 + 0.2) * 1000);
```

❌ **Non-UTC timestamps**
```csharp
// Bad: Timezone-dependent
var timestamp = DateTime.Now.ToString();

// Good: Always UTC with 'Z'
var timestamp = DateTimeOffset.UtcNow.ToString("o");
```

## Determinism Test Checklist

When writing determinism tests, verify:

- [ ] Multiple invocations produce identical output
- [ ] JSON has stable property ordering
- [ ] File paths are sorted lexicographically
- [ ] Timestamps are UTC ISO 8601 with 'Z' suffix
- [ ] No random GUIDs (use content-addressing)
- [ ] Collections are explicitly ordered
- [ ] No system time/random usage (use DeterministicClock/DeterministicRandom)

## Related Documentation

- TestKit README: `src/__Libraries/StellaOps.TestKit/README.md`
- Testing Strategy: `docs/testing/testing-strategy-models.md`
- Test Catalog: `docs/testing/TEST_CATALOG.yml`