git.stella-ops.org/docs/testing/determinism-gates.md

Determinism Gates

Determinism is a core principle of StellaOps - all artifact generation (SBOM, VEX, attestations) must be reproducible. This document describes how to test for determinism.

Why Determinism Matters

• Reproducible builds: Same input → same output, always
• Cryptographic verification: Hash-based integrity depends on byte-for-byte reproducibility
• Audit trails: Deterministic timestamps and ordering for compliance
• Offline operation: No reliance on external randomness or timestamps

Using Determinism Gates

Add StellaOps.TestKit to your test project and use the DeterminismGate class:

using StellaOps.TestKit.Determinism;
using StellaOps.TestKit.Traits;
using Xunit;

public class SbomGeneratorTests
{
    [Fact]
    [UnitTest]
    [DeterminismTest]
    public void SbomGenerationIsDeterministic()
    {
        // Verify that calling the function 3 times produces identical output
        DeterminismGate.AssertDeterministic(() =>
        {
            return GenerateSbom();
        }, iterations: 3);
    }

    [Fact]
    [UnitTest]
    [DeterminismTest]
    public void SbomBinaryIsDeterministic()
    {
        // Verify binary reproducibility
        DeterminismGate.AssertDeterministic(() =>
        {
            return GenerateSbomBytes();
        }, iterations: 3);
    }
}

JSON Determinism

JSON output must have:

• Stable property ordering (alphabetical or schema-defined)
• Consistent whitespace/formatting
• No random IDs or timestamps (unless explicitly from deterministic clock)

[Fact]
[UnitTest]
[DeterminismTest]
public void VexDocumentJsonIsDeterministic()
{
    // Verifies JSON canonicalization and property ordering
    DeterminismGate.AssertJsonDeterministic(() =>
    {
        var vex = GenerateVexDocument();
        return JsonSerializer.Serialize(vex);
    });
}

[Fact]
[UnitTest]
[DeterminismTest]
public void VerdictObjectIsDeterministic()
{
    // Verifies object serialization is deterministic
    DeterminismGate.AssertJsonDeterministic(() =>
    {
        return GenerateVerdict();
    });
}

Canonical Equality

Compare two objects for canonical equivalence:

[Fact]
[UnitTest]
public void VerdictFromDifferentPathsAreCanonicallyEqual()
{
    var verdict1 = GenerateVerdictFromSbom();
    var verdict2 = GenerateVerdictFromCache();

    // Asserts that both produce identical canonical JSON
    DeterminismGate.AssertCanonicallyEqual(verdict1, verdict2);
}

Hash-Based Regression Testing

Compute stable hashes for regression detection:

[Fact]
[UnitTest]
[DeterminismTest]
public void SbomHashMatchesBaseline()
{
    var sbom = GenerateSbom();
    var hash = DeterminismGate.ComputeHash(sbom);

    // This hash should NEVER change unless SBOM format changes intentionally
    const string expectedHash = "abc123...";
    Assert.Equal(expectedHash, hash);
}

Path Ordering

File paths in manifests must be sorted:

[Fact]
[UnitTest]
[DeterminismTest]
public void SbomFilePathsAreSorted()
{
    var sbom = GenerateSbom();
    var filePaths = ExtractFilePaths(sbom);

    // Asserts paths are in deterministic (lexicographic) order
    DeterminismGate.AssertSortedPaths(filePaths);
}

Timestamp Validation

All timestamps must be UTC ISO 8601:

[Fact]
[UnitTest]
[DeterminismTest]
public void AttestationTimestampIsUtcIso8601()
{
    var attestation = GenerateAttestation();

    // Asserts timestamp is UTC with 'Z' suffix
    DeterminismGate.AssertUtcIso8601(attestation.Timestamp);
}

Determin

istic Time in Tests

Use DeterministicClock for reproducible timestamps:

using StellaOps.TestKit.Time;

[Fact]
[UnitTest]
[DeterminismTest]
public void AttestationWithDeterministicTime()
{
    var clock = new DeterministicClock();

    // All operations using this clock will get the same time
    var attestation1 = GenerateAttestation(clock);
    var attestation2 = GenerateAttestation(clock);

    Assert.Equal(attestation1.Timestamp, attestation2.Timestamp);
}

Deterministic Random in Tests

Use DeterministicRandom for reproducible randomness:

using StellaOps.TestKit.Random;

[Fact]
[UnitTest]
[DeterminismTest]
public void GeneratedIdsAreReproducible()
{
    var rng1 = DeterministicRandomExtensions.WithTestSeed();
    var id1 = GenerateId(rng1);

    var rng2 = DeterministicRandomExtensions.WithTestSeed();
    var id2 = GenerateId(rng2);

    // Same seed → same output
    Assert.Equal(id1, id2);
}

Module-Specific Gates

Scanner Determinism

• SBOM file path ordering
• Component hash stability
• Dependency graph ordering

Concelier Determinism

• Advisory normalization (same advisory → same canonical form)
• Vulnerability merge determinism
• No lattice ordering dependencies

Excititor Determinism

• VEX document format stability
• Preserve/prune decision ordering
• No lattice dependencies

Policy Determinism

• Verdict reproducibility (same inputs → same verdict)
• Policy evaluation ordering
• Unknown budget calculations

Attestor Determinism

• DSSE envelope canonical bytes
• Signature ordering (multiple signers)
• Rekor receipt stability

Common Determinism Violations

❌ Timestamps from system clock

// Bad: Uses system time
var timestamp = DateTimeOffset.UtcNow.ToString("o");

// Good: Uses injected clock
var timestamp = clock.UtcNow.ToString("o");

❌ Random GUIDs

// Bad: Non-deterministic
var id = Guid.NewGuid().ToString();

// Good: Deterministic or content-addressed
var id = ComputeContentHash(data);

❌ Unordered collections

// Bad: Dictionary iteration order is undefined
foreach (var (key, value) in dict) { ... }

// Good: Explicit ordering
foreach (var (key, value) in dict.OrderBy(x => x.Key)) { ... }

❌ Floating-point comparisons

// Bad: Floating-point can differ across platforms
var score = 0.1 + 0.2;  // Might not equal 0.3 exactly

// Good: Use fixed-point or integers
var scoreInt = (int)((0.1 + 0.2) * 1000);

❌ Non-UTC timestamps

// Bad: Timezone-dependent
var timestamp = DateTime.Now.ToString();

// Good: Always UTC with 'Z'
var timestamp = DateTimeOffset.UtcNow.ToString("o");

Determinism Test Checklist

When writing determinism tests, verify:

• Multiple invocations produce identical output
• JSON has stable property ordering
• File paths are sorted lexicographically
• Timestamps are UTC ISO 8601 with 'Z' suffix
• No random GUIDs (use content-addressing)
• Collections are explicitly ordered
• No system time/random usage (use DeterministicClock/DeterministicRandom)

Related Documentation

• TestKit README: src/__Libraries/StellaOps.TestKit/README.md
• Testing Strategy: docs/testing/testing-strategy-models.md
• Test Catalog: docs/testing/TEST_CATALOG.yml