stella-ops.org/git.stella-ops.org
1
0
Fork 0
Code Issues Pull Requests Actions 2 Releases Wiki Activity
Files
e716bc6adca8d1a3ac5a24ad491e5fa99e31cf1c
git.stella-ops.org/docs/features/unchecked/tests/determinism-property-based-testing.md

3.2 KiB
Raw Blame History

Determinism Property-Based Testing

Module

__Tests

Status

IMPLEMENTED

Description

Comprehensive determinism property-based tests covering unicode normalization, SBOM/VEX ordering, floating-point stability, digest computation, and canonical JSON to ensure reproducible verdicts.

Implementation Details

  • Unicode Normalization Properties: src/__Tests/__Libraries/StellaOps.Testing.Determinism.Properties/UnicodeNormalizationDeterminismProperties.cs -- FsCheck property-based tests verifying Unicode normalization produces identical output for equivalent Unicode representations.
  • SBOM/VEX Ordering Properties: src/__Tests/__Libraries/StellaOps.Testing.Determinism.Properties/SbomVexOrderingDeterminismProperties.cs -- verifies SBOM and VEX document processing produces identical results regardless of element ordering.
  • Floating-Point Stability Properties: src/__Tests/__Libraries/StellaOps.Testing.Determinism.Properties/FloatingPointStabilityProperties.cs -- verifies floating-point computations (scores, percentages) produce identical results across platforms and evaluation orders.
  • Digest Computation Properties: src/__Tests/__Libraries/StellaOps.Testing.Determinism.Properties/DigestComputationDeterminismProperties.cs -- verifies SHA-256 digest computations are deterministic for identical inputs.
  • Canonical JSON Properties: src/__Tests/__Libraries/StellaOps.Testing.Determinism.Properties/CanonicalJsonDeterminismProperties.cs -- verifies RFC 8785 canonical JSON serialization produces identical byte output for semantically equivalent JSON documents.
  • JSON Object Arbitraries: src/__Tests/__Libraries/StellaOps.Testing.Determinism.Properties/JsonObjectArbitraries.cs -- FsCheck arbitrary generators for producing random JSON structures for property-based testing.
  • Determinism Gate: src/__Tests/__Libraries/StellaOps.Testing.Determinism/Determinism/DeterminismGate.cs -- CI gate that fails the build if determinism properties are violated.
  • Determinism Manifest: src/__Tests/__Libraries/StellaOps.Testing.Determinism/Determinism/DeterminismManifest.cs -- captures determinism verification results as a test artifact.
  • Determinism Baseline Store: src/__Tests/__Libraries/StellaOps.Testing.Determinism/Determinism/DeterminismBaselineStore.cs -- stores determinism baselines for comparison across test runs.

E2E Test Plan

  • Run the Unicode normalization property tests with 1000 randomly generated Unicode strings and verify all produce identical normalized output
  • Run the SBOM ordering property tests with randomly shuffled SBOM components and verify the digest is identical regardless of input order
  • Run the floating-point stability properties and verify score computations produce identical results when operands are reordered
  • Run the canonical JSON properties with randomly generated JSON objects and verify RFC 8785 canonicalization produces identical output for equivalent inputs
  • Verify the determinism gate: introduce a non-deterministic computation and confirm the gate blocks the build
  • Verify determinism manifest: run the full property suite and confirm the manifest captures all property results with pass/fail status