stella-ops.org/git.stella-ops.org
1
0
Fork 0
Code Issues Pull Requests Actions 2 Releases Wiki Activity
Files
951a38d561d095a3ebf6454c94b0fe00ad55bdad
git.stella-ops.org/bench/determinism/README.md
master 951a38d561 Add Canonical JSON serialization library with tests and documentation 
- Implemented CanonJson class for deterministic JSON serialization and hashing.
- Added unit tests for CanonJson functionality, covering various scenarios including key sorting, handling of nested objects, arrays, and special characters.
- Created project files for the Canonical JSON library and its tests, including necessary package references.
- Added README.md for library usage and API reference.
- Introduced RabbitMqIntegrationFactAttribute for conditional RabbitMQ integration tests.
2025-12-19 15:35:00 +02:00

2.9 KiB
Raw Blame History

Determinism Benchmark Suite

Purpose: Verify that StellaOps produces bit-identical results across replays. Status: Active Sprint: SPRINT_3850_0001_0001 (Competitive Gap Closure)

Overview

Determinism is a core differentiator for StellaOps:

  • Same inputs → same outputs (bit-identical)
  • Replay manifests enable audit verification
  • No hidden state or environment leakage

What Gets Tested

Canonical JSON

  • Object key ordering (alphabetical)
  • Number formatting consistency
  • UTF-8 encoding without BOM
  • No whitespace variation

Scan Manifests

  • Same artifact + same feeds → same manifest hash
  • Seed values propagate correctly
  • Timestamp handling (fixed UTC)

Proof Bundles

  • Root hash computation
  • DSSE envelope determinism
  • ProofLedger node ordering

Score Computation

  • Same manifest → same score
  • Lattice merge is associative/commutative
  • Policy rule ordering doesn't affect outcome

Test Cases

TC-001: Canonical JSON Determinism

# Run same object through CanonJson 100 times
# All hashes must match

TC-002: Manifest Hash Stability

# Create manifest with identical inputs
# Verify ComputeHash() returns same value

TC-003: Cross-Platform Determinism

# Run on Linux, Windows, macOS
# Compare output hashes

TC-004: Feed Snapshot Determinism

# Same feed snapshot hash → same scan results

Fixtures

fixtures/
├── sample-manifest.json
├── sample-ledger.json
├── expected-hashes.json
└── cross-platform/
    ├── linux-x64.hashes.json
    ├── windows-x64.hashes.json
    └── macos-arm64.hashes.json

Running the Suite

# Run determinism tests
dotnet test tests/StellaOps.Determinism.Tests

# Run replay verification
./run-replay.sh --manifest fixtures/sample-manifest.json --runs 10

# Cross-platform verification (requires CI matrix)
./verify-cross-platform.sh

Metrics

Metric Target Description
Hash stability 100% All runs produce identical hash
Replay success 100% All replays match original
Cross-platform parity 100% Same hash across OS/arch

Integration with CI

# .gitea/workflows/bench-determinism.yaml
name: Determinism Benchmark
on:
  push:
    paths:
      - 'src/__Libraries/StellaOps.Canonical.Json/**'
      - 'src/Scanner/__Libraries/StellaOps.Scanner.Core/**'
      - 'bench/determinism/**'

jobs:
  determinism:
    strategy:
      matrix:
        os: [ubuntu-latest, windows-latest, macos-latest]
    runs-on: ${{ matrix.os }}
    steps:
      - uses: actions/checkout@v4
      - name: Run Determinism Tests
        run: dotnet test tests/StellaOps.Determinism.Tests
      - name: Capture Hashes
        run: ./bench/determinism/capture-hashes.sh
      - name: Upload Hashes
        uses: actions/upload-artifact@v4
        with:
          name: hashes-${{ matrix.os }}
          path: bench/determinism/results/