951a38d561
- Implemented CanonJson class for deterministic JSON serialization and hashing. - Added unit tests for CanonJson functionality, covering various scenarios including key sorting, handling of nested objects, arrays, and special characters. - Created project files for the Canonical JSON library and its tests, including necessary package references. - Added README.md for library usage and API reference. - Introduced RabbitMqIntegrationFactAttribute for conditional RabbitMQ integration tests.
130 lines
2.9 KiB
Markdown
130 lines
2.9 KiB
Markdown
# Determinism Benchmark Suite
|
|
|
|
> **Purpose:** Verify that StellaOps produces bit-identical results across replays.
|
|
> **Status:** Active
|
|
> **Sprint:** SPRINT_3850_0001_0001 (Competitive Gap Closure)
|
|
|
|
## Overview
|
|
|
|
Determinism is a core differentiator for StellaOps:
|
|
- Same inputs → same outputs (bit-identical)
|
|
- Replay manifests enable audit verification
|
|
- No hidden state or environment leakage
|
|
|
|
## What Gets Tested
|
|
|
|
### Canonical JSON
|
|
- Object key ordering (alphabetical)
|
|
- Number formatting consistency
|
|
- UTF-8 encoding without BOM
|
|
- No whitespace variation
|
|
|
|
### Scan Manifests
|
|
- Same artifact + same feeds → same manifest hash
|
|
- Seed values propagate correctly
|
|
- Timestamp handling (fixed UTC)
|
|
|
|
### Proof Bundles
|
|
- Root hash computation
|
|
- DSSE envelope determinism
|
|
- ProofLedger node ordering
|
|
|
|
### Score Computation
|
|
- Same manifest → same score
|
|
- Lattice merge is associative/commutative
|
|
- Policy rule ordering doesn't affect outcome
|
|
|
|
## Test Cases
|
|
|
|
### TC-001: Canonical JSON Determinism
|
|
|
|
```bash
|
|
# Run same object through CanonJson 100 times
|
|
# All hashes must match
|
|
```
|
|
|
|
### TC-002: Manifest Hash Stability
|
|
|
|
```bash
|
|
# Create manifest with identical inputs
|
|
# Verify ComputeHash() returns same value
|
|
```
|
|
|
|
### TC-003: Cross-Platform Determinism
|
|
|
|
```bash
|
|
# Run on Linux, Windows, macOS
|
|
# Compare output hashes
|
|
```
|
|
|
|
### TC-004: Feed Snapshot Determinism
|
|
|
|
```bash
|
|
# Same feed snapshot hash → same scan results
|
|
```
|
|
|
|
## Fixtures
|
|
|
|
```
|
|
fixtures/
|
|
├── sample-manifest.json
|
|
├── sample-ledger.json
|
|
├── expected-hashes.json
|
|
└── cross-platform/
|
|
├── linux-x64.hashes.json
|
|
├── windows-x64.hashes.json
|
|
└── macos-arm64.hashes.json
|
|
```
|
|
|
|
## Running the Suite
|
|
|
|
```bash
|
|
# Run determinism tests
|
|
dotnet test tests/StellaOps.Determinism.Tests
|
|
|
|
# Run replay verification
|
|
./run-replay.sh --manifest fixtures/sample-manifest.json --runs 10
|
|
|
|
# Cross-platform verification (requires CI matrix)
|
|
./verify-cross-platform.sh
|
|
```
|
|
|
|
## Metrics
|
|
|
|
| Metric | Target | Description |
|
|
|--------|--------|-------------|
|
|
| Hash stability | 100% | All runs produce identical hash |
|
|
| Replay success | 100% | All replays match original |
|
|
| Cross-platform parity | 100% | Same hash across OS/arch |
|
|
|
|
## Integration with CI
|
|
|
|
```yaml
|
|
# .gitea/workflows/bench-determinism.yaml
|
|
name: Determinism Benchmark
|
|
on:
|
|
push:
|
|
paths:
|
|
- 'src/__Libraries/StellaOps.Canonical.Json/**'
|
|
- 'src/Scanner/__Libraries/StellaOps.Scanner.Core/**'
|
|
- 'bench/determinism/**'
|
|
|
|
jobs:
|
|
determinism:
|
|
strategy:
|
|
matrix:
|
|
os: [ubuntu-latest, windows-latest, macos-latest]
|
|
runs-on: ${{ matrix.os }}
|
|
steps:
|
|
- uses: actions/checkout@v4
|
|
- name: Run Determinism Tests
|
|
run: dotnet test tests/StellaOps.Determinism.Tests
|
|
- name: Capture Hashes
|
|
run: ./bench/determinism/capture-hashes.sh
|
|
- name: Upload Hashes
|
|
uses: actions/upload-artifact@v4
|
|
with:
|
|
name: hashes-${{ matrix.os }}
|
|
path: bench/determinism/results/
|
|
```
|