docs consolidation, big sln build fixes, new advisories and sprints/tasks

2026-01-05 18:37:04 +02:00
parent d0a7b88398
commit d7bdca6d97
175 changed files with 10322 additions and 307 deletions
--- a/docs/modules/replay/replay-proof-schema.md
+++ b/docs/modules/replay/replay-proof-schema.md
@@ -0,0 +1,606 @@
+# Replay Proof Schema
+
+> **Ownership:** Replay Guild, Scanner Guild, Attestor Guild
+> **Audience:** Service owners, platform engineers, auditors, compliance teams
+> **Related:** [Platform Architecture](../platform/architecture-overview.md), [Replay Architecture](./architecture.md), [Facet Sealing](../facet/architecture.md), [DSSE Specification](https://github.com/secure-systems-lab/dsse)
+
+This document defines the schema for Replay Proofs - compact, cryptographically verifiable artifacts that attest to deterministic policy evaluation outcomes.
+
+---
+
+## 1. Overview
+
+A **Replay Proof** is a DSSE-signed artifact that proves a policy evaluation produced a specific verdict given a specific set of inputs. Replay proofs enable:
+
+- **Audit trails**: Compact proof that a verdict was computed correctly
+- **Determinism verification**: Re-running with same inputs produces identical output
+- **Time-travel debugging**: Understand why a past decision was made
+- **Compliance evidence**: Cryptographic proof for regulatory requirements
+
+---
+
+## 2. Replay Bundle Structure
+
+A complete replay bundle consists of three artifacts stored in CAS:
+
+```
+cas://replay/<run-id>/
+    manifest.json           # DSSE-signed manifest (this document's focus)
+    inputbundle.tar.zst     # Compressed input artifacts
+    outputbundle.tar.zst    # Compressed output artifacts
+```
+
+### 2.1 Directory Layout
+
+```
+<run-id>/
+    manifest.json
+    inputbundle.tar.zst
+        feeds/
+            nvd/<date>.json
+            osv/<date>.json
+            ghsa/<date>.json
+        policy/
+            bundle.tar
+            version.json
+        sboms/
+            <sbom-id>.spdx.json
+            <sbom-id>.cdx.json
+        vex/
+            <vex-id>.openvex.json
+        config/
+            lattice.json
+            feature-flags.json
+        seeds/
+            random-seeds.json
+            clock-offsets.json
+    outputbundle.tar.zst
+        verdicts/
+            <verdict-id>.json
+        findings/
+            <finding-id>.json
+        merkle/
+            verdict-tree.json
+            finding-tree.json
+        logs/
+            replay.log
+            trace.json
+```
+
+---
+
+## 3. Core Schema Definitions
+
+### 3.1 ReplayProof
+
+The primary proof artifact - a compact summary suitable for verification:
+
+```csharp
+public sealed record ReplayProof
+{
+    // Identity
+    public required Guid ProofId { get; init; }
+    public required Guid RunId { get; init; }
+    public required string Subject { get; init; }           // Image digest or SBOM ID
+
+    // Input digest
+    public required KnowledgeSnapshotDigest InputDigest { get; init; }
+
+    // Output digest
+    public required VerdictDigest OutputDigest { get; init; }
+
+    // Execution metadata
+    public required ExecutionMetadata Execution { get; init; }
+
+    // CAS references
+    public required BundleReferences Bundles { get; init; }
+
+    // Signature
+    public required DateTimeOffset SignedAt { get; init; }
+    public required string SignedBy { get; init; }
+}
+```
+
+### 3.2 KnowledgeSnapshotDigest
+
+Cryptographic digest of all inputs:
+
+```csharp
+public sealed record KnowledgeSnapshotDigest
+{
+    // Component digests
+    public required string SbomsDigest { get; init; }           // SHA-256 of sorted SBOM hashes
+    public required string VexDigest { get; init; }             // SHA-256 of sorted VEX hashes
+    public required string FeedsDigest { get; init; }           // SHA-256 of feed version manifest
+    public required string PolicyDigest { get; init; }          // SHA-256 of policy bundle
+    public required string LatticeDigest { get; init; }         // SHA-256 of lattice config
+    public required string SeedsDigest { get; init; }           // SHA-256 of random seeds
+
+    // Combined root
+    public required string RootDigest { get; init; }            // SHA-256 of all component digests
+
+    // Counts for quick comparison
+    public required int SbomCount { get; init; }
+    public required int VexCount { get; init; }
+    public required int FeedCount { get; init; }
+}
+```
+
+### 3.3 VerdictDigest
+
+Cryptographic digest of all outputs:
+
+```csharp
+public sealed record VerdictDigest
+{
+    public required string VerdictMerkleRoot { get; init; }     // Merkle root of verdicts
+    public required string FindingMerkleRoot { get; init; }     // Merkle root of findings
+    public required int VerdictCount { get; init; }
+    public required int FindingCount { get; init; }
+    public required VerdictSummary Summary { get; init; }
+}
+
+public sealed record VerdictSummary
+{
+    public required int Critical { get; init; }
+    public required int High { get; init; }
+    public required int Medium { get; init; }
+    public required int Low { get; init; }
+    public required int Informational { get; init; }
+    public required int Suppressed { get; init; }
+    public required int Total { get; init; }
+}
+```
+
+### 3.4 ExecutionMetadata
+
+Execution environment and timing:
+
+```csharp
+public sealed record ExecutionMetadata
+{
+    // Timing
+    public required DateTimeOffset StartedAt { get; init; }
+    public required DateTimeOffset CompletedAt { get; init; }
+    public required long DurationMs { get; init; }
+
+    // Engine version
+    public required EngineVersion Engine { get; init; }
+
+    // Environment
+    public required string HostId { get; init; }
+    public required string RuntimeVersion { get; init; }        // e.g., ".NET 10.0.0"
+    public required string Platform { get; init; }              // e.g., "linux-x64"
+
+    // Determinism markers
+    public required bool DeterministicMode { get; init; }
+    public required string ClockMode { get; init; }             // "frozen", "simulated", "real"
+    public required string RandomMode { get; init; }            // "seeded", "recorded", "real"
+}
+
+public sealed record EngineVersion
+{
+    public required string Name { get; init; }                  // e.g., "PolicyEngine"
+    public required string Version { get; init; }               // e.g., "2.1.0"
+    public required string SourceDigest { get; init; }          // SHA-256 of engine source/binary
+}
+```
+
+### 3.5 BundleReferences
+
+CAS URIs to full bundles:
+
+```csharp
+public sealed record BundleReferences
+{
+    public required string ManifestUri { get; init; }           // cas://replay/<run-id>/manifest.json
+    public required string InputBundleUri { get; init; }        // cas://replay/<run-id>/inputbundle.tar.zst
+    public required string OutputBundleUri { get; init; }       // cas://replay/<run-id>/outputbundle.tar.zst
+    public required string ManifestDigest { get; init; }        // SHA-256 of manifest.json
+    public required string InputBundleDigest { get; init; }     // SHA-256 of inputbundle.tar.zst
+    public required string OutputBundleDigest { get; init; }    // SHA-256 of outputbundle.tar.zst
+    public required long InputBundleSize { get; init; }
+    public required long OutputBundleSize { get; init; }
+}
+```
+
+---
+
+## 4. DSSE Envelope
+
+Replay proofs are wrapped in DSSE envelopes for cryptographic binding:
+
+```json
+{
+  "payloadType": "application/vnd.stellaops.replay-proof.v1+json",
+  "payload": "<base64url-encoded canonical JSON>",
+  "signatures": [
+    {
+      "keyid": "sha256:abc123...",
+      "sig": "<base64url-encoded signature>"
+    }
+  ]
+}
+```
+
+### 4.1 Payload Type URI
+
+- **v1**: `application/vnd.stellaops.replay-proof.v1+json`
+- **in-toto compatible**: `https://stellaops.io/ReplayProof/v1`
+
+### 4.2 Canonical JSON Encoding
+
+Payloads MUST be encoded using RFC 8785 canonical JSON:
+
+1. Keys sorted lexicographically using Unicode code points
+2. No whitespace between structural characters
+3. No trailing commas
+4. Numbers without unnecessary decimal points or exponents
+5. Strings with minimal escaping (only required characters)
+
+---
+
+## 5. Full Manifest Schema
+
+The `manifest.json` file contains the complete proof plus additional metadata:
+
+```json
+{
+  "_type": "https://stellaops.io/ReplayManifest/v1",
+  "proofId": "550e8400-e29b-41d4-a716-446655440000",
+  "runId": "660e8400-e29b-41d4-a716-446655440001",
+  "subject": "sha256:abc123def456...",
+  "tenant": "acme-corp",
+  "inputDigest": {
+    "sbomsDigest": "sha256:111...",
+    "vexDigest": "sha256:222...",
+    "feedsDigest": "sha256:333...",
+    "policyDigest": "sha256:444...",
+    "latticeDigest": "sha256:555...",
+    "seedsDigest": "sha256:666...",
+    "rootDigest": "sha256:aaa...",
+    "sbomCount": 1,
+    "vexCount": 5,
+    "feedCount": 3
+  },
+  "outputDigest": {
+    "verdictMerkleRoot": "sha256:bbb...",
+    "findingMerkleRoot": "sha256:ccc...",
+    "verdictCount": 42,
+    "findingCount": 156,
+    "summary": {
+      "critical": 2,
+      "high": 8,
+      "medium": 25,
+      "low": 12,
+      "informational": 3,
+      "suppressed": 106,
+      "total": 156
+    }
+  },
+  "execution": {
+    "startedAt": "2026-01-05T10:00:00.000Z",
+    "completedAt": "2026-01-05T10:00:05.123Z",
+    "durationMs": 5123,
+    "engine": {
+      "name": "PolicyEngine",
+      "version": "2.1.0",
+      "sourceDigest": "sha256:engine123..."
+    },
+    "hostId": "scanner-worker-01",
+    "runtimeVersion": ".NET 10.0.0",
+    "platform": "linux-x64",
+    "deterministicMode": true,
+    "clockMode": "frozen",
+    "randomMode": "seeded"
+  },
+  "bundles": {
+    "manifestUri": "cas://replay/660e8400.../manifest.json",
+    "inputBundleUri": "cas://replay/660e8400.../inputbundle.tar.zst",
+    "outputBundleUri": "cas://replay/660e8400.../outputbundle.tar.zst",
+    "manifestDigest": "sha256:manifest...",
+    "inputBundleDigest": "sha256:input...",
+    "outputBundleDigest": "sha256:output...",
+    "inputBundleSize": 10485760,
+    "outputBundleSize": 2097152
+  },
+  "signedAt": "2026-01-05T10:00:06.000Z",
+  "signedBy": "scanner-worker-01"
+}
+```
+
+---
+
+## 6. Verification Protocol
+
+### 6.1 Quick Verification (Proof Only)
+
+Verify the DSSE signature and check digest consistency:
+
+```csharp
+public async Task<VerificationResult> VerifyProofAsync(
+    ReplayProof proof,
+    DsseEnvelope envelope,
+    CancellationToken ct)
+{
+    // 1. Verify DSSE signature
+    var sigValid = await _dsseVerifier.VerifyAsync(envelope, ct);
+    if (!sigValid)
+        return VerificationResult.Failed("DSSE signature invalid");
+
+    // 2. Verify input digest consistency
+    var inputRoot = ComputeInputRoot(
+        proof.InputDigest.SbomsDigest,
+        proof.InputDigest.VexDigest,
+        proof.InputDigest.FeedsDigest,
+        proof.InputDigest.PolicyDigest,
+        proof.InputDigest.LatticeDigest,
+        proof.InputDigest.SeedsDigest);
+
+    if (inputRoot != proof.InputDigest.RootDigest)
+        return VerificationResult.Failed("Input root digest mismatch");
+
+    return VerificationResult.Passed();
+}
+```
+
+### 6.2 Full Verification (With Replay)
+
+Download bundles and re-execute to verify determinism:
+
+```csharp
+public async Task<VerificationResult> VerifyWithReplayAsync(
+    ReplayProof proof,
+    CancellationToken ct)
+{
+    // 1. Quick verification first
+    var quickResult = await VerifyProofAsync(proof, envelope, ct);
+    if (!quickResult.Passed)
+        return quickResult;
+
+    // 2. Download bundles from CAS
+    var inputBundle = await _cas.DownloadAsync(proof.Bundles.InputBundleUri, ct);
+    var outputBundle = await _cas.DownloadAsync(proof.Bundles.OutputBundleUri, ct);
+
+    // 3. Verify bundle digests
+    if (ComputeDigest(inputBundle) != proof.Bundles.InputBundleDigest)
+        return VerificationResult.Failed("Input bundle digest mismatch");
+    if (ComputeDigest(outputBundle) != proof.Bundles.OutputBundleDigest)
+        return VerificationResult.Failed("Output bundle digest mismatch");
+
+    // 4. Extract and verify individual input digests
+    var inputs = await ExtractInputsAsync(inputBundle, ct);
+    var computedInputDigest = ComputeKnowledgeDigest(inputs);
+    if (computedInputDigest.RootDigest != proof.InputDigest.RootDigest)
+        return VerificationResult.Failed("Computed input digest mismatch");
+
+    // 5. Re-execute policy evaluation
+    var replayResult = await _replayEngine.ExecuteAsync(inputs, ct);
+
+    // 6. Compare output digests
+    var computedOutputDigest = ComputeVerdictDigest(replayResult);
+    if (computedOutputDigest.VerdictMerkleRoot != proof.OutputDigest.VerdictMerkleRoot)
+        return VerificationResult.Failed("Verdict Merkle root mismatch - non-deterministic!");
+
+    if (computedOutputDigest.FindingMerkleRoot != proof.OutputDigest.FindingMerkleRoot)
+        return VerificationResult.Failed("Finding Merkle root mismatch - non-deterministic!");
+
+    return VerificationResult.Passed();
+}
+```
+
+---
+
+## 7. Digest Computation
+
+### 7.1 Input Root Digest
+
+```csharp
+public string ComputeInputRoot(
+    string sbomsDigest,
+    string vexDigest,
+    string feedsDigest,
+    string policyDigest,
+    string latticeDigest,
+    string seedsDigest)
+{
+    // Concatenate in fixed order with separators
+    var combined = string.Join("|",
+        sbomsDigest,
+        vexDigest,
+        feedsDigest,
+        policyDigest,
+        latticeDigest,
+        seedsDigest);
+
+    return ComputeSha256(combined);
+}
+```
+
+### 7.2 SBOM Collection Digest
+
+```csharp
+public string ComputeSbomsDigest(IEnumerable<SbomRef> sboms)
+{
+    // Sort by ID for determinism
+    var sorted = sboms.OrderBy(s => s.SbomId, StringComparer.Ordinal);
+
+    // Concatenate hashes
+    var combined = string.Join("|", sorted.Select(s => s.ContentHash));
+
+    return ComputeSha256(combined);
+}
+```
+
+### 7.3 Verdict Merkle Root
+
+```csharp
+public string ComputeVerdictMerkleRoot(IEnumerable<Verdict> verdicts)
+{
+    // Sort by verdict ID for determinism
+    var sorted = verdicts.OrderBy(v => v.VerdictId, StringComparer.Ordinal);
+
+    // Compute leaf hashes
+    var leaves = sorted.Select(v => ComputeVerdictLeafHash(v)).ToArray();
+
+    // Build Merkle tree
+    return MerkleTreeBuilder.ComputeRoot(leaves);
+}
+
+private string ComputeVerdictLeafHash(Verdict verdict)
+{
+    var canonical = CanonicalJsonSerializer.Serialize(verdict);
+    return ComputeSha256(canonical);
+}
+```
+
+---
+
+## 8. Database Schema
+
+```sql
+-- Replay proof storage
+CREATE TABLE replay_proofs (
+    proof_id            UUID PRIMARY KEY,
+    run_id              UUID NOT NULL,
+    tenant              TEXT NOT NULL,
+    subject             TEXT NOT NULL,
+    input_root_digest   TEXT NOT NULL,
+    output_verdict_root TEXT NOT NULL,
+    output_finding_root TEXT NOT NULL,
+    execution_json      JSONB NOT NULL,
+    bundles_json        JSONB NOT NULL,
+    dsse_envelope       JSONB NOT NULL,
+    signed_at           TIMESTAMPTZ NOT NULL,
+    signed_by           TEXT NOT NULL,
+    created_at          TIMESTAMPTZ NOT NULL DEFAULT NOW(),
+
+    CONSTRAINT uq_replay_run UNIQUE (run_id)
+);
+
+CREATE INDEX ix_replay_proofs_tenant ON replay_proofs (tenant, created_at DESC);
+CREATE INDEX ix_replay_proofs_subject ON replay_proofs (subject);
+CREATE INDEX ix_replay_proofs_input ON replay_proofs (input_root_digest);
+
+-- Replay verification log
+CREATE TABLE replay_verifications (
+    verification_id     UUID PRIMARY KEY,
+    proof_id            UUID NOT NULL REFERENCES replay_proofs(proof_id),
+    tenant              TEXT NOT NULL,
+    verification_type   TEXT NOT NULL,     -- 'quick', 'full'
+    passed              BOOLEAN NOT NULL,
+    failure_reason      TEXT,
+    duration_ms         BIGINT NOT NULL,
+    verified_at         TIMESTAMPTZ NOT NULL,
+    verified_by         TEXT NOT NULL,
+
+    CONSTRAINT fk_proof FOREIGN KEY (proof_id) REFERENCES replay_proofs(proof_id)
+);
+
+CREATE INDEX ix_replay_verifications_proof ON replay_verifications (proof_id);
+```
+
+---
+
+## 9. CLI Integration
+
+```bash
+# Verify a replay proof (quick - signature only)
+stella verify --proof proof.json
+
+# Verify with full replay
+stella verify --proof proof.json --replay
+
+# Verify from CAS URI
+stella verify --bundle cas://replay/660e8400.../manifest.json
+
+# Export proof for audit
+stella replay export --run-id 660e8400-... --output proof.json
+
+# List proofs for an image
+stella replay list --subject sha256:abc123...
+
+# Diff two replay results
+stella replay diff --run-id-a 660e8400... --run-id-b 770e8400...
+```
+
+---
+
+## 10. API Endpoints
+
+```http
+# Get proof by run ID
+GET /api/v1/replay/{runId}/proof
+Response: ReplayProof (JSON)
+
+# Verify proof
+POST /api/v1/replay/{runId}/verify
+Request: { "type": "quick" | "full" }
+Response: VerificationResult
+
+# List proofs for subject
+GET /api/v1/replay/proofs?subject={digest}&tenant={tenant}
+Response: ReplayProofSummary[]
+
+# Download bundle
+GET /api/v1/replay/{runId}/bundles/{type}
+Response: Binary stream (tar.zst)
+
+# Compare two runs
+GET /api/v1/replay/diff?runIdA={id}&runIdB={id}
+Response: ReplayDiffResult
+```
+
+---
+
+## 11. Error Codes
+
+| Code | Description |
+|------|-------------|
+| `REPLAY_001` | Proof not found |
+| `REPLAY_002` | DSSE signature verification failed |
+| `REPLAY_003` | Input digest mismatch |
+| `REPLAY_004` | Output digest mismatch (non-deterministic) |
+| `REPLAY_005` | Bundle not found in CAS |
+| `REPLAY_006` | Bundle digest mismatch |
+| `REPLAY_007` | Engine version mismatch |
+| `REPLAY_008` | Replay execution failed |
+| `REPLAY_009` | Insufficient permissions |
+| `REPLAY_010` | Bundle format invalid |
+
+---
+
+## 12. Migration from v0
+
+If upgrading from pre-v1 replay bundles:
+
+1. **Schema migration**: Run `migrate-replay-schema.sql`
+2. **Re-sign existing proofs**: Use `stella replay migrate --sign` to add DSSE envelopes
+3. **Verify migration**: Run `stella replay verify --all` to check integrity
+4. **Update consumers**: Point to new `/api/v1/replay` endpoints
+
+---
+
+## 13. Security Considerations
+
+1. **Key Management**: Signing keys managed by Authority service with rotation support
+2. **Tenant Isolation**: Proofs scoped to tenants; cross-tenant access prohibited
+3. **Integrity**: All digests use SHA-256; Merkle proofs enable partial verification
+4. **Immutability**: Proofs cannot be modified once signed
+5. **Audit**: All verification attempts logged with correlation IDs
+6. **Air-gap**: Proofs and bundles can be exported for offline verification
+
+---
+
+## 14. References
+
+- [DSSE Specification](https://github.com/secure-systems-lab/dsse)
+- [RFC 8785 - JSON Canonicalization](https://tools.ietf.org/html/rfc8785)
+- [in-toto Attestation Framework](https://github.com/in-toto/attestation)
+- [SLSA Provenance](https://slsa.dev/provenance)
+- [Platform Architecture](../platform/architecture-overview.md)
+- [Facet Sealing Architecture](../facet/architecture.md)
+
+---
+
+*Last updated: 2026-01-05*