# Rekor Verification Technical Design **Document ID**: DOCS-ATTEST-REKOR-001 **Version**: 1.0 **Last Updated**: 2025-12-14 **Status**: Draft --- ## 1. OVERVIEW This document provides the comprehensive technical design for Rekor transparency log verification in StellaOps. It covers three key capabilities: 1. **Merkle Proof Verification** - Cryptographic verification of inclusion proofs 2. **Durable Retry Queue** - Reliable submission with failure recovery 3. **Time Skew Validation** - Replay protection via timestamp validation ### Related Sprints | Sprint | Priority | Description | |--------|----------|-------------| | SPRINT_3000_0001_0001 | P0 | Merkle Proof Verification | | SPRINT_3000_0001_0002 | P1 | Rekor Retry Queue & Metrics | | SPRINT_3000_0001_0003 | P2 | Time Skew Validation | --- ## 2. ARCHITECTURE CONTEXT ### Current State ``` ┌─────────────────────────────────────────────────────────────────────┐ │ Attestor Module │ ├─────────────────────────────────────────────────────────────────────┤ │ │ │ ┌─────────────────────┐ ┌─────────────────────┐ │ │ │ AttestorSubmission │───►│ IRekorClient │ │ │ │ Service │ │ (HttpRekorClient) │ │ │ └─────────────────────┘ └──────────┬──────────┘ │ │ │ │ │ │ ▼ ▼ │ │ ┌─────────────────────┐ ┌─────────────────────┐ │ │ │ IAttestorEntry │ │ Rekor API │ │ │ │ Repository │ │ (External) │ │ │ └─────────────────────┘ └─────────────────────┘ │ │ │ │ Current Limitations: │ │ ✗ Stores proofs but doesn't verify them cryptographically │ │ ✗ Failed submissions are lost (no retry) │ │ ✗ No integrated_time validation │ │ │ └─────────────────────────────────────────────────────────────────────┘ ``` ### Target State ``` ┌─────────────────────────────────────────────────────────────────────┐ │ Attestor Module (Enhanced) │ ├─────────────────────────────────────────────────────────────────────┤ │ │ │ ┌─────────────────────┐ ┌─────────────────────┐ │ │ │ AttestorSubmission │───►│ IRekorClient │ │ │ │ Service │ │ + VerifyInclusion │◄──┐ │ │ └─────────┬───────────┘ └──────────┬──────────┘ │ │ │ │ │ │ │ │ │ (on failure) │ │ │ │ ▼ ▼ │ │ │ ┌─────────────────────┐ ┌─────────────────────┐ │ │ │ │ IRekorSubmission │ │ MerkleProofVerifier │ │ │ │ │ Queue (PostgreSQL) │ │ CheckpointVerifier │ │ │ │ └─────────┬───────────┘ └─────────────────────┘ │ │ │ │ │ │ │ ▼ │ │ │ ┌─────────────────────┐ ┌─────────────────────┐ │ │ │ │ RekorRetryWorker │───►│ Rekor API │───┘ │ │ │ (Background) │ │ (External) │ │ │ └─────────────────────┘ └─────────────────────┘ │ │ │ │ ┌─────────────────────┐ ┌─────────────────────┐ │ │ │ AttestorVerification│───►│ ITimeSkewValidator │ │ │ │ Service │ │ (integrated_time) │ │ │ └─────────────────────┘ └─────────────────────┘ │ │ │ │ Enhancements: │ │ ✓ Cryptographic Merkle proof verification │ │ ✓ Durable retry queue with exponential backoff │ │ ✓ Time skew detection and alerting │ │ │ └─────────────────────────────────────────────────────────────────────┘ ``` --- ## 3. COMPONENT DESIGN ### 3.1 Merkle Proof Verification #### 3.1.1 Algorithm Rekor uses RFC 6962 (Certificate Transparency) Merkle tree structure: ``` Root Hash / \ / \ Hash(0,1) Hash(2,3) / \ / \ H(0) H(1) H(2) H(3) │ │ │ │ Leaf0 Leaf1 Leaf2 Leaf3 ``` **Leaf Hash Computation:** ``` leafHash = SHA256(0x00 || RFC6962_Entry) ``` **Interior Node Computation:** ``` interiorHash = SHA256(0x01 || leftChild || rightChild) ``` **Inclusion Proof Verification:** Given: - `leafIndex`: Position of leaf in tree - `treeSize`: Total number of leaves - `proofPath[]`: Sibling hashes along path to root - `expectedRoot`: Root hash from checkpoint ```python def verify_inclusion(leaf_hash, leaf_index, tree_size, proof_path, expected_root): current_hash = leaf_hash current_index = leaf_index remaining_size = tree_size for sibling_hash in proof_path: if current_index % 2 == 1: # Current node is right child current_hash = sha256(0x01 || sibling_hash || current_hash) else: # Current node is left child current_hash = sha256(0x01 || current_hash || sibling_hash) current_index = current_index // 2 remaining_size = (remaining_size + 1) // 2 return current_hash == expected_root ``` #### 3.1.2 Checkpoint Verification Rekor checkpoints are signed using the log's private key: ``` Checkpoint Format: ───────────────── rekor.sigstore.dev - 1234567 — rekor.sigstore.dev ``` Verification steps: 1. Parse checkpoint text format 2. Extract signature and public key hint 3. Verify Ed25519/ECDSA signature over checkpoint body 4. Extract root hash and tree size #### 3.1.3 Implementation Classes ```csharp /// /// RFC 6962 Merkle proof verification. /// public static class MerkleProofVerifier { private static readonly byte LeafPrefix = 0x00; private static readonly byte NodePrefix = 0x01; public static bool VerifyInclusion( byte[] leafHash, long leafIndex, long treeSize, IReadOnlyList proofPath, byte[] expectedRoot) { ArgumentNullException.ThrowIfNull(leafHash); ArgumentNullException.ThrowIfNull(proofPath); ArgumentNullException.ThrowIfNull(expectedRoot); if (leafHash.Length != 32 || expectedRoot.Length != 32) throw new ArgumentException("Hash must be 32 bytes (SHA-256)"); if (leafIndex < 0 || leafIndex >= treeSize) throw new ArgumentOutOfRangeException(nameof(leafIndex)); var currentHash = leafHash; var currentIndex = leafIndex; var currentSize = treeSize; foreach (var siblingHash in proofPath) { if (siblingHash.Length != 32) throw new ArgumentException("Sibling hash must be 32 bytes"); if (currentIndex % 2 == 1) { // Current is right child currentHash = HashInterior(siblingHash, currentHash); } else { // Current is left child currentHash = HashInterior(currentHash, siblingHash); } currentIndex /= 2; currentSize = (currentSize + 1) / 2; } return currentHash.AsSpan().SequenceEqual(expectedRoot); } private static byte[] HashInterior(byte[] left, byte[] right) { Span buffer = stackalloc byte[1 + 32 + 32]; buffer[0] = NodePrefix; left.CopyTo(buffer.Slice(1, 32)); right.CopyTo(buffer.Slice(33, 32)); return SHA256.HashData(buffer); } public static byte[] ComputeLeafHash(byte[] entryData) { Span buffer = stackalloc byte[1 + entryData.Length]; buffer[0] = LeafPrefix; entryData.CopyTo(buffer.Slice(1)); return SHA256.HashData(buffer); } } ``` ### 3.2 Durable Retry Queue #### 3.2.1 State Machine ``` PENDING │ │ Worker picks up ▼ SUBMITTING / \ / \ (success) (failure) / \ ▼ ▼ SUBMITTED RETRYING │ │ (after backoff delay) ▼ SUBMITTING │ │ (max attempts exceeded) ▼ DEAD_LETTER ``` #### 3.2.2 Exponential Backoff ```csharp public static TimeSpan CalculateBackoff(int attemptCount, RekorQueueOptions options) { var delayMs = options.InitialDelayMs * Math.Pow(options.BackoffMultiplier, attemptCount - 1); var cappedDelayMs = Math.Min(delayMs, options.MaxDelayMs); // Add jitter (±10%) var jitter = Random.Shared.NextDouble() * 0.2 - 0.1; var finalDelayMs = cappedDelayMs * (1 + jitter); return TimeSpan.FromMilliseconds(finalDelayMs); } ``` **Default backoff sequence:** | Attempt | Base Delay | With Jitter | |---------|------------|-------------| | 1 | 1s | 0.9s - 1.1s | | 2 | 2s | 1.8s - 2.2s | | 3 | 4s | 3.6s - 4.4s | | 4 | 8s | 7.2s - 8.8s | | 5 | 16s | 14.4s - 17.6s | #### 3.2.3 Queue Table Design ```sql CREATE TABLE attestor_rekor_queue ( id UUID PRIMARY KEY, tenant_id TEXT NOT NULL, bundle_sha256 TEXT NOT NULL UNIQUE, -- Idempotency key dsse_payload BYTEA NOT NULL, backend TEXT NOT NULL, status TEXT NOT NULL, attempt_count INT NOT NULL DEFAULT 0, max_attempts INT NOT NULL, last_attempt_at TIMESTAMPTZ, last_error TEXT, next_retry_at TIMESTAMPTZ, rekor_uuid TEXT, -- Set on success rekor_log_index BIGINT, -- Set on success created_at TIMESTAMPTZ NOT NULL, updated_at TIMESTAMPTZ NOT NULL ); -- Efficient dequeue query CREATE INDEX idx_rekor_queue_dequeue ON attestor_rekor_queue (next_retry_at, status) WHERE status IN ('pending', 'retrying'); ``` #### 3.2.4 Dequeue Query ```sql -- Atomic dequeue with row locking WITH eligible AS ( SELECT id FROM attestor_rekor_queue WHERE status IN ('pending', 'retrying') AND (next_retry_at IS NULL OR next_retry_at <= NOW()) ORDER BY next_retry_at NULLS FIRST, created_at LIMIT :batch_size FOR UPDATE SKIP LOCKED ) UPDATE attestor_rekor_queue q SET status = 'submitting', updated_at = NOW() FROM eligible e WHERE q.id = e.id RETURNING q.*; ``` ### 3.3 Time Skew Validation #### 3.3.1 Threat Model | Attack | Description | Detection | |--------|-------------|-----------| | Backdated Entry | Attacker inserts entry with old timestamp | Large positive skew | | Future Timestamp | Attacker pre-dates entry | Negative skew (future) | | Log Manipulation | Attacker modifies existing entries | Timestamp inconsistency | #### 3.3.2 Threshold Design ``` Time Skew Detection Zones ───────────────────────────────────────────────────────────────── │ REJECT │ WARN │ OK │ WARN │ REJECT │ │ FUTURE │ FUTURE │ │ PAST │ PAST │ ───────────────────────────────────────────────────────────────── ◄───────────────────────┼───────────────────────► -60s -5m NOW +5m +1h (local time) Default Thresholds: • Future tolerance: 60 seconds (beyond = reject) • Warn threshold: 5 minutes • Reject threshold: 1 hour ``` #### 3.3.3 Validation Flow ```csharp public TimeSkewResult Validate(DateTimeOffset integratedTime, DateTimeOffset localTime) { var skew = integratedTime - localTime; // Future timestamps are highly suspicious if (skew > TimeSpan.Zero) { if (skew > _options.FutureTolerance) { return Rejected($"Future timestamp by {skew}"); } return Ok(skew); // Within future tolerance } // Past timestamps (normal case - Rekor time is in the past) var absSkew = skew.Duration(); if (absSkew >= _options.RejectThreshold) { return Rejected($"Time skew {absSkew} exceeds reject threshold"); } if (absSkew >= _options.WarnThreshold) { return Warning($"Time skew {absSkew} exceeds warn threshold"); } return Ok(skew); } ``` --- ## 4. DATA FLOW ### 4.1 Submission Flow (with Queue) ``` ┌─────────┐ ┌──────────────────┐ ┌───────────┐ │ Client │─────►│ SubmissionService │─────►│ RekorAPI │ └─────────┘ └────────┬─────────┘ └─────┬─────┘ │ │ │ (success) │ ▼ │ ┌─────────────────┐ │ │ Store Entry │◄─────────────┘ │ (status=ok) │ └─────────────────┘ │ (failure) ▼ ┌─────────────────┐ │ Enqueue │ │ (status=pending)│ └────────┬────────┘ │ ▼ ┌─────────────────┐ ┌───────────┐ │ RetryWorker │─────►│ RekorAPI │ │ (background) │ └─────┬─────┘ └─────────────────┘ │ ▲ │ │ │ └─────────────────────┘ (retry loop) ``` ### 4.2 Verification Flow (with Proof Verification) ``` ┌─────────┐ ┌───────────────────┐ ┌─────────────────┐ │ Client │─────►│ VerificationSvc │─────►│ EntryRepository │ └─────────┘ └────────┬──────────┘ └────────┬────────┘ │ │ │◄─────────────────────────┘ │ (AttestorEntry) ▼ ┌─────────────────────┐ │ 1. TimeSkewValidator│ │ (integrated_time)│ └────────┬────────────┘ │ ▼ ┌─────────────────────┐ │ 2. MerkleProof │ │ Verifier │ └────────┬────────────┘ │ ▼ ┌─────────────────────┐ │ 3. Checkpoint │ │ Verifier │ └────────┬────────────┘ │ ▼ ┌─────────────────────┐ │ 4. Aggregate Result │ │ (VerificationRpt)│ └─────────────────────┘ ``` --- ## 5. CONFIGURATION REFERENCE ```yaml # attestor.yaml attestor: rekor: primary: url: https://rekor.sigstore.dev proof_timeout_ms: 15000 poll_interval_ms: 250 max_attempts: 60 mirror: enabled: false url: https://rekor-mirror.internal verification: public_key_path: /etc/stellaops/rekor-pub.pem # Or inline: # public_key_base64: LS0tLS1CRUdJTi... allow_offline_without_signature: false max_checkpoint_age_minutes: 60 queue: enabled: true max_attempts: 5 initial_delay_ms: 1000 max_delay_ms: 60000 backoff_multiplier: 2.0 batch_size: 10 poll_interval_ms: 5000 dead_letter_retention_days: 30 time_skew: enabled: true warn_threshold_seconds: 300 # 5 minutes reject_threshold_seconds: 3600 # 1 hour future_tolerance_seconds: 60 # 1 minute reject_future_timestamps: true skip_in_offline_mode: true ``` --- ## 6. METRICS REFERENCE | Metric | Type | Labels | Description | |--------|------|--------|-------------| | `attestor.inclusion_verify_total` | Counter | `result` | Inclusion proof verifications | | `attestor.inclusion_verify_latency_seconds` | Histogram | | Verification latency | | `attestor.checkpoint_verify_total` | Counter | `result` | Checkpoint signature verifications | | `attestor.rekor_queue_depth` | Gauge | | Current pending + retrying items | | `attestor.rekor_retry_attempts_total` | Counter | `backend`, `attempt` | Retry attempts | | `attestor.rekor_submission_status_total` | Counter | `status`, `backend` | Submission outcomes | | `attestor.rekor_queue_wait_seconds` | Histogram | | Time in queue before submission | | `attestor.time_skew_detected_total` | Counter | `severity`, `action` | Time skew detections | | `attestor.time_skew_seconds` | Histogram | | Observed skew distribution | --- ## 7. ERROR HANDLING ### 7.1 Error Taxonomy | Error Code | Description | Retry? | Action | |------------|-------------|--------|--------| | `rekor_unavailable` | Rekor API not reachable | Yes | Queue for retry | | `rekor_conflict` | Duplicate entry (409) | No | Retrieve existing entry | | `rekor_rate_limited` | Rate limit exceeded (429) | Yes | Backoff and retry | | `rekor_internal_error` | Server error (5xx) | Yes | Queue for retry | | `proof_invalid` | Merkle proof verification failed | No | Reject, log alert | | `checkpoint_signature_invalid` | Checkpoint signature failed | No | Reject, log alert | | `time_skew_rejected` | Time skew exceeds threshold | No | Reject, log warning | ### 7.2 Structured Logging ```json { "timestamp": "2025-12-14T10:30:00Z", "level": "Warning", "message": "Rekor submission failed, queuing for retry", "error_code": "rekor_unavailable", "bundle_sha256": "abc123...", "backend": "primary", "attempt_count": 1, "next_retry_at": "2025-12-14T10:30:02Z", "error_message": "Connection refused" } ``` --- ## 8. SECURITY CONSIDERATIONS ### 8.1 Key Management - Rekor public key must be distributed out-of-band - Support key rotation via versioned key configuration - Store keys in secure location (not in code/config) ### 8.2 Trust Model ``` Trust Hierarchy ─────────────── ┌─────────────┐ │ Rekor Root │ │ Public Key │ └──────┬──────┘ │ signs ▼ ┌─────────────┐ │ Checkpoint │ │ (root hash) │ └──────┬──────┘ │ commits to ▼ ┌─────────────┐ │ Merkle Tree │ │ (entries) │ └──────┬──────┘ │ includes ▼ ┌─────────────┐ │ Attestation │ │ (DSSE) │ └─────────────┘ ``` ### 8.3 Offline Security In air-gapped environments: - Checkpoint must be pre-distributed with offline bundle - Proof verification still works (no network needed) - Time skew validation should be skipped or use bundled reference time --- ## 9. TESTING STRATEGY ### 9.1 Test Categories | Category | Coverage Target | Tools | |----------|-----------------|-------| | Unit | >90% | xUnit, Moq | | Integration | >80% | Testcontainers (PostgreSQL) | | Contract | All public APIs | Snapshot testing | | Performance | Latency P99 | BenchmarkDotNet | ### 9.2 Golden Fixtures Obtain from public Sigstore Rekor instance: ```bash # Get a real Rekor entry for testing rekor-cli get --uuid 24296fb24b8ad77a... --format json > fixtures/rekor-entry-valid.json # Get checkpoint curl https://rekor.sigstore.dev/api/v1/log > fixtures/rekor-checkpoint.json # Get public key curl https://rekor.sigstore.dev/api/v1/log/publicKey > fixtures/rekor-pubkey.pem ``` --- ## 10. MIGRATION GUIDE ### 10.1 Database Migrations Run in order: 1. `00X_rekor_submission_queue.sql` - Queue table 2. Update `AttestorEntry` schema if stored in PostgreSQL ### 10.2 Configuration Migration ```yaml # Before (existing) attestor: rekor: primary: url: https://rekor.sigstore.dev # After (add new sections) attestor: rekor: primary: url: https://rekor.sigstore.dev verification: public_key_path: /etc/stellaops/rekor-pub.pem queue: enabled: true time_skew: enabled: true ``` ### 10.3 Rollback Plan - Queue table can be dropped if not needed - All new features are configurable (can disable) - No breaking changes to existing API contracts --- ## 11. REFERENCES - [RFC 6962: Certificate Transparency](https://datatracker.ietf.org/doc/html/rfc6962) - [Sigstore Rekor](https://github.com/sigstore/rekor) - [Transparency.dev Checkpoint Format](https://github.com/transparency-dev/formats) - [Advisory: Rekor Integration Technical Reference](../../../product-advisories/14-Dec-2025%20-%20Rekor%20Integration%20Technical%20Reference.md)