# Rekor Verification Technical Design

**Document ID**: DOCS-ATTEST-REKOR-001
**Version**: 2.0
**Last Updated**: 2026-01-13
**Status**: Draft

---

## 1. OVERVIEW

This document provides the comprehensive technical design for Rekor transparency log verification in StellaOps. It covers four key capabilities:

1. **Merkle Proof Verification** - Cryptographic verification of inclusion proofs
2. **Durable Retry Queue** - Reliable submission with failure recovery
3. **Time Skew Validation** - Replay protection via timestamp validation
4. **Tile-Based Verification (v2)** - Support for Rekor v2 Sunlight format

### Related Sprints

| Sprint | Priority | Description |
|--------|----------|-------------|
| SPRINT_3000_0001_0001 | P0 | Merkle Proof Verification |
| SPRINT_3000_0001_0002 | P1 | Rekor Retry Queue & Metrics |
| SPRINT_3000_0001_0003 | P2 | Time Skew Validation |
| SPRINT_3000_0001_0004 | P1 | Rekor v2 Tile-Based Verification |

---

## 2. ARCHITECTURE CONTEXT

### Current State

```
┌─────────────────────────────────────────────────────────────────────┐
│                        Attestor Module                               │
├─────────────────────────────────────────────────────────────────────┤
│                                                                      │
│  ┌─────────────────────┐    ┌─────────────────────┐                 │
│  │ AttestorSubmission  │───►│   IRekorClient      │                 │
│  │ Service             │    │   (HttpRekorClient) │                 │
│  └─────────────────────┘    └──────────┬──────────┘                 │
│           │                            │                             │
│           ▼                            ▼                             │
│  ┌─────────────────────┐    ┌─────────────────────┐                 │
│  │ IAttestorEntry      │    │   Rekor API         │                 │
│  │ Repository          │    │   (External)        │                 │
│  └─────────────────────┘    └─────────────────────┘                 │
│                                                                      │
│  Current Limitations:                                                │
│  ✗ Stores proofs but doesn't verify them cryptographically          │
│  ✗ Failed submissions are lost (no retry)                           │
│  ✗ No integrated_time validation                                    │
│                                                                      │
└─────────────────────────────────────────────────────────────────────┘
```

### Target State

```
┌─────────────────────────────────────────────────────────────────────┐
│                        Attestor Module (Enhanced)                    │
├─────────────────────────────────────────────────────────────────────┤
│                                                                      │
│  ┌─────────────────────┐    ┌─────────────────────┐                 │
│  │ AttestorSubmission  │───►│   IRekorClient      │                 │
│  │ Service             │    │   + VerifyInclusion │◄──┐             │
│  └─────────┬───────────┘    └──────────┬──────────┘   │             │
│            │                           │               │             │
│            │ (on failure)              │               │             │
│            ▼                           ▼               │             │
│  ┌─────────────────────┐    ┌─────────────────────┐   │             │
│  │ IRekorSubmission    │    │ MerkleProofVerifier │   │             │
│  │ Queue (PostgreSQL)  │    │ CheckpointVerifier  │   │             │
│  └─────────┬───────────┘    └─────────────────────┘   │             │
│            │                                           │             │
│            ▼                                           │             │
│  ┌─────────────────────┐    ┌─────────────────────┐   │             │
│  │ RekorRetryWorker    │───►│   Rekor API         │───┘             │
│  │ (Background)        │    │   (External)        │                 │
│  └─────────────────────┘    └─────────────────────┘                 │
│                                                                      │
│  ┌─────────────────────┐    ┌─────────────────────┐                 │
│  │ AttestorVerification│───►│ ITimeSkewValidator  │                 │
│  │ Service             │    │ (integrated_time)   │                 │
│  └─────────────────────┘    └─────────────────────┘                 │
│                                                                      │
│  Enhancements:                                                       │
│  ✓ Cryptographic Merkle proof verification                          │
│  ✓ Durable retry queue with exponential backoff                     │
│  ✓ Time skew detection and alerting                                 │
│                                                                      │
└─────────────────────────────────────────────────────────────────────┘
```

---

## 3. COMPONENT DESIGN

### 3.1 Merkle Proof Verification

#### 3.1.1 Algorithm

Rekor uses RFC 6962 (Certificate Transparency) Merkle tree structure:

```
                    Root Hash
                   /         \
                  /           \
            Hash(0,1)       Hash(2,3)
            /     \         /     \
         H(0)   H(1)     H(2)   H(3)
          │       │        │       │
        Leaf0   Leaf1    Leaf2   Leaf3
```

**Leaf Hash Computation:**
```
leafHash = SHA256(0x00 || RFC6962_Entry)
```

**Interior Node Computation:**
```
interiorHash = SHA256(0x01 || leftChild || rightChild)
```

**Inclusion Proof Verification:**

Given:
- `leafIndex`: Position of leaf in tree
- `treeSize`: Total number of leaves
- `proofPath[]`: Sibling hashes along path to root
- `expectedRoot`: Root hash from checkpoint

```python
def verify_inclusion(leaf_hash, leaf_index, tree_size, proof_path, expected_root):
    current_hash = leaf_hash
    current_index = leaf_index
    remaining_size = tree_size

    for sibling_hash in proof_path:
        if current_index % 2 == 1:
            # Current node is right child
            current_hash = sha256(0x01 || sibling_hash || current_hash)
        else:
            # Current node is left child
            current_hash = sha256(0x01 || current_hash || sibling_hash)

        current_index = current_index // 2
        remaining_size = (remaining_size + 1) // 2

    return current_hash == expected_root
```

#### 3.1.2 Checkpoint Verification

Rekor checkpoints are signed using the log's private key:

```
Checkpoint Format:
─────────────────
rekor.sigstore.dev - 1234567
<tree_size>
<root_hash_base64>

— rekor.sigstore.dev <signature>
```

Verification steps:
1. Parse checkpoint text format
2. Extract signature and public key hint
3. Verify Ed25519/ECDSA signature over checkpoint body
4. Extract root hash and tree size

#### 3.1.3 Implementation Classes

```csharp
/// <summary>
/// RFC 6962 Merkle proof verification.
/// </summary>
public static class MerkleProofVerifier
{
    private static readonly byte LeafPrefix = 0x00;
    private static readonly byte NodePrefix = 0x01;

    public static bool VerifyInclusion(
        byte[] leafHash,
        long leafIndex,
        long treeSize,
        IReadOnlyList<byte[]> proofPath,
        byte[] expectedRoot)
    {
        ArgumentNullException.ThrowIfNull(leafHash);
        ArgumentNullException.ThrowIfNull(proofPath);
        ArgumentNullException.ThrowIfNull(expectedRoot);

        if (leafHash.Length != 32 || expectedRoot.Length != 32)
            throw new ArgumentException("Hash must be 32 bytes (SHA-256)");

        if (leafIndex < 0 || leafIndex >= treeSize)
            throw new ArgumentOutOfRangeException(nameof(leafIndex));

        var currentHash = leafHash;
        var currentIndex = leafIndex;
        var currentSize = treeSize;

        foreach (var siblingHash in proofPath)
        {
            if (siblingHash.Length != 32)
                throw new ArgumentException("Sibling hash must be 32 bytes");

            if (currentIndex % 2 == 1)
            {
                // Current is right child
                currentHash = HashInterior(siblingHash, currentHash);
            }
            else
            {
                // Current is left child
                currentHash = HashInterior(currentHash, siblingHash);
            }

            currentIndex /= 2;
            currentSize = (currentSize + 1) / 2;
        }

        return currentHash.AsSpan().SequenceEqual(expectedRoot);
    }

    private static byte[] HashInterior(byte[] left, byte[] right)
    {
        Span<byte> buffer = stackalloc byte[1 + 32 + 32];
        buffer[0] = NodePrefix;
        left.CopyTo(buffer.Slice(1, 32));
        right.CopyTo(buffer.Slice(33, 32));
        return SHA256.HashData(buffer);
    }

    public static byte[] ComputeLeafHash(byte[] entryData)
    {
        Span<byte> buffer = stackalloc byte[1 + entryData.Length];
        buffer[0] = LeafPrefix;
        entryData.CopyTo(buffer.Slice(1));
        return SHA256.HashData(buffer);
    }
}
```

### 3.2 Durable Retry Queue

#### 3.2.1 State Machine

```
                         PENDING
                            │
                            │ Worker picks up
                            ▼
                       SUBMITTING
                      /          \
                     /            \
               (success)        (failure)
                  /                \
                 ▼                  ▼
            SUBMITTED            RETRYING
                                    │
                                    │ (after backoff delay)
                                    ▼
                               SUBMITTING
                                    │
                                    │ (max attempts exceeded)
                                    ▼
                              DEAD_LETTER
```

#### 3.2.2 Exponential Backoff

```csharp
public static TimeSpan CalculateBackoff(int attemptCount, RekorQueueOptions options)
{
    var delayMs = options.InitialDelayMs * Math.Pow(options.BackoffMultiplier, attemptCount - 1);
    var cappedDelayMs = Math.Min(delayMs, options.MaxDelayMs);

    // Add jitter (±10%)
    var jitter = Random.Shared.NextDouble() * 0.2 - 0.1;
    var finalDelayMs = cappedDelayMs * (1 + jitter);

    return TimeSpan.FromMilliseconds(finalDelayMs);
}
```

**Default backoff sequence:**
| Attempt | Base Delay | With Jitter |
|---------|------------|-------------|
| 1 | 1s | 0.9s - 1.1s |
| 2 | 2s | 1.8s - 2.2s |
| 3 | 4s | 3.6s - 4.4s |
| 4 | 8s | 7.2s - 8.8s |
| 5 | 16s | 14.4s - 17.6s |

#### 3.2.3 Queue Table Design

```sql
CREATE TABLE attestor_rekor_queue (
    id                  UUID PRIMARY KEY,
    tenant_id           TEXT NOT NULL,
    bundle_sha256       TEXT NOT NULL UNIQUE,  -- Idempotency key
    dsse_payload        BYTEA NOT NULL,
    backend             TEXT NOT NULL,
    status              TEXT NOT NULL,
    attempt_count       INT NOT NULL DEFAULT 0,
    max_attempts        INT NOT NULL,
    last_attempt_at     TIMESTAMPTZ,
    last_error          TEXT,
    next_retry_at       TIMESTAMPTZ,
    rekor_uuid          TEXT,                  -- Set on success
    rekor_log_index     BIGINT,                -- Set on success
    created_at          TIMESTAMPTZ NOT NULL,
    updated_at          TIMESTAMPTZ NOT NULL
);

-- Efficient dequeue query
CREATE INDEX idx_rekor_queue_dequeue
    ON attestor_rekor_queue (next_retry_at, status)
    WHERE status IN ('pending', 'retrying');
```

#### 3.2.4 Dequeue Query

```sql
-- Atomic dequeue with row locking
WITH eligible AS (
    SELECT id
    FROM attestor_rekor_queue
    WHERE status IN ('pending', 'retrying')
      AND (next_retry_at IS NULL OR next_retry_at <= NOW())
    ORDER BY next_retry_at NULLS FIRST, created_at
    LIMIT :batch_size
    FOR UPDATE SKIP LOCKED
)
UPDATE attestor_rekor_queue q
SET status = 'submitting',
    updated_at = NOW()
FROM eligible e
WHERE q.id = e.id
RETURNING q.*;
```

### 3.3 Time Skew Validation

#### 3.3.1 Threat Model

| Attack | Description | Detection |
|--------|-------------|-----------|
| Backdated Entry | Attacker inserts entry with old timestamp | Large positive skew |
| Future Timestamp | Attacker pre-dates entry | Negative skew (future) |
| Log Manipulation | Attacker modifies existing entries | Timestamp inconsistency |

#### 3.3.2 Threshold Design

```
                    Time Skew Detection Zones
    ─────────────────────────────────────────────────────────────────
    │ REJECT │   WARN   │       OK       │   WARN   │ REJECT │
    │ FUTURE │  FUTURE  │                │   PAST   │  PAST  │
    ─────────────────────────────────────────────────────────────────
    ◄───────────────────────┼───────────────────────►
         -60s     -5m       NOW      +5m      +1h
                                    (local time)

    Default Thresholds:
    • Future tolerance: 60 seconds (beyond = reject)
    • Warn threshold: 5 minutes
    • Reject threshold: 1 hour
```

#### 3.3.3 Validation Flow

```csharp
public TimeSkewResult Validate(DateTimeOffset integratedTime, DateTimeOffset localTime)
{
    var skew = integratedTime - localTime;

    // Future timestamps are highly suspicious
    if (skew > TimeSpan.Zero)
    {
        if (skew > _options.FutureTolerance)
        {
            return Rejected($"Future timestamp by {skew}");
        }
        return Ok(skew);  // Within future tolerance
    }

    // Past timestamps (normal case - Rekor time is in the past)
    var absSkew = skew.Duration();

    if (absSkew >= _options.RejectThreshold)
    {
        return Rejected($"Time skew {absSkew} exceeds reject threshold");
    }

    if (absSkew >= _options.WarnThreshold)
    {
        return Warning($"Time skew {absSkew} exceeds warn threshold");
    }

    return Ok(skew);
}
```

### 3.4 Tile-Based Verification (Rekor v2)

Rekor v2 introduces a tile-based log structure following the Sunlight/C2SP `tlog-tiles` specification. This enables offline-capable verification and more efficient proof computation.

#### 3.4.1 Architecture Overview

In tile-based logs, the Merkle tree is stored in fixed-size chunks (tiles) of 256 entries each:

```
                    Tile Structure (256 entries/tile)
    ───────────────────────────────────────────────────────────
                           Level 2 (root)
                              [Tile]
                             /      \
                    Level 1 (intermediate)
                   [Tile 0]  [Tile 1]  ...
                   /      \
              Level 0 (leaves)
    [Tile 0] [Tile 1] [Tile 2] [Tile 3] ...

    Each tile contains up to 256 hashes (32 bytes each = 8KB max)
```

#### 3.4.2 Log Version Configuration

StellaOps supports automatic selection and explicit version selection:

```csharp
public enum RekorLogVersion
{
    Auto = 0,   // Auto-selects v2 tiles
    V2 = 2      // Tile-based Sunlight format
}
```

**Version Selection Logic:**

| Version | Result |
|---------|--------|
| V2 | Always use tile proofs |
| Auto | Always use tile proofs |

#### 3.4.3 Checkpoint Format

V2 checkpoints follow the `c2sp.org/tlog-tiles` format:

```
rekor.sigstore.dev - 2605736670972794746
<tree_size>
<root_hash_base64>

- rekor.sigstore.dev <signature_base64>
```

**Checkpoint Components:**
- **Line 1**: Origin identifier (log name + instance)
- **Line 2**: Tree size (number of leaves)
- **Line 3**: Root hash (base64-encoded SHA-256)
- **Blank line**: Separator
- **Signature lines**: One or more `- <origin> <signature>` lines

#### 3.4.4 Tile Path Calculation

Tiles are fetched via URL paths following the scheme:

```
GET {tile_base_url}/tile/{level}/{index:03d}[.p/{partial_width}]

Examples:
- /tile/0/000        # Level 0, tile 0 (entries 0-255)
- /tile/0/001        # Level 0, tile 1 (entries 256-511)
- /tile/1/000        # Level 1, tile 0 (intermediate hashes)
- /tile/0/042.p/128  # Partial tile with 128 entries
```

#### 3.4.5 Implementation Classes

**IRekorTileClient Interface:**

```csharp
public interface IRekorTileClient
{
    Task<RekorTileCheckpoint?> GetCheckpointAsync(
        RekorBackend backend,
        CancellationToken cancellationToken = default);

    Task<RekorTileData?> GetTileAsync(
        RekorBackend backend,
        int level,
        long index,
        CancellationToken cancellationToken = default);

    Task<RekorTileEntry?> GetEntryAsync(
        RekorBackend backend,
        long logIndex,
        CancellationToken cancellationToken = default);

    Task<RekorTileInclusionProof?> ComputeInclusionProofAsync(
        RekorBackend backend,
        long logIndex,
        long treeSize,
        CancellationToken cancellationToken = default);
}
```

**RekorTileData Model:**

```csharp
public sealed class RekorTileData
{
    public required int Level { get; init; }
    public required long Index { get; init; }
    public required int Width { get; init; }  // Number of hashes (max 256)
    public required byte[] Hashes { get; init; }  // Width * 32 bytes

    public byte[] GetHash(int position)
    {
        if (position < 0 || position >= Width)
            throw new ArgumentOutOfRangeException(nameof(position));

        var result = new byte[32];
        Array.Copy(Hashes, position * 32, result, 0, 32);
        return result;
    }
}
```

#### 3.4.6 Proof Computation Algorithm

Computing an inclusion proof from tiles:

```python
def compute_inclusion_proof(log_index, tree_size, tile_client):
    """Compute inclusion proof by fetching necessary tiles."""
    proof_path = []
    level = 0
    index = log_index
    size = tree_size

    while size > 1:
        tile_index = index // 256
        position_in_tile = index % 256

        # Determine sibling position
        if index % 2 == 1:
            sibling_pos = position_in_tile - 1
        else:
            sibling_pos = position_in_tile + 1 if position_in_tile + 1 < size else None

        if sibling_pos is not None:
            tile = tile_client.get_tile(level, tile_index)
            proof_path.append(tile.get_hash(sibling_pos))

        index = index // 2
        size = (size + 1) // 2
        level += 1

    return proof_path
```

#### 3.4.7 Configuration

```yaml
attestor:
  rekor:
    primary:
      url: https://rekor.sigstore.dev
      # Version: Auto or V2
      version: Auto
      # Custom tile base URL (optional, defaults to {url}/tile/)
      tile_base_url: ""
      # Log ID for multi-log environments (hex-encoded SHA-256)
      log_id: "c0d23d6ad406973f9559f3ba2d1ca01f84147d8ffc5b8445c224f98b9591801d"
```

**Environment Variables:**

```bash
# Rekor v2 Configuration
REKOR_SERVER_URL=https://rekor.sigstore.dev
REKOR_VERSION=Auto           # Auto or V2
REKOR_TILE_BASE_URL=         # Optional custom tile endpoint
REKOR_LOG_ID=c0d23d6ad406973f9559f3ba2d1ca01f84147d8ffc5b8445c224f98b9591801d
```

#### 3.4.8 Offline Verification Benefits

Tile-based verification enables true offline capability:

1. **Pre-fetch tiles**: Download all necessary tiles during online phase
2. **Bundle checkpoint**: Include signed checkpoint with offline kit
3. **Local proof computation**: Compute proofs entirely from local tile data
4. **No API dependency**: Verification works without Rekor connectivity

```
┌─────────────────────────────────────────────────────────────┐
│                    Offline Verification                      │
├─────────────────────────────────────────────────────────────┤
│                                                              │
│  ┌─────────────┐     ┌──────────────┐     ┌──────────────┐  │
│  │ Checkpoint  │────►│ Tile Cache   │────►│ Proof        │  │
│  │ (signed)    │     │ (local)      │     │ Verifier     │  │
│  └─────────────┘     └──────────────┘     └──────────────┘  │
│                                                              │
│  Advantages:                                                 │
│  - No network round-trips for proof fetching                │
│  - Deterministic verification (same tiles = same proof)     │
│  - Caching efficiency (tiles are immutable)                 │
│  - Air-gap compatible                                        │
│                                                              │
└─────────────────────────────────────────────────────────────┘
```

---

## 4. DATA FLOW

### 4.1 Submission Flow (with Queue)

```
┌─────────┐      ┌──────────────────┐      ┌───────────┐
│ Client  │─────►│ SubmissionService │─────►│ RekorAPI  │
└─────────┘      └────────┬─────────┘      └─────┬─────┘
                          │                       │
                          │ (success)             │
                          ▼                       │
                 ┌─────────────────┐              │
                 │ Store Entry     │◄─────────────┘
                 │ (status=ok)     │
                 └─────────────────┘

                          │ (failure)
                          ▼
                 ┌─────────────────┐
                 │ Enqueue         │
                 │ (status=pending)│
                 └────────┬────────┘
                          │
                          ▼
                 ┌─────────────────┐      ┌───────────┐
                 │ RetryWorker     │─────►│ RekorAPI  │
                 │ (background)    │      └─────┬─────┘
                 └─────────────────┘            │
                          ▲                     │
                          │                     │
                          └─────────────────────┘
                              (retry loop)
```

### 4.2 Verification Flow (with Proof Verification)

```
┌─────────┐      ┌───────────────────┐      ┌─────────────────┐
│ Client  │─────►│ VerificationSvc   │─────►│ EntryRepository │
└─────────┘      └────────┬──────────┘      └────────┬────────┘
                          │                          │
                          │◄─────────────────────────┘
                          │      (AttestorEntry)
                          ▼
                 ┌─────────────────────┐
                 │ 1. TimeSkewValidator│
                 │    (integrated_time)│
                 └────────┬────────────┘
                          │
                          ▼
                 ┌─────────────────────┐
                 │ 2. MerkleProof      │
                 │    Verifier         │
                 └────────┬────────────┘
                          │
                          ▼
                 ┌─────────────────────┐
                 │ 3. Checkpoint       │
                 │    Verifier         │
                 └────────┬────────────┘
                          │
                          ▼
                 ┌─────────────────────┐
                 │ 4. Aggregate Result │
                 │    (VerificationRpt)│
                 └─────────────────────┘
```

---

## 5. CONFIGURATION REFERENCE

```yaml
# attestor.yaml

attestor:
  rekor:
    primary:
      url: https://rekor.sigstore.dev
      proof_timeout_ms: 15000
      poll_interval_ms: 250
      max_attempts: 60

    mirror:
      enabled: false
      url: https://rekor-mirror.internal

    verification:
      public_key_path: /etc/stellaops/rekor-pub.pem
      # Or inline:
      # public_key_base64: LS0tLS1CRUdJTi...
      allow_offline_without_signature: false
      max_checkpoint_age_minutes: 60

    queue:
      enabled: true
      max_attempts: 5
      initial_delay_ms: 1000
      max_delay_ms: 60000
      backoff_multiplier: 2.0
      batch_size: 10
      poll_interval_ms: 5000
      dead_letter_retention_days: 30

    time_skew:
      enabled: true
      warn_threshold_seconds: 300      # 5 minutes
      reject_threshold_seconds: 3600   # 1 hour
      future_tolerance_seconds: 60     # 1 minute
      reject_future_timestamps: true
      skip_in_offline_mode: true
```

---

## 6. METRICS REFERENCE

| Metric | Type | Labels | Description |
|--------|------|--------|-------------|
| `attestor.inclusion_verify_total` | Counter | `result` | Inclusion proof verifications |
| `attestor.inclusion_verify_latency_seconds` | Histogram | | Verification latency |
| `attestor.checkpoint_verify_total` | Counter | `result` | Checkpoint signature verifications |
| `attestor.rekor_queue_depth` | Gauge | | Current pending + retrying items |
| `attestor.rekor_retry_attempts_total` | Counter | `backend`, `attempt` | Retry attempts |
| `attestor.rekor_submission_status_total` | Counter | `status`, `backend` | Submission outcomes |
| `attestor.rekor_queue_wait_seconds` | Histogram | | Time in queue before submission |
| `attestor.time_skew_detected_total` | Counter | `severity`, `action` | Time skew detections |
| `attestor.time_skew_seconds` | Histogram | | Observed skew distribution |

---

## 7. ERROR HANDLING

### 7.1 Error Taxonomy

| Error Code | Description | Retry? | Action |
|------------|-------------|--------|--------|
| `rekor_unavailable` | Rekor API not reachable | Yes | Queue for retry |
| `rekor_conflict` | Duplicate entry (409) | No | Retrieve existing entry |
| `rekor_rate_limited` | Rate limit exceeded (429) | Yes | Backoff and retry |
| `rekor_internal_error` | Server error (5xx) | Yes | Queue for retry |
| `proof_invalid` | Merkle proof verification failed | No | Reject, log alert |
| `checkpoint_signature_invalid` | Checkpoint signature failed | No | Reject, log alert |
| `time_skew_rejected` | Time skew exceeds threshold | No | Reject, log warning |

### 7.2 Structured Logging

```json
{
  "timestamp": "2025-12-14T10:30:00Z",
  "level": "Warning",
  "message": "Rekor submission failed, queuing for retry",
  "error_code": "rekor_unavailable",
  "bundle_sha256": "abc123...",
  "backend": "primary",
  "attempt_count": 1,
  "next_retry_at": "2025-12-14T10:30:02Z",
  "error_message": "Connection refused"
}
```

---

## 8. SECURITY CONSIDERATIONS

### 8.1 Key Management

- Rekor public key must be distributed out-of-band
- Support key rotation via versioned key configuration
- Store keys in secure location (not in code/config)

### 8.2 Trust Model

```
                    Trust Hierarchy
                    ───────────────

                    ┌─────────────┐
                    │ Rekor Root  │
                    │ Public Key  │
                    └──────┬──────┘
                           │ signs
                           ▼
                    ┌─────────────┐
                    │ Checkpoint  │
                    │ (root hash) │
                    └──────┬──────┘
                           │ commits to
                           ▼
                    ┌─────────────┐
                    │ Merkle Tree │
                    │ (entries)   │
                    └──────┬──────┘
                           │ includes
                           ▼
                    ┌─────────────┐
                    │ Attestation │
                    │ (DSSE)      │
                    └─────────────┘
```

### 8.3 Offline Security

In air-gapped environments:
- Checkpoint must be pre-distributed with offline bundle
- Proof verification still works (no network needed)
- Time skew validation should be skipped or use bundled reference time

---

## 9. TESTING STRATEGY

### 9.1 Test Categories

| Category | Coverage Target | Tools |
|----------|-----------------|-------|
| Unit | >90% | xUnit, Moq |
| Integration | >80% | Testcontainers (PostgreSQL) |
| Contract | All public APIs | Snapshot testing |
| Performance | Latency P99 | BenchmarkDotNet |

### 9.2 Golden Fixtures

Obtain from public Sigstore Rekor instance:

```bash
# Get a real Rekor entry for testing
rekor-cli get --uuid 24296fb24b8ad77a... --format json > fixtures/rekor-entry-valid.json

# Get checkpoint
curl https://rekor.sigstore.dev/api/v1/log > fixtures/rekor-checkpoint.json

# Get public key
curl https://rekor.sigstore.dev/api/v1/log/publicKey > fixtures/rekor-pubkey.pem
```

---

## 9A. PERIODIC VERIFICATION (Background Job)

**Sprint Reference**: `SPRINT_20260117_001_ATTESTOR_periodic_rekor_verification`

### 9A.1 Overview

The Periodic Verification system provides continuous validation of previously logged Rekor entries. This addresses the gap where entries are logged but never re-verified, enabling detection of:

- Signature tampering or key compromise
- Merkle tree rollbacks (split-view attacks)
- Time skew violations indicating replay attempts
- Root consistency drift between stored and remote state

### 9A.2 Architecture

```
┌─────────────────────────────────────────────────────────────────────┐
│                   Periodic Verification Job                         │
├─────────────────────────────────────────────────────────────────────┤
│                                                                      │
│  ┌─────────────────────┐    ┌─────────────────────┐                 │
│  │ RekorVerification   │───►│ IRekorVerification  │                 │
│  │ Job (Scheduler)     │    │ Service             │                 │
│  └─────────┬───────────┘    └──────────┬──────────┘                 │
│            │                           │                             │
│            │ batch query               │ verify                      │
│            ▼                           ▼                             │
│  ┌─────────────────────┐    ┌─────────────────────┐                 │
│  │ IRekorEntry         │    │ RekorVerification   │                 │
│  │ Repository          │    │ Metrics             │                 │
│  └─────────────────────┘    └──────────┬──────────┘                 │
│                                        │                             │
│                                        ▼                             │
│                             ┌─────────────────────┐                 │
│                             │ IRekorVerification  │                 │
│                             │ StatusProvider      │                 │
│                             └─────────────────────┘                 │
└─────────────────────────────────────────────────────────────────────┘
```

### 9A.3 Configuration

```yaml
attestor:
  rekor:
    verification:
      enabled: true
      intervalMinutes: 60           # Run every hour
      batchSize: 100                # Entries per batch
      sampleRate: 0.1               # 10% sampling for large deployments
      maxTimeSkewSeconds: 300       # 5 minute tolerance
      alertOnRootInconsistency: true
```

### 9A.4 Verification Checks

| Check | Description | Failure Severity |
|-------|-------------|------------------|
| Signature | Verify entry signature against stored public key | Critical |
| Inclusion Proof | RFC 6962 Merkle inclusion proof verification | Critical |
| Time Skew | Validate integrated_time within tolerance | Warning |
| Root Consistency | Compare stored tree root with remote | Critical |

### 9A.5 Metrics (OpenTelemetry)

```
# Meter: StellaOps.Attestor.RekorVerification

attestor.rekor.verification.runs              # Counter
attestor.rekor.verification.entries.verified  # Counter
attestor.rekor.verification.entries.failed    # Counter
attestor.rekor.verification.entries.skipped   # Counter
attestor.rekor.verification.time_skew_violations # Counter
attestor.rekor.verification.signature_failures   # Counter
attestor.rekor.verification.inclusion_proof_failures # Counter
attestor.rekor.verification.root_consistency_checks  # Counter
attestor.rekor.verification.entry_duration    # Histogram
attestor.rekor.verification.batch_duration    # Histogram
```

### 9A.6 Health Check Integration

The `RekorVerificationHealthCheck` integrates with the Doctor diagnostic system:

```
Check ID: check.attestation.rekor.verification.job

Status Levels:
- Healthy: Last run within expected window, failure rate < 1%
- Degraded: Failure rate 1-5%, or last run overdue
- Unhealthy: Failure rate > 5%, root inconsistency detected, or job not running
```

### 9A.7 Alerting

| Condition | Alert Level | Action |
|-----------|-------------|--------|
| Root inconsistency | P1 Critical | Immediate investigation required |
| Signature failure rate > 5% | P2 High | Review key material |
| Job not running > 3x interval | P3 Medium | Check scheduler |
| Time skew violations > 10% | P3 Medium | Check NTP sync |

### 9A.8 Offline Verification

When network access to Rekor is unavailable, the system falls back to stored inclusion proofs:

1. Read stored `inclusion_proof` from database
2. Verify Merkle path locally against stored root
3. Log verification as "offline" mode
4. Schedule online re-verification when connectivity returns

---

## 10. MIGRATION GUIDE

### 10.1 Database Migrations

Run in order:
1. `00X_rekor_submission_queue.sql` - Queue table
2. Update `AttestorEntry` schema if stored in PostgreSQL

### 10.2 Configuration Migration

```yaml
# Before (existing)
attestor:
  rekor:
    primary:
      url: https://rekor.sigstore.dev

# After (add new sections)
attestor:
  rekor:
    primary:
      url: https://rekor.sigstore.dev
    verification:
      public_key_path: /etc/stellaops/rekor-pub.pem
    queue:
      enabled: true
    time_skew:
      enabled: true
```

### 10.3 Rollback Plan

- Queue table can be dropped if not needed
- All new features are configurable (can disable)
- No breaking changes to existing API contracts

---

## 11. REFERENCES

- [RFC 6962: Certificate Transparency](https://datatracker.ietf.org/doc/html/rfc6962)
- [Sigstore Rekor](https://github.com/sigstore/rekor)
- [Transparency.dev Checkpoint Format](https://github.com/transparency-dev/formats)
- [C2SP tlog-tiles Specification](https://c2sp.org/tlog-tiles) - Tile-based transparency log format
- [Sunlight CT Log](https://github.com/FiloSottile/sunlight) - Reference implementation for tile-based logs
- [Sigstore Rekor v2 Announcement](https://blog.sigstore.dev/) - Official Rekor v2 migration information
- [Advisory: Rekor Integration Technical Reference](../../../product/advisories/14-Dec-2025%20-%20Rekor%20Integration%20Technical%20Reference.md)