Files

master 4ca3ce8fb4 sprints completion. new product advisories prepared

2026-01-16 16:30:03 +02:00

9.6 KiB

Raw Blame History

Rekor Checkpoint Sync Configuration and Operations

This guide covers the configuration and operational procedures for the Rekor periodic checkpoint synchronization service.

Overview

The Rekor sync service maintains a local mirror of Rekor transparency log checkpoints and tiles. This enables:

Offline verification: Verify attestations without network access to Sigstore
Air-gapped operation: Run in environments without internet connectivity
Performance: Reduce latency by using local checkpoint data
Auditability: Maintain local evidence of log state at verification time

Architecture

┌─────────────────────────────────────────────────────────────────┐
│                      RekorSyncBackgroundService                  │
│  ┌──────────────┐     ┌──────────────┐     ┌──────────────┐     │
│  │  Checkpoint  │     │   Signature  │     │    Tile      │     │
│  │   Fetcher    │────▶│   Verifier   │────▶│   Syncer     │     │
│  └──────────────┘     └──────────────┘     └──────────────┘     │
└─────────────────────────────────────────────────────────────────┘
          │                     │                     │
          ▼                     ▼                     ▼
   ┌──────────────┐      ┌──────────────┐     ┌──────────────┐
   │  HTTP Tile   │      │  Checkpoint  │     │    Tile      │
   │   Client     │      │    Store     │     │   Cache      │
   └──────────────┘      │ (PostgreSQL) │     │(File System) │
          │              └──────────────┘     └──────────────┘
          ▼
   ┌──────────────┐
   │    Rekor     │
   │   Server     │
   └──────────────┘

Configuration

Basic Configuration

attestor:
  rekorSync:
    # Enable or disable sync service
    enabled: true

    # How often to fetch new checkpoints
    syncInterval: 5m

    # Delay before first sync after startup
    initialDelay: 30s

    # Enable tile synchronization for full offline support
    enableTileSync: true

    # Maximum tiles to fetch per sync cycle
    maxTilesPerSync: 100

    # Backend configurations
    backends:
      - id: sigstore-prod
        origin: rekor.sigstore.dev
        baseUrl: https://rekor.sigstore.dev
        publicKeyPath: /etc/stella/keys/rekor-sigstore-prod.pub

      - id: sigstore-staging
        origin: rekor.sigstage.dev
        baseUrl: https://rekor.sigstage.dev
        publicKeyPath: /etc/stella/keys/rekor-sigstore-staging.pub

Checkpoint Store Configuration (PostgreSQL)

attestor:
  checkpointStore:
    connectionString: "Host=localhost;Database=stella;Username=stella;Password=secret"
    schema: attestor
    autoInitializeSchema: true

Tile Cache Configuration (File System)

attestor:
  tileCache:
    # Base directory for tile storage
    basePath: /var/lib/stella/attestor/tiles

    # Maximum cache size (0 = unlimited)
    maxCacheSizeBytes: 10737418240  # 10 GB

    # Auto-prune tiles older than this
    autoPruneAfter: 720h  # 30 days

Operational Procedures

Initial Setup

Initialize the checkpoint store schema:

stella attestor checkpoint-store init --connection "Host=localhost;..."

Configure backend(s):

stella attestor backend add sigstore-prod \
  --origin rekor.sigstore.dev \
  --url https://rekor.sigstore.dev \
  --public-key /path/to/rekor.pub

Perform initial sync:

stella attestor sync --backend sigstore-prod --full

Manual Sync Operations

Force immediate sync:

stella attestor sync --backend sigstore-prod

Sync all backends:

stella attestor sync --all

Full tile sync (for offline kit preparation):

stella attestor sync --backend sigstore-prod --full-tiles

Monitoring

Check sync status:

stella attestor sync-status

Output:

Backend         Origin                 Tree Size    Last Sync            Age
sigstore-prod   rekor.sigstore.dev    45,678,901   2026-01-15 12:34:56  2m 15s
sigstore-staging rekor.sigstage.dev   1,234,567    2026-01-15 12:30:00  6m 30s

Check checkpoint history:

stella attestor checkpoints list --backend sigstore-prod --last 10

Check tile cache status:

stella attestor tiles stats --backend sigstore-prod

Output:

Origin: rekor.sigstore.dev
Total Tiles: 45,678
Cache Size: 1.4 GB
Coverage: 100% (tree size 45,678,901)
Oldest Tile: 2026-01-01 00:00:00
Newest Tile: 2026-01-15 12:34:56

Metrics

The sync service exposes the following Prometheus metrics:

# Counter: checkpoints fetched from remote
attestor_rekor_sync_checkpoints_fetched_total{backend="sigstore-prod"} 1234

# Counter: checkpoints stored locally
attestor_rekor_sync_checkpoints_stored_total{backend="sigstore-prod"} 1234

# Counter: tiles fetched from remote
attestor_rekor_sync_tiles_fetched_total{backend="sigstore-prod"} 56789

# Counter: tiles cached locally
attestor_rekor_sync_tiles_cached_total{backend="sigstore-prod"} 56789

# Histogram: checkpoint age at sync time (seconds)
attestor_rekor_sync_checkpoint_age_seconds{backend="sigstore-prod"} 

# Gauge: total tiles cached
attestor_rekor_sync_tiles_cached{backend="sigstore-prod"} 45678

# Gauge: time since last successful sync (seconds)
attestor_rekor_sync_last_success_seconds{backend="sigstore-prod"} 135

# Counter: sync errors
attestor_rekor_sync_errors_total{backend="sigstore-prod",error_type="network"} 5

Alerting Recommendations

groups:
  - name: attestor-rekor-sync
    rules:
      - alert: RekorSyncStale
        expr: attestor_rekor_sync_last_success_seconds > 900
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: Rekor sync is stale
          description: "No successful sync in {{ $value }}s for {{ $labels.backend }}"

      - alert: RekorSyncFailing
        expr: rate(attestor_rekor_sync_errors_total[5m]) > 0.1
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: Rekor sync experiencing errors
          description: "Sync errors detected for {{ $labels.backend }}"

Maintenance Tasks

Prune old checkpoints:

# Keep only last 30 days of checkpoints
stella attestor checkpoints prune --older-than 720h --keep-latest

Prune old tiles:

# Remove tiles for entries no longer needed
stella attestor tiles prune --older-than 720h

Verify checkpoint store integrity:

stella attestor checkpoints verify --backend sigstore-prod

Export checkpoints for air-gap:

stella attestor export \
  --backend sigstore-prod \
  --output /mnt/airgap/attestor-bundle.tar.gz \
  --include-tiles

Troubleshooting

Sync Not Running

Check service logs:
```
journalctl -u stella-attestor -f
```
Verify configuration:
```
stella attestor config validate
```
Check database connectivity:
```
stella attestor checkpoint-store test
```

Signature Verification Failing

Verify public key is correct:

stella attestor backend verify-key sigstore-prod

Check for key rotation:
- Monitor Sigstore announcements
- Update public key if rotated

Compare with direct fetch:

curl -s https://rekor.sigstore.dev/api/v1/log | jq

Tile Cache Issues

Check disk space:
```
df -h /var/lib/stella/attestor/tiles
```
Verify permissions:
```
ls -la /var/lib/stella/attestor/tiles
```

Clear and resync:

stella attestor tiles clear --backend sigstore-prod
stella attestor sync --backend sigstore-prod --full-tiles

Database Issues

Check PostgreSQL connectivity:

psql -h localhost -U stella -d stella -c "SELECT 1"

Verify schema exists:

SELECT * FROM attestor.rekor_checkpoints LIMIT 1;

Reinitialize schema if needed:

stella attestor checkpoint-store init --force

Air-Gap Operations

Preparing an Offline Bundle

Sync to latest checkpoint:

stella attestor sync --backend sigstore-prod --full-tiles

Export bundle:

stella attestor export \
  --backend sigstore-prod \
  --output offline-attestor-bundle.tar.gz \
  --include-tiles \
  --checkpoints-only-verified

Transfer bundle to air-gapped environment

Importing in Air-Gapped Environment

Import the bundle:

stella attestor import offline-attestor-bundle.tar.gz

Verify import:
```
stella attestor sync-status
```
Checkpoints and tiles are now available for offline verification

9.6 KiB Raw Blame History