# Rekor Checkpoint Sync Configuration and Operations This guide covers the configuration and operational procedures for the Rekor periodic checkpoint synchronization service. ## Overview The Rekor sync service maintains a local mirror of Rekor transparency log checkpoints and tiles. This enables: - **Offline verification**: Verify attestations without network access to Sigstore - **Air-gapped operation**: Run in environments without internet connectivity - **Performance**: Reduce latency by using local checkpoint data - **Auditability**: Maintain local evidence of log state at verification time ## Architecture ``` ┌─────────────────────────────────────────────────────────────────┐ │ RekorSyncBackgroundService │ │ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │ │ │ Checkpoint │ │ Signature │ │ Tile │ │ │ │ Fetcher │────▶│ Verifier │────▶│ Syncer │ │ │ └──────────────┘ └──────────────┘ └──────────────┘ │ └─────────────────────────────────────────────────────────────────┘ │ │ │ ▼ ▼ ▼ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │ HTTP Tile │ │ Checkpoint │ │ Tile │ │ Client │ │ Store │ │ Cache │ └──────────────┘ │ (PostgreSQL) │ │(File System) │ │ └──────────────┘ └──────────────┘ ▼ ┌──────────────┐ │ Rekor │ │ Server │ └──────────────┘ ``` ## Configuration ### Basic Configuration ```yaml attestor: rekorSync: # Enable or disable sync service enabled: true # How often to fetch new checkpoints syncInterval: 5m # Delay before first sync after startup initialDelay: 30s # Enable tile synchronization for full offline support enableTileSync: true # Maximum tiles to fetch per sync cycle maxTilesPerSync: 100 # Backend configurations backends: - id: sigstore-prod origin: rekor.sigstore.dev baseUrl: https://rekor.sigstore.dev publicKeyPath: /etc/stella/keys/rekor-sigstore-prod.pub - id: sigstore-staging origin: rekor.sigstage.dev baseUrl: https://rekor.sigstage.dev publicKeyPath: /etc/stella/keys/rekor-sigstore-staging.pub ``` ### Checkpoint Store Configuration (PostgreSQL) ```yaml attestor: checkpointStore: connectionString: "Host=localhost;Database=stella;Username=stella;Password=secret" schema: attestor autoInitializeSchema: true ``` ### Tile Cache Configuration (File System) ```yaml attestor: tileCache: # Base directory for tile storage basePath: /var/lib/stella/attestor/tiles # Maximum cache size (0 = unlimited) maxCacheSizeBytes: 10737418240 # 10 GB # Auto-prune tiles older than this autoPruneAfter: 720h # 30 days ``` ## Operational Procedures ### Initial Setup 1. **Initialize the checkpoint store schema**: ```bash stella attestor checkpoint-store init --connection "Host=localhost;..." ``` 2. **Configure backend(s)**: ```bash stella attestor backend add sigstore-prod \ --origin rekor.sigstore.dev \ --url https://rekor.sigstore.dev \ --public-key /path/to/rekor.pub ``` 3. **Perform initial sync**: ```bash stella attestor sync --backend sigstore-prod --full ``` ### Manual Sync Operations **Force immediate sync**: ```bash stella attestor sync --backend sigstore-prod ``` **Sync all backends**: ```bash stella attestor sync --all ``` **Full tile sync** (for offline kit preparation): ```bash stella attestor sync --backend sigstore-prod --full-tiles ``` ### Monitoring **Check sync status**: ```bash stella attestor sync-status ``` Output: ``` Backend Origin Tree Size Last Sync Age sigstore-prod rekor.sigstore.dev 45,678,901 2026-01-15 12:34:56 2m 15s sigstore-staging rekor.sigstage.dev 1,234,567 2026-01-15 12:30:00 6m 30s ``` **Check checkpoint history**: ```bash stella attestor checkpoints list --backend sigstore-prod --last 10 ``` **Check tile cache status**: ```bash stella attestor tiles stats --backend sigstore-prod ``` Output: ``` Origin: rekor.sigstore.dev Total Tiles: 45,678 Cache Size: 1.4 GB Coverage: 100% (tree size 45,678,901) Oldest Tile: 2026-01-01 00:00:00 Newest Tile: 2026-01-15 12:34:56 ``` ### Metrics The sync service exposes the following Prometheus metrics: ``` # Counter: checkpoints fetched from remote attestor_rekor_sync_checkpoints_fetched_total{backend="sigstore-prod"} 1234 # Counter: checkpoints stored locally attestor_rekor_sync_checkpoints_stored_total{backend="sigstore-prod"} 1234 # Counter: tiles fetched from remote attestor_rekor_sync_tiles_fetched_total{backend="sigstore-prod"} 56789 # Counter: tiles cached locally attestor_rekor_sync_tiles_cached_total{backend="sigstore-prod"} 56789 # Histogram: checkpoint age at sync time (seconds) attestor_rekor_sync_checkpoint_age_seconds{backend="sigstore-prod"} # Gauge: total tiles cached attestor_rekor_sync_tiles_cached{backend="sigstore-prod"} 45678 # Gauge: time since last successful sync (seconds) attestor_rekor_sync_last_success_seconds{backend="sigstore-prod"} 135 # Counter: sync errors attestor_rekor_sync_errors_total{backend="sigstore-prod",error_type="network"} 5 ``` ### Alerting Recommendations ```yaml groups: - name: attestor-rekor-sync rules: - alert: RekorSyncStale expr: attestor_rekor_sync_last_success_seconds > 900 for: 5m labels: severity: warning annotations: summary: Rekor sync is stale description: "No successful sync in {{ $value }}s for {{ $labels.backend }}" - alert: RekorSyncFailing expr: rate(attestor_rekor_sync_errors_total[5m]) > 0.1 for: 10m labels: severity: warning annotations: summary: Rekor sync experiencing errors description: "Sync errors detected for {{ $labels.backend }}" ``` ### Maintenance Tasks **Prune old checkpoints**: ```bash # Keep only last 30 days of checkpoints stella attestor checkpoints prune --older-than 720h --keep-latest ``` **Prune old tiles**: ```bash # Remove tiles for entries no longer needed stella attestor tiles prune --older-than 720h ``` **Verify checkpoint store integrity**: ```bash stella attestor checkpoints verify --backend sigstore-prod ``` **Export checkpoints for air-gap**: ```bash stella attestor export \ --backend sigstore-prod \ --output /mnt/airgap/attestor-bundle.tar.gz \ --include-tiles ``` ## Troubleshooting ### Sync Not Running 1. Check service logs: ```bash journalctl -u stella-attestor -f ``` 2. Verify configuration: ```bash stella attestor config validate ``` 3. Check database connectivity: ```bash stella attestor checkpoint-store test ``` ### Signature Verification Failing 1. Verify public key is correct: ```bash stella attestor backend verify-key sigstore-prod ``` 2. Check for key rotation: - Monitor Sigstore announcements - Update public key if rotated 3. Compare with direct fetch: ```bash curl -s https://rekor.sigstore.dev/api/v1/log | jq ``` ### Tile Cache Issues 1. Check disk space: ```bash df -h /var/lib/stella/attestor/tiles ``` 2. Verify permissions: ```bash ls -la /var/lib/stella/attestor/tiles ``` 3. Clear and resync: ```bash stella attestor tiles clear --backend sigstore-prod stella attestor sync --backend sigstore-prod --full-tiles ``` ### Database Issues 1. Check PostgreSQL connectivity: ```bash psql -h localhost -U stella -d stella -c "SELECT 1" ``` 2. Verify schema exists: ```sql SELECT * FROM attestor.rekor_checkpoints LIMIT 1; ``` 3. Reinitialize schema if needed: ```bash stella attestor checkpoint-store init --force ``` ## Air-Gap Operations ### Preparing an Offline Bundle 1. Sync to latest checkpoint: ```bash stella attestor sync --backend sigstore-prod --full-tiles ``` 2. Export bundle: ```bash stella attestor export \ --backend sigstore-prod \ --output offline-attestor-bundle.tar.gz \ --include-tiles \ --checkpoints-only-verified ``` 3. Transfer bundle to air-gapped environment ### Importing in Air-Gapped Environment 1. Import the bundle: ```bash stella attestor import offline-attestor-bundle.tar.gz ``` 2. Verify import: ```bash stella attestor sync-status ``` 3. Checkpoints and tiles are now available for offline verification ## See Also - [Rekor Verification Design](../modules/attestor/rekor-verification-design.md) - [Checkpoint Divergence Detection](./checkpoint-divergence-runbook.md) - [Offline Kit Preparation](./offline-kit-guide.md) - [Sigstore Rekor Documentation](https://docs.sigstore.dev/rekor/overview/)