363 lines
9.6 KiB
Markdown
363 lines
9.6 KiB
Markdown
# Rekor Checkpoint Sync Configuration and Operations
|
|
|
|
This guide covers the configuration and operational procedures for the Rekor periodic checkpoint synchronization service.
|
|
|
|
## Overview
|
|
|
|
The Rekor sync service maintains a local mirror of Rekor transparency log checkpoints and tiles. This enables:
|
|
|
|
- **Offline verification**: Verify attestations without network access to Sigstore
|
|
- **Air-gapped operation**: Run in environments without internet connectivity
|
|
- **Performance**: Reduce latency by using local checkpoint data
|
|
- **Auditability**: Maintain local evidence of log state at verification time
|
|
|
|
## Architecture
|
|
|
|
```
|
|
┌─────────────────────────────────────────────────────────────────┐
|
|
│ RekorSyncBackgroundService │
|
|
│ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │
|
|
│ │ Checkpoint │ │ Signature │ │ Tile │ │
|
|
│ │ Fetcher │────▶│ Verifier │────▶│ Syncer │ │
|
|
│ └──────────────┘ └──────────────┘ └──────────────┘ │
|
|
└─────────────────────────────────────────────────────────────────┘
|
|
│ │ │
|
|
▼ ▼ ▼
|
|
┌──────────────┐ ┌──────────────┐ ┌──────────────┐
|
|
│ HTTP Tile │ │ Checkpoint │ │ Tile │
|
|
│ Client │ │ Store │ │ Cache │
|
|
└──────────────┘ │ (PostgreSQL) │ │(File System) │
|
|
│ └──────────────┘ └──────────────┘
|
|
▼
|
|
┌──────────────┐
|
|
│ Rekor │
|
|
│ Server │
|
|
└──────────────┘
|
|
```
|
|
|
|
## Configuration
|
|
|
|
### Basic Configuration
|
|
|
|
```yaml
|
|
attestor:
|
|
rekorSync:
|
|
# Enable or disable sync service
|
|
enabled: true
|
|
|
|
# How often to fetch new checkpoints
|
|
syncInterval: 5m
|
|
|
|
# Delay before first sync after startup
|
|
initialDelay: 30s
|
|
|
|
# Enable tile synchronization for full offline support
|
|
enableTileSync: true
|
|
|
|
# Maximum tiles to fetch per sync cycle
|
|
maxTilesPerSync: 100
|
|
|
|
# Backend configurations
|
|
backends:
|
|
- id: sigstore-prod
|
|
origin: rekor.sigstore.dev
|
|
baseUrl: https://rekor.sigstore.dev
|
|
publicKeyPath: /etc/stella/keys/rekor-sigstore-prod.pub
|
|
|
|
- id: sigstore-staging
|
|
origin: rekor.sigstage.dev
|
|
baseUrl: https://rekor.sigstage.dev
|
|
publicKeyPath: /etc/stella/keys/rekor-sigstore-staging.pub
|
|
```
|
|
|
|
### Checkpoint Store Configuration (PostgreSQL)
|
|
|
|
```yaml
|
|
attestor:
|
|
checkpointStore:
|
|
connectionString: "Host=localhost;Database=stella;Username=stella;Password=secret"
|
|
schema: attestor
|
|
autoInitializeSchema: true
|
|
```
|
|
|
|
### Tile Cache Configuration (File System)
|
|
|
|
```yaml
|
|
attestor:
|
|
tileCache:
|
|
# Base directory for tile storage
|
|
basePath: /var/lib/stella/attestor/tiles
|
|
|
|
# Maximum cache size (0 = unlimited)
|
|
maxCacheSizeBytes: 10737418240 # 10 GB
|
|
|
|
# Auto-prune tiles older than this
|
|
autoPruneAfter: 720h # 30 days
|
|
```
|
|
|
|
## Operational Procedures
|
|
|
|
### Initial Setup
|
|
|
|
1. **Initialize the checkpoint store schema**:
|
|
```bash
|
|
stella attestor checkpoint-store init --connection "Host=localhost;..."
|
|
```
|
|
|
|
2. **Configure backend(s)**:
|
|
```bash
|
|
stella attestor backend add sigstore-prod \
|
|
--origin rekor.sigstore.dev \
|
|
--url https://rekor.sigstore.dev \
|
|
--public-key /path/to/rekor.pub
|
|
```
|
|
|
|
3. **Perform initial sync**:
|
|
```bash
|
|
stella attestor sync --backend sigstore-prod --full
|
|
```
|
|
|
|
### Manual Sync Operations
|
|
|
|
**Force immediate sync**:
|
|
```bash
|
|
stella attestor sync --backend sigstore-prod
|
|
```
|
|
|
|
**Sync all backends**:
|
|
```bash
|
|
stella attestor sync --all
|
|
```
|
|
|
|
**Full tile sync** (for offline kit preparation):
|
|
```bash
|
|
stella attestor sync --backend sigstore-prod --full-tiles
|
|
```
|
|
|
|
### Monitoring
|
|
|
|
**Check sync status**:
|
|
```bash
|
|
stella attestor sync-status
|
|
```
|
|
|
|
Output:
|
|
```
|
|
Backend Origin Tree Size Last Sync Age
|
|
sigstore-prod rekor.sigstore.dev 45,678,901 2026-01-15 12:34:56 2m 15s
|
|
sigstore-staging rekor.sigstage.dev 1,234,567 2026-01-15 12:30:00 6m 30s
|
|
```
|
|
|
|
**Check checkpoint history**:
|
|
```bash
|
|
stella attestor checkpoints list --backend sigstore-prod --last 10
|
|
```
|
|
|
|
**Check tile cache status**:
|
|
```bash
|
|
stella attestor tiles stats --backend sigstore-prod
|
|
```
|
|
|
|
Output:
|
|
```
|
|
Origin: rekor.sigstore.dev
|
|
Total Tiles: 45,678
|
|
Cache Size: 1.4 GB
|
|
Coverage: 100% (tree size 45,678,901)
|
|
Oldest Tile: 2026-01-01 00:00:00
|
|
Newest Tile: 2026-01-15 12:34:56
|
|
```
|
|
|
|
### Metrics
|
|
|
|
The sync service exposes the following Prometheus metrics:
|
|
|
|
```
|
|
# Counter: checkpoints fetched from remote
|
|
attestor_rekor_sync_checkpoints_fetched_total{backend="sigstore-prod"} 1234
|
|
|
|
# Counter: checkpoints stored locally
|
|
attestor_rekor_sync_checkpoints_stored_total{backend="sigstore-prod"} 1234
|
|
|
|
# Counter: tiles fetched from remote
|
|
attestor_rekor_sync_tiles_fetched_total{backend="sigstore-prod"} 56789
|
|
|
|
# Counter: tiles cached locally
|
|
attestor_rekor_sync_tiles_cached_total{backend="sigstore-prod"} 56789
|
|
|
|
# Histogram: checkpoint age at sync time (seconds)
|
|
attestor_rekor_sync_checkpoint_age_seconds{backend="sigstore-prod"}
|
|
|
|
# Gauge: total tiles cached
|
|
attestor_rekor_sync_tiles_cached{backend="sigstore-prod"} 45678
|
|
|
|
# Gauge: time since last successful sync (seconds)
|
|
attestor_rekor_sync_last_success_seconds{backend="sigstore-prod"} 135
|
|
|
|
# Counter: sync errors
|
|
attestor_rekor_sync_errors_total{backend="sigstore-prod",error_type="network"} 5
|
|
```
|
|
|
|
### Alerting Recommendations
|
|
|
|
```yaml
|
|
groups:
|
|
- name: attestor-rekor-sync
|
|
rules:
|
|
- alert: RekorSyncStale
|
|
expr: attestor_rekor_sync_last_success_seconds > 900
|
|
for: 5m
|
|
labels:
|
|
severity: warning
|
|
annotations:
|
|
summary: Rekor sync is stale
|
|
description: "No successful sync in {{ $value }}s for {{ $labels.backend }}"
|
|
|
|
- alert: RekorSyncFailing
|
|
expr: rate(attestor_rekor_sync_errors_total[5m]) > 0.1
|
|
for: 10m
|
|
labels:
|
|
severity: warning
|
|
annotations:
|
|
summary: Rekor sync experiencing errors
|
|
description: "Sync errors detected for {{ $labels.backend }}"
|
|
```
|
|
|
|
### Maintenance Tasks
|
|
|
|
**Prune old checkpoints**:
|
|
```bash
|
|
# Keep only last 30 days of checkpoints
|
|
stella attestor checkpoints prune --older-than 720h --keep-latest
|
|
```
|
|
|
|
**Prune old tiles**:
|
|
```bash
|
|
# Remove tiles for entries no longer needed
|
|
stella attestor tiles prune --older-than 720h
|
|
```
|
|
|
|
**Verify checkpoint store integrity**:
|
|
```bash
|
|
stella attestor checkpoints verify --backend sigstore-prod
|
|
```
|
|
|
|
**Export checkpoints for air-gap**:
|
|
```bash
|
|
stella attestor export \
|
|
--backend sigstore-prod \
|
|
--output /mnt/airgap/attestor-bundle.tar.gz \
|
|
--include-tiles
|
|
```
|
|
|
|
## Troubleshooting
|
|
|
|
### Sync Not Running
|
|
|
|
1. Check service logs:
|
|
```bash
|
|
journalctl -u stella-attestor -f
|
|
```
|
|
|
|
2. Verify configuration:
|
|
```bash
|
|
stella attestor config validate
|
|
```
|
|
|
|
3. Check database connectivity:
|
|
```bash
|
|
stella attestor checkpoint-store test
|
|
```
|
|
|
|
### Signature Verification Failing
|
|
|
|
1. Verify public key is correct:
|
|
```bash
|
|
stella attestor backend verify-key sigstore-prod
|
|
```
|
|
|
|
2. Check for key rotation:
|
|
- Monitor Sigstore announcements
|
|
- Update public key if rotated
|
|
|
|
3. Compare with direct fetch:
|
|
```bash
|
|
curl -s https://rekor.sigstore.dev/api/v1/log | jq
|
|
```
|
|
|
|
### Tile Cache Issues
|
|
|
|
1. Check disk space:
|
|
```bash
|
|
df -h /var/lib/stella/attestor/tiles
|
|
```
|
|
|
|
2. Verify permissions:
|
|
```bash
|
|
ls -la /var/lib/stella/attestor/tiles
|
|
```
|
|
|
|
3. Clear and resync:
|
|
```bash
|
|
stella attestor tiles clear --backend sigstore-prod
|
|
stella attestor sync --backend sigstore-prod --full-tiles
|
|
```
|
|
|
|
### Database Issues
|
|
|
|
1. Check PostgreSQL connectivity:
|
|
```bash
|
|
psql -h localhost -U stella -d stella -c "SELECT 1"
|
|
```
|
|
|
|
2. Verify schema exists:
|
|
```sql
|
|
SELECT * FROM attestor.rekor_checkpoints LIMIT 1;
|
|
```
|
|
|
|
3. Reinitialize schema if needed:
|
|
```bash
|
|
stella attestor checkpoint-store init --force
|
|
```
|
|
|
|
## Air-Gap Operations
|
|
|
|
### Preparing an Offline Bundle
|
|
|
|
1. Sync to latest checkpoint:
|
|
```bash
|
|
stella attestor sync --backend sigstore-prod --full-tiles
|
|
```
|
|
|
|
2. Export bundle:
|
|
```bash
|
|
stella attestor export \
|
|
--backend sigstore-prod \
|
|
--output offline-attestor-bundle.tar.gz \
|
|
--include-tiles \
|
|
--checkpoints-only-verified
|
|
```
|
|
|
|
3. Transfer bundle to air-gapped environment
|
|
|
|
### Importing in Air-Gapped Environment
|
|
|
|
1. Import the bundle:
|
|
```bash
|
|
stella attestor import offline-attestor-bundle.tar.gz
|
|
```
|
|
|
|
2. Verify import:
|
|
```bash
|
|
stella attestor sync-status
|
|
```
|
|
|
|
3. Checkpoints and tiles are now available for offline verification
|
|
|
|
## See Also
|
|
|
|
- [Rekor Verification Design](../modules/attestor/rekor-verification-design.md)
|
|
- [Checkpoint Divergence Detection](./checkpoint-divergence-runbook.md)
|
|
- [Offline Kit Preparation](./offline-kit-guide.md)
|
|
- [Sigstore Rekor Documentation](https://docs.sigstore.dev/rekor/overview/)
|