stella-ops.org/git.stella-ops.org
1
0
Fork 0
Code Issues Pull Requests Actions 2 Releases Wiki Activity

sprints completion. new product advisories prepared

Browse Source
This commit is contained in:
master
2026-01-16 16:30:03 +02:00
parent a927d924e3
commit 4ca3ce8fb4
255 changed files with 42434 additions and 1020 deletions
Download Patch File Download Diff File Expand all files Collapse all files

362
docs/operations/rekor-sync-guide.md Normal file
View File

@@ -0,0 +1,362 @@
# Rekor Checkpoint Sync Configuration and Operations
This guide covers the configuration and operational procedures for the Rekor periodic checkpoint synchronization service.
## Overview
The Rekor sync service maintains a local mirror of Rekor transparency log checkpoints and tiles. This enables:
- **Offline verification**: Verify attestations without network access to Sigstore
- **Air-gapped operation**: Run in environments without internet connectivity
- **Performance**: Reduce latency by using local checkpoint data
- **Auditability**: Maintain local evidence of log state at verification time
## Architecture
```
┌─────────────────────────────────────────────────────────────────┐
│ RekorSyncBackgroundService │
│ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │
│ │ Checkpoint │ │ Signature │ │ Tile │ │
│ │ Fetcher │────▶│ Verifier │────▶│ Syncer │ │
│ └──────────────┘ └──────────────┘ └──────────────┘ │
└─────────────────────────────────────────────────────────────────┘
│ │ │
▼ ▼ ▼
┌──────────────┐ ┌──────────────┐ ┌──────────────┐
│ HTTP Tile │ │ Checkpoint │ │ Tile │
│ Client │ │ Store │ │ Cache │
└──────────────┘ │ (PostgreSQL) │ │(File System) │
│ └──────────────┘ └──────────────┘
▼
┌──────────────┐
│ Rekor │
│ Server │
└──────────────┘
```
## Configuration
### Basic Configuration
```yaml
attestor:
rekorSync:
# Enable or disable sync service
enabled: true
# How often to fetch new checkpoints
syncInterval: 5m
# Delay before first sync after startup
initialDelay: 30s
# Enable tile synchronization for full offline support
enableTileSync: true
# Maximum tiles to fetch per sync cycle
maxTilesPerSync: 100
# Backend configurations
backends:
- id: sigstore-prod
origin: rekor.sigstore.dev
baseUrl: https://rekor.sigstore.dev
publicKeyPath: /etc/stella/keys/rekor-sigstore-prod.pub
- id: sigstore-staging
origin: rekor.sigstage.dev
baseUrl: https://rekor.sigstage.dev
publicKeyPath: /etc/stella/keys/rekor-sigstore-staging.pub
```
### Checkpoint Store Configuration (PostgreSQL)
```yaml
attestor:
checkpointStore:
connectionString: "Host=localhost;Database=stella;Username=stella;Password=secret"
schema: attestor
autoInitializeSchema: true
```
### Tile Cache Configuration (File System)
```yaml
attestor:
tileCache:
# Base directory for tile storage
basePath: /var/lib/stella/attestor/tiles
# Maximum cache size (0 = unlimited)
maxCacheSizeBytes: 10737418240 # 10 GB
# Auto-prune tiles older than this
autoPruneAfter: 720h # 30 days
```
## Operational Procedures
### Initial Setup
1. **Initialize the checkpoint store schema**:
```bash
stella attestor checkpoint-store init --connection "Host=localhost;..."
```
2. **Configure backend(s)**:
```bash
stella attestor backend add sigstore-prod \
--origin rekor.sigstore.dev \
--url https://rekor.sigstore.dev \
--public-key /path/to/rekor.pub
```
3. **Perform initial sync**:
```bash
stella attestor sync --backend sigstore-prod --full
```
### Manual Sync Operations
**Force immediate sync**:
```bash
stella attestor sync --backend sigstore-prod
```
**Sync all backends**:
```bash
stella attestor sync --all
```
**Full tile sync** (for offline kit preparation):
```bash
stella attestor sync --backend sigstore-prod --full-tiles
```
### Monitoring
**Check sync status**:
```bash
stella attestor sync-status
```
Output:
```
Backend Origin Tree Size Last Sync Age
sigstore-prod rekor.sigstore.dev 45,678,901 2026-01-15 12:34:56 2m 15s
sigstore-staging rekor.sigstage.dev 1,234,567 2026-01-15 12:30:00 6m 30s
```
**Check checkpoint history**:
```bash
stella attestor checkpoints list --backend sigstore-prod --last 10
```
**Check tile cache status**:
```bash
stella attestor tiles stats --backend sigstore-prod
```
Output:
```
Origin: rekor.sigstore.dev
Total Tiles: 45,678
Cache Size: 1.4 GB
Coverage: 100% (tree size 45,678,901)
Oldest Tile: 2026-01-01 00:00:00
Newest Tile: 2026-01-15 12:34:56
```
### Metrics
The sync service exposes the following Prometheus metrics:
```
# Counter: checkpoints fetched from remote
attestor_rekor_sync_checkpoints_fetched_total{backend="sigstore-prod"} 1234
# Counter: checkpoints stored locally
attestor_rekor_sync_checkpoints_stored_total{backend="sigstore-prod"} 1234
# Counter: tiles fetched from remote
attestor_rekor_sync_tiles_fetched_total{backend="sigstore-prod"} 56789
# Counter: tiles cached locally
attestor_rekor_sync_tiles_cached_total{backend="sigstore-prod"} 56789
# Histogram: checkpoint age at sync time (seconds)
attestor_rekor_sync_checkpoint_age_seconds{backend="sigstore-prod"}
# Gauge: total tiles cached
attestor_rekor_sync_tiles_cached{backend="sigstore-prod"} 45678
# Gauge: time since last successful sync (seconds)
attestor_rekor_sync_last_success_seconds{backend="sigstore-prod"} 135
# Counter: sync errors
attestor_rekor_sync_errors_total{backend="sigstore-prod",error_type="network"} 5
```
### Alerting Recommendations
```yaml
groups:
- name: attestor-rekor-sync
rules:
- alert: RekorSyncStale
expr: attestor_rekor_sync_last_success_seconds > 900
for: 5m
labels:
severity: warning
annotations:
summary: Rekor sync is stale
description: "No successful sync in {{ $value }}s for {{ $labels.backend }}"
- alert: RekorSyncFailing
expr: rate(attestor_rekor_sync_errors_total[5m]) > 0.1
for: 10m
labels:
severity: warning
annotations:
summary: Rekor sync experiencing errors
description: "Sync errors detected for {{ $labels.backend }}"
```
### Maintenance Tasks
**Prune old checkpoints**:
```bash
# Keep only last 30 days of checkpoints
stella attestor checkpoints prune --older-than 720h --keep-latest
```
**Prune old tiles**:
```bash
# Remove tiles for entries no longer needed
stella attestor tiles prune --older-than 720h
```
**Verify checkpoint store integrity**:
```bash
stella attestor checkpoints verify --backend sigstore-prod
```
**Export checkpoints for air-gap**:
```bash
stella attestor export \
--backend sigstore-prod \
--output /mnt/airgap/attestor-bundle.tar.gz \
--include-tiles
```
## Troubleshooting
### Sync Not Running
1. Check service logs:
```bash
journalctl -u stella-attestor -f
```
2. Verify configuration:
```bash
stella attestor config validate
```
3. Check database connectivity:
```bash
stella attestor checkpoint-store test
```
### Signature Verification Failing
1. Verify public key is correct:
```bash
stella attestor backend verify-key sigstore-prod
```
2. Check for key rotation:
- Monitor Sigstore announcements
- Update public key if rotated
3. Compare with direct fetch:
```bash
curl -s https://rekor.sigstore.dev/api/v1/log | jq
```
### Tile Cache Issues
1. Check disk space:
```bash
df -h /var/lib/stella/attestor/tiles
```
2. Verify permissions:
```bash
ls -la /var/lib/stella/attestor/tiles
```
3. Clear and resync:
```bash
stella attestor tiles clear --backend sigstore-prod
stella attestor sync --backend sigstore-prod --full-tiles
```
### Database Issues
1. Check PostgreSQL connectivity:
```bash
psql -h localhost -U stella -d stella -c "SELECT 1"
```
2. Verify schema exists:
```sql
SELECT * FROM attestor.rekor_checkpoints LIMIT 1;
```
3. Reinitialize schema if needed:
```bash
stella attestor checkpoint-store init --force
```
## Air-Gap Operations
### Preparing an Offline Bundle
1. Sync to latest checkpoint:
```bash
stella attestor sync --backend sigstore-prod --full-tiles
```
2. Export bundle:
```bash
stella attestor export \
--backend sigstore-prod \
--output offline-attestor-bundle.tar.gz \
--include-tiles \
--checkpoints-only-verified
```
3. Transfer bundle to air-gapped environment
### Importing in Air-Gapped Environment
1. Import the bundle:
```bash
stella attestor import offline-attestor-bundle.tar.gz
```
2. Verify import:
```bash
stella attestor sync-status
```
3. Checkpoints and tiles are now available for offline verification
## See Also
- [Rekor Verification Design](../modules/attestor/rekor-verification-design.md)
- [Checkpoint Divergence Detection](./checkpoint-divergence-runbook.md)
- [Offline Kit Preparation](./offline-kit-guide.md)
- [Sigstore Rekor Documentation](https://docs.sigstore.dev/rekor/overview/)