git.stella-ops.org/docs/modules/attestor/operations/bundle-rotation.md

Bundle Rotation Operations Guide

This guide covers operational procedures for attestation bundle rotation in StellaOps.

Overview

Bundle rotation is a scheduled process that aggregates attestations from a time period into a verifiable bundle. This enables long-term verification of keyless-signed artifacts beyond their certificate expiry.

Rotation Schedule

Default Schedule

attestor:
  bundling:
    schedule:
      cron: "0 2 1 * *"   # Monthly on the 1st at 02:00 UTC
      cadence: monthly
      timezone: UTC
      skipWeekends: false

Cadence Options

Cadence	Period	Use Case
weekly	Previous 7 days	High-volume environments
monthly	Previous month	Standard deployment (default)
quarterly	Previous quarter	Low-volume, compliance-focused

Manual Rotation

Trigger Immediate Rotation

# Rotate current period
stella attestor bundle rotate

# Rotate specific period
stella attestor bundle rotate --start 2025-12-01 --end 2025-12-31

# Rotate for specific tenant
stella attestor bundle rotate --tenant tenant-gov

API Trigger

POST /api/v1/bundles
Content-Type: application/json

{
  "periodStart": "2025-12-01T00:00:00Z",
  "periodEnd": "2025-12-31T23:59:59Z",
  "tenantId": null,
  "signWithOrgKey": true,
  "orgKeyId": "org-signing-key-2025"
}

Monitoring

Key Metrics

Metric	Description	Alert Threshold
attestor_bundle_created_total	Bundles created	N/A (informational)
attestor_bundle_creation_duration_seconds	Creation time	> 30 minutes
attestor_bundle_attestations_count	Attestations per bundle	> 10,000
attestor_bundle_size_bytes	Bundle size	> 100 MB
attestor_bundle_retention_deleted_total	Expired bundles deleted	N/A

Grafana Dashboard

Import the attestor observability dashboard:

stella observability import --dashboard attestor-bundling

See: docs/modules/attestor/operations/dashboards/attestor-observability.json

Health Check

# Check bundle rotation status
stella attestor bundle status

# Sample output:
# Last Rotation: 2025-12-01T02:00:00Z
# Next Scheduled: 2026-01-01T02:00:00Z
# Bundles This Month: 3
# Total Attestations Bundled: 4,521
# Status: Healthy

Retention Policy

Configuration

attestor:
  bundling:
    retention:
      enabled: true
      defaultMonths: 24
      minimumMonths: 6
      maximumMonths: 120
      expiryAction: delete    # delete | archive | markOnly
      archiveStorageTier: glacier
      gracePeriodDays: 30
      notifyBeforeExpiry: true
      notifyDaysBeforeExpiry: 30
      maxBundlesPerRun: 100

Retention Actions

Action	Behavior
delete	Permanently remove expired bundles
archive	Move to cold storage (S3 Glacier)
markOnly	Mark as expired but retain

Manual Retention Enforcement

# Preview expired bundles
stella attestor bundle retention --dry-run

# Apply retention policy
stella attestor bundle retention --apply

# Force delete specific bundle
stella attestor bundle delete sha256:abc123...

Troubleshooting

Bundle Creation Failed

Symptoms: Rotation job completes with errors

Check:

# View recent rotation logs
stella logs --service attestor --filter "bundle rotation"

# Check attestor health
stella attestor health

Common causes:

1. Database connection issues
2. Insufficient attestations in period
3. Org key unavailable for signing

Large Bundle Size

Symptoms: Bundle exceeds size limits or takes too long

Solutions:

1. Reduce maxAttestationsPerBundle to create multiple smaller bundles
2. Increase queryBatchSize for faster database queries
3. Enable compression for storage

attestor:
  bundling:
    aggregation:
      maxAttestationsPerBundle: 5000
      queryBatchSize: 1000

Org Key Signing Failed

Symptoms: Bundle created without org signature

Check:

# Verify org key availability
stella signer keys list --type org

# Test key signing
stella signer keys test org-signing-key-2025

Solutions:

1. Ensure KMS/HSM connectivity
2. Verify key permissions
3. Check key rotation schedule

Retention Not Running

Symptoms: Expired bundles not being deleted

Check:

# Verify retention is enabled
stella attestor bundle retention --status

# Check for blocked bundles
stella attestor bundle list --status expired

Solutions:

1. Ensure retention.enabled: true
2. Check grace period configuration
3. Verify storage backend permissions

Disaster Recovery

Bundle Export

Export bundles for backup:

# Export all bundles from a period
stella attestor bundle export \
  --start 2025-01-01 \
  --end 2025-12-31 \
  --output /backup/bundles/

# Export specific bundle
stella attestor bundle export sha256:abc123 --output bundle.json

Bundle Import

Restore bundles from backup:

# Import bundle file
stella attestor bundle import /backup/bundles/bundle-sha256-abc123.json

# Bulk import
stella attestor bundle import /backup/bundles/*.json

Verification After Restore

# Verify imported bundle
stella attestor bundle verify sha256:abc123

# Verify all bundles
stella attestor bundle verify --all

Runbooks

Monthly Rotation Check

1. Pre-rotation (1 day before):

   stella attestor bundle preview --period 2025-12
   

2. Post-rotation (rotation day + 1):

   stella attestor bundle list --created-after 2025-12-01
   stella attestor bundle verify --period 2025-12
   

3. Verify notifications sent: Check Slack/Teams/Email for rotation summary

Quarterly Audit

1. List all bundles:

   stella attestor bundle list --format json > audit-report.json
   

2. Verify sample bundles:

   # Random sample of 10%
   stella attestor bundle verify --sample 0.1
   

3. Check retention compliance:

   stella attestor bundle retention --audit
   

Emergency Bundle Access

For urgent verification needs:

# Extract specific attestation from bundle
stella attestor bundle extract sha256:abc123 --entry-id uuid-1

# Verify attestation outside bundle
stella attestor verify --envelope attestation.dsse

Related Documentation

• Bundle Format Specification
• Attestor Architecture
• Observability Guide
• Air-Gap Operations