# Bundle Rotation Operations Guide

This guide covers operational procedures for attestation bundle rotation in StellaOps.

## Overview

Bundle rotation is a scheduled process that aggregates attestations from a time period into a verifiable bundle. This enables long-term verification of keyless-signed artifacts beyond their certificate expiry.

## Rotation Schedule

### Default Schedule

```yaml
attestor:
  bundling:
    schedule:
      cron: "0 2 1 * *"   # Monthly on the 1st at 02:00 UTC
      cadence: monthly
      timezone: UTC
      skipWeekends: false
```

### Cadence Options

| Cadence | Period | Use Case |
|---------|--------|----------|
| `weekly` | Previous 7 days | High-volume environments |
| `monthly` | Previous month | Standard deployment (default) |
| `quarterly` | Previous quarter | Low-volume, compliance-focused |

## Manual Rotation

### Trigger Immediate Rotation

```bash
# Rotate current period
stella attestor bundle rotate

# Rotate specific period
stella attestor bundle rotate --start 2025-12-01 --end 2025-12-31

# Rotate for specific tenant
stella attestor bundle rotate --tenant tenant-gov
```

### API Trigger

```http
POST /api/v1/bundles
Content-Type: application/json

{
  "periodStart": "2025-12-01T00:00:00Z",
  "periodEnd": "2025-12-31T23:59:59Z",
  "tenantId": null,
  "signWithOrgKey": true,
  "orgKeyId": "org-signing-key-2025"
}
```

## Monitoring

### Key Metrics

| Metric | Description | Alert Threshold |
|--------|-------------|-----------------|
| `attestor_bundle_created_total` | Bundles created | N/A (informational) |
| `attestor_bundle_creation_duration_seconds` | Creation time | > 30 minutes |
| `attestor_bundle_attestations_count` | Attestations per bundle | > 10,000 |
| `attestor_bundle_size_bytes` | Bundle size | > 100 MB |
| `attestor_bundle_retention_deleted_total` | Expired bundles deleted | N/A |

### Grafana Dashboard

Import the attestor observability dashboard:
```bash
stella observability import --dashboard attestor-bundling
```

See: `docs/modules/attestor/operations/dashboards/attestor-observability.json`

### Health Check

```bash
# Check bundle rotation status
stella attestor bundle status

# Sample output:
# Last Rotation: 2025-12-01T02:00:00Z
# Next Scheduled: 2026-01-01T02:00:00Z
# Bundles This Month: 3
# Total Attestations Bundled: 4,521
# Status: Healthy
```

## Retention Policy

### Configuration

```yaml
attestor:
  bundling:
    retention:
      enabled: true
      defaultMonths: 24
      minimumMonths: 6
      maximumMonths: 120
      expiryAction: delete    # delete | archive | markOnly
      archiveStorageTier: glacier
      gracePeriodDays: 30
      notifyBeforeExpiry: true
      notifyDaysBeforeExpiry: 30
      maxBundlesPerRun: 100
```

### Retention Actions

| Action | Behavior |
|--------|----------|
| `delete` | Permanently remove expired bundles |
| `archive` | Move to cold storage (S3 Glacier) |
| `markOnly` | Mark as expired but retain |

### Manual Retention Enforcement

```bash
# Preview expired bundles
stella attestor bundle retention --dry-run

# Apply retention policy
stella attestor bundle retention --apply

# Force delete specific bundle
stella attestor bundle delete sha256:abc123...
```

## Troubleshooting

### Bundle Creation Failed

**Symptoms:** Rotation job completes with errors

**Check:**
```bash
# View recent rotation logs
stella logs --service attestor --filter "bundle rotation"

# Check attestor health
stella attestor health
```

**Common causes:**
1. Database connection issues
2. Insufficient attestations in period
3. Org key unavailable for signing

### Large Bundle Size

**Symptoms:** Bundle exceeds size limits or takes too long

**Solutions:**
1. Reduce `maxAttestationsPerBundle` to create multiple smaller bundles
2. Increase `queryBatchSize` for faster database queries
3. Enable compression for storage

```yaml
attestor:
  bundling:
    aggregation:
      maxAttestationsPerBundle: 5000
      queryBatchSize: 1000
```

### Org Key Signing Failed

**Symptoms:** Bundle created without org signature

**Check:**
```bash
# Verify org key availability
stella signer keys list --type org

# Test key signing
stella signer keys test org-signing-key-2025
```

**Solutions:**
1. Ensure KMS/HSM connectivity
2. Verify key permissions
3. Check key rotation schedule

### Retention Not Running

**Symptoms:** Expired bundles not being deleted

**Check:**
```bash
# Verify retention is enabled
stella attestor bundle retention --status

# Check for blocked bundles
stella attestor bundle list --status expired
```

**Solutions:**
1. Ensure `retention.enabled: true`
2. Check grace period configuration
3. Verify storage backend permissions

## Disaster Recovery

### Bundle Export

Export bundles for backup:

```bash
# Export all bundles from a period
stella attestor bundle export \
  --start 2025-01-01 \
  --end 2025-12-31 \
  --output /backup/bundles/

# Export specific bundle
stella attestor bundle export sha256:abc123 --output bundle.json
```

### Bundle Import

Restore bundles from backup:

```bash
# Import bundle file
stella attestor bundle import /backup/bundles/bundle-sha256-abc123.json

# Bulk import
stella attestor bundle import /backup/bundles/*.json
```

### Verification After Restore

```bash
# Verify imported bundle
stella attestor bundle verify sha256:abc123

# Verify all bundles
stella attestor bundle verify --all
```

## Runbooks

### Monthly Rotation Check

1. **Pre-rotation (1 day before):**
   ```bash
   stella attestor bundle preview --period 2025-12
   ```

2. **Post-rotation (rotation day + 1):**
   ```bash
   stella attestor bundle list --created-after 2025-12-01
   stella attestor bundle verify --period 2025-12
   ```

3. **Verify notifications sent:**
   Check Slack/Teams/Email for rotation summary

### Quarterly Audit

1. **List all bundles:**
   ```bash
   stella attestor bundle list --format json > audit-report.json
   ```

2. **Verify sample bundles:**
   ```bash
   # Random sample of 10%
   stella attestor bundle verify --sample 0.1
   ```

3. **Check retention compliance:**
   ```bash
   stella attestor bundle retention --audit
   ```

### Emergency Bundle Access

For urgent verification needs:

```bash
# Extract specific attestation from bundle
stella attestor bundle extract sha256:abc123 --entry-id uuid-1

# Verify attestation outside bundle
stella attestor verify --envelope attestation.dsse
```

## Related Documentation

- [Bundle Format Specification](../bundle-format.md)
- [Attestor Architecture](../architecture.md)
- [Observability Guide](./observability.md)
- [Air-Gap Operations](../airgap.md)