# Secret Detection Bundle Rotation

> **Audience:** Scanner operators, Security Guild, Offline Kit maintainers.
>
> **Related:** [secret-leak-detection.md](./secret-leak-detection.md)

## 1. Overview

Secret detection rule bundles are versioned, immutable artifacts that define the patterns used to detect leaked credentials. This document covers the versioning strategy, rotation procedures, and rollback instructions.

## 2. Versioning strategy

Bundles follow CalVer (Calendar Versioning) with the format `YYYY.MM`:

| Version | Release Type | Notes |
|---------|--------------|-------|
| `2026.01` | Monthly release | Standard monthly update |
| `2026.01.1` | Patch release | Critical rule fix within the month |
| `2026.02` | Monthly release | Next scheduled release |

**Version precedence:**
- `2026.02` > `2026.01.1` > `2026.01`
- Patch versions (`YYYY.MM.N`) are only used for critical fixes
- Monthly releases reset the patch counter

**Custom bundles:**
Organizations creating custom bundles should use a prefix to avoid conflicts:
- `myorg.2026.01` for organization-specific bundles
- Or semantic versioning: `1.0.0`, `1.1.0`, etc.

## 3. Release cadence

| Release Type | Frequency | Notification |
|--------------|-----------|--------------|
| Monthly release | First week of each month | Release notes, changelog |
| Patch release | As needed for critical rules | Security advisory |
| Breaking changes | Major version bump | Migration guide |

## 4. Rotation procedures

### 4.1 Downloading the new bundle

```bash
# From the Export Center or Offline Kit
curl -O https://export.stellaops.io/rules/secrets/2026.02/secrets.ruleset.manifest.json
curl -O https://export.stellaops.io/rules/secrets/2026.02/secrets.ruleset.rules.jsonl
curl -O https://export.stellaops.io/rules/secrets/2026.02/secrets.ruleset.dsse.json
```

For air-gapped environments, obtain the bundle from the Offline Kit media.

### 4.2 Verifying the bundle

Always verify bundles before deployment:

```bash
stella secrets bundle verify \
  --bundle ./2026.02 \
  --shared-secret-file /etc/stellaops/secrets-signing.key \
  --trusted-key-ids stellaops-secrets-signer
```

Expected output:
```
Bundle verified successfully.
  Bundle ID:      secrets.ruleset
  Version:        2026.02
  Rule count:     18
  Enabled rules:  18
  Signed by:      stellaops-secrets-signer
  Signed at:      2026-02-01T00:00:00Z
```

### 4.3 Staged rollout

For production environments, use a staged rollout:

**Stage 1: Canary (1 worker)**
```bash
# Deploy to canary worker
scp -r ./2026.02/* canary-worker:/opt/stellaops/plugins/scanner/analyzers/secrets/
ssh canary-worker 'systemctl restart stellaops-scanner-worker'

# Monitor for 24 hours
# Check logs, metrics, and finding counts
```

**Stage 2: Ring 1 (10% of workers)**
```bash
# Deploy to ring 1 workers
ansible-playbook -l ring1 deploy-secrets-bundle.yml -e bundle_version=2026.02
```

**Stage 3: Full rollout (all workers)**
```bash
# Deploy to all workers
ansible-playbook deploy-secrets-bundle.yml -e bundle_version=2026.02
```

### 4.4 Atomic deployment

For single-worker deployments or when downtime is acceptable:

```bash
# Stop the worker
systemctl stop stellaops-scanner-worker

# Backup current bundle
cp -r /opt/stellaops/plugins/scanner/analyzers/secrets{,.backup}

# Deploy new bundle
cp -r ./2026.02/* /opt/stellaops/plugins/scanner/analyzers/secrets/

# Start the worker
systemctl start stellaops-scanner-worker

# Verify startup
journalctl -u stellaops-scanner-worker | grep SecretsAnalyzerHost
```

### 4.5 Using symlinks (recommended)

For zero-downtime rotations, use the symlink pattern:

```bash
# Directory structure
/opt/stellaops/plugins/scanner/analyzers/secrets/
  bundles/
    2026.01/
      secrets.ruleset.manifest.json
      secrets.ruleset.rules.jsonl
      secrets.ruleset.dsse.json
    2026.02/
      secrets.ruleset.manifest.json
      secrets.ruleset.rules.jsonl
      secrets.ruleset.dsse.json
  current -> bundles/2026.02  # Symlink
```

Rotation with symlinks:
```bash
# Deploy new bundle (no restart needed yet)
cp -r ./2026.02 /opt/stellaops/plugins/scanner/analyzers/secrets/bundles/

# Atomic switch
ln -sfn bundles/2026.02 /opt/stellaops/plugins/scanner/analyzers/secrets/current

# Restart worker to pick up new bundle
systemctl restart stellaops-scanner-worker
```

## 5. Rollback procedures

### 5.1 Quick rollback

If issues are detected after deployment:

```bash
# With symlinks (fastest)
ln -sfn bundles/2026.01 /opt/stellaops/plugins/scanner/analyzers/secrets/current
systemctl restart stellaops-scanner-worker

# Without symlinks
cp -r /opt/stellaops/plugins/scanner/analyzers/secrets.backup/* \
      /opt/stellaops/plugins/scanner/analyzers/secrets/
systemctl restart stellaops-scanner-worker
```

### 5.2 Identifying rollback triggers

Roll back immediately if you observe:

| Symptom | Likely Cause | Action |
|---------|--------------|--------|
| Worker fails to start | Bundle corruption or invalid rules | Rollback + investigate |
| Finding count drops to zero | All rules disabled or regex errors | Rollback + check manifest |
| Finding count spikes 10x+ | Overly broad new patterns | Rollback + review rules |
| High CPU usage | Catastrophic regex backtracking | Rollback + report to Security Guild |
| Signature verification failures | Key mismatch or tampering | Rollback + verify bundle source |

### 5.3 Post-rollback verification

After rolling back:

```bash
# Verify worker is healthy
systemctl status stellaops-scanner-worker

# Check bundle version in logs
journalctl -u stellaops-scanner-worker | grep "Loaded bundle"

# Verify finding generation (run a test scan)
stella scan --target test-image:latest --secrets-only
```

## 6. Bundle retention

Retain previous bundle versions for rollback capability:

| Environment | Retention |
|-------------|-----------|
| Production | Last 3 versions |
| Staging | Last 2 versions |
| Development | Latest only |

Cleanup script:
```bash
#!/bin/bash
BUNDLE_DIR=/opt/stellaops/plugins/scanner/analyzers/secrets/bundles
KEEP=3

ls -dt ${BUNDLE_DIR}/*/ | tail -n +$((KEEP+1)) | xargs rm -rf
```

## 7. Monitoring rotation

Key metrics to monitor during rotation:

| Metric | Baseline | Alert Threshold |
|--------|----------|-----------------|
| `scanner.secret.finding_total` | Varies | +/- 50% from baseline |
| `scanner.secret.scan_duration_ms` | < 100ms | > 500ms |
| `scanner.secret.bundle_load_errors` | 0 | > 0 |
| Worker restart success | 100% | < 100% |

Prometheus alert example:
```yaml
- alert: SecretBundleRotationAnomaly
  expr: |
    abs(
      sum(rate(scanner_secret_finding_total[5m]))
      - sum(rate(scanner_secret_finding_total[5m] offset 1h))
    ) / sum(rate(scanner_secret_finding_total[5m] offset 1h)) > 0.5
  for: 15m
  labels:
    severity: warning
  annotations:
    summary: "Secret finding rate changed significantly after bundle rotation"
```

## 8. Air-gapped rotation

For air-gapped environments:

1. **Obtain bundle from secure media:**
   ```bash
   # Mount offline kit media
   mount /dev/sr0 /mnt/offline-kit

   # Copy bundle
   cp -r /mnt/offline-kit/rules/secrets/2026.02 \
         /opt/stellaops/plugins/scanner/analyzers/secrets/bundles/
   ```

2. **Verify with local secret:**
   ```bash
   stella secrets bundle verify \
     --bundle /opt/stellaops/plugins/scanner/analyzers/secrets/bundles/2026.02 \
     --shared-secret-file /etc/stellaops/offline-signing.key \
     --skip-rekor
   ```

3. **Follow standard rotation procedure (Section 4).**

## 9. Emergency procedures

### 9.1 Disabling secret detection

If secret detection must be disabled entirely:

```bash
# Disable via configuration
echo 'scanner.features.experimental.secret-leak-detection: false' >> /etc/stellaops/scanner.yaml

# Restart worker
systemctl restart stellaops-scanner-worker
```

### 9.2 Emergency rule disable

To disable a specific problematic rule without full rotation:

1. Edit the manifest to set `enabled: false` for the rule
2. This breaks signature verification (expected)
3. Configure worker to skip signature verification temporarily:
   ```yaml
   scanner:
     secrets:
       skipSignatureVerification: true  # TEMPORARY - re-enable after fix
   ```
4. Restart worker
5. Request emergency patch release from Security Guild

## 10. References

- [secret-leak-detection.md](./secret-leak-detection.md) - Main secret detection documentation
- [SPRINT_20260104_003_SCANNER_secret_rule_bundles.md](../../../implplan/SPRINT_20260104_003_SCANNER_secret_rule_bundles.md) - Implementation sprint
- [dsse-rekor-operator-guide.md](./dsse-rekor-operator-guide.md) - DSSE and Rekor verification