stella-ops.org/git.stella-ops.org
1
0
Fork 0
Code Issues Pull Requests Actions 2 Releases Wiki Activity
Files
d7be6ba34be1f39699adf954a164faeff09db8de
git.stella-ops.org/docs/modules/scanner/operations/secrets-bundle-rotation.md
StellaOps Bot 3098e84de4 save progress
2026-01-04 14:54:52 +02:00

8.5 KiB
Raw Blame History

Secret Detection Bundle Rotation

Audience: Scanner operators, Security Guild, Offline Kit maintainers.

Related: secret-leak-detection.md

1. Overview

Secret detection rule bundles are versioned, immutable artifacts that define the patterns used to detect leaked credentials. This document covers the versioning strategy, rotation procedures, and rollback instructions.

2. Versioning strategy

Bundles follow CalVer (Calendar Versioning) with the format YYYY.MM:

Version Release Type Notes
2026.01 Monthly release Standard monthly update
2026.01.1 Patch release Critical rule fix within the month
2026.02 Monthly release Next scheduled release

Version precedence:

  • 2026.02 > 2026.01.1 > 2026.01
  • Patch versions (YYYY.MM.N) are only used for critical fixes
  • Monthly releases reset the patch counter

Custom bundles: Organizations creating custom bundles should use a prefix to avoid conflicts:

  • myorg.2026.01 for organization-specific bundles
  • Or semantic versioning: 1.0.0, 1.1.0, etc.

3. Release cadence

Release Type Frequency Notification
Monthly release First week of each month Release notes, changelog
Patch release As needed for critical rules Security advisory
Breaking changes Major version bump Migration guide

4. Rotation procedures

4.1 Downloading the new bundle

# From the Export Center or Offline Kit
curl -O https://export.stellaops.io/rules/secrets/2026.02/secrets.ruleset.manifest.json
curl -O https://export.stellaops.io/rules/secrets/2026.02/secrets.ruleset.rules.jsonl
curl -O https://export.stellaops.io/rules/secrets/2026.02/secrets.ruleset.dsse.json

For air-gapped environments, obtain the bundle from the Offline Kit media.

4.2 Verifying the bundle

Always verify bundles before deployment:

stella secrets bundle verify \
  --bundle ./2026.02 \
  --shared-secret-file /etc/stellaops/secrets-signing.key \
  --trusted-key-ids stellaops-secrets-signer

Expected output:

Bundle verified successfully.
  Bundle ID:      secrets.ruleset
  Version:        2026.02
  Rule count:     18
  Enabled rules:  18
  Signed by:      stellaops-secrets-signer
  Signed at:      2026-02-01T00:00:00Z

4.3 Staged rollout

For production environments, use a staged rollout:

Stage 1: Canary (1 worker)

# Deploy to canary worker
scp -r ./2026.02/* canary-worker:/opt/stellaops/plugins/scanner/analyzers/secrets/
ssh canary-worker 'systemctl restart stellaops-scanner-worker'

# Monitor for 24 hours
# Check logs, metrics, and finding counts

Stage 2: Ring 1 (10% of workers)

# Deploy to ring 1 workers
ansible-playbook -l ring1 deploy-secrets-bundle.yml -e bundle_version=2026.02

Stage 3: Full rollout (all workers)

# Deploy to all workers
ansible-playbook deploy-secrets-bundle.yml -e bundle_version=2026.02

4.4 Atomic deployment

For single-worker deployments or when downtime is acceptable:

# Stop the worker
systemctl stop stellaops-scanner-worker

# Backup current bundle
cp -r /opt/stellaops/plugins/scanner/analyzers/secrets{,.backup}

# Deploy new bundle
cp -r ./2026.02/* /opt/stellaops/plugins/scanner/analyzers/secrets/

# Start the worker
systemctl start stellaops-scanner-worker

# Verify startup
journalctl -u stellaops-scanner-worker | grep SecretsAnalyzerHost

4.5 Using symlinks (recommended)

For zero-downtime rotations, use the symlink pattern:

# Directory structure
/opt/stellaops/plugins/scanner/analyzers/secrets/
  bundles/
    2026.01/
      secrets.ruleset.manifest.json
      secrets.ruleset.rules.jsonl
      secrets.ruleset.dsse.json
    2026.02/
      secrets.ruleset.manifest.json
      secrets.ruleset.rules.jsonl
      secrets.ruleset.dsse.json
  current -> bundles/2026.02  # Symlink

Rotation with symlinks:

# Deploy new bundle (no restart needed yet)
cp -r ./2026.02 /opt/stellaops/plugins/scanner/analyzers/secrets/bundles/

# Atomic switch
ln -sfn bundles/2026.02 /opt/stellaops/plugins/scanner/analyzers/secrets/current

# Restart worker to pick up new bundle
systemctl restart stellaops-scanner-worker

5. Rollback procedures

5.1 Quick rollback

If issues are detected after deployment:

# With symlinks (fastest)
ln -sfn bundles/2026.01 /opt/stellaops/plugins/scanner/analyzers/secrets/current
systemctl restart stellaops-scanner-worker

# Without symlinks
cp -r /opt/stellaops/plugins/scanner/analyzers/secrets.backup/* \
      /opt/stellaops/plugins/scanner/analyzers/secrets/
systemctl restart stellaops-scanner-worker

5.2 Identifying rollback triggers

Roll back immediately if you observe:

Symptom Likely Cause Action
Worker fails to start Bundle corruption or invalid rules Rollback + investigate
Finding count drops to zero All rules disabled or regex errors Rollback + check manifest
Finding count spikes 10x+ Overly broad new patterns Rollback + review rules
High CPU usage Catastrophic regex backtracking Rollback + report to Security Guild
Signature verification failures Key mismatch or tampering Rollback + verify bundle source

5.3 Post-rollback verification

After rolling back:

# Verify worker is healthy
systemctl status stellaops-scanner-worker

# Check bundle version in logs
journalctl -u stellaops-scanner-worker | grep "Loaded bundle"

# Verify finding generation (run a test scan)
stella scan --target test-image:latest --secrets-only

6. Bundle retention

Retain previous bundle versions for rollback capability:

Environment Retention
Production Last 3 versions
Staging Last 2 versions
Development Latest only

Cleanup script:

#!/bin/bash
BUNDLE_DIR=/opt/stellaops/plugins/scanner/analyzers/secrets/bundles
KEEP=3

ls -dt ${BUNDLE_DIR}/*/ | tail -n +$((KEEP+1)) | xargs rm -rf

7. Monitoring rotation

Key metrics to monitor during rotation:

Metric Baseline Alert Threshold
scanner.secret.finding_total Varies +/- 50% from baseline
scanner.secret.scan_duration_ms < 100ms > 500ms
scanner.secret.bundle_load_errors 0 > 0
Worker restart success 100% < 100%

Prometheus alert example:

- alert: SecretBundleRotationAnomaly
  expr: |
    abs(
      sum(rate(scanner_secret_finding_total[5m]))
      - sum(rate(scanner_secret_finding_total[5m] offset 1h))
    ) / sum(rate(scanner_secret_finding_total[5m] offset 1h)) > 0.5
  for: 15m
  labels:
    severity: warning
  annotations:
    summary: "Secret finding rate changed significantly after bundle rotation"

8. Air-gapped rotation

For air-gapped environments:

  1. Obtain bundle from secure media:

    # Mount offline kit media
mount /dev/sr0 /mnt/offline-kit

# Copy bundle
cp -r /mnt/offline-kit/rules/secrets/2026.02 \
      /opt/stellaops/plugins/scanner/analyzers/secrets/bundles/

  2. Verify with local secret:

    stella secrets bundle verify \
  --bundle /opt/stellaops/plugins/scanner/analyzers/secrets/bundles/2026.02 \
  --shared-secret-file /etc/stellaops/offline-signing.key \
  --skip-rekor

  3. Follow standard rotation procedure (Section 4).

9. Emergency procedures

9.1 Disabling secret detection

If secret detection must be disabled entirely:

# Disable via configuration
echo 'scanner.features.experimental.secret-leak-detection: false' >> /etc/stellaops/scanner.yaml

# Restart worker
systemctl restart stellaops-scanner-worker

9.2 Emergency rule disable

To disable a specific problematic rule without full rotation:

  1. Edit the manifest to set enabled: false for the rule
  2. This breaks signature verification (expected)
  3. Configure worker to skip signature verification temporarily: 
    scanner:
  secrets:
    skipSignatureVerification: true  # TEMPORARY - re-enable after fix
  4. Restart worker
  5. Request emergency patch release from Security Guild

10. References