save progress

2026-01-04 14:54:52 +02:00
parent c49b03a254
commit 3098e84de4
132 changed files with 19783 additions and 31 deletions
--- a/docs/modules/scanner/operations/secrets-bundle-rotation.md
+++ b/docs/modules/scanner/operations/secrets-bundle-rotation.md
@@ -0,0 +1,298 @@
+# Secret Detection Bundle Rotation
+
+> **Audience:** Scanner operators, Security Guild, Offline Kit maintainers.
+>
+> **Related:** [secret-leak-detection.md](./secret-leak-detection.md)
+
+## 1. Overview
+
+Secret detection rule bundles are versioned, immutable artifacts that define the patterns used to detect leaked credentials. This document covers the versioning strategy, rotation procedures, and rollback instructions.
+
+## 2. Versioning strategy
+
+Bundles follow CalVer (Calendar Versioning) with the format `YYYY.MM`:
+
+| Version | Release Type | Notes |
+|---------|--------------|-------|
+| `2026.01` | Monthly release | Standard monthly update |
+| `2026.01.1` | Patch release | Critical rule fix within the month |
+| `2026.02` | Monthly release | Next scheduled release |
+
+**Version precedence:**
+- `2026.02` > `2026.01.1` > `2026.01`
+- Patch versions (`YYYY.MM.N`) are only used for critical fixes
+- Monthly releases reset the patch counter
+
+**Custom bundles:**
+Organizations creating custom bundles should use a prefix to avoid conflicts:
+- `myorg.2026.01` for organization-specific bundles
+- Or semantic versioning: `1.0.0`, `1.1.0`, etc.
+
+## 3. Release cadence
+
+| Release Type | Frequency | Notification |
+|--------------|-----------|--------------|
+| Monthly release | First week of each month | Release notes, changelog |
+| Patch release | As needed for critical rules | Security advisory |
+| Breaking changes | Major version bump | Migration guide |
+
+## 4. Rotation procedures
+
+### 4.1 Downloading the new bundle
+
+```bash
+# From the Export Center or Offline Kit
+curl -O https://export.stellaops.io/rules/secrets/2026.02/secrets.ruleset.manifest.json
+curl -O https://export.stellaops.io/rules/secrets/2026.02/secrets.ruleset.rules.jsonl
+curl -O https://export.stellaops.io/rules/secrets/2026.02/secrets.ruleset.dsse.json
+```
+
+For air-gapped environments, obtain the bundle from the Offline Kit media.
+
+### 4.2 Verifying the bundle
+
+Always verify bundles before deployment:
+
+```bash
+stella secrets bundle verify \
+  --bundle ./2026.02 \
+  --shared-secret-file /etc/stellaops/secrets-signing.key \
+  --trusted-key-ids stellaops-secrets-signer
+```
+
+Expected output:
+```
+Bundle verified successfully.
+  Bundle ID:      secrets.ruleset
+  Version:        2026.02
+  Rule count:     18
+  Enabled rules:  18
+  Signed by:      stellaops-secrets-signer
+  Signed at:      2026-02-01T00:00:00Z
+```
+
+### 4.3 Staged rollout
+
+For production environments, use a staged rollout:
+
+**Stage 1: Canary (1 worker)**
+```bash
+# Deploy to canary worker
+scp -r ./2026.02/* canary-worker:/opt/stellaops/plugins/scanner/analyzers/secrets/
+ssh canary-worker 'systemctl restart stellaops-scanner-worker'
+
+# Monitor for 24 hours
+# Check logs, metrics, and finding counts
+```
+
+**Stage 2: Ring 1 (10% of workers)**
+```bash
+# Deploy to ring 1 workers
+ansible-playbook -l ring1 deploy-secrets-bundle.yml -e bundle_version=2026.02
+```
+
+**Stage 3: Full rollout (all workers)**
+```bash
+# Deploy to all workers
+ansible-playbook deploy-secrets-bundle.yml -e bundle_version=2026.02
+```
+
+### 4.4 Atomic deployment
+
+For single-worker deployments or when downtime is acceptable:
+
+```bash
+# Stop the worker
+systemctl stop stellaops-scanner-worker
+
+# Backup current bundle
+cp -r /opt/stellaops/plugins/scanner/analyzers/secrets{,.backup}
+
+# Deploy new bundle
+cp -r ./2026.02/* /opt/stellaops/plugins/scanner/analyzers/secrets/
+
+# Start the worker
+systemctl start stellaops-scanner-worker
+
+# Verify startup
+journalctl -u stellaops-scanner-worker | grep SecretsAnalyzerHost
+```
+
+### 4.5 Using symlinks (recommended)
+
+For zero-downtime rotations, use the symlink pattern:
+
+```bash
+# Directory structure
+/opt/stellaops/plugins/scanner/analyzers/secrets/
+  bundles/
+    2026.01/
+      secrets.ruleset.manifest.json
+      secrets.ruleset.rules.jsonl
+      secrets.ruleset.dsse.json
+    2026.02/
+      secrets.ruleset.manifest.json
+      secrets.ruleset.rules.jsonl
+      secrets.ruleset.dsse.json
+  current -> bundles/2026.02  # Symlink
+```
+
+Rotation with symlinks:
+```bash
+# Deploy new bundle (no restart needed yet)
+cp -r ./2026.02 /opt/stellaops/plugins/scanner/analyzers/secrets/bundles/
+
+# Atomic switch
+ln -sfn bundles/2026.02 /opt/stellaops/plugins/scanner/analyzers/secrets/current
+
+# Restart worker to pick up new bundle
+systemctl restart stellaops-scanner-worker
+```
+
+## 5. Rollback procedures
+
+### 5.1 Quick rollback
+
+If issues are detected after deployment:
+
+```bash
+# With symlinks (fastest)
+ln -sfn bundles/2026.01 /opt/stellaops/plugins/scanner/analyzers/secrets/current
+systemctl restart stellaops-scanner-worker
+
+# Without symlinks
+cp -r /opt/stellaops/plugins/scanner/analyzers/secrets.backup/* \
+      /opt/stellaops/plugins/scanner/analyzers/secrets/
+systemctl restart stellaops-scanner-worker
+```
+
+### 5.2 Identifying rollback triggers
+
+Roll back immediately if you observe:
+
+| Symptom | Likely Cause | Action |
+|---------|--------------|--------|
+| Worker fails to start | Bundle corruption or invalid rules | Rollback + investigate |
+| Finding count drops to zero | All rules disabled or regex errors | Rollback + check manifest |
+| Finding count spikes 10x+ | Overly broad new patterns | Rollback + review rules |
+| High CPU usage | Catastrophic regex backtracking | Rollback + report to Security Guild |
+| Signature verification failures | Key mismatch or tampering | Rollback + verify bundle source |
+
+### 5.3 Post-rollback verification
+
+After rolling back:
+
+```bash
+# Verify worker is healthy
+systemctl status stellaops-scanner-worker
+
+# Check bundle version in logs
+journalctl -u stellaops-scanner-worker | grep "Loaded bundle"
+
+# Verify finding generation (run a test scan)
+stella scan --target test-image:latest --secrets-only
+```
+
+## 6. Bundle retention
+
+Retain previous bundle versions for rollback capability:
+
+| Environment | Retention |
+|-------------|-----------|
+| Production | Last 3 versions |
+| Staging | Last 2 versions |
+| Development | Latest only |
+
+Cleanup script:
+```bash
+#!/bin/bash
+BUNDLE_DIR=/opt/stellaops/plugins/scanner/analyzers/secrets/bundles
+KEEP=3
+
+ls -dt ${BUNDLE_DIR}/*/ | tail -n +$((KEEP+1)) | xargs rm -rf
+```
+
+## 7. Monitoring rotation
+
+Key metrics to monitor during rotation:
+
+| Metric | Baseline | Alert Threshold |
+|--------|----------|-----------------|
+| `scanner.secret.finding_total` | Varies | +/- 50% from baseline |
+| `scanner.secret.scan_duration_ms` | < 100ms | > 500ms |
+| `scanner.secret.bundle_load_errors` | 0 | > 0 |
+| Worker restart success | 100% | < 100% |
+
+Prometheus alert example:
+```yaml
+- alert: SecretBundleRotationAnomaly
+  expr: |
+    abs(
+      sum(rate(scanner_secret_finding_total[5m]))
+      - sum(rate(scanner_secret_finding_total[5m] offset 1h))
+    ) / sum(rate(scanner_secret_finding_total[5m] offset 1h)) > 0.5
+  for: 15m
+  labels:
+    severity: warning
+  annotations:
+    summary: "Secret finding rate changed significantly after bundle rotation"
+```
+
+## 8. Air-gapped rotation
+
+For air-gapped environments:
+
+1. **Obtain bundle from secure media:**
+   ```bash
+   # Mount offline kit media
+   mount /dev/sr0 /mnt/offline-kit
+
+   # Copy bundle
+   cp -r /mnt/offline-kit/rules/secrets/2026.02 \
+         /opt/stellaops/plugins/scanner/analyzers/secrets/bundles/
+   ```
+
+2. **Verify with local secret:**
+   ```bash
+   stella secrets bundle verify \
+     --bundle /opt/stellaops/plugins/scanner/analyzers/secrets/bundles/2026.02 \
+     --shared-secret-file /etc/stellaops/offline-signing.key \
+     --skip-rekor
+   ```
+
+3. **Follow standard rotation procedure (Section 4).**
+
+## 9. Emergency procedures
+
+### 9.1 Disabling secret detection
+
+If secret detection must be disabled entirely:
+
+```bash
+# Disable via configuration
+echo 'scanner.features.experimental.secret-leak-detection: false' >> /etc/stellaops/scanner.yaml
+
+# Restart worker
+systemctl restart stellaops-scanner-worker
+```
+
+### 9.2 Emergency rule disable
+
+To disable a specific problematic rule without full rotation:
+
+1. Edit the manifest to set `enabled: false` for the rule
+2. This breaks signature verification (expected)
+3. Configure worker to skip signature verification temporarily:
+   ```yaml
+   scanner:
+     secrets:
+       skipSignatureVerification: true  # TEMPORARY - re-enable after fix
+   ```
+4. Restart worker
+5. Request emergency patch release from Security Guild
+
+## 10. References
+
+- [secret-leak-detection.md](./secret-leak-detection.md) - Main secret detection documentation
+- [SPRINT_20260104_003_SCANNER_secret_rule_bundles.md](../../../implplan/SPRINT_20260104_003_SCANNER_secret_rule_bundles.md) - Implementation sprint
+- [dsse-rekor-operator-guide.md](./dsse-rekor-operator-guide.md) - DSSE and Rekor verification