stella-ops.org/git.stella-ops.org
1
0
Fork 0
Code Issues Pull Requests Actions 2 Releases Wiki Activity

save progress

Browse Source
This commit is contained in:
StellaOps Bot
2026-01-04 14:54:52 +02:00
parent c49b03a254
commit 3098e84de4
132 changed files with 19783 additions and 31 deletions
Download Patch File Download Diff File Expand all files Collapse all files

160
docs/modules/scanner/operations/secret-leak-detection.md
View File

@@ -1,9 +1,25 @@
# Secret Leak Detection (Scanner Operations)
> **Status:** Preview (Sprint 132). Requires `SCANNER-ENG-0007`/`POLICY-READINESS-0001` release bundle and the experimental flag `secret-leak-detection`.
> **Status:** PLANNED - Implementation in progress. See implementation sprints below.
>
> **Previous status:** Preview (Sprint 132). Requires `SCANNER-ENG-0007`/`POLICY-READINESS-0001` release bundle and the experimental flag `secret-leak-detection`.
>
> **Audience:** Scanner operators, Security Guild, Docs Guild, Offline Kit maintainers.
## Implementation Status
| Component | Status | Sprint |
|-----------|--------|--------|
| `StellaOps.Scanner.Analyzers.Secrets` plugin | NOT IMPLEMENTED | [SPRINT_20260104_002](../../../implplan/SPRINT_20260104_002_SCANNER_secret_leak_detection_core.md) |
| Rule bundle infrastructure | NOT IMPLEMENTED | [SPRINT_20260104_003](../../../implplan/SPRINT_20260104_003_SCANNER_secret_rule_bundles.md) |
| Policy DSL predicates (`secret.*`) | NOT IMPLEMENTED | [SPRINT_20260104_004](../../../implplan/SPRINT_20260104_004_POLICY_secret_dsl_integration.md) |
| Offline Kit integration | NOT IMPLEMENTED | [SPRINT_20260104_005](../../../implplan/SPRINT_20260104_005_AIRGAP_secret_offline_kit.md) |
| Surface.Secrets (credential delivery) | IMPLEMENTED | N/A (already complete) |
**Note:** The remainder of this document describes the TARGET SPECIFICATION for secret leak detection. The feature is not yet available. Surface.Secrets (operational credential management) is fully implemented and separate from secret leak detection.
---
## 1. Scope & goals
- Introduce the **`StellaOps.Scanner.Analyzers.Secrets`** plug-in, which executes deterministic rule bundles against layer content during scans.
@@ -29,29 +45,107 @@ Rule bundles ship in the Export Center / Offline Kit under `offline/rules/secret
| --- | --- | --- |
| `secrets.ruleset.manifest.json` | Lists rule IDs, versions, severity defaults, and hash digests. | Consume during policy drift audits. |
| `secrets.ruleset.rules.jsonl` | Newline-delimited definitions (regex/entropy metadata, masking hints). | Loaded by the analyzer at startup. |
| `secrets.ruleset.dsse.json` | DSSE envelope (Signer certificate chain + Attestor proof). | Verify before distributing bundles. |
| `secrets.ruleset.dsse.json` | DSSE envelope (HMAC-SHA256 signature). | Verify before distributing bundles. |
Verification checklist (`stella excititor verify` talks to the configured Attestor service):
### 3.1 Creating custom bundles
Organizations can create custom rule bundles with additional detection patterns:
```bash
# Create a bundle from rule definition files
stella secrets bundle create \
--output ./bundles/custom-2026.01 \
--bundle-id custom.secrets.ruleset \
--version 2026.01 \
--rules ./custom-rules/*.json
# Create and sign in one step
stella secrets bundle create \
--output ./bundles/custom-2026.01 \
--bundle-id custom.secrets.ruleset \
--version 2026.01 \
--rules ./custom-rules/*.json \
--sign \
--key-id my-org-secrets-signer \
--shared-secret-file /path/to/signing.key
```
Rule definition files must follow the JSON schema:
```json
{
"id": "myorg.secrets.internal-api-key",
"version": "1.0.0",
"name": "Internal API Key",
"description": "Detects internal API keys with MYORG prefix",
"type": "regex",
"pattern": "MYORG_[A-Z0-9]{32}",
"severity": "high",
"confidence": "high",
"keywords": ["MYORG_"],
"filePatterns": ["*.env", "*.yaml", "*.json"],
"enabled": true
}
```
**Rule ID requirements:**
- Must be namespaced (e.g., `myorg.secrets.rule-name`)
- Must start with lowercase letter
- May contain lowercase letters, digits, dots, and hyphens
### 3.2 Verifying bundles
Verify bundle integrity and signature before deployment:
```bash
# Basic verification (checks SHA-256 integrity)
stella secrets bundle verify \
--bundle ./bundles/2026.01
# Full verification with signature check
stella secrets bundle verify \
--bundle ./bundles/2026.01 \
--shared-secret-file /path/to/signing.key
# Verify with trusted key list
stella secrets bundle verify \
--bundle ./bundles/2026.01 \
--shared-secret-file /path/to/signing.key \
--trusted-key-ids stellaops-secrets-signer,my-org-signer
```
For air-gapped environments, use the local verification mode (no network calls):
```bash
stella secrets bundle verify \
--bundle ./bundles/2026.01 \
--shared-secret-file /path/to/signing.key \
--skip-rekor
```
Alternatively, use `stella excititor verify` for Attestor-based verification:
```bash
stella excititor verify \
--attestation offline/rules/secrets/2025.11/secrets.ruleset.dsse.json \
--digest $(sha256sum offline/rules/secrets/2025.11/secrets.ruleset.rules.jsonl | cut -d' ' -f1)
--attestation offline/rules/secrets/2026.01/secrets.ruleset.dsse.json \
--digest $(sha256sum offline/rules/secrets/2026.01/secrets.ruleset.rules.jsonl | cut -d' ' -f1)
```
For air-gapped environments point the CLI at the Offline Kit Attestor mirror (for example `STELLA_ATTESTOR_URL=http://attestor.offline.local`) before running the command. The Attestor instance validates the DSSE envelope against the mirrored Rekor log and embedded certificate chain; no public network access is required.
### 3.3 Deploying bundles
Once verified, copy the manifest + rules to the worker:
Once verified, copy the bundle to the worker:
```
/opt/stellaops/plugins/scanner/analyzers/secrets/
├── secrets.ruleset.manifest.json
├── secrets.ruleset.rules.jsonl
└── secrets.ruleset.dsse.json
|- secrets.ruleset.manifest.json
|- secrets.ruleset.rules.jsonl
|- secrets.ruleset.dsse.json
```
Restart the worker so the analyzer reloads the updated bundle. Bundles are immutable; upgrading requires replacing all three files and restarting.
See [secrets-bundle-rotation.md](./secrets-bundle-rotation.md) for rotation procedures and rollback instructions.
## 4. Enabling the analyzer
1. **Toggle the feature flag** (WebService + Worker):
@@ -161,6 +255,52 @@ rule low_confidence_warn priority 20 {
| No findings despite seeded secrets | Ensure bundle hash matches manifest. Run worker with `--secrets-trace` (debug build) to log matched rules locally. |
| Policy marks findings as unknown | Upgrade tenant policies to include `secret.*` helpers; older policies silently drop the namespace. |
| Air-gapped verification fails | Ensure `STELLA_ATTESTOR_URL` points to the Offline Kit Attestor mirror and rerun `stella excititor verify --attestation <file> --digest <sha256>`. |
| Signature verification failed | Check shared secret matches signing key. Verify DSSE envelope exists and is not corrupted. See signature troubleshooting below. |
| Bundle integrity check failed | Rules file was modified after signing. Re-download bundle or rebuild from sources. |
| Key not in trusted list | Add signer key ID to `--trusted-key-ids` or update `scanner.secrets.trustedKeyIds` configuration. |
### 7.1 Signature verification troubleshooting
**"Signature verification failed" error:**
1. Verify the shared secret is correct:
```bash
# Check secret file exists and is readable
cat /path/to/signing.key | wc -c
# Should output the key length (typically 32-64 bytes for base64-encoded keys)
```
2. Check the DSSE envelope format:
```bash
# Verify envelope is valid JSON
jq . ./bundles/2026.01/secrets.ruleset.dsse.json
```
3. Confirm manifest matches envelope payload:
```bash
# The envelope payload (base64url-decoded) should match the manifest content
# excluding the signatures field
```
**"Rules file integrity check failed" error:**
1. Recompute the SHA-256 hash:
```bash
sha256sum ./bundles/2026.01/secrets.ruleset.rules.jsonl
```
2. Compare with manifest:
```bash
jq -r '.integrity.rulesSha256' ./bundles/2026.01/secrets.ruleset.manifest.json
```
3. If hashes differ, the rules file was modified. Re-download or rebuild the bundle.
**"Bundle is not signed" error:**
The bundle was created without the `--sign` flag. Either:
- Rebuild with signing: `stella secrets bundle create ... --sign --key-id <key>`
- Skip signature verification: `--skip-signature-verification` (not recommended for production)
## 8. References

298
docs/modules/scanner/operations/secrets-bundle-rotation.md Normal file
View File

@@ -0,0 +1,298 @@
# Secret Detection Bundle Rotation
> **Audience:** Scanner operators, Security Guild, Offline Kit maintainers.
>
> **Related:** [secret-leak-detection.md](./secret-leak-detection.md)
## 1. Overview
Secret detection rule bundles are versioned, immutable artifacts that define the patterns used to detect leaked credentials. This document covers the versioning strategy, rotation procedures, and rollback instructions.
## 2. Versioning strategy
Bundles follow CalVer (Calendar Versioning) with the format `YYYY.MM`:
| Version | Release Type | Notes |
|---------|--------------|-------|
| `2026.01` | Monthly release | Standard monthly update |
| `2026.01.1` | Patch release | Critical rule fix within the month |
| `2026.02` | Monthly release | Next scheduled release |
**Version precedence:**
- `2026.02` > `2026.01.1` > `2026.01`
- Patch versions (`YYYY.MM.N`) are only used for critical fixes
- Monthly releases reset the patch counter
**Custom bundles:**
Organizations creating custom bundles should use a prefix to avoid conflicts:
- `myorg.2026.01` for organization-specific bundles
- Or semantic versioning: `1.0.0`, `1.1.0`, etc.
## 3. Release cadence
| Release Type | Frequency | Notification |
|--------------|-----------|--------------|
| Monthly release | First week of each month | Release notes, changelog |
| Patch release | As needed for critical rules | Security advisory |
| Breaking changes | Major version bump | Migration guide |
## 4. Rotation procedures
### 4.1 Downloading the new bundle
```bash
# From the Export Center or Offline Kit
curl -O https://export.stellaops.io/rules/secrets/2026.02/secrets.ruleset.manifest.json
curl -O https://export.stellaops.io/rules/secrets/2026.02/secrets.ruleset.rules.jsonl
curl -O https://export.stellaops.io/rules/secrets/2026.02/secrets.ruleset.dsse.json
```
For air-gapped environments, obtain the bundle from the Offline Kit media.
### 4.2 Verifying the bundle
Always verify bundles before deployment:
```bash
stella secrets bundle verify \
--bundle ./2026.02 \
--shared-secret-file /etc/stellaops/secrets-signing.key \
--trusted-key-ids stellaops-secrets-signer
```
Expected output:
```
Bundle verified successfully.
Bundle ID: secrets.ruleset
Version: 2026.02
Rule count: 18
Enabled rules: 18
Signed by: stellaops-secrets-signer
Signed at: 2026-02-01T00:00:00Z
```
### 4.3 Staged rollout
For production environments, use a staged rollout:
**Stage 1: Canary (1 worker)**
```bash
# Deploy to canary worker
scp -r ./2026.02/* canary-worker:/opt/stellaops/plugins/scanner/analyzers/secrets/
ssh canary-worker 'systemctl restart stellaops-scanner-worker'
# Monitor for 24 hours
# Check logs, metrics, and finding counts
```
**Stage 2: Ring 1 (10% of workers)**
```bash
# Deploy to ring 1 workers
ansible-playbook -l ring1 deploy-secrets-bundle.yml -e bundle_version=2026.02
```
**Stage 3: Full rollout (all workers)**
```bash
# Deploy to all workers
ansible-playbook deploy-secrets-bundle.yml -e bundle_version=2026.02
```
### 4.4 Atomic deployment
For single-worker deployments or when downtime is acceptable:
```bash
# Stop the worker
systemctl stop stellaops-scanner-worker
# Backup current bundle
cp -r /opt/stellaops/plugins/scanner/analyzers/secrets{,.backup}
# Deploy new bundle
cp -r ./2026.02/* /opt/stellaops/plugins/scanner/analyzers/secrets/
# Start the worker
systemctl start stellaops-scanner-worker
# Verify startup
journalctl -u stellaops-scanner-worker | grep SecretsAnalyzerHost
```
### 4.5 Using symlinks (recommended)
For zero-downtime rotations, use the symlink pattern:
```bash
# Directory structure
/opt/stellaops/plugins/scanner/analyzers/secrets/
bundles/
2026.01/
secrets.ruleset.manifest.json
secrets.ruleset.rules.jsonl
secrets.ruleset.dsse.json
2026.02/
secrets.ruleset.manifest.json
secrets.ruleset.rules.jsonl
secrets.ruleset.dsse.json
current -> bundles/2026.02 # Symlink
```
Rotation with symlinks:
```bash
# Deploy new bundle (no restart needed yet)
cp -r ./2026.02 /opt/stellaops/plugins/scanner/analyzers/secrets/bundles/
# Atomic switch
ln -sfn bundles/2026.02 /opt/stellaops/plugins/scanner/analyzers/secrets/current
# Restart worker to pick up new bundle
systemctl restart stellaops-scanner-worker
```
## 5. Rollback procedures
### 5.1 Quick rollback
If issues are detected after deployment:
```bash
# With symlinks (fastest)
ln -sfn bundles/2026.01 /opt/stellaops/plugins/scanner/analyzers/secrets/current
systemctl restart stellaops-scanner-worker
# Without symlinks
cp -r /opt/stellaops/plugins/scanner/analyzers/secrets.backup/* \
/opt/stellaops/plugins/scanner/analyzers/secrets/
systemctl restart stellaops-scanner-worker
```
### 5.2 Identifying rollback triggers
Roll back immediately if you observe:
| Symptom | Likely Cause | Action |
|---------|--------------|--------|
| Worker fails to start | Bundle corruption or invalid rules | Rollback + investigate |
| Finding count drops to zero | All rules disabled or regex errors | Rollback + check manifest |
| Finding count spikes 10x+ | Overly broad new patterns | Rollback + review rules |
| High CPU usage | Catastrophic regex backtracking | Rollback + report to Security Guild |
| Signature verification failures | Key mismatch or tampering | Rollback + verify bundle source |
### 5.3 Post-rollback verification
After rolling back:
```bash
# Verify worker is healthy
systemctl status stellaops-scanner-worker
# Check bundle version in logs
journalctl -u stellaops-scanner-worker | grep "Loaded bundle"
# Verify finding generation (run a test scan)
stella scan --target test-image:latest --secrets-only
```
## 6. Bundle retention
Retain previous bundle versions for rollback capability:
| Environment | Retention |
|-------------|-----------|
| Production | Last 3 versions |
| Staging | Last 2 versions |
| Development | Latest only |
Cleanup script:
```bash
#!/bin/bash
BUNDLE_DIR=/opt/stellaops/plugins/scanner/analyzers/secrets/bundles
KEEP=3
ls -dt ${BUNDLE_DIR}/*/ | tail -n +$((KEEP+1)) | xargs rm -rf
```
## 7. Monitoring rotation
Key metrics to monitor during rotation:
| Metric | Baseline | Alert Threshold |
|--------|----------|-----------------|
| `scanner.secret.finding_total` | Varies | +/- 50% from baseline |
| `scanner.secret.scan_duration_ms` | < 100ms | > 500ms |
| `scanner.secret.bundle_load_errors` | 0 | > 0 |
| Worker restart success | 100% | < 100% |
Prometheus alert example:
```yaml
- alert: SecretBundleRotationAnomaly
expr: |
abs(
sum(rate(scanner_secret_finding_total[5m]))
- sum(rate(scanner_secret_finding_total[5m] offset 1h))
) / sum(rate(scanner_secret_finding_total[5m] offset 1h)) > 0.5
for: 15m
labels:
severity: warning
annotations:
summary: "Secret finding rate changed significantly after bundle rotation"
```
## 8. Air-gapped rotation
For air-gapped environments:
1. **Obtain bundle from secure media:**
```bash
# Mount offline kit media
mount /dev/sr0 /mnt/offline-kit
# Copy bundle
cp -r /mnt/offline-kit/rules/secrets/2026.02 \
/opt/stellaops/plugins/scanner/analyzers/secrets/bundles/
```
2. **Verify with local secret:**
```bash
stella secrets bundle verify \
--bundle /opt/stellaops/plugins/scanner/analyzers/secrets/bundles/2026.02 \
--shared-secret-file /etc/stellaops/offline-signing.key \
--skip-rekor
```
3. **Follow standard rotation procedure (Section 4).**
## 9. Emergency procedures
### 9.1 Disabling secret detection
If secret detection must be disabled entirely:
```bash
# Disable via configuration
echo 'scanner.features.experimental.secret-leak-detection: false' >> /etc/stellaops/scanner.yaml
# Restart worker
systemctl restart stellaops-scanner-worker
```
### 9.2 Emergency rule disable
To disable a specific problematic rule without full rotation:
1. Edit the manifest to set `enabled: false` for the rule
2. This breaks signature verification (expected)
3. Configure worker to skip signature verification temporarily:
```yaml
scanner:
secrets:
skipSignatureVerification: true # TEMPORARY - re-enable after fix
```
4. Restart worker
5. Request emergency patch release from Security Guild
## 10. References
- [secret-leak-detection.md](./secret-leak-detection.md) - Main secret detection documentation
- [SPRINT_20260104_003_SCANNER_secret_rule_bundles.md](../../../implplan/SPRINT_20260104_003_SCANNER_secret_rule_bundles.md) - Implementation sprint
- [dsse-rekor-operator-guide.md](./dsse-rekor-operator-guide.md) - DSSE and Rekor verification