Update AGENTS.md files across multiple modules to standardize task status update instructions and introduce a new document for Secret Leak Detection operations.
- Modified task status update instructions in AGENTS.md files to refer to corresponding sprint files as `/docs/implplan/SPRINT_*.md` instead of `docs/implplan/SPRINTS.md`. - Added a comprehensive document for Secret Leak Detection operations detailing scope, prerequisites, rule bundle lifecycle, enabling the analyzer, policy patterns, observability, troubleshooting, and references.
This commit is contained in:
master
170
docs/modules/scanner/operations/secret-leak-detection.md
Normal file
170
docs/modules/scanner/operations/secret-leak-detection.md
Normal file
@@ -0,0 +1,170 @@
|
||||
# Secret Leak Detection (Scanner Operations)
|
||||
|
||||
> **Status:** Preview (Sprint 132). Requires `SCANNER-ENG-0007`/`POLICY-READINESS-0001` release bundle and the experimental flag `secret-leak-detection`.
|
||||
>
|
||||
> **Audience:** Scanner operators, Security Guild, Docs Guild, Offline Kit maintainers.
|
||||
|
||||
## 1. Scope & goals
|
||||
|
||||
- Introduce the **`StellaOps.Scanner.Analyzers.Secrets`** plug-in, which executes deterministic rule bundles against layer content during scans.
|
||||
- Ensure every finding is reproducible: rule bundles are DSSE-signed, versioned, and shipped with the Offline Kit.
|
||||
- Surface policy-ready evidence (`secret.leak`) so tenants can enforce block/warn flows using `stella-dsl@1` predicates.
|
||||
- Preserve sovereignty: rule bundles install locally, no outbound telemetry, masking is enforced before data leaves the worker.
|
||||
|
||||
## 2. Prerequisites
|
||||
|
||||
| Requirement | Notes |
|
||||
| --- | --- |
|
||||
| Analyzer binaries | Deploy `StellaOps.Scanner.Analyzers.Secrets` alongside Scanner Worker (packaged with the standard container images). |
|
||||
| Surface libraries | `Surface.Secrets`, `Surface.Validation`, and `Surface.Env` must already be configured (see `surface-secrets.md`). |
|
||||
| Experimental flag | Enable `scanner.features.experimental["secret-leak-detection"] = true` on both WebService and Worker. |
|
||||
| Policy readiness | Import predicates from `docs/modules/policy/secret-leak-detection-readiness.md` into tenant policy packs. |
|
||||
| Offline Kit | Update to an Offline Kit that includes `offline/rules/secrets/<release>/` before enabling production scans. |
|
||||
|
||||
## 3. Rule bundle lifecycle
|
||||
|
||||
Rule bundles ship in the Export Center / Offline Kit under `offline/rules/secrets/<release>/`.
|
||||
|
||||
| File | Purpose | Notes |
|
||||
| --- | --- | --- |
|
||||
| `secrets.ruleset.manifest.json` | Lists rule IDs, versions, severity defaults, and hash digests. | Consume during policy drift audits. |
|
||||
| `secrets.ruleset.rules.jsonl` | Newline-delimited definitions (regex/entropy metadata, masking hints). | Loaded by the analyzer at startup. |
|
||||
| `secrets.ruleset.dsse.json` | DSSE envelope (Signer certificate chain + Attestor proof). | Verify before distributing bundles. |
|
||||
|
||||
Verification checklist (`stella excititor verify` talks to the configured Attestor service):
|
||||
|
||||
```
|
||||
stella excititor verify \
|
||||
--attestation offline/rules/secrets/2025.11/secrets.ruleset.dsse.json \
|
||||
--digest $(sha256sum offline/rules/secrets/2025.11/secrets.ruleset.rules.jsonl | cut -d' ' -f1)
|
||||
```
|
||||
|
||||
For air-gapped environments point the CLI at the Offline Kit Attestor mirror (for example `STELLA_ATTESTOR_URL=http://attestor.offline.local`) before running the command. The Attestor instance validates the DSSE envelope against the mirrored Rekor log and embedded certificate chain; no public network access is required.
|
||||
|
||||
Once verified, copy the manifest + rules to the worker:
|
||||
|
||||
```
|
||||
/opt/stellaops/plugins/scanner/analyzers/secrets/
|
||||
├── secrets.ruleset.manifest.json
|
||||
├── secrets.ruleset.rules.jsonl
|
||||
└── secrets.ruleset.dsse.json
|
||||
```
|
||||
|
||||
Restart the worker so the analyzer reloads the updated bundle. Bundles are immutable; upgrading requires replacing all three files and restarting.
|
||||
|
||||
## 4. Enabling the analyzer
|
||||
|
||||
1. **Toggle the feature flag** (WebService + Worker):
|
||||
|
||||
```yaml
|
||||
scanner:
|
||||
features:
|
||||
experimental:
|
||||
secret-leak-detection: true
|
||||
```
|
||||
|
||||
(Environment alternative: `SCANNER__FEATURES__EXPERIMENTAL__secret-leak-detection=true`.)
|
||||
|
||||
2. **Configure retention** (WebService):
|
||||
|
||||
```yaml
|
||||
scanner:
|
||||
storage:
|
||||
migrations:
|
||||
- Scanner.Analysis.SecretFindingsTtl
|
||||
```
|
||||
|
||||
The migration adds `secretFindings` documents to `ScanAnalysisStore` with the standard TTL (default 90 days). Adjust Mongo TTL via the deployment overlay if longer retention is required.
|
||||
|
||||
3. **Activate policy ingestion** (WebService):
|
||||
|
||||
```yaml
|
||||
scanner:
|
||||
runtime:
|
||||
enableSecretFindings: true
|
||||
```
|
||||
|
||||
(Experimental builds gate secret evidence behind this toggle to avoid surprising downstream consumers.)
|
||||
|
||||
4. **Roll scanner hosts**. Apply the configuration, roll WebService first, then Workers. Verify the startup logs contain `SecretsAnalyzerHost` and `SecretLeakDetection: Enabled`.
|
||||
|
||||
## 5. Policy patterns
|
||||
|
||||
The analyzer emits `secret.leak` evidence with the shape:
|
||||
|
||||
```json
|
||||
{
|
||||
"ruleId": "stellaops.secrets.aws-access-key",
|
||||
"ruleVersion": "2025.11.0",
|
||||
"severity": "high",
|
||||
"confidence": "high",
|
||||
"file": "/app/config.yml",
|
||||
"line": 42,
|
||||
"mask": "AKIA********B7",
|
||||
"bundleId": "secrets.ruleset",
|
||||
"bundleVersion": "2025.11"
|
||||
}
|
||||
```
|
||||
|
||||
Policy DSL helpers introduced with this release:
|
||||
|
||||
| Helper | Description |
|
||||
| --- | --- |
|
||||
| `secret.hasFinding(ruleId?, severity?, confidence?)` | Returns true if any finding matches the filter. |
|
||||
| `secret.bundle.version(requiredVersion)` | Ensures the active bundle meets or exceeds a version. |
|
||||
| `secret.match.count(ruleId?)` | Returns the number of findings (useful for thresholds). |
|
||||
|
||||
Sample policy (`policies/secret-blocker.stella`):
|
||||
|
||||
```dsl
|
||||
policy "Secret Leak Guard" syntax "stella-dsl@1" {
|
||||
metadata {
|
||||
description = "Block high-confidence secret leaks"
|
||||
tags = ["secrets","compliance"]
|
||||
}
|
||||
|
||||
rule block_high_confidence priority 10 {
|
||||
when secret.hasFinding(severity: "high", confidence: "high")
|
||||
then escalate to "block";
|
||||
because "High severity secret leak detected";
|
||||
}
|
||||
|
||||
rule require_current_bundle priority 5 {
|
||||
when not secret.bundle.version("2025.11")
|
||||
then warn message "Secret leak bundle out of date";
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Tenants that prefer staged rollout can downgrade low-confidence findings:
|
||||
|
||||
```dsl
|
||||
rule low_confidence_warn priority 20 {
|
||||
when secret.hasFinding(confidence: "low")
|
||||
then annotate decision.notes := "Investigate masked payload";
|
||||
else ignore;
|
||||
}
|
||||
```
|
||||
|
||||
## 6. Observability & reporting
|
||||
|
||||
- **Metrics:** `scanner.secret.finding_total{tenant,ruleId,severity,confidence}` increments per finding. Add Prometheus alerts for spikes.
|
||||
- **Logs:** `SecretsAnalyzerHost` logs bundle version on load and emits warnings when masking fails (payload never leaves memory).
|
||||
- **Traces:** Each analyzer run adds a `scanner.secrets.scan` span with rule counts and wall-clock timing.
|
||||
- **Reports / CLI:** Scan reports include a `secretFindings` array; CLI diff/export surfaces render masked snippets plus remediation guidance.
|
||||
|
||||
## 7. Troubleshooting
|
||||
|
||||
| Symptom | Resolution |
|
||||
| --- | --- |
|
||||
| Analyzer disabled at startup | Confirm feature flag and bundle files exist; check `plugins/scanner/analyzers/secrets` permissions (`640`). |
|
||||
| No findings despite seeded secrets | Ensure bundle hash matches manifest. Run worker with `--secrets-trace` (debug build) to log matched rules locally. |
|
||||
| Policy marks findings as unknown | Upgrade tenant policies to include `secret.*` helpers; older policies silently drop the namespace. |
|
||||
| Air-gapped verification fails | Ensure `STELLA_ATTESTOR_URL` points to the Offline Kit Attestor mirror and rerun `stella excititor verify --attestation <file> --digest <sha256>`. |
|
||||
|
||||
## 8. References
|
||||
|
||||
- `docs/modules/policy/secret-leak-detection-readiness.md`
|
||||
- `docs/benchmarks/scanner/deep-dives/secrets.md`
|
||||
- `docs/modules/scanner/design/surface-secrets.md`
|
||||
- `docs/07_HIGH_LEVEL_ARCHITECTURE.md` §1.1 Runtime inventory (Scanner)
|
||||
Reference in New Issue
Block a user