Update AGENTS.md files across multiple modules to standardize task status update instructions and introduce a new document for Secret Leak Detection operations.
- Modified task status update instructions in AGENTS.md files to refer to corresponding sprint files as `/docs/implplan/SPRINT_*.md` instead of `docs/implplan/SPRINTS.md`. - Added a comprehensive document for Secret Leak Detection operations detailing scope, prerequisites, rule bundle lifecycle, enabling the analyzer, policy patterns, observability, troubleshooting, and references.
This commit is contained in:
master
@@ -16,7 +16,7 @@ Scanner analyses container images layer-by-layer, producing deterministic SBOM f
|
||||
- [Benchmarks overview](../../benchmarks/scanner/README.md)
|
||||
|
||||
## How to get started
|
||||
1. Open ../../implplan/SPRINTS.md and locate the stories referencing this module.
|
||||
1. Open sprint file `/docs/implplan/SPRINT_*.md` and locate the stories referencing this module.
|
||||
2. Review ./TASKS.md for local follow-ups and confirm status transitions (TODO → DOING → DONE/BLOCKED).
|
||||
3. Read the architecture and README for domain context before editing code or docs.
|
||||
4. Coordinate cross-module changes in the main /AGENTS.md description and through the sprint plan.
|
||||
@@ -33,7 +33,7 @@ Scanner analyses container images layer-by-layer, producing deterministic SBOM f
|
||||
- `docs/modules/platform/architecture-overview.md`
|
||||
|
||||
## Working Agreement
|
||||
- 1. Update task status to `DOING`/`DONE` in both `docs/implplan/SPRINTS.md` and the local `TASKS.md` when you start or finish work.
|
||||
- 1. Update task status to `DOING`/`DONE` in both correspoding sprint file `/docs/implplan/SPRINT_*.md` and the local `TASKS.md` when you start or finish work.
|
||||
- 2. Review this charter and the Required Reading documents before coding; confirm prerequisites are met.
|
||||
- 3. Keep changes deterministic (stable ordering, timestamps, hashes) and align with offline/air-gap expectations.
|
||||
- 4. Coordinate doc updates, tests, and cross-guild communication whenever contracts or workflows change.
|
||||
|
||||
@@ -28,6 +28,7 @@ Scanner analyses container images layer-by-layer, producing deterministic SBOM f
|
||||
- ./operations/analyzers-grafana-dashboard.json
|
||||
- ./operations/rustfs-migration.md
|
||||
- ./operations/entrypoint.md
|
||||
- ./operations/secret-leak-detection.md
|
||||
- ./design/macos-analyzer.md
|
||||
- ./design/windows-analyzer.md
|
||||
- ../benchmarks/scanner/deep-dives/macos.md
|
||||
|
||||
@@ -37,4 +37,4 @@
|
||||
| SCANNER-ENG-0026 | TODO | Scanner Guild (Windows Packages Squad) | Implement Chocolatey & registry collectors per `design/windows-analyzer.md` §3.3–3.4. | Harvest nuspec metadata and registry uninstall/service evidence; merge with filesystem artefacts; align with feed decisions from POLICY-READINESS-0002. |
|
||||
| SCANNER-ENG-0027 | TODO | Scanner Guild, Policy Guild, Offline Kit Guild | Deliver Windows policy/offline integration per `design/windows-analyzer.md` §5–6. | Define predicates, CLI/Offline docs, and packaging for feeds/certs; start after POLICY-READINESS-0002 sign-off. |
|
||||
| SCANNER-OPS-0001 | TODO | Ops Guild | Review runbooks/observability assets after next sprint demo. | Sync outcomes back to ../../TASKS.md |
|
||||
| SCANNER-ENG-0001 | TODO | Module Team | Cross-check implementation plan milestones against ../../implplan/SPRINTS.md. | Update status via ./AGENTS.md workflow |
|
||||
| SCANNER-ENG-0001 | TODO | Module Team | Cross-check implementation plan milestones against `/docs/implplan/SPRINT_*.md`. | Update status via ./AGENTS.md workflow |
|
||||
|
||||
170
docs/modules/scanner/operations/secret-leak-detection.md
Normal file
170
docs/modules/scanner/operations/secret-leak-detection.md
Normal file
@@ -0,0 +1,170 @@
|
||||
# Secret Leak Detection (Scanner Operations)
|
||||
|
||||
> **Status:** Preview (Sprint 132). Requires `SCANNER-ENG-0007`/`POLICY-READINESS-0001` release bundle and the experimental flag `secret-leak-detection`.
|
||||
>
|
||||
> **Audience:** Scanner operators, Security Guild, Docs Guild, Offline Kit maintainers.
|
||||
|
||||
## 1. Scope & goals
|
||||
|
||||
- Introduce the **`StellaOps.Scanner.Analyzers.Secrets`** plug-in, which executes deterministic rule bundles against layer content during scans.
|
||||
- Ensure every finding is reproducible: rule bundles are DSSE-signed, versioned, and shipped with the Offline Kit.
|
||||
- Surface policy-ready evidence (`secret.leak`) so tenants can enforce block/warn flows using `stella-dsl@1` predicates.
|
||||
- Preserve sovereignty: rule bundles install locally, no outbound telemetry, masking is enforced before data leaves the worker.
|
||||
|
||||
## 2. Prerequisites
|
||||
|
||||
| Requirement | Notes |
|
||||
| --- | --- |
|
||||
| Analyzer binaries | Deploy `StellaOps.Scanner.Analyzers.Secrets` alongside Scanner Worker (packaged with the standard container images). |
|
||||
| Surface libraries | `Surface.Secrets`, `Surface.Validation`, and `Surface.Env` must already be configured (see `surface-secrets.md`). |
|
||||
| Experimental flag | Enable `scanner.features.experimental["secret-leak-detection"] = true` on both WebService and Worker. |
|
||||
| Policy readiness | Import predicates from `docs/modules/policy/secret-leak-detection-readiness.md` into tenant policy packs. |
|
||||
| Offline Kit | Update to an Offline Kit that includes `offline/rules/secrets/<release>/` before enabling production scans. |
|
||||
|
||||
## 3. Rule bundle lifecycle
|
||||
|
||||
Rule bundles ship in the Export Center / Offline Kit under `offline/rules/secrets/<release>/`.
|
||||
|
||||
| File | Purpose | Notes |
|
||||
| --- | --- | --- |
|
||||
| `secrets.ruleset.manifest.json` | Lists rule IDs, versions, severity defaults, and hash digests. | Consume during policy drift audits. |
|
||||
| `secrets.ruleset.rules.jsonl` | Newline-delimited definitions (regex/entropy metadata, masking hints). | Loaded by the analyzer at startup. |
|
||||
| `secrets.ruleset.dsse.json` | DSSE envelope (Signer certificate chain + Attestor proof). | Verify before distributing bundles. |
|
||||
|
||||
Verification checklist (`stella excititor verify` talks to the configured Attestor service):
|
||||
|
||||
```
|
||||
stella excititor verify \
|
||||
--attestation offline/rules/secrets/2025.11/secrets.ruleset.dsse.json \
|
||||
--digest $(sha256sum offline/rules/secrets/2025.11/secrets.ruleset.rules.jsonl | cut -d' ' -f1)
|
||||
```
|
||||
|
||||
For air-gapped environments point the CLI at the Offline Kit Attestor mirror (for example `STELLA_ATTESTOR_URL=http://attestor.offline.local`) before running the command. The Attestor instance validates the DSSE envelope against the mirrored Rekor log and embedded certificate chain; no public network access is required.
|
||||
|
||||
Once verified, copy the manifest + rules to the worker:
|
||||
|
||||
```
|
||||
/opt/stellaops/plugins/scanner/analyzers/secrets/
|
||||
├── secrets.ruleset.manifest.json
|
||||
├── secrets.ruleset.rules.jsonl
|
||||
└── secrets.ruleset.dsse.json
|
||||
```
|
||||
|
||||
Restart the worker so the analyzer reloads the updated bundle. Bundles are immutable; upgrading requires replacing all three files and restarting.
|
||||
|
||||
## 4. Enabling the analyzer
|
||||
|
||||
1. **Toggle the feature flag** (WebService + Worker):
|
||||
|
||||
```yaml
|
||||
scanner:
|
||||
features:
|
||||
experimental:
|
||||
secret-leak-detection: true
|
||||
```
|
||||
|
||||
(Environment alternative: `SCANNER__FEATURES__EXPERIMENTAL__secret-leak-detection=true`.)
|
||||
|
||||
2. **Configure retention** (WebService):
|
||||
|
||||
```yaml
|
||||
scanner:
|
||||
storage:
|
||||
migrations:
|
||||
- Scanner.Analysis.SecretFindingsTtl
|
||||
```
|
||||
|
||||
The migration adds `secretFindings` documents to `ScanAnalysisStore` with the standard TTL (default 90 days). Adjust Mongo TTL via the deployment overlay if longer retention is required.
|
||||
|
||||
3. **Activate policy ingestion** (WebService):
|
||||
|
||||
```yaml
|
||||
scanner:
|
||||
runtime:
|
||||
enableSecretFindings: true
|
||||
```
|
||||
|
||||
(Experimental builds gate secret evidence behind this toggle to avoid surprising downstream consumers.)
|
||||
|
||||
4. **Roll scanner hosts**. Apply the configuration, roll WebService first, then Workers. Verify the startup logs contain `SecretsAnalyzerHost` and `SecretLeakDetection: Enabled`.
|
||||
|
||||
## 5. Policy patterns
|
||||
|
||||
The analyzer emits `secret.leak` evidence with the shape:
|
||||
|
||||
```json
|
||||
{
|
||||
"ruleId": "stellaops.secrets.aws-access-key",
|
||||
"ruleVersion": "2025.11.0",
|
||||
"severity": "high",
|
||||
"confidence": "high",
|
||||
"file": "/app/config.yml",
|
||||
"line": 42,
|
||||
"mask": "AKIA********B7",
|
||||
"bundleId": "secrets.ruleset",
|
||||
"bundleVersion": "2025.11"
|
||||
}
|
||||
```
|
||||
|
||||
Policy DSL helpers introduced with this release:
|
||||
|
||||
| Helper | Description |
|
||||
| --- | --- |
|
||||
| `secret.hasFinding(ruleId?, severity?, confidence?)` | Returns true if any finding matches the filter. |
|
||||
| `secret.bundle.version(requiredVersion)` | Ensures the active bundle meets or exceeds a version. |
|
||||
| `secret.match.count(ruleId?)` | Returns the number of findings (useful for thresholds). |
|
||||
|
||||
Sample policy (`policies/secret-blocker.stella`):
|
||||
|
||||
```dsl
|
||||
policy "Secret Leak Guard" syntax "stella-dsl@1" {
|
||||
metadata {
|
||||
description = "Block high-confidence secret leaks"
|
||||
tags = ["secrets","compliance"]
|
||||
}
|
||||
|
||||
rule block_high_confidence priority 10 {
|
||||
when secret.hasFinding(severity: "high", confidence: "high")
|
||||
then escalate to "block";
|
||||
because "High severity secret leak detected";
|
||||
}
|
||||
|
||||
rule require_current_bundle priority 5 {
|
||||
when not secret.bundle.version("2025.11")
|
||||
then warn message "Secret leak bundle out of date";
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Tenants that prefer staged rollout can downgrade low-confidence findings:
|
||||
|
||||
```dsl
|
||||
rule low_confidence_warn priority 20 {
|
||||
when secret.hasFinding(confidence: "low")
|
||||
then annotate decision.notes := "Investigate masked payload";
|
||||
else ignore;
|
||||
}
|
||||
```
|
||||
|
||||
## 6. Observability & reporting
|
||||
|
||||
- **Metrics:** `scanner.secret.finding_total{tenant,ruleId,severity,confidence}` increments per finding. Add Prometheus alerts for spikes.
|
||||
- **Logs:** `SecretsAnalyzerHost` logs bundle version on load and emits warnings when masking fails (payload never leaves memory).
|
||||
- **Traces:** Each analyzer run adds a `scanner.secrets.scan` span with rule counts and wall-clock timing.
|
||||
- **Reports / CLI:** Scan reports include a `secretFindings` array; CLI diff/export surfaces render masked snippets plus remediation guidance.
|
||||
|
||||
## 7. Troubleshooting
|
||||
|
||||
| Symptom | Resolution |
|
||||
| --- | --- |
|
||||
| Analyzer disabled at startup | Confirm feature flag and bundle files exist; check `plugins/scanner/analyzers/secrets` permissions (`640`). |
|
||||
| No findings despite seeded secrets | Ensure bundle hash matches manifest. Run worker with `--secrets-trace` (debug build) to log matched rules locally. |
|
||||
| Policy marks findings as unknown | Upgrade tenant policies to include `secret.*` helpers; older policies silently drop the namespace. |
|
||||
| Air-gapped verification fails | Ensure `STELLA_ATTESTOR_URL` points to the Offline Kit Attestor mirror and rerun `stella excititor verify --attestation <file> --digest <sha256>`. |
|
||||
|
||||
## 8. References
|
||||
|
||||
- `docs/modules/policy/secret-leak-detection-readiness.md`
|
||||
- `docs/benchmarks/scanner/deep-dives/secrets.md`
|
||||
- `docs/modules/scanner/design/surface-secrets.md`
|
||||
- `docs/07_HIGH_LEVEL_ARCHITECTURE.md` §1.1 Runtime inventory (Scanner)
|
||||
Reference in New Issue
Block a user