feat: Implement Runtime Facts ingestion service and NDJSON reader
Some checks failed
Docs CI / lint-and-preview (push) Has been cancelled
Some checks failed
Docs CI / lint-and-preview (push) Has been cancelled
- Added RuntimeFactsNdjsonReader for reading NDJSON formatted runtime facts. - Introduced IRuntimeFactsIngestionService interface and its implementation. - Enhanced Program.cs to register new services and endpoints for runtime facts. - Updated CallgraphIngestionService to include CAS URI in stored artifacts. - Created RuntimeFactsValidationException for validation errors during ingestion. - Added tests for RuntimeFactsIngestionService and RuntimeFactsNdjsonReader. - Implemented SignalsSealedModeMonitor for compliance checks in sealed mode. - Updated project dependencies for testing utilities.
This commit is contained in:
master
@@ -2,7 +2,10 @@
|
||||
|
||||
Scanner analyses container images layer-by-layer, producing deterministic SBOM fragments, diffs, and signed reports.
|
||||
|
||||
## Latest updates (2025-11-06)
|
||||
## Latest updates (2025-11-09)
|
||||
- Node analyzer now ingests npm/yarn/pnpm lockfiles, emitting `DeclaredOnly` components with lock provenance. The CLI companion command `stella node lock-validate` runs the collector offline, surfaces declared-only or missing-lock packages, and emits telemetry via `stellaops.cli.node.lock_validate.count`.
|
||||
- Python analyzer picks up `requirements*.txt`, `Pipfile.lock`, and `poetry.lock`, tagging installed distributions with lock provenance and generating declared-only components for policy. Use `stella python lock-validate` to run the same checks locally before images are built.
|
||||
- Java analyzer now parses `gradle.lockfile`, `gradle/dependency-locks/**/*.lockfile`, and `pom.xml` dependencies via the new `JavaLockFileCollector`, merging lock metadata onto jar evidence and emitting declared-only components when jars are absent. The new CLI verb `stella java lock-validate` reuses that collector offline (table/JSON output) and records `stellaops.cli.java.lock_validate.count{outcome}` for observability.
|
||||
- Worker/WebService now resolve cache roots and feature flags via `StellaOps.Scanner.Surface.Env`; misconfiguration warnings are documented in `docs/modules/scanner/design/surface-env.md` and surfaced through startup validation.
|
||||
- Platform events rollout (2025-10-19) continues to publish scanner.report.ready@1 and scanner.scan.completed@1 envelopes with embedded DSSE payloads (see docs/updates/2025-10-19-scanner-policy.md and docs/updates/2025-10-19-platform-events.md). Service and consumer tests should round-trip the canonical samples under docs/events/samples/.
|
||||
|
||||
@@ -33,6 +36,7 @@ Scanner analyses container images layer-by-layer, producing deterministic SBOM f
|
||||
- ./operations/rustfs-migration.md
|
||||
- ./operations/entrypoint.md
|
||||
- ./operations/secret-leak-detection.md
|
||||
- ./operations/dsse-rekor-operator-guide.md
|
||||
- ./design/macos-analyzer.md
|
||||
- ./design/windows-analyzer.md
|
||||
- ../benchmarks/scanner/deep-dives/macos.md
|
||||
|
||||
171
docs/modules/scanner/operations/dsse-rekor-operator-guide.md
Normal file
171
docs/modules/scanner/operations/dsse-rekor-operator-guide.md
Normal file
@@ -0,0 +1,171 @@
|
||||
# DSSE & Rekor Operator Enablement Guide
|
||||
|
||||
> **Audience.** Scanner / Export Center operators, platform SREs, and field engineers bringing DSSE attestations + Rekor proofs into production (online or air-gapped).
|
||||
>
|
||||
> **Sources.** Aligns with Sprint 138 (SCANNER-ENG-0015) gap analysis (§DSSE/Rekor operator enablement) and Scanner architecture specs.
|
||||
|
||||
---
|
||||
|
||||
## 1. Why this matters
|
||||
|
||||
- **Evidence on demand.** Every SBOM, diff, and report can be bound to a DSSE envelope issued by `StellaOps.Signer`, logged to Rekor via `StellaOps.Attestor`, and bundled for export/offline use.
|
||||
- **Policy leverage.** Policy Engine predicates gate releases until attestations exist *and* their Rekor proofs verify, reducing “unsigned” drift.
|
||||
- **Regulatory readiness.** Operators need a deterministic playbook to satisfy PCI, FedRAMP, EU CRA, and national sovereignty requirements without phoning home.
|
||||
|
||||
---
|
||||
|
||||
## 2. Components & responsibilities
|
||||
|
||||
| Component | Role | Key references |
|
||||
|-----------|------|----------------|
|
||||
| `StellaOps.Signer` | Issues DSSE envelopes using PoE-scoped keys (Fulcio or BYO KMS/HSM). | `ops/devops/signing/` |
|
||||
| `StellaOps.Attestor` | Submits DSSE payloads to Rekor v2, caches `{uuid,index,proof}` and mirrors proofs offline. | `docs/modules/attestor/architecture.md` |
|
||||
| Rekor v2 (managed or self-hosted) | Transparency log providing UUIDs + inclusion proofs. | `docs/ops/rekor/README.md` (if self-hosted) |
|
||||
| `StellaOps.Scanner` (WebService/Worker) | Requests attestations per scan, stores Rekor metadata next to SBOM artefacts. | `docs/modules/scanner/architecture.md` |
|
||||
| Export Center | Packages DSSE payloads + proofs into Offline Kit bundles and mirrors license notices. | `docs/modules/export-center/architecture.md` |
|
||||
| Policy Engine + CLI | Enforce “attested only” promotion, expose CLI verification verbs. | `docs/modules/policy/architecture.md`, `docs/09_API_CLI_REFERENCE.md` |
|
||||
|
||||
---
|
||||
|
||||
## 3. Prerequisites checklist
|
||||
|
||||
1. **Keys & trust roots**
|
||||
- Fulcio / KMS credentials available to `StellaOps.Signer`.
|
||||
- Rekor public key pinned (`rekor.pub`) for verification jobs and CLI tooling.
|
||||
2. **Service wiring**
|
||||
- `scanner.attestation.signerEndpoint` → internal Signer base URL.
|
||||
- `scanner.attestation.attestorEndpoint` → Attestor base URL.
|
||||
- `attestor.rekor.api` & `attestor.rekor.pubkey` set for the target log.
|
||||
3. **Storage**
|
||||
- Mongo collections `attestations` & `rekorProofs` sized for retention (7–30 days recommended).
|
||||
- Object store tier with at-rest encryption for DSSE payloads.
|
||||
4. **Observability**
|
||||
- Metrics: `attestor_rekor_success_total`, `attestor_rekor_retry_total`, `rekor_inclusion_latency`.
|
||||
- Logs shipped to your SIEM for compliance (Signer request/response IDs, Rekor UUIDs).
|
||||
5. **Offline readiness**
|
||||
- Export Center profile with `attestations.bundle=true`.
|
||||
- Rekor log snapshots mirrored (ORAS bundle or rsync of `/var/log/rekor`) for disconnected verification.
|
||||
|
||||
---
|
||||
|
||||
## 4. Enablement workflow
|
||||
|
||||
### 4.1 Configure Signer & Attestor
|
||||
|
||||
```yaml
|
||||
signer:
|
||||
schemaVersion: 2
|
||||
keyProvider: kms-fleet
|
||||
attestorEndpoint: https://attestor.internal
|
||||
defaultPredicate: https://stella-ops.org/attestations/sbom/1
|
||||
|
||||
attestor:
|
||||
schemaVersion: 1
|
||||
rekor:
|
||||
api: https://rekor.internal
|
||||
publicKeyPath: /etc/rekor/rekor.pub
|
||||
offlineMirrorPath: /var/lib/rekor/snapshots
|
||||
retry:
|
||||
maxAttempts: 5
|
||||
backoffSeconds: 15
|
||||
```
|
||||
|
||||
### 4.2 Turn on Scanner enforcement
|
||||
|
||||
```yaml
|
||||
scanner:
|
||||
schemaVersion: 2
|
||||
attestation:
|
||||
requireDsse: true # fail scans when Signer/Attestor errors occur
|
||||
signerEndpoint: https://signer.internal
|
||||
attestorEndpoint: https://attestor.internal
|
||||
uploadArtifacts: true # store DSSE + proof next to SBOM artefacts
|
||||
```
|
||||
|
||||
Set `requireDsse=false` during observation, then flip to `true` once Rekor health SLOs are green.
|
||||
|
||||
### 4.3 Policy templates
|
||||
|
||||
Add Policy Engine predicates (Rego snippet):
|
||||
|
||||
```rego
|
||||
package stella.policies.attestation
|
||||
|
||||
deny[msg] {
|
||||
not input.attestations.rekor_verified
|
||||
msg := sprintf("missing Rekor proof for %s", [input.scan_id])
|
||||
}
|
||||
|
||||
warn[msg] {
|
||||
input.attestations.rekor_age_hours > 24
|
||||
msg := sprintf("Rekor proof older than 24h for %s", [input.scan_id])
|
||||
}
|
||||
```
|
||||
|
||||
Tie Scheduler or CI promotion gates to the `deny` result.
|
||||
|
||||
### 4.4 CLI and verification
|
||||
|
||||
- `stellaops-cli runtime policy test --image <digest> --json` already surfaces `attestation.uuid` and `rekorVerified` fields.
|
||||
- To validate bundles offline: `stellaops-cli attest verify --bundle path/to/export.tar --rekor-key rekor.pub`.
|
||||
|
||||
Document these flows for AppSec teams so they can self-serve proofs during audits.
|
||||
|
||||
### 4.5 Export Center profile
|
||||
|
||||
```yaml
|
||||
exportProfiles:
|
||||
secure-default:
|
||||
includeSboms: true
|
||||
includeAttestations: true
|
||||
includeRekorProofs: true
|
||||
policy:
|
||||
requireAttestations: true
|
||||
allowUnsigned: false
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 5. Rollout levers & phases
|
||||
|
||||
| Phase | Toggle | Goal |
|
||||
|-------|--------|------|
|
||||
| **Observe** | `scanner.attestation.requireDsse=false`, policies in `warn` mode. | Validate plumbing without blocking builds; capture metrics. |
|
||||
| **Enforce** | Flip `requireDsse=true`, policy `deny` for missing proofs, Rekor SLO alerts live. | Block unsigned artefacts; auto-retry attestor failures. |
|
||||
| **Escalate** | Export Center profile `includeAttestations=true`, CLI docs distributed, Notify alerts wired. | Broad communication + audit evidence ready. |
|
||||
|
||||
Roll forward per environment; keep the previous phase’s toggles for hot rollback.
|
||||
|
||||
---
|
||||
|
||||
## 6. Offline / air-gap guidance
|
||||
|
||||
1. **Mirror Rekor**: take log snapshots daily (`rekor-cli log export`) and add to the Offline Kit.
|
||||
2. **Bundle proofs**: Export Center must include `*.rekor.json` and `rekor-chain.pem` alongside DSSE envelopes.
|
||||
3. **CLI verification offline**:
|
||||
```bash
|
||||
stellaops-cli attest verify --bundle offline-kit.tar \
|
||||
--rekor-root hashsum.txt --rekor-tree treehead.json --rekor-key rekor.pub
|
||||
```
|
||||
4. **Fallback**: When Rekor connectivity is unavailable, Attestor queues submissions locally and emits `attestationPending=true`; policy can allow waivers for a limited TTL via `policy.attestations.deferHours`.
|
||||
|
||||
---
|
||||
|
||||
## 7. Troubleshooting
|
||||
|
||||
| Symptom | Checks | Resolution |
|
||||
|---------|--------|------------|
|
||||
| `attestationPending` flag stays true | `attestor_rekor_retry_total`, Attestor logs, Rekor `/healthz`. | Verify Rekor endpoint & certs; rotate API tokens; replay queued DSSE payloads via `attestor replay`. |
|
||||
| Policy denies despite DSSE | Confirm Rekor proof bundle stored under `/artifacts/<scanId>/rekor/`. | Re-run `stellaops-cli attest verify`, ensure Policy Engine has the new schema (`attestations.rekor_verified`). |
|
||||
| CLI verification fails offline | Ensure Rekor snapshot + `rekor.pub` shipped together; check timestamp gap. | Regenerate snapshot, or import Rekor entries into the isolated log before verifying. |
|
||||
|
||||
---
|
||||
|
||||
## References
|
||||
|
||||
- Gap analysis: `docs/benchmarks/scanner/scanning-gaps-stella-misses-from-competitors.md#dsse-rekor-operator-enablement-trivy-grype-snyk`
|
||||
- Scanner architecture (§Signer → Attestor → Rekor): `docs/modules/scanner/architecture.md`
|
||||
- Export Center profiles: `docs/modules/export-center/architecture.md`
|
||||
- Policy Engine predicates: `docs/modules/policy/architecture.md`
|
||||
- CLI reference: `docs/09_API_CLI_REFERENCE.md`
|
||||
|
||||
Reference in New Issue
Block a user