feat: Implement Runtime Facts ingestion service and NDJSON reader
Some checks failed
Docs CI / lint-and-preview (push) Has been cancelled
Some checks failed
Docs CI / lint-and-preview (push) Has been cancelled
- Added RuntimeFactsNdjsonReader for reading NDJSON formatted runtime facts. - Introduced IRuntimeFactsIngestionService interface and its implementation. - Enhanced Program.cs to register new services and endpoints for runtime facts. - Updated CallgraphIngestionService to include CAS URI in stored artifacts. - Created RuntimeFactsValidationException for validation errors during ingestion. - Added tests for RuntimeFactsIngestionService and RuntimeFactsNdjsonReader. - Implemented SignalsSealedModeMonitor for compliance checks in sealed mode. - Updated project dependencies for testing utilities.
This commit is contained in:
master
@@ -189,6 +189,137 @@ Replays the AOC guard against stored raw documents. By default it checks all adv
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 4 · `stella node lock-validate`
|
||||
|
||||
### 4.1 Synopsis
|
||||
|
||||
```bash
|
||||
stella node lock-validate \
|
||||
[--path <directory>] \
|
||||
[--format table|json] \
|
||||
[--verbose]
|
||||
```
|
||||
|
||||
### 4.2 Description
|
||||
|
||||
Runs the Node analyzer locally against a working directory to compare lockfiles (`package-lock.json`, `pnpm-lock.yaml`, `yarn.lock`) with what is actually present in `node_modules`. The command is read-only and never schedules a scan; it reuses the same deterministic collector that powers Scanner so results match backend evidence. Output highlights two conditions that policy cares about:
|
||||
|
||||
- **Declared Only** – packages present in lockfiles but missing from the filesystem or final image.
|
||||
- **Missing Lock** – packages discovered at runtime without corresponding lock metadata (no registry provenance, integrity hash, or repository information).
|
||||
|
||||
This helps catch drift before images are built, keeps lockfiles trustworthy, and feeds policy predicates such as `node.lock.declaredMissing`.
|
||||
|
||||
### 4.3 Options
|
||||
|
||||
| Option | Description |
|
||||
|--------|-------------|
|
||||
| `--path`, `-p` | Directory containing `package.json` and lockfiles. Defaults to the current working directory. |
|
||||
| `--format table|json` | `table` (default) renders a Spectre table with status badges; `json` prints the underlying report for CI automation. |
|
||||
| `--verbose` | Enables detailed logging (shared root option). |
|
||||
|
||||
### 4.4 Output & exit codes
|
||||
|
||||
- `table` mode prints a summary row and two sections: `Declared Only` (red) and `Missing Lock` (yellow). Columns show package, version, lock source/locator, and filesystem path so engineers can reconcile quickly.
|
||||
- `json` mode emits `{ declaredOnly: [], missingLockMetadata: [], totalDeclared, totalInstalled }`, mirroring the analyzer telemetry.
|
||||
|
||||
Exit codes:
|
||||
|
||||
| Code | Meaning |
|
||||
|------|---------|
|
||||
| `0` | No inconsistencies detected. |
|
||||
| `1` | Declared-only or missing-lock packages were found. |
|
||||
| `71` | The requested directory could not be read (missing path, permissions, etc.). |
|
||||
|
||||
The CLI also records `stellaops.cli.node.lock_validate.count{outcome}` so operators can monitor adoption in telemetry.
|
||||
|
||||
### 4.5 Offline notes
|
||||
|
||||
- Works entirely offline; point `--path` at a workspace checked out from an Offline Kit or build cache.
|
||||
- Honors the same `Surface.Validation` limits configured for Scanner once those knobs (`scanner.lockfiles.node.*`) are deployed cluster-wide.
|
||||
- Combine with `stella scan` by running lock validation in CI before images are built to fail fast on inconsistent manifests.
|
||||
|
||||
---
|
||||
|
||||
## 5 · `stella python lock-validate`
|
||||
|
||||
### 5.1 Synopsis
|
||||
|
||||
```bash
|
||||
stella python lock-validate \
|
||||
[--path <directory>] \
|
||||
[--format table|json] \
|
||||
[--verbose]
|
||||
```
|
||||
|
||||
### 5.2 Description
|
||||
|
||||
Validates Python lockfiles (currently `requirements*.txt`, `Pipfile.lock`, and `poetry.lock`) against what exists in `site-packages`. It uses the same analyzer Scanner runs so declared-only packages, missing locks, and editable installs are detected deterministically and without internet access. This catches drift between lock manifests and baked images before scanners or policy gates fail later.
|
||||
|
||||
### 5.3 Options
|
||||
|
||||
| Option | Description |
|
||||
|--------|-------------|
|
||||
| `--path`, `-p` | Directory containing `lib/python*/site-packages` and lockfiles. Defaults to `$PWD`. |
|
||||
| `--format table|json` | `table` (default) prints a human summary; `json` emits the raw report for CI. |
|
||||
| `--verbose` | Enables detailed logging. |
|
||||
|
||||
### 5.4 Output & exit codes
|
||||
|
||||
Output shape mirrors the Node command: declared-only packages are shown with lock provenance, and runtime packages missing lock metadata are highlighted separately. JSON mode returns the same object schema `{ declaredOnly, missingLockMetadata, totalDeclared, totalInstalled }`.
|
||||
|
||||
Exit codes follow the same contract (`0` success, `1` violations, `71` for unreadable path). Telemetry is published via `stellaops.cli.python.lock_validate.count{outcome}`.
|
||||
|
||||
### 5.5 Offline notes
|
||||
|
||||
- Works entirely offline—lockfiles and `site-packages` must already be present (from a venv snapshot, container rootfs, or Offline Kit).
|
||||
- Honors upcoming `scanner.lockfiles.python.*` guardrails once Surface.Validation is wired in so CLI + Scanner enforce the same registry/size limits.
|
||||
- Recommended CI flow: run `stella python lock-validate` before building containers and fail fast when declared-only packages remain.
|
||||
|
||||
## 6 · `stella java lock-validate`
|
||||
|
||||
### 6.1 Synopsis
|
||||
|
||||
```bash
|
||||
stella java lock-validate \\
|
||||
[--path <directory>] \\
|
||||
[--format table|json] \\
|
||||
[--verbose]
|
||||
```
|
||||
|
||||
### 6.2 Description
|
||||
|
||||
Executes the Java language analyzer locally so Gradle `gradle.lockfile`, `gradle/dependency-locks/**/*.lockfile`, and `pom.xml` declarations can be compared with the jars that actually ship in a workspace. The command reuses the new `JavaLockFileCollector` plus the `JavaLanguageAnalyzer` merge logic, so it emits the same `DeclaredOnly` and `Missing Lock` evidence that Scanner and Policy consume. Engineers can see which coordinates exist only in lockfiles (no jar on disk) and which installed jars lack lock metadata (no repository/provenance) before a scan ever runs.
|
||||
|
||||
### 6.3 Options
|
||||
|
||||
| Option | Description |
|
||||
|--------|-------------|
|
||||
| `--path`, `-p` | Directory containing jars (e.g., `build/libs`) and lockfiles. Defaults to the current working directory. |
|
||||
| `--format table|json` | `table` (default) renders the Spectre table; `json` outputs the raw `LockValidationReport`. |
|
||||
| `--verbose` | Enables detailed logging and surfaces the analyzer paths being inspected. |
|
||||
|
||||
### 6.4 Output & exit codes
|
||||
|
||||
Output mirrors the Node/Python verbs: `Declared Only` rows include the lock source/locator (e.g., `gradle.lockfile`, `gradle/dependency-locks/app.lockfile`) plus configuration/repository hints, while `Missing Lock` rows highlight jars that Scanner would tag with `lockMissing=true`. JSON responses return `{ declaredOnly, missingLockMetadata, totalDeclared, totalInstalled }`.
|
||||
|
||||
Exit codes align with the other lock validators:
|
||||
|
||||
| Code | Meaning |
|
||||
|------|---------|
|
||||
| `0` | No inconsistencies detected. |
|
||||
| `1` | Declared-only or missing-lock jars detected. |
|
||||
| `71` | Directory could not be read. |
|
||||
|
||||
Telemetry is recorded via `stellaops.cli.java.lock_validate.count{outcome}` so adoption can be monitored alongside the Node/Python verbs.
|
||||
|
||||
### 6.5 Offline notes
|
||||
|
||||
- Works with any workspace (Gradle, Maven, or extracted container layers) – no network access or build tool metadata is required at runtime.
|
||||
- Honors forthcoming `scanner.lockfiles.java.*` Surface.Validation limits once they are deployed so CLI + Scanner stay in lockstep.
|
||||
- Recommended CI flow: run `stella java lock-validate` before packaging containers to surface missing locks/declared-only coordinates early.
|
||||
|
||||
### 3.5 Exit codes
|
||||
|
||||
| Exit code | Meaning |
|
||||
|
||||
@@ -2,7 +2,10 @@
|
||||
|
||||
Scanner analyses container images layer-by-layer, producing deterministic SBOM fragments, diffs, and signed reports.
|
||||
|
||||
## Latest updates (2025-11-06)
|
||||
## Latest updates (2025-11-09)
|
||||
- Node analyzer now ingests npm/yarn/pnpm lockfiles, emitting `DeclaredOnly` components with lock provenance. The CLI companion command `stella node lock-validate` runs the collector offline, surfaces declared-only or missing-lock packages, and emits telemetry via `stellaops.cli.node.lock_validate.count`.
|
||||
- Python analyzer picks up `requirements*.txt`, `Pipfile.lock`, and `poetry.lock`, tagging installed distributions with lock provenance and generating declared-only components for policy. Use `stella python lock-validate` to run the same checks locally before images are built.
|
||||
- Java analyzer now parses `gradle.lockfile`, `gradle/dependency-locks/**/*.lockfile`, and `pom.xml` dependencies via the new `JavaLockFileCollector`, merging lock metadata onto jar evidence and emitting declared-only components when jars are absent. The new CLI verb `stella java lock-validate` reuses that collector offline (table/JSON output) and records `stellaops.cli.java.lock_validate.count{outcome}` for observability.
|
||||
- Worker/WebService now resolve cache roots and feature flags via `StellaOps.Scanner.Surface.Env`; misconfiguration warnings are documented in `docs/modules/scanner/design/surface-env.md` and surfaced through startup validation.
|
||||
- Platform events rollout (2025-10-19) continues to publish scanner.report.ready@1 and scanner.scan.completed@1 envelopes with embedded DSSE payloads (see docs/updates/2025-10-19-scanner-policy.md and docs/updates/2025-10-19-platform-events.md). Service and consumer tests should round-trip the canonical samples under docs/events/samples/.
|
||||
|
||||
@@ -33,6 +36,7 @@ Scanner analyses container images layer-by-layer, producing deterministic SBOM f
|
||||
- ./operations/rustfs-migration.md
|
||||
- ./operations/entrypoint.md
|
||||
- ./operations/secret-leak-detection.md
|
||||
- ./operations/dsse-rekor-operator-guide.md
|
||||
- ./design/macos-analyzer.md
|
||||
- ./design/windows-analyzer.md
|
||||
- ../benchmarks/scanner/deep-dives/macos.md
|
||||
|
||||
171
docs/modules/scanner/operations/dsse-rekor-operator-guide.md
Normal file
171
docs/modules/scanner/operations/dsse-rekor-operator-guide.md
Normal file
@@ -0,0 +1,171 @@
|
||||
# DSSE & Rekor Operator Enablement Guide
|
||||
|
||||
> **Audience.** Scanner / Export Center operators, platform SREs, and field engineers bringing DSSE attestations + Rekor proofs into production (online or air-gapped).
|
||||
>
|
||||
> **Sources.** Aligns with Sprint 138 (SCANNER-ENG-0015) gap analysis (§DSSE/Rekor operator enablement) and Scanner architecture specs.
|
||||
|
||||
---
|
||||
|
||||
## 1. Why this matters
|
||||
|
||||
- **Evidence on demand.** Every SBOM, diff, and report can be bound to a DSSE envelope issued by `StellaOps.Signer`, logged to Rekor via `StellaOps.Attestor`, and bundled for export/offline use.
|
||||
- **Policy leverage.** Policy Engine predicates gate releases until attestations exist *and* their Rekor proofs verify, reducing “unsigned” drift.
|
||||
- **Regulatory readiness.** Operators need a deterministic playbook to satisfy PCI, FedRAMP, EU CRA, and national sovereignty requirements without phoning home.
|
||||
|
||||
---
|
||||
|
||||
## 2. Components & responsibilities
|
||||
|
||||
| Component | Role | Key references |
|
||||
|-----------|------|----------------|
|
||||
| `StellaOps.Signer` | Issues DSSE envelopes using PoE-scoped keys (Fulcio or BYO KMS/HSM). | `ops/devops/signing/` |
|
||||
| `StellaOps.Attestor` | Submits DSSE payloads to Rekor v2, caches `{uuid,index,proof}` and mirrors proofs offline. | `docs/modules/attestor/architecture.md` |
|
||||
| Rekor v2 (managed or self-hosted) | Transparency log providing UUIDs + inclusion proofs. | `docs/ops/rekor/README.md` (if self-hosted) |
|
||||
| `StellaOps.Scanner` (WebService/Worker) | Requests attestations per scan, stores Rekor metadata next to SBOM artefacts. | `docs/modules/scanner/architecture.md` |
|
||||
| Export Center | Packages DSSE payloads + proofs into Offline Kit bundles and mirrors license notices. | `docs/modules/export-center/architecture.md` |
|
||||
| Policy Engine + CLI | Enforce “attested only” promotion, expose CLI verification verbs. | `docs/modules/policy/architecture.md`, `docs/09_API_CLI_REFERENCE.md` |
|
||||
|
||||
---
|
||||
|
||||
## 3. Prerequisites checklist
|
||||
|
||||
1. **Keys & trust roots**
|
||||
- Fulcio / KMS credentials available to `StellaOps.Signer`.
|
||||
- Rekor public key pinned (`rekor.pub`) for verification jobs and CLI tooling.
|
||||
2. **Service wiring**
|
||||
- `scanner.attestation.signerEndpoint` → internal Signer base URL.
|
||||
- `scanner.attestation.attestorEndpoint` → Attestor base URL.
|
||||
- `attestor.rekor.api` & `attestor.rekor.pubkey` set for the target log.
|
||||
3. **Storage**
|
||||
- Mongo collections `attestations` & `rekorProofs` sized for retention (7–30 days recommended).
|
||||
- Object store tier with at-rest encryption for DSSE payloads.
|
||||
4. **Observability**
|
||||
- Metrics: `attestor_rekor_success_total`, `attestor_rekor_retry_total`, `rekor_inclusion_latency`.
|
||||
- Logs shipped to your SIEM for compliance (Signer request/response IDs, Rekor UUIDs).
|
||||
5. **Offline readiness**
|
||||
- Export Center profile with `attestations.bundle=true`.
|
||||
- Rekor log snapshots mirrored (ORAS bundle or rsync of `/var/log/rekor`) for disconnected verification.
|
||||
|
||||
---
|
||||
|
||||
## 4. Enablement workflow
|
||||
|
||||
### 4.1 Configure Signer & Attestor
|
||||
|
||||
```yaml
|
||||
signer:
|
||||
schemaVersion: 2
|
||||
keyProvider: kms-fleet
|
||||
attestorEndpoint: https://attestor.internal
|
||||
defaultPredicate: https://stella-ops.org/attestations/sbom/1
|
||||
|
||||
attestor:
|
||||
schemaVersion: 1
|
||||
rekor:
|
||||
api: https://rekor.internal
|
||||
publicKeyPath: /etc/rekor/rekor.pub
|
||||
offlineMirrorPath: /var/lib/rekor/snapshots
|
||||
retry:
|
||||
maxAttempts: 5
|
||||
backoffSeconds: 15
|
||||
```
|
||||
|
||||
### 4.2 Turn on Scanner enforcement
|
||||
|
||||
```yaml
|
||||
scanner:
|
||||
schemaVersion: 2
|
||||
attestation:
|
||||
requireDsse: true # fail scans when Signer/Attestor errors occur
|
||||
signerEndpoint: https://signer.internal
|
||||
attestorEndpoint: https://attestor.internal
|
||||
uploadArtifacts: true # store DSSE + proof next to SBOM artefacts
|
||||
```
|
||||
|
||||
Set `requireDsse=false` during observation, then flip to `true` once Rekor health SLOs are green.
|
||||
|
||||
### 4.3 Policy templates
|
||||
|
||||
Add Policy Engine predicates (Rego snippet):
|
||||
|
||||
```rego
|
||||
package stella.policies.attestation
|
||||
|
||||
deny[msg] {
|
||||
not input.attestations.rekor_verified
|
||||
msg := sprintf("missing Rekor proof for %s", [input.scan_id])
|
||||
}
|
||||
|
||||
warn[msg] {
|
||||
input.attestations.rekor_age_hours > 24
|
||||
msg := sprintf("Rekor proof older than 24h for %s", [input.scan_id])
|
||||
}
|
||||
```
|
||||
|
||||
Tie Scheduler or CI promotion gates to the `deny` result.
|
||||
|
||||
### 4.4 CLI and verification
|
||||
|
||||
- `stellaops-cli runtime policy test --image <digest> --json` already surfaces `attestation.uuid` and `rekorVerified` fields.
|
||||
- To validate bundles offline: `stellaops-cli attest verify --bundle path/to/export.tar --rekor-key rekor.pub`.
|
||||
|
||||
Document these flows for AppSec teams so they can self-serve proofs during audits.
|
||||
|
||||
### 4.5 Export Center profile
|
||||
|
||||
```yaml
|
||||
exportProfiles:
|
||||
secure-default:
|
||||
includeSboms: true
|
||||
includeAttestations: true
|
||||
includeRekorProofs: true
|
||||
policy:
|
||||
requireAttestations: true
|
||||
allowUnsigned: false
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 5. Rollout levers & phases
|
||||
|
||||
| Phase | Toggle | Goal |
|
||||
|-------|--------|------|
|
||||
| **Observe** | `scanner.attestation.requireDsse=false`, policies in `warn` mode. | Validate plumbing without blocking builds; capture metrics. |
|
||||
| **Enforce** | Flip `requireDsse=true`, policy `deny` for missing proofs, Rekor SLO alerts live. | Block unsigned artefacts; auto-retry attestor failures. |
|
||||
| **Escalate** | Export Center profile `includeAttestations=true`, CLI docs distributed, Notify alerts wired. | Broad communication + audit evidence ready. |
|
||||
|
||||
Roll forward per environment; keep the previous phase’s toggles for hot rollback.
|
||||
|
||||
---
|
||||
|
||||
## 6. Offline / air-gap guidance
|
||||
|
||||
1. **Mirror Rekor**: take log snapshots daily (`rekor-cli log export`) and add to the Offline Kit.
|
||||
2. **Bundle proofs**: Export Center must include `*.rekor.json` and `rekor-chain.pem` alongside DSSE envelopes.
|
||||
3. **CLI verification offline**:
|
||||
```bash
|
||||
stellaops-cli attest verify --bundle offline-kit.tar \
|
||||
--rekor-root hashsum.txt --rekor-tree treehead.json --rekor-key rekor.pub
|
||||
```
|
||||
4. **Fallback**: When Rekor connectivity is unavailable, Attestor queues submissions locally and emits `attestationPending=true`; policy can allow waivers for a limited TTL via `policy.attestations.deferHours`.
|
||||
|
||||
---
|
||||
|
||||
## 7. Troubleshooting
|
||||
|
||||
| Symptom | Checks | Resolution |
|
||||
|---------|--------|------------|
|
||||
| `attestationPending` flag stays true | `attestor_rekor_retry_total`, Attestor logs, Rekor `/healthz`. | Verify Rekor endpoint & certs; rotate API tokens; replay queued DSSE payloads via `attestor replay`. |
|
||||
| Policy denies despite DSSE | Confirm Rekor proof bundle stored under `/artifacts/<scanId>/rekor/`. | Re-run `stellaops-cli attest verify`, ensure Policy Engine has the new schema (`attestations.rekor_verified`). |
|
||||
| CLI verification fails offline | Ensure Rekor snapshot + `rekor.pub` shipped together; check timestamp gap. | Regenerate snapshot, or import Rekor entries into the isolated log before verifying. |
|
||||
|
||||
---
|
||||
|
||||
## References
|
||||
|
||||
- Gap analysis: `docs/benchmarks/scanner/scanning-gaps-stella-misses-from-competitors.md#dsse-rekor-operator-enablement-trivy-grype-snyk`
|
||||
- Scanner architecture (§Signer → Attestor → Rekor): `docs/modules/scanner/architecture.md`
|
||||
- Export Center profiles: `docs/modules/export-center/architecture.md`
|
||||
- Policy Engine predicates: `docs/modules/policy/architecture.md`
|
||||
- CLI reference: `docs/09_API_CLI_REFERENCE.md`
|
||||
|
||||
Reference in New Issue
Block a user