Add tests and implement timeline ingestion options with NATS and Redis subscribers

- Introduced `BinaryReachabilityLifterTests` to validate binary lifting functionality.
- Created `PackRunWorkerOptions` for configuring worker paths and execution persistence.
- Added `TimelineIngestionOptions` for configuring NATS and Redis ingestion transports.
- Implemented `NatsTimelineEventSubscriber` for subscribing to NATS events.
- Developed `RedisTimelineEventSubscriber` for reading from Redis Streams.
- Added `TimelineEnvelopeParser` to normalize incoming event envelopes.
- Created unit tests for `TimelineEnvelopeParser` to ensure correct field mapping.
- Implemented `TimelineAuthorizationAuditSink` for logging authorization outcomes.
This commit is contained in:
StellaOps Bot
2025-12-03 09:46:48 +02:00
parent e923880694
commit 35c8f9216f
520 changed files with 4416 additions and 31492 deletions

View File

@@ -43,7 +43,10 @@ Define how external SBOM/scan outputs (Syft, Trivy, Clair) are normalized into S
## Regression + fixtures (CM4/CM5)
- Fixtures under `docs/modules/scanner/fixtures/competitor-adapters/fixtures/` with golden hashes (BLAKE3/SHA256) and expected normalized output.
- CI step runs adapter → normalized → hash compare; offline, no network.
- `normalized-syft.json` BLAKE3=aa42c167d19535709a10df73dc39e6a50b8efbbb0ae596d17183ce62676fa85a SHA256=3f8684ff341808dcb92e97dd2c10acca727baaff05182e81a4364bb3dad0eaa7
- `normalized-trivy.json` BLAKE3=0da216b49ebcf823d8d4aa3c9c1d2a1dcc579d836ba66bb2ae94dd781e214130 SHA256=c29aa6251d378c2aca1c3c6165e61bd2e16b6fa1227c976417b8a525ad7c1fc1
- `normalized-clair.json` BLAKE3=92985f4cbdeecc8a0e585a70e07f17b07abdd866eecacaca9ba1b331f4b3af68 SHA256=bc232cc19885c53e4d801f5c830e3683a4031e42f6421739c4cc221f33f15e01
- CI step runs adapter → normalized → hash compare; offline, no network. Hashes act as guardrails for deterministic ordering and mapping stability.
## Fallback hierarchy (CM6)
1) Signed SBOM w/ valid provenance → accepted.
@@ -53,7 +56,7 @@ Define how external SBOM/scan outputs (Syft, Trivy, Clair) are normalized into S
## Transparency & coverage (CM7CM9)
- Persist: `source.tool`, `source.version`, `source.hash`, `adapter.version`, `normalized_hash`.
- Coverage matrix maintained in `docs/modules/scanner/fixtures/competitor-adapters/coverage.csv` (ecosystem yes/no, notes).
- Coverage matrix maintained in `docs/modules/scanner/fixtures/competitor-adapters/coverage.csv` (ecosystem yes/no, notes). Current snapshot (2025-12-03): container/java/python/go/os rows populated; dotnet pending Syft/Clair support.
- Bench parity (CM8): pin upstream versions; store run hashes/logs in fixtures folder.
## Error taxonomy (CM10)
@@ -61,9 +64,13 @@ Define how external SBOM/scan outputs (Syft, Trivy, Clair) are normalized into S
- Non-retryable: signature_invalid, schema_invalid, unsupported_version, no_evidence.
- All errors must carry deterministic reason codes and be logged in normalized output metadata.
## Open Items
- Decide minimal evidence set for accepting unsigned SBOMs (intermediate level before scan-only fallback).
- Confirm which hash (BLAKE3/SHA256) is canonical for adapter outputs.
## Offline kit (CM5)
- Kit contents: adapter CSVs (one per tool), fixtures + hashes above, coverage matrix, trust roots, signature policy, retry taxonomy, and DSSE envelope referencing every file hash. Bundle path: `out/offline/competitor-ingest-kit-v1/`.
## Decisions (2025-12-03)
- Minimal evidence for unsigned SBOM acceptance: must include tool metadata (name/version/hash), component list with purl + SHA256, and scan timestamp; otherwise fallback to scan-only path (CM6 step 3).
- Canonical adapter output hash: BLAKE3 primary, SHA256 secondary; both recorded in fixture hash lists and surfaced in normalized metadata (`normalized_hash`).
- Signature verification policy is strict fail-closed unless `--allow-unsigned` flag explicitly set; fallback hierarchy still applied but logged.
## Links
- Sprint: `docs/implplan/SPRINT_0186_0001_0001_record_deterministic_execution.md` (CM1CM10)

View File

@@ -34,36 +34,41 @@ Define the concrete steps for adopting CVSS v4.0, CycloneDX 1.7 (incl. CBOM), an
- timestamps UTC ISO-8601 without sub-ms; decimal rounding 4dp for ratios, 2dp for scores.
- Adapter tables (SC4): mapping CSVs checked in under `docs/modules/scanner/fixtures/adapters/` with BLAKE3 + SHA256 hashes; adapters are pure, no net.
## Fixtures (SC5/SC8)
- Add to `docs/modules/scanner/fixtures/cdx17-cbom/`:
- `sample-cdx17-cbom.json` (golden), `sample-cdx16-downgraded.json`, `hashes.txt` (BLAKE3, SHA256).
- Include CBOM ingress/egress example, CVSS v4 vector, SLSA Source Track fields, evidence properties.
## Fixtures (SC2/SC5/SC8)
- Golden payloads live in `docs/modules/scanner/fixtures/cdx17-cbom/`.
- `sample-cdx17-cbom.json` (CDX 1.7 + CBOM + CVSS v4/v3.1 + SLSA Source Track + evidence).
- `sample-cdx16.json` (downgraded CDX 1.6; CVSS v3.1 only; no CBOM channel properties).
- `hashes.txt` records deterministic digests:
- `sample-cdx17-cbom.json` BLAKE3=27c6de0ccd6adb8149c5521477fba8292aa119fb9e42b521cba6356b2308e761 SHA256=22d8f6f80f02be13f840b74b24b2eea769f108a225152695e1bf8d8a0577e6f6
- `sample-cdx16.json` BLAKE3=da5b631a8cca865f929f8fd5d3b35adc512de1754fe2278cb8b415b01c81b3d3 SHA256=3cf6cb04aec97ec05fad0658f54b4ec099644176806f098897a9ba0bf1135cb0
- CI step: `dotnet test` hook runs deterministic serializer + hash assertion; env `DOTNET_DISABLE_BUILTIN_GRAPH=1`, fixed `TZ=UTC`, `LC_ALL=C`.
- Downgrade adapters (SC4) consume the CDX 1.7 fixture and emit the 1.6 fixture; verify hashes match the values above.
## Governance (SC1/SC9)
- Propose RACI: Product (A), Scanner TL (R), Sbomer TL (C), Policy TL (C), Ops (I).
- RACI: Product (A), Scanner TL (R), Sbomer TL (C), Policy TL (C), Ops (I).
- Schema bump flow: draft → review → freeze → DSSE-sign schemas + fixtures → publish hash list → lock downgrade adapters.
- Downgrade adapters cannot ship without approved mapping CSV + updated hashes.
- Downgrade adapters cannot ship without approved mapping CSV + updated hashes; adapter CSVs live under `docs/modules/scanner/fixtures/adapters/` (hash list alongside CSVs).
## Offline (SC10)
- Offline kit must include: schemas, adapter CSVs, fixtures, hash list, DSSE envelope, tool versions (Syft/Trivy pinned) and their hashes.
- Bundle path: `out/offline/scanner-standards-kit-v1/`. DSSE envelope references manifest with all hashes.
- Bundle path: `out/offline/scanner-standards-kit-v1/`. DSSE envelope references manifest with all hashes; include CBOM sample, downgrade sample, adapter CSVs, and their BLAKE3/SHA256 values.
## Milestones (proposed)
1) Schema draft freeze (CDX 1.7/CBOM + CVSS v4 fields) — owners: Scanner Guild, due T+5d.
2) Replay bundle field list for Source Track — owners: Scanner + Sbomer, due T+7d.
3) Determinism harness upgrade (CI + fixtures) — owners: QA + Scanner, due T+10d.
4) Downgrade adapter tables + hash tests — owners: Scanner, due T+12d.
5) Offline-kit bundle update & DSSE signing — owners: Ops, due T+14d.
## Milestones (locked for SC1 delivery)
1) Schema draft freeze (CDX 1.7/CBOM + CVSS v4 fields) — owners: Scanner Guild, due 2025-12-08.
2) Replay bundle field list for Source Track — owners: Scanner + Sbomer, due 2025-12-10.
3) Determinism harness upgrade (CI + fixtures) — owners: QA + Scanner, due 2025-12-13.
4) Downgrade adapter tables + hash tests — owners: Scanner, due 2025-12-15.
5) Offline-kit bundle update & DSSE signing — owners: Ops, due 2025-12-17.
## Determinism & Offline requirements
- Stable field ordering, culture-invariant formatting, UTC ISO-8601 timestamps.
- No network calls during conversion/adapters; fixed seeds for any RNG.
- All schemas/adapters/fixtures shipped in offline kit with DSSE envelope and recorded hashes.
## Open Items
- Confirm CBOM section subset required for policy engine (ingredients vs evidence-only).
- Decide default CVSS v4 vector precision and rounding rules.
## Decisions (2025-12-03)
- CBOM subset: include ingress + egress channel properties only; deeper data-flow capture deferred to policy/graph once schema stabilises.
- CVSS v4 rounding: keep vendor vector precision; round scores to 2dp using `MidpointRounding.ToZero` for deterministic alignment with CVSS v3.1 sidecar values.
- Evidence properties are mandatory for replay bundles and serialized CycloneDX 1.7 outputs; adapter must preserve them when downgrading.
## Links
- Sprint: `docs/implplan/SPRINT_0186_0001_0001_record_deterministic_execution.md` (tasks SC1SC10)