Add unit tests for SBOM ingestion and transformation
Some checks failed
Docs CI / lint-and-preview (push) Has been cancelled

- Implement `SbomIngestServiceCollectionExtensionsTests` to verify the SBOM ingestion pipeline exports snapshots correctly.
- Create `SbomIngestTransformerTests` to ensure the transformation produces expected nodes and edges, including deduplication of license nodes and normalization of timestamps.
- Add `SbomSnapshotExporterTests` to test the export functionality for manifest, adjacency, nodes, and edges.
- Introduce `VexOverlayTransformerTests` to validate the transformation of VEX nodes and edges.
- Set up project file for the test project with necessary dependencies and configurations.
- Include JSON fixture files for testing purposes.
This commit is contained in:
master
2025-11-04 07:49:39 +02:00
parent f72c5c513a
commit 2eb6852d34
491 changed files with 39445 additions and 3917 deletions

View File

@@ -5,12 +5,13 @@ Depends on: Sprint 120.A - AirGap, Sprint 130.A - Scanner
Summary: Runtime & Signals focus on Graph).
Task ID | State | Task description | Owners (Source)
--- | --- | --- | ---
GRAPH-INDEX-28-001 | TODO | Define canonical node/edge schemas, attribute dictionaries, identity rules, and seed fixtures; publish schema doc. | Graph Indexer Guild (src/Graph/StellaOps.Graph.Indexer/TASKS.md)
GRAPH-INDEX-28-002 | TODO | Implement SBOM ingest consumer producing artifact/package/file nodes and edges with `valid_from/valid_to`, scope metadata, and provenance links. Dependencies: GRAPH-INDEX-28-001. | Graph Indexer Guild (src/Graph/StellaOps.Graph.Indexer/TASKS.md)
GRAPH-INDEX-28-003 | TODO | Project Concelier linksets into overlay tiles (`affected_by` edges, evidence refs) without mutating source observations; keep advisory aggregates in overlay store only. Dependencies: GRAPH-INDEX-28-002. | Graph Indexer Guild (src/Graph/StellaOps.Graph.Indexer/TASKS.md)
GRAPH-INDEX-28-004 | TODO | Integrate VEX statements (`vex_exempts` edges) with justification metadata and precedence markers for overlays. Dependencies: GRAPH-INDEX-28-003. | Graph Indexer Guild (src/Graph/StellaOps.Graph.Indexer/TASKS.md)
GRAPH-INDEX-28-005 | TODO | Hydrate policy overlays into graph (`governs_with` nodes/edges) referencing effective findings and explain hashes for sampled nodes. Dependencies: GRAPH-INDEX-28-004. | Graph Indexer Guild, Policy Guild (src/Graph/StellaOps.Graph.Indexer/TASKS.md)
GRAPH-INDEX-28-006 | TODO | Generate graph snapshots per SBOM with lineage (`derived_from`), adjacency manifests, and metadata for diff jobs. Dependencies: GRAPH-INDEX-28-005. | Graph Indexer Guild (src/Graph/StellaOps.Graph.Indexer/TASKS.md)
GRAPH-INDEX-28-001 | DONE (2025-11-03) | Define canonical node/edge schemas, attribute dictionaries, identity rules, and seed fixtures; publish schema doc.<br>2025-11-03: Published `docs/modules/graph/schema.md` v1, refreshed fixtures (`nodes.json`, `edges.json`), and aligned GraphIdentity determinism tests. | Graph Indexer Guild (src/Graph/StellaOps.Graph.Indexer/TASKS.md)
GRAPH-INDEX-28-002 | DONE (2025-11-03) | Implement SBOM ingest consumer producing artifact/package/file nodes and edges with `valid_from/valid_to`, scope metadata, and provenance links. Dependencies: GRAPH-INDEX-28-001.<br>2025-11-03: Snapshot models restored, provenance resolution tightened, ingest processor + metrics added, transformer/fixtures/tests expanded for license + base artifact determinism. | Graph Indexer Guild (src/Graph/StellaOps.Graph.Indexer/TASKS.md)
GRAPH-INDEX-28-003 | DONE (2025-11-03) | Project Concelier linksets into overlay tiles (`affected_by` edges, evidence refs) without mutating source observations; keep advisory aggregates in overlay store only. Dependencies: GRAPH-INDEX-28-002. | Graph Indexer Guild (src/Graph/StellaOps.Graph.Indexer/TASKS.md)<br>2025-11-03: Advisory linkset snapshot/transformer coded with AFFECTED_BY edges plus fixture-backed tests; overlay persistence wiring landed; graph overlay suite green.
GRAPH-INDEX-28-004 | DONE (2025-11-03) | Integrate VEX statements (`vex_exempts` edges) with justification metadata and precedence markers for overlays. Dependencies: GRAPH-INDEX-28-003. | Graph Indexer Guild (src/Graph/StellaOps.Graph.Indexer/TASKS.md)<br>2025-11-03: VEX snapshot + transformer emit deterministic VEX_EXEMPTS overlays with provenance hashes; fixtures/tests updated; full graph indexer tests pass.
GRAPH-INDEX-28-005 | DONE (2025-11-03) | Hydrate policy overlays into graph (`governs_with` nodes/edges) referencing effective findings and explain hashes for sampled nodes. Dependencies: GRAPH-INDEX-28-004. | Graph Indexer Guild, Policy Guild (src/Graph/StellaOps.Graph.Indexer/TASKS.md)<br>2025-11-03: Policy snapshot/transformer emit deterministic policy_version nodes and GOVERNS_WITH edges; fixtures/tests updated; targeted transformer tests pass (full Mongo-backed suite requires local mongod).<br>2025-11-03: Processor + OTEL metrics wired to Mongo writer with unit coverage for success/failure; Mongo2Go-backed writer tests now fall back to `STELLAOPS_TEST_MONGO_URI` or `mongodb://127.0.0.1:27017` when available, otherwise they skip with guidance.
GRAPH-INDEX-28-006 | DONE (2025-11-03) | Generate graph snapshots per SBOM with lineage (`derived_from`), adjacency manifests, and metadata for diff jobs. Dependencies: GRAPH-INDEX-28-005. | Graph Indexer Guild (src/Graph/StellaOps.Graph.Indexer/TASKS.md)<br>2025-11-03: Snapshot builder + adjacency manifest added with hashed metadata, tests covering lineage/edges landed, docs note required `STELLAOPS_TEST_MONGO_URI`.<br>2025-11-03: Snapshot exporter writes manifest/adjacency/nodes/edges to snapshot directory with deterministic ordering.
GRAPH-INDEX-28-011 | DONE (2025-11-04) | Wire SBOM ingest runtime to emit snapshot artifacts and align dev/CI Mongo availability. Dependencies: GRAPH-INDEX-28-006. | Graph Indexer Guild (src/Graph/StellaOps.Graph.Indexer/TASKS.md)<br>2025-11-04: Added `AddSbomIngestPipeline` DI wiring with configurable snapshot root (`STELLAOPS_GRAPH_SNAPSHOT_DIR`), updated docs for Mongo/snapshot env vars, and ran Graph Indexer tests (Mongo writer skipped when URI absent).
GRAPH-INDEX-28-007 | TODO | Implement clustering/centrality background jobs (Louvain/degree/betweenness approximations) with configurable schedules and store cluster ids on nodes. Dependencies: GRAPH-INDEX-28-006. | Graph Indexer Guild, Observability Guild (src/Graph/StellaOps.Graph.Indexer/TASKS.md)
GRAPH-INDEX-28-008 | TODO | Provide incremental update + backfill pipeline with change streams, retry/backoff, idempotent operations, and backlog metrics. Dependencies: GRAPH-INDEX-28-007. | Graph Indexer Guild (src/Graph/StellaOps.Graph.Indexer/TASKS.md)
GRAPH-INDEX-28-009 | TODO | Add unit/property/integration tests, synthetic large graph fixtures, chaos testing (missing overlays, cycles), and determinism checks across runs. Dependencies: GRAPH-INDEX-28-008. | Graph Indexer Guild, QA Guild (src/Graph/StellaOps.Graph.Indexer/TASKS.md)
@@ -19,7 +20,7 @@ GRAPH-INDEX-28-010 | TODO | Package deployment artifacts (Helm/Compose), offline
[Runtime & Signals] 140.B) SbomService
Depends on: Sprint 120.A - AirGap, Sprint 130.A - Scanner
Summary: Runtime & Signals focus on SbomService).
Summary: Runtime & Signals focus on SBOM Service — projections, APIs, and orchestrator integration.
Task ID | State | Task description | Owners (Source)
--- | --- | --- | ---
SBOM-AIAI-31-001 | TODO | Provide `GET /sbom/paths?purl=...` and version timeline endpoints optimized for Advisory AI (incl. env flags, blast radius metadata). | SBOM Service Guild (src/SbomService/StellaOps.SbomService/TASKS.md)
@@ -29,10 +30,10 @@ SBOM-CONSOLE-23-002 | TODO | Deliver component lookup endpoints powering global
SBOM-ORCH-32-001 | TODO | Register SBOM ingest/index sources with orchestrator, embed worker SDK, and emit artifact hashes + job metadata. | SBOM Service Guild (src/SbomService/StellaOps.SbomService/TASKS.md)
SBOM-ORCH-33-001 | TODO | Report backpressure metrics, honor orchestrator pause/throttle signals, and classify error outputs for sbom jobs. Dependencies: SBOM-ORCH-32-001. | SBOM Service Guild (src/SbomService/StellaOps.SbomService/TASKS.md)
SBOM-ORCH-34-001 | TODO | Implement orchestrator backfill + watermark reconciliation for SBOM ingest/index, ensuring idempotent artifact reuse. Dependencies: SBOM-ORCH-33-001. | SBOM Service Guild (src/SbomService/StellaOps.SbomService/TASKS.md)
SBOM-SERVICE-21-001 | BLOCKED (2025-10-27) | Publish normalized SBOM projection schema (components, relationships, scopes, entrypoints) and implement read API with pagination + tenant enforcement. | SBOM Service Guild, Cartographer Guild (src/SbomService/StellaOps.SbomService/TASKS.md)
SBOM-SERVICE-21-002 | BLOCKED (2025-10-27) | Emit change events (`sbom.version.created`) carrying digest/version metadata for Graph Indexer builds; add replay/backfill tooling. Dependencies: SBOM-SERVICE-21-001. | SBOM Service Guild, Scheduler Guild (src/SbomService/StellaOps.SbomService/TASKS.md)
SBOM-SERVICE-21-003 | BLOCKED (2025-10-27) | Provide entrypoint/service node management API (list/update overrides) feeding Cartographer path relevance with deterministic defaults. Dependencies: SBOM-SERVICE-21-002. | SBOM Service Guild (src/SbomService/StellaOps.SbomService/TASKS.md)
SBOM-SERVICE-21-004 | BLOCKED (2025-10-27) | Wire observability: metrics (`sbom_projection_seconds`, `sbom_projection_size`), traces, structured logs with tenant info; set alerts for backlog. Dependencies: SBOM-SERVICE-21-003. | SBOM Service Guild, Observability Guild (src/SbomService/StellaOps.SbomService/TASKS.md)
SBOM-SERVICE-21-001 | BLOCKED (2025-10-27) | Publish normalized SBOM projection schema (components, relationships, scopes, entrypoints) and implement read API with pagination + tenant enforcement.<br>2025-10-27: Awaiting projection schema from Concelier (`CONCELIER-GRAPH-21-001`) before finalizing API payloads and fixtures. | SBOM Service Guild, Cartographer Guild (src/SbomService/StellaOps.SbomService/TASKS.md)
SBOM-SERVICE-21-002 | BLOCKED (2025-10-27) | Emit change events (`sbom.version.created`) carrying digest/version metadata for Graph Indexer builds; add replay/backfill tooling. Dependencies: SBOM-SERVICE-21-001.<br>2025-10-27: Blocked until `SBOM-SERVICE-21-001` defines projection schema and endpoints. | SBOM Service Guild, Scheduler Guild (src/SbomService/StellaOps.SbomService/TASKS.md)
SBOM-SERVICE-21-003 | BLOCKED (2025-10-27) | Provide entrypoint/service node management API (list/update overrides) feeding Cartographer path relevance with deterministic defaults. Dependencies: SBOM-SERVICE-21-002.<br>2025-10-27: Depends on base projection schema (`SBOM-SERVICE-21-001`) which is blocked. | SBOM Service Guild (src/SbomService/StellaOps.SbomService/TASKS.md)
SBOM-SERVICE-21-004 | BLOCKED (2025-10-27) | Wire observability: metrics (`sbom_projection_seconds`, `sbom_projection_size`), traces, structured logs with tenant info; set alerts for backlog. Dependencies: SBOM-SERVICE-21-003.<br>2025-10-27: Projection pipeline not in place yet; will follow once `SBOM-SERVICE-21-001` unblocks. | SBOM Service Guild, Observability Guild (src/SbomService/StellaOps.SbomService/TASKS.md)
SBOM-SERVICE-23-001 | TODO | Extend projections to include asset metadata (criticality, owner, environment, exposure flags) required by policy rules; update schema docs. Dependencies: SBOM-SERVICE-21-004. | SBOM Service Guild, Policy Guild (src/SbomService/StellaOps.SbomService/TASKS.md)
SBOM-SERVICE-23-002 | TODO | Emit `sbom.asset.updated` events when metadata changes; ensure idempotent payloads and documentation. Dependencies: SBOM-SERVICE-23-001. | SBOM Service Guild, Platform Events Guild (src/SbomService/StellaOps.SbomService/TASKS.md)
SBOM-VULN-29-001 | TODO | Emit inventory evidence with `scope`, `runtime_flag`, dependency paths, and nearest safe version hints, streaming change events for resolver jobs. | SBOM Service Guild (src/SbomService/StellaOps.SbomService/TASKS.md)
@@ -41,24 +42,27 @@ SBOM-VULN-29-002 | TODO | Provide resolver feed (artifact, purl, version, paths)
[Runtime & Signals] 140.C) Signals
Depends on: Sprint 120.A - AirGap, Sprint 130.A - Scanner
Summary: Runtime & Signals focus on Signals).
Summary: Runtime & Signals focus on Signals — reachability ingestion and scoring.
Notes:
- 2025-10-29: Skeleton live with scope policies, stub endpoints, and integration tests; sample configuration committed under `etc/signals.yaml.sample`.
- 2025-10-29: JSON parsers for Java/Node.js/Python/Go implemented; artifacts stored on filesystem with SHA-256 and callgraphs upserted into Mongo.
Task ID | State | Task description | Owners (Source)
--- | --- | --- | ---
SIGNALS-24-003 | BLOCKED (2025-10-27) | Implement runtime facts ingestion endpoint and normalizer (process, sockets, container metadata) populating `context_facts` with AOC provenance. | Signals Guild, Runtime Guild (src/Signals/StellaOps.Signals/TASKS.md)
SIGNALS-24-004 | BLOCKED (2025-10-27) | Deliver reachability scoring engine producing states/scores and writing to `reachability_facts`; expose configuration for weights. Dependencies: SIGNALS-24-003. | Signals Guild, Data Science (src/Signals/StellaOps.Signals/TASKS.md)
SIGNALS-24-005 | BLOCKED (2025-10-27) | Implement Redis caches (`reachability_cache:*`), invalidation on new facts, and publish `signals.fact.updated` events. Dependencies: SIGNALS-24-004. | Signals Guild, Platform Events Guild (src/Signals/StellaOps.Signals/TASKS.md)
SIGNALS-24-003 | BLOCKED (2025-10-27) | Implement runtime facts ingestion endpoint and normalizer (process, sockets, container metadata) populating `context_facts` with AOC provenance.<br>2025-10-27: Depends on `SIGNALS-24-001` for base API host and authentication plumbing. | Signals Guild, Runtime Guild (src/Signals/StellaOps.Signals/TASKS.md)
SIGNALS-24-004 | BLOCKED (2025-10-27) | Deliver reachability scoring engine producing states/scores and writing to `reachability_facts`; expose configuration for weights. Dependencies: SIGNALS-24-003.<br>2025-10-27: Upstream ingestion pipelines (`SIGNALS-24-002/003`) blocked; scoring engine cannot proceed. | Signals Guild, Data Science (src/Signals/StellaOps.Signals/TASKS.md)
SIGNALS-24-005 | BLOCKED (2025-10-27) | Implement Redis caches (`reachability_cache:*`), invalidation on new facts, and publish `signals.fact.updated` events. Dependencies: SIGNALS-24-004.<br>2025-10-27: Awaiting scoring engine and ingestion layers before wiring cache/events. | Signals Guild, Platform Events Guild (src/Signals/StellaOps.Signals/TASKS.md)
[Runtime & Signals] 140.D) Zastava
Depends on: Sprint 120.A - AirGap, Sprint 130.A - Scanner
Summary: Runtime & Signals focus on Zastava).
Summary: Runtime & Signals focus on Zastava — observer and webhook Surface integration.
Task ID | State | Task description | Owners (Source)
--- | --- | --- | ---
ZASTAVA-ENV-01 | TODO | Adopt Surface.Env helpers for cache endpoints, secret refs, and feature toggles. | Zastava Observer Guild (src/Zastava/StellaOps.Zastava.Observer/TASKS.md)
ZASTAVA-ENV-02 | TODO | Switch to Surface.Env helpers for webhook configuration (cache endpoint, secret refs, feature toggles). Dependencies: ZASTAVA-ENV-01. | Zastava Webhook Guild (src/Zastava/StellaOps.Zastava.Webhook/TASKS.md)
ZASTAVA-SECRETS-01 | TODO | Retrieve CAS/attestation access via Surface.Secrets instead of inline secret stores. | Zastava Observer Guild, Security Guild (src/Zastava/StellaOps.Zastava.Observer/TASKS.md)
ZASTAVA-SECRETS-02 | TODO | Retrieve attestation verification secrets via Surface.Secrets. Dependencies: ZASTAVA-SECRETS-01. | Zastava Webhook Guild, Security Guild (src/Zastava/StellaOps.Zastava.Webhook/TASKS.md)
ZASTAVA-SURFACE-01 | TODO | Integrate Surface.FS client for runtime drift detection (lookup cached layer hashes/entry traces). | Zastava Observer Guild (src/Zastava/StellaOps.Zastava.Observer/TASKS.md)
ZASTAVA-SURFACE-01 | TODO | Integrate Surface.FS client for runtime drift detection (lookup cached layer hashes/entry traces).<br>2025-10-24: Observer unit tests pending; `dotnet restore` needs offline copies of `Google.Protobuf`, `Grpc.Net.Client`, and `Grpc.Tools` in `local-nuget` before verification. | Zastava Observer Guild (src/Zastava/StellaOps.Zastava.Observer/TASKS.md)
ZASTAVA-SURFACE-02 | TODO | Enforce Surface.FS availability during admission (deny when cache missing/stale) and embed pointer checks in webhook response. Dependencies: ZASTAVA-SURFACE-01. | Zastava Webhook Guild (src/Zastava/StellaOps.Zastava.Webhook/TASKS.md)