Refactor code structure for improved readability and maintainability; optimize performance in key functions.
This commit is contained in:
master
@@ -8,6 +8,7 @@
|
||||
- Does not perform scanning; consumes Scanner outputs or supplied SPDX/CycloneDX blobs.
|
||||
- Does not author verdicts/policy; supplies evidence and projections to Policy/Concelier/Graph.
|
||||
- Append-only SBOM versions; mutations happen via new versions, never in-place edits.
|
||||
- Owns the SBOM lineage ledger for versioned uploads, diffs, and retention pruning.
|
||||
|
||||
## 2) Project layout
|
||||
- `src/SbomService/StellaOps.SbomService` — REST API + event emitters + orchestrator integration.
|
||||
@@ -25,7 +26,7 @@
|
||||
The service now owns an idempotent spine that converts OCI images into SBOMs and provenance bundles with DSSE and in-toto. The flow is intentionally air-gap ready:
|
||||
|
||||
- **Extract** OCI manifest/layers (hash becomes `contentAddress`).
|
||||
- **Build SBOM** in CycloneDX 1.6 and/or SPDX 3.0.1; canonicalize JSON before hashing (`sbomHash`).
|
||||
- **Build SBOM** in CycloneDX 1.7 and/or SPDX 3.0.1; canonicalize JSON before hashing (`sbomHash`).
|
||||
- **Sign** outputs as DSSE envelopes; predicate uses in-toto Statement with SLSA Provenance v1.
|
||||
- **Publish** attestations optionally to a transparency backend: `rekor`, `local-merkle`, or `null` (no-op). Local Merkle log keeps proofs for later sync when online.
|
||||
|
||||
@@ -42,23 +43,41 @@ Operational rules:
|
||||
|
||||
## 3) APIs (first wave)
|
||||
- `GET /sbom/paths?purl=...&artifact=...&scope=...&env=...` — returns ordered paths with runtime_flag/blast_radius and nearest-safe-version hint; supports `cursor` pagination.
|
||||
- `GET /sbom/versions?artifact=...` — time-ordered SBOM version timeline for Advisory AI; include provenance and source bundle hash.
|
||||
- `GET /console/sboms` — Console catalog with filters (artifact, license, scope, asset tags), cursor pagination, evaluation metadata, immutable JSON projection for drawer views.
|
||||
- `GET /components/lookup?purl=...` — component neighborhood for global search/Graph overlays; returns caches hints + tenant enforcement.
|
||||
- `POST /entrypoints` / `GET /entrypoints` — manage entrypoint/service node overrides feeding Cartographer relevance; deterministic defaults when unset.
|
||||
- `GET /sboms/{snapshotId}/projection` — Link-Not-Merge v1 projection returning hashes plus asset metadata (criticality, owner, environment, exposure flags, tags) alongside package/component graph.
|
||||
- `GET /sbom/versions?artifact=...` – time-ordered SBOM version timeline for Advisory AI; include provenance and source bundle hash.
|
||||
- `POST /sbom/upload` – BYOS upload endpoint; validates/normalizes SPDX 2.3/3.0 or CycloneDX 1.4–1.6 and registers a ledger version.
|
||||
- `GET /sbom/ledger/history` – list version history for an artifact (cursor pagination).
|
||||
- `GET /sbom/ledger/point` – resolve the SBOM version at a specific timestamp.
|
||||
- `GET /sbom/ledger/range` – query versions within a time range.
|
||||
- `GET /sbom/ledger/diff` – component/version/license diff between two versions.
|
||||
- `GET /sbom/ledger/lineage` – parent/child lineage edges for an artifact chain.
|
||||
- `GET /console/sboms` – Console catalog with filters (artifact, license, scope, asset tags), cursor pagination, evaluation metadata, immutable JSON projection for drawer views.
|
||||
- `GET /components/lookup?purl=...` – component neighborhood for global search/Graph overlays; returns caches hints + tenant enforcement.
|
||||
- `POST /entrypoints` / `GET /entrypoints` – manage entrypoint/service node overrides feeding Cartographer relevance; deterministic defaults when unset.
|
||||
- `GET /sboms/{snapshotId}/projection` – Link-Not-Merge v1 projection returning hashes plus asset metadata (criticality, owner, environment, exposure flags, tags) alongside package/component graph.
|
||||
- `GET /internal/sbom/events` — internal diagnostics endpoint returning the in-memory event outbox for validation.
|
||||
- `POST /internal/sbom/events/backfill` — replays existing projections into the event stream; deterministic ordering, clock abstraction for tests.
|
||||
- `GET /internal/sbom/asset-events` — diagnostics endpoint returning emitted `sbom.asset.updated` envelopes for validation and air-gap parity checks.
|
||||
- `GET/POST /internal/orchestrator/sources` — list/register orchestrator ingest/index sources (deterministic seeds; idempotent on artifactDigest+sourceType).
|
||||
- `GET/POST /internal/orchestrator/control` — manage pause/throttle/backpressure signals per tenant; metrics emitted for control updates.
|
||||
- `GET/POST /internal/orchestrator/watermarks` — fetch/set backfill watermarks for reconciliation and deterministic replays.
|
||||
- `GET /internal/sbom/resolver-feed` — list resolver candidates (artifact, purl, version, paths, scope, runtime_flag, nearest_safe_version).
|
||||
- `POST /internal/sbom/resolver-feed/backfill` — clear and repopulate resolver feed from current projections.
|
||||
- `GET /internal/sbom/resolver-feed/export` — NDJSON export of resolver candidates for air-gap delivery.
|
||||
- `GET /internal/sbom/resolver-feed` – list resolver candidates (artifact, purl, version, paths, scope, runtime_flag, nearest_safe_version).
|
||||
- `POST /internal/sbom/resolver-feed/backfill` – clear and repopulate resolver feed from current projections.
|
||||
- `GET /internal/sbom/resolver-feed/export` – NDJSON export of resolver candidates for air-gap delivery.
|
||||
- `GET /internal/sbom/ledger/audit` – audit trail for ledger changes (created/pruned).
|
||||
- `GET /internal/sbom/analysis/jobs` – list analysis jobs triggered by BYOS uploads.
|
||||
- `POST /internal/sbom/retention/prune` – apply retention policy and emit audit entries.
|
||||
|
||||
## 3.1) Ledger + BYOS workflow (Sprint 4600)
|
||||
- Uploads are validated, normalized, and stored as ledger versions chained per artifact identity.
|
||||
- Diffs compare normalized component keys and surface version/license deltas with deterministic ordering.
|
||||
- Lineage is derived from parent version references and emitted for Graph lineage edges.
|
||||
- Lineage relationships include parent links plus build links (shared CI build IDs when provided).
|
||||
- Retention policy prunes old versions while preserving audit entries and minimum keep counts.
|
||||
- See `docs/modules/sbomservice/ledger-lineage.md` for request/response examples.
|
||||
- See `docs/modules/sbomservice/byos-ingestion.md` for supported formats and troubleshooting.
|
||||
|
||||
## 4) Ingestion & orchestrator integration
|
||||
- Ingest sources: Scanner pipeline (preferred) or uploaded SPDX 3.0.1/CycloneDX 1.6 bundles.
|
||||
- Ingest sources: Scanner pipeline (preferred) or uploaded SPDX 2.3/3.0 and CycloneDX 1.4–1.6 bundles.
|
||||
- Orchestrator: register SBOM ingest/index jobs; worker SDK emits artifact hash + job metadata; honor pause/throttle; report backpressure metrics; support watermark-based backfill for idempotent replays.
|
||||
- Idempotency: combine `(tenant, artifactDigest, sbomVersion)` as primary key; duplicate ingests short-circuit.
|
||||
|
||||
@@ -80,7 +99,7 @@ Operational rules:
|
||||
- Input validation: schema-validate incoming SBOMs; reject oversized/unsupported media types early.
|
||||
|
||||
## 8) Observability
|
||||
- Metrics: `sbom_projection_seconds`, `sbom_projection_size_bytes`, `sbom_projection_queries_total`, `sbom_paths_latency_seconds`, `sbom_paths_cache_hit_ratio`, `sbom_events_backlog`.
|
||||
- Metrics: `sbom_projection_seconds`, `sbom_projection_size_bytes`, `sbom_projection_queries_total`, `sbom_paths_latency_seconds`, `sbom_paths_cache_hit_ratio`, `sbom_events_backlog`, `sbom_ledger_uploads_total`, `sbom_ledger_diffs_total`, `sbom_ledger_retention_pruned_total`.
|
||||
- Tracing: ActivitySource `StellaOps.SbomService` (entrypoints, component lookup, console catalog, projections, events).
|
||||
- Traces: wrap ingest, projection build, and API handlers; propagate orchestrator job IDs.
|
||||
- Logs: structured, include tenant + artifact digest + sbomVersion; classify ingest failures (schema, storage, orchestrator, validation).
|
||||
@@ -90,6 +109,7 @@ Operational rules:
|
||||
- Enable PostgreSQL storage for `/console/sboms` and `/components/lookup` by setting `SbomService:PostgreSQL:ConnectionString` (env: `SBOM_SbomService__PostgreSQL__ConnectionString`).
|
||||
- Optional overrides: `SbomService:PostgreSQL:Schema`, `SbomService:PostgreSQL:CatalogTable`, `SbomService:PostgreSQL:ComponentLookupTable`; defaults are `sbom_service`, `sbom_catalog`, `sbom_component_neighbors`.
|
||||
- When the connection string is absent the service falls back to fixture JSON or deterministic in-memory seeds to keep air-gapped workflows alive.
|
||||
- Ledger retention settings (env prefix `SBOM_SbomService__Ledger__`): `MaxVersionsPerArtifact`, `MaxAgeDays`, `MinVersionsToKeep`.
|
||||
|
||||
## 10) Open questions / dependencies
|
||||
- Confirm orchestrator pause/backfill contract (shared with Runtime & Signals 140-series).
|
||||
@@ -97,3 +117,5 @@ Operational rules:
|
||||
- Publish canonical LNM v1 fixtures and JSON schemas for projections and asset metadata.
|
||||
|
||||
- See `docs/modules/sbomservice/api/projection-read.md` for `/sboms/{snapshotId}/projection` (LNM v1, tenant-scoped, hash-returning).
|
||||
- See `docs/modules/sbomservice/lineage-ledger.md` for ledger endpoints and lineage relationships.
|
||||
- See `docs/modules/sbomservice/retention-policy.md` for retention configuration and audit expectations.
|
||||
|
||||
33
docs/modules/sbomservice/byos-ingestion.md
Normal file
33
docs/modules/sbomservice/byos-ingestion.md
Normal file
@@ -0,0 +1,33 @@
|
||||
# BYOS SBOM Ingestion
|
||||
|
||||
## Overview
|
||||
Bring-your-own SBOM (BYOS) uploads accept SPDX and CycloneDX JSON and register them in the SBOM ledger for analysis.
|
||||
|
||||
## Supported formats
|
||||
- CycloneDX JSON: 1.4, 1.5, 1.6
|
||||
- SPDX JSON: 2.3, 3.0
|
||||
|
||||
## Upload endpoint
|
||||
- `POST /sbom/upload` or `POST /api/v1/sbom/upload`
|
||||
- Required: `artifactRef`, plus `sbom` (JSON object) or `sbomBase64`.
|
||||
- Optional: `format` hint (`cyclonedx` or `spdx`) and `source` metadata.
|
||||
|
||||
Example:
|
||||
```json
|
||||
{
|
||||
"artifactRef": "acme/app:2.0",
|
||||
"sbom": { "spdxVersion": "SPDX-2.3", "packages": [] },
|
||||
"source": { "tool": "syft", "version": "1.9.0" }
|
||||
}
|
||||
```
|
||||
|
||||
## Validation notes
|
||||
- CycloneDX requires `bomFormat` and supported `specVersion`.
|
||||
- SPDX requires `spdxVersion` and a supported version number.
|
||||
- Quality scoring prefers components with PURL, version, and license metadata.
|
||||
|
||||
## Troubleshooting
|
||||
- **"sbom or sbomBase64 is required"**: include an SBOM payload in the request.
|
||||
- **"Unable to detect SBOM format"**: set `format` explicitly or include required root fields.
|
||||
- **Unsupported SBOM format/version**: ensure CycloneDX 1.4–1.6 or SPDX 2.3/3.0.
|
||||
- **Low quality scores**: include PURLs, versions, and license declarations where possible.
|
||||
41
docs/modules/sbomservice/ledger-lineage.md
Normal file
41
docs/modules/sbomservice/ledger-lineage.md
Normal file
@@ -0,0 +1,41 @@
|
||||
# SBOM Lineage Ledger Guide
|
||||
|
||||
## Purpose
|
||||
- Track historical SBOM versions per artifact with deterministic diffs and lineage edges.
|
||||
- Support BYOS uploads for SPDX/CycloneDX inputs while preserving audit history.
|
||||
|
||||
## Upload workflow
|
||||
- Endpoint: `POST /sbom/upload` or `POST /api/v1/sbom/upload`.
|
||||
- Required fields: `artifactRef`, `sbom` (JSON object) or `sbomBase64`.
|
||||
|
||||
Example payload:
|
||||
```json
|
||||
{
|
||||
"artifactRef": "acme/app:1.0",
|
||||
"sbom": { "bomFormat": "CycloneDX", "specVersion": "1.6", "components": [] },
|
||||
"format": "cyclonedx",
|
||||
"source": { "tool": "syft", "version": "1.0.0" }
|
||||
}
|
||||
```
|
||||
|
||||
## Ledger queries
|
||||
- `GET /sbom/ledger/history?artifact=...&limit=...&cursor=...`
|
||||
- `GET /sbom/ledger/point?artifact=...&at=...`
|
||||
- `GET /sbom/ledger/range?artifact=...&start=...&end=...&limit=...&cursor=...`
|
||||
- `GET /sbom/ledger/diff?before=...&after=...`
|
||||
- `GET /sbom/ledger/lineage?artifact=...`
|
||||
|
||||
## Lineage semantics
|
||||
- Versions are chained per artifact; parent references create lineage edges.
|
||||
- Graph Indexer maps ledger lineage into:
|
||||
- `SBOM_VERSION_OF` edges from SBOM to artifact.
|
||||
- `SBOM_LINEAGE_*` edges (e.g., `SBOM_LINEAGE_PARENT`).
|
||||
|
||||
## Retention policy
|
||||
- Apply with `POST /internal/sbom/retention/prune`.
|
||||
- Settings: `SbomService:Ledger:MaxVersionsPerArtifact`, `MaxAgeDays`, `MinVersionsToKeep`.
|
||||
- Audit is available via `GET /internal/sbom/ledger/audit`.
|
||||
|
||||
## Determinism
|
||||
- Versions are ordered by sequence and UTC timestamps.
|
||||
- Diffs are ordered by component key for stable output.
|
||||
30
docs/modules/sbomservice/lineage-ledger.md
Normal file
30
docs/modules/sbomservice/lineage-ledger.md
Normal file
@@ -0,0 +1,30 @@
|
||||
# SBOM lineage ledger
|
||||
|
||||
## Overview
|
||||
- Tracks immutable SBOM versions per artifact reference.
|
||||
- Exposes history, temporal queries, and deterministic diffs.
|
||||
- Emits lineage edges to support graph joins and audit trails.
|
||||
|
||||
## Endpoints
|
||||
- `GET /sbom/ledger/history?artifact=<ref>&limit=50&cursor=0`
|
||||
- `GET /sbom/ledger/point?artifact=<ref>&at=<iso8601>`
|
||||
- `GET /sbom/ledger/range?artifact=<ref>&start=<iso8601>&end=<iso8601>`
|
||||
- `GET /sbom/ledger/diff?before=<versionId>&after=<versionId>`
|
||||
- `GET /sbom/ledger/lineage?artifact=<ref>`
|
||||
|
||||
## Lineage relationships
|
||||
- `parent`: explicit parent version link (supplied at ingest).
|
||||
- `build`: versions emitted from the same CI build ID (from upload provenance).
|
||||
|
||||
## Example lineage response
|
||||
```json
|
||||
{
|
||||
"artifactRef": "example.com/app:1.2.3",
|
||||
"nodes": [{ "versionId": "v1", "sequenceNumber": 1, "digest": "sha256:..." }],
|
||||
"edges": [{ "fromVersionId": "v1", "toVersionId": "v2", "relationship": "build" }]
|
||||
}
|
||||
```
|
||||
|
||||
## Notes
|
||||
- Ledger storage is in-memory until PostgreSQL-backed persistence is wired.
|
||||
- Ordering is deterministic by sequence number, then timestamp.
|
||||
18
docs/modules/sbomservice/retention-policy.md
Normal file
18
docs/modules/sbomservice/retention-policy.md
Normal file
@@ -0,0 +1,18 @@
|
||||
# SBOM ledger retention policy
|
||||
|
||||
## Purpose
|
||||
Retention keeps ledger history bounded while preserving audit trails for compliance.
|
||||
|
||||
## Configuration
|
||||
Settings are bound from `SbomService:Ledger` (env prefix `SBOM_SbomService__Ledger__`):
|
||||
- `MaxVersionsPerArtifact`: max ledger versions retained per artifact (default 50).
|
||||
- `MaxAgeDays`: prune versions older than N days (0 disables age pruning).
|
||||
- `MinVersionsToKeep`: minimum versions always retained per artifact.
|
||||
|
||||
## Operations
|
||||
- `POST /internal/sbom/retention/prune` applies retention rules and returns a summary.
|
||||
- `GET /internal/sbom/ledger/audit?artifact=<ref>` returns audit entries for create/prune actions.
|
||||
|
||||
## Guarantees
|
||||
- Audit entries are append-only and preserved even when versions are pruned.
|
||||
- Deterministic ordering is used when selecting versions to prune.
|
||||
Reference in New Issue
Block a user