docs: module dossier + install/quickstart sync for truthful cutover sprints

- API_CLI_REFERENCE.md, INSTALL_GUIDE.md, quickstart.md, architecture/integrations.md, dev/DEV_ENVIRONMENT_SETUP.md, integrations/LOCAL_SERVICES.md: reflect real-service wiring.
- docs/modules/**: module dossier updates across the modules touched by SPRINT_20260415_001..007 + SPRINT_20260416_003..017 + SPRINT_20260417_018..024 + SPRINT_20260418_025 + SPRINT_20260419_026.
- docs/features/checked/web/**: update feature notes where UI changed.
- docs/qa/feature-checks/runs/web/evidence-presentation-ux/: QA evidence artifacts.
- docs/setup/**, docs/technical/**: align with setup wizard contracts.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
master
2026-04-19 14:45:09 +03:00
parent ad62ba7f76
commit fdf95e0f46
67 changed files with 590 additions and 360 deletions

View File

@@ -1,176 +1,103 @@
# SBOM Service architecture (2025Q4)
# SBOM Service Architecture
> Scope: canonical SBOM projections, lookup and timeline APIs, asset metadata overlays, and events feeding Advisory AI, Console, Graph, Policy, and Vuln Explorer.
> Scope: canonical SBOM projections, ledger history, lineage, registry-source state, and durable SBOM event backlogs consumed by Console, Policy, Concelier, Graph, and Findings.
## 1) Mission & boundaries
- Mission: serve deterministic, tenant-scoped SBOM projections (Link-Not-Merge v1) and related metadata for downstream reasoning and overlays.
- Boundaries:
- Does not perform scanning; consumes Scanner outputs or supplied SPDX/CycloneDX blobs.
- Does not author verdicts/policy; supplies evidence and projections to Policy/Concelier/Graph.
- Append-only SBOM versions; mutations happen via new versions, never in-place edits.
- Owns the SBOM lineage ledger for versioned uploads, diffs, and retention pruning.
## 1) Mission and boundaries
- Serve deterministic, tenant-scoped SBOM projections and ledger history.
- Accept scanner-produced SBOMs or operator-supplied SPDX/CycloneDX uploads.
- Own append-only SBOM versioning, lineage edges, retention, and related audit state.
- Do not perform scanning itself and do not author policy verdicts.
## 2) Project layout
- `src/SbomService/StellaOps.SbomService` — REST API + event emitters + orchestrator integration.
- Storage: PostgreSQL tables (proposed)
- `sbom_snapshots` (immutable versions; tenant + artifact + digest + createdAt)
- `sbom_projections` (materialised views keyed by snapshotId, entrypoint/service node flags)
- `sbom_assets` (asset metadata, criticality/owner/env/exposure; append-only history)
- `sbom_catalog` (console catalog surface; indexed by artifact, scope, license, assetTags.*, createdAt for deterministic pagination)
- `sbom_component_neighbors` (component lookup graph edges; indexed by purl+artifact for cursor pagination)
- `sbom_paths` (resolved dependency paths with runtime flags, blast-radius hints)
- `sbom_events` (outbox for event delivery + watermark/backfill tracking)
## 2) Runtime layout
- Host: `src/SbomService/StellaOps.SbomService`
- Persistence: `src/SbomService/__Libraries/StellaOps.SbomService.Persistence`
- Tests: `src/SbomService/__Tests/StellaOps.SbomService.Persistence.Tests`
### 2.1) SBOM + provenance spine (Nov 2026)
### 2.1 Durable state
- Canonical PostgreSQL tables now include:
- `sbom.projections`, `sbom.catalog`, `sbom.component_neighbors`
- `sbom.entrypoints`, `sbom.orchestrator_sources`, `sbom.orchestrator_control`
- `sbom.ledger_versions`, `sbom.ledger_audit`, `sbom.analysis_jobs`, `sbom.watermarks`
- `sbom.lineage_edges`
- `sbom.version_events`, `sbom.asset_events`, `sbom.inventory_events`, `sbom.resolver_candidates`
- `sbom.registry_sources`, `sbom.registry_source_runs`
- Startup migrations are embedded in the persistence assembly and run automatically on host startup.
The service now owns an idempotent spine that converts OCI images into SBOMs and provenance bundles with DSSE and in-toto. The flow is intentionally air-gap ready:
### 2.2 Truthful unsupported surface
- The live durable runtime no longer fabricates dependency-path data from in-memory seeds.
- Until a durable dependency-path backend exists:
- `GET /sbom/paths` returns `501 sbom_paths_unsupported`
- dependency-path portions of `GET /sbom/context` return `501 sbom_paths_unsupported`
- `POST /internal/sbom/inventory/backfill` returns `501 sbom_inventory_backfill_unsupported`
- `POST /internal/sbom/resolver-feed/backfill` returns `501 sbom_resolver_feed_backfill_unsupported`
- **Extract** OCI manifest/layers (hash becomes `contentAddress`).
- **Build SBOM** in CycloneDX 1.7 and/or SPDX 3.0.1; canonicalize JSON before hashing (`sbomHash`).
- **Sign** outputs as DSSE envelopes; predicate uses in-toto Statement with SLSA Provenance v1.
- **Publish** attestations optionally to a transparency backend: `rekor`, `local-merkle`, or `null` (no-op). Local Merkle log keeps proofs for later sync when online.
## 3) Primary APIs
- `POST /sbom/upload` validates and normalizes SPDX 2.3/3.0 or CycloneDX 1.4-1.7 and appends a new ledger version.
- `GET /sbom/versions?artifact=...` returns the durable timeline for an artifact.
- `GET /sbom/ledger/history`, `/point`, `/range`, `/diff`, `/lineage` expose ledger and lineage state.
- `GET /console/sboms` reads the durable catalog view.
- `GET /components/lookup?purl=...` reads the durable component-neighbor view.
- `POST /entrypoints` and `GET /entrypoints` manage tenant-scoped entrypoint overrides.
- `GET /sboms/{snapshotId}/projection` returns the stored Link-Not-Merge projection.
Minimal APIs exposed by SbomService (idempotent by hash):
## 4) Internal APIs
- `GET /internal/sbom/events` returns durable `sbom.version.created` backlog state.
- `POST /internal/sbom/events/backfill` replays stored projections into the durable event backlog.
- `GET /internal/sbom/asset-events` returns durable `sbom.asset.updated` backlog state.
- `GET /internal/jobengine/sources` and `POST /internal/jobengine/sources` manage ingest sources.
- `GET /internal/jobengine/control` and `POST /internal/jobengine/control` manage pause/throttle/backpressure state.
- `GET /internal/jobengine/watermarks` and `POST /internal/jobengine/watermarks` manage durable watermarks.
- `GET /internal/sbom/resolver-feed` and `GET /internal/sbom/resolver-feed/export` expose the persisted resolver candidate store.
- `GET /internal/sbom/ledger/audit` and `GET /internal/sbom/analysis/jobs` expose durable ledger audit and analysis job history.
- `POST /internal/sbom/retention/prune` applies durable retention policy.
- `POST /sbom/ingest` `{ imageDigest, sbom, format, dsseSignature? }``{ sbomId, status: stored|already_present, sbomHash }` keyed by `contentAddress + sbomHash`.
- `POST /attest/verify` `{ dsseEnvelope, expectedSubjects[] }``{ verified, predicateType, logIndex?, inclusionProof? }` and records attestation when verified.
## 5) Ingestion and ledger behavior
- Uploads are normalized into deterministic component lists before persistence.
- Each artifact keeps an append-only chain of `SbomLedgerVersion` records.
- Audit entries are written for ledger mutations.
- Analysis jobs are persisted immediately when upload-triggered work is queued.
- Lineage edges are persisted for parent, base-image, and shared-build relationships.
- Retention pruning removes old versions while preserving auditability.
Operational rules:
## 6) Events and derived state
- Durable event backlogs are stored in:
- `sbom.version_events`
- `sbom.asset_events`
- `sbom.inventory_events`
- `sbom.resolver_candidates`
- `sbom.version.created` and `sbom.asset.updated` are canonical durable outputs today.
- Inventory and resolver feeds remain durable stores, but automatic rebuild of those feeds is blocked on the missing durable dependency-path backend.
- Default media types: `application/vnd.cyclonedx+json`, `application/spdx+json`, `application/dsse+json`, `application/vnd.in-toto+json`.
- If the same SBOM/attestation arrives again, return HTTP 200 with `"status":"already_present"` and do not create a new version.
- Offline posture: no external calls required; Rekor publish remains optional and retryable when connectivity is restored.
## 7) Registry source management
- Registry source definitions and run history are durable PostgreSQL state.
- `RegistrySource` stores URL, filters, trigger mode, auth references, and tenant ownership.
- `RegistrySourceRun` stores discovery/scan counts, trigger metadata, and completion state.
## 3) APIs (first wave)
- `GET /sbom/paths?purl=...&artifact=...&scope=...&env=...` — returns ordered paths with runtime_flag/blast_radius and nearest-safe-version hint; supports `cursor` pagination.
- `GET /sbom/versions?artifact=...` time-ordered SBOM version timeline for Advisory AI; include provenance and source bundle hash.
- `POST /sbom/upload` BYOS upload endpoint; validates/normalizes SPDX 2.3/3.0.1 or CycloneDX 1.41.7 and registers a ledger version.
- `GET /sbom/ledger/history` list version history for an artifact (cursor pagination).
- `GET /sbom/ledger/point` resolve the SBOM version at a specific timestamp.
- `GET /sbom/ledger/range` query versions within a time range.
- `GET /sbom/ledger/diff` component/version/license diff between two versions.
- `GET /sbom/ledger/lineage` parent/child lineage edges for an artifact chain.
- `GET /api/v1/lineage/compare?a=...&b=...&tenant=...` canonical release-investigation comparison endpoint returning normalized component, VEX, and reachability deltas for deploy-diff.
- `POST /api/change-traces/build` compatibility endpoint that materializes a release-investigation change trace from `fromDigest`, `toDigest`, and tenant context.
- `GET /api/change-traces/{traceId}` stateless compatibility read endpoint; rehydrates the change trace from an encoded trace id and the current lineage compare result.
- `GET /console/sboms` Console catalog with filters (artifact, license, scope, asset tags), cursor pagination, evaluation metadata, immutable JSON projection for drawer views.
- `GET /components/lookup?purl=...` component neighborhood for global search/Graph overlays; returns caches hints + tenant enforcement.
- `POST /entrypoints` / `GET /entrypoints` manage entrypoint/service node overrides feeding Cartographer relevance; deterministic defaults when unset.
- `GET /sboms/{snapshotId}/projection` Link-Not-Merge v1 projection returning hashes plus asset metadata (criticality, owner, environment, exposure flags, tags) alongside package/component graph.
- `GET /internal/sbom/events` — internal diagnostics endpoint returning the in-memory event outbox for validation.
- `POST /internal/sbom/events/backfill` — replays existing projections into the event stream; deterministic ordering, clock abstraction for tests.
- `GET /internal/sbom/asset-events` — diagnostics endpoint returning emitted `sbom.asset.updated` envelopes for validation and air-gap parity checks.
- `GET/POST /internal/jobengine/sources` — list/register orchestrator ingest/index sources (deterministic seeds; idempotent on artifactDigest+sourceType).
- `GET/POST /internal/jobengine/control` — manage pause/throttle/backpressure signals per tenant; metrics emitted for control updates.
- `GET/POST /internal/jobengine/watermarks` — fetch/set backfill watermarks for reconciliation and deterministic replays.
- `GET /internal/sbom/resolver-feed` list resolver candidates (artifact, purl, version, paths, scope, runtime_flag, nearest_safe_version).
- `POST /internal/sbom/resolver-feed/backfill` clear and repopulate resolver feed from current projections.
- `GET /internal/sbom/resolver-feed/export` NDJSON export of resolver candidates for air-gap delivery.
- `GET /internal/sbom/ledger/audit` audit trail for ledger changes (created/pruned).
- `GET /internal/sbom/analysis/jobs` list analysis jobs triggered by BYOS uploads.
- `POST /internal/sbom/retention/prune` apply retention policy and emit audit entries.
## 8) Determinism and offline posture
- Stable ordering, UTC timestamps, and canonical hashing are required for all read models and ledger state.
- The service remains offline-friendly: no external calls are required for persistence or replay.
- `Testing` is the only supported environment for fixture-backed or in-memory fallback runtime composition when PostgreSQL is intentionally absent.
- Development, staging, and production runtimes now fail startup if `SbomService:PostgreSQL` is not configured.
## 3.1) Ledger + BYOS workflow (Sprint 4600)
- Uploads are validated, normalized, and stored as ledger versions chained per artifact identity.
- Diffs compare normalized component keys and surface version/license deltas with deterministic ordering.
- Lineage is derived from parent version references and emitted for Graph lineage edges.
- Lineage relationships include parent links plus build links (shared CI build IDs when provided).
- Retention policy prunes old versions while preserving audit entries and minimum keep counts.
- See `docs/modules/sbomservice/ledger-lineage.md` for request/response examples.
- See `docs/modules/sbomservice/byos-ingestion.md` for supported formats and troubleshooting.
## 9) Configuration
- `SbomService:PostgreSQL:ConnectionString` enables the live durable runtime.
- `SbomService:PostgreSQL:Schema` optionally overrides the default schema `sbom`.
- `SbomService:Ledger:*` controls retention:
- `MaxVersionsPerArtifact`
- `MaxAgeDays`
- `MinVersionsToKeep`
- `SbomService:RegistryHttp`, `SbomService:ScannerHttp`, and `SbomService:RegistrySourceQuery` configure registry discovery and scan emission behavior.
## 4) Ingestion & orchestrator integration
- Ingest sources: Scanner pipeline (preferred) or uploaded SPDX 2.3/3.0.1 and CycloneDX 1.41.7 bundles.
- Orchestrator: register SBOM ingest/index jobs; worker SDK emits artifact hash + job metadata; honor pause/throttle; report backpressure metrics; support watermark-based backfill for idempotent replays.
- Idempotency: combine `(tenant, artifactDigest, sbomVersion)` as primary key; duplicate ingests short-circuit.
## 10) Proof status
- Durable host build: `StellaOps.SbomService` builds cleanly with PostgreSQL-backed canonical stores.
- Durable persistence proof: `StellaOps.SbomService.Persistence.Tests` covers ledger, event store, registry-source persistence, and repository-backed timeline rebuild across service instances.
## 5) Events & streaming
- `sbom.version.created` — emitted per new SBOM snapshot; payload: tenant, artifact digest, sbomVersion, projection hash, source bundle hash, import provenance; replay/backfill via outbox with watermark.
- `sbom.asset.updated` — emitted when asset metadata changes; idempotent payload keyed by `(tenant, assetId, version)`.
- Inventory/resolver feeds — queue/topic delivering `(artifact, purl, version, paths, runtime_flag, scope, nearest_safe_version)` for Vuln Explorer/Findings Ledger.
- Current implementation uses an in-memory event store/publisher (with clock abstraction) plus `/internal/sbom/events` + `/internal/sbom/events/backfill` to validate envelopes until the PostgreSQL-backed outbox is wired.
- Entrypoint/service node overrides are exposed via `/entrypoints` (tenant-scoped) and should be mirrored into Cartographer relevance jobs when the outbox lands.
## 11) Open gaps
- Add a durable dependency-path backend so `/sbom/paths`, dependency-path context assembly, and inventory/resolver feed backfill can leave truthful `501` mode.
- Normalize the shared contract surface further if the persistence assembly keeps absorbing host-owned types.
## 5.1) Release Investigation Compatibility
- The Releases workspace consumes lineage compare as the source of truth for A/B deploy comparison.
- `/api/change-traces/*` exists as a compatibility layer for the web change-trace viewer and gateway routing. It does not persist trace documents; trace ids encode the tenant, digest pair, and byte-diff mode, and the service deterministically rebuilds the document on read.
- When no lineage comparison exists for the selected digests, the service returns `404` so the web workspace can surface an explicit recovery state instead of pretending data exists.
## 6) Determinism & offline posture
- Stable ordering for projections and paths; timestamps in UTC ISO-8601; hash inputs canonicalised.
- Add-only evolution for schemas; LNM v1 fixtures published alongside API docs and replayable tests.
- Offline-friendly: uses mirrored packages, avoids external calls during projection; exports NDJSON bundles for air-gapped replay.
## 7) Tenancy & security
- All APIs require tenant context (token claims or mTLS binding); collection filters must include tenant keys.
- Enforce least-privilege queries; avoid cross-tenant caches; log tenant IDs in structured logs.
- Input validation: schema-validate incoming SBOMs; reject oversized/unsupported media types early.
## 8) Observability
- Metrics: `sbom_projection_seconds`, `sbom_projection_size_bytes`, `sbom_projection_queries_total`, `sbom_paths_latency_seconds`, `sbom_paths_cache_hit_ratio`, `sbom_events_backlog`, `sbom_ledger_uploads_total`, `sbom_ledger_diffs_total`, `sbom_ledger_retention_pruned_total`.
- Tracing: ActivitySource `StellaOps.SbomService` (entrypoints, component lookup, console catalog, projections, events).
- Traces: wrap ingest, projection build, and API handlers; propagate orchestrator job IDs.
- Logs: structured, include tenant + artifact digest + sbomVersion; classify ingest failures (schema, storage, orchestrator, validation).
- Alerts: backlog thresholds for outbox/event delivery; high latency on path/timeline endpoints.
## 8.1) Registry Source Management (Sprint 012)
The service manages container registry sources for automated image discovery and scanning:
### Models
- `RegistrySource` — registry connection with URL, filters, schedule, credentials (via AuthRef).
- `RegistrySourceRun` — run history with status, discovered images, triggered scans, error details.
- `RegistrySourceStatus``Draft`, `Active`, `Paused`, `Error`, `Deleted`.
- `RegistrySourceProvider``Generic`, `Harbor`, `DockerHub`, `ACR`, `ECR`, `GCR`, `GHCR`.
### APIs
- `GET/POST/PUT/DELETE /api/v1/registry-sources` — CRUD operations.
- `POST /api/v1/registry-sources/{id}/test` — test registry connection and credentials.
- `POST /api/v1/registry-sources/{id}/trigger` — manually trigger discovery and scanning.
- `POST /api/v1/registry-sources/{id}/pause` / `/resume` — pause/resume scheduled scans.
- `GET /api/v1/registry-sources/{id}/runs` — run history with health metrics.
- `GET /api/v1/registry-sources/{id}/discover/repositories` — discover repositories matching filters.
- `GET /api/v1/registry-sources/{id}/discover/tags/{repository}` — discover tags for a repository.
- `GET /api/v1/registry-sources/{id}/discover/images` — full image discovery.
- `POST /api/v1/registry-sources/{id}/discover-and-scan` — discover and submit scan jobs.
### Webhook Ingestion
- `POST /api/v1/webhooks/registry/{sourceId}` — receive push notifications from registries.
- Supported providers: Harbor, DockerHub, ACR, ECR, GCR, GHCR.
- HMAC-SHA256 signature validation using webhook secret from AuthRef.
- Auto-detection of provider from request headers.
### Discovery Service
- OCI Distribution Spec compliant repository/tag enumeration.
- Pagination via RFC 5988 Link headers.
- Allowlist/denylist filtering for repositories and tags (glob patterns).
- Manifest digest retrieval via HEAD requests.
### Scan Job Emission
- Batch submission to Scanner service with rate limiting.
- Deduplication (skips if job already exists).
- Metadata includes source ID, trigger type, client request ID.
### Configuration
- `SbomService:ScannerUrl` — Scanner service endpoint (default: `http://localhost:5100`).
- `SbomService:BatchScanSize` — max images per batch (default: 10).
- `SbomService:BatchScanDelayMs` — delay between batch submissions (default: 100ms).
### Credentials
- All credentials via AuthRef URIs: `authref://{vault}/{path}#{key}`.
- Supports basic auth (`basic:user:pass`) and bearer tokens (`bearer:token`) for development.
## 9) Configuration (PostgreSQL-backed catalog & lookup)
- Enable PostgreSQL storage for `/console/sboms` and `/components/lookup` by setting `SbomService:PostgreSQL:ConnectionString` (env: `SBOM_SbomService__PostgreSQL__ConnectionString`).
- Optional overrides: `SbomService:PostgreSQL:Schema`, `SbomService:PostgreSQL:CatalogTable`, `SbomService:PostgreSQL:ComponentLookupTable`; defaults are `sbom_service`, `sbom_catalog`, `sbom_component_neighbors`.
- When the connection string is absent the service falls back to fixture JSON or deterministic in-memory seeds to keep air-gapped workflows alive.
- Ledger retention settings (env prefix `SBOM_SbomService__Ledger__`): `MaxVersionsPerArtifact`, `MaxAgeDays`, `MinVersionsToKeep`.
## 10) Open questions / dependencies
- Confirm orchestrator pause/backfill contract (shared with Runtime & Signals 140-series).
- Finalise storage table names and indexes (compound on tenant+artifactDigest+version, TTL for transient staging).
- Publish canonical LNM v1 fixtures and JSON schemas for projections and asset metadata.
- See `docs/modules/sbomservice/api/projection-read.md` for `/sboms/{snapshotId}/projection` (LNM v1, tenant-scoped, hash-returning).
- See `docs/modules/sbomservice/lineage-ledger.md` for ledger endpoints and lineage relationships.
- See `docs/modules/sbomservice/retention-policy.md` for retention configuration and audit expectations.
## 12) Related docs
- `docs/modules/sbomservice/ledger-lineage.md`
- `docs/modules/sbomservice/byos-ingestion.md`
- `docs/modules/sbomservice/retention-policy.md`
- `docs/modules/sbomservice/api/projection-read.md`