432 lines
20 KiB
Markdown
432 lines
20 KiB
Markdown
Here’s Epic 5 in the same paste‑into‑repo, implementation‑ready format as the prior epics. It’s exhaustive, formal, and designed to slot into AOC, Policy Engine, Conseiller/Excitator, and the Console.
|
||
|
||
---
|
||
|
||
# Epic 5: SBOM Graph Explorer
|
||
|
||
> Short name: **Graph Explorer**
|
||
> Services touched: **SBOM Service**, **Graph Indexer** (new), **Graph API** (new), **Policy Engine**, **Conseiller (Feedser)**, **Excitator (Vexer)**, **Web API Gateway**, **Authority** (authN/Z), **Workers/Scheduler**, **Telemetry**
|
||
> Surfaces: **Console (Web UI)** graph module, **CLI**, **Exports**
|
||
> Deliverables: Interactive graph UI with semantic zoom, saved queries, policy/VEX/advisory overlays, diff views, impact analysis, exports
|
||
|
||
---
|
||
|
||
## 1) What it is
|
||
|
||
**SBOM Graph Explorer** is the interactive, tenant‑scoped view of all supply‑chain relationships the platform knows about, rendered as a navigable graph. It connects:
|
||
|
||
* **Artifacts** (applications, images, libs), **Packages/Versions**, **Files/Paths**, **Licenses**, **Advisories** (from Conseiller), **VEX statements** (from Excitator), **Provenance** (builds, sources), and **Policies** (overlays of determinations)
|
||
* **Edges** like `depends_on`, `contains`, `built_from`, `declared_in`, `affected_by`, `vex_exempts`, `governs_with`
|
||
* **Time/version** dimension: multiple SBOM snapshots with diffs
|
||
|
||
It’s built for investigation and review: find where a vulnerable package enters; see which apps are impacted; understand why a finding exists; simulate a policy version and see the delta. The explorer observes **AOC enforcement**: it never mutates facts; it aggregates and visualizes them. Only the Policy Engine may classify, and classification is displayed as overlays.
|
||
|
||
---
|
||
|
||
## 2) Why
|
||
|
||
* SBOMs are graphs. Tables flatten what matters and hide transitive risk.
|
||
* Engineers, security, and auditors need impact answers quickly: “What pulls in `log4j:2.17` and where is it at runtime?”
|
||
* Policy/VEX/advisory interactions are nuanced. A visual overlay makes precedence and outcomes obvious.
|
||
* Review is collaborative; you need saved queries, deep links, exports, and consistent evidence.
|
||
|
||
---
|
||
|
||
## 3) How it should work (maximum detail)
|
||
|
||
### 3.1 Domain model
|
||
|
||
**Nodes** (typed, versioned, tenant‑scoped):
|
||
|
||
* `Artifact`: application, service, container image, library, module
|
||
* `Package`: name + ecosystem (purl), `PackageVersion` with resolved version
|
||
* `File`: path within artifact or image layer
|
||
* `License`: SPDX id
|
||
* `Advisory`: normalized advisory id (GHSA, CVE, vendor), source = Conseiller
|
||
* `VEX`: statement with product context, status, justification, source = Excitator
|
||
* `SBOM`: ingestion unit; includes metadata (tool, sha, build info)
|
||
* `PolicyDetermination`: materialized view of Policy Engine results (read‑only overlay)
|
||
* `Build`: provenance, commit, workflow run
|
||
* `Source`: repo, tag, commit
|
||
|
||
**Edges** (directed):
|
||
|
||
* `declared_in` (PackageVersion → SBOM)
|
||
* `contains` (Artifact → PackageVersion | File)
|
||
* `depends_on` (PackageVersion → PackageVersion) with scope attr (prod|dev|test|optional)
|
||
* `built_from` (Artifact → Build), `provenance_of` (Build → Source)
|
||
* `affected_by` (PackageVersion → Advisory) with range semantics
|
||
* `vex_exempts` (Advisory ↔ VEX) scoped by product/component
|
||
* `licensed_under` (Artifact|PackageVersion → License)
|
||
* `governs_with` (Artifact|PackageVersion → PolicyDetermination)
|
||
* `derived_from` (SBOM → SBOM) for superseding snapshots
|
||
|
||
**Identity & versioning**
|
||
|
||
* Every node has a stable key: `{tenant}:{type}:{natural_id}` (e.g., purl for packages, digest for images).
|
||
* SBOM snapshots are immutable; edges carry `valid_from`/`valid_to` for time travel and diffing.
|
||
|
||
> **Imposed rule:** Work of this type or tasks of this type on this component must also be applied everywhere else it should be applied.
|
||
|
||
### 3.2 User capabilities (end‑to‑end)
|
||
|
||
* **Search & Navigate**: global search (purls, CVEs, repos, licenses), keyboard nav, breadcrumbs, semantic zoom.
|
||
* **Lenses**: toggle views (Security, License, Provenance, Runtime vs Dev, Policy effect).
|
||
* **Overlays**:
|
||
|
||
* **Advisory overlay**: show affected nodes/edges with source, severity, ranges.
|
||
* **VEX overlay**: show suppressions/justifications; collapse exempted paths.
|
||
* **Policy overlay**: choose a policy version; nodes/edges reflect determinations (severity, status) with explain sampling.
|
||
* **Impact analysis**: pick a vulnerable node; highlight upstream/downstream dependents, scope filters, shortest/all paths with constraints.
|
||
* **Diff view**: compare SBOM A vs B; show added/removed nodes/edges, changed versions, changed determinations.
|
||
* **Saved queries**: visual builder + JSON query; shareable permalinks scoped by tenant and environment.
|
||
* **Exports**: GraphML, CSV edge list, NDJSON of findings, PNG/SVG snapshot.
|
||
* **Evidence details**: side panel with raw facts, advisory links, VEX statements, policy explain trace, provenance.
|
||
* **Accessibility**: tab‑navigable, high‑contrast, screen‑reader labels for nodes and sidebars.
|
||
|
||
### 3.3 Query model
|
||
|
||
* **Visual builder** for common queries:
|
||
|
||
* “Show all paths from Artifact X to Advisory Y up to depth 6.”
|
||
* “All runtime dependencies with license = GPL‑3.0.”
|
||
* “All artifacts affected by GHSA‑… with no applicable VEX.”
|
||
* “Which SBOMs introduced/removed `openssl` between build 120 and 130?”
|
||
* **JSON query** (internal, POST body) with:
|
||
|
||
* `start`: list of node selectors (type + id or attributes)
|
||
* `expand`: edge types and depth, direction, scope filters
|
||
* `where`: predicates on node/edge attributes
|
||
* `overlay`: policy version id, advisory sources, VEX filters
|
||
* `limit`: nodes, edges, timebox, cost budget
|
||
* **Cost control**: server estimates cost, denies or pages results; UI streams partial graph tiles.
|
||
|
||
### 3.4 UI architecture (Console)
|
||
|
||
* **Canvas**: WebGL renderer with level‑of‑detail, edge bundling, and label culling; deterministic layout when possible (seeded).
|
||
* **Semantic zoom**:
|
||
|
||
* Far: clusters by artifact/repo/ecosystem, color by lens
|
||
* Mid: package groups, advisory badges, license swatches
|
||
* Near: concrete versions, direct edges, inline badges for policy determinations
|
||
* **Panels**:
|
||
|
||
* Left: search, filters, lens selector, saved queries
|
||
* Right: details, explain trace, evidence tabs (Advisory/VEX/Policy/Provenance)
|
||
* Bottom: query expression, diagnostics, performance/stream status
|
||
* **Diff mode**: split or overlay, color legend (add/remove/changed), filter by node type.
|
||
* **Deep links**: URL encodes query + viewport; shareable respecting RBAC.
|
||
* **Keyboard**: space drag, +/- zoom, F to focus, G to expand neighbors, P to show paths.
|
||
|
||
### 3.5 Back‑end architecture
|
||
|
||
**Graph Indexer (new)**
|
||
|
||
* Consumes SBOM ingests, Conseiller advisories, Excitator VEX statements, Policy Engine determinations (read‑only).
|
||
* Projects facts into a **property graph** persisted in:
|
||
|
||
* Primary: document store + adjacency sets (e.g., Mongo collections + compressed adjacency lists)
|
||
* Optional driver for graph DB backends if needed (pluggable)
|
||
* Maintains materialized aggregates: degree, critical paths cache, affected artifact counts, license distribution.
|
||
* Emits **graph snapshots** per SBOM with lineage to original ingestion.
|
||
|
||
**Graph API (new)**
|
||
|
||
* Endpoints for search, neighbor expansion, path queries, diffs, overlays, exports.
|
||
* Streaming responses for large graphs (chunked NDJSON tiles).
|
||
* Cost accounting + quotas per tenant.
|
||
|
||
**Workers**
|
||
|
||
* **Centrality & clustering** precompute on idle: betweenness approximations, connected components, Louvain clusters.
|
||
* **Diff compute** on new SBOM ingestion pairs (previous vs current).
|
||
* **Overlay materialization** cache for popular policy versions.
|
||
|
||
**Policy Engine integration**
|
||
|
||
* Graph API requests can specify a policy version.
|
||
* For sampled nodes, the API fetches explain traces; for counts, uses precomputed overlay materializations where available.
|
||
|
||
**AOC enforcement**
|
||
|
||
* Graph Indexer never merges or edits advisories/VEX; it links them and exposes overlays that the Policy Engine evaluates.
|
||
* Conseiller and Excitator remain authoritative sources; severities come from Policy‑governed normalization.
|
||
|
||
### 3.6 APIs (representative)
|
||
|
||
* `GET /graph/search?q=...&type=package|artifact|advisory|license`
|
||
* `POST /graph/query` ⇒ stream tiles `{nodes[], edges[], stats, cursor}`
|
||
* `POST /graph/paths` body: `{from, to, depth<=6, constraints{scope, runtime_only}}`
|
||
* `POST /graph/diff` body: `{sbom_a, sbom_b, filters}`
|
||
* `GET /graph/snapshots/{sbom_id}` ⇒ graph metadata, counts, top advisories
|
||
* `POST /graph/export` body: `{format: graphml|csv|ndjson|png|svg, query|snapshot}`
|
||
* `GET /graph/saved` / `POST /graph/saved` save and list tenant queries
|
||
* `GET /graph/overlays/policy/{version_id}` ⇒ summary stats for caching
|
||
|
||
All endpoints tenant‑scoped, RBAC‑checked. Timeouts and pagination by server. Errors return structured diagnostics.
|
||
|
||
### 3.7 CLI
|
||
|
||
```
|
||
stella sbom graph search "purl:pkg:maven/org.apache.logging.log4j/log4j-core"
|
||
stella sbom graph query --file ./query.json --export graphml > graph.graphml
|
||
stella sbom graph impacted --advisory GHSA-xxxx --runtime-only --limit 100
|
||
stella sbom graph paths --from artifact:service-a --to advisory:GHSA-xxxx --depth 5 --policy 1.3.0
|
||
stella sbom graph diff --sbom-a 2025-03-15T10:00Z --sbom-b 2025-03-22T10:00Z --export csv > diff.csv
|
||
stella sbom graph save --name "openssl-runtime" --file ./query.json
|
||
```
|
||
|
||
Exit codes: 0 ok, 2 query validation error, 3 over‑budget, 4 not found, 5 RBAC denied.
|
||
|
||
> **Imposed rule:** Work of this type or tasks of this type on this component must also be applied everywhere else it should be applied.
|
||
|
||
### 3.8 Performance & scale
|
||
|
||
* **Progressive loading**: server pages tiles by BFS frontier; client renders incrementally.
|
||
* **Viewport culling**: only visible nodes/edges in canvas; offscreen demoted to aggregates.
|
||
* **Level‑of‑detail**: simplified glyphs and collapsed clusters at distance.
|
||
* **Query budgets**: per‑tenant rate + node/edge caps; interactive paths limited to depth ≤ 6.
|
||
* **Caching**: hot queries memoized per tenant + overlay version; diffs precomputed for consecutive SBOMs.
|
||
|
||
### 3.9 Security
|
||
|
||
* Multi‑tenant isolation at storage and API layers.
|
||
* RBAC roles:
|
||
|
||
* **Viewer**: browse graphs, saved queries
|
||
* **Investigator**: run queries, export data
|
||
* **Operator**: configure budgets, purge caches
|
||
* **Auditor**: download evidence bundles
|
||
* Input validation for query JSON; deny disallowed edge traversals; strict CSP in web app.
|
||
|
||
### 3.10 Observability
|
||
|
||
* Metrics: tile latency, nodes/edges per tile, cache hit rate, query denials, memory pressure.
|
||
* Logs: structured, include query hash, cost, truncation flags.
|
||
* Traces: server spans per stage (parse, plan, fetch, overlay, stream).
|
||
|
||
### 3.11 Accessibility & UX guarantees
|
||
|
||
* Keyboard complete, ARIA roles for graph and panels, high‑contrast theme.
|
||
* Deterministic layout on reload for shareable investigations.
|
||
|
||
### 3.12 Data retention
|
||
|
||
* Graph nodes derived from SBOMs share retention with SBOM artifacts; overlays are ephemeral caches.
|
||
* Saved queries retained until deleted; references to missing objects show warnings.
|
||
|
||
---
|
||
|
||
## 4) Implementation plan
|
||
|
||
### 4.1 Services
|
||
|
||
* **Graph Indexer (new microservice)**
|
||
|
||
* Subscribes to SBOM ingest events, Conseiller advisory updates, Excitator VEX updates, Policy overlay materializations.
|
||
* Builds adjacency lists and node documents; computes aggregates and clusters.
|
||
|
||
* **Graph API (new microservice)**
|
||
|
||
* Validates and executes queries; streams tiles; composes overlays; serves diffs and exports.
|
||
* Integrates with Policy Engine for explain sample retrieval.
|
||
|
||
* **SBOM Service (existing)**
|
||
|
||
* Emits ingestion events with SBOM ids and lineage; exposes SBOM metadata to Graph API.
|
||
|
||
* **Web API Gateway**
|
||
|
||
* Routes `/graph/*`, injects tenant context, enforces RBAC.
|
||
|
||
### 4.2 Console (Web UI) feature module
|
||
|
||
* `packages/features/graph-explorer`
|
||
|
||
* Canvas renderer (WebGL), panels, query builder, diff mode, overlays, exports.
|
||
* Deep‑link router and viewport state serializer.
|
||
|
||
### 4.3 Workers
|
||
|
||
* Centrality/clustering worker, diff worker, overlay materialization worker.
|
||
* Schedules on low‑traffic windows; backpressure aware.
|
||
|
||
### 4.4 Data model (storage)
|
||
|
||
* Collections:
|
||
|
||
* `graph_nodes`: `{_id, tenant, type, natural_id, attrs, degree, created_at, updated_at}`
|
||
* `graph_edges`: `{_id, tenant, from_id, to_id, type, attrs, valid_from, valid_to}`
|
||
* `graph_snapshots`: per‑SBOM node/edge references
|
||
* `graph_saved_queries`: `{_id, tenant, name, query_json, created_by}`
|
||
* `graph_overlays_cache`: keyed by `{tenant, policy_version, hash(query)}`
|
||
* Indexes: compound on `{tenant, type, natural_id}`, `{tenant, from_id}`, `{tenant, to_id}`, time bounds.
|
||
|
||
---
|
||
|
||
## 5) Documentation changes (create/update)
|
||
|
||
1. **`/docs/sbom/graph-explorer-overview.md`**
|
||
|
||
* Concepts, node/edge taxonomy, lenses, overlays, roles, limitations.
|
||
2. **`/docs/sbom/graph-using-the-console.md`**
|
||
|
||
* Walkthroughs: search, navigate, impact, diff, export; screenshots and keyboard cheatsheet.
|
||
3. **`/docs/sbom/graph-query-language.md`**
|
||
|
||
* JSON schema, examples, constraints, cost/budget rules.
|
||
4. **`/docs/sbom/graph-api.md`**
|
||
|
||
* REST endpoints, request/response examples, streaming and pagination.
|
||
5. **`/docs/sbom/graph-cli.md`**
|
||
|
||
* CLI command reference and example pipelines.
|
||
6. **`/docs/policy/graph-overlays.md`**
|
||
|
||
* How policy versions render in Graph; explain sampling; AOC guardrails.
|
||
7. **`/docs/vex/graph-integration.md`**
|
||
|
||
* How VEX suppressions appear and how to validate product scoping.
|
||
8. **`/docs/advisories/graph-integration.md`**
|
||
|
||
* Advisory linkage and severity normalization by policy.
|
||
9. **`/docs/architecture/graph-services.md`**
|
||
|
||
* Graph Indexer, Graph API, storage choices, failure modes.
|
||
10. **`/docs/observability/graph-telemetry.md`**
|
||
|
||
* Metrics, logs, tracing, dashboards.
|
||
11. **`/docs/runbooks/graph-incidents.md`**
|
||
|
||
* Handling runaway queries, cache poisoning, degraded render.
|
||
12. **`/docs/security/graph-rbac.md`**
|
||
|
||
* Permissions matrix, multi‑tenant boundaries.
|
||
|
||
Every doc should end with a “Compliance checklist.”
|
||
**Imposed rule:** Work of this type or tasks of this type on this component must also be applied everywhere else it should be applied.
|
||
|
||
---
|
||
|
||
## 6) Tasks
|
||
|
||
### 6.1 Backend: Graph Indexer
|
||
|
||
* [ ] Define node/edge schemas and attribute dictionaries for each type.
|
||
* [ ] Implement event consumers for SBOM ingests, Conseiller updates, Excitator updates.
|
||
* [ ] Build ingestion pipeline that populates nodes/edges and maintains `valid_from/valid_to`.
|
||
* [ ] Implement aggregate counters and degree metrics.
|
||
* [ ] Implement clustering job and persist cluster ids per node.
|
||
* [ ] Implement snapshot materialization per SBOM and lineage tracking.
|
||
* [ ] Unit tests for each node/edge builder; property‑based tests for identity stability.
|
||
|
||
### 6.2 Backend: Graph API
|
||
|
||
* [ ] Implement `/graph/search` with prefix and exact match across node types.
|
||
* [ ] Implement `/graph/query` with validation, planning, cost estimation, and streaming tile results.
|
||
* [ ] Implement `/graph/paths` with constraints and depth limits; shortest path heuristic.
|
||
* [ ] Implement `/graph/diff` computing adds/removes/changed versions; stream results.
|
||
* [ ] Implement overlays: advisory join, VEX join, policy materialization and explain sampling.
|
||
* [ ] Implement exports: GraphML, CSV edge list, NDJSON findings, PNG/SVG snapshots.
|
||
* [ ] RBAC middleware integration; multi‑tenant scoping.
|
||
* [ ] Load tests with synthetic large SBOMs; define default budgets.
|
||
|
||
### 6.3 Policy Engine integration
|
||
|
||
* [ ] Add endpoint to fetch explain traces for specific node ids in batch.
|
||
* [ ] Add materialization export that Graph API can cache per policy version.
|
||
|
||
### 6.4 Console (Web UI)
|
||
|
||
* [ ] Create `graph-explorer` module with routes `/graph`, `/graph/diff`, `/graph/q/:id`.
|
||
* [ ] Implement WebGL canvas with LOD, culling, edge bundling, deterministic layout seed.
|
||
* [ ] Build search, filter, lens, and overlay toolbars.
|
||
* [ ] Side panels: details, evidence tabs, explain viewer.
|
||
* [ ] Diff mode: split/overlay toggles and color legend.
|
||
* [ ] Saved queries: create, update, run; deep links.
|
||
* [ ] Export UI: formats, server round‑trip, progress indicators.
|
||
* [ ] a11y audit and keyboard‑only flow.
|
||
|
||
### 6.5 CLI
|
||
|
||
* [ ] Implement `stella sbom graph *` subcommands with JSON IO and piping support.
|
||
* [ ] Document examples and stable output schemas for CI consumption.
|
||
|
||
### 6.6 Observability & Ops
|
||
|
||
* [ ] Dashboards for tile latency, query denials, cache hit rate, memory.
|
||
* [ ] Alerting on query error spikes, OOM risk, cache churn.
|
||
* [ ] Runbooks in `/docs/runbooks/graph-incidents.md`.
|
||
|
||
### 6.7 Docs
|
||
|
||
* [ ] Author all docs in section 5, link from Console contextual help.
|
||
* [ ] Add end‑to‑end tutorial: “Investigate GHSA‑XXXX across prod artifacts.”
|
||
|
||
> **Imposed rule:** Work of this type or tasks of this type on this component must also be applied everywhere else it should be applied.
|
||
|
||
---
|
||
|
||
## 7) Acceptance criteria
|
||
|
||
* Console renders large SBOM graphs with semantic zoom, overlays, and responsive interactions.
|
||
* Users can run impact and path queries with bounded depth and get results within budget.
|
||
* VEX suppressions and advisory severities appear correctly and are consistent with policy.
|
||
* Diff view clearly shows added/removed/changed nodes/edges between two SBOMs.
|
||
* Saved queries and deep links reproduce the same view deterministically (given same data).
|
||
* Exports produce valid GraphML/CSV/NDJSON and image snapshots.
|
||
* CLI supports search, query, paths, impacted, diff, and export with stable schemas.
|
||
* AOC guardrails: explorer never mutates facts; overlays reflect Policy Engine decisions.
|
||
* RBAC enforced; all actions logged and observable.
|
||
|
||
---
|
||
|
||
## 8) Risks & mitigations
|
||
|
||
* **Graph explosion on large monorepos** → tiling, clustering, budgets, and strict depth limits.
|
||
* **Inconsistent identities across tools** → canonicalize purls/digests; property‑based tests for identity stability.
|
||
* **Policy overlay latency** → precompute materializations for hot policy versions; sample explains only on focus.
|
||
* **User confusion** → strong lens defaults, deterministic layouts, legends, in‑context help.
|
||
|
||
---
|
||
|
||
## 9) Test plan
|
||
|
||
* **Unit**: node/edge builders, identity normalization, cost estimator.
|
||
* **Integration**: ingest SBOM + advisories + VEX, verify overlays and counts.
|
||
* **E2E**: Playwright flows for search→impact→diff→export; deep link determinism.
|
||
* **Performance**: simulate 500k nodes/2M edges; measure tile latency and memory.
|
||
* **Security**: RBAC matrix; tenant isolation tests; query validation fuzzing.
|
||
* **Determinism**: snapshot round‑trip: same query and seed produce identical layout and stats.
|
||
|
||
---
|
||
|
||
## 10) Feature flags
|
||
|
||
* `graph.explorer` (UI feature module)
|
||
* `graph.paths` (advanced path queries)
|
||
* `graph.diff` (SBOM diff mode)
|
||
* `graph.overlays.policy` (policy overlay + explain sampling)
|
||
* `graph.export` (exports enabled)
|
||
|
||
Documented in `/docs/observability/graph-telemetry.md`.
|
||
|
||
---
|
||
|
||
## 11) Non‑goals (this epic)
|
||
|
||
* Real‑time process/runtime call graphs.
|
||
* Full substitution for text reports; Explorer complements Reports.
|
||
* Cross‑tenant graphs; all queries are tenant‑scoped.
|
||
|
||
---
|
||
|
||
## 12) Philosophy
|
||
|
||
* **See the system**: security and license risk are structural. If you cannot see structure, you will miss risk.
|
||
* **Evidence over assertion**: every colored node corresponds to raw facts and explainable determinations.
|
||
* **Bounded interactivity**: fast, partial answers beat slow “complete” ones.
|
||
* **Immutability**: graphs mirror SBOM snapshots and are never rewritten; we add context, not edits.
|
||
|
||
> Final reminder: **Work of this type or tasks of this type on this component must also be applied everywhere else it should be applied.**
|