stella-ops.org/git.stella-ops.org
1
0
Fork 0
Code Issues Pull Requests Actions 1 Releases Wiki Activity
Files
fac626db8dc0d8143a3f3c92de78bc00f45fc053
git.stella-ops.org/docs/implplan/EPIC_5.md
master d870da18ce Restructure solution layout by module
2025-10-28 15:10:40 +02:00

20 KiB
Raw Blame History

Here’s Epic 5 in the same paste‑into‑repo, implementation‑ready format as the prior epics. It’s exhaustive, formal, and designed to slot into AOC, Policy Engine, Conseiller/Excitator, and the Console.

Epic 5: SBOM Graph Explorer

Short name: Graph Explorer Services touched: SBOM Service, Graph Indexer (new), Graph API (new), Policy Engine, Conseiller (Feedser), Excitator (Vexer), Web API Gateway, Authority (authN/Z), Workers/Scheduler, Telemetry Surfaces: Console (Web UI) graph module, CLI, Exports Deliverables: Interactive graph UI with semantic zoom, saved queries, policy/VEX/advisory overlays, diff views, impact analysis, exports

1) What it is

SBOM Graph Explorer is the interactive, tenant‑scoped view of all supply‑chain relationships the platform knows about, rendered as a navigable graph. It connects:

  • Artifacts (applications, images, libs), Packages/Versions, Files/Paths, Licenses, Advisories (from Conseiller), VEX statements (from Excitator), Provenance (builds, sources), and Policies (overlays of determinations)
  • Edges like depends_on, contains, built_from, declared_in, affected_by, vex_exempts, governs_with
  • Time/version dimension: multiple SBOM snapshots with diffs

It’s built for investigation and review: find where a vulnerable package enters; see which apps are impacted; understand why a finding exists; simulate a policy version and see the delta. The explorer observes AOC enforcement: it never mutates facts; it aggregates and visualizes them. Only the Policy Engine may classify, and classification is displayed as overlays.

2) Why

  • SBOMs are graphs. Tables flatten what matters and hide transitive risk.
  • Engineers, security, and auditors need impact answers quickly: “What pulls in log4j:2.17 and where is it at runtime?”
  • Policy/VEX/advisory interactions are nuanced. A visual overlay makes precedence and outcomes obvious.
  • Review is collaborative; you need saved queries, deep links, exports, and consistent evidence.

3) How it should work (maximum detail)

3.1 Domain model

Nodes (typed, versioned, tenant‑scoped):

  • Artifact: application, service, container image, library, module
  • Package: name + ecosystem (purl), PackageVersion with resolved version
  • File: path within artifact or image layer
  • License: SPDX id
  • Advisory: normalized advisory id (GHSA, CVE, vendor), source = Conseiller
  • VEX: statement with product context, status, justification, source = Excitator
  • SBOM: ingestion unit; includes metadata (tool, sha, build info)
  • PolicyDetermination: materialized view of Policy Engine results (read‑only overlay)
  • Build: provenance, commit, workflow run
  • Source: repo, tag, commit

Edges (directed):

  • declared_in (PackageVersion → SBOM)
  • contains (Artifact → PackageVersion | File)
  • depends_on (PackageVersion → PackageVersion) with scope attr (prod|dev|test|optional)
  • built_from (Artifact → Build), provenance_of (Build → Source)
  • affected_by (PackageVersion → Advisory) with range semantics
  • vex_exempts (Advisory ↔ VEX) scoped by product/component
  • licensed_under (Artifact|PackageVersion → License)
  • governs_with (Artifact|PackageVersion → PolicyDetermination)
  • derived_from (SBOM → SBOM) for superseding snapshots

Identity & versioning

  • Every node has a stable key: {tenant}:{type}:{natural_id} (e.g., purl for packages, digest for images).
  • SBOM snapshots are immutable; edges carry valid_from/valid_to for time travel and diffing.

Imposed rule: Work of this type or tasks of this type on this component must also be applied everywhere else it should be applied.

3.2 User capabilities (end‑to‑end)

  • Search & Navigate: global search (purls, CVEs, repos, licenses), keyboard nav, breadcrumbs, semantic zoom.

  • Lenses: toggle views (Security, License, Provenance, Runtime vs Dev, Policy effect).

  • Overlays:

    • Advisory overlay: show affected nodes/edges with source, severity, ranges.
    • VEX overlay: show suppressions/justifications; collapse exempted paths.
    • Policy overlay: choose a policy version; nodes/edges reflect determinations (severity, status) with explain sampling.

  • Impact analysis: pick a vulnerable node; highlight upstream/downstream dependents, scope filters, shortest/all paths with constraints.

  • Diff view: compare SBOM A vs B; show added/removed nodes/edges, changed versions, changed determinations.

  • Saved queries: visual builder + JSON query; shareable permalinks scoped by tenant and environment.

  • Exports: GraphML, CSV edge list, NDJSON of findings, PNG/SVG snapshot.

  • Evidence details: side panel with raw facts, advisory links, VEX statements, policy explain trace, provenance.

  • Accessibility: tab‑navigable, high‑contrast, screen‑reader labels for nodes and sidebars.

3.3 Query model

  • Visual builder for common queries:

    • “Show all paths from Artifact X to Advisory Y up to depth 6.”
    • “All runtime dependencies with license = GPL‑3.0.”
    • “All artifacts affected by GHSA‑… with no applicable VEX.”
    • “Which SBOMs introduced/removed openssl between build 120 and 130?”

  • JSON query (internal, POST body) with:

    • start: list of node selectors (type + id or attributes)
    • expand: edge types and depth, direction, scope filters
    • where: predicates on node/edge attributes
    • overlay: policy version id, advisory sources, VEX filters
    • limit: nodes, edges, timebox, cost budget

  • Cost control: server estimates cost, denies or pages results; UI streams partial graph tiles.

3.4 UI architecture (Console)

  • Canvas: WebGL renderer with level‑of‑detail, edge bundling, and label culling; deterministic layout when possible (seeded).

  • Semantic zoom:

    • Far: clusters by artifact/repo/ecosystem, color by lens
    • Mid: package groups, advisory badges, license swatches
    • Near: concrete versions, direct edges, inline badges for policy determinations

  • Panels:

    • Left: search, filters, lens selector, saved queries
    • Right: details, explain trace, evidence tabs (Advisory/VEX/Policy/Provenance)
    • Bottom: query expression, diagnostics, performance/stream status

  • Diff mode: split or overlay, color legend (add/remove/changed), filter by node type.

  • Deep links: URL encodes query + viewport; shareable respecting RBAC.

  • Keyboard: space drag, +/- zoom, F to focus, G to expand neighbors, P to show paths.

3.5 Back‑end architecture

Graph Indexer (new)

  • Consumes SBOM ingests, Conseiller advisories, Excitator VEX statements, Policy Engine determinations (read‑only).

  • Projects facts into a property graph persisted in:

    • Primary: document store + adjacency sets (e.g., Mongo collections + compressed adjacency lists)
    • Optional driver for graph DB backends if needed (pluggable)

  • Maintains materialized aggregates: degree, critical paths cache, affected artifact counts, license distribution.

  • Emits graph snapshots per SBOM with lineage to original ingestion.

Graph API (new)

  • Endpoints for search, neighbor expansion, path queries, diffs, overlays, exports.
  • Streaming responses for large graphs (chunked NDJSON tiles).
  • Cost accounting + quotas per tenant.

Workers

  • Centrality & clustering precompute on idle: betweenness approximations, connected components, Louvain clusters.
  • Diff compute on new SBOM ingestion pairs (previous vs current).
  • Overlay materialization cache for popular policy versions.

Policy Engine integration

  • Graph API requests can specify a policy version.
  • For sampled nodes, the API fetches explain traces; for counts, uses precomputed overlay materializations where available.

AOC enforcement

  • Graph Indexer never merges or edits advisories/VEX; it links them and exposes overlays that the Policy Engine evaluates.
  • Conseiller and Excitator remain authoritative sources; severities come from Policy‑governed normalization.

3.6 APIs (representative)

  • GET /graph/search?q=...&type=package|artifact|advisory|license
  • POST /graph/query ⇒ stream tiles {nodes[], edges[], stats, cursor}
  • POST /graph/paths body: {from, to, depth<=6, constraints{scope, runtime_only}}
  • POST /graph/diff body: {sbom_a, sbom_b, filters}
  • GET /graph/snapshots/{sbom_id} ⇒ graph metadata, counts, top advisories
  • POST /graph/export body: {format: graphml|csv|ndjson|png|svg, query|snapshot}
  • GET /graph/saved / POST /graph/saved save and list tenant queries
  • GET /graph/overlays/policy/{version_id} ⇒ summary stats for caching

All endpoints tenant‑scoped, RBAC‑checked. Timeouts and pagination by server. Errors return structured diagnostics.

3.7 CLI

stella sbom graph search "purl:pkg:maven/org.apache.logging.log4j/log4j-core"
stella sbom graph query --file ./query.json --export graphml > graph.graphml
stella sbom graph impacted --advisory GHSA-xxxx --runtime-only --limit 100
stella sbom graph paths --from artifact:service-a --to advisory:GHSA-xxxx --depth 5 --policy 1.3.0
stella sbom graph diff --sbom-a 2025-03-15T10:00Z --sbom-b 2025-03-22T10:00Z --export csv > diff.csv
stella sbom graph save --name "openssl-runtime" --file ./query.json

Exit codes: 0 ok, 2 query validation error, 3 over‑budget, 4 not found, 5 RBAC denied.

Imposed rule: Work of this type or tasks of this type on this component must also be applied everywhere else it should be applied.

3.8 Performance & scale

  • Progressive loading: server pages tiles by BFS frontier; client renders incrementally.
  • Viewport culling: only visible nodes/edges in canvas; offscreen demoted to aggregates.
  • Level‑of‑detail: simplified glyphs and collapsed clusters at distance.
  • Query budgets: per‑tenant rate + node/edge caps; interactive paths limited to depth ≤ 6.
  • Caching: hot queries memoized per tenant + overlay version; diffs precomputed for consecutive SBOMs.

3.9 Security

  • Multi‑tenant isolation at storage and API layers.

  • RBAC roles:

    • Viewer: browse graphs, saved queries
    • Investigator: run queries, export data
    • Operator: configure budgets, purge caches
    • Auditor: download evidence bundles

  • Input validation for query JSON; deny disallowed edge traversals; strict CSP in web app.

3.10 Observability

  • Metrics: tile latency, nodes/edges per tile, cache hit rate, query denials, memory pressure.
  • Logs: structured, include query hash, cost, truncation flags.
  • Traces: server spans per stage (parse, plan, fetch, overlay, stream).

3.11 Accessibility & UX guarantees

  • Keyboard complete, ARIA roles for graph and panels, high‑contrast theme.
  • Deterministic layout on reload for shareable investigations.

3.12 Data retention

  • Graph nodes derived from SBOMs share retention with SBOM artifacts; overlays are ephemeral caches.
  • Saved queries retained until deleted; references to missing objects show warnings.

4) Implementation plan

4.1 Services

  • Graph Indexer (new microservice)

    • Subscribes to SBOM ingest events, Conseiller advisory updates, Excitator VEX updates, Policy overlay materializations.
    • Builds adjacency lists and node documents; computes aggregates and clusters.

  • Graph API (new microservice)

    • Validates and executes queries; streams tiles; composes overlays; serves diffs and exports.
    • Integrates with Policy Engine for explain sample retrieval.

  • SBOM Service (existing)

    • Emits ingestion events with SBOM ids and lineage; exposes SBOM metadata to Graph API.

  • Web API Gateway

    • Routes /graph/*, injects tenant context, enforces RBAC.

4.2 Console (Web UI) feature module

  • packages/features/graph-explorer

    • Canvas renderer (WebGL), panels, query builder, diff mode, overlays, exports.
    • Deep‑link router and viewport state serializer.

4.3 Workers

  • Centrality/clustering worker, diff worker, overlay materialization worker.
  • Schedules on low‑traffic windows; backpressure aware.

4.4 Data model (storage)

  • Collections:

    • graph_nodes: {_id, tenant, type, natural_id, attrs, degree, created_at, updated_at}
    • graph_edges: {_id, tenant, from_id, to_id, type, attrs, valid_from, valid_to}
    • graph_snapshots: per‑SBOM node/edge references
    • graph_saved_queries: {_id, tenant, name, query_json, created_by}
    • graph_overlays_cache: keyed by {tenant, policy_version, hash(query)}

  • Indexes: compound on {tenant, type, natural_id}, {tenant, from_id}, {tenant, to_id}, time bounds.

5) Documentation changes (create/update)

  1. /docs/sbom/graph-explorer-overview.md

    • Concepts, node/edge taxonomy, lenses, overlays, roles, limitations.

  2. /docs/sbom/graph-using-the-console.md

    • Walkthroughs: search, navigate, impact, diff, export; screenshots and keyboard cheatsheet.

  3. /docs/sbom/graph-query-language.md

    • JSON schema, examples, constraints, cost/budget rules.

  4. /docs/sbom/graph-api.md

    • REST endpoints, request/response examples, streaming and pagination.

  5. /docs/sbom/graph-cli.md

    • CLI command reference and example pipelines.

  6. /docs/policy/graph-overlays.md

    • How policy versions render in Graph; explain sampling; AOC guardrails.

  7. /docs/vex/graph-integration.md

    • How VEX suppressions appear and how to validate product scoping.

  8. /docs/advisories/graph-integration.md

    • Advisory linkage and severity normalization by policy.

  9. /docs/architecture/graph-services.md

    • Graph Indexer, Graph API, storage choices, failure modes.

  10. /docs/observability/graph-telemetry.md

    • Metrics, logs, tracing, dashboards.

  11. /docs/runbooks/graph-incidents.md

    • Handling runaway queries, cache poisoning, degraded render.

  12. /docs/security/graph-rbac.md

    • Permissions matrix, multi‑tenant boundaries.

Every doc should end with a “Compliance checklist.” Imposed rule: Work of this type or tasks of this type on this component must also be applied everywhere else it should be applied.

6) Tasks

6.1 Backend: Graph Indexer

  • Define node/edge schemas and attribute dictionaries for each type.
  • Implement event consumers for SBOM ingests, Conseiller updates, Excitator updates.
  • Build ingestion pipeline that populates nodes/edges and maintains valid_from/valid_to.
  • Implement aggregate counters and degree metrics.
  • Implement clustering job and persist cluster ids per node.
  • Implement snapshot materialization per SBOM and lineage tracking.
  • Unit tests for each node/edge builder; property‑based tests for identity stability.

6.2 Backend: Graph API

  • Implement /graph/search with prefix and exact match across node types.
  • Implement /graph/query with validation, planning, cost estimation, and streaming tile results.
  • Implement /graph/paths with constraints and depth limits; shortest path heuristic.
  • Implement /graph/diff computing adds/removes/changed versions; stream results.
  • Implement overlays: advisory join, VEX join, policy materialization and explain sampling.
  • Implement exports: GraphML, CSV edge list, NDJSON findings, PNG/SVG snapshots.
  • RBAC middleware integration; multi‑tenant scoping.
  • Load tests with synthetic large SBOMs; define default budgets.

6.3 Policy Engine integration

  • Add endpoint to fetch explain traces for specific node ids in batch.
  • Add materialization export that Graph API can cache per policy version.

6.4 Console (Web UI)

  • Create graph-explorer module with routes /graph, /graph/diff, /graph/q/:id.
  • Implement WebGL canvas with LOD, culling, edge bundling, deterministic layout seed.
  • Build search, filter, lens, and overlay toolbars.
  • Side panels: details, evidence tabs, explain viewer.
  • Diff mode: split/overlay toggles and color legend.
  • Saved queries: create, update, run; deep links.
  • Export UI: formats, server round‑trip, progress indicators.
  • a11y audit and keyboard‑only flow.

6.5 CLI

  • Implement stella sbom graph * subcommands with JSON IO and piping support.
  • Document examples and stable output schemas for CI consumption.

6.6 Observability & Ops

  • Dashboards for tile latency, query denials, cache hit rate, memory.
  • Alerting on query error spikes, OOM risk, cache churn.
  • Runbooks in /docs/runbooks/graph-incidents.md.

6.7 Docs

  • Author all docs in section 5, link from Console contextual help.
  • Add end‑to‑end tutorial: “Investigate GHSA‑XXXX across prod artifacts.”

Imposed rule: Work of this type or tasks of this type on this component must also be applied everywhere else it should be applied.

7) Acceptance criteria

  • Console renders large SBOM graphs with semantic zoom, overlays, and responsive interactions.
  • Users can run impact and path queries with bounded depth and get results within budget.
  • VEX suppressions and advisory severities appear correctly and are consistent with policy.
  • Diff view clearly shows added/removed/changed nodes/edges between two SBOMs.
  • Saved queries and deep links reproduce the same view deterministically (given same data).
  • Exports produce valid GraphML/CSV/NDJSON and image snapshots.
  • CLI supports search, query, paths, impacted, diff, and export with stable schemas.
  • AOC guardrails: explorer never mutates facts; overlays reflect Policy Engine decisions.
  • RBAC enforced; all actions logged and observable.

8) Risks & mitigations

  • Graph explosion on large monorepos → tiling, clustering, budgets, and strict depth limits.
  • Inconsistent identities across tools → canonicalize purls/digests; property‑based tests for identity stability.
  • Policy overlay latency → precompute materializations for hot policy versions; sample explains only on focus.
  • User confusion → strong lens defaults, deterministic layouts, legends, in‑context help.

9) Test plan

  • Unit: node/edge builders, identity normalization, cost estimator.
  • Integration: ingest SBOM + advisories + VEX, verify overlays and counts.
  • E2E: Playwright flows for search→impact→diff→export; deep link determinism.
  • Performance: simulate 500k nodes/2M edges; measure tile latency and memory.
  • Security: RBAC matrix; tenant isolation tests; query validation fuzzing.
  • Determinism: snapshot round‑trip: same query and seed produce identical layout and stats.

10) Feature flags

  • graph.explorer (UI feature module)
  • graph.paths (advanced path queries)
  • graph.diff (SBOM diff mode)
  • graph.overlays.policy (policy overlay + explain sampling)
  • graph.export (exports enabled)

Documented in /docs/observability/graph-telemetry.md.

11) Non‑goals (this epic)

  • Real‑time process/runtime call graphs.
  • Full substitution for text reports; Explorer complements Reports.
  • Cross‑tenant graphs; all queries are tenant‑scoped.

12) Philosophy

  • See the system: security and license risk are structural. If you cannot see structure, you will miss risk.
  • Evidence over assertion: every colored node corresponds to raw facts and explainable determinations.
  • Bounded interactivity: fast, partial answers beat slow “complete” ones.
  • Immutability: graphs mirror SBOM snapshots and are never rewritten; we add context, not edits.

Final reminder: Work of this type or tasks of this type on this component must also be applied everywhere else it should be applied.