Restructure solution layout by module

docs/implplan/EPIC_5.md

@@ -0,0 +1,431 @@
+Here’s Epic 5 in the same paste‑into‑repo, implementation‑ready format as the prior epics. It’s exhaustive, formal, and designed to slot into AOC, Policy Engine, Conseiller/Excitator, and the Console.
+
+---
+
+# Epic 5: SBOM Graph Explorer
+
+> Short name: **Graph Explorer**
+> Services touched: **SBOM Service**, **Graph Indexer** (new), **Graph API** (new), **Policy Engine**, **Conseiller (Feedser)**, **Excitator (Vexer)**, **Web API Gateway**, **Authority** (authN/Z), **Workers/Scheduler**, **Telemetry**
+> Surfaces: **Console (Web UI)** graph module, **CLI**, **Exports**
+> Deliverables: Interactive graph UI with semantic zoom, saved queries, policy/VEX/advisory overlays, diff views, impact analysis, exports
+
+---
+
+## 1) What it is
+
+**SBOM Graph Explorer** is the interactive, tenant‑scoped view of all supply‑chain relationships the platform knows about, rendered as a navigable graph. It connects:
+
+* **Artifacts** (applications, images, libs), **Packages/Versions**, **Files/Paths**, **Licenses**, **Advisories** (from Conseiller), **VEX statements** (from Excitator), **Provenance** (builds, sources), and **Policies** (overlays of determinations)
+* **Edges** like `depends_on`, `contains`, `built_from`, `declared_in`, `affected_by`, `vex_exempts`, `governs_with`
+* **Time/version** dimension: multiple SBOM snapshots with diffs
+
+It’s built for investigation and review: find where a vulnerable package enters; see which apps are impacted; understand why a finding exists; simulate a policy version and see the delta. The explorer observes **AOC enforcement**: it never mutates facts; it aggregates and visualizes them. Only the Policy Engine may classify, and classification is displayed as overlays.
+
+---
+
+## 2) Why
+
+* SBOMs are graphs. Tables flatten what matters and hide transitive risk.
+* Engineers, security, and auditors need impact answers quickly: “What pulls in `log4j:2.17` and where is it at runtime?”
+* Policy/VEX/advisory interactions are nuanced. A visual overlay makes precedence and outcomes obvious.
+* Review is collaborative; you need saved queries, deep links, exports, and consistent evidence.
+
+---
+
+## 3) How it should work (maximum detail)
+
+### 3.1 Domain model
+
+**Nodes** (typed, versioned, tenant‑scoped):
+
+* `Artifact`: application, service, container image, library, module
+* `Package`: name + ecosystem (purl), `PackageVersion` with resolved version
+* `File`: path within artifact or image layer
+* `License`: SPDX id
+* `Advisory`: normalized advisory id (GHSA, CVE, vendor), source = Conseiller
+* `VEX`: statement with product context, status, justification, source = Excitator
+* `SBOM`: ingestion unit; includes metadata (tool, sha, build info)
+* `PolicyDetermination`: materialized view of Policy Engine results (read‑only overlay)
+* `Build`: provenance, commit, workflow run
+* `Source`: repo, tag, commit
+
+**Edges** (directed):
+
+* `declared_in` (PackageVersion → SBOM)
+* `contains` (Artifact → PackageVersion | File)
+* `depends_on` (PackageVersion → PackageVersion) with scope attr (prod|dev|test|optional)
+* `built_from` (Artifact → Build), `provenance_of` (Build → Source)
+* `affected_by` (PackageVersion → Advisory) with range semantics
+* `vex_exempts` (Advisory ↔ VEX) scoped by product/component
+* `licensed_under` (Artifact|PackageVersion → License)
+* `governs_with` (Artifact|PackageVersion → PolicyDetermination)
+* `derived_from` (SBOM → SBOM) for superseding snapshots
+
+**Identity & versioning**
+
+* Every node has a stable key: `{tenant}:{type}:{natural_id}` (e.g., purl for packages, digest for images).
+* SBOM snapshots are immutable; edges carry `valid_from`/`valid_to` for time travel and diffing.
+
+> **Imposed rule:** Work of this type or tasks of this type on this component must also be applied everywhere else it should be applied.
+
+### 3.2 User capabilities (end‑to‑end)
+
+* **Search & Navigate**: global search (purls, CVEs, repos, licenses), keyboard nav, breadcrumbs, semantic zoom.
+* **Lenses**: toggle views (Security, License, Provenance, Runtime vs Dev, Policy effect).
+* **Overlays**:
+
+  * **Advisory overlay**: show affected nodes/edges with source, severity, ranges.
+  * **VEX overlay**: show suppressions/justifications; collapse exempted paths.
+  * **Policy overlay**: choose a policy version; nodes/edges reflect determinations (severity, status) with explain sampling.
+* **Impact analysis**: pick a vulnerable node; highlight upstream/downstream dependents, scope filters, shortest/all paths with constraints.
+* **Diff view**: compare SBOM A vs B; show added/removed nodes/edges, changed versions, changed determinations.
+* **Saved queries**: visual builder + JSON query; shareable permalinks scoped by tenant and environment.
+* **Exports**: GraphML, CSV edge list, NDJSON of findings, PNG/SVG snapshot.
+* **Evidence details**: side panel with raw facts, advisory links, VEX statements, policy explain trace, provenance.
+* **Accessibility**: tab‑navigable, high‑contrast, screen‑reader labels for nodes and sidebars.
+
+### 3.3 Query model
+
+* **Visual builder** for common queries:
+
+  * “Show all paths from Artifact X to Advisory Y up to depth 6.”
+  * “All runtime dependencies with license = GPL‑3.0.”
+  * “All artifacts affected by GHSA‑… with no applicable VEX.”
+  * “Which SBOMs introduced/removed `openssl` between build 120 and 130?”
+* **JSON query** (internal, POST body) with:
+
+  * `start`: list of node selectors (type + id or attributes)
+  * `expand`: edge types and depth, direction, scope filters
+  * `where`: predicates on node/edge attributes
+  * `overlay`: policy version id, advisory sources, VEX filters
+  * `limit`: nodes, edges, timebox, cost budget
+* **Cost control**: server estimates cost, denies or pages results; UI streams partial graph tiles.
+
+### 3.4 UI architecture (Console)
+
+* **Canvas**: WebGL renderer with level‑of‑detail, edge bundling, and label culling; deterministic layout when possible (seeded).
+* **Semantic zoom**:
+
+  * Far: clusters by artifact/repo/ecosystem, color by lens
+  * Mid: package groups, advisory badges, license swatches
+  * Near: concrete versions, direct edges, inline badges for policy determinations
+* **Panels**:
+
+  * Left: search, filters, lens selector, saved queries
+  * Right: details, explain trace, evidence tabs (Advisory/VEX/Policy/Provenance)
+  * Bottom: query expression, diagnostics, performance/stream status
+* **Diff mode**: split or overlay, color legend (add/remove/changed), filter by node type.
+* **Deep links**: URL encodes query + viewport; shareable respecting RBAC.
+* **Keyboard**: space drag, +/- zoom, F to focus, G to expand neighbors, P to show paths.
+
+### 3.5 Back‑end architecture
+
+**Graph Indexer (new)**
+
+* Consumes SBOM ingests, Conseiller advisories, Excitator VEX statements, Policy Engine determinations (read‑only).
+* Projects facts into a **property graph** persisted in:
+
+  * Primary: document store + adjacency sets (e.g., Mongo collections + compressed adjacency lists)
+  * Optional driver for graph DB backends if needed (pluggable)
+* Maintains materialized aggregates: degree, critical paths cache, affected artifact counts, license distribution.
+* Emits **graph snapshots** per SBOM with lineage to original ingestion.
+
+**Graph API (new)**
+
+* Endpoints for search, neighbor expansion, path queries, diffs, overlays, exports.
+* Streaming responses for large graphs (chunked NDJSON tiles).
+* Cost accounting + quotas per tenant.
+
+**Workers**
+
+* **Centrality & clustering** precompute on idle: betweenness approximations, connected components, Louvain clusters.
+* **Diff compute** on new SBOM ingestion pairs (previous vs current).
+* **Overlay materialization** cache for popular policy versions.
+
+**Policy Engine integration**
+
+* Graph API requests can specify a policy version.
+* For sampled nodes, the API fetches explain traces; for counts, uses precomputed overlay materializations where available.
+
+**AOC enforcement**
+
+* Graph Indexer never merges or edits advisories/VEX; it links them and exposes overlays that the Policy Engine evaluates.
+* Conseiller and Excitator remain authoritative sources; severities come from Policy‑governed normalization.
+
+### 3.6 APIs (representative)
+
+* `GET /graph/search?q=...&type=package|artifact|advisory|license`
+* `POST /graph/query` ⇒ stream tiles `{nodes[], edges[], stats, cursor}`
+* `POST /graph/paths` body: `{from, to, depth<=6, constraints{scope, runtime_only}}`
+* `POST /graph/diff` body: `{sbom_a, sbom_b, filters}`
+* `GET /graph/snapshots/{sbom_id}` ⇒ graph metadata, counts, top advisories
+* `POST /graph/export` body: `{format: graphml|csv|ndjson|png|svg, query|snapshot}`
+* `GET /graph/saved` / `POST /graph/saved` save and list tenant queries
+* `GET /graph/overlays/policy/{version_id}` ⇒ summary stats for caching
+
+All endpoints tenant‑scoped, RBAC‑checked. Timeouts and pagination by server. Errors return structured diagnostics.
+
+### 3.7 CLI
+
+```
+stella sbom graph search "purl:pkg:maven/org.apache.logging.log4j/log4j-core"
+stella sbom graph query --file ./query.json --export graphml > graph.graphml
+stella sbom graph impacted --advisory GHSA-xxxx --runtime-only --limit 100
+stella sbom graph paths --from artifact:service-a --to advisory:GHSA-xxxx --depth 5 --policy 1.3.0
+stella sbom graph diff --sbom-a 2025-03-15T10:00Z --sbom-b 2025-03-22T10:00Z --export csv > diff.csv
+stella sbom graph save --name "openssl-runtime" --file ./query.json
+```
+
+Exit codes: 0 ok, 2 query validation error, 3 over‑budget, 4 not found, 5 RBAC denied.
+
+> **Imposed rule:** Work of this type or tasks of this type on this component must also be applied everywhere else it should be applied.
+
+### 3.8 Performance & scale
+
+* **Progressive loading**: server pages tiles by BFS frontier; client renders incrementally.
+* **Viewport culling**: only visible nodes/edges in canvas; offscreen demoted to aggregates.
+* **Level‑of‑detail**: simplified glyphs and collapsed clusters at distance.
+* **Query budgets**: per‑tenant rate + node/edge caps; interactive paths limited to depth ≤ 6.
+* **Caching**: hot queries memoized per tenant + overlay version; diffs precomputed for consecutive SBOMs.
+
+### 3.9 Security
+
+* Multi‑tenant isolation at storage and API layers.
+* RBAC roles:
+
+  * **Viewer**: browse graphs, saved queries
+  * **Investigator**: run queries, export data
+  * **Operator**: configure budgets, purge caches
+  * **Auditor**: download evidence bundles
+* Input validation for query JSON; deny disallowed edge traversals; strict CSP in web app.
+
+### 3.10 Observability
+
+* Metrics: tile latency, nodes/edges per tile, cache hit rate, query denials, memory pressure.
+* Logs: structured, include query hash, cost, truncation flags.
+* Traces: server spans per stage (parse, plan, fetch, overlay, stream).
+
+### 3.11 Accessibility & UX guarantees
+
+* Keyboard complete, ARIA roles for graph and panels, high‑contrast theme.
+* Deterministic layout on reload for shareable investigations.
+
+### 3.12 Data retention
+
+* Graph nodes derived from SBOMs share retention with SBOM artifacts; overlays are ephemeral caches.
+* Saved queries retained until deleted; references to missing objects show warnings.
+
+---
+
+## 4) Implementation plan
+
+### 4.1 Services
+
+* **Graph Indexer (new microservice)**
+
+  * Subscribes to SBOM ingest events, Conseiller advisory updates, Excitator VEX updates, Policy overlay materializations.
+  * Builds adjacency lists and node documents; computes aggregates and clusters.
+
+* **Graph API (new microservice)**
+
+  * Validates and executes queries; streams tiles; composes overlays; serves diffs and exports.
+  * Integrates with Policy Engine for explain sample retrieval.
+
+* **SBOM Service (existing)**
+
+  * Emits ingestion events with SBOM ids and lineage; exposes SBOM metadata to Graph API.
+
+* **Web API Gateway**
+
+  * Routes `/graph/*`, injects tenant context, enforces RBAC.
+
+### 4.2 Console (Web UI) feature module
+
+* `packages/features/graph-explorer`
+
+  * Canvas renderer (WebGL), panels, query builder, diff mode, overlays, exports.
+  * Deep‑link router and viewport state serializer.
+
+### 4.3 Workers
+
+* Centrality/clustering worker, diff worker, overlay materialization worker.
+* Schedules on low‑traffic windows; backpressure aware.
+
+### 4.4 Data model (storage)
+
+* Collections:
+
+  * `graph_nodes`: `{_id, tenant, type, natural_id, attrs, degree, created_at, updated_at}`
+  * `graph_edges`: `{_id, tenant, from_id, to_id, type, attrs, valid_from, valid_to}`
+  * `graph_snapshots`: per‑SBOM node/edge references
+  * `graph_saved_queries`: `{_id, tenant, name, query_json, created_by}`
+  * `graph_overlays_cache`: keyed by `{tenant, policy_version, hash(query)}`
+* Indexes: compound on `{tenant, type, natural_id}`, `{tenant, from_id}`, `{tenant, to_id}`, time bounds.
+
+---
+
+## 5) Documentation changes (create/update)
+
+1. **`/docs/sbom/graph-explorer-overview.md`**
+
+   * Concepts, node/edge taxonomy, lenses, overlays, roles, limitations.
+2. **`/docs/sbom/graph-using-the-console.md`**
+
+   * Walkthroughs: search, navigate, impact, diff, export; screenshots and keyboard cheatsheet.
+3. **`/docs/sbom/graph-query-language.md`**
+
+   * JSON schema, examples, constraints, cost/budget rules.
+4. **`/docs/sbom/graph-api.md`**
+
+   * REST endpoints, request/response examples, streaming and pagination.
+5. **`/docs/sbom/graph-cli.md`**
+
+   * CLI command reference and example pipelines.
+6. **`/docs/policy/graph-overlays.md`**
+
+   * How policy versions render in Graph; explain sampling; AOC guardrails.
+7. **`/docs/vex/graph-integration.md`**
+
+   * How VEX suppressions appear and how to validate product scoping.
+8. **`/docs/advisories/graph-integration.md`**
+
+   * Advisory linkage and severity normalization by policy.
+9. **`/docs/architecture/graph-services.md`**
+
+   * Graph Indexer, Graph API, storage choices, failure modes.
+10. **`/docs/observability/graph-telemetry.md`**
+
+    * Metrics, logs, tracing, dashboards.
+11. **`/docs/runbooks/graph-incidents.md`**
+
+    * Handling runaway queries, cache poisoning, degraded render.
+12. **`/docs/security/graph-rbac.md`**
+
+    * Permissions matrix, multi‑tenant boundaries.
+
+Every doc should end with a “Compliance checklist.”
+**Imposed rule:** Work of this type or tasks of this type on this component must also be applied everywhere else it should be applied.
+
+---
+
+## 6) Tasks
+
+### 6.1 Backend: Graph Indexer
+
+* [ ] Define node/edge schemas and attribute dictionaries for each type.
+* [ ] Implement event consumers for SBOM ingests, Conseiller updates, Excitator updates.
+* [ ] Build ingestion pipeline that populates nodes/edges and maintains `valid_from/valid_to`.
+* [ ] Implement aggregate counters and degree metrics.
+* [ ] Implement clustering job and persist cluster ids per node.
+* [ ] Implement snapshot materialization per SBOM and lineage tracking.
+* [ ] Unit tests for each node/edge builder; property‑based tests for identity stability.
+
+### 6.2 Backend: Graph API
+
+* [ ] Implement `/graph/search` with prefix and exact match across node types.
+* [ ] Implement `/graph/query` with validation, planning, cost estimation, and streaming tile results.
+* [ ] Implement `/graph/paths` with constraints and depth limits; shortest path heuristic.
+* [ ] Implement `/graph/diff` computing adds/removes/changed versions; stream results.
+* [ ] Implement overlays: advisory join, VEX join, policy materialization and explain sampling.
+* [ ] Implement exports: GraphML, CSV edge list, NDJSON findings, PNG/SVG snapshots.
+* [ ] RBAC middleware integration; multi‑tenant scoping.
+* [ ] Load tests with synthetic large SBOMs; define default budgets.
+
+### 6.3 Policy Engine integration
+
+* [ ] Add endpoint to fetch explain traces for specific node ids in batch.
+* [ ] Add materialization export that Graph API can cache per policy version.
+
+### 6.4 Console (Web UI)
+
+* [ ] Create `graph-explorer` module with routes `/graph`, `/graph/diff`, `/graph/q/:id`.
+* [ ] Implement WebGL canvas with LOD, culling, edge bundling, deterministic layout seed.
+* [ ] Build search, filter, lens, and overlay toolbars.
+* [ ] Side panels: details, evidence tabs, explain viewer.
+* [ ] Diff mode: split/overlay toggles and color legend.
+* [ ] Saved queries: create, update, run; deep links.
+* [ ] Export UI: formats, server round‑trip, progress indicators.
+* [ ] a11y audit and keyboard‑only flow.
+
+### 6.5 CLI
+
+* [ ] Implement `stella sbom graph *` subcommands with JSON IO and piping support.
+* [ ] Document examples and stable output schemas for CI consumption.
+
+### 6.6 Observability & Ops
+
+* [ ] Dashboards for tile latency, query denials, cache hit rate, memory.
+* [ ] Alerting on query error spikes, OOM risk, cache churn.
+* [ ] Runbooks in `/docs/runbooks/graph-incidents.md`.
+
+### 6.7 Docs
+
+* [ ] Author all docs in section 5, link from Console contextual help.
+* [ ] Add end‑to‑end tutorial: “Investigate GHSA‑XXXX across prod artifacts.”
+
+> **Imposed rule:** Work of this type or tasks of this type on this component must also be applied everywhere else it should be applied.
+
+---
+
+## 7) Acceptance criteria
+
+* Console renders large SBOM graphs with semantic zoom, overlays, and responsive interactions.
+* Users can run impact and path queries with bounded depth and get results within budget.
+* VEX suppressions and advisory severities appear correctly and are consistent with policy.
+* Diff view clearly shows added/removed/changed nodes/edges between two SBOMs.
+* Saved queries and deep links reproduce the same view deterministically (given same data).
+* Exports produce valid GraphML/CSV/NDJSON and image snapshots.
+* CLI supports search, query, paths, impacted, diff, and export with stable schemas.
+* AOC guardrails: explorer never mutates facts; overlays reflect Policy Engine decisions.
+* RBAC enforced; all actions logged and observable.
+
+---
+
+## 8) Risks & mitigations
+
+* **Graph explosion on large monorepos** → tiling, clustering, budgets, and strict depth limits.
+* **Inconsistent identities across tools** → canonicalize purls/digests; property‑based tests for identity stability.
+* **Policy overlay latency** → precompute materializations for hot policy versions; sample explains only on focus.
+* **User confusion** → strong lens defaults, deterministic layouts, legends, in‑context help.
+
+---
+
+## 9) Test plan
+
+* **Unit**: node/edge builders, identity normalization, cost estimator.
+* **Integration**: ingest SBOM + advisories + VEX, verify overlays and counts.
+* **E2E**: Playwright flows for search→impact→diff→export; deep link determinism.
+* **Performance**: simulate 500k nodes/2M edges; measure tile latency and memory.
+* **Security**: RBAC matrix; tenant isolation tests; query validation fuzzing.
+* **Determinism**: snapshot round‑trip: same query and seed produce identical layout and stats.
+
+---
+
+## 10) Feature flags
+
+* `graph.explorer` (UI feature module)
+* `graph.paths` (advanced path queries)
+* `graph.diff` (SBOM diff mode)
+* `graph.overlays.policy` (policy overlay + explain sampling)
+* `graph.export` (exports enabled)
+
+Documented in `/docs/observability/graph-telemetry.md`.
+
+---
+
+## 11) Non‑goals (this epic)
+
+* Real‑time process/runtime call graphs.
+* Full substitution for text reports; Explorer complements Reports.
+* Cross‑tenant graphs; all queries are tenant‑scoped.
+
+---
+
+## 12) Philosophy
+
+* **See the system**: security and license risk are structural. If you cannot see structure, you will miss risk.
+* **Evidence over assertion**: every colored node corresponds to raw facts and explainable determinations.
+* **Bounded interactivity**: fast, partial answers beat slow “complete” ones.
+* **Immutability**: graphs mirror SBOM snapshots and are never rewritten; we add context, not edits.
+
+> Final reminder: **Work of this type or tasks of this type on this component must also be applied everywhere else it should be applied.**