stella-ops.org/git.stella-ops.org
1
0
Fork 0
Code Issues Pull Requests Actions 1 Releases Wiki Activity

Align AOC tasks for Excititor and Concelier

Browse Source
This commit is contained in:
master
2025-10-31 18:50:15 +02:00
committed by root
parent 9e6d9fbae8
commit 8da4e12a90
334 changed files with 35528 additions and 34546 deletions
Download Patch File Download Diff File Expand all files Collapse all files

42
docs/modules/graph/AGENTS.md
View File

@@ -1,22 +1,22 @@
# Graph agent guide
## Mission
Graph module (upcoming) will power graph-indexed queries for SBOM relationships, lineage, and blast-radius analysis.
## Key docs
- [Module README](./README.md)
- [Architecture](./architecture.md)
- [Implementation plan](./implementation_plan.md)
- [Task board](./TASKS.md)
## How to get started
1. Open ../../implplan/SPRINTS.md and locate the stories referencing this module.
2. Review ./TASKS.md for local follow-ups and confirm status transitions (TODO → DOING → DONE/BLOCKED).
3. Read the architecture and README for domain context before editing code or docs.
4. Coordinate cross-module changes in the main /AGENTS.md description and through the sprint plan.
## Guardrails
- Honour the Aggregation-Only Contract where applicable (see ../../ingestion/aggregation-only-contract.md).
- Preserve determinism: sort outputs, normalise timestamps (UTC ISO-8601), and avoid machine-specific artefacts.
- Keep Offline Kit parity in mind—document air-gapped workflows for any new feature.
# Graph agent guide
## Mission
Graph module (upcoming) will power graph-indexed queries for SBOM relationships, lineage, and blast-radius analysis.
## Key docs
- [Module README](./README.md)
- [Architecture](./architecture.md)
- [Implementation plan](./implementation_plan.md)
- [Task board](./TASKS.md)
## How to get started
1. Open ../../implplan/SPRINTS.md and locate the stories referencing this module.
2. Review ./TASKS.md for local follow-ups and confirm status transitions (TODO → DOING → DONE/BLOCKED).
3. Read the architecture and README for domain context before editing code or docs.
4. Coordinate cross-module changes in the main /AGENTS.md description and through the sprint plan.
## Guardrails
- Honour the Aggregation-Only Contract where applicable (see ../../ingestion/aggregation-only-contract.md).
- Preserve determinism: sort outputs, normalise timestamps (UTC ISO-8601), and avoid machine-specific artefacts.
- Keep Offline Kit parity in mind—document air-gapped workflows for any new feature.
- Update runbooks/observability assets when operational characteristics change.

62
docs/modules/graph/README.md
View File

@@ -1,31 +1,31 @@
# StellaOps Graph
Graph module (upcoming) will power graph-indexed queries for SBOM relationships, lineage, and blast-radius analysis.
## Responsibilities
- Model SBOM and advisory entities as a navigable graph.
- Provide APIs for dependency impact, provenance chains, and reachability analysis.
- Integrate with Scheduler/Policy for graph-driven re-evaluation.
- Expose tooling for offline explorers.
### Domain highlights (Epic 5)
- **Nodes:** artifacts/images, SBOM components, packages/versions, files/paths, licences, advisories, VEX statements, provenance attestations, policy versions.
- **Edges:** `depends_on`, `contains`, `built_from`, `declared_in`, `affected_by`, `vex_exempts`, `governs_with`, `produced_by`, each timestamped and tenant-scoped.
- **Overlays:** policy verdict overlays, VEX consensus, runtime telemetry, and export-ready snapshots with diff support.
- **Queries:** reachability (`impact(graph, advisory)`), blast radius (`reverseDepends(component)`), provenance timeline, saved query library with semantic zoom for Console.
## Key components
- Planned services documented in implementation plan (to be delivered).
## Integrations & dependencies
- SBOM Service / Cartographer for data ingestion.
- Policy & CLI for query surfaces.
## Operational notes
- Pending — see implementation plan for staged milestones.
## Backlog references
- DOCS-GRAPH-24-003 (architecture index) and SCHED-MODELS-21-001 tasks.
## Epic alignment
- **Epic 5 – SBOM Graph Explorer:** deliver graph indexer, API, Console explorer, saved queries, overlays, and exports.
# StellaOps Graph
Graph module (upcoming) will power graph-indexed queries for SBOM relationships, lineage, and blast-radius analysis.
## Responsibilities
- Model SBOM and advisory entities as a navigable graph.
- Provide APIs for dependency impact, provenance chains, and reachability analysis.
- Integrate with Scheduler/Policy for graph-driven re-evaluation.
- Expose tooling for offline explorers.
### Domain highlights (Epic 5)
- **Nodes:** artifacts/images, SBOM components, packages/versions, files/paths, licences, advisories, VEX statements, provenance attestations, policy versions.
- **Edges:** `depends_on`, `contains`, `built_from`, `declared_in`, `affected_by`, `vex_exempts`, `governs_with`, `produced_by`, each timestamped and tenant-scoped.
- **Overlays:** policy verdict overlays, VEX consensus, runtime telemetry, and export-ready snapshots with diff support.
- **Queries:** reachability (`impact(graph, advisory)`), blast radius (`reverseDepends(component)`), provenance timeline, saved query library with semantic zoom for Console.
## Key components
- Planned services documented in implementation plan (to be delivered).
## Integrations & dependencies
- SBOM Service / Cartographer for data ingestion.
- Policy & CLI for query surfaces.
## Operational notes
- Pending — see implementation plan for staged milestones.
## Backlog references
- DOCS-GRAPH-24-003 (architecture index) and SCHED-MODELS-21-001 tasks.
## Epic alignment
- **Epic 5 – SBOM Graph Explorer:** deliver graph indexer, API, Console explorer, saved queries, overlays, and exports.

18
docs/modules/graph/TASKS.md
View File

@@ -1,9 +1,9 @@
# Task board — Graph
> Local tasks should link back to ./AGENTS.md and mirror status updates into ../../TASKS.md when applicable.
| ID | Status | Owner(s) | Description | Notes |
|----|--------|----------|-------------|-------|
| GRAPH-DOCS-0001 | DOING (2025-10-29) | Docs Guild | Validate that ./README.md aligns with the latest release notes. | See ./AGENTS.md |
| GRAPH-OPS-0001 | TODO | Ops Guild | Review runbooks/observability assets after next sprint demo. | Sync outcomes back to ../../TASKS.md |
| GRAPH-ENG-0001 | TODO | Module Team | Cross-check implementation plan milestones against ../../implplan/SPRINTS.md. | Update status via ./AGENTS.md workflow |
# Task board — Graph
> Local tasks should link back to ./AGENTS.md and mirror status updates into ../../TASKS.md when applicable.
| ID | Status | Owner(s) | Description | Notes |
|----|--------|----------|-------------|-------|
| GRAPH-DOCS-0001 | DOING (2025-10-29) | Docs Guild | Validate that ./README.md aligns with the latest release notes. | See ./AGENTS.md |
| GRAPH-OPS-0001 | TODO | Ops Guild | Review runbooks/observability assets after next sprint demo. | Sync outcomes back to ../../TASKS.md |
| GRAPH-ENG-0001 | TODO | Module Team | Cross-check implementation plan milestones against ../../implplan/SPRINTS.md. | Update status via ./AGENTS.md workflow |

112
docs/modules/graph/architecture.md
View File

@@ -1,56 +1,56 @@
# Graph architecture
> Derived from Epic 5 – SBOM Graph Explorer; this section captures the core model, pipeline, and API expectations. Extend with diagrams as implementation matures.
## 1) Core model
- **Nodes:**
- `Artifact` (application/image digest) with metadata (tenant, environment, labels).
- `Component` (package/version, purl, ecosystem).
- `File`/`Path` (source files, binary paths) with hash/time metadata.
- `License` nodes linked to components and SBOM attestations.
- `Advisory` and `VEXStatement` nodes linking to Concelier/Excititor records via digests.
- `PolicyVersion` nodes representing signed policy packs.
- **Edges:** directed, timestamped relationships such as `DEPENDS_ON`, `BUILT_FROM`, `DECLARED_IN`, `AFFECTED_BY`, `VEX_EXEMPTS`, `GOVERNS_WITH`, `OBSERVED_RUNTIME`. Each edge carries provenance (SRM hash, SBOM digest, policy run ID).
- **Overlays:** computed index tables providing fast access to reachability, blast radius, and differential views (e.g., `graph_overlay/vuln/{tenant}/{advisoryKey}`).
## 2) Pipelines
1. **Ingestion:** Cartographer/SBOM Service emit SBOM snapshots (`sbom_snapshot` events) captured by the Graph Indexer. Advisories/VEX from Concelier/Excititor generate edge updates, policy runs attach overlay metadata.
2. **ETL:** Normalises nodes/edges into canonical IDs, deduplicates, enforces tenant partitions, and writes to the graph store (planned: Neo4j-compatible or document + adjacency lists in Mongo).
3. **Overlay computation:** Batch workers build materialised views for frequently used queries (impact lists, saved queries, policy overlays) and store as immutable blobs for Offline Kit exports.
4. **Diffing:** `graph_diff` jobs compare two snapshots (e.g., pre/post deploy) and generate signed diff manifests for UI/CLI consumption.
## 3) APIs
- `GET /graph/nodes/{id}` — fetch node with metadata and attached provenance.
- `POST /graph/query/saved` — execute saved query (Cypher-like DSL) with tenant filtering; supports paging, citation metadata, and `explain` traces.
- `GET /graph/impact/{advisoryKey}` — returns impacted artifacts with path context and policy/vex overlays.
- `GET /graph/diff/{snapshotA}/{snapshotB}` — streaming API returning diff manifest including new/removed edges, risk summary, and export references.
- `POST /graph/overlay/policy` — create or retrieve overlay for policy version + advisory set, referencing `effective_finding` results.
## 4) Storage considerations
- Backed by either:
- **Document + adjacency** (Mongo collections `graph_nodes`, `graph_edges`, `graph_overlays`) with deterministic ordering and streaming exports.
- Or **Graph DB** (e.g., Neo4j/Cosmos Gremlin) behind an abstraction layer; choice depends on deployment footprint.
- All storages require tenant partitioning, append-only change logs, and export manifests for Offline Kits.
## 5) Offline & export
- Each snapshot packages `nodes.jsonl`, `edges.jsonl`, `overlays/` plus manifest with hash, counts, and provenance. Export Center consumes these artefacts for graph-specific bundles.
- Saved queries and overlays include deterministic IDs so Offline Kit consumers can import and replay results.
## 6) Observability
- Metrics: ingestion lag (`graph_ingest_lag_seconds`), node/edge counts, query latency per saved query, overlay generation duration.
- Logs: structured events for ETL stages and query execution (with trace IDs).
- Traces: ETL pipeline spans, query engine spans.
## 7) Rollout notes
- Phase 1: ingest SBOM + advisories, deliver impact queries.
- Phase 2: add VEX overlays, policy overlays, diff tooling.
- Phase 3: expose runtime/Zastava edges and AI-assisted recommendations (future).
Refer to the module README and implementation plan for immediate context, and update this document once component boundaries and data flows are finalised.
# Graph architecture
> Derived from Epic 5 – SBOM Graph Explorer; this section captures the core model, pipeline, and API expectations. Extend with diagrams as implementation matures.
## 1) Core model
- **Nodes:**
- `Artifact` (application/image digest) with metadata (tenant, environment, labels).
- `Component` (package/version, purl, ecosystem).
- `File`/`Path` (source files, binary paths) with hash/time metadata.
- `License` nodes linked to components and SBOM attestations.
- `Advisory` and `VEXStatement` nodes linking to Concelier/Excititor records via digests.
- `PolicyVersion` nodes representing signed policy packs.
- **Edges:** directed, timestamped relationships such as `DEPENDS_ON`, `BUILT_FROM`, `DECLARED_IN`, `AFFECTED_BY`, `VEX_EXEMPTS`, `GOVERNS_WITH`, `OBSERVED_RUNTIME`. Each edge carries provenance (SRM hash, SBOM digest, policy run ID).
- **Overlays:** computed index tables providing fast access to reachability, blast radius, and differential views (e.g., `graph_overlay/vuln/{tenant}/{advisoryKey}`).
## 2) Pipelines
1. **Ingestion:** Cartographer/SBOM Service emit SBOM snapshots (`sbom_snapshot` events) captured by the Graph Indexer. Advisories/VEX from Concelier/Excititor generate edge updates, policy runs attach overlay metadata.
2. **ETL:** Normalises nodes/edges into canonical IDs, deduplicates, enforces tenant partitions, and writes to the graph store (planned: Neo4j-compatible or document + adjacency lists in Mongo).
3. **Overlay computation:** Batch workers build materialised views for frequently used queries (impact lists, saved queries, policy overlays) and store as immutable blobs for Offline Kit exports.
4. **Diffing:** `graph_diff` jobs compare two snapshots (e.g., pre/post deploy) and generate signed diff manifests for UI/CLI consumption.
## 3) APIs
- `GET /graph/nodes/{id}` — fetch node with metadata and attached provenance.
- `POST /graph/query/saved` — execute saved query (Cypher-like DSL) with tenant filtering; supports paging, citation metadata, and `explain` traces.
- `GET /graph/impact/{advisoryKey}` — returns impacted artifacts with path context and policy/vex overlays.
- `GET /graph/diff/{snapshotA}/{snapshotB}` — streaming API returning diff manifest including new/removed edges, risk summary, and export references.
- `POST /graph/overlay/policy` — create or retrieve overlay for policy version + advisory set, referencing `effective_finding` results.
## 4) Storage considerations
- Backed by either:
- **Document + adjacency** (Mongo collections `graph_nodes`, `graph_edges`, `graph_overlays`) with deterministic ordering and streaming exports.
- Or **Graph DB** (e.g., Neo4j/Cosmos Gremlin) behind an abstraction layer; choice depends on deployment footprint.
- All storages require tenant partitioning, append-only change logs, and export manifests for Offline Kits.
## 5) Offline & export
- Each snapshot packages `nodes.jsonl`, `edges.jsonl`, `overlays/` plus manifest with hash, counts, and provenance. Export Center consumes these artefacts for graph-specific bundles.
- Saved queries and overlays include deterministic IDs so Offline Kit consumers can import and replay results.
## 6) Observability
- Metrics: ingestion lag (`graph_ingest_lag_seconds`), node/edge counts, query latency per saved query, overlay generation duration.
- Logs: structured events for ETL stages and query execution (with trace IDs).
- Traces: ETL pipeline spans, query engine spans.
## 7) Rollout notes
- Phase 1: ingest SBOM + advisories, deliver impact queries.
- Phase 2: add VEX overlays, policy overlays, diff tooling.
- Phase 3: expose runtime/Zastava edges and AI-assisted recommendations (future).
Refer to the module README and implementation plan for immediate context, and update this document once component boundaries and data flows are finalised.

128
docs/modules/graph/implementation_plan.md
View File

@@ -1,64 +1,64 @@
# Implementation plan — Graph
## Delivery phases
- **Phase 1 – Graph Indexer foundations**
Stand up Graph Indexer service, node/edge schemas, ingestion from SBOM/Concelier/Excititor events, identity stability, and snapshot materialisation.
- **Phase 2 – Graph API service**
Expose search, query, path, impact, diff, and overlay endpoints with RBAC, cost controls, and streaming responses.
- **Phase 3 – Console & CLI experiences**
Ship Graph Explorer UI (WebGL canvas, filters, diff mode, overlays) and CLI (`stella sbom graph ...`) for automation pipelines.
- **Phase 4 – Advanced analytics**
Implement clustering, centrality, saved queries, overlay caching, and Policy Engine explain integration.
- **Phase 5 – Exports & offline**
Deliver GraphML/CSV/NDJSON exports, Offline Kit bundles (`nodes.jsonl`, `edges.jsonl`, overlays), and deterministic manifests.
- **Phase 6 – Observability & hardening**
Complete dashboards, alerts, runbooks, load/perf testing, and a11y/accessibility review.
## Work breakdown
- **Services**
- Graph Indexer: event consumers, node/edge builders, snapshot/version handling, aggregate metrics.
- Graph API: validation, planner/cost guard, streaming tile engine, diff/overlay builder, exports.
- Worker jobs: clustering, diff, overlay materialisation with backpressure awareness.
- **Data model & storage**
- Collections/tables (`graph_nodes`, `graph_edges`, `graph_snapshots`, `graph_saved_queries`, `graph_overlays_cache`), indexes, tenant partitioning, append-only change logs.
- Evaluate document + adjacency vs graph DB abstraction; ensure deterministic serialization for exports.
- **Console**
- Feature module `graph-explorer` with routes, canvas renderer, panels, diff UI, saved queries, export workflows, a11y pass.
- Telemetry instrumentation for user interactions and query budgets.
- **CLI & SDK**
- `stella sbom graph query|diff|impact|export`, with JSON schema and piping support.
- SDK utilities for automation and CI pipelines.
- **Policy & VEX integration**
- Fetch explain traces for policy overlays, integrate VEX suppressions, align with Policy Engine & VEX Lens data models.
- **Observability & Ops**
- Metrics (ingest lag, query latency, cache hit rate), log/traces, dashboards, alerting for runaway queries and OOM.
- Runbooks for incident classes (query denial, cache poisoning, degraded render).
- **Documentation**
- Maintain overview, API, query language, console guide, CLI reference, policy/VEX integration docs with compliance checklists.
## Acceptance criteria
- Graph Indexer ingests SBOM/advisory/VEX events deterministically with tenant isolation and append-only provenance.
- Graph API serves search/query/path/diff/overlay endpoints within budgeted latency and enforces cost limits + RBAC.
- Console explorer visualises topology, overlays, diffs, saved queries; CLI commands mirror functionality for automation.
- Exports (GraphML/CSV/NDJSON) and Offline Kit bundles reproduce snapshots and overlays with signed manifests.
- Observability dashboards/alerts detect ingest lag, query failures, cache churn, and memory pressure; runbooks guide remediation.
- Policy/VEX overlays align with Policy Engine explain traces and VEX suppressions.
## Risks & mitigations
- **Graph scale/complexity:** adopt adjacency compression, cached overlays, streaming pagination, enforced query budgets.
- **Tenant bleed:** strict tenant filters, fuzz tests, data masking, compliance reviews.
- **Runaway queries/visualization:** cost planner, query timeout, UI hints, safe mode renders.
- **Cache poisoning:** input validation, schema versioning, eviction policies.
- **Offline parity gaps:** deterministic export pipeline, integration tests for Offline Kit import.
## Test strategy
- **Unit:** node/edge builders, identifier stability, overlay computations, query planner, diff engine.
- **Integration:** end-to-end ingest + query flows across SBOM/advisory/VEX, saved query execution, diff exports.
- **Performance:** large SBOM datasets, concurrency, memory profiling, WebGL rendering.
- **Security:** tenant isolation tests, RBAC, query cost abuse.
- **Offline:** export/import verification, manifest hashing, CLI replay.
## Definition of done
- All phases delivered with telemetry, documentation, runbooks, and Offline Kit parity.
- Console/CLI parity validated; a11y review complete.
- ./TASKS.md and ../../TASKS.md updated; README/architecture/plan kept current with imposed rule references.
# Implementation plan — Graph
## Delivery phases
- **Phase 1 – Graph Indexer foundations**
Stand up Graph Indexer service, node/edge schemas, ingestion from SBOM/Concelier/Excititor events, identity stability, and snapshot materialisation.
- **Phase 2 – Graph API service**
Expose search, query, path, impact, diff, and overlay endpoints with RBAC, cost controls, and streaming responses.
- **Phase 3 – Console & CLI experiences**
Ship Graph Explorer UI (WebGL canvas, filters, diff mode, overlays) and CLI (`stella sbom graph ...`) for automation pipelines.
- **Phase 4 – Advanced analytics**
Implement clustering, centrality, saved queries, overlay caching, and Policy Engine explain integration.
- **Phase 5 – Exports & offline**
Deliver GraphML/CSV/NDJSON exports, Offline Kit bundles (`nodes.jsonl`, `edges.jsonl`, overlays), and deterministic manifests.
- **Phase 6 – Observability & hardening**
Complete dashboards, alerts, runbooks, load/perf testing, and a11y/accessibility review.
## Work breakdown
- **Services**
- Graph Indexer: event consumers, node/edge builders, snapshot/version handling, aggregate metrics.
- Graph API: validation, planner/cost guard, streaming tile engine, diff/overlay builder, exports.
- Worker jobs: clustering, diff, overlay materialisation with backpressure awareness.
- **Data model & storage**
- Collections/tables (`graph_nodes`, `graph_edges`, `graph_snapshots`, `graph_saved_queries`, `graph_overlays_cache`), indexes, tenant partitioning, append-only change logs.
- Evaluate document + adjacency vs graph DB abstraction; ensure deterministic serialization for exports.
- **Console**
- Feature module `graph-explorer` with routes, canvas renderer, panels, diff UI, saved queries, export workflows, a11y pass.
- Telemetry instrumentation for user interactions and query budgets.
- **CLI & SDK**
- `stella sbom graph query|diff|impact|export`, with JSON schema and piping support.
- SDK utilities for automation and CI pipelines.
- **Policy & VEX integration**
- Fetch explain traces for policy overlays, integrate VEX suppressions, align with Policy Engine & VEX Lens data models.
- **Observability & Ops**
- Metrics (ingest lag, query latency, cache hit rate), log/traces, dashboards, alerting for runaway queries and OOM.
- Runbooks for incident classes (query denial, cache poisoning, degraded render).
- **Documentation**
- Maintain overview, API, query language, console guide, CLI reference, policy/VEX integration docs with compliance checklists.
## Acceptance criteria
- Graph Indexer ingests SBOM/advisory/VEX events deterministically with tenant isolation and append-only provenance.
- Graph API serves search/query/path/diff/overlay endpoints within budgeted latency and enforces cost limits + RBAC.
- Console explorer visualises topology, overlays, diffs, saved queries; CLI commands mirror functionality for automation.
- Exports (GraphML/CSV/NDJSON) and Offline Kit bundles reproduce snapshots and overlays with signed manifests.
- Observability dashboards/alerts detect ingest lag, query failures, cache churn, and memory pressure; runbooks guide remediation.
- Policy/VEX overlays align with Policy Engine explain traces and VEX suppressions.
## Risks & mitigations
- **Graph scale/complexity:** adopt adjacency compression, cached overlays, streaming pagination, enforced query budgets.
- **Tenant bleed:** strict tenant filters, fuzz tests, data masking, compliance reviews.
- **Runaway queries/visualization:** cost planner, query timeout, UI hints, safe mode renders.
- **Cache poisoning:** input validation, schema versioning, eviction policies.
- **Offline parity gaps:** deterministic export pipeline, integration tests for Offline Kit import.
## Test strategy
- **Unit:** node/edge builders, identifier stability, overlay computations, query planner, diff engine.
- **Integration:** end-to-end ingest + query flows across SBOM/advisory/VEX, saved query execution, diff exports.
- **Performance:** large SBOM datasets, concurrency, memory profiling, WebGL rendering.
- **Security:** tenant isolation tests, RBAC, query cost abuse.
- **Offline:** export/import verification, manifest hashing, CLI replay.
## Definition of done
- All phases delivered with telemetry, documentation, runbooks, and Offline Kit parity.
- Console/CLI parity validated; a11y review complete.
- ./TASKS.md and ../../TASKS.md updated; README/architecture/plan kept current with imposed rule references.