Restructure solution layout by module

This commit is contained in:
master
2025-10-28 15:10:40 +02:00
parent 95daa159c4
commit d870da18ce
4103 changed files with 192899 additions and 187024 deletions

View File

@@ -0,0 +1,33 @@
# Graph Indexer Guild Charter (Epic 5)
## Mission
Project SBOM, advisory, VEX, and policy overlay data into a tenant-scoped property graph powering the SBOM Graph Explorer. Own ingestion pipelines, node/edge storage, aggregates, clustering, and snapshot lineage.
## Scope
- Service source under `src/Graph/StellaOps.Graph.Indexer` (workers, ingestion pipelines, schema builders).
- Mongo collections/object storage for `graph_nodes`, `graph_edges`, `graph_snapshots`, clustering metadata.
- Event consumers: SBOM ingest, Conseiller advisories, Excitator VEX, Policy overlay materials.
- Incremental rebuild, diff, and cache warmers for graph overlays.
## Principles
1. **Immutability** Graph mirrors SBOM snapshots; new data creates new snapshots rather than mutating historical records.
2. **Determinism** Given identical inputs, node/edge ids, hashes, and aggregates remain stable across runs.
3. **Tenant isolation** Enforce isolation at ingestion, storage, and job levels; no cross-tenant leakage.
4. **AOC alignment** Indexer links facts; it never mutates advisories/VEX/policy outcomes. Conseiller/Excitator/Policy Engine remain authoritative.
5. **Performance & telemetry** Every job emits metrics (latency, node/edge counts, queue lag) and structured logs.
## Collaboration
- Keep `src/Graph/StellaOps.Graph.Indexer/TASKS.md`, `../../docs/implplan/SPRINTS.md` synchronized.
- Coordinate with SBOM Service, Policy Engine, Conseiller, Excitator, Scheduler, Web Gateway, and Console teams.
- Publish schema docs and fixtures for clients; share cost/identity conventions across services.
## Tooling
- .NET 10 preview workers (HostedService + channel pipelines).
- MongoDB for node/edge storage; S3-compatible buckets for layout tiles/snapshots if needed.
- Scheduler integration (jobs, change streams) to handle incremental updates.
## Definition of Done
- Pipelines deterministic and tested; fixtures validated.
- Metrics/logs/traces wired with tenant context.
- Schema docs + OpenAPI (where applicable) updated; compliance checklist appended.
- Offline kit includes seed data for air-gapped installs.

View File

@@ -0,0 +1,13 @@
# Graph Indexer Task Board — Epic 5: SBOM Graph Explorer
| ID | Status | Owner(s) | Depends on | Description | Exit Criteria |
|----|--------|----------|------------|-------------|---------------|
| GRAPH-INDEX-28-001 | TODO | Graph Indexer Guild | SBOM-SERVICE-21-001, CARTO-GRAPH-21-001 | Define canonical node/edge schemas, attribute dictionaries, identity rules, and seed fixtures; publish schema doc. | Schema doc merged; identity property tests pass; fixtures committed for CI usage. |
| GRAPH-INDEX-28-002 | TODO | Graph Indexer Guild | GRAPH-INDEX-28-001, SBOM-SERVICE-21-002 | Implement SBOM ingest consumer producing artifact/package/file nodes and edges with `valid_from/valid_to`, scope metadata, and provenance links. | Ingest pipeline processes sample SBOMs deterministically; metrics recorded; unit tests cover identity stability. |
| GRAPH-INDEX-28-003 | TODO | Graph Indexer Guild | GRAPH-INDEX-28-001, CONCELIER-CONSOLE-23-001 | Project Concelier linksets into overlay tiles (`affected_by` edges, evidence refs) without mutating source observations; keep advisory aggregates in overlay store only. | Overlay documents generated deterministically; raw node/edge collections remain immutable; tests cover overlay refresh and eviction. |
| GRAPH-INDEX-28-004 | TODO | Graph Indexer Guild | GRAPH-INDEX-28-001, EXCITITOR-CONSOLE-23-001 | Integrate VEX statements (`vex_exempts` edges) with justification metadata and precedence markers for overlays. | VEX edges generated; conflicts resolved deterministically; tests cover status transitions. |
| GRAPH-INDEX-28-005 | TODO | Graph Indexer Guild, Policy Guild | POLICY-ENGINE-27-001, POLICY-ENGINE-27-002 | Hydrate policy overlays into graph (`governs_with` nodes/edges) referencing effective findings and explain hashes for sampled nodes. | Overlay nodes stored with policy version id, severity, status; explain references captured; validation tests pass. |
| GRAPH-INDEX-28-006 | TODO | Graph Indexer Guild | GRAPH-INDEX-28-002..005 | Generate graph snapshots per SBOM with lineage (`derived_from`), adjacency manifests, and metadata for diff jobs. | Snapshot documents produced; lineage recorded; tests assert diff readiness; metrics emitted. |
| GRAPH-INDEX-28-007 | TODO | Graph Indexer Guild, Observability Guild | GRAPH-INDEX-28-002..006 | Implement clustering/centrality background jobs (Louvain/degree/betweenness approximations) with configurable schedules and store cluster ids on nodes. | Clustering jobs run on fixtures; metrics logged; cluster ids accessible via API; SLA documented. |
| GRAPH-INDEX-28-008 | TODO | Graph Indexer Guild | GRAPH-INDEX-28-002..007 | Provide incremental update + backfill pipeline with change streams, retry/backoff, idempotent operations, and backlog metrics. | Incremental updates replay sample change logs; retries/backoff validated; backlog metrics exported. |
| GRAPH-INDEX-28-009 | TODO | Graph Indexer Guild, QA Guild | GRAPH-INDEX-28-002..008 | Add unit/property/integration tests, synthetic large graph fixtures, chaos testing (missing overlays, cycles), and determinism checks across runs. | Test suite green; determinism harness passes across two runs; perf metrics recorded. |
| GRAPH-INDEX-28-010 | TODO | Graph Indexer Guild, DevOps Guild | GRAPH-INDEX-28-008 | Package deployment artifacts (Helm/Compose), offline seed bundles, and configuration docs; integrate Offline Kit. | Deployment descriptors merged; offline seed bundle documented; smoke deploy tested. |