semi implemented and features implemented save checkpoint

This commit is contained in:
master
2026-02-08 18:00:49 +02:00
parent 04360dff63
commit 1bf6bbf395
20895 changed files with 716795 additions and 64 deletions

View File

@@ -0,0 +1,33 @@
# Graph Analytics Engine
## Module
Graph
## Status
IMPLEMENTED
## Description
Graph analytics with engine, pipeline, DI extensions, and Postgres persistence for analytics results.
## Implementation Details
- **Analytics engine**: `src/Graph/StellaOps.Graph.Indexer/Analytics/GraphAnalyticsEngine.cs` -- core graph analytics computation engine
- **Analytics pipeline**: `src/Graph/StellaOps.Graph.Indexer/Analytics/GraphAnalyticsPipeline.cs` -- multi-stage analytics pipeline orchestration
- **Hosted service**: `src/Graph/StellaOps.Graph.Indexer/Analytics/GraphAnalyticsHostedService.cs` -- background service running analytics on schedule
- **Analytics types**: `src/Graph/StellaOps.Graph.Indexer/Analytics/GraphAnalyticsTypes.cs` -- clustering, centrality, and analytics result types
- **Metrics**: `src/Graph/StellaOps.Graph.Indexer/Analytics/GraphAnalyticsMetrics.cs` -- Prometheus-compatible analytics execution metrics
- **Options**: `src/Graph/StellaOps.Graph.Indexer/Analytics/GraphAnalyticsOptions.cs` -- configurable analytics parameters
- **Writer options**: `src/Graph/StellaOps.Graph.Indexer/Analytics/GraphAnalyticsWriterOptions.cs` -- result persistence configuration
- **Overlay exporter**: `src/Graph/StellaOps.Graph.Indexer/Analytics/GraphOverlayExporter.cs` -- exports analytics results as graph overlays
- **In-memory writer**: `src/Graph/StellaOps.Graph.Indexer/Analytics/InMemoryGraphAnalyticsWriter.cs` -- test analytics writer
- **In-memory snapshot**: `src/Graph/StellaOps.Graph.Indexer/Analytics/InMemoryGraphSnapshotProvider.cs` -- test snapshot provider
- **DI extensions**: `src/Graph/StellaOps.Graph.Indexer/Analytics/GraphAnalyticsServiceCollectionExtensions.cs` -- DI registration
- **Postgres persistence**: `src/Graph/__Libraries/StellaOps.Graph.Indexer.Persistence/Postgres/Repositories/PostgresGraphAnalyticsWriter.cs` -- PostgreSQL analytics result storage
- **Tests**: `src/Graph/__Tests/StellaOps.Graph.Indexer.Tests/GraphAnalyticsEngineTests.cs`, `GraphAnalyticsPipelineTests.cs`, `GraphOverlayExporterTests.cs`, `GraphAnalyticsTestData.cs`
- **Source**: Feature matrix scan
## E2E Test Plan
- [ ] Verify analytics engine computes clustering and centrality scores
- [ ] Test pipeline executes multi-stage analytics in correct order
- [ ] Verify hosted service runs analytics on configured schedule
- [ ] Test Postgres persistence stores analytics results correctly
- [ ] Verify overlay exporter generates valid overlay data from analytics

View File

@@ -0,0 +1,35 @@
# Graph Explorer API with Streaming Tiles
## Module
Graph
## Status
IMPLEMENTED
## Description
Graph query and visualization API providing streaming tile-based graph rendering, path queries, diff computation between graph revisions, RBAC-enforced exports (SVG/PNG/GraphML), and overlay support for policy/VEX/reachability annotations.
## Implementation Details
- **API program**: `src/Graph/StellaOps.Graph.Api/Program.cs` -- minimal API endpoints for graph queries, diffs, exports, search, paths, lineage, overlays
- **Query service**: `src/Graph/StellaOps.Graph.Api/Services/IGraphQueryService.cs`, `InMemoryGraphQueryService.cs` -- graph node/edge query
- **Search service**: `src/Graph/StellaOps.Graph.Api/Services/IGraphSearchService.cs`, `InMemoryGraphSearchService.cs` -- full-text graph search
- **Path service**: `src/Graph/StellaOps.Graph.Api/Services/IGraphPathService.cs`, `InMemoryGraphPathService.cs` -- graph path traversal
- **Diff service**: `src/Graph/StellaOps.Graph.Api/Services/IGraphDiffService.cs`, `InMemoryGraphDiffService.cs` -- diff computation between graph revisions
- **Export service**: `src/Graph/StellaOps.Graph.Api/Services/IGraphExportService.cs`, `InMemoryGraphExportService.cs` -- RBAC-enforced export (SVG/PNG/GraphML)
- **Lineage service**: `src/Graph/StellaOps.Graph.Api/Services/IGraphLineageService.cs`, `InMemoryGraphLineageService.cs` -- lineage traversal
- **Overlay service**: `src/Graph/StellaOps.Graph.Api/Services/IOverlayService.cs`, `InMemoryOverlayService.cs` -- policy/VEX/reachability overlays
- **Reachability delta**: `src/Graph/StellaOps.Graph.Api/Services/IReachabilityDeltaService.cs`, `InMemoryReachabilityDeltaService.cs` -- reachability delta computation
- **Rate limiting**: `src/Graph/StellaOps.Graph.Api/Services/RateLimiterService.cs` -- API rate limiting
- **Metrics**: `src/Graph/StellaOps.Graph.Api/Services/GraphMetrics.cs` -- Prometheus-compatible API metrics
- **Audit**: `src/Graph/StellaOps.Graph.Api/Services/IAuditLogger.cs` -- audit logging for graph access
- **Contracts**: `src/Graph/StellaOps.Graph.Api/Contracts/SearchContracts.cs`, `LineageContracts.cs`, `ReachabilityContracts.cs`
- **Tests**: `src/Graph/__Tests/StellaOps.Graph.Api.Tests/` -- QueryServiceTests, SearchServiceTests, PathServiceTests, DiffServiceTests, ExportServiceTests, LineageServiceTests, LoadTests, MetricsTests, RateLimiterServiceTests, GraphApiContractTests
- **Source**: SPRINT_0207_0001_0001_graph.md
## E2E Test Plan
- [ ] Verify graph query API returns nodes and edges for given criteria
- [ ] Test streaming tile rendering for large graphs
- [ ] Verify diff computation between two graph revisions
- [ ] Test RBAC-enforced export in SVG/PNG/GraphML formats
- [ ] Verify overlay annotations for policy/VEX/reachability layers
- [ ] Test search API returns relevant results with ranking

View File

@@ -0,0 +1,29 @@
# Graph Indexer Clustering and Centrality Background Jobs
## Module
Graph
## Status
IMPLEMENTED
## Description
Background hosted service that runs graph analytics (Louvain community detection, betweenness/closeness centrality) on the dependency graph, producing cluster assignments and centrality scores for risk prioritization.
## Implementation Details
- **Hosted service**: `src/Graph/StellaOps.Graph.Indexer/Analytics/GraphAnalyticsHostedService.cs` -- background service that schedules clustering and centrality jobs
- **Analytics engine**: `src/Graph/StellaOps.Graph.Indexer/Analytics/GraphAnalyticsEngine.cs` -- Louvain community detection and betweenness/closeness centrality algorithms
- **Analytics pipeline**: `src/Graph/StellaOps.Graph.Indexer/Analytics/GraphAnalyticsPipeline.cs` -- orchestrates multi-stage analytics (clustering -> centrality -> export)
- **Analytics types**: `src/Graph/StellaOps.Graph.Indexer/Analytics/GraphAnalyticsTypes.cs` -- `ClusterAssignment`, centrality score models
- **Metrics**: `src/Graph/StellaOps.Graph.Indexer/Analytics/GraphAnalyticsMetrics.cs` -- job execution metrics (duration, cluster count, centrality stats)
- **Options**: `src/Graph/StellaOps.Graph.Indexer/Analytics/GraphAnalyticsOptions.cs` -- configurable job intervals and algorithm parameters
- **Snapshot provider**: `src/Graph/StellaOps.Graph.Indexer/Analytics/InMemoryGraphSnapshotProvider.cs` -- provides graph snapshot for analytics input
- **Postgres writer**: `src/Graph/__Libraries/StellaOps.Graph.Indexer.Persistence/Postgres/Repositories/PostgresGraphAnalyticsWriter.cs` -- persists cluster and centrality results
- **Tests**: `src/Graph/__Tests/StellaOps.Graph.Indexer.Tests/GraphAnalyticsEngineTests.cs`, `GraphAnalyticsPipelineTests.cs`
- **Source**: SPRINT_0141_0001_0001_graph_indexer.md
## E2E Test Plan
- [ ] Verify Louvain community detection produces stable cluster assignments
- [ ] Test betweenness and closeness centrality score computation
- [ ] Verify background service runs on configured schedule
- [ ] Test analytics results are persisted to PostgreSQL
- [ ] Verify metrics are emitted for job duration and cluster counts

View File

@@ -0,0 +1,31 @@
# Graph Indexer Incremental Update Pipeline
## Module
Graph
## Status
IMPLEMENTED
## Description
Change-stream processor for incremental graph updates, consuming SBOM/scan events and applying delta mutations to the indexed graph with idempotency tracking and backfill metrics.
## Implementation Details
- **Change stream processor**: `src/Graph/StellaOps.Graph.Indexer/Incremental/GraphChangeStreamProcessor.cs` -- consumes SBOM/scan events and applies delta mutations to indexed graph
- **Change event model**: `src/Graph/StellaOps.Graph.Indexer/Incremental/GraphChangeEvent.cs` -- event model for graph mutations
- **Stream options**: `src/Graph/StellaOps.Graph.Indexer/Incremental/GraphChangeStreamOptions.cs` -- configurable stream processing parameters
- **Idempotency store**: `src/Graph/StellaOps.Graph.Indexer/Incremental/InMemoryIdempotencyStore.cs` -- in-memory deduplication for event processing
- **Postgres idempotency**: `src/Graph/__Libraries/StellaOps.Graph.Indexer.Persistence/Postgres/Repositories/PostgresIdempotencyStore.cs` -- PostgreSQL-backed idempotency tracking
- **No-op event source**: `src/Graph/StellaOps.Graph.Indexer/Incremental/NoOpGraphChangeEventSource.cs` -- stub event source for testing
- **Backfill metrics**: `src/Graph/StellaOps.Graph.Indexer/Incremental/GraphBackfillMetrics.cs` -- metrics for backfill operations
- **DI extensions**: `src/Graph/StellaOps.Graph.Indexer/Incremental/GraphChangeStreamServiceCollectionExtensions.cs` -- DI registration
- **SBOM ingestion**: `src/Graph/StellaOps.Graph.Indexer/Ingestion/Sbom/SbomIngestProcessor.cs`, `SbomIngestTransformer.cs` -- SBOM event processing and graph transformation
- **Tests**: `src/Graph/__Tests/StellaOps.Graph.Indexer.Tests/GraphChangeStreamProcessorTests.cs`, `GraphIndexerEndToEndTests.cs`
- **Persistence tests**: `src/Graph/__Tests/StellaOps.Graph.Indexer.Persistence.Tests/PostgresIdempotencyStoreTests.cs`
- **Source**: SPRINT_0141_0001_0001_graph_indexer.md
## E2E Test Plan
- [ ] Verify change stream processor applies delta mutations correctly
- [ ] Test idempotency ensures duplicate events are not processed
- [ ] Verify backfill metrics track progress accurately
- [ ] Test SBOM ingestion transforms events into graph updates
- [ ] Verify PostgreSQL idempotency store persists across restarts

View File

@@ -0,0 +1,30 @@
# Graph Overlay System (Policy, VEX, Reachability)
## Module
Graph
## Status
IMPLEMENTED
## Description
Overlay system with exporter, in-memory overlay service, and tests for layering policy/VEX/reachability data onto dependency graphs.
## Implementation Details
- **Overlay service interface**: `src/Graph/StellaOps.Graph.Api/Services/IOverlayService.cs` -- overlay query contract
- **In-memory overlay service**: `src/Graph/StellaOps.Graph.Api/Services/InMemoryOverlayService.cs` -- in-memory overlay implementation for testing
- **Overlay exporter**: `src/Graph/StellaOps.Graph.Indexer/Analytics/GraphOverlayExporter.cs` -- exports analytics results as overlay layers
- **Policy overlay processor**: `src/Graph/StellaOps.Graph.Indexer/Ingestion/Policy/PolicyOverlayProcessor.cs` -- ingests policy decisions as graph overlays
- **Policy overlay transformer**: `src/Graph/StellaOps.Graph.Indexer/Ingestion/Policy/PolicyOverlayTransformer.cs` -- transforms policy data for graph overlay
- **Policy overlay snapshot**: `src/Graph/StellaOps.Graph.Indexer/Ingestion/Policy/PolicyOverlaySnapshot.cs` -- policy overlay state
- **Policy overlay metrics**: `src/Graph/StellaOps.Graph.Indexer/Ingestion/Policy/PolicyOverlayMetrics.cs`, `IPolicyOverlayMetrics.cs` -- overlay processing metrics
- **VEX overlay**: `src/Graph/StellaOps.Graph.Indexer/Ingestion/Vex/VexOverlayTransformer.cs`, `VexOverlaySnapshot.cs` -- VEX verdict overlays on graph
- **Reachability delta**: `src/Graph/StellaOps.Graph.Api/Services/IReachabilityDeltaService.cs`, `InMemoryReachabilityDeltaService.cs` -- reachability annotation overlays
- **Tests**: `src/Graph/__Tests/StellaOps.Graph.Indexer.Tests/GraphOverlayExporterTests.cs`
- **Source**: Feature matrix scan
## E2E Test Plan
- [ ] Verify policy overlay renders policy decisions on graph nodes
- [ ] Test VEX overlay annotates graph with VEX verdict data
- [ ] Verify reachability overlay shows reachability status per edge
- [ ] Test overlay exporter generates valid overlay from analytics results
- [ ] Verify overlay stacking (multiple overlays on same graph)

View File

@@ -0,0 +1,29 @@
# Graph Query and Search API
## Module
Graph
## Status
IMPLEMENTED
## Description
Graph API with query, search, and path services for traversing and querying dependency graphs.
## Implementation Details
- **Query service**: `src/Graph/StellaOps.Graph.Api/Services/IGraphQueryService.cs`, `InMemoryGraphQueryService.cs` -- graph node/edge query with filtering
- **Search service**: `src/Graph/StellaOps.Graph.Api/Services/IGraphSearchService.cs`, `InMemoryGraphSearchService.cs` -- full-text search across graph entities
- **Path service**: `src/Graph/StellaOps.Graph.Api/Services/IGraphPathService.cs`, `InMemoryGraphPathService.cs` -- shortest-path and reachability path queries
- **Repository**: `src/Graph/StellaOps.Graph.Api/Services/InMemoryGraphRepository.cs` -- in-memory graph data store
- **Search contracts**: `src/Graph/StellaOps.Graph.Api/Contracts/SearchContracts.cs` -- search request/response DTOs
- **Lineage contracts**: `src/Graph/StellaOps.Graph.Api/Contracts/LineageContracts.cs` -- lineage query contracts
- **Reachability contracts**: `src/Graph/StellaOps.Graph.Api/Contracts/ReachabilityContracts.cs` -- reachability query contracts
- **Rate limiting**: `src/Graph/StellaOps.Graph.Api/Services/RateLimiterService.cs` -- query rate limiting
- **Tests**: `src/Graph/__Tests/StellaOps.Graph.Api.Tests/QueryServiceTests.cs`, `SearchServiceTests.cs`, `PathServiceTests.cs`, `RateLimiterServiceTests.cs`
- **Source**: Feature matrix scan
## E2E Test Plan
- [ ] Verify query API returns nodes and edges matching filter criteria
- [ ] Test full-text search returns ranked results across graph entities
- [ ] Verify path queries find shortest paths between nodes
- [ ] Test rate limiting prevents query abuse
- [ ] Verify search contracts handle empty results and pagination