32 lines
2.3 KiB
Markdown
32 lines
2.3 KiB
Markdown
# Graph Indexer Incremental Update Pipeline
|
|
|
|
## Module
|
|
Graph
|
|
|
|
## Status
|
|
IMPLEMENTED
|
|
|
|
## Description
|
|
Change-stream processor for incremental graph updates, consuming SBOM/scan events and applying delta mutations to the indexed graph with idempotency tracking and backfill metrics.
|
|
|
|
## Implementation Details
|
|
- **Change stream processor**: `src/Graph/StellaOps.Graph.Indexer/Incremental/GraphChangeStreamProcessor.cs` -- consumes SBOM/scan events and applies delta mutations to indexed graph
|
|
- **Change event model**: `src/Graph/StellaOps.Graph.Indexer/Incremental/GraphChangeEvent.cs` -- event model for graph mutations
|
|
- **Stream options**: `src/Graph/StellaOps.Graph.Indexer/Incremental/GraphChangeStreamOptions.cs` -- configurable stream processing parameters
|
|
- **Idempotency store**: `src/Graph/StellaOps.Graph.Indexer/Incremental/InMemoryIdempotencyStore.cs` -- in-memory deduplication for event processing
|
|
- **Postgres idempotency**: `src/Graph/__Libraries/StellaOps.Graph.Indexer.Persistence/Postgres/Repositories/PostgresIdempotencyStore.cs` -- PostgreSQL-backed idempotency tracking
|
|
- **No-op event source**: `src/Graph/StellaOps.Graph.Indexer/Incremental/NoOpGraphChangeEventSource.cs` -- stub event source for testing
|
|
- **Backfill metrics**: `src/Graph/StellaOps.Graph.Indexer/Incremental/GraphBackfillMetrics.cs` -- metrics for backfill operations
|
|
- **DI extensions**: `src/Graph/StellaOps.Graph.Indexer/Incremental/GraphChangeStreamServiceCollectionExtensions.cs` -- DI registration
|
|
- **SBOM ingestion**: `src/Graph/StellaOps.Graph.Indexer/Ingestion/Sbom/SbomIngestProcessor.cs`, `SbomIngestTransformer.cs` -- SBOM event processing and graph transformation
|
|
- **Tests**: `src/Graph/__Tests/StellaOps.Graph.Indexer.Tests/GraphChangeStreamProcessorTests.cs`, `GraphIndexerEndToEndTests.cs`
|
|
- **Persistence tests**: `src/Graph/__Tests/StellaOps.Graph.Indexer.Persistence.Tests/PostgresIdempotencyStoreTests.cs`
|
|
- **Source**: SPRINT_0141_0001_0001_graph_indexer.md
|
|
|
|
## E2E Test Plan
|
|
- [ ] Verify change stream processor applies delta mutations correctly
|
|
- [ ] Test idempotency ensures duplicate events are not processed
|
|
- [ ] Verify backfill metrics track progress accurately
|
|
- [ ] Test SBOM ingestion transforms events into graph updates
|
|
- [ ] Verify PostgreSQL idempotency store persists across restarts
|