stella-ops.org/git.stella-ops.org
1
0
Fork 0
Code Issues Pull Requests Actions 2 Releases Wiki Activity
Files
4bdc298ec1fc821ee0d9dda6d2d4643dcedf6b35
git.stella-ops.org/docs/features/unchecked/graph/graph-indexer-incremental-update-pipeline.md

2.3 KiB
Raw Blame History

Graph Indexer Incremental Update Pipeline

Module

Graph

Status

IMPLEMENTED

Description

Change-stream processor for incremental graph updates, consuming SBOM/scan events and applying delta mutations to the indexed graph with idempotency tracking and backfill metrics.

Implementation Details

  • Change stream processor: src/Graph/StellaOps.Graph.Indexer/Incremental/GraphChangeStreamProcessor.cs -- consumes SBOM/scan events and applies delta mutations to indexed graph
  • Change event model: src/Graph/StellaOps.Graph.Indexer/Incremental/GraphChangeEvent.cs -- event model for graph mutations
  • Stream options: src/Graph/StellaOps.Graph.Indexer/Incremental/GraphChangeStreamOptions.cs -- configurable stream processing parameters
  • Idempotency store: src/Graph/StellaOps.Graph.Indexer/Incremental/InMemoryIdempotencyStore.cs -- in-memory deduplication for event processing
  • Postgres idempotency: src/Graph/__Libraries/StellaOps.Graph.Indexer.Persistence/Postgres/Repositories/PostgresIdempotencyStore.cs -- PostgreSQL-backed idempotency tracking
  • No-op event source: src/Graph/StellaOps.Graph.Indexer/Incremental/NoOpGraphChangeEventSource.cs -- stub event source for testing
  • Backfill metrics: src/Graph/StellaOps.Graph.Indexer/Incremental/GraphBackfillMetrics.cs -- metrics for backfill operations
  • DI extensions: src/Graph/StellaOps.Graph.Indexer/Incremental/GraphChangeStreamServiceCollectionExtensions.cs -- DI registration
  • SBOM ingestion: src/Graph/StellaOps.Graph.Indexer/Ingestion/Sbom/SbomIngestProcessor.cs, SbomIngestTransformer.cs -- SBOM event processing and graph transformation
  • Tests: src/Graph/__Tests/StellaOps.Graph.Indexer.Tests/GraphChangeStreamProcessorTests.cs, GraphIndexerEndToEndTests.cs
  • Persistence tests: src/Graph/__Tests/StellaOps.Graph.Indexer.Persistence.Tests/PostgresIdempotencyStoreTests.cs
  • Source: SPRINT_0141_0001_0001_graph_indexer.md

E2E Test Plan

  • Verify change stream processor applies delta mutations correctly
  • Test idempotency ensures duplicate events are not processed
  • Verify backfill metrics track progress accurately
  • Test SBOM ingestion transforms events into graph updates
  • Verify PostgreSQL idempotency store persists across restarts