108 lines
7.1 KiB
Markdown
108 lines
7.1 KiB
Markdown
# Graph Indexer Incremental Update Pipeline
|
|
|
|
## Module
|
|
Graph
|
|
|
|
## Status
|
|
VERIFIED
|
|
|
|
## Description
|
|
Change-stream processor for incremental graph updates, consuming SBOM/scan events and applying delta mutations to the indexed graph with idempotency tracking and backfill metrics.
|
|
|
|
## Implementation Details
|
|
- **Change stream processor**: `src/Graph/StellaOps.Graph.Indexer/Incremental/GraphChangeStreamProcessor.cs` -- consumes SBOM/scan events and applies delta mutations to indexed graph
|
|
- **Change event model**: `src/Graph/StellaOps.Graph.Indexer/Incremental/GraphChangeEvent.cs` -- event model for graph mutations
|
|
- **Stream options**: `src/Graph/StellaOps.Graph.Indexer/Incremental/GraphChangeStreamOptions.cs` -- configurable stream processing parameters
|
|
- **Idempotency store**: `src/Graph/StellaOps.Graph.Indexer/Incremental/InMemoryIdempotencyStore.cs` -- in-memory deduplication for event processing
|
|
- **Postgres idempotency**: `src/Graph/__Libraries/StellaOps.Graph.Indexer.Persistence/Postgres/Repositories/PostgresIdempotencyStore.cs` -- PostgreSQL-backed idempotency tracking
|
|
- **No-op event source**: `src/Graph/StellaOps.Graph.Indexer/Incremental/NoOpGraphChangeEventSource.cs` -- stub event source for testing
|
|
- **Backfill metrics**: `src/Graph/StellaOps.Graph.Indexer/Incremental/GraphBackfillMetrics.cs` -- metrics for backfill operations
|
|
- **DI extensions**: `src/Graph/StellaOps.Graph.Indexer/Incremental/GraphChangeStreamServiceCollectionExtensions.cs` -- DI registration
|
|
- **SBOM ingestion**: `src/Graph/StellaOps.Graph.Indexer/Ingestion/Sbom/SbomIngestProcessor.cs`, `SbomIngestTransformer.cs` -- SBOM event processing and graph transformation
|
|
- **Tests**: `src/Graph/__Tests/StellaOps.Graph.Indexer.Tests/GraphChangeStreamProcessorTests.cs`, `GraphIndexerEndToEndTests.cs`
|
|
- **Persistence tests**: `src/Graph/__Tests/StellaOps.Graph.Indexer.Persistence.Tests/PostgresIdempotencyStoreTests.cs`
|
|
- **Source**: SPRINT_0141_0001_0001_graph_indexer.md
|
|
|
|
## E2E Test Plan
|
|
- [x] Verify change stream processor applies delta mutations correctly
|
|
- [x] Test idempotency ensures duplicate events are not processed
|
|
- [x] Verify backfill metrics track progress accurately
|
|
- [x] Test SBOM ingestion transforms events into graph updates
|
|
- [x] Verify PostgreSQL idempotency store persists across restarts
|
|
|
|
## Verification
|
|
- **Run ID**: run-001
|
|
- **Date**: 2026-02-09T16:00:00Z
|
|
- **Tier**: 1 (Build + Test)
|
|
- **Result**: PASS
|
|
- **Evidence**: Graph.Indexer.Tests 37/37 pass. 4 PostgresIdempotencyStore tests skipped (Docker unavailable, env_issue). All source files verified (13/13).
|
|
|
|
### Tier 2 Recheck (Behavioral Integration)
|
|
- **Run ID**: run-002
|
|
- **Date**: 2026-02-10T11:41:00Z
|
|
- **Tier**: 2 (Behavioral integration verification)
|
|
- **Result**: PASS
|
|
- **Evidence**: Revalidated incremental update and idempotency behavior across indexer and persistence suites (`Graph.Indexer.Tests` 37/37 and `Graph.Indexer.Persistence.Tests` 17/17). Artifact: `docs/qa/feature-checks/runs/graph/graph-indexer-incremental-update-pipeline/run-002/tier2-integration-check.json`.
|
|
|
|
### Tier 2 Recheck (Docker-Restored Persistence Replay)
|
|
- **Run ID**: run-003
|
|
- **Date**: 2026-02-10T16:37:52Z
|
|
- **Tier**: 2 (Behavioral integration verification)
|
|
- **Result**: PASS
|
|
- **Evidence**: Replayed indexer + persistence suites with Docker available (Graph.Indexer.Tests 37/37, Graph.Indexer.Persistence.Tests 17/17). Artifact: `docs/qa/feature-checks/runs/graph/graph-indexer-incremental-update-pipeline/run-003/tier2-integration-check.json`.
|
|
|
|
## Recheck (Run-005)
|
|
- **Verified**: 2026-02-10
|
|
- **Method**: Tier 2d deterministic integration replay.
|
|
- **Tests**: PASS (`src/Graph/__Tests/StellaOps.Graph.Indexer.Tests`: 37/37; `src/Graph/__Tests/StellaOps.Graph.Indexer.Persistence.Tests`: 17/17).
|
|
- **Tier 2 Evidence**: `docs/qa/feature-checks/runs/graph/graph-indexer-incremental-update-pipeline/run-005/tier2-integration-check.json`
|
|
- **Outcome**: Incremental update and idempotency behavior remains healthy.
|
|
|
|
## Recheck (Run-006)
|
|
- **Verified**: 2026-02-10
|
|
- **Method**: Tier 2 replay validated via Graph API/indexer suites.
|
|
- **Tests**: PASS (src/Graph/__Tests/StellaOps.Graph.Api.Tests: 66/66; src/Graph/__Tests/StellaOps.Graph.Indexer.Tests: 37/37; src/Graph/__Tests/StellaOps.Graph.Indexer.Persistence.Tests: 17/17).
|
|
- **Tier 2 Evidence**: docs/qa/feature-checks/runs/graph/graph-indexer-incremental-update-pipeline/run-006/tier2-integration-check.json
|
|
- **Outcome**: Incremental indexing and persistence idempotency behavior remain healthy.
|
|
|
|
## Recheck (Run-007)
|
|
- **Verified**: 2026-02-10
|
|
- **Method**: Tier 2 replay validated via Graph API/indexer suites.
|
|
- **Tests**: PASS (src/Graph/__Tests/StellaOps.Graph.Api.Tests: 66/66; src/Graph/__Tests/StellaOps.Graph.Indexer.Tests: 37/37; src/Graph/__Tests/StellaOps.Graph.Indexer.Persistence.Tests: 17/17).
|
|
- **Tier 2 Evidence**: docs/qa/feature-checks/runs/graph/graph-indexer-incremental-update-pipeline/run-007/tier2-integration-check.json
|
|
- **Outcome**: Incremental indexing and persistence idempotency behavior remain healthy.
|
|
|
|
## Recheck (Run-008)
|
|
- **Verified**: 2026-02-10
|
|
- **Method**: Tier 2 replay validated via Graph API/indexer suites.
|
|
- **Tests**: PASS (src/Graph/__Tests/StellaOps.Graph.Api.Tests: 66/66; src/Graph/__Tests/StellaOps.Graph.Indexer.Tests: 37/37; src/Graph/__Tests/StellaOps.Graph.Indexer.Persistence.Tests: 17/17).
|
|
- **Tier 2 Evidence**: docs/qa/feature-checks/runs/graph/graph-indexer-incremental-update-pipeline/run-008/tier2-integration-check.json
|
|
- **Outcome**: Incremental indexing and persistence idempotency behavior remain healthy.
|
|
|
|
## Recheck (Run-009)
|
|
- **Verified**: 2026-02-10
|
|
- **Method**: Tier 2d deterministic integration replay.
|
|
- **Tests**: PASS (Graph.Api.Tests 66/66, Graph.Indexer.Tests 37/37, Graph.Indexer.Persistence.Tests 17/17).
|
|
- **Tier 2 Evidence**: docs/qa/feature-checks/runs/graph/graph-indexer-incremental-update-pipeline/run-009/tier2-integration-check.json
|
|
- **Outcome**: Checked Graph behavior remains healthy in continued replay.
|
|
|
|
|
|
## Recheck (Run-010)
|
|
- **Verified**: 2026-02-10
|
|
- **Method**: Tier 2d deterministic integration replay.
|
|
- **Tests**: PASS (Graph Api 66/66, Indexer 37/37, Indexer.Persistence 17/17; total 120/120).
|
|
- **Tier 2 Evidence**: docs/qa/feature-checks/runs/graph/graph-indexer-incremental-update-pipeline/run-010/tier2-integration-check.json
|
|
- **Outcome**: Checked graph behavior remains healthy in continued replay.
|
|
## Recheck (Run-011)
|
|
- **Verified**: 2026-02-10
|
|
- **Method**: Tier 2d deterministic integration replay.
|
|
- **Tests**: PASS (Graph Api 66/66, Indexer 37/37, Indexer.Persistence 17/17; total 120/120).
|
|
- **Tier 2 Evidence**: docs/qa/feature-checks/runs/graph/graph-indexer-incremental-update-pipeline/run-011/tier2-integration-check.json
|
|
- **Outcome**: Checked graph behavior remains healthy in continued replay.
|
|
## Recheck (Run-012)
|
|
- **Verified**: 2026-02-10
|
|
- **Method**: Tier 2 replay + deterministic Graph suite replay.
|
|
- **Tests**: PASS (Graph.Api 66/66, Graph.Indexer 37/37, Graph.Indexer.Persistence 17/17; total 120/120).
|
|
- **Tier 2 Evidence**: docs/qa/feature-checks/runs/graph/graph-indexer-incremental-update-pipeline/run-012/tier2-integration-check.json
|
|
- **Outcome**: Checked graph behavior remains healthy in continued replay.
|