diff --git a/docs/implplan/SPRINT_20260419_027_Concelier_durable_affected_symbol_runtime.md b/docs/implplan/SPRINT_20260419_027_Concelier_durable_affected_symbol_runtime.md new file mode 100644 index 000000000..712a4edbc --- /dev/null +++ b/docs/implplan/SPRINT_20260419_027_Concelier_durable_affected_symbol_runtime.md @@ -0,0 +1,51 @@ +# Sprint 20260419-027 - Concelier Durable Affected Symbol Runtime + +## Topic & Scope +- Replace the live affected-symbol truthful `501` fallback with a durable PostgreSQL-backed runtime. +- Persist advisory observations from the real raw-ingest path so live read models and signal writers stop depending on no-op observation services. +- Wire ingest-time affected-symbol extraction from persisted advisory observations without changing AOC raw-fact ownership. +- Working directory: `src/Concelier/`. +- Cross-module touchpoints explicitly allowed for this sprint: `docs/modules/concelier/`, `docs/implplan/`. +- Expected evidence: focused persistence tests, targeted runtime/DI proof, and synced Concelier architecture/task-board notes. + +## Dependencies & Concurrency +- Depends on `docs/implplan/SPRINT_20260417_023_Concelier_truthful_affected_symbol_runtime.md`, which intentionally moved `/v1/signals/symbols/*` to explicit `501` until a durable backend existed. +- Safe to execute in parallel with unrelated Excititor or non-Concelier work because the write scope is limited to Concelier persistence, the Concelier raw-ingest pipeline, signals runtime composition, and their tests/docs. + +## Documentation Prerequisites +- `docs/modules/concelier/architecture.md` +- `src/Concelier/StellaOps.Concelier.WebService/TASKS.md` +- `src/Concelier/__Libraries/StellaOps.Concelier.Persistence/TASKS.md` +- `src/Concelier/__Tests/StellaOps.Concelier.WebService.Tests/TASKS.md` +- `src/Concelier/__Tests/StellaOps.Concelier.Persistence.Tests/TASKS.md` + +## Delivery Tracker + +### REALPLAN-007-F - Durable advisory observation and affected-symbol runtime +Status: DOING +Dependency: REALPLAN-007-E +Owners: Developer, Test Automation, Documentation author +Task description: +- The live Concelier raw-ingest path already materializes `AdvisoryObservation` instances inside `AdvisoryRawService`, but the observation pipeline still resolves `IAdvisoryObservationSink` to a null sink and has no persistent `IAdvisoryObservationLookup`. That leaves live linkset/read-model consumers dependent on testing-only replacements instead of a durable runtime owner. +- The affected-symbol surface also has no durable backend or writer path. This sprint adds PostgreSQL-backed advisory-observation persistence plus a PostgreSQL-backed affected-symbol store, then hooks the real ingest path to extract and persist symbols from advisory observations so `/v1/signals/symbols/*` can stop returning truthful `501`. + +Completion criteria: +- [ ] `AddConcelierPostgresStorage` registers durable advisory-observation lookup/sink services and a durable affected-symbol store. +- [ ] The live raw-ingest path persists advisory observations and stores extracted affected symbols from those observations. +- [ ] Non-testing Concelier runtime resolves the durable affected-symbol services instead of `UnsupportedAffectedSymbol*`. +- [ ] Focused persistence/runtime proof covers observation persistence, affected-symbol persistence/query behavior, and ingest-time symbol extraction. +- [ ] Concelier docs and task boards describe the implemented durable runtime contract. + +## Execution Log +| Date (UTC) | Update | Owner | +| --- | --- | --- | +| 2026-04-19 | Sprint created and set `DOING` after reviewing the truthful-runtime follow-up gap: the real raw-ingest path already emits `AdvisoryObservation`, but live persistence still uses a null observation sink/lookup and no durable affected-symbol backend exists. | Developer | + +## Decisions & Risks +- Decision: include durable advisory-observation persistence in the same slice as affected-symbol durability because the live writer hook already materializes observations inside `AdvisoryRawService`; leaving that path null would keep the affected-symbol writer detached from the real runtime. +- Risk: advisory IDs can arrive under multiple aliases (`CVE`, `GHSA`, vendor IDs). The durable affected-symbol store must preserve a primary advisory key while still allowing alias-based lookup so the runtime does not regress query behavior. +- Documentation sync target: `docs/modules/concelier/architecture.md` plus the relevant Concelier task boards once the runtime contract is finalized. + +## Next Checkpoints +- Finish durable observation + affected-symbol persistence. +- Replace the explicit `501` affected-symbol runtime proof with focused durable-runtime verification. diff --git a/src/Concelier/__Libraries/StellaOps.Concelier.Persistence/Extensions/ConcelierPersistenceExtensions.cs b/src/Concelier/__Libraries/StellaOps.Concelier.Persistence/Extensions/ConcelierPersistenceExtensions.cs index ae23ebe35..ee9d2bed4 100644 --- a/src/Concelier/__Libraries/StellaOps.Concelier.Persistence/Extensions/ConcelierPersistenceExtensions.cs +++ b/src/Concelier/__Libraries/StellaOps.Concelier.Persistence/Extensions/ConcelierPersistenceExtensions.cs @@ -7,6 +7,8 @@ using Microsoft.Extensions.Configuration; using Microsoft.Extensions.DependencyInjection; using PsirtContracts = StellaOps.Concelier.Storage.PsirtFlags; using StellaOps.Concelier.Core.Linksets; +using StellaOps.Concelier.Core.Observations; +using StellaOps.Concelier.Core.Signals; using StellaOps.Concelier.Merge.Backport; using StellaOps.Concelier.Persistence.Postgres; using StellaOps.Concelier.Persistence.Postgres.Advisories; @@ -60,8 +62,14 @@ public static class ConcelierPersistenceExtensions services.AddScoped(); services.AddScoped(); services.AddScoped(); - services.AddScoped(); - services.AddScoped(sp => sp.GetRequiredService()); + services.AddSingleton(); + services.AddSingleton(sp => sp.GetRequiredService()); + services.AddSingleton(sp => sp.GetRequiredService()); + services.AddSingleton(); + services.AddSingleton(sp => sp.GetRequiredService()); + services.AddSingleton(); + services.AddSingleton(sp => sp.GetRequiredService()); + services.AddSingleton(sp => sp.GetRequiredService()); services.AddScoped(); services.AddScoped(); services.AddScoped(); @@ -112,8 +120,14 @@ public static class ConcelierPersistenceExtensions services.AddScoped(); services.AddScoped(); services.AddScoped(); - services.AddScoped(); - services.AddScoped(sp => sp.GetRequiredService()); + services.AddSingleton(); + services.AddSingleton(sp => sp.GetRequiredService()); + services.AddSingleton(sp => sp.GetRequiredService()); + services.AddSingleton(); + services.AddSingleton(sp => sp.GetRequiredService()); + services.AddSingleton(); + services.AddSingleton(sp => sp.GetRequiredService()); + services.AddSingleton(sp => sp.GetRequiredService()); services.AddScoped(); services.AddScoped(); services.AddScoped(); diff --git a/src/Concelier/__Libraries/StellaOps.Concelier.Persistence/Migrations/008_add_advisory_observations_and_affected_symbols.sql b/src/Concelier/__Libraries/StellaOps.Concelier.Persistence/Migrations/008_add_advisory_observations_and_affected_symbols.sql new file mode 100644 index 000000000..ecbcdd1f8 --- /dev/null +++ b/src/Concelier/__Libraries/StellaOps.Concelier.Persistence/Migrations/008_add_advisory_observations_and_affected_symbols.sql @@ -0,0 +1,52 @@ +CREATE TABLE IF NOT EXISTS vuln.advisory_observations ( + tenant_id TEXT NOT NULL, + observation_id TEXT NOT NULL, + source_vendor TEXT NOT NULL, + upstream_id TEXT NOT NULL, + linkset_aliases TEXT[] NOT NULL DEFAULT ARRAY[]::TEXT[], + linkset_alias_keys TEXT[] NOT NULL DEFAULT ARRAY[]::TEXT[], + linkset_purls TEXT[] NOT NULL DEFAULT ARRAY[]::TEXT[], + linkset_cpes TEXT[] NOT NULL DEFAULT ARRAY[]::TEXT[], + created_at TIMESTAMPTZ NOT NULL, + observation_json JSONB NOT NULL, + PRIMARY KEY (tenant_id, observation_id) +); + +CREATE INDEX IF NOT EXISTS idx_advisory_observations_tenant_created + ON vuln.advisory_observations(tenant_id, created_at DESC, observation_id ASC); +CREATE INDEX IF NOT EXISTS idx_advisory_observations_tenant_upstream + ON vuln.advisory_observations(tenant_id, upstream_id); +CREATE INDEX IF NOT EXISTS idx_advisory_observations_alias_keys + ON vuln.advisory_observations USING GIN(linkset_alias_keys); +CREATE INDEX IF NOT EXISTS idx_advisory_observations_purls + ON vuln.advisory_observations USING GIN(linkset_purls); +CREATE INDEX IF NOT EXISTS idx_advisory_observations_cpes + ON vuln.advisory_observations USING GIN(linkset_cpes); + +CREATE TABLE IF NOT EXISTS vuln.affected_symbols ( + id BIGSERIAL PRIMARY KEY, + tenant_id TEXT NOT NULL, + advisory_id TEXT NOT NULL, + advisory_id_key TEXT NOT NULL, + observation_id TEXT NOT NULL, + canonical_id TEXT NOT NULL, + symbol_type TEXT NOT NULL, + purl TEXT, + purl_key TEXT NOT NULL DEFAULT '', + provenance_source_key TEXT NOT NULL, + has_source_location BOOLEAN NOT NULL DEFAULT FALSE, + extracted_at TIMESTAMPTZ NOT NULL, + symbol_json JSONB NOT NULL +); + +CREATE INDEX IF NOT EXISTS idx_affected_symbols_tenant_advisory + ON vuln.affected_symbols(tenant_id, advisory_id_key, extracted_at DESC, canonical_id ASC); +CREATE INDEX IF NOT EXISTS idx_affected_symbols_tenant_package + ON vuln.affected_symbols(tenant_id, purl_key, extracted_at DESC, canonical_id ASC) + WHERE purl_key <> ''; +CREATE INDEX IF NOT EXISTS idx_affected_symbols_tenant_observation + ON vuln.affected_symbols(tenant_id, observation_id); +CREATE INDEX IF NOT EXISTS idx_affected_symbols_tenant_source + ON vuln.affected_symbols(tenant_id, provenance_source_key, extracted_at DESC); +CREATE INDEX IF NOT EXISTS idx_affected_symbols_tenant_location + ON vuln.affected_symbols(tenant_id, has_source_location, extracted_at DESC); diff --git a/src/Concelier/__Libraries/StellaOps.Concelier.Persistence/Postgres/ServiceCollectionExtensions.cs b/src/Concelier/__Libraries/StellaOps.Concelier.Persistence/Postgres/ServiceCollectionExtensions.cs index 536e094ef..132764f40 100644 --- a/src/Concelier/__Libraries/StellaOps.Concelier.Persistence/Postgres/ServiceCollectionExtensions.cs +++ b/src/Concelier/__Libraries/StellaOps.Concelier.Persistence/Postgres/ServiceCollectionExtensions.cs @@ -9,6 +9,8 @@ using PsirtContracts = StellaOps.Concelier.Storage.PsirtFlags; using StellaOps.Concelier.Core.Canonical; using StellaOps.Concelier.Core.Jobs; using StellaOps.Concelier.Core.Linksets; +using StellaOps.Concelier.Core.Observations; +using StellaOps.Concelier.Core.Signals; using StellaOps.Concelier.Merge.Backport; using StellaOps.Concelier.Persistence.Postgres.Advisories; using StellaOps.Concelier.Persistence.Postgres.Repositories; @@ -68,8 +70,14 @@ public static class ServiceCollectionExtensions services.AddScoped(); services.AddScoped(); services.AddScoped(); - services.AddScoped(); - services.AddScoped(sp => sp.GetRequiredService()); + services.AddSingleton(); + services.AddSingleton(sp => sp.GetRequiredService()); + services.AddSingleton(sp => sp.GetRequiredService()); + services.AddSingleton(); + services.AddSingleton(sp => sp.GetRequiredService()); + services.AddSingleton(); + services.AddSingleton(sp => sp.GetRequiredService()); + services.AddSingleton(sp => sp.GetRequiredService()); services.AddScoped(); services.AddScoped(); services.AddScoped(); @@ -127,8 +135,14 @@ public static class ServiceCollectionExtensions services.AddScoped(); services.AddScoped(); services.AddScoped(); - services.AddScoped(); - services.AddScoped(sp => sp.GetRequiredService()); + services.AddSingleton(); + services.AddSingleton(sp => sp.GetRequiredService()); + services.AddSingleton(sp => sp.GetRequiredService()); + services.AddSingleton(); + services.AddSingleton(sp => sp.GetRequiredService()); + services.AddSingleton(); + services.AddSingleton(sp => sp.GetRequiredService()); + services.AddSingleton(sp => sp.GetRequiredService()); services.AddScoped(); services.AddScoped(); services.AddScoped();