diff --git a/docs/implplan/SPRINT_20260419_027_Concelier_durable_affected_symbol_runtime.md b/docs-archived/implplan/SPRINT_20260419_027_Concelier_durable_affected_symbol_runtime.md similarity index 77% rename from docs/implplan/SPRINT_20260419_027_Concelier_durable_affected_symbol_runtime.md rename to docs-archived/implplan/SPRINT_20260419_027_Concelier_durable_affected_symbol_runtime.md index 712a4edbc..62721cfcd 100644 --- a/docs/implplan/SPRINT_20260419_027_Concelier_durable_affected_symbol_runtime.md +++ b/docs-archived/implplan/SPRINT_20260419_027_Concelier_durable_affected_symbol_runtime.md @@ -22,7 +22,7 @@ ## Delivery Tracker ### REALPLAN-007-F - Durable advisory observation and affected-symbol runtime -Status: DOING +Status: DONE Dependency: REALPLAN-007-E Owners: Developer, Test Automation, Documentation author Task description: @@ -30,16 +30,17 @@ Task description: - The affected-symbol surface also has no durable backend or writer path. This sprint adds PostgreSQL-backed advisory-observation persistence plus a PostgreSQL-backed affected-symbol store, then hooks the real ingest path to extract and persist symbols from advisory observations so `/v1/signals/symbols/*` can stop returning truthful `501`. Completion criteria: -- [ ] `AddConcelierPostgresStorage` registers durable advisory-observation lookup/sink services and a durable affected-symbol store. -- [ ] The live raw-ingest path persists advisory observations and stores extracted affected symbols from those observations. -- [ ] Non-testing Concelier runtime resolves the durable affected-symbol services instead of `UnsupportedAffectedSymbol*`. -- [ ] Focused persistence/runtime proof covers observation persistence, affected-symbol persistence/query behavior, and ingest-time symbol extraction. -- [ ] Concelier docs and task boards describe the implemented durable runtime contract. +- [x] `AddConcelierPostgresStorage` registers durable advisory-observation lookup/sink services and a durable affected-symbol store. +- [x] The live raw-ingest path persists advisory observations and stores extracted affected symbols from those observations. +- [x] Non-testing Concelier runtime resolves the durable affected-symbol services instead of `UnsupportedAffectedSymbol*`. +- [x] Focused persistence/runtime proof covers observation persistence, affected-symbol persistence/query behavior, and ingest-time symbol extraction. +- [x] Concelier docs and task boards describe the implemented durable runtime contract. ## Execution Log | Date (UTC) | Update | Owner | | --- | --- | --- | | 2026-04-19 | Sprint created and set `DOING` after reviewing the truthful-runtime follow-up gap: the real raw-ingest path already emits `AdvisoryObservation`, but live persistence still uses a null observation sink/lookup and no durable affected-symbol backend exists. | Developer | +| 2026-04-19 | Shipped migration 008 (advisory_observations + affected_symbols), `PostgresAdvisoryObservationStore` (implements both lookup + sink and delegates to `IAffectedSymbolExtractor` during upsert), and `PostgresAffectedSymbolStore`. Wired via `AddConcelierPostgresStorage` in `ConcelierPersistenceExtensions` and registered before `AddConcelierObservationPipeline` so `TryAddSingleton` null-fallbacks do not override. Removed the non-testing `UnsupportedAffectedSymbolStore/Provider` registration in `Program.cs`. Targeted xUnit runs via `scripts/test-targeted-xunit.ps1`: `PostgresAdvisoryObservationStoreTests` 2/2 ✓, `PostgresAffectedSymbolStoreTests` 2/2 ✓, `ConcelierInfrastructureRegistrationTests` 3/3 ✓, `UnsupportedRuntimeWiringTests` 9/9 ✓. `docs/modules/concelier/architecture.md` updated to reflect the durable contract. | Codex | ## Decisions & Risks - Decision: include durable advisory-observation persistence in the same slice as affected-symbol durability because the live writer hook already materializes observations inside `AdvisoryRawService`; leaving that path null would keep the affected-symbol writer detached from the real runtime. diff --git a/docs/modules/concelier/architecture.md b/docs/modules/concelier/architecture.md index d85b1a3bc..e2cc35e34 100644 --- a/docs/modules/concelier/architecture.md +++ b/docs/modules/concelier/architecture.md @@ -109,7 +109,7 @@ Running the same export job twice against the same snapshot must yield byte-iden **Process shape:** single ASP.NET Core service `StellaOps.Concelier.WebService` hosting: -* **Scheduler** with distributed PostgreSQL leases backed by `vuln.job_leases`. Lease coordination is durable; job-run history and internal orchestrator registry state are not yet durably implemented in the live host. +* **Scheduler** with distributed PostgreSQL leases backed by `vuln.job_leases`. Lease coordination is durable; job-run history and internal orchestrator registry state are not yet durably implemented in the live host. * **Connectors** (fetch/parse/map) that emit immutable observation candidates. * **Observation writer** enforcing AOC invariants via `AOCWriteGuard`. * **Linkset builder** that correlates observations into `advisory_linksets` and annotates conflicts. @@ -362,11 +362,11 @@ Events are emitted via Valkey Streams. Consumers acknowledge idempotently using --- -## 7) Storage schema (PostgreSQL) - -Fresh blank databases must enable PostgreSQL extension prerequisites before Concelier's initial schema applies. In practice, Concelier now carries a dedicated pre-schema startup migration for `pg_trgm`, so trigram-backed GIN indexes converge on first boot without relying on external manual database prep. - -### Tables & indexes (LNM path) +## 7) Storage schema (PostgreSQL) + +Fresh blank databases must enable PostgreSQL extension prerequisites before Concelier's initial schema applies. In practice, Concelier now carries a dedicated pre-schema startup migration for `pg_trgm`, so trigram-backed GIN indexes converge on first boot without relying on external manual database prep. + +### Tables & indexes (LNM path) * `concelier.sources` `{_id, type, baseUrl, enabled, notes}` — connector catalog. * `concelier.source_state` `{sourceName(unique), enabled, cursor, lastSuccess, backoffUntil, paceOverrides}` — run-state (TTL indexes on `backoffUntil`). @@ -429,8 +429,8 @@ Fresh blank databases must enable PostgreSQL extension prerequisites before Conc * TTL index on `occurredAt` (configurable retention), `{type:1, occurredAt:-1}` for replay. * `concelier.export_state` `{_id(exportKind), baseExportId?, baseDigest?, lastFullDigest?, lastDeltaDigest?, cursor, files[]}` -* `job_leases` `{lease_key, holder, acquired_at, heartbeat_at, lease_ms, ttl_at}` used by live scheduler coordination; expired leases can be stolen safely by another runner. -* `jobs` `{_id, type, args, state, startedAt, heartbeatAt, endedAt, error}` is the planned durable owner for future job-run history. Current live runtime does not persist `/jobs` or `/internal/orch/*` state and returns `501` until a durable job/orchestrator registry backend lands. +* `job_leases` `{lease_key, holder, acquired_at, heartbeat_at, lease_ms, ttl_at}` used by live scheduler coordination; expired leases can be stolen safely by another runner. +* `jobs` `{_id, type, args, state, startedAt, heartbeatAt, endedAt, error}` is the planned durable owner for future job-run history. Current live runtime does not persist `/jobs` or `/internal/orch/*` state and returns `501` until a durable job/orchestrator registry backend lands. **Legacy tables** (`advisory`, `alias`, `affected`, `reference`, `merge_event`) remain read-only during the migration window to support back-compat exports. New code must not write to them; scheduled cleanup removes them after Link-Not-Merge GA. @@ -461,8 +461,8 @@ Fresh blank databases must enable PostgreSQL extension prerequisites before Conc ``` * Optional ORAS push (OCI layout) for registries. * Offline kit bundles include Trivy DB + JSON tree + export manifest. -* Mirror-ready bundles: when `concelier.trivy.mirror` defines domains, the exporter emits `mirror/index.json` plus per-domain `manifest.json`, `metadata.json`, and `db.tar.gz` files with SHA-256 digests so Concelier mirrors can expose domain-scoped download endpoints. -* Concelier.WebService serves `/concelier/exports/index.json` and `/concelier/exports/mirror/{domain}/…` directly from the export tree with hour-long budgets (index: 60 s, bundles: 300 s, immutable) and per-domain rate limiting. Live downloads now require persisted mirror-domain state from the mirror read model rather than env-seeded domain lists; `Testing` can still seed that state through the management API. +* Mirror-ready bundles: when `concelier.trivy.mirror` defines domains, the exporter emits `mirror/index.json` plus per-domain `manifest.json`, `metadata.json`, and `db.tar.gz` files with SHA-256 digests so Concelier mirrors can expose domain-scoped download endpoints. +* Concelier.WebService serves `/concelier/exports/index.json` and `/concelier/exports/mirror/{domain}/…` directly from the export tree with hour-long budgets (index: 60 s, bundles: 300 s, immutable) and per-domain rate limiting. Live downloads now require persisted mirror-domain state from the mirror read model rather than env-seeded domain lists; `Testing` can still seed that state through the management API. ### 7.3 Hand‑off to Signer/Attestor (optional) @@ -482,19 +482,19 @@ GET /healthz | /readyz GET /status → sources, last runs, export cursors ``` -**Sources & jobs** +**Sources & jobs** ``` GET /sources → list of configured sources POST /sources/{name}/trigger → { jobId } POST /sources/{name}/pause | /resume → toggle -GET /jobs/{id} → job status -``` - -Current runtime note: `/jobs`, `/internal/orch/*`, and the coordinator-backed manual sync compatibility routes (`/api/v1/advisory-sources/{sourceId}/sync`, `/api/v1/advisory-sources/sync`, `/api/v1/concelier/mirrors/{mirrorId}/sync`) are not durably implemented in the live host. Outside `Testing` they return explicit `501` responses rather than falling back to in-memory state. -Current signals note: `/v1/signals/symbols/*` is also not durably implemented in the live host. Outside `Testing` it returns explicit `501` responses rather than falling back to the process-local affected-symbol store. -Current advisory source note: the live host also exposes `/api/v1/advisory-sources/*` for operator/UI source status, enablement, health checks, and sync triggers. Enabled state is now persisted in the Concelier source store so restarts, setup skip/apply flows, and later integrations-page toggles all observe the same source truth. Platform setup uses bootstrap-key-protected `/internal/setup/advisory-sources/{probe,apply}` endpoints to seed initial source configuration without requiring a tenant session. -Mirror bootstrap truthfulness note: `/internal/setup/advisory-sources/{probe,apply}` now validates the selected mirror/source configuration before apply succeeds. Mirror mode probes `/concelier/exports/index.json` with normal runtime TLS rules, so certificate/hostname mismatches are returned as actionable setup failures instead of "background sync" warnings. +GET /jobs/{id} → job status +``` + +Current runtime note: `/jobs`, `/internal/orch/*`, and the coordinator-backed manual sync compatibility routes (`/api/v1/advisory-sources/{sourceId}/sync`, `/api/v1/advisory-sources/sync`, `/api/v1/concelier/mirrors/{mirrorId}/sync`) are not durably implemented in the live host. Outside `Testing` they return explicit `501` responses rather than falling back to in-memory state. +Current signals note: `/v1/signals/symbols/*` is backed by a durable PostgreSQL affected-symbol store (`PostgresAffectedSymbolStore`). Advisory observations are persisted via `PostgresAdvisoryObservationStore`, which invokes the registered `IAffectedSymbolExtractor` chain during `UpsertAsync` to project ingest-time symbols into the affected-symbol store. The legacy non-testing `501` fallback has been removed; in-memory implementations remain only for `Testing` environments. See `SPRINT_20260419_027_Concelier_durable_affected_symbol_runtime` for the cutover record. +Current advisory source note: the live host also exposes `/api/v1/advisory-sources/*` for operator/UI source status, enablement, health checks, and sync triggers. Enabled state is now persisted in the Concelier source store so restarts, setup skip/apply flows, and later integrations-page toggles all observe the same source truth. Platform setup uses bootstrap-key-protected `/internal/setup/advisory-sources/{probe,apply}` endpoints to seed initial source configuration without requiring a tenant session. +Mirror bootstrap truthfulness note: `/internal/setup/advisory-sources/{probe,apply}` now validates the selected mirror/source configuration before apply succeeds. Mirror mode probes `/concelier/exports/index.json` with normal runtime TLS rules, so certificate/hostname mismatches are returned as actionable setup failures instead of "background sync" warnings. **Exports** @@ -502,13 +502,13 @@ Mirror bootstrap truthfulness note: `/internal/setup/advisory-sources/{probe,app POST /exports/json { full?:bool, force?:bool, attest?:bool } → { exportId, digest, rekor? } POST /exports/trivy { full?:bool, force?:bool, publish?:bool, attest?:bool } → { exportId, digest, rekor? } GET /exports/{id} → export metadata (kind, digest, createdAt, rekor?) -GET /concelier/exports/index.json → mirror index describing available domains/bundles -GET /concelier/exports/mirror/{domain}/manifest.json -GET /concelier/exports/mirror/{domain}/bundle.json -GET /concelier/exports/mirror/{domain}/bundle.json.jws -``` - -Current mirror runtime note: the public mirror download surface no longer trusts `ConcelierOptions.Mirror.Domains` in live runtime. Outside `Testing`, index/download requests succeed only when the requested domain exists in the persisted mirror-domain store; config-only seeded domains are ignored. +GET /concelier/exports/index.json → mirror index describing available domains/bundles +GET /concelier/exports/mirror/{domain}/manifest.json +GET /concelier/exports/mirror/{domain}/bundle.json +GET /concelier/exports/mirror/{domain}/bundle.json.jws +``` + +Current mirror runtime note: the public mirror download surface no longer trusts `ConcelierOptions.Mirror.Domains` in live runtime. Outside `Testing`, index/download requests succeed only when the requested domain exists in the persisted mirror-domain store; config-only seeded domains are ignored. **Search (operator debugging)** @@ -518,7 +518,7 @@ GET /advisories?scheme=CVE&value=CVE-2025-12345 GET /affected?productKey=pkg:rpm/openssl&limit=100 ``` -**Mirror domain management** (under `/api/v1/advisory-sources/mirror`) +**Mirror domain management** (under `/api/v1/advisory-sources/mirror`) ``` GET /config → current mirror config (mode, signing, refresh interval) @@ -535,7 +535,7 @@ GET /domains/{domainId}/status → domain sync status (last gener POST /test → test mirror endpoint connectivity ``` -**Mirror consumer configuration** (under `/api/v1/advisory-sources/mirror`) +**Mirror consumer configuration** (under `/api/v1/advisory-sources/mirror`) ``` GET /consumer → current consumer connector configuration (base address, domain, signature, timeout, connection status, last sync) @@ -544,18 +544,18 @@ POST /consumer/discover → fetch mirror index from base a POST /consumer/verify-signature → fetch JWS header from selected domain's bundle, return detected algorithm, key ID, and provider ``` -The consumer endpoints configure the `StellaOpsMirrorConnector` at runtime without requiring service restarts. Configuration is persisted via `IMirrorConsumerConfigStore` in PostgreSQL. The `/consumer/discover` endpoint enables the UI setup wizard to present operators with a list of available domains before committing to a configuration. +The consumer endpoints configure the `StellaOpsMirrorConnector` at runtime without requiring service restarts. Configuration is persisted via `IMirrorConsumerConfigStore` in PostgreSQL. The `/consumer/discover` endpoint enables the UI setup wizard to present operators with a list of available domains before committing to a configuration. -**Air-gap bundle import** (under `/api/v1/advisory-sources/mirror`) - -``` -POST /import → import a mirror bundle from a local filesystem path { bundlePath, verifyChecksums, verifyDsse, trustRootsPath? } -GET /import/status → import progress and result (exports imported, total size, errors, warnings) -``` - -The current HTTP mirror import path is durably implemented in the live host. `/api/v1/advisory-sources/mirror/import` validates a local mirror bundle, persists import status, projects `manifest.json` and `bundle.json` into the live `/concelier/exports/mirror/` surface, and refreshes the public `index.json`. `/api/v1/advisory-sources/mirror/import/status` reads the latest persisted import status instead of fabricating progress from a filesystem inspection pass. Both `bundlePath` and `trustRootsPath` must resolve under the configured `Mirror.ImportRoot`; relative paths are resolved against that allowlisted root, and paths outside it are rejected. Detached JWS verification is supported when callers provide a `trustRootsPath`; unsigned bundles still import truthfully with a warning rather than a synthetic success banner. - -Mirror domains group export plans with shared rate limits and authentication rules. Exports support multi-value filter shorthands: `sourceCategory` (e.g., `"Distribution"` resolves to all distro sources), `sourceTag` (e.g., `"linux"`), and comma-separated `sourceVendor` values. Domain configuration is persisted in `excititor.mirror_domains` / `excititor.mirror_exports` tables. The `MirrorExportScheduler` background service periodically regenerates stale bundles (configurable via `RefreshIntervalMinutes`, default 60 minutes). +**Air-gap bundle import** (under `/api/v1/advisory-sources/mirror`) + +``` +POST /import → import a mirror bundle from a local filesystem path { bundlePath, verifyChecksums, verifyDsse, trustRootsPath? } +GET /import/status → import progress and result (exports imported, total size, errors, warnings) +``` + +The current HTTP mirror import path is durably implemented in the live host. `/api/v1/advisory-sources/mirror/import` validates a local mirror bundle, persists import status, projects `manifest.json` and `bundle.json` into the live `/concelier/exports/mirror/` surface, and refreshes the public `index.json`. `/api/v1/advisory-sources/mirror/import/status` reads the latest persisted import status instead of fabricating progress from a filesystem inspection pass. Both `bundlePath` and `trustRootsPath` must resolve under the configured `Mirror.ImportRoot`; relative paths are resolved against that allowlisted root, and paths outside it are rejected. Detached JWS verification is supported when callers provide a `trustRootsPath`; unsigned bundles still import truthfully with a warning rather than a synthetic success banner. + +Mirror domains group export plans with shared rate limits and authentication rules. Exports support multi-value filter shorthands: `sourceCategory` (e.g., `"Distribution"` resolves to all distro sources), `sourceTag` (e.g., `"linux"`), and comma-separated `sourceVendor` values. Domain configuration is persisted in `excititor.mirror_domains` / `excititor.mirror_exports` tables. The `MirrorExportScheduler` background service periodically regenerates stale bundles (configurable via `RefreshIntervalMinutes`, default 60 minutes). **AuthN/Z:** Authority tokens (OpTok) with roles: `concelier.read`, `concelier.admin`, `concelier.export`.