feat(concelier): multi-sprint batch (mirror domain + advisory sources + durable runtime + credentials)
Bundled commit covering pre-session work from multiple Concelier sprints already archived or in-flight: - SPRINT_20260419_006: mirror domain / source key validation - SPRINT_20260419_029 / 030: durable jobs orchestrator runtime + endpoint verification - SPRINT_20260421_001: advisory source projection truthful counts - SPRINT_20260421_002: FE advisory source consistency (connector-side bits) - SPRINT_20260421_003: advisory connector runtime alignment - SPRINT_20260422_003: source credential entry paths (in-flight) Includes connector internals (ACSC / Adobe / CERT-BUND / Chromium / Cisco / CVE-KEV / GHSA / JVN / KISA / MSRC / Oracle / Ubuntu), source management endpoints, mirror domain management, federation endpoints, topology setup, job registration, and associated dossier updates under docs/modules/concelier/. This commit groups ~229 file changes that accumulated across the above sprints; individual changes are preserved at file granularity so blame remains useful. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
master
@@ -109,7 +109,7 @@ Running the same export job twice against the same snapshot must yield byte-iden
|
||||
|
||||
**Process shape:** single ASP.NET Core service `StellaOps.Concelier.WebService` hosting:
|
||||
|
||||
* **Scheduler** with distributed PostgreSQL leases backed by `vuln.job_leases`. Lease coordination is durable; job-run history and internal orchestrator registry state are not yet durably implemented in the live host.
|
||||
* **Scheduler** with distributed PostgreSQL leases backed by `vuln.job_leases`, durable job-run history in `vuln.job_runs`, and durable internal orchestrator runtime state in `vuln.orchestrator_registry`, `vuln.orchestrator_heartbeats`, `vuln.orchestrator_commands`, and `vuln.orchestrator_manifests`.
|
||||
* **Connectors** (fetch/parse/map) that emit immutable observation candidates.
|
||||
* **Observation writer** enforcing AOC invariants via `AOCWriteGuard`.
|
||||
* **Linkset builder** that correlates observations into `advisory_linksets` and annotates conflicts.
|
||||
@@ -240,12 +240,14 @@ Legacy `Advisory`, `Affected`, and merge-centric entities remain in the reposito
|
||||
|
||||
## 4) Source families & precedence
|
||||
|
||||
The source catalog contains **75 definitions** across **14 categories**. The authoritative definition lives in `src/Concelier/__Libraries/StellaOps.Concelier.Core/Sources/SourceDefinitions.cs`; for the full connector index see `docs/modules/concelier/connectors.md`.
|
||||
The source catalog contains **78 definitions** across **14 categories**. The authoritative definition lives in `src/Concelier/__Libraries/StellaOps.Concelier.Core/Sources/SourceDefinitions.cs`; for the full connector index see `docs/modules/concelier/connectors.md`.
|
||||
|
||||
Runtime surfaces intentionally distinguish between catalog breadth and runnable support. `/api/v1/advisory-sources/catalog` and `/api/v1/advisory-sources/status` expose `syncSupported` plus the resolved `fetchJobKind`, and only sources with a registered fetch pipeline can be enabled or synced through the live host. Operators should use the canonical runtime IDs from the catalog, including `jpcert`, `auscert`, `krcert`, `cert-de`, `adobe`, and `chromium`; legacy aliases such as `jvn`, `acsc`, `kisa`, `cert-bund`, `vndr-adobe`, and `vndr-chromium` remain compatibility fallbacks for configuration binding and source normalization only.
|
||||
|
||||
### 3.1 Families
|
||||
|
||||
* **Primary databases**: NVD, OSV, GHSA, CVE.org (MITRE).
|
||||
* **Vendor PSIRTs**: Microsoft, Oracle, Cisco, Apple, VMware, Fortinet, Juniper, Palo Alto, plus cloud providers (AWS, Azure, GCP).
|
||||
* **Vendor PSIRTs**: Red Hat, Microsoft, Oracle, Adobe, Apple, Chromium, Cisco, VMware, Fortinet, Juniper, Palo Alto, plus cloud providers (AWS, Azure, GCP).
|
||||
* **Linux distros**: Debian, Ubuntu, Alpine, SUSE, RHEL, CentOS, Fedora, Arch, Gentoo, Astra Linux.
|
||||
* **OSS ecosystems**: npm, PyPI, Go, RubyGems, NuGet, Maven, Crates.io, Packagist, Hex.pm.
|
||||
* **Package manager native**: RustSec (cargo-audit), PyPA (pip-audit), Go Vuln DB (govulncheck), Ruby Advisory DB (bundler-audit).
|
||||
@@ -429,8 +431,12 @@ Fresh blank databases must enable PostgreSQL extension prerequisites before Conc
|
||||
|
||||
* TTL index on `occurredAt` (configurable retention), `{type:1, occurredAt:-1}` for replay.
|
||||
* `concelier.export_state` `{_id(exportKind), baseExportId?, baseDigest?, lastFullDigest?, lastDeltaDigest?, cursor, files[]}`
|
||||
* `job_leases` `{lease_key, holder, acquired_at, heartbeat_at, lease_ms, ttl_at}` used by live scheduler coordination; expired leases can be stolen safely by another runner.
|
||||
* `jobs` `{_id, type, args, state, startedAt, heartbeatAt, endedAt, error}` is the planned durable owner for future job-run history. Current live runtime does not persist `/jobs` or `/internal/orch/*` state and returns `501` until a durable job/orchestrator registry backend lands.
|
||||
* `job_leases` `{lease_key, holder, acquired_at, heartbeat_at, lease_ms, ttl_at}` used by live scheduler coordination; expired leases can be stolen safely by another runner.
|
||||
* `job_runs` `{run_id, kind, status, created_at, started_at, completed_at, trigger, parameters_hash, error, timeout_ms, lease_duration_ms, parameters_json}` stores durable scheduler history surfaced by `/jobs`.
|
||||
* `orchestrator_registry` `{tenant_id, connector_id, source, capabilities[], auth_ref, schedule_json, rate_policy_json, artifact_kinds[], lock_key, egress_guard_json, created_at, updated_at}` stores durable connector runtime registrations.
|
||||
* `orchestrator_heartbeats` `{tenant_id, connector_id, run_id, sequence, status, progress, queue_depth, last_artifact_hash, last_artifact_kind, error_code, retry_after_seconds, timestamp_utc}` stores append-only heartbeat history for `/internal/orch/heartbeat`.
|
||||
* `orchestrator_commands` `{tenant_id, connector_id, run_id, sequence, command, throttle_json, backfill_json, created_at, expires_at}` stores pending internal orchestrator commands with expiry-aware reads.
|
||||
* `orchestrator_manifests` `{tenant_id, connector_id, run_id, cursor_range_json, artifact_hashes[], dsse_envelope_hash, completed_at}` stores durable completed-run manifests for internal orchestrator consumers.
|
||||
|
||||
**Legacy tables** (`advisory`, `alias`, `affected`, `reference`, `merge_event`) remain read-only during the migration window to support back-compat exports. New code must not write to them; scheduled cleanup removes them after Link-Not-Merge GA.
|
||||
|
||||
@@ -491,9 +497,17 @@ POST /sources/{name}/pause | /resume → toggle
|
||||
GET /jobs/{id} → job status
|
||||
```
|
||||
|
||||
Current runtime note: `/jobs`, `/internal/orch/*`, and the coordinator-backed manual sync compatibility routes (`/api/v1/advisory-sources/{sourceId}/sync`, `/api/v1/advisory-sources/sync`, `/api/v1/concelier/mirrors/{mirrorId}/sync`) are not durably implemented in the live host. Outside `Testing` they return explicit `501` responses rather than falling back to in-memory state.
|
||||
Current runtime note: `/jobs`, `/internal/orch/*`, and the coordinator-backed manual sync compatibility routes (`/api/v1/advisory-sources/{sourceId}/sync`, `/api/v1/advisory-sources/sync`, `/api/v1/concelier/mirrors/{mirrorId}/sync`) now run against durable PostgreSQL-backed runtime services (`PostgresJobStore`, `JobCoordinator`, and `PostgresOrchestratorRegistryStore`) in the live host. Process-local placeholder implementations remain limited to `Testing` harnesses. Restart-safe endpoint verification now covers persisted job runs, orchestrator registry records, heartbeats, and queued commands through the live HTTP surface, and the obsolete `UnsupportedJobCoordinator` / `UnsupportedOrchestratorRegistryStore` live-host guards have been removed. See `SPRINT_20260419_029_Concelier_durable_jobs_orchestrator_runtime` for the cutover record and `SPRINT_20260419_030_Concelier_durable_runtime_endpoint_verification` for the endpoint proof.
|
||||
Current signals note: `/v1/signals/symbols/*` is backed by a durable PostgreSQL affected-symbol store (`PostgresAffectedSymbolStore`). Advisory observations are persisted via `PostgresAdvisoryObservationStore`, which invokes the registered `IAffectedSymbolExtractor` chain during `UpsertAsync` to project ingest-time symbols into the affected-symbol store. The legacy non-testing `501` fallback has been removed; in-memory implementations remain only for `Testing` environments. See `SPRINT_20260419_027_Concelier_durable_affected_symbol_runtime` for the cutover record.
|
||||
Current advisory source note: the live host also exposes `/api/v1/advisory-sources/*` for operator/UI source status, enablement, health checks, and sync triggers. Enabled state is now persisted in the Concelier source store so restarts, setup skip/apply flows, and later integrations-page toggles all observe the same source truth. Platform setup uses bootstrap-key-protected `/internal/setup/advisory-sources/{probe,apply}` endpoints to seed initial source configuration without requiring a tenant session.
|
||||
Current advisory source note: the live host also exposes `/api/v1/advisory-sources/*` for operator/UI source status, enablement, health checks, sync triggers, and visible review counts. Enabled state is now persisted in the Concelier source store so restarts, setup skip/apply flows, and later integrations-page toggles all observe the same source truth. Advisory totals and signature rollups are projected from distinct `source_advisory_id` values in `vuln.advisory_source_edge`, so the freshness surface reflects the canonical ingest path rather than the legacy `vuln.advisories.source_id` compatibility rows. Platform setup uses bootstrap-key-protected `/internal/setup/advisory-sources/{probe,apply}` endpoints to seed initial source configuration without requiring a tenant session.
|
||||
|
||||
The operator-facing review counters exposed by `/api/v1/advisory-sources` now have fixed semantics:
|
||||
- `sourceDocumentCount` and the backward-compatible `totalAdvisories` alias are both the count of distinct upstream `source_advisory_id` values linked to the source.
|
||||
- `canonicalAdvisoryCount` is the count of distinct canonical advisory ids reached through `vuln.advisory_source_edge`.
|
||||
- `cveCount` is the count of distinct non-empty canonical CVE ids associated with those linked advisories.
|
||||
- `vexDocumentCount` is the count of distinct linked source advisory documents carrying non-null vendor-status/VEX statements.
|
||||
|
||||
Web operators should derive visible summary cards and per-source detail panels from this same advisory-source contract so Integrations, Security, and setup review screens do not drift in headline counts or terminology.
|
||||
Mirror bootstrap truthfulness note: `/internal/setup/advisory-sources/{probe,apply}` now validates the selected mirror/source configuration before apply succeeds. Mirror mode probes `/concelier/exports/index.json` with normal runtime TLS rules, so certificate/hostname mismatches are returned as actionable setup failures instead of "background sync" warnings.
|
||||
|
||||
**Exports**
|
||||
@@ -533,9 +547,11 @@ DELETE /domains/{domainId}/exports/{exportKey} → remove an export from a domai
|
||||
POST /domains/{domainId}/generate → trigger on-demand bundle generation
|
||||
GET /domains/{domainId}/status → domain sync status (last generate, staleness)
|
||||
POST /test → test mirror endpoint connectivity
|
||||
```
|
||||
|
||||
**Mirror consumer configuration** (under `/api/v1/advisory-sources/mirror`)
|
||||
```
|
||||
|
||||
Mirror domain `sourceIds` are validated against the registered advisory source catalog exposed by `/api/v1/advisory-sources/catalog`. Operators can create domains from catalog keys such as `nvd` and `osv` even on a fresh database before any source-status workflow has persisted rows into `vuln.sources`; unknown keys are still rejected.
|
||||
|
||||
**Mirror consumer configuration** (under `/api/v1/advisory-sources/mirror`)
|
||||
|
||||
```
|
||||
GET /consumer → current consumer connector configuration (base address, domain, signature, timeout, connection status, last sync)
|
||||
|
||||
Reference in New Issue
Block a user