docs: module dossier + install/quickstart sync for truthful cutover sprints
- API_CLI_REFERENCE.md, INSTALL_GUIDE.md, quickstart.md, architecture/integrations.md, dev/DEV_ENVIRONMENT_SETUP.md, integrations/LOCAL_SERVICES.md: reflect real-service wiring. - docs/modules/**: module dossier updates across the modules touched by SPRINT_20260415_001..007 + SPRINT_20260416_003..017 + SPRINT_20260417_018..024 + SPRINT_20260418_025 + SPRINT_20260419_026. - docs/features/checked/web/**: update feature notes where UI changed. - docs/qa/feature-checks/runs/web/evidence-presentation-ux/: QA evidence artifacts. - docs/setup/**, docs/technical/**: align with setup wizard contracts. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
master
@@ -109,7 +109,7 @@ Running the same export job twice against the same snapshot must yield byte-iden
|
||||
|
||||
**Process shape:** single ASP.NET Core service `StellaOps.Concelier.WebService` hosting:
|
||||
|
||||
* **Scheduler** with distributed locks (PostgreSQL backed).
|
||||
* **Scheduler** with distributed PostgreSQL leases backed by `vuln.job_leases`. Lease coordination is durable; job-run history and internal orchestrator registry state are not yet durably implemented in the live host.
|
||||
* **Connectors** (fetch/parse/map) that emit immutable observation candidates.
|
||||
* **Observation writer** enforcing AOC invariants via `AOCWriteGuard`.
|
||||
* **Linkset builder** that correlates observations into `advisory_linksets` and annotates conflicts.
|
||||
@@ -362,9 +362,11 @@ Events are emitted via Valkey Streams. Consumers acknowledge idempotently using
|
||||
|
||||
---
|
||||
|
||||
## 7) Storage schema (PostgreSQL)
|
||||
|
||||
### Tables & indexes (LNM path)
|
||||
## 7) Storage schema (PostgreSQL)
|
||||
|
||||
Fresh blank databases must enable PostgreSQL extension prerequisites before Concelier's initial schema applies. In practice, Concelier now carries a dedicated pre-schema startup migration for `pg_trgm`, so trigram-backed GIN indexes converge on first boot without relying on external manual database prep.
|
||||
|
||||
### Tables & indexes (LNM path)
|
||||
|
||||
* `concelier.sources` `{_id, type, baseUrl, enabled, notes}` — connector catalog.
|
||||
* `concelier.source_state` `{sourceName(unique), enabled, cursor, lastSuccess, backoffUntil, paceOverrides}` — run-state (TTL indexes on `backoffUntil`).
|
||||
@@ -427,8 +429,8 @@ Events are emitted via Valkey Streams. Consumers acknowledge idempotently using
|
||||
|
||||
* TTL index on `occurredAt` (configurable retention), `{type:1, occurredAt:-1}` for replay.
|
||||
* `concelier.export_state` `{_id(exportKind), baseExportId?, baseDigest?, lastFullDigest?, lastDeltaDigest?, cursor, files[]}`
|
||||
* `locks` `{_id(jobKey), holder, acquiredAt, heartbeatAt, leaseMs, ttlAt}` (TTL cleans dead locks)
|
||||
* `jobs` `{_id, type, args, state, startedAt, heartbeatAt, endedAt, error}`
|
||||
* `job_leases` `{lease_key, holder, acquired_at, heartbeat_at, lease_ms, ttl_at}` used by live scheduler coordination; expired leases can be stolen safely by another runner.
|
||||
* `jobs` `{_id, type, args, state, startedAt, heartbeatAt, endedAt, error}` is the planned durable owner for future job-run history. Current live runtime does not persist `/jobs` or `/internal/orch/*` state and returns `501` until a durable job/orchestrator registry backend lands.
|
||||
|
||||
**Legacy tables** (`advisory`, `alias`, `affected`, `reference`, `merge_event`) remain read-only during the migration window to support back-compat exports. New code must not write to them; scheduled cleanup removes them after Link-Not-Merge GA.
|
||||
|
||||
@@ -459,8 +461,8 @@ Events are emitted via Valkey Streams. Consumers acknowledge idempotently using
|
||||
```
|
||||
* Optional ORAS push (OCI layout) for registries.
|
||||
* Offline kit bundles include Trivy DB + JSON tree + export manifest.
|
||||
* Mirror-ready bundles: when `concelier.trivy.mirror` defines domains, the exporter emits `mirror/index.json` plus per-domain `manifest.json`, `metadata.json`, and `db.tar.gz` files with SHA-256 digests so Concelier mirrors can expose domain-scoped download endpoints.
|
||||
* Concelier.WebService serves `/concelier/exports/index.json` and `/concelier/exports/mirror/{domain}/…` directly from the export tree with hour-long budgets (index: 60 s, bundles: 300 s, immutable) and per-domain rate limiting; the endpoints honour Stella Ops Authority or CIDR bypass lists depending on mirror topology.
|
||||
* Mirror-ready bundles: when `concelier.trivy.mirror` defines domains, the exporter emits `mirror/index.json` plus per-domain `manifest.json`, `metadata.json`, and `db.tar.gz` files with SHA-256 digests so Concelier mirrors can expose domain-scoped download endpoints.
|
||||
* Concelier.WebService serves `/concelier/exports/index.json` and `/concelier/exports/mirror/{domain}/…` directly from the export tree with hour-long budgets (index: 60 s, bundles: 300 s, immutable) and per-domain rate limiting. Live downloads now require persisted mirror-domain state from the mirror read model rather than env-seeded domain lists; `Testing` can still seed that state through the management API.
|
||||
|
||||
### 7.3 Hand‑off to Signer/Attestor (optional)
|
||||
|
||||
@@ -480,14 +482,19 @@ GET /healthz | /readyz
|
||||
GET /status → sources, last runs, export cursors
|
||||
```
|
||||
|
||||
**Sources & jobs**
|
||||
**Sources & jobs**
|
||||
|
||||
```
|
||||
GET /sources → list of configured sources
|
||||
POST /sources/{name}/trigger → { jobId }
|
||||
POST /sources/{name}/pause | /resume → toggle
|
||||
GET /jobs/{id} → job status
|
||||
```
|
||||
GET /jobs/{id} → job status
|
||||
```
|
||||
|
||||
Current runtime note: `/jobs`, `/internal/orch/*`, and the coordinator-backed manual sync compatibility routes (`/api/v1/advisory-sources/{sourceId}/sync`, `/api/v1/advisory-sources/sync`, `/api/v1/concelier/mirrors/{mirrorId}/sync`) are not durably implemented in the live host. Outside `Testing` they return explicit `501` responses rather than falling back to in-memory state.
|
||||
Current signals note: `/v1/signals/symbols/*` is also not durably implemented in the live host. Outside `Testing` it returns explicit `501` responses rather than falling back to the process-local affected-symbol store.
|
||||
Current advisory source note: the live host also exposes `/api/v1/advisory-sources/*` for operator/UI source status, enablement, health checks, and sync triggers. Enabled state is now persisted in the Concelier source store so restarts, setup skip/apply flows, and later integrations-page toggles all observe the same source truth. Platform setup uses bootstrap-key-protected `/internal/setup/advisory-sources/{probe,apply}` endpoints to seed initial source configuration without requiring a tenant session.
|
||||
Mirror bootstrap truthfulness note: `/internal/setup/advisory-sources/{probe,apply}` now validates the selected mirror/source configuration before apply succeeds. Mirror mode probes `/concelier/exports/index.json` with normal runtime TLS rules, so certificate/hostname mismatches are returned as actionable setup failures instead of "background sync" warnings.
|
||||
|
||||
**Exports**
|
||||
|
||||
@@ -495,11 +502,13 @@ GET /jobs/{id} → job status
|
||||
POST /exports/json { full?:bool, force?:bool, attest?:bool } → { exportId, digest, rekor? }
|
||||
POST /exports/trivy { full?:bool, force?:bool, publish?:bool, attest?:bool } → { exportId, digest, rekor? }
|
||||
GET /exports/{id} → export metadata (kind, digest, createdAt, rekor?)
|
||||
GET /concelier/exports/index.json → mirror index describing available domains/bundles
|
||||
GET /concelier/exports/mirror/{domain}/manifest.json
|
||||
GET /concelier/exports/mirror/{domain}/bundle.json
|
||||
GET /concelier/exports/mirror/{domain}/bundle.json.jws
|
||||
```
|
||||
GET /concelier/exports/index.json → mirror index describing available domains/bundles
|
||||
GET /concelier/exports/mirror/{domain}/manifest.json
|
||||
GET /concelier/exports/mirror/{domain}/bundle.json
|
||||
GET /concelier/exports/mirror/{domain}/bundle.json.jws
|
||||
```
|
||||
|
||||
Current mirror runtime note: the public mirror download surface no longer trusts `ConcelierOptions.Mirror.Domains` in live runtime. Outside `Testing`, index/download requests succeed only when the requested domain exists in the persisted mirror-domain store; config-only seeded domains are ignored.
|
||||
|
||||
**Search (operator debugging)**
|
||||
|
||||
@@ -509,7 +518,7 @@ GET /advisories?scheme=CVE&value=CVE-2025-12345
|
||||
GET /affected?productKey=pkg:rpm/openssl&limit=100
|
||||
```
|
||||
|
||||
**Mirror domain management** (under `/api/v1/mirror`)
|
||||
**Mirror domain management** (under `/api/v1/advisory-sources/mirror`)
|
||||
|
||||
```
|
||||
GET /config → current mirror config (mode, signing, refresh interval)
|
||||
@@ -526,7 +535,7 @@ GET /domains/{domainId}/status → domain sync status (last gener
|
||||
POST /test → test mirror endpoint connectivity
|
||||
```
|
||||
|
||||
**Mirror consumer configuration** (under `/api/v1/mirror`)
|
||||
**Mirror consumer configuration** (under `/api/v1/advisory-sources/mirror`)
|
||||
|
||||
```
|
||||
GET /consumer → current consumer connector configuration (base address, domain, signature, timeout, connection status, last sync)
|
||||
@@ -535,18 +544,18 @@ POST /consumer/discover → fetch mirror index from base a
|
||||
POST /consumer/verify-signature → fetch JWS header from selected domain's bundle, return detected algorithm, key ID, and provider
|
||||
```
|
||||
|
||||
The consumer endpoints configure the `StellaOpsMirrorConnector` at runtime without requiring service restarts. Configuration is persisted via `IMirrorConsumerConfigStore` (in-memory, with planned DB backend). The `/consumer/discover` endpoint enables the UI setup wizard to present operators with a list of available domains before committing to a configuration.
|
||||
The consumer endpoints configure the `StellaOpsMirrorConnector` at runtime without requiring service restarts. Configuration is persisted via `IMirrorConsumerConfigStore` in PostgreSQL. The `/consumer/discover` endpoint enables the UI setup wizard to present operators with a list of available domains before committing to a configuration.
|
||||
|
||||
**Air-gap bundle import** (under `/api/v1/mirror`)
|
||||
|
||||
```
|
||||
POST /import → import a mirror bundle from a local filesystem path { bundlePath, verifyChecksums, verifyDsse, trustRootsPath? }
|
||||
GET /import/status → import progress and result (exports imported, total size, errors, warnings)
|
||||
```
|
||||
|
||||
The import endpoint triggers an async import of a mirror bundle directory accessible to the Concelier container. It parses the bundle manifest, verifies SHA-256 checksums (when `verifyChecksums` is true), detects DSSE envelopes (when `verifyDsse` is true), and copies artifacts into the local data store. Import state is tracked by `IMirrorBundleImportStore`. This exposes the same functionality as the CLI `MirrorBundleImportService` via HTTP.
|
||||
|
||||
Mirror domains group export plans with shared rate limits and authentication rules. Exports support multi-value filter shorthands: `sourceCategory` (e.g., `"Distribution"` resolves to all distro sources), `sourceTag` (e.g., `"linux"`), and comma-separated `sourceVendor` values. Domain configuration is persisted in `excititor.mirror_domains` / `excititor.mirror_exports` tables, with env-var config as fallback. The `MirrorExportScheduler` background service periodically regenerates stale bundles (configurable via `RefreshIntervalMinutes`, default 60 minutes).
|
||||
**Air-gap bundle import** (under `/api/v1/advisory-sources/mirror`)
|
||||
|
||||
```
|
||||
POST /import → import a mirror bundle from a local filesystem path { bundlePath, verifyChecksums, verifyDsse, trustRootsPath? }
|
||||
GET /import/status → import progress and result (exports imported, total size, errors, warnings)
|
||||
```
|
||||
|
||||
The current HTTP mirror import path is durably implemented in the live host. `/api/v1/advisory-sources/mirror/import` validates a local mirror bundle, persists import status, projects `manifest.json` and `bundle.json` into the live `/concelier/exports/mirror/<domain>` surface, and refreshes the public `index.json`. `/api/v1/advisory-sources/mirror/import/status` reads the latest persisted import status instead of fabricating progress from a filesystem inspection pass. Both `bundlePath` and `trustRootsPath` must resolve under the configured `Mirror.ImportRoot`; relative paths are resolved against that allowlisted root, and paths outside it are rejected. Detached JWS verification is supported when callers provide a `trustRootsPath`; unsigned bundles still import truthfully with a warning rather than a synthetic success banner.
|
||||
|
||||
Mirror domains group export plans with shared rate limits and authentication rules. Exports support multi-value filter shorthands: `sourceCategory` (e.g., `"Distribution"` resolves to all distro sources), `sourceTag` (e.g., `"linux"`), and comma-separated `sourceVendor` values. Domain configuration is persisted in `excititor.mirror_domains` / `excititor.mirror_exports` tables. The `MirrorExportScheduler` background service periodically regenerates stale bundles (configurable via `RefreshIntervalMinutes`, default 60 minutes).
|
||||
|
||||
**AuthN/Z:** Authority tokens (OpTok) with roles: `concelier.read`, `concelier.admin`, `concelier.export`.
|
||||
|
||||
|
||||
Reference in New Issue
Block a user