chore(docs+devops): cross-module doc sync + sprint archival moves + compose updates
Bundled pre-session doc + ops work: - docs/modules/**: sync across advisory-ai, airgap, cli, excititor, export-center, findings-ledger, notifier, notify, platform, router, sbom-service, ui, web (architectural + operational updates) - docs/features/**: updates to checked excititor vex pipeline, developer workspace, quick verify drawer - docs top-level: README, quickstart, API_CLI_REFERENCE, UI_GUIDE, code-of-conduct/TESTING_PRACTICES updates - docs/qa/feature-checks/: FLOW.md + excititor state update - docs/implplan/: remaining sprint updates + new Concelier source credentials sprint (SPRINT_20260422_003) - docs-archived/implplan/: 30 sprint archival moves (ElkSharp series, misc completed sprints) - devops/compose: .env + services compose + env example + router gateway config updates File-level granularity preserved. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -120,6 +120,8 @@ These tuples allow VEX Lens to compute deterministic consensus without re-parsin
|
||||
|
||||
Excititor workers now hydrate signature metadata with issuer trust data retrieved from the Issuer Directory service. The worker-side IssuerDirectoryClient performs tenant-aware lookups (including global fallbacks) and caches responses offline so attestation verification exposes an effective trust weight alongside the cryptographic details captured on ingest.
|
||||
|
||||
For the live WebService/runtime path, enabled VEX signature verification now requires a real `VexSignatureVerification:IssuerDirectory:ServiceUrl`. The default dependency-injection wiring no longer seeds `InMemoryIssuerDirectoryClient` or a process-local verification cache when verification is enabled. Operators may optionally enable `VexSignatureVerification:Valkey:*` to persist verification results in Valkey via `StackExchange.Redis`; if that configuration is absent, verification executes without a cache instead of silently falling back to in-memory state.
|
||||
|
||||
### 1.4 AI-ready citations
|
||||
|
||||
`GET /v1/vex/statements/{advisory_key}` produces sorted JSON responses containing raw statement metadata (`issuer`, `content_hash`, `signature`), normalised tuples, and provenance pointers. Advisory AI consumes this endpoint to build retrieval contexts with explicit citations.
|
||||
@@ -160,6 +162,7 @@ Schema: `vex`
|
||||
- **Observations/linksets** - use the append-only Postgres linkset schema already defined for `IAppendOnlyLinksetStore` (tables `vex_linksets`, `vex_linkset_observations`, `vex_linkset_disagreements`, `vex_linkset_mutations`) with indexes on `(tenant, vulnerability_id, product_key)` and `updated_at`.
|
||||
- **Claims** - `vex.claims` stores normalized, queryable claim projections keyed by deterministic `claim_hash`, with JSONB columns for product/document metadata plus indexes on `(tenant, provider_id, vulnerability_id, product_key, last_seen)` and `(tenant, vulnerability_id, last_seen)`.
|
||||
- **Attestations** - `vex.attestations` stores durable DSSE/VEX attestation envelopes keyed by `(tenant, attestation_id)` with manifest lookup and attested-at indexes. Startup migration `005_vex_attestations.sql` creates the table in the active runtime schema so isolated test schemas and the default `vex` schema use the same contract.
|
||||
- **Evidence links** - `vex.evidence_links` stores durable VEX-to-evidence associations keyed by deterministic `link_id`, with the evidence envelope metadata in JSONB plus lookup indexes on `vex_entry_id` and `envelope_digest`. Live runtime wiring resolves `IVexEvidenceLinkStore` from `StellaOps.Excititor.Persistence`; `AddVexEvidenceLinking(...)` no longer injects an in-memory fallback when no real Excititor PostgreSQL connection is configured.
|
||||
- **Graph overlays** - materialized cache table `vex.graph_overlays` (tenant, purl, advisory_id, source) storing JSONB payloads that follow `docs/modules/excititor/schemas/vex_overlay.schema.json` (schemaVersion 1.0.0). Live WebService runtime resolves `IGraphOverlayStore` to the Postgres-backed store; there is no in-memory production fallback.
|
||||
|
||||
**Canonicalisation & hashing**
|
||||
@@ -179,9 +182,10 @@ List/query `/vex/raw` via `SELECT ... FROM vex.vex_raw_documents WHERE tenant=@t
|
||||
|
||||
**Runtime convergence**
|
||||
|
||||
1. `StellaOps.Excititor.WebService` and `StellaOps.Excititor.Worker` resolve `IVexProviderStore`, `IVexConnectorStateRepository`, `IVexClaimStore`, `IVexAttestationStore`, and the WebService graph overlay store from persisted services; the live hosts do not register in-memory fallbacks for these paths.
|
||||
2. `StellaOps.Excititor.Persistence` owns startup migrations for the active runtime schema, including claims, attestation storage, graph overlays, and cleanup of historical demo rows from older local installs.
|
||||
1. `StellaOps.Excititor.WebService`, `StellaOps.Excititor.Worker`, and CLI hosts that enable VEX evidence linking resolve `IVexProviderStore`, `IVexConnectorStateRepository`, `IVexClaimStore`, `IVexAttestationStore`, `IVexEvidenceLinkStore`, and the WebService graph overlay store from persisted services; the live hosts do not register in-memory fallbacks for these paths.
|
||||
2. `StellaOps.Excititor.Persistence` owns startup migrations for the active runtime schema, including providers, observations, observation timeline events, checkpoint state, statements, deltas, claims, attestation storage, graph overlays, and cleanup of historical demo rows from older local installs.
|
||||
3. The Excititor migration assembly embeds only active top-level SQL files. Archived pre-1.0 scripts and demo-seed SQL are excluded so startup/test migration loaders do not replay historical or fake runtime state.
|
||||
4. Worker completion metadata (`LastSuccessAt`, heartbeat fields, checkpoint hashes) must not overwrite connector-managed cursor fields. Connectors own `LastUpdated`, digests, and resume tokens because those values may reflect upstream timestamps or fallback-path checkpoints rather than the wall-clock job completion time.
|
||||
|
||||
---
|
||||
|
||||
@@ -866,6 +870,13 @@ GET /providers/{id}/status → last fetch, doc counts, signature stats
|
||||
|
||||
---
|
||||
|
||||
**Runtime defaults (local mirror / compose):**
|
||||
|
||||
- When `Excititor:Worker:Providers` is empty, the worker seeds the public built-in providers `excititor:redhat`, `excititor:ubuntu`, `excititor:oracle`, and `excititor:cisco`.
|
||||
- Initial delays are intentionally staggered (`00:05:00`, `00:07:00`, `00:09:00`, `00:11:00`) so a fresh mirror does not burst all upstream VEX sources at once.
|
||||
- When Red Hat/Cisco metadata discovery or the Ubuntu root index is unavailable and no offline snapshot exists, the public bootstrap path falls back to deterministic built-in metadata/catalog inference so mirror startup does not fail solely because the discovery document is missing.
|
||||
- `excititor:msrc`, `excititor:suse:ranchervexhub`, and `excititor:oci:openvex:attest` remain opt-in defaults because they require tighter allowlisting, image-scoped configuration, or credential-sensitive setup.
|
||||
|
||||
## 9) Attestation integration
|
||||
|
||||
* Exports can be **DSSE‑signed** via **Signer** and logged to **Rekor v2** via **Attestor** (optional but recommended for regulated pipelines).
|
||||
@@ -936,6 +947,8 @@ With storage configured, the WebService exposes the following ingress and diagno
|
||||
* `POST /v1/attestations/verify` – verifies Evidence Locker attestations for exports/chunks using `IVexAttestationVerifier`; returns `{ valid, diagnostics }` (deterministic key order). Aligns with Evidence Locker contract v1.
|
||||
* `POST /excititor/resolve` – requires `vex.read` scope; accepts up to 256 `(vulnId, productKey)` pairs via `productKeys` or `purls` and returns deterministic consensus results, decision telemetry, and a signed envelope (`artifact` digest, optional signer signature, optional attestation metadata + DSSE envelope). Returns **409 Conflict** when the requested `policyRevisionId` mismatches the active snapshot.
|
||||
|
||||
For the standard compose/local mirror environment, the WebService may derive `Authority:ResourceServer:Authority` and the OpenID metadata address from `Excititor:Authority:BaseUrls:default` when the explicit `Authority:ResourceServer:*` keys are absent. This keeps `/excititor/status` and other Authority-backed routes operational in the default stack without duplicating authority configuration in two places.
|
||||
|
||||
Run the ingestion endpoint once after applying migration `20251019-consensus-signals-statements` to repopulate historical statements with the new severity/KEV/EPSS signal fields.
|
||||
|
||||
* `weights.ceiling` raises the deterministic clamp applied to provider tiers/overrides (range 1.0‒5.0). Values outside the range are clamped with warnings so operators can spot typos.
|
||||
|
||||
@@ -8,7 +8,8 @@ Provide a living plan for Excititor deliverables, dependencies, and evidence.
|
||||
- Update this file when new scoped work is approved.
|
||||
|
||||
## Near-term deliverables
|
||||
- TBD (add when sprint is staffed).
|
||||
- `SPRINT_20260420_001_Concelier_excititor_verification_runtime_no_stub_fallbacks.md`: retire the live Excititor verification fallback path so enabled signature verification requires a real IssuerDirectory URL and only uses Valkey when explicitly configured.
|
||||
- `SPRINT_20260420_002_Cli_evidence_linking_no_inmemory_store.md`: remove the CLI/runtime in-memory `IVexEvidenceLinkStore` fallback so `stella vex gen --link-evidence` only runs when real Excititor PostgreSQL persistence is configured.
|
||||
|
||||
## Dependencies
|
||||
- `docs/modules/excititor/architecture.md`
|
||||
|
||||
@@ -14,6 +14,7 @@
|
||||
| `...:Channels` | `["stable"]` | List of channel names to poll. Order preserved for deterministic cursoring. |
|
||||
| `...:MetadataCacheDuration` | `4h` | How long to cache catalog metadata before re-fetching. |
|
||||
| `...:PreferOfflineSnapshot` / `OfflineSnapshotPath` / `PersistOfflineSnapshot` | `false` / `null` / `true` | Enable when running from Offline Kit bundles. Snapshot path must be reachable/read-only under sealed deployments. |
|
||||
| `...:AllowBuiltInSnapshotFallback` | `true` | When the public Canonical root index is unavailable and no offline snapshot exists, Excititor infers the known public channel catalog URLs from `IndexUri` so bootstrap does not fail on a missing discovery document alone. |
|
||||
| `...:TrustWeight` | `0.75` | Baseline trust weight (0–1). Lens multiplies this by freshness/justification modifiers. |
|
||||
| `...:TrustTier` | `"distro"` | Friendly tier label surfaced via `vex.provenance.trust.tier` (e.g., `distro-trusted`, `community`). |
|
||||
| `...:CosignIssuer` / `CosignIdentityPattern` | `null` | Supply when Ubuntu publishes cosign attestations (issuer URL and identity regex). Required together. |
|
||||
@@ -61,6 +62,7 @@ Excititor__Connectors__Ubuntu__CosignIdentityPattern=spiffe://ubuntu/vex/*
|
||||
5. **Offline mode** – populate `OfflineSnapshotPath` via Offline Kit bundles before toggling `PreferOfflineSnapshot`. Keep snapshots in the sealed `/opt/stella/offline` hierarchy for auditability.
|
||||
|
||||
## Troubleshooting
|
||||
- **Root index returns 404** – with `AllowBuiltInSnapshotFallback=true` Excititor infers the known public Ubuntu channel catalogs (for example `stable/catalog.json`) from the configured `IndexUri`. If you mirror Ubuntu to a custom layout, either preserve the same directory shape or disable the fallback and provide an offline snapshot explicitly.
|
||||
- **Connector refuses to start** – check logs for `InvalidOperationException` referencing `CosignIssuer`/`CosignIdentityPattern` or missing snapshot path; the validator enforces complete pairs and on-disk paths.
|
||||
- **Lens still sees default weights** – confirm the Excititor deployment picked up the new settings (view `/excititor/health` JSON → `connectors.providers[].options`). Lens only overrides when the provenance payload includes `vex.provenance.trust.*` fields.
|
||||
- **PGP mismatch alerts** – if Lens reports fingerprint mismatches, ensure the list ordering matches Canonical’s published order; duplicates are trimmed, so provide each fingerprint once.
|
||||
|
||||
Reference in New Issue
Block a user