Some checks failed
		
		
	
	Docs CI / lint-and-preview (push) Has been cancelled
				
			- Added `SchedulerWorkerOptions` class to encapsulate configuration for the scheduler worker. - Introduced `PlannerBackgroundService` to manage the planner loop, fetching and processing planning runs. - Created `PlannerExecutionService` to handle the execution logic for planning runs, including impact targeting and run persistence. - Developed `PlannerExecutionResult` and `PlannerExecutionStatus` to standardize execution outcomes. - Implemented validation logic within `SchedulerWorkerOptions` to ensure proper configuration. - Added documentation for the planner loop and impact targeting features. - Established health check endpoints and authentication mechanisms for the Signals service. - Created unit tests for the Signals API to ensure proper functionality and response handling. - Configured options for authority integration and fallback authentication methods.
		
			
				
	
	
		
			651 lines
		
	
	
		
			25 KiB
		
	
	
	
		
			Markdown
		
	
	
	
	
	
			
		
		
	
	
			651 lines
		
	
	
		
			25 KiB
		
	
	
	
		
			Markdown
		
	
	
	
	
	
| # component_architecture_excititor.md — **Stella Ops Excititor** (Sprint 22)
 | ||
| 
 | ||
| > **Scope.** This document specifies the **Excititor** service: its purpose, trust model, data structures, observation/linkset pipelines, APIs, plug-in contracts, storage schema, performance budgets, testing matrix, and how it integrates with Concelier, Policy Engine, and evidence surfaces. It is implementation-ready.
 | ||
| 
 | ||
| ---
 | ||
| 
 | ||
| ## 0) Mission & role in the platform
 | ||
| 
 | ||
| **Mission.** Convert heterogeneous **VEX** statements (OpenVEX, CSAF VEX, CycloneDX VEX; vendor/distro/platform sources) into immutable **VEX observations**, correlate them into **linksets** that retain provenance/conflicts without precedence, and publish deterministic evidence exports and events that Policy Engine, Console, and CLI use to suppress or explain findings.
 | ||
| 
 | ||
| **Boundaries.**
 | ||
| 
 | ||
| * Excititor **does not** decide PASS/FAIL. It supplies **evidence** (statuses + justifications + provenance weights).
 | ||
| * Excititor preserves **conflicting observations** unchanged; consensus (when enabled) merely annotates how policy might choose, but raw evidence remains exportable.
 | ||
| * VEX consumption is **backend‑only**: Scanner never applies VEX. The backend’s **Policy Engine** asks Excititor for status evidence and then decides what to show.
 | ||
| 
 | ||
| ---
 | ||
| 
 | ||
| ## 1) Inputs, outputs & canonical domain
 | ||
| 
 | ||
| ### 1.1 Accepted input formats (ingest)
 | ||
| 
 | ||
| * **OpenVEX** JSON documents (attested or raw).
 | ||
| * **CSAF VEX** 2.x (vendor PSIRTs and distros commonly publish CSAF).
 | ||
| * **CycloneDX VEX** 1.4+ (standalone VEX or embedded VEX blocks).
 | ||
| * **OCI‑attached attestations** (VEX statements shipped as OCI referrers) — optional connectors.
 | ||
| 
 | ||
| All connectors register **source metadata**: provider identity, trust tier, signature expectations (PGP/cosign/PKI), fetch windows, rate limits, and time anchors.
 | ||
| 
 | ||
| ### 1.2 Canonical model (observations & linksets)
 | ||
| 
 | ||
| #### VexObservation
 | ||
| 
 | ||
| ```jsonc
 | ||
| observationId       // {tenant}:{providerId}:{upstreamId}:{revision}
 | ||
| tenant
 | ||
| providerId          // e.g., redhat, suse, ubuntu, osv
 | ||
| streamId            // connector stream (csaf, openvex, cyclonedx, attestation)
 | ||
| upstream{
 | ||
|     upstreamId,
 | ||
|     documentVersion?,
 | ||
|     fetchedAt,
 | ||
|     receivedAt,
 | ||
|     contentHash,
 | ||
|     signature{present, format?, keyId?, signature?}
 | ||
| }
 | ||
| statements[
 | ||
|   {
 | ||
|     vulnerabilityId,
 | ||
|     productKey,
 | ||
|     status,                    // affected | not_affected | fixed | under_investigation
 | ||
|     justification?,
 | ||
|     introducedVersion?,
 | ||
|     fixedVersion?,
 | ||
|     lastObserved,
 | ||
|     locator?,                  // JSON Pointer/line for provenance
 | ||
|     evidence?[]
 | ||
|   }
 | ||
| ]
 | ||
| content{
 | ||
|     format,
 | ||
|     specVersion?,
 | ||
|     raw
 | ||
| }
 | ||
| linkset{
 | ||
|     aliases[],                 // CVE/GHSA/vendor IDs
 | ||
|     purls[],
 | ||
|     cpes[],
 | ||
|     references[{type,url}],
 | ||
|     reconciledFrom[]
 | ||
| }
 | ||
| supersedes?
 | ||
| createdAt
 | ||
| attributes?
 | ||
| ```
 | ||
| 
 | ||
| #### VexLinkset
 | ||
| 
 | ||
| ```jsonc
 | ||
| linksetId           // sha256 over sorted (tenant, vulnId, productKey, observationIds)
 | ||
| tenant
 | ||
| key{
 | ||
|     vulnerabilityId,
 | ||
|     productKey,
 | ||
|     confidence          // low|medium|high
 | ||
| }
 | ||
| observations[] = [
 | ||
|   {
 | ||
|     observationId,
 | ||
|     providerId,
 | ||
|     status,
 | ||
|     justification?,
 | ||
|     introducedVersion?,
 | ||
|     fixedVersion?,
 | ||
|     evidence?,
 | ||
|     collectedAt
 | ||
|   }
 | ||
| ]
 | ||
| aliases{
 | ||
|     primary,
 | ||
|     others[]
 | ||
| }
 | ||
| purls[]
 | ||
| cpes[]
 | ||
| conflicts[]?        // see VexLinksetConflict
 | ||
| createdAt
 | ||
| updatedAt
 | ||
| ```
 | ||
| 
 | ||
| #### VexLinksetConflict
 | ||
| 
 | ||
| ```jsonc
 | ||
| conflictId
 | ||
| type                // status-mismatch | justification-divergence | version-range-clash | non-joinable-overlap | metadata-gap
 | ||
| field?              // optional pointer for UI rendering
 | ||
| statements[]        // per-observation values with providerId + status/justification/version data
 | ||
| confidence
 | ||
| detectedAt
 | ||
| ```
 | ||
| 
 | ||
| #### VexConsensus (optional)
 | ||
| 
 | ||
| ```jsonc
 | ||
| consensusId         // sha256(vulnerabilityId, productKey, policyRevisionId)
 | ||
| vulnerabilityId
 | ||
| productKey
 | ||
| rollupStatus        // derived by Excititor policy adapter (linkset aware)
 | ||
| sources[]           // observation references with weight, accepted flag, reason
 | ||
| policyRevisionId
 | ||
| evaluatedAt
 | ||
| consensusDigest
 | ||
| ```
 | ||
| 
 | ||
| Consensus persists only when Excititor policy adapters require pre-computed rollups (e.g., Offline Kit). Policy Engine can also compute consensus on demand from linksets.
 | ||
| 
 | ||
| ### 1.3 Exports & evidence bundles
 | ||
| 
 | ||
| * **Raw observations** — JSON tree per observation for auditing/offline.
 | ||
| * **Linksets** — grouped evidence for policy/Console/CLI consumption.
 | ||
| * **Consensus (optional)** — if enabled, mirrors existing API contracts.
 | ||
| * **Provider snapshots** — last N days of observations per provider to support diagnostics.
 | ||
| * **Index** — `(productKey, vulnerabilityId) → {status candidates, confidence, observationIds}` for high-speed joins.
 | ||
| 
 | ||
| All exports remain deterministic and, when configured, attested via DSSE + Rekor v2.
 | ||
| 
 | ||
| ---
 | ||
| 
 | ||
| ## 2) Identity model — products & joins
 | ||
| 
 | ||
| ### 2.1 Vuln identity
 | ||
| 
 | ||
| * Accepts **CVE**, **GHSA**, vendor IDs (MSRC, RHSA…), distro IDs (DSA/USN/RHSA…) — normalized to `vulnId` with alias sets.
 | ||
| * **Alias graph** maintained (from Concelier) to map vendor/distro IDs → CVE (primary) and to **GHSA** where applicable.
 | ||
| 
 | ||
| ### 2.2 Product identity (`productKey`)
 | ||
| 
 | ||
| * **Primary:** `purl` (Package URL).
 | ||
| * **Secondary links:** `cpe`, **OS package NVRA/EVR**, NuGet/Maven/Golang identity, and **OS package name** when purl unavailable.
 | ||
| * **Fallback:** `oci:<registry>/<repo>@<digest>` for image‑level VEX.
 | ||
| * **Special cases:** kernel modules, firmware, platforms → provider‑specific mapping helpers (connector captures provider’s product taxonomy → canonical `productKey`).
 | ||
| 
 | ||
| > Excititor does not invent identities. If a provider cannot be mapped to purl/CPE/NVRA deterministically, we keep the native **product string** and mark the claim as **non‑joinable**; the backend will ignore it unless a policy explicitly whitelists that provider mapping.
 | ||
| 
 | ||
| ---
 | ||
| 
 | ||
| ## 3) Storage schema (MongoDB)
 | ||
| 
 | ||
| Database: `excititor`
 | ||
| 
 | ||
| ### 3.1 Collections
 | ||
| 
 | ||
| **`vex.providers`**
 | ||
| 
 | ||
| ```
 | ||
| _id: providerId
 | ||
| name, homepage, contact
 | ||
| trustTier: enum {vendor, distro, platform, hub, attestation}
 | ||
| signaturePolicy: { type: pgp|cosign|x509|none, keys[], certs[], cosignKeylessRoots[] }
 | ||
| fetch: { baseUrl, kind: http|oci|file, rateLimit, etagSupport, windowDays }
 | ||
| enabled: bool
 | ||
| createdAt, modifiedAt
 | ||
| ```
 | ||
| 
 | ||
| **`vex.raw`** (immutable raw documents)
 | ||
| 
 | ||
| ```
 | ||
| _id: sha256(doc bytes)
 | ||
| providerId
 | ||
| uri
 | ||
| ingestedAt
 | ||
| contentType
 | ||
| sig: { verified: bool, method: pgp|cosign|x509|none, keyId|certSubject, bundle? }
 | ||
| payload: GridFS pointer (if large)
 | ||
| disposition: kept|replaced|superseded
 | ||
| correlation: { replaces?: sha256, replacedBy?: sha256 }
 | ||
| ```
 | ||
| 
 | ||
| **`vex.observations`**
 | ||
| 
 | ||
| ```
 | ||
| {
 | ||
|   _id: "tenant:providerId:upstreamId:revision",
 | ||
|   tenant,
 | ||
|   providerId,
 | ||
|   streamId,
 | ||
|   upstream: { upstreamId, documentVersion?, fetchedAt, receivedAt, contentHash, signature },
 | ||
|   statements: [
 | ||
|     {
 | ||
|       vulnerabilityId,
 | ||
|       productKey,
 | ||
|       status,
 | ||
|       justification?,
 | ||
|       introducedVersion?,
 | ||
|       fixedVersion?,
 | ||
|       lastObserved,
 | ||
|       locator?,
 | ||
|       evidence?
 | ||
|     }
 | ||
|   ],
 | ||
|   content: { format, specVersion?, raw },
 | ||
|   linkset: { aliases[], purls[], cpes[], references[], reconciledFrom[] },
 | ||
|   supersedes?,
 | ||
|   createdAt,
 | ||
|   attributes?
 | ||
| }
 | ||
| ```
 | ||
| 
 | ||
|   * Indexes: `{tenant:1, providerId:1, upstream.upstreamId:1}`, `{tenant:1, statements.vulnerabilityId:1}`, `{tenant:1, linkset.purls:1}`, `{tenant:1, createdAt:-1}`.
 | ||
| 
 | ||
| **`vex.linksets`**
 | ||
| 
 | ||
| ```
 | ||
| {
 | ||
|   _id: "sha256:...",
 | ||
|   tenant,
 | ||
|   key: { vulnerabilityId, productKey, confidence },
 | ||
|   observations: [
 | ||
|     { observationId, providerId, status, justification?, introducedVersion?, fixedVersion?, evidence?, collectedAt }
 | ||
|   ],
 | ||
|   aliases: { primary, others: [] },
 | ||
|   purls: [],
 | ||
|   cpes: [],
 | ||
|   conflicts: [],
 | ||
|   createdAt,
 | ||
|   updatedAt
 | ||
| }
 | ||
| ```
 | ||
| 
 | ||
|   * Indexes: `{tenant:1, key.vulnerabilityId:1, key.productKey:1}`, `{tenant:1, purls:1}`, `{tenant:1, updatedAt:-1}`.
 | ||
| 
 | ||
| **`vex.events`** (observation/linkset events, optional long retention)
 | ||
| 
 | ||
| ```
 | ||
| {
 | ||
|   _id: ObjectId,
 | ||
|   tenant,
 | ||
|   type: "vex.observation.updated" | "vex.linkset.updated",
 | ||
|   key,
 | ||
|   delta,
 | ||
|   hash,
 | ||
|   occurredAt
 | ||
| }
 | ||
| ```
 | ||
| 
 | ||
|   * Indexes: `{type:1, occurredAt:-1}`, TTL on `occurredAt` for configurable retention.
 | ||
| 
 | ||
| **`vex.consensus`** (optional rollups)
 | ||
| 
 | ||
| ```
 | ||
| _id: sha256(canonical(vulnerabilityId, productKey, policyRevisionId))
 | ||
| vulnerabilityId
 | ||
| productKey
 | ||
| rollupStatus
 | ||
| sources[]      // observation references with weights/reasons
 | ||
| policyRevisionId
 | ||
| evaluatedAt
 | ||
| signals?       // optional severity/kev/epss hints
 | ||
| consensusDigest
 | ||
| ```
 | ||
| 
 | ||
|   * Indexes: `{vulnerabilityId:1, productKey:1}`, `{policyRevisionId:1, evaluatedAt:-1}`.
 | ||
| 
 | ||
| **`vex.exports`** (manifest of emitted artifacts)
 | ||
| 
 | ||
| ```
 | ||
| _id
 | ||
| querySignature
 | ||
| format: raw|consensus|index
 | ||
| artifactSha256
 | ||
| rekor { uuid, index, url }?
 | ||
| createdAt
 | ||
| policyRevisionId
 | ||
| cacheable: bool
 | ||
| ```
 | ||
| 
 | ||
| **`vex.cache`** — observation/linkset export cache: `{querySignature, exportId, ttl, hits}`.
 | ||
| 
 | ||
| **`vex.migrations`** — ordered migrations ensuring new indexes (`20251027-linksets-introduced`, etc.).
 | ||
| 
 | ||
| ### 3.2 Indexing strategy
 | ||
| 
 | ||
| * Hot path queries rely on `{tenant, key.vulnerabilityId, key.productKey}` covering linkset lookup.
 | ||
| * Observability queries use `{tenant, updatedAt}` to monitor staleness.
 | ||
| * Consensus (if enabled) keyed by `{vulnerabilityId, productKey, policyRevisionId}` for deterministic reuse.
 | ||
| 
 | ||
| ---
 | ||
| 
 | ||
| ## 4) Ingestion pipeline
 | ||
| 
 | ||
| ### 4.1 Connector contract
 | ||
| 
 | ||
| ```csharp
 | ||
| public interface IVexConnector
 | ||
| {
 | ||
|     string ProviderId { get; }
 | ||
|     Task FetchAsync(VexConnectorContext ctx, CancellationToken ct);   // raw docs
 | ||
|     Task NormalizeAsync(VexConnectorContext ctx, CancellationToken ct); // raw -> ObservationStatements[]
 | ||
| }
 | ||
| ```
 | ||
| 
 | ||
| * **Fetch** must implement: window scheduling, conditional GET (ETag/If‑Modified‑Since), rate limiting, retry/backoff.
 | ||
| * **Normalize** parses the format, validates schema, maps product identities deterministically, emits observation statements with **provenance** metadata (locator, justification, version ranges).
 | ||
| 
 | ||
| ### 4.2 Signature verification (per provider)
 | ||
| 
 | ||
| * **cosign (keyless or keyful)** for OCI referrers or HTTP‑served JSON with Sigstore bundles.
 | ||
| * **PGP** (provider keyrings) for distro/vendor feeds that sign docs.
 | ||
| * **x509** (mutual TLS / provider‑pinned certs) where applicable.
 | ||
| * Signature state is stored on **vex.raw.sig** and copied into `statements[].signatureState` so downstream policy can gate by verification result.
 | ||
| 
 | ||
| > Observation statements from sources failing signature policy are marked `"signatureState.verified=false"` and policy can down-weight or ignore them.
 | ||
| 
 | ||
| ### 4.3 Time discipline
 | ||
| 
 | ||
| * For each doc, prefer **provider’s document timestamp**; if absent, use fetch time.
 | ||
| * Statements carry `lastObserved` which drives **tie-breaking** within equal weight tiers.
 | ||
| 
 | ||
| ---
 | ||
| 
 | ||
| ## 5) Normalization: product & status semantics
 | ||
| 
 | ||
| ### 5.1 Product mapping
 | ||
| 
 | ||
| * **purl** first; **cpe** second; OS package NVRA/EVR mapping helpers (distro connectors) produce purls via canonical tables (e.g., rpm→purl:rpm, deb→purl:deb).
 | ||
| * Where a provider publishes **platform‑level** VEX (e.g., “RHEL 9 not affected”), connectors expand to known product inventory rules (e.g., map to sets of packages/components shipped in the platform). Expansion tables are versioned and kept per provider; every expansion emits **evidence** indicating the rule applied.
 | ||
| * If expansion would be speculative, the statement remains **platform-scoped** with `productKey="platform:redhat:rhel:9"` and is flagged **non-joinable**; backend can decide to use platform VEX only when Scanner proves the platform runtime.
 | ||
| 
 | ||
| ### 5.2 Status + justification mapping
 | ||
| 
 | ||
| * Canonical **status**: `affected | not_affected | fixed | under_investigation`.
 | ||
| * **Justifications** normalized to a controlled vocabulary (CISA‑aligned), e.g.:
 | ||
| 
 | ||
|   * `component_not_present`
 | ||
|   * `vulnerable_code_not_in_execute_path`
 | ||
|   * `vulnerable_configuration_unused`
 | ||
|   * `inline_mitigation_applied`
 | ||
|   * `fix_available` (with `fixedVersion`)
 | ||
|   * `under_investigation`
 | ||
| * Providers with free‑text justifications are mapped by deterministic tables; raw text preserved as `evidence`.
 | ||
| 
 | ||
| ---
 | ||
| 
 | ||
| ## 6) Consensus algorithm
 | ||
| 
 | ||
| **Goal:** produce a **stable**, explainable `rollupStatus` per `(vulnId, productKey)` when consumers opt into Excititor-managed consensus derived from linksets.
 | ||
| 
 | ||
| ### 6.1 Inputs
 | ||
| 
 | ||
| * Set **S** of observation statements drawn from the current `VexLinkset` for `(tenant, vulnId, productKey)`.
 | ||
| * **Excititor policy snapshot**:
 | ||
| 
 | ||
|   * **weights** per provider tier and per provider overrides.
 | ||
|   * **justification gates** (e.g., require justification for `not_affected` to be acceptable).
 | ||
|   * **minEvidence** rules (e.g., `not_affected` must come from ≥1 vendor or 2 distros).
 | ||
|   * **signature requirements** (e.g., require verified signature for ‘fixed’ to be considered).
 | ||
| 
 | ||
| ### 6.2 Steps
 | ||
| 
 | ||
| 1. **Filter invalid** statements by signature policy & justification gates → set `S'`.
 | ||
| 2. **Score** each statement:
 | ||
|    `score = weight(provider) * freshnessFactor(lastObserved)` where freshnessFactor ∈ [0.8, 1.0] for staleness decay (configurable; small effect). Observations lacking verified signatures receive policy-configured penalties.
 | ||
| 3. **Aggregate** scores per status: `W(status) = Σ score(statements with that status)`.
 | ||
| 4. **Pick** `rollupStatus = argmax_status W(status)`.
 | ||
| 5. **Tie‑breakers** (in order):
 | ||
| 
 | ||
|    * Higher **max single** provider score wins (vendor > distro > platform > hub).
 | ||
|    * More **recent** lastObserved wins.
 | ||
|    * Deterministic lexicographic order of status (`fixed` > `not_affected` > `under_investigation` > `affected`) as final tiebreaker.
 | ||
| 6. **Explain**: mark accepted observations (`accepted=true; reason="weight"`/`"freshness"`/`"confidence"`) and rejected ones with explicit `reason` (`"insufficient_justification"`, `"signature_unverified"`, `"lower_weight"`, `"low_confidence_linkset"`).
 | ||
| 
 | ||
| > The algorithm is **pure** given `S` and policy snapshot; result is reproducible and hashed into `consensusDigest`.
 | ||
| 
 | ||
| ---
 | ||
| 
 | ||
| ## 7) Query & export APIs
 | ||
| 
 | ||
| All endpoints are versioned under `/api/v1/vex`.
 | ||
| 
 | ||
| ### 7.1 Query (online)
 | ||
| 
 | ||
| ```
 | ||
| POST /observations/search
 | ||
|   body: { vulnIds?: string[], productKeys?: string[], providers?: string[], since?: timestamp, limit?: int, pageToken?: string }
 | ||
|   → { observations[], nextPageToken? }
 | ||
| 
 | ||
| POST /linksets/search
 | ||
|   body: { vulnIds?: string[], productKeys?: string[], confidence?: string[], since?: timestamp, limit?: int, pageToken?: string }
 | ||
|   → { linksets[], nextPageToken? }
 | ||
| 
 | ||
| POST /consensus/search
 | ||
|   body: { vulnIds?: string[], productKeys?: string[], policyRevisionId?: string, since?: timestamp, limit?: int, pageToken?: string }
 | ||
|   → { entries[], nextPageToken? }
 | ||
| 
 | ||
| POST /excititor/resolve (scope: vex.read)
 | ||
|   body: { productKeys?: string[], purls?: string[], vulnerabilityIds: string[], policyRevisionId?: string }
 | ||
|   → { policy, resolvedAt, results: [ { vulnerabilityId, productKey, status, observations[], conflicts[], linksetConfidence, consensus?, signals?, envelope? } ] }
 | ||
| ```
 | ||
| 
 | ||
| ### 7.2 Exports (cacheable snapshots)
 | ||
| 
 | ||
| ```
 | ||
| POST /exports
 | ||
|   body: { signature: { vulnFilter?, productFilter?, providers?, since? }, format: raw|consensus|index, policyRevisionId?: string, force?: bool }
 | ||
|   → { exportId, artifactSha256, rekor? }
 | ||
| 
 | ||
| GET  /exports/{exportId}        → bytes (application/json or binary index)
 | ||
| GET  /exports/{exportId}/meta   → { signature, policyRevisionId, createdAt, artifactSha256, rekor? }
 | ||
| ```
 | ||
| 
 | ||
| ### 7.3 Provider operations
 | ||
| 
 | ||
| ```
 | ||
| GET  /providers                  → provider list & signature policy
 | ||
| POST /providers/{id}/refresh     → trigger fetch/normalize window
 | ||
| GET  /providers/{id}/status      → last fetch, doc counts, signature stats
 | ||
| ```
 | ||
| 
 | ||
| **Auth:** service‑to‑service via Authority tokens; operator operations via UI/CLI with RBAC.
 | ||
| 
 | ||
| ---
 | ||
| 
 | ||
| ## 8) Attestation integration
 | ||
| 
 | ||
| * Exports can be **DSSE‑signed** via **Signer** and logged to **Rekor v2** via **Attestor** (optional but recommended for regulated pipelines).
 | ||
| * `vex.exports.rekor` stores `{uuid, index, url}` when present.
 | ||
| * **Predicate type**: `https://stella-ops.org/attestations/vex-export/1` with fields:
 | ||
| 
 | ||
|   * `querySignature`, `policyRevisionId`, `artifactSha256`, `createdAt`.
 | ||
| 
 | ||
| ---
 | ||
| 
 | ||
| ## 9) Configuration (YAML)
 | ||
| 
 | ||
| ```yaml
 | ||
| excititor:
 | ||
|   mongo: { uri: "mongodb://mongo/excititor" }
 | ||
|   s3:
 | ||
|     endpoint: http://minio:9000
 | ||
|     bucket: stellaops
 | ||
|   policy:
 | ||
|     weights:
 | ||
|       vendor: 1.0
 | ||
|       distro: 0.9
 | ||
|       platform: 0.7
 | ||
|       hub: 0.5
 | ||
|       attestation: 0.6
 | ||
|       ceiling: 1.25
 | ||
|     scoring:
 | ||
|       alpha: 0.25
 | ||
|       beta: 0.5
 | ||
|     providerOverrides:
 | ||
|       redhat: 1.0
 | ||
|       suse: 0.95
 | ||
|     requireJustificationForNotAffected: true
 | ||
|     signatureRequiredForFixed: true
 | ||
|     minEvidence:
 | ||
|       not_affected:
 | ||
|         vendorOrTwoDistros: true
 | ||
|   connectors:
 | ||
|     - providerId: redhat
 | ||
|       kind: csaf
 | ||
|       baseUrl: https://access.redhat.com/security/data/csaf/v2/
 | ||
|       signaturePolicy: { type: pgp, keys: [ "…redhat-pgp-key…" ] }
 | ||
|       windowDays: 7
 | ||
|     - providerId: suse
 | ||
|       kind: csaf
 | ||
|       baseUrl: https://ftp.suse.com/pub/projects/security/csaf/
 | ||
|       signaturePolicy: { type: pgp, keys: [ "…suse-pgp-key…" ] }
 | ||
|     - providerId: ubuntu
 | ||
|       kind: openvex
 | ||
|       baseUrl: https://…/vex/
 | ||
|       signaturePolicy: { type: none }
 | ||
|     - providerId: vendorX
 | ||
|       kind: cyclonedx-vex
 | ||
|       ociRef: ghcr.io/vendorx/vex@sha256:…
 | ||
|       signaturePolicy: { type: cosign, cosignKeylessRoots: [ "sigstore-root" ] }
 | ||
| ```
 | ||
| 
 | ||
| ### 9.1 WebService endpoints
 | ||
| 
 | ||
| With storage configured, the WebService exposes the following ingress and diagnostic APIs:
 | ||
| 
 | ||
| * `GET /excititor/status` – returns the active storage configuration and registered artifact stores.
 | ||
| * `GET /excititor/health` – simple liveness probe.
 | ||
| * `POST /excititor/statements` – accepts normalized VEX statements and persists them via `IVexClaimStore`; use this for migrations/backfills.
 | ||
| * `GET /excititor/statements/{vulnId}/{productKey}?since=` – returns the immutable statement log for a vulnerability/product pair.
 | ||
| * `POST /excititor/resolve` – requires `vex.read` scope; accepts up to 256 `(vulnId, productKey)` pairs via `productKeys` or `purls` and returns deterministic consensus results, decision telemetry, and a signed envelope (`artifact` digest, optional signer signature, optional attestation metadata + DSSE envelope). Returns **409 Conflict** when the requested `policyRevisionId` mismatches the active snapshot.
 | ||
| 
 | ||
| Run the ingestion endpoint once after applying migration `20251019-consensus-signals-statements` to repopulate historical statements with the new severity/KEV/EPSS signal fields.
 | ||
| 
 | ||
| * `weights.ceiling` raises the deterministic clamp applied to provider tiers/overrides (range 1.0‒5.0). Values outside the range are clamped with warnings so operators can spot typos.
 | ||
| * `scoring.alpha` / `scoring.beta` configure KEV/EPSS boosts for the Phase 1 → Phase 2 scoring pipeline. Defaults (0.25, 0.5) preserve prior behaviour; negative or excessively large values fall back with diagnostics.
 | ||
| 
 | ||
| ---
 | ||
| 
 | ||
| ## 10) Security model
 | ||
| 
 | ||
| * **Input signature verification** enforced per provider policy (PGP, cosign, x509).
 | ||
| * **Connector allowlists**: outbound fetch constrained to configured domains.
 | ||
| * **Tenant isolation**: per‑tenant DB prefixes or separate DBs; per‑tenant S3 prefixes; per‑tenant policies.
 | ||
| * **AuthN/Z**: Authority‑issued OpToks; RBAC roles (`vex.read`, `vex.admin`, `vex.export`).
 | ||
| * **No secrets in logs**; deterministic logging contexts include providerId, docDigest, observationId, and linksetId.
 | ||
| 
 | ||
| ---
 | ||
| 
 | ||
| ## 11) Performance & scale
 | ||
| 
 | ||
| * **Targets:**
 | ||
| 
 | ||
|   * Normalize 10k observation statements/minute/core.
 | ||
|   * Linkset rebuild ≤ 20 ms P95 for 1k unique `(vuln, product)` pairs in hot cache.
 | ||
|   * Consensus (when enabled) compute ≤ 50 ms for 1k unique `(vuln, product)` pairs.
 | ||
|   * Export (observations + linksets) 1M rows in ≤ 60 s on 8 cores with streaming writer.
 | ||
| 
 | ||
| * **Scaling:**
 | ||
| 
 | ||
|   * WebService handles control APIs; **Worker** background services (same image) execute fetch/normalize in parallel with rate‑limits; Mongo writes batched; upserts by natural keys.
 | ||
|   * Exports stream straight to S3 (MinIO) with rolling buffers.
 | ||
| 
 | ||
| * **Caching:**
 | ||
| 
 | ||
|   * `vex.cache` maps query signatures → export; TTL to avoid stampedes; optimistic reuse unless `force`.
 | ||
| 
 | ||
| ### 11.1 Worker TTL refresh controls
 | ||
| 
 | ||
| Excititor.Worker ships with a background refresh service that re-evaluates stale consensus rows and applies stability dampers before publishing status flips. Operators can tune its behaviour through the following configuration (shown in `appsettings.json` syntax):
 | ||
| 
 | ||
| ```jsonc
 | ||
| {
 | ||
|   "Excititor": {
 | ||
|     "Worker": {
 | ||
|       "Refresh": {
 | ||
|         "Enabled": true,
 | ||
|         "ConsensusTtl": "02:00:00",       // refresh consensus older than 2 hours
 | ||
|         "ScanInterval": "00:10:00",       // sweep cadence
 | ||
|         "ScanBatchSize": 250,              // max documents examined per sweep
 | ||
|         "Damper": {
 | ||
|           "Minimum": "1.00:00:00",       // lower bound before status flip publishes
 | ||
|           "Maximum": "2.00:00:00",       // upper bound guardrail
 | ||
|           "DefaultDuration": "1.12:00:00",
 | ||
|           "Rules": [
 | ||
|             { "MinWeight": 0.90, "Duration": "1.00:00:00" },
 | ||
|             { "MinWeight": 0.75, "Duration": "1.06:00:00" },
 | ||
|             { "MinWeight": 0.50, "Duration": "1.12:00:00" }
 | ||
|           ]
 | ||
|         }
 | ||
|       }
 | ||
|     }
 | ||
|   }
 | ||
| }
 | ||
| ```
 | ||
| 
 | ||
| * `ConsensusTtl` governs when the worker issues a fresh resolve for cached consensus data.
 | ||
| * `Damper` lengths are clamped between `Minimum`/`Maximum`; duration is bypassed when component fingerprints (`VexProduct.ComponentIdentifiers`) change.
 | ||
| * The same keys are available through environment variables (e.g., `Excititor__Worker__Refresh__ConsensusTtl=02:00:00`).
 | ||
| 
 | ||
| ---
 | ||
| 
 | ||
| ## 12) Observability
 | ||
| 
 | ||
| * **Metrics:**
 | ||
| 
 | ||
|   * `vex.fetch.requests_total{provider}` / `vex.fetch.bytes_total{provider}`
 | ||
|   * `vex.fetch.failures_total{provider,reason}` / `vex.signature.failures_total{provider,method}`
 | ||
|   * `vex.normalize.statements_total{provider}`
 | ||
|   * `vex.observations.write_total{result}`
 | ||
|   * `vex.linksets.updated_total{result}` / `vex.linksets.conflicts_total{type}`
 | ||
|   * `vex.consensus.rollup_total{status}` (when enabled)
 | ||
|   * `vex.exports.bytes_total{format}` / `vex.exports.latency_seconds{format}`
 | ||
| * **Tracing:** spans for fetch, verify, parse, map, observe, linkset, consensus, export.
 | ||
| * **Dashboards:** provider staleness, linkset conflict hot spots, signature posture, export cache hit-rate.
 | ||
| 
 | ||
| ---
 | ||
| 
 | ||
| ## 13) Testing matrix
 | ||
| 
 | ||
| * **Connectors:** golden raw docs → deterministic observation statements (fixtures per provider/format).
 | ||
| * **Signature policies:** valid/invalid PGP/cosign/x509 samples; ensure rejects are recorded but not accepted.
 | ||
| * **Normalization edge cases:** platform-scoped statements, free-text justifications, non-purl products.
 | ||
| * **Linksets:** conflict scenarios across tiers; verify confidence scoring + conflict payload stability.
 | ||
| * **Consensus (optional):** ensure tie-breakers honour policy weights/justification gates.
 | ||
| * **Performance:** 1M-row observation/linkset export timing; memory ceilings; stream correctness.
 | ||
| * **Determinism:** same inputs + policy → identical linkset hashes, conflict payloads, optional `consensusDigest`, and export bytes.
 | ||
| * **API contract tests:** pagination, filters, RBAC, rate limits.
 | ||
| 
 | ||
| ---
 | ||
| 
 | ||
| ## 14) Integration points
 | ||
| 
 | ||
| * **Backend Policy Engine** (in Scanner.WebService): calls `POST /excititor/resolve` (scope `vex.read`) with batched `(purl, vulnId)` pairs to fetch `rollupStatus + sources`.
 | ||
| * **Concelier**: provides alias graph (CVE↔vendor IDs) and may supply VEX‑adjacent metadata (e.g., KEV flag) for policy escalation.
 | ||
| * **UI**: VEX explorer screens use `/observations/search`, `/linksets/search`, and `/consensus/search`; show conflicts & provenance.
 | ||
| * **CLI**: `stella vex linksets export --since 7d --out vex-linksets.json` (optionally `--include-consensus`) for audits and Offline Kit parity.
 | ||
| 
 | ||
| ---
 | ||
| 
 | ||
| ## 15) Failure modes & fallback
 | ||
| 
 | ||
| * **Provider unreachable:** stale thresholds trigger warnings; policy can down‑weight stale providers automatically (freshness factor).
 | ||
| * **Signature outage:** continue to ingest but mark `signatureState.verified=false`; consensus will likely exclude or down‑weight per policy.
 | ||
| * **Schema drift:** unknown fields are preserved as `evidence`; normalization rejects only on **invalid identity** or **status**.
 | ||
| 
 | ||
| ---
 | ||
| 
 | ||
| ## 16) Rollout plan (incremental)
 | ||
| 
 | ||
| 1. **MVP**: OpenVEX + CSAF connectors for 3 major providers (e.g., Red Hat/SUSE/Ubuntu), normalization + consensus + `/excititor/resolve`.
 | ||
| 2. **Signature policies**: PGP for distros; cosign for OCI.
 | ||
| 3. **Exports + optional attestation**.
 | ||
| 4. **CycloneDX VEX** connectors; platform claim expansion tables; UI explorer.
 | ||
| 5. **Scale hardening**: export indexes; conflict analytics.
 | ||
| 
 | ||
| ---
 | ||
| 
 | ||
| ## 17) Operational runbooks
 | ||
| 
 | ||
| * **Statement backfill** — see `docs/dev/EXCITITOR_STATEMENT_BACKFILL.md` for the CLI workflow, required permissions, observability guidance, and rollback steps.
 | ||
| 
 | ||
| ---
 | ||
| 
 | ||
| ## 18) Appendix — canonical JSON (stable ordering)
 | ||
| 
 | ||
| All exports and consensus entries are serialized via `VexCanonicalJsonSerializer`:
 | ||
| 
 | ||
| * UTF‑8 without BOM;
 | ||
| * keys sorted (ASCII);
 | ||
| * arrays sorted by `(providerId, vulnId, productKey, lastObserved)` unless semantic order mandated;
 | ||
| * timestamps in `YYYY‑MM‑DDThh:mm:ssZ`;
 | ||
| * no insignificant whitespace.
 | ||
| 
 |