git.stella-ops.org/docs/modules/excititor/mirrors.md

architecture_excititor_mirrors.md — Excititor Mirror Distribution

Status: Draft (Sprint 7). Complements docs/modules/excititor/architecture.md by describing the mirror export surface exposed by Excititor.WebService and the configuration hooks used by operators and downstream mirrors.

---

0) Purpose

Excititor publishes canonical VEX consensus data. Operators (or StellaOps-managed mirrors) need a deterministic way to sync those exports into downstream environments. Mirror distribution provides:

• A declarative map of export bundles (json, jsonl, openvex, csaf) reachable via signed HTTP endpoints under /excititor/mirror.
• Thin quota/authentication controls on top of the existing export cache so mirrors cannot starve the web service.
• Stable payload shapes that downstream automation can monitor (index → fetch updates → download artifact → verify signature).

Mirror endpoints are intentionally read-only. Write paths (export generation, attestation, cache) remain the responsibility of the export pipeline.

---

1) Configuration model

The web service reads mirror configuration from Excititor:Mirror (YAML/JSON/appsettings). Each domain groups a set of exports that share rate limits and authentication rules.

Excititor:
  Mirror:
    Domains:
      - id: primary
        displayName: Primary Mirror
        requireAuthentication: false
        maxIndexRequestsPerHour: 600
        maxDownloadRequestsPerHour: 1200
        exports:
          - key: consensus
            format: json
            filters:
              vulnId: CVE-2025-0001
              productKey: pkg:test/demo
            sort:
              createdAt: false     # descending
            limit: 1000
          - key: consensus-openvex
            format: openvex
            filters:
              vulnId: CVE-2025-0001

Root settings

Field	Required	Description
outputRoot	–	Filesystem root where mirror artefacts are written. Defaults to the Excititor file-system artifact store root when omitted.
directoryName	–	Optional subdirectory created under outputRoot; defaults to mirror.
targetRepository	–	Hint propagated to manifests/index files indicating the operator-visible location (for example s3://mirror/excititor).
signing	–	Bundle signing configuration. When enabled, the exporter emits a detached JWS (bundle.json.jws) alongside each domain bundle.

signing supports the following fields:

Field	Required	Description
enabled	–	Toggles detached signing for domain bundles.
algorithm	–	Signing algorithm identifier (default ES256).
keyId	✅ (when enabled)	Signing key identifier resolved via the configured crypto provider registry.
provider	–	Optional provider hint when multiple registries are available.
keyPath	–	Optional PEM path used to seed the provider when the key is not already loaded.

Domain field reference

Field	Required	Description
id	✅	Stable identifier. Appears in URLs (/excititor/mirror/domains/{id}) and download filenames.
displayName	–	Human-friendly label surfaced in the /domains listing. Falls back to id.
requireAuthentication	–	When true the service enforces that the caller is authenticated (Authority token).
maxIndexRequestsPerHour	–	Per-domain quota for index endpoints. 0/negative disables the guard.
maxDownloadRequestsPerHour	–	Per-domain quota for artifact downloads.
exports	✅	Collection of export projections.

Export-level fields:

Field	Required	Description
key	✅	Unique key within the domain. Used in URLs (/exports/{key}) and filenames/bundle entries.
format	✅	One of json, jsonl, openvex, csaf. Maps to VexExportFormat.
filters	–	Key/value pairs executed via VexQueryFilter. Keys must match export data source columns (e.g., vulnId, productKey).
sort	–	Key/boolean map (false = descending).
limit, offset, view	–	Optional query bounds passed through to the export query.

⚠️ Misconfiguration: invalid formats or missing keys cause exports to be flagged with status in the index response; they are not exposed downstream.

---

2) HTTP surface

Routes are grouped under /excititor/mirror.

Method	Path	Description
GET	/domains	Returns configured domains with quota metadata.
GET	/domains/{domainId}	Domain detail (auth/quota + export keys). 404 for unknown domains.
GET	/domains/{domainId}/index	Lists exports with exportId, query signature, format, artifact digest, attestation metadata, and size. Applies index quota.
GET	/domains/{domainId}/exports/{exportKey}	Returns manifest metadata (single export). 404 if unknown/missing.
GET	/domains/{domainId}/exports/{exportKey}/download	Streams export content from the artifact store. Applies download quota.

Responses are serialized via VexCanonicalJsonSerializer ensuring stable ordering. Download responses include a content-disposition header naming the file <domain>-<export>.<ext>.

Error handling

• 401 – authentication required (requireAuthentication=true).
• 404 – domain/export not found or manifest not persisted.
• 429 – per-domain quota exceeded (Retry-After header set in seconds).
• 503 – export misconfiguration (invalid format/query).

---

3) Rate limiting

MirrorRateLimiter implements a simple rolling 1-hour window using IMemoryCache. Each domain has two quotas:

• index scope → maxIndexRequestsPerHour
• download scope → maxDownloadRequestsPerHour

0 or negative limits disable enforcement. Quotas are best-effort (per-instance). For HA deployments, configure sticky routing at the ingress or replace the limiter with a distributed implementation.

---

4) Interaction with export pipeline

Mirror endpoints consume manifests produced by the export engine (MongoVexExportStore). They can also be generated on demand via the mirror domain management API (POST /api/v1/mirror/domains/{domainId}/generate).

4.1 MirrorExportScheduler (background bundle refresh)

MirrorExportScheduler is a BackgroundService that periodically checks configured mirror domains for stale export bundles and triggers regeneration when source data has been updated since the last bundle generation.

Behavior:

1. On startup, loads all configured mirror domains from DB and config.
2. Every N minutes (configurable via RefreshIntervalMinutes, default 60), iterates all domains.
3. For each domain, checks whether any source in the export filters has been updated since the last bundle generation by comparing connector state timestamps.
4. If stale, triggers bundle regeneration via IMirrorBundleRegenerator.
5. Logs bundle generation metrics (duration, advisory count, stale/regenerated counts).
6. Exposes per-domain staleness via GetDomainStatuses() and the /api/v1/mirror/domains/{domainId}/status endpoint.

Configuration:

Setting	Default	Description
AutoRefreshEnabled	true	Set to false to disable the scheduler entirely. Required for air-gap deployments where bundles are imported, not generated locally.
RefreshIntervalMinutes	60	How often the scheduler checks for stale exports. Minimum 1 minute.
Enabled	true	Global mirror distribution enable flag. When false, the scheduler skips refresh cycles.

Staleness detection uses IVexConnectorStateRepository to find the latest source update timestamp across all connectors. If any source was updated after the last bundle generation timestamp, the export is considered stale.

4.2 Export workflow

Recommended workflow:

1. Define mirror domains via the UI wizard or API (POST /api/v1/mirror/domains).
2. Configure exports with filter shorthands (see 4.3 below).
3. Automatic refresh: MirrorExportScheduler regenerates stale bundles periodically.
4. On-demand generation: POST /api/v1/mirror/domains/{domainId}/generate for immediate bundle creation.
5. Downstream mirror automation:
   • GET /domains/{id}/index
   • Compare exportId / consensusRevision
   • GET /download when new
   • Verify digest + attestation

When the export engine runs, it materializes the following artefacts under outputRoot/<directoryName>:

• index.json -- canonical index listing each configured domain, manifest/bundle descriptors (with SHA-256 digests), and available export keys.
• <domain>/manifest.json -- per-domain summary with export metadata (query signature, consensus/score digests, source providers) and a descriptor pointing at the bundle.
• <domain>/bundle.json -- canonical payload containing serialized consensus, score envelopes, and normalized VEX claims for the matching export definitions.
• <domain>/bundle.json.jws -- optional detached JWS when signing is enabled.

Downstream automation reads manifest.json/bundle.json directly, while /excititor/mirror endpoints stream the same artefacts through authenticated HTTP.

4.3 Multi-value filter support and category shorthands

Export filters in MirrorExportOptions.Filters support three expansion modes via the ResolveFilters() method:

sourceCategory shorthand -- resolves a category name (or comma-separated list) to all source IDs in that category:

{
  "Key": "linux-distros",
  "Format": "openvex",
  "Filters": { "sourceCategory": "Distribution" }
}

This resolves to all 10 distribution sources (Debian, Ubuntu, Alpine, SUSE, RHEL, CentOS, Fedora, Arch, Gentoo, Astra Linux). Multiple categories are comma-separated: "Exploit,Container,Ics,PackageManager".

sourceTag shorthand -- resolves a tag (or comma-separated list) to all sources carrying that tag:

{
  "Key": "linux-all",
  "Format": "json",
  "Filters": { "sourceTag": "linux" }
}

Comma-separated sourceVendor -- OR semantics across multiple vendors:

{
  "Key": "custom-selection",
  "Format": "json",
  "Filters": { "sourceVendor": "debian,ubuntu,alpine" }
}

All resolved values are sorted alphabetically for deterministic query signatures. Existing single-value filters remain backward-compatible.

Supported categories (14 total, matching the SourceCategory enum):

Primary, Vendor, Distribution, Ecosystem, Cert, Csaf, Threat, Exploit, Container, Hardware, Ics, PackageManager, Mirror, Other.

---

5) Mirror configuration UI

The mirror configuration UI provides three Angular components for managing mirror domains from the StellaOps Console, accessible under the Advisory & VEX Sources integration hub.

5.1 Mirror Domain Builder wizard

Route: advisory-vex-sources/mirror/new

A 3-step wizard for creating mirror domains from the source catalog:

Step 1 -- Select Sources: Displays the full source catalog grouped by the 14 categories. Operators can select sources individually or by category (checking a category header selects all sources in that category). Shorthand buttons provide quick selections: "All Primary", "All Distributions", "All Ecosystem", "All CERTs", "Everything". A live summary panel shows selected source count by category.

Step 2 -- Configure Domain: Auto-generates a domain ID and display name from the selection. Operators configure the export format (JSON, JSONL, OpenVEX, CSAF, CycloneDX), rate limits (index requests/hour, download requests/hour), authentication requirement, signing options, and optionally create multiple exports per domain.

Step 3 -- Review & Create: Summary card showing domain name, source count across categories, export format, and rate limits. Displays the resolved filter JSON that will be stored. "Create Domain" calls POST /api/v1/mirror/domains. An optional "Generate Now" checkbox triggers immediate bundle generation after creation.

5.2 Mirror Dashboard

Route: advisory-vex-sources/mirror

Displays all configured mirror domains as status cards:

• Top bar: Mirror mode indicator (Direct/Mirror/Hybrid), "Create Domain" button, global mirror health summary.
• Domain cards: Each card shows domain name, export count and format, source count with category pills, last generated timestamp, bundle size, and staleness indicator (fresh/stale/never generated). Actions: Regenerate, Edit, Delete, View Endpoints.
• Consumer config panel: For Hybrid or Mirror mode, shows the consumer mirror URL, connection status, and last sync time.
• Empty state: If no domains are configured, displays a "Create your first mirror domain" call-to-action linking to the wizard.

5.3 Catalog mirror integration

The source catalog header (advisory-source-catalog.component.ts) includes mirror context:

• Mirror mode badge (Direct/Mirror/Hybrid) in the catalog header.
• "Configure Mirror" link navigating to the mirror dashboard.
• "Create Mirror Domain" button navigating to the wizard with pre-selected sources based on currently enabled sources.
• Mirror domain count and total bundle advisory count in the stats bar (when domains exist).

---

6) Operational guidance

• Track quota utilisation via HTTP 429 metrics (configure structured logging or OTEL counters when rate limiting triggers).
• Mirror domains can be deployed per tenant (e.g., tenant-a, tenant-b) with different auth requirements.
• Ensure the underlying artifact stores (FileSystem, S3, offline bundle) retain artefacts long enough for mirrors to sync.
• For air-gapped mirrors, combine mirror endpoints with the Offline Kit (see docs/OFFLINE_KIT.md).

---

7) Future alignment

• Replace manual export definitions with generated mirror bundle manifests once EXCITITOR-EXPORT-01-007 ships.
• Extend /index payload with quiet-provenance when EXCITITOR-EXPORT-01-006 adds that metadata.
• Integrate domain manifests with DevOps mirror profiles (DEVOPS-MIRROR-08-001) so helm/compose overlays can enable or disable domains declaratively.

---

8) Runbook & observability checklist (Sprint 22 demo refresh · 2025-11-07)

Daily / on-call checks

1. Index freshness – watch excitor_mirror_export_latency_seconds (p95 < 180) grouped by domainId. If latency grows past 10 minutes, verify the export worker queue (stellaops-export-worker logs) and ensure PostgreSQL vex.exports has entries newer than now()-10m.
2. Quota exhaustion – alert on excitor_mirror_quota_exhausted_total{scope="download"} increases. When triggered, inspect structured logs (MirrorDomainId, QuotaScope, RemoteIp) and either raise limits or throttle abusive clients.
3. Bundle signature health – metric excitor_mirror_bundle_signature_verified_total should match download counts when signing enabled. Deltas indicate missing .jws files; rebuild the bundle via export job or copy artefacts from the authority mirror cache.
4. HTTP errors – dashboards should track 4xx/5xx rates split by route; repeated 503 statuses imply misconfigured exports. Check mirror/index logs for status=misconfigured.

Incident steps

1. Use GET /excititor/mirror/domains/{id}/index to capture current manifests. Attach the response to the incident log for reproducibility.
2. For quota incidents, temporarily raise maxIndexRequestsPerHour/maxDownloadRequestsPerHour via the Excititor:Mirror:Domains config override, redeploy, then work with the consuming team on caching.
3. For stale exports, trigger the export job (Excititor.ExportRunner) and confirm the artefacts are written to outputRoot/<domain>.
4. Validate DSSE artefacts by running cosign verify-blob --certificate-rekor-url=<rekor> --bundle <domain>/bundle.json --signature <domain>/bundle.json.jws.

Logging fields (structured)

Field	Description
MirrorDomainId	Domain handling the request (matches id in config).
QuotaScope	index / download, useful when alerting on quota events.
ExportKey	Included in download logs to pinpoint misconfigured exports.
BundleDigest	SHA-256 of the artefact; compare with index payload when debugging corruption.

OTEL signals

• Counters: excitor.mirror.requests, excitor.mirror.quota_blocked, excitor.mirror.signature.failures.
• Histograms: excitor.mirror.download.duration, excitor.mirror.export.latency.
• Spans: mirror.index, mirror.download include attributes mirror.domain, mirror.export.key, and mirror.quota.remaining.

Add these instruments via the MirrorEndpoints middleware; see StellaOps.Excititor.WebService/Telemetry/MirrorMetrics.cs.