git.stella-ops.org/docs/modules/excititor/mirrors.md

architecture_excititor_mirrors.md — Excititor Mirror Distribution

Status: Draft (Sprint 7). Complements docs/modules/excititor/architecture.md by describing the mirror export surface exposed by Excititor.WebService and the configuration hooks used by operators and downstream mirrors.

---

0) Purpose

Excititor publishes canonical VEX consensus data. Operators (or StellaOps-managed mirrors) need a deterministic way to sync those exports into downstream environments. Mirror distribution provides:

• A declarative map of export bundles (json, jsonl, openvex, csaf) reachable via signed HTTP endpoints under /excititor/mirror.
• Thin quota/authentication controls on top of the existing export cache so mirrors cannot starve the web service.
• Stable payload shapes that downstream automation can monitor (index → fetch updates → download artifact → verify signature).

Mirror endpoints are intentionally read-only. Write paths (export generation, attestation, cache) remain the responsibility of the export pipeline.

---

1) Configuration model

The web service reads mirror configuration from Excititor:Mirror (YAML/JSON/appsettings). Each domain groups a set of exports that share rate limits and authentication rules.

Excititor:
  Mirror:
    Domains:
      - id: primary
        displayName: Primary Mirror
        requireAuthentication: false
        maxIndexRequestsPerHour: 600
        maxDownloadRequestsPerHour: 1200
        exports:
          - key: consensus
            format: json
            filters:
              vulnId: CVE-2025-0001
              productKey: pkg:test/demo
            sort:
              createdAt: false     # descending
            limit: 1000
          - key: consensus-openvex
            format: openvex
            filters:
              vulnId: CVE-2025-0001

Root settings

Field	Required	Description
outputRoot	–	Filesystem root where mirror artefacts are written. Defaults to the Excititor file-system artifact store root when omitted.
directoryName	–	Optional subdirectory created under outputRoot; defaults to mirror.
targetRepository	–	Hint propagated to manifests/index files indicating the operator-visible location (for example s3://mirror/excititor).
signing	–	Bundle signing configuration. When enabled, the exporter emits a detached JWS (bundle.json.jws) alongside each domain bundle.

signing supports the following fields:

Field	Required	Description
enabled	–	Toggles detached signing for domain bundles.
algorithm	–	Signing algorithm identifier (default ES256).
keyId	✅ (when enabled)	Signing key identifier resolved via the configured crypto provider registry.
provider	–	Optional provider hint when multiple registries are available.
keyPath	–	Optional PEM path used to seed the provider when the key is not already loaded.

Domain field reference

Field	Required	Description
id	✅	Stable identifier. Appears in URLs (/excititor/mirror/domains/{id}) and download filenames.
displayName	–	Human-friendly label surfaced in the /domains listing. Falls back to id.
requireAuthentication	–	When true the service enforces that the caller is authenticated (Authority token).
maxIndexRequestsPerHour	–	Per-domain quota for index endpoints. 0/negative disables the guard.
maxDownloadRequestsPerHour	–	Per-domain quota for artifact downloads.
exports	✅	Collection of export projections.

Export-level fields:

Field	Required	Description
key	✅	Unique key within the domain. Used in URLs (/exports/{key}) and filenames/bundle entries.
format	✅	One of json, jsonl, openvex, csaf. Maps to VexExportFormat.
filters	–	Key/value pairs executed via VexQueryFilter. Keys must match export data source columns (e.g., vulnId, productKey).
sort	–	Key/boolean map (false = descending).
limit, offset, view	–	Optional query bounds passed through to the export query.

⚠️ Misconfiguration: invalid formats or missing keys cause exports to be flagged with status in the index response; they are not exposed downstream.

---

2) HTTP surface

Routes are grouped under /excititor/mirror.

Method	Path	Description
GET	/domains	Returns configured domains with quota metadata.
GET	/domains/{domainId}	Domain detail (auth/quota + export keys). 404 for unknown domains.
GET	/domains/{domainId}/index	Lists exports with exportId, query signature, format, artifact digest, attestation metadata, and size. Applies index quota.
GET	/domains/{domainId}/exports/{exportKey}	Returns manifest metadata (single export). 404 if unknown/missing.
GET	/domains/{domainId}/exports/{exportKey}/download	Streams export content from the artifact store. Applies download quota.

Responses are serialized via VexCanonicalJsonSerializer ensuring stable ordering. Download responses include a content-disposition header naming the file <domain>-<export>.<ext>.

Error handling

• 401 – authentication required (requireAuthentication=true).
• 404 – domain/export not found or manifest not persisted.
• 429 – per-domain quota exceeded (Retry-After header set in seconds).
• 503 – export misconfiguration (invalid format/query).

---

3) Rate limiting

MirrorRateLimiter implements a simple rolling 1-hour window using IMemoryCache. Each domain has two quotas:

• index scope → maxIndexRequestsPerHour
• download scope → maxDownloadRequestsPerHour

0 or negative limits disable enforcement. Quotas are best-effort (per-instance). For HA deployments, configure sticky routing at the ingress or replace the limiter with a distributed implementation.

---

4) Interaction with export pipeline

Mirror endpoints consume manifests produced by the export engine (MongoVexExportStore). They do not trigger new exports. Operators must configure connectors/exporters to keep targeted exports fresh (see EXCITITOR-EXPORT-01-005/006/007).

Recommended workflow:

1. Define export plans at the export layer (JSON/OpenVEX/CSAF).
2. Configure mirror domains mapping to those plans.
3. Downstream mirror automation:
   • GET /domains/{id}/index
   • Compare exportId / consensusRevision
   • GET /download when new
   • Verify digest + attestation

When the export engine runs, it materializes the following artefacts under outputRoot/<directoryName>:

• index.json – canonical index listing each configured domain, manifest/bundle descriptors (with SHA-256 digests), and available export keys.
• <domain>/manifest.json – per-domain summary with export metadata (query signature, consensus/score digests, source providers) and a descriptor pointing at the bundle.
• <domain>/bundle.json – canonical payload containing serialized consensus, score envelopes, and normalized VEX claims for the matching export definitions.
• <domain>/bundle.json.jws – optional detached JWS when signing is enabled.

Downstream automation reads manifest.json/bundle.json directly, while /excititor/mirror endpoints stream the same artefacts through authenticated HTTP.

---

5) Operational guidance

• Track quota utilisation via HTTP 429 metrics (configure structured logging or OTEL counters when rate limiting triggers).
• Mirror domains can be deployed per tenant (e.g., tenant-a, tenant-b) with different auth requirements.
• Ensure the underlying artifact stores (FileSystem, S3, offline bundle) retain artefacts long enough for mirrors to sync.
• For air-gapped mirrors, combine mirror endpoints with the Offline Kit (see docs/24_OFFLINE_KIT.md).

---

6) Future alignment

• Replace manual export definitions with generated mirror bundle manifests once EXCITITOR-EXPORT-01-007 ships.
• Extend /index payload with quiet-provenance when EXCITITOR-EXPORT-01-006 adds that metadata.
• Integrate domain manifests with DevOps mirror profiles (DEVOPS-MIRROR-08-001) so helm/compose overlays can enable or disable domains declaratively.