stella-ops.org/git.stella-ops.org
1
0
Fork 0
Code Issues Pull Requests Actions 1 Releases Wiki Activity
Files
96d52884e86650452c9c998d343469ea00fafdce
git.stella-ops.org/docs/ARCHITECTURE_EXCITITOR_MIRRORS.md
master cfaea5efd9
Some checks failed
Docs CI / lint-and-preview (push) Has been cancelled
Details
Update
2025-10-21 18:54:26 +03:00

8.3 KiB
Raw Blame History

architecture_excititor_mirrors.md — Excititor Mirror Distribution

Status: Draft (Sprint 7). Complements docs/ARCHITECTURE_EXCITITOR.md by describing the mirror export surface exposed by Excititor.WebService and the configuration hooks used by operators and downstream mirrors.

0) Purpose

Excititor publishes canonical VEX consensus data. Operators (or StellaOps-managed mirrors) need a deterministic way to sync those exports into downstream environments. Mirror distribution provides:

  • A declarative map of export bundles (json, jsonl, openvex, csaf) reachable via signed HTTP endpoints under /excititor/mirror.
  • Thin quota/authentication controls on top of the existing export cache so mirrors cannot starve the web service.
  • Stable payload shapes that downstream automation can monitor (index → fetch updates → download artifact → verify signature).

Mirror endpoints are intentionally read-only. Write paths (export generation, attestation, cache) remain the responsibility of the export pipeline.

1) Configuration model

The web service reads mirror configuration from Excititor:Mirror (YAML/JSON/appsettings). Each domain groups a set of exports that share rate limits and authentication rules.

Excititor:
  Mirror:
    Domains:
      - id: primary
        displayName: Primary Mirror
        requireAuthentication: false
        maxIndexRequestsPerHour: 600
        maxDownloadRequestsPerHour: 1200
        exports:
          - key: consensus
            format: json
            filters:
              vulnId: CVE-2025-0001
              productKey: pkg:test/demo
            sort:
              createdAt: false     # descending
            limit: 1000
          - key: consensus-openvex
            format: openvex
            filters:
              vulnId: CVE-2025-0001

Root settings

Field Required Description
outputRoot – Filesystem root where mirror artefacts are written. Defaults to the Excititor file-system artifact store root when omitted.
directoryName – Optional subdirectory created under outputRoot; defaults to mirror.
targetRepository – Hint propagated to manifests/index files indicating the operator-visible location (for example s3://mirror/excititor).
signing – Bundle signing configuration. When enabled, the exporter emits a detached JWS (bundle.json.jws) alongside each domain bundle.

signing supports the following fields:

Field Required Description
enabled – Toggles detached signing for domain bundles.
algorithm – Signing algorithm identifier (default ES256).
keyId ✅ (when enabled) Signing key identifier resolved via the configured crypto provider registry.
provider – Optional provider hint when multiple registries are available.
keyPath – Optional PEM path used to seed the provider when the key is not already loaded.

Domain field reference

Field Required Description
id ✅ Stable identifier. Appears in URLs (/excititor/mirror/domains/{id}) and download filenames.
displayName – Human-friendly label surfaced in the /domains listing. Falls back to id.
requireAuthentication – When true the service enforces that the caller is authenticated (Authority token).
maxIndexRequestsPerHour – Per-domain quota for index endpoints. 0/negative disables the guard.
maxDownloadRequestsPerHour – Per-domain quota for artifact downloads.
exports ✅ Collection of export projections.

Export-level fields:

Field Required Description
key ✅ Unique key within the domain. Used in URLs (/exports/{key}) and filenames/bundle entries.
format ✅ One of json, jsonl, openvex, csaf. Maps to VexExportFormat.
filters – Key/value pairs executed via VexQueryFilter. Keys must match export data source columns (e.g., vulnId, productKey).
sort – Key/boolean map (false = descending).
limit, offset, view – Optional query bounds passed through to the export query.

⚠️ Misconfiguration: invalid formats or missing keys cause exports to be flagged with status in the index response; they are not exposed downstream.

2) HTTP surface

Routes are grouped under /excititor/mirror.

Method Path Description
GET /domains Returns configured domains with quota metadata.
GET /domains/{domainId} Domain detail (auth/quota + export keys). 404 for unknown domains.
GET /domains/{domainId}/index Lists exports with exportId, query signature, format, artifact digest, attestation metadata, and size. Applies index quota.
GET /domains/{domainId}/exports/{exportKey} Returns manifest metadata (single export). 404 if unknown/missing.
GET /domains/{domainId}/exports/{exportKey}/download Streams export content from the artifact store. Applies download quota.

Responses are serialized via VexCanonicalJsonSerializer ensuring stable ordering. Download responses include a content-disposition header naming the file <domain>-<export>.<ext>.

Error handling

  • 401 – authentication required (requireAuthentication=true).
  • 404 – domain/export not found or manifest not persisted.
  • 429 – per-domain quota exceeded (Retry-After header set in seconds).
  • 503 – export misconfiguration (invalid format/query).

3) Rate limiting

MirrorRateLimiter implements a simple rolling 1-hour window using IMemoryCache. Each domain has two quotas:

  • index scope → maxIndexRequestsPerHour
  • download scope → maxDownloadRequestsPerHour

0 or negative limits disable enforcement. Quotas are best-effort (per-instance). For HA deployments, configure sticky routing at the ingress or replace the limiter with a distributed implementation.

4) Interaction with export pipeline

Mirror endpoints consume manifests produced by the export engine (MongoVexExportStore). They do not trigger new exports. Operators must configure connectors/exporters to keep targeted exports fresh (see EXCITITOR-EXPORT-01-005/006/007).

Recommended workflow:

  1. Define export plans at the export layer (JSON/OpenVEX/CSAF).
  2. Configure mirror domains mapping to those plans.
  3. Downstream mirror automation:
    • GET /domains/{id}/index
    • Compare exportId / consensusRevision
    • GET /download when new
    • Verify digest + attestation

When the export engine runs, it materializes the following artefacts under outputRoot/<directoryName>:

  • index.json – canonical index listing each configured domain, manifest/bundle descriptors (with SHA-256 digests), and available export keys.
  • <domain>/manifest.json – per-domain summary with export metadata (query signature, consensus/score digests, source providers) and a descriptor pointing at the bundle.
  • <domain>/bundle.json – canonical payload containing serialized consensus, score envelopes, and normalized VEX claims for the matching export definitions.
  • <domain>/bundle.json.jws – optional detached JWS when signing is enabled.

Downstream automation reads manifest.json/bundle.json directly, while /excititor/mirror endpoints stream the same artefacts through authenticated HTTP.

5) Operational guidance

  • Track quota utilisation via HTTP 429 metrics (configure structured logging or OTEL counters when rate limiting triggers).
  • Mirror domains can be deployed per tenant (e.g., tenant-a, tenant-b) with different auth requirements.
  • Ensure the underlying artifact stores (FileSystem, S3, offline bundle) retain artefacts long enough for mirrors to sync.
  • For air-gapped mirrors, combine mirror endpoints with the Offline Kit (see docs/24_OFFLINE_KIT.md).

6) Future alignment

  • Replace manual export definitions with generated mirror bundle manifests once EXCITITOR-EXPORT-01-007 ships.
  • Extend /index payload with quiet-provenance when EXCITITOR-EXPORT-01-006 adds that metadata.
  • Integrate domain manifests with DevOps mirror profiles (DEVOPS-MIRROR-08-001) so helm/compose overlays can enable or disable domains declaratively.