stella-ops.org/git.stella-ops.org
1
0
Fork 0
Code Issues Pull Requests Actions 1 Releases Wiki Activity
Files
8e7ce55542fea7fabb0e31cb3721357af1a42a8d
git.stella-ops.org/docs/ARCHITECTURE_EXCITITOR_MIRRORS.md
master 5ce40d2eeb feat: Initialize Zastava Webhook service with TLS and Authority authentication 
- Added Program.cs to set up the web application with Serilog for logging, health check endpoints, and a placeholder admission endpoint.
- Configured Kestrel server to use TLS 1.3 and handle client certificates appropriately.
- Created StellaOps.Zastava.Webhook.csproj with necessary dependencies including Serilog and Polly.
- Documented tasks in TASKS.md for the Zastava Webhook project, outlining current work and exit criteria for each task.
2025-10-19 18:36:22 +03:00

6.4 KiB
Raw Blame History

architecture_excititor_mirrors.md — Excititor Mirror Distribution

Status: Draft (Sprint 7). Complements docs/ARCHITECTURE_EXCITITOR.md by describing the mirror export surface exposed by Excititor.WebService and the configuration hooks used by operators and downstream mirrors.

0) Purpose

Excititor publishes canonical VEX consensus data. Operators (or StellaOps-managed mirrors) need a deterministic way to sync those exports into downstream environments. Mirror distribution provides:

  • A declarative map of export bundles (json, jsonl, openvex, csaf) reachable via signed HTTP endpoints under /excititor/mirror.
  • Thin quota/authentication controls on top of the existing export cache so mirrors cannot starve the web service.
  • Stable payload shapes that downstream automation can monitor (index → fetch updates → download artifact → verify signature).

Mirror endpoints are intentionally read-only. Write paths (export generation, attestation, cache) remain the responsibility of the export pipeline.

1) Configuration model

The web service reads mirror configuration from Excititor:Mirror (YAML/JSON/appsettings). Each domain groups a set of exports that share rate limits and authentication rules.

Excititor:
  Mirror:
    Domains:
      - id: primary
        displayName: Primary Mirror
        requireAuthentication: false
        maxIndexRequestsPerHour: 600
        maxDownloadRequestsPerHour: 1200
        exports:
          - key: consensus
            format: json
            filters:
              vulnId: CVE-2025-0001
              productKey: pkg:test/demo
            sort:
              createdAt: false     # descending
            limit: 1000
          - key: consensus-openvex
            format: openvex
            filters:
              vulnId: CVE-2025-0001

Field reference

Field Required Description
id ✅ Stable identifier. Appears in URLs (/excititor/mirror/domains/{id}) and download filenames.
displayName – Human-friendly label surfaced in the /domains listing. Falls back to id.
requireAuthentication – When true the service enforces that the caller is authenticated (Authority token).
maxIndexRequestsPerHour – Per-domain quota for index endpoints. 0/negative disables the guard.
maxDownloadRequestsPerHour – Per-domain quota for artifact downloads.
exports ✅ Collection of export projections.

Export-level fields:

Field Required Description
key ✅ Unique key within the domain. Used in URLs (/exports/{key}) and filenames.
format ✅ One of json, jsonl, openvex, csaf. Maps to VexExportFormat.
filters – Key/value pairs executed via VexQueryFilter. Keys must match export data source columns (e.g., vulnId, productKey).
sort – Key/boolean map (false = descending).
limit, offset, view – Optional query bounds passed through to the export query.

⚠️ Misconfiguration: invalid formats or missing keys cause exports to be flagged with status in the index response; they are not exposed downstream.

2) HTTP surface

Routes are grouped under /excititor/mirror.

Method Path Description
GET /domains Returns configured domains with quota metadata.
GET /domains/{domainId} Domain detail (auth/quota + export keys). 404 for unknown domains.
GET /domains/{domainId}/index Lists exports with exportId, query signature, format, artifact digest, attestation metadata, and size. Applies index quota.
GET /domains/{domainId}/exports/{exportKey} Returns manifest metadata (single export). 404 if unknown/missing.
GET /domains/{domainId}/exports/{exportKey}/download Streams export content from the artifact store. Applies download quota.

Responses are serialized via VexCanonicalJsonSerializer ensuring stable ordering. Download responses include a content-disposition header naming the file <domain>-<export>.<ext>.

Error handling

  • 401 – authentication required (requireAuthentication=true).
  • 404 – domain/export not found or manifest not persisted.
  • 429 – per-domain quota exceeded (Retry-After header set in seconds).
  • 503 – export misconfiguration (invalid format/query).

3) Rate limiting

MirrorRateLimiter implements a simple rolling 1-hour window using IMemoryCache. Each domain has two quotas:

  • index scope → maxIndexRequestsPerHour
  • download scope → maxDownloadRequestsPerHour

0 or negative limits disable enforcement. Quotas are best-effort (per-instance). For HA deployments, configure sticky routing at the ingress or replace the limiter with a distributed implementation.

4) Interaction with export pipeline

Mirror endpoints consume manifests produced by the export engine (MongoVexExportStore). They do not trigger new exports. Operators must configure connectors/exporters to keep targeted exports fresh (see EXCITITOR-EXPORT-01-005/006/007).

Recommended workflow:

  1. Define export plans at the export layer (JSON/OpenVEX/CSAF).
  2. Configure mirror domains mapping to those plans.
  3. Downstream mirror automation:
    • GET /domains/{id}/index
    • Compare exportId / consensusRevision
    • GET /download when new
    • Verify digest + attestation

When the export team lands deterministic mirror bundles (Sprint 7 tasks 01-005/006/007), these configurations can be generated automatically.

5) Operational guidance

  • Track quota utilisation via HTTP 429 metrics (configure structured logging or OTEL counters when rate limiting triggers).
  • Mirror domains can be deployed per tenant (e.g., tenant-a, tenant-b) with different auth requirements.
  • Ensure the underlying artifact stores (FileSystem, S3, offline bundle) retain artefacts long enough for mirrors to sync.
  • For air-gapped mirrors, combine mirror endpoints with the Offline Kit (see docs/24_OFFLINE_KIT.md).

6) Future alignment

  • Replace manual export definitions with generated mirror bundle manifests once EXCITITOR-EXPORT-01-007 ships.
  • Extend /index payload with quiet-provenance when EXCITITOR-EXPORT-01-006 adds that metadata.
  • Integrate domain manifests with DevOps mirror profiles (DEVOPS-MIRROR-08-001) so helm/compose overlays can enable or disable domains declaratively.