165 lines
8.3 KiB
Markdown
165 lines
8.3 KiB
Markdown
# architecture_excititor_mirrors.md — Excititor Mirror Distribution
|
||
|
||
> **Status:** Draft (Sprint 7). Complements `docs/ARCHITECTURE_EXCITITOR.md` by describing the mirror export surface exposed by `Excititor.WebService` and the configuration hooks used by operators and downstream mirrors.
|
||
|
||
---
|
||
|
||
## 0) Purpose
|
||
|
||
Excititor publishes canonical VEX consensus data. Operators (or StellaOps-managed mirrors) need a deterministic way to sync those exports into downstream environments. Mirror distribution provides:
|
||
|
||
* A declarative map of export bundles (`json`, `jsonl`, `openvex`, `csaf`) reachable via signed HTTP endpoints under `/excititor/mirror`.
|
||
* Thin quota/authentication controls on top of the existing export cache so mirrors cannot starve the web service.
|
||
* Stable payload shapes that downstream automation can monitor (index → fetch updates → download artifact → verify signature).
|
||
|
||
Mirror endpoints are intentionally **read-only**. Write paths (export generation, attestation, cache) remain the responsibility of the export pipeline.
|
||
|
||
---
|
||
|
||
## 1) Configuration model
|
||
|
||
The web service reads mirror configuration from `Excititor:Mirror` (YAML/JSON/appsettings). Each domain groups a set of exports that share rate limits and authentication rules.
|
||
|
||
```yaml
|
||
Excititor:
|
||
Mirror:
|
||
Domains:
|
||
- id: primary
|
||
displayName: Primary Mirror
|
||
requireAuthentication: false
|
||
maxIndexRequestsPerHour: 600
|
||
maxDownloadRequestsPerHour: 1200
|
||
exports:
|
||
- key: consensus
|
||
format: json
|
||
filters:
|
||
vulnId: CVE-2025-0001
|
||
productKey: pkg:test/demo
|
||
sort:
|
||
createdAt: false # descending
|
||
limit: 1000
|
||
- key: consensus-openvex
|
||
format: openvex
|
||
filters:
|
||
vulnId: CVE-2025-0001
|
||
```
|
||
|
||
### Root settings
|
||
|
||
| Field | Required | Description |
|
||
| --- | --- | --- |
|
||
| `outputRoot` | – | Filesystem root where mirror artefacts are written. Defaults to the Excititor file-system artifact store root when omitted. |
|
||
| `directoryName` | – | Optional subdirectory created under `outputRoot`; defaults to `mirror`. |
|
||
| `targetRepository` | – | Hint propagated to manifests/index files indicating the operator-visible location (for example `s3://mirror/excititor`). |
|
||
| `signing` | – | Bundle signing configuration. When enabled, the exporter emits a detached JWS (`bundle.json.jws`) alongside each domain bundle. |
|
||
|
||
`signing` supports the following fields:
|
||
|
||
| Field | Required | Description |
|
||
| --- | --- | --- |
|
||
| `enabled` | – | Toggles detached signing for domain bundles. |
|
||
| `algorithm` | – | Signing algorithm identifier (default `ES256`). |
|
||
| `keyId` | ✅ (when `enabled`) | Signing key identifier resolved via the configured crypto provider registry. |
|
||
| `provider` | – | Optional provider hint when multiple registries are available. |
|
||
| `keyPath` | – | Optional PEM path used to seed the provider when the key is not already loaded. |
|
||
|
||
### Domain field reference
|
||
|
||
| Field | Required | Description |
|
||
| --- | --- | --- |
|
||
| `id` | ✅ | Stable identifier. Appears in URLs (`/excititor/mirror/domains/{id}`) and download filenames. |
|
||
| `displayName` | – | Human-friendly label surfaced in the `/domains` listing. Falls back to `id`. |
|
||
| `requireAuthentication` | – | When `true` the service enforces that the caller is authenticated (Authority token). |
|
||
| `maxIndexRequestsPerHour` | – | Per-domain quota for index endpoints. `0`/negative disables the guard. |
|
||
| `maxDownloadRequestsPerHour` | – | Per-domain quota for artifact downloads. |
|
||
| `exports` | ✅ | Collection of export projections. |
|
||
|
||
Export-level fields:
|
||
|
||
| Field | Required | Description |
|
||
| --- | --- | --- |
|
||
| `key` | ✅ | Unique key within the domain. Used in URLs (`/exports/{key}`) and filenames/bundle entries. |
|
||
| `format` | ✅ | One of `json`, `jsonl`, `openvex`, `csaf`. Maps to `VexExportFormat`. |
|
||
| `filters` | – | Key/value pairs executed via `VexQueryFilter`. Keys must match export data source columns (e.g., `vulnId`, `productKey`). |
|
||
| `sort` | – | Key/boolean map (false = descending). |
|
||
| `limit`, `offset`, `view` | – | Optional query bounds passed through to the export query. |
|
||
|
||
⚠️ **Misconfiguration:** invalid formats or missing keys cause exports to be flagged with `status` in the index response; they are not exposed downstream.
|
||
|
||
---
|
||
|
||
## 2) HTTP surface
|
||
|
||
Routes are grouped under `/excititor/mirror`.
|
||
|
||
| Method | Path | Description |
|
||
| --- | --- | --- |
|
||
| `GET` | `/domains` | Returns configured domains with quota metadata. |
|
||
| `GET` | `/domains/{domainId}` | Domain detail (auth/quota + export keys). `404` for unknown domains. |
|
||
| `GET` | `/domains/{domainId}/index` | Lists exports with exportId, query signature, format, artifact digest, attestation metadata, and size. Applies index quota. |
|
||
| `GET` | `/domains/{domainId}/exports/{exportKey}` | Returns manifest metadata (single export). `404` if unknown/missing. |
|
||
| `GET` | `/domains/{domainId}/exports/{exportKey}/download` | Streams export content from the artifact store. Applies download quota. |
|
||
|
||
Responses are serialized via `VexCanonicalJsonSerializer` ensuring stable ordering. Download responses include a content-disposition header naming the file `<domain>-<export>.<ext>`.
|
||
|
||
### Error handling
|
||
|
||
* `401` – authentication required (`requireAuthentication=true`).
|
||
* `404` – domain/export not found or manifest not persisted.
|
||
* `429` – per-domain quota exceeded (`Retry-After` header set in seconds).
|
||
* `503` – export misconfiguration (invalid format/query).
|
||
|
||
---
|
||
|
||
## 3) Rate limiting
|
||
|
||
`MirrorRateLimiter` implements a simple rolling 1-hour window using `IMemoryCache`. Each domain has two quotas:
|
||
|
||
* `index` scope → `maxIndexRequestsPerHour`
|
||
* `download` scope → `maxDownloadRequestsPerHour`
|
||
|
||
`0` or negative limits disable enforcement. Quotas are best-effort (per-instance). For HA deployments, configure sticky routing at the ingress or replace the limiter with a distributed implementation.
|
||
|
||
---
|
||
|
||
## 4) Interaction with export pipeline
|
||
|
||
Mirror endpoints consume manifests produced by the export engine (`MongoVexExportStore`). They do **not** trigger new exports. Operators must configure connectors/exporters to keep targeted exports fresh (see `EXCITITOR-EXPORT-01-005/006/007`).
|
||
|
||
Recommended workflow:
|
||
|
||
1. Define export plans at the export layer (JSON/OpenVEX/CSAF).
|
||
2. Configure mirror domains mapping to those plans.
|
||
3. Downstream mirror automation:
|
||
* `GET /domains/{id}/index`
|
||
* Compare `exportId` / `consensusRevision`
|
||
* `GET /download` when new
|
||
* Verify digest + attestation
|
||
|
||
When the export engine runs, it materializes the following artefacts under `outputRoot/<directoryName>`:
|
||
|
||
- `index.json` – canonical index listing each configured domain, manifest/bundle descriptors (with SHA-256 digests), and available export keys.
|
||
- `<domain>/manifest.json` – per-domain summary with export metadata (query signature, consensus/score digests, source providers) and a descriptor pointing at the bundle.
|
||
- `<domain>/bundle.json` – canonical payload containing serialized consensus, score envelopes, and normalized VEX claims for the matching export definitions.
|
||
- `<domain>/bundle.json.jws` – optional detached JWS when signing is enabled.
|
||
|
||
Downstream automation reads `manifest.json`/`bundle.json` directly, while `/excititor/mirror` endpoints stream the same artefacts through authenticated HTTP.
|
||
|
||
---
|
||
|
||
## 5) Operational guidance
|
||
|
||
* Track quota utilisation via HTTP 429 metrics (configure structured logging or OTEL counters when rate limiting triggers).
|
||
* Mirror domains can be deployed per tenant (e.g., `tenant-a`, `tenant-b`) with different auth requirements.
|
||
* Ensure the underlying artifact stores (`FileSystem`, `S3`, offline bundle) retain artefacts long enough for mirrors to sync.
|
||
* For air-gapped mirrors, combine mirror endpoints with the Offline Kit (see `docs/24_OFFLINE_KIT.md`).
|
||
|
||
---
|
||
|
||
## 6) Future alignment
|
||
|
||
* Replace manual export definitions with generated mirror bundle manifests once `EXCITITOR-EXPORT-01-007` ships.
|
||
* Extend `/index` payload with quiet-provenance when `EXCITITOR-EXPORT-01-006` adds that metadata.
|
||
* Integrate domain manifests with DevOps mirror profiles (`DEVOPS-MIRROR-08-001`) so helm/compose overlays can enable or disable domains declaratively.
|
||
|