Resolve Concelier/Excititor merge conflicts
This commit is contained in:
root
138
docs/ARCHITECTURE_EXCITITOR_MIRRORS.md
Normal file
138
docs/ARCHITECTURE_EXCITITOR_MIRRORS.md
Normal file
@@ -0,0 +1,138 @@
|
||||
# architecture_excititor_mirrors.md — Excititor Mirror Distribution
|
||||
|
||||
> **Status:** Draft (Sprint 7). Complements `docs/ARCHITECTURE_EXCITITOR.md` by describing the mirror export surface exposed by `Excititor.WebService` and the configuration hooks used by operators and downstream mirrors.
|
||||
|
||||
---
|
||||
|
||||
## 0) Purpose
|
||||
|
||||
Excititor publishes canonical VEX consensus data. Operators (or StellaOps-managed mirrors) need a deterministic way to sync those exports into downstream environments. Mirror distribution provides:
|
||||
|
||||
* A declarative map of export bundles (`json`, `jsonl`, `openvex`, `csaf`) reachable via signed HTTP endpoints under `/excititor/mirror`.
|
||||
* Thin quota/authentication controls on top of the existing export cache so mirrors cannot starve the web service.
|
||||
* Stable payload shapes that downstream automation can monitor (index → fetch updates → download artifact → verify signature).
|
||||
|
||||
Mirror endpoints are intentionally **read-only**. Write paths (export generation, attestation, cache) remain the responsibility of the export pipeline.
|
||||
|
||||
---
|
||||
|
||||
## 1) Configuration model
|
||||
|
||||
The web service reads mirror configuration from `Excititor:Mirror` (YAML/JSON/appsettings). Each domain groups a set of exports that share rate limits and authentication rules.
|
||||
|
||||
```yaml
|
||||
Excititor:
|
||||
Mirror:
|
||||
Domains:
|
||||
- id: primary
|
||||
displayName: Primary Mirror
|
||||
requireAuthentication: false
|
||||
maxIndexRequestsPerHour: 600
|
||||
maxDownloadRequestsPerHour: 1200
|
||||
exports:
|
||||
- key: consensus
|
||||
format: json
|
||||
filters:
|
||||
vulnId: CVE-2025-0001
|
||||
productKey: pkg:test/demo
|
||||
sort:
|
||||
createdAt: false # descending
|
||||
limit: 1000
|
||||
- key: consensus-openvex
|
||||
format: openvex
|
||||
filters:
|
||||
vulnId: CVE-2025-0001
|
||||
```
|
||||
|
||||
### Field reference
|
||||
|
||||
| Field | Required | Description |
|
||||
| --- | --- | --- |
|
||||
| `id` | ✅ | Stable identifier. Appears in URLs (`/excititor/mirror/domains/{id}`) and download filenames. |
|
||||
| `displayName` | – | Human-friendly label surfaced in the `/domains` listing. Falls back to `id`. |
|
||||
| `requireAuthentication` | – | When `true` the service enforces that the caller is authenticated (Authority token). |
|
||||
| `maxIndexRequestsPerHour` | – | Per-domain quota for index endpoints. `0`/negative disables the guard. |
|
||||
| `maxDownloadRequestsPerHour` | – | Per-domain quota for artifact downloads. |
|
||||
| `exports` | ✅ | Collection of export projections. |
|
||||
|
||||
Export-level fields:
|
||||
|
||||
| Field | Required | Description |
|
||||
| --- | --- | --- |
|
||||
| `key` | ✅ | Unique key within the domain. Used in URLs (`/exports/{key}`) and filenames. |
|
||||
| `format` | ✅ | One of `json`, `jsonl`, `openvex`, `csaf`. Maps to `VexExportFormat`. |
|
||||
| `filters` | – | Key/value pairs executed via `VexQueryFilter`. Keys must match export data source columns (e.g., `vulnId`, `productKey`). |
|
||||
| `sort` | – | Key/boolean map (false = descending). |
|
||||
| `limit`, `offset`, `view` | – | Optional query bounds passed through to the export query. |
|
||||
|
||||
⚠️ **Misconfiguration:** invalid formats or missing keys cause exports to be flagged with `status` in the index response; they are not exposed downstream.
|
||||
|
||||
---
|
||||
|
||||
## 2) HTTP surface
|
||||
|
||||
Routes are grouped under `/excititor/mirror`.
|
||||
|
||||
| Method | Path | Description |
|
||||
| --- | --- | --- |
|
||||
| `GET` | `/domains` | Returns configured domains with quota metadata. |
|
||||
| `GET` | `/domains/{domainId}` | Domain detail (auth/quota + export keys). `404` for unknown domains. |
|
||||
| `GET` | `/domains/{domainId}/index` | Lists exports with exportId, query signature, format, artifact digest, attestation metadata, and size. Applies index quota. |
|
||||
| `GET` | `/domains/{domainId}/exports/{exportKey}` | Returns manifest metadata (single export). `404` if unknown/missing. |
|
||||
| `GET` | `/domains/{domainId}/exports/{exportKey}/download` | Streams export content from the artifact store. Applies download quota. |
|
||||
|
||||
Responses are serialized via `VexCanonicalJsonSerializer` ensuring stable ordering. Download responses include a content-disposition header naming the file `<domain>-<export>.<ext>`.
|
||||
|
||||
### Error handling
|
||||
|
||||
* `401` – authentication required (`requireAuthentication=true`).
|
||||
* `404` – domain/export not found or manifest not persisted.
|
||||
* `429` – per-domain quota exceeded (`Retry-After` header set in seconds).
|
||||
* `503` – export misconfiguration (invalid format/query).
|
||||
|
||||
---
|
||||
|
||||
## 3) Rate limiting
|
||||
|
||||
`MirrorRateLimiter` implements a simple rolling 1-hour window using `IMemoryCache`. Each domain has two quotas:
|
||||
|
||||
* `index` scope → `maxIndexRequestsPerHour`
|
||||
* `download` scope → `maxDownloadRequestsPerHour`
|
||||
|
||||
`0` or negative limits disable enforcement. Quotas are best-effort (per-instance). For HA deployments, configure sticky routing at the ingress or replace the limiter with a distributed implementation.
|
||||
|
||||
---
|
||||
|
||||
## 4) Interaction with export pipeline
|
||||
|
||||
Mirror endpoints consume manifests produced by the export engine (`MongoVexExportStore`). They do **not** trigger new exports. Operators must configure connectors/exporters to keep targeted exports fresh (see `EXCITITOR-EXPORT-01-005/006/007`).
|
||||
|
||||
Recommended workflow:
|
||||
|
||||
1. Define export plans at the export layer (JSON/OpenVEX/CSAF).
|
||||
2. Configure mirror domains mapping to those plans.
|
||||
3. Downstream mirror automation:
|
||||
* `GET /domains/{id}/index`
|
||||
* Compare `exportId` / `consensusRevision`
|
||||
* `GET /download` when new
|
||||
* Verify digest + attestation
|
||||
|
||||
When the export team lands deterministic mirror bundles (Sprint 7 tasks 01-005/006/007), these configurations can be generated automatically.
|
||||
|
||||
---
|
||||
|
||||
## 5) Operational guidance
|
||||
|
||||
* Track quota utilisation via HTTP 429 metrics (configure structured logging or OTEL counters when rate limiting triggers).
|
||||
* Mirror domains can be deployed per tenant (e.g., `tenant-a`, `tenant-b`) with different auth requirements.
|
||||
* Ensure the underlying artifact stores (`FileSystem`, `S3`, offline bundle) retain artefacts long enough for mirrors to sync.
|
||||
* For air-gapped mirrors, combine mirror endpoints with the Offline Kit (see `docs/24_OFFLINE_KIT.md`).
|
||||
|
||||
---
|
||||
|
||||
## 6) Future alignment
|
||||
|
||||
* Replace manual export definitions with generated mirror bundle manifests once `EXCITITOR-EXPORT-01-007` ships.
|
||||
* Extend `/index` payload with quiet-provenance when `EXCITITOR-EXPORT-01-006` adds that metadata.
|
||||
* Integrate domain manifests with DevOps mirror profiles (`DEVOPS-MIRROR-08-001`) so helm/compose overlays can enable or disable domains declaratively.
|
||||
|
||||
Reference in New Issue
Block a user