master
5ce40d2eeb
- Added Program.cs to set up the web application with Serilog for logging, health check endpoints, and a placeholder admission endpoint. - Configured Kestrel server to use TLS 1.3 and handle client certificates appropriately. - Created StellaOps.Zastava.Webhook.csproj with necessary dependencies including Serilog and Polly. - Documented tasks in TASKS.md for the Zastava Webhook project, outlining current work and exit criteria for each task.
139 lines
6.4 KiB
Markdown
139 lines
6.4 KiB
Markdown
# architecture_excititor_mirrors.md — Excititor Mirror Distribution
|
||
|
||
> **Status:** Draft (Sprint 7). Complements `docs/ARCHITECTURE_EXCITITOR.md` by describing the mirror export surface exposed by `Excititor.WebService` and the configuration hooks used by operators and downstream mirrors.
|
||
|
||
---
|
||
|
||
## 0) Purpose
|
||
|
||
Excititor publishes canonical VEX consensus data. Operators (or StellaOps-managed mirrors) need a deterministic way to sync those exports into downstream environments. Mirror distribution provides:
|
||
|
||
* A declarative map of export bundles (`json`, `jsonl`, `openvex`, `csaf`) reachable via signed HTTP endpoints under `/excititor/mirror`.
|
||
* Thin quota/authentication controls on top of the existing export cache so mirrors cannot starve the web service.
|
||
* Stable payload shapes that downstream automation can monitor (index → fetch updates → download artifact → verify signature).
|
||
|
||
Mirror endpoints are intentionally **read-only**. Write paths (export generation, attestation, cache) remain the responsibility of the export pipeline.
|
||
|
||
---
|
||
|
||
## 1) Configuration model
|
||
|
||
The web service reads mirror configuration from `Excititor:Mirror` (YAML/JSON/appsettings). Each domain groups a set of exports that share rate limits and authentication rules.
|
||
|
||
```yaml
|
||
Excititor:
|
||
Mirror:
|
||
Domains:
|
||
- id: primary
|
||
displayName: Primary Mirror
|
||
requireAuthentication: false
|
||
maxIndexRequestsPerHour: 600
|
||
maxDownloadRequestsPerHour: 1200
|
||
exports:
|
||
- key: consensus
|
||
format: json
|
||
filters:
|
||
vulnId: CVE-2025-0001
|
||
productKey: pkg:test/demo
|
||
sort:
|
||
createdAt: false # descending
|
||
limit: 1000
|
||
- key: consensus-openvex
|
||
format: openvex
|
||
filters:
|
||
vulnId: CVE-2025-0001
|
||
```
|
||
|
||
### Field reference
|
||
|
||
| Field | Required | Description |
|
||
| --- | --- | --- |
|
||
| `id` | ✅ | Stable identifier. Appears in URLs (`/excititor/mirror/domains/{id}`) and download filenames. |
|
||
| `displayName` | – | Human-friendly label surfaced in the `/domains` listing. Falls back to `id`. |
|
||
| `requireAuthentication` | – | When `true` the service enforces that the caller is authenticated (Authority token). |
|
||
| `maxIndexRequestsPerHour` | – | Per-domain quota for index endpoints. `0`/negative disables the guard. |
|
||
| `maxDownloadRequestsPerHour` | – | Per-domain quota for artifact downloads. |
|
||
| `exports` | ✅ | Collection of export projections. |
|
||
|
||
Export-level fields:
|
||
|
||
| Field | Required | Description |
|
||
| --- | --- | --- |
|
||
| `key` | ✅ | Unique key within the domain. Used in URLs (`/exports/{key}`) and filenames. |
|
||
| `format` | ✅ | One of `json`, `jsonl`, `openvex`, `csaf`. Maps to `VexExportFormat`. |
|
||
| `filters` | – | Key/value pairs executed via `VexQueryFilter`. Keys must match export data source columns (e.g., `vulnId`, `productKey`). |
|
||
| `sort` | – | Key/boolean map (false = descending). |
|
||
| `limit`, `offset`, `view` | – | Optional query bounds passed through to the export query. |
|
||
|
||
⚠️ **Misconfiguration:** invalid formats or missing keys cause exports to be flagged with `status` in the index response; they are not exposed downstream.
|
||
|
||
---
|
||
|
||
## 2) HTTP surface
|
||
|
||
Routes are grouped under `/excititor/mirror`.
|
||
|
||
| Method | Path | Description |
|
||
| --- | --- | --- |
|
||
| `GET` | `/domains` | Returns configured domains with quota metadata. |
|
||
| `GET` | `/domains/{domainId}` | Domain detail (auth/quota + export keys). `404` for unknown domains. |
|
||
| `GET` | `/domains/{domainId}/index` | Lists exports with exportId, query signature, format, artifact digest, attestation metadata, and size. Applies index quota. |
|
||
| `GET` | `/domains/{domainId}/exports/{exportKey}` | Returns manifest metadata (single export). `404` if unknown/missing. |
|
||
| `GET` | `/domains/{domainId}/exports/{exportKey}/download` | Streams export content from the artifact store. Applies download quota. |
|
||
|
||
Responses are serialized via `VexCanonicalJsonSerializer` ensuring stable ordering. Download responses include a content-disposition header naming the file `<domain>-<export>.<ext>`.
|
||
|
||
### Error handling
|
||
|
||
* `401` – authentication required (`requireAuthentication=true`).
|
||
* `404` – domain/export not found or manifest not persisted.
|
||
* `429` – per-domain quota exceeded (`Retry-After` header set in seconds).
|
||
* `503` – export misconfiguration (invalid format/query).
|
||
|
||
---
|
||
|
||
## 3) Rate limiting
|
||
|
||
`MirrorRateLimiter` implements a simple rolling 1-hour window using `IMemoryCache`. Each domain has two quotas:
|
||
|
||
* `index` scope → `maxIndexRequestsPerHour`
|
||
* `download` scope → `maxDownloadRequestsPerHour`
|
||
|
||
`0` or negative limits disable enforcement. Quotas are best-effort (per-instance). For HA deployments, configure sticky routing at the ingress or replace the limiter with a distributed implementation.
|
||
|
||
---
|
||
|
||
## 4) Interaction with export pipeline
|
||
|
||
Mirror endpoints consume manifests produced by the export engine (`MongoVexExportStore`). They do **not** trigger new exports. Operators must configure connectors/exporters to keep targeted exports fresh (see `EXCITITOR-EXPORT-01-005/006/007`).
|
||
|
||
Recommended workflow:
|
||
|
||
1. Define export plans at the export layer (JSON/OpenVEX/CSAF).
|
||
2. Configure mirror domains mapping to those plans.
|
||
3. Downstream mirror automation:
|
||
* `GET /domains/{id}/index`
|
||
* Compare `exportId` / `consensusRevision`
|
||
* `GET /download` when new
|
||
* Verify digest + attestation
|
||
|
||
When the export team lands deterministic mirror bundles (Sprint 7 tasks 01-005/006/007), these configurations can be generated automatically.
|
||
|
||
---
|
||
|
||
## 5) Operational guidance
|
||
|
||
* Track quota utilisation via HTTP 429 metrics (configure structured logging or OTEL counters when rate limiting triggers).
|
||
* Mirror domains can be deployed per tenant (e.g., `tenant-a`, `tenant-b`) with different auth requirements.
|
||
* Ensure the underlying artifact stores (`FileSystem`, `S3`, offline bundle) retain artefacts long enough for mirrors to sync.
|
||
* For air-gapped mirrors, combine mirror endpoints with the Offline Kit (see `docs/24_OFFLINE_KIT.md`).
|
||
|
||
---
|
||
|
||
## 6) Future alignment
|
||
|
||
* Replace manual export definitions with generated mirror bundle manifests once `EXCITITOR-EXPORT-01-007` ships.
|
||
* Extend `/index` payload with quiet-provenance when `EXCITITOR-EXPORT-01-006` adds that metadata.
|
||
* Integrate domain manifests with DevOps mirror profiles (`DEVOPS-MIRROR-08-001`) so helm/compose overlays can enable or disable domains declaratively.
|
||
|