Update documentation for 75-source catalog and mirror management
connectors.md: categorized index of all 75 sources across 14 categories with descriptions, auth requirements, priorities, regions, and status. FSTEC BDU, NKCKI, and Kaspersky ICS promoted from beta to stable. architecture.md: updated source families (75 sources, 14 categories), added mirror domain management API (12 endpoints) to REST APIs section. mirrors.md: added MirrorExportScheduler docs, multi-value filter support (sourceCategory/sourceTag shorthands), mirror config UI sections (wizard, dashboard, catalog integration). docker.md: added section 7 with mirror env var reference (11 vars), domain config via env vars, filter shorthand documentation. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
master
@@ -124,13 +124,40 @@ Responses are serialized via `VexCanonicalJsonSerializer` ensuring stable orderi
|
||||
|
||||
## 4) Interaction with export pipeline
|
||||
|
||||
Mirror endpoints consume manifests produced by the export engine (`MongoVexExportStore`). They do **not** trigger new exports. Operators must configure connectors/exporters to keep targeted exports fresh (see `EXCITITOR-EXPORT-01-005/006/007`).
|
||||
Mirror endpoints consume manifests produced by the export engine (`MongoVexExportStore`). They can also be generated on demand via the mirror domain management API (`POST /api/v1/mirror/domains/{domainId}/generate`).
|
||||
|
||||
### 4.1 MirrorExportScheduler (background bundle refresh)
|
||||
|
||||
`MirrorExportScheduler` is a `BackgroundService` that periodically checks configured mirror domains for stale export bundles and triggers regeneration when source data has been updated since the last bundle generation.
|
||||
|
||||
**Behavior:**
|
||||
|
||||
1. On startup, loads all configured mirror domains from DB and config.
|
||||
2. Every N minutes (configurable via `RefreshIntervalMinutes`, default 60), iterates all domains.
|
||||
3. For each domain, checks whether any source in the export filters has been updated since the last bundle generation by comparing connector state timestamps.
|
||||
4. If stale, triggers bundle regeneration via `IMirrorBundleRegenerator`.
|
||||
5. Logs bundle generation metrics (duration, advisory count, stale/regenerated counts).
|
||||
6. Exposes per-domain staleness via `GetDomainStatuses()` and the `/api/v1/mirror/domains/{domainId}/status` endpoint.
|
||||
|
||||
**Configuration:**
|
||||
|
||||
| Setting | Default | Description |
|
||||
| --- | --- | --- |
|
||||
| `AutoRefreshEnabled` | `true` | Set to `false` to disable the scheduler entirely. Required for air-gap deployments where bundles are imported, not generated locally. |
|
||||
| `RefreshIntervalMinutes` | `60` | How often the scheduler checks for stale exports. Minimum 1 minute. |
|
||||
| `Enabled` | `true` | Global mirror distribution enable flag. When `false`, the scheduler skips refresh cycles. |
|
||||
|
||||
**Staleness detection** uses `IVexConnectorStateRepository` to find the latest source update timestamp across all connectors. If any source was updated after the last bundle generation timestamp, the export is considered stale.
|
||||
|
||||
### 4.2 Export workflow
|
||||
|
||||
Recommended workflow:
|
||||
|
||||
1. Define export plans at the export layer (JSON/OpenVEX/CSAF).
|
||||
2. Configure mirror domains mapping to those plans.
|
||||
3. Downstream mirror automation:
|
||||
1. **Define mirror domains** via the UI wizard or API (`POST /api/v1/mirror/domains`).
|
||||
2. **Configure exports** with filter shorthands (see 4.3 below).
|
||||
3. **Automatic refresh**: `MirrorExportScheduler` regenerates stale bundles periodically.
|
||||
4. **On-demand generation**: `POST /api/v1/mirror/domains/{domainId}/generate` for immediate bundle creation.
|
||||
5. **Downstream mirror automation**:
|
||||
* `GET /domains/{id}/index`
|
||||
* Compare `exportId` / `consensusRevision`
|
||||
* `GET /download` when new
|
||||
@@ -138,16 +165,96 @@ Recommended workflow:
|
||||
|
||||
When the export engine runs, it materializes the following artefacts under `outputRoot/<directoryName>`:
|
||||
|
||||
- `index.json` – canonical index listing each configured domain, manifest/bundle descriptors (with SHA-256 digests), and available export keys.
|
||||
- `<domain>/manifest.json` – per-domain summary with export metadata (query signature, consensus/score digests, source providers) and a descriptor pointing at the bundle.
|
||||
- `<domain>/bundle.json` – canonical payload containing serialized consensus, score envelopes, and normalized VEX claims for the matching export definitions.
|
||||
- `<domain>/bundle.json.jws` – optional detached JWS when signing is enabled.
|
||||
- `index.json` -- canonical index listing each configured domain, manifest/bundle descriptors (with SHA-256 digests), and available export keys.
|
||||
- `<domain>/manifest.json` -- per-domain summary with export metadata (query signature, consensus/score digests, source providers) and a descriptor pointing at the bundle.
|
||||
- `<domain>/bundle.json` -- canonical payload containing serialized consensus, score envelopes, and normalized VEX claims for the matching export definitions.
|
||||
- `<domain>/bundle.json.jws` -- optional detached JWS when signing is enabled.
|
||||
|
||||
Downstream automation reads `manifest.json`/`bundle.json` directly, while `/excititor/mirror` endpoints stream the same artefacts through authenticated HTTP.
|
||||
|
||||
### 4.3 Multi-value filter support and category shorthands
|
||||
|
||||
Export filters in `MirrorExportOptions.Filters` support three expansion modes via the `ResolveFilters()` method:
|
||||
|
||||
**sourceCategory shorthand** -- resolves a category name (or comma-separated list) to all source IDs in that category:
|
||||
|
||||
```json
|
||||
{
|
||||
"Key": "linux-distros",
|
||||
"Format": "openvex",
|
||||
"Filters": { "sourceCategory": "Distribution" }
|
||||
}
|
||||
```
|
||||
|
||||
This resolves to all 10 distribution sources (Debian, Ubuntu, Alpine, SUSE, RHEL, CentOS, Fedora, Arch, Gentoo, Astra Linux). Multiple categories are comma-separated: `"Exploit,Container,Ics,PackageManager"`.
|
||||
|
||||
**sourceTag shorthand** -- resolves a tag (or comma-separated list) to all sources carrying that tag:
|
||||
|
||||
```json
|
||||
{
|
||||
"Key": "linux-all",
|
||||
"Format": "json",
|
||||
"Filters": { "sourceTag": "linux" }
|
||||
}
|
||||
```
|
||||
|
||||
**Comma-separated sourceVendor** -- OR semantics across multiple vendors:
|
||||
|
||||
```json
|
||||
{
|
||||
"Key": "custom-selection",
|
||||
"Format": "json",
|
||||
"Filters": { "sourceVendor": "debian,ubuntu,alpine" }
|
||||
}
|
||||
```
|
||||
|
||||
All resolved values are sorted alphabetically for deterministic query signatures. Existing single-value filters remain backward-compatible.
|
||||
|
||||
**Supported categories** (14 total, matching the `SourceCategory` enum):
|
||||
|
||||
`Primary`, `Vendor`, `Distribution`, `Ecosystem`, `Cert`, `Csaf`, `Threat`, `Exploit`, `Container`, `Hardware`, `Ics`, `PackageManager`, `Mirror`, `Other`.
|
||||
|
||||
---
|
||||
|
||||
## 5) Operational guidance
|
||||
## 5) Mirror configuration UI
|
||||
|
||||
The mirror configuration UI provides three Angular components for managing mirror domains from the StellaOps Console, accessible under the Advisory & VEX Sources integration hub.
|
||||
|
||||
### 5.1 Mirror Domain Builder wizard
|
||||
|
||||
Route: `advisory-vex-sources/mirror/new`
|
||||
|
||||
A 3-step wizard for creating mirror domains from the source catalog:
|
||||
|
||||
**Step 1 -- Select Sources**: Displays the full source catalog grouped by the 14 categories. Operators can select sources individually or by category (checking a category header selects all sources in that category). Shorthand buttons provide quick selections: "All Primary", "All Distributions", "All Ecosystem", "All CERTs", "Everything". A live summary panel shows selected source count by category.
|
||||
|
||||
**Step 2 -- Configure Domain**: Auto-generates a domain ID and display name from the selection. Operators configure the export format (JSON, JSONL, OpenVEX, CSAF, CycloneDX), rate limits (index requests/hour, download requests/hour), authentication requirement, signing options, and optionally create multiple exports per domain.
|
||||
|
||||
**Step 3 -- Review & Create**: Summary card showing domain name, source count across categories, export format, and rate limits. Displays the resolved filter JSON that will be stored. "Create Domain" calls `POST /api/v1/mirror/domains`. An optional "Generate Now" checkbox triggers immediate bundle generation after creation.
|
||||
|
||||
### 5.2 Mirror Dashboard
|
||||
|
||||
Route: `advisory-vex-sources/mirror`
|
||||
|
||||
Displays all configured mirror domains as status cards:
|
||||
|
||||
* **Top bar**: Mirror mode indicator (Direct/Mirror/Hybrid), "Create Domain" button, global mirror health summary.
|
||||
* **Domain cards**: Each card shows domain name, export count and format, source count with category pills, last generated timestamp, bundle size, and staleness indicator (fresh/stale/never generated). Actions: Regenerate, Edit, Delete, View Endpoints.
|
||||
* **Consumer config panel**: For Hybrid or Mirror mode, shows the consumer mirror URL, connection status, and last sync time.
|
||||
* **Empty state**: If no domains are configured, displays a "Create your first mirror domain" call-to-action linking to the wizard.
|
||||
|
||||
### 5.3 Catalog mirror integration
|
||||
|
||||
The source catalog header (`advisory-source-catalog.component.ts`) includes mirror context:
|
||||
|
||||
* Mirror mode badge (Direct/Mirror/Hybrid) in the catalog header.
|
||||
* "Configure Mirror" link navigating to the mirror dashboard.
|
||||
* "Create Mirror Domain" button navigating to the wizard with pre-selected sources based on currently enabled sources.
|
||||
* Mirror domain count and total bundle advisory count in the stats bar (when domains exist).
|
||||
|
||||
---
|
||||
|
||||
## 6) Operational guidance
|
||||
|
||||
* Track quota utilisation via HTTP 429 metrics (configure structured logging or OTEL counters when rate limiting triggers).
|
||||
* Mirror domains can be deployed per tenant (e.g., `tenant-a`, `tenant-b`) with different auth requirements.
|
||||
@@ -156,7 +263,7 @@ Downstream automation reads `manifest.json`/`bundle.json` directly, while `/exci
|
||||
|
||||
---
|
||||
|
||||
## 6) Future alignment
|
||||
## 7) Future alignment
|
||||
|
||||
* Replace manual export definitions with generated mirror bundle manifests once `EXCITITOR-EXPORT-01-007` ships.
|
||||
* Extend `/index` payload with quiet-provenance when `EXCITITOR-EXPORT-01-006` adds that metadata.
|
||||
@@ -164,7 +271,7 @@ Downstream automation reads `manifest.json`/`bundle.json` directly, while `/exci
|
||||
|
||||
---
|
||||
|
||||
## 7) Runbook & observability checklist (Sprint 22 demo refresh · 2025-11-07)
|
||||
## 8) Runbook & observability checklist (Sprint 22 demo refresh · 2025-11-07)
|
||||
|
||||
### Daily / on-call checks
|
||||
1. **Index freshness** – watch `excitor_mirror_export_latency_seconds` (p95 < 180) grouped by `domainId`. If latency grows past 10 minutes, verify the export worker queue (`stellaops-export-worker` logs) and ensure PostgreSQL `vex.exports` has entries newer than `now()-10m`.
|
||||
|
||||
Reference in New Issue
Block a user