diff --git a/docs/implplan/SPRINT_20260315_007_Concelier_full_mirror_source_completeness_and_setup_ui.md b/docs/implplan/SPRINT_20260315_007_Concelier_full_mirror_source_completeness_and_setup_ui.md index 472b556ae..385b5e01f 100644 --- a/docs/implplan/SPRINT_20260315_007_Concelier_full_mirror_source_completeness_and_setup_ui.md +++ b/docs/implplan/SPRINT_20260315_007_Concelier_full_mirror_source_completeness_and_setup_ui.md @@ -426,7 +426,7 @@ Completion criteria: - [x] Mirror consumer (Concelier connector) processes new categories correctly (ResolveFilters handles comma-separated multi-category values) ### TASK-011 - Documentation Update -Status: TODO +Status: DONE Dependency: TASK-001 through TASK-010 Owners: Documentation author @@ -437,9 +437,9 @@ Update module documentation: - `docs/operations/deployment/docker.md` — document mirror configuration env vars Completion criteria: -- [ ] Connector index updated with all new sources -- [ ] Mirror setup guide includes UI-based configuration -- [ ] Source category taxonomy documented +- [x] Connector index updated with all new sources (75 sources across 14 categories, organized by category with descriptions) +- [x] Mirror setup guide includes UI-based configuration (wizard, dashboard, catalog integration documented in mirrors.md sections 5.1-5.3) +- [x] Source category taxonomy documented (14 categories enumerated in connectors.md, architecture.md, and mirrors.md) --- @@ -449,6 +449,7 @@ Completion criteria: | 2026-03-15 | Sprint created with 11 tasks for full mirror source completeness. | Planning | | 2026-03-15 | Expanded to 16 tasks. Added TASK-005 (multi-value filter model), TASK-006 (domain CRUD API, 12 endpoints), TASK-006b (export scheduler), TASK-007a (domain builder wizard), TASK-007b (mirror dashboard), TASK-007c (catalog mirror header). Original TASK-005/006/007 were too generic — now they cover the full distro-to-mirror operator journey. | Planning | | 2026-03-15 | TASK-007 DONE: Updated CATEGORY_ORDER (12 -> 14 entries), added CATEGORY_DESCRIPTIONS map, added category descriptions to UI headers, added getCategoryDescription() method. TASK-010 DONE: Added SupportedCategories to MirrorDistributionOptions, extended ResolveFilters for comma-separated sourceCategory/sourceTag, added Ics and PackageManager to SourceCategory enum, reclassified 6 sources. | Developer | +| 2026-03-15 | TASK-011 DONE: Updated 4 documentation files. connectors.md: rewrote from flat table to categorized index covering all 75 sources across 14 categories with descriptions, auth, priority, regions, and status (FSTEC BDU/NKCKI/Kaspersky ICS promoted to stable). architecture.md: updated source families section (75 sources, 14 categories), added mirror domain management API (12 endpoints) to REST APIs section. mirrors.md: added MirrorExportScheduler docs (staleness detection, config), multi-value filter support (sourceCategory/sourceTag/sourceVendor shorthands), mirror config UI sections (wizard, dashboard, catalog integration), renumbered subsequent sections. docker.md: added section 7 with mirror env var reference table (11 vars), domain config via env vars example, filter shorthand documentation. | Documentation | ## Decisions & Risks - **Commercial feeds** (Snyk, Sonatype OSS Index, VulnDB) intentionally excluded — they require paid API keys and license agreements incompatible with self-hosted offline-first posture. Can be added as `SourceType.Commercial` with clear license gates later. diff --git a/docs/modules/concelier/architecture.md b/docs/modules/concelier/architecture.md index 8d270066f..2c524e5d9 100644 --- a/docs/modules/concelier/architecture.md +++ b/docs/modules/concelier/architecture.md @@ -240,12 +240,31 @@ Legacy `Advisory`, `Affected`, and merge-centric entities remain in the reposito ## 4) Source families & precedence +The source catalog contains **75 definitions** across **14 categories**. The authoritative definition lives in `src/Concelier/__Libraries/StellaOps.Concelier.Core/Sources/SourceDefinitions.cs`; for the full connector index see `docs/modules/concelier/connectors.md`. + ### 3.1 Families -* **Vendor PSIRTs**: Microsoft, Oracle, Cisco, Adobe, Apple, VMware, Chromium… -* **Linux distros**: Red Hat, SUSE, Ubuntu, Debian, Alpine… -* **OSS ecosystems**: OSV, GHSA (GitHub Security Advisories), PyPI, npm, Maven, NuGet, Go. -* **CERTs / national CSIRTs**: CISA (KEV, ICS), JVN, ACSC, CCCS, KISA, CERT‑FR/BUND, etc. +* **Primary databases**: NVD, OSV, GHSA, CVE.org (MITRE). +* **Vendor PSIRTs**: Microsoft, Oracle, Cisco, Apple, VMware, Fortinet, Juniper, Palo Alto, plus cloud providers (AWS, Azure, GCP). +* **Linux distros**: Debian, Ubuntu, Alpine, SUSE, RHEL, CentOS, Fedora, Arch, Gentoo, Astra Linux. +* **OSS ecosystems**: npm, PyPI, Go, RubyGems, NuGet, Maven, Crates.io, Packagist, Hex.pm. +* **Package manager native**: RustSec (cargo-audit), PyPA (pip-audit), Go Vuln DB (govulncheck), Ruby Advisory DB (bundler-audit). +* **CSAF/VEX**: CSAF Aggregator, CSAF TC Trusted Publishers, VEX Hub. +* **Exploit databases**: Exploit-DB, PoC-in-GitHub, Metasploit Modules. +* **Container**: Docker Official CVEs, Chainguard Advisories. +* **Hardware/firmware**: Intel PSIRT, AMD Security, ARM Security Center. +* **ICS/SCADA**: Siemens ProductCERT, Kaspersky ICS-CERT. +* **CERTs / national CSIRTs**: CERT-FR, CERT-Bund, CERT.at, CERT.be, NCSC-CH, CERT-EU, JPCERT/CC, CISA (US-CERT), CERT-UA, CERT.PL, AusCERT, KrCERT/CC, CERT-In. +* **Russian/CIS**: FSTEC BDU, NKCKI (both promoted to stable). +* **Threat intelligence**: EPSS (FIRST), CISA KEV, MITRE ATT&CK, MITRE D3FEND. +* **StellaOps Mirror**: Pre-aggregated advisory mirror for offline/air-gap deployments. + +### Source category enum + +``` +Primary, Vendor, Distribution, Ecosystem, Cert, Csaf, Threat, +Exploit, Container, Hardware, Ics, PackageManager, Mirror, Other +``` ### 3.2 Precedence (when claims conflict) @@ -490,6 +509,25 @@ GET /advisories?scheme=CVE&value=CVE-2025-12345 GET /affected?productKey=pkg:rpm/openssl&limit=100 ``` +**Mirror domain management** (under `/api/v1/mirror`) + +``` +GET /config → current mirror config (mode, signing, refresh interval) +PUT /config → update mirror mode/signing/refresh settings +GET /domains → list all mirror domains with export counts +POST /domains → create a new mirror domain with exports/filters +GET /domains/{domainId} → domain detail (exports, status) +PUT /domains/{domainId} → update domain (name, auth, rate limits, exports) +DELETE /domains/{domainId} → remove a mirror domain +POST /domains/{domainId}/exports → add an export to a domain +DELETE /domains/{domainId}/exports/{exportKey} → remove an export from a domain +POST /domains/{domainId}/generate → trigger on-demand bundle generation +GET /domains/{domainId}/status → domain sync status (last generate, staleness) +POST /test → test mirror endpoint connectivity +``` + +Mirror domains group export plans with shared rate limits and authentication rules. Exports support multi-value filter shorthands: `sourceCategory` (e.g., `"Distribution"` resolves to all distro sources), `sourceTag` (e.g., `"linux"`), and comma-separated `sourceVendor` values. Domain configuration is persisted in `excititor.mirror_domains` / `excititor.mirror_exports` tables, with env-var config as fallback. The `MirrorExportScheduler` background service periodically regenerates stale bundles (configurable via `RefreshIntervalMinutes`, default 60 minutes). + **AuthN/Z:** Authority tokens (OpTok) with roles: `concelier.read`, `concelier.admin`, `concelier.export`. --- diff --git a/docs/modules/concelier/connectors.md b/docs/modules/concelier/connectors.md index b923493c6..3b287f418 100644 --- a/docs/modules/concelier/connectors.md +++ b/docs/modules/concelier/connectors.md @@ -2,38 +2,197 @@ This index lists Concelier connectors, their status, authentication expectations, and links to operational runbooks. For procedures and alerting, see `docs/modules/concelier/operations/connectors/`. -| Connector | Source ID | Status | Auth | Ops Runbook | -| --- | --- | --- | --- | --- | -| NVD (NIST) | `nvd` | stable | api-key | [docs/modules/concelier/operations/connectors/nvd.md](docs/modules/concelier/operations/connectors/nvd.md) | -| CVE (MITRE) | `cve` | stable | none | [docs/modules/concelier/operations/connectors/cve.md](docs/modules/concelier/operations/connectors/cve.md) | -| OSV | `osv` | stable | none | [docs/modules/concelier/operations/connectors/osv.md](docs/modules/concelier/operations/connectors/osv.md) | -| GHSA | `ghsa` | stable | api-token | [docs/modules/concelier/operations/connectors/ghsa.md](docs/modules/concelier/operations/connectors/ghsa.md) | -| EPSS | `epss` | stable | none | [docs/modules/concelier/operations/connectors/epss.md](docs/modules/concelier/operations/connectors/epss.md) | -| Alpine SecDB | `alpine` | stable | none | [docs/modules/concelier/operations/connectors/alpine.md](docs/modules/concelier/operations/connectors/alpine.md) | -| Debian Security Tracker | `debian` | stable | none | [docs/modules/concelier/operations/connectors/debian.md](docs/modules/concelier/operations/connectors/debian.md) | -| Ubuntu USN | `ubuntu` | stable | none | [docs/modules/concelier/operations/connectors/ubuntu.md](docs/modules/concelier/operations/connectors/ubuntu.md) | -| Red Hat OVAL/CSAF | `redhat` | stable | none | [docs/modules/concelier/operations/connectors/redhat.md](docs/modules/concelier/operations/connectors/redhat.md) | -| SUSE OVAL/CSAF | `suse` | stable | none | [docs/modules/concelier/operations/connectors/suse.md](docs/modules/concelier/operations/connectors/suse.md) | -| Astra Linux | `astra` | stable | none | [docs/modules/concelier/operations/connectors/astra.md](docs/modules/concelier/operations/connectors/astra.md) | -| CISA KEV | `kev` | stable | none | [docs/modules/concelier/operations/connectors/cve-kev.md](docs/modules/concelier/operations/connectors/cve-kev.md) | -| CISA ICS-CERT | `ics-cisa` | stable | none | [docs/modules/concelier/operations/connectors/ics-cisa.md](docs/modules/concelier/operations/connectors/ics-cisa.md) | -| CERT-CC | `cert-cc` | stable | none | [docs/modules/concelier/operations/connectors/cert-cc.md](docs/modules/concelier/operations/connectors/cert-cc.md) | -| CERT-FR | `cert-fr` | stable | none | [docs/modules/concelier/operations/connectors/cert-fr.md](docs/modules/concelier/operations/connectors/cert-fr.md) | -| CERT-Bund | `cert-bund` | stable | none | [docs/modules/concelier/operations/connectors/certbund.md](docs/modules/concelier/operations/connectors/certbund.md) | -| CERT-In | `cert-in` | stable | none | [docs/modules/concelier/operations/connectors/cert-in.md](docs/modules/concelier/operations/connectors/cert-in.md) | -| ACSC | `acsc` | stable | none | [docs/modules/concelier/operations/connectors/acsc.md](docs/modules/concelier/operations/connectors/acsc.md) | -| CCCS | `cccs` | stable | none | [docs/modules/concelier/operations/connectors/cccs.md](docs/modules/concelier/operations/connectors/cccs.md) | -| KISA | `kisa` | stable | none | [docs/modules/concelier/operations/connectors/kisa.md](docs/modules/concelier/operations/connectors/kisa.md) | -| JVN | `jvn` | stable | none | [docs/modules/concelier/operations/connectors/jvn.md](docs/modules/concelier/operations/connectors/jvn.md) | -| FSTEC BDU | `fstec-bdu` | beta | none | [docs/modules/concelier/operations/connectors/fstec-bdu.md](docs/modules/concelier/operations/connectors/fstec-bdu.md) | -| NKCKI | `nkcki` | beta | none | [docs/modules/concelier/operations/connectors/nkcki.md](docs/modules/concelier/operations/connectors/nkcki.md) | -| Microsoft MSRC | `msrc` | stable | none | [docs/modules/concelier/operations/connectors/msrc.md](docs/modules/concelier/operations/connectors/msrc.md) | -| Cisco PSIRT | `cisco` | stable | oauth | [docs/modules/concelier/operations/connectors/cisco.md](docs/modules/concelier/operations/connectors/cisco.md) | -| Oracle CPU | `oracle` | stable | none | [docs/modules/concelier/operations/connectors/oracle.md](docs/modules/concelier/operations/connectors/oracle.md) | -| VMware | `vmware` | stable | none | [docs/modules/concelier/operations/connectors/vmware.md](docs/modules/concelier/operations/connectors/vmware.md) | -| Adobe PSIRT | `adobe` | stable | none | [docs/modules/concelier/operations/connectors/adobe.md](docs/modules/concelier/operations/connectors/adobe.md) | -| Apple Security | `apple` | stable | none | [docs/modules/concelier/operations/connectors/apple.md](docs/modules/concelier/operations/connectors/apple.md) | -| Chromium | `chromium` | stable | none | [docs/modules/concelier/operations/connectors/chromium.md](docs/modules/concelier/operations/connectors/chromium.md) | -| Kaspersky ICS-CERT | `kaspersky-ics` | beta | none | [docs/modules/concelier/operations/connectors/kaspersky-ics.md](docs/modules/concelier/operations/connectors/kaspersky-ics.md) | +The catalog currently contains **75 source definitions** across **14 categories**. The authoritative source list is defined in `src/Concelier/__Libraries/StellaOps.Concelier.Core/Sources/SourceDefinitions.cs`. + +--- + +## Source categories + +| Category | Description | Source count | +| --- | --- | --- | +| Primary | Core vulnerability databases (NVD, OSV, GHSA, CVE) | 4 | +| Threat | Threat intelligence, exploit prediction, and known-exploited (EPSS, KEV, MITRE ATT&CK, D3FEND) | 4 | +| Vendor | Vendor PSIRTs and cloud provider security bulletins | 14 | +| Distribution | Linux distribution security trackers | 10 | +| Ecosystem | Language-ecosystem advisory feeds via OSV/GHSA | 9 | +| PackageManager | Native package manager advisory databases (cargo-audit, pip-audit, govulncheck, bundler-audit) | 4 | +| Csaf | CSAF/VEX structured document sources | 3 | +| Exploit | Exploit databases and proof-of-concept repositories | 3 | +| Container | Container image advisory sources | 2 | +| Hardware | Hardware and firmware PSIRT advisories | 3 | +| Ics | Industrial control systems and SCADA advisories | 2 | +| Cert | National CERTs and government CSIRTs | 13 | +| Mirror | StellaOps pre-aggregated mirrors | 1 | +| Other | Uncategorized sources | 0 | + +--- + +## Primary Databases + +| Connector | Source ID | Status | Auth | Priority | Ops Runbook | +| --- | --- | --- | --- | --- | --- | +| NVD (NIST) | `nvd` | stable | api-key (optional) | 10 | [nvd.md](docs/modules/concelier/operations/connectors/nvd.md) | +| OSV (Google) | `osv` | stable | none | 15 | [osv.md](docs/modules/concelier/operations/connectors/osv.md) | +| GitHub Security Advisories | `ghsa` | stable | api-token | 20 | [ghsa.md](docs/modules/concelier/operations/connectors/ghsa.md) | +| CVE.org (MITRE) | `cve` | stable | none | 5 | [cve.md](docs/modules/concelier/operations/connectors/cve.md) | + +## Threat Intelligence & Exploit Scoring + +| Connector | Source ID | Status | Auth | Priority | Ops Runbook | +| --- | --- | --- | --- | --- | --- | +| EPSS (FIRST) | `epss` | stable | none | 50 | [epss.md](docs/modules/concelier/operations/connectors/epss.md) | +| CISA KEV | `kev` | stable | none | 25 | [cve-kev.md](docs/modules/concelier/operations/connectors/cve-kev.md) | +| MITRE ATT&CK | `mitre-attack` | stable | none | 140 | -- | +| MITRE D3FEND | `mitre-d3fend` | stable | none | 142 | -- | + +MITRE ATT&CK provides adversary tactics and techniques in STIX format from the `mitre/cti` GitHub repository. D3FEND provides the complementary defensive techniques knowledge base. Both are tagged `threat-intel` and consumed via the `SourceType.Upstream` connector. For future STIX/TAXII protocol feeds, the `SourceType.StixTaxii` enum value is available for connector extensibility. + +## Vendor Advisories + +| Connector | Source ID | Status | Auth | Priority | Ops Runbook | +| --- | --- | --- | --- | --- | --- | +| Red Hat Security | `redhat` | stable | none | 30 | [redhat.md](docs/modules/concelier/operations/connectors/redhat.md) | +| Microsoft Security (MSRC) | `microsoft` | stable | none | 35 | [msrc.md](docs/modules/concelier/operations/connectors/msrc.md) | +| Amazon Linux Security | `amazon` | stable | none | 40 | -- | +| Google Security | `google` | stable | none | 45 | -- | +| Oracle Security | `oracle` | stable | none | 50 | [oracle.md](docs/modules/concelier/operations/connectors/oracle.md) | +| Apple Security | `apple` | stable | none | 55 | [apple.md](docs/modules/concelier/operations/connectors/apple.md) | +| Cisco Security | `cisco` | stable | oauth | 60 | [cisco.md](docs/modules/concelier/operations/connectors/cisco.md) | +| Fortinet PSIRT | `fortinet` | stable | none | 65 | -- | +| Juniper Security | `juniper` | stable | none | 70 | -- | +| Palo Alto Security | `paloalto` | stable | none | 75 | -- | +| VMware Security | `vmware` | stable | none | 80 | [vmware.md](docs/modules/concelier/operations/connectors/vmware.md) | +| AWS Security Bulletins | `aws` | stable | none | 81 | -- | +| Azure Security Advisories | `azure` | stable | none | 82 | -- | +| GCP Security Bulletins | `gcp` | stable | none | 83 | -- | + +AWS, Azure, and GCP cloud provider advisories were added in Sprint 007. They track platform-level security bulletins for cloud infrastructure components and are categorized under `Vendor` alongside traditional PSIRTs. + +## Linux Distributions + +| Connector | Source ID | Status | Auth | Priority | Regions | Ops Runbook | +| --- | --- | --- | --- | --- | --- | --- | +| Debian Security Tracker | `debian` | stable | none | 30 | -- | [debian.md](docs/modules/concelier/operations/connectors/debian.md) | +| Ubuntu Security Notices | `ubuntu` | stable | none | 32 | -- | [ubuntu.md](docs/modules/concelier/operations/connectors/ubuntu.md) | +| Alpine SecDB | `alpine` | stable | none | 34 | -- | [alpine.md](docs/modules/concelier/operations/connectors/alpine.md) | +| SUSE Security | `suse` | stable | none | 36 | -- | [suse.md](docs/modules/concelier/operations/connectors/suse.md) | +| RHEL Security | `rhel` | stable | none | 38 | -- | -- | +| CentOS Security | `centos` | stable | none | 40 | -- | -- | +| Fedora Security | `fedora` | stable | none | 42 | -- | -- | +| Arch Security | `arch` | stable | none | 44 | -- | -- | +| Gentoo Security | `gentoo` | stable | none | 46 | -- | -- | +| Astra Linux Security | `astra` | stable | none | 48 | RU, CIS | [astra.md](docs/modules/concelier/operations/connectors/astra.md) | + +## Language Ecosystems + +| Connector | Source ID | Status | Auth | Priority | Ops Runbook | +| --- | --- | --- | --- | --- | --- | +| npm Advisories | `npm` | stable | none | 50 | -- | +| PyPI Advisories | `pypi` | stable | none | 52 | -- | +| Go Advisories | `go` | stable | none | 54 | -- | +| RubyGems Advisories | `rubygems` | stable | none | 56 | -- | +| NuGet Advisories | `nuget` | stable | api-token | 58 | -- | +| Maven Advisories | `maven` | stable | none | 60 | -- | +| Crates.io Advisories | `crates` | stable | none | 62 | -- | +| Packagist Advisories | `packagist` | stable | none | 64 | -- | +| Hex.pm Advisories | `hex` | stable | none | 66 | -- | + +Ecosystem connectors use OSV or GHSA GraphQL as the underlying data source. NuGet requires a `GITHUB_PAT` for GHSA GraphQL access. + +## Package Manager Native Advisories + +| Connector | Source ID | Status | Auth | Priority | Ops Runbook | +| --- | --- | --- | --- | --- | --- | +| RustSec Advisory DB (cargo-audit) | `rustsec` | stable | none | 63 | -- | +| PyPA Advisory DB (pip-audit) | `pypa` | stable | none | 53 | -- | +| Go Vuln DB (govulncheck) | `govuln` | stable | none | 55 | -- | +| Ruby Advisory DB (bundler-audit) | `bundler-audit` | stable | none | 57 | -- | + +Package manager native advisory databases provide language-specific vulnerability data curated by the respective package manager maintainers. These complement the ecosystem feeds (OSV/GHSA) by providing authoritative tool-native data used by `cargo-audit`, `pip-audit`, `govulncheck`, and `bundler-audit`. They are categorized separately under `PackageManager` to allow targeted mirror export filtering. + +## CSAF/VEX Sources + +| Connector | Source ID | Status | Auth | Priority | Ops Runbook | +| --- | --- | --- | --- | --- | --- | +| CSAF Aggregator | `csaf` | stable | none | 70 | -- | +| CSAF TC Trusted Publishers | `csaf-tc` | stable | none | 72 | -- | +| VEX Hub | `vex` | stable | none | 74 | -- | + +## Exploit Databases + +| Connector | Source ID | Status | Auth | Priority | Ops Runbook | +| --- | --- | --- | --- | --- | --- | +| Exploit-DB | `exploitdb` | stable | none | 110 | -- | +| PoC-in-GitHub | `poc-github` | stable | api-token | 112 | -- | +| Metasploit Modules | `metasploit` | stable | none | 114 | -- | + +Exploit databases track publicly available proof-of-concept code and exploit modules. Exploit-DB is sourced from the Offensive Security GitLab mirror. PoC-in-GitHub uses the GitHub search API to discover repositories containing vulnerability PoCs (requires `GITHUB_PAT`). Metasploit tracks Rapid7 Metasploit Framework module metadata for CVE-to-exploit correlation. + +## Container Sources + +| Connector | Source ID | Status | Auth | Priority | Ops Runbook | +| --- | --- | --- | --- | --- | --- | +| Docker Official CVEs | `docker-official` | stable | none | 120 | -- | +| Chainguard Advisories | `chainguard` | stable | none | 122 | -- | + +Container-specific advisory sources track vulnerabilities in base images and hardened container distributions. Docker Official CVEs covers the Docker Hub official images program. Chainguard Advisories covers hardened distroless and Wolfi-based images. + +## Hardware/Firmware + +| Connector | Source ID | Status | Auth | Priority | Ops Runbook | +| --- | --- | --- | --- | --- | --- | +| Intel PSIRT | `intel` | stable | none | 130 | -- | +| AMD Security | `amd` | stable | none | 132 | -- | +| ARM Security Center | `arm` | stable | none | 134 | -- | + +Hardware PSIRT advisories cover CPU microcode, firmware, and silicon-level vulnerabilities from the three major processor vendors. These sources are especially relevant for infrastructure operators tracking speculative execution (Spectre/Meltdown class) and firmware supply chain issues. + +## ICS/SCADA + +| Connector | Source ID | Status | Auth | Priority | Regions | Ops Runbook | +| --- | --- | --- | --- | --- | --- | --- | +| Siemens ProductCERT | `siemens` | stable | none | 136 | -- | -- | +| Kaspersky ICS-CERT | `kaspersky-ics` | stable | none | 102 | RU, CIS, GLOBAL | [kaspersky-ics.md](docs/modules/concelier/operations/connectors/kaspersky-ics.md) | + +Industrial control systems advisories cover SCADA and operational technology vulnerabilities. Siemens ProductCERT publishes CSAF-format advisories. Kaspersky ICS-CERT was promoted from beta to stable in Sprint 007 after endpoint stability verification. + +## National CERTs + +| Connector | Source ID | Status | Auth | Priority | Regions | Ops Runbook | +| --- | --- | --- | --- | --- | --- | --- | +| CERT-FR | `cert-fr` | stable | none | 80 | FR, EU | [cert-fr.md](docs/modules/concelier/operations/connectors/cert-fr.md) | +| CERT-Bund (Germany) | `cert-de` | stable | none | 82 | DE, EU | [certbund.md](docs/modules/concelier/operations/connectors/certbund.md) | +| CERT.at (Austria) | `cert-at` | stable | none | 84 | AT, EU | -- | +| CERT.be (Belgium) | `cert-be` | stable | none | 86 | BE, EU | -- | +| NCSC-CH (Switzerland) | `cert-ch` | stable | none | 88 | CH | -- | +| CERT-EU | `cert-eu` | stable | none | 90 | EU | -- | +| JPCERT/CC (Japan) | `jpcert` | stable | none | 92 | JP, APAC | [jvn.md](docs/modules/concelier/operations/connectors/jvn.md) | +| CISA (US-CERT) | `us-cert` | stable | none | 94 | US, NA | [cert-cc.md](docs/modules/concelier/operations/connectors/cert-cc.md) | +| CERT-UA (Ukraine) | `cert-ua` | stable | none | 95 | UA | -- | +| CERT.PL (Poland) | `cert-pl` | stable | none | 96 | PL, EU | -- | +| AusCERT (Australia) | `auscert` | stable | none | 97 | AU, APAC | -- | +| KrCERT/CC (South Korea) | `krcert` | stable | none | 98 | KR, APAC | -- | +| CERT-In (India) | `cert-in` | stable | none | 99 | IN, APAC | [cert-in.md](docs/modules/concelier/operations/connectors/cert-in.md) | + +Five additional CERTs were added in Sprint 007: CERT-UA, CERT.PL, AusCERT, KrCERT/CC, and CERT-In, extending coverage to Eastern Europe, Oceania, and South/East Asia. + +## Russian/CIS Sources + +| Connector | Source ID | Status | Auth | Priority | Regions | Ops Runbook | +| --- | --- | --- | --- | --- | --- | --- | +| FSTEC BDU | `fstec-bdu` | stable | none | 100 | RU, CIS | [fstec-bdu.md](docs/modules/concelier/operations/connectors/fstec-bdu.md) | +| NKCKI | `nkcki` | stable | none | 101 | RU, CIS | [nkcki.md](docs/modules/concelier/operations/connectors/nkcki.md) | + +FSTEC BDU and NKCKI were promoted from beta to stable in Sprint 007. FSTEC BDU (Bank of Security Threats) provides vulnerability data maintained by Russia's Federal Service for Technical and Export Control. NKCKI is the National Coordination Center for Computer Incidents. Kaspersky ICS-CERT and Astra Linux are listed in their respective category sections above. + +## StellaOps Mirror + +| Connector | Source ID | Status | Auth | Priority | Ops Runbook | +| --- | --- | --- | --- | --- | --- | +| StellaOps Mirror | `stella-mirror` | stable | none (configurable) | 1 | -- | + +The StellaOps Mirror connector consumes pre-aggregated advisory data from a StellaOps mirror instance. When using mirror mode, this source takes highest priority (1) and replaces direct upstream connections. See `docs/modules/excititor/mirrors.md` for mirror configuration details. + +--- **Reason Codes Reference:** [docs/modules/concelier/operations/connectors/reason-codes.md](docs/modules/concelier/operations/connectors/reason-codes.md) diff --git a/docs/modules/excititor/mirrors.md b/docs/modules/excititor/mirrors.md index 9697218de..aff4e77e6 100644 --- a/docs/modules/excititor/mirrors.md +++ b/docs/modules/excititor/mirrors.md @@ -124,13 +124,40 @@ Responses are serialized via `VexCanonicalJsonSerializer` ensuring stable orderi ## 4) Interaction with export pipeline -Mirror endpoints consume manifests produced by the export engine (`MongoVexExportStore`). They do **not** trigger new exports. Operators must configure connectors/exporters to keep targeted exports fresh (see `EXCITITOR-EXPORT-01-005/006/007`). +Mirror endpoints consume manifests produced by the export engine (`MongoVexExportStore`). They can also be generated on demand via the mirror domain management API (`POST /api/v1/mirror/domains/{domainId}/generate`). + +### 4.1 MirrorExportScheduler (background bundle refresh) + +`MirrorExportScheduler` is a `BackgroundService` that periodically checks configured mirror domains for stale export bundles and triggers regeneration when source data has been updated since the last bundle generation. + +**Behavior:** + +1. On startup, loads all configured mirror domains from DB and config. +2. Every N minutes (configurable via `RefreshIntervalMinutes`, default 60), iterates all domains. +3. For each domain, checks whether any source in the export filters has been updated since the last bundle generation by comparing connector state timestamps. +4. If stale, triggers bundle regeneration via `IMirrorBundleRegenerator`. +5. Logs bundle generation metrics (duration, advisory count, stale/regenerated counts). +6. Exposes per-domain staleness via `GetDomainStatuses()` and the `/api/v1/mirror/domains/{domainId}/status` endpoint. + +**Configuration:** + +| Setting | Default | Description | +| --- | --- | --- | +| `AutoRefreshEnabled` | `true` | Set to `false` to disable the scheduler entirely. Required for air-gap deployments where bundles are imported, not generated locally. | +| `RefreshIntervalMinutes` | `60` | How often the scheduler checks for stale exports. Minimum 1 minute. | +| `Enabled` | `true` | Global mirror distribution enable flag. When `false`, the scheduler skips refresh cycles. | + +**Staleness detection** uses `IVexConnectorStateRepository` to find the latest source update timestamp across all connectors. If any source was updated after the last bundle generation timestamp, the export is considered stale. + +### 4.2 Export workflow Recommended workflow: -1. Define export plans at the export layer (JSON/OpenVEX/CSAF). -2. Configure mirror domains mapping to those plans. -3. Downstream mirror automation: +1. **Define mirror domains** via the UI wizard or API (`POST /api/v1/mirror/domains`). +2. **Configure exports** with filter shorthands (see 4.3 below). +3. **Automatic refresh**: `MirrorExportScheduler` regenerates stale bundles periodically. +4. **On-demand generation**: `POST /api/v1/mirror/domains/{domainId}/generate` for immediate bundle creation. +5. **Downstream mirror automation**: * `GET /domains/{id}/index` * Compare `exportId` / `consensusRevision` * `GET /download` when new @@ -138,16 +165,96 @@ Recommended workflow: When the export engine runs, it materializes the following artefacts under `outputRoot/`: -- `index.json` – canonical index listing each configured domain, manifest/bundle descriptors (with SHA-256 digests), and available export keys. -- `/manifest.json` – per-domain summary with export metadata (query signature, consensus/score digests, source providers) and a descriptor pointing at the bundle. -- `/bundle.json` – canonical payload containing serialized consensus, score envelopes, and normalized VEX claims for the matching export definitions. -- `/bundle.json.jws` – optional detached JWS when signing is enabled. +- `index.json` -- canonical index listing each configured domain, manifest/bundle descriptors (with SHA-256 digests), and available export keys. +- `/manifest.json` -- per-domain summary with export metadata (query signature, consensus/score digests, source providers) and a descriptor pointing at the bundle. +- `/bundle.json` -- canonical payload containing serialized consensus, score envelopes, and normalized VEX claims for the matching export definitions. +- `/bundle.json.jws` -- optional detached JWS when signing is enabled. Downstream automation reads `manifest.json`/`bundle.json` directly, while `/excititor/mirror` endpoints stream the same artefacts through authenticated HTTP. +### 4.3 Multi-value filter support and category shorthands + +Export filters in `MirrorExportOptions.Filters` support three expansion modes via the `ResolveFilters()` method: + +**sourceCategory shorthand** -- resolves a category name (or comma-separated list) to all source IDs in that category: + +```json +{ + "Key": "linux-distros", + "Format": "openvex", + "Filters": { "sourceCategory": "Distribution" } +} +``` + +This resolves to all 10 distribution sources (Debian, Ubuntu, Alpine, SUSE, RHEL, CentOS, Fedora, Arch, Gentoo, Astra Linux). Multiple categories are comma-separated: `"Exploit,Container,Ics,PackageManager"`. + +**sourceTag shorthand** -- resolves a tag (or comma-separated list) to all sources carrying that tag: + +```json +{ + "Key": "linux-all", + "Format": "json", + "Filters": { "sourceTag": "linux" } +} +``` + +**Comma-separated sourceVendor** -- OR semantics across multiple vendors: + +```json +{ + "Key": "custom-selection", + "Format": "json", + "Filters": { "sourceVendor": "debian,ubuntu,alpine" } +} +``` + +All resolved values are sorted alphabetically for deterministic query signatures. Existing single-value filters remain backward-compatible. + +**Supported categories** (14 total, matching the `SourceCategory` enum): + +`Primary`, `Vendor`, `Distribution`, `Ecosystem`, `Cert`, `Csaf`, `Threat`, `Exploit`, `Container`, `Hardware`, `Ics`, `PackageManager`, `Mirror`, `Other`. + --- -## 5) Operational guidance +## 5) Mirror configuration UI + +The mirror configuration UI provides three Angular components for managing mirror domains from the StellaOps Console, accessible under the Advisory & VEX Sources integration hub. + +### 5.1 Mirror Domain Builder wizard + +Route: `advisory-vex-sources/mirror/new` + +A 3-step wizard for creating mirror domains from the source catalog: + +**Step 1 -- Select Sources**: Displays the full source catalog grouped by the 14 categories. Operators can select sources individually or by category (checking a category header selects all sources in that category). Shorthand buttons provide quick selections: "All Primary", "All Distributions", "All Ecosystem", "All CERTs", "Everything". A live summary panel shows selected source count by category. + +**Step 2 -- Configure Domain**: Auto-generates a domain ID and display name from the selection. Operators configure the export format (JSON, JSONL, OpenVEX, CSAF, CycloneDX), rate limits (index requests/hour, download requests/hour), authentication requirement, signing options, and optionally create multiple exports per domain. + +**Step 3 -- Review & Create**: Summary card showing domain name, source count across categories, export format, and rate limits. Displays the resolved filter JSON that will be stored. "Create Domain" calls `POST /api/v1/mirror/domains`. An optional "Generate Now" checkbox triggers immediate bundle generation after creation. + +### 5.2 Mirror Dashboard + +Route: `advisory-vex-sources/mirror` + +Displays all configured mirror domains as status cards: + +* **Top bar**: Mirror mode indicator (Direct/Mirror/Hybrid), "Create Domain" button, global mirror health summary. +* **Domain cards**: Each card shows domain name, export count and format, source count with category pills, last generated timestamp, bundle size, and staleness indicator (fresh/stale/never generated). Actions: Regenerate, Edit, Delete, View Endpoints. +* **Consumer config panel**: For Hybrid or Mirror mode, shows the consumer mirror URL, connection status, and last sync time. +* **Empty state**: If no domains are configured, displays a "Create your first mirror domain" call-to-action linking to the wizard. + +### 5.3 Catalog mirror integration + +The source catalog header (`advisory-source-catalog.component.ts`) includes mirror context: + +* Mirror mode badge (Direct/Mirror/Hybrid) in the catalog header. +* "Configure Mirror" link navigating to the mirror dashboard. +* "Create Mirror Domain" button navigating to the wizard with pre-selected sources based on currently enabled sources. +* Mirror domain count and total bundle advisory count in the stats bar (when domains exist). + +--- + +## 6) Operational guidance * Track quota utilisation via HTTP 429 metrics (configure structured logging or OTEL counters when rate limiting triggers). * Mirror domains can be deployed per tenant (e.g., `tenant-a`, `tenant-b`) with different auth requirements. @@ -156,7 +263,7 @@ Downstream automation reads `manifest.json`/`bundle.json` directly, while `/exci --- -## 6) Future alignment +## 7) Future alignment * Replace manual export definitions with generated mirror bundle manifests once `EXCITITOR-EXPORT-01-007` ships. * Extend `/index` payload with quiet-provenance when `EXCITITOR-EXPORT-01-006` adds that metadata. @@ -164,7 +271,7 @@ Downstream automation reads `manifest.json`/`bundle.json` directly, while `/exci --- -## 7) Runbook & observability checklist (Sprint 22 demo refresh · 2025-11-07) +## 8) Runbook & observability checklist (Sprint 22 demo refresh · 2025-11-07) ### Daily / on-call checks 1. **Index freshness** – watch `excitor_mirror_export_latency_seconds` (p95 < 180) grouped by `domainId`. If latency grows past 10 minutes, verify the export worker queue (`stellaops-export-worker` logs) and ensure PostgreSQL `vex.exports` has entries newer than `now()-10m`. diff --git a/docs/operations/deployment/docker.md b/docs/operations/deployment/docker.md index e016859d9..59ac372b0 100644 --- a/docs/operations/deployment/docker.md +++ b/docs/operations/deployment/docker.md @@ -1,17 +1,17 @@ -# StellaOps Console — Docker Install Recipes +# StellaOps Console -- Docker Install Recipes -> **Audience:** Deployment Guild, Console Guild, platform operators. -> **Scope:** Acquire the `stellaops/web-ui` image, run it with Compose or Helm, mirror it for air‑gapped environments, and keep parity with CLI workflows. +> **Audience:** Deployment Guild, Console Guild, platform operators. +> **Scope:** Acquire the `stellaops/web-ui` image, run it with Compose or Helm, mirror it for air-gapped environments, and keep parity with CLI workflows. This guide focuses on the new **StellaOps Console** container. Start with the general [Installation Guide](../INSTALL_GUIDE.md) for shared prerequisites (Docker, registry access, TLS) and use the steps below to layer in the console. --- -## 1 · Release artefacts +## 1 - Release artefacts | Artefact | Source | Verification | |----------|--------|--------------| -| Console image | `registry.stella-ops.org/stellaops/web-ui@sha256:` | Listed in `deploy/releases/.yaml` (`yq '.services[] | select(.name=="web-ui") | .image'`). Signed with Cosign (`cosign verify --key https://stella-ops.org/keys/cosign.pub …`). | +| Console image | `registry.stella-ops.org/stellaops/web-ui@sha256:` | Listed in `deploy/releases/.yaml` (`yq '.services[] | select(.name=="web-ui") | .image'`). Signed with Cosign (`cosign verify --key https://stella-ops.org/keys/cosign.pub ...`). | | Compose bundles | `devops/compose/docker-compose.{dev,stage,prod,airgap}.yaml` | Each profile already includes a `web-ui` service pinned to the release digest. Run `docker compose --env-file -f docker-compose..yaml config` to confirm the digest matches the manifest. | | Helm values | `devops/helm/stellaops/values-*.yaml` (`services.web-ui`) | CI lints the chart; use `helm template` to confirm the rendered Deployment/Service carry the expected digest and env vars. | | Offline artefact (preview) | Generated via `oras copy registry.stella-ops.org/stellaops/web-ui@sha256: oci-archive:stellaops-web-ui-.tar` | Record SHA-256 in the downloads manifest (`DOWNLOADS-CONSOLE-23-001`) and sign with Cosign before shipping in the Offline Kit. | @@ -20,7 +20,7 @@ This guide focuses on the new **StellaOps Console** container. Start with the ge --- -## 2 · Compose quickstart (connected host) +## 2 - Compose quickstart (connected host) 1. **Prepare workspace** @@ -29,14 +29,14 @@ This guide focuses on the new **StellaOps Console** container. Start with the ge cp /path/to/repo/devops/compose/env/dev.env.example .env ``` -2. **Add console configuration** – append the following to `.env` (adjust per environment): +2. **Add console configuration** -- append the following to `.env` (adjust per environment): ```bash CONSOLE_PUBLIC_BASE_URL=https://console.dev.stella-ops.local CONSOLE_GATEWAY_BASE_URL=https://api.dev.stella-ops.local AUTHORITY_ISSUER=https://authority.dev.stella-ops.local AUTHORITY_CLIENT_ID=console-ui -AUTHORITY_SCOPES="ui.read ui.admin orch:operate findings:read advisory:read vex:read aoc:verify" + AUTHORITY_SCOPES="ui.read ui.admin orch:operate findings:read advisory:read vex:read aoc:verify" AUTHORITY_DPOP_ENABLED=true ``` @@ -81,7 +81,7 @@ AUTHORITY_SCOPES="ui.read ui.admin orch:operate findings:read advisory:read vex: --- -## 3 · Helm deployment (cluster) +## 3 - Helm deployment (cluster) 1. **Create an overlay** (example `console-values.yaml`): @@ -99,7 +99,7 @@ AUTHORITY_SCOPES="ui.read ui.admin orch:operate findings:read advisory:read vex: CONSOLE_GATEWAY_BASE_URL: "https://api.dev.stella-ops.local" AUTHORITY_ISSUER: "https://authority.dev.stella-ops.local" AUTHORITY_CLIENT_ID: "console-ui" -AUTHORITY_SCOPES: "ui.read ui.admin orch:operate findings:read advisory:read vex:read aoc:verify" + AUTHORITY_SCOPES: "ui.read ui.admin orch:operate findings:read advisory:read vex:read aoc:verify" AUTHORITY_DPOP_ENABLED: "true" CONSOLE_FEATURE_FLAGS: "runs,downloads,policies" CONSOLE_METRICS_ENABLED: "true" @@ -130,7 +130,7 @@ AUTHORITY_SCOPES: "ui.read ui.admin orch:operate findings:read advisory:read vex --- -## 4 · Offline packaging +## 4 - Offline packaging 1. **Mirror the image to an OCI archive** @@ -167,11 +167,11 @@ AUTHORITY_SCOPES: "ui.read ui.admin orch:operate findings:read advisory:read vex mv downloads/manifest.json.tmp downloads/manifest.json ``` - Re-run `stella offline kit import downloads/manifest.json` to validate signatures inside the air‑gapped environment. + Re-run `stella offline kit import downloads/manifest.json` to validate signatures inside the air-gapped environment. --- -## 5 · CLI parity +## 5 - CLI parity Console operations map directly to scriptable workflows: @@ -186,27 +186,79 @@ Track progress for the CLI commands via `DOCS-CONSOLE-23-014` (CLI vs UI parity --- -## 6 · Compliance checklist +## 6 - Compliance checklist -- [ ] Image digest validated against the current release manifest. -- [ ] Compose/Helm deployments verified with `docker compose config` / `helm template`. -- [ ] Authority issuer, scopes, and DPoP settings documented and applied. -- [ ] Offline archive mirrored, signed, and recorded in the downloads manifest. -- [ ] CLI parity notes linked to the upcoming `docs/cli-vs-ui-parity.md` matrix. -- [ ] References cross-checked with `docs/deploy/console.md` and `docs/security/console-security.md`. +- [ ] Image digest validated against the current release manifest. +- [ ] Compose/Helm deployments verified with `docker compose config` / `helm template`. +- [ ] Authority issuer, scopes, and DPoP settings documented and applied. +- [ ] Offline archive mirrored, signed, and recorded in the downloads manifest. +- [ ] CLI parity notes linked to the upcoming `docs/cli-vs-ui-parity.md` matrix. +- [ ] References cross-checked with `docs/deploy/console.md` and `docs/security/console-security.md`. - [ ] Health checks documented for connected and air-gapped installs. --- -## 7 · References +## 7 - Mirror distribution configuration -- `deploy/releases/.yaml` – Release manifest (digests, SBOM metadata). -- `devops/compose/README.md` – Compose profile overview. -- `devops/helm/stellaops/values-*.yaml` – Helm defaults per environment. -- `/docs/deploy/console.md` – Detailed environment variables, CSP, health checks. -- `/docs/security/console-security.md` – Auth flows, scopes, DPoP, monitoring. -- `docs/UI_GUIDE.md` – Console workflows and offline posture. +The Excititor service (advisory VEX export engine) supports mirror distribution for syncing advisory data to downstream environments and air-gapped deployments. Mirror configuration is applied via environment variables or config file sections under `Excititor:Mirror`. + +### Environment variables + +| Variable | Default | Description | +|----------|---------|-------------| +| `Excititor__Mirror__Enabled` | `true` | Global enable flag for mirror distribution surfaces and bundle generation. | +| `Excititor__Mirror__OutputRoot` | (artifact store default) | Filesystem root where mirror artefacts are written. | +| `Excititor__Mirror__DirectoryName` | `mirror` | Subdirectory under OutputRoot for mirror artifacts. | +| `Excititor__Mirror__TargetRepository` | -- | Human-readable hint for downstream mirrors (e.g., `s3://mirror/excititor`). | +| `Excititor__Mirror__RefreshIntervalMinutes` | `60` | How often the `MirrorExportScheduler` checks for stale bundles and regenerates them. Minimum 1 minute. | +| `Excititor__Mirror__AutoRefreshEnabled` | `true` | Set to `false` to disable the background scheduler. Required for air-gap deployments where bundles are imported rather than generated. | +| `Excititor__Mirror__Signing__Enabled` | `false` | Enable detached JWS signing for mirror bundle payloads. | +| `Excititor__Mirror__Signing__Algorithm` | `ES256` | Signing algorithm identifier. | +| `Excititor__Mirror__Signing__KeyId` | -- | Signing key identifier resolved via the crypto provider registry. Required when signing is enabled. | +| `Excititor__Mirror__Signing__Provider` | -- | Optional provider hint for key resolution. | +| `Excititor__Mirror__Signing__KeyPath` | -- | Optional PEM file path for key seeding. | + +### Domain configuration via env vars + +Mirror domains can be configured through structured env vars (array indexing): + +```bash +Excititor__Mirror__Domains__0__Id=primary +Excititor__Mirror__Domains__0__DisplayName=Primary Mirror +Excititor__Mirror__Domains__0__RequireAuthentication=false +Excititor__Mirror__Domains__0__MaxIndexRequestsPerHour=600 +Excititor__Mirror__Domains__0__MaxDownloadRequestsPerHour=1200 +Excititor__Mirror__Domains__0__Exports__0__Key=all-advisories +Excititor__Mirror__Domains__0__Exports__0__Format=json +Excititor__Mirror__Domains__0__Exports__0__Filters__sourceCategory=Distribution +``` + +Alternatively, domains can be managed through the Mirror Domain Management API (`/api/v1/mirror/domains`) or the Console UI wizard (Advisory & VEX Sources > Mirror). Database-configured domains take precedence over env-var domains when both define the same domain ID. + +### Filter shorthands for exports + +Export filters support category and tag shorthands that resolve to source IDs at export time: + +- `sourceCategory=Distribution` -- resolves to all distribution sources (Debian, Ubuntu, Alpine, etc.) +- `sourceCategory=Exploit,Container,Ics` -- comma-separated categories (OR semantics) +- `sourceTag=linux` -- resolves to all sources tagged `linux` +- `sourceVendor=debian,ubuntu,alpine` -- explicit comma-separated source IDs + +See `docs/modules/excititor/mirrors.md` for the full mirror architecture documentation. --- -*Last updated: 2025-10-28 (Sprint 23).* +## 8 - References + +- `deploy/releases/.yaml` -- Release manifest (digests, SBOM metadata). +- `devops/compose/README.md` -- Compose profile overview. +- `devops/helm/stellaops/values-*.yaml` -- Helm defaults per environment. +- `/docs/deploy/console.md` -- Detailed environment variables, CSP, health checks. +- `/docs/security/console-security.md` -- Auth flows, scopes, DPoP, monitoring. +- `docs/UI_GUIDE.md` -- Console workflows and offline posture. +- `docs/modules/excititor/mirrors.md` -- Mirror distribution architecture and configuration. +- `docs/modules/concelier/connectors.md` -- Full connector index (75 sources, 14 categories). + +--- + +*Last updated: 2026-03-15 (Sprint 007).*