prep the Feedser
This commit is contained in:
Vladimir Moushkov
609
src/Feedser/AGENTS.md
Normal file
609
src/Feedser/AGENTS.md
Normal file
@@ -0,0 +1,609 @@
|
||||
# AGENTS.md — Feedser (Stella Ops)
|
||||
|
||||
> YOU ARE: a senior C#/.NET 8 engineer‑agent tasked with building **Feedser**, a CLI that fetches, normalizes, reconciles, and packages *primary, non‑aggregated* vulnerability intelligence into a single **feed‑merge** database and exports a **Trivy‑compatible** DB (OCI artifact) for offline/self‑hosted scanning.
|
||||
> MODE: factual, deterministic, test‑first, with strict provenance and reproducibility.
|
||||
> BOUNDARIES: implement architecture & code in this repo only. No secret exfiltration. Default offline.
|
||||
|
||||
---
|
||||
|
||||
## 0) Stella Ops — condensed context
|
||||
|
||||
**Stella Ops** is a container & infra security platform built for sovereign/offline operation. Key storylines: **Δ‑SBOM warm path**, nightly re‑checks, policy‑as‑code, **signed artifacts**, optional **AI** remediation, and regional/air‑gapped operation.
|
||||
**Feedser** is foundational: it powers the scanner by producing a unified, deduped, explainable vulnerability database and a **self‑hosted Trivy DB**.
|
||||
|
||||
**Sibling components (stable contracts, no code here):**
|
||||
- **Scanner** (`stellaops.module.scanning`) — consumes Trivy‑compatible DB → findings + SBOM digests.
|
||||
- **Policy Engine**, **Signed Artifacts Service** (cosign), **AIRE** (AI suggestions), **SecretsScanner**, **MailDaemon**, **Offline Kit**, **RU/EEU adapters** (CryptoPro TLS, RU cert chains), **UI Shell**.
|
||||
|
||||
---
|
||||
|
||||
## 1) Problem statement
|
||||
|
||||
1) **Fetch** authoritative *primary* sources (global + regional + PSIRT + distro + CERTs + ICS).
|
||||
2) **Parse & Normalize** to a **UnifiedVuln** model.
|
||||
3) **Reconcile/Deduplicate** deterministically across sources with precedence rules.
|
||||
4) **Persist** into **feed‑merge DB** with both **bootstrap‑from‑scratch** and **incremental refresh**.
|
||||
5) **Package & Publish**:
|
||||
- **Trivy DB (v2) OCI artifact** for scanners (`--db-repository`),
|
||||
- optional **vuln‑list–shaped JSON** tree (to reuse `trivy-db` builder),
|
||||
- optional **signed offline bundle**.
|
||||
|
||||
Non‑goals v0: building a new scanner or a custom Java DB; we only ensure Scanner can target our self‑hosted DB.
|
||||
|
||||
---
|
||||
|
||||
## 2) High‑level architecture
|
||||
|
||||
```
|
||||
|
||||
[Connectors] ──► [Source DTO validation] ──► [Normalizer → UnifiedVuln]
|
||||
CVE/NVD, GHSA/OSV, JVN, CERT/CC, CISA KEV, KISA, CERT-In, ANSSI (CERT-FR),
|
||||
BSI (CERT-Bund WID), ACSC, CCCS, RU: BDU + NKCKI, Vendor PSIRTs (MSRC, Cisco,
|
||||
Oracle CPU, Adobe APSB, Apple, Chromium, VMware), Distros (Red Hat, Ubuntu,
|
||||
Debian, SUSE), ICS (CISA ICS, Kaspersky ICS-CERT)
|
||||
│
|
||||
▼
|
||||
[Merge/Reconcile Engine]
|
||||
(aliases, precedence, ranges, KEV flags, PSIRT flags)
|
||||
│
|
||||
▼
|
||||
[Feed‑Merge DB (SQLite→Postgres)]
|
||||
│
|
||||
┌──────────────────┴──────────────────┐
|
||||
▼ ▼
|
||||
[Export: vuln‑list JSON] [Packager: Trivy DB v2]
|
||||
│ │
|
||||
(CI) [ORAS push / offline tar]
|
||||
|
||||
```
|
||||
|
||||
**Principles**
|
||||
- Determinism (same inputs → same outputs, hashed) and provenance per field.
|
||||
- OVAL (vendor/distro) **overrides** generic ranges for OS packages.
|
||||
- Regional feeds **enrich** rather than blindly override unless they carry stronger package‑level truth.
|
||||
|
||||
---
|
||||
|
||||
## 3) Repository layout (create exactly)
|
||||
|
||||
```
|
||||
|
||||
src/Feedser/
|
||||
Feedser.Cli/ # .NET 8 console (System.CommandLine)
|
||||
Feedser.Core/ # domain model & orchestration
|
||||
Feedser.Storage/ # EF Core migrations (SQLite dev/CI; Postgres prod)
|
||||
Feedser.Connectors/
|
||||
Common/ # HTTP, pagination, ETag, backoff, schema validators
|
||||
Cve/ # CVE registry (id+refs)
|
||||
Nvd/ # NVD API v2 windows
|
||||
Ghsa/ # GHSA REST/GraphQL
|
||||
Osm.Osv/ # OSV API
|
||||
Jvn/ # MyJVN (JVNRSS/VULDEF)
|
||||
CertCc/ # CERT/CC Vulnerability Notes
|
||||
Kev/ # CISA Known Exploited
|
||||
Kr.Kisa/ # KISA/KrCERT advisories
|
||||
In.CertIn/ # CERT-In advisories
|
||||
Fr.CertFr/ # ANSSI CERT-FR avis/alertes
|
||||
De.CertBund/ # BSI CERT-Bund WID
|
||||
Au.Acsc/ # ACSC advisories
|
||||
Ca.Cccs/ # CCCS advisories
|
||||
Ru.Bdu/ # FSTEC BDU (HTML→schema; LLM fallback gated)
|
||||
Ru.Nkcki/ # NKCKI bulletins (HTML/PDF→text)
|
||||
Vndr.Msrc/ # MSRC CVRF
|
||||
Vndr.Cisco/ # Cisco PSIRT openVuln
|
||||
Vndr.Oracle/ # Oracle CPU/advisories
|
||||
Vndr.Adobe/ # Adobe APSB/APA
|
||||
Vndr.Apple/ # Apple HT201222 feed
|
||||
Vndr.Chromium/ # Chrome Releases security posts
|
||||
Vndr.Vmware/ # VMSA (Broadcom portal)
|
||||
Distro.RedHat/ # Red Hat Security Data API + OVAL
|
||||
Distro.Ubuntu/ # USN + Security API
|
||||
Distro.Debian/ # Debian Security Tracker JSON
|
||||
Distro.Suse/ # SUSE Update Advisories
|
||||
Ics.Cisa/ # CISA ICS advisories (ICSA-*)
|
||||
Ics.Kaspersky/ # Kaspersky ICS-CERT advisories
|
||||
Feedser.Merge/ # dedupe/aliases/precedence/version-ranges
|
||||
Feedser.Export.VulnList/ # optional vuln-list JSON renderer
|
||||
Feedser.Packagers.TrivyDb/ # db.tar.gz + metadata.json + ORAS push
|
||||
Feedser.Signing/ # cosign integration
|
||||
Feedser.Tests/
|
||||
etc/
|
||||
feedser.yaml # config template (extended, see §11)
|
||||
schemas/ # JSON Schema/XSD for inputs & internal payloads
|
||||
samples/ # golden fixtures per source
|
||||
|
||||
````
|
||||
|
||||
---
|
||||
|
||||
## 4) Unified data model (relational + evented)
|
||||
|
||||
**Storage default**: **SQLite** (dev/CI), **Postgres** (prod). EF Core migrations. Dapper for hot paths if needed.
|
||||
|
||||
**Tables (no change from v1 + PSIRT/CERT flags)**
|
||||
|
||||
- `source(id, name, type, base_url, auth_mode, notes)`
|
||||
- `watermark(source_id, cursor, updated_at)` ← **incremental windows per source**
|
||||
- `document(id, source_id, uri, fetched_at, content_sha256, content_type, status, raw_blob?, metadata_json)`
|
||||
- `advisory(id, advisory_key, title, summary, lang, published, modified,
|
||||
severity_cvss_v3?, severity_cvss_v4?, vendor_severity?,
|
||||
exploit_known bool)`
|
||||
- `alias(advisory_id, scheme, value)` — **schemes** include: CVE, GHSA, OSV, JVN, BDU, VU (CERT/CC), MSRC, CISCO‑SA, ORACLE‑CPU, APSB/APA, APPLE‑HT, CHROMIUM‑POST, VMSA, RHSA, USN, DSA, SUSE‑SU, ICSA, CWE, CPE, PURL, etc.
|
||||
- `affected(advisory_id, platform, name, version_range, cpe?, purl?, fixed_by?, introduced_version?)`
|
||||
- `reference(advisory_id, url, kind, source_tag)` — kind examples: advisory, patch, bulletin, kb, blog, vendor, exploit
|
||||
- `provenance(advisory_id, document_id, extraction, confidence, fields_mask)`
|
||||
- `kev_flag(advisory_id, kev_id, added_date, due_date?)`
|
||||
- `ru_flags(advisory_id, bdu_id?, nkcki_ids_json?, ru_severity?, notes?)`
|
||||
- `jp_flags(advisory_id, jvndb_id?, jvn_category?, vendor_status?)`
|
||||
- `psirt_flags(advisory_id, vendor, advisory_id_text, product_tags_json?)`
|
||||
- `merge_event(id, advisory_key, before_hash, after_hash, merged_at)`
|
||||
|
||||
**Indexes**: unique(advisory_key); index(scheme,value); index(platform,name); index(published); index(modified).
|
||||
|
||||
### 4.1) Alternate storage (MongoDB) — mapping (per your plan)
|
||||
If a **MongoDB** deployment is preferred, mirror the relational shape **as collections** with analogous names (`source`, `watermark`, `document`, `advisory`, `alias`, `affected`, `reference`, `provenance`, `kev_flag`, `ru_flags`, `jp_flags`, `psirt_flags`, `merge_event`).
|
||||
- Keep **advisory documents** flat and **embed** `aliases[]`, `affected[]`, `references[]` when practical; store **provenance** entries as embedded or sidecar collection depending on document growth.
|
||||
- Maintain **deterministic canonical JSON** for merges; hash stored in `merge_event`.
|
||||
- Incremental refreshes rely on the same **per‑source watermarks**.
|
||||
|
||||
---
|
||||
|
||||
## 5) Source connectors — contracts & incremental strategy
|
||||
|
||||
**Common interface**
|
||||
```csharp
|
||||
public interface IFeedConnector {
|
||||
string SourceName { get; }
|
||||
Task FetchAsync(FeedserContext db, CancellationToken ct); // populate document rows
|
||||
Task ParseAsync(FeedserContext db, CancellationToken ct); // document -> DTOs (validated)
|
||||
Task MapAsync(FeedserContext db, CancellationToken ct); // DTOs -> UnifiedVuln tables + provenance
|
||||
}
|
||||
````
|
||||
|
||||
### 5.1 Registries & cross‑ecosystem
|
||||
|
||||
* **CVE (cve.org)** — *identifier registry*. Fetch for alias cross‑checks; minimal fields only. Watermark by last seen ID/time.
|
||||
* **NVD API v2** — sliding **modified windows** (e.g., 6–12h) with backoff and pagination. Persist CVSS/CWE/CPE as aliases; capture change history if present. Watermark = last successful `modified_end`.
|
||||
* **GHSA** — **REST** “global security advisories” + **GraphQL** for richer fields; **note**: `cvss` → `cvss_severities` deprecation → map accordingly. Watermark by updated timestamp/ID cursor.
|
||||
* **OSV** — fetch per eco or time range; map PURL + SemVer ranges.
|
||||
|
||||
### 5.2 National CERTs (incremental via RSS/API/pages)
|
||||
|
||||
* **CERT/CC Vulnerability Notes** — scrape/archive pages (VU#), and/or GitHub data archive when suitable. Watermark by VU publish date/ID.
|
||||
* **JVN / MyJVN (Japan)** — **MyJVN API**: JVNRSS (overview) + VULDEF (detail). Watermark by `dateFirstPublished`/`dateLastUpdated`. Map **JVNDB** IDs, CVE aliases, vendor status.
|
||||
* **RU‑CERT** — advisory/news portal; treat as **enrichment references** (aliases+refs), not a primary package range source. Watermark by post date.
|
||||
* **KISA (KrCERT/KRCERT)** — advisories/notices portal. Watermark by advisory date/ID.
|
||||
* **CERT‑In (India)** — **CIAD** advisories via portal pages; Watermark by advisory code/date.
|
||||
* **ANSSI/CERT‑FR** — *avis/alertes* RSS and list pages; Watermark by advisory ID/date.
|
||||
* **BSI CERT‑Bund (WID)** — “Technische Sicherheitshinweise” pages/feeds; Watermark by bulletin ID/date.
|
||||
* **ACSC (Australia)** — alerts/advisories; Watermark by publish date/slug.
|
||||
* **CCCS (Canada)** — advisories page; Watermark by date/slug.
|
||||
|
||||
### 5.3 Russia‑specific
|
||||
|
||||
* **FSTEC BDU** — **hybrid**: primary **HTML parser** → validate against our **internal XML schema**; if validation fails → **LLM extraction fallback** (strictly gated; see §7). Also support **bulk DB ingests** if official XML/Excel exports are available in the environment. Watermark by BDU ID/date.
|
||||
* **NKCKI** — bulletins list (HTML/PDF). Extract structured fields via PDF→text pipeline + post‑validation. Watermark by bulletin ID/date.
|
||||
|
||||
### 5.4 Vendor PSIRTs (canonical)
|
||||
|
||||
* **MSRC** — **CVRF API** monthly and per‑advisory endpoints. Watermark by month + last modified. Alias: `MSRC:<YYYY-MMM>`; references to KBs/CVEs.
|
||||
* **Cisco PSIRT (openVuln API)** — REST; filter by last published/updated. Alias: `CISCO-SA:<slug>`; map fixed releases.
|
||||
* **Oracle CPU / Security Alerts** — quarterly schedule (3rd Tue of Jan/Apr/Jul/Oct). Scrape CPU pages and advisories. Alias: `CPU:<YYYY-QQ>`; link per‑product CVEs. Watermark by CPU cycle.
|
||||
* **Adobe APSB/APA** — advisory index pages + product feeds. Alias: `APSB-YYYY-XX`.
|
||||
* **Apple** — **HT201222/“About Apple security releases”** index page(s). Alias: `APPLE-HT:HT201222:<yyyy-mm-dd>` + per‑product pages.
|
||||
* **Google Chromium** — **Chrome Releases** blog “Stable Channel Update” posts with security fix lists. Alias: `CHROMIUM-POST:<date>`.
|
||||
* **VMware (VMSA)** — Broadcom support portal VMSA pages; parse ID + affected products + CVEs. Alias: `VMSA-YYYY-XXXX`.
|
||||
|
||||
### 5.5 Linux distributions
|
||||
|
||||
* **Red Hat Security Data API** (CSAF/OVAL/CVE); plus OVAL content. **Precedence** for OS packages. Watermark via API `last_modified`/etag. Alias: `RHSA-YYYY:NNNN`.
|
||||
* **Ubuntu USN** — USN list + **Security API**; Watermark by USN ID/date. Alias: `USN-####-#`.
|
||||
* **Debian Security Tracker** — JSON dataset for CVE↔package↔suite; Watermark by file etag/commit. Alias: `DSA-####-#` (when present).
|
||||
* **SUSE** — security/update advisories pages; Watermark by SUSE‑SU ID/date. Alias: `SUSE-SU-YYYY:NNNN`.
|
||||
|
||||
### 5.6 Specialized / ICS
|
||||
|
||||
* **CISA ICS advisories (ICSA)** — list feeds; Watermark by ICSA‑ID. Alias: `ICSA-YY-###-##`.
|
||||
* **Kaspersky ICS‑CERT** — advisories list; Watermark by advisory ID/date; treat as authoritative vendor ICS source for impacted OT products.
|
||||
|
||||
### 5.7 Exploitation & enrichment
|
||||
|
||||
* **CISA KEV** — JSON catalog; set exploitation flag (`exploit_known=true`), store `kev_id`, `added_date`, `due_date`.
|
||||
|
||||
---
|
||||
|
||||
## 6) Normalization details
|
||||
|
||||
**UnifiedVuln JSON (internal canonical)**
|
||||
|
||||
```json
|
||||
{
|
||||
"advisory_key": "CVE-2025-12345",
|
||||
"ids": { "cve": "CVE-2025-12345", "ghsa": "GHSA-xxxx", "bdu": "BDU:2025-06025", "jvndb": "JVNDB-2025-000123", "msrc": "2025-Jan" },
|
||||
"titles": [{ "text": "Buffer overflow in foo()", "lang": "en" }],
|
||||
"summary": { "text": "...", "lang": "en" },
|
||||
"published": "2025-06-21T12:00:00Z",
|
||||
"modified": "2025-07-03T09:00:00Z",
|
||||
"severity": {
|
||||
"cvss_v3": { "base": 9.8, "vector": "CVSS:3.1/..." },
|
||||
"cvss_v4": null,
|
||||
"vendor": "Critical"
|
||||
},
|
||||
"affected": [
|
||||
{ "platform": "os-distro", "name": "ubuntu:20.04",
|
||||
"cpe": "cpe:/o:canonical:ubuntu_linux:20.04",
|
||||
"version_range": "pkg:deb/ubuntu/foo<1.2.3-0ubuntu0.20.04.1",
|
||||
"fixed_by": "1.2.3-0ubuntu0.20.04.1"
|
||||
}
|
||||
],
|
||||
"references": [
|
||||
{ "url": "https://msrc.microsoft.com/update-guide", "kind": "advisory", "source": "MSRC" }
|
||||
],
|
||||
"exploitation": { "cisa_kev": true, "nkcki": false },
|
||||
"provenance": [
|
||||
{ "source": "RedHat", "document": "https://...", "method": "parser", "confidence": 1.0 }
|
||||
],
|
||||
"psirt": [{ "vendor": "Cisco", "advisory": "cisco-sa-..." }]
|
||||
}
|
||||
```
|
||||
|
||||
**Ranges**
|
||||
|
||||
* **OS packages**: distro semantics (Debian **EVR**, RPM **NEVRA**). Prefer OVAL/PSIRT source whenever available.
|
||||
* **Language ecosystems**: **SemVer** ranges with **PURL** coordinates; use OSV/GHSA fields for introduced/fixed events.
|
||||
* **Severity**: keep **all** CVSS sources; compute a max/consensus for display but preserve originals.
|
||||
|
||||
---
|
||||
|
||||
## 7) FSTEC BDU hybrid extraction (HTML→schema with gated LLM fallback)
|
||||
|
||||
1. **HTML parser** extracts into `BduHtmlExtract`.
|
||||
2. Validate against **internal XML schema** (XSD). Rules: `bdu_id` format `^BDU:\d{4}-\d{5}$`; CVE regex; date parse; severity enumeration.
|
||||
3. On validation failure: run **LLM extraction** (temperature 0) to the same JSON Schema; accept **only** if post‑validation passes and `confidence ≥ minConfidence`. Mark `provenance.method = "llm"`.
|
||||
4. Keep audit logs locally; default **offline model** in sovereign builds.
|
||||
|
||||
---
|
||||
|
||||
## 8) Merge & reconciliation (deterministic)
|
||||
|
||||
* **Identity**: prefer **CVE**; fallback to other keys (BDU/JVN/GHSA/MSRC/CISCO‑SA/VMSA/USN/DSA/SUSE‑SU/ICSA). Canonical `advisory_key`.
|
||||
* **Aliases**: store all cross‑refs (CVE, GHSA, OSV, JVN, BDU, MSRC, CISCO‑SA, ORACLE‑CPU, APSB, APPLE‑HT, CHROMIUM‑POST, VMSA, RHSA, USN, DSA, SUSE‑SU, ICSA, CWE, CPE, PURL…).
|
||||
* **Precedence**:
|
||||
|
||||
* OVAL/PSIRT **override** NVD for OS package ranges.
|
||||
* **KEV** sets exploitation flags only (no severity override).
|
||||
* Regional feeds **enrich** (severity text, mitigation, local notes).
|
||||
* **Determinism**: merged canonical JSON is hashed; store in `merge_event`.
|
||||
|
||||
---
|
||||
|
||||
## 9) Packaging & publishing
|
||||
|
||||
**v0**: render **vuln‑list–shaped JSON** → invoke stock **`trivy-db`** builder to get `metadata.json` + `trivy.db` → tar to `db.tar.gz` → **ORAS push** to your registry with **Trivy DB media types**.
|
||||
**v1**: native C# packager writing BoltDB + `metadata.json` and pushing via ORAS directly.
|
||||
|
||||
**Output contracts**
|
||||
|
||||
* **OCI media types**: layer `application/vnd.aquasec.trivy.db.layer.v1.tar+gzip`; config `application/vnd.aquasec.trivy.config.v1+json`.
|
||||
* Consumers point Trivy at your repo: `--db-repository REGISTRY/PATH`; for air‑gap ship `db.tar.gz`.
|
||||
|
||||
---
|
||||
|
||||
## 10) CLI (idempotent)
|
||||
|
||||
```
|
||||
feedser init
|
||||
feedser fetch --source nvd|cve|ghsa|osv|jvn|certcc|kev|kisa|certin|certfr|certbund|acsc|cccs|bdu|nkcki|msrc|cisco|oracle|adobe|apple|chromium|vmware|redhat|ubuntu|debian|suse [--since ...]
|
||||
feedser parse --source ...
|
||||
feedser merge
|
||||
feedser export vuln-list --out ./out/vuln-list/
|
||||
feedser pack trivy-db --out ./out/db.tar.gz
|
||||
feedser push trivy-db --repo registry.local/security/trivy-db --tag 2 [--auth env|file]
|
||||
feedser sign --artifact ./out/db.tar.gz --key cosign.key
|
||||
feedser status
|
||||
feedser gc --keep-raw 3
|
||||
feedser doctor # media types, registry auth, schema checks
|
||||
```
|
||||
|
||||
Exit codes: non‑zero on schema failure, network failure after retries, or merge non‑determinism.
|
||||
|
||||
---
|
||||
|
||||
## 11) Config (`etc/feedser.yaml`) — extended
|
||||
|
||||
```yaml
|
||||
storage:
|
||||
driver: sqlite
|
||||
dsn: "Data Source=feedser.db"
|
||||
|
||||
sources:
|
||||
cve: { enabled: true }
|
||||
nvd: { enabled: true, window_hours: 6 }
|
||||
ghsa: { enabled: true, github_token: "${GITHUB_TOKEN:-}", api: "rest+graphql" }
|
||||
osv: { enabled: true }
|
||||
jvn:
|
||||
enabled: true
|
||||
api_base: "https://jvndb.jvn.jp/en/apis/"
|
||||
window_days: 7
|
||||
certcc: { enabled: true }
|
||||
kev: { enabled: true }
|
||||
kisa: { enabled: false } # enable when endpoints/feeds are reachable in environment
|
||||
certin: { enabled: true }
|
||||
certfr: { enabled: true }
|
||||
certbund: { enabled: true }
|
||||
acsc: { enabled: true }
|
||||
cccs: { enabled: true }
|
||||
ru:
|
||||
bdu:
|
||||
enabled: true
|
||||
htmlFallback: true
|
||||
llmFallback: "gated"
|
||||
minConfidence: 0.85
|
||||
nkcki:
|
||||
enabled: true
|
||||
msrc: { enabled: true }
|
||||
cisco: { enabled: true, token: "${CISCO_OPENVULN_TOKEN:-}" }
|
||||
oracle: { enabled: true }
|
||||
adobe: { enabled: true }
|
||||
apple: { enabled: true }
|
||||
chromium: { enabled: true }
|
||||
vmware: { enabled: true }
|
||||
redhat:
|
||||
enabled: true
|
||||
api_base: "https://access.redhat.com/hydra/rest/securitydata"
|
||||
ubuntu:
|
||||
enabled: true
|
||||
api_base: "https://ubuntu.com/security/api"
|
||||
debian: { enabled: true }
|
||||
suse: { enabled: true }
|
||||
|
||||
packaging:
|
||||
trivy:
|
||||
publish: true
|
||||
repo: "registry.local/security/trivy-db"
|
||||
tag: "2"
|
||||
offline_bundle: true
|
||||
|
||||
observability:
|
||||
metrics: "stdout"
|
||||
logs: "json"
|
||||
level: "Information"
|
||||
tracing: "otlp"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 12) Observability & performance
|
||||
|
||||
* **Logs**: structured (Serilog); include `source`, `uri`, `status`, `parseMs`, `mappedCount`, `mergeDelta`.
|
||||
* **Metrics**: fetch latency, parse/validation failures, dedupe ratio, DB compaction time, package size, **per‑source rate‑limit counters**.
|
||||
* **Tracing**: OpenTelemetry spans per connector/step.
|
||||
* **Perf**: bounded parallelism per source; streaming XML; content‑hash short‑circuit for unchanged docs.
|
||||
|
||||
---
|
||||
|
||||
## 13) Tests & quality gates
|
||||
|
||||
* **Schema validation** for each connector (external JSON/XML → DTOs).
|
||||
* **Golden fixtures** per source (NVD page, GHSA JSON, OSV, JVN JVNRSS/VULDEF, CERT/CC VU HTML, BDU HTML, NKCKI PDF→text, MSRC CVRF, Cisco openVuln JSON, Oracle CPU HTML, Adobe APSB HTML, Apple HT list, Chrome Releases HTML, VMSA HTML, Red Hat API JSON, USN JSON, Debian JSON, SUSE HTML).
|
||||
* **Merge determinism** (hash‑stable).
|
||||
* **Parity scans**: compare Trivy scan using our DB vs upstream baseline on a reference set of images (differences expected where OVAL narrows ranges).
|
||||
* **Media‑type conformance** (OCI).
|
||||
* **Reproducible packaging**: build ID = hash(vuln‑list tree).
|
||||
|
||||
**Connector DoD**: watermarking; retries/backoff; schema‑validated parsing; mapping; unit tests; goldens; incremental pass; metrics.
|
||||
|
||||
---
|
||||
|
||||
## 14) Security & compliance
|
||||
|
||||
* Default **offline**; explicit allowlist per source host.
|
||||
* **LLM usage isolated** to BDU fallback; no external calls unless configured; redact logs; audit stored locally.
|
||||
* **cosign** signing for artifacts; store SHA256 and manifest digests.
|
||||
* Respect robots/ToS; prefer official APIs/feeds where available.
|
||||
|
||||
---
|
||||
|
||||
## 15) Concrete TODOs (first sprints)
|
||||
|
||||
1. **Storage**: EF Core models & migrations; `watermark` infra; repositories.
|
||||
2. **NVD**: windowed fetch; JSON Schema validation; mapper; watermark.
|
||||
3. **OVAL/Distros**: Red Hat (API+OVAL), Ubuntu (USN+API), Debian (JSON), SUSE (advisories).
|
||||
4. **KEV**: JSON ingest → `exploit_known=true`.
|
||||
5. **GHSA/OSV**: REST + GraphQL; map PURL/semver; handle `cvss_severities`.
|
||||
6. **JVN**: JVNRSS + VULDEF; alias mapping; watermark.
|
||||
7. **RU**: BDU HTML parser + XSD + LLM fallback; NKCKI bulletins harvester.
|
||||
8. **PSIRTs**: MSRC CVRF; Cisco openVuln; Oracle CPU; Adobe APSB; Apple; Chromium; VMware (VMSA).
|
||||
9. **Merge Engine**: aliasing + precedence; canonical JSON + hashing.
|
||||
10. **Export/Pack**: vuln‑list renderer; Trivy DB packaging; **ORAS push**; **offline bundle**.
|
||||
11. **CLI & doctor**; **cosign sign**; **status**.
|
||||
|
||||
---
|
||||
|
||||
## 16) MASTER SOURCE CATALOG (as provided — preserved verbatim)
|
||||
|
||||
### Primary Vulnerability Databases / Advisory Portals
|
||||
|
||||
| Vulnerability DB | Who Supports It | Type | URL | DB Type | What Data It Has |
|
||||
| ---------------------------------------------- | ---------------------------------------------------------- | ----------------------------------- | ----------------------------------------------------------------------------------------------------------------------------- | ---------------------- | ------------------------------------------------------------------ |
|
||||
| **CVE (Common Vulnerabilities and Exposures)** | MITRE (with CNA partners) | Identifier registry | [https://cve.org](https://cve.org) | Global ID registry | CVE IDs, basic description, references, assigner info |
|
||||
| **CERT/CC Vulnerability Notes** | Carnegie Mellon CERT/CC | National CERT / coordination center | [https://kb.cert.org/vuls](https://kb.cert.org/vuls) | Vulnerability Notes DB | VU# IDs, description, impact, vendors affected, references |
|
||||
| **JVN (Japan Vulnerability Notes)** | JPCERT/CC + IPA (Japan) | National CERT | [https://jvn.jp/en/](https://jvn.jp/en/) | Advisory DB | JVN IDs, affected products, mitigation, CVE mappings |
|
||||
| **RU-CERT** | Coordination Center for .RU / Russian CERT | National CERT | [https://www.cert.ru](https://www.cert.ru) | Advisory DB | Russian advisories, incident/vulnerability notes |
|
||||
| **CISA KEV Catalog & Advisories** | US CISA (DHS) | Government CERT / advisories | [https://www.cisa.gov/known-exploited-vulnerabilities](https://www.cisa.gov/known-exploited-vulnerabilities) | Catalog | KEV IDs, CVE links, exploited-in-wild status, remediation deadline |
|
||||
| **KISA (Korean CERT)** | Korea Internet & Security Agency | National CERT | [https://www.krcert.or.kr](https://www.krcert.or.kr) / [https://www.boho.or.kr/en/main.do](https://www.boho.or.kr/en/main.do) | Advisory portal | Korean advisories, CVE refs, guidance |
|
||||
| **CERT-In (India)** | Ministry of Electronics & IT | National CERT | [https://www.cert-in.org.in](https://www.cert-in.org.in) | Advisory portal | Indian CERT advisories, affected vendors, CVEs |
|
||||
| **ANSSI (France)** | Agence nationale de la sécurité des systèmes d'information | National CERT | [https://www.cert.ssi.gouv.fr](https://www.cert.ssi.gouv.fr) | Advisory portal | French advisories, technical notes, CVE refs |
|
||||
| **BSI (Germany, CERT-Bund)** | German Federal Office for Information Security | National CERT | [https://www.bsi.bund.de](https://www.bsi.bund.de) | Advisory portal | Vulnerability advisories, vendor notifications |
|
||||
| **ACSC (Australia)** | Australian Cyber Security Centre | National CERT | [https://www.cyber.gov.au](https://www.cyber.gov.au) | Advisory portal | Australian advisories, CVE refs, guidance |
|
||||
| **CCCS (Canada)** | Canadian Centre for Cyber Security | National CERT | [https://www.cyber.gc.ca](https://www.cyber.gc.ca) | Advisory portal | Canadian advisories, CVE refs |
|
||||
|
||||
### Vendor / PSIRT Databases (Primary)
|
||||
|
||||
| Vulnerability DB | Who Supports It | Type | URL | DB Type | What Data It Has |
|
||||
| --------------------------------------------- | --------------- | ------------ | ---------------------------------------------------------------------------------------------------------------------------- | ----------------------- | ----------------------------------------------------------- |
|
||||
| **Microsoft Security Response Center (MSRC)** | Microsoft | Vendor PSIRT | [https://msrc.microsoft.com/update-guide](https://msrc.microsoft.com/update-guide) | Advisory portal | MSRC IDs, CVE mappings, affected products/versions, patches |
|
||||
| **Cisco PSIRT** | Cisco Systems | Vendor PSIRT | [https://tools.cisco.com/security/center/publicationListing.x](https://tools.cisco.com/security/center/publicationListing.x) | Advisory DB | Cisco advisories, CVEs, product impact, fixes |
|
||||
| **Oracle CPU / Security Alerts** | Oracle | Vendor PSIRT | [https://www.oracle.com/security-alerts/](https://www.oracle.com/security-alerts/) | Advisory DB | Oracle CPUs, CVEs, affected products/versions, patches |
|
||||
| **Adobe Security Bulletins & Advisories** | Adobe | Vendor PSIRT | [https://helpx.adobe.com/security.html](https://helpx.adobe.com/security.html) | Advisory DB | APSB/APA IDs, CVEs, affected software, patches |
|
||||
| **Apple Security Updates** | Apple | Vendor PSIRT | [https://support.apple.com/en-us/HT201222](https://support.apple.com/en-us/HT201222) (security updates index) | Advisory portal | Apple advisories, CVEs, product versions, patches |
|
||||
| **Google Chromium Security** | Google | Vendor PSIRT | [https://chromereleases.googleblog.com](https://chromereleases.googleblog.com) | Advisory blog / tracker | Chromium/Android advisories, CVEs, fixes |
|
||||
| **VMware Security Advisories (VMSA)** | VMware/Broadcom | Vendor PSIRT | [https://www.broadcom.com/support/vmware-security-advisories](https://www.broadcom.com/support/vmware-security-advisories) | Advisory DB | VMSA IDs, CVEs, product versions, fixes |
|
||||
|
||||
### Linux Distribution Security Trackers (Primary)
|
||||
|
||||
| Vulnerability DB | Who Supports It | Type | URL | DB Type | What Data It Has |
|
||||
| ------------------------------------------- | --------------- | ------------------ | ---------------------------------------------------------------------------------------- | ------------------------------------- | --------------------------------------------------------- |
|
||||
| **Red Hat Security Data / RHSA** | Red Hat | Distro Security DB | [https://access.redhat.com/security/updates](https://access.redhat.com/security/updates) | Security advisories & OVAL/JSON feeds | RHSA IDs, CVEs, fixed package versions, affected products |
|
||||
| **Canonical Ubuntu Security Notices (USN)** | Canonical | Distro Security DB | [https://ubuntu.com/security/notices](https://ubuntu.com/security/notices) | Advisory DB | USN IDs, CVEs, affected packages, patches |
|
||||
| **Debian Security Tracker (DSA)** | Debian Project | Distro Security DB | [https://security-tracker.debian.org](https://security-tracker.debian.org) | Tracker + Advisories | DSA IDs, CVEs, package status per release |
|
||||
| **SUSE Security Announcements** | SUSE | Distro Security DB | [https://www.suse.com/support/security/](https://www.suse.com/support/security/) | Advisory DB | SUSE-SA/Update IDs, CVEs, package fix versions |
|
||||
|
||||
### Open Source Ecosystem Advisory Databases (Primary)
|
||||
|
||||
| Vulnerability DB | Who Supports It | Type | URL | DB Type | What Data It Has |
|
||||
| ----------------------------------------- | ---------------------- | ----------------------- | -------------------------------------------------------------- | --------------- | ------------------------------------------------------------------------------------------------- |
|
||||
| **GitHub Security Advisories (GHSA)** | GitHub (Microsoft) | Open Source Advisory DB | [https://github.com/advisories](https://github.com/advisories) | Advisory DB | GHSA IDs, CVEs, affected repos/packages, patches, severity |
|
||||
| **OSV.dev (Open Source Vulnerabilities)** | Google / OSS community | Open Source Advisory DB | [https://osv.dev](https://osv.dev) | Schema-based DB | OSV IDs, CVEs, affected ecosystems (npm, PyPI, Go, crates.io, Maven, etc.), version ranges, fixes |
|
||||
|
||||
### Specialized (ICS / Sectoral)
|
||||
|
||||
| Vulnerability DB | Who Supports It | Type | URL | DB Type | What Data It Has |
|
||||
| ----------------------- | ------------------ | ------------- | ---------------------------------------------------------------------------------------- | --------------- | --------------------------------------------------------------------- |
|
||||
| **CISA ICS Advisories** | US CISA (ICS‑CERT) | ICS sector DB | [https://www.cisa.gov/ics/advisories](https://www.cisa.gov/ics/advisories) | Advisory DB | ICS advisory IDs, CVEs, affected vendors, exploitability, mitigations |
|
||||
| **Kaspersky ICS CERT** | Kaspersky Lab | ICS CERT | [https://ics-cert.kaspersky.com/advisories/](https://ics-cert.kaspersky.com/advisories/) | Advisory portal | ICS advisories, CVEs, technical detail, mitigations |
|
||||
|
||||
---
|
||||
|
||||
## 17) Field‑mapping guide (per family)
|
||||
|
||||
**PSIRT**: set `psirt_flags.vendor` + vendor advisory ID in `alias` and `psirt_flags.advisory_id_text`. Always attach **patch references** and **fixed versions** into `affected.fixed_by`.
|
||||
**Distros**: treat **OVAL/JSON** as range authority; `alias` with RHSA/USN/DSA/SUSE‑SU; attach per‑suite/package status.
|
||||
**CERTs**: attach `reference(kind=bulletin)` and severity text; use as enrichment unless they include authoritative package ranges.
|
||||
**ICS**: map vendor & model families into `affected.platform="ics-vendor"` with product tags.
|
||||
**KEV**: set exploitation flags only.
|
||||
**BDU/JVN**: include local IDs (BDU, JVNDB) in `alias` and specific flags in `ru_flags`/`jp_flags`.
|
||||
|
||||
---
|
||||
|
||||
## 18) Reference commands & snippets
|
||||
|
||||
**ORAS push (Trivy DB v2)**
|
||||
|
||||
```bash
|
||||
oras push --artifact-type application/vnd.aquasec.trivy.config.v1+json \
|
||||
"registry.local/security/trivy-db:2" \
|
||||
db.tar.gz:application/vnd.aquasec.trivy.db.layer.v1.tar+gzip
|
||||
```
|
||||
|
||||
**Point Trivy at our repo**
|
||||
|
||||
```bash
|
||||
trivy image --db-repository registry.local/security/trivy-db --download-db-only
|
||||
```
|
||||
|
||||
**BDU LLM fallback gate (pseudo)**
|
||||
|
||||
```csharp
|
||||
if (!BduSchemaValidator.IsValid(parsed)) {
|
||||
var json = LlmExtractToJson(rawText, schema: BduSchema, temperature: 0);
|
||||
if (!BduSchemaValidator.IsValid(json) || Confidence(json) < minConfidence) Fail("BDU: low confidence");
|
||||
Save(json, provenance: "llm");
|
||||
} else {
|
||||
Save(parsed, provenance: "parser");
|
||||
}
|
||||
```
|
||||
|
||||
## Reference notes (authoritative links for the agent)
|
||||
|
||||
**Trivy self‑hosting / DB media types / vuln‑list**
|
||||
|
||||
* Trivy self‑hosting databases and `--db-repository` flag. ([trivy.dev][1])
|
||||
* DB repository & required OCI media type (`application/vnd.aquasec.trivy.db.layer.v1.tar+gzip`). ([Aqua Security][2])
|
||||
* `vuln-list` and `vuln-list-update` (inputs/build). ([GitHub][3])
|
||||
* `trivy-db` tool (builder/DB format). ([GitHub][4])
|
||||
* GitLab registry media‑type support for trivy‑db (confirmation of the two media types). ([about.gitlab.com][5])
|
||||
|
||||
**Global registries / cross‑ecosystem**
|
||||
|
||||
* CVE program (official). ([CVE][6])
|
||||
* NVD general/search. ([NVD][7])
|
||||
* GHSA DB and APIs (REST/GraphQL + deprecation notice). ([GitHub][8])
|
||||
* OSV.dev (DB + data sources). ([OSV][9])
|
||||
|
||||
**National CERTs**
|
||||
|
||||
* CERT/CC Vulnerability Notes + docs. ([CERT Coordination Center][10])
|
||||
* JVN / MyJVN API (Japan). ([JVN iPedia][11])
|
||||
* RU‑CERT (coordination center profile & site). ([cctld.ru][12])
|
||||
* KISA/KrCERT portals and examples. ([boho.or.kr][13])
|
||||
* CERT‑In (site, CNA role, sample advisory). ([CERT-IN][14])
|
||||
* ANSSI CERT‑FR portal and *avis*. ([cert.ssi.gouv.fr][15])
|
||||
* BSI CERT‑Bund WID pages. ([wid.cert-bund.de][16])
|
||||
* ACSC advisories hub. ([cyber.gov.au][17])
|
||||
* CCCS advisories hub. ([Canadian Centre for Cyber Security][18])
|
||||
|
||||
**Russia‑specific**
|
||||
|
||||
* BDU site and documentation of XML/Excel dumps (context). ([bdu.fstec.ru][19])
|
||||
* NKCKI vulnerability bulletins list. ([safe-surf.ru][20])
|
||||
|
||||
**Vendor PSIRTs**
|
||||
|
||||
* MSRC Security Update Guide + CVRF API examples. ([msrc.microsoft.com][21])
|
||||
* Cisco PSIRT advisories + openVuln API. ([Cisco][22])
|
||||
* Oracle CPU schedule / advisories. ([Oracle][23])
|
||||
* Adobe security advisories (index + product). ([Adobe Help Center][24])
|
||||
* Apple security releases index (HT201222 lineage). ([Apple Support][25])
|
||||
* Chrome Releases (stable updates with security fixes). ([Chrome Releases][26])
|
||||
* VMware Security Advisories (VMSA) on Broadcom; move notice. ([Broadcom][27])
|
||||
|
||||
**Linux distributions**
|
||||
|
||||
* Red Hat Security Data API (+ changelog/pointers). ([Red Hat Docs][28])
|
||||
* Ubuntu Security Notices & Security API. ([Ubuntu][29])
|
||||
* Debian Security Tracker (docs + JSON). ([Debian Security Tracker][30])
|
||||
* SUSE advisories. ([SUSE][31])
|
||||
|
||||
**Exploitation & ICS**
|
||||
|
||||
* CISA KEV catalog. ([CISA][32])
|
||||
* CISA ICS advisories hub (ICSA). ([CISA][33])
|
||||
* Kaspersky ICS‑CERT advisories. ([Kaspersky ICS-CERT][34])
|
||||
|
||||
If you want me to produce **starter EF models + migrations** and a **full `feedser.yaml`** file reflecting all of the above, I can output those files now.
|
||||
|
||||
[1]: https://trivy.dev/v0.60/docs/advanced/self-hosting/?utm_source=chatgpt.com "Self-Hosting Trivy's Databases"
|
||||
[2]: https://aquasecurity.github.io/trivy/v0.56/docs/configuration/db/?utm_source=chatgpt.com "DB"
|
||||
[3]: https://github.com/aquasecurity/vuln-list?utm_source=chatgpt.com "aquasecurity/vuln-list: NVD, Ubuntu, Alpine"
|
||||
[4]: https://github.com/aquasecurity/trivy-db?utm_source=chatgpt.com "aquasecurity/trivy-db"
|
||||
[5]: https://gitlab.com/gitlab-org/container-registry/-/merge_requests/957?utm_source=chatgpt.com "Add trivy-db media types - container-registry"
|
||||
[6]: https://www.cve.org/?utm_source=chatgpt.com "CVE: Common Vulnerabilities and Exposures"
|
||||
[7]: https://nvd.nist.gov/vuln/search?utm_source=chatgpt.com "NVD - Search and Statistics"
|
||||
[8]: https://github.com/advisories?utm_source=chatgpt.com "GitHub Advisory Database"
|
||||
[9]: https://osv.dev/?utm_source=chatgpt.com "OSV - Open Source Vulnerabilities"
|
||||
[10]: https://www.kb.cert.org/?utm_source=chatgpt.com "CERT Vulnerability Notes Database"
|
||||
[11]: https://jvndb.jvn.jp/en/apis/index.html?utm_source=chatgpt.com "MyJVN API"
|
||||
[12]: https://cctld.ru/files/pdf/RU-CERT.pdf?utm_source=chatgpt.com "RU-CERT.pdf"
|
||||
[13]: https://www.boho.or.kr/en/main.do?utm_source=chatgpt.com "KISA 인터넷 보호나라&KrCERT"
|
||||
[14]: https://www.cert-in.org.in/CNA.jsp?utm_source=chatgpt.com "CVE Numbering Authority (CNA) at CERT-In"
|
||||
[15]: https://www.cert.ssi.gouv.fr/?utm_source=chatgpt.com "CERT-FR – Centre gouvernemental de veille, d ... - l'ANSSI"
|
||||
[16]: https://wid.cert-bund.de/?utm_source=chatgpt.com "Warn- und Informationsdienst - Startseite - CERT-Bund"
|
||||
[17]: https://www.cyber.gov.au/about-us/view-all-content/alerts-and-advisories?utm_source=chatgpt.com "Alerts and advisories"
|
||||
[18]: https://www.cyber.gc.ca/en/alerts-advisories?utm_source=chatgpt.com "Alerts and advisories"
|
||||
[19]: https://bdu.fstec.ru/vul?utm_source=chatgpt.com "Уязвимости - БДУ"
|
||||
[20]: https://safe-surf.ru/specialists/bulletins-nkcki/?utm_source=chatgpt.com "Список новых уязвимостей ПО | Уведомления НКЦКИ"
|
||||
[21]: https://msrc.microsoft.com/update-guide?utm_source=chatgpt.com "Security Update Guide"
|
||||
[22]: https://sec.cloudapps.cisco.com/security/center/publicationListing.x?utm_source=chatgpt.com "Cisco Security Advisories"
|
||||
[23]: https://www.oracle.com/security-alerts/?utm_source=chatgpt.com "Critical Patch Updates, Security Alerts and Bulletins"
|
||||
[24]: https://helpx.adobe.com/security/security-bulletin.html?utm_source=chatgpt.com "Security Bulletins and Advisories"
|
||||
[25]: https://support.apple.com/en-us/100100?utm_source=chatgpt.com "Apple security releases"
|
||||
[26]: https://chromereleases.googleblog.com/?utm_source=chatgpt.com "Chrome Releases"
|
||||
[27]: https://www.broadcom.com/support/vmware-security-advisories?utm_source=chatgpt.com "VMware Security Advisories"
|
||||
[28]: https://docs.redhat.com/en/documentation/red_hat_security_data_api/1.0/html-single/red_hat_security_data_api/index?utm_source=chatgpt.com "Red Hat Security Data API | 1.0"
|
||||
[29]: https://ubuntu.com/security/notices?utm_source=chatgpt.com "Ubuntu Security Notices"
|
||||
[30]: https://security-tracker.debian.org/?utm_source=chatgpt.com "Security Bug Tracker - Debian"
|
||||
[31]: https://www.suse.com/support/update/?utm_source=chatgpt.com "SUSE:Update Advisories"
|
||||
[32]: https://www.cisa.gov/known-exploited-vulnerabilities-catalog?utm_source=chatgpt.com "Known Exploited Vulnerabilities Catalog"
|
||||
[33]: https://www.cisa.gov/news-events/ics-advisories?utm_source=chatgpt.com "ICS Advisories"
|
||||
[34]: https://ics-cert.kaspersky.com/advisories/?utm_source=chatgpt.com "Advisories"
|
||||
Reference in New Issue
Block a user