Files

Vladimir Moushkov 192b0add77 feat: introduce plugin host and DI infrastructure

2025-10-03 19:37:10 +03:00

50 KiB

Raw Blame History

AGENTS.md — Feedser (Stella Ops)

YOU ARE: a senior C#/.NET 10 (Preview 7, SDK 10.0.100-preview.7.25380.108) engineer‑agent tasked with building Feedser, a CLI that fetches, normalizes, reconciles, and packages primary, non‑aggregated vulnerability intelligence into a single feed‑merge database and exports a Trivy‑compatible DB (OCI artifact) for offline/self‑hosted scanning.
MODE: factual, deterministic, test‑first, with strict provenance and reproducibility.
BOUNDARIES: implement architecture & code in this repo only. No secret exfiltration. Default offline.

0) Stella Ops — condensed context

Stella Ops is a container & infra security platform built for sovereign/offline operation. Key storylines: Δ‑SBOM warm path, nightly re‑checks, policy‑as‑code, signed artifacts, optional AI remediation, and regional/air‑gapped operation.
Feedser is foundational: it powers the scanner by producing a unified, deduped, explainable vulnerability database and a self‑hosted Trivy DB.

Sibling components (stable contracts, no code here):

Scanner (stellaops.module.scanning) — consumes Trivy‑compatible DB → findings + SBOM digests.
Policy Engine, Signed Artifacts Service (cosign), AIRE (AI suggestions), SecretsScanner, MailDaemon, Offline Kit, RU/EEU adapters (CryptoPro TLS, RU cert chains), UI Shell.

1) Problem statement

Fetch authoritative primary sources (global + regional + PSIRT + distro + CERTs + ICS).
Parse & Normalize to a UnifiedVuln model.
Reconcile/Deduplicate deterministically across sources with precedence rules.
Persist into feed‑merge DB with both bootstrap‑from‑scratch and incremental refresh.
Package & Publish:
- Trivy DB (v2) OCI artifact for scanners (--db-repository),
- optional vuln‑list–shaped JSON tree (to reuse trivy-db builder),
- optional signed offline bundle.

Non‑goals v0: building a new scanner or a custom Java DB; we only ensure Scanner can target our self‑hosted DB.

2) High‑level architecture


[Connectors]  ──►  [Source DTO validation] ──► [Normalizer → UnifiedVuln]
CVE/NVD, GHSA/OSV, JVN, CERT/CC, CISA KEV, KISA, CERT-In, ANSSI (CERT-FR),
BSI (CERT-Bund WID), ACSC, CCCS, RU: BDU + NKCKI, Vendor PSIRTs (MSRC, Cisco,
Oracle CPU, Adobe APSB, Apple, Chromium, VMware), Distros (Red Hat, Ubuntu,
Debian, SUSE), ICS (CISA ICS, Kaspersky ICS-CERT)
│
▼
[Merge/Reconcile Engine]
(aliases, precedence, ranges, KEV flags, PSIRT flags)
│
▼
[Feed‑Merge DB (SQLite→Postgres)]
│
┌──────────────────┴──────────────────┐
▼                                     ▼
[Export: vuln‑list JSON]             [Packager: Trivy DB v2]
│                                     │
(CI)                          [ORAS push / offline tar]

Principles

Determinism (same inputs → same outputs, hashed) and provenance per field.
OVAL (vendor/distro) overrides generic ranges for OS packages.
Regional feeds enrich rather than blindly override unless they carry stronger package‑level truth.

3) Repository layout (create exactly)


src/Feedser/
Feedser.Cli/                     # .NET 10 preview console (System.CommandLine)
Feedser.Core/                    # domain model & orchestration
Feedser.Storage/                 # EF Core migrations (SQLite dev/CI; Postgres prod)
Feedser.Connectors/
Common/                        # HTTP, pagination, ETag, backoff, schema validators
Cve/                           # CVE registry (id+refs)
Nvd/                           # NVD API v2 windows
Ghsa/                          # GHSA REST/GraphQL
Osm.Osv/                       # OSV API
Jvn/                           # MyJVN (JVNRSS/VULDEF)
CertCc/                        # CERT/CC Vulnerability Notes
Kev/                           # CISA Known Exploited
Kr.Kisa/                       # KISA/KrCERT advisories
In.CertIn/                     # CERT-In advisories
Fr.CertFr/                     # ANSSI CERT-FR avis/alertes
De.CertBund/                   # BSI CERT-Bund WID
Au.Acsc/                       # ACSC advisories
Ca.Cccs/                       # CCCS advisories
Ru.Bdu/                        # FSTEC BDU (HTML→schema; LLM fallback gated)
Ru.Nkcki/                      # NKCKI bulletins (HTML/PDF→text)
Vndr.Msrc/                     # MSRC CVRF
Vndr.Cisco/                    # Cisco PSIRT openVuln
Vndr.Oracle/                   # Oracle CPU/advisories
Vndr.Adobe/                    # Adobe APSB/APA
Vndr.Apple/                    # Apple HT201222 feed
Vndr.Chromium/                 # Chrome Releases security posts
Vndr.Vmware/                   # VMSA (Broadcom portal)
Distro.RedHat/                 # Red Hat Security Data API + OVAL
Distro.Ubuntu/                 # USN + Security API
Distro.Debian/                 # Debian Security Tracker JSON
Distro.Suse/                   # SUSE Update Advisories
Ics.Cisa/                      # CISA ICS advisories (ICSA-*)
Ics.Kaspersky/                 # Kaspersky ICS-CERT advisories
Feedser.Merge/                   # dedupe/aliases/precedence/version-ranges
Feedser.Export.VulnList/         # optional vuln-list JSON renderer
Feedser.Packagers.TrivyDb/       # db.tar.gz + metadata.json + ORAS push
Feedser.Signing/                 # cosign integration
Feedser.Tests/
etc/
feedser.yaml                   # config template (extended, see §11)
schemas/                       # JSON Schema/XSD for inputs & internal payloads
samples/                       # golden fixtures per source

4) Unified data model (relational + evented)

Storage default: SQLite (dev/CI), Postgres (prod). EF Core migrations. Dapper for hot paths if needed.

Tables (no change from v1 + PSIRT/CERT flags)

source(id, name, type, base_url, auth_mode, notes)
watermark(source_id, cursor, updated_at) ← incremental windows per source
document(id, source_id, uri, fetched_at, content_sha256, content_type, status, raw_blob?, metadata_json)
advisory(id, advisory_key, title, summary, lang, published, modified, severity_cvss_v3?, severity_cvss_v4?, vendor_severity?, exploit_known bool)
alias(advisory_id, scheme, value) — schemes include: CVE, GHSA, OSV, JVN, BDU, VU (CERT/CC), MSRC, CISCO‑SA, ORACLE‑CPU, APSB/APA, APPLE‑HT, CHROMIUM‑POST, VMSA, RHSA, USN, DSA, SUSE‑SU, ICSA, CWE, CPE, PURL, etc.
affected(advisory_id, platform, name, version_range, cpe?, purl?, fixed_by?, introduced_version?)
reference(advisory_id, url, kind, source_tag) — kind examples: advisory, patch, bulletin, kb, blog, vendor, exploit
provenance(advisory_id, document_id, extraction, confidence, fields_mask)
kev_flag(advisory_id, kev_id, added_date, due_date?)
ru_flags(advisory_id, bdu_id?, nkcki_ids_json?, ru_severity?, notes?)
jp_flags(advisory_id, jvndb_id?, jvn_category?, vendor_status?)
psirt_flags(advisory_id, vendor, advisory_id_text, product_tags_json?)
merge_event(id, advisory_key, before_hash, after_hash, merged_at)

Indexes: unique(advisory_key); index(scheme,value); index(platform,name); index(published); index(modified).

4.1) Alternate storage (MongoDB) — mapping (per your plan)

If a MongoDB deployment is preferred, mirror the relational shape as collections with analogous names (source, watermark, document, advisory, alias, affected, reference, provenance, kev_flag, ru_flags, jp_flags, psirt_flags, merge_event).

Keep advisory documents flat and embed aliases[], affected[], references[] when practical; store provenance entries as embedded or sidecar collection depending on document growth.
Maintain deterministic canonical JSON for merges; hash stored in merge_event.
Incremental refreshes rely on the same per‑source watermarks.

5) Source connectors — contracts & incremental strategy

Common interface

public interface IFeedConnector {
  string SourceName { get; }
  Task FetchAsync(FeedserContext db, CancellationToken ct);   // populate document rows
  Task ParseAsync(FeedserContext db, CancellationToken ct);   // document -> DTOs (validated)
  Task MapAsync(FeedserContext db, CancellationToken ct);     // DTOs -> UnifiedVuln tables + provenance
}

5.1 Registries & cross‑ecosystem

CVE (cve.org) — identifier registry. Fetch for alias cross‑checks; minimal fields only. Watermark by last seen ID/time.
NVD API v2 — sliding modified windows (e.g., 6–12h) with backoff and pagination. Persist CVSS/CWE/CPE as aliases; capture change history if present. Watermark = last successful modified_end.
GHSA — REST “global security advisories” + GraphQL for richer fields; note: cvss → cvss_severities deprecation → map accordingly. Watermark by updated timestamp/ID cursor.
OSV — fetch per eco or time range; map PURL + SemVer ranges.

5.2 National CERTs (incremental via RSS/API/pages)

CERT/CC Vulnerability Notes — scrape/archive pages (VU#), and/or GitHub data archive when suitable. Watermark by VU publish date/ID.
JVN / MyJVN (Japan) — MyJVN API: JVNRSS (overview) + VULDEF (detail). Watermark by dateFirstPublished/dateLastUpdated. Map JVNDB IDs, CVE aliases, vendor status.
RU‑CERT — advisory/news portal; treat as enrichment references (aliases+refs), not a primary package range source. Watermark by post date.
KISA (KrCERT/KRCERT) — advisories/notices portal. Watermark by advisory date/ID.
CERT‑In (India) — CIAD advisories via portal pages; Watermark by advisory code/date.
ANSSI/CERT‑FR — avis/alertes RSS and list pages; Watermark by advisory ID/date.
BSI CERT‑Bund (WID) — “Technische Sicherheitshinweise” pages/feeds; Watermark by bulletin ID/date.
ACSC (Australia) — alerts/advisories; Watermark by publish date/slug.
CCCS (Canada) — advisories page; Watermark by date/slug.

5.3 Russia‑specific

FSTEC BDU — hybrid: primary HTML parser → validate against our internal XML schema; if validation fails → LLM extraction fallback (strictly gated; see §7). Also support bulk DB ingests if official XML/Excel exports are available in the environment. Watermark by BDU ID/date.
NKCKI — bulletins list (HTML/PDF). Extract structured fields via PDF→text pipeline + post‑validation. Watermark by bulletin ID/date.

5.4 Vendor PSIRTs (canonical)

MSRC — CVRF API monthly and per‑advisory endpoints. Watermark by month + last modified. Alias: MSRC:<YYYY-MMM>; references to KBs/CVEs.
Cisco PSIRT (openVuln API) — REST; filter by last published/updated. Alias: CISCO-SA:<slug>; map fixed releases.
Oracle CPU / Security Alerts — quarterly schedule (3rd Tue of Jan/Apr/Jul/Oct). Scrape CPU pages and advisories. Alias: CPU:<YYYY-QQ>; link per‑product CVEs. Watermark by CPU cycle.
Adobe APSB/APA — advisory index pages + product feeds. Alias: APSB-YYYY-XX.
Apple — HT201222/“About Apple security releases” index page(s). Alias: APPLE-HT:HT201222:<yyyy-mm-dd> + per‑product pages.
Google Chromium — Chrome Releases blog “Stable Channel Update” posts with security fix lists. Alias: CHROMIUM-POST:<date>.
VMware (VMSA) — Broadcom support portal VMSA pages; parse ID + affected products + CVEs. Alias: VMSA-YYYY-XXXX.

5.5 Linux distributions

Red Hat Security Data API (CSAF/OVAL/CVE); plus OVAL content. Precedence for OS packages. Watermark via API last_modified/etag. Alias: RHSA-YYYY:NNNN.
Ubuntu USN — USN list + Security API; Watermark by USN ID/date. Alias: USN-####-#.
Debian Security Tracker — JSON dataset for CVE↔package↔suite; Watermark by file etag/commit. Alias: DSA-####-# (when present).
SUSE — security/update advisories pages; Watermark by SUSE‑SU ID/date. Alias: SUSE-SU-YYYY:NNNN.

5.6 Specialized / ICS

CISA ICS advisories (ICSA) — list feeds; Watermark by ICSA‑ID. Alias: ICSA-YY-###-##.
Kaspersky ICS‑CERT — advisories list; Watermark by advisory ID/date; treat as authoritative vendor ICS source for impacted OT products.

5.7 Exploitation & enrichment

CISA KEV — JSON catalog; set exploitation flag (exploit_known=true), store kev_id, added_date, due_date.

6) Normalization details

UnifiedVuln JSON (internal canonical)

{
  "advisory_key": "CVE-2025-12345",
  "ids": { "cve": "CVE-2025-12345", "ghsa": "GHSA-xxxx", "bdu": "BDU:2025-06025", "jvndb": "JVNDB-2025-000123", "msrc": "2025-Jan" },
  "titles": [{ "text": "Buffer overflow in foo()", "lang": "en" }],
  "summary": { "text": "...", "lang": "en" },
  "published": "2025-06-21T12:00:00Z",
  "modified": "2025-07-03T09:00:00Z",
  "severity": {
    "cvss_v3": { "base": 9.8, "vector": "CVSS:3.1/..." },
    "cvss_v4": null,
    "vendor": "Critical"
  },
  "affected": [
    { "platform": "os-distro", "name": "ubuntu:20.04",
      "cpe": "cpe:/o:canonical:ubuntu_linux:20.04",
      "version_range": "pkg:deb/ubuntu/foo<1.2.3-0ubuntu0.20.04.1",
      "fixed_by": "1.2.3-0ubuntu0.20.04.1"
    }
  ],
  "references": [
    { "url": "https://msrc.microsoft.com/update-guide", "kind": "advisory", "source": "MSRC" }
  ],
  "exploitation": { "cisa_kev": true, "nkcki": false },
  "provenance": [
    { "source": "RedHat", "document": "https://...", "method": "parser", "confidence": 1.0 }
  ],
  "psirt": [{ "vendor": "Cisco", "advisory": "cisco-sa-..." }]
}

Ranges

OS packages: distro semantics (Debian EVR, RPM NEVRA). Prefer OVAL/PSIRT source whenever available.
Language ecosystems: SemVer ranges with PURL coordinates; use OSV/GHSA fields for introduced/fixed events.
Severity: keep all CVSS sources; compute a max/consensus for display but preserve originals.

7) FSTEC BDU hybrid extraction (HTML→schema with gated LLM fallback)

HTML parser extracts into BduHtmlExtract.
Validate against internal XML schema (XSD). Rules: bdu_id format ^BDU:\d{4}-\d{5}$; CVE regex; date parse; severity enumeration.
On validation failure: run LLM extraction (temperature 0) to the same JSON Schema; accept only if post‑validation passes and confidence ≥ minConfidence. Mark provenance.method = "llm".
Keep audit logs locally; default offline model in sovereign builds.

8) Merge & reconciliation (deterministic)

Identity: prefer CVE; fallback to other keys (BDU/JVN/GHSA/MSRC/CISCO‑SA/VMSA/USN/DSA/SUSE‑SU/ICSA). Canonical advisory_key.
Aliases: store all cross‑refs (CVE, GHSA, OSV, JVN, BDU, MSRC, CISCO‑SA, ORACLE‑CPU, APSB, APPLE‑HT, CHROMIUM‑POST, VMSA, RHSA, USN, DSA, SUSE‑SU, ICSA, CWE, CPE, PURL…).
Precedence:
- OVAL/PSIRT override NVD for OS package ranges.
- KEV sets exploitation flags only (no severity override).
- Regional feeds enrich (severity text, mitigation, local notes).
Determinism: merged canonical JSON is hashed; store in merge_event.

9) Packaging & publishing

v0: render vuln‑list–shaped JSON → invoke stock trivy-db builder to get metadata.json + trivy.db → tar to db.tar.gz → ORAS push to your registry with Trivy DB media types. v1: native C# packager writing BoltDB + metadata.json and pushing via ORAS directly.

Output contracts

OCI media types: layer application/vnd.aquasec.trivy.db.layer.v1.tar+gzip; config application/vnd.aquasec.trivy.config.v1+json.
Consumers point Trivy at your repo: --db-repository REGISTRY/PATH; for air‑gap ship db.tar.gz.

10) CLI (idempotent)

feedser init
feedser fetch   --source nvd|cve|ghsa|osv|jvn|certcc|kev|kisa|certin|certfr|certbund|acsc|cccs|bdu|nkcki|msrc|cisco|oracle|adobe|apple|chromium|vmware|redhat|ubuntu|debian|suse [--since ...]
feedser parse   --source ...
feedser merge
feedser export vuln-list  --out ./out/vuln-list/
feedser pack   trivy-db   --out ./out/db.tar.gz
feedser push   trivy-db   --repo registry.local/security/trivy-db --tag 2 [--auth env|file]
feedser sign             --artifact ./out/db.tar.gz --key cosign.key
feedser status
feedser gc               --keep-raw 3
feedser doctor           # media types, registry auth, schema checks

Exit codes: non‑zero on schema failure, network failure after retries, or merge non‑determinism.

11) Config (`etc/feedser.yaml`) — extended

storage:
  driver: sqlite
  dsn: "Data Source=feedser.db"

sources:
  cve:    { enabled: true }
  nvd:    { enabled: true, window_hours: 6 }
  ghsa:   { enabled: true, github_token: "${GITHUB_TOKEN:-}", api: "rest+graphql" }
  osv:    { enabled: true }
  jvn:
    enabled: true
    api_base: "https://jvndb.jvn.jp/en/apis/"
    window_days: 7
  certcc: { enabled: true }
  kev:    { enabled: true }
  kisa:   { enabled: false }   # enable when endpoints/feeds are reachable in environment
  certin: { enabled: true }
  certfr: { enabled: true }
  certbund: { enabled: true }
  acsc:   { enabled: true }
  cccs:   { enabled: true }
  ru:
    bdu:
      enabled: true
      htmlFallback: true
      llmFallback: "gated"
      minConfidence: 0.85
    nkcki:
      enabled: true
  msrc:   { enabled: true }
  cisco:  { enabled: true, token: "${CISCO_OPENVULN_TOKEN:-}" }
  oracle: { enabled: true }
  adobe:  { enabled: true }
  apple:  { enabled: true }
  chromium: { enabled: true }
  vmware: { enabled: true }
  redhat:
    enabled: true
    api_base: "https://access.redhat.com/hydra/rest/securitydata"
  ubuntu:
    enabled: true
    api_base: "https://ubuntu.com/security/api"
  debian: { enabled: true }
  suse:   { enabled: true }

packaging:
  trivy:
    publish: true
    repo: "registry.local/security/trivy-db"
    tag: "2"
    offline_bundle: true

observability:
  metrics: "stdout"
  logs: "json"
  level: "Information"
  tracing: "otlp"

12) Observability & performance

Logs: structured (Serilog); include source, uri, status, parseMs, mappedCount, mergeDelta.
Metrics: fetch latency, parse/validation failures, dedupe ratio, DB compaction time, package size, per‑source rate‑limit counters.
Tracing: OpenTelemetry spans per connector/step.
Perf: bounded parallelism per source; streaming XML; content‑hash short‑circuit for unchanged docs.

13) Tests & quality gates

Schema validation for each connector (external JSON/XML → DTOs).
Golden fixtures per source (NVD page, GHSA JSON, OSV, JVN JVNRSS/VULDEF, CERT/CC VU HTML, BDU HTML, NKCKI PDF→text, MSRC CVRF, Cisco openVuln JSON, Oracle CPU HTML, Adobe APSB HTML, Apple HT list, Chrome Releases HTML, VMSA HTML, Red Hat API JSON, USN JSON, Debian JSON, SUSE HTML).
Merge determinism (hash‑stable).
Parity scans: compare Trivy scan using our DB vs upstream baseline on a reference set of images (differences expected where OVAL narrows ranges).
Media‑type conformance (OCI).
Reproducible packaging: build ID = hash(vuln‑list tree).

Connector DoD: watermarking; retries/backoff; schema‑validated parsing; mapping; unit tests; goldens; incremental pass; metrics.

14) Security & compliance

Default offline; explicit allowlist per source host.
LLM usage isolated to BDU fallback; no external calls unless configured; redact logs; audit stored locally.
cosign signing for artifacts; store SHA256 and manifest digests.
Respect robots/ToS; prefer official APIs/feeds where available.

15) Concrete TODOs (first sprints)

Storage: EF Core models & migrations; watermark infra; repositories.
NVD: windowed fetch; JSON Schema validation; mapper; watermark.
OVAL/Distros: Red Hat (API+OVAL), Ubuntu (USN+API), Debian (JSON), SUSE (advisories).
KEV: JSON ingest → exploit_known=true.
GHSA/OSV: REST + GraphQL; map PURL/semver; handle cvss_severities.
JVN: JVNRSS + VULDEF; alias mapping; watermark.
RU: BDU HTML parser + XSD + LLM fallback; NKCKI bulletins harvester.
PSIRTs: MSRC CVRF; Cisco openVuln; Oracle CPU; Adobe APSB; Apple; Chromium; VMware (VMSA).
Merge Engine: aliasing + precedence; canonical JSON + hashing.
Export/Pack: vuln‑list renderer; Trivy DB packaging; ORAS push; offline bundle.
CLI & doctor; cosign sign; status.

16) MASTER SOURCE CATALOG (as provided — preserved verbatim)

Primary Vulnerability Databases / Advisory Portals

Vulnerability DB	Who Supports It	Type	URL	DB Type	What Data It Has
CVE (Common Vulnerabilities and Exposures)	MITRE (with CNA partners)	Identifier registry	https://cve.org	Global ID registry	CVE IDs, basic description, references, assigner info
CERT/CC Vulnerability Notes	Carnegie Mellon CERT/CC	National CERT / coordination center	https://kb.cert.org/vuls	Vulnerability Notes DB	VU# IDs, description, impact, vendors affected, references
JVN (Japan Vulnerability Notes)	JPCERT/CC + IPA (Japan)	National CERT	https://jvn.jp/en/	Advisory DB	JVN IDs, affected products, mitigation, CVE mappings
RU-CERT	Coordination Center for .RU / Russian CERT	National CERT	https://www.cert.ru	Advisory DB	Russian advisories, incident/vulnerability notes
CISA KEV Catalog & Advisories	US CISA (DHS)	Government CERT / advisories	https://www.cisa.gov/known-exploited-vulnerabilities	Catalog	KEV IDs, CVE links, exploited-in-wild status, remediation deadline
KISA (Korean CERT)	Korea Internet & Security Agency	National CERT	https://www.krcert.or.kr / https://www.boho.or.kr/en/main.do	Advisory portal	Korean advisories, CVE refs, guidance
CERT-In (India)	Ministry of Electronics & IT	National CERT	https://www.cert-in.org.in	Advisory portal	Indian CERT advisories, affected vendors, CVEs
ANSSI (France)	Agence nationale de la sécurité des systèmes d'information	National CERT	https://www.cert.ssi.gouv.fr	Advisory portal	French advisories, technical notes, CVE refs
BSI (Germany, CERT-Bund)	German Federal Office for Information Security	National CERT	https://www.bsi.bund.de	Advisory portal	Vulnerability advisories, vendor notifications
ACSC (Australia)	Australian Cyber Security Centre	National CERT	https://www.cyber.gov.au	Advisory portal	Australian advisories, CVE refs, guidance
CCCS (Canada)	Canadian Centre for Cyber Security	National CERT	https://www.cyber.gc.ca	Advisory portal	Canadian advisories, CVE refs

Vendor / PSIRT Databases (Primary)

Vulnerability DB	Who Supports It	Type	URL	DB Type	What Data It Has
Microsoft Security Response Center (MSRC)	Microsoft	Vendor PSIRT	https://msrc.microsoft.com/update-guide	Advisory portal	MSRC IDs, CVE mappings, affected products/versions, patches
Cisco PSIRT	Cisco Systems	Vendor PSIRT	https://tools.cisco.com/security/center/publicationListing.x	Advisory DB	Cisco advisories, CVEs, product impact, fixes
Oracle CPU / Security Alerts	Oracle	Vendor PSIRT	https://www.oracle.com/security-alerts/	Advisory DB	Oracle CPUs, CVEs, affected products/versions, patches
Adobe Security Bulletins & Advisories	Adobe	Vendor PSIRT	https://helpx.adobe.com/security.html	Advisory DB	APSB/APA IDs, CVEs, affected software, patches
Apple Security Updates	Apple	Vendor PSIRT	https://support.apple.com/en-us/HT201222 (security updates index)	Advisory portal	Apple advisories, CVEs, product versions, patches
Google Chromium Security	Google	Vendor PSIRT	https://chromereleases.googleblog.com	Advisory blog / tracker	Chromium/Android advisories, CVEs, fixes
VMware Security Advisories (VMSA)	VMware/Broadcom	Vendor PSIRT	https://www.broadcom.com/support/vmware-security-advisories	Advisory DB	VMSA IDs, CVEs, product versions, fixes

Linux Distribution Security Trackers (Primary)

Vulnerability DB	Who Supports It	Type	URL	DB Type	What Data It Has
Red Hat Security Data / RHSA	Red Hat	Distro Security DB	https://access.redhat.com/security/updates	Security advisories & OVAL/JSON feeds	RHSA IDs, CVEs, fixed package versions, affected products
Canonical Ubuntu Security Notices (USN)	Canonical	Distro Security DB	https://ubuntu.com/security/notices	Advisory DB	USN IDs, CVEs, affected packages, patches
Debian Security Tracker (DSA)	Debian Project	Distro Security DB	https://security-tracker.debian.org	Tracker + Advisories	DSA IDs, CVEs, package status per release
SUSE Security Announcements	SUSE	Distro Security DB	https://www.suse.com/support/security/	Advisory DB	SUSE-SA/Update IDs, CVEs, package fix versions

Open Source Ecosystem Advisory Databases (Primary)

Vulnerability DB	Who Supports It	Type	URL	DB Type	What Data It Has
GitHub Security Advisories (GHSA)	GitHub (Microsoft)	Open Source Advisory DB	https://github.com/advisories	Advisory DB	GHSA IDs, CVEs, affected repos/packages, patches, severity
OSV.dev (Open Source Vulnerabilities)	Google / OSS community	Open Source Advisory DB	https://osv.dev	Schema-based DB	OSV IDs, CVEs, affected ecosystems (npm, PyPI, Go, crates.io, Maven, etc.), version ranges, fixes

Specialized (ICS / Sectoral)

Vulnerability DB	Who Supports It	Type	URL	DB Type	What Data It Has
CISA ICS Advisories	US CISA (ICS‑CERT)	ICS sector DB	https://www.cisa.gov/ics/advisories	Advisory DB	ICS advisory IDs, CVEs, affected vendors, exploitability, mitigations
Kaspersky ICS CERT	Kaspersky Lab	ICS CERT	https://ics-cert.kaspersky.com/advisories/	Advisory portal	ICS advisories, CVEs, technical detail, mitigations

17) Field‑mapping guide (per family)

PSIRT: set psirt_flags.vendor + vendor advisory ID in alias and psirt_flags.advisory_id_text. Always attach patch references and fixed versions into affected.fixed_by. Distros: treat OVAL/JSON as range authority; alias with RHSA/USN/DSA/SUSE‑SU; attach per‑suite/package status. CERTs: attach reference(kind=bulletin) and severity text; use as enrichment unless they include authoritative package ranges. ICS: map vendor & model families into affected.platform="ics-vendor" with product tags. KEV: set exploitation flags only. BDU/JVN: include local IDs (BDU, JVNDB) in alias and specific flags in ru_flags/jp_flags.

18) Reference commands & snippets

ORAS push (Trivy DB v2)

oras push --artifact-type application/vnd.aquasec.trivy.config.v1+json \
  "registry.local/security/trivy-db:2" \
  db.tar.gz:application/vnd.aquasec.trivy.db.layer.v1.tar+gzip

Point Trivy at our repo

trivy image --db-repository registry.local/security/trivy-db --download-db-only

BDU LLM fallback gate (pseudo)

if (!BduSchemaValidator.IsValid(parsed)) {
  var json = LlmExtractToJson(rawText, schema: BduSchema, temperature: 0);
  if (!BduSchemaValidator.IsValid(json) || Confidence(json) < minConfidence) Fail("BDU: low confidence");
  Save(json, provenance: "llm");
} else {
  Save(parsed, provenance: "parser");
}

Reference notes (authoritative links for the agent)

Trivy self‑hosting / DB media types / vuln‑list

Trivy self‑hosting databases and --db-repository flag. (trivy.dev)
DB repository & required OCI media type (application/vnd.aquasec.trivy.db.layer.v1.tar+gzip). (Aqua Security)
vuln-list and vuln-list-update (inputs/build). (GitHub)
trivy-db tool (builder/DB format). (GitHub)
GitLab registry media‑type support for trivy‑db (confirmation of the two media types). (about.gitlab.com)

Global registries / cross‑ecosystem

CVE program (official). (CVE)
NVD general/search. (NVD)
GHSA DB and APIs (REST/GraphQL + deprecation notice). (GitHub)
OSV.dev (DB + data sources). (OSV)

National CERTs

CERT/CC Vulnerability Notes + docs. (CERT Coordination Center)
JVN / MyJVN API (Japan). (JVN iPedia)
RU‑CERT (coordination center profile & site). (cctld.ru)
KISA/KrCERT portals and examples. (boho.or.kr)
CERT‑In (site, CNA role, sample advisory). (CERT-IN)
ANSSI CERT‑FR portal and avis. (cert.ssi.gouv.fr)
BSI CERT‑Bund WID pages. (wid.cert-bund.de)
ACSC advisories hub. (cyber.gov.au)
CCCS advisories hub. (Canadian Centre for Cyber Security)

Russia‑specific

BDU site and documentation of XML/Excel dumps (context). (bdu.fstec.ru)
NKCKI vulnerability bulletins list. (safe-surf.ru)

Vendor PSIRTs

MSRC Security Update Guide + CVRF API examples. (msrc.microsoft.com)
Cisco PSIRT advisories + openVuln API. (Cisco)
Oracle CPU schedule / advisories. (Oracle)
Adobe security advisories (index + product). (Adobe Help Center)
Apple security releases index (HT201222 lineage). (Apple Support)
Chrome Releases (stable updates with security fixes). (Chrome Releases)
VMware Security Advisories (VMSA) on Broadcom; move notice. (Broadcom)

Linux distributions

Red Hat Security Data API (+ changelog/pointers). (Red Hat Docs)
Ubuntu Security Notices & Security API. (Ubuntu)
Debian Security Tracker (docs + JSON). (Debian Security Tracker)
SUSE advisories. (SUSE)

Exploitation & ICS

CISA KEV catalog. (CISA)
CISA ICS advisories hub (ICSA). (CISA)
Kaspersky ICS‑CERT advisories. (Kaspersky ICS-CERT)

If you want me to produce starter EF models + migrations and a full feedser.yaml file reflecting all of the above, I can output those files now.

19) Role Kickstart Playbooks (aligned with ARCHITECTURE.md & IMPLEMENTATION.md)

19.1 Shared pre-flight checklist

Read ARCHITECTURE.md §§1–3 and IMPLEMENTATION.md §§0–3 before writing code; treat this document as the quick-start guide and the others as depth references.
Confirm local toolchain: .NET 10 Preview 7 SDK (10.0.100-preview.7.25380.108), Docker, MongoDB (local or container), oras CLI, cosign, yq, jq, Chrome/Chromium for HTML schema inspection, pdftotext for NKCKI extracts.
Sync repo structure with IMPLEMENTATION.md naming (StellaOps.Feedser.* projects, Mongo storage) even if the CLI-first layout above still exists; prefer additive commits that converge both plans until deprecation is agreed.
Establish secrets handling: load tokens (GitHub, Cisco openVuln, etc.) via environment variables referenced in etc/feedser.yaml.
Instrument everything: Serilog + OpenTelemetry hooks should be wired during the first implementation of any loop so QA can observe behaviour from day one.
Definition of Done (all roles): schema validation in tests, deterministic outputs (hash snapshot checked in), logging/metrics assertions, and hand-off notes in /docs/handbook/ROLE/<source>.md.

19.2 BE-Base — Platform & Pipeline owner

Sprint 0 focus (IMPLEMENTATION.md §1): create the StellaOps.Feedser.sln, add the WebService, Core, Models, Storage.Mongo, Source.Common, Exporter.Json, Exporter.TrivyDb projects; seed Directory.Build.props/targets, .editorconfig, and analyzer packages (StyleCop.Analyzers, nullable enable, treat warnings as errors in CI).
Wire CI (.github/workflows/feedser-ci.yml or internal equivalent) running dotnet restore/build/test, lint (StyleCop), and container build check; artifacts should include the WebService image and test results.
Produce devcontainer + Dockerfile aligned with the Mongo-first run mode (ARCHITECTURE.md §2). Ensure mongo sidecar is declared in devcontainer.json for immediate onboarding.
Establish configuration plumbing: bind appsettings.json + feedser.yaml into strongly typed options, configure reload-on-change, and hydrate SourceState/ExportState repositories.
Create Mongo collections/indexes exactly as catalogued in ARCHITECTURE.md §3; provide integration tests under StellaOps.Feedser.Tests/Storage that assert index presence and TTL semantics.
Publish contribution docs: /docs/contribute.md summarizing coding standards, release tagging, and commit style (Implementation §1.5–1.6).
Hand-off: once WebService boots with /health and /ready endpoints, scheduler skeleton, and Mongo indices created on startup, notify BE-Conn/BE-Merge/BE-Export via project board and land a baseline tag (v0.1.0-alpha1).

19.3 BE-Conn-X — Source Connector engineers

Priority waves (IMPLEMENTATION.md §4 + §5): registries (CVE/NVD/GHSA/OSV), national CERTs, vendor PSIRTs, distros, KEV, ICS. Pick sources in order of dependency on precedence rules (e.g., Red Hat before Debian for RPM logic).
Workflow per source:
1. Extend StellaOps.Feedser.Source.Common with fetch helpers (rate-limit, retries) if not already present; reuse IConnectorClock to respect windowed crawls.
2. Implement FetchAsync to persist documents into Mongo document collection with SHA256 + metadata; follow the watermark guidance in §5 of this file.
3. Validate raw payloads against schemas (JSON Schema, XSD, or Joi equivalent) and store sanitized DTOs in dto. Record validation stats in metrics.
4. Map DTOs into canonical advisories using StellaOps.Feedser.Models. Guarantee alias completeness and provenance entries (parser vs llm).
5. Provide golden fixtures in samples/<source>/ and component tests under StellaOps.Feedser.Tests/Source.<SourceName> that cover fetch (with canned HTTP responses), parse, map, and incremental resume.
Definition of Ready: Base pipeline live, HTTP client registered, schema stub written, test scaffolding ready, tokens/keys documented. Coordinate with BE-Base for any additional shared tooling (HTML to XML transforms, PDF text extraction).
Definition of Done: deterministic DTO + map outputs (snapshot hashed, stored under Feedser.Tests/__snapshots__), metrics counters added, resume cursor unit tests, and documentation entry in /docs/sources/<source>.md describing rate limits, cursor logic, tested fixtures.
Engage QA early: schedule schema reviews before mapper coding to catch field omissions.

19.4 BE-Merge — Canonical merge & dedupe

Start after BE-Base lands canonical models scaffold (IMPLEMENTATION.md §5). Own the StellaOps.Feedser.Models definitions, canonical serialization, and hash calculator used by merge_event.
Implement version range utilities (RPM NEVRA, Debian EVR, SemVer) with exhaustive tests covering edge cases (epoch handling, tilde comparisons, wildcard SUSE ranges). Use fixtures from distro connectors to validate precedence rules.
Build the identity graph: CVE-first resolution with fallback to other alias schemes (outlined in §8 of this file). Guarantee deterministic ordering (sort keys + stable merges) and record beforeHash/afterHash deltas.
Enforce precedence policies: PSIRT/OVAL override generic ranges, KEV toggles exploitation flags without modifying severity, regional feeds enrich severity text but do not downgrade vendor truth. Cover these with integration tests using fused fixture sets.
Expose a Merge service with idempotent MergeAsync(IEnumerable<Guid> advisoryIds) that writes both canonical document and merge_event records per run. Provide metrics (merge.delta.count, merge.identity.conflicts).
Deliver initial merge deterministic test: same fixture set processed twice yields identical hashes; store hash snapshot under /tests/data/merge/expected-hash.json.

19.5 BE-Export — JSON & Trivy DB packaging

After BE-Merge exposes stable canonical output, implement JSON exporter mirroring aquasecurity/vuln-list layout (Implementation §5 step 4; ARCHITECTURE.md stage 3). Ensure directory determinism (sorted keys, newline conventions) and record export cursor consumption.
For Trivy DB exporter: wrap the official trivy-db builder initially; orchestrate invocation in-process or via CLI with reproducible environment variables. Persist resulting metadata.json, BoltDB file, and tarball in export_state with digests.
Integrate ORAS push pipeline guarded by dry-run flag; support offline bundle packaging (zip/tar) for air-gapped delivery. Provide config-driven repo/tag resolution from feedser.yaml.
Metrics to emit: export.duration, export.records, export.size_bytes, export.delta counts, ORAS push success/failure.
Tests: snapshot JSON tree, verify OCI manifest media types, simulate incremental export using seeded advisory records. Provide CLI smoke test hitting POST /jobs/export/trivydb (or CLI equivalent once added).

19.6 QA — Validation & observability lead

Build and maintain test matrix covering connectors, merge, export; enforce schema evolution workflows (any schema change must include updated fixtures, docs, and backward-compat diff summary).
Own component test harness utilities (HTTP canned server, PDF→text conversion mocks, time-travel clock). Ensure connectors implement IClock dependency for deterministic tests.
Set up golden snapshot review cadence; add tooling (dotnet test --filter Category=Golden -- Dump) to regenerate fixtures and compare diffs.
Monitor observability baselines: define default alert thresholds for fetch failures, merge conflicts, export delays. Provide dashboards or documented query templates (Grafana/Prometheus) referenced in /docs/observability.md.
Verify reproducibility: rerun end-to-end pipeline twice and compare exported digests; document discrepancies and feed them back to BE-Merge/BE-Export.

19.7 Coordination & delivery cadence

Sprint naming follows Implementation plan (Sprint 0–4+). Maintain a shared Kanban board with swimlanes per role; entries should reference the numbered tasks from IMPLEMENTATION.md for traceability.
Hold daily 15-minute sync focused on blockers, schema changes, and source-specific rate limits discovered. Escalate major schema/API shifts immediately and capture in /docs/incidents/YYYY-MM-DD.md.
Milestone gates:
1. Platform GA — BE-Base tasks 1–6 closed; Mongo + scheduler operational.
2. Core connectors online — NVD, GHSA, OSV, KEV delivering mapped advisories.
3. Distro precedence — Red Hat + Ubuntu connectors validating version precedence vs NVD.
4. Merge deterministic — hash stability test green.
5. Export ready — JSON + Trivy DB artifacts validated and pushed to staging registry.

19.8 Hand-off artifacts & documentation expectations

Every completed task must append/adjust documentation: connector guides, merge rules, exporter usage. Store under /docs mirroring role ownership; include sample commands, curl invocations, and expected outputs.
Check in sample configs (etc/feedser.local.yaml.example) per role with placeholders for sensitive values.
Capture post-task retros in docs/retros/<sprint>-<task>.md summarizing lessons, API quirks, schema diffs, and follow-up tickets.
Maintain a shared glossary (docs/glossary.md) covering abbreviations (KEV, OVAL, EVR, NEVRA, GridFS) to speed up onboarding for new engineers.

The sections above convert the role taxonomy from IMPLEMENTATION.md into actionable playbooks while keeping the authoritative architecture and data contracts inline with this guide. Use them to bootstrap work, then dive into ARCHITECTURE.md and IMPLEMENTATION.md for exhaustive references and sprint-by-sprint sequencing.

Dependency Injection Registration Baseline

Every product or library that needs DI registration must expose a <ProductNamespace>.DependencyInjection namespace/folder containing one or more implementations of StellaOps.DependencyInjection.IDependencyInjectionRoutine.
When distributing only a subset of connectors/exporters, create a dedicated solution that references just the desired plugin projects; the shared build rules always emit every StellaOps.Feedser.Source.* and StellaOps.Feedser.Exporter.* assembly into PluginBinaries by default.
The IDependencyInjectionRoutine interface lives in src/__Libraries/StellaOps.DependencyInjection and enables both static opt-in registration and reflection-driven root composition.
Each library should provide static helpers that wrap its DI registrations and a thin routine class that forwards to those helpers, for example:

public interface IDependencyInjectionRoutine
{
    IServiceCollection Register(
        IServiceCollection services,
        IConfiguration configuration);
}

public static class NamespaceLibrary
{
    public static IServiceCollection RegisterNamespaceLibrary(
        IServiceCollection services,
        IConfiguration configuration)
    {
        // ...
        return services;
    }
}

public sealed class DependencyInjectionRoutine : IDependencyInjectionRoutine
{
    public IServiceCollection Register(
        IServiceCollection services,
        IConfiguration configuration)
    {
        return NamespaceLibrary.RegisterNamespaceLibrary(services, configuration);
    }
}

50 KiB Raw Blame History Unescape Escape

AGENTS.md — Feedser (Stella Ops)

0) Stella Ops — condensed context