up
This commit is contained in:
Vladimir Moushkov
2
.gitignore
vendored
2
.gitignore
vendored
@@ -23,3 +23,5 @@ TestResults/
|
||||
seed-data/ics-cisa/*.csv
|
||||
seed-data/ics-cisa/*.xlsx
|
||||
seed-data/ics-cisa/*.sha256
|
||||
seed-data/cert-bund/**/*.json
|
||||
seed-data/cert-bund/**/*.sha256
|
||||
|
||||
42
SPRINTS.md
42
SPRINTS.md
@@ -105,8 +105,8 @@
|
||||
| Sprint 4 | Schema Parity & Freshness Alignment | src/StellaOps.Feedser.Merge/TASKS.md | DONE (2025-10-15) | Team Merge & QA Enforcement | FEEDMERGE-ENGINE-04-005 | Connector coordination for new advisory fields<br>GHSA/NVD/OSV connectors now ship description, CWE, and canonical metric data with refreshed fixtures; merge coordination log updated and exporters notified. |
|
||||
| Sprint 4 | Schema Parity & Freshness Alignment | src/StellaOps.Feedser.Exporter.Json/TASKS.md | DONE (2025-10-15) | Team Exporters – JSON | FEEDEXPORT-JSON-04-001 | Surface new advisory fields in JSON exporter<br>Update schemas/offline bundle + fixtures once model/core parity lands.<br>2025-10-15: `dotnet test src/StellaOps.Feedser.Exporter.Json.Tests` validated canonical metric/CWE emission. |
|
||||
| Sprint 4 | Schema Parity & Freshness Alignment | src/StellaOps.Feedser.Exporter.TrivyDb/TASKS.md | DONE (2025-10-15) | Team Exporters – Trivy DB | FEEDEXPORT-TRIVY-04-001 | Propagate new advisory fields into Trivy DB package<br>Extend Bolt builder, metadata, and regression tests for the expanded schema.<br>2025-10-15: `dotnet test src/StellaOps.Feedser.Exporter.TrivyDb.Tests` confirmed canonical metric/CWE propagation. |
|
||||
| Sprint 4 | Schema Parity & Freshness Alignment | src/StellaOps.Feedser.Source.Ghsa/TASKS.md | TODO | Team Connector Regression Fixtures | FEEDCONN-GHSA-04-004 | Harden CVSS fallback so canonical metric ids persist when GitHub omits vectors; extend fixtures and document severity precedence hand-off to Merge. |
|
||||
| Sprint 4 | Schema Parity & Freshness Alignment | src/StellaOps.Feedser.Source.Osv/TASKS.md | TODO | Team Connector Expansion – GHSA/NVD/OSV | FEEDCONN-OSV-04-005 | Map OSV advisories lacking CVSS vectors to canonical metric ids/notes and document CWE provenance quirks; schedule parity fixture updates. |
|
||||
| Sprint 4 | Schema Parity & Freshness Alignment | src/StellaOps.Feedser.Source.Ghsa/TASKS.md | DONE (2025-10-16) | Team Connector Regression Fixtures | FEEDCONN-GHSA-04-004 | Harden CVSS fallback so canonical metric ids persist when GitHub omits vectors; extend fixtures and document severity precedence hand-off to Merge. |
|
||||
| Sprint 4 | Schema Parity & Freshness Alignment | src/StellaOps.Feedser.Source.Osv/TASKS.md | DONE (2025-10-16) | Team Connector Expansion – GHSA/NVD/OSV | FEEDCONN-OSV-04-005 | Map OSV advisories lacking CVSS vectors to canonical metric ids/notes and document CWE provenance quirks; schedule parity fixture updates. |
|
||||
| Sprint 5 | Vexer Core Foundations | src/StellaOps.Vexer.Core/TASKS.md | DONE (2025-10-15) | Team Vexer Core & Policy | VEXER-CORE-01-001 | Stand up canonical VEX claim/consensus records with deterministic serializers so Storage/Exports share a stable contract. |
|
||||
| Sprint 5 | Vexer Core Foundations | src/StellaOps.Vexer.Core/TASKS.md | DONE (2025-10-15) | Team Vexer Core & Policy | VEXER-CORE-01-002 | Implement trust-weighted consensus resolver with baseline policy weights, justification gates, telemetry output, and majority/tie handling. |
|
||||
| Sprint 5 | Vexer Core Foundations | src/StellaOps.Vexer.Core/TASKS.md | DONE (2025-10-15) | Team Vexer Core & Policy | VEXER-CORE-01-003 | Publish shared connector/exporter/attestation abstractions and deterministic query signature utilities for cache/attestation workflows. |
|
||||
@@ -116,29 +116,35 @@
|
||||
| Sprint 5 | Vexer Core Foundations | src/StellaOps.Vexer.Policy/TASKS.md | DONE (2025-10-16) | Team Vexer Policy | VEXER-POLICY-01-004 | Implement YAML/JSON schema validation and deterministic diagnostics for operator bundles. |
|
||||
| Sprint 5 | Vexer Core Foundations | src/StellaOps.Vexer.Policy/TASKS.md | DONE (2025-10-16) | Team Vexer Policy | VEXER-POLICY-01-005 | Add policy change tracking, snapshot digests, and telemetry/logging hooks. |
|
||||
| Sprint 5 | Vexer Core Foundations | src/StellaOps.Vexer.Storage.Mongo/TASKS.md | DONE (2025-10-15) | Team Vexer Storage | VEXER-STORAGE-01-001 | Mongo mapping registry plus raw/export entities and DI extensions in place. |
|
||||
| Sprint 5 | Vexer Core Foundations | src/StellaOps.Vexer.Storage.Mongo/TASKS.md | TODO | Team Vexer Storage | VEXER-STORAGE-01-004 | Build provider/consensus/cache class maps and related collections. |
|
||||
| Sprint 5 | Vexer Core Foundations | src/StellaOps.Vexer.Storage.Mongo/TASKS.md | DONE (2025-10-16) | Team Vexer Storage | VEXER-STORAGE-01-004 | Build provider/consensus/cache class maps and related collections. |
|
||||
| Sprint 5 | Vexer Core Foundations | src/StellaOps.Vexer.Export/TASKS.md | DONE (2025-10-15) | Team Vexer Export | VEXER-EXPORT-01-001 | Export engine delivers cache lookup, manifest creation, and policy integration. |
|
||||
| Sprint 5 | Vexer Core Foundations | src/StellaOps.Vexer.Export/TASKS.md | TODO | Team Vexer Export | VEXER-EXPORT-01-004 | Connect export engine to attestation client and persist Rekor metadata. |
|
||||
| Sprint 5 | Vexer Core Foundations | src/StellaOps.Vexer.Attestation/TASKS.md | TODO | Team Vexer Attestation | VEXER-ATTEST-01-001 | Implement in-toto predicate + DSSE builder providing envelopes for export attestation. |
|
||||
| Sprint 5 | Vexer Core Foundations | src/StellaOps.Vexer.Connectors.Abstractions/TASKS.md | TODO | Team Vexer Connectors | VEXER-CONN-ABS-01-001 | Deliver shared connector context/base classes so provider plug-ins can be activated via WebService/Worker. |
|
||||
| Sprint 5 | Vexer Core Foundations | src/StellaOps.Vexer.WebService/TASKS.md | TODO | Team Vexer WebService | VEXER-WEB-01-001 | Scaffold minimal API host, DI, and `/vexer/status` endpoint integrating policy, storage, export, and attestation services. |
|
||||
| Sprint 6 | Vexer Ingest & Formats | src/StellaOps.Vexer.Worker/TASKS.md | TODO | Team Vexer Worker | VEXER-WORKER-01-001 | Create Worker host with provider scheduling and logging to drive recurring pulls/reconciliation. |
|
||||
| Sprint 6 | Vexer Ingest & Formats | src/StellaOps.Vexer.Formats.CSAF/TASKS.md | TODO | Team Vexer Formats | VEXER-FMT-CSAF-01-001 | Implement CSAF normalizer foundation translating provider documents into `VexClaim` entries. |
|
||||
| Sprint 6 | Vexer Ingest & Formats | src/StellaOps.Vexer.Formats.CycloneDX/TASKS.md | TODO | Team Vexer Formats | VEXER-FMT-CYCLONE-01-001 | Implement CycloneDX VEX normalizer capturing `analysis` state and component references. |
|
||||
| Sprint 6 | Vexer Ingest & Formats | src/StellaOps.Vexer.Formats.OpenVEX/TASKS.md | TODO | Team Vexer Formats | VEXER-FMT-OPENVEX-01-001 | Implement OpenVEX normalizer to ingest attestations into canonical claims with provenance. |
|
||||
| Sprint 6 | Vexer Ingest & Formats | src/StellaOps.Vexer.Connectors.RedHat.CSAF/TASKS.md | TODO | Team Vexer Connectors – Red Hat | VEXER-CONN-RH-01-001 | Ship Red Hat CSAF provider metadata discovery enabling incremental pulls. |
|
||||
| Sprint 6 | Vexer Ingest & Formats | src/StellaOps.Vexer.Connectors.Cisco.CSAF/TASKS.md | TODO | Team Vexer Connectors – Cisco | VEXER-CONN-CISCO-01-001 | Implement Cisco CSAF endpoint discovery/auth to unlock paginated pulls. |
|
||||
| Sprint 6 | Vexer Ingest & Formats | src/StellaOps.Vexer.Connectors.SUSE.RancherVEXHub/TASKS.md | TODO | Team Vexer Connectors – SUSE | VEXER-CONN-SUSE-01-001 | Build Rancher VEX Hub discovery/subscription path with offline snapshot support. |
|
||||
| Sprint 6 | Vexer Ingest & Formats | src/StellaOps.Vexer.Connectors.MSRC.CSAF/TASKS.md | TODO | Team Vexer Connectors – MSRC | VEXER-CONN-MS-01-001 | Deliver AAD onboarding/token cache for MSRC CSAF ingestion. |
|
||||
| Sprint 6 | Vexer Ingest & Formats | src/StellaOps.Vexer.Connectors.Oracle.CSAF/TASKS.md | TODO | Team Vexer Connectors – Oracle | VEXER-CONN-ORACLE-01-001 | Implement Oracle CSAF catalogue discovery with CPU calendar awareness. |
|
||||
| Sprint 6 | Vexer Ingest & Formats | src/StellaOps.Vexer.Connectors.Ubuntu.CSAF/TASKS.md | TODO | Team Vexer Connectors – Ubuntu | VEXER-CONN-UBUNTU-01-001 | Implement Ubuntu CSAF discovery and channel selection for USN ingestion. |
|
||||
| Sprint 5 | Vexer Core Foundations | src/StellaOps.Vexer.Export/TASKS.md | DONE (2025-10-17) | Team Vexer Export | VEXER-EXPORT-01-004 | Connect export engine to attestation client and persist Rekor metadata. |
|
||||
| Sprint 5 | Vexer Core Foundations | src/StellaOps.Vexer.Attestation/TASKS.md | DONE (2025-10-16) | Team Vexer Attestation | VEXER-ATTEST-01-001 | Implement in-toto predicate + DSSE builder providing envelopes for export attestation. |
|
||||
| Sprint 5 | Vexer Core Foundations | src/StellaOps.Vexer.Connectors.Abstractions/TASKS.md | DONE (2025-10-17) | Team Vexer Connectors | VEXER-CONN-ABS-01-001 | Deliver shared connector context/base classes so provider plug-ins can be activated via WebService/Worker. |
|
||||
| Sprint 5 | Vexer Core Foundations | src/StellaOps.Vexer.WebService/TASKS.md | DONE (2025-10-17) | Team Vexer WebService | VEXER-WEB-01-001 | Scaffold minimal API host, DI, and `/vexer/status` endpoint integrating policy, storage, export, and attestation services. |
|
||||
| Sprint 6 | Vexer Ingest & Formats | src/StellaOps.Vexer.Worker/TASKS.md | DONE (2025-10-17) | Team Vexer Worker | VEXER-WORKER-01-001 | Create Worker host with provider scheduling and logging to drive recurring pulls/reconciliation. |
|
||||
| Sprint 6 | Vexer Ingest & Formats | src/StellaOps.Vexer.Formats.CSAF/TASKS.md | DONE (2025-10-17) | Team Vexer Formats | VEXER-FMT-CSAF-01-001 | Implement CSAF normalizer foundation translating provider documents into `VexClaim` entries. |
|
||||
| Sprint 6 | Vexer Ingest & Formats | src/StellaOps.Vexer.Formats.CycloneDX/TASKS.md | DONE (2025-10-17) | Team Vexer Formats | VEXER-FMT-CYCLONE-01-001 | Implement CycloneDX VEX normalizer capturing `analysis` state and component references. |
|
||||
| Sprint 6 | Vexer Ingest & Formats | src/StellaOps.Vexer.Formats.OpenVEX/TASKS.md | DONE (2025-10-17) | Team Vexer Formats | VEXER-FMT-OPENVEX-01-001 | Implement OpenVEX normalizer to ingest attestations into canonical claims with provenance. |
|
||||
| Sprint 6 | Vexer Ingest & Formats | src/StellaOps.Vexer.Connectors.RedHat.CSAF/TASKS.md | DONE (2025-10-17) | Team Vexer Connectors – Red Hat | VEXER-CONN-RH-01-001 | Ship Red Hat CSAF provider metadata discovery enabling incremental pulls. |
|
||||
| Sprint 6 | Vexer Ingest & Formats | src/StellaOps.Vexer.Connectors.RedHat.CSAF/TASKS.md | DONE (2025-10-17) | Team Vexer Connectors – Red Hat | VEXER-CONN-RH-01-002 | Fetch CSAF windows with ETag handling, resume tokens, quarantine on schema errors, and persist raw docs. |
|
||||
| Sprint 6 | Vexer Ingest & Formats | src/StellaOps.Vexer.Connectors.RedHat.CSAF/TASKS.md | DONE (2025-10-17) | Team Vexer Connectors – Red Hat | VEXER-CONN-RH-01-003 | Populate provider trust overrides (cosign issuer, identity regex) and provenance hints for policy evaluation/logging. |
|
||||
| Sprint 6 | Vexer Ingest & Formats | src/StellaOps.Vexer.Connectors.RedHat.CSAF/TASKS.md | DONE (2025-10-17) | Team Vexer Connectors – Red Hat | VEXER-CONN-RH-01-004 | Persist resume cursors (last updated timestamp/document hashes) in storage and reload during fetch to avoid duplicates. |
|
||||
| Sprint 6 | Vexer Ingest & Formats | src/StellaOps.Vexer.Connectors.RedHat.CSAF/TASKS.md | DONE (2025-10-17) | Team Vexer Connectors – Red Hat | VEXER-CONN-RH-01-005 | Register connector in Worker/WebService DI, add scheduled jobs, and document CLI triggers for Red Hat CSAF pulls. |
|
||||
| Sprint 6 | Vexer Ingest & Formats | src/StellaOps.Vexer.Connectors.RedHat.CSAF/TASKS.md | DONE (2025-10-17) | Team Vexer Connectors – Red Hat | VEXER-CONN-RH-01-006 | Add CSAF normalization parity fixtures ensuring RHSA-specific metadata is preserved. |
|
||||
| Sprint 6 | Vexer Ingest & Formats | src/StellaOps.Vexer.Connectors.Cisco.CSAF/TASKS.md | DONE (2025-10-17) | Team Vexer Connectors – Cisco | VEXER-CONN-CISCO-01-001 | Implement Cisco CSAF endpoint discovery/auth to unlock paginated pulls. |
|
||||
| Sprint 6 | Vexer Ingest & Formats | src/StellaOps.Vexer.Connectors.Cisco.CSAF/TASKS.md | DONE (2025-10-17) | Team Vexer Connectors – Cisco | VEXER-CONN-CISCO-01-002 | Implement Cisco CSAF paginated fetch loop with dedupe and raw persistence support. |
|
||||
| Sprint 6 | Vexer Ingest & Formats | src/StellaOps.Vexer.Connectors.SUSE.RancherVEXHub/TASKS.md | DONE (2025-10-17) | Team Vexer Connectors – SUSE | VEXER-CONN-SUSE-01-001 | Build Rancher VEX Hub discovery/subscription path with offline snapshot support. |
|
||||
| Sprint 6 | Vexer Ingest & Formats | src/StellaOps.Vexer.Connectors.MSRC.CSAF/TASKS.md | DONE (2025-10-17) | Team Vexer Connectors – MSRC | VEXER-CONN-MS-01-001 | Deliver AAD onboarding/token cache for MSRC CSAF ingestion. |
|
||||
| Sprint 6 | Vexer Ingest & Formats | src/StellaOps.Vexer.Connectors.Oracle.CSAF/TASKS.md | DONE (2025-10-17) | Team Vexer Connectors – Oracle | VEXER-CONN-ORACLE-01-001 | Implement Oracle CSAF catalogue discovery with CPU calendar awareness. |
|
||||
| Sprint 6 | Vexer Ingest & Formats | src/StellaOps.Vexer.Connectors.Ubuntu.CSAF/TASKS.md | DONE (2025-10-17) | Team Vexer Connectors – Ubuntu | VEXER-CONN-UBUNTU-01-001 | Implement Ubuntu CSAF discovery and channel selection for USN ingestion. |
|
||||
| Sprint 6 | Vexer Ingest & Formats | src/StellaOps.Vexer.Connectors.OCI.OpenVEX.Attest/TASKS.md | TODO | Team Vexer Connectors – OCI | VEXER-CONN-OCI-01-001 | Wire OCI discovery/auth to fetch OpenVEX attestations for configured images. |
|
||||
| Sprint 6 | Vexer Ingest & Formats | src/StellaOps.Cli/TASKS.md | TODO | DevEx/CLI | VEXER-CLI-01-001 | Add `vexer` CLI verbs bridging to WebService with consistent auth and offline UX. |
|
||||
| Sprint 7 | Contextual Truth Foundations | src/StellaOps.Vexer.Core/TASKS.md | TODO | Team Vexer Core & Policy | VEXER-CORE-02-001 | Context signal schema prep – extend consensus models with severity/KEV/EPSS fields and update canonical serializers. |
|
||||
| Sprint 7 | Contextual Truth Foundations | src/StellaOps.Vexer.Policy/TASKS.md | TODO | Team Vexer Policy | VEXER-POLICY-02-001 | Scoring coefficients & weight ceilings – add α/β options, weight boosts, and validation guidance. |
|
||||
| Sprint 7 | Contextual Truth Foundations | src/StellaOps.Vexer.Storage.Mongo/TASKS.md | TODO | Team Vexer Storage | VEXER-STORAGE-02-001 | Statement events & scoring signals – create immutable VEX statement store plus consensus extensions with indexes/migrations. |
|
||||
| Sprint 7 | Contextual Truth Foundations | src/StellaOps.Vexer.WebService/TASKS.md | TODO | Team Vexer WebService | VEXER-WEB-01-004 | Resolve API & signed responses – expose `/vexer/resolve`, return signed consensus/score envelopes, document auth. |
|
||||
| Sprint 7 | Contextual Truth Foundations | src/StellaOps.Vexer.Attestation/TASKS.md | TODO | Team Vexer Attestation | VEXER-ATTEST-01-002 | Rekor v2 client integration – ship transparency log client with retries and offline queue. |
|
||||
| Sprint 7 | Contextual Truth Foundations | src/StellaOps.Vexer.Attestation/TASKS.md | DONE (2025-10-16) | Team Vexer Attestation | VEXER-ATTEST-01-002 | Rekor v2 client integration – ship transparency log client with retries and offline queue. |
|
||||
| Sprint 7 | Contextual Truth Foundations | src/StellaOps.Vexer.Worker/TASKS.md | TODO | Team Vexer Worker | VEXER-WORKER-01-004 | TTL refresh & stability damper – schedule re-resolve loops and guard against status flapping. |
|
||||
| Sprint 7 | Contextual Truth Foundations | src/StellaOps.Vexer.Export/TASKS.md | TODO | Team Vexer Export | VEXER-EXPORT-01-005 | Score & resolve envelope surfaces – include signed consensus/score artifacts in exports. |
|
||||
| Sprint 7 | Contextual Truth Foundations | src/StellaOps.Feedser.Core/TASKS.md | TODO | Team Core Engine & Storage Analytics | FEEDCORE-ENGINE-07-001 | Advisory event log & asOf queries – surface immutable statements and replay capability. |
|
||||
|
||||
52
seed-data/cert-bund/README.md
Normal file
52
seed-data/cert-bund/README.md
Normal file
@@ -0,0 +1,52 @@
|
||||
# CERT-Bund Offline Kit Seed Data
|
||||
|
||||
This directory stores **offline snapshots** for the CERT-Bund connector.
|
||||
The artefacts mirror the public JSON search and export endpoints so
|
||||
air‑gapped deployments can hydrate the connector without contacting the
|
||||
portal.
|
||||
|
||||
> ⚠️ **Distribution notice** – CERT-Bund advisories are published by BSI
|
||||
> (Federal Office for Information Security, Germany). Review the portal
|
||||
> terms of use before redistributing the snapshots. Always keep the JSON
|
||||
> payloads and accompanying SHA-256 sums together.
|
||||
|
||||
## Recommended layout
|
||||
|
||||
```
|
||||
seed-data/cert-bund/
|
||||
├── search/ # paginated search JSON files
|
||||
│ ├── certbund-search-page-00.json
|
||||
│ └── …
|
||||
├── export/ # yearly export JSON files
|
||||
│ ├── certbund-export-2014.json
|
||||
│ └── …
|
||||
├── manifest/
|
||||
│ └── certbund-offline-manifest.json
|
||||
└── certbund-offline-manifest.sha256
|
||||
```
|
||||
|
||||
Use `certbund-offline-manifest.json` to feed the Offline Kit build: every
|
||||
entry contains `source`, `from`, `to`, `sha256`, `capturedAt`, and the
|
||||
relative file path. The manifest is deterministic when regenerated with
|
||||
the tooling described below.
|
||||
|
||||
## Tooling
|
||||
|
||||
Run the helper under `tools/` to capture fresh snapshots or regenerate
|
||||
the manifest:
|
||||
|
||||
```
|
||||
python tools/certbund_offline_snapshot.py --output seed-data/cert-bund
|
||||
```
|
||||
|
||||
See the connector operations guide
|
||||
(`docs/ops/feedser-certbund-operations.md`) for detailed usage,
|
||||
including how to provide cookies/tokens when the portal requires manual
|
||||
authentication.
|
||||
|
||||
## Git hygiene
|
||||
|
||||
- JSON payloads and checksums are **ignored by Git**. Generate them
|
||||
locally when preparing an Offline Kit bundle.
|
||||
- Commit documentation, scripts, and manifest templates only – never the
|
||||
exported advisory data itself.
|
||||
@@ -9,4 +9,4 @@
|
||||
|FEEDCONN-CERTBUND-02-006 Telemetry & documentation|DevEx|Docs|**DONE (2025-10-15)** – Added `CertBundDiagnostics` (meter `StellaOps.Feedser.Source.CertBund`) with fetch/parse/map counters + histograms, recorded coverage days, wired stage summary logs, and published the ops runbook (`docs/ops/feedser-certbund-operations.md`).|
|
||||
|FEEDCONN-CERTBUND-02-007 Feed history & locale assessment|BE-Conn-CERTBUND|Research|**DONE (2025-10-15)** – Measured RSS retention (~6 days/≈250 items), captured connector-driven backfill guidance in the runbook, and aligned locale guidance (preserve `language=de`, Docs glossary follow-up). **Next:** coordinate with Tools to land the state-seeding helper so scripted backfills replace manual Mongo tweaks.|
|
||||
|FEEDCONN-CERTBUND-02-008 Session bootstrap & cookie strategy|BE-Conn-CERTBUND|Source.Common|**DONE (2025-10-14)** – Feed client primes the portal session (cookie container via `SocketsHttpHandler`), shares cookies across detail requests, and documents bootstrap behaviour in options (`PortalBootstrapUri`).|
|
||||
|FEEDCONN-CERTBUND-02-009 Offline Kit export packaging|BE-Conn-CERTBUND, Docs|Offline Kit|**TODO** – Capture JSON search/export snapshots (per-year splits), generate manifest fields (`source`,`from`,`to`,`sha256`,`capturedAt`), and update Offline Kit docs so air-gapped deployments can seed historical CERT-Bund advisories without live fetching. **Remark:** follow the interim workflow documented in `docs/ops/feedser-certbund-operations.md` §3.3 until the packaged artefacts ship.|
|
||||
|FEEDCONN-CERTBUND-02-009 Offline Kit export packaging|BE-Conn-CERTBUND, Docs|Offline Kit|**DONE (2025-10-17)** – Added `tools/certbund_offline_snapshot.py` to capture search/export JSON, emit deterministic manifests + SHA files, and refreshed docs (`docs/ops/feedser-certbund-operations.md`, `docs/24_OFFLINE_KIT.md`) with offline-kit instructions and manifest layout guidance. Seed data README/ignore rules cover local snapshot hygiene.|
|
||||
|
||||
@@ -5,12 +5,12 @@ Defines shared connector infrastructure for Vexer, including base contexts, resu
|
||||
- `IVexConnector` context implementation, raw store helpers, verification hooks, and telemetry utilities.
|
||||
- Configuration primitives (YAML parsing, secrets handling guidelines) and options validation.
|
||||
- Connector lifecycle helpers for retries, paging, `.well-known` discovery, and resume markers.
|
||||
- Documentation for connector packaging, plugin manifest metadata, and DI registration.
|
||||
- Documentation for connector packaging, plugin manifest metadata, and DI registration (see `docs/dev/30_VEXER_CONNECTOR_GUIDE.md` and `docs/dev/templates/vexer-connector/`).
|
||||
## Participants
|
||||
- All Vexer connector projects reference this module to obtain base classes and context services.
|
||||
- WebService/Worker instantiate connectors via plugin loader leveraging abstractions defined here.
|
||||
## Interfaces & contracts
|
||||
- Connector context, result, and telemetry interfaces; `ConnectorDescriptor`, `ConnectorOptions`, authentication helpers.
|
||||
- Connector context, result, and telemetry interfaces; `VexConnectorDescriptor`, `VexConnectorBase`, options binder/validators, authentication helpers.
|
||||
- Utility classes for HTTP clients, throttling, and deterministic logging.
|
||||
## In/Out of scope
|
||||
In: shared abstractions, helper utilities, configuration binding, documentation for connector authors.
|
||||
|
||||
@@ -0,0 +1,12 @@
|
||||
using System.Collections.Generic;
|
||||
|
||||
namespace StellaOps.Vexer.Connectors.Abstractions;
|
||||
|
||||
/// <summary>
|
||||
/// Custom validator hook executed after connector options are bound.
|
||||
/// </summary>
|
||||
/// <typeparam name="TOptions">Connector-specific options type.</typeparam>
|
||||
public interface IVexConnectorOptionsValidator<in TOptions>
|
||||
{
|
||||
void Validate(VexConnectorDescriptor descriptor, TOptions options, IList<string> errors);
|
||||
}
|
||||
@@ -0,0 +1,17 @@
|
||||
<Project Sdk="Microsoft.NET.Sdk">
|
||||
<PropertyGroup>
|
||||
<TargetFramework>net10.0</TargetFramework>
|
||||
<LangVersion>preview</LangVersion>
|
||||
<Nullable>enable</Nullable>
|
||||
<ImplicitUsings>enable</ImplicitUsings>
|
||||
<TreatWarningsAsErrors>true</TreatWarningsAsErrors>
|
||||
</PropertyGroup>
|
||||
<ItemGroup>
|
||||
<ProjectReference Include="..\StellaOps.Vexer.Core\StellaOps.Vexer.Core.csproj" />
|
||||
</ItemGroup>
|
||||
<ItemGroup>
|
||||
<PackageReference Include="Microsoft.Extensions.Configuration" Version="8.0.0" />
|
||||
<PackageReference Include="Microsoft.Extensions.Configuration.Binder" Version="8.0.1" />
|
||||
<PackageReference Include="Microsoft.Extensions.Logging.Abstractions" Version="8.0.0" />
|
||||
</ItemGroup>
|
||||
</Project>
|
||||
@@ -2,6 +2,6 @@ If you are working on this file you need to read docs/ARCHITECTURE_VEXER.md and
|
||||
# TASKS
|
||||
| Task | Owner(s) | Depends on | Notes |
|
||||
|---|---|---|---|
|
||||
|VEXER-CONN-ABS-01-001 – Connector context & base classes|Team Vexer Connectors|VEXER-CORE-01-003|TODO – Implement `VexConnectorContext`, result types, helper base classes, and deterministic logging helpers for connectors.|
|
||||
|VEXER-CONN-ABS-01-002 – YAML options & validation|Team Vexer Connectors|VEXER-CONN-ABS-01-001|TODO – Provide strongly-typed options binding/validation for connector YAML definitions with offline-safe defaults.|
|
||||
|VEXER-CONN-ABS-01-003 – Plugin packaging & docs|Team Vexer Connectors|VEXER-CONN-ABS-01-001|TODO – Document connector packaging (NuGet manifest, plugin loader metadata) and supply reference templates for downstream connector modules.|
|
||||
|VEXER-CONN-ABS-01-001 – Connector context & base classes|Team Vexer Connectors|VEXER-CORE-01-003|**DONE (2025-10-17)** – Added `StellaOps.Vexer.Connectors.Abstractions` project with `VexConnectorBase`, deterministic logging scopes, metadata builder helpers, and connector descriptors; docs updated to highlight the shared abstractions.|
|
||||
|VEXER-CONN-ABS-01-002 – YAML options & validation|Team Vexer Connectors|VEXER-CONN-ABS-01-001|**DONE (2025-10-17)** – Delivered `VexConnectorOptionsBinder` + binder options/validators, environment-variable expansion, data-annotation checks, and custom validation hooks with documentation updates covering the workflow.|
|
||||
|VEXER-CONN-ABS-01-003 – Plugin packaging & docs|Team Vexer Connectors|VEXER-CONN-ABS-01-001|**DONE (2025-10-17)** – Authored `docs/dev/30_VEXER_CONNECTOR_GUIDE.md`, added quick-start template under `docs/dev/templates/vexer-connector/`, and updated module docs to reference the packaging workflow.|
|
||||
|
||||
@@ -0,0 +1,99 @@
|
||||
using System.Collections.Immutable;
|
||||
using System.Security.Cryptography;
|
||||
using Microsoft.Extensions.Logging;
|
||||
using StellaOps.Vexer.Core;
|
||||
|
||||
namespace StellaOps.Vexer.Connectors.Abstractions;
|
||||
|
||||
/// <summary>
|
||||
/// Convenience base class for implementing <see cref="IVexConnector" />.
|
||||
/// </summary>
|
||||
public abstract class VexConnectorBase : IVexConnector
|
||||
{
|
||||
protected VexConnectorBase(VexConnectorDescriptor descriptor, ILogger logger, TimeProvider? timeProvider = null)
|
||||
{
|
||||
Descriptor = descriptor ?? throw new ArgumentNullException(nameof(descriptor));
|
||||
Logger = logger ?? throw new ArgumentNullException(nameof(logger));
|
||||
TimeProvider = timeProvider ?? TimeProvider.System;
|
||||
}
|
||||
|
||||
/// <inheritdoc />
|
||||
public string Id => Descriptor.Id;
|
||||
|
||||
/// <inheritdoc />
|
||||
public VexProviderKind Kind => Descriptor.Kind;
|
||||
|
||||
protected VexConnectorDescriptor Descriptor { get; }
|
||||
|
||||
protected ILogger Logger { get; }
|
||||
|
||||
protected TimeProvider TimeProvider { get; }
|
||||
|
||||
protected DateTimeOffset UtcNow() => TimeProvider.GetUtcNow();
|
||||
|
||||
protected VexRawDocument CreateRawDocument(
|
||||
VexDocumentFormat format,
|
||||
Uri sourceUri,
|
||||
ReadOnlyMemory<byte> content,
|
||||
ImmutableDictionary<string, string>? metadata = null)
|
||||
{
|
||||
if (sourceUri is null)
|
||||
{
|
||||
throw new ArgumentNullException(nameof(sourceUri));
|
||||
}
|
||||
|
||||
var digest = ComputeSha256(content.Span);
|
||||
var captured = TimeProvider.GetUtcNow();
|
||||
return new VexRawDocument(
|
||||
Descriptor.Id,
|
||||
format,
|
||||
sourceUri,
|
||||
captured,
|
||||
digest,
|
||||
content,
|
||||
metadata ?? ImmutableDictionary<string, string>.Empty);
|
||||
}
|
||||
|
||||
protected IDisposable BeginConnectorScope(string operation, IReadOnlyDictionary<string, object?>? metadata = null)
|
||||
=> VexConnectorLogScope.Begin(Logger, Descriptor, operation, metadata);
|
||||
|
||||
protected void LogConnectorEvent(LogLevel level, string eventName, string message, IReadOnlyDictionary<string, object?>? metadata = null, Exception? exception = null)
|
||||
{
|
||||
using var scope = BeginConnectorScope(eventName, metadata);
|
||||
if (exception is null)
|
||||
{
|
||||
Logger.Log(level, "{Message}", message);
|
||||
}
|
||||
else
|
||||
{
|
||||
Logger.Log(level, exception, "{Message}", message);
|
||||
}
|
||||
}
|
||||
|
||||
protected ImmutableDictionary<string, string> BuildMetadata(Action<VexConnectorMetadataBuilder> configure)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(configure);
|
||||
var builder = new VexConnectorMetadataBuilder();
|
||||
configure(builder);
|
||||
return builder.Build();
|
||||
}
|
||||
|
||||
private static string ComputeSha256(ReadOnlySpan<byte> content)
|
||||
{
|
||||
Span<byte> buffer = stackalloc byte[32];
|
||||
if (SHA256.TryHashData(content, buffer, out _))
|
||||
{
|
||||
return "sha256:" + Convert.ToHexString(buffer).ToLowerInvariant();
|
||||
}
|
||||
|
||||
using var sha = SHA256.Create();
|
||||
var hash = sha.ComputeHash(content.ToArray());
|
||||
return "sha256:" + Convert.ToHexString(hash).ToLowerInvariant();
|
||||
}
|
||||
|
||||
public abstract ValueTask ValidateAsync(VexConnectorSettings settings, CancellationToken cancellationToken);
|
||||
|
||||
public abstract IAsyncEnumerable<VexRawDocument> FetchAsync(VexConnectorContext context, CancellationToken cancellationToken);
|
||||
|
||||
public abstract ValueTask<VexClaimBatch> NormalizeAsync(VexRawDocument document, CancellationToken cancellationToken);
|
||||
}
|
||||
@@ -0,0 +1,54 @@
|
||||
using System.Collections.Immutable;
|
||||
using StellaOps.Vexer.Core;
|
||||
|
||||
namespace StellaOps.Vexer.Connectors.Abstractions;
|
||||
|
||||
/// <summary>
|
||||
/// Static descriptor for a Vexer connector plug-in.
|
||||
/// </summary>
|
||||
public sealed record VexConnectorDescriptor
|
||||
{
|
||||
public VexConnectorDescriptor(string id, VexProviderKind kind, string displayName)
|
||||
{
|
||||
if (string.IsNullOrWhiteSpace(id))
|
||||
{
|
||||
throw new ArgumentException("Connector id must be provided.", nameof(id));
|
||||
}
|
||||
|
||||
Id = id;
|
||||
Kind = kind;
|
||||
DisplayName = string.IsNullOrWhiteSpace(displayName) ? id : displayName;
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Stable connector identifier (matches provider id).
|
||||
/// </summary>
|
||||
public string Id { get; }
|
||||
|
||||
/// <summary>
|
||||
/// Provider kind served by the connector.
|
||||
/// </summary>
|
||||
public VexProviderKind Kind { get; }
|
||||
|
||||
/// <summary>
|
||||
/// Human friendly name used in logs/diagnostics.
|
||||
/// </summary>
|
||||
public string DisplayName { get; }
|
||||
|
||||
/// <summary>
|
||||
/// Optional friendly description.
|
||||
/// </summary>
|
||||
public string? Description { get; init; }
|
||||
|
||||
/// <summary>
|
||||
/// Document formats the connector is expected to emit.
|
||||
/// </summary>
|
||||
public ImmutableArray<VexDocumentFormat> SupportedFormats { get; init; } = ImmutableArray<VexDocumentFormat>.Empty;
|
||||
|
||||
/// <summary>
|
||||
/// Optional tags surfaced in diagnostics (e.g. "beta", "offline").
|
||||
/// </summary>
|
||||
public ImmutableArray<string> Tags { get; init; } = ImmutableArray<string>.Empty;
|
||||
|
||||
public override string ToString() => $"{Id} ({Kind})";
|
||||
}
|
||||
@@ -0,0 +1,50 @@
|
||||
using System.Linq;
|
||||
using Microsoft.Extensions.Logging;
|
||||
using StellaOps.Vexer.Core;
|
||||
|
||||
namespace StellaOps.Vexer.Connectors.Abstractions;
|
||||
|
||||
/// <summary>
|
||||
/// Helper to establish deterministic logging scopes for connector operations.
|
||||
/// </summary>
|
||||
public static class VexConnectorLogScope
|
||||
{
|
||||
public static IDisposable Begin(ILogger logger, VexConnectorDescriptor descriptor, string operation, IReadOnlyDictionary<string, object?>? metadata = null)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(logger);
|
||||
ArgumentNullException.ThrowIfNull(descriptor);
|
||||
ArgumentException.ThrowIfNullOrEmpty(operation);
|
||||
|
||||
var scopeValues = new List<KeyValuePair<string, object?>>
|
||||
{
|
||||
new("vex.connector.id", descriptor.Id),
|
||||
new("vex.connector.kind", descriptor.Kind.ToString()),
|
||||
new("vex.connector.operation", operation),
|
||||
};
|
||||
|
||||
if (!string.Equals(descriptor.DisplayName, descriptor.Id, StringComparison.Ordinal))
|
||||
{
|
||||
scopeValues.Add(new KeyValuePair<string, object?>("vex.connector.displayName", descriptor.DisplayName));
|
||||
}
|
||||
|
||||
if (!string.IsNullOrWhiteSpace(descriptor.Description))
|
||||
{
|
||||
scopeValues.Add(new KeyValuePair<string, object?>("vex.connector.description", descriptor.Description));
|
||||
}
|
||||
|
||||
if (!descriptor.Tags.IsDefaultOrEmpty)
|
||||
{
|
||||
scopeValues.Add(new KeyValuePair<string, object?>("vex.connector.tags", string.Join(",", descriptor.Tags)));
|
||||
}
|
||||
|
||||
if (metadata is not null)
|
||||
{
|
||||
foreach (var kvp in metadata.OrderBy(static pair => pair.Key, StringComparer.Ordinal))
|
||||
{
|
||||
scopeValues.Add(new KeyValuePair<string, object?>($"vex.{kvp.Key}", kvp.Value));
|
||||
}
|
||||
}
|
||||
|
||||
return logger.BeginScope(scopeValues)!;
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,37 @@
|
||||
using System.Collections.Immutable;
|
||||
|
||||
namespace StellaOps.Vexer.Connectors.Abstractions;
|
||||
|
||||
/// <summary>
|
||||
/// Builds deterministic metadata dictionaries for raw documents and logging scopes.
|
||||
/// </summary>
|
||||
public sealed class VexConnectorMetadataBuilder
|
||||
{
|
||||
private readonly SortedDictionary<string, string> _values = new(StringComparer.Ordinal);
|
||||
|
||||
public VexConnectorMetadataBuilder Add(string key, string? value)
|
||||
{
|
||||
if (!string.IsNullOrWhiteSpace(key) && !string.IsNullOrWhiteSpace(value))
|
||||
{
|
||||
_values[key] = value!;
|
||||
}
|
||||
|
||||
return this;
|
||||
}
|
||||
|
||||
public VexConnectorMetadataBuilder Add(string key, DateTimeOffset value)
|
||||
=> Add(key, value.ToUniversalTime().ToString("O"));
|
||||
|
||||
public VexConnectorMetadataBuilder AddRange(IEnumerable<KeyValuePair<string, string?>> items)
|
||||
{
|
||||
foreach (var item in items)
|
||||
{
|
||||
Add(item.Key, item.Value);
|
||||
}
|
||||
|
||||
return this;
|
||||
}
|
||||
|
||||
public ImmutableDictionary<string, string> Build()
|
||||
=> _values.ToImmutableDictionary(static pair => pair.Key, static pair => pair.Value, StringComparer.Ordinal);
|
||||
}
|
||||
@@ -0,0 +1,157 @@
|
||||
using System.Collections.Immutable;
|
||||
using System.ComponentModel.DataAnnotations;
|
||||
using System.Linq;
|
||||
using Microsoft.Extensions.Configuration;
|
||||
using StellaOps.Vexer.Core;
|
||||
|
||||
namespace StellaOps.Vexer.Connectors.Abstractions;
|
||||
|
||||
/// <summary>
|
||||
/// Provides strongly typed binding and validation for connector options.
|
||||
/// </summary>
|
||||
public static class VexConnectorOptionsBinder
|
||||
{
|
||||
public static TOptions Bind<TOptions>(
|
||||
VexConnectorDescriptor descriptor,
|
||||
VexConnectorSettings settings,
|
||||
VexConnectorOptionsBinderOptions? options = null,
|
||||
IEnumerable<IVexConnectorOptionsValidator<TOptions>>? validators = null)
|
||||
where TOptions : class, new()
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(descriptor);
|
||||
ArgumentNullException.ThrowIfNull(settings);
|
||||
|
||||
var binderSettings = options ?? new VexConnectorOptionsBinderOptions();
|
||||
var transformed = TransformValues(settings, binderSettings);
|
||||
|
||||
var configuration = BuildConfiguration(transformed);
|
||||
|
||||
var result = new TOptions();
|
||||
var errors = new List<string>();
|
||||
|
||||
try
|
||||
{
|
||||
configuration.Bind(
|
||||
result,
|
||||
binderOptions => binderOptions.ErrorOnUnknownConfiguration = !binderSettings.AllowUnknownKeys);
|
||||
}
|
||||
catch (InvalidOperationException ex) when (!binderSettings.AllowUnknownKeys)
|
||||
{
|
||||
errors.Add(ex.Message);
|
||||
}
|
||||
|
||||
binderSettings.PostConfigure?.Invoke(result);
|
||||
|
||||
if (binderSettings.ValidateDataAnnotations)
|
||||
{
|
||||
ValidateDataAnnotations(result, errors);
|
||||
}
|
||||
|
||||
if (validators is not null)
|
||||
{
|
||||
foreach (var validator in validators)
|
||||
{
|
||||
validator?.Validate(descriptor, result, errors);
|
||||
}
|
||||
}
|
||||
|
||||
if (errors.Count > 0)
|
||||
{
|
||||
throw new VexConnectorOptionsValidationException(descriptor.Id, errors);
|
||||
}
|
||||
|
||||
return result;
|
||||
}
|
||||
|
||||
private static ImmutableDictionary<string, string?> TransformValues(
|
||||
VexConnectorSettings settings,
|
||||
VexConnectorOptionsBinderOptions binderOptions)
|
||||
{
|
||||
var builder = ImmutableDictionary.CreateBuilder<string, string?>(StringComparer.OrdinalIgnoreCase);
|
||||
foreach (var kvp in settings.Values)
|
||||
{
|
||||
var value = kvp.Value;
|
||||
if (binderOptions.TrimWhitespace && value is not null)
|
||||
{
|
||||
value = value.Trim();
|
||||
}
|
||||
|
||||
if (binderOptions.TreatEmptyAsNull && string.IsNullOrEmpty(value))
|
||||
{
|
||||
value = null;
|
||||
}
|
||||
|
||||
if (value is not null && binderOptions.ExpandEnvironmentVariables)
|
||||
{
|
||||
value = Environment.ExpandEnvironmentVariables(value);
|
||||
}
|
||||
|
||||
if (binderOptions.ValueTransformer is not null)
|
||||
{
|
||||
value = binderOptions.ValueTransformer.Invoke(kvp.Key, value);
|
||||
}
|
||||
|
||||
builder[kvp.Key] = value;
|
||||
}
|
||||
|
||||
return builder.ToImmutable();
|
||||
}
|
||||
|
||||
private static IConfiguration BuildConfiguration(ImmutableDictionary<string, string?> values)
|
||||
{
|
||||
var sources = new List<KeyValuePair<string, string?>>();
|
||||
foreach (var kvp in values)
|
||||
{
|
||||
if (kvp.Value is not null)
|
||||
{
|
||||
sources.Add(new KeyValuePair<string, string?>(kvp.Key, kvp.Value));
|
||||
}
|
||||
}
|
||||
|
||||
var configurationBuilder = new ConfigurationBuilder();
|
||||
configurationBuilder.Add(new DictionaryConfigurationSource(sources));
|
||||
return configurationBuilder.Build();
|
||||
}
|
||||
|
||||
private static void ValidateDataAnnotations<TOptions>(TOptions options, IList<string> errors)
|
||||
{
|
||||
var validationResults = new List<ValidationResult>();
|
||||
var validationContext = new ValidationContext(options!);
|
||||
if (!Validator.TryValidateObject(options!, validationContext, validationResults, validateAllProperties: true))
|
||||
{
|
||||
foreach (var validationResult in validationResults)
|
||||
{
|
||||
if (!string.IsNullOrWhiteSpace(validationResult.ErrorMessage))
|
||||
{
|
||||
errors.Add(validationResult.ErrorMessage);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
private sealed class DictionaryConfigurationSource : IConfigurationSource
|
||||
{
|
||||
private readonly IReadOnlyList<KeyValuePair<string, string?>> _data;
|
||||
|
||||
public DictionaryConfigurationSource(IEnumerable<KeyValuePair<string, string?>> data)
|
||||
{
|
||||
_data = data?.ToList() ?? new List<KeyValuePair<string, string?>>();
|
||||
}
|
||||
|
||||
public IConfigurationProvider Build(IConfigurationBuilder builder) => new DictionaryConfigurationProvider(_data);
|
||||
}
|
||||
|
||||
private sealed class DictionaryConfigurationProvider : ConfigurationProvider
|
||||
{
|
||||
public DictionaryConfigurationProvider(IEnumerable<KeyValuePair<string, string?>> data)
|
||||
{
|
||||
foreach (var pair in data)
|
||||
{
|
||||
if (pair.Value is not null)
|
||||
{
|
||||
Data[pair.Key] = pair.Value;
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,45 @@
|
||||
namespace StellaOps.Vexer.Connectors.Abstractions;
|
||||
|
||||
/// <summary>
|
||||
/// Customisation options for connector options binding.
|
||||
/// </summary>
|
||||
public sealed class VexConnectorOptionsBinderOptions
|
||||
{
|
||||
/// <summary>
|
||||
/// Indicates whether environment variables should be expanded in option values.
|
||||
/// Defaults to <c>true</c>.
|
||||
/// </summary>
|
||||
public bool ExpandEnvironmentVariables { get; set; } = true;
|
||||
|
||||
/// <summary>
|
||||
/// When <c>true</c> the binder trims whitespace around option values.
|
||||
/// </summary>
|
||||
public bool TrimWhitespace { get; set; } = true;
|
||||
|
||||
/// <summary>
|
||||
/// Converts empty strings to <c>null</c> before binding. Default: <c>true</c>.
|
||||
/// </summary>
|
||||
public bool TreatEmptyAsNull { get; set; } = true;
|
||||
|
||||
/// <summary>
|
||||
/// When <c>false</c>, binding fails if unknown configuration keys are provided.
|
||||
/// Default: <c>true</c> (permitting unknown keys).
|
||||
/// </summary>
|
||||
public bool AllowUnknownKeys { get; set; } = true;
|
||||
|
||||
/// <summary>
|
||||
/// Enables <see cref="System.ComponentModel.DataAnnotations"/> validation after binding.
|
||||
/// Default: <c>true</c>.
|
||||
/// </summary>
|
||||
public bool ValidateDataAnnotations { get; set; } = true;
|
||||
|
||||
/// <summary>
|
||||
/// Optional post-configuration callback executed after binding.
|
||||
/// </summary>
|
||||
public Action<object>? PostConfigure { get; set; }
|
||||
|
||||
/// <summary>
|
||||
/// Optional hook to transform raw configuration values before binding.
|
||||
/// </summary>
|
||||
public Func<string, string?, string?>? ValueTransformer { get; set; }
|
||||
}
|
||||
@@ -0,0 +1,36 @@
|
||||
using System.Collections.Immutable;
|
||||
|
||||
namespace StellaOps.Vexer.Connectors.Abstractions;
|
||||
|
||||
public sealed class VexConnectorOptionsValidationException : Exception
|
||||
{
|
||||
public VexConnectorOptionsValidationException(
|
||||
string connectorId,
|
||||
IEnumerable<string> errors)
|
||||
: base(BuildMessage(connectorId, errors))
|
||||
{
|
||||
ConnectorId = connectorId;
|
||||
Errors = errors?.ToImmutableArray() ?? ImmutableArray<string>.Empty;
|
||||
}
|
||||
|
||||
public string ConnectorId { get; }
|
||||
|
||||
public ImmutableArray<string> Errors { get; }
|
||||
|
||||
private static string BuildMessage(string connectorId, IEnumerable<string> errors)
|
||||
{
|
||||
var builder = new System.Text.StringBuilder();
|
||||
builder.Append("Connector options validation failed for '");
|
||||
builder.Append(connectorId);
|
||||
builder.Append("'.");
|
||||
|
||||
var list = errors?.ToImmutableArray() ?? ImmutableArray<string>.Empty;
|
||||
if (!list.IsDefaultOrEmpty)
|
||||
{
|
||||
builder.Append(" Errors: ");
|
||||
builder.Append(string.Join("; ", list));
|
||||
}
|
||||
|
||||
return builder.ToString();
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,214 @@
|
||||
using System.Collections.Generic;
|
||||
using System.Net;
|
||||
using System.Net.Http;
|
||||
using System.Text;
|
||||
using FluentAssertions;
|
||||
using Microsoft.Extensions.Caching.Memory;
|
||||
using Microsoft.Extensions.Logging.Abstractions;
|
||||
using Microsoft.Extensions.Options;
|
||||
using Microsoft.Extensions.DependencyInjection;
|
||||
using StellaOps.Vexer.Connectors.Abstractions;
|
||||
using StellaOps.Vexer.Connectors.Cisco.CSAF;
|
||||
using StellaOps.Vexer.Connectors.Cisco.CSAF.Configuration;
|
||||
using StellaOps.Vexer.Connectors.Cisco.CSAF.Metadata;
|
||||
using StellaOps.Vexer.Core;
|
||||
using StellaOps.Vexer.Storage.Mongo;
|
||||
using System.Collections.Immutable;
|
||||
using System.IO.Abstractions.TestingHelpers;
|
||||
using Xunit;
|
||||
using System.Threading;
|
||||
|
||||
namespace StellaOps.Vexer.Connectors.Cisco.CSAF.Tests.Connectors;
|
||||
|
||||
public sealed class CiscoCsafConnectorTests
|
||||
{
|
||||
[Fact]
|
||||
public async Task FetchAsync_NewAdvisory_StoresDocumentAndUpdatesState()
|
||||
{
|
||||
var responses = new Dictionary<Uri, Queue<HttpResponseMessage>>
|
||||
{
|
||||
[new Uri("https://api.cisco.test/.well-known/csaf/provider-metadata.json")] = QueueResponses("""
|
||||
{
|
||||
"metadata": {
|
||||
"publisher": {
|
||||
"name": "Cisco",
|
||||
"category": "vendor",
|
||||
"contact_details": { "id": "vexer:cisco" }
|
||||
}
|
||||
},
|
||||
"distributions": {
|
||||
"directories": [ "https://api.cisco.test/csaf/" ]
|
||||
}
|
||||
}
|
||||
"""),
|
||||
[new Uri("https://api.cisco.test/csaf/index.json")] = QueueResponses("""
|
||||
{
|
||||
"advisories": [
|
||||
{
|
||||
"id": "cisco-sa-2025",
|
||||
"url": "https://api.cisco.test/csaf/cisco-sa-2025.json",
|
||||
"published": "2025-10-01T00:00:00Z",
|
||||
"lastModified": "2025-10-02T00:00:00Z",
|
||||
"sha256": "cafebabe"
|
||||
}
|
||||
]
|
||||
}
|
||||
"""),
|
||||
[new Uri("https://api.cisco.test/csaf/cisco-sa-2025.json")] = QueueResponses("{ \"document\": \"payload\" }")
|
||||
};
|
||||
|
||||
var handler = new RoutingHttpMessageHandler(responses);
|
||||
var httpClient = new HttpClient(handler);
|
||||
var factory = new SingleHttpClientFactory(httpClient);
|
||||
var metadataLoader = new CiscoProviderMetadataLoader(
|
||||
factory,
|
||||
new MemoryCache(new MemoryCacheOptions()),
|
||||
Options.Create(new CiscoConnectorOptions
|
||||
{
|
||||
MetadataUri = "https://api.cisco.test/.well-known/csaf/provider-metadata.json",
|
||||
PersistOfflineSnapshot = false,
|
||||
}),
|
||||
NullLogger<CiscoProviderMetadataLoader>.Instance,
|
||||
new MockFileSystem());
|
||||
|
||||
var stateRepository = new InMemoryConnectorStateRepository();
|
||||
var connector = new CiscoCsafConnector(
|
||||
metadataLoader,
|
||||
factory,
|
||||
stateRepository,
|
||||
new[] { new CiscoConnectorOptionsValidator() },
|
||||
NullLogger<CiscoCsafConnector>.Instance,
|
||||
TimeProvider.System);
|
||||
|
||||
var settings = new VexConnectorSettings(ImmutableDictionary<string, string>.Empty);
|
||||
await connector.ValidateAsync(settings, CancellationToken.None);
|
||||
|
||||
var sink = new InMemoryRawSink();
|
||||
var context = new VexConnectorContext(null, VexConnectorSettings.Empty, sink, new NoopSignatureVerifier(), new NoopNormalizerRouter(), new ServiceCollection().BuildServiceProvider());
|
||||
|
||||
var documents = new List<VexRawDocument>();
|
||||
await foreach (var doc in connector.FetchAsync(context, CancellationToken.None))
|
||||
{
|
||||
documents.Add(doc);
|
||||
}
|
||||
|
||||
documents.Should().HaveCount(1);
|
||||
sink.Documents.Should().HaveCount(1);
|
||||
stateRepository.CurrentState.Should().NotBeNull();
|
||||
stateRepository.CurrentState!.DocumentDigests.Should().HaveCount(1);
|
||||
|
||||
// second run should not refetch documents
|
||||
sink.Documents.Clear();
|
||||
documents.Clear();
|
||||
|
||||
await foreach (var doc in connector.FetchAsync(context, CancellationToken.None))
|
||||
{
|
||||
documents.Add(doc);
|
||||
}
|
||||
|
||||
documents.Should().BeEmpty();
|
||||
sink.Documents.Should().BeEmpty();
|
||||
}
|
||||
|
||||
private static Queue<HttpResponseMessage> QueueResponses(string payload)
|
||||
=> new(new[]
|
||||
{
|
||||
new HttpResponseMessage(HttpStatusCode.OK)
|
||||
{
|
||||
Content = new StringContent(payload, Encoding.UTF8, "application/json"),
|
||||
}
|
||||
});
|
||||
|
||||
private sealed class RoutingHttpMessageHandler : HttpMessageHandler
|
||||
{
|
||||
private readonly Dictionary<Uri, Queue<HttpResponseMessage>> _responses;
|
||||
|
||||
public RoutingHttpMessageHandler(Dictionary<Uri, Queue<HttpResponseMessage>> responses)
|
||||
{
|
||||
_responses = responses;
|
||||
}
|
||||
|
||||
protected override Task<HttpResponseMessage> SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
|
||||
{
|
||||
if (request.RequestUri is not null && _responses.TryGetValue(request.RequestUri, out var queue) && queue.Count > 0)
|
||||
{
|
||||
var response = queue.Peek();
|
||||
return Task.FromResult(response.Clone());
|
||||
}
|
||||
|
||||
return Task.FromResult(new HttpResponseMessage(HttpStatusCode.NotFound)
|
||||
{
|
||||
Content = new StringContent($"No response configured for {request.RequestUri}"),
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
private sealed class SingleHttpClientFactory : IHttpClientFactory
|
||||
{
|
||||
private readonly HttpClient _client;
|
||||
|
||||
public SingleHttpClientFactory(HttpClient client)
|
||||
{
|
||||
_client = client;
|
||||
}
|
||||
|
||||
public HttpClient CreateClient(string name) => _client;
|
||||
}
|
||||
|
||||
private sealed class InMemoryConnectorStateRepository : IVexConnectorStateRepository
|
||||
{
|
||||
public VexConnectorState? CurrentState { get; private set; }
|
||||
|
||||
public ValueTask<VexConnectorState?> GetAsync(string connectorId, CancellationToken cancellationToken)
|
||||
=> ValueTask.FromResult(CurrentState);
|
||||
|
||||
public ValueTask SaveAsync(VexConnectorState state, CancellationToken cancellationToken)
|
||||
{
|
||||
CurrentState = state;
|
||||
return ValueTask.CompletedTask;
|
||||
}
|
||||
}
|
||||
|
||||
private sealed class InMemoryRawSink : IVexRawDocumentSink
|
||||
{
|
||||
public List<VexRawDocument> Documents { get; } = new();
|
||||
|
||||
public ValueTask StoreAsync(VexRawDocument document, CancellationToken cancellationToken)
|
||||
{
|
||||
Documents.Add(document);
|
||||
return ValueTask.CompletedTask;
|
||||
}
|
||||
}
|
||||
|
||||
private sealed class NoopSignatureVerifier : IVexSignatureVerifier
|
||||
{
|
||||
public ValueTask<VexSignatureMetadata?> VerifyAsync(VexRawDocument document, CancellationToken cancellationToken)
|
||||
=> ValueTask.FromResult<VexSignatureMetadata?>(null);
|
||||
}
|
||||
|
||||
private sealed class NoopNormalizerRouter : IVexNormalizerRouter
|
||||
{
|
||||
public ValueTask<VexClaimBatch> NormalizeAsync(VexRawDocument document, CancellationToken cancellationToken)
|
||||
=> ValueTask.FromResult(new VexClaimBatch(document, ImmutableArray<VexClaim>.Empty, ImmutableDictionary<string, string>.Empty));
|
||||
}
|
||||
}
|
||||
|
||||
internal static class HttpResponseMessageExtensions
|
||||
{
|
||||
public static HttpResponseMessage Clone(this HttpResponseMessage response)
|
||||
{
|
||||
var clone = new HttpResponseMessage(response.StatusCode);
|
||||
foreach (var header in response.Headers)
|
||||
{
|
||||
clone.Headers.TryAddWithoutValidation(header.Key, header.Value);
|
||||
}
|
||||
|
||||
if (response.Content is not null)
|
||||
{
|
||||
var payload = response.Content.ReadAsStringAsync().GetAwaiter().GetResult();
|
||||
clone.Content = new StringContent(payload, Encoding.UTF8, response.Content.Headers.ContentType?.MediaType);
|
||||
}
|
||||
|
||||
return clone;
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,149 @@
|
||||
using System.Net;
|
||||
using System.Net.Http;
|
||||
using System.Text;
|
||||
using FluentAssertions;
|
||||
using Microsoft.Extensions.Caching.Memory;
|
||||
using Microsoft.Extensions.Logging.Abstractions;
|
||||
using Microsoft.Extensions.Options;
|
||||
using StellaOps.Vexer.Connectors.Cisco.CSAF.Configuration;
|
||||
using StellaOps.Vexer.Connectors.Cisco.CSAF.Metadata;
|
||||
using StellaOps.Vexer.Core;
|
||||
using System.IO.Abstractions.TestingHelpers;
|
||||
|
||||
namespace StellaOps.Vexer.Connectors.Cisco.CSAF.Tests.Metadata;
|
||||
|
||||
public sealed class CiscoProviderMetadataLoaderTests
|
||||
{
|
||||
[Fact]
|
||||
public async Task LoadAsync_FetchesFromNetworkWithBearerToken()
|
||||
{
|
||||
var payload = """
|
||||
{
|
||||
"metadata": {
|
||||
"publisher": {
|
||||
"name": "Cisco CSAF",
|
||||
"category": "vendor",
|
||||
"contact_details": {
|
||||
"id": "vexer:cisco"
|
||||
}
|
||||
}
|
||||
},
|
||||
"distributions": {
|
||||
"directories": [
|
||||
"https://api.security.cisco.com/csaf/v2/advisories/"
|
||||
]
|
||||
},
|
||||
"discovery": {
|
||||
"well_known": "https://api.security.cisco.com/.well-known/csaf/provider-metadata.json",
|
||||
"rolie": "https://api.security.cisco.com/csaf/rolie/feed"
|
||||
},
|
||||
"trust": {
|
||||
"weight": 0.9,
|
||||
"cosign": {
|
||||
"issuer": "https://oidc.security.cisco.com",
|
||||
"identity_pattern": "spiffe://cisco/*"
|
||||
},
|
||||
"pgp_fingerprints": [ "1234ABCD" ]
|
||||
}
|
||||
}
|
||||
""";
|
||||
|
||||
HttpRequestMessage? capturedRequest = null;
|
||||
var handler = new FakeHttpMessageHandler(request =>
|
||||
{
|
||||
capturedRequest = request;
|
||||
return new HttpResponseMessage(HttpStatusCode.OK)
|
||||
{
|
||||
Content = new StringContent(payload, Encoding.UTF8, "application/json"),
|
||||
Headers = { ETag = new System.Net.Http.Headers.EntityTagHeaderValue("\"etag1\"") }
|
||||
};
|
||||
});
|
||||
|
||||
var httpClient = new HttpClient(handler);
|
||||
var factory = new SingleHttpClientFactory(httpClient);
|
||||
var cache = new MemoryCache(new MemoryCacheOptions());
|
||||
var options = Options.Create(new CiscoConnectorOptions
|
||||
{
|
||||
ApiToken = "token-123",
|
||||
MetadataUri = "https://api.security.cisco.com/.well-known/csaf/provider-metadata.json",
|
||||
});
|
||||
var fileSystem = new MockFileSystem();
|
||||
var loader = new CiscoProviderMetadataLoader(factory, cache, options, NullLogger<CiscoProviderMetadataLoader>.Instance, fileSystem);
|
||||
|
||||
var result = await loader.LoadAsync(CancellationToken.None);
|
||||
|
||||
result.Provider.Id.Should().Be("vexer:cisco");
|
||||
result.Provider.BaseUris.Should().ContainSingle(uri => uri.ToString() == "https://api.security.cisco.com/csaf/v2/advisories/");
|
||||
result.Provider.Discovery.RolIeService.Should().Be(new Uri("https://api.security.cisco.com/csaf/rolie/feed"));
|
||||
result.ServedFromCache.Should().BeFalse();
|
||||
capturedRequest.Should().NotBeNull();
|
||||
capturedRequest!.Headers.Authorization.Should().NotBeNull();
|
||||
capturedRequest.Headers.Authorization!.Parameter.Should().Be("token-123");
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task LoadAsync_FallsBackToOfflineSnapshot()
|
||||
{
|
||||
var payload = """
|
||||
{
|
||||
"metadata": {
|
||||
"publisher": {
|
||||
"name": "Cisco CSAF",
|
||||
"category": "vendor",
|
||||
"contact_details": {
|
||||
"id": "vexer:cisco"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
""";
|
||||
|
||||
var handler = new FakeHttpMessageHandler(_ => new HttpResponseMessage(HttpStatusCode.InternalServerError));
|
||||
var httpClient = new HttpClient(handler);
|
||||
var factory = new SingleHttpClientFactory(httpClient);
|
||||
var cache = new MemoryCache(new MemoryCacheOptions());
|
||||
var options = Options.Create(new CiscoConnectorOptions
|
||||
{
|
||||
MetadataUri = "https://api.security.cisco.com/.well-known/csaf/provider-metadata.json",
|
||||
PreferOfflineSnapshot = true,
|
||||
OfflineSnapshotPath = "/snapshots/cisco.json",
|
||||
});
|
||||
var fileSystem = new MockFileSystem(new Dictionary<string, MockFileData>
|
||||
{
|
||||
["/snapshots/cisco.json"] = new MockFileData(payload),
|
||||
});
|
||||
var loader = new CiscoProviderMetadataLoader(factory, cache, options, NullLogger<CiscoProviderMetadataLoader>.Instance, fileSystem);
|
||||
|
||||
var result = await loader.LoadAsync(CancellationToken.None);
|
||||
|
||||
result.FromOfflineSnapshot.Should().BeTrue();
|
||||
result.Provider.Id.Should().Be("vexer:cisco");
|
||||
}
|
||||
|
||||
private sealed class SingleHttpClientFactory : IHttpClientFactory
|
||||
{
|
||||
private readonly HttpClient _client;
|
||||
|
||||
public SingleHttpClientFactory(HttpClient client)
|
||||
{
|
||||
_client = client;
|
||||
}
|
||||
|
||||
public HttpClient CreateClient(string name) => _client;
|
||||
}
|
||||
|
||||
private sealed class FakeHttpMessageHandler : HttpMessageHandler
|
||||
{
|
||||
private readonly Func<HttpRequestMessage, HttpResponseMessage> _responder;
|
||||
|
||||
public FakeHttpMessageHandler(Func<HttpRequestMessage, HttpResponseMessage> responder)
|
||||
{
|
||||
_responder = responder;
|
||||
}
|
||||
|
||||
protected override Task<HttpResponseMessage> SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
|
||||
{
|
||||
return Task.FromResult(_responder(request));
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,16 @@
|
||||
<Project Sdk="Microsoft.NET.Sdk">
|
||||
<PropertyGroup>
|
||||
<TargetFramework>net10.0</TargetFramework>
|
||||
<LangVersion>preview</LangVersion>
|
||||
<Nullable>enable</Nullable>
|
||||
<ImplicitUsings>enable</ImplicitUsings>
|
||||
<TreatWarningsAsErrors>true</TreatWarningsAsErrors>
|
||||
</PropertyGroup>
|
||||
<ItemGroup>
|
||||
<PackageReference Include="FluentAssertions" Version="6.12.0" />
|
||||
<PackageReference Include="System.IO.Abstractions.TestingHelpers" Version="20.0.28" />
|
||||
</ItemGroup>
|
||||
<ItemGroup>
|
||||
<ProjectReference Include="..\StellaOps.Vexer.Connectors.Cisco.CSAF\StellaOps.Vexer.Connectors.Cisco.CSAF.csproj" />
|
||||
</ItemGroup>
|
||||
</Project>
|
||||
247
src/StellaOps.Vexer.Connectors.Cisco.CSAF/CiscoCsafConnector.cs
Normal file
247
src/StellaOps.Vexer.Connectors.Cisco.CSAF/CiscoCsafConnector.cs
Normal file
@@ -0,0 +1,247 @@
|
||||
using System.Collections.Generic;
|
||||
using System.Collections.Immutable;
|
||||
using System.Linq;
|
||||
using System.Net.Http;
|
||||
using System.Runtime.CompilerServices;
|
||||
using System.Text.Json;
|
||||
using Microsoft.Extensions.Logging;
|
||||
using StellaOps.Vexer.Connectors.Abstractions;
|
||||
using StellaOps.Vexer.Connectors.Cisco.CSAF.Configuration;
|
||||
using StellaOps.Vexer.Connectors.Cisco.CSAF.Metadata;
|
||||
using StellaOps.Vexer.Core;
|
||||
using StellaOps.Vexer.Storage.Mongo;
|
||||
|
||||
namespace StellaOps.Vexer.Connectors.Cisco.CSAF;
|
||||
|
||||
public sealed class CiscoCsafConnector : VexConnectorBase
|
||||
{
|
||||
private static readonly VexConnectorDescriptor DescriptorInstance = new(
|
||||
id: "vexer:cisco",
|
||||
kind: VexProviderKind.Vendor,
|
||||
displayName: "Cisco CSAF")
|
||||
{
|
||||
Tags = ImmutableArray.Create("cisco", "csaf"),
|
||||
};
|
||||
|
||||
private readonly CiscoProviderMetadataLoader _metadataLoader;
|
||||
private readonly IHttpClientFactory _httpClientFactory;
|
||||
private readonly IVexConnectorStateRepository _stateRepository;
|
||||
private readonly IEnumerable<IVexConnectorOptionsValidator<CiscoConnectorOptions>> _validators;
|
||||
private readonly JsonSerializerOptions _serializerOptions = new(JsonSerializerDefaults.Web);
|
||||
|
||||
private CiscoConnectorOptions? _options;
|
||||
private CiscoProviderMetadataResult? _providerMetadata;
|
||||
|
||||
public CiscoCsafConnector(
|
||||
CiscoProviderMetadataLoader metadataLoader,
|
||||
IHttpClientFactory httpClientFactory,
|
||||
IVexConnectorStateRepository stateRepository,
|
||||
IEnumerable<IVexConnectorOptionsValidator<CiscoConnectorOptions>>? validators,
|
||||
ILogger<CiscoCsafConnector> logger,
|
||||
TimeProvider timeProvider)
|
||||
: base(DescriptorInstance, logger, timeProvider)
|
||||
{
|
||||
_metadataLoader = metadataLoader ?? throw new ArgumentNullException(nameof(metadataLoader));
|
||||
_httpClientFactory = httpClientFactory ?? throw new ArgumentNullException(nameof(httpClientFactory));
|
||||
_stateRepository = stateRepository ?? throw new ArgumentNullException(nameof(stateRepository));
|
||||
_validators = validators ?? Array.Empty<IVexConnectorOptionsValidator<CiscoConnectorOptions>>();
|
||||
}
|
||||
|
||||
public override async ValueTask ValidateAsync(VexConnectorSettings settings, CancellationToken cancellationToken)
|
||||
{
|
||||
_options = VexConnectorOptionsBinder.Bind(
|
||||
Descriptor,
|
||||
settings,
|
||||
validators: _validators);
|
||||
|
||||
_providerMetadata = await _metadataLoader.LoadAsync(cancellationToken).ConfigureAwait(false);
|
||||
LogConnectorEvent(LogLevel.Information, "validate", "Cisco CSAF metadata loaded.", new Dictionary<string, object?>
|
||||
{
|
||||
["baseUriCount"] = _providerMetadata.Provider.BaseUris.Length,
|
||||
["fromOffline"] = _providerMetadata.FromOfflineSnapshot,
|
||||
});
|
||||
}
|
||||
|
||||
public override async IAsyncEnumerable<VexRawDocument> FetchAsync(VexConnectorContext context, [EnumeratorCancellation] CancellationToken cancellationToken)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(context);
|
||||
|
||||
if (_options is null)
|
||||
{
|
||||
throw new InvalidOperationException("Connector must be validated before fetch operations.");
|
||||
}
|
||||
|
||||
if (_providerMetadata is null)
|
||||
{
|
||||
_providerMetadata = await _metadataLoader.LoadAsync(cancellationToken).ConfigureAwait(false);
|
||||
}
|
||||
|
||||
var state = await _stateRepository.GetAsync(Descriptor.Id, cancellationToken).ConfigureAwait(false);
|
||||
var knownDigests = state?.DocumentDigests ?? ImmutableArray<string>.Empty;
|
||||
var digestSet = new HashSet<string>(knownDigests, StringComparer.OrdinalIgnoreCase);
|
||||
var digestList = new List<string>(knownDigests);
|
||||
var since = context.Since ?? state?.LastUpdated ?? DateTimeOffset.MinValue;
|
||||
var latestTimestamp = state?.LastUpdated ?? since;
|
||||
var stateChanged = false;
|
||||
|
||||
var client = _httpClientFactory.CreateClient(CiscoConnectorOptions.HttpClientName);
|
||||
foreach (var directory in _providerMetadata.Provider.BaseUris)
|
||||
{
|
||||
await foreach (var advisory in EnumerateCatalogAsync(client, directory, cancellationToken).ConfigureAwait(false))
|
||||
{
|
||||
var published = advisory.LastModified ?? advisory.Published ?? DateTimeOffset.MinValue;
|
||||
if (published <= since)
|
||||
{
|
||||
continue;
|
||||
}
|
||||
|
||||
using var contentResponse = await client.GetAsync(advisory.DocumentUri, HttpCompletionOption.ResponseHeadersRead, cancellationToken).ConfigureAwait(false);
|
||||
contentResponse.EnsureSuccessStatusCode();
|
||||
var payload = await contentResponse.Content.ReadAsByteArrayAsync(cancellationToken).ConfigureAwait(false);
|
||||
|
||||
var rawDocument = CreateRawDocument(
|
||||
VexDocumentFormat.Csaf,
|
||||
advisory.DocumentUri,
|
||||
payload,
|
||||
BuildMetadata(builder => builder
|
||||
.Add("cisco.csaf.advisoryId", advisory.Id)
|
||||
.Add("cisco.csaf.revision", advisory.Revision)
|
||||
.Add("cisco.csaf.published", advisory.Published?.ToString("O"))
|
||||
.Add("cisco.csaf.modified", advisory.LastModified?.ToString("O"))
|
||||
.Add("cisco.csaf.sha256", advisory.Sha256)));
|
||||
|
||||
if (!digestSet.Add(rawDocument.Digest))
|
||||
{
|
||||
continue;
|
||||
}
|
||||
|
||||
await context.RawSink.StoreAsync(rawDocument, cancellationToken).ConfigureAwait(false);
|
||||
digestList.Add(rawDocument.Digest);
|
||||
stateChanged = true;
|
||||
if (published > latestTimestamp)
|
||||
{
|
||||
latestTimestamp = published;
|
||||
}
|
||||
|
||||
yield return rawDocument;
|
||||
}
|
||||
}
|
||||
|
||||
if (stateChanged)
|
||||
{
|
||||
var newState = new VexConnectorState(
|
||||
Descriptor.Id,
|
||||
latestTimestamp == DateTimeOffset.MinValue ? state?.LastUpdated : latestTimestamp,
|
||||
digestList.ToImmutableArray());
|
||||
await _stateRepository.SaveAsync(newState, cancellationToken).ConfigureAwait(false);
|
||||
}
|
||||
}
|
||||
|
||||
public override ValueTask<VexClaimBatch> NormalizeAsync(VexRawDocument document, CancellationToken cancellationToken)
|
||||
=> throw new NotSupportedException("CiscoCsafConnector relies on CSAF normalizers for document processing.");
|
||||
|
||||
private async IAsyncEnumerable<CiscoAdvisoryEntry> EnumerateCatalogAsync(HttpClient client, Uri directory, [EnumeratorCancellation] CancellationToken cancellationToken)
|
||||
{
|
||||
var nextUri = BuildIndexUri(directory, null);
|
||||
while (nextUri is not null)
|
||||
{
|
||||
using var response = await client.GetAsync(nextUri, HttpCompletionOption.ResponseHeadersRead, cancellationToken).ConfigureAwait(false);
|
||||
response.EnsureSuccessStatusCode();
|
||||
var json = await response.Content.ReadAsStringAsync(cancellationToken).ConfigureAwait(false);
|
||||
var page = JsonSerializer.Deserialize<CiscoAdvisoryIndex>(json, _serializerOptions);
|
||||
if (page?.Advisories is null)
|
||||
{
|
||||
yield break;
|
||||
}
|
||||
|
||||
foreach (var advisory in page.Advisories)
|
||||
{
|
||||
if (string.IsNullOrWhiteSpace(advisory.Url))
|
||||
{
|
||||
continue;
|
||||
}
|
||||
|
||||
if (!Uri.TryCreate(advisory.Url, UriKind.RelativeOrAbsolute, out var documentUri))
|
||||
{
|
||||
continue;
|
||||
}
|
||||
|
||||
if (!documentUri.IsAbsoluteUri)
|
||||
{
|
||||
documentUri = new Uri(directory, documentUri);
|
||||
}
|
||||
|
||||
yield return new CiscoAdvisoryEntry(
|
||||
advisory.Id ?? documentUri.Segments.LastOrDefault()?.Trim('/') ?? documentUri.ToString(),
|
||||
documentUri,
|
||||
advisory.Revision,
|
||||
advisory.Published,
|
||||
advisory.LastModified,
|
||||
advisory.Sha256);
|
||||
}
|
||||
|
||||
nextUri = ResolveNextUri(directory, page.Next);
|
||||
}
|
||||
}
|
||||
|
||||
private static Uri BuildIndexUri(Uri directory, string? relative)
|
||||
{
|
||||
if (string.IsNullOrWhiteSpace(relative))
|
||||
{
|
||||
var baseText = directory.ToString();
|
||||
if (!baseText.EndsWith('/'))
|
||||
{
|
||||
baseText += "/";
|
||||
}
|
||||
|
||||
return new Uri(new Uri(baseText, UriKind.Absolute), "index.json");
|
||||
}
|
||||
|
||||
if (Uri.TryCreate(relative, UriKind.Absolute, out var absolute))
|
||||
{
|
||||
return absolute;
|
||||
}
|
||||
|
||||
var baseTextRelative = directory.ToString();
|
||||
if (!baseTextRelative.EndsWith('/'))
|
||||
{
|
||||
baseTextRelative += "/";
|
||||
}
|
||||
|
||||
return new Uri(new Uri(baseTextRelative, UriKind.Absolute), relative);
|
||||
}
|
||||
|
||||
private static Uri? ResolveNextUri(Uri directory, string? next)
|
||||
{
|
||||
if (string.IsNullOrWhiteSpace(next))
|
||||
{
|
||||
return null;
|
||||
}
|
||||
|
||||
return BuildIndexUri(directory, next);
|
||||
}
|
||||
|
||||
private sealed record CiscoAdvisoryIndex
|
||||
{
|
||||
public List<CiscoAdvisory>? Advisories { get; init; }
|
||||
public string? Next { get; init; }
|
||||
}
|
||||
|
||||
private sealed record CiscoAdvisory
|
||||
{
|
||||
public string? Id { get; init; }
|
||||
public string? Url { get; init; }
|
||||
public string? Revision { get; init; }
|
||||
public DateTimeOffset? Published { get; init; }
|
||||
public DateTimeOffset? LastModified { get; init; }
|
||||
public string? Sha256 { get; init; }
|
||||
}
|
||||
|
||||
private sealed record CiscoAdvisoryEntry(
|
||||
string Id,
|
||||
Uri DocumentUri,
|
||||
string? Revision,
|
||||
DateTimeOffset? Published,
|
||||
DateTimeOffset? LastModified,
|
||||
string? Sha256);
|
||||
}
|
||||
@@ -0,0 +1,58 @@
|
||||
using System.ComponentModel.DataAnnotations;
|
||||
|
||||
namespace StellaOps.Vexer.Connectors.Cisco.CSAF.Configuration;
|
||||
|
||||
public sealed class CiscoConnectorOptions : IValidatableObject
|
||||
{
|
||||
public const string HttpClientName = "cisco-csaf";
|
||||
|
||||
/// <summary>
|
||||
/// Endpoint for Cisco CSAF provider metadata discovery.
|
||||
/// </summary>
|
||||
[Required]
|
||||
public string MetadataUri { get; set; } = "https://api.security.cisco.com/.well-known/csaf/provider-metadata.json";
|
||||
|
||||
/// <summary>
|
||||
/// Optional bearer token used when Cisco endpoints require authentication.
|
||||
/// </summary>
|
||||
public string? ApiToken { get; set; }
|
||||
|
||||
/// <summary>
|
||||
/// How long provider metadata remains cached.
|
||||
/// </summary>
|
||||
public TimeSpan MetadataCacheDuration { get; set; } = TimeSpan.FromHours(6);
|
||||
|
||||
/// <summary>
|
||||
/// Whether to prefer offline snapshots when fetching metadata.
|
||||
/// </summary>
|
||||
public bool PreferOfflineSnapshot { get; set; }
|
||||
|
||||
/// <summary>
|
||||
/// When set, provider metadata will be persisted to the given file path.
|
||||
/// </summary>
|
||||
public bool PersistOfflineSnapshot { get; set; }
|
||||
|
||||
public string? OfflineSnapshotPath { get; set; }
|
||||
|
||||
public IEnumerable<ValidationResult> Validate(ValidationContext validationContext)
|
||||
{
|
||||
if (string.IsNullOrWhiteSpace(MetadataUri))
|
||||
{
|
||||
yield return new ValidationResult("MetadataUri must be provided.", new[] { nameof(MetadataUri) });
|
||||
}
|
||||
else if (!Uri.TryCreate(MetadataUri, UriKind.Absolute, out _))
|
||||
{
|
||||
yield return new ValidationResult("MetadataUri must be an absolute URI.", new[] { nameof(MetadataUri) });
|
||||
}
|
||||
|
||||
if (MetadataCacheDuration <= TimeSpan.Zero)
|
||||
{
|
||||
yield return new ValidationResult("MetadataCacheDuration must be greater than zero.", new[] { nameof(MetadataCacheDuration) });
|
||||
}
|
||||
|
||||
if (PersistOfflineSnapshot && string.IsNullOrWhiteSpace(OfflineSnapshotPath))
|
||||
{
|
||||
yield return new ValidationResult("OfflineSnapshotPath must be provided when PersistOfflineSnapshot is enabled.", new[] { nameof(OfflineSnapshotPath) });
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,25 @@
|
||||
using System;
|
||||
using System.Collections.Generic;
|
||||
using System.ComponentModel.DataAnnotations;
|
||||
using StellaOps.Vexer.Connectors.Abstractions;
|
||||
|
||||
namespace StellaOps.Vexer.Connectors.Cisco.CSAF.Configuration;
|
||||
|
||||
public sealed class CiscoConnectorOptionsValidator : IVexConnectorOptionsValidator<CiscoConnectorOptions>
|
||||
{
|
||||
public void Validate(VexConnectorDescriptor descriptor, CiscoConnectorOptions options, IList<string> errors)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(descriptor);
|
||||
ArgumentNullException.ThrowIfNull(options);
|
||||
ArgumentNullException.ThrowIfNull(errors);
|
||||
|
||||
var validationResults = new List<ValidationResult>();
|
||||
if (!Validator.TryValidateObject(options, new ValidationContext(options), validationResults, validateAllProperties: true))
|
||||
{
|
||||
foreach (var result in validationResults)
|
||||
{
|
||||
errors.Add(result.ErrorMessage ?? "Cisco connector options validation failed.");
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,52 @@
|
||||
using System.ComponentModel.DataAnnotations;
|
||||
using System.Net.Http.Headers;
|
||||
using Microsoft.Extensions.Caching.Memory;
|
||||
using Microsoft.Extensions.DependencyInjection;
|
||||
using Microsoft.Extensions.DependencyInjection.Extensions;
|
||||
using Microsoft.Extensions.Options;
|
||||
using StellaOps.Vexer.Connectors.Cisco.CSAF.Configuration;
|
||||
using StellaOps.Vexer.Connectors.Cisco.CSAF.Metadata;
|
||||
using StellaOps.Vexer.Connectors.Abstractions;
|
||||
using StellaOps.Vexer.Core;
|
||||
using System.IO.Abstractions;
|
||||
|
||||
namespace StellaOps.Vexer.Connectors.Cisco.CSAF.DependencyInjection;
|
||||
|
||||
public static class CiscoConnectorServiceCollectionExtensions
|
||||
{
|
||||
public static IServiceCollection AddCiscoCsafConnector(this IServiceCollection services, Action<CiscoConnectorOptions>? configure = null)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(services);
|
||||
|
||||
services.AddOptions<CiscoConnectorOptions>()
|
||||
.Configure(options =>
|
||||
{
|
||||
configure?.Invoke(options);
|
||||
})
|
||||
.PostConfigure(options =>
|
||||
{
|
||||
Validator.ValidateObject(options, new ValidationContext(options), validateAllProperties: true);
|
||||
});
|
||||
|
||||
services.TryAddSingleton<IMemoryCache, MemoryCache>();
|
||||
services.TryAddSingleton<IFileSystem, FileSystem>();
|
||||
services.TryAddEnumerable(ServiceDescriptor.Singleton<IVexConnectorOptionsValidator<CiscoConnectorOptions>, CiscoConnectorOptionsValidator>());
|
||||
|
||||
services.AddHttpClient(CiscoConnectorOptions.HttpClientName)
|
||||
.ConfigureHttpClient((provider, client) =>
|
||||
{
|
||||
var options = provider.GetRequiredService<IOptions<CiscoConnectorOptions>>().Value;
|
||||
client.Timeout = TimeSpan.FromSeconds(30);
|
||||
client.DefaultRequestHeaders.Accept.ParseAdd("application/json");
|
||||
if (!string.IsNullOrWhiteSpace(options.ApiToken))
|
||||
{
|
||||
client.DefaultRequestHeaders.Authorization = new AuthenticationHeaderValue("Bearer", options.ApiToken);
|
||||
}
|
||||
});
|
||||
|
||||
services.AddSingleton<CiscoProviderMetadataLoader>();
|
||||
services.AddSingleton<IVexConnector, CiscoCsafConnector>();
|
||||
|
||||
return services;
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,332 @@
|
||||
using System.Collections.Immutable;
|
||||
using System.Net;
|
||||
using System.Net.Http.Headers;
|
||||
using System.Text.Json;
|
||||
using Microsoft.Extensions.Caching.Memory;
|
||||
using Microsoft.Extensions.Logging;
|
||||
using Microsoft.Extensions.Options;
|
||||
using StellaOps.Vexer.Connectors.Cisco.CSAF.Configuration;
|
||||
using StellaOps.Vexer.Core;
|
||||
using System.IO.Abstractions;
|
||||
|
||||
namespace StellaOps.Vexer.Connectors.Cisco.CSAF.Metadata;
|
||||
|
||||
public sealed class CiscoProviderMetadataLoader
|
||||
{
|
||||
public const string CacheKey = "StellaOps.Vexer.Connectors.Cisco.CSAF.Metadata";
|
||||
|
||||
private readonly IHttpClientFactory _httpClientFactory;
|
||||
private readonly IMemoryCache _memoryCache;
|
||||
private readonly ILogger<CiscoProviderMetadataLoader> _logger;
|
||||
private readonly CiscoConnectorOptions _options;
|
||||
private readonly IFileSystem _fileSystem;
|
||||
private readonly JsonSerializerOptions _serializerOptions;
|
||||
private readonly SemaphoreSlim _semaphore = new(1, 1);
|
||||
|
||||
public CiscoProviderMetadataLoader(
|
||||
IHttpClientFactory httpClientFactory,
|
||||
IMemoryCache memoryCache,
|
||||
IOptions<CiscoConnectorOptions> options,
|
||||
ILogger<CiscoProviderMetadataLoader> logger,
|
||||
IFileSystem? fileSystem = null)
|
||||
{
|
||||
_httpClientFactory = httpClientFactory ?? throw new ArgumentNullException(nameof(httpClientFactory));
|
||||
_memoryCache = memoryCache ?? throw new ArgumentNullException(nameof(memoryCache));
|
||||
_logger = logger ?? throw new ArgumentNullException(nameof(logger));
|
||||
ArgumentNullException.ThrowIfNull(options);
|
||||
_options = options.Value ?? throw new ArgumentNullException(nameof(options));
|
||||
_fileSystem = fileSystem ?? new FileSystem();
|
||||
_serializerOptions = new JsonSerializerOptions(JsonSerializerDefaults.Web)
|
||||
{
|
||||
PropertyNameCaseInsensitive = true,
|
||||
ReadCommentHandling = JsonCommentHandling.Skip,
|
||||
};
|
||||
}
|
||||
|
||||
public async Task<CiscoProviderMetadataResult> LoadAsync(CancellationToken cancellationToken)
|
||||
{
|
||||
if (_memoryCache.TryGetValue<CacheEntry>(CacheKey, out var cached) && cached is not null && !cached.IsExpired())
|
||||
{
|
||||
_logger.LogDebug("Returning cached Cisco provider metadata (expires {Expires}).", cached.ExpiresAt);
|
||||
return new CiscoProviderMetadataResult(cached.Provider, cached.FetchedAt, cached.FromOffline, true);
|
||||
}
|
||||
|
||||
await _semaphore.WaitAsync(cancellationToken).ConfigureAwait(false);
|
||||
try
|
||||
{
|
||||
if (_memoryCache.TryGetValue<CacheEntry>(CacheKey, out cached) && cached is not null && !cached.IsExpired())
|
||||
{
|
||||
return new CiscoProviderMetadataResult(cached.Provider, cached.FetchedAt, cached.FromOffline, true);
|
||||
}
|
||||
|
||||
CacheEntry? previous = cached;
|
||||
|
||||
if (!_options.PreferOfflineSnapshot)
|
||||
{
|
||||
var network = await TryFetchFromNetworkAsync(previous, cancellationToken).ConfigureAwait(false);
|
||||
if (network is not null)
|
||||
{
|
||||
StoreCache(network);
|
||||
return new CiscoProviderMetadataResult(network.Provider, network.FetchedAt, false, false);
|
||||
}
|
||||
}
|
||||
|
||||
var offline = TryLoadFromOffline();
|
||||
if (offline is not null)
|
||||
{
|
||||
var entry = offline with
|
||||
{
|
||||
FetchedAt = DateTimeOffset.UtcNow,
|
||||
ExpiresAt = DateTimeOffset.UtcNow + _options.MetadataCacheDuration,
|
||||
FromOffline = true,
|
||||
};
|
||||
StoreCache(entry);
|
||||
return new CiscoProviderMetadataResult(entry.Provider, entry.FetchedAt, true, false);
|
||||
}
|
||||
|
||||
throw new InvalidOperationException("Unable to load Cisco CSAF provider metadata from network or offline snapshot.");
|
||||
}
|
||||
finally
|
||||
{
|
||||
_semaphore.Release();
|
||||
}
|
||||
}
|
||||
|
||||
private async Task<CacheEntry?> TryFetchFromNetworkAsync(CacheEntry? previous, CancellationToken cancellationToken)
|
||||
{
|
||||
try
|
||||
{
|
||||
var client = _httpClientFactory.CreateClient(CiscoConnectorOptions.HttpClientName);
|
||||
using var request = new HttpRequestMessage(HttpMethod.Get, _options.MetadataUri);
|
||||
if (!string.IsNullOrWhiteSpace(_options.ApiToken))
|
||||
{
|
||||
request.Headers.Authorization = new AuthenticationHeaderValue("Bearer", _options.ApiToken);
|
||||
}
|
||||
|
||||
if (!string.IsNullOrWhiteSpace(previous?.ETag) && EntityTagHeaderValue.TryParse(previous.ETag, out var etag))
|
||||
{
|
||||
request.Headers.IfNoneMatch.Add(etag);
|
||||
}
|
||||
|
||||
using var response = await client.SendAsync(request, HttpCompletionOption.ResponseHeadersRead, cancellationToken).ConfigureAwait(false);
|
||||
|
||||
if (response.StatusCode == HttpStatusCode.NotModified && previous is not null)
|
||||
{
|
||||
_logger.LogDebug("Cisco provider metadata not modified (etag {ETag}).", previous.ETag);
|
||||
return previous with
|
||||
{
|
||||
FetchedAt = DateTimeOffset.UtcNow,
|
||||
ExpiresAt = DateTimeOffset.UtcNow + _options.MetadataCacheDuration,
|
||||
};
|
||||
}
|
||||
|
||||
response.EnsureSuccessStatusCode();
|
||||
var payload = await response.Content.ReadAsStringAsync(cancellationToken).ConfigureAwait(false);
|
||||
var provider = ParseProvider(payload);
|
||||
var etagHeader = response.Headers.ETag?.ToString();
|
||||
|
||||
if (_options.PersistOfflineSnapshot && !string.IsNullOrWhiteSpace(_options.OfflineSnapshotPath))
|
||||
{
|
||||
try
|
||||
{
|
||||
_fileSystem.File.WriteAllText(_options.OfflineSnapshotPath, payload);
|
||||
_logger.LogDebug("Persisted Cisco metadata snapshot to {Path}.", _options.OfflineSnapshotPath);
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
_logger.LogWarning(ex, "Failed to persist Cisco metadata snapshot to {Path}.", _options.OfflineSnapshotPath);
|
||||
}
|
||||
}
|
||||
|
||||
return new CacheEntry(
|
||||
provider,
|
||||
DateTimeOffset.UtcNow,
|
||||
DateTimeOffset.UtcNow + _options.MetadataCacheDuration,
|
||||
etagHeader,
|
||||
FromOffline: false);
|
||||
}
|
||||
catch (Exception ex) when (ex is not OperationCanceledException && !_options.PreferOfflineSnapshot)
|
||||
{
|
||||
_logger.LogWarning(ex, "Failed to fetch Cisco provider metadata from {Uri}; falling back to offline snapshot when available.", _options.MetadataUri);
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
private CacheEntry? TryLoadFromOffline()
|
||||
{
|
||||
if (string.IsNullOrWhiteSpace(_options.OfflineSnapshotPath))
|
||||
{
|
||||
return null;
|
||||
}
|
||||
|
||||
if (!_fileSystem.File.Exists(_options.OfflineSnapshotPath))
|
||||
{
|
||||
_logger.LogWarning("Cisco offline snapshot path {Path} does not exist.", _options.OfflineSnapshotPath);
|
||||
return null;
|
||||
}
|
||||
|
||||
try
|
||||
{
|
||||
var payload = _fileSystem.File.ReadAllText(_options.OfflineSnapshotPath);
|
||||
var provider = ParseProvider(payload);
|
||||
return new CacheEntry(provider, DateTimeOffset.UtcNow, DateTimeOffset.UtcNow + _options.MetadataCacheDuration, null, true);
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
_logger.LogError(ex, "Failed to load Cisco provider metadata from offline snapshot {Path}.", _options.OfflineSnapshotPath);
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
private VexProvider ParseProvider(string payload)
|
||||
{
|
||||
if (string.IsNullOrWhiteSpace(payload))
|
||||
{
|
||||
throw new InvalidOperationException("Cisco provider metadata payload was empty.");
|
||||
}
|
||||
|
||||
ProviderMetadataDocument? document;
|
||||
try
|
||||
{
|
||||
document = JsonSerializer.Deserialize<ProviderMetadataDocument>(payload, _serializerOptions);
|
||||
}
|
||||
catch (JsonException ex)
|
||||
{
|
||||
throw new InvalidOperationException("Failed to parse Cisco provider metadata.", ex);
|
||||
}
|
||||
|
||||
if (document?.Metadata?.Publisher?.ContactDetails is null || string.IsNullOrWhiteSpace(document.Metadata.Publisher.ContactDetails.Id))
|
||||
{
|
||||
throw new InvalidOperationException("Cisco provider metadata did not include a publisher identifier.");
|
||||
}
|
||||
|
||||
var discovery = new VexProviderDiscovery(document.Discovery?.WellKnown, document.Discovery?.RolIe);
|
||||
var trust = document.Trust is null
|
||||
? VexProviderTrust.Default
|
||||
: new VexProviderTrust(
|
||||
document.Trust.Weight ?? 1.0,
|
||||
document.Trust.Cosign is null ? null : new VexCosignTrust(document.Trust.Cosign.Issuer ?? string.Empty, document.Trust.Cosign.IdentityPattern ?? string.Empty),
|
||||
document.Trust.PgpFingerprints ?? Enumerable.Empty<string>());
|
||||
|
||||
var directories = document.Distributions?.Directories is null
|
||||
? Enumerable.Empty<Uri>()
|
||||
: document.Distributions.Directories
|
||||
.Where(static s => !string.IsNullOrWhiteSpace(s))
|
||||
.Select(static s => Uri.TryCreate(s, UriKind.Absolute, out var uri) ? uri : null)
|
||||
.Where(static uri => uri is not null)!
|
||||
.Select(static uri => uri!);
|
||||
|
||||
return new VexProvider(
|
||||
id: document.Metadata.Publisher.ContactDetails.Id,
|
||||
displayName: document.Metadata.Publisher.Name ?? document.Metadata.Publisher.ContactDetails.Id,
|
||||
kind: document.Metadata.Publisher.Category?.Equals("vendor", StringComparison.OrdinalIgnoreCase) == true ? VexProviderKind.Vendor : VexProviderKind.Hub,
|
||||
baseUris: directories,
|
||||
discovery: discovery,
|
||||
trust: trust,
|
||||
enabled: true);
|
||||
}
|
||||
|
||||
private void StoreCache(CacheEntry entry)
|
||||
{
|
||||
var options = new MemoryCacheEntryOptions
|
||||
{
|
||||
AbsoluteExpiration = entry.ExpiresAt,
|
||||
};
|
||||
_memoryCache.Set(CacheKey, entry, options);
|
||||
}
|
||||
|
||||
private sealed record CacheEntry(
|
||||
VexProvider Provider,
|
||||
DateTimeOffset FetchedAt,
|
||||
DateTimeOffset ExpiresAt,
|
||||
string? ETag,
|
||||
bool FromOffline)
|
||||
{
|
||||
public bool IsExpired() => DateTimeOffset.UtcNow >= ExpiresAt;
|
||||
}
|
||||
}
|
||||
|
||||
public sealed record CiscoProviderMetadataResult(
|
||||
VexProvider Provider,
|
||||
DateTimeOffset FetchedAt,
|
||||
bool FromOfflineSnapshot,
|
||||
bool ServedFromCache);
|
||||
|
||||
#region document models
|
||||
|
||||
internal sealed class ProviderMetadataDocument
|
||||
{
|
||||
[System.Text.Json.Serialization.JsonPropertyName("metadata")]
|
||||
public ProviderMetadataMetadata Metadata { get; set; } = new();
|
||||
|
||||
[System.Text.Json.Serialization.JsonPropertyName("discovery")]
|
||||
public ProviderMetadataDiscovery? Discovery { get; set; }
|
||||
|
||||
[System.Text.Json.Serialization.JsonPropertyName("trust")]
|
||||
public ProviderMetadataTrust? Trust { get; set; }
|
||||
|
||||
[System.Text.Json.Serialization.JsonPropertyName("distributions")]
|
||||
public ProviderMetadataDistributions? Distributions { get; set; }
|
||||
}
|
||||
|
||||
internal sealed class ProviderMetadataMetadata
|
||||
{
|
||||
[System.Text.Json.Serialization.JsonPropertyName("publisher")]
|
||||
public ProviderMetadataPublisher Publisher { get; set; } = new();
|
||||
}
|
||||
|
||||
internal sealed class ProviderMetadataPublisher
|
||||
{
|
||||
[System.Text.Json.Serialization.JsonPropertyName("name")]
|
||||
public string? Name { get; set; }
|
||||
|
||||
[System.Text.Json.Serialization.JsonPropertyName("category")]
|
||||
public string? Category { get; set; }
|
||||
|
||||
[System.Text.Json.Serialization.JsonPropertyName("contact_details")]
|
||||
public ProviderMetadataPublisherContact ContactDetails { get; set; } = new();
|
||||
}
|
||||
|
||||
internal sealed class ProviderMetadataPublisherContact
|
||||
{
|
||||
[System.Text.Json.Serialization.JsonPropertyName("id")]
|
||||
public string? Id { get; set; }
|
||||
}
|
||||
|
||||
internal sealed class ProviderMetadataDiscovery
|
||||
{
|
||||
[System.Text.Json.Serialization.JsonPropertyName("well_known")]
|
||||
public Uri? WellKnown { get; set; }
|
||||
|
||||
[System.Text.Json.Serialization.JsonPropertyName("rolie")]
|
||||
public Uri? RolIe { get; set; }
|
||||
}
|
||||
|
||||
internal sealed class ProviderMetadataTrust
|
||||
{
|
||||
[System.Text.Json.Serialization.JsonPropertyName("weight")]
|
||||
public double? Weight { get; set; }
|
||||
|
||||
[System.Text.Json.Serialization.JsonPropertyName("cosign")]
|
||||
public ProviderMetadataTrustCosign? Cosign { get; set; }
|
||||
|
||||
[System.Text.Json.Serialization.JsonPropertyName("pgp_fingerprints")]
|
||||
public string[]? PgpFingerprints { get; set; }
|
||||
}
|
||||
|
||||
internal sealed class ProviderMetadataTrustCosign
|
||||
{
|
||||
[System.Text.Json.Serialization.JsonPropertyName("issuer")]
|
||||
public string? Issuer { get; set; }
|
||||
|
||||
[System.Text.Json.Serialization.JsonPropertyName("identity_pattern")]
|
||||
public string? IdentityPattern { get; set; }
|
||||
}
|
||||
|
||||
internal sealed class ProviderMetadataDistributions
|
||||
{
|
||||
[System.Text.Json.Serialization.JsonPropertyName("directories")]
|
||||
public string[]? Directories { get; set; }
|
||||
}
|
||||
|
||||
#endregion
|
||||
@@ -0,0 +1,20 @@
|
||||
<Project Sdk="Microsoft.NET.Sdk">
|
||||
<PropertyGroup>
|
||||
<TargetFramework>net10.0</TargetFramework>
|
||||
<LangVersion>preview</LangVersion>
|
||||
<Nullable>enable</Nullable>
|
||||
<ImplicitUsings>enable</ImplicitUsings>
|
||||
<TreatWarningsAsErrors>true</TreatWarningsAsErrors>
|
||||
</PropertyGroup>
|
||||
<ItemGroup>
|
||||
<ProjectReference Include="..\StellaOps.Vexer.Connectors.Abstractions\StellaOps.Vexer.Connectors.Abstractions.csproj" />
|
||||
<ProjectReference Include="..\StellaOps.Vexer.Core\StellaOps.Vexer.Core.csproj" />
|
||||
<ProjectReference Include="..\StellaOps.Vexer.Storage.Mongo\StellaOps.Vexer.Storage.Mongo.csproj" />
|
||||
</ItemGroup>
|
||||
<ItemGroup>
|
||||
<PackageReference Include="Microsoft.Extensions.Caching.Memory" Version="8.0.0" />
|
||||
<PackageReference Include="Microsoft.Extensions.Http" Version="8.0.0" />
|
||||
<PackageReference Include="Microsoft.Extensions.Options" Version="8.0.0" />
|
||||
<PackageReference Include="System.IO.Abstractions" Version="20.0.28" />
|
||||
</ItemGroup>
|
||||
</Project>
|
||||
@@ -2,6 +2,6 @@ If you are working on this file you need to read docs/ARCHITECTURE_VEXER.md and
|
||||
# TASKS
|
||||
| Task | Owner(s) | Depends on | Notes |
|
||||
|---|---|---|---|
|
||||
|VEXER-CONN-CISCO-01-001 – Endpoint discovery & auth plumbing|Team Vexer Connectors – Cisco|VEXER-CONN-ABS-01-001|TODO – Resolve Cisco CSAF collection URLs, configure optional token auth, and validate discovery metadata for offline caching.|
|
||||
|VEXER-CONN-CISCO-01-002 – CSAF pull loop & pagination|Team Vexer Connectors – Cisco|VEXER-CONN-CISCO-01-001, VEXER-STORAGE-01-003|TODO – Implement paginated fetch with retries/backoff, checksum validation, and raw document persistence.|
|
||||
|VEXER-CONN-CISCO-01-001 – Endpoint discovery & auth plumbing|Team Vexer Connectors – Cisco|VEXER-CONN-ABS-01-001|**DONE (2025-10-17)** – Added `CiscoProviderMetadataLoader` with bearer token support, offline snapshot fallback, DI helpers, and tests covering network/offline discovery to unblock subsequent fetch work.|
|
||||
|VEXER-CONN-CISCO-01-002 – CSAF pull loop & pagination|Team Vexer Connectors – Cisco|VEXER-CONN-CISCO-01-001, VEXER-STORAGE-01-003|**DONE (2025-10-17)** – Implemented paginated advisory fetch using provider directories, raw document persistence with dedupe/state tracking, offline resiliency, and unit coverage.|
|
||||
|VEXER-CONN-CISCO-01-003 – Provider trust metadata|Team Vexer Connectors – Cisco|VEXER-CONN-CISCO-01-002, VEXER-POLICY-01-001|TODO – Emit cosign/PGP trust metadata and advisory provenance hints for policy weighting.|
|
||||
|
||||
@@ -0,0 +1,176 @@
|
||||
using System.Net;
|
||||
using System.Net.Http;
|
||||
using System.Text;
|
||||
using FluentAssertions;
|
||||
using Microsoft.Extensions.Caching.Memory;
|
||||
using Microsoft.Extensions.Logging.Abstractions;
|
||||
using Microsoft.Extensions.Options;
|
||||
using NSubstitute;
|
||||
using StellaOps.Vexer.Connectors.MSRC.CSAF.Authentication;
|
||||
using StellaOps.Vexer.Connectors.MSRC.CSAF.Configuration;
|
||||
using System.IO.Abstractions.TestingHelpers;
|
||||
using Xunit;
|
||||
|
||||
namespace StellaOps.Vexer.Connectors.MSRC.CSAF.Tests.Authentication;
|
||||
|
||||
public sealed class MsrcTokenProviderTests
|
||||
{
|
||||
[Fact]
|
||||
public async Task GetAccessTokenAsync_CachesUntilExpiry()
|
||||
{
|
||||
var handler = new TestHttpMessageHandler(new[]
|
||||
{
|
||||
CreateTokenResponse("token-1"),
|
||||
CreateTokenResponse("token-2"),
|
||||
});
|
||||
var client = new HttpClient(handler) { BaseAddress = new Uri("https://login.microsoftonline.com/") };
|
||||
var factory = new SingleClientHttpClientFactory(client);
|
||||
var cache = new MemoryCache(new MemoryCacheOptions());
|
||||
var fileSystem = new MockFileSystem();
|
||||
var options = Options.Create(new MsrcConnectorOptions
|
||||
{
|
||||
TenantId = "contoso.onmicrosoft.com",
|
||||
ClientId = "client-id",
|
||||
ClientSecret = "secret",
|
||||
});
|
||||
|
||||
var timeProvider = new AdjustableTimeProvider();
|
||||
var provider = new MsrcTokenProvider(factory, cache, fileSystem, options, NullLogger<MsrcTokenProvider>.Instance, timeProvider);
|
||||
|
||||
var first = await provider.GetAccessTokenAsync(CancellationToken.None);
|
||||
first.Value.Should().Be("token-1");
|
||||
handler.InvocationCount.Should().Be(1);
|
||||
|
||||
var second = await provider.GetAccessTokenAsync(CancellationToken.None);
|
||||
second.Value.Should().Be("token-1");
|
||||
handler.InvocationCount.Should().Be(1);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task GetAccessTokenAsync_RefreshesWhenExpired()
|
||||
{
|
||||
var handler = new TestHttpMessageHandler(new[]
|
||||
{
|
||||
CreateTokenResponse("token-1", expiresIn: 120),
|
||||
CreateTokenResponse("token-2", expiresIn: 3600),
|
||||
});
|
||||
var client = new HttpClient(handler) { BaseAddress = new Uri("https://login.microsoftonline.com/") };
|
||||
var factory = new SingleClientHttpClientFactory(client);
|
||||
var cache = new MemoryCache(new MemoryCacheOptions());
|
||||
var fileSystem = new MockFileSystem();
|
||||
var options = Options.Create(new MsrcConnectorOptions
|
||||
{
|
||||
TenantId = "contoso.onmicrosoft.com",
|
||||
ClientId = "client-id",
|
||||
ClientSecret = "secret",
|
||||
ExpiryLeewaySeconds = 60,
|
||||
});
|
||||
|
||||
var timeProvider = new AdjustableTimeProvider();
|
||||
var provider = new MsrcTokenProvider(factory, cache, fileSystem, options, NullLogger<MsrcTokenProvider>.Instance, timeProvider);
|
||||
|
||||
var first = await provider.GetAccessTokenAsync(CancellationToken.None);
|
||||
first.Value.Should().Be("token-1");
|
||||
handler.InvocationCount.Should().Be(1);
|
||||
|
||||
timeProvider.Advance(TimeSpan.FromMinutes(2));
|
||||
var second = await provider.GetAccessTokenAsync(CancellationToken.None);
|
||||
second.Value.Should().Be("token-2");
|
||||
handler.InvocationCount.Should().Be(2);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task GetAccessTokenAsync_OfflineStaticToken()
|
||||
{
|
||||
var factory = Substitute.For<IHttpClientFactory>();
|
||||
var cache = new MemoryCache(new MemoryCacheOptions());
|
||||
var fileSystem = new MockFileSystem();
|
||||
var options = Options.Create(new MsrcConnectorOptions
|
||||
{
|
||||
PreferOfflineToken = true,
|
||||
StaticAccessToken = "offline-token",
|
||||
});
|
||||
|
||||
var provider = new MsrcTokenProvider(factory, cache, fileSystem, options, NullLogger<MsrcTokenProvider>.Instance);
|
||||
var token = await provider.GetAccessTokenAsync(CancellationToken.None);
|
||||
token.Value.Should().Be("offline-token");
|
||||
token.ExpiresAt.Should().Be(DateTimeOffset.MaxValue);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task GetAccessTokenAsync_OfflineFileToken()
|
||||
{
|
||||
var factory = Substitute.For<IHttpClientFactory>();
|
||||
var cache = new MemoryCache(new MemoryCacheOptions());
|
||||
var fileSystem = new MockFileSystem();
|
||||
var offlinePath = fileSystem.Path.Combine("/tokens", "msrc.txt");
|
||||
fileSystem.AddFile(offlinePath, new MockFileData("file-token"));
|
||||
|
||||
var options = Options.Create(new MsrcConnectorOptions
|
||||
{
|
||||
PreferOfflineToken = true,
|
||||
OfflineTokenPath = offlinePath,
|
||||
});
|
||||
|
||||
var provider = new MsrcTokenProvider(factory, cache, fileSystem, options, NullLogger<MsrcTokenProvider>.Instance);
|
||||
var token = await provider.GetAccessTokenAsync(CancellationToken.None);
|
||||
token.Value.Should().Be("file-token");
|
||||
token.ExpiresAt.Should().Be(DateTimeOffset.MaxValue);
|
||||
}
|
||||
|
||||
private static HttpResponseMessage CreateTokenResponse(string token, int expiresIn = 3600)
|
||||
{
|
||||
var json = $"{{\"access_token\":\"{token}\",\"token_type\":\"Bearer\",\"expires_in\":{expiresIn}}}";
|
||||
return new HttpResponseMessage(HttpStatusCode.OK)
|
||||
{
|
||||
Content = new StringContent(json, Encoding.UTF8, "application/json"),
|
||||
};
|
||||
}
|
||||
|
||||
private sealed class AdjustableTimeProvider : TimeProvider
|
||||
{
|
||||
private DateTimeOffset _now = DateTimeOffset.UtcNow;
|
||||
|
||||
public override DateTimeOffset GetUtcNow() => _now;
|
||||
|
||||
public void Advance(TimeSpan span) => _now = _now.Add(span);
|
||||
}
|
||||
|
||||
private sealed class SingleClientHttpClientFactory : IHttpClientFactory
|
||||
{
|
||||
private readonly HttpClient _client;
|
||||
|
||||
public SingleClientHttpClientFactory(HttpClient client)
|
||||
{
|
||||
_client = client;
|
||||
}
|
||||
|
||||
public HttpClient CreateClient(string name) => _client;
|
||||
}
|
||||
|
||||
private sealed class TestHttpMessageHandler : HttpMessageHandler
|
||||
{
|
||||
private readonly Queue<HttpResponseMessage> _responses;
|
||||
|
||||
public TestHttpMessageHandler(IEnumerable<HttpResponseMessage> responses)
|
||||
{
|
||||
_responses = new Queue<HttpResponseMessage>(responses);
|
||||
}
|
||||
|
||||
public int InvocationCount { get; private set; }
|
||||
|
||||
protected override Task<HttpResponseMessage> SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
|
||||
{
|
||||
InvocationCount++;
|
||||
if (_responses.Count == 0)
|
||||
{
|
||||
return Task.FromResult(new HttpResponseMessage(HttpStatusCode.InternalServerError)
|
||||
{
|
||||
Content = new StringContent("no responses remaining"),
|
||||
});
|
||||
}
|
||||
|
||||
return Task.FromResult(_responses.Dequeue());
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,18 @@
|
||||
<Project Sdk="Microsoft.NET.Sdk">
|
||||
<PropertyGroup>
|
||||
<TargetFramework>net10.0</TargetFramework>
|
||||
<LangVersion>preview</LangVersion>
|
||||
<Nullable>enable</Nullable>
|
||||
<ImplicitUsings>enable</ImplicitUsings>
|
||||
<TreatWarningsAsErrors>true</TreatWarningsAsErrors>
|
||||
</PropertyGroup>
|
||||
<ItemGroup>
|
||||
<ProjectReference Include="..\StellaOps.Vexer.Connectors.MSRC.CSAF\StellaOps.Vexer.Connectors.MSRC.CSAF.csproj" />
|
||||
</ItemGroup>
|
||||
<ItemGroup>
|
||||
<PackageReference Include="FluentAssertions" Version="6.12.0" />
|
||||
<PackageReference Include="Microsoft.Extensions.Logging.Abstractions" Version="8.0.0" />
|
||||
<PackageReference Include="NSubstitute" Version="5.1.0" />
|
||||
<PackageReference Include="System.IO.Abstractions.TestingHelpers" Version="20.0.28" />
|
||||
</ItemGroup>
|
||||
</Project>
|
||||
@@ -0,0 +1,185 @@
|
||||
using System;
|
||||
using System.Collections.Generic;
|
||||
using System.IO.Abstractions;
|
||||
using System.Net.Http;
|
||||
using System.Net.Http.Json;
|
||||
using System.Text.Json.Serialization;
|
||||
using System.Threading;
|
||||
using System.Threading.Tasks;
|
||||
using Microsoft.Extensions.Caching.Memory;
|
||||
using Microsoft.Extensions.Logging;
|
||||
using Microsoft.Extensions.Options;
|
||||
using StellaOps.Vexer.Connectors.MSRC.CSAF.Configuration;
|
||||
|
||||
namespace StellaOps.Vexer.Connectors.MSRC.CSAF.Authentication;
|
||||
|
||||
public interface IMsrcTokenProvider
|
||||
{
|
||||
ValueTask<MsrcAccessToken> GetAccessTokenAsync(CancellationToken cancellationToken);
|
||||
}
|
||||
|
||||
public sealed class MsrcTokenProvider : IMsrcTokenProvider, IDisposable
|
||||
{
|
||||
private const string CachePrefix = "StellaOps.Vexer.Connectors.MSRC.CSAF.Token";
|
||||
|
||||
private readonly IHttpClientFactory _httpClientFactory;
|
||||
private readonly IMemoryCache _cache;
|
||||
private readonly IFileSystem _fileSystem;
|
||||
private readonly ILogger<MsrcTokenProvider> _logger;
|
||||
private readonly TimeProvider _timeProvider;
|
||||
private readonly MsrcConnectorOptions _options;
|
||||
private readonly SemaphoreSlim _refreshLock = new(1, 1);
|
||||
|
||||
public MsrcTokenProvider(
|
||||
IHttpClientFactory httpClientFactory,
|
||||
IMemoryCache cache,
|
||||
IFileSystem fileSystem,
|
||||
IOptions<MsrcConnectorOptions> options,
|
||||
ILogger<MsrcTokenProvider> logger,
|
||||
TimeProvider? timeProvider = null)
|
||||
{
|
||||
_httpClientFactory = httpClientFactory ?? throw new ArgumentNullException(nameof(httpClientFactory));
|
||||
_cache = cache ?? throw new ArgumentNullException(nameof(cache));
|
||||
_fileSystem = fileSystem ?? throw new ArgumentNullException(nameof(fileSystem));
|
||||
ArgumentNullException.ThrowIfNull(options);
|
||||
_options = options.Value ?? throw new ArgumentNullException(nameof(options));
|
||||
_options.Validate(_fileSystem);
|
||||
_logger = logger ?? throw new ArgumentNullException(nameof(logger));
|
||||
_timeProvider = timeProvider ?? TimeProvider.System;
|
||||
}
|
||||
|
||||
public async ValueTask<MsrcAccessToken> GetAccessTokenAsync(CancellationToken cancellationToken)
|
||||
{
|
||||
if (_options.PreferOfflineToken)
|
||||
{
|
||||
return LoadOfflineToken();
|
||||
}
|
||||
|
||||
var cacheKey = CreateCacheKey();
|
||||
if (_cache.TryGetValue<MsrcAccessToken>(cacheKey, out var cachedToken) &&
|
||||
cachedToken is not null &&
|
||||
!cachedToken.IsExpired(_timeProvider.GetUtcNow()))
|
||||
{
|
||||
return cachedToken;
|
||||
}
|
||||
|
||||
await _refreshLock.WaitAsync(cancellationToken).ConfigureAwait(false);
|
||||
try
|
||||
{
|
||||
if (_cache.TryGetValue<MsrcAccessToken>(cacheKey, out cachedToken) &&
|
||||
cachedToken is not null &&
|
||||
!cachedToken.IsExpired(_timeProvider.GetUtcNow()))
|
||||
{
|
||||
return cachedToken;
|
||||
}
|
||||
|
||||
var token = await RequestTokenAsync(cancellationToken).ConfigureAwait(false);
|
||||
var absoluteExpiration = token.ExpiresAt == DateTimeOffset.MaxValue
|
||||
? (DateTimeOffset?)null
|
||||
: token.ExpiresAt;
|
||||
|
||||
var options = new MemoryCacheEntryOptions();
|
||||
if (absoluteExpiration.HasValue)
|
||||
{
|
||||
options.AbsoluteExpiration = absoluteExpiration.Value;
|
||||
}
|
||||
|
||||
_cache.Set(cacheKey, token, options);
|
||||
return token;
|
||||
}
|
||||
finally
|
||||
{
|
||||
_refreshLock.Release();
|
||||
}
|
||||
}
|
||||
|
||||
private MsrcAccessToken LoadOfflineToken()
|
||||
{
|
||||
if (!string.IsNullOrWhiteSpace(_options.StaticAccessToken))
|
||||
{
|
||||
return new MsrcAccessToken(_options.StaticAccessToken!, "Bearer", DateTimeOffset.MaxValue);
|
||||
}
|
||||
|
||||
if (string.IsNullOrWhiteSpace(_options.OfflineTokenPath))
|
||||
{
|
||||
throw new InvalidOperationException("Offline token mode is enabled but no token was provided.");
|
||||
}
|
||||
|
||||
if (!_fileSystem.File.Exists(_options.OfflineTokenPath))
|
||||
{
|
||||
throw new InvalidOperationException($"Offline token path '{_options.OfflineTokenPath}' does not exist.");
|
||||
}
|
||||
|
||||
var token = _fileSystem.File.ReadAllText(_options.OfflineTokenPath).Trim();
|
||||
if (string.IsNullOrEmpty(token))
|
||||
{
|
||||
throw new InvalidOperationException("Offline token file was empty.");
|
||||
}
|
||||
|
||||
return new MsrcAccessToken(token, "Bearer", DateTimeOffset.MaxValue);
|
||||
}
|
||||
|
||||
private async Task<MsrcAccessToken> RequestTokenAsync(CancellationToken cancellationToken)
|
||||
{
|
||||
_logger.LogInformation("Fetching MSRC AAD access token for tenant {TenantId}.", _options.TenantId);
|
||||
|
||||
var client = _httpClientFactory.CreateClient(MsrcConnectorOptions.TokenClientName);
|
||||
using var request = new HttpRequestMessage(HttpMethod.Post, BuildTokenUri())
|
||||
{
|
||||
Content = new FormUrlEncodedContent(new Dictionary<string, string>
|
||||
{
|
||||
["client_id"] = _options.ClientId,
|
||||
["client_secret"] = _options.ClientSecret!,
|
||||
["grant_type"] = "client_credentials",
|
||||
["scope"] = string.IsNullOrWhiteSpace(_options.Scope) ? MsrcConnectorOptions.DefaultScope : _options.Scope,
|
||||
}),
|
||||
};
|
||||
|
||||
using var response = await client.SendAsync(request, cancellationToken).ConfigureAwait(false);
|
||||
if (!response.IsSuccessStatusCode)
|
||||
{
|
||||
var payload = await response.Content.ReadAsStringAsync(cancellationToken).ConfigureAwait(false);
|
||||
throw new InvalidOperationException($"Failed to acquire MSRC access token ({(int)response.StatusCode}). Response: {payload}");
|
||||
}
|
||||
|
||||
var tokenResponse = await response.Content.ReadFromJsonAsync<TokenResponse>(cancellationToken: cancellationToken).ConfigureAwait(false)
|
||||
?? throw new InvalidOperationException("Token endpoint returned an empty payload.");
|
||||
|
||||
if (string.IsNullOrWhiteSpace(tokenResponse.AccessToken))
|
||||
{
|
||||
throw new InvalidOperationException("Token endpoint response did not include an access_token.");
|
||||
}
|
||||
|
||||
var now = _timeProvider.GetUtcNow();
|
||||
var expiresAt = tokenResponse.ExpiresIn > _options.ExpiryLeewaySeconds
|
||||
? now.AddSeconds(tokenResponse.ExpiresIn - _options.ExpiryLeewaySeconds)
|
||||
: now.AddMinutes(5);
|
||||
|
||||
return new MsrcAccessToken(tokenResponse.AccessToken!, tokenResponse.TokenType ?? "Bearer", expiresAt);
|
||||
}
|
||||
|
||||
private string CreateCacheKey()
|
||||
=> $"{CachePrefix}:{_options.TenantId}:{_options.ClientId}:{_options.Scope}";
|
||||
|
||||
private Uri BuildTokenUri()
|
||||
=> new($"https://login.microsoftonline.com/{_options.TenantId}/oauth2/v2.0/token");
|
||||
|
||||
public void Dispose() => _refreshLock.Dispose();
|
||||
|
||||
private sealed record TokenResponse
|
||||
{
|
||||
[JsonPropertyName("access_token")]
|
||||
public string? AccessToken { get; init; }
|
||||
|
||||
[JsonPropertyName("token_type")]
|
||||
public string? TokenType { get; init; }
|
||||
|
||||
[JsonPropertyName("expires_in")]
|
||||
public int ExpiresIn { get; init; }
|
||||
}
|
||||
}
|
||||
|
||||
public sealed record MsrcAccessToken(string Value, string Type, DateTimeOffset ExpiresAt)
|
||||
{
|
||||
public bool IsExpired(DateTimeOffset now) => now >= ExpiresAt;
|
||||
}
|
||||
@@ -0,0 +1,95 @@
|
||||
using System;
|
||||
using System.IO;
|
||||
using System.IO.Abstractions;
|
||||
|
||||
namespace StellaOps.Vexer.Connectors.MSRC.CSAF.Configuration;
|
||||
|
||||
public sealed class MsrcConnectorOptions
|
||||
{
|
||||
public const string TokenClientName = "vexer.connector.msrc.token";
|
||||
public const string DefaultScope = "https://api.msrc.microsoft.com/.default";
|
||||
|
||||
/// <summary>
|
||||
/// Azure AD tenant identifier (GUID or domain).
|
||||
/// </summary>
|
||||
public string TenantId { get; set; } = string.Empty;
|
||||
|
||||
/// <summary>
|
||||
/// Azure AD application (client) identifier.
|
||||
/// </summary>
|
||||
public string ClientId { get; set; } = string.Empty;
|
||||
|
||||
/// <summary>
|
||||
/// Azure AD application secret for client credential flow.
|
||||
/// </summary>
|
||||
public string? ClientSecret { get; set; }
|
||||
/// <summary>
|
||||
/// OAuth scope requested for MSRC API access.
|
||||
/// </summary>
|
||||
public string Scope { get; set; } = DefaultScope;
|
||||
|
||||
/// <summary>
|
||||
/// When true, token acquisition is skipped and the connector expects offline handling.
|
||||
/// </summary>
|
||||
public bool PreferOfflineToken { get; set; }
|
||||
/// <summary>
|
||||
/// Optional path to a pre-provisioned bearer token used when <see cref="PreferOfflineToken"/> is enabled.
|
||||
/// </summary>
|
||||
public string? OfflineTokenPath { get; set; }
|
||||
/// <summary>
|
||||
/// Optional fixed bearer token for constrained environments (e.g., short-lived offline bundles).
|
||||
/// </summary>
|
||||
public string? StaticAccessToken { get; set; }
|
||||
/// <summary>
|
||||
/// Minimum buffer (seconds) subtracted from token expiry before refresh.
|
||||
/// </summary>
|
||||
public int ExpiryLeewaySeconds { get; set; } = 60;
|
||||
|
||||
public void Validate(IFileSystem? fileSystem = null)
|
||||
{
|
||||
if (PreferOfflineToken)
|
||||
{
|
||||
if (string.IsNullOrWhiteSpace(OfflineTokenPath) && string.IsNullOrWhiteSpace(StaticAccessToken))
|
||||
{
|
||||
throw new InvalidOperationException("OfflineTokenPath or StaticAccessToken must be provided when PreferOfflineToken is enabled.");
|
||||
}
|
||||
}
|
||||
else
|
||||
{
|
||||
if (string.IsNullOrWhiteSpace(TenantId))
|
||||
{
|
||||
throw new InvalidOperationException("TenantId is required when not operating in offline token mode.");
|
||||
}
|
||||
|
||||
if (string.IsNullOrWhiteSpace(ClientId))
|
||||
{
|
||||
throw new InvalidOperationException("ClientId is required when not operating in offline token mode.");
|
||||
}
|
||||
|
||||
if (string.IsNullOrWhiteSpace(ClientSecret))
|
||||
{
|
||||
throw new InvalidOperationException("ClientSecret is required when not operating in offline token mode.");
|
||||
}
|
||||
}
|
||||
|
||||
if (string.IsNullOrWhiteSpace(Scope))
|
||||
{
|
||||
Scope = DefaultScope;
|
||||
}
|
||||
|
||||
if (ExpiryLeewaySeconds < 10)
|
||||
{
|
||||
ExpiryLeewaySeconds = 10;
|
||||
}
|
||||
|
||||
if (!string.IsNullOrWhiteSpace(OfflineTokenPath))
|
||||
{
|
||||
var fs = fileSystem ?? new FileSystem();
|
||||
var directory = Path.GetDirectoryName(OfflineTokenPath);
|
||||
if (!string.IsNullOrWhiteSpace(directory) && !fs.Directory.Exists(directory))
|
||||
{
|
||||
fs.Directory.CreateDirectory(directory);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,40 @@
|
||||
using System;
|
||||
using System.Net;
|
||||
using System.Net.Http;
|
||||
using Microsoft.Extensions.Caching.Memory;
|
||||
using Microsoft.Extensions.DependencyInjection;
|
||||
using Microsoft.Extensions.DependencyInjection.Extensions;
|
||||
using StellaOps.Vexer.Connectors.MSRC.CSAF.Authentication;
|
||||
using StellaOps.Vexer.Connectors.MSRC.CSAF.Configuration;
|
||||
using System.IO.Abstractions;
|
||||
|
||||
namespace StellaOps.Vexer.Connectors.MSRC.CSAF.DependencyInjection;
|
||||
|
||||
public static class MsrcConnectorServiceCollectionExtensions
|
||||
{
|
||||
public static IServiceCollection AddMsrcCsafConnector(this IServiceCollection services, Action<MsrcConnectorOptions>? configure = null)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(services);
|
||||
|
||||
services.TryAddSingleton<IMemoryCache, MemoryCache>();
|
||||
services.TryAddSingleton<IFileSystem, FileSystem>();
|
||||
|
||||
services.AddOptions<MsrcConnectorOptions>()
|
||||
.Configure(options => configure?.Invoke(options));
|
||||
|
||||
services.AddHttpClient(MsrcConnectorOptions.TokenClientName, client =>
|
||||
{
|
||||
client.Timeout = TimeSpan.FromSeconds(30);
|
||||
client.DefaultRequestHeaders.UserAgent.ParseAdd("StellaOps.Vexer.Connectors.MSRC.CSAF/1.0");
|
||||
client.DefaultRequestHeaders.Accept.ParseAdd("application/json");
|
||||
})
|
||||
.ConfigurePrimaryHttpMessageHandler(static () => new HttpClientHandler
|
||||
{
|
||||
AutomaticDecompression = DecompressionMethods.All,
|
||||
});
|
||||
|
||||
services.AddSingleton<IMsrcTokenProvider, MsrcTokenProvider>();
|
||||
|
||||
return services;
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,18 @@
|
||||
<Project Sdk="Microsoft.NET.Sdk">
|
||||
<PropertyGroup>
|
||||
<TargetFramework>net10.0</TargetFramework>
|
||||
<LangVersion>preview</LangVersion>
|
||||
<Nullable>enable</Nullable>
|
||||
<ImplicitUsings>enable</ImplicitUsings>
|
||||
<TreatWarningsAsErrors>true</TreatWarningsAsErrors>
|
||||
</PropertyGroup>
|
||||
<ItemGroup>
|
||||
<ProjectReference Include="..\StellaOps.Vexer.Connectors.Abstractions\StellaOps.Vexer.Connectors.Abstractions.csproj" />
|
||||
</ItemGroup>
|
||||
<ItemGroup>
|
||||
<PackageReference Include="Microsoft.Extensions.Caching.Memory" Version="8.0.0" />
|
||||
<PackageReference Include="Microsoft.Extensions.Http" Version="8.0.0" />
|
||||
<PackageReference Include="Microsoft.Extensions.Options" Version="8.0.0" />
|
||||
<PackageReference Include="System.IO.Abstractions" Version="20.0.28" />
|
||||
</ItemGroup>
|
||||
</Project>
|
||||
@@ -2,6 +2,6 @@ If you are working on this file you need to read docs/ARCHITECTURE_VEXER.md and
|
||||
# TASKS
|
||||
| Task | Owner(s) | Depends on | Notes |
|
||||
|---|---|---|---|
|
||||
|VEXER-CONN-MS-01-001 – AAD onboarding & token cache|Team Vexer Connectors – MSRC|VEXER-CONN-ABS-01-001|TODO – Implement Azure AD credential flow, token caching, and validation for MSRC CSAF access with offline fallback guidance.|
|
||||
|VEXER-CONN-MS-01-001 – AAD onboarding & token cache|Team Vexer Connectors – MSRC|VEXER-CONN-ABS-01-001|**DONE (2025-10-17)** – Added MSRC connector project with configurable AAD options, token provider (offline/online modes), DI wiring, and unit tests covering caching and fallback scenarios.|
|
||||
|VEXER-CONN-MS-01-002 – CSAF download pipeline|Team Vexer Connectors – MSRC|VEXER-CONN-MS-01-001, VEXER-STORAGE-01-003|TODO – Fetch CSAF packages with retry/backoff, checksum verification, and raw document persistence plus quarantine for schema failures.|
|
||||
|VEXER-CONN-MS-01-003 – Trust metadata & provenance hints|Team Vexer Connectors – MSRC|VEXER-CONN-MS-01-002, VEXER-POLICY-01-001|TODO – Emit cosign/AAD issuer metadata, attach provenance details, and document policy integration.|
|
||||
|
||||
@@ -0,0 +1,205 @@
|
||||
using System.Collections.Generic;
|
||||
using System.Net;
|
||||
using System.Net.Http;
|
||||
using System.Text;
|
||||
using FluentAssertions;
|
||||
using Microsoft.Extensions.Caching.Memory;
|
||||
using Microsoft.Extensions.Logging.Abstractions;
|
||||
using StellaOps.Vexer.Connectors.Oracle.CSAF.Configuration;
|
||||
using StellaOps.Vexer.Connectors.Oracle.CSAF.Metadata;
|
||||
using System.IO.Abstractions.TestingHelpers;
|
||||
using Xunit;
|
||||
using System.Threading;
|
||||
|
||||
namespace StellaOps.Vexer.Connectors.Oracle.CSAF.Tests.Metadata;
|
||||
|
||||
public sealed class OracleCatalogLoaderTests
|
||||
{
|
||||
private const string SampleCatalog = """
|
||||
{
|
||||
"generated": "2025-09-30T18:00:00Z",
|
||||
"catalog": [
|
||||
{
|
||||
"id": "CPU2025Oct",
|
||||
"title": "Oracle Critical Patch Update Advisory - October 2025",
|
||||
"published": "2025-10-15T00:00:00Z",
|
||||
"revision": "2025-10-15T00:00:00Z",
|
||||
"document": {
|
||||
"url": "https://updates.oracle.com/cpu/2025-10/cpu2025oct.json",
|
||||
"sha256": "abc123",
|
||||
"size": 1024
|
||||
},
|
||||
"products": ["Oracle Database", "Java SE"]
|
||||
}
|
||||
],
|
||||
"schedule": [
|
||||
{ "window": "2025-Oct", "releaseDate": "2025-10-15T00:00:00Z" }
|
||||
]
|
||||
}
|
||||
""";
|
||||
|
||||
private const string SampleCalendar = """
|
||||
{
|
||||
"cpuWindows": [
|
||||
{ "name": "2026-Jan", "releaseDate": "2026-01-21T00:00:00Z" }
|
||||
]
|
||||
}
|
||||
""";
|
||||
|
||||
[Fact]
|
||||
public async Task LoadAsync_FetchesAndCachesCatalog()
|
||||
{
|
||||
var handler = new TestHttpMessageHandler(new Dictionary<Uri, HttpResponseMessage>
|
||||
{
|
||||
[new Uri("https://www.oracle.com/security-alerts/cpu/csaf/catalog.json")] = CreateResponse(SampleCatalog),
|
||||
[new Uri("https://www.oracle.com/security-alerts/cpu/cal.json")] = CreateResponse(SampleCalendar),
|
||||
});
|
||||
var client = new HttpClient(handler);
|
||||
var factory = new SingleClientHttpClientFactory(client);
|
||||
var cache = new MemoryCache(new MemoryCacheOptions());
|
||||
var fileSystem = new MockFileSystem();
|
||||
var loader = new OracleCatalogLoader(factory, cache, fileSystem, NullLogger<OracleCatalogLoader>.Instance, new AdjustableTimeProvider());
|
||||
|
||||
var options = new OracleConnectorOptions
|
||||
{
|
||||
CatalogUri = new Uri("https://www.oracle.com/security-alerts/cpu/csaf/catalog.json"),
|
||||
CpuCalendarUri = new Uri("https://www.oracle.com/security-alerts/cpu/cal.json"),
|
||||
OfflineSnapshotPath = "/snapshots/oracle-catalog.json",
|
||||
};
|
||||
|
||||
var result = await loader.LoadAsync(options, CancellationToken.None);
|
||||
result.Metadata.Entries.Should().HaveCount(1);
|
||||
result.Metadata.CpuSchedule.Should().Contain(r => r.Window == "2025-Oct");
|
||||
result.Metadata.CpuSchedule.Should().Contain(r => r.Window == "2026-Jan");
|
||||
result.FromCache.Should().BeFalse();
|
||||
fileSystem.FileExists(options.OfflineSnapshotPath!).Should().BeTrue();
|
||||
|
||||
handler.ResetInvocationCount();
|
||||
var cached = await loader.LoadAsync(options, CancellationToken.None);
|
||||
cached.FromCache.Should().BeTrue();
|
||||
handler.InvocationCount.Should().Be(0);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task LoadAsync_UsesOfflineSnapshotWhenNetworkFails()
|
||||
{
|
||||
var handler = new TestHttpMessageHandler(new Dictionary<Uri, HttpResponseMessage>());
|
||||
var client = new HttpClient(handler);
|
||||
var factory = new SingleClientHttpClientFactory(client);
|
||||
var cache = new MemoryCache(new MemoryCacheOptions());
|
||||
var fileSystem = new MockFileSystem();
|
||||
var offlineJson = """
|
||||
{
|
||||
"metadata": {
|
||||
"generatedAt": "2025-09-30T18:00:00Z",
|
||||
"entries": [
|
||||
{
|
||||
"id": "CPU2025Oct",
|
||||
"title": "Oracle Critical Patch Update Advisory - October 2025",
|
||||
"documentUri": "https://updates.oracle.com/cpu/2025-10/cpu2025oct.json",
|
||||
"publishedAt": "2025-10-15T00:00:00Z",
|
||||
"revision": "2025-10-15T00:00:00Z",
|
||||
"sha256": "abc123",
|
||||
"size": 1024,
|
||||
"products": [ "Oracle Database" ]
|
||||
}
|
||||
],
|
||||
"cpuSchedule": [
|
||||
{ "window": "2025-Oct", "releaseDate": "2025-10-15T00:00:00Z" }
|
||||
]
|
||||
},
|
||||
"fetchedAt": "2025-10-01T00:00:00Z"
|
||||
}
|
||||
""";
|
||||
fileSystem.AddFile("/snapshots/oracle-catalog.json", new MockFileData(offlineJson));
|
||||
var loader = new OracleCatalogLoader(factory, cache, fileSystem, NullLogger<OracleCatalogLoader>.Instance, new AdjustableTimeProvider());
|
||||
|
||||
var options = new OracleConnectorOptions
|
||||
{
|
||||
CatalogUri = new Uri("https://www.oracle.com/security-alerts/cpu/csaf/catalog.json"),
|
||||
OfflineSnapshotPath = "/snapshots/oracle-catalog.json",
|
||||
PreferOfflineSnapshot = true,
|
||||
};
|
||||
|
||||
var result = await loader.LoadAsync(options, CancellationToken.None);
|
||||
result.FromOfflineSnapshot.Should().BeTrue();
|
||||
result.Metadata.Entries.Should().NotBeEmpty();
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task LoadAsync_ThrowsWhenOfflinePreferredButMissing()
|
||||
{
|
||||
var handler = new TestHttpMessageHandler(new Dictionary<Uri, HttpResponseMessage>());
|
||||
var client = new HttpClient(handler);
|
||||
var factory = new SingleClientHttpClientFactory(client);
|
||||
var cache = new MemoryCache(new MemoryCacheOptions());
|
||||
var fileSystem = new MockFileSystem();
|
||||
var loader = new OracleCatalogLoader(factory, cache, fileSystem, NullLogger<OracleCatalogLoader>.Instance);
|
||||
|
||||
var options = new OracleConnectorOptions
|
||||
{
|
||||
CatalogUri = new Uri("https://www.oracle.com/security-alerts/cpu/csaf/catalog.json"),
|
||||
PreferOfflineSnapshot = true,
|
||||
OfflineSnapshotPath = "/missing/oracle-catalog.json",
|
||||
};
|
||||
|
||||
await Assert.ThrowsAsync<InvalidOperationException>(() => loader.LoadAsync(options, CancellationToken.None));
|
||||
}
|
||||
|
||||
private static HttpResponseMessage CreateResponse(string payload)
|
||||
=> new(HttpStatusCode.OK)
|
||||
{
|
||||
Content = new StringContent(payload, Encoding.UTF8, "application/json"),
|
||||
};
|
||||
|
||||
private sealed class SingleClientHttpClientFactory : IHttpClientFactory
|
||||
{
|
||||
private readonly HttpClient _client;
|
||||
|
||||
public SingleClientHttpClientFactory(HttpClient client)
|
||||
{
|
||||
_client = client;
|
||||
}
|
||||
|
||||
public HttpClient CreateClient(string name) => _client;
|
||||
}
|
||||
|
||||
private sealed class AdjustableTimeProvider : TimeProvider
|
||||
{
|
||||
private DateTimeOffset _now = DateTimeOffset.UtcNow;
|
||||
|
||||
public override DateTimeOffset GetUtcNow() => _now;
|
||||
}
|
||||
|
||||
private sealed class TestHttpMessageHandler : HttpMessageHandler
|
||||
{
|
||||
private readonly Dictionary<Uri, HttpResponseMessage> _responses;
|
||||
|
||||
public TestHttpMessageHandler(Dictionary<Uri, HttpResponseMessage> responses)
|
||||
{
|
||||
_responses = responses;
|
||||
}
|
||||
|
||||
public int InvocationCount { get; private set; }
|
||||
|
||||
public void ResetInvocationCount() => InvocationCount = 0;
|
||||
|
||||
protected override async Task<HttpResponseMessage> SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
|
||||
{
|
||||
InvocationCount++;
|
||||
if (request.RequestUri is not null && _responses.TryGetValue(request.RequestUri, out var response))
|
||||
{
|
||||
var payload = await response.Content.ReadAsStringAsync(cancellationToken).ConfigureAwait(false);
|
||||
return new HttpResponseMessage(response.StatusCode)
|
||||
{
|
||||
Content = new StringContent(payload, Encoding.UTF8, "application/json"),
|
||||
};
|
||||
}
|
||||
|
||||
return new HttpResponseMessage(HttpStatusCode.InternalServerError)
|
||||
{
|
||||
Content = new StringContent("unexpected request"),
|
||||
};
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,17 @@
|
||||
<Project Sdk="Microsoft.NET.Sdk">
|
||||
<PropertyGroup>
|
||||
<TargetFramework>net10.0</TargetFramework>
|
||||
<LangVersion>preview</LangVersion>
|
||||
<Nullable>enable</Nullable>
|
||||
<ImplicitUsings>enable</ImplicitUsings>
|
||||
<TreatWarningsAsErrors>true</TreatWarningsAsErrors>
|
||||
</PropertyGroup>
|
||||
<ItemGroup>
|
||||
<ProjectReference Include="..\StellaOps.Vexer.Connectors.Oracle.CSAF\StellaOps.Vexer.Connectors.Oracle.CSAF.csproj" />
|
||||
</ItemGroup>
|
||||
<ItemGroup>
|
||||
<PackageReference Include="FluentAssertions" Version="6.12.0" />
|
||||
<PackageReference Include="Microsoft.Extensions.Logging.Abstractions" Version="8.0.0" />
|
||||
<PackageReference Include="System.IO.Abstractions.TestingHelpers" Version="20.0.28" />
|
||||
</ItemGroup>
|
||||
</Project>
|
||||
@@ -0,0 +1,85 @@
|
||||
using System;
|
||||
using System.IO;
|
||||
using System.IO.Abstractions;
|
||||
|
||||
namespace StellaOps.Vexer.Connectors.Oracle.CSAF.Configuration;
|
||||
|
||||
public sealed class OracleConnectorOptions
|
||||
{
|
||||
public const string HttpClientName = "vexer.connector.oracle.catalog";
|
||||
|
||||
/// <summary>
|
||||
/// Oracle CSAF catalog endpoint hosting advisory metadata.
|
||||
/// </summary>
|
||||
public Uri CatalogUri { get; set; } = new("https://www.oracle.com/security-alerts/cpu/csaf/catalog.json");
|
||||
|
||||
/// <summary>
|
||||
/// Optional CPU calendar endpoint providing upcoming release dates.
|
||||
/// </summary>
|
||||
public Uri? CpuCalendarUri { get; set; }
|
||||
/// <summary>
|
||||
/// Duration the discovery metadata should be cached before refresh.
|
||||
/// </summary>
|
||||
public TimeSpan MetadataCacheDuration { get; set; } = TimeSpan.FromHours(6);
|
||||
|
||||
/// <summary>
|
||||
/// When true, the loader will prefer offline snapshot data over network fetches.
|
||||
/// </summary>
|
||||
public bool PreferOfflineSnapshot { get; set; }
|
||||
/// <summary>
|
||||
/// Optional file path for persisting or ingesting catalog snapshots.
|
||||
/// </summary>
|
||||
public string? OfflineSnapshotPath { get; set; }
|
||||
/// <summary>
|
||||
/// Enables writing fresh catalog responses to <see cref="OfflineSnapshotPath"/>.
|
||||
/// </summary>
|
||||
public bool PersistOfflineSnapshot { get; set; } = true;
|
||||
|
||||
/// <summary>
|
||||
/// Optional request delay when iterating catalogue entries (for rate limiting).
|
||||
/// </summary>
|
||||
public TimeSpan RequestDelay { get; set; } = TimeSpan.FromMilliseconds(250);
|
||||
|
||||
public void Validate(IFileSystem? fileSystem = null)
|
||||
{
|
||||
if (CatalogUri is null || !CatalogUri.IsAbsoluteUri)
|
||||
{
|
||||
throw new InvalidOperationException("CatalogUri must be an absolute URI.");
|
||||
}
|
||||
|
||||
if (CatalogUri.Scheme is not ("http" or "https"))
|
||||
{
|
||||
throw new InvalidOperationException("CatalogUri must use HTTP or HTTPS.");
|
||||
}
|
||||
|
||||
if (CpuCalendarUri is not null && (!CpuCalendarUri.IsAbsoluteUri || CpuCalendarUri.Scheme is not ("http" or "https")))
|
||||
{
|
||||
throw new InvalidOperationException("CpuCalendarUri must be an absolute HTTP(S) URI when provided.");
|
||||
}
|
||||
|
||||
if (MetadataCacheDuration <= TimeSpan.Zero)
|
||||
{
|
||||
throw new InvalidOperationException("MetadataCacheDuration must be positive.");
|
||||
}
|
||||
|
||||
if (RequestDelay < TimeSpan.Zero)
|
||||
{
|
||||
throw new InvalidOperationException("RequestDelay cannot be negative.");
|
||||
}
|
||||
|
||||
if (PreferOfflineSnapshot && string.IsNullOrWhiteSpace(OfflineSnapshotPath))
|
||||
{
|
||||
throw new InvalidOperationException("OfflineSnapshotPath must be provided when PreferOfflineSnapshot is enabled.");
|
||||
}
|
||||
|
||||
if (!string.IsNullOrWhiteSpace(OfflineSnapshotPath))
|
||||
{
|
||||
var fs = fileSystem ?? new FileSystem();
|
||||
var directory = Path.GetDirectoryName(OfflineSnapshotPath);
|
||||
if (!string.IsNullOrWhiteSpace(directory) && !fs.Directory.Exists(directory))
|
||||
{
|
||||
fs.Directory.CreateDirectory(directory);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,32 @@
|
||||
using System;
|
||||
using System.Collections.Generic;
|
||||
using System.IO.Abstractions;
|
||||
using StellaOps.Vexer.Connectors.Abstractions;
|
||||
|
||||
namespace StellaOps.Vexer.Connectors.Oracle.CSAF.Configuration;
|
||||
|
||||
public sealed class OracleConnectorOptionsValidator : IVexConnectorOptionsValidator<OracleConnectorOptions>
|
||||
{
|
||||
private readonly IFileSystem _fileSystem;
|
||||
|
||||
public OracleConnectorOptionsValidator(IFileSystem fileSystem)
|
||||
{
|
||||
_fileSystem = fileSystem ?? throw new ArgumentNullException(nameof(fileSystem));
|
||||
}
|
||||
|
||||
public void Validate(VexConnectorDescriptor descriptor, OracleConnectorOptions options, IList<string> errors)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(descriptor);
|
||||
ArgumentNullException.ThrowIfNull(options);
|
||||
ArgumentNullException.ThrowIfNull(errors);
|
||||
|
||||
try
|
||||
{
|
||||
options.Validate(_fileSystem);
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
errors.Add(ex.Message);
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,45 @@
|
||||
using System;
|
||||
using System.Net;
|
||||
using System.Net.Http;
|
||||
using Microsoft.Extensions.Caching.Memory;
|
||||
using Microsoft.Extensions.DependencyInjection;
|
||||
using Microsoft.Extensions.DependencyInjection.Extensions;
|
||||
using StellaOps.Vexer.Connectors.Abstractions;
|
||||
using StellaOps.Vexer.Connectors.Oracle.CSAF.Configuration;
|
||||
using StellaOps.Vexer.Connectors.Oracle.CSAF.Metadata;
|
||||
using StellaOps.Vexer.Core;
|
||||
using System.IO.Abstractions;
|
||||
|
||||
namespace StellaOps.Vexer.Connectors.Oracle.CSAF.DependencyInjection;
|
||||
|
||||
public static class OracleConnectorServiceCollectionExtensions
|
||||
{
|
||||
public static IServiceCollection AddOracleCsafConnector(this IServiceCollection services, Action<OracleConnectorOptions>? configure = null)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(services);
|
||||
|
||||
services.TryAddSingleton<IMemoryCache, MemoryCache>();
|
||||
services.TryAddSingleton<IFileSystem, FileSystem>();
|
||||
|
||||
services.AddOptions<OracleConnectorOptions>()
|
||||
.Configure(options => configure?.Invoke(options));
|
||||
|
||||
services.AddSingleton<IVexConnectorOptionsValidator<OracleConnectorOptions>, OracleConnectorOptionsValidator>();
|
||||
|
||||
services.AddHttpClient(OracleConnectorOptions.HttpClientName, client =>
|
||||
{
|
||||
client.Timeout = TimeSpan.FromSeconds(60);
|
||||
client.DefaultRequestHeaders.UserAgent.ParseAdd("StellaOps.Vexer.Connectors.Oracle.CSAF/1.0");
|
||||
client.DefaultRequestHeaders.Accept.ParseAdd("application/json");
|
||||
})
|
||||
.ConfigurePrimaryHttpMessageHandler(static () => new HttpClientHandler
|
||||
{
|
||||
AutomaticDecompression = DecompressionMethods.All,
|
||||
});
|
||||
|
||||
services.AddSingleton<OracleCatalogLoader>();
|
||||
services.AddSingleton<IVexConnector, OracleCsafConnector>();
|
||||
|
||||
return services;
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,418 @@
|
||||
using System;
|
||||
using System.Collections.Generic;
|
||||
using System.Collections.Immutable;
|
||||
using System.IO.Abstractions;
|
||||
using System.Net;
|
||||
using System.Net.Http;
|
||||
using System.Text.Json;
|
||||
using System.Threading;
|
||||
using System.Threading.Tasks;
|
||||
using Microsoft.Extensions.Caching.Memory;
|
||||
using Microsoft.Extensions.Logging;
|
||||
using StellaOps.Vexer.Connectors.Oracle.CSAF.Configuration;
|
||||
|
||||
namespace StellaOps.Vexer.Connectors.Oracle.CSAF.Metadata;
|
||||
|
||||
public sealed class OracleCatalogLoader
|
||||
{
|
||||
public const string CachePrefix = "StellaOps.Vexer.Connectors.Oracle.CSAF.Catalog";
|
||||
|
||||
private readonly IHttpClientFactory _httpClientFactory;
|
||||
private readonly IMemoryCache _memoryCache;
|
||||
private readonly IFileSystem _fileSystem;
|
||||
private readonly ILogger<OracleCatalogLoader> _logger;
|
||||
private readonly TimeProvider _timeProvider;
|
||||
private readonly SemaphoreSlim _semaphore = new(1, 1);
|
||||
private readonly JsonSerializerOptions _serializerOptions = new(JsonSerializerDefaults.Web);
|
||||
|
||||
public OracleCatalogLoader(
|
||||
IHttpClientFactory httpClientFactory,
|
||||
IMemoryCache memoryCache,
|
||||
IFileSystem fileSystem,
|
||||
ILogger<OracleCatalogLoader> logger,
|
||||
TimeProvider? timeProvider = null)
|
||||
{
|
||||
_httpClientFactory = httpClientFactory ?? throw new ArgumentNullException(nameof(httpClientFactory));
|
||||
_memoryCache = memoryCache ?? throw new ArgumentNullException(nameof(memoryCache));
|
||||
_fileSystem = fileSystem ?? throw new ArgumentNullException(nameof(fileSystem));
|
||||
_logger = logger ?? throw new ArgumentNullException(nameof(logger));
|
||||
_timeProvider = timeProvider ?? TimeProvider.System;
|
||||
}
|
||||
|
||||
public async Task<OracleCatalogResult> LoadAsync(OracleConnectorOptions options, CancellationToken cancellationToken)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(options);
|
||||
options.Validate(_fileSystem);
|
||||
|
||||
var cacheKey = CreateCacheKey(options);
|
||||
if (_memoryCache.TryGetValue<CacheEntry>(cacheKey, out var cached) && cached is not null && !cached.IsExpired(_timeProvider.GetUtcNow()))
|
||||
{
|
||||
return cached.ToResult(fromCache: true);
|
||||
}
|
||||
|
||||
await _semaphore.WaitAsync(cancellationToken).ConfigureAwait(false);
|
||||
try
|
||||
{
|
||||
if (_memoryCache.TryGetValue<CacheEntry>(cacheKey, out cached) && cached is not null && !cached.IsExpired(_timeProvider.GetUtcNow()))
|
||||
{
|
||||
return cached.ToResult(fromCache: true);
|
||||
}
|
||||
|
||||
CacheEntry? entry = null;
|
||||
if (options.PreferOfflineSnapshot)
|
||||
{
|
||||
entry = LoadFromOffline(options);
|
||||
if (entry is null)
|
||||
{
|
||||
throw new InvalidOperationException("PreferOfflineSnapshot is enabled but no offline snapshot was found or could be loaded.");
|
||||
}
|
||||
}
|
||||
else
|
||||
{
|
||||
entry = await TryFetchFromNetworkAsync(options, cancellationToken).ConfigureAwait(false)
|
||||
?? LoadFromOffline(options);
|
||||
}
|
||||
|
||||
if (entry is null)
|
||||
{
|
||||
throw new InvalidOperationException("Unable to load Oracle CSAF catalog from network or offline snapshot.");
|
||||
}
|
||||
|
||||
var expiration = entry.MetadataCacheDuration == TimeSpan.Zero
|
||||
? (DateTimeOffset?)null
|
||||
: _timeProvider.GetUtcNow().Add(entry.MetadataCacheDuration);
|
||||
|
||||
var cacheEntryOptions = new MemoryCacheEntryOptions();
|
||||
if (expiration.HasValue)
|
||||
{
|
||||
cacheEntryOptions.AbsoluteExpiration = expiration.Value;
|
||||
}
|
||||
|
||||
_memoryCache.Set(cacheKey, entry with { CachedAt = _timeProvider.GetUtcNow() }, cacheEntryOptions);
|
||||
return entry.ToResult(fromCache: false);
|
||||
}
|
||||
finally
|
||||
{
|
||||
_semaphore.Release();
|
||||
}
|
||||
}
|
||||
|
||||
private async Task<CacheEntry?> TryFetchFromNetworkAsync(OracleConnectorOptions options, CancellationToken cancellationToken)
|
||||
{
|
||||
try
|
||||
{
|
||||
var client = _httpClientFactory.CreateClient(OracleConnectorOptions.HttpClientName);
|
||||
using var response = await client.GetAsync(options.CatalogUri, HttpCompletionOption.ResponseHeadersRead, cancellationToken).ConfigureAwait(false);
|
||||
response.EnsureSuccessStatusCode();
|
||||
var catalogPayload = await response.Content.ReadAsStringAsync(cancellationToken).ConfigureAwait(false);
|
||||
|
||||
string? calendarPayload = null;
|
||||
if (options.CpuCalendarUri is not null)
|
||||
{
|
||||
try
|
||||
{
|
||||
using var calendarResponse = await client.GetAsync(options.CpuCalendarUri, HttpCompletionOption.ResponseHeadersRead, cancellationToken).ConfigureAwait(false);
|
||||
calendarResponse.EnsureSuccessStatusCode();
|
||||
calendarPayload = await calendarResponse.Content.ReadAsStringAsync(cancellationToken).ConfigureAwait(false);
|
||||
}
|
||||
catch (Exception ex) when (ex is not OperationCanceledException)
|
||||
{
|
||||
_logger.LogWarning(ex, "Failed to fetch Oracle CPU calendar from {Uri}; continuing without schedule.", options.CpuCalendarUri);
|
||||
}
|
||||
}
|
||||
|
||||
var metadata = ParseMetadata(catalogPayload, calendarPayload);
|
||||
var fetchedAt = _timeProvider.GetUtcNow();
|
||||
var entry = new CacheEntry(metadata, fetchedAt, fetchedAt, options.MetadataCacheDuration, false);
|
||||
|
||||
PersistSnapshotIfNeeded(options, metadata, fetchedAt);
|
||||
return entry;
|
||||
}
|
||||
catch (Exception ex) when (ex is not OperationCanceledException)
|
||||
{
|
||||
_logger.LogWarning(ex, "Failed to fetch Oracle CSAF catalog from {Uri}; attempting offline fallback if available.", options.CatalogUri);
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
private CacheEntry? LoadFromOffline(OracleConnectorOptions options)
|
||||
{
|
||||
if (string.IsNullOrWhiteSpace(options.OfflineSnapshotPath))
|
||||
{
|
||||
return null;
|
||||
}
|
||||
|
||||
if (!_fileSystem.File.Exists(options.OfflineSnapshotPath))
|
||||
{
|
||||
_logger.LogWarning("Oracle offline snapshot path {Path} does not exist.", options.OfflineSnapshotPath);
|
||||
return null;
|
||||
}
|
||||
|
||||
try
|
||||
{
|
||||
var payload = _fileSystem.File.ReadAllText(options.OfflineSnapshotPath);
|
||||
var snapshot = JsonSerializer.Deserialize<OracleCatalogSnapshot>(payload, _serializerOptions);
|
||||
if (snapshot is null)
|
||||
{
|
||||
throw new InvalidOperationException("Offline snapshot payload was empty.");
|
||||
}
|
||||
|
||||
return new CacheEntry(snapshot.Metadata, snapshot.FetchedAt, _timeProvider.GetUtcNow(), options.MetadataCacheDuration, true);
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
_logger.LogError(ex, "Failed to load Oracle CSAF catalog from offline snapshot {Path}.", options.OfflineSnapshotPath);
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
private OracleCatalogMetadata ParseMetadata(string catalogPayload, string? calendarPayload)
|
||||
{
|
||||
if (string.IsNullOrWhiteSpace(catalogPayload))
|
||||
{
|
||||
throw new InvalidOperationException("Oracle catalog payload was empty.");
|
||||
}
|
||||
|
||||
using var document = JsonDocument.Parse(catalogPayload);
|
||||
var root = document.RootElement;
|
||||
|
||||
var generatedAt = root.TryGetProperty("generated", out var generatedElement) && generatedElement.ValueKind == JsonValueKind.String && DateTimeOffset.TryParse(generatedElement.GetString(), out var generated)
|
||||
? generated
|
||||
: _timeProvider.GetUtcNow();
|
||||
|
||||
var entries = ParseEntries(root);
|
||||
var schedule = ParseSchedule(root);
|
||||
|
||||
if (!string.IsNullOrWhiteSpace(calendarPayload))
|
||||
{
|
||||
schedule = MergeSchedule(schedule, calendarPayload);
|
||||
}
|
||||
|
||||
return new OracleCatalogMetadata(generatedAt, entries, schedule);
|
||||
}
|
||||
|
||||
private ImmutableArray<OracleCatalogEntry> ParseEntries(JsonElement root)
|
||||
{
|
||||
if (!root.TryGetProperty("catalog", out var catalogElement) || catalogElement.ValueKind is not JsonValueKind.Array)
|
||||
{
|
||||
return ImmutableArray<OracleCatalogEntry>.Empty;
|
||||
}
|
||||
|
||||
var builder = ImmutableArray.CreateBuilder<OracleCatalogEntry>();
|
||||
foreach (var entry in catalogElement.EnumerateArray())
|
||||
{
|
||||
var id = entry.TryGetProperty("id", out var idElement) && idElement.ValueKind == JsonValueKind.String ? idElement.GetString() : null;
|
||||
var title = entry.TryGetProperty("title", out var titleElement) && titleElement.ValueKind == JsonValueKind.String ? titleElement.GetString() : null;
|
||||
if (string.IsNullOrWhiteSpace(id) || string.IsNullOrWhiteSpace(title))
|
||||
{
|
||||
continue;
|
||||
}
|
||||
|
||||
DateTimeOffset publishedAt = default;
|
||||
if (entry.TryGetProperty("published", out var publishedElement) && publishedElement.ValueKind == JsonValueKind.String && DateTimeOffset.TryParse(publishedElement.GetString(), out var publishedParsed))
|
||||
{
|
||||
publishedAt = publishedParsed;
|
||||
}
|
||||
|
||||
string? revision = null;
|
||||
if (entry.TryGetProperty("revision", out var revisionElement) && revisionElement.ValueKind == JsonValueKind.String)
|
||||
{
|
||||
revision = revisionElement.GetString();
|
||||
}
|
||||
|
||||
ImmutableArray<string> products = ImmutableArray<string>.Empty;
|
||||
if (entry.TryGetProperty("products", out var productsElement))
|
||||
{
|
||||
products = ParseStringArray(productsElement);
|
||||
}
|
||||
|
||||
Uri? documentUri = null;
|
||||
string? sha256 = null;
|
||||
long? size = null;
|
||||
|
||||
if (entry.TryGetProperty("document", out var documentElement) && documentElement.ValueKind == JsonValueKind.Object)
|
||||
{
|
||||
if (documentElement.TryGetProperty("url", out var urlElement) && urlElement.ValueKind == JsonValueKind.String && Uri.TryCreate(urlElement.GetString(), UriKind.Absolute, out var parsedUri))
|
||||
{
|
||||
documentUri = parsedUri;
|
||||
}
|
||||
|
||||
if (documentElement.TryGetProperty("sha256", out var hashElement) && hashElement.ValueKind == JsonValueKind.String)
|
||||
{
|
||||
sha256 = hashElement.GetString();
|
||||
}
|
||||
|
||||
if (documentElement.TryGetProperty("size", out var sizeElement) && sizeElement.ValueKind == JsonValueKind.Number && sizeElement.TryGetInt64(out var parsedSize))
|
||||
{
|
||||
size = parsedSize;
|
||||
}
|
||||
}
|
||||
|
||||
if (documentUri is null)
|
||||
{
|
||||
continue;
|
||||
}
|
||||
|
||||
builder.Add(new OracleCatalogEntry(id!, title!, documentUri, publishedAt, revision, sha256, size, products));
|
||||
}
|
||||
|
||||
return builder.ToImmutable();
|
||||
}
|
||||
|
||||
private ImmutableArray<OracleCpuRelease> ParseSchedule(JsonElement root)
|
||||
{
|
||||
if (!root.TryGetProperty("schedule", out var scheduleElement) || scheduleElement.ValueKind is not JsonValueKind.Array)
|
||||
{
|
||||
return ImmutableArray<OracleCpuRelease>.Empty;
|
||||
}
|
||||
|
||||
var builder = ImmutableArray.CreateBuilder<OracleCpuRelease>();
|
||||
foreach (var item in scheduleElement.EnumerateArray())
|
||||
{
|
||||
var window = item.TryGetProperty("window", out var windowElement) && windowElement.ValueKind == JsonValueKind.String ? windowElement.GetString() : null;
|
||||
if (string.IsNullOrWhiteSpace(window))
|
||||
{
|
||||
continue;
|
||||
}
|
||||
|
||||
DateTimeOffset releaseDate = default;
|
||||
if (item.TryGetProperty("releaseDate", out var releaseElement) && releaseElement.ValueKind == JsonValueKind.String && DateTimeOffset.TryParse(releaseElement.GetString(), out var parsed))
|
||||
{
|
||||
releaseDate = parsed;
|
||||
}
|
||||
|
||||
builder.Add(new OracleCpuRelease(window!, releaseDate));
|
||||
}
|
||||
|
||||
return builder.ToImmutable();
|
||||
}
|
||||
|
||||
private ImmutableArray<OracleCpuRelease> MergeSchedule(ImmutableArray<OracleCpuRelease> existing, string calendarPayload)
|
||||
{
|
||||
try
|
||||
{
|
||||
using var document = JsonDocument.Parse(calendarPayload);
|
||||
var root = document.RootElement;
|
||||
if (!root.TryGetProperty("cpuWindows", out var windowsElement) || windowsElement.ValueKind is not JsonValueKind.Array)
|
||||
{
|
||||
return existing;
|
||||
}
|
||||
|
||||
var builder = existing.ToBuilder();
|
||||
var known = new HashSet<string>(StringComparer.OrdinalIgnoreCase);
|
||||
foreach (var item in builder)
|
||||
{
|
||||
known.Add(item.Window);
|
||||
}
|
||||
|
||||
foreach (var windowElement in windowsElement.EnumerateArray())
|
||||
{
|
||||
var name = windowElement.TryGetProperty("name", out var nameElement) && nameElement.ValueKind == JsonValueKind.String ? nameElement.GetString() : null;
|
||||
if (string.IsNullOrWhiteSpace(name))
|
||||
{
|
||||
continue;
|
||||
}
|
||||
|
||||
if (!known.Add(name))
|
||||
{
|
||||
continue;
|
||||
}
|
||||
|
||||
DateTimeOffset releaseDate = default;
|
||||
if (windowElement.TryGetProperty("releaseDate", out var releaseElement) && releaseElement.ValueKind == JsonValueKind.String && DateTimeOffset.TryParse(releaseElement.GetString(), out var parsed))
|
||||
{
|
||||
releaseDate = parsed;
|
||||
}
|
||||
|
||||
builder.Add(new OracleCpuRelease(name!, releaseDate));
|
||||
}
|
||||
|
||||
return builder.ToImmutable();
|
||||
}
|
||||
catch (Exception ex) when (ex is not OperationCanceledException)
|
||||
{
|
||||
_logger.LogWarning(ex, "Failed to parse Oracle CPU calendar payload; continuing with existing schedule data.");
|
||||
return existing;
|
||||
}
|
||||
}
|
||||
|
||||
private ImmutableArray<string> ParseStringArray(JsonElement element)
|
||||
{
|
||||
if (element.ValueKind is not JsonValueKind.Array)
|
||||
{
|
||||
return ImmutableArray<string>.Empty;
|
||||
}
|
||||
|
||||
var builder = ImmutableArray.CreateBuilder<string>();
|
||||
foreach (var item in element.EnumerateArray())
|
||||
{
|
||||
if (item.ValueKind == JsonValueKind.String)
|
||||
{
|
||||
var value = item.GetString();
|
||||
if (!string.IsNullOrWhiteSpace(value))
|
||||
{
|
||||
builder.Add(value);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
return builder.ToImmutable();
|
||||
}
|
||||
|
||||
private void PersistSnapshotIfNeeded(OracleConnectorOptions options, OracleCatalogMetadata metadata, DateTimeOffset fetchedAt)
|
||||
{
|
||||
if (!options.PersistOfflineSnapshot || string.IsNullOrWhiteSpace(options.OfflineSnapshotPath))
|
||||
{
|
||||
return;
|
||||
}
|
||||
|
||||
try
|
||||
{
|
||||
var snapshot = new OracleCatalogSnapshot(metadata, fetchedAt);
|
||||
var payload = JsonSerializer.Serialize(snapshot, _serializerOptions);
|
||||
_fileSystem.File.WriteAllText(options.OfflineSnapshotPath, payload);
|
||||
_logger.LogDebug("Persisted Oracle CSAF catalog snapshot to {Path}.", options.OfflineSnapshotPath);
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
_logger.LogWarning(ex, "Failed to persist Oracle CSAF catalog snapshot to {Path}.", options.OfflineSnapshotPath);
|
||||
}
|
||||
}
|
||||
|
||||
private static string CreateCacheKey(OracleConnectorOptions options)
|
||||
=> $"{CachePrefix}:{options.CatalogUri}:{options.CpuCalendarUri}";
|
||||
|
||||
private sealed record CacheEntry(OracleCatalogMetadata Metadata, DateTimeOffset FetchedAt, DateTimeOffset CachedAt, TimeSpan MetadataCacheDuration, bool FromOfflineSnapshot)
|
||||
{
|
||||
public bool IsExpired(DateTimeOffset now)
|
||||
=> MetadataCacheDuration > TimeSpan.Zero && now >= CachedAt + MetadataCacheDuration;
|
||||
|
||||
public OracleCatalogResult ToResult(bool fromCache)
|
||||
=> new(Metadata, FetchedAt, fromCache, FromOfflineSnapshot);
|
||||
}
|
||||
|
||||
private sealed record OracleCatalogSnapshot(OracleCatalogMetadata Metadata, DateTimeOffset FetchedAt);
|
||||
}
|
||||
|
||||
public sealed record OracleCatalogMetadata(
|
||||
DateTimeOffset GeneratedAt,
|
||||
ImmutableArray<OracleCatalogEntry> Entries,
|
||||
ImmutableArray<OracleCpuRelease> CpuSchedule);
|
||||
|
||||
public sealed record OracleCatalogEntry(
|
||||
string Id,
|
||||
string Title,
|
||||
Uri DocumentUri,
|
||||
DateTimeOffset PublishedAt,
|
||||
string? Revision,
|
||||
string? Sha256,
|
||||
long? Size,
|
||||
ImmutableArray<string> Products);
|
||||
|
||||
public sealed record OracleCpuRelease(string Window, DateTimeOffset ReleaseDate);
|
||||
|
||||
public sealed record OracleCatalogResult(
|
||||
OracleCatalogMetadata Metadata,
|
||||
DateTimeOffset FetchedAt,
|
||||
bool FromCache,
|
||||
bool FromOfflineSnapshot);
|
||||
@@ -0,0 +1,81 @@
|
||||
using System.Collections.Generic;
|
||||
using System.Collections.Immutable;
|
||||
using System.Runtime.CompilerServices;
|
||||
using Microsoft.Extensions.Logging;
|
||||
using StellaOps.Vexer.Connectors.Abstractions;
|
||||
using StellaOps.Vexer.Connectors.Oracle.CSAF.Configuration;
|
||||
using StellaOps.Vexer.Connectors.Oracle.CSAF.Metadata;
|
||||
using StellaOps.Vexer.Core;
|
||||
|
||||
namespace StellaOps.Vexer.Connectors.Oracle.CSAF;
|
||||
|
||||
public sealed class OracleCsafConnector : VexConnectorBase
|
||||
{
|
||||
private static readonly VexConnectorDescriptor DescriptorInstance = new(
|
||||
id: "vexer:oracle",
|
||||
kind: VexProviderKind.Vendor,
|
||||
displayName: "Oracle CSAF")
|
||||
{
|
||||
Tags = ImmutableArray.Create("oracle", "csaf", "cpu"),
|
||||
};
|
||||
|
||||
private readonly OracleCatalogLoader _catalogLoader;
|
||||
private readonly IEnumerable<IVexConnectorOptionsValidator<OracleConnectorOptions>> _validators;
|
||||
|
||||
private OracleConnectorOptions? _options;
|
||||
private OracleCatalogResult? _catalog;
|
||||
|
||||
public OracleCsafConnector(
|
||||
OracleCatalogLoader catalogLoader,
|
||||
IEnumerable<IVexConnectorOptionsValidator<OracleConnectorOptions>> validators,
|
||||
ILogger<OracleCsafConnector> logger,
|
||||
TimeProvider timeProvider)
|
||||
: base(DescriptorInstance, logger, timeProvider)
|
||||
{
|
||||
_catalogLoader = catalogLoader ?? throw new ArgumentNullException(nameof(catalogLoader));
|
||||
_validators = validators ?? Array.Empty<IVexConnectorOptionsValidator<OracleConnectorOptions>>();
|
||||
}
|
||||
|
||||
public override async ValueTask ValidateAsync(VexConnectorSettings settings, CancellationToken cancellationToken)
|
||||
{
|
||||
_options = VexConnectorOptionsBinder.Bind(
|
||||
Descriptor,
|
||||
settings,
|
||||
validators: _validators);
|
||||
|
||||
_catalog = await _catalogLoader.LoadAsync(_options, cancellationToken).ConfigureAwait(false);
|
||||
LogConnectorEvent(LogLevel.Information, "validate", "Oracle CSAF catalogue loaded.", new Dictionary<string, object?>
|
||||
{
|
||||
["catalogEntryCount"] = _catalog.Metadata.Entries.Length,
|
||||
["scheduleCount"] = _catalog.Metadata.CpuSchedule.Length,
|
||||
["fromOffline"] = _catalog.FromOfflineSnapshot,
|
||||
});
|
||||
}
|
||||
|
||||
public override async IAsyncEnumerable<VexRawDocument> FetchAsync(VexConnectorContext context, [EnumeratorCancellation] CancellationToken cancellationToken)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(context);
|
||||
|
||||
if (_options is null)
|
||||
{
|
||||
throw new InvalidOperationException("Connector must be validated before fetch operations.");
|
||||
}
|
||||
|
||||
if (_catalog is null)
|
||||
{
|
||||
_catalog = await _catalogLoader.LoadAsync(_options, cancellationToken).ConfigureAwait(false);
|
||||
}
|
||||
|
||||
LogConnectorEvent(LogLevel.Debug, "fetch", "Oracle CSAF discovery ready; document ingestion handled by follow-up task.", new Dictionary<string, object?>
|
||||
{
|
||||
["since"] = context.Since?.ToString("O"),
|
||||
});
|
||||
|
||||
yield break;
|
||||
}
|
||||
|
||||
public override ValueTask<VexClaimBatch> NormalizeAsync(VexRawDocument document, CancellationToken cancellationToken)
|
||||
=> throw new NotSupportedException("OracleCsafConnector relies on dedicated CSAF normalizers.");
|
||||
|
||||
public OracleCatalogResult? GetCachedCatalog() => _catalog;
|
||||
}
|
||||
@@ -0,0 +1,18 @@
|
||||
<Project Sdk="Microsoft.NET.Sdk">
|
||||
<PropertyGroup>
|
||||
<TargetFramework>net10.0</TargetFramework>
|
||||
<LangVersion>preview</LangVersion>
|
||||
<Nullable>enable</Nullable>
|
||||
<ImplicitUsings>enable</ImplicitUsings>
|
||||
<TreatWarningsAsErrors>true</TreatWarningsAsErrors>
|
||||
</PropertyGroup>
|
||||
<ItemGroup>
|
||||
<ProjectReference Include="..\StellaOps.Vexer.Connectors.Abstractions\StellaOps.Vexer.Connectors.Abstractions.csproj" />
|
||||
</ItemGroup>
|
||||
<ItemGroup>
|
||||
<PackageReference Include="Microsoft.Extensions.Caching.Memory" Version="8.0.0" />
|
||||
<PackageReference Include="Microsoft.Extensions.Http" Version="8.0.0" />
|
||||
<PackageReference Include="Microsoft.Extensions.Options" Version="8.0.0" />
|
||||
<PackageReference Include="System.IO.Abstractions" Version="20.0.28" />
|
||||
</ItemGroup>
|
||||
</Project>
|
||||
@@ -2,6 +2,6 @@ If you are working on this file you need to read docs/ARCHITECTURE_VEXER.md and
|
||||
# TASKS
|
||||
| Task | Owner(s) | Depends on | Notes |
|
||||
|---|---|---|---|
|
||||
|VEXER-CONN-ORACLE-01-001 – Oracle CSAF catalogue discovery|Team Vexer Connectors – Oracle|VEXER-CONN-ABS-01-001|TODO – Implement catalogue discovery, CPU calendar awareness, and offline snapshot import for Oracle CSAF feeds.|
|
||||
|VEXER-CONN-ORACLE-01-001 – Oracle CSAF catalogue discovery|Team Vexer Connectors – Oracle|VEXER-CONN-ABS-01-001|DOING (2025-10-17) – Implement catalogue discovery, CPU calendar awareness, and offline snapshot import for Oracle CSAF feeds.|
|
||||
|VEXER-CONN-ORACLE-01-002 – CSAF download & dedupe pipeline|Team Vexer Connectors – Oracle|VEXER-CONN-ORACLE-01-001, VEXER-STORAGE-01-003|TODO – Fetch CSAF documents with retry/backoff, checksum validation, revision deduplication, and raw persistence.|
|
||||
|VEXER-CONN-ORACLE-01-003 – Trust metadata + provenance|Team Vexer Connectors – Oracle|VEXER-CONN-ORACLE-01-002, VEXER-POLICY-01-001|TODO – Emit Oracle signing metadata (PGP/cosign) and provenance hints for consensus weighting.|
|
||||
|
||||
@@ -0,0 +1,277 @@
|
||||
using System.Collections.Generic;
|
||||
using System.Collections.Immutable;
|
||||
using System.Net;
|
||||
using System.Net.Http;
|
||||
using System.Text;
|
||||
using FluentAssertions;
|
||||
using Microsoft.Extensions.Caching.Memory;
|
||||
using Microsoft.Extensions.DependencyInjection;
|
||||
using Microsoft.Extensions.Logging.Abstractions;
|
||||
using Microsoft.Extensions.Options;
|
||||
using StellaOps.Vexer.Connectors.Abstractions;
|
||||
using StellaOps.Vexer.Connectors.RedHat.CSAF.Configuration;
|
||||
using StellaOps.Vexer.Connectors.RedHat.CSAF.Metadata;
|
||||
using StellaOps.Vexer.Core;
|
||||
using StellaOps.Vexer.Storage.Mongo;
|
||||
|
||||
namespace StellaOps.Vexer.Connectors.RedHat.CSAF.Tests.Connectors;
|
||||
|
||||
public sealed class RedHatCsafConnectorTests
|
||||
{
|
||||
private static readonly VexConnectorDescriptor Descriptor = new("vexer:redhat", VexProviderKind.Distro, "Red Hat CSAF");
|
||||
|
||||
[Fact]
|
||||
public async Task FetchAsync_EmitsDocumentsAfterSince()
|
||||
{
|
||||
var metadata = """
|
||||
{
|
||||
"metadata": {
|
||||
"provider": { "name": "Red Hat Product Security" }
|
||||
},
|
||||
"distributions": [
|
||||
{ "directory": "https://example.com/security/data/csaf/v2/advisories/" }
|
||||
],
|
||||
"rolie": {
|
||||
"feeds": [
|
||||
{ "url": "https://example.com/security/data/csaf/v2/advisories/rolie/feed.atom" }
|
||||
]
|
||||
}
|
||||
}
|
||||
""";
|
||||
|
||||
var feed = """
|
||||
<feed xmlns="http://www.w3.org/2005/Atom">
|
||||
<entry>
|
||||
<id>urn:redhat:1</id>
|
||||
<updated>2025-10-16T10:00:00Z</updated>
|
||||
<link href="https://example.com/doc1.json" rel="enclosure" />
|
||||
</entry>
|
||||
<entry>
|
||||
<id>urn:redhat:2</id>
|
||||
<updated>2025-10-17T10:00:00Z</updated>
|
||||
<link href="https://example.com/doc2.json" rel="enclosure" />
|
||||
</entry>
|
||||
</feed>
|
||||
""";
|
||||
|
||||
var handler = TestHttpMessageHandler.Create(
|
||||
request => Response(HttpStatusCode.OK, metadata, "application/json"),
|
||||
request => Response(HttpStatusCode.OK, feed, "application/atom+xml"),
|
||||
request => Response(HttpStatusCode.OK, "{ \"csaf\": 1 }", "application/json"));
|
||||
|
||||
var httpClient = new HttpClient(handler)
|
||||
{
|
||||
BaseAddress = new Uri("https://example.com/"),
|
||||
};
|
||||
|
||||
var factory = new SingleClientHttpClientFactory(httpClient);
|
||||
var cache = new MemoryCache(new MemoryCacheOptions());
|
||||
var options = Options.Create(new RedHatConnectorOptions());
|
||||
var metadataLoader = new RedHatProviderMetadataLoader(factory, cache, options, NullLogger<RedHatProviderMetadataLoader>.Instance);
|
||||
var stateRepository = new InMemoryConnectorStateRepository();
|
||||
var connector = new RedHatCsafConnector(Descriptor, metadataLoader, factory, stateRepository, NullLogger<RedHatCsafConnector>.Instance, TimeProvider.System);
|
||||
|
||||
var rawSink = new CapturingRawSink();
|
||||
var context = new VexConnectorContext(
|
||||
new DateTimeOffset(2025, 10, 16, 12, 0, 0, TimeSpan.Zero),
|
||||
VexConnectorSettings.Empty,
|
||||
rawSink,
|
||||
new NoopSignatureVerifier(),
|
||||
new NoopNormalizerRouter(),
|
||||
new ServiceCollection().BuildServiceProvider());
|
||||
|
||||
var results = new List<VexRawDocument>();
|
||||
await foreach (var document in connector.FetchAsync(context, CancellationToken.None))
|
||||
{
|
||||
results.Add(document);
|
||||
}
|
||||
|
||||
Assert.Single(results);
|
||||
Assert.Single(rawSink.Documents);
|
||||
Assert.Equal("https://example.com/doc2.json", results[0].SourceUri.ToString());
|
||||
Assert.Equal("https://example.com/doc2.json", rawSink.Documents[0].SourceUri.ToString());
|
||||
Assert.Equal(3, handler.CallCount);
|
||||
stateRepository.State.Should().NotBeNull();
|
||||
stateRepository.State!.LastUpdated.Should().Be(new DateTimeOffset(2025, 10, 17, 10, 0, 0, TimeSpan.Zero));
|
||||
stateRepository.State.DocumentDigests.Should().HaveCount(1);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task FetchAsync_UsesStateToSkipDuplicateDocuments()
|
||||
{
|
||||
var metadata = """
|
||||
{
|
||||
"metadata": {
|
||||
"provider": { "name": "Red Hat Product Security" }
|
||||
},
|
||||
"distributions": [
|
||||
{ "directory": "https://example.com/security/data/csaf/v2/advisories/" }
|
||||
],
|
||||
"rolie": {
|
||||
"feeds": [
|
||||
{ "url": "https://example.com/security/data/csaf/v2/advisories/rolie/feed.atom" }
|
||||
]
|
||||
}
|
||||
}
|
||||
""";
|
||||
|
||||
var feed = """
|
||||
<feed xmlns="http://www.w3.org/2005/Atom">
|
||||
<entry>
|
||||
<id>urn:redhat:1</id>
|
||||
<updated>2025-10-17T10:00:00Z</updated>
|
||||
<link href="https://example.com/doc1.json" rel="enclosure" />
|
||||
</entry>
|
||||
</feed>
|
||||
""";
|
||||
|
||||
var handler1 = TestHttpMessageHandler.Create(
|
||||
_ => Response(HttpStatusCode.OK, metadata, "application/json"),
|
||||
_ => Response(HttpStatusCode.OK, feed, "application/atom+xml"),
|
||||
_ => Response(HttpStatusCode.OK, "{ \"csaf\": 1 }", "application/json"));
|
||||
|
||||
var stateRepository = new InMemoryConnectorStateRepository();
|
||||
await ExecuteFetchAsync(handler1, stateRepository);
|
||||
|
||||
stateRepository.State.Should().NotBeNull();
|
||||
var previousState = stateRepository.State!;
|
||||
|
||||
var handler2 = TestHttpMessageHandler.Create(
|
||||
_ => Response(HttpStatusCode.OK, metadata, "application/json"),
|
||||
_ => Response(HttpStatusCode.OK, feed, "application/atom+xml"),
|
||||
_ => Response(HttpStatusCode.OK, "{ \"csaf\": 1 }", "application/json"));
|
||||
|
||||
var (results, rawSink) = await ExecuteFetchAsync(handler2, stateRepository);
|
||||
|
||||
results.Should().BeEmpty();
|
||||
rawSink.Documents.Should().BeEmpty();
|
||||
stateRepository.State!.DocumentDigests.Should().Equal(previousState.DocumentDigests);
|
||||
}
|
||||
|
||||
private static HttpResponseMessage Response(HttpStatusCode statusCode, string content, string contentType)
|
||||
=> new(statusCode)
|
||||
{
|
||||
Content = new StringContent(content, Encoding.UTF8, contentType),
|
||||
};
|
||||
|
||||
private sealed class CapturingRawSink : IVexRawDocumentSink
|
||||
{
|
||||
public List<VexRawDocument> Documents { get; } = new();
|
||||
|
||||
public ValueTask StoreAsync(VexRawDocument document, CancellationToken cancellationToken)
|
||||
{
|
||||
Documents.Add(document);
|
||||
return ValueTask.CompletedTask;
|
||||
}
|
||||
}
|
||||
|
||||
private sealed class NoopSignatureVerifier : IVexSignatureVerifier
|
||||
{
|
||||
public ValueTask<VexSignatureMetadata?> VerifyAsync(VexRawDocument document, CancellationToken cancellationToken)
|
||||
=> ValueTask.FromResult<VexSignatureMetadata?>(null);
|
||||
}
|
||||
|
||||
private sealed class NoopNormalizerRouter : IVexNormalizerRouter
|
||||
{
|
||||
public ValueTask<VexClaimBatch> NormalizeAsync(VexRawDocument document, CancellationToken cancellationToken)
|
||||
=> ValueTask.FromResult(new VexClaimBatch(document, ImmutableArray<VexClaim>.Empty, ImmutableDictionary<string, string>.Empty));
|
||||
}
|
||||
|
||||
private sealed class SingleClientHttpClientFactory : IHttpClientFactory
|
||||
{
|
||||
private readonly HttpClient _client;
|
||||
|
||||
public SingleClientHttpClientFactory(HttpClient client)
|
||||
{
|
||||
_client = client;
|
||||
}
|
||||
|
||||
public HttpClient CreateClient(string name) => _client;
|
||||
}
|
||||
|
||||
private sealed class TestHttpMessageHandler : HttpMessageHandler
|
||||
{
|
||||
private readonly Queue<Func<HttpRequestMessage, HttpResponseMessage>> _responders;
|
||||
|
||||
private TestHttpMessageHandler(IEnumerable<Func<HttpRequestMessage, HttpResponseMessage>> responders)
|
||||
{
|
||||
_responders = new Queue<Func<HttpRequestMessage, HttpResponseMessage>>(responders);
|
||||
}
|
||||
|
||||
public int CallCount { get; private set; }
|
||||
|
||||
public static TestHttpMessageHandler Create(params Func<HttpRequestMessage, HttpResponseMessage>[] responders)
|
||||
=> new(responders);
|
||||
|
||||
protected override Task<HttpResponseMessage> SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
|
||||
{
|
||||
CallCount++;
|
||||
if (_responders.Count == 0)
|
||||
{
|
||||
throw new InvalidOperationException("No responder configured for request.");
|
||||
}
|
||||
|
||||
var responder = _responders.Count > 1
|
||||
? _responders.Dequeue()
|
||||
: _responders.Peek();
|
||||
|
||||
var response = responder(request);
|
||||
response.RequestMessage = request;
|
||||
return Task.FromResult(response);
|
||||
}
|
||||
}
|
||||
|
||||
private static async Task<(List<VexRawDocument> Documents, CapturingRawSink Sink)> ExecuteFetchAsync(
|
||||
TestHttpMessageHandler handler,
|
||||
InMemoryConnectorStateRepository stateRepository)
|
||||
{
|
||||
var httpClient = new HttpClient(handler)
|
||||
{
|
||||
BaseAddress = new Uri("https://example.com/"),
|
||||
};
|
||||
|
||||
var factory = new SingleClientHttpClientFactory(httpClient);
|
||||
var cache = new MemoryCache(new MemoryCacheOptions());
|
||||
var options = Options.Create(new RedHatConnectorOptions());
|
||||
var metadataLoader = new RedHatProviderMetadataLoader(factory, cache, options, NullLogger<RedHatProviderMetadataLoader>.Instance);
|
||||
var connector = new RedHatCsafConnector(Descriptor, metadataLoader, factory, stateRepository, NullLogger<RedHatCsafConnector>.Instance, TimeProvider.System);
|
||||
|
||||
var rawSink = new CapturingRawSink();
|
||||
var context = new VexConnectorContext(
|
||||
null,
|
||||
VexConnectorSettings.Empty,
|
||||
rawSink,
|
||||
new NoopSignatureVerifier(),
|
||||
new NoopNormalizerRouter(),
|
||||
new ServiceCollection().BuildServiceProvider());
|
||||
|
||||
var documents = new List<VexRawDocument>();
|
||||
await foreach (var document in connector.FetchAsync(context, CancellationToken.None))
|
||||
{
|
||||
documents.Add(document);
|
||||
}
|
||||
|
||||
return (documents, rawSink);
|
||||
}
|
||||
|
||||
private sealed class InMemoryConnectorStateRepository : IVexConnectorStateRepository
|
||||
{
|
||||
public VexConnectorState? State { get; private set; }
|
||||
|
||||
public ValueTask<VexConnectorState?> GetAsync(string connectorId, CancellationToken cancellationToken)
|
||||
{
|
||||
if (State is not null && string.Equals(State.ConnectorId, connectorId, StringComparison.OrdinalIgnoreCase))
|
||||
{
|
||||
return ValueTask.FromResult<VexConnectorState?>(State);
|
||||
}
|
||||
|
||||
return ValueTask.FromResult<VexConnectorState?>(null);
|
||||
}
|
||||
|
||||
public ValueTask SaveAsync(VexConnectorState state, CancellationToken cancellationToken)
|
||||
{
|
||||
State = state;
|
||||
return ValueTask.CompletedTask;
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,235 @@
|
||||
using System.Collections.Generic;
|
||||
using System.Net;
|
||||
using System.Net.Http;
|
||||
using System.Net.Http.Headers;
|
||||
using System.Text;
|
||||
using Microsoft.Extensions.Caching.Memory;
|
||||
using Microsoft.Extensions.Logging.Abstractions;
|
||||
using Microsoft.Extensions.Options;
|
||||
using StellaOps.Vexer.Connectors.RedHat.CSAF.Configuration;
|
||||
using StellaOps.Vexer.Connectors.RedHat.CSAF.Metadata;
|
||||
using System.IO.Abstractions.TestingHelpers;
|
||||
|
||||
namespace StellaOps.Vexer.Connectors.RedHat.CSAF.Tests.Metadata;
|
||||
|
||||
public sealed class RedHatProviderMetadataLoaderTests
|
||||
{
|
||||
private const string SampleJson = """
|
||||
{
|
||||
"metadata": {
|
||||
"provider": {
|
||||
"name": "Red Hat Product Security"
|
||||
}
|
||||
},
|
||||
"distributions": [
|
||||
{ "directory": "https://access.redhat.com/security/data/csaf/v2/advisories/" }
|
||||
],
|
||||
"rolie": {
|
||||
"feeds": [
|
||||
{ "url": "https://access.redhat.com/security/data/csaf/v2/advisories/rolie/feed.atom" }
|
||||
]
|
||||
}
|
||||
}
|
||||
""";
|
||||
|
||||
[Fact]
|
||||
public async Task LoadAsync_FetchesMetadataAndCaches()
|
||||
{
|
||||
var handler = TestHttpMessageHandler.RespondWith(_ =>
|
||||
{
|
||||
var response = new HttpResponseMessage(HttpStatusCode.OK)
|
||||
{
|
||||
Content = new StringContent(SampleJson, Encoding.UTF8, "application/json"),
|
||||
};
|
||||
response.Headers.ETag = new EntityTagHeaderValue("\"abc\"");
|
||||
return response;
|
||||
});
|
||||
|
||||
var httpClient = new HttpClient(handler)
|
||||
{
|
||||
BaseAddress = new Uri("https://access.redhat.com/"),
|
||||
};
|
||||
|
||||
var factory = new SingleClientHttpClientFactory(httpClient);
|
||||
var cache = new MemoryCache(new MemoryCacheOptions());
|
||||
var fileSystem = new MockFileSystem();
|
||||
var options = new RedHatConnectorOptions
|
||||
{
|
||||
MetadataUri = new Uri("https://access.redhat.com/.well-known/csaf/provider-metadata.json"),
|
||||
OfflineSnapshotPath = fileSystem.Path.Combine("/offline", "redhat-provider.json"),
|
||||
CosignIssuer = "https://sigstore.dev/redhat",
|
||||
CosignIdentityPattern = "^spiffe://redhat/.+$",
|
||||
};
|
||||
options.PgpFingerprints.Add("A1B2C3D4E5F6");
|
||||
options.Validate(fileSystem);
|
||||
|
||||
var loader = new RedHatProviderMetadataLoader(factory, cache, Options.Create(options), NullLogger<RedHatProviderMetadataLoader>.Instance, fileSystem);
|
||||
|
||||
var result = await loader.LoadAsync(CancellationToken.None);
|
||||
|
||||
Assert.Equal("Red Hat Product Security", result.Provider.DisplayName);
|
||||
Assert.False(result.FromCache);
|
||||
Assert.False(result.FromOfflineSnapshot);
|
||||
Assert.Single(result.Provider.BaseUris);
|
||||
Assert.Equal("https://access.redhat.com/security/data/csaf/v2/advisories/", result.Provider.BaseUris[0].ToString());
|
||||
Assert.Equal("https://access.redhat.com/.well-known/csaf/provider-metadata.json", result.Provider.Discovery.WellKnownMetadata?.ToString());
|
||||
Assert.Equal("https://access.redhat.com/security/data/csaf/v2/advisories/rolie/feed.atom", result.Provider.Discovery.RolIeService?.ToString());
|
||||
Assert.Equal(1.0, result.Provider.Trust.Weight);
|
||||
Assert.NotNull(result.Provider.Trust.Cosign);
|
||||
Assert.Equal("https://sigstore.dev/redhat", result.Provider.Trust.Cosign!.Issuer);
|
||||
Assert.Equal("^spiffe://redhat/.+$", result.Provider.Trust.Cosign.IdentityPattern);
|
||||
Assert.Contains("A1B2C3D4E5F6", result.Provider.Trust.PgpFingerprints);
|
||||
Assert.True(fileSystem.FileExists(options.OfflineSnapshotPath));
|
||||
Assert.Equal(1, handler.CallCount);
|
||||
|
||||
var second = await loader.LoadAsync(CancellationToken.None);
|
||||
Assert.True(second.FromCache);
|
||||
Assert.False(second.FromOfflineSnapshot);
|
||||
Assert.Equal(1, handler.CallCount);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task LoadAsync_UsesOfflineSnapshotWhenPreferred()
|
||||
{
|
||||
var handler = TestHttpMessageHandler.RespondWith(_ => throw new InvalidOperationException("HTTP should not be called"));
|
||||
|
||||
var httpClient = new HttpClient(handler);
|
||||
var factory = new SingleClientHttpClientFactory(httpClient);
|
||||
var cache = new MemoryCache(new MemoryCacheOptions());
|
||||
|
||||
var fileSystem = new MockFileSystem(new Dictionary<string, MockFileData>
|
||||
{
|
||||
["/snapshots/redhat.json"] = new MockFileData(SampleJson),
|
||||
});
|
||||
|
||||
var options = new RedHatConnectorOptions
|
||||
{
|
||||
MetadataUri = new Uri("https://access.redhat.com/.well-known/csaf/provider-metadata.json"),
|
||||
OfflineSnapshotPath = "/snapshots/redhat.json",
|
||||
PreferOfflineSnapshot = true,
|
||||
PersistOfflineSnapshot = false,
|
||||
};
|
||||
options.Validate(fileSystem);
|
||||
|
||||
var loader = new RedHatProviderMetadataLoader(factory, cache, Options.Create(options), NullLogger<RedHatProviderMetadataLoader>.Instance, fileSystem);
|
||||
|
||||
var result = await loader.LoadAsync(CancellationToken.None);
|
||||
|
||||
Assert.Equal("Red Hat Product Security", result.Provider.DisplayName);
|
||||
Assert.False(result.FromCache);
|
||||
Assert.True(result.FromOfflineSnapshot);
|
||||
Assert.Equal(0, handler.CallCount);
|
||||
|
||||
var second = await loader.LoadAsync(CancellationToken.None);
|
||||
Assert.True(second.FromCache);
|
||||
Assert.True(second.FromOfflineSnapshot);
|
||||
Assert.Equal(0, handler.CallCount);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task LoadAsync_UsesETagForConditionalRequest()
|
||||
{
|
||||
var handler = TestHttpMessageHandler.Create(
|
||||
_ =>
|
||||
{
|
||||
var response = new HttpResponseMessage(HttpStatusCode.OK)
|
||||
{
|
||||
Content = new StringContent(SampleJson, Encoding.UTF8, "application/json"),
|
||||
};
|
||||
response.Headers.ETag = new EntityTagHeaderValue("\"abc\"");
|
||||
return response;
|
||||
},
|
||||
request =>
|
||||
{
|
||||
Assert.Contains(request.Headers.IfNoneMatch, etag => etag.Tag == "\"abc\"");
|
||||
return new HttpResponseMessage(HttpStatusCode.NotModified);
|
||||
});
|
||||
|
||||
var httpClient = new HttpClient(handler);
|
||||
var factory = new SingleClientHttpClientFactory(httpClient);
|
||||
var cache = new MemoryCache(new MemoryCacheOptions());
|
||||
var fileSystem = new MockFileSystem();
|
||||
var options = new RedHatConnectorOptions
|
||||
{
|
||||
MetadataUri = new Uri("https://access.redhat.com/.well-known/csaf/provider-metadata.json"),
|
||||
OfflineSnapshotPath = "/offline/redhat.json",
|
||||
MetadataCacheDuration = TimeSpan.FromMinutes(1),
|
||||
};
|
||||
options.Validate(fileSystem);
|
||||
|
||||
var loader = new RedHatProviderMetadataLoader(factory, cache, Options.Create(options), NullLogger<RedHatProviderMetadataLoader>.Instance, fileSystem);
|
||||
|
||||
var first = await loader.LoadAsync(CancellationToken.None);
|
||||
Assert.False(first.FromCache);
|
||||
Assert.False(first.FromOfflineSnapshot);
|
||||
|
||||
Assert.True(cache.TryGetValue(RedHatProviderMetadataLoader.CacheKey, out var entryObj));
|
||||
Assert.NotNull(entryObj);
|
||||
|
||||
var entryType = entryObj!.GetType();
|
||||
var provider = entryType.GetProperty("Provider")!.GetValue(entryObj);
|
||||
var fetchedAt = entryType.GetProperty("FetchedAt")!.GetValue(entryObj);
|
||||
var etag = entryType.GetProperty("ETag")!.GetValue(entryObj);
|
||||
var fromOffline = entryType.GetProperty("FromOffline")!.GetValue(entryObj);
|
||||
|
||||
var expiredEntry = Activator.CreateInstance(entryType, provider, fetchedAt, DateTimeOffset.UtcNow - TimeSpan.FromSeconds(1), etag, fromOffline);
|
||||
cache.Set(RedHatProviderMetadataLoader.CacheKey, expiredEntry!, new MemoryCacheEntryOptions
|
||||
{
|
||||
AbsoluteExpiration = DateTimeOffset.UtcNow + TimeSpan.FromMinutes(1),
|
||||
});
|
||||
|
||||
var second = await loader.LoadAsync(CancellationToken.None);
|
||||
|
||||
var third = await loader.LoadAsync(CancellationToken.None);
|
||||
Assert.True(third.FromCache);
|
||||
|
||||
Assert.Equal(2, handler.CallCount);
|
||||
}
|
||||
|
||||
private sealed class SingleClientHttpClientFactory : IHttpClientFactory
|
||||
{
|
||||
private readonly HttpClient _client;
|
||||
|
||||
public SingleClientHttpClientFactory(HttpClient client)
|
||||
{
|
||||
_client = client ?? throw new ArgumentNullException(nameof(client));
|
||||
}
|
||||
|
||||
public HttpClient CreateClient(string name) => _client;
|
||||
}
|
||||
|
||||
private sealed class TestHttpMessageHandler : HttpMessageHandler
|
||||
{
|
||||
private readonly Queue<Func<HttpRequestMessage, HttpResponseMessage>> _responders;
|
||||
|
||||
private TestHttpMessageHandler(IEnumerable<Func<HttpRequestMessage, HttpResponseMessage>> responders)
|
||||
{
|
||||
_responders = new Queue<Func<HttpRequestMessage, HttpResponseMessage>>(responders);
|
||||
}
|
||||
|
||||
public int CallCount { get; private set; }
|
||||
|
||||
public static TestHttpMessageHandler RespondWith(Func<HttpRequestMessage, HttpResponseMessage> responder)
|
||||
=> new(new[] { responder });
|
||||
|
||||
public static TestHttpMessageHandler Create(params Func<HttpRequestMessage, HttpResponseMessage>[] responders)
|
||||
=> new(responders);
|
||||
|
||||
protected override Task<HttpResponseMessage> SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
|
||||
{
|
||||
CallCount++;
|
||||
if (_responders.Count == 0)
|
||||
{
|
||||
throw new InvalidOperationException("No responder configured for request.");
|
||||
}
|
||||
|
||||
var responder = _responders.Count > 1
|
||||
? _responders.Dequeue()
|
||||
: _responders.Peek();
|
||||
|
||||
var response = responder(request);
|
||||
response.RequestMessage = request;
|
||||
return Task.FromResult(response);
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,17 @@
|
||||
<Project Sdk="Microsoft.NET.Sdk">
|
||||
<PropertyGroup>
|
||||
<TargetFramework>net10.0</TargetFramework>
|
||||
<LangVersion>preview</LangVersion>
|
||||
<Nullable>enable</Nullable>
|
||||
<ImplicitUsings>enable</ImplicitUsings>
|
||||
<TreatWarningsAsErrors>true</TreatWarningsAsErrors>
|
||||
</PropertyGroup>
|
||||
<ItemGroup>
|
||||
<ProjectReference Include="..\StellaOps.Vexer.Connectors.RedHat.CSAF\StellaOps.Vexer.Connectors.RedHat.CSAF.csproj" />
|
||||
<ProjectReference Include="..\StellaOps.Vexer.Storage.Mongo\StellaOps.Vexer.Storage.Mongo.csproj" />
|
||||
</ItemGroup>
|
||||
<ItemGroup>
|
||||
<PackageReference Include="FluentAssertions" Version="6.12.0" />
|
||||
<PackageReference Include="System.IO.Abstractions.TestingHelpers" Version="20.0.28" />
|
||||
</ItemGroup>
|
||||
</Project>
|
||||
@@ -3,6 +3,8 @@
|
||||
Connector for Red Hat CSAF VEX feeds, fetching provider metadata, CSAF documents, and projecting them into raw storage for normalization.
|
||||
## Scope
|
||||
- Discovery via `/.well-known/csaf/provider-metadata.json`, scheduling windows, and ETag-aware HTTP fetches.
|
||||
- `RedHatProviderMetadataLoader` handles `.well-known` metadata with caching, schema validation, and offline snapshots.
|
||||
- `RedHatCsafConnector` consumes ROLIE feeds to fetch incremental CSAF documents, honours `context.Since`, and streams raw advisories to storage.
|
||||
- Mapping Red Hat CSAF specifics (product tree aliases, RHSA identifiers, revision history) into raw documents.
|
||||
- Emitting structured telemetry and resume markers for incremental pulls.
|
||||
- Supplying Red Hat-specific trust overrides and provenance hints to normalization.
|
||||
|
||||
@@ -0,0 +1,104 @@
|
||||
using System.Collections.Generic;
|
||||
using System.IO.Abstractions;
|
||||
|
||||
namespace StellaOps.Vexer.Connectors.RedHat.CSAF.Configuration;
|
||||
|
||||
public sealed class RedHatConnectorOptions
|
||||
{
|
||||
public static readonly Uri DefaultMetadataUri = new("https://access.redhat.com/.well-known/csaf/provider-metadata.json");
|
||||
|
||||
/// <summary>
|
||||
/// HTTP client name registered for the connector.
|
||||
/// </summary>
|
||||
public const string HttpClientName = "vexer.connector.redhat";
|
||||
|
||||
/// <summary>
|
||||
/// URI of the CSAF provider metadata document.
|
||||
/// </summary>
|
||||
public Uri MetadataUri { get; set; } = DefaultMetadataUri;
|
||||
|
||||
/// <summary>
|
||||
/// Duration to cache loaded metadata before refreshing.
|
||||
/// </summary>
|
||||
public TimeSpan MetadataCacheDuration { get; set; } = TimeSpan.FromHours(1);
|
||||
|
||||
/// <summary>
|
||||
/// Optional file path used to store or source offline metadata snapshots.
|
||||
/// </summary>
|
||||
public string? OfflineSnapshotPath { get; set; }
|
||||
|
||||
/// <summary>
|
||||
/// When true, the loader prefers the offline snapshot without attempting a network fetch.
|
||||
/// </summary>
|
||||
public bool PreferOfflineSnapshot { get; set; }
|
||||
|
||||
/// <summary>
|
||||
/// Enables writing fresh metadata responses to <see cref="OfflineSnapshotPath"/>.
|
||||
/// </summary>
|
||||
public bool PersistOfflineSnapshot { get; set; } = true;
|
||||
|
||||
/// <summary>
|
||||
/// Explicit trust weight override applied to the provider entry.
|
||||
/// </summary>
|
||||
public double TrustWeight { get; set; } = 1.0;
|
||||
|
||||
/// <summary>
|
||||
/// Sigstore/Cosign issuer used to verify CSAF signatures, if published.
|
||||
/// </summary>
|
||||
public string? CosignIssuer { get; set; } = "https://access.redhat.com";
|
||||
|
||||
/// <summary>
|
||||
/// Identity pattern matched against the Cosign certificate subject.
|
||||
/// </summary>
|
||||
public string? CosignIdentityPattern { get; set; } = "^https://access\\.redhat\\.com/.+$";
|
||||
|
||||
/// <summary>
|
||||
/// Optional list of PGP fingerprints recognised for Red Hat CSAF artifacts.
|
||||
/// </summary>
|
||||
public IList<string> PgpFingerprints { get; } = new List<string>();
|
||||
|
||||
public void Validate(IFileSystem? fileSystem = null)
|
||||
{
|
||||
if (MetadataUri is null || !MetadataUri.IsAbsoluteUri)
|
||||
{
|
||||
throw new InvalidOperationException("Metadata URI must be absolute.");
|
||||
}
|
||||
|
||||
if (MetadataUri.Scheme is not ("http" or "https"))
|
||||
{
|
||||
throw new InvalidOperationException("Metadata URI must use HTTP or HTTPS.");
|
||||
}
|
||||
|
||||
if (MetadataCacheDuration <= TimeSpan.Zero)
|
||||
{
|
||||
throw new InvalidOperationException("Metadata cache duration must be positive.");
|
||||
}
|
||||
|
||||
if (!string.IsNullOrWhiteSpace(OfflineSnapshotPath))
|
||||
{
|
||||
var fs = fileSystem ?? new FileSystem();
|
||||
var directory = Path.GetDirectoryName(OfflineSnapshotPath);
|
||||
if (!string.IsNullOrWhiteSpace(directory) && !fs.Directory.Exists(directory))
|
||||
{
|
||||
fs.Directory.CreateDirectory(directory);
|
||||
}
|
||||
}
|
||||
|
||||
if (double.IsNaN(TrustWeight) || double.IsInfinity(TrustWeight) || TrustWeight <= 0)
|
||||
{
|
||||
TrustWeight = 1.0;
|
||||
}
|
||||
else if (TrustWeight > 1.0)
|
||||
{
|
||||
TrustWeight = 1.0;
|
||||
}
|
||||
|
||||
if (CosignIssuer is not null)
|
||||
{
|
||||
if (string.IsNullOrWhiteSpace(CosignIdentityPattern))
|
||||
{
|
||||
throw new InvalidOperationException("CosignIdentityPattern must be provided when CosignIssuer is specified.");
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,45 @@
|
||||
using System.Net;
|
||||
using Microsoft.Extensions.Caching.Memory;
|
||||
using Microsoft.Extensions.DependencyInjection;
|
||||
using Microsoft.Extensions.DependencyInjection.Extensions;
|
||||
using StellaOps.Vexer.Connectors.RedHat.CSAF.Configuration;
|
||||
using StellaOps.Vexer.Connectors.RedHat.CSAF.Metadata;
|
||||
using StellaOps.Vexer.Core;
|
||||
using StellaOps.Vexer.Storage.Mongo;
|
||||
using System.IO.Abstractions;
|
||||
|
||||
namespace StellaOps.Vexer.Connectors.RedHat.CSAF.DependencyInjection;
|
||||
|
||||
public static class RedHatConnectorServiceCollectionExtensions
|
||||
{
|
||||
public static IServiceCollection AddRedHatCsafConnector(this IServiceCollection services, Action<RedHatConnectorOptions>? configure = null)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(services);
|
||||
|
||||
services.AddOptions<RedHatConnectorOptions>()
|
||||
.Configure(options =>
|
||||
{
|
||||
configure?.Invoke(options);
|
||||
})
|
||||
.PostConfigure(options => options.Validate());
|
||||
|
||||
services.TryAddSingleton<IMemoryCache, MemoryCache>();
|
||||
services.TryAddSingleton<IFileSystem, FileSystem>();
|
||||
|
||||
services.AddHttpClient(RedHatConnectorOptions.HttpClientName, client =>
|
||||
{
|
||||
client.Timeout = TimeSpan.FromSeconds(30);
|
||||
client.DefaultRequestHeaders.UserAgent.ParseAdd("StellaOps.Vexer.Connectors.RedHat/1.0");
|
||||
client.DefaultRequestHeaders.Accept.ParseAdd("application/json");
|
||||
})
|
||||
.ConfigurePrimaryHttpMessageHandler(() => new HttpClientHandler
|
||||
{
|
||||
AutomaticDecompression = DecompressionMethods.All,
|
||||
});
|
||||
|
||||
services.AddSingleton<RedHatProviderMetadataLoader>();
|
||||
services.AddSingleton<IVexConnector, RedHatCsafConnector>();
|
||||
|
||||
return services;
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,312 @@
|
||||
using System.Collections.Immutable;
|
||||
using System.Linq;
|
||||
using System.Net;
|
||||
using System.Net.Http;
|
||||
using System.Net.Http.Headers;
|
||||
using System.Text.Json;
|
||||
using System.Text.Json.Serialization;
|
||||
using Microsoft.Extensions.Caching.Memory;
|
||||
using Microsoft.Extensions.Logging;
|
||||
using Microsoft.Extensions.Options;
|
||||
using StellaOps.Vexer.Connectors.RedHat.CSAF.Configuration;
|
||||
using StellaOps.Vexer.Core;
|
||||
using System.IO.Abstractions;
|
||||
|
||||
namespace StellaOps.Vexer.Connectors.RedHat.CSAF.Metadata;
|
||||
|
||||
public sealed class RedHatProviderMetadataLoader
|
||||
{
|
||||
public const string CacheKey = "StellaOps.Vexer.Connectors.RedHat.CSAF.Metadata";
|
||||
|
||||
private readonly IHttpClientFactory _httpClientFactory;
|
||||
private readonly IMemoryCache _cache;
|
||||
private readonly ILogger<RedHatProviderMetadataLoader> _logger;
|
||||
private readonly RedHatConnectorOptions _options;
|
||||
private readonly IFileSystem _fileSystem;
|
||||
private readonly JsonSerializerOptions _serializerOptions;
|
||||
private readonly SemaphoreSlim _refreshSemaphore = new(1, 1);
|
||||
|
||||
public RedHatProviderMetadataLoader(
|
||||
IHttpClientFactory httpClientFactory,
|
||||
IMemoryCache memoryCache,
|
||||
IOptions<RedHatConnectorOptions> options,
|
||||
ILogger<RedHatProviderMetadataLoader> logger,
|
||||
IFileSystem? fileSystem = null)
|
||||
{
|
||||
_httpClientFactory = httpClientFactory ?? throw new ArgumentNullException(nameof(httpClientFactory));
|
||||
_cache = memoryCache ?? throw new ArgumentNullException(nameof(memoryCache));
|
||||
_logger = logger ?? throw new ArgumentNullException(nameof(logger));
|
||||
_options = (options ?? throw new ArgumentNullException(nameof(options))).Value ?? throw new ArgumentNullException(nameof(options));
|
||||
_fileSystem = fileSystem ?? new FileSystem();
|
||||
_serializerOptions = new JsonSerializerOptions(JsonSerializerDefaults.Web)
|
||||
{
|
||||
PropertyNameCaseInsensitive = true,
|
||||
ReadCommentHandling = JsonCommentHandling.Skip,
|
||||
};
|
||||
}
|
||||
|
||||
public async Task<RedHatProviderMetadataResult> LoadAsync(CancellationToken cancellationToken)
|
||||
{
|
||||
if (_cache.TryGetValue<CacheEntry>(CacheKey, out var cached) && cached is { } cachedEntry && !cachedEntry.IsExpired())
|
||||
{
|
||||
_logger.LogDebug("Returning cached Red Hat provider metadata (expires {Expires}).", cachedEntry.ExpiresAt);
|
||||
return new RedHatProviderMetadataResult(cachedEntry.Provider, cachedEntry.FetchedAt, true, cachedEntry.FromOffline);
|
||||
}
|
||||
|
||||
await _refreshSemaphore.WaitAsync(cancellationToken).ConfigureAwait(false);
|
||||
try
|
||||
{
|
||||
if (_cache.TryGetValue<CacheEntry>(CacheKey, out cached) && cached is { } cachedAfterLock && !cachedAfterLock.IsExpired())
|
||||
{
|
||||
return new RedHatProviderMetadataResult(cachedAfterLock.Provider, cachedAfterLock.FetchedAt, true, cachedAfterLock.FromOffline);
|
||||
}
|
||||
|
||||
CacheEntry? previous = cached;
|
||||
|
||||
// Attempt live fetch unless offline preferred.
|
||||
if (!_options.PreferOfflineSnapshot)
|
||||
{
|
||||
var httpResult = await TryFetchFromNetworkAsync(previous, cancellationToken).ConfigureAwait(false);
|
||||
if (httpResult is not null)
|
||||
{
|
||||
StoreCache(httpResult);
|
||||
return new RedHatProviderMetadataResult(httpResult.Provider, httpResult.FetchedAt, false, false);
|
||||
}
|
||||
}
|
||||
|
||||
var offlineResult = TryLoadFromOffline();
|
||||
if (offlineResult is not null)
|
||||
{
|
||||
var offlineEntry = offlineResult with
|
||||
{
|
||||
FetchedAt = DateTimeOffset.UtcNow,
|
||||
ExpiresAt = DateTimeOffset.UtcNow + _options.MetadataCacheDuration,
|
||||
FromOffline = true,
|
||||
};
|
||||
StoreCache(offlineEntry);
|
||||
return new RedHatProviderMetadataResult(offlineEntry.Provider, offlineEntry.FetchedAt, false, true);
|
||||
}
|
||||
|
||||
throw new InvalidOperationException("Unable to load Red Hat CSAF provider metadata from network or offline snapshot.");
|
||||
}
|
||||
finally
|
||||
{
|
||||
_refreshSemaphore.Release();
|
||||
}
|
||||
}
|
||||
|
||||
private void StoreCache(CacheEntry entry)
|
||||
{
|
||||
var cacheEntryOptions = new MemoryCacheEntryOptions
|
||||
{
|
||||
AbsoluteExpiration = entry.ExpiresAt,
|
||||
};
|
||||
_cache.Set(CacheKey, entry, cacheEntryOptions);
|
||||
}
|
||||
|
||||
private async Task<CacheEntry?> TryFetchFromNetworkAsync(CacheEntry? previous, CancellationToken cancellationToken)
|
||||
{
|
||||
try
|
||||
{
|
||||
var client = _httpClientFactory.CreateClient(RedHatConnectorOptions.HttpClientName);
|
||||
using var request = new HttpRequestMessage(HttpMethod.Get, _options.MetadataUri);
|
||||
if (!string.IsNullOrWhiteSpace(previous?.ETag))
|
||||
{
|
||||
if (EntityTagHeaderValue.TryParse(previous.ETag, out var etag))
|
||||
{
|
||||
request.Headers.IfNoneMatch.Add(etag);
|
||||
}
|
||||
}
|
||||
|
||||
using var response = await client.SendAsync(request, HttpCompletionOption.ResponseHeadersRead, cancellationToken).ConfigureAwait(false);
|
||||
|
||||
if (response.StatusCode == HttpStatusCode.NotModified && previous is not null)
|
||||
{
|
||||
_logger.LogDebug("Red Hat provider metadata not modified (etag {ETag}).", previous.ETag);
|
||||
return previous with
|
||||
{
|
||||
FetchedAt = DateTimeOffset.UtcNow,
|
||||
ExpiresAt = DateTimeOffset.UtcNow + _options.MetadataCacheDuration,
|
||||
};
|
||||
}
|
||||
|
||||
response.EnsureSuccessStatusCode();
|
||||
var payload = await response.Content.ReadAsStringAsync(cancellationToken).ConfigureAwait(false);
|
||||
|
||||
var provider = ParseAndValidate(payload);
|
||||
var etagHeader = response.Headers.ETag?.ToString();
|
||||
|
||||
if (_options.PersistOfflineSnapshot && !string.IsNullOrWhiteSpace(_options.OfflineSnapshotPath))
|
||||
{
|
||||
try
|
||||
{
|
||||
_fileSystem.File.WriteAllText(_options.OfflineSnapshotPath, payload);
|
||||
_logger.LogDebug("Persisted Red Hat metadata snapshot to {Path}.", _options.OfflineSnapshotPath);
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
_logger.LogWarning(ex, "Failed to persist Red Hat metadata snapshot to {Path}.", _options.OfflineSnapshotPath);
|
||||
}
|
||||
}
|
||||
|
||||
return new CacheEntry(
|
||||
provider,
|
||||
DateTimeOffset.UtcNow,
|
||||
DateTimeOffset.UtcNow + _options.MetadataCacheDuration,
|
||||
etagHeader,
|
||||
FromOffline: false);
|
||||
}
|
||||
catch (Exception ex) when (ex is not OperationCanceledException && !_options.PreferOfflineSnapshot)
|
||||
{
|
||||
_logger.LogWarning(ex, "Failed to fetch Red Hat provider metadata from {Uri}, will attempt offline snapshot.", _options.MetadataUri);
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
private CacheEntry? TryLoadFromOffline()
|
||||
{
|
||||
if (string.IsNullOrWhiteSpace(_options.OfflineSnapshotPath))
|
||||
{
|
||||
return null;
|
||||
}
|
||||
|
||||
if (!_fileSystem.File.Exists(_options.OfflineSnapshotPath))
|
||||
{
|
||||
_logger.LogWarning("Offline snapshot path {Path} does not exist.", _options.OfflineSnapshotPath);
|
||||
return null;
|
||||
}
|
||||
|
||||
try
|
||||
{
|
||||
var payload = _fileSystem.File.ReadAllText(_options.OfflineSnapshotPath);
|
||||
var provider = ParseAndValidate(payload);
|
||||
return new CacheEntry(provider, DateTimeOffset.UtcNow, DateTimeOffset.UtcNow + _options.MetadataCacheDuration, ETag: null, FromOffline: true);
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
_logger.LogError(ex, "Failed to load Red Hat provider metadata from offline snapshot {Path}.", _options.OfflineSnapshotPath);
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
private VexProvider ParseAndValidate(string payload)
|
||||
{
|
||||
if (string.IsNullOrWhiteSpace(payload))
|
||||
{
|
||||
throw new InvalidOperationException("Provider metadata payload was empty.");
|
||||
}
|
||||
|
||||
ProviderMetadataDocument? document;
|
||||
try
|
||||
{
|
||||
document = JsonSerializer.Deserialize<ProviderMetadataDocument>(payload, _serializerOptions);
|
||||
}
|
||||
catch (JsonException ex)
|
||||
{
|
||||
throw new InvalidOperationException("Provider metadata payload could not be parsed.", ex);
|
||||
}
|
||||
|
||||
if (document is null)
|
||||
{
|
||||
throw new InvalidOperationException("Provider metadata payload was null after parsing.");
|
||||
}
|
||||
|
||||
if (document.Metadata?.Provider?.Name is null)
|
||||
{
|
||||
throw new InvalidOperationException("Provider metadata missing provider name.");
|
||||
}
|
||||
|
||||
var distributions = document.Distributions?
|
||||
.Select(static d => d.Directory)
|
||||
.Where(static s => !string.IsNullOrWhiteSpace(s))
|
||||
.Select(static s => CreateUri(s!, nameof(ProviderMetadataDistribution.Directory)))
|
||||
.ToImmutableArray() ?? ImmutableArray<Uri>.Empty;
|
||||
|
||||
if (distributions.IsDefaultOrEmpty)
|
||||
{
|
||||
throw new InvalidOperationException("Provider metadata did not include any valid distribution directories.");
|
||||
}
|
||||
|
||||
Uri? rolieFeed = null;
|
||||
if (document.Rolie?.Feeds is not null)
|
||||
{
|
||||
foreach (var feed in document.Rolie.Feeds)
|
||||
{
|
||||
if (!string.IsNullOrWhiteSpace(feed.Url))
|
||||
{
|
||||
rolieFeed = CreateUri(feed.Url, "rolie.feeds[].url");
|
||||
break;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
var trust = BuildTrust();
|
||||
return new VexProvider(
|
||||
id: "vexer:redhat",
|
||||
displayName: document.Metadata.Provider.Name,
|
||||
kind: VexProviderKind.Distro,
|
||||
baseUris: distributions,
|
||||
discovery: new VexProviderDiscovery(_options.MetadataUri, rolieFeed),
|
||||
trust: trust);
|
||||
}
|
||||
|
||||
private VexProviderTrust BuildTrust()
|
||||
{
|
||||
VexCosignTrust? cosign = null;
|
||||
if (!string.IsNullOrWhiteSpace(_options.CosignIssuer) && !string.IsNullOrWhiteSpace(_options.CosignIdentityPattern))
|
||||
{
|
||||
cosign = new VexCosignTrust(_options.CosignIssuer!, _options.CosignIdentityPattern!);
|
||||
}
|
||||
|
||||
return new VexProviderTrust(
|
||||
_options.TrustWeight,
|
||||
cosign,
|
||||
_options.PgpFingerprints);
|
||||
}
|
||||
|
||||
private static Uri CreateUri(string value, string propertyName)
|
||||
{
|
||||
if (!Uri.TryCreate(value, UriKind.Absolute, out var uri) || uri.Scheme is not ("http" or "https"))
|
||||
{
|
||||
throw new InvalidOperationException($"Provider metadata field '{propertyName}' must be an absolute HTTP(S) URI.");
|
||||
}
|
||||
|
||||
return uri;
|
||||
}
|
||||
|
||||
private sealed record ProviderMetadataDocument(
|
||||
[property: JsonPropertyName("metadata")] ProviderMetadata? Metadata,
|
||||
[property: JsonPropertyName("distributions")] IReadOnlyList<ProviderMetadataDistribution>? Distributions,
|
||||
[property: JsonPropertyName("rolie")] ProviderMetadataRolie? Rolie);
|
||||
|
||||
private sealed record ProviderMetadata(
|
||||
[property: JsonPropertyName("provider")] ProviderMetadataProvider? Provider);
|
||||
|
||||
private sealed record ProviderMetadataProvider(
|
||||
[property: JsonPropertyName("name")] string? Name);
|
||||
|
||||
private sealed record ProviderMetadataDistribution(
|
||||
[property: JsonPropertyName("directory")] string? Directory);
|
||||
|
||||
private sealed record ProviderMetadataRolie(
|
||||
[property: JsonPropertyName("feeds")] IReadOnlyList<ProviderMetadataRolieFeed>? Feeds);
|
||||
|
||||
private sealed record ProviderMetadataRolieFeed(
|
||||
[property: JsonPropertyName("url")] string? Url);
|
||||
|
||||
private sealed record CacheEntry(
|
||||
VexProvider Provider,
|
||||
DateTimeOffset FetchedAt,
|
||||
DateTimeOffset ExpiresAt,
|
||||
string? ETag,
|
||||
bool FromOffline)
|
||||
{
|
||||
public bool IsExpired() => DateTimeOffset.UtcNow >= ExpiresAt;
|
||||
}
|
||||
}
|
||||
|
||||
public sealed record RedHatProviderMetadataResult(
|
||||
VexProvider Provider,
|
||||
DateTimeOffset FetchedAt,
|
||||
bool FromCache,
|
||||
bool FromOfflineSnapshot);
|
||||
@@ -0,0 +1,186 @@
|
||||
using System.Collections.Generic;
|
||||
using System.Collections.Immutable;
|
||||
using System.Linq;
|
||||
using System.Net.Http;
|
||||
using System.Runtime.CompilerServices;
|
||||
using System.Text.Json;
|
||||
using System.Xml.Linq;
|
||||
using Microsoft.Extensions.Logging;
|
||||
using Microsoft.Extensions.Options;
|
||||
using StellaOps.Vexer.Connectors.Abstractions;
|
||||
using StellaOps.Vexer.Connectors.RedHat.CSAF.Configuration;
|
||||
using StellaOps.Vexer.Connectors.RedHat.CSAF.Metadata;
|
||||
using StellaOps.Vexer.Core;
|
||||
using StellaOps.Vexer.Storage.Mongo;
|
||||
|
||||
namespace StellaOps.Vexer.Connectors.RedHat.CSAF;
|
||||
|
||||
public sealed class RedHatCsafConnector : VexConnectorBase
|
||||
{
|
||||
private readonly RedHatProviderMetadataLoader _metadataLoader;
|
||||
private readonly IHttpClientFactory _httpClientFactory;
|
||||
private readonly IVexConnectorStateRepository _stateRepository;
|
||||
public RedHatCsafConnector(
|
||||
VexConnectorDescriptor descriptor,
|
||||
RedHatProviderMetadataLoader metadataLoader,
|
||||
IHttpClientFactory httpClientFactory,
|
||||
IVexConnectorStateRepository stateRepository,
|
||||
ILogger<RedHatCsafConnector> logger,
|
||||
TimeProvider timeProvider)
|
||||
: base(descriptor, logger, timeProvider)
|
||||
{
|
||||
_metadataLoader = metadataLoader ?? throw new ArgumentNullException(nameof(metadataLoader));
|
||||
_httpClientFactory = httpClientFactory ?? throw new ArgumentNullException(nameof(httpClientFactory));
|
||||
_stateRepository = stateRepository ?? throw new ArgumentNullException(nameof(stateRepository));
|
||||
}
|
||||
|
||||
public override ValueTask ValidateAsync(VexConnectorSettings settings, CancellationToken cancellationToken)
|
||||
{
|
||||
// No connector-specific settings yet.
|
||||
return ValueTask.CompletedTask;
|
||||
}
|
||||
|
||||
public override async IAsyncEnumerable<VexRawDocument> FetchAsync(VexConnectorContext context, [EnumeratorCancellation] CancellationToken cancellationToken)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(context);
|
||||
|
||||
var metadataResult = await _metadataLoader.LoadAsync(cancellationToken).ConfigureAwait(false);
|
||||
if (metadataResult.Provider.Discovery.RolIeService is null)
|
||||
{
|
||||
throw new InvalidOperationException("Red Hat provider metadata did not specify a ROLIE feed.");
|
||||
}
|
||||
|
||||
var state = await _stateRepository.GetAsync(Descriptor.Id, cancellationToken).ConfigureAwait(false);
|
||||
|
||||
var sinceTimestamp = context.Since;
|
||||
if (state?.LastUpdated is { } persisted && (sinceTimestamp is null || persisted > sinceTimestamp))
|
||||
{
|
||||
sinceTimestamp = persisted;
|
||||
}
|
||||
|
||||
var knownDigests = state?.DocumentDigests ?? ImmutableArray<string>.Empty;
|
||||
var digestList = new List<string>(knownDigests);
|
||||
var digestSet = new HashSet<string>(knownDigests, StringComparer.OrdinalIgnoreCase);
|
||||
var latestUpdated = state?.LastUpdated ?? sinceTimestamp ?? DateTimeOffset.MinValue;
|
||||
var stateChanged = false;
|
||||
|
||||
foreach (var entry in await FetchRolieEntriesAsync(metadataResult.Provider.Discovery.RolIeService, cancellationToken).ConfigureAwait(false))
|
||||
{
|
||||
if (sinceTimestamp is not null && entry.Updated is DateTimeOffset updated && updated <= sinceTimestamp)
|
||||
{
|
||||
continue;
|
||||
}
|
||||
|
||||
if (entry.DocumentUri is null)
|
||||
{
|
||||
Logger.LogDebug("Skipping ROLIE entry {Id} because no document link was provided.", entry.Id);
|
||||
continue;
|
||||
}
|
||||
|
||||
var rawDocument = await DownloadCsafDocumentAsync(entry, cancellationToken).ConfigureAwait(false);
|
||||
|
||||
if (!digestSet.Add(rawDocument.Digest))
|
||||
{
|
||||
Logger.LogDebug("Skipping CSAF document {Uri} because digest {Digest} was already processed.", rawDocument.SourceUri, rawDocument.Digest);
|
||||
continue;
|
||||
}
|
||||
|
||||
await context.RawSink.StoreAsync(rawDocument, cancellationToken).ConfigureAwait(false);
|
||||
digestList.Add(rawDocument.Digest);
|
||||
stateChanged = true;
|
||||
|
||||
if (entry.Updated is DateTimeOffset entryUpdated && entryUpdated > latestUpdated)
|
||||
{
|
||||
latestUpdated = entryUpdated;
|
||||
}
|
||||
|
||||
yield return rawDocument;
|
||||
}
|
||||
|
||||
if (stateChanged)
|
||||
{
|
||||
var newLastUpdated = latestUpdated == DateTimeOffset.MinValue ? state?.LastUpdated : latestUpdated;
|
||||
var updatedState = new VexConnectorState(
|
||||
Descriptor.Id,
|
||||
newLastUpdated,
|
||||
digestList.ToImmutableArray());
|
||||
|
||||
await _stateRepository.SaveAsync(updatedState, cancellationToken).ConfigureAwait(false);
|
||||
}
|
||||
}
|
||||
|
||||
public override ValueTask<VexClaimBatch> NormalizeAsync(VexRawDocument document, CancellationToken cancellationToken)
|
||||
{
|
||||
// This connector relies on format-specific normalizers registered elsewhere.
|
||||
throw new NotSupportedException("RedHatCsafConnector does not perform in-line normalization; use the CSAF normalizer component.");
|
||||
}
|
||||
|
||||
private async Task<IReadOnlyList<RolieEntry>> FetchRolieEntriesAsync(Uri feedUri, CancellationToken cancellationToken)
|
||||
{
|
||||
var client = _httpClientFactory.CreateClient(RedHatConnectorOptions.HttpClientName);
|
||||
using var response = await client.GetAsync(feedUri, cancellationToken).ConfigureAwait(false);
|
||||
response.EnsureSuccessStatusCode();
|
||||
|
||||
await using var stream = await response.Content.ReadAsStreamAsync(cancellationToken).ConfigureAwait(false);
|
||||
var document = XDocument.Load(stream);
|
||||
var ns = document.Root?.Name.Namespace ?? "http://www.w3.org/2005/Atom";
|
||||
|
||||
var entries = document.Root?
|
||||
.Elements(ns + "entry")
|
||||
.Select(e => new RolieEntry(
|
||||
Id: (string?)e.Element(ns + "id"),
|
||||
Updated: ParseUpdated((string?)e.Element(ns + "updated")),
|
||||
DocumentUri: ParseDocumentLink(e, ns)))
|
||||
.Where(entry => entry.Id is not null && entry.Updated is not null)
|
||||
.OrderBy(entry => entry.Updated)
|
||||
.ToList() ?? new List<RolieEntry>();
|
||||
|
||||
return entries;
|
||||
}
|
||||
|
||||
private static DateTimeOffset? ParseUpdated(string? value)
|
||||
=> DateTimeOffset.TryParse(value, out var parsed) ? parsed : null;
|
||||
|
||||
private static Uri? ParseDocumentLink(XElement entry, XNamespace ns)
|
||||
{
|
||||
var linkElements = entry.Elements(ns + "link");
|
||||
foreach (var link in linkElements)
|
||||
{
|
||||
var rel = (string?)link.Attribute("rel");
|
||||
var href = (string?)link.Attribute("href");
|
||||
if (string.IsNullOrWhiteSpace(href))
|
||||
{
|
||||
continue;
|
||||
}
|
||||
|
||||
if (rel is null || rel.Equals("enclosure", StringComparison.OrdinalIgnoreCase) || rel.Equals("alternate", StringComparison.OrdinalIgnoreCase))
|
||||
{
|
||||
if (Uri.TryCreate(href, UriKind.Absolute, out var uri))
|
||||
{
|
||||
return uri;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
return null;
|
||||
}
|
||||
|
||||
private async Task<VexRawDocument> DownloadCsafDocumentAsync(RolieEntry entry, CancellationToken cancellationToken)
|
||||
{
|
||||
var documentUri = entry.DocumentUri ?? throw new InvalidOperationException("ROLIE entry missing document URI.");
|
||||
|
||||
var client = _httpClientFactory.CreateClient(RedHatConnectorOptions.HttpClientName);
|
||||
using var response = await client.GetAsync(documentUri, cancellationToken).ConfigureAwait(false);
|
||||
response.EnsureSuccessStatusCode();
|
||||
|
||||
var contentBytes = await response.Content.ReadAsByteArrayAsync(cancellationToken).ConfigureAwait(false);
|
||||
var metadata = BuildMetadata(builder => builder
|
||||
.Add("redhat.csaf.entryId", entry.Id)
|
||||
.Add("redhat.csaf.documentUri", documentUri.ToString())
|
||||
.Add("redhat.csaf.updated", entry.Updated?.ToString("O")));
|
||||
|
||||
return CreateRawDocument(VexDocumentFormat.Csaf, documentUri, contentBytes, metadata);
|
||||
}
|
||||
|
||||
private sealed record RolieEntry(string? Id, DateTimeOffset? Updated, Uri? DocumentUri);
|
||||
}
|
||||
@@ -0,0 +1,19 @@
|
||||
<Project Sdk="Microsoft.NET.Sdk">
|
||||
<PropertyGroup>
|
||||
<TargetFramework>net10.0</TargetFramework>
|
||||
<LangVersion>preview</LangVersion>
|
||||
<Nullable>enable</Nullable>
|
||||
<ImplicitUsings>enable</ImplicitUsings>
|
||||
<TreatWarningsAsErrors>true</TreatWarningsAsErrors>
|
||||
</PropertyGroup>
|
||||
<ItemGroup>
|
||||
<ProjectReference Include="..\StellaOps.Vexer.Connectors.Abstractions\StellaOps.Vexer.Connectors.Abstractions.csproj" />
|
||||
<ProjectReference Include="..\StellaOps.Vexer.Storage.Mongo\StellaOps.Vexer.Storage.Mongo.csproj" />
|
||||
</ItemGroup>
|
||||
<ItemGroup>
|
||||
<PackageReference Include="Microsoft.Extensions.Caching.Memory" Version="8.0.0" />
|
||||
<PackageReference Include="Microsoft.Extensions.Http" Version="8.0.0" />
|
||||
<PackageReference Include="Microsoft.Extensions.Options" Version="8.0.0" />
|
||||
<PackageReference Include="System.IO.Abstractions" Version="20.0.28" />
|
||||
</ItemGroup>
|
||||
</Project>
|
||||
@@ -2,6 +2,9 @@ If you are working on this file you need to read docs/ARCHITECTURE_VEXER.md and
|
||||
# TASKS
|
||||
| Task | Owner(s) | Depends on | Notes |
|
||||
|---|---|---|---|
|
||||
|VEXER-CONN-RH-01-001 – Provider metadata discovery|Team Vexer Connectors – Red Hat|VEXER-CONN-ABS-01-001|TODO – Implement `.well-known` metadata loader with caching, schema validation, and offline snapshot support.|
|
||||
|VEXER-CONN-RH-01-002 – Incremental CSAF pulls|Team Vexer Connectors – Red Hat|VEXER-CONN-RH-01-001, VEXER-STORAGE-01-003|TODO – Fetch CSAF windows with ETag handling, resume tokens, quarantine on schema errors, and persist raw docs.|
|
||||
|VEXER-CONN-RH-01-003 – Trust metadata emission|Team Vexer Connectors – Red Hat|VEXER-CONN-RH-01-002, VEXER-POLICY-01-001|TODO – Populate provider trust overrides (cosign issuer, identity regex) and provenance hints for policy evaluation/logging.|
|
||||
|VEXER-CONN-RH-01-001 – Provider metadata discovery|Team Vexer Connectors – Red Hat|VEXER-CONN-ABS-01-001|**DONE (2025-10-17)** – Added `RedHatProviderMetadataLoader` with HTTP/ETag caching, offline snapshot handling, and validation; exposed DI helper + tests covering live, cached, and offline scenarios.|
|
||||
|VEXER-CONN-RH-01-002 – Incremental CSAF pulls|Team Vexer Connectors – Red Hat|VEXER-CONN-RH-01-001, VEXER-STORAGE-01-003|**DONE (2025-10-17)** – Implemented `RedHatCsafConnector` with ROLIE feed parsing, incremental filtering via `context.Since`, CSAF document download + metadata capture, and persistence through `IVexRawDocumentSink`; tests cover live fetch/cache/offline scenarios with ETag handling.|
|
||||
|VEXER-CONN-RH-01-003 – Trust metadata emission|Team Vexer Connectors – Red Hat|VEXER-CONN-RH-01-002, VEXER-POLICY-01-001|**DONE (2025-10-17)** – Provider metadata loader now emits trust overrides (weight, cosign issuer/pattern, PGP fingerprints) and the connector surfaces provenance hints for policy/consensus layers.|
|
||||
|VEXER-CONN-RH-01-004 – Resume state persistence|Team Vexer Connectors – Red Hat|VEXER-CONN-RH-01-002, VEXER-STORAGE-01-003|**DONE (2025-10-17)** – Connector now loads/saves resume state via `IVexConnectorStateRepository`, tracking last update timestamp and recent document digests to avoid duplicate CSAF ingestion; regression covers state persistence and duplicate skips.|
|
||||
|VEXER-CONN-RH-01-005 – Worker/WebService integration|Team Vexer Connectors – Red Hat|VEXER-CONN-RH-01-002|**DONE (2025-10-17)** – Worker/WebService now call `AddRedHatCsafConnector`, register the connector + state repo, and default worker scheduling adds the `vexer:redhat` provider so background jobs and orchestration can activate the connector without extra wiring.|
|
||||
|VEXER-CONN-RH-01-006 – CSAF normalization parity tests|Team Vexer Connectors – Red Hat|VEXER-CONN-RH-01-002, VEXER-FMT-CSAF-01-001|**DONE (2025-10-17)** – Added RHSA fixture-driven regression verifying CSAF normalizer retains Red Hat product metadata, tracking fields, and timestamps (`rhsa-sample.json` + `CsafNormalizerTests.NormalizeAsync_PreservesRedHatSpecificMetadata`).|
|
||||
|
||||
@@ -0,0 +1,138 @@
|
||||
using System;
|
||||
using System.Net;
|
||||
using System.Net.Http;
|
||||
using System.Text;
|
||||
using System.Threading;
|
||||
using FluentAssertions;
|
||||
using Microsoft.Extensions.Caching.Memory;
|
||||
using Microsoft.Extensions.Logging.Abstractions;
|
||||
using StellaOps.Vexer.Connectors.SUSE.RancherVEXHub.Authentication;
|
||||
using StellaOps.Vexer.Connectors.SUSE.RancherVEXHub.Configuration;
|
||||
using Xunit;
|
||||
|
||||
namespace StellaOps.Vexer.Connectors.SUSE.RancherVEXHub.Tests.Authentication;
|
||||
|
||||
public sealed class RancherHubTokenProviderTests
|
||||
{
|
||||
private const string TokenResponse = "{\"access_token\":\"abc123\",\"token_type\":\"Bearer\",\"expires_in\":3600}";
|
||||
|
||||
[Fact]
|
||||
public async Task GetAccessTokenAsync_RequestsAndCachesToken()
|
||||
{
|
||||
var handler = TestHttpMessageHandler.RespondWith(request =>
|
||||
{
|
||||
request.Headers.Authorization.Should().NotBeNull();
|
||||
request.Content.Should().NotBeNull();
|
||||
|
||||
return new HttpResponseMessage(HttpStatusCode.OK)
|
||||
{
|
||||
Content = new StringContent(TokenResponse, Encoding.UTF8, "application/json"),
|
||||
};
|
||||
});
|
||||
|
||||
var client = new HttpClient(handler)
|
||||
{
|
||||
BaseAddress = new Uri("https://identity.suse.com"),
|
||||
};
|
||||
|
||||
var factory = new SingleClientHttpClientFactory(client);
|
||||
var cache = new MemoryCache(new MemoryCacheOptions());
|
||||
var provider = new RancherHubTokenProvider(factory, cache, NullLogger<RancherHubTokenProvider>.Instance);
|
||||
|
||||
var options = new RancherHubConnectorOptions
|
||||
{
|
||||
ClientId = "client",
|
||||
ClientSecret = "secret",
|
||||
TokenEndpoint = new Uri("https://identity.suse.com/oauth/token"),
|
||||
Audience = "https://vexhub.suse.com",
|
||||
};
|
||||
options.Scopes.Clear();
|
||||
options.Scopes.Add("hub.read");
|
||||
options.Scopes.Add("hub.events");
|
||||
|
||||
var token = await provider.GetAccessTokenAsync(options, CancellationToken.None);
|
||||
token.Should().NotBeNull();
|
||||
token!.Value.Should().Be("abc123");
|
||||
|
||||
var cached = await provider.GetAccessTokenAsync(options, CancellationToken.None);
|
||||
cached.Should().NotBeNull();
|
||||
handler.InvocationCount.Should().Be(1);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task GetAccessTokenAsync_ReturnsNullWhenOfflinePreferred()
|
||||
{
|
||||
var handler = TestHttpMessageHandler.RespondWith(_ => new HttpResponseMessage(HttpStatusCode.OK));
|
||||
var client = new HttpClient(handler)
|
||||
{
|
||||
BaseAddress = new Uri("https://identity.suse.com"),
|
||||
};
|
||||
|
||||
var factory = new SingleClientHttpClientFactory(client);
|
||||
var cache = new MemoryCache(new MemoryCacheOptions());
|
||||
var provider = new RancherHubTokenProvider(factory, cache, NullLogger<RancherHubTokenProvider>.Instance);
|
||||
var options = new RancherHubConnectorOptions
|
||||
{
|
||||
PreferOfflineSnapshot = true,
|
||||
ClientId = "client",
|
||||
ClientSecret = "secret",
|
||||
TokenEndpoint = new Uri("https://identity.suse.com/oauth/token"),
|
||||
};
|
||||
|
||||
var token = await provider.GetAccessTokenAsync(options, CancellationToken.None);
|
||||
token.Should().BeNull();
|
||||
handler.InvocationCount.Should().Be(0);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task GetAccessTokenAsync_ReturnsNullWithoutCredentials()
|
||||
{
|
||||
var handler = TestHttpMessageHandler.RespondWith(_ => new HttpResponseMessage(HttpStatusCode.OK));
|
||||
var client = new HttpClient(handler)
|
||||
{
|
||||
BaseAddress = new Uri("https://identity.suse.com"),
|
||||
};
|
||||
|
||||
var factory = new SingleClientHttpClientFactory(client);
|
||||
var cache = new MemoryCache(new MemoryCacheOptions());
|
||||
var provider = new RancherHubTokenProvider(factory, cache, NullLogger<RancherHubTokenProvider>.Instance);
|
||||
var options = new RancherHubConnectorOptions();
|
||||
|
||||
var token = await provider.GetAccessTokenAsync(options, CancellationToken.None);
|
||||
token.Should().BeNull();
|
||||
handler.InvocationCount.Should().Be(0);
|
||||
}
|
||||
|
||||
private sealed class SingleClientHttpClientFactory : IHttpClientFactory
|
||||
{
|
||||
private readonly HttpClient _client;
|
||||
|
||||
public SingleClientHttpClientFactory(HttpClient client)
|
||||
{
|
||||
_client = client;
|
||||
}
|
||||
|
||||
public HttpClient CreateClient(string name) => _client;
|
||||
}
|
||||
|
||||
private sealed class TestHttpMessageHandler : HttpMessageHandler
|
||||
{
|
||||
private readonly Func<HttpRequestMessage, HttpResponseMessage> _responseFactory;
|
||||
|
||||
private TestHttpMessageHandler(Func<HttpRequestMessage, HttpResponseMessage> responseFactory)
|
||||
{
|
||||
_responseFactory = responseFactory;
|
||||
}
|
||||
|
||||
public int InvocationCount { get; private set; }
|
||||
|
||||
public static TestHttpMessageHandler RespondWith(Func<HttpRequestMessage, HttpResponseMessage> responseFactory)
|
||||
=> new(responseFactory);
|
||||
|
||||
protected override Task<HttpResponseMessage> SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
|
||||
{
|
||||
InvocationCount++;
|
||||
return Task.FromResult(_responseFactory(request));
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,178 @@
|
||||
using System;
|
||||
using System.Collections.Generic;
|
||||
using System.Net;
|
||||
using System.Net.Http;
|
||||
using System.Net.Http.Headers;
|
||||
using System.Text;
|
||||
using FluentAssertions;
|
||||
using Microsoft.Extensions.Caching.Memory;
|
||||
using Microsoft.Extensions.Logging.Abstractions;
|
||||
using StellaOps.Vexer.Connectors.SUSE.RancherVEXHub.Authentication;
|
||||
using StellaOps.Vexer.Connectors.SUSE.RancherVEXHub.Configuration;
|
||||
using StellaOps.Vexer.Connectors.SUSE.RancherVEXHub.Metadata;
|
||||
using System.IO.Abstractions.TestingHelpers;
|
||||
using System.Threading;
|
||||
using Xunit;
|
||||
|
||||
namespace StellaOps.Vexer.Connectors.SUSE.RancherVEXHub.Tests.Metadata;
|
||||
|
||||
public sealed class RancherHubMetadataLoaderTests
|
||||
{
|
||||
private const string SampleDiscovery = """
|
||||
{
|
||||
"hubId": "vexer:suse.rancher",
|
||||
"title": "SUSE Rancher VEX Hub",
|
||||
"subscription": {
|
||||
"eventsUri": "https://vexhub.suse.com/api/v1/events",
|
||||
"checkpointUri": "https://vexhub.suse.com/api/v1/checkpoints",
|
||||
"requiresAuthentication": true,
|
||||
"channels": ["rke2", "k3s"],
|
||||
"scopes": ["hub.read", "hub.events"]
|
||||
},
|
||||
"authentication": {
|
||||
"tokenUri": "https://identity.suse.com/oauth2/token",
|
||||
"audience": "https://vexhub.suse.com"
|
||||
},
|
||||
"offline": {
|
||||
"snapshotUri": "https://downloads.suse.com/vexhub/snapshot.json",
|
||||
"sha256": "deadbeef",
|
||||
"updated": "2025-10-10T12:00:00Z"
|
||||
}
|
||||
}
|
||||
""";
|
||||
|
||||
[Fact]
|
||||
public async Task LoadAsync_FetchesAndCachesMetadata()
|
||||
{
|
||||
var handler = TestHttpMessageHandler.RespondWith(_ =>
|
||||
{
|
||||
var response = new HttpResponseMessage(HttpStatusCode.OK)
|
||||
{
|
||||
Content = new StringContent(SampleDiscovery, Encoding.UTF8, "application/json"),
|
||||
};
|
||||
response.Headers.ETag = new EntityTagHeaderValue("\"abc\"");
|
||||
return response;
|
||||
});
|
||||
|
||||
var client = new HttpClient(handler)
|
||||
{
|
||||
BaseAddress = new Uri("https://vexhub.suse.com"),
|
||||
};
|
||||
|
||||
var factory = new SingleClientHttpClientFactory(client);
|
||||
var cache = new MemoryCache(new MemoryCacheOptions());
|
||||
var fileSystem = new MockFileSystem();
|
||||
var offlinePath = fileSystem.Path.Combine(@"C:\offline", "rancher-hub.json");
|
||||
var options = new RancherHubConnectorOptions
|
||||
{
|
||||
DiscoveryUri = new Uri("https://vexhub.suse.com/.well-known/vex/rancher-hub.json"),
|
||||
OfflineSnapshotPath = offlinePath,
|
||||
};
|
||||
|
||||
var tokenProvider = new RancherHubTokenProvider(factory, cache, NullLogger<RancherHubTokenProvider>.Instance);
|
||||
var loader = new RancherHubMetadataLoader(factory, cache, tokenProvider, fileSystem, NullLogger<RancherHubMetadataLoader>.Instance);
|
||||
|
||||
var result = await loader.LoadAsync(options, CancellationToken.None);
|
||||
|
||||
result.FromCache.Should().BeFalse();
|
||||
result.FromOfflineSnapshot.Should().BeFalse();
|
||||
result.Metadata.Provider.DisplayName.Should().Be("SUSE Rancher VEX Hub");
|
||||
result.Metadata.Subscription.EventsUri.Should().Be(new Uri("https://vexhub.suse.com/api/v1/events"));
|
||||
result.Metadata.Authentication.TokenEndpoint.Should().Be(new Uri("https://identity.suse.com/oauth2/token"));
|
||||
|
||||
// Second call should be served from cache (no additional HTTP invocation).
|
||||
handler.ResetInvocationCount();
|
||||
await loader.LoadAsync(options, CancellationToken.None);
|
||||
handler.InvocationCount.Should().Be(0);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task LoadAsync_UsesOfflineSnapshotWhenNetworkFails()
|
||||
{
|
||||
var handler = TestHttpMessageHandler.RespondWith(_ => throw new HttpRequestException("network down"));
|
||||
var client = new HttpClient(handler)
|
||||
{
|
||||
BaseAddress = new Uri("https://vexhub.suse.com"),
|
||||
};
|
||||
|
||||
var factory = new SingleClientHttpClientFactory(client);
|
||||
var cache = new MemoryCache(new MemoryCacheOptions());
|
||||
var fileSystem = new MockFileSystem();
|
||||
var offlinePath = fileSystem.Path.Combine(@"C:\offline", "rancher-hub.json");
|
||||
fileSystem.AddFile(offlinePath, new MockFileData(SampleDiscovery));
|
||||
|
||||
var options = new RancherHubConnectorOptions
|
||||
{
|
||||
DiscoveryUri = new Uri("https://vexhub.suse.com/.well-known/vex/rancher-hub.json"),
|
||||
OfflineSnapshotPath = offlinePath,
|
||||
};
|
||||
|
||||
var tokenProvider = new RancherHubTokenProvider(factory, cache, NullLogger<RancherHubTokenProvider>.Instance);
|
||||
var loader = new RancherHubMetadataLoader(factory, cache, tokenProvider, fileSystem, NullLogger<RancherHubMetadataLoader>.Instance);
|
||||
|
||||
var result = await loader.LoadAsync(options, CancellationToken.None);
|
||||
result.FromOfflineSnapshot.Should().BeTrue();
|
||||
result.Metadata.Subscription.RequiresAuthentication.Should().BeTrue();
|
||||
result.Metadata.OfflineSnapshot.Should().NotBeNull();
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task LoadAsync_ThrowsWhenOfflinePreferredButMissing()
|
||||
{
|
||||
var handler = TestHttpMessageHandler.RespondWith(_ => throw new HttpRequestException("network down"));
|
||||
var client = new HttpClient(handler)
|
||||
{
|
||||
BaseAddress = new Uri("https://vexhub.suse.com"),
|
||||
};
|
||||
|
||||
var factory = new SingleClientHttpClientFactory(client);
|
||||
var cache = new MemoryCache(new MemoryCacheOptions());
|
||||
var fileSystem = new MockFileSystem();
|
||||
var options = new RancherHubConnectorOptions
|
||||
{
|
||||
DiscoveryUri = new Uri("https://vexhub.suse.com/.well-known/vex/rancher-hub.json"),
|
||||
OfflineSnapshotPath = "/offline/missing.json",
|
||||
PreferOfflineSnapshot = true,
|
||||
};
|
||||
|
||||
var tokenProvider = new RancherHubTokenProvider(factory, cache, NullLogger<RancherHubTokenProvider>.Instance);
|
||||
var loader = new RancherHubMetadataLoader(factory, cache, tokenProvider, fileSystem, NullLogger<RancherHubMetadataLoader>.Instance);
|
||||
|
||||
await Assert.ThrowsAsync<InvalidOperationException>(() => loader.LoadAsync(options, CancellationToken.None));
|
||||
}
|
||||
|
||||
private sealed class SingleClientHttpClientFactory : IHttpClientFactory
|
||||
{
|
||||
private readonly HttpClient _client;
|
||||
|
||||
public SingleClientHttpClientFactory(HttpClient client)
|
||||
{
|
||||
_client = client;
|
||||
}
|
||||
|
||||
public HttpClient CreateClient(string name) => _client;
|
||||
}
|
||||
|
||||
private sealed class TestHttpMessageHandler : HttpMessageHandler
|
||||
{
|
||||
private readonly Func<HttpRequestMessage, HttpResponseMessage> _responseFactory;
|
||||
|
||||
private TestHttpMessageHandler(Func<HttpRequestMessage, HttpResponseMessage> responseFactory)
|
||||
{
|
||||
_responseFactory = responseFactory;
|
||||
}
|
||||
|
||||
public int InvocationCount { get; private set; }
|
||||
|
||||
public static TestHttpMessageHandler RespondWith(Func<HttpRequestMessage, HttpResponseMessage> responseFactory)
|
||||
=> new(responseFactory);
|
||||
|
||||
public void ResetInvocationCount() => InvocationCount = 0;
|
||||
|
||||
protected override Task<HttpResponseMessage> SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
|
||||
{
|
||||
InvocationCount++;
|
||||
return Task.FromResult(_responseFactory(request));
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,17 @@
|
||||
<Project Sdk="Microsoft.NET.Sdk">
|
||||
<PropertyGroup>
|
||||
<TargetFramework>net10.0</TargetFramework>
|
||||
<LangVersion>preview</LangVersion>
|
||||
<Nullable>enable</Nullable>
|
||||
<ImplicitUsings>enable</ImplicitUsings>
|
||||
<TreatWarningsAsErrors>true</TreatWarningsAsErrors>
|
||||
</PropertyGroup>
|
||||
<ItemGroup>
|
||||
<ProjectReference Include="..\StellaOps.Vexer.Connectors.SUSE.RancherVEXHub\StellaOps.Vexer.Connectors.SUSE.RancherVEXHub.csproj" />
|
||||
<ProjectReference Include="..\StellaOps.Vexer.Storage.Mongo\StellaOps.Vexer.Storage.Mongo.csproj" />
|
||||
</ItemGroup>
|
||||
<ItemGroup>
|
||||
<PackageReference Include="FluentAssertions" Version="6.12.0" />
|
||||
<PackageReference Include="System.IO.Abstractions.TestingHelpers" Version="20.0.28" />
|
||||
</ItemGroup>
|
||||
</Project>
|
||||
@@ -0,0 +1,171 @@
|
||||
using System;
|
||||
using System.Collections.Generic;
|
||||
using System.Net.Http;
|
||||
using System.Net.Http.Headers;
|
||||
using System.Text;
|
||||
using System.Text.Json;
|
||||
using System.Threading;
|
||||
using System.Threading.Tasks;
|
||||
using Microsoft.Extensions.Caching.Memory;
|
||||
using Microsoft.Extensions.Logging;
|
||||
using StellaOps.Vexer.Connectors.SUSE.RancherVEXHub.Configuration;
|
||||
|
||||
namespace StellaOps.Vexer.Connectors.SUSE.RancherVEXHub.Authentication;
|
||||
|
||||
public sealed class RancherHubTokenProvider
|
||||
{
|
||||
private const string CachePrefix = "StellaOps.Vexer.Connectors.SUSE.RancherVEXHub.Token";
|
||||
|
||||
private readonly IHttpClientFactory _httpClientFactory;
|
||||
private readonly IMemoryCache _cache;
|
||||
private readonly ILogger<RancherHubTokenProvider> _logger;
|
||||
private readonly SemaphoreSlim _semaphore = new(1, 1);
|
||||
|
||||
public RancherHubTokenProvider(IHttpClientFactory httpClientFactory, IMemoryCache cache, ILogger<RancherHubTokenProvider> logger)
|
||||
{
|
||||
_httpClientFactory = httpClientFactory ?? throw new ArgumentNullException(nameof(httpClientFactory));
|
||||
_cache = cache ?? throw new ArgumentNullException(nameof(cache));
|
||||
_logger = logger ?? throw new ArgumentNullException(nameof(logger));
|
||||
}
|
||||
|
||||
public async ValueTask<RancherHubAccessToken?> GetAccessTokenAsync(RancherHubConnectorOptions options, CancellationToken cancellationToken)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(options);
|
||||
|
||||
if (options.PreferOfflineSnapshot)
|
||||
{
|
||||
_logger.LogDebug("Skipping token request because PreferOfflineSnapshot is enabled.");
|
||||
return null;
|
||||
}
|
||||
|
||||
var hasCredentials = !string.IsNullOrWhiteSpace(options.ClientId) &&
|
||||
!string.IsNullOrWhiteSpace(options.ClientSecret) &&
|
||||
options.TokenEndpoint is not null;
|
||||
|
||||
if (!hasCredentials)
|
||||
{
|
||||
if (!options.AllowAnonymousDiscovery)
|
||||
{
|
||||
_logger.LogDebug("No Rancher hub credentials configured; proceeding without Authorization header.");
|
||||
}
|
||||
|
||||
return null;
|
||||
}
|
||||
|
||||
var cacheKey = $"{CachePrefix}:{options.ClientId}";
|
||||
if (_cache.TryGetValue<RancherHubAccessToken>(cacheKey, out var cachedToken) && cachedToken is not null && !cachedToken.IsExpired())
|
||||
{
|
||||
return cachedToken;
|
||||
}
|
||||
|
||||
await _semaphore.WaitAsync(cancellationToken).ConfigureAwait(false);
|
||||
try
|
||||
{
|
||||
if (_cache.TryGetValue<RancherHubAccessToken>(cacheKey, out cachedToken) && cachedToken is not null && !cachedToken.IsExpired())
|
||||
{
|
||||
return cachedToken;
|
||||
}
|
||||
|
||||
var token = await RequestTokenAsync(options, cancellationToken).ConfigureAwait(false);
|
||||
if (token is not null)
|
||||
{
|
||||
var lifetime = token.ExpiresAt - DateTimeOffset.UtcNow;
|
||||
if (lifetime <= TimeSpan.Zero)
|
||||
{
|
||||
lifetime = TimeSpan.FromMinutes(5);
|
||||
}
|
||||
|
||||
var absoluteExpiration = lifetime > TimeSpan.FromSeconds(30)
|
||||
? DateTimeOffset.UtcNow + lifetime - TimeSpan.FromSeconds(30)
|
||||
: DateTimeOffset.UtcNow + TimeSpan.FromSeconds(10);
|
||||
|
||||
_cache.Set(cacheKey, token, new MemoryCacheEntryOptions
|
||||
{
|
||||
AbsoluteExpiration = absoluteExpiration,
|
||||
});
|
||||
}
|
||||
|
||||
return token;
|
||||
}
|
||||
finally
|
||||
{
|
||||
_semaphore.Release();
|
||||
}
|
||||
}
|
||||
|
||||
private async Task<RancherHubAccessToken?> RequestTokenAsync(RancherHubConnectorOptions options, CancellationToken cancellationToken)
|
||||
{
|
||||
using var request = new HttpRequestMessage(HttpMethod.Post, options.TokenEndpoint);
|
||||
request.Headers.Accept.ParseAdd("application/json");
|
||||
|
||||
var parameters = new Dictionary<string, string>
|
||||
{
|
||||
["grant_type"] = "client_credentials",
|
||||
};
|
||||
|
||||
if (options.Scopes.Count > 0)
|
||||
{
|
||||
parameters["scope"] = string.Join(' ', options.Scopes);
|
||||
}
|
||||
|
||||
if (!string.IsNullOrWhiteSpace(options.Audience))
|
||||
{
|
||||
parameters["audience"] = options.Audience!;
|
||||
}
|
||||
|
||||
if (string.Equals(options.ClientAuthenticationScheme, "client_secret_basic", StringComparison.Ordinal))
|
||||
{
|
||||
var credentials = Convert.ToBase64String(Encoding.UTF8.GetBytes($"{options.ClientId}:{options.ClientSecret}"));
|
||||
request.Headers.Authorization = new AuthenticationHeaderValue("Basic", credentials);
|
||||
}
|
||||
else
|
||||
{
|
||||
parameters["client_id"] = options.ClientId!;
|
||||
parameters["client_secret"] = options.ClientSecret!;
|
||||
}
|
||||
|
||||
request.Content = new FormUrlEncodedContent(parameters);
|
||||
|
||||
var client = _httpClientFactory.CreateClient(RancherHubConnectorOptions.HttpClientName);
|
||||
using var response = await client.SendAsync(request, cancellationToken).ConfigureAwait(false);
|
||||
if (!response.IsSuccessStatusCode)
|
||||
{
|
||||
var payload = await response.Content.ReadAsStringAsync(cancellationToken).ConfigureAwait(false);
|
||||
throw new InvalidOperationException($"Failed to acquire Rancher hub access token ({response.StatusCode}): {payload}");
|
||||
}
|
||||
|
||||
await using var stream = await response.Content.ReadAsStreamAsync(cancellationToken).ConfigureAwait(false);
|
||||
using var document = await JsonDocument.ParseAsync(stream, cancellationToken: cancellationToken).ConfigureAwait(false);
|
||||
var root = document.RootElement;
|
||||
|
||||
if (!root.TryGetProperty("access_token", out var accessTokenProperty) || accessTokenProperty.ValueKind is not JsonValueKind.String)
|
||||
{
|
||||
throw new InvalidOperationException("Token endpoint response missing access_token.");
|
||||
}
|
||||
|
||||
var token = accessTokenProperty.GetString();
|
||||
if (string.IsNullOrWhiteSpace(token))
|
||||
{
|
||||
throw new InvalidOperationException("Token endpoint response contained an empty access_token.");
|
||||
}
|
||||
|
||||
var tokenType = root.TryGetProperty("token_type", out var tokenTypeElement) && tokenTypeElement.ValueKind is JsonValueKind.String
|
||||
? tokenTypeElement.GetString() ?? "Bearer"
|
||||
: "Bearer";
|
||||
|
||||
var expires = root.TryGetProperty("expires_in", out var expiresElement) &&
|
||||
expiresElement.ValueKind is JsonValueKind.Number &&
|
||||
expiresElement.TryGetInt32(out var expiresSeconds)
|
||||
? DateTimeOffset.UtcNow + TimeSpan.FromSeconds(Math.Max(30, expiresSeconds))
|
||||
: DateTimeOffset.UtcNow + TimeSpan.FromMinutes(30);
|
||||
|
||||
_logger.LogDebug("Acquired Rancher hub access token (expires {Expires}).", expires);
|
||||
|
||||
return new RancherHubAccessToken(token, tokenType, expires);
|
||||
}
|
||||
}
|
||||
|
||||
public sealed record RancherHubAccessToken(string Value, string TokenType, DateTimeOffset ExpiresAt)
|
||||
{
|
||||
public bool IsExpired() => DateTimeOffset.UtcNow >= ExpiresAt - TimeSpan.FromMinutes(1);
|
||||
}
|
||||
@@ -0,0 +1,186 @@
|
||||
using System;
|
||||
using System.Collections.Generic;
|
||||
using System.IO;
|
||||
using System.IO.Abstractions;
|
||||
|
||||
namespace StellaOps.Vexer.Connectors.SUSE.RancherVEXHub.Configuration;
|
||||
|
||||
public sealed class RancherHubConnectorOptions
|
||||
{
|
||||
public static readonly Uri DefaultDiscoveryUri = new("https://vexhub.suse.com/.well-known/vex/rancher-hub.json");
|
||||
|
||||
/// <summary>
|
||||
/// HTTP client name registered for the connector.
|
||||
/// </summary>
|
||||
public const string HttpClientName = "vexer.connector.suse.rancherhub";
|
||||
|
||||
/// <summary>
|
||||
/// URI for the Rancher VEX hub discovery document.
|
||||
/// </summary>
|
||||
public Uri DiscoveryUri { get; set; } = DefaultDiscoveryUri;
|
||||
|
||||
/// <summary>
|
||||
/// Optional OAuth2/OIDC token endpoint used for hub authentication.
|
||||
/// </summary>
|
||||
public Uri? TokenEndpoint { get; set; }
|
||||
|
||||
/// <summary>
|
||||
/// Client identifier used when requesting hub access tokens.
|
||||
/// </summary>
|
||||
public string? ClientId { get; set; }
|
||||
|
||||
/// <summary>
|
||||
/// Client secret used when requesting hub access tokens.
|
||||
/// </summary>
|
||||
public string? ClientSecret { get; set; }
|
||||
|
||||
/// <summary>
|
||||
/// OAuth scopes requested for hub access; defaults align with Rancher hub reader role.
|
||||
/// </summary>
|
||||
public IList<string> Scopes { get; } = new List<string> { "hub.read" };
|
||||
|
||||
/// <summary>
|
||||
/// Optional audience claim passed when requesting tokens (client credential grant).
|
||||
/// </summary>
|
||||
public string? Audience { get; set; }
|
||||
|
||||
/// <summary>
|
||||
/// Preferred authentication scheme. Supported: client_secret_basic (default) or client_secret_post.
|
||||
/// </summary>
|
||||
public string ClientAuthenticationScheme { get; set; } = "client_secret_basic";
|
||||
|
||||
/// <summary>
|
||||
/// Duration to cache discovery metadata before re-fetching.
|
||||
/// </summary>
|
||||
public TimeSpan MetadataCacheDuration { get; set; } = TimeSpan.FromMinutes(30);
|
||||
|
||||
/// <summary>
|
||||
/// Optional file path for discovery metadata snapshots.
|
||||
/// </summary>
|
||||
public string? OfflineSnapshotPath { get; set; }
|
||||
|
||||
/// <summary>
|
||||
/// When true, the loader prefers the offline snapshot prior to attempting network discovery.
|
||||
/// </summary>
|
||||
public bool PreferOfflineSnapshot { get; set; }
|
||||
|
||||
/// <summary>
|
||||
/// Enables persisting freshly fetched discovery documents to <see cref="OfflineSnapshotPath"/>.
|
||||
/// </summary>
|
||||
public bool PersistOfflineSnapshot { get; set; } = true;
|
||||
|
||||
/// <summary>
|
||||
/// Weight applied to the provider entry; hubs default below direct vendor feeds.
|
||||
/// </summary>
|
||||
public double TrustWeight { get; set; } = 0.6;
|
||||
|
||||
/// <summary>
|
||||
/// Optional Sigstore/Cosign issuer for verifying hub-delivered attestations.
|
||||
/// </summary>
|
||||
public string? CosignIssuer { get; set; }
|
||||
|
||||
/// <summary>
|
||||
/// Cosign identity pattern matched against transparency log subjects.
|
||||
/// </summary>
|
||||
public string? CosignIdentityPattern { get; set; }
|
||||
|
||||
/// <summary>
|
||||
/// Additional trusted PGP fingerprints declared by the hub.
|
||||
/// </summary>
|
||||
public IList<string> PgpFingerprints { get; } = new List<string>();
|
||||
|
||||
/// <summary>
|
||||
/// Allows falling back to unauthenticated discovery requests when credentials are absent.
|
||||
/// </summary>
|
||||
public bool AllowAnonymousDiscovery { get; set; }
|
||||
|
||||
public void Validate(IFileSystem? fileSystem = null)
|
||||
{
|
||||
if (DiscoveryUri is null || !DiscoveryUri.IsAbsoluteUri)
|
||||
{
|
||||
throw new InvalidOperationException("DiscoveryUri must be an absolute URI.");
|
||||
}
|
||||
|
||||
if (DiscoveryUri.Scheme is not ("http" or "https"))
|
||||
{
|
||||
throw new InvalidOperationException("DiscoveryUri must use HTTP or HTTPS.");
|
||||
}
|
||||
|
||||
if (MetadataCacheDuration <= TimeSpan.Zero)
|
||||
{
|
||||
throw new InvalidOperationException("MetadataCacheDuration must be positive.");
|
||||
}
|
||||
|
||||
if (!string.IsNullOrWhiteSpace(OfflineSnapshotPath))
|
||||
{
|
||||
var fs = fileSystem ?? new FileSystem();
|
||||
var directory = Path.GetDirectoryName(OfflineSnapshotPath);
|
||||
if (!string.IsNullOrWhiteSpace(directory) && !fs.Directory.Exists(directory))
|
||||
{
|
||||
fs.Directory.CreateDirectory(directory);
|
||||
}
|
||||
}
|
||||
|
||||
if (PreferOfflineSnapshot && string.IsNullOrWhiteSpace(OfflineSnapshotPath))
|
||||
{
|
||||
throw new InvalidOperationException("OfflineSnapshotPath must be provided when PreferOfflineSnapshot is enabled.");
|
||||
}
|
||||
|
||||
var hasClientId = !string.IsNullOrWhiteSpace(ClientId);
|
||||
var hasClientSecret = !string.IsNullOrWhiteSpace(ClientSecret);
|
||||
var hasTokenEndpoint = TokenEndpoint is not null;
|
||||
if (hasClientId || hasClientSecret || hasTokenEndpoint)
|
||||
{
|
||||
if (!(hasClientId && hasClientSecret && hasTokenEndpoint))
|
||||
{
|
||||
throw new InvalidOperationException("ClientId, ClientSecret, and TokenEndpoint must be provided together for authenticated discovery.");
|
||||
}
|
||||
|
||||
if (TokenEndpoint is not null && (!TokenEndpoint.IsAbsoluteUri || TokenEndpoint.Scheme is not ("http" or "https")))
|
||||
{
|
||||
throw new InvalidOperationException("TokenEndpoint must be an absolute HTTP(S) URI.");
|
||||
}
|
||||
}
|
||||
|
||||
if (double.IsNaN(TrustWeight) || double.IsInfinity(TrustWeight))
|
||||
{
|
||||
TrustWeight = 0.6;
|
||||
}
|
||||
else if (TrustWeight <= 0)
|
||||
{
|
||||
TrustWeight = 0.1;
|
||||
}
|
||||
else if (TrustWeight > 1.0)
|
||||
{
|
||||
TrustWeight = 1.0;
|
||||
}
|
||||
|
||||
if (!string.IsNullOrWhiteSpace(CosignIssuer) && string.IsNullOrWhiteSpace(CosignIdentityPattern))
|
||||
{
|
||||
throw new InvalidOperationException("CosignIdentityPattern must be provided when CosignIssuer is specified.");
|
||||
}
|
||||
|
||||
if (!string.Equals(ClientAuthenticationScheme, "client_secret_basic", StringComparison.Ordinal) &&
|
||||
!string.Equals(ClientAuthenticationScheme, "client_secret_post", StringComparison.Ordinal))
|
||||
{
|
||||
throw new InvalidOperationException("ClientAuthenticationScheme must be 'client_secret_basic' or 'client_secret_post'.");
|
||||
}
|
||||
|
||||
// Remove any empty scopes to avoid token request issues.
|
||||
if (Scopes.Count > 0)
|
||||
{
|
||||
for (var i = Scopes.Count - 1; i >= 0; i--)
|
||||
{
|
||||
if (string.IsNullOrWhiteSpace(Scopes[i]))
|
||||
{
|
||||
Scopes.RemoveAt(i);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
if (Scopes.Count == 0)
|
||||
{
|
||||
Scopes.Add("hub.read");
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,32 @@
|
||||
using System;
|
||||
using System.Collections.Generic;
|
||||
using System.IO.Abstractions;
|
||||
using StellaOps.Vexer.Connectors.Abstractions;
|
||||
|
||||
namespace StellaOps.Vexer.Connectors.SUSE.RancherVEXHub.Configuration;
|
||||
|
||||
public sealed class RancherHubConnectorOptionsValidator : IVexConnectorOptionsValidator<RancherHubConnectorOptions>
|
||||
{
|
||||
private readonly IFileSystem _fileSystem;
|
||||
|
||||
public RancherHubConnectorOptionsValidator(IFileSystem fileSystem)
|
||||
{
|
||||
_fileSystem = fileSystem ?? throw new ArgumentNullException(nameof(fileSystem));
|
||||
}
|
||||
|
||||
public void Validate(VexConnectorDescriptor descriptor, RancherHubConnectorOptions options, IList<string> errors)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(descriptor);
|
||||
ArgumentNullException.ThrowIfNull(options);
|
||||
ArgumentNullException.ThrowIfNull(errors);
|
||||
|
||||
try
|
||||
{
|
||||
options.Validate(_fileSystem);
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
errors.Add(ex.Message);
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,49 @@
|
||||
using System;
|
||||
using System.Net;
|
||||
using System.Net.Http;
|
||||
using Microsoft.Extensions.Caching.Memory;
|
||||
using Microsoft.Extensions.DependencyInjection;
|
||||
using Microsoft.Extensions.DependencyInjection.Extensions;
|
||||
using StellaOps.Vexer.Connectors.Abstractions;
|
||||
using StellaOps.Vexer.Connectors.SUSE.RancherVEXHub.Authentication;
|
||||
using StellaOps.Vexer.Connectors.SUSE.RancherVEXHub.Configuration;
|
||||
using StellaOps.Vexer.Connectors.SUSE.RancherVEXHub.Metadata;
|
||||
using StellaOps.Vexer.Core;
|
||||
using System.IO.Abstractions;
|
||||
|
||||
namespace StellaOps.Vexer.Connectors.SUSE.RancherVEXHub.DependencyInjection;
|
||||
|
||||
public static class RancherHubConnectorServiceCollectionExtensions
|
||||
{
|
||||
public static IServiceCollection AddRancherHubConnector(this IServiceCollection services, Action<RancherHubConnectorOptions>? configure = null)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(services);
|
||||
|
||||
services.TryAddSingleton<IMemoryCache, MemoryCache>();
|
||||
services.TryAddSingleton<IFileSystem, FileSystem>();
|
||||
|
||||
services.AddOptions<RancherHubConnectorOptions>()
|
||||
.Configure(options =>
|
||||
{
|
||||
configure?.Invoke(options);
|
||||
});
|
||||
|
||||
services.AddSingleton<IVexConnectorOptionsValidator<RancherHubConnectorOptions>, RancherHubConnectorOptionsValidator>();
|
||||
services.AddSingleton<RancherHubTokenProvider>();
|
||||
services.AddSingleton<RancherHubMetadataLoader>();
|
||||
services.AddSingleton<IVexConnector, RancherHubConnector>();
|
||||
|
||||
services.AddHttpClient(RancherHubConnectorOptions.HttpClientName, client =>
|
||||
{
|
||||
client.Timeout = TimeSpan.FromSeconds(30);
|
||||
client.DefaultRequestHeaders.UserAgent.ParseAdd("StellaOps.Vexer.Connectors.SUSE.RancherVEXHub/1.0");
|
||||
client.DefaultRequestHeaders.Accept.ParseAdd("application/json");
|
||||
})
|
||||
.ConfigurePrimaryHttpMessageHandler(() => new HttpClientHandler
|
||||
{
|
||||
AutomaticDecompression = DecompressionMethods.All,
|
||||
});
|
||||
|
||||
return services;
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,455 @@
|
||||
using System;
|
||||
using System.Collections.Generic;
|
||||
using System.Collections.Immutable;
|
||||
using System.IO.Abstractions;
|
||||
using System.Net;
|
||||
using System.Net.Http;
|
||||
using System.Net.Http.Headers;
|
||||
using System.Text.Json;
|
||||
using System.Threading;
|
||||
using System.Threading.Tasks;
|
||||
using Microsoft.Extensions.Caching.Memory;
|
||||
using Microsoft.Extensions.Logging;
|
||||
using StellaOps.Vexer.Connectors.SUSE.RancherVEXHub.Authentication;
|
||||
using StellaOps.Vexer.Connectors.SUSE.RancherVEXHub.Configuration;
|
||||
using StellaOps.Vexer.Core;
|
||||
|
||||
namespace StellaOps.Vexer.Connectors.SUSE.RancherVEXHub.Metadata;
|
||||
|
||||
public sealed class RancherHubMetadataLoader
|
||||
{
|
||||
public const string CachePrefix = "StellaOps.Vexer.Connectors.SUSE.RancherVEXHub.Metadata";
|
||||
|
||||
private readonly IHttpClientFactory _httpClientFactory;
|
||||
private readonly IMemoryCache _memoryCache;
|
||||
private readonly RancherHubTokenProvider _tokenProvider;
|
||||
private readonly IFileSystem _fileSystem;
|
||||
private readonly ILogger<RancherHubMetadataLoader> _logger;
|
||||
private readonly SemaphoreSlim _semaphore = new(1, 1);
|
||||
private readonly JsonDocumentOptions _documentOptions;
|
||||
|
||||
public RancherHubMetadataLoader(
|
||||
IHttpClientFactory httpClientFactory,
|
||||
IMemoryCache memoryCache,
|
||||
RancherHubTokenProvider tokenProvider,
|
||||
IFileSystem fileSystem,
|
||||
ILogger<RancherHubMetadataLoader> logger)
|
||||
{
|
||||
_httpClientFactory = httpClientFactory ?? throw new ArgumentNullException(nameof(httpClientFactory));
|
||||
_memoryCache = memoryCache ?? throw new ArgumentNullException(nameof(memoryCache));
|
||||
_tokenProvider = tokenProvider ?? throw new ArgumentNullException(nameof(tokenProvider));
|
||||
_fileSystem = fileSystem ?? throw new ArgumentNullException(nameof(fileSystem));
|
||||
_logger = logger ?? throw new ArgumentNullException(nameof(logger));
|
||||
_documentOptions = new JsonDocumentOptions
|
||||
{
|
||||
CommentHandling = JsonCommentHandling.Skip,
|
||||
AllowTrailingCommas = true,
|
||||
};
|
||||
}
|
||||
|
||||
public async Task<RancherHubMetadataResult> LoadAsync(RancherHubConnectorOptions options, CancellationToken cancellationToken)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(options);
|
||||
|
||||
var cacheKey = CreateCacheKey(options);
|
||||
if (_memoryCache.TryGetValue<CacheEntry>(cacheKey, out var cached) && cached is not null && !cached.IsExpired())
|
||||
{
|
||||
_logger.LogDebug("Returning cached Rancher hub metadata (expires {Expires}).", cached.ExpiresAt);
|
||||
return new RancherHubMetadataResult(cached.Metadata, cached.FetchedAt, FromCache: true, cached.FromOfflineSnapshot);
|
||||
}
|
||||
|
||||
await _semaphore.WaitAsync(cancellationToken).ConfigureAwait(false);
|
||||
try
|
||||
{
|
||||
if (_memoryCache.TryGetValue<CacheEntry>(cacheKey, out cached) && cached is not null && !cached.IsExpired())
|
||||
{
|
||||
return new RancherHubMetadataResult(cached.Metadata, cached.FetchedAt, FromCache: true, cached.FromOfflineSnapshot);
|
||||
}
|
||||
|
||||
CacheEntry? previous = cached;
|
||||
CacheEntry? entry = null;
|
||||
|
||||
if (options.PreferOfflineSnapshot)
|
||||
{
|
||||
entry = TryLoadFromOffline(options);
|
||||
if (entry is null)
|
||||
{
|
||||
throw new InvalidOperationException("PreferOfflineSnapshot is enabled but no offline discovery snapshot was found.");
|
||||
}
|
||||
}
|
||||
else
|
||||
{
|
||||
entry = await TryFetchFromNetworkAsync(options, previous, cancellationToken).ConfigureAwait(false);
|
||||
if (entry is null)
|
||||
{
|
||||
entry = TryLoadFromOffline(options);
|
||||
}
|
||||
}
|
||||
|
||||
if (entry is null)
|
||||
{
|
||||
throw new InvalidOperationException("Unable to load Rancher hub discovery metadata from network or offline snapshot.");
|
||||
}
|
||||
|
||||
_memoryCache.Set(cacheKey, entry, new MemoryCacheEntryOptions
|
||||
{
|
||||
AbsoluteExpirationRelativeToNow = options.MetadataCacheDuration,
|
||||
});
|
||||
|
||||
return new RancherHubMetadataResult(entry.Metadata, entry.FetchedAt, FromCache: false, entry.FromOfflineSnapshot);
|
||||
}
|
||||
finally
|
||||
{
|
||||
_semaphore.Release();
|
||||
}
|
||||
}
|
||||
|
||||
private async Task<CacheEntry?> TryFetchFromNetworkAsync(RancherHubConnectorOptions options, CacheEntry? previous, CancellationToken cancellationToken)
|
||||
{
|
||||
try
|
||||
{
|
||||
var client = _httpClientFactory.CreateClient(RancherHubConnectorOptions.HttpClientName);
|
||||
using var request = new HttpRequestMessage(HttpMethod.Get, options.DiscoveryUri);
|
||||
request.Headers.Accept.ParseAdd("application/json");
|
||||
|
||||
if (!string.IsNullOrWhiteSpace(previous?.ETag) && EntityTagHeaderValue.TryParse(previous.ETag, out var etag))
|
||||
{
|
||||
request.Headers.IfNoneMatch.Add(etag);
|
||||
}
|
||||
|
||||
var token = await _tokenProvider.GetAccessTokenAsync(options, cancellationToken).ConfigureAwait(false);
|
||||
if (token is not null)
|
||||
{
|
||||
var scheme = string.IsNullOrWhiteSpace(token.TokenType) ? "Bearer" : token.TokenType;
|
||||
request.Headers.Authorization = new AuthenticationHeaderValue(scheme, token.Value);
|
||||
}
|
||||
|
||||
using var response = await client.SendAsync(request, HttpCompletionOption.ResponseHeadersRead, cancellationToken).ConfigureAwait(false);
|
||||
|
||||
if (response.StatusCode == HttpStatusCode.NotModified && previous is not null)
|
||||
{
|
||||
_logger.LogDebug("Rancher hub discovery document not modified (etag {ETag}).", previous.ETag);
|
||||
return previous with
|
||||
{
|
||||
FetchedAt = DateTimeOffset.UtcNow,
|
||||
ExpiresAt = DateTimeOffset.UtcNow + options.MetadataCacheDuration,
|
||||
FromOfflineSnapshot = false,
|
||||
};
|
||||
}
|
||||
|
||||
response.EnsureSuccessStatusCode();
|
||||
var payload = await response.Content.ReadAsStringAsync(cancellationToken).ConfigureAwait(false);
|
||||
var metadata = ParseMetadata(payload, options);
|
||||
var entry = new CacheEntry(
|
||||
metadata,
|
||||
DateTimeOffset.UtcNow,
|
||||
DateTimeOffset.UtcNow + options.MetadataCacheDuration,
|
||||
response.Headers.ETag?.ToString(),
|
||||
FromOfflineSnapshot: false,
|
||||
Payload: payload);
|
||||
|
||||
PersistOfflineSnapshot(options, payload);
|
||||
return entry;
|
||||
}
|
||||
catch (Exception ex) when (ex is not OperationCanceledException && !options.PreferOfflineSnapshot)
|
||||
{
|
||||
_logger.LogWarning(ex, "Failed to fetch Rancher hub discovery document; attempting offline snapshot fallback.");
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
private CacheEntry? TryLoadFromOffline(RancherHubConnectorOptions options)
|
||||
{
|
||||
if (string.IsNullOrWhiteSpace(options.OfflineSnapshotPath))
|
||||
{
|
||||
return null;
|
||||
}
|
||||
|
||||
if (!_fileSystem.File.Exists(options.OfflineSnapshotPath))
|
||||
{
|
||||
_logger.LogWarning("Rancher hub offline snapshot path {Path} does not exist.", options.OfflineSnapshotPath);
|
||||
return null;
|
||||
}
|
||||
|
||||
try
|
||||
{
|
||||
var payload = _fileSystem.File.ReadAllText(options.OfflineSnapshotPath);
|
||||
var metadata = ParseMetadata(payload, options);
|
||||
return new CacheEntry(
|
||||
metadata,
|
||||
DateTimeOffset.UtcNow,
|
||||
DateTimeOffset.UtcNow + options.MetadataCacheDuration,
|
||||
ETag: null,
|
||||
FromOfflineSnapshot: true,
|
||||
Payload: payload);
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
_logger.LogError(ex, "Failed to load Rancher hub discovery metadata from offline snapshot {Path}.", options.OfflineSnapshotPath);
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
private void PersistOfflineSnapshot(RancherHubConnectorOptions options, string payload)
|
||||
{
|
||||
if (!options.PersistOfflineSnapshot || string.IsNullOrWhiteSpace(options.OfflineSnapshotPath))
|
||||
{
|
||||
return;
|
||||
}
|
||||
|
||||
try
|
||||
{
|
||||
var directory = _fileSystem.Path.GetDirectoryName(options.OfflineSnapshotPath);
|
||||
if (!string.IsNullOrWhiteSpace(directory))
|
||||
{
|
||||
_fileSystem.Directory.CreateDirectory(directory);
|
||||
}
|
||||
|
||||
_fileSystem.File.WriteAllText(options.OfflineSnapshotPath, payload);
|
||||
_logger.LogDebug("Persisted Rancher hub discovery snapshot to {Path}.", options.OfflineSnapshotPath);
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
_logger.LogWarning(ex, "Failed to persist Rancher hub discovery snapshot to {Path}.", options.OfflineSnapshotPath);
|
||||
}
|
||||
}
|
||||
|
||||
private RancherHubMetadata ParseMetadata(string payload, RancherHubConnectorOptions options)
|
||||
{
|
||||
if (string.IsNullOrWhiteSpace(payload))
|
||||
{
|
||||
throw new InvalidOperationException("Rancher hub discovery payload was empty.");
|
||||
}
|
||||
|
||||
try
|
||||
{
|
||||
using var document = JsonDocument.Parse(payload, _documentOptions);
|
||||
var root = document.RootElement;
|
||||
|
||||
var hubId = ReadString(root, "hubId") ?? "vexer:suse:rancher";
|
||||
var title = ReadString(root, "title") ?? ReadString(root, "displayName") ?? "SUSE Rancher VEX Hub";
|
||||
var baseUri = ReadUri(root, "baseUri");
|
||||
|
||||
var subscriptionElement = TryGetProperty(root, "subscription");
|
||||
if (!subscriptionElement.HasValue)
|
||||
{
|
||||
throw new InvalidOperationException("Discovery payload missing subscription section.");
|
||||
}
|
||||
|
||||
var subscription = subscriptionElement.Value;
|
||||
var eventsUri = ReadRequiredUri(subscription, "eventsUri", "eventsUrl", "eventsEndpoint");
|
||||
var checkpointUri = ReadUri(subscription, "checkpointUri", "checkpointUrl", "checkpointEndpoint");
|
||||
var channels = ReadStringArray(subscription, "channels", "defaultChannels", "products");
|
||||
var scopes = ReadStringArray(subscription, "scopes", "defaultScopes");
|
||||
var requiresAuth = ReadBoolean(subscription, "requiresAuthentication", defaultValue: options.TokenEndpoint is not null);
|
||||
|
||||
var authenticationElement = TryGetProperty(root, "authentication");
|
||||
var tokenEndpointFromMetadata = authenticationElement.HasValue
|
||||
? ReadUri(authenticationElement.Value, "tokenUri", "tokenEndpoint") ?? options.TokenEndpoint
|
||||
: options.TokenEndpoint;
|
||||
var audience = authenticationElement.HasValue
|
||||
? ReadString(authenticationElement.Value, "audience", "aud") ?? options.Audience
|
||||
: options.Audience;
|
||||
|
||||
var offlineElement = TryGetProperty(root, "offline", "snapshot");
|
||||
var offlineSnapshot = offlineElement.HasValue
|
||||
? BuildOfflineSnapshot(offlineElement.Value, options)
|
||||
: null;
|
||||
|
||||
var provider = BuildProvider(hubId, title, baseUri, eventsUri, options);
|
||||
var subscriptionMetadata = new RancherHubSubscriptionMetadata(eventsUri, checkpointUri, channels, scopes, requiresAuth);
|
||||
var authenticationMetadata = new RancherHubAuthenticationMetadata(tokenEndpointFromMetadata, audience);
|
||||
|
||||
return new RancherHubMetadata(provider, subscriptionMetadata, authenticationMetadata, offlineSnapshot);
|
||||
}
|
||||
catch (JsonException ex)
|
||||
{
|
||||
throw new InvalidOperationException("Failed to parse Rancher hub discovery payload.", ex);
|
||||
}
|
||||
}
|
||||
|
||||
private RancherHubOfflineSnapshotMetadata? BuildOfflineSnapshot(JsonElement element, RancherHubConnectorOptions options)
|
||||
{
|
||||
var snapshotUri = ReadUri(element, "snapshotUri", "uri", "url");
|
||||
if (snapshotUri is null)
|
||||
{
|
||||
return null;
|
||||
}
|
||||
|
||||
var checksum = ReadString(element, "sha256", "checksum", "digest");
|
||||
DateTimeOffset? updatedAt = null;
|
||||
var updatedString = ReadString(element, "updated", "lastModified", "timestamp");
|
||||
if (!string.IsNullOrWhiteSpace(updatedString) && DateTimeOffset.TryParse(updatedString, out var parsed))
|
||||
{
|
||||
updatedAt = parsed;
|
||||
}
|
||||
|
||||
return new RancherHubOfflineSnapshotMetadata(snapshotUri, checksum, updatedAt);
|
||||
}
|
||||
|
||||
private VexProvider BuildProvider(string hubId, string title, Uri? baseUri, Uri eventsUri, RancherHubConnectorOptions options)
|
||||
{
|
||||
var baseUris = new List<Uri>();
|
||||
if (baseUri is not null)
|
||||
{
|
||||
baseUris.Add(baseUri);
|
||||
}
|
||||
baseUris.Add(eventsUri);
|
||||
|
||||
VexCosignTrust? cosign = null;
|
||||
if (!string.IsNullOrWhiteSpace(options.CosignIssuer) && !string.IsNullOrWhiteSpace(options.CosignIdentityPattern))
|
||||
{
|
||||
cosign = new VexCosignTrust(options.CosignIssuer!, options.CosignIdentityPattern!);
|
||||
}
|
||||
|
||||
var trust = new VexProviderTrust(options.TrustWeight, cosign, options.PgpFingerprints);
|
||||
return new VexProvider(hubId, title, VexProviderKind.Hub, baseUris, new VexProviderDiscovery(options.DiscoveryUri, null), trust);
|
||||
}
|
||||
|
||||
private static string CreateCacheKey(RancherHubConnectorOptions options)
|
||||
=> $"{CachePrefix}:{options.DiscoveryUri}";
|
||||
|
||||
private static JsonElement? TryGetProperty(JsonElement element, params string[] propertyNames)
|
||||
{
|
||||
foreach (var name in propertyNames)
|
||||
{
|
||||
if (element.TryGetProperty(name, out var value) && value.ValueKind is not JsonValueKind.Null and not JsonValueKind.Undefined)
|
||||
{
|
||||
return value;
|
||||
}
|
||||
}
|
||||
|
||||
return null;
|
||||
}
|
||||
|
||||
private static string? ReadString(JsonElement element, params string[] propertyNames)
|
||||
{
|
||||
var property = TryGetProperty(element, propertyNames);
|
||||
if (property is null || property.Value.ValueKind is not JsonValueKind.String)
|
||||
{
|
||||
return null;
|
||||
}
|
||||
|
||||
var value = property.Value.GetString();
|
||||
return string.IsNullOrWhiteSpace(value) ? null : value;
|
||||
}
|
||||
|
||||
private static bool ReadBoolean(JsonElement element, string propertyName, bool defaultValue)
|
||||
{
|
||||
if (!element.TryGetProperty(propertyName, out var property))
|
||||
{
|
||||
return defaultValue;
|
||||
}
|
||||
|
||||
return property.ValueKind switch
|
||||
{
|
||||
JsonValueKind.True => true,
|
||||
JsonValueKind.False => false,
|
||||
JsonValueKind.String when bool.TryParse(property.GetString(), out var parsed) => parsed,
|
||||
_ => defaultValue,
|
||||
};
|
||||
}
|
||||
|
||||
private static ImmutableArray<string> ReadStringArray(JsonElement element, params string[] propertyNames)
|
||||
{
|
||||
var property = TryGetProperty(element, propertyNames);
|
||||
if (property is null)
|
||||
{
|
||||
return ImmutableArray<string>.Empty;
|
||||
}
|
||||
|
||||
if (property.Value.ValueKind is JsonValueKind.Array)
|
||||
{
|
||||
var builder = ImmutableArray.CreateBuilder<string>();
|
||||
foreach (var item in property.Value.EnumerateArray())
|
||||
{
|
||||
if (item.ValueKind is JsonValueKind.String)
|
||||
{
|
||||
var value = item.GetString();
|
||||
if (!string.IsNullOrWhiteSpace(value))
|
||||
{
|
||||
builder.Add(value!);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
return builder.Count == 0 ? ImmutableArray<string>.Empty : builder.ToImmutable();
|
||||
}
|
||||
|
||||
if (property.Value.ValueKind is JsonValueKind.String)
|
||||
{
|
||||
var single = property.Value.GetString();
|
||||
return string.IsNullOrWhiteSpace(single)
|
||||
? ImmutableArray<string>.Empty
|
||||
: ImmutableArray.Create(single!);
|
||||
}
|
||||
|
||||
return ImmutableArray<string>.Empty;
|
||||
}
|
||||
|
||||
private static Uri? ReadUri(JsonElement element, params string[] propertyNames)
|
||||
{
|
||||
var value = ReadString(element, propertyNames);
|
||||
if (string.IsNullOrWhiteSpace(value))
|
||||
{
|
||||
return null;
|
||||
}
|
||||
|
||||
if (!Uri.TryCreate(value, UriKind.Absolute, out var uri) || uri.Scheme is not ("http" or "https"))
|
||||
{
|
||||
throw new InvalidOperationException($"Discovery field '{string.Join("/", propertyNames)}' must be an absolute HTTP(S) URI.");
|
||||
}
|
||||
|
||||
return uri;
|
||||
}
|
||||
|
||||
private static Uri ReadRequiredUri(JsonElement element, params string[] propertyNames)
|
||||
{
|
||||
var uri = ReadUri(element, propertyNames);
|
||||
if (uri is null)
|
||||
{
|
||||
throw new InvalidOperationException($"Discovery payload missing required URI field '{string.Join("/", propertyNames)}'.");
|
||||
}
|
||||
|
||||
return uri;
|
||||
}
|
||||
|
||||
private sealed record CacheEntry(
|
||||
RancherHubMetadata Metadata,
|
||||
DateTimeOffset FetchedAt,
|
||||
DateTimeOffset ExpiresAt,
|
||||
string? ETag,
|
||||
bool FromOfflineSnapshot,
|
||||
string? Payload)
|
||||
{
|
||||
public bool IsExpired() => DateTimeOffset.UtcNow >= ExpiresAt;
|
||||
}
|
||||
}
|
||||
|
||||
public sealed record RancherHubMetadata(
|
||||
VexProvider Provider,
|
||||
RancherHubSubscriptionMetadata Subscription,
|
||||
RancherHubAuthenticationMetadata Authentication,
|
||||
RancherHubOfflineSnapshotMetadata? OfflineSnapshot);
|
||||
|
||||
public sealed record RancherHubSubscriptionMetadata(
|
||||
Uri EventsUri,
|
||||
Uri? CheckpointUri,
|
||||
ImmutableArray<string> Channels,
|
||||
ImmutableArray<string> Scopes,
|
||||
bool RequiresAuthentication);
|
||||
|
||||
public sealed record RancherHubAuthenticationMetadata(
|
||||
Uri? TokenEndpoint,
|
||||
string? Audience);
|
||||
|
||||
public sealed record RancherHubOfflineSnapshotMetadata(
|
||||
Uri SnapshotUri,
|
||||
string? Sha256,
|
||||
DateTimeOffset? UpdatedAt);
|
||||
|
||||
public sealed record RancherHubMetadataResult(
|
||||
RancherHubMetadata Metadata,
|
||||
DateTimeOffset FetchedAt,
|
||||
bool FromCache,
|
||||
bool FromOfflineSnapshot);
|
||||
@@ -0,0 +1,85 @@
|
||||
using System;
|
||||
using System.Collections.Generic;
|
||||
using System.Collections.Immutable;
|
||||
using System.Runtime.CompilerServices;
|
||||
using Microsoft.Extensions.Logging;
|
||||
using StellaOps.Vexer.Connectors.Abstractions;
|
||||
using StellaOps.Vexer.Connectors.SUSE.RancherVEXHub.Configuration;
|
||||
using StellaOps.Vexer.Connectors.SUSE.RancherVEXHub.Metadata;
|
||||
using StellaOps.Vexer.Core;
|
||||
|
||||
namespace StellaOps.Vexer.Connectors.SUSE.RancherVEXHub;
|
||||
|
||||
public sealed class RancherHubConnector : VexConnectorBase
|
||||
{
|
||||
private static readonly VexConnectorDescriptor StaticDescriptor = new(
|
||||
id: "vexer:suse.rancher",
|
||||
kind: VexProviderKind.Hub,
|
||||
displayName: "SUSE Rancher VEX Hub")
|
||||
{
|
||||
Tags = ImmutableArray.Create("hub", "suse", "offline"),
|
||||
};
|
||||
|
||||
private readonly RancherHubMetadataLoader _metadataLoader;
|
||||
private readonly IEnumerable<IVexConnectorOptionsValidator<RancherHubConnectorOptions>> _validators;
|
||||
|
||||
private RancherHubConnectorOptions? _options;
|
||||
private RancherHubMetadataResult? _metadata;
|
||||
|
||||
public RancherHubConnector(
|
||||
RancherHubMetadataLoader metadataLoader,
|
||||
ILogger<RancherHubConnector> logger,
|
||||
TimeProvider timeProvider,
|
||||
IEnumerable<IVexConnectorOptionsValidator<RancherHubConnectorOptions>>? validators = null)
|
||||
: base(StaticDescriptor, logger, timeProvider)
|
||||
{
|
||||
_metadataLoader = metadataLoader ?? throw new ArgumentNullException(nameof(metadataLoader));
|
||||
_validators = validators ?? Array.Empty<IVexConnectorOptionsValidator<RancherHubConnectorOptions>>();
|
||||
}
|
||||
|
||||
public override async ValueTask ValidateAsync(VexConnectorSettings settings, CancellationToken cancellationToken)
|
||||
{
|
||||
_options = VexConnectorOptionsBinder.Bind(
|
||||
Descriptor,
|
||||
settings,
|
||||
validators: _validators);
|
||||
|
||||
_metadata = await _metadataLoader.LoadAsync(_options, cancellationToken).ConfigureAwait(false);
|
||||
|
||||
LogConnectorEvent(LogLevel.Information, "validate", "Rancher hub discovery loaded.", new Dictionary<string, object?>
|
||||
{
|
||||
["discoveryUri"] = _options.DiscoveryUri.ToString(),
|
||||
["subscriptionUri"] = _metadata.Metadata.Subscription.EventsUri.ToString(),
|
||||
["requiresAuth"] = _metadata.Metadata.Subscription.RequiresAuthentication,
|
||||
["fromOffline"] = _metadata.FromOfflineSnapshot,
|
||||
});
|
||||
}
|
||||
|
||||
public override async IAsyncEnumerable<VexRawDocument> FetchAsync(VexConnectorContext context, [EnumeratorCancellation] CancellationToken cancellationToken)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(context);
|
||||
|
||||
if (_options is null)
|
||||
{
|
||||
throw new InvalidOperationException("Connector must be validated before fetch operations.");
|
||||
}
|
||||
|
||||
if (_metadata is null)
|
||||
{
|
||||
_metadata = await _metadataLoader.LoadAsync(_options, cancellationToken).ConfigureAwait(false);
|
||||
}
|
||||
|
||||
LogConnectorEvent(LogLevel.Debug, "fetch", "Rancher hub connector discovery ready; event ingestion will be implemented in VEXER-CONN-SUSE-01-002.", new Dictionary<string, object?>
|
||||
{
|
||||
["since"] = context.Since?.ToString("O"),
|
||||
["subscriptionUri"] = _metadata.Metadata.Subscription.EventsUri.ToString(),
|
||||
});
|
||||
|
||||
yield break;
|
||||
}
|
||||
|
||||
public override ValueTask<VexClaimBatch> NormalizeAsync(VexRawDocument document, CancellationToken cancellationToken)
|
||||
=> throw new NotSupportedException("RancherHubConnector relies on format-specific normalizers for CSAF/OpenVEX payloads.");
|
||||
|
||||
public RancherHubMetadata? GetCachedMetadata() => _metadata?.Metadata;
|
||||
}
|
||||
@@ -0,0 +1,19 @@
|
||||
<Project Sdk="Microsoft.NET.Sdk">
|
||||
<PropertyGroup>
|
||||
<TargetFramework>net10.0</TargetFramework>
|
||||
<LangVersion>preview</LangVersion>
|
||||
<Nullable>enable</Nullable>
|
||||
<ImplicitUsings>enable</ImplicitUsings>
|
||||
<TreatWarningsAsErrors>true</TreatWarningsAsErrors>
|
||||
</PropertyGroup>
|
||||
<ItemGroup>
|
||||
<ProjectReference Include="..\StellaOps.Vexer.Connectors.Abstractions\StellaOps.Vexer.Connectors.Abstractions.csproj" />
|
||||
<ProjectReference Include="..\StellaOps.Vexer.Storage.Mongo\StellaOps.Vexer.Storage.Mongo.csproj" />
|
||||
</ItemGroup>
|
||||
<ItemGroup>
|
||||
<PackageReference Include="Microsoft.Extensions.Caching.Memory" Version="8.0.0" />
|
||||
<PackageReference Include="Microsoft.Extensions.Http" Version="8.0.0" />
|
||||
<PackageReference Include="Microsoft.Extensions.Options" Version="8.0.0" />
|
||||
<PackageReference Include="System.IO.Abstractions" Version="20.0.28" />
|
||||
</ItemGroup>
|
||||
</Project>
|
||||
@@ -2,6 +2,6 @@ If you are working on this file you need to read docs/ARCHITECTURE_VEXER.md and
|
||||
# TASKS
|
||||
| Task | Owner(s) | Depends on | Notes |
|
||||
|---|---|---|---|
|
||||
|VEXER-CONN-SUSE-01-001 – Rancher hub discovery & auth|Team Vexer Connectors – SUSE|VEXER-CONN-ABS-01-001|TODO – Implement hub discovery/subscription setup with credential handling and offline snapshot support.|
|
||||
|VEXER-CONN-SUSE-01-001 – Rancher hub discovery & auth|Team Vexer Connectors – SUSE|VEXER-CONN-ABS-01-001|**DONE (2025-10-17)** – Added Rancher hub options/token provider, discovery metadata loader with offline snapshots + caching, connector shell, DI wiring, and unit tests covering network/offline paths.|
|
||||
|VEXER-CONN-SUSE-01-002 – Checkpointed event ingestion|Team Vexer Connectors – SUSE|VEXER-CONN-SUSE-01-001, VEXER-STORAGE-01-003|TODO – Process hub events with resume checkpoints, deduplication, and quarantine path for malformed payloads.|
|
||||
|VEXER-CONN-SUSE-01-003 – Trust metadata & policy hints|Team Vexer Connectors – SUSE|VEXER-CONN-SUSE-01-002, VEXER-POLICY-01-001|TODO – Emit provider trust configuration (signers, weight overrides) and attach provenance hints for consensus engine.|
|
||||
|
||||
@@ -0,0 +1,172 @@
|
||||
using System.Collections.Generic;
|
||||
using System.Net;
|
||||
using System.Net.Http;
|
||||
using System.Text;
|
||||
using FluentAssertions;
|
||||
using Microsoft.Extensions.Caching.Memory;
|
||||
using Microsoft.Extensions.Logging.Abstractions;
|
||||
using StellaOps.Vexer.Connectors.Ubuntu.CSAF.Configuration;
|
||||
using StellaOps.Vexer.Connectors.Ubuntu.CSAF.Metadata;
|
||||
using System.IO.Abstractions.TestingHelpers;
|
||||
using Xunit;
|
||||
using System.Threading;
|
||||
|
||||
namespace StellaOps.Vexer.Connectors.Ubuntu.CSAF.Tests.Metadata;
|
||||
|
||||
public sealed class UbuntuCatalogLoaderTests
|
||||
{
|
||||
private const string SampleIndex = """
|
||||
{
|
||||
"generated": "2025-10-10T00:00:00Z",
|
||||
"channels": [
|
||||
{
|
||||
"name": "stable",
|
||||
"catalogUrl": "https://ubuntu.com/security/csaf/stable/catalog.json",
|
||||
"sha256": "abc",
|
||||
"lastUpdated": "2025-10-09T10:00:00Z"
|
||||
},
|
||||
{
|
||||
"name": "esm",
|
||||
"catalogUrl": "https://ubuntu.com/security/csaf/esm/catalog.json",
|
||||
"sha256": "def",
|
||||
"lastUpdated": "2025-10-08T10:00:00Z"
|
||||
}
|
||||
]
|
||||
}
|
||||
""";
|
||||
|
||||
[Fact]
|
||||
public async Task LoadAsync_FetchesAndCachesIndex()
|
||||
{
|
||||
var handler = new TestHttpMessageHandler(new Dictionary<Uri, HttpResponseMessage>
|
||||
{
|
||||
[new Uri("https://ubuntu.com/security/csaf/index.json")] = CreateResponse(SampleIndex),
|
||||
});
|
||||
var client = new HttpClient(handler);
|
||||
var factory = new SingleClientHttpClientFactory(client);
|
||||
var cache = new MemoryCache(new MemoryCacheOptions());
|
||||
var fileSystem = new MockFileSystem();
|
||||
var loader = new UbuntuCatalogLoader(factory, cache, fileSystem, NullLogger<UbuntuCatalogLoader>.Instance, new AdjustableTimeProvider());
|
||||
|
||||
var options = new UbuntuConnectorOptions
|
||||
{
|
||||
IndexUri = new Uri("https://ubuntu.com/security/csaf/index.json"),
|
||||
OfflineSnapshotPath = "/snapshots/ubuntu-index.json",
|
||||
};
|
||||
|
||||
var result = await loader.LoadAsync(options, CancellationToken.None);
|
||||
result.Metadata.Channels.Should().HaveCount(1);
|
||||
result.Metadata.Channels[0].Name.Should().Be("stable");
|
||||
fileSystem.FileExists(options.OfflineSnapshotPath!).Should().BeTrue();
|
||||
result.FromCache.Should().BeFalse();
|
||||
|
||||
handler.ResetInvocationCount();
|
||||
var cached = await loader.LoadAsync(options, CancellationToken.None);
|
||||
cached.FromCache.Should().BeTrue();
|
||||
handler.InvocationCount.Should().Be(0);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task LoadAsync_UsesOfflineSnapshotWhenPreferred()
|
||||
{
|
||||
var handler = new TestHttpMessageHandler(new Dictionary<Uri, HttpResponseMessage>());
|
||||
var client = new HttpClient(handler);
|
||||
var factory = new SingleClientHttpClientFactory(client);
|
||||
var cache = new MemoryCache(new MemoryCacheOptions());
|
||||
var fileSystem = new MockFileSystem();
|
||||
fileSystem.AddFile("/snapshots/ubuntu-index.json", new MockFileData($"{{\"metadata\":{SampleIndex},\"fetchedAt\":\"2025-10-10T00:00:00Z\"}}"));
|
||||
var loader = new UbuntuCatalogLoader(factory, cache, fileSystem, NullLogger<UbuntuCatalogLoader>.Instance, new AdjustableTimeProvider());
|
||||
|
||||
var options = new UbuntuConnectorOptions
|
||||
{
|
||||
IndexUri = new Uri("https://ubuntu.com/security/csaf/index.json"),
|
||||
OfflineSnapshotPath = "/snapshots/ubuntu-index.json",
|
||||
PreferOfflineSnapshot = true,
|
||||
Channels = { "stable" }
|
||||
};
|
||||
|
||||
var result = await loader.LoadAsync(options, CancellationToken.None);
|
||||
result.FromOfflineSnapshot.Should().BeTrue();
|
||||
result.Metadata.Channels.Should().NotBeEmpty();
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task LoadAsync_ThrowsWhenNoChannelsMatch()
|
||||
{
|
||||
var handler = new TestHttpMessageHandler(new Dictionary<Uri, HttpResponseMessage>
|
||||
{
|
||||
[new Uri("https://ubuntu.com/security/csaf/index.json")] = CreateResponse(SampleIndex),
|
||||
});
|
||||
var client = new HttpClient(handler);
|
||||
var factory = new SingleClientHttpClientFactory(client);
|
||||
var cache = new MemoryCache(new MemoryCacheOptions());
|
||||
var fileSystem = new MockFileSystem();
|
||||
var loader = new UbuntuCatalogLoader(factory, cache, fileSystem, NullLogger<UbuntuCatalogLoader>.Instance, new AdjustableTimeProvider());
|
||||
|
||||
var options = new UbuntuConnectorOptions
|
||||
{
|
||||
IndexUri = new Uri("https://ubuntu.com/security/csaf/index.json"),
|
||||
};
|
||||
options.Channels.Clear();
|
||||
options.Channels.Add("nonexistent");
|
||||
|
||||
await Assert.ThrowsAsync<InvalidOperationException>(() => loader.LoadAsync(options, CancellationToken.None));
|
||||
}
|
||||
|
||||
private static HttpResponseMessage CreateResponse(string payload)
|
||||
=> new(HttpStatusCode.OK)
|
||||
{
|
||||
Content = new StringContent(payload, Encoding.UTF8, "application/json"),
|
||||
};
|
||||
|
||||
private sealed class SingleClientHttpClientFactory : IHttpClientFactory
|
||||
{
|
||||
private readonly HttpClient _client;
|
||||
|
||||
public SingleClientHttpClientFactory(HttpClient client)
|
||||
{
|
||||
_client = client;
|
||||
}
|
||||
|
||||
public HttpClient CreateClient(string name) => _client;
|
||||
}
|
||||
|
||||
private sealed class AdjustableTimeProvider : TimeProvider
|
||||
{
|
||||
private DateTimeOffset _now = DateTimeOffset.UtcNow;
|
||||
|
||||
public override DateTimeOffset GetUtcNow() => _now;
|
||||
}
|
||||
|
||||
private sealed class TestHttpMessageHandler : HttpMessageHandler
|
||||
{
|
||||
private readonly Dictionary<Uri, HttpResponseMessage> _responses;
|
||||
|
||||
public TestHttpMessageHandler(Dictionary<Uri, HttpResponseMessage> responses)
|
||||
{
|
||||
_responses = responses;
|
||||
}
|
||||
|
||||
public int InvocationCount { get; private set; }
|
||||
|
||||
public void ResetInvocationCount() => InvocationCount = 0;
|
||||
|
||||
protected override async Task<HttpResponseMessage> SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
|
||||
{
|
||||
InvocationCount++;
|
||||
if (request.RequestUri is not null && _responses.TryGetValue(request.RequestUri, out var response))
|
||||
{
|
||||
var payload = await response.Content.ReadAsStringAsync(cancellationToken).ConfigureAwait(false);
|
||||
return new HttpResponseMessage(response.StatusCode)
|
||||
{
|
||||
Content = new StringContent(payload, Encoding.UTF8, "application/json"),
|
||||
};
|
||||
}
|
||||
|
||||
return new HttpResponseMessage(HttpStatusCode.InternalServerError)
|
||||
{
|
||||
Content = new StringContent("unexpected request"),
|
||||
};
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,17 @@
|
||||
<Project Sdk="Microsoft.NET.Sdk">
|
||||
<PropertyGroup>
|
||||
<TargetFramework>net10.0</TargetFramework>
|
||||
<LangVersion>preview</LangVersion>
|
||||
<Nullable>enable</Nullable>
|
||||
<ImplicitUsings>enable</ImplicitUsings>
|
||||
<TreatWarningsAsErrors>true</TreatWarningsAsErrors>
|
||||
</PropertyGroup>
|
||||
<ItemGroup>
|
||||
<ProjectReference Include="..\StellaOps.Vexer.Connectors.Ubuntu.CSAF\StellaOps.Vexer.Connectors.Ubuntu.CSAF.csproj" />
|
||||
</ItemGroup>
|
||||
<ItemGroup>
|
||||
<PackageReference Include="FluentAssertions" Version="6.12.0" />
|
||||
<PackageReference Include="Microsoft.Extensions.Logging.Abstractions" Version="8.0.0" />
|
||||
<PackageReference Include="System.IO.Abstractions.TestingHelpers" Version="20.0.28" />
|
||||
</ItemGroup>
|
||||
</Project>
|
||||
@@ -0,0 +1,90 @@
|
||||
using System;
|
||||
using System.Collections.Generic;
|
||||
using System.IO;
|
||||
using System.IO.Abstractions;
|
||||
|
||||
namespace StellaOps.Vexer.Connectors.Ubuntu.CSAF.Configuration;
|
||||
|
||||
public sealed class UbuntuConnectorOptions
|
||||
{
|
||||
public const string HttpClientName = "vexer.connector.ubuntu.catalog";
|
||||
|
||||
/// <summary>
|
||||
/// Root index that lists Ubuntu CSAF channels.
|
||||
/// </summary>
|
||||
public Uri IndexUri { get; set; } = new("https://ubuntu.com/security/csaf/index.json");
|
||||
|
||||
/// <summary>
|
||||
/// Channels to include (e.g. stable, esm, lts).
|
||||
/// </summary>
|
||||
public IList<string> Channels { get; } = new List<string> { "stable" };
|
||||
|
||||
/// <summary>
|
||||
/// Duration to cache discovery metadata.
|
||||
/// </summary>
|
||||
public TimeSpan MetadataCacheDuration { get; set; } = TimeSpan.FromHours(4);
|
||||
|
||||
/// <summary>
|
||||
/// Prefer offline snapshot when available.
|
||||
/// </summary>
|
||||
public bool PreferOfflineSnapshot { get; set; }
|
||||
/// <summary>
|
||||
/// Optional file path for offline index snapshot.
|
||||
/// </summary>
|
||||
public string? OfflineSnapshotPath { get; set; }
|
||||
/// <summary>
|
||||
/// Controls persistence of network responses to <see cref="OfflineSnapshotPath"/>.
|
||||
/// </summary>
|
||||
public bool PersistOfflineSnapshot { get; set; } = true;
|
||||
|
||||
public void Validate(IFileSystem? fileSystem = null)
|
||||
{
|
||||
if (IndexUri is null || !IndexUri.IsAbsoluteUri)
|
||||
{
|
||||
throw new InvalidOperationException("IndexUri must be an absolute URI.");
|
||||
}
|
||||
|
||||
if (IndexUri.Scheme is not ("http" or "https"))
|
||||
{
|
||||
throw new InvalidOperationException("IndexUri must use HTTP or HTTPS.");
|
||||
}
|
||||
|
||||
if (Channels.Count == 0)
|
||||
{
|
||||
throw new InvalidOperationException("At least one channel must be specified.");
|
||||
}
|
||||
|
||||
for (var i = Channels.Count - 1; i >= 0; i--)
|
||||
{
|
||||
if (string.IsNullOrWhiteSpace(Channels[i]))
|
||||
{
|
||||
Channels.RemoveAt(i);
|
||||
}
|
||||
}
|
||||
|
||||
if (Channels.Count == 0)
|
||||
{
|
||||
throw new InvalidOperationException("Channel names cannot be empty.");
|
||||
}
|
||||
|
||||
if (MetadataCacheDuration <= TimeSpan.Zero)
|
||||
{
|
||||
throw new InvalidOperationException("MetadataCacheDuration must be positive.");
|
||||
}
|
||||
|
||||
if (PreferOfflineSnapshot && string.IsNullOrWhiteSpace(OfflineSnapshotPath))
|
||||
{
|
||||
throw new InvalidOperationException("OfflineSnapshotPath must be provided when PreferOfflineSnapshot is enabled.");
|
||||
}
|
||||
|
||||
if (!string.IsNullOrWhiteSpace(OfflineSnapshotPath))
|
||||
{
|
||||
var fs = fileSystem ?? new FileSystem();
|
||||
var directory = Path.GetDirectoryName(OfflineSnapshotPath);
|
||||
if (!string.IsNullOrWhiteSpace(directory) && !fs.Directory.Exists(directory))
|
||||
{
|
||||
fs.Directory.CreateDirectory(directory);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,32 @@
|
||||
using System;
|
||||
using System.Collections.Generic;
|
||||
using System.IO.Abstractions;
|
||||
using StellaOps.Vexer.Connectors.Abstractions;
|
||||
|
||||
namespace StellaOps.Vexer.Connectors.Ubuntu.CSAF.Configuration;
|
||||
|
||||
public sealed class UbuntuConnectorOptionsValidator : IVexConnectorOptionsValidator<UbuntuConnectorOptions>
|
||||
{
|
||||
private readonly IFileSystem _fileSystem;
|
||||
|
||||
public UbuntuConnectorOptionsValidator(IFileSystem fileSystem)
|
||||
{
|
||||
_fileSystem = fileSystem ?? throw new ArgumentNullException(nameof(fileSystem));
|
||||
}
|
||||
|
||||
public void Validate(VexConnectorDescriptor descriptor, UbuntuConnectorOptions options, IList<string> errors)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(descriptor);
|
||||
ArgumentNullException.ThrowIfNull(options);
|
||||
ArgumentNullException.ThrowIfNull(errors);
|
||||
|
||||
try
|
||||
{
|
||||
options.Validate(_fileSystem);
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
errors.Add(ex.Message);
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,45 @@
|
||||
using System;
|
||||
using System.Net;
|
||||
using System.Net.Http;
|
||||
using Microsoft.Extensions.Caching.Memory;
|
||||
using Microsoft.Extensions.DependencyInjection;
|
||||
using Microsoft.Extensions.DependencyInjection.Extensions;
|
||||
using StellaOps.Vexer.Connectors.Abstractions;
|
||||
using StellaOps.Vexer.Connectors.Ubuntu.CSAF.Configuration;
|
||||
using StellaOps.Vexer.Connectors.Ubuntu.CSAF.Metadata;
|
||||
using StellaOps.Vexer.Core;
|
||||
using System.IO.Abstractions;
|
||||
|
||||
namespace StellaOps.Vexer.Connectors.Ubuntu.CSAF.DependencyInjection;
|
||||
|
||||
public static class UbuntuConnectorServiceCollectionExtensions
|
||||
{
|
||||
public static IServiceCollection AddUbuntuCsafConnector(this IServiceCollection services, Action<UbuntuConnectorOptions>? configure = null)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(services);
|
||||
|
||||
services.TryAddSingleton<IMemoryCache, MemoryCache>();
|
||||
services.TryAddSingleton<IFileSystem, FileSystem>();
|
||||
|
||||
services.AddOptions<UbuntuConnectorOptions>()
|
||||
.Configure(options => configure?.Invoke(options));
|
||||
|
||||
services.AddSingleton<IVexConnectorOptionsValidator<UbuntuConnectorOptions>, UbuntuConnectorOptionsValidator>();
|
||||
|
||||
services.AddHttpClient(UbuntuConnectorOptions.HttpClientName, client =>
|
||||
{
|
||||
client.Timeout = TimeSpan.FromSeconds(60);
|
||||
client.DefaultRequestHeaders.UserAgent.ParseAdd("StellaOps.Vexer.Connectors.Ubuntu.CSAF/1.0");
|
||||
client.DefaultRequestHeaders.Accept.ParseAdd("application/json");
|
||||
})
|
||||
.ConfigurePrimaryHttpMessageHandler(static () => new HttpClientHandler
|
||||
{
|
||||
AutomaticDecompression = DecompressionMethods.All,
|
||||
});
|
||||
|
||||
services.AddSingleton<UbuntuCatalogLoader>();
|
||||
services.AddSingleton<IVexConnector, UbuntuCsafConnector>();
|
||||
|
||||
return services;
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,248 @@
|
||||
using System;
|
||||
using System.Collections.Generic;
|
||||
using System.Collections.Immutable;
|
||||
using System.IO.Abstractions;
|
||||
using System.Net;
|
||||
using System.Net.Http;
|
||||
using System.Text.Json;
|
||||
using System.Threading;
|
||||
using System.Threading.Tasks;
|
||||
using Microsoft.Extensions.Caching.Memory;
|
||||
using Microsoft.Extensions.Logging;
|
||||
using StellaOps.Vexer.Connectors.Ubuntu.CSAF.Configuration;
|
||||
|
||||
namespace StellaOps.Vexer.Connectors.Ubuntu.CSAF.Metadata;
|
||||
|
||||
public sealed class UbuntuCatalogLoader
|
||||
{
|
||||
public const string CachePrefix = "StellaOps.Vexer.Connectors.Ubuntu.CSAF.Index";
|
||||
|
||||
private readonly IHttpClientFactory _httpClientFactory;
|
||||
private readonly IMemoryCache _memoryCache;
|
||||
private readonly IFileSystem _fileSystem;
|
||||
private readonly ILogger<UbuntuCatalogLoader> _logger;
|
||||
private readonly TimeProvider _timeProvider;
|
||||
private readonly SemaphoreSlim _semaphore = new(1, 1);
|
||||
private readonly JsonSerializerOptions _serializerOptions = new(JsonSerializerDefaults.Web);
|
||||
|
||||
public UbuntuCatalogLoader(
|
||||
IHttpClientFactory httpClientFactory,
|
||||
IMemoryCache memoryCache,
|
||||
IFileSystem fileSystem,
|
||||
ILogger<UbuntuCatalogLoader> logger,
|
||||
TimeProvider? timeProvider = null)
|
||||
{
|
||||
_httpClientFactory = httpClientFactory ?? throw new ArgumentNullException(nameof(httpClientFactory));
|
||||
_memoryCache = memoryCache ?? throw new ArgumentNullException(nameof(memoryCache));
|
||||
_fileSystem = fileSystem ?? throw new ArgumentNullException(nameof(fileSystem));
|
||||
_logger = logger ?? throw new ArgumentNullException(nameof(logger));
|
||||
_timeProvider = timeProvider ?? TimeProvider.System;
|
||||
}
|
||||
|
||||
public async Task<UbuntuCatalogResult> LoadAsync(UbuntuConnectorOptions options, CancellationToken cancellationToken)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(options);
|
||||
options.Validate(_fileSystem);
|
||||
|
||||
var cacheKey = CreateCacheKey(options);
|
||||
if (_memoryCache.TryGetValue<CacheEntry>(cacheKey, out var cached) && cached is not null && !cached.IsExpired(_timeProvider.GetUtcNow()))
|
||||
{
|
||||
return cached.ToResult(fromCache: true);
|
||||
}
|
||||
|
||||
await _semaphore.WaitAsync(cancellationToken).ConfigureAwait(false);
|
||||
try
|
||||
{
|
||||
if (_memoryCache.TryGetValue<CacheEntry>(cacheKey, out cached) && cached is not null && !cached.IsExpired(_timeProvider.GetUtcNow()))
|
||||
{
|
||||
return cached.ToResult(fromCache: true);
|
||||
}
|
||||
|
||||
CacheEntry? entry = null;
|
||||
if (options.PreferOfflineSnapshot)
|
||||
{
|
||||
entry = LoadFromOffline(options);
|
||||
if (entry is null)
|
||||
{
|
||||
throw new InvalidOperationException("PreferOfflineSnapshot is enabled but no offline Ubuntu snapshot was found.");
|
||||
}
|
||||
}
|
||||
else
|
||||
{
|
||||
entry = await TryFetchFromNetworkAsync(options, cancellationToken).ConfigureAwait(false)
|
||||
?? LoadFromOffline(options);
|
||||
}
|
||||
|
||||
if (entry is null)
|
||||
{
|
||||
throw new InvalidOperationException("Unable to load Ubuntu CSAF index from network or offline snapshot.");
|
||||
}
|
||||
|
||||
var cacheOptions = new MemoryCacheEntryOptions();
|
||||
if (entry.MetadataCacheDuration > TimeSpan.Zero)
|
||||
{
|
||||
cacheOptions.AbsoluteExpiration = _timeProvider.GetUtcNow().Add(entry.MetadataCacheDuration);
|
||||
}
|
||||
|
||||
_memoryCache.Set(cacheKey, entry with { CachedAt = _timeProvider.GetUtcNow() }, cacheOptions);
|
||||
return entry.ToResult(fromCache: false);
|
||||
}
|
||||
finally
|
||||
{
|
||||
_semaphore.Release();
|
||||
}
|
||||
}
|
||||
|
||||
private async Task<CacheEntry?> TryFetchFromNetworkAsync(UbuntuConnectorOptions options, CancellationToken cancellationToken)
|
||||
{
|
||||
try
|
||||
{
|
||||
var client = _httpClientFactory.CreateClient(UbuntuConnectorOptions.HttpClientName);
|
||||
using var response = await client.GetAsync(options.IndexUri, HttpCompletionOption.ResponseHeadersRead, cancellationToken).ConfigureAwait(false);
|
||||
response.EnsureSuccessStatusCode();
|
||||
var payload = await response.Content.ReadAsStringAsync(cancellationToken).ConfigureAwait(false);
|
||||
|
||||
var metadata = ParseMetadata(payload, options.Channels);
|
||||
var now = _timeProvider.GetUtcNow();
|
||||
var entry = new CacheEntry(metadata, now, now, options.MetadataCacheDuration, false);
|
||||
|
||||
PersistSnapshotIfNeeded(options, metadata, now);
|
||||
return entry;
|
||||
}
|
||||
catch (Exception ex) when (ex is not OperationCanceledException)
|
||||
{
|
||||
_logger.LogWarning(ex, "Failed to fetch Ubuntu CSAF index from {Uri}; attempting offline fallback if available.", options.IndexUri);
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
private CacheEntry? LoadFromOffline(UbuntuConnectorOptions options)
|
||||
{
|
||||
if (string.IsNullOrWhiteSpace(options.OfflineSnapshotPath))
|
||||
{
|
||||
return null;
|
||||
}
|
||||
|
||||
if (!_fileSystem.File.Exists(options.OfflineSnapshotPath))
|
||||
{
|
||||
_logger.LogWarning("Ubuntu offline snapshot path {Path} does not exist.", options.OfflineSnapshotPath);
|
||||
return null;
|
||||
}
|
||||
|
||||
try
|
||||
{
|
||||
var payload = _fileSystem.File.ReadAllText(options.OfflineSnapshotPath);
|
||||
var snapshot = JsonSerializer.Deserialize<UbuntuCatalogSnapshot>(payload, _serializerOptions);
|
||||
if (snapshot is null)
|
||||
{
|
||||
throw new InvalidOperationException("Offline snapshot payload was empty.");
|
||||
}
|
||||
|
||||
return new CacheEntry(snapshot.Metadata, snapshot.FetchedAt, _timeProvider.GetUtcNow(), options.MetadataCacheDuration, true);
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
_logger.LogError(ex, "Failed to load Ubuntu CSAF index from offline snapshot {Path}.", options.OfflineSnapshotPath);
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
private UbuntuCatalogMetadata ParseMetadata(string payload, IList<string> channels)
|
||||
{
|
||||
if (string.IsNullOrWhiteSpace(payload))
|
||||
{
|
||||
throw new InvalidOperationException("Ubuntu index payload was empty.");
|
||||
}
|
||||
|
||||
using var document = JsonDocument.Parse(payload);
|
||||
var root = document.RootElement;
|
||||
|
||||
var generatedAt = root.TryGetProperty("generated", out var generatedElement) && generatedElement.ValueKind == JsonValueKind.String && DateTimeOffset.TryParse(generatedElement.GetString(), out var generated)
|
||||
? generated
|
||||
: _timeProvider.GetUtcNow();
|
||||
|
||||
var channelSet = new HashSet<string>(channels, StringComparer.OrdinalIgnoreCase);
|
||||
|
||||
if (!root.TryGetProperty("channels", out var channelsElement) || channelsElement.ValueKind is not JsonValueKind.Array)
|
||||
{
|
||||
throw new InvalidOperationException("Ubuntu index did not include a channels array.");
|
||||
}
|
||||
|
||||
var builder = ImmutableArray.CreateBuilder<UbuChannelCatalog>();
|
||||
foreach (var channelElement in channelsElement.EnumerateArray())
|
||||
{
|
||||
var name = channelElement.TryGetProperty("name", out var nameElement) && nameElement.ValueKind == JsonValueKind.String ? nameElement.GetString() : null;
|
||||
if (string.IsNullOrWhiteSpace(name) || !channelSet.Contains(name))
|
||||
{
|
||||
continue;
|
||||
}
|
||||
|
||||
if (!channelElement.TryGetProperty("catalogUrl", out var urlElement) || urlElement.ValueKind != JsonValueKind.String || !Uri.TryCreate(urlElement.GetString(), UriKind.Absolute, out var catalogUri))
|
||||
{
|
||||
_logger.LogWarning("Channel {Channel} did not specify a valid catalogUrl.", name);
|
||||
continue;
|
||||
}
|
||||
|
||||
string? sha256 = null;
|
||||
if (channelElement.TryGetProperty("sha256", out var shaElement) && shaElement.ValueKind == JsonValueKind.String)
|
||||
{
|
||||
sha256 = shaElement.GetString();
|
||||
}
|
||||
|
||||
DateTimeOffset? lastUpdated = null;
|
||||
if (channelElement.TryGetProperty("lastUpdated", out var updatedElement) && updatedElement.ValueKind == JsonValueKind.String && DateTimeOffset.TryParse(updatedElement.GetString(), out var updated))
|
||||
{
|
||||
lastUpdated = updated;
|
||||
}
|
||||
|
||||
builder.Add(new UbuChannelCatalog(name!, catalogUri, sha256, lastUpdated));
|
||||
}
|
||||
|
||||
if (builder.Count == 0)
|
||||
{
|
||||
throw new InvalidOperationException("None of the requested Ubuntu channels were present in the index.");
|
||||
}
|
||||
|
||||
return new UbuntuCatalogMetadata(generatedAt, builder.ToImmutable());
|
||||
}
|
||||
|
||||
private void PersistSnapshotIfNeeded(UbuntuConnectorOptions options, UbuntuCatalogMetadata metadata, DateTimeOffset fetchedAt)
|
||||
{
|
||||
if (!options.PersistOfflineSnapshot || string.IsNullOrWhiteSpace(options.OfflineSnapshotPath))
|
||||
{
|
||||
return;
|
||||
}
|
||||
|
||||
try
|
||||
{
|
||||
var snapshot = new UbuntuCatalogSnapshot(metadata, fetchedAt);
|
||||
var payload = JsonSerializer.Serialize(snapshot, _serializerOptions);
|
||||
_fileSystem.File.WriteAllText(options.OfflineSnapshotPath, payload);
|
||||
_logger.LogDebug("Persisted Ubuntu CSAF index snapshot to {Path}.", options.OfflineSnapshotPath);
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
_logger.LogWarning(ex, "Failed to persist Ubuntu CSAF index snapshot to {Path}.", options.OfflineSnapshotPath);
|
||||
}
|
||||
}
|
||||
|
||||
private static string CreateCacheKey(UbuntuConnectorOptions options)
|
||||
=> $"{CachePrefix}:{options.IndexUri}:{string.Join(',', options.Channels)}";
|
||||
|
||||
private sealed record CacheEntry(UbuntuCatalogMetadata Metadata, DateTimeOffset FetchedAt, DateTimeOffset CachedAt, TimeSpan MetadataCacheDuration, bool FromOfflineSnapshot)
|
||||
{
|
||||
public bool IsExpired(DateTimeOffset now)
|
||||
=> MetadataCacheDuration > TimeSpan.Zero && now >= CachedAt + MetadataCacheDuration;
|
||||
|
||||
public UbuntuCatalogResult ToResult(bool fromCache)
|
||||
=> new(Metadata, FetchedAt, fromCache, FromOfflineSnapshot);
|
||||
}
|
||||
|
||||
private sealed record UbuntuCatalogSnapshot(UbuntuCatalogMetadata Metadata, DateTimeOffset FetchedAt);
|
||||
}
|
||||
|
||||
public sealed record UbuntuCatalogMetadata(DateTimeOffset GeneratedAt, ImmutableArray<UbuChannelCatalog> Channels);
|
||||
|
||||
public sealed record UbuChannelCatalog(string Name, Uri CatalogUri, string? Sha256, DateTimeOffset? LastUpdated);
|
||||
|
||||
public sealed record UbuntuCatalogResult(UbuntuCatalogMetadata Metadata, DateTimeOffset FetchedAt, bool FromCache, bool FromOfflineSnapshot);
|
||||
@@ -0,0 +1,18 @@
|
||||
<Project Sdk="Microsoft.NET.Sdk">
|
||||
<PropertyGroup>
|
||||
<TargetFramework>net10.0</TargetFramework>
|
||||
<LangVersion>preview</LangVersion>
|
||||
<Nullable>enable</Nullable>
|
||||
<ImplicitUsings>enable</ImplicitUsings>
|
||||
<TreatWarningsAsErrors>true</TreatWarningsAsErrors>
|
||||
</PropertyGroup>
|
||||
<ItemGroup>
|
||||
<ProjectReference Include="..\StellaOps.Vexer.Connectors.Abstractions\StellaOps.Vexer.Connectors.Abstractions.csproj" />
|
||||
</ItemGroup>
|
||||
<ItemGroup>
|
||||
<PackageReference Include="Microsoft.Extensions.Caching.Memory" Version="8.0.0" />
|
||||
<PackageReference Include="Microsoft.Extensions.Http" Version="8.0.0" />
|
||||
<PackageReference Include="Microsoft.Extensions.Options" Version="8.0.0" />
|
||||
<PackageReference Include="System.IO.Abstractions" Version="20.0.28" />
|
||||
</ItemGroup>
|
||||
</Project>
|
||||
@@ -2,6 +2,6 @@ If you are working on this file you need to read docs/ARCHITECTURE_VEXER.md and
|
||||
# TASKS
|
||||
| Task | Owner(s) | Depends on | Notes |
|
||||
|---|---|---|---|
|
||||
|VEXER-CONN-UBUNTU-01-001 – Ubuntu CSAF discovery & channels|Team Vexer Connectors – Ubuntu|VEXER-CONN-ABS-01-001|TODO – Implement discovery of Ubuntu CSAF catalogs, channel selection (stable/LTS), and offline snapshot import.|
|
||||
|VEXER-CONN-UBUNTU-01-001 – Ubuntu CSAF discovery & channels|Team Vexer Connectors – Ubuntu|VEXER-CONN-ABS-01-001|**DONE (2025-10-17)** – Added Ubuntu connector project with configurable channel options, catalog loader (network/offline), DI wiring, and discovery unit tests.|
|
||||
|VEXER-CONN-UBUNTU-01-002 – Incremental fetch & deduplication|Team Vexer Connectors – Ubuntu|VEXER-CONN-UBUNTU-01-001, VEXER-STORAGE-01-003|TODO – Fetch CSAF bundles with ETag handling, checksum validation, deduplication, and raw persistence.|
|
||||
|VEXER-CONN-UBUNTU-01-003 – Trust metadata & provenance|Team Vexer Connectors – Ubuntu|VEXER-CONN-UBUNTU-01-002, VEXER-POLICY-01-001|TODO – Emit Ubuntu signing metadata (GPG fingerprints) plus provenance hints for policy weighting and diagnostics.|
|
||||
|
||||
@@ -0,0 +1,80 @@
|
||||
using System.Collections.Generic;
|
||||
using System.Collections.Immutable;
|
||||
using System.Runtime.CompilerServices;
|
||||
using Microsoft.Extensions.Logging;
|
||||
using StellaOps.Vexer.Connectors.Abstractions;
|
||||
using StellaOps.Vexer.Connectors.Ubuntu.CSAF.Configuration;
|
||||
using StellaOps.Vexer.Connectors.Ubuntu.CSAF.Metadata;
|
||||
using StellaOps.Vexer.Core;
|
||||
|
||||
namespace StellaOps.Vexer.Connectors.Ubuntu.CSAF;
|
||||
|
||||
public sealed class UbuntuCsafConnector : VexConnectorBase
|
||||
{
|
||||
private static readonly VexConnectorDescriptor DescriptorInstance = new(
|
||||
id: "vexer:ubuntu",
|
||||
kind: VexProviderKind.Distro,
|
||||
displayName: "Ubuntu CSAF")
|
||||
{
|
||||
Tags = ImmutableArray.Create("ubuntu", "csaf", "usn"),
|
||||
};
|
||||
|
||||
private readonly UbuntuCatalogLoader _catalogLoader;
|
||||
private readonly IEnumerable<IVexConnectorOptionsValidator<UbuntuConnectorOptions>> _validators;
|
||||
|
||||
private UbuntuConnectorOptions? _options;
|
||||
private UbuntuCatalogResult? _catalog;
|
||||
|
||||
public UbuntuCsafConnector(
|
||||
UbuntuCatalogLoader catalogLoader,
|
||||
IEnumerable<IVexConnectorOptionsValidator<UbuntuConnectorOptions>> validators,
|
||||
ILogger<UbuntuCsafConnector> logger,
|
||||
TimeProvider timeProvider)
|
||||
: base(DescriptorInstance, logger, timeProvider)
|
||||
{
|
||||
_catalogLoader = catalogLoader ?? throw new ArgumentNullException(nameof(catalogLoader));
|
||||
_validators = validators ?? Array.Empty<IVexConnectorOptionsValidator<UbuntuConnectorOptions>>();
|
||||
}
|
||||
|
||||
public override async ValueTask ValidateAsync(VexConnectorSettings settings, CancellationToken cancellationToken)
|
||||
{
|
||||
_options = VexConnectorOptionsBinder.Bind(
|
||||
Descriptor,
|
||||
settings,
|
||||
validators: _validators);
|
||||
|
||||
_catalog = await _catalogLoader.LoadAsync(_options, cancellationToken).ConfigureAwait(false);
|
||||
LogConnectorEvent(LogLevel.Information, "validate", "Ubuntu CSAF index loaded.", new Dictionary<string, object?>
|
||||
{
|
||||
["channelCount"] = _catalog.Metadata.Channels.Length,
|
||||
["fromOffline"] = _catalog.FromOfflineSnapshot,
|
||||
});
|
||||
}
|
||||
|
||||
public override async IAsyncEnumerable<VexRawDocument> FetchAsync(VexConnectorContext context, [EnumeratorCancellation] CancellationToken cancellationToken)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(context);
|
||||
|
||||
if (_options is null)
|
||||
{
|
||||
throw new InvalidOperationException("Connector must be validated before fetch operations.");
|
||||
}
|
||||
|
||||
if (_catalog is null)
|
||||
{
|
||||
_catalog = await _catalogLoader.LoadAsync(_options, cancellationToken).ConfigureAwait(false);
|
||||
}
|
||||
|
||||
LogConnectorEvent(LogLevel.Debug, "fetch", "Ubuntu CSAF discovery ready; channel catalogs handled in subsequent task.", new Dictionary<string, object?>
|
||||
{
|
||||
["since"] = context.Since?.ToString("O"),
|
||||
});
|
||||
|
||||
yield break;
|
||||
}
|
||||
|
||||
public override ValueTask<VexClaimBatch> NormalizeAsync(VexRawDocument document, CancellationToken cancellationToken)
|
||||
=> throw new NotSupportedException("UbuntuCsafConnector relies on CSAF normalizers for document processing.");
|
||||
|
||||
public UbuntuCatalogResult? GetCachedCatalog() => _catalog;
|
||||
}
|
||||
@@ -88,10 +88,46 @@ public sealed class ExportEngineTests
|
||||
Assert.Equal(1, recorder2.SaveCount);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task ExportAsync_AttachesAttestationMetadata()
|
||||
{
|
||||
var store = new InMemoryExportStore();
|
||||
var evaluator = new StaticPolicyEvaluator("baseline/v1");
|
||||
var dataSource = new InMemoryExportDataSource();
|
||||
var exporter = new DummyExporter(VexExportFormat.Json);
|
||||
var attestation = new RecordingAttestationClient();
|
||||
var engine = new VexExportEngine(
|
||||
store,
|
||||
evaluator,
|
||||
dataSource,
|
||||
new[] { exporter },
|
||||
NullLogger<VexExportEngine>.Instance,
|
||||
cacheIndex: null,
|
||||
artifactStores: null,
|
||||
attestationClient: attestation);
|
||||
|
||||
var query = VexQuery.Create(new[] { new VexQueryFilter("vulnId", "CVE-2025-0001") });
|
||||
var requestedAt = DateTimeOffset.UtcNow;
|
||||
var context = new VexExportRequestContext(query, VexExportFormat.Json, requestedAt);
|
||||
|
||||
var manifest = await engine.ExportAsync(context, CancellationToken.None);
|
||||
|
||||
Assert.NotNull(attestation.LastRequest);
|
||||
Assert.Equal(manifest.ExportId, attestation.LastRequest!.ExportId);
|
||||
Assert.NotNull(manifest.Attestation);
|
||||
Assert.Equal(attestation.Response.Attestation.EnvelopeDigest, manifest.Attestation!.EnvelopeDigest);
|
||||
Assert.Equal(attestation.Response.Attestation.PredicateType, manifest.Attestation.PredicateType);
|
||||
|
||||
Assert.NotNull(store.LastSavedManifest);
|
||||
Assert.Equal(manifest.Attestation, store.LastSavedManifest!.Attestation);
|
||||
}
|
||||
|
||||
private sealed class InMemoryExportStore : IVexExportStore
|
||||
{
|
||||
private readonly Dictionary<string, VexExportManifest> _store = new(StringComparer.Ordinal);
|
||||
|
||||
public VexExportManifest? LastSavedManifest { get; private set; }
|
||||
|
||||
public ValueTask<VexExportManifest?> FindAsync(VexQuerySignature signature, VexExportFormat format, CancellationToken cancellationToken)
|
||||
{
|
||||
var key = CreateKey(signature.Value, format);
|
||||
@@ -103,6 +139,7 @@ public sealed class ExportEngineTests
|
||||
{
|
||||
var key = CreateKey(manifest.QuerySignature.Value, manifest.Format);
|
||||
_store[key] = manifest;
|
||||
LastSavedManifest = manifest;
|
||||
return ValueTask.CompletedTask;
|
||||
}
|
||||
|
||||
@@ -110,6 +147,28 @@ public sealed class ExportEngineTests
|
||||
=> FormattableString.Invariant($"{signature}|{format}");
|
||||
}
|
||||
|
||||
private sealed class RecordingAttestationClient : IVexAttestationClient
|
||||
{
|
||||
public VexAttestationRequest? LastRequest { get; private set; }
|
||||
|
||||
public VexAttestationResponse Response { get; } = new VexAttestationResponse(
|
||||
new VexAttestationMetadata(
|
||||
predicateType: "https://stella-ops.org/attestations/vex-export",
|
||||
rekor: new VexRekorReference("0.2", "rekor://entry", "123"),
|
||||
envelopeDigest: "sha256:envelope",
|
||||
signedAt: DateTimeOffset.UnixEpoch),
|
||||
ImmutableDictionary<string, string>.Empty);
|
||||
|
||||
public ValueTask<VexAttestationResponse> SignAsync(VexAttestationRequest request, CancellationToken cancellationToken)
|
||||
{
|
||||
LastRequest = request;
|
||||
return ValueTask.FromResult(Response);
|
||||
}
|
||||
|
||||
public ValueTask<VexAttestationVerification> VerifyAsync(VexAttestationRequest request, CancellationToken cancellationToken)
|
||||
=> ValueTask.FromResult(new VexAttestationVerification(true, ImmutableDictionary<string, string>.Empty));
|
||||
}
|
||||
|
||||
private sealed class RecordingCacheIndex : IVexCacheIndex
|
||||
{
|
||||
public Dictionary<(string Signature, VexExportFormat Format), bool> RemoveCalls { get; } = new();
|
||||
@@ -213,4 +272,5 @@ public sealed class ExportEngineTests
|
||||
return ValueTask.FromResult(new VexExportResult(new VexContentAddress("sha256", "deadbeef"), bytes.Length, ImmutableDictionary<string, string>.Empty));
|
||||
}
|
||||
}
|
||||
|
||||
}
|
||||
|
||||
@@ -39,6 +39,7 @@ public sealed class VexExportEngine : IExportEngine
|
||||
private readonly ILogger<VexExportEngine> _logger;
|
||||
private readonly IVexCacheIndex? _cacheIndex;
|
||||
private readonly IReadOnlyList<IVexArtifactStore> _artifactStores;
|
||||
private readonly IVexAttestationClient? _attestationClient;
|
||||
|
||||
public VexExportEngine(
|
||||
IVexExportStore exportStore,
|
||||
@@ -47,7 +48,8 @@ public sealed class VexExportEngine : IExportEngine
|
||||
IEnumerable<IVexExporter> exporters,
|
||||
ILogger<VexExportEngine> logger,
|
||||
IVexCacheIndex? cacheIndex = null,
|
||||
IEnumerable<IVexArtifactStore>? artifactStores = null)
|
||||
IEnumerable<IVexArtifactStore>? artifactStores = null,
|
||||
IVexAttestationClient? attestationClient = null)
|
||||
{
|
||||
_exportStore = exportStore ?? throw new ArgumentNullException(nameof(exportStore));
|
||||
_policyEvaluator = policyEvaluator ?? throw new ArgumentNullException(nameof(policyEvaluator));
|
||||
@@ -55,6 +57,7 @@ public sealed class VexExportEngine : IExportEngine
|
||||
_logger = logger ?? throw new ArgumentNullException(nameof(logger));
|
||||
_cacheIndex = cacheIndex;
|
||||
_artifactStores = artifactStores?.ToArray() ?? Array.Empty<IVexArtifactStore>();
|
||||
_attestationClient = attestationClient;
|
||||
|
||||
if (exporters is null)
|
||||
{
|
||||
@@ -105,6 +108,7 @@ public sealed class VexExportEngine : IExportEngine
|
||||
context.RequestedAt);
|
||||
|
||||
var digest = exporter.Digest(exportRequest);
|
||||
var exportId = FormattableString.Invariant($"exports/{context.RequestedAt:yyyyMMddTHHmmssfffZ}/{digest.Digest}");
|
||||
|
||||
await using var buffer = new MemoryStream();
|
||||
var result = await exporter.SerializeAsync(exportRequest, buffer, cancellationToken).ConfigureAwait(false);
|
||||
@@ -134,7 +138,36 @@ public sealed class VexExportEngine : IExportEngine
|
||||
}
|
||||
}
|
||||
|
||||
var exportId = FormattableString.Invariant($"exports/{context.RequestedAt:yyyyMMddTHHmmssfffZ}/{digest.Digest}");
|
||||
VexAttestationMetadata? attestationMetadata = null;
|
||||
if (_attestationClient is not null)
|
||||
{
|
||||
var attestationRequest = new VexAttestationRequest(
|
||||
exportId,
|
||||
signature,
|
||||
digest,
|
||||
context.Format,
|
||||
context.RequestedAt,
|
||||
dataset.SourceProviders,
|
||||
result.Metadata);
|
||||
|
||||
var response = await _attestationClient.SignAsync(attestationRequest, cancellationToken).ConfigureAwait(false);
|
||||
attestationMetadata = response.Attestation;
|
||||
|
||||
if (!response.Diagnostics.IsEmpty)
|
||||
{
|
||||
foreach (var diagnostic in response.Diagnostics)
|
||||
{
|
||||
_logger.LogDebug(
|
||||
"Attestation diagnostic {Key}={Value} for export {ExportId}",
|
||||
diagnostic.Key,
|
||||
diagnostic.Value,
|
||||
exportId);
|
||||
}
|
||||
}
|
||||
|
||||
_logger.LogInformation("Attestation generated for export {ExportId}", exportId);
|
||||
}
|
||||
|
||||
var manifest = new VexExportManifest(
|
||||
exportId,
|
||||
signature,
|
||||
@@ -145,7 +178,7 @@ public sealed class VexExportEngine : IExportEngine
|
||||
dataset.SourceProviders,
|
||||
fromCache: false,
|
||||
consensusRevision: _policyEvaluator.Version,
|
||||
attestation: null,
|
||||
attestation: attestationMetadata,
|
||||
sizeBytes: result.BytesWritten);
|
||||
|
||||
await _exportStore.SaveAsync(manifest, cancellationToken).ConfigureAwait(false);
|
||||
|
||||
@@ -5,5 +5,5 @@ If you are working on this file you need to read docs/ARCHITECTURE_VEXER.md and
|
||||
|VEXER-EXPORT-01-001 – Export engine orchestration|Team Vexer Export|VEXER-CORE-01-003|DONE (2025-10-15) – Export engine scaffolding with cache lookup, data source hooks, and deterministic manifest emission.|
|
||||
|VEXER-EXPORT-01-002 – Cache index & eviction hooks|Team Vexer Export|VEXER-EXPORT-01-001, VEXER-STORAGE-01-003|**DONE (2025-10-16)** – Export engine now invalidates cache entries on force refresh, cache services expose prune/invalidate APIs, and storage maintenance trims expired/dangling records with Mongo2Go coverage.|
|
||||
|VEXER-EXPORT-01-003 – Artifact store adapters|Team Vexer Export|VEXER-EXPORT-01-001|**DONE (2025-10-16)** – Implemented multi-store pipeline with filesystem, S3-compatible, and offline bundle adapters (hash verification + manifest/zip output) plus unit coverage and DI hooks.|
|
||||
|VEXER-EXPORT-01-004 – Attestation handoff integration|Team Vexer Export|VEXER-EXPORT-01-001, VEXER-ATTEST-01-001|TODO – Connect export engine to attestation client, persist Rekor metadata, and reuse cached attestations.|
|
||||
|VEXER-EXPORT-01-004 – Attestation handoff integration|Team Vexer Export|VEXER-EXPORT-01-001, VEXER-ATTEST-01-001|**DONE (2025-10-17)** – Export engine now invokes attestation client, logs diagnostics, and persists Rekor/envelope metadata on manifests; regression coverage added in `ExportEngineTests.ExportAsync_AttachesAttestationMetadata`.|
|
||||
|VEXER-EXPORT-01-005 – Score & resolve envelope surfaces|Team Vexer Export|VEXER-EXPORT-01-004, VEXER-CORE-02-001|TODO – Emit consensus+score envelopes in export manifests, include policy/scoring digests, and update offline bundle/ORAS layouts to carry signed VEX responses.|
|
||||
|
||||
131
src/StellaOps.Vexer.Formats.CSAF.Tests/CsafNormalizerTests.cs
Normal file
131
src/StellaOps.Vexer.Formats.CSAF.Tests/CsafNormalizerTests.cs
Normal file
@@ -0,0 +1,131 @@
|
||||
using System.Collections.Immutable;
|
||||
using System.Linq;
|
||||
using System.Text;
|
||||
using System.IO;
|
||||
using FluentAssertions;
|
||||
using Microsoft.Extensions.Logging.Abstractions;
|
||||
using StellaOps.Vexer.Core;
|
||||
using StellaOps.Vexer.Formats.CSAF;
|
||||
|
||||
namespace StellaOps.Vexer.Formats.CSAF.Tests;
|
||||
|
||||
public sealed class CsafNormalizerTests
|
||||
{
|
||||
[Fact]
|
||||
public async Task NormalizeAsync_ProducesClaimsPerProductStatus()
|
||||
{
|
||||
var json = """
|
||||
{
|
||||
"document": {
|
||||
"tracking": {
|
||||
"id": "RHSA-2025:0001",
|
||||
"version": "3",
|
||||
"revision": "3",
|
||||
"status": "final",
|
||||
"initial_release_date": "2025-10-01T00:00:00Z",
|
||||
"current_release_date": "2025-10-10T00:00:00Z"
|
||||
},
|
||||
"publisher": {
|
||||
"name": "Red Hat Product Security",
|
||||
"category": "vendor"
|
||||
}
|
||||
},
|
||||
"product_tree": {
|
||||
"full_product_names": [
|
||||
{
|
||||
"product_id": "CSAFPID-0001",
|
||||
"name": "Red Hat Enterprise Linux 9",
|
||||
"product_identification_helper": {
|
||||
"cpe": "cpe:/o:redhat:enterprise_linux:9",
|
||||
"purl": "pkg:rpm/redhat/enterprise-linux@9"
|
||||
}
|
||||
}
|
||||
]
|
||||
},
|
||||
"vulnerabilities": [
|
||||
{
|
||||
"cve": "CVE-2025-0001",
|
||||
"title": "Kernel vulnerability",
|
||||
"product_status": {
|
||||
"known_affected": [ "CSAFPID-0001" ]
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "VEX-0002",
|
||||
"title": "Library issue",
|
||||
"product_status": {
|
||||
"known_not_affected": [ "CSAFPID-0001" ]
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
""";
|
||||
|
||||
var rawDocument = new VexRawDocument(
|
||||
ProviderId: "vexer:redhat",
|
||||
VexDocumentFormat.Csaf,
|
||||
new Uri("https://example.com/csaf/rhsa-2025-0001.json"),
|
||||
new DateTimeOffset(2025, 10, 11, 0, 0, 0, TimeSpan.Zero),
|
||||
"sha256:dummydigest",
|
||||
Encoding.UTF8.GetBytes(json),
|
||||
ImmutableDictionary<string, string>.Empty);
|
||||
|
||||
var provider = new VexProvider("vexer:redhat", "Red Hat CSAF", VexProviderKind.Distro);
|
||||
var normalizer = new CsafNormalizer(NullLogger<CsafNormalizer>.Instance);
|
||||
|
||||
var batch = await normalizer.NormalizeAsync(rawDocument, provider, CancellationToken.None);
|
||||
|
||||
batch.Claims.Should().HaveCount(2);
|
||||
|
||||
var affectedClaim = batch.Claims.First(c => c.VulnerabilityId == "CVE-2025-0001");
|
||||
affectedClaim.Status.Should().Be(VexClaimStatus.Affected);
|
||||
affectedClaim.Product.Key.Should().Be("CSAFPID-0001");
|
||||
affectedClaim.Product.Purl.Should().Be("pkg:rpm/redhat/enterprise-linux@9");
|
||||
affectedClaim.Product.Cpe.Should().Be("cpe:/o:redhat:enterprise_linux:9");
|
||||
affectedClaim.Document.Revision.Should().Be("3");
|
||||
affectedClaim.FirstSeen.Should().Be(new DateTimeOffset(2025, 10, 1, 0, 0, 0, TimeSpan.Zero));
|
||||
affectedClaim.LastSeen.Should().Be(new DateTimeOffset(2025, 10, 10, 0, 0, 0, TimeSpan.Zero));
|
||||
affectedClaim.AdditionalMetadata.Should().ContainKey("csaf.tracking.id");
|
||||
affectedClaim.AdditionalMetadata["csaf.publisher.name"].Should().Be("Red Hat Product Security");
|
||||
|
||||
var notAffectedClaim = batch.Claims.First(c => c.VulnerabilityId == "VEX-0002");
|
||||
notAffectedClaim.Status.Should().Be(VexClaimStatus.NotAffected);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task NormalizeAsync_PreservesRedHatSpecificMetadata()
|
||||
{
|
||||
var path = Path.Combine(AppContext.BaseDirectory, "Fixtures", "rhsa-sample.json");
|
||||
var json = await File.ReadAllTextAsync(path);
|
||||
|
||||
var rawDocument = new VexRawDocument(
|
||||
ProviderId: "vexer:redhat",
|
||||
VexDocumentFormat.Csaf,
|
||||
new Uri("https://security.example.com/rhsa-2025-1001.json"),
|
||||
new DateTimeOffset(2025, 10, 6, 0, 0, 0, TimeSpan.Zero),
|
||||
"sha256:rhdadigest",
|
||||
Encoding.UTF8.GetBytes(json),
|
||||
ImmutableDictionary<string, string>.Empty);
|
||||
|
||||
var provider = new VexProvider("vexer:redhat", "Red Hat CSAF", VexProviderKind.Distro);
|
||||
var normalizer = new CsafNormalizer(NullLogger<CsafNormalizer>.Instance);
|
||||
|
||||
var batch = await normalizer.NormalizeAsync(rawDocument, provider, CancellationToken.None);
|
||||
batch.Claims.Should().ContainSingle();
|
||||
|
||||
var claim = batch.Claims[0];
|
||||
claim.VulnerabilityId.Should().Be("CVE-2025-1234");
|
||||
claim.Status.Should().Be(VexClaimStatus.Affected);
|
||||
claim.Product.Key.Should().Be("rh-enterprise-linux-9");
|
||||
claim.Product.Name.Should().Be("Red Hat Enterprise Linux 9");
|
||||
claim.Product.Purl.Should().Be("pkg:rpm/redhat/enterprise-linux@9");
|
||||
claim.Product.Cpe.Should().Be("cpe:/o:redhat:enterprise_linux:9");
|
||||
claim.FirstSeen.Should().Be(new DateTimeOffset(2025, 10, 1, 12, 0, 0, TimeSpan.Zero));
|
||||
claim.LastSeen.Should().Be(new DateTimeOffset(2025, 10, 5, 10, 0, 0, TimeSpan.Zero));
|
||||
|
||||
claim.AdditionalMetadata.Should().ContainKey("csaf.tracking.id");
|
||||
claim.AdditionalMetadata["csaf.tracking.id"].Should().Be("RHSA-2025:1001");
|
||||
claim.AdditionalMetadata["csaf.tracking.status"].Should().Be("final");
|
||||
claim.AdditionalMetadata["csaf.publisher.name"].Should().Be("Red Hat Product Security");
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,47 @@
|
||||
{
|
||||
"document": {
|
||||
"publisher": {
|
||||
"name": "Red Hat Product Security",
|
||||
"category": "vendor"
|
||||
},
|
||||
"tracking": {
|
||||
"id": "RHSA-2025:1001",
|
||||
"status": "final",
|
||||
"version": "3",
|
||||
"initial_release_date": "2025-10-01T12:00:00Z",
|
||||
"current_release_date": "2025-10-05T10:00:00Z"
|
||||
}
|
||||
},
|
||||
"product_tree": {
|
||||
"full_product_names": [
|
||||
{
|
||||
"product_id": "rh-enterprise-linux-9",
|
||||
"name": "Red Hat Enterprise Linux 9",
|
||||
"product_identification_helper": {
|
||||
"cpe": "cpe:/o:redhat:enterprise_linux:9",
|
||||
"purl": "pkg:rpm/redhat/enterprise-linux@9"
|
||||
}
|
||||
}
|
||||
],
|
||||
"branches": [
|
||||
{
|
||||
"name": "Red Hat Enterprise Linux",
|
||||
"product": {
|
||||
"product_id": "rh-enterprise-linux-9",
|
||||
"name": "Red Hat Enterprise Linux 9"
|
||||
}
|
||||
}
|
||||
]
|
||||
},
|
||||
"vulnerabilities": [
|
||||
{
|
||||
"cve": "CVE-2025-1234",
|
||||
"title": "Kernel privilege escalation",
|
||||
"product_status": {
|
||||
"known_affected": [
|
||||
"rh-enterprise-linux-9"
|
||||
]
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,20 @@
|
||||
<Project Sdk="Microsoft.NET.Sdk">
|
||||
<PropertyGroup>
|
||||
<TargetFramework>net10.0</TargetFramework>
|
||||
<Nullable>enable</Nullable>
|
||||
<ImplicitUsings>enable</ImplicitUsings>
|
||||
</PropertyGroup>
|
||||
<ItemGroup>
|
||||
<PackageReference Include="FluentAssertions" Version="6.12.0" />
|
||||
</ItemGroup>
|
||||
<ItemGroup>
|
||||
<Using Include="Xunit" />
|
||||
</ItemGroup>
|
||||
<ItemGroup>
|
||||
<ProjectReference Include="..\StellaOps.Vexer.Core\StellaOps.Vexer.Core.csproj" />
|
||||
<ProjectReference Include="..\StellaOps.Vexer.Formats.CSAF\StellaOps.Vexer.Formats.CSAF.csproj" />
|
||||
</ItemGroup>
|
||||
<ItemGroup>
|
||||
<None Include="Fixtures\**\*" CopyToOutputDirectory="Always" />
|
||||
</ItemGroup>
|
||||
</Project>
|
||||
532
src/StellaOps.Vexer.Formats.CSAF/CsafNormalizer.cs
Normal file
532
src/StellaOps.Vexer.Formats.CSAF/CsafNormalizer.cs
Normal file
@@ -0,0 +1,532 @@
|
||||
using System;
|
||||
using System.Collections.Generic;
|
||||
using System.Collections.Immutable;
|
||||
using System.Linq;
|
||||
using System.Text.Json;
|
||||
using System.Threading;
|
||||
using System.Threading.Tasks;
|
||||
using Microsoft.Extensions.Logging;
|
||||
using StellaOps.Vexer.Core;
|
||||
|
||||
namespace StellaOps.Vexer.Formats.CSAF;
|
||||
|
||||
public sealed class CsafNormalizer : IVexNormalizer
|
||||
{
|
||||
private static readonly ImmutableDictionary<VexClaimStatus, int> StatusPrecedence = new Dictionary<VexClaimStatus, int>
|
||||
{
|
||||
[VexClaimStatus.UnderInvestigation] = 0,
|
||||
[VexClaimStatus.Affected] = 1,
|
||||
[VexClaimStatus.NotAffected] = 2,
|
||||
[VexClaimStatus.Fixed] = 3,
|
||||
}.ToImmutableDictionary();
|
||||
|
||||
private readonly ILogger<CsafNormalizer> _logger;
|
||||
|
||||
public CsafNormalizer(ILogger<CsafNormalizer> logger)
|
||||
{
|
||||
_logger = logger ?? throw new ArgumentNullException(nameof(logger));
|
||||
}
|
||||
|
||||
public string Format => VexDocumentFormat.Csaf.ToString().ToLowerInvariant();
|
||||
|
||||
public bool CanHandle(VexRawDocument document)
|
||||
=> document is not null && document.Format == VexDocumentFormat.Csaf;
|
||||
|
||||
public ValueTask<VexClaimBatch> NormalizeAsync(VexRawDocument document, VexProvider provider, CancellationToken cancellationToken)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(document);
|
||||
ArgumentNullException.ThrowIfNull(provider);
|
||||
|
||||
cancellationToken.ThrowIfCancellationRequested();
|
||||
|
||||
try
|
||||
{
|
||||
var result = CsafParser.Parse(document);
|
||||
var claims = ImmutableArray.CreateBuilder<VexClaim>(result.Claims.Length);
|
||||
foreach (var entry in result.Claims)
|
||||
{
|
||||
var product = new VexProduct(
|
||||
entry.Product.ProductId,
|
||||
entry.Product.Name,
|
||||
entry.Product.Version,
|
||||
entry.Product.Purl,
|
||||
entry.Product.Cpe);
|
||||
|
||||
var claimDocument = new VexClaimDocument(
|
||||
VexDocumentFormat.Csaf,
|
||||
document.Digest,
|
||||
document.SourceUri,
|
||||
result.Revision,
|
||||
signature: null);
|
||||
|
||||
var claim = new VexClaim(
|
||||
entry.VulnerabilityId,
|
||||
provider.Id,
|
||||
product,
|
||||
entry.Status,
|
||||
claimDocument,
|
||||
result.FirstRelease,
|
||||
result.LastRelease,
|
||||
justification: null,
|
||||
detail: entry.Detail,
|
||||
confidence: null,
|
||||
additionalMetadata: result.Metadata);
|
||||
|
||||
claims.Add(claim);
|
||||
}
|
||||
|
||||
var orderedClaims = claims
|
||||
.ToImmutable()
|
||||
.OrderBy(static claim => claim.VulnerabilityId, StringComparer.Ordinal)
|
||||
.ThenBy(static claim => claim.Product.Key, StringComparer.Ordinal)
|
||||
.ToImmutableArray();
|
||||
|
||||
_logger.LogInformation(
|
||||
"Normalized CSAF document {Source} into {ClaimCount} claim(s).",
|
||||
document.SourceUri,
|
||||
orderedClaims.Length);
|
||||
|
||||
return ValueTask.FromResult(new VexClaimBatch(
|
||||
document,
|
||||
orderedClaims,
|
||||
ImmutableDictionary<string, string>.Empty));
|
||||
}
|
||||
catch (JsonException ex)
|
||||
{
|
||||
_logger.LogError(ex, "Failed to parse CSAF document {SourceUri}", document.SourceUri);
|
||||
throw;
|
||||
}
|
||||
}
|
||||
|
||||
private static class CsafParser
|
||||
{
|
||||
public static CsafParseResult Parse(VexRawDocument document)
|
||||
{
|
||||
using var json = JsonDocument.Parse(document.Content.ToArray());
|
||||
var root = json.RootElement;
|
||||
|
||||
var tracking = root.TryGetProperty("document", out var documentElement) &&
|
||||
documentElement.ValueKind == JsonValueKind.Object &&
|
||||
documentElement.TryGetProperty("tracking", out var trackingElement)
|
||||
? trackingElement
|
||||
: default;
|
||||
|
||||
var firstRelease = ParseDate(tracking, "initial_release_date") ?? document.RetrievedAt;
|
||||
var lastRelease = ParseDate(tracking, "current_release_date") ?? firstRelease;
|
||||
|
||||
if (lastRelease < firstRelease)
|
||||
{
|
||||
lastRelease = firstRelease;
|
||||
}
|
||||
|
||||
var metadataBuilder = ImmutableDictionary.CreateBuilder<string, string>(StringComparer.Ordinal);
|
||||
AddIfPresent(metadataBuilder, tracking, "id", "csaf.tracking.id");
|
||||
AddIfPresent(metadataBuilder, tracking, "version", "csaf.tracking.version");
|
||||
AddIfPresent(metadataBuilder, tracking, "status", "csaf.tracking.status");
|
||||
AddPublisherMetadata(metadataBuilder, documentElement);
|
||||
|
||||
var revision = TryGetString(tracking, "revision");
|
||||
|
||||
var productCatalog = CollectProducts(root);
|
||||
var claimsBuilder = ImmutableArray.CreateBuilder<CsafClaimEntry>();
|
||||
|
||||
if (root.TryGetProperty("vulnerabilities", out var vulnerabilitiesElement) &&
|
||||
vulnerabilitiesElement.ValueKind == JsonValueKind.Array)
|
||||
{
|
||||
foreach (var vulnerability in vulnerabilitiesElement.EnumerateArray())
|
||||
{
|
||||
var vulnerabilityId = ResolveVulnerabilityId(vulnerability);
|
||||
if (string.IsNullOrWhiteSpace(vulnerabilityId))
|
||||
{
|
||||
continue;
|
||||
}
|
||||
|
||||
var detail = ResolveDetail(vulnerability);
|
||||
var productClaims = BuildClaimsForVulnerability(
|
||||
vulnerabilityId,
|
||||
vulnerability,
|
||||
productCatalog,
|
||||
detail);
|
||||
|
||||
claimsBuilder.AddRange(productClaims);
|
||||
}
|
||||
}
|
||||
|
||||
return new CsafParseResult(
|
||||
firstRelease,
|
||||
lastRelease,
|
||||
revision,
|
||||
metadataBuilder.ToImmutable(),
|
||||
claimsBuilder.ToImmutable());
|
||||
}
|
||||
|
||||
private static IReadOnlyList<CsafClaimEntry> BuildClaimsForVulnerability(
|
||||
string vulnerabilityId,
|
||||
JsonElement vulnerability,
|
||||
IReadOnlyDictionary<string, CsafProductInfo> productCatalog,
|
||||
string? detail)
|
||||
{
|
||||
if (!vulnerability.TryGetProperty("product_status", out var statusElement) ||
|
||||
statusElement.ValueKind != JsonValueKind.Object)
|
||||
{
|
||||
return Array.Empty<CsafClaimEntry>();
|
||||
}
|
||||
|
||||
var claims = new Dictionary<string, CsafClaimEntryBuilder>(StringComparer.OrdinalIgnoreCase);
|
||||
|
||||
foreach (var statusProperty in statusElement.EnumerateObject())
|
||||
{
|
||||
var status = MapStatus(statusProperty.Name);
|
||||
if (status is null)
|
||||
{
|
||||
continue;
|
||||
}
|
||||
|
||||
if (statusProperty.Value.ValueKind != JsonValueKind.Array)
|
||||
{
|
||||
continue;
|
||||
}
|
||||
|
||||
foreach (var productIdElement in statusProperty.Value.EnumerateArray())
|
||||
{
|
||||
var productId = productIdElement.GetString();
|
||||
if (string.IsNullOrWhiteSpace(productId))
|
||||
{
|
||||
continue;
|
||||
}
|
||||
|
||||
var product = ResolveProduct(productCatalog, productId);
|
||||
UpdateClaim(claims, product, status.Value, detail);
|
||||
}
|
||||
}
|
||||
|
||||
if (claims.Count == 0)
|
||||
{
|
||||
return Array.Empty<CsafClaimEntry>();
|
||||
}
|
||||
|
||||
return claims.Values
|
||||
.Select(builder => new CsafClaimEntry(
|
||||
vulnerabilityId,
|
||||
builder.Product,
|
||||
builder.Status,
|
||||
builder.Detail))
|
||||
.ToArray();
|
||||
}
|
||||
|
||||
private static void UpdateClaim(
|
||||
IDictionary<string, CsafClaimEntryBuilder> claims,
|
||||
CsafProductInfo product,
|
||||
VexClaimStatus status,
|
||||
string? detail)
|
||||
{
|
||||
if (!claims.TryGetValue(product.ProductId, out var existing) ||
|
||||
StatusPrecedence[status] > StatusPrecedence[existing.Status])
|
||||
{
|
||||
claims[product.ProductId] = new CsafClaimEntryBuilder(product, status, detail);
|
||||
}
|
||||
}
|
||||
|
||||
private static CsafProductInfo ResolveProduct(
|
||||
IReadOnlyDictionary<string, CsafProductInfo> catalog,
|
||||
string productId)
|
||||
{
|
||||
if (catalog.TryGetValue(productId, out var product))
|
||||
{
|
||||
return product;
|
||||
}
|
||||
|
||||
return new CsafProductInfo(productId, productId, null, null, null);
|
||||
}
|
||||
|
||||
private static string ResolveVulnerabilityId(JsonElement vulnerability)
|
||||
{
|
||||
var id = TryGetString(vulnerability, "cve")
|
||||
?? TryGetString(vulnerability, "id")
|
||||
?? TryGetString(vulnerability, "vuln_id");
|
||||
|
||||
return string.IsNullOrWhiteSpace(id) ? string.Empty : id.Trim();
|
||||
}
|
||||
|
||||
private static string? ResolveDetail(JsonElement vulnerability)
|
||||
{
|
||||
var title = TryGetString(vulnerability, "title");
|
||||
if (!string.IsNullOrWhiteSpace(title))
|
||||
{
|
||||
return title.Trim();
|
||||
}
|
||||
|
||||
if (vulnerability.TryGetProperty("notes", out var notesElement) &&
|
||||
notesElement.ValueKind == JsonValueKind.Array)
|
||||
{
|
||||
foreach (var note in notesElement.EnumerateArray())
|
||||
{
|
||||
if (note.ValueKind != JsonValueKind.Object)
|
||||
{
|
||||
continue;
|
||||
}
|
||||
|
||||
var category = TryGetString(note, "category");
|
||||
if (!string.IsNullOrWhiteSpace(category) &&
|
||||
!string.Equals(category, "description", StringComparison.OrdinalIgnoreCase))
|
||||
{
|
||||
continue;
|
||||
}
|
||||
|
||||
var text = TryGetString(note, "text");
|
||||
if (!string.IsNullOrWhiteSpace(text))
|
||||
{
|
||||
return text.Trim();
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
return null;
|
||||
}
|
||||
|
||||
private static Dictionary<string, CsafProductInfo> CollectProducts(JsonElement root)
|
||||
{
|
||||
var products = new Dictionary<string, CsafProductInfo>(StringComparer.OrdinalIgnoreCase);
|
||||
|
||||
if (!root.TryGetProperty("product_tree", out var productTree) ||
|
||||
productTree.ValueKind != JsonValueKind.Object)
|
||||
{
|
||||
return products;
|
||||
}
|
||||
|
||||
if (productTree.TryGetProperty("full_product_names", out var fullNames) &&
|
||||
fullNames.ValueKind == JsonValueKind.Array)
|
||||
{
|
||||
foreach (var productEntry in fullNames.EnumerateArray())
|
||||
{
|
||||
var product = ParseProduct(productEntry, parentBranchName: null);
|
||||
if (product is not null)
|
||||
{
|
||||
AddOrUpdate(product);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
if (productTree.TryGetProperty("branches", out var branches) &&
|
||||
branches.ValueKind == JsonValueKind.Array)
|
||||
{
|
||||
foreach (var branch in branches.EnumerateArray())
|
||||
{
|
||||
VisitBranch(branch, parentBranchName: null);
|
||||
}
|
||||
}
|
||||
|
||||
return products;
|
||||
|
||||
void VisitBranch(JsonElement branch, string? parentBranchName)
|
||||
{
|
||||
if (branch.ValueKind != JsonValueKind.Object)
|
||||
{
|
||||
return;
|
||||
}
|
||||
|
||||
var branchName = TryGetString(branch, "name") ?? parentBranchName;
|
||||
|
||||
if (branch.TryGetProperty("product", out var productElement))
|
||||
{
|
||||
var product = ParseProduct(productElement, branchName);
|
||||
if (product is not null)
|
||||
{
|
||||
AddOrUpdate(product);
|
||||
}
|
||||
}
|
||||
|
||||
if (branch.TryGetProperty("branches", out var childBranches) &&
|
||||
childBranches.ValueKind == JsonValueKind.Array)
|
||||
{
|
||||
foreach (var childBranch in childBranches.EnumerateArray())
|
||||
{
|
||||
VisitBranch(childBranch, branchName);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
void AddOrUpdate(CsafProductInfo product)
|
||||
{
|
||||
if (products.TryGetValue(product.ProductId, out var existing))
|
||||
{
|
||||
products[product.ProductId] = MergeProducts(existing, product);
|
||||
}
|
||||
else
|
||||
{
|
||||
products[product.ProductId] = product;
|
||||
}
|
||||
}
|
||||
|
||||
static CsafProductInfo MergeProducts(CsafProductInfo existing, CsafProductInfo incoming)
|
||||
{
|
||||
static string ChooseName(string incoming, string fallback)
|
||||
=> string.IsNullOrWhiteSpace(incoming) ? fallback : incoming;
|
||||
|
||||
static string? ChooseOptional(string? incoming, string? fallback)
|
||||
=> string.IsNullOrWhiteSpace(incoming) ? fallback : incoming;
|
||||
|
||||
return new CsafProductInfo(
|
||||
existing.ProductId,
|
||||
ChooseName(incoming.Name, existing.Name),
|
||||
ChooseOptional(incoming.Version, existing.Version),
|
||||
ChooseOptional(incoming.Purl, existing.Purl),
|
||||
ChooseOptional(incoming.Cpe, existing.Cpe));
|
||||
}
|
||||
}
|
||||
|
||||
private static CsafProductInfo? ParseProduct(JsonElement element, string? parentBranchName)
|
||||
{
|
||||
if (element.ValueKind != JsonValueKind.Object)
|
||||
{
|
||||
return null;
|
||||
}
|
||||
|
||||
JsonElement productElement = element;
|
||||
if (!element.TryGetProperty("product_id", out var idElement) &&
|
||||
element.TryGetProperty("product", out var nestedProduct) &&
|
||||
nestedProduct.ValueKind == JsonValueKind.Object &&
|
||||
nestedProduct.TryGetProperty("product_id", out idElement))
|
||||
{
|
||||
productElement = nestedProduct;
|
||||
}
|
||||
|
||||
var productId = idElement.GetString();
|
||||
if (string.IsNullOrWhiteSpace(productId))
|
||||
{
|
||||
return null;
|
||||
}
|
||||
|
||||
var name = TryGetString(productElement, "name")
|
||||
?? TryGetString(element, "name")
|
||||
?? parentBranchName
|
||||
?? productId;
|
||||
|
||||
var version = TryGetString(productElement, "product_version")
|
||||
?? TryGetString(productElement, "version")
|
||||
?? TryGetString(element, "product_version");
|
||||
|
||||
string? cpe = null;
|
||||
string? purl = null;
|
||||
if (productElement.TryGetProperty("product_identification_helper", out var helper) &&
|
||||
helper.ValueKind == JsonValueKind.Object)
|
||||
{
|
||||
cpe = TryGetString(helper, "cpe");
|
||||
purl = TryGetString(helper, "purl");
|
||||
}
|
||||
|
||||
return new CsafProductInfo(productId.Trim(), name.Trim(), version?.Trim(), purl?.Trim(), cpe?.Trim());
|
||||
}
|
||||
|
||||
private static VexClaimStatus? MapStatus(string statusName)
|
||||
{
|
||||
if (string.IsNullOrWhiteSpace(statusName))
|
||||
{
|
||||
return null;
|
||||
}
|
||||
|
||||
return statusName switch
|
||||
{
|
||||
"known_affected" or "fixed_after_release" or "first_affected" or "last_affected" => VexClaimStatus.Affected,
|
||||
"known_not_affected" or "last_not_affected" or "first_not_affected" => VexClaimStatus.NotAffected,
|
||||
"fixed" or "first_fixed" or "last_fixed" => VexClaimStatus.Fixed,
|
||||
"under_investigation" or "investigating" => VexClaimStatus.UnderInvestigation,
|
||||
_ => null,
|
||||
};
|
||||
}
|
||||
|
||||
private static DateTimeOffset? ParseDate(JsonElement element, string propertyName)
|
||||
{
|
||||
if (element.ValueKind != JsonValueKind.Object)
|
||||
{
|
||||
return null;
|
||||
}
|
||||
|
||||
if (!element.TryGetProperty(propertyName, out var dateElement))
|
||||
{
|
||||
return null;
|
||||
}
|
||||
|
||||
var value = dateElement.GetString();
|
||||
if (string.IsNullOrWhiteSpace(value))
|
||||
{
|
||||
return null;
|
||||
}
|
||||
|
||||
if (DateTimeOffset.TryParse(value, out var parsed))
|
||||
{
|
||||
return parsed;
|
||||
}
|
||||
|
||||
return null;
|
||||
}
|
||||
|
||||
private static string? TryGetString(JsonElement element, string propertyName)
|
||||
{
|
||||
if (element.ValueKind != JsonValueKind.Object)
|
||||
{
|
||||
return null;
|
||||
}
|
||||
|
||||
if (!element.TryGetProperty(propertyName, out var property))
|
||||
{
|
||||
return null;
|
||||
}
|
||||
|
||||
return property.ValueKind == JsonValueKind.String ? property.GetString() : null;
|
||||
}
|
||||
|
||||
private static void AddIfPresent(
|
||||
ImmutableDictionary<string, string>.Builder builder,
|
||||
JsonElement element,
|
||||
string propertyName,
|
||||
string metadataKey)
|
||||
{
|
||||
var value = TryGetString(element, propertyName);
|
||||
if (!string.IsNullOrWhiteSpace(value))
|
||||
{
|
||||
builder[metadataKey] = value.Trim();
|
||||
}
|
||||
}
|
||||
|
||||
private static void AddPublisherMetadata(
|
||||
ImmutableDictionary<string, string>.Builder builder,
|
||||
JsonElement documentElement)
|
||||
{
|
||||
if (documentElement.ValueKind != JsonValueKind.Object ||
|
||||
!documentElement.TryGetProperty("publisher", out var publisher) ||
|
||||
publisher.ValueKind != JsonValueKind.Object)
|
||||
{
|
||||
return;
|
||||
}
|
||||
|
||||
AddIfPresent(builder, publisher, "name", "csaf.publisher.name");
|
||||
AddIfPresent(builder, publisher, "category", "csaf.publisher.category");
|
||||
}
|
||||
|
||||
private readonly record struct CsafClaimEntryBuilder(
|
||||
CsafProductInfo Product,
|
||||
VexClaimStatus Status,
|
||||
string? Detail);
|
||||
}
|
||||
|
||||
private sealed record CsafParseResult(
|
||||
DateTimeOffset FirstRelease,
|
||||
DateTimeOffset LastRelease,
|
||||
string? Revision,
|
||||
ImmutableDictionary<string, string> Metadata,
|
||||
ImmutableArray<CsafClaimEntry> Claims);
|
||||
|
||||
private sealed record CsafClaimEntry(
|
||||
string VulnerabilityId,
|
||||
CsafProductInfo Product,
|
||||
VexClaimStatus Status,
|
||||
string? Detail);
|
||||
|
||||
private sealed record CsafProductInfo(
|
||||
string ProductId,
|
||||
string Name,
|
||||
string? Version,
|
||||
string? Purl,
|
||||
string? Cpe);
|
||||
}
|
||||
@@ -0,0 +1,14 @@
|
||||
using Microsoft.Extensions.DependencyInjection;
|
||||
using StellaOps.Vexer.Core;
|
||||
|
||||
namespace StellaOps.Vexer.Formats.CSAF;
|
||||
|
||||
public static class CsafFormatsServiceCollectionExtensions
|
||||
{
|
||||
public static IServiceCollection AddCsafNormalizer(this IServiceCollection services)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(services);
|
||||
services.AddSingleton<IVexNormalizer, CsafNormalizer>();
|
||||
return services;
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,16 @@
|
||||
<Project Sdk="Microsoft.NET.Sdk">
|
||||
<PropertyGroup>
|
||||
<TargetFramework>net10.0</TargetFramework>
|
||||
<LangVersion>preview</LangVersion>
|
||||
<Nullable>enable</Nullable>
|
||||
<ImplicitUsings>enable</ImplicitUsings>
|
||||
<TreatWarningsAsErrors>true</TreatWarningsAsErrors>
|
||||
</PropertyGroup>
|
||||
<ItemGroup>
|
||||
<PackageReference Include="Microsoft.Extensions.DependencyInjection.Abstractions" Version="8.0.0" />
|
||||
<PackageReference Include="Microsoft.Extensions.Logging.Abstractions" Version="8.0.0" />
|
||||
</ItemGroup>
|
||||
<ItemGroup>
|
||||
<ProjectReference Include="..\StellaOps.Vexer.Core\StellaOps.Vexer.Core.csproj" />
|
||||
</ItemGroup>
|
||||
</Project>
|
||||
@@ -2,6 +2,6 @@ If you are working on this file you need to read docs/ARCHITECTURE_VEXER.md and
|
||||
# TASKS
|
||||
| Task | Owner(s) | Depends on | Notes |
|
||||
|---|---|---|---|
|
||||
|VEXER-FMT-CSAF-01-001 – CSAF normalizer foundation|Team Vexer Formats|VEXER-CORE-01-001|TODO – Implement CSAF parser covering provider metadata, document tracking, and vulnerability/product mapping into `VexClaim`.|
|
||||
|VEXER-FMT-CSAF-01-001 – CSAF normalizer foundation|Team Vexer Formats|VEXER-CORE-01-001|**DONE (2025-10-17)** – Implemented CSAF normalizer + DI hook, parsing tracking metadata, product tree branches/full names, and mapping product statuses into canonical `VexClaim`s with baseline precedence. Regression added in `CsafNormalizerTests`.|
|
||||
|VEXER-FMT-CSAF-01-002 – Status/justification mapping|Team Vexer Formats|VEXER-FMT-CSAF-01-001, VEXER-POLICY-01-001|TODO – Normalize CSAF `product_status` + `justification` values into policy-aware enums with audit diagnostics for unsupported codes.|
|
||||
|VEXER-FMT-CSAF-01-003 – CSAF export adapter|Team Vexer Formats|VEXER-EXPORT-01-001, VEXER-FMT-CSAF-01-001|TODO – Provide CSAF export writer producing deterministic documents (per vuln/product) and manifest metadata for attestation.|
|
||||
|
||||
@@ -0,0 +1,93 @@
|
||||
using System.Collections.Immutable;
|
||||
using System.Linq;
|
||||
using System.Text;
|
||||
using FluentAssertions;
|
||||
using Microsoft.Extensions.Logging.Abstractions;
|
||||
using StellaOps.Vexer.Core;
|
||||
using StellaOps.Vexer.Formats.CycloneDX;
|
||||
|
||||
namespace StellaOps.Vexer.Formats.CycloneDX.Tests;
|
||||
|
||||
public sealed class CycloneDxNormalizerTests
|
||||
{
|
||||
[Fact]
|
||||
public async Task NormalizeAsync_MapsAnalysisStateAndJustification()
|
||||
{
|
||||
var json = """
|
||||
{
|
||||
"bomFormat": "CycloneDX",
|
||||
"specVersion": "1.4",
|
||||
"serialNumber": "urn:uuid:1234",
|
||||
"version": "7",
|
||||
"metadata": {
|
||||
"timestamp": "2025-10-15T12:00:00Z"
|
||||
},
|
||||
"components": [
|
||||
{
|
||||
"bom-ref": "pkg:npm/acme/lib@1.0.0",
|
||||
"name": "acme-lib",
|
||||
"version": "1.0.0",
|
||||
"purl": "pkg:npm/acme/lib@1.0.0"
|
||||
}
|
||||
],
|
||||
"vulnerabilities": [
|
||||
{
|
||||
"id": "CVE-2025-1000",
|
||||
"detail": "Library issue",
|
||||
"analysis": {
|
||||
"state": "not_affected",
|
||||
"justification": "code_not_present",
|
||||
"response": [ "can_not_fix", "will_not_fix" ]
|
||||
},
|
||||
"affects": [
|
||||
{ "ref": "pkg:npm/acme/lib@1.0.0" }
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "CVE-2025-1001",
|
||||
"description": "Investigating impact",
|
||||
"analysis": {
|
||||
"state": "in_triage"
|
||||
},
|
||||
"affects": [
|
||||
{ "ref": "pkg:npm/missing/component@2.0.0" }
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
""";
|
||||
|
||||
var rawDocument = new VexRawDocument(
|
||||
"vexer:cyclonedx",
|
||||
VexDocumentFormat.CycloneDx,
|
||||
new Uri("https://example.org/vex.json"),
|
||||
new DateTimeOffset(2025, 10, 16, 0, 0, 0, TimeSpan.Zero),
|
||||
"sha256:dummydigest",
|
||||
Encoding.UTF8.GetBytes(json),
|
||||
ImmutableDictionary<string, string>.Empty);
|
||||
|
||||
var provider = new VexProvider("vexer:cyclonedx", "CycloneDX Provider", VexProviderKind.Vendor);
|
||||
var normalizer = new CycloneDxNormalizer(NullLogger<CycloneDxNormalizer>.Instance);
|
||||
|
||||
var batch = await normalizer.NormalizeAsync(rawDocument, provider, CancellationToken.None);
|
||||
|
||||
batch.Claims.Should().HaveCount(2);
|
||||
|
||||
var notAffected = batch.Claims.Single(c => c.VulnerabilityId == "CVE-2025-1000");
|
||||
notAffected.Status.Should().Be(VexClaimStatus.NotAffected);
|
||||
notAffected.Justification.Should().Be(VexJustification.CodeNotPresent);
|
||||
notAffected.Product.Key.Should().Be("pkg:npm/acme/lib@1.0.0");
|
||||
notAffected.Product.Purl.Should().Be("pkg:npm/acme/lib@1.0.0");
|
||||
notAffected.Document.Revision.Should().Be("7");
|
||||
notAffected.AdditionalMetadata["cyclonedx.specVersion"].Should().Be("1.4");
|
||||
notAffected.AdditionalMetadata["cyclonedx.analysis.state"].Should().Be("not_affected");
|
||||
notAffected.AdditionalMetadata["cyclonedx.analysis.response"].Should().Be("can_not_fix,will_not_fix");
|
||||
|
||||
var investigating = batch.Claims.Single(c => c.VulnerabilityId == "CVE-2025-1001");
|
||||
investigating.Status.Should().Be(VexClaimStatus.UnderInvestigation);
|
||||
investigating.Justification.Should().BeNull();
|
||||
investigating.Product.Key.Should().Be("pkg:npm/missing/component@2.0.0");
|
||||
investigating.Product.Name.Should().Be("pkg:npm/missing/component@2.0.0");
|
||||
investigating.AdditionalMetadata.Should().ContainKey("cyclonedx.specVersion");
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,17 @@
|
||||
<Project Sdk="Microsoft.NET.Sdk">
|
||||
<PropertyGroup>
|
||||
<TargetFramework>net10.0</TargetFramework>
|
||||
<Nullable>enable</Nullable>
|
||||
<ImplicitUsings>enable</ImplicitUsings>
|
||||
</PropertyGroup>
|
||||
<ItemGroup>
|
||||
<PackageReference Include="FluentAssertions" Version="6.12.0" />
|
||||
</ItemGroup>
|
||||
<ItemGroup>
|
||||
<Using Include="Xunit" />
|
||||
</ItemGroup>
|
||||
<ItemGroup>
|
||||
<ProjectReference Include="..\StellaOps.Vexer.Core\StellaOps.Vexer.Core.csproj" />
|
||||
<ProjectReference Include="..\StellaOps.Vexer.Formats.CycloneDX\StellaOps.Vexer.Formats.CycloneDX.csproj" />
|
||||
</ItemGroup>
|
||||
</Project>
|
||||
459
src/StellaOps.Vexer.Formats.CycloneDX/CycloneDxNormalizer.cs
Normal file
459
src/StellaOps.Vexer.Formats.CycloneDX/CycloneDxNormalizer.cs
Normal file
@@ -0,0 +1,459 @@
|
||||
using System;
|
||||
using System.Collections.Generic;
|
||||
using System.Collections.Immutable;
|
||||
using System.Linq;
|
||||
using System.Text.Json;
|
||||
using System.Threading;
|
||||
using System.Threading.Tasks;
|
||||
using Microsoft.Extensions.Logging;
|
||||
using StellaOps.Vexer.Core;
|
||||
|
||||
namespace StellaOps.Vexer.Formats.CycloneDX;
|
||||
|
||||
public sealed class CycloneDxNormalizer : IVexNormalizer
|
||||
{
|
||||
private static readonly ImmutableDictionary<string, VexClaimStatus> StateMap = new Dictionary<string, VexClaimStatus>(StringComparer.OrdinalIgnoreCase)
|
||||
{
|
||||
["not_affected"] = VexClaimStatus.NotAffected,
|
||||
["resolved"] = VexClaimStatus.Fixed,
|
||||
["resolved_with_patches"] = VexClaimStatus.Fixed,
|
||||
["resolved_no_fix"] = VexClaimStatus.Fixed,
|
||||
["fixed"] = VexClaimStatus.Fixed,
|
||||
["affected"] = VexClaimStatus.Affected,
|
||||
["known_affected"] = VexClaimStatus.Affected,
|
||||
["exploitable"] = VexClaimStatus.Affected,
|
||||
["in_triage"] = VexClaimStatus.UnderInvestigation,
|
||||
["under_investigation"] = VexClaimStatus.UnderInvestigation,
|
||||
["unknown"] = VexClaimStatus.UnderInvestigation,
|
||||
}.ToImmutableDictionary(StringComparer.OrdinalIgnoreCase);
|
||||
|
||||
private static readonly ImmutableDictionary<string, VexJustification> JustificationMap = new Dictionary<string, VexJustification>(StringComparer.OrdinalIgnoreCase)
|
||||
{
|
||||
["code_not_present"] = VexJustification.CodeNotPresent,
|
||||
["code_not_reachable"] = VexJustification.CodeNotReachable,
|
||||
["component_not_present"] = VexJustification.ComponentNotPresent,
|
||||
["component_not_configured"] = VexJustification.ComponentNotConfigured,
|
||||
["vulnerable_code_not_present"] = VexJustification.VulnerableCodeNotPresent,
|
||||
["vulnerable_code_not_in_execute_path"] = VexJustification.VulnerableCodeNotInExecutePath,
|
||||
["vulnerable_code_cannot_be_controlled_by_adversary"] = VexJustification.VulnerableCodeCannotBeControlledByAdversary,
|
||||
["inline_mitigations_already_exist"] = VexJustification.InlineMitigationsAlreadyExist,
|
||||
["protected_by_mitigating_control"] = VexJustification.ProtectedByMitigatingControl,
|
||||
["protected_by_compensating_control"] = VexJustification.ProtectedByCompensatingControl,
|
||||
["requires_configuration"] = VexJustification.RequiresConfiguration,
|
||||
["requires_dependency"] = VexJustification.RequiresDependency,
|
||||
["requires_environment"] = VexJustification.RequiresEnvironment,
|
||||
}.ToImmutableDictionary(StringComparer.OrdinalIgnoreCase);
|
||||
|
||||
private readonly ILogger<CycloneDxNormalizer> _logger;
|
||||
|
||||
public CycloneDxNormalizer(ILogger<CycloneDxNormalizer> logger)
|
||||
{
|
||||
_logger = logger ?? throw new ArgumentNullException(nameof(logger));
|
||||
}
|
||||
|
||||
public string Format => VexDocumentFormat.CycloneDx.ToString().ToLowerInvariant();
|
||||
|
||||
public bool CanHandle(VexRawDocument document)
|
||||
=> document is not null && document.Format == VexDocumentFormat.CycloneDx;
|
||||
|
||||
public ValueTask<VexClaimBatch> NormalizeAsync(VexRawDocument document, VexProvider provider, CancellationToken cancellationToken)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(document);
|
||||
ArgumentNullException.ThrowIfNull(provider);
|
||||
cancellationToken.ThrowIfCancellationRequested();
|
||||
|
||||
try
|
||||
{
|
||||
var parseResult = CycloneDxParser.Parse(document);
|
||||
var baseMetadata = parseResult.Metadata;
|
||||
var claimsBuilder = ImmutableArray.CreateBuilder<VexClaim>();
|
||||
|
||||
foreach (var vulnerability in parseResult.Vulnerabilities)
|
||||
{
|
||||
cancellationToken.ThrowIfCancellationRequested();
|
||||
|
||||
var state = MapState(vulnerability.AnalysisState, out var stateRaw);
|
||||
var justification = MapJustification(vulnerability.AnalysisJustification);
|
||||
var responses = vulnerability.AnalysisResponses;
|
||||
|
||||
foreach (var affect in vulnerability.Affects)
|
||||
{
|
||||
var productInfo = parseResult.ResolveProduct(affect.ComponentRef);
|
||||
var product = new VexProduct(
|
||||
productInfo.Key,
|
||||
productInfo.Name,
|
||||
productInfo.Version,
|
||||
productInfo.Purl,
|
||||
productInfo.Cpe);
|
||||
|
||||
var metadata = baseMetadata;
|
||||
if (!string.IsNullOrWhiteSpace(stateRaw))
|
||||
{
|
||||
metadata = metadata.SetItem("cyclonedx.analysis.state", stateRaw);
|
||||
}
|
||||
|
||||
if (!string.IsNullOrWhiteSpace(vulnerability.AnalysisJustification))
|
||||
{
|
||||
metadata = metadata.SetItem("cyclonedx.analysis.justification", vulnerability.AnalysisJustification);
|
||||
}
|
||||
|
||||
if (responses.Length > 0)
|
||||
{
|
||||
metadata = metadata.SetItem("cyclonedx.analysis.response", string.Join(",", responses));
|
||||
}
|
||||
|
||||
if (!string.IsNullOrWhiteSpace(affect.ComponentRef))
|
||||
{
|
||||
metadata = metadata.SetItem("cyclonedx.affects.ref", affect.ComponentRef);
|
||||
}
|
||||
|
||||
var claimDocument = new VexClaimDocument(
|
||||
VexDocumentFormat.CycloneDx,
|
||||
document.Digest,
|
||||
document.SourceUri,
|
||||
parseResult.BomVersion,
|
||||
signature: null);
|
||||
|
||||
var claim = new VexClaim(
|
||||
vulnerability.VulnerabilityId,
|
||||
provider.Id,
|
||||
product,
|
||||
state,
|
||||
claimDocument,
|
||||
parseResult.FirstObserved,
|
||||
parseResult.LastObserved,
|
||||
justification,
|
||||
vulnerability.Detail,
|
||||
confidence: null,
|
||||
additionalMetadata: metadata);
|
||||
|
||||
claimsBuilder.Add(claim);
|
||||
}
|
||||
}
|
||||
|
||||
var orderedClaims = claimsBuilder
|
||||
.ToImmutable()
|
||||
.OrderBy(static c => c.VulnerabilityId, StringComparer.Ordinal)
|
||||
.ThenBy(static c => c.Product.Key, StringComparer.Ordinal)
|
||||
.ToImmutableArray();
|
||||
|
||||
_logger.LogInformation(
|
||||
"Normalized CycloneDX document {Source} into {ClaimCount} claim(s).",
|
||||
document.SourceUri,
|
||||
orderedClaims.Length);
|
||||
|
||||
return ValueTask.FromResult(new VexClaimBatch(
|
||||
document,
|
||||
orderedClaims,
|
||||
ImmutableDictionary<string, string>.Empty));
|
||||
}
|
||||
catch (JsonException ex)
|
||||
{
|
||||
_logger.LogError(ex, "Failed to parse CycloneDX VEX document {SourceUri}", document.SourceUri);
|
||||
throw;
|
||||
}
|
||||
}
|
||||
|
||||
private static VexClaimStatus MapState(string? state, out string? raw)
|
||||
{
|
||||
raw = state?.Trim();
|
||||
|
||||
if (!string.IsNullOrWhiteSpace(state) && StateMap.TryGetValue(state.Trim(), out var mapped))
|
||||
{
|
||||
return mapped;
|
||||
}
|
||||
|
||||
return VexClaimStatus.UnderInvestigation;
|
||||
}
|
||||
|
||||
private static VexJustification? MapJustification(string? justification)
|
||||
{
|
||||
if (string.IsNullOrWhiteSpace(justification))
|
||||
{
|
||||
return null;
|
||||
}
|
||||
|
||||
return JustificationMap.TryGetValue(justification.Trim(), out var mapped)
|
||||
? mapped
|
||||
: null;
|
||||
}
|
||||
|
||||
private sealed class CycloneDxParser
|
||||
{
|
||||
public static CycloneDxParseResult Parse(VexRawDocument document)
|
||||
{
|
||||
using var json = JsonDocument.Parse(document.Content.ToArray());
|
||||
var root = json.RootElement;
|
||||
|
||||
var specVersion = TryGetString(root, "specVersion");
|
||||
var bomVersion = TryGetString(root, "version");
|
||||
var serialNumber = TryGetString(root, "serialNumber");
|
||||
|
||||
var metadataTimestamp = ParseDate(TryGetProperty(root, "metadata"), "timestamp");
|
||||
var observedTimestamp = metadataTimestamp ?? document.RetrievedAt;
|
||||
|
||||
var metadataBuilder = ImmutableDictionary.CreateBuilder<string, string>(StringComparer.Ordinal);
|
||||
if (!string.IsNullOrWhiteSpace(specVersion))
|
||||
{
|
||||
metadataBuilder["cyclonedx.specVersion"] = specVersion!;
|
||||
}
|
||||
|
||||
if (!string.IsNullOrWhiteSpace(bomVersion))
|
||||
{
|
||||
metadataBuilder["cyclonedx.version"] = bomVersion!;
|
||||
}
|
||||
|
||||
if (!string.IsNullOrWhiteSpace(serialNumber))
|
||||
{
|
||||
metadataBuilder["cyclonedx.serialNumber"] = serialNumber!;
|
||||
}
|
||||
|
||||
var components = CollectComponents(root);
|
||||
var vulnerabilities = CollectVulnerabilities(root);
|
||||
|
||||
return new CycloneDxParseResult(
|
||||
metadataBuilder.ToImmutable(),
|
||||
bomVersion,
|
||||
observedTimestamp,
|
||||
observedTimestamp,
|
||||
components,
|
||||
vulnerabilities);
|
||||
}
|
||||
|
||||
private static ImmutableDictionary<string, CycloneDxComponent> CollectComponents(JsonElement root)
|
||||
{
|
||||
var builder = ImmutableDictionary.CreateBuilder<string, CycloneDxComponent>(StringComparer.Ordinal);
|
||||
|
||||
if (root.TryGetProperty("components", out var components) && components.ValueKind == JsonValueKind.Array)
|
||||
{
|
||||
foreach (var component in components.EnumerateArray())
|
||||
{
|
||||
if (component.ValueKind != JsonValueKind.Object)
|
||||
{
|
||||
continue;
|
||||
}
|
||||
|
||||
var reference = TryGetString(component, "bom-ref") ?? TryGetString(component, "bomRef");
|
||||
if (string.IsNullOrWhiteSpace(reference))
|
||||
{
|
||||
continue;
|
||||
}
|
||||
|
||||
var name = TryGetString(component, "name") ?? reference;
|
||||
var version = TryGetString(component, "version");
|
||||
var purl = TryGetString(component, "purl");
|
||||
|
||||
string? cpe = null;
|
||||
if (component.TryGetProperty("externalReferences", out var externalRefs) && externalRefs.ValueKind == JsonValueKind.Array)
|
||||
{
|
||||
foreach (var referenceEntry in externalRefs.EnumerateArray())
|
||||
{
|
||||
if (referenceEntry.ValueKind != JsonValueKind.Object)
|
||||
{
|
||||
continue;
|
||||
}
|
||||
|
||||
var type = TryGetString(referenceEntry, "type");
|
||||
if (!string.Equals(type, "cpe", StringComparison.OrdinalIgnoreCase))
|
||||
{
|
||||
continue;
|
||||
}
|
||||
|
||||
if (referenceEntry.TryGetProperty("url", out var url) && url.ValueKind == JsonValueKind.String)
|
||||
{
|
||||
cpe = url.GetString();
|
||||
break;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
builder[reference!] = new CycloneDxComponent(reference!, name ?? reference!, version, purl, cpe);
|
||||
}
|
||||
}
|
||||
|
||||
return builder.ToImmutable();
|
||||
}
|
||||
|
||||
private static ImmutableArray<CycloneDxVulnerability> CollectVulnerabilities(JsonElement root)
|
||||
{
|
||||
if (!root.TryGetProperty("vulnerabilities", out var vulnerabilitiesElement) ||
|
||||
vulnerabilitiesElement.ValueKind != JsonValueKind.Array)
|
||||
{
|
||||
return ImmutableArray<CycloneDxVulnerability>.Empty;
|
||||
}
|
||||
|
||||
var builder = ImmutableArray.CreateBuilder<CycloneDxVulnerability>();
|
||||
|
||||
foreach (var vulnerability in vulnerabilitiesElement.EnumerateArray())
|
||||
{
|
||||
if (vulnerability.ValueKind != JsonValueKind.Object)
|
||||
{
|
||||
continue;
|
||||
}
|
||||
|
||||
var vulnerabilityId =
|
||||
TryGetString(vulnerability, "id") ??
|
||||
TryGetString(vulnerability, "bom-ref") ??
|
||||
TryGetString(vulnerability, "bomRef") ??
|
||||
TryGetString(vulnerability, "cve");
|
||||
|
||||
if (string.IsNullOrWhiteSpace(vulnerabilityId))
|
||||
{
|
||||
continue;
|
||||
}
|
||||
|
||||
var detail = TryGetString(vulnerability, "detail") ?? TryGetString(vulnerability, "description");
|
||||
|
||||
var analysis = TryGetProperty(vulnerability, "analysis");
|
||||
var analysisState = TryGetString(analysis, "state");
|
||||
var analysisJustification = TryGetString(analysis, "justification");
|
||||
var analysisResponses = CollectResponses(analysis);
|
||||
|
||||
var affects = CollectAffects(vulnerability);
|
||||
if (affects.Length == 0)
|
||||
{
|
||||
continue;
|
||||
}
|
||||
|
||||
builder.Add(new CycloneDxVulnerability(
|
||||
vulnerabilityId.Trim(),
|
||||
detail?.Trim(),
|
||||
analysisState,
|
||||
analysisJustification,
|
||||
analysisResponses,
|
||||
affects));
|
||||
}
|
||||
|
||||
return builder.ToImmutable();
|
||||
}
|
||||
|
||||
private static ImmutableArray<string> CollectResponses(JsonElement analysis)
|
||||
{
|
||||
if (analysis.ValueKind != JsonValueKind.Object ||
|
||||
!analysis.TryGetProperty("response", out var responseElement) ||
|
||||
responseElement.ValueKind != JsonValueKind.Array)
|
||||
{
|
||||
return ImmutableArray<string>.Empty;
|
||||
}
|
||||
|
||||
var responses = new SortedSet<string>(StringComparer.OrdinalIgnoreCase);
|
||||
foreach (var response in responseElement.EnumerateArray())
|
||||
{
|
||||
if (response.ValueKind == JsonValueKind.String)
|
||||
{
|
||||
var value = response.GetString();
|
||||
if (!string.IsNullOrWhiteSpace(value))
|
||||
{
|
||||
responses.Add(value.Trim());
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
return responses.Count == 0 ? ImmutableArray<string>.Empty : responses.ToImmutableArray();
|
||||
}
|
||||
|
||||
private static ImmutableArray<CycloneDxAffect> CollectAffects(JsonElement vulnerability)
|
||||
{
|
||||
if (!vulnerability.TryGetProperty("affects", out var affectsElement) ||
|
||||
affectsElement.ValueKind != JsonValueKind.Array)
|
||||
{
|
||||
return ImmutableArray<CycloneDxAffect>.Empty;
|
||||
}
|
||||
|
||||
var builder = ImmutableArray.CreateBuilder<CycloneDxAffect>();
|
||||
foreach (var affect in affectsElement.EnumerateArray())
|
||||
{
|
||||
if (affect.ValueKind != JsonValueKind.Object)
|
||||
{
|
||||
continue;
|
||||
}
|
||||
|
||||
var reference = TryGetString(affect, "ref");
|
||||
if (string.IsNullOrWhiteSpace(reference))
|
||||
{
|
||||
continue;
|
||||
}
|
||||
|
||||
builder.Add(new CycloneDxAffect(reference.Trim()));
|
||||
}
|
||||
|
||||
return builder.ToImmutable();
|
||||
}
|
||||
|
||||
private static JsonElement TryGetProperty(JsonElement element, string propertyName)
|
||||
=> element.ValueKind == JsonValueKind.Object && element.TryGetProperty(propertyName, out var value)
|
||||
? value
|
||||
: default;
|
||||
|
||||
private static string? TryGetString(JsonElement element, string propertyName)
|
||||
{
|
||||
if (element.ValueKind != JsonValueKind.Object)
|
||||
{
|
||||
return null;
|
||||
}
|
||||
|
||||
if (!element.TryGetProperty(propertyName, out var value))
|
||||
{
|
||||
return null;
|
||||
}
|
||||
|
||||
return value.ValueKind == JsonValueKind.String ? value.GetString() : null;
|
||||
}
|
||||
|
||||
private static DateTimeOffset? ParseDate(JsonElement element, string propertyName)
|
||||
{
|
||||
var value = TryGetString(element, propertyName);
|
||||
if (string.IsNullOrWhiteSpace(value))
|
||||
{
|
||||
return null;
|
||||
}
|
||||
|
||||
return DateTimeOffset.TryParse(value, out var parsed) ? parsed : null;
|
||||
}
|
||||
}
|
||||
|
||||
private sealed record CycloneDxParseResult(
|
||||
ImmutableDictionary<string, string> Metadata,
|
||||
string? BomVersion,
|
||||
DateTimeOffset FirstObserved,
|
||||
DateTimeOffset LastObserved,
|
||||
ImmutableDictionary<string, CycloneDxComponent> Components,
|
||||
ImmutableArray<CycloneDxVulnerability> Vulnerabilities)
|
||||
{
|
||||
public CycloneDxProductInfo ResolveProduct(string? componentRef)
|
||||
{
|
||||
if (!string.IsNullOrWhiteSpace(componentRef) &&
|
||||
Components.TryGetValue(componentRef.Trim(), out var component))
|
||||
{
|
||||
return new CycloneDxProductInfo(component.Reference, component.Name, component.Version, component.Purl, component.Cpe);
|
||||
}
|
||||
|
||||
var key = string.IsNullOrWhiteSpace(componentRef) ? "unknown-component" : componentRef.Trim();
|
||||
return new CycloneDxProductInfo(key, key, null, null, null);
|
||||
}
|
||||
}
|
||||
|
||||
private sealed record CycloneDxComponent(
|
||||
string Reference,
|
||||
string Name,
|
||||
string? Version,
|
||||
string? Purl,
|
||||
string? Cpe);
|
||||
|
||||
private sealed record CycloneDxVulnerability(
|
||||
string VulnerabilityId,
|
||||
string? Detail,
|
||||
string? AnalysisState,
|
||||
string? AnalysisJustification,
|
||||
ImmutableArray<string> AnalysisResponses,
|
||||
ImmutableArray<CycloneDxAffect> Affects);
|
||||
|
||||
private sealed record CycloneDxAffect(string ComponentRef);
|
||||
|
||||
private sealed record CycloneDxProductInfo(
|
||||
string Key,
|
||||
string Name,
|
||||
string? Version,
|
||||
string? Purl,
|
||||
string? Cpe);
|
||||
}
|
||||
@@ -0,0 +1,14 @@
|
||||
using Microsoft.Extensions.DependencyInjection;
|
||||
using StellaOps.Vexer.Core;
|
||||
|
||||
namespace StellaOps.Vexer.Formats.CycloneDX;
|
||||
|
||||
public static class CycloneDxFormatsServiceCollectionExtensions
|
||||
{
|
||||
public static IServiceCollection AddCycloneDxNormalizer(this IServiceCollection services)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(services);
|
||||
services.AddSingleton<IVexNormalizer, CycloneDxNormalizer>();
|
||||
return services;
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,16 @@
|
||||
<Project Sdk="Microsoft.NET.Sdk">
|
||||
<PropertyGroup>
|
||||
<TargetFramework>net10.0</TargetFramework>
|
||||
<LangVersion>preview</LangVersion>
|
||||
<Nullable>enable</Nullable>
|
||||
<ImplicitUsings>enable</ImplicitUsings>
|
||||
<TreatWarningsAsErrors>true</TreatWarningsAsErrors>
|
||||
</PropertyGroup>
|
||||
<ItemGroup>
|
||||
<PackageReference Include="Microsoft.Extensions.DependencyInjection.Abstractions" Version="8.0.0" />
|
||||
<PackageReference Include="Microsoft.Extensions.Logging.Abstractions" Version="8.0.0" />
|
||||
</ItemGroup>
|
||||
<ItemGroup>
|
||||
<ProjectReference Include="..\StellaOps.Vexer.Core\StellaOps.Vexer.Core.csproj" />
|
||||
</ItemGroup>
|
||||
</Project>
|
||||
@@ -2,6 +2,6 @@ If you are working on this file you need to read docs/ARCHITECTURE_VEXER.md and
|
||||
# TASKS
|
||||
| Task | Owner(s) | Depends on | Notes |
|
||||
|---|---|---|---|
|
||||
|VEXER-FMT-CYCLONE-01-001 – CycloneDX VEX normalizer|Team Vexer Formats|VEXER-CORE-01-001|TODO – Parse CycloneDX `analysis` data into `VexClaim` entries with deterministic component identifiers.|
|
||||
|VEXER-FMT-CYCLONE-01-001 – CycloneDX VEX normalizer|Team Vexer Formats|VEXER-CORE-01-001|**DONE (2025-10-17)** – CycloneDX normalizer parses `analysis` data, resolves component references, and emits canonical `VexClaim`s; regression lives in `CycloneDxNormalizerTests`.|
|
||||
|VEXER-FMT-CYCLONE-01-002 – Component reference reconciliation|Team Vexer Formats|VEXER-FMT-CYCLONE-01-001|TODO – Implement helpers to reconcile component/service references against policy expectations and emit diagnostics for missing SBOM links.|
|
||||
|VEXER-FMT-CYCLONE-01-003 – CycloneDX export serializer|Team Vexer Formats|VEXER-EXPORT-01-001, VEXER-FMT-CYCLONE-01-001|TODO – Provide exporters producing CycloneDX VEX output with canonical ordering and hash-stable manifests.|
|
||||
|
||||
@@ -0,0 +1,87 @@
|
||||
using System.Collections.Immutable;
|
||||
using System.Linq;
|
||||
using System.Text;
|
||||
using FluentAssertions;
|
||||
using Microsoft.Extensions.Logging.Abstractions;
|
||||
using StellaOps.Vexer.Core;
|
||||
using StellaOps.Vexer.Formats.OpenVEX;
|
||||
|
||||
namespace StellaOps.Vexer.Formats.OpenVEX.Tests;
|
||||
|
||||
public sealed class OpenVexNormalizerTests
|
||||
{
|
||||
[Fact]
|
||||
public async Task NormalizeAsync_ProducesClaimsForStatements()
|
||||
{
|
||||
var json = """
|
||||
{
|
||||
"document": {
|
||||
"author": "Acme Security",
|
||||
"version": "1",
|
||||
"issued": "2025-10-01T00:00:00Z",
|
||||
"last_updated": "2025-10-05T00:00:00Z"
|
||||
},
|
||||
"statements": [
|
||||
{
|
||||
"id": "statement-1",
|
||||
"vulnerability": "CVE-2025-2000",
|
||||
"status": "not_affected",
|
||||
"justification": "code_not_present",
|
||||
"products": [
|
||||
{
|
||||
"id": "acme-widget@1.2.3",
|
||||
"name": "Acme Widget",
|
||||
"version": "1.2.3",
|
||||
"purl": "pkg:acme/widget@1.2.3",
|
||||
"cpe": "cpe:/a:acme:widget:1.2.3"
|
||||
}
|
||||
],
|
||||
"statement": "The vulnerable code was never shipped."
|
||||
},
|
||||
{
|
||||
"id": "statement-2",
|
||||
"vulnerability": "CVE-2025-2001",
|
||||
"status": "affected",
|
||||
"products": [
|
||||
"pkg:acme/widget@2.0.0"
|
||||
],
|
||||
"remediation": "Upgrade to 2.1.0"
|
||||
}
|
||||
]
|
||||
}
|
||||
""";
|
||||
|
||||
var rawDocument = new VexRawDocument(
|
||||
"vexer:openvex",
|
||||
VexDocumentFormat.OpenVex,
|
||||
new Uri("https://example.com/openvex.json"),
|
||||
new DateTimeOffset(2025, 10, 6, 0, 0, 0, TimeSpan.Zero),
|
||||
"sha256:dummydigest",
|
||||
Encoding.UTF8.GetBytes(json),
|
||||
ImmutableDictionary<string, string>.Empty);
|
||||
|
||||
var provider = new VexProvider("vexer:openvex", "OpenVEX Provider", VexProviderKind.Vendor);
|
||||
var normalizer = new OpenVexNormalizer(NullLogger<OpenVexNormalizer>.Instance);
|
||||
|
||||
var batch = await normalizer.NormalizeAsync(rawDocument, provider, CancellationToken.None);
|
||||
|
||||
batch.Claims.Should().HaveCount(2);
|
||||
|
||||
var notAffected = batch.Claims.Single(c => c.VulnerabilityId == "CVE-2025-2000");
|
||||
notAffected.Status.Should().Be(VexClaimStatus.NotAffected);
|
||||
notAffected.Justification.Should().Be(VexJustification.CodeNotPresent);
|
||||
notAffected.Product.Key.Should().Be("acme-widget@1.2.3");
|
||||
notAffected.Product.Purl.Should().Be("pkg:acme/widget@1.2.3");
|
||||
notAffected.Document.Revision.Should().Be("1");
|
||||
notAffected.AdditionalMetadata["openvex.document.author"].Should().Be("Acme Security");
|
||||
notAffected.AdditionalMetadata["openvex.statement.status"].Should().Be("not_affected");
|
||||
notAffected.Detail.Should().Be("The vulnerable code was never shipped.");
|
||||
|
||||
var affected = batch.Claims.Single(c => c.VulnerabilityId == "CVE-2025-2001");
|
||||
affected.Status.Should().Be(VexClaimStatus.Affected);
|
||||
affected.Justification.Should().BeNull();
|
||||
affected.Product.Key.Should().Be("pkg:acme/widget@2.0.0");
|
||||
affected.Product.Name.Should().Be("pkg:acme/widget@2.0.0");
|
||||
affected.Detail.Should().Be("Upgrade to 2.1.0");
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,17 @@
|
||||
<Project Sdk="Microsoft.NET.Sdk">
|
||||
<PropertyGroup>
|
||||
<TargetFramework>net10.0</TargetFramework>
|
||||
<Nullable>enable</Nullable>
|
||||
<ImplicitUsings>enable</ImplicitUsings>
|
||||
</PropertyGroup>
|
||||
<ItemGroup>
|
||||
<PackageReference Include="FluentAssertions" Version="6.12.0" />
|
||||
</ItemGroup>
|
||||
<ItemGroup>
|
||||
<Using Include="Xunit" />
|
||||
</ItemGroup>
|
||||
<ItemGroup>
|
||||
<ProjectReference Include="..\StellaOps.Vexer.Core\StellaOps.Vexer.Core.csproj" />
|
||||
<ProjectReference Include="..\StellaOps.Vexer.Formats.OpenVEX\StellaOps.Vexer.Formats.OpenVEX.csproj" />
|
||||
</ItemGroup>
|
||||
</Project>
|
||||
367
src/StellaOps.Vexer.Formats.OpenVEX/OpenVexNormalizer.cs
Normal file
367
src/StellaOps.Vexer.Formats.OpenVEX/OpenVexNormalizer.cs
Normal file
@@ -0,0 +1,367 @@
|
||||
using System;
|
||||
using System.Collections.Generic;
|
||||
using System.Collections.Immutable;
|
||||
using System.Linq;
|
||||
using System.Text.Json;
|
||||
using System.Threading;
|
||||
using System.Threading.Tasks;
|
||||
using Microsoft.Extensions.Logging;
|
||||
using StellaOps.Vexer.Core;
|
||||
|
||||
namespace StellaOps.Vexer.Formats.OpenVEX;
|
||||
|
||||
public sealed class OpenVexNormalizer : IVexNormalizer
|
||||
{
|
||||
private static readonly ImmutableDictionary<string, VexClaimStatus> StatusMap = new Dictionary<string, VexClaimStatus>(StringComparer.OrdinalIgnoreCase)
|
||||
{
|
||||
["affected"] = VexClaimStatus.Affected,
|
||||
["not_affected"] = VexClaimStatus.NotAffected,
|
||||
["fixed"] = VexClaimStatus.Fixed,
|
||||
["under_investigation"] = VexClaimStatus.UnderInvestigation,
|
||||
}.ToImmutableDictionary(StringComparer.OrdinalIgnoreCase);
|
||||
|
||||
private static readonly ImmutableDictionary<string, VexJustification> JustificationMap = new Dictionary<string, VexJustification>(StringComparer.OrdinalIgnoreCase)
|
||||
{
|
||||
["component_not_present"] = VexJustification.ComponentNotPresent,
|
||||
["component_not_configured"] = VexJustification.ComponentNotConfigured,
|
||||
["vulnerable_code_not_present"] = VexJustification.VulnerableCodeNotPresent,
|
||||
["vulnerable_code_not_in_execute_path"] = VexJustification.VulnerableCodeNotInExecutePath,
|
||||
["vulnerable_code_cannot_be_controlled_by_adversary"] = VexJustification.VulnerableCodeCannotBeControlledByAdversary,
|
||||
["inline_mitigations_already_exist"] = VexJustification.InlineMitigationsAlreadyExist,
|
||||
["protected_by_mitigating_control"] = VexJustification.ProtectedByMitigatingControl,
|
||||
["protected_by_compensating_control"] = VexJustification.ProtectedByCompensatingControl,
|
||||
["code_not_present"] = VexJustification.CodeNotPresent,
|
||||
["code_not_reachable"] = VexJustification.CodeNotReachable,
|
||||
["requires_configuration"] = VexJustification.RequiresConfiguration,
|
||||
["requires_dependency"] = VexJustification.RequiresDependency,
|
||||
["requires_environment"] = VexJustification.RequiresEnvironment,
|
||||
}.ToImmutableDictionary(StringComparer.OrdinalIgnoreCase);
|
||||
|
||||
private readonly ILogger<OpenVexNormalizer> _logger;
|
||||
|
||||
public OpenVexNormalizer(ILogger<OpenVexNormalizer> logger)
|
||||
{
|
||||
_logger = logger ?? throw new ArgumentNullException(nameof(logger));
|
||||
}
|
||||
|
||||
public string Format => VexDocumentFormat.OpenVex.ToString().ToLowerInvariant();
|
||||
|
||||
public bool CanHandle(VexRawDocument document)
|
||||
=> document is not null && document.Format == VexDocumentFormat.OpenVex;
|
||||
|
||||
public ValueTask<VexClaimBatch> NormalizeAsync(VexRawDocument document, VexProvider provider, CancellationToken cancellationToken)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(document);
|
||||
ArgumentNullException.ThrowIfNull(provider);
|
||||
cancellationToken.ThrowIfCancellationRequested();
|
||||
|
||||
try
|
||||
{
|
||||
var result = OpenVexParser.Parse(document);
|
||||
var claims = ImmutableArray.CreateBuilder<VexClaim>(result.Statements.Length);
|
||||
|
||||
foreach (var statement in result.Statements)
|
||||
{
|
||||
cancellationToken.ThrowIfCancellationRequested();
|
||||
|
||||
var status = MapStatus(statement.Status);
|
||||
var justification = MapJustification(statement.Justification);
|
||||
|
||||
foreach (var product in statement.Products)
|
||||
{
|
||||
var vexProduct = new VexProduct(
|
||||
product.Key,
|
||||
product.Name,
|
||||
product.Version,
|
||||
product.Purl,
|
||||
product.Cpe);
|
||||
|
||||
var metadata = result.Metadata;
|
||||
|
||||
metadata = metadata.SetItem("openvex.statement.id", statement.Id);
|
||||
if (!string.IsNullOrWhiteSpace(statement.Status))
|
||||
{
|
||||
metadata = metadata.SetItem("openvex.statement.status", statement.Status!);
|
||||
}
|
||||
|
||||
if (!string.IsNullOrWhiteSpace(statement.Justification))
|
||||
{
|
||||
metadata = metadata.SetItem("openvex.statement.justification", statement.Justification!);
|
||||
}
|
||||
|
||||
if (!string.IsNullOrWhiteSpace(product.OriginalId))
|
||||
{
|
||||
metadata = metadata.SetItem("openvex.product.source", product.OriginalId!);
|
||||
}
|
||||
|
||||
var claimDocument = new VexClaimDocument(
|
||||
VexDocumentFormat.OpenVex,
|
||||
document.Digest,
|
||||
document.SourceUri,
|
||||
result.DocumentVersion,
|
||||
signature: null);
|
||||
|
||||
var claim = new VexClaim(
|
||||
statement.Vulnerability,
|
||||
provider.Id,
|
||||
vexProduct,
|
||||
status,
|
||||
claimDocument,
|
||||
result.FirstObserved,
|
||||
result.LastObserved,
|
||||
justification,
|
||||
statement.Remarks,
|
||||
confidence: null,
|
||||
additionalMetadata: metadata);
|
||||
|
||||
claims.Add(claim);
|
||||
}
|
||||
}
|
||||
|
||||
var orderedClaims = claims
|
||||
.ToImmutable()
|
||||
.OrderBy(static claim => claim.VulnerabilityId, StringComparer.Ordinal)
|
||||
.ThenBy(static claim => claim.Product.Key, StringComparer.Ordinal)
|
||||
.ToImmutableArray();
|
||||
|
||||
_logger.LogInformation(
|
||||
"Normalized OpenVEX document {Source} into {ClaimCount} claim(s).",
|
||||
document.SourceUri,
|
||||
orderedClaims.Length);
|
||||
|
||||
return ValueTask.FromResult(new VexClaimBatch(
|
||||
document,
|
||||
orderedClaims,
|
||||
ImmutableDictionary<string, string>.Empty));
|
||||
}
|
||||
catch (JsonException ex)
|
||||
{
|
||||
_logger.LogError(ex, "Failed to parse OpenVEX document {SourceUri}", document.SourceUri);
|
||||
throw;
|
||||
}
|
||||
}
|
||||
|
||||
private static VexClaimStatus MapStatus(string? status)
|
||||
{
|
||||
if (!string.IsNullOrWhiteSpace(status) && StatusMap.TryGetValue(status.Trim(), out var mapped))
|
||||
{
|
||||
return mapped;
|
||||
}
|
||||
|
||||
return VexClaimStatus.UnderInvestigation;
|
||||
}
|
||||
|
||||
private static VexJustification? MapJustification(string? justification)
|
||||
{
|
||||
if (string.IsNullOrWhiteSpace(justification))
|
||||
{
|
||||
return null;
|
||||
}
|
||||
|
||||
return JustificationMap.TryGetValue(justification.Trim(), out var mapped)
|
||||
? mapped
|
||||
: null;
|
||||
}
|
||||
|
||||
private static class OpenVexParser
|
||||
{
|
||||
public static OpenVexParseResult Parse(VexRawDocument document)
|
||||
{
|
||||
using var json = JsonDocument.Parse(document.Content.ToArray());
|
||||
var root = json.RootElement;
|
||||
|
||||
var metadata = ImmutableDictionary.CreateBuilder<string, string>(StringComparer.Ordinal);
|
||||
|
||||
var documentElement = TryGetProperty(root, "document");
|
||||
var version = TryGetString(documentElement, "version");
|
||||
var author = TryGetString(documentElement, "author");
|
||||
|
||||
if (!string.IsNullOrWhiteSpace(version))
|
||||
{
|
||||
metadata["openvex.document.version"] = version!;
|
||||
}
|
||||
|
||||
if (!string.IsNullOrWhiteSpace(author))
|
||||
{
|
||||
metadata["openvex.document.author"] = author!;
|
||||
}
|
||||
|
||||
var issued = ParseDate(documentElement, "issued");
|
||||
var lastUpdated = ParseDate(documentElement, "last_updated") ?? issued ?? document.RetrievedAt;
|
||||
var effectiveDate = ParseDate(documentElement, "effective_date") ?? issued ?? document.RetrievedAt;
|
||||
|
||||
var statements = CollectStatements(root);
|
||||
|
||||
return new OpenVexParseResult(
|
||||
metadata.ToImmutable(),
|
||||
version,
|
||||
effectiveDate,
|
||||
lastUpdated,
|
||||
statements);
|
||||
}
|
||||
|
||||
private static ImmutableArray<OpenVexStatement> CollectStatements(JsonElement root)
|
||||
{
|
||||
if (!root.TryGetProperty("statements", out var statementsElement) ||
|
||||
statementsElement.ValueKind != JsonValueKind.Array)
|
||||
{
|
||||
return ImmutableArray<OpenVexStatement>.Empty;
|
||||
}
|
||||
|
||||
var builder = ImmutableArray.CreateBuilder<OpenVexStatement>();
|
||||
foreach (var statement in statementsElement.EnumerateArray())
|
||||
{
|
||||
if (statement.ValueKind != JsonValueKind.Object)
|
||||
{
|
||||
continue;
|
||||
}
|
||||
|
||||
var vulnerability = TryGetString(statement, "vulnerability") ?? TryGetString(statement, "vuln") ?? string.Empty;
|
||||
if (string.IsNullOrWhiteSpace(vulnerability))
|
||||
{
|
||||
continue;
|
||||
}
|
||||
|
||||
var id = TryGetString(statement, "id") ?? Guid.NewGuid().ToString();
|
||||
var status = TryGetString(statement, "status");
|
||||
var justification = TryGetString(statement, "justification");
|
||||
var remarks = TryGetString(statement, "remediation") ?? TryGetString(statement, "statement");
|
||||
var products = CollectProducts(statement);
|
||||
if (products.Length == 0)
|
||||
{
|
||||
continue;
|
||||
}
|
||||
|
||||
builder.Add(new OpenVexStatement(
|
||||
id,
|
||||
vulnerability.Trim(),
|
||||
status,
|
||||
justification,
|
||||
remarks,
|
||||
products));
|
||||
}
|
||||
|
||||
return builder.ToImmutable();
|
||||
}
|
||||
|
||||
private static ImmutableArray<OpenVexProduct> CollectProducts(JsonElement statement)
|
||||
{
|
||||
if (!statement.TryGetProperty("products", out var productsElement) ||
|
||||
productsElement.ValueKind != JsonValueKind.Array)
|
||||
{
|
||||
return ImmutableArray<OpenVexProduct>.Empty;
|
||||
}
|
||||
|
||||
var builder = ImmutableArray.CreateBuilder<OpenVexProduct>();
|
||||
foreach (var product in productsElement.EnumerateArray())
|
||||
{
|
||||
if (product.ValueKind != JsonValueKind.String && product.ValueKind != JsonValueKind.Object)
|
||||
{
|
||||
continue;
|
||||
}
|
||||
|
||||
if (product.ValueKind == JsonValueKind.String)
|
||||
{
|
||||
var value = product.GetString();
|
||||
if (string.IsNullOrWhiteSpace(value))
|
||||
{
|
||||
continue;
|
||||
}
|
||||
|
||||
builder.Add(OpenVexProduct.FromString(value.Trim()));
|
||||
continue;
|
||||
}
|
||||
|
||||
var id = TryGetString(product, "id") ?? TryGetString(product, "product_id");
|
||||
var name = TryGetString(product, "name");
|
||||
var version = TryGetString(product, "version");
|
||||
var purl = TryGetString(product, "purl");
|
||||
var cpe = TryGetString(product, "cpe");
|
||||
|
||||
if (string.IsNullOrWhiteSpace(id) && string.IsNullOrWhiteSpace(purl))
|
||||
{
|
||||
continue;
|
||||
}
|
||||
|
||||
builder.Add(new OpenVexProduct(
|
||||
id ?? purl!,
|
||||
name ?? id ?? purl!,
|
||||
version,
|
||||
purl,
|
||||
cpe,
|
||||
OriginalId: id));
|
||||
}
|
||||
|
||||
return builder.ToImmutable();
|
||||
}
|
||||
|
||||
private static JsonElement TryGetProperty(JsonElement element, string propertyName)
|
||||
=> element.ValueKind == JsonValueKind.Object && element.TryGetProperty(propertyName, out var value)
|
||||
? value
|
||||
: default;
|
||||
|
||||
private static string? TryGetString(JsonElement element, string propertyName)
|
||||
{
|
||||
if (element.ValueKind != JsonValueKind.Object)
|
||||
{
|
||||
return null;
|
||||
}
|
||||
|
||||
if (!element.TryGetProperty(propertyName, out var value))
|
||||
{
|
||||
return null;
|
||||
}
|
||||
|
||||
return value.ValueKind == JsonValueKind.String ? value.GetString() : null;
|
||||
}
|
||||
|
||||
private static DateTimeOffset? ParseDate(JsonElement element, string propertyName)
|
||||
{
|
||||
var value = TryGetString(element, propertyName);
|
||||
if (string.IsNullOrWhiteSpace(value))
|
||||
{
|
||||
return null;
|
||||
}
|
||||
|
||||
return DateTimeOffset.TryParse(value, out var parsed) ? parsed : null;
|
||||
}
|
||||
}
|
||||
|
||||
private sealed record OpenVexParseResult(
|
||||
ImmutableDictionary<string, string> Metadata,
|
||||
string? DocumentVersion,
|
||||
DateTimeOffset FirstObserved,
|
||||
DateTimeOffset LastObserved,
|
||||
ImmutableArray<OpenVexStatement> Statements);
|
||||
|
||||
private sealed record OpenVexStatement(
|
||||
string Id,
|
||||
string Vulnerability,
|
||||
string? Status,
|
||||
string? Justification,
|
||||
string? Remarks,
|
||||
ImmutableArray<OpenVexProduct> Products);
|
||||
|
||||
private sealed record OpenVexProduct(
|
||||
string Key,
|
||||
string Name,
|
||||
string? Version,
|
||||
string? Purl,
|
||||
string? Cpe,
|
||||
string? OriginalId)
|
||||
{
|
||||
public static OpenVexProduct FromString(string value)
|
||||
{
|
||||
var key = value;
|
||||
string? purl = null;
|
||||
string? name = value;
|
||||
|
||||
if (value.StartsWith("pkg:", StringComparison.OrdinalIgnoreCase))
|
||||
{
|
||||
purl = value;
|
||||
}
|
||||
|
||||
return new OpenVexProduct(key, name, null, purl, null, OriginalId: value);
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,14 @@
|
||||
using Microsoft.Extensions.DependencyInjection;
|
||||
using StellaOps.Vexer.Core;
|
||||
|
||||
namespace StellaOps.Vexer.Formats.OpenVEX;
|
||||
|
||||
public static class OpenVexFormatsServiceCollectionExtensions
|
||||
{
|
||||
public static IServiceCollection AddOpenVexNormalizer(this IServiceCollection services)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(services);
|
||||
services.AddSingleton<IVexNormalizer, OpenVexNormalizer>();
|
||||
return services;
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,16 @@
|
||||
<Project Sdk="Microsoft.NET.Sdk">
|
||||
<PropertyGroup>
|
||||
<TargetFramework>net10.0</TargetFramework>
|
||||
<LangVersion>preview</LangVersion>
|
||||
<Nullable>enable</Nullable>
|
||||
<ImplicitUsings>enable</ImplicitUsings>
|
||||
<TreatWarningsAsErrors>true</TreatWarningsAsErrors>
|
||||
</PropertyGroup>
|
||||
<ItemGroup>
|
||||
<PackageReference Include="Microsoft.Extensions.DependencyInjection.Abstractions" Version="8.0.0" />
|
||||
<PackageReference Include="Microsoft.Extensions.Logging.Abstractions" Version="8.0.0" />
|
||||
</ItemGroup>
|
||||
<ItemGroup>
|
||||
<ProjectReference Include="..\StellaOps.Vexer.Core\StellaOps.Vexer.Core.csproj" />
|
||||
</ItemGroup>
|
||||
</Project>
|
||||
@@ -2,6 +2,6 @@ If you are working on this file you need to read docs/ARCHITECTURE_VEXER.md and
|
||||
# TASKS
|
||||
| Task | Owner(s) | Depends on | Notes |
|
||||
|---|---|---|---|
|
||||
|VEXER-FMT-OPENVEX-01-001 – OpenVEX normalizer|Team Vexer Formats|VEXER-CORE-01-001|TODO – Implement OpenVEX parser covering statements, products, and `status/justification` mapping with provenance metadata.|
|
||||
|VEXER-FMT-OPENVEX-01-001 – OpenVEX normalizer|Team Vexer Formats|VEXER-CORE-01-001|**DONE (2025-10-17)** – OpenVEX normalizer parses statements/products, maps status/justification, and surfaces provenance metadata; coverage in `OpenVexNormalizerTests`.|
|
||||
|VEXER-FMT-OPENVEX-01-002 – Statement merge utilities|Team Vexer Formats|VEXER-FMT-OPENVEX-01-001|TODO – Add reducers merging multiple OpenVEX statements, resolving conflicts deterministically, and emitting policy diagnostics.|
|
||||
|VEXER-FMT-OPENVEX-01-003 – OpenVEX export writer|Team Vexer Formats|VEXER-EXPORT-01-001, VEXER-FMT-OPENVEX-01-001|TODO – Provide export serializer generating canonical OpenVEX documents with optional SBOM references and hash-stable ordering.|
|
||||
|
||||
@@ -1,5 +1,6 @@
|
||||
using System;
|
||||
using System.Collections.Generic;
|
||||
using System.Collections.Immutable;
|
||||
using System.Threading;
|
||||
using System.Threading.Tasks;
|
||||
using StellaOps.Vexer.Core;
|
||||
@@ -24,6 +25,18 @@ public interface IVexConsensusStore
|
||||
ValueTask SaveAsync(VexConsensus consensus, CancellationToken cancellationToken);
|
||||
}
|
||||
|
||||
public sealed record VexConnectorState(
|
||||
string ConnectorId,
|
||||
DateTimeOffset? LastUpdated,
|
||||
ImmutableArray<string> DocumentDigests);
|
||||
|
||||
public interface IVexConnectorStateRepository
|
||||
{
|
||||
ValueTask<VexConnectorState?> GetAsync(string connectorId, CancellationToken cancellationToken);
|
||||
|
||||
ValueTask SaveAsync(VexConnectorState state, CancellationToken cancellationToken);
|
||||
}
|
||||
|
||||
public interface IVexCacheIndex
|
||||
{
|
||||
ValueTask<VexCacheEntry?> FindAsync(VexQuerySignature signature, VexExportFormat format, CancellationToken cancellationToken);
|
||||
|
||||
@@ -0,0 +1,55 @@
|
||||
using System;
|
||||
using System.Collections.Immutable;
|
||||
using System.Linq;
|
||||
using System.Threading;
|
||||
using System.Threading.Tasks;
|
||||
using MongoDB.Driver;
|
||||
|
||||
namespace StellaOps.Vexer.Storage.Mongo;
|
||||
|
||||
public sealed class MongoVexConnectorStateRepository : IVexConnectorStateRepository
|
||||
{
|
||||
private readonly IMongoCollection<VexConnectorStateDocument> _collection;
|
||||
|
||||
public MongoVexConnectorStateRepository(IMongoDatabase database)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(database);
|
||||
VexMongoMappingRegistry.Register();
|
||||
_collection = database.GetCollection<VexConnectorStateDocument>(VexMongoCollectionNames.ConnectorState);
|
||||
}
|
||||
|
||||
public async ValueTask<VexConnectorState?> GetAsync(string connectorId, CancellationToken cancellationToken)
|
||||
{
|
||||
ArgumentException.ThrowIfNullOrWhiteSpace(connectorId);
|
||||
|
||||
var filter = Builders<VexConnectorStateDocument>.Filter.Eq(x => x.ConnectorId, connectorId.Trim());
|
||||
var document = await _collection.Find(filter).FirstOrDefaultAsync(cancellationToken).ConfigureAwait(false);
|
||||
return document?.ToRecord();
|
||||
}
|
||||
|
||||
public async ValueTask SaveAsync(VexConnectorState state, CancellationToken cancellationToken)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(state);
|
||||
|
||||
var document = VexConnectorStateDocument.FromRecord(state.WithNormalizedDigests());
|
||||
var filter = Builders<VexConnectorStateDocument>.Filter.Eq(x => x.ConnectorId, document.ConnectorId);
|
||||
await _collection.ReplaceOneAsync(filter, document, new ReplaceOptions { IsUpsert = true }, cancellationToken).ConfigureAwait(false);
|
||||
}
|
||||
}
|
||||
|
||||
internal static class VexConnectorStateExtensions
|
||||
{
|
||||
private const int MaxDigestHistory = 200;
|
||||
|
||||
public static VexConnectorState WithNormalizedDigests(this VexConnectorState state)
|
||||
{
|
||||
var digests = state.DocumentDigests;
|
||||
if (digests.Length <= MaxDigestHistory)
|
||||
{
|
||||
return state;
|
||||
}
|
||||
|
||||
var trimmed = digests.Skip(digests.Length - MaxDigestHistory).ToImmutableArray();
|
||||
return state with { DocumentDigests = trimmed };
|
||||
}
|
||||
}
|
||||
@@ -16,6 +16,7 @@ public static class VexMongoServiceCollectionExtensions
|
||||
services.AddSingleton<IVexConsensusStore, MongoVexConsensusStore>();
|
||||
services.AddSingleton<IVexCacheIndex, MongoVexCacheIndex>();
|
||||
services.AddSingleton<IVexCacheMaintenance, MongoVexCacheMaintenance>();
|
||||
services.AddSingleton<IVexConnectorStateRepository, MongoVexConnectorStateRepository>();
|
||||
services.AddSingleton<IVexMongoMigration, VexInitialIndexMigration>();
|
||||
services.AddSingleton<VexMongoMigrationRunner>();
|
||||
services.AddHostedService<VexMongoMigrationHostedService>();
|
||||
|
||||
@@ -39,6 +39,7 @@ public static class VexMongoMappingRegistry
|
||||
RegisterClassMap<VexConsensusConflictDocument>();
|
||||
RegisterClassMap<VexConfidenceDocument>();
|
||||
RegisterClassMap<VexCacheEntryRecord>();
|
||||
RegisterClassMap<VexConnectorStateDocument>();
|
||||
}
|
||||
|
||||
private static void RegisterClassMap<TDocument>()
|
||||
@@ -66,4 +67,5 @@ public static class VexMongoCollectionNames
|
||||
public const string Consensus = "vex.consensus";
|
||||
public const string Exports = "vex.exports";
|
||||
public const string Cache = "vex.cache";
|
||||
public const string ConnectorState = "vex.connector_state";
|
||||
}
|
||||
|
||||
@@ -570,3 +570,35 @@ internal sealed class VexCacheEntryRecord
|
||||
expires);
|
||||
}
|
||||
}
|
||||
|
||||
[BsonIgnoreExtraElements]
|
||||
internal sealed class VexConnectorStateDocument
|
||||
{
|
||||
[BsonId]
|
||||
public string ConnectorId { get; set; } = default!;
|
||||
|
||||
public DateTime? LastUpdated { get; set; }
|
||||
= null;
|
||||
|
||||
public List<string> DocumentDigests { get; set; } = new();
|
||||
|
||||
public static VexConnectorStateDocument FromRecord(VexConnectorState state)
|
||||
=> new()
|
||||
{
|
||||
ConnectorId = state.ConnectorId,
|
||||
LastUpdated = state.LastUpdated?.UtcDateTime,
|
||||
DocumentDigests = state.DocumentDigests.ToList(),
|
||||
};
|
||||
|
||||
public VexConnectorState ToRecord()
|
||||
{
|
||||
var lastUpdated = LastUpdated.HasValue
|
||||
? new DateTimeOffset(DateTime.SpecifyKind(LastUpdated.Value, DateTimeKind.Utc))
|
||||
: (DateTimeOffset?)null;
|
||||
|
||||
return new VexConnectorState(
|
||||
ConnectorId,
|
||||
lastUpdated,
|
||||
DocumentDigests.ToImmutableArray());
|
||||
}
|
||||
}
|
||||
|
||||
@@ -7,6 +7,7 @@ using StellaOps.Vexer.Attestation.Transparency;
|
||||
using StellaOps.Vexer.ArtifactStores.S3.Extensions;
|
||||
using StellaOps.Vexer.Export;
|
||||
using StellaOps.Vexer.Storage.Mongo;
|
||||
using StellaOps.Vexer.Connectors.RedHat.CSAF.DependencyInjection;
|
||||
|
||||
var builder = WebApplication.CreateBuilder(args);
|
||||
var configuration = builder.Configuration;
|
||||
@@ -21,6 +22,7 @@ services.AddVexExportEngine();
|
||||
services.AddVexExportCacheServices();
|
||||
services.AddVexAttestation();
|
||||
services.Configure<VexAttestationClientOptions>(configuration.GetSection("Vexer:Attestation:Client"));
|
||||
services.AddRedHatCsafConnector();
|
||||
|
||||
var rekorSection = configuration.GetSection("Vexer:Attestation:Rekor");
|
||||
if (rekorSection.Exists())
|
||||
|
||||
@@ -2,7 +2,7 @@ If you are working on this file you need to read docs/ARCHITECTURE_VEXER.md and
|
||||
# TASKS
|
||||
| Task | Owner(s) | Depends on | Notes |
|
||||
|---|---|---|---|
|
||||
|VEXER-WEB-01-001 – Minimal API bootstrap & DI|Team Vexer WebService|VEXER-CORE-01-003, VEXER-STORAGE-01-003|TODO – Scaffold ASP.NET host, register connectors/normalizers via plugin loader, bind policy/storage/attestation services, and expose `/vexer/status`.|
|
||||
|VEXER-WEB-01-001 – Minimal API bootstrap & DI|Team Vexer WebService|VEXER-CORE-01-003, VEXER-STORAGE-01-003|**DONE (2025-10-17)** – Minimal API host composes storage/export/attestation/artifact stores, binds Mongo/attestation options, and exposes `/vexer/status` + health endpoints with regression coverage in `StatusEndpointTests`.|
|
||||
|VEXER-WEB-01-002 – Ingest & reconcile endpoints|Team Vexer WebService|VEXER-WEB-01-001|TODO – Implement `/vexer/init`, `/vexer/ingest/run`, `/vexer/ingest/resume`, `/vexer/reconcile` with token scope enforcement and structured run telemetry.|
|
||||
|VEXER-WEB-01-003 – Export & verify endpoints|Team Vexer WebService|VEXER-WEB-01-001, VEXER-EXPORT-01-001, VEXER-ATTEST-01-001|TODO – Add `/vexer/export`, `/vexer/export/{id}`, `/vexer/export/{id}/download`, `/vexer/verify`, returning artifact + attestation metadata with cache awareness.|
|
||||
|VEXER-WEB-01-004 – Resolve API & signed responses|Team Vexer WebService|VEXER-WEB-01-001, VEXER-ATTEST-01-002|TODO – Deliver `/vexer/resolve` (subject/context), return consensus + score envelopes, attach cosign/Rekor metadata, and document auth + rate guardrails.|
|
||||
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user