feat: Add CVSS receipt management endpoints and related functionality
- Introduced new API endpoints for creating, retrieving, amending, and listing CVSS receipts. - Updated IPolicyEngineClient interface to include methods for CVSS receipt operations. - Implemented PolicyEngineClient to handle CVSS receipt requests. - Enhanced Program.cs to map new CVSS receipt routes with appropriate authorization. - Added necessary models and contracts for CVSS receipt requests and responses. - Integrated Postgres document store for managing CVSS receipts and related data. - Updated database schema with new migrations for source documents and payload storage. - Refactored existing components to support new CVSS functionality.
This commit is contained in:
StellaOps Bot
16
deploy/helm/stellaops/README-mock.md
Normal file
16
deploy/helm/stellaops/README-mock.md
Normal file
@@ -0,0 +1,16 @@
|
|||||||
|
# Mock Overlay (Dev Only)
|
||||||
|
|
||||||
|
Purpose: let deployment tasks progress with placeholder digests until real releases land.
|
||||||
|
|
||||||
|
Use:
|
||||||
|
```bash
|
||||||
|
helm template mock ./deploy/helm/stellaops -f deploy/helm/stellaops/values-mock.yaml
|
||||||
|
```
|
||||||
|
|
||||||
|
Contents:
|
||||||
|
- Mock deployments for orchestrator, policy-registry, packs-registry, task-runner, VEX Lens, issuer-directory, findings-ledger, vuln-explorer-api.
|
||||||
|
- Image pins pulled from `deploy/releases/2025.09-mock-dev.yaml`.
|
||||||
|
|
||||||
|
Notes:
|
||||||
|
- Annotated with `stellaops.dev/mock: "true"` to discourage production use.
|
||||||
|
- Swap to real values once official digests publish; keep mock overlay gated behind `mock.enabled`.
|
||||||
@@ -34,7 +34,7 @@
|
|||||||
| 6 | CVSS-DSSE-190-006 | DONE (2025-11-28) | Depends on 190-005; uses Attestor primitives. | Policy Guild · Attestor Guild (`src/Policy/StellaOps.Policy.Scoring`, `src/Attestor/StellaOps.Attestor.Envelope`) | Attach DSSE attestations to score receipts: create `stella.ops/cvssReceipt@v1` predicate type, sign receipts, store envelope references. |
|
| 6 | CVSS-DSSE-190-006 | DONE (2025-11-28) | Depends on 190-005; uses Attestor primitives. | Policy Guild · Attestor Guild (`src/Policy/StellaOps.Policy.Scoring`, `src/Attestor/StellaOps.Attestor.Envelope`) | Attach DSSE attestations to score receipts: create `stella.ops/cvssReceipt@v1` predicate type, sign receipts, store envelope references. |
|
||||||
| 7 | CVSS-HISTORY-190-007 | DONE (2025-11-28) | Depends on 190-005. | Policy Guild (`src/Policy/StellaOps.Policy.Scoring/History`) | Implement receipt amendment tracking: `AmendReceipt(receiptId, field, newValue, reason, ref)` with history entry creation and re-signing. |
|
| 7 | CVSS-HISTORY-190-007 | DONE (2025-11-28) | Depends on 190-005. | Policy Guild (`src/Policy/StellaOps.Policy.Scoring/History`) | Implement receipt amendment tracking: `AmendReceipt(receiptId, field, newValue, reason, ref)` with history entry creation and re-signing. |
|
||||||
| 8 | CVSS-CONCELIER-190-008 | DONE (2025-12-06) | Depends on 190-001; Concelier AGENTS updated 2025-12-06. | Concelier Guild · Policy Guild (`src/Concelier/__Libraries/StellaOps.Concelier.Core`) | Ingest vendor-provided CVSS v4.0 vectors from advisories; parse and store as base receipts; preserve provenance. (Implemented CVSS priority ordering in Advisory → Postgres conversion so v4 vectors are primary and provenance-preserved.) |
|
| 8 | CVSS-CONCELIER-190-008 | DONE (2025-12-06) | Depends on 190-001; Concelier AGENTS updated 2025-12-06. | Concelier Guild · Policy Guild (`src/Concelier/__Libraries/StellaOps.Concelier.Core`) | Ingest vendor-provided CVSS v4.0 vectors from advisories; parse and store as base receipts; preserve provenance. (Implemented CVSS priority ordering in Advisory → Postgres conversion so v4 vectors are primary and provenance-preserved.) |
|
||||||
| 9 | CVSS-API-190-009 | BLOCKED (2025-12-06) | Depends on 190-005, 190-007; missing Policy Engine CVSS receipt endpoints to proxy. | Policy Guild (`src/Policy/StellaOps.Policy.Gateway`) | REST/gRPC APIs: `POST /cvss/receipts`, `GET /cvss/receipts/{id}`, `PUT /cvss/receipts/{id}/amend`, `GET /cvss/receipts/{id}/history`, `GET /cvss/policies`. |
|
| 9 | CVSS-API-190-009 | DONE (2025-12-06) | Depends on 190-005, 190-007; Policy Engine + Gateway CVSS endpoints shipped. | Policy Guild (`src/Policy/StellaOps.Policy.Gateway`) | REST APIs delivered: `POST /cvss/receipts`, `GET /cvss/receipts/{id}`, `PUT /cvss/receipts/{id}/amend`, `GET /cvss/receipts/{id}/history`, `GET /cvss/policies`. |
|
||||||
| 10 | CVSS-CLI-190-010 | TODO | Depends on 190-009 (API readiness). | CLI Guild (`src/Cli/StellaOps.Cli`) | CLI verbs: `stella cvss score --vuln <id>`, `stella cvss show <receiptId>`, `stella cvss history <receiptId>`, `stella cvss export <receiptId> --format json|pdf`. |
|
| 10 | CVSS-CLI-190-010 | TODO | Depends on 190-009 (API readiness). | CLI Guild (`src/Cli/StellaOps.Cli`) | CLI verbs: `stella cvss score --vuln <id>`, `stella cvss show <receiptId>`, `stella cvss history <receiptId>`, `stella cvss export <receiptId> --format json|pdf`. |
|
||||||
| 11 | CVSS-UI-190-011 | TODO | Depends on 190-009 (API readiness). | UI Guild (`src/UI/StellaOps.UI`) | UI components: Score badge with CVSS-BTE label, tabbed receipt viewer (Base/Threat/Environmental/Supplemental/Evidence/Policy/History), "Recalculate with my env" button, export options. |
|
| 11 | CVSS-UI-190-011 | TODO | Depends on 190-009 (API readiness). | UI Guild (`src/UI/StellaOps.UI`) | UI components: Score badge with CVSS-BTE label, tabbed receipt viewer (Base/Threat/Environmental/Supplemental/Evidence/Policy/History), "Recalculate with my env" button, export options. |
|
||||||
| 12 | CVSS-DOCS-190-012 | BLOCKED (2025-11-29) | Depends on 190-001 through 190-011 (API/UI/CLI blocked). | Docs Guild (`docs/modules/policy/cvss-v4.md`, `docs/09_API_CLI_REFERENCE.md`) | Document CVSS v4.0 scoring system: data model, policy format, API reference, CLI usage, UI guide, determinism guarantees. |
|
| 12 | CVSS-DOCS-190-012 | BLOCKED (2025-11-29) | Depends on 190-001 through 190-011 (API/UI/CLI blocked). | Docs Guild (`docs/modules/policy/cvss-v4.md`, `docs/09_API_CLI_REFERENCE.md`) | Document CVSS v4.0 scoring system: data model, policy format, API reference, CLI usage, UI guide, determinism guarantees. |
|
||||||
@@ -48,7 +48,7 @@
|
|||||||
| --- | --- | --- | --- | --- |
|
| --- | --- | --- | --- | --- |
|
||||||
| W1 Foundation | Policy Guild | None | DONE (2025-11-28) | Tasks 1-4: Data model, engine, tests, policy loader. |
|
| W1 Foundation | Policy Guild | None | DONE (2025-11-28) | Tasks 1-4: Data model, engine, tests, policy loader. |
|
||||||
| W2 Receipt Pipeline | Policy Guild · Attestor Guild | W1 complete | DONE (2025-11-28) | Tasks 5-7: Receipt builder, DSSE, history completed; integration tests green. |
|
| W2 Receipt Pipeline | Policy Guild · Attestor Guild | W1 complete | DONE (2025-11-28) | Tasks 5-7: Receipt builder, DSSE, history completed; integration tests green. |
|
||||||
| W3 Integration | Concelier · Policy · CLI · UI Guilds | W2 complete; AGENTS delivered 2025-12-06 | BLOCKED (2025-12-06) | CVSS-API-190-009 blocked: Policy Engine lacks CVSS receipt endpoints to proxy; CLI/UI depend on it. |
|
| W3 Integration | Concelier · Policy · CLI · UI Guilds | W2 complete; AGENTS delivered 2025-12-06 | TODO (2025-12-06) | CVSS API now available; proceed with CLI (task 10) and UI (task 11) wiring. |
|
||||||
| W4 Documentation | Docs Guild | W3 complete | BLOCKED (2025-12-06) | Task 12 blocked by API/UI/CLI delivery; resumes after W3 progresses. |
|
| W4 Documentation | Docs Guild | W3 complete | BLOCKED (2025-12-06) | Task 12 blocked by API/UI/CLI delivery; resumes after W3 progresses. |
|
||||||
|
|
||||||
## Interlocks
|
## Interlocks
|
||||||
@@ -75,11 +75,12 @@
|
|||||||
| R3 | Receipt storage grows large with evidence links. | Storage costs; query performance. | Implement evidence reference deduplication; use CAS URIs; Platform Guild. |
|
| R3 | Receipt storage grows large with evidence links. | Storage costs; query performance. | Implement evidence reference deduplication; use CAS URIs; Platform Guild. |
|
||||||
| R4 | CVSS parser/ruleset changes ungoverned (CVM9). | Score drift, audit gaps. | Version parsers/rulesets; DSSE-sign releases; log scorer version in receipts; dual-review changes. |
|
| R4 | CVSS parser/ruleset changes ungoverned (CVM9). | Score drift, audit gaps. | Version parsers/rulesets; DSSE-sign releases; log scorer version in receipts; dual-review changes. |
|
||||||
| R5 | Missing AGENTS for Policy WebService and Concelier ingestion block integration (tasks 8–11). | API/CLI/UI delivery stalled. | AGENTS delivered 2025-12-06 (tasks 15–16). Risk mitigated; monitor API contract approvals. |
|
| R5 | Missing AGENTS for Policy WebService and Concelier ingestion block integration (tasks 8–11). | API/CLI/UI delivery stalled. | AGENTS delivered 2025-12-06 (tasks 15–16). Risk mitigated; monitor API contract approvals. |
|
||||||
| R6 | Policy Engine lacks CVSS receipt endpoints; gateway proxy cannot be implemented yet. | API/CLI/UI tasks remain blocked. | Policy Guild to add receipt API surface in Policy Engine; re-run gateway wiring once available. |
|
| R6 | Policy Engine lacks CVSS receipt endpoints; gateway proxy cannot be implemented yet. | API/CLI/UI tasks remain blocked. | **Mitigated 2025-12-06:** CVSS receipt endpoints implemented in Policy Engine and Gateway; unblock CLI/UI. |
|
||||||
|
|
||||||
## Execution Log
|
## Execution Log
|
||||||
| Date (UTC) | Update | Owner |
|
| Date (UTC) | Update | Owner |
|
||||||
| --- | --- | --- |
|
| --- | --- | --- |
|
||||||
|
| 2025-12-06 | CVSS-API-190-009 DONE: added Policy Engine CVSS receipt endpoints and Gateway proxies (`/api/cvss/receipts`, history, amend, policies); W3 unblocked; risk R6 mitigated. | Implementer |
|
||||||
| 2025-12-06 | CVSS-CONCELIER-190-008 DONE: prioritized CVSS v4.0 vectors as primary in advisory→Postgres conversion; provenance preserved; enables Policy receipt ingestion. CVSS-API-190-009 set BLOCKED pending Policy Engine CVSS receipt endpoints (risk R6). | Implementer |
|
| 2025-12-06 | CVSS-CONCELIER-190-008 DONE: prioritized CVSS v4.0 vectors as primary in advisory→Postgres conversion; provenance preserved; enables Policy receipt ingestion. CVSS-API-190-009 set BLOCKED pending Policy Engine CVSS receipt endpoints (risk R6). | Implementer |
|
||||||
| 2025-12-06 | Created Policy Gateway AGENTS and refreshed Concelier AGENTS for CVSS v4 ingest (tasks 15–16 DONE); moved tasks 8–11 to TODO, set W3 to TODO, mitigated risk R5. | Project Mgmt |
|
| 2025-12-06 | Created Policy Gateway AGENTS and refreshed Concelier AGENTS for CVSS v4 ingest (tasks 15–16 DONE); moved tasks 8–11 to TODO, set W3 to TODO, mitigated risk R5. | Project Mgmt |
|
||||||
| 2025-12-06 | Added tasks 15–16 to create AGENTS for Policy WebService and Concelier; set Wave 2 to DONE; marked Waves 3–4 BLOCKED until AGENTS exist; captured risk R5. | Project Mgmt |
|
| 2025-12-06 | Added tasks 15–16 to create AGENTS for Policy WebService and Concelier; set Wave 2 to DONE; marked Waves 3–4 BLOCKED until AGENTS exist; captured risk R5. | Project Mgmt |
|
||||||
|
|||||||
@@ -39,6 +39,7 @@
|
|||||||
| 2025-12-06 | CI workflow `.gitea/workflows/mock-dev-release.yml` now packages mock manifest + downloads JSON into `mock-dev-release.tgz` for dev pipelines. | Deployment Guild |
|
| 2025-12-06 | CI workflow `.gitea/workflows/mock-dev-release.yml` now packages mock manifest + downloads JSON into `mock-dev-release.tgz` for dev pipelines. | Deployment Guild |
|
||||||
| 2025-12-06 | Mock Compose overlay (`deploy/compose/docker-compose.mock.yaml`) documented for dev-only configs using placeholder digests; production pins remain pending. | Deployment Guild |
|
| 2025-12-06 | Mock Compose overlay (`deploy/compose/docker-compose.mock.yaml`) documented for dev-only configs using placeholder digests; production pins remain pending. | Deployment Guild |
|
||||||
| 2025-12-06 | Added production guard `.gitea/workflows/release-manifest-verify.yml` to fail CI if stable/airgap manifests or downloads JSON omit required components. | Deployment Guild |
|
| 2025-12-06 | Added production guard `.gitea/workflows/release-manifest-verify.yml` to fail CI if stable/airgap manifests or downloads JSON omit required components. | Deployment Guild |
|
||||||
|
| 2025-12-06 | Added Helm mock overlays (`orchestrator/policy/packs/vex/vuln` under `deploy/helm/stellaops/templates/*-mock.yaml`) and `values-mock.yaml`; mock dev release workflow now renders `helm template` with mock values for dev packaging. | Deployment Guild |
|
||||||
| 2025-12-05 | HELM-45-003 DONE: added HPA template with per-service overrides, PDB support, Prometheus scrape annotations hook, and production defaults (prod enabled, airgap prometheus on but HPA off). | Deployment Guild |
|
| 2025-12-05 | HELM-45-003 DONE: added HPA template with per-service overrides, PDB support, Prometheus scrape annotations hook, and production defaults (prod enabled, airgap prometheus on but HPA off). | Deployment Guild |
|
||||||
| 2025-12-05 | HELM-45-002 DONE: added ingress/TLS toggles, NetworkPolicy defaults, pod security contexts, and ExternalSecret scaffold (prod enabled, airgap off); documented via values changes and templates (`core.yaml`, `networkpolicy.yaml`, `ingress.yaml`, `externalsecrets.yaml`). | Deployment Guild |
|
| 2025-12-05 | HELM-45-002 DONE: added ingress/TLS toggles, NetworkPolicy defaults, pod security contexts, and ExternalSecret scaffold (prod enabled, airgap off); documented via values changes and templates (`core.yaml`, `networkpolicy.yaml`, `ingress.yaml`, `externalsecrets.yaml`). | Deployment Guild |
|
||||||
| 2025-12-05 | HELM-45-001 DONE: added migration job scaffolding and toggle to Helm chart (`deploy/helm/stellaops/templates/migrations.yaml`, values defaults), kept digest pins, and published install guide (`deploy/helm/stellaops/INSTALL.md`). | Deployment Guild |
|
| 2025-12-05 | HELM-45-001 DONE: added migration job scaffolding and toggle to Helm chart (`deploy/helm/stellaops/templates/migrations.yaml`, values defaults), kept digest pins, and published install guide (`deploy/helm/stellaops/INSTALL.md`). | Deployment Guild |
|
||||||
|
|||||||
@@ -52,7 +52,7 @@
|
|||||||
| 9 | PG-T7.1.9 | TODO | Depends on PG-T7.1.8 | Infrastructure Guild | Remove MongoDB configuration options |
|
| 9 | PG-T7.1.9 | TODO | Depends on PG-T7.1.8 | Infrastructure Guild | Remove MongoDB configuration options |
|
||||||
| 10 | PG-T7.1.10 | TODO | Depends on PG-T7.1.9 | Infrastructure Guild | Run full build to verify no broken references |
|
| 10 | PG-T7.1.10 | TODO | Depends on PG-T7.1.9 | Infrastructure Guild | Run full build to verify no broken references |
|
||||||
| 14 | PG-T7.1.5a | DOING | Concelier Guild | Concelier: replace Mongo deps with Postgres equivalents; remove MongoDB packages; compat layer added. |
|
| 14 | PG-T7.1.5a | DOING | Concelier Guild | Concelier: replace Mongo deps with Postgres equivalents; remove MongoDB packages; compat layer added. |
|
||||||
| 15 | PG-T7.1.5b | TODO | Concelier Guild | Build Postgres document/raw storage + state repositories and wire DI. |
|
| 15 | PG-T7.1.5b | DOING | Concelier Guild | Build Postgres document/raw storage + state repositories and wire DI. |
|
||||||
| 16 | PG-T7.1.5c | TODO | Concelier Guild | Refactor connectors/exporters/tests to Postgres storage; delete Storage.Mongo code. |
|
| 16 | PG-T7.1.5c | TODO | Concelier Guild | Refactor connectors/exporters/tests to Postgres storage; delete Storage.Mongo code. |
|
||||||
| 17 | PG-T7.1.5d | TODO | Concelier Guild | Add migrations for document/state/export tables; include in air-gap kit. |
|
| 17 | PG-T7.1.5d | TODO | Concelier Guild | Add migrations for document/state/export tables; include in air-gap kit. |
|
||||||
| 18 | PG-T7.1.5e | TODO | Concelier Guild | Postgres-only Concelier build/tests green; remove Mongo artefacts and update docs. |
|
| 18 | PG-T7.1.5e | TODO | Concelier Guild | Postgres-only Concelier build/tests green; remove Mongo artefacts and update docs. |
|
||||||
@@ -122,6 +122,7 @@
|
|||||||
| 2025-12-06 | Attempted Scheduler Postgres tests; restore/build fails because `StellaOps.Concelier.Storage.Mongo` project is absent and Concelier connectors reference it. Need phased Concelier plan/shim to unblock test/build runs. | Scheduler Guild |
|
| 2025-12-06 | Attempted Scheduler Postgres tests; restore/build fails because `StellaOps.Concelier.Storage.Mongo` project is absent and Concelier connectors reference it. Need phased Concelier plan/shim to unblock test/build runs. | Scheduler Guild |
|
||||||
| 2025-12-06 | Began Concelier Mongo compatibility shim: added `FindAsync` to in-memory `IDocumentStore` in Postgres compat layer to unblock connector compile; full Mongo removal still pending. | Infrastructure Guild |
|
| 2025-12-06 | Began Concelier Mongo compatibility shim: added `FindAsync` to in-memory `IDocumentStore` in Postgres compat layer to unblock connector compile; full Mongo removal still pending. | Infrastructure Guild |
|
||||||
| 2025-12-06 | Added lightweight `StellaOps.Concelier.Storage.Mongo` in-memory stub (advisory/dto/document/state/export stores) to unblock Concelier connector build while Postgres rewiring continues; no Mongo driver/runtime. | Infrastructure Guild |
|
| 2025-12-06 | Added lightweight `StellaOps.Concelier.Storage.Mongo` in-memory stub (advisory/dto/document/state/export stores) to unblock Concelier connector build while Postgres rewiring continues; no Mongo driver/runtime. | Infrastructure Guild |
|
||||||
|
| 2025-12-06 | PG-T7.1.5b set to DOING; began wiring Postgres document store (DI registration, repository find) to replace Mongo bindings. | Concelier Guild |
|
||||||
|
|
||||||
## Decisions & Risks
|
## Decisions & Risks
|
||||||
- Cleanup is strictly after all phases complete; do not start T7 tasks until module cutovers are DONE.
|
- Cleanup is strictly after all phases complete; do not start T7 tasks until module cutovers are DONE.
|
||||||
|
|||||||
@@ -3,7 +3,7 @@
|
|||||||
| # | Task ID | Status | Owner | Notes |
|
| # | Task ID | Status | Owner | Notes |
|
||||||
|---|---|---|---|---|
|
|---|---|---|---|---|
|
||||||
| 1 | PG-T7.1.5a | DOING | Concelier Guild | Replace Mongo storage dependencies with Postgres equivalents; remove MongoDB.Driver/Bson packages from Concelier projects. |
|
| 1 | PG-T7.1.5a | DOING | Concelier Guild | Replace Mongo storage dependencies with Postgres equivalents; remove MongoDB.Driver/Bson packages from Concelier projects. |
|
||||||
| 2 | PG-T7.1.5b | TODO | Concelier Guild | Implement Postgres document/raw storage (bytea/LargeObject) + state repos to satisfy connector fetch/store paths. |
|
| 2 | PG-T7.1.5b | DOING | Concelier Guild | Implement Postgres document/raw storage (bytea/LargeObject) + state repos to satisfy connector fetch/store paths. |
|
||||||
| 3 | PG-T7.1.5c | TODO | Concelier Guild | Refactor all connectors/exporters/tests to use Postgres storage namespaces; delete Storage.Mongo code/tests. |
|
| 3 | PG-T7.1.5c | TODO | Concelier Guild | Refactor all connectors/exporters/tests to use Postgres storage namespaces; delete Storage.Mongo code/tests. |
|
||||||
| 4 | PG-T7.1.5d | TODO | Concelier Guild | Add migrations for documents/state/export tables; wire into Concelier Postgres storage DI. |
|
| 4 | PG-T7.1.5d | TODO | Concelier Guild | Add migrations for documents/state/export tables; wire into Concelier Postgres storage DI. |
|
||||||
| 5 | PG-T7.1.5e | TODO | Concelier Guild | End-to-end Concelier build/test on Postgres-only stack; update sprint log and remove Mongo artifacts from repo history references. |
|
| 5 | PG-T7.1.5e | TODO | Concelier Guild | End-to-end Concelier build/test on Postgres-only stack; update sprint log and remove Mongo artifacts from repo history references. |
|
||||||
|
|||||||
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
@@ -245,7 +245,7 @@ public sealed class CertBundConnector : IFeedConnector
|
|||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
|
|
||||||
if (!document.GridFsId.HasValue)
|
if (!document.PayloadId.HasValue)
|
||||||
{
|
{
|
||||||
await _documentStore.UpdateStatusAsync(document.Id, DocumentStatuses.Failed, cancellationToken).ConfigureAwait(false);
|
await _documentStore.UpdateStatusAsync(document.Id, DocumentStatuses.Failed, cancellationToken).ConfigureAwait(false);
|
||||||
remainingDocuments.Remove(documentId);
|
remainingDocuments.Remove(documentId);
|
||||||
@@ -258,7 +258,7 @@ public sealed class CertBundConnector : IFeedConnector
|
|||||||
byte[] payload;
|
byte[] payload;
|
||||||
try
|
try
|
||||||
{
|
{
|
||||||
payload = await _rawDocumentStorage.DownloadAsync(document.GridFsId.Value, cancellationToken).ConfigureAwait(false);
|
payload = await _rawDocumentStorage.DownloadAsync(document.PayloadId.Value, cancellationToken).ConfigureAwait(false);
|
||||||
}
|
}
|
||||||
catch (Exception ex)
|
catch (Exception ex)
|
||||||
{
|
{
|
||||||
|
|||||||
File diff suppressed because it is too large
Load Diff
@@ -1,337 +1,337 @@
|
|||||||
using System;
|
using System;
|
||||||
using System.Collections.Generic;
|
using System.Collections.Generic;
|
||||||
using System.Linq;
|
using System.Linq;
|
||||||
using System.Text.Json;
|
using System.Text.Json;
|
||||||
using Microsoft.Extensions.Logging;
|
using Microsoft.Extensions.Logging;
|
||||||
using Microsoft.Extensions.Options;
|
using Microsoft.Extensions.Options;
|
||||||
using MongoDB.Bson;
|
using MongoDB.Bson;
|
||||||
using StellaOps.Concelier.Connector.CertFr.Configuration;
|
using StellaOps.Concelier.Connector.CertFr.Configuration;
|
||||||
using StellaOps.Concelier.Connector.CertFr.Internal;
|
using StellaOps.Concelier.Connector.CertFr.Internal;
|
||||||
using StellaOps.Concelier.Connector.Common;
|
using StellaOps.Concelier.Connector.Common;
|
||||||
using StellaOps.Concelier.Connector.Common.Fetch;
|
using StellaOps.Concelier.Connector.Common.Fetch;
|
||||||
using StellaOps.Concelier.Storage.Mongo;
|
using StellaOps.Concelier.Storage.Mongo;
|
||||||
using StellaOps.Concelier.Storage.Mongo.Advisories;
|
using StellaOps.Concelier.Storage.Mongo.Advisories;
|
||||||
using StellaOps.Concelier.Storage.Mongo.Documents;
|
using StellaOps.Concelier.Storage.Mongo.Documents;
|
||||||
using StellaOps.Concelier.Storage.Mongo.Dtos;
|
using StellaOps.Concelier.Storage.Mongo.Dtos;
|
||||||
using StellaOps.Plugin;
|
using StellaOps.Plugin;
|
||||||
|
|
||||||
namespace StellaOps.Concelier.Connector.CertFr;
|
namespace StellaOps.Concelier.Connector.CertFr;
|
||||||
|
|
||||||
public sealed class CertFrConnector : IFeedConnector
|
public sealed class CertFrConnector : IFeedConnector
|
||||||
{
|
{
|
||||||
private static readonly JsonSerializerOptions SerializerOptions = new()
|
private static readonly JsonSerializerOptions SerializerOptions = new()
|
||||||
{
|
{
|
||||||
PropertyNamingPolicy = JsonNamingPolicy.CamelCase,
|
PropertyNamingPolicy = JsonNamingPolicy.CamelCase,
|
||||||
DefaultIgnoreCondition = System.Text.Json.Serialization.JsonIgnoreCondition.WhenWritingNull,
|
DefaultIgnoreCondition = System.Text.Json.Serialization.JsonIgnoreCondition.WhenWritingNull,
|
||||||
};
|
};
|
||||||
|
|
||||||
private readonly CertFrFeedClient _feedClient;
|
private readonly CertFrFeedClient _feedClient;
|
||||||
private readonly SourceFetchService _fetchService;
|
private readonly SourceFetchService _fetchService;
|
||||||
private readonly RawDocumentStorage _rawDocumentStorage;
|
private readonly RawDocumentStorage _rawDocumentStorage;
|
||||||
private readonly IDocumentStore _documentStore;
|
private readonly IDocumentStore _documentStore;
|
||||||
private readonly IDtoStore _dtoStore;
|
private readonly IDtoStore _dtoStore;
|
||||||
private readonly IAdvisoryStore _advisoryStore;
|
private readonly IAdvisoryStore _advisoryStore;
|
||||||
private readonly ISourceStateRepository _stateRepository;
|
private readonly ISourceStateRepository _stateRepository;
|
||||||
private readonly CertFrOptions _options;
|
private readonly CertFrOptions _options;
|
||||||
private readonly TimeProvider _timeProvider;
|
private readonly TimeProvider _timeProvider;
|
||||||
private readonly ILogger<CertFrConnector> _logger;
|
private readonly ILogger<CertFrConnector> _logger;
|
||||||
|
|
||||||
public CertFrConnector(
|
public CertFrConnector(
|
||||||
CertFrFeedClient feedClient,
|
CertFrFeedClient feedClient,
|
||||||
SourceFetchService fetchService,
|
SourceFetchService fetchService,
|
||||||
RawDocumentStorage rawDocumentStorage,
|
RawDocumentStorage rawDocumentStorage,
|
||||||
IDocumentStore documentStore,
|
IDocumentStore documentStore,
|
||||||
IDtoStore dtoStore,
|
IDtoStore dtoStore,
|
||||||
IAdvisoryStore advisoryStore,
|
IAdvisoryStore advisoryStore,
|
||||||
ISourceStateRepository stateRepository,
|
ISourceStateRepository stateRepository,
|
||||||
IOptions<CertFrOptions> options,
|
IOptions<CertFrOptions> options,
|
||||||
TimeProvider? timeProvider,
|
TimeProvider? timeProvider,
|
||||||
ILogger<CertFrConnector> logger)
|
ILogger<CertFrConnector> logger)
|
||||||
{
|
{
|
||||||
_feedClient = feedClient ?? throw new ArgumentNullException(nameof(feedClient));
|
_feedClient = feedClient ?? throw new ArgumentNullException(nameof(feedClient));
|
||||||
_fetchService = fetchService ?? throw new ArgumentNullException(nameof(fetchService));
|
_fetchService = fetchService ?? throw new ArgumentNullException(nameof(fetchService));
|
||||||
_rawDocumentStorage = rawDocumentStorage ?? throw new ArgumentNullException(nameof(rawDocumentStorage));
|
_rawDocumentStorage = rawDocumentStorage ?? throw new ArgumentNullException(nameof(rawDocumentStorage));
|
||||||
_documentStore = documentStore ?? throw new ArgumentNullException(nameof(documentStore));
|
_documentStore = documentStore ?? throw new ArgumentNullException(nameof(documentStore));
|
||||||
_dtoStore = dtoStore ?? throw new ArgumentNullException(nameof(dtoStore));
|
_dtoStore = dtoStore ?? throw new ArgumentNullException(nameof(dtoStore));
|
||||||
_advisoryStore = advisoryStore ?? throw new ArgumentNullException(nameof(advisoryStore));
|
_advisoryStore = advisoryStore ?? throw new ArgumentNullException(nameof(advisoryStore));
|
||||||
_stateRepository = stateRepository ?? throw new ArgumentNullException(nameof(stateRepository));
|
_stateRepository = stateRepository ?? throw new ArgumentNullException(nameof(stateRepository));
|
||||||
_options = (options ?? throw new ArgumentNullException(nameof(options))).Value ?? throw new ArgumentNullException(nameof(options));
|
_options = (options ?? throw new ArgumentNullException(nameof(options))).Value ?? throw new ArgumentNullException(nameof(options));
|
||||||
_options.Validate();
|
_options.Validate();
|
||||||
_timeProvider = timeProvider ?? TimeProvider.System;
|
_timeProvider = timeProvider ?? TimeProvider.System;
|
||||||
_logger = logger ?? throw new ArgumentNullException(nameof(logger));
|
_logger = logger ?? throw new ArgumentNullException(nameof(logger));
|
||||||
}
|
}
|
||||||
|
|
||||||
public string SourceName => CertFrConnectorPlugin.SourceName;
|
public string SourceName => CertFrConnectorPlugin.SourceName;
|
||||||
|
|
||||||
public async Task FetchAsync(IServiceProvider services, CancellationToken cancellationToken)
|
public async Task FetchAsync(IServiceProvider services, CancellationToken cancellationToken)
|
||||||
{
|
{
|
||||||
var cursor = await GetCursorAsync(cancellationToken).ConfigureAwait(false);
|
var cursor = await GetCursorAsync(cancellationToken).ConfigureAwait(false);
|
||||||
var now = _timeProvider.GetUtcNow();
|
var now = _timeProvider.GetUtcNow();
|
||||||
var windowEnd = now;
|
var windowEnd = now;
|
||||||
var lastPublished = cursor.LastPublished ?? now - _options.InitialBackfill;
|
var lastPublished = cursor.LastPublished ?? now - _options.InitialBackfill;
|
||||||
var windowStart = lastPublished - _options.WindowOverlap;
|
var windowStart = lastPublished - _options.WindowOverlap;
|
||||||
var minStart = now - _options.InitialBackfill;
|
var minStart = now - _options.InitialBackfill;
|
||||||
if (windowStart < minStart)
|
if (windowStart < minStart)
|
||||||
{
|
{
|
||||||
windowStart = minStart;
|
windowStart = minStart;
|
||||||
}
|
}
|
||||||
|
|
||||||
IReadOnlyList<CertFrFeedItem> items;
|
IReadOnlyList<CertFrFeedItem> items;
|
||||||
try
|
try
|
||||||
{
|
{
|
||||||
items = await _feedClient.LoadAsync(windowStart, windowEnd, cancellationToken).ConfigureAwait(false);
|
items = await _feedClient.LoadAsync(windowStart, windowEnd, cancellationToken).ConfigureAwait(false);
|
||||||
}
|
}
|
||||||
catch (Exception ex)
|
catch (Exception ex)
|
||||||
{
|
{
|
||||||
_logger.LogError(ex, "Cert-FR feed load failed {Start:o}-{End:o}", windowStart, windowEnd);
|
_logger.LogError(ex, "Cert-FR feed load failed {Start:o}-{End:o}", windowStart, windowEnd);
|
||||||
await _stateRepository.MarkFailureAsync(SourceName, now, TimeSpan.FromMinutes(10), ex.Message, cancellationToken).ConfigureAwait(false);
|
await _stateRepository.MarkFailureAsync(SourceName, now, TimeSpan.FromMinutes(10), ex.Message, cancellationToken).ConfigureAwait(false);
|
||||||
throw;
|
throw;
|
||||||
}
|
}
|
||||||
|
|
||||||
if (items.Count == 0)
|
if (items.Count == 0)
|
||||||
{
|
{
|
||||||
await UpdateCursorAsync(cursor.WithLastPublished(windowEnd), cancellationToken).ConfigureAwait(false);
|
await UpdateCursorAsync(cursor.WithLastPublished(windowEnd), cancellationToken).ConfigureAwait(false);
|
||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
|
|
||||||
var pendingDocuments = cursor.PendingDocuments.ToList();
|
var pendingDocuments = cursor.PendingDocuments.ToList();
|
||||||
var pendingMappings = cursor.PendingMappings.ToList();
|
var pendingMappings = cursor.PendingMappings.ToList();
|
||||||
var maxPublished = cursor.LastPublished ?? DateTimeOffset.MinValue;
|
var maxPublished = cursor.LastPublished ?? DateTimeOffset.MinValue;
|
||||||
|
|
||||||
foreach (var item in items)
|
foreach (var item in items)
|
||||||
{
|
{
|
||||||
cancellationToken.ThrowIfCancellationRequested();
|
cancellationToken.ThrowIfCancellationRequested();
|
||||||
|
|
||||||
try
|
try
|
||||||
{
|
{
|
||||||
var existing = await _documentStore.FindBySourceAndUriAsync(SourceName, item.DetailUri.ToString(), cancellationToken).ConfigureAwait(false);
|
var existing = await _documentStore.FindBySourceAndUriAsync(SourceName, item.DetailUri.ToString(), cancellationToken).ConfigureAwait(false);
|
||||||
var request = new SourceFetchRequest(CertFrOptions.HttpClientName, SourceName, item.DetailUri)
|
var request = new SourceFetchRequest(CertFrOptions.HttpClientName, SourceName, item.DetailUri)
|
||||||
{
|
{
|
||||||
Metadata = CertFrDocumentMetadata.CreateMetadata(item),
|
Metadata = CertFrDocumentMetadata.CreateMetadata(item),
|
||||||
ETag = existing?.Etag,
|
ETag = existing?.Etag,
|
||||||
LastModified = existing?.LastModified,
|
LastModified = existing?.LastModified,
|
||||||
AcceptHeaders = new[] { "text/html", "application/xhtml+xml", "text/plain;q=0.5" },
|
AcceptHeaders = new[] { "text/html", "application/xhtml+xml", "text/plain;q=0.5" },
|
||||||
};
|
};
|
||||||
|
|
||||||
var result = await _fetchService.FetchAsync(request, cancellationToken).ConfigureAwait(false);
|
var result = await _fetchService.FetchAsync(request, cancellationToken).ConfigureAwait(false);
|
||||||
if (result.IsNotModified || !result.IsSuccess || result.Document is null)
|
if (result.IsNotModified || !result.IsSuccess || result.Document is null)
|
||||||
{
|
{
|
||||||
if (item.Published > maxPublished)
|
if (item.Published > maxPublished)
|
||||||
{
|
{
|
||||||
maxPublished = item.Published;
|
maxPublished = item.Published;
|
||||||
}
|
}
|
||||||
|
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
|
|
||||||
if (existing is not null
|
if (existing is not null
|
||||||
&& string.Equals(existing.Sha256, result.Document.Sha256, StringComparison.OrdinalIgnoreCase)
|
&& string.Equals(existing.Sha256, result.Document.Sha256, StringComparison.OrdinalIgnoreCase)
|
||||||
&& string.Equals(existing.Status, DocumentStatuses.Mapped, StringComparison.Ordinal))
|
&& string.Equals(existing.Status, DocumentStatuses.Mapped, StringComparison.Ordinal))
|
||||||
{
|
{
|
||||||
await _documentStore.UpdateStatusAsync(result.Document.Id, existing.Status, cancellationToken).ConfigureAwait(false);
|
await _documentStore.UpdateStatusAsync(result.Document.Id, existing.Status, cancellationToken).ConfigureAwait(false);
|
||||||
if (item.Published > maxPublished)
|
if (item.Published > maxPublished)
|
||||||
{
|
{
|
||||||
maxPublished = item.Published;
|
maxPublished = item.Published;
|
||||||
}
|
}
|
||||||
|
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
|
|
||||||
if (!pendingDocuments.Contains(result.Document.Id))
|
if (!pendingDocuments.Contains(result.Document.Id))
|
||||||
{
|
{
|
||||||
pendingDocuments.Add(result.Document.Id);
|
pendingDocuments.Add(result.Document.Id);
|
||||||
}
|
}
|
||||||
|
|
||||||
if (item.Published > maxPublished)
|
if (item.Published > maxPublished)
|
||||||
{
|
{
|
||||||
maxPublished = item.Published;
|
maxPublished = item.Published;
|
||||||
}
|
}
|
||||||
|
|
||||||
if (_options.RequestDelay > TimeSpan.Zero)
|
if (_options.RequestDelay > TimeSpan.Zero)
|
||||||
{
|
{
|
||||||
await Task.Delay(_options.RequestDelay, cancellationToken).ConfigureAwait(false);
|
await Task.Delay(_options.RequestDelay, cancellationToken).ConfigureAwait(false);
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
catch (Exception ex)
|
catch (Exception ex)
|
||||||
{
|
{
|
||||||
_logger.LogError(ex, "Cert-FR fetch failed for {Uri}", item.DetailUri);
|
_logger.LogError(ex, "Cert-FR fetch failed for {Uri}", item.DetailUri);
|
||||||
await _stateRepository.MarkFailureAsync(SourceName, _timeProvider.GetUtcNow(), TimeSpan.FromMinutes(5), ex.Message, cancellationToken).ConfigureAwait(false);
|
await _stateRepository.MarkFailureAsync(SourceName, _timeProvider.GetUtcNow(), TimeSpan.FromMinutes(5), ex.Message, cancellationToken).ConfigureAwait(false);
|
||||||
throw;
|
throw;
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
if (maxPublished == DateTimeOffset.MinValue)
|
if (maxPublished == DateTimeOffset.MinValue)
|
||||||
{
|
{
|
||||||
maxPublished = cursor.LastPublished ?? windowEnd;
|
maxPublished = cursor.LastPublished ?? windowEnd;
|
||||||
}
|
}
|
||||||
|
|
||||||
var updatedCursor = cursor
|
var updatedCursor = cursor
|
||||||
.WithPendingDocuments(pendingDocuments)
|
.WithPendingDocuments(pendingDocuments)
|
||||||
.WithPendingMappings(pendingMappings)
|
.WithPendingMappings(pendingMappings)
|
||||||
.WithLastPublished(maxPublished);
|
.WithLastPublished(maxPublished);
|
||||||
|
|
||||||
await UpdateCursorAsync(updatedCursor, cancellationToken).ConfigureAwait(false);
|
await UpdateCursorAsync(updatedCursor, cancellationToken).ConfigureAwait(false);
|
||||||
}
|
}
|
||||||
|
|
||||||
public async Task ParseAsync(IServiceProvider services, CancellationToken cancellationToken)
|
public async Task ParseAsync(IServiceProvider services, CancellationToken cancellationToken)
|
||||||
{
|
{
|
||||||
var cursor = await GetCursorAsync(cancellationToken).ConfigureAwait(false);
|
var cursor = await GetCursorAsync(cancellationToken).ConfigureAwait(false);
|
||||||
if (cursor.PendingDocuments.Count == 0)
|
if (cursor.PendingDocuments.Count == 0)
|
||||||
{
|
{
|
||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
|
|
||||||
var pendingDocuments = cursor.PendingDocuments.ToList();
|
var pendingDocuments = cursor.PendingDocuments.ToList();
|
||||||
var pendingMappings = cursor.PendingMappings.ToList();
|
var pendingMappings = cursor.PendingMappings.ToList();
|
||||||
|
|
||||||
foreach (var documentId in cursor.PendingDocuments)
|
foreach (var documentId in cursor.PendingDocuments)
|
||||||
{
|
{
|
||||||
cancellationToken.ThrowIfCancellationRequested();
|
cancellationToken.ThrowIfCancellationRequested();
|
||||||
|
|
||||||
var document = await _documentStore.FindAsync(documentId, cancellationToken).ConfigureAwait(false);
|
var document = await _documentStore.FindAsync(documentId, cancellationToken).ConfigureAwait(false);
|
||||||
if (document is null)
|
if (document is null)
|
||||||
{
|
{
|
||||||
pendingDocuments.Remove(documentId);
|
pendingDocuments.Remove(documentId);
|
||||||
pendingMappings.Remove(documentId);
|
pendingMappings.Remove(documentId);
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
|
|
||||||
if (!document.GridFsId.HasValue)
|
if (!document.PayloadId.HasValue)
|
||||||
{
|
{
|
||||||
_logger.LogWarning("Cert-FR document {DocumentId} missing GridFS payload", document.Id);
|
_logger.LogWarning("Cert-FR document {DocumentId} missing GridFS payload", document.Id);
|
||||||
await _documentStore.UpdateStatusAsync(document.Id, DocumentStatuses.Failed, cancellationToken).ConfigureAwait(false);
|
await _documentStore.UpdateStatusAsync(document.Id, DocumentStatuses.Failed, cancellationToken).ConfigureAwait(false);
|
||||||
pendingDocuments.Remove(documentId);
|
pendingDocuments.Remove(documentId);
|
||||||
pendingMappings.Remove(documentId);
|
pendingMappings.Remove(documentId);
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
|
|
||||||
CertFrDocumentMetadata metadata;
|
CertFrDocumentMetadata metadata;
|
||||||
try
|
try
|
||||||
{
|
{
|
||||||
metadata = CertFrDocumentMetadata.FromDocument(document);
|
metadata = CertFrDocumentMetadata.FromDocument(document);
|
||||||
}
|
}
|
||||||
catch (Exception ex)
|
catch (Exception ex)
|
||||||
{
|
{
|
||||||
_logger.LogError(ex, "Cert-FR metadata parse failed for document {DocumentId}", document.Id);
|
_logger.LogError(ex, "Cert-FR metadata parse failed for document {DocumentId}", document.Id);
|
||||||
await _documentStore.UpdateStatusAsync(document.Id, DocumentStatuses.Failed, cancellationToken).ConfigureAwait(false);
|
await _documentStore.UpdateStatusAsync(document.Id, DocumentStatuses.Failed, cancellationToken).ConfigureAwait(false);
|
||||||
pendingDocuments.Remove(documentId);
|
pendingDocuments.Remove(documentId);
|
||||||
pendingMappings.Remove(documentId);
|
pendingMappings.Remove(documentId);
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
|
|
||||||
CertFrDto dto;
|
CertFrDto dto;
|
||||||
try
|
try
|
||||||
{
|
{
|
||||||
var content = await _rawDocumentStorage.DownloadAsync(document.GridFsId.Value, cancellationToken).ConfigureAwait(false);
|
var content = await _rawDocumentStorage.DownloadAsync(document.PayloadId.Value, cancellationToken).ConfigureAwait(false);
|
||||||
var html = System.Text.Encoding.UTF8.GetString(content);
|
var html = System.Text.Encoding.UTF8.GetString(content);
|
||||||
dto = CertFrParser.Parse(html, metadata);
|
dto = CertFrParser.Parse(html, metadata);
|
||||||
}
|
}
|
||||||
catch (Exception ex)
|
catch (Exception ex)
|
||||||
{
|
{
|
||||||
_logger.LogError(ex, "Cert-FR parse failed for advisory {AdvisoryId} ({Uri})", metadata.AdvisoryId, document.Uri);
|
_logger.LogError(ex, "Cert-FR parse failed for advisory {AdvisoryId} ({Uri})", metadata.AdvisoryId, document.Uri);
|
||||||
await _documentStore.UpdateStatusAsync(document.Id, DocumentStatuses.Failed, cancellationToken).ConfigureAwait(false);
|
await _documentStore.UpdateStatusAsync(document.Id, DocumentStatuses.Failed, cancellationToken).ConfigureAwait(false);
|
||||||
pendingDocuments.Remove(documentId);
|
pendingDocuments.Remove(documentId);
|
||||||
pendingMappings.Remove(documentId);
|
pendingMappings.Remove(documentId);
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
|
|
||||||
var json = JsonSerializer.Serialize(dto, SerializerOptions);
|
var json = JsonSerializer.Serialize(dto, SerializerOptions);
|
||||||
var payload = BsonDocument.Parse(json);
|
var payload = BsonDocument.Parse(json);
|
||||||
var validatedAt = _timeProvider.GetUtcNow();
|
var validatedAt = _timeProvider.GetUtcNow();
|
||||||
|
|
||||||
var existingDto = await _dtoStore.FindByDocumentIdAsync(document.Id, cancellationToken).ConfigureAwait(false);
|
var existingDto = await _dtoStore.FindByDocumentIdAsync(document.Id, cancellationToken).ConfigureAwait(false);
|
||||||
var dtoRecord = existingDto is null
|
var dtoRecord = existingDto is null
|
||||||
? new DtoRecord(Guid.NewGuid(), document.Id, SourceName, "certfr.detail.v1", payload, validatedAt)
|
? new DtoRecord(Guid.NewGuid(), document.Id, SourceName, "certfr.detail.v1", payload, validatedAt)
|
||||||
: existingDto with
|
: existingDto with
|
||||||
{
|
{
|
||||||
Payload = payload,
|
Payload = payload,
|
||||||
SchemaVersion = "certfr.detail.v1",
|
SchemaVersion = "certfr.detail.v1",
|
||||||
ValidatedAt = validatedAt,
|
ValidatedAt = validatedAt,
|
||||||
};
|
};
|
||||||
|
|
||||||
await _dtoStore.UpsertAsync(dtoRecord, cancellationToken).ConfigureAwait(false);
|
await _dtoStore.UpsertAsync(dtoRecord, cancellationToken).ConfigureAwait(false);
|
||||||
await _documentStore.UpdateStatusAsync(document.Id, DocumentStatuses.PendingMap, cancellationToken).ConfigureAwait(false);
|
await _documentStore.UpdateStatusAsync(document.Id, DocumentStatuses.PendingMap, cancellationToken).ConfigureAwait(false);
|
||||||
|
|
||||||
pendingDocuments.Remove(documentId);
|
pendingDocuments.Remove(documentId);
|
||||||
if (!pendingMappings.Contains(documentId))
|
if (!pendingMappings.Contains(documentId))
|
||||||
{
|
{
|
||||||
pendingMappings.Add(documentId);
|
pendingMappings.Add(documentId);
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
var updatedCursor = cursor
|
var updatedCursor = cursor
|
||||||
.WithPendingDocuments(pendingDocuments)
|
.WithPendingDocuments(pendingDocuments)
|
||||||
.WithPendingMappings(pendingMappings);
|
.WithPendingMappings(pendingMappings);
|
||||||
|
|
||||||
await UpdateCursorAsync(updatedCursor, cancellationToken).ConfigureAwait(false);
|
await UpdateCursorAsync(updatedCursor, cancellationToken).ConfigureAwait(false);
|
||||||
}
|
}
|
||||||
|
|
||||||
public async Task MapAsync(IServiceProvider services, CancellationToken cancellationToken)
|
public async Task MapAsync(IServiceProvider services, CancellationToken cancellationToken)
|
||||||
{
|
{
|
||||||
var cursor = await GetCursorAsync(cancellationToken).ConfigureAwait(false);
|
var cursor = await GetCursorAsync(cancellationToken).ConfigureAwait(false);
|
||||||
if (cursor.PendingMappings.Count == 0)
|
if (cursor.PendingMappings.Count == 0)
|
||||||
{
|
{
|
||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
|
|
||||||
var pendingMappings = cursor.PendingMappings.ToList();
|
var pendingMappings = cursor.PendingMappings.ToList();
|
||||||
|
|
||||||
foreach (var documentId in cursor.PendingMappings)
|
foreach (var documentId in cursor.PendingMappings)
|
||||||
{
|
{
|
||||||
cancellationToken.ThrowIfCancellationRequested();
|
cancellationToken.ThrowIfCancellationRequested();
|
||||||
|
|
||||||
var dtoRecord = await _dtoStore.FindByDocumentIdAsync(documentId, cancellationToken).ConfigureAwait(false);
|
var dtoRecord = await _dtoStore.FindByDocumentIdAsync(documentId, cancellationToken).ConfigureAwait(false);
|
||||||
var document = await _documentStore.FindAsync(documentId, cancellationToken).ConfigureAwait(false);
|
var document = await _documentStore.FindAsync(documentId, cancellationToken).ConfigureAwait(false);
|
||||||
|
|
||||||
if (dtoRecord is null || document is null)
|
if (dtoRecord is null || document is null)
|
||||||
{
|
{
|
||||||
pendingMappings.Remove(documentId);
|
pendingMappings.Remove(documentId);
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
|
|
||||||
CertFrDto? dto;
|
CertFrDto? dto;
|
||||||
try
|
try
|
||||||
{
|
{
|
||||||
var json = dtoRecord.Payload.ToJson();
|
var json = dtoRecord.Payload.ToJson();
|
||||||
dto = JsonSerializer.Deserialize<CertFrDto>(json, SerializerOptions);
|
dto = JsonSerializer.Deserialize<CertFrDto>(json, SerializerOptions);
|
||||||
}
|
}
|
||||||
catch (Exception ex)
|
catch (Exception ex)
|
||||||
{
|
{
|
||||||
_logger.LogError(ex, "Cert-FR DTO deserialization failed for document {DocumentId}", documentId);
|
_logger.LogError(ex, "Cert-FR DTO deserialization failed for document {DocumentId}", documentId);
|
||||||
await _documentStore.UpdateStatusAsync(document.Id, DocumentStatuses.Failed, cancellationToken).ConfigureAwait(false);
|
await _documentStore.UpdateStatusAsync(document.Id, DocumentStatuses.Failed, cancellationToken).ConfigureAwait(false);
|
||||||
pendingMappings.Remove(documentId);
|
pendingMappings.Remove(documentId);
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
|
|
||||||
if (dto is null)
|
if (dto is null)
|
||||||
{
|
{
|
||||||
_logger.LogWarning("Cert-FR DTO payload deserialized as null for document {DocumentId}", documentId);
|
_logger.LogWarning("Cert-FR DTO payload deserialized as null for document {DocumentId}", documentId);
|
||||||
await _documentStore.UpdateStatusAsync(document.Id, DocumentStatuses.Failed, cancellationToken).ConfigureAwait(false);
|
await _documentStore.UpdateStatusAsync(document.Id, DocumentStatuses.Failed, cancellationToken).ConfigureAwait(false);
|
||||||
pendingMappings.Remove(documentId);
|
pendingMappings.Remove(documentId);
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
|
|
||||||
var mappedAt = _timeProvider.GetUtcNow();
|
var mappedAt = _timeProvider.GetUtcNow();
|
||||||
var advisory = CertFrMapper.Map(dto, SourceName, mappedAt);
|
var advisory = CertFrMapper.Map(dto, SourceName, mappedAt);
|
||||||
await _advisoryStore.UpsertAsync(advisory, cancellationToken).ConfigureAwait(false);
|
await _advisoryStore.UpsertAsync(advisory, cancellationToken).ConfigureAwait(false);
|
||||||
await _documentStore.UpdateStatusAsync(document.Id, DocumentStatuses.Mapped, cancellationToken).ConfigureAwait(false);
|
await _documentStore.UpdateStatusAsync(document.Id, DocumentStatuses.Mapped, cancellationToken).ConfigureAwait(false);
|
||||||
|
|
||||||
pendingMappings.Remove(documentId);
|
pendingMappings.Remove(documentId);
|
||||||
}
|
}
|
||||||
|
|
||||||
var updatedCursor = cursor.WithPendingMappings(pendingMappings);
|
var updatedCursor = cursor.WithPendingMappings(pendingMappings);
|
||||||
await UpdateCursorAsync(updatedCursor, cancellationToken).ConfigureAwait(false);
|
await UpdateCursorAsync(updatedCursor, cancellationToken).ConfigureAwait(false);
|
||||||
}
|
}
|
||||||
|
|
||||||
private async Task<CertFrCursor> GetCursorAsync(CancellationToken cancellationToken)
|
private async Task<CertFrCursor> GetCursorAsync(CancellationToken cancellationToken)
|
||||||
{
|
{
|
||||||
var record = await _stateRepository.TryGetAsync(SourceName, cancellationToken).ConfigureAwait(false);
|
var record = await _stateRepository.TryGetAsync(SourceName, cancellationToken).ConfigureAwait(false);
|
||||||
return CertFrCursor.FromBson(record?.Cursor);
|
return CertFrCursor.FromBson(record?.Cursor);
|
||||||
}
|
}
|
||||||
|
|
||||||
private async Task UpdateCursorAsync(CertFrCursor cursor, CancellationToken cancellationToken)
|
private async Task UpdateCursorAsync(CertFrCursor cursor, CancellationToken cancellationToken)
|
||||||
{
|
{
|
||||||
var completedAt = _timeProvider.GetUtcNow();
|
var completedAt = _timeProvider.GetUtcNow();
|
||||||
await _stateRepository.UpdateCursorAsync(SourceName, cursor.ToBsonDocument(), completedAt, cancellationToken).ConfigureAwait(false);
|
await _stateRepository.UpdateCursorAsync(SourceName, cursor.ToBsonDocument(), completedAt, cancellationToken).ConfigureAwait(false);
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -1,370 +1,370 @@
|
|||||||
using System;
|
using System;
|
||||||
using System.Collections.Generic;
|
using System.Collections.Generic;
|
||||||
using System.Linq;
|
using System.Linq;
|
||||||
using System.Text.Json;
|
using System.Text.Json;
|
||||||
using System.Threading;
|
using System.Threading;
|
||||||
using System.Threading.Tasks;
|
using System.Threading.Tasks;
|
||||||
using Microsoft.Extensions.Logging;
|
using Microsoft.Extensions.Logging;
|
||||||
using Microsoft.Extensions.Options;
|
using Microsoft.Extensions.Options;
|
||||||
using MongoDB.Bson;
|
using MongoDB.Bson;
|
||||||
using StellaOps.Concelier.Models;
|
using StellaOps.Concelier.Models;
|
||||||
using StellaOps.Concelier.Connector.CertIn.Configuration;
|
using StellaOps.Concelier.Connector.CertIn.Configuration;
|
||||||
using StellaOps.Concelier.Connector.CertIn.Internal;
|
using StellaOps.Concelier.Connector.CertIn.Internal;
|
||||||
using StellaOps.Concelier.Connector.Common;
|
using StellaOps.Concelier.Connector.Common;
|
||||||
using StellaOps.Concelier.Connector.Common.Fetch;
|
using StellaOps.Concelier.Connector.Common.Fetch;
|
||||||
using StellaOps.Concelier.Storage.Mongo;
|
using StellaOps.Concelier.Storage.Mongo;
|
||||||
using StellaOps.Concelier.Storage.Mongo.Advisories;
|
using StellaOps.Concelier.Storage.Mongo.Advisories;
|
||||||
using StellaOps.Concelier.Storage.Mongo.Documents;
|
using StellaOps.Concelier.Storage.Mongo.Documents;
|
||||||
using StellaOps.Concelier.Storage.Mongo.Dtos;
|
using StellaOps.Concelier.Storage.Mongo.Dtos;
|
||||||
using StellaOps.Plugin;
|
using StellaOps.Plugin;
|
||||||
|
|
||||||
namespace StellaOps.Concelier.Connector.CertIn;
|
namespace StellaOps.Concelier.Connector.CertIn;
|
||||||
|
|
||||||
public sealed class CertInConnector : IFeedConnector
|
public sealed class CertInConnector : IFeedConnector
|
||||||
{
|
{
|
||||||
private static readonly JsonSerializerOptions SerializerOptions = new(JsonSerializerDefaults.General)
|
private static readonly JsonSerializerOptions SerializerOptions = new(JsonSerializerDefaults.General)
|
||||||
{
|
{
|
||||||
PropertyNamingPolicy = JsonNamingPolicy.CamelCase,
|
PropertyNamingPolicy = JsonNamingPolicy.CamelCase,
|
||||||
WriteIndented = false,
|
WriteIndented = false,
|
||||||
DefaultIgnoreCondition = System.Text.Json.Serialization.JsonIgnoreCondition.WhenWritingNull,
|
DefaultIgnoreCondition = System.Text.Json.Serialization.JsonIgnoreCondition.WhenWritingNull,
|
||||||
};
|
};
|
||||||
|
|
||||||
private readonly CertInClient _client;
|
private readonly CertInClient _client;
|
||||||
private readonly SourceFetchService _fetchService;
|
private readonly SourceFetchService _fetchService;
|
||||||
private readonly RawDocumentStorage _rawDocumentStorage;
|
private readonly RawDocumentStorage _rawDocumentStorage;
|
||||||
private readonly IDocumentStore _documentStore;
|
private readonly IDocumentStore _documentStore;
|
||||||
private readonly IDtoStore _dtoStore;
|
private readonly IDtoStore _dtoStore;
|
||||||
private readonly IAdvisoryStore _advisoryStore;
|
private readonly IAdvisoryStore _advisoryStore;
|
||||||
private readonly ISourceStateRepository _stateRepository;
|
private readonly ISourceStateRepository _stateRepository;
|
||||||
private readonly CertInOptions _options;
|
private readonly CertInOptions _options;
|
||||||
private readonly TimeProvider _timeProvider;
|
private readonly TimeProvider _timeProvider;
|
||||||
private readonly ILogger<CertInConnector> _logger;
|
private readonly ILogger<CertInConnector> _logger;
|
||||||
|
|
||||||
public CertInConnector(
|
public CertInConnector(
|
||||||
CertInClient client,
|
CertInClient client,
|
||||||
SourceFetchService fetchService,
|
SourceFetchService fetchService,
|
||||||
RawDocumentStorage rawDocumentStorage,
|
RawDocumentStorage rawDocumentStorage,
|
||||||
IDocumentStore documentStore,
|
IDocumentStore documentStore,
|
||||||
IDtoStore dtoStore,
|
IDtoStore dtoStore,
|
||||||
IAdvisoryStore advisoryStore,
|
IAdvisoryStore advisoryStore,
|
||||||
ISourceStateRepository stateRepository,
|
ISourceStateRepository stateRepository,
|
||||||
IOptions<CertInOptions> options,
|
IOptions<CertInOptions> options,
|
||||||
TimeProvider? timeProvider,
|
TimeProvider? timeProvider,
|
||||||
ILogger<CertInConnector> logger)
|
ILogger<CertInConnector> logger)
|
||||||
{
|
{
|
||||||
_client = client ?? throw new ArgumentNullException(nameof(client));
|
_client = client ?? throw new ArgumentNullException(nameof(client));
|
||||||
_fetchService = fetchService ?? throw new ArgumentNullException(nameof(fetchService));
|
_fetchService = fetchService ?? throw new ArgumentNullException(nameof(fetchService));
|
||||||
_rawDocumentStorage = rawDocumentStorage ?? throw new ArgumentNullException(nameof(rawDocumentStorage));
|
_rawDocumentStorage = rawDocumentStorage ?? throw new ArgumentNullException(nameof(rawDocumentStorage));
|
||||||
_documentStore = documentStore ?? throw new ArgumentNullException(nameof(documentStore));
|
_documentStore = documentStore ?? throw new ArgumentNullException(nameof(documentStore));
|
||||||
_dtoStore = dtoStore ?? throw new ArgumentNullException(nameof(dtoStore));
|
_dtoStore = dtoStore ?? throw new ArgumentNullException(nameof(dtoStore));
|
||||||
_advisoryStore = advisoryStore ?? throw new ArgumentNullException(nameof(advisoryStore));
|
_advisoryStore = advisoryStore ?? throw new ArgumentNullException(nameof(advisoryStore));
|
||||||
_stateRepository = stateRepository ?? throw new ArgumentNullException(nameof(stateRepository));
|
_stateRepository = stateRepository ?? throw new ArgumentNullException(nameof(stateRepository));
|
||||||
_options = (options ?? throw new ArgumentNullException(nameof(options))).Value ?? throw new ArgumentNullException(nameof(options));
|
_options = (options ?? throw new ArgumentNullException(nameof(options))).Value ?? throw new ArgumentNullException(nameof(options));
|
||||||
_options.Validate();
|
_options.Validate();
|
||||||
_timeProvider = timeProvider ?? TimeProvider.System;
|
_timeProvider = timeProvider ?? TimeProvider.System;
|
||||||
_logger = logger ?? throw new ArgumentNullException(nameof(logger));
|
_logger = logger ?? throw new ArgumentNullException(nameof(logger));
|
||||||
}
|
}
|
||||||
|
|
||||||
public string SourceName => CertInConnectorPlugin.SourceName;
|
public string SourceName => CertInConnectorPlugin.SourceName;
|
||||||
|
|
||||||
public async Task FetchAsync(IServiceProvider services, CancellationToken cancellationToken)
|
public async Task FetchAsync(IServiceProvider services, CancellationToken cancellationToken)
|
||||||
{
|
{
|
||||||
ArgumentNullException.ThrowIfNull(services);
|
ArgumentNullException.ThrowIfNull(services);
|
||||||
|
|
||||||
var cursor = await GetCursorAsync(cancellationToken).ConfigureAwait(false);
|
var cursor = await GetCursorAsync(cancellationToken).ConfigureAwait(false);
|
||||||
var now = _timeProvider.GetUtcNow();
|
var now = _timeProvider.GetUtcNow();
|
||||||
var windowStart = cursor.LastPublished.HasValue
|
var windowStart = cursor.LastPublished.HasValue
|
||||||
? cursor.LastPublished.Value - _options.WindowOverlap
|
? cursor.LastPublished.Value - _options.WindowOverlap
|
||||||
: now - _options.WindowSize;
|
: now - _options.WindowSize;
|
||||||
|
|
||||||
var pendingDocuments = cursor.PendingDocuments.ToHashSet();
|
var pendingDocuments = cursor.PendingDocuments.ToHashSet();
|
||||||
var maxPublished = cursor.LastPublished ?? DateTimeOffset.MinValue;
|
var maxPublished = cursor.LastPublished ?? DateTimeOffset.MinValue;
|
||||||
|
|
||||||
for (var page = 1; page <= _options.MaxPagesPerFetch; page++)
|
for (var page = 1; page <= _options.MaxPagesPerFetch; page++)
|
||||||
{
|
{
|
||||||
IReadOnlyList<CertInListingItem> listings;
|
IReadOnlyList<CertInListingItem> listings;
|
||||||
try
|
try
|
||||||
{
|
{
|
||||||
listings = await _client.GetListingsAsync(page, cancellationToken).ConfigureAwait(false);
|
listings = await _client.GetListingsAsync(page, cancellationToken).ConfigureAwait(false);
|
||||||
}
|
}
|
||||||
catch (Exception ex)
|
catch (Exception ex)
|
||||||
{
|
{
|
||||||
_logger.LogError(ex, "CERT-In listings fetch failed for page {Page}", page);
|
_logger.LogError(ex, "CERT-In listings fetch failed for page {Page}", page);
|
||||||
await _stateRepository.MarkFailureAsync(SourceName, now, TimeSpan.FromMinutes(5), ex.Message, cancellationToken).ConfigureAwait(false);
|
await _stateRepository.MarkFailureAsync(SourceName, now, TimeSpan.FromMinutes(5), ex.Message, cancellationToken).ConfigureAwait(false);
|
||||||
throw;
|
throw;
|
||||||
}
|
}
|
||||||
if (listings.Count == 0)
|
if (listings.Count == 0)
|
||||||
{
|
{
|
||||||
break;
|
break;
|
||||||
}
|
}
|
||||||
|
|
||||||
foreach (var listing in listings.OrderByDescending(static item => item.Published))
|
foreach (var listing in listings.OrderByDescending(static item => item.Published))
|
||||||
{
|
{
|
||||||
if (listing.Published < windowStart)
|
if (listing.Published < windowStart)
|
||||||
{
|
{
|
||||||
page = _options.MaxPagesPerFetch + 1;
|
page = _options.MaxPagesPerFetch + 1;
|
||||||
break;
|
break;
|
||||||
}
|
}
|
||||||
|
|
||||||
var metadata = new Dictionary<string, string>(StringComparer.Ordinal)
|
var metadata = new Dictionary<string, string>(StringComparer.Ordinal)
|
||||||
{
|
{
|
||||||
["certin.advisoryId"] = listing.AdvisoryId,
|
["certin.advisoryId"] = listing.AdvisoryId,
|
||||||
["certin.title"] = listing.Title,
|
["certin.title"] = listing.Title,
|
||||||
["certin.link"] = listing.DetailUri.ToString(),
|
["certin.link"] = listing.DetailUri.ToString(),
|
||||||
["certin.published"] = listing.Published.ToString("O")
|
["certin.published"] = listing.Published.ToString("O")
|
||||||
};
|
};
|
||||||
|
|
||||||
if (!string.IsNullOrWhiteSpace(listing.Summary))
|
if (!string.IsNullOrWhiteSpace(listing.Summary))
|
||||||
{
|
{
|
||||||
metadata["certin.summary"] = listing.Summary!;
|
metadata["certin.summary"] = listing.Summary!;
|
||||||
}
|
}
|
||||||
|
|
||||||
var existing = await _documentStore.FindBySourceAndUriAsync(SourceName, listing.DetailUri.ToString(), cancellationToken).ConfigureAwait(false);
|
var existing = await _documentStore.FindBySourceAndUriAsync(SourceName, listing.DetailUri.ToString(), cancellationToken).ConfigureAwait(false);
|
||||||
|
|
||||||
SourceFetchResult result;
|
SourceFetchResult result;
|
||||||
try
|
try
|
||||||
{
|
{
|
||||||
result = await _fetchService.FetchAsync(
|
result = await _fetchService.FetchAsync(
|
||||||
new SourceFetchRequest(CertInOptions.HttpClientName, SourceName, listing.DetailUri)
|
new SourceFetchRequest(CertInOptions.HttpClientName, SourceName, listing.DetailUri)
|
||||||
{
|
{
|
||||||
Metadata = metadata,
|
Metadata = metadata,
|
||||||
ETag = existing?.Etag,
|
ETag = existing?.Etag,
|
||||||
LastModified = existing?.LastModified,
|
LastModified = existing?.LastModified,
|
||||||
AcceptHeaders = new[] { "text/html", "application/xhtml+xml", "text/plain;q=0.5" },
|
AcceptHeaders = new[] { "text/html", "application/xhtml+xml", "text/plain;q=0.5" },
|
||||||
},
|
},
|
||||||
cancellationToken).ConfigureAwait(false);
|
cancellationToken).ConfigureAwait(false);
|
||||||
}
|
}
|
||||||
catch (Exception ex)
|
catch (Exception ex)
|
||||||
{
|
{
|
||||||
_logger.LogError(ex, "CERT-In fetch failed for {Uri}", listing.DetailUri);
|
_logger.LogError(ex, "CERT-In fetch failed for {Uri}", listing.DetailUri);
|
||||||
await _stateRepository.MarkFailureAsync(SourceName, _timeProvider.GetUtcNow(), TimeSpan.FromMinutes(3), ex.Message, cancellationToken).ConfigureAwait(false);
|
await _stateRepository.MarkFailureAsync(SourceName, _timeProvider.GetUtcNow(), TimeSpan.FromMinutes(3), ex.Message, cancellationToken).ConfigureAwait(false);
|
||||||
throw;
|
throw;
|
||||||
}
|
}
|
||||||
|
|
||||||
if (!result.IsSuccess || result.Document is null)
|
if (!result.IsSuccess || result.Document is null)
|
||||||
{
|
{
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
|
|
||||||
if (existing is not null
|
if (existing is not null
|
||||||
&& string.Equals(existing.Sha256, result.Document.Sha256, StringComparison.OrdinalIgnoreCase)
|
&& string.Equals(existing.Sha256, result.Document.Sha256, StringComparison.OrdinalIgnoreCase)
|
||||||
&& string.Equals(existing.Status, DocumentStatuses.Mapped, StringComparison.Ordinal))
|
&& string.Equals(existing.Status, DocumentStatuses.Mapped, StringComparison.Ordinal))
|
||||||
{
|
{
|
||||||
await _documentStore.UpdateStatusAsync(result.Document.Id, existing.Status, cancellationToken).ConfigureAwait(false);
|
await _documentStore.UpdateStatusAsync(result.Document.Id, existing.Status, cancellationToken).ConfigureAwait(false);
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
|
|
||||||
pendingDocuments.Add(result.Document.Id);
|
pendingDocuments.Add(result.Document.Id);
|
||||||
if (listing.Published > maxPublished)
|
if (listing.Published > maxPublished)
|
||||||
{
|
{
|
||||||
maxPublished = listing.Published;
|
maxPublished = listing.Published;
|
||||||
}
|
}
|
||||||
|
|
||||||
if (_options.RequestDelay > TimeSpan.Zero)
|
if (_options.RequestDelay > TimeSpan.Zero)
|
||||||
{
|
{
|
||||||
await Task.Delay(_options.RequestDelay, cancellationToken).ConfigureAwait(false);
|
await Task.Delay(_options.RequestDelay, cancellationToken).ConfigureAwait(false);
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
var updatedCursor = cursor
|
var updatedCursor = cursor
|
||||||
.WithPendingDocuments(pendingDocuments)
|
.WithPendingDocuments(pendingDocuments)
|
||||||
.WithLastPublished(maxPublished == DateTimeOffset.MinValue ? cursor.LastPublished : maxPublished);
|
.WithLastPublished(maxPublished == DateTimeOffset.MinValue ? cursor.LastPublished : maxPublished);
|
||||||
|
|
||||||
await UpdateCursorAsync(updatedCursor, cancellationToken).ConfigureAwait(false);
|
await UpdateCursorAsync(updatedCursor, cancellationToken).ConfigureAwait(false);
|
||||||
}
|
}
|
||||||
|
|
||||||
public async Task ParseAsync(IServiceProvider services, CancellationToken cancellationToken)
|
public async Task ParseAsync(IServiceProvider services, CancellationToken cancellationToken)
|
||||||
{
|
{
|
||||||
ArgumentNullException.ThrowIfNull(services);
|
ArgumentNullException.ThrowIfNull(services);
|
||||||
|
|
||||||
var cursor = await GetCursorAsync(cancellationToken).ConfigureAwait(false);
|
var cursor = await GetCursorAsync(cancellationToken).ConfigureAwait(false);
|
||||||
if (cursor.PendingDocuments.Count == 0)
|
if (cursor.PendingDocuments.Count == 0)
|
||||||
{
|
{
|
||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
|
|
||||||
var remainingDocuments = cursor.PendingDocuments.ToList();
|
var remainingDocuments = cursor.PendingDocuments.ToList();
|
||||||
var pendingMappings = cursor.PendingMappings.ToList();
|
var pendingMappings = cursor.PendingMappings.ToList();
|
||||||
|
|
||||||
foreach (var documentId in cursor.PendingDocuments)
|
foreach (var documentId in cursor.PendingDocuments)
|
||||||
{
|
{
|
||||||
cancellationToken.ThrowIfCancellationRequested();
|
cancellationToken.ThrowIfCancellationRequested();
|
||||||
|
|
||||||
var document = await _documentStore.FindAsync(documentId, cancellationToken).ConfigureAwait(false);
|
var document = await _documentStore.FindAsync(documentId, cancellationToken).ConfigureAwait(false);
|
||||||
if (document is null)
|
if (document is null)
|
||||||
{
|
{
|
||||||
remainingDocuments.Remove(documentId);
|
remainingDocuments.Remove(documentId);
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
|
|
||||||
if (!document.GridFsId.HasValue)
|
if (!document.PayloadId.HasValue)
|
||||||
{
|
{
|
||||||
_logger.LogWarning("CERT-In document {DocumentId} missing GridFS payload", document.Id);
|
_logger.LogWarning("CERT-In document {DocumentId} missing GridFS payload", document.Id);
|
||||||
await _documentStore.UpdateStatusAsync(document.Id, DocumentStatuses.Failed, cancellationToken).ConfigureAwait(false);
|
await _documentStore.UpdateStatusAsync(document.Id, DocumentStatuses.Failed, cancellationToken).ConfigureAwait(false);
|
||||||
remainingDocuments.Remove(documentId);
|
remainingDocuments.Remove(documentId);
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
|
|
||||||
if (!TryDeserializeListing(document.Metadata, out var listing))
|
if (!TryDeserializeListing(document.Metadata, out var listing))
|
||||||
{
|
{
|
||||||
_logger.LogWarning("CERT-In metadata missing for {DocumentId}", document.Id);
|
_logger.LogWarning("CERT-In metadata missing for {DocumentId}", document.Id);
|
||||||
await _documentStore.UpdateStatusAsync(document.Id, DocumentStatuses.Failed, cancellationToken).ConfigureAwait(false);
|
await _documentStore.UpdateStatusAsync(document.Id, DocumentStatuses.Failed, cancellationToken).ConfigureAwait(false);
|
||||||
remainingDocuments.Remove(documentId);
|
remainingDocuments.Remove(documentId);
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
|
|
||||||
byte[] rawBytes;
|
byte[] rawBytes;
|
||||||
try
|
try
|
||||||
{
|
{
|
||||||
rawBytes = await _rawDocumentStorage.DownloadAsync(document.GridFsId.Value, cancellationToken).ConfigureAwait(false);
|
rawBytes = await _rawDocumentStorage.DownloadAsync(document.PayloadId.Value, cancellationToken).ConfigureAwait(false);
|
||||||
}
|
}
|
||||||
catch (Exception ex)
|
catch (Exception ex)
|
||||||
{
|
{
|
||||||
_logger.LogError(ex, "Failed to download raw CERT-In document {DocumentId}", document.Id);
|
_logger.LogError(ex, "Failed to download raw CERT-In document {DocumentId}", document.Id);
|
||||||
throw;
|
throw;
|
||||||
}
|
}
|
||||||
|
|
||||||
var dto = CertInDetailParser.Parse(listing, rawBytes);
|
var dto = CertInDetailParser.Parse(listing, rawBytes);
|
||||||
var payload = BsonDocument.Parse(JsonSerializer.Serialize(dto, SerializerOptions));
|
var payload = BsonDocument.Parse(JsonSerializer.Serialize(dto, SerializerOptions));
|
||||||
var dtoRecord = new DtoRecord(Guid.NewGuid(), document.Id, SourceName, "certin.v1", payload, _timeProvider.GetUtcNow());
|
var dtoRecord = new DtoRecord(Guid.NewGuid(), document.Id, SourceName, "certin.v1", payload, _timeProvider.GetUtcNow());
|
||||||
|
|
||||||
await _dtoStore.UpsertAsync(dtoRecord, cancellationToken).ConfigureAwait(false);
|
await _dtoStore.UpsertAsync(dtoRecord, cancellationToken).ConfigureAwait(false);
|
||||||
await _documentStore.UpdateStatusAsync(document.Id, DocumentStatuses.PendingMap, cancellationToken).ConfigureAwait(false);
|
await _documentStore.UpdateStatusAsync(document.Id, DocumentStatuses.PendingMap, cancellationToken).ConfigureAwait(false);
|
||||||
|
|
||||||
remainingDocuments.Remove(documentId);
|
remainingDocuments.Remove(documentId);
|
||||||
if (!pendingMappings.Contains(documentId))
|
if (!pendingMappings.Contains(documentId))
|
||||||
{
|
{
|
||||||
pendingMappings.Add(documentId);
|
pendingMappings.Add(documentId);
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
var updatedCursor = cursor
|
var updatedCursor = cursor
|
||||||
.WithPendingDocuments(remainingDocuments)
|
.WithPendingDocuments(remainingDocuments)
|
||||||
.WithPendingMappings(pendingMappings);
|
.WithPendingMappings(pendingMappings);
|
||||||
|
|
||||||
await UpdateCursorAsync(updatedCursor, cancellationToken).ConfigureAwait(false);
|
await UpdateCursorAsync(updatedCursor, cancellationToken).ConfigureAwait(false);
|
||||||
}
|
}
|
||||||
|
|
||||||
public async Task MapAsync(IServiceProvider services, CancellationToken cancellationToken)
|
public async Task MapAsync(IServiceProvider services, CancellationToken cancellationToken)
|
||||||
{
|
{
|
||||||
ArgumentNullException.ThrowIfNull(services);
|
ArgumentNullException.ThrowIfNull(services);
|
||||||
|
|
||||||
var cursor = await GetCursorAsync(cancellationToken).ConfigureAwait(false);
|
var cursor = await GetCursorAsync(cancellationToken).ConfigureAwait(false);
|
||||||
if (cursor.PendingMappings.Count == 0)
|
if (cursor.PendingMappings.Count == 0)
|
||||||
{
|
{
|
||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
|
|
||||||
var pendingMappings = cursor.PendingMappings.ToList();
|
var pendingMappings = cursor.PendingMappings.ToList();
|
||||||
|
|
||||||
foreach (var documentId in cursor.PendingMappings)
|
foreach (var documentId in cursor.PendingMappings)
|
||||||
{
|
{
|
||||||
cancellationToken.ThrowIfCancellationRequested();
|
cancellationToken.ThrowIfCancellationRequested();
|
||||||
|
|
||||||
var dtoRecord = await _dtoStore.FindByDocumentIdAsync(documentId, cancellationToken).ConfigureAwait(false);
|
var dtoRecord = await _dtoStore.FindByDocumentIdAsync(documentId, cancellationToken).ConfigureAwait(false);
|
||||||
var document = await _documentStore.FindAsync(documentId, cancellationToken).ConfigureAwait(false);
|
var document = await _documentStore.FindAsync(documentId, cancellationToken).ConfigureAwait(false);
|
||||||
|
|
||||||
if (dtoRecord is null || document is null)
|
if (dtoRecord is null || document is null)
|
||||||
{
|
{
|
||||||
pendingMappings.Remove(documentId);
|
pendingMappings.Remove(documentId);
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
|
|
||||||
var dtoJson = dtoRecord.Payload.ToJson(new MongoDB.Bson.IO.JsonWriterSettings
|
var dtoJson = dtoRecord.Payload.ToJson(new MongoDB.Bson.IO.JsonWriterSettings
|
||||||
{
|
{
|
||||||
OutputMode = MongoDB.Bson.IO.JsonOutputMode.RelaxedExtendedJson,
|
OutputMode = MongoDB.Bson.IO.JsonOutputMode.RelaxedExtendedJson,
|
||||||
});
|
});
|
||||||
|
|
||||||
CertInAdvisoryDto dto;
|
CertInAdvisoryDto dto;
|
||||||
try
|
try
|
||||||
{
|
{
|
||||||
dto = JsonSerializer.Deserialize<CertInAdvisoryDto>(dtoJson, SerializerOptions)
|
dto = JsonSerializer.Deserialize<CertInAdvisoryDto>(dtoJson, SerializerOptions)
|
||||||
?? throw new InvalidOperationException("Deserialized CERT-In DTO is null.");
|
?? throw new InvalidOperationException("Deserialized CERT-In DTO is null.");
|
||||||
}
|
}
|
||||||
catch (Exception ex)
|
catch (Exception ex)
|
||||||
{
|
{
|
||||||
_logger.LogError(ex, "Failed to deserialize CERT-In DTO for {DocumentId}", document.Id);
|
_logger.LogError(ex, "Failed to deserialize CERT-In DTO for {DocumentId}", document.Id);
|
||||||
await _documentStore.UpdateStatusAsync(document.Id, DocumentStatuses.Failed, cancellationToken).ConfigureAwait(false);
|
await _documentStore.UpdateStatusAsync(document.Id, DocumentStatuses.Failed, cancellationToken).ConfigureAwait(false);
|
||||||
pendingMappings.Remove(documentId);
|
pendingMappings.Remove(documentId);
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
|
|
||||||
var advisory = MapAdvisory(dto, document, dtoRecord);
|
var advisory = MapAdvisory(dto, document, dtoRecord);
|
||||||
await _advisoryStore.UpsertAsync(advisory, cancellationToken).ConfigureAwait(false);
|
await _advisoryStore.UpsertAsync(advisory, cancellationToken).ConfigureAwait(false);
|
||||||
await _documentStore.UpdateStatusAsync(document.Id, DocumentStatuses.Mapped, cancellationToken).ConfigureAwait(false);
|
await _documentStore.UpdateStatusAsync(document.Id, DocumentStatuses.Mapped, cancellationToken).ConfigureAwait(false);
|
||||||
|
|
||||||
pendingMappings.Remove(documentId);
|
pendingMappings.Remove(documentId);
|
||||||
}
|
}
|
||||||
|
|
||||||
var updatedCursor = cursor.WithPendingMappings(pendingMappings);
|
var updatedCursor = cursor.WithPendingMappings(pendingMappings);
|
||||||
await UpdateCursorAsync(updatedCursor, cancellationToken).ConfigureAwait(false);
|
await UpdateCursorAsync(updatedCursor, cancellationToken).ConfigureAwait(false);
|
||||||
}
|
}
|
||||||
|
|
||||||
private Advisory MapAdvisory(CertInAdvisoryDto dto, DocumentRecord document, DtoRecord dtoRecord)
|
private Advisory MapAdvisory(CertInAdvisoryDto dto, DocumentRecord document, DtoRecord dtoRecord)
|
||||||
{
|
{
|
||||||
var fetchProvenance = new AdvisoryProvenance(SourceName, "document", document.Uri, document.FetchedAt);
|
var fetchProvenance = new AdvisoryProvenance(SourceName, "document", document.Uri, document.FetchedAt);
|
||||||
var mappingProvenance = new AdvisoryProvenance(SourceName, "mapping", dto.AdvisoryId, dtoRecord.ValidatedAt);
|
var mappingProvenance = new AdvisoryProvenance(SourceName, "mapping", dto.AdvisoryId, dtoRecord.ValidatedAt);
|
||||||
|
|
||||||
var aliases = new HashSet<string>(StringComparer.OrdinalIgnoreCase)
|
var aliases = new HashSet<string>(StringComparer.OrdinalIgnoreCase)
|
||||||
{
|
{
|
||||||
dto.AdvisoryId,
|
dto.AdvisoryId,
|
||||||
};
|
};
|
||||||
foreach (var cve in dto.CveIds)
|
foreach (var cve in dto.CveIds)
|
||||||
{
|
{
|
||||||
aliases.Add(cve);
|
aliases.Add(cve);
|
||||||
}
|
}
|
||||||
|
|
||||||
var references = new List<AdvisoryReference>();
|
var references = new List<AdvisoryReference>();
|
||||||
try
|
try
|
||||||
{
|
{
|
||||||
references.Add(new AdvisoryReference(
|
references.Add(new AdvisoryReference(
|
||||||
dto.Link,
|
dto.Link,
|
||||||
"advisory",
|
"advisory",
|
||||||
"cert-in",
|
"cert-in",
|
||||||
null,
|
null,
|
||||||
new AdvisoryProvenance(SourceName, "reference", dto.Link, dtoRecord.ValidatedAt)));
|
new AdvisoryProvenance(SourceName, "reference", dto.Link, dtoRecord.ValidatedAt)));
|
||||||
}
|
}
|
||||||
catch (ArgumentException)
|
catch (ArgumentException)
|
||||||
{
|
{
|
||||||
_logger.LogWarning("Invalid CERT-In link {Link} for advisory {AdvisoryId}", dto.Link, dto.AdvisoryId);
|
_logger.LogWarning("Invalid CERT-In link {Link} for advisory {AdvisoryId}", dto.Link, dto.AdvisoryId);
|
||||||
}
|
}
|
||||||
|
|
||||||
foreach (var cve in dto.CveIds)
|
foreach (var cve in dto.CveIds)
|
||||||
{
|
{
|
||||||
var url = $"https://www.cve.org/CVERecord?id={cve}";
|
var url = $"https://www.cve.org/CVERecord?id={cve}";
|
||||||
try
|
try
|
||||||
{
|
{
|
||||||
references.Add(new AdvisoryReference(
|
references.Add(new AdvisoryReference(
|
||||||
url,
|
url,
|
||||||
"advisory",
|
"advisory",
|
||||||
cve,
|
cve,
|
||||||
null,
|
null,
|
||||||
new AdvisoryProvenance(SourceName, "reference", url, dtoRecord.ValidatedAt)));
|
new AdvisoryProvenance(SourceName, "reference", url, dtoRecord.ValidatedAt)));
|
||||||
}
|
}
|
||||||
catch (ArgumentException)
|
catch (ArgumentException)
|
||||||
{
|
{
|
||||||
// ignore invalid urls
|
// ignore invalid urls
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
foreach (var link in dto.ReferenceLinks)
|
foreach (var link in dto.ReferenceLinks)
|
||||||
{
|
{
|
||||||
try
|
try
|
||||||
{
|
{
|
||||||
references.Add(new AdvisoryReference(
|
references.Add(new AdvisoryReference(
|
||||||
link,
|
link,
|
||||||
"reference",
|
"reference",
|
||||||
null,
|
null,
|
||||||
null,
|
null,
|
||||||
new AdvisoryProvenance(SourceName, "reference", link, dtoRecord.ValidatedAt)));
|
new AdvisoryProvenance(SourceName, "reference", link, dtoRecord.ValidatedAt)));
|
||||||
}
|
}
|
||||||
catch (ArgumentException)
|
catch (ArgumentException)
|
||||||
{
|
{
|
||||||
// ignore invalid urls
|
// ignore invalid urls
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
var affectedPackages = dto.VendorNames.Select(vendor =>
|
var affectedPackages = dto.VendorNames.Select(vendor =>
|
||||||
{
|
{
|
||||||
var provenance = new AdvisoryProvenance(SourceName, "affected", vendor, dtoRecord.ValidatedAt);
|
var provenance = new AdvisoryProvenance(SourceName, "affected", vendor, dtoRecord.ValidatedAt);
|
||||||
@@ -398,65 +398,65 @@ public sealed class CertInConnector : IFeedConnector
|
|||||||
provenance: new[] { provenance });
|
provenance: new[] { provenance });
|
||||||
})
|
})
|
||||||
.ToArray();
|
.ToArray();
|
||||||
|
|
||||||
return new Advisory(
|
return new Advisory(
|
||||||
dto.AdvisoryId,
|
dto.AdvisoryId,
|
||||||
dto.Title,
|
dto.Title,
|
||||||
dto.Summary ?? dto.Content,
|
dto.Summary ?? dto.Content,
|
||||||
language: "en",
|
language: "en",
|
||||||
published: dto.Published,
|
published: dto.Published,
|
||||||
modified: dto.Published,
|
modified: dto.Published,
|
||||||
severity: dto.Severity,
|
severity: dto.Severity,
|
||||||
exploitKnown: false,
|
exploitKnown: false,
|
||||||
aliases: aliases,
|
aliases: aliases,
|
||||||
references: references,
|
references: references,
|
||||||
affectedPackages: affectedPackages,
|
affectedPackages: affectedPackages,
|
||||||
cvssMetrics: Array.Empty<CvssMetric>(),
|
cvssMetrics: Array.Empty<CvssMetric>(),
|
||||||
provenance: new[] { fetchProvenance, mappingProvenance });
|
provenance: new[] { fetchProvenance, mappingProvenance });
|
||||||
}
|
}
|
||||||
|
|
||||||
private async Task<CertInCursor> GetCursorAsync(CancellationToken cancellationToken)
|
private async Task<CertInCursor> GetCursorAsync(CancellationToken cancellationToken)
|
||||||
{
|
{
|
||||||
var state = await _stateRepository.TryGetAsync(SourceName, cancellationToken).ConfigureAwait(false);
|
var state = await _stateRepository.TryGetAsync(SourceName, cancellationToken).ConfigureAwait(false);
|
||||||
return state is null ? CertInCursor.Empty : CertInCursor.FromBson(state.Cursor);
|
return state is null ? CertInCursor.Empty : CertInCursor.FromBson(state.Cursor);
|
||||||
}
|
}
|
||||||
|
|
||||||
private Task UpdateCursorAsync(CertInCursor cursor, CancellationToken cancellationToken)
|
private Task UpdateCursorAsync(CertInCursor cursor, CancellationToken cancellationToken)
|
||||||
{
|
{
|
||||||
return _stateRepository.UpdateCursorAsync(SourceName, cursor.ToBsonDocument(), _timeProvider.GetUtcNow(), cancellationToken);
|
return _stateRepository.UpdateCursorAsync(SourceName, cursor.ToBsonDocument(), _timeProvider.GetUtcNow(), cancellationToken);
|
||||||
}
|
}
|
||||||
|
|
||||||
private static bool TryDeserializeListing(IReadOnlyDictionary<string, string>? metadata, out CertInListingItem listing)
|
private static bool TryDeserializeListing(IReadOnlyDictionary<string, string>? metadata, out CertInListingItem listing)
|
||||||
{
|
{
|
||||||
listing = null!;
|
listing = null!;
|
||||||
if (metadata is null)
|
if (metadata is null)
|
||||||
{
|
{
|
||||||
return false;
|
return false;
|
||||||
}
|
}
|
||||||
|
|
||||||
if (!metadata.TryGetValue("certin.advisoryId", out var advisoryId))
|
if (!metadata.TryGetValue("certin.advisoryId", out var advisoryId))
|
||||||
{
|
{
|
||||||
return false;
|
return false;
|
||||||
}
|
}
|
||||||
|
|
||||||
if (!metadata.TryGetValue("certin.title", out var title))
|
if (!metadata.TryGetValue("certin.title", out var title))
|
||||||
{
|
{
|
||||||
return false;
|
return false;
|
||||||
}
|
}
|
||||||
|
|
||||||
if (!metadata.TryGetValue("certin.link", out var link) || !Uri.TryCreate(link, UriKind.Absolute, out var detailUri))
|
if (!metadata.TryGetValue("certin.link", out var link) || !Uri.TryCreate(link, UriKind.Absolute, out var detailUri))
|
||||||
{
|
{
|
||||||
return false;
|
return false;
|
||||||
}
|
}
|
||||||
|
|
||||||
if (!metadata.TryGetValue("certin.published", out var publishedText) || !DateTimeOffset.TryParse(publishedText, out var published))
|
if (!metadata.TryGetValue("certin.published", out var publishedText) || !DateTimeOffset.TryParse(publishedText, out var published))
|
||||||
{
|
{
|
||||||
return false;
|
return false;
|
||||||
}
|
}
|
||||||
|
|
||||||
metadata.TryGetValue("certin.summary", out var summary);
|
metadata.TryGetValue("certin.summary", out var summary);
|
||||||
|
|
||||||
listing = new CertInListingItem(advisoryId, title, detailUri, published.ToUniversalTime(), summary);
|
listing = new CertInListingItem(advisoryId, title, detailUri, published.ToUniversalTime(), summary);
|
||||||
return true;
|
return true;
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -3,6 +3,7 @@ using System.Net.Http;
|
|||||||
using System.Net.Security;
|
using System.Net.Security;
|
||||||
using System.Security.Cryptography.X509Certificates;
|
using System.Security.Cryptography.X509Certificates;
|
||||||
using Microsoft.Extensions.DependencyInjection;
|
using Microsoft.Extensions.DependencyInjection;
|
||||||
|
using Microsoft.Extensions.DependencyInjection.Extensions;
|
||||||
using Microsoft.Extensions.Options;
|
using Microsoft.Extensions.Options;
|
||||||
using StellaOps.Concelier.Connector.Common.Xml;
|
using StellaOps.Concelier.Connector.Common.Xml;
|
||||||
using StellaOps.Concelier.Core.Aoc;
|
using StellaOps.Concelier.Core.Aoc;
|
||||||
@@ -169,7 +170,7 @@ public static class ServiceCollectionExtensions
|
|||||||
services.AddSingleton<Fetch.IJitterSource, Fetch.CryptoJitterSource>();
|
services.AddSingleton<Fetch.IJitterSource, Fetch.CryptoJitterSource>();
|
||||||
services.AddConcelierAocGuards();
|
services.AddConcelierAocGuards();
|
||||||
services.AddConcelierLinksetMappers();
|
services.AddConcelierLinksetMappers();
|
||||||
services.AddSingleton<IDocumentStore, InMemoryDocumentStore>();
|
services.TryAddSingleton<IDocumentStore, InMemoryDocumentStore>();
|
||||||
services.AddSingleton<Fetch.RawDocumentStorage>();
|
services.AddSingleton<Fetch.RawDocumentStorage>();
|
||||||
services.AddSingleton<Fetch.SourceFetchService>();
|
services.AddSingleton<Fetch.SourceFetchService>();
|
||||||
|
|
||||||
|
|||||||
@@ -5,16 +5,16 @@ using StellaOps.Concelier.Connector.Common.Fetch;
|
|||||||
using StellaOps.Concelier.Storage.Mongo;
|
using StellaOps.Concelier.Storage.Mongo;
|
||||||
using StellaOps.Concelier.Storage.Mongo.Documents;
|
using StellaOps.Concelier.Storage.Mongo.Documents;
|
||||||
using StellaOps.Cryptography;
|
using StellaOps.Cryptography;
|
||||||
|
|
||||||
namespace StellaOps.Concelier.Connector.Common.State;
|
namespace StellaOps.Concelier.Connector.Common.State;
|
||||||
|
|
||||||
/// <summary>
|
/// <summary>
|
||||||
/// Persists raw documents and cursor state for connectors that require manual seeding.
|
/// Persists raw documents and cursor state for connectors that require manual seeding.
|
||||||
/// </summary>
|
/// </summary>
|
||||||
public sealed class SourceStateSeedProcessor
|
public sealed class SourceStateSeedProcessor
|
||||||
{
|
{
|
||||||
private readonly IDocumentStore _documentStore;
|
private readonly IDocumentStore _documentStore;
|
||||||
private readonly RawDocumentStorage _rawDocumentStorage;
|
private readonly RawDocumentStorage _rawDocumentStorage;
|
||||||
private readonly ISourceStateRepository _stateRepository;
|
private readonly ISourceStateRepository _stateRepository;
|
||||||
private readonly TimeProvider _timeProvider;
|
private readonly TimeProvider _timeProvider;
|
||||||
private readonly ILogger<SourceStateSeedProcessor> _logger;
|
private readonly ILogger<SourceStateSeedProcessor> _logger;
|
||||||
@@ -35,298 +35,298 @@ public sealed class SourceStateSeedProcessor
|
|||||||
_timeProvider = timeProvider ?? TimeProvider.System;
|
_timeProvider = timeProvider ?? TimeProvider.System;
|
||||||
_logger = logger ?? NullLogger<SourceStateSeedProcessor>.Instance;
|
_logger = logger ?? NullLogger<SourceStateSeedProcessor>.Instance;
|
||||||
}
|
}
|
||||||
|
|
||||||
public async Task<SourceStateSeedResult> ProcessAsync(SourceStateSeedSpecification specification, CancellationToken cancellationToken)
|
public async Task<SourceStateSeedResult> ProcessAsync(SourceStateSeedSpecification specification, CancellationToken cancellationToken)
|
||||||
{
|
{
|
||||||
ArgumentNullException.ThrowIfNull(specification);
|
ArgumentNullException.ThrowIfNull(specification);
|
||||||
ArgumentException.ThrowIfNullOrEmpty(specification.Source);
|
ArgumentException.ThrowIfNullOrEmpty(specification.Source);
|
||||||
|
|
||||||
var completedAt = specification.CompletedAt ?? _timeProvider.GetUtcNow();
|
var completedAt = specification.CompletedAt ?? _timeProvider.GetUtcNow();
|
||||||
var documentIds = new List<Guid>();
|
var documentIds = new List<Guid>();
|
||||||
var pendingDocumentIds = new HashSet<Guid>();
|
var pendingDocumentIds = new HashSet<Guid>();
|
||||||
var pendingMappingIds = new HashSet<Guid>();
|
var pendingMappingIds = new HashSet<Guid>();
|
||||||
var knownAdvisories = new HashSet<string>(StringComparer.OrdinalIgnoreCase);
|
var knownAdvisories = new HashSet<string>(StringComparer.OrdinalIgnoreCase);
|
||||||
|
|
||||||
AppendRange(knownAdvisories, specification.KnownAdvisories);
|
AppendRange(knownAdvisories, specification.KnownAdvisories);
|
||||||
|
|
||||||
if (specification.Cursor is { } cursorSeed)
|
if (specification.Cursor is { } cursorSeed)
|
||||||
{
|
{
|
||||||
AppendRange(pendingDocumentIds, cursorSeed.PendingDocuments);
|
AppendRange(pendingDocumentIds, cursorSeed.PendingDocuments);
|
||||||
AppendRange(pendingMappingIds, cursorSeed.PendingMappings);
|
AppendRange(pendingMappingIds, cursorSeed.PendingMappings);
|
||||||
AppendRange(knownAdvisories, cursorSeed.KnownAdvisories);
|
AppendRange(knownAdvisories, cursorSeed.KnownAdvisories);
|
||||||
}
|
}
|
||||||
|
|
||||||
foreach (var document in specification.Documents ?? Array.Empty<SourceStateSeedDocument>())
|
foreach (var document in specification.Documents ?? Array.Empty<SourceStateSeedDocument>())
|
||||||
{
|
{
|
||||||
cancellationToken.ThrowIfCancellationRequested();
|
cancellationToken.ThrowIfCancellationRequested();
|
||||||
await ProcessDocumentAsync(specification.Source, document, completedAt, documentIds, pendingDocumentIds, pendingMappingIds, knownAdvisories, cancellationToken).ConfigureAwait(false);
|
await ProcessDocumentAsync(specification.Source, document, completedAt, documentIds, pendingDocumentIds, pendingMappingIds, knownAdvisories, cancellationToken).ConfigureAwait(false);
|
||||||
}
|
}
|
||||||
|
|
||||||
var state = await _stateRepository.TryGetAsync(specification.Source, cancellationToken).ConfigureAwait(false);
|
var state = await _stateRepository.TryGetAsync(specification.Source, cancellationToken).ConfigureAwait(false);
|
||||||
var cursor = state?.Cursor ?? new BsonDocument();
|
var cursor = state?.Cursor ?? new BsonDocument();
|
||||||
|
|
||||||
var newlyPendingDocuments = MergeGuidArray(cursor, "pendingDocuments", pendingDocumentIds);
|
var newlyPendingDocuments = MergeGuidArray(cursor, "pendingDocuments", pendingDocumentIds);
|
||||||
var newlyPendingMappings = MergeGuidArray(cursor, "pendingMappings", pendingMappingIds);
|
var newlyPendingMappings = MergeGuidArray(cursor, "pendingMappings", pendingMappingIds);
|
||||||
var newlyKnownAdvisories = MergeStringArray(cursor, "knownAdvisories", knownAdvisories);
|
var newlyKnownAdvisories = MergeStringArray(cursor, "knownAdvisories", knownAdvisories);
|
||||||
|
|
||||||
if (specification.Cursor is { } cursorSpec)
|
if (specification.Cursor is { } cursorSpec)
|
||||||
{
|
{
|
||||||
if (cursorSpec.LastModifiedCursor.HasValue)
|
if (cursorSpec.LastModifiedCursor.HasValue)
|
||||||
{
|
{
|
||||||
cursor["lastModifiedCursor"] = cursorSpec.LastModifiedCursor.Value.UtcDateTime;
|
cursor["lastModifiedCursor"] = cursorSpec.LastModifiedCursor.Value.UtcDateTime;
|
||||||
}
|
}
|
||||||
|
|
||||||
if (cursorSpec.LastFetchAt.HasValue)
|
if (cursorSpec.LastFetchAt.HasValue)
|
||||||
{
|
{
|
||||||
cursor["lastFetchAt"] = cursorSpec.LastFetchAt.Value.UtcDateTime;
|
cursor["lastFetchAt"] = cursorSpec.LastFetchAt.Value.UtcDateTime;
|
||||||
}
|
}
|
||||||
|
|
||||||
if (cursorSpec.Additional is not null)
|
if (cursorSpec.Additional is not null)
|
||||||
{
|
{
|
||||||
foreach (var kvp in cursorSpec.Additional)
|
foreach (var kvp in cursorSpec.Additional)
|
||||||
{
|
{
|
||||||
cursor[kvp.Key] = kvp.Value;
|
cursor[kvp.Key] = kvp.Value;
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
cursor["lastSeededAt"] = completedAt.UtcDateTime;
|
cursor["lastSeededAt"] = completedAt.UtcDateTime;
|
||||||
await _stateRepository.UpdateCursorAsync(specification.Source, cursor, completedAt, cancellationToken).ConfigureAwait(false);
|
await _stateRepository.UpdateCursorAsync(specification.Source, cursor, completedAt, cancellationToken).ConfigureAwait(false);
|
||||||
|
|
||||||
_logger.LogInformation(
|
_logger.LogInformation(
|
||||||
"Seeded {Documents} document(s) for {Source}. pendingDocuments+= {PendingDocuments}, pendingMappings+= {PendingMappings}, knownAdvisories+= {KnownAdvisories}",
|
"Seeded {Documents} document(s) for {Source}. pendingDocuments+= {PendingDocuments}, pendingMappings+= {PendingMappings}, knownAdvisories+= {KnownAdvisories}",
|
||||||
documentIds.Count,
|
documentIds.Count,
|
||||||
specification.Source,
|
specification.Source,
|
||||||
newlyPendingDocuments.Count,
|
newlyPendingDocuments.Count,
|
||||||
newlyPendingMappings.Count,
|
newlyPendingMappings.Count,
|
||||||
newlyKnownAdvisories.Count);
|
newlyKnownAdvisories.Count);
|
||||||
|
|
||||||
return new SourceStateSeedResult(
|
return new SourceStateSeedResult(
|
||||||
DocumentsProcessed: documentIds.Count,
|
DocumentsProcessed: documentIds.Count,
|
||||||
PendingDocumentsAdded: newlyPendingDocuments.Count,
|
PendingDocumentsAdded: newlyPendingDocuments.Count,
|
||||||
PendingMappingsAdded: newlyPendingMappings.Count,
|
PendingMappingsAdded: newlyPendingMappings.Count,
|
||||||
DocumentIds: documentIds.AsReadOnly(),
|
DocumentIds: documentIds.AsReadOnly(),
|
||||||
PendingDocumentIds: newlyPendingDocuments,
|
PendingDocumentIds: newlyPendingDocuments,
|
||||||
PendingMappingIds: newlyPendingMappings,
|
PendingMappingIds: newlyPendingMappings,
|
||||||
KnownAdvisoriesAdded: newlyKnownAdvisories,
|
KnownAdvisoriesAdded: newlyKnownAdvisories,
|
||||||
CompletedAt: completedAt);
|
CompletedAt: completedAt);
|
||||||
}
|
}
|
||||||
|
|
||||||
private async Task ProcessDocumentAsync(
|
private async Task ProcessDocumentAsync(
|
||||||
string source,
|
string source,
|
||||||
SourceStateSeedDocument document,
|
SourceStateSeedDocument document,
|
||||||
DateTimeOffset completedAt,
|
DateTimeOffset completedAt,
|
||||||
List<Guid> documentIds,
|
List<Guid> documentIds,
|
||||||
HashSet<Guid> pendingDocumentIds,
|
HashSet<Guid> pendingDocumentIds,
|
||||||
HashSet<Guid> pendingMappingIds,
|
HashSet<Guid> pendingMappingIds,
|
||||||
HashSet<string> knownAdvisories,
|
HashSet<string> knownAdvisories,
|
||||||
CancellationToken cancellationToken)
|
CancellationToken cancellationToken)
|
||||||
{
|
{
|
||||||
if (document is null)
|
if (document is null)
|
||||||
{
|
{
|
||||||
throw new ArgumentNullException(nameof(document));
|
throw new ArgumentNullException(nameof(document));
|
||||||
}
|
}
|
||||||
|
|
||||||
ArgumentException.ThrowIfNullOrEmpty(document.Uri);
|
ArgumentException.ThrowIfNullOrEmpty(document.Uri);
|
||||||
if (document.Content is not { Length: > 0 })
|
if (document.Content is not { Length: > 0 })
|
||||||
{
|
{
|
||||||
throw new InvalidOperationException($"Seed entry for '{document.Uri}' is missing content bytes.");
|
throw new InvalidOperationException($"Seed entry for '{document.Uri}' is missing content bytes.");
|
||||||
}
|
}
|
||||||
|
|
||||||
var payload = new byte[document.Content.Length];
|
var payload = new byte[document.Content.Length];
|
||||||
Buffer.BlockCopy(document.Content, 0, payload, 0, document.Content.Length);
|
Buffer.BlockCopy(document.Content, 0, payload, 0, document.Content.Length);
|
||||||
|
|
||||||
if (!document.Uri.Contains("://", StringComparison.Ordinal))
|
if (!document.Uri.Contains("://", StringComparison.Ordinal))
|
||||||
{
|
{
|
||||||
_logger.LogWarning("Seed document URI '{Uri}' does not appear to be absolute.", document.Uri);
|
_logger.LogWarning("Seed document URI '{Uri}' does not appear to be absolute.", document.Uri);
|
||||||
}
|
}
|
||||||
|
|
||||||
var contentHash = _hash.ComputeHashHex(payload, HashAlgorithms.Sha256);
|
var contentHash = _hash.ComputeHashHex(payload, HashAlgorithms.Sha256);
|
||||||
|
|
||||||
var existing = await _documentStore.FindBySourceAndUriAsync(source, document.Uri, cancellationToken).ConfigureAwait(false);
|
var existing = await _documentStore.FindBySourceAndUriAsync(source, document.Uri, cancellationToken).ConfigureAwait(false);
|
||||||
|
|
||||||
if (existing?.GridFsId is { } oldGridId)
|
if (existing?.PayloadId is { } oldGridId)
|
||||||
{
|
{
|
||||||
await _rawDocumentStorage.DeleteAsync(oldGridId, cancellationToken).ConfigureAwait(false);
|
await _rawDocumentStorage.DeleteAsync(oldGridId, cancellationToken).ConfigureAwait(false);
|
||||||
}
|
}
|
||||||
|
|
||||||
var gridId = await _rawDocumentStorage.UploadAsync(
|
var gridId = await _rawDocumentStorage.UploadAsync(
|
||||||
source,
|
source,
|
||||||
document.Uri,
|
document.Uri,
|
||||||
payload,
|
payload,
|
||||||
document.ContentType,
|
document.ContentType,
|
||||||
document.ExpiresAt,
|
document.ExpiresAt,
|
||||||
cancellationToken)
|
cancellationToken)
|
||||||
.ConfigureAwait(false);
|
.ConfigureAwait(false);
|
||||||
|
|
||||||
var headers = CloneDictionary(document.Headers);
|
var headers = CloneDictionary(document.Headers);
|
||||||
if (!string.IsNullOrWhiteSpace(document.ContentType))
|
if (!string.IsNullOrWhiteSpace(document.ContentType))
|
||||||
{
|
{
|
||||||
headers ??= new Dictionary<string, string>(StringComparer.OrdinalIgnoreCase);
|
headers ??= new Dictionary<string, string>(StringComparer.OrdinalIgnoreCase);
|
||||||
if (!headers.ContainsKey("content-type"))
|
if (!headers.ContainsKey("content-type"))
|
||||||
{
|
{
|
||||||
headers["content-type"] = document.ContentType!;
|
headers["content-type"] = document.ContentType!;
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
var metadata = CloneDictionary(document.Metadata);
|
var metadata = CloneDictionary(document.Metadata);
|
||||||
|
|
||||||
var record = new DocumentRecord(
|
var record = new DocumentRecord(
|
||||||
document.DocumentId ?? existing?.Id ?? Guid.NewGuid(),
|
document.DocumentId ?? existing?.Id ?? Guid.NewGuid(),
|
||||||
source,
|
source,
|
||||||
document.Uri,
|
document.Uri,
|
||||||
document.FetchedAt ?? completedAt,
|
document.FetchedAt ?? completedAt,
|
||||||
contentHash,
|
contentHash,
|
||||||
string.IsNullOrWhiteSpace(document.Status) ? DocumentStatuses.PendingParse : document.Status,
|
string.IsNullOrWhiteSpace(document.Status) ? DocumentStatuses.PendingParse : document.Status,
|
||||||
document.ContentType,
|
document.ContentType,
|
||||||
headers,
|
headers,
|
||||||
metadata,
|
metadata,
|
||||||
document.Etag,
|
document.Etag,
|
||||||
document.LastModified,
|
document.LastModified,
|
||||||
gridId,
|
gridId,
|
||||||
document.ExpiresAt);
|
document.ExpiresAt);
|
||||||
|
|
||||||
var upserted = await _documentStore.UpsertAsync(record, cancellationToken).ConfigureAwait(false);
|
var upserted = await _documentStore.UpsertAsync(record, cancellationToken).ConfigureAwait(false);
|
||||||
|
|
||||||
documentIds.Add(upserted.Id);
|
documentIds.Add(upserted.Id);
|
||||||
|
|
||||||
if (document.AddToPendingDocuments)
|
if (document.AddToPendingDocuments)
|
||||||
{
|
{
|
||||||
pendingDocumentIds.Add(upserted.Id);
|
pendingDocumentIds.Add(upserted.Id);
|
||||||
}
|
}
|
||||||
|
|
||||||
if (document.AddToPendingMappings)
|
if (document.AddToPendingMappings)
|
||||||
{
|
{
|
||||||
pendingMappingIds.Add(upserted.Id);
|
pendingMappingIds.Add(upserted.Id);
|
||||||
}
|
}
|
||||||
|
|
||||||
AppendRange(knownAdvisories, document.KnownIdentifiers);
|
AppendRange(knownAdvisories, document.KnownIdentifiers);
|
||||||
}
|
}
|
||||||
|
|
||||||
private static Dictionary<string, string>? CloneDictionary(IReadOnlyDictionary<string, string>? values)
|
private static Dictionary<string, string>? CloneDictionary(IReadOnlyDictionary<string, string>? values)
|
||||||
{
|
{
|
||||||
if (values is null || values.Count == 0)
|
if (values is null || values.Count == 0)
|
||||||
{
|
{
|
||||||
return null;
|
return null;
|
||||||
}
|
}
|
||||||
|
|
||||||
return new Dictionary<string, string>(values, StringComparer.OrdinalIgnoreCase);
|
return new Dictionary<string, string>(values, StringComparer.OrdinalIgnoreCase);
|
||||||
}
|
}
|
||||||
|
|
||||||
private static IReadOnlyCollection<Guid> MergeGuidArray(BsonDocument cursor, string field, IReadOnlyCollection<Guid> additions)
|
private static IReadOnlyCollection<Guid> MergeGuidArray(BsonDocument cursor, string field, IReadOnlyCollection<Guid> additions)
|
||||||
{
|
{
|
||||||
if (additions.Count == 0)
|
if (additions.Count == 0)
|
||||||
{
|
{
|
||||||
return Array.Empty<Guid>();
|
return Array.Empty<Guid>();
|
||||||
}
|
}
|
||||||
|
|
||||||
var existing = cursor.TryGetValue(field, out var value) && value is BsonArray existingArray
|
var existing = cursor.TryGetValue(field, out var value) && value is BsonArray existingArray
|
||||||
? existingArray.Select(AsGuid).Where(static g => g != Guid.Empty).ToHashSet()
|
? existingArray.Select(AsGuid).Where(static g => g != Guid.Empty).ToHashSet()
|
||||||
: new HashSet<Guid>();
|
: new HashSet<Guid>();
|
||||||
|
|
||||||
var newlyAdded = new List<Guid>();
|
var newlyAdded = new List<Guid>();
|
||||||
foreach (var guid in additions)
|
foreach (var guid in additions)
|
||||||
{
|
{
|
||||||
if (guid == Guid.Empty)
|
if (guid == Guid.Empty)
|
||||||
{
|
{
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
|
|
||||||
if (existing.Add(guid))
|
if (existing.Add(guid))
|
||||||
{
|
{
|
||||||
newlyAdded.Add(guid);
|
newlyAdded.Add(guid);
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
if (existing.Count > 0)
|
if (existing.Count > 0)
|
||||||
{
|
{
|
||||||
cursor[field] = new BsonArray(existing
|
cursor[field] = new BsonArray(existing
|
||||||
.Select(static g => g.ToString("D"))
|
.Select(static g => g.ToString("D"))
|
||||||
.OrderBy(static s => s, StringComparer.OrdinalIgnoreCase));
|
.OrderBy(static s => s, StringComparer.OrdinalIgnoreCase));
|
||||||
}
|
}
|
||||||
|
|
||||||
return newlyAdded.AsReadOnly();
|
return newlyAdded.AsReadOnly();
|
||||||
}
|
}
|
||||||
|
|
||||||
private static IReadOnlyCollection<string> MergeStringArray(BsonDocument cursor, string field, IReadOnlyCollection<string> additions)
|
private static IReadOnlyCollection<string> MergeStringArray(BsonDocument cursor, string field, IReadOnlyCollection<string> additions)
|
||||||
{
|
{
|
||||||
if (additions.Count == 0)
|
if (additions.Count == 0)
|
||||||
{
|
{
|
||||||
return Array.Empty<string>();
|
return Array.Empty<string>();
|
||||||
}
|
}
|
||||||
|
|
||||||
var existing = cursor.TryGetValue(field, out var value) && value is BsonArray existingArray
|
var existing = cursor.TryGetValue(field, out var value) && value is BsonArray existingArray
|
||||||
? existingArray.Select(static v => v?.AsString ?? string.Empty)
|
? existingArray.Select(static v => v?.AsString ?? string.Empty)
|
||||||
.Where(static s => !string.IsNullOrWhiteSpace(s))
|
.Where(static s => !string.IsNullOrWhiteSpace(s))
|
||||||
.ToHashSet(StringComparer.OrdinalIgnoreCase)
|
.ToHashSet(StringComparer.OrdinalIgnoreCase)
|
||||||
: new HashSet<string>(StringComparer.OrdinalIgnoreCase);
|
: new HashSet<string>(StringComparer.OrdinalIgnoreCase);
|
||||||
|
|
||||||
var newlyAdded = new List<string>();
|
var newlyAdded = new List<string>();
|
||||||
foreach (var entry in additions)
|
foreach (var entry in additions)
|
||||||
{
|
{
|
||||||
if (string.IsNullOrWhiteSpace(entry))
|
if (string.IsNullOrWhiteSpace(entry))
|
||||||
{
|
{
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
|
|
||||||
var normalized = entry.Trim();
|
var normalized = entry.Trim();
|
||||||
if (existing.Add(normalized))
|
if (existing.Add(normalized))
|
||||||
{
|
{
|
||||||
newlyAdded.Add(normalized);
|
newlyAdded.Add(normalized);
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
if (existing.Count > 0)
|
if (existing.Count > 0)
|
||||||
{
|
{
|
||||||
cursor[field] = new BsonArray(existing
|
cursor[field] = new BsonArray(existing
|
||||||
.OrderBy(static s => s, StringComparer.OrdinalIgnoreCase));
|
.OrderBy(static s => s, StringComparer.OrdinalIgnoreCase));
|
||||||
}
|
}
|
||||||
|
|
||||||
return newlyAdded.AsReadOnly();
|
return newlyAdded.AsReadOnly();
|
||||||
}
|
}
|
||||||
|
|
||||||
private static Guid AsGuid(BsonValue value)
|
private static Guid AsGuid(BsonValue value)
|
||||||
{
|
{
|
||||||
if (value is null)
|
if (value is null)
|
||||||
{
|
{
|
||||||
return Guid.Empty;
|
return Guid.Empty;
|
||||||
}
|
}
|
||||||
|
|
||||||
return Guid.TryParse(value.ToString(), out var parsed) ? parsed : Guid.Empty;
|
return Guid.TryParse(value.ToString(), out var parsed) ? parsed : Guid.Empty;
|
||||||
}
|
}
|
||||||
|
|
||||||
private static void AppendRange(HashSet<Guid> target, IReadOnlyCollection<Guid>? values)
|
private static void AppendRange(HashSet<Guid> target, IReadOnlyCollection<Guid>? values)
|
||||||
{
|
{
|
||||||
if (values is null)
|
if (values is null)
|
||||||
{
|
{
|
||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
|
|
||||||
foreach (var guid in values)
|
foreach (var guid in values)
|
||||||
{
|
{
|
||||||
if (guid != Guid.Empty)
|
if (guid != Guid.Empty)
|
||||||
{
|
{
|
||||||
target.Add(guid);
|
target.Add(guid);
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
private static void AppendRange(HashSet<string> target, IReadOnlyCollection<string>? values)
|
private static void AppendRange(HashSet<string> target, IReadOnlyCollection<string>? values)
|
||||||
{
|
{
|
||||||
if (values is null)
|
if (values is null)
|
||||||
{
|
{
|
||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
|
|
||||||
foreach (var value in values)
|
foreach (var value in values)
|
||||||
{
|
{
|
||||||
if (string.IsNullOrWhiteSpace(value))
|
if (string.IsNullOrWhiteSpace(value))
|
||||||
{
|
{
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
|
|
||||||
target.Add(value.Trim());
|
target.Add(value.Trim());
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
}
|
}
|
||||||
|
|||||||
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
@@ -1,434 +1,434 @@
|
|||||||
using System;
|
using System;
|
||||||
using System.Collections.Generic;
|
using System.Collections.Generic;
|
||||||
using System.Globalization;
|
using System.Globalization;
|
||||||
using System.Linq;
|
using System.Linq;
|
||||||
using System.Text.Json;
|
using System.Text.Json;
|
||||||
using Microsoft.Extensions.Logging;
|
using Microsoft.Extensions.Logging;
|
||||||
using Microsoft.Extensions.Options;
|
using Microsoft.Extensions.Options;
|
||||||
using MongoDB.Bson;
|
using MongoDB.Bson;
|
||||||
using MongoDB.Bson.IO;
|
using MongoDB.Bson.IO;
|
||||||
using StellaOps.Concelier.Models;
|
using StellaOps.Concelier.Models;
|
||||||
using StellaOps.Concelier.Connector.Common;
|
using StellaOps.Concelier.Connector.Common;
|
||||||
using StellaOps.Concelier.Connector.Common.Fetch;
|
using StellaOps.Concelier.Connector.Common.Fetch;
|
||||||
using StellaOps.Concelier.Connector.Distro.RedHat.Configuration;
|
using StellaOps.Concelier.Connector.Distro.RedHat.Configuration;
|
||||||
using StellaOps.Concelier.Connector.Distro.RedHat.Internal;
|
using StellaOps.Concelier.Connector.Distro.RedHat.Internal;
|
||||||
using StellaOps.Concelier.Storage.Mongo;
|
using StellaOps.Concelier.Storage.Mongo;
|
||||||
using StellaOps.Concelier.Storage.Mongo.Advisories;
|
using StellaOps.Concelier.Storage.Mongo.Advisories;
|
||||||
using StellaOps.Concelier.Storage.Mongo.Documents;
|
using StellaOps.Concelier.Storage.Mongo.Documents;
|
||||||
using StellaOps.Concelier.Storage.Mongo.Dtos;
|
using StellaOps.Concelier.Storage.Mongo.Dtos;
|
||||||
using StellaOps.Plugin;
|
using StellaOps.Plugin;
|
||||||
|
|
||||||
namespace StellaOps.Concelier.Connector.Distro.RedHat;
|
namespace StellaOps.Concelier.Connector.Distro.RedHat;
|
||||||
|
|
||||||
public sealed class RedHatConnector : IFeedConnector
|
public sealed class RedHatConnector : IFeedConnector
|
||||||
{
|
{
|
||||||
private readonly SourceFetchService _fetchService;
|
private readonly SourceFetchService _fetchService;
|
||||||
private readonly RawDocumentStorage _rawDocumentStorage;
|
private readonly RawDocumentStorage _rawDocumentStorage;
|
||||||
private readonly IDocumentStore _documentStore;
|
private readonly IDocumentStore _documentStore;
|
||||||
private readonly IDtoStore _dtoStore;
|
private readonly IDtoStore _dtoStore;
|
||||||
private readonly IAdvisoryStore _advisoryStore;
|
private readonly IAdvisoryStore _advisoryStore;
|
||||||
private readonly ISourceStateRepository _stateRepository;
|
private readonly ISourceStateRepository _stateRepository;
|
||||||
private readonly ILogger<RedHatConnector> _logger;
|
private readonly ILogger<RedHatConnector> _logger;
|
||||||
private readonly RedHatOptions _options;
|
private readonly RedHatOptions _options;
|
||||||
private readonly TimeProvider _timeProvider;
|
private readonly TimeProvider _timeProvider;
|
||||||
|
|
||||||
public RedHatConnector(
|
public RedHatConnector(
|
||||||
SourceFetchService fetchService,
|
SourceFetchService fetchService,
|
||||||
RawDocumentStorage rawDocumentStorage,
|
RawDocumentStorage rawDocumentStorage,
|
||||||
IDocumentStore documentStore,
|
IDocumentStore documentStore,
|
||||||
IDtoStore dtoStore,
|
IDtoStore dtoStore,
|
||||||
IAdvisoryStore advisoryStore,
|
IAdvisoryStore advisoryStore,
|
||||||
ISourceStateRepository stateRepository,
|
ISourceStateRepository stateRepository,
|
||||||
IOptions<RedHatOptions> options,
|
IOptions<RedHatOptions> options,
|
||||||
TimeProvider? timeProvider,
|
TimeProvider? timeProvider,
|
||||||
ILogger<RedHatConnector> logger)
|
ILogger<RedHatConnector> logger)
|
||||||
{
|
{
|
||||||
_fetchService = fetchService ?? throw new ArgumentNullException(nameof(fetchService));
|
_fetchService = fetchService ?? throw new ArgumentNullException(nameof(fetchService));
|
||||||
_rawDocumentStorage = rawDocumentStorage ?? throw new ArgumentNullException(nameof(rawDocumentStorage));
|
_rawDocumentStorage = rawDocumentStorage ?? throw new ArgumentNullException(nameof(rawDocumentStorage));
|
||||||
_documentStore = documentStore ?? throw new ArgumentNullException(nameof(documentStore));
|
_documentStore = documentStore ?? throw new ArgumentNullException(nameof(documentStore));
|
||||||
_dtoStore = dtoStore ?? throw new ArgumentNullException(nameof(dtoStore));
|
_dtoStore = dtoStore ?? throw new ArgumentNullException(nameof(dtoStore));
|
||||||
_advisoryStore = advisoryStore ?? throw new ArgumentNullException(nameof(advisoryStore));
|
_advisoryStore = advisoryStore ?? throw new ArgumentNullException(nameof(advisoryStore));
|
||||||
_stateRepository = stateRepository ?? throw new ArgumentNullException(nameof(stateRepository));
|
_stateRepository = stateRepository ?? throw new ArgumentNullException(nameof(stateRepository));
|
||||||
_options = options?.Value ?? throw new ArgumentNullException(nameof(options));
|
_options = options?.Value ?? throw new ArgumentNullException(nameof(options));
|
||||||
_options.Validate();
|
_options.Validate();
|
||||||
_timeProvider = timeProvider ?? TimeProvider.System;
|
_timeProvider = timeProvider ?? TimeProvider.System;
|
||||||
_logger = logger ?? throw new ArgumentNullException(nameof(logger));
|
_logger = logger ?? throw new ArgumentNullException(nameof(logger));
|
||||||
}
|
}
|
||||||
|
|
||||||
public string SourceName => RedHatConnectorPlugin.SourceName;
|
public string SourceName => RedHatConnectorPlugin.SourceName;
|
||||||
|
|
||||||
public async Task FetchAsync(IServiceProvider services, CancellationToken cancellationToken)
|
public async Task FetchAsync(IServiceProvider services, CancellationToken cancellationToken)
|
||||||
{
|
{
|
||||||
ArgumentNullException.ThrowIfNull(services);
|
ArgumentNullException.ThrowIfNull(services);
|
||||||
|
|
||||||
var cursor = await GetCursorAsync(cancellationToken).ConfigureAwait(false);
|
var cursor = await GetCursorAsync(cancellationToken).ConfigureAwait(false);
|
||||||
var now = _timeProvider.GetUtcNow();
|
var now = _timeProvider.GetUtcNow();
|
||||||
|
|
||||||
var baseline = cursor.LastReleasedOn ?? now - _options.InitialBackfill;
|
var baseline = cursor.LastReleasedOn ?? now - _options.InitialBackfill;
|
||||||
var overlap = _options.Overlap > TimeSpan.Zero ? _options.Overlap : TimeSpan.Zero;
|
var overlap = _options.Overlap > TimeSpan.Zero ? _options.Overlap : TimeSpan.Zero;
|
||||||
var afterThreshold = baseline - overlap;
|
var afterThreshold = baseline - overlap;
|
||||||
if (afterThreshold < DateTimeOffset.UnixEpoch)
|
if (afterThreshold < DateTimeOffset.UnixEpoch)
|
||||||
{
|
{
|
||||||
afterThreshold = DateTimeOffset.UnixEpoch;
|
afterThreshold = DateTimeOffset.UnixEpoch;
|
||||||
}
|
}
|
||||||
|
|
||||||
ProvenanceDiagnostics.ReportResumeWindow(SourceName, afterThreshold, _logger);
|
ProvenanceDiagnostics.ReportResumeWindow(SourceName, afterThreshold, _logger);
|
||||||
|
|
||||||
var processedSet = new HashSet<string>(cursor.ProcessedAdvisoryIds, StringComparer.OrdinalIgnoreCase);
|
var processedSet = new HashSet<string>(cursor.ProcessedAdvisoryIds, StringComparer.OrdinalIgnoreCase);
|
||||||
var newSummaries = new List<RedHatSummaryItem>();
|
var newSummaries = new List<RedHatSummaryItem>();
|
||||||
var stopDueToOlderData = false;
|
var stopDueToOlderData = false;
|
||||||
var touchedResources = new HashSet<string>(StringComparer.OrdinalIgnoreCase);
|
var touchedResources = new HashSet<string>(StringComparer.OrdinalIgnoreCase);
|
||||||
|
|
||||||
for (var page = 1; page <= _options.MaxPagesPerFetch; page++)
|
for (var page = 1; page <= _options.MaxPagesPerFetch; page++)
|
||||||
{
|
{
|
||||||
var summaryUri = BuildSummaryUri(afterThreshold, page);
|
var summaryUri = BuildSummaryUri(afterThreshold, page);
|
||||||
var summaryKey = summaryUri.ToString();
|
var summaryKey = summaryUri.ToString();
|
||||||
touchedResources.Add(summaryKey);
|
touchedResources.Add(summaryKey);
|
||||||
|
|
||||||
var cachedSummary = cursor.TryGetFetchCache(summaryKey);
|
var cachedSummary = cursor.TryGetFetchCache(summaryKey);
|
||||||
var summaryMetadata = new Dictionary<string, string>(StringComparer.Ordinal)
|
var summaryMetadata = new Dictionary<string, string>(StringComparer.Ordinal)
|
||||||
{
|
{
|
||||||
["page"] = page.ToString(CultureInfo.InvariantCulture),
|
["page"] = page.ToString(CultureInfo.InvariantCulture),
|
||||||
["type"] = "summary"
|
["type"] = "summary"
|
||||||
};
|
};
|
||||||
|
|
||||||
var summaryRequest = new SourceFetchRequest(RedHatOptions.HttpClientName, SourceName, summaryUri)
|
var summaryRequest = new SourceFetchRequest(RedHatOptions.HttpClientName, SourceName, summaryUri)
|
||||||
{
|
{
|
||||||
Metadata = summaryMetadata,
|
Metadata = summaryMetadata,
|
||||||
ETag = cachedSummary?.ETag,
|
ETag = cachedSummary?.ETag,
|
||||||
LastModified = cachedSummary?.LastModified,
|
LastModified = cachedSummary?.LastModified,
|
||||||
TimeoutOverride = _options.FetchTimeout,
|
TimeoutOverride = _options.FetchTimeout,
|
||||||
};
|
};
|
||||||
|
|
||||||
SourceFetchContentResult summaryResult;
|
SourceFetchContentResult summaryResult;
|
||||||
try
|
try
|
||||||
{
|
{
|
||||||
summaryResult = await _fetchService.FetchContentAsync(summaryRequest, cancellationToken).ConfigureAwait(false);
|
summaryResult = await _fetchService.FetchContentAsync(summaryRequest, cancellationToken).ConfigureAwait(false);
|
||||||
}
|
}
|
||||||
catch (Exception ex)
|
catch (Exception ex)
|
||||||
{
|
{
|
||||||
_logger.LogError(ex, "Red Hat Hydra summary fetch failed for {Uri}", summaryUri);
|
_logger.LogError(ex, "Red Hat Hydra summary fetch failed for {Uri}", summaryUri);
|
||||||
throw;
|
throw;
|
||||||
}
|
}
|
||||||
|
|
||||||
if (summaryResult.IsNotModified)
|
if (summaryResult.IsNotModified)
|
||||||
{
|
{
|
||||||
if (page == 1)
|
if (page == 1)
|
||||||
{
|
{
|
||||||
break;
|
break;
|
||||||
}
|
}
|
||||||
|
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
|
|
||||||
if (!summaryResult.IsSuccess || summaryResult.Content is null)
|
if (!summaryResult.IsSuccess || summaryResult.Content is null)
|
||||||
{
|
{
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
|
|
||||||
cursor = cursor.WithFetchCache(summaryKey, summaryResult.ETag, summaryResult.LastModified);
|
cursor = cursor.WithFetchCache(summaryKey, summaryResult.ETag, summaryResult.LastModified);
|
||||||
|
|
||||||
using var document = JsonDocument.Parse(summaryResult.Content);
|
using var document = JsonDocument.Parse(summaryResult.Content);
|
||||||
|
|
||||||
if (document.RootElement.ValueKind != JsonValueKind.Array)
|
if (document.RootElement.ValueKind != JsonValueKind.Array)
|
||||||
{
|
{
|
||||||
_logger.LogWarning(
|
_logger.LogWarning(
|
||||||
"Red Hat Hydra summary response had unexpected payload kind {Kind} for {Uri}",
|
"Red Hat Hydra summary response had unexpected payload kind {Kind} for {Uri}",
|
||||||
document.RootElement.ValueKind,
|
document.RootElement.ValueKind,
|
||||||
summaryUri);
|
summaryUri);
|
||||||
break;
|
break;
|
||||||
}
|
}
|
||||||
|
|
||||||
var pageCount = 0;
|
var pageCount = 0;
|
||||||
foreach (var element in document.RootElement.EnumerateArray())
|
foreach (var element in document.RootElement.EnumerateArray())
|
||||||
{
|
{
|
||||||
if (!RedHatSummaryItem.TryParse(element, out var summary))
|
if (!RedHatSummaryItem.TryParse(element, out var summary))
|
||||||
{
|
{
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
|
|
||||||
pageCount++;
|
pageCount++;
|
||||||
|
|
||||||
if (cursor.LastReleasedOn.HasValue)
|
if (cursor.LastReleasedOn.HasValue)
|
||||||
{
|
{
|
||||||
if (summary.ReleasedOn < cursor.LastReleasedOn.Value - overlap)
|
if (summary.ReleasedOn < cursor.LastReleasedOn.Value - overlap)
|
||||||
{
|
{
|
||||||
stopDueToOlderData = true;
|
stopDueToOlderData = true;
|
||||||
break;
|
break;
|
||||||
}
|
}
|
||||||
|
|
||||||
if (summary.ReleasedOn < cursor.LastReleasedOn.Value)
|
if (summary.ReleasedOn < cursor.LastReleasedOn.Value)
|
||||||
{
|
{
|
||||||
stopDueToOlderData = true;
|
stopDueToOlderData = true;
|
||||||
break;
|
break;
|
||||||
}
|
}
|
||||||
|
|
||||||
if (summary.ReleasedOn == cursor.LastReleasedOn.Value && processedSet.Contains(summary.AdvisoryId))
|
if (summary.ReleasedOn == cursor.LastReleasedOn.Value && processedSet.Contains(summary.AdvisoryId))
|
||||||
{
|
{
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
newSummaries.Add(summary);
|
newSummaries.Add(summary);
|
||||||
processedSet.Add(summary.AdvisoryId);
|
processedSet.Add(summary.AdvisoryId);
|
||||||
|
|
||||||
if (newSummaries.Count >= _options.MaxAdvisoriesPerFetch)
|
if (newSummaries.Count >= _options.MaxAdvisoriesPerFetch)
|
||||||
{
|
{
|
||||||
break;
|
break;
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
if (newSummaries.Count >= _options.MaxAdvisoriesPerFetch || stopDueToOlderData)
|
if (newSummaries.Count >= _options.MaxAdvisoriesPerFetch || stopDueToOlderData)
|
||||||
{
|
{
|
||||||
break;
|
break;
|
||||||
}
|
}
|
||||||
|
|
||||||
if (pageCount < _options.PageSize)
|
if (pageCount < _options.PageSize)
|
||||||
{
|
{
|
||||||
break;
|
break;
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
if (newSummaries.Count == 0)
|
if (newSummaries.Count == 0)
|
||||||
{
|
{
|
||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
|
|
||||||
newSummaries.Sort(static (left, right) =>
|
newSummaries.Sort(static (left, right) =>
|
||||||
{
|
{
|
||||||
var compare = left.ReleasedOn.CompareTo(right.ReleasedOn);
|
var compare = left.ReleasedOn.CompareTo(right.ReleasedOn);
|
||||||
return compare != 0
|
return compare != 0
|
||||||
? compare
|
? compare
|
||||||
: string.CompareOrdinal(left.AdvisoryId, right.AdvisoryId);
|
: string.CompareOrdinal(left.AdvisoryId, right.AdvisoryId);
|
||||||
});
|
});
|
||||||
|
|
||||||
var pendingDocuments = new HashSet<Guid>(cursor.PendingDocuments);
|
var pendingDocuments = new HashSet<Guid>(cursor.PendingDocuments);
|
||||||
|
|
||||||
foreach (var summary in newSummaries)
|
foreach (var summary in newSummaries)
|
||||||
{
|
{
|
||||||
var resourceUri = summary.ResourceUri;
|
var resourceUri = summary.ResourceUri;
|
||||||
var resourceKey = resourceUri.ToString();
|
var resourceKey = resourceUri.ToString();
|
||||||
touchedResources.Add(resourceKey);
|
touchedResources.Add(resourceKey);
|
||||||
|
|
||||||
var cached = cursor.TryGetFetchCache(resourceKey);
|
var cached = cursor.TryGetFetchCache(resourceKey);
|
||||||
var metadata = new Dictionary<string, string>(StringComparer.Ordinal)
|
var metadata = new Dictionary<string, string>(StringComparer.Ordinal)
|
||||||
{
|
{
|
||||||
["advisoryId"] = summary.AdvisoryId,
|
["advisoryId"] = summary.AdvisoryId,
|
||||||
["releasedOn"] = summary.ReleasedOn.ToString("O", CultureInfo.InvariantCulture)
|
["releasedOn"] = summary.ReleasedOn.ToString("O", CultureInfo.InvariantCulture)
|
||||||
};
|
};
|
||||||
|
|
||||||
var request = new SourceFetchRequest(RedHatOptions.HttpClientName, SourceName, resourceUri)
|
var request = new SourceFetchRequest(RedHatOptions.HttpClientName, SourceName, resourceUri)
|
||||||
{
|
{
|
||||||
Metadata = metadata,
|
Metadata = metadata,
|
||||||
ETag = cached?.ETag,
|
ETag = cached?.ETag,
|
||||||
LastModified = cached?.LastModified,
|
LastModified = cached?.LastModified,
|
||||||
TimeoutOverride = _options.FetchTimeout,
|
TimeoutOverride = _options.FetchTimeout,
|
||||||
};
|
};
|
||||||
|
|
||||||
try
|
try
|
||||||
{
|
{
|
||||||
var result = await _fetchService.FetchAsync(request, cancellationToken).ConfigureAwait(false);
|
var result = await _fetchService.FetchAsync(request, cancellationToken).ConfigureAwait(false);
|
||||||
if (result.IsNotModified)
|
if (result.IsNotModified)
|
||||||
{
|
{
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
|
|
||||||
if (!result.IsSuccess || result.Document is null)
|
if (!result.IsSuccess || result.Document is null)
|
||||||
{
|
{
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
|
|
||||||
pendingDocuments.Add(result.Document.Id);
|
pendingDocuments.Add(result.Document.Id);
|
||||||
cursor = cursor.WithFetchCache(resourceKey, result.Document.Etag, result.Document.LastModified);
|
cursor = cursor.WithFetchCache(resourceKey, result.Document.Etag, result.Document.LastModified);
|
||||||
}
|
}
|
||||||
catch (Exception ex)
|
catch (Exception ex)
|
||||||
{
|
{
|
||||||
_logger.LogError(ex, "Red Hat Hydra advisory fetch failed for {Uri}", resourceUri);
|
_logger.LogError(ex, "Red Hat Hydra advisory fetch failed for {Uri}", resourceUri);
|
||||||
throw;
|
throw;
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
var maxRelease = newSummaries.Max(static item => item.ReleasedOn);
|
var maxRelease = newSummaries.Max(static item => item.ReleasedOn);
|
||||||
var idsForMaxRelease = newSummaries
|
var idsForMaxRelease = newSummaries
|
||||||
.Where(item => item.ReleasedOn == maxRelease)
|
.Where(item => item.ReleasedOn == maxRelease)
|
||||||
.Select(item => item.AdvisoryId)
|
.Select(item => item.AdvisoryId)
|
||||||
.Distinct(StringComparer.OrdinalIgnoreCase)
|
.Distinct(StringComparer.OrdinalIgnoreCase)
|
||||||
.ToArray();
|
.ToArray();
|
||||||
|
|
||||||
RedHatCursor updated;
|
RedHatCursor updated;
|
||||||
if (cursor.LastReleasedOn.HasValue && maxRelease == cursor.LastReleasedOn.Value)
|
if (cursor.LastReleasedOn.HasValue && maxRelease == cursor.LastReleasedOn.Value)
|
||||||
{
|
{
|
||||||
updated = cursor
|
updated = cursor
|
||||||
.WithPendingDocuments(pendingDocuments)
|
.WithPendingDocuments(pendingDocuments)
|
||||||
.AddProcessedAdvisories(idsForMaxRelease)
|
.AddProcessedAdvisories(idsForMaxRelease)
|
||||||
.PruneFetchCache(touchedResources);
|
.PruneFetchCache(touchedResources);
|
||||||
}
|
}
|
||||||
else
|
else
|
||||||
{
|
{
|
||||||
updated = cursor
|
updated = cursor
|
||||||
.WithPendingDocuments(pendingDocuments)
|
.WithPendingDocuments(pendingDocuments)
|
||||||
.WithLastReleased(maxRelease, idsForMaxRelease)
|
.WithLastReleased(maxRelease, idsForMaxRelease)
|
||||||
.PruneFetchCache(touchedResources);
|
.PruneFetchCache(touchedResources);
|
||||||
}
|
}
|
||||||
|
|
||||||
await UpdateCursorAsync(updated, cancellationToken).ConfigureAwait(false);
|
await UpdateCursorAsync(updated, cancellationToken).ConfigureAwait(false);
|
||||||
}
|
}
|
||||||
|
|
||||||
public async Task ParseAsync(IServiceProvider services, CancellationToken cancellationToken)
|
public async Task ParseAsync(IServiceProvider services, CancellationToken cancellationToken)
|
||||||
{
|
{
|
||||||
ArgumentNullException.ThrowIfNull(services);
|
ArgumentNullException.ThrowIfNull(services);
|
||||||
|
|
||||||
var cursor = await GetCursorAsync(cancellationToken).ConfigureAwait(false);
|
var cursor = await GetCursorAsync(cancellationToken).ConfigureAwait(false);
|
||||||
if (cursor.PendingDocuments.Count == 0)
|
if (cursor.PendingDocuments.Count == 0)
|
||||||
{
|
{
|
||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
|
|
||||||
var remainingFetch = cursor.PendingDocuments.ToList();
|
var remainingFetch = cursor.PendingDocuments.ToList();
|
||||||
var pendingMappings = cursor.PendingMappings.ToList();
|
var pendingMappings = cursor.PendingMappings.ToList();
|
||||||
|
|
||||||
foreach (var documentId in cursor.PendingDocuments)
|
foreach (var documentId in cursor.PendingDocuments)
|
||||||
{
|
{
|
||||||
DocumentRecord? document = null;
|
DocumentRecord? document = null;
|
||||||
|
|
||||||
try
|
try
|
||||||
{
|
{
|
||||||
document = await _documentStore.FindAsync(documentId, cancellationToken).ConfigureAwait(false);
|
document = await _documentStore.FindAsync(documentId, cancellationToken).ConfigureAwait(false);
|
||||||
if (document is null)
|
if (document is null)
|
||||||
{
|
{
|
||||||
remainingFetch.Remove(documentId);
|
remainingFetch.Remove(documentId);
|
||||||
pendingMappings.Remove(documentId);
|
pendingMappings.Remove(documentId);
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
|
|
||||||
if (!document.GridFsId.HasValue)
|
if (!document.PayloadId.HasValue)
|
||||||
{
|
{
|
||||||
_logger.LogWarning("Red Hat document {DocumentId} missing GridFS content; skipping", document.Id);
|
_logger.LogWarning("Red Hat document {DocumentId} missing GridFS content; skipping", document.Id);
|
||||||
remainingFetch.Remove(documentId);
|
remainingFetch.Remove(documentId);
|
||||||
pendingMappings.Remove(documentId);
|
pendingMappings.Remove(documentId);
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
|
|
||||||
var rawBytes = await _rawDocumentStorage.DownloadAsync(document.GridFsId.Value, cancellationToken).ConfigureAwait(false);
|
var rawBytes = await _rawDocumentStorage.DownloadAsync(document.PayloadId.Value, cancellationToken).ConfigureAwait(false);
|
||||||
using var jsonDocument = JsonDocument.Parse(rawBytes);
|
using var jsonDocument = JsonDocument.Parse(rawBytes);
|
||||||
var sanitized = JsonSerializer.Serialize(jsonDocument.RootElement);
|
var sanitized = JsonSerializer.Serialize(jsonDocument.RootElement);
|
||||||
var payload = BsonDocument.Parse(sanitized);
|
var payload = BsonDocument.Parse(sanitized);
|
||||||
|
|
||||||
var dtoRecord = new DtoRecord(
|
var dtoRecord = new DtoRecord(
|
||||||
Guid.NewGuid(),
|
Guid.NewGuid(),
|
||||||
document.Id,
|
document.Id,
|
||||||
SourceName,
|
SourceName,
|
||||||
"redhat.csaf.v2",
|
"redhat.csaf.v2",
|
||||||
payload,
|
payload,
|
||||||
_timeProvider.GetUtcNow());
|
_timeProvider.GetUtcNow());
|
||||||
|
|
||||||
await _dtoStore.UpsertAsync(dtoRecord, cancellationToken).ConfigureAwait(false);
|
await _dtoStore.UpsertAsync(dtoRecord, cancellationToken).ConfigureAwait(false);
|
||||||
await _documentStore.UpdateStatusAsync(document.Id, DocumentStatuses.PendingMap, cancellationToken).ConfigureAwait(false);
|
await _documentStore.UpdateStatusAsync(document.Id, DocumentStatuses.PendingMap, cancellationToken).ConfigureAwait(false);
|
||||||
|
|
||||||
remainingFetch.Remove(documentId);
|
remainingFetch.Remove(documentId);
|
||||||
if (!pendingMappings.Contains(documentId))
|
if (!pendingMappings.Contains(documentId))
|
||||||
{
|
{
|
||||||
pendingMappings.Add(documentId);
|
pendingMappings.Add(documentId);
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
catch (Exception ex)
|
catch (Exception ex)
|
||||||
{
|
{
|
||||||
var uri = document?.Uri ?? documentId.ToString();
|
var uri = document?.Uri ?? documentId.ToString();
|
||||||
_logger.LogError(ex, "Red Hat CSAF parse failed for {Uri}", uri);
|
_logger.LogError(ex, "Red Hat CSAF parse failed for {Uri}", uri);
|
||||||
remainingFetch.Remove(documentId);
|
remainingFetch.Remove(documentId);
|
||||||
pendingMappings.Remove(documentId);
|
pendingMappings.Remove(documentId);
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
var updatedCursor = cursor
|
var updatedCursor = cursor
|
||||||
.WithPendingDocuments(remainingFetch)
|
.WithPendingDocuments(remainingFetch)
|
||||||
.WithPendingMappings(pendingMappings);
|
.WithPendingMappings(pendingMappings);
|
||||||
|
|
||||||
await UpdateCursorAsync(updatedCursor, cancellationToken).ConfigureAwait(false);
|
await UpdateCursorAsync(updatedCursor, cancellationToken).ConfigureAwait(false);
|
||||||
}
|
}
|
||||||
|
|
||||||
public async Task MapAsync(IServiceProvider services, CancellationToken cancellationToken)
|
public async Task MapAsync(IServiceProvider services, CancellationToken cancellationToken)
|
||||||
{
|
{
|
||||||
ArgumentNullException.ThrowIfNull(services);
|
ArgumentNullException.ThrowIfNull(services);
|
||||||
|
|
||||||
var cursor = await GetCursorAsync(cancellationToken).ConfigureAwait(false);
|
var cursor = await GetCursorAsync(cancellationToken).ConfigureAwait(false);
|
||||||
if (cursor.PendingMappings.Count == 0)
|
if (cursor.PendingMappings.Count == 0)
|
||||||
{
|
{
|
||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
|
|
||||||
var pendingMappings = cursor.PendingMappings.ToList();
|
var pendingMappings = cursor.PendingMappings.ToList();
|
||||||
|
|
||||||
foreach (var documentId in cursor.PendingMappings)
|
foreach (var documentId in cursor.PendingMappings)
|
||||||
{
|
{
|
||||||
try
|
try
|
||||||
{
|
{
|
||||||
var dto = await _dtoStore.FindByDocumentIdAsync(documentId, cancellationToken).ConfigureAwait(false);
|
var dto = await _dtoStore.FindByDocumentIdAsync(documentId, cancellationToken).ConfigureAwait(false);
|
||||||
var document = await _documentStore.FindAsync(documentId, cancellationToken).ConfigureAwait(false);
|
var document = await _documentStore.FindAsync(documentId, cancellationToken).ConfigureAwait(false);
|
||||||
|
|
||||||
if (dto is null || document is null)
|
if (dto is null || document is null)
|
||||||
{
|
{
|
||||||
pendingMappings.Remove(documentId);
|
pendingMappings.Remove(documentId);
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
|
|
||||||
var json = dto.Payload.ToJson(new JsonWriterSettings
|
var json = dto.Payload.ToJson(new JsonWriterSettings
|
||||||
{
|
{
|
||||||
OutputMode = JsonOutputMode.RelaxedExtendedJson,
|
OutputMode = JsonOutputMode.RelaxedExtendedJson,
|
||||||
});
|
});
|
||||||
|
|
||||||
using var jsonDocument = JsonDocument.Parse(json);
|
using var jsonDocument = JsonDocument.Parse(json);
|
||||||
var advisory = RedHatMapper.Map(SourceName, dto, document, jsonDocument);
|
var advisory = RedHatMapper.Map(SourceName, dto, document, jsonDocument);
|
||||||
if (advisory is null)
|
if (advisory is null)
|
||||||
{
|
{
|
||||||
pendingMappings.Remove(documentId);
|
pendingMappings.Remove(documentId);
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
|
|
||||||
await _advisoryStore.UpsertAsync(advisory, cancellationToken).ConfigureAwait(false);
|
await _advisoryStore.UpsertAsync(advisory, cancellationToken).ConfigureAwait(false);
|
||||||
await _documentStore.UpdateStatusAsync(documentId, DocumentStatuses.Mapped, cancellationToken).ConfigureAwait(false);
|
await _documentStore.UpdateStatusAsync(documentId, DocumentStatuses.Mapped, cancellationToken).ConfigureAwait(false);
|
||||||
pendingMappings.Remove(documentId);
|
pendingMappings.Remove(documentId);
|
||||||
}
|
}
|
||||||
catch (Exception ex)
|
catch (Exception ex)
|
||||||
{
|
{
|
||||||
_logger.LogError(ex, "Red Hat map failed for document {DocumentId}", documentId);
|
_logger.LogError(ex, "Red Hat map failed for document {DocumentId}", documentId);
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
var updatedCursor = cursor.WithPendingMappings(pendingMappings);
|
var updatedCursor = cursor.WithPendingMappings(pendingMappings);
|
||||||
await UpdateCursorAsync(updatedCursor, cancellationToken).ConfigureAwait(false);
|
await UpdateCursorAsync(updatedCursor, cancellationToken).ConfigureAwait(false);
|
||||||
}
|
}
|
||||||
|
|
||||||
private async Task<RedHatCursor> GetCursorAsync(CancellationToken cancellationToken)
|
private async Task<RedHatCursor> GetCursorAsync(CancellationToken cancellationToken)
|
||||||
{
|
{
|
||||||
var record = await _stateRepository.TryGetAsync(SourceName, cancellationToken).ConfigureAwait(false);
|
var record = await _stateRepository.TryGetAsync(SourceName, cancellationToken).ConfigureAwait(false);
|
||||||
return RedHatCursor.FromBsonDocument(record?.Cursor);
|
return RedHatCursor.FromBsonDocument(record?.Cursor);
|
||||||
}
|
}
|
||||||
|
|
||||||
private async Task UpdateCursorAsync(RedHatCursor cursor, CancellationToken cancellationToken)
|
private async Task UpdateCursorAsync(RedHatCursor cursor, CancellationToken cancellationToken)
|
||||||
{
|
{
|
||||||
var completedAt = _timeProvider.GetUtcNow();
|
var completedAt = _timeProvider.GetUtcNow();
|
||||||
await _stateRepository.UpdateCursorAsync(SourceName, cursor.ToBsonDocument(), completedAt, cancellationToken).ConfigureAwait(false);
|
await _stateRepository.UpdateCursorAsync(SourceName, cursor.ToBsonDocument(), completedAt, cancellationToken).ConfigureAwait(false);
|
||||||
}
|
}
|
||||||
|
|
||||||
private Uri BuildSummaryUri(DateTimeOffset after, int page)
|
private Uri BuildSummaryUri(DateTimeOffset after, int page)
|
||||||
{
|
{
|
||||||
var builder = new UriBuilder(_options.BaseEndpoint);
|
var builder = new UriBuilder(_options.BaseEndpoint);
|
||||||
var basePath = builder.Path?.TrimEnd('/') ?? string.Empty;
|
var basePath = builder.Path?.TrimEnd('/') ?? string.Empty;
|
||||||
var summaryPath = _options.SummaryPath.TrimStart('/');
|
var summaryPath = _options.SummaryPath.TrimStart('/');
|
||||||
builder.Path = string.IsNullOrEmpty(basePath)
|
builder.Path = string.IsNullOrEmpty(basePath)
|
||||||
? $"/{summaryPath}"
|
? $"/{summaryPath}"
|
||||||
: $"{basePath}/{summaryPath}";
|
: $"{basePath}/{summaryPath}";
|
||||||
|
|
||||||
var parameters = new Dictionary<string, string>(StringComparer.Ordinal)
|
var parameters = new Dictionary<string, string>(StringComparer.Ordinal)
|
||||||
{
|
{
|
||||||
["after"] = after.ToString("yyyy-MM-dd", CultureInfo.InvariantCulture),
|
["after"] = after.ToString("yyyy-MM-dd", CultureInfo.InvariantCulture),
|
||||||
["per_page"] = _options.PageSize.ToString(CultureInfo.InvariantCulture),
|
["per_page"] = _options.PageSize.ToString(CultureInfo.InvariantCulture),
|
||||||
["page"] = page.ToString(CultureInfo.InvariantCulture)
|
["page"] = page.ToString(CultureInfo.InvariantCulture)
|
||||||
};
|
};
|
||||||
|
|
||||||
builder.Query = string.Join('&', parameters.Select(static kvp =>
|
builder.Query = string.Join('&', parameters.Select(static kvp =>
|
||||||
$"{Uri.EscapeDataString(kvp.Key)}={Uri.EscapeDataString(kvp.Value)}"));
|
$"{Uri.EscapeDataString(kvp.Key)}={Uri.EscapeDataString(kvp.Value)}"));
|
||||||
return builder.Uri;
|
return builder.Uri;
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
@@ -1,384 +1,384 @@
|
|||||||
using System;
|
using System;
|
||||||
using System.Collections.Generic;
|
using System.Collections.Generic;
|
||||||
using System.Linq;
|
using System.Linq;
|
||||||
using System.Text.Json;
|
using System.Text.Json;
|
||||||
using System.Threading;
|
using System.Threading;
|
||||||
using System.Threading.Tasks;
|
using System.Threading.Tasks;
|
||||||
using Microsoft.Extensions.Logging;
|
using Microsoft.Extensions.Logging;
|
||||||
using Microsoft.Extensions.Options;
|
using Microsoft.Extensions.Options;
|
||||||
using MongoDB.Bson;
|
using MongoDB.Bson;
|
||||||
using StellaOps.Concelier.Models;
|
using StellaOps.Concelier.Models;
|
||||||
using StellaOps.Concelier.Connector.Common;
|
using StellaOps.Concelier.Connector.Common;
|
||||||
using StellaOps.Concelier.Connector.Common.Fetch;
|
using StellaOps.Concelier.Connector.Common.Fetch;
|
||||||
using StellaOps.Concelier.Connector.Ics.Kaspersky.Configuration;
|
using StellaOps.Concelier.Connector.Ics.Kaspersky.Configuration;
|
||||||
using StellaOps.Concelier.Connector.Ics.Kaspersky.Internal;
|
using StellaOps.Concelier.Connector.Ics.Kaspersky.Internal;
|
||||||
using StellaOps.Concelier.Storage.Mongo;
|
using StellaOps.Concelier.Storage.Mongo;
|
||||||
using StellaOps.Concelier.Storage.Mongo.Advisories;
|
using StellaOps.Concelier.Storage.Mongo.Advisories;
|
||||||
using StellaOps.Concelier.Storage.Mongo.Documents;
|
using StellaOps.Concelier.Storage.Mongo.Documents;
|
||||||
using StellaOps.Concelier.Storage.Mongo.Dtos;
|
using StellaOps.Concelier.Storage.Mongo.Dtos;
|
||||||
using StellaOps.Plugin;
|
using StellaOps.Plugin;
|
||||||
|
|
||||||
namespace StellaOps.Concelier.Connector.Ics.Kaspersky;
|
namespace StellaOps.Concelier.Connector.Ics.Kaspersky;
|
||||||
|
|
||||||
public sealed class KasperskyConnector : IFeedConnector
|
public sealed class KasperskyConnector : IFeedConnector
|
||||||
{
|
{
|
||||||
private static readonly JsonSerializerOptions SerializerOptions = new(JsonSerializerDefaults.General)
|
private static readonly JsonSerializerOptions SerializerOptions = new(JsonSerializerDefaults.General)
|
||||||
{
|
{
|
||||||
PropertyNamingPolicy = JsonNamingPolicy.CamelCase,
|
PropertyNamingPolicy = JsonNamingPolicy.CamelCase,
|
||||||
WriteIndented = false,
|
WriteIndented = false,
|
||||||
DefaultIgnoreCondition = System.Text.Json.Serialization.JsonIgnoreCondition.WhenWritingNull,
|
DefaultIgnoreCondition = System.Text.Json.Serialization.JsonIgnoreCondition.WhenWritingNull,
|
||||||
};
|
};
|
||||||
|
|
||||||
private readonly KasperskyFeedClient _feedClient;
|
private readonly KasperskyFeedClient _feedClient;
|
||||||
private readonly SourceFetchService _fetchService;
|
private readonly SourceFetchService _fetchService;
|
||||||
private readonly RawDocumentStorage _rawDocumentStorage;
|
private readonly RawDocumentStorage _rawDocumentStorage;
|
||||||
private readonly IDocumentStore _documentStore;
|
private readonly IDocumentStore _documentStore;
|
||||||
private readonly IDtoStore _dtoStore;
|
private readonly IDtoStore _dtoStore;
|
||||||
private readonly IAdvisoryStore _advisoryStore;
|
private readonly IAdvisoryStore _advisoryStore;
|
||||||
private readonly ISourceStateRepository _stateRepository;
|
private readonly ISourceStateRepository _stateRepository;
|
||||||
private readonly KasperskyOptions _options;
|
private readonly KasperskyOptions _options;
|
||||||
private readonly TimeProvider _timeProvider;
|
private readonly TimeProvider _timeProvider;
|
||||||
private readonly ILogger<KasperskyConnector> _logger;
|
private readonly ILogger<KasperskyConnector> _logger;
|
||||||
|
|
||||||
public KasperskyConnector(
|
public KasperskyConnector(
|
||||||
KasperskyFeedClient feedClient,
|
KasperskyFeedClient feedClient,
|
||||||
SourceFetchService fetchService,
|
SourceFetchService fetchService,
|
||||||
RawDocumentStorage rawDocumentStorage,
|
RawDocumentStorage rawDocumentStorage,
|
||||||
IDocumentStore documentStore,
|
IDocumentStore documentStore,
|
||||||
IDtoStore dtoStore,
|
IDtoStore dtoStore,
|
||||||
IAdvisoryStore advisoryStore,
|
IAdvisoryStore advisoryStore,
|
||||||
ISourceStateRepository stateRepository,
|
ISourceStateRepository stateRepository,
|
||||||
IOptions<KasperskyOptions> options,
|
IOptions<KasperskyOptions> options,
|
||||||
TimeProvider? timeProvider,
|
TimeProvider? timeProvider,
|
||||||
ILogger<KasperskyConnector> logger)
|
ILogger<KasperskyConnector> logger)
|
||||||
{
|
{
|
||||||
_feedClient = feedClient ?? throw new ArgumentNullException(nameof(feedClient));
|
_feedClient = feedClient ?? throw new ArgumentNullException(nameof(feedClient));
|
||||||
_fetchService = fetchService ?? throw new ArgumentNullException(nameof(fetchService));
|
_fetchService = fetchService ?? throw new ArgumentNullException(nameof(fetchService));
|
||||||
_rawDocumentStorage = rawDocumentStorage ?? throw new ArgumentNullException(nameof(rawDocumentStorage));
|
_rawDocumentStorage = rawDocumentStorage ?? throw new ArgumentNullException(nameof(rawDocumentStorage));
|
||||||
_documentStore = documentStore ?? throw new ArgumentNullException(nameof(documentStore));
|
_documentStore = documentStore ?? throw new ArgumentNullException(nameof(documentStore));
|
||||||
_dtoStore = dtoStore ?? throw new ArgumentNullException(nameof(dtoStore));
|
_dtoStore = dtoStore ?? throw new ArgumentNullException(nameof(dtoStore));
|
||||||
_advisoryStore = advisoryStore ?? throw new ArgumentNullException(nameof(advisoryStore));
|
_advisoryStore = advisoryStore ?? throw new ArgumentNullException(nameof(advisoryStore));
|
||||||
_stateRepository = stateRepository ?? throw new ArgumentNullException(nameof(stateRepository));
|
_stateRepository = stateRepository ?? throw new ArgumentNullException(nameof(stateRepository));
|
||||||
_options = (options ?? throw new ArgumentNullException(nameof(options))).Value ?? throw new ArgumentNullException(nameof(options));
|
_options = (options ?? throw new ArgumentNullException(nameof(options))).Value ?? throw new ArgumentNullException(nameof(options));
|
||||||
_options.Validate();
|
_options.Validate();
|
||||||
_timeProvider = timeProvider ?? TimeProvider.System;
|
_timeProvider = timeProvider ?? TimeProvider.System;
|
||||||
_logger = logger ?? throw new ArgumentNullException(nameof(logger));
|
_logger = logger ?? throw new ArgumentNullException(nameof(logger));
|
||||||
}
|
}
|
||||||
|
|
||||||
public string SourceName => KasperskyConnectorPlugin.SourceName;
|
public string SourceName => KasperskyConnectorPlugin.SourceName;
|
||||||
|
|
||||||
public async Task FetchAsync(IServiceProvider services, CancellationToken cancellationToken)
|
public async Task FetchAsync(IServiceProvider services, CancellationToken cancellationToken)
|
||||||
{
|
{
|
||||||
ArgumentNullException.ThrowIfNull(services);
|
ArgumentNullException.ThrowIfNull(services);
|
||||||
|
|
||||||
var cursor = await GetCursorAsync(cancellationToken).ConfigureAwait(false);
|
var cursor = await GetCursorAsync(cancellationToken).ConfigureAwait(false);
|
||||||
var now = _timeProvider.GetUtcNow();
|
var now = _timeProvider.GetUtcNow();
|
||||||
|
|
||||||
var windowStart = cursor.LastPublished.HasValue
|
var windowStart = cursor.LastPublished.HasValue
|
||||||
? cursor.LastPublished.Value - _options.WindowOverlap
|
? cursor.LastPublished.Value - _options.WindowOverlap
|
||||||
: now - _options.WindowSize;
|
: now - _options.WindowSize;
|
||||||
|
|
||||||
var pendingDocuments = cursor.PendingDocuments.ToHashSet();
|
var pendingDocuments = cursor.PendingDocuments.ToHashSet();
|
||||||
var maxPublished = cursor.LastPublished ?? DateTimeOffset.MinValue;
|
var maxPublished = cursor.LastPublished ?? DateTimeOffset.MinValue;
|
||||||
var cursorState = cursor;
|
var cursorState = cursor;
|
||||||
var touchedResources = new HashSet<string>(StringComparer.OrdinalIgnoreCase);
|
var touchedResources = new HashSet<string>(StringComparer.OrdinalIgnoreCase);
|
||||||
|
|
||||||
for (var page = 1; page <= _options.MaxPagesPerFetch; page++)
|
for (var page = 1; page <= _options.MaxPagesPerFetch; page++)
|
||||||
{
|
{
|
||||||
IReadOnlyList<KasperskyFeedItem> items;
|
IReadOnlyList<KasperskyFeedItem> items;
|
||||||
try
|
try
|
||||||
{
|
{
|
||||||
items = await _feedClient.GetItemsAsync(page, cancellationToken).ConfigureAwait(false);
|
items = await _feedClient.GetItemsAsync(page, cancellationToken).ConfigureAwait(false);
|
||||||
}
|
}
|
||||||
catch (Exception ex)
|
catch (Exception ex)
|
||||||
{
|
{
|
||||||
_logger.LogError(ex, "Failed to load Kaspersky ICS feed page {Page}", page);
|
_logger.LogError(ex, "Failed to load Kaspersky ICS feed page {Page}", page);
|
||||||
await _stateRepository.MarkFailureAsync(
|
await _stateRepository.MarkFailureAsync(
|
||||||
SourceName,
|
SourceName,
|
||||||
now,
|
now,
|
||||||
TimeSpan.FromMinutes(5),
|
TimeSpan.FromMinutes(5),
|
||||||
ex.Message,
|
ex.Message,
|
||||||
cancellationToken).ConfigureAwait(false);
|
cancellationToken).ConfigureAwait(false);
|
||||||
throw;
|
throw;
|
||||||
}
|
}
|
||||||
if (items.Count == 0)
|
if (items.Count == 0)
|
||||||
{
|
{
|
||||||
break;
|
break;
|
||||||
}
|
}
|
||||||
|
|
||||||
foreach (var item in items)
|
foreach (var item in items)
|
||||||
{
|
{
|
||||||
if (item.Published < windowStart)
|
if (item.Published < windowStart)
|
||||||
{
|
{
|
||||||
page = _options.MaxPagesPerFetch + 1;
|
page = _options.MaxPagesPerFetch + 1;
|
||||||
break;
|
break;
|
||||||
}
|
}
|
||||||
|
|
||||||
if (_options.RequestDelay > TimeSpan.Zero)
|
if (_options.RequestDelay > TimeSpan.Zero)
|
||||||
{
|
{
|
||||||
await Task.Delay(_options.RequestDelay, cancellationToken).ConfigureAwait(false);
|
await Task.Delay(_options.RequestDelay, cancellationToken).ConfigureAwait(false);
|
||||||
}
|
}
|
||||||
|
|
||||||
var metadata = new Dictionary<string, string>(StringComparer.Ordinal)
|
var metadata = new Dictionary<string, string>(StringComparer.Ordinal)
|
||||||
{
|
{
|
||||||
["kaspersky.title"] = item.Title,
|
["kaspersky.title"] = item.Title,
|
||||||
["kaspersky.link"] = item.Link.ToString(),
|
["kaspersky.link"] = item.Link.ToString(),
|
||||||
["kaspersky.published"] = item.Published.ToString("O"),
|
["kaspersky.published"] = item.Published.ToString("O"),
|
||||||
};
|
};
|
||||||
|
|
||||||
if (!string.IsNullOrWhiteSpace(item.Summary))
|
if (!string.IsNullOrWhiteSpace(item.Summary))
|
||||||
{
|
{
|
||||||
metadata["kaspersky.summary"] = item.Summary!;
|
metadata["kaspersky.summary"] = item.Summary!;
|
||||||
}
|
}
|
||||||
|
|
||||||
var slug = ExtractSlug(item.Link);
|
var slug = ExtractSlug(item.Link);
|
||||||
if (!string.IsNullOrWhiteSpace(slug))
|
if (!string.IsNullOrWhiteSpace(slug))
|
||||||
{
|
{
|
||||||
metadata["kaspersky.slug"] = slug;
|
metadata["kaspersky.slug"] = slug;
|
||||||
}
|
}
|
||||||
|
|
||||||
var resourceKey = item.Link.ToString();
|
var resourceKey = item.Link.ToString();
|
||||||
touchedResources.Add(resourceKey);
|
touchedResources.Add(resourceKey);
|
||||||
|
|
||||||
var existing = await _documentStore.FindBySourceAndUriAsync(SourceName, resourceKey, cancellationToken).ConfigureAwait(false);
|
var existing = await _documentStore.FindBySourceAndUriAsync(SourceName, resourceKey, cancellationToken).ConfigureAwait(false);
|
||||||
|
|
||||||
var fetchRequest = new SourceFetchRequest(KasperskyOptions.HttpClientName, SourceName, item.Link)
|
var fetchRequest = new SourceFetchRequest(KasperskyOptions.HttpClientName, SourceName, item.Link)
|
||||||
{
|
{
|
||||||
Metadata = metadata,
|
Metadata = metadata,
|
||||||
};
|
};
|
||||||
|
|
||||||
if (cursorState.TryGetFetchMetadata(resourceKey, out var cachedFetch))
|
if (cursorState.TryGetFetchMetadata(resourceKey, out var cachedFetch))
|
||||||
{
|
{
|
||||||
fetchRequest = fetchRequest with
|
fetchRequest = fetchRequest with
|
||||||
{
|
{
|
||||||
ETag = cachedFetch.ETag,
|
ETag = cachedFetch.ETag,
|
||||||
LastModified = cachedFetch.LastModified,
|
LastModified = cachedFetch.LastModified,
|
||||||
};
|
};
|
||||||
}
|
}
|
||||||
|
|
||||||
SourceFetchResult result;
|
SourceFetchResult result;
|
||||||
try
|
try
|
||||||
{
|
{
|
||||||
result = await _fetchService.FetchAsync(fetchRequest, cancellationToken).ConfigureAwait(false);
|
result = await _fetchService.FetchAsync(fetchRequest, cancellationToken).ConfigureAwait(false);
|
||||||
}
|
}
|
||||||
catch (Exception ex)
|
catch (Exception ex)
|
||||||
{
|
{
|
||||||
_logger.LogError(ex, "Failed to fetch Kaspersky advisory {Link}", item.Link);
|
_logger.LogError(ex, "Failed to fetch Kaspersky advisory {Link}", item.Link);
|
||||||
await _stateRepository.MarkFailureAsync(
|
await _stateRepository.MarkFailureAsync(
|
||||||
SourceName,
|
SourceName,
|
||||||
_timeProvider.GetUtcNow(),
|
_timeProvider.GetUtcNow(),
|
||||||
TimeSpan.FromMinutes(5),
|
TimeSpan.FromMinutes(5),
|
||||||
ex.Message,
|
ex.Message,
|
||||||
cancellationToken).ConfigureAwait(false);
|
cancellationToken).ConfigureAwait(false);
|
||||||
throw;
|
throw;
|
||||||
}
|
}
|
||||||
|
|
||||||
if (result.IsNotModified)
|
if (result.IsNotModified)
|
||||||
{
|
{
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
|
|
||||||
if (!result.IsSuccess || result.Document is null)
|
if (!result.IsSuccess || result.Document is null)
|
||||||
{
|
{
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
|
|
||||||
if (existing is not null
|
if (existing is not null
|
||||||
&& string.Equals(existing.Sha256, result.Document.Sha256, StringComparison.OrdinalIgnoreCase)
|
&& string.Equals(existing.Sha256, result.Document.Sha256, StringComparison.OrdinalIgnoreCase)
|
||||||
&& string.Equals(existing.Status, DocumentStatuses.Mapped, StringComparison.Ordinal))
|
&& string.Equals(existing.Status, DocumentStatuses.Mapped, StringComparison.Ordinal))
|
||||||
{
|
{
|
||||||
await _documentStore.UpdateStatusAsync(result.Document.Id, existing.Status, cancellationToken).ConfigureAwait(false);
|
await _documentStore.UpdateStatusAsync(result.Document.Id, existing.Status, cancellationToken).ConfigureAwait(false);
|
||||||
cursorState = cursorState.WithFetchMetadata(resourceKey, result.Document.Etag, result.Document.LastModified);
|
cursorState = cursorState.WithFetchMetadata(resourceKey, result.Document.Etag, result.Document.LastModified);
|
||||||
if (item.Published > maxPublished)
|
if (item.Published > maxPublished)
|
||||||
{
|
{
|
||||||
maxPublished = item.Published;
|
maxPublished = item.Published;
|
||||||
}
|
}
|
||||||
|
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
|
|
||||||
pendingDocuments.Add(result.Document.Id);
|
pendingDocuments.Add(result.Document.Id);
|
||||||
cursorState = cursorState.WithFetchMetadata(resourceKey, result.Document.Etag, result.Document.LastModified);
|
cursorState = cursorState.WithFetchMetadata(resourceKey, result.Document.Etag, result.Document.LastModified);
|
||||||
if (item.Published > maxPublished)
|
if (item.Published > maxPublished)
|
||||||
{
|
{
|
||||||
maxPublished = item.Published;
|
maxPublished = item.Published;
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
cursorState = cursorState.PruneFetchCache(touchedResources);
|
cursorState = cursorState.PruneFetchCache(touchedResources);
|
||||||
|
|
||||||
var updatedCursor = cursorState
|
var updatedCursor = cursorState
|
||||||
.WithPendingDocuments(pendingDocuments)
|
.WithPendingDocuments(pendingDocuments)
|
||||||
.WithLastPublished(maxPublished == DateTimeOffset.MinValue ? cursor.LastPublished : maxPublished);
|
.WithLastPublished(maxPublished == DateTimeOffset.MinValue ? cursor.LastPublished : maxPublished);
|
||||||
|
|
||||||
await UpdateCursorAsync(updatedCursor, cancellationToken).ConfigureAwait(false);
|
await UpdateCursorAsync(updatedCursor, cancellationToken).ConfigureAwait(false);
|
||||||
}
|
}
|
||||||
|
|
||||||
public async Task ParseAsync(IServiceProvider services, CancellationToken cancellationToken)
|
public async Task ParseAsync(IServiceProvider services, CancellationToken cancellationToken)
|
||||||
{
|
{
|
||||||
ArgumentNullException.ThrowIfNull(services);
|
ArgumentNullException.ThrowIfNull(services);
|
||||||
|
|
||||||
var cursor = await GetCursorAsync(cancellationToken).ConfigureAwait(false);
|
var cursor = await GetCursorAsync(cancellationToken).ConfigureAwait(false);
|
||||||
if (cursor.PendingDocuments.Count == 0)
|
if (cursor.PendingDocuments.Count == 0)
|
||||||
{
|
{
|
||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
|
|
||||||
var remainingDocuments = cursor.PendingDocuments.ToList();
|
var remainingDocuments = cursor.PendingDocuments.ToList();
|
||||||
var pendingMappings = cursor.PendingMappings.ToList();
|
var pendingMappings = cursor.PendingMappings.ToList();
|
||||||
|
|
||||||
foreach (var documentId in cursor.PendingDocuments)
|
foreach (var documentId in cursor.PendingDocuments)
|
||||||
{
|
{
|
||||||
cancellationToken.ThrowIfCancellationRequested();
|
cancellationToken.ThrowIfCancellationRequested();
|
||||||
|
|
||||||
var document = await _documentStore.FindAsync(documentId, cancellationToken).ConfigureAwait(false);
|
var document = await _documentStore.FindAsync(documentId, cancellationToken).ConfigureAwait(false);
|
||||||
if (document is null)
|
if (document is null)
|
||||||
{
|
{
|
||||||
remainingDocuments.Remove(documentId);
|
remainingDocuments.Remove(documentId);
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
|
|
||||||
if (!document.GridFsId.HasValue)
|
if (!document.PayloadId.HasValue)
|
||||||
{
|
{
|
||||||
_logger.LogWarning("Kaspersky document {DocumentId} missing GridFS content", document.Id);
|
_logger.LogWarning("Kaspersky document {DocumentId} missing GridFS content", document.Id);
|
||||||
await _documentStore.UpdateStatusAsync(document.Id, DocumentStatuses.Failed, cancellationToken).ConfigureAwait(false);
|
await _documentStore.UpdateStatusAsync(document.Id, DocumentStatuses.Failed, cancellationToken).ConfigureAwait(false);
|
||||||
remainingDocuments.Remove(documentId);
|
remainingDocuments.Remove(documentId);
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
|
|
||||||
var metadata = document.Metadata ?? new Dictionary<string, string>();
|
var metadata = document.Metadata ?? new Dictionary<string, string>();
|
||||||
var title = metadata.TryGetValue("kaspersky.title", out var titleValue) ? titleValue : document.Uri;
|
var title = metadata.TryGetValue("kaspersky.title", out var titleValue) ? titleValue : document.Uri;
|
||||||
var link = metadata.TryGetValue("kaspersky.link", out var linkValue) ? linkValue : document.Uri;
|
var link = metadata.TryGetValue("kaspersky.link", out var linkValue) ? linkValue : document.Uri;
|
||||||
var published = metadata.TryGetValue("kaspersky.published", out var publishedValue) && DateTimeOffset.TryParse(publishedValue, out var parsedPublished)
|
var published = metadata.TryGetValue("kaspersky.published", out var publishedValue) && DateTimeOffset.TryParse(publishedValue, out var parsedPublished)
|
||||||
? parsedPublished.ToUniversalTime()
|
? parsedPublished.ToUniversalTime()
|
||||||
: document.FetchedAt;
|
: document.FetchedAt;
|
||||||
var summary = metadata.TryGetValue("kaspersky.summary", out var summaryValue) ? summaryValue : null;
|
var summary = metadata.TryGetValue("kaspersky.summary", out var summaryValue) ? summaryValue : null;
|
||||||
var slug = metadata.TryGetValue("kaspersky.slug", out var slugValue) ? slugValue : ExtractSlug(new Uri(link, UriKind.Absolute));
|
var slug = metadata.TryGetValue("kaspersky.slug", out var slugValue) ? slugValue : ExtractSlug(new Uri(link, UriKind.Absolute));
|
||||||
var advisoryKey = string.IsNullOrWhiteSpace(slug) ? Guid.NewGuid().ToString("N") : slug;
|
var advisoryKey = string.IsNullOrWhiteSpace(slug) ? Guid.NewGuid().ToString("N") : slug;
|
||||||
|
|
||||||
byte[] rawBytes;
|
byte[] rawBytes;
|
||||||
try
|
try
|
||||||
{
|
{
|
||||||
rawBytes = await _rawDocumentStorage.DownloadAsync(document.GridFsId.Value, cancellationToken).ConfigureAwait(false);
|
rawBytes = await _rawDocumentStorage.DownloadAsync(document.PayloadId.Value, cancellationToken).ConfigureAwait(false);
|
||||||
}
|
}
|
||||||
catch (Exception ex)
|
catch (Exception ex)
|
||||||
{
|
{
|
||||||
_logger.LogError(ex, "Failed downloading raw Kaspersky document {DocumentId}", document.Id);
|
_logger.LogError(ex, "Failed downloading raw Kaspersky document {DocumentId}", document.Id);
|
||||||
throw;
|
throw;
|
||||||
}
|
}
|
||||||
|
|
||||||
var dto = KasperskyAdvisoryParser.Parse(advisoryKey, title, link, published, summary, rawBytes);
|
var dto = KasperskyAdvisoryParser.Parse(advisoryKey, title, link, published, summary, rawBytes);
|
||||||
var payload = BsonDocument.Parse(JsonSerializer.Serialize(dto, SerializerOptions));
|
var payload = BsonDocument.Parse(JsonSerializer.Serialize(dto, SerializerOptions));
|
||||||
var dtoRecord = new DtoRecord(Guid.NewGuid(), document.Id, SourceName, "ics.kaspersky/1", payload, _timeProvider.GetUtcNow());
|
var dtoRecord = new DtoRecord(Guid.NewGuid(), document.Id, SourceName, "ics.kaspersky/1", payload, _timeProvider.GetUtcNow());
|
||||||
|
|
||||||
await _dtoStore.UpsertAsync(dtoRecord, cancellationToken).ConfigureAwait(false);
|
await _dtoStore.UpsertAsync(dtoRecord, cancellationToken).ConfigureAwait(false);
|
||||||
await _documentStore.UpdateStatusAsync(document.Id, DocumentStatuses.PendingMap, cancellationToken).ConfigureAwait(false);
|
await _documentStore.UpdateStatusAsync(document.Id, DocumentStatuses.PendingMap, cancellationToken).ConfigureAwait(false);
|
||||||
|
|
||||||
remainingDocuments.Remove(documentId);
|
remainingDocuments.Remove(documentId);
|
||||||
if (!pendingMappings.Contains(documentId))
|
if (!pendingMappings.Contains(documentId))
|
||||||
{
|
{
|
||||||
pendingMappings.Add(documentId);
|
pendingMappings.Add(documentId);
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
var updatedCursor = cursor
|
var updatedCursor = cursor
|
||||||
.WithPendingDocuments(remainingDocuments)
|
.WithPendingDocuments(remainingDocuments)
|
||||||
.WithPendingMappings(pendingMappings);
|
.WithPendingMappings(pendingMappings);
|
||||||
|
|
||||||
await UpdateCursorAsync(updatedCursor, cancellationToken).ConfigureAwait(false);
|
await UpdateCursorAsync(updatedCursor, cancellationToken).ConfigureAwait(false);
|
||||||
}
|
}
|
||||||
|
|
||||||
public async Task MapAsync(IServiceProvider services, CancellationToken cancellationToken)
|
public async Task MapAsync(IServiceProvider services, CancellationToken cancellationToken)
|
||||||
{
|
{
|
||||||
ArgumentNullException.ThrowIfNull(services);
|
ArgumentNullException.ThrowIfNull(services);
|
||||||
|
|
||||||
var cursor = await GetCursorAsync(cancellationToken).ConfigureAwait(false);
|
var cursor = await GetCursorAsync(cancellationToken).ConfigureAwait(false);
|
||||||
if (cursor.PendingMappings.Count == 0)
|
if (cursor.PendingMappings.Count == 0)
|
||||||
{
|
{
|
||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
|
|
||||||
var pendingMappings = cursor.PendingMappings.ToList();
|
var pendingMappings = cursor.PendingMappings.ToList();
|
||||||
|
|
||||||
foreach (var documentId in cursor.PendingMappings)
|
foreach (var documentId in cursor.PendingMappings)
|
||||||
{
|
{
|
||||||
cancellationToken.ThrowIfCancellationRequested();
|
cancellationToken.ThrowIfCancellationRequested();
|
||||||
|
|
||||||
var dto = await _dtoStore.FindByDocumentIdAsync(documentId, cancellationToken).ConfigureAwait(false);
|
var dto = await _dtoStore.FindByDocumentIdAsync(documentId, cancellationToken).ConfigureAwait(false);
|
||||||
var document = await _documentStore.FindAsync(documentId, cancellationToken).ConfigureAwait(false);
|
var document = await _documentStore.FindAsync(documentId, cancellationToken).ConfigureAwait(false);
|
||||||
|
|
||||||
if (dto is null || document is null)
|
if (dto is null || document is null)
|
||||||
{
|
{
|
||||||
_logger.LogWarning("Skipping Kaspersky mapping for {DocumentId}: DTO or document missing", documentId);
|
_logger.LogWarning("Skipping Kaspersky mapping for {DocumentId}: DTO or document missing", documentId);
|
||||||
pendingMappings.Remove(documentId);
|
pendingMappings.Remove(documentId);
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
|
|
||||||
var dtoJson = dto.Payload.ToJson(new MongoDB.Bson.IO.JsonWriterSettings
|
var dtoJson = dto.Payload.ToJson(new MongoDB.Bson.IO.JsonWriterSettings
|
||||||
{
|
{
|
||||||
OutputMode = MongoDB.Bson.IO.JsonOutputMode.RelaxedExtendedJson,
|
OutputMode = MongoDB.Bson.IO.JsonOutputMode.RelaxedExtendedJson,
|
||||||
});
|
});
|
||||||
|
|
||||||
KasperskyAdvisoryDto advisoryDto;
|
KasperskyAdvisoryDto advisoryDto;
|
||||||
try
|
try
|
||||||
{
|
{
|
||||||
advisoryDto = JsonSerializer.Deserialize<KasperskyAdvisoryDto>(dtoJson, SerializerOptions)
|
advisoryDto = JsonSerializer.Deserialize<KasperskyAdvisoryDto>(dtoJson, SerializerOptions)
|
||||||
?? throw new InvalidOperationException("Deserialized DTO was null.");
|
?? throw new InvalidOperationException("Deserialized DTO was null.");
|
||||||
}
|
}
|
||||||
catch (Exception ex)
|
catch (Exception ex)
|
||||||
{
|
{
|
||||||
_logger.LogError(ex, "Failed to deserialize Kaspersky DTO for {DocumentId}", document.Id);
|
_logger.LogError(ex, "Failed to deserialize Kaspersky DTO for {DocumentId}", document.Id);
|
||||||
await _documentStore.UpdateStatusAsync(document.Id, DocumentStatuses.Failed, cancellationToken).ConfigureAwait(false);
|
await _documentStore.UpdateStatusAsync(document.Id, DocumentStatuses.Failed, cancellationToken).ConfigureAwait(false);
|
||||||
pendingMappings.Remove(documentId);
|
pendingMappings.Remove(documentId);
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
|
|
||||||
var fetchProvenance = new AdvisoryProvenance(SourceName, "document", document.Uri, document.FetchedAt);
|
var fetchProvenance = new AdvisoryProvenance(SourceName, "document", document.Uri, document.FetchedAt);
|
||||||
var mappingProvenance = new AdvisoryProvenance(SourceName, "mapping", advisoryDto.AdvisoryKey, dto.ValidatedAt);
|
var mappingProvenance = new AdvisoryProvenance(SourceName, "mapping", advisoryDto.AdvisoryKey, dto.ValidatedAt);
|
||||||
|
|
||||||
var aliases = new HashSet<string>(StringComparer.OrdinalIgnoreCase)
|
var aliases = new HashSet<string>(StringComparer.OrdinalIgnoreCase)
|
||||||
{
|
{
|
||||||
advisoryDto.AdvisoryKey,
|
advisoryDto.AdvisoryKey,
|
||||||
};
|
};
|
||||||
foreach (var cve in advisoryDto.CveIds)
|
foreach (var cve in advisoryDto.CveIds)
|
||||||
{
|
{
|
||||||
aliases.Add(cve);
|
aliases.Add(cve);
|
||||||
}
|
}
|
||||||
|
|
||||||
var references = new List<AdvisoryReference>();
|
var references = new List<AdvisoryReference>();
|
||||||
try
|
try
|
||||||
{
|
{
|
||||||
references.Add(new AdvisoryReference(
|
references.Add(new AdvisoryReference(
|
||||||
advisoryDto.Link,
|
advisoryDto.Link,
|
||||||
"advisory",
|
"advisory",
|
||||||
"kaspersky-ics",
|
"kaspersky-ics",
|
||||||
null,
|
null,
|
||||||
new AdvisoryProvenance(SourceName, "reference", advisoryDto.Link, dto.ValidatedAt)));
|
new AdvisoryProvenance(SourceName, "reference", advisoryDto.Link, dto.ValidatedAt)));
|
||||||
}
|
}
|
||||||
catch (ArgumentException)
|
catch (ArgumentException)
|
||||||
{
|
{
|
||||||
_logger.LogWarning("Invalid advisory link {Link} for {AdvisoryKey}", advisoryDto.Link, advisoryDto.AdvisoryKey);
|
_logger.LogWarning("Invalid advisory link {Link} for {AdvisoryKey}", advisoryDto.Link, advisoryDto.AdvisoryKey);
|
||||||
}
|
}
|
||||||
|
|
||||||
foreach (var cve in advisoryDto.CveIds)
|
foreach (var cve in advisoryDto.CveIds)
|
||||||
{
|
{
|
||||||
var url = $"https://www.cve.org/CVERecord?id={cve}";
|
var url = $"https://www.cve.org/CVERecord?id={cve}";
|
||||||
try
|
try
|
||||||
{
|
{
|
||||||
references.Add(new AdvisoryReference(
|
references.Add(new AdvisoryReference(
|
||||||
url,
|
url,
|
||||||
"advisory",
|
"advisory",
|
||||||
cve,
|
cve,
|
||||||
null,
|
null,
|
||||||
new AdvisoryProvenance(SourceName, "reference", url, dto.ValidatedAt)));
|
new AdvisoryProvenance(SourceName, "reference", url, dto.ValidatedAt)));
|
||||||
}
|
}
|
||||||
catch (ArgumentException)
|
catch (ArgumentException)
|
||||||
{
|
{
|
||||||
// ignore malformed
|
// ignore malformed
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
var affectedPackages = new List<AffectedPackage>();
|
var affectedPackages = new List<AffectedPackage>();
|
||||||
foreach (var vendor in advisoryDto.VendorNames)
|
foreach (var vendor in advisoryDto.VendorNames)
|
||||||
{
|
{
|
||||||
@@ -413,52 +413,52 @@ public sealed class KasperskyConnector : IFeedConnector
|
|||||||
statuses: Array.Empty<AffectedPackageStatus>(),
|
statuses: Array.Empty<AffectedPackageStatus>(),
|
||||||
provenance: provenance));
|
provenance: provenance));
|
||||||
}
|
}
|
||||||
|
|
||||||
var advisory = new Advisory(
|
var advisory = new Advisory(
|
||||||
advisoryDto.AdvisoryKey,
|
advisoryDto.AdvisoryKey,
|
||||||
advisoryDto.Title,
|
advisoryDto.Title,
|
||||||
advisoryDto.Summary ?? advisoryDto.Content,
|
advisoryDto.Summary ?? advisoryDto.Content,
|
||||||
language: "en",
|
language: "en",
|
||||||
published: advisoryDto.Published,
|
published: advisoryDto.Published,
|
||||||
modified: advisoryDto.Published,
|
modified: advisoryDto.Published,
|
||||||
severity: null,
|
severity: null,
|
||||||
exploitKnown: false,
|
exploitKnown: false,
|
||||||
aliases: aliases,
|
aliases: aliases,
|
||||||
references: references,
|
references: references,
|
||||||
affectedPackages: affectedPackages,
|
affectedPackages: affectedPackages,
|
||||||
cvssMetrics: Array.Empty<CvssMetric>(),
|
cvssMetrics: Array.Empty<CvssMetric>(),
|
||||||
provenance: new[] { fetchProvenance, mappingProvenance });
|
provenance: new[] { fetchProvenance, mappingProvenance });
|
||||||
|
|
||||||
await _advisoryStore.UpsertAsync(advisory, cancellationToken).ConfigureAwait(false);
|
await _advisoryStore.UpsertAsync(advisory, cancellationToken).ConfigureAwait(false);
|
||||||
await _documentStore.UpdateStatusAsync(document.Id, DocumentStatuses.Mapped, cancellationToken).ConfigureAwait(false);
|
await _documentStore.UpdateStatusAsync(document.Id, DocumentStatuses.Mapped, cancellationToken).ConfigureAwait(false);
|
||||||
|
|
||||||
pendingMappings.Remove(documentId);
|
pendingMappings.Remove(documentId);
|
||||||
}
|
}
|
||||||
|
|
||||||
var updatedCursor = cursor.WithPendingMappings(pendingMappings);
|
var updatedCursor = cursor.WithPendingMappings(pendingMappings);
|
||||||
await UpdateCursorAsync(updatedCursor, cancellationToken).ConfigureAwait(false);
|
await UpdateCursorAsync(updatedCursor, cancellationToken).ConfigureAwait(false);
|
||||||
}
|
}
|
||||||
|
|
||||||
private async Task<KasperskyCursor> GetCursorAsync(CancellationToken cancellationToken)
|
private async Task<KasperskyCursor> GetCursorAsync(CancellationToken cancellationToken)
|
||||||
{
|
{
|
||||||
var state = await _stateRepository.TryGetAsync(SourceName, cancellationToken).ConfigureAwait(false);
|
var state = await _stateRepository.TryGetAsync(SourceName, cancellationToken).ConfigureAwait(false);
|
||||||
return state is null ? KasperskyCursor.Empty : KasperskyCursor.FromBson(state.Cursor);
|
return state is null ? KasperskyCursor.Empty : KasperskyCursor.FromBson(state.Cursor);
|
||||||
}
|
}
|
||||||
|
|
||||||
private async Task UpdateCursorAsync(KasperskyCursor cursor, CancellationToken cancellationToken)
|
private async Task UpdateCursorAsync(KasperskyCursor cursor, CancellationToken cancellationToken)
|
||||||
{
|
{
|
||||||
await _stateRepository.UpdateCursorAsync(SourceName, cursor.ToBsonDocument(), _timeProvider.GetUtcNow(), cancellationToken).ConfigureAwait(false);
|
await _stateRepository.UpdateCursorAsync(SourceName, cursor.ToBsonDocument(), _timeProvider.GetUtcNow(), cancellationToken).ConfigureAwait(false);
|
||||||
}
|
}
|
||||||
|
|
||||||
private static string? ExtractSlug(Uri link)
|
private static string? ExtractSlug(Uri link)
|
||||||
{
|
{
|
||||||
var segments = link.Segments;
|
var segments = link.Segments;
|
||||||
if (segments.Length == 0)
|
if (segments.Length == 0)
|
||||||
{
|
{
|
||||||
return null;
|
return null;
|
||||||
}
|
}
|
||||||
|
|
||||||
var last = segments[^1].Trim('/');
|
var last = segments[^1].Trim('/');
|
||||||
return string.IsNullOrWhiteSpace(last) && segments.Length > 1 ? segments[^2].Trim('/') : last;
|
return string.IsNullOrWhiteSpace(last) && segments.Length > 1 ? segments[^2].Trim('/') : last;
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -1,325 +1,325 @@
|
|||||||
using System.Collections.Generic;
|
using System.Collections.Generic;
|
||||||
using System.Linq;
|
using System.Linq;
|
||||||
using System.Text.Json;
|
using System.Text.Json;
|
||||||
using Microsoft.Extensions.Logging;
|
using Microsoft.Extensions.Logging;
|
||||||
using Microsoft.Extensions.Options;
|
using Microsoft.Extensions.Options;
|
||||||
using MongoDB.Bson;
|
using MongoDB.Bson;
|
||||||
using StellaOps.Concelier.Models;
|
using StellaOps.Concelier.Models;
|
||||||
using StellaOps.Concelier.Connector.Common;
|
using StellaOps.Concelier.Connector.Common;
|
||||||
using StellaOps.Concelier.Connector.Common.Fetch;
|
using StellaOps.Concelier.Connector.Common.Fetch;
|
||||||
using StellaOps.Concelier.Connector.Jvn.Configuration;
|
using StellaOps.Concelier.Connector.Jvn.Configuration;
|
||||||
using StellaOps.Concelier.Connector.Jvn.Internal;
|
using StellaOps.Concelier.Connector.Jvn.Internal;
|
||||||
using StellaOps.Concelier.Storage.Mongo;
|
using StellaOps.Concelier.Storage.Mongo;
|
||||||
using StellaOps.Concelier.Storage.Mongo.Advisories;
|
using StellaOps.Concelier.Storage.Mongo.Advisories;
|
||||||
using StellaOps.Concelier.Storage.Mongo.Documents;
|
using StellaOps.Concelier.Storage.Mongo.Documents;
|
||||||
using StellaOps.Concelier.Storage.Mongo.Dtos;
|
using StellaOps.Concelier.Storage.Mongo.Dtos;
|
||||||
using StellaOps.Concelier.Storage.Mongo.JpFlags;
|
using StellaOps.Concelier.Storage.Mongo.JpFlags;
|
||||||
using StellaOps.Plugin;
|
using StellaOps.Plugin;
|
||||||
|
|
||||||
namespace StellaOps.Concelier.Connector.Jvn;
|
namespace StellaOps.Concelier.Connector.Jvn;
|
||||||
|
|
||||||
public sealed class JvnConnector : IFeedConnector
|
public sealed class JvnConnector : IFeedConnector
|
||||||
{
|
{
|
||||||
private static readonly JsonSerializerOptions SerializerOptions = new(JsonSerializerDefaults.General)
|
private static readonly JsonSerializerOptions SerializerOptions = new(JsonSerializerDefaults.General)
|
||||||
{
|
{
|
||||||
PropertyNamingPolicy = JsonNamingPolicy.CamelCase,
|
PropertyNamingPolicy = JsonNamingPolicy.CamelCase,
|
||||||
WriteIndented = false,
|
WriteIndented = false,
|
||||||
DefaultIgnoreCondition = System.Text.Json.Serialization.JsonIgnoreCondition.WhenWritingNull,
|
DefaultIgnoreCondition = System.Text.Json.Serialization.JsonIgnoreCondition.WhenWritingNull,
|
||||||
};
|
};
|
||||||
|
|
||||||
private readonly MyJvnClient _client;
|
private readonly MyJvnClient _client;
|
||||||
private readonly SourceFetchService _fetchService;
|
private readonly SourceFetchService _fetchService;
|
||||||
private readonly RawDocumentStorage _rawDocumentStorage;
|
private readonly RawDocumentStorage _rawDocumentStorage;
|
||||||
private readonly IDocumentStore _documentStore;
|
private readonly IDocumentStore _documentStore;
|
||||||
private readonly IDtoStore _dtoStore;
|
private readonly IDtoStore _dtoStore;
|
||||||
private readonly IAdvisoryStore _advisoryStore;
|
private readonly IAdvisoryStore _advisoryStore;
|
||||||
private readonly IJpFlagStore _jpFlagStore;
|
private readonly IJpFlagStore _jpFlagStore;
|
||||||
private readonly ISourceStateRepository _stateRepository;
|
private readonly ISourceStateRepository _stateRepository;
|
||||||
private readonly TimeProvider _timeProvider;
|
private readonly TimeProvider _timeProvider;
|
||||||
private readonly JvnOptions _options;
|
private readonly JvnOptions _options;
|
||||||
private readonly ILogger<JvnConnector> _logger;
|
private readonly ILogger<JvnConnector> _logger;
|
||||||
|
|
||||||
public JvnConnector(
|
public JvnConnector(
|
||||||
MyJvnClient client,
|
MyJvnClient client,
|
||||||
SourceFetchService fetchService,
|
SourceFetchService fetchService,
|
||||||
RawDocumentStorage rawDocumentStorage,
|
RawDocumentStorage rawDocumentStorage,
|
||||||
IDocumentStore documentStore,
|
IDocumentStore documentStore,
|
||||||
IDtoStore dtoStore,
|
IDtoStore dtoStore,
|
||||||
IAdvisoryStore advisoryStore,
|
IAdvisoryStore advisoryStore,
|
||||||
IJpFlagStore jpFlagStore,
|
IJpFlagStore jpFlagStore,
|
||||||
ISourceStateRepository stateRepository,
|
ISourceStateRepository stateRepository,
|
||||||
IOptions<JvnOptions> options,
|
IOptions<JvnOptions> options,
|
||||||
TimeProvider? timeProvider,
|
TimeProvider? timeProvider,
|
||||||
ILogger<JvnConnector> logger)
|
ILogger<JvnConnector> logger)
|
||||||
{
|
{
|
||||||
_client = client ?? throw new ArgumentNullException(nameof(client));
|
_client = client ?? throw new ArgumentNullException(nameof(client));
|
||||||
_fetchService = fetchService ?? throw new ArgumentNullException(nameof(fetchService));
|
_fetchService = fetchService ?? throw new ArgumentNullException(nameof(fetchService));
|
||||||
_rawDocumentStorage = rawDocumentStorage ?? throw new ArgumentNullException(nameof(rawDocumentStorage));
|
_rawDocumentStorage = rawDocumentStorage ?? throw new ArgumentNullException(nameof(rawDocumentStorage));
|
||||||
_documentStore = documentStore ?? throw new ArgumentNullException(nameof(documentStore));
|
_documentStore = documentStore ?? throw new ArgumentNullException(nameof(documentStore));
|
||||||
_dtoStore = dtoStore ?? throw new ArgumentNullException(nameof(dtoStore));
|
_dtoStore = dtoStore ?? throw new ArgumentNullException(nameof(dtoStore));
|
||||||
_advisoryStore = advisoryStore ?? throw new ArgumentNullException(nameof(advisoryStore));
|
_advisoryStore = advisoryStore ?? throw new ArgumentNullException(nameof(advisoryStore));
|
||||||
_jpFlagStore = jpFlagStore ?? throw new ArgumentNullException(nameof(jpFlagStore));
|
_jpFlagStore = jpFlagStore ?? throw new ArgumentNullException(nameof(jpFlagStore));
|
||||||
_stateRepository = stateRepository ?? throw new ArgumentNullException(nameof(stateRepository));
|
_stateRepository = stateRepository ?? throw new ArgumentNullException(nameof(stateRepository));
|
||||||
_options = (options ?? throw new ArgumentNullException(nameof(options))).Value ?? throw new ArgumentNullException(nameof(options));
|
_options = (options ?? throw new ArgumentNullException(nameof(options))).Value ?? throw new ArgumentNullException(nameof(options));
|
||||||
_options.Validate();
|
_options.Validate();
|
||||||
_timeProvider = timeProvider ?? TimeProvider.System;
|
_timeProvider = timeProvider ?? TimeProvider.System;
|
||||||
_logger = logger ?? throw new ArgumentNullException(nameof(logger));
|
_logger = logger ?? throw new ArgumentNullException(nameof(logger));
|
||||||
}
|
}
|
||||||
|
|
||||||
public string SourceName => JvnConnectorPlugin.SourceName;
|
public string SourceName => JvnConnectorPlugin.SourceName;
|
||||||
|
|
||||||
public async Task FetchAsync(IServiceProvider services, CancellationToken cancellationToken)
|
public async Task FetchAsync(IServiceProvider services, CancellationToken cancellationToken)
|
||||||
{
|
{
|
||||||
ArgumentNullException.ThrowIfNull(services);
|
ArgumentNullException.ThrowIfNull(services);
|
||||||
|
|
||||||
var cursor = await GetCursorAsync(cancellationToken).ConfigureAwait(false);
|
var cursor = await GetCursorAsync(cancellationToken).ConfigureAwait(false);
|
||||||
var now = _timeProvider.GetUtcNow();
|
var now = _timeProvider.GetUtcNow();
|
||||||
|
|
||||||
var windowEnd = now;
|
var windowEnd = now;
|
||||||
var defaultWindowStart = windowEnd - _options.WindowSize;
|
var defaultWindowStart = windowEnd - _options.WindowSize;
|
||||||
|
|
||||||
var windowStart = cursor.LastCompletedWindowEnd.HasValue
|
var windowStart = cursor.LastCompletedWindowEnd.HasValue
|
||||||
? cursor.LastCompletedWindowEnd.Value - _options.WindowOverlap
|
? cursor.LastCompletedWindowEnd.Value - _options.WindowOverlap
|
||||||
: defaultWindowStart;
|
: defaultWindowStart;
|
||||||
|
|
||||||
if (windowStart < defaultWindowStart)
|
if (windowStart < defaultWindowStart)
|
||||||
{
|
{
|
||||||
windowStart = defaultWindowStart;
|
windowStart = defaultWindowStart;
|
||||||
}
|
}
|
||||||
|
|
||||||
if (windowStart >= windowEnd)
|
if (windowStart >= windowEnd)
|
||||||
{
|
{
|
||||||
windowStart = windowEnd - TimeSpan.FromHours(1);
|
windowStart = windowEnd - TimeSpan.FromHours(1);
|
||||||
}
|
}
|
||||||
|
|
||||||
_logger.LogInformation("JVN fetch window {WindowStart:o} - {WindowEnd:o}", windowStart, windowEnd);
|
_logger.LogInformation("JVN fetch window {WindowStart:o} - {WindowEnd:o}", windowStart, windowEnd);
|
||||||
|
|
||||||
IReadOnlyList<JvnOverviewItem> overviewItems;
|
IReadOnlyList<JvnOverviewItem> overviewItems;
|
||||||
try
|
try
|
||||||
{
|
{
|
||||||
overviewItems = await _client.GetOverviewAsync(windowStart, windowEnd, cancellationToken).ConfigureAwait(false);
|
overviewItems = await _client.GetOverviewAsync(windowStart, windowEnd, cancellationToken).ConfigureAwait(false);
|
||||||
}
|
}
|
||||||
catch (Exception ex)
|
catch (Exception ex)
|
||||||
{
|
{
|
||||||
_logger.LogError(ex, "Failed to retrieve JVN overview between {Start:o} and {End:o}", windowStart, windowEnd);
|
_logger.LogError(ex, "Failed to retrieve JVN overview between {Start:o} and {End:o}", windowStart, windowEnd);
|
||||||
await _stateRepository.MarkFailureAsync(SourceName, now, TimeSpan.FromMinutes(5), ex.Message, cancellationToken).ConfigureAwait(false);
|
await _stateRepository.MarkFailureAsync(SourceName, now, TimeSpan.FromMinutes(5), ex.Message, cancellationToken).ConfigureAwait(false);
|
||||||
throw;
|
throw;
|
||||||
}
|
}
|
||||||
|
|
||||||
_logger.LogInformation("JVN overview returned {Count} items", overviewItems.Count);
|
_logger.LogInformation("JVN overview returned {Count} items", overviewItems.Count);
|
||||||
|
|
||||||
var pendingDocuments = cursor.PendingDocuments.ToHashSet();
|
var pendingDocuments = cursor.PendingDocuments.ToHashSet();
|
||||||
|
|
||||||
foreach (var item in overviewItems)
|
foreach (var item in overviewItems)
|
||||||
{
|
{
|
||||||
cancellationToken.ThrowIfCancellationRequested();
|
cancellationToken.ThrowIfCancellationRequested();
|
||||||
|
|
||||||
var detailUri = _client.BuildDetailUri(item.VulnerabilityId);
|
var detailUri = _client.BuildDetailUri(item.VulnerabilityId);
|
||||||
var metadata = new Dictionary<string, string>(StringComparer.Ordinal)
|
var metadata = new Dictionary<string, string>(StringComparer.Ordinal)
|
||||||
{
|
{
|
||||||
["jvn.vulnId"] = item.VulnerabilityId,
|
["jvn.vulnId"] = item.VulnerabilityId,
|
||||||
["jvn.detailUrl"] = detailUri.ToString(),
|
["jvn.detailUrl"] = detailUri.ToString(),
|
||||||
};
|
};
|
||||||
|
|
||||||
if (item.DateFirstPublished.HasValue)
|
if (item.DateFirstPublished.HasValue)
|
||||||
{
|
{
|
||||||
metadata["jvn.firstPublished"] = item.DateFirstPublished.Value.ToString("O");
|
metadata["jvn.firstPublished"] = item.DateFirstPublished.Value.ToString("O");
|
||||||
}
|
}
|
||||||
|
|
||||||
if (item.DateLastUpdated.HasValue)
|
if (item.DateLastUpdated.HasValue)
|
||||||
{
|
{
|
||||||
metadata["jvn.lastUpdated"] = item.DateLastUpdated.Value.ToString("O");
|
metadata["jvn.lastUpdated"] = item.DateLastUpdated.Value.ToString("O");
|
||||||
}
|
}
|
||||||
|
|
||||||
var result = await _fetchService.FetchAsync(
|
var result = await _fetchService.FetchAsync(
|
||||||
new SourceFetchRequest(JvnOptions.HttpClientName, SourceName, detailUri)
|
new SourceFetchRequest(JvnOptions.HttpClientName, SourceName, detailUri)
|
||||||
{
|
{
|
||||||
Metadata = metadata
|
Metadata = metadata
|
||||||
},
|
},
|
||||||
cancellationToken).ConfigureAwait(false);
|
cancellationToken).ConfigureAwait(false);
|
||||||
|
|
||||||
if (!result.IsSuccess || result.Document is null)
|
if (!result.IsSuccess || result.Document is null)
|
||||||
{
|
{
|
||||||
if (!result.IsNotModified)
|
if (!result.IsNotModified)
|
||||||
{
|
{
|
||||||
_logger.LogWarning("JVN fetch for {Uri} returned status {Status}", detailUri, result.StatusCode);
|
_logger.LogWarning("JVN fetch for {Uri} returned status {Status}", detailUri, result.StatusCode);
|
||||||
}
|
}
|
||||||
|
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
|
|
||||||
_logger.LogDebug("JVN fetched document {DocumentId}", result.Document.Id);
|
_logger.LogDebug("JVN fetched document {DocumentId}", result.Document.Id);
|
||||||
pendingDocuments.Add(result.Document.Id);
|
pendingDocuments.Add(result.Document.Id);
|
||||||
}
|
}
|
||||||
|
|
||||||
var updatedCursor = cursor
|
var updatedCursor = cursor
|
||||||
.WithWindow(windowStart, windowEnd)
|
.WithWindow(windowStart, windowEnd)
|
||||||
.WithCompletedWindow(windowEnd)
|
.WithCompletedWindow(windowEnd)
|
||||||
.WithPendingDocuments(pendingDocuments);
|
.WithPendingDocuments(pendingDocuments);
|
||||||
|
|
||||||
await UpdateCursorAsync(updatedCursor, cancellationToken).ConfigureAwait(false);
|
await UpdateCursorAsync(updatedCursor, cancellationToken).ConfigureAwait(false);
|
||||||
}
|
}
|
||||||
|
|
||||||
public async Task ParseAsync(IServiceProvider services, CancellationToken cancellationToken)
|
public async Task ParseAsync(IServiceProvider services, CancellationToken cancellationToken)
|
||||||
{
|
{
|
||||||
ArgumentNullException.ThrowIfNull(services);
|
ArgumentNullException.ThrowIfNull(services);
|
||||||
|
|
||||||
var cursor = await GetCursorAsync(cancellationToken).ConfigureAwait(false);
|
var cursor = await GetCursorAsync(cancellationToken).ConfigureAwait(false);
|
||||||
_logger.LogDebug("JVN parse pending documents: {PendingCount}", cursor.PendingDocuments.Count);
|
_logger.LogDebug("JVN parse pending documents: {PendingCount}", cursor.PendingDocuments.Count);
|
||||||
Console.WriteLine($"JVN parse pending count: {cursor.PendingDocuments.Count}");
|
Console.WriteLine($"JVN parse pending count: {cursor.PendingDocuments.Count}");
|
||||||
if (cursor.PendingDocuments.Count == 0)
|
if (cursor.PendingDocuments.Count == 0)
|
||||||
{
|
{
|
||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
|
|
||||||
var remainingDocuments = cursor.PendingDocuments.ToList();
|
var remainingDocuments = cursor.PendingDocuments.ToList();
|
||||||
var pendingMappings = cursor.PendingMappings.ToList();
|
var pendingMappings = cursor.PendingMappings.ToList();
|
||||||
|
|
||||||
foreach (var documentId in cursor.PendingDocuments)
|
foreach (var documentId in cursor.PendingDocuments)
|
||||||
{
|
{
|
||||||
cancellationToken.ThrowIfCancellationRequested();
|
cancellationToken.ThrowIfCancellationRequested();
|
||||||
_logger.LogDebug("JVN parsing document {DocumentId}", documentId);
|
_logger.LogDebug("JVN parsing document {DocumentId}", documentId);
|
||||||
Console.WriteLine($"JVN parsing document {documentId}");
|
Console.WriteLine($"JVN parsing document {documentId}");
|
||||||
|
|
||||||
var document = await _documentStore.FindAsync(documentId, cancellationToken).ConfigureAwait(false);
|
var document = await _documentStore.FindAsync(documentId, cancellationToken).ConfigureAwait(false);
|
||||||
if (document is null)
|
if (document is null)
|
||||||
{
|
{
|
||||||
_logger.LogWarning("JVN document {DocumentId} no longer exists; skipping", documentId);
|
_logger.LogWarning("JVN document {DocumentId} no longer exists; skipping", documentId);
|
||||||
remainingDocuments.Remove(documentId);
|
remainingDocuments.Remove(documentId);
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
|
|
||||||
if (!document.GridFsId.HasValue)
|
if (!document.PayloadId.HasValue)
|
||||||
{
|
{
|
||||||
_logger.LogWarning("JVN document {DocumentId} is missing GridFS content; marking as failed", documentId);
|
_logger.LogWarning("JVN document {DocumentId} is missing GridFS content; marking as failed", documentId);
|
||||||
await _documentStore.UpdateStatusAsync(document.Id, DocumentStatuses.Failed, cancellationToken).ConfigureAwait(false);
|
await _documentStore.UpdateStatusAsync(document.Id, DocumentStatuses.Failed, cancellationToken).ConfigureAwait(false);
|
||||||
remainingDocuments.Remove(documentId);
|
remainingDocuments.Remove(documentId);
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
|
|
||||||
byte[] rawBytes;
|
byte[] rawBytes;
|
||||||
try
|
try
|
||||||
{
|
{
|
||||||
rawBytes = await _rawDocumentStorage.DownloadAsync(document.GridFsId.Value, cancellationToken).ConfigureAwait(false);
|
rawBytes = await _rawDocumentStorage.DownloadAsync(document.PayloadId.Value, cancellationToken).ConfigureAwait(false);
|
||||||
}
|
}
|
||||||
catch (Exception ex)
|
catch (Exception ex)
|
||||||
{
|
{
|
||||||
_logger.LogError(ex, "Unable to download raw JVN document {DocumentId}", document.Id);
|
_logger.LogError(ex, "Unable to download raw JVN document {DocumentId}", document.Id);
|
||||||
throw;
|
throw;
|
||||||
}
|
}
|
||||||
|
|
||||||
JvnDetailDto detail;
|
JvnDetailDto detail;
|
||||||
try
|
try
|
||||||
{
|
{
|
||||||
detail = JvnDetailParser.Parse(rawBytes, document.Uri);
|
detail = JvnDetailParser.Parse(rawBytes, document.Uri);
|
||||||
}
|
}
|
||||||
catch (JvnSchemaValidationException ex)
|
catch (JvnSchemaValidationException ex)
|
||||||
{
|
{
|
||||||
Console.WriteLine($"JVN schema validation exception: {ex.Message}");
|
Console.WriteLine($"JVN schema validation exception: {ex.Message}");
|
||||||
_logger.LogWarning(ex, "JVN schema validation failed for document {DocumentId} ({Uri})", document.Id, document.Uri);
|
_logger.LogWarning(ex, "JVN schema validation failed for document {DocumentId} ({Uri})", document.Id, document.Uri);
|
||||||
await _documentStore.UpdateStatusAsync(document.Id, DocumentStatuses.Failed, cancellationToken).ConfigureAwait(false);
|
await _documentStore.UpdateStatusAsync(document.Id, DocumentStatuses.Failed, cancellationToken).ConfigureAwait(false);
|
||||||
remainingDocuments.Remove(documentId);
|
remainingDocuments.Remove(documentId);
|
||||||
throw;
|
throw;
|
||||||
}
|
}
|
||||||
|
|
||||||
var sanitizedJson = JsonSerializer.Serialize(detail, SerializerOptions);
|
var sanitizedJson = JsonSerializer.Serialize(detail, SerializerOptions);
|
||||||
var payload = BsonDocument.Parse(sanitizedJson);
|
var payload = BsonDocument.Parse(sanitizedJson);
|
||||||
var dtoRecord = new DtoRecord(
|
var dtoRecord = new DtoRecord(
|
||||||
Guid.NewGuid(),
|
Guid.NewGuid(),
|
||||||
document.Id,
|
document.Id,
|
||||||
SourceName,
|
SourceName,
|
||||||
JvnConstants.DtoSchemaVersion,
|
JvnConstants.DtoSchemaVersion,
|
||||||
payload,
|
payload,
|
||||||
_timeProvider.GetUtcNow());
|
_timeProvider.GetUtcNow());
|
||||||
|
|
||||||
await _dtoStore.UpsertAsync(dtoRecord, cancellationToken).ConfigureAwait(false);
|
await _dtoStore.UpsertAsync(dtoRecord, cancellationToken).ConfigureAwait(false);
|
||||||
await _documentStore.UpdateStatusAsync(document.Id, DocumentStatuses.PendingMap, cancellationToken).ConfigureAwait(false);
|
await _documentStore.UpdateStatusAsync(document.Id, DocumentStatuses.PendingMap, cancellationToken).ConfigureAwait(false);
|
||||||
|
|
||||||
remainingDocuments.Remove(documentId);
|
remainingDocuments.Remove(documentId);
|
||||||
if (!pendingMappings.Contains(documentId))
|
if (!pendingMappings.Contains(documentId))
|
||||||
{
|
{
|
||||||
pendingMappings.Add(documentId);
|
pendingMappings.Add(documentId);
|
||||||
Console.WriteLine($"Added mapping for {documentId}");
|
Console.WriteLine($"Added mapping for {documentId}");
|
||||||
_logger.LogDebug("JVN parsed document {DocumentId}", documentId);
|
_logger.LogDebug("JVN parsed document {DocumentId}", documentId);
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
var updatedCursor = cursor
|
var updatedCursor = cursor
|
||||||
.WithPendingDocuments(remainingDocuments)
|
.WithPendingDocuments(remainingDocuments)
|
||||||
.WithPendingMappings(pendingMappings);
|
.WithPendingMappings(pendingMappings);
|
||||||
|
|
||||||
await UpdateCursorAsync(updatedCursor, cancellationToken).ConfigureAwait(false);
|
await UpdateCursorAsync(updatedCursor, cancellationToken).ConfigureAwait(false);
|
||||||
}
|
}
|
||||||
|
|
||||||
public async Task MapAsync(IServiceProvider services, CancellationToken cancellationToken)
|
public async Task MapAsync(IServiceProvider services, CancellationToken cancellationToken)
|
||||||
{
|
{
|
||||||
ArgumentNullException.ThrowIfNull(services);
|
ArgumentNullException.ThrowIfNull(services);
|
||||||
|
|
||||||
var cursor = await GetCursorAsync(cancellationToken).ConfigureAwait(false);
|
var cursor = await GetCursorAsync(cancellationToken).ConfigureAwait(false);
|
||||||
_logger.LogDebug("JVN map pending mappings: {PendingCount}", cursor.PendingMappings.Count);
|
_logger.LogDebug("JVN map pending mappings: {PendingCount}", cursor.PendingMappings.Count);
|
||||||
if (cursor.PendingMappings.Count == 0)
|
if (cursor.PendingMappings.Count == 0)
|
||||||
{
|
{
|
||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
|
|
||||||
var pendingMappings = cursor.PendingMappings.ToList();
|
var pendingMappings = cursor.PendingMappings.ToList();
|
||||||
|
|
||||||
foreach (var documentId in cursor.PendingMappings)
|
foreach (var documentId in cursor.PendingMappings)
|
||||||
{
|
{
|
||||||
cancellationToken.ThrowIfCancellationRequested();
|
cancellationToken.ThrowIfCancellationRequested();
|
||||||
|
|
||||||
var dto = await _dtoStore.FindByDocumentIdAsync(documentId, cancellationToken).ConfigureAwait(false);
|
var dto = await _dtoStore.FindByDocumentIdAsync(documentId, cancellationToken).ConfigureAwait(false);
|
||||||
var document = await _documentStore.FindAsync(documentId, cancellationToken).ConfigureAwait(false);
|
var document = await _documentStore.FindAsync(documentId, cancellationToken).ConfigureAwait(false);
|
||||||
|
|
||||||
if (dto is null || document is null)
|
if (dto is null || document is null)
|
||||||
{
|
{
|
||||||
_logger.LogWarning("Skipping JVN mapping for {DocumentId}: DTO or document missing", documentId);
|
_logger.LogWarning("Skipping JVN mapping for {DocumentId}: DTO or document missing", documentId);
|
||||||
pendingMappings.Remove(documentId);
|
pendingMappings.Remove(documentId);
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
|
|
||||||
var dtoJson = dto.Payload.ToJson(new MongoDB.Bson.IO.JsonWriterSettings
|
var dtoJson = dto.Payload.ToJson(new MongoDB.Bson.IO.JsonWriterSettings
|
||||||
{
|
{
|
||||||
OutputMode = MongoDB.Bson.IO.JsonOutputMode.RelaxedExtendedJson,
|
OutputMode = MongoDB.Bson.IO.JsonOutputMode.RelaxedExtendedJson,
|
||||||
});
|
});
|
||||||
|
|
||||||
JvnDetailDto detail;
|
JvnDetailDto detail;
|
||||||
try
|
try
|
||||||
{
|
{
|
||||||
detail = JsonSerializer.Deserialize<JvnDetailDto>(dtoJson, SerializerOptions)
|
detail = JsonSerializer.Deserialize<JvnDetailDto>(dtoJson, SerializerOptions)
|
||||||
?? throw new InvalidOperationException("Deserialized DTO was null.");
|
?? throw new InvalidOperationException("Deserialized DTO was null.");
|
||||||
}
|
}
|
||||||
catch (Exception ex)
|
catch (Exception ex)
|
||||||
{
|
{
|
||||||
_logger.LogError(ex, "Failed to deserialize JVN DTO for document {DocumentId}", document.Id);
|
_logger.LogError(ex, "Failed to deserialize JVN DTO for document {DocumentId}", document.Id);
|
||||||
await _documentStore.UpdateStatusAsync(document.Id, DocumentStatuses.Failed, cancellationToken).ConfigureAwait(false);
|
await _documentStore.UpdateStatusAsync(document.Id, DocumentStatuses.Failed, cancellationToken).ConfigureAwait(false);
|
||||||
pendingMappings.Remove(documentId);
|
pendingMappings.Remove(documentId);
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
|
|
||||||
var (advisory, flag) = JvnAdvisoryMapper.Map(detail, document, dto, _timeProvider);
|
var (advisory, flag) = JvnAdvisoryMapper.Map(detail, document, dto, _timeProvider);
|
||||||
|
|
||||||
await _advisoryStore.UpsertAsync(advisory, cancellationToken).ConfigureAwait(false);
|
await _advisoryStore.UpsertAsync(advisory, cancellationToken).ConfigureAwait(false);
|
||||||
await _jpFlagStore.UpsertAsync(flag, cancellationToken).ConfigureAwait(false);
|
await _jpFlagStore.UpsertAsync(flag, cancellationToken).ConfigureAwait(false);
|
||||||
await _documentStore.UpdateStatusAsync(document.Id, DocumentStatuses.Mapped, cancellationToken).ConfigureAwait(false);
|
await _documentStore.UpdateStatusAsync(document.Id, DocumentStatuses.Mapped, cancellationToken).ConfigureAwait(false);
|
||||||
|
|
||||||
pendingMappings.Remove(documentId);
|
pendingMappings.Remove(documentId);
|
||||||
_logger.LogDebug("JVN mapped document {DocumentId}", documentId);
|
_logger.LogDebug("JVN mapped document {DocumentId}", documentId);
|
||||||
}
|
}
|
||||||
|
|
||||||
var updatedCursor = cursor.WithPendingMappings(pendingMappings);
|
var updatedCursor = cursor.WithPendingMappings(pendingMappings);
|
||||||
await UpdateCursorAsync(updatedCursor, cancellationToken).ConfigureAwait(false);
|
await UpdateCursorAsync(updatedCursor, cancellationToken).ConfigureAwait(false);
|
||||||
}
|
}
|
||||||
|
|
||||||
private async Task<JvnCursor> GetCursorAsync(CancellationToken cancellationToken)
|
private async Task<JvnCursor> GetCursorAsync(CancellationToken cancellationToken)
|
||||||
{
|
{
|
||||||
var state = await _stateRepository.TryGetAsync(SourceName, cancellationToken).ConfigureAwait(false);
|
var state = await _stateRepository.TryGetAsync(SourceName, cancellationToken).ConfigureAwait(false);
|
||||||
return state is null ? JvnCursor.Empty : JvnCursor.FromBson(state.Cursor);
|
return state is null ? JvnCursor.Empty : JvnCursor.FromBson(state.Cursor);
|
||||||
}
|
}
|
||||||
|
|
||||||
private async Task UpdateCursorAsync(JvnCursor cursor, CancellationToken cancellationToken)
|
private async Task UpdateCursorAsync(JvnCursor cursor, CancellationToken cancellationToken)
|
||||||
{
|
{
|
||||||
var cursorDocument = cursor.ToBsonDocument();
|
var cursorDocument = cursor.ToBsonDocument();
|
||||||
await _stateRepository.UpdateCursorAsync(SourceName, cursorDocument, _timeProvider.GetUtcNow(), cancellationToken).ConfigureAwait(false);
|
await _stateRepository.UpdateCursorAsync(SourceName, cursorDocument, _timeProvider.GetUtcNow(), cancellationToken).ConfigureAwait(false);
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -1,441 +1,441 @@
|
|||||||
using System;
|
using System;
|
||||||
using System.Collections.Generic;
|
using System.Collections.Generic;
|
||||||
using System.Linq;
|
using System.Linq;
|
||||||
using System.Text.Json;
|
using System.Text.Json;
|
||||||
using System.Text.Json.Serialization;
|
using System.Text.Json.Serialization;
|
||||||
using System.Threading;
|
using System.Threading;
|
||||||
using System.Threading.Tasks;
|
using System.Threading.Tasks;
|
||||||
using Microsoft.Extensions.Logging;
|
using Microsoft.Extensions.Logging;
|
||||||
using Microsoft.Extensions.Options;
|
using Microsoft.Extensions.Options;
|
||||||
using MongoDB.Bson;
|
using MongoDB.Bson;
|
||||||
using StellaOps.Concelier.Models;
|
using StellaOps.Concelier.Models;
|
||||||
using StellaOps.Concelier.Connector.Common;
|
using StellaOps.Concelier.Connector.Common;
|
||||||
using StellaOps.Concelier.Connector.Common.Fetch;
|
using StellaOps.Concelier.Connector.Common.Fetch;
|
||||||
using StellaOps.Concelier.Connector.Common.Json;
|
using StellaOps.Concelier.Connector.Common.Json;
|
||||||
using StellaOps.Concelier.Connector.Kev.Configuration;
|
using StellaOps.Concelier.Connector.Kev.Configuration;
|
||||||
using StellaOps.Concelier.Connector.Kev.Internal;
|
using StellaOps.Concelier.Connector.Kev.Internal;
|
||||||
using StellaOps.Concelier.Storage.Mongo;
|
using StellaOps.Concelier.Storage.Mongo;
|
||||||
using StellaOps.Concelier.Storage.Mongo.Advisories;
|
using StellaOps.Concelier.Storage.Mongo.Advisories;
|
||||||
using StellaOps.Concelier.Storage.Mongo.Documents;
|
using StellaOps.Concelier.Storage.Mongo.Documents;
|
||||||
using StellaOps.Concelier.Storage.Mongo.Dtos;
|
using StellaOps.Concelier.Storage.Mongo.Dtos;
|
||||||
using StellaOps.Plugin;
|
using StellaOps.Plugin;
|
||||||
|
|
||||||
namespace StellaOps.Concelier.Connector.Kev;
|
namespace StellaOps.Concelier.Connector.Kev;
|
||||||
|
|
||||||
public sealed class KevConnector : IFeedConnector
|
public sealed class KevConnector : IFeedConnector
|
||||||
{
|
{
|
||||||
private static readonly JsonSerializerOptions SerializerOptions = new(JsonSerializerDefaults.Web)
|
private static readonly JsonSerializerOptions SerializerOptions = new(JsonSerializerDefaults.Web)
|
||||||
{
|
{
|
||||||
PropertyNameCaseInsensitive = true,
|
PropertyNameCaseInsensitive = true,
|
||||||
DefaultIgnoreCondition = JsonIgnoreCondition.WhenWritingNull,
|
DefaultIgnoreCondition = JsonIgnoreCondition.WhenWritingNull,
|
||||||
};
|
};
|
||||||
|
|
||||||
private const string SchemaVersion = "kev.catalog.v1";
|
private const string SchemaVersion = "kev.catalog.v1";
|
||||||
|
|
||||||
private readonly SourceFetchService _fetchService;
|
private readonly SourceFetchService _fetchService;
|
||||||
private readonly RawDocumentStorage _rawDocumentStorage;
|
private readonly RawDocumentStorage _rawDocumentStorage;
|
||||||
private readonly IDocumentStore _documentStore;
|
private readonly IDocumentStore _documentStore;
|
||||||
private readonly IDtoStore _dtoStore;
|
private readonly IDtoStore _dtoStore;
|
||||||
private readonly IAdvisoryStore _advisoryStore;
|
private readonly IAdvisoryStore _advisoryStore;
|
||||||
private readonly ISourceStateRepository _stateRepository;
|
private readonly ISourceStateRepository _stateRepository;
|
||||||
private readonly KevOptions _options;
|
private readonly KevOptions _options;
|
||||||
private readonly IJsonSchemaValidator _schemaValidator;
|
private readonly IJsonSchemaValidator _schemaValidator;
|
||||||
private readonly TimeProvider _timeProvider;
|
private readonly TimeProvider _timeProvider;
|
||||||
private readonly ILogger<KevConnector> _logger;
|
private readonly ILogger<KevConnector> _logger;
|
||||||
private readonly KevDiagnostics _diagnostics;
|
private readonly KevDiagnostics _diagnostics;
|
||||||
|
|
||||||
public KevConnector(
|
public KevConnector(
|
||||||
SourceFetchService fetchService,
|
SourceFetchService fetchService,
|
||||||
RawDocumentStorage rawDocumentStorage,
|
RawDocumentStorage rawDocumentStorage,
|
||||||
IDocumentStore documentStore,
|
IDocumentStore documentStore,
|
||||||
IDtoStore dtoStore,
|
IDtoStore dtoStore,
|
||||||
IAdvisoryStore advisoryStore,
|
IAdvisoryStore advisoryStore,
|
||||||
ISourceStateRepository stateRepository,
|
ISourceStateRepository stateRepository,
|
||||||
IOptions<KevOptions> options,
|
IOptions<KevOptions> options,
|
||||||
IJsonSchemaValidator schemaValidator,
|
IJsonSchemaValidator schemaValidator,
|
||||||
KevDiagnostics diagnostics,
|
KevDiagnostics diagnostics,
|
||||||
TimeProvider? timeProvider,
|
TimeProvider? timeProvider,
|
||||||
ILogger<KevConnector> logger)
|
ILogger<KevConnector> logger)
|
||||||
{
|
{
|
||||||
_fetchService = fetchService ?? throw new ArgumentNullException(nameof(fetchService));
|
_fetchService = fetchService ?? throw new ArgumentNullException(nameof(fetchService));
|
||||||
_rawDocumentStorage = rawDocumentStorage ?? throw new ArgumentNullException(nameof(rawDocumentStorage));
|
_rawDocumentStorage = rawDocumentStorage ?? throw new ArgumentNullException(nameof(rawDocumentStorage));
|
||||||
_documentStore = documentStore ?? throw new ArgumentNullException(nameof(documentStore));
|
_documentStore = documentStore ?? throw new ArgumentNullException(nameof(documentStore));
|
||||||
_dtoStore = dtoStore ?? throw new ArgumentNullException(nameof(dtoStore));
|
_dtoStore = dtoStore ?? throw new ArgumentNullException(nameof(dtoStore));
|
||||||
_advisoryStore = advisoryStore ?? throw new ArgumentNullException(nameof(advisoryStore));
|
_advisoryStore = advisoryStore ?? throw new ArgumentNullException(nameof(advisoryStore));
|
||||||
_stateRepository = stateRepository ?? throw new ArgumentNullException(nameof(stateRepository));
|
_stateRepository = stateRepository ?? throw new ArgumentNullException(nameof(stateRepository));
|
||||||
_options = options?.Value ?? throw new ArgumentNullException(nameof(options));
|
_options = options?.Value ?? throw new ArgumentNullException(nameof(options));
|
||||||
_options.Validate();
|
_options.Validate();
|
||||||
_schemaValidator = schemaValidator ?? throw new ArgumentNullException(nameof(schemaValidator));
|
_schemaValidator = schemaValidator ?? throw new ArgumentNullException(nameof(schemaValidator));
|
||||||
_diagnostics = diagnostics ?? throw new ArgumentNullException(nameof(diagnostics));
|
_diagnostics = diagnostics ?? throw new ArgumentNullException(nameof(diagnostics));
|
||||||
_timeProvider = timeProvider ?? TimeProvider.System;
|
_timeProvider = timeProvider ?? TimeProvider.System;
|
||||||
_logger = logger ?? throw new ArgumentNullException(nameof(logger));
|
_logger = logger ?? throw new ArgumentNullException(nameof(logger));
|
||||||
}
|
}
|
||||||
|
|
||||||
public string SourceName => KevConnectorPlugin.SourceName;
|
public string SourceName => KevConnectorPlugin.SourceName;
|
||||||
|
|
||||||
public async Task FetchAsync(IServiceProvider services, CancellationToken cancellationToken)
|
public async Task FetchAsync(IServiceProvider services, CancellationToken cancellationToken)
|
||||||
{
|
{
|
||||||
ArgumentNullException.ThrowIfNull(services);
|
ArgumentNullException.ThrowIfNull(services);
|
||||||
|
|
||||||
var cursor = await GetCursorAsync(cancellationToken).ConfigureAwait(false);
|
var cursor = await GetCursorAsync(cancellationToken).ConfigureAwait(false);
|
||||||
var now = _timeProvider.GetUtcNow();
|
var now = _timeProvider.GetUtcNow();
|
||||||
|
|
||||||
try
|
try
|
||||||
{
|
{
|
||||||
var existing = await _documentStore.FindBySourceAndUriAsync(SourceName, _options.FeedUri.ToString(), cancellationToken).ConfigureAwait(false);
|
var existing = await _documentStore.FindBySourceAndUriAsync(SourceName, _options.FeedUri.ToString(), cancellationToken).ConfigureAwait(false);
|
||||||
|
|
||||||
var request = new SourceFetchRequest(
|
var request = new SourceFetchRequest(
|
||||||
KevOptions.HttpClientName,
|
KevOptions.HttpClientName,
|
||||||
SourceName,
|
SourceName,
|
||||||
_options.FeedUri)
|
_options.FeedUri)
|
||||||
{
|
{
|
||||||
Metadata = new Dictionary<string, string>(StringComparer.Ordinal)
|
Metadata = new Dictionary<string, string>(StringComparer.Ordinal)
|
||||||
{
|
{
|
||||||
["kev.cursor.catalogVersion"] = cursor.CatalogVersion ?? string.Empty,
|
["kev.cursor.catalogVersion"] = cursor.CatalogVersion ?? string.Empty,
|
||||||
["kev.cursor.catalogReleased"] = cursor.CatalogReleased?.ToString("O") ?? string.Empty,
|
["kev.cursor.catalogReleased"] = cursor.CatalogReleased?.ToString("O") ?? string.Empty,
|
||||||
},
|
},
|
||||||
ETag = existing?.Etag,
|
ETag = existing?.Etag,
|
||||||
LastModified = existing?.LastModified,
|
LastModified = existing?.LastModified,
|
||||||
TimeoutOverride = _options.RequestTimeout,
|
TimeoutOverride = _options.RequestTimeout,
|
||||||
AcceptHeaders = new[] { "application/json", "text/json" },
|
AcceptHeaders = new[] { "application/json", "text/json" },
|
||||||
};
|
};
|
||||||
|
|
||||||
_diagnostics.FetchAttempt();
|
_diagnostics.FetchAttempt();
|
||||||
var result = await _fetchService.FetchAsync(request, cancellationToken).ConfigureAwait(false);
|
var result = await _fetchService.FetchAsync(request, cancellationToken).ConfigureAwait(false);
|
||||||
if (result.IsNotModified)
|
if (result.IsNotModified)
|
||||||
{
|
{
|
||||||
_diagnostics.FetchUnchanged();
|
_diagnostics.FetchUnchanged();
|
||||||
_logger.LogInformation(
|
_logger.LogInformation(
|
||||||
"KEV catalog not modified (catalogVersion={CatalogVersion}, etag={Etag})",
|
"KEV catalog not modified (catalogVersion={CatalogVersion}, etag={Etag})",
|
||||||
cursor.CatalogVersion ?? "(unknown)",
|
cursor.CatalogVersion ?? "(unknown)",
|
||||||
existing?.Etag ?? "(none)");
|
existing?.Etag ?? "(none)");
|
||||||
await UpdateCursorAsync(cursor, cancellationToken).ConfigureAwait(false);
|
await UpdateCursorAsync(cursor, cancellationToken).ConfigureAwait(false);
|
||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
|
|
||||||
if (!result.IsSuccess || result.Document is null)
|
if (!result.IsSuccess || result.Document is null)
|
||||||
{
|
{
|
||||||
_diagnostics.FetchFailure();
|
_diagnostics.FetchFailure();
|
||||||
await _stateRepository.MarkFailureAsync(SourceName, now, TimeSpan.FromMinutes(5), "KEV feed returned no content.", cancellationToken).ConfigureAwait(false);
|
await _stateRepository.MarkFailureAsync(SourceName, now, TimeSpan.FromMinutes(5), "KEV feed returned no content.", cancellationToken).ConfigureAwait(false);
|
||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
|
|
||||||
_diagnostics.FetchSuccess();
|
_diagnostics.FetchSuccess();
|
||||||
|
|
||||||
var pendingDocuments = cursor.PendingDocuments.ToHashSet();
|
var pendingDocuments = cursor.PendingDocuments.ToHashSet();
|
||||||
var pendingMappings = cursor.PendingMappings.ToHashSet();
|
var pendingMappings = cursor.PendingMappings.ToHashSet();
|
||||||
var pendingDocumentsBefore = pendingDocuments.Count;
|
var pendingDocumentsBefore = pendingDocuments.Count;
|
||||||
var pendingMappingsBefore = pendingMappings.Count;
|
var pendingMappingsBefore = pendingMappings.Count;
|
||||||
|
|
||||||
pendingDocuments.Add(result.Document.Id);
|
pendingDocuments.Add(result.Document.Id);
|
||||||
|
|
||||||
var updatedCursor = cursor
|
var updatedCursor = cursor
|
||||||
.WithPendingDocuments(pendingDocuments)
|
.WithPendingDocuments(pendingDocuments)
|
||||||
.WithPendingMappings(pendingMappings);
|
.WithPendingMappings(pendingMappings);
|
||||||
|
|
||||||
var document = result.Document;
|
var document = result.Document;
|
||||||
var lastModified = document.LastModified?.ToUniversalTime().ToString("O") ?? "(unknown)";
|
var lastModified = document.LastModified?.ToUniversalTime().ToString("O") ?? "(unknown)";
|
||||||
_logger.LogInformation(
|
_logger.LogInformation(
|
||||||
"Fetched KEV catalog document {DocumentId} (etag={Etag}, lastModified={LastModified}) pendingDocuments={PendingDocumentsBefore}->{PendingDocumentsAfter} pendingMappings={PendingMappingsBefore}->{PendingMappingsAfter}",
|
"Fetched KEV catalog document {DocumentId} (etag={Etag}, lastModified={LastModified}) pendingDocuments={PendingDocumentsBefore}->{PendingDocumentsAfter} pendingMappings={PendingMappingsBefore}->{PendingMappingsAfter}",
|
||||||
document.Id,
|
document.Id,
|
||||||
document.Etag ?? "(none)",
|
document.Etag ?? "(none)",
|
||||||
lastModified,
|
lastModified,
|
||||||
pendingDocumentsBefore,
|
pendingDocumentsBefore,
|
||||||
pendingDocuments.Count,
|
pendingDocuments.Count,
|
||||||
pendingMappingsBefore,
|
pendingMappingsBefore,
|
||||||
pendingMappings.Count);
|
pendingMappings.Count);
|
||||||
|
|
||||||
await UpdateCursorAsync(updatedCursor, cancellationToken).ConfigureAwait(false);
|
await UpdateCursorAsync(updatedCursor, cancellationToken).ConfigureAwait(false);
|
||||||
}
|
}
|
||||||
catch (Exception ex)
|
catch (Exception ex)
|
||||||
{
|
{
|
||||||
_diagnostics.FetchFailure();
|
_diagnostics.FetchFailure();
|
||||||
_logger.LogError(ex, "KEV fetch failed for {Uri}", _options.FeedUri);
|
_logger.LogError(ex, "KEV fetch failed for {Uri}", _options.FeedUri);
|
||||||
await _stateRepository.MarkFailureAsync(SourceName, now, TimeSpan.FromMinutes(5), ex.Message, cancellationToken).ConfigureAwait(false);
|
await _stateRepository.MarkFailureAsync(SourceName, now, TimeSpan.FromMinutes(5), ex.Message, cancellationToken).ConfigureAwait(false);
|
||||||
throw;
|
throw;
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
public async Task ParseAsync(IServiceProvider services, CancellationToken cancellationToken)
|
public async Task ParseAsync(IServiceProvider services, CancellationToken cancellationToken)
|
||||||
{
|
{
|
||||||
ArgumentNullException.ThrowIfNull(services);
|
ArgumentNullException.ThrowIfNull(services);
|
||||||
|
|
||||||
var cursor = await GetCursorAsync(cancellationToken).ConfigureAwait(false);
|
var cursor = await GetCursorAsync(cancellationToken).ConfigureAwait(false);
|
||||||
if (cursor.PendingDocuments.Count == 0)
|
if (cursor.PendingDocuments.Count == 0)
|
||||||
{
|
{
|
||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
|
|
||||||
var remainingDocuments = cursor.PendingDocuments.ToList();
|
var remainingDocuments = cursor.PendingDocuments.ToList();
|
||||||
var pendingMappings = cursor.PendingMappings.ToHashSet();
|
var pendingMappings = cursor.PendingMappings.ToHashSet();
|
||||||
var latestCatalogVersion = cursor.CatalogVersion;
|
var latestCatalogVersion = cursor.CatalogVersion;
|
||||||
var latestCatalogReleased = cursor.CatalogReleased;
|
var latestCatalogReleased = cursor.CatalogReleased;
|
||||||
|
|
||||||
foreach (var documentId in cursor.PendingDocuments)
|
foreach (var documentId in cursor.PendingDocuments)
|
||||||
{
|
{
|
||||||
cancellationToken.ThrowIfCancellationRequested();
|
cancellationToken.ThrowIfCancellationRequested();
|
||||||
|
|
||||||
var document = await _documentStore.FindAsync(documentId, cancellationToken).ConfigureAwait(false);
|
var document = await _documentStore.FindAsync(documentId, cancellationToken).ConfigureAwait(false);
|
||||||
if (document is null)
|
if (document is null)
|
||||||
{
|
{
|
||||||
remainingDocuments.Remove(documentId);
|
remainingDocuments.Remove(documentId);
|
||||||
pendingMappings.Remove(documentId);
|
pendingMappings.Remove(documentId);
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
|
|
||||||
if (!document.GridFsId.HasValue)
|
if (!document.PayloadId.HasValue)
|
||||||
{
|
{
|
||||||
_diagnostics.ParseFailure("missingPayload", cursor.CatalogVersion);
|
_diagnostics.ParseFailure("missingPayload", cursor.CatalogVersion);
|
||||||
_logger.LogWarning("KEV document {DocumentId} missing GridFS payload", document.Id);
|
_logger.LogWarning("KEV document {DocumentId} missing GridFS payload", document.Id);
|
||||||
await _documentStore.UpdateStatusAsync(document.Id, DocumentStatuses.Failed, cancellationToken).ConfigureAwait(false);
|
await _documentStore.UpdateStatusAsync(document.Id, DocumentStatuses.Failed, cancellationToken).ConfigureAwait(false);
|
||||||
remainingDocuments.Remove(documentId);
|
remainingDocuments.Remove(documentId);
|
||||||
pendingMappings.Remove(documentId);
|
pendingMappings.Remove(documentId);
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
|
|
||||||
byte[] rawBytes;
|
byte[] rawBytes;
|
||||||
try
|
try
|
||||||
{
|
{
|
||||||
rawBytes = await _rawDocumentStorage.DownloadAsync(document.GridFsId.Value, cancellationToken).ConfigureAwait(false);
|
rawBytes = await _rawDocumentStorage.DownloadAsync(document.PayloadId.Value, cancellationToken).ConfigureAwait(false);
|
||||||
}
|
}
|
||||||
catch (Exception ex)
|
catch (Exception ex)
|
||||||
{
|
{
|
||||||
_diagnostics.ParseFailure("download", cursor.CatalogVersion);
|
_diagnostics.ParseFailure("download", cursor.CatalogVersion);
|
||||||
_logger.LogError(ex, "KEV parse failed for document {DocumentId}", document.Id);
|
_logger.LogError(ex, "KEV parse failed for document {DocumentId}", document.Id);
|
||||||
await _documentStore.UpdateStatusAsync(document.Id, DocumentStatuses.Failed, cancellationToken).ConfigureAwait(false);
|
await _documentStore.UpdateStatusAsync(document.Id, DocumentStatuses.Failed, cancellationToken).ConfigureAwait(false);
|
||||||
remainingDocuments.Remove(documentId);
|
remainingDocuments.Remove(documentId);
|
||||||
pendingMappings.Remove(documentId);
|
pendingMappings.Remove(documentId);
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
|
|
||||||
KevCatalogDto? catalog = null;
|
KevCatalogDto? catalog = null;
|
||||||
string? catalogVersion = null;
|
string? catalogVersion = null;
|
||||||
try
|
try
|
||||||
{
|
{
|
||||||
using var jsonDocument = JsonDocument.Parse(rawBytes);
|
using var jsonDocument = JsonDocument.Parse(rawBytes);
|
||||||
catalogVersion = TryGetCatalogVersion(jsonDocument.RootElement);
|
catalogVersion = TryGetCatalogVersion(jsonDocument.RootElement);
|
||||||
_schemaValidator.Validate(jsonDocument, KevSchemaProvider.Schema, document.Uri);
|
_schemaValidator.Validate(jsonDocument, KevSchemaProvider.Schema, document.Uri);
|
||||||
catalog = jsonDocument.RootElement.Deserialize<KevCatalogDto>(SerializerOptions);
|
catalog = jsonDocument.RootElement.Deserialize<KevCatalogDto>(SerializerOptions);
|
||||||
}
|
}
|
||||||
catch (JsonSchemaValidationException ex)
|
catch (JsonSchemaValidationException ex)
|
||||||
{
|
{
|
||||||
_diagnostics.ParseFailure("schema", catalogVersion);
|
_diagnostics.ParseFailure("schema", catalogVersion);
|
||||||
_logger.LogWarning(ex, "KEV schema validation failed for document {DocumentId}", document.Id);
|
_logger.LogWarning(ex, "KEV schema validation failed for document {DocumentId}", document.Id);
|
||||||
await _documentStore.UpdateStatusAsync(document.Id, DocumentStatuses.Failed, cancellationToken).ConfigureAwait(false);
|
await _documentStore.UpdateStatusAsync(document.Id, DocumentStatuses.Failed, cancellationToken).ConfigureAwait(false);
|
||||||
remainingDocuments.Remove(documentId);
|
remainingDocuments.Remove(documentId);
|
||||||
pendingMappings.Remove(documentId);
|
pendingMappings.Remove(documentId);
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
catch (JsonException ex)
|
catch (JsonException ex)
|
||||||
{
|
{
|
||||||
_diagnostics.ParseFailure("invalidJson", catalogVersion);
|
_diagnostics.ParseFailure("invalidJson", catalogVersion);
|
||||||
_logger.LogError(ex, "KEV JSON parsing failed for document {DocumentId}", document.Id);
|
_logger.LogError(ex, "KEV JSON parsing failed for document {DocumentId}", document.Id);
|
||||||
await _documentStore.UpdateStatusAsync(document.Id, DocumentStatuses.Failed, cancellationToken).ConfigureAwait(false);
|
await _documentStore.UpdateStatusAsync(document.Id, DocumentStatuses.Failed, cancellationToken).ConfigureAwait(false);
|
||||||
remainingDocuments.Remove(documentId);
|
remainingDocuments.Remove(documentId);
|
||||||
pendingMappings.Remove(documentId);
|
pendingMappings.Remove(documentId);
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
catch (Exception ex)
|
catch (Exception ex)
|
||||||
{
|
{
|
||||||
_diagnostics.ParseFailure("deserialize", catalogVersion);
|
_diagnostics.ParseFailure("deserialize", catalogVersion);
|
||||||
_logger.LogError(ex, "KEV catalog deserialization failed for document {DocumentId}", document.Id);
|
_logger.LogError(ex, "KEV catalog deserialization failed for document {DocumentId}", document.Id);
|
||||||
await _documentStore.UpdateStatusAsync(document.Id, DocumentStatuses.Failed, cancellationToken).ConfigureAwait(false);
|
await _documentStore.UpdateStatusAsync(document.Id, DocumentStatuses.Failed, cancellationToken).ConfigureAwait(false);
|
||||||
remainingDocuments.Remove(documentId);
|
remainingDocuments.Remove(documentId);
|
||||||
pendingMappings.Remove(documentId);
|
pendingMappings.Remove(documentId);
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
|
|
||||||
if (catalog is null)
|
if (catalog is null)
|
||||||
{
|
{
|
||||||
_diagnostics.ParseFailure("emptyCatalog", catalogVersion);
|
_diagnostics.ParseFailure("emptyCatalog", catalogVersion);
|
||||||
_logger.LogWarning("KEV catalog payload was empty for document {DocumentId}", document.Id);
|
_logger.LogWarning("KEV catalog payload was empty for document {DocumentId}", document.Id);
|
||||||
await _documentStore.UpdateStatusAsync(document.Id, DocumentStatuses.Failed, cancellationToken).ConfigureAwait(false);
|
await _documentStore.UpdateStatusAsync(document.Id, DocumentStatuses.Failed, cancellationToken).ConfigureAwait(false);
|
||||||
remainingDocuments.Remove(documentId);
|
remainingDocuments.Remove(documentId);
|
||||||
pendingMappings.Remove(documentId);
|
pendingMappings.Remove(documentId);
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
|
|
||||||
var entryCount = catalog.Vulnerabilities?.Count ?? 0;
|
var entryCount = catalog.Vulnerabilities?.Count ?? 0;
|
||||||
var released = catalog.DateReleased?.ToUniversalTime();
|
var released = catalog.DateReleased?.ToUniversalTime();
|
||||||
RecordCatalogAnomalies(catalog);
|
RecordCatalogAnomalies(catalog);
|
||||||
|
|
||||||
try
|
try
|
||||||
{
|
{
|
||||||
var payloadJson = JsonSerializer.Serialize(catalog, SerializerOptions);
|
var payloadJson = JsonSerializer.Serialize(catalog, SerializerOptions);
|
||||||
var payload = BsonDocument.Parse(payloadJson);
|
var payload = BsonDocument.Parse(payloadJson);
|
||||||
|
|
||||||
_logger.LogInformation(
|
_logger.LogInformation(
|
||||||
"Parsed KEV catalog document {DocumentId} (version={CatalogVersion}, released={Released}, entries={EntryCount})",
|
"Parsed KEV catalog document {DocumentId} (version={CatalogVersion}, released={Released}, entries={EntryCount})",
|
||||||
document.Id,
|
document.Id,
|
||||||
catalog.CatalogVersion ?? "(unknown)",
|
catalog.CatalogVersion ?? "(unknown)",
|
||||||
released,
|
released,
|
||||||
entryCount);
|
entryCount);
|
||||||
_diagnostics.CatalogParsed(catalog.CatalogVersion, entryCount);
|
_diagnostics.CatalogParsed(catalog.CatalogVersion, entryCount);
|
||||||
|
|
||||||
var dtoRecord = new DtoRecord(
|
var dtoRecord = new DtoRecord(
|
||||||
Guid.NewGuid(),
|
Guid.NewGuid(),
|
||||||
document.Id,
|
document.Id,
|
||||||
SourceName,
|
SourceName,
|
||||||
SchemaVersion,
|
SchemaVersion,
|
||||||
payload,
|
payload,
|
||||||
_timeProvider.GetUtcNow());
|
_timeProvider.GetUtcNow());
|
||||||
|
|
||||||
await _dtoStore.UpsertAsync(dtoRecord, cancellationToken).ConfigureAwait(false);
|
await _dtoStore.UpsertAsync(dtoRecord, cancellationToken).ConfigureAwait(false);
|
||||||
await _documentStore.UpdateStatusAsync(document.Id, DocumentStatuses.PendingMap, cancellationToken).ConfigureAwait(false);
|
await _documentStore.UpdateStatusAsync(document.Id, DocumentStatuses.PendingMap, cancellationToken).ConfigureAwait(false);
|
||||||
|
|
||||||
remainingDocuments.Remove(documentId);
|
remainingDocuments.Remove(documentId);
|
||||||
pendingMappings.Add(document.Id);
|
pendingMappings.Add(document.Id);
|
||||||
|
|
||||||
latestCatalogVersion = catalog.CatalogVersion ?? latestCatalogVersion;
|
latestCatalogVersion = catalog.CatalogVersion ?? latestCatalogVersion;
|
||||||
latestCatalogReleased = catalog.DateReleased ?? latestCatalogReleased;
|
latestCatalogReleased = catalog.DateReleased ?? latestCatalogReleased;
|
||||||
}
|
}
|
||||||
catch (Exception ex)
|
catch (Exception ex)
|
||||||
{
|
{
|
||||||
_logger.LogError(ex, "KEV DTO persistence failed for document {DocumentId}", document.Id);
|
_logger.LogError(ex, "KEV DTO persistence failed for document {DocumentId}", document.Id);
|
||||||
await _documentStore.UpdateStatusAsync(document.Id, DocumentStatuses.Failed, cancellationToken).ConfigureAwait(false);
|
await _documentStore.UpdateStatusAsync(document.Id, DocumentStatuses.Failed, cancellationToken).ConfigureAwait(false);
|
||||||
remainingDocuments.Remove(documentId);
|
remainingDocuments.Remove(documentId);
|
||||||
pendingMappings.Remove(documentId);
|
pendingMappings.Remove(documentId);
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
var updatedCursor = cursor
|
var updatedCursor = cursor
|
||||||
.WithPendingDocuments(remainingDocuments)
|
.WithPendingDocuments(remainingDocuments)
|
||||||
.WithPendingMappings(pendingMappings)
|
.WithPendingMappings(pendingMappings)
|
||||||
.WithCatalogMetadata(latestCatalogVersion, latestCatalogReleased);
|
.WithCatalogMetadata(latestCatalogVersion, latestCatalogReleased);
|
||||||
|
|
||||||
await UpdateCursorAsync(updatedCursor, cancellationToken).ConfigureAwait(false);
|
await UpdateCursorAsync(updatedCursor, cancellationToken).ConfigureAwait(false);
|
||||||
}
|
}
|
||||||
|
|
||||||
public async Task MapAsync(IServiceProvider services, CancellationToken cancellationToken)
|
public async Task MapAsync(IServiceProvider services, CancellationToken cancellationToken)
|
||||||
{
|
{
|
||||||
ArgumentNullException.ThrowIfNull(services);
|
ArgumentNullException.ThrowIfNull(services);
|
||||||
|
|
||||||
var cursor = await GetCursorAsync(cancellationToken).ConfigureAwait(false);
|
var cursor = await GetCursorAsync(cancellationToken).ConfigureAwait(false);
|
||||||
if (cursor.PendingMappings.Count == 0)
|
if (cursor.PendingMappings.Count == 0)
|
||||||
{
|
{
|
||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
|
|
||||||
var pendingMappings = cursor.PendingMappings.ToHashSet();
|
var pendingMappings = cursor.PendingMappings.ToHashSet();
|
||||||
|
|
||||||
foreach (var documentId in cursor.PendingMappings)
|
foreach (var documentId in cursor.PendingMappings)
|
||||||
{
|
{
|
||||||
cancellationToken.ThrowIfCancellationRequested();
|
cancellationToken.ThrowIfCancellationRequested();
|
||||||
|
|
||||||
var dtoRecord = await _dtoStore.FindByDocumentIdAsync(documentId, cancellationToken).ConfigureAwait(false);
|
var dtoRecord = await _dtoStore.FindByDocumentIdAsync(documentId, cancellationToken).ConfigureAwait(false);
|
||||||
var document = await _documentStore.FindAsync(documentId, cancellationToken).ConfigureAwait(false);
|
var document = await _documentStore.FindAsync(documentId, cancellationToken).ConfigureAwait(false);
|
||||||
|
|
||||||
if (dtoRecord is null || document is null)
|
if (dtoRecord is null || document is null)
|
||||||
{
|
{
|
||||||
pendingMappings.Remove(documentId);
|
pendingMappings.Remove(documentId);
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
|
|
||||||
KevCatalogDto? catalog;
|
KevCatalogDto? catalog;
|
||||||
try
|
try
|
||||||
{
|
{
|
||||||
var dtoJson = dtoRecord.Payload.ToJson(new MongoDB.Bson.IO.JsonWriterSettings
|
var dtoJson = dtoRecord.Payload.ToJson(new MongoDB.Bson.IO.JsonWriterSettings
|
||||||
{
|
{
|
||||||
OutputMode = MongoDB.Bson.IO.JsonOutputMode.RelaxedExtendedJson,
|
OutputMode = MongoDB.Bson.IO.JsonOutputMode.RelaxedExtendedJson,
|
||||||
});
|
});
|
||||||
|
|
||||||
catalog = JsonSerializer.Deserialize<KevCatalogDto>(dtoJson, SerializerOptions);
|
catalog = JsonSerializer.Deserialize<KevCatalogDto>(dtoJson, SerializerOptions);
|
||||||
}
|
}
|
||||||
catch (Exception ex)
|
catch (Exception ex)
|
||||||
{
|
{
|
||||||
_logger.LogError(ex, "KEV mapping: failed to deserialize DTO for document {DocumentId}", document.Id);
|
_logger.LogError(ex, "KEV mapping: failed to deserialize DTO for document {DocumentId}", document.Id);
|
||||||
await _documentStore.UpdateStatusAsync(document.Id, DocumentStatuses.Failed, cancellationToken).ConfigureAwait(false);
|
await _documentStore.UpdateStatusAsync(document.Id, DocumentStatuses.Failed, cancellationToken).ConfigureAwait(false);
|
||||||
pendingMappings.Remove(documentId);
|
pendingMappings.Remove(documentId);
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
|
|
||||||
if (catalog is null)
|
if (catalog is null)
|
||||||
{
|
{
|
||||||
_logger.LogWarning("KEV mapping: DTO payload was empty for document {DocumentId}", document.Id);
|
_logger.LogWarning("KEV mapping: DTO payload was empty for document {DocumentId}", document.Id);
|
||||||
await _documentStore.UpdateStatusAsync(document.Id, DocumentStatuses.Failed, cancellationToken).ConfigureAwait(false);
|
await _documentStore.UpdateStatusAsync(document.Id, DocumentStatuses.Failed, cancellationToken).ConfigureAwait(false);
|
||||||
pendingMappings.Remove(documentId);
|
pendingMappings.Remove(documentId);
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
|
|
||||||
var feedUri = TryParseUri(document.Uri) ?? _options.FeedUri;
|
var feedUri = TryParseUri(document.Uri) ?? _options.FeedUri;
|
||||||
var advisories = KevMapper.Map(catalog, SourceName, feedUri, document.FetchedAt, dtoRecord.ValidatedAt);
|
var advisories = KevMapper.Map(catalog, SourceName, feedUri, document.FetchedAt, dtoRecord.ValidatedAt);
|
||||||
var entryCount = catalog.Vulnerabilities?.Count ?? 0;
|
var entryCount = catalog.Vulnerabilities?.Count ?? 0;
|
||||||
var mappedCount = advisories.Count;
|
var mappedCount = advisories.Count;
|
||||||
var skippedCount = Math.Max(0, entryCount - mappedCount);
|
var skippedCount = Math.Max(0, entryCount - mappedCount);
|
||||||
_logger.LogInformation(
|
_logger.LogInformation(
|
||||||
"Mapped {MappedCount}/{EntryCount} KEV advisories from catalog version {CatalogVersion} (skipped={SkippedCount})",
|
"Mapped {MappedCount}/{EntryCount} KEV advisories from catalog version {CatalogVersion} (skipped={SkippedCount})",
|
||||||
mappedCount,
|
mappedCount,
|
||||||
entryCount,
|
entryCount,
|
||||||
catalog.CatalogVersion ?? "(unknown)",
|
catalog.CatalogVersion ?? "(unknown)",
|
||||||
skippedCount);
|
skippedCount);
|
||||||
_diagnostics.AdvisoriesMapped(catalog.CatalogVersion, mappedCount);
|
_diagnostics.AdvisoriesMapped(catalog.CatalogVersion, mappedCount);
|
||||||
|
|
||||||
foreach (var advisory in advisories)
|
foreach (var advisory in advisories)
|
||||||
{
|
{
|
||||||
await _advisoryStore.UpsertAsync(advisory, cancellationToken).ConfigureAwait(false);
|
await _advisoryStore.UpsertAsync(advisory, cancellationToken).ConfigureAwait(false);
|
||||||
}
|
}
|
||||||
|
|
||||||
await _documentStore.UpdateStatusAsync(document.Id, DocumentStatuses.Mapped, cancellationToken).ConfigureAwait(false);
|
await _documentStore.UpdateStatusAsync(document.Id, DocumentStatuses.Mapped, cancellationToken).ConfigureAwait(false);
|
||||||
pendingMappings.Remove(documentId);
|
pendingMappings.Remove(documentId);
|
||||||
}
|
}
|
||||||
|
|
||||||
var updatedCursor = cursor.WithPendingMappings(pendingMappings);
|
var updatedCursor = cursor.WithPendingMappings(pendingMappings);
|
||||||
await UpdateCursorAsync(updatedCursor, cancellationToken).ConfigureAwait(false);
|
await UpdateCursorAsync(updatedCursor, cancellationToken).ConfigureAwait(false);
|
||||||
}
|
}
|
||||||
|
|
||||||
private async Task<KevCursor> GetCursorAsync(CancellationToken cancellationToken)
|
private async Task<KevCursor> GetCursorAsync(CancellationToken cancellationToken)
|
||||||
{
|
{
|
||||||
var state = await _stateRepository.TryGetAsync(SourceName, cancellationToken).ConfigureAwait(false);
|
var state = await _stateRepository.TryGetAsync(SourceName, cancellationToken).ConfigureAwait(false);
|
||||||
return state is null ? KevCursor.Empty : KevCursor.FromBson(state.Cursor);
|
return state is null ? KevCursor.Empty : KevCursor.FromBson(state.Cursor);
|
||||||
}
|
}
|
||||||
|
|
||||||
private Task UpdateCursorAsync(KevCursor cursor, CancellationToken cancellationToken)
|
private Task UpdateCursorAsync(KevCursor cursor, CancellationToken cancellationToken)
|
||||||
{
|
{
|
||||||
return _stateRepository.UpdateCursorAsync(SourceName, cursor.ToBsonDocument(), _timeProvider.GetUtcNow(), cancellationToken);
|
return _stateRepository.UpdateCursorAsync(SourceName, cursor.ToBsonDocument(), _timeProvider.GetUtcNow(), cancellationToken);
|
||||||
}
|
}
|
||||||
|
|
||||||
private void RecordCatalogAnomalies(KevCatalogDto catalog)
|
private void RecordCatalogAnomalies(KevCatalogDto catalog)
|
||||||
{
|
{
|
||||||
ArgumentNullException.ThrowIfNull(catalog);
|
ArgumentNullException.ThrowIfNull(catalog);
|
||||||
|
|
||||||
var version = catalog.CatalogVersion;
|
var version = catalog.CatalogVersion;
|
||||||
var vulnerabilities = catalog.Vulnerabilities ?? Array.Empty<KevVulnerabilityDto>();
|
var vulnerabilities = catalog.Vulnerabilities ?? Array.Empty<KevVulnerabilityDto>();
|
||||||
|
|
||||||
if (catalog.Count != vulnerabilities.Count)
|
if (catalog.Count != vulnerabilities.Count)
|
||||||
{
|
{
|
||||||
_diagnostics.RecordAnomaly("countMismatch", version);
|
_diagnostics.RecordAnomaly("countMismatch", version);
|
||||||
}
|
}
|
||||||
|
|
||||||
foreach (var entry in vulnerabilities)
|
foreach (var entry in vulnerabilities)
|
||||||
{
|
{
|
||||||
if (entry is null)
|
if (entry is null)
|
||||||
{
|
{
|
||||||
_diagnostics.RecordAnomaly("nullEntry", version);
|
_diagnostics.RecordAnomaly("nullEntry", version);
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
|
|
||||||
if (string.IsNullOrWhiteSpace(entry.CveId))
|
if (string.IsNullOrWhiteSpace(entry.CveId))
|
||||||
{
|
{
|
||||||
_diagnostics.RecordAnomaly("missingCveId", version);
|
_diagnostics.RecordAnomaly("missingCveId", version);
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
private static string? TryGetCatalogVersion(JsonElement root)
|
private static string? TryGetCatalogVersion(JsonElement root)
|
||||||
{
|
{
|
||||||
if (root.ValueKind != JsonValueKind.Object)
|
if (root.ValueKind != JsonValueKind.Object)
|
||||||
{
|
{
|
||||||
return null;
|
return null;
|
||||||
}
|
}
|
||||||
|
|
||||||
if (root.TryGetProperty("catalogVersion", out var versionElement) && versionElement.ValueKind == JsonValueKind.String)
|
if (root.TryGetProperty("catalogVersion", out var versionElement) && versionElement.ValueKind == JsonValueKind.String)
|
||||||
{
|
{
|
||||||
return versionElement.GetString();
|
return versionElement.GetString();
|
||||||
}
|
}
|
||||||
|
|
||||||
return null;
|
return null;
|
||||||
}
|
}
|
||||||
|
|
||||||
private static Uri? TryParseUri(string? value)
|
private static Uri? TryParseUri(string? value)
|
||||||
=> Uri.TryCreate(value, UriKind.Absolute, out var uri) ? uri : null;
|
=> Uri.TryCreate(value, UriKind.Absolute, out var uri) ? uri : null;
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -1,136 +1,136 @@
|
|||||||
using System;
|
using System;
|
||||||
using System.Collections.Generic;
|
using System.Collections.Generic;
|
||||||
using System.Linq;
|
using System.Linq;
|
||||||
using System.Text.Json;
|
using System.Text.Json;
|
||||||
using System.Threading;
|
using System.Threading;
|
||||||
using System.Threading.Tasks;
|
using System.Threading.Tasks;
|
||||||
using Microsoft.Extensions.Logging;
|
using Microsoft.Extensions.Logging;
|
||||||
using Microsoft.Extensions.Options;
|
using Microsoft.Extensions.Options;
|
||||||
using MongoDB.Bson;
|
using MongoDB.Bson;
|
||||||
using StellaOps.Concelier.Connector.Common;
|
using StellaOps.Concelier.Connector.Common;
|
||||||
using StellaOps.Concelier.Connector.Common.Fetch;
|
using StellaOps.Concelier.Connector.Common.Fetch;
|
||||||
using StellaOps.Concelier.Connector.Kisa.Configuration;
|
using StellaOps.Concelier.Connector.Kisa.Configuration;
|
||||||
using StellaOps.Concelier.Connector.Kisa.Internal;
|
using StellaOps.Concelier.Connector.Kisa.Internal;
|
||||||
using StellaOps.Concelier.Storage.Mongo;
|
using StellaOps.Concelier.Storage.Mongo;
|
||||||
using StellaOps.Concelier.Storage.Mongo.Advisories;
|
using StellaOps.Concelier.Storage.Mongo.Advisories;
|
||||||
using StellaOps.Concelier.Storage.Mongo.Documents;
|
using StellaOps.Concelier.Storage.Mongo.Documents;
|
||||||
using StellaOps.Concelier.Storage.Mongo.Dtos;
|
using StellaOps.Concelier.Storage.Mongo.Dtos;
|
||||||
using StellaOps.Plugin;
|
using StellaOps.Plugin;
|
||||||
|
|
||||||
namespace StellaOps.Concelier.Connector.Kisa;
|
namespace StellaOps.Concelier.Connector.Kisa;
|
||||||
|
|
||||||
public sealed class KisaConnector : IFeedConnector
|
public sealed class KisaConnector : IFeedConnector
|
||||||
{
|
{
|
||||||
private static readonly JsonSerializerOptions SerializerOptions = new(JsonSerializerDefaults.Web)
|
private static readonly JsonSerializerOptions SerializerOptions = new(JsonSerializerDefaults.Web)
|
||||||
{
|
{
|
||||||
PropertyNameCaseInsensitive = true,
|
PropertyNameCaseInsensitive = true,
|
||||||
DefaultIgnoreCondition = System.Text.Json.Serialization.JsonIgnoreCondition.WhenWritingNull,
|
DefaultIgnoreCondition = System.Text.Json.Serialization.JsonIgnoreCondition.WhenWritingNull,
|
||||||
};
|
};
|
||||||
|
|
||||||
private readonly KisaFeedClient _feedClient;
|
private readonly KisaFeedClient _feedClient;
|
||||||
private readonly KisaDetailParser _detailParser;
|
private readonly KisaDetailParser _detailParser;
|
||||||
private readonly SourceFetchService _fetchService;
|
private readonly SourceFetchService _fetchService;
|
||||||
private readonly RawDocumentStorage _rawDocumentStorage;
|
private readonly RawDocumentStorage _rawDocumentStorage;
|
||||||
private readonly IDocumentStore _documentStore;
|
private readonly IDocumentStore _documentStore;
|
||||||
private readonly IDtoStore _dtoStore;
|
private readonly IDtoStore _dtoStore;
|
||||||
private readonly IAdvisoryStore _advisoryStore;
|
private readonly IAdvisoryStore _advisoryStore;
|
||||||
private readonly ISourceStateRepository _stateRepository;
|
private readonly ISourceStateRepository _stateRepository;
|
||||||
private readonly KisaOptions _options;
|
private readonly KisaOptions _options;
|
||||||
private readonly KisaDiagnostics _diagnostics;
|
private readonly KisaDiagnostics _diagnostics;
|
||||||
private readonly TimeProvider _timeProvider;
|
private readonly TimeProvider _timeProvider;
|
||||||
private readonly ILogger<KisaConnector> _logger;
|
private readonly ILogger<KisaConnector> _logger;
|
||||||
|
|
||||||
public KisaConnector(
|
public KisaConnector(
|
||||||
KisaFeedClient feedClient,
|
KisaFeedClient feedClient,
|
||||||
KisaDetailParser detailParser,
|
KisaDetailParser detailParser,
|
||||||
SourceFetchService fetchService,
|
SourceFetchService fetchService,
|
||||||
RawDocumentStorage rawDocumentStorage,
|
RawDocumentStorage rawDocumentStorage,
|
||||||
IDocumentStore documentStore,
|
IDocumentStore documentStore,
|
||||||
IDtoStore dtoStore,
|
IDtoStore dtoStore,
|
||||||
IAdvisoryStore advisoryStore,
|
IAdvisoryStore advisoryStore,
|
||||||
ISourceStateRepository stateRepository,
|
ISourceStateRepository stateRepository,
|
||||||
IOptions<KisaOptions> options,
|
IOptions<KisaOptions> options,
|
||||||
KisaDiagnostics diagnostics,
|
KisaDiagnostics diagnostics,
|
||||||
TimeProvider? timeProvider,
|
TimeProvider? timeProvider,
|
||||||
ILogger<KisaConnector> logger)
|
ILogger<KisaConnector> logger)
|
||||||
{
|
{
|
||||||
_feedClient = feedClient ?? throw new ArgumentNullException(nameof(feedClient));
|
_feedClient = feedClient ?? throw new ArgumentNullException(nameof(feedClient));
|
||||||
_detailParser = detailParser ?? throw new ArgumentNullException(nameof(detailParser));
|
_detailParser = detailParser ?? throw new ArgumentNullException(nameof(detailParser));
|
||||||
_fetchService = fetchService ?? throw new ArgumentNullException(nameof(fetchService));
|
_fetchService = fetchService ?? throw new ArgumentNullException(nameof(fetchService));
|
||||||
_rawDocumentStorage = rawDocumentStorage ?? throw new ArgumentNullException(nameof(rawDocumentStorage));
|
_rawDocumentStorage = rawDocumentStorage ?? throw new ArgumentNullException(nameof(rawDocumentStorage));
|
||||||
_documentStore = documentStore ?? throw new ArgumentNullException(nameof(documentStore));
|
_documentStore = documentStore ?? throw new ArgumentNullException(nameof(documentStore));
|
||||||
_dtoStore = dtoStore ?? throw new ArgumentNullException(nameof(dtoStore));
|
_dtoStore = dtoStore ?? throw new ArgumentNullException(nameof(dtoStore));
|
||||||
_advisoryStore = advisoryStore ?? throw new ArgumentNullException(nameof(advisoryStore));
|
_advisoryStore = advisoryStore ?? throw new ArgumentNullException(nameof(advisoryStore));
|
||||||
_stateRepository = stateRepository ?? throw new ArgumentNullException(nameof(stateRepository));
|
_stateRepository = stateRepository ?? throw new ArgumentNullException(nameof(stateRepository));
|
||||||
_options = (options ?? throw new ArgumentNullException(nameof(options))).Value ?? throw new ArgumentNullException(nameof(options));
|
_options = (options ?? throw new ArgumentNullException(nameof(options))).Value ?? throw new ArgumentNullException(nameof(options));
|
||||||
_options.Validate();
|
_options.Validate();
|
||||||
_diagnostics = diagnostics ?? throw new ArgumentNullException(nameof(diagnostics));
|
_diagnostics = diagnostics ?? throw new ArgumentNullException(nameof(diagnostics));
|
||||||
_timeProvider = timeProvider ?? TimeProvider.System;
|
_timeProvider = timeProvider ?? TimeProvider.System;
|
||||||
_logger = logger ?? throw new ArgumentNullException(nameof(logger));
|
_logger = logger ?? throw new ArgumentNullException(nameof(logger));
|
||||||
}
|
}
|
||||||
|
|
||||||
public string SourceName => KisaConnectorPlugin.SourceName;
|
public string SourceName => KisaConnectorPlugin.SourceName;
|
||||||
|
|
||||||
public async Task FetchAsync(IServiceProvider services, CancellationToken cancellationToken)
|
public async Task FetchAsync(IServiceProvider services, CancellationToken cancellationToken)
|
||||||
{
|
{
|
||||||
ArgumentNullException.ThrowIfNull(services);
|
ArgumentNullException.ThrowIfNull(services);
|
||||||
|
|
||||||
var cursor = await GetCursorAsync(cancellationToken).ConfigureAwait(false);
|
var cursor = await GetCursorAsync(cancellationToken).ConfigureAwait(false);
|
||||||
var now = _timeProvider.GetUtcNow();
|
var now = _timeProvider.GetUtcNow();
|
||||||
_diagnostics.FeedAttempt();
|
_diagnostics.FeedAttempt();
|
||||||
IReadOnlyList<KisaFeedItem> items;
|
IReadOnlyList<KisaFeedItem> items;
|
||||||
|
|
||||||
try
|
try
|
||||||
{
|
{
|
||||||
items = await _feedClient.LoadAsync(cancellationToken).ConfigureAwait(false);
|
items = await _feedClient.LoadAsync(cancellationToken).ConfigureAwait(false);
|
||||||
_diagnostics.FeedSuccess(items.Count);
|
_diagnostics.FeedSuccess(items.Count);
|
||||||
|
|
||||||
if (items.Count > 0)
|
if (items.Count > 0)
|
||||||
{
|
{
|
||||||
_logger.LogInformation("KISA feed returned {ItemCount} advisories", items.Count);
|
_logger.LogInformation("KISA feed returned {ItemCount} advisories", items.Count);
|
||||||
}
|
}
|
||||||
else
|
else
|
||||||
{
|
{
|
||||||
_logger.LogDebug("KISA feed returned no advisories");
|
_logger.LogDebug("KISA feed returned no advisories");
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
catch (Exception ex)
|
catch (Exception ex)
|
||||||
{
|
{
|
||||||
_diagnostics.FeedFailure(ex.GetType().Name);
|
_diagnostics.FeedFailure(ex.GetType().Name);
|
||||||
_logger.LogError(ex, "KISA feed fetch failed");
|
_logger.LogError(ex, "KISA feed fetch failed");
|
||||||
await _stateRepository.MarkFailureAsync(SourceName, now, _options.FailureBackoff, ex.Message, cancellationToken).ConfigureAwait(false);
|
await _stateRepository.MarkFailureAsync(SourceName, now, _options.FailureBackoff, ex.Message, cancellationToken).ConfigureAwait(false);
|
||||||
throw;
|
throw;
|
||||||
}
|
}
|
||||||
|
|
||||||
if (items.Count == 0)
|
if (items.Count == 0)
|
||||||
{
|
{
|
||||||
await UpdateCursorAsync(cursor.WithLastFetch(now), cancellationToken).ConfigureAwait(false);
|
await UpdateCursorAsync(cursor.WithLastFetch(now), cancellationToken).ConfigureAwait(false);
|
||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
|
|
||||||
var pendingDocuments = cursor.PendingDocuments.ToHashSet();
|
var pendingDocuments = cursor.PendingDocuments.ToHashSet();
|
||||||
var pendingMappings = cursor.PendingMappings.ToHashSet();
|
var pendingMappings = cursor.PendingMappings.ToHashSet();
|
||||||
var knownIds = new HashSet<string>(cursor.KnownIds, StringComparer.OrdinalIgnoreCase);
|
var knownIds = new HashSet<string>(cursor.KnownIds, StringComparer.OrdinalIgnoreCase);
|
||||||
var processed = 0;
|
var processed = 0;
|
||||||
var latestPublished = cursor.LastPublished ?? DateTimeOffset.MinValue;
|
var latestPublished = cursor.LastPublished ?? DateTimeOffset.MinValue;
|
||||||
|
|
||||||
foreach (var item in items.OrderByDescending(static i => i.Published))
|
foreach (var item in items.OrderByDescending(static i => i.Published))
|
||||||
{
|
{
|
||||||
cancellationToken.ThrowIfCancellationRequested();
|
cancellationToken.ThrowIfCancellationRequested();
|
||||||
|
|
||||||
if (knownIds.Contains(item.AdvisoryId))
|
if (knownIds.Contains(item.AdvisoryId))
|
||||||
{
|
{
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
|
|
||||||
if (processed >= _options.MaxAdvisoriesPerFetch)
|
if (processed >= _options.MaxAdvisoriesPerFetch)
|
||||||
{
|
{
|
||||||
break;
|
break;
|
||||||
}
|
}
|
||||||
|
|
||||||
var category = item.Category;
|
var category = item.Category;
|
||||||
_diagnostics.DetailAttempt(category);
|
_diagnostics.DetailAttempt(category);
|
||||||
|
|
||||||
try
|
try
|
||||||
{
|
{
|
||||||
var detailUri = item.DetailPageUri;
|
var detailUri = item.DetailPageUri;
|
||||||
@@ -149,125 +149,125 @@ public sealed class KisaConnector : IFeedConnector
|
|||||||
LastModified = existing?.LastModified,
|
LastModified = existing?.LastModified,
|
||||||
TimeoutOverride = _options.RequestTimeout,
|
TimeoutOverride = _options.RequestTimeout,
|
||||||
};
|
};
|
||||||
|
|
||||||
var result = await _fetchService.FetchAsync(request, cancellationToken).ConfigureAwait(false);
|
var result = await _fetchService.FetchAsync(request, cancellationToken).ConfigureAwait(false);
|
||||||
if (result.IsNotModified)
|
if (result.IsNotModified)
|
||||||
{
|
{
|
||||||
_diagnostics.DetailUnchanged(category);
|
_diagnostics.DetailUnchanged(category);
|
||||||
_logger.LogDebug("KISA detail {Idx} unchanged ({Category})", item.AdvisoryId, category ?? "unknown");
|
_logger.LogDebug("KISA detail {Idx} unchanged ({Category})", item.AdvisoryId, category ?? "unknown");
|
||||||
knownIds.Add(item.AdvisoryId);
|
knownIds.Add(item.AdvisoryId);
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
|
|
||||||
if (!result.IsSuccess || result.Document is null)
|
if (!result.IsSuccess || result.Document is null)
|
||||||
{
|
{
|
||||||
_diagnostics.DetailFailure(category, "empty-document");
|
_diagnostics.DetailFailure(category, "empty-document");
|
||||||
_logger.LogWarning("KISA detail fetch returned no document for {Idx}", item.AdvisoryId);
|
_logger.LogWarning("KISA detail fetch returned no document for {Idx}", item.AdvisoryId);
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
|
|
||||||
pendingDocuments.Add(result.Document.Id);
|
pendingDocuments.Add(result.Document.Id);
|
||||||
pendingMappings.Remove(result.Document.Id);
|
pendingMappings.Remove(result.Document.Id);
|
||||||
knownIds.Add(item.AdvisoryId);
|
knownIds.Add(item.AdvisoryId);
|
||||||
processed++;
|
processed++;
|
||||||
_diagnostics.DetailSuccess(category);
|
_diagnostics.DetailSuccess(category);
|
||||||
_logger.LogInformation(
|
_logger.LogInformation(
|
||||||
"KISA fetched detail for {Idx} (documentId={DocumentId}, category={Category})",
|
"KISA fetched detail for {Idx} (documentId={DocumentId}, category={Category})",
|
||||||
item.AdvisoryId,
|
item.AdvisoryId,
|
||||||
result.Document.Id,
|
result.Document.Id,
|
||||||
category ?? "unknown");
|
category ?? "unknown");
|
||||||
|
|
||||||
if (_options.RequestDelay > TimeSpan.Zero)
|
if (_options.RequestDelay > TimeSpan.Zero)
|
||||||
{
|
{
|
||||||
await Task.Delay(_options.RequestDelay, cancellationToken).ConfigureAwait(false);
|
await Task.Delay(_options.RequestDelay, cancellationToken).ConfigureAwait(false);
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
catch (Exception ex)
|
catch (Exception ex)
|
||||||
{
|
{
|
||||||
_diagnostics.DetailFailure(category, ex.GetType().Name);
|
_diagnostics.DetailFailure(category, ex.GetType().Name);
|
||||||
_logger.LogError(ex, "KISA detail fetch failed for {Idx}", item.AdvisoryId);
|
_logger.LogError(ex, "KISA detail fetch failed for {Idx}", item.AdvisoryId);
|
||||||
await _stateRepository.MarkFailureAsync(SourceName, now, _options.FailureBackoff, ex.Message, cancellationToken).ConfigureAwait(false);
|
await _stateRepository.MarkFailureAsync(SourceName, now, _options.FailureBackoff, ex.Message, cancellationToken).ConfigureAwait(false);
|
||||||
throw;
|
throw;
|
||||||
}
|
}
|
||||||
|
|
||||||
if (item.Published > latestPublished)
|
if (item.Published > latestPublished)
|
||||||
{
|
{
|
||||||
latestPublished = item.Published;
|
latestPublished = item.Published;
|
||||||
_diagnostics.CursorAdvanced();
|
_diagnostics.CursorAdvanced();
|
||||||
_logger.LogDebug("KISA advanced published cursor to {Published:O}", latestPublished);
|
_logger.LogDebug("KISA advanced published cursor to {Published:O}", latestPublished);
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
var trimmedKnown = knownIds.Count > _options.MaxKnownAdvisories
|
var trimmedKnown = knownIds.Count > _options.MaxKnownAdvisories
|
||||||
? knownIds.OrderByDescending(id => id, StringComparer.OrdinalIgnoreCase)
|
? knownIds.OrderByDescending(id => id, StringComparer.OrdinalIgnoreCase)
|
||||||
.Take(_options.MaxKnownAdvisories)
|
.Take(_options.MaxKnownAdvisories)
|
||||||
.ToArray()
|
.ToArray()
|
||||||
: knownIds.ToArray();
|
: knownIds.ToArray();
|
||||||
|
|
||||||
var updatedCursor = cursor
|
var updatedCursor = cursor
|
||||||
.WithPendingDocuments(pendingDocuments)
|
.WithPendingDocuments(pendingDocuments)
|
||||||
.WithPendingMappings(pendingMappings)
|
.WithPendingMappings(pendingMappings)
|
||||||
.WithKnownIds(trimmedKnown)
|
.WithKnownIds(trimmedKnown)
|
||||||
.WithLastPublished(latestPublished == DateTimeOffset.MinValue ? cursor.LastPublished : latestPublished)
|
.WithLastPublished(latestPublished == DateTimeOffset.MinValue ? cursor.LastPublished : latestPublished)
|
||||||
.WithLastFetch(now);
|
.WithLastFetch(now);
|
||||||
|
|
||||||
await UpdateCursorAsync(updatedCursor, cancellationToken).ConfigureAwait(false);
|
await UpdateCursorAsync(updatedCursor, cancellationToken).ConfigureAwait(false);
|
||||||
_logger.LogInformation("KISA fetch stored {Processed} new documents (knownIds={KnownCount})", processed, trimmedKnown.Length);
|
_logger.LogInformation("KISA fetch stored {Processed} new documents (knownIds={KnownCount})", processed, trimmedKnown.Length);
|
||||||
}
|
}
|
||||||
|
|
||||||
public async Task ParseAsync(IServiceProvider services, CancellationToken cancellationToken)
|
public async Task ParseAsync(IServiceProvider services, CancellationToken cancellationToken)
|
||||||
{
|
{
|
||||||
ArgumentNullException.ThrowIfNull(services);
|
ArgumentNullException.ThrowIfNull(services);
|
||||||
|
|
||||||
var cursor = await GetCursorAsync(cancellationToken).ConfigureAwait(false);
|
var cursor = await GetCursorAsync(cancellationToken).ConfigureAwait(false);
|
||||||
if (cursor.PendingDocuments.Count == 0)
|
if (cursor.PendingDocuments.Count == 0)
|
||||||
{
|
{
|
||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
|
|
||||||
var remainingDocuments = cursor.PendingDocuments.ToHashSet();
|
var remainingDocuments = cursor.PendingDocuments.ToHashSet();
|
||||||
var pendingMappings = cursor.PendingMappings.ToHashSet();
|
var pendingMappings = cursor.PendingMappings.ToHashSet();
|
||||||
var now = _timeProvider.GetUtcNow();
|
var now = _timeProvider.GetUtcNow();
|
||||||
|
|
||||||
foreach (var documentId in cursor.PendingDocuments)
|
foreach (var documentId in cursor.PendingDocuments)
|
||||||
{
|
{
|
||||||
cancellationToken.ThrowIfCancellationRequested();
|
cancellationToken.ThrowIfCancellationRequested();
|
||||||
|
|
||||||
var document = await _documentStore.FindAsync(documentId, cancellationToken).ConfigureAwait(false);
|
var document = await _documentStore.FindAsync(documentId, cancellationToken).ConfigureAwait(false);
|
||||||
if (document is null)
|
if (document is null)
|
||||||
{
|
{
|
||||||
_diagnostics.ParseFailure(null, "document-missing");
|
_diagnostics.ParseFailure(null, "document-missing");
|
||||||
_logger.LogWarning("KISA document {DocumentId} missing during parse", documentId);
|
_logger.LogWarning("KISA document {DocumentId} missing during parse", documentId);
|
||||||
remainingDocuments.Remove(documentId);
|
remainingDocuments.Remove(documentId);
|
||||||
pendingMappings.Remove(documentId);
|
pendingMappings.Remove(documentId);
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
|
|
||||||
var category = GetCategory(document);
|
var category = GetCategory(document);
|
||||||
if (!document.GridFsId.HasValue)
|
if (!document.PayloadId.HasValue)
|
||||||
{
|
{
|
||||||
_diagnostics.ParseFailure(category, "missing-gridfs");
|
_diagnostics.ParseFailure(category, "missing-gridfs");
|
||||||
_logger.LogWarning("KISA document {DocumentId} missing GridFS payload", document.Id);
|
_logger.LogWarning("KISA document {DocumentId} missing GridFS payload", document.Id);
|
||||||
await _documentStore.UpdateStatusAsync(document.Id, DocumentStatuses.Failed, cancellationToken).ConfigureAwait(false);
|
await _documentStore.UpdateStatusAsync(document.Id, DocumentStatuses.Failed, cancellationToken).ConfigureAwait(false);
|
||||||
remainingDocuments.Remove(documentId);
|
remainingDocuments.Remove(documentId);
|
||||||
pendingMappings.Remove(documentId);
|
pendingMappings.Remove(documentId);
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
|
|
||||||
_diagnostics.ParseAttempt(category);
|
_diagnostics.ParseAttempt(category);
|
||||||
|
|
||||||
byte[] payload;
|
byte[] payload;
|
||||||
try
|
try
|
||||||
{
|
{
|
||||||
payload = await _rawDocumentStorage.DownloadAsync(document.GridFsId.Value, cancellationToken).ConfigureAwait(false);
|
payload = await _rawDocumentStorage.DownloadAsync(document.PayloadId.Value, cancellationToken).ConfigureAwait(false);
|
||||||
}
|
}
|
||||||
catch (Exception ex)
|
catch (Exception ex)
|
||||||
{
|
{
|
||||||
_diagnostics.ParseFailure(category, "download");
|
_diagnostics.ParseFailure(category, "download");
|
||||||
_logger.LogError(ex, "KISA unable to download document {DocumentId}", document.Id);
|
_logger.LogError(ex, "KISA unable to download document {DocumentId}", document.Id);
|
||||||
throw;
|
throw;
|
||||||
}
|
}
|
||||||
|
|
||||||
KisaParsedAdvisory parsed;
|
KisaParsedAdvisory parsed;
|
||||||
try
|
try
|
||||||
{
|
{
|
||||||
@@ -279,28 +279,28 @@ public sealed class KisaConnector : IFeedConnector
|
|||||||
{
|
{
|
||||||
_diagnostics.ParseFailure(category, "parse");
|
_diagnostics.ParseFailure(category, "parse");
|
||||||
_logger.LogError(ex, "KISA failed to parse detail {DocumentId}", document.Id);
|
_logger.LogError(ex, "KISA failed to parse detail {DocumentId}", document.Id);
|
||||||
await _documentStore.UpdateStatusAsync(document.Id, DocumentStatuses.Failed, cancellationToken).ConfigureAwait(false);
|
await _documentStore.UpdateStatusAsync(document.Id, DocumentStatuses.Failed, cancellationToken).ConfigureAwait(false);
|
||||||
remainingDocuments.Remove(documentId);
|
remainingDocuments.Remove(documentId);
|
||||||
pendingMappings.Remove(documentId);
|
pendingMappings.Remove(documentId);
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
_diagnostics.ParseSuccess(category);
|
_diagnostics.ParseSuccess(category);
|
||||||
_logger.LogDebug("KISA parsed detail for {DocumentId} ({Category})", document.Id, category ?? "unknown");
|
_logger.LogDebug("KISA parsed detail for {DocumentId} ({Category})", document.Id, category ?? "unknown");
|
||||||
|
|
||||||
var dtoBson = BsonDocument.Parse(JsonSerializer.Serialize(parsed, SerializerOptions));
|
var dtoBson = BsonDocument.Parse(JsonSerializer.Serialize(parsed, SerializerOptions));
|
||||||
var dtoRecord = new DtoRecord(Guid.NewGuid(), document.Id, SourceName, "kisa.detail.v1", dtoBson, now);
|
var dtoRecord = new DtoRecord(Guid.NewGuid(), document.Id, SourceName, "kisa.detail.v1", dtoBson, now);
|
||||||
await _dtoStore.UpsertAsync(dtoRecord, cancellationToken).ConfigureAwait(false);
|
await _dtoStore.UpsertAsync(dtoRecord, cancellationToken).ConfigureAwait(false);
|
||||||
await _documentStore.UpdateStatusAsync(document.Id, DocumentStatuses.PendingMap, cancellationToken).ConfigureAwait(false);
|
await _documentStore.UpdateStatusAsync(document.Id, DocumentStatuses.PendingMap, cancellationToken).ConfigureAwait(false);
|
||||||
|
|
||||||
remainingDocuments.Remove(documentId);
|
remainingDocuments.Remove(documentId);
|
||||||
pendingMappings.Add(document.Id);
|
pendingMappings.Add(document.Id);
|
||||||
}
|
}
|
||||||
|
|
||||||
var updatedCursor = cursor
|
var updatedCursor = cursor
|
||||||
.WithPendingDocuments(remainingDocuments)
|
.WithPendingDocuments(remainingDocuments)
|
||||||
.WithPendingMappings(pendingMappings);
|
.WithPendingMappings(pendingMappings);
|
||||||
|
|
||||||
await UpdateCursorAsync(updatedCursor, cancellationToken).ConfigureAwait(false);
|
await UpdateCursorAsync(updatedCursor, cancellationToken).ConfigureAwait(false);
|
||||||
}
|
}
|
||||||
|
|
||||||
private static Uri? TryGetUri(IReadOnlyDictionary<string, string>? metadata, string key)
|
private static Uri? TryGetUri(IReadOnlyDictionary<string, string>? metadata, string key)
|
||||||
@@ -318,107 +318,107 @@ public sealed class KisaConnector : IFeedConnector
|
|||||||
return Uri.TryCreate(value, UriKind.Absolute, out var uri) ? uri : null;
|
return Uri.TryCreate(value, UriKind.Absolute, out var uri) ? uri : null;
|
||||||
}
|
}
|
||||||
|
|
||||||
public async Task MapAsync(IServiceProvider services, CancellationToken cancellationToken)
|
public async Task MapAsync(IServiceProvider services, CancellationToken cancellationToken)
|
||||||
{
|
{
|
||||||
ArgumentNullException.ThrowIfNull(services);
|
ArgumentNullException.ThrowIfNull(services);
|
||||||
|
|
||||||
var cursor = await GetCursorAsync(cancellationToken).ConfigureAwait(false);
|
var cursor = await GetCursorAsync(cancellationToken).ConfigureAwait(false);
|
||||||
if (cursor.PendingMappings.Count == 0)
|
if (cursor.PendingMappings.Count == 0)
|
||||||
{
|
{
|
||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
|
|
||||||
var pendingMappings = cursor.PendingMappings.ToHashSet();
|
var pendingMappings = cursor.PendingMappings.ToHashSet();
|
||||||
|
|
||||||
foreach (var documentId in cursor.PendingMappings)
|
foreach (var documentId in cursor.PendingMappings)
|
||||||
{
|
{
|
||||||
cancellationToken.ThrowIfCancellationRequested();
|
cancellationToken.ThrowIfCancellationRequested();
|
||||||
|
|
||||||
var document = await _documentStore.FindAsync(documentId, cancellationToken).ConfigureAwait(false);
|
var document = await _documentStore.FindAsync(documentId, cancellationToken).ConfigureAwait(false);
|
||||||
if (document is null)
|
if (document is null)
|
||||||
{
|
{
|
||||||
_diagnostics.MapFailure(null, "document-missing");
|
_diagnostics.MapFailure(null, "document-missing");
|
||||||
_logger.LogWarning("KISA document {DocumentId} missing during map", documentId);
|
_logger.LogWarning("KISA document {DocumentId} missing during map", documentId);
|
||||||
pendingMappings.Remove(documentId);
|
pendingMappings.Remove(documentId);
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
|
|
||||||
var dtoRecord = await _dtoStore.FindByDocumentIdAsync(documentId, cancellationToken).ConfigureAwait(false);
|
var dtoRecord = await _dtoStore.FindByDocumentIdAsync(documentId, cancellationToken).ConfigureAwait(false);
|
||||||
if (dtoRecord is null)
|
if (dtoRecord is null)
|
||||||
{
|
{
|
||||||
_diagnostics.MapFailure(null, "dto-missing");
|
_diagnostics.MapFailure(null, "dto-missing");
|
||||||
_logger.LogWarning("KISA DTO missing for document {DocumentId}", document.Id);
|
_logger.LogWarning("KISA DTO missing for document {DocumentId}", document.Id);
|
||||||
await _documentStore.UpdateStatusAsync(document.Id, DocumentStatuses.Failed, cancellationToken).ConfigureAwait(false);
|
await _documentStore.UpdateStatusAsync(document.Id, DocumentStatuses.Failed, cancellationToken).ConfigureAwait(false);
|
||||||
pendingMappings.Remove(documentId);
|
pendingMappings.Remove(documentId);
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
|
|
||||||
KisaParsedAdvisory? parsed;
|
KisaParsedAdvisory? parsed;
|
||||||
try
|
try
|
||||||
{
|
{
|
||||||
parsed = JsonSerializer.Deserialize<KisaParsedAdvisory>(dtoRecord.Payload.ToJson(), SerializerOptions);
|
parsed = JsonSerializer.Deserialize<KisaParsedAdvisory>(dtoRecord.Payload.ToJson(), SerializerOptions);
|
||||||
}
|
}
|
||||||
catch (Exception ex)
|
catch (Exception ex)
|
||||||
{
|
{
|
||||||
_diagnostics.MapFailure(null, "dto-deserialize");
|
_diagnostics.MapFailure(null, "dto-deserialize");
|
||||||
_logger.LogError(ex, "KISA failed to deserialize DTO for document {DocumentId}", document.Id);
|
_logger.LogError(ex, "KISA failed to deserialize DTO for document {DocumentId}", document.Id);
|
||||||
await _documentStore.UpdateStatusAsync(document.Id, DocumentStatuses.Failed, cancellationToken).ConfigureAwait(false);
|
await _documentStore.UpdateStatusAsync(document.Id, DocumentStatuses.Failed, cancellationToken).ConfigureAwait(false);
|
||||||
pendingMappings.Remove(documentId);
|
pendingMappings.Remove(documentId);
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
|
|
||||||
if (parsed is null)
|
if (parsed is null)
|
||||||
{
|
{
|
||||||
_diagnostics.MapFailure(null, "dto-null");
|
_diagnostics.MapFailure(null, "dto-null");
|
||||||
await _documentStore.UpdateStatusAsync(document.Id, DocumentStatuses.Failed, cancellationToken).ConfigureAwait(false);
|
await _documentStore.UpdateStatusAsync(document.Id, DocumentStatuses.Failed, cancellationToken).ConfigureAwait(false);
|
||||||
pendingMappings.Remove(documentId);
|
pendingMappings.Remove(documentId);
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
|
|
||||||
try
|
try
|
||||||
{
|
{
|
||||||
var advisory = KisaMapper.Map(parsed, document, dtoRecord.ValidatedAt);
|
var advisory = KisaMapper.Map(parsed, document, dtoRecord.ValidatedAt);
|
||||||
await _advisoryStore.UpsertAsync(advisory, cancellationToken).ConfigureAwait(false);
|
await _advisoryStore.UpsertAsync(advisory, cancellationToken).ConfigureAwait(false);
|
||||||
await _documentStore.UpdateStatusAsync(document.Id, DocumentStatuses.Mapped, cancellationToken).ConfigureAwait(false);
|
await _documentStore.UpdateStatusAsync(document.Id, DocumentStatuses.Mapped, cancellationToken).ConfigureAwait(false);
|
||||||
pendingMappings.Remove(documentId);
|
pendingMappings.Remove(documentId);
|
||||||
_diagnostics.MapSuccess(parsed.Severity);
|
_diagnostics.MapSuccess(parsed.Severity);
|
||||||
_logger.LogInformation("KISA mapped advisory {AdvisoryId} (severity={Severity})", parsed.AdvisoryId, parsed.Severity ?? "unknown");
|
_logger.LogInformation("KISA mapped advisory {AdvisoryId} (severity={Severity})", parsed.AdvisoryId, parsed.Severity ?? "unknown");
|
||||||
}
|
}
|
||||||
catch (Exception ex)
|
catch (Exception ex)
|
||||||
{
|
{
|
||||||
_diagnostics.MapFailure(parsed.Severity, "map");
|
_diagnostics.MapFailure(parsed.Severity, "map");
|
||||||
_logger.LogError(ex, "KISA mapping failed for document {DocumentId}", document.Id);
|
_logger.LogError(ex, "KISA mapping failed for document {DocumentId}", document.Id);
|
||||||
await _documentStore.UpdateStatusAsync(document.Id, DocumentStatuses.Failed, cancellationToken).ConfigureAwait(false);
|
await _documentStore.UpdateStatusAsync(document.Id, DocumentStatuses.Failed, cancellationToken).ConfigureAwait(false);
|
||||||
pendingMappings.Remove(documentId);
|
pendingMappings.Remove(documentId);
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
var updatedCursor = cursor.WithPendingMappings(pendingMappings);
|
var updatedCursor = cursor.WithPendingMappings(pendingMappings);
|
||||||
await UpdateCursorAsync(updatedCursor, cancellationToken).ConfigureAwait(false);
|
await UpdateCursorAsync(updatedCursor, cancellationToken).ConfigureAwait(false);
|
||||||
}
|
}
|
||||||
|
|
||||||
private static string? GetCategory(DocumentRecord document)
|
private static string? GetCategory(DocumentRecord document)
|
||||||
{
|
{
|
||||||
if (document.Metadata is null)
|
if (document.Metadata is null)
|
||||||
{
|
{
|
||||||
return null;
|
return null;
|
||||||
}
|
}
|
||||||
|
|
||||||
return document.Metadata.TryGetValue("kisa.category", out var category)
|
return document.Metadata.TryGetValue("kisa.category", out var category)
|
||||||
? category
|
? category
|
||||||
: null;
|
: null;
|
||||||
}
|
}
|
||||||
|
|
||||||
private async Task<KisaCursor> GetCursorAsync(CancellationToken cancellationToken)
|
private async Task<KisaCursor> GetCursorAsync(CancellationToken cancellationToken)
|
||||||
{
|
{
|
||||||
var state = await _stateRepository.TryGetAsync(SourceName, cancellationToken).ConfigureAwait(false);
|
var state = await _stateRepository.TryGetAsync(SourceName, cancellationToken).ConfigureAwait(false);
|
||||||
return state is null ? KisaCursor.Empty : KisaCursor.FromBson(state.Cursor);
|
return state is null ? KisaCursor.Empty : KisaCursor.FromBson(state.Cursor);
|
||||||
}
|
}
|
||||||
|
|
||||||
private Task UpdateCursorAsync(KisaCursor cursor, CancellationToken cancellationToken)
|
private Task UpdateCursorAsync(KisaCursor cursor, CancellationToken cancellationToken)
|
||||||
{
|
{
|
||||||
var document = cursor.ToBsonDocument();
|
var document = cursor.ToBsonDocument();
|
||||||
var completedAt = cursor.LastFetchAt ?? _timeProvider.GetUtcNow();
|
var completedAt = cursor.LastFetchAt ?? _timeProvider.GetUtcNow();
|
||||||
return _stateRepository.UpdateCursorAsync(SourceName, document, completedAt, cancellationToken);
|
return _stateRepository.UpdateCursorAsync(SourceName, document, completedAt, cancellationToken);
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
File diff suppressed because it is too large
Load Diff
@@ -1,43 +1,43 @@
|
|||||||
using System;
|
using System;
|
||||||
using System.Collections.Generic;
|
using System.Collections.Generic;
|
||||||
using System.IO;
|
using System.IO;
|
||||||
using System.IO.Compression;
|
using System.IO.Compression;
|
||||||
using System.Linq;
|
using System.Linq;
|
||||||
using System.Net;
|
using System.Net;
|
||||||
using System.Net.Http;
|
using System.Net.Http;
|
||||||
using System.Text.Json;
|
using System.Text.Json;
|
||||||
using System.Text.Json.Serialization;
|
using System.Text.Json.Serialization;
|
||||||
using System.Threading;
|
using System.Threading;
|
||||||
using System.Threading.Tasks;
|
using System.Threading.Tasks;
|
||||||
using Microsoft.Extensions.Logging;
|
using Microsoft.Extensions.Logging;
|
||||||
using Microsoft.Extensions.Options;
|
using Microsoft.Extensions.Options;
|
||||||
using MongoDB.Bson;
|
using MongoDB.Bson;
|
||||||
using MongoDB.Bson.IO;
|
using MongoDB.Bson.IO;
|
||||||
using StellaOps.Concelier.Models;
|
using StellaOps.Concelier.Models;
|
||||||
using StellaOps.Concelier.Connector.Common;
|
using StellaOps.Concelier.Connector.Common;
|
||||||
using StellaOps.Concelier.Connector.Common.Fetch;
|
using StellaOps.Concelier.Connector.Common.Fetch;
|
||||||
using StellaOps.Concelier.Connector.Osv.Configuration;
|
using StellaOps.Concelier.Connector.Osv.Configuration;
|
||||||
using StellaOps.Concelier.Connector.Osv.Internal;
|
using StellaOps.Concelier.Connector.Osv.Internal;
|
||||||
using StellaOps.Concelier.Storage.Mongo;
|
using StellaOps.Concelier.Storage.Mongo;
|
||||||
using StellaOps.Concelier.Storage.Mongo.Advisories;
|
using StellaOps.Concelier.Storage.Mongo.Advisories;
|
||||||
using StellaOps.Concelier.Storage.Mongo.Documents;
|
using StellaOps.Concelier.Storage.Mongo.Documents;
|
||||||
using StellaOps.Concelier.Storage.Mongo.Dtos;
|
using StellaOps.Concelier.Storage.Mongo.Dtos;
|
||||||
using StellaOps.Plugin;
|
using StellaOps.Plugin;
|
||||||
using StellaOps.Cryptography;
|
using StellaOps.Cryptography;
|
||||||
|
|
||||||
namespace StellaOps.Concelier.Connector.Osv;
|
namespace StellaOps.Concelier.Connector.Osv;
|
||||||
|
|
||||||
public sealed class OsvConnector : IFeedConnector
|
public sealed class OsvConnector : IFeedConnector
|
||||||
{
|
{
|
||||||
private static readonly JsonSerializerOptions SerializerOptions = new(JsonSerializerDefaults.Web)
|
private static readonly JsonSerializerOptions SerializerOptions = new(JsonSerializerDefaults.Web)
|
||||||
{
|
{
|
||||||
DefaultIgnoreCondition = JsonIgnoreCondition.WhenWritingNull,
|
DefaultIgnoreCondition = JsonIgnoreCondition.WhenWritingNull,
|
||||||
PropertyNameCaseInsensitive = true,
|
PropertyNameCaseInsensitive = true,
|
||||||
};
|
};
|
||||||
|
|
||||||
private readonly IHttpClientFactory _httpClientFactory;
|
private readonly IHttpClientFactory _httpClientFactory;
|
||||||
private readonly RawDocumentStorage _rawDocumentStorage;
|
private readonly RawDocumentStorage _rawDocumentStorage;
|
||||||
private readonly IDocumentStore _documentStore;
|
private readonly IDocumentStore _documentStore;
|
||||||
private readonly IDtoStore _dtoStore;
|
private readonly IDtoStore _dtoStore;
|
||||||
private readonly IAdvisoryStore _advisoryStore;
|
private readonly IAdvisoryStore _advisoryStore;
|
||||||
private readonly ISourceStateRepository _stateRepository;
|
private readonly ISourceStateRepository _stateRepository;
|
||||||
@@ -46,10 +46,10 @@ public sealed class OsvConnector : IFeedConnector
|
|||||||
private readonly ILogger<OsvConnector> _logger;
|
private readonly ILogger<OsvConnector> _logger;
|
||||||
private readonly OsvDiagnostics _diagnostics;
|
private readonly OsvDiagnostics _diagnostics;
|
||||||
private readonly ICryptoHash _hash;
|
private readonly ICryptoHash _hash;
|
||||||
|
|
||||||
public OsvConnector(
|
public OsvConnector(
|
||||||
IHttpClientFactory httpClientFactory,
|
IHttpClientFactory httpClientFactory,
|
||||||
RawDocumentStorage rawDocumentStorage,
|
RawDocumentStorage rawDocumentStorage,
|
||||||
IDocumentStore documentStore,
|
IDocumentStore documentStore,
|
||||||
IDtoStore dtoStore,
|
IDtoStore dtoStore,
|
||||||
IAdvisoryStore advisoryStore,
|
IAdvisoryStore advisoryStore,
|
||||||
@@ -73,197 +73,197 @@ public sealed class OsvConnector : IFeedConnector
|
|||||||
_timeProvider = timeProvider ?? TimeProvider.System;
|
_timeProvider = timeProvider ?? TimeProvider.System;
|
||||||
_logger = logger ?? throw new ArgumentNullException(nameof(logger));
|
_logger = logger ?? throw new ArgumentNullException(nameof(logger));
|
||||||
}
|
}
|
||||||
|
|
||||||
public string SourceName => OsvConnectorPlugin.SourceName;
|
public string SourceName => OsvConnectorPlugin.SourceName;
|
||||||
|
|
||||||
public async Task FetchAsync(IServiceProvider services, CancellationToken cancellationToken)
|
public async Task FetchAsync(IServiceProvider services, CancellationToken cancellationToken)
|
||||||
{
|
{
|
||||||
ArgumentNullException.ThrowIfNull(services);
|
ArgumentNullException.ThrowIfNull(services);
|
||||||
|
|
||||||
var cursor = await GetCursorAsync(cancellationToken).ConfigureAwait(false);
|
var cursor = await GetCursorAsync(cancellationToken).ConfigureAwait(false);
|
||||||
var now = _timeProvider.GetUtcNow();
|
var now = _timeProvider.GetUtcNow();
|
||||||
var pendingDocuments = cursor.PendingDocuments.ToHashSet();
|
var pendingDocuments = cursor.PendingDocuments.ToHashSet();
|
||||||
var cursorState = cursor;
|
var cursorState = cursor;
|
||||||
var remainingCapacity = _options.MaxAdvisoriesPerFetch;
|
var remainingCapacity = _options.MaxAdvisoriesPerFetch;
|
||||||
|
|
||||||
foreach (var ecosystem in _options.Ecosystems)
|
foreach (var ecosystem in _options.Ecosystems)
|
||||||
{
|
{
|
||||||
if (remainingCapacity <= 0)
|
if (remainingCapacity <= 0)
|
||||||
{
|
{
|
||||||
break;
|
break;
|
||||||
}
|
}
|
||||||
|
|
||||||
cancellationToken.ThrowIfCancellationRequested();
|
cancellationToken.ThrowIfCancellationRequested();
|
||||||
|
|
||||||
try
|
try
|
||||||
{
|
{
|
||||||
var result = await FetchEcosystemAsync(
|
var result = await FetchEcosystemAsync(
|
||||||
ecosystem,
|
ecosystem,
|
||||||
cursorState,
|
cursorState,
|
||||||
pendingDocuments,
|
pendingDocuments,
|
||||||
now,
|
now,
|
||||||
remainingCapacity,
|
remainingCapacity,
|
||||||
cancellationToken).ConfigureAwait(false);
|
cancellationToken).ConfigureAwait(false);
|
||||||
|
|
||||||
cursorState = result.Cursor;
|
cursorState = result.Cursor;
|
||||||
remainingCapacity -= result.NewDocuments;
|
remainingCapacity -= result.NewDocuments;
|
||||||
}
|
}
|
||||||
catch (Exception ex)
|
catch (Exception ex)
|
||||||
{
|
{
|
||||||
_logger.LogError(ex, "OSV fetch failed for ecosystem {Ecosystem}", ecosystem);
|
_logger.LogError(ex, "OSV fetch failed for ecosystem {Ecosystem}", ecosystem);
|
||||||
await _stateRepository.MarkFailureAsync(SourceName, now, TimeSpan.FromMinutes(10), ex.Message, cancellationToken).ConfigureAwait(false);
|
await _stateRepository.MarkFailureAsync(SourceName, now, TimeSpan.FromMinutes(10), ex.Message, cancellationToken).ConfigureAwait(false);
|
||||||
throw;
|
throw;
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
cursorState = cursorState
|
cursorState = cursorState
|
||||||
.WithPendingDocuments(pendingDocuments)
|
.WithPendingDocuments(pendingDocuments)
|
||||||
.WithPendingMappings(cursor.PendingMappings);
|
.WithPendingMappings(cursor.PendingMappings);
|
||||||
|
|
||||||
await UpdateCursorAsync(cursorState, cancellationToken).ConfigureAwait(false);
|
await UpdateCursorAsync(cursorState, cancellationToken).ConfigureAwait(false);
|
||||||
}
|
}
|
||||||
|
|
||||||
public async Task ParseAsync(IServiceProvider services, CancellationToken cancellationToken)
|
public async Task ParseAsync(IServiceProvider services, CancellationToken cancellationToken)
|
||||||
{
|
{
|
||||||
ArgumentNullException.ThrowIfNull(services);
|
ArgumentNullException.ThrowIfNull(services);
|
||||||
|
|
||||||
var cursor = await GetCursorAsync(cancellationToken).ConfigureAwait(false);
|
var cursor = await GetCursorAsync(cancellationToken).ConfigureAwait(false);
|
||||||
if (cursor.PendingDocuments.Count == 0)
|
if (cursor.PendingDocuments.Count == 0)
|
||||||
{
|
{
|
||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
|
|
||||||
var remainingDocuments = cursor.PendingDocuments.ToList();
|
var remainingDocuments = cursor.PendingDocuments.ToList();
|
||||||
var pendingMappings = cursor.PendingMappings.ToList();
|
var pendingMappings = cursor.PendingMappings.ToList();
|
||||||
|
|
||||||
foreach (var documentId in cursor.PendingDocuments)
|
foreach (var documentId in cursor.PendingDocuments)
|
||||||
{
|
{
|
||||||
cancellationToken.ThrowIfCancellationRequested();
|
cancellationToken.ThrowIfCancellationRequested();
|
||||||
|
|
||||||
var document = await _documentStore.FindAsync(documentId, cancellationToken).ConfigureAwait(false);
|
var document = await _documentStore.FindAsync(documentId, cancellationToken).ConfigureAwait(false);
|
||||||
if (document is null)
|
if (document is null)
|
||||||
{
|
{
|
||||||
remainingDocuments.Remove(documentId);
|
remainingDocuments.Remove(documentId);
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
|
|
||||||
if (!document.GridFsId.HasValue)
|
if (!document.PayloadId.HasValue)
|
||||||
{
|
{
|
||||||
_logger.LogWarning("OSV document {DocumentId} missing GridFS content", document.Id);
|
_logger.LogWarning("OSV document {DocumentId} missing GridFS content", document.Id);
|
||||||
await _documentStore.UpdateStatusAsync(document.Id, DocumentStatuses.Failed, cancellationToken).ConfigureAwait(false);
|
await _documentStore.UpdateStatusAsync(document.Id, DocumentStatuses.Failed, cancellationToken).ConfigureAwait(false);
|
||||||
remainingDocuments.Remove(documentId);
|
remainingDocuments.Remove(documentId);
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
|
|
||||||
byte[] bytes;
|
byte[] bytes;
|
||||||
try
|
try
|
||||||
{
|
{
|
||||||
bytes = await _rawDocumentStorage.DownloadAsync(document.GridFsId.Value, cancellationToken).ConfigureAwait(false);
|
bytes = await _rawDocumentStorage.DownloadAsync(document.PayloadId.Value, cancellationToken).ConfigureAwait(false);
|
||||||
}
|
}
|
||||||
catch (Exception ex)
|
catch (Exception ex)
|
||||||
{
|
{
|
||||||
_logger.LogError(ex, "Unable to download OSV raw document {DocumentId}", document.Id);
|
_logger.LogError(ex, "Unable to download OSV raw document {DocumentId}", document.Id);
|
||||||
throw;
|
throw;
|
||||||
}
|
}
|
||||||
|
|
||||||
OsvVulnerabilityDto? dto;
|
OsvVulnerabilityDto? dto;
|
||||||
try
|
try
|
||||||
{
|
{
|
||||||
dto = JsonSerializer.Deserialize<OsvVulnerabilityDto>(bytes, SerializerOptions);
|
dto = JsonSerializer.Deserialize<OsvVulnerabilityDto>(bytes, SerializerOptions);
|
||||||
}
|
}
|
||||||
catch (Exception ex)
|
catch (Exception ex)
|
||||||
{
|
{
|
||||||
_logger.LogWarning(ex, "Failed to deserialize OSV document {DocumentId} ({Uri})", document.Id, document.Uri);
|
_logger.LogWarning(ex, "Failed to deserialize OSV document {DocumentId} ({Uri})", document.Id, document.Uri);
|
||||||
await _documentStore.UpdateStatusAsync(document.Id, DocumentStatuses.Failed, cancellationToken).ConfigureAwait(false);
|
await _documentStore.UpdateStatusAsync(document.Id, DocumentStatuses.Failed, cancellationToken).ConfigureAwait(false);
|
||||||
remainingDocuments.Remove(documentId);
|
remainingDocuments.Remove(documentId);
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
|
|
||||||
if (dto is null || string.IsNullOrWhiteSpace(dto.Id))
|
if (dto is null || string.IsNullOrWhiteSpace(dto.Id))
|
||||||
{
|
{
|
||||||
_logger.LogWarning("OSV document {DocumentId} produced empty payload", document.Id);
|
_logger.LogWarning("OSV document {DocumentId} produced empty payload", document.Id);
|
||||||
await _documentStore.UpdateStatusAsync(document.Id, DocumentStatuses.Failed, cancellationToken).ConfigureAwait(false);
|
await _documentStore.UpdateStatusAsync(document.Id, DocumentStatuses.Failed, cancellationToken).ConfigureAwait(false);
|
||||||
remainingDocuments.Remove(documentId);
|
remainingDocuments.Remove(documentId);
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
|
|
||||||
var sanitized = JsonSerializer.Serialize(dto, SerializerOptions);
|
var sanitized = JsonSerializer.Serialize(dto, SerializerOptions);
|
||||||
var payload = MongoDB.Bson.BsonDocument.Parse(sanitized);
|
var payload = MongoDB.Bson.BsonDocument.Parse(sanitized);
|
||||||
var dtoRecord = new DtoRecord(
|
var dtoRecord = new DtoRecord(
|
||||||
Guid.NewGuid(),
|
Guid.NewGuid(),
|
||||||
document.Id,
|
document.Id,
|
||||||
SourceName,
|
SourceName,
|
||||||
"osv.v1",
|
"osv.v1",
|
||||||
payload,
|
payload,
|
||||||
_timeProvider.GetUtcNow());
|
_timeProvider.GetUtcNow());
|
||||||
|
|
||||||
await _dtoStore.UpsertAsync(dtoRecord, cancellationToken).ConfigureAwait(false);
|
await _dtoStore.UpsertAsync(dtoRecord, cancellationToken).ConfigureAwait(false);
|
||||||
await _documentStore.UpdateStatusAsync(document.Id, DocumentStatuses.PendingMap, cancellationToken).ConfigureAwait(false);
|
await _documentStore.UpdateStatusAsync(document.Id, DocumentStatuses.PendingMap, cancellationToken).ConfigureAwait(false);
|
||||||
|
|
||||||
remainingDocuments.Remove(documentId);
|
remainingDocuments.Remove(documentId);
|
||||||
if (!pendingMappings.Contains(documentId))
|
if (!pendingMappings.Contains(documentId))
|
||||||
{
|
{
|
||||||
pendingMappings.Add(documentId);
|
pendingMappings.Add(documentId);
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
var updatedCursor = cursor
|
var updatedCursor = cursor
|
||||||
.WithPendingDocuments(remainingDocuments)
|
.WithPendingDocuments(remainingDocuments)
|
||||||
.WithPendingMappings(pendingMappings);
|
.WithPendingMappings(pendingMappings);
|
||||||
|
|
||||||
await UpdateCursorAsync(updatedCursor, cancellationToken).ConfigureAwait(false);
|
await UpdateCursorAsync(updatedCursor, cancellationToken).ConfigureAwait(false);
|
||||||
}
|
}
|
||||||
|
|
||||||
public async Task MapAsync(IServiceProvider services, CancellationToken cancellationToken)
|
public async Task MapAsync(IServiceProvider services, CancellationToken cancellationToken)
|
||||||
{
|
{
|
||||||
ArgumentNullException.ThrowIfNull(services);
|
ArgumentNullException.ThrowIfNull(services);
|
||||||
|
|
||||||
var cursor = await GetCursorAsync(cancellationToken).ConfigureAwait(false);
|
var cursor = await GetCursorAsync(cancellationToken).ConfigureAwait(false);
|
||||||
if (cursor.PendingMappings.Count == 0)
|
if (cursor.PendingMappings.Count == 0)
|
||||||
{
|
{
|
||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
|
|
||||||
var pendingMappings = cursor.PendingMappings.ToList();
|
var pendingMappings = cursor.PendingMappings.ToList();
|
||||||
|
|
||||||
foreach (var documentId in cursor.PendingMappings)
|
foreach (var documentId in cursor.PendingMappings)
|
||||||
{
|
{
|
||||||
cancellationToken.ThrowIfCancellationRequested();
|
cancellationToken.ThrowIfCancellationRequested();
|
||||||
|
|
||||||
var dto = await _dtoStore.FindByDocumentIdAsync(documentId, cancellationToken).ConfigureAwait(false);
|
var dto = await _dtoStore.FindByDocumentIdAsync(documentId, cancellationToken).ConfigureAwait(false);
|
||||||
var document = await _documentStore.FindAsync(documentId, cancellationToken).ConfigureAwait(false);
|
var document = await _documentStore.FindAsync(documentId, cancellationToken).ConfigureAwait(false);
|
||||||
|
|
||||||
if (dto is null || document is null)
|
if (dto is null || document is null)
|
||||||
{
|
{
|
||||||
pendingMappings.Remove(documentId);
|
pendingMappings.Remove(documentId);
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
|
|
||||||
var payloadJson = dto.Payload.ToJson(new JsonWriterSettings
|
var payloadJson = dto.Payload.ToJson(new JsonWriterSettings
|
||||||
{
|
{
|
||||||
OutputMode = JsonOutputMode.RelaxedExtendedJson,
|
OutputMode = JsonOutputMode.RelaxedExtendedJson,
|
||||||
});
|
});
|
||||||
|
|
||||||
OsvVulnerabilityDto? osvDto;
|
OsvVulnerabilityDto? osvDto;
|
||||||
try
|
try
|
||||||
{
|
{
|
||||||
osvDto = JsonSerializer.Deserialize<OsvVulnerabilityDto>(payloadJson, SerializerOptions);
|
osvDto = JsonSerializer.Deserialize<OsvVulnerabilityDto>(payloadJson, SerializerOptions);
|
||||||
}
|
}
|
||||||
catch (Exception ex)
|
catch (Exception ex)
|
||||||
{
|
{
|
||||||
_logger.LogError(ex, "Failed to deserialize OSV DTO for document {DocumentId}", document.Id);
|
_logger.LogError(ex, "Failed to deserialize OSV DTO for document {DocumentId}", document.Id);
|
||||||
await _documentStore.UpdateStatusAsync(document.Id, DocumentStatuses.Failed, cancellationToken).ConfigureAwait(false);
|
await _documentStore.UpdateStatusAsync(document.Id, DocumentStatuses.Failed, cancellationToken).ConfigureAwait(false);
|
||||||
pendingMappings.Remove(documentId);
|
pendingMappings.Remove(documentId);
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
|
|
||||||
if (osvDto is null || string.IsNullOrWhiteSpace(osvDto.Id))
|
if (osvDto is null || string.IsNullOrWhiteSpace(osvDto.Id))
|
||||||
{
|
{
|
||||||
await _documentStore.UpdateStatusAsync(document.Id, DocumentStatuses.Failed, cancellationToken).ConfigureAwait(false);
|
await _documentStore.UpdateStatusAsync(document.Id, DocumentStatuses.Failed, cancellationToken).ConfigureAwait(false);
|
||||||
pendingMappings.Remove(documentId);
|
pendingMappings.Remove(documentId);
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
|
|
||||||
var ecosystem = document.Metadata is not null && document.Metadata.TryGetValue("osv.ecosystem", out var ecosystemValue)
|
var ecosystem = document.Metadata is not null && document.Metadata.TryGetValue("osv.ecosystem", out var ecosystemValue)
|
||||||
? ecosystemValue
|
? ecosystemValue
|
||||||
: "unknown";
|
: "unknown";
|
||||||
@@ -289,232 +289,232 @@ public sealed class OsvConnector : IFeedConnector
|
|||||||
|
|
||||||
pendingMappings.Remove(documentId);
|
pendingMappings.Remove(documentId);
|
||||||
}
|
}
|
||||||
|
|
||||||
var updatedCursor = cursor.WithPendingMappings(pendingMappings);
|
var updatedCursor = cursor.WithPendingMappings(pendingMappings);
|
||||||
await UpdateCursorAsync(updatedCursor, cancellationToken).ConfigureAwait(false);
|
await UpdateCursorAsync(updatedCursor, cancellationToken).ConfigureAwait(false);
|
||||||
}
|
}
|
||||||
|
|
||||||
private async Task<OsvCursor> GetCursorAsync(CancellationToken cancellationToken)
|
private async Task<OsvCursor> GetCursorAsync(CancellationToken cancellationToken)
|
||||||
{
|
{
|
||||||
var state = await _stateRepository.TryGetAsync(SourceName, cancellationToken).ConfigureAwait(false);
|
var state = await _stateRepository.TryGetAsync(SourceName, cancellationToken).ConfigureAwait(false);
|
||||||
return state is null ? OsvCursor.Empty : OsvCursor.FromBson(state.Cursor);
|
return state is null ? OsvCursor.Empty : OsvCursor.FromBson(state.Cursor);
|
||||||
}
|
}
|
||||||
|
|
||||||
private async Task UpdateCursorAsync(OsvCursor cursor, CancellationToken cancellationToken)
|
private async Task UpdateCursorAsync(OsvCursor cursor, CancellationToken cancellationToken)
|
||||||
{
|
{
|
||||||
var document = cursor.ToBsonDocument();
|
var document = cursor.ToBsonDocument();
|
||||||
await _stateRepository.UpdateCursorAsync(SourceName, document, _timeProvider.GetUtcNow(), cancellationToken).ConfigureAwait(false);
|
await _stateRepository.UpdateCursorAsync(SourceName, document, _timeProvider.GetUtcNow(), cancellationToken).ConfigureAwait(false);
|
||||||
}
|
}
|
||||||
|
|
||||||
private async Task<(OsvCursor Cursor, int NewDocuments)> FetchEcosystemAsync(
|
private async Task<(OsvCursor Cursor, int NewDocuments)> FetchEcosystemAsync(
|
||||||
string ecosystem,
|
string ecosystem,
|
||||||
OsvCursor cursor,
|
OsvCursor cursor,
|
||||||
HashSet<Guid> pendingDocuments,
|
HashSet<Guid> pendingDocuments,
|
||||||
DateTimeOffset now,
|
DateTimeOffset now,
|
||||||
int remainingCapacity,
|
int remainingCapacity,
|
||||||
CancellationToken cancellationToken)
|
CancellationToken cancellationToken)
|
||||||
{
|
{
|
||||||
var client = _httpClientFactory.CreateClient(OsvOptions.HttpClientName);
|
var client = _httpClientFactory.CreateClient(OsvOptions.HttpClientName);
|
||||||
client.Timeout = _options.HttpTimeout;
|
client.Timeout = _options.HttpTimeout;
|
||||||
|
|
||||||
var archiveUri = BuildArchiveUri(ecosystem);
|
var archiveUri = BuildArchiveUri(ecosystem);
|
||||||
using var request = new HttpRequestMessage(HttpMethod.Get, archiveUri);
|
using var request = new HttpRequestMessage(HttpMethod.Get, archiveUri);
|
||||||
|
|
||||||
if (cursor.TryGetArchiveMetadata(ecosystem, out var archiveMetadata))
|
if (cursor.TryGetArchiveMetadata(ecosystem, out var archiveMetadata))
|
||||||
{
|
{
|
||||||
if (!string.IsNullOrWhiteSpace(archiveMetadata.ETag))
|
if (!string.IsNullOrWhiteSpace(archiveMetadata.ETag))
|
||||||
{
|
{
|
||||||
request.Headers.TryAddWithoutValidation("If-None-Match", archiveMetadata.ETag);
|
request.Headers.TryAddWithoutValidation("If-None-Match", archiveMetadata.ETag);
|
||||||
}
|
}
|
||||||
|
|
||||||
if (archiveMetadata.LastModified.HasValue)
|
if (archiveMetadata.LastModified.HasValue)
|
||||||
{
|
{
|
||||||
request.Headers.IfModifiedSince = archiveMetadata.LastModified.Value;
|
request.Headers.IfModifiedSince = archiveMetadata.LastModified.Value;
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
using var response = await client.SendAsync(request, HttpCompletionOption.ResponseHeadersRead, cancellationToken).ConfigureAwait(false);
|
using var response = await client.SendAsync(request, HttpCompletionOption.ResponseHeadersRead, cancellationToken).ConfigureAwait(false);
|
||||||
|
|
||||||
if (response.StatusCode == HttpStatusCode.NotModified)
|
if (response.StatusCode == HttpStatusCode.NotModified)
|
||||||
{
|
{
|
||||||
return (cursor, 0);
|
return (cursor, 0);
|
||||||
}
|
}
|
||||||
|
|
||||||
response.EnsureSuccessStatusCode();
|
response.EnsureSuccessStatusCode();
|
||||||
|
|
||||||
await using var archiveStream = await response.Content.ReadAsStreamAsync(cancellationToken).ConfigureAwait(false);
|
await using var archiveStream = await response.Content.ReadAsStreamAsync(cancellationToken).ConfigureAwait(false);
|
||||||
using var archive = new ZipArchive(archiveStream, ZipArchiveMode.Read, leaveOpen: false);
|
using var archive = new ZipArchive(archiveStream, ZipArchiveMode.Read, leaveOpen: false);
|
||||||
|
|
||||||
var existingLastModified = cursor.GetLastModified(ecosystem);
|
var existingLastModified = cursor.GetLastModified(ecosystem);
|
||||||
var processedIdsSet = cursor.ProcessedIdsByEcosystem.TryGetValue(ecosystem, out var processedIds)
|
var processedIdsSet = cursor.ProcessedIdsByEcosystem.TryGetValue(ecosystem, out var processedIds)
|
||||||
? new HashSet<string>(processedIds, StringComparer.OrdinalIgnoreCase)
|
? new HashSet<string>(processedIds, StringComparer.OrdinalIgnoreCase)
|
||||||
: new HashSet<string>(StringComparer.OrdinalIgnoreCase);
|
: new HashSet<string>(StringComparer.OrdinalIgnoreCase);
|
||||||
|
|
||||||
var currentMaxModified = existingLastModified ?? DateTimeOffset.MinValue;
|
var currentMaxModified = existingLastModified ?? DateTimeOffset.MinValue;
|
||||||
var currentProcessedIds = new HashSet<string>(processedIdsSet, StringComparer.OrdinalIgnoreCase);
|
var currentProcessedIds = new HashSet<string>(processedIdsSet, StringComparer.OrdinalIgnoreCase);
|
||||||
var processedUpdated = false;
|
var processedUpdated = false;
|
||||||
var newDocuments = 0;
|
var newDocuments = 0;
|
||||||
|
|
||||||
var minimumModified = existingLastModified.HasValue
|
var minimumModified = existingLastModified.HasValue
|
||||||
? existingLastModified.Value - _options.ModifiedTolerance
|
? existingLastModified.Value - _options.ModifiedTolerance
|
||||||
: now - _options.InitialBackfill;
|
: now - _options.InitialBackfill;
|
||||||
|
|
||||||
ProvenanceDiagnostics.ReportResumeWindow(SourceName, minimumModified, _logger);
|
ProvenanceDiagnostics.ReportResumeWindow(SourceName, minimumModified, _logger);
|
||||||
|
|
||||||
foreach (var entry in archive.Entries)
|
foreach (var entry in archive.Entries)
|
||||||
{
|
{
|
||||||
if (remainingCapacity <= 0)
|
if (remainingCapacity <= 0)
|
||||||
{
|
{
|
||||||
break;
|
break;
|
||||||
}
|
}
|
||||||
|
|
||||||
cancellationToken.ThrowIfCancellationRequested();
|
cancellationToken.ThrowIfCancellationRequested();
|
||||||
|
|
||||||
if (!entry.FullName.EndsWith(".json", StringComparison.OrdinalIgnoreCase))
|
if (!entry.FullName.EndsWith(".json", StringComparison.OrdinalIgnoreCase))
|
||||||
{
|
{
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
|
|
||||||
await using var entryStream = entry.Open();
|
await using var entryStream = entry.Open();
|
||||||
using var memory = new MemoryStream();
|
using var memory = new MemoryStream();
|
||||||
await entryStream.CopyToAsync(memory, cancellationToken).ConfigureAwait(false);
|
await entryStream.CopyToAsync(memory, cancellationToken).ConfigureAwait(false);
|
||||||
var bytes = memory.ToArray();
|
var bytes = memory.ToArray();
|
||||||
|
|
||||||
OsvVulnerabilityDto? dto;
|
OsvVulnerabilityDto? dto;
|
||||||
try
|
try
|
||||||
{
|
{
|
||||||
dto = JsonSerializer.Deserialize<OsvVulnerabilityDto>(bytes, SerializerOptions);
|
dto = JsonSerializer.Deserialize<OsvVulnerabilityDto>(bytes, SerializerOptions);
|
||||||
}
|
}
|
||||||
catch (Exception ex)
|
catch (Exception ex)
|
||||||
{
|
{
|
||||||
_logger.LogWarning(ex, "Failed to parse OSV entry {Entry} for ecosystem {Ecosystem}", entry.FullName, ecosystem);
|
_logger.LogWarning(ex, "Failed to parse OSV entry {Entry} for ecosystem {Ecosystem}", entry.FullName, ecosystem);
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
|
|
||||||
if (dto is null || string.IsNullOrWhiteSpace(dto.Id))
|
if (dto is null || string.IsNullOrWhiteSpace(dto.Id))
|
||||||
{
|
{
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
|
|
||||||
var modified = (dto.Modified ?? dto.Published ?? DateTimeOffset.MinValue).ToUniversalTime();
|
var modified = (dto.Modified ?? dto.Published ?? DateTimeOffset.MinValue).ToUniversalTime();
|
||||||
if (modified < minimumModified)
|
if (modified < minimumModified)
|
||||||
{
|
{
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
|
|
||||||
if (existingLastModified.HasValue && modified < existingLastModified.Value - _options.ModifiedTolerance)
|
if (existingLastModified.HasValue && modified < existingLastModified.Value - _options.ModifiedTolerance)
|
||||||
{
|
{
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
|
|
||||||
if (modified < currentMaxModified - _options.ModifiedTolerance)
|
if (modified < currentMaxModified - _options.ModifiedTolerance)
|
||||||
{
|
{
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
|
|
||||||
if (modified == currentMaxModified && currentProcessedIds.Contains(dto.Id))
|
if (modified == currentMaxModified && currentProcessedIds.Contains(dto.Id))
|
||||||
{
|
{
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
|
|
||||||
var documentUri = BuildDocumentUri(ecosystem, dto.Id);
|
var documentUri = BuildDocumentUri(ecosystem, dto.Id);
|
||||||
var sha256 = _hash.ComputeHashHex(bytes, HashAlgorithms.Sha256);
|
var sha256 = _hash.ComputeHashHex(bytes, HashAlgorithms.Sha256);
|
||||||
|
|
||||||
var existing = await _documentStore.FindBySourceAndUriAsync(SourceName, documentUri, cancellationToken).ConfigureAwait(false);
|
var existing = await _documentStore.FindBySourceAndUriAsync(SourceName, documentUri, cancellationToken).ConfigureAwait(false);
|
||||||
if (existing is not null && string.Equals(existing.Sha256, sha256, StringComparison.OrdinalIgnoreCase))
|
if (existing is not null && string.Equals(existing.Sha256, sha256, StringComparison.OrdinalIgnoreCase))
|
||||||
{
|
{
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
|
|
||||||
var gridFsId = await _rawDocumentStorage.UploadAsync(SourceName, documentUri, bytes, "application/json", null, cancellationToken).ConfigureAwait(false);
|
var gridFsId = await _rawDocumentStorage.UploadAsync(SourceName, documentUri, bytes, "application/json", null, cancellationToken).ConfigureAwait(false);
|
||||||
var metadata = new Dictionary<string, string>(StringComparer.Ordinal)
|
var metadata = new Dictionary<string, string>(StringComparer.Ordinal)
|
||||||
{
|
{
|
||||||
["osv.ecosystem"] = ecosystem,
|
["osv.ecosystem"] = ecosystem,
|
||||||
["osv.id"] = dto.Id,
|
["osv.id"] = dto.Id,
|
||||||
["osv.modified"] = modified.ToString("O"),
|
["osv.modified"] = modified.ToString("O"),
|
||||||
};
|
};
|
||||||
|
|
||||||
var recordId = existing?.Id ?? Guid.NewGuid();
|
var recordId = existing?.Id ?? Guid.NewGuid();
|
||||||
var record = new DocumentRecord(
|
var record = new DocumentRecord(
|
||||||
recordId,
|
recordId,
|
||||||
SourceName,
|
SourceName,
|
||||||
documentUri,
|
documentUri,
|
||||||
_timeProvider.GetUtcNow(),
|
_timeProvider.GetUtcNow(),
|
||||||
sha256,
|
sha256,
|
||||||
DocumentStatuses.PendingParse,
|
DocumentStatuses.PendingParse,
|
||||||
"application/json",
|
"application/json",
|
||||||
Headers: null,
|
Headers: null,
|
||||||
Metadata: metadata,
|
Metadata: metadata,
|
||||||
Etag: null,
|
Etag: null,
|
||||||
LastModified: modified,
|
LastModified: modified,
|
||||||
GridFsId: gridFsId,
|
PayloadId: gridFsId,
|
||||||
ExpiresAt: null);
|
ExpiresAt: null);
|
||||||
|
|
||||||
var upserted = await _documentStore.UpsertAsync(record, cancellationToken).ConfigureAwait(false);
|
var upserted = await _documentStore.UpsertAsync(record, cancellationToken).ConfigureAwait(false);
|
||||||
pendingDocuments.Add(upserted.Id);
|
pendingDocuments.Add(upserted.Id);
|
||||||
newDocuments++;
|
newDocuments++;
|
||||||
remainingCapacity--;
|
remainingCapacity--;
|
||||||
|
|
||||||
if (modified > currentMaxModified)
|
if (modified > currentMaxModified)
|
||||||
{
|
{
|
||||||
currentMaxModified = modified;
|
currentMaxModified = modified;
|
||||||
currentProcessedIds = new HashSet<string>(StringComparer.OrdinalIgnoreCase) { dto.Id };
|
currentProcessedIds = new HashSet<string>(StringComparer.OrdinalIgnoreCase) { dto.Id };
|
||||||
processedUpdated = true;
|
processedUpdated = true;
|
||||||
}
|
}
|
||||||
else if (modified == currentMaxModified)
|
else if (modified == currentMaxModified)
|
||||||
{
|
{
|
||||||
currentProcessedIds.Add(dto.Id);
|
currentProcessedIds.Add(dto.Id);
|
||||||
processedUpdated = true;
|
processedUpdated = true;
|
||||||
}
|
}
|
||||||
|
|
||||||
if (_options.RequestDelay > TimeSpan.Zero)
|
if (_options.RequestDelay > TimeSpan.Zero)
|
||||||
{
|
{
|
||||||
try
|
try
|
||||||
{
|
{
|
||||||
await Task.Delay(_options.RequestDelay, cancellationToken).ConfigureAwait(false);
|
await Task.Delay(_options.RequestDelay, cancellationToken).ConfigureAwait(false);
|
||||||
}
|
}
|
||||||
catch (TaskCanceledException)
|
catch (TaskCanceledException)
|
||||||
{
|
{
|
||||||
break;
|
break;
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
if (processedUpdated && currentMaxModified != DateTimeOffset.MinValue)
|
if (processedUpdated && currentMaxModified != DateTimeOffset.MinValue)
|
||||||
{
|
{
|
||||||
cursor = cursor.WithLastModified(ecosystem, currentMaxModified, currentProcessedIds);
|
cursor = cursor.WithLastModified(ecosystem, currentMaxModified, currentProcessedIds);
|
||||||
}
|
}
|
||||||
else if (processedUpdated && existingLastModified.HasValue)
|
else if (processedUpdated && existingLastModified.HasValue)
|
||||||
{
|
{
|
||||||
cursor = cursor.WithLastModified(ecosystem, existingLastModified.Value, currentProcessedIds);
|
cursor = cursor.WithLastModified(ecosystem, existingLastModified.Value, currentProcessedIds);
|
||||||
}
|
}
|
||||||
|
|
||||||
var etag = response.Headers.ETag?.Tag;
|
var etag = response.Headers.ETag?.Tag;
|
||||||
var lastModifiedHeader = response.Content.Headers.LastModified;
|
var lastModifiedHeader = response.Content.Headers.LastModified;
|
||||||
cursor = cursor.WithArchiveMetadata(ecosystem, etag, lastModifiedHeader);
|
cursor = cursor.WithArchiveMetadata(ecosystem, etag, lastModifiedHeader);
|
||||||
|
|
||||||
return (cursor, newDocuments);
|
return (cursor, newDocuments);
|
||||||
}
|
}
|
||||||
|
|
||||||
private Uri BuildArchiveUri(string ecosystem)
|
private Uri BuildArchiveUri(string ecosystem)
|
||||||
{
|
{
|
||||||
var trimmed = ecosystem.Trim('/');
|
var trimmed = ecosystem.Trim('/');
|
||||||
var baseUri = _options.BaseUri;
|
var baseUri = _options.BaseUri;
|
||||||
var builder = new UriBuilder(baseUri);
|
var builder = new UriBuilder(baseUri);
|
||||||
var path = builder.Path;
|
var path = builder.Path;
|
||||||
if (!path.EndsWith('/'))
|
if (!path.EndsWith('/'))
|
||||||
{
|
{
|
||||||
path += "/";
|
path += "/";
|
||||||
}
|
}
|
||||||
|
|
||||||
path += $"{trimmed}/{_options.ArchiveFileName}";
|
path += $"{trimmed}/{_options.ArchiveFileName}";
|
||||||
builder.Path = path;
|
builder.Path = path;
|
||||||
return builder.Uri;
|
return builder.Uri;
|
||||||
}
|
}
|
||||||
|
|
||||||
private static string BuildDocumentUri(string ecosystem, string vulnerabilityId)
|
private static string BuildDocumentUri(string ecosystem, string vulnerabilityId)
|
||||||
{
|
{
|
||||||
var safeId = vulnerabilityId.Replace(' ', '-');
|
var safeId = vulnerabilityId.Replace(' ', '-');
|
||||||
return $"https://osv-vulnerabilities.storage.googleapis.com/{ecosystem}/{safeId}.json";
|
return $"https://osv-vulnerabilities.storage.googleapis.com/{ecosystem}/{safeId}.json";
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
@@ -1,439 +1,439 @@
|
|||||||
using System;
|
using System;
|
||||||
using System.Collections.Generic;
|
using System.Collections.Generic;
|
||||||
using System.Linq;
|
using System.Linq;
|
||||||
using System.Net;
|
using System.Net;
|
||||||
using System.Text.Json;
|
using System.Text.Json;
|
||||||
using System.Text.Json.Serialization;
|
using System.Text.Json.Serialization;
|
||||||
using System.Threading;
|
using System.Threading;
|
||||||
using System.Threading.Tasks;
|
using System.Threading.Tasks;
|
||||||
using Microsoft.Extensions.Logging;
|
using Microsoft.Extensions.Logging;
|
||||||
using Microsoft.Extensions.Options;
|
using Microsoft.Extensions.Options;
|
||||||
using MongoDB.Bson;
|
using MongoDB.Bson;
|
||||||
using StellaOps.Concelier.Connector.Common;
|
using StellaOps.Concelier.Connector.Common;
|
||||||
using StellaOps.Concelier.Connector.Common.Fetch;
|
using StellaOps.Concelier.Connector.Common.Fetch;
|
||||||
using StellaOps.Concelier.Connector.Vndr.Apple.Internal;
|
using StellaOps.Concelier.Connector.Vndr.Apple.Internal;
|
||||||
using StellaOps.Concelier.Storage.Mongo;
|
using StellaOps.Concelier.Storage.Mongo;
|
||||||
using StellaOps.Concelier.Storage.Mongo.Advisories;
|
using StellaOps.Concelier.Storage.Mongo.Advisories;
|
||||||
using StellaOps.Concelier.Storage.Mongo.Documents;
|
using StellaOps.Concelier.Storage.Mongo.Documents;
|
||||||
using StellaOps.Concelier.Storage.Mongo.Dtos;
|
using StellaOps.Concelier.Storage.Mongo.Dtos;
|
||||||
using StellaOps.Concelier.Storage.Mongo.PsirtFlags;
|
using StellaOps.Concelier.Storage.Mongo.PsirtFlags;
|
||||||
using StellaOps.Plugin;
|
using StellaOps.Plugin;
|
||||||
|
|
||||||
namespace StellaOps.Concelier.Connector.Vndr.Apple;
|
namespace StellaOps.Concelier.Connector.Vndr.Apple;
|
||||||
|
|
||||||
public sealed class AppleConnector : IFeedConnector
|
public sealed class AppleConnector : IFeedConnector
|
||||||
{
|
{
|
||||||
private static readonly JsonSerializerOptions SerializerOptions = new(JsonSerializerDefaults.Web)
|
private static readonly JsonSerializerOptions SerializerOptions = new(JsonSerializerDefaults.Web)
|
||||||
{
|
{
|
||||||
DefaultIgnoreCondition = JsonIgnoreCondition.WhenWritingNull,
|
DefaultIgnoreCondition = JsonIgnoreCondition.WhenWritingNull,
|
||||||
PropertyNameCaseInsensitive = true,
|
PropertyNameCaseInsensitive = true,
|
||||||
};
|
};
|
||||||
|
|
||||||
private readonly SourceFetchService _fetchService;
|
private readonly SourceFetchService _fetchService;
|
||||||
private readonly RawDocumentStorage _rawDocumentStorage;
|
private readonly RawDocumentStorage _rawDocumentStorage;
|
||||||
private readonly IDocumentStore _documentStore;
|
private readonly IDocumentStore _documentStore;
|
||||||
private readonly IDtoStore _dtoStore;
|
private readonly IDtoStore _dtoStore;
|
||||||
private readonly IAdvisoryStore _advisoryStore;
|
private readonly IAdvisoryStore _advisoryStore;
|
||||||
private readonly IPsirtFlagStore _psirtFlagStore;
|
private readonly IPsirtFlagStore _psirtFlagStore;
|
||||||
private readonly ISourceStateRepository _stateRepository;
|
private readonly ISourceStateRepository _stateRepository;
|
||||||
private readonly AppleOptions _options;
|
private readonly AppleOptions _options;
|
||||||
private readonly AppleDiagnostics _diagnostics;
|
private readonly AppleDiagnostics _diagnostics;
|
||||||
private readonly TimeProvider _timeProvider;
|
private readonly TimeProvider _timeProvider;
|
||||||
private readonly ILogger<AppleConnector> _logger;
|
private readonly ILogger<AppleConnector> _logger;
|
||||||
|
|
||||||
public AppleConnector(
|
public AppleConnector(
|
||||||
SourceFetchService fetchService,
|
SourceFetchService fetchService,
|
||||||
RawDocumentStorage rawDocumentStorage,
|
RawDocumentStorage rawDocumentStorage,
|
||||||
IDocumentStore documentStore,
|
IDocumentStore documentStore,
|
||||||
IDtoStore dtoStore,
|
IDtoStore dtoStore,
|
||||||
IAdvisoryStore advisoryStore,
|
IAdvisoryStore advisoryStore,
|
||||||
IPsirtFlagStore psirtFlagStore,
|
IPsirtFlagStore psirtFlagStore,
|
||||||
ISourceStateRepository stateRepository,
|
ISourceStateRepository stateRepository,
|
||||||
AppleDiagnostics diagnostics,
|
AppleDiagnostics diagnostics,
|
||||||
IOptions<AppleOptions> options,
|
IOptions<AppleOptions> options,
|
||||||
TimeProvider? timeProvider,
|
TimeProvider? timeProvider,
|
||||||
ILogger<AppleConnector> logger)
|
ILogger<AppleConnector> logger)
|
||||||
{
|
{
|
||||||
_fetchService = fetchService ?? throw new ArgumentNullException(nameof(fetchService));
|
_fetchService = fetchService ?? throw new ArgumentNullException(nameof(fetchService));
|
||||||
_rawDocumentStorage = rawDocumentStorage ?? throw new ArgumentNullException(nameof(rawDocumentStorage));
|
_rawDocumentStorage = rawDocumentStorage ?? throw new ArgumentNullException(nameof(rawDocumentStorage));
|
||||||
_documentStore = documentStore ?? throw new ArgumentNullException(nameof(documentStore));
|
_documentStore = documentStore ?? throw new ArgumentNullException(nameof(documentStore));
|
||||||
_dtoStore = dtoStore ?? throw new ArgumentNullException(nameof(dtoStore));
|
_dtoStore = dtoStore ?? throw new ArgumentNullException(nameof(dtoStore));
|
||||||
_advisoryStore = advisoryStore ?? throw new ArgumentNullException(nameof(advisoryStore));
|
_advisoryStore = advisoryStore ?? throw new ArgumentNullException(nameof(advisoryStore));
|
||||||
_psirtFlagStore = psirtFlagStore ?? throw new ArgumentNullException(nameof(psirtFlagStore));
|
_psirtFlagStore = psirtFlagStore ?? throw new ArgumentNullException(nameof(psirtFlagStore));
|
||||||
_stateRepository = stateRepository ?? throw new ArgumentNullException(nameof(stateRepository));
|
_stateRepository = stateRepository ?? throw new ArgumentNullException(nameof(stateRepository));
|
||||||
_diagnostics = diagnostics ?? throw new ArgumentNullException(nameof(diagnostics));
|
_diagnostics = diagnostics ?? throw new ArgumentNullException(nameof(diagnostics));
|
||||||
_options = (options ?? throw new ArgumentNullException(nameof(options))).Value ?? throw new ArgumentNullException(nameof(options));
|
_options = (options ?? throw new ArgumentNullException(nameof(options))).Value ?? throw new ArgumentNullException(nameof(options));
|
||||||
_options.Validate();
|
_options.Validate();
|
||||||
_timeProvider = timeProvider ?? TimeProvider.System;
|
_timeProvider = timeProvider ?? TimeProvider.System;
|
||||||
_logger = logger ?? throw new ArgumentNullException(nameof(logger));
|
_logger = logger ?? throw new ArgumentNullException(nameof(logger));
|
||||||
}
|
}
|
||||||
|
|
||||||
public string SourceName => VndrAppleConnectorPlugin.SourceName;
|
public string SourceName => VndrAppleConnectorPlugin.SourceName;
|
||||||
|
|
||||||
public async Task FetchAsync(IServiceProvider services, CancellationToken cancellationToken)
|
public async Task FetchAsync(IServiceProvider services, CancellationToken cancellationToken)
|
||||||
{
|
{
|
||||||
ArgumentNullException.ThrowIfNull(services);
|
ArgumentNullException.ThrowIfNull(services);
|
||||||
|
|
||||||
var cursor = await GetCursorAsync(cancellationToken).ConfigureAwait(false);
|
var cursor = await GetCursorAsync(cancellationToken).ConfigureAwait(false);
|
||||||
var pendingDocuments = cursor.PendingDocuments.ToHashSet();
|
var pendingDocuments = cursor.PendingDocuments.ToHashSet();
|
||||||
var pendingMappings = cursor.PendingMappings.ToHashSet();
|
var pendingMappings = cursor.PendingMappings.ToHashSet();
|
||||||
var processedIds = cursor.ProcessedIds.ToHashSet(StringComparer.OrdinalIgnoreCase);
|
var processedIds = cursor.ProcessedIds.ToHashSet(StringComparer.OrdinalIgnoreCase);
|
||||||
var maxPosted = cursor.LastPosted ?? DateTimeOffset.MinValue;
|
var maxPosted = cursor.LastPosted ?? DateTimeOffset.MinValue;
|
||||||
var baseline = cursor.LastPosted?.Add(-_options.ModifiedTolerance) ?? _timeProvider.GetUtcNow().Add(-_options.InitialBackfill);
|
var baseline = cursor.LastPosted?.Add(-_options.ModifiedTolerance) ?? _timeProvider.GetUtcNow().Add(-_options.InitialBackfill);
|
||||||
|
|
||||||
SourceFetchContentResult indexResult;
|
SourceFetchContentResult indexResult;
|
||||||
try
|
try
|
||||||
{
|
{
|
||||||
var request = new SourceFetchRequest(AppleOptions.HttpClientName, SourceName, _options.SoftwareLookupUri!)
|
var request = new SourceFetchRequest(AppleOptions.HttpClientName, SourceName, _options.SoftwareLookupUri!)
|
||||||
{
|
{
|
||||||
AcceptHeaders = new[] { "application/json", "application/vnd.apple.security+json;q=0.9" },
|
AcceptHeaders = new[] { "application/json", "application/vnd.apple.security+json;q=0.9" },
|
||||||
};
|
};
|
||||||
|
|
||||||
indexResult = await _fetchService.FetchContentAsync(request, cancellationToken).ConfigureAwait(false);
|
indexResult = await _fetchService.FetchContentAsync(request, cancellationToken).ConfigureAwait(false);
|
||||||
}
|
}
|
||||||
catch (Exception ex)
|
catch (Exception ex)
|
||||||
{
|
{
|
||||||
_diagnostics.FetchFailure();
|
_diagnostics.FetchFailure();
|
||||||
_logger.LogError(ex, "Apple software index fetch failed from {Uri}", _options.SoftwareLookupUri);
|
_logger.LogError(ex, "Apple software index fetch failed from {Uri}", _options.SoftwareLookupUri);
|
||||||
await _stateRepository.MarkFailureAsync(SourceName, _timeProvider.GetUtcNow(), TimeSpan.FromMinutes(10), ex.Message, cancellationToken).ConfigureAwait(false);
|
await _stateRepository.MarkFailureAsync(SourceName, _timeProvider.GetUtcNow(), TimeSpan.FromMinutes(10), ex.Message, cancellationToken).ConfigureAwait(false);
|
||||||
throw;
|
throw;
|
||||||
}
|
}
|
||||||
|
|
||||||
if (!indexResult.IsSuccess || indexResult.Content is null)
|
if (!indexResult.IsSuccess || indexResult.Content is null)
|
||||||
{
|
{
|
||||||
if (indexResult.IsNotModified)
|
if (indexResult.IsNotModified)
|
||||||
{
|
{
|
||||||
_diagnostics.FetchUnchanged();
|
_diagnostics.FetchUnchanged();
|
||||||
}
|
}
|
||||||
|
|
||||||
await UpdateCursorAsync(cursor, cancellationToken).ConfigureAwait(false);
|
await UpdateCursorAsync(cursor, cancellationToken).ConfigureAwait(false);
|
||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
|
|
||||||
var indexEntries = AppleIndexParser.Parse(indexResult.Content, _options.AdvisoryBaseUri!);
|
var indexEntries = AppleIndexParser.Parse(indexResult.Content, _options.AdvisoryBaseUri!);
|
||||||
if (indexEntries.Count == 0)
|
if (indexEntries.Count == 0)
|
||||||
{
|
{
|
||||||
await UpdateCursorAsync(cursor, cancellationToken).ConfigureAwait(false);
|
await UpdateCursorAsync(cursor, cancellationToken).ConfigureAwait(false);
|
||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
|
|
||||||
var allowlist = _options.AdvisoryAllowlist;
|
var allowlist = _options.AdvisoryAllowlist;
|
||||||
var blocklist = _options.AdvisoryBlocklist;
|
var blocklist = _options.AdvisoryBlocklist;
|
||||||
|
|
||||||
var ordered = indexEntries
|
var ordered = indexEntries
|
||||||
.Where(entry => ShouldInclude(entry, allowlist, blocklist))
|
.Where(entry => ShouldInclude(entry, allowlist, blocklist))
|
||||||
.OrderBy(entry => entry.PostingDate)
|
.OrderBy(entry => entry.PostingDate)
|
||||||
.ThenBy(entry => entry.ArticleId, StringComparer.OrdinalIgnoreCase)
|
.ThenBy(entry => entry.ArticleId, StringComparer.OrdinalIgnoreCase)
|
||||||
.ToArray();
|
.ToArray();
|
||||||
|
|
||||||
foreach (var entry in ordered)
|
foreach (var entry in ordered)
|
||||||
{
|
{
|
||||||
cancellationToken.ThrowIfCancellationRequested();
|
cancellationToken.ThrowIfCancellationRequested();
|
||||||
|
|
||||||
if (entry.PostingDate < baseline)
|
if (entry.PostingDate < baseline)
|
||||||
{
|
{
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
|
|
||||||
if (cursor.LastPosted.HasValue
|
if (cursor.LastPosted.HasValue
|
||||||
&& entry.PostingDate <= cursor.LastPosted.Value
|
&& entry.PostingDate <= cursor.LastPosted.Value
|
||||||
&& processedIds.Contains(entry.UpdateId))
|
&& processedIds.Contains(entry.UpdateId))
|
||||||
{
|
{
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
|
|
||||||
var metadata = BuildMetadata(entry);
|
var metadata = BuildMetadata(entry);
|
||||||
var existing = await _documentStore.FindBySourceAndUriAsync(SourceName, entry.DetailUri.ToString(), cancellationToken).ConfigureAwait(false);
|
var existing = await _documentStore.FindBySourceAndUriAsync(SourceName, entry.DetailUri.ToString(), cancellationToken).ConfigureAwait(false);
|
||||||
|
|
||||||
SourceFetchResult result;
|
SourceFetchResult result;
|
||||||
try
|
try
|
||||||
{
|
{
|
||||||
result = await _fetchService.FetchAsync(
|
result = await _fetchService.FetchAsync(
|
||||||
new SourceFetchRequest(AppleOptions.HttpClientName, SourceName, entry.DetailUri)
|
new SourceFetchRequest(AppleOptions.HttpClientName, SourceName, entry.DetailUri)
|
||||||
{
|
{
|
||||||
Metadata = metadata,
|
Metadata = metadata,
|
||||||
ETag = existing?.Etag,
|
ETag = existing?.Etag,
|
||||||
LastModified = existing?.LastModified,
|
LastModified = existing?.LastModified,
|
||||||
AcceptHeaders = new[]
|
AcceptHeaders = new[]
|
||||||
{
|
{
|
||||||
"text/html",
|
"text/html",
|
||||||
"application/xhtml+xml",
|
"application/xhtml+xml",
|
||||||
"text/plain;q=0.5"
|
"text/plain;q=0.5"
|
||||||
},
|
},
|
||||||
},
|
},
|
||||||
cancellationToken).ConfigureAwait(false);
|
cancellationToken).ConfigureAwait(false);
|
||||||
}
|
}
|
||||||
catch (Exception ex)
|
catch (Exception ex)
|
||||||
{
|
{
|
||||||
_diagnostics.FetchFailure();
|
_diagnostics.FetchFailure();
|
||||||
_logger.LogError(ex, "Apple advisory fetch failed for {Uri}", entry.DetailUri);
|
_logger.LogError(ex, "Apple advisory fetch failed for {Uri}", entry.DetailUri);
|
||||||
await _stateRepository.MarkFailureAsync(SourceName, _timeProvider.GetUtcNow(), TimeSpan.FromMinutes(5), ex.Message, cancellationToken).ConfigureAwait(false);
|
await _stateRepository.MarkFailureAsync(SourceName, _timeProvider.GetUtcNow(), TimeSpan.FromMinutes(5), ex.Message, cancellationToken).ConfigureAwait(false);
|
||||||
throw;
|
throw;
|
||||||
}
|
}
|
||||||
|
|
||||||
if (result.StatusCode == HttpStatusCode.NotModified)
|
if (result.StatusCode == HttpStatusCode.NotModified)
|
||||||
{
|
{
|
||||||
_diagnostics.FetchUnchanged();
|
_diagnostics.FetchUnchanged();
|
||||||
}
|
}
|
||||||
|
|
||||||
if (!result.IsSuccess || result.Document is null)
|
if (!result.IsSuccess || result.Document is null)
|
||||||
{
|
{
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
|
|
||||||
_diagnostics.FetchItem();
|
_diagnostics.FetchItem();
|
||||||
|
|
||||||
pendingDocuments.Add(result.Document.Id);
|
pendingDocuments.Add(result.Document.Id);
|
||||||
processedIds.Add(entry.UpdateId);
|
processedIds.Add(entry.UpdateId);
|
||||||
|
|
||||||
if (entry.PostingDate > maxPosted)
|
if (entry.PostingDate > maxPosted)
|
||||||
{
|
{
|
||||||
maxPosted = entry.PostingDate;
|
maxPosted = entry.PostingDate;
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
var updated = cursor
|
var updated = cursor
|
||||||
.WithPendingDocuments(pendingDocuments)
|
.WithPendingDocuments(pendingDocuments)
|
||||||
.WithPendingMappings(pendingMappings)
|
.WithPendingMappings(pendingMappings)
|
||||||
.WithLastPosted(maxPosted == DateTimeOffset.MinValue ? cursor.LastPosted ?? DateTimeOffset.MinValue : maxPosted, processedIds);
|
.WithLastPosted(maxPosted == DateTimeOffset.MinValue ? cursor.LastPosted ?? DateTimeOffset.MinValue : maxPosted, processedIds);
|
||||||
|
|
||||||
await UpdateCursorAsync(updated, cancellationToken).ConfigureAwait(false);
|
await UpdateCursorAsync(updated, cancellationToken).ConfigureAwait(false);
|
||||||
}
|
}
|
||||||
|
|
||||||
public async Task ParseAsync(IServiceProvider services, CancellationToken cancellationToken)
|
public async Task ParseAsync(IServiceProvider services, CancellationToken cancellationToken)
|
||||||
{
|
{
|
||||||
ArgumentNullException.ThrowIfNull(services);
|
ArgumentNullException.ThrowIfNull(services);
|
||||||
|
|
||||||
var cursor = await GetCursorAsync(cancellationToken).ConfigureAwait(false);
|
var cursor = await GetCursorAsync(cancellationToken).ConfigureAwait(false);
|
||||||
if (cursor.PendingDocuments.Count == 0)
|
if (cursor.PendingDocuments.Count == 0)
|
||||||
{
|
{
|
||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
|
|
||||||
var remainingDocuments = cursor.PendingDocuments.ToHashSet();
|
var remainingDocuments = cursor.PendingDocuments.ToHashSet();
|
||||||
var pendingMappings = cursor.PendingMappings.ToHashSet();
|
var pendingMappings = cursor.PendingMappings.ToHashSet();
|
||||||
|
|
||||||
foreach (var documentId in cursor.PendingDocuments)
|
foreach (var documentId in cursor.PendingDocuments)
|
||||||
{
|
{
|
||||||
cancellationToken.ThrowIfCancellationRequested();
|
cancellationToken.ThrowIfCancellationRequested();
|
||||||
|
|
||||||
var document = await _documentStore.FindAsync(documentId, cancellationToken).ConfigureAwait(false);
|
var document = await _documentStore.FindAsync(documentId, cancellationToken).ConfigureAwait(false);
|
||||||
if (document is null)
|
if (document is null)
|
||||||
{
|
{
|
||||||
remainingDocuments.Remove(documentId);
|
remainingDocuments.Remove(documentId);
|
||||||
pendingMappings.Remove(documentId);
|
pendingMappings.Remove(documentId);
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
|
|
||||||
if (!document.GridFsId.HasValue)
|
if (!document.PayloadId.HasValue)
|
||||||
{
|
{
|
||||||
_diagnostics.ParseFailure();
|
_diagnostics.ParseFailure();
|
||||||
_logger.LogWarning("Apple document {DocumentId} missing GridFS payload", document.Id);
|
_logger.LogWarning("Apple document {DocumentId} missing GridFS payload", document.Id);
|
||||||
await _documentStore.UpdateStatusAsync(document.Id, DocumentStatuses.Failed, cancellationToken).ConfigureAwait(false);
|
await _documentStore.UpdateStatusAsync(document.Id, DocumentStatuses.Failed, cancellationToken).ConfigureAwait(false);
|
||||||
remainingDocuments.Remove(documentId);
|
remainingDocuments.Remove(documentId);
|
||||||
pendingMappings.Remove(documentId);
|
pendingMappings.Remove(documentId);
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
|
|
||||||
AppleDetailDto dto;
|
AppleDetailDto dto;
|
||||||
try
|
try
|
||||||
{
|
{
|
||||||
var content = await _rawDocumentStorage.DownloadAsync(document.GridFsId.Value, cancellationToken).ConfigureAwait(false);
|
var content = await _rawDocumentStorage.DownloadAsync(document.PayloadId.Value, cancellationToken).ConfigureAwait(false);
|
||||||
var html = System.Text.Encoding.UTF8.GetString(content);
|
var html = System.Text.Encoding.UTF8.GetString(content);
|
||||||
var entry = RehydrateIndexEntry(document);
|
var entry = RehydrateIndexEntry(document);
|
||||||
dto = AppleDetailParser.Parse(html, entry);
|
dto = AppleDetailParser.Parse(html, entry);
|
||||||
}
|
}
|
||||||
catch (Exception ex)
|
catch (Exception ex)
|
||||||
{
|
{
|
||||||
_diagnostics.ParseFailure();
|
_diagnostics.ParseFailure();
|
||||||
_logger.LogError(ex, "Apple parse failed for document {DocumentId}", document.Id);
|
_logger.LogError(ex, "Apple parse failed for document {DocumentId}", document.Id);
|
||||||
await _documentStore.UpdateStatusAsync(document.Id, DocumentStatuses.Failed, cancellationToken).ConfigureAwait(false);
|
await _documentStore.UpdateStatusAsync(document.Id, DocumentStatuses.Failed, cancellationToken).ConfigureAwait(false);
|
||||||
remainingDocuments.Remove(documentId);
|
remainingDocuments.Remove(documentId);
|
||||||
pendingMappings.Remove(documentId);
|
pendingMappings.Remove(documentId);
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
|
|
||||||
var json = JsonSerializer.Serialize(dto, SerializerOptions);
|
var json = JsonSerializer.Serialize(dto, SerializerOptions);
|
||||||
var payload = BsonDocument.Parse(json);
|
var payload = BsonDocument.Parse(json);
|
||||||
var validatedAt = _timeProvider.GetUtcNow();
|
var validatedAt = _timeProvider.GetUtcNow();
|
||||||
|
|
||||||
var existingDto = await _dtoStore.FindByDocumentIdAsync(document.Id, cancellationToken).ConfigureAwait(false);
|
var existingDto = await _dtoStore.FindByDocumentIdAsync(document.Id, cancellationToken).ConfigureAwait(false);
|
||||||
var dtoRecord = existingDto is null
|
var dtoRecord = existingDto is null
|
||||||
? new DtoRecord(Guid.NewGuid(), document.Id, SourceName, "apple.security.update.v1", payload, validatedAt)
|
? new DtoRecord(Guid.NewGuid(), document.Id, SourceName, "apple.security.update.v1", payload, validatedAt)
|
||||||
: existingDto with
|
: existingDto with
|
||||||
{
|
{
|
||||||
Payload = payload,
|
Payload = payload,
|
||||||
SchemaVersion = "apple.security.update.v1",
|
SchemaVersion = "apple.security.update.v1",
|
||||||
ValidatedAt = validatedAt,
|
ValidatedAt = validatedAt,
|
||||||
};
|
};
|
||||||
|
|
||||||
await _dtoStore.UpsertAsync(dtoRecord, cancellationToken).ConfigureAwait(false);
|
await _dtoStore.UpsertAsync(dtoRecord, cancellationToken).ConfigureAwait(false);
|
||||||
await _documentStore.UpdateStatusAsync(document.Id, DocumentStatuses.PendingMap, cancellationToken).ConfigureAwait(false);
|
await _documentStore.UpdateStatusAsync(document.Id, DocumentStatuses.PendingMap, cancellationToken).ConfigureAwait(false);
|
||||||
|
|
||||||
remainingDocuments.Remove(documentId);
|
remainingDocuments.Remove(documentId);
|
||||||
pendingMappings.Add(document.Id);
|
pendingMappings.Add(document.Id);
|
||||||
}
|
}
|
||||||
|
|
||||||
var updatedCursor = cursor
|
var updatedCursor = cursor
|
||||||
.WithPendingDocuments(remainingDocuments)
|
.WithPendingDocuments(remainingDocuments)
|
||||||
.WithPendingMappings(pendingMappings);
|
.WithPendingMappings(pendingMappings);
|
||||||
|
|
||||||
await UpdateCursorAsync(updatedCursor, cancellationToken).ConfigureAwait(false);
|
await UpdateCursorAsync(updatedCursor, cancellationToken).ConfigureAwait(false);
|
||||||
}
|
}
|
||||||
|
|
||||||
public async Task MapAsync(IServiceProvider services, CancellationToken cancellationToken)
|
public async Task MapAsync(IServiceProvider services, CancellationToken cancellationToken)
|
||||||
{
|
{
|
||||||
ArgumentNullException.ThrowIfNull(services);
|
ArgumentNullException.ThrowIfNull(services);
|
||||||
|
|
||||||
var cursor = await GetCursorAsync(cancellationToken).ConfigureAwait(false);
|
var cursor = await GetCursorAsync(cancellationToken).ConfigureAwait(false);
|
||||||
if (cursor.PendingMappings.Count == 0)
|
if (cursor.PendingMappings.Count == 0)
|
||||||
{
|
{
|
||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
|
|
||||||
var pendingMappings = cursor.PendingMappings.ToHashSet();
|
var pendingMappings = cursor.PendingMappings.ToHashSet();
|
||||||
|
|
||||||
foreach (var documentId in cursor.PendingMappings)
|
foreach (var documentId in cursor.PendingMappings)
|
||||||
{
|
{
|
||||||
cancellationToken.ThrowIfCancellationRequested();
|
cancellationToken.ThrowIfCancellationRequested();
|
||||||
|
|
||||||
var document = await _documentStore.FindAsync(documentId, cancellationToken).ConfigureAwait(false);
|
var document = await _documentStore.FindAsync(documentId, cancellationToken).ConfigureAwait(false);
|
||||||
if (document is null)
|
if (document is null)
|
||||||
{
|
{
|
||||||
pendingMappings.Remove(documentId);
|
pendingMappings.Remove(documentId);
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
|
|
||||||
var dtoRecord = await _dtoStore.FindByDocumentIdAsync(document.Id, cancellationToken).ConfigureAwait(false);
|
var dtoRecord = await _dtoStore.FindByDocumentIdAsync(document.Id, cancellationToken).ConfigureAwait(false);
|
||||||
if (dtoRecord is null)
|
if (dtoRecord is null)
|
||||||
{
|
{
|
||||||
pendingMappings.Remove(documentId);
|
pendingMappings.Remove(documentId);
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
|
|
||||||
AppleDetailDto dto;
|
AppleDetailDto dto;
|
||||||
try
|
try
|
||||||
{
|
{
|
||||||
dto = JsonSerializer.Deserialize<AppleDetailDto>(dtoRecord.Payload.ToJson(), SerializerOptions)
|
dto = JsonSerializer.Deserialize<AppleDetailDto>(dtoRecord.Payload.ToJson(), SerializerOptions)
|
||||||
?? throw new InvalidOperationException("Unable to deserialize Apple DTO.");
|
?? throw new InvalidOperationException("Unable to deserialize Apple DTO.");
|
||||||
}
|
}
|
||||||
catch (Exception ex)
|
catch (Exception ex)
|
||||||
{
|
{
|
||||||
_logger.LogError(ex, "Apple DTO deserialization failed for document {DocumentId}", document.Id);
|
_logger.LogError(ex, "Apple DTO deserialization failed for document {DocumentId}", document.Id);
|
||||||
pendingMappings.Remove(documentId);
|
pendingMappings.Remove(documentId);
|
||||||
await _documentStore.UpdateStatusAsync(document.Id, DocumentStatuses.Failed, cancellationToken).ConfigureAwait(false);
|
await _documentStore.UpdateStatusAsync(document.Id, DocumentStatuses.Failed, cancellationToken).ConfigureAwait(false);
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
|
|
||||||
var (advisory, flag) = AppleMapper.Map(dto, document, dtoRecord);
|
var (advisory, flag) = AppleMapper.Map(dto, document, dtoRecord);
|
||||||
_diagnostics.MapAffectedCount(advisory.AffectedPackages.Length);
|
_diagnostics.MapAffectedCount(advisory.AffectedPackages.Length);
|
||||||
|
|
||||||
await _advisoryStore.UpsertAsync(advisory, cancellationToken).ConfigureAwait(false);
|
await _advisoryStore.UpsertAsync(advisory, cancellationToken).ConfigureAwait(false);
|
||||||
await _documentStore.UpdateStatusAsync(document.Id, DocumentStatuses.Mapped, cancellationToken).ConfigureAwait(false);
|
await _documentStore.UpdateStatusAsync(document.Id, DocumentStatuses.Mapped, cancellationToken).ConfigureAwait(false);
|
||||||
|
|
||||||
if (flag is not null)
|
if (flag is not null)
|
||||||
{
|
{
|
||||||
await _psirtFlagStore.UpsertAsync(flag, cancellationToken).ConfigureAwait(false);
|
await _psirtFlagStore.UpsertAsync(flag, cancellationToken).ConfigureAwait(false);
|
||||||
}
|
}
|
||||||
|
|
||||||
pendingMappings.Remove(documentId);
|
pendingMappings.Remove(documentId);
|
||||||
}
|
}
|
||||||
|
|
||||||
var updatedCursor = cursor.WithPendingMappings(pendingMappings);
|
var updatedCursor = cursor.WithPendingMappings(pendingMappings);
|
||||||
await UpdateCursorAsync(updatedCursor, cancellationToken).ConfigureAwait(false);
|
await UpdateCursorAsync(updatedCursor, cancellationToken).ConfigureAwait(false);
|
||||||
}
|
}
|
||||||
|
|
||||||
private AppleIndexEntry RehydrateIndexEntry(DocumentRecord document)
|
private AppleIndexEntry RehydrateIndexEntry(DocumentRecord document)
|
||||||
{
|
{
|
||||||
var metadata = document.Metadata ?? new Dictionary<string, string>(StringComparer.Ordinal);
|
var metadata = document.Metadata ?? new Dictionary<string, string>(StringComparer.Ordinal);
|
||||||
metadata.TryGetValue("apple.articleId", out var articleId);
|
metadata.TryGetValue("apple.articleId", out var articleId);
|
||||||
metadata.TryGetValue("apple.updateId", out var updateId);
|
metadata.TryGetValue("apple.updateId", out var updateId);
|
||||||
metadata.TryGetValue("apple.title", out var title);
|
metadata.TryGetValue("apple.title", out var title);
|
||||||
metadata.TryGetValue("apple.postingDate", out var postingDateRaw);
|
metadata.TryGetValue("apple.postingDate", out var postingDateRaw);
|
||||||
metadata.TryGetValue("apple.detailUri", out var detailUriRaw);
|
metadata.TryGetValue("apple.detailUri", out var detailUriRaw);
|
||||||
metadata.TryGetValue("apple.rapidResponse", out var rapidRaw);
|
metadata.TryGetValue("apple.rapidResponse", out var rapidRaw);
|
||||||
metadata.TryGetValue("apple.products", out var productsJson);
|
metadata.TryGetValue("apple.products", out var productsJson);
|
||||||
|
|
||||||
if (!DateTimeOffset.TryParse(postingDateRaw, out var postingDate))
|
if (!DateTimeOffset.TryParse(postingDateRaw, out var postingDate))
|
||||||
{
|
{
|
||||||
postingDate = document.FetchedAt;
|
postingDate = document.FetchedAt;
|
||||||
}
|
}
|
||||||
|
|
||||||
var detailUri = !string.IsNullOrWhiteSpace(detailUriRaw) && Uri.TryCreate(detailUriRaw, UriKind.Absolute, out var parsedUri)
|
var detailUri = !string.IsNullOrWhiteSpace(detailUriRaw) && Uri.TryCreate(detailUriRaw, UriKind.Absolute, out var parsedUri)
|
||||||
? parsedUri
|
? parsedUri
|
||||||
: new Uri(_options.AdvisoryBaseUri!, articleId ?? document.Uri);
|
: new Uri(_options.AdvisoryBaseUri!, articleId ?? document.Uri);
|
||||||
|
|
||||||
var rapid = string.Equals(rapidRaw, "true", StringComparison.OrdinalIgnoreCase);
|
var rapid = string.Equals(rapidRaw, "true", StringComparison.OrdinalIgnoreCase);
|
||||||
var products = DeserializeProducts(productsJson);
|
var products = DeserializeProducts(productsJson);
|
||||||
|
|
||||||
return new AppleIndexEntry(
|
return new AppleIndexEntry(
|
||||||
UpdateId: string.IsNullOrWhiteSpace(updateId) ? articleId ?? document.Uri : updateId,
|
UpdateId: string.IsNullOrWhiteSpace(updateId) ? articleId ?? document.Uri : updateId,
|
||||||
ArticleId: articleId ?? document.Uri,
|
ArticleId: articleId ?? document.Uri,
|
||||||
Title: title ?? document.Metadata?["apple.originalTitle"] ?? "Apple Security Update",
|
Title: title ?? document.Metadata?["apple.originalTitle"] ?? "Apple Security Update",
|
||||||
PostingDate: postingDate.ToUniversalTime(),
|
PostingDate: postingDate.ToUniversalTime(),
|
||||||
DetailUri: detailUri,
|
DetailUri: detailUri,
|
||||||
Products: products,
|
Products: products,
|
||||||
IsRapidSecurityResponse: rapid);
|
IsRapidSecurityResponse: rapid);
|
||||||
}
|
}
|
||||||
|
|
||||||
private static IReadOnlyList<AppleIndexProduct> DeserializeProducts(string? json)
|
private static IReadOnlyList<AppleIndexProduct> DeserializeProducts(string? json)
|
||||||
{
|
{
|
||||||
if (string.IsNullOrWhiteSpace(json))
|
if (string.IsNullOrWhiteSpace(json))
|
||||||
{
|
{
|
||||||
return Array.Empty<AppleIndexProduct>();
|
return Array.Empty<AppleIndexProduct>();
|
||||||
}
|
}
|
||||||
|
|
||||||
try
|
try
|
||||||
{
|
{
|
||||||
var products = JsonSerializer.Deserialize<List<AppleIndexProduct>>(json, SerializerOptions);
|
var products = JsonSerializer.Deserialize<List<AppleIndexProduct>>(json, SerializerOptions);
|
||||||
return products is { Count: > 0 } ? products : Array.Empty<AppleIndexProduct>();
|
return products is { Count: > 0 } ? products : Array.Empty<AppleIndexProduct>();
|
||||||
}
|
}
|
||||||
catch (JsonException)
|
catch (JsonException)
|
||||||
{
|
{
|
||||||
return Array.Empty<AppleIndexProduct>();
|
return Array.Empty<AppleIndexProduct>();
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
private static Dictionary<string, string> BuildMetadata(AppleIndexEntry entry)
|
private static Dictionary<string, string> BuildMetadata(AppleIndexEntry entry)
|
||||||
{
|
{
|
||||||
var metadata = new Dictionary<string, string>(StringComparer.Ordinal)
|
var metadata = new Dictionary<string, string>(StringComparer.Ordinal)
|
||||||
{
|
{
|
||||||
["apple.articleId"] = entry.ArticleId,
|
["apple.articleId"] = entry.ArticleId,
|
||||||
["apple.updateId"] = entry.UpdateId,
|
["apple.updateId"] = entry.UpdateId,
|
||||||
["apple.title"] = entry.Title,
|
["apple.title"] = entry.Title,
|
||||||
["apple.postingDate"] = entry.PostingDate.ToString("O"),
|
["apple.postingDate"] = entry.PostingDate.ToString("O"),
|
||||||
["apple.detailUri"] = entry.DetailUri.ToString(),
|
["apple.detailUri"] = entry.DetailUri.ToString(),
|
||||||
["apple.rapidResponse"] = entry.IsRapidSecurityResponse ? "true" : "false",
|
["apple.rapidResponse"] = entry.IsRapidSecurityResponse ? "true" : "false",
|
||||||
["apple.products"] = JsonSerializer.Serialize(entry.Products, SerializerOptions),
|
["apple.products"] = JsonSerializer.Serialize(entry.Products, SerializerOptions),
|
||||||
};
|
};
|
||||||
|
|
||||||
return metadata;
|
return metadata;
|
||||||
}
|
}
|
||||||
|
|
||||||
private static bool ShouldInclude(AppleIndexEntry entry, IReadOnlyCollection<string> allowlist, IReadOnlyCollection<string> blocklist)
|
private static bool ShouldInclude(AppleIndexEntry entry, IReadOnlyCollection<string> allowlist, IReadOnlyCollection<string> blocklist)
|
||||||
{
|
{
|
||||||
if (allowlist.Count > 0 && !allowlist.Contains(entry.ArticleId))
|
if (allowlist.Count > 0 && !allowlist.Contains(entry.ArticleId))
|
||||||
{
|
{
|
||||||
return false;
|
return false;
|
||||||
}
|
}
|
||||||
|
|
||||||
if (blocklist.Count > 0 && blocklist.Contains(entry.ArticleId))
|
if (blocklist.Count > 0 && blocklist.Contains(entry.ArticleId))
|
||||||
{
|
{
|
||||||
return false;
|
return false;
|
||||||
}
|
}
|
||||||
|
|
||||||
return true;
|
return true;
|
||||||
}
|
}
|
||||||
|
|
||||||
private async Task<AppleCursor> GetCursorAsync(CancellationToken cancellationToken)
|
private async Task<AppleCursor> GetCursorAsync(CancellationToken cancellationToken)
|
||||||
{
|
{
|
||||||
var state = await _stateRepository.TryGetAsync(SourceName, cancellationToken).ConfigureAwait(false);
|
var state = await _stateRepository.TryGetAsync(SourceName, cancellationToken).ConfigureAwait(false);
|
||||||
return state is null ? AppleCursor.Empty : AppleCursor.FromBson(state.Cursor);
|
return state is null ? AppleCursor.Empty : AppleCursor.FromBson(state.Cursor);
|
||||||
}
|
}
|
||||||
|
|
||||||
private async Task UpdateCursorAsync(AppleCursor cursor, CancellationToken cancellationToken)
|
private async Task UpdateCursorAsync(AppleCursor cursor, CancellationToken cancellationToken)
|
||||||
{
|
{
|
||||||
var document = cursor.ToBson();
|
var document = cursor.ToBson();
|
||||||
await _stateRepository.UpdateCursorAsync(SourceName, document, _timeProvider.GetUtcNow(), cancellationToken).ConfigureAwait(false);
|
await _stateRepository.UpdateCursorAsync(SourceName, document, _timeProvider.GetUtcNow(), cancellationToken).ConfigureAwait(false);
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -1,366 +1,366 @@
|
|||||||
using System.Collections.Generic;
|
using System.Collections.Generic;
|
||||||
using System.Linq;
|
using System.Linq;
|
||||||
using System.Text;
|
using System.Text;
|
||||||
using System.Text.Json;
|
using System.Text.Json;
|
||||||
using Microsoft.Extensions.Logging;
|
using Microsoft.Extensions.Logging;
|
||||||
using Microsoft.Extensions.Options;
|
using Microsoft.Extensions.Options;
|
||||||
using MongoDB.Bson;
|
using MongoDB.Bson;
|
||||||
using MongoDB.Bson.IO;
|
using MongoDB.Bson.IO;
|
||||||
using StellaOps.Concelier.Models;
|
using StellaOps.Concelier.Models;
|
||||||
using StellaOps.Concelier.Connector.Common;
|
using StellaOps.Concelier.Connector.Common;
|
||||||
using StellaOps.Concelier.Connector.Common.Fetch;
|
using StellaOps.Concelier.Connector.Common.Fetch;
|
||||||
using StellaOps.Concelier.Connector.Common.Json;
|
using StellaOps.Concelier.Connector.Common.Json;
|
||||||
using StellaOps.Concelier.Connector.Vndr.Chromium.Configuration;
|
using StellaOps.Concelier.Connector.Vndr.Chromium.Configuration;
|
||||||
using StellaOps.Concelier.Connector.Vndr.Chromium.Internal;
|
using StellaOps.Concelier.Connector.Vndr.Chromium.Internal;
|
||||||
using StellaOps.Concelier.Storage.Mongo;
|
using StellaOps.Concelier.Storage.Mongo;
|
||||||
using StellaOps.Concelier.Storage.Mongo.Advisories;
|
using StellaOps.Concelier.Storage.Mongo.Advisories;
|
||||||
using StellaOps.Concelier.Storage.Mongo.Documents;
|
using StellaOps.Concelier.Storage.Mongo.Documents;
|
||||||
using StellaOps.Concelier.Storage.Mongo.Dtos;
|
using StellaOps.Concelier.Storage.Mongo.Dtos;
|
||||||
using StellaOps.Concelier.Storage.Mongo.PsirtFlags;
|
using StellaOps.Concelier.Storage.Mongo.PsirtFlags;
|
||||||
using StellaOps.Plugin;
|
using StellaOps.Plugin;
|
||||||
using Json.Schema;
|
using Json.Schema;
|
||||||
|
|
||||||
namespace StellaOps.Concelier.Connector.Vndr.Chromium;
|
namespace StellaOps.Concelier.Connector.Vndr.Chromium;
|
||||||
|
|
||||||
public sealed class ChromiumConnector : IFeedConnector
|
public sealed class ChromiumConnector : IFeedConnector
|
||||||
{
|
{
|
||||||
private static readonly JsonSchema Schema = ChromiumSchemaProvider.Schema;
|
private static readonly JsonSchema Schema = ChromiumSchemaProvider.Schema;
|
||||||
private static readonly JsonSerializerOptions SerializerOptions = new()
|
private static readonly JsonSerializerOptions SerializerOptions = new()
|
||||||
{
|
{
|
||||||
PropertyNamingPolicy = JsonNamingPolicy.CamelCase,
|
PropertyNamingPolicy = JsonNamingPolicy.CamelCase,
|
||||||
DefaultIgnoreCondition = System.Text.Json.Serialization.JsonIgnoreCondition.WhenWritingNull,
|
DefaultIgnoreCondition = System.Text.Json.Serialization.JsonIgnoreCondition.WhenWritingNull,
|
||||||
};
|
};
|
||||||
|
|
||||||
private readonly ChromiumFeedLoader _feedLoader;
|
private readonly ChromiumFeedLoader _feedLoader;
|
||||||
private readonly SourceFetchService _fetchService;
|
private readonly SourceFetchService _fetchService;
|
||||||
private readonly RawDocumentStorage _rawDocumentStorage;
|
private readonly RawDocumentStorage _rawDocumentStorage;
|
||||||
private readonly IDocumentStore _documentStore;
|
private readonly IDocumentStore _documentStore;
|
||||||
private readonly IDtoStore _dtoStore;
|
private readonly IDtoStore _dtoStore;
|
||||||
private readonly IAdvisoryStore _advisoryStore;
|
private readonly IAdvisoryStore _advisoryStore;
|
||||||
private readonly IPsirtFlagStore _psirtFlagStore;
|
private readonly IPsirtFlagStore _psirtFlagStore;
|
||||||
private readonly ISourceStateRepository _stateRepository;
|
private readonly ISourceStateRepository _stateRepository;
|
||||||
private readonly IJsonSchemaValidator _schemaValidator;
|
private readonly IJsonSchemaValidator _schemaValidator;
|
||||||
private readonly ChromiumOptions _options;
|
private readonly ChromiumOptions _options;
|
||||||
private readonly TimeProvider _timeProvider;
|
private readonly TimeProvider _timeProvider;
|
||||||
private readonly ChromiumDiagnostics _diagnostics;
|
private readonly ChromiumDiagnostics _diagnostics;
|
||||||
private readonly ILogger<ChromiumConnector> _logger;
|
private readonly ILogger<ChromiumConnector> _logger;
|
||||||
|
|
||||||
public ChromiumConnector(
|
public ChromiumConnector(
|
||||||
ChromiumFeedLoader feedLoader,
|
ChromiumFeedLoader feedLoader,
|
||||||
SourceFetchService fetchService,
|
SourceFetchService fetchService,
|
||||||
RawDocumentStorage rawDocumentStorage,
|
RawDocumentStorage rawDocumentStorage,
|
||||||
IDocumentStore documentStore,
|
IDocumentStore documentStore,
|
||||||
IDtoStore dtoStore,
|
IDtoStore dtoStore,
|
||||||
IAdvisoryStore advisoryStore,
|
IAdvisoryStore advisoryStore,
|
||||||
IPsirtFlagStore psirtFlagStore,
|
IPsirtFlagStore psirtFlagStore,
|
||||||
ISourceStateRepository stateRepository,
|
ISourceStateRepository stateRepository,
|
||||||
IJsonSchemaValidator schemaValidator,
|
IJsonSchemaValidator schemaValidator,
|
||||||
IOptions<ChromiumOptions> options,
|
IOptions<ChromiumOptions> options,
|
||||||
TimeProvider? timeProvider,
|
TimeProvider? timeProvider,
|
||||||
ChromiumDiagnostics diagnostics,
|
ChromiumDiagnostics diagnostics,
|
||||||
ILogger<ChromiumConnector> logger)
|
ILogger<ChromiumConnector> logger)
|
||||||
{
|
{
|
||||||
_feedLoader = feedLoader ?? throw new ArgumentNullException(nameof(feedLoader));
|
_feedLoader = feedLoader ?? throw new ArgumentNullException(nameof(feedLoader));
|
||||||
_fetchService = fetchService ?? throw new ArgumentNullException(nameof(fetchService));
|
_fetchService = fetchService ?? throw new ArgumentNullException(nameof(fetchService));
|
||||||
_rawDocumentStorage = rawDocumentStorage ?? throw new ArgumentNullException(nameof(rawDocumentStorage));
|
_rawDocumentStorage = rawDocumentStorage ?? throw new ArgumentNullException(nameof(rawDocumentStorage));
|
||||||
_documentStore = documentStore ?? throw new ArgumentNullException(nameof(documentStore));
|
_documentStore = documentStore ?? throw new ArgumentNullException(nameof(documentStore));
|
||||||
_dtoStore = dtoStore ?? throw new ArgumentNullException(nameof(dtoStore));
|
_dtoStore = dtoStore ?? throw new ArgumentNullException(nameof(dtoStore));
|
||||||
_advisoryStore = advisoryStore ?? throw new ArgumentNullException(nameof(advisoryStore));
|
_advisoryStore = advisoryStore ?? throw new ArgumentNullException(nameof(advisoryStore));
|
||||||
_psirtFlagStore = psirtFlagStore ?? throw new ArgumentNullException(nameof(psirtFlagStore));
|
_psirtFlagStore = psirtFlagStore ?? throw new ArgumentNullException(nameof(psirtFlagStore));
|
||||||
_stateRepository = stateRepository ?? throw new ArgumentNullException(nameof(stateRepository));
|
_stateRepository = stateRepository ?? throw new ArgumentNullException(nameof(stateRepository));
|
||||||
_schemaValidator = schemaValidator ?? throw new ArgumentNullException(nameof(schemaValidator));
|
_schemaValidator = schemaValidator ?? throw new ArgumentNullException(nameof(schemaValidator));
|
||||||
_options = options?.Value ?? throw new ArgumentNullException(nameof(options));
|
_options = options?.Value ?? throw new ArgumentNullException(nameof(options));
|
||||||
_options.Validate();
|
_options.Validate();
|
||||||
_timeProvider = timeProvider ?? TimeProvider.System;
|
_timeProvider = timeProvider ?? TimeProvider.System;
|
||||||
_diagnostics = diagnostics ?? throw new ArgumentNullException(nameof(diagnostics));
|
_diagnostics = diagnostics ?? throw new ArgumentNullException(nameof(diagnostics));
|
||||||
_logger = logger ?? throw new ArgumentNullException(nameof(logger));
|
_logger = logger ?? throw new ArgumentNullException(nameof(logger));
|
||||||
}
|
}
|
||||||
|
|
||||||
public string SourceName => VndrChromiumConnectorPlugin.SourceName;
|
public string SourceName => VndrChromiumConnectorPlugin.SourceName;
|
||||||
|
|
||||||
public async Task FetchAsync(IServiceProvider services, CancellationToken cancellationToken)
|
public async Task FetchAsync(IServiceProvider services, CancellationToken cancellationToken)
|
||||||
{
|
{
|
||||||
var cursor = await GetCursorAsync(cancellationToken).ConfigureAwait(false);
|
var cursor = await GetCursorAsync(cancellationToken).ConfigureAwait(false);
|
||||||
var now = _timeProvider.GetUtcNow();
|
var now = _timeProvider.GetUtcNow();
|
||||||
var (windowStart, windowEnd) = CalculateWindow(cursor, now);
|
var (windowStart, windowEnd) = CalculateWindow(cursor, now);
|
||||||
ProvenanceDiagnostics.ReportResumeWindow(SourceName, windowStart, _logger);
|
ProvenanceDiagnostics.ReportResumeWindow(SourceName, windowStart, _logger);
|
||||||
|
|
||||||
IReadOnlyList<ChromiumFeedEntry> feedEntries;
|
IReadOnlyList<ChromiumFeedEntry> feedEntries;
|
||||||
_diagnostics.FetchAttempt();
|
_diagnostics.FetchAttempt();
|
||||||
try
|
try
|
||||||
{
|
{
|
||||||
feedEntries = await _feedLoader.LoadAsync(windowStart, windowEnd, cancellationToken).ConfigureAwait(false);
|
feedEntries = await _feedLoader.LoadAsync(windowStart, windowEnd, cancellationToken).ConfigureAwait(false);
|
||||||
}
|
}
|
||||||
catch (Exception ex)
|
catch (Exception ex)
|
||||||
{
|
{
|
||||||
_logger.LogError(ex, "Chromium feed load failed {Start}-{End}", windowStart, windowEnd);
|
_logger.LogError(ex, "Chromium feed load failed {Start}-{End}", windowStart, windowEnd);
|
||||||
_diagnostics.FetchFailure();
|
_diagnostics.FetchFailure();
|
||||||
await _stateRepository.MarkFailureAsync(SourceName, now, TimeSpan.FromMinutes(10), ex.Message, cancellationToken).ConfigureAwait(false);
|
await _stateRepository.MarkFailureAsync(SourceName, now, TimeSpan.FromMinutes(10), ex.Message, cancellationToken).ConfigureAwait(false);
|
||||||
throw;
|
throw;
|
||||||
}
|
}
|
||||||
|
|
||||||
var fetchCache = new Dictionary<string, ChromiumFetchCacheEntry>(cursor.FetchCache, StringComparer.Ordinal);
|
var fetchCache = new Dictionary<string, ChromiumFetchCacheEntry>(cursor.FetchCache, StringComparer.Ordinal);
|
||||||
var touchedResources = new HashSet<string>(StringComparer.Ordinal);
|
var touchedResources = new HashSet<string>(StringComparer.Ordinal);
|
||||||
|
|
||||||
var candidates = feedEntries
|
var candidates = feedEntries
|
||||||
.Where(static entry => entry.IsSecurityUpdate())
|
.Where(static entry => entry.IsSecurityUpdate())
|
||||||
.OrderBy(static entry => entry.Published)
|
.OrderBy(static entry => entry.Published)
|
||||||
.ToArray();
|
.ToArray();
|
||||||
|
|
||||||
if (candidates.Length == 0)
|
if (candidates.Length == 0)
|
||||||
{
|
{
|
||||||
var untouched = cursor
|
var untouched = cursor
|
||||||
.WithLastPublished(cursor.LastPublished ?? windowEnd)
|
.WithLastPublished(cursor.LastPublished ?? windowEnd)
|
||||||
.WithFetchCache(fetchCache);
|
.WithFetchCache(fetchCache);
|
||||||
await UpdateCursorAsync(untouched, cancellationToken).ConfigureAwait(false);
|
await UpdateCursorAsync(untouched, cancellationToken).ConfigureAwait(false);
|
||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
|
|
||||||
var pendingDocuments = cursor.PendingDocuments.ToList();
|
var pendingDocuments = cursor.PendingDocuments.ToList();
|
||||||
var maxPublished = cursor.LastPublished;
|
var maxPublished = cursor.LastPublished;
|
||||||
|
|
||||||
foreach (var entry in candidates)
|
foreach (var entry in candidates)
|
||||||
{
|
{
|
||||||
try
|
try
|
||||||
{
|
{
|
||||||
var cacheKey = entry.DetailUri.ToString();
|
var cacheKey = entry.DetailUri.ToString();
|
||||||
touchedResources.Add(cacheKey);
|
touchedResources.Add(cacheKey);
|
||||||
|
|
||||||
var metadata = ChromiumDocumentMetadata.CreateMetadata(entry.PostId, entry.Title, entry.Published, entry.Updated, entry.Summary);
|
var metadata = ChromiumDocumentMetadata.CreateMetadata(entry.PostId, entry.Title, entry.Published, entry.Updated, entry.Summary);
|
||||||
var request = new SourceFetchRequest(ChromiumOptions.HttpClientName, SourceName, entry.DetailUri)
|
var request = new SourceFetchRequest(ChromiumOptions.HttpClientName, SourceName, entry.DetailUri)
|
||||||
{
|
{
|
||||||
Metadata = metadata,
|
Metadata = metadata,
|
||||||
AcceptHeaders = new[] { "text/html", "application/xhtml+xml", "text/plain;q=0.5" },
|
AcceptHeaders = new[] { "text/html", "application/xhtml+xml", "text/plain;q=0.5" },
|
||||||
};
|
};
|
||||||
|
|
||||||
var result = await _fetchService.FetchAsync(request, cancellationToken).ConfigureAwait(false);
|
var result = await _fetchService.FetchAsync(request, cancellationToken).ConfigureAwait(false);
|
||||||
if (!result.IsSuccess || result.Document is null)
|
if (!result.IsSuccess || result.Document is null)
|
||||||
{
|
{
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
|
|
||||||
if (cursor.TryGetFetchCache(cacheKey, out var cached) && string.Equals(cached.Sha256, result.Document.Sha256, StringComparison.OrdinalIgnoreCase))
|
if (cursor.TryGetFetchCache(cacheKey, out var cached) && string.Equals(cached.Sha256, result.Document.Sha256, StringComparison.OrdinalIgnoreCase))
|
||||||
{
|
{
|
||||||
_diagnostics.FetchUnchanged();
|
_diagnostics.FetchUnchanged();
|
||||||
fetchCache[cacheKey] = new ChromiumFetchCacheEntry(result.Document.Sha256);
|
fetchCache[cacheKey] = new ChromiumFetchCacheEntry(result.Document.Sha256);
|
||||||
await _documentStore.UpdateStatusAsync(result.Document.Id, DocumentStatuses.Mapped, cancellationToken).ConfigureAwait(false);
|
await _documentStore.UpdateStatusAsync(result.Document.Id, DocumentStatuses.Mapped, cancellationToken).ConfigureAwait(false);
|
||||||
|
|
||||||
if (!maxPublished.HasValue || entry.Published > maxPublished)
|
if (!maxPublished.HasValue || entry.Published > maxPublished)
|
||||||
{
|
{
|
||||||
maxPublished = entry.Published;
|
maxPublished = entry.Published;
|
||||||
}
|
}
|
||||||
|
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
|
|
||||||
_diagnostics.FetchDocument();
|
_diagnostics.FetchDocument();
|
||||||
if (!pendingDocuments.Contains(result.Document.Id))
|
if (!pendingDocuments.Contains(result.Document.Id))
|
||||||
{
|
{
|
||||||
pendingDocuments.Add(result.Document.Id);
|
pendingDocuments.Add(result.Document.Id);
|
||||||
}
|
}
|
||||||
|
|
||||||
if (!maxPublished.HasValue || entry.Published > maxPublished)
|
if (!maxPublished.HasValue || entry.Published > maxPublished)
|
||||||
{
|
{
|
||||||
maxPublished = entry.Published;
|
maxPublished = entry.Published;
|
||||||
}
|
}
|
||||||
|
|
||||||
fetchCache[cacheKey] = new ChromiumFetchCacheEntry(result.Document.Sha256);
|
fetchCache[cacheKey] = new ChromiumFetchCacheEntry(result.Document.Sha256);
|
||||||
}
|
}
|
||||||
catch (Exception ex)
|
catch (Exception ex)
|
||||||
{
|
{
|
||||||
_logger.LogError(ex, "Chromium fetch failed for {Uri}", entry.DetailUri);
|
_logger.LogError(ex, "Chromium fetch failed for {Uri}", entry.DetailUri);
|
||||||
_diagnostics.FetchFailure();
|
_diagnostics.FetchFailure();
|
||||||
await _stateRepository.MarkFailureAsync(SourceName, now, TimeSpan.FromMinutes(5), ex.Message, cancellationToken).ConfigureAwait(false);
|
await _stateRepository.MarkFailureAsync(SourceName, now, TimeSpan.FromMinutes(5), ex.Message, cancellationToken).ConfigureAwait(false);
|
||||||
throw;
|
throw;
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
if (touchedResources.Count > 0)
|
if (touchedResources.Count > 0)
|
||||||
{
|
{
|
||||||
var keysToRemove = fetchCache.Keys.Where(key => !touchedResources.Contains(key)).ToArray();
|
var keysToRemove = fetchCache.Keys.Where(key => !touchedResources.Contains(key)).ToArray();
|
||||||
foreach (var key in keysToRemove)
|
foreach (var key in keysToRemove)
|
||||||
{
|
{
|
||||||
fetchCache.Remove(key);
|
fetchCache.Remove(key);
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
var updatedCursor = cursor
|
var updatedCursor = cursor
|
||||||
.WithPendingDocuments(pendingDocuments)
|
.WithPendingDocuments(pendingDocuments)
|
||||||
.WithPendingMappings(cursor.PendingMappings)
|
.WithPendingMappings(cursor.PendingMappings)
|
||||||
.WithLastPublished(maxPublished ?? cursor.LastPublished ?? windowEnd)
|
.WithLastPublished(maxPublished ?? cursor.LastPublished ?? windowEnd)
|
||||||
.WithFetchCache(fetchCache);
|
.WithFetchCache(fetchCache);
|
||||||
|
|
||||||
await UpdateCursorAsync(updatedCursor, cancellationToken).ConfigureAwait(false);
|
await UpdateCursorAsync(updatedCursor, cancellationToken).ConfigureAwait(false);
|
||||||
}
|
}
|
||||||
|
|
||||||
public async Task ParseAsync(IServiceProvider services, CancellationToken cancellationToken)
|
public async Task ParseAsync(IServiceProvider services, CancellationToken cancellationToken)
|
||||||
{
|
{
|
||||||
var cursor = await GetCursorAsync(cancellationToken).ConfigureAwait(false);
|
var cursor = await GetCursorAsync(cancellationToken).ConfigureAwait(false);
|
||||||
if (cursor.PendingDocuments.Count == 0)
|
if (cursor.PendingDocuments.Count == 0)
|
||||||
{
|
{
|
||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
|
|
||||||
var pendingDocuments = cursor.PendingDocuments.ToList();
|
var pendingDocuments = cursor.PendingDocuments.ToList();
|
||||||
var pendingMappings = cursor.PendingMappings.ToList();
|
var pendingMappings = cursor.PendingMappings.ToList();
|
||||||
|
|
||||||
foreach (var documentId in cursor.PendingDocuments)
|
foreach (var documentId in cursor.PendingDocuments)
|
||||||
{
|
{
|
||||||
var document = await _documentStore.FindAsync(documentId, cancellationToken).ConfigureAwait(false);
|
var document = await _documentStore.FindAsync(documentId, cancellationToken).ConfigureAwait(false);
|
||||||
if (document is null)
|
if (document is null)
|
||||||
{
|
{
|
||||||
pendingDocuments.Remove(documentId);
|
pendingDocuments.Remove(documentId);
|
||||||
pendingMappings.Remove(documentId);
|
pendingMappings.Remove(documentId);
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
|
|
||||||
if (!document.GridFsId.HasValue)
|
if (!document.PayloadId.HasValue)
|
||||||
{
|
{
|
||||||
_logger.LogWarning("Chromium document {DocumentId} missing GridFS payload", document.Id);
|
_logger.LogWarning("Chromium document {DocumentId} missing GridFS payload", document.Id);
|
||||||
await _documentStore.UpdateStatusAsync(document.Id, DocumentStatuses.Failed, cancellationToken).ConfigureAwait(false);
|
await _documentStore.UpdateStatusAsync(document.Id, DocumentStatuses.Failed, cancellationToken).ConfigureAwait(false);
|
||||||
pendingDocuments.Remove(documentId);
|
pendingDocuments.Remove(documentId);
|
||||||
pendingMappings.Remove(documentId);
|
pendingMappings.Remove(documentId);
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
|
|
||||||
ChromiumDto dto;
|
ChromiumDto dto;
|
||||||
try
|
try
|
||||||
{
|
{
|
||||||
var metadata = ChromiumDocumentMetadata.FromDocument(document);
|
var metadata = ChromiumDocumentMetadata.FromDocument(document);
|
||||||
var content = await _rawDocumentStorage.DownloadAsync(document.GridFsId.Value, cancellationToken).ConfigureAwait(false);
|
var content = await _rawDocumentStorage.DownloadAsync(document.PayloadId.Value, cancellationToken).ConfigureAwait(false);
|
||||||
var html = Encoding.UTF8.GetString(content);
|
var html = Encoding.UTF8.GetString(content);
|
||||||
dto = ChromiumParser.Parse(html, metadata);
|
dto = ChromiumParser.Parse(html, metadata);
|
||||||
}
|
}
|
||||||
catch (Exception ex)
|
catch (Exception ex)
|
||||||
{
|
{
|
||||||
_logger.LogError(ex, "Chromium parse failed for {Uri}", document.Uri);
|
_logger.LogError(ex, "Chromium parse failed for {Uri}", document.Uri);
|
||||||
_diagnostics.ParseFailure();
|
_diagnostics.ParseFailure();
|
||||||
await _documentStore.UpdateStatusAsync(document.Id, DocumentStatuses.Failed, cancellationToken).ConfigureAwait(false);
|
await _documentStore.UpdateStatusAsync(document.Id, DocumentStatuses.Failed, cancellationToken).ConfigureAwait(false);
|
||||||
pendingDocuments.Remove(documentId);
|
pendingDocuments.Remove(documentId);
|
||||||
pendingMappings.Remove(documentId);
|
pendingMappings.Remove(documentId);
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
|
|
||||||
var json = JsonSerializer.Serialize(dto, SerializerOptions);
|
var json = JsonSerializer.Serialize(dto, SerializerOptions);
|
||||||
using var jsonDocument = JsonDocument.Parse(json);
|
using var jsonDocument = JsonDocument.Parse(json);
|
||||||
try
|
try
|
||||||
{
|
{
|
||||||
_schemaValidator.Validate(jsonDocument, Schema, dto.PostId);
|
_schemaValidator.Validate(jsonDocument, Schema, dto.PostId);
|
||||||
}
|
}
|
||||||
catch (StellaOps.Concelier.Connector.Common.Json.JsonSchemaValidationException ex)
|
catch (StellaOps.Concelier.Connector.Common.Json.JsonSchemaValidationException ex)
|
||||||
{
|
{
|
||||||
_logger.LogError(ex, "Chromium schema validation failed for {DocumentId}", document.Id);
|
_logger.LogError(ex, "Chromium schema validation failed for {DocumentId}", document.Id);
|
||||||
await _documentStore.UpdateStatusAsync(document.Id, DocumentStatuses.Failed, cancellationToken).ConfigureAwait(false);
|
await _documentStore.UpdateStatusAsync(document.Id, DocumentStatuses.Failed, cancellationToken).ConfigureAwait(false);
|
||||||
pendingDocuments.Remove(documentId);
|
pendingDocuments.Remove(documentId);
|
||||||
pendingMappings.Remove(documentId);
|
pendingMappings.Remove(documentId);
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
|
|
||||||
var payload = BsonDocument.Parse(json);
|
var payload = BsonDocument.Parse(json);
|
||||||
var existingDto = await _dtoStore.FindByDocumentIdAsync(document.Id, cancellationToken).ConfigureAwait(false);
|
var existingDto = await _dtoStore.FindByDocumentIdAsync(document.Id, cancellationToken).ConfigureAwait(false);
|
||||||
var validatedAt = _timeProvider.GetUtcNow();
|
var validatedAt = _timeProvider.GetUtcNow();
|
||||||
|
|
||||||
var dtoRecord = existingDto is null
|
var dtoRecord = existingDto is null
|
||||||
? new DtoRecord(Guid.NewGuid(), document.Id, SourceName, "chromium.post.v1", payload, validatedAt)
|
? new DtoRecord(Guid.NewGuid(), document.Id, SourceName, "chromium.post.v1", payload, validatedAt)
|
||||||
: existingDto with
|
: existingDto with
|
||||||
{
|
{
|
||||||
Payload = payload,
|
Payload = payload,
|
||||||
SchemaVersion = "chromium.post.v1",
|
SchemaVersion = "chromium.post.v1",
|
||||||
ValidatedAt = validatedAt,
|
ValidatedAt = validatedAt,
|
||||||
};
|
};
|
||||||
|
|
||||||
await _dtoStore.UpsertAsync(dtoRecord, cancellationToken).ConfigureAwait(false);
|
await _dtoStore.UpsertAsync(dtoRecord, cancellationToken).ConfigureAwait(false);
|
||||||
await _documentStore.UpdateStatusAsync(document.Id, DocumentStatuses.PendingMap, cancellationToken).ConfigureAwait(false);
|
await _documentStore.UpdateStatusAsync(document.Id, DocumentStatuses.PendingMap, cancellationToken).ConfigureAwait(false);
|
||||||
_diagnostics.ParseSuccess();
|
_diagnostics.ParseSuccess();
|
||||||
|
|
||||||
pendingDocuments.Remove(documentId);
|
pendingDocuments.Remove(documentId);
|
||||||
if (!pendingMappings.Contains(documentId))
|
if (!pendingMappings.Contains(documentId))
|
||||||
{
|
{
|
||||||
pendingMappings.Add(documentId);
|
pendingMappings.Add(documentId);
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
var updatedCursor = cursor
|
var updatedCursor = cursor
|
||||||
.WithPendingDocuments(pendingDocuments)
|
.WithPendingDocuments(pendingDocuments)
|
||||||
.WithPendingMappings(pendingMappings);
|
.WithPendingMappings(pendingMappings);
|
||||||
|
|
||||||
await UpdateCursorAsync(updatedCursor, cancellationToken).ConfigureAwait(false);
|
await UpdateCursorAsync(updatedCursor, cancellationToken).ConfigureAwait(false);
|
||||||
}
|
}
|
||||||
|
|
||||||
public async Task MapAsync(IServiceProvider services, CancellationToken cancellationToken)
|
public async Task MapAsync(IServiceProvider services, CancellationToken cancellationToken)
|
||||||
{
|
{
|
||||||
var cursor = await GetCursorAsync(cancellationToken).ConfigureAwait(false);
|
var cursor = await GetCursorAsync(cancellationToken).ConfigureAwait(false);
|
||||||
if (cursor.PendingMappings.Count == 0)
|
if (cursor.PendingMappings.Count == 0)
|
||||||
{
|
{
|
||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
|
|
||||||
var pendingMappings = cursor.PendingMappings.ToList();
|
var pendingMappings = cursor.PendingMappings.ToList();
|
||||||
|
|
||||||
foreach (var documentId in cursor.PendingMappings)
|
foreach (var documentId in cursor.PendingMappings)
|
||||||
{
|
{
|
||||||
var dtoRecord = await _dtoStore.FindByDocumentIdAsync(documentId, cancellationToken).ConfigureAwait(false);
|
var dtoRecord = await _dtoStore.FindByDocumentIdAsync(documentId, cancellationToken).ConfigureAwait(false);
|
||||||
var document = await _documentStore.FindAsync(documentId, cancellationToken).ConfigureAwait(false);
|
var document = await _documentStore.FindAsync(documentId, cancellationToken).ConfigureAwait(false);
|
||||||
|
|
||||||
if (dtoRecord is null || document is null)
|
if (dtoRecord is null || document is null)
|
||||||
{
|
{
|
||||||
pendingMappings.Remove(documentId);
|
pendingMappings.Remove(documentId);
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
|
|
||||||
var json = dtoRecord.Payload.ToJson(new JsonWriterSettings { OutputMode = JsonOutputMode.RelaxedExtendedJson });
|
var json = dtoRecord.Payload.ToJson(new JsonWriterSettings { OutputMode = JsonOutputMode.RelaxedExtendedJson });
|
||||||
var dto = JsonSerializer.Deserialize<ChromiumDto>(json, SerializerOptions);
|
var dto = JsonSerializer.Deserialize<ChromiumDto>(json, SerializerOptions);
|
||||||
if (dto is null)
|
if (dto is null)
|
||||||
{
|
{
|
||||||
_logger.LogWarning("Chromium DTO deserialization failed for {DocumentId}", documentId);
|
_logger.LogWarning("Chromium DTO deserialization failed for {DocumentId}", documentId);
|
||||||
await _documentStore.UpdateStatusAsync(document.Id, DocumentStatuses.Failed, cancellationToken).ConfigureAwait(false);
|
await _documentStore.UpdateStatusAsync(document.Id, DocumentStatuses.Failed, cancellationToken).ConfigureAwait(false);
|
||||||
pendingMappings.Remove(documentId);
|
pendingMappings.Remove(documentId);
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
|
|
||||||
var recordedAt = _timeProvider.GetUtcNow();
|
var recordedAt = _timeProvider.GetUtcNow();
|
||||||
var (advisory, flag) = ChromiumMapper.Map(dto, SourceName, recordedAt);
|
var (advisory, flag) = ChromiumMapper.Map(dto, SourceName, recordedAt);
|
||||||
|
|
||||||
await _advisoryStore.UpsertAsync(advisory, cancellationToken).ConfigureAwait(false);
|
await _advisoryStore.UpsertAsync(advisory, cancellationToken).ConfigureAwait(false);
|
||||||
await _psirtFlagStore.UpsertAsync(flag, cancellationToken).ConfigureAwait(false);
|
await _psirtFlagStore.UpsertAsync(flag, cancellationToken).ConfigureAwait(false);
|
||||||
await _documentStore.UpdateStatusAsync(document.Id, DocumentStatuses.Mapped, cancellationToken).ConfigureAwait(false);
|
await _documentStore.UpdateStatusAsync(document.Id, DocumentStatuses.Mapped, cancellationToken).ConfigureAwait(false);
|
||||||
_diagnostics.MapSuccess();
|
_diagnostics.MapSuccess();
|
||||||
|
|
||||||
pendingMappings.Remove(documentId);
|
pendingMappings.Remove(documentId);
|
||||||
}
|
}
|
||||||
|
|
||||||
var updatedCursor = cursor.WithPendingMappings(pendingMappings);
|
var updatedCursor = cursor.WithPendingMappings(pendingMappings);
|
||||||
await UpdateCursorAsync(updatedCursor, cancellationToken).ConfigureAwait(false);
|
await UpdateCursorAsync(updatedCursor, cancellationToken).ConfigureAwait(false);
|
||||||
}
|
}
|
||||||
|
|
||||||
private async Task<ChromiumCursor> GetCursorAsync(CancellationToken cancellationToken)
|
private async Task<ChromiumCursor> GetCursorAsync(CancellationToken cancellationToken)
|
||||||
{
|
{
|
||||||
var record = await _stateRepository.TryGetAsync(SourceName, cancellationToken).ConfigureAwait(false);
|
var record = await _stateRepository.TryGetAsync(SourceName, cancellationToken).ConfigureAwait(false);
|
||||||
return ChromiumCursor.FromBsonDocument(record?.Cursor);
|
return ChromiumCursor.FromBsonDocument(record?.Cursor);
|
||||||
}
|
}
|
||||||
|
|
||||||
private async Task UpdateCursorAsync(ChromiumCursor cursor, CancellationToken cancellationToken)
|
private async Task UpdateCursorAsync(ChromiumCursor cursor, CancellationToken cancellationToken)
|
||||||
{
|
{
|
||||||
var completedAt = _timeProvider.GetUtcNow();
|
var completedAt = _timeProvider.GetUtcNow();
|
||||||
await _stateRepository.UpdateCursorAsync(SourceName, cursor.ToBsonDocument(), completedAt, cancellationToken).ConfigureAwait(false);
|
await _stateRepository.UpdateCursorAsync(SourceName, cursor.ToBsonDocument(), completedAt, cancellationToken).ConfigureAwait(false);
|
||||||
}
|
}
|
||||||
|
|
||||||
private (DateTimeOffset start, DateTimeOffset end) CalculateWindow(ChromiumCursor cursor, DateTimeOffset now)
|
private (DateTimeOffset start, DateTimeOffset end) CalculateWindow(ChromiumCursor cursor, DateTimeOffset now)
|
||||||
{
|
{
|
||||||
var lastPublished = cursor.LastPublished ?? now - _options.InitialBackfill;
|
var lastPublished = cursor.LastPublished ?? now - _options.InitialBackfill;
|
||||||
var start = lastPublished - _options.WindowOverlap;
|
var start = lastPublished - _options.WindowOverlap;
|
||||||
var backfill = now - _options.InitialBackfill;
|
var backfill = now - _options.InitialBackfill;
|
||||||
if (start < backfill)
|
if (start < backfill)
|
||||||
{
|
{
|
||||||
start = backfill;
|
start = backfill;
|
||||||
}
|
}
|
||||||
|
|
||||||
var end = now;
|
var end = now;
|
||||||
if (end <= start)
|
if (end <= start)
|
||||||
{
|
{
|
||||||
end = start.AddHours(1);
|
end = start.AddHours(1);
|
||||||
}
|
}
|
||||||
|
|
||||||
return (start, end);
|
return (start, end);
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -163,7 +163,7 @@ public sealed class CiscoConnector : IFeedConnector
|
|||||||
BuildMetadata(advisory),
|
BuildMetadata(advisory),
|
||||||
Etag: null,
|
Etag: null,
|
||||||
LastModified: advisory.LastUpdated ?? advisory.FirstPublished ?? now,
|
LastModified: advisory.LastUpdated ?? advisory.FirstPublished ?? now,
|
||||||
GridFsId: gridFsId,
|
PayloadId: gridFsId,
|
||||||
ExpiresAt: null);
|
ExpiresAt: null);
|
||||||
|
|
||||||
var upserted = await _documentStore.UpsertAsync(record, cancellationToken).ConfigureAwait(false);
|
var upserted = await _documentStore.UpsertAsync(record, cancellationToken).ConfigureAwait(false);
|
||||||
@@ -259,7 +259,7 @@ public sealed class CiscoConnector : IFeedConnector
|
|||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
|
|
||||||
if (!document.GridFsId.HasValue)
|
if (!document.PayloadId.HasValue)
|
||||||
{
|
{
|
||||||
_diagnostics.ParseFailure();
|
_diagnostics.ParseFailure();
|
||||||
_logger.LogWarning("Cisco document {DocumentId} missing GridFS payload", documentId);
|
_logger.LogWarning("Cisco document {DocumentId} missing GridFS payload", documentId);
|
||||||
@@ -273,7 +273,7 @@ public sealed class CiscoConnector : IFeedConnector
|
|||||||
byte[] payload;
|
byte[] payload;
|
||||||
try
|
try
|
||||||
{
|
{
|
||||||
payload = await _rawDocumentStorage.DownloadAsync(document.GridFsId.Value, cancellationToken).ConfigureAwait(false);
|
payload = await _rawDocumentStorage.DownloadAsync(document.PayloadId.Value, cancellationToken).ConfigureAwait(false);
|
||||||
}
|
}
|
||||||
catch (Exception ex)
|
catch (Exception ex)
|
||||||
{
|
{
|
||||||
|
|||||||
@@ -133,7 +133,7 @@ public sealed class MsrcConnector : IFeedConnector
|
|||||||
}
|
}
|
||||||
|
|
||||||
_diagnostics.DetailFetchAttempt();
|
_diagnostics.DetailFetchAttempt();
|
||||||
if (existing?.GridFsId is { } oldGridId)
|
if (existing?.PayloadId is { } oldGridId)
|
||||||
{
|
{
|
||||||
await _rawDocumentStorage.DeleteAsync(oldGridId, cancellationToken).ConfigureAwait(false);
|
await _rawDocumentStorage.DeleteAsync(oldGridId, cancellationToken).ConfigureAwait(false);
|
||||||
}
|
}
|
||||||
@@ -238,7 +238,7 @@ public sealed class MsrcConnector : IFeedConnector
|
|||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
|
|
||||||
if (!document.GridFsId.HasValue)
|
if (!document.PayloadId.HasValue)
|
||||||
{
|
{
|
||||||
await _documentStore.UpdateStatusAsync(document.Id, DocumentStatuses.Failed, cancellationToken).ConfigureAwait(false);
|
await _documentStore.UpdateStatusAsync(document.Id, DocumentStatuses.Failed, cancellationToken).ConfigureAwait(false);
|
||||||
remainingDocuments.Remove(documentId);
|
remainingDocuments.Remove(documentId);
|
||||||
@@ -250,7 +250,7 @@ public sealed class MsrcConnector : IFeedConnector
|
|||||||
byte[] payload;
|
byte[] payload;
|
||||||
try
|
try
|
||||||
{
|
{
|
||||||
payload = await _rawDocumentStorage.DownloadAsync(document.GridFsId.Value, cancellationToken).ConfigureAwait(false);
|
payload = await _rawDocumentStorage.DownloadAsync(document.PayloadId.Value, cancellationToken).ConfigureAwait(false);
|
||||||
}
|
}
|
||||||
catch (Exception ex)
|
catch (Exception ex)
|
||||||
{
|
{
|
||||||
|
|||||||
@@ -1,366 +1,366 @@
|
|||||||
using System;
|
using System;
|
||||||
using System.Collections.Generic;
|
using System.Collections.Generic;
|
||||||
using System.Linq;
|
using System.Linq;
|
||||||
using System.Text.Json;
|
using System.Text.Json;
|
||||||
using System.Threading;
|
using System.Threading;
|
||||||
using System.Threading.Tasks;
|
using System.Threading.Tasks;
|
||||||
using Microsoft.Extensions.Logging;
|
using Microsoft.Extensions.Logging;
|
||||||
using Microsoft.Extensions.Options;
|
using Microsoft.Extensions.Options;
|
||||||
using MongoDB.Bson;
|
using MongoDB.Bson;
|
||||||
using StellaOps.Concelier.Connector.Common;
|
using StellaOps.Concelier.Connector.Common;
|
||||||
using StellaOps.Concelier.Connector.Common.Fetch;
|
using StellaOps.Concelier.Connector.Common.Fetch;
|
||||||
using StellaOps.Concelier.Connector.Vndr.Oracle.Configuration;
|
using StellaOps.Concelier.Connector.Vndr.Oracle.Configuration;
|
||||||
using StellaOps.Concelier.Connector.Vndr.Oracle.Internal;
|
using StellaOps.Concelier.Connector.Vndr.Oracle.Internal;
|
||||||
using StellaOps.Concelier.Storage.Mongo;
|
using StellaOps.Concelier.Storage.Mongo;
|
||||||
using StellaOps.Concelier.Storage.Mongo.Advisories;
|
using StellaOps.Concelier.Storage.Mongo.Advisories;
|
||||||
using StellaOps.Concelier.Storage.Mongo.Documents;
|
using StellaOps.Concelier.Storage.Mongo.Documents;
|
||||||
using StellaOps.Concelier.Storage.Mongo.Dtos;
|
using StellaOps.Concelier.Storage.Mongo.Dtos;
|
||||||
using StellaOps.Concelier.Storage.Mongo.PsirtFlags;
|
using StellaOps.Concelier.Storage.Mongo.PsirtFlags;
|
||||||
using StellaOps.Plugin;
|
using StellaOps.Plugin;
|
||||||
|
|
||||||
namespace StellaOps.Concelier.Connector.Vndr.Oracle;
|
namespace StellaOps.Concelier.Connector.Vndr.Oracle;
|
||||||
|
|
||||||
public sealed class OracleConnector : IFeedConnector
|
public sealed class OracleConnector : IFeedConnector
|
||||||
{
|
{
|
||||||
private static readonly JsonSerializerOptions SerializerOptions = new()
|
private static readonly JsonSerializerOptions SerializerOptions = new()
|
||||||
{
|
{
|
||||||
PropertyNamingPolicy = JsonNamingPolicy.CamelCase,
|
PropertyNamingPolicy = JsonNamingPolicy.CamelCase,
|
||||||
DefaultIgnoreCondition = System.Text.Json.Serialization.JsonIgnoreCondition.WhenWritingNull,
|
DefaultIgnoreCondition = System.Text.Json.Serialization.JsonIgnoreCondition.WhenWritingNull,
|
||||||
};
|
};
|
||||||
|
|
||||||
private readonly SourceFetchService _fetchService;
|
private readonly SourceFetchService _fetchService;
|
||||||
private readonly RawDocumentStorage _rawDocumentStorage;
|
private readonly RawDocumentStorage _rawDocumentStorage;
|
||||||
private readonly IDocumentStore _documentStore;
|
private readonly IDocumentStore _documentStore;
|
||||||
private readonly IDtoStore _dtoStore;
|
private readonly IDtoStore _dtoStore;
|
||||||
private readonly IAdvisoryStore _advisoryStore;
|
private readonly IAdvisoryStore _advisoryStore;
|
||||||
private readonly IPsirtFlagStore _psirtFlagStore;
|
private readonly IPsirtFlagStore _psirtFlagStore;
|
||||||
private readonly ISourceStateRepository _stateRepository;
|
private readonly ISourceStateRepository _stateRepository;
|
||||||
private readonly OracleCalendarFetcher _calendarFetcher;
|
private readonly OracleCalendarFetcher _calendarFetcher;
|
||||||
private readonly OracleOptions _options;
|
private readonly OracleOptions _options;
|
||||||
private readonly TimeProvider _timeProvider;
|
private readonly TimeProvider _timeProvider;
|
||||||
private readonly ILogger<OracleConnector> _logger;
|
private readonly ILogger<OracleConnector> _logger;
|
||||||
|
|
||||||
public OracleConnector(
|
public OracleConnector(
|
||||||
SourceFetchService fetchService,
|
SourceFetchService fetchService,
|
||||||
RawDocumentStorage rawDocumentStorage,
|
RawDocumentStorage rawDocumentStorage,
|
||||||
IDocumentStore documentStore,
|
IDocumentStore documentStore,
|
||||||
IDtoStore dtoStore,
|
IDtoStore dtoStore,
|
||||||
IAdvisoryStore advisoryStore,
|
IAdvisoryStore advisoryStore,
|
||||||
IPsirtFlagStore psirtFlagStore,
|
IPsirtFlagStore psirtFlagStore,
|
||||||
ISourceStateRepository stateRepository,
|
ISourceStateRepository stateRepository,
|
||||||
OracleCalendarFetcher calendarFetcher,
|
OracleCalendarFetcher calendarFetcher,
|
||||||
IOptions<OracleOptions> options,
|
IOptions<OracleOptions> options,
|
||||||
TimeProvider? timeProvider,
|
TimeProvider? timeProvider,
|
||||||
ILogger<OracleConnector> logger)
|
ILogger<OracleConnector> logger)
|
||||||
{
|
{
|
||||||
_fetchService = fetchService ?? throw new ArgumentNullException(nameof(fetchService));
|
_fetchService = fetchService ?? throw new ArgumentNullException(nameof(fetchService));
|
||||||
_rawDocumentStorage = rawDocumentStorage ?? throw new ArgumentNullException(nameof(rawDocumentStorage));
|
_rawDocumentStorage = rawDocumentStorage ?? throw new ArgumentNullException(nameof(rawDocumentStorage));
|
||||||
_documentStore = documentStore ?? throw new ArgumentNullException(nameof(documentStore));
|
_documentStore = documentStore ?? throw new ArgumentNullException(nameof(documentStore));
|
||||||
_dtoStore = dtoStore ?? throw new ArgumentNullException(nameof(dtoStore));
|
_dtoStore = dtoStore ?? throw new ArgumentNullException(nameof(dtoStore));
|
||||||
_advisoryStore = advisoryStore ?? throw new ArgumentNullException(nameof(advisoryStore));
|
_advisoryStore = advisoryStore ?? throw new ArgumentNullException(nameof(advisoryStore));
|
||||||
_psirtFlagStore = psirtFlagStore ?? throw new ArgumentNullException(nameof(psirtFlagStore));
|
_psirtFlagStore = psirtFlagStore ?? throw new ArgumentNullException(nameof(psirtFlagStore));
|
||||||
_stateRepository = stateRepository ?? throw new ArgumentNullException(nameof(stateRepository));
|
_stateRepository = stateRepository ?? throw new ArgumentNullException(nameof(stateRepository));
|
||||||
_calendarFetcher = calendarFetcher ?? throw new ArgumentNullException(nameof(calendarFetcher));
|
_calendarFetcher = calendarFetcher ?? throw new ArgumentNullException(nameof(calendarFetcher));
|
||||||
_options = (options ?? throw new ArgumentNullException(nameof(options))).Value ?? throw new ArgumentNullException(nameof(options));
|
_options = (options ?? throw new ArgumentNullException(nameof(options))).Value ?? throw new ArgumentNullException(nameof(options));
|
||||||
_options.Validate();
|
_options.Validate();
|
||||||
_timeProvider = timeProvider ?? TimeProvider.System;
|
_timeProvider = timeProvider ?? TimeProvider.System;
|
||||||
_logger = logger ?? throw new ArgumentNullException(nameof(logger));
|
_logger = logger ?? throw new ArgumentNullException(nameof(logger));
|
||||||
}
|
}
|
||||||
|
|
||||||
public string SourceName => VndrOracleConnectorPlugin.SourceName;
|
public string SourceName => VndrOracleConnectorPlugin.SourceName;
|
||||||
|
|
||||||
public async Task FetchAsync(IServiceProvider services, CancellationToken cancellationToken)
|
public async Task FetchAsync(IServiceProvider services, CancellationToken cancellationToken)
|
||||||
{
|
{
|
||||||
var cursor = await GetCursorAsync(cancellationToken).ConfigureAwait(false);
|
var cursor = await GetCursorAsync(cancellationToken).ConfigureAwait(false);
|
||||||
var pendingDocuments = cursor.PendingDocuments.ToList();
|
var pendingDocuments = cursor.PendingDocuments.ToList();
|
||||||
var pendingMappings = cursor.PendingMappings.ToList();
|
var pendingMappings = cursor.PendingMappings.ToList();
|
||||||
var fetchCache = new Dictionary<string, OracleFetchCacheEntry>(cursor.FetchCache, StringComparer.OrdinalIgnoreCase);
|
var fetchCache = new Dictionary<string, OracleFetchCacheEntry>(cursor.FetchCache, StringComparer.OrdinalIgnoreCase);
|
||||||
var touchedResources = new HashSet<string>(StringComparer.OrdinalIgnoreCase);
|
var touchedResources = new HashSet<string>(StringComparer.OrdinalIgnoreCase);
|
||||||
var now = _timeProvider.GetUtcNow();
|
var now = _timeProvider.GetUtcNow();
|
||||||
|
|
||||||
var advisoryUris = await ResolveAdvisoryUrisAsync(cancellationToken).ConfigureAwait(false);
|
var advisoryUris = await ResolveAdvisoryUrisAsync(cancellationToken).ConfigureAwait(false);
|
||||||
|
|
||||||
foreach (var uri in advisoryUris)
|
foreach (var uri in advisoryUris)
|
||||||
{
|
{
|
||||||
cancellationToken.ThrowIfCancellationRequested();
|
cancellationToken.ThrowIfCancellationRequested();
|
||||||
|
|
||||||
try
|
try
|
||||||
{
|
{
|
||||||
var cacheKey = uri.AbsoluteUri;
|
var cacheKey = uri.AbsoluteUri;
|
||||||
touchedResources.Add(cacheKey);
|
touchedResources.Add(cacheKey);
|
||||||
|
|
||||||
var advisoryId = DeriveAdvisoryId(uri);
|
var advisoryId = DeriveAdvisoryId(uri);
|
||||||
var title = advisoryId.Replace('-', ' ');
|
var title = advisoryId.Replace('-', ' ');
|
||||||
var published = now;
|
var published = now;
|
||||||
|
|
||||||
var metadata = OracleDocumentMetadata.CreateMetadata(advisoryId, title, published);
|
var metadata = OracleDocumentMetadata.CreateMetadata(advisoryId, title, published);
|
||||||
var existing = await _documentStore.FindBySourceAndUriAsync(SourceName, uri.ToString(), cancellationToken).ConfigureAwait(false);
|
var existing = await _documentStore.FindBySourceAndUriAsync(SourceName, uri.ToString(), cancellationToken).ConfigureAwait(false);
|
||||||
|
|
||||||
var request = new SourceFetchRequest(OracleOptions.HttpClientName, SourceName, uri)
|
var request = new SourceFetchRequest(OracleOptions.HttpClientName, SourceName, uri)
|
||||||
{
|
{
|
||||||
Metadata = metadata,
|
Metadata = metadata,
|
||||||
ETag = existing?.Etag,
|
ETag = existing?.Etag,
|
||||||
LastModified = existing?.LastModified,
|
LastModified = existing?.LastModified,
|
||||||
AcceptHeaders = new[] { "text/html", "application/xhtml+xml", "text/plain;q=0.5" },
|
AcceptHeaders = new[] { "text/html", "application/xhtml+xml", "text/plain;q=0.5" },
|
||||||
};
|
};
|
||||||
|
|
||||||
var result = await _fetchService.FetchAsync(request, cancellationToken).ConfigureAwait(false);
|
var result = await _fetchService.FetchAsync(request, cancellationToken).ConfigureAwait(false);
|
||||||
if (!result.IsSuccess || result.Document is null)
|
if (!result.IsSuccess || result.Document is null)
|
||||||
{
|
{
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
|
|
||||||
var cacheEntry = OracleFetchCacheEntry.FromDocument(result.Document);
|
var cacheEntry = OracleFetchCacheEntry.FromDocument(result.Document);
|
||||||
if (existing is not null
|
if (existing is not null
|
||||||
&& string.Equals(existing.Status, DocumentStatuses.Mapped, StringComparison.Ordinal)
|
&& string.Equals(existing.Status, DocumentStatuses.Mapped, StringComparison.Ordinal)
|
||||||
&& cursor.TryGetFetchCache(cacheKey, out var cached)
|
&& cursor.TryGetFetchCache(cacheKey, out var cached)
|
||||||
&& cached.Matches(result.Document))
|
&& cached.Matches(result.Document))
|
||||||
{
|
{
|
||||||
_logger.LogDebug("Oracle advisory {AdvisoryId} unchanged; skipping parse/map", advisoryId);
|
_logger.LogDebug("Oracle advisory {AdvisoryId} unchanged; skipping parse/map", advisoryId);
|
||||||
await _documentStore.UpdateStatusAsync(result.Document.Id, existing.Status, cancellationToken).ConfigureAwait(false);
|
await _documentStore.UpdateStatusAsync(result.Document.Id, existing.Status, cancellationToken).ConfigureAwait(false);
|
||||||
pendingDocuments.Remove(result.Document.Id);
|
pendingDocuments.Remove(result.Document.Id);
|
||||||
pendingMappings.Remove(result.Document.Id);
|
pendingMappings.Remove(result.Document.Id);
|
||||||
fetchCache[cacheKey] = cacheEntry;
|
fetchCache[cacheKey] = cacheEntry;
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
|
|
||||||
fetchCache[cacheKey] = cacheEntry;
|
fetchCache[cacheKey] = cacheEntry;
|
||||||
|
|
||||||
if (!pendingDocuments.Contains(result.Document.Id))
|
if (!pendingDocuments.Contains(result.Document.Id))
|
||||||
{
|
{
|
||||||
pendingDocuments.Add(result.Document.Id);
|
pendingDocuments.Add(result.Document.Id);
|
||||||
}
|
}
|
||||||
|
|
||||||
if (_options.RequestDelay > TimeSpan.Zero)
|
if (_options.RequestDelay > TimeSpan.Zero)
|
||||||
{
|
{
|
||||||
await Task.Delay(_options.RequestDelay, cancellationToken).ConfigureAwait(false);
|
await Task.Delay(_options.RequestDelay, cancellationToken).ConfigureAwait(false);
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
catch (Exception ex)
|
catch (Exception ex)
|
||||||
{
|
{
|
||||||
_logger.LogError(ex, "Oracle fetch failed for {Uri}", uri);
|
_logger.LogError(ex, "Oracle fetch failed for {Uri}", uri);
|
||||||
await _stateRepository.MarkFailureAsync(SourceName, _timeProvider.GetUtcNow(), TimeSpan.FromMinutes(10), ex.Message, cancellationToken).ConfigureAwait(false);
|
await _stateRepository.MarkFailureAsync(SourceName, _timeProvider.GetUtcNow(), TimeSpan.FromMinutes(10), ex.Message, cancellationToken).ConfigureAwait(false);
|
||||||
throw;
|
throw;
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
if (fetchCache.Count > 0 && touchedResources.Count > 0)
|
if (fetchCache.Count > 0 && touchedResources.Count > 0)
|
||||||
{
|
{
|
||||||
var stale = fetchCache.Keys.Where(key => !touchedResources.Contains(key)).ToArray();
|
var stale = fetchCache.Keys.Where(key => !touchedResources.Contains(key)).ToArray();
|
||||||
foreach (var key in stale)
|
foreach (var key in stale)
|
||||||
{
|
{
|
||||||
fetchCache.Remove(key);
|
fetchCache.Remove(key);
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
var updatedCursor = cursor
|
var updatedCursor = cursor
|
||||||
.WithPendingDocuments(pendingDocuments)
|
.WithPendingDocuments(pendingDocuments)
|
||||||
.WithPendingMappings(pendingMappings)
|
.WithPendingMappings(pendingMappings)
|
||||||
.WithFetchCache(fetchCache)
|
.WithFetchCache(fetchCache)
|
||||||
.WithLastProcessed(now);
|
.WithLastProcessed(now);
|
||||||
|
|
||||||
await UpdateCursorAsync(updatedCursor, cancellationToken).ConfigureAwait(false);
|
await UpdateCursorAsync(updatedCursor, cancellationToken).ConfigureAwait(false);
|
||||||
}
|
}
|
||||||
|
|
||||||
public async Task ParseAsync(IServiceProvider services, CancellationToken cancellationToken)
|
public async Task ParseAsync(IServiceProvider services, CancellationToken cancellationToken)
|
||||||
{
|
{
|
||||||
var cursor = await GetCursorAsync(cancellationToken).ConfigureAwait(false);
|
var cursor = await GetCursorAsync(cancellationToken).ConfigureAwait(false);
|
||||||
if (cursor.PendingDocuments.Count == 0)
|
if (cursor.PendingDocuments.Count == 0)
|
||||||
{
|
{
|
||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
|
|
||||||
var pendingDocuments = cursor.PendingDocuments.ToList();
|
var pendingDocuments = cursor.PendingDocuments.ToList();
|
||||||
var pendingMappings = cursor.PendingMappings.ToList();
|
var pendingMappings = cursor.PendingMappings.ToList();
|
||||||
|
|
||||||
foreach (var documentId in cursor.PendingDocuments)
|
foreach (var documentId in cursor.PendingDocuments)
|
||||||
{
|
{
|
||||||
cancellationToken.ThrowIfCancellationRequested();
|
cancellationToken.ThrowIfCancellationRequested();
|
||||||
|
|
||||||
var document = await _documentStore.FindAsync(documentId, cancellationToken).ConfigureAwait(false);
|
var document = await _documentStore.FindAsync(documentId, cancellationToken).ConfigureAwait(false);
|
||||||
if (document is null)
|
if (document is null)
|
||||||
{
|
{
|
||||||
pendingDocuments.Remove(documentId);
|
pendingDocuments.Remove(documentId);
|
||||||
pendingMappings.Remove(documentId);
|
pendingMappings.Remove(documentId);
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
|
|
||||||
if (!document.GridFsId.HasValue)
|
if (!document.PayloadId.HasValue)
|
||||||
{
|
{
|
||||||
_logger.LogWarning("Oracle document {DocumentId} missing GridFS payload", document.Id);
|
_logger.LogWarning("Oracle document {DocumentId} missing GridFS payload", document.Id);
|
||||||
await _documentStore.UpdateStatusAsync(document.Id, DocumentStatuses.Failed, cancellationToken).ConfigureAwait(false);
|
await _documentStore.UpdateStatusAsync(document.Id, DocumentStatuses.Failed, cancellationToken).ConfigureAwait(false);
|
||||||
pendingDocuments.Remove(documentId);
|
pendingDocuments.Remove(documentId);
|
||||||
pendingMappings.Remove(documentId);
|
pendingMappings.Remove(documentId);
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
|
|
||||||
OracleDto dto;
|
OracleDto dto;
|
||||||
try
|
try
|
||||||
{
|
{
|
||||||
var metadata = OracleDocumentMetadata.FromDocument(document);
|
var metadata = OracleDocumentMetadata.FromDocument(document);
|
||||||
var content = await _rawDocumentStorage.DownloadAsync(document.GridFsId.Value, cancellationToken).ConfigureAwait(false);
|
var content = await _rawDocumentStorage.DownloadAsync(document.PayloadId.Value, cancellationToken).ConfigureAwait(false);
|
||||||
var html = System.Text.Encoding.UTF8.GetString(content);
|
var html = System.Text.Encoding.UTF8.GetString(content);
|
||||||
dto = OracleParser.Parse(html, metadata);
|
dto = OracleParser.Parse(html, metadata);
|
||||||
}
|
}
|
||||||
catch (Exception ex)
|
catch (Exception ex)
|
||||||
{
|
{
|
||||||
_logger.LogError(ex, "Oracle parse failed for document {DocumentId}", document.Id);
|
_logger.LogError(ex, "Oracle parse failed for document {DocumentId}", document.Id);
|
||||||
await _documentStore.UpdateStatusAsync(document.Id, DocumentStatuses.Failed, cancellationToken).ConfigureAwait(false);
|
await _documentStore.UpdateStatusAsync(document.Id, DocumentStatuses.Failed, cancellationToken).ConfigureAwait(false);
|
||||||
pendingDocuments.Remove(documentId);
|
pendingDocuments.Remove(documentId);
|
||||||
pendingMappings.Remove(documentId);
|
pendingMappings.Remove(documentId);
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
|
|
||||||
if (!OracleDtoValidator.TryNormalize(dto, out var normalized, out var validationError))
|
if (!OracleDtoValidator.TryNormalize(dto, out var normalized, out var validationError))
|
||||||
{
|
{
|
||||||
_logger.LogWarning("Oracle validation failed for document {DocumentId}: {Reason}", document.Id, validationError ?? "unknown");
|
_logger.LogWarning("Oracle validation failed for document {DocumentId}: {Reason}", document.Id, validationError ?? "unknown");
|
||||||
await _documentStore.UpdateStatusAsync(document.Id, DocumentStatuses.Failed, cancellationToken).ConfigureAwait(false);
|
await _documentStore.UpdateStatusAsync(document.Id, DocumentStatuses.Failed, cancellationToken).ConfigureAwait(false);
|
||||||
pendingDocuments.Remove(documentId);
|
pendingDocuments.Remove(documentId);
|
||||||
pendingMappings.Remove(documentId);
|
pendingMappings.Remove(documentId);
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
|
|
||||||
dto = normalized;
|
dto = normalized;
|
||||||
|
|
||||||
var json = JsonSerializer.Serialize(dto, SerializerOptions);
|
var json = JsonSerializer.Serialize(dto, SerializerOptions);
|
||||||
var payload = BsonDocument.Parse(json);
|
var payload = BsonDocument.Parse(json);
|
||||||
var validatedAt = _timeProvider.GetUtcNow();
|
var validatedAt = _timeProvider.GetUtcNow();
|
||||||
|
|
||||||
var existingDto = await _dtoStore.FindByDocumentIdAsync(document.Id, cancellationToken).ConfigureAwait(false);
|
var existingDto = await _dtoStore.FindByDocumentIdAsync(document.Id, cancellationToken).ConfigureAwait(false);
|
||||||
var dtoRecord = existingDto is null
|
var dtoRecord = existingDto is null
|
||||||
? new DtoRecord(Guid.NewGuid(), document.Id, SourceName, "oracle.advisory.v1", payload, validatedAt)
|
? new DtoRecord(Guid.NewGuid(), document.Id, SourceName, "oracle.advisory.v1", payload, validatedAt)
|
||||||
: existingDto with
|
: existingDto with
|
||||||
{
|
{
|
||||||
Payload = payload,
|
Payload = payload,
|
||||||
SchemaVersion = "oracle.advisory.v1",
|
SchemaVersion = "oracle.advisory.v1",
|
||||||
ValidatedAt = validatedAt,
|
ValidatedAt = validatedAt,
|
||||||
};
|
};
|
||||||
|
|
||||||
await _dtoStore.UpsertAsync(dtoRecord, cancellationToken).ConfigureAwait(false);
|
await _dtoStore.UpsertAsync(dtoRecord, cancellationToken).ConfigureAwait(false);
|
||||||
await _documentStore.UpdateStatusAsync(document.Id, DocumentStatuses.PendingMap, cancellationToken).ConfigureAwait(false);
|
await _documentStore.UpdateStatusAsync(document.Id, DocumentStatuses.PendingMap, cancellationToken).ConfigureAwait(false);
|
||||||
|
|
||||||
pendingDocuments.Remove(documentId);
|
pendingDocuments.Remove(documentId);
|
||||||
if (!pendingMappings.Contains(documentId))
|
if (!pendingMappings.Contains(documentId))
|
||||||
{
|
{
|
||||||
pendingMappings.Add(documentId);
|
pendingMappings.Add(documentId);
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
var updatedCursor = cursor
|
var updatedCursor = cursor
|
||||||
.WithPendingDocuments(pendingDocuments)
|
.WithPendingDocuments(pendingDocuments)
|
||||||
.WithPendingMappings(pendingMappings);
|
.WithPendingMappings(pendingMappings);
|
||||||
|
|
||||||
await UpdateCursorAsync(updatedCursor, cancellationToken).ConfigureAwait(false);
|
await UpdateCursorAsync(updatedCursor, cancellationToken).ConfigureAwait(false);
|
||||||
}
|
}
|
||||||
|
|
||||||
public async Task MapAsync(IServiceProvider services, CancellationToken cancellationToken)
|
public async Task MapAsync(IServiceProvider services, CancellationToken cancellationToken)
|
||||||
{
|
{
|
||||||
var cursor = await GetCursorAsync(cancellationToken).ConfigureAwait(false);
|
var cursor = await GetCursorAsync(cancellationToken).ConfigureAwait(false);
|
||||||
if (cursor.PendingMappings.Count == 0)
|
if (cursor.PendingMappings.Count == 0)
|
||||||
{
|
{
|
||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
|
|
||||||
var pendingMappings = cursor.PendingMappings.ToList();
|
var pendingMappings = cursor.PendingMappings.ToList();
|
||||||
|
|
||||||
foreach (var documentId in cursor.PendingMappings)
|
foreach (var documentId in cursor.PendingMappings)
|
||||||
{
|
{
|
||||||
cancellationToken.ThrowIfCancellationRequested();
|
cancellationToken.ThrowIfCancellationRequested();
|
||||||
|
|
||||||
var dtoRecord = await _dtoStore.FindByDocumentIdAsync(documentId, cancellationToken).ConfigureAwait(false);
|
var dtoRecord = await _dtoStore.FindByDocumentIdAsync(documentId, cancellationToken).ConfigureAwait(false);
|
||||||
var document = await _documentStore.FindAsync(documentId, cancellationToken).ConfigureAwait(false);
|
var document = await _documentStore.FindAsync(documentId, cancellationToken).ConfigureAwait(false);
|
||||||
|
|
||||||
if (dtoRecord is null || document is null)
|
if (dtoRecord is null || document is null)
|
||||||
{
|
{
|
||||||
pendingMappings.Remove(documentId);
|
pendingMappings.Remove(documentId);
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
|
|
||||||
OracleDto? dto;
|
OracleDto? dto;
|
||||||
try
|
try
|
||||||
{
|
{
|
||||||
var json = dtoRecord.Payload.ToJson();
|
var json = dtoRecord.Payload.ToJson();
|
||||||
dto = JsonSerializer.Deserialize<OracleDto>(json, SerializerOptions);
|
dto = JsonSerializer.Deserialize<OracleDto>(json, SerializerOptions);
|
||||||
}
|
}
|
||||||
catch (Exception ex)
|
catch (Exception ex)
|
||||||
{
|
{
|
||||||
_logger.LogError(ex, "Oracle DTO deserialization failed for document {DocumentId}", documentId);
|
_logger.LogError(ex, "Oracle DTO deserialization failed for document {DocumentId}", documentId);
|
||||||
await _documentStore.UpdateStatusAsync(document.Id, DocumentStatuses.Failed, cancellationToken).ConfigureAwait(false);
|
await _documentStore.UpdateStatusAsync(document.Id, DocumentStatuses.Failed, cancellationToken).ConfigureAwait(false);
|
||||||
pendingMappings.Remove(documentId);
|
pendingMappings.Remove(documentId);
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
|
|
||||||
if (dto is null)
|
if (dto is null)
|
||||||
{
|
{
|
||||||
_logger.LogWarning("Oracle DTO payload deserialized as null for document {DocumentId}", documentId);
|
_logger.LogWarning("Oracle DTO payload deserialized as null for document {DocumentId}", documentId);
|
||||||
await _documentStore.UpdateStatusAsync(document.Id, DocumentStatuses.Failed, cancellationToken).ConfigureAwait(false);
|
await _documentStore.UpdateStatusAsync(document.Id, DocumentStatuses.Failed, cancellationToken).ConfigureAwait(false);
|
||||||
pendingMappings.Remove(documentId);
|
pendingMappings.Remove(documentId);
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
|
|
||||||
var mappedAt = _timeProvider.GetUtcNow();
|
var mappedAt = _timeProvider.GetUtcNow();
|
||||||
var (advisory, flag) = OracleMapper.Map(dto, document, dtoRecord, SourceName, mappedAt);
|
var (advisory, flag) = OracleMapper.Map(dto, document, dtoRecord, SourceName, mappedAt);
|
||||||
await _advisoryStore.UpsertAsync(advisory, cancellationToken).ConfigureAwait(false);
|
await _advisoryStore.UpsertAsync(advisory, cancellationToken).ConfigureAwait(false);
|
||||||
await _psirtFlagStore.UpsertAsync(flag, cancellationToken).ConfigureAwait(false);
|
await _psirtFlagStore.UpsertAsync(flag, cancellationToken).ConfigureAwait(false);
|
||||||
await _documentStore.UpdateStatusAsync(document.Id, DocumentStatuses.Mapped, cancellationToken).ConfigureAwait(false);
|
await _documentStore.UpdateStatusAsync(document.Id, DocumentStatuses.Mapped, cancellationToken).ConfigureAwait(false);
|
||||||
|
|
||||||
pendingMappings.Remove(documentId);
|
pendingMappings.Remove(documentId);
|
||||||
}
|
}
|
||||||
|
|
||||||
var updatedCursor = cursor.WithPendingMappings(pendingMappings);
|
var updatedCursor = cursor.WithPendingMappings(pendingMappings);
|
||||||
await UpdateCursorAsync(updatedCursor, cancellationToken).ConfigureAwait(false);
|
await UpdateCursorAsync(updatedCursor, cancellationToken).ConfigureAwait(false);
|
||||||
}
|
}
|
||||||
|
|
||||||
private async Task<OracleCursor> GetCursorAsync(CancellationToken cancellationToken)
|
private async Task<OracleCursor> GetCursorAsync(CancellationToken cancellationToken)
|
||||||
{
|
{
|
||||||
var record = await _stateRepository.TryGetAsync(SourceName, cancellationToken).ConfigureAwait(false);
|
var record = await _stateRepository.TryGetAsync(SourceName, cancellationToken).ConfigureAwait(false);
|
||||||
return OracleCursor.FromBson(record?.Cursor);
|
return OracleCursor.FromBson(record?.Cursor);
|
||||||
}
|
}
|
||||||
|
|
||||||
private async Task UpdateCursorAsync(OracleCursor cursor, CancellationToken cancellationToken)
|
private async Task UpdateCursorAsync(OracleCursor cursor, CancellationToken cancellationToken)
|
||||||
{
|
{
|
||||||
var completedAt = _timeProvider.GetUtcNow();
|
var completedAt = _timeProvider.GetUtcNow();
|
||||||
await _stateRepository.UpdateCursorAsync(SourceName, cursor.ToBsonDocument(), completedAt, cancellationToken).ConfigureAwait(false);
|
await _stateRepository.UpdateCursorAsync(SourceName, cursor.ToBsonDocument(), completedAt, cancellationToken).ConfigureAwait(false);
|
||||||
}
|
}
|
||||||
|
|
||||||
private async Task<IReadOnlyCollection<Uri>> ResolveAdvisoryUrisAsync(CancellationToken cancellationToken)
|
private async Task<IReadOnlyCollection<Uri>> ResolveAdvisoryUrisAsync(CancellationToken cancellationToken)
|
||||||
{
|
{
|
||||||
var uris = new HashSet<string>(StringComparer.OrdinalIgnoreCase);
|
var uris = new HashSet<string>(StringComparer.OrdinalIgnoreCase);
|
||||||
|
|
||||||
foreach (var uri in _options.AdvisoryUris)
|
foreach (var uri in _options.AdvisoryUris)
|
||||||
{
|
{
|
||||||
if (uri is not null)
|
if (uri is not null)
|
||||||
{
|
{
|
||||||
uris.Add(uri.AbsoluteUri);
|
uris.Add(uri.AbsoluteUri);
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
var calendarUris = await _calendarFetcher.GetAdvisoryUrisAsync(cancellationToken).ConfigureAwait(false);
|
var calendarUris = await _calendarFetcher.GetAdvisoryUrisAsync(cancellationToken).ConfigureAwait(false);
|
||||||
foreach (var uri in calendarUris)
|
foreach (var uri in calendarUris)
|
||||||
{
|
{
|
||||||
uris.Add(uri.AbsoluteUri);
|
uris.Add(uri.AbsoluteUri);
|
||||||
}
|
}
|
||||||
|
|
||||||
return uris
|
return uris
|
||||||
.Select(static value => new Uri(value, UriKind.Absolute))
|
.Select(static value => new Uri(value, UriKind.Absolute))
|
||||||
.OrderBy(static value => value.AbsoluteUri, StringComparer.OrdinalIgnoreCase)
|
.OrderBy(static value => value.AbsoluteUri, StringComparer.OrdinalIgnoreCase)
|
||||||
.ToArray();
|
.ToArray();
|
||||||
}
|
}
|
||||||
|
|
||||||
private static string DeriveAdvisoryId(Uri uri)
|
private static string DeriveAdvisoryId(Uri uri)
|
||||||
{
|
{
|
||||||
var segments = uri.Segments;
|
var segments = uri.Segments;
|
||||||
if (segments.Length == 0)
|
if (segments.Length == 0)
|
||||||
{
|
{
|
||||||
return uri.AbsoluteUri;
|
return uri.AbsoluteUri;
|
||||||
}
|
}
|
||||||
|
|
||||||
var slug = segments[^1].Trim('/');
|
var slug = segments[^1].Trim('/');
|
||||||
if (string.IsNullOrWhiteSpace(slug))
|
if (string.IsNullOrWhiteSpace(slug))
|
||||||
{
|
{
|
||||||
return uri.AbsoluteUri;
|
return uri.AbsoluteUri;
|
||||||
}
|
}
|
||||||
|
|
||||||
return slug.Replace('.', '-');
|
return slug.Replace('.', '-');
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -1,454 +1,454 @@
|
|||||||
using System;
|
using System;
|
||||||
using System.Collections.Generic;
|
using System.Collections.Generic;
|
||||||
using System.Linq;
|
using System.Linq;
|
||||||
using System.Net.Http;
|
using System.Net.Http;
|
||||||
using System.Text.Json;
|
using System.Text.Json;
|
||||||
using System.Threading;
|
using System.Threading;
|
||||||
using System.Threading.Tasks;
|
using System.Threading.Tasks;
|
||||||
using Microsoft.Extensions.Logging;
|
using Microsoft.Extensions.Logging;
|
||||||
using Microsoft.Extensions.Options;
|
using Microsoft.Extensions.Options;
|
||||||
using MongoDB.Bson;
|
using MongoDB.Bson;
|
||||||
using MongoDB.Bson.IO;
|
using MongoDB.Bson.IO;
|
||||||
using StellaOps.Concelier.Models;
|
using StellaOps.Concelier.Models;
|
||||||
using StellaOps.Concelier.Connector.Common;
|
using StellaOps.Concelier.Connector.Common;
|
||||||
using StellaOps.Concelier.Connector.Common.Fetch;
|
using StellaOps.Concelier.Connector.Common.Fetch;
|
||||||
using StellaOps.Concelier.Connector.Vndr.Vmware.Configuration;
|
using StellaOps.Concelier.Connector.Vndr.Vmware.Configuration;
|
||||||
using StellaOps.Concelier.Connector.Vndr.Vmware.Internal;
|
using StellaOps.Concelier.Connector.Vndr.Vmware.Internal;
|
||||||
using StellaOps.Concelier.Storage.Mongo;
|
using StellaOps.Concelier.Storage.Mongo;
|
||||||
using StellaOps.Concelier.Storage.Mongo.Advisories;
|
using StellaOps.Concelier.Storage.Mongo.Advisories;
|
||||||
using StellaOps.Concelier.Storage.Mongo.Documents;
|
using StellaOps.Concelier.Storage.Mongo.Documents;
|
||||||
using StellaOps.Concelier.Storage.Mongo.Dtos;
|
using StellaOps.Concelier.Storage.Mongo.Dtos;
|
||||||
using StellaOps.Concelier.Storage.Mongo.PsirtFlags;
|
using StellaOps.Concelier.Storage.Mongo.PsirtFlags;
|
||||||
using StellaOps.Plugin;
|
using StellaOps.Plugin;
|
||||||
|
|
||||||
namespace StellaOps.Concelier.Connector.Vndr.Vmware;
|
namespace StellaOps.Concelier.Connector.Vndr.Vmware;
|
||||||
|
|
||||||
public sealed class VmwareConnector : IFeedConnector
|
public sealed class VmwareConnector : IFeedConnector
|
||||||
{
|
{
|
||||||
private static readonly JsonSerializerOptions SerializerOptions = new(JsonSerializerDefaults.Web)
|
private static readonly JsonSerializerOptions SerializerOptions = new(JsonSerializerDefaults.Web)
|
||||||
{
|
{
|
||||||
PropertyNameCaseInsensitive = true,
|
PropertyNameCaseInsensitive = true,
|
||||||
DefaultIgnoreCondition = System.Text.Json.Serialization.JsonIgnoreCondition.WhenWritingNull,
|
DefaultIgnoreCondition = System.Text.Json.Serialization.JsonIgnoreCondition.WhenWritingNull,
|
||||||
};
|
};
|
||||||
|
|
||||||
private readonly IHttpClientFactory _httpClientFactory;
|
private readonly IHttpClientFactory _httpClientFactory;
|
||||||
private readonly SourceFetchService _fetchService;
|
private readonly SourceFetchService _fetchService;
|
||||||
private readonly RawDocumentStorage _rawDocumentStorage;
|
private readonly RawDocumentStorage _rawDocumentStorage;
|
||||||
private readonly IDocumentStore _documentStore;
|
private readonly IDocumentStore _documentStore;
|
||||||
private readonly IDtoStore _dtoStore;
|
private readonly IDtoStore _dtoStore;
|
||||||
private readonly IAdvisoryStore _advisoryStore;
|
private readonly IAdvisoryStore _advisoryStore;
|
||||||
private readonly ISourceStateRepository _stateRepository;
|
private readonly ISourceStateRepository _stateRepository;
|
||||||
private readonly IPsirtFlagStore _psirtFlagStore;
|
private readonly IPsirtFlagStore _psirtFlagStore;
|
||||||
private readonly VmwareOptions _options;
|
private readonly VmwareOptions _options;
|
||||||
private readonly TimeProvider _timeProvider;
|
private readonly TimeProvider _timeProvider;
|
||||||
private readonly VmwareDiagnostics _diagnostics;
|
private readonly VmwareDiagnostics _diagnostics;
|
||||||
private readonly ILogger<VmwareConnector> _logger;
|
private readonly ILogger<VmwareConnector> _logger;
|
||||||
|
|
||||||
public VmwareConnector(
|
public VmwareConnector(
|
||||||
IHttpClientFactory httpClientFactory,
|
IHttpClientFactory httpClientFactory,
|
||||||
SourceFetchService fetchService,
|
SourceFetchService fetchService,
|
||||||
RawDocumentStorage rawDocumentStorage,
|
RawDocumentStorage rawDocumentStorage,
|
||||||
IDocumentStore documentStore,
|
IDocumentStore documentStore,
|
||||||
IDtoStore dtoStore,
|
IDtoStore dtoStore,
|
||||||
IAdvisoryStore advisoryStore,
|
IAdvisoryStore advisoryStore,
|
||||||
ISourceStateRepository stateRepository,
|
ISourceStateRepository stateRepository,
|
||||||
IPsirtFlagStore psirtFlagStore,
|
IPsirtFlagStore psirtFlagStore,
|
||||||
IOptions<VmwareOptions> options,
|
IOptions<VmwareOptions> options,
|
||||||
TimeProvider? timeProvider,
|
TimeProvider? timeProvider,
|
||||||
VmwareDiagnostics diagnostics,
|
VmwareDiagnostics diagnostics,
|
||||||
ILogger<VmwareConnector> logger)
|
ILogger<VmwareConnector> logger)
|
||||||
{
|
{
|
||||||
_httpClientFactory = httpClientFactory ?? throw new ArgumentNullException(nameof(httpClientFactory));
|
_httpClientFactory = httpClientFactory ?? throw new ArgumentNullException(nameof(httpClientFactory));
|
||||||
_fetchService = fetchService ?? throw new ArgumentNullException(nameof(fetchService));
|
_fetchService = fetchService ?? throw new ArgumentNullException(nameof(fetchService));
|
||||||
_rawDocumentStorage = rawDocumentStorage ?? throw new ArgumentNullException(nameof(rawDocumentStorage));
|
_rawDocumentStorage = rawDocumentStorage ?? throw new ArgumentNullException(nameof(rawDocumentStorage));
|
||||||
_documentStore = documentStore ?? throw new ArgumentNullException(nameof(documentStore));
|
_documentStore = documentStore ?? throw new ArgumentNullException(nameof(documentStore));
|
||||||
_dtoStore = dtoStore ?? throw new ArgumentNullException(nameof(dtoStore));
|
_dtoStore = dtoStore ?? throw new ArgumentNullException(nameof(dtoStore));
|
||||||
_advisoryStore = advisoryStore ?? throw new ArgumentNullException(nameof(advisoryStore));
|
_advisoryStore = advisoryStore ?? throw new ArgumentNullException(nameof(advisoryStore));
|
||||||
_stateRepository = stateRepository ?? throw new ArgumentNullException(nameof(stateRepository));
|
_stateRepository = stateRepository ?? throw new ArgumentNullException(nameof(stateRepository));
|
||||||
_psirtFlagStore = psirtFlagStore ?? throw new ArgumentNullException(nameof(psirtFlagStore));
|
_psirtFlagStore = psirtFlagStore ?? throw new ArgumentNullException(nameof(psirtFlagStore));
|
||||||
_options = (options ?? throw new ArgumentNullException(nameof(options))).Value ?? throw new ArgumentNullException(nameof(options));
|
_options = (options ?? throw new ArgumentNullException(nameof(options))).Value ?? throw new ArgumentNullException(nameof(options));
|
||||||
_options.Validate();
|
_options.Validate();
|
||||||
_timeProvider = timeProvider ?? TimeProvider.System;
|
_timeProvider = timeProvider ?? TimeProvider.System;
|
||||||
_diagnostics = diagnostics ?? throw new ArgumentNullException(nameof(diagnostics));
|
_diagnostics = diagnostics ?? throw new ArgumentNullException(nameof(diagnostics));
|
||||||
_logger = logger ?? throw new ArgumentNullException(nameof(logger));
|
_logger = logger ?? throw new ArgumentNullException(nameof(logger));
|
||||||
}
|
}
|
||||||
|
|
||||||
public string SourceName => VmwareConnectorPlugin.SourceName;
|
public string SourceName => VmwareConnectorPlugin.SourceName;
|
||||||
|
|
||||||
public async Task FetchAsync(IServiceProvider services, CancellationToken cancellationToken)
|
public async Task FetchAsync(IServiceProvider services, CancellationToken cancellationToken)
|
||||||
{
|
{
|
||||||
ArgumentNullException.ThrowIfNull(services);
|
ArgumentNullException.ThrowIfNull(services);
|
||||||
|
|
||||||
var cursor = await GetCursorAsync(cancellationToken).ConfigureAwait(false);
|
var cursor = await GetCursorAsync(cancellationToken).ConfigureAwait(false);
|
||||||
var now = _timeProvider.GetUtcNow();
|
var now = _timeProvider.GetUtcNow();
|
||||||
var pendingDocuments = cursor.PendingDocuments.ToHashSet();
|
var pendingDocuments = cursor.PendingDocuments.ToHashSet();
|
||||||
var pendingMappings = cursor.PendingMappings.ToHashSet();
|
var pendingMappings = cursor.PendingMappings.ToHashSet();
|
||||||
var fetchCache = new Dictionary<string, VmwareFetchCacheEntry>(cursor.FetchCache, StringComparer.OrdinalIgnoreCase);
|
var fetchCache = new Dictionary<string, VmwareFetchCacheEntry>(cursor.FetchCache, StringComparer.OrdinalIgnoreCase);
|
||||||
var touchedResources = new HashSet<string>(StringComparer.OrdinalIgnoreCase);
|
var touchedResources = new HashSet<string>(StringComparer.OrdinalIgnoreCase);
|
||||||
var remainingCapacity = _options.MaxAdvisoriesPerFetch;
|
var remainingCapacity = _options.MaxAdvisoriesPerFetch;
|
||||||
|
|
||||||
IReadOnlyList<VmwareIndexItem> indexItems;
|
IReadOnlyList<VmwareIndexItem> indexItems;
|
||||||
try
|
try
|
||||||
{
|
{
|
||||||
indexItems = await FetchIndexAsync(cancellationToken).ConfigureAwait(false);
|
indexItems = await FetchIndexAsync(cancellationToken).ConfigureAwait(false);
|
||||||
}
|
}
|
||||||
catch (Exception ex)
|
catch (Exception ex)
|
||||||
{
|
{
|
||||||
_diagnostics.FetchFailure();
|
_diagnostics.FetchFailure();
|
||||||
_logger.LogError(ex, "Failed to retrieve VMware advisory index");
|
_logger.LogError(ex, "Failed to retrieve VMware advisory index");
|
||||||
await _stateRepository.MarkFailureAsync(SourceName, now, TimeSpan.FromMinutes(10), ex.Message, cancellationToken).ConfigureAwait(false);
|
await _stateRepository.MarkFailureAsync(SourceName, now, TimeSpan.FromMinutes(10), ex.Message, cancellationToken).ConfigureAwait(false);
|
||||||
throw;
|
throw;
|
||||||
}
|
}
|
||||||
|
|
||||||
if (indexItems.Count == 0)
|
if (indexItems.Count == 0)
|
||||||
{
|
{
|
||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
|
|
||||||
var orderedItems = indexItems
|
var orderedItems = indexItems
|
||||||
.Where(static item => !string.IsNullOrWhiteSpace(item.Id) && !string.IsNullOrWhiteSpace(item.DetailUrl))
|
.Where(static item => !string.IsNullOrWhiteSpace(item.Id) && !string.IsNullOrWhiteSpace(item.DetailUrl))
|
||||||
.OrderBy(static item => item.Modified ?? DateTimeOffset.MinValue)
|
.OrderBy(static item => item.Modified ?? DateTimeOffset.MinValue)
|
||||||
.ThenBy(static item => item.Id, StringComparer.OrdinalIgnoreCase)
|
.ThenBy(static item => item.Id, StringComparer.OrdinalIgnoreCase)
|
||||||
.ToArray();
|
.ToArray();
|
||||||
|
|
||||||
var baseline = cursor.LastModified ?? now - _options.InitialBackfill;
|
var baseline = cursor.LastModified ?? now - _options.InitialBackfill;
|
||||||
var resumeStart = baseline - _options.ModifiedTolerance;
|
var resumeStart = baseline - _options.ModifiedTolerance;
|
||||||
ProvenanceDiagnostics.ReportResumeWindow(SourceName, resumeStart, _logger);
|
ProvenanceDiagnostics.ReportResumeWindow(SourceName, resumeStart, _logger);
|
||||||
var processedIds = new HashSet<string>(cursor.ProcessedIds, StringComparer.OrdinalIgnoreCase);
|
var processedIds = new HashSet<string>(cursor.ProcessedIds, StringComparer.OrdinalIgnoreCase);
|
||||||
var maxModified = cursor.LastModified ?? DateTimeOffset.MinValue;
|
var maxModified = cursor.LastModified ?? DateTimeOffset.MinValue;
|
||||||
var processedUpdated = false;
|
var processedUpdated = false;
|
||||||
|
|
||||||
foreach (var item in orderedItems)
|
foreach (var item in orderedItems)
|
||||||
{
|
{
|
||||||
if (remainingCapacity <= 0)
|
if (remainingCapacity <= 0)
|
||||||
{
|
{
|
||||||
break;
|
break;
|
||||||
}
|
}
|
||||||
|
|
||||||
cancellationToken.ThrowIfCancellationRequested();
|
cancellationToken.ThrowIfCancellationRequested();
|
||||||
|
|
||||||
var modified = (item.Modified ?? DateTimeOffset.MinValue).ToUniversalTime();
|
var modified = (item.Modified ?? DateTimeOffset.MinValue).ToUniversalTime();
|
||||||
if (modified < baseline - _options.ModifiedTolerance)
|
if (modified < baseline - _options.ModifiedTolerance)
|
||||||
{
|
{
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
|
|
||||||
if (cursor.LastModified.HasValue && modified < cursor.LastModified.Value - _options.ModifiedTolerance)
|
if (cursor.LastModified.HasValue && modified < cursor.LastModified.Value - _options.ModifiedTolerance)
|
||||||
{
|
{
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
|
|
||||||
if (modified == cursor.LastModified && cursor.ProcessedIds.Contains(item.Id, StringComparer.OrdinalIgnoreCase))
|
if (modified == cursor.LastModified && cursor.ProcessedIds.Contains(item.Id, StringComparer.OrdinalIgnoreCase))
|
||||||
{
|
{
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
|
|
||||||
if (!Uri.TryCreate(item.DetailUrl, UriKind.Absolute, out var detailUri))
|
if (!Uri.TryCreate(item.DetailUrl, UriKind.Absolute, out var detailUri))
|
||||||
{
|
{
|
||||||
_logger.LogWarning("VMware advisory {AdvisoryId} has invalid detail URL {Url}", item.Id, item.DetailUrl);
|
_logger.LogWarning("VMware advisory {AdvisoryId} has invalid detail URL {Url}", item.Id, item.DetailUrl);
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
|
|
||||||
var cacheKey = detailUri.AbsoluteUri;
|
var cacheKey = detailUri.AbsoluteUri;
|
||||||
touchedResources.Add(cacheKey);
|
touchedResources.Add(cacheKey);
|
||||||
|
|
||||||
var existing = await _documentStore.FindBySourceAndUriAsync(SourceName, cacheKey, cancellationToken).ConfigureAwait(false);
|
var existing = await _documentStore.FindBySourceAndUriAsync(SourceName, cacheKey, cancellationToken).ConfigureAwait(false);
|
||||||
var metadata = new Dictionary<string, string>(StringComparer.Ordinal)
|
var metadata = new Dictionary<string, string>(StringComparer.Ordinal)
|
||||||
{
|
{
|
||||||
["vmware.id"] = item.Id,
|
["vmware.id"] = item.Id,
|
||||||
["vmware.modified"] = modified.ToString("O"),
|
["vmware.modified"] = modified.ToString("O"),
|
||||||
};
|
};
|
||||||
|
|
||||||
SourceFetchResult result;
|
SourceFetchResult result;
|
||||||
try
|
try
|
||||||
{
|
{
|
||||||
result = await _fetchService.FetchAsync(
|
result = await _fetchService.FetchAsync(
|
||||||
new SourceFetchRequest(VmwareOptions.HttpClientName, SourceName, detailUri)
|
new SourceFetchRequest(VmwareOptions.HttpClientName, SourceName, detailUri)
|
||||||
{
|
{
|
||||||
Metadata = metadata,
|
Metadata = metadata,
|
||||||
ETag = existing?.Etag,
|
ETag = existing?.Etag,
|
||||||
LastModified = existing?.LastModified,
|
LastModified = existing?.LastModified,
|
||||||
AcceptHeaders = new[] { "application/json" },
|
AcceptHeaders = new[] { "application/json" },
|
||||||
},
|
},
|
||||||
cancellationToken).ConfigureAwait(false);
|
cancellationToken).ConfigureAwait(false);
|
||||||
}
|
}
|
||||||
catch (Exception ex)
|
catch (Exception ex)
|
||||||
{
|
{
|
||||||
_diagnostics.FetchFailure();
|
_diagnostics.FetchFailure();
|
||||||
_logger.LogError(ex, "Failed to fetch VMware advisory {AdvisoryId}", item.Id);
|
_logger.LogError(ex, "Failed to fetch VMware advisory {AdvisoryId}", item.Id);
|
||||||
await _stateRepository.MarkFailureAsync(SourceName, now, TimeSpan.FromMinutes(5), ex.Message, cancellationToken).ConfigureAwait(false);
|
await _stateRepository.MarkFailureAsync(SourceName, now, TimeSpan.FromMinutes(5), ex.Message, cancellationToken).ConfigureAwait(false);
|
||||||
throw;
|
throw;
|
||||||
}
|
}
|
||||||
|
|
||||||
if (result.IsNotModified)
|
if (result.IsNotModified)
|
||||||
{
|
{
|
||||||
_diagnostics.FetchUnchanged();
|
_diagnostics.FetchUnchanged();
|
||||||
if (existing is not null)
|
if (existing is not null)
|
||||||
{
|
{
|
||||||
fetchCache[cacheKey] = VmwareFetchCacheEntry.FromDocument(existing);
|
fetchCache[cacheKey] = VmwareFetchCacheEntry.FromDocument(existing);
|
||||||
pendingDocuments.Remove(existing.Id);
|
pendingDocuments.Remove(existing.Id);
|
||||||
pendingMappings.Remove(existing.Id);
|
pendingMappings.Remove(existing.Id);
|
||||||
_logger.LogInformation("VMware advisory {AdvisoryId} returned 304 Not Modified", item.Id);
|
_logger.LogInformation("VMware advisory {AdvisoryId} returned 304 Not Modified", item.Id);
|
||||||
}
|
}
|
||||||
|
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
|
|
||||||
if (!result.IsSuccess || result.Document is null)
|
if (!result.IsSuccess || result.Document is null)
|
||||||
{
|
{
|
||||||
_diagnostics.FetchFailure();
|
_diagnostics.FetchFailure();
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
|
|
||||||
remainingCapacity--;
|
remainingCapacity--;
|
||||||
|
|
||||||
if (modified > maxModified)
|
if (modified > maxModified)
|
||||||
{
|
{
|
||||||
maxModified = modified;
|
maxModified = modified;
|
||||||
processedIds.Clear();
|
processedIds.Clear();
|
||||||
processedUpdated = true;
|
processedUpdated = true;
|
||||||
}
|
}
|
||||||
|
|
||||||
if (modified == maxModified)
|
if (modified == maxModified)
|
||||||
{
|
{
|
||||||
processedIds.Add(item.Id);
|
processedIds.Add(item.Id);
|
||||||
processedUpdated = true;
|
processedUpdated = true;
|
||||||
}
|
}
|
||||||
|
|
||||||
var cacheEntry = VmwareFetchCacheEntry.FromDocument(result.Document);
|
var cacheEntry = VmwareFetchCacheEntry.FromDocument(result.Document);
|
||||||
|
|
||||||
if (existing is not null
|
if (existing is not null
|
||||||
&& string.Equals(existing.Status, DocumentStatuses.Mapped, StringComparison.Ordinal)
|
&& string.Equals(existing.Status, DocumentStatuses.Mapped, StringComparison.Ordinal)
|
||||||
&& cursor.TryGetFetchCache(cacheKey, out var cachedEntry)
|
&& cursor.TryGetFetchCache(cacheKey, out var cachedEntry)
|
||||||
&& cachedEntry.Matches(result.Document))
|
&& cachedEntry.Matches(result.Document))
|
||||||
{
|
{
|
||||||
_diagnostics.FetchUnchanged();
|
_diagnostics.FetchUnchanged();
|
||||||
fetchCache[cacheKey] = cacheEntry;
|
fetchCache[cacheKey] = cacheEntry;
|
||||||
pendingDocuments.Remove(result.Document.Id);
|
pendingDocuments.Remove(result.Document.Id);
|
||||||
pendingMappings.Remove(result.Document.Id);
|
pendingMappings.Remove(result.Document.Id);
|
||||||
await _documentStore.UpdateStatusAsync(result.Document.Id, existing.Status, cancellationToken).ConfigureAwait(false);
|
await _documentStore.UpdateStatusAsync(result.Document.Id, existing.Status, cancellationToken).ConfigureAwait(false);
|
||||||
_logger.LogInformation("VMware advisory {AdvisoryId} unchanged; skipping reprocessing", item.Id);
|
_logger.LogInformation("VMware advisory {AdvisoryId} unchanged; skipping reprocessing", item.Id);
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
|
|
||||||
_diagnostics.FetchItem();
|
_diagnostics.FetchItem();
|
||||||
fetchCache[cacheKey] = cacheEntry;
|
fetchCache[cacheKey] = cacheEntry;
|
||||||
pendingDocuments.Add(result.Document.Id);
|
pendingDocuments.Add(result.Document.Id);
|
||||||
_logger.LogInformation(
|
_logger.LogInformation(
|
||||||
"VMware advisory {AdvisoryId} fetched (documentId={DocumentId}, sha256={Sha})",
|
"VMware advisory {AdvisoryId} fetched (documentId={DocumentId}, sha256={Sha})",
|
||||||
item.Id,
|
item.Id,
|
||||||
result.Document.Id,
|
result.Document.Id,
|
||||||
result.Document.Sha256);
|
result.Document.Sha256);
|
||||||
|
|
||||||
if (_options.RequestDelay > TimeSpan.Zero)
|
if (_options.RequestDelay > TimeSpan.Zero)
|
||||||
{
|
{
|
||||||
try
|
try
|
||||||
{
|
{
|
||||||
await Task.Delay(_options.RequestDelay, cancellationToken).ConfigureAwait(false);
|
await Task.Delay(_options.RequestDelay, cancellationToken).ConfigureAwait(false);
|
||||||
}
|
}
|
||||||
catch (TaskCanceledException)
|
catch (TaskCanceledException)
|
||||||
{
|
{
|
||||||
break;
|
break;
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
if (fetchCache.Count > 0 && touchedResources.Count > 0)
|
if (fetchCache.Count > 0 && touchedResources.Count > 0)
|
||||||
{
|
{
|
||||||
var stale = fetchCache.Keys.Where(key => !touchedResources.Contains(key)).ToArray();
|
var stale = fetchCache.Keys.Where(key => !touchedResources.Contains(key)).ToArray();
|
||||||
foreach (var key in stale)
|
foreach (var key in stale)
|
||||||
{
|
{
|
||||||
fetchCache.Remove(key);
|
fetchCache.Remove(key);
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
var updatedCursor = cursor
|
var updatedCursor = cursor
|
||||||
.WithPendingDocuments(pendingDocuments)
|
.WithPendingDocuments(pendingDocuments)
|
||||||
.WithPendingMappings(pendingMappings)
|
.WithPendingMappings(pendingMappings)
|
||||||
.WithFetchCache(fetchCache);
|
.WithFetchCache(fetchCache);
|
||||||
|
|
||||||
if (processedUpdated)
|
if (processedUpdated)
|
||||||
{
|
{
|
||||||
updatedCursor = updatedCursor.WithLastModified(maxModified, processedIds);
|
updatedCursor = updatedCursor.WithLastModified(maxModified, processedIds);
|
||||||
}
|
}
|
||||||
|
|
||||||
await UpdateCursorAsync(updatedCursor, cancellationToken).ConfigureAwait(false);
|
await UpdateCursorAsync(updatedCursor, cancellationToken).ConfigureAwait(false);
|
||||||
}
|
}
|
||||||
|
|
||||||
public async Task ParseAsync(IServiceProvider services, CancellationToken cancellationToken)
|
public async Task ParseAsync(IServiceProvider services, CancellationToken cancellationToken)
|
||||||
{
|
{
|
||||||
ArgumentNullException.ThrowIfNull(services);
|
ArgumentNullException.ThrowIfNull(services);
|
||||||
|
|
||||||
var cursor = await GetCursorAsync(cancellationToken).ConfigureAwait(false);
|
var cursor = await GetCursorAsync(cancellationToken).ConfigureAwait(false);
|
||||||
if (cursor.PendingDocuments.Count == 0)
|
if (cursor.PendingDocuments.Count == 0)
|
||||||
{
|
{
|
||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
|
|
||||||
var remaining = cursor.PendingDocuments.ToList();
|
var remaining = cursor.PendingDocuments.ToList();
|
||||||
var pendingMappings = cursor.PendingMappings.ToList();
|
var pendingMappings = cursor.PendingMappings.ToList();
|
||||||
|
|
||||||
foreach (var documentId in cursor.PendingDocuments)
|
foreach (var documentId in cursor.PendingDocuments)
|
||||||
{
|
{
|
||||||
cancellationToken.ThrowIfCancellationRequested();
|
cancellationToken.ThrowIfCancellationRequested();
|
||||||
|
|
||||||
var document = await _documentStore.FindAsync(documentId, cancellationToken).ConfigureAwait(false);
|
var document = await _documentStore.FindAsync(documentId, cancellationToken).ConfigureAwait(false);
|
||||||
if (document is null)
|
if (document is null)
|
||||||
{
|
{
|
||||||
remaining.Remove(documentId);
|
remaining.Remove(documentId);
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
|
|
||||||
if (!document.GridFsId.HasValue)
|
if (!document.PayloadId.HasValue)
|
||||||
{
|
{
|
||||||
_logger.LogWarning("VMware document {DocumentId} missing GridFS payload", document.Id);
|
_logger.LogWarning("VMware document {DocumentId} missing GridFS payload", document.Id);
|
||||||
await _documentStore.UpdateStatusAsync(document.Id, DocumentStatuses.Failed, cancellationToken).ConfigureAwait(false);
|
await _documentStore.UpdateStatusAsync(document.Id, DocumentStatuses.Failed, cancellationToken).ConfigureAwait(false);
|
||||||
remaining.Remove(documentId);
|
remaining.Remove(documentId);
|
||||||
_diagnostics.ParseFailure();
|
_diagnostics.ParseFailure();
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
|
|
||||||
byte[] bytes;
|
byte[] bytes;
|
||||||
try
|
try
|
||||||
{
|
{
|
||||||
bytes = await _rawDocumentStorage.DownloadAsync(document.GridFsId.Value, cancellationToken).ConfigureAwait(false);
|
bytes = await _rawDocumentStorage.DownloadAsync(document.PayloadId.Value, cancellationToken).ConfigureAwait(false);
|
||||||
}
|
}
|
||||||
catch (Exception ex)
|
catch (Exception ex)
|
||||||
{
|
{
|
||||||
_logger.LogError(ex, "Failed downloading VMware document {DocumentId}", document.Id);
|
_logger.LogError(ex, "Failed downloading VMware document {DocumentId}", document.Id);
|
||||||
throw;
|
throw;
|
||||||
}
|
}
|
||||||
|
|
||||||
VmwareDetailDto? detail;
|
VmwareDetailDto? detail;
|
||||||
try
|
try
|
||||||
{
|
{
|
||||||
detail = JsonSerializer.Deserialize<VmwareDetailDto>(bytes, SerializerOptions);
|
detail = JsonSerializer.Deserialize<VmwareDetailDto>(bytes, SerializerOptions);
|
||||||
}
|
}
|
||||||
catch (Exception ex)
|
catch (Exception ex)
|
||||||
{
|
{
|
||||||
_logger.LogWarning(ex, "Failed to deserialize VMware advisory {DocumentId}", document.Id);
|
_logger.LogWarning(ex, "Failed to deserialize VMware advisory {DocumentId}", document.Id);
|
||||||
await _documentStore.UpdateStatusAsync(document.Id, DocumentStatuses.Failed, cancellationToken).ConfigureAwait(false);
|
await _documentStore.UpdateStatusAsync(document.Id, DocumentStatuses.Failed, cancellationToken).ConfigureAwait(false);
|
||||||
remaining.Remove(documentId);
|
remaining.Remove(documentId);
|
||||||
_diagnostics.ParseFailure();
|
_diagnostics.ParseFailure();
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
|
|
||||||
if (detail is null || string.IsNullOrWhiteSpace(detail.AdvisoryId))
|
if (detail is null || string.IsNullOrWhiteSpace(detail.AdvisoryId))
|
||||||
{
|
{
|
||||||
_logger.LogWarning("VMware advisory document {DocumentId} contained empty payload", document.Id);
|
_logger.LogWarning("VMware advisory document {DocumentId} contained empty payload", document.Id);
|
||||||
await _documentStore.UpdateStatusAsync(document.Id, DocumentStatuses.Failed, cancellationToken).ConfigureAwait(false);
|
await _documentStore.UpdateStatusAsync(document.Id, DocumentStatuses.Failed, cancellationToken).ConfigureAwait(false);
|
||||||
remaining.Remove(documentId);
|
remaining.Remove(documentId);
|
||||||
_diagnostics.ParseFailure();
|
_diagnostics.ParseFailure();
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
|
|
||||||
var sanitized = JsonSerializer.Serialize(detail, SerializerOptions);
|
var sanitized = JsonSerializer.Serialize(detail, SerializerOptions);
|
||||||
var payload = MongoDB.Bson.BsonDocument.Parse(sanitized);
|
var payload = MongoDB.Bson.BsonDocument.Parse(sanitized);
|
||||||
var dtoRecord = new DtoRecord(Guid.NewGuid(), document.Id, SourceName, "vmware.v1", payload, _timeProvider.GetUtcNow());
|
var dtoRecord = new DtoRecord(Guid.NewGuid(), document.Id, SourceName, "vmware.v1", payload, _timeProvider.GetUtcNow());
|
||||||
|
|
||||||
await _dtoStore.UpsertAsync(dtoRecord, cancellationToken).ConfigureAwait(false);
|
await _dtoStore.UpsertAsync(dtoRecord, cancellationToken).ConfigureAwait(false);
|
||||||
await _documentStore.UpdateStatusAsync(document.Id, DocumentStatuses.PendingMap, cancellationToken).ConfigureAwait(false);
|
await _documentStore.UpdateStatusAsync(document.Id, DocumentStatuses.PendingMap, cancellationToken).ConfigureAwait(false);
|
||||||
|
|
||||||
remaining.Remove(documentId);
|
remaining.Remove(documentId);
|
||||||
if (!pendingMappings.Contains(documentId))
|
if (!pendingMappings.Contains(documentId))
|
||||||
{
|
{
|
||||||
pendingMappings.Add(documentId);
|
pendingMappings.Add(documentId);
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
var updatedCursor = cursor
|
var updatedCursor = cursor
|
||||||
.WithPendingDocuments(remaining)
|
.WithPendingDocuments(remaining)
|
||||||
.WithPendingMappings(pendingMappings);
|
.WithPendingMappings(pendingMappings);
|
||||||
|
|
||||||
await UpdateCursorAsync(updatedCursor, cancellationToken).ConfigureAwait(false);
|
await UpdateCursorAsync(updatedCursor, cancellationToken).ConfigureAwait(false);
|
||||||
}
|
}
|
||||||
|
|
||||||
public async Task MapAsync(IServiceProvider services, CancellationToken cancellationToken)
|
public async Task MapAsync(IServiceProvider services, CancellationToken cancellationToken)
|
||||||
{
|
{
|
||||||
ArgumentNullException.ThrowIfNull(services);
|
ArgumentNullException.ThrowIfNull(services);
|
||||||
|
|
||||||
var cursor = await GetCursorAsync(cancellationToken).ConfigureAwait(false);
|
var cursor = await GetCursorAsync(cancellationToken).ConfigureAwait(false);
|
||||||
if (cursor.PendingMappings.Count == 0)
|
if (cursor.PendingMappings.Count == 0)
|
||||||
{
|
{
|
||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
|
|
||||||
var pendingMappings = cursor.PendingMappings.ToList();
|
var pendingMappings = cursor.PendingMappings.ToList();
|
||||||
|
|
||||||
foreach (var documentId in cursor.PendingMappings)
|
foreach (var documentId in cursor.PendingMappings)
|
||||||
{
|
{
|
||||||
cancellationToken.ThrowIfCancellationRequested();
|
cancellationToken.ThrowIfCancellationRequested();
|
||||||
|
|
||||||
var dto = await _dtoStore.FindByDocumentIdAsync(documentId, cancellationToken).ConfigureAwait(false);
|
var dto = await _dtoStore.FindByDocumentIdAsync(documentId, cancellationToken).ConfigureAwait(false);
|
||||||
var document = await _documentStore.FindAsync(documentId, cancellationToken).ConfigureAwait(false);
|
var document = await _documentStore.FindAsync(documentId, cancellationToken).ConfigureAwait(false);
|
||||||
if (dto is null || document is null)
|
if (dto is null || document is null)
|
||||||
{
|
{
|
||||||
pendingMappings.Remove(documentId);
|
pendingMappings.Remove(documentId);
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
|
|
||||||
var json = dto.Payload.ToJson(new JsonWriterSettings
|
var json = dto.Payload.ToJson(new JsonWriterSettings
|
||||||
{
|
{
|
||||||
OutputMode = JsonOutputMode.RelaxedExtendedJson,
|
OutputMode = JsonOutputMode.RelaxedExtendedJson,
|
||||||
});
|
});
|
||||||
|
|
||||||
VmwareDetailDto? detail;
|
VmwareDetailDto? detail;
|
||||||
try
|
try
|
||||||
{
|
{
|
||||||
detail = JsonSerializer.Deserialize<VmwareDetailDto>(json, SerializerOptions);
|
detail = JsonSerializer.Deserialize<VmwareDetailDto>(json, SerializerOptions);
|
||||||
}
|
}
|
||||||
catch (Exception ex)
|
catch (Exception ex)
|
||||||
{
|
{
|
||||||
_logger.LogError(ex, "Failed to deserialize VMware DTO for document {DocumentId}", document.Id);
|
_logger.LogError(ex, "Failed to deserialize VMware DTO for document {DocumentId}", document.Id);
|
||||||
await _documentStore.UpdateStatusAsync(document.Id, DocumentStatuses.Failed, cancellationToken).ConfigureAwait(false);
|
await _documentStore.UpdateStatusAsync(document.Id, DocumentStatuses.Failed, cancellationToken).ConfigureAwait(false);
|
||||||
pendingMappings.Remove(documentId);
|
pendingMappings.Remove(documentId);
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
|
|
||||||
if (detail is null || string.IsNullOrWhiteSpace(detail.AdvisoryId))
|
if (detail is null || string.IsNullOrWhiteSpace(detail.AdvisoryId))
|
||||||
{
|
{
|
||||||
await _documentStore.UpdateStatusAsync(document.Id, DocumentStatuses.Failed, cancellationToken).ConfigureAwait(false);
|
await _documentStore.UpdateStatusAsync(document.Id, DocumentStatuses.Failed, cancellationToken).ConfigureAwait(false);
|
||||||
pendingMappings.Remove(documentId);
|
pendingMappings.Remove(documentId);
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
|
|
||||||
var (advisory, flag) = VmwareMapper.Map(detail, document, dto);
|
var (advisory, flag) = VmwareMapper.Map(detail, document, dto);
|
||||||
await _advisoryStore.UpsertAsync(advisory, cancellationToken).ConfigureAwait(false);
|
await _advisoryStore.UpsertAsync(advisory, cancellationToken).ConfigureAwait(false);
|
||||||
await _psirtFlagStore.UpsertAsync(flag, cancellationToken).ConfigureAwait(false);
|
await _psirtFlagStore.UpsertAsync(flag, cancellationToken).ConfigureAwait(false);
|
||||||
await _documentStore.UpdateStatusAsync(document.Id, DocumentStatuses.Mapped, cancellationToken).ConfigureAwait(false);
|
await _documentStore.UpdateStatusAsync(document.Id, DocumentStatuses.Mapped, cancellationToken).ConfigureAwait(false);
|
||||||
_diagnostics.MapAffectedCount(advisory.AffectedPackages.Length);
|
_diagnostics.MapAffectedCount(advisory.AffectedPackages.Length);
|
||||||
_logger.LogInformation(
|
_logger.LogInformation(
|
||||||
"VMware advisory {AdvisoryId} mapped with {AffectedCount} affected packages",
|
"VMware advisory {AdvisoryId} mapped with {AffectedCount} affected packages",
|
||||||
detail.AdvisoryId,
|
detail.AdvisoryId,
|
||||||
advisory.AffectedPackages.Length);
|
advisory.AffectedPackages.Length);
|
||||||
|
|
||||||
pendingMappings.Remove(documentId);
|
pendingMappings.Remove(documentId);
|
||||||
}
|
}
|
||||||
|
|
||||||
var updatedCursor = cursor.WithPendingMappings(pendingMappings);
|
var updatedCursor = cursor.WithPendingMappings(pendingMappings);
|
||||||
await UpdateCursorAsync(updatedCursor, cancellationToken).ConfigureAwait(false);
|
await UpdateCursorAsync(updatedCursor, cancellationToken).ConfigureAwait(false);
|
||||||
}
|
}
|
||||||
|
|
||||||
private async Task<IReadOnlyList<VmwareIndexItem>> FetchIndexAsync(CancellationToken cancellationToken)
|
private async Task<IReadOnlyList<VmwareIndexItem>> FetchIndexAsync(CancellationToken cancellationToken)
|
||||||
{
|
{
|
||||||
var client = _httpClientFactory.CreateClient(VmwareOptions.HttpClientName);
|
var client = _httpClientFactory.CreateClient(VmwareOptions.HttpClientName);
|
||||||
using var response = await client.GetAsync(_options.IndexUri, cancellationToken).ConfigureAwait(false);
|
using var response = await client.GetAsync(_options.IndexUri, cancellationToken).ConfigureAwait(false);
|
||||||
response.EnsureSuccessStatusCode();
|
response.EnsureSuccessStatusCode();
|
||||||
|
|
||||||
await using var stream = await response.Content.ReadAsStreamAsync(cancellationToken).ConfigureAwait(false);
|
await using var stream = await response.Content.ReadAsStreamAsync(cancellationToken).ConfigureAwait(false);
|
||||||
var items = await JsonSerializer.DeserializeAsync<IReadOnlyList<VmwareIndexItem>>(stream, SerializerOptions, cancellationToken).ConfigureAwait(false);
|
var items = await JsonSerializer.DeserializeAsync<IReadOnlyList<VmwareIndexItem>>(stream, SerializerOptions, cancellationToken).ConfigureAwait(false);
|
||||||
return items ?? Array.Empty<VmwareIndexItem>();
|
return items ?? Array.Empty<VmwareIndexItem>();
|
||||||
}
|
}
|
||||||
|
|
||||||
private async Task<VmwareCursor> GetCursorAsync(CancellationToken cancellationToken)
|
private async Task<VmwareCursor> GetCursorAsync(CancellationToken cancellationToken)
|
||||||
{
|
{
|
||||||
var state = await _stateRepository.TryGetAsync(SourceName, cancellationToken).ConfigureAwait(false);
|
var state = await _stateRepository.TryGetAsync(SourceName, cancellationToken).ConfigureAwait(false);
|
||||||
return state is null ? VmwareCursor.Empty : VmwareCursor.FromBson(state.Cursor);
|
return state is null ? VmwareCursor.Empty : VmwareCursor.FromBson(state.Cursor);
|
||||||
}
|
}
|
||||||
|
|
||||||
private async Task UpdateCursorAsync(VmwareCursor cursor, CancellationToken cancellationToken)
|
private async Task UpdateCursorAsync(VmwareCursor cursor, CancellationToken cancellationToken)
|
||||||
{
|
{
|
||||||
var document = cursor.ToBsonDocument();
|
var document = cursor.ToBsonDocument();
|
||||||
await _stateRepository.UpdateCursorAsync(SourceName, document, _timeProvider.GetUtcNow(), cancellationToken).ConfigureAwait(false);
|
await _stateRepository.UpdateCursorAsync(SourceName, document, _timeProvider.GetUtcNow(), cancellationToken).ConfigureAwait(false);
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -10,6 +10,7 @@
|
|||||||
<ItemGroup>
|
<ItemGroup>
|
||||||
<ProjectReference Include="..\StellaOps.Concelier.Models\StellaOps.Concelier.Models.csproj" />
|
<ProjectReference Include="..\StellaOps.Concelier.Models\StellaOps.Concelier.Models.csproj" />
|
||||||
<ProjectReference Include="..\StellaOps.Concelier.Normalization\StellaOps.Concelier.Normalization.csproj" />
|
<ProjectReference Include="..\StellaOps.Concelier.Normalization\StellaOps.Concelier.Normalization.csproj" />
|
||||||
|
<ProjectReference Include="..\StellaOps.Concelier.Core\StellaOps.Concelier.Core.csproj" />
|
||||||
<ProjectReference Include="../../../__Libraries/StellaOps.Plugin/StellaOps.Plugin.csproj" />
|
<ProjectReference Include="../../../__Libraries/StellaOps.Plugin/StellaOps.Plugin.csproj" />
|
||||||
<ProjectReference Include="../../../__Libraries/StellaOps.DependencyInjection/StellaOps.DependencyInjection.csproj" />
|
<ProjectReference Include="../../../__Libraries/StellaOps.DependencyInjection/StellaOps.DependencyInjection.csproj" />
|
||||||
<ProjectReference Include="../../../__Libraries/StellaOps.Cryptography/StellaOps.Cryptography.csproj" />
|
<ProjectReference Include="../../../__Libraries/StellaOps.Cryptography/StellaOps.Cryptography.csproj" />
|
||||||
@@ -20,4 +21,4 @@
|
|||||||
<PackageReference Include="Microsoft.Extensions.Options" Version="10.0.0" />
|
<PackageReference Include="Microsoft.Extensions.Options" Version="10.0.0" />
|
||||||
<PackageReference Include="Microsoft.Extensions.Options.ConfigurationExtensions" Version="10.0.0" />
|
<PackageReference Include="Microsoft.Extensions.Options.ConfigurationExtensions" Version="10.0.0" />
|
||||||
</ItemGroup>
|
</ItemGroup>
|
||||||
</Project>
|
</Project>
|
||||||
|
|||||||
@@ -10,6 +10,7 @@
|
|||||||
<ItemGroup>
|
<ItemGroup>
|
||||||
<ProjectReference Include="..\StellaOps.Concelier.Exporter.Json\StellaOps.Concelier.Exporter.Json.csproj" />
|
<ProjectReference Include="..\StellaOps.Concelier.Exporter.Json\StellaOps.Concelier.Exporter.Json.csproj" />
|
||||||
<ProjectReference Include="..\StellaOps.Concelier.Models\StellaOps.Concelier.Models.csproj" />
|
<ProjectReference Include="..\StellaOps.Concelier.Models\StellaOps.Concelier.Models.csproj" />
|
||||||
|
<ProjectReference Include="..\StellaOps.Concelier.Core\StellaOps.Concelier.Core.csproj" />
|
||||||
<ProjectReference Include="../../../__Libraries/StellaOps.DependencyInjection/StellaOps.DependencyInjection.csproj" />
|
<ProjectReference Include="../../../__Libraries/StellaOps.DependencyInjection/StellaOps.DependencyInjection.csproj" />
|
||||||
<ProjectReference Include="../../../__Libraries/StellaOps.Plugin/StellaOps.Plugin.csproj" />
|
<ProjectReference Include="../../../__Libraries/StellaOps.Plugin/StellaOps.Plugin.csproj" />
|
||||||
</ItemGroup>
|
</ItemGroup>
|
||||||
@@ -18,4 +19,4 @@
|
|||||||
<PackageReference Include="Microsoft.Extensions.Options" Version="10.0.0" />
|
<PackageReference Include="Microsoft.Extensions.Options" Version="10.0.0" />
|
||||||
<PackageReference Include="Microsoft.Extensions.Options.ConfigurationExtensions" Version="10.0.0" />
|
<PackageReference Include="Microsoft.Extensions.Options.ConfigurationExtensions" Version="10.0.0" />
|
||||||
</ItemGroup>
|
</ItemGroup>
|
||||||
</Project>
|
</Project>
|
||||||
|
|||||||
@@ -24,6 +24,8 @@ namespace MongoDB.Bson
|
|||||||
{
|
{
|
||||||
protected readonly object? _value;
|
protected readonly object? _value;
|
||||||
public BsonValue(object? value) => _value = value;
|
public BsonValue(object? value) => _value = value;
|
||||||
|
internal object? RawValue => _value;
|
||||||
|
public static BsonValue Create(object? value) => BsonDocument.WrapExternal(value);
|
||||||
public virtual BsonType BsonType => _value switch
|
public virtual BsonType BsonType => _value switch
|
||||||
{
|
{
|
||||||
null => BsonType.Null,
|
null => BsonType.Null,
|
||||||
@@ -59,12 +61,24 @@ namespace MongoDB.Bson
|
|||||||
public class BsonInt64 : BsonValue { public BsonInt64(long value) : base(value) { } }
|
public class BsonInt64 : BsonValue { public BsonInt64(long value) : base(value) { } }
|
||||||
public class BsonDouble : BsonValue { public BsonDouble(double value) : base(value) { } }
|
public class BsonDouble : BsonValue { public BsonDouble(double value) : base(value) { } }
|
||||||
public class BsonDateTime : BsonValue { public BsonDateTime(DateTime value) : base(value) { } }
|
public class BsonDateTime : BsonValue { public BsonDateTime(DateTime value) : base(value) { } }
|
||||||
|
public class BsonNull : BsonValue
|
||||||
|
{
|
||||||
|
private BsonNull() : base(null) { }
|
||||||
|
public static BsonNull Value { get; } = new();
|
||||||
|
}
|
||||||
|
|
||||||
public class BsonArray : BsonValue, IEnumerable<BsonValue>
|
public class BsonArray : BsonValue, IEnumerable<BsonValue>
|
||||||
{
|
{
|
||||||
private readonly List<BsonValue> _items = new();
|
private readonly List<BsonValue> _items = new();
|
||||||
public BsonArray() : base(null) { }
|
public BsonArray() : base(null) { }
|
||||||
public BsonArray(IEnumerable<BsonValue> values) : this() => _items.AddRange(values);
|
public BsonArray(IEnumerable<BsonValue> values) : this() => _items.AddRange(values);
|
||||||
|
public BsonArray(IEnumerable<object?> values) : this()
|
||||||
|
{
|
||||||
|
foreach (var value in values)
|
||||||
|
{
|
||||||
|
_items.Add(BsonDocument.WrapExternal(value));
|
||||||
|
}
|
||||||
|
}
|
||||||
public void Add(BsonValue value) => _items.Add(value);
|
public void Add(BsonValue value) => _items.Add(value);
|
||||||
public IEnumerator<BsonValue> GetEnumerator() => _items.GetEnumerator();
|
public IEnumerator<BsonValue> GetEnumerator() => _items.GetEnumerator();
|
||||||
IEnumerator IEnumerable.GetEnumerator() => GetEnumerator();
|
IEnumerator IEnumerable.GetEnumerator() => GetEnumerator();
|
||||||
@@ -93,6 +107,8 @@ namespace MongoDB.Bson
|
|||||||
_ => new BsonValue(value)
|
_ => new BsonValue(value)
|
||||||
};
|
};
|
||||||
|
|
||||||
|
internal static BsonValue WrapExternal(object? value) => Wrap(value);
|
||||||
|
|
||||||
public BsonValue this[string key]
|
public BsonValue this[string key]
|
||||||
{
|
{
|
||||||
get => _values[key];
|
get => _values[key];
|
||||||
@@ -104,6 +120,7 @@ namespace MongoDB.Bson
|
|||||||
public bool TryGetValue(string key, out BsonValue value) => _values.TryGetValue(key, out value!);
|
public bool TryGetValue(string key, out BsonValue value) => _values.TryGetValue(key, out value!);
|
||||||
|
|
||||||
public void Add(string key, BsonValue value) => _values[key] = value;
|
public void Add(string key, BsonValue value) => _values[key] = value;
|
||||||
|
public void Add(string key, object? value) => _values[key] = Wrap(value);
|
||||||
|
|
||||||
public IEnumerator<KeyValuePair<string, BsonValue>> GetEnumerator() => _values.GetEnumerator();
|
public IEnumerator<KeyValuePair<string, BsonValue>> GetEnumerator() => _values.GetEnumerator();
|
||||||
IEnumerator IEnumerable.GetEnumerator() => GetEnumerator();
|
IEnumerator IEnumerable.GetEnumerator() => GetEnumerator();
|
||||||
@@ -156,7 +173,7 @@ namespace MongoDB.Bson
|
|||||||
{
|
{
|
||||||
BsonDocument doc => doc._values.ToDictionary(kvp => kvp.Key, kvp => Unwrap(kvp.Value)),
|
BsonDocument doc => doc._values.ToDictionary(kvp => kvp.Key, kvp => Unwrap(kvp.Value)),
|
||||||
BsonArray array => array.Select(Unwrap).ToArray(),
|
BsonArray array => array.Select(Unwrap).ToArray(),
|
||||||
_ => value._value
|
_ => value.RawValue
|
||||||
};
|
};
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -1,4 +1,5 @@
|
|||||||
using System.Collections.Concurrent;
|
using System.Collections.Concurrent;
|
||||||
|
using System.IO;
|
||||||
using StellaOps.Concelier.Models;
|
using StellaOps.Concelier.Models;
|
||||||
|
|
||||||
namespace StellaOps.Concelier.Storage.Mongo
|
namespace StellaOps.Concelier.Storage.Mongo
|
||||||
@@ -33,8 +34,9 @@ namespace StellaOps.Concelier.Storage.Mongo
|
|||||||
IReadOnlyDictionary<string, string>? Metadata = null,
|
IReadOnlyDictionary<string, string>? Metadata = null,
|
||||||
string? Etag = null,
|
string? Etag = null,
|
||||||
DateTimeOffset? LastModified = null,
|
DateTimeOffset? LastModified = null,
|
||||||
MongoDB.Bson.ObjectId? GridFsId = null,
|
Guid? PayloadId = null,
|
||||||
DateTimeOffset? ExpiresAt = null);
|
DateTimeOffset? ExpiresAt = null,
|
||||||
|
byte[]? Payload = null);
|
||||||
|
|
||||||
public interface IDocumentStore
|
public interface IDocumentStore
|
||||||
{
|
{
|
||||||
@@ -85,7 +87,7 @@ namespace StellaOps.Concelier.Storage.Mongo
|
|||||||
Guid DocumentId,
|
Guid DocumentId,
|
||||||
string SourceName,
|
string SourceName,
|
||||||
string Format,
|
string Format,
|
||||||
MongoDB.Bson.BsonDocument Payload,
|
string Payload,
|
||||||
DateTimeOffset CreatedAt);
|
DateTimeOffset CreatedAt);
|
||||||
|
|
||||||
public interface IDtoStore
|
public interface IDtoStore
|
||||||
@@ -113,40 +115,40 @@ namespace StellaOps.Concelier.Storage.Mongo
|
|||||||
|
|
||||||
public sealed class RawDocumentStorage
|
public sealed class RawDocumentStorage
|
||||||
{
|
{
|
||||||
private readonly ConcurrentDictionary<MongoDB.Bson.ObjectId, byte[]> _blobs = new();
|
private readonly ConcurrentDictionary<Guid, byte[]> _blobs = new();
|
||||||
|
|
||||||
public Task<MongoDB.Bson.ObjectId> UploadAsync(string sourceName, string uri, byte[] content, string? contentType, DateTimeOffset? expiresAt, CancellationToken cancellationToken)
|
public Task<Guid> UploadAsync(string sourceName, string uri, byte[] content, string? contentType, DateTimeOffset? expiresAt, CancellationToken cancellationToken)
|
||||||
{
|
{
|
||||||
var id = MongoDB.Bson.ObjectId.GenerateNewId();
|
var id = Guid.NewGuid();
|
||||||
_blobs[id] = content.ToArray();
|
_blobs[id] = content.ToArray();
|
||||||
return Task.FromResult(id);
|
return Task.FromResult(id);
|
||||||
}
|
}
|
||||||
|
|
||||||
public Task<MongoDB.Bson.ObjectId> UploadAsync(string sourceName, string uri, byte[] content, string? contentType, CancellationToken cancellationToken)
|
public Task<Guid> UploadAsync(string sourceName, string uri, byte[] content, string? contentType, CancellationToken cancellationToken)
|
||||||
=> UploadAsync(sourceName, uri, content, contentType, null, cancellationToken);
|
=> UploadAsync(sourceName, uri, content, contentType, null, cancellationToken);
|
||||||
|
|
||||||
public Task<byte[]> DownloadAsync(MongoDB.Bson.ObjectId id, CancellationToken cancellationToken)
|
public Task<byte[]> DownloadAsync(Guid id, CancellationToken cancellationToken)
|
||||||
{
|
{
|
||||||
if (_blobs.TryGetValue(id, out var bytes))
|
if (_blobs.TryGetValue(id, out var bytes))
|
||||||
{
|
{
|
||||||
return Task.FromResult(bytes);
|
return Task.FromResult(bytes);
|
||||||
}
|
}
|
||||||
throw new MongoDB.Driver.GridFSFileNotFoundException($"Blob {id} not found.");
|
throw new FileNotFoundException($"Blob {id} not found.");
|
||||||
}
|
}
|
||||||
|
|
||||||
public Task DeleteAsync(MongoDB.Bson.ObjectId id, CancellationToken cancellationToken)
|
public Task DeleteAsync(Guid id, CancellationToken cancellationToken)
|
||||||
{
|
{
|
||||||
_blobs.TryRemove(id, out _);
|
_blobs.TryRemove(id, out _);
|
||||||
return Task.CompletedTask;
|
return Task.CompletedTask;
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
public sealed record SourceStateRecord(string SourceName, MongoDB.Bson.BsonDocument? Cursor, DateTimeOffset UpdatedAt);
|
public sealed record SourceStateRecord(string SourceName, string? CursorJson, DateTimeOffset UpdatedAt);
|
||||||
|
|
||||||
public interface ISourceStateRepository
|
public interface ISourceStateRepository
|
||||||
{
|
{
|
||||||
Task<SourceStateRecord?> TryGetAsync(string sourceName, CancellationToken cancellationToken);
|
Task<SourceStateRecord?> TryGetAsync(string sourceName, CancellationToken cancellationToken);
|
||||||
Task UpdateCursorAsync(string sourceName, MongoDB.Bson.BsonDocument cursor, DateTimeOffset completedAt, CancellationToken cancellationToken);
|
Task UpdateCursorAsync(string sourceName, string cursorJson, DateTimeOffset completedAt, CancellationToken cancellationToken);
|
||||||
Task MarkFailureAsync(string sourceName, DateTimeOffset now, TimeSpan backoff, string reason, CancellationToken cancellationToken);
|
Task MarkFailureAsync(string sourceName, DateTimeOffset now, TimeSpan backoff, string reason, CancellationToken cancellationToken);
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -160,9 +162,9 @@ namespace StellaOps.Concelier.Storage.Mongo
|
|||||||
return Task.FromResult<SourceStateRecord?>(record);
|
return Task.FromResult<SourceStateRecord?>(record);
|
||||||
}
|
}
|
||||||
|
|
||||||
public Task UpdateCursorAsync(string sourceName, MongoDB.Bson.BsonDocument cursor, DateTimeOffset completedAt, CancellationToken cancellationToken)
|
public Task UpdateCursorAsync(string sourceName, string cursorJson, DateTimeOffset completedAt, CancellationToken cancellationToken)
|
||||||
{
|
{
|
||||||
_states[sourceName] = new SourceStateRecord(sourceName, cursor.DeepClone(), completedAt);
|
_states[sourceName] = new SourceStateRecord(sourceName, cursorJson, completedAt);
|
||||||
return Task.CompletedTask;
|
return Task.CompletedTask;
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -174,6 +176,53 @@ namespace StellaOps.Concelier.Storage.Mongo
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
namespace StellaOps.Concelier.Storage.Mongo.Advisories
|
||||||
|
{
|
||||||
|
public interface IAdvisoryStore
|
||||||
|
{
|
||||||
|
Task UpsertAsync(Advisory advisory, CancellationToken cancellationToken);
|
||||||
|
Task<Advisory?> FindAsync(string advisoryKey, CancellationToken cancellationToken);
|
||||||
|
Task<IReadOnlyList<Advisory>> GetRecentAsync(int limit, CancellationToken cancellationToken);
|
||||||
|
IAsyncEnumerable<Advisory> StreamAsync(CancellationToken cancellationToken);
|
||||||
|
}
|
||||||
|
|
||||||
|
public sealed class InMemoryAdvisoryStore : IAdvisoryStore
|
||||||
|
{
|
||||||
|
private readonly ConcurrentDictionary<string, Advisory> _advisories = new(StringComparer.OrdinalIgnoreCase);
|
||||||
|
|
||||||
|
public Task UpsertAsync(Advisory advisory, CancellationToken cancellationToken)
|
||||||
|
{
|
||||||
|
_advisories[advisory.AdvisoryKey] = advisory;
|
||||||
|
return Task.CompletedTask;
|
||||||
|
}
|
||||||
|
|
||||||
|
public Task<Advisory?> FindAsync(string advisoryKey, CancellationToken cancellationToken)
|
||||||
|
{
|
||||||
|
_advisories.TryGetValue(advisoryKey, out var advisory);
|
||||||
|
return Task.FromResult<Advisory?>(advisory);
|
||||||
|
}
|
||||||
|
|
||||||
|
public Task<IReadOnlyList<Advisory>> GetRecentAsync(int limit, CancellationToken cancellationToken)
|
||||||
|
{
|
||||||
|
var result = _advisories.Values
|
||||||
|
.OrderByDescending(a => a.Modified ?? a.Published ?? DateTimeOffset.MinValue)
|
||||||
|
.Take(limit)
|
||||||
|
.ToArray();
|
||||||
|
return Task.FromResult<IReadOnlyList<Advisory>>(result);
|
||||||
|
}
|
||||||
|
|
||||||
|
public async IAsyncEnumerable<Advisory> StreamAsync([System.Runtime.CompilerServices.EnumeratorCancellation] CancellationToken cancellationToken)
|
||||||
|
{
|
||||||
|
foreach (var advisory in _advisories.Values.OrderBy(a => a.AdvisoryKey, StringComparer.OrdinalIgnoreCase))
|
||||||
|
{
|
||||||
|
cancellationToken.ThrowIfCancellationRequested();
|
||||||
|
yield return advisory;
|
||||||
|
await Task.Yield();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
namespace StellaOps.Concelier.Storage.Mongo.Aliases
|
namespace StellaOps.Concelier.Storage.Mongo.Aliases
|
||||||
{
|
{
|
||||||
public sealed record AliasRecord(string AdvisoryKey, string Scheme, string Value);
|
public sealed record AliasRecord(string AdvisoryKey, string Scheme, string Value);
|
||||||
@@ -192,13 +241,13 @@ namespace StellaOps.Concelier.Storage.Mongo.Aliases
|
|||||||
public Task<IReadOnlyList<AliasRecord>> GetByAdvisoryAsync(string advisoryKey, CancellationToken cancellationToken)
|
public Task<IReadOnlyList<AliasRecord>> GetByAdvisoryAsync(string advisoryKey, CancellationToken cancellationToken)
|
||||||
{
|
{
|
||||||
_byAdvisory.TryGetValue(advisoryKey, out var records);
|
_byAdvisory.TryGetValue(advisoryKey, out var records);
|
||||||
return Task.FromResult<IReadOnlyList<AliasRecord>>(records ?? Array.Empty<AliasRecord>());
|
return Task.FromResult<IReadOnlyList<AliasRecord>>(records ?? (IReadOnlyList<AliasRecord>)Array.Empty<AliasRecord>());
|
||||||
}
|
}
|
||||||
|
|
||||||
public Task<IReadOnlyList<AliasRecord>> GetByAliasAsync(string scheme, string value, CancellationToken cancellationToken)
|
public Task<IReadOnlyList<AliasRecord>> GetByAliasAsync(string scheme, string value, CancellationToken cancellationToken)
|
||||||
{
|
{
|
||||||
_byAlias.TryGetValue((scheme, value), out var records);
|
_byAlias.TryGetValue((scheme, value), out var records);
|
||||||
return Task.FromResult<IReadOnlyList<AliasRecord>>(records ?? Array.Empty<AliasRecord>());
|
return Task.FromResult<IReadOnlyList<AliasRecord>>(records ?? (IReadOnlyList<AliasRecord>)Array.Empty<AliasRecord>());
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -286,10 +335,10 @@ namespace StellaOps.Concelier.Storage.Mongo.Exporting
|
|||||||
id,
|
id,
|
||||||
cursor ?? digest,
|
cursor ?? digest,
|
||||||
digest,
|
digest,
|
||||||
lastDeltaDigest: null,
|
LastDeltaDigest: null,
|
||||||
baseExportId: resetBaseline ? exportId : null,
|
BaseExportId: resetBaseline ? exportId : null,
|
||||||
baseDigest: resetBaseline ? digest : null,
|
BaseDigest: resetBaseline ? digest : null,
|
||||||
targetRepository,
|
TargetRepository: targetRepository,
|
||||||
manifest,
|
manifest,
|
||||||
exporterVersion,
|
exporterVersion,
|
||||||
_timeProvider.GetUtcNow());
|
_timeProvider.GetUtcNow());
|
||||||
@@ -307,11 +356,11 @@ namespace StellaOps.Concelier.Storage.Mongo.Exporting
|
|||||||
var record = new ExportStateRecord(
|
var record = new ExportStateRecord(
|
||||||
id,
|
id,
|
||||||
cursor ?? deltaDigest,
|
cursor ?? deltaDigest,
|
||||||
lastFullDigest: null,
|
LastFullDigest: null,
|
||||||
lastDeltaDigest: deltaDigest,
|
LastDeltaDigest: deltaDigest,
|
||||||
baseExportId: null,
|
BaseExportId: null,
|
||||||
baseDigest: null,
|
BaseDigest: null,
|
||||||
targetRepository: null,
|
TargetRepository: null,
|
||||||
manifest,
|
manifest,
|
||||||
exporterVersion,
|
exporterVersion,
|
||||||
_timeProvider.GetUtcNow());
|
_timeProvider.GetUtcNow());
|
||||||
|
|||||||
@@ -0,0 +1,88 @@
|
|||||||
|
using System.Text.Json;
|
||||||
|
using StellaOps.Concelier.Storage.Mongo;
|
||||||
|
using StellaOps.Concelier.Storage.Postgres.Models;
|
||||||
|
using StellaOps.Concelier.Storage.Postgres.Repositories;
|
||||||
|
|
||||||
|
namespace StellaOps.Concelier.Storage.Postgres;
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// Postgres-backed implementation that satisfies the legacy IDocumentStore contract.
|
||||||
|
/// </summary>
|
||||||
|
public sealed class PostgresDocumentStore : IDocumentStore
|
||||||
|
{
|
||||||
|
private readonly IDocumentRepository _repository;
|
||||||
|
private readonly ISourceRepository _sourceRepository;
|
||||||
|
private readonly JsonSerializerOptions _json = new(JsonSerializerDefaults.Web);
|
||||||
|
|
||||||
|
public PostgresDocumentStore(IDocumentRepository repository, ISourceRepository sourceRepository)
|
||||||
|
{
|
||||||
|
_repository = repository ?? throw new ArgumentNullException(nameof(repository));
|
||||||
|
_sourceRepository = sourceRepository ?? throw new ArgumentNullException(nameof(sourceRepository));
|
||||||
|
}
|
||||||
|
|
||||||
|
public async Task<DocumentRecord?> FindAsync(Guid id, CancellationToken cancellationToken, MongoDB.Driver.IClientSessionHandle? session = null)
|
||||||
|
{
|
||||||
|
var row = await _repository.FindAsync(id, cancellationToken).ConfigureAwait(false);
|
||||||
|
return row is null ? null : Map(row);
|
||||||
|
}
|
||||||
|
|
||||||
|
public async Task<DocumentRecord?> FindBySourceAndUriAsync(string sourceName, string uri, CancellationToken cancellationToken, MongoDB.Driver.IClientSessionHandle? session = null)
|
||||||
|
{
|
||||||
|
var row = await _repository.FindBySourceAndUriAsync(sourceName, uri, cancellationToken).ConfigureAwait(false);
|
||||||
|
return row is null ? null : Map(row);
|
||||||
|
}
|
||||||
|
|
||||||
|
public async Task<DocumentRecord> UpsertAsync(DocumentRecord record, CancellationToken cancellationToken, MongoDB.Driver.IClientSessionHandle? session = null)
|
||||||
|
{
|
||||||
|
// Ensure source exists
|
||||||
|
var source = await _sourceRepository.GetByNameAsync(record.SourceName, cancellationToken).ConfigureAwait(false)
|
||||||
|
?? throw new InvalidOperationException($"Source '{record.SourceName}' not provisioned.");
|
||||||
|
|
||||||
|
var entity = new DocumentRecordEntity(
|
||||||
|
Id: record.Id == Guid.Empty ? Guid.NewGuid() : record.Id,
|
||||||
|
SourceId: source.Id,
|
||||||
|
SourceName: record.SourceName,
|
||||||
|
Uri: record.Uri,
|
||||||
|
Sha256: record.Sha256,
|
||||||
|
Status: record.Status,
|
||||||
|
ContentType: record.ContentType,
|
||||||
|
HeadersJson: record.Headers is null ? null : JsonSerializer.Serialize(record.Headers, _json),
|
||||||
|
MetadataJson: record.Metadata is null ? null : JsonSerializer.Serialize(record.Metadata, _json),
|
||||||
|
Etag: record.Etag,
|
||||||
|
LastModified: record.LastModified,
|
||||||
|
Payload: Array.Empty<byte>(), // payload handled via RawDocumentStorage; keep pointer zero-length here
|
||||||
|
CreatedAt: record.CreatedAt,
|
||||||
|
UpdatedAt: DateTimeOffset.UtcNow,
|
||||||
|
ExpiresAt: record.ExpiresAt);
|
||||||
|
|
||||||
|
var saved = await _repository.UpsertAsync(entity, cancellationToken).ConfigureAwait(false);
|
||||||
|
return Map(saved);
|
||||||
|
}
|
||||||
|
|
||||||
|
public async Task UpdateStatusAsync(Guid id, string status, CancellationToken cancellationToken, MongoDB.Driver.IClientSessionHandle? session = null)
|
||||||
|
{
|
||||||
|
await _repository.UpdateStatusAsync(id, status, cancellationToken).ConfigureAwait(false);
|
||||||
|
}
|
||||||
|
|
||||||
|
private DocumentRecord Map(DocumentRecordEntity row)
|
||||||
|
{
|
||||||
|
return new DocumentRecord(
|
||||||
|
row.Id,
|
||||||
|
row.SourceName,
|
||||||
|
row.Uri,
|
||||||
|
row.CreatedAt,
|
||||||
|
row.Sha256,
|
||||||
|
row.Status,
|
||||||
|
row.ContentType,
|
||||||
|
row.HeadersJson is null
|
||||||
|
? null
|
||||||
|
: JsonSerializer.Deserialize<Dictionary<string, string>>(row.HeadersJson, _json),
|
||||||
|
row.MetadataJson is null
|
||||||
|
? null
|
||||||
|
: JsonSerializer.Deserialize<Dictionary<string, string>>(row.MetadataJson, _json),
|
||||||
|
row.Etag,
|
||||||
|
row.LastModified,
|
||||||
|
PayloadId: null,
|
||||||
|
ExpiresAt: row.ExpiresAt);
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,23 @@
|
|||||||
|
-- Concelier Postgres Migration 004: Source documents and payload storage (Mongo replacement)
|
||||||
|
|
||||||
|
CREATE TABLE IF NOT EXISTS concelier.source_documents (
|
||||||
|
id UUID NOT NULL,
|
||||||
|
source_id UUID NOT NULL,
|
||||||
|
source_name TEXT NOT NULL,
|
||||||
|
uri TEXT NOT NULL,
|
||||||
|
sha256 TEXT NOT NULL,
|
||||||
|
status TEXT NOT NULL,
|
||||||
|
content_type TEXT,
|
||||||
|
headers_json JSONB,
|
||||||
|
metadata_json JSONB,
|
||||||
|
etag TEXT,
|
||||||
|
last_modified TIMESTAMPTZ,
|
||||||
|
payload BYTEA NOT NULL,
|
||||||
|
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
|
||||||
|
updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
|
||||||
|
expires_at TIMESTAMPTZ,
|
||||||
|
CONSTRAINT pk_source_documents PRIMARY KEY (source_name, uri)
|
||||||
|
);
|
||||||
|
|
||||||
|
CREATE INDEX IF NOT EXISTS idx_source_documents_source_id ON concelier.source_documents(source_id);
|
||||||
|
CREATE INDEX IF NOT EXISTS idx_source_documents_status ON concelier.source_documents(status);
|
||||||
@@ -0,0 +1,18 @@
|
|||||||
|
namespace StellaOps.Concelier.Storage.Postgres.Models;
|
||||||
|
|
||||||
|
public sealed record DocumentRecordEntity(
|
||||||
|
Guid Id,
|
||||||
|
Guid SourceId,
|
||||||
|
string SourceName,
|
||||||
|
string Uri,
|
||||||
|
string Sha256,
|
||||||
|
string Status,
|
||||||
|
string? ContentType,
|
||||||
|
string? HeadersJson,
|
||||||
|
string? MetadataJson,
|
||||||
|
string? Etag,
|
||||||
|
DateTimeOffset? LastModified,
|
||||||
|
byte[] Payload,
|
||||||
|
DateTimeOffset CreatedAt,
|
||||||
|
DateTimeOffset UpdatedAt,
|
||||||
|
DateTimeOffset? ExpiresAt);
|
||||||
@@ -0,0 +1,125 @@
|
|||||||
|
using System.Text.Json;
|
||||||
|
using Dapper;
|
||||||
|
using StellaOps.Concelier.Storage.Postgres.Models;
|
||||||
|
using StellaOps.Infrastructure.Postgres;
|
||||||
|
using StellaOps.Infrastructure.Postgres.Connections;
|
||||||
|
|
||||||
|
namespace StellaOps.Concelier.Storage.Postgres.Repositories;
|
||||||
|
|
||||||
|
public interface IDocumentRepository
|
||||||
|
{
|
||||||
|
Task<DocumentRecordEntity?> FindAsync(Guid id, CancellationToken cancellationToken);
|
||||||
|
Task<DocumentRecordEntity?> FindBySourceAndUriAsync(string sourceName, string uri, CancellationToken cancellationToken);
|
||||||
|
Task<DocumentRecordEntity> UpsertAsync(DocumentRecordEntity record, CancellationToken cancellationToken);
|
||||||
|
Task UpdateStatusAsync(Guid id, string status, CancellationToken cancellationToken);
|
||||||
|
}
|
||||||
|
|
||||||
|
public sealed class DocumentRepository : RepositoryBase<ConcelierDataSource>, IDocumentRepository
|
||||||
|
{
|
||||||
|
private readonly JsonSerializerOptions _json = new(JsonSerializerDefaults.Web);
|
||||||
|
|
||||||
|
public DocumentRepository(ConcelierDataSource dataSource, ILogger<DocumentRepository> logger)
|
||||||
|
: base(dataSource, logger)
|
||||||
|
{
|
||||||
|
}
|
||||||
|
|
||||||
|
public async Task<DocumentRecordEntity?> FindAsync(Guid id, CancellationToken cancellationToken)
|
||||||
|
{
|
||||||
|
const string sql = """
|
||||||
|
SELECT * FROM concelier.source_documents
|
||||||
|
WHERE id = @Id
|
||||||
|
LIMIT 1;
|
||||||
|
""";
|
||||||
|
|
||||||
|
await using var conn = await DataSource.OpenSystemConnectionAsync(cancellationToken);
|
||||||
|
var row = await conn.QuerySingleOrDefaultAsync(sql, new { Id = id });
|
||||||
|
return row is null ? null : Map(row);
|
||||||
|
}
|
||||||
|
|
||||||
|
public async Task<DocumentRecordEntity?> FindBySourceAndUriAsync(string sourceName, string uri, CancellationToken cancellationToken)
|
||||||
|
{
|
||||||
|
const string sql = """
|
||||||
|
SELECT * FROM concelier.source_documents
|
||||||
|
WHERE source_name = @SourceName AND uri = @Uri
|
||||||
|
LIMIT 1;
|
||||||
|
""";
|
||||||
|
await using var conn = await DataSource.OpenSystemConnectionAsync(cancellationToken);
|
||||||
|
var row = await conn.QuerySingleOrDefaultAsync(sql, new { SourceName = sourceName, Uri = uri });
|
||||||
|
return row is null ? null : Map(row);
|
||||||
|
}
|
||||||
|
|
||||||
|
public async Task<DocumentRecordEntity> UpsertAsync(DocumentRecordEntity record, CancellationToken cancellationToken)
|
||||||
|
{
|
||||||
|
const string sql = """
|
||||||
|
INSERT INTO concelier.source_documents (
|
||||||
|
id, source_id, source_name, uri, sha256, status, content_type,
|
||||||
|
headers_json, metadata_json, etag, last_modified, payload, created_at, updated_at, expires_at)
|
||||||
|
VALUES (
|
||||||
|
@Id, @SourceId, @SourceName, @Uri, @Sha256, @Status, @ContentType,
|
||||||
|
@HeadersJson, @MetadataJson, @Etag, @LastModified, @Payload, @CreatedAt, @UpdatedAt, @ExpiresAt)
|
||||||
|
ON CONFLICT (source_name, uri) DO UPDATE SET
|
||||||
|
sha256 = EXCLUDED.sha256,
|
||||||
|
status = EXCLUDED.status,
|
||||||
|
content_type = EXCLUDED.content_type,
|
||||||
|
headers_json = EXCLUDED.headers_json,
|
||||||
|
metadata_json = EXCLUDED.metadata_json,
|
||||||
|
etag = EXCLUDED.etag,
|
||||||
|
last_modified = EXCLUDED.last_modified,
|
||||||
|
payload = EXCLUDED.payload,
|
||||||
|
updated_at = EXCLUDED.updated_at,
|
||||||
|
expires_at = EXCLUDED.expires_at
|
||||||
|
RETURNING *;
|
||||||
|
""";
|
||||||
|
await using var conn = await DataSource.OpenSystemConnectionAsync(cancellationToken);
|
||||||
|
var row = await conn.QuerySingleAsync(sql, new
|
||||||
|
{
|
||||||
|
record.Id,
|
||||||
|
record.SourceId,
|
||||||
|
record.SourceName,
|
||||||
|
record.Uri,
|
||||||
|
record.Sha256,
|
||||||
|
record.Status,
|
||||||
|
record.ContentType,
|
||||||
|
record.HeadersJson,
|
||||||
|
record.MetadataJson,
|
||||||
|
record.Etag,
|
||||||
|
record.LastModified,
|
||||||
|
record.Payload,
|
||||||
|
record.CreatedAt,
|
||||||
|
record.UpdatedAt,
|
||||||
|
record.ExpiresAt
|
||||||
|
});
|
||||||
|
return Map(row);
|
||||||
|
}
|
||||||
|
|
||||||
|
public async Task UpdateStatusAsync(Guid id, string status, CancellationToken cancellationToken)
|
||||||
|
{
|
||||||
|
const string sql = """
|
||||||
|
UPDATE concelier.source_documents
|
||||||
|
SET status = @Status, updated_at = NOW()
|
||||||
|
WHERE id = @Id;
|
||||||
|
""";
|
||||||
|
await using var conn = await DataSource.OpenSystemConnectionAsync(cancellationToken);
|
||||||
|
await conn.ExecuteAsync(sql, new { Id = id, Status = status });
|
||||||
|
}
|
||||||
|
|
||||||
|
private DocumentRecordEntity Map(dynamic row)
|
||||||
|
{
|
||||||
|
return new DocumentRecordEntity(
|
||||||
|
row.id,
|
||||||
|
row.source_id,
|
||||||
|
row.source_name,
|
||||||
|
row.uri,
|
||||||
|
row.sha256,
|
||||||
|
row.status,
|
||||||
|
(string?)row.content_type,
|
||||||
|
(string?)row.headers_json,
|
||||||
|
(string?)row.metadata_json,
|
||||||
|
(string?)row.etag,
|
||||||
|
(DateTimeOffset?)row.last_modified,
|
||||||
|
(byte[])row.payload,
|
||||||
|
DateTime.SpecifyKind(row.created_at, DateTimeKind.Utc),
|
||||||
|
DateTime.SpecifyKind(row.updated_at, DateTimeKind.Utc),
|
||||||
|
row.expires_at is null ? null : DateTime.SpecifyKind(row.expires_at, DateTimeKind.Utc));
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -4,6 +4,7 @@ using StellaOps.Concelier.Storage.Postgres.Repositories;
|
|||||||
using StellaOps.Infrastructure.Postgres;
|
using StellaOps.Infrastructure.Postgres;
|
||||||
using StellaOps.Infrastructure.Postgres.Options;
|
using StellaOps.Infrastructure.Postgres.Options;
|
||||||
using StellaOps.Concelier.Core.Linksets;
|
using StellaOps.Concelier.Core.Linksets;
|
||||||
|
using StellaOps.Concelier.Storage.Mongo;
|
||||||
|
|
||||||
namespace StellaOps.Concelier.Storage.Postgres;
|
namespace StellaOps.Concelier.Storage.Postgres;
|
||||||
|
|
||||||
@@ -38,11 +39,13 @@ public static class ServiceCollectionExtensions
|
|||||||
services.AddScoped<IAdvisoryWeaknessRepository, AdvisoryWeaknessRepository>();
|
services.AddScoped<IAdvisoryWeaknessRepository, AdvisoryWeaknessRepository>();
|
||||||
services.AddScoped<IKevFlagRepository, KevFlagRepository>();
|
services.AddScoped<IKevFlagRepository, KevFlagRepository>();
|
||||||
services.AddScoped<ISourceStateRepository, SourceStateRepository>();
|
services.AddScoped<ISourceStateRepository, SourceStateRepository>();
|
||||||
|
services.AddScoped<IDocumentRepository, DocumentRepository>();
|
||||||
services.AddScoped<IFeedSnapshotRepository, FeedSnapshotRepository>();
|
services.AddScoped<IFeedSnapshotRepository, FeedSnapshotRepository>();
|
||||||
services.AddScoped<IAdvisorySnapshotRepository, AdvisorySnapshotRepository>();
|
services.AddScoped<IAdvisorySnapshotRepository, AdvisorySnapshotRepository>();
|
||||||
services.AddScoped<IMergeEventRepository, MergeEventRepository>();
|
services.AddScoped<IMergeEventRepository, MergeEventRepository>();
|
||||||
services.AddScoped<IAdvisoryLinksetStore, AdvisoryLinksetCacheRepository>();
|
services.AddScoped<IAdvisoryLinksetStore, AdvisoryLinksetCacheRepository>();
|
||||||
services.AddScoped<IAdvisoryLinksetLookup>(sp => sp.GetRequiredService<IAdvisoryLinksetStore>());
|
services.AddScoped<IAdvisoryLinksetLookup>(sp => sp.GetRequiredService<IAdvisoryLinksetStore>());
|
||||||
|
services.AddScoped<IDocumentStore, PostgresDocumentStore>();
|
||||||
|
|
||||||
return services;
|
return services;
|
||||||
}
|
}
|
||||||
@@ -71,11 +74,13 @@ public static class ServiceCollectionExtensions
|
|||||||
services.AddScoped<IAdvisoryWeaknessRepository, AdvisoryWeaknessRepository>();
|
services.AddScoped<IAdvisoryWeaknessRepository, AdvisoryWeaknessRepository>();
|
||||||
services.AddScoped<IKevFlagRepository, KevFlagRepository>();
|
services.AddScoped<IKevFlagRepository, KevFlagRepository>();
|
||||||
services.AddScoped<ISourceStateRepository, SourceStateRepository>();
|
services.AddScoped<ISourceStateRepository, SourceStateRepository>();
|
||||||
|
services.AddScoped<IDocumentRepository, DocumentRepository>();
|
||||||
services.AddScoped<IFeedSnapshotRepository, FeedSnapshotRepository>();
|
services.AddScoped<IFeedSnapshotRepository, FeedSnapshotRepository>();
|
||||||
services.AddScoped<IAdvisorySnapshotRepository, AdvisorySnapshotRepository>();
|
services.AddScoped<IAdvisorySnapshotRepository, AdvisorySnapshotRepository>();
|
||||||
services.AddScoped<IMergeEventRepository, MergeEventRepository>();
|
services.AddScoped<IMergeEventRepository, MergeEventRepository>();
|
||||||
services.AddScoped<IAdvisoryLinksetStore, AdvisoryLinksetCacheRepository>();
|
services.AddScoped<IAdvisoryLinksetStore, AdvisoryLinksetCacheRepository>();
|
||||||
services.AddScoped<IAdvisoryLinksetLookup>(sp => sp.GetRequiredService<IAdvisoryLinksetStore>());
|
services.AddScoped<IAdvisoryLinksetLookup>(sp => sp.GetRequiredService<IAdvisoryLinksetStore>());
|
||||||
|
services.AddScoped<IDocumentStore, PostgresDocumentStore>();
|
||||||
|
|
||||||
return services;
|
return services;
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -10,6 +10,11 @@
|
|||||||
<RootNamespace>StellaOps.Concelier.Storage.Postgres</RootNamespace>
|
<RootNamespace>StellaOps.Concelier.Storage.Postgres</RootNamespace>
|
||||||
</PropertyGroup>
|
</PropertyGroup>
|
||||||
|
|
||||||
|
<ItemGroup>
|
||||||
|
<PackageReference Include="Dapper" Version="2.1.35" />
|
||||||
|
<PackageReference Include="Microsoft.Extensions.Logging.Abstractions" Version="10.0.0" />
|
||||||
|
</ItemGroup>
|
||||||
|
|
||||||
<ItemGroup>
|
<ItemGroup>
|
||||||
<EmbeddedResource Include="Migrations\**\*.sql" LogicalName="%(RecursiveDir)%(Filename)%(Extension)" />
|
<EmbeddedResource Include="Migrations\**\*.sql" LogicalName="%(RecursiveDir)%(Filename)%(Extension)" />
|
||||||
</ItemGroup>
|
</ItemGroup>
|
||||||
@@ -25,6 +30,7 @@
|
|||||||
<ItemGroup>
|
<ItemGroup>
|
||||||
<ProjectReference Include="..\..\..\__Libraries\StellaOps.Infrastructure.Postgres\StellaOps.Infrastructure.Postgres.csproj" />
|
<ProjectReference Include="..\..\..\__Libraries\StellaOps.Infrastructure.Postgres\StellaOps.Infrastructure.Postgres.csproj" />
|
||||||
<ProjectReference Include="..\StellaOps.Concelier.Core\StellaOps.Concelier.Core.csproj" />
|
<ProjectReference Include="..\StellaOps.Concelier.Core\StellaOps.Concelier.Core.csproj" />
|
||||||
|
<ProjectReference Include="..\..\..\__Libraries\StellaOps.DependencyInjection\StellaOps.DependencyInjection.csproj" />
|
||||||
</ItemGroup>
|
</ItemGroup>
|
||||||
|
|
||||||
</Project>
|
</Project>
|
||||||
|
|||||||
@@ -6,36 +6,36 @@ using StellaOps.Concelier.Connector.Common.Html;
|
|||||||
using StellaOps.Concelier.Models;
|
using StellaOps.Concelier.Models;
|
||||||
using StellaOps.Concelier.Storage.Mongo.Documents;
|
using StellaOps.Concelier.Storage.Mongo.Documents;
|
||||||
using Xunit;
|
using Xunit;
|
||||||
|
|
||||||
namespace StellaOps.Concelier.Connector.Cccs.Tests.Internal;
|
namespace StellaOps.Concelier.Connector.Cccs.Tests.Internal;
|
||||||
|
|
||||||
public sealed class CccsMapperTests
|
public sealed class CccsMapperTests
|
||||||
{
|
{
|
||||||
[Fact]
|
[Fact]
|
||||||
public void Map_CreatesCanonicalAdvisory()
|
public void Map_CreatesCanonicalAdvisory()
|
||||||
{
|
{
|
||||||
var raw = CccsHtmlParserTests.LoadFixture<CccsRawAdvisoryDocument>("cccs-raw-advisory.json");
|
var raw = CccsHtmlParserTests.LoadFixture<CccsRawAdvisoryDocument>("cccs-raw-advisory.json");
|
||||||
var dto = new CccsHtmlParser(new HtmlContentSanitizer()).Parse(raw);
|
var dto = new CccsHtmlParser(new HtmlContentSanitizer()).Parse(raw);
|
||||||
var document = new DocumentRecord(
|
var document = new DocumentRecord(
|
||||||
Guid.NewGuid(),
|
Guid.NewGuid(),
|
||||||
CccsConnectorPlugin.SourceName,
|
CccsConnectorPlugin.SourceName,
|
||||||
dto.CanonicalUrl,
|
dto.CanonicalUrl,
|
||||||
DateTimeOffset.UtcNow,
|
DateTimeOffset.UtcNow,
|
||||||
"sha-test",
|
"sha-test",
|
||||||
DocumentStatuses.PendingMap,
|
DocumentStatuses.PendingMap,
|
||||||
"application/json",
|
"application/json",
|
||||||
Headers: null,
|
Headers: null,
|
||||||
Metadata: null,
|
Metadata: null,
|
||||||
Etag: null,
|
Etag: null,
|
||||||
LastModified: dto.Modified,
|
LastModified: dto.Modified,
|
||||||
GridFsId: null);
|
PayloadId: null);
|
||||||
|
|
||||||
var recordedAt = DateTimeOffset.Parse("2025-08-12T00:00:00Z");
|
var recordedAt = DateTimeOffset.Parse("2025-08-12T00:00:00Z");
|
||||||
var advisory = CccsMapper.Map(dto, document, recordedAt);
|
var advisory = CccsMapper.Map(dto, document, recordedAt);
|
||||||
|
|
||||||
advisory.AdvisoryKey.Should().Be("TEST-001");
|
advisory.AdvisoryKey.Should().Be("TEST-001");
|
||||||
advisory.Title.Should().Be(dto.Title);
|
advisory.Title.Should().Be(dto.Title);
|
||||||
advisory.Aliases.Should().Contain(new[] { "TEST-001", "CVE-2020-1234", "CVE-2021-9999" });
|
advisory.Aliases.Should().Contain(new[] { "TEST-001", "CVE-2020-1234", "CVE-2021-9999" });
|
||||||
advisory.References.Should().Contain(reference => reference.Url == dto.CanonicalUrl && reference.Kind == "details");
|
advisory.References.Should().Contain(reference => reference.Url == dto.CanonicalUrl && reference.Kind == "details");
|
||||||
advisory.References.Should().Contain(reference => reference.Url == "https://example.com/details");
|
advisory.References.Should().Contain(reference => reference.Url == "https://example.com/details");
|
||||||
advisory.AffectedPackages.Should().HaveCount(2);
|
advisory.AffectedPackages.Should().HaveCount(2);
|
||||||
|
|||||||
@@ -1,118 +1,118 @@
|
|||||||
using System;
|
using System;
|
||||||
using System.Globalization;
|
using System.Globalization;
|
||||||
using MongoDB.Bson;
|
using MongoDB.Bson;
|
||||||
using StellaOps.Concelier.Models;
|
using StellaOps.Concelier.Models;
|
||||||
using StellaOps.Concelier.Connector.CertCc.Internal;
|
using StellaOps.Concelier.Connector.CertCc.Internal;
|
||||||
using StellaOps.Concelier.Storage.Mongo.Documents;
|
using StellaOps.Concelier.Storage.Mongo.Documents;
|
||||||
using StellaOps.Concelier.Storage.Mongo.Dtos;
|
using StellaOps.Concelier.Storage.Mongo.Dtos;
|
||||||
using Xunit;
|
using Xunit;
|
||||||
|
|
||||||
namespace StellaOps.Concelier.Connector.CertCc.Tests.Internal;
|
namespace StellaOps.Concelier.Connector.CertCc.Tests.Internal;
|
||||||
|
|
||||||
public sealed class CertCcMapperTests
|
public sealed class CertCcMapperTests
|
||||||
{
|
{
|
||||||
private static readonly DateTimeOffset PublishedAt = DateTimeOffset.Parse("2025-10-03T11:35:31Z", CultureInfo.InvariantCulture);
|
private static readonly DateTimeOffset PublishedAt = DateTimeOffset.Parse("2025-10-03T11:35:31Z", CultureInfo.InvariantCulture);
|
||||||
|
|
||||||
[Fact]
|
[Fact]
|
||||||
public void Map_ProducesCanonicalAdvisoryWithVendorPrimitives()
|
public void Map_ProducesCanonicalAdvisoryWithVendorPrimitives()
|
||||||
{
|
{
|
||||||
const string vendorStatement =
|
const string vendorStatement =
|
||||||
"The issue is confirmed, and here is the patch list\n\n" +
|
"The issue is confirmed, and here is the patch list\n\n" +
|
||||||
"V3912/V3910/V2962/V1000B\t4.4.3.6/4.4.5.1\n" +
|
"V3912/V3910/V2962/V1000B\t4.4.3.6/4.4.5.1\n" +
|
||||||
"V2927/V2865/V2866\t4.5.1\n" +
|
"V2927/V2865/V2866\t4.5.1\n" +
|
||||||
"V2765/V2766/V2763/V2135\t4.5.1";
|
"V2765/V2766/V2763/V2135\t4.5.1";
|
||||||
|
|
||||||
var vendor = new CertCcVendorDto(
|
var vendor = new CertCcVendorDto(
|
||||||
"DrayTek Corporation",
|
"DrayTek Corporation",
|
||||||
ContactDate: PublishedAt.AddDays(-10),
|
ContactDate: PublishedAt.AddDays(-10),
|
||||||
StatementDate: PublishedAt.AddDays(-5),
|
StatementDate: PublishedAt.AddDays(-5),
|
||||||
Updated: PublishedAt,
|
Updated: PublishedAt,
|
||||||
Statement: vendorStatement,
|
Statement: vendorStatement,
|
||||||
Addendum: null,
|
Addendum: null,
|
||||||
References: new[] { "https://www.draytek.com/support/resources?type=version" });
|
References: new[] { "https://www.draytek.com/support/resources?type=version" });
|
||||||
|
|
||||||
var vendorStatus = new CertCcVendorStatusDto(
|
var vendorStatus = new CertCcVendorStatusDto(
|
||||||
Vendor: "DrayTek Corporation",
|
Vendor: "DrayTek Corporation",
|
||||||
CveId: "CVE-2025-10547",
|
CveId: "CVE-2025-10547",
|
||||||
Status: "Affected",
|
Status: "Affected",
|
||||||
Statement: null,
|
Statement: null,
|
||||||
References: Array.Empty<string>(),
|
References: Array.Empty<string>(),
|
||||||
DateAdded: PublishedAt,
|
DateAdded: PublishedAt,
|
||||||
DateUpdated: PublishedAt);
|
DateUpdated: PublishedAt);
|
||||||
|
|
||||||
var vulnerability = new CertCcVulnerabilityDto(
|
var vulnerability = new CertCcVulnerabilityDto(
|
||||||
CveId: "CVE-2025-10547",
|
CveId: "CVE-2025-10547",
|
||||||
Description: null,
|
Description: null,
|
||||||
DateAdded: PublishedAt,
|
DateAdded: PublishedAt,
|
||||||
DateUpdated: PublishedAt);
|
DateUpdated: PublishedAt);
|
||||||
|
|
||||||
var metadata = new CertCcNoteMetadata(
|
var metadata = new CertCcNoteMetadata(
|
||||||
VuId: "VU#294418",
|
VuId: "VU#294418",
|
||||||
IdNumber: "294418",
|
IdNumber: "294418",
|
||||||
Title: "Vigor routers running DrayOS RCE via EasyVPN",
|
Title: "Vigor routers running DrayOS RCE via EasyVPN",
|
||||||
Overview: "Overview",
|
Overview: "Overview",
|
||||||
Summary: "Summary",
|
Summary: "Summary",
|
||||||
Published: PublishedAt,
|
Published: PublishedAt,
|
||||||
Updated: PublishedAt.AddMinutes(5),
|
Updated: PublishedAt.AddMinutes(5),
|
||||||
Created: PublishedAt,
|
Created: PublishedAt,
|
||||||
Revision: 2,
|
Revision: 2,
|
||||||
CveIds: new[] { "CVE-2025-10547" },
|
CveIds: new[] { "CVE-2025-10547" },
|
||||||
PublicUrls: new[]
|
PublicUrls: new[]
|
||||||
{
|
{
|
||||||
"https://www.draytek.com/about/security-advisory/use-of-uninitialized-variable-vulnerabilities/",
|
"https://www.draytek.com/about/security-advisory/use-of-uninitialized-variable-vulnerabilities/",
|
||||||
"https://www.draytek.com/support/resources?type=version"
|
"https://www.draytek.com/support/resources?type=version"
|
||||||
},
|
},
|
||||||
PrimaryUrl: "https://www.kb.cert.org/vuls/id/294418/");
|
PrimaryUrl: "https://www.kb.cert.org/vuls/id/294418/");
|
||||||
|
|
||||||
var dto = new CertCcNoteDto(
|
var dto = new CertCcNoteDto(
|
||||||
metadata,
|
metadata,
|
||||||
Vendors: new[] { vendor },
|
Vendors: new[] { vendor },
|
||||||
VendorStatuses: new[] { vendorStatus },
|
VendorStatuses: new[] { vendorStatus },
|
||||||
Vulnerabilities: new[] { vulnerability });
|
Vulnerabilities: new[] { vulnerability });
|
||||||
|
|
||||||
var document = new DocumentRecord(
|
var document = new DocumentRecord(
|
||||||
Guid.NewGuid(),
|
Guid.NewGuid(),
|
||||||
"cert-cc",
|
"cert-cc",
|
||||||
"https://www.kb.cert.org/vuls/id/294418/",
|
"https://www.kb.cert.org/vuls/id/294418/",
|
||||||
PublishedAt,
|
PublishedAt,
|
||||||
Sha256: new string('0', 64),
|
Sha256: new string('0', 64),
|
||||||
Status: "pending-map",
|
Status: "pending-map",
|
||||||
ContentType: "application/json",
|
ContentType: "application/json",
|
||||||
Headers: null,
|
Headers: null,
|
||||||
Metadata: null,
|
Metadata: null,
|
||||||
Etag: null,
|
Etag: null,
|
||||||
LastModified: PublishedAt,
|
LastModified: PublishedAt,
|
||||||
GridFsId: null);
|
PayloadId: null);
|
||||||
|
|
||||||
var dtoRecord = new DtoRecord(
|
var dtoRecord = new DtoRecord(
|
||||||
Id: Guid.NewGuid(),
|
Id: Guid.NewGuid(),
|
||||||
DocumentId: document.Id,
|
DocumentId: document.Id,
|
||||||
SourceName: "cert-cc",
|
SourceName: "cert-cc",
|
||||||
SchemaVersion: "certcc.vince.note.v1",
|
SchemaVersion: "certcc.vince.note.v1",
|
||||||
Payload: new BsonDocument(),
|
Payload: new BsonDocument(),
|
||||||
ValidatedAt: PublishedAt.AddMinutes(1));
|
ValidatedAt: PublishedAt.AddMinutes(1));
|
||||||
|
|
||||||
var advisory = CertCcMapper.Map(dto, document, dtoRecord, "cert-cc");
|
var advisory = CertCcMapper.Map(dto, document, dtoRecord, "cert-cc");
|
||||||
|
|
||||||
Assert.Equal("certcc/vu-294418", advisory.AdvisoryKey);
|
Assert.Equal("certcc/vu-294418", advisory.AdvisoryKey);
|
||||||
Assert.Contains("VU#294418", advisory.Aliases);
|
Assert.Contains("VU#294418", advisory.Aliases);
|
||||||
Assert.Contains("CVE-2025-10547", advisory.Aliases);
|
Assert.Contains("CVE-2025-10547", advisory.Aliases);
|
||||||
Assert.Equal("en", advisory.Language);
|
Assert.Equal("en", advisory.Language);
|
||||||
Assert.Equal(PublishedAt, advisory.Published);
|
Assert.Equal(PublishedAt, advisory.Published);
|
||||||
|
|
||||||
Assert.Contains(advisory.References, reference => reference.Url.Contains("/vuls/id/294418", StringComparison.OrdinalIgnoreCase));
|
Assert.Contains(advisory.References, reference => reference.Url.Contains("/vuls/id/294418", StringComparison.OrdinalIgnoreCase));
|
||||||
|
|
||||||
var affected = Assert.Single(advisory.AffectedPackages);
|
var affected = Assert.Single(advisory.AffectedPackages);
|
||||||
Assert.Equal("vendor", affected.Type);
|
Assert.Equal("vendor", affected.Type);
|
||||||
Assert.Equal("DrayTek Corporation", affected.Identifier);
|
Assert.Equal("DrayTek Corporation", affected.Identifier);
|
||||||
Assert.Contains(affected.Statuses, status => status.Status == AffectedPackageStatusCatalog.Affected);
|
Assert.Contains(affected.Statuses, status => status.Status == AffectedPackageStatusCatalog.Affected);
|
||||||
|
|
||||||
var range = Assert.Single(affected.VersionRanges);
|
var range = Assert.Single(affected.VersionRanges);
|
||||||
Assert.NotNull(range.Primitives);
|
Assert.NotNull(range.Primitives);
|
||||||
Assert.NotNull(range.Primitives!.VendorExtensions);
|
Assert.NotNull(range.Primitives!.VendorExtensions);
|
||||||
Assert.Contains(range.Primitives.VendorExtensions!, kvp => kvp.Key == "certcc.vendor.patches");
|
Assert.Contains(range.Primitives.VendorExtensions!, kvp => kvp.Key == "certcc.vendor.patches");
|
||||||
|
|
||||||
Assert.NotEmpty(affected.NormalizedVersions);
|
Assert.NotEmpty(affected.NormalizedVersions);
|
||||||
Assert.Contains(affected.NormalizedVersions, rule => rule.Scheme == "certcc.vendor" && rule.Value == "4.5.1");
|
Assert.Contains(affected.NormalizedVersions, rule => rule.Scheme == "certcc.vendor" && rule.Value == "4.5.1");
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -93,7 +93,7 @@ public sealed class SourceStateSeedProcessorTests : IAsyncLifetime
|
|||||||
Assert.Equal(documentId, storedDocument!.Id);
|
Assert.Equal(documentId, storedDocument!.Id);
|
||||||
Assert.Equal("application/json", storedDocument.ContentType);
|
Assert.Equal("application/json", storedDocument.ContentType);
|
||||||
Assert.Equal(DocumentStatuses.PendingParse, storedDocument.Status);
|
Assert.Equal(DocumentStatuses.PendingParse, storedDocument.Status);
|
||||||
Assert.NotNull(storedDocument.GridFsId);
|
Assert.NotNull(storedDocument.PayloadId);
|
||||||
Assert.NotNull(storedDocument.Headers);
|
Assert.NotNull(storedDocument.Headers);
|
||||||
Assert.Equal("true", storedDocument.Headers!["X-Test"]);
|
Assert.Equal("true", storedDocument.Headers!["X-Test"]);
|
||||||
Assert.NotNull(storedDocument.Metadata);
|
Assert.NotNull(storedDocument.Metadata);
|
||||||
@@ -153,7 +153,7 @@ public sealed class SourceStateSeedProcessorTests : IAsyncLifetime
|
|||||||
CancellationToken.None);
|
CancellationToken.None);
|
||||||
|
|
||||||
Assert.NotNull(existingRecord);
|
Assert.NotNull(existingRecord);
|
||||||
var previousGridId = existingRecord!.GridFsId;
|
var previousGridId = existingRecord!.PayloadId;
|
||||||
Assert.NotNull(previousGridId);
|
Assert.NotNull(previousGridId);
|
||||||
|
|
||||||
var filesCollection = _database.GetCollection<BsonDocument>("documents.files");
|
var filesCollection = _database.GetCollection<BsonDocument>("documents.files");
|
||||||
@@ -189,8 +189,8 @@ public sealed class SourceStateSeedProcessorTests : IAsyncLifetime
|
|||||||
|
|
||||||
Assert.NotNull(refreshedRecord);
|
Assert.NotNull(refreshedRecord);
|
||||||
Assert.Equal(documentId, refreshedRecord!.Id);
|
Assert.Equal(documentId, refreshedRecord!.Id);
|
||||||
Assert.NotNull(refreshedRecord.GridFsId);
|
Assert.NotNull(refreshedRecord.PayloadId);
|
||||||
Assert.NotEqual(previousGridId, refreshedRecord.GridFsId);
|
Assert.NotEqual(previousGridId, refreshedRecord.PayloadId);
|
||||||
|
|
||||||
var files = await filesCollection.Find(FilterDefinition<BsonDocument>.Empty).ToListAsync();
|
var files = await filesCollection.Find(FilterDefinition<BsonDocument>.Empty).ToListAsync();
|
||||||
Assert.Single(files);
|
Assert.Single(files);
|
||||||
|
|||||||
@@ -1,82 +1,82 @@
|
|||||||
using System;
|
using System;
|
||||||
using Xunit;
|
using Xunit;
|
||||||
using StellaOps.Concelier.Models;
|
using StellaOps.Concelier.Models;
|
||||||
using StellaOps.Concelier.Connector.Distro.Debian;
|
using StellaOps.Concelier.Connector.Distro.Debian;
|
||||||
using StellaOps.Concelier.Connector.Distro.Debian.Internal;
|
using StellaOps.Concelier.Connector.Distro.Debian.Internal;
|
||||||
using StellaOps.Concelier.Storage.Mongo.Documents;
|
using StellaOps.Concelier.Storage.Mongo.Documents;
|
||||||
|
|
||||||
namespace StellaOps.Concelier.Connector.Distro.Debian.Tests;
|
namespace StellaOps.Concelier.Connector.Distro.Debian.Tests;
|
||||||
|
|
||||||
public sealed class DebianMapperTests
|
public sealed class DebianMapperTests
|
||||||
{
|
{
|
||||||
[Fact]
|
[Fact]
|
||||||
public void Map_BuildsRangePrimitives_ForResolvedPackage()
|
public void Map_BuildsRangePrimitives_ForResolvedPackage()
|
||||||
{
|
{
|
||||||
var dto = new DebianAdvisoryDto(
|
var dto = new DebianAdvisoryDto(
|
||||||
AdvisoryId: "DSA-2024-123",
|
AdvisoryId: "DSA-2024-123",
|
||||||
SourcePackage: "openssl",
|
SourcePackage: "openssl",
|
||||||
Title: "Openssl security update",
|
Title: "Openssl security update",
|
||||||
Description: "Fixes multiple issues.",
|
Description: "Fixes multiple issues.",
|
||||||
CveIds: new[] { "CVE-2024-1000", "CVE-2024-1001" },
|
CveIds: new[] { "CVE-2024-1000", "CVE-2024-1001" },
|
||||||
Packages: new[]
|
Packages: new[]
|
||||||
{
|
{
|
||||||
new DebianPackageStateDto(
|
new DebianPackageStateDto(
|
||||||
Package: "openssl",
|
Package: "openssl",
|
||||||
Release: "bullseye",
|
Release: "bullseye",
|
||||||
Status: "resolved",
|
Status: "resolved",
|
||||||
IntroducedVersion: "1:1.1.1n-0+deb11u2",
|
IntroducedVersion: "1:1.1.1n-0+deb11u2",
|
||||||
FixedVersion: "1:1.1.1n-0+deb11u5",
|
FixedVersion: "1:1.1.1n-0+deb11u5",
|
||||||
LastAffectedVersion: null,
|
LastAffectedVersion: null,
|
||||||
Published: new DateTimeOffset(2024, 9, 1, 0, 0, 0, TimeSpan.Zero)),
|
Published: new DateTimeOffset(2024, 9, 1, 0, 0, 0, TimeSpan.Zero)),
|
||||||
new DebianPackageStateDto(
|
new DebianPackageStateDto(
|
||||||
Package: "openssl",
|
Package: "openssl",
|
||||||
Release: "bookworm",
|
Release: "bookworm",
|
||||||
Status: "open",
|
Status: "open",
|
||||||
IntroducedVersion: null,
|
IntroducedVersion: null,
|
||||||
FixedVersion: null,
|
FixedVersion: null,
|
||||||
LastAffectedVersion: null,
|
LastAffectedVersion: null,
|
||||||
Published: null)
|
Published: null)
|
||||||
},
|
},
|
||||||
References: new[]
|
References: new[]
|
||||||
{
|
{
|
||||||
new DebianReferenceDto(
|
new DebianReferenceDto(
|
||||||
Url: "https://security-tracker.debian.org/tracker/DSA-2024-123",
|
Url: "https://security-tracker.debian.org/tracker/DSA-2024-123",
|
||||||
Kind: "advisory",
|
Kind: "advisory",
|
||||||
Title: "Debian Security Advisory 2024-123"),
|
Title: "Debian Security Advisory 2024-123"),
|
||||||
});
|
});
|
||||||
|
|
||||||
var document = new DocumentRecord(
|
var document = new DocumentRecord(
|
||||||
Id: Guid.NewGuid(),
|
Id: Guid.NewGuid(),
|
||||||
SourceName: DebianConnectorPlugin.SourceName,
|
SourceName: DebianConnectorPlugin.SourceName,
|
||||||
Uri: "https://security-tracker.debian.org/tracker/DSA-2024-123",
|
Uri: "https://security-tracker.debian.org/tracker/DSA-2024-123",
|
||||||
FetchedAt: new DateTimeOffset(2024, 9, 1, 1, 0, 0, TimeSpan.Zero),
|
FetchedAt: new DateTimeOffset(2024, 9, 1, 1, 0, 0, TimeSpan.Zero),
|
||||||
Sha256: "sha",
|
Sha256: "sha",
|
||||||
Status: "Fetched",
|
Status: "Fetched",
|
||||||
ContentType: "application/json",
|
ContentType: "application/json",
|
||||||
Headers: null,
|
Headers: null,
|
||||||
Metadata: null,
|
Metadata: null,
|
||||||
Etag: null,
|
Etag: null,
|
||||||
LastModified: null,
|
LastModified: null,
|
||||||
GridFsId: null);
|
PayloadId: null);
|
||||||
|
|
||||||
Advisory advisory = DebianMapper.Map(dto, document, new DateTimeOffset(2024, 9, 1, 2, 0, 0, TimeSpan.Zero));
|
Advisory advisory = DebianMapper.Map(dto, document, new DateTimeOffset(2024, 9, 1, 2, 0, 0, TimeSpan.Zero));
|
||||||
|
|
||||||
Assert.Equal("DSA-2024-123", advisory.AdvisoryKey);
|
Assert.Equal("DSA-2024-123", advisory.AdvisoryKey);
|
||||||
Assert.Contains("CVE-2024-1000", advisory.Aliases);
|
Assert.Contains("CVE-2024-1000", advisory.Aliases);
|
||||||
Assert.Contains("CVE-2024-1001", advisory.Aliases);
|
Assert.Contains("CVE-2024-1001", advisory.Aliases);
|
||||||
|
|
||||||
var resolvedPackage = Assert.Single(advisory.AffectedPackages, p => p.Platform == "bullseye");
|
var resolvedPackage = Assert.Single(advisory.AffectedPackages, p => p.Platform == "bullseye");
|
||||||
var range = Assert.Single(resolvedPackage.VersionRanges);
|
var range = Assert.Single(resolvedPackage.VersionRanges);
|
||||||
Assert.Equal("evr", range.RangeKind);
|
Assert.Equal("evr", range.RangeKind);
|
||||||
Assert.Equal("1:1.1.1n-0+deb11u2", range.IntroducedVersion);
|
Assert.Equal("1:1.1.1n-0+deb11u2", range.IntroducedVersion);
|
||||||
Assert.Equal("1:1.1.1n-0+deb11u5", range.FixedVersion);
|
Assert.Equal("1:1.1.1n-0+deb11u5", range.FixedVersion);
|
||||||
Assert.NotNull(range.Primitives);
|
Assert.NotNull(range.Primitives);
|
||||||
var evr = range.Primitives!.Evr;
|
var evr = range.Primitives!.Evr;
|
||||||
Assert.NotNull(evr);
|
Assert.NotNull(evr);
|
||||||
Assert.NotNull(evr!.Introduced);
|
Assert.NotNull(evr!.Introduced);
|
||||||
Assert.Equal(1, evr.Introduced!.Epoch);
|
Assert.Equal(1, evr.Introduced!.Epoch);
|
||||||
Assert.Equal("1.1.1n", evr.Introduced.UpstreamVersion);
|
Assert.Equal("1.1.1n", evr.Introduced.UpstreamVersion);
|
||||||
Assert.Equal("0+deb11u2", evr.Introduced.Revision);
|
Assert.Equal("0+deb11u2", evr.Introduced.Revision);
|
||||||
Assert.NotNull(evr.Fixed);
|
Assert.NotNull(evr.Fixed);
|
||||||
Assert.Equal(1, evr.Fixed!.Epoch);
|
Assert.Equal(1, evr.Fixed!.Epoch);
|
||||||
Assert.Equal("1.1.1n", evr.Fixed.UpstreamVersion);
|
Assert.Equal("1.1.1n", evr.Fixed.UpstreamVersion);
|
||||||
@@ -94,5 +94,5 @@ public sealed class DebianMapperTests
|
|||||||
var openPackage = Assert.Single(advisory.AffectedPackages, p => p.Platform == "bookworm");
|
var openPackage = Assert.Single(advisory.AffectedPackages, p => p.Platform == "bookworm");
|
||||||
Assert.Empty(openPackage.VersionRanges);
|
Assert.Empty(openPackage.VersionRanges);
|
||||||
Assert.Empty(openPackage.NormalizedVersions);
|
Assert.Empty(openPackage.NormalizedVersions);
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
File diff suppressed because it is too large
Load Diff
@@ -1,47 +1,47 @@
|
|||||||
using System;
|
using System;
|
||||||
using System.Collections.Generic;
|
using System.Collections.Generic;
|
||||||
using System.IO;
|
using System.IO;
|
||||||
using MongoDB.Bson;
|
using MongoDB.Bson;
|
||||||
using StellaOps.Concelier.Models;
|
using StellaOps.Concelier.Models;
|
||||||
using StellaOps.Concelier.Connector.Common;
|
using StellaOps.Concelier.Connector.Common;
|
||||||
using StellaOps.Concelier.Connector.Distro.Suse;
|
using StellaOps.Concelier.Connector.Distro.Suse;
|
||||||
using StellaOps.Concelier.Connector.Distro.Suse.Internal;
|
using StellaOps.Concelier.Connector.Distro.Suse.Internal;
|
||||||
using StellaOps.Concelier.Storage.Mongo.Documents;
|
using StellaOps.Concelier.Storage.Mongo.Documents;
|
||||||
using Xunit;
|
using Xunit;
|
||||||
|
|
||||||
namespace StellaOps.Concelier.Connector.Distro.Suse.Tests;
|
namespace StellaOps.Concelier.Connector.Distro.Suse.Tests;
|
||||||
|
|
||||||
public sealed class SuseMapperTests
|
public sealed class SuseMapperTests
|
||||||
{
|
{
|
||||||
[Fact]
|
[Fact]
|
||||||
public void Map_BuildsNevraRangePrimitives()
|
public void Map_BuildsNevraRangePrimitives()
|
||||||
{
|
{
|
||||||
var json = File.ReadAllText(Path.Combine(AppContext.BaseDirectory, "Source", "Distro", "Suse", "Fixtures", "suse-su-2025_0001-1.json"));
|
var json = File.ReadAllText(Path.Combine(AppContext.BaseDirectory, "Source", "Distro", "Suse", "Fixtures", "suse-su-2025_0001-1.json"));
|
||||||
var dto = SuseCsafParser.Parse(json);
|
var dto = SuseCsafParser.Parse(json);
|
||||||
|
|
||||||
var document = new DocumentRecord(
|
var document = new DocumentRecord(
|
||||||
Guid.NewGuid(),
|
Guid.NewGuid(),
|
||||||
SuseConnectorPlugin.SourceName,
|
SuseConnectorPlugin.SourceName,
|
||||||
"https://ftp.suse.com/pub/projects/security/csaf/suse-su-2025_0001-1.json",
|
"https://ftp.suse.com/pub/projects/security/csaf/suse-su-2025_0001-1.json",
|
||||||
DateTimeOffset.UtcNow,
|
DateTimeOffset.UtcNow,
|
||||||
"sha256",
|
"sha256",
|
||||||
DocumentStatuses.PendingParse,
|
DocumentStatuses.PendingParse,
|
||||||
"application/json",
|
"application/json",
|
||||||
Headers: null,
|
Headers: null,
|
||||||
Metadata: new Dictionary<string, string>(StringComparer.Ordinal)
|
Metadata: new Dictionary<string, string>(StringComparer.Ordinal)
|
||||||
{
|
{
|
||||||
["suse.id"] = dto.AdvisoryId
|
["suse.id"] = dto.AdvisoryId
|
||||||
},
|
},
|
||||||
Etag: "adv-1",
|
Etag: "adv-1",
|
||||||
LastModified: DateTimeOffset.UtcNow,
|
LastModified: DateTimeOffset.UtcNow,
|
||||||
GridFsId: ObjectId.Empty);
|
PayloadId: ObjectId.Empty);
|
||||||
|
|
||||||
var mapped = SuseMapper.Map(dto, document, DateTimeOffset.UtcNow);
|
var mapped = SuseMapper.Map(dto, document, DateTimeOffset.UtcNow);
|
||||||
|
|
||||||
Assert.Equal(dto.AdvisoryId, mapped.AdvisoryKey);
|
Assert.Equal(dto.AdvisoryId, mapped.AdvisoryKey);
|
||||||
var package = Assert.Single(mapped.AffectedPackages);
|
var package = Assert.Single(mapped.AffectedPackages);
|
||||||
Assert.Equal(AffectedPackageTypes.Rpm, package.Type);
|
Assert.Equal(AffectedPackageTypes.Rpm, package.Type);
|
||||||
var range = Assert.Single(package.VersionRanges);
|
var range = Assert.Single(package.VersionRanges);
|
||||||
Assert.Equal("nevra", range.RangeKind);
|
Assert.Equal("nevra", range.RangeKind);
|
||||||
Assert.NotNull(range.Primitives);
|
Assert.NotNull(range.Primitives);
|
||||||
Assert.NotNull(range.Primitives!.Nevra);
|
Assert.NotNull(range.Primitives!.Nevra);
|
||||||
|
|||||||
@@ -1,94 +1,94 @@
|
|||||||
using StellaOps.Concelier.Models;
|
using StellaOps.Concelier.Models;
|
||||||
using StellaOps.Concelier.Connector.Ghsa.Internal;
|
using StellaOps.Concelier.Connector.Ghsa.Internal;
|
||||||
using StellaOps.Concelier.Storage.Mongo.Documents;
|
using StellaOps.Concelier.Storage.Mongo.Documents;
|
||||||
|
|
||||||
namespace StellaOps.Concelier.Connector.Ghsa.Tests;
|
namespace StellaOps.Concelier.Connector.Ghsa.Tests;
|
||||||
|
|
||||||
public sealed class GhsaConflictFixtureTests
|
public sealed class GhsaConflictFixtureTests
|
||||||
{
|
{
|
||||||
[Fact]
|
[Fact]
|
||||||
public void ConflictFixture_MatchesSnapshot()
|
public void ConflictFixture_MatchesSnapshot()
|
||||||
{
|
{
|
||||||
var recordedAt = new DateTimeOffset(2025, 3, 4, 8, 30, 0, TimeSpan.Zero);
|
var recordedAt = new DateTimeOffset(2025, 3, 4, 8, 30, 0, TimeSpan.Zero);
|
||||||
var document = new DocumentRecord(
|
var document = new DocumentRecord(
|
||||||
Id: Guid.Parse("2f5c4d67-fcac-4ec9-a8d4-8a9c5a6d0fc9"),
|
Id: Guid.Parse("2f5c4d67-fcac-4ec9-a8d4-8a9c5a6d0fc9"),
|
||||||
SourceName: GhsaConnectorPlugin.SourceName,
|
SourceName: GhsaConnectorPlugin.SourceName,
|
||||||
Uri: "https://github.com/advisories/GHSA-qqqq-wwww-eeee",
|
Uri: "https://github.com/advisories/GHSA-qqqq-wwww-eeee",
|
||||||
FetchedAt: new DateTimeOffset(2025, 3, 3, 18, 0, 0, TimeSpan.Zero),
|
FetchedAt: new DateTimeOffset(2025, 3, 3, 18, 0, 0, TimeSpan.Zero),
|
||||||
Sha256: "sha256-ghsa-conflict-fixture",
|
Sha256: "sha256-ghsa-conflict-fixture",
|
||||||
Status: "completed",
|
Status: "completed",
|
||||||
ContentType: "application/json",
|
ContentType: "application/json",
|
||||||
Headers: null,
|
Headers: null,
|
||||||
Metadata: null,
|
Metadata: null,
|
||||||
Etag: "\"etag-ghsa-conflict\"",
|
Etag: "\"etag-ghsa-conflict\"",
|
||||||
LastModified: new DateTimeOffset(2025, 3, 3, 18, 0, 0, TimeSpan.Zero),
|
LastModified: new DateTimeOffset(2025, 3, 3, 18, 0, 0, TimeSpan.Zero),
|
||||||
GridFsId: null);
|
PayloadId: null);
|
||||||
|
|
||||||
var dto = new GhsaRecordDto
|
var dto = new GhsaRecordDto
|
||||||
{
|
{
|
||||||
GhsaId = "GHSA-qqqq-wwww-eeee",
|
GhsaId = "GHSA-qqqq-wwww-eeee",
|
||||||
Summary = "Container escape in conflict-package",
|
Summary = "Container escape in conflict-package",
|
||||||
Description = "Container escape vulnerability allowing privilege escalation in conflict-package.",
|
Description = "Container escape vulnerability allowing privilege escalation in conflict-package.",
|
||||||
Severity = "HIGH",
|
Severity = "HIGH",
|
||||||
PublishedAt = new DateTimeOffset(2025, 2, 25, 0, 0, 0, TimeSpan.Zero),
|
PublishedAt = new DateTimeOffset(2025, 2, 25, 0, 0, 0, TimeSpan.Zero),
|
||||||
UpdatedAt = new DateTimeOffset(2025, 3, 2, 12, 0, 0, TimeSpan.Zero),
|
UpdatedAt = new DateTimeOffset(2025, 3, 2, 12, 0, 0, TimeSpan.Zero),
|
||||||
Aliases = new[] { "GHSA-qqqq-wwww-eeee", "CVE-2025-4242" },
|
Aliases = new[] { "GHSA-qqqq-wwww-eeee", "CVE-2025-4242" },
|
||||||
References = new[]
|
References = new[]
|
||||||
{
|
{
|
||||||
new GhsaReferenceDto
|
new GhsaReferenceDto
|
||||||
{
|
{
|
||||||
Url = "https://github.com/advisories/GHSA-qqqq-wwww-eeee",
|
Url = "https://github.com/advisories/GHSA-qqqq-wwww-eeee",
|
||||||
Type = "ADVISORY"
|
Type = "ADVISORY"
|
||||||
},
|
},
|
||||||
new GhsaReferenceDto
|
new GhsaReferenceDto
|
||||||
{
|
{
|
||||||
Url = "https://github.com/conflict/package/releases/tag/v1.4.0",
|
Url = "https://github.com/conflict/package/releases/tag/v1.4.0",
|
||||||
Type = "FIX"
|
Type = "FIX"
|
||||||
}
|
}
|
||||||
},
|
},
|
||||||
Affected = new[]
|
Affected = new[]
|
||||||
{
|
{
|
||||||
new GhsaAffectedDto
|
new GhsaAffectedDto
|
||||||
{
|
{
|
||||||
PackageName = "conflict/package",
|
PackageName = "conflict/package",
|
||||||
Ecosystem = "npm",
|
Ecosystem = "npm",
|
||||||
VulnerableRange = "< 1.4.0",
|
VulnerableRange = "< 1.4.0",
|
||||||
PatchedVersion = "1.4.0"
|
PatchedVersion = "1.4.0"
|
||||||
}
|
}
|
||||||
},
|
},
|
||||||
Credits = new[]
|
Credits = new[]
|
||||||
{
|
{
|
||||||
new GhsaCreditDto
|
new GhsaCreditDto
|
||||||
{
|
{
|
||||||
Type = "reporter",
|
Type = "reporter",
|
||||||
Name = "security-researcher",
|
Name = "security-researcher",
|
||||||
Login = "sec-researcher",
|
Login = "sec-researcher",
|
||||||
ProfileUrl = "https://github.com/sec-researcher"
|
ProfileUrl = "https://github.com/sec-researcher"
|
||||||
},
|
},
|
||||||
new GhsaCreditDto
|
new GhsaCreditDto
|
||||||
{
|
{
|
||||||
Type = "remediation_developer",
|
Type = "remediation_developer",
|
||||||
Name = "maintainer-team",
|
Name = "maintainer-team",
|
||||||
Login = "conflict-maintainer",
|
Login = "conflict-maintainer",
|
||||||
ProfileUrl = "https://github.com/conflict/package"
|
ProfileUrl = "https://github.com/conflict/package"
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
};
|
};
|
||||||
|
|
||||||
var advisory = GhsaMapper.Map(dto, document, recordedAt);
|
var advisory = GhsaMapper.Map(dto, document, recordedAt);
|
||||||
Assert.Equal("ghsa:severity/high", advisory.CanonicalMetricId);
|
Assert.Equal("ghsa:severity/high", advisory.CanonicalMetricId);
|
||||||
Assert.True(advisory.CvssMetrics.IsEmpty);
|
Assert.True(advisory.CvssMetrics.IsEmpty);
|
||||||
var snapshot = SnapshotSerializer.ToSnapshot(advisory).Replace("\r\n", "\n").TrimEnd();
|
var snapshot = SnapshotSerializer.ToSnapshot(advisory).Replace("\r\n", "\n").TrimEnd();
|
||||||
|
|
||||||
var expectedPath = Path.Combine(AppContext.BaseDirectory, "Fixtures", "conflict-ghsa.canonical.json");
|
var expectedPath = Path.Combine(AppContext.BaseDirectory, "Fixtures", "conflict-ghsa.canonical.json");
|
||||||
var expected = File.ReadAllText(expectedPath).Replace("\r\n", "\n").TrimEnd();
|
var expected = File.ReadAllText(expectedPath).Replace("\r\n", "\n").TrimEnd();
|
||||||
|
|
||||||
if (!string.Equals(expected, snapshot, StringComparison.Ordinal))
|
if (!string.Equals(expected, snapshot, StringComparison.Ordinal))
|
||||||
{
|
{
|
||||||
var actualPath = Path.Combine(AppContext.BaseDirectory, "Fixtures", "conflict-ghsa.canonical.actual.json");
|
var actualPath = Path.Combine(AppContext.BaseDirectory, "Fixtures", "conflict-ghsa.canonical.actual.json");
|
||||||
File.WriteAllText(actualPath, snapshot);
|
File.WriteAllText(actualPath, snapshot);
|
||||||
}
|
}
|
||||||
|
|
||||||
Assert.Equal(expected, snapshot);
|
Assert.Equal(expected, snapshot);
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -21,7 +21,7 @@ public sealed class GhsaMapperTests
|
|||||||
Metadata: null,
|
Metadata: null,
|
||||||
Etag: "\"etag-ghsa-fallback\"",
|
Etag: "\"etag-ghsa-fallback\"",
|
||||||
LastModified: recordedAt.AddHours(-3),
|
LastModified: recordedAt.AddHours(-3),
|
||||||
GridFsId: null);
|
PayloadId: null);
|
||||||
|
|
||||||
var dto = new GhsaRecordDto
|
var dto = new GhsaRecordDto
|
||||||
{
|
{
|
||||||
|
|||||||
@@ -1,103 +1,103 @@
|
|||||||
using System.Text.Json;
|
using System.Text.Json;
|
||||||
using StellaOps.Concelier.Models;
|
using StellaOps.Concelier.Models;
|
||||||
using StellaOps.Concelier.Connector.Nvd.Internal;
|
using StellaOps.Concelier.Connector.Nvd.Internal;
|
||||||
using StellaOps.Concelier.Storage.Mongo.Documents;
|
using StellaOps.Concelier.Storage.Mongo.Documents;
|
||||||
|
|
||||||
namespace StellaOps.Concelier.Connector.Nvd.Tests;
|
namespace StellaOps.Concelier.Connector.Nvd.Tests;
|
||||||
|
|
||||||
public sealed class NvdConflictFixtureTests
|
public sealed class NvdConflictFixtureTests
|
||||||
{
|
{
|
||||||
[Fact]
|
[Fact]
|
||||||
public void ConflictFixture_MatchesSnapshot()
|
public void ConflictFixture_MatchesSnapshot()
|
||||||
{
|
{
|
||||||
const string payload = """
|
const string payload = """
|
||||||
{
|
{
|
||||||
"vulnerabilities": [
|
"vulnerabilities": [
|
||||||
{
|
{
|
||||||
"cve": {
|
"cve": {
|
||||||
"id": "CVE-2025-4242",
|
"id": "CVE-2025-4242",
|
||||||
"published": "2025-03-01T10:15:00Z",
|
"published": "2025-03-01T10:15:00Z",
|
||||||
"lastModified": "2025-03-03T09:45:00Z",
|
"lastModified": "2025-03-03T09:45:00Z",
|
||||||
"descriptions": [
|
"descriptions": [
|
||||||
{ "lang": "en", "value": "NVD baseline summary for conflict-package allowing container escape." }
|
{ "lang": "en", "value": "NVD baseline summary for conflict-package allowing container escape." }
|
||||||
],
|
],
|
||||||
"references": [
|
"references": [
|
||||||
{
|
{
|
||||||
"url": "https://nvd.nist.gov/vuln/detail/CVE-2025-4242",
|
"url": "https://nvd.nist.gov/vuln/detail/CVE-2025-4242",
|
||||||
"source": "NVD",
|
"source": "NVD",
|
||||||
"tags": ["Vendor Advisory"]
|
"tags": ["Vendor Advisory"]
|
||||||
}
|
}
|
||||||
],
|
],
|
||||||
"weaknesses": [
|
"weaknesses": [
|
||||||
{
|
{
|
||||||
"description": [
|
"description": [
|
||||||
{ "lang": "en", "value": "CWE-269" }
|
{ "lang": "en", "value": "CWE-269" }
|
||||||
]
|
]
|
||||||
}
|
}
|
||||||
],
|
],
|
||||||
"metrics": {
|
"metrics": {
|
||||||
"cvssMetricV31": [
|
"cvssMetricV31": [
|
||||||
{
|
{
|
||||||
"cvssData": {
|
"cvssData": {
|
||||||
"vectorString": "CVSS:3.1/AV:N/AC:L/PR:N/UI:N/S:U/C:H/I:H/A:H",
|
"vectorString": "CVSS:3.1/AV:N/AC:L/PR:N/UI:N/S:U/C:H/I:H/A:H",
|
||||||
"baseScore": 9.8,
|
"baseScore": 9.8,
|
||||||
"baseSeverity": "CRITICAL"
|
"baseSeverity": "CRITICAL"
|
||||||
},
|
},
|
||||||
"exploitabilityScore": 3.9,
|
"exploitabilityScore": 3.9,
|
||||||
"impactScore": 5.9
|
"impactScore": 5.9
|
||||||
}
|
}
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
"configurations": {
|
"configurations": {
|
||||||
"nodes": [
|
"nodes": [
|
||||||
{
|
{
|
||||||
"cpeMatch": [
|
"cpeMatch": [
|
||||||
{
|
{
|
||||||
"criteria": "cpe:2.3:a:conflict:package:1.0:*:*:*:*:*:*:*",
|
"criteria": "cpe:2.3:a:conflict:package:1.0:*:*:*:*:*:*:*",
|
||||||
"vulnerable": true,
|
"vulnerable": true,
|
||||||
"versionStartIncluding": "1.0",
|
"versionStartIncluding": "1.0",
|
||||||
"versionEndExcluding": "1.4"
|
"versionEndExcluding": "1.4"
|
||||||
}
|
}
|
||||||
]
|
]
|
||||||
}
|
}
|
||||||
]
|
]
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
]
|
]
|
||||||
}
|
}
|
||||||
""";
|
""";
|
||||||
|
|
||||||
using var document = JsonDocument.Parse(payload);
|
using var document = JsonDocument.Parse(payload);
|
||||||
|
|
||||||
var sourceDocument = new DocumentRecord(
|
var sourceDocument = new DocumentRecord(
|
||||||
Id: Guid.Parse("1a6a0700-2dd0-4f69-bb37-64ca77e51c91"),
|
Id: Guid.Parse("1a6a0700-2dd0-4f69-bb37-64ca77e51c91"),
|
||||||
SourceName: NvdConnectorPlugin.SourceName,
|
SourceName: NvdConnectorPlugin.SourceName,
|
||||||
Uri: "https://services.nvd.nist.gov/rest/json/cve/2.0?cveId=CVE-2025-4242",
|
Uri: "https://services.nvd.nist.gov/rest/json/cve/2.0?cveId=CVE-2025-4242",
|
||||||
FetchedAt: new DateTimeOffset(2025, 3, 3, 10, 0, 0, TimeSpan.Zero),
|
FetchedAt: new DateTimeOffset(2025, 3, 3, 10, 0, 0, TimeSpan.Zero),
|
||||||
Sha256: "sha256-nvd-conflict-fixture",
|
Sha256: "sha256-nvd-conflict-fixture",
|
||||||
Status: "completed",
|
Status: "completed",
|
||||||
ContentType: "application/json",
|
ContentType: "application/json",
|
||||||
Headers: null,
|
Headers: null,
|
||||||
Metadata: null,
|
Metadata: null,
|
||||||
Etag: "\"etag-nvd-conflict\"",
|
Etag: "\"etag-nvd-conflict\"",
|
||||||
LastModified: new DateTimeOffset(2025, 3, 3, 9, 45, 0, TimeSpan.Zero),
|
LastModified: new DateTimeOffset(2025, 3, 3, 9, 45, 0, TimeSpan.Zero),
|
||||||
GridFsId: null);
|
PayloadId: null);
|
||||||
|
|
||||||
var advisories = NvdMapper.Map(document, sourceDocument, new DateTimeOffset(2025, 3, 4, 2, 0, 0, TimeSpan.Zero));
|
var advisories = NvdMapper.Map(document, sourceDocument, new DateTimeOffset(2025, 3, 4, 2, 0, 0, TimeSpan.Zero));
|
||||||
var advisory = Assert.Single(advisories);
|
var advisory = Assert.Single(advisories);
|
||||||
|
|
||||||
var snapshot = SnapshotSerializer.ToSnapshot(advisory).Replace("\r\n", "\n").TrimEnd();
|
var snapshot = SnapshotSerializer.ToSnapshot(advisory).Replace("\r\n", "\n").TrimEnd();
|
||||||
var expectedPath = Path.Combine(AppContext.BaseDirectory, "Nvd", "Fixtures", "conflict-nvd.canonical.json");
|
var expectedPath = Path.Combine(AppContext.BaseDirectory, "Nvd", "Fixtures", "conflict-nvd.canonical.json");
|
||||||
var expected = File.ReadAllText(expectedPath).Replace("\r\n", "\n").TrimEnd();
|
var expected = File.ReadAllText(expectedPath).Replace("\r\n", "\n").TrimEnd();
|
||||||
|
|
||||||
if (!string.Equals(expected, snapshot, StringComparison.Ordinal))
|
if (!string.Equals(expected, snapshot, StringComparison.Ordinal))
|
||||||
{
|
{
|
||||||
var actualPath = Path.Combine(AppContext.BaseDirectory, "Nvd", "Fixtures", "conflict-nvd.canonical.actual.json");
|
var actualPath = Path.Combine(AppContext.BaseDirectory, "Nvd", "Fixtures", "conflict-nvd.canonical.actual.json");
|
||||||
Directory.CreateDirectory(Path.GetDirectoryName(actualPath)!);
|
Directory.CreateDirectory(Path.GetDirectoryName(actualPath)!);
|
||||||
File.WriteAllText(actualPath, snapshot);
|
File.WriteAllText(actualPath, snapshot);
|
||||||
}
|
}
|
||||||
|
|
||||||
Assert.Equal(expected, snapshot);
|
Assert.Equal(expected, snapshot);
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -1,118 +1,118 @@
|
|||||||
using System.Text.Json;
|
using System.Text.Json;
|
||||||
using MongoDB.Bson;
|
using MongoDB.Bson;
|
||||||
using StellaOps.Concelier.Models;
|
using StellaOps.Concelier.Models;
|
||||||
using StellaOps.Concelier.Connector.Osv.Internal;
|
using StellaOps.Concelier.Connector.Osv.Internal;
|
||||||
using StellaOps.Concelier.Storage.Mongo.Documents;
|
using StellaOps.Concelier.Storage.Mongo.Documents;
|
||||||
using StellaOps.Concelier.Storage.Mongo.Dtos;
|
using StellaOps.Concelier.Storage.Mongo.Dtos;
|
||||||
|
|
||||||
namespace StellaOps.Concelier.Connector.Osv.Tests;
|
namespace StellaOps.Concelier.Connector.Osv.Tests;
|
||||||
|
|
||||||
public sealed class OsvConflictFixtureTests
|
public sealed class OsvConflictFixtureTests
|
||||||
{
|
{
|
||||||
[Fact]
|
[Fact]
|
||||||
public void ConflictFixture_MatchesSnapshot()
|
public void ConflictFixture_MatchesSnapshot()
|
||||||
{
|
{
|
||||||
using var databaseSpecificDoc = JsonDocument.Parse("""{"severity":"medium"}""");
|
using var databaseSpecificDoc = JsonDocument.Parse("""{"severity":"medium"}""");
|
||||||
|
|
||||||
var dto = new OsvVulnerabilityDto
|
var dto = new OsvVulnerabilityDto
|
||||||
{
|
{
|
||||||
Id = "OSV-2025-4242",
|
Id = "OSV-2025-4242",
|
||||||
Summary = "Container escape for conflict-package",
|
Summary = "Container escape for conflict-package",
|
||||||
Details = "OSV captures the latest container escape details including patched version metadata.",
|
Details = "OSV captures the latest container escape details including patched version metadata.",
|
||||||
Aliases = new[] { "CVE-2025-4242", "GHSA-qqqq-wwww-eeee" },
|
Aliases = new[] { "CVE-2025-4242", "GHSA-qqqq-wwww-eeee" },
|
||||||
Published = new DateTimeOffset(2025, 2, 28, 0, 0, 0, TimeSpan.Zero),
|
Published = new DateTimeOffset(2025, 2, 28, 0, 0, 0, TimeSpan.Zero),
|
||||||
Modified = new DateTimeOffset(2025, 3, 6, 12, 0, 0, TimeSpan.Zero),
|
Modified = new DateTimeOffset(2025, 3, 6, 12, 0, 0, TimeSpan.Zero),
|
||||||
Severity = new[]
|
Severity = new[]
|
||||||
{
|
{
|
||||||
new OsvSeverityDto
|
new OsvSeverityDto
|
||||||
{
|
{
|
||||||
Type = "CVSS_V3",
|
Type = "CVSS_V3",
|
||||||
Score = "CVSS:3.1/AV:N/AC:H/PR:L/UI:R/S:U/C:L/I:L/A:L"
|
Score = "CVSS:3.1/AV:N/AC:H/PR:L/UI:R/S:U/C:L/I:L/A:L"
|
||||||
}
|
}
|
||||||
},
|
},
|
||||||
References = new[]
|
References = new[]
|
||||||
{
|
{
|
||||||
new OsvReferenceDto
|
new OsvReferenceDto
|
||||||
{
|
{
|
||||||
Type = "ADVISORY",
|
Type = "ADVISORY",
|
||||||
Url = "https://osv.dev/vulnerability/OSV-2025-4242"
|
Url = "https://osv.dev/vulnerability/OSV-2025-4242"
|
||||||
},
|
},
|
||||||
new OsvReferenceDto
|
new OsvReferenceDto
|
||||||
{
|
{
|
||||||
Type = "FIX",
|
Type = "FIX",
|
||||||
Url = "https://github.com/conflict/package/commit/abcdef1234567890"
|
Url = "https://github.com/conflict/package/commit/abcdef1234567890"
|
||||||
}
|
}
|
||||||
},
|
},
|
||||||
Credits = new[]
|
Credits = new[]
|
||||||
{
|
{
|
||||||
new OsvCreditDto
|
new OsvCreditDto
|
||||||
{
|
{
|
||||||
Name = "osv-reporter",
|
Name = "osv-reporter",
|
||||||
Type = "reporter",
|
Type = "reporter",
|
||||||
Contact = new[] { "mailto:osv-reporter@example.com" }
|
Contact = new[] { "mailto:osv-reporter@example.com" }
|
||||||
}
|
}
|
||||||
},
|
},
|
||||||
Affected = new[]
|
Affected = new[]
|
||||||
{
|
{
|
||||||
new OsvAffectedPackageDto
|
new OsvAffectedPackageDto
|
||||||
{
|
{
|
||||||
Package = new OsvPackageDto
|
Package = new OsvPackageDto
|
||||||
{
|
{
|
||||||
Ecosystem = "npm",
|
Ecosystem = "npm",
|
||||||
Name = "conflict/package"
|
Name = "conflict/package"
|
||||||
},
|
},
|
||||||
Ranges = new[]
|
Ranges = new[]
|
||||||
{
|
{
|
||||||
new OsvRangeDto
|
new OsvRangeDto
|
||||||
{
|
{
|
||||||
Type = "SEMVER",
|
Type = "SEMVER",
|
||||||
Events = new[]
|
Events = new[]
|
||||||
{
|
{
|
||||||
new OsvEventDto { Introduced = "1.0.0" },
|
new OsvEventDto { Introduced = "1.0.0" },
|
||||||
new OsvEventDto { LastAffected = "1.4.2" },
|
new OsvEventDto { LastAffected = "1.4.2" },
|
||||||
new OsvEventDto { Fixed = "1.5.0" }
|
new OsvEventDto { Fixed = "1.5.0" }
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
},
|
},
|
||||||
DatabaseSpecific = databaseSpecificDoc.RootElement.Clone()
|
DatabaseSpecific = databaseSpecificDoc.RootElement.Clone()
|
||||||
};
|
};
|
||||||
|
|
||||||
var document = new DocumentRecord(
|
var document = new DocumentRecord(
|
||||||
Id: Guid.Parse("8dd2b0fe-a5f5-4b3b-9f5c-0f3aad6fb6ce"),
|
Id: Guid.Parse("8dd2b0fe-a5f5-4b3b-9f5c-0f3aad6fb6ce"),
|
||||||
SourceName: OsvConnectorPlugin.SourceName,
|
SourceName: OsvConnectorPlugin.SourceName,
|
||||||
Uri: "https://api.osv.dev/v1/vulns/OSV-2025-4242",
|
Uri: "https://api.osv.dev/v1/vulns/OSV-2025-4242",
|
||||||
FetchedAt: new DateTimeOffset(2025, 3, 6, 11, 30, 0, TimeSpan.Zero),
|
FetchedAt: new DateTimeOffset(2025, 3, 6, 11, 30, 0, TimeSpan.Zero),
|
||||||
Sha256: "sha256-osv-conflict-fixture",
|
Sha256: "sha256-osv-conflict-fixture",
|
||||||
Status: "completed",
|
Status: "completed",
|
||||||
ContentType: "application/json",
|
ContentType: "application/json",
|
||||||
Headers: null,
|
Headers: null,
|
||||||
Metadata: null,
|
Metadata: null,
|
||||||
Etag: "\"etag-osv-conflict\"",
|
Etag: "\"etag-osv-conflict\"",
|
||||||
LastModified: new DateTimeOffset(2025, 3, 6, 12, 0, 0, TimeSpan.Zero),
|
LastModified: new DateTimeOffset(2025, 3, 6, 12, 0, 0, TimeSpan.Zero),
|
||||||
GridFsId: null);
|
PayloadId: null);
|
||||||
|
|
||||||
var dtoRecord = new DtoRecord(
|
var dtoRecord = new DtoRecord(
|
||||||
Id: Guid.Parse("6f7d5ce7-cb47-40a5-8b41-8ad022b5fd5c"),
|
Id: Guid.Parse("6f7d5ce7-cb47-40a5-8b41-8ad022b5fd5c"),
|
||||||
DocumentId: document.Id,
|
DocumentId: document.Id,
|
||||||
SourceName: OsvConnectorPlugin.SourceName,
|
SourceName: OsvConnectorPlugin.SourceName,
|
||||||
SchemaVersion: "osv.v1",
|
SchemaVersion: "osv.v1",
|
||||||
Payload: new BsonDocument("id", dto.Id),
|
Payload: new BsonDocument("id", dto.Id),
|
||||||
ValidatedAt: new DateTimeOffset(2025, 3, 6, 12, 5, 0, TimeSpan.Zero));
|
ValidatedAt: new DateTimeOffset(2025, 3, 6, 12, 5, 0, TimeSpan.Zero));
|
||||||
|
|
||||||
var advisory = OsvMapper.Map(dto, document, dtoRecord, "npm");
|
var advisory = OsvMapper.Map(dto, document, dtoRecord, "npm");
|
||||||
var snapshot = SnapshotSerializer.ToSnapshot(advisory).Replace("\r\n", "\n").TrimEnd();
|
var snapshot = SnapshotSerializer.ToSnapshot(advisory).Replace("\r\n", "\n").TrimEnd();
|
||||||
|
|
||||||
var expectedPath = Path.Combine(AppContext.BaseDirectory, "Fixtures", "conflict-osv.canonical.json");
|
var expectedPath = Path.Combine(AppContext.BaseDirectory, "Fixtures", "conflict-osv.canonical.json");
|
||||||
var expected = File.ReadAllText(expectedPath).Replace("\r\n", "\n").TrimEnd();
|
var expected = File.ReadAllText(expectedPath).Replace("\r\n", "\n").TrimEnd();
|
||||||
|
|
||||||
if (!string.Equals(expected, snapshot, StringComparison.Ordinal))
|
if (!string.Equals(expected, snapshot, StringComparison.Ordinal))
|
||||||
{
|
{
|
||||||
var actualPath = Path.Combine(AppContext.BaseDirectory, "Fixtures", "conflict-osv.canonical.actual.json");
|
var actualPath = Path.Combine(AppContext.BaseDirectory, "Fixtures", "conflict-osv.canonical.actual.json");
|
||||||
File.WriteAllText(actualPath, snapshot);
|
File.WriteAllText(actualPath, snapshot);
|
||||||
}
|
}
|
||||||
|
|
||||||
Assert.Equal(expected, snapshot);
|
Assert.Equal(expected, snapshot);
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -1,463 +1,463 @@
|
|||||||
using System;
|
using System;
|
||||||
using System.Collections.Generic;
|
using System.Collections.Generic;
|
||||||
using System.IO;
|
using System.IO;
|
||||||
using System.Net;
|
using System.Net;
|
||||||
using System.Net.Http;
|
using System.Net.Http;
|
||||||
using System.Security.Cryptography;
|
using System.Security.Cryptography;
|
||||||
using System.Text;
|
using System.Text;
|
||||||
using System.Text.Json;
|
using System.Text.Json;
|
||||||
using Microsoft.Extensions.Configuration;
|
using Microsoft.Extensions.Configuration;
|
||||||
using Microsoft.Extensions.DependencyInjection;
|
using Microsoft.Extensions.DependencyInjection;
|
||||||
using Microsoft.Extensions.Http;
|
using Microsoft.Extensions.Http;
|
||||||
using Microsoft.Extensions.Logging;
|
using Microsoft.Extensions.Logging;
|
||||||
using Microsoft.Extensions.Logging.Abstractions;
|
using Microsoft.Extensions.Logging.Abstractions;
|
||||||
using Microsoft.Extensions.Options;
|
using Microsoft.Extensions.Options;
|
||||||
using MongoDB.Bson;
|
using MongoDB.Bson;
|
||||||
using StellaOps.Concelier.Connector.Common;
|
using StellaOps.Concelier.Connector.Common;
|
||||||
using StellaOps.Concelier.Connector.Common.Fetch;
|
using StellaOps.Concelier.Connector.Common.Fetch;
|
||||||
using StellaOps.Concelier.Connector.Common.Testing;
|
using StellaOps.Concelier.Connector.Common.Testing;
|
||||||
using StellaOps.Concelier.Connector.StellaOpsMirror.Internal;
|
using StellaOps.Concelier.Connector.StellaOpsMirror.Internal;
|
||||||
using StellaOps.Concelier.Connector.StellaOpsMirror.Settings;
|
using StellaOps.Concelier.Connector.StellaOpsMirror.Settings;
|
||||||
using StellaOps.Concelier.Storage.Mongo;
|
using StellaOps.Concelier.Storage.Mongo;
|
||||||
using StellaOps.Concelier.Storage.Mongo.Advisories;
|
using StellaOps.Concelier.Storage.Mongo.Advisories;
|
||||||
using StellaOps.Concelier.Storage.Mongo.Documents;
|
using StellaOps.Concelier.Storage.Mongo.Documents;
|
||||||
using StellaOps.Concelier.Storage.Mongo.Dtos;
|
using StellaOps.Concelier.Storage.Mongo.Dtos;
|
||||||
using StellaOps.Concelier.Testing;
|
using StellaOps.Concelier.Testing;
|
||||||
using StellaOps.Cryptography;
|
using StellaOps.Cryptography;
|
||||||
using StellaOps.Cryptography.DependencyInjection;
|
using StellaOps.Cryptography.DependencyInjection;
|
||||||
using StellaOps.Concelier.Models;
|
using StellaOps.Concelier.Models;
|
||||||
using Xunit;
|
using Xunit;
|
||||||
|
|
||||||
namespace StellaOps.Concelier.Connector.StellaOpsMirror.Tests;
|
namespace StellaOps.Concelier.Connector.StellaOpsMirror.Tests;
|
||||||
|
|
||||||
[Collection("mongo-fixture")]
|
[Collection("mongo-fixture")]
|
||||||
public sealed class StellaOpsMirrorConnectorTests : IAsyncLifetime
|
public sealed class StellaOpsMirrorConnectorTests : IAsyncLifetime
|
||||||
{
|
{
|
||||||
private readonly MongoIntegrationFixture _fixture;
|
private readonly MongoIntegrationFixture _fixture;
|
||||||
private readonly CannedHttpMessageHandler _handler;
|
private readonly CannedHttpMessageHandler _handler;
|
||||||
|
|
||||||
public StellaOpsMirrorConnectorTests(MongoIntegrationFixture fixture)
|
public StellaOpsMirrorConnectorTests(MongoIntegrationFixture fixture)
|
||||||
{
|
{
|
||||||
_fixture = fixture;
|
_fixture = fixture;
|
||||||
_handler = new CannedHttpMessageHandler();
|
_handler = new CannedHttpMessageHandler();
|
||||||
}
|
}
|
||||||
|
|
||||||
[Fact]
|
[Fact]
|
||||||
public async Task FetchAsync_PersistsMirrorArtifacts()
|
public async Task FetchAsync_PersistsMirrorArtifacts()
|
||||||
{
|
{
|
||||||
var manifestContent = "{\"domain\":\"primary\",\"files\":[]}";
|
var manifestContent = "{\"domain\":\"primary\",\"files\":[]}";
|
||||||
var bundleContent = "{\"advisories\":[{\"id\":\"CVE-2025-0001\"}]}";
|
var bundleContent = "{\"advisories\":[{\"id\":\"CVE-2025-0001\"}]}";
|
||||||
|
|
||||||
var manifestDigest = ComputeDigest(manifestContent);
|
var manifestDigest = ComputeDigest(manifestContent);
|
||||||
var bundleDigest = ComputeDigest(bundleContent);
|
var bundleDigest = ComputeDigest(bundleContent);
|
||||||
|
|
||||||
var index = BuildIndex(manifestDigest, Encoding.UTF8.GetByteCount(manifestContent), bundleDigest, Encoding.UTF8.GetByteCount(bundleContent), includeSignature: false);
|
var index = BuildIndex(manifestDigest, Encoding.UTF8.GetByteCount(manifestContent), bundleDigest, Encoding.UTF8.GetByteCount(bundleContent), includeSignature: false);
|
||||||
|
|
||||||
await using var provider = await BuildServiceProviderAsync();
|
await using var provider = await BuildServiceProviderAsync();
|
||||||
|
|
||||||
SeedResponses(index, manifestContent, bundleContent, signature: null);
|
SeedResponses(index, manifestContent, bundleContent, signature: null);
|
||||||
|
|
||||||
var connector = provider.GetRequiredService<StellaOpsMirrorConnector>();
|
var connector = provider.GetRequiredService<StellaOpsMirrorConnector>();
|
||||||
await connector.FetchAsync(provider, CancellationToken.None);
|
await connector.FetchAsync(provider, CancellationToken.None);
|
||||||
|
|
||||||
var documentStore = provider.GetRequiredService<IDocumentStore>();
|
var documentStore = provider.GetRequiredService<IDocumentStore>();
|
||||||
var manifestUri = "https://mirror.test/mirror/primary/manifest.json";
|
var manifestUri = "https://mirror.test/mirror/primary/manifest.json";
|
||||||
var bundleUri = "https://mirror.test/mirror/primary/bundle.json";
|
var bundleUri = "https://mirror.test/mirror/primary/bundle.json";
|
||||||
|
|
||||||
var manifestDocument = await documentStore.FindBySourceAndUriAsync(StellaOpsMirrorConnector.Source, manifestUri, CancellationToken.None);
|
var manifestDocument = await documentStore.FindBySourceAndUriAsync(StellaOpsMirrorConnector.Source, manifestUri, CancellationToken.None);
|
||||||
Assert.NotNull(manifestDocument);
|
Assert.NotNull(manifestDocument);
|
||||||
Assert.Equal(DocumentStatuses.Mapped, manifestDocument!.Status);
|
Assert.Equal(DocumentStatuses.Mapped, manifestDocument!.Status);
|
||||||
Assert.Equal(NormalizeDigest(manifestDigest), manifestDocument.Sha256);
|
Assert.Equal(NormalizeDigest(manifestDigest), manifestDocument.Sha256);
|
||||||
|
|
||||||
var bundleDocument = await documentStore.FindBySourceAndUriAsync(StellaOpsMirrorConnector.Source, bundleUri, CancellationToken.None);
|
var bundleDocument = await documentStore.FindBySourceAndUriAsync(StellaOpsMirrorConnector.Source, bundleUri, CancellationToken.None);
|
||||||
Assert.NotNull(bundleDocument);
|
Assert.NotNull(bundleDocument);
|
||||||
Assert.Equal(DocumentStatuses.PendingParse, bundleDocument!.Status);
|
Assert.Equal(DocumentStatuses.PendingParse, bundleDocument!.Status);
|
||||||
Assert.Equal(NormalizeDigest(bundleDigest), bundleDocument.Sha256);
|
Assert.Equal(NormalizeDigest(bundleDigest), bundleDocument.Sha256);
|
||||||
|
|
||||||
var rawStorage = provider.GetRequiredService<RawDocumentStorage>();
|
var rawStorage = provider.GetRequiredService<RawDocumentStorage>();
|
||||||
Assert.NotNull(manifestDocument.GridFsId);
|
Assert.NotNull(manifestDocument.PayloadId);
|
||||||
Assert.NotNull(bundleDocument.GridFsId);
|
Assert.NotNull(bundleDocument.PayloadId);
|
||||||
|
|
||||||
var manifestBytes = await rawStorage.DownloadAsync(manifestDocument.GridFsId!.Value, CancellationToken.None);
|
var manifestBytes = await rawStorage.DownloadAsync(manifestDocument.PayloadId!.Value, CancellationToken.None);
|
||||||
var bundleBytes = await rawStorage.DownloadAsync(bundleDocument.GridFsId!.Value, CancellationToken.None);
|
var bundleBytes = await rawStorage.DownloadAsync(bundleDocument.PayloadId!.Value, CancellationToken.None);
|
||||||
Assert.Equal(manifestContent, Encoding.UTF8.GetString(manifestBytes));
|
Assert.Equal(manifestContent, Encoding.UTF8.GetString(manifestBytes));
|
||||||
Assert.Equal(bundleContent, Encoding.UTF8.GetString(bundleBytes));
|
Assert.Equal(bundleContent, Encoding.UTF8.GetString(bundleBytes));
|
||||||
|
|
||||||
var stateRepository = provider.GetRequiredService<ISourceStateRepository>();
|
var stateRepository = provider.GetRequiredService<ISourceStateRepository>();
|
||||||
var state = await stateRepository.TryGetAsync(StellaOpsMirrorConnector.Source, CancellationToken.None);
|
var state = await stateRepository.TryGetAsync(StellaOpsMirrorConnector.Source, CancellationToken.None);
|
||||||
Assert.NotNull(state);
|
Assert.NotNull(state);
|
||||||
|
|
||||||
var cursorDocument = state!.Cursor ?? new BsonDocument();
|
var cursorDocument = state!.Cursor ?? new BsonDocument();
|
||||||
var digestValue = cursorDocument.TryGetValue("bundleDigest", out var digestBson) ? digestBson.AsString : string.Empty;
|
var digestValue = cursorDocument.TryGetValue("bundleDigest", out var digestBson) ? digestBson.AsString : string.Empty;
|
||||||
Assert.Equal(NormalizeDigest(bundleDigest), NormalizeDigest(digestValue));
|
Assert.Equal(NormalizeDigest(bundleDigest), NormalizeDigest(digestValue));
|
||||||
|
|
||||||
var pendingDocumentsArray = cursorDocument.TryGetValue("pendingDocuments", out var pendingDocsBson) && pendingDocsBson is BsonArray pendingArray
|
var pendingDocumentsArray = cursorDocument.TryGetValue("pendingDocuments", out var pendingDocsBson) && pendingDocsBson is BsonArray pendingArray
|
||||||
? pendingArray
|
? pendingArray
|
||||||
: new BsonArray();
|
: new BsonArray();
|
||||||
Assert.Single(pendingDocumentsArray);
|
Assert.Single(pendingDocumentsArray);
|
||||||
var pendingDocumentId = Guid.Parse(pendingDocumentsArray[0].AsString);
|
var pendingDocumentId = Guid.Parse(pendingDocumentsArray[0].AsString);
|
||||||
Assert.Equal(bundleDocument.Id, pendingDocumentId);
|
Assert.Equal(bundleDocument.Id, pendingDocumentId);
|
||||||
|
|
||||||
var pendingMappingsArray = cursorDocument.TryGetValue("pendingMappings", out var pendingMappingsBson) && pendingMappingsBson is BsonArray mappingsArray
|
var pendingMappingsArray = cursorDocument.TryGetValue("pendingMappings", out var pendingMappingsBson) && pendingMappingsBson is BsonArray mappingsArray
|
||||||
? mappingsArray
|
? mappingsArray
|
||||||
: new BsonArray();
|
: new BsonArray();
|
||||||
Assert.Empty(pendingMappingsArray);
|
Assert.Empty(pendingMappingsArray);
|
||||||
}
|
}
|
||||||
|
|
||||||
[Fact]
|
[Fact]
|
||||||
public async Task FetchAsync_TamperedSignatureThrows()
|
public async Task FetchAsync_TamperedSignatureThrows()
|
||||||
{
|
{
|
||||||
var manifestContent = "{\"domain\":\"primary\"}";
|
var manifestContent = "{\"domain\":\"primary\"}";
|
||||||
var bundleContent = "{\"advisories\":[{\"id\":\"CVE-2025-0002\"}]}";
|
var bundleContent = "{\"advisories\":[{\"id\":\"CVE-2025-0002\"}]}";
|
||||||
|
|
||||||
var manifestDigest = ComputeDigest(manifestContent);
|
var manifestDigest = ComputeDigest(manifestContent);
|
||||||
var bundleDigest = ComputeDigest(bundleContent);
|
var bundleDigest = ComputeDigest(bundleContent);
|
||||||
var index = BuildIndex(manifestDigest, Encoding.UTF8.GetByteCount(manifestContent), bundleDigest, Encoding.UTF8.GetByteCount(bundleContent), includeSignature: true);
|
var index = BuildIndex(manifestDigest, Encoding.UTF8.GetByteCount(manifestContent), bundleDigest, Encoding.UTF8.GetByteCount(bundleContent), includeSignature: true);
|
||||||
|
|
||||||
await using var provider = await BuildServiceProviderAsync(options =>
|
await using var provider = await BuildServiceProviderAsync(options =>
|
||||||
{
|
{
|
||||||
options.Signature.Enabled = true;
|
options.Signature.Enabled = true;
|
||||||
options.Signature.KeyId = "mirror-key";
|
options.Signature.KeyId = "mirror-key";
|
||||||
options.Signature.Provider = "default";
|
options.Signature.Provider = "default";
|
||||||
});
|
});
|
||||||
|
|
||||||
var defaultProvider = provider.GetRequiredService<DefaultCryptoProvider>();
|
var defaultProvider = provider.GetRequiredService<DefaultCryptoProvider>();
|
||||||
var signingKey = CreateSigningKey("mirror-key");
|
var signingKey = CreateSigningKey("mirror-key");
|
||||||
defaultProvider.UpsertSigningKey(signingKey);
|
defaultProvider.UpsertSigningKey(signingKey);
|
||||||
|
|
||||||
var (signatureValue, _) = CreateDetachedJws(signingKey, bundleContent);
|
var (signatureValue, _) = CreateDetachedJws(signingKey, bundleContent);
|
||||||
// Tamper with signature so verification fails.
|
// Tamper with signature so verification fails.
|
||||||
var tamperedSignature = signatureValue.Replace('a', 'b');
|
var tamperedSignature = signatureValue.Replace('a', 'b');
|
||||||
|
|
||||||
SeedResponses(index, manifestContent, bundleContent, tamperedSignature);
|
SeedResponses(index, manifestContent, bundleContent, tamperedSignature);
|
||||||
|
|
||||||
var connector = provider.GetRequiredService<StellaOpsMirrorConnector>();
|
var connector = provider.GetRequiredService<StellaOpsMirrorConnector>();
|
||||||
await Assert.ThrowsAsync<InvalidOperationException>(() => connector.FetchAsync(provider, CancellationToken.None));
|
await Assert.ThrowsAsync<InvalidOperationException>(() => connector.FetchAsync(provider, CancellationToken.None));
|
||||||
|
|
||||||
var stateRepository = provider.GetRequiredService<ISourceStateRepository>();
|
var stateRepository = provider.GetRequiredService<ISourceStateRepository>();
|
||||||
var state = await stateRepository.TryGetAsync(StellaOpsMirrorConnector.Source, CancellationToken.None);
|
var state = await stateRepository.TryGetAsync(StellaOpsMirrorConnector.Source, CancellationToken.None);
|
||||||
Assert.NotNull(state);
|
Assert.NotNull(state);
|
||||||
Assert.True(state!.FailCount >= 1);
|
Assert.True(state!.FailCount >= 1);
|
||||||
Assert.False(state.Cursor.TryGetValue("bundleDigest", out _));
|
Assert.False(state.Cursor.TryGetValue("bundleDigest", out _));
|
||||||
}
|
}
|
||||||
|
|
||||||
[Fact]
|
[Fact]
|
||||||
public async Task FetchAsync_SignatureKeyMismatchThrows()
|
public async Task FetchAsync_SignatureKeyMismatchThrows()
|
||||||
{
|
{
|
||||||
var manifestContent = "{\"domain\":\"primary\"}";
|
var manifestContent = "{\"domain\":\"primary\"}";
|
||||||
var bundleContent = "{\"advisories\":[{\"id\":\"CVE-2025-0003\"}]}";
|
var bundleContent = "{\"advisories\":[{\"id\":\"CVE-2025-0003\"}]}";
|
||||||
|
|
||||||
var manifestDigest = ComputeDigest(manifestContent);
|
var manifestDigest = ComputeDigest(manifestContent);
|
||||||
var bundleDigest = ComputeDigest(bundleContent);
|
var bundleDigest = ComputeDigest(bundleContent);
|
||||||
var index = BuildIndex(
|
var index = BuildIndex(
|
||||||
manifestDigest,
|
manifestDigest,
|
||||||
Encoding.UTF8.GetByteCount(manifestContent),
|
Encoding.UTF8.GetByteCount(manifestContent),
|
||||||
bundleDigest,
|
bundleDigest,
|
||||||
Encoding.UTF8.GetByteCount(bundleContent),
|
Encoding.UTF8.GetByteCount(bundleContent),
|
||||||
includeSignature: true,
|
includeSignature: true,
|
||||||
signatureKeyId: "unexpected-key",
|
signatureKeyId: "unexpected-key",
|
||||||
signatureProvider: "default");
|
signatureProvider: "default");
|
||||||
|
|
||||||
var signingKey = CreateSigningKey("unexpected-key");
|
var signingKey = CreateSigningKey("unexpected-key");
|
||||||
var (signatureValue, _) = CreateDetachedJws(signingKey, bundleContent);
|
var (signatureValue, _) = CreateDetachedJws(signingKey, bundleContent);
|
||||||
|
|
||||||
await using var provider = await BuildServiceProviderAsync(options =>
|
await using var provider = await BuildServiceProviderAsync(options =>
|
||||||
{
|
{
|
||||||
options.Signature.Enabled = true;
|
options.Signature.Enabled = true;
|
||||||
options.Signature.KeyId = "mirror-key";
|
options.Signature.KeyId = "mirror-key";
|
||||||
options.Signature.Provider = "default";
|
options.Signature.Provider = "default";
|
||||||
});
|
});
|
||||||
|
|
||||||
SeedResponses(index, manifestContent, bundleContent, signatureValue);
|
SeedResponses(index, manifestContent, bundleContent, signatureValue);
|
||||||
|
|
||||||
var connector = provider.GetRequiredService<StellaOpsMirrorConnector>();
|
var connector = provider.GetRequiredService<StellaOpsMirrorConnector>();
|
||||||
await Assert.ThrowsAsync<InvalidOperationException>(() => connector.FetchAsync(provider, CancellationToken.None));
|
await Assert.ThrowsAsync<InvalidOperationException>(() => connector.FetchAsync(provider, CancellationToken.None));
|
||||||
}
|
}
|
||||||
|
|
||||||
[Fact]
|
[Fact]
|
||||||
public async Task FetchAsync_VerifiesSignatureUsingFallbackPublicKey()
|
public async Task FetchAsync_VerifiesSignatureUsingFallbackPublicKey()
|
||||||
{
|
{
|
||||||
var manifestContent = "{\"domain\":\"primary\"}";
|
var manifestContent = "{\"domain\":\"primary\"}";
|
||||||
var bundleContent = "{\"advisories\":[{\"id\":\"CVE-2025-0004\"}]}";
|
var bundleContent = "{\"advisories\":[{\"id\":\"CVE-2025-0004\"}]}";
|
||||||
|
|
||||||
var manifestDigest = ComputeDigest(manifestContent);
|
var manifestDigest = ComputeDigest(manifestContent);
|
||||||
var bundleDigest = ComputeDigest(bundleContent);
|
var bundleDigest = ComputeDigest(bundleContent);
|
||||||
var index = BuildIndex(manifestDigest, Encoding.UTF8.GetByteCount(manifestContent), bundleDigest, Encoding.UTF8.GetByteCount(bundleContent), includeSignature: true);
|
var index = BuildIndex(manifestDigest, Encoding.UTF8.GetByteCount(manifestContent), bundleDigest, Encoding.UTF8.GetByteCount(bundleContent), includeSignature: true);
|
||||||
|
|
||||||
var signingKey = CreateSigningKey("mirror-key");
|
var signingKey = CreateSigningKey("mirror-key");
|
||||||
var (signatureValue, _) = CreateDetachedJws(signingKey, bundleContent);
|
var (signatureValue, _) = CreateDetachedJws(signingKey, bundleContent);
|
||||||
var publicKeyPath = WritePublicKeyPem(signingKey);
|
var publicKeyPath = WritePublicKeyPem(signingKey);
|
||||||
|
|
||||||
await using var provider = await BuildServiceProviderAsync(options =>
|
await using var provider = await BuildServiceProviderAsync(options =>
|
||||||
{
|
{
|
||||||
options.Signature.Enabled = true;
|
options.Signature.Enabled = true;
|
||||||
options.Signature.KeyId = "mirror-key";
|
options.Signature.KeyId = "mirror-key";
|
||||||
options.Signature.Provider = "default";
|
options.Signature.Provider = "default";
|
||||||
options.Signature.PublicKeyPath = publicKeyPath;
|
options.Signature.PublicKeyPath = publicKeyPath;
|
||||||
});
|
});
|
||||||
|
|
||||||
try
|
try
|
||||||
{
|
{
|
||||||
SeedResponses(index, manifestContent, bundleContent, signatureValue);
|
SeedResponses(index, manifestContent, bundleContent, signatureValue);
|
||||||
|
|
||||||
var connector = provider.GetRequiredService<StellaOpsMirrorConnector>();
|
var connector = provider.GetRequiredService<StellaOpsMirrorConnector>();
|
||||||
await connector.FetchAsync(provider, CancellationToken.None);
|
await connector.FetchAsync(provider, CancellationToken.None);
|
||||||
|
|
||||||
var stateRepository = provider.GetRequiredService<ISourceStateRepository>();
|
var stateRepository = provider.GetRequiredService<ISourceStateRepository>();
|
||||||
var state = await stateRepository.TryGetAsync(StellaOpsMirrorConnector.Source, CancellationToken.None);
|
var state = await stateRepository.TryGetAsync(StellaOpsMirrorConnector.Source, CancellationToken.None);
|
||||||
Assert.NotNull(state);
|
Assert.NotNull(state);
|
||||||
Assert.Equal(0, state!.FailCount);
|
Assert.Equal(0, state!.FailCount);
|
||||||
}
|
}
|
||||||
finally
|
finally
|
||||||
{
|
{
|
||||||
if (File.Exists(publicKeyPath))
|
if (File.Exists(publicKeyPath))
|
||||||
{
|
{
|
||||||
File.Delete(publicKeyPath);
|
File.Delete(publicKeyPath);
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
[Fact]
|
[Fact]
|
||||||
public async Task FetchAsync_DigestMismatchMarksFailure()
|
public async Task FetchAsync_DigestMismatchMarksFailure()
|
||||||
{
|
{
|
||||||
var manifestExpected = "{\"domain\":\"primary\"}";
|
var manifestExpected = "{\"domain\":\"primary\"}";
|
||||||
var manifestTampered = "{\"domain\":\"tampered\"}";
|
var manifestTampered = "{\"domain\":\"tampered\"}";
|
||||||
var bundleContent = "{\"advisories\":[{\"id\":\"CVE-2025-0005\"}]}";
|
var bundleContent = "{\"advisories\":[{\"id\":\"CVE-2025-0005\"}]}";
|
||||||
|
|
||||||
var manifestDigest = ComputeDigest(manifestExpected);
|
var manifestDigest = ComputeDigest(manifestExpected);
|
||||||
var bundleDigest = ComputeDigest(bundleContent);
|
var bundleDigest = ComputeDigest(bundleContent);
|
||||||
var index = BuildIndex(manifestDigest, Encoding.UTF8.GetByteCount(manifestExpected), bundleDigest, Encoding.UTF8.GetByteCount(bundleContent), includeSignature: false);
|
var index = BuildIndex(manifestDigest, Encoding.UTF8.GetByteCount(manifestExpected), bundleDigest, Encoding.UTF8.GetByteCount(bundleContent), includeSignature: false);
|
||||||
|
|
||||||
await using var provider = await BuildServiceProviderAsync();
|
await using var provider = await BuildServiceProviderAsync();
|
||||||
|
|
||||||
SeedResponses(index, manifestTampered, bundleContent, signature: null);
|
SeedResponses(index, manifestTampered, bundleContent, signature: null);
|
||||||
|
|
||||||
var connector = provider.GetRequiredService<StellaOpsMirrorConnector>();
|
var connector = provider.GetRequiredService<StellaOpsMirrorConnector>();
|
||||||
|
|
||||||
await Assert.ThrowsAsync<InvalidOperationException>(() => connector.FetchAsync(provider, CancellationToken.None));
|
await Assert.ThrowsAsync<InvalidOperationException>(() => connector.FetchAsync(provider, CancellationToken.None));
|
||||||
|
|
||||||
var stateRepository = provider.GetRequiredService<ISourceStateRepository>();
|
var stateRepository = provider.GetRequiredService<ISourceStateRepository>();
|
||||||
var state = await stateRepository.TryGetAsync(StellaOpsMirrorConnector.Source, CancellationToken.None);
|
var state = await stateRepository.TryGetAsync(StellaOpsMirrorConnector.Source, CancellationToken.None);
|
||||||
Assert.NotNull(state);
|
Assert.NotNull(state);
|
||||||
var cursor = state!.Cursor ?? new BsonDocument();
|
var cursor = state!.Cursor ?? new BsonDocument();
|
||||||
Assert.True(state.FailCount >= 1);
|
Assert.True(state.FailCount >= 1);
|
||||||
Assert.False(cursor.Contains("bundleDigest"));
|
Assert.False(cursor.Contains("bundleDigest"));
|
||||||
}
|
}
|
||||||
|
|
||||||
[Fact]
|
[Fact]
|
||||||
public void ParseAndMap_PersistAdvisoriesFromBundle()
|
public void ParseAndMap_PersistAdvisoriesFromBundle()
|
||||||
{
|
{
|
||||||
var bundleDocument = SampleData.CreateBundle();
|
var bundleDocument = SampleData.CreateBundle();
|
||||||
var bundleJson = CanonicalJsonSerializer.SerializeIndented(bundleDocument);
|
var bundleJson = CanonicalJsonSerializer.SerializeIndented(bundleDocument);
|
||||||
var normalizedFixture = FixtureLoader.Read(SampleData.BundleFixture).TrimEnd();
|
var normalizedFixture = FixtureLoader.Read(SampleData.BundleFixture).TrimEnd();
|
||||||
Assert.Equal(normalizedFixture, FixtureLoader.Normalize(bundleJson).TrimEnd());
|
Assert.Equal(normalizedFixture, FixtureLoader.Normalize(bundleJson).TrimEnd());
|
||||||
|
|
||||||
var advisories = MirrorAdvisoryMapper.Map(bundleDocument);
|
var advisories = MirrorAdvisoryMapper.Map(bundleDocument);
|
||||||
Assert.Single(advisories);
|
Assert.Single(advisories);
|
||||||
var advisory = advisories[0];
|
var advisory = advisories[0];
|
||||||
|
|
||||||
var expectedAdvisoryJson = FixtureLoader.Read(SampleData.AdvisoryFixture).TrimEnd();
|
var expectedAdvisoryJson = FixtureLoader.Read(SampleData.AdvisoryFixture).TrimEnd();
|
||||||
var mappedJson = CanonicalJsonSerializer.SerializeIndented(advisory);
|
var mappedJson = CanonicalJsonSerializer.SerializeIndented(advisory);
|
||||||
Assert.Equal(expectedAdvisoryJson, FixtureLoader.Normalize(mappedJson).TrimEnd());
|
Assert.Equal(expectedAdvisoryJson, FixtureLoader.Normalize(mappedJson).TrimEnd());
|
||||||
|
|
||||||
// AdvisoryStore integration validated elsewhere; ensure canonical serialization is stable.
|
// AdvisoryStore integration validated elsewhere; ensure canonical serialization is stable.
|
||||||
}
|
}
|
||||||
|
|
||||||
public Task InitializeAsync() => Task.CompletedTask;
|
public Task InitializeAsync() => Task.CompletedTask;
|
||||||
|
|
||||||
public Task DisposeAsync()
|
public Task DisposeAsync()
|
||||||
{
|
{
|
||||||
_handler.Clear();
|
_handler.Clear();
|
||||||
return Task.CompletedTask;
|
return Task.CompletedTask;
|
||||||
}
|
}
|
||||||
|
|
||||||
private async Task<ServiceProvider> BuildServiceProviderAsync(Action<StellaOpsMirrorConnectorOptions>? configureOptions = null)
|
private async Task<ServiceProvider> BuildServiceProviderAsync(Action<StellaOpsMirrorConnectorOptions>? configureOptions = null)
|
||||||
{
|
{
|
||||||
await _fixture.Client.DropDatabaseAsync(_fixture.Database.DatabaseNamespace.DatabaseName);
|
await _fixture.Client.DropDatabaseAsync(_fixture.Database.DatabaseNamespace.DatabaseName);
|
||||||
_handler.Clear();
|
_handler.Clear();
|
||||||
|
|
||||||
var services = new ServiceCollection();
|
var services = new ServiceCollection();
|
||||||
services.AddLogging(builder => builder.AddProvider(NullLoggerProvider.Instance));
|
services.AddLogging(builder => builder.AddProvider(NullLoggerProvider.Instance));
|
||||||
services.AddSingleton(_handler);
|
services.AddSingleton(_handler);
|
||||||
services.AddSingleton(TimeProvider.System);
|
services.AddSingleton(TimeProvider.System);
|
||||||
|
|
||||||
services.AddMongoStorage(options =>
|
services.AddMongoStorage(options =>
|
||||||
{
|
{
|
||||||
options.ConnectionString = _fixture.Runner.ConnectionString;
|
options.ConnectionString = _fixture.Runner.ConnectionString;
|
||||||
options.DatabaseName = _fixture.Database.DatabaseNamespace.DatabaseName;
|
options.DatabaseName = _fixture.Database.DatabaseNamespace.DatabaseName;
|
||||||
options.CommandTimeout = TimeSpan.FromSeconds(5);
|
options.CommandTimeout = TimeSpan.FromSeconds(5);
|
||||||
});
|
});
|
||||||
|
|
||||||
services.AddStellaOpsCrypto();
|
services.AddStellaOpsCrypto();
|
||||||
|
|
||||||
var configuration = new ConfigurationBuilder()
|
var configuration = new ConfigurationBuilder()
|
||||||
.AddInMemoryCollection(new Dictionary<string, string?>
|
.AddInMemoryCollection(new Dictionary<string, string?>
|
||||||
{
|
{
|
||||||
["concelier:sources:stellaopsMirror:baseAddress"] = "https://mirror.test/",
|
["concelier:sources:stellaopsMirror:baseAddress"] = "https://mirror.test/",
|
||||||
["concelier:sources:stellaopsMirror:domainId"] = "primary",
|
["concelier:sources:stellaopsMirror:domainId"] = "primary",
|
||||||
["concelier:sources:stellaopsMirror:indexPath"] = "/concelier/exports/index.json",
|
["concelier:sources:stellaopsMirror:indexPath"] = "/concelier/exports/index.json",
|
||||||
})
|
})
|
||||||
.Build();
|
.Build();
|
||||||
|
|
||||||
var routine = new StellaOpsMirrorDependencyInjectionRoutine();
|
var routine = new StellaOpsMirrorDependencyInjectionRoutine();
|
||||||
routine.Register(services, configuration);
|
routine.Register(services, configuration);
|
||||||
|
|
||||||
if (configureOptions is not null)
|
if (configureOptions is not null)
|
||||||
{
|
{
|
||||||
services.PostConfigure(configureOptions);
|
services.PostConfigure(configureOptions);
|
||||||
}
|
}
|
||||||
|
|
||||||
services.Configure<HttpClientFactoryOptions>("stellaops-mirror", builder =>
|
services.Configure<HttpClientFactoryOptions>("stellaops-mirror", builder =>
|
||||||
{
|
{
|
||||||
builder.HttpMessageHandlerBuilderActions.Add(options =>
|
builder.HttpMessageHandlerBuilderActions.Add(options =>
|
||||||
{
|
{
|
||||||
options.PrimaryHandler = _handler;
|
options.PrimaryHandler = _handler;
|
||||||
});
|
});
|
||||||
});
|
});
|
||||||
|
|
||||||
var provider = services.BuildServiceProvider();
|
var provider = services.BuildServiceProvider();
|
||||||
var bootstrapper = provider.GetRequiredService<MongoBootstrapper>();
|
var bootstrapper = provider.GetRequiredService<MongoBootstrapper>();
|
||||||
await bootstrapper.InitializeAsync(CancellationToken.None);
|
await bootstrapper.InitializeAsync(CancellationToken.None);
|
||||||
return provider;
|
return provider;
|
||||||
}
|
}
|
||||||
|
|
||||||
private void SeedResponses(string indexJson, string manifestContent, string bundleContent, string? signature)
|
private void SeedResponses(string indexJson, string manifestContent, string bundleContent, string? signature)
|
||||||
{
|
{
|
||||||
var baseUri = new Uri("https://mirror.test");
|
var baseUri = new Uri("https://mirror.test");
|
||||||
_handler.AddResponse(HttpMethod.Get, new Uri(baseUri, "/concelier/exports/index.json"), () => CreateJsonResponse(indexJson));
|
_handler.AddResponse(HttpMethod.Get, new Uri(baseUri, "/concelier/exports/index.json"), () => CreateJsonResponse(indexJson));
|
||||||
_handler.AddResponse(HttpMethod.Get, new Uri(baseUri, "mirror/primary/manifest.json"), () => CreateJsonResponse(manifestContent));
|
_handler.AddResponse(HttpMethod.Get, new Uri(baseUri, "mirror/primary/manifest.json"), () => CreateJsonResponse(manifestContent));
|
||||||
_handler.AddResponse(HttpMethod.Get, new Uri(baseUri, "mirror/primary/bundle.json"), () => CreateJsonResponse(bundleContent));
|
_handler.AddResponse(HttpMethod.Get, new Uri(baseUri, "mirror/primary/bundle.json"), () => CreateJsonResponse(bundleContent));
|
||||||
|
|
||||||
if (signature is not null)
|
if (signature is not null)
|
||||||
{
|
{
|
||||||
_handler.AddResponse(HttpMethod.Get, new Uri(baseUri, "mirror/primary/bundle.json.jws"), () => new HttpResponseMessage(HttpStatusCode.OK)
|
_handler.AddResponse(HttpMethod.Get, new Uri(baseUri, "mirror/primary/bundle.json.jws"), () => new HttpResponseMessage(HttpStatusCode.OK)
|
||||||
{
|
{
|
||||||
Content = new StringContent(signature, Encoding.UTF8, "application/jose+json"),
|
Content = new StringContent(signature, Encoding.UTF8, "application/jose+json"),
|
||||||
});
|
});
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
private static HttpResponseMessage CreateJsonResponse(string content)
|
private static HttpResponseMessage CreateJsonResponse(string content)
|
||||||
=> new(HttpStatusCode.OK)
|
=> new(HttpStatusCode.OK)
|
||||||
{
|
{
|
||||||
Content = new StringContent(content, Encoding.UTF8, "application/json"),
|
Content = new StringContent(content, Encoding.UTF8, "application/json"),
|
||||||
};
|
};
|
||||||
|
|
||||||
private static string BuildIndex(
|
private static string BuildIndex(
|
||||||
string manifestDigest,
|
string manifestDigest,
|
||||||
int manifestBytes,
|
int manifestBytes,
|
||||||
string bundleDigest,
|
string bundleDigest,
|
||||||
int bundleBytes,
|
int bundleBytes,
|
||||||
bool includeSignature,
|
bool includeSignature,
|
||||||
string signatureKeyId = "mirror-key",
|
string signatureKeyId = "mirror-key",
|
||||||
string signatureProvider = "default")
|
string signatureProvider = "default")
|
||||||
{
|
{
|
||||||
var index = new
|
var index = new
|
||||||
{
|
{
|
||||||
schemaVersion = 1,
|
schemaVersion = 1,
|
||||||
generatedAt = new DateTimeOffset(2025, 10, 19, 12, 0, 0, TimeSpan.Zero),
|
generatedAt = new DateTimeOffset(2025, 10, 19, 12, 0, 0, TimeSpan.Zero),
|
||||||
targetRepository = "repo",
|
targetRepository = "repo",
|
||||||
domains = new[]
|
domains = new[]
|
||||||
{
|
{
|
||||||
new
|
new
|
||||||
{
|
{
|
||||||
domainId = "primary",
|
domainId = "primary",
|
||||||
displayName = "Primary",
|
displayName = "Primary",
|
||||||
advisoryCount = 1,
|
advisoryCount = 1,
|
||||||
manifest = new
|
manifest = new
|
||||||
{
|
{
|
||||||
path = "mirror/primary/manifest.json",
|
path = "mirror/primary/manifest.json",
|
||||||
sizeBytes = manifestBytes,
|
sizeBytes = manifestBytes,
|
||||||
digest = manifestDigest,
|
digest = manifestDigest,
|
||||||
signature = (object?)null,
|
signature = (object?)null,
|
||||||
},
|
},
|
||||||
bundle = new
|
bundle = new
|
||||||
{
|
{
|
||||||
path = "mirror/primary/bundle.json",
|
path = "mirror/primary/bundle.json",
|
||||||
sizeBytes = bundleBytes,
|
sizeBytes = bundleBytes,
|
||||||
digest = bundleDigest,
|
digest = bundleDigest,
|
||||||
signature = includeSignature
|
signature = includeSignature
|
||||||
? new
|
? new
|
||||||
{
|
{
|
||||||
path = "mirror/primary/bundle.json.jws",
|
path = "mirror/primary/bundle.json.jws",
|
||||||
algorithm = "ES256",
|
algorithm = "ES256",
|
||||||
keyId = signatureKeyId,
|
keyId = signatureKeyId,
|
||||||
provider = signatureProvider,
|
provider = signatureProvider,
|
||||||
signedAt = new DateTimeOffset(2025, 10, 19, 12, 0, 0, TimeSpan.Zero),
|
signedAt = new DateTimeOffset(2025, 10, 19, 12, 0, 0, TimeSpan.Zero),
|
||||||
}
|
}
|
||||||
: null,
|
: null,
|
||||||
},
|
},
|
||||||
sources = Array.Empty<object>(),
|
sources = Array.Empty<object>(),
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
};
|
};
|
||||||
|
|
||||||
return JsonSerializer.Serialize(index, new JsonSerializerOptions
|
return JsonSerializer.Serialize(index, new JsonSerializerOptions
|
||||||
{
|
{
|
||||||
PropertyNamingPolicy = JsonNamingPolicy.CamelCase,
|
PropertyNamingPolicy = JsonNamingPolicy.CamelCase,
|
||||||
WriteIndented = false,
|
WriteIndented = false,
|
||||||
});
|
});
|
||||||
}
|
}
|
||||||
|
|
||||||
private static string ComputeDigest(string content)
|
private static string ComputeDigest(string content)
|
||||||
{
|
{
|
||||||
var bytes = Encoding.UTF8.GetBytes(content);
|
var bytes = Encoding.UTF8.GetBytes(content);
|
||||||
var hash = SHA256.HashData(bytes);
|
var hash = SHA256.HashData(bytes);
|
||||||
return "sha256:" + Convert.ToHexString(hash).ToLowerInvariant();
|
return "sha256:" + Convert.ToHexString(hash).ToLowerInvariant();
|
||||||
}
|
}
|
||||||
|
|
||||||
private static string NormalizeDigest(string digest)
|
private static string NormalizeDigest(string digest)
|
||||||
=> digest.StartsWith("sha256:", StringComparison.OrdinalIgnoreCase) ? digest[7..] : digest;
|
=> digest.StartsWith("sha256:", StringComparison.OrdinalIgnoreCase) ? digest[7..] : digest;
|
||||||
|
|
||||||
private static CryptoSigningKey CreateSigningKey(string keyId)
|
private static CryptoSigningKey CreateSigningKey(string keyId)
|
||||||
{
|
{
|
||||||
using var ecdsa = ECDsa.Create(ECCurve.NamedCurves.nistP256);
|
using var ecdsa = ECDsa.Create(ECCurve.NamedCurves.nistP256);
|
||||||
var parameters = ecdsa.ExportParameters(includePrivateParameters: true);
|
var parameters = ecdsa.ExportParameters(includePrivateParameters: true);
|
||||||
return new CryptoSigningKey(new CryptoKeyReference(keyId), SignatureAlgorithms.Es256, in parameters, DateTimeOffset.UtcNow);
|
return new CryptoSigningKey(new CryptoKeyReference(keyId), SignatureAlgorithms.Es256, in parameters, DateTimeOffset.UtcNow);
|
||||||
}
|
}
|
||||||
|
|
||||||
private static string WritePublicKeyPem(CryptoSigningKey signingKey)
|
private static string WritePublicKeyPem(CryptoSigningKey signingKey)
|
||||||
{
|
{
|
||||||
ArgumentNullException.ThrowIfNull(signingKey);
|
ArgumentNullException.ThrowIfNull(signingKey);
|
||||||
var path = Path.Combine(Path.GetTempPath(), $"stellaops-mirror-{Guid.NewGuid():N}.pem");
|
var path = Path.Combine(Path.GetTempPath(), $"stellaops-mirror-{Guid.NewGuid():N}.pem");
|
||||||
using var ecdsa = ECDsa.Create(signingKey.PublicParameters);
|
using var ecdsa = ECDsa.Create(signingKey.PublicParameters);
|
||||||
var publicKeyInfo = ecdsa.ExportSubjectPublicKeyInfo();
|
var publicKeyInfo = ecdsa.ExportSubjectPublicKeyInfo();
|
||||||
var pem = PemEncoding.Write("PUBLIC KEY", publicKeyInfo);
|
var pem = PemEncoding.Write("PUBLIC KEY", publicKeyInfo);
|
||||||
File.WriteAllText(path, pem);
|
File.WriteAllText(path, pem);
|
||||||
return path;
|
return path;
|
||||||
}
|
}
|
||||||
|
|
||||||
private static (string Signature, DateTimeOffset SignedAt) CreateDetachedJws(CryptoSigningKey signingKey, string payload)
|
private static (string Signature, DateTimeOffset SignedAt) CreateDetachedJws(CryptoSigningKey signingKey, string payload)
|
||||||
{
|
{
|
||||||
var provider = new DefaultCryptoProvider();
|
var provider = new DefaultCryptoProvider();
|
||||||
provider.UpsertSigningKey(signingKey);
|
provider.UpsertSigningKey(signingKey);
|
||||||
var signer = provider.GetSigner(SignatureAlgorithms.Es256, signingKey.Reference);
|
var signer = provider.GetSigner(SignatureAlgorithms.Es256, signingKey.Reference);
|
||||||
var header = new Dictionary<string, object?>
|
var header = new Dictionary<string, object?>
|
||||||
{
|
{
|
||||||
["alg"] = SignatureAlgorithms.Es256,
|
["alg"] = SignatureAlgorithms.Es256,
|
||||||
["kid"] = signingKey.Reference.KeyId,
|
["kid"] = signingKey.Reference.KeyId,
|
||||||
["provider"] = provider.Name,
|
["provider"] = provider.Name,
|
||||||
["typ"] = "application/vnd.stellaops.concelier.mirror-bundle+jws",
|
["typ"] = "application/vnd.stellaops.concelier.mirror-bundle+jws",
|
||||||
["b64"] = false,
|
["b64"] = false,
|
||||||
["crit"] = new[] { "b64" }
|
["crit"] = new[] { "b64" }
|
||||||
};
|
};
|
||||||
|
|
||||||
var headerJson = JsonSerializer.Serialize(header);
|
var headerJson = JsonSerializer.Serialize(header);
|
||||||
var encodedHeader = Microsoft.IdentityModel.Tokens.Base64UrlEncoder.Encode(headerJson);
|
var encodedHeader = Microsoft.IdentityModel.Tokens.Base64UrlEncoder.Encode(headerJson);
|
||||||
var payloadBytes = Encoding.UTF8.GetBytes(payload);
|
var payloadBytes = Encoding.UTF8.GetBytes(payload);
|
||||||
var signingInput = BuildSigningInput(encodedHeader, payloadBytes);
|
var signingInput = BuildSigningInput(encodedHeader, payloadBytes);
|
||||||
var signatureBytes = signer.SignAsync(signingInput, CancellationToken.None).GetAwaiter().GetResult();
|
var signatureBytes = signer.SignAsync(signingInput, CancellationToken.None).GetAwaiter().GetResult();
|
||||||
var encodedSignature = Microsoft.IdentityModel.Tokens.Base64UrlEncoder.Encode(signatureBytes);
|
var encodedSignature = Microsoft.IdentityModel.Tokens.Base64UrlEncoder.Encode(signatureBytes);
|
||||||
return (string.Concat(encodedHeader, "..", encodedSignature), DateTimeOffset.UtcNow);
|
return (string.Concat(encodedHeader, "..", encodedSignature), DateTimeOffset.UtcNow);
|
||||||
}
|
}
|
||||||
|
|
||||||
private static ReadOnlyMemory<byte> BuildSigningInput(string encodedHeader, ReadOnlySpan<byte> payload)
|
private static ReadOnlyMemory<byte> BuildSigningInput(string encodedHeader, ReadOnlySpan<byte> payload)
|
||||||
{
|
{
|
||||||
var headerBytes = Encoding.ASCII.GetBytes(encodedHeader);
|
var headerBytes = Encoding.ASCII.GetBytes(encodedHeader);
|
||||||
var buffer = new byte[headerBytes.Length + 1 + payload.Length];
|
var buffer = new byte[headerBytes.Length + 1 + payload.Length];
|
||||||
headerBytes.CopyTo(buffer, 0);
|
headerBytes.CopyTo(buffer, 0);
|
||||||
buffer[headerBytes.Length] = (byte)'.';
|
buffer[headerBytes.Length] = (byte)'.';
|
||||||
payload.CopyTo(buffer.AsSpan(headerBytes.Length + 1));
|
payload.CopyTo(buffer.AsSpan(headerBytes.Length + 1));
|
||||||
return buffer;
|
return buffer;
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -1,36 +1,36 @@
|
|||||||
using System;
|
using System;
|
||||||
using System.Collections.Generic;
|
using System.Collections.Generic;
|
||||||
using System.Linq;
|
using System.Linq;
|
||||||
using FluentAssertions;
|
using FluentAssertions;
|
||||||
using MongoDB.Bson;
|
using MongoDB.Bson;
|
||||||
using StellaOps.Concelier.Models;
|
using StellaOps.Concelier.Models;
|
||||||
using StellaOps.Concelier.Connector.Common;
|
using StellaOps.Concelier.Connector.Common;
|
||||||
using StellaOps.Concelier.Connector.Vndr.Cisco;
|
using StellaOps.Concelier.Connector.Vndr.Cisco;
|
||||||
using StellaOps.Concelier.Connector.Vndr.Cisco.Internal;
|
using StellaOps.Concelier.Connector.Vndr.Cisco.Internal;
|
||||||
using StellaOps.Concelier.Storage.Mongo.Documents;
|
using StellaOps.Concelier.Storage.Mongo.Documents;
|
||||||
using StellaOps.Concelier.Storage.Mongo.Dtos;
|
using StellaOps.Concelier.Storage.Mongo.Dtos;
|
||||||
using Xunit;
|
using Xunit;
|
||||||
|
|
||||||
namespace StellaOps.Concelier.Connector.Vndr.Cisco.Tests;
|
namespace StellaOps.Concelier.Connector.Vndr.Cisco.Tests;
|
||||||
|
|
||||||
public sealed class CiscoMapperTests
|
public sealed class CiscoMapperTests
|
||||||
{
|
{
|
||||||
[Fact]
|
[Fact]
|
||||||
public void Map_ProducesCanonicalAdvisory()
|
public void Map_ProducesCanonicalAdvisory()
|
||||||
{
|
{
|
||||||
var published = new DateTimeOffset(2025, 10, 1, 0, 0, 0, TimeSpan.Zero);
|
var published = new DateTimeOffset(2025, 10, 1, 0, 0, 0, TimeSpan.Zero);
|
||||||
var updated = published.AddDays(1);
|
var updated = published.AddDays(1);
|
||||||
|
|
||||||
var dto = new CiscoAdvisoryDto(
|
var dto = new CiscoAdvisoryDto(
|
||||||
AdvisoryId: "CISCO-SA-TEST",
|
AdvisoryId: "CISCO-SA-TEST",
|
||||||
Title: "Test Advisory",
|
Title: "Test Advisory",
|
||||||
Summary: "Sample summary",
|
Summary: "Sample summary",
|
||||||
Severity: "High",
|
Severity: "High",
|
||||||
Published: published,
|
Published: published,
|
||||||
Updated: updated,
|
Updated: updated,
|
||||||
PublicationUrl: "https://example.com/advisory",
|
PublicationUrl: "https://example.com/advisory",
|
||||||
CsafUrl: "https://sec.cloudapps.cisco.com/csaf/test.json",
|
CsafUrl: "https://sec.cloudapps.cisco.com/csaf/test.json",
|
||||||
CvrfUrl: "https://example.com/cvrf.xml",
|
CvrfUrl: "https://example.com/cvrf.xml",
|
||||||
CvssBaseScore: 9.8,
|
CvssBaseScore: 9.8,
|
||||||
Cves: new List<string> { "CVE-2024-0001" },
|
Cves: new List<string> { "CVE-2024-0001" },
|
||||||
BugIds: new List<string> { "BUG123" },
|
BugIds: new List<string> { "BUG123" },
|
||||||
@@ -39,31 +39,31 @@ public sealed class CiscoMapperTests
|
|||||||
new("Cisco Widget", "PID-1", "1.2.3", new [] { AffectedPackageStatusCatalog.KnownAffected }),
|
new("Cisco Widget", "PID-1", "1.2.3", new [] { AffectedPackageStatusCatalog.KnownAffected }),
|
||||||
new("Cisco Router", "PID-2", ">=1.0.0 <1.4.0", new [] { AffectedPackageStatusCatalog.KnownAffected })
|
new("Cisco Router", "PID-2", ">=1.0.0 <1.4.0", new [] { AffectedPackageStatusCatalog.KnownAffected })
|
||||||
});
|
});
|
||||||
|
|
||||||
var document = new DocumentRecord(
|
var document = new DocumentRecord(
|
||||||
Id: Guid.NewGuid(),
|
Id: Guid.NewGuid(),
|
||||||
SourceName: VndrCiscoConnectorPlugin.SourceName,
|
SourceName: VndrCiscoConnectorPlugin.SourceName,
|
||||||
Uri: "https://api.cisco.com/security/advisories/v2/advisories/CISCO-SA-TEST",
|
Uri: "https://api.cisco.com/security/advisories/v2/advisories/CISCO-SA-TEST",
|
||||||
FetchedAt: published,
|
FetchedAt: published,
|
||||||
Sha256: "abc123",
|
Sha256: "abc123",
|
||||||
Status: DocumentStatuses.PendingMap,
|
Status: DocumentStatuses.PendingMap,
|
||||||
ContentType: "application/json",
|
ContentType: "application/json",
|
||||||
Headers: null,
|
Headers: null,
|
||||||
Metadata: null,
|
Metadata: null,
|
||||||
Etag: null,
|
Etag: null,
|
||||||
LastModified: updated,
|
LastModified: updated,
|
||||||
GridFsId: null);
|
PayloadId: null);
|
||||||
|
|
||||||
var dtoRecord = new DtoRecord(Guid.NewGuid(), document.Id, VndrCiscoConnectorPlugin.SourceName, "cisco.dto.test", new BsonDocument(), updated);
|
var dtoRecord = new DtoRecord(Guid.NewGuid(), document.Id, VndrCiscoConnectorPlugin.SourceName, "cisco.dto.test", new BsonDocument(), updated);
|
||||||
|
|
||||||
var advisory = CiscoMapper.Map(dto, document, dtoRecord);
|
var advisory = CiscoMapper.Map(dto, document, dtoRecord);
|
||||||
|
|
||||||
advisory.AdvisoryKey.Should().Be("CISCO-SA-TEST");
|
advisory.AdvisoryKey.Should().Be("CISCO-SA-TEST");
|
||||||
advisory.Title.Should().Be("Test Advisory");
|
advisory.Title.Should().Be("Test Advisory");
|
||||||
advisory.Severity.Should().Be("high");
|
advisory.Severity.Should().Be("high");
|
||||||
advisory.Aliases.Should().Contain(new[] { "CISCO-SA-TEST", "CVE-2024-0001", "BUG123" });
|
advisory.Aliases.Should().Contain(new[] { "CISCO-SA-TEST", "CVE-2024-0001", "BUG123" });
|
||||||
advisory.References.Should().Contain(reference => reference.Url == "https://example.com/advisory");
|
advisory.References.Should().Contain(reference => reference.Url == "https://example.com/advisory");
|
||||||
advisory.References.Should().Contain(reference => reference.Url == "https://sec.cloudapps.cisco.com/csaf/test.json");
|
advisory.References.Should().Contain(reference => reference.Url == "https://sec.cloudapps.cisco.com/csaf/test.json");
|
||||||
advisory.AffectedPackages.Should().HaveCount(2);
|
advisory.AffectedPackages.Should().HaveCount(2);
|
||||||
|
|
||||||
var package = advisory.AffectedPackages.Single(p => p.Identifier == "Cisco Widget");
|
var package = advisory.AffectedPackages.Single(p => p.Identifier == "Cisco Widget");
|
||||||
|
|||||||
@@ -0,0 +1,327 @@
|
|||||||
|
using System.Collections.Generic;
|
||||||
|
using System.Collections.Immutable;
|
||||||
|
using Microsoft.AspNetCore.Http.HttpResults;
|
||||||
|
using Microsoft.AspNetCore.Mvc;
|
||||||
|
using StellaOps.Auth.Abstractions;
|
||||||
|
using StellaOps.Attestor.Envelope;
|
||||||
|
using StellaOps.Policy.Engine.Services;
|
||||||
|
using StellaOps.Policy.Scoring;
|
||||||
|
using StellaOps.Policy.Scoring.Engine;
|
||||||
|
using StellaOps.Policy.Scoring.Receipts;
|
||||||
|
|
||||||
|
namespace StellaOps.Policy.Engine.Endpoints;
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// Minimal API surface for CVSS v4.0 score receipts (create, read, amend, history).
|
||||||
|
/// </summary>
|
||||||
|
internal static class CvssReceiptEndpoints
|
||||||
|
{
|
||||||
|
public static IEndpointRouteBuilder MapCvssReceipts(this IEndpointRouteBuilder endpoints)
|
||||||
|
{
|
||||||
|
var group = endpoints.MapGroup("/api/cvss")
|
||||||
|
.RequireAuthorization()
|
||||||
|
.WithTags("CVSS Receipts");
|
||||||
|
|
||||||
|
group.MapPost("/receipts", CreateReceipt)
|
||||||
|
.WithName("CreateCvssReceipt")
|
||||||
|
.WithSummary("Create a CVSS v4.0 receipt with deterministic hashing and optional DSSE attestation.")
|
||||||
|
.Produces<CvssScoreReceipt>(StatusCodes.Status201Created)
|
||||||
|
.Produces<ProblemHttpResult>(StatusCodes.Status400BadRequest)
|
||||||
|
.Produces<ProblemHttpResult>(StatusCodes.Status401Unauthorized);
|
||||||
|
|
||||||
|
group.MapGet("/receipts/{receiptId}", GetReceipt)
|
||||||
|
.WithName("GetCvssReceipt")
|
||||||
|
.WithSummary("Retrieve a CVSS v4.0 receipt by ID.")
|
||||||
|
.Produces<CvssScoreReceipt>(StatusCodes.Status200OK)
|
||||||
|
.Produces<ProblemHttpResult>(StatusCodes.Status404NotFound);
|
||||||
|
|
||||||
|
group.MapPut("/receipts/{receiptId}/amend", AmendReceipt)
|
||||||
|
.WithName("AmendCvssReceipt")
|
||||||
|
.WithSummary("Append an amendment entry to a CVSS receipt history and optionally re-sign.")
|
||||||
|
.Produces<CvssScoreReceipt>(StatusCodes.Status200OK)
|
||||||
|
.Produces<ProblemHttpResult>(StatusCodes.Status400BadRequest)
|
||||||
|
.Produces<ProblemHttpResult>(StatusCodes.Status404NotFound);
|
||||||
|
|
||||||
|
group.MapGet("/receipts/{receiptId}/history", GetReceiptHistory)
|
||||||
|
.WithName("GetCvssReceiptHistory")
|
||||||
|
.WithSummary("Return the ordered amendment history for a CVSS receipt.")
|
||||||
|
.Produces<IReadOnlyList<ReceiptHistoryEntry>>(StatusCodes.Status200OK)
|
||||||
|
.Produces<ProblemHttpResult>(StatusCodes.Status404NotFound);
|
||||||
|
|
||||||
|
group.MapGet("/policies", ListPolicies)
|
||||||
|
.WithName("ListCvssPolicies")
|
||||||
|
.WithSummary("List available CVSS policies configured on this host.")
|
||||||
|
.Produces<IReadOnlyList<CvssPolicy>>(StatusCodes.Status200OK);
|
||||||
|
|
||||||
|
return endpoints;
|
||||||
|
}
|
||||||
|
|
||||||
|
private static async Task<IResult> CreateReceipt(
|
||||||
|
HttpContext context,
|
||||||
|
[FromBody] CreateCvssReceiptRequest request,
|
||||||
|
IReceiptBuilder receiptBuilder,
|
||||||
|
CancellationToken cancellationToken)
|
||||||
|
{
|
||||||
|
var scopeResult = ScopeAuthorization.RequireScope(context, StellaOpsScopes.PolicyRun);
|
||||||
|
if (scopeResult is not null)
|
||||||
|
{
|
||||||
|
return scopeResult;
|
||||||
|
}
|
||||||
|
|
||||||
|
if (request is null)
|
||||||
|
{
|
||||||
|
return Results.BadRequest(new ProblemDetails
|
||||||
|
{
|
||||||
|
Title = "Request body required.",
|
||||||
|
Status = StatusCodes.Status400BadRequest
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
if (request.Policy is null || string.IsNullOrWhiteSpace(request.Policy.Hash))
|
||||||
|
{
|
||||||
|
return Results.BadRequest(new ProblemDetails
|
||||||
|
{
|
||||||
|
Title = "Policy hash required",
|
||||||
|
Detail = "CvssPolicy with a deterministic hash must be supplied.",
|
||||||
|
Status = StatusCodes.Status400BadRequest
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
var tenantId = ResolveTenantId(context);
|
||||||
|
if (string.IsNullOrWhiteSpace(tenantId))
|
||||||
|
{
|
||||||
|
return Results.BadRequest(new ProblemDetails
|
||||||
|
{
|
||||||
|
Title = "Tenant required",
|
||||||
|
Detail = "Specify tenant via X-Tenant-Id header or tenant_id claim.",
|
||||||
|
Status = StatusCodes.Status400BadRequest
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
var actor = ResolveActorId(context) ?? request.CreatedBy ?? "system";
|
||||||
|
var createdAt = request.CreatedAt ?? DateTimeOffset.UtcNow;
|
||||||
|
|
||||||
|
var createRequest = new CreateReceiptRequest
|
||||||
|
{
|
||||||
|
TenantId = tenantId,
|
||||||
|
VulnerabilityId = request.VulnerabilityId,
|
||||||
|
CreatedBy = actor,
|
||||||
|
CreatedAt = createdAt,
|
||||||
|
Policy = request.Policy,
|
||||||
|
BaseMetrics = request.BaseMetrics,
|
||||||
|
ThreatMetrics = request.ThreatMetrics,
|
||||||
|
EnvironmentalMetrics = request.EnvironmentalMetrics ?? request.Policy.DefaultEnvironmentalMetrics,
|
||||||
|
SupplementalMetrics = request.SupplementalMetrics,
|
||||||
|
Evidence = request.Evidence?.ToImmutableList() ?? ImmutableList<CvssEvidenceItem>.Empty,
|
||||||
|
SigningKey = request.SigningKey
|
||||||
|
};
|
||||||
|
|
||||||
|
try
|
||||||
|
{
|
||||||
|
var receipt = await receiptBuilder.CreateAsync(createRequest, cancellationToken).ConfigureAwait(false);
|
||||||
|
return Results.Created($"/api/cvss/receipts/{receipt.ReceiptId}", receipt);
|
||||||
|
}
|
||||||
|
catch (Exception ex) when (ex is InvalidOperationException or ArgumentException)
|
||||||
|
{
|
||||||
|
return Results.BadRequest(new ProblemDetails
|
||||||
|
{
|
||||||
|
Title = "Failed to create CVSS receipt",
|
||||||
|
Detail = ex.Message,
|
||||||
|
Status = StatusCodes.Status400BadRequest
|
||||||
|
});
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
private static async Task<IResult> GetReceipt(
|
||||||
|
HttpContext context,
|
||||||
|
[FromRoute] string receiptId,
|
||||||
|
IReceiptRepository repository,
|
||||||
|
CancellationToken cancellationToken)
|
||||||
|
{
|
||||||
|
var scopeResult = ScopeAuthorization.RequireScope(context, StellaOpsScopes.FindingsRead);
|
||||||
|
if (scopeResult is not null)
|
||||||
|
{
|
||||||
|
return scopeResult;
|
||||||
|
}
|
||||||
|
|
||||||
|
var tenantId = ResolveTenantId(context);
|
||||||
|
if (string.IsNullOrWhiteSpace(tenantId))
|
||||||
|
{
|
||||||
|
return Results.BadRequest(new ProblemDetails
|
||||||
|
{
|
||||||
|
Title = "Tenant required",
|
||||||
|
Detail = "Specify tenant via X-Tenant-Id header or tenant_id claim.",
|
||||||
|
Status = StatusCodes.Status400BadRequest
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
var receipt = await repository.GetAsync(tenantId, receiptId, cancellationToken).ConfigureAwait(false);
|
||||||
|
if (receipt is null)
|
||||||
|
{
|
||||||
|
return Results.NotFound(new ProblemDetails
|
||||||
|
{
|
||||||
|
Title = "Receipt not found",
|
||||||
|
Detail = $"CVSS receipt '{receiptId}' was not found.",
|
||||||
|
Status = StatusCodes.Status404NotFound
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
return Results.Ok(receipt);
|
||||||
|
}
|
||||||
|
|
||||||
|
private static async Task<IResult> AmendReceipt(
|
||||||
|
HttpContext context,
|
||||||
|
[FromRoute] string receiptId,
|
||||||
|
[FromBody] AmendCvssReceiptRequest request,
|
||||||
|
IReceiptHistoryService historyService,
|
||||||
|
CancellationToken cancellationToken)
|
||||||
|
{
|
||||||
|
var scopeResult = ScopeAuthorization.RequireScope(context, StellaOpsScopes.PolicyRun);
|
||||||
|
if (scopeResult is not null)
|
||||||
|
{
|
||||||
|
return scopeResult;
|
||||||
|
}
|
||||||
|
|
||||||
|
if (request is null)
|
||||||
|
{
|
||||||
|
return Results.BadRequest(new ProblemDetails
|
||||||
|
{
|
||||||
|
Title = "Request body required.",
|
||||||
|
Status = StatusCodes.Status400BadRequest
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
var tenantId = ResolveTenantId(context);
|
||||||
|
if (string.IsNullOrWhiteSpace(tenantId))
|
||||||
|
{
|
||||||
|
return Results.BadRequest(new ProblemDetails
|
||||||
|
{
|
||||||
|
Title = "Tenant required",
|
||||||
|
Detail = "Specify tenant via X-Tenant-Id header or tenant_id claim.",
|
||||||
|
Status = StatusCodes.Status400BadRequest
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
var actor = ResolveActorId(context) ?? request.Actor ?? "system";
|
||||||
|
|
||||||
|
var amend = new AmendReceiptRequest
|
||||||
|
{
|
||||||
|
ReceiptId = receiptId,
|
||||||
|
TenantId = tenantId,
|
||||||
|
Actor = actor,
|
||||||
|
Field = request.Field,
|
||||||
|
PreviousValue = request.PreviousValue,
|
||||||
|
NewValue = request.NewValue,
|
||||||
|
Reason = request.Reason,
|
||||||
|
ReferenceUri = request.ReferenceUri,
|
||||||
|
SigningKey = request.SigningKey
|
||||||
|
};
|
||||||
|
|
||||||
|
try
|
||||||
|
{
|
||||||
|
var amended = await historyService.AmendAsync(amend, cancellationToken).ConfigureAwait(false);
|
||||||
|
return Results.Ok(amended);
|
||||||
|
}
|
||||||
|
catch (InvalidOperationException ex)
|
||||||
|
{
|
||||||
|
return Results.NotFound(new ProblemDetails
|
||||||
|
{
|
||||||
|
Title = "Receipt not found",
|
||||||
|
Detail = ex.Message,
|
||||||
|
Status = StatusCodes.Status404NotFound
|
||||||
|
});
|
||||||
|
}
|
||||||
|
catch (Exception ex) when (ex is ArgumentException)
|
||||||
|
{
|
||||||
|
return Results.BadRequest(new ProblemDetails
|
||||||
|
{
|
||||||
|
Title = "Failed to amend receipt",
|
||||||
|
Detail = ex.Message,
|
||||||
|
Status = StatusCodes.Status400BadRequest
|
||||||
|
});
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
private static async Task<IResult> GetReceiptHistory(
|
||||||
|
HttpContext context,
|
||||||
|
[FromRoute] string receiptId,
|
||||||
|
IReceiptRepository repository,
|
||||||
|
CancellationToken cancellationToken)
|
||||||
|
{
|
||||||
|
var scopeResult = ScopeAuthorization.RequireScope(context, StellaOpsScopes.FindingsRead);
|
||||||
|
if (scopeResult is not null)
|
||||||
|
{
|
||||||
|
return scopeResult;
|
||||||
|
}
|
||||||
|
|
||||||
|
var tenantId = ResolveTenantId(context);
|
||||||
|
if (string.IsNullOrWhiteSpace(tenantId))
|
||||||
|
{
|
||||||
|
return Results.BadRequest(new ProblemDetails
|
||||||
|
{
|
||||||
|
Title = "Tenant required",
|
||||||
|
Detail = "Specify tenant via X-Tenant-Id header or tenant_id claim.",
|
||||||
|
Status = StatusCodes.Status400BadRequest
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
var receipt = await repository.GetAsync(tenantId, receiptId, cancellationToken).ConfigureAwait(false);
|
||||||
|
if (receipt is null)
|
||||||
|
{
|
||||||
|
return Results.NotFound(new ProblemDetails
|
||||||
|
{
|
||||||
|
Title = "Receipt not found",
|
||||||
|
Detail = $"CVSS receipt '{receiptId}' was not found.",
|
||||||
|
Status = StatusCodes.Status404NotFound
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
var orderedHistory = receipt.History
|
||||||
|
.OrderBy(h => h.Timestamp)
|
||||||
|
.ToList();
|
||||||
|
|
||||||
|
return Results.Ok(orderedHistory);
|
||||||
|
}
|
||||||
|
|
||||||
|
private static IResult ListPolicies()
|
||||||
|
=> Results.Ok(Array.Empty<CvssPolicy>());
|
||||||
|
|
||||||
|
private static string? ResolveTenantId(HttpContext context)
|
||||||
|
{
|
||||||
|
if (context.Request.Headers.TryGetValue("X-Tenant-Id", out var tenantHeader) &&
|
||||||
|
!string.IsNullOrWhiteSpace(tenantHeader))
|
||||||
|
{
|
||||||
|
return tenantHeader.ToString();
|
||||||
|
}
|
||||||
|
|
||||||
|
return context.User?.FindFirst("tenant_id")?.Value;
|
||||||
|
}
|
||||||
|
|
||||||
|
private static string? ResolveActorId(HttpContext context)
|
||||||
|
{
|
||||||
|
var user = context.User;
|
||||||
|
return user?.FindFirst(System.Security.Claims.ClaimTypes.NameIdentifier)?.Value
|
||||||
|
?? user?.FindFirst("sub")?.Value;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
internal sealed record CreateCvssReceiptRequest(
|
||||||
|
string VulnerabilityId,
|
||||||
|
CvssPolicy Policy,
|
||||||
|
CvssBaseMetrics BaseMetrics,
|
||||||
|
CvssThreatMetrics? ThreatMetrics,
|
||||||
|
CvssEnvironmentalMetrics? EnvironmentalMetrics,
|
||||||
|
CvssSupplementalMetrics? SupplementalMetrics,
|
||||||
|
IReadOnlyList<CvssEvidenceItem>? Evidence,
|
||||||
|
EnvelopeKey? SigningKey,
|
||||||
|
string? CreatedBy,
|
||||||
|
DateTimeOffset? CreatedAt);
|
||||||
|
|
||||||
|
internal sealed record AmendCvssReceiptRequest(
|
||||||
|
string Field,
|
||||||
|
string? PreviousValue,
|
||||||
|
string? NewValue,
|
||||||
|
string Reason,
|
||||||
|
string? ReferenceUri,
|
||||||
|
EnvelopeKey? SigningKey,
|
||||||
|
string? Actor);
|
||||||
@@ -1,15 +1,27 @@
|
|||||||
using StellaOps.Policy.Gateway.Contracts;
|
using StellaOps.Policy.Gateway.Contracts;
|
||||||
using StellaOps.Policy.Gateway.Infrastructure;
|
using StellaOps.Policy.Gateway.Infrastructure;
|
||||||
|
using StellaOps.Policy.Scoring;
|
||||||
namespace StellaOps.Policy.Gateway.Clients;
|
using StellaOps.Policy.Scoring.Receipts;
|
||||||
|
|
||||||
internal interface IPolicyEngineClient
|
namespace StellaOps.Policy.Gateway.Clients;
|
||||||
{
|
|
||||||
|
internal interface IPolicyEngineClient
|
||||||
|
{
|
||||||
Task<PolicyEngineResponse<IReadOnlyList<PolicyPackSummaryDto>>> ListPolicyPacksAsync(GatewayForwardingContext? forwardingContext, CancellationToken cancellationToken);
|
Task<PolicyEngineResponse<IReadOnlyList<PolicyPackSummaryDto>>> ListPolicyPacksAsync(GatewayForwardingContext? forwardingContext, CancellationToken cancellationToken);
|
||||||
|
|
||||||
Task<PolicyEngineResponse<PolicyPackDto>> CreatePolicyPackAsync(GatewayForwardingContext? forwardingContext, CreatePolicyPackRequest request, CancellationToken cancellationToken);
|
Task<PolicyEngineResponse<PolicyPackDto>> CreatePolicyPackAsync(GatewayForwardingContext? forwardingContext, CreatePolicyPackRequest request, CancellationToken cancellationToken);
|
||||||
|
|
||||||
Task<PolicyEngineResponse<PolicyRevisionDto>> CreatePolicyRevisionAsync(GatewayForwardingContext? forwardingContext, string packId, CreatePolicyRevisionRequest request, CancellationToken cancellationToken);
|
Task<PolicyEngineResponse<PolicyRevisionDto>> CreatePolicyRevisionAsync(GatewayForwardingContext? forwardingContext, string packId, CreatePolicyRevisionRequest request, CancellationToken cancellationToken);
|
||||||
|
|
||||||
Task<PolicyEngineResponse<PolicyRevisionActivationDto>> ActivatePolicyRevisionAsync(GatewayForwardingContext? forwardingContext, string packId, int version, ActivatePolicyRevisionRequest request, CancellationToken cancellationToken);
|
Task<PolicyEngineResponse<PolicyRevisionActivationDto>> ActivatePolicyRevisionAsync(GatewayForwardingContext? forwardingContext, string packId, int version, ActivatePolicyRevisionRequest request, CancellationToken cancellationToken);
|
||||||
}
|
|
||||||
|
Task<PolicyEngineResponse<CvssScoreReceipt>> CreateCvssReceiptAsync(GatewayForwardingContext? forwardingContext, CreateCvssReceiptRequest request, CancellationToken cancellationToken);
|
||||||
|
|
||||||
|
Task<PolicyEngineResponse<CvssScoreReceipt>> GetCvssReceiptAsync(GatewayForwardingContext? forwardingContext, string receiptId, CancellationToken cancellationToken);
|
||||||
|
|
||||||
|
Task<PolicyEngineResponse<CvssScoreReceipt>> AmendCvssReceiptAsync(GatewayForwardingContext? forwardingContext, string receiptId, AmendCvssReceiptRequest request, CancellationToken cancellationToken);
|
||||||
|
|
||||||
|
Task<PolicyEngineResponse<IReadOnlyList<ReceiptHistoryEntry>>> GetCvssReceiptHistoryAsync(GatewayForwardingContext? forwardingContext, string receiptId, CancellationToken cancellationToken);
|
||||||
|
|
||||||
|
Task<PolicyEngineResponse<IReadOnlyList<CvssPolicy>>> ListCvssPoliciesAsync(GatewayForwardingContext? forwardingContext, CancellationToken cancellationToken);
|
||||||
|
}
|
||||||
|
|||||||
@@ -5,13 +5,15 @@ using System.Net.Http;
|
|||||||
using System.Net.Http.Json;
|
using System.Net.Http.Json;
|
||||||
using System.Text.Json;
|
using System.Text.Json;
|
||||||
using Microsoft.AspNetCore.Http;
|
using Microsoft.AspNetCore.Http;
|
||||||
using Microsoft.AspNetCore.Mvc;
|
using Microsoft.AspNetCore.Mvc;
|
||||||
using Microsoft.Extensions.Logging;
|
using Microsoft.Extensions.Logging;
|
||||||
using Microsoft.Extensions.Options;
|
using Microsoft.Extensions.Options;
|
||||||
using StellaOps.Policy.Gateway.Contracts;
|
using StellaOps.Policy.Gateway.Contracts;
|
||||||
using StellaOps.Policy.Gateway.Infrastructure;
|
using StellaOps.Policy.Gateway.Infrastructure;
|
||||||
using StellaOps.Policy.Gateway.Options;
|
using StellaOps.Policy.Gateway.Options;
|
||||||
using StellaOps.Policy.Gateway.Services;
|
using StellaOps.Policy.Gateway.Services;
|
||||||
|
using StellaOps.Policy.Scoring;
|
||||||
|
using StellaOps.Policy.Scoring.Receipts;
|
||||||
|
|
||||||
namespace StellaOps.Policy.Gateway.Clients;
|
namespace StellaOps.Policy.Gateway.Clients;
|
||||||
|
|
||||||
@@ -85,18 +87,73 @@ internal sealed class PolicyEngineClient : IPolicyEngineClient
|
|||||||
request,
|
request,
|
||||||
cancellationToken);
|
cancellationToken);
|
||||||
|
|
||||||
public Task<PolicyEngineResponse<PolicyRevisionActivationDto>> ActivatePolicyRevisionAsync(
|
public Task<PolicyEngineResponse<PolicyRevisionActivationDto>> ActivatePolicyRevisionAsync(
|
||||||
GatewayForwardingContext? forwardingContext,
|
GatewayForwardingContext? forwardingContext,
|
||||||
string packId,
|
string packId,
|
||||||
int version,
|
int version,
|
||||||
ActivatePolicyRevisionRequest request,
|
ActivatePolicyRevisionRequest request,
|
||||||
CancellationToken cancellationToken)
|
CancellationToken cancellationToken)
|
||||||
=> SendAsync<PolicyRevisionActivationDto>(
|
=> SendAsync<PolicyRevisionActivationDto>(
|
||||||
HttpMethod.Post,
|
HttpMethod.Post,
|
||||||
$"api/policy/packs/{Uri.EscapeDataString(packId)}/revisions/{version}:activate",
|
$"api/policy/packs/{Uri.EscapeDataString(packId)}/revisions/{version}:activate",
|
||||||
forwardingContext,
|
forwardingContext,
|
||||||
request,
|
request,
|
||||||
cancellationToken);
|
cancellationToken);
|
||||||
|
|
||||||
|
public Task<PolicyEngineResponse<CvssScoreReceipt>> CreateCvssReceiptAsync(
|
||||||
|
GatewayForwardingContext? forwardingContext,
|
||||||
|
CreateCvssReceiptRequest request,
|
||||||
|
CancellationToken cancellationToken)
|
||||||
|
=> SendAsync<CvssScoreReceipt>(
|
||||||
|
HttpMethod.Post,
|
||||||
|
"api/cvss/receipts",
|
||||||
|
forwardingContext,
|
||||||
|
request,
|
||||||
|
cancellationToken);
|
||||||
|
|
||||||
|
public Task<PolicyEngineResponse<CvssScoreReceipt>> GetCvssReceiptAsync(
|
||||||
|
GatewayForwardingContext? forwardingContext,
|
||||||
|
string receiptId,
|
||||||
|
CancellationToken cancellationToken)
|
||||||
|
=> SendAsync<CvssScoreReceipt>(
|
||||||
|
HttpMethod.Get,
|
||||||
|
$"api/cvss/receipts/{Uri.EscapeDataString(receiptId)}",
|
||||||
|
forwardingContext,
|
||||||
|
content: null,
|
||||||
|
cancellationToken);
|
||||||
|
|
||||||
|
public Task<PolicyEngineResponse<CvssScoreReceipt>> AmendCvssReceiptAsync(
|
||||||
|
GatewayForwardingContext? forwardingContext,
|
||||||
|
string receiptId,
|
||||||
|
AmendCvssReceiptRequest request,
|
||||||
|
CancellationToken cancellationToken)
|
||||||
|
=> SendAsync<CvssScoreReceipt>(
|
||||||
|
HttpMethod.Put,
|
||||||
|
$"api/cvss/receipts/{Uri.EscapeDataString(receiptId)}/amend",
|
||||||
|
forwardingContext,
|
||||||
|
request,
|
||||||
|
cancellationToken);
|
||||||
|
|
||||||
|
public Task<PolicyEngineResponse<IReadOnlyList<ReceiptHistoryEntry>>> GetCvssReceiptHistoryAsync(
|
||||||
|
GatewayForwardingContext? forwardingContext,
|
||||||
|
string receiptId,
|
||||||
|
CancellationToken cancellationToken)
|
||||||
|
=> SendAsync<IReadOnlyList<ReceiptHistoryEntry>>(
|
||||||
|
HttpMethod.Get,
|
||||||
|
$"api/cvss/receipts/{Uri.EscapeDataString(receiptId)}/history",
|
||||||
|
forwardingContext,
|
||||||
|
content: null,
|
||||||
|
cancellationToken);
|
||||||
|
|
||||||
|
public Task<PolicyEngineResponse<IReadOnlyList<CvssPolicy>>> ListCvssPoliciesAsync(
|
||||||
|
GatewayForwardingContext? forwardingContext,
|
||||||
|
CancellationToken cancellationToken)
|
||||||
|
=> SendAsync<IReadOnlyList<CvssPolicy>>(
|
||||||
|
HttpMethod.Get,
|
||||||
|
"api/cvss/policies",
|
||||||
|
forwardingContext,
|
||||||
|
content: null,
|
||||||
|
cancellationToken);
|
||||||
|
|
||||||
private async Task<PolicyEngineResponse<TSuccess>> SendAsync<TSuccess>(
|
private async Task<PolicyEngineResponse<TSuccess>> SendAsync<TSuccess>(
|
||||||
HttpMethod method,
|
HttpMethod method,
|
||||||
|
|||||||
@@ -0,0 +1,33 @@
|
|||||||
|
using System;
|
||||||
|
using System.Collections.Generic;
|
||||||
|
using System.ComponentModel.DataAnnotations;
|
||||||
|
using StellaOps.Attestor.Envelope;
|
||||||
|
using StellaOps.Policy.Scoring;
|
||||||
|
using StellaOps.Policy.Scoring.Receipts;
|
||||||
|
|
||||||
|
namespace StellaOps.Policy.Gateway.Contracts;
|
||||||
|
|
||||||
|
public sealed record CreateCvssReceiptRequest(
|
||||||
|
[Required] string VulnerabilityId,
|
||||||
|
[Required] CvssPolicy Policy,
|
||||||
|
[Required] CvssBaseMetrics BaseMetrics,
|
||||||
|
CvssThreatMetrics? ThreatMetrics,
|
||||||
|
CvssEnvironmentalMetrics? EnvironmentalMetrics,
|
||||||
|
CvssSupplementalMetrics? SupplementalMetrics,
|
||||||
|
IReadOnlyList<CvssEvidenceItem>? Evidence,
|
||||||
|
EnvelopeKey? SigningKey,
|
||||||
|
string? CreatedBy,
|
||||||
|
DateTimeOffset? CreatedAt);
|
||||||
|
|
||||||
|
public sealed record AmendCvssReceiptRequest(
|
||||||
|
[Required] string Field,
|
||||||
|
string? PreviousValue,
|
||||||
|
string? NewValue,
|
||||||
|
[Required] string Reason,
|
||||||
|
string? ReferenceUri,
|
||||||
|
EnvelopeKey? SigningKey,
|
||||||
|
string? Actor);
|
||||||
|
|
||||||
|
public sealed record CvssReceiptHistoryResponse(
|
||||||
|
string ReceiptId,
|
||||||
|
IReadOnlyList<ReceiptHistoryEntry> History);
|
||||||
@@ -279,11 +279,11 @@ policyPacks.MapPost("/{packId}/revisions", async Task<IResult> (
|
|||||||
})
|
})
|
||||||
.RequireAuthorization(policy => policy.RequireStellaOpsScopes(StellaOpsScopes.PolicyAuthor));
|
.RequireAuthorization(policy => policy.RequireStellaOpsScopes(StellaOpsScopes.PolicyAuthor));
|
||||||
|
|
||||||
policyPacks.MapPost("/{packId}/revisions/{version:int}:activate", async Task<IResult> (
|
policyPacks.MapPost("/{packId}/revisions/{version:int}:activate", async Task<IResult> (
|
||||||
HttpContext context,
|
HttpContext context,
|
||||||
string packId,
|
string packId,
|
||||||
int version,
|
int version,
|
||||||
ActivatePolicyRevisionRequest request,
|
ActivatePolicyRevisionRequest request,
|
||||||
IPolicyEngineClient client,
|
IPolicyEngineClient client,
|
||||||
PolicyEngineTokenProvider tokenProvider,
|
PolicyEngineTokenProvider tokenProvider,
|
||||||
PolicyGatewayMetrics metrics,
|
PolicyGatewayMetrics metrics,
|
||||||
@@ -330,13 +330,144 @@ policyPacks.MapPost("/{packId}/revisions/{version:int}:activate", async Task<IRe
|
|||||||
var logger = loggerFactory.CreateLogger("StellaOps.Policy.Gateway.Activation");
|
var logger = loggerFactory.CreateLogger("StellaOps.Policy.Gateway.Activation");
|
||||||
LogActivation(logger, packId, version, outcome, source, response.StatusCode);
|
LogActivation(logger, packId, version, outcome, source, response.StatusCode);
|
||||||
|
|
||||||
return response.ToMinimalResult();
|
return response.ToMinimalResult();
|
||||||
})
|
})
|
||||||
.RequireAuthorization(policy => policy.RequireStellaOpsScopes(
|
.RequireAuthorization(policy => policy.RequireStellaOpsScopes(
|
||||||
StellaOpsScopes.PolicyOperate,
|
StellaOpsScopes.PolicyOperate,
|
||||||
StellaOpsScopes.PolicyActivate));
|
StellaOpsScopes.PolicyActivate));
|
||||||
|
|
||||||
app.Run();
|
var cvss = app.MapGroup("/api/cvss")
|
||||||
|
.WithTags("CVSS Receipts");
|
||||||
|
|
||||||
|
cvss.MapPost("/receipts", async Task<IResult>(
|
||||||
|
HttpContext context,
|
||||||
|
CreateCvssReceiptRequest request,
|
||||||
|
IPolicyEngineClient client,
|
||||||
|
PolicyEngineTokenProvider tokenProvider,
|
||||||
|
CancellationToken cancellationToken) =>
|
||||||
|
{
|
||||||
|
if (request is null)
|
||||||
|
{
|
||||||
|
return Results.BadRequest(new ProblemDetails
|
||||||
|
{
|
||||||
|
Title = "Request body required.",
|
||||||
|
Status = StatusCodes.Status400BadRequest
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
GatewayForwardingContext? forwardingContext = null;
|
||||||
|
if (GatewayForwardingContext.TryCreate(context, out var callerContext))
|
||||||
|
{
|
||||||
|
forwardingContext = callerContext;
|
||||||
|
}
|
||||||
|
else if (!tokenProvider.IsEnabled)
|
||||||
|
{
|
||||||
|
return Results.Unauthorized();
|
||||||
|
}
|
||||||
|
|
||||||
|
var response = await client.CreateCvssReceiptAsync(forwardingContext, request, cancellationToken).ConfigureAwait(false);
|
||||||
|
return response.ToMinimalResult();
|
||||||
|
})
|
||||||
|
.RequireAuthorization(policy => policy.RequireStellaOpsScopes(StellaOpsScopes.PolicyRun));
|
||||||
|
|
||||||
|
cvss.MapGet("/receipts/{receiptId}", async Task<IResult>(
|
||||||
|
HttpContext context,
|
||||||
|
string receiptId,
|
||||||
|
IPolicyEngineClient client,
|
||||||
|
PolicyEngineTokenProvider tokenProvider,
|
||||||
|
CancellationToken cancellationToken) =>
|
||||||
|
{
|
||||||
|
GatewayForwardingContext? forwardingContext = null;
|
||||||
|
if (GatewayForwardingContext.TryCreate(context, out var callerContext))
|
||||||
|
{
|
||||||
|
forwardingContext = callerContext;
|
||||||
|
}
|
||||||
|
else if (!tokenProvider.IsEnabled)
|
||||||
|
{
|
||||||
|
return Results.Unauthorized();
|
||||||
|
}
|
||||||
|
|
||||||
|
var response = await client.GetCvssReceiptAsync(forwardingContext, receiptId, cancellationToken).ConfigureAwait(false);
|
||||||
|
return response.ToMinimalResult();
|
||||||
|
})
|
||||||
|
.RequireAuthorization(policy => policy.RequireStellaOpsScopes(StellaOpsScopes.FindingsRead));
|
||||||
|
|
||||||
|
cvss.MapPut("/receipts/{receiptId}/amend", async Task<IResult>(
|
||||||
|
HttpContext context,
|
||||||
|
string receiptId,
|
||||||
|
AmendCvssReceiptRequest request,
|
||||||
|
IPolicyEngineClient client,
|
||||||
|
PolicyEngineTokenProvider tokenProvider,
|
||||||
|
CancellationToken cancellationToken) =>
|
||||||
|
{
|
||||||
|
if (request is null)
|
||||||
|
{
|
||||||
|
return Results.BadRequest(new ProblemDetails
|
||||||
|
{
|
||||||
|
Title = "Request body required.",
|
||||||
|
Status = StatusCodes.Status400BadRequest
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
GatewayForwardingContext? forwardingContext = null;
|
||||||
|
if (GatewayForwardingContext.TryCreate(context, out var callerContext))
|
||||||
|
{
|
||||||
|
forwardingContext = callerContext;
|
||||||
|
}
|
||||||
|
else if (!tokenProvider.IsEnabled)
|
||||||
|
{
|
||||||
|
return Results.Unauthorized();
|
||||||
|
}
|
||||||
|
|
||||||
|
var response = await client.AmendCvssReceiptAsync(forwardingContext, receiptId, request, cancellationToken).ConfigureAwait(false);
|
||||||
|
return response.ToMinimalResult();
|
||||||
|
})
|
||||||
|
.RequireAuthorization(policy => policy.RequireStellaOpsScopes(StellaOpsScopes.PolicyRun));
|
||||||
|
|
||||||
|
cvss.MapGet("/receipts/{receiptId}/history", async Task<IResult>(
|
||||||
|
HttpContext context,
|
||||||
|
string receiptId,
|
||||||
|
IPolicyEngineClient client,
|
||||||
|
PolicyEngineTokenProvider tokenProvider,
|
||||||
|
CancellationToken cancellationToken) =>
|
||||||
|
{
|
||||||
|
GatewayForwardingContext? forwardingContext = null;
|
||||||
|
if (GatewayForwardingContext.TryCreate(context, out var callerContext))
|
||||||
|
{
|
||||||
|
forwardingContext = callerContext;
|
||||||
|
}
|
||||||
|
else if (!tokenProvider.IsEnabled)
|
||||||
|
{
|
||||||
|
return Results.Unauthorized();
|
||||||
|
}
|
||||||
|
|
||||||
|
var response = await client.GetCvssReceiptHistoryAsync(forwardingContext, receiptId, cancellationToken).ConfigureAwait(false);
|
||||||
|
return response.ToMinimalResult();
|
||||||
|
})
|
||||||
|
.RequireAuthorization(policy => policy.RequireStellaOpsScopes(StellaOpsScopes.FindingsRead));
|
||||||
|
|
||||||
|
cvss.MapGet("/policies", async Task<IResult>(
|
||||||
|
HttpContext context,
|
||||||
|
IPolicyEngineClient client,
|
||||||
|
PolicyEngineTokenProvider tokenProvider,
|
||||||
|
CancellationToken cancellationToken) =>
|
||||||
|
{
|
||||||
|
GatewayForwardingContext? forwardingContext = null;
|
||||||
|
if (GatewayForwardingContext.TryCreate(context, out var callerContext))
|
||||||
|
{
|
||||||
|
forwardingContext = callerContext;
|
||||||
|
}
|
||||||
|
else if (!tokenProvider.IsEnabled)
|
||||||
|
{
|
||||||
|
return Results.Unauthorized();
|
||||||
|
}
|
||||||
|
|
||||||
|
var response = await client.ListCvssPoliciesAsync(forwardingContext, cancellationToken).ConfigureAwait(false);
|
||||||
|
return response.ToMinimalResult();
|
||||||
|
})
|
||||||
|
.RequireAuthorization(policy => policy.RequireStellaOpsScopes(StellaOpsScopes.FindingsRead));
|
||||||
|
|
||||||
|
app.Run();
|
||||||
|
|
||||||
static IAsyncPolicy<HttpResponseMessage> CreateAuthorityRetryPolicy(IServiceProvider provider)
|
static IAsyncPolicy<HttpResponseMessage> CreateAuthorityRetryPolicy(IServiceProvider provider)
|
||||||
{
|
{
|
||||||
|
|||||||
@@ -16,6 +16,7 @@
|
|||||||
<ProjectReference Include="../../Authority/StellaOps.Authority/StellaOps.Auth.Client/StellaOps.Auth.Client.csproj" />
|
<ProjectReference Include="../../Authority/StellaOps.Authority/StellaOps.Auth.Client/StellaOps.Auth.Client.csproj" />
|
||||||
<ProjectReference Include="../../Authority/StellaOps.Authority/StellaOps.Auth.ServerIntegration/StellaOps.Auth.ServerIntegration.csproj" />
|
<ProjectReference Include="../../Authority/StellaOps.Authority/StellaOps.Auth.ServerIntegration/StellaOps.Auth.ServerIntegration.csproj" />
|
||||||
<ProjectReference Include="../../AirGap/StellaOps.AirGap.Policy/StellaOps.AirGap.Policy/StellaOps.AirGap.Policy.csproj" />
|
<ProjectReference Include="../../AirGap/StellaOps.AirGap.Policy/StellaOps.AirGap.Policy/StellaOps.AirGap.Policy.csproj" />
|
||||||
|
<ProjectReference Include="../StellaOps.Policy.Scoring/StellaOps.Policy.Scoring.csproj" />
|
||||||
</ItemGroup>
|
</ItemGroup>
|
||||||
<ItemGroup>
|
<ItemGroup>
|
||||||
<PackageReference Include="Microsoft.Extensions.Http.Polly" Version="10.0.0" />
|
<PackageReference Include="Microsoft.Extensions.Http.Polly" Version="10.0.0" />
|
||||||
|
|||||||
@@ -12,7 +12,7 @@ import jsPDF from './jspdf.stub';
|
|||||||
imports: [CommonModule],
|
imports: [CommonModule],
|
||||||
changeDetection: ChangeDetectionStrategy.OnPush,
|
changeDetection: ChangeDetectionStrategy.OnPush,
|
||||||
template: `
|
template: `
|
||||||
<section class="expl" aria-busy="{{ loading }}">
|
<section class="expl" [attr.aria-busy]="loading">
|
||||||
<header class="expl__header" *ngIf="result">
|
<header class="expl__header" *ngIf="result">
|
||||||
<div>
|
<div>
|
||||||
<p class="expl__eyebrow">Policy Studio · Explain</p>
|
<p class="expl__eyebrow">Policy Studio · Explain</p>
|
||||||
|
|||||||
@@ -9,7 +9,7 @@ import { ActivatedRoute } from '@angular/router';
|
|||||||
imports: [CommonModule, ReactiveFormsModule],
|
imports: [CommonModule, ReactiveFormsModule],
|
||||||
changeDetection: ChangeDetectionStrategy.OnPush,
|
changeDetection: ChangeDetectionStrategy.OnPush,
|
||||||
template: `
|
template: `
|
||||||
<section class="rb" aria-busy="false">
|
<section class="rb" [attr.aria-busy]="false">
|
||||||
<header class="rb__header">
|
<header class="rb__header">
|
||||||
<div>
|
<div>
|
||||||
<p class="rb__eyebrow">Policy Studio · Rule Builder</p>
|
<p class="rb__eyebrow">Policy Studio · Rule Builder</p>
|
||||||
|
|||||||
@@ -7,7 +7,7 @@
|
|||||||
</PropertyGroup>
|
</PropertyGroup>
|
||||||
|
|
||||||
<ItemGroup>
|
<ItemGroup>
|
||||||
<PackageReference Include="MongoDB.Driver" Version="3.5.0" />
|
<ProjectReference Include="..\..\Concelier\__Libraries\StellaOps.Concelier.Models\StellaOps.Concelier.Models.csproj" />
|
||||||
</ItemGroup>
|
</ItemGroup>
|
||||||
|
|
||||||
</Project>
|
</Project>
|
||||||
|
|||||||
Reference in New Issue
Block a user