Rename Concelier Source modules to Connector
This commit is contained in:
40
src/StellaOps.Concelier.Connector.CertBund/AGENTS.md
Normal file
40
src/StellaOps.Concelier.Connector.CertBund/AGENTS.md
Normal file
@@ -0,0 +1,40 @@
|
||||
# AGENTS
|
||||
## Role
|
||||
Deliver a connector for Germany’s CERT-Bund advisories so Concelier can ingest, normalise, and enrich BSI alerts alongside other national feeds.
|
||||
|
||||
## Scope
|
||||
- Identify the authoritative CERT-Bund advisory feed(s) (RSS/Atom, JSON, CSV, or HTML).
|
||||
- Implement fetch/cursor logic with proper windowing, dedupe, and failure backoff.
|
||||
- Parse advisory detail pages for summary, affected products/vendors, mitigation, and references.
|
||||
- Map advisories into canonical `Advisory` objects including aliases, references, affected packages, and provenance/range primitives.
|
||||
- Provide deterministic fixtures and regression tests.
|
||||
|
||||
## Participants
|
||||
- `Source.Common` (HTTP/fetch utilities, DTO storage).
|
||||
- `Storage.Mongo` (raw/document/DTO/advisory stores, source state).
|
||||
- `Concelier.Models` (canonical data model).
|
||||
- `Concelier.Testing` (integration harness, snapshot utilities).
|
||||
|
||||
## Interfaces & Contracts
|
||||
- Job kinds: `certbund:fetch`, `certbund:parse`, `certbund:map`.
|
||||
- Persist upstream metadata (ETag/Last-Modified) if provided.
|
||||
- Alias set should include CERT-Bund ID and referenced CVE entries.
|
||||
|
||||
## In/Out of scope
|
||||
In scope:
|
||||
- End-to-end connector implementation with deterministic tests and range primitive coverage.
|
||||
- Baseline logging/metrics for pipeline observability.
|
||||
|
||||
Out of scope:
|
||||
- Non-advisory CERT-Bund digests or newsletters.
|
||||
- Downstream exporter changes.
|
||||
|
||||
## Observability & Security Expectations
|
||||
- Log fetch attempts, item counts, and mapping metrics.
|
||||
- Sanitize HTML thoroughly before persistence.
|
||||
- Handle transient failures gracefully with exponential backoff and failure records in source state.
|
||||
|
||||
## Tests
|
||||
- Add `StellaOps.Concelier.Connector.CertBund.Tests` covering fetch/parse/map with canned fixtures.
|
||||
- Snapshot canonical advisories; support regeneration via environment flag.
|
||||
- Ensure deterministic ordering, casing, and timestamps.
|
||||
435
src/StellaOps.Concelier.Connector.CertBund/CertBundConnector.cs
Normal file
435
src/StellaOps.Concelier.Connector.CertBund/CertBundConnector.cs
Normal file
@@ -0,0 +1,435 @@
|
||||
using System;
|
||||
using System.Collections.Generic;
|
||||
using System.Linq;
|
||||
using System.Text.Json;
|
||||
using System.Threading;
|
||||
using System.Threading.Tasks;
|
||||
using Microsoft.Extensions.Logging;
|
||||
using Microsoft.Extensions.Options;
|
||||
using MongoDB.Bson;
|
||||
using StellaOps.Concelier.Connector.CertBund.Configuration;
|
||||
using StellaOps.Concelier.Connector.CertBund.Internal;
|
||||
using StellaOps.Concelier.Connector.Common;
|
||||
using StellaOps.Concelier.Connector.Common.Fetch;
|
||||
using StellaOps.Concelier.Connector.Common.Html;
|
||||
using StellaOps.Concelier.Storage.Mongo;
|
||||
using StellaOps.Concelier.Storage.Mongo.Advisories;
|
||||
using StellaOps.Concelier.Storage.Mongo.Documents;
|
||||
using StellaOps.Concelier.Storage.Mongo.Dtos;
|
||||
using StellaOps.Plugin;
|
||||
|
||||
namespace StellaOps.Concelier.Connector.CertBund;
|
||||
|
||||
public sealed class CertBundConnector : IFeedConnector
|
||||
{
|
||||
private static readonly JsonSerializerOptions SerializerOptions = new(JsonSerializerDefaults.Web)
|
||||
{
|
||||
PropertyNameCaseInsensitive = true,
|
||||
DefaultIgnoreCondition = System.Text.Json.Serialization.JsonIgnoreCondition.WhenWritingNull,
|
||||
};
|
||||
|
||||
private readonly CertBundFeedClient _feedClient;
|
||||
private readonly CertBundDetailParser _detailParser;
|
||||
private readonly SourceFetchService _fetchService;
|
||||
private readonly RawDocumentStorage _rawDocumentStorage;
|
||||
private readonly IDocumentStore _documentStore;
|
||||
private readonly IDtoStore _dtoStore;
|
||||
private readonly IAdvisoryStore _advisoryStore;
|
||||
private readonly ISourceStateRepository _stateRepository;
|
||||
private readonly CertBundOptions _options;
|
||||
private readonly TimeProvider _timeProvider;
|
||||
private readonly CertBundDiagnostics _diagnostics;
|
||||
private readonly ILogger<CertBundConnector> _logger;
|
||||
|
||||
public CertBundConnector(
|
||||
CertBundFeedClient feedClient,
|
||||
CertBundDetailParser detailParser,
|
||||
SourceFetchService fetchService,
|
||||
RawDocumentStorage rawDocumentStorage,
|
||||
IDocumentStore documentStore,
|
||||
IDtoStore dtoStore,
|
||||
IAdvisoryStore advisoryStore,
|
||||
ISourceStateRepository stateRepository,
|
||||
IOptions<CertBundOptions> options,
|
||||
CertBundDiagnostics diagnostics,
|
||||
TimeProvider? timeProvider,
|
||||
ILogger<CertBundConnector> logger)
|
||||
{
|
||||
_feedClient = feedClient ?? throw new ArgumentNullException(nameof(feedClient));
|
||||
_detailParser = detailParser ?? throw new ArgumentNullException(nameof(detailParser));
|
||||
_fetchService = fetchService ?? throw new ArgumentNullException(nameof(fetchService));
|
||||
_rawDocumentStorage = rawDocumentStorage ?? throw new ArgumentNullException(nameof(rawDocumentStorage));
|
||||
_documentStore = documentStore ?? throw new ArgumentNullException(nameof(documentStore));
|
||||
_dtoStore = dtoStore ?? throw new ArgumentNullException(nameof(dtoStore));
|
||||
_advisoryStore = advisoryStore ?? throw new ArgumentNullException(nameof(advisoryStore));
|
||||
_stateRepository = stateRepository ?? throw new ArgumentNullException(nameof(stateRepository));
|
||||
_options = (options ?? throw new ArgumentNullException(nameof(options))).Value ?? throw new ArgumentNullException(nameof(options));
|
||||
_options.Validate();
|
||||
_diagnostics = diagnostics ?? throw new ArgumentNullException(nameof(diagnostics));
|
||||
_timeProvider = timeProvider ?? TimeProvider.System;
|
||||
_logger = logger ?? throw new ArgumentNullException(nameof(logger));
|
||||
}
|
||||
|
||||
public string SourceName => CertBundConnectorPlugin.SourceName;
|
||||
|
||||
public async Task FetchAsync(IServiceProvider services, CancellationToken cancellationToken)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(services);
|
||||
|
||||
var cursor = await GetCursorAsync(cancellationToken).ConfigureAwait(false);
|
||||
var now = _timeProvider.GetUtcNow();
|
||||
IReadOnlyList<CertBundFeedItem> feedItems;
|
||||
|
||||
_diagnostics.FeedFetchAttempt();
|
||||
try
|
||||
{
|
||||
feedItems = await _feedClient.LoadAsync(cancellationToken).ConfigureAwait(false);
|
||||
_diagnostics.FeedFetchSuccess(feedItems.Count);
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
_logger.LogError(ex, "CERT-Bund feed fetch failed");
|
||||
_diagnostics.FeedFetchFailure();
|
||||
await _stateRepository.MarkFailureAsync(SourceName, now, _options.FailureBackoff, ex.Message, cancellationToken).ConfigureAwait(false);
|
||||
throw;
|
||||
}
|
||||
|
||||
var coverageDays = CalculateCoverageDays(feedItems, now);
|
||||
_diagnostics.RecordFeedCoverage(coverageDays);
|
||||
|
||||
if (feedItems.Count == 0)
|
||||
{
|
||||
await UpdateCursorAsync(cursor.WithLastFetch(now), cancellationToken).ConfigureAwait(false);
|
||||
return;
|
||||
}
|
||||
|
||||
var pendingDocuments = cursor.PendingDocuments.ToHashSet();
|
||||
var pendingMappings = cursor.PendingMappings.ToHashSet();
|
||||
var knownAdvisories = new HashSet<string>(cursor.KnownAdvisories, StringComparer.OrdinalIgnoreCase);
|
||||
var processed = 0;
|
||||
var alreadyKnown = 0;
|
||||
var notModified = 0;
|
||||
var detailFailures = 0;
|
||||
var truncated = false;
|
||||
var latestPublished = cursor.LastPublished ?? DateTimeOffset.MinValue;
|
||||
|
||||
foreach (var item in feedItems.OrderByDescending(static i => i.Published))
|
||||
{
|
||||
cancellationToken.ThrowIfCancellationRequested();
|
||||
|
||||
if (knownAdvisories.Contains(item.AdvisoryId))
|
||||
{
|
||||
alreadyKnown++;
|
||||
continue;
|
||||
}
|
||||
|
||||
if (processed >= _options.MaxAdvisoriesPerFetch)
|
||||
{
|
||||
truncated = true;
|
||||
break;
|
||||
}
|
||||
|
||||
try
|
||||
{
|
||||
_diagnostics.DetailFetchAttempt();
|
||||
var existing = await _documentStore.FindBySourceAndUriAsync(SourceName, item.DetailUri.ToString(), cancellationToken).ConfigureAwait(false);
|
||||
var request = new SourceFetchRequest(CertBundOptions.HttpClientName, SourceName, item.DetailUri)
|
||||
{
|
||||
AcceptHeaders = new[] { "application/json", "text/json" },
|
||||
Metadata = CertBundDocumentMetadata.CreateMetadata(item),
|
||||
ETag = existing?.Etag,
|
||||
LastModified = existing?.LastModified,
|
||||
TimeoutOverride = _options.RequestTimeout,
|
||||
};
|
||||
|
||||
var result = await _fetchService.FetchAsync(request, cancellationToken).ConfigureAwait(false);
|
||||
if (result.IsNotModified)
|
||||
{
|
||||
_diagnostics.DetailFetchNotModified();
|
||||
notModified++;
|
||||
knownAdvisories.Add(item.AdvisoryId);
|
||||
continue;
|
||||
}
|
||||
|
||||
if (!result.IsSuccess || result.Document is null)
|
||||
{
|
||||
_diagnostics.DetailFetchFailure("skipped");
|
||||
detailFailures++;
|
||||
continue;
|
||||
}
|
||||
|
||||
_diagnostics.DetailFetchSuccess();
|
||||
pendingDocuments.Add(result.Document.Id);
|
||||
pendingMappings.Remove(result.Document.Id);
|
||||
knownAdvisories.Add(item.AdvisoryId);
|
||||
processed++;
|
||||
|
||||
if (_options.RequestDelay > TimeSpan.Zero)
|
||||
{
|
||||
await Task.Delay(_options.RequestDelay, cancellationToken).ConfigureAwait(false);
|
||||
}
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
_logger.LogError(ex, "CERT-Bund detail fetch failed for {AdvisoryId}", item.AdvisoryId);
|
||||
_diagnostics.DetailFetchFailure("exception");
|
||||
detailFailures++;
|
||||
await _stateRepository.MarkFailureAsync(SourceName, now, _options.FailureBackoff, ex.Message, cancellationToken).ConfigureAwait(false);
|
||||
throw;
|
||||
}
|
||||
|
||||
if (item.Published > latestPublished)
|
||||
{
|
||||
latestPublished = item.Published;
|
||||
}
|
||||
}
|
||||
|
||||
_diagnostics.DetailFetchEnqueued(processed);
|
||||
|
||||
if (feedItems.Count > 0 || processed > 0 || detailFailures > 0)
|
||||
{
|
||||
_logger.LogInformation(
|
||||
"CERT-Bund fetch cycle: feed items {FeedItems}, enqueued {Enqueued}, already known {Known}, not modified {NotModified}, detail failures {DetailFailures}, pending documents {PendingDocuments}, pending mappings {PendingMappings}, truncated {Truncated}, coverageDays={CoverageDays}",
|
||||
feedItems.Count,
|
||||
processed,
|
||||
alreadyKnown,
|
||||
notModified,
|
||||
detailFailures,
|
||||
pendingDocuments.Count,
|
||||
pendingMappings.Count,
|
||||
truncated,
|
||||
coverageDays ?? double.NaN);
|
||||
}
|
||||
|
||||
var trimmedKnown = knownAdvisories.Count > _options.MaxKnownAdvisories
|
||||
? knownAdvisories.OrderByDescending(id => id, StringComparer.OrdinalIgnoreCase)
|
||||
.Take(_options.MaxKnownAdvisories)
|
||||
.ToArray()
|
||||
: knownAdvisories.ToArray();
|
||||
|
||||
var updatedCursor = cursor
|
||||
.WithPendingDocuments(pendingDocuments)
|
||||
.WithPendingMappings(pendingMappings)
|
||||
.WithKnownAdvisories(trimmedKnown)
|
||||
.WithLastPublished(latestPublished == DateTimeOffset.MinValue ? cursor.LastPublished : latestPublished)
|
||||
.WithLastFetch(now);
|
||||
|
||||
await UpdateCursorAsync(updatedCursor, cancellationToken).ConfigureAwait(false);
|
||||
}
|
||||
|
||||
public async Task ParseAsync(IServiceProvider services, CancellationToken cancellationToken)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(services);
|
||||
|
||||
var cursor = await GetCursorAsync(cancellationToken).ConfigureAwait(false);
|
||||
if (cursor.PendingDocuments.Count == 0)
|
||||
{
|
||||
return;
|
||||
}
|
||||
|
||||
var remainingDocuments = cursor.PendingDocuments.ToHashSet();
|
||||
var pendingMappings = cursor.PendingMappings.ToHashSet();
|
||||
var now = _timeProvider.GetUtcNow();
|
||||
var parsedCount = 0;
|
||||
var failedCount = 0;
|
||||
|
||||
foreach (var documentId in cursor.PendingDocuments)
|
||||
{
|
||||
cancellationToken.ThrowIfCancellationRequested();
|
||||
|
||||
var document = await _documentStore.FindAsync(documentId, cancellationToken).ConfigureAwait(false);
|
||||
if (document is null)
|
||||
{
|
||||
remainingDocuments.Remove(documentId);
|
||||
pendingMappings.Remove(documentId);
|
||||
continue;
|
||||
}
|
||||
|
||||
if (!document.GridFsId.HasValue)
|
||||
{
|
||||
await _documentStore.UpdateStatusAsync(document.Id, DocumentStatuses.Failed, cancellationToken).ConfigureAwait(false);
|
||||
remainingDocuments.Remove(documentId);
|
||||
pendingMappings.Remove(documentId);
|
||||
_diagnostics.ParseFailure("missing_payload");
|
||||
failedCount++;
|
||||
continue;
|
||||
}
|
||||
|
||||
byte[] payload;
|
||||
try
|
||||
{
|
||||
payload = await _rawDocumentStorage.DownloadAsync(document.GridFsId.Value, cancellationToken).ConfigureAwait(false);
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
_logger.LogError(ex, "CERT-Bund unable to download document {DocumentId}", document.Id);
|
||||
_diagnostics.ParseFailure("download_failed");
|
||||
throw;
|
||||
}
|
||||
|
||||
CertBundAdvisoryDto dto;
|
||||
try
|
||||
{
|
||||
dto = _detailParser.Parse(new Uri(document.Uri), new Uri(document.Metadata?["certbund.portalUri"] ?? document.Uri), payload);
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
_logger.LogError(ex, "CERT-Bund failed to parse advisory detail {DocumentId}", document.Id);
|
||||
await _documentStore.UpdateStatusAsync(document.Id, DocumentStatuses.Failed, cancellationToken).ConfigureAwait(false);
|
||||
remainingDocuments.Remove(documentId);
|
||||
pendingMappings.Remove(documentId);
|
||||
_diagnostics.ParseFailure("parse_error");
|
||||
failedCount++;
|
||||
continue;
|
||||
}
|
||||
|
||||
_diagnostics.ParseSuccess(dto.Products.Count, dto.CveIds.Count);
|
||||
parsedCount++;
|
||||
|
||||
var bson = BsonDocument.Parse(JsonSerializer.Serialize(dto, SerializerOptions));
|
||||
var dtoRecord = new DtoRecord(Guid.NewGuid(), document.Id, SourceName, "cert-bund.detail.v1", bson, now);
|
||||
await _dtoStore.UpsertAsync(dtoRecord, cancellationToken).ConfigureAwait(false);
|
||||
await _documentStore.UpdateStatusAsync(document.Id, DocumentStatuses.PendingMap, cancellationToken).ConfigureAwait(false);
|
||||
|
||||
remainingDocuments.Remove(documentId);
|
||||
pendingMappings.Add(document.Id);
|
||||
}
|
||||
|
||||
if (cursor.PendingDocuments.Count > 0)
|
||||
{
|
||||
_logger.LogInformation(
|
||||
"CERT-Bund parse cycle: parsed {Parsed}, failures {Failures}, remaining documents {RemainingDocuments}, pending mappings {PendingMappings}",
|
||||
parsedCount,
|
||||
failedCount,
|
||||
remainingDocuments.Count,
|
||||
pendingMappings.Count);
|
||||
}
|
||||
|
||||
var updatedCursor = cursor
|
||||
.WithPendingDocuments(remainingDocuments)
|
||||
.WithPendingMappings(pendingMappings);
|
||||
|
||||
await UpdateCursorAsync(updatedCursor, cancellationToken).ConfigureAwait(false);
|
||||
}
|
||||
|
||||
public async Task MapAsync(IServiceProvider services, CancellationToken cancellationToken)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(services);
|
||||
|
||||
var cursor = await GetCursorAsync(cancellationToken).ConfigureAwait(false);
|
||||
if (cursor.PendingMappings.Count == 0)
|
||||
{
|
||||
return;
|
||||
}
|
||||
|
||||
var pendingMappings = cursor.PendingMappings.ToHashSet();
|
||||
var mappedCount = 0;
|
||||
var failedCount = 0;
|
||||
|
||||
foreach (var documentId in cursor.PendingMappings)
|
||||
{
|
||||
cancellationToken.ThrowIfCancellationRequested();
|
||||
|
||||
var document = await _documentStore.FindAsync(documentId, cancellationToken).ConfigureAwait(false);
|
||||
if (document is null)
|
||||
{
|
||||
pendingMappings.Remove(documentId);
|
||||
continue;
|
||||
}
|
||||
|
||||
var dtoRecord = await _dtoStore.FindByDocumentIdAsync(documentId, cancellationToken).ConfigureAwait(false);
|
||||
if (dtoRecord is null)
|
||||
{
|
||||
await _documentStore.UpdateStatusAsync(document.Id, DocumentStatuses.Failed, cancellationToken).ConfigureAwait(false);
|
||||
pendingMappings.Remove(documentId);
|
||||
_diagnostics.MapFailure("missing_dto");
|
||||
failedCount++;
|
||||
continue;
|
||||
}
|
||||
|
||||
CertBundAdvisoryDto? dto;
|
||||
try
|
||||
{
|
||||
dto = JsonSerializer.Deserialize<CertBundAdvisoryDto>(dtoRecord.Payload.ToJson(), SerializerOptions);
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
_logger.LogError(ex, "CERT-Bund failed to deserialize DTO for document {DocumentId}", document.Id);
|
||||
await _documentStore.UpdateStatusAsync(document.Id, DocumentStatuses.Failed, cancellationToken).ConfigureAwait(false);
|
||||
pendingMappings.Remove(documentId);
|
||||
_diagnostics.MapFailure("deserialize_failed");
|
||||
failedCount++;
|
||||
continue;
|
||||
}
|
||||
|
||||
if (dto is null)
|
||||
{
|
||||
await _documentStore.UpdateStatusAsync(document.Id, DocumentStatuses.Failed, cancellationToken).ConfigureAwait(false);
|
||||
pendingMappings.Remove(documentId);
|
||||
_diagnostics.MapFailure("null_dto");
|
||||
failedCount++;
|
||||
continue;
|
||||
}
|
||||
|
||||
try
|
||||
{
|
||||
var advisory = CertBundMapper.Map(dto, document, dtoRecord.ValidatedAt);
|
||||
await _advisoryStore.UpsertAsync(advisory, cancellationToken).ConfigureAwait(false);
|
||||
await _documentStore.UpdateStatusAsync(document.Id, DocumentStatuses.Mapped, cancellationToken).ConfigureAwait(false);
|
||||
pendingMappings.Remove(documentId);
|
||||
_diagnostics.MapSuccess(advisory.AffectedPackages.Length, advisory.Aliases.Length);
|
||||
mappedCount++;
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
_logger.LogError(ex, "CERT-Bund mapping failed for document {DocumentId}", document.Id);
|
||||
await _documentStore.UpdateStatusAsync(document.Id, DocumentStatuses.Failed, cancellationToken).ConfigureAwait(false);
|
||||
pendingMappings.Remove(documentId);
|
||||
_diagnostics.MapFailure("exception");
|
||||
failedCount++;
|
||||
}
|
||||
}
|
||||
|
||||
if (cursor.PendingMappings.Count > 0)
|
||||
{
|
||||
_logger.LogInformation(
|
||||
"CERT-Bund map cycle: mapped {Mapped}, failures {Failures}, remaining pending mappings {PendingMappings}",
|
||||
mappedCount,
|
||||
failedCount,
|
||||
pendingMappings.Count);
|
||||
}
|
||||
|
||||
var updatedCursor = cursor.WithPendingMappings(pendingMappings);
|
||||
await UpdateCursorAsync(updatedCursor, cancellationToken).ConfigureAwait(false);
|
||||
}
|
||||
|
||||
private static double? CalculateCoverageDays(IReadOnlyList<CertBundFeedItem> items, DateTimeOffset fetchedAt)
|
||||
{
|
||||
if (items is null || items.Count == 0)
|
||||
{
|
||||
return null;
|
||||
}
|
||||
|
||||
var oldest = items.Min(static item => item.Published);
|
||||
if (oldest == DateTimeOffset.MinValue)
|
||||
{
|
||||
return null;
|
||||
}
|
||||
|
||||
var span = fetchedAt - oldest;
|
||||
return span >= TimeSpan.Zero ? span.TotalDays : null;
|
||||
}
|
||||
|
||||
private async Task<CertBundCursor> GetCursorAsync(CancellationToken cancellationToken)
|
||||
{
|
||||
var state = await _stateRepository.TryGetAsync(SourceName, cancellationToken).ConfigureAwait(false);
|
||||
return state is null ? CertBundCursor.Empty : CertBundCursor.FromBson(state.Cursor);
|
||||
}
|
||||
|
||||
private Task UpdateCursorAsync(CertBundCursor cursor, CancellationToken cancellationToken)
|
||||
{
|
||||
var document = cursor.ToBsonDocument();
|
||||
var completedAt = cursor.LastFetchAt ?? _timeProvider.GetUtcNow();
|
||||
return _stateRepository.UpdateCursorAsync(SourceName, document, completedAt, cancellationToken);
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,21 @@
|
||||
using System;
|
||||
using Microsoft.Extensions.DependencyInjection;
|
||||
using StellaOps.Plugin;
|
||||
|
||||
namespace StellaOps.Concelier.Connector.CertBund;
|
||||
|
||||
public sealed class CertBundConnectorPlugin : IConnectorPlugin
|
||||
{
|
||||
public const string SourceName = "cert-bund";
|
||||
|
||||
public string Name => SourceName;
|
||||
|
||||
public bool IsAvailable(IServiceProvider services)
|
||||
=> services.GetService<CertBundConnector>() is not null;
|
||||
|
||||
public IFeedConnector Create(IServiceProvider services)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(services);
|
||||
return services.GetRequiredService<CertBundConnector>();
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,50 @@
|
||||
using System;
|
||||
using Microsoft.Extensions.Configuration;
|
||||
using Microsoft.Extensions.DependencyInjection;
|
||||
using StellaOps.DependencyInjection;
|
||||
using StellaOps.Concelier.Core.Jobs;
|
||||
using StellaOps.Concelier.Connector.CertBund.Configuration;
|
||||
|
||||
namespace StellaOps.Concelier.Connector.CertBund;
|
||||
|
||||
public sealed class CertBundDependencyInjectionRoutine : IDependencyInjectionRoutine
|
||||
{
|
||||
private const string ConfigurationSection = "concelier:sources:cert-bund";
|
||||
|
||||
public IServiceCollection Register(IServiceCollection services, IConfiguration configuration)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(services);
|
||||
ArgumentNullException.ThrowIfNull(configuration);
|
||||
|
||||
services.AddCertBundConnector(options =>
|
||||
{
|
||||
configuration.GetSection(ConfigurationSection).Bind(options);
|
||||
options.Validate();
|
||||
});
|
||||
|
||||
services.AddTransient<CertBundFetchJob>();
|
||||
|
||||
services.PostConfigure<JobSchedulerOptions>(options =>
|
||||
{
|
||||
EnsureJob(options, CertBundJobKinds.Fetch, typeof(CertBundFetchJob));
|
||||
});
|
||||
|
||||
return services;
|
||||
}
|
||||
|
||||
private static void EnsureJob(JobSchedulerOptions options, string kind, Type jobType)
|
||||
{
|
||||
if (options.Definitions.ContainsKey(kind))
|
||||
{
|
||||
return;
|
||||
}
|
||||
|
||||
options.Definitions[kind] = new JobDefinition(
|
||||
kind,
|
||||
jobType,
|
||||
options.DefaultTimeout,
|
||||
options.DefaultLeaseDuration,
|
||||
CronExpression: null,
|
||||
Enabled: true);
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,48 @@
|
||||
using System;
|
||||
using System.Net;
|
||||
using Microsoft.Extensions.DependencyInjection;
|
||||
using Microsoft.Extensions.DependencyInjection.Extensions;
|
||||
using Microsoft.Extensions.Options;
|
||||
using StellaOps.Concelier.Connector.CertBund.Configuration;
|
||||
using StellaOps.Concelier.Connector.CertBund.Internal;
|
||||
using StellaOps.Concelier.Connector.Common.Html;
|
||||
using StellaOps.Concelier.Connector.Common.Http;
|
||||
|
||||
namespace StellaOps.Concelier.Connector.CertBund;
|
||||
|
||||
public static class CertBundServiceCollectionExtensions
|
||||
{
|
||||
public static IServiceCollection AddCertBundConnector(this IServiceCollection services, Action<CertBundOptions> configure)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(services);
|
||||
ArgumentNullException.ThrowIfNull(configure);
|
||||
|
||||
services.AddOptions<CertBundOptions>()
|
||||
.Configure(configure)
|
||||
.PostConfigure(static options => options.Validate());
|
||||
|
||||
services.AddSourceHttpClient(CertBundOptions.HttpClientName, static (sp, clientOptions) =>
|
||||
{
|
||||
var options = sp.GetRequiredService<IOptions<CertBundOptions>>().Value;
|
||||
clientOptions.Timeout = options.RequestTimeout;
|
||||
clientOptions.UserAgent = "StellaOps.Concelier.CertBund/1.0";
|
||||
clientOptions.AllowedHosts.Clear();
|
||||
clientOptions.AllowedHosts.Add(options.FeedUri.Host);
|
||||
clientOptions.AllowedHosts.Add(options.DetailApiUri.Host);
|
||||
clientOptions.AllowedHosts.Add(options.PortalBootstrapUri.Host);
|
||||
clientOptions.ConfigureHandler = handler =>
|
||||
{
|
||||
handler.AutomaticDecompression = DecompressionMethods.All;
|
||||
handler.UseCookies = true;
|
||||
handler.CookieContainer = new System.Net.CookieContainer();
|
||||
};
|
||||
});
|
||||
|
||||
services.TryAddSingleton<HtmlContentSanitizer>();
|
||||
services.TryAddSingleton<CertBundDiagnostics>();
|
||||
services.TryAddSingleton<CertBundFeedClient>();
|
||||
services.TryAddSingleton<CertBundDetailParser>();
|
||||
services.AddTransient<CertBundConnector>();
|
||||
return services;
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,104 @@
|
||||
using System.Net;
|
||||
|
||||
namespace StellaOps.Concelier.Connector.CertBund.Configuration;
|
||||
|
||||
public sealed class CertBundOptions
|
||||
{
|
||||
public const string HttpClientName = "concelier.source.certbund";
|
||||
|
||||
/// <summary>
|
||||
/// RSS feed providing the latest CERT-Bund advisories.
|
||||
/// </summary>
|
||||
public Uri FeedUri { get; set; } = new("https://wid.cert-bund.de/content/public/securityAdvisory/rss");
|
||||
|
||||
/// <summary>
|
||||
/// Portal endpoint used to bootstrap session cookies (required for the SPA JSON API).
|
||||
/// </summary>
|
||||
public Uri PortalBootstrapUri { get; set; } = new("https://wid.cert-bund.de/portal/");
|
||||
|
||||
/// <summary>
|
||||
/// Detail API endpoint template; advisory identifier is appended as the <c>name</c> query parameter.
|
||||
/// </summary>
|
||||
public Uri DetailApiUri { get; set; } = new("https://wid.cert-bund.de/portal/api/securityadvisory");
|
||||
|
||||
/// <summary>
|
||||
/// Optional timeout override for feed/detail requests.
|
||||
/// </summary>
|
||||
public TimeSpan RequestTimeout { get; set; } = TimeSpan.FromSeconds(30);
|
||||
|
||||
/// <summary>
|
||||
/// Delay applied between successive detail fetches to respect upstream politeness.
|
||||
/// </summary>
|
||||
public TimeSpan RequestDelay { get; set; } = TimeSpan.FromMilliseconds(250);
|
||||
|
||||
/// <summary>
|
||||
/// Backoff recorded in source state when a fetch attempt fails.
|
||||
/// </summary>
|
||||
public TimeSpan FailureBackoff { get; set; } = TimeSpan.FromMinutes(5);
|
||||
|
||||
/// <summary>
|
||||
/// Maximum number of advisories to enqueue per fetch iteration.
|
||||
/// </summary>
|
||||
public int MaxAdvisoriesPerFetch { get; set; } = 50;
|
||||
|
||||
/// <summary>
|
||||
/// Maximum number of advisory identifiers remembered to prevent re-processing.
|
||||
/// </summary>
|
||||
public int MaxKnownAdvisories { get; set; } = 512;
|
||||
|
||||
public void Validate()
|
||||
{
|
||||
if (FeedUri is null || !FeedUri.IsAbsoluteUri)
|
||||
{
|
||||
throw new InvalidOperationException("CERT-Bund feed URI must be an absolute URI.");
|
||||
}
|
||||
|
||||
if (PortalBootstrapUri is null || !PortalBootstrapUri.IsAbsoluteUri)
|
||||
{
|
||||
throw new InvalidOperationException("CERT-Bund portal bootstrap URI must be an absolute URI.");
|
||||
}
|
||||
|
||||
if (DetailApiUri is null || !DetailApiUri.IsAbsoluteUri)
|
||||
{
|
||||
throw new InvalidOperationException("CERT-Bund detail API URI must be an absolute URI.");
|
||||
}
|
||||
|
||||
if (RequestTimeout <= TimeSpan.Zero)
|
||||
{
|
||||
throw new InvalidOperationException($"{nameof(RequestTimeout)} must be positive.");
|
||||
}
|
||||
|
||||
if (RequestDelay < TimeSpan.Zero)
|
||||
{
|
||||
throw new InvalidOperationException($"{nameof(RequestDelay)} cannot be negative.");
|
||||
}
|
||||
|
||||
if (FailureBackoff <= TimeSpan.Zero)
|
||||
{
|
||||
throw new InvalidOperationException($"{nameof(FailureBackoff)} must be positive.");
|
||||
}
|
||||
|
||||
if (MaxAdvisoriesPerFetch <= 0)
|
||||
{
|
||||
throw new InvalidOperationException($"{nameof(MaxAdvisoriesPerFetch)} must be greater than zero.");
|
||||
}
|
||||
|
||||
if (MaxKnownAdvisories <= 0)
|
||||
{
|
||||
throw new InvalidOperationException($"{nameof(MaxKnownAdvisories)} must be greater than zero.");
|
||||
}
|
||||
}
|
||||
|
||||
public Uri BuildDetailUri(string advisoryId)
|
||||
{
|
||||
if (string.IsNullOrWhiteSpace(advisoryId))
|
||||
{
|
||||
throw new ArgumentException("Advisory identifier must be provided.", nameof(advisoryId));
|
||||
}
|
||||
|
||||
var builder = new UriBuilder(DetailApiUri);
|
||||
var queryPrefix = string.IsNullOrEmpty(builder.Query) ? string.Empty : builder.Query.TrimStart('?') + "&";
|
||||
builder.Query = $"{queryPrefix}name={Uri.EscapeDataString(advisoryId)}";
|
||||
return builder.Uri;
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,68 @@
|
||||
using System;
|
||||
using System.Collections.Generic;
|
||||
using System.Text.Json.Serialization;
|
||||
|
||||
namespace StellaOps.Concelier.Connector.CertBund.Internal;
|
||||
|
||||
public sealed record CertBundAdvisoryDto
|
||||
{
|
||||
[JsonPropertyName("advisoryId")]
|
||||
public string AdvisoryId { get; init; } = string.Empty;
|
||||
|
||||
[JsonPropertyName("title")]
|
||||
public string Title { get; init; } = string.Empty;
|
||||
|
||||
[JsonPropertyName("summary")]
|
||||
public string? Summary { get; init; }
|
||||
|
||||
[JsonPropertyName("contentHtml")]
|
||||
public string ContentHtml { get; init; } = string.Empty;
|
||||
|
||||
[JsonPropertyName("severity")]
|
||||
public string? Severity { get; init; }
|
||||
|
||||
[JsonPropertyName("language")]
|
||||
public string Language { get; init; } = "de";
|
||||
|
||||
[JsonPropertyName("published")]
|
||||
public DateTimeOffset? Published { get; init; }
|
||||
|
||||
[JsonPropertyName("modified")]
|
||||
public DateTimeOffset? Modified { get; init; }
|
||||
|
||||
[JsonPropertyName("portalUri")]
|
||||
public Uri PortalUri { get; init; } = new("https://wid.cert-bund.de/");
|
||||
|
||||
[JsonPropertyName("detailUri")]
|
||||
public Uri DetailUri { get; init; } = new("https://wid.cert-bund.de/");
|
||||
|
||||
[JsonPropertyName("cveIds")]
|
||||
public IReadOnlyList<string> CveIds { get; init; } = Array.Empty<string>();
|
||||
|
||||
[JsonPropertyName("products")]
|
||||
public IReadOnlyList<CertBundProductDto> Products { get; init; } = Array.Empty<CertBundProductDto>();
|
||||
|
||||
[JsonPropertyName("references")]
|
||||
public IReadOnlyList<CertBundReferenceDto> References { get; init; } = Array.Empty<CertBundReferenceDto>();
|
||||
}
|
||||
|
||||
public sealed record CertBundProductDto
|
||||
{
|
||||
[JsonPropertyName("vendor")]
|
||||
public string? Vendor { get; init; }
|
||||
|
||||
[JsonPropertyName("name")]
|
||||
public string? Name { get; init; }
|
||||
|
||||
[JsonPropertyName("versions")]
|
||||
public string? Versions { get; init; }
|
||||
}
|
||||
|
||||
public sealed record CertBundReferenceDto
|
||||
{
|
||||
[JsonPropertyName("url")]
|
||||
public string Url { get; init; } = string.Empty;
|
||||
|
||||
[JsonPropertyName("label")]
|
||||
public string? Label { get; init; }
|
||||
}
|
||||
@@ -0,0 +1,118 @@
|
||||
using System;
|
||||
using System.Linq;
|
||||
using MongoDB.Bson;
|
||||
|
||||
namespace StellaOps.Concelier.Connector.CertBund.Internal;
|
||||
|
||||
internal sealed record CertBundCursor(
|
||||
IReadOnlyCollection<Guid> PendingDocuments,
|
||||
IReadOnlyCollection<Guid> PendingMappings,
|
||||
IReadOnlyCollection<string> KnownAdvisories,
|
||||
DateTimeOffset? LastPublished,
|
||||
DateTimeOffset? LastFetchAt)
|
||||
{
|
||||
private static readonly IReadOnlyCollection<Guid> EmptyGuids = Array.Empty<Guid>();
|
||||
private static readonly IReadOnlyCollection<string> EmptyStrings = Array.Empty<string>();
|
||||
|
||||
public static CertBundCursor Empty { get; } = new(EmptyGuids, EmptyGuids, EmptyStrings, null, null);
|
||||
|
||||
public CertBundCursor WithPendingDocuments(IEnumerable<Guid> documents)
|
||||
=> this with { PendingDocuments = Distinct(documents) };
|
||||
|
||||
public CertBundCursor WithPendingMappings(IEnumerable<Guid> mappings)
|
||||
=> this with { PendingMappings = Distinct(mappings) };
|
||||
|
||||
public CertBundCursor WithKnownAdvisories(IEnumerable<string> advisories)
|
||||
=> this with { KnownAdvisories = advisories?.Distinct(StringComparer.OrdinalIgnoreCase).ToArray() ?? EmptyStrings };
|
||||
|
||||
public CertBundCursor WithLastPublished(DateTimeOffset? published)
|
||||
=> this with { LastPublished = published };
|
||||
|
||||
public CertBundCursor WithLastFetch(DateTimeOffset? timestamp)
|
||||
=> this with { LastFetchAt = timestamp };
|
||||
|
||||
public BsonDocument ToBsonDocument()
|
||||
{
|
||||
var document = new BsonDocument
|
||||
{
|
||||
["pendingDocuments"] = new BsonArray(PendingDocuments.Select(id => id.ToString())),
|
||||
["pendingMappings"] = new BsonArray(PendingMappings.Select(id => id.ToString())),
|
||||
["knownAdvisories"] = new BsonArray(KnownAdvisories),
|
||||
};
|
||||
|
||||
if (LastPublished.HasValue)
|
||||
{
|
||||
document["lastPublished"] = LastPublished.Value.UtcDateTime;
|
||||
}
|
||||
|
||||
if (LastFetchAt.HasValue)
|
||||
{
|
||||
document["lastFetchAt"] = LastFetchAt.Value.UtcDateTime;
|
||||
}
|
||||
|
||||
return document;
|
||||
}
|
||||
|
||||
public static CertBundCursor FromBson(BsonDocument? document)
|
||||
{
|
||||
if (document is null || document.ElementCount == 0)
|
||||
{
|
||||
return Empty;
|
||||
}
|
||||
|
||||
var pendingDocuments = ReadGuidArray(document, "pendingDocuments");
|
||||
var pendingMappings = ReadGuidArray(document, "pendingMappings");
|
||||
var knownAdvisories = ReadStringArray(document, "knownAdvisories");
|
||||
var lastPublished = document.TryGetValue("lastPublished", out var publishedValue)
|
||||
? ParseDate(publishedValue)
|
||||
: null;
|
||||
var lastFetch = document.TryGetValue("lastFetchAt", out var fetchValue)
|
||||
? ParseDate(fetchValue)
|
||||
: null;
|
||||
|
||||
return new CertBundCursor(pendingDocuments, pendingMappings, knownAdvisories, lastPublished, lastFetch);
|
||||
}
|
||||
|
||||
private static IReadOnlyCollection<Guid> Distinct(IEnumerable<Guid>? values)
|
||||
=> values?.Distinct().ToArray() ?? EmptyGuids;
|
||||
|
||||
private static IReadOnlyCollection<Guid> ReadGuidArray(BsonDocument document, string field)
|
||||
{
|
||||
if (!document.TryGetValue(field, out var value) || value is not BsonArray array)
|
||||
{
|
||||
return EmptyGuids;
|
||||
}
|
||||
|
||||
var items = new List<Guid>(array.Count);
|
||||
foreach (var element in array)
|
||||
{
|
||||
if (Guid.TryParse(element?.ToString(), out var id))
|
||||
{
|
||||
items.Add(id);
|
||||
}
|
||||
}
|
||||
|
||||
return items;
|
||||
}
|
||||
|
||||
private static IReadOnlyCollection<string> ReadStringArray(BsonDocument document, string field)
|
||||
{
|
||||
if (!document.TryGetValue(field, out var value) || value is not BsonArray array)
|
||||
{
|
||||
return EmptyStrings;
|
||||
}
|
||||
|
||||
return array.Select(element => element?.ToString() ?? string.Empty)
|
||||
.Where(static s => !string.IsNullOrWhiteSpace(s))
|
||||
.Distinct(StringComparer.OrdinalIgnoreCase)
|
||||
.ToArray();
|
||||
}
|
||||
|
||||
private static DateTimeOffset? ParseDate(BsonValue value)
|
||||
=> value.BsonType switch
|
||||
{
|
||||
BsonType.DateTime => DateTime.SpecifyKind(value.ToUniversalTime(), DateTimeKind.Utc),
|
||||
BsonType.String when DateTimeOffset.TryParse(value.AsString, out var parsed) => parsed.ToUniversalTime(),
|
||||
_ => null,
|
||||
};
|
||||
}
|
||||
@@ -0,0 +1,87 @@
|
||||
using System;
|
||||
using System.Collections.Generic;
|
||||
using System.Linq;
|
||||
using System.Text.Json;
|
||||
using System.Text.Json.Serialization;
|
||||
using StellaOps.Concelier.Connector.Common.Html;
|
||||
|
||||
namespace StellaOps.Concelier.Connector.CertBund.Internal;
|
||||
|
||||
public sealed class CertBundDetailParser
|
||||
{
|
||||
private static readonly JsonSerializerOptions SerializerOptions = new(JsonSerializerDefaults.Web)
|
||||
{
|
||||
PropertyNameCaseInsensitive = true,
|
||||
DefaultIgnoreCondition = JsonIgnoreCondition.WhenWritingNull,
|
||||
};
|
||||
|
||||
private readonly HtmlContentSanitizer _sanitizer;
|
||||
|
||||
public CertBundDetailParser(HtmlContentSanitizer sanitizer)
|
||||
=> _sanitizer = sanitizer ?? throw new ArgumentNullException(nameof(sanitizer));
|
||||
|
||||
public CertBundAdvisoryDto Parse(Uri detailUri, Uri portalUri, byte[] payload)
|
||||
{
|
||||
var detail = JsonSerializer.Deserialize<CertBundDetailResponse>(payload, SerializerOptions)
|
||||
?? throw new InvalidOperationException("CERT-Bund detail payload deserialized to null.");
|
||||
|
||||
var advisoryId = detail.Name ?? throw new InvalidOperationException("CERT-Bund detail missing advisory name.");
|
||||
var contentHtml = _sanitizer.Sanitize(detail.Description ?? string.Empty, portalUri);
|
||||
|
||||
return new CertBundAdvisoryDto
|
||||
{
|
||||
AdvisoryId = advisoryId,
|
||||
Title = detail.Title ?? advisoryId,
|
||||
Summary = detail.Summary,
|
||||
ContentHtml = contentHtml,
|
||||
Severity = detail.Severity,
|
||||
Language = string.IsNullOrWhiteSpace(detail.Language) ? "de" : detail.Language!,
|
||||
Published = detail.Published,
|
||||
Modified = detail.Updated ?? detail.Published,
|
||||
PortalUri = portalUri,
|
||||
DetailUri = detailUri,
|
||||
CveIds = detail.CveIds?.Where(static id => !string.IsNullOrWhiteSpace(id))
|
||||
.Select(static id => id!.Trim())
|
||||
.Distinct(StringComparer.OrdinalIgnoreCase)
|
||||
.ToArray() ?? Array.Empty<string>(),
|
||||
References = MapReferences(detail.References),
|
||||
Products = MapProducts(detail.Products),
|
||||
};
|
||||
}
|
||||
|
||||
private static IReadOnlyList<CertBundReferenceDto> MapReferences(CertBundDetailReference[]? references)
|
||||
{
|
||||
if (references is null || references.Length == 0)
|
||||
{
|
||||
return Array.Empty<CertBundReferenceDto>();
|
||||
}
|
||||
|
||||
return references
|
||||
.Where(static reference => !string.IsNullOrWhiteSpace(reference.Url))
|
||||
.Select(reference => new CertBundReferenceDto
|
||||
{
|
||||
Url = reference.Url!,
|
||||
Label = reference.Label,
|
||||
})
|
||||
.DistinctBy(static reference => reference.Url, StringComparer.OrdinalIgnoreCase)
|
||||
.ToArray();
|
||||
}
|
||||
|
||||
private static IReadOnlyList<CertBundProductDto> MapProducts(CertBundDetailProduct[]? products)
|
||||
{
|
||||
if (products is null || products.Length == 0)
|
||||
{
|
||||
return Array.Empty<CertBundProductDto>();
|
||||
}
|
||||
|
||||
return products
|
||||
.Where(static product => !string.IsNullOrWhiteSpace(product.Vendor) || !string.IsNullOrWhiteSpace(product.Name))
|
||||
.Select(product => new CertBundProductDto
|
||||
{
|
||||
Vendor = product.Vendor,
|
||||
Name = product.Name,
|
||||
Versions = product.Versions,
|
||||
})
|
||||
.ToArray();
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,60 @@
|
||||
using System.Text.Json.Serialization;
|
||||
|
||||
namespace StellaOps.Concelier.Connector.CertBund.Internal;
|
||||
|
||||
internal sealed record CertBundDetailResponse
|
||||
{
|
||||
[JsonPropertyName("name")]
|
||||
public string? Name { get; init; }
|
||||
|
||||
[JsonPropertyName("title")]
|
||||
public string? Title { get; init; }
|
||||
|
||||
[JsonPropertyName("summary")]
|
||||
public string? Summary { get; init; }
|
||||
|
||||
[JsonPropertyName("description")]
|
||||
public string? Description { get; init; }
|
||||
|
||||
[JsonPropertyName("severity")]
|
||||
public string? Severity { get; init; }
|
||||
|
||||
[JsonPropertyName("language")]
|
||||
public string? Language { get; init; }
|
||||
|
||||
[JsonPropertyName("published")]
|
||||
public DateTimeOffset? Published { get; init; }
|
||||
|
||||
[JsonPropertyName("updated")]
|
||||
public DateTimeOffset? Updated { get; init; }
|
||||
|
||||
[JsonPropertyName("cveIds")]
|
||||
public string[]? CveIds { get; init; }
|
||||
|
||||
[JsonPropertyName("references")]
|
||||
public CertBundDetailReference[]? References { get; init; }
|
||||
|
||||
[JsonPropertyName("products")]
|
||||
public CertBundDetailProduct[]? Products { get; init; }
|
||||
}
|
||||
|
||||
internal sealed record CertBundDetailReference
|
||||
{
|
||||
[JsonPropertyName("url")]
|
||||
public string? Url { get; init; }
|
||||
|
||||
[JsonPropertyName("label")]
|
||||
public string? Label { get; init; }
|
||||
}
|
||||
|
||||
internal sealed record CertBundDetailProduct
|
||||
{
|
||||
[JsonPropertyName("vendor")]
|
||||
public string? Vendor { get; init; }
|
||||
|
||||
[JsonPropertyName("name")]
|
||||
public string? Name { get; init; }
|
||||
|
||||
[JsonPropertyName("versions")]
|
||||
public string? Versions { get; init; }
|
||||
}
|
||||
@@ -0,0 +1,191 @@
|
||||
using System;
|
||||
using System.Collections.Generic;
|
||||
using System.Diagnostics.Metrics;
|
||||
|
||||
namespace StellaOps.Concelier.Connector.CertBund.Internal;
|
||||
|
||||
/// <summary>
|
||||
/// Emits OpenTelemetry counters and histograms for the CERT-Bund connector.
|
||||
/// </summary>
|
||||
public sealed class CertBundDiagnostics : IDisposable
|
||||
{
|
||||
private const string MeterName = "StellaOps.Concelier.Connector.CertBund";
|
||||
private const string MeterVersion = "1.0.0";
|
||||
|
||||
private readonly Meter _meter;
|
||||
private readonly Counter<long> _feedFetchAttempts;
|
||||
private readonly Counter<long> _feedFetchSuccess;
|
||||
private readonly Counter<long> _feedFetchFailures;
|
||||
private readonly Histogram<long> _feedItemCount;
|
||||
private readonly Histogram<long> _feedEnqueuedCount;
|
||||
private readonly Histogram<double> _feedCoverageDays;
|
||||
private readonly Counter<long> _detailFetchAttempts;
|
||||
private readonly Counter<long> _detailFetchSuccess;
|
||||
private readonly Counter<long> _detailFetchNotModified;
|
||||
private readonly Counter<long> _detailFetchFailures;
|
||||
private readonly Counter<long> _parseSuccess;
|
||||
private readonly Counter<long> _parseFailures;
|
||||
private readonly Histogram<long> _parseProductCount;
|
||||
private readonly Histogram<long> _parseCveCount;
|
||||
private readonly Counter<long> _mapSuccess;
|
||||
private readonly Counter<long> _mapFailures;
|
||||
private readonly Histogram<long> _mapPackageCount;
|
||||
private readonly Histogram<long> _mapAliasCount;
|
||||
|
||||
public CertBundDiagnostics()
|
||||
{
|
||||
_meter = new Meter(MeterName, MeterVersion);
|
||||
_feedFetchAttempts = _meter.CreateCounter<long>(
|
||||
name: "certbund.feed.fetch.attempts",
|
||||
unit: "operations",
|
||||
description: "Number of RSS feed load attempts.");
|
||||
_feedFetchSuccess = _meter.CreateCounter<long>(
|
||||
name: "certbund.feed.fetch.success",
|
||||
unit: "operations",
|
||||
description: "Number of successful RSS feed loads.");
|
||||
_feedFetchFailures = _meter.CreateCounter<long>(
|
||||
name: "certbund.feed.fetch.failures",
|
||||
unit: "operations",
|
||||
description: "Number of RSS feed load failures.");
|
||||
_feedItemCount = _meter.CreateHistogram<long>(
|
||||
name: "certbund.feed.items.count",
|
||||
unit: "items",
|
||||
description: "Distribution of RSS item counts per fetch.");
|
||||
_feedEnqueuedCount = _meter.CreateHistogram<long>(
|
||||
name: "certbund.feed.enqueued.count",
|
||||
unit: "documents",
|
||||
description: "Distribution of advisory documents enqueued per fetch.");
|
||||
_feedCoverageDays = _meter.CreateHistogram<double>(
|
||||
name: "certbund.feed.coverage.days",
|
||||
unit: "days",
|
||||
description: "Coverage window in days between fetch time and the oldest published advisory in the feed.");
|
||||
_detailFetchAttempts = _meter.CreateCounter<long>(
|
||||
name: "certbund.detail.fetch.attempts",
|
||||
unit: "operations",
|
||||
description: "Number of detail fetch attempts.");
|
||||
_detailFetchSuccess = _meter.CreateCounter<long>(
|
||||
name: "certbund.detail.fetch.success",
|
||||
unit: "operations",
|
||||
description: "Number of detail fetches that persisted a document.");
|
||||
_detailFetchNotModified = _meter.CreateCounter<long>(
|
||||
name: "certbund.detail.fetch.not_modified",
|
||||
unit: "operations",
|
||||
description: "Number of detail fetches returning HTTP 304.");
|
||||
_detailFetchFailures = _meter.CreateCounter<long>(
|
||||
name: "certbund.detail.fetch.failures",
|
||||
unit: "operations",
|
||||
description: "Number of detail fetches that failed.");
|
||||
_parseSuccess = _meter.CreateCounter<long>(
|
||||
name: "certbund.parse.success",
|
||||
unit: "documents",
|
||||
description: "Number of documents parsed into CERT-Bund DTOs.");
|
||||
_parseFailures = _meter.CreateCounter<long>(
|
||||
name: "certbund.parse.failures",
|
||||
unit: "documents",
|
||||
description: "Number of documents that failed to parse.");
|
||||
_parseProductCount = _meter.CreateHistogram<long>(
|
||||
name: "certbund.parse.products.count",
|
||||
unit: "products",
|
||||
description: "Distribution of product entries captured per advisory.");
|
||||
_parseCveCount = _meter.CreateHistogram<long>(
|
||||
name: "certbund.parse.cve.count",
|
||||
unit: "aliases",
|
||||
description: "Distribution of CVE identifiers captured per advisory.");
|
||||
_mapSuccess = _meter.CreateCounter<long>(
|
||||
name: "certbund.map.success",
|
||||
unit: "advisories",
|
||||
description: "Number of canonical advisories emitted by the mapper.");
|
||||
_mapFailures = _meter.CreateCounter<long>(
|
||||
name: "certbund.map.failures",
|
||||
unit: "advisories",
|
||||
description: "Number of mapping failures.");
|
||||
_mapPackageCount = _meter.CreateHistogram<long>(
|
||||
name: "certbund.map.affected.count",
|
||||
unit: "packages",
|
||||
description: "Distribution of affected packages emitted per advisory.");
|
||||
_mapAliasCount = _meter.CreateHistogram<long>(
|
||||
name: "certbund.map.aliases.count",
|
||||
unit: "aliases",
|
||||
description: "Distribution of alias counts per advisory.");
|
||||
}
|
||||
|
||||
public void FeedFetchAttempt() => _feedFetchAttempts.Add(1);
|
||||
|
||||
public void FeedFetchSuccess(int itemCount)
|
||||
{
|
||||
_feedFetchSuccess.Add(1);
|
||||
if (itemCount >= 0)
|
||||
{
|
||||
_feedItemCount.Record(itemCount);
|
||||
}
|
||||
}
|
||||
|
||||
public void FeedFetchFailure(string reason = "error")
|
||||
=> _feedFetchFailures.Add(1, ReasonTag(reason));
|
||||
|
||||
public void RecordFeedCoverage(double? coverageDays)
|
||||
{
|
||||
if (coverageDays is { } days && days >= 0)
|
||||
{
|
||||
_feedCoverageDays.Record(days);
|
||||
}
|
||||
}
|
||||
|
||||
public void DetailFetchAttempt() => _detailFetchAttempts.Add(1);
|
||||
|
||||
public void DetailFetchSuccess() => _detailFetchSuccess.Add(1);
|
||||
|
||||
public void DetailFetchNotModified() => _detailFetchNotModified.Add(1);
|
||||
|
||||
public void DetailFetchFailure(string reason = "error")
|
||||
=> _detailFetchFailures.Add(1, ReasonTag(reason));
|
||||
|
||||
public void DetailFetchEnqueued(int count)
|
||||
{
|
||||
if (count >= 0)
|
||||
{
|
||||
_feedEnqueuedCount.Record(count);
|
||||
}
|
||||
}
|
||||
|
||||
public void ParseSuccess(int productCount, int cveCount)
|
||||
{
|
||||
_parseSuccess.Add(1);
|
||||
|
||||
if (productCount >= 0)
|
||||
{
|
||||
_parseProductCount.Record(productCount);
|
||||
}
|
||||
|
||||
if (cveCount >= 0)
|
||||
{
|
||||
_parseCveCount.Record(cveCount);
|
||||
}
|
||||
}
|
||||
|
||||
public void ParseFailure(string reason = "error")
|
||||
=> _parseFailures.Add(1, ReasonTag(reason));
|
||||
|
||||
public void MapSuccess(int affectedPackages, int aliasCount)
|
||||
{
|
||||
_mapSuccess.Add(1);
|
||||
|
||||
if (affectedPackages >= 0)
|
||||
{
|
||||
_mapPackageCount.Record(affectedPackages);
|
||||
}
|
||||
|
||||
if (aliasCount >= 0)
|
||||
{
|
||||
_mapAliasCount.Record(aliasCount);
|
||||
}
|
||||
}
|
||||
|
||||
public void MapFailure(string reason = "error")
|
||||
=> _mapFailures.Add(1, ReasonTag(reason));
|
||||
|
||||
private static KeyValuePair<string, object?> ReasonTag(string reason)
|
||||
=> new("reason", string.IsNullOrWhiteSpace(reason) ? "unknown" : reason.ToLowerInvariant());
|
||||
|
||||
public void Dispose() => _meter.Dispose();
|
||||
}
|
||||
@@ -0,0 +1,29 @@
|
||||
using System;
|
||||
using System.Collections.Generic;
|
||||
|
||||
namespace StellaOps.Concelier.Connector.CertBund.Internal;
|
||||
|
||||
internal static class CertBundDocumentMetadata
|
||||
{
|
||||
public static Dictionary<string, string> CreateMetadata(CertBundFeedItem item)
|
||||
{
|
||||
var metadata = new Dictionary<string, string>(StringComparer.OrdinalIgnoreCase)
|
||||
{
|
||||
["certbund.advisoryId"] = item.AdvisoryId,
|
||||
["certbund.portalUri"] = item.PortalUri.ToString(),
|
||||
["certbund.published"] = item.Published.ToString("O"),
|
||||
};
|
||||
|
||||
if (!string.IsNullOrWhiteSpace(item.Category))
|
||||
{
|
||||
metadata["certbund.category"] = item.Category!;
|
||||
}
|
||||
|
||||
if (!string.IsNullOrWhiteSpace(item.Title))
|
||||
{
|
||||
metadata["certbund.title"] = item.Title!;
|
||||
}
|
||||
|
||||
return metadata;
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,143 @@
|
||||
using System;
|
||||
using System.Collections.Generic;
|
||||
using System.Globalization;
|
||||
using System.Linq;
|
||||
using System.Net.Http;
|
||||
using System.Threading;
|
||||
using System.Threading.Tasks;
|
||||
using System.Xml.Linq;
|
||||
using Microsoft.Extensions.Logging;
|
||||
using Microsoft.Extensions.Options;
|
||||
using StellaOps.Concelier.Connector.CertBund.Configuration;
|
||||
|
||||
namespace StellaOps.Concelier.Connector.CertBund.Internal;
|
||||
|
||||
public sealed class CertBundFeedClient
|
||||
{
|
||||
private readonly IHttpClientFactory _httpClientFactory;
|
||||
private readonly CertBundOptions _options;
|
||||
private readonly ILogger<CertBundFeedClient> _logger;
|
||||
private readonly SemaphoreSlim _bootstrapSemaphore = new(1, 1);
|
||||
private volatile bool _bootstrapped;
|
||||
|
||||
public CertBundFeedClient(
|
||||
IHttpClientFactory httpClientFactory,
|
||||
IOptions<CertBundOptions> options,
|
||||
ILogger<CertBundFeedClient> logger)
|
||||
{
|
||||
_httpClientFactory = httpClientFactory ?? throw new ArgumentNullException(nameof(httpClientFactory));
|
||||
_options = (options ?? throw new ArgumentNullException(nameof(options))).Value ?? throw new ArgumentNullException(nameof(options));
|
||||
_options.Validate();
|
||||
_logger = logger ?? throw new ArgumentNullException(nameof(logger));
|
||||
}
|
||||
|
||||
public async Task<IReadOnlyList<CertBundFeedItem>> LoadAsync(CancellationToken cancellationToken)
|
||||
{
|
||||
var client = _httpClientFactory.CreateClient(CertBundOptions.HttpClientName);
|
||||
await EnsureSessionAsync(client, cancellationToken).ConfigureAwait(false);
|
||||
|
||||
using var request = new HttpRequestMessage(HttpMethod.Get, _options.FeedUri);
|
||||
request.Headers.TryAddWithoutValidation("Accept", "application/rss+xml, application/xml;q=0.9, text/xml;q=0.8");
|
||||
using var response = await client.SendAsync(request, HttpCompletionOption.ResponseHeadersRead, cancellationToken).ConfigureAwait(false);
|
||||
response.EnsureSuccessStatusCode();
|
||||
|
||||
await using var stream = await response.Content.ReadAsStreamAsync(cancellationToken).ConfigureAwait(false);
|
||||
var document = XDocument.Load(stream);
|
||||
|
||||
var items = new List<CertBundFeedItem>();
|
||||
foreach (var element in document.Descendants("item"))
|
||||
{
|
||||
cancellationToken.ThrowIfCancellationRequested();
|
||||
|
||||
var linkValue = element.Element("link")?.Value?.Trim();
|
||||
if (string.IsNullOrWhiteSpace(linkValue) || !Uri.TryCreate(linkValue, UriKind.Absolute, out var portalUri))
|
||||
{
|
||||
continue;
|
||||
}
|
||||
|
||||
var advisoryId = TryExtractNameParameter(portalUri);
|
||||
if (string.IsNullOrWhiteSpace(advisoryId))
|
||||
{
|
||||
continue;
|
||||
}
|
||||
|
||||
var detailUri = _options.BuildDetailUri(advisoryId);
|
||||
var pubDateText = element.Element("pubDate")?.Value;
|
||||
var published = ParseDate(pubDateText);
|
||||
var title = element.Element("title")?.Value?.Trim();
|
||||
var category = element.Element("category")?.Value?.Trim();
|
||||
|
||||
items.Add(new CertBundFeedItem(advisoryId, detailUri, portalUri, published, title, category));
|
||||
}
|
||||
|
||||
return items;
|
||||
}
|
||||
|
||||
private async Task EnsureSessionAsync(HttpClient client, CancellationToken cancellationToken)
|
||||
{
|
||||
if (_bootstrapped)
|
||||
{
|
||||
return;
|
||||
}
|
||||
|
||||
await _bootstrapSemaphore.WaitAsync(cancellationToken).ConfigureAwait(false);
|
||||
try
|
||||
{
|
||||
if (_bootstrapped)
|
||||
{
|
||||
return;
|
||||
}
|
||||
|
||||
using var request = new HttpRequestMessage(HttpMethod.Get, _options.PortalBootstrapUri);
|
||||
request.Headers.TryAddWithoutValidation("Accept", "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8");
|
||||
using var response = await client.SendAsync(request, HttpCompletionOption.ResponseHeadersRead, cancellationToken).ConfigureAwait(false);
|
||||
response.EnsureSuccessStatusCode();
|
||||
|
||||
_bootstrapped = true;
|
||||
}
|
||||
finally
|
||||
{
|
||||
_bootstrapSemaphore.Release();
|
||||
}
|
||||
}
|
||||
|
||||
private static string? TryExtractNameParameter(Uri portalUri)
|
||||
{
|
||||
if (portalUri is null)
|
||||
{
|
||||
return null;
|
||||
}
|
||||
|
||||
var query = portalUri.Query;
|
||||
if (string.IsNullOrEmpty(query))
|
||||
{
|
||||
return null;
|
||||
}
|
||||
|
||||
var trimmed = query.TrimStart('?');
|
||||
foreach (var pair in trimmed.Split('&', StringSplitOptions.RemoveEmptyEntries))
|
||||
{
|
||||
var separatorIndex = pair.IndexOf('=');
|
||||
if (separatorIndex <= 0)
|
||||
{
|
||||
continue;
|
||||
}
|
||||
|
||||
var key = pair[..separatorIndex].Trim();
|
||||
if (!key.Equals("name", StringComparison.OrdinalIgnoreCase))
|
||||
{
|
||||
continue;
|
||||
}
|
||||
|
||||
var value = pair[(separatorIndex + 1)..];
|
||||
return Uri.UnescapeDataString(value);
|
||||
}
|
||||
|
||||
return null;
|
||||
}
|
||||
|
||||
private static DateTimeOffset ParseDate(string? value)
|
||||
=> DateTimeOffset.TryParse(value, CultureInfo.InvariantCulture, DateTimeStyles.AssumeUniversal | DateTimeStyles.AdjustToUniversal, out var parsed)
|
||||
? parsed
|
||||
: DateTimeOffset.UtcNow;
|
||||
}
|
||||
@@ -0,0 +1,11 @@
|
||||
namespace StellaOps.Concelier.Connector.CertBund.Internal;
|
||||
|
||||
using System;
|
||||
|
||||
public sealed record CertBundFeedItem(
|
||||
string AdvisoryId,
|
||||
Uri DetailUri,
|
||||
Uri PortalUri,
|
||||
DateTimeOffset Published,
|
||||
string? Title,
|
||||
string? Category);
|
||||
@@ -0,0 +1,168 @@
|
||||
using System;
|
||||
using System.Collections.Generic;
|
||||
using System.Linq;
|
||||
using StellaOps.Concelier.Models;
|
||||
using StellaOps.Concelier.Storage.Mongo.Documents;
|
||||
|
||||
namespace StellaOps.Concelier.Connector.CertBund.Internal;
|
||||
|
||||
internal static class CertBundMapper
|
||||
{
|
||||
public static Advisory Map(CertBundAdvisoryDto dto, DocumentRecord document, DateTimeOffset recordedAt)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(dto);
|
||||
ArgumentNullException.ThrowIfNull(document);
|
||||
|
||||
var aliases = BuildAliases(dto);
|
||||
var references = BuildReferences(dto, recordedAt);
|
||||
var packages = BuildPackages(dto, recordedAt);
|
||||
var provenance = new AdvisoryProvenance(
|
||||
CertBundConnectorPlugin.SourceName,
|
||||
"advisory",
|
||||
dto.AdvisoryId,
|
||||
recordedAt,
|
||||
new[] { ProvenanceFieldMasks.Advisory });
|
||||
|
||||
return new Advisory(
|
||||
advisoryKey: dto.AdvisoryId,
|
||||
title: dto.Title,
|
||||
summary: dto.Summary,
|
||||
language: dto.Language?.ToLowerInvariant() ?? "de",
|
||||
published: dto.Published,
|
||||
modified: dto.Modified,
|
||||
severity: MapSeverity(dto.Severity),
|
||||
exploitKnown: false,
|
||||
aliases: aliases,
|
||||
references: references,
|
||||
affectedPackages: packages,
|
||||
cvssMetrics: Array.Empty<CvssMetric>(),
|
||||
provenance: new[] { provenance });
|
||||
}
|
||||
|
||||
private static IReadOnlyList<string> BuildAliases(CertBundAdvisoryDto dto)
|
||||
{
|
||||
var aliases = new List<string>(capacity: 4) { dto.AdvisoryId };
|
||||
foreach (var cve in dto.CveIds)
|
||||
{
|
||||
if (!string.IsNullOrWhiteSpace(cve))
|
||||
{
|
||||
aliases.Add(cve);
|
||||
}
|
||||
}
|
||||
|
||||
return aliases
|
||||
.Where(static alias => !string.IsNullOrWhiteSpace(alias))
|
||||
.Distinct(StringComparer.OrdinalIgnoreCase)
|
||||
.OrderBy(static alias => alias, StringComparer.OrdinalIgnoreCase)
|
||||
.ToArray();
|
||||
}
|
||||
|
||||
private static IReadOnlyList<AdvisoryReference> BuildReferences(CertBundAdvisoryDto dto, DateTimeOffset recordedAt)
|
||||
{
|
||||
var references = new List<AdvisoryReference>
|
||||
{
|
||||
new(dto.DetailUri.ToString(), "details", "cert-bund", null, new AdvisoryProvenance(
|
||||
CertBundConnectorPlugin.SourceName,
|
||||
"reference",
|
||||
dto.DetailUri.ToString(),
|
||||
recordedAt,
|
||||
new[] { ProvenanceFieldMasks.References }))
|
||||
};
|
||||
|
||||
foreach (var reference in dto.References)
|
||||
{
|
||||
if (string.IsNullOrWhiteSpace(reference.Url))
|
||||
{
|
||||
continue;
|
||||
}
|
||||
|
||||
references.Add(new AdvisoryReference(
|
||||
reference.Url,
|
||||
kind: "reference",
|
||||
sourceTag: "cert-bund",
|
||||
summary: reference.Label,
|
||||
provenance: new AdvisoryProvenance(
|
||||
CertBundConnectorPlugin.SourceName,
|
||||
"reference",
|
||||
reference.Url,
|
||||
recordedAt,
|
||||
new[] { ProvenanceFieldMasks.References })));
|
||||
}
|
||||
|
||||
return references
|
||||
.DistinctBy(static reference => reference.Url, StringComparer.Ordinal)
|
||||
.OrderBy(static reference => reference.Url, StringComparer.Ordinal)
|
||||
.ToArray();
|
||||
}
|
||||
|
||||
private static IReadOnlyList<AffectedPackage> BuildPackages(CertBundAdvisoryDto dto, DateTimeOffset recordedAt)
|
||||
{
|
||||
if (dto.Products.Count == 0)
|
||||
{
|
||||
return Array.Empty<AffectedPackage>();
|
||||
}
|
||||
|
||||
var packages = new List<AffectedPackage>(dto.Products.Count);
|
||||
foreach (var product in dto.Products)
|
||||
{
|
||||
var vendor = Validation.TrimToNull(product.Vendor) ?? "Unspecified";
|
||||
var name = Validation.TrimToNull(product.Name);
|
||||
var identifier = name is null ? vendor : $"{vendor} {name}";
|
||||
|
||||
var provenance = new AdvisoryProvenance(
|
||||
CertBundConnectorPlugin.SourceName,
|
||||
"package",
|
||||
identifier,
|
||||
recordedAt,
|
||||
new[] { ProvenanceFieldMasks.AffectedPackages });
|
||||
|
||||
var ranges = string.IsNullOrWhiteSpace(product.Versions)
|
||||
? Array.Empty<AffectedVersionRange>()
|
||||
: new[]
|
||||
{
|
||||
new AffectedVersionRange(
|
||||
rangeKind: "string",
|
||||
introducedVersion: null,
|
||||
fixedVersion: null,
|
||||
lastAffectedVersion: null,
|
||||
rangeExpression: product.Versions,
|
||||
provenance: new AdvisoryProvenance(
|
||||
CertBundConnectorPlugin.SourceName,
|
||||
"package-range",
|
||||
product.Versions,
|
||||
recordedAt,
|
||||
new[] { ProvenanceFieldMasks.VersionRanges }))
|
||||
};
|
||||
|
||||
packages.Add(new AffectedPackage(
|
||||
AffectedPackageTypes.Vendor,
|
||||
identifier,
|
||||
platform: null,
|
||||
versionRanges: ranges,
|
||||
statuses: Array.Empty<AffectedPackageStatus>(),
|
||||
provenance: new[] { provenance },
|
||||
normalizedVersions: Array.Empty<NormalizedVersionRule>()));
|
||||
}
|
||||
|
||||
return packages
|
||||
.DistinctBy(static package => package.Identifier, StringComparer.OrdinalIgnoreCase)
|
||||
.OrderBy(static package => package.Identifier, StringComparer.OrdinalIgnoreCase)
|
||||
.ToArray();
|
||||
}
|
||||
|
||||
private static string? MapSeverity(string? severity)
|
||||
{
|
||||
if (string.IsNullOrWhiteSpace(severity))
|
||||
{
|
||||
return null;
|
||||
}
|
||||
|
||||
return severity.ToLowerInvariant() switch
|
||||
{
|
||||
"hoch" or "high" => "high",
|
||||
"mittel" or "medium" => "medium",
|
||||
"gering" or "low" => "low",
|
||||
_ => severity.ToLowerInvariant(),
|
||||
};
|
||||
}
|
||||
}
|
||||
22
src/StellaOps.Concelier.Connector.CertBund/Jobs.cs
Normal file
22
src/StellaOps.Concelier.Connector.CertBund/Jobs.cs
Normal file
@@ -0,0 +1,22 @@
|
||||
using System;
|
||||
using System.Threading;
|
||||
using System.Threading.Tasks;
|
||||
using StellaOps.Concelier.Core.Jobs;
|
||||
|
||||
namespace StellaOps.Concelier.Connector.CertBund;
|
||||
|
||||
internal static class CertBundJobKinds
|
||||
{
|
||||
public const string Fetch = "source:cert-bund:fetch";
|
||||
}
|
||||
|
||||
internal sealed class CertBundFetchJob : IJob
|
||||
{
|
||||
private readonly CertBundConnector _connector;
|
||||
|
||||
public CertBundFetchJob(CertBundConnector connector)
|
||||
=> _connector = connector ?? throw new ArgumentNullException(nameof(connector));
|
||||
|
||||
public Task ExecuteAsync(JobExecutionContext context, CancellationToken cancellationToken)
|
||||
=> _connector.FetchAsync(context.Services, cancellationToken);
|
||||
}
|
||||
39
src/StellaOps.Concelier.Connector.CertBund/README.md
Normal file
39
src/StellaOps.Concelier.Connector.CertBund/README.md
Normal file
@@ -0,0 +1,39 @@
|
||||
# CERT-Bund Security Advisories – Connector Notes
|
||||
|
||||
## Publication endpoints
|
||||
- **RSS feed (latest 250 advisories)** – `https://wid.cert-bund.de/content/public/securityAdvisory/rss`. The feed refreshes quickly; the current window spans roughly 6 days of activity, so fetch jobs must run frequently to avoid churn.
|
||||
- **Portal bootstrap** – `https://wid.cert-bund.de/portal/` is hit once per process start to prime the session (`client_config` cookie) before any API calls.
|
||||
- **Detail API** – `https://wid.cert-bund.de/portal/api/securityadvisory?name=<ID>`. The connector reuses the bootstrapped `SocketsHttpHandler` so cookies and headers match the Angular SPA. Manual reproduction requires the same cookie container; otherwise the endpoint responds with the shell HTML document.
|
||||
|
||||
## Telemetry
|
||||
The OpenTelemetry meter is `StellaOps.Concelier.Connector.CertBund`. Key instruments:
|
||||
|
||||
| Metric | Type | Notes |
|
||||
| --- | --- | --- |
|
||||
| `certbund.feed.fetch.attempts` / `.success` / `.failures` | counter | Feed poll lifecycle. |
|
||||
| `certbund.feed.items.count` | histogram | Items returned per RSS fetch. |
|
||||
| `certbund.feed.enqueued.count` | histogram | Detail documents queued per cycle (post-dedupe, before truncation). |
|
||||
| `certbund.feed.coverage.days` | histogram | Rolling window (fetch time − oldest published entry). Useful to alert when feed depth contracts. |
|
||||
| `certbund.detail.fetch.*` | counter | Attempts, successes, HTTP 304, and failure counts; failures are tagged by reason (`skipped`, `exception`). |
|
||||
| `certbund.parse.success` / `.failures` | counter | Parsing outcomes; histograms capture product and CVE counts. |
|
||||
| `certbund.map.success` / `.failures` | counter | Canonical mapping results; histograms capture affected-package and alias volume. |
|
||||
|
||||
Dashboards should chart coverage days and enqueued counts alongside fetch failures: sharp drops indicate the upstream window tightened or parsing stalled.
|
||||
|
||||
## Logging signals
|
||||
- `CERT-Bund fetch cycle: feed items …` summarises each RSS run (enqueued, already-known, HTTP 304, failures, coverage window).
|
||||
- Parse and map stages log corresponding counts when work remains in the cursor.
|
||||
- Errors include advisory/document identifiers to simplify replays.
|
||||
|
||||
## Historical coverage
|
||||
- RSS contains the newest **250** items (≈6 days at the current publication rate). The connector prunes the “known advisory” set to 512 IDs to avoid unbounded memory but retains enough headroom for short-term replay.
|
||||
- Older advisories remain accessible through the same detail API (`WID-SEC-<year>-<sequence>` identifiers). For deep backfills run a scripted sweep that queues historical IDs in descending order; the connector will persist any payloads that still resolve. Document these batches under source state comments so Merge/Docs can track provenance.
|
||||
|
||||
## Locale & translation stance
|
||||
- CERT-Bund publishes advisory titles and summaries **only in German** (language tag `de`). The connector preserves original casing/content and sets `Advisory.Language = "de"`.
|
||||
- Operator guidance:
|
||||
1. Front-line analysts consuming Concelier data should maintain German literacy or rely on approved machine-translation pipelines.
|
||||
2. When mirroring advisories into English dashboards, store translations outside the canonical advisory payload to keep determinism. Suggested approach: create an auxiliary collection keyed by advisory ID with timestamped translated snippets.
|
||||
3. Offline Kit bundles must document that CERT-Bund content is untranslated to avoid surprise during audits.
|
||||
|
||||
The Docs guild will surface the translation policy (retain German source, optionally layer operator-provided translations) in the broader i18n section; this README is the connector-level reference.
|
||||
@@ -0,0 +1,15 @@
|
||||
<Project Sdk="Microsoft.NET.Sdk">
|
||||
|
||||
<PropertyGroup>
|
||||
<TargetFramework>net10.0</TargetFramework>
|
||||
<ImplicitUsings>enable</ImplicitUsings>
|
||||
<Nullable>enable</Nullable>
|
||||
</PropertyGroup>
|
||||
|
||||
<ItemGroup>
|
||||
<ProjectReference Include="../StellaOps.Plugin/StellaOps.Plugin.csproj" />
|
||||
|
||||
<ProjectReference Include="../StellaOps.Concelier.Connector.Common/StellaOps.Concelier.Connector.Common.csproj" />
|
||||
<ProjectReference Include="../StellaOps.Concelier.Models/StellaOps.Concelier.Models.csproj" />
|
||||
</ItemGroup>
|
||||
</Project>
|
||||
12
src/StellaOps.Concelier.Connector.CertBund/TASKS.md
Normal file
12
src/StellaOps.Concelier.Connector.CertBund/TASKS.md
Normal file
@@ -0,0 +1,12 @@
|
||||
# TASKS
|
||||
| Task | Owner(s) | Depends on | Notes |
|
||||
|---|---|---|---|
|
||||
|FEEDCONN-CERTBUND-02-001 Research CERT-Bund advisory endpoints|BE-Conn-CERTBUND|Research|**DONE (2025-10-11)** – Confirmed public RSS at `https://wid.cert-bund.de/content/public/securityAdvisory/rss` (HTTP 200 w/out cookies), 250-item window, German titles/categories, and detail links pointing to Angular SPA. Captured header profile (no cache hints) and logged open item to discover the JSON API used by `portal` frontend.|
|
||||
|FEEDCONN-CERTBUND-02-002 Fetch job & state persistence|BE-Conn-CERTBUND|Source.Common, Storage.Mongo|**DONE (2025-10-14)** – `CertBundConnector.FetchAsync` consumes RSS via session-bootstrapped client, stores per-advisory JSON documents with metadata + SHA, throttles detail requests, and maintains cursor state (pending docs/mappings, known advisory IDs, last published).|
|
||||
|FEEDCONN-CERTBUND-02-003 Parser/DTO implementation|BE-Conn-CERTBUND|Source.Common|**DONE (2025-10-14)** – Detail JSON piped through `CertBundDetailParser` (raw DOM sanitised to HTML), capturing severity, CVEs, product list, and references into DTO records (`cert-bund.detail.v1`).|
|
||||
|FEEDCONN-CERTBUND-02-004 Canonical mapping & range primitives|BE-Conn-CERTBUND|Models|**DONE (2025-10-14)** – `CertBundMapper` emits canonical advisories (aliases, references, vendor package ranges, provenance) with severity normalisation and deterministic ordering.|
|
||||
|FEEDCONN-CERTBUND-02-005 Regression fixtures & tests|QA|Testing|**DONE (2025-10-14)** – Added `StellaOps.Concelier.Connector.CertBund.Tests` covering fetch→parse→map against canned RSS/JSON fixtures; integration harness uses Mongo2Go + canned HTTP handler; fixtures regenerate via `UPDATE_CERTBUND_FIXTURES=1`.|
|
||||
|FEEDCONN-CERTBUND-02-006 Telemetry & documentation|DevEx|Docs|**DONE (2025-10-15)** – Added `CertBundDiagnostics` (meter `StellaOps.Concelier.Connector.CertBund`) with fetch/parse/map counters + histograms, recorded coverage days, wired stage summary logs, and published the ops runbook (`docs/ops/concelier-certbund-operations.md`).|
|
||||
|FEEDCONN-CERTBUND-02-007 Feed history & locale assessment|BE-Conn-CERTBUND|Research|**DONE (2025-10-15)** – Measured RSS retention (~6 days/≈250 items), captured connector-driven backfill guidance in the runbook, and aligned locale guidance (preserve `language=de`, Docs glossary follow-up). **Next:** coordinate with Tools to land the state-seeding helper so scripted backfills replace manual Mongo tweaks.|
|
||||
|FEEDCONN-CERTBUND-02-008 Session bootstrap & cookie strategy|BE-Conn-CERTBUND|Source.Common|**DONE (2025-10-14)** – Feed client primes the portal session (cookie container via `SocketsHttpHandler`), shares cookies across detail requests, and documents bootstrap behaviour in options (`PortalBootstrapUri`).|
|
||||
|FEEDCONN-CERTBUND-02-009 Offline Kit export packaging|BE-Conn-CERTBUND, Docs|Offline Kit|**DONE (2025-10-17)** – Added `tools/certbund_offline_snapshot.py` to capture search/export JSON, emit deterministic manifests + SHA files, and refreshed docs (`docs/ops/concelier-certbund-operations.md`, `docs/24_OFFLINE_KIT.md`) with offline-kit instructions and manifest layout guidance. Seed data README/ignore rules cover local snapshot hygiene.|
|
||||
Reference in New Issue
Block a user