Restructure solution layout by module
Some checks failed
Docs CI / lint-and-preview (push) Has been cancelled

This commit is contained in:
root
2025-10-28 15:10:40 +02:00
parent 4e3e575db5
commit 68da90a11a
4103 changed files with 192899 additions and 187024 deletions

View File

@@ -0,0 +1,39 @@
# AGENTS
## Role
Implement the CISA ICS advisory connector to ingest US CISA Industrial Control Systems advisories (distinct from the general CERT feed).
## Scope
- Locate the official CISA ICS advisory feed/API (currently HTML/RSS) and define fetch cadence/windowing.
- Build fetch/cursor pipeline with retry/backoff and raw document storage.
- Parse advisory content for summary, impacted vendors/products, mitigation, CVEs.
- Map advisories into canonical `Advisory` records with aliases, references, affected ICS packages, and range primitives.
- Provide deterministic fixtures and automated regression tests.
## Participants
- `Source.Common` (HTTP/fetch utilities, DTO storage).
- `Storage.Mongo` (raw/document/DTO/advisory stores + source state).
- `Concelier.Models` (canonical advisory structures).
- `Concelier.Testing` (integration fixtures and snapshots).
## Interfaces & Contracts
- Job kinds: `ics-cisa:fetch`, `ics-cisa:parse`, `ics-cisa:map`.
- Persist upstream caching metadata (ETag/Last-Modified) when available.
- Alias set should include CISA ICS advisory IDs and referenced CVE IDs.
## In/Out of scope
In scope:
- ICS-specific advisories from CISA.
- Range primitives capturing vendor/equipment metadata.
Out of scope:
- General CISA alerts (covered elsewhere).
## Observability & Security Expectations
- Log fetch attempts, advisory counts, and mapping results.
- Sanitize HTML, removing scripts/styles before persistence.
- Honour upstream rate limits with exponential backoff.
## Tests
- Add `StellaOps.Concelier.Connector.Ics.Cisa.Tests` to cover fetch/parse/map with canned fixtures.
- Snapshot canonical advisories; support fixture regeneration via env flag.
- Ensure deterministic ordering/time normalisation.

View File

@@ -0,0 +1,182 @@
using System;
using System.Collections.Generic;
using System.Diagnostics.CodeAnalysis;
using System.Linq;
using System.Net;
using System.Net.Http;
namespace StellaOps.Concelier.Connector.Ics.Cisa.Configuration;
public sealed class IcsCisaOptions
{
public static string HttpClientName => "source.ics.cisa";
/// <summary>
/// GovDelivery topics RSS endpoint. Feed URIs are constructed from this base.
/// </summary>
public Uri TopicsEndpoint { get; set; } = new("https://content.govdelivery.com/accounts/USDHSCISA/topics.rss", UriKind.Absolute);
/// <summary>
/// GovDelivery personalised subscription code (code=...).
/// </summary>
public string GovDeliveryCode { get; set; } = string.Empty;
/// <summary>
/// Topic identifiers to pull (e.g. USDHSCISA_16 for general ICS advisories).
/// </summary>
public IList<string> TopicIds { get; } = new List<string>
{
"USDHSCISA_16", // ICS advisories (ICSA)
"USDHSCISA_19", // ICS medical advisories (ICSMA)
"USDHSCISA_17", // ICS alerts
};
/// <summary>
/// Optional delay between sequential topic fetches to appease GovDelivery throttling.
/// </summary>
public TimeSpan RequestDelay { get; set; } = TimeSpan.FromMilliseconds(250);
public TimeSpan FailureBackoff { get; set; } = TimeSpan.FromMinutes(5);
public TimeSpan DocumentExpiry { get; set; } = TimeSpan.FromDays(30);
/// <summary>
/// Optional proxy endpoint used when Akamai blocks direct pulls.
/// </summary>
public Uri? ProxyUri { get; set; }
/// <summary>
/// HTTP version requested when contacting GovDelivery.
/// </summary>
public Version RequestVersion { get; set; } = HttpVersion.Version11;
/// <summary>
/// Negotiation policy applied to HTTP requests.
/// </summary>
public HttpVersionPolicy RequestVersionPolicy { get; set; } = HttpVersionPolicy.RequestVersionOrLower;
/// <summary>
/// Maximum number of retry attempts for RSS fetches.
/// </summary>
public int MaxAttempts { get; set; } = 4;
/// <summary>
/// Base delay used for exponential backoff between attempts.
/// </summary>
public TimeSpan BaseDelay { get; set; } = TimeSpan.FromSeconds(3);
/// <summary>
/// Base URI used when fetching HTML detail pages.
/// </summary>
public Uri DetailBaseUri { get; set; } = new("https://www.cisa.gov/", UriKind.Absolute);
/// <summary>
/// Optional timeout override applied to detail page fetches.
/// </summary>
public TimeSpan DetailRequestTimeout { get; set; } = TimeSpan.FromSeconds(25);
/// <summary>
/// Additional hosts allowed by the connector (detail pages, attachments).
/// </summary>
public IList<string> AdditionalHosts { get; } = new List<string>
{
"www.cisa.gov",
"cisa.gov"
};
public bool EnableDetailScrape { get; set; } = true;
public bool CaptureAttachments { get; set; } = true;
[MemberNotNull(nameof(TopicsEndpoint), nameof(GovDeliveryCode))]
public void Validate()
{
if (TopicsEndpoint is null || !TopicsEndpoint.IsAbsoluteUri)
{
throw new InvalidOperationException("TopicsEndpoint must be an absolute URI.");
}
if (string.IsNullOrWhiteSpace(GovDeliveryCode))
{
throw new InvalidOperationException("GovDeliveryCode must be provided.");
}
if (TopicIds.Count == 0)
{
throw new InvalidOperationException("At least one GovDelivery topic identifier is required.");
}
foreach (var topic in TopicIds)
{
if (string.IsNullOrWhiteSpace(topic))
{
throw new InvalidOperationException("Topic identifiers cannot be blank.");
}
}
if (RequestDelay < TimeSpan.Zero)
{
throw new InvalidOperationException("RequestDelay cannot be negative.");
}
if (FailureBackoff <= TimeSpan.Zero)
{
throw new InvalidOperationException("FailureBackoff must be greater than zero.");
}
if (DocumentExpiry <= TimeSpan.Zero)
{
throw new InvalidOperationException("DocumentExpiry must be greater than zero.");
}
if (MaxAttempts <= 0)
{
throw new InvalidOperationException("MaxAttempts must be positive.");
}
if (BaseDelay <= TimeSpan.Zero)
{
throw new InvalidOperationException("BaseDelay must be greater than zero.");
}
if (DetailBaseUri is null || !DetailBaseUri.IsAbsoluteUri)
{
throw new InvalidOperationException("DetailBaseUri must be an absolute URI.");
}
if (DetailRequestTimeout <= TimeSpan.Zero)
{
throw new InvalidOperationException("DetailRequestTimeout must be greater than zero.");
}
if (ProxyUri is not null && !ProxyUri.IsAbsoluteUri)
{
throw new InvalidOperationException("ProxyUri must be an absolute URI when specified.");
}
foreach (var host in AdditionalHosts)
{
if (string.IsNullOrWhiteSpace(host))
{
throw new InvalidOperationException("Additional host entries cannot be blank.");
}
}
}
public Uri BuildTopicUri(string topicId)
{
ArgumentException.ThrowIfNullOrEmpty(topicId);
Validate();
var builder = new UriBuilder(TopicsEndpoint);
var query = new Dictionary<string, string>(StringComparer.OrdinalIgnoreCase)
{
["code"] = GovDeliveryCode,
["format"] = "xml",
["topic_id"] = topicId.Trim(),
};
builder.Query = string.Join("&", query.Select(pair => $"{Uri.EscapeDataString(pair.Key)}={Uri.EscapeDataString(pair.Value)}"));
return builder.Uri;
}
}

View File

@@ -0,0 +1,21 @@
# ICS CISA Connector Status (2025-10-16)
## Context
- Proxy plumbing for GovDelivery (`SourceHttpClientOptions.Proxy*`) is implemented and covered by `SourceHttpClientBuilderTests.AddSourceHttpClient_LoadsProxyConfiguration`.
- Detail enrichment now extracts mitigation paragraphs/bullets, merges them with feed data, and emits `mitigation` references plus combined alias sets.
- `BuildAffectedPackages` parses product/version pairs and now persists SemVer exact values for canonical ranges via the advisory store.
## Current Outcomes
- Feed parser fixtures were refreshed so vendor PDFs stay surfaced as attachments; DTO references continue including canonical links.
- SemVer primitive deserialisation now restores `exactValue` (e.g., `"4.2"``"4.2.0"`), keeping connector snapshots deterministic.
- Console debugging noise was removed from connector/parser code.
- Ops runbook documents attachment + SemVer validation steps for dry runs.
- `dotnet test src/Concelier/__Tests/StellaOps.Concelier.Connector.Ics.Cisa.Tests/StellaOps.Concelier.Connector.Ics.Cisa.Tests.csproj` passes (2025-10-16).
## Outstanding Items
- None. Continue monitoring Akamai access decisions and proxy requirements via Ops feedback.
## Verification Checklist
-`dotnet test src/Concelier/__Tests/StellaOps.Concelier.Connector.Ics.Cisa.Tests/StellaOps.Concelier.Connector.Ics.Cisa.Tests.csproj`
-`dotnet test src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.Common.Tests/StellaOps.Concelier.Connector.Common.Tests.csproj` (proxy support) — rerun when Source.Common changes land.
- Keep this summary aligned with `TASKS.md` as further work emerges.

View File

@@ -0,0 +1,19 @@
using Microsoft.Extensions.DependencyInjection;
using StellaOps.Plugin;
namespace StellaOps.Concelier.Connector.Ics.Cisa;
public sealed class IcsCisaConnectorPlugin : IConnectorPlugin
{
public const string SourceName = "ics-cisa";
public string Name => SourceName;
public bool IsAvailable(IServiceProvider services) => services is not null;
public IFeedConnector Create(IServiceProvider services)
{
ArgumentNullException.ThrowIfNull(services);
return ActivatorUtilities.CreateInstance<IcsCisaConnector>(services);
}
}

View File

@@ -0,0 +1,56 @@
using System;
using Microsoft.Extensions.Configuration;
using Microsoft.Extensions.DependencyInjection;
using StellaOps.DependencyInjection;
using StellaOps.Concelier.Core.Jobs;
using StellaOps.Concelier.Connector.Ics.Cisa.Configuration;
namespace StellaOps.Concelier.Connector.Ics.Cisa;
public sealed class IcsCisaDependencyInjectionRoutine : IDependencyInjectionRoutine
{
private const string ConfigurationSection = "concelier:sources:ics-cisa";
public IServiceCollection Register(IServiceCollection services, IConfiguration configuration)
{
ArgumentNullException.ThrowIfNull(services);
ArgumentNullException.ThrowIfNull(configuration);
services.AddIcsCisaConnector(options =>
{
configuration.GetSection(ConfigurationSection).Bind(options);
options.Validate();
});
services.AddTransient<IcsCisaFetchJob>();
services.AddTransient<IcsCisaParseJob>();
services.AddTransient<IcsCisaMapJob>();
services.PostConfigure<JobSchedulerOptions>(options =>
{
EnsureJob(options, IcsCisaJobKinds.Fetch, typeof(IcsCisaFetchJob));
EnsureJob(options, IcsCisaJobKinds.Parse, typeof(IcsCisaParseJob));
EnsureJob(options, IcsCisaJobKinds.Map, typeof(IcsCisaMapJob));
});
return services;
}
private static void EnsureJob(JobSchedulerOptions options, string kind, Type jobType)
{
ArgumentNullException.ThrowIfNull(options);
if (options.Definitions.ContainsKey(kind))
{
return;
}
options.Definitions[kind] = new JobDefinition(
kind,
jobType,
options.DefaultTimeout,
options.DefaultLeaseDuration,
CronExpression: null,
Enabled: true);
}
}

View File

@@ -0,0 +1,60 @@
using System;
using System.Net;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Options;
using StellaOps.Concelier.Connector.Common.Http;
using StellaOps.Concelier.Connector.Ics.Cisa.Configuration;
using StellaOps.Concelier.Connector.Ics.Cisa.Internal;
namespace StellaOps.Concelier.Connector.Ics.Cisa;
public static class IcsCisaServiceCollectionExtensions
{
public static IServiceCollection AddIcsCisaConnector(this IServiceCollection services, Action<IcsCisaOptions> configure)
{
ArgumentNullException.ThrowIfNull(services);
ArgumentNullException.ThrowIfNull(configure);
services.AddOptions<IcsCisaOptions>()
.Configure(configure)
.PostConfigure(static opts => opts.Validate());
services.AddSourceHttpClient(IcsCisaOptions.HttpClientName, (sp, clientOptions) =>
{
var options = sp.GetRequiredService<IOptions<IcsCisaOptions>>().Value;
clientOptions.BaseAddress = new Uri(options.TopicsEndpoint.GetLeftPart(UriPartial.Authority));
clientOptions.Timeout = TimeSpan.FromSeconds(45);
clientOptions.UserAgent = "StellaOps.Concelier.IcsCisa/1.0";
clientOptions.AllowedHosts.Clear();
clientOptions.AllowedHosts.Add(options.TopicsEndpoint.Host);
clientOptions.AllowedHosts.Add(options.DetailBaseUri.Host);
foreach (var host in options.AdditionalHosts)
{
clientOptions.AllowedHosts.Add(host);
}
clientOptions.DefaultRequestHeaders["Accept"] = "application/rss+xml";
clientOptions.RequestVersion = options.RequestVersion;
clientOptions.VersionPolicy = options.RequestVersionPolicy;
clientOptions.MaxAttempts = options.MaxAttempts;
clientOptions.BaseDelay = options.BaseDelay;
clientOptions.EnableMultipleHttp2Connections = false;
clientOptions.ConfigureHandler = handler =>
{
handler.AutomaticDecompression = DecompressionMethods.All;
};
if (options.ProxyUri is not null)
{
clientOptions.ProxyAddress = options.ProxyUri;
clientOptions.ProxyBypassOnLocal = false;
}
});
services.AddSingleton<IcsCisaDiagnostics>();
services.AddSingleton<IcsCisaFeedParser>();
services.AddTransient<IcsCisaConnector>();
return services;
}
}

View File

@@ -0,0 +1,56 @@
using System;
using System.Collections.Generic;
using System.Text.Json.Serialization;
namespace StellaOps.Concelier.Connector.Ics.Cisa.Internal;
public sealed record IcsCisaAdvisoryDto
{
[JsonPropertyName("advisoryId")]
public required string AdvisoryId { get; init; }
[JsonPropertyName("title")]
public required string Title { get; init; }
[JsonPropertyName("link")]
public required string Link { get; init; }
[JsonPropertyName("summary")]
public string? Summary { get; init; }
[JsonPropertyName("descriptionHtml")]
public string DescriptionHtml { get; init; } = string.Empty;
[JsonPropertyName("published")]
public DateTimeOffset Published { get; init; }
[JsonPropertyName("updated")]
public DateTimeOffset? Updated { get; init; }
[JsonPropertyName("medical")]
public bool IsMedical { get; init; }
[JsonPropertyName("aliases")]
public IReadOnlyCollection<string> Aliases { get; init; } = Array.Empty<string>();
[JsonPropertyName("cveIds")]
public IReadOnlyCollection<string> CveIds { get; init; } = Array.Empty<string>();
[JsonPropertyName("vendors")]
public IReadOnlyCollection<string> Vendors { get; init; } = Array.Empty<string>();
[JsonPropertyName("products")]
public IReadOnlyCollection<string> Products { get; init; } = Array.Empty<string>();
[JsonPropertyName("references")]
public IReadOnlyCollection<string> References { get; init; } = Array.Empty<string>();
[JsonPropertyName("attachments")]
public IReadOnlyCollection<IcsCisaAttachmentDto> Attachments { get; init; } = Array.Empty<IcsCisaAttachmentDto>();
[JsonPropertyName("mitigations")]
public IReadOnlyCollection<string> Mitigations { get; init; } = Array.Empty<string>();
[JsonPropertyName("detailHtml")]
public string? DetailHtml { get; init; }
}

View File

@@ -0,0 +1,12 @@
using System.Text.Json.Serialization;
namespace StellaOps.Concelier.Connector.Ics.Cisa.Internal;
public sealed record IcsCisaAttachmentDto
{
[JsonPropertyName("title")]
public string? Title { get; init; }
[JsonPropertyName("url")]
public required string Url { get; init; }
}

View File

@@ -0,0 +1,88 @@
using System;
using System.Collections.Generic;
using System.Linq;
using MongoDB.Bson;
namespace StellaOps.Concelier.Connector.Ics.Cisa.Internal;
internal sealed record IcsCisaCursor(
DateTimeOffset? LastPublished,
IReadOnlyCollection<Guid> PendingDocuments,
IReadOnlyCollection<Guid> PendingMappings)
{
public static IcsCisaCursor Empty { get; } = new(null, Array.Empty<Guid>(), Array.Empty<Guid>());
public BsonDocument ToBsonDocument()
{
var document = new BsonDocument
{
["pendingDocuments"] = new BsonArray(PendingDocuments.Select(static id => id.ToString())),
["pendingMappings"] = new BsonArray(PendingMappings.Select(static id => id.ToString())),
};
if (LastPublished.HasValue)
{
document["lastPublished"] = LastPublished.Value.UtcDateTime;
}
return document;
}
public static IcsCisaCursor FromBson(BsonDocument? document)
{
if (document is null || document.ElementCount == 0)
{
return Empty;
}
var lastPublished = document.TryGetValue("lastPublished", out var publishedValue)
? ParseDate(publishedValue)
: null;
return new IcsCisaCursor(
lastPublished,
ReadGuidArray(document, "pendingDocuments"),
ReadGuidArray(document, "pendingMappings"));
}
public IcsCisaCursor WithPendingDocuments(IEnumerable<Guid> ids)
=> this with { PendingDocuments = ids?.Distinct().ToArray() ?? Array.Empty<Guid>() };
public IcsCisaCursor WithPendingMappings(IEnumerable<Guid> ids)
=> this with { PendingMappings = ids?.Distinct().ToArray() ?? Array.Empty<Guid>() };
public IcsCisaCursor WithLastPublished(DateTimeOffset? published)
=> this with { LastPublished = published?.ToUniversalTime() };
private static DateTimeOffset? ParseDate(BsonValue value)
=> value.BsonType switch
{
BsonType.DateTime => DateTime.SpecifyKind(value.ToUniversalTime(), DateTimeKind.Utc),
BsonType.String when DateTimeOffset.TryParse(value.AsString, out var parsed) => parsed.ToUniversalTime(),
_ => null,
};
private static IReadOnlyCollection<Guid> ReadGuidArray(BsonDocument document, string field)
{
if (!document.TryGetValue(field, out var value) || value is not BsonArray array)
{
return Array.Empty<Guid>();
}
var results = new List<Guid>(array.Count);
foreach (var element in array)
{
if (element is null)
{
continue;
}
if (Guid.TryParse(element.ToString(), out var guid))
{
results.Add(guid);
}
}
return results;
}
}

View File

@@ -0,0 +1,171 @@
using System;
using System.Collections.Generic;
using System.Diagnostics.Metrics;
namespace StellaOps.Concelier.Connector.Ics.Cisa.Internal;
public sealed class IcsCisaDiagnostics : IDisposable
{
private const string MeterName = "StellaOps.Concelier.Connector.Ics.Cisa";
private const string MeterVersion = "1.0.0";
private readonly Meter _meter;
private readonly Counter<long> _fetchAttempts;
private readonly Counter<long> _fetchSuccess;
private readonly Counter<long> _fetchFailures;
private readonly Counter<long> _fetchNotModified;
private readonly Counter<long> _fetchFallbacks;
private readonly Histogram<long> _fetchDocuments;
private readonly Counter<long> _parseSuccess;
private readonly Counter<long> _parseFailures;
private readonly Histogram<long> _parseAdvisoryCount;
private readonly Histogram<long> _parseAttachmentCount;
private readonly Histogram<long> _parseDetailCount;
private readonly Counter<long> _detailSuccess;
private readonly Counter<long> _detailFailures;
private readonly Counter<long> _mapSuccess;
private readonly Counter<long> _mapFailures;
private readonly Histogram<long> _mapReferenceCount;
private readonly Histogram<long> _mapPackageCount;
private readonly Histogram<long> _mapAliasCount;
public IcsCisaDiagnostics()
{
_meter = new Meter(MeterName, MeterVersion);
_fetchAttempts = _meter.CreateCounter<long>("icscisa.fetch.attempts", unit: "operations");
_fetchSuccess = _meter.CreateCounter<long>("icscisa.fetch.success", unit: "operations");
_fetchFailures = _meter.CreateCounter<long>("icscisa.fetch.failures", unit: "operations");
_fetchNotModified = _meter.CreateCounter<long>("icscisa.fetch.not_modified", unit: "operations");
_fetchFallbacks = _meter.CreateCounter<long>("icscisa.fetch.fallbacks", unit: "operations");
_fetchDocuments = _meter.CreateHistogram<long>("icscisa.fetch.documents", unit: "documents");
_parseSuccess = _meter.CreateCounter<long>("icscisa.parse.success", unit: "documents");
_parseFailures = _meter.CreateCounter<long>("icscisa.parse.failures", unit: "documents");
_parseAdvisoryCount = _meter.CreateHistogram<long>("icscisa.parse.advisories", unit: "advisories");
_parseAttachmentCount = _meter.CreateHistogram<long>("icscisa.parse.attachments", unit: "attachments");
_parseDetailCount = _meter.CreateHistogram<long>("icscisa.parse.detail_fetches", unit: "fetches");
_detailSuccess = _meter.CreateCounter<long>("icscisa.detail.success", unit: "operations");
_detailFailures = _meter.CreateCounter<long>("icscisa.detail.failures", unit: "operations");
_mapSuccess = _meter.CreateCounter<long>("icscisa.map.success", unit: "advisories");
_mapFailures = _meter.CreateCounter<long>("icscisa.map.failures", unit: "advisories");
_mapReferenceCount = _meter.CreateHistogram<long>("icscisa.map.references", unit: "references");
_mapPackageCount = _meter.CreateHistogram<long>("icscisa.map.packages", unit: "packages");
_mapAliasCount = _meter.CreateHistogram<long>("icscisa.map.aliases", unit: "aliases");
}
public void FetchAttempt(string topicId)
{
_fetchAttempts.Add(1, BuildTopicTags(topicId));
}
public void FetchSuccess(string topicId, int documentsAdded)
{
var tags = BuildTopicTags(topicId);
_fetchSuccess.Add(1, tags);
if (documentsAdded > 0)
{
_fetchDocuments.Record(documentsAdded, tags);
}
}
public void FetchNotModified(string topicId)
{
_fetchNotModified.Add(1, BuildTopicTags(topicId));
}
public void FetchFallback(string topicId)
{
_fetchFallbacks.Add(1, BuildTopicTags(topicId));
}
public void FetchFailure(string topicId)
{
_fetchFailures.Add(1, BuildTopicTags(topicId));
}
public void ParseSuccess(string topicId, int advisoryCount, int attachmentCount, int detailFetchCount)
{
var tags = BuildTopicTags(topicId);
_parseSuccess.Add(1, tags);
if (advisoryCount >= 0)
{
_parseAdvisoryCount.Record(advisoryCount, tags);
}
if (attachmentCount >= 0)
{
_parseAttachmentCount.Record(attachmentCount, tags);
}
if (detailFetchCount >= 0)
{
_parseDetailCount.Record(detailFetchCount, tags);
}
}
public void ParseFailure(string topicId)
{
_parseFailures.Add(1, BuildTopicTags(topicId));
}
public void DetailFetchSuccess(string advisoryId)
{
_detailSuccess.Add(1, BuildAdvisoryTags(advisoryId));
}
public void DetailFetchFailure(string advisoryId)
{
_detailFailures.Add(1, BuildAdvisoryTags(advisoryId));
}
public void MapSuccess(string advisoryId, int referenceCount, int packageCount, int aliasCount)
{
var tags = BuildAdvisoryTags(advisoryId);
_mapSuccess.Add(1, tags);
if (referenceCount >= 0)
{
_mapReferenceCount.Record(referenceCount, tags);
}
if (packageCount >= 0)
{
_mapPackageCount.Record(packageCount, tags);
}
if (aliasCount >= 0)
{
_mapAliasCount.Record(aliasCount, tags);
}
}
public void MapFailure(string advisoryId)
{
_mapFailures.Add(1, BuildAdvisoryTags(advisoryId));
}
private static KeyValuePair<string, object?>[] BuildTopicTags(string? topicId)
=> new[]
{
new KeyValuePair<string, object?>("concelier.source", IcsCisaConnectorPlugin.SourceName),
new KeyValuePair<string, object?>("icscisa.topic", string.IsNullOrWhiteSpace(topicId) ? "unknown" : topicId)
};
private static KeyValuePair<string, object?>[] BuildAdvisoryTags(string? advisoryId)
=> new[]
{
new KeyValuePair<string, object?>("concelier.source", IcsCisaConnectorPlugin.SourceName),
new KeyValuePair<string, object?>("icscisa.advisory", string.IsNullOrWhiteSpace(advisoryId) ? "unknown" : advisoryId)
};
public void Dispose()
{
_meter.Dispose();
}
}

View File

@@ -0,0 +1,16 @@
using System.Collections.Generic;
using System.Text.Json.Serialization;
namespace StellaOps.Concelier.Connector.Ics.Cisa.Internal;
public sealed record IcsCisaFeedDto
{
[JsonPropertyName("topicId")]
public required string TopicId { get; init; }
[JsonPropertyName("feedUri")]
public required string FeedUri { get; init; }
[JsonPropertyName("advisories")]
public IReadOnlyCollection<IcsCisaAdvisoryDto> Advisories { get; init; } = new List<IcsCisaAdvisoryDto>();
}

View File

@@ -0,0 +1,402 @@
using System;
using System.Collections.Generic;
using System.IO;
using System.Linq;
using System.ServiceModel.Syndication;
using System.Text;
using System.Text.RegularExpressions;
using System.Xml;
using AngleSharp.Html.Parser;
using AngleSharp.Html.Dom;
using StellaOps.Concelier.Connector.Common.Html;
namespace StellaOps.Concelier.Connector.Ics.Cisa.Internal;
public sealed class IcsCisaFeedParser
{
private static readonly Regex AdvisoryIdRegex = new(@"^(?<id>ICS[AM]?A?-?\d{2}-\d{3}[A-Z]?(-\d{2})?)", RegexOptions.IgnoreCase | RegexOptions.Compiled);
private static readonly Regex CveRegex = new(@"CVE-\d{4}-\d{4,}", RegexOptions.IgnoreCase | RegexOptions.Compiled);
private readonly HtmlContentSanitizer _sanitizer = new();
private readonly HtmlParser _htmlParser = new();
public IReadOnlyCollection<IcsCisaAdvisoryDto> Parse(Stream rssStream, bool isMedicalTopic, Uri? topicUri)
{
if (rssStream is null)
{
return Array.Empty<IcsCisaAdvisoryDto>();
}
using var reader = XmlReader.Create(rssStream, new XmlReaderSettings
{
DtdProcessing = DtdProcessing.Ignore,
IgnoreComments = true,
IgnoreProcessingInstructions = true,
});
var feed = SyndicationFeed.Load(reader);
if (feed is null || feed.Items is null)
{
return Array.Empty<IcsCisaAdvisoryDto>();
}
var advisories = new List<IcsCisaAdvisoryDto>();
foreach (var item in feed.Items)
{
var dto = ConvertItem(item, isMedicalTopic, topicUri);
if (dto is not null)
{
advisories.Add(dto);
}
}
return advisories;
}
private IcsCisaAdvisoryDto? ConvertItem(SyndicationItem item, bool isMedicalTopic, Uri? topicUri)
{
if (item is null)
{
return null;
}
var title = item.Title?.Text?.Trim();
if (string.IsNullOrWhiteSpace(title))
{
return null;
}
var advisoryId = ExtractAdvisoryId(title);
if (string.IsNullOrWhiteSpace(advisoryId))
{
return null;
}
var linkUri = item.Links.FirstOrDefault()?.Uri;
if (linkUri is null && !string.IsNullOrWhiteSpace(item.Id) && Uri.TryCreate(item.Id, UriKind.Absolute, out var fallback))
{
linkUri = fallback;
}
if (linkUri is null)
{
return null;
}
var contentHtml = ExtractContentHtml(item);
var sanitizedHtml = _sanitizer.Sanitize(contentHtml, linkUri);
var textContent = ExtractTextContent(sanitizedHtml);
var aliases = new HashSet<string>(StringComparer.OrdinalIgnoreCase) { advisoryId };
var cveIds = ExtractCveIds(textContent, aliases);
var vendors = ExtractList(sanitizedHtml, textContent, "Vendor");
var products = ExtractList(sanitizedHtml, textContent, "Products");
if (products.Count == 0)
{
products = ExtractList(sanitizedHtml, textContent, "Product");
}
var attachments = ExtractAttachments(sanitizedHtml, linkUri);
var references = ExtractReferences(sanitizedHtml, linkUri);
var published = item.PublishDate != DateTimeOffset.MinValue
? item.PublishDate.ToUniversalTime()
: item.LastUpdatedTime.ToUniversalTime();
var updated = item.LastUpdatedTime != DateTimeOffset.MinValue
? item.LastUpdatedTime.ToUniversalTime()
: (DateTimeOffset?)null;
return new IcsCisaAdvisoryDto
{
AdvisoryId = advisoryId,
Title = title,
Link = linkUri.ToString(),
Summary = item.Summary?.Text?.Trim(),
DescriptionHtml = sanitizedHtml,
Published = published,
Updated = updated,
IsMedical = isMedicalTopic || advisoryId.StartsWith("ICSMA", StringComparison.OrdinalIgnoreCase),
Aliases = aliases.ToArray(),
CveIds = cveIds,
Vendors = vendors,
Products = products,
References = references,
Attachments = attachments,
};
}
private static string ExtractAdvisoryId(string title)
{
if (string.IsNullOrWhiteSpace(title))
{
return string.Empty;
}
var colonIndex = title.IndexOf(':');
var candidate = colonIndex > 0 ? title[..colonIndex] : title;
var match = AdvisoryIdRegex.Match(candidate);
if (match.Success)
{
var id = match.Groups["id"].Value.Trim();
return id.ToUpperInvariant();
}
return candidate.Trim();
}
private static string ExtractContentHtml(SyndicationItem item)
{
if (item.Content is TextSyndicationContent textContent)
{
return textContent.Text ?? string.Empty;
}
if (item.Summary is not null)
{
return item.Summary.Text ?? string.Empty;
}
if (item.ElementExtensions is not null)
{
foreach (var extension in item.ElementExtensions)
{
try
{
var value = extension.GetObject<string>();
if (!string.IsNullOrWhiteSpace(value))
{
return value;
}
}
catch
{
// ignore malformed extensions
}
}
}
return string.Empty;
}
private static IReadOnlyCollection<string> ExtractCveIds(string text, HashSet<string> aliases)
{
if (string.IsNullOrWhiteSpace(text))
{
return Array.Empty<string>();
}
var matches = CveRegex.Matches(text);
if (matches.Count == 0)
{
return Array.Empty<string>();
}
var ids = new HashSet<string>(StringComparer.OrdinalIgnoreCase);
foreach (Match match in matches)
{
if (!match.Success)
{
continue;
}
var value = match.Value.ToUpperInvariant();
if (ids.Add(value))
{
aliases.Add(value);
}
}
return ids.ToArray();
}
private IReadOnlyCollection<string> ExtractList(string sanitizedHtml, string textContent, string key)
{
if (string.IsNullOrWhiteSpace(sanitizedHtml))
{
return Array.Empty<string>();
}
var items = new HashSet<string>(StringComparer.OrdinalIgnoreCase);
try
{
var document = _htmlParser.ParseDocument(sanitizedHtml);
foreach (var element in document.All)
{
if (element is IHtmlParagraphElement or IHtmlDivElement or IHtmlSpanElement or IHtmlListItemElement)
{
var content = element.TextContent?.Trim();
if (string.IsNullOrWhiteSpace(content))
{
continue;
}
if (content.StartsWith($"{key}:", StringComparison.OrdinalIgnoreCase))
{
var line = content[(key.Length + 1)..].Trim();
foreach (var part in line.Split(new[] { ',', ';' }, StringSplitOptions.RemoveEmptyEntries))
{
var value = part.Trim();
if (!string.IsNullOrWhiteSpace(value))
{
items.Add(value);
}
}
}
}
}
}
catch
{
// ignore HTML parsing failures; fallback to text processing below
}
if (items.Count == 0 && !string.IsNullOrWhiteSpace(textContent))
{
using var reader = new StringReader(textContent);
string? line;
while ((line = reader.ReadLine()) is not null)
{
if (line.StartsWith($"{key}:", StringComparison.OrdinalIgnoreCase))
{
var raw = line[(key.Length + 1)..].Trim();
foreach (var part in raw.Split(new[] { ',', ';' }, StringSplitOptions.RemoveEmptyEntries))
{
var value = part.Trim();
if (!string.IsNullOrWhiteSpace(value))
{
items.Add(value);
}
}
}
}
}
return items.ToArray();
}
private IReadOnlyCollection<IcsCisaAttachmentDto> ExtractAttachments(string sanitizedHtml, Uri linkUri)
{
if (string.IsNullOrWhiteSpace(sanitizedHtml))
{
return Array.Empty<IcsCisaAttachmentDto>();
}
try
{
var document = _htmlParser.ParseDocument(sanitizedHtml);
var attachments = new List<IcsCisaAttachmentDto>();
foreach (var anchor in document.QuerySelectorAll("a"))
{
var href = anchor.GetAttribute("href");
if (string.IsNullOrWhiteSpace(href))
{
continue;
}
if (!Uri.TryCreate(linkUri, href, out var resolved))
{
continue;
}
var url = resolved.ToString();
if (!url.EndsWith(".pdf", StringComparison.OrdinalIgnoreCase) &&
!url.Contains("/pdf", StringComparison.OrdinalIgnoreCase))
{
continue;
}
attachments.Add(new IcsCisaAttachmentDto
{
Title = anchor.TextContent?.Trim(),
Url = url,
});
}
return attachments.Count == 0 ? Array.Empty<IcsCisaAttachmentDto>() : attachments;
}
catch
{
return Array.Empty<IcsCisaAttachmentDto>();
}
}
private IReadOnlyCollection<string> ExtractReferences(string sanitizedHtml, Uri linkUri)
{
if (string.IsNullOrWhiteSpace(sanitizedHtml))
{
return new[] { linkUri.ToString() };
}
try
{
var document = _htmlParser.ParseDocument(sanitizedHtml);
var links = new HashSet<string>(StringComparer.OrdinalIgnoreCase)
{
linkUri.ToString()
};
foreach (var anchor in document.QuerySelectorAll("a"))
{
var href = anchor.GetAttribute("href");
if (string.IsNullOrWhiteSpace(href))
{
continue;
}
if (Uri.TryCreate(linkUri, href, out var resolved))
{
links.Add(resolved.ToString());
}
}
return links.ToArray();
}
catch
{
return new[] { linkUri.ToString() };
}
}
private string ExtractTextContent(string sanitizedHtml)
{
if (string.IsNullOrWhiteSpace(sanitizedHtml))
{
return string.Empty;
}
try
{
var document = _htmlParser.ParseDocument(sanitizedHtml);
var builder = new StringBuilder();
var body = document.Body ?? document.DocumentElement;
if (body is null)
{
return string.Empty;
}
foreach (var node in body.ChildNodes)
{
var text = node.TextContent;
if (string.IsNullOrWhiteSpace(text))
{
continue;
}
if (builder.Length > 0)
{
builder.AppendLine();
}
builder.Append(text.Trim());
}
return builder.ToString();
}
catch
{
return sanitizedHtml;
}
}
}

View File

@@ -0,0 +1,46 @@
using System;
using System.Threading;
using System.Threading.Tasks;
using StellaOps.Concelier.Core.Jobs;
namespace StellaOps.Concelier.Connector.Ics.Cisa;
internal static class IcsCisaJobKinds
{
public const string Fetch = "source:ics-cisa:fetch";
public const string Parse = "source:ics-cisa:parse";
public const string Map = "source:ics-cisa:map";
}
internal sealed class IcsCisaFetchJob : IJob
{
private readonly IcsCisaConnector _connector;
public IcsCisaFetchJob(IcsCisaConnector connector)
=> _connector = connector ?? throw new ArgumentNullException(nameof(connector));
public Task ExecuteAsync(JobExecutionContext context, CancellationToken cancellationToken)
=> _connector.FetchAsync(context.Services, cancellationToken);
}
internal sealed class IcsCisaParseJob : IJob
{
private readonly IcsCisaConnector _connector;
public IcsCisaParseJob(IcsCisaConnector connector)
=> _connector = connector ?? throw new ArgumentNullException(nameof(connector));
public Task ExecuteAsync(JobExecutionContext context, CancellationToken cancellationToken)
=> _connector.ParseAsync(context.Services, cancellationToken);
}
internal sealed class IcsCisaMapJob : IJob
{
private readonly IcsCisaConnector _connector;
public IcsCisaMapJob(IcsCisaConnector connector)
=> _connector = connector ?? throw new ArgumentNullException(nameof(connector));
public Task ExecuteAsync(JobExecutionContext context, CancellationToken cancellationToken)
=> _connector.MapAsync(context.Services, cancellationToken);
}

View File

@@ -0,0 +1,29 @@
<?xml version='1.0' encoding='utf-8'?>
<Project Sdk="Microsoft.NET.Sdk">
<PropertyGroup>
<TargetFramework>net10.0</TargetFramework>
<ImplicitUsings>enable</ImplicitUsings>
<Nullable>enable</Nullable>
</PropertyGroup>
<ItemGroup>
<ProjectReference Include="../../../__Libraries/StellaOps.Plugin/StellaOps.Plugin.csproj" />
<ProjectReference Include="../StellaOps.Concelier.Connector.Common/StellaOps.Concelier.Connector.Common.csproj" />
<ProjectReference Include="../StellaOps.Concelier.Core/StellaOps.Concelier.Core.csproj" />
<ProjectReference Include="../StellaOps.Concelier.Models/StellaOps.Concelier.Models.csproj" />
<ProjectReference Include="../StellaOps.Concelier.Storage.Mongo/StellaOps.Concelier.Storage.Mongo.csproj" />
</ItemGroup>
<ItemGroup>
<PackageReference Include="System.ServiceModel.Syndication" Version="8.0.0" />
</ItemGroup>
<ItemGroup>
<AssemblyAttribute Include="System.Runtime.CompilerServices.InternalsVisibleTo">
<_Parameter1>StellaOps.Concelier.Connector.Ics.Cisa.Tests</_Parameter1>
</AssemblyAttribute>
</ItemGroup>
</Project>

View File

@@ -0,0 +1,15 @@
# TASKS
| Task | Owner(s) | Depends on | Notes |
|---|---|---|---|
|FEEDCONN-ICSCISA-02-001 Document CISA ICS feed contract|BE-Conn-ICS-CISA|Research|**DONE (2025-10-11)** `https://www.cisa.gov/cybersecurity-advisories/ics-advisories.xml` and legacy `/sites/default/files/feeds/...` return Akamai 403 even with browser UA; HTML landing page blocked as well. Logged full headers (x-reference-error, AkamaiGHost) in `docs/concelier-connector-research-20251011.md` and initiated GovDelivery access request.|
|FEEDCONN-ICSCISA-02-002 Fetch pipeline & cursor storage|BE-Conn-ICS-CISA|Source.Common, Storage.Mongo|**DONE (2025-10-16)** Confirmed proxy knobs + cursor state behave with the refreshed fixtures; ops runbook now captures proxy usage/validation so the fetch stage is production-ready.|
|FEEDCONN-ICSCISA-02-003 DTO/parser implementation|BE-Conn-ICS-CISA|Source.Common|**DONE (2025-10-16)** Feed parser fixtures updated to retain vendor PDFs as attachments while maintaining reference coverage; console diagnostics removed.|
|FEEDCONN-ICSCISA-02-004 Canonical mapping & range primitives|BE-Conn-ICS-CISA|Models|**DONE (2025-10-16)** `TryCreateSemVerPrimitive` flow + Mongo deserialiser now persist `exactValue` (`4.2``4.2.0`), unblocking canonical snapshots.|
|FEEDCONN-ICSCISA-02-005 Deterministic fixtures/tests|QA|Testing|**DONE (2025-10-16)** `dotnet test src/Concelier/__Tests/StellaOps.Concelier.Connector.Ics.Cisa.Tests/...` passes; fixtures assert attachment handling + SemVer semantics.|
|FEEDCONN-ICSCISA-02-006 Telemetry & documentation|DevEx|Docs|**DONE (2025-10-16)** Ops guide documents attachment checks, SemVer exact values, and proxy guidance; diagnostics remain unchanged.|
|FEEDCONN-ICSCISA-02-007 Detail document inventory|BE-Conn-ICS-CISA|Research|**DONE (2025-10-16)** Validated canned detail pages vs feed output so attachment inventories stay aligned; archived expectations noted in `HANDOVER.md`.|
|FEEDCONN-ICSCISA-02-008 Distribution fallback strategy|BE-Conn-ICS-CISA|Research|**DONE (2025-10-11)** Outlined GovDelivery token request, HTML scrape + email digest fallback, and dependency on Ops for credential workflow; awaiting decision before fetch implementation.|
|FEEDCONN-ICSCISA-02-009 GovDelivery credential onboarding|Ops, BE-Conn-ICS-CISA|Ops|**DONE (2025-10-14)** GovDelivery onboarding runbook captured in `docs/ops/concelier-icscisa-operations.md`; secret vault path and Offline Kit handling documented.|
|FEEDCONN-ICSCISA-02-010 Mitigation & SemVer polish|BE-Conn-ICS-CISA|02-003, 02-004|**DONE (2025-10-16)** Attachment + mitigation references now land as expected and SemVer primitives carry exact values; end-to-end suite green (see `HANDOVER.md`).|
|FEEDCONN-ICSCISA-02-011 Docs & telemetry refresh|DevEx|02-006|**DONE (2025-10-16)** Ops documentation refreshed (attachments, SemVer validation, proxy knobs) and telemetry notes verified.|
|FEEDCONN-ICSCISA-02-012 Normalized version decision|BE-Conn-ICS-CISA|Merge coordination (`FEEDMERGE-COORD-02-900`)|**TODO (due 2025-10-23)** Promote existing `SemVerPrimitive` exact values into `NormalizedVersions` via `.ToNormalizedVersionRule("ics-cisa:{advisoryId}:{product}")`, add regression coverage, and open Models ticket if non-SemVer firmware requires a new scheme.|