Restructure solution layout by module
Some checks failed
Docs CI / lint-and-preview (push) Has been cancelled
Some checks failed
Docs CI / lint-and-preview (push) Has been cancelled
This commit is contained in:
@@ -0,0 +1,39 @@
|
||||
# AGENTS
|
||||
## Role
|
||||
Implement the CISA ICS advisory connector to ingest US CISA Industrial Control Systems advisories (distinct from the general CERT feed).
|
||||
|
||||
## Scope
|
||||
- Locate the official CISA ICS advisory feed/API (currently HTML/RSS) and define fetch cadence/windowing.
|
||||
- Build fetch/cursor pipeline with retry/backoff and raw document storage.
|
||||
- Parse advisory content for summary, impacted vendors/products, mitigation, CVEs.
|
||||
- Map advisories into canonical `Advisory` records with aliases, references, affected ICS packages, and range primitives.
|
||||
- Provide deterministic fixtures and automated regression tests.
|
||||
|
||||
## Participants
|
||||
- `Source.Common` (HTTP/fetch utilities, DTO storage).
|
||||
- `Storage.Mongo` (raw/document/DTO/advisory stores + source state).
|
||||
- `Concelier.Models` (canonical advisory structures).
|
||||
- `Concelier.Testing` (integration fixtures and snapshots).
|
||||
|
||||
## Interfaces & Contracts
|
||||
- Job kinds: `ics-cisa:fetch`, `ics-cisa:parse`, `ics-cisa:map`.
|
||||
- Persist upstream caching metadata (ETag/Last-Modified) when available.
|
||||
- Alias set should include CISA ICS advisory IDs and referenced CVE IDs.
|
||||
|
||||
## In/Out of scope
|
||||
In scope:
|
||||
- ICS-specific advisories from CISA.
|
||||
- Range primitives capturing vendor/equipment metadata.
|
||||
|
||||
Out of scope:
|
||||
- General CISA alerts (covered elsewhere).
|
||||
|
||||
## Observability & Security Expectations
|
||||
- Log fetch attempts, advisory counts, and mapping results.
|
||||
- Sanitize HTML, removing scripts/styles before persistence.
|
||||
- Honour upstream rate limits with exponential backoff.
|
||||
|
||||
## Tests
|
||||
- Add `StellaOps.Concelier.Connector.Ics.Cisa.Tests` to cover fetch/parse/map with canned fixtures.
|
||||
- Snapshot canonical advisories; support fixture regeneration via env flag.
|
||||
- Ensure deterministic ordering/time normalisation.
|
||||
@@ -0,0 +1,182 @@
|
||||
using System;
|
||||
using System.Collections.Generic;
|
||||
using System.Diagnostics.CodeAnalysis;
|
||||
using System.Linq;
|
||||
using System.Net;
|
||||
using System.Net.Http;
|
||||
|
||||
namespace StellaOps.Concelier.Connector.Ics.Cisa.Configuration;
|
||||
|
||||
public sealed class IcsCisaOptions
|
||||
{
|
||||
public static string HttpClientName => "source.ics.cisa";
|
||||
|
||||
/// <summary>
|
||||
/// GovDelivery topics RSS endpoint. Feed URIs are constructed from this base.
|
||||
/// </summary>
|
||||
public Uri TopicsEndpoint { get; set; } = new("https://content.govdelivery.com/accounts/USDHSCISA/topics.rss", UriKind.Absolute);
|
||||
|
||||
/// <summary>
|
||||
/// GovDelivery personalised subscription code (code=...).
|
||||
/// </summary>
|
||||
public string GovDeliveryCode { get; set; } = string.Empty;
|
||||
|
||||
/// <summary>
|
||||
/// Topic identifiers to pull (e.g. USDHSCISA_16 for general ICS advisories).
|
||||
/// </summary>
|
||||
public IList<string> TopicIds { get; } = new List<string>
|
||||
{
|
||||
"USDHSCISA_16", // ICS advisories (ICSA)
|
||||
"USDHSCISA_19", // ICS medical advisories (ICSMA)
|
||||
"USDHSCISA_17", // ICS alerts
|
||||
};
|
||||
|
||||
/// <summary>
|
||||
/// Optional delay between sequential topic fetches to appease GovDelivery throttling.
|
||||
/// </summary>
|
||||
public TimeSpan RequestDelay { get; set; } = TimeSpan.FromMilliseconds(250);
|
||||
|
||||
public TimeSpan FailureBackoff { get; set; } = TimeSpan.FromMinutes(5);
|
||||
|
||||
public TimeSpan DocumentExpiry { get; set; } = TimeSpan.FromDays(30);
|
||||
|
||||
/// <summary>
|
||||
/// Optional proxy endpoint used when Akamai blocks direct pulls.
|
||||
/// </summary>
|
||||
public Uri? ProxyUri { get; set; }
|
||||
|
||||
/// <summary>
|
||||
/// HTTP version requested when contacting GovDelivery.
|
||||
/// </summary>
|
||||
public Version RequestVersion { get; set; } = HttpVersion.Version11;
|
||||
|
||||
/// <summary>
|
||||
/// Negotiation policy applied to HTTP requests.
|
||||
/// </summary>
|
||||
public HttpVersionPolicy RequestVersionPolicy { get; set; } = HttpVersionPolicy.RequestVersionOrLower;
|
||||
|
||||
/// <summary>
|
||||
/// Maximum number of retry attempts for RSS fetches.
|
||||
/// </summary>
|
||||
public int MaxAttempts { get; set; } = 4;
|
||||
|
||||
/// <summary>
|
||||
/// Base delay used for exponential backoff between attempts.
|
||||
/// </summary>
|
||||
public TimeSpan BaseDelay { get; set; } = TimeSpan.FromSeconds(3);
|
||||
|
||||
/// <summary>
|
||||
/// Base URI used when fetching HTML detail pages.
|
||||
/// </summary>
|
||||
public Uri DetailBaseUri { get; set; } = new("https://www.cisa.gov/", UriKind.Absolute);
|
||||
|
||||
/// <summary>
|
||||
/// Optional timeout override applied to detail page fetches.
|
||||
/// </summary>
|
||||
public TimeSpan DetailRequestTimeout { get; set; } = TimeSpan.FromSeconds(25);
|
||||
|
||||
/// <summary>
|
||||
/// Additional hosts allowed by the connector (detail pages, attachments).
|
||||
/// </summary>
|
||||
public IList<string> AdditionalHosts { get; } = new List<string>
|
||||
{
|
||||
"www.cisa.gov",
|
||||
"cisa.gov"
|
||||
};
|
||||
|
||||
public bool EnableDetailScrape { get; set; } = true;
|
||||
|
||||
public bool CaptureAttachments { get; set; } = true;
|
||||
|
||||
[MemberNotNull(nameof(TopicsEndpoint), nameof(GovDeliveryCode))]
|
||||
public void Validate()
|
||||
{
|
||||
if (TopicsEndpoint is null || !TopicsEndpoint.IsAbsoluteUri)
|
||||
{
|
||||
throw new InvalidOperationException("TopicsEndpoint must be an absolute URI.");
|
||||
}
|
||||
|
||||
if (string.IsNullOrWhiteSpace(GovDeliveryCode))
|
||||
{
|
||||
throw new InvalidOperationException("GovDeliveryCode must be provided.");
|
||||
}
|
||||
|
||||
if (TopicIds.Count == 0)
|
||||
{
|
||||
throw new InvalidOperationException("At least one GovDelivery topic identifier is required.");
|
||||
}
|
||||
|
||||
foreach (var topic in TopicIds)
|
||||
{
|
||||
if (string.IsNullOrWhiteSpace(topic))
|
||||
{
|
||||
throw new InvalidOperationException("Topic identifiers cannot be blank.");
|
||||
}
|
||||
}
|
||||
|
||||
if (RequestDelay < TimeSpan.Zero)
|
||||
{
|
||||
throw new InvalidOperationException("RequestDelay cannot be negative.");
|
||||
}
|
||||
|
||||
if (FailureBackoff <= TimeSpan.Zero)
|
||||
{
|
||||
throw new InvalidOperationException("FailureBackoff must be greater than zero.");
|
||||
}
|
||||
|
||||
if (DocumentExpiry <= TimeSpan.Zero)
|
||||
{
|
||||
throw new InvalidOperationException("DocumentExpiry must be greater than zero.");
|
||||
}
|
||||
|
||||
if (MaxAttempts <= 0)
|
||||
{
|
||||
throw new InvalidOperationException("MaxAttempts must be positive.");
|
||||
}
|
||||
|
||||
if (BaseDelay <= TimeSpan.Zero)
|
||||
{
|
||||
throw new InvalidOperationException("BaseDelay must be greater than zero.");
|
||||
}
|
||||
|
||||
if (DetailBaseUri is null || !DetailBaseUri.IsAbsoluteUri)
|
||||
{
|
||||
throw new InvalidOperationException("DetailBaseUri must be an absolute URI.");
|
||||
}
|
||||
|
||||
if (DetailRequestTimeout <= TimeSpan.Zero)
|
||||
{
|
||||
throw new InvalidOperationException("DetailRequestTimeout must be greater than zero.");
|
||||
}
|
||||
|
||||
if (ProxyUri is not null && !ProxyUri.IsAbsoluteUri)
|
||||
{
|
||||
throw new InvalidOperationException("ProxyUri must be an absolute URI when specified.");
|
||||
}
|
||||
|
||||
foreach (var host in AdditionalHosts)
|
||||
{
|
||||
if (string.IsNullOrWhiteSpace(host))
|
||||
{
|
||||
throw new InvalidOperationException("Additional host entries cannot be blank.");
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
public Uri BuildTopicUri(string topicId)
|
||||
{
|
||||
ArgumentException.ThrowIfNullOrEmpty(topicId);
|
||||
Validate();
|
||||
|
||||
var builder = new UriBuilder(TopicsEndpoint);
|
||||
var query = new Dictionary<string, string>(StringComparer.OrdinalIgnoreCase)
|
||||
{
|
||||
["code"] = GovDeliveryCode,
|
||||
["format"] = "xml",
|
||||
["topic_id"] = topicId.Trim(),
|
||||
};
|
||||
|
||||
builder.Query = string.Join("&", query.Select(pair => $"{Uri.EscapeDataString(pair.Key)}={Uri.EscapeDataString(pair.Value)}"));
|
||||
return builder.Uri;
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,21 @@
|
||||
# ICS CISA Connector – Status (2025-10-16)
|
||||
|
||||
## Context
|
||||
- Proxy plumbing for GovDelivery (`SourceHttpClientOptions.Proxy*`) is implemented and covered by `SourceHttpClientBuilderTests.AddSourceHttpClient_LoadsProxyConfiguration`.
|
||||
- Detail enrichment now extracts mitigation paragraphs/bullets, merges them with feed data, and emits `mitigation` references plus combined alias sets.
|
||||
- `BuildAffectedPackages` parses product/version pairs and now persists SemVer exact values for canonical ranges via the advisory store.
|
||||
|
||||
## Current Outcomes
|
||||
- Feed parser fixtures were refreshed so vendor PDFs stay surfaced as attachments; DTO references continue including canonical links.
|
||||
- SemVer primitive deserialisation now restores `exactValue` (e.g., `"4.2"` → `"4.2.0"`), keeping connector snapshots deterministic.
|
||||
- Console debugging noise was removed from connector/parser code.
|
||||
- Ops runbook documents attachment + SemVer validation steps for dry runs.
|
||||
- `dotnet test src/Concelier/__Tests/StellaOps.Concelier.Connector.Ics.Cisa.Tests/StellaOps.Concelier.Connector.Ics.Cisa.Tests.csproj` passes (2025-10-16).
|
||||
|
||||
## Outstanding Items
|
||||
- None. Continue monitoring Akamai access decisions and proxy requirements via Ops feedback.
|
||||
|
||||
## Verification Checklist
|
||||
- ✅ `dotnet test src/Concelier/__Tests/StellaOps.Concelier.Connector.Ics.Cisa.Tests/StellaOps.Concelier.Connector.Ics.Cisa.Tests.csproj`
|
||||
- ☐ `dotnet test src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.Common.Tests/StellaOps.Concelier.Connector.Common.Tests.csproj` (proxy support) — rerun when Source.Common changes land.
|
||||
- Keep this summary aligned with `TASKS.md` as further work emerges.
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,19 @@
|
||||
using Microsoft.Extensions.DependencyInjection;
|
||||
using StellaOps.Plugin;
|
||||
|
||||
namespace StellaOps.Concelier.Connector.Ics.Cisa;
|
||||
|
||||
public sealed class IcsCisaConnectorPlugin : IConnectorPlugin
|
||||
{
|
||||
public const string SourceName = "ics-cisa";
|
||||
|
||||
public string Name => SourceName;
|
||||
|
||||
public bool IsAvailable(IServiceProvider services) => services is not null;
|
||||
|
||||
public IFeedConnector Create(IServiceProvider services)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(services);
|
||||
return ActivatorUtilities.CreateInstance<IcsCisaConnector>(services);
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,56 @@
|
||||
using System;
|
||||
using Microsoft.Extensions.Configuration;
|
||||
using Microsoft.Extensions.DependencyInjection;
|
||||
using StellaOps.DependencyInjection;
|
||||
using StellaOps.Concelier.Core.Jobs;
|
||||
using StellaOps.Concelier.Connector.Ics.Cisa.Configuration;
|
||||
|
||||
namespace StellaOps.Concelier.Connector.Ics.Cisa;
|
||||
|
||||
public sealed class IcsCisaDependencyInjectionRoutine : IDependencyInjectionRoutine
|
||||
{
|
||||
private const string ConfigurationSection = "concelier:sources:ics-cisa";
|
||||
|
||||
public IServiceCollection Register(IServiceCollection services, IConfiguration configuration)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(services);
|
||||
ArgumentNullException.ThrowIfNull(configuration);
|
||||
|
||||
services.AddIcsCisaConnector(options =>
|
||||
{
|
||||
configuration.GetSection(ConfigurationSection).Bind(options);
|
||||
options.Validate();
|
||||
});
|
||||
|
||||
services.AddTransient<IcsCisaFetchJob>();
|
||||
services.AddTransient<IcsCisaParseJob>();
|
||||
services.AddTransient<IcsCisaMapJob>();
|
||||
|
||||
services.PostConfigure<JobSchedulerOptions>(options =>
|
||||
{
|
||||
EnsureJob(options, IcsCisaJobKinds.Fetch, typeof(IcsCisaFetchJob));
|
||||
EnsureJob(options, IcsCisaJobKinds.Parse, typeof(IcsCisaParseJob));
|
||||
EnsureJob(options, IcsCisaJobKinds.Map, typeof(IcsCisaMapJob));
|
||||
});
|
||||
|
||||
return services;
|
||||
}
|
||||
|
||||
private static void EnsureJob(JobSchedulerOptions options, string kind, Type jobType)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(options);
|
||||
|
||||
if (options.Definitions.ContainsKey(kind))
|
||||
{
|
||||
return;
|
||||
}
|
||||
|
||||
options.Definitions[kind] = new JobDefinition(
|
||||
kind,
|
||||
jobType,
|
||||
options.DefaultTimeout,
|
||||
options.DefaultLeaseDuration,
|
||||
CronExpression: null,
|
||||
Enabled: true);
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,60 @@
|
||||
using System;
|
||||
using System.Net;
|
||||
using Microsoft.Extensions.DependencyInjection;
|
||||
using Microsoft.Extensions.Options;
|
||||
using StellaOps.Concelier.Connector.Common.Http;
|
||||
using StellaOps.Concelier.Connector.Ics.Cisa.Configuration;
|
||||
using StellaOps.Concelier.Connector.Ics.Cisa.Internal;
|
||||
|
||||
namespace StellaOps.Concelier.Connector.Ics.Cisa;
|
||||
|
||||
public static class IcsCisaServiceCollectionExtensions
|
||||
{
|
||||
public static IServiceCollection AddIcsCisaConnector(this IServiceCollection services, Action<IcsCisaOptions> configure)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(services);
|
||||
ArgumentNullException.ThrowIfNull(configure);
|
||||
|
||||
services.AddOptions<IcsCisaOptions>()
|
||||
.Configure(configure)
|
||||
.PostConfigure(static opts => opts.Validate());
|
||||
|
||||
services.AddSourceHttpClient(IcsCisaOptions.HttpClientName, (sp, clientOptions) =>
|
||||
{
|
||||
var options = sp.GetRequiredService<IOptions<IcsCisaOptions>>().Value;
|
||||
clientOptions.BaseAddress = new Uri(options.TopicsEndpoint.GetLeftPart(UriPartial.Authority));
|
||||
clientOptions.Timeout = TimeSpan.FromSeconds(45);
|
||||
clientOptions.UserAgent = "StellaOps.Concelier.IcsCisa/1.0";
|
||||
clientOptions.AllowedHosts.Clear();
|
||||
clientOptions.AllowedHosts.Add(options.TopicsEndpoint.Host);
|
||||
clientOptions.AllowedHosts.Add(options.DetailBaseUri.Host);
|
||||
foreach (var host in options.AdditionalHosts)
|
||||
{
|
||||
clientOptions.AllowedHosts.Add(host);
|
||||
}
|
||||
clientOptions.DefaultRequestHeaders["Accept"] = "application/rss+xml";
|
||||
clientOptions.RequestVersion = options.RequestVersion;
|
||||
clientOptions.VersionPolicy = options.RequestVersionPolicy;
|
||||
clientOptions.MaxAttempts = options.MaxAttempts;
|
||||
clientOptions.BaseDelay = options.BaseDelay;
|
||||
clientOptions.EnableMultipleHttp2Connections = false;
|
||||
|
||||
clientOptions.ConfigureHandler = handler =>
|
||||
{
|
||||
handler.AutomaticDecompression = DecompressionMethods.All;
|
||||
};
|
||||
|
||||
if (options.ProxyUri is not null)
|
||||
{
|
||||
clientOptions.ProxyAddress = options.ProxyUri;
|
||||
clientOptions.ProxyBypassOnLocal = false;
|
||||
}
|
||||
});
|
||||
|
||||
services.AddSingleton<IcsCisaDiagnostics>();
|
||||
services.AddSingleton<IcsCisaFeedParser>();
|
||||
services.AddTransient<IcsCisaConnector>();
|
||||
|
||||
return services;
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,56 @@
|
||||
using System;
|
||||
using System.Collections.Generic;
|
||||
using System.Text.Json.Serialization;
|
||||
|
||||
namespace StellaOps.Concelier.Connector.Ics.Cisa.Internal;
|
||||
|
||||
public sealed record IcsCisaAdvisoryDto
|
||||
{
|
||||
[JsonPropertyName("advisoryId")]
|
||||
public required string AdvisoryId { get; init; }
|
||||
|
||||
[JsonPropertyName("title")]
|
||||
public required string Title { get; init; }
|
||||
|
||||
[JsonPropertyName("link")]
|
||||
public required string Link { get; init; }
|
||||
|
||||
[JsonPropertyName("summary")]
|
||||
public string? Summary { get; init; }
|
||||
|
||||
[JsonPropertyName("descriptionHtml")]
|
||||
public string DescriptionHtml { get; init; } = string.Empty;
|
||||
|
||||
[JsonPropertyName("published")]
|
||||
public DateTimeOffset Published { get; init; }
|
||||
|
||||
[JsonPropertyName("updated")]
|
||||
public DateTimeOffset? Updated { get; init; }
|
||||
|
||||
[JsonPropertyName("medical")]
|
||||
public bool IsMedical { get; init; }
|
||||
|
||||
[JsonPropertyName("aliases")]
|
||||
public IReadOnlyCollection<string> Aliases { get; init; } = Array.Empty<string>();
|
||||
|
||||
[JsonPropertyName("cveIds")]
|
||||
public IReadOnlyCollection<string> CveIds { get; init; } = Array.Empty<string>();
|
||||
|
||||
[JsonPropertyName("vendors")]
|
||||
public IReadOnlyCollection<string> Vendors { get; init; } = Array.Empty<string>();
|
||||
|
||||
[JsonPropertyName("products")]
|
||||
public IReadOnlyCollection<string> Products { get; init; } = Array.Empty<string>();
|
||||
|
||||
[JsonPropertyName("references")]
|
||||
public IReadOnlyCollection<string> References { get; init; } = Array.Empty<string>();
|
||||
|
||||
[JsonPropertyName("attachments")]
|
||||
public IReadOnlyCollection<IcsCisaAttachmentDto> Attachments { get; init; } = Array.Empty<IcsCisaAttachmentDto>();
|
||||
|
||||
[JsonPropertyName("mitigations")]
|
||||
public IReadOnlyCollection<string> Mitigations { get; init; } = Array.Empty<string>();
|
||||
|
||||
[JsonPropertyName("detailHtml")]
|
||||
public string? DetailHtml { get; init; }
|
||||
}
|
||||
@@ -0,0 +1,12 @@
|
||||
using System.Text.Json.Serialization;
|
||||
|
||||
namespace StellaOps.Concelier.Connector.Ics.Cisa.Internal;
|
||||
|
||||
public sealed record IcsCisaAttachmentDto
|
||||
{
|
||||
[JsonPropertyName("title")]
|
||||
public string? Title { get; init; }
|
||||
|
||||
[JsonPropertyName("url")]
|
||||
public required string Url { get; init; }
|
||||
}
|
||||
@@ -0,0 +1,88 @@
|
||||
using System;
|
||||
using System.Collections.Generic;
|
||||
using System.Linq;
|
||||
using MongoDB.Bson;
|
||||
|
||||
namespace StellaOps.Concelier.Connector.Ics.Cisa.Internal;
|
||||
|
||||
internal sealed record IcsCisaCursor(
|
||||
DateTimeOffset? LastPublished,
|
||||
IReadOnlyCollection<Guid> PendingDocuments,
|
||||
IReadOnlyCollection<Guid> PendingMappings)
|
||||
{
|
||||
public static IcsCisaCursor Empty { get; } = new(null, Array.Empty<Guid>(), Array.Empty<Guid>());
|
||||
|
||||
public BsonDocument ToBsonDocument()
|
||||
{
|
||||
var document = new BsonDocument
|
||||
{
|
||||
["pendingDocuments"] = new BsonArray(PendingDocuments.Select(static id => id.ToString())),
|
||||
["pendingMappings"] = new BsonArray(PendingMappings.Select(static id => id.ToString())),
|
||||
};
|
||||
|
||||
if (LastPublished.HasValue)
|
||||
{
|
||||
document["lastPublished"] = LastPublished.Value.UtcDateTime;
|
||||
}
|
||||
|
||||
return document;
|
||||
}
|
||||
|
||||
public static IcsCisaCursor FromBson(BsonDocument? document)
|
||||
{
|
||||
if (document is null || document.ElementCount == 0)
|
||||
{
|
||||
return Empty;
|
||||
}
|
||||
|
||||
var lastPublished = document.TryGetValue("lastPublished", out var publishedValue)
|
||||
? ParseDate(publishedValue)
|
||||
: null;
|
||||
|
||||
return new IcsCisaCursor(
|
||||
lastPublished,
|
||||
ReadGuidArray(document, "pendingDocuments"),
|
||||
ReadGuidArray(document, "pendingMappings"));
|
||||
}
|
||||
|
||||
public IcsCisaCursor WithPendingDocuments(IEnumerable<Guid> ids)
|
||||
=> this with { PendingDocuments = ids?.Distinct().ToArray() ?? Array.Empty<Guid>() };
|
||||
|
||||
public IcsCisaCursor WithPendingMappings(IEnumerable<Guid> ids)
|
||||
=> this with { PendingMappings = ids?.Distinct().ToArray() ?? Array.Empty<Guid>() };
|
||||
|
||||
public IcsCisaCursor WithLastPublished(DateTimeOffset? published)
|
||||
=> this with { LastPublished = published?.ToUniversalTime() };
|
||||
|
||||
private static DateTimeOffset? ParseDate(BsonValue value)
|
||||
=> value.BsonType switch
|
||||
{
|
||||
BsonType.DateTime => DateTime.SpecifyKind(value.ToUniversalTime(), DateTimeKind.Utc),
|
||||
BsonType.String when DateTimeOffset.TryParse(value.AsString, out var parsed) => parsed.ToUniversalTime(),
|
||||
_ => null,
|
||||
};
|
||||
|
||||
private static IReadOnlyCollection<Guid> ReadGuidArray(BsonDocument document, string field)
|
||||
{
|
||||
if (!document.TryGetValue(field, out var value) || value is not BsonArray array)
|
||||
{
|
||||
return Array.Empty<Guid>();
|
||||
}
|
||||
|
||||
var results = new List<Guid>(array.Count);
|
||||
foreach (var element in array)
|
||||
{
|
||||
if (element is null)
|
||||
{
|
||||
continue;
|
||||
}
|
||||
|
||||
if (Guid.TryParse(element.ToString(), out var guid))
|
||||
{
|
||||
results.Add(guid);
|
||||
}
|
||||
}
|
||||
|
||||
return results;
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,171 @@
|
||||
using System;
|
||||
using System.Collections.Generic;
|
||||
using System.Diagnostics.Metrics;
|
||||
|
||||
namespace StellaOps.Concelier.Connector.Ics.Cisa.Internal;
|
||||
|
||||
public sealed class IcsCisaDiagnostics : IDisposable
|
||||
{
|
||||
private const string MeterName = "StellaOps.Concelier.Connector.Ics.Cisa";
|
||||
private const string MeterVersion = "1.0.0";
|
||||
|
||||
private readonly Meter _meter;
|
||||
|
||||
private readonly Counter<long> _fetchAttempts;
|
||||
private readonly Counter<long> _fetchSuccess;
|
||||
private readonly Counter<long> _fetchFailures;
|
||||
private readonly Counter<long> _fetchNotModified;
|
||||
private readonly Counter<long> _fetchFallbacks;
|
||||
private readonly Histogram<long> _fetchDocuments;
|
||||
|
||||
private readonly Counter<long> _parseSuccess;
|
||||
private readonly Counter<long> _parseFailures;
|
||||
private readonly Histogram<long> _parseAdvisoryCount;
|
||||
private readonly Histogram<long> _parseAttachmentCount;
|
||||
private readonly Histogram<long> _parseDetailCount;
|
||||
|
||||
private readonly Counter<long> _detailSuccess;
|
||||
private readonly Counter<long> _detailFailures;
|
||||
|
||||
private readonly Counter<long> _mapSuccess;
|
||||
private readonly Counter<long> _mapFailures;
|
||||
private readonly Histogram<long> _mapReferenceCount;
|
||||
private readonly Histogram<long> _mapPackageCount;
|
||||
private readonly Histogram<long> _mapAliasCount;
|
||||
|
||||
public IcsCisaDiagnostics()
|
||||
{
|
||||
_meter = new Meter(MeterName, MeterVersion);
|
||||
|
||||
_fetchAttempts = _meter.CreateCounter<long>("icscisa.fetch.attempts", unit: "operations");
|
||||
_fetchSuccess = _meter.CreateCounter<long>("icscisa.fetch.success", unit: "operations");
|
||||
_fetchFailures = _meter.CreateCounter<long>("icscisa.fetch.failures", unit: "operations");
|
||||
_fetchNotModified = _meter.CreateCounter<long>("icscisa.fetch.not_modified", unit: "operations");
|
||||
_fetchFallbacks = _meter.CreateCounter<long>("icscisa.fetch.fallbacks", unit: "operations");
|
||||
_fetchDocuments = _meter.CreateHistogram<long>("icscisa.fetch.documents", unit: "documents");
|
||||
|
||||
_parseSuccess = _meter.CreateCounter<long>("icscisa.parse.success", unit: "documents");
|
||||
_parseFailures = _meter.CreateCounter<long>("icscisa.parse.failures", unit: "documents");
|
||||
_parseAdvisoryCount = _meter.CreateHistogram<long>("icscisa.parse.advisories", unit: "advisories");
|
||||
_parseAttachmentCount = _meter.CreateHistogram<long>("icscisa.parse.attachments", unit: "attachments");
|
||||
_parseDetailCount = _meter.CreateHistogram<long>("icscisa.parse.detail_fetches", unit: "fetches");
|
||||
|
||||
_detailSuccess = _meter.CreateCounter<long>("icscisa.detail.success", unit: "operations");
|
||||
_detailFailures = _meter.CreateCounter<long>("icscisa.detail.failures", unit: "operations");
|
||||
|
||||
_mapSuccess = _meter.CreateCounter<long>("icscisa.map.success", unit: "advisories");
|
||||
_mapFailures = _meter.CreateCounter<long>("icscisa.map.failures", unit: "advisories");
|
||||
_mapReferenceCount = _meter.CreateHistogram<long>("icscisa.map.references", unit: "references");
|
||||
_mapPackageCount = _meter.CreateHistogram<long>("icscisa.map.packages", unit: "packages");
|
||||
_mapAliasCount = _meter.CreateHistogram<long>("icscisa.map.aliases", unit: "aliases");
|
||||
}
|
||||
|
||||
public void FetchAttempt(string topicId)
|
||||
{
|
||||
_fetchAttempts.Add(1, BuildTopicTags(topicId));
|
||||
}
|
||||
|
||||
public void FetchSuccess(string topicId, int documentsAdded)
|
||||
{
|
||||
var tags = BuildTopicTags(topicId);
|
||||
_fetchSuccess.Add(1, tags);
|
||||
if (documentsAdded > 0)
|
||||
{
|
||||
_fetchDocuments.Record(documentsAdded, tags);
|
||||
}
|
||||
}
|
||||
|
||||
public void FetchNotModified(string topicId)
|
||||
{
|
||||
_fetchNotModified.Add(1, BuildTopicTags(topicId));
|
||||
}
|
||||
|
||||
public void FetchFallback(string topicId)
|
||||
{
|
||||
_fetchFallbacks.Add(1, BuildTopicTags(topicId));
|
||||
}
|
||||
|
||||
public void FetchFailure(string topicId)
|
||||
{
|
||||
_fetchFailures.Add(1, BuildTopicTags(topicId));
|
||||
}
|
||||
|
||||
public void ParseSuccess(string topicId, int advisoryCount, int attachmentCount, int detailFetchCount)
|
||||
{
|
||||
var tags = BuildTopicTags(topicId);
|
||||
_parseSuccess.Add(1, tags);
|
||||
if (advisoryCount >= 0)
|
||||
{
|
||||
_parseAdvisoryCount.Record(advisoryCount, tags);
|
||||
}
|
||||
|
||||
if (attachmentCount >= 0)
|
||||
{
|
||||
_parseAttachmentCount.Record(attachmentCount, tags);
|
||||
}
|
||||
|
||||
if (detailFetchCount >= 0)
|
||||
{
|
||||
_parseDetailCount.Record(detailFetchCount, tags);
|
||||
}
|
||||
}
|
||||
|
||||
public void ParseFailure(string topicId)
|
||||
{
|
||||
_parseFailures.Add(1, BuildTopicTags(topicId));
|
||||
}
|
||||
|
||||
public void DetailFetchSuccess(string advisoryId)
|
||||
{
|
||||
_detailSuccess.Add(1, BuildAdvisoryTags(advisoryId));
|
||||
}
|
||||
|
||||
public void DetailFetchFailure(string advisoryId)
|
||||
{
|
||||
_detailFailures.Add(1, BuildAdvisoryTags(advisoryId));
|
||||
}
|
||||
|
||||
public void MapSuccess(string advisoryId, int referenceCount, int packageCount, int aliasCount)
|
||||
{
|
||||
var tags = BuildAdvisoryTags(advisoryId);
|
||||
_mapSuccess.Add(1, tags);
|
||||
if (referenceCount >= 0)
|
||||
{
|
||||
_mapReferenceCount.Record(referenceCount, tags);
|
||||
}
|
||||
|
||||
if (packageCount >= 0)
|
||||
{
|
||||
_mapPackageCount.Record(packageCount, tags);
|
||||
}
|
||||
|
||||
if (aliasCount >= 0)
|
||||
{
|
||||
_mapAliasCount.Record(aliasCount, tags);
|
||||
}
|
||||
}
|
||||
|
||||
public void MapFailure(string advisoryId)
|
||||
{
|
||||
_mapFailures.Add(1, BuildAdvisoryTags(advisoryId));
|
||||
}
|
||||
|
||||
private static KeyValuePair<string, object?>[] BuildTopicTags(string? topicId)
|
||||
=> new[]
|
||||
{
|
||||
new KeyValuePair<string, object?>("concelier.source", IcsCisaConnectorPlugin.SourceName),
|
||||
new KeyValuePair<string, object?>("icscisa.topic", string.IsNullOrWhiteSpace(topicId) ? "unknown" : topicId)
|
||||
};
|
||||
|
||||
private static KeyValuePair<string, object?>[] BuildAdvisoryTags(string? advisoryId)
|
||||
=> new[]
|
||||
{
|
||||
new KeyValuePair<string, object?>("concelier.source", IcsCisaConnectorPlugin.SourceName),
|
||||
new KeyValuePair<string, object?>("icscisa.advisory", string.IsNullOrWhiteSpace(advisoryId) ? "unknown" : advisoryId)
|
||||
};
|
||||
|
||||
public void Dispose()
|
||||
{
|
||||
_meter.Dispose();
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,16 @@
|
||||
using System.Collections.Generic;
|
||||
using System.Text.Json.Serialization;
|
||||
|
||||
namespace StellaOps.Concelier.Connector.Ics.Cisa.Internal;
|
||||
|
||||
public sealed record IcsCisaFeedDto
|
||||
{
|
||||
[JsonPropertyName("topicId")]
|
||||
public required string TopicId { get; init; }
|
||||
|
||||
[JsonPropertyName("feedUri")]
|
||||
public required string FeedUri { get; init; }
|
||||
|
||||
[JsonPropertyName("advisories")]
|
||||
public IReadOnlyCollection<IcsCisaAdvisoryDto> Advisories { get; init; } = new List<IcsCisaAdvisoryDto>();
|
||||
}
|
||||
@@ -0,0 +1,402 @@
|
||||
using System;
|
||||
using System.Collections.Generic;
|
||||
using System.IO;
|
||||
using System.Linq;
|
||||
using System.ServiceModel.Syndication;
|
||||
using System.Text;
|
||||
using System.Text.RegularExpressions;
|
||||
using System.Xml;
|
||||
using AngleSharp.Html.Parser;
|
||||
using AngleSharp.Html.Dom;
|
||||
using StellaOps.Concelier.Connector.Common.Html;
|
||||
|
||||
namespace StellaOps.Concelier.Connector.Ics.Cisa.Internal;
|
||||
|
||||
public sealed class IcsCisaFeedParser
|
||||
{
|
||||
private static readonly Regex AdvisoryIdRegex = new(@"^(?<id>ICS[AM]?A?-?\d{2}-\d{3}[A-Z]?(-\d{2})?)", RegexOptions.IgnoreCase | RegexOptions.Compiled);
|
||||
private static readonly Regex CveRegex = new(@"CVE-\d{4}-\d{4,}", RegexOptions.IgnoreCase | RegexOptions.Compiled);
|
||||
|
||||
private readonly HtmlContentSanitizer _sanitizer = new();
|
||||
private readonly HtmlParser _htmlParser = new();
|
||||
|
||||
public IReadOnlyCollection<IcsCisaAdvisoryDto> Parse(Stream rssStream, bool isMedicalTopic, Uri? topicUri)
|
||||
{
|
||||
if (rssStream is null)
|
||||
{
|
||||
return Array.Empty<IcsCisaAdvisoryDto>();
|
||||
}
|
||||
|
||||
using var reader = XmlReader.Create(rssStream, new XmlReaderSettings
|
||||
{
|
||||
DtdProcessing = DtdProcessing.Ignore,
|
||||
IgnoreComments = true,
|
||||
IgnoreProcessingInstructions = true,
|
||||
});
|
||||
|
||||
var feed = SyndicationFeed.Load(reader);
|
||||
if (feed is null || feed.Items is null)
|
||||
{
|
||||
return Array.Empty<IcsCisaAdvisoryDto>();
|
||||
}
|
||||
|
||||
var advisories = new List<IcsCisaAdvisoryDto>();
|
||||
foreach (var item in feed.Items)
|
||||
{
|
||||
var dto = ConvertItem(item, isMedicalTopic, topicUri);
|
||||
if (dto is not null)
|
||||
{
|
||||
advisories.Add(dto);
|
||||
}
|
||||
}
|
||||
|
||||
return advisories;
|
||||
}
|
||||
|
||||
private IcsCisaAdvisoryDto? ConvertItem(SyndicationItem item, bool isMedicalTopic, Uri? topicUri)
|
||||
{
|
||||
if (item is null)
|
||||
{
|
||||
return null;
|
||||
}
|
||||
|
||||
var title = item.Title?.Text?.Trim();
|
||||
if (string.IsNullOrWhiteSpace(title))
|
||||
{
|
||||
return null;
|
||||
}
|
||||
|
||||
var advisoryId = ExtractAdvisoryId(title);
|
||||
if (string.IsNullOrWhiteSpace(advisoryId))
|
||||
{
|
||||
return null;
|
||||
}
|
||||
|
||||
var linkUri = item.Links.FirstOrDefault()?.Uri;
|
||||
if (linkUri is null && !string.IsNullOrWhiteSpace(item.Id) && Uri.TryCreate(item.Id, UriKind.Absolute, out var fallback))
|
||||
{
|
||||
linkUri = fallback;
|
||||
}
|
||||
|
||||
if (linkUri is null)
|
||||
{
|
||||
return null;
|
||||
}
|
||||
|
||||
var contentHtml = ExtractContentHtml(item);
|
||||
var sanitizedHtml = _sanitizer.Sanitize(contentHtml, linkUri);
|
||||
var textContent = ExtractTextContent(sanitizedHtml);
|
||||
|
||||
var aliases = new HashSet<string>(StringComparer.OrdinalIgnoreCase) { advisoryId };
|
||||
var cveIds = ExtractCveIds(textContent, aliases);
|
||||
var vendors = ExtractList(sanitizedHtml, textContent, "Vendor");
|
||||
var products = ExtractList(sanitizedHtml, textContent, "Products");
|
||||
if (products.Count == 0)
|
||||
{
|
||||
products = ExtractList(sanitizedHtml, textContent, "Product");
|
||||
}
|
||||
var attachments = ExtractAttachments(sanitizedHtml, linkUri);
|
||||
var references = ExtractReferences(sanitizedHtml, linkUri);
|
||||
|
||||
var published = item.PublishDate != DateTimeOffset.MinValue
|
||||
? item.PublishDate.ToUniversalTime()
|
||||
: item.LastUpdatedTime.ToUniversalTime();
|
||||
|
||||
var updated = item.LastUpdatedTime != DateTimeOffset.MinValue
|
||||
? item.LastUpdatedTime.ToUniversalTime()
|
||||
: (DateTimeOffset?)null;
|
||||
|
||||
return new IcsCisaAdvisoryDto
|
||||
{
|
||||
AdvisoryId = advisoryId,
|
||||
Title = title,
|
||||
Link = linkUri.ToString(),
|
||||
Summary = item.Summary?.Text?.Trim(),
|
||||
DescriptionHtml = sanitizedHtml,
|
||||
Published = published,
|
||||
Updated = updated,
|
||||
IsMedical = isMedicalTopic || advisoryId.StartsWith("ICSMA", StringComparison.OrdinalIgnoreCase),
|
||||
Aliases = aliases.ToArray(),
|
||||
CveIds = cveIds,
|
||||
Vendors = vendors,
|
||||
Products = products,
|
||||
References = references,
|
||||
Attachments = attachments,
|
||||
};
|
||||
}
|
||||
|
||||
private static string ExtractAdvisoryId(string title)
|
||||
{
|
||||
if (string.IsNullOrWhiteSpace(title))
|
||||
{
|
||||
return string.Empty;
|
||||
}
|
||||
|
||||
var colonIndex = title.IndexOf(':');
|
||||
var candidate = colonIndex > 0 ? title[..colonIndex] : title;
|
||||
var match = AdvisoryIdRegex.Match(candidate);
|
||||
if (match.Success)
|
||||
{
|
||||
var id = match.Groups["id"].Value.Trim();
|
||||
return id.ToUpperInvariant();
|
||||
}
|
||||
|
||||
return candidate.Trim();
|
||||
}
|
||||
|
||||
private static string ExtractContentHtml(SyndicationItem item)
|
||||
{
|
||||
if (item.Content is TextSyndicationContent textContent)
|
||||
{
|
||||
return textContent.Text ?? string.Empty;
|
||||
}
|
||||
|
||||
if (item.Summary is not null)
|
||||
{
|
||||
return item.Summary.Text ?? string.Empty;
|
||||
}
|
||||
|
||||
if (item.ElementExtensions is not null)
|
||||
{
|
||||
foreach (var extension in item.ElementExtensions)
|
||||
{
|
||||
try
|
||||
{
|
||||
var value = extension.GetObject<string>();
|
||||
if (!string.IsNullOrWhiteSpace(value))
|
||||
{
|
||||
return value;
|
||||
}
|
||||
}
|
||||
catch
|
||||
{
|
||||
// ignore malformed extensions
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
return string.Empty;
|
||||
}
|
||||
|
||||
private static IReadOnlyCollection<string> ExtractCveIds(string text, HashSet<string> aliases)
|
||||
{
|
||||
if (string.IsNullOrWhiteSpace(text))
|
||||
{
|
||||
return Array.Empty<string>();
|
||||
}
|
||||
|
||||
var matches = CveRegex.Matches(text);
|
||||
if (matches.Count == 0)
|
||||
{
|
||||
return Array.Empty<string>();
|
||||
}
|
||||
|
||||
var ids = new HashSet<string>(StringComparer.OrdinalIgnoreCase);
|
||||
foreach (Match match in matches)
|
||||
{
|
||||
if (!match.Success)
|
||||
{
|
||||
continue;
|
||||
}
|
||||
|
||||
var value = match.Value.ToUpperInvariant();
|
||||
if (ids.Add(value))
|
||||
{
|
||||
aliases.Add(value);
|
||||
}
|
||||
}
|
||||
|
||||
return ids.ToArray();
|
||||
}
|
||||
|
||||
private IReadOnlyCollection<string> ExtractList(string sanitizedHtml, string textContent, string key)
|
||||
{
|
||||
if (string.IsNullOrWhiteSpace(sanitizedHtml))
|
||||
{
|
||||
return Array.Empty<string>();
|
||||
}
|
||||
|
||||
var items = new HashSet<string>(StringComparer.OrdinalIgnoreCase);
|
||||
|
||||
try
|
||||
{
|
||||
var document = _htmlParser.ParseDocument(sanitizedHtml);
|
||||
foreach (var element in document.All)
|
||||
{
|
||||
if (element is IHtmlParagraphElement or IHtmlDivElement or IHtmlSpanElement or IHtmlListItemElement)
|
||||
{
|
||||
var content = element.TextContent?.Trim();
|
||||
if (string.IsNullOrWhiteSpace(content))
|
||||
{
|
||||
continue;
|
||||
}
|
||||
|
||||
if (content.StartsWith($"{key}:", StringComparison.OrdinalIgnoreCase))
|
||||
{
|
||||
var line = content[(key.Length + 1)..].Trim();
|
||||
foreach (var part in line.Split(new[] { ',', ';' }, StringSplitOptions.RemoveEmptyEntries))
|
||||
{
|
||||
var value = part.Trim();
|
||||
if (!string.IsNullOrWhiteSpace(value))
|
||||
{
|
||||
items.Add(value);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
catch
|
||||
{
|
||||
// ignore HTML parsing failures; fallback to text processing below
|
||||
}
|
||||
|
||||
if (items.Count == 0 && !string.IsNullOrWhiteSpace(textContent))
|
||||
{
|
||||
using var reader = new StringReader(textContent);
|
||||
string? line;
|
||||
while ((line = reader.ReadLine()) is not null)
|
||||
{
|
||||
if (line.StartsWith($"{key}:", StringComparison.OrdinalIgnoreCase))
|
||||
{
|
||||
var raw = line[(key.Length + 1)..].Trim();
|
||||
foreach (var part in raw.Split(new[] { ',', ';' }, StringSplitOptions.RemoveEmptyEntries))
|
||||
{
|
||||
var value = part.Trim();
|
||||
if (!string.IsNullOrWhiteSpace(value))
|
||||
{
|
||||
items.Add(value);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
return items.ToArray();
|
||||
}
|
||||
|
||||
private IReadOnlyCollection<IcsCisaAttachmentDto> ExtractAttachments(string sanitizedHtml, Uri linkUri)
|
||||
{
|
||||
if (string.IsNullOrWhiteSpace(sanitizedHtml))
|
||||
{
|
||||
return Array.Empty<IcsCisaAttachmentDto>();
|
||||
}
|
||||
|
||||
try
|
||||
{
|
||||
var document = _htmlParser.ParseDocument(sanitizedHtml);
|
||||
var attachments = new List<IcsCisaAttachmentDto>();
|
||||
|
||||
foreach (var anchor in document.QuerySelectorAll("a"))
|
||||
{
|
||||
var href = anchor.GetAttribute("href");
|
||||
if (string.IsNullOrWhiteSpace(href))
|
||||
{
|
||||
continue;
|
||||
}
|
||||
|
||||
if (!Uri.TryCreate(linkUri, href, out var resolved))
|
||||
{
|
||||
continue;
|
||||
}
|
||||
|
||||
var url = resolved.ToString();
|
||||
if (!url.EndsWith(".pdf", StringComparison.OrdinalIgnoreCase) &&
|
||||
!url.Contains("/pdf", StringComparison.OrdinalIgnoreCase))
|
||||
{
|
||||
continue;
|
||||
}
|
||||
|
||||
attachments.Add(new IcsCisaAttachmentDto
|
||||
{
|
||||
Title = anchor.TextContent?.Trim(),
|
||||
Url = url,
|
||||
});
|
||||
}
|
||||
|
||||
return attachments.Count == 0 ? Array.Empty<IcsCisaAttachmentDto>() : attachments;
|
||||
}
|
||||
catch
|
||||
{
|
||||
return Array.Empty<IcsCisaAttachmentDto>();
|
||||
}
|
||||
}
|
||||
|
||||
private IReadOnlyCollection<string> ExtractReferences(string sanitizedHtml, Uri linkUri)
|
||||
{
|
||||
if (string.IsNullOrWhiteSpace(sanitizedHtml))
|
||||
{
|
||||
return new[] { linkUri.ToString() };
|
||||
}
|
||||
|
||||
try
|
||||
{
|
||||
var document = _htmlParser.ParseDocument(sanitizedHtml);
|
||||
var links = new HashSet<string>(StringComparer.OrdinalIgnoreCase)
|
||||
{
|
||||
linkUri.ToString()
|
||||
};
|
||||
|
||||
foreach (var anchor in document.QuerySelectorAll("a"))
|
||||
{
|
||||
var href = anchor.GetAttribute("href");
|
||||
if (string.IsNullOrWhiteSpace(href))
|
||||
{
|
||||
continue;
|
||||
}
|
||||
|
||||
if (Uri.TryCreate(linkUri, href, out var resolved))
|
||||
{
|
||||
links.Add(resolved.ToString());
|
||||
}
|
||||
}
|
||||
|
||||
return links.ToArray();
|
||||
}
|
||||
catch
|
||||
{
|
||||
return new[] { linkUri.ToString() };
|
||||
}
|
||||
}
|
||||
|
||||
private string ExtractTextContent(string sanitizedHtml)
|
||||
{
|
||||
if (string.IsNullOrWhiteSpace(sanitizedHtml))
|
||||
{
|
||||
return string.Empty;
|
||||
}
|
||||
|
||||
try
|
||||
{
|
||||
var document = _htmlParser.ParseDocument(sanitizedHtml);
|
||||
var builder = new StringBuilder();
|
||||
var body = document.Body ?? document.DocumentElement;
|
||||
if (body is null)
|
||||
{
|
||||
return string.Empty;
|
||||
}
|
||||
|
||||
foreach (var node in body.ChildNodes)
|
||||
{
|
||||
var text = node.TextContent;
|
||||
if (string.IsNullOrWhiteSpace(text))
|
||||
{
|
||||
continue;
|
||||
}
|
||||
|
||||
if (builder.Length > 0)
|
||||
{
|
||||
builder.AppendLine();
|
||||
}
|
||||
|
||||
builder.Append(text.Trim());
|
||||
}
|
||||
|
||||
return builder.ToString();
|
||||
}
|
||||
catch
|
||||
{
|
||||
return sanitizedHtml;
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,46 @@
|
||||
using System;
|
||||
using System.Threading;
|
||||
using System.Threading.Tasks;
|
||||
using StellaOps.Concelier.Core.Jobs;
|
||||
|
||||
namespace StellaOps.Concelier.Connector.Ics.Cisa;
|
||||
|
||||
internal static class IcsCisaJobKinds
|
||||
{
|
||||
public const string Fetch = "source:ics-cisa:fetch";
|
||||
public const string Parse = "source:ics-cisa:parse";
|
||||
public const string Map = "source:ics-cisa:map";
|
||||
}
|
||||
|
||||
internal sealed class IcsCisaFetchJob : IJob
|
||||
{
|
||||
private readonly IcsCisaConnector _connector;
|
||||
|
||||
public IcsCisaFetchJob(IcsCisaConnector connector)
|
||||
=> _connector = connector ?? throw new ArgumentNullException(nameof(connector));
|
||||
|
||||
public Task ExecuteAsync(JobExecutionContext context, CancellationToken cancellationToken)
|
||||
=> _connector.FetchAsync(context.Services, cancellationToken);
|
||||
}
|
||||
|
||||
internal sealed class IcsCisaParseJob : IJob
|
||||
{
|
||||
private readonly IcsCisaConnector _connector;
|
||||
|
||||
public IcsCisaParseJob(IcsCisaConnector connector)
|
||||
=> _connector = connector ?? throw new ArgumentNullException(nameof(connector));
|
||||
|
||||
public Task ExecuteAsync(JobExecutionContext context, CancellationToken cancellationToken)
|
||||
=> _connector.ParseAsync(context.Services, cancellationToken);
|
||||
}
|
||||
|
||||
internal sealed class IcsCisaMapJob : IJob
|
||||
{
|
||||
private readonly IcsCisaConnector _connector;
|
||||
|
||||
public IcsCisaMapJob(IcsCisaConnector connector)
|
||||
=> _connector = connector ?? throw new ArgumentNullException(nameof(connector));
|
||||
|
||||
public Task ExecuteAsync(JobExecutionContext context, CancellationToken cancellationToken)
|
||||
=> _connector.MapAsync(context.Services, cancellationToken);
|
||||
}
|
||||
@@ -0,0 +1,29 @@
|
||||
<?xml version='1.0' encoding='utf-8'?>
|
||||
<Project Sdk="Microsoft.NET.Sdk">
|
||||
|
||||
<PropertyGroup>
|
||||
<TargetFramework>net10.0</TargetFramework>
|
||||
<ImplicitUsings>enable</ImplicitUsings>
|
||||
<Nullable>enable</Nullable>
|
||||
</PropertyGroup>
|
||||
|
||||
<ItemGroup>
|
||||
<ProjectReference Include="../../../__Libraries/StellaOps.Plugin/StellaOps.Plugin.csproj" />
|
||||
|
||||
<ProjectReference Include="../StellaOps.Concelier.Connector.Common/StellaOps.Concelier.Connector.Common.csproj" />
|
||||
<ProjectReference Include="../StellaOps.Concelier.Core/StellaOps.Concelier.Core.csproj" />
|
||||
<ProjectReference Include="../StellaOps.Concelier.Models/StellaOps.Concelier.Models.csproj" />
|
||||
<ProjectReference Include="../StellaOps.Concelier.Storage.Mongo/StellaOps.Concelier.Storage.Mongo.csproj" />
|
||||
</ItemGroup>
|
||||
|
||||
<ItemGroup>
|
||||
<PackageReference Include="System.ServiceModel.Syndication" Version="8.0.0" />
|
||||
</ItemGroup>
|
||||
|
||||
<ItemGroup>
|
||||
<AssemblyAttribute Include="System.Runtime.CompilerServices.InternalsVisibleTo">
|
||||
<_Parameter1>StellaOps.Concelier.Connector.Ics.Cisa.Tests</_Parameter1>
|
||||
</AssemblyAttribute>
|
||||
</ItemGroup>
|
||||
|
||||
</Project>
|
||||
@@ -0,0 +1,15 @@
|
||||
# TASKS
|
||||
| Task | Owner(s) | Depends on | Notes |
|
||||
|---|---|---|---|
|
||||
|FEEDCONN-ICSCISA-02-001 Document CISA ICS feed contract|BE-Conn-ICS-CISA|Research|**DONE (2025-10-11)** – `https://www.cisa.gov/cybersecurity-advisories/ics-advisories.xml` and legacy `/sites/default/files/feeds/...` return Akamai 403 even with browser UA; HTML landing page blocked as well. Logged full headers (x-reference-error, AkamaiGHost) in `docs/concelier-connector-research-20251011.md` and initiated GovDelivery access request.|
|
||||
|FEEDCONN-ICSCISA-02-002 Fetch pipeline & cursor storage|BE-Conn-ICS-CISA|Source.Common, Storage.Mongo|**DONE (2025-10-16)** – Confirmed proxy knobs + cursor state behave with the refreshed fixtures; ops runbook now captures proxy usage/validation so the fetch stage is production-ready.|
|
||||
|FEEDCONN-ICSCISA-02-003 DTO/parser implementation|BE-Conn-ICS-CISA|Source.Common|**DONE (2025-10-16)** – Feed parser fixtures updated to retain vendor PDFs as attachments while maintaining reference coverage; console diagnostics removed.|
|
||||
|FEEDCONN-ICSCISA-02-004 Canonical mapping & range primitives|BE-Conn-ICS-CISA|Models|**DONE (2025-10-16)** – `TryCreateSemVerPrimitive` flow + Mongo deserialiser now persist `exactValue` (`4.2` → `4.2.0`), unblocking canonical snapshots.|
|
||||
|FEEDCONN-ICSCISA-02-005 Deterministic fixtures/tests|QA|Testing|**DONE (2025-10-16)** – `dotnet test src/Concelier/__Tests/StellaOps.Concelier.Connector.Ics.Cisa.Tests/...` passes; fixtures assert attachment handling + SemVer semantics.|
|
||||
|FEEDCONN-ICSCISA-02-006 Telemetry & documentation|DevEx|Docs|**DONE (2025-10-16)** – Ops guide documents attachment checks, SemVer exact values, and proxy guidance; diagnostics remain unchanged.|
|
||||
|FEEDCONN-ICSCISA-02-007 Detail document inventory|BE-Conn-ICS-CISA|Research|**DONE (2025-10-16)** – Validated canned detail pages vs feed output so attachment inventories stay aligned; archived expectations noted in `HANDOVER.md`.|
|
||||
|FEEDCONN-ICSCISA-02-008 Distribution fallback strategy|BE-Conn-ICS-CISA|Research|**DONE (2025-10-11)** – Outlined GovDelivery token request, HTML scrape + email digest fallback, and dependency on Ops for credential workflow; awaiting decision before fetch implementation.|
|
||||
|FEEDCONN-ICSCISA-02-009 GovDelivery credential onboarding|Ops, BE-Conn-ICS-CISA|Ops|**DONE (2025-10-14)** – GovDelivery onboarding runbook captured in `docs/ops/concelier-icscisa-operations.md`; secret vault path and Offline Kit handling documented.|
|
||||
|FEEDCONN-ICSCISA-02-010 Mitigation & SemVer polish|BE-Conn-ICS-CISA|02-003, 02-004|**DONE (2025-10-16)** – Attachment + mitigation references now land as expected and SemVer primitives carry exact values; end-to-end suite green (see `HANDOVER.md`).|
|
||||
|FEEDCONN-ICSCISA-02-011 Docs & telemetry refresh|DevEx|02-006|**DONE (2025-10-16)** – Ops documentation refreshed (attachments, SemVer validation, proxy knobs) and telemetry notes verified.|
|
||||
|FEEDCONN-ICSCISA-02-012 Normalized version decision|BE-Conn-ICS-CISA|Merge coordination (`FEEDMERGE-COORD-02-900`)|**TODO (due 2025-10-23)** – Promote existing `SemVerPrimitive` exact values into `NormalizedVersions` via `.ToNormalizedVersionRule("ics-cisa:{advisoryId}:{product}")`, add regression coverage, and open Models ticket if non-SemVer firmware requires a new scheme.|
|
||||
Reference in New Issue
Block a user