Add MergeUsageAnalyzer to detect legacy merge service usage
Some checks failed
Docs CI / lint-and-preview (push) Has been cancelled
Some checks failed
Docs CI / lint-and-preview (push) Has been cancelled
- Implemented MergeUsageAnalyzer to flag usage of AdvisoryMergeService and AddMergeModule. - Created AnalyzerReleases.Shipped.md and AnalyzerReleases.Unshipped.md for release documentation. - Added tests for MergeUsageAnalyzer to ensure correct diagnostics for various scenarios. - Updated project files for analyzers and tests to include necessary dependencies and configurations. - Introduced a sample report structure for scanner output.
This commit is contained in:
@@ -29,7 +29,7 @@ public sealed class AdvisoryObservationQueryService : IAdvisoryObservationQueryS
|
||||
|
||||
var normalizedTenant = NormalizeTenant(options.Tenant);
|
||||
var normalizedObservationIds = NormalizeSet(options.ObservationIds, static value => value, StringComparer.Ordinal);
|
||||
var normalizedAliases = NormalizeSet(options.Aliases, static value => value.ToLowerInvariant(), StringComparer.Ordinal);
|
||||
var normalizedAliases = NormalizeSet(options.Aliases, static value => value, StringComparer.OrdinalIgnoreCase);
|
||||
var normalizedPurls = NormalizeSet(options.Purls, static value => value, StringComparer.Ordinal);
|
||||
var normalizedCpes = NormalizeSet(options.Cpes, static value => value, StringComparer.Ordinal);
|
||||
|
||||
|
||||
@@ -9,7 +9,7 @@
|
||||
> Docs alignment (2025-10-26): Linkset expectations detailed in AOC reference §4 and policy-engine architecture §2.1.
|
||||
> 2025-10-28: Advisory raw ingestion now strips client-supplied supersedes hints, logs ignored pointers, and surfaces repository-supplied supersedes identifiers; service tests cover duplicate handling and append-only semantics.
|
||||
> Docs alignment (2025-10-26): Deployment guide + observability guide describe supersedes metrics; ensure implementation emits `aoc_violation_total` on failure.
|
||||
| CONCELIER-CORE-AOC-19-004 `Remove ingestion normalization` | DOING (2025-10-28) | Concelier Core Guild | CONCELIER-CORE-AOC-19-002, POLICY-AOC-19-003 | Strip normalization/dedup/severity logic from ingestion pipelines, delegate derived computations to Policy Engine, and update exporters/tests to consume raw documents only.<br>2025-10-29 19:05Z: Audit completed for `AdvisoryRawService`/Mongo repo to confirm alias order/dedup removal persists; identified remaining normalization in observation/linkset factory that will be revised to surface raw duplicates for Policy ingestion. Change sketch + regression matrix drafted under `docs/dev/aoc-normalization-removal-notes.md` (pending commit).<br>2025-10-31 20:45Z: Added raw linkset projection to observations/storage, exposing canonical+raw views, refreshed fixtures/tests, and documented behaviour in models/doc factory.<br>2025-10-31 21:10Z: Coordinated with Policy Engine (POLICY-ENGINE-20-003) on adoption timeline; backfill + consumer readiness tracked in `docs/dev/raw-linkset-backfill-plan.md`.<br>2025-11-05 14:25Z: Resuming to document merge-dependent normalization paths and prepare implementation notes for `noMergeEnabled` gating before code changes land.<br>2025-11-05 19:20Z: Observation factory/linkset now preserve upstream ordering + duplicates; canonicalisation responsibility shifts to downstream consumers with refreshed unit coverage.<br>2025-11-06 16:10Z: Updated AOC reference/backfill docs with raw vs canonical guidance and cross-linked analyzer guardrails. |
|
||||
| CONCELIER-CORE-AOC-19-004 `Remove ingestion normalization` | DONE (2025-11-06) | Concelier Core Guild | CONCELIER-CORE-AOC-19-002, POLICY-AOC-19-003 | Strip normalization/dedup/severity logic from ingestion pipelines, delegate derived computations to Policy Engine, and update exporters/tests to consume raw documents only.<br>2025-10-29 19:05Z: Audit completed for `AdvisoryRawService`/Mongo repo to confirm alias order/dedup removal persists; identified remaining normalization in observation/linkset factory that will be revised to surface raw duplicates for Policy ingestion. Change sketch + regression matrix drafted under `docs/dev/aoc-normalization-removal-notes.md` (pending commit).<br>2025-10-31 20:45Z: Added raw linkset projection to observations/storage, exposing canonical+raw views, refreshed fixtures/tests, and documented behaviour in models/doc factory.<br>2025-10-31 21:10Z: Coordinated with Policy Engine (POLICY-ENGINE-20-003) on adoption timeline; backfill + consumer readiness tracked in `docs/dev/raw-linkset-backfill-plan.md`.<br>2025-11-05 14:25Z: Resuming to document merge-dependent normalization paths and prepare implementation notes for `noMergeEnabled` gating before code changes land.<br>2025-11-05 19:20Z: Observation factory/linkset now preserve upstream ordering + duplicates; canonicalisation responsibility shifts to downstream consumers with refreshed unit coverage.<br>2025-11-06 16:10Z: Updated AOC reference/backfill docs with raw vs canonical guidance and cross-linked analyzer guardrails.<br>2025-11-06 23:40Z: Final pass preserves raw alias casing/whitespace end-to-end; query filters now compare case-insensitively, exporter fixtures refreshed, and docs aligned. Tests: `StellaOps.Concelier.Models/Core/Storage.Mongo.Tests` green on .NET 10 preview. |
|
||||
> Docs alignment (2025-10-26): Architecture overview emphasises policy-only derivation; coordinate with Policy Engine guild for rollout.
|
||||
> 2025-10-29: `AdvisoryRawService` now preserves upstream alias/linkset ordering (trim-only) and updated AOC documentation reflects the behaviour; follow-up to ensure policy consumers handle duplicates remains open.
|
||||
| CONCELIER-CORE-AOC-19-013 `Authority tenant scope smoke coverage` | TODO | Concelier Core Guild | AUTH-AOC-19-002 | Extend Concelier smoke/e2e fixtures to configure `requiredTenants` and assert cross-tenant rejection with updated Authority tokens. | Coordinate deliverable so Authority docs (`AUTH-AOC-19-003`) can close once tests are in place. |
|
||||
|
||||
@@ -13,15 +13,21 @@ namespace StellaOps.Concelier.Merge;
|
||||
[Obsolete("Legacy merge module is deprecated; prefer Link-Not-Merge linkset pipelines. Track MERGE-LNM-21-002 and set concelier:features:noMergeEnabled=true to disable registration.", DiagnosticId = "CONCELIER0001", UrlFormat = "https://stella-ops.org/docs/migration/no-merge")]
|
||||
public static class MergeServiceCollectionExtensions
|
||||
{
|
||||
public static IServiceCollection AddMergeModule(this IServiceCollection services, IConfiguration configuration)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(services);
|
||||
ArgumentNullException.ThrowIfNull(configuration);
|
||||
|
||||
public static IServiceCollection AddMergeModule(this IServiceCollection services, IConfiguration configuration)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(services);
|
||||
ArgumentNullException.ThrowIfNull(configuration);
|
||||
|
||||
var noMergeEnabled = configuration.GetValue<bool?>("concelier:features:noMergeEnabled");
|
||||
if (noMergeEnabled is true)
|
||||
{
|
||||
return services;
|
||||
}
|
||||
|
||||
services.TryAddSingleton<CanonicalHashCalculator>();
|
||||
services.TryAddSingleton<CanonicalMerger>();
|
||||
services.TryAddSingleton<AliasGraphResolver>();
|
||||
services.TryAddSingleton<AffectedPackagePrecedenceResolver>(sp =>
|
||||
services.TryAddSingleton<AffectedPackagePrecedenceResolver>(sp =>
|
||||
{
|
||||
var options = configuration.GetSection("concelier:merge:precedence").Get<AdvisoryPrecedenceOptions>();
|
||||
return options is null ? new AffectedPackagePrecedenceResolver() : new AffectedPackagePrecedenceResolver(options);
|
||||
|
||||
@@ -1,15 +1,15 @@
|
||||
# TASKS
|
||||
| Task | Owner(s) | Depends on | Notes |
|
||||
# TASKS
|
||||
| Task | Owner(s) | Depends on | Notes |
|
||||
|---|---|---|---|
|
||||
|Link-Not-Merge version provenance coordination|BE-Merge|CONCELIER-LNM-21-001|**DONE (2025-11-04)** – Coordinated connector rollout: updated `docs/dev/normalized-rule-recipes.md` with a per-connector status table + follow-up IDs, enabled `Normalized version rules missing` diagnostics in `AdvisoryPrecedenceMerger`, and confirmed Linkset validation metrics reflect remaining upstream gaps (ACSC/CCCS/CERTBUND/Cisco/RU-BDU awaiting structured ranges).|
|
||||
|FEEDMERGE-COORD-02-901 Connector deadline check-ins|BE-Merge|FEEDMERGE-COORD-02-900|**TODO (due 2025-10-21)** – Confirm Cccs/Cisco version-provenance updates land, capture `LinksetVersionCoverage` dashboard snapshots (expect zero missing-range warnings), and update coordination docs with the results.<br>2025-10-29: Observation metrics now surface `version_entries_total`/`missing_version_entries_total`; include screenshots for both when closing this task.|
|
||||
|FEEDMERGE-COORD-02-902 ICS-CISA version comparison support|BE-Merge, Models|FEEDMERGE-COORD-02-900|**TODO (due 2025-10-23)** – Review ICS-CISA sample advisories, validate reuse of existing comparison helpers, and pre-stage Models ticket template only if a new firmware comparator is required. Document the outcome and observation coverage logs in coordination docs + tracker files.<br>2025-10-29: `docs/dev/normalized-rule-recipes.md` (§2–§3) now covers observation entries; attach decision summary + log sample when handing off to Models.|
|
||||
|FEEDMERGE-COORD-02-903 KISA firmware scheme review|BE-Merge, Models|FEEDMERGE-COORD-02-900|**TODO (due 2025-10-24)** – Pair with KISA team on proposed firmware comparison helper (`kisa.build` or variant), ensure observation mapper alignment, and open Models ticket only if a new comparator is required. Log the final helper signature and observation coverage metrics in coordination docs + tracker files.|
|
||||
|
||||
## Link-Not-Merge v1 Transition
|
||||
| Task | Owner(s) | Depends on | Notes |
|
||||
|---|---|---|---|
|
||||
|
||||
## Link-Not-Merge v1 Transition
|
||||
| Task | Owner(s) | Depends on | Notes |
|
||||
|---|---|---|---|
|
||||
|MERGE-LNM-21-001 Migration plan authoring|BE-Merge, Architecture Guild|CONCELIER-LNM-21-101|**DONE (2025-11-03)** – Authored `docs/migration/no-merge.md` with rollout phases, backfill/validation checklists, rollback guidance, and ownership matrix for the Link-Not-Merge cutover.|
|
||||
|MERGE-LNM-21-002 Merge service deprecation|BE-Merge|MERGE-LNM-21-001|**DOING (2025-11-03)** – Auditing service registrations, DI bindings, and tests consuming `AdvisoryMergeService`; drafting deprecation plan and analyzer scope prior to code removal.<br>2025-11-05 14:42Z: Implementing `concelier:features:noMergeEnabled` gate, merge job allowlist checks, `[Obsolete]` markings, and analyzer scaffolding to steer consumers toward linkset APIs.<br>2025-11-06 16:10Z: Introduced Roslyn analyzer (`CONCELIER0002`) referenced by Concelier WebService + tests, documented suppression guidance, and updated migration playbook.|
|
||||
|MERGE-LNM-21-002 Merge service deprecation|BE-Merge|MERGE-LNM-21-001|**DOING (2025-11-03)** – Auditing service registrations, DI bindings, and tests consuming `AdvisoryMergeService`; drafting deprecation plan and analyzer scope prior to code removal.<br>2025-11-05 14:42Z: Implementing `concelier:features:noMergeEnabled` gate, merge job allowlist checks, `[Obsolete]` markings, and analyzer scaffolding to steer consumers toward linkset APIs.<br>2025-11-06 16:10Z: Introduced Roslyn analyzer (`CONCELIER0002`) referenced by Concelier WebService + tests, documented suppression guidance, and updated migration playbook.<br>2025-11-06 23:45Z: Analyzer enforcement merged; DI removal + feature-flag default change remain. Analyzer tests compile locally but restore blocked offline (`Microsoft.Bcl.AsyncInterfaces >= 8.0` absent) — capture follow-up once nuget mirror updated.|
|
||||
> 2025-11-03: Catalogued call sites (WebService Program `AddMergeModule`, built-in job registration `merge:reconcile`, `MergeReconcileJob`) and confirmed unit tests are the only direct `MergeAsync` callers; next step is to define analyzer + replacement observability coverage.
|
||||
|MERGE-LNM-21-003 Determinism/test updates|QA Guild, BE-Merge|MERGE-LNM-21-002|Replace merge determinism suites with observation/linkset regression tests verifying no data mutation and conflicts remain visible.|
|
||||
|MERGE-LNM-21-003 Determinism/test updates|QA Guild, BE-Merge|MERGE-LNM-21-002|Replace merge determinism suites with observation/linkset regression tests verifying no data mutation and conflicts remain visible.|
|
||||
|
||||
@@ -259,17 +259,27 @@ public sealed record AdvisoryObservationContent
|
||||
}
|
||||
}
|
||||
|
||||
public sealed record AdvisoryObservationReference
|
||||
{
|
||||
public AdvisoryObservationReference(string type, string url)
|
||||
{
|
||||
Type = Validation.EnsureNotNullOrWhiteSpace(type, nameof(type)).ToLowerInvariant();
|
||||
Url = Validation.EnsureNotNullOrWhiteSpace(url, nameof(url));
|
||||
}
|
||||
|
||||
public string Type { get; }
|
||||
|
||||
public string Url { get; }
|
||||
public sealed record AdvisoryObservationReference
|
||||
{
|
||||
public AdvisoryObservationReference(string type, string url)
|
||||
{
|
||||
if (string.IsNullOrWhiteSpace(type))
|
||||
{
|
||||
throw new ArgumentException("Reference type cannot be null or whitespace.", nameof(type));
|
||||
}
|
||||
|
||||
if (string.IsNullOrWhiteSpace(url))
|
||||
{
|
||||
throw new ArgumentException("Reference url cannot be null or whitespace.", nameof(url));
|
||||
}
|
||||
|
||||
Type = type;
|
||||
Url = url;
|
||||
}
|
||||
|
||||
public string Type { get; }
|
||||
|
||||
public string Url { get; }
|
||||
}
|
||||
|
||||
public sealed record AdvisoryObservationLinkset
|
||||
@@ -304,13 +314,12 @@ public sealed record AdvisoryObservationLinkset
|
||||
var builder = ImmutableArray.CreateBuilder<string>();
|
||||
foreach (var value in values)
|
||||
{
|
||||
var trimmed = Validation.TrimToNull(value);
|
||||
if (trimmed is null)
|
||||
if (value is null)
|
||||
{
|
||||
continue;
|
||||
}
|
||||
|
||||
builder.Add(trimmed);
|
||||
builder.Add(value);
|
||||
}
|
||||
|
||||
return builder.Count == 0 ? ImmutableArray<string>.Empty : builder.ToImmutable();
|
||||
|
||||
@@ -1,11 +1,13 @@
|
||||
using System;
|
||||
using System.Collections.Generic;
|
||||
using System.Linq;
|
||||
using MongoDB.Driver;
|
||||
using StellaOps.Concelier.Core.Observations;
|
||||
using StellaOps.Concelier.Models.Observations;
|
||||
|
||||
namespace StellaOps.Concelier.Storage.Mongo.Observations;
|
||||
using System;
|
||||
using System.Collections.Generic;
|
||||
using System.Linq;
|
||||
using System.Text.RegularExpressions;
|
||||
using MongoDB.Bson;
|
||||
using MongoDB.Driver;
|
||||
using StellaOps.Concelier.Core.Observations;
|
||||
using StellaOps.Concelier.Models.Observations;
|
||||
|
||||
namespace StellaOps.Concelier.Storage.Mongo.Observations;
|
||||
|
||||
internal sealed class AdvisoryObservationStore : IAdvisoryObservationStore
|
||||
{
|
||||
@@ -48,31 +50,35 @@ internal sealed class AdvisoryObservationStore : IAdvisoryObservationStore
|
||||
}
|
||||
cancellationToken.ThrowIfCancellationRequested();
|
||||
|
||||
var normalizedTenant = tenant.ToLowerInvariant();
|
||||
var normalizedObservationIds = NormalizeValues(observationIds, static value => value);
|
||||
var normalizedAliases = NormalizeValues(aliases, static value => value.ToLowerInvariant());
|
||||
var normalizedPurls = NormalizeValues(purls, static value => value);
|
||||
var normalizedCpes = NormalizeValues(cpes, static value => value);
|
||||
|
||||
var builder = Builders<AdvisoryObservationDocument>.Filter;
|
||||
var filters = new List<FilterDefinition<AdvisoryObservationDocument>>
|
||||
{
|
||||
builder.Eq(document => document.Tenant, normalizedTenant)
|
||||
var normalizedTenant = tenant.ToLowerInvariant();
|
||||
var normalizedObservationIds = NormalizeValues(observationIds, static value => value);
|
||||
var normalizedAliases = NormalizeAliasFilters(aliases);
|
||||
var normalizedPurls = NormalizeValues(purls, static value => value);
|
||||
var normalizedCpes = NormalizeValues(cpes, static value => value);
|
||||
|
||||
var builder = Builders<AdvisoryObservationDocument>.Filter;
|
||||
var filters = new List<FilterDefinition<AdvisoryObservationDocument>>
|
||||
{
|
||||
builder.Eq(document => document.Tenant, normalizedTenant)
|
||||
};
|
||||
|
||||
if (normalizedObservationIds.Length > 0)
|
||||
{
|
||||
filters.Add(builder.In(document => document.Id, normalizedObservationIds));
|
||||
}
|
||||
|
||||
if (normalizedAliases.Length > 0)
|
||||
{
|
||||
filters.Add(builder.In("linkset.aliases", normalizedAliases));
|
||||
}
|
||||
|
||||
if (normalizedPurls.Length > 0)
|
||||
{
|
||||
filters.Add(builder.In("linkset.purls", normalizedPurls));
|
||||
|
||||
if (normalizedAliases.Length > 0)
|
||||
{
|
||||
var aliasFilters = normalizedAliases
|
||||
.Select(alias => CreateAliasFilter(builder, alias))
|
||||
.ToList();
|
||||
|
||||
filters.Add(builder.Or(aliasFilters));
|
||||
}
|
||||
|
||||
if (normalizedPurls.Length > 0)
|
||||
{
|
||||
filters.Add(builder.In("linkset.purls", normalizedPurls));
|
||||
}
|
||||
|
||||
if (normalizedCpes.Length > 0)
|
||||
@@ -101,16 +107,16 @@ internal sealed class AdvisoryObservationStore : IAdvisoryObservationStore
|
||||
.Limit(limit)
|
||||
.ToListAsync(cancellationToken)
|
||||
.ConfigureAwait(false);
|
||||
|
||||
return documents.Select(AdvisoryObservationDocumentFactory.ToModel).ToArray();
|
||||
}
|
||||
|
||||
private static string[] NormalizeValues(IEnumerable<string>? values, Func<string, string> projector)
|
||||
{
|
||||
if (values is null)
|
||||
{
|
||||
return Array.Empty<string>();
|
||||
}
|
||||
|
||||
return documents.Select(AdvisoryObservationDocumentFactory.ToModel).ToArray();
|
||||
}
|
||||
|
||||
private static string[] NormalizeValues(IEnumerable<string>? values, Func<string, string> projector)
|
||||
{
|
||||
if (values is null)
|
||||
{
|
||||
return Array.Empty<string>();
|
||||
}
|
||||
|
||||
var set = new HashSet<string>(StringComparer.Ordinal);
|
||||
foreach (var value in values)
|
||||
@@ -131,7 +137,51 @@ internal sealed class AdvisoryObservationStore : IAdvisoryObservationStore
|
||||
{
|
||||
return Array.Empty<string>();
|
||||
}
|
||||
|
||||
return set.ToArray();
|
||||
}
|
||||
}
|
||||
|
||||
return set.ToArray();
|
||||
}
|
||||
|
||||
private static string[] NormalizeAliasFilters(IEnumerable<string>? aliases)
|
||||
{
|
||||
if (aliases is null)
|
||||
{
|
||||
return Array.Empty<string>();
|
||||
}
|
||||
|
||||
var set = new HashSet<string>(StringComparer.OrdinalIgnoreCase);
|
||||
var list = new List<string>();
|
||||
|
||||
foreach (var alias in aliases)
|
||||
{
|
||||
if (alias is null)
|
||||
{
|
||||
continue;
|
||||
}
|
||||
|
||||
var trimmed = alias.Trim();
|
||||
if (trimmed.Length == 0)
|
||||
{
|
||||
continue;
|
||||
}
|
||||
|
||||
if (set.Add(trimmed))
|
||||
{
|
||||
list.Add(trimmed);
|
||||
}
|
||||
}
|
||||
|
||||
return list.Count == 0 ? Array.Empty<string>() : list.ToArray();
|
||||
}
|
||||
|
||||
private static FilterDefinition<AdvisoryObservationDocument> CreateAliasFilter(
|
||||
FilterDefinitionBuilder<AdvisoryObservationDocument> builder,
|
||||
string alias)
|
||||
{
|
||||
var escaped = Regex.Escape(alias);
|
||||
var regex = new BsonRegularExpression($"^{escaped}$", "i");
|
||||
|
||||
return builder.Or(
|
||||
builder.Regex("rawLinkset.aliases", regex),
|
||||
builder.Regex("linkset.aliases", regex));
|
||||
}
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user