Add MergeUsageAnalyzer to detect legacy merge service usage
Some checks failed
Docs CI / lint-and-preview (push) Has been cancelled

- Implemented MergeUsageAnalyzer to flag usage of AdvisoryMergeService and AddMergeModule.
- Created AnalyzerReleases.Shipped.md and AnalyzerReleases.Unshipped.md for release documentation.
- Added tests for MergeUsageAnalyzer to ensure correct diagnostics for various scenarios.
- Updated project files for analyzers and tests to include necessary dependencies and configurations.
- Introduced a sample report structure for scanner output.
This commit is contained in:
master
2025-11-06 15:03:39 +02:00
parent 5a923d968c
commit 950f238a93
45 changed files with 1291 additions and 623 deletions

View File

@@ -29,7 +29,7 @@ public sealed class AdvisoryObservationQueryService : IAdvisoryObservationQueryS
var normalizedTenant = NormalizeTenant(options.Tenant);
var normalizedObservationIds = NormalizeSet(options.ObservationIds, static value => value, StringComparer.Ordinal);
var normalizedAliases = NormalizeSet(options.Aliases, static value => value.ToLowerInvariant(), StringComparer.Ordinal);
var normalizedAliases = NormalizeSet(options.Aliases, static value => value, StringComparer.OrdinalIgnoreCase);
var normalizedPurls = NormalizeSet(options.Purls, static value => value, StringComparer.Ordinal);
var normalizedCpes = NormalizeSet(options.Cpes, static value => value, StringComparer.Ordinal);

View File

@@ -9,7 +9,7 @@
> Docs alignment (2025-10-26): Linkset expectations detailed in AOC reference §4 and policy-engine architecture §2.1.
> 2025-10-28: Advisory raw ingestion now strips client-supplied supersedes hints, logs ignored pointers, and surfaces repository-supplied supersedes identifiers; service tests cover duplicate handling and append-only semantics.
> Docs alignment (2025-10-26): Deployment guide + observability guide describe supersedes metrics; ensure implementation emits `aoc_violation_total` on failure.
| CONCELIER-CORE-AOC-19-004 `Remove ingestion normalization` | DOING (2025-10-28) | Concelier Core Guild | CONCELIER-CORE-AOC-19-002, POLICY-AOC-19-003 | Strip normalization/dedup/severity logic from ingestion pipelines, delegate derived computations to Policy Engine, and update exporters/tests to consume raw documents only.<br>2025-10-29 19:05Z: Audit completed for `AdvisoryRawService`/Mongo repo to confirm alias order/dedup removal persists; identified remaining normalization in observation/linkset factory that will be revised to surface raw duplicates for Policy ingestion. Change sketch + regression matrix drafted under `docs/dev/aoc-normalization-removal-notes.md` (pending commit).<br>2025-10-31 20:45Z: Added raw linkset projection to observations/storage, exposing canonical+raw views, refreshed fixtures/tests, and documented behaviour in models/doc factory.<br>2025-10-31 21:10Z: Coordinated with Policy Engine (POLICY-ENGINE-20-003) on adoption timeline; backfill + consumer readiness tracked in `docs/dev/raw-linkset-backfill-plan.md`.<br>2025-11-05 14:25Z: Resuming to document merge-dependent normalization paths and prepare implementation notes for `noMergeEnabled` gating before code changes land.<br>2025-11-05 19:20Z: Observation factory/linkset now preserve upstream ordering + duplicates; canonicalisation responsibility shifts to downstream consumers with refreshed unit coverage.<br>2025-11-06 16:10Z: Updated AOC reference/backfill docs with raw vs canonical guidance and cross-linked analyzer guardrails. |
| CONCELIER-CORE-AOC-19-004 `Remove ingestion normalization` | DONE (2025-11-06) | Concelier Core Guild | CONCELIER-CORE-AOC-19-002, POLICY-AOC-19-003 | Strip normalization/dedup/severity logic from ingestion pipelines, delegate derived computations to Policy Engine, and update exporters/tests to consume raw documents only.<br>2025-10-29 19:05Z: Audit completed for `AdvisoryRawService`/Mongo repo to confirm alias order/dedup removal persists; identified remaining normalization in observation/linkset factory that will be revised to surface raw duplicates for Policy ingestion. Change sketch + regression matrix drafted under `docs/dev/aoc-normalization-removal-notes.md` (pending commit).<br>2025-10-31 20:45Z: Added raw linkset projection to observations/storage, exposing canonical+raw views, refreshed fixtures/tests, and documented behaviour in models/doc factory.<br>2025-10-31 21:10Z: Coordinated with Policy Engine (POLICY-ENGINE-20-003) on adoption timeline; backfill + consumer readiness tracked in `docs/dev/raw-linkset-backfill-plan.md`.<br>2025-11-05 14:25Z: Resuming to document merge-dependent normalization paths and prepare implementation notes for `noMergeEnabled` gating before code changes land.<br>2025-11-05 19:20Z: Observation factory/linkset now preserve upstream ordering + duplicates; canonicalisation responsibility shifts to downstream consumers with refreshed unit coverage.<br>2025-11-06 16:10Z: Updated AOC reference/backfill docs with raw vs canonical guidance and cross-linked analyzer guardrails.<br>2025-11-06 23:40Z: Final pass preserves raw alias casing/whitespace end-to-end; query filters now compare case-insensitively, exporter fixtures refreshed, and docs aligned. Tests: `StellaOps.Concelier.Models/Core/Storage.Mongo.Tests` green on .NET 10 preview. |
> Docs alignment (2025-10-26): Architecture overview emphasises policy-only derivation; coordinate with Policy Engine guild for rollout.
> 2025-10-29: `AdvisoryRawService` now preserves upstream alias/linkset ordering (trim-only) and updated AOC documentation reflects the behaviour; follow-up to ensure policy consumers handle duplicates remains open.
| CONCELIER-CORE-AOC-19-013 `Authority tenant scope smoke coverage` | TODO | Concelier Core Guild | AUTH-AOC-19-002 | Extend Concelier smoke/e2e fixtures to configure `requiredTenants` and assert cross-tenant rejection with updated Authority tokens. | Coordinate deliverable so Authority docs (`AUTH-AOC-19-003`) can close once tests are in place. |

View File

@@ -13,15 +13,21 @@ namespace StellaOps.Concelier.Merge;
[Obsolete("Legacy merge module is deprecated; prefer Link-Not-Merge linkset pipelines. Track MERGE-LNM-21-002 and set concelier:features:noMergeEnabled=true to disable registration.", DiagnosticId = "CONCELIER0001", UrlFormat = "https://stella-ops.org/docs/migration/no-merge")]
public static class MergeServiceCollectionExtensions
{
public static IServiceCollection AddMergeModule(this IServiceCollection services, IConfiguration configuration)
{
ArgumentNullException.ThrowIfNull(services);
ArgumentNullException.ThrowIfNull(configuration);
public static IServiceCollection AddMergeModule(this IServiceCollection services, IConfiguration configuration)
{
ArgumentNullException.ThrowIfNull(services);
ArgumentNullException.ThrowIfNull(configuration);
var noMergeEnabled = configuration.GetValue<bool?>("concelier:features:noMergeEnabled");
if (noMergeEnabled is true)
{
return services;
}
services.TryAddSingleton<CanonicalHashCalculator>();
services.TryAddSingleton<CanonicalMerger>();
services.TryAddSingleton<AliasGraphResolver>();
services.TryAddSingleton<AffectedPackagePrecedenceResolver>(sp =>
services.TryAddSingleton<AffectedPackagePrecedenceResolver>(sp =>
{
var options = configuration.GetSection("concelier:merge:precedence").Get<AdvisoryPrecedenceOptions>();
return options is null ? new AffectedPackagePrecedenceResolver() : new AffectedPackagePrecedenceResolver(options);

View File

@@ -1,15 +1,15 @@
# TASKS
| Task | Owner(s) | Depends on | Notes |
# TASKS
| Task | Owner(s) | Depends on | Notes |
|---|---|---|---|
|Link-Not-Merge version provenance coordination|BE-Merge|CONCELIER-LNM-21-001|**DONE (2025-11-04)** Coordinated connector rollout: updated `docs/dev/normalized-rule-recipes.md` with a per-connector status table + follow-up IDs, enabled `Normalized version rules missing` diagnostics in `AdvisoryPrecedenceMerger`, and confirmed Linkset validation metrics reflect remaining upstream gaps (ACSC/CCCS/CERTBUND/Cisco/RU-BDU awaiting structured ranges).|
|FEEDMERGE-COORD-02-901 Connector deadline check-ins|BE-Merge|FEEDMERGE-COORD-02-900|**TODO (due 2025-10-21)** Confirm Cccs/Cisco version-provenance updates land, capture `LinksetVersionCoverage` dashboard snapshots (expect zero missing-range warnings), and update coordination docs with the results.<br>2025-10-29: Observation metrics now surface `version_entries_total`/`missing_version_entries_total`; include screenshots for both when closing this task.|
|FEEDMERGE-COORD-02-902 ICS-CISA version comparison support|BE-Merge, Models|FEEDMERGE-COORD-02-900|**TODO (due 2025-10-23)** Review ICS-CISA sample advisories, validate reuse of existing comparison helpers, and pre-stage Models ticket template only if a new firmware comparator is required. Document the outcome and observation coverage logs in coordination docs + tracker files.<br>2025-10-29: `docs/dev/normalized-rule-recipes.md` (§2§3) now covers observation entries; attach decision summary + log sample when handing off to Models.|
|FEEDMERGE-COORD-02-903 KISA firmware scheme review|BE-Merge, Models|FEEDMERGE-COORD-02-900|**TODO (due 2025-10-24)** Pair with KISA team on proposed firmware comparison helper (`kisa.build` or variant), ensure observation mapper alignment, and open Models ticket only if a new comparator is required. Log the final helper signature and observation coverage metrics in coordination docs + tracker files.|
## Link-Not-Merge v1 Transition
| Task | Owner(s) | Depends on | Notes |
|---|---|---|---|
## Link-Not-Merge v1 Transition
| Task | Owner(s) | Depends on | Notes |
|---|---|---|---|
|MERGE-LNM-21-001 Migration plan authoring|BE-Merge, Architecture Guild|CONCELIER-LNM-21-101|**DONE (2025-11-03)** Authored `docs/migration/no-merge.md` with rollout phases, backfill/validation checklists, rollback guidance, and ownership matrix for the Link-Not-Merge cutover.|
|MERGE-LNM-21-002 Merge service deprecation|BE-Merge|MERGE-LNM-21-001|**DOING (2025-11-03)** Auditing service registrations, DI bindings, and tests consuming `AdvisoryMergeService`; drafting deprecation plan and analyzer scope prior to code removal.<br>2025-11-05 14:42Z: Implementing `concelier:features:noMergeEnabled` gate, merge job allowlist checks, `[Obsolete]` markings, and analyzer scaffolding to steer consumers toward linkset APIs.<br>2025-11-06 16:10Z: Introduced Roslyn analyzer (`CONCELIER0002`) referenced by Concelier WebService + tests, documented suppression guidance, and updated migration playbook.|
|MERGE-LNM-21-002 Merge service deprecation|BE-Merge|MERGE-LNM-21-001|**DOING (2025-11-03)** Auditing service registrations, DI bindings, and tests consuming `AdvisoryMergeService`; drafting deprecation plan and analyzer scope prior to code removal.<br>2025-11-05 14:42Z: Implementing `concelier:features:noMergeEnabled` gate, merge job allowlist checks, `[Obsolete]` markings, and analyzer scaffolding to steer consumers toward linkset APIs.<br>2025-11-06 16:10Z: Introduced Roslyn analyzer (`CONCELIER0002`) referenced by Concelier WebService + tests, documented suppression guidance, and updated migration playbook.<br>2025-11-06 23:45Z: Analyzer enforcement merged; DI removal + feature-flag default change remain. Analyzer tests compile locally but restore blocked offline (`Microsoft.Bcl.AsyncInterfaces >= 8.0` absent) — capture follow-up once nuget mirror updated.|
> 2025-11-03: Catalogued call sites (WebService Program `AddMergeModule`, built-in job registration `merge:reconcile`, `MergeReconcileJob`) and confirmed unit tests are the only direct `MergeAsync` callers; next step is to define analyzer + replacement observability coverage.
|MERGE-LNM-21-003 Determinism/test updates|QA Guild, BE-Merge|MERGE-LNM-21-002|Replace merge determinism suites with observation/linkset regression tests verifying no data mutation and conflicts remain visible.|
|MERGE-LNM-21-003 Determinism/test updates|QA Guild, BE-Merge|MERGE-LNM-21-002|Replace merge determinism suites with observation/linkset regression tests verifying no data mutation and conflicts remain visible.|

View File

@@ -259,17 +259,27 @@ public sealed record AdvisoryObservationContent
}
}
public sealed record AdvisoryObservationReference
{
public AdvisoryObservationReference(string type, string url)
{
Type = Validation.EnsureNotNullOrWhiteSpace(type, nameof(type)).ToLowerInvariant();
Url = Validation.EnsureNotNullOrWhiteSpace(url, nameof(url));
}
public string Type { get; }
public string Url { get; }
public sealed record AdvisoryObservationReference
{
public AdvisoryObservationReference(string type, string url)
{
if (string.IsNullOrWhiteSpace(type))
{
throw new ArgumentException("Reference type cannot be null or whitespace.", nameof(type));
}
if (string.IsNullOrWhiteSpace(url))
{
throw new ArgumentException("Reference url cannot be null or whitespace.", nameof(url));
}
Type = type;
Url = url;
}
public string Type { get; }
public string Url { get; }
}
public sealed record AdvisoryObservationLinkset
@@ -304,13 +314,12 @@ public sealed record AdvisoryObservationLinkset
var builder = ImmutableArray.CreateBuilder<string>();
foreach (var value in values)
{
var trimmed = Validation.TrimToNull(value);
if (trimmed is null)
if (value is null)
{
continue;
}
builder.Add(trimmed);
builder.Add(value);
}
return builder.Count == 0 ? ImmutableArray<string>.Empty : builder.ToImmutable();

View File

@@ -1,11 +1,13 @@
using System;
using System.Collections.Generic;
using System.Linq;
using MongoDB.Driver;
using StellaOps.Concelier.Core.Observations;
using StellaOps.Concelier.Models.Observations;
namespace StellaOps.Concelier.Storage.Mongo.Observations;
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text.RegularExpressions;
using MongoDB.Bson;
using MongoDB.Driver;
using StellaOps.Concelier.Core.Observations;
using StellaOps.Concelier.Models.Observations;
namespace StellaOps.Concelier.Storage.Mongo.Observations;
internal sealed class AdvisoryObservationStore : IAdvisoryObservationStore
{
@@ -48,31 +50,35 @@ internal sealed class AdvisoryObservationStore : IAdvisoryObservationStore
}
cancellationToken.ThrowIfCancellationRequested();
var normalizedTenant = tenant.ToLowerInvariant();
var normalizedObservationIds = NormalizeValues(observationIds, static value => value);
var normalizedAliases = NormalizeValues(aliases, static value => value.ToLowerInvariant());
var normalizedPurls = NormalizeValues(purls, static value => value);
var normalizedCpes = NormalizeValues(cpes, static value => value);
var builder = Builders<AdvisoryObservationDocument>.Filter;
var filters = new List<FilterDefinition<AdvisoryObservationDocument>>
{
builder.Eq(document => document.Tenant, normalizedTenant)
var normalizedTenant = tenant.ToLowerInvariant();
var normalizedObservationIds = NormalizeValues(observationIds, static value => value);
var normalizedAliases = NormalizeAliasFilters(aliases);
var normalizedPurls = NormalizeValues(purls, static value => value);
var normalizedCpes = NormalizeValues(cpes, static value => value);
var builder = Builders<AdvisoryObservationDocument>.Filter;
var filters = new List<FilterDefinition<AdvisoryObservationDocument>>
{
builder.Eq(document => document.Tenant, normalizedTenant)
};
if (normalizedObservationIds.Length > 0)
{
filters.Add(builder.In(document => document.Id, normalizedObservationIds));
}
if (normalizedAliases.Length > 0)
{
filters.Add(builder.In("linkset.aliases", normalizedAliases));
}
if (normalizedPurls.Length > 0)
{
filters.Add(builder.In("linkset.purls", normalizedPurls));
if (normalizedAliases.Length > 0)
{
var aliasFilters = normalizedAliases
.Select(alias => CreateAliasFilter(builder, alias))
.ToList();
filters.Add(builder.Or(aliasFilters));
}
if (normalizedPurls.Length > 0)
{
filters.Add(builder.In("linkset.purls", normalizedPurls));
}
if (normalizedCpes.Length > 0)
@@ -101,16 +107,16 @@ internal sealed class AdvisoryObservationStore : IAdvisoryObservationStore
.Limit(limit)
.ToListAsync(cancellationToken)
.ConfigureAwait(false);
return documents.Select(AdvisoryObservationDocumentFactory.ToModel).ToArray();
}
private static string[] NormalizeValues(IEnumerable<string>? values, Func<string, string> projector)
{
if (values is null)
{
return Array.Empty<string>();
}
return documents.Select(AdvisoryObservationDocumentFactory.ToModel).ToArray();
}
private static string[] NormalizeValues(IEnumerable<string>? values, Func<string, string> projector)
{
if (values is null)
{
return Array.Empty<string>();
}
var set = new HashSet<string>(StringComparer.Ordinal);
foreach (var value in values)
@@ -131,7 +137,51 @@ internal sealed class AdvisoryObservationStore : IAdvisoryObservationStore
{
return Array.Empty<string>();
}
return set.ToArray();
}
}
return set.ToArray();
}
private static string[] NormalizeAliasFilters(IEnumerable<string>? aliases)
{
if (aliases is null)
{
return Array.Empty<string>();
}
var set = new HashSet<string>(StringComparer.OrdinalIgnoreCase);
var list = new List<string>();
foreach (var alias in aliases)
{
if (alias is null)
{
continue;
}
var trimmed = alias.Trim();
if (trimmed.Length == 0)
{
continue;
}
if (set.Add(trimmed))
{
list.Add(trimmed);
}
}
return list.Count == 0 ? Array.Empty<string>() : list.ToArray();
}
private static FilterDefinition<AdvisoryObservationDocument> CreateAliasFilter(
FilterDefinitionBuilder<AdvisoryObservationDocument> builder,
string alias)
{
var escaped = Regex.Escape(alias);
var regex = new BsonRegularExpression($"^{escaped}$", "i");
return builder.Or(
builder.Regex("rawLinkset.aliases", regex),
builder.Regex("linkset.aliases", regex));
}
}