up

src/StellaOps.Feedser.Merge/RANGE_PRIMITIVES_COORDINATION.md

@@ -23,7 +23,10 @@ Until these blocks land, connectors should stage changes behind a feature flag o
 | CertBund | BE-Conn-CERTBUND | All tasks still TODO | Ensure canonical mapper emits vendor range primitives plus normalized rules for product firmware. | Needs language/localisation guidance; coordinate with Localization WG for deterministic casing. |
 | CertCc | BE-Conn-CERTCC | Fetch in progress, mapping TODO | Map VINCE vendor/product data into `RangePrimitives` with `certcc.vendor` extensions; build normalized SemVer ranges when version strings surface. | Follow up on 2025-10-14 to review VINCE payload examples and confirm builder requirements. |
 | Cve | BE-Conn-CVE | Mapping/tests DONE (legacy SemVer) | Refactor `CveMapper` to call the shared builder and populate `NormalizedVersions` + provenance notes once models land. | Prepare MR behind `ENABLE_NORMALIZED_VERSIONS` flag; regression fixtures already cover version ranges—extend snapshots to cover rule arrays. |
-| Ghsa | BE-Conn-GHSA | Mapping/tests DONE; normalized rule task TODO | Switch to `SemVerRangeRuleBuilder`, populate `NormalizedVersions`, and extend fixtures with rule/provenance fields. | Target code review window 2025-10-15; needs builder API from Normalization team by 2025-10-13. |
+| Ghsa | BE-Conn-GHSA | Normalized rules emitted (2025-10-11) | Maintain SemVer builder integration; share regression diffs if schema shifts occur. | Fixtures refreshed with `ghsa:{identifier}` notes; OSV rollout next in queue—await connector handoff update. |
+| Osv | BE-Conn-OSV | Normalized rules emitted (2025-10-11) | Keep SemVer builder wiring current; extend notes if new ecosystems appear. | npm/PyPI parity snapshots updated with `osv:{ecosystem}:{advisoryId}:{identifier}` notes; merge analytics notified. |
+| Nvd | BE-Conn-NVD | Normalized rules emitted (2025-10-11) | Maintain SemVer coverage for ecosystem ranges; keep notes aligned with CVE IDs. | CPE ranges now emit semver primitives when versions parse; fixtures refreshed, report sent to FEEDMERGE-COORD-02-900. |
+| Cve | BE-Conn-CVE | Normalized rules emitted (2025-10-11) | Maintain SemVer notes for vendor ecosystems; backfill additional fixture coverage as CVE payloads expand. | Connector outputs `cve:{cveId}:{identifier}` notes; npm parity test fixtures updated and merge ping acknowledged. |
 | Ics.Cisa | BE-Conn-ICS-CISA | All tasks TODO | When defining product schema, plan for SemVer or vendor version rules (many advisories use firmware revisions). | Gather sample advisories and confirm whether ranges are SemVer or vendor-specific so we can introduce scheme identifiers early. |
 | Kisa | BE-Conn-KISA | All tasks TODO | Ensure DTO parsing captures version strings despite localisation; feed into normalized rule builder once ready. | Requires translation samples; request help from Localization WG before mapper implementation. |
 | Ru.Bdu | BE-Conn-BDU | All tasks TODO | Map product releases into normalized rules; add provenance notes referencing BDU advisory identifiers. | Verify we have UTF-8 safe handling in builder; share sample sanitized inputs. |
@@ -32,6 +35,53 @@ Until these blocks land, connectors should stage changes behind a feature flag o
 | Vndr.Cisco | BE-Conn-Cisco | All tasks TODO | When parser lands, normalise IOS/ASA version strings into SemVer-style or vendor-specific ranges and supply normalized arrays. | Identify whether ranges require custom comparer (maybe `ios.semver` style); escalate to Models if new scheme required. |
 | Vndr.Msrc | BE-Conn-MSRC | All tasks TODO | Canonical mapper must output product/build coverage as normalized rules (likely `msrc.patch` scheme) with provenance referencing KB IDs. | Sync with Models on adding scheme identifiers for MSRC packages; plan fixture coverage for monthly rollups. |
 
+## Storage alignment quick reference (2025-10-11)
+- `NormalizedVersionDocumentFactory` copies each `NormalizedVersionRule` into Mongo with the shape `{ packageId, packageType, scheme, type, style, min, minInclusive, max, maxInclusive, value, notes, decisionReason, constraint, source, recordedAt }`. `style` is currently a direct echo of `type` but reserved for future vendor comparers—no connector action required.
+- `constraint` is hydrated only when `NormalizedVersionRule` matches a legacy `VersionRange` primitive. Preserve `notes` (e.g., `nvd:cve-2025-1234`) so storage can join rules back to their provenance and carry decision reasoning.
+- Valid `scheme` values today are `semver`, `nevra`, and `evr`. Raise a Models ticket before introducing additional scheme identifiers (e.g., `apple.build`, `ios.semver`).
+- Prefer normalized `type` tokens from `NormalizedVersionRuleTypes` (`range`, `exact`, `lt`, `lte`, `gt`, `gte`). Builders already coerce casing/format—avoid custom strings.
+- Ensure `AffectedPackage.Identifier`/`Type` and `Provenance` collections are populated; storage falls back to package-level provenance if range-level data is absent, but loses traceability if both are empty.
+- Snapshot of an emitted document (SemVer range) for reference:
+  ```json
+  {
+    "packageId": "pkg:npm/example",
+    "packageType": "npm",
+    "scheme": "semver",
+    "type": "range",
+    "style": "range",
+    "min": "1.2.3",
+    "minInclusive": true,
+    "max": "2.0.0",
+    "maxInclusive": false,
+    "value": null,
+    "notes": "ghsa:GHSA-xxxx-yyyy",
+    "decisionReason": "ghsa-precedence-over-nvd",
+    "constraint": ">= 1.2.3 < 2.0.0",
+    "source": "ghsa",
+    "recordedAt": "2025-10-11T00:00:00Z"
+  }
+  ```
+- For distro sources emitting NEVRA/EVR primitives, expect the same envelope with `scheme` swapped accordingly. Example (`nevra`):
+  ```json
+  {
+    "packageId": "bash",
+    "packageType": "rpm",
+    "scheme": "nevra",
+    "type": "range",
+    "style": "range",
+    "min": "0:4.4.18-2.el7",
+    "minInclusive": true,
+    "max": "0:4.4.20-1.el7",
+    "maxInclusive": false,
+    "value": null,
+    "notes": "redhat:RHSA-2025:1234",
+    "decisionReason": "rhel-priority-over-nvd",
+    "constraint": "<= 0:4.4.20-1.el7",
+    "source": "redhat",
+    "recordedAt": "2025-10-11T00:00:00Z"
+  }
+  ```
+
 ## Immediate next steps
 - Normalization team to share draft `SemVerRangeRuleBuilder` API by **2025-10-13** for review; Merge will circulate feedback within 24 hours.
 - Connector owners to prepare fixture pull requests demonstrating sample normalized rule arrays (even if feature-flagged) by **2025-10-17**.
@@ -39,6 +89,7 @@ Until these blocks land, connectors should stage changes behind a feature flag o
 - Schedule held for **2025-10-14 14:00 UTC** to review the CERT/CC staging VINCE advisory sample once `enableDetailMapping` is flipped; capture findings in `#feedser-merge` with snapshot diffs.
 
 ## Tracking & follow-up
-- Capture connector progress updates in stand-ups twice per week; link PRs/issues back to this document.
+- Capture connector progress updates in stand-ups twice per week; link PRs/issues back to this document and the rollout dashboard (`docs/dev/normalized_versions_rollout.md`).
+- Monitor merge counters `feedser.merge.normalized_rules` and `feedser.merge.normalized_rules_missing` to spot advisories that still lack normalized arrays after precedence merge.
 - When a connector is ready to emit normalized rules, update its module `TASKS.md` status and ping Merge in `#feedser-merge` with fixture diff screenshots.
 - If new schemes or comparer logic is required (e.g., Cisco IOS), open a Models issue referencing `FEEDMODELS-SCHEMA-02-900` before implementing.

src/StellaOps.Feedser.Merge/Services/AdvisoryPrecedenceMerger.cs

@@ -31,10 +31,20 @@ public sealed class AdvisoryPrecedenceMerger
         unit: "count",
         description: "Number of affected-package range overrides performed during precedence merge.");
 
-    private static readonly Counter<long> ConflictCounter = MergeMeter.CreateCounter<long>(
-        "feedser.merge.conflicts",
-        unit: "count",
-        description: "Number of precedence conflicts detected (severity, rank ties, etc.).");
+    private static readonly Counter<long> ConflictCounter = MergeMeter.CreateCounter<long>(
+        "feedser.merge.conflicts",
+        unit: "count",
+        description: "Number of precedence conflicts detected (severity, rank ties, etc.).");
+
+    private static readonly Counter<long> NormalizedRuleCounter = MergeMeter.CreateCounter<long>(
+        "feedser.merge.normalized_rules",
+        unit: "rule",
+        description: "Number of normalized version rules retained after precedence merge.");
+
+    private static readonly Counter<long> MissingNormalizedRuleCounter = MergeMeter.CreateCounter<long>(
+        "feedser.merge.normalized_rules_missing",
+        unit: "package",
+        description: "Number of affected packages with version ranges but no normalized rules.");
 
     private static readonly Action<ILogger, MergeOverrideAudit, Exception?> OverrideLogged = LoggerMessage.Define<MergeOverrideAudit>(
         LogLevel.Information,
@@ -151,8 +161,9 @@ public sealed class AdvisoryPrecedenceMerger
             .Distinct()
             .ToArray();
 
-        var packageResult = _packageResolver.Merge(ordered.SelectMany(entry => entry.Advisory.AffectedPackages));
-        var affectedPackages = packageResult.Packages;
+        var packageResult = _packageResolver.Merge(ordered.SelectMany(entry => entry.Advisory.AffectedPackages));
+        RecordNormalizedRuleMetrics(packageResult.Packages);
+        var affectedPackages = packageResult.Packages;
         var cvssMetrics = ordered
             .SelectMany(entry => entry.Advisory.CvssMetrics)
             .Distinct()
@@ -186,13 +197,13 @@ public sealed class AdvisoryPrecedenceMerger
         LogPackageOverrides(advisoryKey, packageResult.Overrides);
         RecordFieldConflicts(advisoryKey, ordered);
 
-        return new Advisory(
-            advisoryKey,
-            title,
-            summary,
-            language,
-            published,
-            modified,
+        return new Advisory(
+            advisoryKey,
+            title,
+            summary,
+            language,
+            published,
+            modified,
             severity,
             exploitKnown,
             aliases,
@@ -201,13 +212,49 @@ public sealed class AdvisoryPrecedenceMerger
             affectedPackages,
             cvssMetrics,
             provenance);
-    }
-
-    private string? PickString(IEnumerable<AdvisoryEntry> ordered, Func<Advisory, string?> selector)
-    {
-        foreach (var entry in ordered)
-        {
-            var value = selector(entry.Advisory);
+    }
+
+    private static void RecordNormalizedRuleMetrics(IReadOnlyList<AffectedPackage> packages)
+    {
+        if (packages.Count == 0)
+        {
+            return;
+        }
+
+        foreach (var package in packages)
+        {
+            var packageType = package.Type ?? string.Empty;
+            var normalizedVersions = package.NormalizedVersions;
+            if (normalizedVersions.Length > 0)
+            {
+                foreach (var rule in normalizedVersions)
+                {
+                    var tags = new KeyValuePair<string, object?>[]
+                    {
+                        new("package_type", packageType),
+                        new("scheme", rule.Scheme ?? string.Empty),
+                    };
+
+                    NormalizedRuleCounter.Add(1, tags);
+                }
+            }
+            else if (package.VersionRanges.Length > 0)
+            {
+                var tags = new KeyValuePair<string, object?>[]
+                {
+                    new("package_type", packageType),
+                };
+
+                MissingNormalizedRuleCounter.Add(1, tags);
+            }
+        }
+    }
+
+    private string? PickString(IEnumerable<AdvisoryEntry> ordered, Func<Advisory, string?> selector)
+    {
+        foreach (var entry in ordered)
+        {
+            var value = selector(entry.Advisory);
             if (!string.IsNullOrWhiteSpace(value))
             {
                 return value.Trim();

src/StellaOps.Feedser.Merge/Services/AffectedPackagePrecedenceResolver.cs

@@ -60,15 +60,21 @@ public sealed class AffectedPackagePrecedenceResolver
                 .Distinct()
                 .ToImmutableArray();
 
-            var statuses = ordered
-                .SelectMany(static entry => entry.Package.Statuses)
-                .Distinct(AffectedPackageStatusEqualityComparer.Instance)
-                .ToImmutableArray();
-
-            foreach (var candidate in ordered.Skip(1))
-            {
-                if (candidate.Package.VersionRanges.Length == 0)
-                {
+            var statuses = ordered
+                .SelectMany(static entry => entry.Package.Statuses)
+                .Distinct(AffectedPackageStatusEqualityComparer.Instance)
+                .ToImmutableArray();
+
+            var normalizedRules = ordered
+                .SelectMany(static entry => entry.Package.NormalizedVersions)
+                .Distinct(NormalizedVersionRuleEqualityComparer.Instance)
+                .OrderBy(static rule => rule, NormalizedVersionRuleComparer.Instance)
+                .ToImmutableArray();
+
+            foreach (var candidate in ordered.Skip(1))
+            {
+                if (candidate.Package.VersionRanges.Length == 0)
+                {
                     continue;
                 }
 
@@ -84,16 +90,17 @@ public sealed class AffectedPackagePrecedenceResolver
                     candidate.Package.VersionRanges.Length));
             }
 
-            var merged = new AffectedPackage(
-                primary.Type,
-                primary.Identifier,
-                string.IsNullOrWhiteSpace(primary.Platform) ? null : primary.Platform,
-                primary.Package.VersionRanges,
-                statuses,
-                provenance);
-
-            resolved.Add(merged);
-        }
+            var merged = new AffectedPackage(
+                primary.Type,
+                primary.Identifier,
+                string.IsNullOrWhiteSpace(primary.Platform) ? null : primary.Platform,
+                primary.Package.VersionRanges,
+                statuses,
+                provenance,
+                normalizedRules);
+
+            resolved.Add(merged);
+        }
 
         var packagesResult = resolved
             .OrderBy(static pkg => pkg.Type, StringComparer.Ordinal)

src/StellaOps.Feedser.Merge/TASKS.md

@@ -15,4 +15,4 @@
 |FEEDMERGE-QA-04-001 End-to-end conflict regression suite|QA|Merge|DONE – `AdvisoryMergeServiceTests.MergeAsync_AppliesCanonicalRulesAndPersistsDecisions` exercises GHSA/NVD/OSV conflict path and merge-event analytics. **Reminder:** QA to sync with connector teams once new fixture triples land.|
 |Override audit logging|BE-Merge|Observability|DONE – override audits now emit structured logs plus bounded-tag metrics suitable for prod telemetry.|
 |Configurable precedence table|BE-Merge|Architecture|DONE – precedence options bind via feedser:merge:precedence:ranks with docs/tests covering operator workflow.|
-|Range primitives backlog|BE-Merge|Connector WGs|**DOING** – Coordinate remaining connectors (`Acsc`, `Cccs`, `CertBund`, `CertCc`, `Cve`, `Ghsa`, `Ics.Cisa`, `Kisa`, `Ru.Bdu`, `Ru.Nkcki`, `Vndr.Apple`, `Vndr.Cisco`, `Vndr.Msrc`) to emit canonical RangePrimitives with provenance tags; track progress/fixtures here.| 
+|Range primitives backlog|BE-Merge|Connector WGs|**DOING** – Coordinate remaining connectors (`Acsc`, `Cccs`, `CertBund`, `CertCc`, `Cve`, `Ghsa`, `Ics.Cisa`, `Kisa`, `Ru.Bdu`, `Ru.Nkcki`, `Vndr.Apple`, `Vndr.Cisco`, `Vndr.Msrc`) to emit canonical RangePrimitives with provenance tags; track progress/fixtures here.<br>2025-10-11: Storage alignment notes + sample normalized rule JSON now captured in `RANGE_PRIMITIVES_COORDINATION.md` (see “Storage alignment quick reference”).<br>2025-10-11 18:45Z: GHSA normalized rules landed; OSV connector picked up next for rollout.<br>2025-10-11 21:10Z: `docs/dev/merge_semver_playbook.md` Section 8 now documents the persisted Mongo projection (SemVer + NEVRA) for connector reviewers.<br>2025-10-11 21:30Z: Added `docs/dev/normalized_versions_rollout.md` dashboard to centralize connector status and upcoming milestones.<br>2025-10-11 21:55Z: Merge now emits `feedser.merge.normalized_rules*` counters and unions connector-provided normalized arrays; see new test coverage in `AdvisoryPrecedenceMergerTests.Merge_RecordsNormalizedRuleMetrics`.<br>2025-10-12 17:05Z: CVE + KEV normalized rule verification complete; OSV parity fixtures revalidated—downstream parity/monitoring tasks may proceed.|