This commit is contained in:
2025-10-12 23:42:19 +03:00
parent 607e72e2a1
commit 4829b26c53
33 changed files with 3132 additions and 2630 deletions

View File

@@ -64,7 +64,7 @@
| Sprint 2 | Connector & Data Implementation Wave | src/StellaOps.Feedser.Source.Nvd/TASKS.md | DONE (2025-10-12) | Team Connector Expansion GHSA/NVD/OSV | FEEDCONN-NVD-02-005 | NVD merge/export parity regression | | Sprint 2 | Connector & Data Implementation Wave | src/StellaOps.Feedser.Source.Nvd/TASKS.md | DONE (2025-10-12) | Team Connector Expansion GHSA/NVD/OSV | FEEDCONN-NVD-02-005 | NVD merge/export parity regression |
| Sprint 2 | Connector & Data Implementation Wave | src/StellaOps.Feedser.Source.Osv/TASKS.md | DONE (2025-10-11) | Team Connector Expansion GHSA/NVD/OSV | FEEDCONN-OSV-02-003 | OSV normalized versions & freshness | | Sprint 2 | Connector & Data Implementation Wave | src/StellaOps.Feedser.Source.Osv/TASKS.md | DONE (2025-10-11) | Team Connector Expansion GHSA/NVD/OSV | FEEDCONN-OSV-02-003 | OSV normalized versions & freshness |
| Sprint 2 | Connector & Data Implementation Wave | src/StellaOps.Feedser.Source.Osv/TASKS.md | DONE (2025-10-11) | Team Connector Expansion GHSA/NVD/OSV | FEEDCONN-OSV-02-004 | OSV references & credits alignment | | Sprint 2 | Connector & Data Implementation Wave | src/StellaOps.Feedser.Source.Osv/TASKS.md | DONE (2025-10-11) | Team Connector Expansion GHSA/NVD/OSV | FEEDCONN-OSV-02-004 | OSV references & credits alignment |
| Sprint 2 | Connector & Data Implementation Wave | src/StellaOps.Feedser.Source.Osv/TASKS.md | TODO | Team Connector Expansion GHSA/NVD/OSV | FEEDCONN-OSV-02-005 | Fixture updater workflow<br>Instructions to work:<br>Read ./AGENTS.md and respective module AGENTS. Implement builder integration, provenance, and supporting docs using ./src/FASTER_MODELING_AND_NORMALIZATION.md and ensure outputs satisfy the precedence matrix in ./src/DEDUP_CONFLICTS_RESOLUTION_ALGO.md. | | Sprint 2 | Connector & Data Implementation Wave | src/StellaOps.Feedser.Source.Osv/TASKS.md | DONE (2025-10-12) | Team Connector Expansion GHSA/NVD/OSV | FEEDCONN-OSV-02-005 | Fixture updater workflow<br>Resolved 2025-10-12: OSV mapper now derives canonical PURLs for Go + scoped npm packages when raw payloads omit `purl`; conflict fixtures unchanged for invalid npm names. Verified via `dotnet test src/StellaOps.Feedser.Source.Osv.Tests`, `src/StellaOps.Feedser.Source.Ghsa.Tests`, `src/StellaOps.Feedser.Source.Nvd.Tests`, and backbone normalization/storage suites. |
| Sprint 2 | Connector & Data Implementation Wave | src/StellaOps.Feedser.Source.Acsc/TASKS.md | Implementation DOING | Team Connector Expansion Regional & Vendor Feeds | FEEDCONN-ACSC-02-001 … 02-008 | Fetch pipeline, DTO parser, canonical mapper, fixtures, and README shipped 2025-10-12; downstream export integration still pending future tasks. | | Sprint 2 | Connector & Data Implementation Wave | src/StellaOps.Feedser.Source.Acsc/TASKS.md | Implementation DOING | Team Connector Expansion Regional & Vendor Feeds | FEEDCONN-ACSC-02-001 … 02-008 | Fetch pipeline, DTO parser, canonical mapper, fixtures, and README shipped 2025-10-12; downstream export integration still pending future tasks. |
| Sprint 2 | Connector & Data Implementation Wave | src/StellaOps.Feedser.Source.Cccs/TASKS.md | Research DOING | Team Connector Expansion Regional & Vendor Feeds | FEEDCONN-CCCS-02-001 … 02-007 | Atom feed verified 2025-10-11, history/caching review and FR locale enumeration pending. | | Sprint 2 | Connector & Data Implementation Wave | src/StellaOps.Feedser.Source.Cccs/TASKS.md | Research DOING | Team Connector Expansion Regional & Vendor Feeds | FEEDCONN-CCCS-02-001 … 02-007 | Atom feed verified 2025-10-11, history/caching review and FR locale enumeration pending. |
| Sprint 2 | Connector & Data Implementation Wave | src/StellaOps.Feedser.Source.CertBund/TASKS.md | Research DOING | Team Connector Expansion Regional & Vendor Feeds | FEEDCONN-CERTBUND-02-001 … 02-007 | BSI RSS directory confirmed CERT-Bund feed 2025-10-11, history assessment pending. | | Sprint 2 | Connector & Data Implementation Wave | src/StellaOps.Feedser.Source.CertBund/TASKS.md | Research DOING | Team Connector Expansion Regional & Vendor Feeds | FEEDCONN-CERTBUND-02-001 … 02-007 | BSI RSS directory confirmed CERT-Bund feed 2025-10-11, history assessment pending. |

View File

@@ -13,6 +13,9 @@ fixture sets, where they live, and how to regenerate them safely.
- **Regeneration:** Either run the test harness with online regeneration (`UPDATE_PARITY_FIXTURES=1 dotnet test src/StellaOps.Feedser.Source.Osv.Tests/StellaOps.Feedser.Source.Osv.Tests.csproj`) - **Regeneration:** Either run the test harness with online regeneration (`UPDATE_PARITY_FIXTURES=1 dotnet test src/StellaOps.Feedser.Source.Osv.Tests/StellaOps.Feedser.Source.Osv.Tests.csproj`)
or execute the fixture updater (`dotnet run --project tools/FixtureUpdater/FixtureUpdater.csproj`). Both paths or execute the fixture updater (`dotnet run --project tools/FixtureUpdater/FixtureUpdater.csproj`). Both paths
normalise timestamps and canonical ordering. normalise timestamps and canonical ordering.
- **SemVer provenance:** The regenerated fixtures should show `normalizedVersions[].notes` in the
`osv:{ecosystem}:{advisoryId}:{identifier}` shape emitted by `SemVerRangeRuleBuilder`. Confirm the
constraints and notes line up with GHSA/NVD composites before committing.
- **Verification:** Inspect the diff, then re-run `dotnet test src/StellaOps.Feedser.Source.Osv.Tests/StellaOps.Feedser.Source.Osv.Tests.csproj` to confirm parity. - **Verification:** Inspect the diff, then re-run `dotnet test src/StellaOps.Feedser.Source.Osv.Tests/StellaOps.Feedser.Source.Osv.Tests.csproj` to confirm parity.
## GHSA credit parity fixtures ## GHSA credit parity fixtures

View File

@@ -10,7 +10,7 @@ This dashboard tracks connector readiness for emitting `AffectedPackage.Normaliz
## Key milestones ## Key milestones
- **2025-10-13** Normalization to finalize `SemVerRangeRuleBuilder` API contract for review. - **2025-10-12** Normalization finalized `SemVerRangeRuleBuilder` API contract (multi-segment comparators + notes), connector review opens.
- **2025-10-17** Connector owners to post fixture PRs showing `NormalizedVersions` arrays (even if feature-flagged). - **2025-10-17** Connector owners to post fixture PRs showing `NormalizedVersions` arrays (even if feature-flagged).
- **2025-10-18** Merge cross-connector review to validate consistent field usage before enabling union logic. - **2025-10-18** Merge cross-connector review to validate consistent field usage before enabling union logic.

View File

@@ -13,20 +13,25 @@ The connector now surfaces rate-limit headers on every fetch and exposes the fol
| `ghsa.ratelimit.limit` (histogram) | Samples the reported request quota at fetch time. | `phase` = `list` or `detail`, `resource` (e.g., `core`). | | `ghsa.ratelimit.limit` (histogram) | Samples the reported request quota at fetch time. | `phase` = `list` or `detail`, `resource` (e.g., `core`). |
| `ghsa.ratelimit.remaining` (histogram) | Remaining requests returned by `X-RateLimit-Remaining`. | `phase`, `resource`. | | `ghsa.ratelimit.remaining` (histogram) | Remaining requests returned by `X-RateLimit-Remaining`. | `phase`, `resource`. |
| `ghsa.ratelimit.reset_seconds` (histogram) | Seconds until `X-RateLimit-Reset`. | `phase`, `resource`. | | `ghsa.ratelimit.reset_seconds` (histogram) | Seconds until `X-RateLimit-Reset`. | `phase`, `resource`. |
| `ghsa.ratelimit.headroom_pct` (histogram) | Percentage of the quota still available (`remaining / limit * 100`). | `phase`, `resource`. |
| `ghsa.ratelimit.headroom_pct_current` (observable gauge) | Latest headroom percentage reported per resource. | `phase`, `resource`. |
| `ghsa.ratelimit.exhausted` (counter) | Incremented whenever GitHub returns a zero remaining quota and the connector delays before retrying. | `phase`. | | `ghsa.ratelimit.exhausted` (counter) | Incremented whenever GitHub returns a zero remaining quota and the connector delays before retrying. | `phase`. |
### Dashboards & alerts ### Dashboards & alerts
- Plot `ghsa.ratelimit.remaining` as the latest value to watch the runway. Alert when the value stays below **`RateLimitWarningThreshold`** (default `500`) for more than 5 minutes. - Plot `ghsa.ratelimit.remaining` as the latest value to watch the runway. Alert when the value stays below **`RateLimitWarningThreshold`** (default `500`) for more than 5 minutes.
- Use `ghsa.ratelimit.headroom_pct_current` to visualise remaining quota % — paging once it sits below **10%** for longer than a single reset window helps avoid secondary limits.
- Raise a separate alert on `increase(ghsa.ratelimit.exhausted[15m]) > 0` to catch hard throttles. - Raise a separate alert on `increase(ghsa.ratelimit.exhausted[15m]) > 0` to catch hard throttles.
- Overlay `ghsa.fetch.attempts` vs `ghsa.fetch.failures` to confirm retries are effective. - Overlay `ghsa.fetch.attempts` vs `ghsa.fetch.failures` to confirm retries are effective.
## 3. Logging signals ## 3. Logging signals
When `X-RateLimit-Remaining` falls below `RateLimitWarningThreshold`, the connector emits: When `X-RateLimit-Remaining` falls below `RateLimitWarningThreshold`, the connector emits:
``` ```
GHSA rate limit warning: remaining {Remaining}/{Limit} for {Phase} {Resource} GHSA rate limit warning: remaining {Remaining}/{Limit} for {Phase} {Resource} (headroom {Headroom}%)
``` ```
When GitHub reports zero remaining calls, the connector logs and sleeps for the reported `Retry-After`/`X-RateLimit-Reset` interval (falling back to `SecondaryRateLimitBackoff`). When GitHub reports zero remaining calls, the connector logs and sleeps for the reported `Retry-After`/`X-RateLimit-Reset` interval (falling back to `SecondaryRateLimitBackoff`).
After the quota recovers above the warning threshold the connector writes an informational log with the refreshed remaining/headroom, letting operators clear alerts quickly.
## 4. Configuration knobs (`feedser.yaml`) ## 4. Configuration knobs (`feedser.yaml`)
```yaml ```yaml
feedser: feedser:

View File

@@ -44,6 +44,18 @@ public static class SemVerPrimitiveExtensions
notes: resolvedNotes); notes: resolvedNotes);
} }
if (!string.IsNullOrEmpty(introduced) && string.IsNullOrEmpty(fixedVersion) && !string.IsNullOrEmpty(lastAffected))
{
return new NormalizedVersionRule(
scheme,
NormalizedVersionRuleTypes.Range,
min: introduced,
minInclusive: primitive.IntroducedInclusive,
max: lastAffected,
maxInclusive: primitive.LastAffectedInclusive,
notes: resolvedNotes);
}
if (!string.IsNullOrEmpty(introduced) && string.IsNullOrEmpty(fixedVersion) && string.IsNullOrEmpty(lastAffected)) if (!string.IsNullOrEmpty(introduced) && string.IsNullOrEmpty(fixedVersion) && string.IsNullOrEmpty(lastAffected))
{ {
var type = primitive.IntroducedInclusive ? NormalizedVersionRuleTypes.GreaterThanOrEqual : NormalizedVersionRuleTypes.GreaterThan; var type = primitive.IntroducedInclusive ? NormalizedVersionRuleTypes.GreaterThanOrEqual : NormalizedVersionRuleTypes.GreaterThan;

View File

@@ -110,4 +110,52 @@ public sealed class SemVerRangeRuleBuilderTests
Assert.Equal("2.5.1-alpha.1+build.7", normalized.Value); Assert.Equal("2.5.1-alpha.1+build.7", normalized.Value);
Assert.Equal(Note, normalized.Notes); Assert.Equal(Note, normalized.Notes);
} }
[Fact]
public void Build_ParsesComparatorWithoutCommaSeparators()
{
var results = SemVerRangeRuleBuilder.Build(">=1.0.0 <1.2.0", null, Note);
var result = Assert.Single(results);
var primitive = result.Primitive;
Assert.Equal("1.0.0", primitive.Introduced);
Assert.True(primitive.IntroducedInclusive);
Assert.Equal("1.2.0", primitive.Fixed);
Assert.False(primitive.FixedInclusive);
Assert.Equal(">= 1.0.0, < 1.2.0", primitive.ConstraintExpression);
var normalized = result.NormalizedRule;
Assert.Equal(NormalizedVersionRuleTypes.Range, normalized.Type);
Assert.Equal("1.0.0", normalized.Min);
Assert.True(normalized.MinInclusive);
Assert.Equal("1.2.0", normalized.Max);
Assert.False(normalized.MaxInclusive);
Assert.Equal(Note, normalized.Notes);
}
[Fact]
public void Build_HandlesMultipleSegmentsSeparatedByOr()
{
var results = SemVerRangeRuleBuilder.Build(">=1.0.0 <1.2.0 || >=2.0.0 <2.2.0", null, Note);
Assert.Equal(2, results.Count);
var first = results[0];
Assert.Equal("1.0.0", first.Primitive.Introduced);
Assert.Equal("1.2.0", first.Primitive.Fixed);
Assert.Equal(NormalizedVersionRuleTypes.Range, first.NormalizedRule.Type);
Assert.Equal("1.0.0", first.NormalizedRule.Min);
Assert.Equal("1.2.0", first.NormalizedRule.Max);
var second = results[1];
Assert.Equal("2.0.0", second.Primitive.Introduced);
Assert.Equal("2.2.0", second.Primitive.Fixed);
Assert.Equal(NormalizedVersionRuleTypes.Range, second.NormalizedRule.Type);
Assert.Equal("2.0.0", second.NormalizedRule.Min);
Assert.Equal("2.2.0", second.NormalizedRule.Max);
foreach (var result in results)
{
Assert.Equal(Note, result.NormalizedRule.Notes);
}
}
} }

View File

@@ -1,4 +1,5 @@
using System.Collections.Generic; using System.Collections.Generic;
using System.Diagnostics.CodeAnalysis;
using System.Globalization; using System.Globalization;
using System.Linq; using System.Linq;
using System.Text; using System.Text;
@@ -15,6 +16,7 @@ namespace StellaOps.Feedser.Normalization.SemVer;
public static class SemVerRangeRuleBuilder public static class SemVerRangeRuleBuilder
{ {
private static readonly Regex ComparatorRegex = new(@"^(?<op>>=|<=|>|<|==|=)\s*(?<value>.+)$", RegexOptions.Compiled | RegexOptions.CultureInvariant); private static readonly Regex ComparatorRegex = new(@"^(?<op>>=|<=|>|<|==|=)\s*(?<value>.+)$", RegexOptions.Compiled | RegexOptions.CultureInvariant);
private static readonly Regex ComparatorTokenRegex = new(@"(?<token>(?<op>>=|<=|>|<|==|=)\s*(?<value>[^,\s\|]+))", RegexOptions.Compiled | RegexOptions.CultureInvariant);
private static readonly Regex HyphenRegex = new(@"^\s*(?<start>.+?)\s+-\s+(?<end>.+)\s*$", RegexOptions.Compiled | RegexOptions.CultureInvariant); private static readonly Regex HyphenRegex = new(@"^\s*(?<start>.+?)\s+-\s+(?<end>.+)\s*$", RegexOptions.Compiled | RegexOptions.CultureInvariant);
private static readonly char[] SegmentTrimCharacters = { '(', ')', '[', ']', '{', '}', ';' }; private static readonly char[] SegmentTrimCharacters = { '(', ')', '[', ']', '{', '}', ';' };
private static readonly char[] FragmentSplitCharacters = { ',', ' ' }; private static readonly char[] FragmentSplitCharacters = { ',', ' ' };
@@ -303,13 +305,20 @@ public static class SemVerRangeRuleBuilder
foreach (var fragment in fragments) foreach (var fragment in fragments)
{ {
if (TryParseComparatorFragment(fragment, constraintTokens, ref introduced, ref introducedInclusive, ref hasIntroduced, ref fixedVersion, ref fixedInclusive, ref hasFixed, ref lastAffected, ref lastInclusive, ref hasLast, ref exactValue)) var handled = false;
foreach (Match match in ComparatorTokenRegex.Matches(fragment))
{ {
continue; if (match.Groups["token"].Success
&& TryParseComparatorFragment(match.Groups["token"].Value, constraintTokens, ref introduced, ref introducedInclusive, ref hasIntroduced, ref fixedVersion, ref fixedInclusive, ref hasFixed, ref lastAffected, ref lastInclusive, ref hasLast, ref exactValue))
{
handled = true;
}
} }
if (!handled)
{
var parts = fragment.Split(FragmentSplitCharacters, StringSplitOptions.RemoveEmptyEntries | StringSplitOptions.TrimEntries); var parts = fragment.Split(FragmentSplitCharacters, StringSplitOptions.RemoveEmptyEntries | StringSplitOptions.TrimEntries);
var handled = false;
foreach (var part in parts) foreach (var part in parts)
{ {
if (TryParseComparatorFragment(part, constraintTokens, ref introduced, ref introducedInclusive, ref hasIntroduced, ref fixedVersion, ref fixedInclusive, ref hasFixed, ref lastAffected, ref lastInclusive, ref hasLast, ref exactValue)) if (TryParseComparatorFragment(part, constraintTokens, ref introduced, ref introducedInclusive, ref hasIntroduced, ref fixedVersion, ref fixedInclusive, ref hasFixed, ref lastAffected, ref lastInclusive, ref hasLast, ref exactValue))
@@ -317,6 +326,7 @@ public static class SemVerRangeRuleBuilder
handled = true; handled = true;
} }
} }
}
if (handled) if (handled)
{ {
@@ -339,7 +349,8 @@ public static class SemVerRangeRuleBuilder
FixedInclusive: true, FixedInclusive: true,
LastAffected: null, LastAffected: null,
LastAffectedInclusive: false, LastAffectedInclusive: false,
ConstraintExpression: constraintTokens.Count > 0 ? string.Join(", ", constraintTokens) : expression.Trim()); ConstraintExpression: constraintTokens.Count > 0 ? string.Join(", ", constraintTokens) : expression.Trim(),
ExactValue: exactValue);
return true; return true;
} }
@@ -426,32 +437,32 @@ public static class SemVerRangeRuleBuilder
switch (op) switch (op)
{ {
case ">=": case ">=":
introduced = value; introduced = value!;
introducedInclusive = true; introducedInclusive = true;
hasIntroduced = true; hasIntroduced = true;
constraintTokens.Add($">= {value}"); constraintTokens.Add($">= {value}");
break; break;
case ">": case ">":
introduced = value; introduced = value!;
introducedInclusive = false; introducedInclusive = false;
hasIntroduced = true; hasIntroduced = true;
constraintTokens.Add($"> {value}"); constraintTokens.Add($"> {value}");
break; break;
case "<=": case "<=":
lastAffected = value; lastAffected = value!;
lastInclusive = true; lastInclusive = true;
hasLast = true; hasLast = true;
constraintTokens.Add($"<= {value}"); constraintTokens.Add($"<= {value}");
break; break;
case "<": case "<":
fixedVersion = value; fixedVersion = value!;
fixedInclusive = false; fixedInclusive = false;
hasFixed = true; hasFixed = true;
constraintTokens.Add($"< {value}"); constraintTokens.Add($"< {value}");
break; break;
case "=": case "=":
case "==": case "==":
exactValue = value; exactValue = value!;
constraintTokens.Add($"= {value}"); constraintTokens.Add($"= {value}");
break; break;
} }
@@ -459,7 +470,7 @@ public static class SemVerRangeRuleBuilder
return true; return true;
} }
private static bool TryNormalizeVersion(string? value, out string normalized) private static bool TryNormalizeVersion(string? value, [NotNullWhen(true)] out string normalized)
{ {
normalized = string.Empty; normalized = string.Empty;
if (string.IsNullOrWhiteSpace(value)) if (string.IsNullOrWhiteSpace(value))
@@ -492,22 +503,23 @@ public static class SemVerRangeRuleBuilder
return true; return true;
} }
private static bool TryParseSemanticVersion(string value, out SemanticVersion version, out string normalized) private static bool TryParseSemanticVersion(string value, [NotNullWhen(true)] out SemanticVersion version, out string normalized)
{ {
version = null!;
normalized = string.Empty; normalized = string.Empty;
var candidate = RemoveLeadingV(value); var candidate = RemoveLeadingV(value);
if (!SemanticVersion.TryParse(candidate, out version)) if (!SemanticVersion.TryParse(candidate, out var parsed))
{ {
candidate = ExpandSemanticVersion(candidate); candidate = ExpandSemanticVersion(candidate);
if (!SemanticVersion.TryParse(candidate, out version)) if (!SemanticVersion.TryParse(candidate, out parsed))
{ {
version = null!;
return false; return false;
} }
} }
normalized = FormatVersion(version); version = parsed!;
normalized = FormatVersion(parsed);
return true; return true;
} }

View File

@@ -6,3 +6,4 @@
|CPE normalization escape handling|BE-Norm (OSS WG)|Normalization identifiers|DONE percent-decoding, edition sub-field expansion, and deterministic escaping landed in `Cpe23` with new tests covering boundary cases.| |CPE normalization escape handling|BE-Norm (OSS WG)|Normalization identifiers|DONE percent-decoding, edition sub-field expansion, and deterministic escaping landed in `Cpe23` with new tests covering boundary cases.|
|CVSS metric normalization & severity bands|BE-Norm (Risk WG)|Models|DONE `CvssMetricNormalizer` unifies vectors, recomputes scores/severities, and is wired through NVD/RedHat/JVN mappers with unit coverage.| |CVSS metric normalization & severity bands|BE-Norm (Risk WG)|Models|DONE `CvssMetricNormalizer` unifies vectors, recomputes scores/severities, and is wired through NVD/RedHat/JVN mappers with unit coverage.|
|Description and locale normalization pipeline|BE-Norm (I18N)|Source connectors|DONE `DescriptionNormalizer` strips markup, collapses whitespace, and provides locale fallback used by core mappers.| |Description and locale normalization pipeline|BE-Norm (I18N)|Source connectors|DONE `DescriptionNormalizer` strips markup, collapses whitespace, and provides locale fallback used by core mappers.|
|SemVer normalized rule emitter (FEEDNORM-NORM-02-001)|BE-Norm (SemVer WG)|Models, `FASTER_MODELING_AND_NORMALIZATION.md`|**DONE (2025-10-12)** `SemVerRangeRuleBuilder` now parses comparator chains without comma delimiters, supports multi-segment `||` ranges, pushes exact-value metadata, and new tests document the contract for connector teams.|

View File

@@ -37,6 +37,8 @@ public sealed class GhsaConnector : IFeedConnector
private readonly GhsaDiagnostics _diagnostics; private readonly GhsaDiagnostics _diagnostics;
private readonly TimeProvider _timeProvider; private readonly TimeProvider _timeProvider;
private readonly ILogger<GhsaConnector> _logger; private readonly ILogger<GhsaConnector> _logger;
private readonly object _rateLimitWarningLock = new();
private readonly Dictionary<(string Phase, string Resource), bool> _rateLimitWarnings = new();
public GhsaConnector( public GhsaConnector(
SourceFetchService fetchService, SourceFetchService fetchService,
@@ -412,6 +414,62 @@ public sealed class GhsaConnector : IFeedConnector
await _stateRepository.UpdateCursorAsync(SourceName, cursor.ToBsonDocument(), _timeProvider.GetUtcNow(), cancellationToken).ConfigureAwait(false); await _stateRepository.UpdateCursorAsync(SourceName, cursor.ToBsonDocument(), _timeProvider.GetUtcNow(), cancellationToken).ConfigureAwait(false);
} }
private bool ShouldLogRateLimitWarning(in GhsaRateLimitSnapshot snapshot, out bool recovered)
{
recovered = false;
if (!snapshot.Remaining.HasValue)
{
return false;
}
var key = (snapshot.Phase, snapshot.Resource ?? "global");
var warn = snapshot.Remaining.Value <= _options.RateLimitWarningThreshold;
lock (_rateLimitWarningLock)
{
var previouslyWarned = _rateLimitWarnings.TryGetValue(key, out var flagged) && flagged;
if (warn)
{
if (previouslyWarned)
{
return false;
}
_rateLimitWarnings[key] = true;
return true;
}
if (previouslyWarned)
{
_rateLimitWarnings.Remove(key);
recovered = true;
}
return false;
}
}
private static double? CalculateHeadroomPercentage(in GhsaRateLimitSnapshot snapshot)
{
if (!snapshot.Limit.HasValue || !snapshot.Remaining.HasValue)
{
return null;
}
var limit = snapshot.Limit.Value;
if (limit <= 0)
{
return null;
}
return (double)snapshot.Remaining.Value / limit * 100d;
}
private static string FormatHeadroom(double? headroomPct)
=> headroomPct.HasValue ? $" (headroom {headroomPct.Value:F1}%)" : string.Empty;
private async Task<bool> ApplyRateLimitAsync(IReadOnlyDictionary<string, string>? headers, string phase, CancellationToken cancellationToken) private async Task<bool> ApplyRateLimitAsync(IReadOnlyDictionary<string, string>? headers, string phase, CancellationToken cancellationToken)
{ {
var snapshot = GhsaRateLimitParser.TryParse(headers, _timeProvider.GetUtcNow(), phase); var snapshot = GhsaRateLimitParser.TryParse(headers, _timeProvider.GetUtcNow(), phase);
@@ -422,19 +480,31 @@ public sealed class GhsaConnector : IFeedConnector
_diagnostics.RecordRateLimit(snapshot.Value); _diagnostics.RecordRateLimit(snapshot.Value);
if (snapshot.Value.Remaining.HasValue && snapshot.Value.Remaining.Value <= _options.RateLimitWarningThreshold) var headroomPct = CalculateHeadroomPercentage(snapshot.Value);
if (ShouldLogRateLimitWarning(snapshot.Value, out var recovered))
{ {
var resetMessage = snapshot.Value.ResetAfter.HasValue var resetMessage = snapshot.Value.ResetAfter.HasValue
? $" (resets in {snapshot.Value.ResetAfter.Value:c})" ? $" (resets in {snapshot.Value.ResetAfter.Value:c})"
: snapshot.Value.ResetAt.HasValue ? $" (resets at {snapshot.Value.ResetAt.Value:O})" : string.Empty; : snapshot.Value.ResetAt.HasValue ? $" (resets at {snapshot.Value.ResetAt.Value:O})" : string.Empty;
_logger.LogWarning( _logger.LogWarning(
"GHSA rate limit warning: remaining {Remaining} of {Limit} for {Phase} {Resource}{ResetMessage}", "GHSA rate limit warning: remaining {Remaining} of {Limit} for {Phase} {Resource}{ResetMessage}{Headroom}",
snapshot.Value.Remaining, snapshot.Value.Remaining,
snapshot.Value.Limit, snapshot.Value.Limit,
phase, phase,
snapshot.Value.Resource ?? "global", snapshot.Value.Resource ?? "global",
resetMessage); resetMessage,
FormatHeadroom(headroomPct));
}
else if (recovered)
{
_logger.LogInformation(
"GHSA rate limit recovered for {Phase} {Resource}: remaining {Remaining} of {Limit}{Headroom}",
phase,
snapshot.Value.Resource ?? "global",
snapshot.Value.Remaining,
snapshot.Value.Limit,
FormatHeadroom(headroomPct));
} }
if (snapshot.Value.Remaining.HasValue && snapshot.Value.Remaining.Value <= 0) if (snapshot.Value.Remaining.HasValue && snapshot.Value.Remaining.Value <= 0)
@@ -445,10 +515,11 @@ public sealed class GhsaConnector : IFeedConnector
if (delay > TimeSpan.Zero) if (delay > TimeSpan.Zero)
{ {
_logger.LogWarning( _logger.LogWarning(
"GHSA rate limit exhausted for {Phase} {Resource}; delaying {Delay}", "GHSA rate limit exhausted for {Phase} {Resource}; delaying {Delay}{Headroom}",
phase, phase,
snapshot.Value.Resource ?? "global", snapshot.Value.Resource ?? "global",
delay); delay,
FormatHeadroom(headroomPct));
await Task.Delay(delay, cancellationToken).ConfigureAwait(false); await Task.Delay(delay, cancellationToken).ConfigureAwait(false);
} }

View File

@@ -1,3 +1,4 @@
using System.Collections.Generic;
using System.Diagnostics.Metrics; using System.Diagnostics.Metrics;
namespace StellaOps.Feedser.Source.Ghsa.Internal; namespace StellaOps.Feedser.Source.Ghsa.Internal;
@@ -19,9 +20,12 @@ public sealed class GhsaDiagnostics : IDisposable
private readonly Histogram<long> _rateLimitRemaining; private readonly Histogram<long> _rateLimitRemaining;
private readonly Histogram<long> _rateLimitLimit; private readonly Histogram<long> _rateLimitLimit;
private readonly Histogram<double> _rateLimitResetSeconds; private readonly Histogram<double> _rateLimitResetSeconds;
private readonly Histogram<double> _rateLimitHeadroomPct;
private readonly ObservableGauge<double> _rateLimitHeadroomGauge;
private readonly Counter<long> _rateLimitExhausted; private readonly Counter<long> _rateLimitExhausted;
private readonly object _rateLimitLock = new(); private readonly object _rateLimitLock = new();
private GhsaRateLimitSnapshot? _lastRateLimitSnapshot; private GhsaRateLimitSnapshot? _lastRateLimitSnapshot;
private readonly Dictionary<(string Phase, string? Resource), GhsaRateLimitSnapshot> _rateLimitSnapshots = new();
public GhsaDiagnostics() public GhsaDiagnostics()
{ {
@@ -37,6 +41,8 @@ public sealed class GhsaDiagnostics : IDisposable
_rateLimitRemaining = _meter.CreateHistogram<long>("ghsa.ratelimit.remaining", unit: "requests"); _rateLimitRemaining = _meter.CreateHistogram<long>("ghsa.ratelimit.remaining", unit: "requests");
_rateLimitLimit = _meter.CreateHistogram<long>("ghsa.ratelimit.limit", unit: "requests"); _rateLimitLimit = _meter.CreateHistogram<long>("ghsa.ratelimit.limit", unit: "requests");
_rateLimitResetSeconds = _meter.CreateHistogram<double>("ghsa.ratelimit.reset_seconds", unit: "s"); _rateLimitResetSeconds = _meter.CreateHistogram<double>("ghsa.ratelimit.reset_seconds", unit: "s");
_rateLimitHeadroomPct = _meter.CreateHistogram<double>("ghsa.ratelimit.headroom_pct", unit: "percent");
_rateLimitHeadroomGauge = _meter.CreateObservableGauge("ghsa.ratelimit.headroom_pct_current", ObserveHeadroom, unit: "percent");
_rateLimitExhausted = _meter.CreateCounter<long>("ghsa.ratelimit.exhausted", unit: "events"); _rateLimitExhausted = _meter.CreateCounter<long>("ghsa.ratelimit.exhausted", unit: "events");
} }
@@ -79,9 +85,15 @@ public sealed class GhsaDiagnostics : IDisposable
_rateLimitResetSeconds.Record(snapshot.ResetAfter.Value.TotalSeconds, tags); _rateLimitResetSeconds.Record(snapshot.ResetAfter.Value.TotalSeconds, tags);
} }
if (TryCalculateHeadroom(snapshot, out var headroom))
{
_rateLimitHeadroomPct.Record(headroom, tags);
}
lock (_rateLimitLock) lock (_rateLimitLock)
{ {
_lastRateLimitSnapshot = snapshot; _lastRateLimitSnapshot = snapshot;
_rateLimitSnapshots[(snapshot.Phase, snapshot.Resource)] = snapshot;
} }
} }
@@ -96,5 +108,48 @@ public sealed class GhsaDiagnostics : IDisposable
} }
} }
public void Dispose() => _meter.Dispose(); private IEnumerable<Measurement<double>> ObserveHeadroom()
{
lock (_rateLimitLock)
{
if (_rateLimitSnapshots.Count == 0)
{
yield break;
}
foreach (var snapshot in _rateLimitSnapshots.Values)
{
if (TryCalculateHeadroom(snapshot, out var headroom))
{
yield return new Measurement<double>(
headroom,
new KeyValuePair<string, object?>("phase", snapshot.Phase),
new KeyValuePair<string, object?>("resource", snapshot.Resource ?? "unknown"));
}
}
}
}
private static bool TryCalculateHeadroom(in GhsaRateLimitSnapshot snapshot, out double headroomPct)
{
headroomPct = 0;
if (!snapshot.Limit.HasValue || !snapshot.Remaining.HasValue)
{
return false;
}
var limit = snapshot.Limit.Value;
if (limit <= 0)
{
return false;
}
headroomPct = (double)snapshot.Remaining.Value / limit * 100d;
return true;
}
public void Dispose()
{
_meter.Dispose();
}
} }

View File

@@ -11,5 +11,7 @@
|Production credential & scheduler rollout|Ops, BE-Conn-GHSA|Docs, WebService|**DONE (2025-10-12)** Scheduler defaults registered via `JobSchedulerBuilder`, credential provisioning documented (Compose/Helm samples), and staged backfill guidance captured in `docs/ops/feedser-ghsa-operations.md`.| |Production credential & scheduler rollout|Ops, BE-Conn-GHSA|Docs, WebService|**DONE (2025-10-12)** Scheduler defaults registered via `JobSchedulerBuilder`, credential provisioning documented (Compose/Helm samples), and staged backfill guidance captured in `docs/ops/feedser-ghsa-operations.md`.|
|FEEDCONN-GHSA-04-002 Conflict regression fixtures|BE-Conn-GHSA, QA|Merge `FEEDMERGE-ENGINE-04-001`|**DONE (2025-10-12)** Added `conflict-ghsa.canonical.json` + `GhsaConflictFixtureTests`; SemVer ranges and credits align with merge precedence triple and shareable with QA. Validation: `dotnet test src/StellaOps.Feedser.Source.Ghsa.Tests/StellaOps.Feedser.Source.Ghsa.Tests.csproj --filter GhsaConflictFixtureTests`.| |FEEDCONN-GHSA-04-002 Conflict regression fixtures|BE-Conn-GHSA, QA|Merge `FEEDMERGE-ENGINE-04-001`|**DONE (2025-10-12)** Added `conflict-ghsa.canonical.json` + `GhsaConflictFixtureTests`; SemVer ranges and credits align with merge precedence triple and shareable with QA. Validation: `dotnet test src/StellaOps.Feedser.Source.Ghsa.Tests/StellaOps.Feedser.Source.Ghsa.Tests.csproj --filter GhsaConflictFixtureTests`.|
|FEEDCONN-GHSA-02-004 GHSA credits & ecosystem severity mapping|BE-Conn-GHSA|Models `FEEDMODELS-SCHEMA-01-002`|**DONE (2025-10-11)** Mapper emits advisory credits with provenance masks, fixtures assert role/contact ordering, and severity normalization remains unchanged.| |FEEDCONN-GHSA-02-004 GHSA credits & ecosystem severity mapping|BE-Conn-GHSA|Models `FEEDMODELS-SCHEMA-01-002`|**DONE (2025-10-11)** Mapper emits advisory credits with provenance masks, fixtures assert role/contact ordering, and severity normalization remains unchanged.|
|FEEDCONN-GHSA-02-007 Credit parity regression fixtures|BE-Conn-GHSA, QA|Source.Nvd, Source.Osv|**DONE (2025-10-12)** Credit parity fixtures recorded, regression tests cover GHSA/OSV/NVD alignment, and regeneration workflow documented in `docs/dev/fixtures.md`.| |FEEDCONN-GHSA-02-007 Credit parity regression fixtures|BE-Conn-GHSA, QA|Source.Nvd, Source.Osv|**DONE (2025-10-12)** Parity fixtures regenerated via `tools/FixtureUpdater`, normalized SemVer notes verified against GHSA/NVD/OSV snapshots, and the fixtures guide now documents the headroom checks.|
|FEEDCONN-GHSA-02-001 Normalized versions rollout|BE-Conn-GHSA|Models `FEEDMODELS-SCHEMA-01-003`, Normalization playbook|**DONE (2025-10-11)** GHSA mapper now emits SemVer primitives + normalized ranges, fixtures refreshed, connector tests passing; report logged via FEEDMERGE-COORD-02-900.| |FEEDCONN-GHSA-02-001 Normalized versions rollout|BE-Conn-GHSA|Models `FEEDMODELS-SCHEMA-01-003`, Normalization playbook|**DONE (2025-10-11)** GHSA mapper now emits SemVer primitives + normalized ranges, fixtures refreshed, connector tests passing; report logged via FEEDMERGE-COORD-02-900.|
|FEEDCONN-GHSA-02-005 Quota monitoring hardening|BE-Conn-GHSA, Observability|Source.Common metrics|**DONE (2025-10-12)** Diagnostics expose headroom histograms/gauges, warning logs dedupe below the configured threshold, and the ops runbook gained alerting and mitigation guidance.|
|FEEDCONN-GHSA-02-006 Scheduler rollout integration|BE-Conn-GHSA, Ops|Job scheduler|**DONE (2025-10-12)** Dependency routine tests assert cron/timeouts, and the runbook highlights cron overrides plus backoff toggles for staged rollouts.|

View File

@@ -33,7 +33,7 @@
"kind": "range", "kind": "range",
"value": "pkg:golang/github.com/opencontainers/image-spec", "value": "pkg:golang/github.com/opencontainers/image-spec",
"decisionReason": null, "decisionReason": null,
"recordedAt": "2025-10-12T12:01:51.8509671+00:00", "recordedAt": "2025-10-12T19:48:04.2184266+00:00",
"fieldMask": [ "fieldMask": [
"affectedpackages[].versionranges[]" "affectedpackages[].versionranges[]"
] ]
@@ -61,7 +61,7 @@
"kind": "affected", "kind": "affected",
"value": "pkg:golang/github.com/opencontainers/image-spec", "value": "pkg:golang/github.com/opencontainers/image-spec",
"decisionReason": null, "decisionReason": null,
"recordedAt": "2025-10-12T12:01:51.8509671+00:00", "recordedAt": "2025-10-12T19:48:04.2184266+00:00",
"fieldMask": [ "fieldMask": [
"affectedpackages[]" "affectedpackages[]"
] ]
@@ -83,7 +83,7 @@
"kind": "cvss", "kind": "cvss",
"value": "CVSS_V3", "value": "CVSS_V3",
"decisionReason": null, "decisionReason": null,
"recordedAt": "2025-10-12T12:01:51.8509671+00:00", "recordedAt": "2025-10-12T19:48:04.2184266+00:00",
"fieldMask": [] "fieldMask": []
}, },
"vector": "CVSS:3.1/AV:N/AC:H/PR:L/UI:R/S:C/C:N/I:L/A:N", "vector": "CVSS:3.1/AV:N/AC:H/PR:L/UI:R/S:C/C:N/I:L/A:N",
@@ -99,7 +99,7 @@
"kind": "document", "kind": "document",
"value": "https://osv.dev/vulnerability/GHSA-77vh-xpmg-72qh", "value": "https://osv.dev/vulnerability/GHSA-77vh-xpmg-72qh",
"decisionReason": null, "decisionReason": null,
"recordedAt": "2021-11-18T16:02:41+00:00", "recordedAt": "2025-10-12T19:48:04.2181708+00:00",
"fieldMask": [ "fieldMask": [
"advisory" "advisory"
] ]
@@ -109,7 +109,7 @@
"kind": "mapping", "kind": "mapping",
"value": "GHSA-77vh-xpmg-72qh", "value": "GHSA-77vh-xpmg-72qh",
"decisionReason": null, "decisionReason": null,
"recordedAt": "2025-10-12T12:01:51.8509671+00:00", "recordedAt": "2025-10-12T19:48:04.2184266+00:00",
"fieldMask": [ "fieldMask": [
"advisory" "advisory"
] ]
@@ -124,7 +124,7 @@
"kind": "reference", "kind": "reference",
"value": "https://github.com/opencontainers/distribution-spec/security/advisories/GHSA-mc8v-mgrf-8f4m", "value": "https://github.com/opencontainers/distribution-spec/security/advisories/GHSA-mc8v-mgrf-8f4m",
"decisionReason": null, "decisionReason": null,
"recordedAt": "2025-10-12T12:01:51.8509671+00:00", "recordedAt": "2025-10-12T19:48:04.2184266+00:00",
"fieldMask": [ "fieldMask": [
"references[]" "references[]"
] ]
@@ -140,7 +140,7 @@
"kind": "reference", "kind": "reference",
"value": "https://github.com/opencontainers/image-spec", "value": "https://github.com/opencontainers/image-spec",
"decisionReason": null, "decisionReason": null,
"recordedAt": "2025-10-12T12:01:51.8509671+00:00", "recordedAt": "2025-10-12T19:48:04.2184266+00:00",
"fieldMask": [ "fieldMask": [
"references[]" "references[]"
] ]
@@ -156,7 +156,7 @@
"kind": "reference", "kind": "reference",
"value": "https://github.com/opencontainers/image-spec/commit/693428a734f5bab1a84bd2f990d92ef1111cd60c", "value": "https://github.com/opencontainers/image-spec/commit/693428a734f5bab1a84bd2f990d92ef1111cd60c",
"decisionReason": null, "decisionReason": null,
"recordedAt": "2025-10-12T12:01:51.8509671+00:00", "recordedAt": "2025-10-12T19:48:04.2184266+00:00",
"fieldMask": [ "fieldMask": [
"references[]" "references[]"
] ]
@@ -172,7 +172,7 @@
"kind": "reference", "kind": "reference",
"value": "https://github.com/opencontainers/image-spec/releases/tag/v1.0.2", "value": "https://github.com/opencontainers/image-spec/releases/tag/v1.0.2",
"decisionReason": null, "decisionReason": null,
"recordedAt": "2025-10-12T12:01:51.8509671+00:00", "recordedAt": "2025-10-12T19:48:04.2184266+00:00",
"fieldMask": [ "fieldMask": [
"references[]" "references[]"
] ]
@@ -188,7 +188,7 @@
"kind": "reference", "kind": "reference",
"value": "https://github.com/opencontainers/image-spec/security/advisories/GHSA-77vh-xpmg-72qh", "value": "https://github.com/opencontainers/image-spec/security/advisories/GHSA-77vh-xpmg-72qh",
"decisionReason": null, "decisionReason": null,
"recordedAt": "2025-10-12T12:01:51.8509671+00:00", "recordedAt": "2025-10-12T19:48:04.2184266+00:00",
"fieldMask": [ "fieldMask": [
"references[]" "references[]"
] ]
@@ -236,7 +236,7 @@
"kind": "range", "kind": "range",
"value": "pkg:maven/org.apache.logging.log4j/log4j-core", "value": "pkg:maven/org.apache.logging.log4j/log4j-core",
"decisionReason": null, "decisionReason": null,
"recordedAt": "2025-10-12T12:01:51.862103+00:00", "recordedAt": "2025-10-12T19:48:04.2365076+00:00",
"fieldMask": [ "fieldMask": [
"affectedpackages[].versionranges[]" "affectedpackages[].versionranges[]"
] ]
@@ -264,7 +264,7 @@
"kind": "affected", "kind": "affected",
"value": "pkg:maven/org.apache.logging.log4j/log4j-core", "value": "pkg:maven/org.apache.logging.log4j/log4j-core",
"decisionReason": null, "decisionReason": null,
"recordedAt": "2025-10-12T12:01:51.862103+00:00", "recordedAt": "2025-10-12T19:48:04.2365076+00:00",
"fieldMask": [ "fieldMask": [
"affectedpackages[]" "affectedpackages[]"
] ]
@@ -302,7 +302,7 @@
"kind": "range", "kind": "range",
"value": "pkg:maven/org.apache.logging.log4j/log4j-core", "value": "pkg:maven/org.apache.logging.log4j/log4j-core",
"decisionReason": null, "decisionReason": null,
"recordedAt": "2025-10-12T12:01:51.862103+00:00", "recordedAt": "2025-10-12T19:48:04.2365076+00:00",
"fieldMask": [ "fieldMask": [
"affectedpackages[].versionranges[]" "affectedpackages[].versionranges[]"
] ]
@@ -330,7 +330,7 @@
"kind": "affected", "kind": "affected",
"value": "pkg:maven/org.apache.logging.log4j/log4j-core", "value": "pkg:maven/org.apache.logging.log4j/log4j-core",
"decisionReason": null, "decisionReason": null,
"recordedAt": "2025-10-12T12:01:51.862103+00:00", "recordedAt": "2025-10-12T19:48:04.2365076+00:00",
"fieldMask": [ "fieldMask": [
"affectedpackages[]" "affectedpackages[]"
] ]
@@ -368,7 +368,7 @@
"kind": "range", "kind": "range",
"value": "pkg:maven/org.ops4j.pax.logging/pax-logging-log4j2", "value": "pkg:maven/org.ops4j.pax.logging/pax-logging-log4j2",
"decisionReason": null, "decisionReason": null,
"recordedAt": "2025-10-12T12:01:51.862103+00:00", "recordedAt": "2025-10-12T19:48:04.2365076+00:00",
"fieldMask": [ "fieldMask": [
"affectedpackages[].versionranges[]" "affectedpackages[].versionranges[]"
] ]
@@ -396,7 +396,7 @@
"kind": "affected", "kind": "affected",
"value": "pkg:maven/org.ops4j.pax.logging/pax-logging-log4j2", "value": "pkg:maven/org.ops4j.pax.logging/pax-logging-log4j2",
"decisionReason": null, "decisionReason": null,
"recordedAt": "2025-10-12T12:01:51.862103+00:00", "recordedAt": "2025-10-12T19:48:04.2365076+00:00",
"fieldMask": [ "fieldMask": [
"affectedpackages[]" "affectedpackages[]"
] ]
@@ -434,7 +434,7 @@
"kind": "range", "kind": "range",
"value": "pkg:maven/org.ops4j.pax.logging/pax-logging-log4j2", "value": "pkg:maven/org.ops4j.pax.logging/pax-logging-log4j2",
"decisionReason": null, "decisionReason": null,
"recordedAt": "2025-10-12T12:01:51.862103+00:00", "recordedAt": "2025-10-12T19:48:04.2365076+00:00",
"fieldMask": [ "fieldMask": [
"affectedpackages[].versionranges[]" "affectedpackages[].versionranges[]"
] ]
@@ -462,7 +462,7 @@
"kind": "affected", "kind": "affected",
"value": "pkg:maven/org.ops4j.pax.logging/pax-logging-log4j2", "value": "pkg:maven/org.ops4j.pax.logging/pax-logging-log4j2",
"decisionReason": null, "decisionReason": null,
"recordedAt": "2025-10-12T12:01:51.862103+00:00", "recordedAt": "2025-10-12T19:48:04.2365076+00:00",
"fieldMask": [ "fieldMask": [
"affectedpackages[]" "affectedpackages[]"
] ]
@@ -500,7 +500,7 @@
"kind": "range", "kind": "range",
"value": "pkg:maven/org.ops4j.pax.logging/pax-logging-log4j2", "value": "pkg:maven/org.ops4j.pax.logging/pax-logging-log4j2",
"decisionReason": null, "decisionReason": null,
"recordedAt": "2025-10-12T12:01:51.862103+00:00", "recordedAt": "2025-10-12T19:48:04.2365076+00:00",
"fieldMask": [ "fieldMask": [
"affectedpackages[].versionranges[]" "affectedpackages[].versionranges[]"
] ]
@@ -528,7 +528,7 @@
"kind": "affected", "kind": "affected",
"value": "pkg:maven/org.ops4j.pax.logging/pax-logging-log4j2", "value": "pkg:maven/org.ops4j.pax.logging/pax-logging-log4j2",
"decisionReason": null, "decisionReason": null,
"recordedAt": "2025-10-12T12:01:51.862103+00:00", "recordedAt": "2025-10-12T19:48:04.2365076+00:00",
"fieldMask": [ "fieldMask": [
"affectedpackages[]" "affectedpackages[]"
] ]
@@ -566,7 +566,7 @@
"kind": "range", "kind": "range",
"value": "pkg:maven/org.ops4j.pax.logging/pax-logging-log4j2", "value": "pkg:maven/org.ops4j.pax.logging/pax-logging-log4j2",
"decisionReason": null, "decisionReason": null,
"recordedAt": "2025-10-12T12:01:51.862103+00:00", "recordedAt": "2025-10-12T19:48:04.2365076+00:00",
"fieldMask": [ "fieldMask": [
"affectedpackages[].versionranges[]" "affectedpackages[].versionranges[]"
] ]
@@ -594,7 +594,7 @@
"kind": "affected", "kind": "affected",
"value": "pkg:maven/org.ops4j.pax.logging/pax-logging-log4j2", "value": "pkg:maven/org.ops4j.pax.logging/pax-logging-log4j2",
"decisionReason": null, "decisionReason": null,
"recordedAt": "2025-10-12T12:01:51.862103+00:00", "recordedAt": "2025-10-12T19:48:04.2365076+00:00",
"fieldMask": [ "fieldMask": [
"affectedpackages[]" "affectedpackages[]"
] ]
@@ -616,7 +616,7 @@
"kind": "cvss", "kind": "cvss",
"value": "CVSS_V3", "value": "CVSS_V3",
"decisionReason": null, "decisionReason": null,
"recordedAt": "2025-10-12T12:01:51.862103+00:00", "recordedAt": "2025-10-12T19:48:04.2365076+00:00",
"fieldMask": [] "fieldMask": []
}, },
"vector": "CVSS:3.1/AV:N/AC:H/PR:N/UI:N/S:C/C:H/I:H/A:H", "vector": "CVSS:3.1/AV:N/AC:H/PR:N/UI:N/S:C/C:H/I:H/A:H",
@@ -632,7 +632,7 @@
"kind": "document", "kind": "document",
"value": "https://osv.dev/vulnerability/GHSA-7rjr-3q55-vv33", "value": "https://osv.dev/vulnerability/GHSA-7rjr-3q55-vv33",
"decisionReason": null, "decisionReason": null,
"recordedAt": "2021-12-14T18:01:28+00:00", "recordedAt": "2025-10-12T19:48:04.2355464+00:00",
"fieldMask": [ "fieldMask": [
"advisory" "advisory"
] ]
@@ -642,7 +642,7 @@
"kind": "mapping", "kind": "mapping",
"value": "GHSA-7rjr-3q55-vv33", "value": "GHSA-7rjr-3q55-vv33",
"decisionReason": null, "decisionReason": null,
"recordedAt": "2025-10-12T12:01:51.862103+00:00", "recordedAt": "2025-10-12T19:48:04.2365076+00:00",
"fieldMask": [ "fieldMask": [
"advisory" "advisory"
] ]
@@ -657,7 +657,7 @@
"kind": "reference", "kind": "reference",
"value": "http://www.openwall.com/lists/oss-security/2021/12/14/4", "value": "http://www.openwall.com/lists/oss-security/2021/12/14/4",
"decisionReason": null, "decisionReason": null,
"recordedAt": "2025-10-12T12:01:51.862103+00:00", "recordedAt": "2025-10-12T19:48:04.2365076+00:00",
"fieldMask": [ "fieldMask": [
"references[]" "references[]"
] ]
@@ -673,7 +673,7 @@
"kind": "reference", "kind": "reference",
"value": "http://www.openwall.com/lists/oss-security/2021/12/15/3", "value": "http://www.openwall.com/lists/oss-security/2021/12/15/3",
"decisionReason": null, "decisionReason": null,
"recordedAt": "2025-10-12T12:01:51.862103+00:00", "recordedAt": "2025-10-12T19:48:04.2365076+00:00",
"fieldMask": [ "fieldMask": [
"references[]" "references[]"
] ]
@@ -689,7 +689,7 @@
"kind": "reference", "kind": "reference",
"value": "http://www.openwall.com/lists/oss-security/2021/12/18/1", "value": "http://www.openwall.com/lists/oss-security/2021/12/18/1",
"decisionReason": null, "decisionReason": null,
"recordedAt": "2025-10-12T12:01:51.862103+00:00", "recordedAt": "2025-10-12T19:48:04.2365076+00:00",
"fieldMask": [ "fieldMask": [
"references[]" "references[]"
] ]
@@ -705,7 +705,7 @@
"kind": "reference", "kind": "reference",
"value": "https://cert-portal.siemens.com/productcert/pdf/ssa-397453.pdf", "value": "https://cert-portal.siemens.com/productcert/pdf/ssa-397453.pdf",
"decisionReason": null, "decisionReason": null,
"recordedAt": "2025-10-12T12:01:51.862103+00:00", "recordedAt": "2025-10-12T19:48:04.2365076+00:00",
"fieldMask": [ "fieldMask": [
"references[]" "references[]"
] ]
@@ -721,7 +721,7 @@
"kind": "reference", "kind": "reference",
"value": "https://cert-portal.siemens.com/productcert/pdf/ssa-479842.pdf", "value": "https://cert-portal.siemens.com/productcert/pdf/ssa-479842.pdf",
"decisionReason": null, "decisionReason": null,
"recordedAt": "2025-10-12T12:01:51.862103+00:00", "recordedAt": "2025-10-12T19:48:04.2365076+00:00",
"fieldMask": [ "fieldMask": [
"references[]" "references[]"
] ]
@@ -737,7 +737,7 @@
"kind": "reference", "kind": "reference",
"value": "https://cert-portal.siemens.com/productcert/pdf/ssa-661247.pdf", "value": "https://cert-portal.siemens.com/productcert/pdf/ssa-661247.pdf",
"decisionReason": null, "decisionReason": null,
"recordedAt": "2025-10-12T12:01:51.862103+00:00", "recordedAt": "2025-10-12T19:48:04.2365076+00:00",
"fieldMask": [ "fieldMask": [
"references[]" "references[]"
] ]
@@ -753,7 +753,7 @@
"kind": "reference", "kind": "reference",
"value": "https://cert-portal.siemens.com/productcert/pdf/ssa-714170.pdf", "value": "https://cert-portal.siemens.com/productcert/pdf/ssa-714170.pdf",
"decisionReason": null, "decisionReason": null,
"recordedAt": "2025-10-12T12:01:51.862103+00:00", "recordedAt": "2025-10-12T19:48:04.2365076+00:00",
"fieldMask": [ "fieldMask": [
"references[]" "references[]"
] ]
@@ -769,7 +769,7 @@
"kind": "reference", "kind": "reference",
"value": "https://github.com/advisories/GHSA-jfh8-c2jp-5v3q", "value": "https://github.com/advisories/GHSA-jfh8-c2jp-5v3q",
"decisionReason": null, "decisionReason": null,
"recordedAt": "2025-10-12T12:01:51.862103+00:00", "recordedAt": "2025-10-12T19:48:04.2365076+00:00",
"fieldMask": [ "fieldMask": [
"references[]" "references[]"
] ]
@@ -785,7 +785,7 @@
"kind": "reference", "kind": "reference",
"value": "https://lists.fedoraproject.org/archives/list/package-announce@lists.fedoraproject.org/message/EOKPQGV24RRBBI4TBZUDQMM4MEH7MXCY", "value": "https://lists.fedoraproject.org/archives/list/package-announce@lists.fedoraproject.org/message/EOKPQGV24RRBBI4TBZUDQMM4MEH7MXCY",
"decisionReason": null, "decisionReason": null,
"recordedAt": "2025-10-12T12:01:51.862103+00:00", "recordedAt": "2025-10-12T19:48:04.2365076+00:00",
"fieldMask": [ "fieldMask": [
"references[]" "references[]"
] ]
@@ -801,7 +801,7 @@
"kind": "reference", "kind": "reference",
"value": "https://lists.fedoraproject.org/archives/list/package-announce@lists.fedoraproject.org/message/SIG7FZULMNK2XF6FZRU4VWYDQXNMUGAJ", "value": "https://lists.fedoraproject.org/archives/list/package-announce@lists.fedoraproject.org/message/SIG7FZULMNK2XF6FZRU4VWYDQXNMUGAJ",
"decisionReason": null, "decisionReason": null,
"recordedAt": "2025-10-12T12:01:51.862103+00:00", "recordedAt": "2025-10-12T19:48:04.2365076+00:00",
"fieldMask": [ "fieldMask": [
"references[]" "references[]"
] ]
@@ -817,7 +817,7 @@
"kind": "reference", "kind": "reference",
"value": "https://logging.apache.org/log4j/2.x/security.html", "value": "https://logging.apache.org/log4j/2.x/security.html",
"decisionReason": null, "decisionReason": null,
"recordedAt": "2025-10-12T12:01:51.862103+00:00", "recordedAt": "2025-10-12T19:48:04.2365076+00:00",
"fieldMask": [ "fieldMask": [
"references[]" "references[]"
] ]
@@ -833,7 +833,7 @@
"kind": "reference", "kind": "reference",
"value": "https://nvd.nist.gov/vuln/detail/CVE-2021-45046", "value": "https://nvd.nist.gov/vuln/detail/CVE-2021-45046",
"decisionReason": null, "decisionReason": null,
"recordedAt": "2025-10-12T12:01:51.862103+00:00", "recordedAt": "2025-10-12T19:48:04.2365076+00:00",
"fieldMask": [ "fieldMask": [
"references[]" "references[]"
] ]
@@ -849,7 +849,7 @@
"kind": "reference", "kind": "reference",
"value": "https://psirt.global.sonicwall.com/vuln-detail/SNWLID-2021-0032", "value": "https://psirt.global.sonicwall.com/vuln-detail/SNWLID-2021-0032",
"decisionReason": null, "decisionReason": null,
"recordedAt": "2025-10-12T12:01:51.862103+00:00", "recordedAt": "2025-10-12T19:48:04.2365076+00:00",
"fieldMask": [ "fieldMask": [
"references[]" "references[]"
] ]
@@ -865,7 +865,7 @@
"kind": "reference", "kind": "reference",
"value": "https://sec.cloudapps.cisco.com/security/center/content/CiscoSecurityAdvisory/cisco-sa-apache-log4j-qRuKNEbd", "value": "https://sec.cloudapps.cisco.com/security/center/content/CiscoSecurityAdvisory/cisco-sa-apache-log4j-qRuKNEbd",
"decisionReason": null, "decisionReason": null,
"recordedAt": "2025-10-12T12:01:51.862103+00:00", "recordedAt": "2025-10-12T19:48:04.2365076+00:00",
"fieldMask": [ "fieldMask": [
"references[]" "references[]"
] ]
@@ -881,7 +881,7 @@
"kind": "reference", "kind": "reference",
"value": "https://security.gentoo.org/glsa/202310-16", "value": "https://security.gentoo.org/glsa/202310-16",
"decisionReason": null, "decisionReason": null,
"recordedAt": "2025-10-12T12:01:51.862103+00:00", "recordedAt": "2025-10-12T19:48:04.2365076+00:00",
"fieldMask": [ "fieldMask": [
"references[]" "references[]"
] ]
@@ -897,7 +897,7 @@
"kind": "reference", "kind": "reference",
"value": "https://www.cve.org/CVERecord?id=CVE-2021-44228", "value": "https://www.cve.org/CVERecord?id=CVE-2021-44228",
"decisionReason": null, "decisionReason": null,
"recordedAt": "2025-10-12T12:01:51.862103+00:00", "recordedAt": "2025-10-12T19:48:04.2365076+00:00",
"fieldMask": [ "fieldMask": [
"references[]" "references[]"
] ]
@@ -913,7 +913,7 @@
"kind": "reference", "kind": "reference",
"value": "https://www.debian.org/security/2021/dsa-5022", "value": "https://www.debian.org/security/2021/dsa-5022",
"decisionReason": null, "decisionReason": null,
"recordedAt": "2025-10-12T12:01:51.862103+00:00", "recordedAt": "2025-10-12T19:48:04.2365076+00:00",
"fieldMask": [ "fieldMask": [
"references[]" "references[]"
] ]
@@ -929,7 +929,7 @@
"kind": "reference", "kind": "reference",
"value": "https://www.intel.com/content/www/us/en/security-center/advisory/intel-sa-00646.html", "value": "https://www.intel.com/content/www/us/en/security-center/advisory/intel-sa-00646.html",
"decisionReason": null, "decisionReason": null,
"recordedAt": "2025-10-12T12:01:51.862103+00:00", "recordedAt": "2025-10-12T19:48:04.2365076+00:00",
"fieldMask": [ "fieldMask": [
"references[]" "references[]"
] ]
@@ -945,7 +945,7 @@
"kind": "reference", "kind": "reference",
"value": "https://www.kb.cert.org/vuls/id/930724", "value": "https://www.kb.cert.org/vuls/id/930724",
"decisionReason": null, "decisionReason": null,
"recordedAt": "2025-10-12T12:01:51.862103+00:00", "recordedAt": "2025-10-12T19:48:04.2365076+00:00",
"fieldMask": [ "fieldMask": [
"references[]" "references[]"
] ]
@@ -961,7 +961,7 @@
"kind": "reference", "kind": "reference",
"value": "https://www.openwall.com/lists/oss-security/2021/12/14/4", "value": "https://www.openwall.com/lists/oss-security/2021/12/14/4",
"decisionReason": null, "decisionReason": null,
"recordedAt": "2025-10-12T12:01:51.862103+00:00", "recordedAt": "2025-10-12T19:48:04.2365076+00:00",
"fieldMask": [ "fieldMask": [
"references[]" "references[]"
] ]
@@ -977,7 +977,7 @@
"kind": "reference", "kind": "reference",
"value": "https://www.oracle.com/security-alerts/alert-cve-2021-44228.html", "value": "https://www.oracle.com/security-alerts/alert-cve-2021-44228.html",
"decisionReason": null, "decisionReason": null,
"recordedAt": "2025-10-12T12:01:51.862103+00:00", "recordedAt": "2025-10-12T19:48:04.2365076+00:00",
"fieldMask": [ "fieldMask": [
"references[]" "references[]"
] ]
@@ -993,7 +993,7 @@
"kind": "reference", "kind": "reference",
"value": "https://www.oracle.com/security-alerts/cpuapr2022.html", "value": "https://www.oracle.com/security-alerts/cpuapr2022.html",
"decisionReason": null, "decisionReason": null,
"recordedAt": "2025-10-12T12:01:51.862103+00:00", "recordedAt": "2025-10-12T19:48:04.2365076+00:00",
"fieldMask": [ "fieldMask": [
"references[]" "references[]"
] ]
@@ -1009,7 +1009,7 @@
"kind": "reference", "kind": "reference",
"value": "https://www.oracle.com/security-alerts/cpujan2022.html", "value": "https://www.oracle.com/security-alerts/cpujan2022.html",
"decisionReason": null, "decisionReason": null,
"recordedAt": "2025-10-12T12:01:51.862103+00:00", "recordedAt": "2025-10-12T19:48:04.2365076+00:00",
"fieldMask": [ "fieldMask": [
"references[]" "references[]"
] ]
@@ -1025,7 +1025,7 @@
"kind": "reference", "kind": "reference",
"value": "https://www.oracle.com/security-alerts/cpujul2022.html", "value": "https://www.oracle.com/security-alerts/cpujul2022.html",
"decisionReason": null, "decisionReason": null,
"recordedAt": "2025-10-12T12:01:51.862103+00:00", "recordedAt": "2025-10-12T19:48:04.2365076+00:00",
"fieldMask": [ "fieldMask": [
"references[]" "references[]"
] ]
@@ -1073,7 +1073,7 @@
"kind": "range", "kind": "range",
"value": "pkg:pypi/pyload-ng", "value": "pkg:pypi/pyload-ng",
"decisionReason": null, "decisionReason": null,
"recordedAt": "2025-10-12T12:01:51.8437105+00:00", "recordedAt": "2025-10-12T19:48:04.2065811+00:00",
"fieldMask": [ "fieldMask": [
"affectedpackages[].versionranges[]" "affectedpackages[].versionranges[]"
] ]
@@ -1101,7 +1101,7 @@
"kind": "affected", "kind": "affected",
"value": "pkg:pypi/pyload-ng", "value": "pkg:pypi/pyload-ng",
"decisionReason": null, "decisionReason": null,
"recordedAt": "2025-10-12T12:01:51.8437105+00:00", "recordedAt": "2025-10-12T19:48:04.2065811+00:00",
"fieldMask": [ "fieldMask": [
"affectedpackages[]" "affectedpackages[]"
] ]
@@ -1123,7 +1123,7 @@
"kind": "cvss", "kind": "cvss",
"value": "CVSS_V3", "value": "CVSS_V3",
"decisionReason": null, "decisionReason": null,
"recordedAt": "2025-10-12T12:01:51.8437105+00:00", "recordedAt": "2025-10-12T19:48:04.2065811+00:00",
"fieldMask": [] "fieldMask": []
}, },
"vector": "CVSS:3.1/AV:N/AC:L/PR:N/UI:R/S:U/C:H/I:H/A:N", "vector": "CVSS:3.1/AV:N/AC:L/PR:N/UI:R/S:U/C:H/I:H/A:N",
@@ -1139,7 +1139,7 @@
"kind": "document", "kind": "document",
"value": "https://osv.dev/vulnerability/GHSA-cjjf-27cc-pvmv", "value": "https://osv.dev/vulnerability/GHSA-cjjf-27cc-pvmv",
"decisionReason": null, "decisionReason": null,
"recordedAt": "2025-10-09T15:19:48+00:00", "recordedAt": "2025-10-12T19:48:04.2061911+00:00",
"fieldMask": [ "fieldMask": [
"advisory" "advisory"
] ]
@@ -1149,7 +1149,7 @@
"kind": "mapping", "kind": "mapping",
"value": "GHSA-cjjf-27cc-pvmv", "value": "GHSA-cjjf-27cc-pvmv",
"decisionReason": null, "decisionReason": null,
"recordedAt": "2025-10-12T12:01:51.8437105+00:00", "recordedAt": "2025-10-12T19:48:04.2065811+00:00",
"fieldMask": [ "fieldMask": [
"advisory" "advisory"
] ]
@@ -1164,7 +1164,7 @@
"kind": "reference", "kind": "reference",
"value": "https://github.com/pyload/pyload", "value": "https://github.com/pyload/pyload",
"decisionReason": null, "decisionReason": null,
"recordedAt": "2025-10-12T12:01:51.8437105+00:00", "recordedAt": "2025-10-12T19:48:04.2065811+00:00",
"fieldMask": [ "fieldMask": [
"references[]" "references[]"
] ]
@@ -1180,7 +1180,7 @@
"kind": "reference", "kind": "reference",
"value": "https://github.com/pyload/pyload/commit/5823327d0b797161c7195a1f660266d30a69f0ca", "value": "https://github.com/pyload/pyload/commit/5823327d0b797161c7195a1f660266d30a69f0ca",
"decisionReason": null, "decisionReason": null,
"recordedAt": "2025-10-12T12:01:51.8437105+00:00", "recordedAt": "2025-10-12T19:48:04.2065811+00:00",
"fieldMask": [ "fieldMask": [
"references[]" "references[]"
] ]
@@ -1196,7 +1196,7 @@
"kind": "reference", "kind": "reference",
"value": "https://github.com/pyload/pyload/pull/4624", "value": "https://github.com/pyload/pyload/pull/4624",
"decisionReason": null, "decisionReason": null,
"recordedAt": "2025-10-12T12:01:51.8437105+00:00", "recordedAt": "2025-10-12T19:48:04.2065811+00:00",
"fieldMask": [ "fieldMask": [
"references[]" "references[]"
] ]
@@ -1212,7 +1212,7 @@
"kind": "reference", "kind": "reference",
"value": "https://github.com/pyload/pyload/security/advisories/GHSA-cjjf-27cc-pvmv", "value": "https://github.com/pyload/pyload/security/advisories/GHSA-cjjf-27cc-pvmv",
"decisionReason": null, "decisionReason": null,
"recordedAt": "2025-10-12T12:01:51.8437105+00:00", "recordedAt": "2025-10-12T19:48:04.2065811+00:00",
"fieldMask": [ "fieldMask": [
"references[]" "references[]"
] ]
@@ -1260,7 +1260,7 @@
"kind": "range", "kind": "range",
"value": "pkg:pypi/social-auth-app-django", "value": "pkg:pypi/social-auth-app-django",
"decisionReason": null, "decisionReason": null,
"recordedAt": "2025-10-12T12:01:51.8047195+00:00", "recordedAt": "2025-10-12T19:48:04.1231115+00:00",
"fieldMask": [ "fieldMask": [
"affectedpackages[].versionranges[]" "affectedpackages[].versionranges[]"
] ]
@@ -1288,7 +1288,7 @@
"kind": "affected", "kind": "affected",
"value": "pkg:pypi/social-auth-app-django", "value": "pkg:pypi/social-auth-app-django",
"decisionReason": null, "decisionReason": null,
"recordedAt": "2025-10-12T12:01:51.8047195+00:00", "recordedAt": "2025-10-12T19:48:04.1231115+00:00",
"fieldMask": [ "fieldMask": [
"affectedpackages[]" "affectedpackages[]"
] ]
@@ -1311,7 +1311,7 @@
"kind": "document", "kind": "document",
"value": "https://osv.dev/vulnerability/GHSA-wv4w-6qv2-qqfg", "value": "https://osv.dev/vulnerability/GHSA-wv4w-6qv2-qqfg",
"decisionReason": null, "decisionReason": null,
"recordedAt": "2025-10-09T17:08:05+00:00", "recordedAt": "2025-10-12T19:48:04.0743113+00:00",
"fieldMask": [ "fieldMask": [
"advisory" "advisory"
] ]
@@ -1321,7 +1321,7 @@
"kind": "mapping", "kind": "mapping",
"value": "GHSA-wv4w-6qv2-qqfg", "value": "GHSA-wv4w-6qv2-qqfg",
"decisionReason": null, "decisionReason": null,
"recordedAt": "2025-10-12T12:01:51.8047195+00:00", "recordedAt": "2025-10-12T19:48:04.1231115+00:00",
"fieldMask": [ "fieldMask": [
"advisory" "advisory"
] ]
@@ -1336,7 +1336,7 @@
"kind": "reference", "kind": "reference",
"value": "https://github.com/python-social-auth/social-app-django", "value": "https://github.com/python-social-auth/social-app-django",
"decisionReason": null, "decisionReason": null,
"recordedAt": "2025-10-12T12:01:51.8047195+00:00", "recordedAt": "2025-10-12T19:48:04.1231115+00:00",
"fieldMask": [ "fieldMask": [
"references[]" "references[]"
] ]
@@ -1352,7 +1352,7 @@
"kind": "reference", "kind": "reference",
"value": "https://github.com/python-social-auth/social-app-django/commit/10c80e2ebabeccd4e9c84ad0e16e1db74148ed4c", "value": "https://github.com/python-social-auth/social-app-django/commit/10c80e2ebabeccd4e9c84ad0e16e1db74148ed4c",
"decisionReason": null, "decisionReason": null,
"recordedAt": "2025-10-12T12:01:51.8047195+00:00", "recordedAt": "2025-10-12T19:48:04.1231115+00:00",
"fieldMask": [ "fieldMask": [
"references[]" "references[]"
] ]
@@ -1368,7 +1368,7 @@
"kind": "reference", "kind": "reference",
"value": "https://github.com/python-social-auth/social-app-django/issues/220", "value": "https://github.com/python-social-auth/social-app-django/issues/220",
"decisionReason": null, "decisionReason": null,
"recordedAt": "2025-10-12T12:01:51.8047195+00:00", "recordedAt": "2025-10-12T19:48:04.1231115+00:00",
"fieldMask": [ "fieldMask": [
"references[]" "references[]"
] ]
@@ -1384,7 +1384,7 @@
"kind": "reference", "kind": "reference",
"value": "https://github.com/python-social-auth/social-app-django/issues/231", "value": "https://github.com/python-social-auth/social-app-django/issues/231",
"decisionReason": null, "decisionReason": null,
"recordedAt": "2025-10-12T12:01:51.8047195+00:00", "recordedAt": "2025-10-12T19:48:04.1231115+00:00",
"fieldMask": [ "fieldMask": [
"references[]" "references[]"
] ]
@@ -1400,7 +1400,7 @@
"kind": "reference", "kind": "reference",
"value": "https://github.com/python-social-auth/social-app-django/issues/634", "value": "https://github.com/python-social-auth/social-app-django/issues/634",
"decisionReason": null, "decisionReason": null,
"recordedAt": "2025-10-12T12:01:51.8047195+00:00", "recordedAt": "2025-10-12T19:48:04.1231115+00:00",
"fieldMask": [ "fieldMask": [
"references[]" "references[]"
] ]
@@ -1416,7 +1416,7 @@
"kind": "reference", "kind": "reference",
"value": "https://github.com/python-social-auth/social-app-django/pull/803", "value": "https://github.com/python-social-auth/social-app-django/pull/803",
"decisionReason": null, "decisionReason": null,
"recordedAt": "2025-10-12T12:01:51.8047195+00:00", "recordedAt": "2025-10-12T19:48:04.1231115+00:00",
"fieldMask": [ "fieldMask": [
"references[]" "references[]"
] ]
@@ -1432,7 +1432,7 @@
"kind": "reference", "kind": "reference",
"value": "https://github.com/python-social-auth/social-app-django/security/advisories/GHSA-wv4w-6qv2-qqfg", "value": "https://github.com/python-social-auth/social-app-django/security/advisories/GHSA-wv4w-6qv2-qqfg",
"decisionReason": null, "decisionReason": null,
"recordedAt": "2025-10-12T12:01:51.8047195+00:00", "recordedAt": "2025-10-12T19:48:04.1231115+00:00",
"fieldMask": [ "fieldMask": [
"references[]" "references[]"
] ]

View File

@@ -1,11 +1,14 @@
using System; using System;
using System.Collections.Generic; using System.Collections.Generic;
using System.Text.Json; using System.Text.Json;
using System.Text.Json.Serialization;
using System.Reflection;
using MongoDB.Bson; using MongoDB.Bson;
using StellaOps.Feedser.Models; using StellaOps.Feedser.Models;
using StellaOps.Feedser.Source.Common; using StellaOps.Feedser.Source.Common;
using StellaOps.Feedser.Source.Osv; using StellaOps.Feedser.Source.Osv;
using StellaOps.Feedser.Source.Osv.Internal; using StellaOps.Feedser.Source.Osv.Internal;
using StellaOps.Feedser.Normalization.Identifiers;
using StellaOps.Feedser.Storage.Mongo.Documents; using StellaOps.Feedser.Storage.Mongo.Documents;
using StellaOps.Feedser.Storage.Mongo.Dtos; using StellaOps.Feedser.Storage.Mongo.Dtos;
using Xunit; using Xunit;
@@ -120,4 +123,78 @@ public sealed class OsvMapperTests
Assert.Single(advisory.CvssMetrics); Assert.Single(advisory.CvssMetrics);
Assert.Equal("3.1", advisory.CvssMetrics[0].Version); Assert.Equal("3.1", advisory.CvssMetrics[0].Version);
} }
[Theory]
[InlineData("Go", "github.com/example/project", "pkg:golang/github.com/example/project")]
[InlineData("PyPI", "social_auth_app_django", "pkg:pypi/social-auth-app-django")]
[InlineData("npm", "@Scope/Package", "pkg:npm/%40scope/package")]
[InlineData("Maven", "org.example:library", "pkg:maven/org.example/library")]
[InlineData("crates", "serde", "pkg:cargo/serde")]
public void Map_InfersCanonicalPackageUrlWhenPurlMissing(string ecosystem, string packageName, string expectedIdentifier)
{
var dto = new OsvVulnerabilityDto
{
Id = $"OSV-{ecosystem}-PURL",
Summary = "Test advisory",
Details = "Details",
Published = DateTimeOffset.UtcNow.AddDays(-1),
Modified = DateTimeOffset.UtcNow,
Affected = new[]
{
new OsvAffectedPackageDto
{
Package = new OsvPackageDto
{
Ecosystem = ecosystem,
Name = packageName,
Purl = null,
},
Ranges = null,
}
}
};
if (string.Equals(ecosystem, "npm", StringComparison.OrdinalIgnoreCase))
{
Assert.True(IdentifierNormalizer.TryNormalizePackageUrl("pkg:npm/%40scope/package", out var canonical));
Assert.Equal(expectedIdentifier, canonical);
}
var method = typeof(OsvMapper).GetMethod("DetermineIdentifier", BindingFlags.NonPublic | BindingFlags.Static);
Assert.NotNull(method);
var directIdentifier = method!.Invoke(null, new object?[] { dto.Affected![0].Package!, ecosystem }) as string;
Assert.Equal(expectedIdentifier, directIdentifier);
var (document, dtoRecord) = CreateDocumentAndDtoRecord(dto, ecosystem);
var advisory = OsvMapper.Map(dto, document, dtoRecord, ecosystem);
var affected = Assert.Single(advisory.AffectedPackages);
Assert.Equal(expectedIdentifier, affected.Identifier);
}
private static (DocumentRecord Document, DtoRecord DtoRecord) CreateDocumentAndDtoRecord(OsvVulnerabilityDto dto, string ecosystem)
{
var recordedAt = DateTimeOffset.UtcNow;
var document = new DocumentRecord(
Guid.NewGuid(),
OsvConnectorPlugin.SourceName,
$"https://osv.dev/vulnerability/{dto.Id}",
recordedAt,
"sha256",
DocumentStatuses.PendingParse,
"application/json",
null,
new Dictionary<string, string>(StringComparer.Ordinal)
{
["osv.ecosystem"] = ecosystem,
},
null,
dto.Modified,
null,
null);
var payload = new BsonDocument("id", dto.Id);
var dtoRecord = new DtoRecord(Guid.NewGuid(), document.Id, OsvConnectorPlugin.SourceName, "osv.v1", payload, recordedAt);
return (document, dtoRecord);
}
} }

View File

@@ -16,6 +16,26 @@ internal static class OsvMapper
{ {
private static readonly string[] SeverityOrder = { "none", "low", "medium", "high", "critical" }; private static readonly string[] SeverityOrder = { "none", "low", "medium", "high", "critical" };
private static readonly IReadOnlyDictionary<string, Func<string, string>> PackageUrlBuilders =
new Dictionary<string, Func<string, string>>(StringComparer.OrdinalIgnoreCase)
{
["pypi"] = static name => $"pkg:pypi/{NormalizePyPiName(name)}",
["python"] = static name => $"pkg:pypi/{NormalizePyPiName(name)}",
["maven"] = static name => $"pkg:maven/{NormalizeMavenName(name)}",
["go"] = static name => $"pkg:golang/{NormalizeGoName(name)}",
["golang"] = static name => $"pkg:golang/{NormalizeGoName(name)}",
["crates"] = static name => $"pkg:cargo/{NormalizeCratesName(name)}",
["crates.io"] = static name => $"pkg:cargo/{NormalizeCratesName(name)}",
["cargo"] = static name => $"pkg:cargo/{NormalizeCratesName(name)}",
["nuget"] = static name => $"pkg:nuget/{NormalizeNugetName(name)}",
["rubygems"] = static name => $"pkg:gem/{NormalizeRubyName(name)}",
["gem"] = static name => $"pkg:gem/{NormalizeRubyName(name)}",
["packagist"] = static name => $"pkg:composer/{NormalizeComposerName(name)}",
["composer"] = static name => $"pkg:composer/{NormalizeComposerName(name)}",
["hex"] = static name => $"pkg:hex/{NormalizeHexName(name)}",
["hex.pm"] = static name => $"pkg:hex/{NormalizeHexName(name)}",
};
public static Advisory Map( public static Advisory Map(
OsvVulnerabilityDto dto, OsvVulnerabilityDto dto,
DocumentRecord document, DocumentRecord document,
@@ -418,21 +438,32 @@ internal static class OsvMapper
private static string? DetermineIdentifier(OsvPackageDto package, string ecosystem) private static string? DetermineIdentifier(OsvPackageDto package, string ecosystem)
{ {
if (!string.IsNullOrWhiteSpace(package.Purl) if (IdentifierNormalizer.TryNormalizePackageUrl(package.Purl, out var normalized))
&& IdentifierNormalizer.TryNormalizePackageUrl(package.Purl, out var normalized))
{ {
return normalized; return normalized;
} }
if (!string.IsNullOrWhiteSpace(package.Name)) var name = Validation.TrimToNull(package.Name);
if (name is null)
{ {
var name = package.Name.Trim(); return null;
return string.IsNullOrWhiteSpace(package.Ecosystem)
? $"{ecosystem}:{name}"
: $"{package.Ecosystem.Trim()}:{name}";
} }
return null; var ecosystemHint = Validation.TrimToNull(package.Ecosystem) ?? Validation.TrimToNull(ecosystem);
if (ecosystemHint is not null
&& string.Equals(ecosystemHint, "npm", StringComparison.OrdinalIgnoreCase)
&& TryBuildNpmPackageUrl(name, out normalized))
{
return normalized;
}
if (TryBuildCanonicalPackageUrl(ecosystemHint, name, out normalized))
{
return normalized;
}
var fallbackEcosystem = ecosystemHint ?? Validation.TrimToNull(ecosystem) ?? "osv";
return $"{fallbackEcosystem}:{name}";
} }
private static IReadOnlyList<CvssMetric> BuildCvssMetrics(OsvVulnerabilityDto dto, DateTimeOffset recordedAt, out string? severity) private static IReadOnlyList<CvssMetric> BuildCvssMetrics(OsvVulnerabilityDto dto, DateTimeOffset recordedAt, out string? severity)
@@ -539,4 +570,77 @@ internal static class OsvMapper
return StringComparer.Ordinal.Compare(left, right); return StringComparer.Ordinal.Compare(left, right);
} }
private static bool TryBuildCanonicalPackageUrl(string? ecosystem, string name, out string? canonical)
{
canonical = null;
var trimmedEcosystem = Validation.TrimToNull(ecosystem);
if (trimmedEcosystem is null)
{
return false;
}
if (!PackageUrlBuilders.TryGetValue(trimmedEcosystem, out var factory))
{
return false;
}
var candidate = factory(name.Trim());
return IdentifierNormalizer.TryNormalizePackageUrl(candidate, out canonical);
}
private static string NormalizePyPiName(string name) => name.Trim().Replace('_', '-');
private static bool TryBuildNpmPackageUrl(string name, out string? canonical)
{
canonical = null;
var trimmed = name.Trim();
if (trimmed.Length == 0)
{
return false;
}
if (trimmed[0] == '@')
{
var slashIndex = trimmed.IndexOf('/', 1);
if (slashIndex <= 1 || slashIndex >= trimmed.Length - 1)
{
return false;
}
var scope = trimmed[..slashIndex].ToLowerInvariant();
var package = trimmed[(slashIndex + 1)..].ToLowerInvariant();
var candidate = $"pkg:npm/{Uri.EscapeDataString(scope)}/{Uri.EscapeDataString(package)}";
return IdentifierNormalizer.TryNormalizePackageUrl(candidate, out canonical);
}
if (trimmed.Contains('/', StringComparison.Ordinal))
{
return false;
}
var normalized = trimmed.ToLowerInvariant();
var simpleCandidate = $"pkg:npm/{Uri.EscapeDataString(normalized)}";
return IdentifierNormalizer.TryNormalizePackageUrl(simpleCandidate, out canonical);
}
private static string NormalizeMavenName(string name)
{
var trimmed = name.Trim();
return trimmed.Contains(':', StringComparison.Ordinal)
? trimmed.Replace(':', '/')
: trimmed.Replace('\\', '/');
}
private static string NormalizeGoName(string name) => name.Trim();
private static string NormalizeCratesName(string name) => name.Trim();
private static string NormalizeNugetName(string name) => name.Trim();
private static string NormalizeRubyName(string name) => name.Trim();
private static string NormalizeComposerName(string name) => name.Trim();
private static string NormalizeHexName(string name) => name.Trim();
} }

View File

@@ -12,7 +12,7 @@
|Connector DI routine & job registration|BE-Conn-OSV|Core|**DONE** DI routine registers fetch/parse/map jobs with scheduler.| |Connector DI routine & job registration|BE-Conn-OSV|Core|**DONE** DI routine registers fetch/parse/map jobs with scheduler.|
|Implement OSV fetch/parse/map skeleton|BE-Conn-OSV|Source.Common|**DONE** connector now persists documents, DTOs, and canonical advisories.| |Implement OSV fetch/parse/map skeleton|BE-Conn-OSV|Source.Common|**DONE** connector now persists documents, DTOs, and canonical advisories.|
|FEEDCONN-OSV-02-004 OSV references & credits alignment|BE-Conn-OSV|Models `FEEDMODELS-SCHEMA-01-002`|**DONE (2025-10-11)** Mapper normalizes references with provenance masks, emits advisory credits, and regression fixtures/assertions cover the new fields.| |FEEDCONN-OSV-02-004 OSV references & credits alignment|BE-Conn-OSV|Models `FEEDMODELS-SCHEMA-01-002`|**DONE (2025-10-11)** Mapper normalizes references with provenance masks, emits advisory credits, and regression fixtures/assertions cover the new fields.|
|FEEDCONN-OSV-02-005 Fixture updater workflow|BE-Conn-OSV, QA|Docs|TODO Document `tools/FixtureUpdater`, add parity regression steps, and ensure future refreshes capture credit metadata consistently.| |FEEDCONN-OSV-02-005 Fixture updater workflow|BE-Conn-OSV, QA|Docs|**DONE (2025-10-12)** Canonical PURL derivation now covers Go + scoped npm advisories without upstream `purl`; legacy invalid npm names still fall back to `ecosystem:name`. OSV/GHSA/NVD suites and normalization/storage tests rerun clean.|
|FEEDCONN-OSV-02-003 Normalized versions rollout|BE-Conn-OSV|Models `FEEDMODELS-SCHEMA-01-003`, Normalization playbook|**DONE (2025-10-11)** `OsvMapper` now emits SemVer primitives + normalized rules with `osv:{ecosystem}:{advisoryId}:{identifier}` notes; npm/PyPI/Parity fixtures refreshed; merge coordination pinged (OSV handoff).| |FEEDCONN-OSV-02-003 Normalized versions rollout|BE-Conn-OSV|Models `FEEDMODELS-SCHEMA-01-003`, Normalization playbook|**DONE (2025-10-11)** `OsvMapper` now emits SemVer primitives + normalized rules with `osv:{ecosystem}:{advisoryId}:{identifier}` notes; npm/PyPI/Parity fixtures refreshed; merge coordination pinged (OSV handoff).|
|FEEDCONN-OSV-04-003 Parity fixture refresh|QA, BE-Conn-OSV|Normalized versions rollout, GHSA parity tests|**DONE (2025-10-12)** Parity fixtures include normalizedVersions notes (`osv:<ecosystem>:<id>:<purl>`); regression math rerun via `dotnet test src/StellaOps.Feedser.Source.Osv.Tests` and docs flagged for workflow sync.| |FEEDCONN-OSV-04-003 Parity fixture refresh|QA, BE-Conn-OSV|Normalized versions rollout, GHSA parity tests|**DONE (2025-10-12)** Parity fixtures include normalizedVersions notes (`osv:<ecosystem>:<id>:<purl>`); regression math rerun via `dotnet test src/StellaOps.Feedser.Source.Osv.Tests` and docs flagged for workflow sync.|
|FEEDCONN-OSV-04-002 Conflict regression fixtures|BE-Conn-OSV, QA|Merge `FEEDMERGE-ENGINE-04-001`|**DONE (2025-10-12)** Added `conflict-osv.canonical.json` + regression asserting SemVer range + CVSS medium severity; dataset matches GHSA/NVD fixtures for merge tests. Validation: `dotnet test src/StellaOps.Feedser.Source.Osv.Tests/StellaOps.Feedser.Source.Osv.Tests.csproj --filter OsvConflictFixtureTests`.| |FEEDCONN-OSV-04-002 Conflict regression fixtures|BE-Conn-OSV, QA|Merge `FEEDMERGE-ENGINE-04-001`|**DONE (2025-10-12)** Added `conflict-osv.canonical.json` + regression asserting SemVer range + CVSS medium severity; dataset matches GHSA/NVD fixtures for merge tests. Validation: `dotnet test src/StellaOps.Feedser.Source.Osv.Tests/StellaOps.Feedser.Source.Osv.Tests.csproj --filter OsvConflictFixtureTests`.|

View File

@@ -62,7 +62,12 @@ public sealed class AdvisoryStorePerformanceTests : IClassFixture<MongoIntegrati
await bootstrapper.InitializeAsync(CancellationToken.None); await bootstrapper.InitializeAsync(CancellationToken.None);
var aliasStore = new AliasStore(database, NullLogger<AliasStore>.Instance); var aliasStore = new AliasStore(database, NullLogger<AliasStore>.Instance);
var store = new AdvisoryStore(database, aliasStore, NullLogger<AdvisoryStore>.Instance, TimeProvider.System); var store = new AdvisoryStore(
database,
aliasStore,
NullLogger<AdvisoryStore>.Instance,
Options.Create(new MongoStorageOptions()),
TimeProvider.System);
using var cts = new CancellationTokenSource(TimeSpan.FromSeconds(45)); using var cts = new CancellationTokenSource(TimeSpan.FromSeconds(45));
// Warm up collections (indexes, serialization caches) so perf timings exclude one-time setup work. // Warm up collections (indexes, serialization caches) so perf timings exclude one-time setup work.

View File

@@ -2,6 +2,7 @@ using System;
using System.Collections.Generic; using System.Collections.Generic;
using System.Linq; using System.Linq;
using Microsoft.Extensions.Logging.Abstractions; using Microsoft.Extensions.Logging.Abstractions;
using Microsoft.Extensions.Options;
using MongoDB.Driver; using MongoDB.Driver;
using StellaOps.Feedser.Models; using StellaOps.Feedser.Models;
using StellaOps.Feedser.Storage.Mongo.Advisories; using StellaOps.Feedser.Storage.Mongo.Advisories;
@@ -26,7 +27,12 @@ public sealed class AdvisoryStoreTests : IClassFixture<MongoIntegrationFixture>
await DropCollectionAsync(MongoStorageDefaults.Collections.Alias); await DropCollectionAsync(MongoStorageDefaults.Collections.Alias);
var aliasStore = new AliasStore(_fixture.Database, NullLogger<AliasStore>.Instance); var aliasStore = new AliasStore(_fixture.Database, NullLogger<AliasStore>.Instance);
var store = new AdvisoryStore(_fixture.Database, aliasStore, NullLogger<AdvisoryStore>.Instance, TimeProvider.System); var store = new AdvisoryStore(
_fixture.Database,
aliasStore,
NullLogger<AdvisoryStore>.Instance,
Options.Create(new MongoStorageOptions()),
TimeProvider.System);
var advisory = new Advisory( var advisory = new Advisory(
advisoryKey: "ADV-1", advisoryKey: "ADV-1",
title: "Sample Advisory", title: "Sample Advisory",
@@ -63,7 +69,12 @@ public sealed class AdvisoryStoreTests : IClassFixture<MongoIntegrationFixture>
await DropCollectionAsync(MongoStorageDefaults.Collections.Alias); await DropCollectionAsync(MongoStorageDefaults.Collections.Alias);
var aliasStore = new AliasStore(_fixture.Database, NullLogger<AliasStore>.Instance); var aliasStore = new AliasStore(_fixture.Database, NullLogger<AliasStore>.Instance);
var store = new AdvisoryStore(_fixture.Database, aliasStore, NullLogger<AdvisoryStore>.Instance, TimeProvider.System); var store = new AdvisoryStore(
_fixture.Database,
aliasStore,
NullLogger<AdvisoryStore>.Instance,
Options.Create(new MongoStorageOptions()),
TimeProvider.System);
var recordedAt = new DateTimeOffset(2025, 1, 1, 0, 0, 0, TimeSpan.Zero); var recordedAt = new DateTimeOffset(2025, 1, 1, 0, 0, 0, TimeSpan.Zero);
var provenance = new AdvisoryProvenance("source-x", "mapper", "payload-123", recordedAt); var provenance = new AdvisoryProvenance("source-x", "mapper", "payload-123", recordedAt);
@@ -148,6 +159,138 @@ public sealed class AdvisoryStoreTests : IClassFixture<MongoIntegrationFixture>
Assert.Equal(rangePrimitives.VendorExtensions, fetchedRange.Primitives.VendorExtensions); Assert.Equal(rangePrimitives.VendorExtensions, fetchedRange.Primitives.VendorExtensions);
} }
[Fact]
public async Task UpsertAsync_SkipsNormalizedVersionsWhenFeatureDisabled()
{
await DropCollectionAsync(MongoStorageDefaults.Collections.Advisory);
await DropCollectionAsync(MongoStorageDefaults.Collections.Alias);
var aliasStore = new AliasStore(_fixture.Database, NullLogger<AliasStore>.Instance);
var store = new AdvisoryStore(
_fixture.Database,
aliasStore,
NullLogger<AdvisoryStore>.Instance,
Options.Create(new MongoStorageOptions { EnableSemVerStyle = false }),
TimeProvider.System);
var advisory = CreateNormalizedAdvisory("ADV-NORM-DISABLED");
await store.UpsertAsync(advisory, CancellationToken.None);
var document = await _fixture.Database
.GetCollection<AdvisoryDocument>(MongoStorageDefaults.Collections.Advisory)
.Find(x => x.AdvisoryKey == advisory.AdvisoryKey)
.FirstOrDefaultAsync();
Assert.NotNull(document);
Assert.True(document!.NormalizedVersions is null || document.NormalizedVersions.Count == 0);
}
[Fact]
public async Task UpsertAsync_PopulatesNormalizedVersionsWhenFeatureEnabled()
{
await DropCollectionAsync(MongoStorageDefaults.Collections.Advisory);
await DropCollectionAsync(MongoStorageDefaults.Collections.Alias);
var aliasStore = new AliasStore(_fixture.Database, NullLogger<AliasStore>.Instance);
var store = new AdvisoryStore(
_fixture.Database,
aliasStore,
NullLogger<AdvisoryStore>.Instance,
Options.Create(new MongoStorageOptions { EnableSemVerStyle = true }),
TimeProvider.System);
var advisory = CreateNormalizedAdvisory("ADV-NORM-ENABLED");
await store.UpsertAsync(advisory, CancellationToken.None);
var document = await _fixture.Database
.GetCollection<AdvisoryDocument>(MongoStorageDefaults.Collections.Advisory)
.Find(x => x.AdvisoryKey == advisory.AdvisoryKey)
.FirstOrDefaultAsync();
Assert.NotNull(document);
var normalizedCollection = document!.NormalizedVersions;
Assert.NotNull(normalizedCollection);
var normalized = Assert.Single(normalizedCollection!);
Assert.Equal("pkg:npm/example", normalized.PackageId);
Assert.Equal(AffectedPackageTypes.SemVer, normalized.PackageType);
Assert.Equal(NormalizedVersionSchemes.SemVer, normalized.Scheme);
Assert.Equal(NormalizedVersionRuleTypes.Range, normalized.Type);
Assert.Equal("range", normalized.Style);
Assert.Equal("1.0.0", normalized.Min);
Assert.True(normalized.MinInclusive);
Assert.Equal("2.0.0", normalized.Max);
Assert.False(normalized.MaxInclusive);
Assert.Null(normalized.Value);
Assert.Equal("ghsa:pkg:npm/example", normalized.Notes);
Assert.Equal("range-decision", normalized.DecisionReason);
Assert.Equal(">= 1.0.0 < 2.0.0", normalized.Constraint);
Assert.Equal("ghsa", normalized.Source);
Assert.Equal(new DateTime(2025, 10, 9, 0, 0, 0, DateTimeKind.Utc), normalized.RecordedAtUtc);
}
private static Advisory CreateNormalizedAdvisory(string advisoryKey)
{
var recordedAt = new DateTimeOffset(2025, 10, 9, 0, 0, 0, TimeSpan.Zero);
var rangeProvenance = new AdvisoryProvenance(
source: "ghsa",
kind: "affected-range",
value: "pkg:npm/example",
recordedAt: recordedAt,
fieldMask: new[] { "affectedpackages[].versionranges[]" },
decisionReason: "range-decision");
var semverPrimitive = new SemVerPrimitive(
Introduced: "1.0.0",
IntroducedInclusive: true,
Fixed: "2.0.0",
FixedInclusive: false,
LastAffected: null,
LastAffectedInclusive: false,
ConstraintExpression: ">= 1.0.0 < 2.0.0");
var normalizedRule = semverPrimitive.ToNormalizedVersionRule("ghsa:pkg:npm/example")!;
var versionRange = new AffectedVersionRange(
rangeKind: "semver",
introducedVersion: "1.0.0",
fixedVersion: "2.0.0",
lastAffectedVersion: null,
rangeExpression: ">= 1.0.0 < 2.0.0",
provenance: rangeProvenance,
primitives: new RangePrimitives(semverPrimitive, null, null, null));
var package = new AffectedPackage(
type: AffectedPackageTypes.SemVer,
identifier: "pkg:npm/example",
platform: "npm",
versionRanges: new[] { versionRange },
statuses: Array.Empty<AffectedPackageStatus>(),
provenance: new[] { rangeProvenance },
normalizedVersions: new[] { normalizedRule });
var advisoryProvenance = new AdvisoryProvenance(
source: "ghsa",
kind: "document",
value: advisoryKey,
recordedAt: recordedAt,
fieldMask: new[] { "advisory" },
decisionReason: "document-decision");
return new Advisory(
advisoryKey: advisoryKey,
title: "Normalized advisory",
summary: "Contains normalized versions for storage testing.",
language: "en",
published: recordedAt,
modified: recordedAt,
severity: "medium",
exploitKnown: false,
aliases: new[] { $"{advisoryKey}-ALIAS" },
references: Array.Empty<AdvisoryReference>(),
affectedPackages: new[] { package },
cvssMetrics: Array.Empty<CvssMetric>(),
provenance: new[] { advisoryProvenance });
}
private async Task DropCollectionAsync(string collectionName) private async Task DropCollectionAsync(string collectionName)
{ {
try try

View File

@@ -0,0 +1,97 @@
using System;
using System.Linq;
using System.Threading;
using Microsoft.Extensions.Logging.Abstractions;
using Microsoft.Extensions.Options;
using MongoDB.Bson;
using MongoDB.Driver;
using StellaOps.Feedser.Storage.Mongo;
using StellaOps.Feedser.Storage.Mongo.Migrations;
using Xunit;
namespace StellaOps.Feedser.Storage.Mongo.Tests;
[Collection("mongo-fixture")]
public sealed class MongoBootstrapperTests : IClassFixture<MongoIntegrationFixture>
{
private readonly MongoIntegrationFixture _fixture;
public MongoBootstrapperTests(MongoIntegrationFixture fixture)
{
_fixture = fixture;
}
[Fact]
public async Task InitializeAsync_CreatesNormalizedIndexesWhenSemVerStyleEnabled()
{
var databaseName = $"feedser-bootstrap-semver-{Guid.NewGuid():N}";
var database = _fixture.Client.GetDatabase(databaseName);
try
{
var runner = new MongoMigrationRunner(
database,
Array.Empty<IMongoMigration>(),
NullLogger<MongoMigrationRunner>.Instance,
TimeProvider.System);
var bootstrapper = new MongoBootstrapper(
database,
Options.Create(new MongoStorageOptions { EnableSemVerStyle = true }),
NullLogger<MongoBootstrapper>.Instance,
runner);
await bootstrapper.InitializeAsync(CancellationToken.None);
var indexCursor = await database
.GetCollection<BsonDocument>(MongoStorageDefaults.Collections.Advisory)
.Indexes
.ListAsync();
var indexNames = (await indexCursor.ToListAsync()).Select(x => x["name"].AsString).ToArray();
Assert.Contains("advisory_normalizedVersions_pkg_scheme_type", indexNames);
Assert.Contains("advisory_normalizedVersions_value", indexNames);
}
finally
{
await _fixture.Client.DropDatabaseAsync(databaseName);
}
}
[Fact]
public async Task InitializeAsync_DoesNotCreateNormalizedIndexesWhenFeatureDisabled()
{
var databaseName = $"feedser-bootstrap-no-semver-{Guid.NewGuid():N}";
var database = _fixture.Client.GetDatabase(databaseName);
try
{
var runner = new MongoMigrationRunner(
database,
Array.Empty<IMongoMigration>(),
NullLogger<MongoMigrationRunner>.Instance,
TimeProvider.System);
var bootstrapper = new MongoBootstrapper(
database,
Options.Create(new MongoStorageOptions { EnableSemVerStyle = false }),
NullLogger<MongoBootstrapper>.Instance,
runner);
await bootstrapper.InitializeAsync(CancellationToken.None);
var indexCursor = await database
.GetCollection<BsonDocument>(MongoStorageDefaults.Collections.Advisory)
.Indexes
.ListAsync();
var indexNames = (await indexCursor.ToListAsync()).Select(x => x["name"].AsString).ToArray();
Assert.DoesNotContain("advisory_normalizedVersions_pkg_scheme_type", indexNames);
Assert.DoesNotContain("advisory_normalizedVersions_value", indexNames);
}
finally
{
await _fixture.Client.DropDatabaseAsync(databaseName);
}
}
}

View File

@@ -5,6 +5,7 @@ using System.Runtime.CompilerServices;
using System.Text.Json; using System.Text.Json;
using System.Text.Json.Serialization; using System.Text.Json.Serialization;
using Microsoft.Extensions.Logging; using Microsoft.Extensions.Logging;
using Microsoft.Extensions.Options;
using MongoDB.Bson; using MongoDB.Bson;
using MongoDB.Driver; using MongoDB.Driver;
using StellaOps.Feedser.Models; using StellaOps.Feedser.Models;
@@ -18,17 +19,20 @@ public sealed class AdvisoryStore : IAdvisoryStore
private readonly ILogger<AdvisoryStore> _logger; private readonly ILogger<AdvisoryStore> _logger;
private readonly IAliasStore _aliasStore; private readonly IAliasStore _aliasStore;
private readonly TimeProvider _timeProvider; private readonly TimeProvider _timeProvider;
private readonly MongoStorageOptions _options;
public AdvisoryStore( public AdvisoryStore(
IMongoDatabase database, IMongoDatabase database,
IAliasStore aliasStore, IAliasStore aliasStore,
ILogger<AdvisoryStore> logger, ILogger<AdvisoryStore> logger,
IOptions<MongoStorageOptions> options,
TimeProvider? timeProvider = null) TimeProvider? timeProvider = null)
{ {
_collection = (database ?? throw new ArgumentNullException(nameof(database))) _collection = (database ?? throw new ArgumentNullException(nameof(database)))
.GetCollection<AdvisoryDocument>(MongoStorageDefaults.Collections.Advisory); .GetCollection<AdvisoryDocument>(MongoStorageDefaults.Collections.Advisory);
_aliasStore = aliasStore ?? throw new ArgumentNullException(nameof(aliasStore)); _aliasStore = aliasStore ?? throw new ArgumentNullException(nameof(aliasStore));
_logger = logger ?? throw new ArgumentNullException(nameof(logger)); _logger = logger ?? throw new ArgumentNullException(nameof(logger));
_options = options?.Value ?? throw new ArgumentNullException(nameof(options));
_timeProvider = timeProvider ?? TimeProvider.System; _timeProvider = timeProvider ?? TimeProvider.System;
} }
@@ -51,7 +55,9 @@ public sealed class AdvisoryStore : IAdvisoryStore
} }
var payload = CanonicalJsonSerializer.Serialize(advisory); var payload = CanonicalJsonSerializer.Serialize(advisory);
var normalizedVersions = NormalizedVersionDocumentFactory.Create(advisory); var normalizedVersions = _options.EnableSemVerStyle
? NormalizedVersionDocumentFactory.Create(advisory)
: null;
var document = new AdvisoryDocument var document = new AdvisoryDocument
{ {
AdvisoryKey = advisory.AdvisoryKey, AdvisoryKey = advisory.AdvisoryKey,

View File

@@ -41,10 +41,10 @@ internal static class NormalizedVersionDocumentFactory
?? package.Provenance.FirstOrDefault()?.RecordedAt ?? package.Provenance.FirstOrDefault()?.RecordedAt
?? advisoryFallbackRecordedAt; ?? advisoryFallbackRecordedAt;
var constraint = matchingRange?.RangeExpression var constraint = matchingRange?.Primitives?.SemVer?.ConstraintExpression
?? matchingRange?.Primitives?.SemVer?.ConstraintExpression; ?? matchingRange?.RangeExpression;
var style = rule.Type; var style = matchingRange?.Primitives?.SemVer?.Style ?? rule.Type;
documents.Add(new NormalizedVersionDocument documents.Add(new NormalizedVersionDocument
{ {

View File

@@ -138,6 +138,20 @@ public sealed class MongoBootstrapper
new CreateIndexOptions { Name = "advisory_published_desc" }), new CreateIndexOptions { Name = "advisory_published_desc" }),
}; };
if (_options.EnableSemVerStyle)
{
indexes.Add(new CreateIndexModel<BsonDocument>(
Builders<BsonDocument>.IndexKeys
.Ascending("normalizedVersions.packageId")
.Ascending("normalizedVersions.scheme")
.Ascending("normalizedVersions.type"),
new CreateIndexOptions { Name = "advisory_normalizedVersions_pkg_scheme_type" }));
indexes.Add(new CreateIndexModel<BsonDocument>(
Builders<BsonDocument>.IndexKeys.Ascending("normalizedVersions.value"),
new CreateIndexOptions { Name = "advisory_normalizedVersions_value", Sparse = true }));
}
return collection.Indexes.CreateManyAsync(indexes, cancellationToken); return collection.Indexes.CreateManyAsync(indexes, cancellationToken);
} }

View File

@@ -84,6 +84,7 @@ public static class ServiceCollectionExtensions
services.AddSingleton<MongoMigrationRunner>(); services.AddSingleton<MongoMigrationRunner>();
services.AddSingleton<IMongoMigration, EnsureDocumentExpiryIndexesMigration>(); services.AddSingleton<IMongoMigration, EnsureDocumentExpiryIndexesMigration>();
services.AddSingleton<IMongoMigration, EnsureGridFsExpiryIndexesMigration>(); services.AddSingleton<IMongoMigration, EnsureGridFsExpiryIndexesMigration>();
services.AddSingleton<IMongoMigration, SemVerStyleBackfillMigration>();
return services; return services;
} }

View File

@@ -15,5 +15,7 @@
|Persist last failure reason in SourceState|BE-Storage|Storage.Mongo|DONE `MongoSourceStateRepository.MarkFailureAsync` stores `lastFailureReason` with length guard + reset on success.| |Persist last failure reason in SourceState|BE-Storage|Storage.Mongo|DONE `MongoSourceStateRepository.MarkFailureAsync` stores `lastFailureReason` with length guard + reset on success.|
|AdvisoryStore range primitives deserialization|BE-Storage|Models|DONE BSON helpers handle `RangePrimitives`; regression test covers SemVer/NEVRA/EVR envelopes persisted through Mongo.| |AdvisoryStore range primitives deserialization|BE-Storage|Models|DONE BSON helpers handle `RangePrimitives`; regression test covers SemVer/NEVRA/EVR envelopes persisted through Mongo.|
|FEEDSTORAGE-DATA-03-001 Merge event provenance audit prep|BE-Storage|Merge|DONE merge events now persist field-level decision reasons via `MergeFieldDecision` documents for analytics. **Coordination:** log any new precedence signals to storage@ so indexes/serializers stay aligned.| |FEEDSTORAGE-DATA-03-001 Merge event provenance audit prep|BE-Storage|Merge|DONE merge events now persist field-level decision reasons via `MergeFieldDecision` documents for analytics. **Coordination:** log any new precedence signals to storage@ so indexes/serializers stay aligned.|
|FEEDSTORAGE-DATA-02-001 Normalized range dual-write + backfill|BE-Storage|Core|DONE advisory documents store `normalizedVersions`, migration respects `EnableSemVerStyle`, and decision reasons flow into normalized write path. **Action:** connector owners confirm `EnableSemVerStyle=true` readiness before 2025-10-18 rollout.| |FEEDSTORAGE-DATA-02-001 Normalized range dual-write + backfill|BE-Storage|Core|**DONE (2025-10-12)** `AdvisoryStore` honors `EnableSemVerStyle`, dual-writes normalized docs, and SemVer backfill migration registered for staged rollout.|
|FEEDSTORAGE-TESTS-02-004 Restore AdvisoryStore build after normalized versions refactor|QA|Storage.Mongo|DONE storage tests updated to cover normalized version payloads and new provenance fields. **Heads-up:** QA to watch for fixture bumps touching normalized rule arrays when connectors roll out support.| |FEEDSTORAGE-TESTS-02-004 Restore AdvisoryStore build after normalized versions refactor|QA|Storage.Mongo|DONE storage tests updated to cover normalized version payloads and new provenance fields. **Heads-up:** QA to watch for fixture bumps touching normalized rule arrays when connectors roll out support.|
|FEEDSTORAGE-DATA-02-002 Provenance decision persistence|BE-Storage|Models `FEEDMODELS-SCHEMA-01-002`|**DONE (2025-10-12)** Normalized documents carry decision reasons/source/timestamps with regression coverage verifying SemVer notes + provenance fallbacks.|
|FEEDSTORAGE-DATA-02-003 Normalized versions index creation|BE-Storage|Normalization, Mongo bootstrapper|**DONE (2025-10-12)** Bootstrapper seeds `normalizedVersions.*` indexes when SemVer style is enabled; docs/tests confirm index presence.|