stella-ops.org/git.stella-ops.org
1
0
Fork 0
Code Issues Pull Requests Actions 16 Releases Wiki Activity

Align AOC tasks for Excititor and Concelier

Browse Source
This commit is contained in:
master
2025-10-31 18:50:15 +02:00
committed by root
parent 9e6d9fbae8
commit 8da4e12a90
334 changed files with 35528 additions and 34546 deletions
Download Patch File Download Diff File Expand all files Collapse all files

118
src/Concelier/__Libraries/StellaOps.Concelier.Connector.CertCc/FEEDCONN-CERTCC-02-009_PLAN.md
View File

@@ -1,59 +1,59 @@
# FEEDCONN-CERTCC-02-009 – VINCE Detail & Map Reintegration Plan
- **Author:** BE-Conn-CERTCC (current on-call)
- **Date:** 2025-10-11
- **Scope:** Restore VINCE detail parsing and canonical mapping in Concelier without destabilising downstream Merge/Export pipelines.
## 1. Current State Snapshot (2025-10-11)
- ✅ Fetch pipeline, VINCE summary planner, and detail queue are live; documents land with `DocumentStatuses.PendingParse`.
- ✅ DTO aggregate (`CertCcNoteDto`) plus mapper emit vendor-centric `normalizedVersions` (`scheme=certcc.vendor`) and provenance aligned with `src/Concelier/__Libraries/StellaOps.Concelier.Models/PROVENANCE_GUIDELINES.md`.
- ✅ Regression coverage exists for fetch/parse/map flows (`CertCcConnectorSnapshotTests`), but snapshot regeneration is gated on harness refresh (FEEDCONN-CERTCC-02-007) and QA handoff (FEEDCONN-CERTCC-02-008).
- ⚠️ Parse/map jobs are not scheduled; production still operates in fetch-only mode.
- ⚠️ Downstream Merge team is finalising normalized range ingestion per `src/FASTER_MODELING_AND_NORMALIZATION.md`; we must avoid publishing canonical records until they certify compatibility.
## 2. Required Dependencies & Coordinated Tasks
| Dependency | Owner(s) | Blocking Condition | Handshake |
|------------|----------|--------------------|-----------|
| FEEDCONN-CERTCC-02-004 (Canonical mapping & range primitives hardening) | BE-Conn-CERTCC + Models | Ensure mapper emits deterministic `normalizedVersions` array and provenance field masks | Daily sync with Models/Merge leads; share fixture diff before each enablement phase |
| FEEDCONN-CERTCC-02-007 (Connector test harness remediation) | BE-Conn-CERTCC, QA | Restore `AddSourceCommon` harness + canned VINCE fixtures so we can shadow-run parse/map | Required before Phase 1 |
| FEEDCONN-CERTCC-02-008 (Snapshot coverage handoff) | QA | Snapshot refresh process green to surface regressions | Required before Phase 2 |
| FEEDCONN-CERTCC-02-010 (Partial-detail graceful degradation) | BE-Conn-CERTCC | Resiliency for missing VINCE endpoints to avoid job wedging after reintegration | Should land before Phase 2 cutover |
## 3. Phased Rollout Plan
| Phase | Window (UTC) | Actions | Success Signals | Rollback |
|-------|--------------|---------|-----------------|----------|
| **0 – Pre-flight validation** | 2025-10-11 → 2025-10-12 | • Finish FEEDCONN-CERTCC-02-007 harness fixes and regenerate fixtures.<br>• Run `dotnet test src/Concelier/__Tests/StellaOps.Concelier.Connector.CertCc.Tests` with `UPDATE_CERTCC_FIXTURES=0` to confirm deterministic baselines.<br>• Generate sample advisory batch (`dotnet test … --filter SnapshotSmoke`) and deliver JSON diff to Merge for schema verification (`normalizedVersions[].scheme == certcc.vendor`, provenance masks populated). | • Harness tests green locally and in CI.<br>• Merge sign-off that sample advisories conform to `FASTER_MODELING_AND_NORMALIZATION.md`. | N/A (no production enablement yet). |
| **1 – Shadow parse/map in staging** | Target start 2025-10-13 | • Register `source:cert-cc:parse` and `source:cert-cc:map` jobs, but gate them behind new config flag `concelier:sources:cert-cc:enableDetailMapping` (default `false`).<br>• Deploy (restart required for options rebinding), enable flag, and point connector at staging Mongo with isolated collection (`advisories_certcc_shadow`).<br>• Run connector for ≥2 cycles; compare advisory counts vs. fetch-only baseline and validate `concelier.range.primitives` metrics include `scheme=certcc.vendor`. | • No uncaught exceptions in staging logs.<br>• Shadow advisories match expected vendor counts (±5%).<br>• `certcc.summary.fetch.*` + new `certcc.map.duration.ms` metrics stable. | Disable flag; staging returns to fetch-only. No production impact. |
| **2 – Controlled production enablement** | Target start 2025-10-14 | • Redeploy production with flag enabled, start with job concurrency `1`, and reduce `MaxNotesPerFetch` to 5 for first 24 h.<br>• Observe metrics dashboards hourly (fetch/map latency, pending queues, Mongo write throughput).<br>• QA to replay latest snapshots and confirm no deterministic drift.<br>• Publish advisory sample (top 10 changed docs) to Merge Slack channel for validation. | • Pending parse/mapping queues drain within expected SLA (<30 min).<br>• No increase in merge dedupe anomalies.<br>• Mongo writes stay within 10% of baseline. | Toggle flag off, re-run fetch-only. Clear `pendingMappings` via connector cursor reset if stuck. |
| **3 – Full production & cleanup** | Target start 2025-10-15 | • Restore `MaxNotesPerFetch` to configured default (20).<br>• Remove temporary throttles and leave flag enabled by default.<br>• Update `README.md` rollout notes; close FEEDCONN-CERTCC-02-009.<br>• Kick off post-merge audit with Merge to ensure new advisories dedupe with other sources. | • Stable operations for ≥48 h, no degradation alerts.<br>• Merge confirms conflict resolver behaviour unchanged. | If regression detected, revert to Phase 2 state or disable jobs; retain plan for reuse. |
## 4. Monitoring & Validation Checklist
- Dashboards: `certcc.*` meters (plan, summary fetch, detail fetch) plus `concelier.range.primitives` with tag `scheme=certcc.vendor`.
- Logs: ensure Parse/Map jobs emit `correlationId` aligned with fetch events for traceability.
- Data QA: run `src/Tools/dump_advisory` against two VINCE notes (one multi-vendor, one single-vendor) every phase to spot-check normalized versions ordering and provenance.
- Storage: verify Mongo TTL/size for `raw_documents` and `dtos`—detail payload volume increases by ~3× when mapping resumes.
## 5. Rollback / Contingency Playbook
1. Disable `concelier:sources:cert-cc:enableDetailMapping` flag (and optionally set `MaxNotesPerFetch=0` for a single cycle) to halt new detail ingestion.
2. Run connector once to update cursor; verify `pendingMappings` drains.
3. If advisories already persisted, coordinate with Merge to soft-delete affected `certcc/*` advisories by advisory key hash (no schema rollback required).
4. Re-run Phase 1 shadow validation before retrying.
## 6. Communication Cadence
- Daily check-in with Models/Merge leads (09:30 EDT) to surface normalizedVersions/provenance diffs.
- Post-phase reports in `#concelier-certcc` Slack channel summarising metrics, advisory counts, and outstanding issues.
- Escalate blockers >12 h via Runbook SEV-3 path and annotate `TASKS.md`.
## 7. Open Questions / Next Actions
- [ ] Confirm whether Merge requires additional provenance field masks before Phase 2 (waiting on feedback from 2025-10-11 sample).
- [ ] Decide if CSAF endpoint ingestion (optional) should piggyback on Phase 3 or stay deferred.
- [ ] Validate that FEEDCONN-CERTCC-02-010 coverage handles mixed 200/404 VINCE endpoints during partial outages.
Once Dependencies (Section 2) are cleared and Phase 3 completes, update `src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.CertCc/TASKS.md` and close FEEDCONN-CERTCC-02-009.
# FEEDCONN-CERTCC-02-009 – VINCE Detail & Map Reintegration Plan
- **Author:** BE-Conn-CERTCC (current on-call)
- **Date:** 2025-10-11
- **Scope:** Restore VINCE detail parsing and canonical mapping in Concelier without destabilising downstream Merge/Export pipelines.
## 1. Current State Snapshot (2025-10-11)
- ✅ Fetch pipeline, VINCE summary planner, and detail queue are live; documents land with `DocumentStatuses.PendingParse`.
- ✅ DTO aggregate (`CertCcNoteDto`) plus mapper emit vendor-centric `normalizedVersions` (`scheme=certcc.vendor`) and provenance aligned with `src/Concelier/__Libraries/StellaOps.Concelier.Models/PROVENANCE_GUIDELINES.md`.
- ✅ Regression coverage exists for fetch/parse/map flows (`CertCcConnectorSnapshotTests`), but snapshot regeneration is gated on harness refresh (FEEDCONN-CERTCC-02-007) and QA handoff (FEEDCONN-CERTCC-02-008).
- ⚠️ Parse/map jobs are not scheduled; production still operates in fetch-only mode.
- ⚠️ Downstream Merge team is finalising normalized range ingestion per `src/FASTER_MODELING_AND_NORMALIZATION.md`; we must avoid publishing canonical records until they certify compatibility.
## 2. Required Dependencies & Coordinated Tasks
| Dependency | Owner(s) | Blocking Condition | Handshake |
|------------|----------|--------------------|-----------|
| FEEDCONN-CERTCC-02-004 (Canonical mapping & range primitives hardening) | BE-Conn-CERTCC + Models | Ensure mapper emits deterministic `normalizedVersions` array and provenance field masks | Daily sync with Models/Merge leads; share fixture diff before each enablement phase |
| FEEDCONN-CERTCC-02-007 (Connector test harness remediation) | BE-Conn-CERTCC, QA | Restore `AddSourceCommon` harness + canned VINCE fixtures so we can shadow-run parse/map | Required before Phase 1 |
| FEEDCONN-CERTCC-02-008 (Snapshot coverage handoff) | QA | Snapshot refresh process green to surface regressions | Required before Phase 2 |
| FEEDCONN-CERTCC-02-010 (Partial-detail graceful degradation) | BE-Conn-CERTCC | Resiliency for missing VINCE endpoints to avoid job wedging after reintegration | Should land before Phase 2 cutover |
## 3. Phased Rollout Plan
| Phase | Window (UTC) | Actions | Success Signals | Rollback |
|-------|--------------|---------|-----------------|----------|
| **0 – Pre-flight validation** | 2025-10-11 → 2025-10-12 | • Finish FEEDCONN-CERTCC-02-007 harness fixes and regenerate fixtures.<br>• Run `dotnet test src/Concelier/__Tests/StellaOps.Concelier.Connector.CertCc.Tests` with `UPDATE_CERTCC_FIXTURES=0` to confirm deterministic baselines.<br>• Generate sample advisory batch (`dotnet test … --filter SnapshotSmoke`) and deliver JSON diff to Merge for schema verification (`normalizedVersions[].scheme == certcc.vendor`, provenance masks populated). | • Harness tests green locally and in CI.<br>• Merge sign-off that sample advisories conform to `FASTER_MODELING_AND_NORMALIZATION.md`. | N/A (no production enablement yet). |
| **1 – Shadow parse/map in staging** | Target start 2025-10-13 | • Register `source:cert-cc:parse` and `source:cert-cc:map` jobs, but gate them behind new config flag `concelier:sources:cert-cc:enableDetailMapping` (default `false`).<br>• Deploy (restart required for options rebinding), enable flag, and point connector at staging Mongo with isolated collection (`advisories_certcc_shadow`).<br>• Run connector for ≥2 cycles; compare advisory counts vs. fetch-only baseline and validate `concelier.range.primitives` metrics include `scheme=certcc.vendor`. | • No uncaught exceptions in staging logs.<br>• Shadow advisories match expected vendor counts (±5%).<br>• `certcc.summary.fetch.*` + new `certcc.map.duration.ms` metrics stable. | Disable flag; staging returns to fetch-only. No production impact. |
| **2 – Controlled production enablement** | Target start 2025-10-14 | • Redeploy production with flag enabled, start with job concurrency `1`, and reduce `MaxNotesPerFetch` to 5 for first 24 h.<br>• Observe metrics dashboards hourly (fetch/map latency, pending queues, Mongo write throughput).<br>• QA to replay latest snapshots and confirm no deterministic drift.<br>• Publish advisory sample (top 10 changed docs) to Merge Slack channel for validation. | • Pending parse/mapping queues drain within expected SLA (<30 min).<br>• No increase in merge dedupe anomalies.<br>• Mongo writes stay within 10% of baseline. | Toggle flag off, re-run fetch-only. Clear `pendingMappings` via connector cursor reset if stuck. |
| **3 – Full production & cleanup** | Target start 2025-10-15 | • Restore `MaxNotesPerFetch` to configured default (20).<br>• Remove temporary throttles and leave flag enabled by default.<br>• Update `README.md` rollout notes; close FEEDCONN-CERTCC-02-009.<br>• Kick off post-merge audit with Merge to ensure new advisories dedupe with other sources. | • Stable operations for ≥48 h, no degradation alerts.<br>• Merge confirms conflict resolver behaviour unchanged. | If regression detected, revert to Phase 2 state or disable jobs; retain plan for reuse. |
## 4. Monitoring & Validation Checklist
- Dashboards: `certcc.*` meters (plan, summary fetch, detail fetch) plus `concelier.range.primitives` with tag `scheme=certcc.vendor`.
- Logs: ensure Parse/Map jobs emit `correlationId` aligned with fetch events for traceability.
- Data QA: run `src/Tools/dump_advisory` against two VINCE notes (one multi-vendor, one single-vendor) every phase to spot-check normalized versions ordering and provenance.
- Storage: verify Mongo TTL/size for `raw_documents` and `dtos`—detail payload volume increases by ~3× when mapping resumes.
## 5. Rollback / Contingency Playbook
1. Disable `concelier:sources:cert-cc:enableDetailMapping` flag (and optionally set `MaxNotesPerFetch=0` for a single cycle) to halt new detail ingestion.
2. Run connector once to update cursor; verify `pendingMappings` drains.
3. If advisories already persisted, coordinate with Merge to soft-delete affected `certcc/*` advisories by advisory key hash (no schema rollback required).
4. Re-run Phase 1 shadow validation before retrying.
## 6. Communication Cadence
- Daily check-in with Models/Merge leads (09:30 EDT) to surface normalizedVersions/provenance diffs.
- Post-phase reports in `#concelier-certcc` Slack channel summarising metrics, advisory counts, and outstanding issues.
- Escalate blockers >12 h via Runbook SEV-3 path and annotate `TASKS.md`.
## 7. Open Questions / Next Actions
- [ ] Confirm whether Merge requires additional provenance field masks before Phase 2 (waiting on feedback from 2025-10-11 sample).
- [ ] Decide if CSAF endpoint ingestion (optional) should piggyback on Phase 3 or stay deferred.
- [ ] Validate that FEEDCONN-CERTCC-02-010 coverage handles mixed 200/404 VINCE endpoints during partial outages.
Once Dependencies (Section 2) are cleared and Phase 3 completes, update `src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.CertCc/TASKS.md` and close FEEDCONN-CERTCC-02-009.

98
src/Concelier/__Libraries/StellaOps.Concelier.Core/Linksets/AdvisoryObservationFactory.cs
View File

@@ -19,23 +19,25 @@ internal sealed class AdvisoryObservationFactory : IAdvisoryObservationFactory
ArgumentNullException.ThrowIfNull(rawDocument);
var source = CreateSource(rawDocument.Source, rawDocument.Upstream);
var upstream = CreateUpstream(rawDocument.Upstream);
var content = CreateContent(rawDocument.Content);
var linkset = CreateLinkset(rawDocument.Identifiers, rawDocument.Linkset);
var attributes = CreateAttributes(rawDocument);
var createdAt = (observedAt ?? rawDocument.Upstream.RetrievedAt).ToUniversalTime();
return new AdvisoryObservation(
var upstream = CreateUpstream(rawDocument.Upstream);
var content = CreateContent(rawDocument.Content);
var linkset = CreateLinkset(rawDocument.Identifiers, rawDocument.Linkset);
var rawLinkset = CreateRawLinkset(rawDocument.Identifiers, rawDocument.Linkset);
var attributes = CreateAttributes(rawDocument);
var createdAt = (observedAt ?? rawDocument.Upstream.RetrievedAt).ToUniversalTime();
return new AdvisoryObservation(
observationId: BuildObservationId(rawDocument),
tenant: rawDocument.Tenant,
source: source,
upstream: upstream,
content: content,
linkset: linkset,
createdAt: createdAt,
attributes: attributes);
}
upstream: upstream,
content: content,
linkset: linkset,
rawLinkset: rawLinkset,
createdAt: createdAt,
attributes: attributes);
}
private static AdvisoryObservationSource CreateSource(RawSourceMetadata source, RawUpstreamMetadata upstream)
{
@@ -110,16 +112,64 @@ internal sealed class AdvisoryObservationFactory : IAdvisoryObservationFactory
return JsonNode.Parse(document.RootElement.GetRawText()) ?? JsonNode.Parse("{}")!;
}
private static AdvisoryObservationLinkset CreateLinkset(RawIdentifiers identifiers, RawLinkset linkset)
{
var aliases = NormalizeAliases(identifiers, linkset);
var purls = NormalizePackageUrls(linkset.PackageUrls);
var cpes = NormalizeCpes(linkset.Cpes);
var references = NormalizeReferences(linkset.References);
return new AdvisoryObservationLinkset(aliases, purls, cpes, references);
}
private static AdvisoryObservationLinkset CreateLinkset(RawIdentifiers identifiers, RawLinkset linkset)
{
var aliases = NormalizeAliases(identifiers, linkset);
var purls = NormalizePackageUrls(linkset.PackageUrls);
var cpes = NormalizeCpes(linkset.Cpes);
var references = NormalizeReferences(linkset.References);
return new AdvisoryObservationLinkset(aliases, purls, cpes, references);
}
private static RawLinkset CreateRawLinkset(RawIdentifiers identifiers, RawLinkset linkset)
{
var aliasBuilder = ImmutableArray.CreateBuilder<string>();
if (!string.IsNullOrWhiteSpace(identifiers.PrimaryId))
{
aliasBuilder.Add(identifiers.PrimaryId);
}
if (!identifiers.Aliases.IsDefaultOrEmpty)
{
foreach (var alias in identifiers.Aliases)
{
if (!string.IsNullOrEmpty(alias))
{
aliasBuilder.Add(alias);
}
}
}
if (!linkset.Aliases.IsDefaultOrEmpty)
{
foreach (var alias in linkset.Aliases)
{
if (!string.IsNullOrEmpty(alias))
{
aliasBuilder.Add(alias);
}
}
}
static ImmutableArray<string> EnsureArray(ImmutableArray<string> values)
=> values.IsDefault ? ImmutableArray<string>.Empty : values;
static ImmutableArray<RawReference> EnsureReferences(ImmutableArray<RawReference> values)
=> values.IsDefault ? ImmutableArray<RawReference>.Empty : values;
return linkset with
{
Aliases = aliasBuilder.ToImmutable(),
PackageUrls = EnsureArray(linkset.PackageUrls),
Cpes = EnsureArray(linkset.Cpes),
References = EnsureReferences(linkset.References),
ReconciledFrom = EnsureArray(linkset.ReconciledFrom),
Notes = linkset.Notes ?? ImmutableDictionary<string, string>.Empty
};
}
private static IEnumerable<string> NormalizeAliases(RawIdentifiers identifiers, RawLinkset linkset)
{
var aliases = new HashSet<string>(StringComparer.OrdinalIgnoreCase);

111
src/Concelier/__Libraries/StellaOps.Concelier.Models/Observations/AdvisoryObservation.cs
View File

@@ -1,6 +1,7 @@
using System.Collections.Immutable;
using System.Text.Json;
using System.Text.Json.Nodes;
using System.Collections.Immutable;
using System.Text.Json;
using System.Text.Json.Nodes;
using StellaOps.Concelier.RawModels;
namespace StellaOps.Concelier.Models.Observations;
@@ -12,19 +13,21 @@ public sealed record AdvisoryObservation
AdvisoryObservationSource source,
AdvisoryObservationUpstream upstream,
AdvisoryObservationContent content,
AdvisoryObservationLinkset linkset,
DateTimeOffset createdAt,
ImmutableDictionary<string, string>? attributes = null)
{
ObservationId = Validation.EnsureNotNullOrWhiteSpace(observationId, nameof(observationId));
Tenant = Validation.EnsureNotNullOrWhiteSpace(tenant, nameof(tenant)).ToLowerInvariant();
Source = source ?? throw new ArgumentNullException(nameof(source));
Upstream = upstream ?? throw new ArgumentNullException(nameof(upstream));
Content = content ?? throw new ArgumentNullException(nameof(content));
Linkset = linkset ?? throw new ArgumentNullException(nameof(linkset));
CreatedAt = createdAt.ToUniversalTime();
Attributes = NormalizeAttributes(attributes);
}
AdvisoryObservationLinkset linkset,
RawLinkset rawLinkset,
DateTimeOffset createdAt,
ImmutableDictionary<string, string>? attributes = null)
{
ObservationId = Validation.EnsureNotNullOrWhiteSpace(observationId, nameof(observationId));
Tenant = Validation.EnsureNotNullOrWhiteSpace(tenant, nameof(tenant)).ToLowerInvariant();
Source = source ?? throw new ArgumentNullException(nameof(source));
Upstream = upstream ?? throw new ArgumentNullException(nameof(upstream));
Content = content ?? throw new ArgumentNullException(nameof(content));
Linkset = linkset ?? throw new ArgumentNullException(nameof(linkset));
RawLinkset = SanitizeRawLinkset(rawLinkset);
CreatedAt = createdAt.ToUniversalTime();
Attributes = NormalizeAttributes(attributes);
}
public string ObservationId { get; }
@@ -34,15 +37,17 @@ public sealed record AdvisoryObservation
public AdvisoryObservationUpstream Upstream { get; }
public AdvisoryObservationContent Content { get; }
public AdvisoryObservationLinkset Linkset { get; }
public DateTimeOffset CreatedAt { get; }
public ImmutableDictionary<string, string> Attributes { get; }
private static ImmutableDictionary<string, string> NormalizeAttributes(ImmutableDictionary<string, string>? attributes)
public AdvisoryObservationContent Content { get; }
public AdvisoryObservationLinkset Linkset { get; }
public RawLinkset RawLinkset { get; }
public DateTimeOffset CreatedAt { get; }
public ImmutableDictionary<string, string> Attributes { get; }
private static ImmutableDictionary<string, string> NormalizeAttributes(ImmutableDictionary<string, string>? attributes)
{
if (attributes is null || attributes.Count == 0)
{
@@ -59,10 +64,58 @@ public sealed record AdvisoryObservation
builder[pair.Key.Trim()] = pair.Value;
}
return builder.ToImmutable();
}
}
return builder.ToImmutable();
}
private static RawLinkset SanitizeRawLinkset(RawLinkset? rawLinkset)
{
if (rawLinkset is null)
{
return new RawLinkset();
}
static ImmutableArray<string> SanitizeStrings(ImmutableArray<string> values)
{
if (values.IsDefault)
{
return ImmutableArray<string>.Empty;
}
return values;
}
static ImmutableArray<RawReference> SanitizeReferences(ImmutableArray<RawReference> references)
{
if (references.IsDefault)
{
return ImmutableArray<RawReference>.Empty;
}
return references;
}
static ImmutableDictionary<string, string> SanitizeNotes(ImmutableDictionary<string, string>? notes)
{
if (notes is null || notes.Count == 0)
{
return ImmutableDictionary<string, string>.Empty;
}
return notes;
}
return rawLinkset with
{
Aliases = SanitizeStrings(rawLinkset.Aliases),
PackageUrls = SanitizeStrings(rawLinkset.PackageUrls),
Cpes = SanitizeStrings(rawLinkset.Cpes),
References = SanitizeReferences(rawLinkset.References),
ReconciledFrom = SanitizeStrings(rawLinkset.ReconciledFrom),
Notes = SanitizeNotes(rawLinkset.Notes)
};
}
}
public sealed record AdvisoryObservationSource
{

25
src/Concelier/__Libraries/StellaOps.Concelier.Models/StellaOps.Concelier.Models.csproj
View File

@@ -1,12 +1,15 @@
<Project Sdk="Microsoft.NET.Sdk">
<PropertyGroup>
<TargetFramework>net10.0</TargetFramework>
<LangVersion>preview</LangVersion>
<Nullable>enable</Nullable>
<ImplicitUsings>enable</ImplicitUsings>
<TreatWarningsAsErrors>true</TreatWarningsAsErrors>
</PropertyGroup>
<ItemGroup>
<PackageReference Include="Microsoft.Extensions.Logging.Abstractions" Version="10.0.0-rc.2.25502.107" />
</ItemGroup>
</Project>
<PropertyGroup>
<TargetFramework>net10.0</TargetFramework>
<LangVersion>preview</LangVersion>
<Nullable>enable</Nullable>
<ImplicitUsings>enable</ImplicitUsings>
<TreatWarningsAsErrors>true</TreatWarningsAsErrors>
</PropertyGroup>
<ItemGroup>
<PackageReference Include="Microsoft.Extensions.Logging.Abstractions" Version="10.0.0-rc.2.25502.107" />
</ItemGroup>
<ItemGroup>
<ProjectReference Include="..\StellaOps.Concelier.RawModels\StellaOps.Concelier.RawModels.csproj" />
</ItemGroup>
</Project>

1
src/Concelier/__Libraries/StellaOps.Concelier.Storage.Mongo/MIGRATIONS.md
View File

@@ -26,6 +26,7 @@ This module owns the persistent shape of Concelier's MongoDB database. Upgrades
| `20251028_advisory_raw_idempotency_index` | Applies compound unique index on `(source.vendor, upstream.upstream_id, upstream.content_hash, tenant)` after verifying no duplicates exist. |
| `20251028_advisory_supersedes_backfill` | Renames legacy `advisory` collection to a read-only backup view and backfills `supersedes` chains across `advisory_raw`. |
| `20251028_advisory_raw_validator` | Applies Aggregation-Only Contract JSON schema validator to the `advisory_raw` collection with configurable enforcement level. |
| `20251104_advisory_observations_raw_linkset` | Backfills `rawLinkset` on `advisory_observations` using stored `advisory_raw` documents so canonical and raw projections co-exist for downstream policy joins. |
## Operator Runbook

442
src/Concelier/__Libraries/StellaOps.Concelier.Storage.Mongo/Migrations/EnsureAdvisoryObservationsRawLinksetMigration.cs Normal file
View File

@@ -0,0 +1,442 @@
using System;
using System.Collections.Generic;
using System.Collections.Immutable;
using System.Globalization;
using System.Linq;
using System.Text.Json;
using System.Threading;
using System.Threading.Tasks;
using MongoDB.Bson;
using MongoDB.Bson.IO;
using MongoDB.Driver;
using StellaOps.Concelier.RawModels;
namespace StellaOps.Concelier.Storage.Mongo.Migrations;
/// <summary>
/// Backfills the raw linkset projection on advisory observations so downstream services
/// can rely on both canonical and raw linkset shapes.
/// </summary>
public sealed class EnsureAdvisoryObservationsRawLinksetMigration : IMongoMigration
{
private const string MigrationId = "20251104_advisory_observations_raw_linkset";
private const int BulkBatchSize = 500;
public string Id => MigrationId;
public string Description => "Populate rawLinkset field for advisory observations using stored advisory_raw documents.";
public async Task ApplyAsync(IMongoDatabase database, CancellationToken cancellationToken)
{
ArgumentNullException.ThrowIfNull(database);
var observations = database.GetCollection<BsonDocument>(MongoStorageDefaults.Collections.AdvisoryObservations);
var rawCollection = database.GetCollection<BsonDocument>(MongoStorageDefaults.Collections.AdvisoryRaw);
var filter = Builders<BsonDocument>.Filter.Exists("rawLinkset", false) |
Builders<BsonDocument>.Filter.Type("rawLinkset", BsonType.Null);
using var cursor = await observations
.Find(filter)
.ToCursorAsync(cancellationToken)
.ConfigureAwait(false);
var updates = new List<WriteModel<BsonDocument>>(BulkBatchSize);
var missingRawDocuments = new List<string>();
while (await cursor.MoveNextAsync(cancellationToken).ConfigureAwait(false))
{
foreach (var observationDocument in cursor.Current)
{
cancellationToken.ThrowIfCancellationRequested();
if (!TryExtractObservationKey(observationDocument, out var key))
{
continue;
}
var rawFilter = Builders<BsonDocument>.Filter.Eq("tenant", key.Tenant) &
Builders<BsonDocument>.Filter.Eq("source.vendor", key.Vendor) &
Builders<BsonDocument>.Filter.Eq("upstream.upstream_id", key.UpstreamId) &
Builders<BsonDocument>.Filter.Eq("upstream.content_hash", key.ContentHash);
var rawDocument = await rawCollection
.Find(rawFilter)
.Sort(Builders<BsonDocument>.Sort.Descending("ingested_at").Descending("_id"))
.Limit(1)
.FirstOrDefaultAsync(cancellationToken)
.ConfigureAwait(false);
if (rawDocument is null)
{
missingRawDocuments.Add(key.ToString());
continue;
}
var advisoryRaw = MapToRawDocument(rawDocument);
var rawLinkset = BuildRawLinkset(advisoryRaw.Identifiers, advisoryRaw.Linkset);
var rawLinksetDocument = BuildRawLinksetBson(rawLinkset);
var update = Builders<BsonDocument>.Update.Set("rawLinkset", rawLinksetDocument);
updates.Add(new UpdateOneModel<BsonDocument>(
Builders<BsonDocument>.Filter.Eq("_id", observationDocument["_id"].AsString),
update));
if (updates.Count >= BulkBatchSize)
{
await observations.BulkWriteAsync(updates, cancellationToken: cancellationToken).ConfigureAwait(false);
updates.Clear();
}
}
}
if (updates.Count > 0)
{
await observations.BulkWriteAsync(updates, cancellationToken: cancellationToken).ConfigureAwait(false);
}
if (missingRawDocuments.Count > 0)
{
throw new InvalidOperationException(
$"Unable to locate advisory_raw documents for {missingRawDocuments.Count} observations: {string.Join(", ", missingRawDocuments.Take(10))}");
}
}
private static bool TryExtractObservationKey(BsonDocument observation, out ObservationKey key)
{
key = default;
if (!observation.TryGetValue("tenant", out var tenantValue) || tenantValue.IsBsonNull)
{
return false;
}
if (!observation.TryGetValue("source", out var sourceValue) || sourceValue is not BsonDocument sourceDocument)
{
return false;
}
if (!observation.TryGetValue("upstream", out var upstreamValue) || upstreamValue is not BsonDocument upstreamDocument)
{
return false;
}
var tenant = tenantValue.AsString;
var vendor = sourceDocument.GetValue("vendor", BsonString.Empty).AsString;
var upstreamId = upstreamDocument.GetValue("upstream_id", BsonString.Empty).AsString;
var contentHash = upstreamDocument.GetValue("contentHash", BsonString.Empty).AsString;
var createdAt = observation.GetValue("createdAt", BsonNull.Value);
key = new ObservationKey(
tenant,
vendor,
upstreamId,
contentHash,
BsonValueToDateTimeOffset(createdAt) ?? DateTimeOffset.UtcNow);
return !string.IsNullOrWhiteSpace(tenant) &&
!string.IsNullOrWhiteSpace(vendor) &&
!string.IsNullOrWhiteSpace(upstreamId) &&
!string.IsNullOrWhiteSpace(contentHash);
}
private static AdvisoryRawDocument MapToRawDocument(BsonDocument document)
{
var tenant = GetRequiredString(document, "tenant");
var source = MapSource(document["source"].AsBsonDocument);
var upstream = MapUpstream(document["upstream"].AsBsonDocument);
var content = MapContent(document["content"].AsBsonDocument);
var identifiers = MapIdentifiers(document["identifiers"].AsBsonDocument);
var linkset = MapLinkset(document["linkset"].AsBsonDocument);
var supersedes = document.GetValue("supersedes", BsonNull.Value);
return new AdvisoryRawDocument(
tenant,
source,
upstream,
content,
identifiers,
linkset,
supersedes.IsBsonNull ? null : supersedes.AsString);
}
private static RawSourceMetadata MapSource(BsonDocument source)
{
return new RawSourceMetadata(
GetRequiredString(source, "vendor"),
GetRequiredString(source, "connector"),
GetRequiredString(source, "version"),
GetOptionalString(source, "stream"));
}
private static RawUpstreamMetadata MapUpstream(BsonDocument upstream)
{
var provenanceBuilder = ImmutableDictionary.CreateBuilder<string, string>(StringComparer.Ordinal);
if (upstream.TryGetValue("provenance", out var provenanceValue) && provenanceValue.IsBsonDocument)
{
foreach (var element in provenanceValue.AsBsonDocument)
{
provenanceBuilder[element.Name] = BsonValueToString(element.Value);
}
}
var signatureDocument = upstream["signature"].AsBsonDocument;
var signature = new RawSignatureMetadata(
signatureDocument.GetValue("present", BsonBoolean.False).AsBoolean,
signatureDocument.TryGetValue("format", out var format) && !format.IsBsonNull ? format.AsString : null,
signatureDocument.TryGetValue("key_id", out var keyId) && !keyId.IsBsonNull ? keyId.AsString : null,
signatureDocument.TryGetValue("sig", out var sig) && !sig.IsBsonNull ? sig.AsString : null,
signatureDocument.TryGetValue("certificate", out var certificate) && !certificate.IsBsonNull ? certificate.AsString : null,
signatureDocument.TryGetValue("digest", out var digest) && !digest.IsBsonNull ? digest.AsString : null);
return new RawUpstreamMetadata(
GetRequiredString(upstream, "upstream_id"),
upstream.TryGetValue("document_version", out var version) && !version.IsBsonNull ? version.AsString : null,
GetDateTimeOffset(upstream, "retrieved_at", DateTimeOffset.UtcNow),
GetRequiredString(upstream, "content_hash"),
signature,
provenanceBuilder.ToImmutable());
}
private static RawContent MapContent(BsonDocument content)
{
var rawValue = content.GetValue("raw", BsonNull.Value);
string rawJson;
if (rawValue.IsBsonNull)
{
rawJson = "{}";
}
else if (rawValue.IsString)
{
rawJson = rawValue.AsString ?? "{}";
}
else
{
rawJson = rawValue.ToJson(new JsonWriterSettings { OutputMode = JsonOutputMode.RelaxedExtendedJson });
}
using var document = JsonDocument.Parse(string.IsNullOrWhiteSpace(rawJson) ? "{}" : rawJson);
return new RawContent(
GetRequiredString(content, "format"),
content.TryGetValue("spec_version", out var specVersion) && !specVersion.IsBsonNull ? specVersion.AsString : null,
document.RootElement.Clone(),
content.TryGetValue("encoding", out var encoding) && !encoding.IsBsonNull ? encoding.AsString : null);
}
private static RawIdentifiers MapIdentifiers(BsonDocument identifiers)
{
var aliases = identifiers.TryGetValue("aliases", out var aliasesValue) && aliasesValue.IsBsonArray
? aliasesValue.AsBsonArray.Select(BsonValueToString).ToImmutableArray()
: ImmutableArray<string>.Empty;
return new RawIdentifiers(
aliases,
GetRequiredString(identifiers, "primary"));
}
private static RawLinkset MapLinkset(BsonDocument linkset)
{
var aliases = linkset.TryGetValue("aliases", out var aliasesValue) && aliasesValue.IsBsonArray
? aliasesValue.AsBsonArray.Select(BsonValueToString).ToImmutableArray()
: ImmutableArray<string>.Empty;
var purls = linkset.TryGetValue("purls", out var purlsValue) && purlsValue.IsBsonArray
? purlsValue.AsBsonArray.Select(BsonValueToString).ToImmutableArray()
: ImmutableArray<string>.Empty;
var cpes = linkset.TryGetValue("cpes", out var cpesValue) && cpesValue.IsBsonArray
? cpesValue.AsBsonArray.Select(BsonValueToString).ToImmutableArray()
: ImmutableArray<string>.Empty;
var references = linkset.TryGetValue("references", out var referencesValue) && referencesValue.IsBsonArray
? referencesValue.AsBsonArray
.Where(static value => value.IsBsonDocument)
.Select(value =>
{
var doc = value.AsBsonDocument;
return new RawReference(
GetRequiredString(doc, "type"),
GetRequiredString(doc, "url"),
doc.TryGetValue("source", out var source) && !source.IsBsonNull ? source.AsString : null);
})
.ToImmutableArray()
: ImmutableArray<RawReference>.Empty;
var reconciledFrom = linkset.TryGetValue("reconciled_from", out var reconciledValue) && reconciledValue.IsBsonArray
? reconciledValue.AsBsonArray.Select(BsonValueToString).ToImmutableArray()
: ImmutableArray<string>.Empty;
var notesBuilder = ImmutableDictionary.CreateBuilder<string, string>(StringComparer.Ordinal);
if (linkset.TryGetValue("notes", out var notesValue) && notesValue.IsBsonDocument)
{
foreach (var element in notesValue.AsBsonDocument)
{
notesBuilder[element.Name] = BsonValueToString(element.Value);
}
}
return new RawLinkset
{
Aliases = aliases,
PackageUrls = purls,
Cpes = cpes,
References = references,
ReconciledFrom = reconciledFrom,
Notes = notesBuilder.ToImmutable()
};
}
private static RawLinkset BuildRawLinkset(RawIdentifiers identifiers, RawLinkset linkset)
{
var aliasBuilder = ImmutableArray.CreateBuilder<string>();
if (!string.IsNullOrWhiteSpace(identifiers.PrimaryId))
{
aliasBuilder.Add(identifiers.PrimaryId);
}
if (!identifiers.Aliases.IsDefaultOrEmpty)
{
foreach (var alias in identifiers.Aliases)
{
if (!string.IsNullOrEmpty(alias))
{
aliasBuilder.Add(alias);
}
}
}
if (!linkset.Aliases.IsDefaultOrEmpty)
{
foreach (var alias in linkset.Aliases)
{
if (!string.IsNullOrEmpty(alias))
{
aliasBuilder.Add(alias);
}
}
}
static ImmutableArray<string> EnsureArray(ImmutableArray<string> values)
=> values.IsDefault ? ImmutableArray<string>.Empty : values;
static ImmutableArray<RawReference> EnsureReferences(ImmutableArray<RawReference> values)
=> values.IsDefault ? ImmutableArray<RawReference>.Empty : values;
return linkset with
{
Aliases = aliasBuilder.ToImmutable(),
PackageUrls = EnsureArray(linkset.PackageUrls),
Cpes = EnsureArray(linkset.Cpes),
References = EnsureReferences(linkset.References),
ReconciledFrom = EnsureArray(linkset.ReconciledFrom),
Notes = linkset.Notes ?? ImmutableDictionary<string, string>.Empty
};
}
private static BsonDocument BuildRawLinksetBson(RawLinkset rawLinkset)
{
var references = new BsonArray(rawLinkset.References.Select(reference =>
{
var referenceDocument = new BsonDocument
{
{ "type", reference.Type },
{ "url", reference.Url }
};
if (!string.IsNullOrWhiteSpace(reference.Source))
{
referenceDocument["source"] = reference.Source;
}
return referenceDocument;
}));
var notes = new BsonDocument();
if (rawLinkset.Notes is not null)
{
foreach (var entry in rawLinkset.Notes)
{
notes[entry.Key] = entry.Value;
}
}
return new BsonDocument
{
{ "aliases", new BsonArray(rawLinkset.Aliases) },
{ "purls", new BsonArray(rawLinkset.PackageUrls) },
{ "cpes", new BsonArray(rawLinkset.Cpes) },
{ "references", references },
{ "reconciled_from", new BsonArray(rawLinkset.ReconciledFrom) },
{ "notes", notes }
};
}
private static string GetRequiredString(BsonDocument document, string key)
{
if (!document.TryGetValue(key, out var value) || value.IsBsonNull)
{
return string.Empty;
}
return value.IsString ? value.AsString : value.ToString() ?? string.Empty;
}
private static string? GetOptionalString(BsonDocument document, string key)
{
if (!document.TryGetValue(key, out var value) || value.IsBsonNull)
{
return null;
}
return value.IsString ? value.AsString : value.ToString();
}
private static string BsonValueToString(BsonValue value)
{
if (value.IsString)
{
return value.AsString ?? string.Empty;
}
if (value.IsBsonNull)
{
return string.Empty;
}
return value.ToString() ?? string.Empty;
}
private static DateTimeOffset GetDateTimeOffset(BsonDocument document, string field, DateTimeOffset fallback)
{
if (!document.TryGetValue(field, out var value) || value.IsBsonNull)
{
return fallback;
}
return BsonValueToDateTimeOffset(value) ?? fallback;
}
private static DateTimeOffset? BsonValueToDateTimeOffset(BsonValue value)
{
return value.BsonType switch
{
BsonType.DateTime => new DateTimeOffset(DateTime.SpecifyKind(value.ToUniversalTime(), DateTimeKind.Utc)),
BsonType.String when DateTimeOffset.TryParse(value.AsString, CultureInfo.InvariantCulture, DateTimeStyles.AssumeUniversal, out var parsed)
=> parsed.ToUniversalTime(),
BsonType.Int64 => DateTimeOffset.FromUnixTimeMilliseconds(value.AsInt64).ToUniversalTime(),
_ => null
};
}
private readonly record struct ObservationKey(
string Tenant,
string Vendor,
string UpstreamId,
string ContentHash,
DateTimeOffset CreatedAt)
{
public override string ToString()
=> $"{Tenant}:{Vendor}:{UpstreamId}:{ContentHash}";
}
}

92
src/Concelier/__Libraries/StellaOps.Concelier.Storage.Mongo/Observations/AdvisoryObservationDocument.cs
View File

@@ -24,11 +24,16 @@ public sealed class AdvisoryObservationDocument
public AdvisoryObservationContentDocument Content { get; set; } = new();
[BsonElement("linkset")]
public AdvisoryObservationLinksetDocument Linkset { get; set; } = new();
[BsonElement("createdAt")]
public DateTime CreatedAt { get; set; }
= DateTime.UtcNow;
public AdvisoryObservationLinksetDocument Linkset { get; set; } = new();
[BsonElement("rawLinkset")]
[BsonIgnoreIfNull]
public AdvisoryObservationRawLinksetDocument? RawLinkset { get; set; }
= null;
[BsonElement("createdAt")]
public DateTime CreatedAt { get; set; }
= DateTime.UtcNow;
[BsonElement("attributes")]
[BsonIgnoreIfNull]
@@ -129,11 +134,11 @@ public sealed class AdvisoryObservationContentDocument
}
[BsonIgnoreExtraElements]
public sealed class AdvisoryObservationLinksetDocument
{
[BsonElement("aliases")]
[BsonIgnoreIfNull]
public List<string>? Aliases { get; set; }
public sealed class AdvisoryObservationLinksetDocument
{
[BsonElement("aliases")]
[BsonIgnoreIfNull]
public List<string>? Aliases { get; set; }
= new();
[BsonElement("purls")]
@@ -153,11 +158,62 @@ public sealed class AdvisoryObservationLinksetDocument
}
[BsonIgnoreExtraElements]
public sealed class AdvisoryObservationReferenceDocument
{
[BsonElement("type")]
public string Type { get; set; } = string.Empty;
[BsonElement("url")]
public string Url { get; set; } = string.Empty;
}
public sealed class AdvisoryObservationReferenceDocument
{
[BsonElement("type")]
public string Type { get; set; } = string.Empty;
[BsonElement("url")]
public string Url { get; set; } = string.Empty;
}
[BsonIgnoreExtraElements]
public sealed class AdvisoryObservationRawLinksetDocument
{
[BsonElement("aliases")]
[BsonIgnoreIfNull]
public List<string>? Aliases { get; set; }
= new();
[BsonElement("purls")]
[BsonIgnoreIfNull]
public List<string>? PackageUrls { get; set; }
= new();
[BsonElement("cpes")]
[BsonIgnoreIfNull]
public List<string>? Cpes { get; set; }
= new();
[BsonElement("references")]
[BsonIgnoreIfNull]
public List<AdvisoryObservationRawReferenceDocument>? References { get; set; }
= new();
[BsonElement("reconciled_from")]
[BsonIgnoreIfNull]
public List<string>? ReconciledFrom { get; set; }
= new();
[BsonElement("notes")]
[BsonIgnoreIfNull]
public Dictionary<string, string>? Notes { get; set; }
= new(StringComparer.Ordinal);
}
[BsonIgnoreExtraElements]
public sealed class AdvisoryObservationRawReferenceDocument
{
[BsonElement("type")]
[BsonIgnoreIfNull]
public string? Type { get; set; }
= null;
[BsonElement("url")]
public string Url { get; set; } = string.Empty;
[BsonElement("source")]
[BsonIgnoreIfNull]
public string? Source { get; set; }
= null;
}

120
src/Concelier/__Libraries/StellaOps.Concelier.Storage.Mongo/Observations/AdvisoryObservationDocumentFactory.cs
View File

@@ -5,7 +5,8 @@ using System.Linq;
using System.Text.Json.Nodes;
using MongoDB.Bson;
using MongoDB.Bson.IO;
using StellaOps.Concelier.Models.Observations;
using StellaOps.Concelier.Models.Observations;
using StellaOps.Concelier.RawModels;
namespace StellaOps.Concelier.Storage.Mongo.Observations;
@@ -22,12 +23,14 @@ internal static class AdvisoryObservationDocumentFactory
var contentMetadata = ToImmutable(document.Content.Metadata);
var upstreamMetadata = ToImmutable(document.Upstream.Metadata);
var observation = new AdvisoryObservation(
document.Id,
document.Tenant,
new AdvisoryObservationSource(
document.Source.Vendor,
document.Source.Stream,
var rawLinkset = ToRawLinkset(document.RawLinkset);
var observation = new AdvisoryObservation(
document.Id,
document.Tenant,
new AdvisoryObservationSource(
document.Source.Vendor,
document.Source.Stream,
document.Source.Api,
document.Source.CollectorVersion),
new AdvisoryObservationUpstream(
@@ -42,21 +45,22 @@ internal static class AdvisoryObservationDocumentFactory
document.Upstream.Signature.KeyId,
document.Upstream.Signature.Signature),
upstreamMetadata),
new AdvisoryObservationContent(
document.Content.Format,
document.Content.SpecVersion,
rawNode,
contentMetadata),
new AdvisoryObservationLinkset(
document.Linkset.Aliases ?? Enumerable.Empty<string>(),
document.Linkset.Purls ?? Enumerable.Empty<string>(),
document.Linkset.Cpes ?? Enumerable.Empty<string>(),
document.Linkset.References?.Select(reference => new AdvisoryObservationReference(reference.Type, reference.Url))),
DateTime.SpecifyKind(document.CreatedAt, DateTimeKind.Utc),
attributes);
return observation;
}
new AdvisoryObservationContent(
document.Content.Format,
document.Content.SpecVersion,
rawNode,
contentMetadata),
new AdvisoryObservationLinkset(
document.Linkset.Aliases ?? Enumerable.Empty<string>(),
document.Linkset.Purls ?? Enumerable.Empty<string>(),
document.Linkset.Cpes ?? Enumerable.Empty<string>(),
document.Linkset.References?.Select(reference => new AdvisoryObservationReference(reference.Type, reference.Url))),
rawLinkset,
DateTime.SpecifyKind(document.CreatedAt, DateTimeKind.Utc),
attributes);
return observation;
}
private static JsonNode ParseJsonNode(BsonDocument raw)
{
@@ -87,6 +91,72 @@ internal static class AdvisoryObservationDocumentFactory
builder[pair.Key.Trim()] = pair.Value;
}
return builder.ToImmutable();
}
}
return builder.ToImmutable();
}
private static RawLinkset ToRawLinkset(AdvisoryObservationRawLinksetDocument? document)
{
if (document is null)
{
return new RawLinkset();
}
static ImmutableArray<string> ToImmutableStringArray(List<string>? values)
{
if (values is null || values.Count == 0)
{
return ImmutableArray<string>.Empty;
}
return values
.Select(static value => value ?? string.Empty)
.ToImmutableArray();
}
static ImmutableArray<RawReference> ToImmutableReferences(List<AdvisoryObservationRawReferenceDocument>? references)
{
if (references is null || references.Count == 0)
{
return ImmutableArray<RawReference>.Empty;
}
return references
.Select(static reference => new RawReference(
reference.Type ?? string.Empty,
reference.Url,
reference.Source))
.ToImmutableArray();
}
static ImmutableDictionary<string, string> ToImmutableDictionary(Dictionary<string, string>? values)
{
if (values is null || values.Count == 0)
{
return ImmutableDictionary<string, string>.Empty;
}
var builder = ImmutableDictionary.CreateBuilder<string, string>(StringComparer.Ordinal);
foreach (var pair in values)
{
if (pair.Key is null)
{
continue;
}
builder[pair.Key] = pair.Value;
}
return builder.ToImmutable();
}
return new RawLinkset
{
Aliases = ToImmutableStringArray(document.Aliases),
PackageUrls = ToImmutableStringArray(document.PackageUrls),
Cpes = ToImmutableStringArray(document.Cpes),
References = ToImmutableReferences(document.References),
ReconciledFrom = ToImmutableStringArray(document.ReconciledFrom),
Notes = ToImmutableDictionary(document.Notes)
};
}
}

1
src/Concelier/__Libraries/StellaOps.Concelier.Storage.Mongo/ServiceCollectionExtensions.cs
View File

@@ -109,6 +109,7 @@ public static class ServiceCollectionExtensions
services.AddSingleton<IMongoMigration, EnsureAdvisoryRawIdempotencyIndexMigration>();
services.AddSingleton<IMongoMigration, EnsureAdvisorySupersedesBackfillMigration>();
services.AddSingleton<IMongoMigration, EnsureAdvisoryRawValidatorMigration>();
services.AddSingleton<IMongoMigration, EnsureAdvisoryObservationsRawLinksetMigration>();
services.AddSingleton<IMongoMigration, EnsureAdvisoryEventCollectionsMigration>();
services.AddSingleton<IMongoMigration, SemVerStyleBackfillMigration>();

1
src/Concelier/__Libraries/StellaOps.Concelier.Storage.Mongo/TASKS.md
View File

@@ -10,6 +10,7 @@
> Docs alignment (2025-10-26): Rollback guidance added to `docs/deploy/containers.md` §6.
> 2025-10-28: Documented duplicate audit + migration workflow in `docs/deploy/containers.md`, Offline Kit guide, and `MIGRATIONS.md`; published `ops/devops/scripts/check-advisory-raw-duplicates.js` for staging/offline clusters.
> Docs alignment (2025-10-26): Offline kit requirements documented in `docs/deploy/containers.md` §5.
| CONCELIER-STORE-AOC-19-005 `Raw linkset backfill` | TODO (2025-11-04) | Concelier Storage Guild, DevOps Guild | CONCELIER-CORE-AOC-19-004 | Plan and execute advisory_observations `rawLinkset` backfill (online + Offline Kit bundles), supply migration scripts + rehearse rollback. Follow the coordination plan in `docs/dev/raw-linkset-backfill-plan.md`. |
## Policy Engine v2

50
src/Concelier/__Tests/StellaOps.Concelier.Core.Tests/Linksets/AdvisoryObservationFactoryTests.cs
View File

@@ -34,12 +34,34 @@ public sealed class AdvisoryObservationFactoryTests
Assert.Equal(SampleTimestamp, observation.CreatedAt);
Assert.Equal(new[] { "cve-2025-0001", "ghsa-xxxx-yyyy" }, observation.Linkset.Aliases);
Assert.Equal(new[] { "pkg:npm/left-pad@1.0.0" }, observation.Linkset.Purls);
Assert.Equal(new[] { "cpe:2.3:a:example:product:1.0:*:*:*:*:*:*:*" }, observation.Linkset.Cpes);
var reference = Assert.Single(observation.Linkset.References);
Assert.Equal("advisory", reference.Type);
Assert.Equal("https://example.test/advisory", reference.Url);
}
Assert.Equal(new[] { "pkg:npm/left-pad@1.0.0" }, observation.Linkset.Purls);
Assert.Equal(new[] { "cpe:2.3:a:example:product:1.0:*:*:*:*:*:*:*" }, observation.Linkset.Cpes);
var reference = Assert.Single(observation.Linkset.References);
Assert.Equal("advisory", reference.Type);
Assert.Equal("https://example.test/advisory", reference.Url);
Assert.Equal(
new[] { "GHSA-XXXX-YYYY", " CVE-2025-0001 ", "ghsa-XXXX-YYYY", " CVE-2025-0001 " },
observation.RawLinkset.Aliases);
Assert.Equal(
new[] { "pkg:NPM/left-pad@1.0.0", "pkg:npm/left-pad@1.0.0?foo=bar" },
observation.RawLinkset.PackageUrls);
Assert.Equal(
new[] { "cpe:/a:Example:Product:1.0", "cpe:/a:example:product:1.0" },
observation.RawLinkset.Cpes);
Assert.Collection(
observation.RawLinkset.References,
first =>
{
Assert.Equal("Advisory", first.Type);
Assert.Equal(" https://example.test/advisory ", first.Url);
},
second =>
{
Assert.Equal("ADVISORY", second.Type);
Assert.Equal("https://example.test/advisory", second.Url);
});
}
[Fact]
public void Create_SetsSourceAndUpstreamFields()
@@ -96,13 +118,15 @@ public sealed class AdvisoryObservationFactoryTests
},
supersedes: "tenant-a:vendor-x:previous:sha256:123");
var observation = factory.Create(rawDocument);
Assert.Equal("1.0.0", observation.Attributes["linkset.note.range-introduced"]);
Assert.Equal("1.0.5", observation.Attributes["linkset.note.range-fixed"]);
Assert.Equal("tenant-a:vendor-x:previous:sha256:123", observation.Attributes["supersedes"]);
Assert.Equal("connector-a;connector-b", observation.Attributes["linkset.reconciled_from"]);
}
var observation = factory.Create(rawDocument);
Assert.Equal("1.0.0", observation.Attributes["linkset.note.range-introduced"]);
Assert.Equal("1.0.5", observation.Attributes["linkset.note.range-fixed"]);
Assert.Equal("tenant-a:vendor-x:previous:sha256:123", observation.Attributes["supersedes"]);
Assert.Equal("connector-a;connector-b", observation.Attributes["linkset.reconciled_from"]);
Assert.Equal(notes, observation.RawLinkset.Notes);
Assert.Equal(new[] { "connector-a", "connector-b" }, observation.RawLinkset.ReconciledFrom);
}
private static AdvisoryRawDocument BuildRawDocument(
RawSourceMetadata? source = null,

42
src/Concelier/__Tests/StellaOps.Concelier.Core.Tests/Observations/AdvisoryObservationQueryServiceTests.cs
View File

@@ -1,7 +1,9 @@
using System.Collections.Immutable;
using System.Collections.Immutable;
using System.Linq;
using System.Text.Json.Nodes;
using StellaOps.Concelier.Core.Observations;
using StellaOps.Concelier.Models.Observations;
using StellaOps.Concelier.Core.Observations;
using StellaOps.Concelier.Models.Observations;
using StellaOps.Concelier.RawModels;
using Xunit;
namespace StellaOps.Concelier.Core.Tests.Observations;
@@ -224,18 +226,28 @@ public sealed class AdvisoryObservationQueryServiceTests
contentHash: $"sha256:{observationId}",
signature: DefaultSignature);
var content = new AdvisoryObservationContent("CSAF", "2.0", raw);
var linkset = new AdvisoryObservationLinkset(aliases, purls, cpes, references);
return new AdvisoryObservation(
observationId,
tenant,
DefaultSource,
upstream,
content,
linkset,
createdAt);
}
var content = new AdvisoryObservationContent("CSAF", "2.0", raw);
var linkset = new AdvisoryObservationLinkset(aliases, purls, cpes, references);
var rawLinkset = new RawLinkset
{
Aliases = aliases.ToImmutableArray(),
PackageUrls = purls.ToImmutableArray(),
Cpes = cpes.ToImmutableArray(),
References = references
.Select(static reference => new RawReference(reference.Type, reference.Url))
.ToImmutableArray()
};
return new AdvisoryObservation(
observationId,
tenant,
DefaultSource,
upstream,
content,
linkset,
rawLinkset,
createdAt);
}
private sealed class InMemoryLookup : IAdvisoryObservationLookup
{

5
src/Concelier/__Tests/StellaOps.Concelier.Core.Tests/Raw/AdvisoryRawServiceTests.cs
View File

@@ -1,5 +1,6 @@
using System;
using System.Collections.Immutable;
using System.Collections.Immutable;
using System.Linq;
using System.Text.Json;
using System.Threading;
using System.Threading.Tasks;
@@ -66,7 +67,7 @@ public sealed class AdvisoryRawServiceTests
await service.IngestAsync(document, CancellationToken.None);
Assert.NotNull(repository.CapturedDocument);
Assert.Equal(aliasSeries, repository.CapturedDocument!.Identifiers.Aliases);
Assert.True(aliasSeries.SequenceEqual(repository.CapturedDocument!.Identifiers.Aliases));
}
private static AdvisoryRawService CreateService(RecordingRepository repository)

127
src/Concelier/__Tests/StellaOps.Concelier.Models.Tests/Fixtures/ghsa-semver.actual.json
View File

@@ -1,127 +0,0 @@
{
"advisoryKey": "GHSA-aaaa-bbbb-cccc",
"affectedPackages": [
{
"type": "semver",
"identifier": "pkg:npm/example-widget",
"platform": null,
"versionRanges": [
{
"fixedVersion": "2.5.1",
"introducedVersion": null,
"lastAffectedVersion": null,
"primitives": null,
"provenance": {
"source": "ghsa",
"kind": "map",
"value": "ghsa-aaaa-bbbb-cccc",
"decisionReason": null,
"recordedAt": "2024-03-05T10:00:00+00:00",
"fieldMask": []
},
"rangeExpression": ">=0.0.0 <2.5.1",
"rangeKind": "semver"
},
{
"fixedVersion": "3.2.4",
"introducedVersion": "3.0.0",
"lastAffectedVersion": null,
"primitives": null,
"provenance": {
"source": "ghsa",
"kind": "map",
"value": "ghsa-aaaa-bbbb-cccc",
"decisionReason": null,
"recordedAt": "2024-03-05T10:00:00+00:00",
"fieldMask": []
},
"rangeExpression": null,
"rangeKind": "semver"
}
],
"normalizedVersions": [],
"statuses": [],
"provenance": [
{
"source": "ghsa",
"kind": "map",
"value": "ghsa-aaaa-bbbb-cccc",
"decisionReason": null,
"recordedAt": "2024-03-05T10:00:00+00:00",
"fieldMask": []
}
]
}
],
"aliases": [
"CVE-2024-2222",
"GHSA-aaaa-bbbb-cccc"
],
"canonicalMetricId": null,
"credits": [],
"cvssMetrics": [
{
"baseScore": 8.8,
"baseSeverity": "high",
"provenance": {
"source": "ghsa",
"kind": "map",
"value": "ghsa-aaaa-bbbb-cccc",
"decisionReason": null,
"recordedAt": "2024-03-05T10:00:00+00:00",
"fieldMask": []
},
"vector": "CVSS:3.1/AV:N/AC:L/PR:N/UI:R/S:U/C:H/I:H/A:H",
"version": "3.1"
}
],
"cwes": [],
"description": null,
"exploitKnown": false,
"language": "en",
"modified": "2024-03-04T12:00:00+00:00",
"provenance": [
{
"source": "ghsa",
"kind": "map",
"value": "ghsa-aaaa-bbbb-cccc",
"decisionReason": null,
"recordedAt": "2024-03-05T10:00:00+00:00",
"fieldMask": []
}
],
"published": "2024-03-04T00:00:00+00:00",
"references": [
{
"kind": "patch",
"provenance": {
"source": "ghsa",
"kind": "map",
"value": "ghsa-aaaa-bbbb-cccc",
"decisionReason": null,
"recordedAt": "2024-03-05T10:00:00+00:00",
"fieldMask": []
},
"sourceTag": "ghsa",
"summary": "Patch commit",
"url": "https://github.com/example/widget/commit/abcd1234"
},
{
"kind": "advisory",
"provenance": {
"source": "ghsa",
"kind": "map",
"value": "ghsa-aaaa-bbbb-cccc",
"decisionReason": null,
"recordedAt": "2024-03-05T10:00:00+00:00",
"fieldMask": []
},
"sourceTag": "ghsa",
"summary": "GitHub Security Advisory",
"url": "https://github.com/example/widget/security/advisories/GHSA-aaaa-bbbb-cccc"
}
],
"severity": "high",
"summary": "A crafted payload can pollute Object.prototype leading to RCE.",
"title": "Prototype pollution in widget.js"
}

249
src/Concelier/__Tests/StellaOps.Concelier.Models.Tests/Fixtures/ghsa-semver.json
View File

@@ -1,124 +1,127 @@
{
"advisoryKey": "GHSA-aaaa-bbbb-cccc",
"affectedPackages": [
{
"type": "semver",
"identifier": "pkg:npm/example-widget",
"platform": null,
"versionRanges": [
{
"fixedVersion": "2.5.1",
"introducedVersion": null,
"lastAffectedVersion": null,
"primitives": null,
"provenance": {
"source": "ghsa",
"kind": "map",
"value": "ghsa-aaaa-bbbb-cccc",
"decisionReason": null,
"recordedAt": "2024-03-05T10:00:00+00:00",
"fieldMask": []
},
"rangeExpression": ">=0.0.0 <2.5.1",
"rangeKind": "semver"
},
{
"fixedVersion": "3.2.4",
"introducedVersion": "3.0.0",
"lastAffectedVersion": null,
"primitives": null,
"provenance": {
"source": "ghsa",
"kind": "map",
"value": "ghsa-aaaa-bbbb-cccc",
"decisionReason": null,
"recordedAt": "2024-03-05T10:00:00+00:00",
"fieldMask": []
},
"rangeExpression": null,
"rangeKind": "semver"
}
],
"normalizedVersions": [],
"statuses": [],
"provenance": [
{
"source": "ghsa",
"kind": "map",
"value": "ghsa-aaaa-bbbb-cccc",
"decisionReason": null,
"recordedAt": "2024-03-05T10:00:00+00:00",
"fieldMask": []
}
]
}
],
"aliases": [
"CVE-2024-2222",
"GHSA-aaaa-bbbb-cccc"
],
"credits": [],
"cvssMetrics": [
{
"baseScore": 8.8,
"baseSeverity": "high",
"provenance": {
"source": "ghsa",
"kind": "map",
"value": "ghsa-aaaa-bbbb-cccc",
"decisionReason": null,
"recordedAt": "2024-03-05T10:00:00+00:00",
"fieldMask": []
},
"vector": "CVSS:3.1/AV:N/AC:L/PR:N/UI:R/S:U/C:H/I:H/A:H",
"version": "3.1"
}
],
"exploitKnown": false,
"language": "en",
"modified": "2024-03-04T12:00:00+00:00",
"provenance": [
{
"source": "ghsa",
"kind": "map",
"value": "ghsa-aaaa-bbbb-cccc",
"decisionReason": null,
"recordedAt": "2024-03-05T10:00:00+00:00",
"fieldMask": []
}
],
"published": "2024-03-04T00:00:00+00:00",
"references": [
{
"kind": "patch",
"provenance": {
"source": "ghsa",
"kind": "map",
"value": "ghsa-aaaa-bbbb-cccc",
"decisionReason": null,
"recordedAt": "2024-03-05T10:00:00+00:00",
"fieldMask": []
},
"sourceTag": "ghsa",
"summary": "Patch commit",
"url": "https://github.com/example/widget/commit/abcd1234"
},
{
"kind": "advisory",
"provenance": {
"source": "ghsa",
"kind": "map",
"value": "ghsa-aaaa-bbbb-cccc",
"decisionReason": null,
"recordedAt": "2024-03-05T10:00:00+00:00",
"fieldMask": []
},
"sourceTag": "ghsa",
"summary": "GitHub Security Advisory",
"url": "https://github.com/example/widget/security/advisories/GHSA-aaaa-bbbb-cccc"
}
],
"severity": "high",
"summary": "A crafted payload can pollute Object.prototype leading to RCE.",
"title": "Prototype pollution in widget.js"
{
"advisoryKey": "GHSA-aaaa-bbbb-cccc",
"affectedPackages": [
{
"type": "semver",
"identifier": "pkg:npm/example-widget",
"platform": null,
"versionRanges": [
{
"fixedVersion": "2.5.1",
"introducedVersion": null,
"lastAffectedVersion": null,
"primitives": null,
"provenance": {
"source": "ghsa",
"kind": "map",
"value": "ghsa-aaaa-bbbb-cccc",
"decisionReason": null,
"recordedAt": "2024-03-05T10:00:00+00:00",
"fieldMask": []
},
"rangeExpression": ">=0.0.0 <2.5.1",
"rangeKind": "semver"
},
{
"fixedVersion": "3.2.4",
"introducedVersion": "3.0.0",
"lastAffectedVersion": null,
"primitives": null,
"provenance": {
"source": "ghsa",
"kind": "map",
"value": "ghsa-aaaa-bbbb-cccc",
"decisionReason": null,
"recordedAt": "2024-03-05T10:00:00+00:00",
"fieldMask": []
},
"rangeExpression": null,
"rangeKind": "semver"
}
],
"normalizedVersions": [],
"statuses": [],
"provenance": [
{
"source": "ghsa",
"kind": "map",
"value": "ghsa-aaaa-bbbb-cccc",
"decisionReason": null,
"recordedAt": "2024-03-05T10:00:00+00:00",
"fieldMask": []
}
]
}
],
"aliases": [
"CVE-2024-2222",
"GHSA-aaaa-bbbb-cccc"
],
"canonicalMetricId": null,
"credits": [],
"cvssMetrics": [
{
"baseScore": 8.8,
"baseSeverity": "high",
"provenance": {
"source": "ghsa",
"kind": "map",
"value": "ghsa-aaaa-bbbb-cccc",
"decisionReason": null,
"recordedAt": "2024-03-05T10:00:00+00:00",
"fieldMask": []
},
"vector": "CVSS:3.1/AV:N/AC:L/PR:N/UI:R/S:U/C:H/I:H/A:H",
"version": "3.1"
}
],
"cwes": [],
"description": null,
"exploitKnown": false,
"language": "en",
"modified": "2024-03-04T12:00:00+00:00",
"provenance": [
{
"source": "ghsa",
"kind": "map",
"value": "ghsa-aaaa-bbbb-cccc",
"decisionReason": null,
"recordedAt": "2024-03-05T10:00:00+00:00",
"fieldMask": []
}
],
"published": "2024-03-04T00:00:00+00:00",
"references": [
{
"kind": "patch",
"provenance": {
"source": "ghsa",
"kind": "map",
"value": "ghsa-aaaa-bbbb-cccc",
"decisionReason": null,
"recordedAt": "2024-03-05T10:00:00+00:00",
"fieldMask": []
},
"sourceTag": "ghsa",
"summary": "Patch commit",
"url": "https://github.com/example/widget/commit/abcd1234"
},
{
"kind": "advisory",
"provenance": {
"source": "ghsa",
"kind": "map",
"value": "ghsa-aaaa-bbbb-cccc",
"decisionReason": null,
"recordedAt": "2024-03-05T10:00:00+00:00",
"fieldMask": []
},
"sourceTag": "ghsa",
"summary": "GitHub Security Advisory",
"url": "https://github.com/example/widget/security/advisories/GHSA-aaaa-bbbb-cccc"
}
],
"severity": "high",
"summary": "A crafted payload can pollute Object.prototype leading to RCE.",
"title": "Prototype pollution in widget.js"
}

45
src/Concelier/__Tests/StellaOps.Concelier.Models.Tests/Fixtures/kev-flag.actual.json
View File

@@ -1,45 +0,0 @@
{
"advisoryKey": "CVE-2023-9999",
"affectedPackages": [],
"aliases": [
"CVE-2023-9999"
],
"canonicalMetricId": null,
"credits": [],
"cvssMetrics": [],
"cwes": [],
"description": null,
"exploitKnown": true,
"language": "en",
"modified": "2024-02-09T16:22:00+00:00",
"provenance": [
{
"source": "cisa-kev",
"kind": "annotate",
"value": "kev",
"decisionReason": null,
"recordedAt": "2024-02-10T09:30:00+00:00",
"fieldMask": []
}
],
"published": "2023-11-20T00:00:00+00:00",
"references": [
{
"kind": "kev",
"provenance": {
"source": "cisa-kev",
"kind": "annotate",
"value": "kev",
"decisionReason": null,
"recordedAt": "2024-02-10T09:30:00+00:00",
"fieldMask": []
},
"sourceTag": "cisa",
"summary": "CISA KEV entry",
"url": "https://www.cisa.gov/known-exploited-vulnerabilities-catalog"
}
],
"severity": "critical",
"summary": "Unauthenticated RCE due to unsafe deserialization.",
"title": "Remote code execution in LegacyServer"
}

85
src/Concelier/__Tests/StellaOps.Concelier.Models.Tests/Fixtures/kev-flag.json
View File

@@ -1,42 +1,45 @@
{
"advisoryKey": "CVE-2023-9999",
"affectedPackages": [],
"aliases": [
"CVE-2023-9999"
],
"credits": [],
"cvssMetrics": [],
"exploitKnown": true,
"language": "en",
"modified": "2024-02-09T16:22:00+00:00",
"provenance": [
{
"source": "cisa-kev",
"kind": "annotate",
"value": "kev",
"decisionReason": null,
"recordedAt": "2024-02-10T09:30:00+00:00",
"fieldMask": []
}
],
"published": "2023-11-20T00:00:00+00:00",
"references": [
{
"kind": "kev",
"provenance": {
"source": "cisa-kev",
"kind": "annotate",
"value": "kev",
"decisionReason": null,
"recordedAt": "2024-02-10T09:30:00+00:00",
"fieldMask": []
},
"sourceTag": "cisa",
"summary": "CISA KEV entry",
"url": "https://www.cisa.gov/known-exploited-vulnerabilities-catalog"
}
],
"severity": "critical",
"summary": "Unauthenticated RCE due to unsafe deserialization.",
"title": "Remote code execution in LegacyServer"
{
"advisoryKey": "CVE-2023-9999",
"affectedPackages": [],
"aliases": [
"CVE-2023-9999"
],
"canonicalMetricId": null,
"credits": [],
"cvssMetrics": [],
"cwes": [],
"description": null,
"exploitKnown": true,
"language": "en",
"modified": "2024-02-09T16:22:00+00:00",
"provenance": [
{
"source": "cisa-kev",
"kind": "annotate",
"value": "kev",
"decisionReason": null,
"recordedAt": "2024-02-10T09:30:00+00:00",
"fieldMask": []
}
],
"published": "2023-11-20T00:00:00+00:00",
"references": [
{
"kind": "kev",
"provenance": {
"source": "cisa-kev",
"kind": "annotate",
"value": "kev",
"decisionReason": null,
"recordedAt": "2024-02-10T09:30:00+00:00",
"fieldMask": []
},
"sourceTag": "cisa",
"summary": "CISA KEV entry",
"url": "https://www.cisa.gov/known-exploited-vulnerabilities-catalog"
}
],
"severity": "critical",
"summary": "Unauthenticated RCE due to unsafe deserialization.",
"title": "Remote code execution in LegacyServer"
}

122
src/Concelier/__Tests/StellaOps.Concelier.Models.Tests/Fixtures/nvd-basic.actual.json
View File

@@ -1,122 +0,0 @@
{
"advisoryKey": "CVE-2024-1234",
"affectedPackages": [
{
"type": "cpe",
"identifier": "cpe:/a:examplecms:examplecms:1.0",
"platform": null,
"versionRanges": [
{
"fixedVersion": "1.0.5",
"introducedVersion": "1.0",
"lastAffectedVersion": null,
"primitives": null,
"provenance": {
"source": "nvd",
"kind": "map",
"value": "cve-2024-1234",
"decisionReason": null,
"recordedAt": "2024-08-01T12:00:00+00:00",
"fieldMask": []
},
"rangeExpression": null,
"rangeKind": "version"
}
],
"normalizedVersions": [],
"statuses": [
{
"provenance": {
"source": "nvd",
"kind": "map",
"value": "cve-2024-1234",
"decisionReason": null,
"recordedAt": "2024-08-01T12:00:00+00:00",
"fieldMask": []
},
"status": "affected"
}
],
"provenance": [
{
"source": "nvd",
"kind": "map",
"value": "cve-2024-1234",
"decisionReason": null,
"recordedAt": "2024-08-01T12:00:00+00:00",
"fieldMask": []
}
]
}
],
"aliases": [
"CVE-2024-1234"
],
"canonicalMetricId": null,
"credits": [],
"cvssMetrics": [
{
"baseScore": 9.8,
"baseSeverity": "critical",
"provenance": {
"source": "nvd",
"kind": "map",
"value": "cve-2024-1234",
"decisionReason": null,
"recordedAt": "2024-08-01T12:00:00+00:00",
"fieldMask": []
},
"vector": "CVSS:3.1/AV:N/AC:L/PR:N/UI:N/S:U/C:H/I:H/A:H",
"version": "3.1"
}
],
"cwes": [],
"description": null,
"exploitKnown": false,
"language": "en",
"modified": "2024-07-16T10:35:00+00:00",
"provenance": [
{
"source": "nvd",
"kind": "map",
"value": "cve-2024-1234",
"decisionReason": null,
"recordedAt": "2024-08-01T12:00:00+00:00",
"fieldMask": []
}
],
"published": "2024-07-15T00:00:00+00:00",
"references": [
{
"kind": "advisory",
"provenance": {
"source": "example",
"kind": "fetch",
"value": "bulletin",
"decisionReason": null,
"recordedAt": "2024-07-14T15:00:00+00:00",
"fieldMask": []
},
"sourceTag": "vendor",
"summary": "Vendor bulletin",
"url": "https://example.org/security/CVE-2024-1234"
},
{
"kind": "advisory",
"provenance": {
"source": "nvd",
"kind": "map",
"value": "cve-2024-1234",
"decisionReason": null,
"recordedAt": "2024-08-01T12:00:00+00:00",
"fieldMask": []
},
"sourceTag": "nvd",
"summary": "NVD entry",
"url": "https://nvd.nist.gov/vuln/detail/CVE-2024-1234"
}
],
"severity": "high",
"summary": "An integer overflow in ExampleCMS allows remote attackers to escalate privileges.",
"title": "Integer overflow in ExampleCMS"
}

239
src/Concelier/__Tests/StellaOps.Concelier.Models.Tests/Fixtures/nvd-basic.json
View File

@@ -1,119 +1,122 @@
{
"advisoryKey": "CVE-2024-1234",
"affectedPackages": [
{
"type": "cpe",
"identifier": "cpe:/a:examplecms:examplecms:1.0",
"platform": null,
"versionRanges": [
{
"fixedVersion": "1.0.5",
"introducedVersion": "1.0",
"lastAffectedVersion": null,
"primitives": null,
"provenance": {
"source": "nvd",
"kind": "map",
"value": "cve-2024-1234",
"decisionReason": null,
"recordedAt": "2024-08-01T12:00:00+00:00",
"fieldMask": []
},
"rangeExpression": null,
"rangeKind": "version"
}
],
"normalizedVersions": [],
"statuses": [
{
"provenance": {
"source": "nvd",
"kind": "map",
"value": "cve-2024-1234",
"decisionReason": null,
"recordedAt": "2024-08-01T12:00:00+00:00",
"fieldMask": []
},
"status": "affected"
}
],
"provenance": [
{
"source": "nvd",
"kind": "map",
"value": "cve-2024-1234",
"decisionReason": null,
"recordedAt": "2024-08-01T12:00:00+00:00",
"fieldMask": []
}
]
}
],
"aliases": [
"CVE-2024-1234"
],
"credits": [],
"cvssMetrics": [
{
"baseScore": 9.8,
"baseSeverity": "critical",
"provenance": {
"source": "nvd",
"kind": "map",
"value": "cve-2024-1234",
"decisionReason": null,
"recordedAt": "2024-08-01T12:00:00+00:00",
"fieldMask": []
},
"vector": "CVSS:3.1/AV:N/AC:L/PR:N/UI:N/S:U/C:H/I:H/A:H",
"version": "3.1"
}
],
"exploitKnown": false,
"language": "en",
"modified": "2024-07-16T10:35:00+00:00",
"provenance": [
{
"source": "nvd",
"kind": "map",
"value": "cve-2024-1234",
"decisionReason": null,
"recordedAt": "2024-08-01T12:00:00+00:00",
"fieldMask": []
}
],
"published": "2024-07-15T00:00:00+00:00",
"references": [
{
"kind": "advisory",
"provenance": {
"source": "example",
"kind": "fetch",
"value": "bulletin",
"decisionReason": null,
"recordedAt": "2024-07-14T15:00:00+00:00",
"fieldMask": []
},
"sourceTag": "vendor",
"summary": "Vendor bulletin",
"url": "https://example.org/security/CVE-2024-1234"
},
{
"kind": "advisory",
"provenance": {
"source": "nvd",
"kind": "map",
"value": "cve-2024-1234",
"decisionReason": null,
"recordedAt": "2024-08-01T12:00:00+00:00",
"fieldMask": []
},
"sourceTag": "nvd",
"summary": "NVD entry",
"url": "https://nvd.nist.gov/vuln/detail/CVE-2024-1234"
}
],
"severity": "high",
"summary": "An integer overflow in ExampleCMS allows remote attackers to escalate privileges.",
"title": "Integer overflow in ExampleCMS"
{
"advisoryKey": "CVE-2024-1234",
"affectedPackages": [
{
"type": "cpe",
"identifier": "cpe:/a:examplecms:examplecms:1.0",
"platform": null,
"versionRanges": [
{
"fixedVersion": "1.0.5",
"introducedVersion": "1.0",
"lastAffectedVersion": null,
"primitives": null,
"provenance": {
"source": "nvd",
"kind": "map",
"value": "cve-2024-1234",
"decisionReason": null,
"recordedAt": "2024-08-01T12:00:00+00:00",
"fieldMask": []
},
"rangeExpression": null,
"rangeKind": "version"
}
],
"normalizedVersions": [],
"statuses": [
{
"provenance": {
"source": "nvd",
"kind": "map",
"value": "cve-2024-1234",
"decisionReason": null,
"recordedAt": "2024-08-01T12:00:00+00:00",
"fieldMask": []
},
"status": "affected"
}
],
"provenance": [
{
"source": "nvd",
"kind": "map",
"value": "cve-2024-1234",
"decisionReason": null,
"recordedAt": "2024-08-01T12:00:00+00:00",
"fieldMask": []
}
]
}
],
"aliases": [
"CVE-2024-1234"
],
"canonicalMetricId": null,
"credits": [],
"cvssMetrics": [
{
"baseScore": 9.8,
"baseSeverity": "critical",
"provenance": {
"source": "nvd",
"kind": "map",
"value": "cve-2024-1234",
"decisionReason": null,
"recordedAt": "2024-08-01T12:00:00+00:00",
"fieldMask": []
},
"vector": "CVSS:3.1/AV:N/AC:L/PR:N/UI:N/S:U/C:H/I:H/A:H",
"version": "3.1"
}
],
"cwes": [],
"description": null,
"exploitKnown": false,
"language": "en",
"modified": "2024-07-16T10:35:00+00:00",
"provenance": [
{
"source": "nvd",
"kind": "map",
"value": "cve-2024-1234",
"decisionReason": null,
"recordedAt": "2024-08-01T12:00:00+00:00",
"fieldMask": []
}
],
"published": "2024-07-15T00:00:00+00:00",
"references": [
{
"kind": "advisory",
"provenance": {
"source": "example",
"kind": "fetch",
"value": "bulletin",
"decisionReason": null,
"recordedAt": "2024-07-14T15:00:00+00:00",
"fieldMask": []
},
"sourceTag": "vendor",
"summary": "Vendor bulletin",
"url": "https://example.org/security/CVE-2024-1234"
},
{
"kind": "advisory",
"provenance": {
"source": "nvd",
"kind": "map",
"value": "cve-2024-1234",
"decisionReason": null,
"recordedAt": "2024-08-01T12:00:00+00:00",
"fieldMask": []
},
"sourceTag": "nvd",
"summary": "NVD entry",
"url": "https://nvd.nist.gov/vuln/detail/CVE-2024-1234"
}
],
"severity": "high",
"summary": "An integer overflow in ExampleCMS allows remote attackers to escalate privileges.",
"title": "Integer overflow in ExampleCMS"
}

125
src/Concelier/__Tests/StellaOps.Concelier.Models.Tests/Fixtures/psirt-overlay.actual.json
View File

@@ -1,125 +0,0 @@
{
"advisoryKey": "RHSA-2024:0252",
"affectedPackages": [
{
"type": "rpm",
"identifier": "kernel-0:4.18.0-553.el8.x86_64",
"platform": "rhel-8",
"versionRanges": [
{
"fixedVersion": null,
"introducedVersion": "0:4.18.0-553.el8",
"lastAffectedVersion": null,
"primitives": null,
"provenance": {
"source": "redhat",
"kind": "map",
"value": "rhsa-2024:0252",
"decisionReason": null,
"recordedAt": "2024-05-11T09:00:00+00:00",
"fieldMask": []
},
"rangeExpression": null,
"rangeKind": "nevra"
}
],
"normalizedVersions": [],
"statuses": [
{
"provenance": {
"source": "redhat",
"kind": "map",
"value": "rhsa-2024:0252",
"decisionReason": null,
"recordedAt": "2024-05-11T09:00:00+00:00",
"fieldMask": []
},
"status": "fixed"
}
],
"provenance": [
{
"source": "redhat",
"kind": "enrich",
"value": "cve-2024-5678",
"decisionReason": null,
"recordedAt": "2024-05-11T09:05:00+00:00",
"fieldMask": []
},
{
"source": "redhat",
"kind": "map",
"value": "rhsa-2024:0252",
"decisionReason": null,
"recordedAt": "2024-05-11T09:00:00+00:00",
"fieldMask": []
}
]
}
],
"aliases": [
"CVE-2024-5678",
"RHSA-2024:0252"
],
"canonicalMetricId": null,
"credits": [],
"cvssMetrics": [
{
"baseScore": 6.7,
"baseSeverity": "medium",
"provenance": {
"source": "redhat",
"kind": "map",
"value": "rhsa-2024:0252",
"decisionReason": null,
"recordedAt": "2024-05-11T09:00:00+00:00",
"fieldMask": []
},
"vector": "CVSS:3.1/AV:L/AC:L/PR:H/UI:N/S:U/C:H/I:H/A:H",
"version": "3.1"
}
],
"cwes": [],
"description": null,
"exploitKnown": false,
"language": "en",
"modified": "2024-05-11T08:15:00+00:00",
"provenance": [
{
"source": "redhat",
"kind": "enrich",
"value": "cve-2024-5678",
"decisionReason": null,
"recordedAt": "2024-05-11T09:05:00+00:00",
"fieldMask": []
},
{
"source": "redhat",
"kind": "map",
"value": "rhsa-2024:0252",
"decisionReason": null,
"recordedAt": "2024-05-11T09:00:00+00:00",
"fieldMask": []
}
],
"published": "2024-05-10T19:28:00+00:00",
"references": [
{
"kind": "advisory",
"provenance": {
"source": "redhat",
"kind": "map",
"value": "rhsa-2024:0252",
"decisionReason": null,
"recordedAt": "2024-05-11T09:00:00+00:00",
"fieldMask": []
},
"sourceTag": "redhat",
"summary": "Red Hat security advisory",
"url": "https://access.redhat.com/errata/RHSA-2024:0252"
}
],
"severity": "critical",
"summary": "Updates the Red Hat Enterprise Linux kernel to address CVE-2024-5678.",
"title": "Important: kernel security update"
}

245
src/Concelier/__Tests/StellaOps.Concelier.Models.Tests/Fixtures/psirt-overlay.json
View File

@@ -1,122 +1,125 @@
{
"advisoryKey": "RHSA-2024:0252",
"affectedPackages": [
{
"type": "rpm",
"identifier": "kernel-0:4.18.0-553.el8.x86_64",
"platform": "rhel-8",
"versionRanges": [
{
"fixedVersion": null,
"introducedVersion": "0:4.18.0-553.el8",
"lastAffectedVersion": null,
"primitives": null,
"provenance": {
"source": "redhat",
"kind": "map",
"value": "rhsa-2024:0252",
"decisionReason": null,
"recordedAt": "2024-05-11T09:00:00+00:00",
"fieldMask": []
},
"rangeExpression": null,
"rangeKind": "nevra"
}
],
"normalizedVersions": [],
"statuses": [
{
"provenance": {
"source": "redhat",
"kind": "map",
"value": "rhsa-2024:0252",
"decisionReason": null,
"recordedAt": "2024-05-11T09:00:00+00:00",
"fieldMask": []
},
"status": "fixed"
}
],
"provenance": [
{
"source": "redhat",
"kind": "enrich",
"value": "cve-2024-5678",
"decisionReason": null,
"recordedAt": "2024-05-11T09:05:00+00:00",
"fieldMask": []
},
{
"source": "redhat",
"kind": "map",
"value": "rhsa-2024:0252",
"decisionReason": null,
"recordedAt": "2024-05-11T09:00:00+00:00",
"fieldMask": []
}
]
}
],
"aliases": [
"CVE-2024-5678",
"RHSA-2024:0252"
],
"credits": [],
"cvssMetrics": [
{
"baseScore": 6.7,
"baseSeverity": "medium",
"provenance": {
"source": "redhat",
"kind": "map",
"value": "rhsa-2024:0252",
"decisionReason": null,
"recordedAt": "2024-05-11T09:00:00+00:00",
"fieldMask": []
},
"vector": "CVSS:3.1/AV:L/AC:L/PR:H/UI:N/S:U/C:H/I:H/A:H",
"version": "3.1"
}
],
"exploitKnown": false,
"language": "en",
"modified": "2024-05-11T08:15:00+00:00",
"provenance": [
{
"source": "redhat",
"kind": "enrich",
"value": "cve-2024-5678",
"decisionReason": null,
"recordedAt": "2024-05-11T09:05:00+00:00",
"fieldMask": []
},
{
"source": "redhat",
"kind": "map",
"value": "rhsa-2024:0252",
"decisionReason": null,
"recordedAt": "2024-05-11T09:00:00+00:00",
"fieldMask": []
}
],
"published": "2024-05-10T19:28:00+00:00",
"references": [
{
"kind": "advisory",
"provenance": {
"source": "redhat",
"kind": "map",
"value": "rhsa-2024:0252",
"decisionReason": null,
"recordedAt": "2024-05-11T09:00:00+00:00",
"fieldMask": []
},
"sourceTag": "redhat",
"summary": "Red Hat security advisory",
"url": "https://access.redhat.com/errata/RHSA-2024:0252"
}
],
"severity": "critical",
"summary": "Updates the Red Hat Enterprise Linux kernel to address CVE-2024-5678.",
"title": "Important: kernel security update"
{
"advisoryKey": "RHSA-2024:0252",
"affectedPackages": [
{
"type": "rpm",
"identifier": "kernel-0:4.18.0-553.el8.x86_64",
"platform": "rhel-8",
"versionRanges": [
{
"fixedVersion": null,
"introducedVersion": "0:4.18.0-553.el8",
"lastAffectedVersion": null,
"primitives": null,
"provenance": {
"source": "redhat",
"kind": "map",
"value": "rhsa-2024:0252",
"decisionReason": null,
"recordedAt": "2024-05-11T09:00:00+00:00",
"fieldMask": []
},
"rangeExpression": null,
"rangeKind": "nevra"
}
],
"normalizedVersions": [],
"statuses": [
{
"provenance": {
"source": "redhat",
"kind": "map",
"value": "rhsa-2024:0252",
"decisionReason": null,
"recordedAt": "2024-05-11T09:00:00+00:00",
"fieldMask": []
},
"status": "fixed"
}
],
"provenance": [
{
"source": "redhat",
"kind": "enrich",
"value": "cve-2024-5678",
"decisionReason": null,
"recordedAt": "2024-05-11T09:05:00+00:00",
"fieldMask": []
},
{
"source": "redhat",
"kind": "map",
"value": "rhsa-2024:0252",
"decisionReason": null,
"recordedAt": "2024-05-11T09:00:00+00:00",
"fieldMask": []
}
]
}
],
"aliases": [
"CVE-2024-5678",
"RHSA-2024:0252"
],
"canonicalMetricId": null,
"credits": [],
"cvssMetrics": [
{
"baseScore": 6.7,
"baseSeverity": "medium",
"provenance": {
"source": "redhat",
"kind": "map",
"value": "rhsa-2024:0252",
"decisionReason": null,
"recordedAt": "2024-05-11T09:00:00+00:00",
"fieldMask": []
},
"vector": "CVSS:3.1/AV:L/AC:L/PR:H/UI:N/S:U/C:H/I:H/A:H",
"version": "3.1"
}
],
"cwes": [],
"description": null,
"exploitKnown": false,
"language": "en",
"modified": "2024-05-11T08:15:00+00:00",
"provenance": [
{
"source": "redhat",
"kind": "enrich",
"value": "cve-2024-5678",
"decisionReason": null,
"recordedAt": "2024-05-11T09:05:00+00:00",
"fieldMask": []
},
{
"source": "redhat",
"kind": "map",
"value": "rhsa-2024:0252",
"decisionReason": null,
"recordedAt": "2024-05-11T09:00:00+00:00",
"fieldMask": []
}
],
"published": "2024-05-10T19:28:00+00:00",
"references": [
{
"kind": "advisory",
"provenance": {
"source": "redhat",
"kind": "map",
"value": "rhsa-2024:0252",
"decisionReason": null,
"recordedAt": "2024-05-11T09:00:00+00:00",
"fieldMask": []
},
"sourceTag": "redhat",
"summary": "Red Hat security advisory",
"url": "https://access.redhat.com/errata/RHSA-2024:0252"
}
],
"severity": "critical",
"summary": "Updates the Red Hat Enterprise Linux kernel to address CVE-2024-5678.",
"title": "Important: kernel security update"
}

44
src/Concelier/__Tests/StellaOps.Concelier.Models.Tests/Observations/AdvisoryObservationTests.cs
View File

@@ -1,7 +1,8 @@
using System.Collections.Generic;
using System.Collections.Immutable;
using System.Text.Json.Nodes;
using StellaOps.Concelier.Models.Observations;
using StellaOps.Concelier.Models.Observations;
using StellaOps.Concelier.RawModels;
using Xunit;
namespace StellaOps.Concelier.Models.Tests.Observations;
@@ -33,21 +34,32 @@ public sealed class AdvisoryObservationTests
new AdvisoryObservationReference("advisory", "https://example.com/advisory")
});
var attributes = ImmutableDictionary.CreateRange(new Dictionary<string, string>
{
[" region "] = "emea",
["pipeline"] = "daily"
});
var observation = new AdvisoryObservation(
observationId: " tenant-a:CVE-2025-1234:1 ",
tenant: " Tenant-A ",
source: source,
upstream: upstream,
content: content,
linkset: linkset,
createdAt: DateTimeOffset.Parse("2025-10-01T01:00:06Z"),
attributes: attributes);
var attributes = ImmutableDictionary.CreateRange(new Dictionary<string, string>
{
[" region "] = "emea",
["pipeline"] = "daily"
});
var rawLinkset = new RawLinkset
{
Aliases = ImmutableArray.Create(" Cve-2025-1234 ", "cve-2025-1234"),
PackageUrls = ImmutableArray.Create("pkg:generic/foo@1.0.0"),
Cpes = ImmutableArray.Create("cpe:/a:vendor:product:1"),
References = ImmutableArray.Create(new RawReference("ADVISORY", "https://example.com/advisory")),
ReconciledFrom = ImmutableArray.Create("pointer-1"),
Notes = ImmutableDictionary.CreateRange(new Dictionary<string, string> { ["note"] = "value" })
};
var observation = new AdvisoryObservation(
observationId: " tenant-a:CVE-2025-1234:1 ",
tenant: " Tenant-A ",
source: source,
upstream: upstream,
content: content,
linkset: linkset,
rawLinkset: rawLinkset,
createdAt: DateTimeOffset.Parse("2025-10-01T01:00:06Z"),
attributes: attributes);
Assert.Equal("tenant-a:CVE-2025-1234:1", observation.ObservationId);
Assert.Equal("tenant-a", observation.Tenant);

337
src/Concelier/__Tests/StellaOps.Concelier.Storage.Mongo.Tests/Migrations/EnsureAdvisoryObservationsRawLinksetMigrationTests.cs Normal file
View File

@@ -0,0 +1,337 @@
using System;
using System.Collections.Generic;
using System.Collections.Immutable;
using System.Linq;
using System.Text.Json;
using System.Threading;
using System.Threading.Tasks;
using Microsoft.Extensions.Logging.Abstractions;
using MongoDB.Bson;
using MongoDB.Bson.Serialization;
using MongoDB.Driver;
using StellaOps.Concelier.RawModels;
using StellaOps.Concelier.Storage.Mongo;
using StellaOps.Concelier.Storage.Mongo.Migrations;
using StellaOps.Concelier.Storage.Mongo.Observations;
using StellaOps.Concelier.Storage.Mongo.Raw;
using Xunit;
namespace StellaOps.Concelier.Storage.Mongo.Tests.Migrations;
[Collection("mongo-fixture")]
public sealed class EnsureAdvisoryObservationsRawLinksetMigrationTests
{
private readonly MongoIntegrationFixture _fixture;
public EnsureAdvisoryObservationsRawLinksetMigrationTests(MongoIntegrationFixture fixture)
{
_fixture = fixture;
}
[Fact]
public async Task ApplyAsync_BackfillsRawLinksetFromRawDocument()
{
var databaseName = $"concelier-rawlinkset-{Guid.NewGuid():N}";
var database = _fixture.Client.GetDatabase(databaseName);
await database.CreateCollectionAsync(MongoStorageDefaults.Collections.Migrations);
await database.CreateCollectionAsync(MongoStorageDefaults.Collections.AdvisoryObservations);
try
{
var rawRepository = new MongoAdvisoryRawRepository(
database,
TimeProvider.System,
NullLogger<MongoAdvisoryRawRepository>.Instance);
var rawDocument = RawDocumentFactory.CreateAdvisory(
tenant: "tenant-a",
source: new RawSourceMetadata("Vendor-X", "connector-y", "1.0.0", "stable"),
upstream: new RawUpstreamMetadata(
UpstreamId: "GHSA-2025-0001",
DocumentVersion: "v1",
RetrievedAt: DateTimeOffset.Parse("2025-10-29T12:34:56Z"),
ContentHash: "sha256:abc123",
Signature: new RawSignatureMetadata(true, "dsse", "key1", "sig1"),
Provenance: ImmutableDictionary.CreateRange(new[] { new KeyValuePair<string, string>("api", "https://example.test/api") })),
content: new RawContent(
Format: "OSV",
SpecVersion: "1.0.0",
Raw: ParseJsonElement("""{"id":"GHSA-2025-0001"}"""),
Encoding: null),
identifiers: new RawIdentifiers(
Aliases: ImmutableArray.Create("CVE-2025-0001", "cve-2025-0001"),
PrimaryId: "CVE-2025-0001"),
linkset: new RawLinkset
{
Aliases = ImmutableArray.Create("GHSA-xxxx-yyyy"),
PackageUrls = ImmutableArray.Create("pkg:npm/example@1.0.0"),
Cpes = ImmutableArray.Create("cpe:/a:example:product:1.0"),
References = ImmutableArray.Create(new RawReference("advisory", "https://example.test/advisory", "vendor")),
ReconciledFrom = ImmutableArray.Create("connector-y"),
Notes = ImmutableDictionary.CreateRange(new[] { new KeyValuePair<string, string>("range-fixed", "1.0.1") })
});
await rawRepository.UpsertAsync(rawDocument, CancellationToken.None);
var expectedRawLinkset = BuildRawLinkset(rawDocument.Identifiers, rawDocument.Linkset);
var canonicalAliases = ImmutableArray.Create("cve-2025-0001", "ghsa-xxxx-yyyy");
var canonicalPurls = rawDocument.Linkset.PackageUrls;
var canonicalCpes = rawDocument.Linkset.Cpes;
var canonicalReferences = rawDocument.Linkset.References;
var observationId = "tenant-a:vendor-x:ghsa-2025-0001:sha256-abc123";
var observationBson = BuildObservationDocument(
observationId,
rawDocument,
canonicalAliases,
canonicalPurls,
canonicalCpes,
canonicalReferences,
rawDocument.Upstream.RetrievedAt,
includeRawLinkset: false);
await database
.GetCollection<BsonDocument>(MongoStorageDefaults.Collections.AdvisoryObservations)
.InsertOneAsync(observationBson);
var migration = new EnsureAdvisoryObservationsRawLinksetMigration();
await migration.ApplyAsync(database, CancellationToken.None);
var storedBson = await database
.GetCollection<BsonDocument>(MongoStorageDefaults.Collections.AdvisoryObservations)
.Find(Builders<BsonDocument>.Filter.Eq("_id", observationId))
.FirstOrDefaultAsync();
Assert.NotNull(storedBson);
Assert.True(storedBson.TryGetValue("rawLinkset", out var rawLinksetValue));
var storedDocument = BsonSerializer.Deserialize<AdvisoryObservationDocument>(storedBson);
var storedObservation = AdvisoryObservationDocumentFactory.ToModel(storedDocument);
Assert.True(expectedRawLinkset.Aliases.SequenceEqual(storedObservation.RawLinkset.Aliases, StringComparer.Ordinal));
Assert.True(expectedRawLinkset.PackageUrls.SequenceEqual(storedObservation.RawLinkset.PackageUrls, StringComparer.Ordinal));
Assert.True(expectedRawLinkset.Cpes.SequenceEqual(storedObservation.RawLinkset.Cpes, StringComparer.Ordinal));
Assert.True(expectedRawLinkset.References.SequenceEqual(storedObservation.RawLinkset.References));
Assert.Equal(expectedRawLinkset.Notes, storedObservation.RawLinkset.Notes);
}
finally
{
await _fixture.Client.DropDatabaseAsync(databaseName);
}
}
[Fact]
public async Task ApplyAsync_ThrowsWhenRawDocumentMissing()
{
var databaseName = $"concelier-rawlinkset-missing-{Guid.NewGuid():N}";
var database = _fixture.Client.GetDatabase(databaseName);
await database.CreateCollectionAsync(MongoStorageDefaults.Collections.Migrations);
await database.CreateCollectionAsync(MongoStorageDefaults.Collections.AdvisoryObservations);
try
{
var rawDocument = RawDocumentFactory.CreateAdvisory(
tenant: "tenant-b",
source: new RawSourceMetadata("Vendor-Y", "connector-z", "2.0.0", "stable"),
upstream: new RawUpstreamMetadata(
UpstreamId: "GHSA-9999-0001",
DocumentVersion: "v2",
RetrievedAt: DateTimeOffset.Parse("2025-10-30T00:00:00Z"),
ContentHash: "sha256:def456",
Signature: new RawSignatureMetadata(false),
Provenance: ImmutableDictionary<string, string>.Empty),
content: new RawContent(
Format: "OSV",
SpecVersion: "1.0.0",
Raw: ParseJsonElement("""{"id":"GHSA-9999-0001"}"""),
Encoding: null),
identifiers: new RawIdentifiers(
Aliases: ImmutableArray<string>.Empty,
PrimaryId: "GHSA-9999-0001"),
linkset: new RawLinkset());
var observationId = "tenant-b:vendor-y:ghsa-9999-0001:sha256-def456";
var document = BuildObservationDocument(
observationId,
rawDocument,
ImmutableArray<string>.Empty,
ImmutableArray<string>.Empty,
ImmutableArray<string>.Empty,
ImmutableArray<RawReference>.Empty,
rawDocument.Upstream.RetrievedAt,
includeRawLinkset: false);
await database
.GetCollection<BsonDocument>(MongoStorageDefaults.Collections.AdvisoryObservations)
.InsertOneAsync(document);
var migration = new EnsureAdvisoryObservationsRawLinksetMigration();
await Assert.ThrowsAsync<InvalidOperationException>(
() => migration.ApplyAsync(database, CancellationToken.None));
}
finally
{
await _fixture.Client.DropDatabaseAsync(databaseName);
}
}
private static BsonDocument BuildObservationDocument(
string observationId,
AdvisoryRawDocument rawDocument,
ImmutableArray<string> canonicalAliases,
ImmutableArray<string> canonicalPurls,
ImmutableArray<string> canonicalCpes,
ImmutableArray<RawReference> canonicalReferences,
DateTimeOffset createdAt,
bool includeRawLinkset,
RawLinkset? rawLinkset = null)
{
var sourceDocument = new BsonDocument
{
{ "vendor", rawDocument.Source.Vendor },
{ "stream", string.IsNullOrWhiteSpace(rawDocument.Source.Stream) ? rawDocument.Source.Connector : rawDocument.Source.Stream! },
{ "api", rawDocument.Upstream.Provenance.TryGetValue("api", out var api) ? api : rawDocument.Source.Connector }
};
if (!string.IsNullOrWhiteSpace(rawDocument.Source.ConnectorVersion))
{
sourceDocument["collectorVersion"] = rawDocument.Source.ConnectorVersion;
}
var signatureDocument = new BsonDocument
{
{ "present", rawDocument.Upstream.Signature.Present }
};
if (!string.IsNullOrWhiteSpace(rawDocument.Upstream.Signature.Format))
{
signatureDocument["format"] = rawDocument.Upstream.Signature.Format;
}
if (!string.IsNullOrWhiteSpace(rawDocument.Upstream.Signature.KeyId))
{
signatureDocument["keyId"] = rawDocument.Upstream.Signature.KeyId;
}
if (!string.IsNullOrWhiteSpace(rawDocument.Upstream.Signature.Signature))
{
signatureDocument["signature"] = rawDocument.Upstream.Signature.Signature;
}
var upstreamDocument = new BsonDocument
{
{ "upstream_id", rawDocument.Upstream.UpstreamId },
{ "document_version", rawDocument.Upstream.DocumentVersion },
{ "fetchedAt", rawDocument.Upstream.RetrievedAt.UtcDateTime },
{ "receivedAt", rawDocument.Upstream.RetrievedAt.UtcDateTime },
{ "contentHash", rawDocument.Upstream.ContentHash },
{ "signature", signatureDocument },
{ "metadata", new BsonDocument(rawDocument.Upstream.Provenance) }
};
var contentDocument = new BsonDocument
{
{ "format", rawDocument.Content.Format },
{ "raw", BsonDocument.Parse(rawDocument.Content.Raw.GetRawText()) }
};
if (!string.IsNullOrWhiteSpace(rawDocument.Content.SpecVersion))
{
contentDocument["specVersion"] = rawDocument.Content.SpecVersion;
}
var canonicalLinkset = new BsonDocument
{
{ "aliases", new BsonArray(canonicalAliases) },
{ "purls", new BsonArray(canonicalPurls) },
{ "cpes", new BsonArray(canonicalCpes) },
{ "references", new BsonArray(canonicalReferences.Select(reference => new BsonDocument
{
{ "type", reference.Type },
{ "url", reference.Url }
})) }
};
var document = new BsonDocument
{
{ "_id", observationId },
{ "tenant", rawDocument.Tenant },
{ "source", sourceDocument },
{ "upstream", upstreamDocument },
{ "content", contentDocument },
{ "linkset", canonicalLinkset },
{ "createdAt", createdAt.UtcDateTime },
{ "attributes", new BsonDocument() }
};
if (includeRawLinkset)
{
var actualRawLinkset = rawLinkset ?? throw new ArgumentNullException(nameof(rawLinkset));
document["rawLinkset"] = new BsonDocument
{
{ "aliases", new BsonArray(actualRawLinkset.Aliases) },
{ "purls", new BsonArray(actualRawLinkset.PackageUrls) },
{ "cpes", new BsonArray(actualRawLinkset.Cpes) },
{ "references", new BsonArray(actualRawLinkset.References.Select(reference => new BsonDocument
{
{ "type", reference.Type },
{ "url", reference.Url },
{ "source", reference.Source }
})) },
{ "reconciled_from", new BsonArray(actualRawLinkset.ReconciledFrom) },
{ "notes", new BsonDocument(actualRawLinkset.Notes) }
};
}
return document;
}
private static JsonElement ParseJsonElement(string json)
{
using var document = JsonDocument.Parse(json);
return document.RootElement.Clone();
}
private static RawLinkset BuildRawLinkset(RawIdentifiers identifiers, RawLinkset linkset)
{
var aliasBuilder = ImmutableArray.CreateBuilder<string>();
if (!string.IsNullOrWhiteSpace(identifiers.PrimaryId))
{
aliasBuilder.Add(identifiers.PrimaryId);
}
if (!identifiers.Aliases.IsDefaultOrEmpty)
{
foreach (var alias in identifiers.Aliases)
{
if (!string.IsNullOrEmpty(alias))
{
aliasBuilder.Add(alias);
}
}
}
if (!linkset.Aliases.IsDefaultOrEmpty)
{
foreach (var alias in linkset.Aliases)
{
if (!string.IsNullOrEmpty(alias))
{
aliasBuilder.Add(alias);
}
}
}
static ImmutableArray<string> EnsureArray(ImmutableArray<string> values)
=> values.IsDefault ? ImmutableArray<string>.Empty : values;
static ImmutableArray<RawReference> EnsureReferences(ImmutableArray<RawReference> values)
=> values.IsDefault ? ImmutableArray<RawReference>.Empty : values;
return linkset with
{
Aliases = aliasBuilder.ToImmutable(),
PackageUrls = EnsureArray(linkset.PackageUrls),
Cpes = EnsureArray(linkset.Cpes),
References = EnsureReferences(linkset.References),
ReconciledFrom = EnsureArray(linkset.ReconciledFrom),
Notes = linkset.Notes ?? ImmutableDictionary<string, string>.Empty
};
}
}

70
src/Concelier/__Tests/StellaOps.Concelier.Storage.Mongo.Tests/Observations/AdvisoryObservationDocumentFactoryTests.cs
View File

@@ -37,32 +37,48 @@ public sealed class AdvisoryObservationDocumentFactoryTests
Signature = "signature"
}
},
Content = new AdvisoryObservationContentDocument
{
Format = "CSAF",
SpecVersion = "2.0",
Raw = BsonDocument.Parse("{\"example\":true}")
},
Linkset = new AdvisoryObservationLinksetDocument
{
Aliases = new List<string> { "CVE-2025-1234" },
Purls = new List<string> { "pkg:generic/foo@1.0.0" },
Cpes = new List<string> { "cpe:/a:vendor:product:1" },
References = new List<AdvisoryObservationReferenceDocument>
{
new() { Type = "advisory", Url = "https://example.com" }
}
}
};
var observation = AdvisoryObservationDocumentFactory.ToModel(document);
Assert.Equal("tenant-a:obs-1", observation.ObservationId);
Content = new AdvisoryObservationContentDocument
{
Format = "CSAF",
SpecVersion = "2.0",
Raw = BsonDocument.Parse("{\"example\":true}")
},
Linkset = new AdvisoryObservationLinksetDocument
{
Aliases = new List<string> { "CVE-2025-1234" },
Purls = new List<string> { "pkg:generic/foo@1.0.0" },
Cpes = new List<string> { "cpe:/a:vendor:product:1" },
References = new List<AdvisoryObservationReferenceDocument>
{
new() { Type = "advisory", Url = "https://example.com" }
}
},
RawLinkset = new AdvisoryObservationRawLinksetDocument
{
Aliases = new List<string> { "CVE-2025-1234", "cve-2025-1234" },
PackageUrls = new List<string> { "pkg:generic/foo@1.0.0" },
Cpes = new List<string> { "cpe:/a:vendor:product:1" },
References = new List<AdvisoryObservationRawReferenceDocument>
{
new() { Type = "Advisory", Url = "https://example.com", Source = "vendor" }
},
ReconciledFrom = new List<string> { "source-a" },
Notes = new Dictionary<string, string> { ["note-key"] = "note-value" }
}
};
var observation = AdvisoryObservationDocumentFactory.ToModel(document);
Assert.Equal("tenant-a:obs-1", observation.ObservationId);
Assert.Equal("tenant-a", observation.Tenant);
Assert.Equal("CVE-2025-1234", observation.Upstream.UpstreamId);
Assert.Contains("pkg:generic/foo@1.0.0", observation.Linkset.Purls);
Assert.Equal("CSAF", observation.Content.Format);
Assert.True(observation.Content.Raw?["example"]?.GetValue<bool>());
Assert.Equal("advisory", observation.Linkset.References[0].Type);
}
}
Assert.Contains("pkg:generic/foo@1.0.0", observation.Linkset.Purls);
Assert.Equal("CSAF", observation.Content.Format);
Assert.True(observation.Content.Raw?["example"]?.GetValue<bool>());
Assert.Equal("advisory", observation.Linkset.References[0].Type);
Assert.Equal(new[] { "CVE-2025-1234", "cve-2025-1234" }, observation.RawLinkset.Aliases);
Assert.Equal("Advisory", observation.RawLinkset.References[0].Type);
Assert.Equal("vendor", observation.RawLinkset.References[0].Source);
Assert.Equal("note-value", observation.RawLinkset.Notes["note-key"]);
}
}