feat: Add DigestUpsertRequest and LockEntity models
Some checks failed
AOC Guard CI / aoc-guard (push) Has been cancelled
AOC Guard CI / aoc-verify (push) Has been cancelled
Concelier Attestation Tests / attestation-tests (push) Has been cancelled
Docs CI / lint-and-preview (push) Has been cancelled
Export Center CI / export-ci (push) Has been cancelled
Mirror Thin Bundle Sign & Verify / mirror-sign (push) Has been cancelled
Some checks failed
AOC Guard CI / aoc-guard (push) Has been cancelled
AOC Guard CI / aoc-verify (push) Has been cancelled
Concelier Attestation Tests / attestation-tests (push) Has been cancelled
Docs CI / lint-and-preview (push) Has been cancelled
Export Center CI / export-ci (push) Has been cancelled
Mirror Thin Bundle Sign & Verify / mirror-sign (push) Has been cancelled
- Introduced DigestUpsertRequest for handling digest upsert requests with properties like ChannelId, Recipient, DigestKey, Events, and CollectUntil. - Created LockEntity to represent a lightweight distributed lock entry with properties such as Id, TenantId, Resource, Owner, ExpiresAt, and CreatedAt. feat: Implement ILockRepository interface and LockRepository class - Defined ILockRepository interface with methods for acquiring and releasing locks. - Implemented LockRepository class with methods to try acquiring a lock and releasing it, using SQL for upsert operations. feat: Add SurfaceManifestPointer record for manifest pointers - Introduced SurfaceManifestPointer to represent a minimal pointer to a Surface.FS manifest associated with an image digest. feat: Create PolicySimulationInputLock and related validation logic - Added PolicySimulationInputLock record to describe policy simulation inputs and expected digests. - Implemented validation logic for policy simulation inputs, including checks for digest drift and shadow mode requirements. test: Add unit tests for ReplayVerificationService and ReplayVerifier - Created ReplayVerificationServiceTests to validate the behavior of the ReplayVerificationService under various scenarios. - Developed ReplayVerifierTests to ensure the correctness of the ReplayVerifier logic. test: Implement PolicySimulationInputLockValidatorTests - Added tests for PolicySimulationInputLockValidator to verify the validation logic against expected inputs and conditions. chore: Add cosign key example and signing scripts - Included a placeholder cosign key example for development purposes. - Added a script for signing Signals artifacts using cosign with support for both v2 and v3. chore: Create script for uploading evidence to the evidence locker - Developed a script to upload evidence to the evidence locker, ensuring required environment variables are set.
This commit is contained in:
@@ -0,0 +1,30 @@
|
||||
# Concelier Storage.Postgres — Agent Charter
|
||||
|
||||
## Mission & Scope
|
||||
- Working directory: `src/Concelier/__Libraries/StellaOps.Concelier.Storage.Postgres`.
|
||||
- Deliver the PostgreSQL storage layer for Concelier vulnerability data (sources, advisories, aliases/CVSS/affected, KEV, states, snapshots, merge audit).
|
||||
- Keep behaviour deterministic, air-gap friendly, and aligned with the Link-Not-Merge (LNM) contract: ingest facts, don’t derive.
|
||||
|
||||
## Roles
|
||||
- **Backend engineer (.NET 10/Postgres):** repositories, migrations, connection plumbing, perf indexes.
|
||||
- **QA engineer:** integration tests using Testcontainers PostgreSQL, determinism and replacement semantics on child tables.
|
||||
|
||||
## Required Reading (treat as read before DOING)
|
||||
- `docs/README.md`, `docs/07_HIGH_LEVEL_ARCHITECTURE.md`
|
||||
- `docs/modules/platform/architecture-overview.md`
|
||||
- `docs/modules/concelier/architecture.md`
|
||||
- `docs/modules/concelier/link-not-merge-schema.md`
|
||||
- `docs/db/README.md`, `docs/db/SPECIFICATION.md` (Section 5.2), `docs/db/RULES.md`
|
||||
- Sprint doc: `docs/implplan/SPRINT_3405_0001_0001_postgres_vulnerabilities.md`
|
||||
|
||||
## Working Agreements
|
||||
- Determinism: stable ordering (ORDER BY in queries/tests), UTC timestamps, no random seeds; JSON kept canonical.
|
||||
- Offline-first: no network in code/tests; fixtures must be self-contained.
|
||||
- Tenant safety: vulnerability data is global; still pass `_system` tenant id through RepositoryBase; no caller-specific state.
|
||||
- Schema changes: update migration SQL and docs; keep search/vector triggers intact.
|
||||
- Status discipline: mirror `TODO → DOING → DONE/BLOCKED` in sprint docs when you start/finish/block tasks.
|
||||
|
||||
## Testing Rules
|
||||
- Use `ConcelierPostgresFixture` (Testcontainers PostgreSQL). Docker daemon must be available.
|
||||
- Before each test, truncate tables via fixture; avoid cross-test coupling.
|
||||
- Cover replacement semantics for child tables (aliases/CVSS/affected/etc.), search, PURL lookups, and source state cursor updates.
|
||||
@@ -0,0 +1,40 @@
|
||||
using StellaOps.Concelier.Storage.Mongo.Advisories;
|
||||
using StellaOps.Concelier.Storage.Postgres.Models;
|
||||
using StellaOps.Concelier.Storage.Postgres.Repositories;
|
||||
|
||||
namespace StellaOps.Concelier.Storage.Postgres.Converters;
|
||||
|
||||
/// <summary>
|
||||
/// Service to convert Mongo advisory documents and persist them into PostgreSQL.
|
||||
/// </summary>
|
||||
public sealed class AdvisoryConversionService
|
||||
{
|
||||
private readonly IAdvisoryRepository _advisories;
|
||||
|
||||
public AdvisoryConversionService(IAdvisoryRepository advisories)
|
||||
{
|
||||
_advisories = advisories;
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Converts a Mongo advisory document and persists it (upsert) with all child rows.
|
||||
/// </summary>
|
||||
public Task<AdvisoryEntity> ConvertAndUpsertAsync(
|
||||
AdvisoryDocument doc,
|
||||
string sourceKey,
|
||||
Guid sourceId,
|
||||
CancellationToken cancellationToken = default)
|
||||
{
|
||||
var result = AdvisoryConverter.Convert(doc, sourceKey, sourceId);
|
||||
return _advisories.UpsertAsync(
|
||||
result.Advisory,
|
||||
result.Aliases,
|
||||
result.Cvss,
|
||||
result.Affected,
|
||||
result.References,
|
||||
result.Credits,
|
||||
result.Weaknesses,
|
||||
result.KevFlags,
|
||||
cancellationToken);
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,297 @@
|
||||
using System.Collections.Immutable;
|
||||
using System.Text.Json;
|
||||
using StellaOps.Concelier.Storage.Mongo.Advisories;
|
||||
using StellaOps.Concelier.Storage.Postgres.Models;
|
||||
|
||||
namespace StellaOps.Concelier.Storage.Postgres.Converters;
|
||||
|
||||
/// <summary>
|
||||
/// Converts Mongo advisory documents to Postgres advisory entities and child collections.
|
||||
/// Deterministic: ordering of child collections is preserved (sorted for stable SQL writes).
|
||||
/// </summary>
|
||||
public static class AdvisoryConverter
|
||||
{
|
||||
public sealed record Result(
|
||||
AdvisoryEntity Advisory,
|
||||
IReadOnlyList<AdvisoryAliasEntity> Aliases,
|
||||
IReadOnlyList<AdvisoryCvssEntity> Cvss,
|
||||
IReadOnlyList<AdvisoryAffectedEntity> Affected,
|
||||
IReadOnlyList<AdvisoryReferenceEntity> References,
|
||||
IReadOnlyList<AdvisoryCreditEntity> Credits,
|
||||
IReadOnlyList<AdvisoryWeaknessEntity> Weaknesses,
|
||||
IReadOnlyList<KevFlagEntity> KevFlags);
|
||||
|
||||
/// <summary>
|
||||
/// Maps a Mongo AdvisoryDocument and its raw payload into Postgres entities.
|
||||
/// </summary>
|
||||
public static Result Convert(
|
||||
AdvisoryDocument doc,
|
||||
string sourceKey,
|
||||
Guid sourceId,
|
||||
string? contentHash = null)
|
||||
{
|
||||
var now = DateTimeOffset.UtcNow;
|
||||
|
||||
// Top-level advisory
|
||||
var advisoryId = Guid.NewGuid();
|
||||
var payloadJson = doc.Payload.ToJson();
|
||||
var provenanceJson = JsonSerializer.Serialize(new { source = sourceKey });
|
||||
|
||||
var advisory = new AdvisoryEntity
|
||||
{
|
||||
Id = advisoryId,
|
||||
AdvisoryKey = doc.AdvisoryKey,
|
||||
PrimaryVulnId = doc.Payload.GetValue("primaryVulnId", doc.AdvisoryKey)?.ToString() ?? doc.AdvisoryKey,
|
||||
SourceId = sourceId,
|
||||
Title = doc.Payload.GetValue("title", null)?.ToString(),
|
||||
Summary = doc.Payload.GetValue("summary", null)?.ToString(),
|
||||
Description = doc.Payload.GetValue("description", null)?.ToString(),
|
||||
Severity = doc.Payload.GetValue("severity", null)?.ToString(),
|
||||
PublishedAt = doc.Published.HasValue ? DateTime.SpecifyKind(doc.Published.Value, DateTimeKind.Utc) : null,
|
||||
ModifiedAt = DateTime.SpecifyKind(doc.Modified, DateTimeKind.Utc),
|
||||
WithdrawnAt = doc.Payload.TryGetValue("withdrawnAt", out var withdrawn) && withdrawn.IsValidDateTime
|
||||
? withdrawn.ToUniversalTime()
|
||||
: null,
|
||||
Provenance = provenanceJson,
|
||||
RawPayload = payloadJson,
|
||||
CreatedAt = now,
|
||||
UpdatedAt = now
|
||||
};
|
||||
|
||||
// Aliases
|
||||
var aliases = doc.Payload.TryGetValue("aliases", out var aliasesBson) && aliasesBson.IsBsonArray
|
||||
? aliasesBson.AsBsonArray.Select(v => v.ToString() ?? string.Empty)
|
||||
: Enumerable.Empty<string>();
|
||||
|
||||
var aliasEntities = aliases
|
||||
.Where(a => !string.IsNullOrWhiteSpace(a))
|
||||
.Distinct(StringComparer.OrdinalIgnoreCase)
|
||||
.OrderBy(a => a, StringComparer.OrdinalIgnoreCase)
|
||||
.Select((alias, idx) => new AdvisoryAliasEntity
|
||||
{
|
||||
Id = Guid.NewGuid(),
|
||||
AdvisoryId = advisoryId,
|
||||
AliasType = alias.StartsWith("CVE-", StringComparison.OrdinalIgnoreCase) ? "CVE" : "OTHER",
|
||||
AliasValue = alias,
|
||||
IsPrimary = idx == 0,
|
||||
CreatedAt = now
|
||||
})
|
||||
.ToArray();
|
||||
|
||||
// CVSS
|
||||
var cvssEntities = BuildCvssEntities(doc, advisoryId, now);
|
||||
|
||||
// Affected
|
||||
var affectedEntities = BuildAffectedEntities(doc, advisoryId, now);
|
||||
|
||||
// References
|
||||
var referencesEntities = BuildReferenceEntities(doc, advisoryId, now);
|
||||
|
||||
// Credits
|
||||
var creditEntities = BuildCreditEntities(doc, advisoryId, now);
|
||||
|
||||
// Weaknesses
|
||||
var weaknessEntities = BuildWeaknessEntities(doc, advisoryId, now);
|
||||
|
||||
// KEV flags (from payload.kev if present)
|
||||
var kevEntities = BuildKevEntities(doc, advisoryId, now);
|
||||
|
||||
return new Result(
|
||||
advisory,
|
||||
aliasEntities,
|
||||
cvssEntities,
|
||||
affectedEntities,
|
||||
referencesEntities,
|
||||
creditEntities,
|
||||
weaknessEntities,
|
||||
kevEntities);
|
||||
}
|
||||
|
||||
private static IReadOnlyList<AdvisoryCvssEntity> BuildCvssEntities(AdvisoryDocument doc, Guid advisoryId, DateTimeOffset now)
|
||||
{
|
||||
if (!doc.Payload.TryGetValue("cvss", out var cvssValue) || !cvssValue.IsBsonArray)
|
||||
{
|
||||
return Array.Empty<AdvisoryCvssEntity>();
|
||||
}
|
||||
|
||||
return cvssValue.AsBsonArray
|
||||
.Where(v => v.IsBsonDocument)
|
||||
.Select(v => v.AsBsonDocument)
|
||||
.Select(d => new AdvisoryCvssEntity
|
||||
{
|
||||
Id = Guid.NewGuid(),
|
||||
AdvisoryId = advisoryId,
|
||||
CvssVersion = d.GetValue("version", "3.1").ToString() ?? "3.1",
|
||||
VectorString = d.GetValue("vector", string.Empty).ToString() ?? string.Empty,
|
||||
BaseScore = d.GetValue("baseScore", 0m).ToDecimal(),
|
||||
BaseSeverity = d.GetValue("baseSeverity", null)?.ToString(),
|
||||
ExploitabilityScore = d.GetValue("exploitabilityScore", null)?.ToNullableDecimal(),
|
||||
ImpactScore = d.GetValue("impactScore", null)?.ToNullableDecimal(),
|
||||
Source = d.GetValue("source", null)?.ToString(),
|
||||
IsPrimary = d.GetValue("isPrimary", false).ToBoolean(),
|
||||
CreatedAt = now
|
||||
})
|
||||
.OrderByDescending(c => c.IsPrimary)
|
||||
.ThenByDescending(c => c.BaseScore)
|
||||
.ThenBy(c => c.Id)
|
||||
.ToArray();
|
||||
}
|
||||
|
||||
private static IReadOnlyList<AdvisoryAffectedEntity> BuildAffectedEntities(AdvisoryDocument doc, Guid advisoryId, DateTimeOffset now)
|
||||
{
|
||||
if (!doc.Payload.TryGetValue("affected", out var affectedValue) || !affectedValue.IsBsonArray)
|
||||
{
|
||||
return Array.Empty<AdvisoryAffectedEntity>();
|
||||
}
|
||||
|
||||
return affectedValue.AsBsonArray
|
||||
.Where(v => v.IsBsonDocument)
|
||||
.Select(v => v.AsBsonDocument)
|
||||
.Select(d => new AdvisoryAffectedEntity
|
||||
{
|
||||
Id = Guid.NewGuid(),
|
||||
AdvisoryId = advisoryId,
|
||||
Ecosystem = d.GetValue("ecosystem", string.Empty).ToString() ?? string.Empty,
|
||||
PackageName = d.GetValue("packageName", string.Empty).ToString() ?? string.Empty,
|
||||
Purl = d.GetValue("purl", null)?.ToString(),
|
||||
VersionRange = d.GetValue("range", "{}").ToString() ?? "{}",
|
||||
VersionsAffected = d.TryGetValue("versionsAffected", out var va) && va.IsBsonArray
|
||||
? va.AsBsonArray.Select(x => x.ToString() ?? string.Empty).ToArray()
|
||||
: null,
|
||||
VersionsFixed = d.TryGetValue("versionsFixed", out var vf) && vf.IsBsonArray
|
||||
? vf.AsBsonArray.Select(x => x.ToString() ?? string.Empty).ToArray()
|
||||
: null,
|
||||
DatabaseSpecific = d.GetValue("databaseSpecific", null)?.ToString(),
|
||||
CreatedAt = now
|
||||
})
|
||||
.OrderBy(a => a.Ecosystem)
|
||||
.ThenBy(a => a.PackageName)
|
||||
.ThenBy(a => a.Purl)
|
||||
.ToArray();
|
||||
}
|
||||
|
||||
private static IReadOnlyList<AdvisoryReferenceEntity> BuildReferenceEntities(AdvisoryDocument doc, Guid advisoryId, DateTimeOffset now)
|
||||
{
|
||||
if (!doc.Payload.TryGetValue("references", out var referencesValue) || !referencesValue.IsBsonArray)
|
||||
{
|
||||
return Array.Empty<AdvisoryReferenceEntity>();
|
||||
}
|
||||
|
||||
return referencesValue.AsBsonArray
|
||||
.Where(v => v.IsBsonDocument)
|
||||
.Select(v => v.AsBsonDocument)
|
||||
.Select(r => new AdvisoryReferenceEntity
|
||||
{
|
||||
Id = Guid.NewGuid(),
|
||||
AdvisoryId = advisoryId,
|
||||
RefType = r.GetValue("type", "advisory").ToString() ?? "advisory",
|
||||
Url = r.GetValue("url", string.Empty).ToString() ?? string.Empty,
|
||||
CreatedAt = now
|
||||
})
|
||||
.OrderBy(r => r.Url)
|
||||
.ToArray();
|
||||
}
|
||||
|
||||
private static IReadOnlyList<AdvisoryCreditEntity> BuildCreditEntities(AdvisoryDocument doc, Guid advisoryId, DateTimeOffset now)
|
||||
{
|
||||
if (!doc.Payload.TryGetValue("credits", out var creditsValue) || !creditsValue.IsBsonArray)
|
||||
{
|
||||
return Array.Empty<AdvisoryCreditEntity>();
|
||||
}
|
||||
|
||||
return creditsValue.AsBsonArray
|
||||
.Where(v => v.IsBsonDocument)
|
||||
.Select(v => v.AsBsonDocument)
|
||||
.Select(c => new AdvisoryCreditEntity
|
||||
{
|
||||
Id = Guid.NewGuid(),
|
||||
AdvisoryId = advisoryId,
|
||||
Name = c.GetValue("name", string.Empty).ToString() ?? string.Empty,
|
||||
Contact = c.GetValue("contact", null)?.ToString(),
|
||||
CreditType = c.GetValue("type", null)?.ToString(),
|
||||
CreatedAt = now
|
||||
})
|
||||
.OrderBy(c => c.Name)
|
||||
.ThenBy(c => c.Contact)
|
||||
.ToArray();
|
||||
}
|
||||
|
||||
private static IReadOnlyList<AdvisoryWeaknessEntity> BuildWeaknessEntities(AdvisoryDocument doc, Guid advisoryId, DateTimeOffset now)
|
||||
{
|
||||
if (!doc.Payload.TryGetValue("weaknesses", out var weaknessesValue) || !weaknessesValue.IsBsonArray)
|
||||
{
|
||||
return Array.Empty<AdvisoryWeaknessEntity>();
|
||||
}
|
||||
|
||||
return weaknessesValue.AsBsonArray
|
||||
.Where(v => v.IsBsonDocument)
|
||||
.Select(v => v.AsBsonDocument)
|
||||
.Select(w => new AdvisoryWeaknessEntity
|
||||
{
|
||||
Id = Guid.NewGuid(),
|
||||
AdvisoryId = advisoryId,
|
||||
CweId = w.GetValue("cweId", string.Empty).ToString() ?? string.Empty,
|
||||
Description = w.GetValue("description", null)?.ToString(),
|
||||
Source = w.GetValue("source", null)?.ToString(),
|
||||
CreatedAt = now
|
||||
})
|
||||
.OrderBy(w => w.CweId)
|
||||
.ToArray();
|
||||
}
|
||||
|
||||
private static IReadOnlyList<KevFlagEntity> BuildKevEntities(AdvisoryDocument doc, Guid advisoryId, DateTimeOffset now)
|
||||
{
|
||||
if (!doc.Payload.TryGetValue("kev", out var kevValue) || !kevValue.IsBsonArray)
|
||||
{
|
||||
return Array.Empty<KevFlagEntity>();
|
||||
}
|
||||
|
||||
var today = DateOnly.FromDateTime(now.UtcDateTime);
|
||||
return kevValue.AsBsonArray
|
||||
.Where(v => v.IsBsonDocument)
|
||||
.Select(v => v.AsBsonDocument)
|
||||
.Select(k => new KevFlagEntity
|
||||
{
|
||||
Id = Guid.NewGuid(),
|
||||
AdvisoryId = advisoryId,
|
||||
CveId = k.GetValue("cveId", string.Empty).ToString() ?? string.Empty,
|
||||
VendorProject = k.GetValue("vendorProject", null)?.ToString(),
|
||||
Product = k.GetValue("product", null)?.ToString(),
|
||||
VulnerabilityName = k.GetValue("name", null)?.ToString(),
|
||||
DateAdded = k.TryGetValue("dateAdded", out var dateAdded) && dateAdded.IsValidDateTime
|
||||
? DateOnly.FromDateTime(dateAdded.ToUniversalTime().Date)
|
||||
: today,
|
||||
DueDate = k.TryGetValue("dueDate", out var dueDate) && dueDate.IsValidDateTime
|
||||
? DateOnly.FromDateTime(dueDate.ToUniversalTime().Date)
|
||||
: null,
|
||||
KnownRansomwareUse = k.GetValue("knownRansomwareUse", false).ToBoolean(),
|
||||
Notes = k.GetValue("notes", null)?.ToString(),
|
||||
CreatedAt = now
|
||||
})
|
||||
.OrderBy(k => k.CveId)
|
||||
.ToArray();
|
||||
}
|
||||
|
||||
private static decimal ToDecimal(this object value)
|
||||
=> value switch
|
||||
{
|
||||
decimal d => d,
|
||||
double d => (decimal)d,
|
||||
float f => (decimal)f,
|
||||
IConvertible c => c.ToDecimal(null),
|
||||
_ => 0m
|
||||
};
|
||||
|
||||
private static decimal? ToNullableDecimal(this object? value)
|
||||
{
|
||||
if (value is null) return null;
|
||||
return value switch
|
||||
{
|
||||
decimal d => d,
|
||||
double d => (decimal)d,
|
||||
float f => (decimal)f,
|
||||
IConvertible c => c.ToDecimal(null),
|
||||
_ => null
|
||||
};
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,66 @@
|
||||
using MongoDB.Driver;
|
||||
using StellaOps.Concelier.Storage.Mongo.Advisories;
|
||||
using StellaOps.Concelier.Storage.Postgres.Models;
|
||||
using StellaOps.Concelier.Storage.Postgres.Repositories;
|
||||
|
||||
namespace StellaOps.Concelier.Storage.Postgres.Converters.Importers;
|
||||
|
||||
/// <summary>
|
||||
/// Imports GHSA/vendor advisories from Mongo into PostgreSQL.
|
||||
/// </summary>
|
||||
public sealed class GhsaImporter
|
||||
{
|
||||
private readonly IMongoCollection<AdvisoryDocument> _collection;
|
||||
private readonly AdvisoryConversionService _conversionService;
|
||||
private readonly IFeedSnapshotRepository _feedSnapshots;
|
||||
private readonly IAdvisorySnapshotRepository _advisorySnapshots;
|
||||
|
||||
public GhsaImporter(
|
||||
IMongoCollection<AdvisoryDocument> collection,
|
||||
AdvisoryConversionService conversionService,
|
||||
IFeedSnapshotRepository feedSnapshots,
|
||||
IAdvisorySnapshotRepository advisorySnapshots)
|
||||
{
|
||||
_collection = collection;
|
||||
_conversionService = conversionService;
|
||||
_feedSnapshots = feedSnapshots;
|
||||
_advisorySnapshots = advisorySnapshots;
|
||||
}
|
||||
|
||||
public async Task ImportSnapshotAsync(
|
||||
Guid sourceId,
|
||||
string sourceKey,
|
||||
string snapshotId,
|
||||
CancellationToken cancellationToken)
|
||||
{
|
||||
var advisories = await _collection
|
||||
.Find(Builders<AdvisoryDocument>.Filter.Empty)
|
||||
.ToListAsync(cancellationToken)
|
||||
.ConfigureAwait(false);
|
||||
|
||||
var feedSnapshot = await _feedSnapshots.InsertAsync(new FeedSnapshotEntity
|
||||
{
|
||||
Id = Guid.NewGuid(),
|
||||
SourceId = sourceId,
|
||||
SnapshotId = snapshotId,
|
||||
AdvisoryCount = advisories.Count,
|
||||
Metadata = $"{{\"source\":\"{sourceKey}\"}}",
|
||||
CreatedAt = DateTimeOffset.UtcNow
|
||||
}, cancellationToken).ConfigureAwait(false);
|
||||
|
||||
foreach (var advisory in advisories)
|
||||
{
|
||||
var stored = await _conversionService.ConvertAndUpsertAsync(advisory, sourceKey, sourceId, cancellationToken)
|
||||
.ConfigureAwait(false);
|
||||
|
||||
await _advisorySnapshots.InsertAsync(new AdvisorySnapshotEntity
|
||||
{
|
||||
Id = Guid.NewGuid(),
|
||||
FeedSnapshotId = feedSnapshot.Id,
|
||||
AdvisoryKey = stored.AdvisoryKey,
|
||||
ContentHash = advisory.Payload.GetValue("hash", advisory.AdvisoryKey)?.ToString() ?? advisory.AdvisoryKey,
|
||||
CreatedAt = DateTimeOffset.UtcNow
|
||||
}, cancellationToken).ConfigureAwait(false);
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,68 @@
|
||||
using System.Text.Json;
|
||||
using MongoDB.Driver;
|
||||
using StellaOps.Concelier.Storage.Mongo.Advisories;
|
||||
using StellaOps.Concelier.Storage.Postgres.Models;
|
||||
using StellaOps.Concelier.Storage.Postgres.Repositories;
|
||||
|
||||
namespace StellaOps.Concelier.Storage.Postgres.Converters.Importers;
|
||||
|
||||
/// <summary>
|
||||
/// Imports NVD advisory documents from Mongo into PostgreSQL using the advisory converter.
|
||||
/// </summary>
|
||||
public sealed class NvdImporter
|
||||
{
|
||||
private readonly IMongoCollection<AdvisoryDocument> _collection;
|
||||
private readonly AdvisoryConversionService _conversionService;
|
||||
private readonly IFeedSnapshotRepository _feedSnapshots;
|
||||
private readonly IAdvisorySnapshotRepository _advisorySnapshots;
|
||||
|
||||
public NvdImporter(
|
||||
IMongoCollection<AdvisoryDocument> collection,
|
||||
AdvisoryConversionService conversionService,
|
||||
IFeedSnapshotRepository feedSnapshots,
|
||||
IAdvisorySnapshotRepository advisorySnapshots)
|
||||
{
|
||||
_collection = collection;
|
||||
_conversionService = conversionService;
|
||||
_feedSnapshots = feedSnapshots;
|
||||
_advisorySnapshots = advisorySnapshots;
|
||||
}
|
||||
|
||||
public async Task ImportSnapshotAsync(
|
||||
Guid sourceId,
|
||||
string sourceKey,
|
||||
string snapshotId,
|
||||
CancellationToken cancellationToken)
|
||||
{
|
||||
var advisories = await _collection
|
||||
.Find(Builders<AdvisoryDocument>.Filter.Empty)
|
||||
.ToListAsync(cancellationToken)
|
||||
.ConfigureAwait(false);
|
||||
|
||||
var feedSnapshot = await _feedSnapshots.InsertAsync(new FeedSnapshotEntity
|
||||
{
|
||||
Id = Guid.NewGuid(),
|
||||
SourceId = sourceId,
|
||||
SnapshotId = snapshotId,
|
||||
AdvisoryCount = advisories.Count,
|
||||
Checksum = null,
|
||||
Metadata = JsonSerializer.Serialize(new { source = sourceKey, snapshot = snapshotId }),
|
||||
CreatedAt = DateTimeOffset.UtcNow
|
||||
}, cancellationToken).ConfigureAwait(false);
|
||||
|
||||
foreach (var advisory in advisories)
|
||||
{
|
||||
var stored = await _conversionService.ConvertAndUpsertAsync(advisory, sourceKey, sourceId, cancellationToken)
|
||||
.ConfigureAwait(false);
|
||||
|
||||
await _advisorySnapshots.InsertAsync(new AdvisorySnapshotEntity
|
||||
{
|
||||
Id = Guid.NewGuid(),
|
||||
FeedSnapshotId = feedSnapshot.Id,
|
||||
AdvisoryKey = stored.AdvisoryKey,
|
||||
ContentHash = advisory.Payload.GetValue("hash", advisory.AdvisoryKey)?.ToString() ?? advisory.AdvisoryKey,
|
||||
CreatedAt = DateTimeOffset.UtcNow
|
||||
}, cancellationToken).ConfigureAwait(false);
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,65 @@
|
||||
using MongoDB.Driver;
|
||||
using StellaOps.Concelier.Storage.Mongo.Advisories;
|
||||
using StellaOps.Concelier.Storage.Postgres.Models;
|
||||
using StellaOps.Concelier.Storage.Postgres.Repositories;
|
||||
|
||||
namespace StellaOps.Concelier.Storage.Postgres.Converters.Importers;
|
||||
|
||||
/// <summary>
|
||||
/// Imports OSV advisories from Mongo into PostgreSQL.
|
||||
/// </summary>
|
||||
public sealed class OsvImporter
|
||||
{
|
||||
private readonly IMongoCollection<AdvisoryDocument> _collection;
|
||||
private readonly AdvisoryConversionService _conversionService;
|
||||
private readonly IFeedSnapshotRepository _feedSnapshots;
|
||||
private readonly IAdvisorySnapshotRepository _advisorySnapshots;
|
||||
|
||||
public OsvImporter(
|
||||
IMongoCollection<AdvisoryDocument> collection,
|
||||
AdvisoryConversionService conversionService,
|
||||
IFeedSnapshotRepository feedSnapshots,
|
||||
IAdvisorySnapshotRepository advisorySnapshots)
|
||||
{
|
||||
_collection = collection;
|
||||
_conversionService = conversionService;
|
||||
_feedSnapshots = feedSnapshots;
|
||||
_advisorySnapshots = advisorySnapshots;
|
||||
}
|
||||
|
||||
public async Task ImportSnapshotAsync(
|
||||
Guid sourceId,
|
||||
string snapshotId,
|
||||
CancellationToken cancellationToken)
|
||||
{
|
||||
var advisories = await _collection
|
||||
.Find(Builders<AdvisoryDocument>.Filter.Empty)
|
||||
.ToListAsync(cancellationToken)
|
||||
.ConfigureAwait(false);
|
||||
|
||||
var feedSnapshot = await _feedSnapshots.InsertAsync(new FeedSnapshotEntity
|
||||
{
|
||||
Id = Guid.NewGuid(),
|
||||
SourceId = sourceId,
|
||||
SnapshotId = snapshotId,
|
||||
AdvisoryCount = advisories.Count,
|
||||
Metadata = "{\"source\":\"osv\"}",
|
||||
CreatedAt = DateTimeOffset.UtcNow
|
||||
}, cancellationToken).ConfigureAwait(false);
|
||||
|
||||
foreach (var advisory in advisories)
|
||||
{
|
||||
var stored = await _conversionService.ConvertAndUpsertAsync(advisory, "osv", sourceId, cancellationToken)
|
||||
.ConfigureAwait(false);
|
||||
|
||||
await _advisorySnapshots.InsertAsync(new AdvisorySnapshotEntity
|
||||
{
|
||||
Id = Guid.NewGuid(),
|
||||
FeedSnapshotId = feedSnapshot.Id,
|
||||
AdvisoryKey = stored.AdvisoryKey,
|
||||
ContentHash = advisory.Payload.GetValue("hash", advisory.AdvisoryKey)?.ToString() ?? advisory.AdvisoryKey,
|
||||
CreatedAt = DateTimeOffset.UtcNow
|
||||
}, cancellationToken).ConfigureAwait(false);
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -16,6 +16,7 @@
|
||||
|
||||
<ItemGroup>
|
||||
<ProjectReference Include="..\..\..\__Libraries\StellaOps.Infrastructure.Postgres\StellaOps.Infrastructure.Postgres.csproj" />
|
||||
<ProjectReference Include="..\StellaOps.Concelier.Storage.Mongo\StellaOps.Concelier.Storage.Mongo.csproj" />
|
||||
</ItemGroup>
|
||||
|
||||
</Project>
|
||||
|
||||
@@ -0,0 +1,90 @@
|
||||
using FluentAssertions;
|
||||
using Microsoft.Extensions.Logging.Abstractions;
|
||||
using Microsoft.Extensions.Options;
|
||||
using MongoDB.Bson;
|
||||
using StellaOps.Concelier.Storage.Mongo.Advisories;
|
||||
using StellaOps.Concelier.Storage.Postgres;
|
||||
using StellaOps.Concelier.Storage.Postgres.Converters;
|
||||
using StellaOps.Concelier.Storage.Postgres.Repositories;
|
||||
using Xunit;
|
||||
|
||||
namespace StellaOps.Concelier.Storage.Postgres.Tests;
|
||||
|
||||
[Collection(ConcelierPostgresCollection.Name)]
|
||||
public sealed class AdvisoryConversionServiceTests : IAsyncLifetime
|
||||
{
|
||||
private readonly ConcelierPostgresFixture _fixture;
|
||||
private readonly AdvisoryConversionService _service;
|
||||
private readonly AdvisoryRepository _advisories;
|
||||
private readonly AdvisoryAliasRepository _aliases;
|
||||
private readonly AdvisoryAffectedRepository _affected;
|
||||
|
||||
public AdvisoryConversionServiceTests(ConcelierPostgresFixture fixture)
|
||||
{
|
||||
_fixture = fixture;
|
||||
var options = fixture.Fixture.CreateOptions();
|
||||
options.SchemaName = fixture.SchemaName;
|
||||
var dataSource = new ConcelierDataSource(Options.Create(options), NullLogger<ConcelierDataSource>.Instance);
|
||||
|
||||
_advisories = new AdvisoryRepository(dataSource, NullLogger<AdvisoryRepository>.Instance);
|
||||
_aliases = new AdvisoryAliasRepository(dataSource, NullLogger<AdvisoryAliasRepository>.Instance);
|
||||
_affected = new AdvisoryAffectedRepository(dataSource, NullLogger<AdvisoryAffectedRepository>.Instance);
|
||||
_service = new AdvisoryConversionService(_advisories);
|
||||
}
|
||||
|
||||
public Task InitializeAsync() => _fixture.TruncateAllTablesAsync();
|
||||
public Task DisposeAsync() => Task.CompletedTask;
|
||||
|
||||
[Fact]
|
||||
public async Task ConvertAndUpsert_PersistsAdvisoryAndChildren()
|
||||
{
|
||||
var doc = CreateDoc();
|
||||
var sourceId = Guid.NewGuid();
|
||||
|
||||
var stored = await _service.ConvertAndUpsertAsync(doc, "osv", sourceId);
|
||||
|
||||
var fetched = await _advisories.GetByKeyAsync(doc.AdvisoryKey);
|
||||
var aliases = await _aliases.GetByAdvisoryAsync(stored.Id);
|
||||
var affected = await _affected.GetByAdvisoryAsync(stored.Id);
|
||||
|
||||
fetched.Should().NotBeNull();
|
||||
fetched!.PrimaryVulnId.Should().Be("CVE-2024-0002");
|
||||
fetched.RawPayload.Should().NotBeNull();
|
||||
fetched.Provenance.Should().Contain("osv");
|
||||
aliases.Should().NotBeEmpty();
|
||||
affected.Should().ContainSingle(a => a.Purl == "pkg:npm/example@2.0.0");
|
||||
affected[0].VersionRange.Should().Contain("introduced");
|
||||
}
|
||||
|
||||
private static AdvisoryDocument CreateDoc()
|
||||
{
|
||||
var payload = new BsonDocument
|
||||
{
|
||||
{ "primaryVulnId", "CVE-2024-0002" },
|
||||
{ "title", "Another advisory" },
|
||||
{ "severity", "medium" },
|
||||
{ "aliases", new BsonArray { "CVE-2024-0002" } },
|
||||
{ "affected", new BsonArray
|
||||
{
|
||||
new BsonDocument
|
||||
{
|
||||
{ "ecosystem", "npm" },
|
||||
{ "packageName", "example" },
|
||||
{ "purl", "pkg:npm/example@2.0.0" },
|
||||
{ "range", "{\"introduced\":\"0\",\"fixed\":\"2.0.1\"}" },
|
||||
{ "versionsAffected", new BsonArray { "2.0.0" } },
|
||||
{ "versionsFixed", new BsonArray { "2.0.1" } }
|
||||
}
|
||||
}
|
||||
}
|
||||
};
|
||||
|
||||
return new AdvisoryDocument
|
||||
{
|
||||
AdvisoryKey = "ADV-2",
|
||||
Payload = payload,
|
||||
Modified = DateTime.UtcNow,
|
||||
Published = DateTime.UtcNow.AddDays(-2)
|
||||
};
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,122 @@
|
||||
using FluentAssertions;
|
||||
using MongoDB.Bson;
|
||||
using StellaOps.Concelier.Storage.Mongo.Advisories;
|
||||
using StellaOps.Concelier.Storage.Postgres.Converters;
|
||||
using Xunit;
|
||||
|
||||
namespace StellaOps.Concelier.Storage.Postgres.Tests;
|
||||
|
||||
public sealed class AdvisoryConverterTests
|
||||
{
|
||||
[Fact]
|
||||
public void Convert_MapsCoreFieldsAndChildren()
|
||||
{
|
||||
var doc = CreateAdvisoryDocument();
|
||||
|
||||
var result = AdvisoryConverter.Convert(doc, sourceKey: "osv", sourceId: Guid.Parse("aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa"));
|
||||
|
||||
result.Advisory.AdvisoryKey.Should().Be("ADV-1");
|
||||
result.Advisory.PrimaryVulnId.Should().Be("CVE-2024-0001");
|
||||
result.Advisory.Severity.Should().Be("high");
|
||||
result.Aliases.Should().ContainSingle(a => a.AliasValue == "CVE-2024-0001");
|
||||
result.Cvss.Should().ContainSingle(c => c.BaseScore == 9.8m && c.BaseSeverity == "critical");
|
||||
result.Affected.Should().ContainSingle(a => a.Purl == "pkg:npm/example@1.0.0");
|
||||
result.References.Should().ContainSingle(r => r.Url == "https://ref.example/test");
|
||||
result.Credits.Should().ContainSingle(c => c.Name == "Researcher One");
|
||||
result.Weaknesses.Should().ContainSingle(w => w.CweId == "CWE-79");
|
||||
result.KevFlags.Should().ContainSingle(k => k.CveId == "CVE-2024-0001");
|
||||
}
|
||||
|
||||
private static AdvisoryDocument CreateAdvisoryDocument()
|
||||
{
|
||||
var payload = new BsonDocument
|
||||
{
|
||||
{ "primaryVulnId", "CVE-2024-0001" },
|
||||
{ "title", "Sample Advisory" },
|
||||
{ "summary", "Summary" },
|
||||
{ "description", "Description" },
|
||||
{ "severity", "high" },
|
||||
{ "aliases", new BsonArray { "CVE-2024-0001", "GHSA-aaaa-bbbb-cccc" } },
|
||||
{ "cvss", new BsonArray
|
||||
{
|
||||
new BsonDocument
|
||||
{
|
||||
{ "version", "3.1" },
|
||||
{ "vector", "CVSS:3.1/AV:N/AC:L/PR:N/UI:N/S:U/C:H/I:H/A:H" },
|
||||
{ "baseScore", 9.8 },
|
||||
{ "baseSeverity", "critical" },
|
||||
{ "exploitabilityScore", 3.9 },
|
||||
{ "impactScore", 5.9 },
|
||||
{ "source", "nvd" },
|
||||
{ "isPrimary", true }
|
||||
}
|
||||
}
|
||||
},
|
||||
{ "affected", new BsonArray
|
||||
{
|
||||
new BsonDocument
|
||||
{
|
||||
{ "ecosystem", "npm" },
|
||||
{ "packageName", "example" },
|
||||
{ "purl", "pkg:npm/example@1.0.0" },
|
||||
{ "range", "{\"introduced\":\"0\",\"fixed\":\"1.0.1\"}" },
|
||||
{ "versionsAffected", new BsonArray { "1.0.0" } },
|
||||
{ "versionsFixed", new BsonArray { "1.0.1" } },
|
||||
{ "databaseSpecific", "{\"severity\":\"high\"}" }
|
||||
}
|
||||
}
|
||||
},
|
||||
{ "references", new BsonArray
|
||||
{
|
||||
new BsonDocument
|
||||
{
|
||||
{ "type", "advisory" },
|
||||
{ "url", "https://ref.example/test" }
|
||||
}
|
||||
}
|
||||
},
|
||||
{ "credits", new BsonArray
|
||||
{
|
||||
new BsonDocument
|
||||
{
|
||||
{ "name", "Researcher One" },
|
||||
{ "contact", "r1@example.test" },
|
||||
{ "type", "finder" }
|
||||
}
|
||||
}
|
||||
},
|
||||
{ "weaknesses", new BsonArray
|
||||
{
|
||||
new BsonDocument
|
||||
{
|
||||
{ "cweId", "CWE-79" },
|
||||
{ "description", "XSS" }
|
||||
}
|
||||
}
|
||||
},
|
||||
{ "kev", new BsonArray
|
||||
{
|
||||
new BsonDocument
|
||||
{
|
||||
{ "cveId", "CVE-2024-0001" },
|
||||
{ "vendorProject", "Example" },
|
||||
{ "product", "Example Product" },
|
||||
{ "name", "Critical vuln" },
|
||||
{ "knownRansomwareUse", false },
|
||||
{ "dateAdded", DateTime.UtcNow },
|
||||
{ "dueDate", DateTime.UtcNow.AddDays(7) },
|
||||
{ "notes", "note" }
|
||||
}
|
||||
}
|
||||
}
|
||||
};
|
||||
|
||||
return new AdvisoryDocument
|
||||
{
|
||||
AdvisoryKey = "ADV-1",
|
||||
Payload = payload,
|
||||
Modified = DateTime.UtcNow,
|
||||
Published = DateTime.UtcNow.AddDays(-1)
|
||||
};
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,81 @@
|
||||
using FluentAssertions;
|
||||
using Microsoft.Extensions.Logging.Abstractions;
|
||||
using Microsoft.Extensions.Options;
|
||||
using MongoDB.Bson;
|
||||
using MongoDB.Driver;
|
||||
using MongoDB.Driver.Linq;
|
||||
using StellaOps.Concelier.Storage.Mongo.Advisories;
|
||||
using StellaOps.Concelier.Storage.Postgres;
|
||||
using StellaOps.Concelier.Storage.Postgres.Converters;
|
||||
using StellaOps.Concelier.Storage.Postgres.Converters.Importers;
|
||||
using StellaOps.Concelier.Storage.Postgres.Repositories;
|
||||
using Xunit;
|
||||
|
||||
namespace StellaOps.Concelier.Storage.Postgres.Tests;
|
||||
|
||||
[Collection(ConcelierPostgresCollection.Name)]
|
||||
public sealed class NvdImporterTests : IAsyncLifetime
|
||||
{
|
||||
private readonly ConcelierPostgresFixture _fixture;
|
||||
private readonly AdvisoryConversionService _conversionService;
|
||||
private readonly IAdvisoryRepository _advisories;
|
||||
private readonly IFeedSnapshotRepository _feedSnapshots;
|
||||
private readonly IAdvisorySnapshotRepository _advisorySnapshots;
|
||||
private readonly IMongoCollection<AdvisoryDocument> _mongo;
|
||||
|
||||
public NvdImporterTests(ConcelierPostgresFixture fixture)
|
||||
{
|
||||
_fixture = fixture;
|
||||
var options = fixture.Fixture.CreateOptions();
|
||||
options.SchemaName = fixture.SchemaName;
|
||||
var dataSource = new ConcelierDataSource(Options.Create(options), NullLogger<ConcelierDataSource>.Instance);
|
||||
|
||||
_advisories = new AdvisoryRepository(dataSource, NullLogger<AdvisoryRepository>.Instance);
|
||||
_feedSnapshots = new FeedSnapshotRepository(dataSource, NullLogger<FeedSnapshotRepository>.Instance);
|
||||
_advisorySnapshots = new AdvisorySnapshotRepository(dataSource, NullLogger<AdvisorySnapshotRepository>.Instance);
|
||||
_conversionService = new AdvisoryConversionService(_advisories);
|
||||
|
||||
// In-memory Mongo (Mock via Mongo2Go would be heavier; here use in-memory collection mock via MongoDB.Driver.Linq IMock).
|
||||
var client = new MongoClient("mongodb://localhost:27017");
|
||||
var db = client.GetDatabase("concelier_test");
|
||||
_mongo = db.GetCollection<AdvisoryDocument>("advisories");
|
||||
db.DropCollection("advisories");
|
||||
_mongo = db.GetCollection<AdvisoryDocument>("advisories");
|
||||
}
|
||||
|
||||
public Task InitializeAsync() => _fixture.TruncateAllTablesAsync();
|
||||
|
||||
public Task DisposeAsync()
|
||||
{
|
||||
// clean mongo collection
|
||||
_mongo.Database.DropCollection("advisories");
|
||||
return Task.CompletedTask;
|
||||
}
|
||||
|
||||
[Fact(Skip = "Requires local Mongo; placeholder for pipeline wiring test")] // To be enabled when Mongo fixture available
|
||||
public async Task ImportSnapshot_UpsertsAdvisoriesAndSnapshots()
|
||||
{
|
||||
var doc = new AdvisoryDocument
|
||||
{
|
||||
AdvisoryKey = "ADV-3",
|
||||
Payload = new BsonDocument
|
||||
{
|
||||
{ "primaryVulnId", "CVE-2024-0003" },
|
||||
{ "aliases", new BsonArray { "CVE-2024-0003" } },
|
||||
{ "affected", new BsonArray { new BsonDocument { { "ecosystem", "npm" }, { "packageName", "pkg" }, { "range", "{}" } } } }
|
||||
},
|
||||
Modified = DateTime.UtcNow,
|
||||
Published = DateTime.UtcNow.AddDays(-3)
|
||||
};
|
||||
await _mongo.InsertOneAsync(doc);
|
||||
|
||||
var importer = new NvdImporter(_mongo, _conversionService, _feedSnapshots, _advisorySnapshots);
|
||||
await importer.ImportSnapshotAsync(Guid.NewGuid(), "nvd", "snap-1", default);
|
||||
|
||||
var stored = await _advisories.GetByKeyAsync("ADV-3");
|
||||
stored.Should().NotBeNull();
|
||||
|
||||
var snapshots = await _advisorySnapshots.GetByFeedSnapshotAsync((await _feedSnapshots.GetBySourceAndIdAsync(stored!.SourceId!.Value, "snap-1"))!.Id);
|
||||
snapshots.Should().NotBeEmpty();
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,367 @@
|
||||
using FluentAssertions;
|
||||
using Microsoft.Extensions.Logging.Abstractions;
|
||||
using Microsoft.Extensions.Options;
|
||||
using StellaOps.Concelier.Storage.Postgres;
|
||||
using StellaOps.Concelier.Storage.Postgres.Models;
|
||||
using StellaOps.Concelier.Storage.Postgres.Repositories;
|
||||
using Xunit;
|
||||
|
||||
namespace StellaOps.Concelier.Storage.Postgres.Tests;
|
||||
|
||||
[Collection(ConcelierPostgresCollection.Name)]
|
||||
public sealed class RepositoryIntegrationTests : IAsyncLifetime
|
||||
{
|
||||
private readonly ConcelierPostgresFixture _fixture;
|
||||
private readonly ConcelierDataSource _dataSource;
|
||||
private readonly SourceRepository _sources;
|
||||
private readonly SourceStateRepository _sourceStates;
|
||||
private readonly FeedSnapshotRepository _feedSnapshots;
|
||||
private readonly AdvisorySnapshotRepository _advisorySnapshots;
|
||||
private readonly AdvisoryRepository _advisories;
|
||||
private readonly AdvisoryAliasRepository _aliases;
|
||||
private readonly AdvisoryCvssRepository _cvss;
|
||||
private readonly AdvisoryAffectedRepository _affected;
|
||||
private readonly AdvisoryReferenceRepository _references;
|
||||
private readonly AdvisoryCreditRepository _credits;
|
||||
private readonly AdvisoryWeaknessRepository _weaknesses;
|
||||
private readonly KevFlagRepository _kevFlags;
|
||||
private readonly MergeEventRepository _mergeEvents;
|
||||
|
||||
public RepositoryIntegrationTests(ConcelierPostgresFixture fixture)
|
||||
{
|
||||
_fixture = fixture;
|
||||
|
||||
var options = fixture.Fixture.CreateOptions();
|
||||
options.SchemaName = fixture.SchemaName;
|
||||
_dataSource = new ConcelierDataSource(Options.Create(options), NullLogger<ConcelierDataSource>.Instance);
|
||||
|
||||
_sources = new SourceRepository(_dataSource, NullLogger<SourceRepository>.Instance);
|
||||
_sourceStates = new SourceStateRepository(_dataSource, NullLogger<SourceStateRepository>.Instance);
|
||||
_feedSnapshots = new FeedSnapshotRepository(_dataSource, NullLogger<FeedSnapshotRepository>.Instance);
|
||||
_advisorySnapshots = new AdvisorySnapshotRepository(_dataSource, NullLogger<AdvisorySnapshotRepository>.Instance);
|
||||
_advisories = new AdvisoryRepository(_dataSource, NullLogger<AdvisoryRepository>.Instance);
|
||||
_aliases = new AdvisoryAliasRepository(_dataSource, NullLogger<AdvisoryAliasRepository>.Instance);
|
||||
_cvss = new AdvisoryCvssRepository(_dataSource, NullLogger<AdvisoryCvssRepository>.Instance);
|
||||
_affected = new AdvisoryAffectedRepository(_dataSource, NullLogger<AdvisoryAffectedRepository>.Instance);
|
||||
_references = new AdvisoryReferenceRepository(_dataSource, NullLogger<AdvisoryReferenceRepository>.Instance);
|
||||
_credits = new AdvisoryCreditRepository(_dataSource, NullLogger<AdvisoryCreditRepository>.Instance);
|
||||
_weaknesses = new AdvisoryWeaknessRepository(_dataSource, NullLogger<AdvisoryWeaknessRepository>.Instance);
|
||||
_kevFlags = new KevFlagRepository(_dataSource, NullLogger<KevFlagRepository>.Instance);
|
||||
_mergeEvents = new MergeEventRepository(_dataSource, NullLogger<MergeEventRepository>.Instance);
|
||||
}
|
||||
|
||||
public Task InitializeAsync() => _fixture.TruncateAllTablesAsync();
|
||||
public Task DisposeAsync() => Task.CompletedTask;
|
||||
|
||||
[Fact]
|
||||
public async Task SourceRepository_RoundTripsAndLists()
|
||||
{
|
||||
var source = CreateSource("osv", priority: 50);
|
||||
|
||||
var upserted = await _sources.UpsertAsync(source);
|
||||
var fetchedByKey = await _sources.GetByKeyAsync(source.Key);
|
||||
var enabled = await _sources.ListAsync(enabled: true);
|
||||
|
||||
upserted.Name.Should().Be("Open Source Vulns");
|
||||
fetchedByKey!.Id.Should().Be(source.Id);
|
||||
enabled.Should().ContainSingle(s => s.Key == source.Key);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task SourceStateRepository_Upsert_ReplacesState()
|
||||
{
|
||||
var source = await _sources.UpsertAsync(CreateSource("ghsa"));
|
||||
var initial = CreateState(source.Id, syncCount: 1, errorCount: 0, cursor: "c1");
|
||||
await _sourceStates.UpsertAsync(initial);
|
||||
|
||||
var updated = await _sourceStates.UpsertAsync(CreateState(
|
||||
source.Id,
|
||||
syncCount: 5,
|
||||
errorCount: 1,
|
||||
cursor: "c2",
|
||||
lastError: "timeout"));
|
||||
|
||||
updated.Cursor.Should().Be("c2");
|
||||
updated.SyncCount.Should().Be(5);
|
||||
updated.ErrorCount.Should().Be(1);
|
||||
updated.LastError.Should().Be("timeout");
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task FeedAndAdvisorySnapshots_RoundTrip()
|
||||
{
|
||||
var source = await _sources.UpsertAsync(CreateSource("nvd"));
|
||||
var feed = new FeedSnapshotEntity
|
||||
{
|
||||
Id = Guid.NewGuid(),
|
||||
SourceId = source.Id,
|
||||
SnapshotId = "2024-12-01",
|
||||
AdvisoryCount = 123,
|
||||
Checksum = "sha256:deadbeef",
|
||||
Metadata = "{\"import\":\"full\"}",
|
||||
CreatedAt = DateTimeOffset.UtcNow
|
||||
};
|
||||
|
||||
var insertedFeed = await _feedSnapshots.InsertAsync(feed);
|
||||
var fetchedFeed = await _feedSnapshots.GetBySourceAndIdAsync(source.Id, feed.SnapshotId);
|
||||
|
||||
var advisorySnapshot = new AdvisorySnapshotEntity
|
||||
{
|
||||
Id = Guid.NewGuid(),
|
||||
FeedSnapshotId = insertedFeed.Id,
|
||||
AdvisoryKey = "GHSA-1234-5678-90ab",
|
||||
ContentHash = "content-hash",
|
||||
CreatedAt = DateTimeOffset.UtcNow
|
||||
};
|
||||
|
||||
await _advisorySnapshots.InsertAsync(advisorySnapshot);
|
||||
var snapshots = await _advisorySnapshots.GetByFeedSnapshotAsync(insertedFeed.Id);
|
||||
|
||||
fetchedFeed!.Checksum.Should().Be("sha256:deadbeef");
|
||||
snapshots.Should().ContainSingle(s => s.AdvisoryKey == advisorySnapshot.AdvisoryKey);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task AdvisoryRepository_Upsert_WithChildren_ReplacesAndQueries()
|
||||
{
|
||||
var source = await _sources.UpsertAsync(CreateSource("vendor-feed"));
|
||||
var advisory = CreateAdvisory(source.Id, "adv-key-1", "CVE-2024-9999");
|
||||
var children = CreateChildren(advisory.Id);
|
||||
|
||||
await _advisories.UpsertAsync(
|
||||
advisory,
|
||||
children.aliases,
|
||||
children.cvss,
|
||||
children.affected,
|
||||
children.references,
|
||||
children.credits,
|
||||
children.weaknesses,
|
||||
children.kevFlags);
|
||||
|
||||
// Replace aliases to assert deletion/replacement semantics
|
||||
var replacementAliases = new[]
|
||||
{
|
||||
new AdvisoryAliasEntity
|
||||
{
|
||||
Id = Guid.NewGuid(),
|
||||
AdvisoryId = advisory.Id,
|
||||
AliasType = "GHSA",
|
||||
AliasValue = "GHSA-aaaa-bbbb-cccc",
|
||||
IsPrimary = false,
|
||||
CreatedAt = DateTimeOffset.UtcNow
|
||||
}
|
||||
};
|
||||
|
||||
await _advisories.UpsertAsync(advisory, replacementAliases, null, null, null, null, null, null);
|
||||
|
||||
var fetched = await _advisories.GetByKeyAsync(advisory.AdvisoryKey);
|
||||
var aliases = await _aliases.GetByAdvisoryAsync(advisory.Id);
|
||||
var cvss = await _cvss.GetByAdvisoryAsync(advisory.Id);
|
||||
var affected = await _affected.GetByAdvisoryAsync(advisory.Id);
|
||||
var references = await _references.GetByAdvisoryAsync(advisory.Id);
|
||||
var credits = await _credits.GetByAdvisoryAsync(advisory.Id);
|
||||
var weaknesses = await _weaknesses.GetByAdvisoryAsync(advisory.Id);
|
||||
var kevFlags = await _kevFlags.GetByAdvisoryAsync(advisory.Id);
|
||||
var countBySeverity = await _advisories.CountBySeverityAsync();
|
||||
|
||||
fetched!.PrimaryVulnId.Should().Be("CVE-2024-9999");
|
||||
aliases.Should().HaveCount(1).And.ContainSingle(a => a.AliasType == "GHSA");
|
||||
cvss.Should().ContainSingle(c => c.BaseScore == 9.8m);
|
||||
affected.Should().ContainSingle(a => a.Purl == "pkg:npm/example@1.0.0");
|
||||
references.Should().ContainSingle(r => r.Url == "https://advisories.example/ref");
|
||||
credits.Should().ContainSingle(c => c.Name == "Researcher Zero");
|
||||
weaknesses.Should().ContainSingle(w => w.CweId == "CWE-79");
|
||||
kevFlags.Should().ContainSingle(k => k.CveId == "CVE-2024-9999");
|
||||
countBySeverity["high"].Should().BeGreaterOrEqualTo(1);
|
||||
|
||||
var purlMatches = await _advisories.GetAffectingPackageAsync("pkg:npm/example@1.0.0");
|
||||
var packageMatches = await _advisories.GetAffectingPackageNameAsync("npm", "example");
|
||||
var searchResults = await _advisories.SearchAsync("Test advisory");
|
||||
|
||||
purlMatches.Should().NotBeEmpty();
|
||||
packageMatches.Should().NotBeEmpty();
|
||||
searchResults.Should().NotBeEmpty();
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task MergeEvents_InsertAndFetch()
|
||||
{
|
||||
var source = await _sources.UpsertAsync(CreateSource("merge-feed"));
|
||||
var advisory = await _advisories.UpsertAsync(CreateAdvisory(source.Id, "merge-adv", "CVE-2024-5555"));
|
||||
|
||||
var evt = new MergeEventEntity
|
||||
{
|
||||
AdvisoryId = advisory.Id,
|
||||
SourceId = source.Id,
|
||||
EventType = "merge",
|
||||
OldValue = "{\"title\":\"old\"}",
|
||||
NewValue = "{\"title\":\"new\"}",
|
||||
CreatedAt = DateTimeOffset.UtcNow
|
||||
};
|
||||
|
||||
await _mergeEvents.InsertAsync(evt);
|
||||
var events = await _mergeEvents.GetByAdvisoryAsync(advisory.Id);
|
||||
|
||||
events.Should().ContainSingle(e => e.EventType == "merge");
|
||||
}
|
||||
|
||||
private static SourceEntity CreateSource(string key, int priority = 10) => new()
|
||||
{
|
||||
Id = Guid.NewGuid(),
|
||||
Key = key,
|
||||
Name = key switch { "osv" => "Open Source Vulns", _ => $"Source {key}" },
|
||||
SourceType = key,
|
||||
Url = "https://example.test",
|
||||
Priority = priority,
|
||||
Enabled = true,
|
||||
Config = "{}",
|
||||
Metadata = "{}",
|
||||
CreatedAt = DateTimeOffset.UtcNow,
|
||||
UpdatedAt = DateTimeOffset.UtcNow
|
||||
};
|
||||
|
||||
private static SourceStateEntity CreateState(Guid sourceId, long syncCount, int errorCount, string cursor, string? lastError = null) => new()
|
||||
{
|
||||
Id = Guid.NewGuid(),
|
||||
SourceId = sourceId,
|
||||
Cursor = cursor,
|
||||
LastSyncAt = DateTimeOffset.UtcNow,
|
||||
LastSuccessAt = DateTimeOffset.UtcNow,
|
||||
LastError = lastError,
|
||||
SyncCount = syncCount,
|
||||
ErrorCount = errorCount,
|
||||
Metadata = "{\"state\":\"ok\"}",
|
||||
UpdatedAt = DateTimeOffset.UtcNow
|
||||
};
|
||||
|
||||
private static AdvisoryEntity CreateAdvisory(Guid? sourceId, string advisoryKey, string vulnId) => new()
|
||||
{
|
||||
Id = Guid.NewGuid(),
|
||||
AdvisoryKey = advisoryKey,
|
||||
PrimaryVulnId = vulnId,
|
||||
SourceId = sourceId,
|
||||
Title = "Test advisory",
|
||||
Summary = "Sample summary",
|
||||
Description = "Full description",
|
||||
Severity = "high",
|
||||
PublishedAt = DateTimeOffset.UtcNow.AddDays(-2),
|
||||
ModifiedAt = DateTimeOffset.UtcNow.AddDays(-1),
|
||||
WithdrawnAt = null,
|
||||
Provenance = "{\"source\":\"unit\"}",
|
||||
RawPayload = "{\"raw\":\"payload\"}",
|
||||
CreatedAt = DateTimeOffset.UtcNow,
|
||||
UpdatedAt = DateTimeOffset.UtcNow
|
||||
};
|
||||
|
||||
private static (
|
||||
AdvisoryAliasEntity[] aliases,
|
||||
AdvisoryCvssEntity[] cvss,
|
||||
AdvisoryAffectedEntity[] affected,
|
||||
AdvisoryReferenceEntity[] references,
|
||||
AdvisoryCreditEntity[] credits,
|
||||
AdvisoryWeaknessEntity[] weaknesses,
|
||||
KevFlagEntity[] kevFlags) CreateChildren(Guid advisoryId)
|
||||
{
|
||||
var now = DateTimeOffset.UtcNow;
|
||||
var today = DateOnly.FromDateTime(now.DateTime);
|
||||
return (
|
||||
aliases: new[]
|
||||
{
|
||||
new AdvisoryAliasEntity
|
||||
{
|
||||
Id = Guid.NewGuid(),
|
||||
AdvisoryId = advisoryId,
|
||||
AliasType = "CVE",
|
||||
AliasValue = "CVE-2024-9999",
|
||||
IsPrimary = true,
|
||||
CreatedAt = now
|
||||
}
|
||||
},
|
||||
cvss: new[]
|
||||
{
|
||||
new AdvisoryCvssEntity
|
||||
{
|
||||
Id = Guid.NewGuid(),
|
||||
AdvisoryId = advisoryId,
|
||||
CvssVersion = "3.1",
|
||||
VectorString = "CVSS:3.1/AV:N/AC:L/PR:N/UI:N/S:U/C:H/I:H/A:H",
|
||||
BaseScore = 9.8m,
|
||||
BaseSeverity = "critical",
|
||||
ExploitabilityScore = 3.9m,
|
||||
ImpactScore = 5.9m,
|
||||
Source = "nvd",
|
||||
IsPrimary = true,
|
||||
CreatedAt = now
|
||||
}
|
||||
},
|
||||
affected: new[]
|
||||
{
|
||||
new AdvisoryAffectedEntity
|
||||
{
|
||||
Id = Guid.NewGuid(),
|
||||
AdvisoryId = advisoryId,
|
||||
Ecosystem = "npm",
|
||||
PackageName = "example",
|
||||
Purl = "pkg:npm/example@1.0.0",
|
||||
VersionRange = "{\"introduced\":\"0\",\"fixed\":\"1.0.1\"}",
|
||||
VersionsAffected = new[] { "1.0.0" },
|
||||
VersionsFixed = new[] { "1.0.1" },
|
||||
DatabaseSpecific = "{\"severity\":\"high\"}",
|
||||
CreatedAt = now
|
||||
}
|
||||
},
|
||||
references: new[]
|
||||
{
|
||||
new AdvisoryReferenceEntity
|
||||
{
|
||||
Id = Guid.NewGuid(),
|
||||
AdvisoryId = advisoryId,
|
||||
RefType = "advisory",
|
||||
Url = "https://advisories.example/ref",
|
||||
CreatedAt = now
|
||||
}
|
||||
},
|
||||
credits: new[]
|
||||
{
|
||||
new AdvisoryCreditEntity
|
||||
{
|
||||
Id = Guid.NewGuid(),
|
||||
AdvisoryId = advisoryId,
|
||||
Name = "Researcher Zero",
|
||||
Contact = "r0@example.test",
|
||||
CreditType = "finder",
|
||||
CreatedAt = now
|
||||
}
|
||||
},
|
||||
weaknesses: new[]
|
||||
{
|
||||
new AdvisoryWeaknessEntity
|
||||
{
|
||||
Id = Guid.NewGuid(),
|
||||
AdvisoryId = advisoryId,
|
||||
CweId = "CWE-79",
|
||||
Description = "XSS",
|
||||
Source = "internal",
|
||||
CreatedAt = now
|
||||
}
|
||||
},
|
||||
kevFlags: new[]
|
||||
{
|
||||
new KevFlagEntity
|
||||
{
|
||||
Id = Guid.NewGuid(),
|
||||
AdvisoryId = advisoryId,
|
||||
CveId = "CVE-2024-9999",
|
||||
VendorProject = "Example",
|
||||
Product = "Example Product",
|
||||
VulnerabilityName = "Critical vuln",
|
||||
DateAdded = today,
|
||||
DueDate = today.AddDays(7),
|
||||
KnownRansomwareUse = false,
|
||||
Notes = "test note",
|
||||
CreatedAt = now
|
||||
}
|
||||
}
|
||||
);
|
||||
}
|
||||
}
|
||||
@@ -12,7 +12,7 @@
|
||||
|
||||
<ItemGroup>
|
||||
<PackageReference Include="FluentAssertions" Version="6.12.0" />
|
||||
<PackageReference Include="Microsoft.NET.Test.Sdk" Version="17.11.1" />
|
||||
<PackageReference Include="Microsoft.NET.Test.Sdk" Version="17.14.0" />
|
||||
<PackageReference Include="Moq" Version="4.20.70" />
|
||||
<PackageReference Include="xunit" Version="2.9.2" />
|
||||
<PackageReference Include="xunit.runner.visualstudio" Version="2.8.2">
|
||||
@@ -28,6 +28,7 @@
|
||||
<ItemGroup>
|
||||
<ProjectReference Include="..\..\__Libraries\StellaOps.Concelier.Storage.Postgres\StellaOps.Concelier.Storage.Postgres.csproj" />
|
||||
<ProjectReference Include="..\..\..\__Libraries\StellaOps.Infrastructure.Postgres.Testing\StellaOps.Infrastructure.Postgres.Testing.csproj" />
|
||||
<ProjectReference Include="..\..\__Libraries\StellaOps.Concelier.Storage.Mongo\StellaOps.Concelier.Storage.Mongo.csproj" />
|
||||
</ItemGroup>
|
||||
|
||||
</Project>
|
||||
|
||||
Reference in New Issue
Block a user