Align AOC tasks for Excititor and Concelier
This commit is contained in:
@@ -14,8 +14,8 @@ Document follow-up actions for CONCELIER-CORE-AOC-19-004 as we unwind the final
|
||||
|
||||
## Next Steps
|
||||
|
||||
1. Introduce a raw linkset projection alongside the existing canonical mapper so Policy Engine can choose which flavour to consume.
|
||||
2. Update observation factory/query tests to assert duplicate handling and ordering with the relaxed projection.
|
||||
1. Introduce a raw linkset projection alongside the existing canonical mapper so Policy Engine can choose which flavour to consume. ✅ 2025-10-31: `AdvisoryObservation` now surfaces `RawLinkset`; Mongo documents store both canonical & raw shapes; tests/goldens updated.
|
||||
2. Update observation factory/query tests to assert duplicate handling and ordering with the relaxed projection. ✅ 2025-10-31.
|
||||
3. Refresh docs (`docs/ingestion/aggregation-only-contract.md`) once behaviour lands to explain the “raw vs canonical linkset” split.
|
||||
4. Coordinate with Policy Guild to validate consumers against the new raw projection before flipping defaults.
|
||||
4. Coordinate with Policy Guild to validate consumers against the new raw projection before flipping defaults. ↺ Ongoing — see action items in `docs/dev/raw-linkset-backfill-plan.md` (2025-10-31 handshake with POLICY-ENGINE-20-003 owners).
|
||||
|
||||
|
||||
56
docs/dev/raw-linkset-backfill-plan.md
Normal file
56
docs/dev/raw-linkset-backfill-plan.md
Normal file
@@ -0,0 +1,56 @@
|
||||
# Raw Linkset Backfill & Adoption Plan
|
||||
|
||||
_Last updated: 2025-10-31_
|
||||
Owners: Concelier Storage Guild, DevOps Guild, Policy Guild
|
||||
|
||||
## Context
|
||||
|
||||
- Concelier observations now emit both a **canonical linkset** (deduped, normalised identifiers) and a **raw linkset** (`rawLinkset`) that preserves upstream ordering, duplicates, and original pointer metadata.
|
||||
- Existing `concelier.advisory_observations` documents created before 2025-10-31 do **not** contain the `rawLinkset` field.
|
||||
- Policy Engine selection joiners (`POLICY-ENGINE-20-003`) will switch to the raw projection once backfill completes and consumers validate fixtures.
|
||||
|
||||
## Objectives
|
||||
|
||||
1. Populate `rawLinkset` for historical observations across online clusters and Offline Kit bundles without breaking append-only guarantees.
|
||||
2. Provide migration scripts + runbook so operators can rehearse in staging (and air-gapped deployments) before production rollout.
|
||||
3. Unblock Policy Engine adoption by guaranteeing dual projections exist for all tenants.
|
||||
|
||||
## Deliverables
|
||||
|
||||
- [ ] **Migration script** (`20251104_advisory_observations_raw_linkset_backfill.csx`)
|
||||
- Iterates observations lacking `rawLinkset`
|
||||
- Rehydrates raw document via existing snapshot (or cached DTO)
|
||||
- Reuses `AdvisoryObservationFactory.CreateRawLinkset`
|
||||
- Writes using `$set` with optimistic retry; preserves `updatedAt` via `setOnInsert`
|
||||
- [ ] **Offline Kit updater** (extend `ops/offline-kit/scripts/export_offline_bundle.py`) to patch bundles in-place
|
||||
- [ ] **Runbook** covering:
|
||||
- Pre-check query: `db.concelier.advisory_observations.countDocuments({ rawLinkset: { $exists: false } })`
|
||||
- Backup procedure (`mongodump` or snapshot requirement)
|
||||
- Dry-run mode limiting batches by tenant
|
||||
- Metrics/telemetry expectations (`concelier.migrations.documents_processed_total`)
|
||||
- Rollback (no-op because field addition; note to retain snapshot for verification)
|
||||
- [ ] **Fixture updates** ensuring storage/CLI/Policy tests include `rawLinkset`
|
||||
- [ ] **Policy Engine follow-up** to flip joiners once `rawLinkset` population reaches 100% (tracked via metrics).
|
||||
|
||||
## Timeline
|
||||
|
||||
| Date (UTC) | Milestone | Notes |
|
||||
|------------|-----------|-------|
|
||||
| 2025-10-31 | Handshake w/ Policy | Agreement to consume `rawLinkset`; this document created. |
|
||||
| 2025-11-01 | Draft migration script | Validate against staging dataset snapshots. |
|
||||
| 2025-11-04 | Storage task CONCELIER-STORE-AOC-19-005 due | Deliver script + runbook for review. |
|
||||
| 2025-11-06 | Staging backfill rehearsal | Target < 30 min runtime on 5M observations. |
|
||||
| 2025-11-08 | Policy fixtures updated | POL engine branch consumes `rawLinkset`. |
|
||||
| 2025-11-11 | Production rollout window | Pending DevOps sign-off after rehearsals. |
|
||||
|
||||
## Open Questions
|
||||
|
||||
- Do we need archival of the canonical-only projection for backwards compatibility exports? (Policy to confirm.)
|
||||
- Offline Kit delta: should we regenerate entire bundle or ship incremental patch? (DevOps reviewing.)
|
||||
- Metrics: add `raw_linkset_missing_total` counter to detect regressions post-backfill?
|
||||
|
||||
## Next Actions
|
||||
|
||||
- [ ] Concelier Storage Guild: prototype migration script, share for review (`2025-11-01`).
|
||||
- [ ] DevOps Guild: schedule staging rehearsal + update `docs/deploy/containers.md` with new runbook section.
|
||||
- [ ] Policy Guild: prepare feature flag/branch to switch joiners once metrics show zero missing `rawLinkset`.
|
||||
@@ -16,7 +16,7 @@ This file describe implementation of Stella Ops (docs/README.md). Implementation
|
||||
| Sprint 19 | Aggregation-Only Contract Enforcement | src/Concelier/__Libraries/StellaOps.Concelier.Core/TASKS.md | TODO | Concelier Core Guild | CONCELIER-CORE-AOC-19-001 | Implement AOC repository guard rejecting forbidden fields. |
|
||||
| Sprint 19 | Aggregation-Only Contract Enforcement | src/Concelier/__Libraries/StellaOps.Concelier.Core/TASKS.md | TODO | Concelier Core Guild | CONCELIER-CORE-AOC-19-002 | Deliver deterministic linkset extraction for advisories. |
|
||||
| Sprint 19 | Aggregation-Only Contract Enforcement | src/Concelier/__Libraries/StellaOps.Concelier.Core/TASKS.md | TODO | Concelier Core Guild | CONCELIER-CORE-AOC-19-003 | Enforce idempotent append-only upsert with supersedes pointers. |
|
||||
| Sprint 19 | Aggregation-Only Contract Enforcement | src/Concelier/__Libraries/StellaOps.Concelier.Core/TASKS.md | TODO | Concelier Core Guild | CONCELIER-CORE-AOC-19-004 | Remove ingestion normalization; defer derived logic to Policy Engine. |
|
||||
| Sprint 19 | Aggregation-Only Contract Enforcement | src/Concelier/__Libraries/StellaOps.Concelier.Core/TASKS.md | DOING (2025-10-28) | Concelier Core Guild | CONCELIER-CORE-AOC-19-004 | Remove ingestion normalization; defer derived logic to Policy Engine. |
|
||||
| Sprint 19 | Aggregation-Only Contract Enforcement | src/Concelier/__Libraries/StellaOps.Concelier.Core/TASKS.md | TODO | Concelier Core Guild | CONCELIER-CORE-AOC-19-013 | Extend smoke coverage to validate tenant-scoped Authority tokens and cross-tenant rejection. |
|
||||
| Sprint 19 | Aggregation-Only Contract Enforcement | src/Concelier/__Libraries/StellaOps.Concelier.Storage.Mongo/TASKS.md | TODO | Concelier Storage Guild | CONCELIER-STORE-AOC-19-001 | Add Mongo schema validator for `advisory_raw`. |
|
||||
| Sprint 19 | Aggregation-Only Contract Enforcement | src/Concelier/__Libraries/StellaOps.Concelier.Storage.Mongo/TASKS.md | TODO | Concelier Storage Guild | CONCELIER-STORE-AOC-19-002 | Create idempotency unique index backed by migration scripts. |
|
||||
|
||||
@@ -25,7 +25,7 @@
|
||||
| 3 | Provenance is mandatory | `source.*`, `upstream.*`, and `signature` metadata must be present; missing provenance triggers `ERR_AOC_004`. | Schema validator, guard, CLI verifier. |
|
||||
| 4 | Idempotent upserts | Writes keyed by `(vendor, upstream_id, content_hash)` either no-op or insert a new revision with `supersedes`. Duplicate hashes map to the same document. | Repository guard, storage unique index, CI smoke tests. |
|
||||
| 5 | Append-only revisions | Updates create a new document with `supersedes` pointer; no in-place mutation of content. | Mongo schema (`supersedes` format), guard, data migration scripts. |
|
||||
| 6 | Linkset only | Ingestion may compute link hints (`purls`, `cpes`, IDs) to accelerate joins, but must not transform or infer severity or policy. | Linkset builders reviewed via fixtures and analyzers. |
|
||||
| 6 | Linkset only | Ingestion may compute link hints (`purls`, `cpes`, IDs) to accelerate joins, but must not transform or infer severity or policy. Observations now persist both canonical linksets (for indexed queries) and raw linksets (preserving upstream order/duplicates) so downstream policy can decide how to normalise. | Linkset builders reviewed via fixtures/analyzers; raw-vs-canonical parity covered by observation fixtures. |
|
||||
| 7 | Policy-only effective findings | Only Policy Engine identities can write `effective_finding_*`; ingestion callers receive `ERR_AOC_006` if they attempt it. | Authority scopes, Policy Engine guard. |
|
||||
| 8 | Schema safety | Unknown top-level keys reject with `ERR_AOC_007`; timestamps use ISO 8601 UTC strings; tenant is required. | Mongo validator, JSON schema tests. |
|
||||
| 9 | Clock discipline | Collectors stamp `fetched_at` and `received_at` monotonically per batch to support reproducibility windows. | Collector contracts, QA fixtures. |
|
||||
|
||||
@@ -335,6 +335,7 @@ Events are emitted via NATS (primary) and Redis Stream (fallback). Consumers ack
|
||||
content: { format, specVersion, raw, metadata? },
|
||||
identifiers: { cve?, ghsa?, vendorIds[], aliases[] },
|
||||
linkset: { purls[], cpes[], aliases[], references[], reconciledFrom[] },
|
||||
rawLinkset: { aliases[], purls[], cpes[], references[], reconciledFrom[], notes? },
|
||||
supersedes?: "prevObservationId",
|
||||
createdAt,
|
||||
attributes?: object
|
||||
|
||||
@@ -22,6 +22,7 @@ internal sealed class AdvisoryObservationFactory : IAdvisoryObservationFactory
|
||||
var upstream = CreateUpstream(rawDocument.Upstream);
|
||||
var content = CreateContent(rawDocument.Content);
|
||||
var linkset = CreateLinkset(rawDocument.Identifiers, rawDocument.Linkset);
|
||||
var rawLinkset = CreateRawLinkset(rawDocument.Identifiers, rawDocument.Linkset);
|
||||
var attributes = CreateAttributes(rawDocument);
|
||||
|
||||
var createdAt = (observedAt ?? rawDocument.Upstream.RetrievedAt).ToUniversalTime();
|
||||
@@ -33,6 +34,7 @@ internal sealed class AdvisoryObservationFactory : IAdvisoryObservationFactory
|
||||
upstream: upstream,
|
||||
content: content,
|
||||
linkset: linkset,
|
||||
rawLinkset: rawLinkset,
|
||||
createdAt: createdAt,
|
||||
attributes: attributes);
|
||||
}
|
||||
@@ -120,6 +122,54 @@ internal sealed class AdvisoryObservationFactory : IAdvisoryObservationFactory
|
||||
return new AdvisoryObservationLinkset(aliases, purls, cpes, references);
|
||||
}
|
||||
|
||||
private static RawLinkset CreateRawLinkset(RawIdentifiers identifiers, RawLinkset linkset)
|
||||
{
|
||||
var aliasBuilder = ImmutableArray.CreateBuilder<string>();
|
||||
|
||||
if (!string.IsNullOrWhiteSpace(identifiers.PrimaryId))
|
||||
{
|
||||
aliasBuilder.Add(identifiers.PrimaryId);
|
||||
}
|
||||
|
||||
if (!identifiers.Aliases.IsDefaultOrEmpty)
|
||||
{
|
||||
foreach (var alias in identifiers.Aliases)
|
||||
{
|
||||
if (!string.IsNullOrEmpty(alias))
|
||||
{
|
||||
aliasBuilder.Add(alias);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
if (!linkset.Aliases.IsDefaultOrEmpty)
|
||||
{
|
||||
foreach (var alias in linkset.Aliases)
|
||||
{
|
||||
if (!string.IsNullOrEmpty(alias))
|
||||
{
|
||||
aliasBuilder.Add(alias);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
static ImmutableArray<string> EnsureArray(ImmutableArray<string> values)
|
||||
=> values.IsDefault ? ImmutableArray<string>.Empty : values;
|
||||
|
||||
static ImmutableArray<RawReference> EnsureReferences(ImmutableArray<RawReference> values)
|
||||
=> values.IsDefault ? ImmutableArray<RawReference>.Empty : values;
|
||||
|
||||
return linkset with
|
||||
{
|
||||
Aliases = aliasBuilder.ToImmutable(),
|
||||
PackageUrls = EnsureArray(linkset.PackageUrls),
|
||||
Cpes = EnsureArray(linkset.Cpes),
|
||||
References = EnsureReferences(linkset.References),
|
||||
ReconciledFrom = EnsureArray(linkset.ReconciledFrom),
|
||||
Notes = linkset.Notes ?? ImmutableDictionary<string, string>.Empty
|
||||
};
|
||||
}
|
||||
|
||||
private static IEnumerable<string> NormalizeAliases(RawIdentifiers identifiers, RawLinkset linkset)
|
||||
{
|
||||
var aliases = new HashSet<string>(StringComparer.OrdinalIgnoreCase);
|
||||
|
||||
@@ -1,6 +1,7 @@
|
||||
using System.Collections.Immutable;
|
||||
using System.Text.Json;
|
||||
using System.Text.Json.Nodes;
|
||||
using StellaOps.Concelier.RawModels;
|
||||
|
||||
namespace StellaOps.Concelier.Models.Observations;
|
||||
|
||||
@@ -13,6 +14,7 @@ public sealed record AdvisoryObservation
|
||||
AdvisoryObservationUpstream upstream,
|
||||
AdvisoryObservationContent content,
|
||||
AdvisoryObservationLinkset linkset,
|
||||
RawLinkset rawLinkset,
|
||||
DateTimeOffset createdAt,
|
||||
ImmutableDictionary<string, string>? attributes = null)
|
||||
{
|
||||
@@ -22,6 +24,7 @@ public sealed record AdvisoryObservation
|
||||
Upstream = upstream ?? throw new ArgumentNullException(nameof(upstream));
|
||||
Content = content ?? throw new ArgumentNullException(nameof(content));
|
||||
Linkset = linkset ?? throw new ArgumentNullException(nameof(linkset));
|
||||
RawLinkset = SanitizeRawLinkset(rawLinkset);
|
||||
CreatedAt = createdAt.ToUniversalTime();
|
||||
Attributes = NormalizeAttributes(attributes);
|
||||
}
|
||||
@@ -38,6 +41,8 @@ public sealed record AdvisoryObservation
|
||||
|
||||
public AdvisoryObservationLinkset Linkset { get; }
|
||||
|
||||
public RawLinkset RawLinkset { get; }
|
||||
|
||||
public DateTimeOffset CreatedAt { get; }
|
||||
|
||||
public ImmutableDictionary<string, string> Attributes { get; }
|
||||
@@ -62,6 +67,54 @@ public sealed record AdvisoryObservation
|
||||
|
||||
return builder.ToImmutable();
|
||||
}
|
||||
|
||||
private static RawLinkset SanitizeRawLinkset(RawLinkset? rawLinkset)
|
||||
{
|
||||
if (rawLinkset is null)
|
||||
{
|
||||
return new RawLinkset();
|
||||
}
|
||||
|
||||
static ImmutableArray<string> SanitizeStrings(ImmutableArray<string> values)
|
||||
{
|
||||
if (values.IsDefault)
|
||||
{
|
||||
return ImmutableArray<string>.Empty;
|
||||
}
|
||||
|
||||
return values;
|
||||
}
|
||||
|
||||
static ImmutableArray<RawReference> SanitizeReferences(ImmutableArray<RawReference> references)
|
||||
{
|
||||
if (references.IsDefault)
|
||||
{
|
||||
return ImmutableArray<RawReference>.Empty;
|
||||
}
|
||||
|
||||
return references;
|
||||
}
|
||||
|
||||
static ImmutableDictionary<string, string> SanitizeNotes(ImmutableDictionary<string, string>? notes)
|
||||
{
|
||||
if (notes is null || notes.Count == 0)
|
||||
{
|
||||
return ImmutableDictionary<string, string>.Empty;
|
||||
}
|
||||
|
||||
return notes;
|
||||
}
|
||||
|
||||
return rawLinkset with
|
||||
{
|
||||
Aliases = SanitizeStrings(rawLinkset.Aliases),
|
||||
PackageUrls = SanitizeStrings(rawLinkset.PackageUrls),
|
||||
Cpes = SanitizeStrings(rawLinkset.Cpes),
|
||||
References = SanitizeReferences(rawLinkset.References),
|
||||
ReconciledFrom = SanitizeStrings(rawLinkset.ReconciledFrom),
|
||||
Notes = SanitizeNotes(rawLinkset.Notes)
|
||||
};
|
||||
}
|
||||
}
|
||||
|
||||
public sealed record AdvisoryObservationSource
|
||||
|
||||
@@ -9,4 +9,7 @@
|
||||
<ItemGroup>
|
||||
<PackageReference Include="Microsoft.Extensions.Logging.Abstractions" Version="10.0.0-rc.2.25502.107" />
|
||||
</ItemGroup>
|
||||
<ItemGroup>
|
||||
<ProjectReference Include="..\StellaOps.Concelier.RawModels\StellaOps.Concelier.RawModels.csproj" />
|
||||
</ItemGroup>
|
||||
</Project>
|
||||
|
||||
@@ -26,6 +26,7 @@ This module owns the persistent shape of Concelier's MongoDB database. Upgrades
|
||||
| `20251028_advisory_raw_idempotency_index` | Applies compound unique index on `(source.vendor, upstream.upstream_id, upstream.content_hash, tenant)` after verifying no duplicates exist. |
|
||||
| `20251028_advisory_supersedes_backfill` | Renames legacy `advisory` collection to a read-only backup view and backfills `supersedes` chains across `advisory_raw`. |
|
||||
| `20251028_advisory_raw_validator` | Applies Aggregation-Only Contract JSON schema validator to the `advisory_raw` collection with configurable enforcement level. |
|
||||
| `20251104_advisory_observations_raw_linkset` | Backfills `rawLinkset` on `advisory_observations` using stored `advisory_raw` documents so canonical and raw projections co-exist for downstream policy joins. |
|
||||
|
||||
## Operator Runbook
|
||||
|
||||
|
||||
@@ -0,0 +1,442 @@
|
||||
using System;
|
||||
using System.Collections.Generic;
|
||||
using System.Collections.Immutable;
|
||||
using System.Globalization;
|
||||
using System.Linq;
|
||||
using System.Text.Json;
|
||||
using System.Threading;
|
||||
using System.Threading.Tasks;
|
||||
using MongoDB.Bson;
|
||||
using MongoDB.Bson.IO;
|
||||
using MongoDB.Driver;
|
||||
using StellaOps.Concelier.RawModels;
|
||||
|
||||
namespace StellaOps.Concelier.Storage.Mongo.Migrations;
|
||||
|
||||
/// <summary>
|
||||
/// Backfills the raw linkset projection on advisory observations so downstream services
|
||||
/// can rely on both canonical and raw linkset shapes.
|
||||
/// </summary>
|
||||
public sealed class EnsureAdvisoryObservationsRawLinksetMigration : IMongoMigration
|
||||
{
|
||||
private const string MigrationId = "20251104_advisory_observations_raw_linkset";
|
||||
private const int BulkBatchSize = 500;
|
||||
|
||||
public string Id => MigrationId;
|
||||
|
||||
public string Description => "Populate rawLinkset field for advisory observations using stored advisory_raw documents.";
|
||||
|
||||
public async Task ApplyAsync(IMongoDatabase database, CancellationToken cancellationToken)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(database);
|
||||
|
||||
var observations = database.GetCollection<BsonDocument>(MongoStorageDefaults.Collections.AdvisoryObservations);
|
||||
var rawCollection = database.GetCollection<BsonDocument>(MongoStorageDefaults.Collections.AdvisoryRaw);
|
||||
|
||||
var filter = Builders<BsonDocument>.Filter.Exists("rawLinkset", false) |
|
||||
Builders<BsonDocument>.Filter.Type("rawLinkset", BsonType.Null);
|
||||
|
||||
using var cursor = await observations
|
||||
.Find(filter)
|
||||
.ToCursorAsync(cancellationToken)
|
||||
.ConfigureAwait(false);
|
||||
|
||||
var updates = new List<WriteModel<BsonDocument>>(BulkBatchSize);
|
||||
var missingRawDocuments = new List<string>();
|
||||
|
||||
while (await cursor.MoveNextAsync(cancellationToken).ConfigureAwait(false))
|
||||
{
|
||||
foreach (var observationDocument in cursor.Current)
|
||||
{
|
||||
cancellationToken.ThrowIfCancellationRequested();
|
||||
|
||||
if (!TryExtractObservationKey(observationDocument, out var key))
|
||||
{
|
||||
continue;
|
||||
}
|
||||
|
||||
var rawFilter = Builders<BsonDocument>.Filter.Eq("tenant", key.Tenant) &
|
||||
Builders<BsonDocument>.Filter.Eq("source.vendor", key.Vendor) &
|
||||
Builders<BsonDocument>.Filter.Eq("upstream.upstream_id", key.UpstreamId) &
|
||||
Builders<BsonDocument>.Filter.Eq("upstream.content_hash", key.ContentHash);
|
||||
|
||||
var rawDocument = await rawCollection
|
||||
.Find(rawFilter)
|
||||
.Sort(Builders<BsonDocument>.Sort.Descending("ingested_at").Descending("_id"))
|
||||
.Limit(1)
|
||||
.FirstOrDefaultAsync(cancellationToken)
|
||||
.ConfigureAwait(false);
|
||||
|
||||
if (rawDocument is null)
|
||||
{
|
||||
missingRawDocuments.Add(key.ToString());
|
||||
continue;
|
||||
}
|
||||
|
||||
var advisoryRaw = MapToRawDocument(rawDocument);
|
||||
var rawLinkset = BuildRawLinkset(advisoryRaw.Identifiers, advisoryRaw.Linkset);
|
||||
var rawLinksetDocument = BuildRawLinksetBson(rawLinkset);
|
||||
|
||||
var update = Builders<BsonDocument>.Update.Set("rawLinkset", rawLinksetDocument);
|
||||
updates.Add(new UpdateOneModel<BsonDocument>(
|
||||
Builders<BsonDocument>.Filter.Eq("_id", observationDocument["_id"].AsString),
|
||||
update));
|
||||
|
||||
if (updates.Count >= BulkBatchSize)
|
||||
{
|
||||
await observations.BulkWriteAsync(updates, cancellationToken: cancellationToken).ConfigureAwait(false);
|
||||
updates.Clear();
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
if (updates.Count > 0)
|
||||
{
|
||||
await observations.BulkWriteAsync(updates, cancellationToken: cancellationToken).ConfigureAwait(false);
|
||||
}
|
||||
|
||||
if (missingRawDocuments.Count > 0)
|
||||
{
|
||||
throw new InvalidOperationException(
|
||||
$"Unable to locate advisory_raw documents for {missingRawDocuments.Count} observations: {string.Join(", ", missingRawDocuments.Take(10))}");
|
||||
}
|
||||
}
|
||||
|
||||
private static bool TryExtractObservationKey(BsonDocument observation, out ObservationKey key)
|
||||
{
|
||||
key = default;
|
||||
|
||||
if (!observation.TryGetValue("tenant", out var tenantValue) || tenantValue.IsBsonNull)
|
||||
{
|
||||
return false;
|
||||
}
|
||||
|
||||
if (!observation.TryGetValue("source", out var sourceValue) || sourceValue is not BsonDocument sourceDocument)
|
||||
{
|
||||
return false;
|
||||
}
|
||||
|
||||
if (!observation.TryGetValue("upstream", out var upstreamValue) || upstreamValue is not BsonDocument upstreamDocument)
|
||||
{
|
||||
return false;
|
||||
}
|
||||
|
||||
var tenant = tenantValue.AsString;
|
||||
var vendor = sourceDocument.GetValue("vendor", BsonString.Empty).AsString;
|
||||
var upstreamId = upstreamDocument.GetValue("upstream_id", BsonString.Empty).AsString;
|
||||
var contentHash = upstreamDocument.GetValue("contentHash", BsonString.Empty).AsString;
|
||||
var createdAt = observation.GetValue("createdAt", BsonNull.Value);
|
||||
|
||||
key = new ObservationKey(
|
||||
tenant,
|
||||
vendor,
|
||||
upstreamId,
|
||||
contentHash,
|
||||
BsonValueToDateTimeOffset(createdAt) ?? DateTimeOffset.UtcNow);
|
||||
|
||||
return !string.IsNullOrWhiteSpace(tenant) &&
|
||||
!string.IsNullOrWhiteSpace(vendor) &&
|
||||
!string.IsNullOrWhiteSpace(upstreamId) &&
|
||||
!string.IsNullOrWhiteSpace(contentHash);
|
||||
}
|
||||
|
||||
private static AdvisoryRawDocument MapToRawDocument(BsonDocument document)
|
||||
{
|
||||
var tenant = GetRequiredString(document, "tenant");
|
||||
var source = MapSource(document["source"].AsBsonDocument);
|
||||
var upstream = MapUpstream(document["upstream"].AsBsonDocument);
|
||||
var content = MapContent(document["content"].AsBsonDocument);
|
||||
var identifiers = MapIdentifiers(document["identifiers"].AsBsonDocument);
|
||||
var linkset = MapLinkset(document["linkset"].AsBsonDocument);
|
||||
var supersedes = document.GetValue("supersedes", BsonNull.Value);
|
||||
|
||||
return new AdvisoryRawDocument(
|
||||
tenant,
|
||||
source,
|
||||
upstream,
|
||||
content,
|
||||
identifiers,
|
||||
linkset,
|
||||
supersedes.IsBsonNull ? null : supersedes.AsString);
|
||||
}
|
||||
|
||||
private static RawSourceMetadata MapSource(BsonDocument source)
|
||||
{
|
||||
return new RawSourceMetadata(
|
||||
GetRequiredString(source, "vendor"),
|
||||
GetRequiredString(source, "connector"),
|
||||
GetRequiredString(source, "version"),
|
||||
GetOptionalString(source, "stream"));
|
||||
}
|
||||
|
||||
private static RawUpstreamMetadata MapUpstream(BsonDocument upstream)
|
||||
{
|
||||
var provenanceBuilder = ImmutableDictionary.CreateBuilder<string, string>(StringComparer.Ordinal);
|
||||
if (upstream.TryGetValue("provenance", out var provenanceValue) && provenanceValue.IsBsonDocument)
|
||||
{
|
||||
foreach (var element in provenanceValue.AsBsonDocument)
|
||||
{
|
||||
provenanceBuilder[element.Name] = BsonValueToString(element.Value);
|
||||
}
|
||||
}
|
||||
|
||||
var signatureDocument = upstream["signature"].AsBsonDocument;
|
||||
var signature = new RawSignatureMetadata(
|
||||
signatureDocument.GetValue("present", BsonBoolean.False).AsBoolean,
|
||||
signatureDocument.TryGetValue("format", out var format) && !format.IsBsonNull ? format.AsString : null,
|
||||
signatureDocument.TryGetValue("key_id", out var keyId) && !keyId.IsBsonNull ? keyId.AsString : null,
|
||||
signatureDocument.TryGetValue("sig", out var sig) && !sig.IsBsonNull ? sig.AsString : null,
|
||||
signatureDocument.TryGetValue("certificate", out var certificate) && !certificate.IsBsonNull ? certificate.AsString : null,
|
||||
signatureDocument.TryGetValue("digest", out var digest) && !digest.IsBsonNull ? digest.AsString : null);
|
||||
|
||||
return new RawUpstreamMetadata(
|
||||
GetRequiredString(upstream, "upstream_id"),
|
||||
upstream.TryGetValue("document_version", out var version) && !version.IsBsonNull ? version.AsString : null,
|
||||
GetDateTimeOffset(upstream, "retrieved_at", DateTimeOffset.UtcNow),
|
||||
GetRequiredString(upstream, "content_hash"),
|
||||
signature,
|
||||
provenanceBuilder.ToImmutable());
|
||||
}
|
||||
|
||||
private static RawContent MapContent(BsonDocument content)
|
||||
{
|
||||
var rawValue = content.GetValue("raw", BsonNull.Value);
|
||||
string rawJson;
|
||||
if (rawValue.IsBsonNull)
|
||||
{
|
||||
rawJson = "{}";
|
||||
}
|
||||
else if (rawValue.IsString)
|
||||
{
|
||||
rawJson = rawValue.AsString ?? "{}";
|
||||
}
|
||||
else
|
||||
{
|
||||
rawJson = rawValue.ToJson(new JsonWriterSettings { OutputMode = JsonOutputMode.RelaxedExtendedJson });
|
||||
}
|
||||
|
||||
using var document = JsonDocument.Parse(string.IsNullOrWhiteSpace(rawJson) ? "{}" : rawJson);
|
||||
|
||||
return new RawContent(
|
||||
GetRequiredString(content, "format"),
|
||||
content.TryGetValue("spec_version", out var specVersion) && !specVersion.IsBsonNull ? specVersion.AsString : null,
|
||||
document.RootElement.Clone(),
|
||||
content.TryGetValue("encoding", out var encoding) && !encoding.IsBsonNull ? encoding.AsString : null);
|
||||
}
|
||||
|
||||
private static RawIdentifiers MapIdentifiers(BsonDocument identifiers)
|
||||
{
|
||||
var aliases = identifiers.TryGetValue("aliases", out var aliasesValue) && aliasesValue.IsBsonArray
|
||||
? aliasesValue.AsBsonArray.Select(BsonValueToString).ToImmutableArray()
|
||||
: ImmutableArray<string>.Empty;
|
||||
|
||||
return new RawIdentifiers(
|
||||
aliases,
|
||||
GetRequiredString(identifiers, "primary"));
|
||||
}
|
||||
|
||||
private static RawLinkset MapLinkset(BsonDocument linkset)
|
||||
{
|
||||
var aliases = linkset.TryGetValue("aliases", out var aliasesValue) && aliasesValue.IsBsonArray
|
||||
? aliasesValue.AsBsonArray.Select(BsonValueToString).ToImmutableArray()
|
||||
: ImmutableArray<string>.Empty;
|
||||
|
||||
var purls = linkset.TryGetValue("purls", out var purlsValue) && purlsValue.IsBsonArray
|
||||
? purlsValue.AsBsonArray.Select(BsonValueToString).ToImmutableArray()
|
||||
: ImmutableArray<string>.Empty;
|
||||
|
||||
var cpes = linkset.TryGetValue("cpes", out var cpesValue) && cpesValue.IsBsonArray
|
||||
? cpesValue.AsBsonArray.Select(BsonValueToString).ToImmutableArray()
|
||||
: ImmutableArray<string>.Empty;
|
||||
|
||||
var references = linkset.TryGetValue("references", out var referencesValue) && referencesValue.IsBsonArray
|
||||
? referencesValue.AsBsonArray
|
||||
.Where(static value => value.IsBsonDocument)
|
||||
.Select(value =>
|
||||
{
|
||||
var doc = value.AsBsonDocument;
|
||||
return new RawReference(
|
||||
GetRequiredString(doc, "type"),
|
||||
GetRequiredString(doc, "url"),
|
||||
doc.TryGetValue("source", out var source) && !source.IsBsonNull ? source.AsString : null);
|
||||
})
|
||||
.ToImmutableArray()
|
||||
: ImmutableArray<RawReference>.Empty;
|
||||
|
||||
var reconciledFrom = linkset.TryGetValue("reconciled_from", out var reconciledValue) && reconciledValue.IsBsonArray
|
||||
? reconciledValue.AsBsonArray.Select(BsonValueToString).ToImmutableArray()
|
||||
: ImmutableArray<string>.Empty;
|
||||
|
||||
var notesBuilder = ImmutableDictionary.CreateBuilder<string, string>(StringComparer.Ordinal);
|
||||
if (linkset.TryGetValue("notes", out var notesValue) && notesValue.IsBsonDocument)
|
||||
{
|
||||
foreach (var element in notesValue.AsBsonDocument)
|
||||
{
|
||||
notesBuilder[element.Name] = BsonValueToString(element.Value);
|
||||
}
|
||||
}
|
||||
|
||||
return new RawLinkset
|
||||
{
|
||||
Aliases = aliases,
|
||||
PackageUrls = purls,
|
||||
Cpes = cpes,
|
||||
References = references,
|
||||
ReconciledFrom = reconciledFrom,
|
||||
Notes = notesBuilder.ToImmutable()
|
||||
};
|
||||
}
|
||||
|
||||
private static RawLinkset BuildRawLinkset(RawIdentifiers identifiers, RawLinkset linkset)
|
||||
{
|
||||
var aliasBuilder = ImmutableArray.CreateBuilder<string>();
|
||||
|
||||
if (!string.IsNullOrWhiteSpace(identifiers.PrimaryId))
|
||||
{
|
||||
aliasBuilder.Add(identifiers.PrimaryId);
|
||||
}
|
||||
|
||||
if (!identifiers.Aliases.IsDefaultOrEmpty)
|
||||
{
|
||||
foreach (var alias in identifiers.Aliases)
|
||||
{
|
||||
if (!string.IsNullOrEmpty(alias))
|
||||
{
|
||||
aliasBuilder.Add(alias);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
if (!linkset.Aliases.IsDefaultOrEmpty)
|
||||
{
|
||||
foreach (var alias in linkset.Aliases)
|
||||
{
|
||||
if (!string.IsNullOrEmpty(alias))
|
||||
{
|
||||
aliasBuilder.Add(alias);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
static ImmutableArray<string> EnsureArray(ImmutableArray<string> values)
|
||||
=> values.IsDefault ? ImmutableArray<string>.Empty : values;
|
||||
|
||||
static ImmutableArray<RawReference> EnsureReferences(ImmutableArray<RawReference> values)
|
||||
=> values.IsDefault ? ImmutableArray<RawReference>.Empty : values;
|
||||
|
||||
return linkset with
|
||||
{
|
||||
Aliases = aliasBuilder.ToImmutable(),
|
||||
PackageUrls = EnsureArray(linkset.PackageUrls),
|
||||
Cpes = EnsureArray(linkset.Cpes),
|
||||
References = EnsureReferences(linkset.References),
|
||||
ReconciledFrom = EnsureArray(linkset.ReconciledFrom),
|
||||
Notes = linkset.Notes ?? ImmutableDictionary<string, string>.Empty
|
||||
};
|
||||
}
|
||||
|
||||
private static BsonDocument BuildRawLinksetBson(RawLinkset rawLinkset)
|
||||
{
|
||||
var references = new BsonArray(rawLinkset.References.Select(reference =>
|
||||
{
|
||||
var referenceDocument = new BsonDocument
|
||||
{
|
||||
{ "type", reference.Type },
|
||||
{ "url", reference.Url }
|
||||
};
|
||||
if (!string.IsNullOrWhiteSpace(reference.Source))
|
||||
{
|
||||
referenceDocument["source"] = reference.Source;
|
||||
}
|
||||
|
||||
return referenceDocument;
|
||||
}));
|
||||
|
||||
var notes = new BsonDocument();
|
||||
if (rawLinkset.Notes is not null)
|
||||
{
|
||||
foreach (var entry in rawLinkset.Notes)
|
||||
{
|
||||
notes[entry.Key] = entry.Value;
|
||||
}
|
||||
}
|
||||
|
||||
return new BsonDocument
|
||||
{
|
||||
{ "aliases", new BsonArray(rawLinkset.Aliases) },
|
||||
{ "purls", new BsonArray(rawLinkset.PackageUrls) },
|
||||
{ "cpes", new BsonArray(rawLinkset.Cpes) },
|
||||
{ "references", references },
|
||||
{ "reconciled_from", new BsonArray(rawLinkset.ReconciledFrom) },
|
||||
{ "notes", notes }
|
||||
};
|
||||
}
|
||||
|
||||
private static string GetRequiredString(BsonDocument document, string key)
|
||||
{
|
||||
if (!document.TryGetValue(key, out var value) || value.IsBsonNull)
|
||||
{
|
||||
return string.Empty;
|
||||
}
|
||||
|
||||
return value.IsString ? value.AsString : value.ToString() ?? string.Empty;
|
||||
}
|
||||
|
||||
private static string? GetOptionalString(BsonDocument document, string key)
|
||||
{
|
||||
if (!document.TryGetValue(key, out var value) || value.IsBsonNull)
|
||||
{
|
||||
return null;
|
||||
}
|
||||
|
||||
return value.IsString ? value.AsString : value.ToString();
|
||||
}
|
||||
|
||||
private static string BsonValueToString(BsonValue value)
|
||||
{
|
||||
if (value.IsString)
|
||||
{
|
||||
return value.AsString ?? string.Empty;
|
||||
}
|
||||
|
||||
if (value.IsBsonNull)
|
||||
{
|
||||
return string.Empty;
|
||||
}
|
||||
|
||||
return value.ToString() ?? string.Empty;
|
||||
}
|
||||
|
||||
private static DateTimeOffset GetDateTimeOffset(BsonDocument document, string field, DateTimeOffset fallback)
|
||||
{
|
||||
if (!document.TryGetValue(field, out var value) || value.IsBsonNull)
|
||||
{
|
||||
return fallback;
|
||||
}
|
||||
|
||||
return BsonValueToDateTimeOffset(value) ?? fallback;
|
||||
}
|
||||
|
||||
private static DateTimeOffset? BsonValueToDateTimeOffset(BsonValue value)
|
||||
{
|
||||
return value.BsonType switch
|
||||
{
|
||||
BsonType.DateTime => new DateTimeOffset(DateTime.SpecifyKind(value.ToUniversalTime(), DateTimeKind.Utc)),
|
||||
BsonType.String when DateTimeOffset.TryParse(value.AsString, CultureInfo.InvariantCulture, DateTimeStyles.AssumeUniversal, out var parsed)
|
||||
=> parsed.ToUniversalTime(),
|
||||
BsonType.Int64 => DateTimeOffset.FromUnixTimeMilliseconds(value.AsInt64).ToUniversalTime(),
|
||||
_ => null
|
||||
};
|
||||
}
|
||||
|
||||
private readonly record struct ObservationKey(
|
||||
string Tenant,
|
||||
string Vendor,
|
||||
string UpstreamId,
|
||||
string ContentHash,
|
||||
DateTimeOffset CreatedAt)
|
||||
{
|
||||
public override string ToString()
|
||||
=> $"{Tenant}:{Vendor}:{UpstreamId}:{ContentHash}";
|
||||
}
|
||||
}
|
||||
@@ -26,6 +26,11 @@ public sealed class AdvisoryObservationDocument
|
||||
[BsonElement("linkset")]
|
||||
public AdvisoryObservationLinksetDocument Linkset { get; set; } = new();
|
||||
|
||||
[BsonElement("rawLinkset")]
|
||||
[BsonIgnoreIfNull]
|
||||
public AdvisoryObservationRawLinksetDocument? RawLinkset { get; set; }
|
||||
= null;
|
||||
|
||||
[BsonElement("createdAt")]
|
||||
public DateTime CreatedAt { get; set; }
|
||||
= DateTime.UtcNow;
|
||||
@@ -161,3 +166,54 @@ public sealed class AdvisoryObservationReferenceDocument
|
||||
[BsonElement("url")]
|
||||
public string Url { get; set; } = string.Empty;
|
||||
}
|
||||
|
||||
[BsonIgnoreExtraElements]
|
||||
public sealed class AdvisoryObservationRawLinksetDocument
|
||||
{
|
||||
[BsonElement("aliases")]
|
||||
[BsonIgnoreIfNull]
|
||||
public List<string>? Aliases { get; set; }
|
||||
= new();
|
||||
|
||||
[BsonElement("purls")]
|
||||
[BsonIgnoreIfNull]
|
||||
public List<string>? PackageUrls { get; set; }
|
||||
= new();
|
||||
|
||||
[BsonElement("cpes")]
|
||||
[BsonIgnoreIfNull]
|
||||
public List<string>? Cpes { get; set; }
|
||||
= new();
|
||||
|
||||
[BsonElement("references")]
|
||||
[BsonIgnoreIfNull]
|
||||
public List<AdvisoryObservationRawReferenceDocument>? References { get; set; }
|
||||
= new();
|
||||
|
||||
[BsonElement("reconciled_from")]
|
||||
[BsonIgnoreIfNull]
|
||||
public List<string>? ReconciledFrom { get; set; }
|
||||
= new();
|
||||
|
||||
[BsonElement("notes")]
|
||||
[BsonIgnoreIfNull]
|
||||
public Dictionary<string, string>? Notes { get; set; }
|
||||
= new(StringComparer.Ordinal);
|
||||
}
|
||||
|
||||
[BsonIgnoreExtraElements]
|
||||
public sealed class AdvisoryObservationRawReferenceDocument
|
||||
{
|
||||
[BsonElement("type")]
|
||||
[BsonIgnoreIfNull]
|
||||
public string? Type { get; set; }
|
||||
= null;
|
||||
|
||||
[BsonElement("url")]
|
||||
public string Url { get; set; } = string.Empty;
|
||||
|
||||
[BsonElement("source")]
|
||||
[BsonIgnoreIfNull]
|
||||
public string? Source { get; set; }
|
||||
= null;
|
||||
}
|
||||
|
||||
@@ -6,6 +6,7 @@ using System.Text.Json.Nodes;
|
||||
using MongoDB.Bson;
|
||||
using MongoDB.Bson.IO;
|
||||
using StellaOps.Concelier.Models.Observations;
|
||||
using StellaOps.Concelier.RawModels;
|
||||
|
||||
namespace StellaOps.Concelier.Storage.Mongo.Observations;
|
||||
|
||||
@@ -22,6 +23,8 @@ internal static class AdvisoryObservationDocumentFactory
|
||||
var contentMetadata = ToImmutable(document.Content.Metadata);
|
||||
var upstreamMetadata = ToImmutable(document.Upstream.Metadata);
|
||||
|
||||
var rawLinkset = ToRawLinkset(document.RawLinkset);
|
||||
|
||||
var observation = new AdvisoryObservation(
|
||||
document.Id,
|
||||
document.Tenant,
|
||||
@@ -52,6 +55,7 @@ internal static class AdvisoryObservationDocumentFactory
|
||||
document.Linkset.Purls ?? Enumerable.Empty<string>(),
|
||||
document.Linkset.Cpes ?? Enumerable.Empty<string>(),
|
||||
document.Linkset.References?.Select(reference => new AdvisoryObservationReference(reference.Type, reference.Url))),
|
||||
rawLinkset,
|
||||
DateTime.SpecifyKind(document.CreatedAt, DateTimeKind.Utc),
|
||||
attributes);
|
||||
|
||||
@@ -89,4 +93,70 @@ internal static class AdvisoryObservationDocumentFactory
|
||||
|
||||
return builder.ToImmutable();
|
||||
}
|
||||
|
||||
private static RawLinkset ToRawLinkset(AdvisoryObservationRawLinksetDocument? document)
|
||||
{
|
||||
if (document is null)
|
||||
{
|
||||
return new RawLinkset();
|
||||
}
|
||||
|
||||
static ImmutableArray<string> ToImmutableStringArray(List<string>? values)
|
||||
{
|
||||
if (values is null || values.Count == 0)
|
||||
{
|
||||
return ImmutableArray<string>.Empty;
|
||||
}
|
||||
|
||||
return values
|
||||
.Select(static value => value ?? string.Empty)
|
||||
.ToImmutableArray();
|
||||
}
|
||||
|
||||
static ImmutableArray<RawReference> ToImmutableReferences(List<AdvisoryObservationRawReferenceDocument>? references)
|
||||
{
|
||||
if (references is null || references.Count == 0)
|
||||
{
|
||||
return ImmutableArray<RawReference>.Empty;
|
||||
}
|
||||
|
||||
return references
|
||||
.Select(static reference => new RawReference(
|
||||
reference.Type ?? string.Empty,
|
||||
reference.Url,
|
||||
reference.Source))
|
||||
.ToImmutableArray();
|
||||
}
|
||||
|
||||
static ImmutableDictionary<string, string> ToImmutableDictionary(Dictionary<string, string>? values)
|
||||
{
|
||||
if (values is null || values.Count == 0)
|
||||
{
|
||||
return ImmutableDictionary<string, string>.Empty;
|
||||
}
|
||||
|
||||
var builder = ImmutableDictionary.CreateBuilder<string, string>(StringComparer.Ordinal);
|
||||
foreach (var pair in values)
|
||||
{
|
||||
if (pair.Key is null)
|
||||
{
|
||||
continue;
|
||||
}
|
||||
|
||||
builder[pair.Key] = pair.Value;
|
||||
}
|
||||
|
||||
return builder.ToImmutable();
|
||||
}
|
||||
|
||||
return new RawLinkset
|
||||
{
|
||||
Aliases = ToImmutableStringArray(document.Aliases),
|
||||
PackageUrls = ToImmutableStringArray(document.PackageUrls),
|
||||
Cpes = ToImmutableStringArray(document.Cpes),
|
||||
References = ToImmutableReferences(document.References),
|
||||
ReconciledFrom = ToImmutableStringArray(document.ReconciledFrom),
|
||||
Notes = ToImmutableDictionary(document.Notes)
|
||||
};
|
||||
}
|
||||
}
|
||||
|
||||
@@ -109,6 +109,7 @@ public static class ServiceCollectionExtensions
|
||||
services.AddSingleton<IMongoMigration, EnsureAdvisoryRawIdempotencyIndexMigration>();
|
||||
services.AddSingleton<IMongoMigration, EnsureAdvisorySupersedesBackfillMigration>();
|
||||
services.AddSingleton<IMongoMigration, EnsureAdvisoryRawValidatorMigration>();
|
||||
services.AddSingleton<IMongoMigration, EnsureAdvisoryObservationsRawLinksetMigration>();
|
||||
services.AddSingleton<IMongoMigration, EnsureAdvisoryEventCollectionsMigration>();
|
||||
services.AddSingleton<IMongoMigration, SemVerStyleBackfillMigration>();
|
||||
|
||||
|
||||
@@ -10,6 +10,7 @@
|
||||
> Docs alignment (2025-10-26): Rollback guidance added to `docs/deploy/containers.md` §6.
|
||||
> 2025-10-28: Documented duplicate audit + migration workflow in `docs/deploy/containers.md`, Offline Kit guide, and `MIGRATIONS.md`; published `ops/devops/scripts/check-advisory-raw-duplicates.js` for staging/offline clusters.
|
||||
> Docs alignment (2025-10-26): Offline kit requirements documented in `docs/deploy/containers.md` §5.
|
||||
| CONCELIER-STORE-AOC-19-005 `Raw linkset backfill` | TODO (2025-11-04) | Concelier Storage Guild, DevOps Guild | CONCELIER-CORE-AOC-19-004 | Plan and execute advisory_observations `rawLinkset` backfill (online + Offline Kit bundles), supply migration scripts + rehearse rollback. Follow the coordination plan in `docs/dev/raw-linkset-backfill-plan.md`. |
|
||||
|
||||
## Policy Engine v2
|
||||
|
||||
|
||||
@@ -39,6 +39,28 @@ public sealed class AdvisoryObservationFactoryTests
|
||||
var reference = Assert.Single(observation.Linkset.References);
|
||||
Assert.Equal("advisory", reference.Type);
|
||||
Assert.Equal("https://example.test/advisory", reference.Url);
|
||||
|
||||
Assert.Equal(
|
||||
new[] { "GHSA-XXXX-YYYY", " CVE-2025-0001 ", "ghsa-XXXX-YYYY", " CVE-2025-0001 " },
|
||||
observation.RawLinkset.Aliases);
|
||||
Assert.Equal(
|
||||
new[] { "pkg:NPM/left-pad@1.0.0", "pkg:npm/left-pad@1.0.0?foo=bar" },
|
||||
observation.RawLinkset.PackageUrls);
|
||||
Assert.Equal(
|
||||
new[] { "cpe:/a:Example:Product:1.0", "cpe:/a:example:product:1.0" },
|
||||
observation.RawLinkset.Cpes);
|
||||
Assert.Collection(
|
||||
observation.RawLinkset.References,
|
||||
first =>
|
||||
{
|
||||
Assert.Equal("Advisory", first.Type);
|
||||
Assert.Equal(" https://example.test/advisory ", first.Url);
|
||||
},
|
||||
second =>
|
||||
{
|
||||
Assert.Equal("ADVISORY", second.Type);
|
||||
Assert.Equal("https://example.test/advisory", second.Url);
|
||||
});
|
||||
}
|
||||
|
||||
[Fact]
|
||||
@@ -102,6 +124,8 @@ public sealed class AdvisoryObservationFactoryTests
|
||||
Assert.Equal("1.0.5", observation.Attributes["linkset.note.range-fixed"]);
|
||||
Assert.Equal("tenant-a:vendor-x:previous:sha256:123", observation.Attributes["supersedes"]);
|
||||
Assert.Equal("connector-a;connector-b", observation.Attributes["linkset.reconciled_from"]);
|
||||
Assert.Equal(notes, observation.RawLinkset.Notes);
|
||||
Assert.Equal(new[] { "connector-a", "connector-b" }, observation.RawLinkset.ReconciledFrom);
|
||||
}
|
||||
|
||||
private static AdvisoryRawDocument BuildRawDocument(
|
||||
|
||||
@@ -1,7 +1,9 @@
|
||||
using System.Collections.Immutable;
|
||||
using System.Linq;
|
||||
using System.Text.Json.Nodes;
|
||||
using StellaOps.Concelier.Core.Observations;
|
||||
using StellaOps.Concelier.Models.Observations;
|
||||
using StellaOps.Concelier.RawModels;
|
||||
using Xunit;
|
||||
|
||||
namespace StellaOps.Concelier.Core.Tests.Observations;
|
||||
@@ -226,6 +228,15 @@ public sealed class AdvisoryObservationQueryServiceTests
|
||||
|
||||
var content = new AdvisoryObservationContent("CSAF", "2.0", raw);
|
||||
var linkset = new AdvisoryObservationLinkset(aliases, purls, cpes, references);
|
||||
var rawLinkset = new RawLinkset
|
||||
{
|
||||
Aliases = aliases.ToImmutableArray(),
|
||||
PackageUrls = purls.ToImmutableArray(),
|
||||
Cpes = cpes.ToImmutableArray(),
|
||||
References = references
|
||||
.Select(static reference => new RawReference(reference.Type, reference.Url))
|
||||
.ToImmutableArray()
|
||||
};
|
||||
|
||||
return new AdvisoryObservation(
|
||||
observationId,
|
||||
@@ -234,6 +245,7 @@ public sealed class AdvisoryObservationQueryServiceTests
|
||||
upstream,
|
||||
content,
|
||||
linkset,
|
||||
rawLinkset,
|
||||
createdAt);
|
||||
}
|
||||
|
||||
|
||||
@@ -1,5 +1,6 @@
|
||||
using System;
|
||||
using System.Collections.Immutable;
|
||||
using System.Linq;
|
||||
using System.Text.Json;
|
||||
using System.Threading;
|
||||
using System.Threading.Tasks;
|
||||
@@ -66,7 +67,7 @@ public sealed class AdvisoryRawServiceTests
|
||||
await service.IngestAsync(document, CancellationToken.None);
|
||||
|
||||
Assert.NotNull(repository.CapturedDocument);
|
||||
Assert.Equal(aliasSeries, repository.CapturedDocument!.Identifiers.Aliases);
|
||||
Assert.True(aliasSeries.SequenceEqual(repository.CapturedDocument!.Identifiers.Aliases));
|
||||
}
|
||||
|
||||
private static AdvisoryRawService CreateService(RecordingRepository repository)
|
||||
|
||||
@@ -1,127 +0,0 @@
|
||||
{
|
||||
"advisoryKey": "GHSA-aaaa-bbbb-cccc",
|
||||
"affectedPackages": [
|
||||
{
|
||||
"type": "semver",
|
||||
"identifier": "pkg:npm/example-widget",
|
||||
"platform": null,
|
||||
"versionRanges": [
|
||||
{
|
||||
"fixedVersion": "2.5.1",
|
||||
"introducedVersion": null,
|
||||
"lastAffectedVersion": null,
|
||||
"primitives": null,
|
||||
"provenance": {
|
||||
"source": "ghsa",
|
||||
"kind": "map",
|
||||
"value": "ghsa-aaaa-bbbb-cccc",
|
||||
"decisionReason": null,
|
||||
"recordedAt": "2024-03-05T10:00:00+00:00",
|
||||
"fieldMask": []
|
||||
},
|
||||
"rangeExpression": ">=0.0.0 <2.5.1",
|
||||
"rangeKind": "semver"
|
||||
},
|
||||
{
|
||||
"fixedVersion": "3.2.4",
|
||||
"introducedVersion": "3.0.0",
|
||||
"lastAffectedVersion": null,
|
||||
"primitives": null,
|
||||
"provenance": {
|
||||
"source": "ghsa",
|
||||
"kind": "map",
|
||||
"value": "ghsa-aaaa-bbbb-cccc",
|
||||
"decisionReason": null,
|
||||
"recordedAt": "2024-03-05T10:00:00+00:00",
|
||||
"fieldMask": []
|
||||
},
|
||||
"rangeExpression": null,
|
||||
"rangeKind": "semver"
|
||||
}
|
||||
],
|
||||
"normalizedVersions": [],
|
||||
"statuses": [],
|
||||
"provenance": [
|
||||
{
|
||||
"source": "ghsa",
|
||||
"kind": "map",
|
||||
"value": "ghsa-aaaa-bbbb-cccc",
|
||||
"decisionReason": null,
|
||||
"recordedAt": "2024-03-05T10:00:00+00:00",
|
||||
"fieldMask": []
|
||||
}
|
||||
]
|
||||
}
|
||||
],
|
||||
"aliases": [
|
||||
"CVE-2024-2222",
|
||||
"GHSA-aaaa-bbbb-cccc"
|
||||
],
|
||||
"canonicalMetricId": null,
|
||||
"credits": [],
|
||||
"cvssMetrics": [
|
||||
{
|
||||
"baseScore": 8.8,
|
||||
"baseSeverity": "high",
|
||||
"provenance": {
|
||||
"source": "ghsa",
|
||||
"kind": "map",
|
||||
"value": "ghsa-aaaa-bbbb-cccc",
|
||||
"decisionReason": null,
|
||||
"recordedAt": "2024-03-05T10:00:00+00:00",
|
||||
"fieldMask": []
|
||||
},
|
||||
"vector": "CVSS:3.1/AV:N/AC:L/PR:N/UI:R/S:U/C:H/I:H/A:H",
|
||||
"version": "3.1"
|
||||
}
|
||||
],
|
||||
"cwes": [],
|
||||
"description": null,
|
||||
"exploitKnown": false,
|
||||
"language": "en",
|
||||
"modified": "2024-03-04T12:00:00+00:00",
|
||||
"provenance": [
|
||||
{
|
||||
"source": "ghsa",
|
||||
"kind": "map",
|
||||
"value": "ghsa-aaaa-bbbb-cccc",
|
||||
"decisionReason": null,
|
||||
"recordedAt": "2024-03-05T10:00:00+00:00",
|
||||
"fieldMask": []
|
||||
}
|
||||
],
|
||||
"published": "2024-03-04T00:00:00+00:00",
|
||||
"references": [
|
||||
{
|
||||
"kind": "patch",
|
||||
"provenance": {
|
||||
"source": "ghsa",
|
||||
"kind": "map",
|
||||
"value": "ghsa-aaaa-bbbb-cccc",
|
||||
"decisionReason": null,
|
||||
"recordedAt": "2024-03-05T10:00:00+00:00",
|
||||
"fieldMask": []
|
||||
},
|
||||
"sourceTag": "ghsa",
|
||||
"summary": "Patch commit",
|
||||
"url": "https://github.com/example/widget/commit/abcd1234"
|
||||
},
|
||||
{
|
||||
"kind": "advisory",
|
||||
"provenance": {
|
||||
"source": "ghsa",
|
||||
"kind": "map",
|
||||
"value": "ghsa-aaaa-bbbb-cccc",
|
||||
"decisionReason": null,
|
||||
"recordedAt": "2024-03-05T10:00:00+00:00",
|
||||
"fieldMask": []
|
||||
},
|
||||
"sourceTag": "ghsa",
|
||||
"summary": "GitHub Security Advisory",
|
||||
"url": "https://github.com/example/widget/security/advisories/GHSA-aaaa-bbbb-cccc"
|
||||
}
|
||||
],
|
||||
"severity": "high",
|
||||
"summary": "A crafted payload can pollute Object.prototype leading to RCE.",
|
||||
"title": "Prototype pollution in widget.js"
|
||||
}
|
||||
@@ -57,6 +57,7 @@
|
||||
"CVE-2024-2222",
|
||||
"GHSA-aaaa-bbbb-cccc"
|
||||
],
|
||||
"canonicalMetricId": null,
|
||||
"credits": [],
|
||||
"cvssMetrics": [
|
||||
{
|
||||
@@ -74,6 +75,8 @@
|
||||
"version": "3.1"
|
||||
}
|
||||
],
|
||||
"cwes": [],
|
||||
"description": null,
|
||||
"exploitKnown": false,
|
||||
"language": "en",
|
||||
"modified": "2024-03-04T12:00:00+00:00",
|
||||
|
||||
@@ -1,45 +0,0 @@
|
||||
{
|
||||
"advisoryKey": "CVE-2023-9999",
|
||||
"affectedPackages": [],
|
||||
"aliases": [
|
||||
"CVE-2023-9999"
|
||||
],
|
||||
"canonicalMetricId": null,
|
||||
"credits": [],
|
||||
"cvssMetrics": [],
|
||||
"cwes": [],
|
||||
"description": null,
|
||||
"exploitKnown": true,
|
||||
"language": "en",
|
||||
"modified": "2024-02-09T16:22:00+00:00",
|
||||
"provenance": [
|
||||
{
|
||||
"source": "cisa-kev",
|
||||
"kind": "annotate",
|
||||
"value": "kev",
|
||||
"decisionReason": null,
|
||||
"recordedAt": "2024-02-10T09:30:00+00:00",
|
||||
"fieldMask": []
|
||||
}
|
||||
],
|
||||
"published": "2023-11-20T00:00:00+00:00",
|
||||
"references": [
|
||||
{
|
||||
"kind": "kev",
|
||||
"provenance": {
|
||||
"source": "cisa-kev",
|
||||
"kind": "annotate",
|
||||
"value": "kev",
|
||||
"decisionReason": null,
|
||||
"recordedAt": "2024-02-10T09:30:00+00:00",
|
||||
"fieldMask": []
|
||||
},
|
||||
"sourceTag": "cisa",
|
||||
"summary": "CISA KEV entry",
|
||||
"url": "https://www.cisa.gov/known-exploited-vulnerabilities-catalog"
|
||||
}
|
||||
],
|
||||
"severity": "critical",
|
||||
"summary": "Unauthenticated RCE due to unsafe deserialization.",
|
||||
"title": "Remote code execution in LegacyServer"
|
||||
}
|
||||
@@ -4,8 +4,11 @@
|
||||
"aliases": [
|
||||
"CVE-2023-9999"
|
||||
],
|
||||
"canonicalMetricId": null,
|
||||
"credits": [],
|
||||
"cvssMetrics": [],
|
||||
"cwes": [],
|
||||
"description": null,
|
||||
"exploitKnown": true,
|
||||
"language": "en",
|
||||
"modified": "2024-02-09T16:22:00+00:00",
|
||||
|
||||
@@ -1,122 +0,0 @@
|
||||
{
|
||||
"advisoryKey": "CVE-2024-1234",
|
||||
"affectedPackages": [
|
||||
{
|
||||
"type": "cpe",
|
||||
"identifier": "cpe:/a:examplecms:examplecms:1.0",
|
||||
"platform": null,
|
||||
"versionRanges": [
|
||||
{
|
||||
"fixedVersion": "1.0.5",
|
||||
"introducedVersion": "1.0",
|
||||
"lastAffectedVersion": null,
|
||||
"primitives": null,
|
||||
"provenance": {
|
||||
"source": "nvd",
|
||||
"kind": "map",
|
||||
"value": "cve-2024-1234",
|
||||
"decisionReason": null,
|
||||
"recordedAt": "2024-08-01T12:00:00+00:00",
|
||||
"fieldMask": []
|
||||
},
|
||||
"rangeExpression": null,
|
||||
"rangeKind": "version"
|
||||
}
|
||||
],
|
||||
"normalizedVersions": [],
|
||||
"statuses": [
|
||||
{
|
||||
"provenance": {
|
||||
"source": "nvd",
|
||||
"kind": "map",
|
||||
"value": "cve-2024-1234",
|
||||
"decisionReason": null,
|
||||
"recordedAt": "2024-08-01T12:00:00+00:00",
|
||||
"fieldMask": []
|
||||
},
|
||||
"status": "affected"
|
||||
}
|
||||
],
|
||||
"provenance": [
|
||||
{
|
||||
"source": "nvd",
|
||||
"kind": "map",
|
||||
"value": "cve-2024-1234",
|
||||
"decisionReason": null,
|
||||
"recordedAt": "2024-08-01T12:00:00+00:00",
|
||||
"fieldMask": []
|
||||
}
|
||||
]
|
||||
}
|
||||
],
|
||||
"aliases": [
|
||||
"CVE-2024-1234"
|
||||
],
|
||||
"canonicalMetricId": null,
|
||||
"credits": [],
|
||||
"cvssMetrics": [
|
||||
{
|
||||
"baseScore": 9.8,
|
||||
"baseSeverity": "critical",
|
||||
"provenance": {
|
||||
"source": "nvd",
|
||||
"kind": "map",
|
||||
"value": "cve-2024-1234",
|
||||
"decisionReason": null,
|
||||
"recordedAt": "2024-08-01T12:00:00+00:00",
|
||||
"fieldMask": []
|
||||
},
|
||||
"vector": "CVSS:3.1/AV:N/AC:L/PR:N/UI:N/S:U/C:H/I:H/A:H",
|
||||
"version": "3.1"
|
||||
}
|
||||
],
|
||||
"cwes": [],
|
||||
"description": null,
|
||||
"exploitKnown": false,
|
||||
"language": "en",
|
||||
"modified": "2024-07-16T10:35:00+00:00",
|
||||
"provenance": [
|
||||
{
|
||||
"source": "nvd",
|
||||
"kind": "map",
|
||||
"value": "cve-2024-1234",
|
||||
"decisionReason": null,
|
||||
"recordedAt": "2024-08-01T12:00:00+00:00",
|
||||
"fieldMask": []
|
||||
}
|
||||
],
|
||||
"published": "2024-07-15T00:00:00+00:00",
|
||||
"references": [
|
||||
{
|
||||
"kind": "advisory",
|
||||
"provenance": {
|
||||
"source": "example",
|
||||
"kind": "fetch",
|
||||
"value": "bulletin",
|
||||
"decisionReason": null,
|
||||
"recordedAt": "2024-07-14T15:00:00+00:00",
|
||||
"fieldMask": []
|
||||
},
|
||||
"sourceTag": "vendor",
|
||||
"summary": "Vendor bulletin",
|
||||
"url": "https://example.org/security/CVE-2024-1234"
|
||||
},
|
||||
{
|
||||
"kind": "advisory",
|
||||
"provenance": {
|
||||
"source": "nvd",
|
||||
"kind": "map",
|
||||
"value": "cve-2024-1234",
|
||||
"decisionReason": null,
|
||||
"recordedAt": "2024-08-01T12:00:00+00:00",
|
||||
"fieldMask": []
|
||||
},
|
||||
"sourceTag": "nvd",
|
||||
"summary": "NVD entry",
|
||||
"url": "https://nvd.nist.gov/vuln/detail/CVE-2024-1234"
|
||||
}
|
||||
],
|
||||
"severity": "high",
|
||||
"summary": "An integer overflow in ExampleCMS allows remote attackers to escalate privileges.",
|
||||
"title": "Integer overflow in ExampleCMS"
|
||||
}
|
||||
@@ -52,6 +52,7 @@
|
||||
"aliases": [
|
||||
"CVE-2024-1234"
|
||||
],
|
||||
"canonicalMetricId": null,
|
||||
"credits": [],
|
||||
"cvssMetrics": [
|
||||
{
|
||||
@@ -69,6 +70,8 @@
|
||||
"version": "3.1"
|
||||
}
|
||||
],
|
||||
"cwes": [],
|
||||
"description": null,
|
||||
"exploitKnown": false,
|
||||
"language": "en",
|
||||
"modified": "2024-07-16T10:35:00+00:00",
|
||||
|
||||
@@ -1,125 +0,0 @@
|
||||
{
|
||||
"advisoryKey": "RHSA-2024:0252",
|
||||
"affectedPackages": [
|
||||
{
|
||||
"type": "rpm",
|
||||
"identifier": "kernel-0:4.18.0-553.el8.x86_64",
|
||||
"platform": "rhel-8",
|
||||
"versionRanges": [
|
||||
{
|
||||
"fixedVersion": null,
|
||||
"introducedVersion": "0:4.18.0-553.el8",
|
||||
"lastAffectedVersion": null,
|
||||
"primitives": null,
|
||||
"provenance": {
|
||||
"source": "redhat",
|
||||
"kind": "map",
|
||||
"value": "rhsa-2024:0252",
|
||||
"decisionReason": null,
|
||||
"recordedAt": "2024-05-11T09:00:00+00:00",
|
||||
"fieldMask": []
|
||||
},
|
||||
"rangeExpression": null,
|
||||
"rangeKind": "nevra"
|
||||
}
|
||||
],
|
||||
"normalizedVersions": [],
|
||||
"statuses": [
|
||||
{
|
||||
"provenance": {
|
||||
"source": "redhat",
|
||||
"kind": "map",
|
||||
"value": "rhsa-2024:0252",
|
||||
"decisionReason": null,
|
||||
"recordedAt": "2024-05-11T09:00:00+00:00",
|
||||
"fieldMask": []
|
||||
},
|
||||
"status": "fixed"
|
||||
}
|
||||
],
|
||||
"provenance": [
|
||||
{
|
||||
"source": "redhat",
|
||||
"kind": "enrich",
|
||||
"value": "cve-2024-5678",
|
||||
"decisionReason": null,
|
||||
"recordedAt": "2024-05-11T09:05:00+00:00",
|
||||
"fieldMask": []
|
||||
},
|
||||
{
|
||||
"source": "redhat",
|
||||
"kind": "map",
|
||||
"value": "rhsa-2024:0252",
|
||||
"decisionReason": null,
|
||||
"recordedAt": "2024-05-11T09:00:00+00:00",
|
||||
"fieldMask": []
|
||||
}
|
||||
]
|
||||
}
|
||||
],
|
||||
"aliases": [
|
||||
"CVE-2024-5678",
|
||||
"RHSA-2024:0252"
|
||||
],
|
||||
"canonicalMetricId": null,
|
||||
"credits": [],
|
||||
"cvssMetrics": [
|
||||
{
|
||||
"baseScore": 6.7,
|
||||
"baseSeverity": "medium",
|
||||
"provenance": {
|
||||
"source": "redhat",
|
||||
"kind": "map",
|
||||
"value": "rhsa-2024:0252",
|
||||
"decisionReason": null,
|
||||
"recordedAt": "2024-05-11T09:00:00+00:00",
|
||||
"fieldMask": []
|
||||
},
|
||||
"vector": "CVSS:3.1/AV:L/AC:L/PR:H/UI:N/S:U/C:H/I:H/A:H",
|
||||
"version": "3.1"
|
||||
}
|
||||
],
|
||||
"cwes": [],
|
||||
"description": null,
|
||||
"exploitKnown": false,
|
||||
"language": "en",
|
||||
"modified": "2024-05-11T08:15:00+00:00",
|
||||
"provenance": [
|
||||
{
|
||||
"source": "redhat",
|
||||
"kind": "enrich",
|
||||
"value": "cve-2024-5678",
|
||||
"decisionReason": null,
|
||||
"recordedAt": "2024-05-11T09:05:00+00:00",
|
||||
"fieldMask": []
|
||||
},
|
||||
{
|
||||
"source": "redhat",
|
||||
"kind": "map",
|
||||
"value": "rhsa-2024:0252",
|
||||
"decisionReason": null,
|
||||
"recordedAt": "2024-05-11T09:00:00+00:00",
|
||||
"fieldMask": []
|
||||
}
|
||||
],
|
||||
"published": "2024-05-10T19:28:00+00:00",
|
||||
"references": [
|
||||
{
|
||||
"kind": "advisory",
|
||||
"provenance": {
|
||||
"source": "redhat",
|
||||
"kind": "map",
|
||||
"value": "rhsa-2024:0252",
|
||||
"decisionReason": null,
|
||||
"recordedAt": "2024-05-11T09:00:00+00:00",
|
||||
"fieldMask": []
|
||||
},
|
||||
"sourceTag": "redhat",
|
||||
"summary": "Red Hat security advisory",
|
||||
"url": "https://access.redhat.com/errata/RHSA-2024:0252"
|
||||
}
|
||||
],
|
||||
"severity": "critical",
|
||||
"summary": "Updates the Red Hat Enterprise Linux kernel to address CVE-2024-5678.",
|
||||
"title": "Important: kernel security update"
|
||||
}
|
||||
@@ -61,6 +61,7 @@
|
||||
"CVE-2024-5678",
|
||||
"RHSA-2024:0252"
|
||||
],
|
||||
"canonicalMetricId": null,
|
||||
"credits": [],
|
||||
"cvssMetrics": [
|
||||
{
|
||||
@@ -78,6 +79,8 @@
|
||||
"version": "3.1"
|
||||
}
|
||||
],
|
||||
"cwes": [],
|
||||
"description": null,
|
||||
"exploitKnown": false,
|
||||
"language": "en",
|
||||
"modified": "2024-05-11T08:15:00+00:00",
|
||||
|
||||
@@ -2,6 +2,7 @@ using System.Collections.Generic;
|
||||
using System.Collections.Immutable;
|
||||
using System.Text.Json.Nodes;
|
||||
using StellaOps.Concelier.Models.Observations;
|
||||
using StellaOps.Concelier.RawModels;
|
||||
using Xunit;
|
||||
|
||||
namespace StellaOps.Concelier.Models.Tests.Observations;
|
||||
@@ -39,6 +40,16 @@ public sealed class AdvisoryObservationTests
|
||||
["pipeline"] = "daily"
|
||||
});
|
||||
|
||||
var rawLinkset = new RawLinkset
|
||||
{
|
||||
Aliases = ImmutableArray.Create(" Cve-2025-1234 ", "cve-2025-1234"),
|
||||
PackageUrls = ImmutableArray.Create("pkg:generic/foo@1.0.0"),
|
||||
Cpes = ImmutableArray.Create("cpe:/a:vendor:product:1"),
|
||||
References = ImmutableArray.Create(new RawReference("ADVISORY", "https://example.com/advisory")),
|
||||
ReconciledFrom = ImmutableArray.Create("pointer-1"),
|
||||
Notes = ImmutableDictionary.CreateRange(new Dictionary<string, string> { ["note"] = "value" })
|
||||
};
|
||||
|
||||
var observation = new AdvisoryObservation(
|
||||
observationId: " tenant-a:CVE-2025-1234:1 ",
|
||||
tenant: " Tenant-A ",
|
||||
@@ -46,6 +57,7 @@ public sealed class AdvisoryObservationTests
|
||||
upstream: upstream,
|
||||
content: content,
|
||||
linkset: linkset,
|
||||
rawLinkset: rawLinkset,
|
||||
createdAt: DateTimeOffset.Parse("2025-10-01T01:00:06Z"),
|
||||
attributes: attributes);
|
||||
|
||||
|
||||
@@ -0,0 +1,337 @@
|
||||
using System;
|
||||
using System.Collections.Generic;
|
||||
using System.Collections.Immutable;
|
||||
using System.Linq;
|
||||
using System.Text.Json;
|
||||
using System.Threading;
|
||||
using System.Threading.Tasks;
|
||||
using Microsoft.Extensions.Logging.Abstractions;
|
||||
using MongoDB.Bson;
|
||||
using MongoDB.Bson.Serialization;
|
||||
using MongoDB.Driver;
|
||||
using StellaOps.Concelier.RawModels;
|
||||
using StellaOps.Concelier.Storage.Mongo;
|
||||
using StellaOps.Concelier.Storage.Mongo.Migrations;
|
||||
using StellaOps.Concelier.Storage.Mongo.Observations;
|
||||
using StellaOps.Concelier.Storage.Mongo.Raw;
|
||||
using Xunit;
|
||||
|
||||
namespace StellaOps.Concelier.Storage.Mongo.Tests.Migrations;
|
||||
|
||||
[Collection("mongo-fixture")]
|
||||
public sealed class EnsureAdvisoryObservationsRawLinksetMigrationTests
|
||||
{
|
||||
private readonly MongoIntegrationFixture _fixture;
|
||||
|
||||
public EnsureAdvisoryObservationsRawLinksetMigrationTests(MongoIntegrationFixture fixture)
|
||||
{
|
||||
_fixture = fixture;
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task ApplyAsync_BackfillsRawLinksetFromRawDocument()
|
||||
{
|
||||
var databaseName = $"concelier-rawlinkset-{Guid.NewGuid():N}";
|
||||
var database = _fixture.Client.GetDatabase(databaseName);
|
||||
await database.CreateCollectionAsync(MongoStorageDefaults.Collections.Migrations);
|
||||
await database.CreateCollectionAsync(MongoStorageDefaults.Collections.AdvisoryObservations);
|
||||
|
||||
try
|
||||
{
|
||||
var rawRepository = new MongoAdvisoryRawRepository(
|
||||
database,
|
||||
TimeProvider.System,
|
||||
NullLogger<MongoAdvisoryRawRepository>.Instance);
|
||||
|
||||
var rawDocument = RawDocumentFactory.CreateAdvisory(
|
||||
tenant: "tenant-a",
|
||||
source: new RawSourceMetadata("Vendor-X", "connector-y", "1.0.0", "stable"),
|
||||
upstream: new RawUpstreamMetadata(
|
||||
UpstreamId: "GHSA-2025-0001",
|
||||
DocumentVersion: "v1",
|
||||
RetrievedAt: DateTimeOffset.Parse("2025-10-29T12:34:56Z"),
|
||||
ContentHash: "sha256:abc123",
|
||||
Signature: new RawSignatureMetadata(true, "dsse", "key1", "sig1"),
|
||||
Provenance: ImmutableDictionary.CreateRange(new[] { new KeyValuePair<string, string>("api", "https://example.test/api") })),
|
||||
content: new RawContent(
|
||||
Format: "OSV",
|
||||
SpecVersion: "1.0.0",
|
||||
Raw: ParseJsonElement("""{"id":"GHSA-2025-0001"}"""),
|
||||
Encoding: null),
|
||||
identifiers: new RawIdentifiers(
|
||||
Aliases: ImmutableArray.Create("CVE-2025-0001", "cve-2025-0001"),
|
||||
PrimaryId: "CVE-2025-0001"),
|
||||
linkset: new RawLinkset
|
||||
{
|
||||
Aliases = ImmutableArray.Create("GHSA-xxxx-yyyy"),
|
||||
PackageUrls = ImmutableArray.Create("pkg:npm/example@1.0.0"),
|
||||
Cpes = ImmutableArray.Create("cpe:/a:example:product:1.0"),
|
||||
References = ImmutableArray.Create(new RawReference("advisory", "https://example.test/advisory", "vendor")),
|
||||
ReconciledFrom = ImmutableArray.Create("connector-y"),
|
||||
Notes = ImmutableDictionary.CreateRange(new[] { new KeyValuePair<string, string>("range-fixed", "1.0.1") })
|
||||
});
|
||||
|
||||
await rawRepository.UpsertAsync(rawDocument, CancellationToken.None);
|
||||
|
||||
var expectedRawLinkset = BuildRawLinkset(rawDocument.Identifiers, rawDocument.Linkset);
|
||||
var canonicalAliases = ImmutableArray.Create("cve-2025-0001", "ghsa-xxxx-yyyy");
|
||||
var canonicalPurls = rawDocument.Linkset.PackageUrls;
|
||||
var canonicalCpes = rawDocument.Linkset.Cpes;
|
||||
var canonicalReferences = rawDocument.Linkset.References;
|
||||
|
||||
var observationId = "tenant-a:vendor-x:ghsa-2025-0001:sha256-abc123";
|
||||
var observationBson = BuildObservationDocument(
|
||||
observationId,
|
||||
rawDocument,
|
||||
canonicalAliases,
|
||||
canonicalPurls,
|
||||
canonicalCpes,
|
||||
canonicalReferences,
|
||||
rawDocument.Upstream.RetrievedAt,
|
||||
includeRawLinkset: false);
|
||||
await database
|
||||
.GetCollection<BsonDocument>(MongoStorageDefaults.Collections.AdvisoryObservations)
|
||||
.InsertOneAsync(observationBson);
|
||||
|
||||
var migration = new EnsureAdvisoryObservationsRawLinksetMigration();
|
||||
await migration.ApplyAsync(database, CancellationToken.None);
|
||||
|
||||
var storedBson = await database
|
||||
.GetCollection<BsonDocument>(MongoStorageDefaults.Collections.AdvisoryObservations)
|
||||
.Find(Builders<BsonDocument>.Filter.Eq("_id", observationId))
|
||||
.FirstOrDefaultAsync();
|
||||
|
||||
Assert.NotNull(storedBson);
|
||||
Assert.True(storedBson.TryGetValue("rawLinkset", out var rawLinksetValue));
|
||||
|
||||
var storedDocument = BsonSerializer.Deserialize<AdvisoryObservationDocument>(storedBson);
|
||||
var storedObservation = AdvisoryObservationDocumentFactory.ToModel(storedDocument);
|
||||
|
||||
Assert.True(expectedRawLinkset.Aliases.SequenceEqual(storedObservation.RawLinkset.Aliases, StringComparer.Ordinal));
|
||||
Assert.True(expectedRawLinkset.PackageUrls.SequenceEqual(storedObservation.RawLinkset.PackageUrls, StringComparer.Ordinal));
|
||||
Assert.True(expectedRawLinkset.Cpes.SequenceEqual(storedObservation.RawLinkset.Cpes, StringComparer.Ordinal));
|
||||
Assert.True(expectedRawLinkset.References.SequenceEqual(storedObservation.RawLinkset.References));
|
||||
Assert.Equal(expectedRawLinkset.Notes, storedObservation.RawLinkset.Notes);
|
||||
}
|
||||
finally
|
||||
{
|
||||
await _fixture.Client.DropDatabaseAsync(databaseName);
|
||||
}
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task ApplyAsync_ThrowsWhenRawDocumentMissing()
|
||||
{
|
||||
var databaseName = $"concelier-rawlinkset-missing-{Guid.NewGuid():N}";
|
||||
var database = _fixture.Client.GetDatabase(databaseName);
|
||||
await database.CreateCollectionAsync(MongoStorageDefaults.Collections.Migrations);
|
||||
await database.CreateCollectionAsync(MongoStorageDefaults.Collections.AdvisoryObservations);
|
||||
|
||||
try
|
||||
{
|
||||
var rawDocument = RawDocumentFactory.CreateAdvisory(
|
||||
tenant: "tenant-b",
|
||||
source: new RawSourceMetadata("Vendor-Y", "connector-z", "2.0.0", "stable"),
|
||||
upstream: new RawUpstreamMetadata(
|
||||
UpstreamId: "GHSA-9999-0001",
|
||||
DocumentVersion: "v2",
|
||||
RetrievedAt: DateTimeOffset.Parse("2025-10-30T00:00:00Z"),
|
||||
ContentHash: "sha256:def456",
|
||||
Signature: new RawSignatureMetadata(false),
|
||||
Provenance: ImmutableDictionary<string, string>.Empty),
|
||||
content: new RawContent(
|
||||
Format: "OSV",
|
||||
SpecVersion: "1.0.0",
|
||||
Raw: ParseJsonElement("""{"id":"GHSA-9999-0001"}"""),
|
||||
Encoding: null),
|
||||
identifiers: new RawIdentifiers(
|
||||
Aliases: ImmutableArray<string>.Empty,
|
||||
PrimaryId: "GHSA-9999-0001"),
|
||||
linkset: new RawLinkset());
|
||||
|
||||
var observationId = "tenant-b:vendor-y:ghsa-9999-0001:sha256-def456";
|
||||
var document = BuildObservationDocument(
|
||||
observationId,
|
||||
rawDocument,
|
||||
ImmutableArray<string>.Empty,
|
||||
ImmutableArray<string>.Empty,
|
||||
ImmutableArray<string>.Empty,
|
||||
ImmutableArray<RawReference>.Empty,
|
||||
rawDocument.Upstream.RetrievedAt,
|
||||
includeRawLinkset: false);
|
||||
|
||||
await database
|
||||
.GetCollection<BsonDocument>(MongoStorageDefaults.Collections.AdvisoryObservations)
|
||||
.InsertOneAsync(document);
|
||||
|
||||
var migration = new EnsureAdvisoryObservationsRawLinksetMigration();
|
||||
|
||||
await Assert.ThrowsAsync<InvalidOperationException>(
|
||||
() => migration.ApplyAsync(database, CancellationToken.None));
|
||||
}
|
||||
finally
|
||||
{
|
||||
await _fixture.Client.DropDatabaseAsync(databaseName);
|
||||
}
|
||||
}
|
||||
|
||||
private static BsonDocument BuildObservationDocument(
|
||||
string observationId,
|
||||
AdvisoryRawDocument rawDocument,
|
||||
ImmutableArray<string> canonicalAliases,
|
||||
ImmutableArray<string> canonicalPurls,
|
||||
ImmutableArray<string> canonicalCpes,
|
||||
ImmutableArray<RawReference> canonicalReferences,
|
||||
DateTimeOffset createdAt,
|
||||
bool includeRawLinkset,
|
||||
RawLinkset? rawLinkset = null)
|
||||
{
|
||||
var sourceDocument = new BsonDocument
|
||||
{
|
||||
{ "vendor", rawDocument.Source.Vendor },
|
||||
{ "stream", string.IsNullOrWhiteSpace(rawDocument.Source.Stream) ? rawDocument.Source.Connector : rawDocument.Source.Stream! },
|
||||
{ "api", rawDocument.Upstream.Provenance.TryGetValue("api", out var api) ? api : rawDocument.Source.Connector }
|
||||
};
|
||||
if (!string.IsNullOrWhiteSpace(rawDocument.Source.ConnectorVersion))
|
||||
{
|
||||
sourceDocument["collectorVersion"] = rawDocument.Source.ConnectorVersion;
|
||||
}
|
||||
|
||||
var signatureDocument = new BsonDocument
|
||||
{
|
||||
{ "present", rawDocument.Upstream.Signature.Present }
|
||||
};
|
||||
if (!string.IsNullOrWhiteSpace(rawDocument.Upstream.Signature.Format))
|
||||
{
|
||||
signatureDocument["format"] = rawDocument.Upstream.Signature.Format;
|
||||
}
|
||||
if (!string.IsNullOrWhiteSpace(rawDocument.Upstream.Signature.KeyId))
|
||||
{
|
||||
signatureDocument["keyId"] = rawDocument.Upstream.Signature.KeyId;
|
||||
}
|
||||
if (!string.IsNullOrWhiteSpace(rawDocument.Upstream.Signature.Signature))
|
||||
{
|
||||
signatureDocument["signature"] = rawDocument.Upstream.Signature.Signature;
|
||||
}
|
||||
|
||||
var upstreamDocument = new BsonDocument
|
||||
{
|
||||
{ "upstream_id", rawDocument.Upstream.UpstreamId },
|
||||
{ "document_version", rawDocument.Upstream.DocumentVersion },
|
||||
{ "fetchedAt", rawDocument.Upstream.RetrievedAt.UtcDateTime },
|
||||
{ "receivedAt", rawDocument.Upstream.RetrievedAt.UtcDateTime },
|
||||
{ "contentHash", rawDocument.Upstream.ContentHash },
|
||||
{ "signature", signatureDocument },
|
||||
{ "metadata", new BsonDocument(rawDocument.Upstream.Provenance) }
|
||||
};
|
||||
|
||||
var contentDocument = new BsonDocument
|
||||
{
|
||||
{ "format", rawDocument.Content.Format },
|
||||
{ "raw", BsonDocument.Parse(rawDocument.Content.Raw.GetRawText()) }
|
||||
};
|
||||
if (!string.IsNullOrWhiteSpace(rawDocument.Content.SpecVersion))
|
||||
{
|
||||
contentDocument["specVersion"] = rawDocument.Content.SpecVersion;
|
||||
}
|
||||
|
||||
var canonicalLinkset = new BsonDocument
|
||||
{
|
||||
{ "aliases", new BsonArray(canonicalAliases) },
|
||||
{ "purls", new BsonArray(canonicalPurls) },
|
||||
{ "cpes", new BsonArray(canonicalCpes) },
|
||||
{ "references", new BsonArray(canonicalReferences.Select(reference => new BsonDocument
|
||||
{
|
||||
{ "type", reference.Type },
|
||||
{ "url", reference.Url }
|
||||
})) }
|
||||
};
|
||||
|
||||
var document = new BsonDocument
|
||||
{
|
||||
{ "_id", observationId },
|
||||
{ "tenant", rawDocument.Tenant },
|
||||
{ "source", sourceDocument },
|
||||
{ "upstream", upstreamDocument },
|
||||
{ "content", contentDocument },
|
||||
{ "linkset", canonicalLinkset },
|
||||
{ "createdAt", createdAt.UtcDateTime },
|
||||
{ "attributes", new BsonDocument() }
|
||||
};
|
||||
|
||||
if (includeRawLinkset)
|
||||
{
|
||||
var actualRawLinkset = rawLinkset ?? throw new ArgumentNullException(nameof(rawLinkset));
|
||||
document["rawLinkset"] = new BsonDocument
|
||||
{
|
||||
{ "aliases", new BsonArray(actualRawLinkset.Aliases) },
|
||||
{ "purls", new BsonArray(actualRawLinkset.PackageUrls) },
|
||||
{ "cpes", new BsonArray(actualRawLinkset.Cpes) },
|
||||
{ "references", new BsonArray(actualRawLinkset.References.Select(reference => new BsonDocument
|
||||
{
|
||||
{ "type", reference.Type },
|
||||
{ "url", reference.Url },
|
||||
{ "source", reference.Source }
|
||||
})) },
|
||||
{ "reconciled_from", new BsonArray(actualRawLinkset.ReconciledFrom) },
|
||||
{ "notes", new BsonDocument(actualRawLinkset.Notes) }
|
||||
};
|
||||
}
|
||||
|
||||
return document;
|
||||
}
|
||||
|
||||
private static JsonElement ParseJsonElement(string json)
|
||||
{
|
||||
using var document = JsonDocument.Parse(json);
|
||||
return document.RootElement.Clone();
|
||||
}
|
||||
|
||||
private static RawLinkset BuildRawLinkset(RawIdentifiers identifiers, RawLinkset linkset)
|
||||
{
|
||||
var aliasBuilder = ImmutableArray.CreateBuilder<string>();
|
||||
|
||||
if (!string.IsNullOrWhiteSpace(identifiers.PrimaryId))
|
||||
{
|
||||
aliasBuilder.Add(identifiers.PrimaryId);
|
||||
}
|
||||
|
||||
if (!identifiers.Aliases.IsDefaultOrEmpty)
|
||||
{
|
||||
foreach (var alias in identifiers.Aliases)
|
||||
{
|
||||
if (!string.IsNullOrEmpty(alias))
|
||||
{
|
||||
aliasBuilder.Add(alias);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
if (!linkset.Aliases.IsDefaultOrEmpty)
|
||||
{
|
||||
foreach (var alias in linkset.Aliases)
|
||||
{
|
||||
if (!string.IsNullOrEmpty(alias))
|
||||
{
|
||||
aliasBuilder.Add(alias);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
static ImmutableArray<string> EnsureArray(ImmutableArray<string> values)
|
||||
=> values.IsDefault ? ImmutableArray<string>.Empty : values;
|
||||
|
||||
static ImmutableArray<RawReference> EnsureReferences(ImmutableArray<RawReference> values)
|
||||
=> values.IsDefault ? ImmutableArray<RawReference>.Empty : values;
|
||||
|
||||
return linkset with
|
||||
{
|
||||
Aliases = aliasBuilder.ToImmutable(),
|
||||
PackageUrls = EnsureArray(linkset.PackageUrls),
|
||||
Cpes = EnsureArray(linkset.Cpes),
|
||||
References = EnsureReferences(linkset.References),
|
||||
ReconciledFrom = EnsureArray(linkset.ReconciledFrom),
|
||||
Notes = linkset.Notes ?? ImmutableDictionary<string, string>.Empty
|
||||
};
|
||||
}
|
||||
}
|
||||
@@ -52,6 +52,18 @@ public sealed class AdvisoryObservationDocumentFactoryTests
|
||||
{
|
||||
new() { Type = "advisory", Url = "https://example.com" }
|
||||
}
|
||||
},
|
||||
RawLinkset = new AdvisoryObservationRawLinksetDocument
|
||||
{
|
||||
Aliases = new List<string> { "CVE-2025-1234", "cve-2025-1234" },
|
||||
PackageUrls = new List<string> { "pkg:generic/foo@1.0.0" },
|
||||
Cpes = new List<string> { "cpe:/a:vendor:product:1" },
|
||||
References = new List<AdvisoryObservationRawReferenceDocument>
|
||||
{
|
||||
new() { Type = "Advisory", Url = "https://example.com", Source = "vendor" }
|
||||
},
|
||||
ReconciledFrom = new List<string> { "source-a" },
|
||||
Notes = new Dictionary<string, string> { ["note-key"] = "note-value" }
|
||||
}
|
||||
};
|
||||
|
||||
@@ -64,5 +76,9 @@ public sealed class AdvisoryObservationDocumentFactoryTests
|
||||
Assert.Equal("CSAF", observation.Content.Format);
|
||||
Assert.True(observation.Content.Raw?["example"]?.GetValue<bool>());
|
||||
Assert.Equal("advisory", observation.Linkset.References[0].Type);
|
||||
Assert.Equal(new[] { "CVE-2025-1234", "cve-2025-1234" }, observation.RawLinkset.Aliases);
|
||||
Assert.Equal("Advisory", observation.RawLinkset.References[0].Type);
|
||||
Assert.Equal("vendor", observation.RawLinkset.References[0].Source);
|
||||
Assert.Equal("note-value", observation.RawLinkset.Notes["note-key"]);
|
||||
}
|
||||
}
|
||||
|
||||
@@ -2,6 +2,6 @@ If you are working on this file you need to read docs/modules/excititor/ARCHITEC
|
||||
# TASKS
|
||||
| Task | Owner(s) | Depends on | Notes |
|
||||
|---|---|---|---|
|
||||
|EXCITITOR-ATTEST-01-003 – Verification suite & observability|Team Excititor Attestation|EXCITITOR-ATTEST-01-002|DOING (2025-10-22) – Continuing implementation: build `IVexAttestationVerifier`, wire metrics/logging, and add regression tests. Draft plan in `EXCITITOR-ATTEST-01-003-plan.md` (2025-10-19) guides scope; updating with worknotes as progress lands.|
|
||||
|EXCITITOR-ATTEST-01-003 – Verification suite & observability|Team Excititor Attestation|EXCITITOR-ATTEST-01-002|DOING (2025-10-22) – Continuing implementation: build `IVexAttestationVerifier`, wire metrics/logging, and add regression tests. Draft plan in `EXCITITOR-ATTEST-01-003-plan.md` (2025-10-19) guides scope; updating with worknotes as progress lands.<br>2025-10-31: Verifier now tolerates duplicate source providers from AOC raw projections, downgrades offline Rekor verification to a degraded result, and enforces trusted signer registry checks with detailed diagnostics/tests.|
|
||||
|
||||
> Remark (2025-10-22): Added verifier implementation + metrics/tests; next steps include wiring into WebService/Worker flows and expanding negative-path coverage.
|
||||
|
||||
@@ -173,7 +173,11 @@ internal sealed class VexAttestationVerifier : IVexAttestationVerifier
|
||||
resultLabel = "degraded";
|
||||
}
|
||||
|
||||
diagnostics["signature.state"] = "present";
|
||||
if (rekorState is "offline" && resultLabel != "invalid")
|
||||
{
|
||||
resultLabel = "degraded";
|
||||
}
|
||||
|
||||
return BuildResult(true);
|
||||
}
|
||||
catch (Exception ex)
|
||||
@@ -206,6 +210,103 @@ internal sealed class VexAttestationVerifier : IVexAttestationVerifier
|
||||
}
|
||||
}
|
||||
|
||||
private async ValueTask<bool> VerifySignaturesAsync(
|
||||
ReadOnlyMemory<byte> payloadBytes,
|
||||
IReadOnlyList<DsseSignature> signatures,
|
||||
ImmutableDictionary<string, string>.Builder diagnostics,
|
||||
CancellationToken cancellationToken)
|
||||
{
|
||||
if (signatures is null || signatures.Count == 0)
|
||||
{
|
||||
diagnostics["signature.state"] = "missing";
|
||||
return false;
|
||||
}
|
||||
|
||||
if (_trustedSigners.Count == 0)
|
||||
{
|
||||
diagnostics["signature.state"] = "skipped";
|
||||
diagnostics["signature.reason"] = "trust_not_configured";
|
||||
return true;
|
||||
}
|
||||
|
||||
if (_cryptoRegistry is null)
|
||||
{
|
||||
diagnostics["signature.state"] = "error";
|
||||
diagnostics["signature.reason"] = "registry_unavailable";
|
||||
return false;
|
||||
}
|
||||
|
||||
foreach (var signature in signatures)
|
||||
{
|
||||
if (string.IsNullOrWhiteSpace(signature.Signature))
|
||||
{
|
||||
diagnostics["signature.state"] = "error";
|
||||
diagnostics["signature.reason"] = "empty_signature";
|
||||
return false;
|
||||
}
|
||||
|
||||
if (string.IsNullOrWhiteSpace(signature.KeyId))
|
||||
{
|
||||
diagnostics["signature.state"] = "error";
|
||||
diagnostics["signature.reason"] = "missing_key_id";
|
||||
return false;
|
||||
}
|
||||
|
||||
if (!_trustedSigners.TryGetValue(signature.KeyId, out var signerOptions))
|
||||
{
|
||||
diagnostics["signature.state"] = "error";
|
||||
diagnostics["signature.reason"] = "untrusted_key";
|
||||
diagnostics["signature.keyId"] = signature.KeyId;
|
||||
return false;
|
||||
}
|
||||
|
||||
byte[] signatureBytes;
|
||||
try
|
||||
{
|
||||
signatureBytes = Convert.FromBase64String(signature.Signature);
|
||||
}
|
||||
catch (FormatException)
|
||||
{
|
||||
diagnostics["signature.state"] = "error";
|
||||
diagnostics["signature.reason"] = "invalid_signature_encoding";
|
||||
diagnostics["signature.keyId"] = signature.KeyId;
|
||||
return false;
|
||||
}
|
||||
|
||||
CryptoSignerResolution resolution;
|
||||
try
|
||||
{
|
||||
resolution = _cryptoRegistry.ResolveSigner(
|
||||
CryptoCapability.Verification,
|
||||
signerOptions.Algorithm,
|
||||
new CryptoKeyReference(signerOptions.KeyReference, signerOptions.ProviderHint));
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
diagnostics["signature.state"] = "error";
|
||||
diagnostics["signature.reason"] = "resolution_failed";
|
||||
diagnostics["signature.error"] = ex.GetType().Name;
|
||||
diagnostics["signature.keyId"] = signature.KeyId;
|
||||
return false;
|
||||
}
|
||||
|
||||
var verified = await resolution.Signer
|
||||
.VerifyAsync(payloadBytes, signatureBytes, cancellationToken)
|
||||
.ConfigureAwait(false);
|
||||
|
||||
if (!verified)
|
||||
{
|
||||
diagnostics["signature.state"] = "error";
|
||||
diagnostics["signature.reason"] = "verification_failed";
|
||||
diagnostics["signature.keyId"] = signature.KeyId;
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
diagnostics["signature.state"] = "verified";
|
||||
return true;
|
||||
}
|
||||
|
||||
private static bool TryDeserializeEnvelope(
|
||||
string envelopeJson,
|
||||
out DsseEnvelope envelope,
|
||||
@@ -473,17 +574,18 @@ internal sealed class VexAttestationVerifier : IVexAttestationVerifier
|
||||
|
||||
private static bool SetEquals(IReadOnlyCollection<string>? left, ImmutableArray<string> right)
|
||||
{
|
||||
if (left is null)
|
||||
if (left is null || left.Count == 0)
|
||||
{
|
||||
return right.IsDefaultOrEmpty;
|
||||
return right.IsDefaultOrEmpty || right.Length == 0;
|
||||
}
|
||||
|
||||
if (left.Count != right.Length)
|
||||
if (right.IsDefaultOrEmpty)
|
||||
{
|
||||
return false;
|
||||
}
|
||||
|
||||
var leftSet = new HashSet<string>(left, StringComparer.Ordinal);
|
||||
return right.All(leftSet.Contains);
|
||||
var rightSet = new HashSet<string>(right, StringComparer.Ordinal);
|
||||
return leftSet.SetEquals(rightSet);
|
||||
}
|
||||
}
|
||||
|
||||
@@ -1,6 +1,9 @@
|
||||
using System.Collections.Immutable;
|
||||
using System.Text;
|
||||
using Microsoft.Extensions.Logging.Abstractions;
|
||||
using Microsoft.Extensions.Options;
|
||||
using Microsoft.IdentityModel.Tokens;
|
||||
using StellaOps.Cryptography;
|
||||
using StellaOps.Excititor.Attestation.Dsse;
|
||||
using StellaOps.Excititor.Attestation.Signing;
|
||||
using StellaOps.Excititor.Attestation.Transparency;
|
||||
@@ -65,6 +68,7 @@ public sealed class VexAttestationVerifierTests : IDisposable
|
||||
|
||||
Assert.True(verification.IsValid);
|
||||
Assert.Equal("offline", verification.Diagnostics["rekor.state"]);
|
||||
Assert.Equal("degraded", verification.Diagnostics["result"]);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
@@ -106,7 +110,80 @@ public sealed class VexAttestationVerifierTests : IDisposable
|
||||
Assert.Equal("invalid", verification.Diagnostics["result"]);
|
||||
}
|
||||
|
||||
private async Task<(VexAttestationRequest Request, VexAttestationMetadata Metadata, string Envelope)> CreateSignedAttestationAsync(bool includeRekor = false)
|
||||
[Fact]
|
||||
public async Task VerifyAsync_HandlesDuplicateSourceProviders()
|
||||
{
|
||||
var (request, metadata, envelope) = await CreateSignedAttestationAsync(
|
||||
includeRekor: false,
|
||||
sourceProviders: ImmutableArray.Create("provider-a", "provider-a"));
|
||||
|
||||
var normalizedRequest = request with { SourceProviders = ImmutableArray.Create("provider-a") };
|
||||
var verifier = CreateVerifier(options => options.RequireTransparencyLog = false);
|
||||
|
||||
var verification = await verifier.VerifyAsync(
|
||||
new VexAttestationVerificationRequest(normalizedRequest, metadata, envelope),
|
||||
CancellationToken.None);
|
||||
|
||||
Assert.True(verification.IsValid);
|
||||
Assert.Equal("valid", verification.Diagnostics["result"]);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task VerifyAsync_ReturnsValid_WhenTrustedSignerConfigured()
|
||||
{
|
||||
var (request, metadata, envelope) = await CreateSignedAttestationAsync(includeRekor: false);
|
||||
var registry = new StubCryptoProviderRegistry(success: true);
|
||||
var verifier = CreateVerifier(options =>
|
||||
{
|
||||
options.RequireTransparencyLog = false;
|
||||
options.RequireSignatureVerification = true;
|
||||
options.TrustedSigners = ImmutableDictionary<string, VexAttestationVerificationOptions.TrustedSignerOptions>.Empty.Add(
|
||||
"key",
|
||||
new VexAttestationVerificationOptions.TrustedSignerOptions
|
||||
{
|
||||
Algorithm = StubCryptoProviderRegistry.Algorithm,
|
||||
KeyReference = StubCryptoProviderRegistry.KeyReference
|
||||
});
|
||||
}, transparency: null, registry: registry);
|
||||
|
||||
var verification = await verifier.VerifyAsync(
|
||||
new VexAttestationVerificationRequest(request, metadata, envelope),
|
||||
CancellationToken.None);
|
||||
|
||||
Assert.True(verification.IsValid);
|
||||
Assert.Equal("verified", verification.Diagnostics["signature.state"]);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task VerifyAsync_ReturnsInvalid_WhenSignatureFailsAndRequired()
|
||||
{
|
||||
var (request, metadata, envelope) = await CreateSignedAttestationAsync(includeRekor: false);
|
||||
var registry = new StubCryptoProviderRegistry(success: false);
|
||||
var verifier = CreateVerifier(options =>
|
||||
{
|
||||
options.RequireTransparencyLog = false;
|
||||
options.RequireSignatureVerification = true;
|
||||
options.TrustedSigners = ImmutableDictionary<string, VexAttestationVerificationOptions.TrustedSignerOptions>.Empty.Add(
|
||||
"key",
|
||||
new VexAttestationVerificationOptions.TrustedSignerOptions
|
||||
{
|
||||
Algorithm = StubCryptoProviderRegistry.Algorithm,
|
||||
KeyReference = StubCryptoProviderRegistry.KeyReference
|
||||
});
|
||||
}, transparency: null, registry: registry);
|
||||
|
||||
var verification = await verifier.VerifyAsync(
|
||||
new VexAttestationVerificationRequest(request, metadata, envelope),
|
||||
CancellationToken.None);
|
||||
|
||||
Assert.False(verification.IsValid);
|
||||
Assert.Equal("error", verification.Diagnostics["signature.state"]);
|
||||
Assert.Equal("verification_failed", verification.Diagnostics["signature.reason"]);
|
||||
}
|
||||
|
||||
private async Task<(VexAttestationRequest Request, VexAttestationMetadata Metadata, string Envelope)> CreateSignedAttestationAsync(
|
||||
bool includeRekor = false,
|
||||
ImmutableArray<string>? sourceProviders = null)
|
||||
{
|
||||
var signer = new FakeSigner();
|
||||
var builder = new VexDsseBuilder(signer, NullLogger<VexDsseBuilder>.Instance);
|
||||
@@ -115,13 +192,14 @@ public sealed class VexAttestationVerifierTests : IDisposable
|
||||
var verifier = CreateVerifier(options => options.RequireTransparencyLog = false);
|
||||
var client = new VexAttestationClient(builder, options, NullLogger<VexAttestationClient>.Instance, verifier, transparency);
|
||||
|
||||
var providers = sourceProviders ?? ImmutableArray.Create("provider-a");
|
||||
var request = new VexAttestationRequest(
|
||||
ExportId: "exports/unit-test",
|
||||
QuerySignature: new VexQuerySignature("filters"),
|
||||
Artifact: new VexContentAddress("sha256", "cafebabe"),
|
||||
Format: VexExportFormat.Json,
|
||||
CreatedAt: DateTimeOffset.UtcNow,
|
||||
SourceProviders: ImmutableArray.Create("provider-a"),
|
||||
SourceProviders: providers,
|
||||
Metadata: ImmutableDictionary<string, string>.Empty);
|
||||
|
||||
var response = await client.SignAsync(request, CancellationToken.None);
|
||||
@@ -129,7 +207,10 @@ public sealed class VexAttestationVerifierTests : IDisposable
|
||||
return (request, response.Attestation, envelope);
|
||||
}
|
||||
|
||||
private VexAttestationVerifier CreateVerifier(Action<VexAttestationVerificationOptions>? configureOptions = null, ITransparencyLogClient? transparency = null)
|
||||
private VexAttestationVerifier CreateVerifier(
|
||||
Action<VexAttestationVerificationOptions>? configureOptions = null,
|
||||
ITransparencyLogClient? transparency = null,
|
||||
ICryptoProviderRegistry? registry = null)
|
||||
{
|
||||
var options = new VexAttestationVerificationOptions();
|
||||
configureOptions?.Invoke(options);
|
||||
@@ -137,7 +218,8 @@ public sealed class VexAttestationVerifierTests : IDisposable
|
||||
NullLogger<VexAttestationVerifier>.Instance,
|
||||
transparency,
|
||||
Options.Create(options),
|
||||
_metrics);
|
||||
_metrics,
|
||||
registry);
|
||||
}
|
||||
|
||||
public void Dispose()
|
||||
@@ -147,8 +229,10 @@ public sealed class VexAttestationVerifierTests : IDisposable
|
||||
|
||||
private sealed class FakeSigner : IVexSigner
|
||||
{
|
||||
internal static readonly string SignatureBase64 = Convert.ToBase64String(Encoding.UTF8.GetBytes("signature"));
|
||||
|
||||
public ValueTask<VexSignedPayload> SignAsync(ReadOnlyMemory<byte> payload, CancellationToken cancellationToken)
|
||||
=> ValueTask.FromResult(new VexSignedPayload("signature", "key"));
|
||||
=> ValueTask.FromResult(new VexSignedPayload(SignatureBase64, "key"));
|
||||
}
|
||||
|
||||
private sealed class FakeTransparencyLogClient : ITransparencyLogClient
|
||||
@@ -168,4 +252,70 @@ public sealed class VexAttestationVerifierTests : IDisposable
|
||||
public ValueTask<bool> VerifyAsync(string entryLocation, CancellationToken cancellationToken)
|
||||
=> throw new HttpRequestException("rekor unavailable");
|
||||
}
|
||||
|
||||
private sealed class StubCryptoProviderRegistry : ICryptoProviderRegistry
|
||||
{
|
||||
public const string Algorithm = "ed25519";
|
||||
public const string KeyReference = "stub-key";
|
||||
|
||||
private readonly StubCryptoSigner _signer;
|
||||
private readonly IReadOnlyCollection<ICryptoProvider> _providers = Array.Empty<ICryptoProvider>();
|
||||
|
||||
public StubCryptoProviderRegistry(bool success)
|
||||
{
|
||||
_signer = new StubCryptoSigner("key", Algorithm, success);
|
||||
}
|
||||
|
||||
public IReadOnlyCollection<ICryptoProvider> Providers => _providers;
|
||||
|
||||
public bool TryResolve(string preferredProvider, out ICryptoProvider provider)
|
||||
{
|
||||
provider = null!;
|
||||
return false;
|
||||
}
|
||||
|
||||
public ICryptoProvider ResolveOrThrow(CryptoCapability capability, string algorithmId)
|
||||
=> throw new NotSupportedException();
|
||||
|
||||
public CryptoSignerResolution ResolveSigner(
|
||||
CryptoCapability capability,
|
||||
string algorithmId,
|
||||
CryptoKeyReference keyReference,
|
||||
string? preferredProvider = null)
|
||||
{
|
||||
if (!string.Equals(keyReference.KeyId, _signer.KeyId, StringComparison.Ordinal))
|
||||
{
|
||||
throw new InvalidOperationException($"Unknown key '{keyReference.KeyId}'.");
|
||||
}
|
||||
|
||||
return new CryptoSignerResolution(_signer, "stub");
|
||||
}
|
||||
}
|
||||
|
||||
private sealed class StubCryptoSigner : ICryptoSigner
|
||||
{
|
||||
private readonly bool _success;
|
||||
private readonly byte[] _expectedSignature;
|
||||
|
||||
public StubCryptoSigner(string keyId, string algorithmId, bool success)
|
||||
{
|
||||
KeyId = keyId;
|
||||
AlgorithmId = algorithmId;
|
||||
_success = success;
|
||||
_expectedSignature = Convert.FromBase64String(FakeSigner.SignatureBase64);
|
||||
}
|
||||
|
||||
public string KeyId { get; }
|
||||
|
||||
public string AlgorithmId { get; }
|
||||
|
||||
public ValueTask<byte[]> SignAsync(ReadOnlyMemory<byte> data, CancellationToken cancellationToken = default)
|
||||
=> throw new NotSupportedException();
|
||||
|
||||
public ValueTask<bool> VerifyAsync(ReadOnlyMemory<byte> data, ReadOnlyMemory<byte> signature, CancellationToken cancellationToken = default)
|
||||
=> ValueTask.FromResult(_success && signature.Span.SequenceEqual(_expectedSignature.Span));
|
||||
|
||||
public JsonWebKey ExportPublicJsonWebKey()
|
||||
=> new JsonWebKey();
|
||||
}
|
||||
}
|
||||
|
||||
@@ -7,6 +7,7 @@
|
||||
> 2025-10-26: Blocked while bootstrapping DSL parser/evaluator; remaining grammar coverage (profile keywords, condition parsing) and rule evaluation semantics still pending to satisfy acceptance tests.
|
||||
| POLICY-ENGINE-20-003 | TODO | Policy Guild, Concelier Core Guild, Excititor Core Guild | POLICY-ENGINE-20-001, CONCELIER-POLICY-20-002, EXCITITOR-POLICY-20-002 | Implement selection joiners resolving SBOM↔advisory↔VEX tuples using linksets and PURL equivalence tables, with deterministic batching. | Joiners fetch correct candidate sets in integration tests; batching meets memory targets; explain traces list input provenance. |
|
||||
> 2025-10-26: Scheduler DTO contracts for runs/diffs/explains available (`src/Scheduler/__Libraries/StellaOps.Scheduler.Models/docs/SCHED-MODELS-20-001-POLICY-RUNS.md`); consume `PolicyRunRequest/Status/DiffSummary` from samples under `samples/api/scheduler/`.
|
||||
> 2025-10-31: Raw Concelier observations expose `rawLinkset`; update joiners/tests to consume it and align rollout/backfill per `docs/dev/raw-linkset-backfill-plan.md`.
|
||||
| POLICY-ENGINE-20-004 | TODO | Policy Guild, Platform Storage Guild | POLICY-ENGINE-20-003, CONCELIER-POLICY-20-003, EXCITITOR-POLICY-20-003 | Ship materialization writer that upserts into `effective_finding_{policyId}` with append-only history, tenant scoping, and trace references. | Writes restricted to Policy Engine identity; idempotent upserts proven via tests; collections indexed per design and docs updated. |
|
||||
| POLICY-ENGINE-20-005 | TODO | Policy Guild, Security Engineering | POLICY-ENGINE-20-002 | Enforce determinism guard banning wall-clock, RNG, and network usage during evaluation via static analysis + runtime sandbox. | Guard blocks forbidden APIs in unit/integration tests; violations emit `ERR_POL_004`; CI analyzer wired. |
|
||||
| POLICY-ENGINE-20-006 | TODO | Policy Guild, Scheduler Worker Guild | POLICY-ENGINE-20-003, POLICY-ENGINE-20-004, SCHED-WORKER-20-301 | Implement incremental orchestrator reacting to advisory/vex/SBOM change streams and scheduling partial policy re-evaluations. | Change stream listeners enqueue affected tuples with dedupe; orchestrator meets 5 min SLA in perf tests; metrics exposed (`policy_run_seconds`). |
|
||||
|
||||
Reference in New Issue
Block a user