Align AOC tasks for Excititor and Concelier

This commit is contained in:
master
2025-10-31 18:50:15 +02:00
committed by root
parent 9e6d9fbae8
commit 8da4e12a90
334 changed files with 35528 additions and 34546 deletions

View File

@@ -14,8 +14,8 @@ Document follow-up actions for CONCELIER-CORE-AOC-19-004 as we unwind the final
## Next Steps
1. Introduce a raw linkset projection alongside the existing canonical mapper so Policy Engine can choose which flavour to consume.
2. Update observation factory/query tests to assert duplicate handling and ordering with the relaxed projection.
1. Introduce a raw linkset projection alongside the existing canonical mapper so Policy Engine can choose which flavour to consume. ✅ 2025-10-31: `AdvisoryObservation` now surfaces `RawLinkset`; Mongo documents store both canonical & raw shapes; tests/goldens updated.
2. Update observation factory/query tests to assert duplicate handling and ordering with the relaxed projection. ✅ 2025-10-31.
3. Refresh docs (`docs/ingestion/aggregation-only-contract.md`) once behaviour lands to explain the “raw vs canonical linkset” split.
4. Coordinate with Policy Guild to validate consumers against the new raw projection before flipping defaults.
4. Coordinate with Policy Guild to validate consumers against the new raw projection before flipping defaults. ↺ Ongoing — see action items in `docs/dev/raw-linkset-backfill-plan.md` (2025-10-31 handshake with POLICY-ENGINE-20-003 owners).

View File

@@ -0,0 +1,56 @@
# Raw Linkset Backfill & Adoption Plan
_Last updated: 2025-10-31_
Owners: Concelier Storage Guild, DevOps Guild, Policy Guild
## Context
- Concelier observations now emit both a **canonical linkset** (deduped, normalised identifiers) and a **raw linkset** (`rawLinkset`) that preserves upstream ordering, duplicates, and original pointer metadata.
- Existing `concelier.advisory_observations` documents created before 2025-10-31 do **not** contain the `rawLinkset` field.
- Policy Engine selection joiners (`POLICY-ENGINE-20-003`) will switch to the raw projection once backfill completes and consumers validate fixtures.
## Objectives
1. Populate `rawLinkset` for historical observations across online clusters and Offline Kit bundles without breaking append-only guarantees.
2. Provide migration scripts + runbook so operators can rehearse in staging (and air-gapped deployments) before production rollout.
3. Unblock Policy Engine adoption by guaranteeing dual projections exist for all tenants.
## Deliverables
- [ ] **Migration script** (`20251104_advisory_observations_raw_linkset_backfill.csx`)
- Iterates observations lacking `rawLinkset`
- Rehydrates raw document via existing snapshot (or cached DTO)
- Reuses `AdvisoryObservationFactory.CreateRawLinkset`
- Writes using `$set` with optimistic retry; preserves `updatedAt` via `setOnInsert`
- [ ] **Offline Kit updater** (extend `ops/offline-kit/scripts/export_offline_bundle.py`) to patch bundles in-place
- [ ] **Runbook** covering:
- Pre-check query: `db.concelier.advisory_observations.countDocuments({ rawLinkset: { $exists: false } })`
- Backup procedure (`mongodump` or snapshot requirement)
- Dry-run mode limiting batches by tenant
- Metrics/telemetry expectations (`concelier.migrations.documents_processed_total`)
- Rollback (no-op because field addition; note to retain snapshot for verification)
- [ ] **Fixture updates** ensuring storage/CLI/Policy tests include `rawLinkset`
- [ ] **Policy Engine follow-up** to flip joiners once `rawLinkset` population reaches 100% (tracked via metrics).
## Timeline
| Date (UTC) | Milestone | Notes |
|------------|-----------|-------|
| 2025-10-31 | Handshake w/ Policy | Agreement to consume `rawLinkset`; this document created. |
| 2025-11-01 | Draft migration script | Validate against staging dataset snapshots. |
| 2025-11-04 | Storage task CONCELIER-STORE-AOC-19-005 due | Deliver script + runbook for review. |
| 2025-11-06 | Staging backfill rehearsal | Target < 30 min runtime on 5M observations. |
| 2025-11-08 | Policy fixtures updated | POL engine branch consumes `rawLinkset`. |
| 2025-11-11 | Production rollout window | Pending DevOps sign-off after rehearsals. |
## Open Questions
- Do we need archival of the canonical-only projection for backwards compatibility exports? (Policy to confirm.)
- Offline Kit delta: should we regenerate entire bundle or ship incremental patch? (DevOps reviewing.)
- Metrics: add `raw_linkset_missing_total` counter to detect regressions post-backfill?
## Next Actions
- [ ] Concelier Storage Guild: prototype migration script, share for review (`2025-11-01`).
- [ ] DevOps Guild: schedule staging rehearsal + update `docs/deploy/containers.md` with new runbook section.
- [ ] Policy Guild: prepare feature flag/branch to switch joiners once metrics show zero missing `rawLinkset`.

View File

@@ -16,7 +16,7 @@ This file describe implementation of Stella Ops (docs/README.md). Implementation
| Sprint 19 | Aggregation-Only Contract Enforcement | src/Concelier/__Libraries/StellaOps.Concelier.Core/TASKS.md | TODO | Concelier Core Guild | CONCELIER-CORE-AOC-19-001 | Implement AOC repository guard rejecting forbidden fields. |
| Sprint 19 | Aggregation-Only Contract Enforcement | src/Concelier/__Libraries/StellaOps.Concelier.Core/TASKS.md | TODO | Concelier Core Guild | CONCELIER-CORE-AOC-19-002 | Deliver deterministic linkset extraction for advisories. |
| Sprint 19 | Aggregation-Only Contract Enforcement | src/Concelier/__Libraries/StellaOps.Concelier.Core/TASKS.md | TODO | Concelier Core Guild | CONCELIER-CORE-AOC-19-003 | Enforce idempotent append-only upsert with supersedes pointers. |
| Sprint 19 | Aggregation-Only Contract Enforcement | src/Concelier/__Libraries/StellaOps.Concelier.Core/TASKS.md | TODO | Concelier Core Guild | CONCELIER-CORE-AOC-19-004 | Remove ingestion normalization; defer derived logic to Policy Engine. |
| Sprint 19 | Aggregation-Only Contract Enforcement | src/Concelier/__Libraries/StellaOps.Concelier.Core/TASKS.md | DOING (2025-10-28) | Concelier Core Guild | CONCELIER-CORE-AOC-19-004 | Remove ingestion normalization; defer derived logic to Policy Engine. |
| Sprint 19 | Aggregation-Only Contract Enforcement | src/Concelier/__Libraries/StellaOps.Concelier.Core/TASKS.md | TODO | Concelier Core Guild | CONCELIER-CORE-AOC-19-013 | Extend smoke coverage to validate tenant-scoped Authority tokens and cross-tenant rejection. |
| Sprint 19 | Aggregation-Only Contract Enforcement | src/Concelier/__Libraries/StellaOps.Concelier.Storage.Mongo/TASKS.md | TODO | Concelier Storage Guild | CONCELIER-STORE-AOC-19-001 | Add Mongo schema validator for `advisory_raw`. |
| Sprint 19 | Aggregation-Only Contract Enforcement | src/Concelier/__Libraries/StellaOps.Concelier.Storage.Mongo/TASKS.md | TODO | Concelier Storage Guild | CONCELIER-STORE-AOC-19-002 | Create idempotency unique index backed by migration scripts. |

View File

@@ -25,7 +25,7 @@
| 3 | Provenance is mandatory | `source.*`, `upstream.*`, and `signature` metadata must be present; missing provenance triggers `ERR_AOC_004`. | Schema validator, guard, CLI verifier. |
| 4 | Idempotent upserts | Writes keyed by `(vendor, upstream_id, content_hash)` either no-op or insert a new revision with `supersedes`. Duplicate hashes map to the same document. | Repository guard, storage unique index, CI smoke tests. |
| 5 | Append-only revisions | Updates create a new document with `supersedes` pointer; no in-place mutation of content. | Mongo schema (`supersedes` format), guard, data migration scripts. |
| 6 | Linkset only | Ingestion may compute link hints (`purls`, `cpes`, IDs) to accelerate joins, but must not transform or infer severity or policy. | Linkset builders reviewed via fixtures and analyzers. |
| 6 | Linkset only | Ingestion may compute link hints (`purls`, `cpes`, IDs) to accelerate joins, but must not transform or infer severity or policy. Observations now persist both canonical linksets (for indexed queries) and raw linksets (preserving upstream order/duplicates) so downstream policy can decide how to normalise. | Linkset builders reviewed via fixtures/analyzers; raw-vs-canonical parity covered by observation fixtures. |
| 7 | Policy-only effective findings | Only Policy Engine identities can write `effective_finding_*`; ingestion callers receive `ERR_AOC_006` if they attempt it. | Authority scopes, Policy Engine guard. |
| 8 | Schema safety | Unknown top-level keys reject with `ERR_AOC_007`; timestamps use ISO 8601 UTC strings; tenant is required. | Mongo validator, JSON schema tests. |
| 9 | Clock discipline | Collectors stamp `fetched_at` and `received_at` monotonically per batch to support reproducibility windows. | Collector contracts, QA fixtures. |

View File

@@ -335,6 +335,7 @@ Events are emitted via NATS (primary) and Redis Stream (fallback). Consumers ack
content: { format, specVersion, raw, metadata? },
identifiers: { cve?, ghsa?, vendorIds[], aliases[] },
linkset: { purls[], cpes[], aliases[], references[], reconciledFrom[] },
rawLinkset: { aliases[], purls[], cpes[], references[], reconciledFrom[], notes? },
supersedes?: "prevObservationId",
createdAt,
attributes?: object

View File

@@ -22,6 +22,7 @@ internal sealed class AdvisoryObservationFactory : IAdvisoryObservationFactory
var upstream = CreateUpstream(rawDocument.Upstream);
var content = CreateContent(rawDocument.Content);
var linkset = CreateLinkset(rawDocument.Identifiers, rawDocument.Linkset);
var rawLinkset = CreateRawLinkset(rawDocument.Identifiers, rawDocument.Linkset);
var attributes = CreateAttributes(rawDocument);
var createdAt = (observedAt ?? rawDocument.Upstream.RetrievedAt).ToUniversalTime();
@@ -33,6 +34,7 @@ internal sealed class AdvisoryObservationFactory : IAdvisoryObservationFactory
upstream: upstream,
content: content,
linkset: linkset,
rawLinkset: rawLinkset,
createdAt: createdAt,
attributes: attributes);
}
@@ -120,6 +122,54 @@ internal sealed class AdvisoryObservationFactory : IAdvisoryObservationFactory
return new AdvisoryObservationLinkset(aliases, purls, cpes, references);
}
private static RawLinkset CreateRawLinkset(RawIdentifiers identifiers, RawLinkset linkset)
{
var aliasBuilder = ImmutableArray.CreateBuilder<string>();
if (!string.IsNullOrWhiteSpace(identifiers.PrimaryId))
{
aliasBuilder.Add(identifiers.PrimaryId);
}
if (!identifiers.Aliases.IsDefaultOrEmpty)
{
foreach (var alias in identifiers.Aliases)
{
if (!string.IsNullOrEmpty(alias))
{
aliasBuilder.Add(alias);
}
}
}
if (!linkset.Aliases.IsDefaultOrEmpty)
{
foreach (var alias in linkset.Aliases)
{
if (!string.IsNullOrEmpty(alias))
{
aliasBuilder.Add(alias);
}
}
}
static ImmutableArray<string> EnsureArray(ImmutableArray<string> values)
=> values.IsDefault ? ImmutableArray<string>.Empty : values;
static ImmutableArray<RawReference> EnsureReferences(ImmutableArray<RawReference> values)
=> values.IsDefault ? ImmutableArray<RawReference>.Empty : values;
return linkset with
{
Aliases = aliasBuilder.ToImmutable(),
PackageUrls = EnsureArray(linkset.PackageUrls),
Cpes = EnsureArray(linkset.Cpes),
References = EnsureReferences(linkset.References),
ReconciledFrom = EnsureArray(linkset.ReconciledFrom),
Notes = linkset.Notes ?? ImmutableDictionary<string, string>.Empty
};
}
private static IEnumerable<string> NormalizeAliases(RawIdentifiers identifiers, RawLinkset linkset)
{
var aliases = new HashSet<string>(StringComparer.OrdinalIgnoreCase);

View File

@@ -1,6 +1,7 @@
using System.Collections.Immutable;
using System.Text.Json;
using System.Text.Json.Nodes;
using StellaOps.Concelier.RawModels;
namespace StellaOps.Concelier.Models.Observations;
@@ -13,6 +14,7 @@ public sealed record AdvisoryObservation
AdvisoryObservationUpstream upstream,
AdvisoryObservationContent content,
AdvisoryObservationLinkset linkset,
RawLinkset rawLinkset,
DateTimeOffset createdAt,
ImmutableDictionary<string, string>? attributes = null)
{
@@ -22,6 +24,7 @@ public sealed record AdvisoryObservation
Upstream = upstream ?? throw new ArgumentNullException(nameof(upstream));
Content = content ?? throw new ArgumentNullException(nameof(content));
Linkset = linkset ?? throw new ArgumentNullException(nameof(linkset));
RawLinkset = SanitizeRawLinkset(rawLinkset);
CreatedAt = createdAt.ToUniversalTime();
Attributes = NormalizeAttributes(attributes);
}
@@ -38,6 +41,8 @@ public sealed record AdvisoryObservation
public AdvisoryObservationLinkset Linkset { get; }
public RawLinkset RawLinkset { get; }
public DateTimeOffset CreatedAt { get; }
public ImmutableDictionary<string, string> Attributes { get; }
@@ -62,6 +67,54 @@ public sealed record AdvisoryObservation
return builder.ToImmutable();
}
private static RawLinkset SanitizeRawLinkset(RawLinkset? rawLinkset)
{
if (rawLinkset is null)
{
return new RawLinkset();
}
static ImmutableArray<string> SanitizeStrings(ImmutableArray<string> values)
{
if (values.IsDefault)
{
return ImmutableArray<string>.Empty;
}
return values;
}
static ImmutableArray<RawReference> SanitizeReferences(ImmutableArray<RawReference> references)
{
if (references.IsDefault)
{
return ImmutableArray<RawReference>.Empty;
}
return references;
}
static ImmutableDictionary<string, string> SanitizeNotes(ImmutableDictionary<string, string>? notes)
{
if (notes is null || notes.Count == 0)
{
return ImmutableDictionary<string, string>.Empty;
}
return notes;
}
return rawLinkset with
{
Aliases = SanitizeStrings(rawLinkset.Aliases),
PackageUrls = SanitizeStrings(rawLinkset.PackageUrls),
Cpes = SanitizeStrings(rawLinkset.Cpes),
References = SanitizeReferences(rawLinkset.References),
ReconciledFrom = SanitizeStrings(rawLinkset.ReconciledFrom),
Notes = SanitizeNotes(rawLinkset.Notes)
};
}
}
public sealed record AdvisoryObservationSource

View File

@@ -9,4 +9,7 @@
<ItemGroup>
<PackageReference Include="Microsoft.Extensions.Logging.Abstractions" Version="10.0.0-rc.2.25502.107" />
</ItemGroup>
<ItemGroup>
<ProjectReference Include="..\StellaOps.Concelier.RawModels\StellaOps.Concelier.RawModels.csproj" />
</ItemGroup>
</Project>

View File

@@ -26,6 +26,7 @@ This module owns the persistent shape of Concelier's MongoDB database. Upgrades
| `20251028_advisory_raw_idempotency_index` | Applies compound unique index on `(source.vendor, upstream.upstream_id, upstream.content_hash, tenant)` after verifying no duplicates exist. |
| `20251028_advisory_supersedes_backfill` | Renames legacy `advisory` collection to a read-only backup view and backfills `supersedes` chains across `advisory_raw`. |
| `20251028_advisory_raw_validator` | Applies Aggregation-Only Contract JSON schema validator to the `advisory_raw` collection with configurable enforcement level. |
| `20251104_advisory_observations_raw_linkset` | Backfills `rawLinkset` on `advisory_observations` using stored `advisory_raw` documents so canonical and raw projections co-exist for downstream policy joins. |
## Operator Runbook

View File

@@ -0,0 +1,442 @@
using System;
using System.Collections.Generic;
using System.Collections.Immutable;
using System.Globalization;
using System.Linq;
using System.Text.Json;
using System.Threading;
using System.Threading.Tasks;
using MongoDB.Bson;
using MongoDB.Bson.IO;
using MongoDB.Driver;
using StellaOps.Concelier.RawModels;
namespace StellaOps.Concelier.Storage.Mongo.Migrations;
/// <summary>
/// Backfills the raw linkset projection on advisory observations so downstream services
/// can rely on both canonical and raw linkset shapes.
/// </summary>
public sealed class EnsureAdvisoryObservationsRawLinksetMigration : IMongoMigration
{
private const string MigrationId = "20251104_advisory_observations_raw_linkset";
private const int BulkBatchSize = 500;
public string Id => MigrationId;
public string Description => "Populate rawLinkset field for advisory observations using stored advisory_raw documents.";
public async Task ApplyAsync(IMongoDatabase database, CancellationToken cancellationToken)
{
ArgumentNullException.ThrowIfNull(database);
var observations = database.GetCollection<BsonDocument>(MongoStorageDefaults.Collections.AdvisoryObservations);
var rawCollection = database.GetCollection<BsonDocument>(MongoStorageDefaults.Collections.AdvisoryRaw);
var filter = Builders<BsonDocument>.Filter.Exists("rawLinkset", false) |
Builders<BsonDocument>.Filter.Type("rawLinkset", BsonType.Null);
using var cursor = await observations
.Find(filter)
.ToCursorAsync(cancellationToken)
.ConfigureAwait(false);
var updates = new List<WriteModel<BsonDocument>>(BulkBatchSize);
var missingRawDocuments = new List<string>();
while (await cursor.MoveNextAsync(cancellationToken).ConfigureAwait(false))
{
foreach (var observationDocument in cursor.Current)
{
cancellationToken.ThrowIfCancellationRequested();
if (!TryExtractObservationKey(observationDocument, out var key))
{
continue;
}
var rawFilter = Builders<BsonDocument>.Filter.Eq("tenant", key.Tenant) &
Builders<BsonDocument>.Filter.Eq("source.vendor", key.Vendor) &
Builders<BsonDocument>.Filter.Eq("upstream.upstream_id", key.UpstreamId) &
Builders<BsonDocument>.Filter.Eq("upstream.content_hash", key.ContentHash);
var rawDocument = await rawCollection
.Find(rawFilter)
.Sort(Builders<BsonDocument>.Sort.Descending("ingested_at").Descending("_id"))
.Limit(1)
.FirstOrDefaultAsync(cancellationToken)
.ConfigureAwait(false);
if (rawDocument is null)
{
missingRawDocuments.Add(key.ToString());
continue;
}
var advisoryRaw = MapToRawDocument(rawDocument);
var rawLinkset = BuildRawLinkset(advisoryRaw.Identifiers, advisoryRaw.Linkset);
var rawLinksetDocument = BuildRawLinksetBson(rawLinkset);
var update = Builders<BsonDocument>.Update.Set("rawLinkset", rawLinksetDocument);
updates.Add(new UpdateOneModel<BsonDocument>(
Builders<BsonDocument>.Filter.Eq("_id", observationDocument["_id"].AsString),
update));
if (updates.Count >= BulkBatchSize)
{
await observations.BulkWriteAsync(updates, cancellationToken: cancellationToken).ConfigureAwait(false);
updates.Clear();
}
}
}
if (updates.Count > 0)
{
await observations.BulkWriteAsync(updates, cancellationToken: cancellationToken).ConfigureAwait(false);
}
if (missingRawDocuments.Count > 0)
{
throw new InvalidOperationException(
$"Unable to locate advisory_raw documents for {missingRawDocuments.Count} observations: {string.Join(", ", missingRawDocuments.Take(10))}");
}
}
private static bool TryExtractObservationKey(BsonDocument observation, out ObservationKey key)
{
key = default;
if (!observation.TryGetValue("tenant", out var tenantValue) || tenantValue.IsBsonNull)
{
return false;
}
if (!observation.TryGetValue("source", out var sourceValue) || sourceValue is not BsonDocument sourceDocument)
{
return false;
}
if (!observation.TryGetValue("upstream", out var upstreamValue) || upstreamValue is not BsonDocument upstreamDocument)
{
return false;
}
var tenant = tenantValue.AsString;
var vendor = sourceDocument.GetValue("vendor", BsonString.Empty).AsString;
var upstreamId = upstreamDocument.GetValue("upstream_id", BsonString.Empty).AsString;
var contentHash = upstreamDocument.GetValue("contentHash", BsonString.Empty).AsString;
var createdAt = observation.GetValue("createdAt", BsonNull.Value);
key = new ObservationKey(
tenant,
vendor,
upstreamId,
contentHash,
BsonValueToDateTimeOffset(createdAt) ?? DateTimeOffset.UtcNow);
return !string.IsNullOrWhiteSpace(tenant) &&
!string.IsNullOrWhiteSpace(vendor) &&
!string.IsNullOrWhiteSpace(upstreamId) &&
!string.IsNullOrWhiteSpace(contentHash);
}
private static AdvisoryRawDocument MapToRawDocument(BsonDocument document)
{
var tenant = GetRequiredString(document, "tenant");
var source = MapSource(document["source"].AsBsonDocument);
var upstream = MapUpstream(document["upstream"].AsBsonDocument);
var content = MapContent(document["content"].AsBsonDocument);
var identifiers = MapIdentifiers(document["identifiers"].AsBsonDocument);
var linkset = MapLinkset(document["linkset"].AsBsonDocument);
var supersedes = document.GetValue("supersedes", BsonNull.Value);
return new AdvisoryRawDocument(
tenant,
source,
upstream,
content,
identifiers,
linkset,
supersedes.IsBsonNull ? null : supersedes.AsString);
}
private static RawSourceMetadata MapSource(BsonDocument source)
{
return new RawSourceMetadata(
GetRequiredString(source, "vendor"),
GetRequiredString(source, "connector"),
GetRequiredString(source, "version"),
GetOptionalString(source, "stream"));
}
private static RawUpstreamMetadata MapUpstream(BsonDocument upstream)
{
var provenanceBuilder = ImmutableDictionary.CreateBuilder<string, string>(StringComparer.Ordinal);
if (upstream.TryGetValue("provenance", out var provenanceValue) && provenanceValue.IsBsonDocument)
{
foreach (var element in provenanceValue.AsBsonDocument)
{
provenanceBuilder[element.Name] = BsonValueToString(element.Value);
}
}
var signatureDocument = upstream["signature"].AsBsonDocument;
var signature = new RawSignatureMetadata(
signatureDocument.GetValue("present", BsonBoolean.False).AsBoolean,
signatureDocument.TryGetValue("format", out var format) && !format.IsBsonNull ? format.AsString : null,
signatureDocument.TryGetValue("key_id", out var keyId) && !keyId.IsBsonNull ? keyId.AsString : null,
signatureDocument.TryGetValue("sig", out var sig) && !sig.IsBsonNull ? sig.AsString : null,
signatureDocument.TryGetValue("certificate", out var certificate) && !certificate.IsBsonNull ? certificate.AsString : null,
signatureDocument.TryGetValue("digest", out var digest) && !digest.IsBsonNull ? digest.AsString : null);
return new RawUpstreamMetadata(
GetRequiredString(upstream, "upstream_id"),
upstream.TryGetValue("document_version", out var version) && !version.IsBsonNull ? version.AsString : null,
GetDateTimeOffset(upstream, "retrieved_at", DateTimeOffset.UtcNow),
GetRequiredString(upstream, "content_hash"),
signature,
provenanceBuilder.ToImmutable());
}
private static RawContent MapContent(BsonDocument content)
{
var rawValue = content.GetValue("raw", BsonNull.Value);
string rawJson;
if (rawValue.IsBsonNull)
{
rawJson = "{}";
}
else if (rawValue.IsString)
{
rawJson = rawValue.AsString ?? "{}";
}
else
{
rawJson = rawValue.ToJson(new JsonWriterSettings { OutputMode = JsonOutputMode.RelaxedExtendedJson });
}
using var document = JsonDocument.Parse(string.IsNullOrWhiteSpace(rawJson) ? "{}" : rawJson);
return new RawContent(
GetRequiredString(content, "format"),
content.TryGetValue("spec_version", out var specVersion) && !specVersion.IsBsonNull ? specVersion.AsString : null,
document.RootElement.Clone(),
content.TryGetValue("encoding", out var encoding) && !encoding.IsBsonNull ? encoding.AsString : null);
}
private static RawIdentifiers MapIdentifiers(BsonDocument identifiers)
{
var aliases = identifiers.TryGetValue("aliases", out var aliasesValue) && aliasesValue.IsBsonArray
? aliasesValue.AsBsonArray.Select(BsonValueToString).ToImmutableArray()
: ImmutableArray<string>.Empty;
return new RawIdentifiers(
aliases,
GetRequiredString(identifiers, "primary"));
}
private static RawLinkset MapLinkset(BsonDocument linkset)
{
var aliases = linkset.TryGetValue("aliases", out var aliasesValue) && aliasesValue.IsBsonArray
? aliasesValue.AsBsonArray.Select(BsonValueToString).ToImmutableArray()
: ImmutableArray<string>.Empty;
var purls = linkset.TryGetValue("purls", out var purlsValue) && purlsValue.IsBsonArray
? purlsValue.AsBsonArray.Select(BsonValueToString).ToImmutableArray()
: ImmutableArray<string>.Empty;
var cpes = linkset.TryGetValue("cpes", out var cpesValue) && cpesValue.IsBsonArray
? cpesValue.AsBsonArray.Select(BsonValueToString).ToImmutableArray()
: ImmutableArray<string>.Empty;
var references = linkset.TryGetValue("references", out var referencesValue) && referencesValue.IsBsonArray
? referencesValue.AsBsonArray
.Where(static value => value.IsBsonDocument)
.Select(value =>
{
var doc = value.AsBsonDocument;
return new RawReference(
GetRequiredString(doc, "type"),
GetRequiredString(doc, "url"),
doc.TryGetValue("source", out var source) && !source.IsBsonNull ? source.AsString : null);
})
.ToImmutableArray()
: ImmutableArray<RawReference>.Empty;
var reconciledFrom = linkset.TryGetValue("reconciled_from", out var reconciledValue) && reconciledValue.IsBsonArray
? reconciledValue.AsBsonArray.Select(BsonValueToString).ToImmutableArray()
: ImmutableArray<string>.Empty;
var notesBuilder = ImmutableDictionary.CreateBuilder<string, string>(StringComparer.Ordinal);
if (linkset.TryGetValue("notes", out var notesValue) && notesValue.IsBsonDocument)
{
foreach (var element in notesValue.AsBsonDocument)
{
notesBuilder[element.Name] = BsonValueToString(element.Value);
}
}
return new RawLinkset
{
Aliases = aliases,
PackageUrls = purls,
Cpes = cpes,
References = references,
ReconciledFrom = reconciledFrom,
Notes = notesBuilder.ToImmutable()
};
}
private static RawLinkset BuildRawLinkset(RawIdentifiers identifiers, RawLinkset linkset)
{
var aliasBuilder = ImmutableArray.CreateBuilder<string>();
if (!string.IsNullOrWhiteSpace(identifiers.PrimaryId))
{
aliasBuilder.Add(identifiers.PrimaryId);
}
if (!identifiers.Aliases.IsDefaultOrEmpty)
{
foreach (var alias in identifiers.Aliases)
{
if (!string.IsNullOrEmpty(alias))
{
aliasBuilder.Add(alias);
}
}
}
if (!linkset.Aliases.IsDefaultOrEmpty)
{
foreach (var alias in linkset.Aliases)
{
if (!string.IsNullOrEmpty(alias))
{
aliasBuilder.Add(alias);
}
}
}
static ImmutableArray<string> EnsureArray(ImmutableArray<string> values)
=> values.IsDefault ? ImmutableArray<string>.Empty : values;
static ImmutableArray<RawReference> EnsureReferences(ImmutableArray<RawReference> values)
=> values.IsDefault ? ImmutableArray<RawReference>.Empty : values;
return linkset with
{
Aliases = aliasBuilder.ToImmutable(),
PackageUrls = EnsureArray(linkset.PackageUrls),
Cpes = EnsureArray(linkset.Cpes),
References = EnsureReferences(linkset.References),
ReconciledFrom = EnsureArray(linkset.ReconciledFrom),
Notes = linkset.Notes ?? ImmutableDictionary<string, string>.Empty
};
}
private static BsonDocument BuildRawLinksetBson(RawLinkset rawLinkset)
{
var references = new BsonArray(rawLinkset.References.Select(reference =>
{
var referenceDocument = new BsonDocument
{
{ "type", reference.Type },
{ "url", reference.Url }
};
if (!string.IsNullOrWhiteSpace(reference.Source))
{
referenceDocument["source"] = reference.Source;
}
return referenceDocument;
}));
var notes = new BsonDocument();
if (rawLinkset.Notes is not null)
{
foreach (var entry in rawLinkset.Notes)
{
notes[entry.Key] = entry.Value;
}
}
return new BsonDocument
{
{ "aliases", new BsonArray(rawLinkset.Aliases) },
{ "purls", new BsonArray(rawLinkset.PackageUrls) },
{ "cpes", new BsonArray(rawLinkset.Cpes) },
{ "references", references },
{ "reconciled_from", new BsonArray(rawLinkset.ReconciledFrom) },
{ "notes", notes }
};
}
private static string GetRequiredString(BsonDocument document, string key)
{
if (!document.TryGetValue(key, out var value) || value.IsBsonNull)
{
return string.Empty;
}
return value.IsString ? value.AsString : value.ToString() ?? string.Empty;
}
private static string? GetOptionalString(BsonDocument document, string key)
{
if (!document.TryGetValue(key, out var value) || value.IsBsonNull)
{
return null;
}
return value.IsString ? value.AsString : value.ToString();
}
private static string BsonValueToString(BsonValue value)
{
if (value.IsString)
{
return value.AsString ?? string.Empty;
}
if (value.IsBsonNull)
{
return string.Empty;
}
return value.ToString() ?? string.Empty;
}
private static DateTimeOffset GetDateTimeOffset(BsonDocument document, string field, DateTimeOffset fallback)
{
if (!document.TryGetValue(field, out var value) || value.IsBsonNull)
{
return fallback;
}
return BsonValueToDateTimeOffset(value) ?? fallback;
}
private static DateTimeOffset? BsonValueToDateTimeOffset(BsonValue value)
{
return value.BsonType switch
{
BsonType.DateTime => new DateTimeOffset(DateTime.SpecifyKind(value.ToUniversalTime(), DateTimeKind.Utc)),
BsonType.String when DateTimeOffset.TryParse(value.AsString, CultureInfo.InvariantCulture, DateTimeStyles.AssumeUniversal, out var parsed)
=> parsed.ToUniversalTime(),
BsonType.Int64 => DateTimeOffset.FromUnixTimeMilliseconds(value.AsInt64).ToUniversalTime(),
_ => null
};
}
private readonly record struct ObservationKey(
string Tenant,
string Vendor,
string UpstreamId,
string ContentHash,
DateTimeOffset CreatedAt)
{
public override string ToString()
=> $"{Tenant}:{Vendor}:{UpstreamId}:{ContentHash}";
}
}

View File

@@ -26,6 +26,11 @@ public sealed class AdvisoryObservationDocument
[BsonElement("linkset")]
public AdvisoryObservationLinksetDocument Linkset { get; set; } = new();
[BsonElement("rawLinkset")]
[BsonIgnoreIfNull]
public AdvisoryObservationRawLinksetDocument? RawLinkset { get; set; }
= null;
[BsonElement("createdAt")]
public DateTime CreatedAt { get; set; }
= DateTime.UtcNow;
@@ -161,3 +166,54 @@ public sealed class AdvisoryObservationReferenceDocument
[BsonElement("url")]
public string Url { get; set; } = string.Empty;
}
[BsonIgnoreExtraElements]
public sealed class AdvisoryObservationRawLinksetDocument
{
[BsonElement("aliases")]
[BsonIgnoreIfNull]
public List<string>? Aliases { get; set; }
= new();
[BsonElement("purls")]
[BsonIgnoreIfNull]
public List<string>? PackageUrls { get; set; }
= new();
[BsonElement("cpes")]
[BsonIgnoreIfNull]
public List<string>? Cpes { get; set; }
= new();
[BsonElement("references")]
[BsonIgnoreIfNull]
public List<AdvisoryObservationRawReferenceDocument>? References { get; set; }
= new();
[BsonElement("reconciled_from")]
[BsonIgnoreIfNull]
public List<string>? ReconciledFrom { get; set; }
= new();
[BsonElement("notes")]
[BsonIgnoreIfNull]
public Dictionary<string, string>? Notes { get; set; }
= new(StringComparer.Ordinal);
}
[BsonIgnoreExtraElements]
public sealed class AdvisoryObservationRawReferenceDocument
{
[BsonElement("type")]
[BsonIgnoreIfNull]
public string? Type { get; set; }
= null;
[BsonElement("url")]
public string Url { get; set; } = string.Empty;
[BsonElement("source")]
[BsonIgnoreIfNull]
public string? Source { get; set; }
= null;
}

View File

@@ -6,6 +6,7 @@ using System.Text.Json.Nodes;
using MongoDB.Bson;
using MongoDB.Bson.IO;
using StellaOps.Concelier.Models.Observations;
using StellaOps.Concelier.RawModels;
namespace StellaOps.Concelier.Storage.Mongo.Observations;
@@ -22,6 +23,8 @@ internal static class AdvisoryObservationDocumentFactory
var contentMetadata = ToImmutable(document.Content.Metadata);
var upstreamMetadata = ToImmutable(document.Upstream.Metadata);
var rawLinkset = ToRawLinkset(document.RawLinkset);
var observation = new AdvisoryObservation(
document.Id,
document.Tenant,
@@ -52,6 +55,7 @@ internal static class AdvisoryObservationDocumentFactory
document.Linkset.Purls ?? Enumerable.Empty<string>(),
document.Linkset.Cpes ?? Enumerable.Empty<string>(),
document.Linkset.References?.Select(reference => new AdvisoryObservationReference(reference.Type, reference.Url))),
rawLinkset,
DateTime.SpecifyKind(document.CreatedAt, DateTimeKind.Utc),
attributes);
@@ -89,4 +93,70 @@ internal static class AdvisoryObservationDocumentFactory
return builder.ToImmutable();
}
private static RawLinkset ToRawLinkset(AdvisoryObservationRawLinksetDocument? document)
{
if (document is null)
{
return new RawLinkset();
}
static ImmutableArray<string> ToImmutableStringArray(List<string>? values)
{
if (values is null || values.Count == 0)
{
return ImmutableArray<string>.Empty;
}
return values
.Select(static value => value ?? string.Empty)
.ToImmutableArray();
}
static ImmutableArray<RawReference> ToImmutableReferences(List<AdvisoryObservationRawReferenceDocument>? references)
{
if (references is null || references.Count == 0)
{
return ImmutableArray<RawReference>.Empty;
}
return references
.Select(static reference => new RawReference(
reference.Type ?? string.Empty,
reference.Url,
reference.Source))
.ToImmutableArray();
}
static ImmutableDictionary<string, string> ToImmutableDictionary(Dictionary<string, string>? values)
{
if (values is null || values.Count == 0)
{
return ImmutableDictionary<string, string>.Empty;
}
var builder = ImmutableDictionary.CreateBuilder<string, string>(StringComparer.Ordinal);
foreach (var pair in values)
{
if (pair.Key is null)
{
continue;
}
builder[pair.Key] = pair.Value;
}
return builder.ToImmutable();
}
return new RawLinkset
{
Aliases = ToImmutableStringArray(document.Aliases),
PackageUrls = ToImmutableStringArray(document.PackageUrls),
Cpes = ToImmutableStringArray(document.Cpes),
References = ToImmutableReferences(document.References),
ReconciledFrom = ToImmutableStringArray(document.ReconciledFrom),
Notes = ToImmutableDictionary(document.Notes)
};
}
}

View File

@@ -109,6 +109,7 @@ public static class ServiceCollectionExtensions
services.AddSingleton<IMongoMigration, EnsureAdvisoryRawIdempotencyIndexMigration>();
services.AddSingleton<IMongoMigration, EnsureAdvisorySupersedesBackfillMigration>();
services.AddSingleton<IMongoMigration, EnsureAdvisoryRawValidatorMigration>();
services.AddSingleton<IMongoMigration, EnsureAdvisoryObservationsRawLinksetMigration>();
services.AddSingleton<IMongoMigration, EnsureAdvisoryEventCollectionsMigration>();
services.AddSingleton<IMongoMigration, SemVerStyleBackfillMigration>();

View File

@@ -10,6 +10,7 @@
> Docs alignment (2025-10-26): Rollback guidance added to `docs/deploy/containers.md` §6.
> 2025-10-28: Documented duplicate audit + migration workflow in `docs/deploy/containers.md`, Offline Kit guide, and `MIGRATIONS.md`; published `ops/devops/scripts/check-advisory-raw-duplicates.js` for staging/offline clusters.
> Docs alignment (2025-10-26): Offline kit requirements documented in `docs/deploy/containers.md` §5.
| CONCELIER-STORE-AOC-19-005 `Raw linkset backfill` | TODO (2025-11-04) | Concelier Storage Guild, DevOps Guild | CONCELIER-CORE-AOC-19-004 | Plan and execute advisory_observations `rawLinkset` backfill (online + Offline Kit bundles), supply migration scripts + rehearse rollback. Follow the coordination plan in `docs/dev/raw-linkset-backfill-plan.md`. |
## Policy Engine v2

View File

@@ -39,6 +39,28 @@ public sealed class AdvisoryObservationFactoryTests
var reference = Assert.Single(observation.Linkset.References);
Assert.Equal("advisory", reference.Type);
Assert.Equal("https://example.test/advisory", reference.Url);
Assert.Equal(
new[] { "GHSA-XXXX-YYYY", " CVE-2025-0001 ", "ghsa-XXXX-YYYY", " CVE-2025-0001 " },
observation.RawLinkset.Aliases);
Assert.Equal(
new[] { "pkg:NPM/left-pad@1.0.0", "pkg:npm/left-pad@1.0.0?foo=bar" },
observation.RawLinkset.PackageUrls);
Assert.Equal(
new[] { "cpe:/a:Example:Product:1.0", "cpe:/a:example:product:1.0" },
observation.RawLinkset.Cpes);
Assert.Collection(
observation.RawLinkset.References,
first =>
{
Assert.Equal("Advisory", first.Type);
Assert.Equal(" https://example.test/advisory ", first.Url);
},
second =>
{
Assert.Equal("ADVISORY", second.Type);
Assert.Equal("https://example.test/advisory", second.Url);
});
}
[Fact]
@@ -102,6 +124,8 @@ public sealed class AdvisoryObservationFactoryTests
Assert.Equal("1.0.5", observation.Attributes["linkset.note.range-fixed"]);
Assert.Equal("tenant-a:vendor-x:previous:sha256:123", observation.Attributes["supersedes"]);
Assert.Equal("connector-a;connector-b", observation.Attributes["linkset.reconciled_from"]);
Assert.Equal(notes, observation.RawLinkset.Notes);
Assert.Equal(new[] { "connector-a", "connector-b" }, observation.RawLinkset.ReconciledFrom);
}
private static AdvisoryRawDocument BuildRawDocument(

View File

@@ -1,7 +1,9 @@
using System.Collections.Immutable;
using System.Linq;
using System.Text.Json.Nodes;
using StellaOps.Concelier.Core.Observations;
using StellaOps.Concelier.Models.Observations;
using StellaOps.Concelier.RawModels;
using Xunit;
namespace StellaOps.Concelier.Core.Tests.Observations;
@@ -226,6 +228,15 @@ public sealed class AdvisoryObservationQueryServiceTests
var content = new AdvisoryObservationContent("CSAF", "2.0", raw);
var linkset = new AdvisoryObservationLinkset(aliases, purls, cpes, references);
var rawLinkset = new RawLinkset
{
Aliases = aliases.ToImmutableArray(),
PackageUrls = purls.ToImmutableArray(),
Cpes = cpes.ToImmutableArray(),
References = references
.Select(static reference => new RawReference(reference.Type, reference.Url))
.ToImmutableArray()
};
return new AdvisoryObservation(
observationId,
@@ -234,6 +245,7 @@ public sealed class AdvisoryObservationQueryServiceTests
upstream,
content,
linkset,
rawLinkset,
createdAt);
}

View File

@@ -1,5 +1,6 @@
using System;
using System.Collections.Immutable;
using System.Linq;
using System.Text.Json;
using System.Threading;
using System.Threading.Tasks;
@@ -66,7 +67,7 @@ public sealed class AdvisoryRawServiceTests
await service.IngestAsync(document, CancellationToken.None);
Assert.NotNull(repository.CapturedDocument);
Assert.Equal(aliasSeries, repository.CapturedDocument!.Identifiers.Aliases);
Assert.True(aliasSeries.SequenceEqual(repository.CapturedDocument!.Identifiers.Aliases));
}
private static AdvisoryRawService CreateService(RecordingRepository repository)

View File

@@ -1,127 +0,0 @@
{
"advisoryKey": "GHSA-aaaa-bbbb-cccc",
"affectedPackages": [
{
"type": "semver",
"identifier": "pkg:npm/example-widget",
"platform": null,
"versionRanges": [
{
"fixedVersion": "2.5.1",
"introducedVersion": null,
"lastAffectedVersion": null,
"primitives": null,
"provenance": {
"source": "ghsa",
"kind": "map",
"value": "ghsa-aaaa-bbbb-cccc",
"decisionReason": null,
"recordedAt": "2024-03-05T10:00:00+00:00",
"fieldMask": []
},
"rangeExpression": ">=0.0.0 <2.5.1",
"rangeKind": "semver"
},
{
"fixedVersion": "3.2.4",
"introducedVersion": "3.0.0",
"lastAffectedVersion": null,
"primitives": null,
"provenance": {
"source": "ghsa",
"kind": "map",
"value": "ghsa-aaaa-bbbb-cccc",
"decisionReason": null,
"recordedAt": "2024-03-05T10:00:00+00:00",
"fieldMask": []
},
"rangeExpression": null,
"rangeKind": "semver"
}
],
"normalizedVersions": [],
"statuses": [],
"provenance": [
{
"source": "ghsa",
"kind": "map",
"value": "ghsa-aaaa-bbbb-cccc",
"decisionReason": null,
"recordedAt": "2024-03-05T10:00:00+00:00",
"fieldMask": []
}
]
}
],
"aliases": [
"CVE-2024-2222",
"GHSA-aaaa-bbbb-cccc"
],
"canonicalMetricId": null,
"credits": [],
"cvssMetrics": [
{
"baseScore": 8.8,
"baseSeverity": "high",
"provenance": {
"source": "ghsa",
"kind": "map",
"value": "ghsa-aaaa-bbbb-cccc",
"decisionReason": null,
"recordedAt": "2024-03-05T10:00:00+00:00",
"fieldMask": []
},
"vector": "CVSS:3.1/AV:N/AC:L/PR:N/UI:R/S:U/C:H/I:H/A:H",
"version": "3.1"
}
],
"cwes": [],
"description": null,
"exploitKnown": false,
"language": "en",
"modified": "2024-03-04T12:00:00+00:00",
"provenance": [
{
"source": "ghsa",
"kind": "map",
"value": "ghsa-aaaa-bbbb-cccc",
"decisionReason": null,
"recordedAt": "2024-03-05T10:00:00+00:00",
"fieldMask": []
}
],
"published": "2024-03-04T00:00:00+00:00",
"references": [
{
"kind": "patch",
"provenance": {
"source": "ghsa",
"kind": "map",
"value": "ghsa-aaaa-bbbb-cccc",
"decisionReason": null,
"recordedAt": "2024-03-05T10:00:00+00:00",
"fieldMask": []
},
"sourceTag": "ghsa",
"summary": "Patch commit",
"url": "https://github.com/example/widget/commit/abcd1234"
},
{
"kind": "advisory",
"provenance": {
"source": "ghsa",
"kind": "map",
"value": "ghsa-aaaa-bbbb-cccc",
"decisionReason": null,
"recordedAt": "2024-03-05T10:00:00+00:00",
"fieldMask": []
},
"sourceTag": "ghsa",
"summary": "GitHub Security Advisory",
"url": "https://github.com/example/widget/security/advisories/GHSA-aaaa-bbbb-cccc"
}
],
"severity": "high",
"summary": "A crafted payload can pollute Object.prototype leading to RCE.",
"title": "Prototype pollution in widget.js"
}

View File

@@ -57,6 +57,7 @@
"CVE-2024-2222",
"GHSA-aaaa-bbbb-cccc"
],
"canonicalMetricId": null,
"credits": [],
"cvssMetrics": [
{
@@ -74,6 +75,8 @@
"version": "3.1"
}
],
"cwes": [],
"description": null,
"exploitKnown": false,
"language": "en",
"modified": "2024-03-04T12:00:00+00:00",

View File

@@ -1,45 +0,0 @@
{
"advisoryKey": "CVE-2023-9999",
"affectedPackages": [],
"aliases": [
"CVE-2023-9999"
],
"canonicalMetricId": null,
"credits": [],
"cvssMetrics": [],
"cwes": [],
"description": null,
"exploitKnown": true,
"language": "en",
"modified": "2024-02-09T16:22:00+00:00",
"provenance": [
{
"source": "cisa-kev",
"kind": "annotate",
"value": "kev",
"decisionReason": null,
"recordedAt": "2024-02-10T09:30:00+00:00",
"fieldMask": []
}
],
"published": "2023-11-20T00:00:00+00:00",
"references": [
{
"kind": "kev",
"provenance": {
"source": "cisa-kev",
"kind": "annotate",
"value": "kev",
"decisionReason": null,
"recordedAt": "2024-02-10T09:30:00+00:00",
"fieldMask": []
},
"sourceTag": "cisa",
"summary": "CISA KEV entry",
"url": "https://www.cisa.gov/known-exploited-vulnerabilities-catalog"
}
],
"severity": "critical",
"summary": "Unauthenticated RCE due to unsafe deserialization.",
"title": "Remote code execution in LegacyServer"
}

View File

@@ -4,8 +4,11 @@
"aliases": [
"CVE-2023-9999"
],
"canonicalMetricId": null,
"credits": [],
"cvssMetrics": [],
"cwes": [],
"description": null,
"exploitKnown": true,
"language": "en",
"modified": "2024-02-09T16:22:00+00:00",

View File

@@ -1,122 +0,0 @@
{
"advisoryKey": "CVE-2024-1234",
"affectedPackages": [
{
"type": "cpe",
"identifier": "cpe:/a:examplecms:examplecms:1.0",
"platform": null,
"versionRanges": [
{
"fixedVersion": "1.0.5",
"introducedVersion": "1.0",
"lastAffectedVersion": null,
"primitives": null,
"provenance": {
"source": "nvd",
"kind": "map",
"value": "cve-2024-1234",
"decisionReason": null,
"recordedAt": "2024-08-01T12:00:00+00:00",
"fieldMask": []
},
"rangeExpression": null,
"rangeKind": "version"
}
],
"normalizedVersions": [],
"statuses": [
{
"provenance": {
"source": "nvd",
"kind": "map",
"value": "cve-2024-1234",
"decisionReason": null,
"recordedAt": "2024-08-01T12:00:00+00:00",
"fieldMask": []
},
"status": "affected"
}
],
"provenance": [
{
"source": "nvd",
"kind": "map",
"value": "cve-2024-1234",
"decisionReason": null,
"recordedAt": "2024-08-01T12:00:00+00:00",
"fieldMask": []
}
]
}
],
"aliases": [
"CVE-2024-1234"
],
"canonicalMetricId": null,
"credits": [],
"cvssMetrics": [
{
"baseScore": 9.8,
"baseSeverity": "critical",
"provenance": {
"source": "nvd",
"kind": "map",
"value": "cve-2024-1234",
"decisionReason": null,
"recordedAt": "2024-08-01T12:00:00+00:00",
"fieldMask": []
},
"vector": "CVSS:3.1/AV:N/AC:L/PR:N/UI:N/S:U/C:H/I:H/A:H",
"version": "3.1"
}
],
"cwes": [],
"description": null,
"exploitKnown": false,
"language": "en",
"modified": "2024-07-16T10:35:00+00:00",
"provenance": [
{
"source": "nvd",
"kind": "map",
"value": "cve-2024-1234",
"decisionReason": null,
"recordedAt": "2024-08-01T12:00:00+00:00",
"fieldMask": []
}
],
"published": "2024-07-15T00:00:00+00:00",
"references": [
{
"kind": "advisory",
"provenance": {
"source": "example",
"kind": "fetch",
"value": "bulletin",
"decisionReason": null,
"recordedAt": "2024-07-14T15:00:00+00:00",
"fieldMask": []
},
"sourceTag": "vendor",
"summary": "Vendor bulletin",
"url": "https://example.org/security/CVE-2024-1234"
},
{
"kind": "advisory",
"provenance": {
"source": "nvd",
"kind": "map",
"value": "cve-2024-1234",
"decisionReason": null,
"recordedAt": "2024-08-01T12:00:00+00:00",
"fieldMask": []
},
"sourceTag": "nvd",
"summary": "NVD entry",
"url": "https://nvd.nist.gov/vuln/detail/CVE-2024-1234"
}
],
"severity": "high",
"summary": "An integer overflow in ExampleCMS allows remote attackers to escalate privileges.",
"title": "Integer overflow in ExampleCMS"
}

View File

@@ -52,6 +52,7 @@
"aliases": [
"CVE-2024-1234"
],
"canonicalMetricId": null,
"credits": [],
"cvssMetrics": [
{
@@ -69,6 +70,8 @@
"version": "3.1"
}
],
"cwes": [],
"description": null,
"exploitKnown": false,
"language": "en",
"modified": "2024-07-16T10:35:00+00:00",

View File

@@ -1,125 +0,0 @@
{
"advisoryKey": "RHSA-2024:0252",
"affectedPackages": [
{
"type": "rpm",
"identifier": "kernel-0:4.18.0-553.el8.x86_64",
"platform": "rhel-8",
"versionRanges": [
{
"fixedVersion": null,
"introducedVersion": "0:4.18.0-553.el8",
"lastAffectedVersion": null,
"primitives": null,
"provenance": {
"source": "redhat",
"kind": "map",
"value": "rhsa-2024:0252",
"decisionReason": null,
"recordedAt": "2024-05-11T09:00:00+00:00",
"fieldMask": []
},
"rangeExpression": null,
"rangeKind": "nevra"
}
],
"normalizedVersions": [],
"statuses": [
{
"provenance": {
"source": "redhat",
"kind": "map",
"value": "rhsa-2024:0252",
"decisionReason": null,
"recordedAt": "2024-05-11T09:00:00+00:00",
"fieldMask": []
},
"status": "fixed"
}
],
"provenance": [
{
"source": "redhat",
"kind": "enrich",
"value": "cve-2024-5678",
"decisionReason": null,
"recordedAt": "2024-05-11T09:05:00+00:00",
"fieldMask": []
},
{
"source": "redhat",
"kind": "map",
"value": "rhsa-2024:0252",
"decisionReason": null,
"recordedAt": "2024-05-11T09:00:00+00:00",
"fieldMask": []
}
]
}
],
"aliases": [
"CVE-2024-5678",
"RHSA-2024:0252"
],
"canonicalMetricId": null,
"credits": [],
"cvssMetrics": [
{
"baseScore": 6.7,
"baseSeverity": "medium",
"provenance": {
"source": "redhat",
"kind": "map",
"value": "rhsa-2024:0252",
"decisionReason": null,
"recordedAt": "2024-05-11T09:00:00+00:00",
"fieldMask": []
},
"vector": "CVSS:3.1/AV:L/AC:L/PR:H/UI:N/S:U/C:H/I:H/A:H",
"version": "3.1"
}
],
"cwes": [],
"description": null,
"exploitKnown": false,
"language": "en",
"modified": "2024-05-11T08:15:00+00:00",
"provenance": [
{
"source": "redhat",
"kind": "enrich",
"value": "cve-2024-5678",
"decisionReason": null,
"recordedAt": "2024-05-11T09:05:00+00:00",
"fieldMask": []
},
{
"source": "redhat",
"kind": "map",
"value": "rhsa-2024:0252",
"decisionReason": null,
"recordedAt": "2024-05-11T09:00:00+00:00",
"fieldMask": []
}
],
"published": "2024-05-10T19:28:00+00:00",
"references": [
{
"kind": "advisory",
"provenance": {
"source": "redhat",
"kind": "map",
"value": "rhsa-2024:0252",
"decisionReason": null,
"recordedAt": "2024-05-11T09:00:00+00:00",
"fieldMask": []
},
"sourceTag": "redhat",
"summary": "Red Hat security advisory",
"url": "https://access.redhat.com/errata/RHSA-2024:0252"
}
],
"severity": "critical",
"summary": "Updates the Red Hat Enterprise Linux kernel to address CVE-2024-5678.",
"title": "Important: kernel security update"
}

View File

@@ -61,6 +61,7 @@
"CVE-2024-5678",
"RHSA-2024:0252"
],
"canonicalMetricId": null,
"credits": [],
"cvssMetrics": [
{
@@ -78,6 +79,8 @@
"version": "3.1"
}
],
"cwes": [],
"description": null,
"exploitKnown": false,
"language": "en",
"modified": "2024-05-11T08:15:00+00:00",

View File

@@ -2,6 +2,7 @@ using System.Collections.Generic;
using System.Collections.Immutable;
using System.Text.Json.Nodes;
using StellaOps.Concelier.Models.Observations;
using StellaOps.Concelier.RawModels;
using Xunit;
namespace StellaOps.Concelier.Models.Tests.Observations;
@@ -39,6 +40,16 @@ public sealed class AdvisoryObservationTests
["pipeline"] = "daily"
});
var rawLinkset = new RawLinkset
{
Aliases = ImmutableArray.Create(" Cve-2025-1234 ", "cve-2025-1234"),
PackageUrls = ImmutableArray.Create("pkg:generic/foo@1.0.0"),
Cpes = ImmutableArray.Create("cpe:/a:vendor:product:1"),
References = ImmutableArray.Create(new RawReference("ADVISORY", "https://example.com/advisory")),
ReconciledFrom = ImmutableArray.Create("pointer-1"),
Notes = ImmutableDictionary.CreateRange(new Dictionary<string, string> { ["note"] = "value" })
};
var observation = new AdvisoryObservation(
observationId: " tenant-a:CVE-2025-1234:1 ",
tenant: " Tenant-A ",
@@ -46,6 +57,7 @@ public sealed class AdvisoryObservationTests
upstream: upstream,
content: content,
linkset: linkset,
rawLinkset: rawLinkset,
createdAt: DateTimeOffset.Parse("2025-10-01T01:00:06Z"),
attributes: attributes);

View File

@@ -0,0 +1,337 @@
using System;
using System.Collections.Generic;
using System.Collections.Immutable;
using System.Linq;
using System.Text.Json;
using System.Threading;
using System.Threading.Tasks;
using Microsoft.Extensions.Logging.Abstractions;
using MongoDB.Bson;
using MongoDB.Bson.Serialization;
using MongoDB.Driver;
using StellaOps.Concelier.RawModels;
using StellaOps.Concelier.Storage.Mongo;
using StellaOps.Concelier.Storage.Mongo.Migrations;
using StellaOps.Concelier.Storage.Mongo.Observations;
using StellaOps.Concelier.Storage.Mongo.Raw;
using Xunit;
namespace StellaOps.Concelier.Storage.Mongo.Tests.Migrations;
[Collection("mongo-fixture")]
public sealed class EnsureAdvisoryObservationsRawLinksetMigrationTests
{
private readonly MongoIntegrationFixture _fixture;
public EnsureAdvisoryObservationsRawLinksetMigrationTests(MongoIntegrationFixture fixture)
{
_fixture = fixture;
}
[Fact]
public async Task ApplyAsync_BackfillsRawLinksetFromRawDocument()
{
var databaseName = $"concelier-rawlinkset-{Guid.NewGuid():N}";
var database = _fixture.Client.GetDatabase(databaseName);
await database.CreateCollectionAsync(MongoStorageDefaults.Collections.Migrations);
await database.CreateCollectionAsync(MongoStorageDefaults.Collections.AdvisoryObservations);
try
{
var rawRepository = new MongoAdvisoryRawRepository(
database,
TimeProvider.System,
NullLogger<MongoAdvisoryRawRepository>.Instance);
var rawDocument = RawDocumentFactory.CreateAdvisory(
tenant: "tenant-a",
source: new RawSourceMetadata("Vendor-X", "connector-y", "1.0.0", "stable"),
upstream: new RawUpstreamMetadata(
UpstreamId: "GHSA-2025-0001",
DocumentVersion: "v1",
RetrievedAt: DateTimeOffset.Parse("2025-10-29T12:34:56Z"),
ContentHash: "sha256:abc123",
Signature: new RawSignatureMetadata(true, "dsse", "key1", "sig1"),
Provenance: ImmutableDictionary.CreateRange(new[] { new KeyValuePair<string, string>("api", "https://example.test/api") })),
content: new RawContent(
Format: "OSV",
SpecVersion: "1.0.0",
Raw: ParseJsonElement("""{"id":"GHSA-2025-0001"}"""),
Encoding: null),
identifiers: new RawIdentifiers(
Aliases: ImmutableArray.Create("CVE-2025-0001", "cve-2025-0001"),
PrimaryId: "CVE-2025-0001"),
linkset: new RawLinkset
{
Aliases = ImmutableArray.Create("GHSA-xxxx-yyyy"),
PackageUrls = ImmutableArray.Create("pkg:npm/example@1.0.0"),
Cpes = ImmutableArray.Create("cpe:/a:example:product:1.0"),
References = ImmutableArray.Create(new RawReference("advisory", "https://example.test/advisory", "vendor")),
ReconciledFrom = ImmutableArray.Create("connector-y"),
Notes = ImmutableDictionary.CreateRange(new[] { new KeyValuePair<string, string>("range-fixed", "1.0.1") })
});
await rawRepository.UpsertAsync(rawDocument, CancellationToken.None);
var expectedRawLinkset = BuildRawLinkset(rawDocument.Identifiers, rawDocument.Linkset);
var canonicalAliases = ImmutableArray.Create("cve-2025-0001", "ghsa-xxxx-yyyy");
var canonicalPurls = rawDocument.Linkset.PackageUrls;
var canonicalCpes = rawDocument.Linkset.Cpes;
var canonicalReferences = rawDocument.Linkset.References;
var observationId = "tenant-a:vendor-x:ghsa-2025-0001:sha256-abc123";
var observationBson = BuildObservationDocument(
observationId,
rawDocument,
canonicalAliases,
canonicalPurls,
canonicalCpes,
canonicalReferences,
rawDocument.Upstream.RetrievedAt,
includeRawLinkset: false);
await database
.GetCollection<BsonDocument>(MongoStorageDefaults.Collections.AdvisoryObservations)
.InsertOneAsync(observationBson);
var migration = new EnsureAdvisoryObservationsRawLinksetMigration();
await migration.ApplyAsync(database, CancellationToken.None);
var storedBson = await database
.GetCollection<BsonDocument>(MongoStorageDefaults.Collections.AdvisoryObservations)
.Find(Builders<BsonDocument>.Filter.Eq("_id", observationId))
.FirstOrDefaultAsync();
Assert.NotNull(storedBson);
Assert.True(storedBson.TryGetValue("rawLinkset", out var rawLinksetValue));
var storedDocument = BsonSerializer.Deserialize<AdvisoryObservationDocument>(storedBson);
var storedObservation = AdvisoryObservationDocumentFactory.ToModel(storedDocument);
Assert.True(expectedRawLinkset.Aliases.SequenceEqual(storedObservation.RawLinkset.Aliases, StringComparer.Ordinal));
Assert.True(expectedRawLinkset.PackageUrls.SequenceEqual(storedObservation.RawLinkset.PackageUrls, StringComparer.Ordinal));
Assert.True(expectedRawLinkset.Cpes.SequenceEqual(storedObservation.RawLinkset.Cpes, StringComparer.Ordinal));
Assert.True(expectedRawLinkset.References.SequenceEqual(storedObservation.RawLinkset.References));
Assert.Equal(expectedRawLinkset.Notes, storedObservation.RawLinkset.Notes);
}
finally
{
await _fixture.Client.DropDatabaseAsync(databaseName);
}
}
[Fact]
public async Task ApplyAsync_ThrowsWhenRawDocumentMissing()
{
var databaseName = $"concelier-rawlinkset-missing-{Guid.NewGuid():N}";
var database = _fixture.Client.GetDatabase(databaseName);
await database.CreateCollectionAsync(MongoStorageDefaults.Collections.Migrations);
await database.CreateCollectionAsync(MongoStorageDefaults.Collections.AdvisoryObservations);
try
{
var rawDocument = RawDocumentFactory.CreateAdvisory(
tenant: "tenant-b",
source: new RawSourceMetadata("Vendor-Y", "connector-z", "2.0.0", "stable"),
upstream: new RawUpstreamMetadata(
UpstreamId: "GHSA-9999-0001",
DocumentVersion: "v2",
RetrievedAt: DateTimeOffset.Parse("2025-10-30T00:00:00Z"),
ContentHash: "sha256:def456",
Signature: new RawSignatureMetadata(false),
Provenance: ImmutableDictionary<string, string>.Empty),
content: new RawContent(
Format: "OSV",
SpecVersion: "1.0.0",
Raw: ParseJsonElement("""{"id":"GHSA-9999-0001"}"""),
Encoding: null),
identifiers: new RawIdentifiers(
Aliases: ImmutableArray<string>.Empty,
PrimaryId: "GHSA-9999-0001"),
linkset: new RawLinkset());
var observationId = "tenant-b:vendor-y:ghsa-9999-0001:sha256-def456";
var document = BuildObservationDocument(
observationId,
rawDocument,
ImmutableArray<string>.Empty,
ImmutableArray<string>.Empty,
ImmutableArray<string>.Empty,
ImmutableArray<RawReference>.Empty,
rawDocument.Upstream.RetrievedAt,
includeRawLinkset: false);
await database
.GetCollection<BsonDocument>(MongoStorageDefaults.Collections.AdvisoryObservations)
.InsertOneAsync(document);
var migration = new EnsureAdvisoryObservationsRawLinksetMigration();
await Assert.ThrowsAsync<InvalidOperationException>(
() => migration.ApplyAsync(database, CancellationToken.None));
}
finally
{
await _fixture.Client.DropDatabaseAsync(databaseName);
}
}
private static BsonDocument BuildObservationDocument(
string observationId,
AdvisoryRawDocument rawDocument,
ImmutableArray<string> canonicalAliases,
ImmutableArray<string> canonicalPurls,
ImmutableArray<string> canonicalCpes,
ImmutableArray<RawReference> canonicalReferences,
DateTimeOffset createdAt,
bool includeRawLinkset,
RawLinkset? rawLinkset = null)
{
var sourceDocument = new BsonDocument
{
{ "vendor", rawDocument.Source.Vendor },
{ "stream", string.IsNullOrWhiteSpace(rawDocument.Source.Stream) ? rawDocument.Source.Connector : rawDocument.Source.Stream! },
{ "api", rawDocument.Upstream.Provenance.TryGetValue("api", out var api) ? api : rawDocument.Source.Connector }
};
if (!string.IsNullOrWhiteSpace(rawDocument.Source.ConnectorVersion))
{
sourceDocument["collectorVersion"] = rawDocument.Source.ConnectorVersion;
}
var signatureDocument = new BsonDocument
{
{ "present", rawDocument.Upstream.Signature.Present }
};
if (!string.IsNullOrWhiteSpace(rawDocument.Upstream.Signature.Format))
{
signatureDocument["format"] = rawDocument.Upstream.Signature.Format;
}
if (!string.IsNullOrWhiteSpace(rawDocument.Upstream.Signature.KeyId))
{
signatureDocument["keyId"] = rawDocument.Upstream.Signature.KeyId;
}
if (!string.IsNullOrWhiteSpace(rawDocument.Upstream.Signature.Signature))
{
signatureDocument["signature"] = rawDocument.Upstream.Signature.Signature;
}
var upstreamDocument = new BsonDocument
{
{ "upstream_id", rawDocument.Upstream.UpstreamId },
{ "document_version", rawDocument.Upstream.DocumentVersion },
{ "fetchedAt", rawDocument.Upstream.RetrievedAt.UtcDateTime },
{ "receivedAt", rawDocument.Upstream.RetrievedAt.UtcDateTime },
{ "contentHash", rawDocument.Upstream.ContentHash },
{ "signature", signatureDocument },
{ "metadata", new BsonDocument(rawDocument.Upstream.Provenance) }
};
var contentDocument = new BsonDocument
{
{ "format", rawDocument.Content.Format },
{ "raw", BsonDocument.Parse(rawDocument.Content.Raw.GetRawText()) }
};
if (!string.IsNullOrWhiteSpace(rawDocument.Content.SpecVersion))
{
contentDocument["specVersion"] = rawDocument.Content.SpecVersion;
}
var canonicalLinkset = new BsonDocument
{
{ "aliases", new BsonArray(canonicalAliases) },
{ "purls", new BsonArray(canonicalPurls) },
{ "cpes", new BsonArray(canonicalCpes) },
{ "references", new BsonArray(canonicalReferences.Select(reference => new BsonDocument
{
{ "type", reference.Type },
{ "url", reference.Url }
})) }
};
var document = new BsonDocument
{
{ "_id", observationId },
{ "tenant", rawDocument.Tenant },
{ "source", sourceDocument },
{ "upstream", upstreamDocument },
{ "content", contentDocument },
{ "linkset", canonicalLinkset },
{ "createdAt", createdAt.UtcDateTime },
{ "attributes", new BsonDocument() }
};
if (includeRawLinkset)
{
var actualRawLinkset = rawLinkset ?? throw new ArgumentNullException(nameof(rawLinkset));
document["rawLinkset"] = new BsonDocument
{
{ "aliases", new BsonArray(actualRawLinkset.Aliases) },
{ "purls", new BsonArray(actualRawLinkset.PackageUrls) },
{ "cpes", new BsonArray(actualRawLinkset.Cpes) },
{ "references", new BsonArray(actualRawLinkset.References.Select(reference => new BsonDocument
{
{ "type", reference.Type },
{ "url", reference.Url },
{ "source", reference.Source }
})) },
{ "reconciled_from", new BsonArray(actualRawLinkset.ReconciledFrom) },
{ "notes", new BsonDocument(actualRawLinkset.Notes) }
};
}
return document;
}
private static JsonElement ParseJsonElement(string json)
{
using var document = JsonDocument.Parse(json);
return document.RootElement.Clone();
}
private static RawLinkset BuildRawLinkset(RawIdentifiers identifiers, RawLinkset linkset)
{
var aliasBuilder = ImmutableArray.CreateBuilder<string>();
if (!string.IsNullOrWhiteSpace(identifiers.PrimaryId))
{
aliasBuilder.Add(identifiers.PrimaryId);
}
if (!identifiers.Aliases.IsDefaultOrEmpty)
{
foreach (var alias in identifiers.Aliases)
{
if (!string.IsNullOrEmpty(alias))
{
aliasBuilder.Add(alias);
}
}
}
if (!linkset.Aliases.IsDefaultOrEmpty)
{
foreach (var alias in linkset.Aliases)
{
if (!string.IsNullOrEmpty(alias))
{
aliasBuilder.Add(alias);
}
}
}
static ImmutableArray<string> EnsureArray(ImmutableArray<string> values)
=> values.IsDefault ? ImmutableArray<string>.Empty : values;
static ImmutableArray<RawReference> EnsureReferences(ImmutableArray<RawReference> values)
=> values.IsDefault ? ImmutableArray<RawReference>.Empty : values;
return linkset with
{
Aliases = aliasBuilder.ToImmutable(),
PackageUrls = EnsureArray(linkset.PackageUrls),
Cpes = EnsureArray(linkset.Cpes),
References = EnsureReferences(linkset.References),
ReconciledFrom = EnsureArray(linkset.ReconciledFrom),
Notes = linkset.Notes ?? ImmutableDictionary<string, string>.Empty
};
}
}

View File

@@ -52,6 +52,18 @@ public sealed class AdvisoryObservationDocumentFactoryTests
{
new() { Type = "advisory", Url = "https://example.com" }
}
},
RawLinkset = new AdvisoryObservationRawLinksetDocument
{
Aliases = new List<string> { "CVE-2025-1234", "cve-2025-1234" },
PackageUrls = new List<string> { "pkg:generic/foo@1.0.0" },
Cpes = new List<string> { "cpe:/a:vendor:product:1" },
References = new List<AdvisoryObservationRawReferenceDocument>
{
new() { Type = "Advisory", Url = "https://example.com", Source = "vendor" }
},
ReconciledFrom = new List<string> { "source-a" },
Notes = new Dictionary<string, string> { ["note-key"] = "note-value" }
}
};
@@ -64,5 +76,9 @@ public sealed class AdvisoryObservationDocumentFactoryTests
Assert.Equal("CSAF", observation.Content.Format);
Assert.True(observation.Content.Raw?["example"]?.GetValue<bool>());
Assert.Equal("advisory", observation.Linkset.References[0].Type);
Assert.Equal(new[] { "CVE-2025-1234", "cve-2025-1234" }, observation.RawLinkset.Aliases);
Assert.Equal("Advisory", observation.RawLinkset.References[0].Type);
Assert.Equal("vendor", observation.RawLinkset.References[0].Source);
Assert.Equal("note-value", observation.RawLinkset.Notes["note-key"]);
}
}

View File

@@ -2,6 +2,6 @@ If you are working on this file you need to read docs/modules/excititor/ARCHITEC
# TASKS
| Task | Owner(s) | Depends on | Notes |
|---|---|---|---|
|EXCITITOR-ATTEST-01-003 Verification suite & observability|Team Excititor Attestation|EXCITITOR-ATTEST-01-002|DOING (2025-10-22) Continuing implementation: build `IVexAttestationVerifier`, wire metrics/logging, and add regression tests. Draft plan in `EXCITITOR-ATTEST-01-003-plan.md` (2025-10-19) guides scope; updating with worknotes as progress lands.|
|EXCITITOR-ATTEST-01-003 Verification suite & observability|Team Excititor Attestation|EXCITITOR-ATTEST-01-002|DOING (2025-10-22) Continuing implementation: build `IVexAttestationVerifier`, wire metrics/logging, and add regression tests. Draft plan in `EXCITITOR-ATTEST-01-003-plan.md` (2025-10-19) guides scope; updating with worknotes as progress lands.<br>2025-10-31: Verifier now tolerates duplicate source providers from AOC raw projections, downgrades offline Rekor verification to a degraded result, and enforces trusted signer registry checks with detailed diagnostics/tests.|
> Remark (2025-10-22): Added verifier implementation + metrics/tests; next steps include wiring into WebService/Worker flows and expanding negative-path coverage.

View File

@@ -173,7 +173,11 @@ internal sealed class VexAttestationVerifier : IVexAttestationVerifier
resultLabel = "degraded";
}
diagnostics["signature.state"] = "present";
if (rekorState is "offline" && resultLabel != "invalid")
{
resultLabel = "degraded";
}
return BuildResult(true);
}
catch (Exception ex)
@@ -206,6 +210,103 @@ internal sealed class VexAttestationVerifier : IVexAttestationVerifier
}
}
private async ValueTask<bool> VerifySignaturesAsync(
ReadOnlyMemory<byte> payloadBytes,
IReadOnlyList<DsseSignature> signatures,
ImmutableDictionary<string, string>.Builder diagnostics,
CancellationToken cancellationToken)
{
if (signatures is null || signatures.Count == 0)
{
diagnostics["signature.state"] = "missing";
return false;
}
if (_trustedSigners.Count == 0)
{
diagnostics["signature.state"] = "skipped";
diagnostics["signature.reason"] = "trust_not_configured";
return true;
}
if (_cryptoRegistry is null)
{
diagnostics["signature.state"] = "error";
diagnostics["signature.reason"] = "registry_unavailable";
return false;
}
foreach (var signature in signatures)
{
if (string.IsNullOrWhiteSpace(signature.Signature))
{
diagnostics["signature.state"] = "error";
diagnostics["signature.reason"] = "empty_signature";
return false;
}
if (string.IsNullOrWhiteSpace(signature.KeyId))
{
diagnostics["signature.state"] = "error";
diagnostics["signature.reason"] = "missing_key_id";
return false;
}
if (!_trustedSigners.TryGetValue(signature.KeyId, out var signerOptions))
{
diagnostics["signature.state"] = "error";
diagnostics["signature.reason"] = "untrusted_key";
diagnostics["signature.keyId"] = signature.KeyId;
return false;
}
byte[] signatureBytes;
try
{
signatureBytes = Convert.FromBase64String(signature.Signature);
}
catch (FormatException)
{
diagnostics["signature.state"] = "error";
diagnostics["signature.reason"] = "invalid_signature_encoding";
diagnostics["signature.keyId"] = signature.KeyId;
return false;
}
CryptoSignerResolution resolution;
try
{
resolution = _cryptoRegistry.ResolveSigner(
CryptoCapability.Verification,
signerOptions.Algorithm,
new CryptoKeyReference(signerOptions.KeyReference, signerOptions.ProviderHint));
}
catch (Exception ex)
{
diagnostics["signature.state"] = "error";
diagnostics["signature.reason"] = "resolution_failed";
diagnostics["signature.error"] = ex.GetType().Name;
diagnostics["signature.keyId"] = signature.KeyId;
return false;
}
var verified = await resolution.Signer
.VerifyAsync(payloadBytes, signatureBytes, cancellationToken)
.ConfigureAwait(false);
if (!verified)
{
diagnostics["signature.state"] = "error";
diagnostics["signature.reason"] = "verification_failed";
diagnostics["signature.keyId"] = signature.KeyId;
return false;
}
}
diagnostics["signature.state"] = "verified";
return true;
}
private static bool TryDeserializeEnvelope(
string envelopeJson,
out DsseEnvelope envelope,
@@ -473,17 +574,18 @@ internal sealed class VexAttestationVerifier : IVexAttestationVerifier
private static bool SetEquals(IReadOnlyCollection<string>? left, ImmutableArray<string> right)
{
if (left is null)
if (left is null || left.Count == 0)
{
return right.IsDefaultOrEmpty;
return right.IsDefaultOrEmpty || right.Length == 0;
}
if (left.Count != right.Length)
if (right.IsDefaultOrEmpty)
{
return false;
}
var leftSet = new HashSet<string>(left, StringComparer.Ordinal);
return right.All(leftSet.Contains);
var rightSet = new HashSet<string>(right, StringComparer.Ordinal);
return leftSet.SetEquals(rightSet);
}
}

View File

@@ -1,6 +1,9 @@
using System.Collections.Immutable;
using System.Text;
using Microsoft.Extensions.Logging.Abstractions;
using Microsoft.Extensions.Options;
using Microsoft.IdentityModel.Tokens;
using StellaOps.Cryptography;
using StellaOps.Excititor.Attestation.Dsse;
using StellaOps.Excititor.Attestation.Signing;
using StellaOps.Excititor.Attestation.Transparency;
@@ -65,6 +68,7 @@ public sealed class VexAttestationVerifierTests : IDisposable
Assert.True(verification.IsValid);
Assert.Equal("offline", verification.Diagnostics["rekor.state"]);
Assert.Equal("degraded", verification.Diagnostics["result"]);
}
[Fact]
@@ -106,7 +110,80 @@ public sealed class VexAttestationVerifierTests : IDisposable
Assert.Equal("invalid", verification.Diagnostics["result"]);
}
private async Task<(VexAttestationRequest Request, VexAttestationMetadata Metadata, string Envelope)> CreateSignedAttestationAsync(bool includeRekor = false)
[Fact]
public async Task VerifyAsync_HandlesDuplicateSourceProviders()
{
var (request, metadata, envelope) = await CreateSignedAttestationAsync(
includeRekor: false,
sourceProviders: ImmutableArray.Create("provider-a", "provider-a"));
var normalizedRequest = request with { SourceProviders = ImmutableArray.Create("provider-a") };
var verifier = CreateVerifier(options => options.RequireTransparencyLog = false);
var verification = await verifier.VerifyAsync(
new VexAttestationVerificationRequest(normalizedRequest, metadata, envelope),
CancellationToken.None);
Assert.True(verification.IsValid);
Assert.Equal("valid", verification.Diagnostics["result"]);
}
[Fact]
public async Task VerifyAsync_ReturnsValid_WhenTrustedSignerConfigured()
{
var (request, metadata, envelope) = await CreateSignedAttestationAsync(includeRekor: false);
var registry = new StubCryptoProviderRegistry(success: true);
var verifier = CreateVerifier(options =>
{
options.RequireTransparencyLog = false;
options.RequireSignatureVerification = true;
options.TrustedSigners = ImmutableDictionary<string, VexAttestationVerificationOptions.TrustedSignerOptions>.Empty.Add(
"key",
new VexAttestationVerificationOptions.TrustedSignerOptions
{
Algorithm = StubCryptoProviderRegistry.Algorithm,
KeyReference = StubCryptoProviderRegistry.KeyReference
});
}, transparency: null, registry: registry);
var verification = await verifier.VerifyAsync(
new VexAttestationVerificationRequest(request, metadata, envelope),
CancellationToken.None);
Assert.True(verification.IsValid);
Assert.Equal("verified", verification.Diagnostics["signature.state"]);
}
[Fact]
public async Task VerifyAsync_ReturnsInvalid_WhenSignatureFailsAndRequired()
{
var (request, metadata, envelope) = await CreateSignedAttestationAsync(includeRekor: false);
var registry = new StubCryptoProviderRegistry(success: false);
var verifier = CreateVerifier(options =>
{
options.RequireTransparencyLog = false;
options.RequireSignatureVerification = true;
options.TrustedSigners = ImmutableDictionary<string, VexAttestationVerificationOptions.TrustedSignerOptions>.Empty.Add(
"key",
new VexAttestationVerificationOptions.TrustedSignerOptions
{
Algorithm = StubCryptoProviderRegistry.Algorithm,
KeyReference = StubCryptoProviderRegistry.KeyReference
});
}, transparency: null, registry: registry);
var verification = await verifier.VerifyAsync(
new VexAttestationVerificationRequest(request, metadata, envelope),
CancellationToken.None);
Assert.False(verification.IsValid);
Assert.Equal("error", verification.Diagnostics["signature.state"]);
Assert.Equal("verification_failed", verification.Diagnostics["signature.reason"]);
}
private async Task<(VexAttestationRequest Request, VexAttestationMetadata Metadata, string Envelope)> CreateSignedAttestationAsync(
bool includeRekor = false,
ImmutableArray<string>? sourceProviders = null)
{
var signer = new FakeSigner();
var builder = new VexDsseBuilder(signer, NullLogger<VexDsseBuilder>.Instance);
@@ -115,13 +192,14 @@ public sealed class VexAttestationVerifierTests : IDisposable
var verifier = CreateVerifier(options => options.RequireTransparencyLog = false);
var client = new VexAttestationClient(builder, options, NullLogger<VexAttestationClient>.Instance, verifier, transparency);
var providers = sourceProviders ?? ImmutableArray.Create("provider-a");
var request = new VexAttestationRequest(
ExportId: "exports/unit-test",
QuerySignature: new VexQuerySignature("filters"),
Artifact: new VexContentAddress("sha256", "cafebabe"),
Format: VexExportFormat.Json,
CreatedAt: DateTimeOffset.UtcNow,
SourceProviders: ImmutableArray.Create("provider-a"),
SourceProviders: providers,
Metadata: ImmutableDictionary<string, string>.Empty);
var response = await client.SignAsync(request, CancellationToken.None);
@@ -129,7 +207,10 @@ public sealed class VexAttestationVerifierTests : IDisposable
return (request, response.Attestation, envelope);
}
private VexAttestationVerifier CreateVerifier(Action<VexAttestationVerificationOptions>? configureOptions = null, ITransparencyLogClient? transparency = null)
private VexAttestationVerifier CreateVerifier(
Action<VexAttestationVerificationOptions>? configureOptions = null,
ITransparencyLogClient? transparency = null,
ICryptoProviderRegistry? registry = null)
{
var options = new VexAttestationVerificationOptions();
configureOptions?.Invoke(options);
@@ -137,7 +218,8 @@ public sealed class VexAttestationVerifierTests : IDisposable
NullLogger<VexAttestationVerifier>.Instance,
transparency,
Options.Create(options),
_metrics);
_metrics,
registry);
}
public void Dispose()
@@ -147,8 +229,10 @@ public sealed class VexAttestationVerifierTests : IDisposable
private sealed class FakeSigner : IVexSigner
{
internal static readonly string SignatureBase64 = Convert.ToBase64String(Encoding.UTF8.GetBytes("signature"));
public ValueTask<VexSignedPayload> SignAsync(ReadOnlyMemory<byte> payload, CancellationToken cancellationToken)
=> ValueTask.FromResult(new VexSignedPayload("signature", "key"));
=> ValueTask.FromResult(new VexSignedPayload(SignatureBase64, "key"));
}
private sealed class FakeTransparencyLogClient : ITransparencyLogClient
@@ -168,4 +252,70 @@ public sealed class VexAttestationVerifierTests : IDisposable
public ValueTask<bool> VerifyAsync(string entryLocation, CancellationToken cancellationToken)
=> throw new HttpRequestException("rekor unavailable");
}
private sealed class StubCryptoProviderRegistry : ICryptoProviderRegistry
{
public const string Algorithm = "ed25519";
public const string KeyReference = "stub-key";
private readonly StubCryptoSigner _signer;
private readonly IReadOnlyCollection<ICryptoProvider> _providers = Array.Empty<ICryptoProvider>();
public StubCryptoProviderRegistry(bool success)
{
_signer = new StubCryptoSigner("key", Algorithm, success);
}
public IReadOnlyCollection<ICryptoProvider> Providers => _providers;
public bool TryResolve(string preferredProvider, out ICryptoProvider provider)
{
provider = null!;
return false;
}
public ICryptoProvider ResolveOrThrow(CryptoCapability capability, string algorithmId)
=> throw new NotSupportedException();
public CryptoSignerResolution ResolveSigner(
CryptoCapability capability,
string algorithmId,
CryptoKeyReference keyReference,
string? preferredProvider = null)
{
if (!string.Equals(keyReference.KeyId, _signer.KeyId, StringComparison.Ordinal))
{
throw new InvalidOperationException($"Unknown key '{keyReference.KeyId}'.");
}
return new CryptoSignerResolution(_signer, "stub");
}
}
private sealed class StubCryptoSigner : ICryptoSigner
{
private readonly bool _success;
private readonly byte[] _expectedSignature;
public StubCryptoSigner(string keyId, string algorithmId, bool success)
{
KeyId = keyId;
AlgorithmId = algorithmId;
_success = success;
_expectedSignature = Convert.FromBase64String(FakeSigner.SignatureBase64);
}
public string KeyId { get; }
public string AlgorithmId { get; }
public ValueTask<byte[]> SignAsync(ReadOnlyMemory<byte> data, CancellationToken cancellationToken = default)
=> throw new NotSupportedException();
public ValueTask<bool> VerifyAsync(ReadOnlyMemory<byte> data, ReadOnlyMemory<byte> signature, CancellationToken cancellationToken = default)
=> ValueTask.FromResult(_success && signature.Span.SequenceEqual(_expectedSignature.Span));
public JsonWebKey ExportPublicJsonWebKey()
=> new JsonWebKey();
}
}

View File

@@ -7,6 +7,7 @@
> 2025-10-26: Blocked while bootstrapping DSL parser/evaluator; remaining grammar coverage (profile keywords, condition parsing) and rule evaluation semantics still pending to satisfy acceptance tests.
| POLICY-ENGINE-20-003 | TODO | Policy Guild, Concelier Core Guild, Excititor Core Guild | POLICY-ENGINE-20-001, CONCELIER-POLICY-20-002, EXCITITOR-POLICY-20-002 | Implement selection joiners resolving SBOM↔advisory↔VEX tuples using linksets and PURL equivalence tables, with deterministic batching. | Joiners fetch correct candidate sets in integration tests; batching meets memory targets; explain traces list input provenance. |
> 2025-10-26: Scheduler DTO contracts for runs/diffs/explains available (`src/Scheduler/__Libraries/StellaOps.Scheduler.Models/docs/SCHED-MODELS-20-001-POLICY-RUNS.md`); consume `PolicyRunRequest/Status/DiffSummary` from samples under `samples/api/scheduler/`.
> 2025-10-31: Raw Concelier observations expose `rawLinkset`; update joiners/tests to consume it and align rollout/backfill per `docs/dev/raw-linkset-backfill-plan.md`.
| POLICY-ENGINE-20-004 | TODO | Policy Guild, Platform Storage Guild | POLICY-ENGINE-20-003, CONCELIER-POLICY-20-003, EXCITITOR-POLICY-20-003 | Ship materialization writer that upserts into `effective_finding_{policyId}` with append-only history, tenant scoping, and trace references. | Writes restricted to Policy Engine identity; idempotent upserts proven via tests; collections indexed per design and docs updated. |
| POLICY-ENGINE-20-005 | TODO | Policy Guild, Security Engineering | POLICY-ENGINE-20-002 | Enforce determinism guard banning wall-clock, RNG, and network usage during evaluation via static analysis + runtime sandbox. | Guard blocks forbidden APIs in unit/integration tests; violations emit `ERR_POL_004`; CI analyzer wired. |
| POLICY-ENGINE-20-006 | TODO | Policy Guild, Scheduler Worker Guild | POLICY-ENGINE-20-003, POLICY-ENGINE-20-004, SCHED-WORKER-20-301 | Implement incremental orchestrator reacting to advisory/vex/SBOM change streams and scheduling partial policy re-evaluations. | Change stream listeners enqueue affected tuples with dedupe; orchestrator meets 5 min SLA in perf tests; metrics exposed (`policy_run_seconds`). |