feat: Add initial implementation of Vulnerability Resolver Jobs
Some checks failed
Docs CI / lint-and-preview (push) Has been cancelled
Some checks failed
Docs CI / lint-and-preview (push) Has been cancelled
- Created project for StellaOps.Scanner.Analyzers.Native.Tests with necessary dependencies. - Documented roles and guidelines in AGENTS.md for Scheduler module. - Implemented IResolverJobService interface and InMemoryResolverJobService for handling resolver jobs. - Added ResolverBacklogNotifier and ResolverBacklogService for monitoring job metrics. - Developed API endpoints for managing resolver jobs and retrieving metrics. - Defined models for resolver job requests and responses. - Integrated dependency injection for resolver job services. - Implemented ImpactIndexSnapshot for persisting impact index data. - Introduced SignalsScoringOptions for configurable scoring weights in reachability scoring. - Added unit tests for ReachabilityScoringService and RuntimeFactsIngestionService. - Created dotnet-filter.sh script to handle command-line arguments for dotnet. - Established nuget-prime project for managing package downloads.
This commit is contained in:
@@ -1,37 +1,40 @@
|
||||
# AGENTS
|
||||
## Role
|
||||
Domain source of truth for VEX statements, consensus rollups, and trust policy orchestration across all Excititor services.
|
||||
# Excititor Core Charter
|
||||
|
||||
## Mission
|
||||
Provide ingestion/domain logic for VEX observations and linksets under the Aggregation-Only Contract: store raw facts, provenance, and precedence pointers without computing consensus or severity.
|
||||
|
||||
## Scope
|
||||
- Records for raw document metadata, normalized claims, consensus projections, and export descriptors.
|
||||
- Policy + weighting engine that projects provider trust tiers into consensus status outcomes.
|
||||
- Connector, normalizer, export, and attestation contracts shared by WebService, Worker, and plug-ins.
|
||||
- Deterministic hashing utilities (query signatures, artifact digests, attestation subjects).
|
||||
## Participants
|
||||
- Excititor WebService uses the models to persist ingress/egress payloads and to perform consensus mutations.
|
||||
- Excititor Worker executes reconciliation and verification routines using policy helpers defined here.
|
||||
- Export/Attestation modules depend on record definitions for envelopes and manifest payloads.
|
||||
## Interfaces & contracts
|
||||
- `IVexConnector`, `INormalizer`, `IExportEngine`, `ITransparencyLogClient`, `IArtifactStore`, and policy abstractions for consensus resolution.
|
||||
- Value objects for provider metadata, VexClaim, VexConsensusEntry, ExportManifest, QuerySignature.
|
||||
- Deterministic comparer utilities and stable JSON serialization helpers for tests and cache keys.
|
||||
## In/Out of scope
|
||||
In: domain invariants, policy evaluation helpers, deterministic serialization, shared abstractions.
|
||||
Out: Mongo persistence implementations, HTTP endpoints, background scheduling, concrete connector logic.
|
||||
## Observability & security expectations
|
||||
- Avoid secret handling; provide structured logging extension methods for consensus decisions.
|
||||
- Emit correlation identifiers and query signatures without embedding PII.
|
||||
- Ensure deterministic logging order to keep reproducibility guarantees intact.
|
||||
## Tests
|
||||
- Unit coverage lives in `../StellaOps.Excititor.Core.Tests` (to be scaffolded) focusing on consensus, policy gates, and serialization determinism.
|
||||
- Golden fixtures must rely on canonical JSON snapshots produced via stable serializers.
|
||||
- Working directory: `src/Excititor/__Libraries/StellaOps.Excititor.Core`
|
||||
- Domain models, validators, linkset extraction, idempotent upserts, tenant guards, and invariants shared by WebService/Worker.
|
||||
- No UI concerns; no policy evaluation.
|
||||
|
||||
## Required Reading
|
||||
- `docs/modules/excititor/architecture.md`
|
||||
- `docs/modules/platform/architecture-overview.md`
|
||||
- `docs/modules/excititor/vex_observations.md`
|
||||
- `docs/ingestion/aggregation-only-contract.md`
|
||||
- `docs/modules/excititor/implementation_plan.md`
|
||||
|
||||
## Working Agreement
|
||||
- 1. Update task status to `DOING`/`DONE` in both correspoding sprint file `/docs/implplan/SPRINT_*.md` and the local `TASKS.md` when you start or finish work.
|
||||
- 2. Review this charter and the Required Reading documents before coding; confirm prerequisites are met.
|
||||
- 3. Keep changes deterministic (stable ordering, timestamps, hashes) and align with offline/air-gap expectations.
|
||||
- 4. Coordinate doc updates, tests, and cross-guild communication whenever contracts or workflows change.
|
||||
- 5. Revert to `TODO` if you pause the task without shipping changes; leave notes in commit/PR descriptions for context.
|
||||
## Roles
|
||||
- Backend library engineer (.NET 10 / C# preview).
|
||||
- QA automation (unit + integration against Mongo fixtures).
|
||||
|
||||
## Working Agreements
|
||||
1. Update sprint status on task transitions; log notable decisions in sprint Execution Log.
|
||||
2. Enforce idempotent ingestion: uniqueness on `(vendor, upstreamId, contentHash, tenant)` and append-only supersede chains.
|
||||
3. Preserve provenance fields and reconciled-from metadata when building linksets; never drop issuer data.
|
||||
4. Tenant isolation is mandatory: all queries/commands include tenant scope; cross-tenant writes must be rejected.
|
||||
5. Offline-first; avoid fetching external resources at runtime.
|
||||
|
||||
## Testing & Determinism
|
||||
- Write deterministic tests: seeded clocks/GUIDs, stable ordering of collections, ISO-8601 UTC timestamps.
|
||||
- Cover linkset extraction ordering, supersede chain construction, and duplicate prevention.
|
||||
- Use Mongo in-memory/test harness fixtures; do not rely on live services.
|
||||
|
||||
## Boundaries
|
||||
- Do not embed Policy Engine rules or Cartographer schemas here; expose contracts for consumers instead.
|
||||
- Keep serialization shapes versioned; document breaking changes in `docs/modules/excititor/changes.md` if created.
|
||||
|
||||
## Ready-to-Start Checklist
|
||||
- Required docs reviewed.
|
||||
- Deterministic test fixtures in place.
|
||||
- Feature flags/config options identified for any behavioral changes.
|
||||
|
||||
@@ -0,0 +1,154 @@
|
||||
using System;
|
||||
using System.Collections.Generic;
|
||||
using System.Collections.Immutable;
|
||||
using System.Linq;
|
||||
|
||||
namespace StellaOps.Excititor.Core.Observations;
|
||||
|
||||
public static class VexLinksetUpdatedEventFactory
|
||||
{
|
||||
public const string EventType = "vex.linkset.updated";
|
||||
|
||||
public static VexLinksetUpdatedEvent Create(
|
||||
string tenant,
|
||||
string linksetId,
|
||||
string vulnerabilityId,
|
||||
string productKey,
|
||||
IEnumerable<VexObservation> observations,
|
||||
IEnumerable<VexObservationDisagreement> disagreements,
|
||||
DateTimeOffset createdAtUtc)
|
||||
{
|
||||
var normalizedTenant = Ensure(tenant, nameof(tenant)).ToLowerInvariant();
|
||||
var normalizedLinksetId = Ensure(linksetId, nameof(linksetId));
|
||||
var normalizedVulnerabilityId = Ensure(vulnerabilityId, nameof(vulnerabilityId));
|
||||
var normalizedProductKey = Ensure(productKey, nameof(productKey));
|
||||
|
||||
var observationRefs = (observations ?? Enumerable.Empty<VexObservation>())
|
||||
.Where(obs => obs is not null)
|
||||
.SelectMany(obs => obs.Statements.Select(statement => new VexLinksetObservationRef(
|
||||
observationId: obs.ObservationId,
|
||||
providerId: obs.ProviderId,
|
||||
status: statement.Status.ToString().ToLowerInvariant(),
|
||||
confidence: statement.Signals?.Severity?.Score)))
|
||||
.Distinct(VexLinksetObservationRefComparer.Instance)
|
||||
.OrderBy(refItem => refItem.ProviderId, StringComparer.OrdinalIgnoreCase)
|
||||
.ThenBy(refItem => refItem.ObservationId, StringComparer.Ordinal)
|
||||
.ToImmutableArray();
|
||||
|
||||
var disagreementList = (disagreements ?? Enumerable.Empty<VexObservationDisagreement>())
|
||||
.Where(d => d is not null)
|
||||
.Select(d => new VexObservationDisagreement(
|
||||
providerId: Normalize(d.ProviderId),
|
||||
status: Normalize(d.Status),
|
||||
justification: VexObservation.TrimToNull(d.Justification),
|
||||
confidence: d.Confidence is null ? null : Math.Clamp(d.Confidence.Value, 0.0, 1.0)))
|
||||
.Distinct(DisagreementComparer.Instance)
|
||||
.OrderBy(d => d.ProviderId, StringComparer.OrdinalIgnoreCase)
|
||||
.ThenBy(d => d.Status, StringComparer.OrdinalIgnoreCase)
|
||||
.ThenBy(d => d.Justification ?? string.Empty, StringComparer.OrdinalIgnoreCase)
|
||||
.ToImmutableArray();
|
||||
|
||||
return new VexLinksetUpdatedEvent(
|
||||
EventType,
|
||||
normalizedTenant,
|
||||
normalizedLinksetId,
|
||||
normalizedVulnerabilityId,
|
||||
normalizedProductKey,
|
||||
observationRefs,
|
||||
disagreementList,
|
||||
createdAtUtc);
|
||||
}
|
||||
|
||||
private static string Normalize(string value) => Ensure(value, nameof(value));
|
||||
|
||||
private static string Ensure(string value, string name)
|
||||
{
|
||||
if (string.IsNullOrWhiteSpace(value))
|
||||
{
|
||||
throw new ArgumentException($"{name} cannot be null or whitespace", name);
|
||||
}
|
||||
return value.Trim();
|
||||
}
|
||||
|
||||
private sealed class VexLinksetObservationRefComparer : IEqualityComparer<VexLinksetObservationRef>
|
||||
{
|
||||
public static readonly VexLinksetObservationRefComparer Instance = new();
|
||||
|
||||
public bool Equals(VexLinksetObservationRef? x, VexLinksetObservationRef? y)
|
||||
{
|
||||
if (ReferenceEquals(x, y))
|
||||
{
|
||||
return true;
|
||||
}
|
||||
|
||||
if (x is null || y is null)
|
||||
{
|
||||
return false;
|
||||
}
|
||||
|
||||
return string.Equals(x.ObservationId, y.ObservationId, StringComparison.Ordinal)
|
||||
&& string.Equals(x.ProviderId, y.ProviderId, StringComparison.OrdinalIgnoreCase)
|
||||
&& string.Equals(x.Status, y.Status, StringComparison.OrdinalIgnoreCase)
|
||||
&& Nullable.Equals(x.Confidence, y.Confidence);
|
||||
}
|
||||
|
||||
public int GetHashCode(VexLinksetObservationRef obj)
|
||||
{
|
||||
var hash = new HashCode();
|
||||
hash.Add(obj.ObservationId, StringComparer.Ordinal);
|
||||
hash.Add(obj.ProviderId, StringComparer.OrdinalIgnoreCase);
|
||||
hash.Add(obj.Status, StringComparer.OrdinalIgnoreCase);
|
||||
hash.Add(obj.Confidence);
|
||||
return hash.ToHashCode();
|
||||
}
|
||||
}
|
||||
|
||||
private sealed class DisagreementComparer : IEqualityComparer<VexObservationDisagreement>
|
||||
{
|
||||
public static readonly DisagreementComparer Instance = new();
|
||||
|
||||
public bool Equals(VexObservationDisagreement? x, VexObservationDisagreement? y)
|
||||
{
|
||||
if (ReferenceEquals(x, y))
|
||||
{
|
||||
return true;
|
||||
}
|
||||
|
||||
if (x is null || y is null)
|
||||
{
|
||||
return false;
|
||||
}
|
||||
|
||||
return string.Equals(x.ProviderId, y.ProviderId, StringComparison.OrdinalIgnoreCase)
|
||||
&& string.Equals(x.Status, y.Status, StringComparison.OrdinalIgnoreCase)
|
||||
&& string.Equals(x.Justification, y.Justification, StringComparison.OrdinalIgnoreCase)
|
||||
&& Nullable.Equals(x.Confidence, y.Confidence);
|
||||
}
|
||||
|
||||
public int GetHashCode(VexObservationDisagreement obj)
|
||||
{
|
||||
var hash = new HashCode();
|
||||
hash.Add(obj.ProviderId, StringComparer.OrdinalIgnoreCase);
|
||||
hash.Add(obj.Status, StringComparer.OrdinalIgnoreCase);
|
||||
hash.Add(obj.Justification, StringComparer.OrdinalIgnoreCase);
|
||||
hash.Add(obj.Confidence);
|
||||
return hash.ToHashCode();
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
public sealed record VexLinksetObservationRef(
|
||||
string ObservationId,
|
||||
string ProviderId,
|
||||
string Status,
|
||||
double? Confidence);
|
||||
|
||||
public sealed record VexLinksetUpdatedEvent(
|
||||
string EventType,
|
||||
string Tenant,
|
||||
string LinksetId,
|
||||
string VulnerabilityId,
|
||||
string ProductKey,
|
||||
ImmutableArray<VexLinksetObservationRef> Observations,
|
||||
ImmutableArray<VexObservationDisagreement> Disagreements,
|
||||
DateTimeOffset CreatedAtUtc);
|
||||
@@ -352,21 +352,23 @@ public sealed record VexObservationStatement
|
||||
}
|
||||
}
|
||||
|
||||
public sealed record VexObservationLinkset
|
||||
{
|
||||
public VexObservationLinkset(
|
||||
IEnumerable<string>? aliases,
|
||||
IEnumerable<string>? purls,
|
||||
IEnumerable<string>? cpes,
|
||||
IEnumerable<VexObservationReference>? references,
|
||||
IEnumerable<string>? reconciledFrom = null)
|
||||
{
|
||||
Aliases = NormalizeSet(aliases, toLower: true);
|
||||
Purls = NormalizeSet(purls, toLower: false);
|
||||
Cpes = NormalizeSet(cpes, toLower: false);
|
||||
References = NormalizeReferences(references);
|
||||
ReconciledFrom = NormalizeSet(reconciledFrom, toLower: false);
|
||||
}
|
||||
public sealed record VexObservationLinkset
|
||||
{
|
||||
public VexObservationLinkset(
|
||||
IEnumerable<string>? aliases,
|
||||
IEnumerable<string>? purls,
|
||||
IEnumerable<string>? cpes,
|
||||
IEnumerable<VexObservationReference>? references,
|
||||
IEnumerable<string>? reconciledFrom = null,
|
||||
IEnumerable<VexObservationDisagreement>? disagreements = null)
|
||||
{
|
||||
Aliases = NormalizeSet(aliases, toLower: true);
|
||||
Purls = NormalizeSet(purls, toLower: false);
|
||||
Cpes = NormalizeSet(cpes, toLower: false);
|
||||
References = NormalizeReferences(references);
|
||||
ReconciledFrom = NormalizeSet(reconciledFrom, toLower: false);
|
||||
Disagreements = NormalizeDisagreements(disagreements);
|
||||
}
|
||||
|
||||
public ImmutableArray<string> Aliases { get; }
|
||||
|
||||
@@ -374,9 +376,11 @@ public sealed record VexObservationLinkset
|
||||
|
||||
public ImmutableArray<string> Cpes { get; }
|
||||
|
||||
public ImmutableArray<VexObservationReference> References { get; }
|
||||
|
||||
public ImmutableArray<string> ReconciledFrom { get; }
|
||||
public ImmutableArray<VexObservationReference> References { get; }
|
||||
|
||||
public ImmutableArray<string> ReconciledFrom { get; }
|
||||
|
||||
public ImmutableArray<VexObservationDisagreement> Disagreements { get; }
|
||||
|
||||
private static ImmutableArray<string> NormalizeSet(IEnumerable<string>? values, bool toLower)
|
||||
{
|
||||
@@ -419,19 +423,116 @@ public sealed record VexObservationLinkset
|
||||
set.Add(reference);
|
||||
}
|
||||
|
||||
return set.Count == 0 ? ImmutableArray<VexObservationReference>.Empty : set.ToImmutableArray();
|
||||
}
|
||||
}
|
||||
return set.Count == 0 ? ImmutableArray<VexObservationReference>.Empty : set.ToImmutableArray();
|
||||
}
|
||||
|
||||
private static ImmutableArray<VexObservationDisagreement> NormalizeDisagreements(
|
||||
IEnumerable<VexObservationDisagreement>? disagreements)
|
||||
{
|
||||
if (disagreements is null)
|
||||
{
|
||||
return ImmutableArray<VexObservationDisagreement>.Empty;
|
||||
}
|
||||
|
||||
var comparer = Comparer<VexObservationDisagreement>.Create(static (left, right) =>
|
||||
{
|
||||
var providerCompare = StringComparer.OrdinalIgnoreCase.Compare(left.ProviderId, right.ProviderId);
|
||||
if (providerCompare != 0)
|
||||
{
|
||||
return providerCompare;
|
||||
}
|
||||
|
||||
return StringComparer.OrdinalIgnoreCase.Compare(left.Status, right.Status);
|
||||
});
|
||||
|
||||
var set = new SortedSet<VexObservationDisagreement>(Comparer<VexObservationDisagreement>.Create((a, b) =>
|
||||
{
|
||||
var providerCompare = StringComparer.OrdinalIgnoreCase.Compare(a.ProviderId, b.ProviderId);
|
||||
if (providerCompare != 0)
|
||||
{
|
||||
return providerCompare;
|
||||
}
|
||||
|
||||
var statusCompare = StringComparer.OrdinalIgnoreCase.Compare(a.Status, b.Status);
|
||||
if (statusCompare != 0)
|
||||
{
|
||||
return statusCompare;
|
||||
}
|
||||
|
||||
var justificationCompare = StringComparer.OrdinalIgnoreCase.Compare(
|
||||
a.Justification ?? string.Empty,
|
||||
b.Justification ?? string.Empty);
|
||||
if (justificationCompare != 0)
|
||||
{
|
||||
return justificationCompare;
|
||||
}
|
||||
|
||||
return Nullable.Compare(a.Confidence, b.Confidence);
|
||||
}));
|
||||
|
||||
foreach (var disagreement in disagreements)
|
||||
{
|
||||
if (disagreement is null)
|
||||
{
|
||||
continue;
|
||||
}
|
||||
|
||||
var normalizedProvider = VexObservation.TrimToNull(disagreement.ProviderId);
|
||||
var normalizedStatus = VexObservation.TrimToNull(disagreement.Status);
|
||||
|
||||
if (normalizedProvider is null || normalizedStatus is null)
|
||||
{
|
||||
continue;
|
||||
}
|
||||
|
||||
var normalizedJustification = VexObservation.TrimToNull(disagreement.Justification);
|
||||
var clampedConfidence = disagreement.Confidence is null
|
||||
? null
|
||||
: Math.Clamp(disagreement.Confidence.Value, 0.0, 1.0);
|
||||
|
||||
set.Add(new VexObservationDisagreement(
|
||||
normalizedProvider,
|
||||
normalizedStatus,
|
||||
normalizedJustification,
|
||||
clampedConfidence));
|
||||
}
|
||||
|
||||
return set.Count == 0 ? ImmutableArray<VexObservationDisagreement>.Empty : set.ToImmutableArray();
|
||||
}
|
||||
}
|
||||
|
||||
public sealed record VexObservationReference
|
||||
{
|
||||
public VexObservationReference(string type, string url)
|
||||
{
|
||||
Type = VexObservation.EnsureNotNullOrWhiteSpace(type, nameof(type));
|
||||
Url = VexObservation.EnsureNotNullOrWhiteSpace(url, nameof(url));
|
||||
}
|
||||
public sealed record VexObservationReference
|
||||
{
|
||||
public VexObservationReference(string type, string url)
|
||||
{
|
||||
Type = VexObservation.EnsureNotNullOrWhiteSpace(type, nameof(type));
|
||||
Url = VexObservation.EnsureNotNullOrWhiteSpace(url, nameof(url));
|
||||
}
|
||||
|
||||
public string Type { get; }
|
||||
|
||||
public string Url { get; }
|
||||
}
|
||||
|
||||
public string Url { get; }
|
||||
}
|
||||
|
||||
public sealed record VexObservationDisagreement
|
||||
{
|
||||
public VexObservationDisagreement(
|
||||
string providerId,
|
||||
string status,
|
||||
string? justification,
|
||||
double? confidence)
|
||||
{
|
||||
ProviderId = VexObservation.EnsureNotNullOrWhiteSpace(providerId, nameof(providerId));
|
||||
Status = VexObservation.EnsureNotNullOrWhiteSpace(status, nameof(status));
|
||||
Justification = justification;
|
||||
Confidence = confidence;
|
||||
}
|
||||
|
||||
public string ProviderId { get; }
|
||||
|
||||
public string Status { get; }
|
||||
|
||||
public string? Justification { get; }
|
||||
|
||||
public double? Confidence { get; }
|
||||
}
|
||||
|
||||
@@ -1,35 +1,37 @@
|
||||
# AGENTS
|
||||
## Role
|
||||
MongoDB persistence layer for Excititor raw documents, claims, consensus snapshots, exports, and cache metadata.
|
||||
# Excititor Storage (Mongo) Charter
|
||||
|
||||
## Mission
|
||||
Provide Mongo-backed persistence for Excititor ingestion, linksets, and observations with deterministic schemas, indexes, and migrations; keep aggregation-only semantics intact.
|
||||
|
||||
## Scope
|
||||
- Collection schemas, Bson class maps, repositories, and transactional write patterns for ingest/export flows.
|
||||
- GridFS integration for raw source documents and artifact metadata persistence.
|
||||
- Migrations, index builders, and bootstrap routines aligned with offline-first deployments.
|
||||
- Deterministic query helpers used by WebService, Worker, and Export modules.
|
||||
## Participants
|
||||
- WebService invokes repositories to store ingest runs, recompute consensus, and register exports.
|
||||
- Worker relies on repositories for resume markers, retry queues, and cache GC flows.
|
||||
- Export/Attestation modules pull stored claims/consensus data for snapshot building.
|
||||
## Interfaces & contracts
|
||||
- Repository abstractions (`IVexRawStore`, `IVexClaimStore`, `IVexConsensusStore`, `IVexExportStore`, `IVexCacheIndex`) and migration host interfaces.
|
||||
- Diagnostics hooks providing collection health metrics and schema validation results.
|
||||
## In/Out of scope
|
||||
In: MongoDB data access, migrations, transactional semantics, schema documentation.
|
||||
Out: domain modeling (Core), policy evaluation (Policy), HTTP surfaces (WebService).
|
||||
## Observability & security expectations
|
||||
- Emit structured logs for collection/migration events including revision ids and elapsed timings.
|
||||
- Expose health metrics (counts, queue backlog) and publish to OpenTelemetry when enabled.
|
||||
- Ensure no raw secret material is logged; mask tokens/URLs in diagnostics.
|
||||
## Tests
|
||||
- Integration fixtures (Mongo runner) and schema regression tests will reside in `../StellaOps.Excititor.Storage.Mongo.Tests`.
|
||||
- Working directory: `src/Excititor/__Libraries/StellaOps.Excititor.Storage.Mongo`
|
||||
- Collections, indexes, migrations, repository abstractions, and data access helpers shared by WebService/Worker/Core.
|
||||
|
||||
## Required Reading
|
||||
- `docs/modules/excititor/architecture.md`
|
||||
- `docs/modules/platform/architecture-overview.md`
|
||||
- `docs/modules/excititor/vex_observations.md`
|
||||
- `docs/ingestion/aggregation-only-contract.md`
|
||||
- `docs/modules/excititor/implementation_plan.md`
|
||||
|
||||
## Working Agreement
|
||||
- 1. Update task status to `DOING`/`DONE` in both correspoding sprint file `/docs/implplan/SPRINT_*.md` and the local `TASKS.md` when you start or finish work.
|
||||
- 2. Review this charter and the Required Reading documents before coding; confirm prerequisites are met.
|
||||
- 3. Keep changes deterministic (stable ordering, timestamps, hashes) and align with offline/air-gap expectations.
|
||||
- 4. Coordinate doc updates, tests, and cross-guild communication whenever contracts or workflows change.
|
||||
- 5. Revert to `TODO` if you pause the task without shipping changes; leave notes in commit/PR descriptions for context.
|
||||
## Roles
|
||||
- Backend/storage engineer (.NET 10, MongoDB driver ≥3.0).
|
||||
- QA automation (repository + migration tests).
|
||||
|
||||
## Working Agreements
|
||||
1. Maintain deterministic migrations; record new indexes and shapes in sprint `Execution Log` and module docs if added.
|
||||
2. Enforce tenant scope in all queries; include partition keys in indexes where applicable.
|
||||
3. No consensus/weighting logic; store raw facts, provenance, and precedence pointers only.
|
||||
4. Offline-first; no runtime external calls.
|
||||
|
||||
## Testing & Determinism
|
||||
- Use Mongo test fixtures/in-memory harness with seeded data; assert index presence and sort stability.
|
||||
- Keep timestamps UTC ISO-8601 and ordering explicit (e.g., vendor, upstreamId, version, createdUtc).
|
||||
- Avoid nondeterministic ObjectId/GUID usage in tests; seed values.
|
||||
|
||||
## Boundaries
|
||||
- Do not embed Policy Engine or Cartographer schemas; consume published contracts.
|
||||
- Config via DI/appsettings; no hard-coded connection strings.
|
||||
|
||||
## Ready-to-Start Checklist
|
||||
- Required docs reviewed.
|
||||
- Test fixture database prepared; migrations scripted and reversible where possible.
|
||||
|
||||
@@ -0,0 +1,79 @@
|
||||
using System.Threading;
|
||||
using System.Threading.Tasks;
|
||||
using MongoDB.Driver;
|
||||
|
||||
namespace StellaOps.Excititor.Storage.Mongo.Migrations;
|
||||
|
||||
internal sealed class VexObservationCollectionsMigration : IVexMongoMigration
|
||||
{
|
||||
public string Id => "20251117-observations-linksets";
|
||||
|
||||
public async ValueTask ExecuteAsync(IMongoDatabase database, CancellationToken cancellationToken)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(database);
|
||||
|
||||
await EnsureObservationsIndexesAsync(database, cancellationToken).ConfigureAwait(false);
|
||||
await EnsureLinksetIndexesAsync(database, cancellationToken).ConfigureAwait(false);
|
||||
}
|
||||
|
||||
private static Task EnsureObservationsIndexesAsync(IMongoDatabase database, CancellationToken cancellationToken)
|
||||
{
|
||||
var collection = database.GetCollection<VexObservationRecord>(VexMongoCollectionNames.Observations);
|
||||
|
||||
var tenantObservationIndex = Builders<VexObservationRecord>.IndexKeys
|
||||
.Ascending(x => x.Tenant)
|
||||
.Ascending(x => x.ObservationId);
|
||||
|
||||
var tenantVulnIndex = Builders<VexObservationRecord>.IndexKeys
|
||||
.Ascending(x => x.Tenant)
|
||||
.Ascending(x => x.VulnerabilityId);
|
||||
|
||||
var tenantProductIndex = Builders<VexObservationRecord>.IndexKeys
|
||||
.Ascending(x => x.Tenant)
|
||||
.Ascending(x => x.ProductKey);
|
||||
|
||||
var tenantDigestIndex = Builders<VexObservationRecord>.IndexKeys
|
||||
.Ascending(x => x.Tenant)
|
||||
.Ascending("Document.Digest");
|
||||
|
||||
var tenantProviderStatusIndex = Builders<VexObservationRecord>.IndexKeys
|
||||
.Ascending(x => x.Tenant)
|
||||
.Ascending(x => x.ProviderId)
|
||||
.Ascending(x => x.Status);
|
||||
|
||||
return Task.WhenAll(
|
||||
collection.Indexes.CreateOneAsync(new CreateIndexModel<VexObservationRecord>(tenantObservationIndex, new CreateIndexOptions { Unique = true }), cancellationToken: cancellationToken),
|
||||
collection.Indexes.CreateOneAsync(new CreateIndexModel<VexObservationRecord>(tenantVulnIndex), cancellationToken: cancellationToken),
|
||||
collection.Indexes.CreateOneAsync(new CreateIndexModel<VexObservationRecord>(tenantProductIndex), cancellationToken: cancellationToken),
|
||||
collection.Indexes.CreateOneAsync(new CreateIndexModel<VexObservationRecord>(tenantDigestIndex), cancellationToken: cancellationToken),
|
||||
collection.Indexes.CreateOneAsync(new CreateIndexModel<VexObservationRecord>(tenantProviderStatusIndex), cancellationToken: cancellationToken));
|
||||
}
|
||||
|
||||
private static Task EnsureLinksetIndexesAsync(IMongoDatabase database, CancellationToken cancellationToken)
|
||||
{
|
||||
var collection = database.GetCollection<VexLinksetRecord>(VexMongoCollectionNames.Linksets);
|
||||
|
||||
var tenantLinksetIndex = Builders<VexLinksetRecord>.IndexKeys
|
||||
.Ascending(x => x.Tenant)
|
||||
.Ascending(x => x.LinksetId);
|
||||
|
||||
var tenantVulnIndex = Builders<VexLinksetRecord>.IndexKeys
|
||||
.Ascending(x => x.Tenant)
|
||||
.Ascending(x => x.VulnerabilityId);
|
||||
|
||||
var tenantProductIndex = Builders<VexLinksetRecord>.IndexKeys
|
||||
.Ascending(x => x.Tenant)
|
||||
.Ascending(x => x.ProductKey);
|
||||
|
||||
var tenantDisagreementProviderIndex = Builders<VexLinksetRecord>.IndexKeys
|
||||
.Ascending(x => x.Tenant)
|
||||
.Ascending("Disagreements.ProviderId")
|
||||
.Ascending("Disagreements.Status");
|
||||
|
||||
return Task.WhenAll(
|
||||
collection.Indexes.CreateOneAsync(new CreateIndexModel<VexLinksetRecord>(tenantLinksetIndex, new CreateIndexOptions { Unique = true }), cancellationToken: cancellationToken),
|
||||
collection.Indexes.CreateOneAsync(new CreateIndexModel<VexLinksetRecord>(tenantVulnIndex), cancellationToken: cancellationToken),
|
||||
collection.Indexes.CreateOneAsync(new CreateIndexModel<VexLinksetRecord>(tenantProductIndex), cancellationToken: cancellationToken),
|
||||
collection.Indexes.CreateOneAsync(new CreateIndexModel<VexLinksetRecord>(tenantDisagreementProviderIndex), cancellationToken: cancellationToken));
|
||||
}
|
||||
}
|
||||
@@ -60,6 +60,7 @@ public static class VexMongoServiceCollectionExtensions
|
||||
services.AddSingleton<IVexMongoMigration, VexInitialIndexMigration>();
|
||||
services.AddSingleton<IVexMongoMigration, VexConsensusSignalsMigration>();
|
||||
services.AddSingleton<IVexMongoMigration, VexConsensusHoldMigration>();
|
||||
services.AddSingleton<IVexMongoMigration, VexObservationCollectionsMigration>();
|
||||
services.AddSingleton<VexMongoMigrationRunner>();
|
||||
services.AddHostedService<VexMongoMigrationHostedService>();
|
||||
return services;
|
||||
|
||||
@@ -64,12 +64,12 @@ public static class VexMongoMappingRegistry
|
||||
}
|
||||
}
|
||||
|
||||
public static class VexMongoCollectionNames
|
||||
{
|
||||
public const string Migrations = "vex.migrations";
|
||||
public const string Providers = "vex.providers";
|
||||
public const string Raw = "vex.raw";
|
||||
public const string Statements = "vex.statements";
|
||||
public static class VexMongoCollectionNames
|
||||
{
|
||||
public const string Migrations = "vex.migrations";
|
||||
public const string Providers = "vex.providers";
|
||||
public const string Raw = "vex.raw";
|
||||
public const string Statements = "vex.statements";
|
||||
public const string Claims = Statements;
|
||||
public const string Consensus = "vex.consensus";
|
||||
public const string Exports = "vex.exports";
|
||||
@@ -77,4 +77,6 @@ public static class VexMongoCollectionNames
|
||||
public const string ConnectorState = "vex.connector_state";
|
||||
public const string ConsensusHolds = "vex.consensus_holds";
|
||||
public const string Attestations = "vex.attestations";
|
||||
public const string Observations = "vex.observations";
|
||||
public const string Linksets = "vex.linksets";
|
||||
}
|
||||
|
||||
@@ -63,8 +63,8 @@ internal sealed class VexRawDocumentRecord
|
||||
}
|
||||
|
||||
[BsonIgnoreExtraElements]
|
||||
internal sealed class VexExportManifestRecord
|
||||
{
|
||||
internal sealed class VexExportManifestRecord
|
||||
{
|
||||
[BsonId]
|
||||
public string Id { get; set; } = default!;
|
||||
|
||||
@@ -164,8 +164,8 @@ internal sealed class VexExportManifestRecord
|
||||
SizeBytes = manifest.SizeBytes,
|
||||
};
|
||||
|
||||
public VexExportManifest ToDomain()
|
||||
{
|
||||
public VexExportManifest ToDomain()
|
||||
{
|
||||
var signedAt = SignedAt.HasValue
|
||||
? new DateTimeOffset(DateTime.SpecifyKind(SignedAt.Value, DateTimeKind.Utc))
|
||||
: (DateTimeOffset?)null;
|
||||
@@ -276,6 +276,73 @@ internal sealed class VexQuietStatementRecord
|
||||
}
|
||||
}
|
||||
|
||||
[BsonIgnoreExtraElements]
|
||||
internal sealed class VexObservationRecord
|
||||
{
|
||||
[BsonId]
|
||||
public string Id { get; set; } = default!; // observationId
|
||||
|
||||
public string Tenant { get; set; } = default!;
|
||||
|
||||
public string ObservationId { get; set; } = default!;
|
||||
|
||||
public string VulnerabilityId { get; set; } = default!;
|
||||
|
||||
public string ProductKey { get; set; } = default!;
|
||||
|
||||
public string ProviderId { get; set; } = default!;
|
||||
|
||||
public string Status { get; set; } = default!;
|
||||
|
||||
public VexObservationDocumentRecord Document { get; set; } = new();
|
||||
|
||||
public DateTime CreatedAt { get; set; } = DateTime.SpecifyKind(DateTime.UtcNow, DateTimeKind.Utc);
|
||||
|
||||
public List<VexLinksetDisagreementRecord> Disagreements { get; set; } = new();
|
||||
}
|
||||
|
||||
[BsonIgnoreExtraElements]
|
||||
internal sealed class VexObservationDocumentRecord
|
||||
{
|
||||
public string Digest { get; set; } = default!;
|
||||
|
||||
public string? SourceUri { get; set; }
|
||||
= null;
|
||||
}
|
||||
|
||||
[BsonIgnoreExtraElements]
|
||||
internal sealed class VexLinksetRecord
|
||||
{
|
||||
[BsonId]
|
||||
public string Id { get; set; } = default!; // linksetId
|
||||
|
||||
public string Tenant { get; set; } = default!;
|
||||
|
||||
public string LinksetId { get; set; } = default!;
|
||||
|
||||
public string VulnerabilityId { get; set; } = default!;
|
||||
|
||||
public string ProductKey { get; set; } = default!;
|
||||
|
||||
public DateTime CreatedAt { get; set; } = DateTime.SpecifyKind(DateTime.UtcNow, DateTimeKind.Utc);
|
||||
|
||||
public List<VexLinksetDisagreementRecord> Disagreements { get; set; } = new();
|
||||
}
|
||||
|
||||
[BsonIgnoreExtraElements]
|
||||
internal sealed class VexLinksetDisagreementRecord
|
||||
{
|
||||
public string ProviderId { get; set; } = default!;
|
||||
|
||||
public string Status { get; set; } = default!;
|
||||
|
||||
public string? Justification { get; set; }
|
||||
= null;
|
||||
|
||||
public double? Confidence { get; set; }
|
||||
= null;
|
||||
}
|
||||
|
||||
[BsonIgnoreExtraElements]
|
||||
internal sealed class VexProviderRecord
|
||||
{
|
||||
|
||||
Reference in New Issue
Block a user