consolidation of some of the modules, localization fixes, product advisories work, qa work
This commit is contained in:
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,26 @@
|
||||
# Excititor Mirror Connector Charter
|
||||
|
||||
## Mission
|
||||
Ingest StellaOps VEX mirror bundles into Excititor, converting them into immutable VEX observations without applying consensus or suppression. The connector must honour the Aggregation-Only Contract, maintain provenance, and support offline replay and incremental updates.
|
||||
|
||||
## Scope
|
||||
- Code in `StellaOps.Excititor.Connectors.StellaOpsMirror`.
|
||||
- Bundle validation (signatures, manifests, Merkle roots) and cursor management.
|
||||
- Integration with Excititor storage and Surface/VEX Lens consumers.
|
||||
- Test fixtures demonstrating deterministic ingest across bundle versions.
|
||||
|
||||
## Required Reading
|
||||
- `docs/modules/excititor/architecture.md`
|
||||
- `docs/modules/concelier/guides/aggregation-only-contract.md`
|
||||
- `docs/modules/excititor/mirrors.md` (if available; otherwise coordinate with Docs to add details)
|
||||
- `docs/modules/airgap/guides/airgap-mode.md`
|
||||
- `docs/modules/concelier/operations/mirror.md` (shared mirror concepts)
|
||||
|
||||
## Working Agreement
|
||||
1. **Status updates**: set tasks to `DOING`/`DONE` in both sprint file `/docs/implplan/SPRINT_*.md` and local `TASKS.md` when work starts/finishes.
|
||||
2. **Provenance preservation**: record bundle IDs, digests, and time anchors in stored observations; avoid derived fields.
|
||||
3. **Deterministic replay**: ensure repeated imports of the same bundle produce identical documents; handle supersedes and delta bundles gracefully.
|
||||
4. **Offline readiness**: no external network calls; provide clear errors for invalid or stale bundles.
|
||||
5. **Testing**: maintain mirror fixtures covering full/delta bundles, supersedes chains, and failure cases.
|
||||
6. **Documentation**: coordinate updates to mirror connector docs and release notes when behaviour or configuration changes.
|
||||
|
||||
@@ -0,0 +1,5 @@
|
||||
# Completed Tasks
|
||||
|
||||
| ID | Status | Owner(s) | Depends on | Description | Exit Criteria |
|
||||
|----|--------|----------|------------|-------------|---------------|
|
||||
| EXCITITOR-CONN-STELLA-07-001 | DONE (2025-10-21) | Excititor Connectors – Stella | EXCITITOR-EXPORT-01-007 | **DONE (2025-10-21)** – Implemented `StellaOpsMirrorConnector` with `MirrorManifestClient` + `MirrorSignatureVerifier`, digest validation, signature enforcement, raw document + DTO persistence, and resume cursor updates. Added fixture-backed tests covering happy path and tampered manifest rejection. | Fetch job downloads mirror manifest, verifies DSSE/signature, stores raw documents + provenance; unit tests cover happy path and tampered manifest failure. |
|
||||
48
src/Concelier/StellaOps.Excititor.WebService/AGENTS.md
Normal file
48
src/Concelier/StellaOps.Excititor.WebService/AGENTS.md
Normal file
@@ -0,0 +1,48 @@
|
||||
# Excititor WebService Charter
|
||||
|
||||
## Mission
|
||||
Expose Excititor APIs (console VEX views, graph/Vuln Explorer feeds, observation intake/health) while honoring the Aggregation-Only Contract (no consensus/severity logic in this service).
|
||||
|
||||
## Scope
|
||||
- Working directory: `src/Excititor/StellaOps.Excititor.WebService`
|
||||
- HTTP APIs, DTOs, controllers, authz filters, composition root, telemetry hooks.
|
||||
- Wiring to Core/Storage libraries; no direct policy or consensus logic.
|
||||
|
||||
## Required Reading
|
||||
- `docs/modules/excititor/architecture.md`
|
||||
- `docs/modules/excititor/README.md`
|
||||
- `docs/modules/excititor/vex_observations.md`
|
||||
- `docs/modules/concelier/guides/aggregation-only-contract.md`
|
||||
- `docs-archived/implplan/implementation-plans/excititor-implementation-plan.md`
|
||||
|
||||
## Roles
|
||||
- Backend developer (.NET 10 / C# preview).
|
||||
- QA automation (integration + API contract tests).
|
||||
|
||||
## Working Agreements
|
||||
1. Update sprint `Delivery Tracker` when tasks move TODO???DOING???DONE/BLOCKED; mirror notes in Execution Log.
|
||||
2. Keep APIs aggregation-only: persist raw observations, provenance, and precedence pointers; never merge/weight/consensus here.
|
||||
3. Enforce tenant scoping and RBAC on all endpoints; default-deny for cross-tenant data.
|
||||
4. Offline-first: no external network calls; rely on cached/mirrored feeds only.
|
||||
5. Observability: structured logs, counters, optional OTEL traces behind configuration flags.
|
||||
|
||||
## Testing
|
||||
- Prefer deterministic API/integration tests under `__Tests` with seeded Postgres fixtures or in-memory stores.
|
||||
- Verify RBAC/tenant isolation, idempotent ingestion, and stable ordering of VEX aggregates.
|
||||
- Use ISO-8601 UTC timestamps and stable sorting in responses; assert on content hashes where applicable.
|
||||
|
||||
## Determinism & Data
|
||||
- Postgres append-only storage is canonical; never apply consensus transformations before persistence.
|
||||
- Ensure paged/list endpoints use explicit sort keys (e.g., vendor, upstreamId, version, createdUtc).
|
||||
- Avoid nondeterministic clocks/randomness; inject clocks and GUID providers for tests.
|
||||
- Evidence/attestation endpoints are temporarily disabled; re-enable only when Postgres-backed stores land (Mongo/BSON removed).
|
||||
|
||||
## Boundaries
|
||||
- Do not modify Policy Engine or Cartographer schemas from here; consume published contracts only.
|
||||
- Configuration via appsettings/environment; no hard-coded secrets.
|
||||
|
||||
## Ready-to-Start Checklist
|
||||
- Required docs reviewed.
|
||||
- Test database/fixtures prepared (no external dependencies).
|
||||
- Feature flags defined for new endpoints before exposing them.
|
||||
|
||||
@@ -0,0 +1,38 @@
|
||||
using System;
|
||||
using System.Text.Json.Serialization;
|
||||
|
||||
namespace StellaOps.Excititor.WebService.Contracts;
|
||||
|
||||
/// <summary>
|
||||
/// Envelope for air-gapped VEX bundle imports.
|
||||
/// Mirrors the thin mirror bundle schema and carries signing metadata.
|
||||
/// </summary>
|
||||
public sealed class AirgapImportRequest
|
||||
{
|
||||
[JsonPropertyName("bundleId")]
|
||||
public string? BundleId { get; init; }
|
||||
|
||||
[JsonPropertyName("mirrorGeneration")]
|
||||
public string? MirrorGeneration { get; init; }
|
||||
|
||||
[JsonPropertyName("signedAt")]
|
||||
public DateTimeOffset? SignedAt { get; init; }
|
||||
|
||||
[JsonPropertyName("publisher")]
|
||||
public string? Publisher { get; init; }
|
||||
|
||||
[JsonPropertyName("tenantId")]
|
||||
public string? TenantId { get; init; }
|
||||
|
||||
[JsonPropertyName("payloadHash")]
|
||||
public string? PayloadHash { get; init; }
|
||||
|
||||
[JsonPropertyName("payloadUrl")]
|
||||
public string? PayloadUrl { get; init; }
|
||||
|
||||
[JsonPropertyName("signature")]
|
||||
public string? Signature { get; init; }
|
||||
|
||||
[JsonPropertyName("transparencyLog")]
|
||||
public string? TransparencyLog { get; init; }
|
||||
}
|
||||
@@ -0,0 +1,199 @@
|
||||
using System;
|
||||
using System.Collections.Generic;
|
||||
using System.Text.Json.Serialization;
|
||||
|
||||
namespace StellaOps.Excititor.WebService.Contracts;
|
||||
|
||||
/// <summary>
|
||||
/// Response listing registered mirror bundles.
|
||||
/// </summary>
|
||||
public sealed record MirrorBundleListResponse(
|
||||
[property: JsonPropertyName("bundles")] IReadOnlyList<MirrorBundleSummary> Bundles,
|
||||
[property: JsonPropertyName("totalCount")] int TotalCount,
|
||||
[property: JsonPropertyName("limit")] int Limit,
|
||||
[property: JsonPropertyName("offset")] int Offset,
|
||||
[property: JsonPropertyName("queriedAt")] DateTimeOffset QueriedAt);
|
||||
|
||||
/// <summary>
|
||||
/// Summary of a registered mirror bundle.
|
||||
/// </summary>
|
||||
public sealed record MirrorBundleSummary(
|
||||
[property: JsonPropertyName("bundleId")] string BundleId,
|
||||
[property: JsonPropertyName("mirrorGeneration")] string MirrorGeneration,
|
||||
[property: JsonPropertyName("publisher")] string Publisher,
|
||||
[property: JsonPropertyName("signedAt")] DateTimeOffset SignedAt,
|
||||
[property: JsonPropertyName("importedAt")] DateTimeOffset ImportedAt,
|
||||
[property: JsonPropertyName("payloadHash")] string PayloadHash,
|
||||
[property: JsonPropertyName("stalenessSeconds")] long StalenessSeconds,
|
||||
[property: JsonPropertyName("status")] string Status);
|
||||
|
||||
/// <summary>
|
||||
/// Detailed response for a registered mirror bundle with provenance.
|
||||
/// </summary>
|
||||
public sealed record MirrorBundleDetailResponse(
|
||||
[property: JsonPropertyName("bundleId")] string BundleId,
|
||||
[property: JsonPropertyName("mirrorGeneration")] string MirrorGeneration,
|
||||
[property: JsonPropertyName("tenantId")] string TenantId,
|
||||
[property: JsonPropertyName("publisher")] string Publisher,
|
||||
[property: JsonPropertyName("signedAt")] DateTimeOffset SignedAt,
|
||||
[property: JsonPropertyName("importedAt")] DateTimeOffset ImportedAt,
|
||||
[property: JsonPropertyName("provenance")] MirrorBundleProvenance Provenance,
|
||||
[property: JsonPropertyName("staleness")] MirrorBundleStaleness Staleness,
|
||||
[property: JsonPropertyName("paths")] MirrorBundlePaths Paths,
|
||||
[property: JsonPropertyName("timeline")] IReadOnlyList<MirrorBundleTimelineEntry> Timeline,
|
||||
[property: JsonPropertyName("queriedAt")] DateTimeOffset QueriedAt);
|
||||
|
||||
/// <summary>
|
||||
/// Provenance metadata for a mirror bundle.
|
||||
/// </summary>
|
||||
public sealed record MirrorBundleProvenance(
|
||||
[property: JsonPropertyName("payloadHash")] string PayloadHash,
|
||||
[property: JsonPropertyName("signature")] string Signature,
|
||||
[property: JsonPropertyName("payloadUrl")] string? PayloadUrl,
|
||||
[property: JsonPropertyName("transparencyLog")] string? TransparencyLog,
|
||||
[property: JsonPropertyName("manifestHash")] string ManifestHash);
|
||||
|
||||
/// <summary>
|
||||
/// Staleness metrics for a mirror bundle.
|
||||
/// </summary>
|
||||
public sealed record MirrorBundleStaleness(
|
||||
[property: JsonPropertyName("sinceSignedSeconds")] long SinceSignedSeconds,
|
||||
[property: JsonPropertyName("sinceImportedSeconds")] long SinceImportedSeconds,
|
||||
[property: JsonPropertyName("signedAgeCategory")] string SignedAgeCategory,
|
||||
[property: JsonPropertyName("importedAgeCategory")] string ImportedAgeCategory);
|
||||
|
||||
/// <summary>
|
||||
/// Storage paths for a mirror bundle.
|
||||
/// </summary>
|
||||
public sealed record MirrorBundlePaths(
|
||||
[property: JsonPropertyName("portableManifestPath")] string PortableManifestPath,
|
||||
[property: JsonPropertyName("evidenceLockerPath")] string EvidenceLockerPath);
|
||||
|
||||
/// <summary>
|
||||
/// Timeline entry for audit trail.
|
||||
/// </summary>
|
||||
public sealed record MirrorBundleTimelineEntry(
|
||||
[property: JsonPropertyName("eventType")] string EventType,
|
||||
[property: JsonPropertyName("createdAt")] DateTimeOffset CreatedAt,
|
||||
[property: JsonPropertyName("stalenessSeconds")] int? StalenessSeconds,
|
||||
[property: JsonPropertyName("errorCode")] string? ErrorCode,
|
||||
[property: JsonPropertyName("message")] string? Message,
|
||||
[property: JsonPropertyName("remediation")] string? Remediation,
|
||||
[property: JsonPropertyName("actor")] string? Actor,
|
||||
[property: JsonPropertyName("scopes")] string? Scopes);
|
||||
|
||||
/// <summary>
|
||||
/// Response for timeline-only query.
|
||||
/// </summary>
|
||||
public sealed record MirrorBundleTimelineResponse(
|
||||
[property: JsonPropertyName("bundleId")] string BundleId,
|
||||
[property: JsonPropertyName("mirrorGeneration")] string MirrorGeneration,
|
||||
[property: JsonPropertyName("timeline")] IReadOnlyList<MirrorBundleTimelineEntry> Timeline,
|
||||
[property: JsonPropertyName("queriedAt")] DateTimeOffset QueriedAt);
|
||||
|
||||
/// <summary>
|
||||
/// Structured error response for sealed-mode and airgap errors.
|
||||
/// </summary>
|
||||
public sealed record AirgapErrorResponse(
|
||||
[property: JsonPropertyName("errorCode")] string ErrorCode,
|
||||
[property: JsonPropertyName("message")] string Message,
|
||||
[property: JsonPropertyName("category")] string Category,
|
||||
[property: JsonPropertyName("retryable")] bool Retryable,
|
||||
[property: JsonPropertyName("details")] IReadOnlyDictionary<string, string>? Details,
|
||||
[property: JsonPropertyName("remediation")] string? Remediation);
|
||||
|
||||
/// <summary>
|
||||
/// Maps sealed-mode error codes to structured error responses.
|
||||
/// </summary>
|
||||
public static class AirgapErrorMapping
|
||||
{
|
||||
public const string CategoryValidation = "validation";
|
||||
public const string CategorySealedMode = "sealed_mode";
|
||||
public const string CategoryTrust = "trust";
|
||||
public const string CategoryDuplicate = "duplicate";
|
||||
public const string CategoryNotFound = "not_found";
|
||||
|
||||
public static AirgapErrorResponse FromErrorCode(string errorCode, string message, IReadOnlyDictionary<string, string>? details = null)
|
||||
{
|
||||
var (category, retryable) = errorCode switch
|
||||
{
|
||||
"AIRGAP_EGRESS_BLOCKED" => (CategorySealedMode, false),
|
||||
"AIRGAP_SOURCE_UNTRUSTED" => (CategoryTrust, false),
|
||||
"AIRGAP_SIGNATURE_MISSING" => (CategoryValidation, false),
|
||||
"AIRGAP_SIGNATURE_INVALID" => (CategoryValidation, false),
|
||||
"AIRGAP_PAYLOAD_STALE" => (CategoryValidation, true),
|
||||
"AIRGAP_PAYLOAD_MISMATCH" => (CategoryTrust, false),
|
||||
"AIRGAP_DUPLICATE_IMPORT" => (CategoryDuplicate, false),
|
||||
"AIRGAP_BUNDLE_NOT_FOUND" => (CategoryNotFound, false),
|
||||
_ when errorCode.StartsWith("bundle_", StringComparison.Ordinal) => (CategoryValidation, false),
|
||||
_ when errorCode.StartsWith("mirror_", StringComparison.Ordinal) => (CategoryValidation, false),
|
||||
_ when errorCode.StartsWith("publisher_", StringComparison.Ordinal) => (CategoryValidation, false),
|
||||
_ when errorCode.StartsWith("payload_", StringComparison.Ordinal) => (CategoryValidation, false),
|
||||
_ when errorCode.StartsWith("signed_", StringComparison.Ordinal) => (CategoryValidation, false),
|
||||
_ => (CategoryValidation, false),
|
||||
};
|
||||
|
||||
var remediation = ResolveRemediation(errorCode);
|
||||
return new AirgapErrorResponse(errorCode, message, category, retryable, details, remediation);
|
||||
}
|
||||
|
||||
public static AirgapErrorResponse DuplicateImport(string bundleId, string mirrorGeneration)
|
||||
=> new(
|
||||
"AIRGAP_DUPLICATE_IMPORT",
|
||||
$"Bundle '{bundleId}' generation '{mirrorGeneration}' has already been imported.",
|
||||
CategoryDuplicate,
|
||||
Retryable: false,
|
||||
new Dictionary<string, string>
|
||||
{
|
||||
["bundleId"] = bundleId,
|
||||
["mirrorGeneration"] = mirrorGeneration,
|
||||
},
|
||||
ResolveRemediation("AIRGAP_DUPLICATE_IMPORT"));
|
||||
|
||||
public static AirgapErrorResponse BundleNotFound(string bundleId, string? mirrorGeneration)
|
||||
=> new(
|
||||
"AIRGAP_BUNDLE_NOT_FOUND",
|
||||
mirrorGeneration is null
|
||||
? $"Bundle '{bundleId}' not found."
|
||||
: $"Bundle '{bundleId}' generation '{mirrorGeneration}' not found.",
|
||||
CategoryNotFound,
|
||||
Retryable: false,
|
||||
new Dictionary<string, string>
|
||||
{
|
||||
["bundleId"] = bundleId,
|
||||
["mirrorGeneration"] = mirrorGeneration ?? string.Empty,
|
||||
},
|
||||
ResolveRemediation("AIRGAP_BUNDLE_NOT_FOUND"));
|
||||
|
||||
private static string? ResolveRemediation(string errorCode) =>
|
||||
errorCode switch
|
||||
{
|
||||
"AIRGAP_EGRESS_BLOCKED" => "Stage bundle via mirror or portable media; remove external URLs before retrying.",
|
||||
"AIRGAP_SOURCE_UNTRUSTED" => "Submit from an allowlisted publisher or add the publisher to TrustedPublishers in Excititor:Airgap settings.",
|
||||
"AIRGAP_SIGNATURE_MISSING" => "Provide DSSE signature for the bundle manifest.",
|
||||
"AIRGAP_SIGNATURE_INVALID" => "Re-sign the bundle manifest with a trusted key.",
|
||||
"AIRGAP_PAYLOAD_STALE" => "Regenerate bundle with fresh signedAt closer to import time.",
|
||||
"AIRGAP_PAYLOAD_MISMATCH" => "Recreate bundle; ensure manifest hash matches payload.",
|
||||
"AIRGAP_DUPLICATE_IMPORT" => "Use a new mirrorGeneration or verify the previous import before retrying.",
|
||||
_ => null
|
||||
};
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Utility for computing staleness categories.
|
||||
/// </summary>
|
||||
public static class StalenessCalculator
|
||||
{
|
||||
public static long ComputeSeconds(DateTimeOffset then, DateTimeOffset now)
|
||||
=> (long)Math.Max(0, Math.Ceiling((now - then).TotalSeconds));
|
||||
|
||||
public static string CategorizeAge(long seconds)
|
||||
=> seconds switch
|
||||
{
|
||||
< 3600 => "fresh", // < 1 hour
|
||||
< 86400 => "recent", // < 1 day
|
||||
< 604800 => "stale", // < 1 week
|
||||
< 2592000 => "old", // < 30 days
|
||||
_ => "very_old", // >= 30 days
|
||||
};
|
||||
}
|
||||
@@ -0,0 +1,46 @@
|
||||
using System;
|
||||
using System.Collections.Generic;
|
||||
|
||||
namespace StellaOps.Excititor.WebService.Contracts;
|
||||
|
||||
public sealed record AttestationVerifyRequest
|
||||
{
|
||||
public string ExportId { get; init; } = string.Empty;
|
||||
public string QuerySignature { get; init; } = string.Empty;
|
||||
public string ArtifactDigest { get; init; } = string.Empty;
|
||||
public string Format { get; init; } = string.Empty;
|
||||
public DateTimeOffset CreatedAt { get; init; }
|
||||
= DateTimeOffset.UnixEpoch;
|
||||
public IReadOnlyList<string> SourceProviders { get; init; }
|
||||
= Array.Empty<string>();
|
||||
public IReadOnlyDictionary<string, string> Metadata { get; init; }
|
||||
= new Dictionary<string, string>(StringComparer.Ordinal);
|
||||
public AttestationVerifyMetadata Attestation { get; init; }
|
||||
= new();
|
||||
public string Envelope { get; init; } = string.Empty;
|
||||
public bool IsReverify { get; init; }
|
||||
= false;
|
||||
}
|
||||
|
||||
public sealed record AttestationVerifyMetadata
|
||||
{
|
||||
public string PredicateType { get; init; } = string.Empty;
|
||||
public string EnvelopeDigest { get; init; } = string.Empty;
|
||||
public DateTimeOffset SignedAt { get; init; } = DateTimeOffset.UnixEpoch;
|
||||
public AttestationRekorReference? Rekor { get; init; }
|
||||
= null;
|
||||
}
|
||||
|
||||
public sealed record AttestationRekorReference
|
||||
{
|
||||
public string? ApiVersion { get; init; }
|
||||
= null;
|
||||
public string? Location { get; init; }
|
||||
= null;
|
||||
public long? LogIndex { get; init; }
|
||||
= null;
|
||||
public Uri? InclusionProofUrl { get; init; }
|
||||
= null;
|
||||
}
|
||||
|
||||
public sealed record AttestationVerifyResponse(bool Valid, IDictionary<string, string> Diagnostics);
|
||||
@@ -0,0 +1,20 @@
|
||||
|
||||
using StellaOps.Excititor.Core.Evidence;
|
||||
using System.Collections.Generic;
|
||||
using System.Text.Json.Serialization;
|
||||
|
||||
namespace StellaOps.Excititor.WebService.Contracts;
|
||||
|
||||
public sealed record EvidenceManifestResponse(
|
||||
[property: JsonPropertyName("manifest")] VexLockerManifest Manifest,
|
||||
[property: JsonPropertyName("attestationId")] string AttestationId,
|
||||
[property: JsonPropertyName("dsseEnvelope")] string DsseEnvelope,
|
||||
[property: JsonPropertyName("dsseEnvelopeHash")] string DsseEnvelopeHash,
|
||||
[property: JsonPropertyName("itemCount")] int ItemCount,
|
||||
[property: JsonPropertyName("generatedAt")] DateTimeOffset GeneratedAt);
|
||||
|
||||
public sealed record EvidenceChunkListResponse(
|
||||
[property: JsonPropertyName("chunks")] IReadOnlyList<VexEvidenceChunkResponse> Chunks,
|
||||
[property: JsonPropertyName("total")] int Total,
|
||||
[property: JsonPropertyName("truncated")] bool Truncated,
|
||||
[property: JsonPropertyName("generatedAt")] DateTimeOffset GeneratedAt);
|
||||
@@ -0,0 +1,40 @@
|
||||
using System;
|
||||
using System.Collections.Generic;
|
||||
using System.Text.Json.Serialization;
|
||||
|
||||
namespace StellaOps.Excititor.WebService.Contracts;
|
||||
|
||||
public sealed record GraphLinkoutsRequest(
|
||||
[property: JsonPropertyName("tenant")] string Tenant,
|
||||
[property: JsonPropertyName("purls")] IReadOnlyList<string> Purls,
|
||||
[property: JsonPropertyName("includeJustifications")] bool IncludeJustifications = false,
|
||||
[property: JsonPropertyName("includeProvenance")] bool IncludeProvenance = true);
|
||||
|
||||
public sealed record GraphLinkoutsResponse(
|
||||
[property: JsonPropertyName("items")] IReadOnlyList<GraphLinkoutItem> Items,
|
||||
[property: JsonPropertyName("notFound")] IReadOnlyList<string> NotFound);
|
||||
|
||||
public sealed record GraphLinkoutItem(
|
||||
[property: JsonPropertyName("purl")] string Purl,
|
||||
[property: JsonPropertyName("advisories")] IReadOnlyList<GraphLinkoutAdvisory> Advisories,
|
||||
[property: JsonPropertyName("conflicts")] IReadOnlyList<GraphLinkoutConflict> Conflicts,
|
||||
[property: JsonPropertyName("truncated")] bool Truncated = false,
|
||||
[property: JsonPropertyName("nextCursor")] string? NextCursor = null);
|
||||
|
||||
public sealed record GraphLinkoutAdvisory(
|
||||
[property: JsonPropertyName("advisoryId")] string AdvisoryId,
|
||||
[property: JsonPropertyName("source")] string Source,
|
||||
[property: JsonPropertyName("status")] string Status,
|
||||
[property: JsonPropertyName("justification")] string? Justification,
|
||||
[property: JsonPropertyName("modifiedAt")] DateTimeOffset ModifiedAt,
|
||||
[property: JsonPropertyName("evidenceHash")] string EvidenceHash,
|
||||
[property: JsonPropertyName("connectorId")] string ConnectorId,
|
||||
[property: JsonPropertyName("dsseEnvelopeHash")] string? DsseEnvelopeHash);
|
||||
|
||||
public sealed record GraphLinkoutConflict(
|
||||
[property: JsonPropertyName("source")] string Source,
|
||||
[property: JsonPropertyName("status")] string Status,
|
||||
[property: JsonPropertyName("justification")] string? Justification,
|
||||
[property: JsonPropertyName("observedAt")] DateTimeOffset ObservedAt,
|
||||
[property: JsonPropertyName("evidenceHash")] string EvidenceHash);
|
||||
|
||||
@@ -0,0 +1,54 @@
|
||||
using System;
|
||||
using System.Collections.Generic;
|
||||
using System.Text.Json.Serialization;
|
||||
|
||||
namespace StellaOps.Excititor.WebService.Contracts;
|
||||
|
||||
public sealed record GraphOverlaysResponse(
|
||||
[property: JsonPropertyName("items")] IReadOnlyList<GraphOverlayItem> Items,
|
||||
[property: JsonPropertyName("cached")] bool Cached,
|
||||
[property: JsonPropertyName("cacheAgeMs")] long? CacheAgeMs);
|
||||
|
||||
public sealed record GraphOverlayItem(
|
||||
[property: JsonPropertyName("schemaVersion")] string SchemaVersion,
|
||||
[property: JsonPropertyName("generatedAt")] DateTimeOffset GeneratedAt,
|
||||
[property: JsonPropertyName("tenant")] string Tenant,
|
||||
[property: JsonPropertyName("purl")] string Purl,
|
||||
[property: JsonPropertyName("advisoryId")] string AdvisoryId,
|
||||
[property: JsonPropertyName("source")] string Source,
|
||||
[property: JsonPropertyName("status")] string Status,
|
||||
[property: JsonPropertyName("justifications")] IReadOnlyList<GraphOverlayJustification> Justifications,
|
||||
[property: JsonPropertyName("conflicts")] IReadOnlyList<GraphOverlayConflict> Conflicts,
|
||||
[property: JsonPropertyName("observations")] IReadOnlyList<GraphOverlayObservation> Observations,
|
||||
[property: JsonPropertyName("provenance")] GraphOverlayProvenance Provenance,
|
||||
[property: JsonPropertyName("cache")] GraphOverlayCache? Cache);
|
||||
|
||||
public sealed record GraphOverlayJustification(
|
||||
[property: JsonPropertyName("kind")] string Kind,
|
||||
[property: JsonPropertyName("reason")] string Reason,
|
||||
[property: JsonPropertyName("evidence")] IReadOnlyList<string>? Evidence,
|
||||
[property: JsonPropertyName("weight")] double? Weight);
|
||||
|
||||
public sealed record GraphOverlayConflict(
|
||||
[property: JsonPropertyName("field")] string Field,
|
||||
[property: JsonPropertyName("reason")] string Reason,
|
||||
[property: JsonPropertyName("values")] IReadOnlyList<string> Values,
|
||||
[property: JsonPropertyName("sourceIds")] IReadOnlyList<string>? SourceIds);
|
||||
|
||||
public sealed record GraphOverlayObservation(
|
||||
[property: JsonPropertyName("id")] string Id,
|
||||
[property: JsonPropertyName("contentHash")] string ContentHash,
|
||||
[property: JsonPropertyName("fetchedAt")] DateTimeOffset FetchedAt);
|
||||
|
||||
public sealed record GraphOverlayProvenance(
|
||||
[property: JsonPropertyName("linksetId")] string LinksetId,
|
||||
[property: JsonPropertyName("linksetHash")] string LinksetHash,
|
||||
[property: JsonPropertyName("observationHashes")] IReadOnlyList<string> ObservationHashes,
|
||||
[property: JsonPropertyName("policyHash")] string? PolicyHash,
|
||||
[property: JsonPropertyName("sbomContextHash")] string? SbomContextHash,
|
||||
[property: JsonPropertyName("planCacheKey")] string? PlanCacheKey);
|
||||
|
||||
public sealed record GraphOverlayCache(
|
||||
[property: JsonPropertyName("cached")] bool Cached,
|
||||
[property: JsonPropertyName("cachedAt")] DateTimeOffset? CachedAt,
|
||||
[property: JsonPropertyName("ttlSeconds")] int? TtlSeconds);
|
||||
@@ -0,0 +1,23 @@
|
||||
using System;
|
||||
using System.Collections.Generic;
|
||||
using System.Text.Json.Serialization;
|
||||
|
||||
namespace StellaOps.Excititor.WebService.Contracts;
|
||||
|
||||
public sealed record GraphStatusResponse(
|
||||
[property: JsonPropertyName("items")] IReadOnlyList<GraphStatusItem> Items,
|
||||
[property: JsonPropertyName("cached")] bool Cached,
|
||||
[property: JsonPropertyName("cacheAgeMs")] long? CacheAgeMs);
|
||||
|
||||
public sealed record GraphStatusItem(
|
||||
[property: JsonPropertyName("purl")] string Purl,
|
||||
[property: JsonPropertyName("summary")] GraphOverlaySummary Summary,
|
||||
[property: JsonPropertyName("latestModifiedAt")] DateTimeOffset? LatestModifiedAt,
|
||||
[property: JsonPropertyName("sources")] IReadOnlyList<string> Sources,
|
||||
[property: JsonPropertyName("lastEvidenceHash")] string? LastEvidenceHash);
|
||||
|
||||
public sealed record GraphOverlaySummary(
|
||||
[property: JsonPropertyName("open")] int Open,
|
||||
[property: JsonPropertyName("not_affected")] int NotAffected,
|
||||
[property: JsonPropertyName("under_investigation")] int UnderInvestigation,
|
||||
[property: JsonPropertyName("no_statement")] int NoStatement);
|
||||
@@ -0,0 +1,25 @@
|
||||
using System;
|
||||
using System.Collections.Generic;
|
||||
using System.Text.Json.Serialization;
|
||||
|
||||
namespace StellaOps.Excititor.WebService.Contracts;
|
||||
|
||||
public sealed record GraphTooltipResponse(
|
||||
[property: JsonPropertyName("items")] IReadOnlyList<GraphTooltipItem> Items,
|
||||
[property: JsonPropertyName("nextCursor")] string? NextCursor,
|
||||
[property: JsonPropertyName("hasMore")] bool HasMore);
|
||||
|
||||
public sealed record GraphTooltipItem(
|
||||
[property: JsonPropertyName("purl")] string Purl,
|
||||
[property: JsonPropertyName("observations")] IReadOnlyList<GraphTooltipObservation> Observations,
|
||||
[property: JsonPropertyName("truncated")] bool Truncated);
|
||||
|
||||
public sealed record GraphTooltipObservation(
|
||||
[property: JsonPropertyName("observationId")] string ObservationId,
|
||||
[property: JsonPropertyName("advisoryId")] string AdvisoryId,
|
||||
[property: JsonPropertyName("status")] string Status,
|
||||
[property: JsonPropertyName("justification")] string? Justification,
|
||||
[property: JsonPropertyName("providerId")] string ProviderId,
|
||||
[property: JsonPropertyName("modifiedAt")] DateTimeOffset ModifiedAt,
|
||||
[property: JsonPropertyName("evidenceHash")] string EvidenceHash,
|
||||
[property: JsonPropertyName("dsseEnvelopeHash")] string? DsseEnvelopeHash);
|
||||
@@ -0,0 +1,52 @@
|
||||
|
||||
using StellaOps.Excititor.Core;
|
||||
using System;
|
||||
using System.Collections.Generic;
|
||||
using System.ComponentModel.DataAnnotations;
|
||||
using System.Text.Json.Serialization;
|
||||
|
||||
namespace StellaOps.Excititor.WebService.Contracts;
|
||||
|
||||
public sealed record PolicyVexLookupRequest
|
||||
{
|
||||
[JsonPropertyName("advisory_keys")]
|
||||
public IReadOnlyList<string> AdvisoryKeys { get; init; } = Array.Empty<string>();
|
||||
|
||||
[JsonPropertyName("purls")]
|
||||
public IReadOnlyList<string> Purls { get; init; } = Array.Empty<string>();
|
||||
|
||||
[JsonPropertyName("providers")]
|
||||
public IReadOnlyList<string> Providers { get; init; } = Array.Empty<string>();
|
||||
|
||||
[JsonPropertyName("statuses")]
|
||||
public IReadOnlyList<string> Statuses { get; init; } = Array.Empty<string>();
|
||||
|
||||
[Range(1, 500)]
|
||||
[JsonPropertyName("limit")]
|
||||
public int Limit { get; init; } = 200;
|
||||
}
|
||||
|
||||
public sealed record PolicyVexLookupResponse(
|
||||
IReadOnlyList<PolicyVexLookupItem> Results,
|
||||
int TotalStatements,
|
||||
DateTimeOffset GeneratedAtUtc);
|
||||
|
||||
public sealed record PolicyVexLookupItem(
|
||||
string AdvisoryKey,
|
||||
IReadOnlyList<string> Aliases,
|
||||
IReadOnlyList<PolicyVexStatement> Statements);
|
||||
|
||||
public sealed record PolicyVexStatement(
|
||||
string ObservationId,
|
||||
string ProviderId,
|
||||
string Status,
|
||||
string ProductKey,
|
||||
string? Purl,
|
||||
string? Cpe,
|
||||
string? Version,
|
||||
string? Justification,
|
||||
string? Detail,
|
||||
DateTimeOffset FirstSeen,
|
||||
DateTimeOffset LastSeen,
|
||||
VexSignatureMetadata? Signature,
|
||||
IReadOnlyDictionary<string, string> Metadata);
|
||||
@@ -0,0 +1,88 @@
|
||||
using System;
|
||||
using System.Collections.Generic;
|
||||
using System.Text.Json.Serialization;
|
||||
|
||||
namespace StellaOps.Excititor.WebService.Contracts;
|
||||
|
||||
/// <summary>
|
||||
/// Response for /attestations/vex/{attestationId} endpoint.
|
||||
/// </summary>
|
||||
public sealed record VexAttestationDetailResponse(
|
||||
[property: JsonPropertyName("attestationId")] string AttestationId,
|
||||
[property: JsonPropertyName("tenant")] string Tenant,
|
||||
[property: JsonPropertyName("createdAt")] DateTimeOffset CreatedAt,
|
||||
[property: JsonPropertyName("predicateType")] string PredicateType,
|
||||
[property: JsonPropertyName("subject")] VexAttestationSubject Subject,
|
||||
[property: JsonPropertyName("builder")] VexAttestationBuilderIdentity Builder,
|
||||
[property: JsonPropertyName("verification")] VexAttestationVerificationState Verification,
|
||||
[property: JsonPropertyName("chainOfCustody")] IReadOnlyList<VexAttestationCustodyLink> ChainOfCustody,
|
||||
[property: JsonPropertyName("metadata")] IReadOnlyDictionary<string, string> Metadata);
|
||||
|
||||
/// <summary>
|
||||
/// Subject of the attestation (what was signed).
|
||||
/// </summary>
|
||||
public sealed record VexAttestationSubject(
|
||||
[property: JsonPropertyName("digest")] string Digest,
|
||||
[property: JsonPropertyName("digestAlgorithm")] string DigestAlgorithm,
|
||||
[property: JsonPropertyName("name")] string? Name,
|
||||
[property: JsonPropertyName("uri")] string? Uri);
|
||||
|
||||
/// <summary>
|
||||
/// Builder identity for the attestation.
|
||||
/// </summary>
|
||||
public sealed record VexAttestationBuilderIdentity(
|
||||
[property: JsonPropertyName("id")] string Id,
|
||||
[property: JsonPropertyName("version")] string? Version,
|
||||
[property: JsonPropertyName("builderId")] string? BuilderId,
|
||||
[property: JsonPropertyName("invocationId")] string? InvocationId);
|
||||
|
||||
/// <summary>
|
||||
/// DSSE verification state.
|
||||
/// </summary>
|
||||
public sealed record VexAttestationVerificationState(
|
||||
[property: JsonPropertyName("valid")] bool Valid,
|
||||
[property: JsonPropertyName("verifiedAt")] DateTimeOffset? VerifiedAt,
|
||||
[property: JsonPropertyName("signatureType")] string? SignatureType,
|
||||
[property: JsonPropertyName("keyId")] string? KeyId,
|
||||
[property: JsonPropertyName("issuer")] string? Issuer,
|
||||
[property: JsonPropertyName("envelopeDigest")] string? EnvelopeDigest,
|
||||
[property: JsonPropertyName("diagnostics")] IReadOnlyDictionary<string, string> Diagnostics);
|
||||
|
||||
/// <summary>
|
||||
/// Chain-of-custody link in the attestation provenance.
|
||||
/// </summary>
|
||||
public sealed record VexAttestationCustodyLink(
|
||||
[property: JsonPropertyName("step")] int Step,
|
||||
[property: JsonPropertyName("actor")] string Actor,
|
||||
[property: JsonPropertyName("action")] string Action,
|
||||
[property: JsonPropertyName("timestamp")] DateTimeOffset Timestamp,
|
||||
[property: JsonPropertyName("reference")] string? Reference);
|
||||
|
||||
/// <summary>
|
||||
/// Response for /attestations/vex/list endpoint.
|
||||
/// </summary>
|
||||
public sealed record VexAttestationListResponse(
|
||||
[property: JsonPropertyName("items")] IReadOnlyList<VexAttestationListItem> Items,
|
||||
[property: JsonPropertyName("cursor")] string? Cursor,
|
||||
[property: JsonPropertyName("hasMore")] bool HasMore,
|
||||
[property: JsonPropertyName("total")] int Total);
|
||||
|
||||
/// <summary>
|
||||
/// Summary item for attestation list.
|
||||
/// </summary>
|
||||
public sealed record VexAttestationListItem(
|
||||
[property: JsonPropertyName("attestationId")] string AttestationId,
|
||||
[property: JsonPropertyName("tenant")] string Tenant,
|
||||
[property: JsonPropertyName("createdAt")] DateTimeOffset CreatedAt,
|
||||
[property: JsonPropertyName("predicateType")] string PredicateType,
|
||||
[property: JsonPropertyName("subjectDigest")] string SubjectDigest,
|
||||
[property: JsonPropertyName("valid")] bool Valid,
|
||||
[property: JsonPropertyName("builderId")] string? BuilderId);
|
||||
|
||||
/// <summary>
|
||||
/// Response for /attestations/vex/lookup endpoint.
|
||||
/// </summary>
|
||||
public sealed record VexAttestationLookupResponse(
|
||||
[property: JsonPropertyName("subjectDigest")] string SubjectDigest,
|
||||
[property: JsonPropertyName("attestations")] IReadOnlyList<VexAttestationListItem> Attestations,
|
||||
[property: JsonPropertyName("queriedAt")] DateTimeOffset QueriedAt);
|
||||
@@ -0,0 +1,142 @@
|
||||
using System.Text.Json.Serialization;
|
||||
|
||||
namespace StellaOps.Excititor.WebService.Contracts;
|
||||
|
||||
/// <summary>
|
||||
/// Request for POST /api/v1/vex/candidates/{candidateId}/approve.
|
||||
/// Sprint: SPRINT_4000_0100_0002 - UI-Driven Vulnerability Annotation.
|
||||
/// </summary>
|
||||
public sealed record VexCandidateApprovalRequest
|
||||
{
|
||||
[JsonPropertyName("status")]
|
||||
public required string Status { get; init; }
|
||||
|
||||
[JsonPropertyName("justification")]
|
||||
public required string Justification { get; init; }
|
||||
|
||||
[JsonPropertyName("justification_text")]
|
||||
public string? JustificationText { get; init; }
|
||||
|
||||
[JsonPropertyName("valid_until")]
|
||||
public DateTimeOffset? ValidUntil { get; init; }
|
||||
|
||||
[JsonPropertyName("approval_notes")]
|
||||
public string? ApprovalNotes { get; init; }
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Request for POST /api/v1/vex/candidates/{candidateId}/reject.
|
||||
/// </summary>
|
||||
public sealed record VexCandidateRejectionRequest
|
||||
{
|
||||
[JsonPropertyName("reason")]
|
||||
public required string Reason { get; init; }
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Response for POST /api/v1/vex/candidates/{candidateId}/approve.
|
||||
/// </summary>
|
||||
public sealed record VexStatementResponse
|
||||
{
|
||||
[JsonPropertyName("statement_id")]
|
||||
public required string StatementId { get; init; }
|
||||
|
||||
[JsonPropertyName("vulnerability_id")]
|
||||
public required string VulnerabilityId { get; init; }
|
||||
|
||||
[JsonPropertyName("product_id")]
|
||||
public required string ProductId { get; init; }
|
||||
|
||||
[JsonPropertyName("status")]
|
||||
public required string Status { get; init; }
|
||||
|
||||
[JsonPropertyName("justification")]
|
||||
public required string Justification { get; init; }
|
||||
|
||||
[JsonPropertyName("justification_text")]
|
||||
public string? JustificationText { get; init; }
|
||||
|
||||
[JsonPropertyName("timestamp")]
|
||||
public required DateTimeOffset Timestamp { get; init; }
|
||||
|
||||
[JsonPropertyName("valid_until")]
|
||||
public DateTimeOffset? ValidUntil { get; init; }
|
||||
|
||||
[JsonPropertyName("approved_by")]
|
||||
public required string ApprovedBy { get; init; }
|
||||
|
||||
[JsonPropertyName("source_candidate")]
|
||||
public string? SourceCandidate { get; init; }
|
||||
|
||||
[JsonPropertyName("dsse_envelope_digest")]
|
||||
public string? DsseEnvelopeDigest { get; init; }
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// VEX candidate summary.
|
||||
/// </summary>
|
||||
public sealed record VexCandidateDto
|
||||
{
|
||||
[JsonPropertyName("candidate_id")]
|
||||
public required string CandidateId { get; init; }
|
||||
|
||||
[JsonPropertyName("finding_id")]
|
||||
public required string FindingId { get; init; }
|
||||
|
||||
[JsonPropertyName("vulnerability_id")]
|
||||
public required string VulnerabilityId { get; init; }
|
||||
|
||||
[JsonPropertyName("product_id")]
|
||||
public required string ProductId { get; init; }
|
||||
|
||||
[JsonPropertyName("suggested_status")]
|
||||
public required string SuggestedStatus { get; init; }
|
||||
|
||||
[JsonPropertyName("suggested_justification")]
|
||||
public required string SuggestedJustification { get; init; }
|
||||
|
||||
[JsonPropertyName("justification_text")]
|
||||
public string? JustificationText { get; init; }
|
||||
|
||||
[JsonPropertyName("confidence")]
|
||||
public double Confidence { get; init; }
|
||||
|
||||
[JsonPropertyName("source")]
|
||||
public required string Source { get; init; }
|
||||
|
||||
[JsonPropertyName("evidence_digests")]
|
||||
public IReadOnlyList<string>? EvidenceDigests { get; init; }
|
||||
|
||||
[JsonPropertyName("created_at")]
|
||||
public required DateTimeOffset CreatedAt { get; init; }
|
||||
|
||||
[JsonPropertyName("expires_at")]
|
||||
public DateTimeOffset? ExpiresAt { get; init; }
|
||||
|
||||
[JsonPropertyName("status")]
|
||||
public required string Status { get; init; }
|
||||
|
||||
[JsonPropertyName("reviewed_by")]
|
||||
public string? ReviewedBy { get; init; }
|
||||
|
||||
[JsonPropertyName("reviewed_at")]
|
||||
public DateTimeOffset? ReviewedAt { get; init; }
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// VEX candidates list response.
|
||||
/// </summary>
|
||||
public sealed record VexCandidatesListResponse
|
||||
{
|
||||
[JsonPropertyName("items")]
|
||||
public required IReadOnlyList<VexCandidateDto> Items { get; init; }
|
||||
|
||||
[JsonPropertyName("total")]
|
||||
public int Total { get; init; }
|
||||
|
||||
[JsonPropertyName("limit")]
|
||||
public int Limit { get; init; }
|
||||
|
||||
[JsonPropertyName("offset")]
|
||||
public int Offset { get; init; }
|
||||
}
|
||||
@@ -0,0 +1,22 @@
|
||||
using System;
|
||||
using System.Collections.Generic;
|
||||
|
||||
namespace StellaOps.Excititor.WebService.Contracts;
|
||||
|
||||
public sealed record VexConsoleStatementDto(
|
||||
string AdvisoryId,
|
||||
string ProductKey,
|
||||
string? Purl,
|
||||
string Status,
|
||||
string? Justification,
|
||||
string ProviderId,
|
||||
string ObservationId,
|
||||
DateTimeOffset CreatedAtUtc,
|
||||
IReadOnlyDictionary<string, string> Attributes);
|
||||
|
||||
public sealed record VexConsolePage(
|
||||
IReadOnlyList<VexConsoleStatementDto> Items,
|
||||
string? Cursor,
|
||||
bool HasMore,
|
||||
int Returned,
|
||||
IReadOnlyDictionary<string, int>? Counters = null);
|
||||
@@ -0,0 +1,44 @@
|
||||
using System;
|
||||
using System.Collections.Generic;
|
||||
using System.Text.Json.Serialization;
|
||||
|
||||
namespace StellaOps.Excititor.WebService.Contracts;
|
||||
|
||||
public sealed record VexEvidenceChunkResponse(
|
||||
[property: JsonPropertyName("observationId")] string ObservationId,
|
||||
[property: JsonPropertyName("linksetId")] string LinksetId,
|
||||
[property: JsonPropertyName("vulnerabilityId")] string VulnerabilityId,
|
||||
[property: JsonPropertyName("productKey")] string ProductKey,
|
||||
[property: JsonPropertyName("providerId")] string ProviderId,
|
||||
[property: JsonPropertyName("status")] string Status,
|
||||
[property: JsonPropertyName("justification")] string? Justification,
|
||||
[property: JsonPropertyName("detail")] string? Detail,
|
||||
[property: JsonPropertyName("scopeScore")] double? ScopeScore,
|
||||
[property: JsonPropertyName("firstSeen")] DateTimeOffset FirstSeen,
|
||||
[property: JsonPropertyName("lastSeen")] DateTimeOffset LastSeen,
|
||||
[property: JsonPropertyName("scope")] VexEvidenceChunkScope Scope,
|
||||
[property: JsonPropertyName("document")] VexEvidenceChunkDocument Document,
|
||||
[property: JsonPropertyName("signature")] VexEvidenceChunkSignature? Signature,
|
||||
[property: JsonPropertyName("metadata")] IReadOnlyDictionary<string, string> Metadata);
|
||||
|
||||
public sealed record VexEvidenceChunkScope(
|
||||
[property: JsonPropertyName("key")] string Key,
|
||||
[property: JsonPropertyName("name")] string? Name,
|
||||
[property: JsonPropertyName("version")] string? Version,
|
||||
[property: JsonPropertyName("purl")] string? Purl,
|
||||
[property: JsonPropertyName("cpe")] string? Cpe,
|
||||
[property: JsonPropertyName("componentIdentifiers")] IReadOnlyList<string> ComponentIdentifiers);
|
||||
|
||||
public sealed record VexEvidenceChunkDocument(
|
||||
[property: JsonPropertyName("digest")] string Digest,
|
||||
[property: JsonPropertyName("format")] string Format,
|
||||
[property: JsonPropertyName("sourceUri")] string SourceUri,
|
||||
[property: JsonPropertyName("revision")] string? Revision);
|
||||
|
||||
public sealed record VexEvidenceChunkSignature(
|
||||
[property: JsonPropertyName("type")] string Type,
|
||||
[property: JsonPropertyName("subject")] string? Subject,
|
||||
[property: JsonPropertyName("issuer")] string? Issuer,
|
||||
[property: JsonPropertyName("keyId")] string? KeyId,
|
||||
[property: JsonPropertyName("verifiedAt")] DateTimeOffset? VerifiedAt,
|
||||
[property: JsonPropertyName("transparencyRef")] string? TransparencyRef);
|
||||
@@ -0,0 +1,171 @@
|
||||
using System;
|
||||
using System.Collections.Generic;
|
||||
using System.Text.Json.Serialization;
|
||||
|
||||
namespace StellaOps.Excititor.WebService.Contracts;
|
||||
|
||||
/// <summary>
|
||||
/// Response for /evidence/vex/bundle/{bundleId} endpoint.
|
||||
/// </summary>
|
||||
public sealed record VexEvidenceBundleResponse(
|
||||
[property: JsonPropertyName("bundleId")] string BundleId,
|
||||
[property: JsonPropertyName("tenant")] string Tenant,
|
||||
[property: JsonPropertyName("createdAt")] DateTimeOffset CreatedAt,
|
||||
[property: JsonPropertyName("contentHash")] string ContentHash,
|
||||
[property: JsonPropertyName("format")] string Format,
|
||||
[property: JsonPropertyName("itemCount")] int ItemCount,
|
||||
[property: JsonPropertyName("verification")] VexEvidenceVerificationMetadata? Verification,
|
||||
[property: JsonPropertyName("metadata")] IReadOnlyDictionary<string, string> Metadata);
|
||||
|
||||
/// <summary>
|
||||
/// Verification metadata for evidence bundles.
|
||||
/// </summary>
|
||||
public sealed record VexEvidenceVerificationMetadata(
|
||||
[property: JsonPropertyName("verified")] bool Verified,
|
||||
[property: JsonPropertyName("verifiedAt")] DateTimeOffset? VerifiedAt,
|
||||
[property: JsonPropertyName("signatureType")] string? SignatureType,
|
||||
[property: JsonPropertyName("keyId")] string? KeyId,
|
||||
[property: JsonPropertyName("issuer")] string? Issuer,
|
||||
[property: JsonPropertyName("transparencyRef")] string? TransparencyRef);
|
||||
|
||||
/// <summary>
|
||||
/// Response for /evidence/vex/list endpoint.
|
||||
/// </summary>
|
||||
public sealed record VexEvidenceListResponse(
|
||||
[property: JsonPropertyName("items")] IReadOnlyList<VexEvidenceListItem> Items,
|
||||
[property: JsonPropertyName("cursor")] string? Cursor,
|
||||
[property: JsonPropertyName("hasMore")] bool HasMore,
|
||||
[property: JsonPropertyName("total")] int Total);
|
||||
|
||||
/// <summary>
|
||||
/// Summary item for evidence list.
|
||||
/// </summary>
|
||||
public sealed record VexEvidenceListItem(
|
||||
[property: JsonPropertyName("bundleId")] string BundleId,
|
||||
[property: JsonPropertyName("tenant")] string Tenant,
|
||||
[property: JsonPropertyName("createdAt")] DateTimeOffset CreatedAt,
|
||||
[property: JsonPropertyName("contentHash")] string ContentHash,
|
||||
[property: JsonPropertyName("format")] string Format,
|
||||
[property: JsonPropertyName("itemCount")] int ItemCount,
|
||||
[property: JsonPropertyName("verified")] bool Verified);
|
||||
|
||||
/// <summary>
|
||||
/// Evidence Locker manifest reference returned by /evidence/vex/locker/*.
|
||||
/// </summary>
|
||||
public sealed record VexEvidenceLockerResponse(
|
||||
[property: JsonPropertyName("bundleId")] string BundleId,
|
||||
[property: JsonPropertyName("mirrorGeneration")] string MirrorGeneration,
|
||||
[property: JsonPropertyName("tenant")] string Tenant,
|
||||
[property: JsonPropertyName("publisher")] string Publisher,
|
||||
[property: JsonPropertyName("payloadHash")] string PayloadHash,
|
||||
[property: JsonPropertyName("manifestPath")] string ManifestPath,
|
||||
[property: JsonPropertyName("manifestHash")] string ManifestHash,
|
||||
[property: JsonPropertyName("evidencePath")] string EvidencePath,
|
||||
[property: JsonPropertyName("evidenceHash")] string? EvidenceHash,
|
||||
[property: JsonPropertyName("manifestSizeBytes")] long? ManifestSizeBytes,
|
||||
[property: JsonPropertyName("evidenceSizeBytes")] long? EvidenceSizeBytes,
|
||||
[property: JsonPropertyName("importedAt")] DateTimeOffset ImportedAt,
|
||||
[property: JsonPropertyName("stalenessSeconds")] int? StalenessSeconds,
|
||||
[property: JsonPropertyName("transparencyLog")] string? TransparencyLog,
|
||||
[property: JsonPropertyName("timeline")] IReadOnlyList<VexEvidenceLockerTimelineEntry> Timeline);
|
||||
|
||||
/// <summary>
|
||||
/// Timeline event for air-gapped imports.
|
||||
/// </summary>
|
||||
public sealed record VexEvidenceLockerTimelineEntry(
|
||||
[property: JsonPropertyName("eventType")] string EventType,
|
||||
[property: JsonPropertyName("createdAt")] DateTimeOffset CreatedAt,
|
||||
[property: JsonPropertyName("errorCode")] string? ErrorCode,
|
||||
[property: JsonPropertyName("message")] string? Message,
|
||||
[property: JsonPropertyName("stalenessSeconds")] int? StalenessSeconds);
|
||||
|
||||
/// <summary>
|
||||
/// Response for /evidence/vex/lookup endpoint.
|
||||
/// </summary>
|
||||
public sealed record VexEvidenceLookupResponse(
|
||||
[property: JsonPropertyName("vulnerabilityId")] string VulnerabilityId,
|
||||
[property: JsonPropertyName("productKey")] string ProductKey,
|
||||
[property: JsonPropertyName("evidenceItems")] IReadOnlyList<VexEvidenceItem> EvidenceItems,
|
||||
[property: JsonPropertyName("queriedAt")] DateTimeOffset QueriedAt);
|
||||
|
||||
/// <summary>
|
||||
/// Individual evidence item for a vuln/product pair.
|
||||
/// </summary>
|
||||
public sealed record VexEvidenceItem(
|
||||
[property: JsonPropertyName("observationId")] string ObservationId,
|
||||
[property: JsonPropertyName("providerId")] string ProviderId,
|
||||
[property: JsonPropertyName("status")] string Status,
|
||||
[property: JsonPropertyName("justification")] string? Justification,
|
||||
[property: JsonPropertyName("firstSeen")] DateTimeOffset FirstSeen,
|
||||
[property: JsonPropertyName("lastSeen")] DateTimeOffset LastSeen,
|
||||
[property: JsonPropertyName("documentDigest")] string DocumentDigest,
|
||||
[property: JsonPropertyName("verification")] VexEvidenceVerificationMetadata? Verification);
|
||||
|
||||
/// <summary>
|
||||
/// Response for /vuln/evidence/vex/{advisory_key} endpoint.
|
||||
/// Returns tenant-scoped raw statements for Vuln Explorer evidence tabs.
|
||||
/// </summary>
|
||||
public sealed record VexAdvisoryEvidenceResponse(
|
||||
[property: JsonPropertyName("advisoryKey")] string AdvisoryKey,
|
||||
[property: JsonPropertyName("canonicalKey")] string CanonicalKey,
|
||||
[property: JsonPropertyName("scope")] string Scope,
|
||||
[property: JsonPropertyName("aliases")] IReadOnlyList<VexAdvisoryLinkResponse> Aliases,
|
||||
[property: JsonPropertyName("statements")] IReadOnlyList<VexAdvisoryStatementResponse> Statements,
|
||||
[property: JsonPropertyName("queriedAt")] DateTimeOffset QueriedAt,
|
||||
[property: JsonPropertyName("totalCount")] int TotalCount);
|
||||
|
||||
/// <summary>
|
||||
/// Advisory link for traceability (CVE, GHSA, RHSA, etc.).
|
||||
/// </summary>
|
||||
public sealed record VexAdvisoryLinkResponse(
|
||||
[property: JsonPropertyName("identifier")] string Identifier,
|
||||
[property: JsonPropertyName("type")] string Type,
|
||||
[property: JsonPropertyName("isOriginal")] bool IsOriginal);
|
||||
|
||||
/// <summary>
|
||||
/// Raw VEX statement for an advisory with provenance and attestation metadata.
|
||||
/// </summary>
|
||||
public sealed record VexAdvisoryStatementResponse(
|
||||
[property: JsonPropertyName("statementId")] string StatementId,
|
||||
[property: JsonPropertyName("providerId")] string ProviderId,
|
||||
[property: JsonPropertyName("product")] VexAdvisoryProductResponse Product,
|
||||
[property: JsonPropertyName("status")] string Status,
|
||||
[property: JsonPropertyName("justification")] string? Justification,
|
||||
[property: JsonPropertyName("detail")] string? Detail,
|
||||
[property: JsonPropertyName("firstSeen")] DateTimeOffset FirstSeen,
|
||||
[property: JsonPropertyName("lastSeen")] DateTimeOffset LastSeen,
|
||||
[property: JsonPropertyName("provenance")] VexAdvisoryProvenanceResponse Provenance,
|
||||
[property: JsonPropertyName("attestation")] VexAdvisoryAttestationResponse? Attestation);
|
||||
|
||||
/// <summary>
|
||||
/// Product information for an advisory statement.
|
||||
/// </summary>
|
||||
public sealed record VexAdvisoryProductResponse(
|
||||
[property: JsonPropertyName("key")] string Key,
|
||||
[property: JsonPropertyName("name")] string? Name,
|
||||
[property: JsonPropertyName("version")] string? Version,
|
||||
[property: JsonPropertyName("purl")] string? Purl,
|
||||
[property: JsonPropertyName("cpe")] string? Cpe);
|
||||
|
||||
/// <summary>
|
||||
/// Provenance metadata for a VEX statement.
|
||||
/// </summary>
|
||||
public sealed record VexAdvisoryProvenanceResponse(
|
||||
[property: JsonPropertyName("documentDigest")] string DocumentDigest,
|
||||
[property: JsonPropertyName("documentFormat")] string DocumentFormat,
|
||||
[property: JsonPropertyName("sourceUri")] string SourceUri,
|
||||
[property: JsonPropertyName("revision")] string? Revision,
|
||||
[property: JsonPropertyName("insertedAt")] DateTimeOffset InsertedAt);
|
||||
|
||||
/// <summary>
|
||||
/// Attestation metadata for signature verification.
|
||||
/// </summary>
|
||||
public sealed record VexAdvisoryAttestationResponse(
|
||||
[property: JsonPropertyName("signatureType")] string SignatureType,
|
||||
[property: JsonPropertyName("issuer")] string? Issuer,
|
||||
[property: JsonPropertyName("subject")] string? Subject,
|
||||
[property: JsonPropertyName("keyId")] string? KeyId,
|
||||
[property: JsonPropertyName("verifiedAt")] DateTimeOffset? VerifiedAt,
|
||||
[property: JsonPropertyName("transparencyLogRef")] string? TransparencyLogRef,
|
||||
[property: JsonPropertyName("trustWeight")] decimal? TrustWeight,
|
||||
[property: JsonPropertyName("trustTier")] string? TrustTier);
|
||||
@@ -0,0 +1,49 @@
|
||||
using System;
|
||||
using System.Collections.Generic;
|
||||
using System.Text.Json.Serialization;
|
||||
|
||||
namespace StellaOps.Excititor.WebService.Contracts;
|
||||
|
||||
public sealed record VexLinksetListResponse(
|
||||
[property: JsonPropertyName("items")] IReadOnlyList<VexLinksetListItem> Items,
|
||||
[property: JsonPropertyName("nextCursor")] string? NextCursor);
|
||||
|
||||
public sealed record VexLinksetListItem(
|
||||
[property: JsonPropertyName("linksetId")] string LinksetId,
|
||||
[property: JsonPropertyName("tenant")] string Tenant,
|
||||
[property: JsonPropertyName("vulnerabilityId")] string VulnerabilityId,
|
||||
[property: JsonPropertyName("productKey")] string ProductKey,
|
||||
[property: JsonPropertyName("scope")] VexLinksetScope Scope,
|
||||
[property: JsonPropertyName("providerIds")] IReadOnlyList<string> ProviderIds,
|
||||
[property: JsonPropertyName("statuses")] IReadOnlyList<string> Statuses,
|
||||
[property: JsonPropertyName("aliases")] IReadOnlyList<string> Aliases,
|
||||
[property: JsonPropertyName("purls")] IReadOnlyList<string> Purls,
|
||||
[property: JsonPropertyName("cpes")] IReadOnlyList<string> Cpes,
|
||||
[property: JsonPropertyName("references")] IReadOnlyList<VexLinksetReference> References,
|
||||
[property: JsonPropertyName("disagreements")] IReadOnlyList<VexLinksetDisagreement> Disagreements,
|
||||
[property: JsonPropertyName("observations")] IReadOnlyList<VexLinksetObservationRef> Observations,
|
||||
[property: JsonPropertyName("createdAt")] DateTimeOffset CreatedAt);
|
||||
|
||||
public sealed record VexLinksetReference(
|
||||
[property: JsonPropertyName("type")] string Type,
|
||||
[property: JsonPropertyName("url")] string Url);
|
||||
|
||||
public sealed record VexLinksetDisagreement(
|
||||
[property: JsonPropertyName("providerId")] string ProviderId,
|
||||
[property: JsonPropertyName("status")] string Status,
|
||||
[property: JsonPropertyName("justification")] string? Justification,
|
||||
[property: JsonPropertyName("confidence")] double? Confidence);
|
||||
|
||||
public sealed record VexLinksetObservationRef(
|
||||
[property: JsonPropertyName("observationId")] string ObservationId,
|
||||
[property: JsonPropertyName("providerId")] string ProviderId,
|
||||
[property: JsonPropertyName("status")] string Status,
|
||||
[property: JsonPropertyName("confidence")] double? Confidence);
|
||||
|
||||
public sealed record VexLinksetScope(
|
||||
[property: JsonPropertyName("productKey")] string ProductKey,
|
||||
[property: JsonPropertyName("type")] string Type,
|
||||
[property: JsonPropertyName("version")] string? Version,
|
||||
[property: JsonPropertyName("purl")] string? Purl,
|
||||
[property: JsonPropertyName("cpe")] string? Cpe,
|
||||
[property: JsonPropertyName("identifiers")] IReadOnlyList<string> Identifiers);
|
||||
@@ -0,0 +1,46 @@
|
||||
using System;
|
||||
using System.Collections.Generic;
|
||||
using System.Text.Json.Serialization;
|
||||
|
||||
namespace StellaOps.Excititor.WebService.Contracts;
|
||||
|
||||
public sealed record VexObservationProjectionResponse(
|
||||
[property: JsonPropertyName("vulnerabilityId")] string VulnerabilityId,
|
||||
[property: JsonPropertyName("productKey")] string ProductKey,
|
||||
[property: JsonPropertyName("generatedAt") ] DateTimeOffset GeneratedAt,
|
||||
[property: JsonPropertyName("totalCount")] int TotalCount,
|
||||
[property: JsonPropertyName("truncated")] bool Truncated,
|
||||
[property: JsonPropertyName("statements")] IReadOnlyList<VexObservationStatementResponse> Statements);
|
||||
|
||||
public sealed record VexObservationStatementResponse(
|
||||
[property: JsonPropertyName("observationId")] string ObservationId,
|
||||
[property: JsonPropertyName("providerId")] string ProviderId,
|
||||
[property: JsonPropertyName("status")] string Status,
|
||||
[property: JsonPropertyName("justification")] string? Justification,
|
||||
[property: JsonPropertyName("detail")] string? Detail,
|
||||
[property: JsonPropertyName("firstSeen")] DateTimeOffset FirstSeen,
|
||||
[property: JsonPropertyName("lastSeen")] DateTimeOffset LastSeen,
|
||||
[property: JsonPropertyName("scope")] VexObservationScopeResponse Scope,
|
||||
[property: JsonPropertyName("anchors")] IReadOnlyList<string> Anchors,
|
||||
[property: JsonPropertyName("document")] VexObservationDocumentResponse Document,
|
||||
[property: JsonPropertyName("signature")] VexObservationSignatureResponse? Signature);
|
||||
|
||||
public sealed record VexObservationScopeResponse(
|
||||
[property: JsonPropertyName("key")] string Key,
|
||||
[property: JsonPropertyName("name")] string? Name,
|
||||
[property: JsonPropertyName("version")] string? Version,
|
||||
[property: JsonPropertyName("purl")] string? Purl,
|
||||
[property: JsonPropertyName("cpe")] string? Cpe,
|
||||
[property: JsonPropertyName("componentIdentifiers")] IReadOnlyList<string> ComponentIdentifiers);
|
||||
|
||||
public sealed record VexObservationDocumentResponse(
|
||||
[property: JsonPropertyName("digest")] string Digest,
|
||||
[property: JsonPropertyName("format")] string Format,
|
||||
[property: JsonPropertyName("revision")] string? Revision,
|
||||
[property: JsonPropertyName("sourceUri")] string SourceUri);
|
||||
|
||||
public sealed record VexObservationSignatureResponse(
|
||||
[property: JsonPropertyName("type")] string Type,
|
||||
[property: JsonPropertyName("keyId")] string? KeyId,
|
||||
[property: JsonPropertyName("issuer")] string? Issuer,
|
||||
[property: JsonPropertyName("verifiedAt")] DateTimeOffset? VerifiedAtUtc);
|
||||
@@ -0,0 +1,20 @@
|
||||
using System;
|
||||
using System.Collections.Generic;
|
||||
using System.Text.Json.Serialization;
|
||||
|
||||
namespace StellaOps.Excititor.WebService.Contracts;
|
||||
|
||||
public sealed record VexObservationListResponse(
|
||||
[property: JsonPropertyName("items")] IReadOnlyList<VexObservationListItem> Items,
|
||||
[property: JsonPropertyName("nextCursor")] string? NextCursor);
|
||||
|
||||
public sealed record VexObservationListItem(
|
||||
[property: JsonPropertyName("observationId")] string ObservationId,
|
||||
[property: JsonPropertyName("tenant")] string Tenant,
|
||||
[property: JsonPropertyName("providerId")] string ProviderId,
|
||||
[property: JsonPropertyName("vulnerabilityId")] string VulnerabilityId,
|
||||
[property: JsonPropertyName("productKey")] string ProductKey,
|
||||
[property: JsonPropertyName("status")] string Status,
|
||||
[property: JsonPropertyName("createdAt")] DateTimeOffset CreatedAt,
|
||||
[property: JsonPropertyName("lastObserved")] DateTimeOffset? LastObserved,
|
||||
[property: JsonPropertyName("purls")] IReadOnlyList<string> Purls);
|
||||
@@ -0,0 +1,115 @@
|
||||
|
||||
using RawVexDocumentModel = StellaOps.Concelier.RawModels.VexRawDocument;
|
||||
using StellaOps.Concelier.RawModels;
|
||||
using System;
|
||||
using System.Collections.Generic;
|
||||
using System.Text.Json;
|
||||
using System.Text.Json.Serialization;
|
||||
|
||||
namespace StellaOps.Excititor.WebService.Contracts;
|
||||
|
||||
public sealed record VexIngestRequest(
|
||||
[property: JsonPropertyName("providerId")] string ProviderId,
|
||||
[property: JsonPropertyName("source")] VexIngestSourceRequest Source,
|
||||
[property: JsonPropertyName("upstream")] VexIngestUpstreamRequest Upstream,
|
||||
[property: JsonPropertyName("content")] VexIngestContentRequest Content,
|
||||
[property: JsonPropertyName("metadata")] IReadOnlyDictionary<string, string>? Metadata);
|
||||
|
||||
public sealed record VexIngestSourceRequest(
|
||||
[property: JsonPropertyName("vendor")] string Vendor,
|
||||
[property: JsonPropertyName("connector")] string Connector,
|
||||
[property: JsonPropertyName("version")] string Version,
|
||||
[property: JsonPropertyName("stream")] string? Stream);
|
||||
|
||||
public sealed record VexIngestUpstreamRequest(
|
||||
[property: JsonPropertyName("sourceUri")] string SourceUri,
|
||||
[property: JsonPropertyName("upstreamId")] string UpstreamId,
|
||||
[property: JsonPropertyName("documentVersion")] string? DocumentVersion,
|
||||
[property: JsonPropertyName("retrievedAt")] DateTimeOffset? RetrievedAt,
|
||||
[property: JsonPropertyName("contentHash")] string? ContentHash,
|
||||
[property: JsonPropertyName("signature")] VexIngestSignatureRequest? Signature,
|
||||
[property: JsonPropertyName("provenance")] IReadOnlyDictionary<string, string>? Provenance);
|
||||
|
||||
public sealed record VexIngestSignatureRequest(
|
||||
[property: JsonPropertyName("present")] bool Present,
|
||||
[property: JsonPropertyName("format")] string? Format,
|
||||
[property: JsonPropertyName("keyId")] string? KeyId,
|
||||
[property: JsonPropertyName("sig")] string? Signature,
|
||||
[property: JsonPropertyName("certificate")] string? Certificate,
|
||||
[property: JsonPropertyName("digest")] string? Digest);
|
||||
|
||||
public sealed record VexIngestContentRequest(
|
||||
[property: JsonPropertyName("format")] string Format,
|
||||
[property: JsonPropertyName("specVersion")] string? SpecVersion,
|
||||
[property: JsonPropertyName("raw")] JsonElement Raw,
|
||||
[property: JsonPropertyName("encoding")] string? Encoding);
|
||||
|
||||
public sealed record VexIngestResponse(
|
||||
[property: JsonPropertyName("digest")] string Digest,
|
||||
[property: JsonPropertyName("inserted")] bool Inserted,
|
||||
[property: JsonPropertyName("tenant")] string Tenant,
|
||||
[property: JsonPropertyName("retrievedAt")] DateTimeOffset RetrievedAt);
|
||||
|
||||
public sealed record VexRawSummaryResponse(
|
||||
[property: JsonPropertyName("digest")] string Digest,
|
||||
[property: JsonPropertyName("providerId")] string ProviderId,
|
||||
[property: JsonPropertyName("format")] string Format,
|
||||
[property: JsonPropertyName("sourceUri")] string SourceUri,
|
||||
[property: JsonPropertyName("retrievedAt")] DateTimeOffset RetrievedAt,
|
||||
[property: JsonPropertyName("inlineContent")] bool InlineContent,
|
||||
[property: JsonPropertyName("metadata")] IReadOnlyDictionary<string, string> Metadata);
|
||||
|
||||
public sealed record VexRawListResponse(
|
||||
[property: JsonPropertyName("records")] IReadOnlyList<VexRawSummaryResponse> Records,
|
||||
[property: JsonPropertyName("nextCursor")] string? NextCursor,
|
||||
[property: JsonPropertyName("hasMore")] bool HasMore);
|
||||
|
||||
public sealed record VexRawRecordResponse(
|
||||
[property: JsonPropertyName("digest")] string Digest,
|
||||
[property: JsonPropertyName("document")] RawVexDocumentModel Document,
|
||||
[property: JsonPropertyName("retrievedAt")] DateTimeOffset RetrievedAt);
|
||||
|
||||
public sealed record VexRawProvenanceResponse(
|
||||
[property: JsonPropertyName("digest")] string Digest,
|
||||
[property: JsonPropertyName("tenant")] string Tenant,
|
||||
[property: JsonPropertyName("source")] RawSourceMetadata Source,
|
||||
[property: JsonPropertyName("upstream")] RawUpstreamMetadata Upstream,
|
||||
[property: JsonPropertyName("retrievedAt")] DateTimeOffset RetrievedAt);
|
||||
|
||||
public sealed record VexAocVerifyRequest(
|
||||
[property: JsonPropertyName("since")] DateTimeOffset? Since,
|
||||
[property: JsonPropertyName("until")] DateTimeOffset? Until,
|
||||
[property: JsonPropertyName("limit")] int? Limit,
|
||||
[property: JsonPropertyName("sources")] IReadOnlyList<string>? Sources,
|
||||
[property: JsonPropertyName("codes")] IReadOnlyList<string>? Codes);
|
||||
|
||||
public sealed record VexAocVerifyResponse(
|
||||
[property: JsonPropertyName("tenant")] string Tenant,
|
||||
[property: JsonPropertyName("window")] VexAocVerifyWindow Window,
|
||||
[property: JsonPropertyName("checked")] VexAocVerifyChecked Checked,
|
||||
[property: JsonPropertyName("violations")] IReadOnlyList<VexAocVerifyViolation> Violations,
|
||||
[property: JsonPropertyName("metrics")] VexAocVerifyMetrics Metrics,
|
||||
[property: JsonPropertyName("truncated")] bool Truncated);
|
||||
|
||||
public sealed record VexAocVerifyWindow(
|
||||
[property: JsonPropertyName("from")] DateTimeOffset From,
|
||||
[property: JsonPropertyName("to")] DateTimeOffset To);
|
||||
|
||||
public sealed record VexAocVerifyChecked(
|
||||
[property: JsonPropertyName("advisories")] int Advisories,
|
||||
[property: JsonPropertyName("vex")] int Vex);
|
||||
|
||||
public sealed record VexAocVerifyMetrics(
|
||||
[property: JsonPropertyName("ingestion_write_total")] int IngestionWriteTotal,
|
||||
[property: JsonPropertyName("aoc_violation_total")] int AocViolationTotal);
|
||||
|
||||
public sealed record VexAocVerifyViolation(
|
||||
[property: JsonPropertyName("code")] string Code,
|
||||
[property: JsonPropertyName("count")] int Count,
|
||||
[property: JsonPropertyName("examples")] IReadOnlyList<VexAocVerifyViolationExample> Examples);
|
||||
|
||||
public sealed record VexAocVerifyViolationExample(
|
||||
[property: JsonPropertyName("source")] string Source,
|
||||
[property: JsonPropertyName("documentId")] string DocumentId,
|
||||
[property: JsonPropertyName("contentHash")] string ContentHash,
|
||||
[property: JsonPropertyName("path")] string Path);
|
||||
@@ -0,0 +1,182 @@
|
||||
|
||||
using Microsoft.AspNetCore.Builder;
|
||||
using Microsoft.AspNetCore.Http;
|
||||
using Microsoft.AspNetCore.Mvc;
|
||||
using Microsoft.Extensions.Options;
|
||||
using static Program;
|
||||
using StellaOps.Auth.ServerIntegration.Tenancy;
|
||||
using StellaOps.Excititor.Core.Evidence;
|
||||
using StellaOps.Excititor.Core.Storage;
|
||||
using StellaOps.Excititor.WebService.Security;
|
||||
using StellaOps.Excititor.WebService.Services;
|
||||
using System.Text.Json.Serialization;
|
||||
|
||||
namespace StellaOps.Excititor.WebService.Endpoints;
|
||||
|
||||
/// <summary>
|
||||
/// Attestation API endpoints for listing and retrieving DSSE attestations.
|
||||
/// </summary>
|
||||
public static class AttestationEndpoints
|
||||
{
|
||||
public static void MapAttestationEndpoints(this WebApplication app)
|
||||
{
|
||||
// GET /attestations/vex/list
|
||||
app.MapGet("/attestations/vex/list", async (
|
||||
HttpContext context,
|
||||
[FromQuery] string? since,
|
||||
[FromQuery] string? until,
|
||||
[FromQuery] int? limit,
|
||||
[FromQuery] int? offset,
|
||||
IOptions<VexStorageOptions> storageOptions,
|
||||
[FromServices] IVexAttestationStore? attestationStore,
|
||||
CancellationToken cancellationToken) =>
|
||||
{
|
||||
var scopeResult = ScopeAuthorization.RequireScope(context, "vex.read");
|
||||
if (scopeResult is not null)
|
||||
{
|
||||
return scopeResult;
|
||||
}
|
||||
|
||||
if (!TryResolveTenant(context, storageOptions.Value, requireHeader: false, out var tenant, out var tenantError))
|
||||
{
|
||||
return tenantError;
|
||||
}
|
||||
|
||||
if (attestationStore is null)
|
||||
{
|
||||
return Results.Problem(
|
||||
detail: "Attestation store is not configured.",
|
||||
statusCode: StatusCodes.Status503ServiceUnavailable,
|
||||
title: "Service unavailable");
|
||||
}
|
||||
|
||||
var parsedSince = ParseTimestamp(since);
|
||||
var parsedUntil = ParseTimestamp(until);
|
||||
|
||||
var query = new VexAttestationQuery(
|
||||
tenant!,
|
||||
parsedSince,
|
||||
parsedUntil,
|
||||
limit ?? 100,
|
||||
offset ?? 0);
|
||||
|
||||
var result = await attestationStore.ListAsync(query, cancellationToken).ConfigureAwait(false);
|
||||
|
||||
var items = result.Items
|
||||
.Select(a => new AttestationListItemDto(
|
||||
a.AttestationId,
|
||||
a.ManifestId,
|
||||
a.MerkleRoot,
|
||||
a.ItemCount,
|
||||
a.AttestedAt))
|
||||
.ToList();
|
||||
|
||||
var response = new AttestationListResponse(
|
||||
items,
|
||||
result.TotalCount,
|
||||
result.HasMore);
|
||||
|
||||
return Results.Ok(response);
|
||||
})
|
||||
.WithName("ListVexAttestations")
|
||||
.WithDescription("Lists DSSE VEX attestations for the tenant with optional time-range filters. Returns a paginated set of attestation summaries including manifest ID, Merkle root, and item counts.")
|
||||
.RequireAuthorization(ExcititorPolicies.VexRead)
|
||||
.RequireTenant();
|
||||
|
||||
// GET /attestations/vex/{attestationId}
|
||||
app.MapGet("/attestations/vex/{attestationId}", async (
|
||||
HttpContext context,
|
||||
string attestationId,
|
||||
IOptions<VexStorageOptions> storageOptions,
|
||||
[FromServices] IVexAttestationStore? attestationStore,
|
||||
CancellationToken cancellationToken) =>
|
||||
{
|
||||
var scopeResult = ScopeAuthorization.RequireScope(context, "vex.read");
|
||||
if (scopeResult is not null)
|
||||
{
|
||||
return scopeResult;
|
||||
}
|
||||
|
||||
if (!TryResolveTenant(context, storageOptions.Value, requireHeader: false, out var tenant, out var tenantError))
|
||||
{
|
||||
return tenantError;
|
||||
}
|
||||
|
||||
if (string.IsNullOrWhiteSpace(attestationId))
|
||||
{
|
||||
return Results.Problem(
|
||||
detail: "attestationId is required.",
|
||||
statusCode: StatusCodes.Status400BadRequest,
|
||||
title: "Validation error");
|
||||
}
|
||||
|
||||
if (attestationStore is null)
|
||||
{
|
||||
return Results.Problem(
|
||||
detail: "Attestation store is not configured.",
|
||||
statusCode: StatusCodes.Status503ServiceUnavailable,
|
||||
title: "Service unavailable");
|
||||
}
|
||||
|
||||
var attestation = await attestationStore.FindByIdAsync(tenant!, attestationId, cancellationToken).ConfigureAwait(false);
|
||||
if (attestation is null)
|
||||
{
|
||||
return Results.NotFound(new
|
||||
{
|
||||
error = new { code = "ERR_ATTESTATION_NOT_FOUND", message = $"Attestation '{attestationId}' not found" }
|
||||
});
|
||||
}
|
||||
|
||||
var response = new AttestationDetailResponse(
|
||||
attestation.AttestationId,
|
||||
attestation.Tenant,
|
||||
attestation.ManifestId,
|
||||
attestation.MerkleRoot,
|
||||
attestation.DsseEnvelopeJson,
|
||||
attestation.DsseEnvelopeHash,
|
||||
attestation.ItemCount,
|
||||
attestation.AttestedAt,
|
||||
attestation.Metadata);
|
||||
|
||||
return Results.Ok(response);
|
||||
})
|
||||
.WithName("GetVexAttestation")
|
||||
.WithDescription("Retrieves the full DSSE attestation envelope for a specific attestation ID, including the Merkle root, DSSE envelope JSON, and envelope hash for offline verification.")
|
||||
.RequireAuthorization(ExcititorPolicies.VexRead)
|
||||
.RequireTenant();
|
||||
}
|
||||
|
||||
private static DateTimeOffset? ParseTimestamp(string? value)
|
||||
{
|
||||
if (string.IsNullOrWhiteSpace(value))
|
||||
{
|
||||
return null;
|
||||
}
|
||||
|
||||
return DateTimeOffset.TryParse(value, out var parsed) ? parsed : null;
|
||||
}
|
||||
}
|
||||
|
||||
// Response DTOs
|
||||
public sealed record AttestationListItemDto(
|
||||
[property: JsonPropertyName("attestationId")] string AttestationId,
|
||||
[property: JsonPropertyName("manifestId")] string ManifestId,
|
||||
[property: JsonPropertyName("merkleRoot")] string MerkleRoot,
|
||||
[property: JsonPropertyName("itemCount")] int ItemCount,
|
||||
[property: JsonPropertyName("attestedAt")] DateTimeOffset AttestedAt);
|
||||
|
||||
public sealed record AttestationListResponse(
|
||||
[property: JsonPropertyName("items")] IReadOnlyList<AttestationListItemDto> Items,
|
||||
[property: JsonPropertyName("totalCount")] int TotalCount,
|
||||
[property: JsonPropertyName("hasMore")] bool HasMore);
|
||||
|
||||
public sealed record AttestationDetailResponse(
|
||||
[property: JsonPropertyName("attestationId")] string AttestationId,
|
||||
[property: JsonPropertyName("tenant")] string Tenant,
|
||||
[property: JsonPropertyName("manifestId")] string ManifestId,
|
||||
[property: JsonPropertyName("merkleRoot")] string MerkleRoot,
|
||||
[property: JsonPropertyName("dsseEnvelopeJson")] string DsseEnvelopeJson,
|
||||
[property: JsonPropertyName("dsseEnvelopeHash")] string DsseEnvelopeHash,
|
||||
[property: JsonPropertyName("itemCount")] int ItemCount,
|
||||
[property: JsonPropertyName("attestedAt")] DateTimeOffset AttestedAt,
|
||||
[property: JsonPropertyName("metadata")] IReadOnlyDictionary<string, string> Metadata);
|
||||
@@ -0,0 +1,408 @@
|
||||
|
||||
using Microsoft.AspNetCore.Builder;
|
||||
using Microsoft.AspNetCore.Http;
|
||||
using Microsoft.AspNetCore.Mvc;
|
||||
using Microsoft.Extensions.Options;
|
||||
using static Program;
|
||||
using StellaOps.Auth.ServerIntegration.Tenancy;
|
||||
using StellaOps.Excititor.Core;
|
||||
using StellaOps.Excititor.Core.Evidence;
|
||||
using StellaOps.Excititor.Core.Storage;
|
||||
using StellaOps.Excititor.WebService.Contracts;
|
||||
using StellaOps.Excititor.WebService.Options;
|
||||
using StellaOps.Excititor.WebService.Security;
|
||||
using StellaOps.Excititor.WebService.Services;
|
||||
using StellaOps.Excititor.WebService.Telemetry;
|
||||
using System.Collections.Immutable;
|
||||
using System.IO;
|
||||
using System.Security.Cryptography;
|
||||
using System.Text;
|
||||
using static StellaOps.Localization.T;
|
||||
|
||||
namespace StellaOps.Excititor.WebService.Endpoints;
|
||||
|
||||
/// <summary>
|
||||
/// Evidence API endpoints (manifest + DSSE attestation + evidence chunks).
|
||||
/// </summary>
|
||||
public static class EvidenceEndpoints
|
||||
{
|
||||
public static void MapEvidenceEndpoints(this WebApplication app)
|
||||
{
|
||||
// GET /evidence/vex/locker/{bundleId}
|
||||
app.MapGet("/evidence/vex/locker/{bundleId}", async (
|
||||
HttpContext context,
|
||||
string bundleId,
|
||||
[FromServices] IOptions<AirgapOptions> airgapOptions,
|
||||
[FromServices] IOptions<VexStorageOptions> storageOptions,
|
||||
[FromServices] IAirgapImportStore importStore,
|
||||
CancellationToken cancellationToken) =>
|
||||
{
|
||||
var scopeResult = ScopeAuthorization.RequireScope(context, "vex.read");
|
||||
if (scopeResult is not null)
|
||||
{
|
||||
return scopeResult;
|
||||
}
|
||||
|
||||
if (!TryResolveTenant(context, storageOptions.Value, requireHeader: false, out var tenant, out var tenantError))
|
||||
{
|
||||
return tenantError;
|
||||
}
|
||||
|
||||
var record = await importStore.FindByBundleIdAsync(tenant!, bundleId, null, cancellationToken).ConfigureAwait(false);
|
||||
if (record is null)
|
||||
{
|
||||
return Results.NotFound();
|
||||
}
|
||||
|
||||
if (string.IsNullOrWhiteSpace(airgapOptions.Value.LockerRootPath))
|
||||
{
|
||||
return Results.StatusCode(StatusCodes.Status503ServiceUnavailable);
|
||||
}
|
||||
|
||||
var manifestPath = Path.Combine(airgapOptions.Value.LockerRootPath!, record.PortableManifestPath ?? string.Empty);
|
||||
if (!File.Exists(manifestPath))
|
||||
{
|
||||
return Results.NotFound();
|
||||
}
|
||||
|
||||
var manifestHash = ComputeSha256(manifestPath, out var manifestSize);
|
||||
string evidenceHash = "sha256:" + Convert.ToHexString(SHA256.HashData(Array.Empty<byte>())).ToLowerInvariant();
|
||||
long? evidenceSize = 0;
|
||||
|
||||
if (!string.IsNullOrWhiteSpace(record.EvidenceLockerPath))
|
||||
{
|
||||
var evidencePath = Path.Combine(airgapOptions.Value.LockerRootPath!, record.EvidenceLockerPath);
|
||||
if (File.Exists(evidencePath))
|
||||
{
|
||||
evidenceHash = ComputeSha256(evidencePath, out var size);
|
||||
evidenceSize = size;
|
||||
}
|
||||
}
|
||||
|
||||
var timeline = record.Timeline
|
||||
.Select(t => new VexEvidenceLockerTimelineEntry(t.EventType, t.CreatedAt, t.ErrorCode, t.Message, t.StalenessSeconds))
|
||||
.ToList();
|
||||
|
||||
var response = new VexEvidenceLockerResponse(
|
||||
record.BundleId,
|
||||
record.MirrorGeneration,
|
||||
record.TenantId,
|
||||
record.Publisher,
|
||||
record.PayloadHash,
|
||||
record.PortableManifestPath ?? string.Empty,
|
||||
manifestHash,
|
||||
record.EvidenceLockerPath ?? string.Empty,
|
||||
evidenceHash,
|
||||
manifestSize,
|
||||
evidenceSize,
|
||||
record.ImportedAt,
|
||||
null,
|
||||
record.TransparencyLog,
|
||||
timeline);
|
||||
|
||||
return Results.Ok(response);
|
||||
})
|
||||
.WithName("GetEvidenceLocker")
|
||||
.WithDescription("Returns the VEX evidence locker record for a specific bundle ID, including manifest path, hashes, transparency log reference, and import timeline events.")
|
||||
.RequireAuthorization(ExcititorPolicies.VexRead)
|
||||
.RequireTenant();
|
||||
|
||||
// GET /evidence/vex/locker/{bundleId}/manifest/file
|
||||
app.MapGet("/evidence/vex/locker/{bundleId}/manifest/file", async (
|
||||
HttpContext context,
|
||||
string bundleId,
|
||||
[FromServices] IOptions<AirgapOptions> airgapOptions,
|
||||
[FromServices] IOptions<VexStorageOptions> storageOptions,
|
||||
[FromServices] IAirgapImportStore importStore,
|
||||
CancellationToken cancellationToken) =>
|
||||
{
|
||||
var scopeResult = ScopeAuthorization.RequireScope(context, "vex.read");
|
||||
if (scopeResult is not null)
|
||||
{
|
||||
return scopeResult;
|
||||
}
|
||||
|
||||
if (!TryResolveTenant(context, storageOptions.Value, requireHeader: false, out var tenant, out var tenantError))
|
||||
{
|
||||
return tenantError;
|
||||
}
|
||||
|
||||
var record = await importStore.FindByBundleIdAsync(tenant!, bundleId, null, cancellationToken).ConfigureAwait(false);
|
||||
if (record is null || string.IsNullOrWhiteSpace(record.PortableManifestPath))
|
||||
{
|
||||
return Results.NotFound();
|
||||
}
|
||||
|
||||
if (string.IsNullOrWhiteSpace(airgapOptions.Value.LockerRootPath))
|
||||
{
|
||||
return Results.StatusCode(StatusCodes.Status503ServiceUnavailable);
|
||||
}
|
||||
|
||||
var manifestPath = Path.Combine(airgapOptions.Value.LockerRootPath!, record.PortableManifestPath);
|
||||
if (!File.Exists(manifestPath))
|
||||
{
|
||||
return Results.NotFound();
|
||||
}
|
||||
|
||||
var etag = ComputeSha256(manifestPath, out _);
|
||||
context.Response.Headers.ETag = $"\"{etag}\"";
|
||||
return Results.File(manifestPath, "application/json");
|
||||
})
|
||||
.WithName("GetEvidenceLockerManifestFile")
|
||||
.WithDescription("Downloads the portable manifest JSON file for a specific VEX bundle from the evidence locker. Suitable for air-gap transfer and offline verification. Returns the raw file with an ETag header.")
|
||||
.RequireAuthorization(ExcititorPolicies.VexRead)
|
||||
.RequireTenant();
|
||||
|
||||
// GET /evidence/vex/list
|
||||
app.MapGet("/evidence/vex/list", async (
|
||||
HttpContext context,
|
||||
[FromQuery(Name = "vulnerabilityId")] string[] vulnerabilityIds,
|
||||
[FromQuery(Name = "productKey")] string[] productKeys,
|
||||
[FromQuery] string? since,
|
||||
[FromQuery] int? limit,
|
||||
IVexClaimStore claimStore,
|
||||
IVexEvidenceLockerService lockerService,
|
||||
IVexEvidenceAttestor attestor,
|
||||
IOptions<VexStorageOptions> storageOptions,
|
||||
ChunkTelemetry chunkTelemetry,
|
||||
TimeProvider timeProvider,
|
||||
CancellationToken cancellationToken) =>
|
||||
{
|
||||
var scopeResult = ScopeAuthorization.RequireScope(context, "vex.read");
|
||||
if (scopeResult is not null)
|
||||
{
|
||||
return scopeResult;
|
||||
}
|
||||
|
||||
if (!TryResolveTenant(context, storageOptions.Value, requireHeader: false, out var tenant, out var tenantError))
|
||||
{
|
||||
return tenantError;
|
||||
}
|
||||
|
||||
var parsedSince = ParseSinceTimestamp(new Microsoft.Extensions.Primitives.StringValues(since));
|
||||
var max = Math.Clamp(limit ?? 500, 1, 1000);
|
||||
|
||||
var pairs = NormalizeValues(vulnerabilityIds).SelectMany(v =>
|
||||
NormalizeValues(productKeys).Select(p => (Vuln: v, Product: p))).ToList();
|
||||
|
||||
if (pairs.Count == 0)
|
||||
{
|
||||
return Results.BadRequest(_t("excititor.validation.vuln_and_product_required"));
|
||||
}
|
||||
|
||||
var claims = new List<VexClaim>();
|
||||
foreach (var pair in pairs)
|
||||
{
|
||||
var found = await claimStore.FindAsync(pair.Vuln, pair.Product, parsedSince, cancellationToken).ConfigureAwait(false);
|
||||
claims.AddRange(found);
|
||||
}
|
||||
|
||||
claims = claims
|
||||
.OrderBy(c => c.VulnerabilityId, StringComparer.OrdinalIgnoreCase)
|
||||
.ThenBy(c => c.Product.Key, StringComparer.OrdinalIgnoreCase)
|
||||
.ThenByDescending(c => c.LastSeen)
|
||||
.Take(max)
|
||||
.ToList();
|
||||
|
||||
if (claims.Count == 0)
|
||||
{
|
||||
return Results.NotFound(_t("excititor.validation.no_claims_available"));
|
||||
}
|
||||
|
||||
var items = claims.Select(claim =>
|
||||
new VexEvidenceSnapshotItem(
|
||||
observationId: FormattableString.Invariant($"{claim.ProviderId}:{claim.Document.Digest}"),
|
||||
providerId: claim.ProviderId,
|
||||
contentHash: claim.Document.Digest,
|
||||
linksetId: FormattableString.Invariant($"{claim.VulnerabilityId}:{claim.Product.Key}"),
|
||||
dsseEnvelopeHash: null,
|
||||
provenance: new VexEvidenceProvenance("ingest")))
|
||||
.ToList();
|
||||
|
||||
var now = timeProvider.GetUtcNow();
|
||||
var manifest = lockerService.BuildManifest(tenant, items, timestamp: now, sequence: 1, isSealed: false);
|
||||
var attestation = await attestor.AttestManifestAsync(manifest, cancellationToken).ConfigureAwait(false);
|
||||
|
||||
chunkTelemetry.RecordIngested(tenant, null, "available", "locker-manifest", claims.Count, 0, 0);
|
||||
var response = new EvidenceManifestResponse(
|
||||
attestation.SignedManifest,
|
||||
attestation.AttestationId,
|
||||
attestation.DsseEnvelopeJson,
|
||||
attestation.DsseEnvelopeHash,
|
||||
attestation.SignedManifest.Items.Length,
|
||||
attestation.AttestedAt);
|
||||
|
||||
return Results.Ok(response);
|
||||
})
|
||||
.WithName("ListVexEvidence")
|
||||
.WithDescription("Builds a signed evidence manifest for the supplied vulnerability and product key pairs. Queries VEX claims, assembles a Merkle-attested manifest, and returns the DSSE attestation envelope for downstream policy evaluation.")
|
||||
.RequireAuthorization(ExcititorPolicies.VexRead)
|
||||
.RequireTenant();
|
||||
|
||||
// GET /evidence/vex/{bundleId}
|
||||
app.MapGet("/evidence/vex/{bundleId}", async (
|
||||
HttpContext context,
|
||||
string bundleId,
|
||||
[FromQuery(Name = "vulnerabilityId")] string[] vulnerabilityIds,
|
||||
[FromQuery(Name = "productKey")] string[] productKeys,
|
||||
[FromQuery] string? since,
|
||||
[FromQuery] int? limit,
|
||||
IVexClaimStore claimStore,
|
||||
IVexEvidenceLockerService lockerService,
|
||||
IVexEvidenceAttestor attestor,
|
||||
IOptions<VexStorageOptions> storageOptions,
|
||||
TimeProvider timeProvider,
|
||||
CancellationToken cancellationToken) =>
|
||||
{
|
||||
var scopeResult = ScopeAuthorization.RequireScope(context, "vex.read");
|
||||
if (scopeResult is not null)
|
||||
{
|
||||
return scopeResult;
|
||||
}
|
||||
|
||||
if (!TryResolveTenant(context, storageOptions.Value, requireHeader: false, out var tenant, out var tenantError))
|
||||
{
|
||||
return tenantError;
|
||||
}
|
||||
|
||||
if (string.IsNullOrWhiteSpace(bundleId))
|
||||
{
|
||||
return Results.Problem(
|
||||
detail: "bundleId is required.",
|
||||
statusCode: StatusCodes.Status400BadRequest,
|
||||
title: "Validation error");
|
||||
}
|
||||
|
||||
var parsedSince = ParseSinceTimestamp(new Microsoft.Extensions.Primitives.StringValues(since));
|
||||
var max = Math.Clamp(limit ?? 500, 1, 1000);
|
||||
var pairs = NormalizeValues(vulnerabilityIds).SelectMany(v =>
|
||||
NormalizeValues(productKeys).Select(p => (Vuln: v, Product: p))).ToList();
|
||||
|
||||
if (pairs.Count == 0)
|
||||
{
|
||||
return Results.BadRequest(_t("excititor.validation.vuln_and_product_required"));
|
||||
}
|
||||
|
||||
var claims = new List<VexClaim>();
|
||||
foreach (var pair in pairs)
|
||||
{
|
||||
var found = await claimStore.FindAsync(pair.Vuln, pair.Product, parsedSince, cancellationToken).ConfigureAwait(false);
|
||||
claims.AddRange(found);
|
||||
}
|
||||
|
||||
claims = claims
|
||||
.OrderBy(c => c.VulnerabilityId, StringComparer.OrdinalIgnoreCase)
|
||||
.ThenBy(c => c.Product.Key, StringComparer.OrdinalIgnoreCase)
|
||||
.ThenByDescending(c => c.LastSeen)
|
||||
.Take(max)
|
||||
.ToList();
|
||||
|
||||
if (claims.Count == 0)
|
||||
{
|
||||
return Results.NotFound(_t("excititor.validation.no_claims_available"));
|
||||
}
|
||||
|
||||
var items = claims.Select(claim =>
|
||||
new VexEvidenceSnapshotItem(
|
||||
observationId: FormattableString.Invariant($"{claim.ProviderId}:{claim.Document.Digest}"),
|
||||
providerId: claim.ProviderId,
|
||||
contentHash: claim.Document.Digest,
|
||||
linksetId: FormattableString.Invariant($"{claim.VulnerabilityId}:{claim.Product.Key}"),
|
||||
dsseEnvelopeHash: null,
|
||||
provenance: new VexEvidenceProvenance("ingest")))
|
||||
.ToList();
|
||||
|
||||
var now = timeProvider.GetUtcNow();
|
||||
var manifest = lockerService.BuildManifest(tenant, items, timestamp: now, sequence: 1, isSealed: false);
|
||||
if (!string.Equals(manifest.ManifestId, bundleId, StringComparison.OrdinalIgnoreCase))
|
||||
{
|
||||
return Results.NotFound(_t("excititor.error.bundle_not_found", bundleId));
|
||||
}
|
||||
|
||||
var attestation = await attestor.AttestManifestAsync(manifest, cancellationToken).ConfigureAwait(false);
|
||||
var response = new EvidenceManifestResponse(
|
||||
attestation.SignedManifest,
|
||||
attestation.AttestationId,
|
||||
attestation.DsseEnvelopeJson,
|
||||
attestation.DsseEnvelopeHash,
|
||||
attestation.SignedManifest.Items.Length,
|
||||
attestation.AttestedAt);
|
||||
|
||||
return Results.Ok(response);
|
||||
})
|
||||
.WithName("GetVexEvidenceBundle")
|
||||
.WithDescription("Retrieves the signed evidence bundle for a specific bundle ID filtered by vulnerability and product keys. Validates the manifest ID against the requested bundle ID before returning the DSSE attestation envelope.")
|
||||
.RequireAuthorization(ExcititorPolicies.VexRead)
|
||||
.RequireTenant();
|
||||
|
||||
// GET /v1/vex/evidence/chunks
|
||||
app.MapGet("/v1/vex/evidence/chunks", async (
|
||||
HttpContext context,
|
||||
[FromQuery] string vulnerabilityId,
|
||||
[FromQuery] string productKey,
|
||||
[FromQuery(Name = "providerId")] string[] providerIds,
|
||||
[FromQuery] string[] status,
|
||||
[FromQuery] string? since,
|
||||
[FromQuery] int? limit,
|
||||
IOptions<VexStorageOptions> storageOptions,
|
||||
IVexEvidenceChunkService chunkService,
|
||||
ChunkTelemetry chunkTelemetry,
|
||||
CancellationToken cancellationToken) =>
|
||||
{
|
||||
var scopeResult = ScopeAuthorization.RequireScope(context, "vex.read");
|
||||
if (scopeResult is not null)
|
||||
{
|
||||
return scopeResult;
|
||||
}
|
||||
|
||||
if (!TryResolveTenant(context, storageOptions.Value, requireHeader: false, out var tenant, out var tenantError))
|
||||
{
|
||||
return tenantError;
|
||||
}
|
||||
|
||||
if (string.IsNullOrWhiteSpace(vulnerabilityId) || string.IsNullOrWhiteSpace(productKey))
|
||||
{
|
||||
return Results.BadRequest(_t("excititor.validation.vuln_and_product_required_short"));
|
||||
}
|
||||
|
||||
var parsedSince = ParseSinceTimestamp(new Microsoft.Extensions.Primitives.StringValues(since));
|
||||
var providers = providerIds?.Length > 0
|
||||
? providerIds.ToImmutableHashSet(StringComparer.OrdinalIgnoreCase)
|
||||
: ImmutableHashSet<string>.Empty;
|
||||
|
||||
var statuses = status?.Length > 0
|
||||
? status
|
||||
.Select(s => Enum.TryParse<VexClaimStatus>(s, true, out var parsed) ? parsed : (VexClaimStatus?)null)
|
||||
.Where(s => s is not null)
|
||||
.Select(s => s!.Value)
|
||||
.ToImmutableHashSet()
|
||||
: ImmutableHashSet<VexClaimStatus>.Empty;
|
||||
|
||||
var req = new VexEvidenceChunkRequest(
|
||||
tenant,
|
||||
vulnerabilityId,
|
||||
productKey,
|
||||
providers,
|
||||
statuses,
|
||||
parsedSince,
|
||||
Math.Clamp(limit ?? 200, 1, 1000));
|
||||
|
||||
var result = await chunkService.QueryAsync(req, cancellationToken).ConfigureAwait(false);
|
||||
chunkTelemetry.RecordIngested(tenant, null, "available", "locker-chunks", result.TotalCount, 0, 0);
|
||||
|
||||
return Results.Ok(new EvidenceChunkListResponse(result.Chunks, result.TotalCount, result.Truncated, result.GeneratedAtUtc));
|
||||
})
|
||||
.WithName("GetVexEvidenceChunks")
|
||||
.WithDescription("Queries VEX evidence chunks for a specific vulnerability and product key, with optional provider and status filters. Returns raw chunk records suitable for incremental sync or audit replay.")
|
||||
.RequireAuthorization(ExcititorPolicies.VexRead)
|
||||
.RequireTenant();
|
||||
}
|
||||
|
||||
private static string ComputeSha256(string path, out long sizeBytes)
|
||||
{
|
||||
var data = File.ReadAllBytes(path);
|
||||
sizeBytes = data.LongLength;
|
||||
var hash = SHA256.HashData(data);
|
||||
return "sha256:" + Convert.ToHexString(hash).ToLowerInvariant();
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,306 @@
|
||||
|
||||
using Microsoft.AspNetCore.Http;
|
||||
using Microsoft.AspNetCore.Routing;
|
||||
using StellaOps.Auth.ServerIntegration.Tenancy;
|
||||
using StellaOps.Excititor.WebService.Security;
|
||||
using StellaOps.Excititor.WebService.Services;
|
||||
using System.Collections.Immutable;
|
||||
using System.Globalization;
|
||||
|
||||
namespace StellaOps.Excititor.WebService.Endpoints;
|
||||
|
||||
internal static class IngestEndpoints
|
||||
{
|
||||
private const string AdminScope = "vex.admin";
|
||||
|
||||
public static void MapIngestEndpoints(IEndpointRouteBuilder app)
|
||||
{
|
||||
var group = app.MapGroup("/excititor")
|
||||
.RequireTenant();
|
||||
|
||||
group.MapPost("/init", HandleInitAsync)
|
||||
.WithName("ExcititorInit")
|
||||
.WithDescription("Initializes VEX ingest providers for the specified provider IDs, establishing connector state and preparing the pipeline for incremental ingestion. Requires vex.admin scope.")
|
||||
.RequireAuthorization(ExcititorPolicies.VexAdmin);
|
||||
|
||||
group.MapPost("/ingest/run", HandleRunAsync)
|
||||
.WithName("ExcititorIngestRun")
|
||||
.WithDescription("Triggers a full or incremental VEX ingest run across specified providers within the given time window. Returns per-provider run summaries including document counts and checkpoint state. Requires vex.admin scope.")
|
||||
.RequireAuthorization(ExcititorPolicies.VexAdmin);
|
||||
|
||||
group.MapPost("/ingest/resume", HandleResumeAsync)
|
||||
.WithName("ExcititorIngestResume")
|
||||
.WithDescription("Resumes a previously interrupted ingest run from the specified checkpoint, replaying from the last known good position without re-fetching earlier data. Requires vex.admin scope.")
|
||||
.RequireAuthorization(ExcititorPolicies.VexAdmin);
|
||||
|
||||
group.MapPost("/reconcile", HandleReconcileAsync)
|
||||
.WithName("ExcititorReconcile")
|
||||
.WithDescription("Reconciles VEX data across providers by re-evaluating entries older than the specified max-age threshold. Stale or conflicting claims are re-fetched and re-normalized. Requires vex.admin scope.")
|
||||
.RequireAuthorization(ExcititorPolicies.VexAdmin);
|
||||
}
|
||||
|
||||
internal static async Task<IResult> HandleInitAsync(
|
||||
HttpContext httpContext,
|
||||
ExcititorInitRequest request,
|
||||
IVexIngestOrchestrator orchestrator,
|
||||
TimeProvider timeProvider,
|
||||
CancellationToken cancellationToken)
|
||||
{
|
||||
var scopeResult = ScopeAuthorization.RequireScope(httpContext, AdminScope);
|
||||
if (scopeResult is not null)
|
||||
{
|
||||
return scopeResult;
|
||||
}
|
||||
|
||||
var providerIds = NormalizeProviders(request.Providers);
|
||||
_ = timeProvider;
|
||||
var options = new IngestInitOptions(providerIds, request.Resume ?? false);
|
||||
|
||||
var summary = await orchestrator.InitializeAsync(options, cancellationToken).ConfigureAwait(false);
|
||||
var message = $"Initialized {summary.ProviderCount} provider(s); {summary.SuccessCount} succeeded, {summary.FailureCount} failed.";
|
||||
|
||||
return TypedResults.Ok<object>(new
|
||||
{
|
||||
message,
|
||||
runId = summary.RunId,
|
||||
startedAt = summary.StartedAt,
|
||||
completedAt = summary.CompletedAt,
|
||||
providers = summary.Providers.Select(static provider => new
|
||||
{
|
||||
providerId = provider.ProviderId,
|
||||
displayName = provider.DisplayName,
|
||||
status = provider.Status,
|
||||
durationMs = provider.Duration.TotalMilliseconds,
|
||||
error = provider.Error
|
||||
})
|
||||
});
|
||||
}
|
||||
|
||||
internal static async Task<IResult> HandleRunAsync(
|
||||
HttpContext httpContext,
|
||||
ExcititorIngestRunRequest request,
|
||||
IVexIngestOrchestrator orchestrator,
|
||||
TimeProvider timeProvider,
|
||||
CancellationToken cancellationToken)
|
||||
{
|
||||
var scopeResult = ScopeAuthorization.RequireScope(httpContext, AdminScope);
|
||||
if (scopeResult is not null)
|
||||
{
|
||||
return scopeResult;
|
||||
}
|
||||
|
||||
if (!TryParseDateTimeOffset(request.Since, out var since, out var sinceError))
|
||||
{
|
||||
return TypedResults.BadRequest<object>(new { message = sinceError });
|
||||
}
|
||||
|
||||
if (!TryParseTimeSpan(request.Window, out var window, out var windowError))
|
||||
{
|
||||
return TypedResults.BadRequest<object>(new { message = windowError });
|
||||
}
|
||||
|
||||
_ = timeProvider;
|
||||
var providerIds = NormalizeProviders(request.Providers);
|
||||
var options = new IngestRunOptions(
|
||||
providerIds,
|
||||
since,
|
||||
window,
|
||||
request.Force ?? false);
|
||||
|
||||
var summary = await orchestrator.RunAsync(options, cancellationToken).ConfigureAwait(false);
|
||||
var message = $"Ingest run completed for {summary.ProviderCount} provider(s); {summary.SuccessCount} succeeded, {summary.FailureCount} failed.";
|
||||
|
||||
return TypedResults.Ok<object>(new
|
||||
{
|
||||
message,
|
||||
runId = summary.RunId,
|
||||
startedAt = summary.StartedAt,
|
||||
completedAt = summary.CompletedAt,
|
||||
durationMs = summary.Duration.TotalMilliseconds,
|
||||
providers = summary.Providers.Select(static provider => new
|
||||
{
|
||||
providerId = provider.ProviderId,
|
||||
status = provider.Status,
|
||||
documents = provider.Documents,
|
||||
claims = provider.Claims,
|
||||
startedAt = provider.StartedAt,
|
||||
completedAt = provider.CompletedAt,
|
||||
durationMs = provider.Duration.TotalMilliseconds,
|
||||
lastDigest = provider.LastDigest,
|
||||
lastUpdated = provider.LastUpdated,
|
||||
checkpoint = provider.Checkpoint,
|
||||
error = provider.Error
|
||||
})
|
||||
});
|
||||
}
|
||||
|
||||
internal static async Task<IResult> HandleResumeAsync(
|
||||
HttpContext httpContext,
|
||||
ExcititorIngestResumeRequest request,
|
||||
IVexIngestOrchestrator orchestrator,
|
||||
TimeProvider timeProvider,
|
||||
CancellationToken cancellationToken)
|
||||
{
|
||||
var scopeResult = ScopeAuthorization.RequireScope(httpContext, AdminScope);
|
||||
if (scopeResult is not null)
|
||||
{
|
||||
return scopeResult;
|
||||
}
|
||||
|
||||
_ = timeProvider;
|
||||
var providerIds = NormalizeProviders(request.Providers);
|
||||
var options = new IngestResumeOptions(providerIds, request.Checkpoint);
|
||||
|
||||
var summary = await orchestrator.ResumeAsync(options, cancellationToken).ConfigureAwait(false);
|
||||
var message = $"Resume run completed for {summary.ProviderCount} provider(s); {summary.SuccessCount} succeeded, {summary.FailureCount} failed.";
|
||||
|
||||
return TypedResults.Ok<object>(new
|
||||
{
|
||||
message,
|
||||
runId = summary.RunId,
|
||||
startedAt = summary.StartedAt,
|
||||
completedAt = summary.CompletedAt,
|
||||
durationMs = summary.Duration.TotalMilliseconds,
|
||||
providers = summary.Providers.Select(static provider => new
|
||||
{
|
||||
providerId = provider.ProviderId,
|
||||
status = provider.Status,
|
||||
documents = provider.Documents,
|
||||
claims = provider.Claims,
|
||||
startedAt = provider.StartedAt,
|
||||
completedAt = provider.CompletedAt,
|
||||
durationMs = provider.Duration.TotalMilliseconds,
|
||||
since = provider.Since,
|
||||
checkpoint = provider.Checkpoint,
|
||||
error = provider.Error
|
||||
})
|
||||
});
|
||||
}
|
||||
|
||||
internal static async Task<IResult> HandleReconcileAsync(
|
||||
HttpContext httpContext,
|
||||
ExcititorReconcileRequest request,
|
||||
IVexIngestOrchestrator orchestrator,
|
||||
TimeProvider timeProvider,
|
||||
CancellationToken cancellationToken)
|
||||
{
|
||||
var scopeResult = ScopeAuthorization.RequireScope(httpContext, AdminScope);
|
||||
if (scopeResult is not null)
|
||||
{
|
||||
return scopeResult;
|
||||
}
|
||||
|
||||
if (!TryParseTimeSpan(request.MaxAge, out var maxAge, out var error))
|
||||
{
|
||||
return TypedResults.BadRequest<object>(new { message = error });
|
||||
}
|
||||
|
||||
_ = timeProvider;
|
||||
var providerIds = NormalizeProviders(request.Providers);
|
||||
var options = new ReconcileOptions(providerIds, maxAge);
|
||||
|
||||
var summary = await orchestrator.ReconcileAsync(options, cancellationToken).ConfigureAwait(false);
|
||||
var message = $"Reconcile completed for {summary.ProviderCount} provider(s); {summary.ReconciledCount} reconciled, {summary.SkippedCount} skipped, {summary.FailureCount} failed.";
|
||||
|
||||
return TypedResults.Ok<object>(new
|
||||
{
|
||||
message,
|
||||
runId = summary.RunId,
|
||||
startedAt = summary.StartedAt,
|
||||
completedAt = summary.CompletedAt,
|
||||
durationMs = summary.Duration.TotalMilliseconds,
|
||||
providers = summary.Providers.Select(static provider => new
|
||||
{
|
||||
providerId = provider.ProviderId,
|
||||
status = provider.Status,
|
||||
action = provider.Action,
|
||||
lastUpdated = provider.LastUpdated,
|
||||
threshold = provider.Threshold,
|
||||
documents = provider.Documents,
|
||||
claims = provider.Claims,
|
||||
error = provider.Error
|
||||
})
|
||||
});
|
||||
}
|
||||
|
||||
internal static ImmutableArray<string> NormalizeProviders(IReadOnlyCollection<string>? providers)
|
||||
{
|
||||
if (providers is null || providers.Count == 0)
|
||||
{
|
||||
return ImmutableArray<string>.Empty;
|
||||
}
|
||||
|
||||
var set = new SortedSet<string>(StringComparer.OrdinalIgnoreCase);
|
||||
foreach (var provider in providers)
|
||||
{
|
||||
if (string.IsNullOrWhiteSpace(provider))
|
||||
{
|
||||
continue;
|
||||
}
|
||||
|
||||
set.Add(provider.Trim());
|
||||
}
|
||||
|
||||
return set.ToImmutableArray();
|
||||
}
|
||||
|
||||
internal static bool TryParseDateTimeOffset(string? value, out DateTimeOffset? result, out string? error)
|
||||
{
|
||||
result = null;
|
||||
error = null;
|
||||
|
||||
if (string.IsNullOrWhiteSpace(value))
|
||||
{
|
||||
return true;
|
||||
}
|
||||
|
||||
if (DateTimeOffset.TryParse(
|
||||
value.Trim(),
|
||||
CultureInfo.InvariantCulture,
|
||||
DateTimeStyles.AssumeUniversal | DateTimeStyles.AdjustToUniversal,
|
||||
out var parsed))
|
||||
{
|
||||
result = parsed;
|
||||
return true;
|
||||
}
|
||||
|
||||
error = "Invalid 'since' value. Use ISO-8601 format (e.g. 2025-10-19T12:30:00Z).";
|
||||
return false;
|
||||
}
|
||||
|
||||
internal static bool TryParseTimeSpan(string? value, out TimeSpan? result, out string? error)
|
||||
{
|
||||
result = null;
|
||||
error = null;
|
||||
|
||||
if (string.IsNullOrWhiteSpace(value))
|
||||
{
|
||||
return true;
|
||||
}
|
||||
|
||||
if (TimeSpan.TryParse(value.Trim(), CultureInfo.InvariantCulture, out var parsed) && parsed >= TimeSpan.Zero)
|
||||
{
|
||||
result = parsed;
|
||||
return true;
|
||||
}
|
||||
|
||||
error = "Invalid duration value. Use TimeSpan format (e.g. 1.00:00:00).";
|
||||
return false;
|
||||
}
|
||||
|
||||
internal sealed record ExcititorInitRequest(IReadOnlyList<string>? Providers, bool? Resume);
|
||||
|
||||
internal sealed record ExcititorIngestRunRequest(
|
||||
IReadOnlyList<string>? Providers,
|
||||
string? Since,
|
||||
string? Window,
|
||||
bool? Force);
|
||||
|
||||
internal sealed record ExcititorIngestResumeRequest(
|
||||
IReadOnlyList<string>? Providers,
|
||||
string? Checkpoint);
|
||||
|
||||
internal sealed record ExcititorReconcileRequest(
|
||||
IReadOnlyList<string>? Providers,
|
||||
string? MaxAge);
|
||||
}
|
||||
@@ -0,0 +1,401 @@
|
||||
|
||||
using DomainVexProductScope = StellaOps.Excititor.Core.Observations.VexProductScope;
|
||||
using Microsoft.AspNetCore.Builder;
|
||||
using Microsoft.AspNetCore.Http;
|
||||
using Microsoft.AspNetCore.Mvc;
|
||||
using Microsoft.Extensions.Options;
|
||||
using StellaOps.Auth.ServerIntegration.Tenancy;
|
||||
using StellaOps.Excititor.Core.Canonicalization;
|
||||
using StellaOps.Excititor.Core.Observations;
|
||||
using StellaOps.Excititor.Core.Storage;
|
||||
using static StellaOps.Localization.T;
|
||||
using StellaOps.Excititor.WebService.Contracts;
|
||||
using StellaOps.Excititor.WebService.Security;
|
||||
using StellaOps.Excititor.WebService.Services;
|
||||
using StellaOps.Excititor.WebService.Telemetry;
|
||||
using System;
|
||||
using System.Collections.Generic;
|
||||
using System.Collections.Immutable;
|
||||
using System.Diagnostics;
|
||||
using System.Linq;
|
||||
using System.Text.Json.Serialization;
|
||||
|
||||
namespace StellaOps.Excititor.WebService.Endpoints;
|
||||
|
||||
/// <summary>
|
||||
/// Linkset API endpoints (EXCITITOR-LNM-21-202).
|
||||
/// Exposes /vex/linksets/* endpoints that surface alias mappings, conflict markers,
|
||||
/// and provenance proofs exactly as stored. Errors map to ERR_AGG_* codes.
|
||||
/// </summary>
|
||||
public static class LinksetEndpoints
|
||||
{
|
||||
public static void MapLinksetEndpoints(this WebApplication app)
|
||||
{
|
||||
var group = app.MapGroup("/vex/linksets")
|
||||
.RequireAuthorization(ExcititorPolicies.VexRead)
|
||||
.RequireTenant();
|
||||
|
||||
// GET /vex/linksets - List linksets with filters
|
||||
group.MapGet("", async (
|
||||
HttpContext context,
|
||||
IOptions<VexStorageOptions> storageOptions,
|
||||
[FromServices] IVexLinksetStore linksetStore,
|
||||
[FromQuery] int? limit,
|
||||
[FromQuery] string? cursor,
|
||||
[FromQuery] string? vulnerabilityId,
|
||||
[FromQuery] string? productKey,
|
||||
[FromQuery] string? providerId,
|
||||
[FromQuery] bool? hasConflicts,
|
||||
CancellationToken cancellationToken) =>
|
||||
{
|
||||
var scopeResult = ScopeAuthorization.RequireScope(context, "vex.read");
|
||||
if (scopeResult is not null)
|
||||
{
|
||||
return scopeResult;
|
||||
}
|
||||
|
||||
if (!TryResolveTenant(context, storageOptions.Value, out var tenant, out var tenantError))
|
||||
{
|
||||
return tenantError;
|
||||
}
|
||||
|
||||
var take = Math.Clamp(limit.GetValueOrDefault(50), 1, 100);
|
||||
|
||||
IReadOnlyList<VexLinkset> linksets;
|
||||
|
||||
// Route to appropriate query method based on filters
|
||||
if (hasConflicts == true)
|
||||
{
|
||||
linksets = await linksetStore
|
||||
.FindWithConflictsAsync(tenant, take, cancellationToken)
|
||||
.ConfigureAwait(false);
|
||||
}
|
||||
else if (!string.IsNullOrWhiteSpace(vulnerabilityId))
|
||||
{
|
||||
linksets = await linksetStore
|
||||
.FindByVulnerabilityAsync(tenant, vulnerabilityId.Trim(), take, cancellationToken)
|
||||
.ConfigureAwait(false);
|
||||
}
|
||||
else if (!string.IsNullOrWhiteSpace(productKey))
|
||||
{
|
||||
linksets = await linksetStore
|
||||
.FindByProductKeyAsync(tenant, productKey.Trim(), take, cancellationToken)
|
||||
.ConfigureAwait(false);
|
||||
}
|
||||
else if (!string.IsNullOrWhiteSpace(providerId))
|
||||
{
|
||||
linksets = await linksetStore
|
||||
.FindByProviderAsync(tenant, providerId.Trim(), take, cancellationToken)
|
||||
.ConfigureAwait(false);
|
||||
}
|
||||
else
|
||||
{
|
||||
return Results.BadRequest(new
|
||||
{
|
||||
error = new
|
||||
{
|
||||
code = "ERR_AGG_PARAMS",
|
||||
message = _t("excititor.validation.linkset_filter_required")
|
||||
}
|
||||
});
|
||||
}
|
||||
|
||||
var items = linksets
|
||||
.Take(take)
|
||||
.Select(ToListItem)
|
||||
.ToList();
|
||||
|
||||
// Record conflict metrics (EXCITITOR-OBS-51-001)
|
||||
foreach (var linkset in linksets.Take(take))
|
||||
{
|
||||
if (linkset.HasConflicts)
|
||||
{
|
||||
LinksetTelemetry.RecordLinksetDisagreements(tenant, linkset);
|
||||
}
|
||||
}
|
||||
|
||||
var hasMore = linksets.Count > take;
|
||||
string? nextCursor = null;
|
||||
if (hasMore && items.Count > 0)
|
||||
{
|
||||
var last = linksets[items.Count - 1];
|
||||
nextCursor = EncodeCursor(last.UpdatedAt.UtcDateTime, last.LinksetId);
|
||||
}
|
||||
|
||||
var response = new VexLinksetListResponse(items, nextCursor);
|
||||
return Results.Ok(response);
|
||||
})
|
||||
.WithName("ListVexLinksets")
|
||||
.WithDescription("Lists VEX linksets for the tenant with optional filters for vulnerability ID, product key, provider, or conflict status. Linksets aggregate provider observations into a canonical vulnerability-product mapping with disagreement tracking.");
|
||||
|
||||
// GET /vex/linksets/{linksetId} - Get linkset by ID
|
||||
group.MapGet("/{linksetId}", async (
|
||||
HttpContext context,
|
||||
string linksetId,
|
||||
IOptions<VexStorageOptions> storageOptions,
|
||||
[FromServices] IVexLinksetStore linksetStore,
|
||||
CancellationToken cancellationToken) =>
|
||||
{
|
||||
var scopeResult = ScopeAuthorization.RequireScope(context, "vex.read");
|
||||
if (scopeResult is not null)
|
||||
{
|
||||
return scopeResult;
|
||||
}
|
||||
|
||||
if (!TryResolveTenant(context, storageOptions.Value, out var tenant, out var tenantError))
|
||||
{
|
||||
return tenantError;
|
||||
}
|
||||
|
||||
if (string.IsNullOrWhiteSpace(linksetId))
|
||||
{
|
||||
return Results.BadRequest(new
|
||||
{
|
||||
error = new { code = "ERR_AGG_PARAMS", message = "linksetId is required" }
|
||||
});
|
||||
}
|
||||
|
||||
var linkset = await linksetStore
|
||||
.GetByIdAsync(tenant, linksetId.Trim(), cancellationToken)
|
||||
.ConfigureAwait(false);
|
||||
|
||||
if (linkset is null)
|
||||
{
|
||||
return Results.NotFound(new
|
||||
{
|
||||
error = new { code = "ERR_AGG_NOT_FOUND", message = $"Linkset '{linksetId}' not found" }
|
||||
});
|
||||
}
|
||||
|
||||
var response = ToDetailResponse(linkset);
|
||||
return Results.Ok(response);
|
||||
})
|
||||
.WithName("GetVexLinkset")
|
||||
.WithDescription("Retrieves the full linkset record for a specific linkset ID, including all provider observations, disagreements, confidence level, and scope details.");
|
||||
|
||||
// GET /vex/linksets/lookup - Lookup linkset by vulnerability and product
|
||||
group.MapGet("/lookup", async (
|
||||
HttpContext context,
|
||||
IOptions<VexStorageOptions> storageOptions,
|
||||
[FromServices] IVexLinksetStore linksetStore,
|
||||
[FromQuery] string? vulnerabilityId,
|
||||
[FromQuery] string? productKey,
|
||||
CancellationToken cancellationToken) =>
|
||||
{
|
||||
var scopeResult = ScopeAuthorization.RequireScope(context, "vex.read");
|
||||
if (scopeResult is not null)
|
||||
{
|
||||
return scopeResult;
|
||||
}
|
||||
|
||||
if (!TryResolveTenant(context, storageOptions.Value, out var tenant, out var tenantError))
|
||||
{
|
||||
return tenantError;
|
||||
}
|
||||
|
||||
if (string.IsNullOrWhiteSpace(vulnerabilityId) || string.IsNullOrWhiteSpace(productKey))
|
||||
{
|
||||
return Results.BadRequest(new
|
||||
{
|
||||
error = new { code = "ERR_AGG_PARAMS", message = "vulnerabilityId and productKey are required" }
|
||||
});
|
||||
}
|
||||
|
||||
var linksetId = VexLinkset.CreateLinksetId(tenant, vulnerabilityId.Trim(), productKey.Trim());
|
||||
var linkset = await linksetStore
|
||||
.GetByIdAsync(tenant, linksetId, cancellationToken)
|
||||
.ConfigureAwait(false);
|
||||
|
||||
if (linkset is null)
|
||||
{
|
||||
return Results.NotFound(new
|
||||
{
|
||||
error = new { code = "ERR_AGG_NOT_FOUND", message = "No linkset found for the specified vulnerability and product" }
|
||||
});
|
||||
}
|
||||
|
||||
var response = ToDetailResponse(linkset);
|
||||
return Results.Ok(response);
|
||||
})
|
||||
.WithName("LookupVexLinkset")
|
||||
.WithDescription("Performs a deterministic lookup of the linkset for a specific vulnerability and product key pair. Returns the same detail response as GetVexLinkset but accepts human-readable IDs rather than the internal linkset ID.");
|
||||
|
||||
// GET /vex/linksets/count - Get linkset counts for tenant
|
||||
group.MapGet("/count", async (
|
||||
HttpContext context,
|
||||
IOptions<VexStorageOptions> storageOptions,
|
||||
[FromServices] IVexLinksetStore linksetStore,
|
||||
CancellationToken cancellationToken) =>
|
||||
{
|
||||
var scopeResult = ScopeAuthorization.RequireScope(context, "vex.read");
|
||||
if (scopeResult is not null)
|
||||
{
|
||||
return scopeResult;
|
||||
}
|
||||
|
||||
if (!TryResolveTenant(context, storageOptions.Value, out var tenant, out var tenantError))
|
||||
{
|
||||
return tenantError;
|
||||
}
|
||||
|
||||
var total = await linksetStore
|
||||
.CountAsync(tenant, cancellationToken)
|
||||
.ConfigureAwait(false);
|
||||
|
||||
var withConflicts = await linksetStore
|
||||
.CountWithConflictsAsync(tenant, cancellationToken)
|
||||
.ConfigureAwait(false);
|
||||
|
||||
return Results.Ok(new LinksetCountResponse(total, withConflicts));
|
||||
})
|
||||
.WithName("CountVexLinksets")
|
||||
.WithDescription("Returns total linkset count and the count of linksets with provider disagreements for the current tenant. Useful for dashboard monitoring and conflict alerting.");
|
||||
|
||||
// GET /vex/linksets/conflicts - List linksets with conflicts (shorthand)
|
||||
group.MapGet("/conflicts", async (
|
||||
HttpContext context,
|
||||
IOptions<VexStorageOptions> storageOptions,
|
||||
[FromServices] IVexLinksetStore linksetStore,
|
||||
[FromQuery] int? limit,
|
||||
CancellationToken cancellationToken) =>
|
||||
{
|
||||
var scopeResult = ScopeAuthorization.RequireScope(context, "vex.read");
|
||||
if (scopeResult is not null)
|
||||
{
|
||||
return scopeResult;
|
||||
}
|
||||
|
||||
if (!TryResolveTenant(context, storageOptions.Value, out var tenant, out var tenantError))
|
||||
{
|
||||
return tenantError;
|
||||
}
|
||||
|
||||
var take = Math.Clamp(limit.GetValueOrDefault(50), 1, 100);
|
||||
|
||||
var linksets = await linksetStore
|
||||
.FindWithConflictsAsync(tenant, take, cancellationToken)
|
||||
.ConfigureAwait(false);
|
||||
|
||||
var items = linksets.Select(ToListItem).ToList();
|
||||
var response = new VexLinksetListResponse(items, null);
|
||||
return Results.Ok(response);
|
||||
})
|
||||
.WithName("ListVexLinksetConflicts")
|
||||
.WithDescription("Lists linksets that have active provider disagreements, where two or more ingest sources report different VEX status verdicts for the same vulnerability-product pair. Intended for triage workflows.");
|
||||
}
|
||||
|
||||
private static VexLinksetListItem ToListItem(VexLinkset linkset)
|
||||
{
|
||||
var scope = BuildScope(linkset.Scope);
|
||||
return new VexLinksetListItem(
|
||||
LinksetId: linkset.LinksetId,
|
||||
Tenant: linkset.Tenant,
|
||||
VulnerabilityId: linkset.VulnerabilityId,
|
||||
ProductKey: linkset.ProductKey,
|
||||
Scope: scope,
|
||||
ProviderIds: linkset.ProviderIds.ToList(),
|
||||
Statuses: linkset.Statuses.ToList(),
|
||||
Aliases: Array.Empty<string>(), // Aliases are in observations, not linksets
|
||||
Purls: Array.Empty<string>(),
|
||||
Cpes: Array.Empty<string>(),
|
||||
References: Array.Empty<VexLinksetReference>(),
|
||||
Disagreements: linkset.Disagreements
|
||||
.Select(d => new VexLinksetDisagreement(d.ProviderId, d.Status, d.Justification, d.Confidence))
|
||||
.ToList(),
|
||||
Observations: linkset.Observations
|
||||
.Select(o => new VexLinksetObservationRef(o.ObservationId, o.ProviderId, o.Status, o.Confidence))
|
||||
.ToList(),
|
||||
CreatedAt: linkset.CreatedAt);
|
||||
}
|
||||
|
||||
private static VexLinksetDetailResponse ToDetailResponse(VexLinkset linkset)
|
||||
{
|
||||
var scope = BuildScope(linkset.Scope);
|
||||
return new VexLinksetDetailResponse(
|
||||
LinksetId: linkset.LinksetId,
|
||||
Tenant: linkset.Tenant,
|
||||
VulnerabilityId: linkset.VulnerabilityId,
|
||||
ProductKey: linkset.ProductKey,
|
||||
Scope: scope,
|
||||
ProviderIds: linkset.ProviderIds.ToList(),
|
||||
Statuses: linkset.Statuses.ToList(),
|
||||
Confidence: linkset.Confidence.ToString().ToLowerInvariant(),
|
||||
HasConflicts: linkset.HasConflicts,
|
||||
Disagreements: linkset.Disagreements
|
||||
.Select(d => new VexLinksetDisagreement(d.ProviderId, d.Status, d.Justification, d.Confidence))
|
||||
.ToList(),
|
||||
Observations: linkset.Observations
|
||||
.Select(o => new VexLinksetObservationRef(o.ObservationId, o.ProviderId, o.Status, o.Confidence))
|
||||
.ToList(),
|
||||
CreatedAt: linkset.CreatedAt,
|
||||
UpdatedAt: linkset.UpdatedAt);
|
||||
}
|
||||
|
||||
private static bool TryResolveTenant(
|
||||
HttpContext context,
|
||||
VexStorageOptions options,
|
||||
out string tenant,
|
||||
out IResult? problem)
|
||||
{
|
||||
problem = null;
|
||||
tenant = string.Empty;
|
||||
|
||||
var headerTenant = context.Request.Headers["X-Stella-Tenant"].FirstOrDefault();
|
||||
if (!string.IsNullOrWhiteSpace(headerTenant))
|
||||
{
|
||||
tenant = headerTenant.Trim().ToLowerInvariant();
|
||||
}
|
||||
else if (!string.IsNullOrWhiteSpace(options.DefaultTenant))
|
||||
{
|
||||
tenant = options.DefaultTenant.Trim().ToLowerInvariant();
|
||||
}
|
||||
else
|
||||
{
|
||||
problem = Results.BadRequest(new
|
||||
{
|
||||
error = new { code = "ERR_TENANT", message = "X-Stella-Tenant header is required" }
|
||||
});
|
||||
return false;
|
||||
}
|
||||
|
||||
return true;
|
||||
}
|
||||
|
||||
private static string EncodeCursor(DateTime timestamp, string id)
|
||||
{
|
||||
var raw = $"{timestamp:O}|{id}";
|
||||
return Convert.ToBase64String(System.Text.Encoding.UTF8.GetBytes(raw));
|
||||
}
|
||||
|
||||
private static VexLinksetScope BuildScope(DomainVexProductScope scope)
|
||||
{
|
||||
return new VexLinksetScope(
|
||||
ProductKey: scope.ProductKey,
|
||||
Type: scope.Type,
|
||||
Version: scope.Version,
|
||||
Purl: scope.Purl,
|
||||
Cpe: scope.Cpe,
|
||||
Identifiers: scope.Identifiers.ToList());
|
||||
}
|
||||
}
|
||||
|
||||
// Detail response for single linkset
|
||||
public sealed record VexLinksetDetailResponse(
|
||||
[property: JsonPropertyName("linksetId")] string LinksetId,
|
||||
[property: JsonPropertyName("tenant")] string Tenant,
|
||||
[property: JsonPropertyName("vulnerabilityId")] string VulnerabilityId,
|
||||
[property: JsonPropertyName("productKey")] string ProductKey,
|
||||
[property: JsonPropertyName("scope")] VexLinksetScope Scope,
|
||||
[property: JsonPropertyName("providerIds")] IReadOnlyList<string> ProviderIds,
|
||||
[property: JsonPropertyName("statuses")] IReadOnlyList<string> Statuses,
|
||||
[property: JsonPropertyName("confidence")] string Confidence,
|
||||
[property: JsonPropertyName("hasConflicts")] bool HasConflicts,
|
||||
[property: JsonPropertyName("disagreements")] IReadOnlyList<VexLinksetDisagreement> Disagreements,
|
||||
[property: JsonPropertyName("observations")] IReadOnlyList<VexLinksetObservationRef> Observations,
|
||||
[property: JsonPropertyName("createdAt")] DateTimeOffset CreatedAt,
|
||||
[property: JsonPropertyName("updatedAt")] DateTimeOffset UpdatedAt);
|
||||
|
||||
// Count response
|
||||
public sealed record LinksetCountResponse(
|
||||
[property: JsonPropertyName("total")] long Total,
|
||||
[property: JsonPropertyName("withConflicts")] long WithConflicts);
|
||||
@@ -0,0 +1,414 @@
|
||||
|
||||
using Microsoft.AspNetCore.Builder;
|
||||
using Microsoft.AspNetCore.Http;
|
||||
using Microsoft.AspNetCore.Mvc;
|
||||
using Microsoft.Extensions.Logging;
|
||||
using Microsoft.Extensions.Options;
|
||||
using StellaOps.Auth.ServerIntegration.Tenancy;
|
||||
using StellaOps.Excititor.Core;
|
||||
using StellaOps.Excititor.Core.Storage;
|
||||
using StellaOps.Excititor.Export;
|
||||
using StellaOps.Excititor.WebService.Security;
|
||||
using StellaOps.Excititor.WebService.Services;
|
||||
using System.Collections.Immutable;
|
||||
using System.Globalization;
|
||||
using System.IO;
|
||||
using System.Text;
|
||||
|
||||
namespace StellaOps.Excititor.WebService.Endpoints;
|
||||
|
||||
internal static class MirrorEndpoints
|
||||
{
|
||||
public static void MapMirrorEndpoints(WebApplication app)
|
||||
{
|
||||
var group = app.MapGroup("/excititor/mirror")
|
||||
.RequireTenant();
|
||||
|
||||
group.MapGet("/domains", HandleListDomainsAsync)
|
||||
.WithName("ListMirrorDomains")
|
||||
.WithDescription("Lists all configured VEX mirror distribution domains and their rate-limit settings. Anonymous access is permitted; per-domain authentication requirements are enforced at index and download operations.")
|
||||
.AllowAnonymous();
|
||||
|
||||
group.MapGet("/domains/{domainId}", HandleDomainDetailAsync)
|
||||
.WithName("GetMirrorDomain")
|
||||
.WithDescription("Returns configuration details for a specific mirror domain, including available export keys and per-operation rate limits.")
|
||||
.AllowAnonymous();
|
||||
|
||||
group.MapGet("/domains/{domainId}/index", HandleDomainIndexAsync)
|
||||
.WithName("GetMirrorDomainIndex")
|
||||
.WithDescription("Returns the current export index for a mirror domain, listing all available exports with their artifact addresses, attestations, and staleness status. Authentication required for domains with RequireAuthentication=true.")
|
||||
.RequireAuthorization(ExcititorPolicies.VexRead);
|
||||
|
||||
group.MapGet("/domains/{domainId}/exports/{exportKey}", HandleExportMetadataAsync)
|
||||
.WithName("GetMirrorExportMetadata")
|
||||
.WithDescription("Returns metadata for a specific export including query signature, artifact content address, size, and source providers. Used by mirrors to verify export integrity before downloading.")
|
||||
.RequireAuthorization(ExcititorPolicies.VexRead);
|
||||
|
||||
group.MapGet("/domains/{domainId}/exports/{exportKey}/download", HandleExportDownloadAsync)
|
||||
.WithName("DownloadMirrorExport")
|
||||
.WithDescription("Streams the export artifact for download in the configured format (JSON, JSONL, OpenVEX, CSAF, CycloneDX). Subject to per-domain download rate limits. Authentication required for protected domains.")
|
||||
.RequireAuthorization(ExcititorPolicies.VexRead);
|
||||
}
|
||||
|
||||
private static async Task<IResult> HandleListDomainsAsync(
|
||||
HttpContext httpContext,
|
||||
IOptions<MirrorDistributionOptions> options,
|
||||
CancellationToken cancellationToken)
|
||||
{
|
||||
var domains = options.Value.Domains
|
||||
.Select(static domain => new MirrorDomainSummary(
|
||||
domain.Id,
|
||||
string.IsNullOrWhiteSpace(domain.DisplayName) ? domain.Id : domain.DisplayName,
|
||||
domain.RequireAuthentication,
|
||||
Math.Max(domain.MaxIndexRequestsPerHour, 0),
|
||||
Math.Max(domain.MaxDownloadRequestsPerHour, 0)))
|
||||
.ToArray();
|
||||
|
||||
await WriteJsonAsync(httpContext, new MirrorDomainListResponse(domains), StatusCodes.Status200OK, cancellationToken).ConfigureAwait(false);
|
||||
return Results.Empty;
|
||||
}
|
||||
|
||||
private static async Task<IResult> HandleDomainDetailAsync(
|
||||
string domainId,
|
||||
HttpContext httpContext,
|
||||
IOptions<MirrorDistributionOptions> options,
|
||||
CancellationToken cancellationToken)
|
||||
{
|
||||
if (!TryFindDomain(options.Value, domainId, out var domain))
|
||||
{
|
||||
return Results.NotFound();
|
||||
}
|
||||
|
||||
var response = new MirrorDomainDetail(
|
||||
domain.Id,
|
||||
string.IsNullOrWhiteSpace(domain.DisplayName) ? domain.Id : domain.DisplayName,
|
||||
domain.RequireAuthentication,
|
||||
Math.Max(domain.MaxIndexRequestsPerHour, 0),
|
||||
Math.Max(domain.MaxDownloadRequestsPerHour, 0),
|
||||
domain.Exports.Select(static export => export.Key).OrderBy(static key => key, StringComparer.Ordinal).ToImmutableArray());
|
||||
|
||||
await WriteJsonAsync(httpContext, response, StatusCodes.Status200OK, cancellationToken).ConfigureAwait(false);
|
||||
return Results.Empty;
|
||||
}
|
||||
|
||||
private static async Task<IResult> HandleDomainIndexAsync(
|
||||
string domainId,
|
||||
HttpContext httpContext,
|
||||
IOptions<MirrorDistributionOptions> options,
|
||||
[FromServices] MirrorRateLimiter rateLimiter,
|
||||
[FromServices] IVexExportStore exportStore,
|
||||
[FromServices] TimeProvider timeProvider,
|
||||
CancellationToken cancellationToken)
|
||||
{
|
||||
if (!TryFindDomain(options.Value, domainId, out var domain))
|
||||
{
|
||||
return Results.NotFound();
|
||||
}
|
||||
|
||||
if (domain.RequireAuthentication && (httpContext.User?.Identity?.IsAuthenticated is not true))
|
||||
{
|
||||
return Results.Unauthorized();
|
||||
}
|
||||
|
||||
if (!rateLimiter.TryAcquire(domain.Id, "index", Math.Max(domain.MaxIndexRequestsPerHour, 0), out var retryAfter))
|
||||
{
|
||||
if (retryAfter is { } retry)
|
||||
{
|
||||
httpContext.Response.Headers.RetryAfter = ((int)Math.Ceiling(retry.TotalSeconds)).ToString(CultureInfo.InvariantCulture);
|
||||
}
|
||||
|
||||
await WritePlainTextAsync(httpContext, "mirror index quota exceeded", StatusCodes.Status429TooManyRequests, cancellationToken).ConfigureAwait(false);
|
||||
return Results.Empty;
|
||||
}
|
||||
|
||||
var resolvedExports = new List<MirrorExportIndexEntry>();
|
||||
foreach (var exportOption in domain.Exports)
|
||||
{
|
||||
if (!MirrorExportPlanner.TryBuild(exportOption, out var plan, out var error))
|
||||
{
|
||||
resolvedExports.Add(new MirrorExportIndexEntry(
|
||||
exportOption.Key,
|
||||
null,
|
||||
null,
|
||||
exportOption.Format,
|
||||
null,
|
||||
null,
|
||||
0,
|
||||
null,
|
||||
null,
|
||||
error ?? "invalid_export_configuration"));
|
||||
continue;
|
||||
}
|
||||
|
||||
var manifest = await exportStore.FindAsync(plan.Signature, plan.Format, cancellationToken).ConfigureAwait(false);
|
||||
|
||||
if (manifest is null)
|
||||
{
|
||||
resolvedExports.Add(new MirrorExportIndexEntry(
|
||||
exportOption.Key,
|
||||
null,
|
||||
plan.Signature.Value,
|
||||
plan.Format.ToString().ToLowerInvariant(),
|
||||
null,
|
||||
null,
|
||||
0,
|
||||
null,
|
||||
null,
|
||||
"manifest_not_found"));
|
||||
continue;
|
||||
}
|
||||
|
||||
resolvedExports.Add(new MirrorExportIndexEntry(
|
||||
exportOption.Key,
|
||||
manifest.ExportId,
|
||||
manifest.QuerySignature.Value,
|
||||
manifest.Format.ToString().ToLowerInvariant(),
|
||||
manifest.CreatedAt,
|
||||
manifest.Artifact,
|
||||
manifest.SizeBytes,
|
||||
manifest.ConsensusRevision,
|
||||
manifest.Attestation is null ? null : new MirrorExportAttestation(manifest.Attestation.PredicateType, manifest.Attestation.Rekor?.Location, manifest.Attestation.EnvelopeDigest, manifest.Attestation.SignedAt),
|
||||
null));
|
||||
}
|
||||
|
||||
var indexResponse = new MirrorDomainIndex(
|
||||
domain.Id,
|
||||
string.IsNullOrWhiteSpace(domain.DisplayName) ? domain.Id : domain.DisplayName,
|
||||
timeProvider.GetUtcNow(),
|
||||
resolvedExports.ToImmutableArray());
|
||||
|
||||
await WriteJsonAsync(httpContext, indexResponse, StatusCodes.Status200OK, cancellationToken).ConfigureAwait(false);
|
||||
return Results.Empty;
|
||||
}
|
||||
|
||||
private static async Task<IResult> HandleExportMetadataAsync(
|
||||
string domainId,
|
||||
string exportKey,
|
||||
HttpContext httpContext,
|
||||
IOptions<MirrorDistributionOptions> options,
|
||||
[FromServices] MirrorRateLimiter rateLimiter,
|
||||
[FromServices] IVexExportStore exportStore,
|
||||
[FromServices] TimeProvider timeProvider,
|
||||
CancellationToken cancellationToken)
|
||||
{
|
||||
if (!TryFindDomain(options.Value, domainId, out var domain))
|
||||
{
|
||||
return Results.NotFound();
|
||||
}
|
||||
|
||||
if (domain.RequireAuthentication && (httpContext.User?.Identity?.IsAuthenticated is not true))
|
||||
{
|
||||
return Results.Unauthorized();
|
||||
}
|
||||
|
||||
if (!TryFindExport(domain, exportKey, out var exportOptions))
|
||||
{
|
||||
return Results.NotFound();
|
||||
}
|
||||
|
||||
if (!MirrorExportPlanner.TryBuild(exportOptions, out var plan, out var error))
|
||||
{
|
||||
await WritePlainTextAsync(httpContext, error ?? "invalid_export_configuration", StatusCodes.Status503ServiceUnavailable, cancellationToken).ConfigureAwait(false);
|
||||
return Results.Empty;
|
||||
}
|
||||
|
||||
var manifest = await exportStore.FindAsync(plan.Signature, plan.Format, cancellationToken).ConfigureAwait(false);
|
||||
if (manifest is null)
|
||||
{
|
||||
return Results.NotFound();
|
||||
}
|
||||
|
||||
var payload = new MirrorExportMetadata(
|
||||
domain.Id,
|
||||
exportOptions.Key,
|
||||
manifest.ExportId,
|
||||
manifest.QuerySignature.Value,
|
||||
manifest.Format.ToString().ToLowerInvariant(),
|
||||
manifest.CreatedAt,
|
||||
manifest.Artifact,
|
||||
manifest.SizeBytes,
|
||||
manifest.SourceProviders,
|
||||
manifest.Attestation is null ? null : new MirrorExportAttestation(manifest.Attestation.PredicateType, manifest.Attestation.Rekor?.Location, manifest.Attestation.EnvelopeDigest, manifest.Attestation.SignedAt));
|
||||
|
||||
await WriteJsonAsync(httpContext, payload, StatusCodes.Status200OK, cancellationToken).ConfigureAwait(false);
|
||||
return Results.Empty;
|
||||
}
|
||||
|
||||
private static async Task<IResult> HandleExportDownloadAsync(
|
||||
string domainId,
|
||||
string exportKey,
|
||||
HttpContext httpContext,
|
||||
IOptions<MirrorDistributionOptions> options,
|
||||
[FromServices] MirrorRateLimiter rateLimiter,
|
||||
[FromServices] IVexExportStore exportStore,
|
||||
[FromServices] IEnumerable<IVexArtifactStore> artifactStores,
|
||||
CancellationToken cancellationToken)
|
||||
{
|
||||
if (!TryFindDomain(options.Value, domainId, out var domain))
|
||||
{
|
||||
return Results.NotFound();
|
||||
}
|
||||
|
||||
if (domain.RequireAuthentication && (httpContext.User?.Identity?.IsAuthenticated is not true))
|
||||
{
|
||||
return Results.Unauthorized();
|
||||
}
|
||||
|
||||
if (!rateLimiter.TryAcquire(domain.Id, "download", Math.Max(domain.MaxDownloadRequestsPerHour, 0), out var retryAfter))
|
||||
{
|
||||
if (retryAfter is { } retry)
|
||||
{
|
||||
httpContext.Response.Headers.RetryAfter = ((int)Math.Ceiling(retry.TotalSeconds)).ToString(CultureInfo.InvariantCulture);
|
||||
}
|
||||
|
||||
await WritePlainTextAsync(httpContext, "mirror download quota exceeded", StatusCodes.Status429TooManyRequests, cancellationToken).ConfigureAwait(false);
|
||||
return Results.Empty;
|
||||
}
|
||||
|
||||
if (!TryFindExport(domain, exportKey, out var exportOptions) || !MirrorExportPlanner.TryBuild(exportOptions, out var plan, out _))
|
||||
{
|
||||
return Results.NotFound();
|
||||
}
|
||||
|
||||
var manifest = await exportStore.FindAsync(plan.Signature, plan.Format, cancellationToken).ConfigureAwait(false);
|
||||
if (manifest is null)
|
||||
{
|
||||
return Results.NotFound();
|
||||
}
|
||||
|
||||
Stream? contentStream = null;
|
||||
foreach (var store in artifactStores)
|
||||
{
|
||||
contentStream = await store.OpenReadAsync(manifest.Artifact, cancellationToken).ConfigureAwait(false);
|
||||
if (contentStream is not null)
|
||||
{
|
||||
break;
|
||||
}
|
||||
}
|
||||
|
||||
if (contentStream is null)
|
||||
{
|
||||
return Results.NotFound();
|
||||
}
|
||||
|
||||
await using (contentStream.ConfigureAwait(false))
|
||||
{
|
||||
var contentType = ResolveContentType(manifest.Format);
|
||||
httpContext.Response.StatusCode = StatusCodes.Status200OK;
|
||||
httpContext.Response.ContentType = contentType;
|
||||
httpContext.Response.Headers.ContentDisposition = FormattableString.Invariant($"attachment; filename=\"{BuildDownloadFileName(domain.Id, exportOptions.Key, manifest.Format)}\"");
|
||||
|
||||
await contentStream.CopyToAsync(httpContext.Response.Body, cancellationToken).ConfigureAwait(false);
|
||||
}
|
||||
|
||||
return Results.Empty;
|
||||
}
|
||||
|
||||
private static bool TryFindDomain(MirrorDistributionOptions options, string domainId, out MirrorDomainOptions domain)
|
||||
{
|
||||
domain = options.Domains.FirstOrDefault(d => string.Equals(d.Id, domainId, StringComparison.OrdinalIgnoreCase))!;
|
||||
return domain is not null;
|
||||
}
|
||||
|
||||
private static bool TryFindExport(MirrorDomainOptions domain, string exportKey, out MirrorExportOptions export)
|
||||
{
|
||||
export = domain.Exports.FirstOrDefault(e => string.Equals(e.Key, exportKey, StringComparison.OrdinalIgnoreCase))!;
|
||||
return export is not null;
|
||||
}
|
||||
|
||||
private static string ResolveContentType(VexExportFormat format)
|
||||
=> format switch
|
||||
{
|
||||
VexExportFormat.Json => "application/json",
|
||||
VexExportFormat.JsonLines => "application/jsonl",
|
||||
VexExportFormat.OpenVex => "application/json",
|
||||
VexExportFormat.Csaf => "application/json",
|
||||
VexExportFormat.CycloneDx => "application/json",
|
||||
_ => "application/octet-stream",
|
||||
};
|
||||
|
||||
private static string BuildDownloadFileName(string domainId, string exportKey, VexExportFormat format)
|
||||
{
|
||||
var builder = new StringBuilder(domainId.Length + exportKey.Length + 8);
|
||||
builder.Append(domainId).Append('-').Append(exportKey);
|
||||
builder.Append(format switch
|
||||
{
|
||||
VexExportFormat.Json => ".json",
|
||||
VexExportFormat.JsonLines => ".jsonl",
|
||||
VexExportFormat.OpenVex => ".openvex.json",
|
||||
VexExportFormat.Csaf => ".csaf.json",
|
||||
VexExportFormat.CycloneDx => ".cyclonedx.json",
|
||||
_ => ".bin",
|
||||
});
|
||||
return builder.ToString();
|
||||
}
|
||||
|
||||
private static async Task WritePlainTextAsync(HttpContext context, string message, int statusCode, CancellationToken cancellationToken)
|
||||
{
|
||||
context.Response.StatusCode = statusCode;
|
||||
context.Response.ContentType = "text/plain";
|
||||
await context.Response.WriteAsync(message, cancellationToken);
|
||||
}
|
||||
|
||||
private static async Task WriteJsonAsync<T>(HttpContext context, T payload, int statusCode, CancellationToken cancellationToken)
|
||||
{
|
||||
context.Response.StatusCode = statusCode;
|
||||
context.Response.ContentType = "application/json";
|
||||
var json = VexCanonicalJsonSerializer.Serialize(payload);
|
||||
await context.Response.WriteAsync(json, cancellationToken);
|
||||
}
|
||||
|
||||
}
|
||||
|
||||
internal sealed record MirrorDomainListResponse(IReadOnlyList<MirrorDomainSummary> Domains);
|
||||
|
||||
internal sealed record MirrorDomainSummary(
|
||||
string Id,
|
||||
string DisplayName,
|
||||
bool RequireAuthentication,
|
||||
int MaxIndexRequestsPerHour,
|
||||
int MaxDownloadRequestsPerHour);
|
||||
|
||||
internal sealed record MirrorDomainDetail(
|
||||
string Id,
|
||||
string DisplayName,
|
||||
bool RequireAuthentication,
|
||||
int MaxIndexRequestsPerHour,
|
||||
int MaxDownloadRequestsPerHour,
|
||||
IReadOnlyList<string> Exports);
|
||||
|
||||
internal sealed record MirrorDomainIndex(
|
||||
string Id,
|
||||
string DisplayName,
|
||||
DateTimeOffset GeneratedAt,
|
||||
IReadOnlyList<MirrorExportIndexEntry> Exports);
|
||||
|
||||
internal sealed record MirrorExportIndexEntry(
|
||||
string ExportKey,
|
||||
string? ExportId,
|
||||
string? QuerySignature,
|
||||
string Format,
|
||||
DateTimeOffset? CreatedAt,
|
||||
VexContentAddress? Artifact,
|
||||
long SizeBytes,
|
||||
string? ConsensusRevision,
|
||||
MirrorExportAttestation? Attestation,
|
||||
string? Status);
|
||||
|
||||
internal sealed record MirrorExportAttestation(
|
||||
string PredicateType,
|
||||
string? RekorLocation,
|
||||
string? EnvelopeDigest,
|
||||
DateTimeOffset? SignedAt);
|
||||
|
||||
internal sealed record MirrorExportMetadata(
|
||||
string DomainId,
|
||||
string ExportKey,
|
||||
string ExportId,
|
||||
string QuerySignature,
|
||||
string Format,
|
||||
DateTimeOffset CreatedAt,
|
||||
VexContentAddress Artifact,
|
||||
long SizeBytes,
|
||||
IReadOnlyList<string> SourceProviders,
|
||||
MirrorExportAttestation? Attestation);
|
||||
@@ -0,0 +1,277 @@
|
||||
|
||||
using Microsoft.AspNetCore.Builder;
|
||||
using Microsoft.AspNetCore.Http;
|
||||
using Microsoft.AspNetCore.Mvc;
|
||||
using Microsoft.Extensions.Logging;
|
||||
using StellaOps.Auth.ServerIntegration.Tenancy;
|
||||
using StellaOps.Excititor.Core;
|
||||
using StellaOps.Excititor.Core.Storage;
|
||||
using StellaOps.Excititor.WebService.Contracts;
|
||||
using StellaOps.Excititor.WebService.Security;
|
||||
using System;
|
||||
using System.Collections.Generic;
|
||||
using System.Linq;
|
||||
using System.Threading;
|
||||
using System.Threading.Tasks;
|
||||
|
||||
namespace StellaOps.Excititor.WebService.Endpoints;
|
||||
|
||||
/// <summary>
|
||||
/// Endpoints for mirror bundle registration, provenance exposure, and timeline queries (EXCITITOR-AIRGAP-56-001).
|
||||
/// </summary>
|
||||
internal static class MirrorRegistrationEndpoints
|
||||
{
|
||||
public static void MapMirrorRegistrationEndpoints(WebApplication app)
|
||||
{
|
||||
var group = app.MapGroup("/airgap/v1/mirror/bundles")
|
||||
.RequireTenant();
|
||||
|
||||
group.MapGet("/", HandleListBundlesAsync)
|
||||
.WithName("ListMirrorBundles")
|
||||
.WithDescription("Lists registered air-gap mirror bundles with optional filters by publisher and import timestamp. Returns staleness metrics and import status per bundle to support synchronization monitoring.")
|
||||
.RequireAuthorization(ExcititorPolicies.VexRead);
|
||||
|
||||
group.MapGet("/{bundleId}", HandleGetBundleAsync)
|
||||
.WithName("GetMirrorBundle")
|
||||
.WithDescription("Returns full detail for a specific mirror bundle including provenance (payload hash, signature, transparency log reference), staleness categorization, and file paths within the evidence locker.")
|
||||
.RequireAuthorization(ExcititorPolicies.VexRead);
|
||||
|
||||
group.MapGet("/{bundleId}/timeline", HandleGetBundleTimelineAsync)
|
||||
.WithName("GetMirrorBundleTimeline")
|
||||
.WithDescription("Returns the ordered event timeline for a mirror bundle, tracking state transitions from import started through completion or failure, with staleness seconds and remediation hints per event.")
|
||||
.RequireAuthorization(ExcititorPolicies.VexRead);
|
||||
}
|
||||
|
||||
private static async Task<IResult> HandleListBundlesAsync(
|
||||
HttpContext httpContext,
|
||||
[FromServices] IAirgapImportStore importStore,
|
||||
[FromServices] TimeProvider timeProvider,
|
||||
[FromServices] ILogger<MirrorRegistrationEndpointsMarker> logger,
|
||||
[FromQuery] string? publisher = null,
|
||||
[FromQuery] string? importedAfter = null,
|
||||
[FromQuery] int limit = 50,
|
||||
[FromQuery] int offset = 0,
|
||||
CancellationToken cancellationToken = default)
|
||||
{
|
||||
var tenantId = ResolveTenantId(httpContext);
|
||||
var now = timeProvider.GetUtcNow();
|
||||
|
||||
DateTimeOffset? afterFilter = null;
|
||||
if (!string.IsNullOrWhiteSpace(importedAfter) && DateTimeOffset.TryParse(importedAfter, out var parsed))
|
||||
{
|
||||
afterFilter = parsed;
|
||||
}
|
||||
|
||||
var clampedLimit = Math.Clamp(limit, 1, 100);
|
||||
var clampedOffset = Math.Max(0, offset);
|
||||
|
||||
var records = await importStore.ListAsync(
|
||||
tenantId,
|
||||
publisher,
|
||||
afterFilter,
|
||||
clampedLimit,
|
||||
clampedOffset,
|
||||
cancellationToken).ConfigureAwait(false);
|
||||
|
||||
var totalCount = await importStore.CountAsync(
|
||||
tenantId,
|
||||
publisher,
|
||||
afterFilter,
|
||||
cancellationToken).ConfigureAwait(false);
|
||||
|
||||
var summaries = records.Select(record =>
|
||||
{
|
||||
var stalenessSeconds = StalenessCalculator.ComputeSeconds(record.SignedAt, now);
|
||||
var status = DetermineStatus(record.Timeline);
|
||||
return new MirrorBundleSummary(
|
||||
record.BundleId,
|
||||
record.MirrorGeneration,
|
||||
record.Publisher,
|
||||
record.SignedAt,
|
||||
record.ImportedAt,
|
||||
record.PayloadHash,
|
||||
stalenessSeconds,
|
||||
status);
|
||||
}).ToList();
|
||||
|
||||
var response = new MirrorBundleListResponse(
|
||||
summaries,
|
||||
totalCount,
|
||||
clampedLimit,
|
||||
clampedOffset,
|
||||
now);
|
||||
|
||||
await WriteJsonAsync(httpContext, response, StatusCodes.Status200OK, cancellationToken).ConfigureAwait(false);
|
||||
return Results.Empty;
|
||||
}
|
||||
|
||||
private static async Task<IResult> HandleGetBundleAsync(
|
||||
string bundleId,
|
||||
HttpContext httpContext,
|
||||
[FromServices] IAirgapImportStore importStore,
|
||||
[FromServices] TimeProvider timeProvider,
|
||||
[FromServices] ILogger<MirrorRegistrationEndpointsMarker> logger,
|
||||
[FromQuery] string? generation = null,
|
||||
CancellationToken cancellationToken = default)
|
||||
{
|
||||
var tenantId = ResolveTenantId(httpContext);
|
||||
var now = timeProvider.GetUtcNow();
|
||||
|
||||
var record = await importStore.FindByBundleIdAsync(
|
||||
tenantId,
|
||||
bundleId,
|
||||
generation,
|
||||
cancellationToken).ConfigureAwait(false);
|
||||
|
||||
if (record is null)
|
||||
{
|
||||
var errorResponse = AirgapErrorMapping.BundleNotFound(bundleId, generation);
|
||||
await WriteJsonAsync(httpContext, errorResponse, StatusCodes.Status404NotFound, cancellationToken).ConfigureAwait(false);
|
||||
return Results.Empty;
|
||||
}
|
||||
|
||||
var sinceSignedSeconds = StalenessCalculator.ComputeSeconds(record.SignedAt, now);
|
||||
var sinceImportedSeconds = StalenessCalculator.ComputeSeconds(record.ImportedAt, now);
|
||||
|
||||
var staleness = new MirrorBundleStaleness(
|
||||
sinceSignedSeconds,
|
||||
sinceImportedSeconds,
|
||||
StalenessCalculator.CategorizeAge(sinceSignedSeconds),
|
||||
StalenessCalculator.CategorizeAge(sinceImportedSeconds));
|
||||
|
||||
var provenance = new MirrorBundleProvenance(
|
||||
record.PayloadHash,
|
||||
record.Signature,
|
||||
record.PayloadUrl,
|
||||
record.TransparencyLog,
|
||||
record.PortableManifestHash ?? string.Empty);
|
||||
|
||||
var paths = new MirrorBundlePaths(
|
||||
record.PortableManifestPath ?? string.Empty,
|
||||
record.EvidenceLockerPath ?? string.Empty);
|
||||
|
||||
var timeline = record.Timeline
|
||||
.OrderByDescending(e => e.CreatedAt)
|
||||
.Select(e => new MirrorBundleTimelineEntry(
|
||||
e.EventType,
|
||||
e.CreatedAt,
|
||||
e.StalenessSeconds,
|
||||
e.ErrorCode,
|
||||
e.Message,
|
||||
e.Remediation,
|
||||
e.Actor,
|
||||
e.Scopes))
|
||||
.ToList();
|
||||
|
||||
var response = new MirrorBundleDetailResponse(
|
||||
record.BundleId,
|
||||
record.MirrorGeneration,
|
||||
record.TenantId,
|
||||
record.Publisher,
|
||||
record.SignedAt,
|
||||
record.ImportedAt,
|
||||
provenance,
|
||||
staleness,
|
||||
paths,
|
||||
timeline,
|
||||
now);
|
||||
|
||||
await WriteJsonAsync(httpContext, response, StatusCodes.Status200OK, cancellationToken).ConfigureAwait(false);
|
||||
return Results.Empty;
|
||||
}
|
||||
|
||||
private static async Task<IResult> HandleGetBundleTimelineAsync(
|
||||
string bundleId,
|
||||
HttpContext httpContext,
|
||||
[FromServices] IAirgapImportStore importStore,
|
||||
[FromServices] TimeProvider timeProvider,
|
||||
[FromServices] ILogger<MirrorRegistrationEndpointsMarker> logger,
|
||||
[FromQuery] string? generation = null,
|
||||
CancellationToken cancellationToken = default)
|
||||
{
|
||||
var tenantId = ResolveTenantId(httpContext);
|
||||
var now = timeProvider.GetUtcNow();
|
||||
|
||||
var record = await importStore.FindByBundleIdAsync(
|
||||
tenantId,
|
||||
bundleId,
|
||||
generation,
|
||||
cancellationToken).ConfigureAwait(false);
|
||||
|
||||
if (record is null)
|
||||
{
|
||||
var errorResponse = AirgapErrorMapping.BundleNotFound(bundleId, generation);
|
||||
await WriteJsonAsync(httpContext, errorResponse, StatusCodes.Status404NotFound, cancellationToken).ConfigureAwait(false);
|
||||
return Results.Empty;
|
||||
}
|
||||
|
||||
var timeline = record.Timeline
|
||||
.OrderByDescending(e => e.CreatedAt)
|
||||
.Select(e => new MirrorBundleTimelineEntry(
|
||||
e.EventType,
|
||||
e.CreatedAt,
|
||||
e.StalenessSeconds,
|
||||
e.ErrorCode,
|
||||
e.Message,
|
||||
e.Remediation,
|
||||
e.Actor,
|
||||
e.Scopes))
|
||||
.ToList();
|
||||
|
||||
var response = new MirrorBundleTimelineResponse(
|
||||
record.BundleId,
|
||||
record.MirrorGeneration,
|
||||
timeline,
|
||||
now);
|
||||
|
||||
await WriteJsonAsync(httpContext, response, StatusCodes.Status200OK, cancellationToken).ConfigureAwait(false);
|
||||
return Results.Empty;
|
||||
}
|
||||
|
||||
private static string ResolveTenantId(HttpContext httpContext)
|
||||
{
|
||||
if (httpContext.Request.Headers.TryGetValue("X-Tenant-Id", out var tenantHeader)
|
||||
&& !string.IsNullOrWhiteSpace(tenantHeader.ToString()))
|
||||
{
|
||||
return tenantHeader.ToString();
|
||||
}
|
||||
|
||||
return "default";
|
||||
}
|
||||
|
||||
private static string DetermineStatus(IEnumerable<AirgapTimelineEntry> timeline)
|
||||
{
|
||||
var entries = timeline.ToList();
|
||||
if (entries.Count == 0)
|
||||
{
|
||||
return "unknown";
|
||||
}
|
||||
|
||||
var latestEvent = entries.MaxBy(e => e.CreatedAt);
|
||||
if (latestEvent is null)
|
||||
{
|
||||
return "unknown";
|
||||
}
|
||||
|
||||
return latestEvent.EventType switch
|
||||
{
|
||||
"airgap.import.completed" => "completed",
|
||||
"airgap.import.failed" => "failed",
|
||||
"airgap.import.started" => "in_progress",
|
||||
_ => "unknown",
|
||||
};
|
||||
}
|
||||
|
||||
private static async Task WriteJsonAsync<T>(HttpContext context, T payload, int statusCode, CancellationToken cancellationToken)
|
||||
{
|
||||
context.Response.StatusCode = statusCode;
|
||||
context.Response.ContentType = "application/json";
|
||||
var json = VexCanonicalJsonSerializer.Serialize(payload);
|
||||
await context.Response.WriteAsync(json, cancellationToken);
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Marker type for logger category resolution.
|
||||
/// </summary>
|
||||
internal sealed class MirrorRegistrationEndpointsMarker { }
|
||||
@@ -0,0 +1,322 @@
|
||||
|
||||
using Microsoft.AspNetCore.Builder;
|
||||
using Microsoft.AspNetCore.Http;
|
||||
using Microsoft.AspNetCore.Mvc;
|
||||
using Microsoft.Extensions.Options;
|
||||
using StellaOps.Auth.ServerIntegration.Tenancy;
|
||||
using StellaOps.Excititor.Core.Observations;
|
||||
using StellaOps.Excititor.Core.Storage;
|
||||
using StellaOps.Excititor.WebService.Contracts;
|
||||
using StellaOps.Excititor.WebService.Security;
|
||||
using static StellaOps.Localization.T;
|
||||
using StellaOps.Excititor.WebService.Services;
|
||||
using System;
|
||||
using System.Collections.Generic;
|
||||
using System.Linq;
|
||||
|
||||
namespace StellaOps.Excititor.WebService.Endpoints;
|
||||
|
||||
/// <summary>
|
||||
/// Observation API endpoints (EXCITITOR-LNM-21-201).
|
||||
/// Exposes /vex/observations/* endpoints with filters for advisory/product/provider,
|
||||
/// strict RBAC, and deterministic pagination (no derived verdict fields).
|
||||
/// </summary>
|
||||
public static class ObservationEndpoints
|
||||
{
|
||||
public static void MapObservationEndpoints(this WebApplication app)
|
||||
{
|
||||
var group = app.MapGroup("/vex/observations")
|
||||
.RequireAuthorization(ExcititorPolicies.VexRead)
|
||||
.RequireTenant();
|
||||
|
||||
// GET /vex/observations - List observations with filters
|
||||
group.MapGet("", async (
|
||||
HttpContext context,
|
||||
IOptions<VexStorageOptions> storageOptions,
|
||||
[FromServices] IVexObservationStore observationStore,
|
||||
TimeProvider timeProvider,
|
||||
[FromQuery] int? limit,
|
||||
[FromQuery] string? cursor,
|
||||
[FromQuery] string? vulnerabilityId,
|
||||
[FromQuery] string? productKey,
|
||||
[FromQuery] string? providerId,
|
||||
CancellationToken cancellationToken) =>
|
||||
{
|
||||
var scopeResult = ScopeAuthorization.RequireScope(context, "vex.read");
|
||||
if (scopeResult is not null)
|
||||
{
|
||||
return scopeResult;
|
||||
}
|
||||
|
||||
if (!TryResolveTenant(context, storageOptions.Value, out var tenant, out var tenantError))
|
||||
{
|
||||
return tenantError;
|
||||
}
|
||||
|
||||
var take = Math.Clamp(limit.GetValueOrDefault(50), 1, 100);
|
||||
|
||||
IReadOnlyList<VexObservation> observations;
|
||||
|
||||
// Route to appropriate query method based on filters
|
||||
if (!string.IsNullOrWhiteSpace(vulnerabilityId) && !string.IsNullOrWhiteSpace(productKey))
|
||||
{
|
||||
observations = await observationStore
|
||||
.FindByVulnerabilityAndProductAsync(tenant, vulnerabilityId.Trim(), productKey.Trim(), cancellationToken)
|
||||
.ConfigureAwait(false);
|
||||
}
|
||||
else if (!string.IsNullOrWhiteSpace(providerId))
|
||||
{
|
||||
observations = await observationStore
|
||||
.FindByProviderAsync(tenant, providerId.Trim(), take, cancellationToken)
|
||||
.ConfigureAwait(false);
|
||||
}
|
||||
else
|
||||
{
|
||||
// No filter - return empty for now (full list requires pagination infrastructure)
|
||||
return Results.BadRequest(new
|
||||
{
|
||||
error = new
|
||||
{
|
||||
code = "ERR_PARAMS",
|
||||
message = _t("excititor.validation.filter_required")
|
||||
}
|
||||
});
|
||||
}
|
||||
|
||||
var items = observations
|
||||
.Take(take)
|
||||
.Select(obs => ToListItem(obs))
|
||||
.ToList();
|
||||
|
||||
var hasMore = observations.Count > take;
|
||||
string? nextCursor = null;
|
||||
if (hasMore && items.Count > 0)
|
||||
{
|
||||
var last = observations[items.Count - 1];
|
||||
nextCursor = EncodeCursor(last.CreatedAt.UtcDateTime, last.ObservationId);
|
||||
}
|
||||
|
||||
var response = new VexObservationListResponse(items, nextCursor);
|
||||
return Results.Ok(response);
|
||||
})
|
||||
.WithName("ListVexObservations")
|
||||
.WithDescription("Lists raw VEX observations from ingested provider documents, filtered by vulnerability ID and product key pair or by provider. Returns statement-level records without consensus or derived severity fields.");
|
||||
|
||||
// GET /vex/observations/{observationId} - Get observation by ID
|
||||
group.MapGet("/{observationId}", async (
|
||||
HttpContext context,
|
||||
string observationId,
|
||||
IOptions<VexStorageOptions> storageOptions,
|
||||
[FromServices] IVexObservationStore observationStore,
|
||||
CancellationToken cancellationToken) =>
|
||||
{
|
||||
var scopeResult = ScopeAuthorization.RequireScope(context, "vex.read");
|
||||
if (scopeResult is not null)
|
||||
{
|
||||
return scopeResult;
|
||||
}
|
||||
|
||||
if (!TryResolveTenant(context, storageOptions.Value, out var tenant, out var tenantError))
|
||||
{
|
||||
return tenantError;
|
||||
}
|
||||
|
||||
if (string.IsNullOrWhiteSpace(observationId))
|
||||
{
|
||||
return Results.BadRequest(new
|
||||
{
|
||||
error = new { code = "ERR_PARAMS", message = _t("excititor.validation.observation_id_required") }
|
||||
});
|
||||
}
|
||||
|
||||
var observation = await observationStore
|
||||
.GetByIdAsync(tenant, observationId.Trim(), cancellationToken)
|
||||
.ConfigureAwait(false);
|
||||
|
||||
if (observation is null)
|
||||
{
|
||||
return Results.NotFound(new
|
||||
{
|
||||
error = new { code = "ERR_NOT_FOUND", message = _t("excititor.error.observation_not_found", observationId) }
|
||||
});
|
||||
}
|
||||
|
||||
var response = ToDetailResponse(observation);
|
||||
return Results.Ok(response);
|
||||
})
|
||||
.WithName("GetVexObservation")
|
||||
.WithDescription("Retrieves the full VEX observation record for a specific observation ID, including upstream document metadata, content format, all statements, linkset aliases, and signature information.");
|
||||
|
||||
// GET /vex/observations/count - Get observation count for tenant
|
||||
group.MapGet("/count", async (
|
||||
HttpContext context,
|
||||
IOptions<VexStorageOptions> storageOptions,
|
||||
[FromServices] IVexObservationStore observationStore,
|
||||
CancellationToken cancellationToken) =>
|
||||
{
|
||||
var scopeResult = ScopeAuthorization.RequireScope(context, "vex.read");
|
||||
if (scopeResult is not null)
|
||||
{
|
||||
return scopeResult;
|
||||
}
|
||||
|
||||
if (!TryResolveTenant(context, storageOptions.Value, out var tenant, out var tenantError))
|
||||
{
|
||||
return tenantError;
|
||||
}
|
||||
|
||||
var count = await observationStore
|
||||
.CountAsync(tenant, cancellationToken)
|
||||
.ConfigureAwait(false);
|
||||
|
||||
return Results.Ok(new { count });
|
||||
})
|
||||
.WithName("CountVexObservations")
|
||||
.WithDescription("Returns the total count of VEX observations stored for the current tenant. Useful for capacity monitoring and detecting ingest pipeline stalls.");
|
||||
}
|
||||
|
||||
private static VexObservationListItem ToListItem(VexObservation obs)
|
||||
{
|
||||
var firstStatement = obs.Statements.FirstOrDefault();
|
||||
return new VexObservationListItem(
|
||||
ObservationId: obs.ObservationId,
|
||||
Tenant: obs.Tenant,
|
||||
ProviderId: obs.ProviderId,
|
||||
VulnerabilityId: firstStatement?.VulnerabilityId ?? string.Empty,
|
||||
ProductKey: firstStatement?.ProductKey ?? string.Empty,
|
||||
Status: firstStatement?.Status.ToString().ToLowerInvariant() ?? "unknown",
|
||||
CreatedAt: obs.CreatedAt,
|
||||
LastObserved: firstStatement?.LastObserved,
|
||||
Purls: obs.Linkset.Purls.ToList());
|
||||
}
|
||||
|
||||
private static VexObservationDetailResponse ToDetailResponse(VexObservation obs)
|
||||
{
|
||||
var upstream = new VexObservationUpstreamResponse(
|
||||
obs.Upstream.UpstreamId,
|
||||
obs.Upstream.DocumentVersion,
|
||||
obs.Upstream.FetchedAt,
|
||||
obs.Upstream.ReceivedAt,
|
||||
obs.Upstream.ContentHash,
|
||||
obs.Upstream.Signature.Present
|
||||
? new VexObservationSignatureResponse(
|
||||
obs.Upstream.Signature.Format ?? "dsse",
|
||||
obs.Upstream.Signature.KeyId,
|
||||
Issuer: null,
|
||||
VerifiedAtUtc: null)
|
||||
: null);
|
||||
|
||||
var content = new VexObservationContentResponse(
|
||||
obs.Content.Format,
|
||||
obs.Content.SpecVersion);
|
||||
|
||||
var statements = obs.Statements
|
||||
.Select(stmt => new VexObservationStatementItem(
|
||||
stmt.VulnerabilityId,
|
||||
stmt.ProductKey,
|
||||
stmt.Status.ToString().ToLowerInvariant(),
|
||||
stmt.LastObserved,
|
||||
stmt.Locator,
|
||||
stmt.Justification?.ToString().ToLowerInvariant(),
|
||||
stmt.IntroducedVersion,
|
||||
stmt.FixedVersion))
|
||||
.ToList();
|
||||
|
||||
var linkset = new VexObservationLinksetResponse(
|
||||
obs.Linkset.Aliases.ToList(),
|
||||
obs.Linkset.Purls.ToList(),
|
||||
obs.Linkset.Cpes.ToList(),
|
||||
obs.Linkset.References.Select(r => new VexObservationReferenceItem(r.Type, r.Url)).ToList());
|
||||
|
||||
return new VexObservationDetailResponse(
|
||||
obs.ObservationId,
|
||||
obs.Tenant,
|
||||
obs.ProviderId,
|
||||
obs.StreamId,
|
||||
upstream,
|
||||
content,
|
||||
statements,
|
||||
linkset,
|
||||
obs.CreatedAt);
|
||||
}
|
||||
|
||||
private static bool TryResolveTenant(
|
||||
HttpContext context,
|
||||
VexStorageOptions options,
|
||||
out string tenant,
|
||||
out IResult? problem)
|
||||
{
|
||||
problem = null;
|
||||
tenant = string.Empty;
|
||||
|
||||
var headerTenant = context.Request.Headers["X-Stella-Tenant"].FirstOrDefault();
|
||||
if (!string.IsNullOrWhiteSpace(headerTenant))
|
||||
{
|
||||
tenant = headerTenant.Trim().ToLowerInvariant();
|
||||
}
|
||||
else if (!string.IsNullOrWhiteSpace(options.DefaultTenant))
|
||||
{
|
||||
tenant = options.DefaultTenant.Trim().ToLowerInvariant();
|
||||
}
|
||||
else
|
||||
{
|
||||
problem = Results.BadRequest(new
|
||||
{
|
||||
error = new { code = "ERR_TENANT", message = _t("excititor.validation.tenant_header_required") }
|
||||
});
|
||||
return false;
|
||||
}
|
||||
|
||||
return true;
|
||||
}
|
||||
|
||||
private static string EncodeCursor(DateTime timestamp, string id)
|
||||
{
|
||||
var raw = $"{timestamp:O}|{id}";
|
||||
return Convert.ToBase64String(System.Text.Encoding.UTF8.GetBytes(raw));
|
||||
}
|
||||
}
|
||||
|
||||
// Additional response DTOs for observation detail
|
||||
public sealed record VexObservationUpstreamResponse(
|
||||
[property: System.Text.Json.Serialization.JsonPropertyName("upstreamId")] string UpstreamId,
|
||||
[property: System.Text.Json.Serialization.JsonPropertyName("documentVersion")] string? DocumentVersion,
|
||||
[property: System.Text.Json.Serialization.JsonPropertyName("fetchedAt")] DateTimeOffset FetchedAt,
|
||||
[property: System.Text.Json.Serialization.JsonPropertyName("receivedAt")] DateTimeOffset ReceivedAt,
|
||||
[property: System.Text.Json.Serialization.JsonPropertyName("contentHash")] string ContentHash,
|
||||
[property: System.Text.Json.Serialization.JsonPropertyName("signature")] VexObservationSignatureResponse? Signature);
|
||||
|
||||
public sealed record VexObservationContentResponse(
|
||||
[property: System.Text.Json.Serialization.JsonPropertyName("format")] string Format,
|
||||
[property: System.Text.Json.Serialization.JsonPropertyName("specVersion")] string? SpecVersion);
|
||||
|
||||
public sealed record VexObservationStatementItem(
|
||||
[property: System.Text.Json.Serialization.JsonPropertyName("vulnerabilityId")] string VulnerabilityId,
|
||||
[property: System.Text.Json.Serialization.JsonPropertyName("productKey")] string ProductKey,
|
||||
[property: System.Text.Json.Serialization.JsonPropertyName("status")] string Status,
|
||||
[property: System.Text.Json.Serialization.JsonPropertyName("lastObserved")] DateTimeOffset? LastObserved,
|
||||
[property: System.Text.Json.Serialization.JsonPropertyName("locator")] string? Locator,
|
||||
[property: System.Text.Json.Serialization.JsonPropertyName("justification")] string? Justification,
|
||||
[property: System.Text.Json.Serialization.JsonPropertyName("introducedVersion")] string? IntroducedVersion,
|
||||
[property: System.Text.Json.Serialization.JsonPropertyName("fixedVersion")] string? FixedVersion);
|
||||
|
||||
public sealed record VexObservationLinksetResponse(
|
||||
[property: System.Text.Json.Serialization.JsonPropertyName("aliases")] IReadOnlyList<string> Aliases,
|
||||
[property: System.Text.Json.Serialization.JsonPropertyName("purls")] IReadOnlyList<string> Purls,
|
||||
[property: System.Text.Json.Serialization.JsonPropertyName("cpes")] IReadOnlyList<string> Cpes,
|
||||
[property: System.Text.Json.Serialization.JsonPropertyName("references")] IReadOnlyList<VexObservationReferenceItem> References);
|
||||
|
||||
public sealed record VexObservationReferenceItem(
|
||||
[property: System.Text.Json.Serialization.JsonPropertyName("type")] string Type,
|
||||
[property: System.Text.Json.Serialization.JsonPropertyName("url")] string Url);
|
||||
|
||||
public sealed record VexObservationDetailResponse(
|
||||
[property: System.Text.Json.Serialization.JsonPropertyName("observationId")] string ObservationId,
|
||||
[property: System.Text.Json.Serialization.JsonPropertyName("tenant")] string Tenant,
|
||||
[property: System.Text.Json.Serialization.JsonPropertyName("providerId")] string ProviderId,
|
||||
[property: System.Text.Json.Serialization.JsonPropertyName("streamId")] string StreamId,
|
||||
[property: System.Text.Json.Serialization.JsonPropertyName("upstream")] VexObservationUpstreamResponse Upstream,
|
||||
[property: System.Text.Json.Serialization.JsonPropertyName("content")] VexObservationContentResponse Content,
|
||||
[property: System.Text.Json.Serialization.JsonPropertyName("statements")] IReadOnlyList<VexObservationStatementItem> Statements,
|
||||
[property: System.Text.Json.Serialization.JsonPropertyName("linkset")] VexObservationLinksetResponse Linkset,
|
||||
[property: System.Text.Json.Serialization.JsonPropertyName("createdAt")] DateTimeOffset CreatedAt);
|
||||
@@ -0,0 +1,283 @@
|
||||
|
||||
using Microsoft.AspNetCore.Builder;
|
||||
using Microsoft.AspNetCore.Http;
|
||||
using Microsoft.AspNetCore.Mvc;
|
||||
using Microsoft.Extensions.Options;
|
||||
using StellaOps.Auth.ServerIntegration.Tenancy;
|
||||
using StellaOps.Excititor.Core;
|
||||
using StellaOps.Excititor.Core.Storage;
|
||||
using StellaOps.Excititor.WebService.Contracts;
|
||||
using StellaOps.Excititor.WebService.Security;
|
||||
using StellaOps.Excititor.WebService.Services;
|
||||
using static StellaOps.Localization.T;
|
||||
using System;
|
||||
using System.Collections.Generic;
|
||||
using System.Collections.Immutable;
|
||||
using System.Linq;
|
||||
using System.Threading;
|
||||
using System.Threading.Tasks;
|
||||
|
||||
namespace StellaOps.Excititor.WebService.Endpoints;
|
||||
|
||||
/// <summary>
|
||||
/// Policy-facing VEX lookup endpoints (EXCITITOR-POLICY-20-001).
|
||||
/// Aggregation-only: returns raw observations/statements without consensus or severity.
|
||||
/// </summary>
|
||||
public static class PolicyEndpoints
|
||||
{
|
||||
public static void MapPolicyEndpoints(this WebApplication app)
|
||||
{
|
||||
app.MapPost("/policy/v1/vex/lookup", LookupVexAsync)
|
||||
.WithName("Policy_VexLookup")
|
||||
.WithDescription("Performs a batch VEX status lookup by advisory key and product PURL for policy evaluation. Returns raw observations and statements aggregated across configured providers without consensus or severity derivation. Results are ordered deterministically for replay compatibility.")
|
||||
.RequireAuthorization(ExcititorPolicies.VexRead)
|
||||
.RequireTenant();
|
||||
}
|
||||
|
||||
private static async Task<IResult> LookupVexAsync(
|
||||
HttpContext context,
|
||||
[FromBody] PolicyVexLookupRequest request,
|
||||
IOptions<VexStorageOptions> storageOptions,
|
||||
[FromServices] IGraphOverlayStore overlayStore,
|
||||
[FromServices] IVexClaimStore? claimStore,
|
||||
TimeProvider timeProvider,
|
||||
CancellationToken cancellationToken)
|
||||
{
|
||||
// AuthN/Z
|
||||
var scopeResult = ScopeAuthorization.RequireScope(context, "vex.read");
|
||||
if (scopeResult is not null)
|
||||
{
|
||||
return scopeResult;
|
||||
}
|
||||
|
||||
if (!TryResolveTenant(context, storageOptions.Value, out var tenant, out var tenantError))
|
||||
{
|
||||
return tenantError!;
|
||||
}
|
||||
|
||||
// Validate input
|
||||
if ((request.AdvisoryKeys.Count == 0) && (request.Purls.Count == 0))
|
||||
{
|
||||
return Results.BadRequest(new { error = new { code = "ERR_REQUEST", message = _t("excititor.validation.advisory_keys_or_purls_required") } });
|
||||
}
|
||||
|
||||
var advisories = request.AdvisoryKeys
|
||||
.Where(a => !string.IsNullOrWhiteSpace(a))
|
||||
.Select(a => a.Trim())
|
||||
.ToList();
|
||||
|
||||
var purls = request.Purls
|
||||
.Where(p => !string.IsNullOrWhiteSpace(p))
|
||||
.Select(p => p.Trim())
|
||||
.ToList();
|
||||
|
||||
var statusFilter = request.Statuses
|
||||
.Where(s => !string.IsNullOrWhiteSpace(s))
|
||||
.Select(s => s.Trim().ToLowerInvariant())
|
||||
.ToImmutableHashSet();
|
||||
|
||||
var providerFilter = request.Providers
|
||||
.Where(p => !string.IsNullOrWhiteSpace(p))
|
||||
.Select(p => p.Trim())
|
||||
.ToImmutableHashSet(StringComparer.OrdinalIgnoreCase);
|
||||
|
||||
var overlays = await ResolveOverlaysAsync(overlayStore, tenant!, advisories, purls, request.Limit, cancellationToken).ConfigureAwait(false);
|
||||
|
||||
var filtered = overlays
|
||||
.Where(o => MatchesProvider(providerFilter, o))
|
||||
.Where(o => MatchesStatus(statusFilter, o))
|
||||
.OrderBy(o => o.AdvisoryId, StringComparer.OrdinalIgnoreCase)
|
||||
.ThenBy(o => o.Purl, StringComparer.OrdinalIgnoreCase)
|
||||
.ThenBy(o => o.Source, StringComparer.OrdinalIgnoreCase)
|
||||
.Take(Math.Clamp(request.Limit, 1, 500))
|
||||
.ToList();
|
||||
|
||||
if (filtered.Count > 0)
|
||||
{
|
||||
var grouped = filtered
|
||||
.GroupBy(o => o.AdvisoryId, StringComparer.OrdinalIgnoreCase)
|
||||
.Select(group => new PolicyVexLookupItem(
|
||||
group.Key,
|
||||
new[] { group.Key },
|
||||
group.Select(MapStatement).ToList()))
|
||||
.ToList();
|
||||
|
||||
var response = new PolicyVexLookupResponse(grouped, filtered.Count, timeProvider.GetUtcNow());
|
||||
return Results.Ok(response);
|
||||
}
|
||||
|
||||
if (claimStore is null)
|
||||
{
|
||||
return Results.Ok(new PolicyVexLookupResponse(Array.Empty<PolicyVexLookupItem>(), 0, timeProvider.GetUtcNow()));
|
||||
}
|
||||
|
||||
var claimResults = await FallbackClaimsAsync(claimStore, advisories, purls, providerFilter, statusFilter, request.Limit, cancellationToken).ConfigureAwait(false);
|
||||
var totalStatements = claimResults.Sum(item => item.Statements.Count);
|
||||
return Results.Ok(new PolicyVexLookupResponse(claimResults, totalStatements, timeProvider.GetUtcNow()));
|
||||
}
|
||||
|
||||
private static async Task<IReadOnlyList<GraphOverlayItem>> ResolveOverlaysAsync(
|
||||
IGraphOverlayStore overlayStore,
|
||||
string tenant,
|
||||
IReadOnlyList<string> advisories,
|
||||
IReadOnlyList<string> purls,
|
||||
int limit,
|
||||
CancellationToken cancellationToken)
|
||||
{
|
||||
if (purls.Count > 0)
|
||||
{
|
||||
var overlays = await overlayStore.FindByPurlsAsync(tenant, purls, cancellationToken).ConfigureAwait(false);
|
||||
if (advisories.Count == 0)
|
||||
{
|
||||
return overlays;
|
||||
}
|
||||
|
||||
return overlays.Where(o => advisories.Contains(o.AdvisoryId, StringComparer.OrdinalIgnoreCase)).ToList();
|
||||
}
|
||||
|
||||
return await overlayStore.FindByAdvisoriesAsync(tenant, advisories, limit, cancellationToken).ConfigureAwait(false);
|
||||
}
|
||||
|
||||
private static bool MatchesProvider(ISet<string> providers, GraphOverlayItem overlay)
|
||||
=> providers.Count == 0 || providers.Contains(overlay.Source, StringComparer.OrdinalIgnoreCase);
|
||||
|
||||
private static bool MatchesStatus(ISet<string> statuses, GraphOverlayItem overlay)
|
||||
=> statuses.Count == 0 || statuses.Contains(overlay.Status, StringComparer.OrdinalIgnoreCase);
|
||||
|
||||
private static PolicyVexStatement MapStatement(GraphOverlayItem overlay)
|
||||
{
|
||||
var firstSeen = overlay.Observations.Count == 0
|
||||
? overlay.GeneratedAt
|
||||
: overlay.Observations.Min(o => o.FetchedAt);
|
||||
|
||||
var lastSeen = overlay.Observations.Count == 0
|
||||
? overlay.GeneratedAt
|
||||
: overlay.Observations.Max(o => o.FetchedAt);
|
||||
|
||||
var metadata = new Dictionary<string, string>(StringComparer.OrdinalIgnoreCase)
|
||||
{
|
||||
["schemaVersion"] = overlay.SchemaVersion,
|
||||
["linksetId"] = overlay.Provenance.LinksetId,
|
||||
["linksetHash"] = overlay.Provenance.LinksetHash,
|
||||
["source"] = overlay.Source
|
||||
};
|
||||
|
||||
if (!string.IsNullOrWhiteSpace(overlay.Provenance.PlanCacheKey))
|
||||
{
|
||||
metadata["planCacheKey"] = overlay.Provenance.PlanCacheKey!;
|
||||
}
|
||||
|
||||
var justification = overlay.Justifications.FirstOrDefault();
|
||||
var primaryObservation = overlay.Observations.FirstOrDefault();
|
||||
|
||||
return new PolicyVexStatement(
|
||||
ObservationId: primaryObservation?.Id ?? $"{overlay.Source}:{overlay.AdvisoryId}",
|
||||
ProviderId: overlay.Source,
|
||||
Status: overlay.Status,
|
||||
ProductKey: overlay.Purl,
|
||||
Purl: overlay.Purl,
|
||||
Cpe: null,
|
||||
Version: null,
|
||||
Justification: justification?.Kind,
|
||||
Detail: justification?.Reason,
|
||||
FirstSeen: firstSeen,
|
||||
LastSeen: lastSeen,
|
||||
Signature: null,
|
||||
Metadata: metadata);
|
||||
}
|
||||
|
||||
private static async Task<List<PolicyVexLookupItem>> FallbackClaimsAsync(
|
||||
IVexClaimStore claimStore,
|
||||
IReadOnlyList<string> advisories,
|
||||
IReadOnlyList<string> purls,
|
||||
ISet<string> providers,
|
||||
ISet<string> statuses,
|
||||
int limit,
|
||||
CancellationToken cancellationToken)
|
||||
{
|
||||
var results = new List<PolicyVexLookupItem>();
|
||||
foreach (var advisory in advisories)
|
||||
{
|
||||
var claims = await claimStore.FindByVulnerabilityAsync(advisory, limit, cancellationToken).ConfigureAwait(false);
|
||||
|
||||
var filtered = claims
|
||||
.Where(c => providers.Count == 0 || providers.Contains(c.ProviderId, StringComparer.OrdinalIgnoreCase))
|
||||
.Where(c => statuses.Count == 0 || statuses.Contains(c.Status.ToString().ToLowerInvariant()))
|
||||
.Where(c => purls.Count == 0
|
||||
|| purls.Contains(c.Product.Key, StringComparer.OrdinalIgnoreCase)
|
||||
|| (!string.IsNullOrWhiteSpace(c.Product.Purl) && purls.Contains(c.Product.Purl, StringComparer.OrdinalIgnoreCase)))
|
||||
.OrderByDescending(c => c.LastSeen)
|
||||
.ThenBy(c => c.ProviderId, StringComparer.Ordinal)
|
||||
.Take(limit)
|
||||
.Select(MapClaimStatement)
|
||||
.ToList();
|
||||
|
||||
if (filtered.Count > 0)
|
||||
{
|
||||
results.Add(new PolicyVexLookupItem(advisory, new[] { advisory }, filtered));
|
||||
}
|
||||
}
|
||||
|
||||
return results;
|
||||
}
|
||||
|
||||
private static PolicyVexStatement MapClaimStatement(VexClaim claim)
|
||||
{
|
||||
var observationId = $"{claim.ProviderId}:{claim.Document.Digest}";
|
||||
var metadata = new Dictionary<string, string>(StringComparer.OrdinalIgnoreCase)
|
||||
{
|
||||
["document_digest"] = claim.Document.Digest,
|
||||
["document_uri"] = claim.Document.SourceUri.ToString()
|
||||
};
|
||||
|
||||
if (!string.IsNullOrWhiteSpace(claim.Document.Revision))
|
||||
{
|
||||
metadata["document_revision"] = claim.Document.Revision!;
|
||||
}
|
||||
|
||||
return new PolicyVexStatement(
|
||||
ObservationId: observationId,
|
||||
ProviderId: claim.ProviderId,
|
||||
Status: claim.Status.ToString(),
|
||||
ProductKey: claim.Product.Key,
|
||||
Purl: claim.Product.Purl,
|
||||
Cpe: claim.Product.Cpe,
|
||||
Version: claim.Product.Version,
|
||||
Justification: claim.Justification?.ToString(),
|
||||
Detail: claim.Detail,
|
||||
FirstSeen: claim.FirstSeen,
|
||||
LastSeen: claim.LastSeen,
|
||||
Signature: claim.Document.Signature,
|
||||
Metadata: metadata);
|
||||
}
|
||||
|
||||
private static bool TryResolveTenant(
|
||||
HttpContext context,
|
||||
VexStorageOptions options,
|
||||
out string tenant,
|
||||
out IResult? problem)
|
||||
{
|
||||
problem = null;
|
||||
tenant = string.Empty;
|
||||
|
||||
var headerTenant = context.Request.Headers["X-Stella-Tenant"].FirstOrDefault();
|
||||
if (!string.IsNullOrWhiteSpace(headerTenant))
|
||||
{
|
||||
tenant = headerTenant.Trim().ToLowerInvariant();
|
||||
}
|
||||
else if (!string.IsNullOrWhiteSpace(options.DefaultTenant))
|
||||
{
|
||||
tenant = options.DefaultTenant.Trim().ToLowerInvariant();
|
||||
}
|
||||
else
|
||||
{
|
||||
problem = Results.BadRequest(new
|
||||
{
|
||||
error = new { code = "ERR_TENANT", message = "X-Stella-Tenant header is required" }
|
||||
});
|
||||
return false;
|
||||
}
|
||||
|
||||
return true;
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,325 @@
|
||||
// -----------------------------------------------------------------------------
|
||||
// RekorAttestationEndpoints.cs
|
||||
// Sprint: SPRINT_20260117_002_EXCITITOR_vex_rekor_linkage
|
||||
// Task: VRL-007 - Create API endpoints for VEX-Rekor attestation management
|
||||
// Description: REST API endpoints for VEX observation attestation to Rekor
|
||||
// -----------------------------------------------------------------------------
|
||||
|
||||
|
||||
using Microsoft.AspNetCore.Builder;
|
||||
using Microsoft.AspNetCore.Http;
|
||||
using Microsoft.AspNetCore.Mvc;
|
||||
using Microsoft.Extensions.Options;
|
||||
using static Program;
|
||||
using StellaOps.Auth.ServerIntegration.Tenancy;
|
||||
using StellaOps.Excititor.Core.Observations;
|
||||
using StellaOps.Excititor.Core.Storage;
|
||||
using StellaOps.Excititor.WebService.Security;
|
||||
using StellaOps.Excititor.WebService.Services;
|
||||
using System.Text.Json.Serialization;
|
||||
using static StellaOps.Localization.T;
|
||||
|
||||
namespace StellaOps.Excititor.WebService.Endpoints;
|
||||
|
||||
/// <summary>
|
||||
/// API endpoints for managing VEX observation attestation to Rekor transparency log.
|
||||
/// </summary>
|
||||
public static class RekorAttestationEndpoints
|
||||
{
|
||||
public static void MapRekorAttestationEndpoints(this WebApplication app)
|
||||
{
|
||||
var group = app.MapGroup("/attestations/rekor")
|
||||
.WithTags("Rekor Attestation")
|
||||
.RequireAuthorization(ExcititorPolicies.VexRead)
|
||||
.RequireTenant();
|
||||
|
||||
// POST /attestations/rekor/observations/{observationId}
|
||||
// Attest a single observation to Rekor
|
||||
group.MapPost("/observations/{observationId}", async (
|
||||
HttpContext context,
|
||||
string observationId,
|
||||
[FromBody] AttestObservationRequest? request,
|
||||
IOptions<VexStorageOptions> storageOptions,
|
||||
[FromServices] IVexObservationAttestationService? attestationService,
|
||||
CancellationToken cancellationToken) =>
|
||||
{
|
||||
var scopeResult = ScopeAuthorization.RequireScope(context, "vex.attest");
|
||||
if (scopeResult is not null)
|
||||
{
|
||||
return scopeResult;
|
||||
}
|
||||
|
||||
if (!TryResolveTenant(context, storageOptions.Value, requireHeader: false, out var tenant, out var tenantError))
|
||||
{
|
||||
return tenantError;
|
||||
}
|
||||
|
||||
if (attestationService is null)
|
||||
{
|
||||
return Results.Problem(
|
||||
detail: _t("excititor.error.attestation_service_unavailable"),
|
||||
statusCode: StatusCodes.Status503ServiceUnavailable,
|
||||
title: "Service unavailable");
|
||||
}
|
||||
|
||||
if (string.IsNullOrWhiteSpace(observationId))
|
||||
{
|
||||
return Results.Problem(
|
||||
detail: _t("excititor.validation.observation_id_required"),
|
||||
statusCode: StatusCodes.Status400BadRequest,
|
||||
title: "Validation error");
|
||||
}
|
||||
|
||||
var options = new VexAttestationOptions
|
||||
{
|
||||
SubmitToRekor = true,
|
||||
RekorUrl = request?.RekorUrl,
|
||||
StoreInclusionProof = request?.StoreInclusionProof ?? true,
|
||||
SigningKeyId = request?.SigningKeyId,
|
||||
TraceId = context.TraceIdentifier
|
||||
};
|
||||
|
||||
// TODO: In real implementation, we'd fetch the observation first and pass it
|
||||
// For now, we use the simpler VerifyLinkageAsync which takes observationId
|
||||
var result = await attestationService.VerifyLinkageAsync(observationId, cancellationToken);
|
||||
|
||||
if (!result.IsValid)
|
||||
{
|
||||
return Results.Problem(
|
||||
detail: result.Message,
|
||||
statusCode: result.Status switch
|
||||
{
|
||||
RekorLinkageVerificationStatus.NoLinkage => StatusCodes.Status404NotFound,
|
||||
RekorLinkageVerificationStatus.EntryNotFound => StatusCodes.Status404NotFound,
|
||||
_ => StatusCodes.Status500InternalServerError
|
||||
},
|
||||
title: "Verification failed");
|
||||
}
|
||||
|
||||
var response = new AttestObservationResponse(
|
||||
observationId,
|
||||
result.Linkage!.Uuid,
|
||||
result.Linkage.LogIndex,
|
||||
result.Linkage.IntegratedTime,
|
||||
null);
|
||||
|
||||
return Results.Ok(response);
|
||||
})
|
||||
.WithName("AttestObservationToRekor")
|
||||
.WithDescription("Attests a single VEX observation to the Rekor transparency log, linking the observation to an immutable log entry. Returns the Rekor entry UUID and log index for inclusion proof tracking.")
|
||||
.RequireAuthorization(ExcititorPolicies.VexAttest);
|
||||
|
||||
// POST /attestations/rekor/observations/batch
|
||||
// Attest multiple observations to Rekor
|
||||
group.MapPost("/observations/batch", async (
|
||||
HttpContext context,
|
||||
[FromBody] BatchAttestRequest request,
|
||||
IOptions<VexStorageOptions> storageOptions,
|
||||
[FromServices] IVexObservationAttestationService? attestationService,
|
||||
CancellationToken cancellationToken) =>
|
||||
{
|
||||
var scopeResult = ScopeAuthorization.RequireScope(context, "vex.attest");
|
||||
if (scopeResult is not null)
|
||||
{
|
||||
return scopeResult;
|
||||
}
|
||||
|
||||
if (!TryResolveTenant(context, storageOptions.Value, requireHeader: false, out var tenant, out var tenantError))
|
||||
{
|
||||
return tenantError;
|
||||
}
|
||||
|
||||
if (attestationService is null)
|
||||
{
|
||||
return Results.Problem(
|
||||
detail: _t("excititor.error.attestation_service_unavailable"),
|
||||
statusCode: StatusCodes.Status503ServiceUnavailable,
|
||||
title: "Service unavailable");
|
||||
}
|
||||
|
||||
if (request.ObservationIds is null || request.ObservationIds.Count == 0)
|
||||
{
|
||||
return Results.Problem(
|
||||
detail: _t("excititor.validation.observation_ids_required"),
|
||||
statusCode: StatusCodes.Status400BadRequest,
|
||||
title: "Validation error");
|
||||
}
|
||||
|
||||
if (request.ObservationIds.Count > 100)
|
||||
{
|
||||
return Results.Problem(
|
||||
detail: _t("excititor.validation.observation_ids_max"),
|
||||
statusCode: StatusCodes.Status400BadRequest,
|
||||
title: "Validation error");
|
||||
}
|
||||
|
||||
var options = new VexAttestationOptions
|
||||
{
|
||||
SubmitToRekor = true,
|
||||
RekorUrl = request.RekorUrl,
|
||||
StoreInclusionProof = request.StoreInclusionProof ?? true,
|
||||
SigningKeyId = request.SigningKeyId,
|
||||
TraceId = context.TraceIdentifier
|
||||
};
|
||||
|
||||
var results = await attestationService.AttestBatchAsync(
|
||||
request.ObservationIds,
|
||||
options,
|
||||
cancellationToken);
|
||||
|
||||
var items = results.Select(r => new BatchAttestResultItem(
|
||||
r.ObservationId,
|
||||
r.Success,
|
||||
r.RekorLinkage?.Uuid,
|
||||
r.RekorLinkage?.LogIndex,
|
||||
r.ErrorMessage,
|
||||
r.ErrorCode?.ToString()
|
||||
)).ToList();
|
||||
|
||||
var response = new BatchAttestResponse(
|
||||
items.Count(i => i.Success),
|
||||
items.Count(i => !i.Success),
|
||||
items);
|
||||
|
||||
return Results.Ok(response);
|
||||
})
|
||||
.WithName("BatchAttestObservationsToRekor")
|
||||
.WithDescription("Attests up to 100 VEX observations to Rekor in a single batch operation. Returns per-observation success/failure results with Rekor entry IDs. Failed items do not roll back successful ones.")
|
||||
.RequireAuthorization(ExcititorPolicies.VexAttest);
|
||||
|
||||
// GET /attestations/rekor/observations/{observationId}/verify
|
||||
// Verify an observation's Rekor linkage
|
||||
group.MapGet("/observations/{observationId}/verify", async (
|
||||
HttpContext context,
|
||||
string observationId,
|
||||
IOptions<VexStorageOptions> storageOptions,
|
||||
[FromServices] IVexObservationAttestationService? attestationService,
|
||||
CancellationToken cancellationToken) =>
|
||||
{
|
||||
var scopeResult = ScopeAuthorization.RequireScope(context, "vex.read");
|
||||
if (scopeResult is not null)
|
||||
{
|
||||
return scopeResult;
|
||||
}
|
||||
|
||||
if (!TryResolveTenant(context, storageOptions.Value, requireHeader: false, out var tenant, out var tenantError))
|
||||
{
|
||||
return tenantError;
|
||||
}
|
||||
|
||||
if (attestationService is null)
|
||||
{
|
||||
return Results.Problem(
|
||||
detail: _t("excititor.error.attestation_service_unavailable"),
|
||||
statusCode: StatusCodes.Status503ServiceUnavailable,
|
||||
title: "Service unavailable");
|
||||
}
|
||||
|
||||
if (string.IsNullOrWhiteSpace(observationId))
|
||||
{
|
||||
return Results.Problem(
|
||||
detail: _t("excititor.validation.observation_id_required"),
|
||||
statusCode: StatusCodes.Status400BadRequest,
|
||||
title: "Validation error");
|
||||
}
|
||||
|
||||
var result = await attestationService.VerifyLinkageAsync(observationId, cancellationToken);
|
||||
|
||||
var response = new VerifyLinkageResponse(
|
||||
observationId,
|
||||
result.IsValid,
|
||||
result.VerifiedAt,
|
||||
result.Linkage?.Uuid,
|
||||
result.Linkage?.LogIndex,
|
||||
result.Message);
|
||||
|
||||
return Results.Ok(response);
|
||||
})
|
||||
.WithName("VerifyObservationRekorLinkage")
|
||||
.WithDescription("Verifies that a VEX observation has a valid Rekor transparency log entry, confirming the inclusion proof and log index. Returns verification status and the linked entry details.");
|
||||
|
||||
// GET /attestations/rekor/pending
|
||||
// Get observations pending attestation
|
||||
group.MapGet("/pending", async (
|
||||
HttpContext context,
|
||||
[FromQuery] int? limit,
|
||||
IOptions<VexStorageOptions> storageOptions,
|
||||
[FromServices] IVexObservationAttestationService? attestationService,
|
||||
CancellationToken cancellationToken) =>
|
||||
{
|
||||
var scopeResult = ScopeAuthorization.RequireScope(context, "vex.read");
|
||||
if (scopeResult is not null)
|
||||
{
|
||||
return scopeResult;
|
||||
}
|
||||
|
||||
if (!TryResolveTenant(context, storageOptions.Value, requireHeader: false, out var tenant, out var tenantError))
|
||||
{
|
||||
return tenantError;
|
||||
}
|
||||
|
||||
if (attestationService is null)
|
||||
{
|
||||
return Results.Problem(
|
||||
detail: _t("excititor.error.attestation_service_unavailable"),
|
||||
statusCode: StatusCodes.Status503ServiceUnavailable,
|
||||
title: "Service unavailable");
|
||||
}
|
||||
|
||||
var pendingIds = await attestationService.GetPendingAttestationsAsync(
|
||||
limit ?? 100,
|
||||
cancellationToken);
|
||||
|
||||
var response = new PendingAttestationsResponse(pendingIds.Count, pendingIds);
|
||||
|
||||
return Results.Ok(response);
|
||||
})
|
||||
.WithName("GetPendingRekorAttestations")
|
||||
.WithDescription("Returns a list of VEX observation IDs that have not yet been submitted to the Rekor transparency log. Used by background workers to drive attestation pipelines.");
|
||||
}
|
||||
}
|
||||
|
||||
// Request DTOs
|
||||
public sealed record AttestObservationRequest(
|
||||
[property: JsonPropertyName("rekorUrl")] string? RekorUrl,
|
||||
[property: JsonPropertyName("storeInclusionProof")] bool? StoreInclusionProof,
|
||||
[property: JsonPropertyName("signingKeyId")] string? SigningKeyId);
|
||||
|
||||
public sealed record BatchAttestRequest(
|
||||
[property: JsonPropertyName("observationIds")] IReadOnlyList<string> ObservationIds,
|
||||
[property: JsonPropertyName("rekorUrl")] string? RekorUrl,
|
||||
[property: JsonPropertyName("storeInclusionProof")] bool? StoreInclusionProof,
|
||||
[property: JsonPropertyName("signingKeyId")] string? SigningKeyId);
|
||||
|
||||
// Response DTOs
|
||||
public sealed record AttestObservationResponse(
|
||||
[property: JsonPropertyName("observationId")] string ObservationId,
|
||||
[property: JsonPropertyName("rekorEntryId")] string RekorEntryId,
|
||||
[property: JsonPropertyName("logIndex")] long LogIndex,
|
||||
[property: JsonPropertyName("integratedTime")] DateTimeOffset IntegratedTime,
|
||||
[property: JsonPropertyName("duration")] TimeSpan? Duration);
|
||||
|
||||
public sealed record BatchAttestResultItem(
|
||||
[property: JsonPropertyName("observationId")] string ObservationId,
|
||||
[property: JsonPropertyName("success")] bool Success,
|
||||
[property: JsonPropertyName("rekorEntryId")] string? RekorEntryId,
|
||||
[property: JsonPropertyName("logIndex")] long? LogIndex,
|
||||
[property: JsonPropertyName("error")] string? Error,
|
||||
[property: JsonPropertyName("errorCode")] string? ErrorCode);
|
||||
|
||||
public sealed record BatchAttestResponse(
|
||||
[property: JsonPropertyName("successCount")] int SuccessCount,
|
||||
[property: JsonPropertyName("failureCount")] int FailureCount,
|
||||
[property: JsonPropertyName("results")] IReadOnlyList<BatchAttestResultItem> Results);
|
||||
|
||||
public sealed record VerifyLinkageResponse(
|
||||
[property: JsonPropertyName("observationId")] string ObservationId,
|
||||
[property: JsonPropertyName("isVerified")] bool IsVerified,
|
||||
[property: JsonPropertyName("verifiedAt")] DateTimeOffset? VerifiedAt,
|
||||
[property: JsonPropertyName("rekorEntryId")] string? RekorEntryId,
|
||||
[property: JsonPropertyName("logIndex")] long? LogIndex,
|
||||
[property: JsonPropertyName("failureReason")] string? FailureReason);
|
||||
|
||||
public sealed record PendingAttestationsResponse(
|
||||
[property: JsonPropertyName("count")] int Count,
|
||||
[property: JsonPropertyName("observationIds")] IReadOnlyList<string> ObservationIds);
|
||||
@@ -0,0 +1,590 @@
|
||||
#pragma warning disable EXCITITOR001 // Consensus logic is deprecated - resolve endpoint uses VexConsensus during transition
|
||||
using Microsoft.AspNetCore.Builder;
|
||||
using Microsoft.AspNetCore.Http;
|
||||
using Microsoft.AspNetCore.Mvc;
|
||||
using Microsoft.Extensions.DependencyInjection;
|
||||
using Microsoft.Extensions.Logging;
|
||||
using StellaOps.Auth.ServerIntegration.Tenancy;
|
||||
using StellaOps.Excititor.Attestation;
|
||||
using StellaOps.Excititor.Attestation.Signing;
|
||||
using StellaOps.Excititor.Core;
|
||||
using StellaOps.Excititor.Core.Dsse;
|
||||
using StellaOps.Excititor.Core.Lattice;
|
||||
using StellaOps.Excititor.Core.Storage;
|
||||
using StellaOps.Excititor.Formats.OpenVEX;
|
||||
using StellaOps.Excititor.Policy;
|
||||
using StellaOps.Excititor.WebService.Security;
|
||||
using StellaOps.Excititor.WebService.Services;
|
||||
using static StellaOps.Localization.T;
|
||||
using System;
|
||||
using System.Collections.Generic;
|
||||
using System.Collections.Immutable;
|
||||
using System.Linq;
|
||||
using System.Security.Cryptography;
|
||||
using System.Text;
|
||||
using System.Text.Json;
|
||||
|
||||
namespace StellaOps.Excititor.WebService.Endpoints;
|
||||
|
||||
|
||||
internal static class ResolveEndpoint
|
||||
{
|
||||
private const int MaxSubjectPairs = 256;
|
||||
private const string ReadScope = "vex.read";
|
||||
|
||||
public static void MapResolveEndpoint(WebApplication app)
|
||||
{
|
||||
app.MapPost("/excititor/resolve", HandleResolveAsync)
|
||||
.WithName("ResolveVexConsensus")
|
||||
.WithDescription("Resolves VEX consensus for one or more vulnerability ID and product key pairs against the active policy snapshot. Applies lattice-based trust weighting, produces a signed DSSE envelope, and optionally persists the consensus result. Requires vex.read scope.")
|
||||
.RequireAuthorization(ExcititorPolicies.VexRead)
|
||||
.RequireTenant();
|
||||
}
|
||||
|
||||
private static async Task<IResult> HandleResolveAsync(
|
||||
VexResolveRequest request,
|
||||
HttpContext httpContext,
|
||||
IVexClaimStore claimStore,
|
||||
[FromServices] IVexConsensusStore? consensusStore,
|
||||
OpenVexStatementMerger merger,
|
||||
IVexLatticeProvider lattice,
|
||||
IVexPolicyProvider policyProvider,
|
||||
TimeProvider timeProvider,
|
||||
ILoggerFactory loggerFactory,
|
||||
IVexAttestationClient? attestationClient,
|
||||
CancellationToken cancellationToken)
|
||||
{
|
||||
var scopeResult = ScopeAuthorization.RequireScope(httpContext, ReadScope);
|
||||
if (scopeResult is not null)
|
||||
{
|
||||
return scopeResult;
|
||||
}
|
||||
|
||||
if (request is null)
|
||||
{
|
||||
return Results.BadRequest(_t("excititor.validation.request_payload_required"));
|
||||
}
|
||||
|
||||
var logger = loggerFactory.CreateLogger("ResolveEndpoint");
|
||||
var signer = httpContext.RequestServices.GetService<IVexSigner>();
|
||||
|
||||
var productKeys = NormalizeValues(request.ProductKeys, request.Purls);
|
||||
var vulnerabilityIds = NormalizeValues(request.VulnerabilityIds);
|
||||
|
||||
if (productKeys.Count == 0)
|
||||
{
|
||||
await WritePlainTextAsync(httpContext, "At least one productKey or purl must be provided.", StatusCodes.Status400BadRequest, cancellationToken);
|
||||
return Results.Empty;
|
||||
}
|
||||
|
||||
if (vulnerabilityIds.Count == 0)
|
||||
{
|
||||
await WritePlainTextAsync(httpContext, "At least one vulnerabilityId must be provided.", StatusCodes.Status400BadRequest, cancellationToken);
|
||||
return Results.Empty;
|
||||
}
|
||||
|
||||
var pairCount = (long)productKeys.Count * vulnerabilityIds.Count;
|
||||
if (pairCount > MaxSubjectPairs)
|
||||
{
|
||||
await WritePlainTextAsync(httpContext, FormattableString.Invariant($"A maximum of {MaxSubjectPairs} subject pairs are allowed per request."), StatusCodes.Status400BadRequest, cancellationToken);
|
||||
return Results.Empty;
|
||||
}
|
||||
|
||||
var snapshot = policyProvider.GetSnapshot();
|
||||
|
||||
if (!string.IsNullOrWhiteSpace(request.PolicyRevisionId) &&
|
||||
!string.Equals(request.PolicyRevisionId.Trim(), snapshot.RevisionId, StringComparison.Ordinal))
|
||||
{
|
||||
var conflictPayload = new
|
||||
{
|
||||
message = $"Requested policy revision '{request.PolicyRevisionId}' does not match active revision '{snapshot.RevisionId}'.",
|
||||
activeRevision = snapshot.RevisionId,
|
||||
requestedRevision = request.PolicyRevisionId,
|
||||
};
|
||||
await WriteJsonAsync(httpContext, conflictPayload, StatusCodes.Status409Conflict, cancellationToken);
|
||||
return Results.Empty;
|
||||
}
|
||||
|
||||
var resolvedAt = timeProvider.GetUtcNow();
|
||||
var results = new List<VexResolveResult>((int)pairCount);
|
||||
|
||||
foreach (var productKey in productKeys)
|
||||
{
|
||||
foreach (var vulnerabilityId in vulnerabilityIds)
|
||||
{
|
||||
var claims = await claimStore.FindAsync(vulnerabilityId, productKey, since: null, cancellationToken)
|
||||
.ConfigureAwait(false);
|
||||
|
||||
var claimArray = claims.Count == 0 ? Array.Empty<VexClaim>() : claims.ToArray();
|
||||
var signals = AggregateSignals(claimArray);
|
||||
var product = ResolveProduct(claimArray, productKey);
|
||||
var (consensus, decisions) = BuildConsensus(
|
||||
vulnerabilityId,
|
||||
claimArray,
|
||||
product,
|
||||
signals,
|
||||
snapshot,
|
||||
merger,
|
||||
lattice,
|
||||
timeProvider);
|
||||
|
||||
if (!string.Equals(consensus.PolicyVersion, snapshot.Version, StringComparison.Ordinal) ||
|
||||
!string.Equals(consensus.PolicyRevisionId, snapshot.RevisionId, StringComparison.Ordinal) ||
|
||||
!string.Equals(consensus.PolicyDigest, snapshot.Digest, StringComparison.Ordinal))
|
||||
{
|
||||
consensus = new VexConsensus(
|
||||
consensus.VulnerabilityId,
|
||||
consensus.Product,
|
||||
consensus.Status,
|
||||
consensus.CalculatedAt,
|
||||
consensus.Sources,
|
||||
consensus.Conflicts,
|
||||
consensus.Signals,
|
||||
snapshot.Version,
|
||||
consensus.Summary,
|
||||
snapshot.RevisionId,
|
||||
snapshot.Digest);
|
||||
}
|
||||
|
||||
if (consensusStore is not null)
|
||||
{
|
||||
await consensusStore.SaveAsync(consensus, cancellationToken).ConfigureAwait(false);
|
||||
}
|
||||
|
||||
var payload = PreparePayload(consensus);
|
||||
var contentSignature = await TrySignAsync(signer, payload, logger, cancellationToken).ConfigureAwait(false);
|
||||
var attestation = await BuildAttestationAsync(
|
||||
attestationClient,
|
||||
consensus,
|
||||
snapshot,
|
||||
payload,
|
||||
logger,
|
||||
cancellationToken).ConfigureAwait(false);
|
||||
|
||||
results.Add(new VexResolveResult(
|
||||
consensus.VulnerabilityId,
|
||||
consensus.Product.Key,
|
||||
consensus.Status,
|
||||
consensus.CalculatedAt,
|
||||
consensus.Sources,
|
||||
consensus.Conflicts,
|
||||
consensus.Signals,
|
||||
consensus.Summary,
|
||||
consensus.PolicyRevisionId ?? snapshot.RevisionId,
|
||||
consensus.PolicyVersion ?? snapshot.Version,
|
||||
consensus.PolicyDigest ?? snapshot.Digest,
|
||||
decisions,
|
||||
new VexResolveEnvelope(
|
||||
payload.Artifact,
|
||||
contentSignature,
|
||||
attestation.Metadata,
|
||||
attestation.Envelope,
|
||||
attestation.Signature)));
|
||||
}
|
||||
}
|
||||
|
||||
var policy = new VexResolvePolicy(
|
||||
snapshot.RevisionId,
|
||||
snapshot.Version,
|
||||
snapshot.Digest,
|
||||
request.PolicyRevisionId?.Trim());
|
||||
|
||||
var response = new VexResolveResponse(resolvedAt, policy, results);
|
||||
await WriteJsonAsync(httpContext, response, StatusCodes.Status200OK, cancellationToken);
|
||||
return Results.Empty;
|
||||
}
|
||||
|
||||
private static List<string> NormalizeValues(params IReadOnlyList<string>?[] sources)
|
||||
{
|
||||
var result = new List<string>();
|
||||
var seen = new HashSet<string>(StringComparer.Ordinal);
|
||||
|
||||
foreach (var source in sources)
|
||||
{
|
||||
if (source is null)
|
||||
{
|
||||
continue;
|
||||
}
|
||||
|
||||
foreach (var value in source)
|
||||
{
|
||||
if (string.IsNullOrWhiteSpace(value))
|
||||
{
|
||||
continue;
|
||||
}
|
||||
|
||||
var normalized = value.Trim();
|
||||
if (seen.Add(normalized))
|
||||
{
|
||||
result.Add(normalized);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
return result;
|
||||
}
|
||||
|
||||
private static VexSignalSnapshot? AggregateSignals(IReadOnlyList<VexClaim> claims)
|
||||
{
|
||||
if (claims.Count == 0)
|
||||
{
|
||||
return null;
|
||||
}
|
||||
|
||||
VexSeveritySignal? bestSeverity = null;
|
||||
double? bestScore = null;
|
||||
bool kevPresent = false;
|
||||
bool kevTrue = false;
|
||||
double? bestEpss = null;
|
||||
|
||||
foreach (var claim in claims)
|
||||
{
|
||||
if (claim.Signals is null)
|
||||
{
|
||||
continue;
|
||||
}
|
||||
|
||||
var severity = claim.Signals.Severity;
|
||||
if (severity is not null)
|
||||
{
|
||||
var score = severity.Score;
|
||||
if (bestSeverity is null ||
|
||||
(score is not null && (bestScore is null || score.Value > bestScore.Value)) ||
|
||||
(score is null && bestScore is null && !string.IsNullOrWhiteSpace(severity.Label) && string.IsNullOrWhiteSpace(bestSeverity.Label)))
|
||||
{
|
||||
bestSeverity = severity;
|
||||
bestScore = severity.Score;
|
||||
}
|
||||
}
|
||||
|
||||
if (claim.Signals.Kev is { } kevValue)
|
||||
{
|
||||
kevPresent = true;
|
||||
if (kevValue)
|
||||
{
|
||||
kevTrue = true;
|
||||
}
|
||||
}
|
||||
|
||||
if (claim.Signals.Epss is { } epss)
|
||||
{
|
||||
if (bestEpss is null || epss > bestEpss.Value)
|
||||
{
|
||||
bestEpss = epss;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
if (bestSeverity is null && !kevPresent && bestEpss is null)
|
||||
{
|
||||
return null;
|
||||
}
|
||||
|
||||
bool? kev = kevTrue ? true : (kevPresent ? false : null);
|
||||
return new VexSignalSnapshot(bestSeverity, kev, bestEpss);
|
||||
}
|
||||
|
||||
private static VexProduct ResolveProduct(IReadOnlyList<VexClaim> claims, string productKey)
|
||||
{
|
||||
if (claims.Count > 0)
|
||||
{
|
||||
return claims[0].Product;
|
||||
}
|
||||
|
||||
var inferredPurl = productKey.StartsWith("pkg:", StringComparison.OrdinalIgnoreCase) ? productKey : null;
|
||||
return new VexProduct(productKey, name: null, version: null, purl: inferredPurl);
|
||||
}
|
||||
|
||||
private static (VexConsensus Consensus, IReadOnlyList<VexConsensusDecisionTelemetry> Decisions) BuildConsensus(
|
||||
string vulnerabilityId,
|
||||
IReadOnlyList<VexClaim> claims,
|
||||
VexProduct product,
|
||||
VexSignalSnapshot? signals,
|
||||
VexPolicySnapshot snapshot,
|
||||
OpenVexStatementMerger merger,
|
||||
IVexLatticeProvider lattice,
|
||||
TimeProvider timeProvider)
|
||||
{
|
||||
var calculatedAt = timeProvider.GetUtcNow();
|
||||
|
||||
if (claims.Count == 0)
|
||||
{
|
||||
var emptyConsensus = new VexConsensus(
|
||||
vulnerabilityId,
|
||||
product,
|
||||
VexConsensusStatus.UnderInvestigation,
|
||||
calculatedAt,
|
||||
Array.Empty<VexConsensusSource>(),
|
||||
Array.Empty<VexConsensusConflict>(),
|
||||
signals,
|
||||
snapshot.Version,
|
||||
"No claims available.",
|
||||
snapshot.RevisionId,
|
||||
snapshot.Digest);
|
||||
|
||||
return (emptyConsensus, Array.Empty<VexConsensusDecisionTelemetry>());
|
||||
}
|
||||
|
||||
var mergeResult = merger.MergeClaims(claims);
|
||||
var consensusStatus = MapConsensusStatus(mergeResult.ResultStatement.Status);
|
||||
var sources = claims
|
||||
.Select(claim => new VexConsensusSource(
|
||||
claim.ProviderId,
|
||||
claim.Status,
|
||||
claim.Document.Digest,
|
||||
(double)lattice.GetTrustWeight(claim),
|
||||
claim.Justification,
|
||||
claim.Detail,
|
||||
claim.Confidence))
|
||||
.ToArray();
|
||||
|
||||
var conflicts = claims
|
||||
.Where(claim => claim.Status != mergeResult.ResultStatement.Status)
|
||||
.Select(claim => new VexConsensusConflict(
|
||||
claim.ProviderId,
|
||||
claim.Status,
|
||||
claim.Document.Digest,
|
||||
claim.Justification,
|
||||
claim.Detail,
|
||||
"status_conflict"))
|
||||
.ToArray();
|
||||
|
||||
var summary = MergeTraceWriter.ToExplanation(mergeResult);
|
||||
var decisions = BuildDecisionLog(claims, lattice);
|
||||
|
||||
var consensus = new VexConsensus(
|
||||
vulnerabilityId,
|
||||
product,
|
||||
consensusStatus,
|
||||
calculatedAt,
|
||||
sources,
|
||||
conflicts,
|
||||
signals,
|
||||
snapshot.Version,
|
||||
summary,
|
||||
snapshot.RevisionId,
|
||||
snapshot.Digest);
|
||||
|
||||
return (consensus, decisions);
|
||||
}
|
||||
|
||||
private static VexConsensusStatus MapConsensusStatus(VexClaimStatus status)
|
||||
=> status switch
|
||||
{
|
||||
VexClaimStatus.Affected => VexConsensusStatus.Affected,
|
||||
VexClaimStatus.NotAffected => VexConsensusStatus.NotAffected,
|
||||
VexClaimStatus.Fixed => VexConsensusStatus.Fixed,
|
||||
_ => VexConsensusStatus.UnderInvestigation,
|
||||
};
|
||||
|
||||
private static IReadOnlyList<VexConsensusDecisionTelemetry> BuildDecisionLog(
|
||||
IReadOnlyList<VexClaim> claims,
|
||||
IVexLatticeProvider lattice)
|
||||
{
|
||||
if (claims.Count == 0)
|
||||
{
|
||||
return Array.Empty<VexConsensusDecisionTelemetry>();
|
||||
}
|
||||
|
||||
var decisions = new List<VexConsensusDecisionTelemetry>(claims.Count);
|
||||
foreach (var claim in claims)
|
||||
{
|
||||
var weight = lattice.GetTrustWeight(claim);
|
||||
var included = weight > 0;
|
||||
var reason = included ? null : "weight_not_positive";
|
||||
|
||||
decisions.Add(new VexConsensusDecisionTelemetry(
|
||||
claim.ProviderId,
|
||||
claim.Document.Digest,
|
||||
claim.Status,
|
||||
included,
|
||||
(double)weight,
|
||||
reason,
|
||||
claim.Justification,
|
||||
claim.Detail));
|
||||
}
|
||||
|
||||
return decisions;
|
||||
}
|
||||
|
||||
private static ConsensusPayload PreparePayload(VexConsensus consensus)
|
||||
{
|
||||
var canonicalJson = VexCanonicalJsonSerializer.Serialize(consensus);
|
||||
var bytes = Encoding.UTF8.GetBytes(canonicalJson);
|
||||
var digest = SHA256.HashData(bytes);
|
||||
var digestHex = Convert.ToHexString(digest).ToLowerInvariant();
|
||||
var address = new VexContentAddress("sha256", digestHex);
|
||||
return new ConsensusPayload(address, bytes, canonicalJson);
|
||||
}
|
||||
|
||||
private static async ValueTask<ResolveSignature?> TrySignAsync(
|
||||
IVexSigner? signer,
|
||||
ConsensusPayload payload,
|
||||
ILogger logger,
|
||||
CancellationToken cancellationToken)
|
||||
{
|
||||
if (signer is null)
|
||||
{
|
||||
return null;
|
||||
}
|
||||
|
||||
try
|
||||
{
|
||||
var signature = await signer.SignAsync(payload.Bytes, cancellationToken).ConfigureAwait(false);
|
||||
return new ResolveSignature(signature.Signature, signature.KeyId);
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
logger.LogWarning(ex, "Failed to sign resolve payload {Digest}", payload.Artifact.ToUri());
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
private static async ValueTask<ResolveAttestation> BuildAttestationAsync(
|
||||
IVexAttestationClient? attestationClient,
|
||||
VexConsensus consensus,
|
||||
VexPolicySnapshot snapshot,
|
||||
ConsensusPayload payload,
|
||||
ILogger logger,
|
||||
CancellationToken cancellationToken)
|
||||
{
|
||||
if (attestationClient is null)
|
||||
{
|
||||
return new ResolveAttestation(null, null, null);
|
||||
}
|
||||
|
||||
try
|
||||
{
|
||||
var exportId = BuildAttestationExportId(consensus.VulnerabilityId, consensus.Product.Key);
|
||||
var filters = new[]
|
||||
{
|
||||
new KeyValuePair<string, string>("vulnerabilityId", consensus.VulnerabilityId),
|
||||
new KeyValuePair<string, string>("productKey", consensus.Product.Key),
|
||||
new KeyValuePair<string, string>("policyRevisionId", snapshot.RevisionId),
|
||||
};
|
||||
|
||||
var querySignature = VexQuerySignature.FromFilters(filters);
|
||||
var providerIds = consensus.Sources
|
||||
.Select(source => source.ProviderId)
|
||||
.Distinct(StringComparer.Ordinal)
|
||||
.ToImmutableArray();
|
||||
|
||||
var metadataBuilder = ImmutableDictionary.CreateBuilder<string, string>(StringComparer.Ordinal);
|
||||
metadataBuilder["consensusDigest"] = payload.Artifact.ToUri();
|
||||
metadataBuilder["policyRevisionId"] = snapshot.RevisionId;
|
||||
metadataBuilder["policyVersion"] = snapshot.Version;
|
||||
if (!string.IsNullOrWhiteSpace(snapshot.Digest))
|
||||
{
|
||||
metadataBuilder["policyDigest"] = snapshot.Digest;
|
||||
}
|
||||
|
||||
var response = await attestationClient.SignAsync(new VexAttestationRequest(
|
||||
exportId,
|
||||
querySignature,
|
||||
payload.Artifact,
|
||||
VexExportFormat.Json,
|
||||
consensus.CalculatedAt,
|
||||
providerIds,
|
||||
metadataBuilder.ToImmutable()), cancellationToken).ConfigureAwait(false);
|
||||
|
||||
var envelopeJson = response.Diagnostics.TryGetValue("envelope", out var envelopeValue)
|
||||
? envelopeValue
|
||||
: null;
|
||||
|
||||
ResolveSignature? signature = null;
|
||||
if (!string.IsNullOrWhiteSpace(envelopeJson))
|
||||
{
|
||||
try
|
||||
{
|
||||
var envelope = JsonSerializer.Deserialize<DsseEnvelope>(envelopeJson);
|
||||
var dsseSignature = envelope?.Signatures?.FirstOrDefault();
|
||||
if (dsseSignature is not null)
|
||||
{
|
||||
signature = new ResolveSignature(dsseSignature.Signature, dsseSignature.KeyId);
|
||||
}
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
logger.LogDebug(ex, "Failed to deserialize DSSE envelope for resolve export {ExportId}", exportId);
|
||||
}
|
||||
}
|
||||
|
||||
return new ResolveAttestation(response.Attestation, envelopeJson, signature);
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
logger.LogWarning(ex, "Unable to produce attestation for {VulnerabilityId}/{ProductKey}", consensus.VulnerabilityId, consensus.Product.Key);
|
||||
return new ResolveAttestation(null, null, null);
|
||||
}
|
||||
}
|
||||
|
||||
private static string BuildAttestationExportId(string vulnerabilityId, string productKey)
|
||||
{
|
||||
var hash = SHA256.HashData(Encoding.UTF8.GetBytes(productKey));
|
||||
var digest = Convert.ToHexString(hash).ToLowerInvariant();
|
||||
return FormattableString.Invariant($"resolve/{vulnerabilityId}/{digest}");
|
||||
}
|
||||
|
||||
private sealed record ConsensusPayload(VexContentAddress Artifact, byte[] Bytes, string CanonicalJson);
|
||||
|
||||
private static async Task WritePlainTextAsync(HttpContext context, string message, int statusCode, CancellationToken cancellationToken)
|
||||
{
|
||||
context.Response.StatusCode = statusCode;
|
||||
context.Response.ContentType = "text/plain";
|
||||
await context.Response.WriteAsync(message, cancellationToken);
|
||||
}
|
||||
|
||||
private static async Task WriteJsonAsync<T>(HttpContext context, T payload, int statusCode, CancellationToken cancellationToken)
|
||||
{
|
||||
context.Response.StatusCode = statusCode;
|
||||
context.Response.ContentType = "application/json";
|
||||
var json = VexCanonicalJsonSerializer.Serialize(payload);
|
||||
await context.Response.WriteAsync(json, cancellationToken);
|
||||
}
|
||||
}
|
||||
|
||||
public sealed record VexResolveRequest(
|
||||
IReadOnlyList<string>? ProductKeys,
|
||||
IReadOnlyList<string>? Purls,
|
||||
IReadOnlyList<string>? VulnerabilityIds,
|
||||
string? PolicyRevisionId);
|
||||
|
||||
internal sealed record VexResolvePolicy(
|
||||
string ActiveRevisionId,
|
||||
string Version,
|
||||
string Digest,
|
||||
string? RequestedRevisionId);
|
||||
|
||||
internal sealed record VexResolveResponse(
|
||||
DateTimeOffset ResolvedAt,
|
||||
VexResolvePolicy Policy,
|
||||
IReadOnlyList<VexResolveResult> Results);
|
||||
|
||||
internal sealed record VexResolveResult(
|
||||
string VulnerabilityId,
|
||||
string ProductKey,
|
||||
VexConsensusStatus Status,
|
||||
DateTimeOffset CalculatedAt,
|
||||
IReadOnlyList<VexConsensusSource> Sources,
|
||||
IReadOnlyList<VexConsensusConflict> Conflicts,
|
||||
VexSignalSnapshot? Signals,
|
||||
string? Summary,
|
||||
string PolicyRevisionId,
|
||||
string PolicyVersion,
|
||||
string PolicyDigest,
|
||||
IReadOnlyList<VexConsensusDecisionTelemetry> Decisions,
|
||||
VexResolveEnvelope Envelope);
|
||||
|
||||
internal sealed record VexResolveEnvelope(
|
||||
VexContentAddress Artifact,
|
||||
ResolveSignature? ContentSignature,
|
||||
VexAttestationMetadata? Attestation,
|
||||
string? AttestationEnvelope,
|
||||
ResolveSignature? AttestationSignature);
|
||||
|
||||
internal sealed record ResolveSignature(string Value, string? KeyId);
|
||||
|
||||
internal sealed record ResolveAttestation(
|
||||
VexAttestationMetadata? Metadata,
|
||||
string? Envelope,
|
||||
ResolveSignature? Signature);
|
||||
@@ -0,0 +1,316 @@
|
||||
|
||||
using Microsoft.AspNetCore.Builder;
|
||||
using Microsoft.AspNetCore.Http;
|
||||
using Microsoft.AspNetCore.Mvc;
|
||||
using Microsoft.Extensions.Options;
|
||||
using StellaOps.Auth.ServerIntegration.Tenancy;
|
||||
using StellaOps.Excititor.Core;
|
||||
using StellaOps.Excititor.Core.RiskFeed;
|
||||
using StellaOps.Excititor.Core.Storage;
|
||||
using StellaOps.Excititor.WebService.Security;
|
||||
using StellaOps.Excititor.WebService.Services;
|
||||
using System.Collections.Immutable;
|
||||
using System.Text.Json.Serialization;
|
||||
|
||||
namespace StellaOps.Excititor.WebService.Endpoints;
|
||||
|
||||
/// <summary>
|
||||
/// Risk feed API endpoints (EXCITITOR-RISK-66-001).
|
||||
/// Publishes risk-engine ready feeds with status, justification, and provenance
|
||||
/// without derived severity (aggregation-only per AOC baseline).
|
||||
/// </summary>
|
||||
public static class RiskFeedEndpoints
|
||||
{
|
||||
public static void MapRiskFeedEndpoints(this WebApplication app)
|
||||
{
|
||||
var group = app.MapGroup("/risk/v1")
|
||||
.RequireAuthorization(ExcititorPolicies.VexRead)
|
||||
.RequireTenant();
|
||||
|
||||
// POST /risk/v1/feed - Generate risk feed
|
||||
group.MapPost("/feed", async (
|
||||
HttpContext context,
|
||||
IOptions<VexStorageOptions> storageOptions,
|
||||
[FromServices] IRiskFeedService riskFeedService,
|
||||
[FromBody] RiskFeedRequestDto request,
|
||||
CancellationToken cancellationToken) =>
|
||||
{
|
||||
var scopeResult = ScopeAuthorization.RequireScope(context, "vex.read");
|
||||
if (scopeResult is not null)
|
||||
{
|
||||
return scopeResult;
|
||||
}
|
||||
|
||||
if (!TryResolveTenant(context, storageOptions.Value, out var tenant, out var tenantError))
|
||||
{
|
||||
return tenantError;
|
||||
}
|
||||
|
||||
if (request is null)
|
||||
{
|
||||
return Results.BadRequest(new
|
||||
{
|
||||
error = new { code = "ERR_RISK_PARAMS", message = "Request body is required" }
|
||||
});
|
||||
}
|
||||
|
||||
var domainRequest = new RiskFeedRequest(
|
||||
tenantId: tenant,
|
||||
advisoryKeys: request.AdvisoryKeys,
|
||||
artifacts: request.Artifacts,
|
||||
since: request.Since,
|
||||
limit: request.Limit ?? 1000);
|
||||
|
||||
var feedResponse = await riskFeedService
|
||||
.GenerateFeedAsync(domainRequest, cancellationToken)
|
||||
.ConfigureAwait(false);
|
||||
|
||||
var responseDto = MapToResponse(feedResponse);
|
||||
return Results.Ok(responseDto);
|
||||
})
|
||||
.WithName("GenerateRiskFeed")
|
||||
.WithDescription("Generates a risk-engine-ready VEX feed for the specified advisory keys and artifacts. Returns aggregated status, justification, and provenance without derived severity scores, suitable for direct consumption by the risk engine.");
|
||||
|
||||
// GET /risk/v1/feed/item - Get single risk feed item
|
||||
group.MapGet("/feed/item", async (
|
||||
HttpContext context,
|
||||
IOptions<VexStorageOptions> storageOptions,
|
||||
[FromServices] IRiskFeedService riskFeedService,
|
||||
[FromQuery] string? advisoryKey,
|
||||
[FromQuery] string? artifact,
|
||||
CancellationToken cancellationToken) =>
|
||||
{
|
||||
var scopeResult = ScopeAuthorization.RequireScope(context, "vex.read");
|
||||
if (scopeResult is not null)
|
||||
{
|
||||
return scopeResult;
|
||||
}
|
||||
|
||||
if (!TryResolveTenant(context, storageOptions.Value, out var tenant, out var tenantError))
|
||||
{
|
||||
return tenantError;
|
||||
}
|
||||
|
||||
if (string.IsNullOrWhiteSpace(advisoryKey) || string.IsNullOrWhiteSpace(artifact))
|
||||
{
|
||||
return Results.BadRequest(new
|
||||
{
|
||||
error = new { code = "ERR_RISK_PARAMS", message = "advisoryKey and artifact query parameters are required" }
|
||||
});
|
||||
}
|
||||
|
||||
var item = await riskFeedService
|
||||
.GetItemAsync(tenant, advisoryKey, artifact, cancellationToken)
|
||||
.ConfigureAwait(false);
|
||||
|
||||
if (item is null)
|
||||
{
|
||||
return Results.NotFound(new
|
||||
{
|
||||
error = new { code = "ERR_RISK_NOT_FOUND", message = "No risk feed item found for the specified advisory and artifact" }
|
||||
});
|
||||
}
|
||||
|
||||
var dto = MapToItemDto(item);
|
||||
return Results.Ok(dto);
|
||||
})
|
||||
.WithName("GetRiskFeedItem")
|
||||
.WithDescription("Retrieves a single risk feed item for a specific advisory key and artifact combination, including status, justification, provenance details, and all contributing source observations.");
|
||||
|
||||
// GET /risk/v1/feed/by-advisory - Get risk feed items by advisory key
|
||||
group.MapGet("/feed/by-advisory/{advisoryKey}", async (
|
||||
HttpContext context,
|
||||
string advisoryKey,
|
||||
IOptions<VexStorageOptions> storageOptions,
|
||||
[FromServices] IRiskFeedService riskFeedService,
|
||||
[FromQuery] int? limit,
|
||||
CancellationToken cancellationToken) =>
|
||||
{
|
||||
var scopeResult = ScopeAuthorization.RequireScope(context, "vex.read");
|
||||
if (scopeResult is not null)
|
||||
{
|
||||
return scopeResult;
|
||||
}
|
||||
|
||||
if (!TryResolveTenant(context, storageOptions.Value, out var tenant, out var tenantError))
|
||||
{
|
||||
return tenantError;
|
||||
}
|
||||
|
||||
if (string.IsNullOrWhiteSpace(advisoryKey))
|
||||
{
|
||||
return Results.BadRequest(new
|
||||
{
|
||||
error = new { code = "ERR_RISK_PARAMS", message = "advisoryKey is required" }
|
||||
});
|
||||
}
|
||||
|
||||
var request = new RiskFeedRequest(
|
||||
tenantId: tenant,
|
||||
advisoryKeys: [advisoryKey],
|
||||
limit: limit ?? 100);
|
||||
|
||||
var feedResponse = await riskFeedService
|
||||
.GenerateFeedAsync(request, cancellationToken)
|
||||
.ConfigureAwait(false);
|
||||
|
||||
var responseDto = MapToResponse(feedResponse);
|
||||
return Results.Ok(responseDto);
|
||||
})
|
||||
.WithName("GetRiskFeedByAdvisory")
|
||||
.WithDescription("Returns all risk feed items for a specific advisory key, listing affected artifacts with their VEX status, justification, and provenance. Useful for advisory-centric risk dashboards.");
|
||||
|
||||
// GET /risk/v1/feed/by-artifact/{artifact} - Get risk feed items by artifact
|
||||
group.MapGet("/feed/by-artifact/{**artifact}", async (
|
||||
HttpContext context,
|
||||
string artifact,
|
||||
IOptions<VexStorageOptions> storageOptions,
|
||||
[FromServices] IRiskFeedService riskFeedService,
|
||||
[FromQuery] int? limit,
|
||||
CancellationToken cancellationToken) =>
|
||||
{
|
||||
var scopeResult = ScopeAuthorization.RequireScope(context, "vex.read");
|
||||
if (scopeResult is not null)
|
||||
{
|
||||
return scopeResult;
|
||||
}
|
||||
|
||||
if (!TryResolveTenant(context, storageOptions.Value, out var tenant, out var tenantError))
|
||||
{
|
||||
return tenantError;
|
||||
}
|
||||
|
||||
if (string.IsNullOrWhiteSpace(artifact))
|
||||
{
|
||||
return Results.BadRequest(new
|
||||
{
|
||||
error = new { code = "ERR_RISK_PARAMS", message = "artifact is required" }
|
||||
});
|
||||
}
|
||||
|
||||
var request = new RiskFeedRequest(
|
||||
tenantId: tenant,
|
||||
artifacts: [artifact],
|
||||
limit: limit ?? 100);
|
||||
|
||||
var feedResponse = await riskFeedService
|
||||
.GenerateFeedAsync(request, cancellationToken)
|
||||
.ConfigureAwait(false);
|
||||
|
||||
var responseDto = MapToResponse(feedResponse);
|
||||
return Results.Ok(responseDto);
|
||||
})
|
||||
.WithName("GetRiskFeedByArtifact")
|
||||
.WithDescription("Returns all risk feed items for a specific artifact PURL or digest, listing all advisories affecting that artifact with their current VEX status and provenance. Supports wildcard path segments.");
|
||||
}
|
||||
|
||||
private static RiskFeedResponseDto MapToResponse(RiskFeedResponse response)
|
||||
{
|
||||
var items = response.Items
|
||||
.Select(MapToItemDto)
|
||||
.ToList();
|
||||
|
||||
return new RiskFeedResponseDto(
|
||||
Items: items,
|
||||
GeneratedAt: response.GeneratedAt,
|
||||
NextPageToken: response.NextPageToken);
|
||||
}
|
||||
|
||||
private static RiskFeedItemDto MapToItemDto(RiskFeedItem item)
|
||||
{
|
||||
var provenance = new RiskFeedProvenanceDto(
|
||||
TenantId: item.Provenance.TenantId,
|
||||
LinksetId: item.Provenance.LinksetId,
|
||||
ContentHash: item.Provenance.ContentHash,
|
||||
Confidence: item.Provenance.Confidence.ToString().ToLowerInvariant(),
|
||||
HasConflicts: item.Provenance.HasConflicts,
|
||||
GeneratedAt: item.Provenance.GeneratedAt,
|
||||
AttestationId: item.Provenance.AttestationId);
|
||||
|
||||
var sources = item.Sources
|
||||
.Select(s => new RiskFeedSourceDto(
|
||||
ObservationId: s.ObservationId,
|
||||
ProviderId: s.ProviderId,
|
||||
Status: s.Status,
|
||||
Justification: s.Justification,
|
||||
Confidence: s.Confidence))
|
||||
.ToList();
|
||||
|
||||
return new RiskFeedItemDto(
|
||||
AdvisoryKey: item.AdvisoryKey,
|
||||
Artifact: item.Artifact,
|
||||
Status: item.Status.ToString().ToLowerInvariant(),
|
||||
Justification: item.Justification?.ToString().ToLowerInvariant(),
|
||||
Provenance: provenance,
|
||||
ObservedAt: item.ObservedAt,
|
||||
Sources: sources);
|
||||
}
|
||||
|
||||
private static bool TryResolveTenant(
|
||||
HttpContext context,
|
||||
VexStorageOptions options,
|
||||
out string tenant,
|
||||
out IResult? problem)
|
||||
{
|
||||
problem = null;
|
||||
tenant = string.Empty;
|
||||
|
||||
var headerTenant = context.Request.Headers["X-Stella-Tenant"].FirstOrDefault();
|
||||
if (!string.IsNullOrWhiteSpace(headerTenant))
|
||||
{
|
||||
tenant = headerTenant.Trim().ToLowerInvariant();
|
||||
}
|
||||
else if (!string.IsNullOrWhiteSpace(options.DefaultTenant))
|
||||
{
|
||||
tenant = options.DefaultTenant.Trim().ToLowerInvariant();
|
||||
}
|
||||
else
|
||||
{
|
||||
problem = Results.BadRequest(new
|
||||
{
|
||||
error = new { code = "ERR_TENANT", message = "X-Stella-Tenant header is required" }
|
||||
});
|
||||
return false;
|
||||
}
|
||||
|
||||
return true;
|
||||
}
|
||||
}
|
||||
|
||||
// Request DTO
|
||||
public sealed record RiskFeedRequestDto(
|
||||
[property: JsonPropertyName("advisoryKeys")] IEnumerable<string>? AdvisoryKeys,
|
||||
[property: JsonPropertyName("artifacts")] IEnumerable<string>? Artifacts,
|
||||
[property: JsonPropertyName("since")] DateTimeOffset? Since,
|
||||
[property: JsonPropertyName("limit")] int? Limit);
|
||||
|
||||
// Response DTOs
|
||||
public sealed record RiskFeedResponseDto(
|
||||
[property: JsonPropertyName("items")] IReadOnlyList<RiskFeedItemDto> Items,
|
||||
[property: JsonPropertyName("generatedAt")] DateTimeOffset GeneratedAt,
|
||||
[property: JsonPropertyName("nextPageToken")] string? NextPageToken);
|
||||
|
||||
public sealed record RiskFeedItemDto(
|
||||
[property: JsonPropertyName("advisoryKey")] string AdvisoryKey,
|
||||
[property: JsonPropertyName("artifact")] string Artifact,
|
||||
[property: JsonPropertyName("status")] string Status,
|
||||
[property: JsonPropertyName("justification")] string? Justification,
|
||||
[property: JsonPropertyName("provenance")] RiskFeedProvenanceDto Provenance,
|
||||
[property: JsonPropertyName("observedAt")] DateTimeOffset ObservedAt,
|
||||
[property: JsonPropertyName("sources")] IReadOnlyList<RiskFeedSourceDto> Sources);
|
||||
|
||||
public sealed record RiskFeedProvenanceDto(
|
||||
[property: JsonPropertyName("tenantId")] string TenantId,
|
||||
[property: JsonPropertyName("linksetId")] string LinksetId,
|
||||
[property: JsonPropertyName("contentHash")] string ContentHash,
|
||||
[property: JsonPropertyName("confidence")] string Confidence,
|
||||
[property: JsonPropertyName("hasConflicts")] bool HasConflicts,
|
||||
[property: JsonPropertyName("generatedAt")] DateTimeOffset GeneratedAt,
|
||||
[property: JsonPropertyName("attestationId")] string? AttestationId);
|
||||
|
||||
public sealed record RiskFeedSourceDto(
|
||||
[property: JsonPropertyName("observationId")] string ObservationId,
|
||||
[property: JsonPropertyName("providerId")] string ProviderId,
|
||||
[property: JsonPropertyName("status")] string Status,
|
||||
[property: JsonPropertyName("justification")] string? Justification,
|
||||
[property: JsonPropertyName("confidence")] double? Confidence);
|
||||
@@ -0,0 +1,77 @@
|
||||
|
||||
using Microsoft.AspNetCore.Builder;
|
||||
using Microsoft.AspNetCore.Http;
|
||||
using Microsoft.Extensions.Primitives;
|
||||
using System.Diagnostics;
|
||||
using System.Threading.Tasks;
|
||||
|
||||
namespace StellaOps.Excititor.WebService.Extensions;
|
||||
|
||||
internal static class ObservabilityExtensions
|
||||
{
|
||||
private const string TraceHeaderName = "X-Stella-TraceId";
|
||||
private const string CorrelationHeaderName = "X-Stella-CorrelationId";
|
||||
private const string LegacyCorrelationHeaderName = "X-Correlation-Id";
|
||||
private const string CorrelationItemKey = "__stella.correlationId";
|
||||
|
||||
public static IApplicationBuilder UseObservabilityHeaders(this IApplicationBuilder app)
|
||||
{
|
||||
return app.Use((context, next) =>
|
||||
{
|
||||
var correlationId = ResolveCorrelationId(context);
|
||||
context.Items[CorrelationItemKey] = correlationId;
|
||||
|
||||
context.Response.OnStarting(state =>
|
||||
{
|
||||
var httpContext = (HttpContext)state;
|
||||
ApplyHeaders(httpContext);
|
||||
return Task.CompletedTask;
|
||||
}, context);
|
||||
|
||||
return next();
|
||||
});
|
||||
}
|
||||
|
||||
private static void ApplyHeaders(HttpContext context)
|
||||
{
|
||||
var traceId = Activity.Current?.TraceId.ToString() ?? context.TraceIdentifier;
|
||||
if (!string.IsNullOrWhiteSpace(traceId))
|
||||
{
|
||||
context.Response.Headers[TraceHeaderName] = traceId;
|
||||
}
|
||||
|
||||
var correlationId = ResolveCorrelationId(context);
|
||||
if (!string.IsNullOrWhiteSpace(correlationId))
|
||||
{
|
||||
context.Response.Headers[CorrelationHeaderName] = correlationId!;
|
||||
}
|
||||
}
|
||||
|
||||
private static string ResolveCorrelationId(HttpContext context)
|
||||
{
|
||||
if (context.Items.TryGetValue(CorrelationItemKey, out var existing) && existing is string cached && !string.IsNullOrWhiteSpace(cached))
|
||||
{
|
||||
return cached;
|
||||
}
|
||||
|
||||
if (TryReadHeader(context.Request.Headers, CorrelationHeaderName, out var headerValue) ||
|
||||
TryReadHeader(context.Request.Headers, LegacyCorrelationHeaderName, out headerValue))
|
||||
{
|
||||
return headerValue!;
|
||||
}
|
||||
|
||||
return context.TraceIdentifier;
|
||||
}
|
||||
|
||||
private static bool TryReadHeader(IHeaderDictionary headers, string name, out string? value)
|
||||
{
|
||||
if (headers.TryGetValue(name, out StringValues header) && !StringValues.IsNullOrEmpty(header))
|
||||
{
|
||||
value = header.ToString();
|
||||
return true;
|
||||
}
|
||||
|
||||
value = null;
|
||||
return false;
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,145 @@
|
||||
|
||||
using Microsoft.AspNetCore.Builder;
|
||||
using Microsoft.Extensions.DependencyInjection;
|
||||
using OpenTelemetry.Metrics;
|
||||
using OpenTelemetry.Resources;
|
||||
using OpenTelemetry.Trace;
|
||||
using StellaOps.Excititor.WebService.Options;
|
||||
using StellaOps.Excititor.WebService.Telemetry;
|
||||
using StellaOps.Ingestion.Telemetry;
|
||||
using System;
|
||||
using System.Collections.Generic;
|
||||
using System.Reflection;
|
||||
|
||||
namespace StellaOps.Excititor.WebService.Extensions;
|
||||
|
||||
internal static class TelemetryExtensions
|
||||
{
|
||||
public static void ConfigureExcititorTelemetry(this WebApplicationBuilder builder)
|
||||
{
|
||||
var telemetryOptions = new ExcititorTelemetryOptions();
|
||||
builder.Configuration.GetSection("Excititor:Telemetry").Bind(telemetryOptions);
|
||||
|
||||
if (!telemetryOptions.Enabled || (!telemetryOptions.EnableTracing && !telemetryOptions.EnableMetrics))
|
||||
{
|
||||
return;
|
||||
}
|
||||
|
||||
var openTelemetry = builder.Services.AddOpenTelemetry();
|
||||
|
||||
openTelemetry.ConfigureResource(resource =>
|
||||
{
|
||||
var serviceName = telemetryOptions.ServiceName ?? builder.Environment.ApplicationName ?? "StellaOps.Excititor.WebService";
|
||||
var version = Assembly.GetExecutingAssembly().GetName().Version?.ToString() ?? "unknown";
|
||||
resource.AddService(serviceName, serviceVersion: version, serviceInstanceId: Environment.MachineName);
|
||||
|
||||
foreach (var attribute in telemetryOptions.ResourceAttributes)
|
||||
{
|
||||
if (string.IsNullOrWhiteSpace(attribute.Key) || string.IsNullOrWhiteSpace(attribute.Value))
|
||||
{
|
||||
continue;
|
||||
}
|
||||
|
||||
resource.AddAttributes(new[]
|
||||
{
|
||||
new KeyValuePair<string, object>(attribute.Key, attribute.Value)
|
||||
});
|
||||
}
|
||||
});
|
||||
|
||||
if (telemetryOptions.EnableTracing)
|
||||
{
|
||||
openTelemetry.WithTracing(tracing =>
|
||||
{
|
||||
tracing
|
||||
.AddSource(IngestionTelemetry.ActivitySourceName)
|
||||
.AddAspNetCoreInstrumentation()
|
||||
.AddHttpClientInstrumentation();
|
||||
|
||||
ConfigureExporters(telemetryOptions, tracing);
|
||||
});
|
||||
}
|
||||
|
||||
if (telemetryOptions.EnableMetrics)
|
||||
{
|
||||
openTelemetry.WithMetrics(metrics =>
|
||||
{
|
||||
metrics
|
||||
.AddMeter(IngestionTelemetry.MeterName)
|
||||
.AddMeter(EvidenceTelemetry.MeterName)
|
||||
.AddMeter(LinksetTelemetry.MeterName)
|
||||
.AddMeter(NormalizationTelemetry.MeterName)
|
||||
.AddAspNetCoreInstrumentation()
|
||||
.AddHttpClientInstrumentation()
|
||||
.AddRuntimeInstrumentation();
|
||||
|
||||
ConfigureExporters(telemetryOptions, metrics);
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
private static void ConfigureExporters(ExcititorTelemetryOptions options, TracerProviderBuilder tracing)
|
||||
{
|
||||
if (!string.IsNullOrWhiteSpace(options.OtlpEndpoint))
|
||||
{
|
||||
tracing.AddOtlpExporter(otlp =>
|
||||
{
|
||||
otlp.Endpoint = new Uri(options.OtlpEndpoint, UriKind.Absolute);
|
||||
var headers = BuildHeaders(options.OtlpHeaders);
|
||||
if (!string.IsNullOrEmpty(headers))
|
||||
{
|
||||
otlp.Headers = headers;
|
||||
}
|
||||
});
|
||||
return;
|
||||
}
|
||||
|
||||
if (options.ExportConsole)
|
||||
{
|
||||
tracing.AddConsoleExporter();
|
||||
}
|
||||
}
|
||||
|
||||
private static void ConfigureExporters(ExcititorTelemetryOptions options, MeterProviderBuilder metrics)
|
||||
{
|
||||
if (!string.IsNullOrWhiteSpace(options.OtlpEndpoint))
|
||||
{
|
||||
metrics.AddOtlpExporter(otlp =>
|
||||
{
|
||||
otlp.Endpoint = new Uri(options.OtlpEndpoint, UriKind.Absolute);
|
||||
var headers = BuildHeaders(options.OtlpHeaders);
|
||||
if (!string.IsNullOrEmpty(headers))
|
||||
{
|
||||
otlp.Headers = headers;
|
||||
}
|
||||
});
|
||||
return;
|
||||
}
|
||||
|
||||
if (options.ExportConsole)
|
||||
{
|
||||
metrics.AddConsoleExporter();
|
||||
}
|
||||
}
|
||||
|
||||
private static string? BuildHeaders(IReadOnlyDictionary<string, string> headers)
|
||||
{
|
||||
if (headers.Count == 0)
|
||||
{
|
||||
return null;
|
||||
}
|
||||
|
||||
var parts = new List<string>(headers.Count);
|
||||
foreach (var header in headers)
|
||||
{
|
||||
if (string.IsNullOrWhiteSpace(header.Key) || string.IsNullOrWhiteSpace(header.Value))
|
||||
{
|
||||
continue;
|
||||
}
|
||||
|
||||
parts.Add($"{header.Key}={header.Value}");
|
||||
}
|
||||
|
||||
return parts.Count == 0 ? null : string.Join(',', parts);
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,72 @@
|
||||
|
||||
using RawModels = StellaOps.Concelier.RawModels;
|
||||
using StellaOps.Excititor.Core;
|
||||
using StellaOps.Excititor.Core.Storage;
|
||||
using System.Collections.Immutable;
|
||||
using System.Text.Json;
|
||||
|
||||
namespace StellaOps.Excititor.WebService.Extensions;
|
||||
|
||||
internal static class VexRawDocumentMapper
|
||||
{
|
||||
public static RawModels.VexRawDocument ToRawModel(VexRawRecord record, string defaultTenant)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(record);
|
||||
|
||||
var metadata = record.Metadata ?? ImmutableDictionary<string, string>.Empty;
|
||||
var tenant = Get(metadata, "tenant", record.Tenant) ?? defaultTenant;
|
||||
|
||||
var source = new RawModels.RawSourceMetadata(
|
||||
Vendor: Get(metadata, "source.vendor", record.ProviderId) ?? record.ProviderId,
|
||||
Connector: Get(metadata, "source.connector", record.ProviderId) ?? record.ProviderId,
|
||||
ConnectorVersion: Get(metadata, "source.connector_version", "unknown") ?? "unknown",
|
||||
Stream: Get(metadata, "source.stream", record.Format.ToString().ToLowerInvariant()));
|
||||
|
||||
var signature = new RawModels.RawSignatureMetadata(
|
||||
Present: string.Equals(Get(metadata, "signature.present"), "true", StringComparison.OrdinalIgnoreCase),
|
||||
Format: Get(metadata, "signature.format"),
|
||||
KeyId: Get(metadata, "signature.key_id"),
|
||||
Signature: Get(metadata, "signature.sig"),
|
||||
Certificate: Get(metadata, "signature.certificate"),
|
||||
Digest: Get(metadata, "signature.digest"));
|
||||
|
||||
var upstream = new RawModels.RawUpstreamMetadata(
|
||||
UpstreamId: Get(metadata, "upstream.id", record.Digest) ?? record.Digest,
|
||||
DocumentVersion: Get(metadata, "upstream.version"),
|
||||
RetrievedAt: record.RetrievedAt,
|
||||
ContentHash: Get(metadata, "upstream.content_hash", record.Digest) ?? record.Digest,
|
||||
Signature: signature,
|
||||
Provenance: metadata);
|
||||
|
||||
var content = new RawModels.RawContent(
|
||||
Format: record.Format.ToString().ToLowerInvariant(),
|
||||
SpecVersion: Get(metadata, "content.spec_version"),
|
||||
Raw: ParseJson(record.Content),
|
||||
Encoding: Get(metadata, "content.encoding"));
|
||||
|
||||
return new RawModels.VexRawDocument(
|
||||
tenant,
|
||||
source,
|
||||
upstream,
|
||||
content,
|
||||
new RawModels.RawLinkset(),
|
||||
Statements: null,
|
||||
Supersedes: record.SupersedesDigest);
|
||||
}
|
||||
|
||||
private static string? Get(IReadOnlyDictionary<string, string> metadata, string key, string? fallback = null)
|
||||
{
|
||||
if (metadata.TryGetValue(key, out var value) && !string.IsNullOrWhiteSpace(value))
|
||||
{
|
||||
return value;
|
||||
}
|
||||
|
||||
return fallback;
|
||||
}
|
||||
|
||||
private static JsonElement ParseJson(ReadOnlyMemory<byte> content)
|
||||
{
|
||||
using var document = JsonDocument.Parse(content);
|
||||
return document.RootElement.Clone();
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,152 @@
|
||||
|
||||
using StellaOps.Excititor.Core;
|
||||
using StellaOps.Excititor.WebService.Contracts;
|
||||
using System;
|
||||
using System.Buffers;
|
||||
using System.Collections.Generic;
|
||||
using System.Collections.Immutable;
|
||||
using System.Globalization;
|
||||
using System.Security.Cryptography;
|
||||
using System.Text;
|
||||
using System.Text.Json;
|
||||
|
||||
namespace StellaOps.Excititor.WebService.Extensions;
|
||||
|
||||
internal static class VexRawRequestMapper
|
||||
{
|
||||
public static VexRawDocument Map(VexIngestRequest request, string tenant, TimeProvider timeProvider)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(request);
|
||||
ArgumentException.ThrowIfNullOrWhiteSpace(request.ProviderId);
|
||||
ArgumentNullException.ThrowIfNull(timeProvider);
|
||||
|
||||
var source = request.Source ?? throw new ArgumentException("source section is required.", nameof(request));
|
||||
var upstream = request.Upstream ?? throw new ArgumentException("upstream section is required.", nameof(request));
|
||||
var content = request.Content ?? throw new ArgumentException("content section is required.", nameof(request));
|
||||
ArgumentException.ThrowIfNullOrWhiteSpace(upstream.SourceUri);
|
||||
ArgumentException.ThrowIfNullOrWhiteSpace(upstream.UpstreamId);
|
||||
|
||||
var providerId = request.ProviderId.Trim();
|
||||
var format = ParseFormat(content.Format);
|
||||
var sourceUri = new Uri(upstream.SourceUri!, UriKind.Absolute);
|
||||
var retrievedAt = upstream.RetrievedAt ?? timeProvider.GetUtcNow();
|
||||
var payload = SerializeContent(content.Raw);
|
||||
var digest = NormalizeDigest(upstream.ContentHash, payload.Span);
|
||||
|
||||
var metadataBuilder = ImmutableDictionary.CreateBuilder<string, string>(StringComparer.Ordinal);
|
||||
CopyMetadata(metadataBuilder, request.Metadata);
|
||||
CopyMetadata(metadataBuilder, upstream.Provenance);
|
||||
|
||||
metadataBuilder["tenant"] = tenant.Trim().ToLowerInvariant();
|
||||
SetIfMissing(metadataBuilder, "source.vendor", source.Vendor);
|
||||
SetIfMissing(metadataBuilder, "source.connector", source.Connector);
|
||||
SetIfMissing(metadataBuilder, "source.connector_version", source.Version);
|
||||
SetIfMissing(metadataBuilder, "source.stream", source.Stream ?? format.ToString().ToLowerInvariant());
|
||||
SetIfMissing(metadataBuilder, "upstream.id", upstream.UpstreamId);
|
||||
SetIfMissing(metadataBuilder, "upstream.version", upstream.DocumentVersion ?? retrievedAt.ToString("O", CultureInfo.InvariantCulture));
|
||||
SetIfMissing(metadataBuilder, "content.spec_version", content.SpecVersion);
|
||||
SetIfMissing(metadataBuilder, "content.encoding", content.Encoding);
|
||||
|
||||
var signature = upstream.Signature;
|
||||
metadataBuilder["signature.present"] = (signature?.Present ?? false).ToString();
|
||||
SetIfMissing(metadataBuilder, "signature.format", signature?.Format);
|
||||
SetIfMissing(metadataBuilder, "signature.key_id", signature?.KeyId);
|
||||
SetIfMissing(metadataBuilder, "signature.sig", signature?.Signature);
|
||||
SetIfMissing(metadataBuilder, "signature.certificate", signature?.Certificate);
|
||||
SetIfMissing(metadataBuilder, "signature.digest", signature?.Digest);
|
||||
|
||||
var metadata = metadataBuilder.ToImmutable();
|
||||
|
||||
return new VexRawDocument(
|
||||
providerId,
|
||||
format,
|
||||
sourceUri,
|
||||
retrievedAt,
|
||||
digest,
|
||||
payload,
|
||||
metadata);
|
||||
}
|
||||
|
||||
private static ReadOnlyMemory<byte> SerializeContent(JsonElement element)
|
||||
{
|
||||
var buffer = new ArrayBufferWriter<byte>();
|
||||
using (var writer = new Utf8JsonWriter(buffer, new JsonWriterOptions { Indented = false }))
|
||||
{
|
||||
if (element.ValueKind == JsonValueKind.Undefined)
|
||||
{
|
||||
writer.WriteStartObject();
|
||||
writer.WriteEndObject();
|
||||
}
|
||||
else
|
||||
{
|
||||
element.WriteTo(writer);
|
||||
}
|
||||
}
|
||||
|
||||
return buffer.WrittenMemory.ToArray();
|
||||
}
|
||||
|
||||
private static VexDocumentFormat ParseFormat(string format)
|
||||
{
|
||||
if (string.IsNullOrWhiteSpace(format))
|
||||
{
|
||||
throw new ArgumentException("content.format is required.", nameof(format));
|
||||
}
|
||||
|
||||
if (Enum.TryParse<VexDocumentFormat>(format, ignoreCase: true, out var parsed))
|
||||
{
|
||||
return parsed;
|
||||
}
|
||||
|
||||
throw new ArgumentException($"Unsupported VEX document format {format}.", nameof(format));
|
||||
}
|
||||
|
||||
private static string NormalizeDigest(string? existingDigest, ReadOnlySpan<byte> payload)
|
||||
{
|
||||
if (!string.IsNullOrWhiteSpace(existingDigest))
|
||||
{
|
||||
return existingDigest.Trim();
|
||||
}
|
||||
|
||||
Span<byte> buffer = stackalloc byte[32];
|
||||
if (SHA256.TryHashData(payload, buffer, out _))
|
||||
{
|
||||
return "sha256:" + Convert.ToHexString(buffer).ToLowerInvariant();
|
||||
}
|
||||
|
||||
using var sha = SHA256.Create();
|
||||
var hash = sha.ComputeHash(payload.ToArray());
|
||||
return "sha256:" + Convert.ToHexString(hash).ToLowerInvariant();
|
||||
}
|
||||
|
||||
private static void CopyMetadata(ImmutableDictionary<string, string>.Builder builder, IReadOnlyDictionary<string, string>? values)
|
||||
{
|
||||
if (values is null)
|
||||
{
|
||||
return;
|
||||
}
|
||||
|
||||
foreach (var kvp in values)
|
||||
{
|
||||
if (string.IsNullOrWhiteSpace(kvp.Key))
|
||||
{
|
||||
continue;
|
||||
}
|
||||
|
||||
builder[kvp.Key.Trim()] = kvp.Value?.Trim() ?? string.Empty;
|
||||
}
|
||||
}
|
||||
|
||||
private static void SetIfMissing(ImmutableDictionary<string, string>.Builder builder, string key, string? value)
|
||||
{
|
||||
if (string.IsNullOrWhiteSpace(value))
|
||||
{
|
||||
return;
|
||||
}
|
||||
|
||||
if (!builder.ContainsKey(key))
|
||||
{
|
||||
builder[key] = value.Trim();
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,247 @@
|
||||
|
||||
using StellaOps.Excititor.Core;
|
||||
using StellaOps.Excititor.Core.Observations;
|
||||
using StellaOps.Excititor.WebService.Contracts;
|
||||
using System;
|
||||
using System.Collections.Generic;
|
||||
using System.Collections.Immutable;
|
||||
using System.Linq;
|
||||
|
||||
namespace StellaOps.Excititor.WebService.Graph;
|
||||
|
||||
internal static class GraphOverlayFactory
|
||||
{
|
||||
public static IReadOnlyList<GraphOverlayItem> Build(
|
||||
string tenant,
|
||||
DateTimeOffset generatedAt,
|
||||
IReadOnlyList<string> orderedPurls,
|
||||
IReadOnlyList<VexObservation> observations,
|
||||
bool includeJustifications)
|
||||
{
|
||||
if (string.IsNullOrWhiteSpace(tenant))
|
||||
{
|
||||
throw new ArgumentException("tenant is required", nameof(tenant));
|
||||
}
|
||||
|
||||
if (orderedPurls is null)
|
||||
{
|
||||
throw new ArgumentNullException(nameof(orderedPurls));
|
||||
}
|
||||
|
||||
if (observations is null)
|
||||
{
|
||||
throw new ArgumentNullException(nameof(observations));
|
||||
}
|
||||
|
||||
var purlOrder = new Dictionary<string, int>(StringComparer.OrdinalIgnoreCase);
|
||||
for (var i = 0; i < orderedPurls.Count; i++)
|
||||
{
|
||||
purlOrder[orderedPurls[i]] = i;
|
||||
}
|
||||
|
||||
var aggregates = new Dictionary<(string Purl, string AdvisoryId, string Source), OverlayAggregate>(new OverlayKeyComparer());
|
||||
|
||||
foreach (var observation in observations.OrderByDescending(o => o.CreatedAt).ThenBy(o => o.ObservationId, StringComparer.Ordinal))
|
||||
{
|
||||
var observationRef = new GraphOverlayObservation(
|
||||
observation.ObservationId,
|
||||
observation.Upstream.ContentHash,
|
||||
observation.Upstream.FetchedAt);
|
||||
|
||||
foreach (var statement in observation.Statements)
|
||||
{
|
||||
var targetPurls = ResolvePurls(statement, observation.Linkset.Purls);
|
||||
foreach (var purl in targetPurls)
|
||||
{
|
||||
if (!purlOrder.ContainsKey(purl))
|
||||
{
|
||||
continue;
|
||||
}
|
||||
|
||||
var key = (purl, statement.VulnerabilityId, observation.ProviderId);
|
||||
if (!aggregates.TryGetValue(key, out var aggregate))
|
||||
{
|
||||
aggregate = new OverlayAggregate(purl, statement.VulnerabilityId, observation.ProviderId);
|
||||
aggregates[key] = aggregate;
|
||||
}
|
||||
|
||||
aggregate.UpdateStatus(statement.Status, observation.CreatedAt);
|
||||
if (includeJustifications && statement.Justification is not null)
|
||||
{
|
||||
aggregate.AddJustification(statement.Justification.Value, observation.ObservationId);
|
||||
}
|
||||
|
||||
aggregate.AddObservation(observationRef);
|
||||
aggregate.AddConflicts(observation.Linkset.Disagreements);
|
||||
aggregate.SetProvenance(
|
||||
observation.StreamId ?? observation.ObservationId,
|
||||
observation.Upstream.ContentHash,
|
||||
observation.Upstream.ContentHash);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
var overlays = aggregates.Values
|
||||
.OrderBy(a => purlOrder[a.Purl])
|
||||
.ThenBy(a => a.AdvisoryId, StringComparer.OrdinalIgnoreCase)
|
||||
.ThenBy(a => a.Source, StringComparer.OrdinalIgnoreCase)
|
||||
.Select(a => a.ToOverlayItem(tenant, generatedAt, includeJustifications))
|
||||
.ToList();
|
||||
|
||||
return overlays;
|
||||
}
|
||||
|
||||
private static IReadOnlyList<string> ResolvePurls(VexObservationStatement stmt, ImmutableArray<string> linksetPurls)
|
||||
{
|
||||
if (!string.IsNullOrWhiteSpace(stmt.Purl))
|
||||
{
|
||||
return new[] { stmt.Purl };
|
||||
}
|
||||
|
||||
if (linksetPurls.IsDefaultOrEmpty)
|
||||
{
|
||||
return Array.Empty<string>();
|
||||
}
|
||||
|
||||
return linksetPurls.Where(p => !string.IsNullOrWhiteSpace(p)).ToArray();
|
||||
}
|
||||
|
||||
private static string MapStatus(VexClaimStatus status)
|
||||
=> status switch
|
||||
{
|
||||
VexClaimStatus.NotAffected => "not_affected",
|
||||
VexClaimStatus.UnderInvestigation => "under_investigation",
|
||||
VexClaimStatus.Fixed => "fixed",
|
||||
_ => "affected"
|
||||
};
|
||||
|
||||
private sealed class OverlayAggregate
|
||||
{
|
||||
private readonly SortedSet<string> _observationHashes = new(StringComparer.Ordinal);
|
||||
private readonly SortedSet<string> _observationIds = new(StringComparer.Ordinal);
|
||||
private readonly List<GraphOverlayObservation> _observations = new();
|
||||
private readonly List<GraphOverlayConflict> _conflicts = new();
|
||||
private readonly List<GraphOverlayJustification> _justifications = new();
|
||||
private DateTimeOffset? _latestCreatedAt;
|
||||
private string? _status;
|
||||
private string? _linksetId;
|
||||
private string? _linksetHash;
|
||||
private string? _policyHash;
|
||||
private string? _sbomContextHash;
|
||||
|
||||
public OverlayAggregate(string purl, string advisoryId, string source)
|
||||
{
|
||||
Purl = purl;
|
||||
AdvisoryId = advisoryId;
|
||||
Source = source;
|
||||
}
|
||||
|
||||
public string Purl { get; }
|
||||
|
||||
public string AdvisoryId { get; }
|
||||
|
||||
public string Source { get; }
|
||||
|
||||
public void UpdateStatus(VexClaimStatus status, DateTimeOffset createdAt)
|
||||
{
|
||||
if (_latestCreatedAt is null || createdAt > _latestCreatedAt.Value)
|
||||
{
|
||||
_latestCreatedAt = createdAt;
|
||||
_status = MapStatus(status);
|
||||
}
|
||||
}
|
||||
|
||||
public void AddJustification(VexJustification justification, string observationId)
|
||||
{
|
||||
var kind = justification.ToString();
|
||||
if (string.IsNullOrWhiteSpace(kind))
|
||||
{
|
||||
return;
|
||||
}
|
||||
|
||||
_justifications.Add(new GraphOverlayJustification(
|
||||
kind,
|
||||
kind,
|
||||
new[] { observationId },
|
||||
null));
|
||||
}
|
||||
|
||||
public void AddObservation(GraphOverlayObservation observation)
|
||||
{
|
||||
if (_observationIds.Add(observation.Id))
|
||||
{
|
||||
_observations.Add(observation);
|
||||
}
|
||||
|
||||
_observationHashes.Add(observation.ContentHash);
|
||||
}
|
||||
|
||||
public void AddConflicts(ImmutableArray<VexObservationDisagreement> disagreements)
|
||||
{
|
||||
if (disagreements.IsDefaultOrEmpty)
|
||||
{
|
||||
return;
|
||||
}
|
||||
|
||||
foreach (var disagreement in disagreements)
|
||||
{
|
||||
_conflicts.Add(new GraphOverlayConflict(
|
||||
"status",
|
||||
disagreement.Justification ?? disagreement.Status,
|
||||
new[] { disagreement.Status },
|
||||
new[] { disagreement.ProviderId }));
|
||||
}
|
||||
}
|
||||
|
||||
public void SetProvenance(string linksetId, string linksetHash, string observationHash)
|
||||
{
|
||||
_linksetId ??= linksetId;
|
||||
_linksetHash ??= linksetHash;
|
||||
_policyHash ??= null;
|
||||
_sbomContextHash ??= null;
|
||||
_observationHashes.Add(observationHash);
|
||||
}
|
||||
|
||||
public GraphOverlayItem ToOverlayItem(string tenant, DateTimeOffset generatedAt, bool includeJustifications)
|
||||
{
|
||||
return new GraphOverlayItem(
|
||||
SchemaVersion: "1.0.0",
|
||||
GeneratedAt: generatedAt,
|
||||
Tenant: tenant,
|
||||
Purl: Purl,
|
||||
AdvisoryId: AdvisoryId,
|
||||
Source: Source,
|
||||
Status: _status ?? "unknown",
|
||||
Justifications: includeJustifications ? _justifications : Array.Empty<GraphOverlayJustification>(),
|
||||
Conflicts: _conflicts,
|
||||
Observations: _observations,
|
||||
Provenance: new GraphOverlayProvenance(
|
||||
LinksetId: _linksetId ?? string.Empty,
|
||||
LinksetHash: _linksetHash ?? string.Empty,
|
||||
ObservationHashes: _observationHashes.ToArray(),
|
||||
PolicyHash: _policyHash,
|
||||
SbomContextHash: _sbomContextHash,
|
||||
PlanCacheKey: null),
|
||||
Cache: null);
|
||||
}
|
||||
}
|
||||
|
||||
private sealed class OverlayKeyComparer : IEqualityComparer<(string Purl, string AdvisoryId, string Source)>
|
||||
{
|
||||
public bool Equals((string Purl, string AdvisoryId, string Source) x, (string Purl, string AdvisoryId, string Source) y)
|
||||
{
|
||||
return string.Equals(x.Purl, y.Purl, StringComparison.OrdinalIgnoreCase)
|
||||
&& string.Equals(x.AdvisoryId, y.AdvisoryId, StringComparison.OrdinalIgnoreCase)
|
||||
&& string.Equals(x.Source, y.Source, StringComparison.OrdinalIgnoreCase);
|
||||
}
|
||||
|
||||
public int GetHashCode((string Purl, string AdvisoryId, string Source) obj)
|
||||
{
|
||||
var hash = new HashCode();
|
||||
hash.Add(obj.Purl, StringComparer.OrdinalIgnoreCase);
|
||||
hash.Add(obj.AdvisoryId, StringComparer.OrdinalIgnoreCase);
|
||||
hash.Add(obj.Source, StringComparer.OrdinalIgnoreCase);
|
||||
return hash.ToHashCode();
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,103 @@
|
||||
|
||||
using StellaOps.Excititor.Core.Observations;
|
||||
using StellaOps.Excititor.WebService.Contracts;
|
||||
using System;
|
||||
using System.Collections.Generic;
|
||||
using System.Linq;
|
||||
|
||||
namespace StellaOps.Excititor.WebService.Graph;
|
||||
|
||||
internal static class GraphStatusFactory
|
||||
{
|
||||
public static IReadOnlyList<GraphStatusItem> Build(
|
||||
string tenant,
|
||||
DateTimeOffset generatedAt,
|
||||
IReadOnlyList<string> orderedPurls,
|
||||
IReadOnlyList<VexObservation> observations)
|
||||
{
|
||||
if (string.IsNullOrWhiteSpace(tenant))
|
||||
{
|
||||
throw new ArgumentException("tenant is required", nameof(tenant));
|
||||
}
|
||||
|
||||
if (orderedPurls is null)
|
||||
{
|
||||
throw new ArgumentNullException(nameof(orderedPurls));
|
||||
}
|
||||
|
||||
if (observations is null)
|
||||
{
|
||||
throw new ArgumentNullException(nameof(observations));
|
||||
}
|
||||
|
||||
var overlays = GraphOverlayFactory.Build(tenant, generatedAt, orderedPurls, observations, includeJustifications: false);
|
||||
|
||||
var items = new List<GraphStatusItem>(orderedPurls.Count);
|
||||
|
||||
foreach (var purl in orderedPurls)
|
||||
{
|
||||
var overlaysForPurl = overlays
|
||||
.Where(o => o.Purl.Equals(purl, StringComparison.OrdinalIgnoreCase))
|
||||
.ToList();
|
||||
|
||||
if (overlaysForPurl.Count == 0)
|
||||
{
|
||||
items.Add(new GraphStatusItem(
|
||||
purl,
|
||||
new GraphOverlaySummary(0, 0, 0, 1),
|
||||
null,
|
||||
Array.Empty<string>(),
|
||||
null));
|
||||
continue;
|
||||
}
|
||||
|
||||
var open = 0;
|
||||
var notAffected = 0;
|
||||
var underInvestigation = 0;
|
||||
var noStatement = 0;
|
||||
var sources = new SortedSet<string>(StringComparer.OrdinalIgnoreCase);
|
||||
var observationRefs = new List<GraphOverlayObservation>();
|
||||
|
||||
foreach (var overlay in overlaysForPurl)
|
||||
{
|
||||
sources.Add(overlay.Source);
|
||||
observationRefs.AddRange(overlay.Observations);
|
||||
switch (overlay.Status)
|
||||
{
|
||||
case "not_affected":
|
||||
notAffected++;
|
||||
break;
|
||||
case "under_investigation":
|
||||
underInvestigation++;
|
||||
break;
|
||||
case "fixed":
|
||||
case "affected":
|
||||
open++;
|
||||
break;
|
||||
default:
|
||||
noStatement++;
|
||||
break;
|
||||
}
|
||||
}
|
||||
|
||||
var latest = observationRefs.Count == 0
|
||||
? (DateTimeOffset?)null
|
||||
: observationRefs.Max(o => o.FetchedAt);
|
||||
|
||||
var lastHash = observationRefs
|
||||
.OrderBy(o => o.FetchedAt)
|
||||
.ThenBy(o => o.Id, StringComparer.Ordinal)
|
||||
.LastOrDefault()
|
||||
?.ContentHash;
|
||||
|
||||
items.Add(new GraphStatusItem(
|
||||
purl,
|
||||
new GraphOverlaySummary(open, notAffected, underInvestigation, noStatement),
|
||||
latest,
|
||||
sources.ToArray(),
|
||||
lastHash));
|
||||
}
|
||||
|
||||
return items;
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,128 @@
|
||||
|
||||
using StellaOps.Excititor.Core.Observations;
|
||||
using StellaOps.Excititor.WebService.Contracts;
|
||||
using System;
|
||||
using System.Collections.Generic;
|
||||
using System.Collections.Immutable;
|
||||
using System.Linq;
|
||||
|
||||
namespace StellaOps.Excititor.WebService.Graph;
|
||||
|
||||
internal static class GraphTooltipFactory
|
||||
{
|
||||
public static IReadOnlyList<GraphTooltipItem> Build(
|
||||
IReadOnlyList<string> orderedPurls,
|
||||
IReadOnlyList<VexObservation> observations,
|
||||
bool includeJustifications,
|
||||
int maxItemsPerPurl)
|
||||
{
|
||||
if (orderedPurls is null)
|
||||
{
|
||||
throw new ArgumentNullException(nameof(orderedPurls));
|
||||
}
|
||||
|
||||
if (observations is null)
|
||||
{
|
||||
throw new ArgumentNullException(nameof(observations));
|
||||
}
|
||||
|
||||
if (maxItemsPerPurl <= 0)
|
||||
{
|
||||
throw new ArgumentOutOfRangeException(nameof(maxItemsPerPurl));
|
||||
}
|
||||
|
||||
var requested = new HashSet<string>(orderedPurls, StringComparer.OrdinalIgnoreCase);
|
||||
var byPurl = orderedPurls.ToDictionary(
|
||||
keySelector: static purl => purl,
|
||||
elementSelector: static _ => new List<GraphTooltipObservation>(),
|
||||
comparer: StringComparer.OrdinalIgnoreCase);
|
||||
|
||||
foreach (var observation in observations)
|
||||
{
|
||||
var linksetPurls = observation.Linkset.Purls;
|
||||
foreach (var statement in observation.Statements)
|
||||
{
|
||||
var targets = CollectTargets(statement, linksetPurls, requested);
|
||||
if (targets.Count == 0)
|
||||
{
|
||||
continue;
|
||||
}
|
||||
|
||||
var payload = new GraphTooltipObservation(
|
||||
observation.ObservationId,
|
||||
statement.VulnerabilityId,
|
||||
statement.Status.ToString().ToLowerInvariant(),
|
||||
includeJustifications ? statement.Justification?.ToString() : null,
|
||||
observation.ProviderId,
|
||||
observation.CreatedAt,
|
||||
observation.Upstream.ContentHash,
|
||||
observation.Upstream.Signature.Signature);
|
||||
|
||||
foreach (var target in targets)
|
||||
{
|
||||
byPurl[target].Add(payload);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
var items = new List<GraphTooltipItem>(orderedPurls.Count);
|
||||
foreach (var purl in orderedPurls)
|
||||
{
|
||||
if (!byPurl.TryGetValue(purl, out var observationsForPurl))
|
||||
{
|
||||
items.Add(new GraphTooltipItem(purl, Array.Empty<GraphTooltipObservation>(), false));
|
||||
continue;
|
||||
}
|
||||
|
||||
var ordered = observationsForPurl
|
||||
.OrderByDescending(static o => o.ModifiedAt)
|
||||
.ThenBy(static o => o.AdvisoryId, StringComparer.Ordinal)
|
||||
.ThenBy(static o => o.ProviderId, StringComparer.OrdinalIgnoreCase)
|
||||
.ThenBy(static o => o.ObservationId, StringComparer.Ordinal)
|
||||
.ToList();
|
||||
|
||||
var truncated = ordered.Count > maxItemsPerPurl;
|
||||
var limited = truncated ? ordered.Take(maxItemsPerPurl).ToList() : ordered;
|
||||
|
||||
items.Add(new GraphTooltipItem(purl, limited, truncated));
|
||||
}
|
||||
|
||||
return items;
|
||||
}
|
||||
|
||||
private static List<string> CollectTargets(
|
||||
VexObservationStatement statement,
|
||||
ImmutableArray<string> linksetPurls,
|
||||
HashSet<string> requested)
|
||||
{
|
||||
var targets = new List<string>();
|
||||
|
||||
if (!string.IsNullOrWhiteSpace(statement.Purl))
|
||||
{
|
||||
var normalized = statement.Purl.ToLowerInvariant();
|
||||
if (requested.Contains(normalized))
|
||||
{
|
||||
targets.Add(normalized);
|
||||
}
|
||||
}
|
||||
|
||||
if (targets.Count > 0)
|
||||
{
|
||||
return targets;
|
||||
}
|
||||
|
||||
if (!linksetPurls.IsDefaultOrEmpty)
|
||||
{
|
||||
foreach (var purl in linksetPurls)
|
||||
{
|
||||
var normalized = purl?.ToLowerInvariant();
|
||||
if (normalized is not null && requested.Contains(normalized))
|
||||
{
|
||||
targets.Add(normalized);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
return targets;
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,33 @@
|
||||
using System.Collections.Generic;
|
||||
|
||||
namespace StellaOps.Excititor.WebService.Options;
|
||||
|
||||
internal sealed class AirgapOptions
|
||||
{
|
||||
public const string SectionName = "Excititor:Airgap";
|
||||
|
||||
/// <summary>
|
||||
/// Enables sealed-mode enforcement for air-gapped imports.
|
||||
/// When true, external payload URLs are rejected and publisher allowlist is applied.
|
||||
/// </summary>
|
||||
public bool SealedMode { get; set; } = false;
|
||||
|
||||
/// <summary>
|
||||
/// When true, imports must originate from mirror/offline sources (no HTTP/HTTPS URLs).
|
||||
/// </summary>
|
||||
public bool MirrorOnly { get; set; } = true;
|
||||
|
||||
/// <summary>
|
||||
/// Optional allowlist of publishers that may submit bundles while sealed mode is enabled.
|
||||
/// Empty list means allow all.
|
||||
/// </summary>
|
||||
public List<string> TrustedPublishers { get; } = new();
|
||||
|
||||
/// <summary>
|
||||
/// Optional root path for locally stored locker artefacts (portable manifest, evidence NDJSON).
|
||||
/// When set, /evidence/vex/locker/* endpoints will attempt to read files from this root to
|
||||
/// compute deterministic hashes and sizes; otherwise only stored hashes are returned.
|
||||
/// </summary>
|
||||
public string? LockerRootPath { get; set; }
|
||||
= null;
|
||||
}
|
||||
@@ -0,0 +1,53 @@
|
||||
using System;
|
||||
|
||||
namespace StellaOps.Excititor.WebService.Options;
|
||||
|
||||
internal sealed class ExcititorObservabilityOptions
|
||||
{
|
||||
public TimeSpan IngestWarningThreshold { get; set; } = TimeSpan.FromHours(6);
|
||||
|
||||
public TimeSpan IngestCriticalThreshold { get; set; } = TimeSpan.FromHours(24);
|
||||
|
||||
public TimeSpan LinkWarningThreshold { get; set; } = TimeSpan.FromMinutes(15);
|
||||
|
||||
public TimeSpan LinkCriticalThreshold { get; set; } = TimeSpan.FromHours(1);
|
||||
|
||||
public TimeSpan SignatureWindow { get; set; } = TimeSpan.FromHours(12);
|
||||
|
||||
public double SignatureHealthyCoverage { get; set; } = 0.8;
|
||||
|
||||
public double SignatureWarningCoverage { get; set; } = 0.5;
|
||||
|
||||
public TimeSpan ConflictTrendWindow { get; set; } = TimeSpan.FromHours(24);
|
||||
|
||||
public int ConflictTrendBucketMinutes { get; set; } = 60;
|
||||
|
||||
public double ConflictWarningRatio { get; set; } = 0.15;
|
||||
|
||||
public double ConflictCriticalRatio { get; set; } = 0.3;
|
||||
|
||||
public int MaxConnectorDetails { get; set; } = 50;
|
||||
|
||||
internal TimeSpan GetPositive(TimeSpan candidate, TimeSpan fallback)
|
||||
=> candidate <= TimeSpan.Zero ? fallback : candidate;
|
||||
|
||||
internal double ClampRatio(double value, double fallback)
|
||||
{
|
||||
if (double.IsNaN(value) || double.IsInfinity(value))
|
||||
{
|
||||
return fallback;
|
||||
}
|
||||
|
||||
if (value < 0)
|
||||
{
|
||||
return 0;
|
||||
}
|
||||
|
||||
if (value > 1)
|
||||
{
|
||||
return 1;
|
||||
}
|
||||
|
||||
return value;
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,23 @@
|
||||
using System;
|
||||
using System.Collections.Generic;
|
||||
|
||||
namespace StellaOps.Excititor.WebService.Options;
|
||||
|
||||
internal sealed class ExcititorTelemetryOptions
|
||||
{
|
||||
public bool Enabled { get; set; } = true;
|
||||
|
||||
public bool EnableTracing { get; set; } = true;
|
||||
|
||||
public bool EnableMetrics { get; set; } = true;
|
||||
|
||||
public bool ExportConsole { get; set; }
|
||||
|
||||
public string? ServiceName { get; set; }
|
||||
|
||||
public string? OtlpEndpoint { get; set; }
|
||||
|
||||
public Dictionary<string, string> OtlpHeaders { get; } = new(StringComparer.OrdinalIgnoreCase);
|
||||
|
||||
public Dictionary<string, string> ResourceAttributes { get; } = new(StringComparer.OrdinalIgnoreCase);
|
||||
}
|
||||
@@ -0,0 +1,14 @@
|
||||
namespace StellaOps.Excititor.WebService.Options;
|
||||
|
||||
/// <summary>
|
||||
/// Configuration for graph linkouts and overlays.
|
||||
/// </summary>
|
||||
public sealed class GraphOptions
|
||||
{
|
||||
public int MaxPurls { get; set; } = 500;
|
||||
public int MaxAdvisoriesPerPurl { get; set; } = 200;
|
||||
public int OverlayTtlSeconds { get; set; } = 300;
|
||||
public bool UsePostgresOverlayStore { get; set; } = true;
|
||||
public int MaxTooltipItemsPerPurl { get; set; } = 50;
|
||||
public int MaxTooltipTotal { get; set; } = 1000;
|
||||
}
|
||||
252
src/Concelier/StellaOps.Excititor.WebService/Program.Helpers.cs
Normal file
252
src/Concelier/StellaOps.Excititor.WebService/Program.Helpers.cs
Normal file
@@ -0,0 +1,252 @@
|
||||
using Microsoft.AspNetCore.Http;
|
||||
using Microsoft.Extensions.Primitives;
|
||||
using StellaOps.Excititor.Core;
|
||||
using StellaOps.Excititor.Core.Aoc;
|
||||
using StellaOps.Excititor.Core.Storage;
|
||||
using StellaOps.Excititor.WebService.Contracts;
|
||||
using StellaOps.Excititor.WebService.Services;
|
||||
using System;
|
||||
using System.Collections.Generic;
|
||||
using System.Collections.Immutable;
|
||||
using System.Globalization;
|
||||
using System.Linq;
|
||||
using System.Text;
|
||||
|
||||
public partial class Program
|
||||
{
|
||||
private const string TenantHeaderName = "X-Stella-Tenant";
|
||||
|
||||
internal static bool TryResolveTenant(HttpContext context, VexStorageOptions options, bool requireHeader, out string tenant, out IResult? problem)
|
||||
{
|
||||
tenant = options.DefaultTenant;
|
||||
problem = null;
|
||||
|
||||
if (context.Request.Headers.TryGetValue(TenantHeaderName, out var headerValues) && headerValues.Count > 0)
|
||||
{
|
||||
var requestedTenant = headerValues[0]?.Trim();
|
||||
if (string.IsNullOrEmpty(requestedTenant))
|
||||
{
|
||||
problem = Results.Problem(detail: "X-Stella-Tenant header must not be empty.", statusCode: StatusCodes.Status400BadRequest, title: "Validation error");
|
||||
return false;
|
||||
}
|
||||
|
||||
if (!string.Equals(requestedTenant, options.DefaultTenant, StringComparison.OrdinalIgnoreCase))
|
||||
{
|
||||
var detail = string.Format(CultureInfo.InvariantCulture, "Tenant '{0}' is not allowed for this Excititor deployment.", requestedTenant);
|
||||
problem = Results.Problem(detail: detail, statusCode: StatusCodes.Status403Forbidden, title: "Forbidden");
|
||||
return false;
|
||||
}
|
||||
|
||||
tenant = requestedTenant;
|
||||
return true;
|
||||
}
|
||||
|
||||
if (requireHeader)
|
||||
{
|
||||
var detail = string.Format(CultureInfo.InvariantCulture, "{0} header is required.", TenantHeaderName);
|
||||
problem = Results.Problem(detail: detail, statusCode: StatusCodes.Status400BadRequest, title: "Validation error");
|
||||
return false;
|
||||
}
|
||||
|
||||
return true;
|
||||
}
|
||||
|
||||
private static bool TryDecodeCursor(string? cursor, out DateTimeOffset timestamp, out string digest)
|
||||
{
|
||||
timestamp = default;
|
||||
digest = string.Empty;
|
||||
if (string.IsNullOrWhiteSpace(cursor))
|
||||
{
|
||||
return false;
|
||||
}
|
||||
|
||||
try
|
||||
{
|
||||
var payload = Encoding.UTF8.GetString(Convert.FromBase64String(cursor));
|
||||
var parts = payload.Split('|');
|
||||
if (parts.Length != 2)
|
||||
{
|
||||
return false;
|
||||
}
|
||||
|
||||
if (!DateTimeOffset.TryParse(parts[0], CultureInfo.InvariantCulture, DateTimeStyles.AssumeUniversal, out timestamp))
|
||||
{
|
||||
return false;
|
||||
}
|
||||
|
||||
digest = parts[1];
|
||||
return true;
|
||||
}
|
||||
catch
|
||||
{
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
private static string EncodeCursor(DateTime timestamp, string digest)
|
||||
{
|
||||
var payload = string.Format(CultureInfo.InvariantCulture, "{0:O}|{1}", timestamp, digest);
|
||||
return Convert.ToBase64String(Encoding.UTF8.GetBytes(payload));
|
||||
}
|
||||
|
||||
private static IResult ValidationProblem(string message)
|
||||
=> Results.Problem(detail: message, statusCode: StatusCodes.Status400BadRequest, title: "Validation error");
|
||||
|
||||
private static IResult MapGuardException(ExcititorAocGuardException exception)
|
||||
{
|
||||
var violations = exception.Violations.Select(violation => new
|
||||
{
|
||||
code = violation.ErrorCode,
|
||||
path = violation.Path,
|
||||
message = violation.Message
|
||||
});
|
||||
|
||||
return Results.Problem(
|
||||
detail: "VEX document failed Aggregation-Only Contract validation.",
|
||||
statusCode: StatusCodes.Status400BadRequest,
|
||||
title: "AOC violation",
|
||||
extensions: new Dictionary<string, object?>
|
||||
{
|
||||
["violations"] = violations.ToArray(),
|
||||
["primaryCode"] = exception.PrimaryErrorCode,
|
||||
});
|
||||
}
|
||||
|
||||
private static ImmutableHashSet<string> BuildStringFilterSet(StringValues values)
|
||||
{
|
||||
if (values.Count == 0)
|
||||
{
|
||||
return ImmutableHashSet<string>.Empty;
|
||||
}
|
||||
|
||||
var builder = ImmutableHashSet.CreateBuilder<string>(StringComparer.OrdinalIgnoreCase);
|
||||
foreach (var value in values)
|
||||
{
|
||||
if (!string.IsNullOrWhiteSpace(value))
|
||||
{
|
||||
builder.Add(value.Trim());
|
||||
}
|
||||
}
|
||||
|
||||
return builder.ToImmutable();
|
||||
}
|
||||
|
||||
private static ImmutableHashSet<VexClaimStatus> BuildStatusFilter(StringValues values)
|
||||
{
|
||||
if (values.Count == 0)
|
||||
{
|
||||
return ImmutableHashSet<VexClaimStatus>.Empty;
|
||||
}
|
||||
|
||||
var builder = ImmutableHashSet.CreateBuilder<VexClaimStatus>();
|
||||
foreach (var value in values)
|
||||
{
|
||||
if (Enum.TryParse<VexClaimStatus>(value, ignoreCase: true, out var status))
|
||||
{
|
||||
builder.Add(status);
|
||||
}
|
||||
}
|
||||
|
||||
return builder.ToImmutable();
|
||||
}
|
||||
|
||||
internal static DateTimeOffset? ParseSinceTimestamp(StringValues values)
|
||||
{
|
||||
if (values.Count == 0)
|
||||
{
|
||||
return null;
|
||||
}
|
||||
|
||||
var candidate = values[0];
|
||||
return DateTimeOffset.TryParse(candidate, CultureInfo.InvariantCulture, DateTimeStyles.AssumeUniversal | DateTimeStyles.AdjustToUniversal, out var parsed)
|
||||
? parsed
|
||||
: null;
|
||||
}
|
||||
|
||||
private static int ResolveLimit(StringValues values, int defaultValue, int min, int max)
|
||||
{
|
||||
if (values.Count == 0)
|
||||
{
|
||||
return defaultValue;
|
||||
}
|
||||
|
||||
if (!int.TryParse(values[0], NumberStyles.Integer, CultureInfo.InvariantCulture, out var parsed))
|
||||
{
|
||||
return defaultValue;
|
||||
}
|
||||
|
||||
return Math.Clamp(parsed, min, max);
|
||||
}
|
||||
|
||||
private static IReadOnlyList<string> NormalizePurls(string[]? purls)
|
||||
{
|
||||
if (purls is null || purls.Length == 0)
|
||||
{
|
||||
return Array.Empty<string>();
|
||||
}
|
||||
|
||||
var seen = new HashSet<string>(StringComparer.OrdinalIgnoreCase);
|
||||
var ordered = new List<string>(purls.Length);
|
||||
foreach (var purl in purls)
|
||||
{
|
||||
var trimmed = purl?.Trim();
|
||||
if (string.IsNullOrWhiteSpace(trimmed))
|
||||
{
|
||||
continue;
|
||||
}
|
||||
|
||||
var normalized = trimmed.ToLowerInvariant();
|
||||
if (seen.Add(normalized))
|
||||
{
|
||||
ordered.Add(normalized);
|
||||
}
|
||||
}
|
||||
|
||||
return ordered;
|
||||
}
|
||||
|
||||
private static VexObservationStatementResponse ToResponse(VexObservationStatementProjection projection)
|
||||
{
|
||||
var scope = projection.Scope;
|
||||
var document = projection.Document;
|
||||
var signature = projection.Signature;
|
||||
|
||||
return new VexObservationStatementResponse(
|
||||
projection.ObservationId,
|
||||
projection.ProviderId,
|
||||
projection.Status.ToString().ToLowerInvariant(),
|
||||
projection.Justification?.ToString().ToLowerInvariant(),
|
||||
projection.Detail,
|
||||
projection.FirstSeen,
|
||||
projection.LastSeen,
|
||||
new VexObservationScopeResponse(
|
||||
scope.Key,
|
||||
scope.Name,
|
||||
scope.Version,
|
||||
scope.Purl,
|
||||
scope.Cpe,
|
||||
scope.ComponentIdentifiers),
|
||||
projection.Anchors,
|
||||
new VexObservationDocumentResponse(
|
||||
document.Digest,
|
||||
document.Format.ToString().ToLowerInvariant(),
|
||||
document.Revision,
|
||||
document.SourceUri.ToString()),
|
||||
signature is null
|
||||
? null
|
||||
: new VexObservationSignatureResponse(
|
||||
signature.Type,
|
||||
signature.KeyId,
|
||||
signature.Issuer,
|
||||
signature.VerifiedAt));
|
||||
}
|
||||
|
||||
private sealed record CachedGraphStatus(
|
||||
IReadOnlyList<GraphStatusItem> Items,
|
||||
DateTimeOffset CachedAt);
|
||||
|
||||
internal static string[] NormalizeValues(StringValues values) =>
|
||||
values.Where(static v => !string.IsNullOrWhiteSpace(v))
|
||||
.Select(static v => v!.Trim())
|
||||
.ToArray();
|
||||
}
|
||||
2494
src/Concelier/StellaOps.Excititor.WebService/Program.cs
Normal file
2494
src/Concelier/StellaOps.Excititor.WebService/Program.cs
Normal file
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,3 @@
|
||||
// NOTE: Unable to update Program.cs usings via apply_patch because of file size and PTY limits.
|
||||
// Desired additions:
|
||||
// using StellaOps.Excititor.WebService.Options;
|
||||
@@ -0,0 +1,4 @@
|
||||
using System.Runtime.CompilerServices;
|
||||
|
||||
[assembly: InternalsVisibleTo("StellaOps.Excititor.WebService.Tests")]
|
||||
[assembly: InternalsVisibleTo("StellaOps.Excititor.Core.UnitTests")]
|
||||
@@ -0,0 +1,14 @@
|
||||
{
|
||||
"profiles": {
|
||||
"StellaOps.Excititor.WebService": {
|
||||
"commandName": "Project",
|
||||
"launchBrowser": true,
|
||||
"environmentVariables": {
|
||||
"ASPNETCORE_ENVIRONMENT": "Development",
|
||||
"STELLAOPS_WEBSERVICES_CORS": "true",
|
||||
"STELLAOPS_WEBSERVICES_CORS_ORIGIN": "https://stella-ops.local,https://stella-ops.local:10000,https://localhost:10000"
|
||||
},
|
||||
"applicationUrl": "https://localhost:10100;http://localhost:10101"
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,20 @@
|
||||
namespace StellaOps.Excititor.WebService.Security;
|
||||
|
||||
/// <summary>
|
||||
/// Named authorization policy constants for the Excititor service.
|
||||
/// These policies map to OAuth2 scopes enforced at the endpoint level.
|
||||
/// </summary>
|
||||
internal static class ExcititorPolicies
|
||||
{
|
||||
/// <summary>Policy requiring the vex.admin scope (approve/reject, ingest control, reconcile).</summary>
|
||||
public const string VexAdmin = "excititor.vex.admin";
|
||||
|
||||
/// <summary>Policy requiring the vex.read scope (read-only VEX data, observations, linksets, attestations).</summary>
|
||||
public const string VexRead = "excititor.vex.read";
|
||||
|
||||
/// <summary>Policy requiring the vex.ingest scope (VEX data ingestion operations).</summary>
|
||||
public const string VexIngest = "excititor.vex.ingest";
|
||||
|
||||
/// <summary>Policy requiring the vex.attest scope (Rekor attestation operations).</summary>
|
||||
public const string VexAttest = "excititor.vex.attest";
|
||||
}
|
||||
@@ -0,0 +1,108 @@
|
||||
|
||||
using StellaOps.Excititor.WebService.Contracts;
|
||||
using System;
|
||||
using System.Collections.Generic;
|
||||
using System.Globalization;
|
||||
using System.Linq;
|
||||
using System.Text.RegularExpressions;
|
||||
|
||||
namespace StellaOps.Excititor.WebService.Services;
|
||||
|
||||
internal sealed class AirgapImportValidator
|
||||
{
|
||||
private static readonly TimeSpan AllowedSkew = TimeSpan.FromSeconds(5);
|
||||
private static readonly Regex Sha256Pattern = new(@"^sha256:[A-Fa-f0-9]{64}$", RegexOptions.Compiled);
|
||||
private static readonly Regex MirrorGenerationPattern = new(@"^[0-9]+$", RegexOptions.Compiled);
|
||||
|
||||
public IReadOnlyList<ValidationError> Validate(AirgapImportRequest request, DateTimeOffset nowUtc)
|
||||
{
|
||||
var errors = new List<ValidationError>();
|
||||
|
||||
if (request is null)
|
||||
{
|
||||
errors.Add(new ValidationError("invalid_request", "Request body is required."));
|
||||
return errors;
|
||||
}
|
||||
|
||||
if (string.IsNullOrWhiteSpace(request.BundleId))
|
||||
{
|
||||
errors.Add(new ValidationError("bundle_id_missing", "bundleId is required."));
|
||||
}
|
||||
else if (request.BundleId.Length > 256)
|
||||
{
|
||||
errors.Add(new ValidationError("bundle_id_too_long", "bundleId must be <= 256 characters."));
|
||||
}
|
||||
|
||||
if (string.IsNullOrWhiteSpace(request.MirrorGeneration))
|
||||
{
|
||||
errors.Add(new ValidationError("mirror_generation_missing", "mirrorGeneration is required."));
|
||||
}
|
||||
else if (!MirrorGenerationPattern.IsMatch(request.MirrorGeneration))
|
||||
{
|
||||
errors.Add(new ValidationError("mirror_generation_invalid", "mirrorGeneration must be a numeric string."));
|
||||
}
|
||||
|
||||
if (string.IsNullOrWhiteSpace(request.Publisher))
|
||||
{
|
||||
errors.Add(new ValidationError("publisher_missing", "publisher is required."));
|
||||
}
|
||||
else if (request.Publisher.Length > 256)
|
||||
{
|
||||
errors.Add(new ValidationError("publisher_too_long", "publisher must be <= 256 characters."));
|
||||
}
|
||||
|
||||
if (string.IsNullOrWhiteSpace(request.PayloadHash))
|
||||
{
|
||||
errors.Add(new ValidationError("payload_hash_missing", "payloadHash is required."));
|
||||
}
|
||||
else if (!Sha256Pattern.IsMatch(request.PayloadHash))
|
||||
{
|
||||
errors.Add(new ValidationError("payload_hash_invalid", "payloadHash must be sha256:<64-hex>."));
|
||||
}
|
||||
|
||||
if (string.IsNullOrWhiteSpace(request.Signature))
|
||||
{
|
||||
errors.Add(new ValidationError("AIRGAP_SIGNATURE_MISSING", "signature is required for air-gapped imports."));
|
||||
}
|
||||
else if (!IsBase64(request.Signature))
|
||||
{
|
||||
errors.Add(new ValidationError("AIRGAP_SIGNATURE_INVALID", "signature must be base64-encoded."));
|
||||
}
|
||||
|
||||
if (request.SignedAt is null)
|
||||
{
|
||||
errors.Add(new ValidationError("signed_at_missing", "signedAt is required."));
|
||||
}
|
||||
else
|
||||
{
|
||||
var delta = (nowUtc - request.SignedAt.Value).Duration();
|
||||
if (delta > AllowedSkew)
|
||||
{
|
||||
errors.Add(new ValidationError(
|
||||
"AIRGAP_PAYLOAD_STALE",
|
||||
$"signedAt exceeds allowed skew of {AllowedSkew.TotalSeconds.ToString(CultureInfo.InvariantCulture)} seconds."));
|
||||
}
|
||||
}
|
||||
|
||||
return errors;
|
||||
}
|
||||
|
||||
private static bool IsBase64(string value)
|
||||
{
|
||||
if (string.IsNullOrWhiteSpace(value) || value.Length % 4 != 0)
|
||||
{
|
||||
return false;
|
||||
}
|
||||
try
|
||||
{
|
||||
_ = Convert.FromBase64String(value);
|
||||
return true;
|
||||
}
|
||||
catch
|
||||
{
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
public readonly record struct ValidationError(string Code, string Message);
|
||||
}
|
||||
@@ -0,0 +1,66 @@
|
||||
|
||||
using Microsoft.Extensions.Logging;
|
||||
using Microsoft.Extensions.Options;
|
||||
using StellaOps.Excititor.WebService.Contracts;
|
||||
using StellaOps.Excititor.WebService.Options;
|
||||
using System;
|
||||
using System.Linq;
|
||||
|
||||
namespace StellaOps.Excititor.WebService.Services;
|
||||
|
||||
internal sealed class AirgapModeEnforcer
|
||||
{
|
||||
private readonly AirgapOptions _options;
|
||||
private readonly ILogger<AirgapModeEnforcer> _logger;
|
||||
|
||||
public AirgapModeEnforcer(IOptions<AirgapOptions> options, ILogger<AirgapModeEnforcer> logger)
|
||||
{
|
||||
_options = options.Value;
|
||||
_logger = logger;
|
||||
}
|
||||
|
||||
public bool Validate(AirgapImportRequest request, out string? errorCode, out string? message)
|
||||
{
|
||||
errorCode = null;
|
||||
message = null;
|
||||
|
||||
if (!_options.SealedMode)
|
||||
{
|
||||
return true;
|
||||
}
|
||||
|
||||
if (_options.MirrorOnly && !string.IsNullOrWhiteSpace(request.PayloadUrl) && LooksLikeExternal(request.PayloadUrl))
|
||||
{
|
||||
errorCode = "AIRGAP_EGRESS_BLOCKED";
|
||||
message = "Sealed mode forbids external payload URLs; stage bundle via mirror/portable media.";
|
||||
_logger.LogWarning("Blocked airgap import because payloadUrl points to external location: {Url}", request.PayloadUrl);
|
||||
return false;
|
||||
}
|
||||
|
||||
if (_options.TrustedPublishers.Count > 0 && !string.IsNullOrWhiteSpace(request.Publisher))
|
||||
{
|
||||
var allowed = _options.TrustedPublishers.Any(p => string.Equals(p, request.Publisher, StringComparison.OrdinalIgnoreCase));
|
||||
if (!allowed)
|
||||
{
|
||||
errorCode = "AIRGAP_SOURCE_UNTRUSTED";
|
||||
message = $"Publisher '{request.Publisher}' is not allowlisted for sealed-mode imports.";
|
||||
_logger.LogWarning("Blocked airgap import because publisher {Publisher} is not allowlisted.", request.Publisher);
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
return true;
|
||||
}
|
||||
|
||||
private static bool LooksLikeExternal(string url)
|
||||
{
|
||||
if (string.IsNullOrWhiteSpace(url))
|
||||
{
|
||||
return false;
|
||||
}
|
||||
|
||||
return url.StartsWith("http://", StringComparison.OrdinalIgnoreCase)
|
||||
|| url.StartsWith("https://", StringComparison.OrdinalIgnoreCase)
|
||||
|| url.StartsWith("ftp://", StringComparison.OrdinalIgnoreCase);
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,83 @@
|
||||
|
||||
using Microsoft.Extensions.Logging;
|
||||
using StellaOps.Excititor.Connectors.Abstractions.Trust;
|
||||
using StellaOps.Excititor.WebService.Contracts;
|
||||
using System;
|
||||
using System.IO;
|
||||
using System.Linq;
|
||||
|
||||
namespace StellaOps.Excititor.WebService.Services;
|
||||
|
||||
internal sealed class AirgapSignerTrustService
|
||||
{
|
||||
private readonly ILogger<AirgapSignerTrustService> _logger;
|
||||
private readonly string? _metadataPath;
|
||||
private ConnectorSignerMetadataSet? _metadata;
|
||||
|
||||
public AirgapSignerTrustService(ILogger<AirgapSignerTrustService> logger)
|
||||
{
|
||||
_logger = logger;
|
||||
_metadataPath = Environment.GetEnvironmentVariable("STELLAOPS_CONNECTOR_SIGNER_METADATA_PATH");
|
||||
}
|
||||
|
||||
public bool Validate(AirgapImportRequest request, out string? errorCode, out string? message)
|
||||
{
|
||||
errorCode = null;
|
||||
message = null;
|
||||
|
||||
if (string.IsNullOrWhiteSpace(_metadataPath) || !File.Exists(_metadataPath))
|
||||
{
|
||||
_logger.LogDebug("Airgap signer metadata not configured; skipping trust enforcement.");
|
||||
return true;
|
||||
}
|
||||
|
||||
_metadata ??= ConnectorSignerMetadataLoader.TryLoad(_metadataPath);
|
||||
if (_metadata is null)
|
||||
{
|
||||
_logger.LogWarning("Failed to load airgap signer metadata from {Path}; allowing import.", _metadataPath);
|
||||
return true;
|
||||
}
|
||||
|
||||
if (string.IsNullOrWhiteSpace(request.Publisher))
|
||||
{
|
||||
errorCode = "AIRGAP_SOURCE_UNTRUSTED";
|
||||
message = "publisher is required for trust enforcement.";
|
||||
return false;
|
||||
}
|
||||
|
||||
if (!_metadata.TryGet(request.Publisher, out var connector))
|
||||
{
|
||||
errorCode = "AIRGAP_SOURCE_UNTRUSTED";
|
||||
message = $"Publisher '{request.Publisher}' is not present in trusted signer metadata.";
|
||||
return false;
|
||||
}
|
||||
|
||||
if (connector.Revoked)
|
||||
{
|
||||
errorCode = "AIRGAP_SOURCE_UNTRUSTED";
|
||||
message = $"Publisher '{request.Publisher}' is revoked.";
|
||||
return false;
|
||||
}
|
||||
|
||||
if (connector.Bundle?.Digest is { } digest && !string.IsNullOrWhiteSpace(digest))
|
||||
{
|
||||
if (!string.Equals(digest.Trim(), request.PayloadHash?.Trim(), StringComparison.OrdinalIgnoreCase))
|
||||
{
|
||||
errorCode = "AIRGAP_PAYLOAD_MISMATCH";
|
||||
message = "Payload hash does not match trusted bundle digest.";
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
// Basic sanity: ensure at least one signer entry exists.
|
||||
if (connector.Signers.IsDefaultOrEmpty || connector.Signers.Sum(s => s.Fingerprints.Length) == 0)
|
||||
{
|
||||
errorCode = "AIRGAP_SOURCE_UNTRUSTED";
|
||||
message = $"Publisher '{request.Publisher}' has no trusted signers configured.";
|
||||
return false;
|
||||
}
|
||||
|
||||
return true;
|
||||
}
|
||||
}
|
||||
|
||||
@@ -0,0 +1,557 @@
|
||||
|
||||
using Microsoft.Extensions.Logging;
|
||||
using Microsoft.Extensions.Options;
|
||||
using StellaOps.Excititor.Connectors.Abstractions;
|
||||
using StellaOps.Excititor.Core;
|
||||
using StellaOps.Excititor.Core.Observations;
|
||||
using StellaOps.Excititor.Core.Storage;
|
||||
using StellaOps.Excititor.WebService.Options;
|
||||
using System.Collections.Generic;
|
||||
using System.Linq;
|
||||
|
||||
namespace StellaOps.Excititor.WebService.Services;
|
||||
|
||||
internal sealed class ExcititorHealthService
|
||||
{
|
||||
private readonly IVexRawStore _rawStore;
|
||||
private readonly IVexLinksetStore _linksetStore;
|
||||
private readonly IVexProviderStore _providerStore;
|
||||
private readonly IVexConnectorStateRepository _stateRepository;
|
||||
private readonly IReadOnlyDictionary<string, VexConnectorDescriptor> _connectors;
|
||||
private readonly TimeProvider _timeProvider;
|
||||
private readonly ExcititorObservabilityOptions _options;
|
||||
private readonly ILogger<ExcititorHealthService> _logger;
|
||||
private readonly string _defaultTenant;
|
||||
|
||||
public ExcititorHealthService(
|
||||
IVexRawStore rawStore,
|
||||
IVexLinksetStore linksetStore,
|
||||
IVexProviderStore providerStore,
|
||||
IVexConnectorStateRepository stateRepository,
|
||||
IEnumerable<IVexConnector> connectors,
|
||||
TimeProvider timeProvider,
|
||||
IOptions<ExcititorObservabilityOptions> options,
|
||||
IOptions<VexStorageOptions> storageOptions,
|
||||
ILogger<ExcititorHealthService> logger)
|
||||
{
|
||||
_rawStore = rawStore ?? throw new ArgumentNullException(nameof(rawStore));
|
||||
_linksetStore = linksetStore ?? throw new ArgumentNullException(nameof(linksetStore));
|
||||
_providerStore = providerStore ?? throw new ArgumentNullException(nameof(providerStore));
|
||||
_stateRepository = stateRepository ?? throw new ArgumentNullException(nameof(stateRepository));
|
||||
_timeProvider = timeProvider ?? TimeProvider.System;
|
||||
_options = options?.Value ?? new ExcititorObservabilityOptions();
|
||||
_logger = logger ?? throw new ArgumentNullException(nameof(logger));
|
||||
var storage = storageOptions?.Value ?? new VexStorageOptions();
|
||||
_defaultTenant = string.IsNullOrWhiteSpace(storage.DefaultTenant)
|
||||
? "default"
|
||||
: storage.DefaultTenant.Trim();
|
||||
|
||||
if (connectors is null)
|
||||
{
|
||||
throw new ArgumentNullException(nameof(connectors));
|
||||
}
|
||||
|
||||
_connectors = connectors
|
||||
.GroupBy(connector => connector.Id, StringComparer.OrdinalIgnoreCase)
|
||||
.ToDictionary(
|
||||
group => group.Key,
|
||||
group => DescribeConnector(group.First()),
|
||||
StringComparer.OrdinalIgnoreCase);
|
||||
}
|
||||
|
||||
public async Task<ExcititorHealthDocument> GetAsync(CancellationToken cancellationToken)
|
||||
{
|
||||
var now = _timeProvider.GetUtcNow();
|
||||
|
||||
var providersTask = _providerStore.ListAsync(cancellationToken).AsTask();
|
||||
var statesTask = _stateRepository.ListAsync(cancellationToken).AsTask();
|
||||
var signatureTask = LoadSignatureSnapshotAsync(now, cancellationToken);
|
||||
var conflictTask = LoadConflictSnapshotAsync(now, cancellationToken);
|
||||
var linkTask = LoadLinkSnapshotAsync(cancellationToken);
|
||||
|
||||
await Task.WhenAll(providersTask, statesTask, signatureTask, conflictTask, linkTask).ConfigureAwait(false);
|
||||
|
||||
var ingest = BuildIngestSection(now, providersTask.Result, statesTask.Result);
|
||||
var link = BuildLinkSection(now, linkTask.Result);
|
||||
var conflicts = BuildConflictSection(conflictTask.Result, link);
|
||||
var signature = BuildSignatureSection(signatureTask.Result);
|
||||
|
||||
return new ExcititorHealthDocument(
|
||||
now,
|
||||
ingest,
|
||||
link,
|
||||
signature,
|
||||
conflicts);
|
||||
}
|
||||
|
||||
private IngestHealthSection BuildIngestSection(
|
||||
DateTimeOffset now,
|
||||
IReadOnlyCollection<VexProvider> providers,
|
||||
IReadOnlyCollection<VexConnectorState> states)
|
||||
{
|
||||
var providerNames = providers
|
||||
.GroupBy(provider => provider.Id, StringComparer.OrdinalIgnoreCase)
|
||||
.ToDictionary(
|
||||
group => group.Key,
|
||||
group => group.First().DisplayName,
|
||||
StringComparer.OrdinalIgnoreCase);
|
||||
|
||||
var stateMap = states
|
||||
.ToDictionary(state => state.ConnectorId, state => state, StringComparer.OrdinalIgnoreCase);
|
||||
|
||||
var warningThreshold = _options.GetPositive(_options.IngestWarningThreshold, TimeSpan.FromHours(6));
|
||||
var criticalThreshold = _options.GetPositive(_options.IngestCriticalThreshold, TimeSpan.FromHours(24));
|
||||
|
||||
var connectorHealth = new List<ConnectorHealth>(_connectors.Count);
|
||||
foreach (var descriptor in _connectors.Values.OrderBy(d => d.Id, StringComparer.OrdinalIgnoreCase))
|
||||
{
|
||||
stateMap.TryGetValue(descriptor.Id, out var state);
|
||||
var displayName = providerNames.TryGetValue(descriptor.Id, out var name)
|
||||
? name
|
||||
: descriptor.DisplayName;
|
||||
|
||||
var lastSuccess = state?.LastSuccessAt ?? state?.LastUpdated;
|
||||
double? lagSeconds = null;
|
||||
if (lastSuccess is not null)
|
||||
{
|
||||
var lag = now - lastSuccess.Value;
|
||||
if (lag < TimeSpan.Zero)
|
||||
{
|
||||
lag = TimeSpan.Zero;
|
||||
}
|
||||
|
||||
lagSeconds = lag.TotalSeconds;
|
||||
}
|
||||
|
||||
var status = DetermineIngestStatus(
|
||||
state?.FailureCount ?? 0,
|
||||
lagSeconds,
|
||||
warningThreshold.TotalSeconds,
|
||||
criticalThreshold.TotalSeconds);
|
||||
|
||||
connectorHealth.Add(new ConnectorHealth(
|
||||
descriptor.Id,
|
||||
displayName,
|
||||
status,
|
||||
lastSuccess,
|
||||
state?.LastUpdated,
|
||||
state?.NextEligibleRun,
|
||||
lagSeconds,
|
||||
state?.FailureCount ?? 0,
|
||||
state?.LastFailureReason));
|
||||
}
|
||||
|
||||
var overallStatus = ReduceStatuses(connectorHealth.Select(ch => ch.Status));
|
||||
double? maxLag = connectorHealth.Any(ch => ch.LagSeconds.HasValue)
|
||||
? connectorHealth.Max(ch => ch.LagSeconds ?? 0d)
|
||||
: null;
|
||||
|
||||
var maxDetails = _options.MaxConnectorDetails <= 0 ? 50 : _options.MaxConnectorDetails;
|
||||
var projected = connectorHealth
|
||||
.OrderByDescending(ch => SeverityRank(ch.Status))
|
||||
.ThenByDescending(ch => ch.LagSeconds ?? -1)
|
||||
.ThenBy(ch => ch.ConnectorId, StringComparer.OrdinalIgnoreCase)
|
||||
.Take(maxDetails)
|
||||
.ToList();
|
||||
|
||||
return new IngestHealthSection(overallStatus, maxLag, projected);
|
||||
}
|
||||
|
||||
private LinkHealthSection BuildLinkSection(DateTimeOffset now, LinkSnapshot snapshot)
|
||||
{
|
||||
TimeSpan? lag = null;
|
||||
if (snapshot.LastUpdatedAt is { } calculatedAt)
|
||||
{
|
||||
lag = now - calculatedAt;
|
||||
if (lag < TimeSpan.Zero)
|
||||
{
|
||||
lag = TimeSpan.Zero;
|
||||
}
|
||||
}
|
||||
|
||||
var warning = _options.GetPositive(_options.LinkWarningThreshold, TimeSpan.FromMinutes(15));
|
||||
var critical = _options.GetPositive(_options.LinkCriticalThreshold, TimeSpan.FromHours(1));
|
||||
|
||||
var status = DetermineLagStatus(lag, warning, critical);
|
||||
|
||||
return new LinkHealthSection(
|
||||
status,
|
||||
snapshot.LastUpdatedAt,
|
||||
lag?.TotalSeconds,
|
||||
snapshot.TotalDocuments,
|
||||
snapshot.DocumentsWithConflicts);
|
||||
}
|
||||
|
||||
private ConflictHealthSection BuildConflictSection(ConflictSnapshot snapshot, LinkHealthSection link)
|
||||
{
|
||||
var warningRatio = _options.ClampRatio(_options.ConflictWarningRatio, 0.15);
|
||||
var criticalRatio = _options.ClampRatio(_options.ConflictCriticalRatio, 0.3);
|
||||
|
||||
string status;
|
||||
if (link.TotalDocuments <= 0)
|
||||
{
|
||||
status = "unknown";
|
||||
}
|
||||
else
|
||||
{
|
||||
var ratio = (double)snapshot.DocumentsWithConflicts / link.TotalDocuments;
|
||||
if (ratio >= criticalRatio)
|
||||
{
|
||||
status = "critical";
|
||||
}
|
||||
else if (ratio >= warningRatio)
|
||||
{
|
||||
status = "warning";
|
||||
}
|
||||
else
|
||||
{
|
||||
status = "healthy";
|
||||
}
|
||||
}
|
||||
|
||||
return new ConflictHealthSection(
|
||||
status,
|
||||
snapshot.WindowStart,
|
||||
snapshot.WindowEnd,
|
||||
snapshot.DocumentsWithConflicts,
|
||||
snapshot.TotalConflicts,
|
||||
snapshot.ByStatus,
|
||||
snapshot.Trend);
|
||||
}
|
||||
|
||||
private SignatureHealthSection BuildSignatureSection(SignatureSnapshot snapshot)
|
||||
{
|
||||
if (snapshot.DocumentsEvaluated == 0)
|
||||
{
|
||||
return new SignatureHealthSection(
|
||||
"unknown",
|
||||
snapshot.WindowStart,
|
||||
snapshot.WindowEnd,
|
||||
0,
|
||||
0,
|
||||
0,
|
||||
0,
|
||||
0,
|
||||
0);
|
||||
}
|
||||
|
||||
var coverage = snapshot.DocumentsEvaluated == 0
|
||||
? 0d
|
||||
: (double)snapshot.Verified / snapshot.DocumentsEvaluated;
|
||||
|
||||
var healthy = _options.ClampRatio(_options.SignatureHealthyCoverage, 0.8);
|
||||
var warning = _options.ClampRatio(_options.SignatureWarningCoverage, 0.5);
|
||||
if (warning > healthy)
|
||||
{
|
||||
warning = healthy;
|
||||
}
|
||||
|
||||
var status = coverage switch
|
||||
{
|
||||
var value when value >= healthy => "healthy",
|
||||
var value when value >= warning => "warning",
|
||||
_ => "critical"
|
||||
};
|
||||
|
||||
var failures = Math.Max(0, snapshot.WithSignatures - snapshot.Verified);
|
||||
var unsigned = Math.Max(0, snapshot.DocumentsEvaluated - snapshot.WithSignatures);
|
||||
|
||||
return new SignatureHealthSection(
|
||||
status,
|
||||
snapshot.WindowStart,
|
||||
snapshot.WindowEnd,
|
||||
snapshot.DocumentsEvaluated,
|
||||
snapshot.WithSignatures,
|
||||
snapshot.Verified,
|
||||
failures,
|
||||
unsigned,
|
||||
coverage);
|
||||
}
|
||||
|
||||
private async Task<SignatureSnapshot> LoadSignatureSnapshotAsync(DateTimeOffset now, CancellationToken cancellationToken)
|
||||
{
|
||||
var window = _options.GetPositive(_options.SignatureWindow, TimeSpan.FromHours(12));
|
||||
var windowStart = now - window;
|
||||
|
||||
var page = await _rawStore.QueryAsync(
|
||||
new VexRawQuery(
|
||||
_defaultTenant,
|
||||
Array.Empty<string>(),
|
||||
Array.Empty<string>(),
|
||||
Array.Empty<VexDocumentFormat>(),
|
||||
windowStart,
|
||||
Until: null,
|
||||
Cursor: null,
|
||||
Limit: 500),
|
||||
cancellationToken).ConfigureAwait(false);
|
||||
|
||||
var evaluated = 0;
|
||||
var withSignatures = 0;
|
||||
var verified = 0;
|
||||
|
||||
foreach (var document in page.Items)
|
||||
{
|
||||
evaluated++;
|
||||
var metadata = document.Metadata;
|
||||
if (metadata.TryGetValue("signature.present", out var presentValue) &&
|
||||
bool.TryParse(presentValue, out var present) &&
|
||||
present)
|
||||
{
|
||||
withSignatures++;
|
||||
}
|
||||
|
||||
if (metadata.TryGetValue("signature.verified", out var verifiedValue) &&
|
||||
bool.TryParse(verifiedValue, out var verifiedFlag) &&
|
||||
verifiedFlag)
|
||||
{
|
||||
verified++;
|
||||
}
|
||||
}
|
||||
|
||||
return new SignatureSnapshot(windowStart, now, evaluated, withSignatures, verified);
|
||||
}
|
||||
|
||||
private async Task<LinkSnapshot> LoadLinkSnapshotAsync(CancellationToken cancellationToken)
|
||||
{
|
||||
long totalDocuments = 0;
|
||||
long conflictDocuments = 0;
|
||||
DateTimeOffset? lastUpdated = null;
|
||||
|
||||
try
|
||||
{
|
||||
totalDocuments = await _linksetStore.CountAsync(_defaultTenant, cancellationToken).ConfigureAwait(false);
|
||||
conflictDocuments = await _linksetStore.CountWithConflictsAsync(_defaultTenant, cancellationToken).ConfigureAwait(false);
|
||||
|
||||
var conflictSample = await _linksetStore.FindWithConflictsAsync(_defaultTenant, 1, cancellationToken).ConfigureAwait(false);
|
||||
if (conflictSample.Count > 0)
|
||||
{
|
||||
lastUpdated = conflictSample[0].UpdatedAt;
|
||||
}
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
_logger.LogWarning(ex, "Failed to compute linkset counts.");
|
||||
}
|
||||
|
||||
return new LinkSnapshot(lastUpdated, totalDocuments, conflictDocuments);
|
||||
}
|
||||
|
||||
private async Task<ConflictSnapshot> LoadConflictSnapshotAsync(DateTimeOffset now, CancellationToken cancellationToken)
|
||||
{
|
||||
var window = _options.GetPositive(_options.ConflictTrendWindow, TimeSpan.FromHours(24));
|
||||
var windowStart = now - window;
|
||||
IReadOnlyList<VexLinkset> linksets;
|
||||
try
|
||||
{
|
||||
// Sample conflicted linksets (ordered by updated_at DESC in Postgres implementation)
|
||||
linksets = await _linksetStore.FindWithConflictsAsync(_defaultTenant, 500, cancellationToken).ConfigureAwait(false);
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
_logger.LogWarning(ex, "Failed to load conflict trend window.");
|
||||
linksets = Array.Empty<VexLinkset>();
|
||||
}
|
||||
|
||||
var byStatus = new Dictionary<string, long>(StringComparer.OrdinalIgnoreCase);
|
||||
var timeline = new SortedDictionary<long, long>();
|
||||
long totalConflicts = 0;
|
||||
long docsWithConflicts = 0;
|
||||
var bucketMinutes = Math.Max(1, _options.ConflictTrendBucketMinutes);
|
||||
var bucketTicks = TimeSpan.FromMinutes(bucketMinutes).Ticks;
|
||||
|
||||
foreach (var linkset in linksets)
|
||||
{
|
||||
if (linkset.Disagreements.Length == 0)
|
||||
{
|
||||
continue;
|
||||
}
|
||||
|
||||
docsWithConflicts++;
|
||||
totalConflicts += linkset.Disagreements.Length;
|
||||
|
||||
foreach (var disagreement in linkset.Disagreements)
|
||||
{
|
||||
var status = string.IsNullOrWhiteSpace(disagreement.Status)
|
||||
? "unknown"
|
||||
: disagreement.Status;
|
||||
|
||||
byStatus[status] = byStatus.TryGetValue(status, out var current)
|
||||
? current + 1
|
||||
: 1;
|
||||
}
|
||||
|
||||
var alignedTicks = AlignTicks(linkset.UpdatedAt.UtcDateTime, bucketTicks);
|
||||
timeline[alignedTicks] = timeline.TryGetValue(alignedTicks, out var currentCount)
|
||||
? currentCount + linkset.Disagreements.Length
|
||||
: linkset.Disagreements.Length;
|
||||
}
|
||||
|
||||
var trend = timeline
|
||||
.Select(pair => new ConflictTrendPoint(
|
||||
new DateTimeOffset(pair.Key, TimeSpan.Zero),
|
||||
pair.Value))
|
||||
.ToList();
|
||||
|
||||
return new ConflictSnapshot(
|
||||
windowStart,
|
||||
now,
|
||||
docsWithConflicts,
|
||||
totalConflicts,
|
||||
new Dictionary<string, long>(byStatus, StringComparer.OrdinalIgnoreCase),
|
||||
trend);
|
||||
}
|
||||
|
||||
private static string DetermineIngestStatus(int failureCount, double? lagSeconds, double warningSeconds, double criticalSeconds)
|
||||
{
|
||||
if (failureCount > 0)
|
||||
{
|
||||
return "failing";
|
||||
}
|
||||
|
||||
if (lagSeconds is null)
|
||||
{
|
||||
return "unknown";
|
||||
}
|
||||
|
||||
if (lagSeconds.Value >= criticalSeconds)
|
||||
{
|
||||
return "critical";
|
||||
}
|
||||
|
||||
if (lagSeconds.Value >= warningSeconds)
|
||||
{
|
||||
return "warning";
|
||||
}
|
||||
|
||||
return "healthy";
|
||||
}
|
||||
|
||||
private static string DetermineLagStatus(TimeSpan? lag, TimeSpan warning, TimeSpan critical)
|
||||
{
|
||||
if (lag is null)
|
||||
{
|
||||
return "unknown";
|
||||
}
|
||||
|
||||
if (lag.Value >= critical)
|
||||
{
|
||||
return "critical";
|
||||
}
|
||||
|
||||
if (lag.Value >= warning)
|
||||
{
|
||||
return "warning";
|
||||
}
|
||||
|
||||
return "healthy";
|
||||
}
|
||||
|
||||
private static string ReduceStatuses(IEnumerable<string> statuses)
|
||||
{
|
||||
var highest = "unknown";
|
||||
var highestRank = -1;
|
||||
foreach (var status in statuses)
|
||||
{
|
||||
var rank = SeverityRank(status);
|
||||
if (rank > highestRank)
|
||||
{
|
||||
highestRank = rank;
|
||||
highest = status;
|
||||
}
|
||||
}
|
||||
|
||||
return highest;
|
||||
}
|
||||
|
||||
private static int SeverityRank(string? status)
|
||||
=> status?.ToLowerInvariant() switch
|
||||
{
|
||||
"critical" => 4,
|
||||
"failing" => 3,
|
||||
"warning" => 2,
|
||||
"unknown" => 1,
|
||||
_ => 0,
|
||||
};
|
||||
|
||||
private static long AlignTicks(DateTime dateTimeUtc, long bucketTicks)
|
||||
{
|
||||
var ticks = dateTimeUtc.Ticks;
|
||||
return ticks - (ticks % bucketTicks);
|
||||
}
|
||||
|
||||
private static VexConnectorDescriptor DescribeConnector(IVexConnector connector)
|
||||
=> connector switch
|
||||
{
|
||||
VexConnectorBase baseConnector => baseConnector.Descriptor,
|
||||
_ => new VexConnectorDescriptor(connector.Id, connector.Kind, connector.Id)
|
||||
};
|
||||
|
||||
private sealed record LinkSnapshot(DateTimeOffset? LastUpdatedAt, long TotalDocuments, long DocumentsWithConflicts);
|
||||
|
||||
private sealed record ConflictSnapshot(
|
||||
DateTimeOffset WindowStart,
|
||||
DateTimeOffset WindowEnd,
|
||||
long DocumentsWithConflicts,
|
||||
long TotalConflicts,
|
||||
IReadOnlyDictionary<string, long> ByStatus,
|
||||
IReadOnlyList<ConflictTrendPoint> Trend);
|
||||
|
||||
private sealed record SignatureSnapshot(
|
||||
DateTimeOffset WindowStart,
|
||||
DateTimeOffset WindowEnd,
|
||||
int DocumentsEvaluated,
|
||||
int WithSignatures,
|
||||
int Verified);
|
||||
}
|
||||
|
||||
internal sealed record ExcititorHealthDocument(
|
||||
DateTimeOffset GeneratedAt,
|
||||
IngestHealthSection Ingest,
|
||||
LinkHealthSection Link,
|
||||
SignatureHealthSection Signature,
|
||||
ConflictHealthSection Conflicts);
|
||||
|
||||
internal sealed record IngestHealthSection(
|
||||
string Status,
|
||||
double? MaxLagSeconds,
|
||||
IReadOnlyList<ConnectorHealth> Connectors);
|
||||
|
||||
internal sealed record ConnectorHealth(
|
||||
string ConnectorId,
|
||||
string DisplayName,
|
||||
string Status,
|
||||
DateTimeOffset? LastSuccessAt,
|
||||
DateTimeOffset? LastUpdated,
|
||||
DateTimeOffset? NextEligibleRun,
|
||||
double? LagSeconds,
|
||||
int FailureCount,
|
||||
string? LastFailureReason);
|
||||
|
||||
internal sealed record LinkHealthSection(
|
||||
string Status,
|
||||
DateTimeOffset? LastConsensusAt,
|
||||
double? LagSeconds,
|
||||
long TotalDocuments,
|
||||
long DocumentsWithConflicts);
|
||||
|
||||
internal sealed record SignatureHealthSection(
|
||||
string Status,
|
||||
DateTimeOffset WindowStart,
|
||||
DateTimeOffset WindowEnd,
|
||||
int DocumentsEvaluated,
|
||||
int WithSignatures,
|
||||
int Verified,
|
||||
int Failures,
|
||||
int Unsigned,
|
||||
double Coverage);
|
||||
|
||||
internal sealed record ConflictHealthSection(
|
||||
string Status,
|
||||
DateTimeOffset WindowStart,
|
||||
DateTimeOffset WindowEnd,
|
||||
long DocumentsWithConflicts,
|
||||
long ConflictStatements,
|
||||
IReadOnlyDictionary<string, long> ByStatus,
|
||||
IReadOnlyList<ConflictTrendPoint> Trend);
|
||||
|
||||
internal sealed record ConflictTrendPoint(DateTimeOffset BucketStart, long Conflicts);
|
||||
@@ -0,0 +1,56 @@
|
||||
using Microsoft.Extensions.Caching.Memory;
|
||||
using Microsoft.Extensions.Options;
|
||||
using StellaOps.Excititor.WebService.Contracts;
|
||||
using StellaOps.Excititor.WebService.Options;
|
||||
|
||||
namespace StellaOps.Excititor.WebService.Services;
|
||||
|
||||
public interface IGraphOverlayCache
|
||||
{
|
||||
ValueTask<GraphOverlayCacheHit?> TryGetAsync(string tenant, bool includeJustifications, IReadOnlyList<string> orderedPurls, CancellationToken cancellationToken);
|
||||
|
||||
ValueTask SaveAsync(string tenant, bool includeJustifications, IReadOnlyList<string> orderedPurls, IReadOnlyList<GraphOverlayItem> items, DateTimeOffset cachedAt, CancellationToken cancellationToken);
|
||||
}
|
||||
|
||||
public sealed record GraphOverlayCacheHit(IReadOnlyList<GraphOverlayItem> Items, long AgeMilliseconds);
|
||||
|
||||
internal sealed class GraphOverlayCacheStore : IGraphOverlayCache
|
||||
{
|
||||
private readonly IMemoryCache _memoryCache;
|
||||
private readonly IOptions<GraphOptions> _options;
|
||||
private readonly TimeProvider _timeProvider;
|
||||
|
||||
public GraphOverlayCacheStore(IMemoryCache memoryCache, IOptions<GraphOptions> options, TimeProvider timeProvider)
|
||||
{
|
||||
_memoryCache = memoryCache ?? throw new ArgumentNullException(nameof(memoryCache));
|
||||
_options = options ?? throw new ArgumentNullException(nameof(options));
|
||||
_timeProvider = timeProvider ?? throw new ArgumentNullException(nameof(timeProvider));
|
||||
}
|
||||
|
||||
public ValueTask<GraphOverlayCacheHit?> TryGetAsync(string tenant, bool includeJustifications, IReadOnlyList<string> orderedPurls, CancellationToken cancellationToken)
|
||||
{
|
||||
cancellationToken.ThrowIfCancellationRequested();
|
||||
var key = BuildKey(tenant, includeJustifications, orderedPurls);
|
||||
if (_memoryCache.TryGetValue<CachedOverlay>(key, out var cached) && cached is not null)
|
||||
{
|
||||
var ageMs = (long)Math.Max(0, (_timeProvider.GetUtcNow() - cached.CachedAt).TotalMilliseconds);
|
||||
return ValueTask.FromResult<GraphOverlayCacheHit?>(new GraphOverlayCacheHit(cached.Items, ageMs));
|
||||
}
|
||||
|
||||
return ValueTask.FromResult<GraphOverlayCacheHit?>(null);
|
||||
}
|
||||
|
||||
public ValueTask SaveAsync(string tenant, bool includeJustifications, IReadOnlyList<string> orderedPurls, IReadOnlyList<GraphOverlayItem> items, DateTimeOffset cachedAt, CancellationToken cancellationToken)
|
||||
{
|
||||
cancellationToken.ThrowIfCancellationRequested();
|
||||
var key = BuildKey(tenant, includeJustifications, orderedPurls);
|
||||
var ttl = TimeSpan.FromSeconds(Math.Max(1, _options.Value.OverlayTtlSeconds));
|
||||
_memoryCache.Set(key, new CachedOverlay(items, cachedAt), ttl);
|
||||
return ValueTask.CompletedTask;
|
||||
}
|
||||
|
||||
private static string BuildKey(string tenant, bool includeJustifications, IReadOnlyList<string> orderedPurls)
|
||||
=> $"graph-overlays:{tenant}:{includeJustifications}:{string.Join('|', orderedPurls)}";
|
||||
|
||||
private sealed record CachedOverlay(IReadOnlyList<GraphOverlayItem> Items, DateTimeOffset CachedAt);
|
||||
}
|
||||
@@ -0,0 +1,154 @@
|
||||
using StellaOps.Excititor.WebService.Contracts;
|
||||
|
||||
namespace StellaOps.Excititor.WebService.Services;
|
||||
|
||||
public interface IGraphOverlayStore
|
||||
{
|
||||
ValueTask SaveAsync(string tenant, IReadOnlyList<GraphOverlayItem> overlays, CancellationToken cancellationToken);
|
||||
|
||||
ValueTask<IReadOnlyList<GraphOverlayItem>> FindByPurlsAsync(string tenant, IReadOnlyCollection<string> purls, CancellationToken cancellationToken);
|
||||
|
||||
ValueTask<IReadOnlyList<GraphOverlayItem>> FindByAdvisoriesAsync(string tenant, IReadOnlyCollection<string> advisories, int limit, CancellationToken cancellationToken);
|
||||
|
||||
ValueTask<IReadOnlyList<GraphOverlayItem>> FindWithConflictsAsync(string tenant, int limit, CancellationToken cancellationToken);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// In-memory overlay store placeholder until Postgres materialization is added.
|
||||
/// </summary>
|
||||
public sealed class InMemoryGraphOverlayStore : IGraphOverlayStore
|
||||
{
|
||||
private readonly Dictionary<string, Dictionary<string, List<GraphOverlayItem>>> _store = new(StringComparer.OrdinalIgnoreCase);
|
||||
private readonly object _lock = new();
|
||||
|
||||
public ValueTask SaveAsync(string tenant, IReadOnlyList<GraphOverlayItem> overlays, CancellationToken cancellationToken)
|
||||
{
|
||||
cancellationToken.ThrowIfCancellationRequested();
|
||||
lock (_lock)
|
||||
{
|
||||
if (!_store.TryGetValue(tenant, out var byPurl))
|
||||
{
|
||||
byPurl = new Dictionary<string, List<GraphOverlayItem>>(StringComparer.OrdinalIgnoreCase);
|
||||
_store[tenant] = byPurl;
|
||||
}
|
||||
|
||||
foreach (var overlay in overlays)
|
||||
{
|
||||
if (!byPurl.TryGetValue(overlay.Purl, out var list))
|
||||
{
|
||||
list = new List<GraphOverlayItem>();
|
||||
byPurl[overlay.Purl] = list;
|
||||
}
|
||||
|
||||
// replace existing advisory/source entry for deterministic latest overlay
|
||||
var existingIndex = list.FindIndex(o =>
|
||||
string.Equals(o.AdvisoryId, overlay.AdvisoryId, StringComparison.OrdinalIgnoreCase) &&
|
||||
string.Equals(o.Source, overlay.Source, StringComparison.OrdinalIgnoreCase));
|
||||
if (existingIndex >= 0)
|
||||
{
|
||||
list[existingIndex] = overlay;
|
||||
}
|
||||
else
|
||||
{
|
||||
list.Add(overlay);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
return ValueTask.CompletedTask;
|
||||
}
|
||||
|
||||
public ValueTask<IReadOnlyList<GraphOverlayItem>> FindByPurlsAsync(string tenant, IReadOnlyCollection<string> purls, CancellationToken cancellationToken)
|
||||
{
|
||||
cancellationToken.ThrowIfCancellationRequested();
|
||||
if (purls.Count == 0)
|
||||
{
|
||||
return ValueTask.FromResult<IReadOnlyList<GraphOverlayItem>>(Array.Empty<GraphOverlayItem>());
|
||||
}
|
||||
|
||||
lock (_lock)
|
||||
{
|
||||
if (!_store.TryGetValue(tenant, out var byPurl))
|
||||
{
|
||||
return ValueTask.FromResult<IReadOnlyList<GraphOverlayItem>>(Array.Empty<GraphOverlayItem>());
|
||||
}
|
||||
|
||||
var ordered = new List<GraphOverlayItem>();
|
||||
foreach (var purl in purls)
|
||||
{
|
||||
if (byPurl.TryGetValue(purl, out var list))
|
||||
{
|
||||
// Order overlays deterministically by advisory + source for stable outputs
|
||||
ordered.AddRange(list
|
||||
.OrderBy(o => o.AdvisoryId, StringComparer.OrdinalIgnoreCase)
|
||||
.ThenBy(o => o.Source, StringComparer.OrdinalIgnoreCase));
|
||||
}
|
||||
}
|
||||
|
||||
return ValueTask.FromResult<IReadOnlyList<GraphOverlayItem>>(ordered);
|
||||
}
|
||||
}
|
||||
|
||||
public ValueTask<IReadOnlyList<GraphOverlayItem>> FindByAdvisoriesAsync(string tenant, IReadOnlyCollection<string> advisories, int limit, CancellationToken cancellationToken)
|
||||
{
|
||||
cancellationToken.ThrowIfCancellationRequested();
|
||||
if (advisories.Count == 0)
|
||||
{
|
||||
return ValueTask.FromResult<IReadOnlyList<GraphOverlayItem>>(Array.Empty<GraphOverlayItem>());
|
||||
}
|
||||
|
||||
lock (_lock)
|
||||
{
|
||||
if (!_store.TryGetValue(tenant, out var byPurl))
|
||||
{
|
||||
return ValueTask.FromResult<IReadOnlyList<GraphOverlayItem>>(Array.Empty<GraphOverlayItem>());
|
||||
}
|
||||
|
||||
var results = new List<GraphOverlayItem>();
|
||||
foreach (var kvp in byPurl)
|
||||
{
|
||||
foreach (var overlay in kvp.Value)
|
||||
{
|
||||
if (advisories.Contains(overlay.AdvisoryId, StringComparer.OrdinalIgnoreCase))
|
||||
{
|
||||
results.Add(overlay);
|
||||
if (results.Count >= limit)
|
||||
{
|
||||
return ValueTask.FromResult<IReadOnlyList<GraphOverlayItem>>(results);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
return ValueTask.FromResult<IReadOnlyList<GraphOverlayItem>>(results
|
||||
.OrderBy(o => o.AdvisoryId, StringComparer.OrdinalIgnoreCase)
|
||||
.ThenBy(o => o.Purl, StringComparer.OrdinalIgnoreCase)
|
||||
.ThenBy(o => o.Source, StringComparer.OrdinalIgnoreCase)
|
||||
.Take(limit)
|
||||
.ToList());
|
||||
}
|
||||
}
|
||||
|
||||
public ValueTask<IReadOnlyList<GraphOverlayItem>> FindWithConflictsAsync(string tenant, int limit, CancellationToken cancellationToken)
|
||||
{
|
||||
cancellationToken.ThrowIfCancellationRequested();
|
||||
lock (_lock)
|
||||
{
|
||||
if (!_store.TryGetValue(tenant, out var byPurl))
|
||||
{
|
||||
return ValueTask.FromResult<IReadOnlyList<GraphOverlayItem>>(Array.Empty<GraphOverlayItem>());
|
||||
}
|
||||
|
||||
var results = byPurl.Values
|
||||
.SelectMany(list => list)
|
||||
.Where(o => o.Conflicts.Count > 0)
|
||||
.OrderBy(o => o.Purl, StringComparer.OrdinalIgnoreCase)
|
||||
.ThenBy(o => o.AdvisoryId, StringComparer.OrdinalIgnoreCase)
|
||||
.ThenBy(o => o.Source, StringComparer.OrdinalIgnoreCase)
|
||||
.Take(limit)
|
||||
.ToList();
|
||||
|
||||
return ValueTask.FromResult<IReadOnlyList<GraphOverlayItem>>(results);
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,76 @@
|
||||
using Microsoft.Extensions.Options;
|
||||
using StellaOps.Excititor.WebService.Contracts;
|
||||
using StellaOps.Excititor.WebService.Options;
|
||||
using StellaOps.Messaging;
|
||||
using StellaOps.Messaging.Abstractions;
|
||||
|
||||
namespace StellaOps.Excititor.WebService.Services;
|
||||
|
||||
/// <summary>
|
||||
/// Graph overlay cache backed by <see cref="IDistributedCache{TValue}"/>.
|
||||
/// Supports any transport (InMemory, Valkey, PostgreSQL) via factory injection.
|
||||
/// </summary>
|
||||
internal sealed class MessagingGraphOverlayCache : IGraphOverlayCache
|
||||
{
|
||||
private readonly IDistributedCache<GraphOverlayCacheEntry> _cache;
|
||||
private readonly IOptions<GraphOptions> _options;
|
||||
private readonly TimeProvider _timeProvider;
|
||||
|
||||
public MessagingGraphOverlayCache(
|
||||
IDistributedCacheFactory cacheFactory,
|
||||
IOptions<GraphOptions> options,
|
||||
TimeProvider timeProvider)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(cacheFactory);
|
||||
_options = options ?? throw new ArgumentNullException(nameof(options));
|
||||
_timeProvider = timeProvider ?? throw new ArgumentNullException(nameof(timeProvider));
|
||||
|
||||
_cache = cacheFactory.Create<GraphOverlayCacheEntry>(new CacheOptions
|
||||
{
|
||||
KeyPrefix = "graph-overlays:",
|
||||
});
|
||||
}
|
||||
|
||||
public async ValueTask<GraphOverlayCacheHit?> TryGetAsync(
|
||||
string tenant,
|
||||
bool includeJustifications,
|
||||
IReadOnlyList<string> orderedPurls,
|
||||
CancellationToken cancellationToken)
|
||||
{
|
||||
var key = BuildKey(tenant, includeJustifications, orderedPurls);
|
||||
var result = await _cache.GetAsync(key, cancellationToken).ConfigureAwait(false);
|
||||
|
||||
if (result.HasValue)
|
||||
{
|
||||
var cached = result.Value;
|
||||
var ageMs = (long)Math.Max(0, (_timeProvider.GetUtcNow() - cached.CachedAt).TotalMilliseconds);
|
||||
return new GraphOverlayCacheHit(cached.Items, ageMs);
|
||||
}
|
||||
|
||||
return null;
|
||||
}
|
||||
|
||||
public async ValueTask SaveAsync(
|
||||
string tenant,
|
||||
bool includeJustifications,
|
||||
IReadOnlyList<string> orderedPurls,
|
||||
IReadOnlyList<GraphOverlayItem> items,
|
||||
DateTimeOffset cachedAt,
|
||||
CancellationToken cancellationToken)
|
||||
{
|
||||
var key = BuildKey(tenant, includeJustifications, orderedPurls);
|
||||
var ttl = TimeSpan.FromSeconds(Math.Max(1, _options.Value.OverlayTtlSeconds));
|
||||
var entry = new GraphOverlayCacheEntry(items.ToList(), cachedAt);
|
||||
var entryOptions = new CacheEntryOptions { TimeToLive = ttl };
|
||||
|
||||
await _cache.SetAsync(key, entry, entryOptions, cancellationToken).ConfigureAwait(false);
|
||||
}
|
||||
|
||||
private static string BuildKey(string tenant, bool includeJustifications, IReadOnlyList<string> orderedPurls)
|
||||
=> $"{tenant}:{includeJustifications}:{string.Join('|', orderedPurls)}";
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Cache entry for graph overlays.
|
||||
/// </summary>
|
||||
internal sealed record GraphOverlayCacheEntry(List<GraphOverlayItem> Items, DateTimeOffset CachedAt);
|
||||
@@ -0,0 +1,57 @@
|
||||
using Microsoft.Extensions.Caching.Memory;
|
||||
|
||||
namespace StellaOps.Excititor.WebService.Services;
|
||||
|
||||
internal sealed class MirrorRateLimiter
|
||||
{
|
||||
private readonly IMemoryCache _cache;
|
||||
private readonly TimeProvider _timeProvider;
|
||||
private static readonly TimeSpan Window = TimeSpan.FromHours(1);
|
||||
|
||||
public MirrorRateLimiter(IMemoryCache cache, TimeProvider timeProvider)
|
||||
{
|
||||
_cache = cache ?? throw new ArgumentNullException(nameof(cache));
|
||||
_timeProvider = timeProvider ?? throw new ArgumentNullException(nameof(timeProvider));
|
||||
}
|
||||
|
||||
public bool TryAcquire(string domainId, string scope, int limit, out TimeSpan? retryAfter)
|
||||
{
|
||||
retryAfter = null;
|
||||
|
||||
if (limit <= 0 || limit == int.MaxValue)
|
||||
{
|
||||
return true;
|
||||
}
|
||||
|
||||
var key = CreateKey(domainId, scope);
|
||||
var now = _timeProvider.GetUtcNow();
|
||||
|
||||
var counter = _cache.Get<Counter>(key);
|
||||
if (counter is null || now - counter.WindowStart >= Window)
|
||||
{
|
||||
counter = new Counter(now, 0);
|
||||
}
|
||||
|
||||
if (counter.Count >= limit)
|
||||
{
|
||||
var windowEnd = counter.WindowStart + Window;
|
||||
retryAfter = windowEnd > now ? windowEnd - now : TimeSpan.Zero;
|
||||
return false;
|
||||
}
|
||||
|
||||
counter = counter with { Count = counter.Count + 1 };
|
||||
var absoluteExpiration = counter.WindowStart + Window;
|
||||
_cache.Set(key, counter, absoluteExpiration);
|
||||
return true;
|
||||
}
|
||||
|
||||
private static string CreateKey(string domainId, string scope)
|
||||
=> string.Create(domainId.Length + scope.Length + 1, (domainId, scope), static (span, state) =>
|
||||
{
|
||||
state.domainId.AsSpan().CopyTo(span);
|
||||
span[state.domainId.Length] = '|';
|
||||
state.scope.AsSpan().CopyTo(span[(state.domainId.Length + 1)..]);
|
||||
});
|
||||
|
||||
private sealed record Counter(DateTimeOffset WindowStart, int Count);
|
||||
}
|
||||
@@ -0,0 +1,171 @@
|
||||
|
||||
using StellaOps.Excititor.Core;
|
||||
using StellaOps.Excititor.Core.Observations;
|
||||
using StellaOps.Excititor.Core.RiskFeed;
|
||||
using StellaOps.Excititor.WebService.Contracts;
|
||||
using System.Collections.Immutable;
|
||||
using System.Linq;
|
||||
|
||||
namespace StellaOps.Excititor.WebService.Services;
|
||||
|
||||
/// <summary>
|
||||
/// Risk feed service backed by graph overlays (EXCITITOR-RISK-66-001).
|
||||
/// </summary>
|
||||
public sealed class OverlayRiskFeedService : IRiskFeedService
|
||||
{
|
||||
private readonly IGraphOverlayStore _overlayStore;
|
||||
private readonly TimeProvider _timeProvider;
|
||||
|
||||
public OverlayRiskFeedService(IGraphOverlayStore overlayStore, TimeProvider timeProvider)
|
||||
{
|
||||
_overlayStore = overlayStore ?? throw new ArgumentNullException(nameof(overlayStore));
|
||||
_timeProvider = timeProvider ?? throw new ArgumentNullException(nameof(timeProvider));
|
||||
}
|
||||
|
||||
public async Task<RiskFeedResponse> GenerateFeedAsync(RiskFeedRequest request, CancellationToken cancellationToken)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(request);
|
||||
|
||||
var overlays = await ResolveOverlaysAsync(request, cancellationToken).ConfigureAwait(false);
|
||||
var filtered = ApplySinceFilter(overlays, request.Since);
|
||||
|
||||
var items = filtered
|
||||
.Select(MapToRiskFeedItem)
|
||||
.Where(item => item is not null)
|
||||
.Cast<RiskFeedItem>()
|
||||
.OrderBy(item => item.AdvisoryKey, StringComparer.OrdinalIgnoreCase)
|
||||
.ThenBy(item => item.Artifact, StringComparer.OrdinalIgnoreCase)
|
||||
.ThenBy(item => item.Provenance.TenantId, StringComparer.OrdinalIgnoreCase)
|
||||
.Take(request.Limit)
|
||||
.ToImmutableArray();
|
||||
|
||||
return new RiskFeedResponse(items, _timeProvider.GetUtcNow());
|
||||
}
|
||||
|
||||
public async Task<RiskFeedItem?> GetItemAsync(string tenantId, string advisoryKey, string artifact, CancellationToken cancellationToken)
|
||||
{
|
||||
ArgumentException.ThrowIfNullOrWhiteSpace(tenantId);
|
||||
ArgumentException.ThrowIfNullOrWhiteSpace(advisoryKey);
|
||||
ArgumentException.ThrowIfNullOrWhiteSpace(artifact);
|
||||
|
||||
var overlays = await _overlayStore
|
||||
.FindByPurlsAsync(tenantId, new[] { artifact }, cancellationToken)
|
||||
.ConfigureAwait(false);
|
||||
|
||||
var match = overlays
|
||||
.Where(o => string.Equals(o.AdvisoryId, advisoryKey, StringComparison.OrdinalIgnoreCase))
|
||||
.OrderBy(o => o.Source, StringComparer.OrdinalIgnoreCase)
|
||||
.FirstOrDefault();
|
||||
|
||||
return match is null ? null : MapToRiskFeedItem(match);
|
||||
}
|
||||
|
||||
private async Task<IReadOnlyList<GraphOverlayItem>> ResolveOverlaysAsync(RiskFeedRequest request, CancellationToken cancellationToken)
|
||||
{
|
||||
if (!request.AdvisoryKeys.IsDefaultOrEmpty)
|
||||
{
|
||||
return await _overlayStore
|
||||
.FindByAdvisoriesAsync(request.TenantId, request.AdvisoryKeys, request.Limit, cancellationToken)
|
||||
.ConfigureAwait(false);
|
||||
}
|
||||
|
||||
if (!request.Artifacts.IsDefaultOrEmpty)
|
||||
{
|
||||
return await _overlayStore
|
||||
.FindByPurlsAsync(request.TenantId, request.Artifacts, cancellationToken)
|
||||
.ConfigureAwait(false);
|
||||
}
|
||||
|
||||
return await _overlayStore
|
||||
.FindWithConflictsAsync(request.TenantId, request.Limit, cancellationToken)
|
||||
.ConfigureAwait(false);
|
||||
}
|
||||
|
||||
private static IEnumerable<GraphOverlayItem> ApplySinceFilter(IEnumerable<GraphOverlayItem> overlays, DateTimeOffset? since)
|
||||
{
|
||||
if (since is null)
|
||||
{
|
||||
return overlays;
|
||||
}
|
||||
|
||||
var threshold = since.Value;
|
||||
return overlays.Where(o => o.GeneratedAt >= threshold);
|
||||
}
|
||||
|
||||
private static RiskFeedItem? MapToRiskFeedItem(GraphOverlayItem overlay)
|
||||
{
|
||||
if (!TryParseStatus(overlay.Status, out var status))
|
||||
{
|
||||
return null;
|
||||
}
|
||||
|
||||
var justification = ParseJustification(overlay.Justifications.FirstOrDefault()?.Kind);
|
||||
var confidence = DeriveConfidence(overlay);
|
||||
var provenance = new RiskFeedProvenance(
|
||||
overlay.Tenant,
|
||||
overlay.Provenance.LinksetId,
|
||||
overlay.Provenance.LinksetHash,
|
||||
confidence,
|
||||
overlay.Conflicts.Count > 0,
|
||||
overlay.GeneratedAt);
|
||||
|
||||
var observedAt = overlay.Observations.Count == 0
|
||||
? overlay.GeneratedAt
|
||||
: overlay.Observations.Max(o => o.FetchedAt);
|
||||
|
||||
var sources = overlay.Observations
|
||||
.OrderBy(o => o.FetchedAt)
|
||||
.Select(o => new RiskFeedObservationSource(
|
||||
o.Id,
|
||||
overlay.Source,
|
||||
overlay.Status,
|
||||
overlay.Justifications.FirstOrDefault()?.Kind,
|
||||
null))
|
||||
.ToImmutableArray();
|
||||
|
||||
return new RiskFeedItem(
|
||||
overlay.AdvisoryId,
|
||||
overlay.Purl,
|
||||
status,
|
||||
justification,
|
||||
provenance,
|
||||
observedAt,
|
||||
sources);
|
||||
}
|
||||
|
||||
private static bool TryParseStatus(string status, out VexClaimStatus parsed)
|
||||
{
|
||||
parsed = status.ToLowerInvariant() switch
|
||||
{
|
||||
"not_affected" => VexClaimStatus.NotAffected,
|
||||
"under_investigation" => VexClaimStatus.UnderInvestigation,
|
||||
"fixed" => VexClaimStatus.Fixed,
|
||||
"affected" => VexClaimStatus.Affected,
|
||||
_ => VexClaimStatus.UnderInvestigation
|
||||
};
|
||||
|
||||
return true;
|
||||
}
|
||||
|
||||
private static VexJustification? ParseJustification(string? value)
|
||||
{
|
||||
if (string.IsNullOrWhiteSpace(value))
|
||||
{
|
||||
return null;
|
||||
}
|
||||
|
||||
return Enum.TryParse<VexJustification>(value, true, out var justification) ? justification : null;
|
||||
}
|
||||
|
||||
private static VexLinksetConfidence DeriveConfidence(GraphOverlayItem overlay)
|
||||
{
|
||||
if (overlay.Conflicts.Count > 0)
|
||||
{
|
||||
return VexLinksetConfidence.Low;
|
||||
}
|
||||
|
||||
return overlay.Observations.Count > 1
|
||||
? VexLinksetConfidence.High
|
||||
: VexLinksetConfidence.Medium;
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,245 @@
|
||||
|
||||
using Microsoft.Extensions.Logging;
|
||||
using Npgsql;
|
||||
using NpgsqlTypes;
|
||||
using StellaOps.Excititor.Persistence.Postgres;
|
||||
using StellaOps.Excititor.WebService.Contracts;
|
||||
using System.Linq;
|
||||
using System.Text.Json;
|
||||
using System.Text.Json.Serialization;
|
||||
|
||||
namespace StellaOps.Excititor.WebService.Services;
|
||||
|
||||
/// <summary>
|
||||
/// Postgres-backed overlay materialization store. Persists overlays per tenant/purl/advisory/source.
|
||||
/// </summary>
|
||||
public sealed class PostgresGraphOverlayStore : IGraphOverlayStore
|
||||
{
|
||||
private static readonly JsonSerializerOptions SerializerOptions = new()
|
||||
{
|
||||
DefaultIgnoreCondition = JsonIgnoreCondition.WhenWritingNull,
|
||||
PropertyNamingPolicy = JsonNamingPolicy.CamelCase
|
||||
};
|
||||
|
||||
private readonly ExcititorDataSource _dataSource;
|
||||
private readonly ILogger<PostgresGraphOverlayStore> _logger;
|
||||
private volatile bool _initialized;
|
||||
private readonly SemaphoreSlim _initLock = new(1, 1);
|
||||
|
||||
public PostgresGraphOverlayStore(ExcititorDataSource dataSource, ILogger<PostgresGraphOverlayStore> logger)
|
||||
{
|
||||
_dataSource = dataSource ?? throw new ArgumentNullException(nameof(dataSource));
|
||||
_logger = logger ?? throw new ArgumentNullException(nameof(logger));
|
||||
}
|
||||
|
||||
public async ValueTask SaveAsync(string tenant, IReadOnlyList<GraphOverlayItem> overlays, CancellationToken cancellationToken)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(tenant);
|
||||
ArgumentNullException.ThrowIfNull(overlays);
|
||||
await EnsureTableAsync(cancellationToken).ConfigureAwait(false);
|
||||
|
||||
await using var connection = await _dataSource.OpenConnectionAsync("public", "writer", cancellationToken).ConfigureAwait(false);
|
||||
const string sql = """
|
||||
INSERT INTO vex.graph_overlays (tenant, purl, advisory_id, source, generated_at, payload)
|
||||
VALUES (@tenant, @purl, @advisory_id, @source, @generated_at, @payload)
|
||||
ON CONFLICT (tenant, purl, advisory_id, source)
|
||||
DO UPDATE SET generated_at = EXCLUDED.generated_at, payload = EXCLUDED.payload;
|
||||
""";
|
||||
|
||||
foreach (var overlay in overlays)
|
||||
{
|
||||
await using var command = new NpgsqlCommand(sql, connection)
|
||||
{
|
||||
CommandTimeout = _dataSource.CommandTimeoutSeconds
|
||||
};
|
||||
|
||||
command.Parameters.AddWithValue("tenant", tenant);
|
||||
command.Parameters.AddWithValue("purl", overlay.Purl);
|
||||
command.Parameters.AddWithValue("advisory_id", overlay.AdvisoryId);
|
||||
command.Parameters.AddWithValue("source", overlay.Source);
|
||||
command.Parameters.AddWithValue("generated_at", overlay.GeneratedAt.UtcDateTime);
|
||||
command.Parameters.Add(new NpgsqlParameter("payload", NpgsqlDbType.Jsonb)
|
||||
{
|
||||
Value = JsonSerializer.Serialize(overlay, SerializerOptions)
|
||||
});
|
||||
|
||||
await command.ExecuteNonQueryAsync(cancellationToken).ConfigureAwait(false);
|
||||
}
|
||||
}
|
||||
|
||||
public async ValueTask<IReadOnlyList<GraphOverlayItem>> FindByPurlsAsync(string tenant, IReadOnlyCollection<string> purls, CancellationToken cancellationToken)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(tenant);
|
||||
ArgumentNullException.ThrowIfNull(purls);
|
||||
if (purls.Count == 0)
|
||||
{
|
||||
return Array.Empty<GraphOverlayItem>();
|
||||
}
|
||||
|
||||
await EnsureTableAsync(cancellationToken).ConfigureAwait(false);
|
||||
await using var connection = await _dataSource.OpenConnectionAsync("public", "reader", cancellationToken).ConfigureAwait(false);
|
||||
|
||||
const string sql = """
|
||||
SELECT payload
|
||||
FROM vex.graph_overlays
|
||||
WHERE tenant = @tenant AND purl = ANY(@purls)
|
||||
ORDER BY purl, advisory_id, source;
|
||||
""";
|
||||
|
||||
await using var command = new NpgsqlCommand(sql, connection)
|
||||
{
|
||||
CommandTimeout = _dataSource.CommandTimeoutSeconds
|
||||
};
|
||||
|
||||
command.Parameters.AddWithValue("tenant", tenant);
|
||||
command.Parameters.Add(new NpgsqlParameter<string[]>("purls", NpgsqlDbType.Array | NpgsqlDbType.Text)
|
||||
{
|
||||
TypedValue = purls.ToArray()
|
||||
});
|
||||
|
||||
var overlays = new List<GraphOverlayItem>();
|
||||
await using var reader = await command.ExecuteReaderAsync(cancellationToken).ConfigureAwait(false);
|
||||
while (await reader.ReadAsync(cancellationToken).ConfigureAwait(false))
|
||||
{
|
||||
var payload = reader.GetString(0);
|
||||
var overlay = JsonSerializer.Deserialize<GraphOverlayItem>(payload, SerializerOptions);
|
||||
if (overlay is not null)
|
||||
{
|
||||
overlays.Add(overlay);
|
||||
}
|
||||
}
|
||||
|
||||
return overlays;
|
||||
}
|
||||
|
||||
public async ValueTask<IReadOnlyList<GraphOverlayItem>> FindByAdvisoriesAsync(string tenant, IReadOnlyCollection<string> advisories, int limit, CancellationToken cancellationToken)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(tenant);
|
||||
ArgumentNullException.ThrowIfNull(advisories);
|
||||
if (advisories.Count == 0)
|
||||
{
|
||||
return Array.Empty<GraphOverlayItem>();
|
||||
}
|
||||
|
||||
await EnsureTableAsync(cancellationToken).ConfigureAwait(false);
|
||||
await using var connection = await _dataSource.OpenConnectionAsync("public", "reader", cancellationToken).ConfigureAwait(false);
|
||||
|
||||
const string sql = """
|
||||
SELECT payload
|
||||
FROM vex.graph_overlays
|
||||
WHERE tenant = @tenant AND advisory_id = ANY(@advisories)
|
||||
ORDER BY advisory_id, purl, source
|
||||
LIMIT @limit;
|
||||
""";
|
||||
|
||||
await using var command = new NpgsqlCommand(sql, connection)
|
||||
{
|
||||
CommandTimeout = _dataSource.CommandTimeoutSeconds
|
||||
};
|
||||
|
||||
command.Parameters.AddWithValue("tenant", tenant);
|
||||
command.Parameters.Add(new NpgsqlParameter<string[]>("advisories", NpgsqlDbType.Array | NpgsqlDbType.Text)
|
||||
{
|
||||
TypedValue = advisories.ToArray()
|
||||
});
|
||||
command.Parameters.AddWithValue("limit", limit);
|
||||
|
||||
var overlays = new List<GraphOverlayItem>();
|
||||
await using var reader = await command.ExecuteReaderAsync(cancellationToken).ConfigureAwait(false);
|
||||
while (await reader.ReadAsync(cancellationToken).ConfigureAwait(false))
|
||||
{
|
||||
var payload = reader.GetString(0);
|
||||
var overlay = JsonSerializer.Deserialize<GraphOverlayItem>(payload, SerializerOptions);
|
||||
if (overlay is not null)
|
||||
{
|
||||
overlays.Add(overlay);
|
||||
}
|
||||
}
|
||||
|
||||
return overlays;
|
||||
}
|
||||
|
||||
public async ValueTask<IReadOnlyList<GraphOverlayItem>> FindWithConflictsAsync(string tenant, int limit, CancellationToken cancellationToken)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(tenant);
|
||||
await EnsureTableAsync(cancellationToken).ConfigureAwait(false);
|
||||
|
||||
await using var connection = await _dataSource.OpenConnectionAsync("public", "reader", cancellationToken).ConfigureAwait(false);
|
||||
const string sql = """
|
||||
SELECT payload
|
||||
FROM vex.graph_overlays
|
||||
WHERE tenant = @tenant
|
||||
AND jsonb_array_length(payload -> 'conflicts') > 0
|
||||
ORDER BY generated_at DESC, purl, advisory_id, source
|
||||
LIMIT @limit;
|
||||
""";
|
||||
|
||||
await using var command = new NpgsqlCommand(sql, connection)
|
||||
{
|
||||
CommandTimeout = _dataSource.CommandTimeoutSeconds
|
||||
};
|
||||
command.Parameters.AddWithValue("tenant", tenant);
|
||||
command.Parameters.AddWithValue("limit", limit);
|
||||
|
||||
var overlays = new List<GraphOverlayItem>();
|
||||
await using var reader = await command.ExecuteReaderAsync(cancellationToken).ConfigureAwait(false);
|
||||
while (await reader.ReadAsync(cancellationToken).ConfigureAwait(false))
|
||||
{
|
||||
var payload = reader.GetString(0);
|
||||
var overlay = JsonSerializer.Deserialize<GraphOverlayItem>(payload, SerializerOptions);
|
||||
if (overlay is not null)
|
||||
{
|
||||
overlays.Add(overlay);
|
||||
}
|
||||
}
|
||||
|
||||
return overlays;
|
||||
}
|
||||
|
||||
private async ValueTask EnsureTableAsync(CancellationToken cancellationToken)
|
||||
{
|
||||
if (_initialized)
|
||||
{
|
||||
return;
|
||||
}
|
||||
|
||||
await _initLock.WaitAsync(cancellationToken).ConfigureAwait(false);
|
||||
try
|
||||
{
|
||||
if (_initialized)
|
||||
{
|
||||
return;
|
||||
}
|
||||
|
||||
await using var connection = await _dataSource.OpenSystemConnectionAsync(cancellationToken).ConfigureAwait(false);
|
||||
const string sql = """
|
||||
CREATE TABLE IF NOT EXISTS vex.graph_overlays (
|
||||
tenant text NOT NULL,
|
||||
purl text NOT NULL,
|
||||
advisory_id text NOT NULL,
|
||||
source text NOT NULL,
|
||||
generated_at timestamptz NOT NULL,
|
||||
payload jsonb NOT NULL,
|
||||
CONSTRAINT pk_graph_overlays PRIMARY KEY (tenant, purl, advisory_id, source)
|
||||
);
|
||||
""";
|
||||
|
||||
await using var command = new NpgsqlCommand(sql, connection)
|
||||
{
|
||||
CommandTimeout = _dataSource.CommandTimeoutSeconds
|
||||
};
|
||||
await command.ExecuteNonQueryAsync(cancellationToken).ConfigureAwait(false);
|
||||
|
||||
_initialized = true;
|
||||
}
|
||||
catch (Exception ex) when (!cancellationToken.IsCancellationRequested)
|
||||
{
|
||||
_logger.LogError(ex, "Failed to ensure graph_overlays table exists.");
|
||||
throw;
|
||||
}
|
||||
finally
|
||||
{
|
||||
_initLock.Release();
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,55 @@
|
||||
|
||||
using Microsoft.AspNetCore.Http;
|
||||
using System.Linq;
|
||||
using System.Security.Claims;
|
||||
|
||||
namespace StellaOps.Excititor.WebService.Services;
|
||||
|
||||
internal static class ScopeAuthorization
|
||||
{
|
||||
public static IResult? RequireScope(HttpContext context, string scope)
|
||||
{
|
||||
if (context is null)
|
||||
{
|
||||
throw new ArgumentNullException(nameof(context));
|
||||
}
|
||||
|
||||
if (string.IsNullOrWhiteSpace(scope))
|
||||
{
|
||||
throw new ArgumentException("Scope must be provided.", nameof(scope));
|
||||
}
|
||||
|
||||
var user = context.User;
|
||||
if (user?.Identity?.IsAuthenticated is not true)
|
||||
{
|
||||
return Results.Unauthorized();
|
||||
}
|
||||
|
||||
if (!HasScope(user, scope))
|
||||
{
|
||||
return Results.Forbid();
|
||||
}
|
||||
|
||||
return null;
|
||||
}
|
||||
|
||||
private static bool HasScope(ClaimsPrincipal user, string requiredScope)
|
||||
{
|
||||
var comparison = StringComparer.OrdinalIgnoreCase;
|
||||
foreach (var claim in user.FindAll("scope").Concat(user.FindAll("scp")))
|
||||
{
|
||||
if (string.IsNullOrWhiteSpace(claim.Value))
|
||||
{
|
||||
continue;
|
||||
}
|
||||
|
||||
var scopes = claim.Value.Split(' ', StringSplitOptions.RemoveEmptyEntries | StringSplitOptions.TrimEntries);
|
||||
if (scopes.Any(scope => comparison.Equals(scope, requiredScope)))
|
||||
{
|
||||
return true;
|
||||
}
|
||||
}
|
||||
|
||||
return false;
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,131 @@
|
||||
|
||||
using StellaOps.Excititor.Core;
|
||||
using StellaOps.Excititor.Core.Storage;
|
||||
using StellaOps.Excititor.WebService.Contracts;
|
||||
using System;
|
||||
using System.Collections.Generic;
|
||||
using System.Collections.Immutable;
|
||||
using System.Globalization;
|
||||
using System.Linq;
|
||||
using System.Threading;
|
||||
using System.Threading.Tasks;
|
||||
|
||||
namespace StellaOps.Excititor.WebService.Services;
|
||||
|
||||
internal interface IVexEvidenceChunkService
|
||||
{
|
||||
Task<VexEvidenceChunkResult> QueryAsync(VexEvidenceChunkRequest request, CancellationToken cancellationToken);
|
||||
}
|
||||
|
||||
internal sealed record VexEvidenceChunkRequest(
|
||||
string Tenant,
|
||||
string VulnerabilityId,
|
||||
string ProductKey,
|
||||
ImmutableHashSet<string> ProviderIds,
|
||||
ImmutableHashSet<VexClaimStatus> Statuses,
|
||||
DateTimeOffset? Since,
|
||||
int Limit);
|
||||
|
||||
internal sealed record VexEvidenceChunkResult(
|
||||
IReadOnlyList<VexEvidenceChunkResponse> Chunks,
|
||||
bool Truncated,
|
||||
int TotalCount,
|
||||
DateTimeOffset GeneratedAtUtc);
|
||||
|
||||
internal sealed class VexEvidenceChunkService : IVexEvidenceChunkService
|
||||
{
|
||||
private readonly IVexClaimStore _claimStore;
|
||||
private readonly TimeProvider _timeProvider;
|
||||
|
||||
public VexEvidenceChunkService(IVexClaimStore claimStore, TimeProvider timeProvider)
|
||||
{
|
||||
_claimStore = claimStore ?? throw new ArgumentNullException(nameof(claimStore));
|
||||
_timeProvider = timeProvider ?? throw new ArgumentNullException(nameof(timeProvider));
|
||||
}
|
||||
|
||||
public async Task<VexEvidenceChunkResult> QueryAsync(VexEvidenceChunkRequest request, CancellationToken cancellationToken)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(request);
|
||||
|
||||
var claims = await _claimStore
|
||||
.FindAsync(request.VulnerabilityId, request.ProductKey, request.Since, cancellationToken)
|
||||
.ConfigureAwait(false);
|
||||
|
||||
var filtered = claims
|
||||
.Where(claim => MatchesProvider(claim, request.ProviderIds))
|
||||
.Where(claim => MatchesStatus(claim, request.Statuses))
|
||||
.OrderByDescending(claim => claim.LastSeen)
|
||||
.ToList();
|
||||
|
||||
var total = filtered.Count;
|
||||
if (filtered.Count > request.Limit)
|
||||
{
|
||||
filtered = filtered.Take(request.Limit).ToList();
|
||||
}
|
||||
|
||||
var chunks = filtered
|
||||
.Select(MapChunk)
|
||||
.ToList();
|
||||
|
||||
return new VexEvidenceChunkResult(
|
||||
chunks,
|
||||
total > request.Limit,
|
||||
total,
|
||||
_timeProvider.GetUtcNow());
|
||||
}
|
||||
|
||||
private static bool MatchesProvider(VexClaim claim, ImmutableHashSet<string> providers)
|
||||
=> providers.Count == 0 || providers.Contains(claim.ProviderId, StringComparer.OrdinalIgnoreCase);
|
||||
|
||||
private static bool MatchesStatus(VexClaim claim, ImmutableHashSet<VexClaimStatus> statuses)
|
||||
=> statuses.Count == 0 || statuses.Contains(claim.Status);
|
||||
|
||||
private static VexEvidenceChunkResponse MapChunk(VexClaim claim)
|
||||
{
|
||||
var observationId = string.Create(CultureInfo.InvariantCulture, $"{claim.ProviderId}:{claim.Document.Digest}");
|
||||
var linksetId = string.Create(CultureInfo.InvariantCulture, $"{claim.VulnerabilityId}:{claim.Product.Key}");
|
||||
|
||||
var scope = new VexEvidenceChunkScope(
|
||||
claim.Product.Key,
|
||||
claim.Product.Name,
|
||||
claim.Product.Version,
|
||||
claim.Product.Purl,
|
||||
claim.Product.Cpe,
|
||||
claim.Product.ComponentIdentifiers);
|
||||
|
||||
var document = new VexEvidenceChunkDocument(
|
||||
claim.Document.Digest,
|
||||
claim.Document.Format.ToString().ToLowerInvariant(),
|
||||
claim.Document.SourceUri.ToString(),
|
||||
claim.Document.Revision);
|
||||
|
||||
var signature = claim.Document.Signature is null
|
||||
? null
|
||||
: new VexEvidenceChunkSignature(
|
||||
claim.Document.Signature.Type,
|
||||
claim.Document.Signature.Subject,
|
||||
claim.Document.Signature.Issuer,
|
||||
claim.Document.Signature.KeyId,
|
||||
claim.Document.Signature.VerifiedAt,
|
||||
claim.Document.Signature.TransparencyLogReference);
|
||||
|
||||
var scopeScore = claim.Confidence?.Score ?? claim.Signals?.Severity?.Score;
|
||||
|
||||
return new VexEvidenceChunkResponse(
|
||||
observationId,
|
||||
linksetId,
|
||||
claim.VulnerabilityId,
|
||||
claim.Product.Key,
|
||||
claim.ProviderId,
|
||||
claim.Status.ToString(),
|
||||
claim.Justification?.ToString(),
|
||||
claim.Detail,
|
||||
scopeScore,
|
||||
claim.FirstSeen,
|
||||
claim.LastSeen,
|
||||
scope,
|
||||
document,
|
||||
signature,
|
||||
claim.AdditionalMetadata);
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,113 @@
|
||||
|
||||
using StellaOps.Cryptography;
|
||||
using System;
|
||||
using System.Security.Cryptography;
|
||||
using System.Text;
|
||||
|
||||
namespace StellaOps.Excititor.WebService.Services;
|
||||
|
||||
/// <summary>
|
||||
/// Service interface for hashing operations in Excititor (CRYPTO-90-001).
|
||||
/// Abstracts hashing implementation to support GOST/SM algorithms via ICryptoProviderRegistry.
|
||||
/// </summary>
|
||||
public interface IVexHashingService
|
||||
{
|
||||
/// <summary>
|
||||
/// Compute hash of a UTF-8 encoded string.
|
||||
/// </summary>
|
||||
string ComputeHash(string value, string algorithm = "sha256");
|
||||
|
||||
/// <summary>
|
||||
/// Compute hash of raw bytes.
|
||||
/// </summary>
|
||||
string ComputeHash(ReadOnlySpan<byte> data, string algorithm = "sha256");
|
||||
|
||||
/// <summary>
|
||||
/// Try to compute hash of raw bytes with stack-allocated buffer optimization.
|
||||
/// </summary>
|
||||
bool TryComputeHash(ReadOnlySpan<byte> data, Span<byte> destination, out int bytesWritten, string algorithm = "sha256");
|
||||
|
||||
/// <summary>
|
||||
/// Format a hash digest with algorithm prefix.
|
||||
/// </summary>
|
||||
string FormatDigest(string algorithm, ReadOnlySpan<byte> digest);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Default implementation of <see cref="IVexHashingService"/> that uses ICryptoProviderRegistry
|
||||
/// when available, falling back to System.Security.Cryptography for SHA-256.
|
||||
/// </summary>
|
||||
public sealed class VexHashingService : IVexHashingService
|
||||
{
|
||||
private readonly ICryptoProviderRegistry? _registry;
|
||||
|
||||
public VexHashingService(ICryptoProviderRegistry? registry = null)
|
||||
{
|
||||
_registry = registry;
|
||||
}
|
||||
|
||||
public string ComputeHash(string value, string algorithm = "sha256")
|
||||
{
|
||||
if (string.IsNullOrEmpty(value))
|
||||
{
|
||||
return string.Empty;
|
||||
}
|
||||
|
||||
var bytes = Encoding.UTF8.GetBytes(value);
|
||||
return ComputeHash(bytes, algorithm);
|
||||
}
|
||||
|
||||
public string ComputeHash(ReadOnlySpan<byte> data, string algorithm = "sha256")
|
||||
{
|
||||
Span<byte> buffer = stackalloc byte[64]; // Large enough for SHA-512 and GOST
|
||||
if (!TryComputeHash(data, buffer, out var written, algorithm))
|
||||
{
|
||||
throw new InvalidOperationException($"Failed to compute {algorithm} hash.");
|
||||
}
|
||||
|
||||
return FormatDigest(algorithm, buffer[..written]);
|
||||
}
|
||||
|
||||
public bool TryComputeHash(ReadOnlySpan<byte> data, Span<byte> destination, out int bytesWritten, string algorithm = "sha256")
|
||||
{
|
||||
bytesWritten = 0;
|
||||
|
||||
// Try to use crypto provider registry first for pluggable algorithms
|
||||
if (_registry is not null)
|
||||
{
|
||||
try
|
||||
{
|
||||
var resolution = _registry.ResolveHasher(algorithm);
|
||||
var hasher = resolution.Hasher;
|
||||
var result = hasher.ComputeHash(data);
|
||||
if (result.Length <= destination.Length)
|
||||
{
|
||||
result.CopyTo(destination);
|
||||
bytesWritten = result.Length;
|
||||
return true;
|
||||
}
|
||||
}
|
||||
catch
|
||||
{
|
||||
// Fall through to built-in implementation
|
||||
}
|
||||
}
|
||||
|
||||
// Fall back to System.Security.Cryptography for standard algorithms
|
||||
var normalizedAlgorithm = algorithm.ToLowerInvariant().Replace("-", string.Empty);
|
||||
return normalizedAlgorithm switch
|
||||
{
|
||||
"sha256" => SHA256.TryHashData(data, destination, out bytesWritten),
|
||||
"sha384" => SHA384.TryHashData(data, destination, out bytesWritten),
|
||||
"sha512" => SHA512.TryHashData(data, destination, out bytesWritten),
|
||||
_ => throw new NotSupportedException($"Unsupported hash algorithm: {algorithm}")
|
||||
};
|
||||
}
|
||||
|
||||
public string FormatDigest(string algorithm, ReadOnlySpan<byte> digest)
|
||||
{
|
||||
var normalizedAlgorithm = algorithm.ToLowerInvariant().Replace("-", string.Empty);
|
||||
var hexDigest = Convert.ToHexString(digest).ToLowerInvariant();
|
||||
return $"{normalizedAlgorithm}:{hexDigest}";
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,590 @@
|
||||
|
||||
using Microsoft.Extensions.Logging;
|
||||
using Microsoft.Extensions.Options;
|
||||
using StellaOps.Excititor.Connectors.Abstractions;
|
||||
using StellaOps.Excititor.Core;
|
||||
using StellaOps.Excititor.Core.Storage;
|
||||
using System.Collections.Generic;
|
||||
using System.Collections.Immutable;
|
||||
using System.Diagnostics;
|
||||
using System.Globalization;
|
||||
using System.Linq;
|
||||
|
||||
namespace StellaOps.Excititor.WebService.Services;
|
||||
|
||||
internal interface IVexIngestOrchestrator
|
||||
{
|
||||
Task<InitSummary> InitializeAsync(IngestInitOptions options, CancellationToken cancellationToken);
|
||||
|
||||
Task<IngestRunSummary> RunAsync(IngestRunOptions options, CancellationToken cancellationToken);
|
||||
|
||||
Task<IngestRunSummary> ResumeAsync(IngestResumeOptions options, CancellationToken cancellationToken);
|
||||
|
||||
Task<ReconcileSummary> ReconcileAsync(ReconcileOptions options, CancellationToken cancellationToken);
|
||||
}
|
||||
|
||||
internal sealed class VexIngestOrchestrator : IVexIngestOrchestrator
|
||||
{
|
||||
private readonly IServiceProvider _serviceProvider;
|
||||
private readonly IReadOnlyDictionary<string, IVexConnector> _connectors;
|
||||
private readonly IVexRawStore _rawStore;
|
||||
private readonly IVexClaimStore _claimStore;
|
||||
private readonly IVexProviderStore _providerStore;
|
||||
private readonly IVexConnectorStateRepository _stateRepository;
|
||||
private readonly IVexNormalizerRouter _normalizerRouter;
|
||||
private readonly IVexSignatureVerifier _signatureVerifier;
|
||||
private readonly TimeProvider _timeProvider;
|
||||
private readonly ILogger<VexIngestOrchestrator> _logger;
|
||||
private readonly string _defaultTenant;
|
||||
|
||||
public VexIngestOrchestrator(
|
||||
IServiceProvider serviceProvider,
|
||||
IEnumerable<IVexConnector> connectors,
|
||||
IVexRawStore rawStore,
|
||||
IVexClaimStore claimStore,
|
||||
IVexProviderStore providerStore,
|
||||
IVexConnectorStateRepository stateRepository,
|
||||
IVexNormalizerRouter normalizerRouter,
|
||||
IVexSignatureVerifier signatureVerifier,
|
||||
TimeProvider timeProvider,
|
||||
IOptions<VexStorageOptions> storageOptions,
|
||||
ILogger<VexIngestOrchestrator> logger)
|
||||
{
|
||||
_serviceProvider = serviceProvider ?? throw new ArgumentNullException(nameof(serviceProvider));
|
||||
_rawStore = rawStore ?? throw new ArgumentNullException(nameof(rawStore));
|
||||
_claimStore = claimStore ?? throw new ArgumentNullException(nameof(claimStore));
|
||||
_providerStore = providerStore ?? throw new ArgumentNullException(nameof(providerStore));
|
||||
_stateRepository = stateRepository ?? throw new ArgumentNullException(nameof(stateRepository));
|
||||
_normalizerRouter = normalizerRouter ?? throw new ArgumentNullException(nameof(normalizerRouter));
|
||||
_signatureVerifier = signatureVerifier ?? throw new ArgumentNullException(nameof(signatureVerifier));
|
||||
_timeProvider = timeProvider ?? throw new ArgumentNullException(nameof(timeProvider));
|
||||
_logger = logger ?? throw new ArgumentNullException(nameof(logger));
|
||||
var optionsValue = (storageOptions ?? throw new ArgumentNullException(nameof(storageOptions))).Value
|
||||
?? throw new ArgumentNullException(nameof(storageOptions));
|
||||
_defaultTenant = string.IsNullOrWhiteSpace(optionsValue.DefaultTenant)
|
||||
? "default"
|
||||
: optionsValue.DefaultTenant.Trim();
|
||||
|
||||
if (connectors is null)
|
||||
{
|
||||
throw new ArgumentNullException(nameof(connectors));
|
||||
}
|
||||
|
||||
_connectors = connectors
|
||||
.GroupBy(connector => connector.Id, StringComparer.OrdinalIgnoreCase)
|
||||
.ToDictionary(group => group.Key, group => group.First(), StringComparer.OrdinalIgnoreCase);
|
||||
}
|
||||
|
||||
public async Task<InitSummary> InitializeAsync(IngestInitOptions options, CancellationToken cancellationToken)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(options);
|
||||
|
||||
var runId = Guid.NewGuid();
|
||||
var startedAt = _timeProvider.GetUtcNow();
|
||||
var results = ImmutableArray.CreateBuilder<InitProviderResult>();
|
||||
|
||||
var (handles, missing) = ResolveConnectors(options.Providers);
|
||||
foreach (var providerId in missing)
|
||||
{
|
||||
results.Add(new InitProviderResult(providerId, providerId, "missing", TimeSpan.Zero, "Provider connector is not registered."));
|
||||
}
|
||||
|
||||
foreach (var handle in handles)
|
||||
{
|
||||
var stopwatch = Stopwatch.StartNew();
|
||||
try
|
||||
{
|
||||
await ValidateConnectorAsync(handle, cancellationToken).ConfigureAwait(false);
|
||||
await EnsureProviderRegistrationAsync(handle.Descriptor, cancellationToken).ConfigureAwait(false);
|
||||
stopwatch.Stop();
|
||||
|
||||
results.Add(new InitProviderResult(
|
||||
handle.Descriptor.Id,
|
||||
handle.Descriptor.DisplayName,
|
||||
"succeeded",
|
||||
stopwatch.Elapsed,
|
||||
Error: null));
|
||||
|
||||
_logger.LogInformation("Excititor init validated provider {ProviderId} in {Duration}ms.", handle.Descriptor.Id, stopwatch.Elapsed.TotalMilliseconds);
|
||||
}
|
||||
catch (OperationCanceledException) when (cancellationToken.IsCancellationRequested)
|
||||
{
|
||||
stopwatch.Stop();
|
||||
results.Add(new InitProviderResult(
|
||||
handle.Descriptor.Id,
|
||||
handle.Descriptor.DisplayName,
|
||||
"cancelled",
|
||||
stopwatch.Elapsed,
|
||||
"Operation cancelled."));
|
||||
_logger.LogWarning("Excititor init cancelled for provider {ProviderId}.", handle.Descriptor.Id);
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
stopwatch.Stop();
|
||||
results.Add(new InitProviderResult(
|
||||
handle.Descriptor.Id,
|
||||
handle.Descriptor.DisplayName,
|
||||
"failed",
|
||||
stopwatch.Elapsed,
|
||||
ex.Message));
|
||||
_logger.LogError(ex, "Excititor init failed for provider {ProviderId}: {Message}", handle.Descriptor.Id, ex.Message);
|
||||
}
|
||||
}
|
||||
|
||||
var completedAt = _timeProvider.GetUtcNow();
|
||||
return new InitSummary(runId, startedAt, completedAt, results.ToImmutable());
|
||||
}
|
||||
|
||||
public async Task<IngestRunSummary> RunAsync(IngestRunOptions options, CancellationToken cancellationToken)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(options);
|
||||
|
||||
var runId = Guid.NewGuid();
|
||||
var startedAt = _timeProvider.GetUtcNow();
|
||||
var since = ResolveSince(options.Since, options.Window, startedAt);
|
||||
var results = ImmutableArray.CreateBuilder<ProviderRunResult>();
|
||||
var (handles, missing) = ResolveConnectors(options.Providers);
|
||||
foreach (var providerId in missing)
|
||||
{
|
||||
results.Add(ProviderRunResult.Missing(providerId, since));
|
||||
}
|
||||
|
||||
foreach (var handle in handles)
|
||||
{
|
||||
var result = await ExecuteRunAsync(runId, handle, since, options.Force, cancellationToken).ConfigureAwait(false);
|
||||
results.Add(result);
|
||||
}
|
||||
|
||||
var completedAt = _timeProvider.GetUtcNow();
|
||||
return new IngestRunSummary(runId, startedAt, completedAt, results.ToImmutable());
|
||||
}
|
||||
|
||||
public async Task<IngestRunSummary> ResumeAsync(IngestResumeOptions options, CancellationToken cancellationToken)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(options);
|
||||
|
||||
var runId = Guid.NewGuid();
|
||||
var startedAt = _timeProvider.GetUtcNow();
|
||||
var results = ImmutableArray.CreateBuilder<ProviderRunResult>();
|
||||
var (handles, missing) = ResolveConnectors(options.Providers);
|
||||
foreach (var providerId in missing)
|
||||
{
|
||||
results.Add(ProviderRunResult.Missing(providerId, since: null));
|
||||
}
|
||||
|
||||
foreach (var handle in handles)
|
||||
{
|
||||
var since = await ResolveResumeSinceAsync(handle.Descriptor.Id, options.Checkpoint, cancellationToken).ConfigureAwait(false);
|
||||
var result = await ExecuteRunAsync(runId, handle, since, force: false, cancellationToken).ConfigureAwait(false);
|
||||
results.Add(result);
|
||||
}
|
||||
|
||||
var completedAt = _timeProvider.GetUtcNow();
|
||||
return new IngestRunSummary(runId, startedAt, completedAt, results.ToImmutable());
|
||||
}
|
||||
|
||||
public async Task<ReconcileSummary> ReconcileAsync(ReconcileOptions options, CancellationToken cancellationToken)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(options);
|
||||
|
||||
var runId = Guid.NewGuid();
|
||||
var startedAt = _timeProvider.GetUtcNow();
|
||||
var threshold = options.MaxAge is null ? (DateTimeOffset?)null : startedAt - options.MaxAge.Value;
|
||||
var results = ImmutableArray.CreateBuilder<ReconcileProviderResult>();
|
||||
var (handles, missing) = ResolveConnectors(options.Providers);
|
||||
foreach (var providerId in missing)
|
||||
{
|
||||
results.Add(new ReconcileProviderResult(providerId, "missing", "missing", null, threshold, 0, 0, "Provider connector is not registered."));
|
||||
}
|
||||
|
||||
foreach (var handle in handles)
|
||||
{
|
||||
try
|
||||
{
|
||||
var state = await _stateRepository.GetAsync(handle.Descriptor.Id, cancellationToken).ConfigureAwait(false);
|
||||
var lastUpdated = state?.LastUpdated;
|
||||
var stale = threshold.HasValue && (lastUpdated is null || lastUpdated < threshold.Value);
|
||||
|
||||
if (stale || state is null)
|
||||
{
|
||||
var since = stale ? threshold : lastUpdated;
|
||||
var result = await ExecuteRunAsync(runId, handle, since, force: false, cancellationToken).ConfigureAwait(false);
|
||||
results.Add(new ReconcileProviderResult(
|
||||
handle.Descriptor.Id,
|
||||
result.Status,
|
||||
"reconciled",
|
||||
result.LastUpdated ?? result.CompletedAt,
|
||||
threshold,
|
||||
result.Documents,
|
||||
result.Claims,
|
||||
result.Error));
|
||||
}
|
||||
else
|
||||
{
|
||||
results.Add(new ReconcileProviderResult(
|
||||
handle.Descriptor.Id,
|
||||
"succeeded",
|
||||
"skipped",
|
||||
lastUpdated,
|
||||
threshold,
|
||||
Documents: 0,
|
||||
Claims: 0,
|
||||
Error: null));
|
||||
}
|
||||
}
|
||||
catch (OperationCanceledException) when (cancellationToken.IsCancellationRequested)
|
||||
{
|
||||
results.Add(new ReconcileProviderResult(
|
||||
handle.Descriptor.Id,
|
||||
"cancelled",
|
||||
"cancelled",
|
||||
null,
|
||||
threshold,
|
||||
0,
|
||||
0,
|
||||
"Operation cancelled."));
|
||||
_logger.LogWarning("Excititor reconcile cancelled for provider {ProviderId}.", handle.Descriptor.Id);
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
results.Add(new ReconcileProviderResult(
|
||||
handle.Descriptor.Id,
|
||||
"failed",
|
||||
"failed",
|
||||
null,
|
||||
threshold,
|
||||
0,
|
||||
0,
|
||||
ex.Message));
|
||||
_logger.LogError(ex, "Excititor reconcile failed for provider {ProviderId}: {Message}", handle.Descriptor.Id, ex.Message);
|
||||
}
|
||||
}
|
||||
|
||||
var completedAt = _timeProvider.GetUtcNow();
|
||||
return new ReconcileSummary(runId, startedAt, completedAt, results.ToImmutable());
|
||||
}
|
||||
|
||||
private async Task ValidateConnectorAsync(ConnectorHandle handle, CancellationToken cancellationToken)
|
||||
{
|
||||
await handle.Connector.ValidateAsync(VexConnectorSettings.Empty, cancellationToken).ConfigureAwait(false);
|
||||
}
|
||||
|
||||
private async Task EnsureProviderRegistrationAsync(VexConnectorDescriptor descriptor, CancellationToken cancellationToken)
|
||||
{
|
||||
var existing = await _providerStore.FindAsync(descriptor.Id, cancellationToken).ConfigureAwait(false);
|
||||
if (existing is not null)
|
||||
{
|
||||
return;
|
||||
}
|
||||
|
||||
var provider = new VexProvider(descriptor.Id, descriptor.DisplayName, descriptor.Kind);
|
||||
await _providerStore.SaveAsync(provider, cancellationToken).ConfigureAwait(false);
|
||||
}
|
||||
|
||||
private async Task<ProviderRunResult> ExecuteRunAsync(
|
||||
Guid runId,
|
||||
ConnectorHandle handle,
|
||||
DateTimeOffset? since,
|
||||
bool force,
|
||||
CancellationToken cancellationToken)
|
||||
{
|
||||
var providerId = handle.Descriptor.Id;
|
||||
var startedAt = _timeProvider.GetUtcNow();
|
||||
var stopwatch = Stopwatch.StartNew();
|
||||
using var scope = _logger.BeginScope(new Dictionary<string, object?>(StringComparer.Ordinal)
|
||||
{
|
||||
["tenant"] = _defaultTenant,
|
||||
["runId"] = runId,
|
||||
["providerId"] = providerId,
|
||||
["window.since"] = since?.ToString("O", CultureInfo.InvariantCulture),
|
||||
["force"] = force,
|
||||
});
|
||||
|
||||
try
|
||||
{
|
||||
await ValidateConnectorAsync(handle, cancellationToken).ConfigureAwait(false);
|
||||
await EnsureProviderRegistrationAsync(handle.Descriptor, cancellationToken).ConfigureAwait(false);
|
||||
|
||||
if (force)
|
||||
{
|
||||
var resetState = new VexConnectorState(providerId, null, ImmutableArray<string>.Empty);
|
||||
await _stateRepository.SaveAsync(resetState, cancellationToken).ConfigureAwait(false);
|
||||
}
|
||||
|
||||
var stateBeforeRun = await _stateRepository.GetAsync(providerId, cancellationToken).ConfigureAwait(false);
|
||||
var resumeTokens = stateBeforeRun?.ResumeTokens ?? ImmutableDictionary<string, string>.Empty;
|
||||
|
||||
var context = new VexConnectorContext(
|
||||
since,
|
||||
VexConnectorSettings.Empty,
|
||||
_rawStore,
|
||||
_signatureVerifier,
|
||||
_normalizerRouter,
|
||||
_serviceProvider,
|
||||
resumeTokens);
|
||||
|
||||
var documents = 0;
|
||||
var claims = 0;
|
||||
string? lastDigest = null;
|
||||
|
||||
await foreach (var document in handle.Connector.FetchAsync(context, cancellationToken).ConfigureAwait(false))
|
||||
{
|
||||
documents++;
|
||||
lastDigest = document.Digest;
|
||||
|
||||
var batch = await _normalizerRouter.NormalizeAsync(document, cancellationToken).ConfigureAwait(false);
|
||||
if (!batch.Claims.IsDefaultOrEmpty && batch.Claims.Length > 0)
|
||||
{
|
||||
claims += batch.Claims.Length;
|
||||
await _claimStore.AppendAsync(batch.Claims, _timeProvider.GetUtcNow(), cancellationToken).ConfigureAwait(false);
|
||||
}
|
||||
}
|
||||
|
||||
stopwatch.Stop();
|
||||
var completedAt = _timeProvider.GetUtcNow();
|
||||
var stateAfterRun = await _stateRepository.GetAsync(providerId, cancellationToken).ConfigureAwait(false);
|
||||
|
||||
var checkpoint = stateAfterRun?.DocumentDigests.IsDefaultOrEmpty == false
|
||||
? stateAfterRun.DocumentDigests[^1]
|
||||
: lastDigest;
|
||||
|
||||
var result = new ProviderRunResult(
|
||||
providerId,
|
||||
"succeeded",
|
||||
documents,
|
||||
claims,
|
||||
startedAt,
|
||||
completedAt,
|
||||
stopwatch.Elapsed,
|
||||
lastDigest,
|
||||
stateAfterRun?.LastUpdated,
|
||||
checkpoint,
|
||||
null,
|
||||
since);
|
||||
|
||||
_logger.LogInformation(
|
||||
"Excititor ingest provider {ProviderId} completed: documents={Documents} claims={Claims} since={Since} duration={Duration}ms",
|
||||
providerId,
|
||||
documents,
|
||||
claims,
|
||||
since?.ToString("O", CultureInfo.InvariantCulture),
|
||||
result.Duration.TotalMilliseconds);
|
||||
|
||||
return result;
|
||||
}
|
||||
catch (OperationCanceledException) when (cancellationToken.IsCancellationRequested)
|
||||
{
|
||||
stopwatch.Stop();
|
||||
var cancelledAt = _timeProvider.GetUtcNow();
|
||||
_logger.LogWarning("Excititor ingest provider {ProviderId} cancelled.", providerId);
|
||||
return new ProviderRunResult(
|
||||
providerId,
|
||||
"cancelled",
|
||||
0,
|
||||
0,
|
||||
startedAt,
|
||||
cancelledAt,
|
||||
stopwatch.Elapsed,
|
||||
null,
|
||||
null,
|
||||
null,
|
||||
"Operation cancelled.",
|
||||
since);
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
stopwatch.Stop();
|
||||
var failedAt = _timeProvider.GetUtcNow();
|
||||
_logger.LogError(ex, "Excititor ingest provider {ProviderId} failed: {Message}", providerId, ex.Message);
|
||||
return new ProviderRunResult(
|
||||
providerId,
|
||||
"failed",
|
||||
0,
|
||||
0,
|
||||
startedAt,
|
||||
failedAt,
|
||||
stopwatch.Elapsed,
|
||||
null,
|
||||
null,
|
||||
null,
|
||||
ex.Message,
|
||||
since);
|
||||
}
|
||||
}
|
||||
|
||||
private async Task<DateTimeOffset?> ResolveResumeSinceAsync(string providerId, string? checkpoint, CancellationToken cancellationToken)
|
||||
{
|
||||
if (!string.IsNullOrWhiteSpace(checkpoint))
|
||||
{
|
||||
if (DateTimeOffset.TryParse(
|
||||
checkpoint.Trim(),
|
||||
CultureInfo.InvariantCulture,
|
||||
DateTimeStyles.AssumeUniversal | DateTimeStyles.AdjustToUniversal,
|
||||
out var parsed))
|
||||
{
|
||||
return parsed;
|
||||
}
|
||||
|
||||
var digest = checkpoint.Trim();
|
||||
var document = await _rawStore.FindByDigestAsync(digest, cancellationToken).ConfigureAwait(false);
|
||||
if (document is not null)
|
||||
{
|
||||
return document.RetrievedAt;
|
||||
}
|
||||
}
|
||||
|
||||
var state = await _stateRepository.GetAsync(providerId, cancellationToken).ConfigureAwait(false);
|
||||
return state?.LastUpdated;
|
||||
}
|
||||
|
||||
private static DateTimeOffset? ResolveSince(DateTimeOffset? since, TimeSpan? window, DateTimeOffset reference)
|
||||
{
|
||||
if (since.HasValue)
|
||||
{
|
||||
return since.Value;
|
||||
}
|
||||
|
||||
if (window is { } duration && duration > TimeSpan.Zero)
|
||||
{
|
||||
var candidate = reference - duration;
|
||||
return candidate < DateTimeOffset.MinValue ? DateTimeOffset.MinValue : candidate;
|
||||
}
|
||||
|
||||
return null;
|
||||
}
|
||||
|
||||
private (IReadOnlyList<ConnectorHandle> Handles, ImmutableArray<string> Missing) ResolveConnectors(ImmutableArray<string> requestedProviders)
|
||||
{
|
||||
var handles = new List<ConnectorHandle>();
|
||||
var missing = ImmutableArray.CreateBuilder<string>();
|
||||
|
||||
if (requestedProviders.IsDefaultOrEmpty || requestedProviders.Length == 0)
|
||||
{
|
||||
foreach (var connector in _connectors.Values.OrderBy(static x => x.Id, StringComparer.OrdinalIgnoreCase))
|
||||
{
|
||||
handles.Add(new ConnectorHandle(connector, CreateDescriptor(connector)));
|
||||
}
|
||||
|
||||
return (handles, missing.ToImmutable());
|
||||
}
|
||||
|
||||
foreach (var providerId in requestedProviders)
|
||||
{
|
||||
if (_connectors.TryGetValue(providerId, out var connector))
|
||||
{
|
||||
handles.Add(new ConnectorHandle(connector, CreateDescriptor(connector)));
|
||||
}
|
||||
else
|
||||
{
|
||||
missing.Add(providerId);
|
||||
}
|
||||
}
|
||||
|
||||
return (handles, missing.ToImmutable());
|
||||
}
|
||||
|
||||
private static VexConnectorDescriptor CreateDescriptor(IVexConnector connector)
|
||||
=> connector switch
|
||||
{
|
||||
VexConnectorBase baseConnector => baseConnector.Descriptor,
|
||||
_ => new VexConnectorDescriptor(connector.Id, connector.Kind, connector.Id)
|
||||
};
|
||||
|
||||
private sealed record ConnectorHandle(IVexConnector Connector, VexConnectorDescriptor Descriptor);
|
||||
}
|
||||
|
||||
internal sealed record IngestInitOptions(
|
||||
ImmutableArray<string> Providers,
|
||||
bool Resume);
|
||||
|
||||
internal sealed record IngestRunOptions(
|
||||
ImmutableArray<string> Providers,
|
||||
DateTimeOffset? Since,
|
||||
TimeSpan? Window,
|
||||
bool Force);
|
||||
|
||||
internal sealed record IngestResumeOptions(
|
||||
ImmutableArray<string> Providers,
|
||||
string? Checkpoint);
|
||||
|
||||
internal sealed record ReconcileOptions(
|
||||
ImmutableArray<string> Providers,
|
||||
TimeSpan? MaxAge);
|
||||
|
||||
internal sealed record InitSummary(
|
||||
Guid RunId,
|
||||
DateTimeOffset StartedAt,
|
||||
DateTimeOffset CompletedAt,
|
||||
ImmutableArray<InitProviderResult> Providers)
|
||||
{
|
||||
public int ProviderCount => Providers.Length;
|
||||
public int SuccessCount => Providers.Count(result => string.Equals(result.Status, "succeeded", StringComparison.OrdinalIgnoreCase));
|
||||
public int FailureCount => Providers.Count(result => string.Equals(result.Status, "failed", StringComparison.OrdinalIgnoreCase));
|
||||
}
|
||||
|
||||
internal sealed record InitProviderResult(
|
||||
string ProviderId,
|
||||
string DisplayName,
|
||||
string Status,
|
||||
TimeSpan Duration,
|
||||
string? Error);
|
||||
|
||||
internal sealed record IngestRunSummary(
|
||||
Guid RunId,
|
||||
DateTimeOffset StartedAt,
|
||||
DateTimeOffset CompletedAt,
|
||||
ImmutableArray<ProviderRunResult> Providers)
|
||||
{
|
||||
public int ProviderCount => Providers.Length;
|
||||
|
||||
public int SuccessCount => Providers.Count(provider => string.Equals(provider.Status, "succeeded", StringComparison.OrdinalIgnoreCase));
|
||||
|
||||
public int FailureCount => Providers.Count(provider => string.Equals(provider.Status, "failed", StringComparison.OrdinalIgnoreCase));
|
||||
|
||||
public TimeSpan Duration => CompletedAt - StartedAt;
|
||||
}
|
||||
|
||||
internal sealed record ProviderRunResult(
|
||||
string ProviderId,
|
||||
string Status,
|
||||
int Documents,
|
||||
int Claims,
|
||||
DateTimeOffset StartedAt,
|
||||
DateTimeOffset CompletedAt,
|
||||
TimeSpan Duration,
|
||||
string? LastDigest,
|
||||
DateTimeOffset? LastUpdated,
|
||||
string? Checkpoint,
|
||||
string? Error,
|
||||
DateTimeOffset? Since)
|
||||
{
|
||||
public static ProviderRunResult Missing(string providerId, DateTimeOffset? since)
|
||||
=> new(providerId, "missing", 0, 0, DateTimeOffset.MinValue, DateTimeOffset.MinValue, TimeSpan.Zero, null, null, null, "Provider connector is not registered.", since);
|
||||
}
|
||||
|
||||
internal sealed record ReconcileSummary(
|
||||
Guid RunId,
|
||||
DateTimeOffset StartedAt,
|
||||
DateTimeOffset CompletedAt,
|
||||
ImmutableArray<ReconcileProviderResult> Providers)
|
||||
{
|
||||
public int ProviderCount => Providers.Length;
|
||||
|
||||
public int ReconciledCount => Providers.Count(result => string.Equals(result.Action, "reconciled", StringComparison.OrdinalIgnoreCase));
|
||||
|
||||
public int SkippedCount => Providers.Count(result => string.Equals(result.Action, "skipped", StringComparison.OrdinalIgnoreCase));
|
||||
|
||||
public int FailureCount => Providers.Count(result => string.Equals(result.Status, "failed", StringComparison.OrdinalIgnoreCase));
|
||||
|
||||
public TimeSpan Duration => CompletedAt - StartedAt;
|
||||
}
|
||||
|
||||
internal sealed record ReconcileProviderResult(
|
||||
string ProviderId,
|
||||
string Status,
|
||||
string Action,
|
||||
DateTimeOffset? LastUpdated,
|
||||
DateTimeOffset? Threshold,
|
||||
int Documents,
|
||||
int Claims,
|
||||
string? Error);
|
||||
@@ -0,0 +1,163 @@
|
||||
|
||||
using StellaOps.Excititor.Core;
|
||||
using StellaOps.Excititor.Core.Storage;
|
||||
using System;
|
||||
using System.Collections.Generic;
|
||||
using System.Collections.Immutable;
|
||||
using System.Globalization;
|
||||
using System.Linq;
|
||||
using System.Threading;
|
||||
using System.Threading.Tasks;
|
||||
|
||||
namespace StellaOps.Excititor.WebService.Services;
|
||||
|
||||
internal interface IVexObservationProjectionService
|
||||
{
|
||||
Task<VexObservationProjectionResult> QueryAsync(
|
||||
VexObservationProjectionRequest request,
|
||||
CancellationToken cancellationToken);
|
||||
}
|
||||
|
||||
internal sealed record VexObservationProjectionRequest(
|
||||
string Tenant,
|
||||
string VulnerabilityId,
|
||||
string ProductKey,
|
||||
ImmutableHashSet<string> ProviderIds,
|
||||
ImmutableHashSet<VexClaimStatus> Statuses,
|
||||
DateTimeOffset? Since,
|
||||
int Limit);
|
||||
|
||||
internal sealed record VexObservationProjectionResult(
|
||||
IReadOnlyList<VexObservationStatementProjection> Statements,
|
||||
bool Truncated,
|
||||
int TotalCount,
|
||||
DateTimeOffset GeneratedAtUtc);
|
||||
|
||||
internal sealed record VexObservationStatementProjection(
|
||||
string ObservationId,
|
||||
string ProviderId,
|
||||
VexClaimStatus Status,
|
||||
VexJustification? Justification,
|
||||
string? Detail,
|
||||
DateTimeOffset FirstSeen,
|
||||
DateTimeOffset LastSeen,
|
||||
VexProductScope Scope,
|
||||
IReadOnlyList<string> Anchors,
|
||||
VexClaimDocument Document,
|
||||
VexSignatureMetadata? Signature);
|
||||
|
||||
internal sealed record VexProductScope(
|
||||
string Key,
|
||||
string? Name,
|
||||
string? Version,
|
||||
string? Purl,
|
||||
string? Cpe,
|
||||
IReadOnlyList<string> ComponentIdentifiers);
|
||||
|
||||
internal sealed class VexObservationProjectionService : IVexObservationProjectionService
|
||||
{
|
||||
private static readonly string[] AnchorKeys =
|
||||
{
|
||||
"json_pointer",
|
||||
"jsonPointer",
|
||||
"statement_locator",
|
||||
"locator",
|
||||
"paragraph",
|
||||
"section",
|
||||
"path"
|
||||
};
|
||||
|
||||
private readonly IVexClaimStore _claimStore;
|
||||
private readonly TimeProvider _timeProvider;
|
||||
|
||||
public VexObservationProjectionService(IVexClaimStore claimStore, TimeProvider? timeProvider = null)
|
||||
{
|
||||
_claimStore = claimStore ?? throw new ArgumentNullException(nameof(claimStore));
|
||||
_timeProvider = timeProvider ?? TimeProvider.System;
|
||||
}
|
||||
|
||||
public async Task<VexObservationProjectionResult> QueryAsync(
|
||||
VexObservationProjectionRequest request,
|
||||
CancellationToken cancellationToken)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(request);
|
||||
cancellationToken.ThrowIfCancellationRequested();
|
||||
|
||||
var claims = await _claimStore.FindAsync(
|
||||
request.VulnerabilityId,
|
||||
request.ProductKey,
|
||||
request.Since,
|
||||
cancellationToken)
|
||||
.ConfigureAwait(false);
|
||||
|
||||
var filtered = claims
|
||||
.Where(claim => MatchesProvider(claim, request.ProviderIds))
|
||||
.Where(claim => MatchesStatus(claim, request.Statuses))
|
||||
.OrderByDescending(claim => claim.LastSeen)
|
||||
.ThenBy(claim => claim.ProviderId, StringComparer.Ordinal)
|
||||
.ToList();
|
||||
|
||||
var total = filtered.Count;
|
||||
var page = filtered.Take(request.Limit).ToList();
|
||||
var statements = page
|
||||
.Select(claim => MapClaim(claim))
|
||||
.ToList();
|
||||
|
||||
return new VexObservationProjectionResult(
|
||||
statements,
|
||||
total > request.Limit,
|
||||
total,
|
||||
_timeProvider.GetUtcNow());
|
||||
}
|
||||
|
||||
private static bool MatchesProvider(VexClaim claim, ImmutableHashSet<string> providers)
|
||||
=> providers.Count == 0 || providers.Contains(claim.ProviderId, StringComparer.OrdinalIgnoreCase);
|
||||
|
||||
private static bool MatchesStatus(VexClaim claim, ImmutableHashSet<VexClaimStatus> statuses)
|
||||
=> statuses.Count == 0 || statuses.Contains(claim.Status);
|
||||
|
||||
private static VexObservationStatementProjection MapClaim(VexClaim claim)
|
||||
{
|
||||
var observationId = string.Create(CultureInfo.InvariantCulture, $"{claim.ProviderId}:{claim.Document.Digest}");
|
||||
var anchors = ExtractAnchors(claim.AdditionalMetadata);
|
||||
var scope = new VexProductScope(
|
||||
claim.Product.Key,
|
||||
claim.Product.Name,
|
||||
claim.Product.Version,
|
||||
claim.Product.Purl,
|
||||
claim.Product.Cpe,
|
||||
claim.Product.ComponentIdentifiers);
|
||||
|
||||
return new VexObservationStatementProjection(
|
||||
observationId,
|
||||
claim.ProviderId,
|
||||
claim.Status,
|
||||
claim.Justification,
|
||||
claim.Detail,
|
||||
claim.FirstSeen,
|
||||
claim.LastSeen,
|
||||
scope,
|
||||
anchors,
|
||||
claim.Document,
|
||||
claim.Document.Signature);
|
||||
}
|
||||
|
||||
private static IReadOnlyList<string> ExtractAnchors(ImmutableSortedDictionary<string, string> metadata)
|
||||
{
|
||||
if (metadata.Count == 0)
|
||||
{
|
||||
return Array.Empty<string>();
|
||||
}
|
||||
|
||||
var anchors = new List<string>();
|
||||
foreach (var key in AnchorKeys)
|
||||
{
|
||||
if (metadata.TryGetValue(key, out var value) && !string.IsNullOrWhiteSpace(value))
|
||||
{
|
||||
anchors.Add(value.Trim());
|
||||
}
|
||||
}
|
||||
|
||||
return anchors.Count == 0 ? Array.Empty<string>() : anchors;
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,142 @@
|
||||
// VexSignatureVerifierV1Adapter - Adapts V2 interface to V1 for backward compatibility
|
||||
// Part of SPRINT_1227_0004_0001: Activate VEX Signature Verification Pipeline
|
||||
|
||||
|
||||
using Microsoft.Extensions.Logging;
|
||||
using Microsoft.Extensions.Options;
|
||||
using StellaOps.Excititor.Core;
|
||||
using StellaOps.Excititor.Core.Verification;
|
||||
using System;
|
||||
using System.Threading;
|
||||
using System.Threading.Tasks;
|
||||
|
||||
namespace StellaOps.Excititor.WebService.Services;
|
||||
|
||||
/// <summary>
|
||||
/// Adapter that bridges the new IVexSignatureVerifierV2 interface to the legacy IVexSignatureVerifier interface.
|
||||
/// This allows gradual migration while maintaining backward compatibility.
|
||||
/// </summary>
|
||||
public sealed class VexSignatureVerifierV1Adapter : IVexSignatureVerifier
|
||||
{
|
||||
private readonly IVexSignatureVerifierV2 _v2Verifier;
|
||||
private readonly VexSignatureVerifierOptions _options;
|
||||
private readonly ILogger<VexSignatureVerifierV1Adapter> _logger;
|
||||
|
||||
public VexSignatureVerifierV1Adapter(
|
||||
IVexSignatureVerifierV2 v2Verifier,
|
||||
IOptions<VexSignatureVerifierOptions> options,
|
||||
ILogger<VexSignatureVerifierV1Adapter> logger)
|
||||
{
|
||||
_v2Verifier = v2Verifier ?? throw new ArgumentNullException(nameof(v2Verifier));
|
||||
_options = options?.Value ?? throw new ArgumentNullException(nameof(options));
|
||||
_logger = logger ?? throw new ArgumentNullException(nameof(logger));
|
||||
}
|
||||
|
||||
/// <inheritdoc />
|
||||
public async ValueTask<VexSignatureMetadata?> VerifyAsync(
|
||||
VexRawDocument document,
|
||||
CancellationToken cancellationToken)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(document);
|
||||
|
||||
try
|
||||
{
|
||||
// Create verification context from options
|
||||
var context = new VexVerificationContext
|
||||
{
|
||||
TenantId = ExtractTenantId(document),
|
||||
CryptoProfile = _options.DefaultProfile,
|
||||
AllowExpiredCerts = _options.AllowExpiredCerts,
|
||||
RequireSignature = _options.RequireSignature,
|
||||
ClockTolerance = _options.ClockTolerance
|
||||
};
|
||||
|
||||
// Call V2 verifier
|
||||
var result = await _v2Verifier.VerifyAsync(document, context, cancellationToken);
|
||||
|
||||
// Convert V2 result to V1 VexSignatureMetadata
|
||||
return ConvertToV1Metadata(result);
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
_logger.LogWarning(ex, "V1 adapter verification failed for document {Digest}", document.Digest);
|
||||
|
||||
// Return null on error (V1 behavior - treat as unsigned)
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
private static string ExtractTenantId(VexRawDocument document)
|
||||
{
|
||||
// Try to extract tenant from document metadata
|
||||
if (document.Metadata.TryGetValue("tenant-id", out var tenantId) &&
|
||||
!string.IsNullOrWhiteSpace(tenantId))
|
||||
{
|
||||
return tenantId;
|
||||
}
|
||||
|
||||
if (document.Metadata.TryGetValue("x-stellaops-tenant", out tenantId) &&
|
||||
!string.IsNullOrWhiteSpace(tenantId))
|
||||
{
|
||||
return tenantId;
|
||||
}
|
||||
|
||||
// Default to global tenant
|
||||
return "@global";
|
||||
}
|
||||
|
||||
private static VexSignatureMetadata? ConvertToV1Metadata(VexSignatureVerificationResult result)
|
||||
{
|
||||
// No signature case
|
||||
if (result.Method == VerificationMethod.None && !result.Verified)
|
||||
{
|
||||
return null;
|
||||
}
|
||||
|
||||
// Failed verification - still return metadata but with details
|
||||
// This allows the caller to see that verification was attempted
|
||||
|
||||
var type = result.Method switch
|
||||
{
|
||||
VerificationMethod.Dsse => "dsse",
|
||||
VerificationMethod.DsseKeyless => "dsse-keyless",
|
||||
VerificationMethod.Cosign => "cosign",
|
||||
VerificationMethod.CosignKeyless => "cosign-keyless",
|
||||
VerificationMethod.Pgp => "pgp",
|
||||
VerificationMethod.X509 => "x509",
|
||||
VerificationMethod.InToto => "in-toto",
|
||||
VerificationMethod.None => "none",
|
||||
_ => "unknown"
|
||||
};
|
||||
|
||||
// Build transparency log reference
|
||||
string? transparencyLogRef = null;
|
||||
if (result.RekorLogIndex.HasValue)
|
||||
{
|
||||
transparencyLogRef = result.RekorLogId != null
|
||||
? $"{result.RekorLogId}:{result.RekorLogIndex}"
|
||||
: $"rekor:{result.RekorLogIndex}";
|
||||
}
|
||||
|
||||
// Build trust metadata if issuer is known
|
||||
VexSignatureTrustMetadata? trust = null;
|
||||
if (!string.IsNullOrEmpty(result.IssuerId))
|
||||
{
|
||||
trust = new VexSignatureTrustMetadata(
|
||||
effectiveWeight: 1.0m, // Full trust for verified signatures
|
||||
tenantId: "@global",
|
||||
issuerId: result.IssuerId,
|
||||
tenantOverrideApplied: false,
|
||||
retrievedAtUtc: result.VerifiedAt);
|
||||
}
|
||||
|
||||
return new VexSignatureMetadata(
|
||||
type: type,
|
||||
subject: result.CertSubject,
|
||||
issuer: result.IssuerName,
|
||||
keyId: result.KeyId,
|
||||
verifiedAt: result.Verified ? result.VerifiedAt : null,
|
||||
transparencyLogReference: transparencyLogRef,
|
||||
trust: trust);
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,31 @@
|
||||
using Microsoft.Extensions.Logging;
|
||||
|
||||
namespace StellaOps.Excititor.WebService.Services;
|
||||
|
||||
public sealed record VexStatementBackfillRequest(int BatchSize = 500);
|
||||
|
||||
public sealed record VexStatementBackfillResult(
|
||||
int DocumentsEvaluated,
|
||||
int DocumentsBackfilled,
|
||||
int ClaimsWritten,
|
||||
int SkippedExisting,
|
||||
int NormalizationFailures);
|
||||
|
||||
/// <summary>
|
||||
/// Placeholder backfill service while legacy statement storage is removed.
|
||||
/// </summary>
|
||||
public sealed class VexStatementBackfillService
|
||||
{
|
||||
private readonly ILogger<VexStatementBackfillService> _logger;
|
||||
|
||||
public VexStatementBackfillService(ILogger<VexStatementBackfillService> logger)
|
||||
{
|
||||
_logger = logger;
|
||||
}
|
||||
|
||||
public ValueTask<VexStatementBackfillResult> RunAsync(VexStatementBackfillRequest request, CancellationToken cancellationToken)
|
||||
{
|
||||
_logger.LogInformation("Vex statement backfill is currently a no-op; batchSize={BatchSize}", request.BatchSize);
|
||||
return ValueTask.FromResult(new VexStatementBackfillResult(0, 0, 0, 0, 0));
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,44 @@
|
||||
<?xml version='1.0' encoding='utf-8'?>
|
||||
<Project Sdk="Microsoft.NET.Sdk.Web">
|
||||
<PropertyGroup>
|
||||
<TargetFramework>net10.0</TargetFramework>
|
||||
<LangVersion>preview</LangVersion>
|
||||
<Nullable>enable</Nullable>
|
||||
<ImplicitUsings>enable</ImplicitUsings>
|
||||
<TreatWarningsAsErrors>true</TreatWarningsAsErrors>
|
||||
</PropertyGroup>
|
||||
<ItemGroup>
|
||||
<PackageReference Include="OpenTelemetry.Exporter.Console" />
|
||||
<PackageReference Include="OpenTelemetry.Exporter.OpenTelemetryProtocol" />
|
||||
<PackageReference Include="OpenTelemetry.Extensions.Hosting" />
|
||||
<PackageReference Include="OpenTelemetry.Instrumentation.AspNetCore" />
|
||||
<PackageReference Include="OpenTelemetry.Instrumentation.Http" />
|
||||
<PackageReference Include="OpenTelemetry.Instrumentation.Runtime" />
|
||||
</ItemGroup>
|
||||
<ItemGroup>
|
||||
<ProjectReference Include="../__Libraries/StellaOps.Excititor.Core/StellaOps.Excititor.Core.csproj" />
|
||||
<ProjectReference Include="../../Router/__Libraries/StellaOps.Messaging/StellaOps.Messaging.csproj" />
|
||||
<ProjectReference Include="../__Libraries/StellaOps.Excititor.Persistence/StellaOps.Excititor.Persistence.csproj" />
|
||||
<ProjectReference Include="../__Libraries/StellaOps.Excititor.Export/StellaOps.Excititor.Export.csproj" />
|
||||
<ProjectReference Include="../__Libraries/StellaOps.Excititor.Connectors.Abstractions/StellaOps.Excititor.Connectors.Abstractions.csproj" />
|
||||
<ProjectReference Include="../__Libraries/StellaOps.Excititor.Policy/StellaOps.Excititor.Policy.csproj" />
|
||||
<ProjectReference Include="../__Libraries/StellaOps.Excititor.Attestation/StellaOps.Excititor.Attestation.csproj" />
|
||||
<ProjectReference Include="../__Libraries/StellaOps.Excititor.ArtifactStores.S3/StellaOps.Excititor.ArtifactStores.S3.csproj" />
|
||||
<ProjectReference Include="../__Libraries/StellaOps.Excititor.Connectors.RedHat.CSAF/StellaOps.Excititor.Connectors.RedHat.CSAF.csproj" />
|
||||
<ProjectReference Include="../__Libraries/StellaOps.Excititor.Formats.CSAF/StellaOps.Excititor.Formats.CSAF.csproj" />
|
||||
<ProjectReference Include="../__Libraries/StellaOps.Excititor.Formats.CycloneDX/StellaOps.Excititor.Formats.CycloneDX.csproj" />
|
||||
<ProjectReference Include="../__Libraries/StellaOps.Excititor.Formats.OpenVEX/StellaOps.Excititor.Formats.OpenVEX.csproj" />
|
||||
<ProjectReference Include="../../__Libraries/StellaOps.Ingestion.Telemetry/StellaOps.Ingestion.Telemetry.csproj" />
|
||||
<ProjectReference Include="../../Aoc/__Libraries/StellaOps.Aoc/StellaOps.Aoc.csproj" />
|
||||
<ProjectReference Include="../../Router/__Libraries/StellaOps.Router.AspNet/StellaOps.Router.AspNet.csproj" />
|
||||
<ProjectReference Include="../../Authority/StellaOps.Authority/StellaOps.Auth.ServerIntegration/StellaOps.Auth.ServerIntegration.csproj" />
|
||||
<ProjectReference Include="../../__Libraries/StellaOps.Localization/StellaOps.Localization.csproj" />
|
||||
</ItemGroup>
|
||||
<ItemGroup>
|
||||
<EmbeddedResource Include="Translations\*.json" />
|
||||
</ItemGroup>
|
||||
<PropertyGroup Label="StellaOpsReleaseVersion">
|
||||
<Version>1.0.0-alpha1</Version>
|
||||
<InformationalVersion>1.0.0-alpha1</InformationalVersion>
|
||||
</PropertyGroup>
|
||||
</Project>
|
||||
10
src/Concelier/StellaOps.Excititor.WebService/TASKS.md
Normal file
10
src/Concelier/StellaOps.Excititor.WebService/TASKS.md
Normal file
@@ -0,0 +1,10 @@
|
||||
# Excititor WebService Task Board
|
||||
|
||||
This board mirrors active sprint tasks for this module.
|
||||
Source of truth: `docs-archived/implplan/2025-12-29-csproj-audit/SPRINT_20251229_049_BE_csproj_audit_maint_tests.md`.
|
||||
|
||||
| Task ID | Status | Notes |
|
||||
| --- | --- | --- |
|
||||
| AUDIT-0327-M | DONE | Revalidated 2026-01-07; maintainability audit for Excititor.WebService. |
|
||||
| AUDIT-0327-T | DONE | Revalidated 2026-01-07; test coverage audit for Excititor.WebService. |
|
||||
| AUDIT-0327-A | TODO | Pending approval (non-test project; revalidated 2026-01-07). |
|
||||
@@ -0,0 +1,49 @@
|
||||
using System.Diagnostics.Metrics;
|
||||
|
||||
namespace StellaOps.Excititor.WebService.Telemetry;
|
||||
|
||||
internal sealed class ChunkTelemetry
|
||||
{
|
||||
private readonly Counter<long> _ingestedTotal;
|
||||
private readonly Histogram<long> _itemCount;
|
||||
private readonly Histogram<long> _payloadBytes;
|
||||
private readonly Histogram<double> _latencyMs;
|
||||
|
||||
public ChunkTelemetry(IMeterFactory meterFactory)
|
||||
{
|
||||
var meter = meterFactory.Create("StellaOps.Excititor.Chunks");
|
||||
_ingestedTotal = meter.CreateCounter<long>(
|
||||
name: "vex_chunks_ingested_total",
|
||||
unit: "chunks",
|
||||
description: "Chunks submitted to Excititor VEX ingestion.");
|
||||
_itemCount = meter.CreateHistogram<long>(
|
||||
name: "vex_chunks_item_count",
|
||||
unit: "items",
|
||||
description: "Item count per submitted chunk.");
|
||||
_payloadBytes = meter.CreateHistogram<long>(
|
||||
name: "vex_chunks_payload_bytes",
|
||||
unit: "bytes",
|
||||
description: "Payload size per submitted chunk.");
|
||||
_latencyMs = meter.CreateHistogram<double>(
|
||||
name: "vex_chunks_latency_ms",
|
||||
unit: "ms",
|
||||
description: "End-to-end processing latency per chunk request.");
|
||||
}
|
||||
|
||||
public void RecordIngested(string? tenant, string? source, string status, string? reason, long itemCount, long payloadBytes, double latencyMs)
|
||||
{
|
||||
var tags = new KeyValuePair<string, object?>[]
|
||||
{
|
||||
new("tenant", tenant ?? string.Empty),
|
||||
new("source", source ?? string.Empty),
|
||||
new("status", status),
|
||||
new("reason", string.IsNullOrWhiteSpace(reason) ? string.Empty : reason)
|
||||
};
|
||||
|
||||
var tagSpan = tags.AsSpan();
|
||||
_ingestedTotal.Add(1, tagSpan);
|
||||
_itemCount.Record(itemCount, tagSpan);
|
||||
_payloadBytes.Record(payloadBytes, tagSpan);
|
||||
_latencyMs.Record(latencyMs, tagSpan);
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,15 @@
|
||||
using System.Diagnostics.Metrics;
|
||||
|
||||
namespace StellaOps.Excititor.WebService.Telemetry;
|
||||
|
||||
internal sealed class ConsoleTelemetry
|
||||
{
|
||||
public const string MeterName = "StellaOps.Excititor.Console";
|
||||
|
||||
private static readonly Meter Meter = new(MeterName);
|
||||
|
||||
public Counter<long> Requests { get; } = Meter.CreateCounter<long>("console.vex.requests");
|
||||
public Counter<long> CacheHits { get; } = Meter.CreateCounter<long>("console.vex.cache_hits");
|
||||
public Counter<long> CacheMisses { get; } = Meter.CreateCounter<long>("console.vex.cache_misses");
|
||||
}
|
||||
|
||||
@@ -0,0 +1,246 @@
|
||||
|
||||
using StellaOps.Excititor.Core.Aoc;
|
||||
using StellaOps.Excititor.WebService.Contracts;
|
||||
using StellaOps.Excititor.WebService.Services;
|
||||
using System;
|
||||
using System.Collections.Generic;
|
||||
using System.Diagnostics.Metrics;
|
||||
using System.Linq;
|
||||
|
||||
namespace StellaOps.Excititor.WebService.Telemetry;
|
||||
|
||||
internal static class EvidenceTelemetry
|
||||
{
|
||||
public const string MeterName = "StellaOps.Excititor.WebService.Evidence";
|
||||
|
||||
private static readonly Meter Meter = new(MeterName);
|
||||
|
||||
private static readonly Counter<long> ObservationRequestCounter =
|
||||
Meter.CreateCounter<long>(
|
||||
"excititor.vex.observation.requests",
|
||||
unit: "requests",
|
||||
description: "Number of observation projection requests handled by the evidence APIs.");
|
||||
|
||||
private static readonly Histogram<int> ObservationStatementHistogram =
|
||||
Meter.CreateHistogram<int>(
|
||||
"excititor.vex.observation.statement_count",
|
||||
unit: "statements",
|
||||
description: "Distribution of statements returned per observation projection request.");
|
||||
|
||||
private static readonly Counter<long> EvidenceRequestCounter =
|
||||
Meter.CreateCounter<long>(
|
||||
"excititor.vex.evidence.requests",
|
||||
unit: "requests",
|
||||
description: "Number of evidence chunk requests handled by the evidence APIs.");
|
||||
|
||||
private static readonly Histogram<int> EvidenceChunkHistogram =
|
||||
Meter.CreateHistogram<int>(
|
||||
"excititor.vex.evidence.chunk_count",
|
||||
unit: "chunks",
|
||||
description: "Distribution of evidence chunks streamed per request.");
|
||||
|
||||
private static readonly Counter<long> SignatureStatusCounter =
|
||||
Meter.CreateCounter<long>(
|
||||
"excititor.vex.signature.status",
|
||||
unit: "statements",
|
||||
description: "Signature verification status counts for observation statements.");
|
||||
|
||||
private static readonly Counter<long> GuardViolationCounter =
|
||||
Meter.CreateCounter<long>(
|
||||
"excititor.vex.aoc.guard_violations",
|
||||
unit: "violations",
|
||||
description: "Aggregated count of AOC guard violations detected by Excititor evidence APIs.");
|
||||
|
||||
public static void RecordObservationOutcome(string? tenant, string outcome, int returnedCount = 0, bool truncated = false)
|
||||
{
|
||||
var normalizedTenant = NormalizeTenant(tenant);
|
||||
var tags = new[]
|
||||
{
|
||||
new KeyValuePair<string, object?>("tenant", normalizedTenant),
|
||||
new KeyValuePair<string, object?>("outcome", outcome),
|
||||
new KeyValuePair<string, object?>("truncated", truncated),
|
||||
};
|
||||
|
||||
ObservationRequestCounter.Add(1, tags);
|
||||
|
||||
if (!string.Equals(outcome, "success", StringComparison.OrdinalIgnoreCase))
|
||||
{
|
||||
return;
|
||||
}
|
||||
|
||||
ObservationStatementHistogram.Record(returnedCount, tags);
|
||||
}
|
||||
|
||||
public static void RecordChunkOutcome(string? tenant, string outcome, int chunkCount = 0, bool truncated = false)
|
||||
{
|
||||
var normalizedTenant = NormalizeTenant(tenant);
|
||||
var tags = new[]
|
||||
{
|
||||
new KeyValuePair<string, object?>("tenant", normalizedTenant),
|
||||
new KeyValuePair<string, object?>("outcome", outcome),
|
||||
new KeyValuePair<string, object?>("truncated", truncated),
|
||||
};
|
||||
|
||||
EvidenceRequestCounter.Add(1, tags);
|
||||
|
||||
if (!string.Equals(outcome, "success", StringComparison.OrdinalIgnoreCase))
|
||||
{
|
||||
return;
|
||||
}
|
||||
|
||||
EvidenceChunkHistogram.Record(chunkCount, tags);
|
||||
}
|
||||
|
||||
public static void RecordSignatureStatus(string? tenant, IReadOnlyList<VexObservationStatementProjection> statements)
|
||||
{
|
||||
if (statements is null || statements.Count == 0)
|
||||
{
|
||||
return;
|
||||
}
|
||||
|
||||
var normalizedTenant = NormalizeTenant(tenant);
|
||||
var missing = 0;
|
||||
var unverified = 0;
|
||||
var verified = 0;
|
||||
|
||||
foreach (var statement in statements)
|
||||
{
|
||||
var signature = statement.Signature;
|
||||
if (signature is null)
|
||||
{
|
||||
missing++;
|
||||
continue;
|
||||
}
|
||||
|
||||
if (signature.VerifiedAt is null)
|
||||
{
|
||||
unverified++;
|
||||
}
|
||||
else
|
||||
{
|
||||
verified++;
|
||||
}
|
||||
}
|
||||
|
||||
if (missing > 0)
|
||||
{
|
||||
SignatureStatusCounter.Add(
|
||||
missing,
|
||||
new[]
|
||||
{
|
||||
new KeyValuePair<string, object?>("tenant", normalizedTenant),
|
||||
new KeyValuePair<string, object?>("status", "missing"),
|
||||
});
|
||||
}
|
||||
|
||||
if (unverified > 0)
|
||||
{
|
||||
SignatureStatusCounter.Add(
|
||||
unverified,
|
||||
BuildSignatureTags(normalizedTenant, "unverified"));
|
||||
}
|
||||
|
||||
if (verified > 0)
|
||||
{
|
||||
SignatureStatusCounter.Add(
|
||||
verified,
|
||||
BuildSignatureTags(normalizedTenant, "verified"));
|
||||
}
|
||||
}
|
||||
|
||||
public static void RecordChunkSignatureStatus(string? tenant, IReadOnlyList<VexEvidenceChunkResponse> chunks)
|
||||
{
|
||||
if (chunks is null || chunks.Count == 0)
|
||||
{
|
||||
return;
|
||||
}
|
||||
|
||||
var normalizedTenant = NormalizeTenant(tenant);
|
||||
|
||||
var unsigned = 0;
|
||||
var unverified = 0;
|
||||
var verified = 0;
|
||||
|
||||
foreach (var chunk in chunks)
|
||||
{
|
||||
var signature = chunk.Signature;
|
||||
if (signature is null)
|
||||
{
|
||||
unsigned++;
|
||||
continue;
|
||||
}
|
||||
|
||||
if (signature.VerifiedAt is null)
|
||||
{
|
||||
unverified++;
|
||||
}
|
||||
else
|
||||
{
|
||||
verified++;
|
||||
}
|
||||
}
|
||||
|
||||
if (unsigned > 0)
|
||||
{
|
||||
SignatureStatusCounter.Add(unsigned, BuildSignatureTags(normalizedTenant, "unsigned"));
|
||||
}
|
||||
|
||||
if (unverified > 0)
|
||||
{
|
||||
SignatureStatusCounter.Add(unverified, BuildSignatureTags(normalizedTenant, "unverified"));
|
||||
}
|
||||
|
||||
if (verified > 0)
|
||||
{
|
||||
SignatureStatusCounter.Add(verified, BuildSignatureTags(normalizedTenant, "verified"));
|
||||
}
|
||||
}
|
||||
|
||||
public static void RecordGuardViolations(string? tenant, string surface, ExcititorAocGuardException exception)
|
||||
{
|
||||
var normalizedTenant = NormalizeTenant(tenant);
|
||||
var normalizedSurface = NormalizeSurface(surface);
|
||||
|
||||
if (exception.Violations.IsDefaultOrEmpty)
|
||||
{
|
||||
GuardViolationCounter.Add(
|
||||
1,
|
||||
new[]
|
||||
{
|
||||
new KeyValuePair<string, object?>("tenant", normalizedTenant),
|
||||
new KeyValuePair<string, object?>("surface", normalizedSurface),
|
||||
new KeyValuePair<string, object?>("code", exception.PrimaryErrorCode),
|
||||
});
|
||||
return;
|
||||
}
|
||||
|
||||
foreach (var violation in exception.Violations)
|
||||
{
|
||||
var code = string.IsNullOrWhiteSpace(violation.ErrorCode)
|
||||
? exception.PrimaryErrorCode
|
||||
: violation.ErrorCode;
|
||||
|
||||
GuardViolationCounter.Add(
|
||||
1,
|
||||
new[]
|
||||
{
|
||||
new KeyValuePair<string, object?>("tenant", normalizedTenant),
|
||||
new KeyValuePair<string, object?>("surface", normalizedSurface),
|
||||
new KeyValuePair<string, object?>("code", code),
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
private static string NormalizeTenant(string? tenant)
|
||||
=> string.IsNullOrWhiteSpace(tenant) ? "default" : tenant;
|
||||
|
||||
private static string NormalizeSurface(string? surface)
|
||||
=> string.IsNullOrWhiteSpace(surface) ? "unknown" : surface.ToLowerInvariant();
|
||||
|
||||
private static KeyValuePair<string, object?>[] BuildSignatureTags(string tenant, string status)
|
||||
=> new[]
|
||||
{
|
||||
new KeyValuePair<string, object?>("tenant", tenant),
|
||||
new KeyValuePair<string, object?>("status", status),
|
||||
};
|
||||
}
|
||||
@@ -0,0 +1,251 @@
|
||||
|
||||
using StellaOps.Excititor.Core.Observations;
|
||||
using System;
|
||||
using System.Collections.Generic;
|
||||
using System.Diagnostics.Metrics;
|
||||
|
||||
namespace StellaOps.Excititor.WebService.Telemetry;
|
||||
|
||||
/// <summary>
|
||||
/// Telemetry metrics for VEX linkset and observation store operations (EXCITITOR-OBS-51-001).
|
||||
/// Tracks ingest latency, scope resolution success, conflict rate, and signature verification
|
||||
/// to support SLO burn alerts for AOC "evidence freshness" mission.
|
||||
/// </summary>
|
||||
internal static class LinksetTelemetry
|
||||
{
|
||||
public const string MeterName = "StellaOps.Excititor.WebService.Linksets";
|
||||
|
||||
private static readonly Meter Meter = new(MeterName);
|
||||
|
||||
// Ingest latency metrics
|
||||
private static readonly Histogram<double> IngestLatencyHistogram =
|
||||
Meter.CreateHistogram<double>(
|
||||
"excititor.vex.ingest.latency_seconds",
|
||||
unit: "s",
|
||||
description: "Latency distribution for VEX observation and linkset store operations.");
|
||||
|
||||
private static readonly Counter<long> IngestOperationCounter =
|
||||
Meter.CreateCounter<long>(
|
||||
"excititor.vex.ingest.operations_total",
|
||||
unit: "operations",
|
||||
description: "Total count of VEX ingest operations by outcome.");
|
||||
|
||||
// Scope resolution metrics
|
||||
private static readonly Counter<long> ScopeResolutionCounter =
|
||||
Meter.CreateCounter<long>(
|
||||
"excititor.vex.scope.resolution_total",
|
||||
unit: "resolutions",
|
||||
description: "Count of scope resolution attempts by outcome (success/failure).");
|
||||
|
||||
private static readonly Histogram<int> ScopeMatchCountHistogram =
|
||||
Meter.CreateHistogram<int>(
|
||||
"excititor.vex.scope.match_count",
|
||||
unit: "matches",
|
||||
description: "Distribution of matched scopes per resolution request.");
|
||||
|
||||
// Conflict/disagreement metrics
|
||||
private static readonly Counter<long> LinksetConflictCounter =
|
||||
Meter.CreateCounter<long>(
|
||||
"excititor.vex.linkset.conflicts_total",
|
||||
unit: "conflicts",
|
||||
description: "Total count of linksets with provider disagreements detected.");
|
||||
|
||||
private static readonly Histogram<int> DisagreementCountHistogram =
|
||||
Meter.CreateHistogram<int>(
|
||||
"excititor.vex.linkset.disagreement_count",
|
||||
unit: "disagreements",
|
||||
description: "Distribution of disagreement count per linkset.");
|
||||
|
||||
private static readonly Counter<long> DisagreementByStatusCounter =
|
||||
Meter.CreateCounter<long>(
|
||||
"excititor.vex.linkset.disagreement_by_status",
|
||||
unit: "disagreements",
|
||||
description: "Disagreement counts broken down by conflicting status values.");
|
||||
|
||||
// Observation store metrics
|
||||
private static readonly Counter<long> ObservationStoreCounter =
|
||||
Meter.CreateCounter<long>(
|
||||
"excititor.vex.observation.store_operations_total",
|
||||
unit: "operations",
|
||||
description: "Total observation store operations by type and outcome.");
|
||||
|
||||
private static readonly Histogram<int> ObservationBatchSizeHistogram =
|
||||
Meter.CreateHistogram<int>(
|
||||
"excititor.vex.observation.batch_size",
|
||||
unit: "observations",
|
||||
description: "Distribution of observation batch sizes for store operations.");
|
||||
|
||||
// Linkset store metrics
|
||||
private static readonly Counter<long> LinksetStoreCounter =
|
||||
Meter.CreateCounter<long>(
|
||||
"excititor.vex.linkset.store_operations_total",
|
||||
unit: "operations",
|
||||
description: "Total linkset store operations by type and outcome.");
|
||||
|
||||
// Confidence metrics
|
||||
private static readonly Histogram<double> LinksetConfidenceHistogram =
|
||||
Meter.CreateHistogram<double>(
|
||||
"excititor.vex.linkset.confidence_score",
|
||||
unit: "score",
|
||||
description: "Distribution of linkset confidence scores (0.0-1.0).");
|
||||
|
||||
/// <summary>
|
||||
/// Records latency for a VEX ingest operation.
|
||||
/// </summary>
|
||||
public static void RecordIngestLatency(string? tenant, string operation, string outcome, double latencySeconds)
|
||||
{
|
||||
var tags = BuildBaseTags(tenant, operation, outcome);
|
||||
IngestLatencyHistogram.Record(latencySeconds, tags);
|
||||
IngestOperationCounter.Add(1, tags);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Records a scope resolution attempt and its outcome.
|
||||
/// </summary>
|
||||
public static void RecordScopeResolution(string? tenant, string outcome, int matchCount = 0)
|
||||
{
|
||||
var normalizedTenant = NormalizeTenant(tenant);
|
||||
var tags = new[]
|
||||
{
|
||||
new KeyValuePair<string, object?>("tenant", normalizedTenant),
|
||||
new KeyValuePair<string, object?>("outcome", outcome),
|
||||
};
|
||||
|
||||
ScopeResolutionCounter.Add(1, tags);
|
||||
|
||||
if (string.Equals(outcome, "success", StringComparison.OrdinalIgnoreCase) && matchCount > 0)
|
||||
{
|
||||
ScopeMatchCountHistogram.Record(matchCount, tags);
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Records conflict detection for a linkset.
|
||||
/// </summary>
|
||||
public static void RecordLinksetConflict(string? tenant, bool hasConflicts, int disagreementCount = 0)
|
||||
{
|
||||
var normalizedTenant = NormalizeTenant(tenant);
|
||||
|
||||
if (hasConflicts)
|
||||
{
|
||||
var conflictTags = new[]
|
||||
{
|
||||
new KeyValuePair<string, object?>("tenant", normalizedTenant),
|
||||
};
|
||||
LinksetConflictCounter.Add(1, conflictTags);
|
||||
|
||||
if (disagreementCount > 0)
|
||||
{
|
||||
DisagreementCountHistogram.Record(disagreementCount, conflictTags);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Records a linkset with detailed disagreement breakdown.
|
||||
/// </summary>
|
||||
public static void RecordLinksetDisagreements(string? tenant, VexLinkset linkset)
|
||||
{
|
||||
if (linkset is null || !linkset.HasConflicts)
|
||||
{
|
||||
return;
|
||||
}
|
||||
|
||||
var normalizedTenant = NormalizeTenant(tenant);
|
||||
RecordLinksetConflict(normalizedTenant, true, linkset.Disagreements.Length);
|
||||
|
||||
// Record disagreements by status
|
||||
foreach (var disagreement in linkset.Disagreements)
|
||||
{
|
||||
var statusTags = new[]
|
||||
{
|
||||
new KeyValuePair<string, object?>("tenant", normalizedTenant),
|
||||
new KeyValuePair<string, object?>("status", disagreement.Status.ToLowerInvariant()),
|
||||
new KeyValuePair<string, object?>("provider", disagreement.ProviderId),
|
||||
};
|
||||
DisagreementByStatusCounter.Add(1, statusTags);
|
||||
}
|
||||
|
||||
// Record confidence score
|
||||
var confidenceScore = linkset.Confidence switch
|
||||
{
|
||||
VexLinksetConfidence.High => 0.9,
|
||||
VexLinksetConfidence.Medium => 0.7,
|
||||
VexLinksetConfidence.Low => 0.4,
|
||||
_ => 0.5
|
||||
};
|
||||
|
||||
var confidenceTags = new[]
|
||||
{
|
||||
new KeyValuePair<string, object?>("tenant", normalizedTenant),
|
||||
new KeyValuePair<string, object?>("has_conflicts", linkset.HasConflicts),
|
||||
};
|
||||
LinksetConfidenceHistogram.Record(confidenceScore, confidenceTags);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Records an observation store operation.
|
||||
/// </summary>
|
||||
public static void RecordObservationStoreOperation(
|
||||
string? tenant,
|
||||
string operation,
|
||||
string outcome,
|
||||
int batchSize = 1)
|
||||
{
|
||||
var tags = BuildBaseTags(tenant, operation, outcome);
|
||||
ObservationStoreCounter.Add(1, tags);
|
||||
|
||||
if (batchSize > 0 && string.Equals(outcome, "success", StringComparison.OrdinalIgnoreCase))
|
||||
{
|
||||
var batchTags = new[]
|
||||
{
|
||||
new KeyValuePair<string, object?>("tenant", NormalizeTenant(tenant)),
|
||||
new KeyValuePair<string, object?>("operation", operation),
|
||||
};
|
||||
ObservationBatchSizeHistogram.Record(batchSize, batchTags);
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Records a linkset store operation.
|
||||
/// </summary>
|
||||
public static void RecordLinksetStoreOperation(string? tenant, string operation, string outcome)
|
||||
{
|
||||
var tags = BuildBaseTags(tenant, operation, outcome);
|
||||
LinksetStoreCounter.Add(1, tags);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Records linkset confidence score distribution.
|
||||
/// </summary>
|
||||
public static void RecordLinksetConfidence(string? tenant, VexLinksetConfidence confidence, bool hasConflicts)
|
||||
{
|
||||
var score = confidence switch
|
||||
{
|
||||
VexLinksetConfidence.High => 0.9,
|
||||
VexLinksetConfidence.Medium => 0.7,
|
||||
VexLinksetConfidence.Low => 0.4,
|
||||
_ => 0.5
|
||||
};
|
||||
|
||||
var tags = new[]
|
||||
{
|
||||
new KeyValuePair<string, object?>("tenant", NormalizeTenant(tenant)),
|
||||
new KeyValuePair<string, object?>("has_conflicts", hasConflicts),
|
||||
new KeyValuePair<string, object?>("confidence_level", confidence.ToString().ToLowerInvariant()),
|
||||
};
|
||||
|
||||
LinksetConfidenceHistogram.Record(score, tags);
|
||||
}
|
||||
|
||||
private static string NormalizeTenant(string? tenant)
|
||||
=> string.IsNullOrWhiteSpace(tenant) ? "default" : tenant;
|
||||
|
||||
private static KeyValuePair<string, object?>[] BuildBaseTags(string? tenant, string operation, string outcome)
|
||||
=> new[]
|
||||
{
|
||||
new KeyValuePair<string, object?>("tenant", NormalizeTenant(tenant)),
|
||||
new KeyValuePair<string, object?>("operation", operation),
|
||||
new KeyValuePair<string, object?>("outcome", outcome),
|
||||
};
|
||||
}
|
||||
@@ -0,0 +1,319 @@
|
||||
|
||||
using StellaOps.Excititor.Core.Canonicalization;
|
||||
using System;
|
||||
using System.Collections.Generic;
|
||||
using System.Diagnostics.Metrics;
|
||||
|
||||
namespace StellaOps.Excititor.WebService.Telemetry;
|
||||
|
||||
/// <summary>
|
||||
/// Telemetry metrics for VEX normalization and canonicalization operations (EXCITITOR-VULN-29-004).
|
||||
/// Tracks advisory/product key canonicalization, normalization errors, suppression scopes,
|
||||
/// and withdrawn statement handling for Vuln Explorer and Advisory AI dashboards.
|
||||
/// </summary>
|
||||
internal static class NormalizationTelemetry
|
||||
{
|
||||
public const string MeterName = "StellaOps.Excititor.WebService.Normalization";
|
||||
|
||||
private static readonly Meter Meter = new(MeterName);
|
||||
|
||||
// Advisory key canonicalization metrics
|
||||
private static readonly Counter<long> AdvisoryKeyCanonicalizeCounter =
|
||||
Meter.CreateCounter<long>(
|
||||
"excititor.vex.canonicalize.advisory_key_total",
|
||||
unit: "operations",
|
||||
description: "Total advisory key canonicalization operations by outcome.");
|
||||
|
||||
private static readonly Counter<long> AdvisoryKeyCanonicalizeErrorCounter =
|
||||
Meter.CreateCounter<long>(
|
||||
"excititor.vex.canonicalize.advisory_key_errors",
|
||||
unit: "errors",
|
||||
description: "Advisory key canonicalization errors by error type.");
|
||||
|
||||
private static readonly Counter<long> AdvisoryKeyScopeCounter =
|
||||
Meter.CreateCounter<long>(
|
||||
"excititor.vex.canonicalize.advisory_key_scope",
|
||||
unit: "keys",
|
||||
description: "Advisory keys processed by scope (global, ecosystem, vendor, distribution, unknown).");
|
||||
|
||||
// Product key canonicalization metrics
|
||||
private static readonly Counter<long> ProductKeyCanonicalizeCounter =
|
||||
Meter.CreateCounter<long>(
|
||||
"excititor.vex.canonicalize.product_key_total",
|
||||
unit: "operations",
|
||||
description: "Total product key canonicalization operations by outcome.");
|
||||
|
||||
private static readonly Counter<long> ProductKeyCanonicalizeErrorCounter =
|
||||
Meter.CreateCounter<long>(
|
||||
"excititor.vex.canonicalize.product_key_errors",
|
||||
unit: "errors",
|
||||
description: "Product key canonicalization errors by error type.");
|
||||
|
||||
private static readonly Counter<long> ProductKeyScopeCounter =
|
||||
Meter.CreateCounter<long>(
|
||||
"excititor.vex.canonicalize.product_key_scope",
|
||||
unit: "keys",
|
||||
description: "Product keys processed by scope (package, component, ospackage, container, platform, unknown).");
|
||||
|
||||
private static readonly Counter<long> ProductKeyTypeCounter =
|
||||
Meter.CreateCounter<long>(
|
||||
"excititor.vex.canonicalize.product_key_type",
|
||||
unit: "keys",
|
||||
description: "Product keys processed by type (purl, cpe, rpm, deb, oci, platform, other).");
|
||||
|
||||
// Evidence retrieval metrics
|
||||
private static readonly Counter<long> EvidenceRetrievalCounter =
|
||||
Meter.CreateCounter<long>(
|
||||
"excititor.vex.evidence.retrieval_total",
|
||||
unit: "requests",
|
||||
description: "Total evidence retrieval requests by outcome.");
|
||||
|
||||
private static readonly Histogram<int> EvidenceStatementCountHistogram =
|
||||
Meter.CreateHistogram<int>(
|
||||
"excititor.vex.evidence.statement_count",
|
||||
unit: "statements",
|
||||
description: "Distribution of statements returned per evidence retrieval request.");
|
||||
|
||||
private static readonly Histogram<double> EvidenceRetrievalLatencyHistogram =
|
||||
Meter.CreateHistogram<double>(
|
||||
"excititor.vex.evidence.retrieval_latency_seconds",
|
||||
unit: "s",
|
||||
description: "Latency distribution for evidence retrieval operations.");
|
||||
|
||||
// Normalization error metrics
|
||||
private static readonly Counter<long> NormalizationErrorCounter =
|
||||
Meter.CreateCounter<long>(
|
||||
"excititor.vex.normalize.errors_total",
|
||||
unit: "errors",
|
||||
description: "Total normalization errors by type and provider.");
|
||||
|
||||
// Suppression scope metrics
|
||||
private static readonly Counter<long> SuppressionScopeCounter =
|
||||
Meter.CreateCounter<long>(
|
||||
"excititor.vex.suppression.scope_total",
|
||||
unit: "suppressions",
|
||||
description: "Suppression scope applications by scope type.");
|
||||
|
||||
private static readonly Counter<long> SuppressionAppliedCounter =
|
||||
Meter.CreateCounter<long>(
|
||||
"excititor.vex.suppression.applied_total",
|
||||
unit: "statements",
|
||||
description: "Statements affected by suppression scopes.");
|
||||
|
||||
// Withdrawn statement metrics
|
||||
private static readonly Counter<long> WithdrawnStatementCounter =
|
||||
Meter.CreateCounter<long>(
|
||||
"excititor.vex.withdrawn.statements_total",
|
||||
unit: "statements",
|
||||
description: "Total withdrawn statement detections by provider.");
|
||||
|
||||
private static readonly Counter<long> WithdrawnReplacementCounter =
|
||||
Meter.CreateCounter<long>(
|
||||
"excititor.vex.withdrawn.replacements_total",
|
||||
unit: "replacements",
|
||||
description: "Withdrawn statement replacements processed.");
|
||||
|
||||
/// <summary>
|
||||
/// Records a successful advisory key canonicalization.
|
||||
/// </summary>
|
||||
public static void RecordAdvisoryKeyCanonicalization(
|
||||
string? tenant,
|
||||
VexCanonicalAdvisoryKey result)
|
||||
{
|
||||
var normalizedTenant = NormalizeTenant(tenant);
|
||||
var scope = result.Scope.ToString().ToLowerInvariant();
|
||||
|
||||
AdvisoryKeyCanonicalizeCounter.Add(1, BuildOutcomeTags(normalizedTenant, "success"));
|
||||
AdvisoryKeyScopeCounter.Add(1, BuildScopeTags(normalizedTenant, scope));
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Records an advisory key canonicalization error.
|
||||
/// </summary>
|
||||
public static void RecordAdvisoryKeyCanonicalizeError(
|
||||
string? tenant,
|
||||
string errorType,
|
||||
string? advisoryKey = null)
|
||||
{
|
||||
var normalizedTenant = NormalizeTenant(tenant);
|
||||
var tags = new[]
|
||||
{
|
||||
new KeyValuePair<string, object?>("tenant", normalizedTenant),
|
||||
new KeyValuePair<string, object?>("error_type", errorType),
|
||||
};
|
||||
|
||||
AdvisoryKeyCanonicalizeCounter.Add(1, BuildOutcomeTags(normalizedTenant, "error"));
|
||||
AdvisoryKeyCanonicalizeErrorCounter.Add(1, tags);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Records a successful product key canonicalization.
|
||||
/// </summary>
|
||||
public static void RecordProductKeyCanonicalization(
|
||||
string? tenant,
|
||||
VexCanonicalProductKey result)
|
||||
{
|
||||
var normalizedTenant = NormalizeTenant(tenant);
|
||||
var scope = result.Scope.ToString().ToLowerInvariant();
|
||||
var keyType = result.KeyType.ToString().ToLowerInvariant();
|
||||
|
||||
ProductKeyCanonicalizeCounter.Add(1, BuildOutcomeTags(normalizedTenant, "success"));
|
||||
ProductKeyScopeCounter.Add(1, BuildScopeTags(normalizedTenant, scope));
|
||||
ProductKeyTypeCounter.Add(1, new[]
|
||||
{
|
||||
new KeyValuePair<string, object?>("tenant", normalizedTenant),
|
||||
new KeyValuePair<string, object?>("key_type", keyType),
|
||||
});
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Records a product key canonicalization error.
|
||||
/// </summary>
|
||||
public static void RecordProductKeyCanonicalizeError(
|
||||
string? tenant,
|
||||
string errorType,
|
||||
string? productKey = null)
|
||||
{
|
||||
var normalizedTenant = NormalizeTenant(tenant);
|
||||
var tags = new[]
|
||||
{
|
||||
new KeyValuePair<string, object?>("tenant", normalizedTenant),
|
||||
new KeyValuePair<string, object?>("error_type", errorType),
|
||||
};
|
||||
|
||||
ProductKeyCanonicalizeCounter.Add(1, BuildOutcomeTags(normalizedTenant, "error"));
|
||||
ProductKeyCanonicalizeErrorCounter.Add(1, tags);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Records an evidence retrieval operation.
|
||||
/// </summary>
|
||||
public static void RecordEvidenceRetrieval(
|
||||
string? tenant,
|
||||
string outcome,
|
||||
int statementCount,
|
||||
double latencySeconds)
|
||||
{
|
||||
var normalizedTenant = NormalizeTenant(tenant);
|
||||
var tags = BuildOutcomeTags(normalizedTenant, outcome);
|
||||
|
||||
EvidenceRetrievalCounter.Add(1, tags);
|
||||
|
||||
if (string.Equals(outcome, "success", StringComparison.OrdinalIgnoreCase))
|
||||
{
|
||||
EvidenceStatementCountHistogram.Record(statementCount, tags);
|
||||
}
|
||||
|
||||
EvidenceRetrievalLatencyHistogram.Record(latencySeconds, tags);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Records a normalization error.
|
||||
/// </summary>
|
||||
public static void RecordNormalizationError(
|
||||
string? tenant,
|
||||
string provider,
|
||||
string errorType,
|
||||
string? detail = null)
|
||||
{
|
||||
var normalizedTenant = NormalizeTenant(tenant);
|
||||
var tags = new[]
|
||||
{
|
||||
new KeyValuePair<string, object?>("tenant", normalizedTenant),
|
||||
new KeyValuePair<string, object?>("provider", string.IsNullOrWhiteSpace(provider) ? "unknown" : provider),
|
||||
new KeyValuePair<string, object?>("error_type", errorType),
|
||||
};
|
||||
|
||||
NormalizationErrorCounter.Add(1, tags);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Records a suppression scope application.
|
||||
/// </summary>
|
||||
public static void RecordSuppressionScope(
|
||||
string? tenant,
|
||||
string scopeType,
|
||||
int affectedStatements)
|
||||
{
|
||||
var normalizedTenant = NormalizeTenant(tenant);
|
||||
var scopeTags = new[]
|
||||
{
|
||||
new KeyValuePair<string, object?>("tenant", normalizedTenant),
|
||||
new KeyValuePair<string, object?>("scope_type", scopeType),
|
||||
};
|
||||
|
||||
SuppressionScopeCounter.Add(1, scopeTags);
|
||||
|
||||
if (affectedStatements > 0)
|
||||
{
|
||||
SuppressionAppliedCounter.Add(affectedStatements, scopeTags);
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Records a withdrawn statement detection.
|
||||
/// </summary>
|
||||
public static void RecordWithdrawnStatement(
|
||||
string? tenant,
|
||||
string provider,
|
||||
string? replacementId = null)
|
||||
{
|
||||
var normalizedTenant = NormalizeTenant(tenant);
|
||||
var tags = new[]
|
||||
{
|
||||
new KeyValuePair<string, object?>("tenant", normalizedTenant),
|
||||
new KeyValuePair<string, object?>("provider", string.IsNullOrWhiteSpace(provider) ? "unknown" : provider),
|
||||
};
|
||||
|
||||
WithdrawnStatementCounter.Add(1, tags);
|
||||
|
||||
if (!string.IsNullOrWhiteSpace(replacementId))
|
||||
{
|
||||
WithdrawnReplacementCounter.Add(1, tags);
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Records batch withdrawn statement processing.
|
||||
/// </summary>
|
||||
public static void RecordWithdrawnStatements(
|
||||
string? tenant,
|
||||
string provider,
|
||||
int totalWithdrawn,
|
||||
int replacements)
|
||||
{
|
||||
var normalizedTenant = NormalizeTenant(tenant);
|
||||
var tags = new[]
|
||||
{
|
||||
new KeyValuePair<string, object?>("tenant", normalizedTenant),
|
||||
new KeyValuePair<string, object?>("provider", string.IsNullOrWhiteSpace(provider) ? "unknown" : provider),
|
||||
};
|
||||
|
||||
if (totalWithdrawn > 0)
|
||||
{
|
||||
WithdrawnStatementCounter.Add(totalWithdrawn, tags);
|
||||
}
|
||||
|
||||
if (replacements > 0)
|
||||
{
|
||||
WithdrawnReplacementCounter.Add(replacements, tags);
|
||||
}
|
||||
}
|
||||
|
||||
private static string NormalizeTenant(string? tenant)
|
||||
=> string.IsNullOrWhiteSpace(tenant) ? "default" : tenant;
|
||||
|
||||
private static KeyValuePair<string, object?>[] BuildOutcomeTags(string tenant, string outcome)
|
||||
=> new[]
|
||||
{
|
||||
new KeyValuePair<string, object?>("tenant", tenant),
|
||||
new KeyValuePair<string, object?>("outcome", outcome),
|
||||
};
|
||||
|
||||
private static KeyValuePair<string, object?>[] BuildScopeTags(string tenant, string scope)
|
||||
=> new[]
|
||||
{
|
||||
new KeyValuePair<string, object?>("tenant", tenant),
|
||||
new KeyValuePair<string, object?>("scope", scope),
|
||||
};
|
||||
}
|
||||
@@ -0,0 +1,87 @@
|
||||
using Microsoft.Extensions.Logging;
|
||||
using StellaOps.Excititor.Core;
|
||||
|
||||
namespace StellaOps.Excititor.WebService.Telemetry;
|
||||
|
||||
/// <summary>
|
||||
/// Implementation of <see cref="IVexNormalizationTelemetryRecorder"/> that bridges to
|
||||
/// <see cref="NormalizationTelemetry"/> static metrics and structured logging (EXCITITOR-VULN-29-004).
|
||||
/// </summary>
|
||||
internal sealed class VexNormalizationTelemetryRecorder : IVexNormalizationTelemetryRecorder
|
||||
{
|
||||
private readonly ILogger<VexNormalizationTelemetryRecorder> _logger;
|
||||
|
||||
public VexNormalizationTelemetryRecorder(ILogger<VexNormalizationTelemetryRecorder> logger)
|
||||
{
|
||||
_logger = logger ?? throw new ArgumentNullException(nameof(logger));
|
||||
}
|
||||
|
||||
public void RecordNormalizationError(string? tenant, string provider, string errorType, string? detail = null)
|
||||
{
|
||||
NormalizationTelemetry.RecordNormalizationError(tenant, provider, errorType, detail);
|
||||
|
||||
_logger.LogWarning(
|
||||
"VEX normalization error: tenant={Tenant} provider={Provider} errorType={ErrorType} detail={Detail}",
|
||||
tenant ?? "default",
|
||||
provider,
|
||||
errorType,
|
||||
detail ?? "(none)");
|
||||
}
|
||||
|
||||
public void RecordSuppressionScope(string? tenant, string scopeType, int affectedStatements)
|
||||
{
|
||||
NormalizationTelemetry.RecordSuppressionScope(tenant, scopeType, affectedStatements);
|
||||
|
||||
if (affectedStatements > 0)
|
||||
{
|
||||
_logger.LogInformation(
|
||||
"VEX suppression scope applied: tenant={Tenant} scopeType={ScopeType} affectedStatements={AffectedStatements}",
|
||||
tenant ?? "default",
|
||||
scopeType,
|
||||
affectedStatements);
|
||||
}
|
||||
else
|
||||
{
|
||||
_logger.LogDebug(
|
||||
"VEX suppression scope checked (no statements affected): tenant={Tenant} scopeType={ScopeType}",
|
||||
tenant ?? "default",
|
||||
scopeType);
|
||||
}
|
||||
}
|
||||
|
||||
public void RecordWithdrawnStatement(string? tenant, string provider, string? replacementId = null)
|
||||
{
|
||||
NormalizationTelemetry.RecordWithdrawnStatement(tenant, provider, replacementId);
|
||||
|
||||
if (string.IsNullOrWhiteSpace(replacementId))
|
||||
{
|
||||
_logger.LogInformation(
|
||||
"VEX withdrawn statement detected: tenant={Tenant} provider={Provider}",
|
||||
tenant ?? "default",
|
||||
provider);
|
||||
}
|
||||
else
|
||||
{
|
||||
_logger.LogInformation(
|
||||
"VEX withdrawn statement superseded: tenant={Tenant} provider={Provider} replacementId={ReplacementId}",
|
||||
tenant ?? "default",
|
||||
provider,
|
||||
replacementId);
|
||||
}
|
||||
}
|
||||
|
||||
public void RecordWithdrawnStatements(string? tenant, string provider, int totalWithdrawn, int replacements)
|
||||
{
|
||||
NormalizationTelemetry.RecordWithdrawnStatements(tenant, provider, totalWithdrawn, replacements);
|
||||
|
||||
if (totalWithdrawn > 0)
|
||||
{
|
||||
_logger.LogInformation(
|
||||
"VEX withdrawn statements batch: tenant={Tenant} provider={Provider} totalWithdrawn={TotalWithdrawn} replacements={Replacements}",
|
||||
tenant ?? "default",
|
||||
provider,
|
||||
totalWithdrawn,
|
||||
replacements);
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,19 @@
|
||||
{
|
||||
"_meta": { "locale": "en-US", "namespace": "excititor", "version": "1.0" },
|
||||
|
||||
"excititor.error.observation_not_found": "Observation '{0}' not found.",
|
||||
"excititor.error.bundle_not_found": "Requested bundleId '{0}' not found for current filters.",
|
||||
"excititor.error.attestation_service_unavailable": "Attestation service is not configured.",
|
||||
|
||||
"excititor.validation.vuln_and_product_required": "At least one vulnerabilityId and productKey are required.",
|
||||
"excititor.validation.vuln_and_product_required_short": "vulnerabilityId and productKey are required.",
|
||||
"excititor.validation.no_claims_available": "No claims available for the requested filters.",
|
||||
"excititor.validation.filter_required": "At least one filter is required: vulnerabilityId+productKey or providerId.",
|
||||
"excititor.validation.linkset_filter_required": "At least one filter is required: vulnerabilityId, productKey, providerId, or hasConflicts=true.",
|
||||
"excititor.validation.advisory_keys_or_purls_required": "advisory_keys or purls must be provided.",
|
||||
"excititor.validation.observation_id_required": "observationId is required.",
|
||||
"excititor.validation.observation_ids_required": "observationIds is required and must not be empty.",
|
||||
"excititor.validation.observation_ids_max": "Maximum 100 observations per batch.",
|
||||
"excititor.validation.request_payload_required": "Request payload is required.",
|
||||
"excititor.validation.tenant_header_required": "X-Stella-Tenant header is required."
|
||||
}
|
||||
40
src/Concelier/StellaOps.Excititor.Worker/AGENTS.md
Normal file
40
src/Concelier/StellaOps.Excititor.Worker/AGENTS.md
Normal file
@@ -0,0 +1,40 @@
|
||||
# Excititor Worker Charter
|
||||
|
||||
## Mission
|
||||
Run Excititor background jobs (ingestion, linkset extraction, dedup/idempotency enforcement) under the Aggregation-Only Contract; orchestrate Core + Storage without applying consensus or severity.
|
||||
|
||||
## Scope
|
||||
- Working directory: `src/Excititor/StellaOps.Excititor.Worker`
|
||||
- Job runners, pipelines, scheduling, DI wiring, health checks, telemetry for background tasks.
|
||||
|
||||
## Required Reading
|
||||
- `docs/modules/excititor/architecture.md`
|
||||
- `docs/modules/excititor/vex_observations.md`
|
||||
- `docs/modules/concelier/guides/aggregation-only-contract.md`
|
||||
- `docs-archived/implplan/implementation-plans/excititor-implementation-plan.md`
|
||||
|
||||
## Roles
|
||||
- Backend/worker engineer (.NET 10).
|
||||
- QA automation (background job + integration tests).
|
||||
|
||||
## Working Agreements
|
||||
1. Track task status in sprint files; log notable operational decisions in Execution Log.
|
||||
2. Respect tenant isolation on all job inputs/outputs; never process cross-tenant data.
|
||||
3. Idempotent processing only: guard against duplicate bundles and repeated messages.
|
||||
4. Offline-first; no external fetches during jobs.
|
||||
5. Observability: structured logs, counters, and optional OTEL traces behind config flags.
|
||||
|
||||
## Testing & Determinism
|
||||
- Provide deterministic job fixtures with seeded clocks/IDs; assert stable ordering of outputs and retries.
|
||||
- Simulate failure/retry paths; ensure idempotent writes in Storage.
|
||||
- Keep timestamps UTC ISO-8601; inject clock/GUID providers for tests.
|
||||
|
||||
## Boundaries
|
||||
- Delegate domain logic to Core and persistence to Storage.Postgres; avoid embedding policy or UI concerns.
|
||||
- Configuration via appsettings/environment; no hard-coded secrets.
|
||||
|
||||
## Ready-to-Start Checklist
|
||||
- Required docs reviewed.
|
||||
- Test harness prepared for background jobs (including retry/backoff settings).
|
||||
- Feature flags defined for new pipelines before enabling in production runs.
|
||||
|
||||
@@ -0,0 +1,8 @@
|
||||
using System.Net.Http;
|
||||
|
||||
namespace StellaOps.Excititor.Worker.Auth;
|
||||
|
||||
public interface ITenantAuthorityClientFactory
|
||||
{
|
||||
HttpClient Create(string tenant);
|
||||
}
|
||||
@@ -0,0 +1,46 @@
|
||||
|
||||
using Microsoft.Extensions.Options;
|
||||
using StellaOps.Excititor.Worker.Options;
|
||||
using System;
|
||||
using System.Net.Http;
|
||||
using System.Net.Http.Headers;
|
||||
|
||||
namespace StellaOps.Excititor.Worker.Auth;
|
||||
|
||||
/// <summary>
|
||||
/// Minimal tenant-scoped Authority client factory.
|
||||
/// Throws if tenant is missing or not configured, enforcing tenant isolation.
|
||||
/// </summary>
|
||||
public sealed class TenantAuthorityClientFactory : ITenantAuthorityClientFactory
|
||||
{
|
||||
internal const string AuthorityClientName = "StellaOps.Excititor.Worker.Authority";
|
||||
private readonly TenantAuthorityOptions _options;
|
||||
private readonly IHttpClientFactory _httpClientFactory;
|
||||
|
||||
public TenantAuthorityClientFactory(IOptions<TenantAuthorityOptions> options, IHttpClientFactory httpClientFactory)
|
||||
{
|
||||
_options = options?.Value ?? throw new ArgumentNullException(nameof(options));
|
||||
_httpClientFactory = httpClientFactory ?? throw new ArgumentNullException(nameof(httpClientFactory));
|
||||
}
|
||||
|
||||
public HttpClient Create(string tenant)
|
||||
{
|
||||
if (string.IsNullOrWhiteSpace(tenant))
|
||||
{
|
||||
throw new ArgumentException("Tenant is required for Authority client creation.", nameof(tenant));
|
||||
}
|
||||
|
||||
if (!_options.BaseUrls.TryGetValue(tenant, out var baseUrl) || string.IsNullOrWhiteSpace(baseUrl))
|
||||
{
|
||||
throw new InvalidOperationException($"Authority base URL not configured for tenant '{tenant}'.");
|
||||
}
|
||||
|
||||
var client = _httpClientFactory.CreateClient(AuthorityClientName);
|
||||
client.BaseAddress = new Uri(baseUrl, UriKind.Absolute);
|
||||
|
||||
client.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue("application/json"));
|
||||
client.DefaultRequestHeaders.Add("X-Tenant", tenant);
|
||||
|
||||
return client;
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,25 @@
|
||||
using System.Collections.Generic;
|
||||
|
||||
namespace StellaOps.Excititor.Worker.Options;
|
||||
|
||||
/// <summary>
|
||||
/// Per-tenant Authority endpoints and client credentials used by worker services.
|
||||
/// When DisableConsensus is true, these settings are still required for tenant-scoped provenance checks.
|
||||
/// </summary>
|
||||
public sealed class TenantAuthorityOptions
|
||||
{
|
||||
/// <summary>
|
||||
/// Map of tenant slug → base URL for Authority.
|
||||
/// </summary>
|
||||
public IDictionary<string, string> BaseUrls { get; } = new Dictionary<string, string>(StringComparer.OrdinalIgnoreCase);
|
||||
|
||||
/// <summary>
|
||||
/// Optional map of tenant slug → clientId.
|
||||
/// </summary>
|
||||
public IDictionary<string, string> ClientIds { get; } = new Dictionary<string, string>(StringComparer.OrdinalIgnoreCase);
|
||||
|
||||
/// <summary>
|
||||
/// Optional map of tenant slug → clientSecret.
|
||||
/// </summary>
|
||||
public IDictionary<string, string> ClientSecrets { get; } = new Dictionary<string, string>(StringComparer.OrdinalIgnoreCase);
|
||||
}
|
||||
@@ -0,0 +1,29 @@
|
||||
using Microsoft.Extensions.Options;
|
||||
|
||||
namespace StellaOps.Excititor.Worker.Options;
|
||||
|
||||
internal sealed class TenantAuthorityOptionsValidator : IValidateOptions<TenantAuthorityOptions>
|
||||
{
|
||||
public ValidateOptionsResult Validate(string? name, TenantAuthorityOptions options)
|
||||
{
|
||||
if (options is null)
|
||||
{
|
||||
return ValidateOptionsResult.Fail("TenantAuthorityOptions is required.");
|
||||
}
|
||||
|
||||
if (options.BaseUrls.Count == 0)
|
||||
{
|
||||
return ValidateOptionsResult.Fail("Excititor:Authority:BaseUrls must define at least one tenant endpoint.");
|
||||
}
|
||||
|
||||
foreach (var kvp in options.BaseUrls)
|
||||
{
|
||||
if (string.IsNullOrWhiteSpace(kvp.Key) || string.IsNullOrWhiteSpace(kvp.Value))
|
||||
{
|
||||
return ValidateOptionsResult.Fail("Excititor:Authority:BaseUrls must include non-empty tenant keys and URLs.");
|
||||
}
|
||||
}
|
||||
|
||||
return ValidateOptionsResult.Success;
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,78 @@
|
||||
|
||||
using StellaOps.Excititor.Core;
|
||||
using StellaOps.Excititor.Worker.Scheduling;
|
||||
using System.Collections.Generic;
|
||||
using System.Collections.Immutable;
|
||||
|
||||
namespace StellaOps.Excititor.Worker.Options;
|
||||
|
||||
public sealed class VexWorkerOptions
|
||||
{
|
||||
public TimeSpan DefaultInterval { get; set; } = TimeSpan.FromHours(1);
|
||||
|
||||
public TimeSpan OfflineInterval { get; set; } = TimeSpan.FromHours(6);
|
||||
|
||||
public TimeSpan DefaultInitialDelay { get; set; } = TimeSpan.FromMinutes(5);
|
||||
|
||||
public bool OfflineMode { get; set; }
|
||||
|
||||
// Aggregation-only cutover: when true, consensus refresh stays disabled to enforce fact-only ingests.
|
||||
public bool DisableConsensus { get; set; } = true;
|
||||
|
||||
public IList<VexWorkerProviderOptions> Providers { get; } = new List<VexWorkerProviderOptions>();
|
||||
|
||||
public VexWorkerRetryOptions Retry { get; } = new();
|
||||
|
||||
public VexWorkerRefreshOptions Refresh { get; } = new();
|
||||
|
||||
internal IReadOnlyList<VexWorkerSchedule> ResolveSchedules()
|
||||
{
|
||||
var schedules = new List<VexWorkerSchedule>();
|
||||
foreach (var provider in Providers)
|
||||
{
|
||||
if (!provider.Enabled)
|
||||
{
|
||||
continue;
|
||||
}
|
||||
|
||||
var providerId = provider.ProviderId?.Trim();
|
||||
if (string.IsNullOrWhiteSpace(providerId))
|
||||
{
|
||||
continue;
|
||||
}
|
||||
|
||||
var interval = provider.Interval ?? (OfflineMode ? OfflineInterval : DefaultInterval);
|
||||
if (interval <= TimeSpan.Zero)
|
||||
{
|
||||
continue;
|
||||
}
|
||||
|
||||
var initialDelay = provider.InitialDelay ?? DefaultInitialDelay;
|
||||
if (initialDelay < TimeSpan.Zero)
|
||||
{
|
||||
initialDelay = TimeSpan.Zero;
|
||||
}
|
||||
|
||||
var connectorSettings = provider.Settings.Count == 0
|
||||
? VexConnectorSettings.Empty
|
||||
: new VexConnectorSettings(provider.Settings.ToImmutableDictionary(StringComparer.Ordinal));
|
||||
|
||||
schedules.Add(new VexWorkerSchedule(providerId, interval, initialDelay, connectorSettings));
|
||||
}
|
||||
|
||||
return schedules;
|
||||
}
|
||||
}
|
||||
|
||||
public sealed class VexWorkerProviderOptions
|
||||
{
|
||||
public string ProviderId { get; set; } = string.Empty;
|
||||
|
||||
public bool Enabled { get; set; } = true;
|
||||
|
||||
public TimeSpan? Interval { get; set; }
|
||||
|
||||
public TimeSpan? InitialDelay { get; set; }
|
||||
|
||||
public IDictionary<string, string> Settings { get; } = new Dictionary<string, string>(StringComparer.Ordinal);
|
||||
}
|
||||
@@ -0,0 +1,32 @@
|
||||
|
||||
using Microsoft.Extensions.Options;
|
||||
using System.Collections.Generic;
|
||||
|
||||
namespace StellaOps.Excititor.Worker.Options;
|
||||
|
||||
public sealed class VexWorkerOptionsValidator : IValidateOptions<VexWorkerOptions>
|
||||
{
|
||||
public ValidateOptionsResult Validate(string? name, VexWorkerOptions options)
|
||||
{
|
||||
if (options is null)
|
||||
{
|
||||
return ValidateOptionsResult.Fail("Excititor.Worker options cannot be null.");
|
||||
}
|
||||
|
||||
var failures = new List<string>();
|
||||
|
||||
if (options.Refresh.ScanBatchSize <= 0)
|
||||
{
|
||||
failures.Add("Excititor.Worker.Refresh.ScanBatchSize must be greater than zero.");
|
||||
}
|
||||
|
||||
if (options.DisableConsensus && options.Refresh.Enabled)
|
||||
{
|
||||
failures.Add("Excititor.Worker.DisableConsensus=true requires Refresh.Enabled=false.");
|
||||
}
|
||||
|
||||
return failures.Count == 0
|
||||
? ValidateOptionsResult.Success
|
||||
: ValidateOptionsResult.Fail(failures);
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,69 @@
|
||||
using System;
|
||||
|
||||
namespace StellaOps.Excititor.Worker.Options;
|
||||
|
||||
/// <summary>
|
||||
/// Configuration options for the orchestrator worker SDK integration.
|
||||
/// </summary>
|
||||
public sealed class VexWorkerOrchestratorOptions
|
||||
{
|
||||
/// <summary>Whether orchestrator integration is enabled.</summary>
|
||||
public bool Enabled { get; set; } = true;
|
||||
|
||||
/// <summary>Base address of the Orchestrator WebService (e.g. "https://orch.local/").</summary>
|
||||
public Uri? BaseAddress { get; set; }
|
||||
|
||||
/// <summary>Logical job type registered with Orchestrator.</summary>
|
||||
public string JobType { get; set; } = "exc-vex-ingest";
|
||||
|
||||
/// <summary>Unique worker identifier presented to Orchestrator.</summary>
|
||||
public string WorkerId { get; set; } = "excititor-worker";
|
||||
|
||||
/// <summary>Optional task runner identifier (e.g. host name or pod).</summary>
|
||||
public string? TaskRunnerId { get; set; }
|
||||
|
||||
/// <summary>Tenant header name; defaults to Orchestrator default.</summary>
|
||||
public string TenantHeader { get; set; } = "X-Tenant-Id";
|
||||
|
||||
/// <summary>Tenant value to present when claiming jobs.</summary>
|
||||
public string DefaultTenant { get; set; } = "default";
|
||||
|
||||
/// <summary>API key header name for worker auth.</summary>
|
||||
public string ApiKeyHeader { get; set; } = "X-Worker-Token";
|
||||
|
||||
/// <summary>Optional API key value.</summary>
|
||||
public string? ApiKey { get; set; }
|
||||
|
||||
/// <summary>Optional bearer token value.</summary>
|
||||
public string? BearerToken { get; set; }
|
||||
|
||||
/// <summary>Interval between heartbeat emissions during job execution.</summary>
|
||||
public TimeSpan HeartbeatInterval { get; set; } = TimeSpan.FromSeconds(30);
|
||||
|
||||
/// <summary>Minimum heartbeat interval (safety floor).</summary>
|
||||
public TimeSpan MinHeartbeatInterval { get; set; } = TimeSpan.FromSeconds(5);
|
||||
|
||||
/// <summary>Maximum heartbeat interval (safety cap).</summary>
|
||||
public TimeSpan MaxHeartbeatInterval { get; set; } = TimeSpan.FromMinutes(2);
|
||||
|
||||
/// <summary>Lease duration requested when claiming jobs.</summary>
|
||||
public TimeSpan DefaultLeaseDuration { get; set; } = TimeSpan.FromMinutes(5);
|
||||
|
||||
/// <summary>Lease extension requested on each heartbeat.</summary>
|
||||
public TimeSpan HeartbeatExtend { get; set; } = TimeSpan.FromMinutes(1);
|
||||
|
||||
/// <summary>HTTP request timeout when talking to Orchestrator.</summary>
|
||||
public TimeSpan RequestTimeout { get; set; } = TimeSpan.FromSeconds(15);
|
||||
|
||||
/// <summary>Enable verbose logging for heartbeat/artifact events.</summary>
|
||||
public bool EnableVerboseLogging { get; set; }
|
||||
|
||||
/// <summary>Maximum number of artifact hashes to retain in state.</summary>
|
||||
public int MaxArtifactHashes { get; set; } = 1000;
|
||||
|
||||
/// <summary>Emit progress events for artifacts while running.</summary>
|
||||
public bool EmitProgressForArtifacts { get; set; } = true;
|
||||
|
||||
/// <summary>Fallback to local state only when orchestrator is unreachable.</summary>
|
||||
public bool AllowLocalFallback { get; set; } = true;
|
||||
}
|
||||
@@ -0,0 +1,21 @@
|
||||
using System;
|
||||
using System.IO;
|
||||
|
||||
namespace StellaOps.Excititor.Worker.Options;
|
||||
|
||||
public sealed class VexWorkerPluginOptions
|
||||
{
|
||||
public string? Directory { get; set; }
|
||||
|
||||
public string? SearchPattern { get; set; }
|
||||
|
||||
internal string ResolveDirectory()
|
||||
=> string.IsNullOrWhiteSpace(Directory)
|
||||
? Path.Combine(AppContext.BaseDirectory, "plugins")
|
||||
: Path.GetFullPath(Directory);
|
||||
|
||||
internal string ResolveSearchPattern()
|
||||
=> string.IsNullOrWhiteSpace(SearchPattern)
|
||||
? "StellaOps.Excititor.Connectors.*.dll"
|
||||
: SearchPattern!;
|
||||
}
|
||||
@@ -0,0 +1,90 @@
|
||||
using System.Collections.Generic;
|
||||
using System.Linq;
|
||||
|
||||
namespace StellaOps.Excititor.Worker.Options;
|
||||
|
||||
public sealed class VexWorkerRefreshOptions
|
||||
{
|
||||
private static readonly TimeSpan DefaultScanInterval = TimeSpan.FromMinutes(10);
|
||||
private static readonly TimeSpan DefaultConsensusTtl = TimeSpan.FromHours(2);
|
||||
|
||||
public bool Enabled { get; set; } = true;
|
||||
|
||||
public TimeSpan ScanInterval { get; set; } = DefaultScanInterval;
|
||||
|
||||
public TimeSpan ConsensusTtl { get; set; } = DefaultConsensusTtl;
|
||||
|
||||
public int ScanBatchSize { get; set; } = 250;
|
||||
|
||||
public VexStabilityDamperOptions Damper { get; } = new();
|
||||
}
|
||||
|
||||
public sealed class VexStabilityDamperOptions
|
||||
{
|
||||
private static readonly TimeSpan DefaultMinimum = TimeSpan.FromHours(24);
|
||||
private static readonly TimeSpan DefaultMaximum = TimeSpan.FromHours(48);
|
||||
private static readonly TimeSpan DefaultDurationBaseline = TimeSpan.FromHours(36);
|
||||
|
||||
public TimeSpan Minimum { get; set; } = DefaultMinimum;
|
||||
|
||||
public TimeSpan Maximum { get; set; } = DefaultMaximum;
|
||||
|
||||
public TimeSpan DefaultDuration { get; set; } = DefaultDurationBaseline;
|
||||
|
||||
public IList<VexStabilityDamperRule> Rules { get; } = new List<VexStabilityDamperRule>
|
||||
{
|
||||
new() { MinWeight = 0.9, Duration = TimeSpan.FromHours(24) },
|
||||
new() { MinWeight = 0.75, Duration = TimeSpan.FromHours(30) },
|
||||
new() { MinWeight = 0.5, Duration = TimeSpan.FromHours(36) },
|
||||
};
|
||||
|
||||
internal TimeSpan ClampDuration(TimeSpan duration)
|
||||
{
|
||||
if (duration < Minimum)
|
||||
{
|
||||
return Minimum;
|
||||
}
|
||||
|
||||
if (duration > Maximum)
|
||||
{
|
||||
return Maximum;
|
||||
}
|
||||
|
||||
return duration;
|
||||
}
|
||||
|
||||
public TimeSpan ResolveDuration(double weight)
|
||||
{
|
||||
if (double.IsNaN(weight) || double.IsInfinity(weight) || weight < 0)
|
||||
{
|
||||
return ClampDuration(DefaultDuration);
|
||||
}
|
||||
|
||||
if (Rules.Count == 0)
|
||||
{
|
||||
return ClampDuration(DefaultDuration);
|
||||
}
|
||||
|
||||
// Evaluate highest weight threshold first.
|
||||
TimeSpan? selected = null;
|
||||
foreach (var rule in Rules.OrderByDescending(static r => r.MinWeight))
|
||||
{
|
||||
if (weight >= rule.MinWeight)
|
||||
{
|
||||
selected = rule.Duration;
|
||||
break;
|
||||
}
|
||||
}
|
||||
|
||||
return ClampDuration(selected ?? DefaultDuration);
|
||||
}
|
||||
}
|
||||
|
||||
public sealed class VexStabilityDamperRule
|
||||
{
|
||||
public double MinWeight { get; set; }
|
||||
= 1.0;
|
||||
|
||||
public TimeSpan Duration { get; set; }
|
||||
= TimeSpan.FromHours(24);
|
||||
}
|
||||
@@ -0,0 +1,20 @@
|
||||
using System.ComponentModel.DataAnnotations;
|
||||
|
||||
namespace StellaOps.Excititor.Worker.Options;
|
||||
|
||||
public sealed class VexWorkerRetryOptions
|
||||
{
|
||||
[Range(1, int.MaxValue)]
|
||||
public int FailureThreshold { get; set; } = 3;
|
||||
|
||||
[Range(typeof(double), "0.0", "1.0")]
|
||||
public double JitterRatio { get; set; } = 0.2;
|
||||
|
||||
public TimeSpan BaseDelay { get; set; } = TimeSpan.FromMinutes(5);
|
||||
|
||||
public TimeSpan MaxDelay { get; set; } = TimeSpan.FromHours(6);
|
||||
|
||||
public TimeSpan QuarantineDuration { get; set; } = TimeSpan.FromHours(12);
|
||||
|
||||
public TimeSpan RetryCap { get; set; } = TimeSpan.FromHours(24);
|
||||
}
|
||||
@@ -0,0 +1,83 @@
|
||||
using Microsoft.Extensions.DependencyInjection;
|
||||
using Microsoft.Extensions.DependencyInjection.Extensions;
|
||||
using StellaOps.Concelier.Core.Orchestration;
|
||||
using StellaOps.Excititor.Worker.Scheduling;
|
||||
|
||||
namespace StellaOps.Excititor.Worker.Orchestration;
|
||||
|
||||
/// <summary>
|
||||
/// Service collection extensions for Excititor orchestrator integration.
|
||||
/// Per EXCITITOR-ORCH-32/33: Adopt orchestrator worker SDK.
|
||||
/// </summary>
|
||||
public static class ExcititorOrchestrationExtensions
|
||||
{
|
||||
/// <summary>
|
||||
/// Adds orchestrator-integrated VEX worker services.
|
||||
/// This wraps the existing provider runner with orchestrator SDK calls
|
||||
/// for heartbeats, progress, and pause/throttle handling.
|
||||
/// </summary>
|
||||
/// <param name="services">The service collection.</param>
|
||||
/// <returns>The service collection for chaining.</returns>
|
||||
public static IServiceCollection AddExcititorOrchestration(this IServiceCollection services)
|
||||
{
|
||||
// Add the Concelier orchestration services (registry, worker factory, backfill)
|
||||
services.AddConcelierOrchestrationServices();
|
||||
|
||||
// Register the orchestrator-aware provider runner as a decorator
|
||||
// This preserves the existing IVexProviderRunner implementation and wraps it
|
||||
services.Decorate<IVexProviderRunner, OrchestratorVexProviderRunner>();
|
||||
|
||||
return services;
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Extension methods for service decoration pattern.
|
||||
/// </summary>
|
||||
internal static class ServiceCollectionDecoratorExtensions
|
||||
{
|
||||
/// <summary>
|
||||
/// Decorates an existing service registration with a decorator implementation.
|
||||
/// </summary>
|
||||
/// <typeparam name="TService">The service interface type.</typeparam>
|
||||
/// <typeparam name="TDecorator">The decorator type that wraps TService.</typeparam>
|
||||
/// <param name="services">The service collection.</param>
|
||||
/// <returns>The service collection for chaining.</returns>
|
||||
public static IServiceCollection Decorate<TService, TDecorator>(this IServiceCollection services)
|
||||
where TService : class
|
||||
where TDecorator : class, TService
|
||||
{
|
||||
// Find the existing registration
|
||||
var existingDescriptor = services.FirstOrDefault(d => d.ServiceType == typeof(TService));
|
||||
if (existingDescriptor is null)
|
||||
{
|
||||
throw new InvalidOperationException(
|
||||
$"Cannot decorate service {typeof(TService).Name}: no existing registration found.");
|
||||
}
|
||||
|
||||
// Remove the original registration
|
||||
services.Remove(existingDescriptor);
|
||||
|
||||
// Create a factory that gets the original implementation and wraps it
|
||||
services.Add(ServiceDescriptor.Describe(
|
||||
typeof(TService),
|
||||
sp =>
|
||||
{
|
||||
// Resolve the original implementation
|
||||
var innerFactory = existingDescriptor.ImplementationFactory;
|
||||
var inner = innerFactory is not null
|
||||
? (TService)innerFactory(sp)
|
||||
: existingDescriptor.ImplementationType is not null
|
||||
? (TService)ActivatorUtilities.CreateInstance(sp, existingDescriptor.ImplementationType)
|
||||
: existingDescriptor.ImplementationInstance is not null
|
||||
? (TService)existingDescriptor.ImplementationInstance
|
||||
: throw new InvalidOperationException("Cannot resolve inner service.");
|
||||
|
||||
// Create the decorator with the inner instance
|
||||
return ActivatorUtilities.CreateInstance<TDecorator>(sp, inner);
|
||||
},
|
||||
existingDescriptor.Lifetime));
|
||||
|
||||
return services;
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,13 @@
|
||||
using System;
|
||||
|
||||
namespace StellaOps.Excititor.Worker.Orchestration;
|
||||
|
||||
internal interface IGuidGenerator
|
||||
{
|
||||
Guid NewGuid();
|
||||
}
|
||||
|
||||
internal sealed class DefaultGuidGenerator : IGuidGenerator
|
||||
{
|
||||
public Guid NewGuid() => Guid.NewGuid();
|
||||
}
|
||||
@@ -0,0 +1,140 @@
|
||||
using Microsoft.Extensions.Logging;
|
||||
using StellaOps.Concelier.Core.Orchestration;
|
||||
using StellaOps.Excititor.Worker.Scheduling;
|
||||
|
||||
namespace StellaOps.Excititor.Worker.Orchestration;
|
||||
|
||||
/// <summary>
|
||||
/// Orchestrator-integrated VEX provider runner.
|
||||
/// Per EXCITITOR-ORCH-32/33: Adopt orchestrator worker SDK; honor pause/throttle/retry with deterministic checkpoints.
|
||||
/// </summary>
|
||||
internal sealed class OrchestratorVexProviderRunner : IVexProviderRunner
|
||||
{
|
||||
private readonly IVexProviderRunner _inner;
|
||||
private readonly IConnectorWorkerFactory _workerFactory;
|
||||
private readonly ILogger<OrchestratorVexProviderRunner> _logger;
|
||||
private readonly TimeProvider _timeProvider;
|
||||
|
||||
public OrchestratorVexProviderRunner(
|
||||
IVexProviderRunner inner,
|
||||
IConnectorWorkerFactory workerFactory,
|
||||
ILogger<OrchestratorVexProviderRunner> logger,
|
||||
TimeProvider timeProvider)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(inner);
|
||||
ArgumentNullException.ThrowIfNull(workerFactory);
|
||||
ArgumentNullException.ThrowIfNull(logger);
|
||||
ArgumentNullException.ThrowIfNull(timeProvider);
|
||||
|
||||
_inner = inner;
|
||||
_workerFactory = workerFactory;
|
||||
_logger = logger;
|
||||
_timeProvider = timeProvider;
|
||||
}
|
||||
|
||||
public async ValueTask RunAsync(VexWorkerSchedule schedule, CancellationToken cancellationToken)
|
||||
{
|
||||
// Derive tenant from schedule (default to global tenant if not specified)
|
||||
var tenant = schedule.Tenant ?? "global";
|
||||
var connectorId = $"excititor-{schedule.ProviderId}".ToLowerInvariant();
|
||||
|
||||
var worker = _workerFactory.CreateWorker(tenant, connectorId);
|
||||
|
||||
try
|
||||
{
|
||||
// Start the orchestrator-tracked run
|
||||
await worker.StartRunAsync(cancellationToken).ConfigureAwait(false);
|
||||
|
||||
_logger.LogInformation(
|
||||
"Orchestrator run {RunId} started for VEX provider {ProviderId}",
|
||||
worker.RunId,
|
||||
schedule.ProviderId);
|
||||
|
||||
// Check for pause/throttle before starting actual work
|
||||
if (!await worker.CheckContinueAsync(cancellationToken).ConfigureAwait(false))
|
||||
{
|
||||
_logger.LogInformation(
|
||||
"Orchestrator run {RunId} paused before execution for {ProviderId}",
|
||||
worker.RunId,
|
||||
schedule.ProviderId);
|
||||
return;
|
||||
}
|
||||
|
||||
// Apply any active throttle
|
||||
var throttle = worker.GetActiveThrottle();
|
||||
if (throttle is not null)
|
||||
{
|
||||
_logger.LogInformation(
|
||||
"Applying throttle override for {ProviderId}: RPM={Rpm}",
|
||||
schedule.ProviderId,
|
||||
throttle.Rpm);
|
||||
}
|
||||
|
||||
// Report initial progress
|
||||
await worker.ReportProgressAsync(0, cancellationToken: cancellationToken).ConfigureAwait(false);
|
||||
|
||||
// Execute the actual provider run
|
||||
var startTime = _timeProvider.GetUtcNow();
|
||||
await _inner.RunAsync(schedule, cancellationToken).ConfigureAwait(false);
|
||||
var elapsed = _timeProvider.GetUtcNow() - startTime;
|
||||
|
||||
// Report completion
|
||||
await worker.ReportProgressAsync(100, cancellationToken: cancellationToken).ConfigureAwait(false);
|
||||
|
||||
await worker.CompleteSuccessAsync(cancellationToken).ConfigureAwait(false);
|
||||
|
||||
_logger.LogInformation(
|
||||
"Orchestrator run {RunId} completed successfully for {ProviderId} in {Duration}",
|
||||
worker.RunId,
|
||||
schedule.ProviderId,
|
||||
elapsed);
|
||||
}
|
||||
catch (OperationCanceledException) when (cancellationToken.IsCancellationRequested)
|
||||
{
|
||||
_logger.LogInformation(
|
||||
"Orchestrator run {RunId} cancelled for {ProviderId}",
|
||||
worker.RunId,
|
||||
schedule.ProviderId);
|
||||
throw;
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
_logger.LogError(
|
||||
ex,
|
||||
"Orchestrator run {RunId} failed for {ProviderId}: {Message}",
|
||||
worker.RunId,
|
||||
schedule.ProviderId,
|
||||
ex.Message);
|
||||
|
||||
// Report failure to orchestrator with retry suggestion
|
||||
await worker.CompleteFailureAsync(
|
||||
GetErrorCode(ex),
|
||||
GetRetryAfterSeconds(ex),
|
||||
cancellationToken).ConfigureAwait(false);
|
||||
|
||||
throw;
|
||||
}
|
||||
}
|
||||
|
||||
private static string GetErrorCode(Exception ex)
|
||||
{
|
||||
return ex switch
|
||||
{
|
||||
HttpRequestException => "HTTP_ERROR",
|
||||
TimeoutException => "TIMEOUT",
|
||||
InvalidOperationException => "INVALID_OPERATION",
|
||||
_ => "UNKNOWN_ERROR"
|
||||
};
|
||||
}
|
||||
|
||||
private static int? GetRetryAfterSeconds(Exception ex)
|
||||
{
|
||||
// Suggest retry delays based on error type
|
||||
return ex switch
|
||||
{
|
||||
HttpRequestException => 60, // Network issues: retry after 1 minute
|
||||
TimeoutException => 120, // Timeout: retry after 2 minutes
|
||||
_ => 300 // Unknown: retry after 5 minutes
|
||||
};
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,156 @@
|
||||
using StellaOps.Concelier.Core.Orchestration;
|
||||
|
||||
namespace StellaOps.Excititor.Worker.Orchestration;
|
||||
|
||||
/// <summary>
|
||||
/// Metadata for well-known VEX connectors.
|
||||
/// Per EXCITITOR-ORCH-32: Register VEX connectors with orchestrator.
|
||||
/// </summary>
|
||||
public static class VexConnectorMetadata
|
||||
{
|
||||
/// <summary>
|
||||
/// Red Hat CSAF connector metadata.
|
||||
/// </summary>
|
||||
public static ConnectorMetadata RedHatCsaf => new()
|
||||
{
|
||||
ConnectorId = "excititor-redhat-csaf",
|
||||
Source = "redhat-csaf",
|
||||
DisplayName = "Red Hat CSAF",
|
||||
Description = "Red Hat CSAF VEX documents",
|
||||
Capabilities = ["observations", "linksets"],
|
||||
ArtifactKinds = ["raw-vex", "normalized", "linkset"],
|
||||
DefaultCron = "0 */6 * * *", // Every 6 hours
|
||||
DefaultRpm = 60,
|
||||
EgressAllowlist = ["access.redhat.com", "www.redhat.com"]
|
||||
};
|
||||
|
||||
/// <summary>
|
||||
/// SUSE Rancher VEX Hub connector metadata.
|
||||
/// </summary>
|
||||
public static ConnectorMetadata SuseRancherVexHub => new()
|
||||
{
|
||||
ConnectorId = "excititor-suse-rancher",
|
||||
Source = "suse-rancher",
|
||||
DisplayName = "SUSE Rancher VEX Hub",
|
||||
Description = "SUSE Rancher VEX Hub documents",
|
||||
Capabilities = ["observations", "linksets", "attestations"],
|
||||
ArtifactKinds = ["raw-vex", "normalized", "linkset", "attestation"],
|
||||
DefaultCron = "0 */4 * * *", // Every 4 hours
|
||||
DefaultRpm = 100,
|
||||
EgressAllowlist = ["rancher.com", "suse.com"]
|
||||
};
|
||||
|
||||
/// <summary>
|
||||
/// Ubuntu CSAF connector metadata.
|
||||
/// </summary>
|
||||
public static ConnectorMetadata UbuntuCsaf => new()
|
||||
{
|
||||
ConnectorId = "excititor-ubuntu-csaf",
|
||||
Source = "ubuntu-csaf",
|
||||
DisplayName = "Ubuntu CSAF",
|
||||
Description = "Ubuntu CSAF VEX documents",
|
||||
Capabilities = ["observations", "linksets"],
|
||||
ArtifactKinds = ["raw-vex", "normalized", "linkset"],
|
||||
DefaultCron = "0 */6 * * *", // Every 6 hours
|
||||
DefaultRpm = 60,
|
||||
EgressAllowlist = ["ubuntu.com", "canonical.com"]
|
||||
};
|
||||
|
||||
/// <summary>
|
||||
/// Oracle CSAF connector metadata.
|
||||
/// </summary>
|
||||
public static ConnectorMetadata OracleCsaf => new()
|
||||
{
|
||||
ConnectorId = "excititor-oracle-csaf",
|
||||
Source = "oracle-csaf",
|
||||
DisplayName = "Oracle CSAF",
|
||||
Description = "Oracle CSAF VEX documents",
|
||||
Capabilities = ["observations", "linksets"],
|
||||
ArtifactKinds = ["raw-vex", "normalized", "linkset"],
|
||||
DefaultCron = "0 */12 * * *", // Every 12 hours
|
||||
DefaultRpm = 30,
|
||||
EgressAllowlist = ["oracle.com"]
|
||||
};
|
||||
|
||||
/// <summary>
|
||||
/// Cisco CSAF connector metadata.
|
||||
/// </summary>
|
||||
public static ConnectorMetadata CiscoCsaf => new()
|
||||
{
|
||||
ConnectorId = "excititor-cisco-csaf",
|
||||
Source = "cisco-csaf",
|
||||
DisplayName = "Cisco CSAF",
|
||||
Description = "Cisco CSAF VEX documents",
|
||||
Capabilities = ["observations", "linksets"],
|
||||
ArtifactKinds = ["raw-vex", "normalized", "linkset"],
|
||||
DefaultCron = "0 */6 * * *", // Every 6 hours
|
||||
DefaultRpm = 60,
|
||||
EgressAllowlist = ["cisco.com", "tools.cisco.com"]
|
||||
};
|
||||
|
||||
/// <summary>
|
||||
/// Microsoft MSRC CSAF connector metadata.
|
||||
/// </summary>
|
||||
public static ConnectorMetadata MsrcCsaf => new()
|
||||
{
|
||||
ConnectorId = "excititor-msrc-csaf",
|
||||
Source = "msrc-csaf",
|
||||
DisplayName = "Microsoft MSRC CSAF",
|
||||
Description = "Microsoft Security Response Center CSAF VEX documents",
|
||||
Capabilities = ["observations", "linksets"],
|
||||
ArtifactKinds = ["raw-vex", "normalized", "linkset"],
|
||||
DefaultCron = "0 */6 * * *", // Every 6 hours
|
||||
DefaultRpm = 30,
|
||||
EgressAllowlist = ["microsoft.com", "msrc.microsoft.com"]
|
||||
};
|
||||
|
||||
/// <summary>
|
||||
/// OCI OpenVEX Attestation connector metadata.
|
||||
/// </summary>
|
||||
public static ConnectorMetadata OciOpenVexAttestation => new()
|
||||
{
|
||||
ConnectorId = "excititor-oci-openvex",
|
||||
Source = "oci-openvex",
|
||||
DisplayName = "OCI OpenVEX Attestations",
|
||||
Description = "OpenVEX attestations from OCI registries",
|
||||
Capabilities = ["observations", "attestations"],
|
||||
ArtifactKinds = ["raw-vex", "attestation"],
|
||||
DefaultCron = "0 */2 * * *", // Every 2 hours (frequently updated)
|
||||
DefaultRpm = 100, // Higher rate for OCI registries
|
||||
EgressAllowlist = [] // Configured per-registry
|
||||
};
|
||||
|
||||
/// <summary>
|
||||
/// Gets metadata for all well-known VEX connectors.
|
||||
/// </summary>
|
||||
public static IReadOnlyList<ConnectorMetadata> All =>
|
||||
[
|
||||
RedHatCsaf,
|
||||
SuseRancherVexHub,
|
||||
UbuntuCsaf,
|
||||
OracleCsaf,
|
||||
CiscoCsaf,
|
||||
MsrcCsaf,
|
||||
OciOpenVexAttestation
|
||||
];
|
||||
|
||||
/// <summary>
|
||||
/// Gets connector metadata by provider ID.
|
||||
/// </summary>
|
||||
/// <param name="providerId">The provider identifier.</param>
|
||||
/// <returns>The connector metadata, or null if not found.</returns>
|
||||
public static ConnectorMetadata? GetByProviderId(string providerId)
|
||||
{
|
||||
return providerId.ToLowerInvariant() switch
|
||||
{
|
||||
"redhat" or "redhat-csaf" => RedHatCsaf,
|
||||
"suse" or "suse-rancher" or "rancher" => SuseRancherVexHub,
|
||||
"ubuntu" or "ubuntu-csaf" => UbuntuCsaf,
|
||||
"oracle" or "oracle-csaf" => OracleCsaf,
|
||||
"cisco" or "cisco-csaf" => CiscoCsaf,
|
||||
"msrc" or "msrc-csaf" or "microsoft" => MsrcCsaf,
|
||||
"oci" or "oci-openvex" or "openvex" => OciOpenVexAttestation,
|
||||
_ => null
|
||||
};
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,153 @@
|
||||
|
||||
using Microsoft.Extensions.Logging;
|
||||
using Microsoft.Extensions.Options;
|
||||
using StellaOps.Excititor.Core.Orchestration;
|
||||
using StellaOps.Excititor.Worker.Options;
|
||||
using System;
|
||||
using System.Threading;
|
||||
using System.Threading.Tasks;
|
||||
|
||||
namespace StellaOps.Excititor.Worker.Orchestration;
|
||||
|
||||
/// <summary>
|
||||
/// Background service that emits periodic heartbeats during job execution.
|
||||
/// </summary>
|
||||
internal sealed class VexWorkerHeartbeatService
|
||||
{
|
||||
private readonly IVexWorkerOrchestratorClient _orchestratorClient;
|
||||
private readonly IOptions<VexWorkerOrchestratorOptions> _options;
|
||||
private readonly TimeProvider _timeProvider;
|
||||
private readonly ILogger<VexWorkerHeartbeatService> _logger;
|
||||
|
||||
public VexWorkerHeartbeatService(
|
||||
IVexWorkerOrchestratorClient orchestratorClient,
|
||||
IOptions<VexWorkerOrchestratorOptions> options,
|
||||
TimeProvider timeProvider,
|
||||
ILogger<VexWorkerHeartbeatService> logger)
|
||||
{
|
||||
_orchestratorClient = orchestratorClient ?? throw new ArgumentNullException(nameof(orchestratorClient));
|
||||
_options = options ?? throw new ArgumentNullException(nameof(options));
|
||||
_timeProvider = timeProvider ?? throw new ArgumentNullException(nameof(timeProvider));
|
||||
_logger = logger ?? throw new ArgumentNullException(nameof(logger));
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Runs the heartbeat loop for the given job context.
|
||||
/// Call this in a background task while the job is running.
|
||||
/// </summary>
|
||||
public async Task RunAsync(
|
||||
VexWorkerJobContext context,
|
||||
Func<VexWorkerHeartbeatStatus> statusProvider,
|
||||
Func<int?> progressProvider,
|
||||
Func<string?> lastArtifactHashProvider,
|
||||
Func<string?> lastArtifactKindProvider,
|
||||
CancellationToken cancellationToken)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(context);
|
||||
ArgumentNullException.ThrowIfNull(statusProvider);
|
||||
|
||||
if (!_options.Value.Enabled)
|
||||
{
|
||||
_logger.LogDebug("Orchestrator heartbeat service disabled; skipping heartbeat loop.");
|
||||
return;
|
||||
}
|
||||
|
||||
var interval = ComputeInterval();
|
||||
_logger.LogDebug(
|
||||
"Starting heartbeat loop for job {RunId} with interval {Interval}",
|
||||
context.RunId,
|
||||
interval);
|
||||
|
||||
await Task.Yield();
|
||||
|
||||
try
|
||||
{
|
||||
using var timer = new PeriodicTimer(interval);
|
||||
|
||||
// Send initial heartbeat
|
||||
await SendHeartbeatAsync(
|
||||
context,
|
||||
statusProvider(),
|
||||
progressProvider?.Invoke(),
|
||||
lastArtifactHashProvider?.Invoke(),
|
||||
lastArtifactKindProvider?.Invoke(),
|
||||
cancellationToken).ConfigureAwait(false);
|
||||
|
||||
while (await timer.WaitForNextTickAsync(cancellationToken).ConfigureAwait(false))
|
||||
{
|
||||
if (cancellationToken.IsCancellationRequested)
|
||||
{
|
||||
break;
|
||||
}
|
||||
|
||||
await SendHeartbeatAsync(
|
||||
context,
|
||||
statusProvider(),
|
||||
progressProvider?.Invoke(),
|
||||
lastArtifactHashProvider?.Invoke(),
|
||||
lastArtifactKindProvider?.Invoke(),
|
||||
cancellationToken).ConfigureAwait(false);
|
||||
}
|
||||
}
|
||||
catch (OperationCanceledException) when (cancellationToken.IsCancellationRequested)
|
||||
{
|
||||
_logger.LogDebug("Heartbeat loop cancelled for job {RunId}", context.RunId);
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
_logger.LogWarning(
|
||||
ex,
|
||||
"Heartbeat loop error for job {RunId}: {Message}",
|
||||
context.RunId,
|
||||
ex.Message);
|
||||
}
|
||||
}
|
||||
|
||||
private async Task SendHeartbeatAsync(
|
||||
VexWorkerJobContext context,
|
||||
VexWorkerHeartbeatStatus status,
|
||||
int? progress,
|
||||
string? lastArtifactHash,
|
||||
string? lastArtifactKind,
|
||||
CancellationToken cancellationToken)
|
||||
{
|
||||
try
|
||||
{
|
||||
var heartbeat = new VexWorkerHeartbeat(
|
||||
status,
|
||||
progress,
|
||||
QueueDepth: null,
|
||||
lastArtifactHash,
|
||||
lastArtifactKind,
|
||||
ErrorCode: null,
|
||||
RetryAfterSeconds: null);
|
||||
|
||||
await _orchestratorClient.SendHeartbeatAsync(context, heartbeat, cancellationToken).ConfigureAwait(false);
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
_logger.LogWarning(
|
||||
ex,
|
||||
"Failed to send heartbeat for job {RunId}: {Message}",
|
||||
context.RunId,
|
||||
ex.Message);
|
||||
}
|
||||
}
|
||||
|
||||
private TimeSpan ComputeInterval()
|
||||
{
|
||||
var opts = _options.Value;
|
||||
var interval = opts.HeartbeatInterval;
|
||||
|
||||
if (interval < opts.MinHeartbeatInterval)
|
||||
{
|
||||
interval = opts.MinHeartbeatInterval;
|
||||
}
|
||||
else if (interval > opts.MaxHeartbeatInterval)
|
||||
{
|
||||
interval = opts.MaxHeartbeatInterval;
|
||||
}
|
||||
|
||||
return interval;
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,849 @@
|
||||
|
||||
using Microsoft.Extensions.Logging;
|
||||
using Microsoft.Extensions.Options;
|
||||
using StellaOps.Excititor.Core;
|
||||
using StellaOps.Excititor.Core.Orchestration;
|
||||
using StellaOps.Excititor.Core.Storage;
|
||||
using StellaOps.Excititor.Worker.Options;
|
||||
using System;
|
||||
using System.Collections.Immutable;
|
||||
using System.Globalization;
|
||||
using System.Net;
|
||||
using System.Net.Http;
|
||||
using System.Net.Http.Headers;
|
||||
using System.Text;
|
||||
using System.Text.Json;
|
||||
using System.Text.Json.Serialization;
|
||||
using System.Threading;
|
||||
using System.Threading.Tasks;
|
||||
|
||||
namespace StellaOps.Excititor.Worker.Orchestration;
|
||||
|
||||
/// <summary>
|
||||
/// Default implementation of <see cref="IVexWorkerOrchestratorClient"/>.
|
||||
/// Stores heartbeats and artifacts locally and, when configured, mirrors them to the Orchestrator worker API.
|
||||
/// Per EXCITITOR-ORCH-32/33: Uses append-only checkpoint store for deterministic persistence and replay.
|
||||
/// </summary>
|
||||
internal sealed class VexWorkerOrchestratorClient : IVexWorkerOrchestratorClient
|
||||
{
|
||||
private readonly IVexConnectorStateRepository _stateRepository;
|
||||
private readonly IAppendOnlyCheckpointStore? _checkpointStore;
|
||||
private readonly TimeProvider _timeProvider;
|
||||
private readonly IGuidGenerator _guidGenerator;
|
||||
private readonly IOptions<VexWorkerOrchestratorOptions> _options;
|
||||
private readonly ILogger<VexWorkerOrchestratorClient> _logger;
|
||||
private readonly HttpClient? _httpClient;
|
||||
private readonly JsonSerializerOptions _serializerOptions;
|
||||
private VexWorkerCommand? _pendingCommand;
|
||||
private long _commandSequence;
|
||||
|
||||
public VexWorkerOrchestratorClient(
|
||||
IVexConnectorStateRepository stateRepository,
|
||||
TimeProvider timeProvider,
|
||||
IGuidGenerator guidGenerator,
|
||||
IOptions<VexWorkerOrchestratorOptions> options,
|
||||
ILogger<VexWorkerOrchestratorClient> logger,
|
||||
HttpClient? httpClient = null,
|
||||
IAppendOnlyCheckpointStore? checkpointStore = null)
|
||||
{
|
||||
_stateRepository = stateRepository ?? throw new ArgumentNullException(nameof(stateRepository));
|
||||
_checkpointStore = checkpointStore;
|
||||
_timeProvider = timeProvider ?? throw new ArgumentNullException(nameof(timeProvider));
|
||||
_guidGenerator = guidGenerator ?? throw new ArgumentNullException(nameof(guidGenerator));
|
||||
_options = options ?? throw new ArgumentNullException(nameof(options));
|
||||
_logger = logger ?? throw new ArgumentNullException(nameof(logger));
|
||||
_httpClient = httpClient;
|
||||
_serializerOptions = new JsonSerializerOptions
|
||||
{
|
||||
PropertyNamingPolicy = JsonNamingPolicy.CamelCase,
|
||||
DefaultIgnoreCondition = JsonIgnoreCondition.WhenWritingNull
|
||||
};
|
||||
}
|
||||
|
||||
public async ValueTask<VexWorkerJobContext> StartJobAsync(
|
||||
string tenant,
|
||||
string connectorId,
|
||||
string? checkpoint,
|
||||
CancellationToken cancellationToken = default)
|
||||
{
|
||||
var startedAt = _timeProvider.GetUtcNow();
|
||||
var fallbackContext = new VexWorkerJobContext(tenant, connectorId, _guidGenerator.NewGuid(), checkpoint, startedAt);
|
||||
|
||||
if (!CanUseRemote())
|
||||
{
|
||||
return fallbackContext;
|
||||
}
|
||||
|
||||
var claimRequest = new ClaimRequest(
|
||||
WorkerId: _options.Value.WorkerId,
|
||||
TaskRunnerId: _options.Value.TaskRunnerId ?? Environment.MachineName,
|
||||
JobType: ResolveJobType(connectorId),
|
||||
LeaseSeconds: ResolveLeaseSeconds(),
|
||||
IdempotencyKey: $"exc-{connectorId}-{startedAt.ToUnixTimeSeconds()}");
|
||||
|
||||
var response = await PostAsync("api/v1/orchestrator/worker/claim", tenant, claimRequest, cancellationToken).ConfigureAwait(false);
|
||||
if (response is null)
|
||||
{
|
||||
return fallbackContext;
|
||||
}
|
||||
|
||||
if (response.StatusCode == HttpStatusCode.NoContent)
|
||||
{
|
||||
_logger.LogInformation("Orchestrator had no jobs for {ConnectorId}; continuing with local execution.", connectorId);
|
||||
return fallbackContext;
|
||||
}
|
||||
|
||||
if (!response.IsSuccessStatusCode)
|
||||
{
|
||||
await HandleErrorResponseAsync("claim", response, connectorId, cancellationToken).ConfigureAwait(false);
|
||||
return fallbackContext;
|
||||
}
|
||||
|
||||
var claim = await DeserializeAsync<ClaimResponse>(response, cancellationToken).ConfigureAwait(false);
|
||||
if (claim is null)
|
||||
{
|
||||
return fallbackContext;
|
||||
}
|
||||
|
||||
_logger.LogInformation(
|
||||
"Orchestrator job claimed: tenant={Tenant} connector={ConnectorId} jobId={JobId} leaseUntil={LeaseUntil:O}",
|
||||
tenant,
|
||||
connectorId,
|
||||
claim.JobId,
|
||||
claim.LeaseUntil);
|
||||
|
||||
return new VexWorkerJobContext(
|
||||
tenant,
|
||||
connectorId,
|
||||
claim.JobId,
|
||||
checkpoint,
|
||||
startedAt,
|
||||
orchestratorJobId: claim.JobId,
|
||||
orchestratorLeaseId: claim.LeaseId,
|
||||
leaseExpiresAt: claim.LeaseUntil,
|
||||
jobType: claim.JobType,
|
||||
correlationId: claim.CorrelationId,
|
||||
orchestratorRunId: claim.RunId);
|
||||
}
|
||||
|
||||
public async ValueTask SendHeartbeatAsync(
|
||||
VexWorkerJobContext context,
|
||||
VexWorkerHeartbeat heartbeat,
|
||||
CancellationToken cancellationToken = default)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(context);
|
||||
ArgumentNullException.ThrowIfNull(heartbeat);
|
||||
|
||||
var sequence = context.NextSequence();
|
||||
var timestamp = _timeProvider.GetUtcNow();
|
||||
|
||||
// Update state with heartbeat info
|
||||
var state = await _stateRepository.GetAsync(context.ConnectorId, cancellationToken).ConfigureAwait(false)
|
||||
?? new VexConnectorState(context.ConnectorId, null, ImmutableArray<string>.Empty);
|
||||
|
||||
var updated = state with
|
||||
{
|
||||
LastHeartbeatAt = timestamp,
|
||||
LastHeartbeatStatus = heartbeat.Status.ToString()
|
||||
};
|
||||
|
||||
await _stateRepository.SaveAsync(updated, cancellationToken).ConfigureAwait(false);
|
||||
|
||||
if (_options.Value.EnableVerboseLogging)
|
||||
{
|
||||
_logger.LogDebug(
|
||||
"Orchestrator heartbeat: runId={RunId} seq={Sequence} status={Status} progress={Progress} artifact={ArtifactHash}",
|
||||
context.RunId,
|
||||
sequence,
|
||||
heartbeat.Status,
|
||||
heartbeat.Progress,
|
||||
heartbeat.LastArtifactHash);
|
||||
}
|
||||
|
||||
// Log to append-only checkpoint store (EXCITITOR-ORCH-32/33)
|
||||
await LogCheckpointMutationAsync(
|
||||
context,
|
||||
CheckpointMutation.Heartbeat(
|
||||
context.RunId,
|
||||
timestamp,
|
||||
cursor: null,
|
||||
heartbeat.LastArtifactHash,
|
||||
heartbeat.LastArtifactKind,
|
||||
idempotencyKey: $"hb-{context.RunId}-{sequence}"),
|
||||
cancellationToken).ConfigureAwait(false);
|
||||
|
||||
await SendRemoteHeartbeatAsync(context, heartbeat, cancellationToken).ConfigureAwait(false);
|
||||
}
|
||||
|
||||
public async ValueTask RecordArtifactAsync(
|
||||
VexWorkerJobContext context,
|
||||
VexWorkerArtifact artifact,
|
||||
CancellationToken cancellationToken = default)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(context);
|
||||
ArgumentNullException.ThrowIfNull(artifact);
|
||||
|
||||
// Track artifact hash in connector state for determinism verification
|
||||
var state = await _stateRepository.GetAsync(context.ConnectorId, cancellationToken).ConfigureAwait(false)
|
||||
?? new VexConnectorState(context.ConnectorId, null, ImmutableArray<string>.Empty);
|
||||
|
||||
var digests = state.DocumentDigests.IsDefault
|
||||
? ImmutableArray<string>.Empty
|
||||
: state.DocumentDigests;
|
||||
|
||||
// Add artifact hash if not already tracked (cap to avoid unbounded growth)
|
||||
var maxDigests = Math.Max(1, _options.Value.MaxArtifactHashes);
|
||||
if (!digests.Contains(artifact.Hash))
|
||||
{
|
||||
digests = digests.Length >= maxDigests
|
||||
? digests.RemoveAt(0).Add(artifact.Hash)
|
||||
: digests.Add(artifact.Hash);
|
||||
}
|
||||
|
||||
var updated = state with
|
||||
{
|
||||
DocumentDigests = digests,
|
||||
LastArtifactHash = artifact.Hash,
|
||||
LastArtifactKind = artifact.Kind
|
||||
};
|
||||
|
||||
await _stateRepository.SaveAsync(updated, cancellationToken).ConfigureAwait(false);
|
||||
|
||||
_logger.LogDebug(
|
||||
"Orchestrator artifact recorded: runId={RunId} hash={Hash} kind={Kind} provider={Provider}",
|
||||
context.RunId,
|
||||
artifact.Hash,
|
||||
artifact.Kind,
|
||||
artifact.ProviderId);
|
||||
|
||||
// Log to append-only checkpoint store (EXCITITOR-ORCH-32/33)
|
||||
await LogCheckpointMutationAsync(
|
||||
context,
|
||||
CheckpointMutation.Artifact(
|
||||
context.RunId,
|
||||
artifact.CreatedAt,
|
||||
artifact.Hash,
|
||||
artifact.Kind,
|
||||
idempotencyKey: $"artifact-{artifact.Hash}"),
|
||||
cancellationToken).ConfigureAwait(false);
|
||||
|
||||
await SendRemoteProgressForArtifactAsync(context, artifact, cancellationToken).ConfigureAwait(false);
|
||||
}
|
||||
|
||||
public async ValueTask CompleteJobAsync(
|
||||
VexWorkerJobContext context,
|
||||
VexWorkerJobResult result,
|
||||
CancellationToken cancellationToken = default)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(context);
|
||||
ArgumentNullException.ThrowIfNull(result);
|
||||
|
||||
var state = await _stateRepository.GetAsync(context.ConnectorId, cancellationToken).ConfigureAwait(false)
|
||||
?? new VexConnectorState(context.ConnectorId, null, ImmutableArray<string>.Empty);
|
||||
|
||||
var updated = state with
|
||||
{
|
||||
LastUpdated = result.CompletedAt,
|
||||
LastSuccessAt = result.CompletedAt,
|
||||
LastHeartbeatAt = result.CompletedAt,
|
||||
LastHeartbeatStatus = VexWorkerHeartbeatStatus.Succeeded.ToString(),
|
||||
LastArtifactHash = result.LastArtifactHash,
|
||||
LastCheckpoint = ParseCheckpoint(result.LastCheckpoint),
|
||||
FailureCount = 0,
|
||||
NextEligibleRun = null,
|
||||
LastFailureReason = null
|
||||
};
|
||||
|
||||
await _stateRepository.SaveAsync(updated, cancellationToken).ConfigureAwait(false);
|
||||
|
||||
var duration = result.CompletedAt - context.StartedAt;
|
||||
_logger.LogInformation(
|
||||
"Orchestrator job completed: runId={RunId} connector={ConnectorId} documents={Documents} claims={Claims} duration={Duration}",
|
||||
context.RunId,
|
||||
context.ConnectorId,
|
||||
result.DocumentsProcessed,
|
||||
result.ClaimsGenerated,
|
||||
duration);
|
||||
|
||||
// Log to append-only checkpoint store (EXCITITOR-ORCH-32/33)
|
||||
await LogCheckpointMutationAsync(
|
||||
context,
|
||||
CheckpointMutation.Completed(
|
||||
context.RunId,
|
||||
result.CompletedAt,
|
||||
result.LastCheckpoint,
|
||||
result.DocumentsProcessed,
|
||||
result.ClaimsGenerated,
|
||||
result.LastArtifactHash,
|
||||
idempotencyKey: $"complete-{context.RunId}"),
|
||||
cancellationToken).ConfigureAwait(false);
|
||||
|
||||
await SendRemoteCompletionAsync(context, result, cancellationToken).ConfigureAwait(false);
|
||||
}
|
||||
|
||||
public async ValueTask FailJobAsync(
|
||||
VexWorkerJobContext context,
|
||||
string errorCode,
|
||||
string? errorMessage,
|
||||
int? retryAfterSeconds,
|
||||
CancellationToken cancellationToken = default)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(context);
|
||||
|
||||
var now = _timeProvider.GetUtcNow();
|
||||
var state = await _stateRepository.GetAsync(context.ConnectorId, cancellationToken).ConfigureAwait(false)
|
||||
?? new VexConnectorState(context.ConnectorId, null, ImmutableArray<string>.Empty);
|
||||
|
||||
var failureCount = state.FailureCount + 1;
|
||||
var nextEligible = retryAfterSeconds.HasValue
|
||||
? now.AddSeconds(retryAfterSeconds.Value)
|
||||
: (DateTimeOffset?)null;
|
||||
|
||||
var updated = state with
|
||||
{
|
||||
LastHeartbeatAt = now,
|
||||
LastHeartbeatStatus = VexWorkerHeartbeatStatus.Failed.ToString(),
|
||||
FailureCount = failureCount,
|
||||
NextEligibleRun = nextEligible,
|
||||
LastFailureReason = Truncate($"{errorCode}: {errorMessage}", 512)
|
||||
};
|
||||
|
||||
await _stateRepository.SaveAsync(updated, cancellationToken).ConfigureAwait(false);
|
||||
|
||||
_logger.LogWarning(
|
||||
"Orchestrator job failed: runId={RunId} connector={ConnectorId} error={ErrorCode} retryAfter={RetryAfter}s",
|
||||
context.RunId,
|
||||
context.ConnectorId,
|
||||
errorCode,
|
||||
retryAfterSeconds);
|
||||
|
||||
// Log to append-only checkpoint store (EXCITITOR-ORCH-32/33)
|
||||
await LogCheckpointMutationAsync(
|
||||
context,
|
||||
CheckpointMutation.Failed(
|
||||
context.RunId,
|
||||
now,
|
||||
errorCode,
|
||||
errorMessage,
|
||||
retryAfterSeconds,
|
||||
state.LastCheckpoint?.ToString("O", CultureInfo.InvariantCulture),
|
||||
idempotencyKey: $"fail-{context.RunId}"),
|
||||
cancellationToken).ConfigureAwait(false);
|
||||
|
||||
await SendRemoteCompletionAsync(
|
||||
context,
|
||||
new VexWorkerJobResult(0, 0, state.LastCheckpoint?.ToString("O", CultureInfo.InvariantCulture), state.LastArtifactHash, now),
|
||||
cancellationToken,
|
||||
success: false,
|
||||
failureReason: Truncate($"{errorCode}: {errorMessage}", 256)).ConfigureAwait(false);
|
||||
}
|
||||
|
||||
public ValueTask FailJobAsync(
|
||||
VexWorkerJobContext context,
|
||||
VexWorkerError error,
|
||||
CancellationToken cancellationToken = default)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(error);
|
||||
|
||||
_logger.LogDebug(
|
||||
"Orchestrator job failed with classified error: runId={RunId} code={Code} category={Category} retryable={Retryable}",
|
||||
context.RunId,
|
||||
error.Code,
|
||||
error.Category,
|
||||
error.Retryable);
|
||||
|
||||
return FailJobAsync(
|
||||
context,
|
||||
error.Code,
|
||||
error.Message,
|
||||
error.Retryable ? error.RetryAfterSeconds : null,
|
||||
cancellationToken);
|
||||
}
|
||||
|
||||
public ValueTask<VexWorkerCommand?> GetPendingCommandAsync(
|
||||
VexWorkerJobContext context,
|
||||
CancellationToken cancellationToken = default)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(context);
|
||||
|
||||
if (!_options.Value.Enabled)
|
||||
{
|
||||
return ValueTask.FromResult<VexWorkerCommand?>(null);
|
||||
}
|
||||
|
||||
var command = Interlocked.Exchange(ref _pendingCommand, null);
|
||||
return ValueTask.FromResult(command);
|
||||
}
|
||||
|
||||
public ValueTask AcknowledgeCommandAsync(
|
||||
VexWorkerJobContext context,
|
||||
long commandSequence,
|
||||
CancellationToken cancellationToken = default)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(context);
|
||||
|
||||
_logger.LogDebug(
|
||||
"Orchestrator command acknowledged: runId={RunId} sequence={Sequence}",
|
||||
context.RunId,
|
||||
commandSequence);
|
||||
|
||||
return ValueTask.CompletedTask;
|
||||
}
|
||||
|
||||
public async ValueTask SaveCheckpointAsync(
|
||||
VexWorkerJobContext context,
|
||||
VexWorkerCheckpoint checkpoint,
|
||||
CancellationToken cancellationToken = default)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(context);
|
||||
ArgumentNullException.ThrowIfNull(checkpoint);
|
||||
|
||||
var now = _timeProvider.GetUtcNow();
|
||||
var state = await _stateRepository.GetAsync(context.ConnectorId, cancellationToken).ConfigureAwait(false)
|
||||
?? new VexConnectorState(context.ConnectorId, null, ImmutableArray<string>.Empty);
|
||||
|
||||
var updated = state with
|
||||
{
|
||||
LastCheckpoint = ParseCheckpoint(checkpoint.Cursor),
|
||||
LastUpdated = checkpoint.LastProcessedAt ?? now,
|
||||
DocumentDigests = checkpoint.ProcessedDigests.IsDefault
|
||||
? ImmutableArray<string>.Empty
|
||||
: checkpoint.ProcessedDigests,
|
||||
ResumeTokens = checkpoint.ResumeTokens.IsEmpty
|
||||
? ImmutableDictionary<string, string>.Empty
|
||||
: checkpoint.ResumeTokens
|
||||
};
|
||||
|
||||
await _stateRepository.SaveAsync(updated, cancellationToken).ConfigureAwait(false);
|
||||
|
||||
_logger.LogDebug(
|
||||
"Orchestrator checkpoint saved: runId={RunId} connector={ConnectorId} cursor={Cursor} digests={DigestCount}",
|
||||
context.RunId,
|
||||
context.ConnectorId,
|
||||
checkpoint.Cursor ?? "(none)",
|
||||
checkpoint.ProcessedDigests.Length);
|
||||
|
||||
// Log to append-only checkpoint store (EXCITITOR-ORCH-32/33)
|
||||
if (!string.IsNullOrEmpty(checkpoint.Cursor))
|
||||
{
|
||||
await LogCheckpointMutationAsync(
|
||||
context,
|
||||
CheckpointMutation.CursorUpdate(
|
||||
context.RunId,
|
||||
checkpoint.LastProcessedAt ?? now,
|
||||
checkpoint.Cursor,
|
||||
checkpoint.ProcessedDigests.Length,
|
||||
idempotencyKey: $"cursor-{context.RunId}-{checkpoint.Cursor}"),
|
||||
cancellationToken).ConfigureAwait(false);
|
||||
}
|
||||
}
|
||||
|
||||
public async ValueTask<VexWorkerCheckpoint?> LoadCheckpointAsync(
|
||||
string connectorId,
|
||||
CancellationToken cancellationToken = default)
|
||||
{
|
||||
ArgumentException.ThrowIfNullOrWhiteSpace(connectorId);
|
||||
|
||||
var state = await _stateRepository.GetAsync(connectorId, cancellationToken).ConfigureAwait(false);
|
||||
if (state is null)
|
||||
{
|
||||
return null;
|
||||
}
|
||||
|
||||
return new VexWorkerCheckpoint(
|
||||
connectorId,
|
||||
state.LastCheckpoint?.ToString("O", CultureInfo.InvariantCulture),
|
||||
state.LastUpdated,
|
||||
state.DocumentDigests.IsDefault ? ImmutableArray<string>.Empty : state.DocumentDigests,
|
||||
state.ResumeTokens.IsEmpty ? ImmutableDictionary<string, string>.Empty : state.ResumeTokens);
|
||||
}
|
||||
|
||||
private bool CanUseRemote()
|
||||
{
|
||||
var opts = _options.Value;
|
||||
return opts.Enabled && _httpClient is not null && opts.BaseAddress is not null;
|
||||
}
|
||||
|
||||
private static string Truncate(string? value, int maxLength)
|
||||
{
|
||||
if (string.IsNullOrEmpty(value))
|
||||
{
|
||||
return string.Empty;
|
||||
}
|
||||
|
||||
return value.Length <= maxLength
|
||||
? value
|
||||
: value[..maxLength];
|
||||
}
|
||||
|
||||
private static DateTimeOffset? ParseCheckpoint(string? checkpoint)
|
||||
{
|
||||
if (string.IsNullOrEmpty(checkpoint))
|
||||
{
|
||||
return null;
|
||||
}
|
||||
|
||||
return DateTimeOffset.TryParse(checkpoint, CultureInfo.InvariantCulture, DateTimeStyles.RoundtripKind, out var parsed)
|
||||
? parsed
|
||||
: null;
|
||||
}
|
||||
|
||||
private int ResolveLeaseSeconds()
|
||||
{
|
||||
var seconds = (int)Math.Round(_options.Value.DefaultLeaseDuration.TotalSeconds);
|
||||
return Math.Clamp(seconds, 30, 3600);
|
||||
}
|
||||
|
||||
private int ResolveHeartbeatExtendSeconds()
|
||||
{
|
||||
var opts = _options.Value;
|
||||
var seconds = (int)Math.Round(opts.HeartbeatExtend.TotalSeconds);
|
||||
var min = (int)Math.Round(opts.MinHeartbeatInterval.TotalSeconds);
|
||||
var max = (int)Math.Round(opts.MaxHeartbeatInterval.TotalSeconds);
|
||||
return Math.Clamp(seconds, min, max);
|
||||
}
|
||||
|
||||
private string ResolveJobType(string connectorId)
|
||||
{
|
||||
return string.IsNullOrWhiteSpace(_options.Value.JobType)
|
||||
? $"exc-vex-{connectorId}"
|
||||
: _options.Value.JobType;
|
||||
}
|
||||
|
||||
private async ValueTask SendRemoteHeartbeatAsync(
|
||||
VexWorkerJobContext context,
|
||||
VexWorkerHeartbeat heartbeat,
|
||||
CancellationToken cancellationToken)
|
||||
{
|
||||
if (!CanUseRemote() || context.OrchestratorJobId is null || context.OrchestratorLeaseId is null)
|
||||
{
|
||||
return;
|
||||
}
|
||||
|
||||
var request = new HeartbeatRequest(
|
||||
context.OrchestratorLeaseId.Value,
|
||||
ResolveHeartbeatExtendSeconds(),
|
||||
IdempotencyKey: $"hb-{context.RunId}-{context.Sequence}");
|
||||
|
||||
var response = await PostAsync(
|
||||
$"api/v1/orchestrator/worker/jobs/{context.OrchestratorJobId}/heartbeat",
|
||||
context.Tenant,
|
||||
request,
|
||||
cancellationToken).ConfigureAwait(false);
|
||||
|
||||
if (response is null)
|
||||
{
|
||||
return;
|
||||
}
|
||||
|
||||
if (response.IsSuccessStatusCode)
|
||||
{
|
||||
var hb = await DeserializeAsync<HeartbeatResponse>(response, cancellationToken).ConfigureAwait(false);
|
||||
if (hb?.LeaseUntil is not null)
|
||||
{
|
||||
context.UpdateLease(hb.LeaseUntil);
|
||||
}
|
||||
return;
|
||||
}
|
||||
|
||||
await HandleErrorResponseAsync("heartbeat", response, context.ConnectorId, cancellationToken).ConfigureAwait(false);
|
||||
}
|
||||
|
||||
private async ValueTask SendRemoteProgressForArtifactAsync(
|
||||
VexWorkerJobContext context,
|
||||
VexWorkerArtifact artifact,
|
||||
CancellationToken cancellationToken)
|
||||
{
|
||||
if (!CanUseRemote() || !_options.Value.EmitProgressForArtifacts || context.OrchestratorJobId is null || context.OrchestratorLeaseId is null)
|
||||
{
|
||||
return;
|
||||
}
|
||||
|
||||
var metadata = Serialize(new
|
||||
{
|
||||
artifact.Hash,
|
||||
artifact.Kind,
|
||||
artifact.ProviderId,
|
||||
artifact.DocumentId,
|
||||
artifact.CreatedAt
|
||||
});
|
||||
|
||||
var request = new ProgressRequest(
|
||||
context.OrchestratorLeaseId.Value,
|
||||
ProgressPercent: null,
|
||||
Message: $"artifact:{artifact.Kind}",
|
||||
Metadata: metadata,
|
||||
IdempotencyKey: $"artifact-{artifact.Hash}");
|
||||
|
||||
var response = await PostAsync(
|
||||
$"api/v1/orchestrator/worker/jobs/{context.OrchestratorJobId}/progress",
|
||||
context.Tenant,
|
||||
request,
|
||||
cancellationToken).ConfigureAwait(false);
|
||||
|
||||
if (response is not null && !response.IsSuccessStatusCode)
|
||||
{
|
||||
await HandleErrorResponseAsync("progress", response, context.ConnectorId, cancellationToken).ConfigureAwait(false);
|
||||
}
|
||||
}
|
||||
|
||||
private async ValueTask SendRemoteCompletionAsync(
|
||||
VexWorkerJobContext context,
|
||||
VexWorkerJobResult result,
|
||||
CancellationToken cancellationToken,
|
||||
bool success = true,
|
||||
string? failureReason = null)
|
||||
{
|
||||
if (!CanUseRemote() || context.OrchestratorJobId is null || context.OrchestratorLeaseId is null)
|
||||
{
|
||||
return;
|
||||
}
|
||||
|
||||
var request = new CompleteRequest(
|
||||
context.OrchestratorLeaseId.Value,
|
||||
success,
|
||||
success ? null : failureReason,
|
||||
Artifacts: Array.Empty<ArtifactInput>(),
|
||||
ResultDigest: result.LastArtifactHash,
|
||||
IdempotencyKey: $"complete-{context.RunId}-{context.Sequence}");
|
||||
|
||||
var response = await PostAsync(
|
||||
$"api/v1/orchestrator/worker/jobs/{context.OrchestratorJobId}/complete",
|
||||
context.Tenant,
|
||||
request,
|
||||
cancellationToken).ConfigureAwait(false);
|
||||
|
||||
if (response is not null && !response.IsSuccessStatusCode)
|
||||
{
|
||||
await HandleErrorResponseAsync("complete", response, context.ConnectorId, cancellationToken).ConfigureAwait(false);
|
||||
}
|
||||
}
|
||||
|
||||
private async Task<HttpResponseMessage?> PostAsync<TPayload>(
|
||||
string path,
|
||||
string tenant,
|
||||
TPayload payload,
|
||||
CancellationToken cancellationToken)
|
||||
{
|
||||
if (!CanUseRemote())
|
||||
{
|
||||
return null;
|
||||
}
|
||||
|
||||
var request = new HttpRequestMessage(HttpMethod.Post, path)
|
||||
{
|
||||
Content = new StringContent(JsonSerializer.Serialize(payload, _serializerOptions), Encoding.UTF8, "application/json")
|
||||
};
|
||||
|
||||
request.Headers.Accept.Add(new MediaTypeWithQualityHeaderValue("application/json"));
|
||||
|
||||
var opts = _options.Value;
|
||||
request.Headers.TryAddWithoutValidation(string.IsNullOrWhiteSpace(opts.TenantHeader) ? "X-Tenant-Id" : opts.TenantHeader, tenant);
|
||||
|
||||
if (!string.IsNullOrWhiteSpace(opts.ApiKey))
|
||||
{
|
||||
request.Headers.TryAddWithoutValidation(string.IsNullOrWhiteSpace(opts.ApiKeyHeader) ? "X-Worker-Token" : opts.ApiKeyHeader, opts.ApiKey);
|
||||
}
|
||||
|
||||
if (!string.IsNullOrWhiteSpace(opts.BearerToken))
|
||||
{
|
||||
request.Headers.Authorization = new AuthenticationHeaderValue("Bearer", opts.BearerToken);
|
||||
}
|
||||
|
||||
try
|
||||
{
|
||||
return await _httpClient!.SendAsync(request, cancellationToken).ConfigureAwait(false);
|
||||
}
|
||||
catch (Exception ex) when (opts.AllowLocalFallback)
|
||||
{
|
||||
_logger.LogWarning(ex, "Failed to contact Orchestrator ({Path}); continuing locally.", path);
|
||||
StorePendingCommand(VexWorkerCommandKind.Retry, reason: "orchestrator_unreachable", retryAfterSeconds: 60);
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
private async ValueTask<T?> DeserializeAsync<T>(HttpResponseMessage response, CancellationToken cancellationToken)
|
||||
{
|
||||
var stream = await response.Content.ReadAsStreamAsync(cancellationToken).ConfigureAwait(false);
|
||||
return await JsonSerializer.DeserializeAsync<T>(stream, _serializerOptions, cancellationToken).ConfigureAwait(false);
|
||||
}
|
||||
|
||||
private async Task HandleErrorResponseAsync(
|
||||
string stage,
|
||||
HttpResponseMessage response,
|
||||
string connectorId,
|
||||
CancellationToken cancellationToken)
|
||||
{
|
||||
ErrorResponse? error = null;
|
||||
|
||||
try
|
||||
{
|
||||
error = await DeserializeAsync<ErrorResponse>(response, cancellationToken).ConfigureAwait(false);
|
||||
}
|
||||
catch
|
||||
{
|
||||
// ignore parse issues; fall back to status code handling
|
||||
}
|
||||
|
||||
var retryAfter = error?.RetryAfterSeconds;
|
||||
|
||||
switch (response.StatusCode)
|
||||
{
|
||||
case HttpStatusCode.TooManyRequests:
|
||||
StorePendingCommand(VexWorkerCommandKind.Throttle, reason: error?.Message ?? "rate_limited", retryAfterSeconds: retryAfter ?? 60);
|
||||
break;
|
||||
case HttpStatusCode.Conflict:
|
||||
StorePendingCommand(VexWorkerCommandKind.Retry, reason: error?.Message ?? "lease_conflict", retryAfterSeconds: retryAfter ?? 30);
|
||||
break;
|
||||
case HttpStatusCode.ServiceUnavailable:
|
||||
case HttpStatusCode.BadGateway:
|
||||
case HttpStatusCode.GatewayTimeout:
|
||||
StorePendingCommand(VexWorkerCommandKind.Pause, reason: error?.Message ?? "orchestrator_unavailable", retryAfterSeconds: retryAfter ?? 120);
|
||||
break;
|
||||
default:
|
||||
StorePendingCommand(VexWorkerCommandKind.Retry, reason: error?.Message ?? "orchestrator_error", retryAfterSeconds: retryAfter ?? 30);
|
||||
break;
|
||||
}
|
||||
|
||||
_logger.LogWarning(
|
||||
"Orchestrator {Stage} call failed for connector {ConnectorId}: {Status} {Error}",
|
||||
stage,
|
||||
connectorId,
|
||||
response.StatusCode,
|
||||
error?.Message ?? response.ReasonPhrase);
|
||||
}
|
||||
|
||||
private void StorePendingCommand(VexWorkerCommandKind kind, string? reason = null, int? retryAfterSeconds = null)
|
||||
{
|
||||
var issuedAt = _timeProvider.GetUtcNow();
|
||||
var sequence = Interlocked.Increment(ref _commandSequence);
|
||||
var expiresAt = retryAfterSeconds.HasValue ? issuedAt.AddSeconds(retryAfterSeconds.Value) : (DateTimeOffset?)null;
|
||||
|
||||
_pendingCommand = new VexWorkerCommand(
|
||||
kind,
|
||||
sequence,
|
||||
issuedAt,
|
||||
expiresAt,
|
||||
Throttle: kind == VexWorkerCommandKind.Throttle && retryAfterSeconds.HasValue
|
||||
? new VexWorkerThrottleParams(null, null, retryAfterSeconds)
|
||||
: null,
|
||||
Reason: reason);
|
||||
}
|
||||
|
||||
private string Serialize(object value) => JsonSerializer.Serialize(value, _serializerOptions);
|
||||
|
||||
/// <summary>
|
||||
/// Logs a checkpoint mutation to the append-only store for deterministic replay.
|
||||
/// Per EXCITITOR-ORCH-32/33: All checkpoint mutations are logged for audit/replay.
|
||||
/// </summary>
|
||||
private async ValueTask LogCheckpointMutationAsync(
|
||||
VexWorkerJobContext context,
|
||||
CheckpointMutation mutation,
|
||||
CancellationToken cancellationToken)
|
||||
{
|
||||
if (_checkpointStore is null)
|
||||
{
|
||||
return;
|
||||
}
|
||||
|
||||
try
|
||||
{
|
||||
var result = await _checkpointStore.AppendAsync(
|
||||
context.Tenant,
|
||||
context.ConnectorId,
|
||||
mutation,
|
||||
cancellationToken).ConfigureAwait(false);
|
||||
|
||||
if (_options.Value.EnableVerboseLogging)
|
||||
{
|
||||
_logger.LogDebug(
|
||||
"Checkpoint mutation logged: runId={RunId} type={Type} seq={Sequence} duplicate={IsDuplicate}",
|
||||
context.RunId,
|
||||
mutation.Type,
|
||||
result.SequenceNumber,
|
||||
result.WasDuplicate);
|
||||
}
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
_logger.LogWarning(ex,
|
||||
"Failed to log checkpoint mutation for connector {ConnectorId}: {Type}",
|
||||
context.ConnectorId,
|
||||
mutation.Type);
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Gets the append-only mutation log for a connector.
|
||||
/// Per EXCITITOR-ORCH-32/33: Enables deterministic replay.
|
||||
/// </summary>
|
||||
public async ValueTask<IReadOnlyList<CheckpointMutationEvent>> GetCheckpointMutationLogAsync(
|
||||
string tenant,
|
||||
string connectorId,
|
||||
long? sinceSequence = null,
|
||||
int limit = 100,
|
||||
CancellationToken cancellationToken = default)
|
||||
{
|
||||
if (_checkpointStore is null)
|
||||
{
|
||||
return Array.Empty<CheckpointMutationEvent>();
|
||||
}
|
||||
|
||||
return await _checkpointStore.GetMutationLogAsync(
|
||||
tenant,
|
||||
connectorId,
|
||||
sinceSequence,
|
||||
limit,
|
||||
cancellationToken).ConfigureAwait(false);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Replays checkpoint mutations to reconstruct state at a specific sequence.
|
||||
/// Per EXCITITOR-ORCH-32/33: Deterministic replay for audit/recovery.
|
||||
/// </summary>
|
||||
public async ValueTask<CheckpointState?> ReplayCheckpointToSequenceAsync(
|
||||
string tenant,
|
||||
string connectorId,
|
||||
long upToSequence,
|
||||
CancellationToken cancellationToken = default)
|
||||
{
|
||||
if (_checkpointStore is null)
|
||||
{
|
||||
return null;
|
||||
}
|
||||
|
||||
return await _checkpointStore.ReplayToSequenceAsync(
|
||||
tenant,
|
||||
connectorId,
|
||||
upToSequence,
|
||||
cancellationToken).ConfigureAwait(false);
|
||||
}
|
||||
|
||||
private sealed record ClaimRequest(string WorkerId, string? TaskRunnerId, string? JobType, int? LeaseSeconds, string? IdempotencyKey);
|
||||
|
||||
private sealed record ClaimResponse(
|
||||
Guid JobId,
|
||||
Guid LeaseId,
|
||||
string JobType,
|
||||
string Payload,
|
||||
string PayloadDigest,
|
||||
int Attempt,
|
||||
int MaxAttempts,
|
||||
DateTimeOffset LeaseUntil,
|
||||
string IdempotencyKey,
|
||||
string? CorrelationId,
|
||||
Guid? RunId,
|
||||
string? ProjectId);
|
||||
|
||||
private sealed record HeartbeatRequest(Guid LeaseId, int? ExtendSeconds, string? IdempotencyKey);
|
||||
|
||||
private sealed record HeartbeatResponse(Guid JobId, Guid LeaseId, DateTimeOffset LeaseUntil, bool Acknowledged);
|
||||
|
||||
private sealed record ProgressRequest(Guid LeaseId, double? ProgressPercent, string? Message, string? Metadata, string? IdempotencyKey);
|
||||
|
||||
private sealed record CompleteRequest(Guid LeaseId, bool Success, string? Reason, IReadOnlyList<ArtifactInput>? Artifacts, string? ResultDigest, string? IdempotencyKey);
|
||||
|
||||
private sealed record ArtifactInput(string ArtifactType, string Uri, string Digest, string? MimeType, long? SizeBytes, string? Metadata);
|
||||
|
||||
private sealed record ErrorResponse(string Error, string Message, Guid? JobId, int? RetryAfterSeconds);
|
||||
}
|
||||
138
src/Concelier/StellaOps.Excititor.Worker/Program.cs
Normal file
138
src/Concelier/StellaOps.Excititor.Worker/Program.cs
Normal file
@@ -0,0 +1,138 @@
|
||||
|
||||
using Microsoft.Extensions.DependencyInjection;
|
||||
using Microsoft.Extensions.DependencyInjection.Extensions;
|
||||
using Microsoft.Extensions.Hosting;
|
||||
using Microsoft.Extensions.Logging;
|
||||
using Microsoft.Extensions.Options;
|
||||
using StellaOps.Excititor.Attestation.Extensions;
|
||||
using StellaOps.Excititor.Attestation.Verification;
|
||||
using StellaOps.Excititor.Connectors.Abstractions;
|
||||
using StellaOps.Excititor.Core;
|
||||
using StellaOps.Excititor.Core.Aoc;
|
||||
using StellaOps.Excititor.Core.Orchestration;
|
||||
using StellaOps.Excititor.Core.Storage;
|
||||
using StellaOps.Excititor.Formats.CSAF;
|
||||
using StellaOps.Excititor.Formats.CycloneDX;
|
||||
using StellaOps.Excititor.Formats.OpenVEX;
|
||||
using StellaOps.Excititor.Persistence.Extensions;
|
||||
using StellaOps.Excititor.Worker.Auth;
|
||||
using StellaOps.Excititor.Worker.Options;
|
||||
using StellaOps.Excititor.Worker.Orchestration;
|
||||
using StellaOps.Excititor.Worker.Plugins;
|
||||
using StellaOps.Excititor.Worker.Scheduling;
|
||||
using StellaOps.Excititor.Worker.Signature;
|
||||
using StellaOps.IssuerDirectory.Client;
|
||||
using StellaOps.Plugin;
|
||||
using StellaOps.Worker.Health;
|
||||
using System.IO;
|
||||
using System.Linq;
|
||||
|
||||
var builder = WebApplication.CreateSlimBuilder(args);
|
||||
var services = builder.Services;
|
||||
var configuration = builder.Configuration;
|
||||
var workerConfig = configuration.GetSection("Excititor:Worker");
|
||||
var workerConfigSnapshot = workerConfig.Get<VexWorkerOptions>() ?? new VexWorkerOptions();
|
||||
services.AddOptions<VexWorkerOptions>()
|
||||
.Bind(workerConfig)
|
||||
.ValidateOnStart();
|
||||
|
||||
services.Configure<VexWorkerPluginOptions>(configuration.GetSection("Excititor:Worker:Plugins"));
|
||||
services.Configure<TenantAuthorityOptions>(configuration.GetSection("Excititor:Authority"));
|
||||
services.AddSingleton<IValidateOptions<TenantAuthorityOptions>, TenantAuthorityOptionsValidator>();
|
||||
services.PostConfigure<VexWorkerOptions>(options =>
|
||||
{
|
||||
if (options.DisableConsensus)
|
||||
{
|
||||
options.Refresh.Enabled = false;
|
||||
}
|
||||
});
|
||||
// VEX connectors are loaded via plugin catalog below
|
||||
// Direct connector registration removed in favor of plugin-based loading
|
||||
|
||||
services.AddOptions<VexStorageOptions>()
|
||||
.Bind(configuration.GetSection("Excititor:Storage"))
|
||||
.ValidateOnStart();
|
||||
|
||||
services.AddExcititorPersistence(configuration);
|
||||
services.TryAddSingleton<IVexProviderStore, InMemoryVexProviderStore>();
|
||||
services.TryAddScoped<IVexConnectorStateRepository, InMemoryVexConnectorStateRepository>();
|
||||
services.TryAddSingleton<IVexClaimStore, InMemoryVexClaimStore>();
|
||||
services.AddCsafNormalizer();
|
||||
services.AddCycloneDxNormalizer();
|
||||
services.AddOpenVexNormalizer();
|
||||
services.AddSingleton<IVexSignatureVerifier, WorkerSignatureVerifier>();
|
||||
services.AddVexAttestation();
|
||||
services.Configure<VexAttestationVerificationOptions>(configuration.GetSection("Excititor:Attestation:Verification"));
|
||||
var issuerDirectorySection = configuration.GetSection("Excititor:IssuerDirectory");
|
||||
if (issuerDirectorySection.Exists())
|
||||
{
|
||||
services.AddIssuerDirectoryClient(issuerDirectorySection);
|
||||
}
|
||||
else
|
||||
{
|
||||
services.AddIssuerDirectoryClient(configuration);
|
||||
}
|
||||
services.PostConfigure<VexAttestationVerificationOptions>(options =>
|
||||
{
|
||||
// Workers operate in offline-first environments; allow verification to succeed without Rekor.
|
||||
options.AllowOfflineTransparency = true;
|
||||
if (!configuration.GetSection("Excititor:Attestation:Verification").Exists())
|
||||
{
|
||||
options.RequireTransparencyLog = false;
|
||||
}
|
||||
});
|
||||
services.AddExcititorAocGuards();
|
||||
|
||||
services.AddSingleton<IValidateOptions<VexWorkerOptions>, VexWorkerOptionsValidator>();
|
||||
services.AddSingleton(TimeProvider.System);
|
||||
services.TryAddSingleton<IGuidGenerator, DefaultGuidGenerator>();
|
||||
services.PostConfigure<VexWorkerOptions>(options =>
|
||||
{
|
||||
if (!options.Providers.Any(provider => string.Equals(provider.ProviderId, "excititor:redhat", StringComparison.OrdinalIgnoreCase)))
|
||||
{
|
||||
options.Providers.Add(new VexWorkerProviderOptions
|
||||
{
|
||||
ProviderId = "excititor:redhat",
|
||||
});
|
||||
}
|
||||
});
|
||||
// Load VEX connector plugins
|
||||
services.AddSingleton<VexWorkerPluginCatalogLoader>();
|
||||
services.AddSingleton<PluginCatalog>(provider =>
|
||||
provider.GetRequiredService<VexWorkerPluginCatalogLoader>().Load().Catalog);
|
||||
|
||||
// Orchestrator worker SDK integration
|
||||
services.AddOptions<VexWorkerOrchestratorOptions>()
|
||||
.Bind(configuration.GetSection("Excititor:Worker:Orchestrator"))
|
||||
.ValidateOnStart();
|
||||
services.AddHttpClient(TenantAuthorityClientFactory.AuthorityClientName);
|
||||
services.AddHttpClient<IVexWorkerOrchestratorClient, VexWorkerOrchestratorClient>((provider, client) =>
|
||||
{
|
||||
var opts = provider.GetRequiredService<IOptions<VexWorkerOrchestratorOptions>>().Value;
|
||||
if (opts.BaseAddress is not null)
|
||||
{
|
||||
client.BaseAddress = opts.BaseAddress;
|
||||
}
|
||||
|
||||
client.Timeout = opts.RequestTimeout;
|
||||
});
|
||||
services.AddSingleton<VexWorkerHeartbeatService>();
|
||||
|
||||
services.AddSingleton<IVexProviderRunner, DefaultVexProviderRunner>();
|
||||
services.AddHostedService<VexWorkerHostedService>();
|
||||
if (!workerConfigSnapshot.DisableConsensus)
|
||||
{
|
||||
services.AddSingleton<VexConsensusRefreshService>();
|
||||
services.AddSingleton<IVexConsensusRefreshScheduler>(static provider => provider.GetRequiredService<VexConsensusRefreshService>());
|
||||
services.AddHostedService(static provider => provider.GetRequiredService<VexConsensusRefreshService>());
|
||||
}
|
||||
services.AddSingleton<ITenantAuthorityClientFactory, TenantAuthorityClientFactory>();
|
||||
|
||||
builder.Services.AddWorkerHealthChecks();
|
||||
|
||||
var app = builder.Build();
|
||||
app.MapWorkerHealthEndpoints();
|
||||
await app.RunAsync();
|
||||
|
||||
// Make Program class file-scoped to prevent it from being exposed to referencing assemblies
|
||||
file sealed partial class Program;
|
||||
@@ -0,0 +1,3 @@
|
||||
using System.Runtime.CompilerServices;
|
||||
|
||||
[assembly: InternalsVisibleTo("StellaOps.Excititor.Worker.Tests")]
|
||||
@@ -0,0 +1,398 @@
|
||||
|
||||
using Microsoft.Extensions.DependencyInjection;
|
||||
using Microsoft.Extensions.Logging;
|
||||
using Microsoft.Extensions.Options;
|
||||
using StellaOps.Excititor.Connectors.Abstractions;
|
||||
using StellaOps.Excititor.Core;
|
||||
using StellaOps.Excititor.Core.Orchestration;
|
||||
using StellaOps.Excititor.Core.Storage;
|
||||
using StellaOps.Excititor.Worker.Options;
|
||||
using StellaOps.Excititor.Worker.Orchestration;
|
||||
using StellaOps.Excititor.Worker.Signature;
|
||||
using StellaOps.Plugin;
|
||||
using System;
|
||||
using System.Buffers.Binary;
|
||||
using System.Collections.Immutable;
|
||||
using System.Globalization;
|
||||
using System.Linq;
|
||||
using System.Security.Cryptography;
|
||||
using System.Text;
|
||||
|
||||
namespace StellaOps.Excititor.Worker.Scheduling;
|
||||
|
||||
internal sealed class DefaultVexProviderRunner : IVexProviderRunner
|
||||
{
|
||||
private readonly IServiceProvider _serviceProvider;
|
||||
private readonly PluginCatalog _pluginCatalog;
|
||||
private readonly IVexWorkerOrchestratorClient _orchestratorClient;
|
||||
private readonly VexWorkerHeartbeatService _heartbeatService;
|
||||
private readonly ILogger<DefaultVexProviderRunner> _logger;
|
||||
private readonly TimeProvider _timeProvider;
|
||||
private readonly VexWorkerRetryOptions _retryOptions;
|
||||
private readonly VexWorkerOrchestratorOptions _orchestratorOptions;
|
||||
|
||||
public DefaultVexProviderRunner(
|
||||
IServiceProvider serviceProvider,
|
||||
PluginCatalog pluginCatalog,
|
||||
IVexWorkerOrchestratorClient orchestratorClient,
|
||||
VexWorkerHeartbeatService heartbeatService,
|
||||
ILogger<DefaultVexProviderRunner> logger,
|
||||
TimeProvider timeProvider,
|
||||
IOptions<VexWorkerOptions> workerOptions,
|
||||
IOptions<VexWorkerOrchestratorOptions> orchestratorOptions)
|
||||
{
|
||||
_serviceProvider = serviceProvider ?? throw new ArgumentNullException(nameof(serviceProvider));
|
||||
_pluginCatalog = pluginCatalog ?? throw new ArgumentNullException(nameof(pluginCatalog));
|
||||
_orchestratorClient = orchestratorClient ?? throw new ArgumentNullException(nameof(orchestratorClient));
|
||||
_heartbeatService = heartbeatService ?? throw new ArgumentNullException(nameof(heartbeatService));
|
||||
_logger = logger ?? throw new ArgumentNullException(nameof(logger));
|
||||
_timeProvider = timeProvider ?? throw new ArgumentNullException(nameof(timeProvider));
|
||||
if (workerOptions is null)
|
||||
{
|
||||
throw new ArgumentNullException(nameof(workerOptions));
|
||||
}
|
||||
|
||||
_retryOptions = workerOptions.Value?.Retry ?? throw new InvalidOperationException("VexWorkerOptions.Retry must be configured.");
|
||||
_orchestratorOptions = orchestratorOptions?.Value ?? new VexWorkerOrchestratorOptions();
|
||||
}
|
||||
|
||||
public async ValueTask RunAsync(VexWorkerSchedule schedule, CancellationToken cancellationToken)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(schedule);
|
||||
ArgumentException.ThrowIfNullOrWhiteSpace(schedule.ProviderId);
|
||||
|
||||
using var scope = _serviceProvider.CreateScope();
|
||||
var availablePlugins = _pluginCatalog.GetAvailableConnectorPlugins(scope.ServiceProvider);
|
||||
var matched = availablePlugins.FirstOrDefault(plugin =>
|
||||
string.Equals(plugin.Name, schedule.ProviderId, StringComparison.OrdinalIgnoreCase));
|
||||
|
||||
if (matched is not null)
|
||||
{
|
||||
_logger.LogInformation(
|
||||
"Connector plugin {PluginName} ({ProviderId}) is available. Execution hooks will be added in subsequent tasks.",
|
||||
matched.Name,
|
||||
schedule.ProviderId);
|
||||
}
|
||||
else
|
||||
{
|
||||
_logger.LogInformation("No legacy connector plugin registered for provider {ProviderId}; falling back to DI-managed connectors.", schedule.ProviderId);
|
||||
}
|
||||
|
||||
var connectors = scope.ServiceProvider.GetServices<IVexConnector>();
|
||||
var connector = connectors.FirstOrDefault(c => string.Equals(c.Id, schedule.ProviderId, StringComparison.OrdinalIgnoreCase));
|
||||
|
||||
if (connector is null)
|
||||
{
|
||||
_logger.LogWarning("No IVexConnector implementation registered for provider {ProviderId}; skipping run.", schedule.ProviderId);
|
||||
return;
|
||||
}
|
||||
|
||||
await ExecuteConnectorAsync(scope.ServiceProvider, connector, schedule.Settings, cancellationToken).ConfigureAwait(false);
|
||||
}
|
||||
|
||||
private async Task ExecuteConnectorAsync(IServiceProvider scopeProvider, IVexConnector connector, VexConnectorSettings settings, CancellationToken cancellationToken)
|
||||
{
|
||||
var effectiveSettings = settings ?? VexConnectorSettings.Empty;
|
||||
var rawStore = scopeProvider.GetRequiredService<IVexRawStore>();
|
||||
var providerStore = scopeProvider.GetRequiredService<IVexProviderStore>();
|
||||
var stateRepository = scopeProvider.GetRequiredService<IVexConnectorStateRepository>();
|
||||
var normalizerRouter = scopeProvider.GetRequiredService<IVexNormalizerRouter>();
|
||||
var signatureVerifier = scopeProvider.GetRequiredService<IVexSignatureVerifier>();
|
||||
|
||||
var descriptor = connector switch
|
||||
{
|
||||
VexConnectorBase baseConnector => baseConnector.Descriptor,
|
||||
_ => new VexConnectorDescriptor(connector.Id, VexProviderKind.Vendor, connector.Id)
|
||||
};
|
||||
|
||||
var provider = await providerStore.FindAsync(descriptor.Id, cancellationToken).ConfigureAwait(false)
|
||||
?? new VexProvider(descriptor.Id, descriptor.DisplayName, descriptor.Kind);
|
||||
|
||||
await providerStore.SaveAsync(provider, cancellationToken).ConfigureAwait(false);
|
||||
|
||||
var stateBeforeRun = await stateRepository.GetAsync(descriptor.Id, cancellationToken).ConfigureAwait(false);
|
||||
var now = _timeProvider.GetUtcNow();
|
||||
|
||||
if (stateBeforeRun?.NextEligibleRun is { } nextEligible && nextEligible > now)
|
||||
{
|
||||
_logger.LogInformation(
|
||||
"Connector {ConnectorId} is in backoff until {NextEligible:O}; skipping run.",
|
||||
connector.Id,
|
||||
nextEligible);
|
||||
return;
|
||||
}
|
||||
|
||||
await connector.ValidateAsync(effectiveSettings, cancellationToken).ConfigureAwait(false);
|
||||
|
||||
var verifyingSink = new VerifyingVexRawDocumentSink(rawStore, signatureVerifier);
|
||||
|
||||
var connectorContext = new VexConnectorContext(
|
||||
Since: stateBeforeRun?.LastUpdated,
|
||||
Settings: effectiveSettings,
|
||||
RawSink: verifyingSink,
|
||||
SignatureVerifier: signatureVerifier,
|
||||
Normalizers: normalizerRouter,
|
||||
Services: scopeProvider,
|
||||
ResumeTokens: stateBeforeRun?.ResumeTokens ?? ImmutableDictionary<string, string>.Empty);
|
||||
|
||||
// Start orchestrator job for heartbeat/progress tracking
|
||||
var jobContext = await _orchestratorClient.StartJobAsync(
|
||||
_orchestratorOptions.DefaultTenant,
|
||||
connector.Id,
|
||||
stateBeforeRun?.LastCheckpoint?.ToString("O", CultureInfo.InvariantCulture),
|
||||
cancellationToken).ConfigureAwait(false);
|
||||
|
||||
var documentCount = 0;
|
||||
string? lastArtifactHash = null;
|
||||
string? lastArtifactKind = null;
|
||||
var currentStatus = VexWorkerHeartbeatStatus.Running;
|
||||
|
||||
// Start heartbeat loop in background
|
||||
using var heartbeatCts = CancellationTokenSource.CreateLinkedTokenSource(cancellationToken);
|
||||
var heartbeatTask = _heartbeatService.RunAsync(
|
||||
jobContext,
|
||||
() => currentStatus,
|
||||
() => null, // Progress not tracked at document level
|
||||
() => lastArtifactHash,
|
||||
() => lastArtifactKind,
|
||||
heartbeatCts.Token);
|
||||
|
||||
try
|
||||
{
|
||||
await foreach (var document in connector.FetchAsync(connectorContext, cancellationToken).ConfigureAwait(false))
|
||||
{
|
||||
documentCount++;
|
||||
lastArtifactHash = document.Digest;
|
||||
lastArtifactKind = "vex-raw-document";
|
||||
|
||||
// Record artifact for determinism tracking
|
||||
if (_orchestratorOptions.Enabled)
|
||||
{
|
||||
var artifact = new VexWorkerArtifact(
|
||||
document.Digest,
|
||||
"vex-raw-document",
|
||||
connector.Id,
|
||||
document.Digest,
|
||||
_timeProvider.GetUtcNow());
|
||||
|
||||
await _orchestratorClient.RecordArtifactAsync(jobContext, artifact, cancellationToken).ConfigureAwait(false);
|
||||
}
|
||||
}
|
||||
|
||||
// Stop heartbeat loop
|
||||
currentStatus = VexWorkerHeartbeatStatus.Succeeded;
|
||||
await heartbeatCts.CancelAsync().ConfigureAwait(false);
|
||||
await SafeWaitForTaskAsync(heartbeatTask).ConfigureAwait(false);
|
||||
|
||||
_logger.LogInformation(
|
||||
"Connector {ConnectorId} persisted {DocumentCount} raw document(s) this run.",
|
||||
connector.Id,
|
||||
documentCount);
|
||||
|
||||
// Complete orchestrator job
|
||||
var completedAt = _timeProvider.GetUtcNow();
|
||||
var result = new VexWorkerJobResult(
|
||||
documentCount,
|
||||
ClaimsGenerated: 0, // Claims generated in separate normalization pass
|
||||
lastArtifactHash,
|
||||
lastArtifactHash,
|
||||
completedAt);
|
||||
|
||||
await _orchestratorClient.CompleteJobAsync(jobContext, result, cancellationToken).ConfigureAwait(false);
|
||||
|
||||
await UpdateSuccessStateAsync(stateRepository, descriptor.Id, completedAt, cancellationToken).ConfigureAwait(false);
|
||||
}
|
||||
catch (OperationCanceledException) when (cancellationToken.IsCancellationRequested)
|
||||
{
|
||||
currentStatus = VexWorkerHeartbeatStatus.Failed;
|
||||
await heartbeatCts.CancelAsync().ConfigureAwait(false);
|
||||
await SafeWaitForTaskAsync(heartbeatTask).ConfigureAwait(false);
|
||||
|
||||
var error = VexWorkerError.Cancelled("Operation cancelled by host");
|
||||
try
|
||||
{
|
||||
await _orchestratorClient.FailJobAsync(jobContext, error, cancellationToken).ConfigureAwait(false);
|
||||
}
|
||||
catch (OperationCanceledException) when (cancellationToken.IsCancellationRequested)
|
||||
{
|
||||
}
|
||||
|
||||
throw;
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
currentStatus = VexWorkerHeartbeatStatus.Failed;
|
||||
await heartbeatCts.CancelAsync().ConfigureAwait(false);
|
||||
await SafeWaitForTaskAsync(heartbeatTask).ConfigureAwait(false);
|
||||
|
||||
// Classify the error for appropriate retry handling
|
||||
var classifiedError = VexWorkerError.FromException(ex, stage: "fetch");
|
||||
|
||||
// Apply backoff delay for retryable errors
|
||||
var retryDelay = classifiedError.Retryable
|
||||
? (int)CalculateDelayWithJitter(connector.Id, (stateBeforeRun?.FailureCount ?? 0) + 1).TotalSeconds
|
||||
: (int?)null;
|
||||
|
||||
var errorWithRetry = classifiedError.Retryable && retryDelay.HasValue
|
||||
? new VexWorkerError(
|
||||
classifiedError.Code,
|
||||
classifiedError.Category,
|
||||
classifiedError.Message,
|
||||
classifiedError.Retryable,
|
||||
retryDelay,
|
||||
classifiedError.Stage,
|
||||
classifiedError.Details)
|
||||
: classifiedError;
|
||||
|
||||
try
|
||||
{
|
||||
await _orchestratorClient.FailJobAsync(jobContext, errorWithRetry, cancellationToken).ConfigureAwait(false);
|
||||
}
|
||||
catch (OperationCanceledException) when (cancellationToken.IsCancellationRequested)
|
||||
{
|
||||
}
|
||||
|
||||
await UpdateFailureStateAsync(stateRepository, descriptor.Id, _timeProvider.GetUtcNow(), ex, classifiedError.Retryable, cancellationToken).ConfigureAwait(false);
|
||||
throw;
|
||||
}
|
||||
}
|
||||
|
||||
private static async Task SafeWaitForTaskAsync(Task task)
|
||||
{
|
||||
try
|
||||
{
|
||||
await task.ConfigureAwait(false);
|
||||
}
|
||||
catch (OperationCanceledException)
|
||||
{
|
||||
// Expected when cancellation is requested
|
||||
}
|
||||
}
|
||||
|
||||
private async Task UpdateSuccessStateAsync(
|
||||
IVexConnectorStateRepository stateRepository,
|
||||
string connectorId,
|
||||
DateTimeOffset completedAt,
|
||||
CancellationToken cancellationToken)
|
||||
{
|
||||
var current = await stateRepository.GetAsync(connectorId, cancellationToken).ConfigureAwait(false)
|
||||
?? new VexConnectorState(connectorId, null, ImmutableArray<string>.Empty);
|
||||
|
||||
var updated = current with
|
||||
{
|
||||
LastSuccessAt = completedAt,
|
||||
FailureCount = 0,
|
||||
NextEligibleRun = null,
|
||||
LastFailureReason = null
|
||||
};
|
||||
|
||||
await stateRepository.SaveAsync(updated, cancellationToken).ConfigureAwait(false);
|
||||
}
|
||||
|
||||
private async Task UpdateFailureStateAsync(
|
||||
IVexConnectorStateRepository stateRepository,
|
||||
string connectorId,
|
||||
DateTimeOffset failureTime,
|
||||
Exception exception,
|
||||
bool retryable,
|
||||
CancellationToken cancellationToken)
|
||||
{
|
||||
var current = await stateRepository.GetAsync(connectorId, cancellationToken).ConfigureAwait(false)
|
||||
?? new VexConnectorState(connectorId, null, ImmutableArray<string>.Empty);
|
||||
|
||||
var failureCount = current.FailureCount + 1;
|
||||
DateTimeOffset? nextEligible;
|
||||
|
||||
if (retryable)
|
||||
{
|
||||
// Apply exponential backoff for retryable errors
|
||||
var delay = CalculateDelayWithJitter(connectorId, failureCount);
|
||||
nextEligible = failureTime + delay;
|
||||
|
||||
if (failureCount >= _retryOptions.FailureThreshold)
|
||||
{
|
||||
var quarantineUntil = failureTime + _retryOptions.QuarantineDuration;
|
||||
if (quarantineUntil > nextEligible)
|
||||
{
|
||||
nextEligible = quarantineUntil;
|
||||
}
|
||||
}
|
||||
|
||||
var retryCap = failureTime + _retryOptions.RetryCap;
|
||||
if (nextEligible > retryCap)
|
||||
{
|
||||
nextEligible = retryCap;
|
||||
}
|
||||
|
||||
if (nextEligible < failureTime)
|
||||
{
|
||||
nextEligible = failureTime;
|
||||
}
|
||||
}
|
||||
else
|
||||
{
|
||||
// Non-retryable errors: apply quarantine immediately
|
||||
nextEligible = failureTime + _retryOptions.QuarantineDuration;
|
||||
}
|
||||
|
||||
var updated = current with
|
||||
{
|
||||
FailureCount = failureCount,
|
||||
NextEligibleRun = nextEligible,
|
||||
LastFailureReason = Truncate(exception.Message, 512)
|
||||
};
|
||||
|
||||
await stateRepository.SaveAsync(updated, cancellationToken).ConfigureAwait(false);
|
||||
|
||||
_logger.LogWarning(
|
||||
exception,
|
||||
"Connector {ConnectorId} failed (attempt {Attempt}, retryable={Retryable}). Next eligible run at {NextEligible:O}.",
|
||||
connectorId,
|
||||
failureCount,
|
||||
retryable,
|
||||
nextEligible);
|
||||
}
|
||||
|
||||
internal TimeSpan CalculateDelayWithJitter(string connectorId, int failureCount)
|
||||
{
|
||||
var exponent = Math.Max(0, failureCount - 1);
|
||||
var factor = Math.Pow(2, exponent);
|
||||
var baselineTicks = (long)Math.Min(_retryOptions.BaseDelay.Ticks * factor, _retryOptions.MaxDelay.Ticks);
|
||||
|
||||
if (_retryOptions.JitterRatio <= 0)
|
||||
{
|
||||
return TimeSpan.FromTicks(baselineTicks);
|
||||
}
|
||||
|
||||
var minFactor = 1.0 - _retryOptions.JitterRatio;
|
||||
var maxFactor = 1.0 + _retryOptions.JitterRatio;
|
||||
var sample = ComputeDeterministicSample(connectorId, failureCount);
|
||||
var jitterFactor = minFactor + (maxFactor - minFactor) * sample;
|
||||
var jitteredTicks = (long)Math.Round(baselineTicks * jitterFactor, MidpointRounding.AwayFromZero);
|
||||
|
||||
jitteredTicks = Math.Clamp(jitteredTicks, _retryOptions.BaseDelay.Ticks, _retryOptions.MaxDelay.Ticks);
|
||||
|
||||
return TimeSpan.FromTicks(jitteredTicks);
|
||||
}
|
||||
|
||||
internal static double ComputeDeterministicSample(string connectorId, int failureCount)
|
||||
{
|
||||
var normalizedId = string.IsNullOrWhiteSpace(connectorId) ? "unknown" : connectorId.Trim();
|
||||
var input = string.Concat(normalizedId, ":", failureCount.ToString(CultureInfo.InvariantCulture));
|
||||
var hash = SHA256.HashData(Encoding.UTF8.GetBytes(input));
|
||||
var value = BinaryPrimitives.ReadUInt64BigEndian(hash.AsSpan(0, 8));
|
||||
return value / (double)ulong.MaxValue;
|
||||
}
|
||||
|
||||
private static string Truncate(string? value, int maxLength)
|
||||
{
|
||||
if (string.IsNullOrEmpty(value))
|
||||
{
|
||||
return string.Empty;
|
||||
}
|
||||
|
||||
return value.Length <= maxLength
|
||||
? value
|
||||
: value[..maxLength];
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,6 @@
|
||||
namespace StellaOps.Excititor.Worker.Scheduling;
|
||||
|
||||
public interface IVexConsensusRefreshScheduler
|
||||
{
|
||||
void ScheduleRefresh(string vulnerabilityId, string productKey);
|
||||
}
|
||||
@@ -0,0 +1,6 @@
|
||||
namespace StellaOps.Excititor.Worker.Scheduling;
|
||||
|
||||
internal interface IVexProviderRunner
|
||||
{
|
||||
ValueTask RunAsync(VexWorkerSchedule schedule, CancellationToken cancellationToken);
|
||||
}
|
||||
@@ -0,0 +1,675 @@
|
||||
#pragma warning disable EXCITITOR001 // Consensus logic is deprecated - refresh service manages VexConsensus during transition
|
||||
|
||||
|
||||
using Microsoft.Extensions.DependencyInjection;
|
||||
using Microsoft.Extensions.Hosting;
|
||||
using Microsoft.Extensions.Logging;
|
||||
using Microsoft.Extensions.Options;
|
||||
using StellaOps.Excititor.Core;
|
||||
using StellaOps.Excititor.Core.Lattice;
|
||||
using StellaOps.Excititor.Core.Storage;
|
||||
using StellaOps.Excititor.Formats.OpenVEX;
|
||||
using StellaOps.Excititor.Policy;
|
||||
using StellaOps.Excititor.Worker.Options;
|
||||
using System.Collections.Concurrent;
|
||||
using System.Collections.Immutable;
|
||||
using System.Linq;
|
||||
using System.Runtime.CompilerServices;
|
||||
using System.Threading.Channels;
|
||||
|
||||
namespace StellaOps.Excititor.Worker.Scheduling;
|
||||
|
||||
internal sealed class VexConsensusRefreshService : BackgroundService, IVexConsensusRefreshScheduler
|
||||
{
|
||||
private readonly IServiceScopeFactory _scopeFactory;
|
||||
private readonly ILogger<VexConsensusRefreshService> _logger;
|
||||
private readonly TimeProvider _timeProvider;
|
||||
private readonly Channel<RefreshRequest> _refreshRequests;
|
||||
private readonly ConcurrentDictionary<string, byte> _scheduledKeys = new(StringComparer.Ordinal);
|
||||
private readonly IDisposable? _optionsSubscription;
|
||||
private RefreshState _refreshState;
|
||||
private volatile bool _disableConsensus;
|
||||
|
||||
public VexConsensusRefreshService(
|
||||
IServiceScopeFactory scopeFactory,
|
||||
IOptionsMonitor<VexWorkerOptions> optionsMonitor,
|
||||
ILogger<VexConsensusRefreshService> logger,
|
||||
TimeProvider timeProvider)
|
||||
{
|
||||
_scopeFactory = scopeFactory ?? throw new ArgumentNullException(nameof(scopeFactory));
|
||||
_logger = logger ?? throw new ArgumentNullException(nameof(logger));
|
||||
_timeProvider = timeProvider ?? throw new ArgumentNullException(nameof(timeProvider));
|
||||
_refreshRequests = Channel.CreateUnbounded<RefreshRequest>(new UnboundedChannelOptions
|
||||
{
|
||||
AllowSynchronousContinuations = false,
|
||||
SingleReader = true,
|
||||
SingleWriter = false,
|
||||
});
|
||||
|
||||
if (optionsMonitor is null)
|
||||
{
|
||||
throw new ArgumentNullException(nameof(optionsMonitor));
|
||||
}
|
||||
|
||||
var options = optionsMonitor.CurrentValue;
|
||||
_disableConsensus = options.DisableConsensus;
|
||||
_refreshState = RefreshState.FromOptions(options.Refresh);
|
||||
_optionsSubscription = optionsMonitor.OnChange(o =>
|
||||
{
|
||||
var state = RefreshState.FromOptions((o?.Refresh) ?? new VexWorkerRefreshOptions());
|
||||
_disableConsensus = o?.DisableConsensus ?? false;
|
||||
Volatile.Write(ref _refreshState, state);
|
||||
_logger.LogInformation(
|
||||
"Consensus refresh options updated: enabled={Enabled}, interval={Interval}, ttl={Ttl}, batch={Batch}",
|
||||
state.Enabled,
|
||||
state.ScanInterval,
|
||||
state.ConsensusTtl,
|
||||
state.ScanBatchSize);
|
||||
});
|
||||
}
|
||||
|
||||
public override void Dispose()
|
||||
{
|
||||
_optionsSubscription?.Dispose();
|
||||
base.Dispose();
|
||||
}
|
||||
|
||||
public void ScheduleRefresh(string vulnerabilityId, string productKey)
|
||||
{
|
||||
if (string.IsNullOrWhiteSpace(vulnerabilityId) || string.IsNullOrWhiteSpace(productKey))
|
||||
{
|
||||
return;
|
||||
}
|
||||
|
||||
if (_disableConsensus)
|
||||
{
|
||||
_logger.LogDebug("Consensus refresh disabled; ignoring schedule request for {VulnerabilityId}/{ProductKey}.", vulnerabilityId, productKey);
|
||||
return;
|
||||
}
|
||||
|
||||
var key = BuildKey(vulnerabilityId, productKey);
|
||||
if (!_scheduledKeys.TryAdd(key, 0))
|
||||
{
|
||||
return;
|
||||
}
|
||||
|
||||
var request = new RefreshRequest(vulnerabilityId.Trim(), productKey.Trim());
|
||||
if (!_refreshRequests.Writer.TryWrite(request))
|
||||
{
|
||||
_scheduledKeys.TryRemove(key, out _);
|
||||
}
|
||||
}
|
||||
|
||||
protected override async Task ExecuteAsync(CancellationToken stoppingToken)
|
||||
{
|
||||
var queueTask = ProcessQueueAsync(stoppingToken);
|
||||
|
||||
try
|
||||
{
|
||||
while (!stoppingToken.IsCancellationRequested)
|
||||
{
|
||||
if (_disableConsensus)
|
||||
{
|
||||
_logger.LogInformation("Consensus refresh disabled via DisableConsensus flag; exiting refresh loop.");
|
||||
break;
|
||||
}
|
||||
|
||||
var options = CurrentOptions;
|
||||
|
||||
try
|
||||
{
|
||||
await ProcessEligibleHoldsAsync(options, stoppingToken).ConfigureAwait(false);
|
||||
if (options.Enabled)
|
||||
{
|
||||
await ProcessTtlRefreshAsync(options, stoppingToken).ConfigureAwait(false);
|
||||
}
|
||||
else
|
||||
{
|
||||
_logger.LogDebug("Consensus refresh disabled; skipping TTL sweep.");
|
||||
}
|
||||
}
|
||||
catch (OperationCanceledException) when (stoppingToken.IsCancellationRequested)
|
||||
{
|
||||
break;
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
_logger.LogError(ex, "Consensus refresh loop failed.");
|
||||
}
|
||||
|
||||
try
|
||||
{
|
||||
await Task.Delay(options.ScanInterval, stoppingToken).ConfigureAwait(false);
|
||||
}
|
||||
catch (OperationCanceledException)
|
||||
{
|
||||
break;
|
||||
}
|
||||
}
|
||||
}
|
||||
finally
|
||||
{
|
||||
_refreshRequests.Writer.TryComplete();
|
||||
try
|
||||
{
|
||||
await queueTask.ConfigureAwait(false);
|
||||
}
|
||||
catch (OperationCanceledException)
|
||||
{
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
private RefreshState CurrentOptions => Volatile.Read(ref _refreshState);
|
||||
|
||||
private async Task ProcessQueueAsync(CancellationToken cancellationToken)
|
||||
{
|
||||
try
|
||||
{
|
||||
while (await _refreshRequests.Reader.WaitToReadAsync(cancellationToken).ConfigureAwait(false))
|
||||
{
|
||||
while (_refreshRequests.Reader.TryRead(out var request))
|
||||
{
|
||||
var key = BuildKey(request.VulnerabilityId, request.ProductKey);
|
||||
try
|
||||
{
|
||||
await ProcessCandidateAsync(request.VulnerabilityId, request.ProductKey, existingConsensus: null, CurrentOptions, cancellationToken).ConfigureAwait(false);
|
||||
}
|
||||
catch (OperationCanceledException) when (cancellationToken.IsCancellationRequested)
|
||||
{
|
||||
return;
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
_logger.LogError(ex, "Failed to refresh consensus for {VulnerabilityId}/{ProductKey} from queue.", request.VulnerabilityId, request.ProductKey);
|
||||
}
|
||||
finally
|
||||
{
|
||||
_scheduledKeys.TryRemove(key, out _);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
catch (OperationCanceledException)
|
||||
{
|
||||
}
|
||||
}
|
||||
|
||||
private async Task ProcessEligibleHoldsAsync(RefreshState options, CancellationToken cancellationToken)
|
||||
{
|
||||
using var scope = _scopeFactory.CreateScope();
|
||||
var holdStore = scope.ServiceProvider.GetRequiredService<IVexConsensusHoldStore>();
|
||||
var consensusStore = scope.ServiceProvider.GetRequiredService<IVexConsensusStore>();
|
||||
|
||||
var now = _timeProvider.GetUtcNow();
|
||||
await foreach (var hold in holdStore.FindEligibleAsync(now, options.ScanBatchSize, cancellationToken).ConfigureAwait(false))
|
||||
{
|
||||
var key = BuildKey(hold.VulnerabilityId, hold.ProductKey);
|
||||
if (!_scheduledKeys.TryAdd(key, 0))
|
||||
{
|
||||
continue;
|
||||
}
|
||||
|
||||
try
|
||||
{
|
||||
await consensusStore.SaveAsync(hold.Candidate with { }, cancellationToken).ConfigureAwait(false);
|
||||
await holdStore.RemoveAsync(hold.VulnerabilityId, hold.ProductKey, cancellationToken).ConfigureAwait(false);
|
||||
_logger.LogInformation(
|
||||
"Promoted consensus hold for {VulnerabilityId}/{ProductKey}; status={Status}, reason={Reason}",
|
||||
hold.VulnerabilityId,
|
||||
hold.ProductKey,
|
||||
hold.Candidate.Status,
|
||||
hold.Reason);
|
||||
}
|
||||
catch (Exception ex) when (!cancellationToken.IsCancellationRequested)
|
||||
{
|
||||
_logger.LogError(
|
||||
ex,
|
||||
"Failed to promote consensus hold for {VulnerabilityId}/{ProductKey}.",
|
||||
hold.VulnerabilityId,
|
||||
hold.ProductKey);
|
||||
}
|
||||
finally
|
||||
{
|
||||
_scheduledKeys.TryRemove(key, out _);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
private async Task ProcessTtlRefreshAsync(RefreshState options, CancellationToken cancellationToken)
|
||||
{
|
||||
var now = _timeProvider.GetUtcNow();
|
||||
var cutoff = now - options.ConsensusTtl;
|
||||
|
||||
using var scope = _scopeFactory.CreateScope();
|
||||
var consensusStore = scope.ServiceProvider.GetRequiredService<IVexConsensusStore>();
|
||||
|
||||
await foreach (var consensus in consensusStore.FindCalculatedBeforeAsync(cutoff, options.ScanBatchSize, cancellationToken).ConfigureAwait(false))
|
||||
{
|
||||
var key = BuildKey(consensus.VulnerabilityId, consensus.Product.Key);
|
||||
if (!_scheduledKeys.TryAdd(key, 0))
|
||||
{
|
||||
continue;
|
||||
}
|
||||
|
||||
try
|
||||
{
|
||||
await ProcessCandidateAsync(consensus.VulnerabilityId, consensus.Product.Key, consensus, options, cancellationToken).ConfigureAwait(false);
|
||||
}
|
||||
catch (Exception ex) when (!cancellationToken.IsCancellationRequested)
|
||||
{
|
||||
_logger.LogError(
|
||||
ex,
|
||||
"Failed to refresh consensus for {VulnerabilityId}/{ProductKey} during TTL sweep.",
|
||||
consensus.VulnerabilityId,
|
||||
consensus.Product.Key);
|
||||
}
|
||||
finally
|
||||
{
|
||||
_scheduledKeys.TryRemove(key, out _);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
private async Task ProcessCandidateAsync(
|
||||
string vulnerabilityId,
|
||||
string productKey,
|
||||
VexConsensus? existingConsensus,
|
||||
RefreshState options,
|
||||
CancellationToken cancellationToken)
|
||||
{
|
||||
using var scope = _scopeFactory.CreateScope();
|
||||
var consensusStore = scope.ServiceProvider.GetRequiredService<IVexConsensusStore>();
|
||||
var holdStore = scope.ServiceProvider.GetRequiredService<IVexConsensusHoldStore>();
|
||||
var claimStore = scope.ServiceProvider.GetRequiredService<IVexClaimStore>();
|
||||
var policyProvider = scope.ServiceProvider.GetRequiredService<IVexPolicyProvider>();
|
||||
var merger = scope.ServiceProvider.GetRequiredService<OpenVexStatementMerger>();
|
||||
var lattice = scope.ServiceProvider.GetRequiredService<IVexLatticeProvider>();
|
||||
|
||||
existingConsensus ??= await consensusStore.FindAsync(vulnerabilityId, productKey, cancellationToken).ConfigureAwait(false);
|
||||
|
||||
var claims = await claimStore.FindAsync(vulnerabilityId, productKey, since: null, cancellationToken).ConfigureAwait(false);
|
||||
if (claims.Count == 0)
|
||||
{
|
||||
_logger.LogDebug("No claims found for {VulnerabilityId}/{ProductKey}; skipping consensus refresh.", vulnerabilityId, productKey);
|
||||
await holdStore.RemoveAsync(vulnerabilityId, productKey, cancellationToken).ConfigureAwait(false);
|
||||
return;
|
||||
}
|
||||
|
||||
var claimList = claims as IReadOnlyList<VexClaim> ?? claims.ToList();
|
||||
|
||||
var snapshot = policyProvider.GetSnapshot();
|
||||
var product = ResolveProduct(claimList, productKey);
|
||||
var candidate = NormalizePolicyMetadata(
|
||||
BuildConsensus(
|
||||
vulnerabilityId,
|
||||
claimList,
|
||||
product,
|
||||
AggregateSignals(claimList),
|
||||
snapshot,
|
||||
merger,
|
||||
lattice,
|
||||
_timeProvider),
|
||||
snapshot);
|
||||
|
||||
await ApplyConsensusAsync(
|
||||
candidate,
|
||||
existingConsensus,
|
||||
holdStore,
|
||||
consensusStore,
|
||||
options.Damper,
|
||||
options,
|
||||
cancellationToken).ConfigureAwait(false);
|
||||
}
|
||||
|
||||
private async Task ApplyConsensusAsync(
|
||||
VexConsensus candidate,
|
||||
VexConsensus? existing,
|
||||
IVexConsensusHoldStore holdStore,
|
||||
IVexConsensusStore consensusStore,
|
||||
DamperState damper,
|
||||
RefreshState options,
|
||||
CancellationToken cancellationToken)
|
||||
{
|
||||
var vulnerabilityId = candidate.VulnerabilityId;
|
||||
var productKey = candidate.Product.Key;
|
||||
|
||||
var componentChanged = HasComponentChange(existing, candidate);
|
||||
var statusChanged = existing is not null && existing.Status != candidate.Status;
|
||||
|
||||
if (existing is null)
|
||||
{
|
||||
await consensusStore.SaveAsync(candidate, cancellationToken).ConfigureAwait(false);
|
||||
await holdStore.RemoveAsync(vulnerabilityId, productKey, cancellationToken).ConfigureAwait(false);
|
||||
_logger.LogInformation("Stored initial consensus for {VulnerabilityId}/{ProductKey} with status {Status}.", vulnerabilityId, productKey, candidate.Status);
|
||||
return;
|
||||
}
|
||||
|
||||
TimeSpan duration = TimeSpan.Zero;
|
||||
if (statusChanged)
|
||||
{
|
||||
if (componentChanged)
|
||||
{
|
||||
duration = TimeSpan.Zero;
|
||||
}
|
||||
else
|
||||
{
|
||||
var mappedStatus = MapConsensusStatus(candidate.Status);
|
||||
var supportingWeight = mappedStatus is null
|
||||
? 0d
|
||||
: candidate.Sources
|
||||
.Where(source => source.Status == mappedStatus.Value)
|
||||
.Sum(source => source.Weight);
|
||||
duration = damper.ResolveDuration(supportingWeight);
|
||||
}
|
||||
}
|
||||
|
||||
var requestedAt = _timeProvider.GetUtcNow();
|
||||
|
||||
if (statusChanged && duration > TimeSpan.Zero)
|
||||
{
|
||||
var eligibleAt = requestedAt + duration;
|
||||
var reason = componentChanged ? "component_change" : "status_change";
|
||||
var newHold = new VexConsensusHold(vulnerabilityId, productKey, candidate, requestedAt, eligibleAt, reason);
|
||||
var existingHold = await holdStore.FindAsync(vulnerabilityId, productKey, cancellationToken).ConfigureAwait(false);
|
||||
|
||||
if (existingHold is null || existingHold.Candidate != candidate || existingHold.EligibleAt != newHold.EligibleAt)
|
||||
{
|
||||
await holdStore.SaveAsync(newHold, cancellationToken).ConfigureAwait(false);
|
||||
_logger.LogInformation(
|
||||
"Deferred consensus update for {VulnerabilityId}/{ProductKey} until {EligibleAt:O}; status {Status} pending (reason={Reason}).",
|
||||
vulnerabilityId,
|
||||
productKey,
|
||||
eligibleAt,
|
||||
candidate.Status,
|
||||
reason);
|
||||
}
|
||||
return;
|
||||
}
|
||||
|
||||
await consensusStore.SaveAsync(candidate, cancellationToken).ConfigureAwait(false);
|
||||
await holdStore.RemoveAsync(vulnerabilityId, productKey, cancellationToken).ConfigureAwait(false);
|
||||
_logger.LogInformation(
|
||||
"Updated consensus for {VulnerabilityId}/{ProductKey}; status={Status}, componentChange={ComponentChanged}.",
|
||||
vulnerabilityId,
|
||||
productKey,
|
||||
candidate.Status,
|
||||
componentChanged);
|
||||
}
|
||||
|
||||
private static bool HasComponentChange(VexConsensus? existing, VexConsensus candidate)
|
||||
{
|
||||
if (existing is null)
|
||||
{
|
||||
return false;
|
||||
}
|
||||
|
||||
var previous = existing.Product.ComponentIdentifiers;
|
||||
var current = candidate.Product.ComponentIdentifiers;
|
||||
|
||||
if (previous.IsDefaultOrEmpty && current.IsDefaultOrEmpty)
|
||||
{
|
||||
return false;
|
||||
}
|
||||
|
||||
if (previous.Length != current.Length)
|
||||
{
|
||||
return true;
|
||||
}
|
||||
|
||||
for (var i = 0; i < previous.Length; i++)
|
||||
{
|
||||
if (!string.Equals(previous[i], current[i], StringComparison.Ordinal))
|
||||
{
|
||||
return true;
|
||||
}
|
||||
}
|
||||
|
||||
return false;
|
||||
}
|
||||
|
||||
private static VexConsensus NormalizePolicyMetadata(VexConsensus consensus, VexPolicySnapshot snapshot)
|
||||
{
|
||||
if (string.Equals(consensus.PolicyVersion, snapshot.Version, StringComparison.Ordinal) &&
|
||||
string.Equals(consensus.PolicyRevisionId, snapshot.RevisionId, StringComparison.Ordinal) &&
|
||||
string.Equals(consensus.PolicyDigest, snapshot.Digest, StringComparison.Ordinal))
|
||||
{
|
||||
return consensus;
|
||||
}
|
||||
|
||||
return new VexConsensus(
|
||||
consensus.VulnerabilityId,
|
||||
consensus.Product,
|
||||
consensus.Status,
|
||||
consensus.CalculatedAt,
|
||||
consensus.Sources,
|
||||
consensus.Conflicts,
|
||||
consensus.Signals,
|
||||
snapshot.Version,
|
||||
consensus.Summary,
|
||||
snapshot.RevisionId,
|
||||
snapshot.Digest);
|
||||
}
|
||||
|
||||
private static VexClaimStatus? MapConsensusStatus(VexConsensusStatus status)
|
||||
=> status switch
|
||||
{
|
||||
VexConsensusStatus.Affected => VexClaimStatus.Affected,
|
||||
VexConsensusStatus.NotAffected => VexClaimStatus.NotAffected,
|
||||
VexConsensusStatus.Fixed => VexClaimStatus.Fixed,
|
||||
_ => null,
|
||||
};
|
||||
|
||||
private static string BuildKey(string vulnerabilityId, string productKey)
|
||||
=> string.Create(
|
||||
vulnerabilityId.Length + productKey.Length + 1,
|
||||
(vulnerabilityId, productKey),
|
||||
static (span, tuple) =>
|
||||
{
|
||||
tuple.vulnerabilityId.AsSpan().CopyTo(span);
|
||||
span[tuple.vulnerabilityId.Length] = '|';
|
||||
tuple.productKey.AsSpan().CopyTo(span[(tuple.vulnerabilityId.Length + 1)..]);
|
||||
});
|
||||
|
||||
private static VexProduct ResolveProduct(IReadOnlyList<VexClaim> claims, string productKey)
|
||||
{
|
||||
if (claims.Count > 0)
|
||||
{
|
||||
return claims[0].Product;
|
||||
}
|
||||
|
||||
var inferredPurl = productKey.StartsWith("pkg:", StringComparison.OrdinalIgnoreCase) ? productKey : null;
|
||||
return new VexProduct(productKey, name: null, version: null, purl: inferredPurl);
|
||||
}
|
||||
|
||||
private static VexConsensus BuildConsensus(
|
||||
string vulnerabilityId,
|
||||
IReadOnlyList<VexClaim> claims,
|
||||
VexProduct product,
|
||||
VexSignalSnapshot? signals,
|
||||
VexPolicySnapshot snapshot,
|
||||
OpenVexStatementMerger merger,
|
||||
IVexLatticeProvider lattice,
|
||||
TimeProvider timeProvider)
|
||||
{
|
||||
var calculatedAt = timeProvider.GetUtcNow();
|
||||
|
||||
if (claims.Count == 0)
|
||||
{
|
||||
return new VexConsensus(
|
||||
vulnerabilityId,
|
||||
product,
|
||||
VexConsensusStatus.UnderInvestigation,
|
||||
calculatedAt,
|
||||
Array.Empty<VexConsensusSource>(),
|
||||
Array.Empty<VexConsensusConflict>(),
|
||||
signals,
|
||||
snapshot.Version,
|
||||
"No claims available.",
|
||||
snapshot.RevisionId,
|
||||
snapshot.Digest);
|
||||
}
|
||||
|
||||
var mergeResult = merger.MergeClaims(claims);
|
||||
var consensusStatus = MapConsensusStatusFromClaim(mergeResult.ResultStatement.Status);
|
||||
var sources = claims
|
||||
.Select(claim => new VexConsensusSource(
|
||||
claim.ProviderId,
|
||||
claim.Status,
|
||||
claim.Document.Digest,
|
||||
(double)lattice.GetTrustWeight(claim),
|
||||
claim.Justification,
|
||||
claim.Detail,
|
||||
claim.Confidence))
|
||||
.ToArray();
|
||||
|
||||
var conflicts = claims
|
||||
.Where(claim => claim.Status != mergeResult.ResultStatement.Status)
|
||||
.Select(claim => new VexConsensusConflict(
|
||||
claim.ProviderId,
|
||||
claim.Status,
|
||||
claim.Document.Digest,
|
||||
claim.Justification,
|
||||
claim.Detail,
|
||||
"status_conflict"))
|
||||
.ToArray();
|
||||
|
||||
var summary = MergeTraceWriter.ToExplanation(mergeResult);
|
||||
|
||||
return new VexConsensus(
|
||||
vulnerabilityId,
|
||||
product,
|
||||
consensusStatus,
|
||||
calculatedAt,
|
||||
sources,
|
||||
conflicts,
|
||||
signals,
|
||||
snapshot.Version,
|
||||
summary,
|
||||
snapshot.RevisionId,
|
||||
snapshot.Digest);
|
||||
}
|
||||
|
||||
private static VexConsensusStatus MapConsensusStatusFromClaim(VexClaimStatus status)
|
||||
=> status switch
|
||||
{
|
||||
VexClaimStatus.Affected => VexConsensusStatus.Affected,
|
||||
VexClaimStatus.NotAffected => VexConsensusStatus.NotAffected,
|
||||
VexClaimStatus.Fixed => VexConsensusStatus.Fixed,
|
||||
_ => VexConsensusStatus.UnderInvestigation,
|
||||
};
|
||||
|
||||
private static VexSignalSnapshot? AggregateSignals(IReadOnlyList<VexClaim> claims)
|
||||
{
|
||||
if (claims.Count == 0)
|
||||
{
|
||||
return null;
|
||||
}
|
||||
|
||||
VexSeveritySignal? bestSeverity = null;
|
||||
double? bestScore = null;
|
||||
bool kevPresent = false;
|
||||
bool kevTrue = false;
|
||||
double? bestEpss = null;
|
||||
|
||||
foreach (var claim in claims)
|
||||
{
|
||||
if (claim.Signals is null)
|
||||
{
|
||||
continue;
|
||||
}
|
||||
|
||||
var severity = claim.Signals.Severity;
|
||||
if (severity is not null)
|
||||
{
|
||||
var score = severity.Score;
|
||||
if (bestSeverity is null ||
|
||||
(score is not null && (bestScore is null || score.Value > bestScore.Value)) ||
|
||||
(score is null && bestScore is null && !string.IsNullOrWhiteSpace(severity.Label) && string.IsNullOrWhiteSpace(bestSeverity.Label)))
|
||||
{
|
||||
bestSeverity = severity;
|
||||
bestScore = severity.Score;
|
||||
}
|
||||
}
|
||||
|
||||
if (claim.Signals.Kev is { } kevValue)
|
||||
{
|
||||
kevPresent = true;
|
||||
if (kevValue)
|
||||
{
|
||||
kevTrue = true;
|
||||
}
|
||||
}
|
||||
|
||||
if (claim.Signals.Epss is { } epss)
|
||||
{
|
||||
if (bestEpss is null || epss > bestEpss.Value)
|
||||
{
|
||||
bestEpss = epss;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
if (bestSeverity is null && !kevPresent && bestEpss is null)
|
||||
{
|
||||
return null;
|
||||
}
|
||||
|
||||
bool? kev = kevTrue ? true : (kevPresent ? false : null);
|
||||
return new VexSignalSnapshot(bestSeverity, kev, bestEpss);
|
||||
}
|
||||
|
||||
private readonly record struct RefreshRequest(string VulnerabilityId, string ProductKey);
|
||||
|
||||
private sealed record RefreshState(
|
||||
bool Enabled,
|
||||
TimeSpan ScanInterval,
|
||||
TimeSpan ConsensusTtl,
|
||||
int ScanBatchSize,
|
||||
DamperState Damper)
|
||||
{
|
||||
public static RefreshState FromOptions(VexWorkerRefreshOptions options)
|
||||
{
|
||||
var interval = options.ScanInterval > TimeSpan.Zero ? options.ScanInterval : TimeSpan.FromMinutes(10);
|
||||
var ttl = options.ConsensusTtl > TimeSpan.Zero ? options.ConsensusTtl : TimeSpan.FromHours(2);
|
||||
var batchSize = options.ScanBatchSize > 0 ? options.ScanBatchSize : 250;
|
||||
var damper = DamperState.FromOptions(options.Damper);
|
||||
return new RefreshState(options.Enabled, interval, ttl, batchSize, damper);
|
||||
}
|
||||
}
|
||||
|
||||
private sealed record DamperState(TimeSpan Minimum, TimeSpan Maximum, TimeSpan DefaultDuration, ImmutableArray<DamperRuleState> Rules)
|
||||
{
|
||||
public static DamperState FromOptions(VexStabilityDamperOptions options)
|
||||
{
|
||||
var minimum = options.Minimum < TimeSpan.Zero ? TimeSpan.Zero : options.Minimum;
|
||||
var maximum = options.Maximum > minimum ? options.Maximum : minimum + TimeSpan.FromHours(1);
|
||||
var defaultDuration = options.ClampDuration(options.DefaultDuration);
|
||||
var rules = options.Rules
|
||||
.Select(rule => new DamperRuleState(Math.Max(0, rule.MinWeight), options.ClampDuration(rule.Duration)))
|
||||
.OrderByDescending(rule => rule.MinWeight)
|
||||
.ToImmutableArray();
|
||||
return new DamperState(minimum, maximum, defaultDuration, rules);
|
||||
}
|
||||
|
||||
public TimeSpan ResolveDuration(double weight)
|
||||
{
|
||||
if (double.IsNaN(weight) || double.IsInfinity(weight) || weight < 0)
|
||||
{
|
||||
return DefaultDuration;
|
||||
}
|
||||
|
||||
foreach (var rule in Rules)
|
||||
{
|
||||
if (weight >= rule.MinWeight)
|
||||
{
|
||||
return rule.Duration;
|
||||
}
|
||||
}
|
||||
|
||||
return DefaultDuration;
|
||||
}
|
||||
}
|
||||
|
||||
private sealed record DamperRuleState(double MinWeight, TimeSpan Duration);
|
||||
}
|
||||
@@ -0,0 +1,111 @@
|
||||
|
||||
using Microsoft.Extensions.Hosting;
|
||||
using Microsoft.Extensions.Logging;
|
||||
using Microsoft.Extensions.Options;
|
||||
using StellaOps.Excititor.Worker.Options;
|
||||
using System;
|
||||
using System.Collections.Generic;
|
||||
using System.Threading;
|
||||
using System.Threading.Tasks;
|
||||
|
||||
namespace StellaOps.Excititor.Worker.Scheduling;
|
||||
|
||||
internal class VexWorkerHostedService : BackgroundService
|
||||
{
|
||||
private readonly IOptions<VexWorkerOptions> _options;
|
||||
private readonly IVexProviderRunner _runner;
|
||||
private readonly ILogger<VexWorkerHostedService> _logger;
|
||||
private readonly TimeProvider _timeProvider;
|
||||
|
||||
public VexWorkerHostedService(
|
||||
IOptions<VexWorkerOptions> options,
|
||||
IVexProviderRunner runner,
|
||||
ILogger<VexWorkerHostedService> logger,
|
||||
TimeProvider timeProvider)
|
||||
{
|
||||
_options = options ?? throw new ArgumentNullException(nameof(options));
|
||||
_runner = runner ?? throw new ArgumentNullException(nameof(runner));
|
||||
_logger = logger ?? throw new ArgumentNullException(nameof(logger));
|
||||
_timeProvider = timeProvider ?? throw new ArgumentNullException(nameof(timeProvider));
|
||||
}
|
||||
|
||||
protected override async Task ExecuteAsync(CancellationToken stoppingToken)
|
||||
{
|
||||
var schedules = _options.Value.ResolveSchedules();
|
||||
if (schedules.Count == 0)
|
||||
{
|
||||
_logger.LogWarning("Excititor worker has no configured provider schedules; the service will remain idle.");
|
||||
await Task.CompletedTask;
|
||||
return;
|
||||
}
|
||||
|
||||
_logger.LogInformation("Excititor worker starting with {ProviderCount} provider schedule(s).", schedules.Count);
|
||||
|
||||
var tasks = new List<Task>(schedules.Count);
|
||||
foreach (var schedule in schedules)
|
||||
{
|
||||
tasks.Add(RunScheduleAsync(schedule, stoppingToken));
|
||||
}
|
||||
|
||||
await Task.WhenAll(tasks);
|
||||
}
|
||||
|
||||
private async Task RunScheduleAsync(VexWorkerSchedule schedule, CancellationToken cancellationToken)
|
||||
{
|
||||
try
|
||||
{
|
||||
if (schedule.InitialDelay > TimeSpan.Zero)
|
||||
{
|
||||
_logger.LogInformation(
|
||||
"Provider {ProviderId} initial delay of {InitialDelay} before first execution.",
|
||||
schedule.ProviderId,
|
||||
schedule.InitialDelay);
|
||||
|
||||
await Task.Delay(schedule.InitialDelay, cancellationToken).ConfigureAwait(false);
|
||||
}
|
||||
|
||||
using var timer = new PeriodicTimer(schedule.Interval, _timeProvider);
|
||||
do
|
||||
{
|
||||
var startedAt = _timeProvider.GetUtcNow();
|
||||
_logger.LogInformation(
|
||||
"Provider {ProviderId} run started at {StartedAt}. Interval={Interval}.",
|
||||
schedule.ProviderId,
|
||||
startedAt,
|
||||
schedule.Interval);
|
||||
|
||||
try
|
||||
{
|
||||
await _runner.RunAsync(schedule, cancellationToken).ConfigureAwait(false);
|
||||
|
||||
var completedAt = _timeProvider.GetUtcNow();
|
||||
var elapsed = completedAt - startedAt;
|
||||
|
||||
_logger.LogInformation(
|
||||
"Provider {ProviderId} run completed at {CompletedAt} (duration {Duration}).",
|
||||
schedule.ProviderId,
|
||||
completedAt,
|
||||
elapsed);
|
||||
}
|
||||
catch (OperationCanceledException) when (cancellationToken.IsCancellationRequested)
|
||||
{
|
||||
_logger.LogInformation("Provider {ProviderId} run cancelled.", schedule.ProviderId);
|
||||
break;
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
_logger.LogError(
|
||||
ex,
|
||||
"Provider {ProviderId} run failed: {Message}",
|
||||
schedule.ProviderId,
|
||||
ex.Message);
|
||||
}
|
||||
}
|
||||
while (await timer.WaitForNextTickAsync(cancellationToken).ConfigureAwait(false));
|
||||
}
|
||||
catch (OperationCanceledException) when (cancellationToken.IsCancellationRequested)
|
||||
{
|
||||
_logger.LogInformation("Provider {ProviderId} schedule cancelled.", schedule.ProviderId);
|
||||
}
|
||||
}
|
||||
}
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user