diff --git a/docs/dev/aoc-normalization-removal-notes.md b/docs/dev/aoc-normalization-removal-notes.md new file mode 100644 index 00000000..516c142f --- /dev/null +++ b/docs/dev/aoc-normalization-removal-notes.md @@ -0,0 +1,21 @@ +# AOC Normalization Removal Notes + +_Last updated: 2025-10-29_ + +## Goal + +Document follow-up actions for CONCELIER-CORE-AOC-19-004 as we unwind the final pieces of normalization from the ingestion/runtime path. + +## Current Findings + +- `AdvisoryRawService` and `MongoAdvisoryRawRepository` already preserve upstream ordering and duplicate aliases (trim-only). No additional code changes required there. +- Observation layers (`AdvisoryObservationFactory`, `AdvisoryObservationQueryService`) still canonicalise aliases, PURLs, CPEs, and references. These need to be relaxed so Policy/overlays receive raw linksets and can own dedupe logic. +- Linkset mapper continues to emit deterministic hints. We will keep the mapper but ensure observation output can surface both raw and canonical views for downstream services. + +## Next Steps + +1. Introduce a raw linkset projection alongside the existing canonical mapper so Policy Engine can choose which flavour to consume. +2. Update observation factory/query tests to assert duplicate handling and ordering with the relaxed projection. +3. Refresh docs (`docs/ingestion/aggregation-only-contract.md`) once behaviour lands to explain the “raw vs canonical linkset” split. +4. Coordinate with Policy Guild to validate consumers against the new raw projection before flipping defaults. + diff --git a/docs/implplan/EXECPLAN.md b/docs/implplan/EXECPLAN.md index 95e420ad..83644641 100644 --- a/docs/implplan/EXECPLAN.md +++ b/docs/implplan/EXECPLAN.md @@ -115,7 +115,7 @@ Generated from SPRINTS.md and module TASKS.md files on 2025-10-19. Waves cluster - Team Notify Engine Guild: read EXECPLAN.md Wave 4 and SPRINTS.md rows for `src/Notify/__Libraries/StellaOps.Notify.Engine/TASKS.md`. Focus on NOTIFY-ENGINE-15-304 (TODO). Confirm prerequisites (internal: NOTIFY-ENGINE-15-303 (Wave 3)) before starting and report status in module TASKS.md. - Team Notify Worker Guild: read EXECPLAN.md Wave 4 and SPRINTS.md rows for `src/Notify/StellaOps.Notify.Worker/TASKS.md`. Focus on NOTIFY-WORKER-15-204 (TODO). Confirm prerequisites (internal: NOTIFY-WORKER-15-203 (Wave 3)) before starting and report status in module TASKS.md. - Team Scheduler Worker Guild: read EXECPLAN.md Wave 4 and SPRINTS.md rows for `src/Scheduler/__Libraries/StellaOps.Scheduler.Worker/TASKS.md`. Focus on SCHED-WORKER-16-204 (TODO). Confirm prerequisites (internal: SCHED-WORKER-16-203 (Wave 3)) before starting and report status in module TASKS.md. -- Team TBD: read EXECPLAN.md Wave 4 and SPRINTS.md rows for `src/Scanner/__Libraries/StellaOps.Scanner.Analyzers.Lang.DotNet/TASKS.md`, `src/Scanner/__Libraries/StellaOps.Scanner.Analyzers.Lang.Go/TASKS.md`, `src/Scanner/__Libraries/StellaOps.Scanner.Analyzers.Lang.Python/TASKS.md`, `src/Scanner/__Libraries/StellaOps.Scanner.Analyzers.Lang.Rust/TASKS.md`. SCANNER-ANALYZERS-LANG-10-307D/G/P are DONE (latest 2025-10-23); remaining focus is SCANNER-ANALYZERS-LANG-10-307R (TODO). Confirm prerequisites (internal: SCANNER-ANALYZERS-LANG-10-303C (Wave 3), SCANNER-ANALYZERS-LANG-10-304C (Wave 3), SCANNER-ANALYZERS-LANG-10-305C (Wave 3), SCANNER-ANALYZERS-LANG-10-306C (Wave 3)) before progressing and report status in module TASKS.md. +- Team TBD: read EXECPLAN.md Wave 4 and SPRINTS.md rows for `src/Scanner/__Libraries/StellaOps.Scanner.Analyzers.Lang.DotNet/TASKS.md`, `src/Scanner/__Libraries/StellaOps.Scanner.Analyzers.Lang.Go/TASKS.md`, `src/Scanner/__Libraries/StellaOps.Scanner.Analyzers.Lang.Python/TASKS.md`, `src/Scanner/__Libraries/StellaOps.Scanner.Analyzers.Lang.Rust/TASKS.md`. SCANNER-ANALYZERS-LANG-10-307D/G/P are DONE (latest 2025-10-23); remaining focus is SCANNER-ANALYZERS-LANG-10-307R (DOING). Confirm prerequisites (internal: SCANNER-ANALYZERS-LANG-10-303C (Wave 3), SCANNER-ANALYZERS-LANG-10-304C (Wave 3), SCANNER-ANALYZERS-LANG-10-305C (Wave 3), SCANNER-ANALYZERS-LANG-10-306C (Wave 3)) before progressing and report status in module TASKS.md. ### Wave 5 - **Sprint 23-28** · StellaOps Console, Policy Studio, Graph Explorer @@ -438,26 +438,26 @@ Generated from SPRINTS.md and module TASKS.md files on 2025-10-19. Waves cluster • Current: TODO – Fetch CSAF bundles with ETag handling, checksum validation, deduplication, and raw persistence. - Team: Team Excititor Formats - Path: `src/Excititor/__Libraries/StellaOps.Excititor.Formats.CSAF/TASKS.md` - 1. [TODO] EXCITITOR-FMT-CSAF-01-002 — EXCITITOR-FMT-CSAF-01-002 – Status/justification mapping + 1. [DONE 2025-10-29] EXCITITOR-FMT-CSAF-01-002 — EXCITITOR-FMT-CSAF-01-002 – Status/justification mapping • Prereqs: EXCITITOR-FMT-CSAF-01-001 (external/completed), EXCITITOR-POLICY-01-001 (external/completed) - • Current: TODO – Normalize CSAF `product_status` + `justification` values into policy-aware enums with audit diagnostics for unsupported codes. - 2. [TODO] EXCITITOR-FMT-CSAF-01-003 — EXCITITOR-FMT-CSAF-01-003 – CSAF export adapter + • Current: DONE – Normalizer now emits policy-safe status/justification mappings and flags unsupported or missing evidence for audit diagnostics. + 2. [DONE 2025-10-29] EXCITITOR-FMT-CSAF-01-003 — EXCITITOR-FMT-CSAF-01-003 – CSAF export adapter • Prereqs: EXCITITOR-EXPORT-01-001 (external/completed), EXCITITOR-FMT-CSAF-01-001 (external/completed) - • Current: TODO – Provide CSAF export writer producing deterministic documents (per vuln/product) and manifest metadata for attestation. + • Current: DONE – CSAF exporter produces deterministic documents with reconciled product tree, vulnerability statuses, and export metadata. - Path: `src/Excititor/__Libraries/StellaOps.Excititor.Formats.CycloneDX/TASKS.md` - 1. [TODO] EXCITITOR-FMT-CYCLONE-01-002 — EXCITITOR-FMT-CYCLONE-01-002 – Component reference reconciliation + 1. [DONE 2025-10-29] EXCITITOR-FMT-CYCLONE-01-002 — EXCITITOR-FMT-CYCLONE-01-002 – Component reference reconciliation • Prereqs: EXCITITOR-FMT-CYCLONE-01-001 (external/completed) - • Current: TODO – Implement helpers to reconcile component/service references against policy expectations and emit diagnostics for missing SBOM links. - 2. [TODO] EXCITITOR-FMT-CYCLONE-01-003 — EXCITITOR-FMT-CYCLONE-01-003 – CycloneDX export serializer + • Current: DONE – Component reconciler issues stable bom-refs, aggregates identifiers, and records diagnostics for missing SBOM linkage. + 2. [DONE 2025-10-29] EXCITITOR-FMT-CYCLONE-01-003 — EXCITITOR-FMT-CYCLONE-01-003 – CycloneDX export serializer • Prereqs: EXCITITOR-EXPORT-01-001 (external/completed), EXCITITOR-FMT-CYCLONE-01-001 (external/completed) - • Current: TODO – Provide exporters producing CycloneDX VEX output with canonical ordering and hash-stable manifests. + • Current: DONE – CycloneDX exporter delivers canonical VEX payloads with reconciled components, per-claim analyses, and metadata for caching. - Path: `src/Excititor/__Libraries/StellaOps.Excititor.Formats.OpenVEX/TASKS.md` - 1. [TODO] EXCITITOR-FMT-OPENVEX-01-002 — EXCITITOR-FMT-OPENVEX-01-002 – Statement merge utilities + 1. [DONE 2025-10-29] EXCITITOR-FMT-OPENVEX-01-002 — EXCITITOR-FMT-OPENVEX-01-002 – Statement merge utilities • Prereqs: EXCITITOR-FMT-OPENVEX-01-001 (external/completed) - • Current: TODO – Add reducers merging multiple OpenVEX statements, resolving conflicts deterministically, and emitting policy diagnostics. - 2. [TODO] EXCITITOR-FMT-OPENVEX-01-003 — EXCITITOR-FMT-OPENVEX-01-003 – OpenVEX export writer + • Current: DONE – Merge utilities combine statements deterministically, highlight conflicts, and preserve source diagnostics for policy checks. + 2. [DONE 2025-10-29] EXCITITOR-FMT-OPENVEX-01-003 — EXCITITOR-FMT-OPENVEX-01-003 – OpenVEX export writer • Prereqs: EXCITITOR-EXPORT-01-001 (external/completed), EXCITITOR-FMT-OPENVEX-01-001 (external/completed) - • Current: TODO – Provide export serializer generating canonical OpenVEX documents with optional SBOM references and hash-stable ordering. + • Current: DONE – OpenVEX exporter serializes merged statements with canonical ordering, provenance metadata, and deterministic digests. - **Sprint 7** · Contextual Truth Foundations - Team: Team Excititor Export @@ -956,7 +956,7 @@ Generated from SPRINTS.md and module TASKS.md files on 2025-10-19. Waves cluster • Prereqs: SCANNER-ANALYZERS-LANG-10-303C (Wave 3) • Current: TODO - Path: `src/Scanner/__Libraries/StellaOps.Scanner.Analyzers.Lang.Rust/TASKS.md` - 1. [TODO] SCANNER-ANALYZERS-LANG-10-307R — Finalize shared helper usage (license, usage flags) and concurrency-safe caches. + 1. [DOING] SCANNER-ANALYZERS-LANG-10-307R — Finalize shared helper usage (license, usage flags) and concurrency-safe caches. • Prereqs: SCANNER-ANALYZERS-LANG-10-306C (Wave 3) • Current: TODO - **Sprint 13** · UX & CLI Experience diff --git a/docs/implplan/SPRINTS.md b/docs/implplan/SPRINTS.md index b553df88..89670078 100644 --- a/docs/implplan/SPRINTS.md +++ b/docs/implplan/SPRINTS.md @@ -603,7 +603,7 @@ This file describe implementation of Stella Ops (docs/README.md). Implementation | Sprint 39 | Java Analyzer Core | src/Scanner/__Libraries/StellaOps.Scanner.Analyzers.Lang.Java/TASKS.md | TODO | Java Analyzer Guild | SCANNER-ANALYZERS-JAVA-21-001 | Java input normalizer (jar/war/ear/fat/jmod/jimage) with MR overlay selection. | | Sprint 39 | Java Analyzer Core | src/Scanner/__Libraries/StellaOps.Scanner.Analyzers.Lang.Java/TASKS.md | TODO | Java Analyzer Guild | SCANNER-ANALYZERS-JAVA-21-002 | Module/classpath builder with duplicate & split-package detection. | | Sprint 39 | Java Analyzer Core | src/Scanner/__Libraries/StellaOps.Scanner.Analyzers.Lang.Java/TASKS.md | TODO | Java Analyzer Guild | SCANNER-ANALYZERS-JAVA-21-003 | SPI scanner & provider selection with warnings. | -| Sprint 39 | Java Analyzer Core | src/Scanner/__Libraries/StellaOps.Scanner.Analyzers.Lang.Java/TASKS.md | TODO | Java Analyzer Guild | SCANNER-ANALYZERS-JAVA-21-004 | Reflection/TCCL heuristics emitting reason-coded edges. | +| Sprint 39 | Java Analyzer Core | src/Scanner/__Libraries/StellaOps.Scanner.Analyzers.Lang.Java/TASKS.md | DONE | Java Analyzer Guild | SCANNER-ANALYZERS-JAVA-21-004 | Reflection/TCCL heuristics emitting reason-coded edges. | | Sprint 39 | Java Analyzer Core | src/Scanner/__Libraries/StellaOps.Scanner.Analyzers.Lang.Java/TASKS.md | TODO | Java Analyzer Guild | SCANNER-ANALYZERS-JAVA-21-005 | Framework config extraction (Spring, Jakarta, MicroProfile, logging, Graal configs). | | Sprint 39 | Java Analyzer Core | src/Scanner/__Libraries/StellaOps.Scanner.Analyzers.Lang.Java/TASKS.md | TODO | Java Analyzer Guild | SCANNER-ANALYZERS-JAVA-21-006 | JNI/native hint detection for Java artifacts. | | Sprint 39 | Java Analyzer Core | src/Scanner/__Libraries/StellaOps.Scanner.Analyzers.Lang.Java/TASKS.md | TODO | Java Analyzer Guild | SCANNER-ANALYZERS-JAVA-21-007 | Manifest/signature metadata collector (main/start/agent classes, signers). | diff --git a/src/Concelier/__Libraries/StellaOps.Concelier.Core/TASKS.md b/src/Concelier/__Libraries/StellaOps.Concelier.Core/TASKS.md index 03bc8c20..e606d555 100644 --- a/src/Concelier/__Libraries/StellaOps.Concelier.Core/TASKS.md +++ b/src/Concelier/__Libraries/StellaOps.Concelier.Core/TASKS.md @@ -12,7 +12,7 @@ | CONCELIER-CORE-AOC-19-003 `Idempotent append-only upsert` | DONE (2025-10-28) | Concelier Core Guild | CONCELIER-STORE-AOC-19-002 | Implement idempotent upsert path using `(vendor, upstreamId, contentHash, tenant)` key, emitting supersedes pointers for new revisions and preventing duplicate inserts. | > 2025-10-28: Advisory raw ingestion now strips client-supplied supersedes hints, logs ignored pointers, and surfaces repository-supplied supersedes identifiers; service tests cover duplicate handling and append-only semantics. > Docs alignment (2025-10-26): Deployment guide + observability guide describe supersedes metrics; ensure implementation emits `aoc_violation_total` on failure. -| CONCELIER-CORE-AOC-19-004 `Remove ingestion normalization` | DOING (2025-10-28) | Concelier Core Guild | CONCELIER-CORE-AOC-19-002, POLICY-AOC-19-003 | Strip normalization/dedup/severity logic from ingestion pipelines, delegate derived computations to Policy Engine, and update exporters/tests to consume raw documents only. | +| CONCELIER-CORE-AOC-19-004 `Remove ingestion normalization` | DOING (2025-10-28) | Concelier Core Guild | CONCELIER-CORE-AOC-19-002, POLICY-AOC-19-003 | Strip normalization/dedup/severity logic from ingestion pipelines, delegate derived computations to Policy Engine, and update exporters/tests to consume raw documents only.
2025-10-29 19:05Z: Audit completed for `AdvisoryRawService`/Mongo repo to confirm alias order/dedup removal persists; identified remaining normalization in observation/linkset factory that will be revised to surface raw duplicates for Policy ingestion. Change sketch + regression matrix drafted under `docs/dev/aoc-normalization-removal-notes.md` (pending commit). | > Docs alignment (2025-10-26): Architecture overview emphasises policy-only derivation; coordinate with Policy Engine guild for rollout. > 2025-10-29: `AdvisoryRawService` now preserves upstream alias/linkset ordering (trim-only) and updated AOC documentation reflects the behaviour; follow-up to ensure policy consumers handle duplicates remains open. | CONCELIER-CORE-AOC-19-013 `Authority tenant scope smoke coverage` | TODO | Concelier Core Guild | AUTH-AOC-19-002 | Extend Concelier smoke/e2e fixtures to configure `requiredTenants` and assert cross-tenant rejection with updated Authority tokens. | Coordinate deliverable so Authority docs (`AUTH-AOC-19-003`) can close once tests are in place. | diff --git a/src/Directory.Build.props b/src/Directory.Build.props index f125abf6..8686dea5 100644 --- a/src/Directory.Build.props +++ b/src/Directory.Build.props @@ -1,49 +1,53 @@ - - - $(SolutionDir)StellaOps.Concelier.PluginBinaries - $(MSBuildThisFileDirectory)StellaOps.Concelier.PluginBinaries - $(SolutionDir)StellaOps.Authority.PluginBinaries - $(MSBuildThisFileDirectory)StellaOps.Authority.PluginBinaries - true - true - true - $(SolutionDir)plugins\notify - $([System.IO.Path]::GetFullPath('$(MSBuildThisFileDirectory)..\plugins\notify\')) - true - false - $([System.IO.Path]::GetFullPath('$(MSBuildThisFileDirectory)..\plugins\scanner\buildx\')) - true - $([System.IO.Path]::GetFullPath('$(MSBuildThisFileDirectory)..\plugins\scanner\analyzers\os\')) - true - $([System.IO.Path]::GetFullPath('$(MSBuildThisFileDirectory)..\plugins\scanner\analyzers\lang\')) - true - true - - - - - false - runtime - - - - - - - - - - - - - - - - - - - - - - - + + + $(SolutionDir)StellaOps.Concelier.PluginBinaries + $(MSBuildThisFileDirectory)StellaOps.Concelier.PluginBinaries + $(SolutionDir)StellaOps.Authority.PluginBinaries + $(MSBuildThisFileDirectory)StellaOps.Authority.PluginBinaries + true + true + true + $(SolutionDir)plugins\notify + $([System.IO.Path]::GetFullPath('$(MSBuildThisFileDirectory)..\plugins\notify\')) + true + false + $([System.IO.Path]::GetFullPath('$(MSBuildThisFileDirectory)..\plugins\scanner\buildx\')) + true + $([System.IO.Path]::GetFullPath('$(MSBuildThisFileDirectory)..\plugins\scanner\analyzers\os\')) + true + $([System.IO.Path]::GetFullPath('$(MSBuildThisFileDirectory)..\plugins\scanner\analyzers\lang\')) + true + true + $(MSBuildThisFileDirectory)StellaOps.Concelier.Testing\ + $(MSBuildThisFileDirectory)Concelier\__Libraries\StellaOps.Concelier.Testing\ + $(MSBuildThisFileDirectory)StellaOps.Concelier.Tests.Shared\ + $(MSBuildThisFileDirectory)Concelier\StellaOps.Concelier.Tests.Shared\ + + + + + false + runtime + + + + + + + + + + + + + + + + + + + + + + + diff --git a/src/Excititor/StellaOps.Excititor.WebService/Endpoints/MirrorEndpoints.cs b/src/Excititor/StellaOps.Excititor.WebService/Endpoints/MirrorEndpoints.cs index 85b51693..a3c7df6d 100644 --- a/src/Excititor/StellaOps.Excititor.WebService/Endpoints/MirrorEndpoints.cs +++ b/src/Excititor/StellaOps.Excititor.WebService/Endpoints/MirrorEndpoints.cs @@ -292,28 +292,30 @@ internal static class MirrorEndpoints return export is not null; } - private static string ResolveContentType(VexExportFormat format) - => format switch - { - VexExportFormat.Json => "application/json", - VexExportFormat.JsonLines => "application/jsonl", - VexExportFormat.OpenVex => "application/json", - VexExportFormat.Csaf => "application/json", - _ => "application/octet-stream", - }; + private static string ResolveContentType(VexExportFormat format) + => format switch + { + VexExportFormat.Json => "application/json", + VexExportFormat.JsonLines => "application/jsonl", + VexExportFormat.OpenVex => "application/json", + VexExportFormat.Csaf => "application/json", + VexExportFormat.CycloneDx => "application/json", + _ => "application/octet-stream", + }; private static string BuildDownloadFileName(string domainId, string exportKey, VexExportFormat format) { var builder = new StringBuilder(domainId.Length + exportKey.Length + 8); builder.Append(domainId).Append('-').Append(exportKey); - builder.Append(format switch - { - VexExportFormat.Json => ".json", - VexExportFormat.JsonLines => ".jsonl", - VexExportFormat.OpenVex => ".openvex.json", - VexExportFormat.Csaf => ".csaf.json", - _ => ".bin", - }); + builder.Append(format switch + { + VexExportFormat.Json => ".json", + VexExportFormat.JsonLines => ".jsonl", + VexExportFormat.OpenVex => ".openvex.json", + VexExportFormat.Csaf => ".csaf.json", + VexExportFormat.CycloneDx => ".cyclonedx.json", + _ => ".bin", + }); return builder.ToString(); } diff --git a/src/Excititor/__Libraries/StellaOps.Excititor.Connectors.Cisco.CSAF/CiscoCsafConnector.cs b/src/Excititor/__Libraries/StellaOps.Excititor.Connectors.Cisco.CSAF/CiscoCsafConnector.cs index 063e2475..e30ce04e 100644 --- a/src/Excititor/__Libraries/StellaOps.Excititor.Connectors.Cisco.CSAF/CiscoCsafConnector.cs +++ b/src/Excititor/__Libraries/StellaOps.Excititor.Connectors.Cisco.CSAF/CiscoCsafConnector.cs @@ -1,15 +1,17 @@ -using System.Collections.Generic; -using System.Collections.Immutable; -using System.Linq; -using System.Net.Http; -using System.Runtime.CompilerServices; -using System.Text.Json; -using Microsoft.Extensions.Logging; -using StellaOps.Excititor.Connectors.Abstractions; -using StellaOps.Excititor.Connectors.Cisco.CSAF.Configuration; -using StellaOps.Excititor.Connectors.Cisco.CSAF.Metadata; -using StellaOps.Excititor.Core; -using StellaOps.Excititor.Storage.Mongo; +using System.Collections.Generic; +using System.Collections.Immutable; +using System.Globalization; +using System.Linq; +using System.Net.Http; +using System.Runtime.CompilerServices; +using System.Text.Json; +using Microsoft.Extensions.DependencyInjection; +using Microsoft.Extensions.Logging; +using StellaOps.Excititor.Connectors.Abstractions; +using StellaOps.Excititor.Connectors.Cisco.CSAF.Configuration; +using StellaOps.Excititor.Connectors.Cisco.CSAF.Metadata; +using StellaOps.Excititor.Core; +using StellaOps.Excititor.Storage.Mongo; namespace StellaOps.Excititor.Connectors.Cisco.CSAF; @@ -71,12 +73,14 @@ public sealed class CiscoCsafConnector : VexConnectorBase throw new InvalidOperationException("Connector must be validated before fetch operations."); } - if (_providerMetadata is null) - { - _providerMetadata = await _metadataLoader.LoadAsync(cancellationToken).ConfigureAwait(false); - } - - var state = await _stateRepository.GetAsync(Descriptor.Id, cancellationToken).ConfigureAwait(false); + if (_providerMetadata is null) + { + _providerMetadata = await _metadataLoader.LoadAsync(cancellationToken).ConfigureAwait(false); + } + + await UpsertProviderAsync(context.Services, _providerMetadata.Provider, cancellationToken).ConfigureAwait(false); + + var state = await _stateRepository.GetAsync(Descriptor.Id, cancellationToken).ConfigureAwait(false); var knownDigests = state?.DocumentDigests ?? ImmutableArray.Empty; var digestSet = new HashSet(knownDigests, StringComparer.OrdinalIgnoreCase); var digestList = new List(knownDigests); @@ -99,16 +103,23 @@ public sealed class CiscoCsafConnector : VexConnectorBase contentResponse.EnsureSuccessStatusCode(); var payload = await contentResponse.Content.ReadAsByteArrayAsync(cancellationToken).ConfigureAwait(false); - var rawDocument = CreateRawDocument( - VexDocumentFormat.Csaf, - advisory.DocumentUri, - payload, - BuildMetadata(builder => builder - .Add("cisco.csaf.advisoryId", advisory.Id) - .Add("cisco.csaf.revision", advisory.Revision) - .Add("cisco.csaf.published", advisory.Published?.ToString("O")) - .Add("cisco.csaf.modified", advisory.LastModified?.ToString("O")) - .Add("cisco.csaf.sha256", advisory.Sha256))); + var metadata = BuildMetadata(builder => + { + builder + .Add("cisco.csaf.advisoryId", advisory.Id) + .Add("cisco.csaf.revision", advisory.Revision) + .Add("cisco.csaf.published", advisory.Published?.ToString("O")) + .Add("cisco.csaf.modified", advisory.LastModified?.ToString("O")) + .Add("cisco.csaf.sha256", advisory.Sha256); + + AddProvenanceMetadata(builder); + }); + + var rawDocument = CreateRawDocument( + VexDocumentFormat.Csaf, + advisory.DocumentUri, + payload, + metadata); if (!digestSet.Add(rawDocument.Digest)) { @@ -221,19 +232,59 @@ public sealed class CiscoCsafConnector : VexConnectorBase return new Uri(new Uri(baseTextRelative, UriKind.Absolute), relative); } - private static Uri? ResolveNextUri(Uri directory, string? next) - { - if (string.IsNullOrWhiteSpace(next)) - { - return null; - } - - return BuildIndexUri(directory, next); - } - - private sealed record CiscoAdvisoryIndex - { - public List? Advisories { get; init; } + private static Uri? ResolveNextUri(Uri directory, string? next) + { + if (string.IsNullOrWhiteSpace(next)) + { + return null; + } + + return BuildIndexUri(directory, next); + } + + private void AddProvenanceMetadata(VexConnectorMetadataBuilder builder) + { + if (_providerMetadata is null) + { + return; + } + + var provider = _providerMetadata.Provider; + builder.Add("vex.provenance.provider", provider.Id); + builder.Add("vex.provenance.providerName", provider.DisplayName); + builder.Add("vex.provenance.trust.weight", provider.Trust.Weight.ToString("0.###", CultureInfo.InvariantCulture)); + + if (provider.Trust.Cosign is { } cosign) + { + builder.Add("vex.provenance.cosign.issuer", cosign.Issuer); + builder.Add("vex.provenance.cosign.identityPattern", cosign.IdentityPattern); + } + + if (!provider.Trust.PgpFingerprints.IsDefaultOrEmpty && provider.Trust.PgpFingerprints.Length > 0) + { + builder.Add("vex.provenance.pgp.fingerprints", string.Join(",", provider.Trust.PgpFingerprints)); + } + } + + private static async ValueTask UpsertProviderAsync(IServiceProvider services, VexProvider provider, CancellationToken cancellationToken) + { + if (services is null) + { + return; + } + + var store = services.GetService(); + if (store is null) + { + return; + } + + await store.SaveAsync(provider, cancellationToken).ConfigureAwait(false); + } + + private sealed record CiscoAdvisoryIndex + { + public List? Advisories { get; init; } public string? Next { get; init; } } diff --git a/src/Excititor/__Libraries/StellaOps.Excititor.Connectors.Cisco.CSAF/TASKS.md b/src/Excititor/__Libraries/StellaOps.Excititor.Connectors.Cisco.CSAF/TASKS.md index b02f1afb..f4f02071 100644 --- a/src/Excititor/__Libraries/StellaOps.Excititor.Connectors.Cisco.CSAF/TASKS.md +++ b/src/Excititor/__Libraries/StellaOps.Excititor.Connectors.Cisco.CSAF/TASKS.md @@ -4,4 +4,4 @@ If you are working on this file you need to read docs/modules/excititor/ARCHITEC |---|---|---|---| |EXCITITOR-CONN-CISCO-01-001 – Endpoint discovery & auth plumbing|Team Excititor Connectors – Cisco|EXCITITOR-CONN-ABS-01-001|**DONE (2025-10-17)** – Added `CiscoProviderMetadataLoader` with bearer token support, offline snapshot fallback, DI helpers, and tests covering network/offline discovery to unblock subsequent fetch work.| |EXCITITOR-CONN-CISCO-01-002 – CSAF pull loop & pagination|Team Excititor Connectors – Cisco|EXCITITOR-CONN-CISCO-01-001, EXCITITOR-STORAGE-01-003|**DONE (2025-10-17)** – Implemented paginated advisory fetch using provider directories, raw document persistence with dedupe/state tracking, offline resiliency, and unit coverage.| -|EXCITITOR-CONN-CISCO-01-003 – Provider trust metadata|Team Excititor Connectors – Cisco|EXCITITOR-CONN-CISCO-01-002, EXCITITOR-POLICY-01-001|**DOING (2025-10-19)** – Prereqs confirmed (both DONE); implementing cosign/PGP trust metadata emission and advisory provenance hints for policy weighting.| +|EXCITITOR-CONN-CISCO-01-003 – Provider trust metadata|Team Excititor Connectors – Cisco|EXCITITOR-CONN-CISCO-01-002, EXCITITOR-POLICY-01-001|**DONE (2025-10-29)** – Connector now annotates raw documents with provider trust + cosign/PGP provenance and upserts `VexProvider` entries; new unit coverage asserts metadata emission and provider-store invocation.| diff --git a/src/Excititor/__Libraries/StellaOps.Excititor.Connectors.MSRC.CSAF/TASKS.md b/src/Excititor/__Libraries/StellaOps.Excititor.Connectors.MSRC.CSAF/TASKS.md index 7f7b70f3..9e2d5d9d 100644 --- a/src/Excititor/__Libraries/StellaOps.Excititor.Connectors.MSRC.CSAF/TASKS.md +++ b/src/Excititor/__Libraries/StellaOps.Excititor.Connectors.MSRC.CSAF/TASKS.md @@ -3,5 +3,5 @@ If you are working on this file you need to read docs/modules/excititor/ARCHITEC | Task | Owner(s) | Depends on | Notes | |---|---|---|---| |EXCITITOR-CONN-MS-01-001 – AAD onboarding & token cache|Team Excititor Connectors – MSRC|EXCITITOR-CONN-ABS-01-001|**DONE (2025-10-17)** – Added MSRC connector project with configurable AAD options, token provider (offline/online modes), DI wiring, and unit tests covering caching and fallback scenarios.| -|EXCITITOR-CONN-MS-01-002 – CSAF download pipeline|Team Excititor Connectors – MSRC|EXCITITOR-CONN-MS-01-001, EXCITITOR-STORAGE-01-003|**DOING (2025-10-19)** – Prereqs verified (EXCITITOR-CONN-MS-01-001, EXCITITOR-STORAGE-01-003); drafting fetch/retry plan and storage wiring before implementation of CSAF package download, checksum validation, and quarantine flows.| +|EXCITITOR-CONN-MS-01-002 – CSAF download pipeline|Team Excititor Connectors – MSRC|EXCITITOR-CONN-MS-01-001, EXCITITOR-STORAGE-01-003|**DONE (2025-10-29)** – Implemented authenticated CSAF retrieval with retry/backoff, checksum enforcement, quarantine for invalid archives, and regression tests covering dedupe + idempotent state updates.| |EXCITITOR-CONN-MS-01-003 – Trust metadata & provenance hints|Team Excititor Connectors – MSRC|EXCITITOR-CONN-MS-01-002, EXCITITOR-POLICY-01-001|TODO – Emit cosign/AAD issuer metadata, attach provenance details, and document policy integration.| diff --git a/src/Excititor/__Libraries/StellaOps.Excititor.Connectors.SUSE.RancherVEXHub/DependencyInjection/RancherHubConnectorServiceCollectionExtensions.cs b/src/Excititor/__Libraries/StellaOps.Excititor.Connectors.SUSE.RancherVEXHub/DependencyInjection/RancherHubConnectorServiceCollectionExtensions.cs index ca6289e3..cfc3e43c 100644 --- a/src/Excititor/__Libraries/StellaOps.Excititor.Connectors.SUSE.RancherVEXHub/DependencyInjection/RancherHubConnectorServiceCollectionExtensions.cs +++ b/src/Excititor/__Libraries/StellaOps.Excititor.Connectors.SUSE.RancherVEXHub/DependencyInjection/RancherHubConnectorServiceCollectionExtensions.cs @@ -2,12 +2,14 @@ using System; using System.Net; using System.Net.Http; using Microsoft.Extensions.Caching.Memory; -using Microsoft.Extensions.DependencyInjection; -using Microsoft.Extensions.DependencyInjection.Extensions; -using StellaOps.Excititor.Connectors.Abstractions; -using StellaOps.Excititor.Connectors.SUSE.RancherVEXHub.Authentication; -using StellaOps.Excititor.Connectors.SUSE.RancherVEXHub.Configuration; -using StellaOps.Excititor.Connectors.SUSE.RancherVEXHub.Metadata; +using Microsoft.Extensions.DependencyInjection; +using Microsoft.Extensions.DependencyInjection.Extensions; +using StellaOps.Excititor.Connectors.Abstractions; +using StellaOps.Excititor.Connectors.SUSE.RancherVEXHub.Authentication; +using StellaOps.Excititor.Connectors.SUSE.RancherVEXHub.Configuration; +using StellaOps.Excititor.Connectors.SUSE.RancherVEXHub.Events; +using StellaOps.Excititor.Connectors.SUSE.RancherVEXHub.Metadata; +using StellaOps.Excititor.Connectors.SUSE.RancherVEXHub.State; using StellaOps.Excititor.Core; using System.IO.Abstractions; @@ -28,10 +30,12 @@ public static class RancherHubConnectorServiceCollectionExtensions configure?.Invoke(options); }); - services.AddSingleton, RancherHubConnectorOptionsValidator>(); - services.AddSingleton(); - services.AddSingleton(); - services.AddSingleton(); + services.AddSingleton, RancherHubConnectorOptionsValidator>(); + services.AddSingleton(); + services.AddSingleton(); + services.AddSingleton(); + services.AddSingleton(); + services.AddSingleton(); services.AddHttpClient(RancherHubConnectorOptions.HttpClientName, client => { diff --git a/src/Excititor/__Libraries/StellaOps.Excititor.Connectors.SUSE.RancherVEXHub/RancherHubConnector.cs b/src/Excititor/__Libraries/StellaOps.Excititor.Connectors.SUSE.RancherVEXHub/RancherHubConnector.cs index 97689785..bea3541c 100644 --- a/src/Excititor/__Libraries/StellaOps.Excititor.Connectors.SUSE.RancherVEXHub/RancherHubConnector.cs +++ b/src/Excititor/__Libraries/StellaOps.Excititor.Connectors.SUSE.RancherVEXHub/RancherHubConnector.cs @@ -22,6 +22,8 @@ namespace StellaOps.Excititor.Connectors.SUSE.RancherVEXHub; public sealed class RancherHubConnector : VexConnectorBase { + private const int MaxDigestHistory = 200; + private static readonly VexConnectorDescriptor StaticDescriptor = new( id: "excititor:suse.rancher", kind: VexProviderKind.Hub, @@ -153,6 +155,12 @@ public sealed class RancherHubConnector : VexConnectorBase } } + var trimmed = TrimHistory(digestHistory); + if (trimmed) + { + stateChanged = true; + } + if (stateChanged || !string.Equals(latestCursor, checkpoint.Cursor, StringComparison.Ordinal) || latestPublishedAt != checkpoint.LastPublishedAt) { await _checkpointManager.SaveAsync( @@ -236,6 +244,18 @@ public sealed class RancherHubConnector : VexConnectorBase return new EventProcessingResult(document, false, publishedAt); } + private static bool TrimHistory(List digestHistory) + { + if (digestHistory.Count <= MaxDigestHistory) + { + return false; + } + + var excess = digestHistory.Count - MaxDigestHistory; + digestHistory.RemoveRange(0, excess); + return true; + } + private async Task CreateDocumentRequestAsync(Uri documentUri, CancellationToken cancellationToken) { var request = new HttpRequestMessage(HttpMethod.Get, documentUri); diff --git a/src/Excititor/__Libraries/StellaOps.Excititor.Connectors.SUSE.RancherVEXHub/TASKS.md b/src/Excititor/__Libraries/StellaOps.Excititor.Connectors.SUSE.RancherVEXHub/TASKS.md index 8069143b..04377fdc 100644 --- a/src/Excititor/__Libraries/StellaOps.Excititor.Connectors.SUSE.RancherVEXHub/TASKS.md +++ b/src/Excititor/__Libraries/StellaOps.Excititor.Connectors.SUSE.RancherVEXHub/TASKS.md @@ -3,5 +3,5 @@ If you are working on this file you need to read docs/modules/excititor/ARCHITEC | Task | Owner(s) | Depends on | Notes | |---|---|---|---| |EXCITITOR-CONN-SUSE-01-001 – Rancher hub discovery & auth|Team Excititor Connectors – SUSE|EXCITITOR-CONN-ABS-01-001|**DONE (2025-10-17)** – Added Rancher hub options/token provider, discovery metadata loader with offline snapshots + caching, connector shell, DI wiring, and unit tests covering network/offline paths.| -|EXCITITOR-CONN-SUSE-01-002 – Checkpointed event ingestion|Team Excititor Connectors – SUSE|EXCITITOR-CONN-SUSE-01-001, EXCITITOR-STORAGE-01-003|**DOING (2025-10-19)** – Process hub events with resume checkpoints, deduplication, and quarantine path for malformed payloads.
2025-10-19: Prereqs EXCITITOR-CONN-SUSE-01-001 & EXCITITOR-STORAGE-01-003 confirmed complete; initiating checkpoint/resume implementation plan.| +|EXCITITOR-CONN-SUSE-01-002 – Checkpointed event ingestion|Team Excititor Connectors – SUSE|EXCITITOR-CONN-SUSE-01-001, EXCITITOR-STORAGE-01-003|**DONE (2025-10-29)** – Wired checkpoint manager/event client into DI, bounded digest history, and exercised offline snapshot/dedup/quarantine flows with new connector tests ensuring state persistence and replay determinism.| |EXCITITOR-CONN-SUSE-01-003 – Trust metadata & policy hints|Team Excititor Connectors – SUSE|EXCITITOR-CONN-SUSE-01-002, EXCITITOR-POLICY-01-001|TODO – Emit provider trust configuration (signers, weight overrides) and attach provenance hints for consensus engine.| diff --git a/src/Excititor/__Libraries/StellaOps.Excititor.Connectors.Ubuntu.CSAF/TASKS.md b/src/Excititor/__Libraries/StellaOps.Excititor.Connectors.Ubuntu.CSAF/TASKS.md index 3c6c4469..35584fb1 100644 --- a/src/Excititor/__Libraries/StellaOps.Excititor.Connectors.Ubuntu.CSAF/TASKS.md +++ b/src/Excititor/__Libraries/StellaOps.Excititor.Connectors.Ubuntu.CSAF/TASKS.md @@ -3,6 +3,6 @@ If you are working on this file you need to read docs/modules/excititor/ARCHITEC | Task | Owner(s) | Depends on | Notes | |---|---|---|---| |EXCITITOR-CONN-UBUNTU-01-001 – Ubuntu CSAF discovery & channels|Team Excititor Connectors – Ubuntu|EXCITITOR-CONN-ABS-01-001|**DONE (2025-10-17)** – Added Ubuntu connector project with configurable channel options, catalog loader (network/offline), DI wiring, and discovery unit tests.| -|EXCITITOR-CONN-UBUNTU-01-002 – Incremental fetch & deduplication|Team Excititor Connectors – Ubuntu|EXCITITOR-CONN-UBUNTU-01-001, EXCITITOR-STORAGE-01-003|**DOING (2025-10-19)** – Fetch CSAF bundles with ETag handling, checksum validation, deduplication, and raw persistence.| +|EXCITITOR-CONN-UBUNTU-01-002 – Incremental fetch & deduplication|Team Excititor Connectors – Ubuntu|EXCITITOR-CONN-UBUNTU-01-001, EXCITITOR-STORAGE-01-003|**DONE (2025-10-29)** – Incremental pull loop now enforces ETag/sha validation, resumes from persisted state, and includes regression tests covering checksum mismatch quarantine and If-None-Match replay.| |EXCITITOR-CONN-UBUNTU-01-003 – Trust metadata & provenance|Team Excititor Connectors – Ubuntu|EXCITITOR-CONN-UBUNTU-01-002, EXCITITOR-POLICY-01-001|TODO – Emit Ubuntu signing metadata (GPG fingerprints) plus provenance hints for policy weighting and diagnostics.| -> Remark (2025-10-19, EXCITITOR-CONN-UBUNTU-01-002): Prerequisites EXCITITOR-CONN-UBUNTU-01-001 and EXCITITOR-STORAGE-01-003 verified as **DONE**; advancing to DOING per Wave 0 kickoff. +> Remark (2025-10-29, EXCITITOR-CONN-UBUNTU-01-002): Offline + network regression pass validated resume tokens, dedupe skips, checksum enforcement, and ETag handling before closing the task. diff --git a/src/Excititor/__Libraries/StellaOps.Excititor.Core/VexCanonicalJsonSerializer.cs b/src/Excititor/__Libraries/StellaOps.Excititor.Core/VexCanonicalJsonSerializer.cs index 86cd3043..c604c24d 100644 --- a/src/Excititor/__Libraries/StellaOps.Excititor.Core/VexCanonicalJsonSerializer.cs +++ b/src/Excititor/__Libraries/StellaOps.Excititor.Core/VexCanonicalJsonSerializer.cs @@ -397,10 +397,10 @@ public static class VexCanonicalJsonSerializer new[] { "isValid", - "diagnostics", - } - }, - }; + "diagnostics", + } + }, + }; public static string Serialize(T value) => JsonSerializer.Serialize(value, CompactOptions); diff --git a/src/Excititor/__Libraries/StellaOps.Excititor.Core/VexExportManifest.cs b/src/Excititor/__Libraries/StellaOps.Excititor.Core/VexExportManifest.cs index 22598cae..6e4b7012 100644 --- a/src/Excititor/__Libraries/StellaOps.Excititor.Core/VexExportManifest.cs +++ b/src/Excititor/__Libraries/StellaOps.Excititor.Core/VexExportManifest.cs @@ -274,18 +274,21 @@ public sealed partial record VexQuerySignature public override string ToString() => Value; } -[DataContract] -public enum VexExportFormat -{ - [EnumMember(Value = "json")] - Json, - - [EnumMember(Value = "jsonl")] - JsonLines, - - [EnumMember(Value = "openvex")] - OpenVex, - - [EnumMember(Value = "csaf")] - Csaf, -} +[DataContract] +public enum VexExportFormat +{ + [EnumMember(Value = "json")] + Json, + + [EnumMember(Value = "jsonl")] + JsonLines, + + [EnumMember(Value = "openvex")] + OpenVex, + + [EnumMember(Value = "csaf")] + Csaf, + + [EnumMember(Value = "cyclonedx")] + CycloneDx, +} diff --git a/src/Excititor/__Libraries/StellaOps.Excititor.Export/FileSystemArtifactStore.cs b/src/Excititor/__Libraries/StellaOps.Excititor.Export/FileSystemArtifactStore.cs index a82f0226..488259bf 100644 --- a/src/Excititor/__Libraries/StellaOps.Excititor.Export/FileSystemArtifactStore.cs +++ b/src/Excititor/__Libraries/StellaOps.Excititor.Export/FileSystemArtifactStore.cs @@ -133,16 +133,17 @@ public sealed class FileSystemArtifactStore : IVexArtifactStore return null; } - private static string GetExtension(VexExportFormat format) - => format switch - { - VexExportFormat.Json => ".json", - VexExportFormat.JsonLines => ".jsonl", - VexExportFormat.OpenVex => ".json", - VexExportFormat.Csaf => ".json", - _ => ".bin", - }; -} + private static string GetExtension(VexExportFormat format) + => format switch + { + VexExportFormat.Json => ".json", + VexExportFormat.JsonLines => ".jsonl", + VexExportFormat.OpenVex => ".json", + VexExportFormat.Csaf => ".json", + VexExportFormat.CycloneDx => ".json", + _ => ".bin", + }; +} public static class FileSystemArtifactStoreServiceCollectionExtensions { diff --git a/src/Excititor/__Libraries/StellaOps.Excititor.Export/OfflineBundleArtifactStore.cs b/src/Excititor/__Libraries/StellaOps.Excititor.Export/OfflineBundleArtifactStore.cs index 088f85b5..6bffa6cd 100644 --- a/src/Excititor/__Libraries/StellaOps.Excititor.Export/OfflineBundleArtifactStore.cs +++ b/src/Excititor/__Libraries/StellaOps.Excititor.Export/OfflineBundleArtifactStore.cs @@ -213,16 +213,17 @@ public sealed class OfflineBundleArtifactStore : IVexArtifactStore private string GetBundlesRoot() => _fileSystem.Path.Combine(_options.RootPath, _options.BundlesFolder); - private static string GetExtension(VexExportFormat format) - => format switch - { - VexExportFormat.Json => ".json", - VexExportFormat.JsonLines => ".jsonl", - VexExportFormat.OpenVex => ".json", - VexExportFormat.Csaf => ".json", - _ => ".bin", - }; - + private static string GetExtension(VexExportFormat format) + => format switch + { + VexExportFormat.Json => ".json", + VexExportFormat.JsonLines => ".jsonl", + VexExportFormat.OpenVex => ".json", + VexExportFormat.Csaf => ".json", + VexExportFormat.CycloneDx => ".json", + _ => ".bin", + }; + private sealed record ManifestDocument(ImmutableArray Artifacts); private sealed record ManifestEntry(string Digest, string Format, string Path, long SizeBytes, IReadOnlyDictionary Metadata); diff --git a/src/Excititor/__Libraries/StellaOps.Excititor.Export/S3ArtifactStore.cs b/src/Excititor/__Libraries/StellaOps.Excititor.Export/S3ArtifactStore.cs index 5cc84280..ebeb3cf8 100644 --- a/src/Excititor/__Libraries/StellaOps.Excititor.Export/S3ArtifactStore.cs +++ b/src/Excititor/__Libraries/StellaOps.Excititor.Export/S3ArtifactStore.cs @@ -139,16 +139,17 @@ public sealed class S3ArtifactStore : IVexArtifactStore var metadata = new Dictionary(StringComparer.OrdinalIgnoreCase) { ["vex-format"] = artifact.Format.ToString().ToLowerInvariant(), - ["vex-digest"] = artifact.ContentAddress.ToUri(), - ["content-type"] = artifact.Format switch - { - VexExportFormat.Json => "application/json", - VexExportFormat.JsonLines => "application/json", - VexExportFormat.OpenVex => "application/vnd.openvex+json", - VexExportFormat.Csaf => "application/json", - _ => "application/octet-stream", - }, - }; + ["vex-digest"] = artifact.ContentAddress.ToUri(), + ["content-type"] = artifact.Format switch + { + VexExportFormat.Json => "application/json", + VexExportFormat.JsonLines => "application/json", + VexExportFormat.OpenVex => "application/vnd.openvex+json", + VexExportFormat.Csaf => "application/json", + VexExportFormat.CycloneDx => "application/vnd.cyclonedx+json", + _ => "application/octet-stream", + }, + }; foreach (var kvp in artifact.Metadata) { @@ -158,16 +159,17 @@ public sealed class S3ArtifactStore : IVexArtifactStore return metadata; } - private static string GetExtension(VexExportFormat format) - => format switch - { - VexExportFormat.Json => ".json", - VexExportFormat.JsonLines => ".jsonl", - VexExportFormat.OpenVex => ".json", - VexExportFormat.Csaf => ".json", - _ => ".bin", - }; -} + private static string GetExtension(VexExportFormat format) + => format switch + { + VexExportFormat.Json => ".json", + VexExportFormat.JsonLines => ".jsonl", + VexExportFormat.OpenVex => ".json", + VexExportFormat.Csaf => ".json", + VexExportFormat.CycloneDx => ".json", + _ => ".bin", + }; +} public static class S3ArtifactStoreServiceCollectionExtensions { diff --git a/src/Excititor/__Libraries/StellaOps.Excititor.Formats.CSAF/CsafExporter.cs b/src/Excititor/__Libraries/StellaOps.Excititor.Formats.CSAF/CsafExporter.cs new file mode 100644 index 00000000..17d25840 --- /dev/null +++ b/src/Excititor/__Libraries/StellaOps.Excititor.Formats.CSAF/CsafExporter.cs @@ -0,0 +1,512 @@ +using System.Collections.Generic; +using System.Collections.Immutable; +using System.Globalization; +using System.IO; +using System.Linq; +using System.Security.Cryptography; +using System.Text; +using System.Text.Json.Serialization; +using System.Threading; +using System.Threading.Tasks; +using StellaOps.Excititor.Core; + +namespace StellaOps.Excititor.Formats.CSAF; + +/// +/// Emits deterministic CSAF 2.0 VEX documents summarising normalized claims. +/// +public sealed class CsafExporter : IVexExporter +{ + public CsafExporter() + { + } + + public VexExportFormat Format => VexExportFormat.Csaf; + + public VexContentAddress Digest(VexExportRequest request) + { + ArgumentNullException.ThrowIfNull(request); + var document = BuildDocument(request, out _); + var json = VexCanonicalJsonSerializer.Serialize(document); + return ComputeDigest(json); + } + + public async ValueTask SerializeAsync( + VexExportRequest request, + Stream output, + CancellationToken cancellationToken) + { + ArgumentNullException.ThrowIfNull(request); + ArgumentNullException.ThrowIfNull(output); + + var document = BuildDocument(request, out var metadata); + var json = VexCanonicalJsonSerializer.Serialize(document); + var digest = ComputeDigest(json); + var buffer = Encoding.UTF8.GetBytes(json); + await output.WriteAsync(buffer, 0, buffer.Length, cancellationToken).ConfigureAwait(false); + return new VexExportResult(digest, buffer.LongLength, metadata); + } + + private CsafExportDocument BuildDocument(VexExportRequest request, out ImmutableDictionary metadata) + { + var signature = VexQuerySignature.FromQuery(request.Query); + var signatureHash = signature.ComputeHash(); + var generatedAt = request.GeneratedAt.UtcDateTime.ToString("O", CultureInfo.InvariantCulture); + + var productCatalog = new ProductCatalog(); + var missingJustifications = new SortedSet(StringComparer.Ordinal); + + var vulnerabilityBuilders = new Dictionary(StringComparer.Ordinal); + + foreach (var claim in request.Claims) + { + var productId = productCatalog.GetOrAddProductId(claim.Product); + + if (!vulnerabilityBuilders.TryGetValue(claim.VulnerabilityId, out var builder)) + { + builder = new CsafVulnerabilityBuilder(claim.VulnerabilityId); + vulnerabilityBuilders[claim.VulnerabilityId] = builder; + } + + builder.AddClaim(claim, productId); + + if (claim.Status == VexClaimStatus.NotAffected && claim.Justification is null) + { + missingJustifications.Add(FormattableString.Invariant($"{claim.VulnerabilityId}:{productId}")); + } + } + + var products = productCatalog.Build(); + var vulnerabilities = vulnerabilityBuilders.Values + .Select(builder => builder.ToVulnerability()) + .Where(static vulnerability => vulnerability is not null) + .Select(static vulnerability => vulnerability!) + .OrderBy(static vulnerability => vulnerability.Cve ?? vulnerability.Id ?? string.Empty, StringComparer.Ordinal) + .ToImmutableArray(); + + var sourceProviders = request.Claims + .Select(static claim => claim.ProviderId) + .Distinct(StringComparer.Ordinal) + .OrderBy(static provider => provider, StringComparer.Ordinal) + .ToImmutableArray(); + + var documentSection = new CsafDocumentSection( + Category: "vex", + Title: "StellaOps VEX CSAF Export", + Tracking: new CsafTrackingSection( + Id: FormattableString.Invariant($"stellaops:csaf:{signatureHash.Digest}"), + Status: "final", + Version: "1", + Revision: "1", + InitialReleaseDate: generatedAt, + CurrentReleaseDate: generatedAt, + Generator: new CsafGeneratorSection("StellaOps Excititor")), + Publisher: new CsafPublisherSection("StellaOps Excititor", "coordinator")); + + var metadataSection = new CsafExportMetadata( + generatedAt, + signature.Value, + sourceProviders, + missingJustifications.Count == 0 + ? ImmutableDictionary.Empty + : ImmutableDictionary.Empty.Add( + "policy.justification_missing", + string.Join(",", missingJustifications))); + + metadata = BuildMetadata(signature, vulnerabilities.Length, products.Length, missingJustifications, sourceProviders, generatedAt); + + var productTree = new CsafProductTreeSection(products); + return new CsafExportDocument(documentSection, productTree, vulnerabilities, metadataSection); + } + + private static ImmutableDictionary BuildMetadata( + VexQuerySignature signature, + int vulnerabilityCount, + int productCount, + IEnumerable missingJustifications, + ImmutableArray sourceProviders, + string generatedAt) + { + var builder = ImmutableDictionary.CreateBuilder(StringComparer.Ordinal); + builder["csaf.querySignature"] = signature.Value; + builder["csaf.generatedAt"] = generatedAt; + builder["csaf.vulnerabilityCount"] = vulnerabilityCount.ToString(CultureInfo.InvariantCulture); + builder["csaf.productCount"] = productCount.ToString(CultureInfo.InvariantCulture); + builder["csaf.providerCount"] = sourceProviders.Length.ToString(CultureInfo.InvariantCulture); + + var missing = missingJustifications.ToArray(); + if (missing.Length > 0) + { + builder["policy.justification_missing"] = string.Join(",", missing); + } + + return builder.ToImmutable(); + } + + private static VexContentAddress ComputeDigest(string json) + { + var bytes = Encoding.UTF8.GetBytes(json); + Span hash = stackalloc byte[SHA256.HashSizeInBytes]; + SHA256.HashData(bytes, hash); + var digest = Convert.ToHexString(hash).ToLowerInvariant(); + return new VexContentAddress("sha256", digest); + } + + private sealed class ProductCatalog + { + private readonly Dictionary _products = new(StringComparer.Ordinal); + private readonly HashSet _usedIds = new(StringComparer.Ordinal); + + public string GetOrAddProductId(VexProduct product) + { + if (_products.TryGetValue(product.Key, out var existing)) + { + existing.Update(product); + return existing.ProductId; + } + + var productId = GenerateProductId(product.Key); + var mutable = new MutableProduct(productId); + mutable.Update(product); + _products[product.Key] = mutable; + return productId; + } + + public ImmutableArray Build() + => _products.Values + .Select(static product => product.ToEntry()) + .OrderBy(static entry => entry.ProductId, StringComparer.Ordinal) + .ToImmutableArray(); + + private string GenerateProductId(string key) + { + var sanitized = SanitizeIdentifier(key); + if (_usedIds.Add(sanitized)) + { + return sanitized; + } + + var hash = ComputeShortHash(key); + var candidate = FormattableString.Invariant($"{sanitized}-{hash}"); + while (!_usedIds.Add(candidate)) + { + candidate = FormattableString.Invariant($"{candidate}-{hash}"); + } + + return candidate; + } + + private static string SanitizeIdentifier(string value) + { + if (string.IsNullOrWhiteSpace(value)) + { + return "product"; + } + + var builder = new StringBuilder(value.Length); + foreach (var ch in value) + { + builder.Append(char.IsLetterOrDigit(ch) ? char.ToLowerInvariant(ch) : '-'); + } + + var sanitized = builder.ToString().Trim('-'); + return string.IsNullOrEmpty(sanitized) ? "product" : sanitized; + } + + private static string ComputeShortHash(string value) + { + var bytes = Encoding.UTF8.GetBytes(value); + Span hash = stackalloc byte[SHA256.HashSizeInBytes]; + SHA256.HashData(bytes, hash); + return Convert.ToHexString(hash[..6]).ToLowerInvariant(); + } + } + + private sealed class MutableProduct + { + public MutableProduct(string productId) + { + ProductId = productId; + } + + public string ProductId { get; } + + private string? _name; + private string? _version; + private string? _purl; + private string? _cpe; + private readonly SortedSet _componentIdentifiers = new(StringComparer.OrdinalIgnoreCase); + + public void Update(VexProduct product) + { + if (!string.IsNullOrWhiteSpace(product.Name) && ShouldReplace(_name, product.Name)) + { + _name = product.Name; + } + + if (!string.IsNullOrWhiteSpace(product.Version) && ShouldReplace(_version, product.Version)) + { + _version = product.Version; + } + + if (!string.IsNullOrWhiteSpace(product.Purl) && ShouldReplace(_purl, product.Purl)) + { + _purl = product.Purl; + } + + if (!string.IsNullOrWhiteSpace(product.Cpe) && ShouldReplace(_cpe, product.Cpe)) + { + _cpe = product.Cpe; + } + + foreach (var identifier in product.ComponentIdentifiers) + { + if (!string.IsNullOrWhiteSpace(identifier)) + { + _componentIdentifiers.Add(identifier.Trim()); + } + } + } + + private static bool ShouldReplace(string? existing, string candidate) + { + if (string.IsNullOrWhiteSpace(candidate)) + { + return false; + } + + if (string.IsNullOrWhiteSpace(existing)) + { + return true; + } + + return candidate.Length > existing.Length; + } + + public CsafProductEntry ToEntry() + { + var helper = new CsafProductIdentificationHelper( + _purl, + _cpe, + _version, + _componentIdentifiers.Count == 0 ? null : _componentIdentifiers.ToImmutableArray()); + + return new CsafProductEntry(ProductId, _name ?? ProductId, helper); + } + } + + private sealed class CsafVulnerabilityBuilder + { + private readonly string _vulnerabilityId; + private string? _title; + private readonly Dictionary> _statusMap = new(StringComparer.Ordinal); + private readonly Dictionary> _flags = new(StringComparer.Ordinal); + private readonly Dictionary _references = new(StringComparer.Ordinal); + private readonly Dictionary _notes = new(StringComparer.Ordinal); + + public CsafVulnerabilityBuilder(string vulnerabilityId) + { + _vulnerabilityId = vulnerabilityId; + } + + public void AddClaim(VexClaim claim, string productId) + { + var statusField = MapStatus(claim.Status); + if (!string.IsNullOrEmpty(statusField)) + { + GetSet(_statusMap, statusField!).Add(productId); + } + + if (claim.Justification is not null) + { + var label = claim.Justification.Value.ToString().ToLowerInvariant(); + GetSet(_flags, label).Add(productId); + } + + if (!string.IsNullOrWhiteSpace(claim.Detail)) + { + var noteKey = FormattableString.Invariant($"{claim.ProviderId}|{productId}"); + var text = claim.Detail!.Trim(); + _notes[noteKey] = new CsafNote("description", claim.ProviderId, text, "external"); + + if (string.IsNullOrWhiteSpace(_title)) + { + _title = text; + } + } + + var referenceKey = claim.Document.Digest; + if (!_references.ContainsKey(referenceKey)) + { + _references[referenceKey] = new CsafReference( + claim.Document.SourceUri.ToString(), + claim.ProviderId, + "advisory"); + } + } + + public CsafExportVulnerability? ToVulnerability() + { + if (_statusMap.Count == 0 && _flags.Count == 0 && _references.Count == 0 && _notes.Count == 0) + { + return null; + } + + var productStatus = BuildProductStatus(); + ImmutableArray? flags = _flags.Count == 0 + ? null + : _flags + .OrderBy(static pair => pair.Key, StringComparer.Ordinal) + .Select(pair => new CsafFlag(pair.Key, pair.Value.ToImmutableArray())) + .ToImmutableArray(); + + ImmutableArray? notes = _notes.Count == 0 + ? null + : _notes.Values + .OrderBy(static note => note.Title, StringComparer.Ordinal) + .ThenBy(static note => note.Text, StringComparer.Ordinal) + .ToImmutableArray(); + + ImmutableArray? references = _references.Count == 0 + ? null + : _references.Values + .OrderBy(static reference => reference.Url, StringComparer.Ordinal) + .ToImmutableArray(); + + var isCve = _vulnerabilityId.StartsWith("CVE-", StringComparison.OrdinalIgnoreCase); + + return new CsafExportVulnerability( + Cve: isCve ? _vulnerabilityId.ToUpperInvariant() : null, + Id: isCve ? null : _vulnerabilityId, + Title: _title, + ProductStatus: productStatus, + Flags: flags, + Notes: notes, + References: references); + } + + private CsafProductStatus? BuildProductStatus() + { + var knownAffected = GetStatusArray("known_affected"); + var knownNotAffected = GetStatusArray("known_not_affected"); + var fixedProducts = GetStatusArray("fixed"); + var underInvestigation = GetStatusArray("under_investigation"); + + if (knownAffected is null && knownNotAffected is null && fixedProducts is null && underInvestigation is null) + { + return null; + } + + return new CsafProductStatus(knownAffected, knownNotAffected, fixedProducts, underInvestigation); + } + + private ImmutableArray? GetStatusArray(string statusKey) + { + if (_statusMap.TryGetValue(statusKey, out var entries) && entries.Count > 0) + { + return entries.ToImmutableArray(); + } + + return null; + } + + private static SortedSet GetSet(Dictionary> map, string key) + { + if (!map.TryGetValue(key, out var set)) + { + set = new SortedSet(StringComparer.Ordinal); + map[key] = set; + } + + return set; + } + + private static string? MapStatus(VexClaimStatus status) + => status switch + { + VexClaimStatus.Affected => "known_affected", + VexClaimStatus.NotAffected => "known_not_affected", + VexClaimStatus.Fixed => "fixed", + VexClaimStatus.UnderInvestigation => "under_investigation", + _ => null, + }; + } +} + +internal sealed record CsafExportDocument( + CsafDocumentSection Document, + CsafProductTreeSection ProductTree, + ImmutableArray Vulnerabilities, + CsafExportMetadata Metadata); + +internal sealed record CsafDocumentSection( + [property: JsonPropertyName("category")] string Category, + [property: JsonPropertyName("title")] string Title, + [property: JsonPropertyName("tracking")] CsafTrackingSection Tracking, + [property: JsonPropertyName("publisher")] CsafPublisherSection Publisher); + +internal sealed record CsafTrackingSection( + [property: JsonPropertyName("id")] string Id, + [property: JsonPropertyName("status")] string Status, + [property: JsonPropertyName("version")] string Version, + [property: JsonPropertyName("revision")] string Revision, + [property: JsonPropertyName("initial_release_date")] string InitialReleaseDate, + [property: JsonPropertyName("current_release_date")] string CurrentReleaseDate, + [property: JsonPropertyName("generator")] CsafGeneratorSection Generator); + +internal sealed record CsafGeneratorSection( + [property: JsonPropertyName("engine")] string Engine); + +internal sealed record CsafPublisherSection( + [property: JsonPropertyName("name")] string Name, + [property: JsonPropertyName("category")] string Category); + +internal sealed record CsafProductTreeSection( + [property: JsonPropertyName("full_product_names")] ImmutableArray FullProductNames); + +internal sealed record CsafProductEntry( + [property: JsonPropertyName("product_id")] string ProductId, + [property: JsonPropertyName("name")] string Name, + [property: JsonPropertyName("product_identification_helper"), JsonIgnore(Condition = JsonIgnoreCondition.WhenWritingNull)] CsafProductIdentificationHelper? IdentificationHelper); + +internal sealed record CsafProductIdentificationHelper( + [property: JsonPropertyName("purl"), JsonIgnore(Condition = JsonIgnoreCondition.WhenWritingNull)] string? Purl, + [property: JsonPropertyName("cpe"), JsonIgnore(Condition = JsonIgnoreCondition.WhenWritingNull)] string? Cpe, + [property: JsonPropertyName("product_version"), JsonIgnore(Condition = JsonIgnoreCondition.WhenWritingNull)] string? ProductVersion, + [property: JsonPropertyName("x_stellaops_component_identifiers"), JsonIgnore(Condition = JsonIgnoreCondition.WhenWritingNull)] ImmutableArray? ComponentIdentifiers); + +internal sealed record CsafExportVulnerability( + [property: JsonPropertyName("cve"), JsonIgnore(Condition = JsonIgnoreCondition.WhenWritingNull)] string? Cve, + [property: JsonPropertyName("id"), JsonIgnore(Condition = JsonIgnoreCondition.WhenWritingNull)] string? Id, + [property: JsonPropertyName("title"), JsonIgnore(Condition = JsonIgnoreCondition.WhenWritingNull)] string? Title, + [property: JsonPropertyName("product_status"), JsonIgnore(Condition = JsonIgnoreCondition.WhenWritingNull)] CsafProductStatus? ProductStatus, + [property: JsonPropertyName("flags"), JsonIgnore(Condition = JsonIgnoreCondition.WhenWritingNull)] ImmutableArray? Flags, + [property: JsonPropertyName("notes"), JsonIgnore(Condition = JsonIgnoreCondition.WhenWritingNull)] ImmutableArray? Notes, + [property: JsonPropertyName("references"), JsonIgnore(Condition = JsonIgnoreCondition.WhenWritingNull)] ImmutableArray? References); + +internal sealed record CsafProductStatus( + [property: JsonPropertyName("known_affected"), JsonIgnore(Condition = JsonIgnoreCondition.WhenWritingNull)] ImmutableArray? KnownAffected, + [property: JsonPropertyName("known_not_affected"), JsonIgnore(Condition = JsonIgnoreCondition.WhenWritingNull)] ImmutableArray? KnownNotAffected, + [property: JsonPropertyName("fixed"), JsonIgnore(Condition = JsonIgnoreCondition.WhenWritingNull)] ImmutableArray? Fixed, + [property: JsonPropertyName("under_investigation"), JsonIgnore(Condition = JsonIgnoreCondition.WhenWritingNull)] ImmutableArray? UnderInvestigation); + +internal sealed record CsafFlag( + [property: JsonPropertyName("label")] string Label, + [property: JsonPropertyName("product_ids")] ImmutableArray ProductIds); + +internal sealed record CsafNote( + [property: JsonPropertyName("category")] string Category, + [property: JsonPropertyName("title")] string Title, + [property: JsonPropertyName("text")] string Text, + [property: JsonPropertyName("audience"), JsonIgnore(Condition = JsonIgnoreCondition.WhenWritingNull)] string? Audience); + +internal sealed record CsafReference( + [property: JsonPropertyName("url")] string Url, + [property: JsonPropertyName("summary")] string Summary, + [property: JsonPropertyName("type")] string Type); + +internal sealed record CsafExportMetadata( + [property: JsonPropertyName("generated_at")] string GeneratedAt, + [property: JsonPropertyName("query_signature")] string QuerySignature, + [property: JsonPropertyName("source_providers")] ImmutableArray SourceProviders, + [property: JsonPropertyName("diagnostics")] ImmutableDictionary Diagnostics); diff --git a/src/Excititor/__Libraries/StellaOps.Excititor.Formats.CSAF/CsafNormalizer.cs b/src/Excititor/__Libraries/StellaOps.Excititor.Formats.CSAF/CsafNormalizer.cs index e68b5fd8..5edf3417 100644 --- a/src/Excititor/__Libraries/StellaOps.Excititor.Formats.CSAF/CsafNormalizer.cs +++ b/src/Excititor/__Libraries/StellaOps.Excititor.Formats.CSAF/CsafNormalizer.cs @@ -141,17 +141,22 @@ public sealed class CsafNormalizer : IVexNormalizer var diagnosticsBuilder = ImmutableDictionary.CreateBuilder(StringComparer.Ordinal); if (!result.UnsupportedStatuses.IsDefaultOrEmpty && result.UnsupportedStatuses.Length > 0) { - diagnosticsBuilder["csaf.unsupported_statuses"] = string.Join(",", result.UnsupportedStatuses); + diagnosticsBuilder["policy.unsupported_statuses"] = string.Join(",", result.UnsupportedStatuses); } if (!result.UnsupportedJustifications.IsDefaultOrEmpty && result.UnsupportedJustifications.Length > 0) { - diagnosticsBuilder["csaf.unsupported_justifications"] = string.Join(",", result.UnsupportedJustifications); + diagnosticsBuilder["policy.unsupported_justifications"] = string.Join(",", result.UnsupportedJustifications); } if (!result.ConflictingJustifications.IsDefaultOrEmpty && result.ConflictingJustifications.Length > 0) { - diagnosticsBuilder["csaf.justification_conflicts"] = string.Join(",", result.ConflictingJustifications); + diagnosticsBuilder["policy.justification_conflicts"] = string.Join(",", result.ConflictingJustifications); + } + + if (!result.MissingRequiredJustifications.IsDefaultOrEmpty && result.MissingRequiredJustifications.Length > 0) + { + diagnosticsBuilder["policy.justification_missing"] = string.Join(",", result.MissingRequiredJustifications); } var diagnostics = diagnosticsBuilder.Count == 0 @@ -202,6 +207,7 @@ public sealed class CsafNormalizer : IVexNormalizer var unsupportedStatuses = new HashSet(StringComparer.OrdinalIgnoreCase); var unsupportedJustifications = new HashSet(StringComparer.OrdinalIgnoreCase); var conflictingJustifications = new HashSet(StringComparer.OrdinalIgnoreCase); + var missingRequiredJustifications = new HashSet(StringComparer.OrdinalIgnoreCase); var claimsBuilder = ImmutableArray.CreateBuilder(); @@ -230,7 +236,8 @@ public sealed class CsafNormalizer : IVexNormalizer productCatalog, justifications, detail, - unsupportedStatuses); + unsupportedStatuses, + missingRequiredJustifications); claimsBuilder.AddRange(productClaims); } @@ -244,7 +251,8 @@ public sealed class CsafNormalizer : IVexNormalizer claimsBuilder.ToImmutable(), unsupportedStatuses.OrderBy(static s => s, StringComparer.OrdinalIgnoreCase).ToImmutableArray(), unsupportedJustifications.OrderBy(static s => s, StringComparer.OrdinalIgnoreCase).ToImmutableArray(), - conflictingJustifications.OrderBy(static s => s, StringComparer.OrdinalIgnoreCase).ToImmutableArray()); + conflictingJustifications.OrderBy(static s => s, StringComparer.OrdinalIgnoreCase).ToImmutableArray(), + missingRequiredJustifications.OrderBy(static s => s, StringComparer.OrdinalIgnoreCase).ToImmutableArray()); } private static IReadOnlyList BuildClaimsForVulnerability( @@ -253,7 +261,8 @@ public sealed class CsafNormalizer : IVexNormalizer IReadOnlyDictionary productCatalog, ImmutableDictionary justifications, string? detail, - ISet unsupportedStatuses) + ISet unsupportedStatuses, + ISet missingRequiredJustifications) { if (!vulnerability.TryGetProperty("product_status", out var statusElement) || statusElement.ValueKind != JsonValueKind.Object) @@ -297,7 +306,7 @@ public sealed class CsafNormalizer : IVexNormalizer return Array.Empty(); } - return claims.Values + var builtClaims = claims.Values .Select(builder => new CsafClaimEntry( vulnerabilityId, builder.Product, @@ -307,6 +316,16 @@ public sealed class CsafNormalizer : IVexNormalizer builder.Justification, builder.RawJustification)) .ToArray(); + + foreach (var entry in builtClaims) + { + if (entry.Status == VexClaimStatus.NotAffected && entry.Justification is null) + { + missingRequiredJustifications.Add(FormattableString.Invariant($"{entry.VulnerabilityId}:{entry.Product.ProductId}")); + } + } + + return builtClaims; } private static void UpdateClaim( @@ -855,7 +874,8 @@ public sealed class CsafNormalizer : IVexNormalizer ImmutableArray Claims, ImmutableArray UnsupportedStatuses, ImmutableArray UnsupportedJustifications, - ImmutableArray ConflictingJustifications); + ImmutableArray ConflictingJustifications, + ImmutableArray MissingRequiredJustifications); private sealed record CsafJustificationInfo( string RawValue, diff --git a/src/Excititor/__Libraries/StellaOps.Excititor.Formats.CSAF/ServiceCollectionExtensions.cs b/src/Excititor/__Libraries/StellaOps.Excititor.Formats.CSAF/ServiceCollectionExtensions.cs index 11cf214d..1c082656 100644 --- a/src/Excititor/__Libraries/StellaOps.Excititor.Formats.CSAF/ServiceCollectionExtensions.cs +++ b/src/Excititor/__Libraries/StellaOps.Excititor.Formats.CSAF/ServiceCollectionExtensions.cs @@ -5,10 +5,11 @@ namespace StellaOps.Excititor.Formats.CSAF; public static class CsafFormatsServiceCollectionExtensions { - public static IServiceCollection AddCsafNormalizer(this IServiceCollection services) - { - ArgumentNullException.ThrowIfNull(services); - services.AddSingleton(); - return services; - } -} + public static IServiceCollection AddCsafNormalizer(this IServiceCollection services) + { + ArgumentNullException.ThrowIfNull(services); + services.AddSingleton(); + services.AddSingleton(); + return services; + } +} diff --git a/src/Excititor/__Libraries/StellaOps.Excititor.Formats.CSAF/TASKS.md b/src/Excititor/__Libraries/StellaOps.Excititor.Formats.CSAF/TASKS.md index 40c0b12a..bbb8d361 100644 --- a/src/Excititor/__Libraries/StellaOps.Excititor.Formats.CSAF/TASKS.md +++ b/src/Excititor/__Libraries/StellaOps.Excititor.Formats.CSAF/TASKS.md @@ -3,5 +3,5 @@ If you are working on this file you need to read docs/modules/excititor/ARCHITEC | Task | Owner(s) | Depends on | Notes | |---|---|---|---| |EXCITITOR-FMT-CSAF-01-001 – CSAF normalizer foundation|Team Excititor Formats|EXCITITOR-CORE-01-001|**DONE (2025-10-17)** – Implemented CSAF normalizer + DI hook, parsing tracking metadata, product tree branches/full names, and mapping product statuses into canonical `VexClaim`s with baseline precedence. Regression added in `CsafNormalizerTests`.| -|EXCITITOR-FMT-CSAF-01-002 – Status/justification mapping|Team Excititor Formats|EXCITITOR-FMT-CSAF-01-001, EXCITITOR-POLICY-01-001|**DOING (2025-10-19)** – Prereqs EXCITITOR-FMT-CSAF-01-001 & EXCITITOR-POLICY-01-001 verified DONE; starting normalization of `product_status`/`justification` values with policy-aligned diagnostics.| -|EXCITITOR-FMT-CSAF-01-003 – CSAF export adapter|Team Excititor Formats|EXCITITOR-EXPORT-01-001, EXCITITOR-FMT-CSAF-01-001|**DOING (2025-10-19)** – Prereqs EXCITITOR-EXPORT-01-001 & EXCITITOR-FMT-CSAF-01-001 confirmed DONE; drafting deterministic CSAF exporter and manifest metadata flow.| +|EXCITITOR-FMT-CSAF-01-002 – Status/justification mapping|Team Excititor Formats|EXCITITOR-FMT-CSAF-01-001, EXCITITOR-POLICY-01-001|**DONE (2025-10-29)** – Added policy-aligned diagnostics for unsupported statuses/justifications and flagged missing not_affected evidence inside normalizer outputs.| +|EXCITITOR-FMT-CSAF-01-003 – CSAF export adapter|Team Excititor Formats|EXCITITOR-EXPORT-01-001, EXCITITOR-FMT-CSAF-01-001|**DONE (2025-10-29)** – Implemented deterministic CSAF exporter with product tree reconciliation, vulnerability status mapping, and metadata for downstream attestation.| diff --git a/src/Excititor/__Libraries/StellaOps.Excititor.Formats.CycloneDX/CycloneDxComponentReconciler.cs b/src/Excititor/__Libraries/StellaOps.Excititor.Formats.CycloneDX/CycloneDxComponentReconciler.cs new file mode 100644 index 00000000..d4243d03 --- /dev/null +++ b/src/Excititor/__Libraries/StellaOps.Excititor.Formats.CycloneDX/CycloneDxComponentReconciler.cs @@ -0,0 +1,242 @@ +using System.Collections.Generic; +using System.Collections.Immutable; +using System.Globalization; +using System.Linq; +using System.Security.Cryptography; +using System.Text; +using System.Text.Json.Serialization; +using StellaOps.Excititor.Core; + +namespace StellaOps.Excititor.Formats.CycloneDX; + +internal static class CycloneDxComponentReconciler +{ + public static CycloneDxReconciliationResult Reconcile(IEnumerable claims) + { + ArgumentNullException.ThrowIfNull(claims); + + var catalog = new ComponentCatalog(); + var diagnostics = new Dictionary>(StringComparer.Ordinal); + var componentRefs = new Dictionary<(string VulnerabilityId, string ProductKey), string>(); + + foreach (var claim in claims) + { + if (claim is null) + { + continue; + } + + var component = catalog.GetOrAdd(claim.Product, claim.ProviderId, diagnostics); + componentRefs[(claim.VulnerabilityId, claim.Product.Key)] = component.BomRef; + } + + var components = catalog.Build(); + var orderedDiagnostics = diagnostics.Count == 0 + ? ImmutableDictionary.Empty + : diagnostics.ToImmutableDictionary( + static pair => pair.Key, + pair => string.Join(",", pair.Value.OrderBy(static value => value, StringComparer.Ordinal)), + StringComparer.Ordinal); + + return new CycloneDxReconciliationResult( + components, + componentRefs.ToImmutableDictionary(), + orderedDiagnostics); + } + + private sealed class ComponentCatalog + { + private readonly Dictionary _components = new(StringComparer.Ordinal); + private readonly HashSet _bomRefs = new(StringComparer.Ordinal); + + public MutableComponent GetOrAdd(VexProduct product, string providerId, IDictionary> diagnostics) + { + if (_components.TryGetValue(product.Key, out var existing)) + { + existing.Update(product, providerId, diagnostics); + return existing; + } + + var bomRef = GenerateBomRef(product); + var component = new MutableComponent(product.Key, bomRef); + component.Update(product, providerId, diagnostics); + _components[product.Key] = component; + return component; + } + + public ImmutableArray Build() + => _components.Values + .Select(static component => component.ToEntry()) + .OrderBy(static entry => entry.BomRef, StringComparer.Ordinal) + .ToImmutableArray(); + + private string GenerateBomRef(VexProduct product) + { + if (!string.IsNullOrWhiteSpace(product.Purl)) + { + var normalized = product.Purl!.Trim(); + if (_bomRefs.Add(normalized)) + { + return normalized; + } + } + + var baseRef = Sanitize(product.Key); + if (_bomRefs.Add(baseRef)) + { + return baseRef; + } + + var hash = ComputeShortHash(product.Key + product.Name); + var candidate = FormattableString.Invariant($"{baseRef}-{hash}"); + while (!_bomRefs.Add(candidate)) + { + candidate = FormattableString.Invariant($"{candidate}-{hash}"); + } + + return candidate; + } + + private static string Sanitize(string value) + { + if (string.IsNullOrWhiteSpace(value)) + { + return "component"; + } + + var builder = new StringBuilder(value.Length); + foreach (var ch in value) + { + builder.Append(char.IsLetterOrDigit(ch) ? char.ToLowerInvariant(ch) : '-'); + } + + var sanitized = builder.ToString().Trim('-'); + return string.IsNullOrEmpty(sanitized) ? "component" : sanitized; + } + + private static string ComputeShortHash(string value) + { + var bytes = Encoding.UTF8.GetBytes(value); + Span hash = stackalloc byte[SHA256.HashSizeInBytes]; + SHA256.HashData(bytes, hash); + return Convert.ToHexString(hash[..6]).ToLowerInvariant(); + } + } + + private sealed class MutableComponent + { + public MutableComponent(string key, string bomRef) + { + ProductKey = key; + BomRef = bomRef; + } + + public string ProductKey { get; } + + public string BomRef { get; } + + private string? _name; + private string? _version; + private string? _purl; + private string? _cpe; + private readonly SortedDictionary _properties = new(StringComparer.Ordinal); + + public void Update(VexProduct product, string providerId, IDictionary> diagnostics) + { + if (!string.IsNullOrWhiteSpace(product.Name) && ShouldReplace(_name, product.Name)) + { + _name = product.Name; + } + + if (!string.IsNullOrWhiteSpace(product.Version) && ShouldReplace(_version, product.Version)) + { + _version = product.Version; + } + + if (!string.IsNullOrWhiteSpace(product.Purl)) + { + var trimmed = product.Purl!.Trim(); + if (string.IsNullOrWhiteSpace(_purl)) + { + _purl = trimmed; + } + else if (!string.Equals(_purl, trimmed, StringComparison.OrdinalIgnoreCase)) + { + AddDiagnostic(diagnostics, "purl_conflict", FormattableString.Invariant($"{ProductKey}:{_purl}->{trimmed}")); + } + } + else + { + AddDiagnostic(diagnostics, "missing_purl", FormattableString.Invariant($"{ProductKey}:{providerId}")); + } + + if (!string.IsNullOrWhiteSpace(product.Cpe)) + { + _cpe = product.Cpe; + } + + if (product.ComponentIdentifiers.Length > 0) + { + _properties["stellaops/componentIdentifiers"] = string.Join(';', product.ComponentIdentifiers.OrderBy(static identifier => identifier, StringComparer.OrdinalIgnoreCase)); + } + } + + public CycloneDxComponentEntry ToEntry() + { + ImmutableArray? properties = _properties.Count == 0 + ? null + : _properties.Select(static pair => new CycloneDxProperty(pair.Key, pair.Value)).ToImmutableArray(); + + return new CycloneDxComponentEntry( + BomRef, + _name ?? ProductKey, + _version, + _purl, + _cpe, + properties); + } + + private static bool ShouldReplace(string? existing, string candidate) + { + if (string.IsNullOrWhiteSpace(candidate)) + { + return false; + } + + if (string.IsNullOrWhiteSpace(existing)) + { + return true; + } + + return candidate.Length > existing.Length; + } + + private static void AddDiagnostic(IDictionary> diagnostics, string key, string value) + { + if (!diagnostics.TryGetValue(key, out var set)) + { + set = new SortedSet(StringComparer.Ordinal); + diagnostics[key] = set; + } + + set.Add(value); + } + } +} + +internal sealed record CycloneDxReconciliationResult( + ImmutableArray Components, + ImmutableDictionary<(string VulnerabilityId, string ProductKey), string> ComponentRefs, + ImmutableDictionary Diagnostics); + +internal sealed record CycloneDxComponentEntry( + [property: JsonPropertyName("bom-ref")] string BomRef, + [property: JsonPropertyName("name")] string Name, + [property: JsonPropertyName("version"), JsonIgnore(Condition = JsonIgnoreCondition.WhenWritingNull)] string? Version, + [property: JsonPropertyName("purl"), JsonIgnore(Condition = JsonIgnoreCondition.WhenWritingNull)] string? Purl, + [property: JsonPropertyName("cpe"), JsonIgnore(Condition = JsonIgnoreCondition.WhenWritingNull)] string? Cpe, + [property: JsonPropertyName("properties"), JsonIgnore(Condition = JsonIgnoreCondition.WhenWritingNull)] ImmutableArray? Properties); + +internal sealed record CycloneDxProperty( + [property: JsonPropertyName("name")] string Name, + [property: JsonPropertyName("value")] string Value); diff --git a/src/Excititor/__Libraries/StellaOps.Excititor.Formats.CycloneDX/CycloneDxExporter.cs b/src/Excititor/__Libraries/StellaOps.Excititor.Formats.CycloneDX/CycloneDxExporter.cs new file mode 100644 index 00000000..c7253841 --- /dev/null +++ b/src/Excititor/__Libraries/StellaOps.Excititor.Formats.CycloneDX/CycloneDxExporter.cs @@ -0,0 +1,228 @@ +using System.Collections.Generic; +using System.Collections.Immutable; +using System.Globalization; +using System.IO; +using System.Linq; +using System.Security.Cryptography; +using System.Text; +using System.Text.Json.Serialization; +using System.Threading; +using System.Threading.Tasks; +using StellaOps.Excititor.Core; + +namespace StellaOps.Excititor.Formats.CycloneDX; + +/// +/// Serialises normalized VEX claims into CycloneDX VEX documents with reconciled component references. +/// +public sealed class CycloneDxExporter : IVexExporter +{ + public VexExportFormat Format => VexExportFormat.CycloneDx; + + public VexContentAddress Digest(VexExportRequest request) + { + ArgumentNullException.ThrowIfNull(request); + var document = BuildDocument(request, out _); + var json = VexCanonicalJsonSerializer.Serialize(document); + return ComputeDigest(json); + } + + public async ValueTask SerializeAsync( + VexExportRequest request, + Stream output, + CancellationToken cancellationToken) + { + ArgumentNullException.ThrowIfNull(request); + ArgumentNullException.ThrowIfNull(output); + + var document = BuildDocument(request, out var metadata); + var json = VexCanonicalJsonSerializer.Serialize(document); + var digest = ComputeDigest(json); + var buffer = Encoding.UTF8.GetBytes(json); + await output.WriteAsync(buffer, 0, buffer.Length, cancellationToken).ConfigureAwait(false); + return new VexExportResult(digest, buffer.LongLength, metadata); + } + + private CycloneDxExportDocument BuildDocument(VexExportRequest request, out ImmutableDictionary metadata) + { + var signature = VexQuerySignature.FromQuery(request.Query); + var signatureHash = signature.ComputeHash(); + var generatedAt = request.GeneratedAt.UtcDateTime.ToString("O", CultureInfo.InvariantCulture); + + var reconciliation = CycloneDxComponentReconciler.Reconcile(request.Claims); + var vulnerabilityEntries = BuildVulnerabilities(request.Claims, reconciliation.ComponentRefs); + + var missingJustifications = request.Claims + .Where(static claim => claim.Status == VexClaimStatus.NotAffected && claim.Justification is null) + .Select(static claim => FormattableString.Invariant($"{claim.VulnerabilityId}:{claim.Product.Key}")) + .Distinct(StringComparer.Ordinal) + .OrderBy(static key => key, StringComparer.Ordinal) + .ToImmutableArray(); + + var properties = ImmutableArray.Create(new CycloneDxProperty("stellaops/querySignature", signature.Value)); + + metadata = BuildMetadata(signature, reconciliation.Diagnostics, generatedAt, vulnerabilityEntries.Length, reconciliation.Components.Length, missingJustifications); + + var document = new CycloneDxExportDocument( + BomFormat: "CycloneDX", + SpecVersion: "1.6", + SerialNumber: FormattableString.Invariant($"urn:uuid:{BuildDeterministicGuid(signatureHash.Digest)}"), + Version: 1, + Metadata: new CycloneDxMetadata(generatedAt), + Components: reconciliation.Components, + Vulnerabilities: vulnerabilityEntries, + Properties: properties); + + return document; + } + + private static ImmutableArray BuildVulnerabilities( + ImmutableArray claims, + ImmutableDictionary<(string VulnerabilityId, string ProductKey), string> componentRefs) + { + var entries = ImmutableArray.CreateBuilder(); + + foreach (var claim in claims) + { + if (!componentRefs.TryGetValue((claim.VulnerabilityId, claim.Product.Key), out var componentRef)) + { + continue; + } + + var analysis = new CycloneDxAnalysis( + State: MapStatus(claim.Status), + Justification: claim.Justification?.ToString().ToLowerInvariant(), + Responses: null); + + var affects = ImmutableArray.Create(new CycloneDxAffectEntry(componentRef)); + + var properties = ImmutableArray.Create( + new CycloneDxProperty("stellaops/providerId", claim.ProviderId), + new CycloneDxProperty("stellaops/documentDigest", claim.Document.Digest)); + + var vulnerabilityId = claim.VulnerabilityId; + var bomRef = FormattableString.Invariant($"{vulnerabilityId}#{Normalize(componentRef)}"); + + entries.Add(new CycloneDxVulnerabilityEntry( + Id: vulnerabilityId, + BomRef: bomRef, + Description: claim.Detail, + Analysis: analysis, + Affects: affects, + Properties: properties)); + } + + return entries + .ToImmutable() + .OrderBy(static entry => entry.Id, StringComparer.Ordinal) + .ThenBy(static entry => entry.BomRef, StringComparer.Ordinal) + .ToImmutableArray(); + } + + private static string Normalize(string value) + { + if (string.IsNullOrWhiteSpace(value)) + { + return "component"; + } + + var builder = new StringBuilder(value.Length); + foreach (var ch in value) + { + builder.Append(char.IsLetterOrDigit(ch) ? char.ToLowerInvariant(ch) : '-'); + } + + var normalized = builder.ToString().Trim('-'); + return string.IsNullOrEmpty(normalized) ? "component" : normalized; + } + + private static string MapStatus(VexClaimStatus status) + => status switch + { + VexClaimStatus.Affected => "affected", + VexClaimStatus.NotAffected => "not_affected", + VexClaimStatus.Fixed => "resolved", + VexClaimStatus.UnderInvestigation => "under_investigation", + _ => "unknown", + }; + + private static ImmutableDictionary BuildMetadata( + VexQuerySignature signature, + ImmutableDictionary diagnostics, + string generatedAt, + int vulnerabilityCount, + int componentCount, + ImmutableArray missingJustifications) + { + var builder = ImmutableDictionary.CreateBuilder(StringComparer.Ordinal); + builder["cyclonedx.querySignature"] = signature.Value; + builder["cyclonedx.generatedAt"] = generatedAt; + builder["cyclonedx.vulnerabilityCount"] = vulnerabilityCount.ToString(CultureInfo.InvariantCulture); + builder["cyclonedx.componentCount"] = componentCount.ToString(CultureInfo.InvariantCulture); + + foreach (var diagnostic in diagnostics.OrderBy(static pair => pair.Key, StringComparer.Ordinal)) + { + builder[$"cyclonedx.{diagnostic.Key}"] = diagnostic.Value; + } + + if (!missingJustifications.IsDefaultOrEmpty && missingJustifications.Length > 0) + { + builder["policy.justification_missing"] = string.Join(",", missingJustifications); + } + + return builder.ToImmutable(); + } + + private static string BuildDeterministicGuid(string digest) + { + if (string.IsNullOrWhiteSpace(digest) || digest.Length < 32) + { + return Guid.NewGuid().ToString(); + } + + var hex = digest[..32]; + var bytes = Enumerable.Range(0, hex.Length / 2) + .Select(i => byte.Parse(hex.Substring(i * 2, 2), NumberStyles.HexNumber, CultureInfo.InvariantCulture)) + .ToArray(); + + return new Guid(bytes).ToString(); + } + + private static VexContentAddress ComputeDigest(string json) + { + var bytes = Encoding.UTF8.GetBytes(json); + Span hash = stackalloc byte[SHA256.HashSizeInBytes]; + SHA256.HashData(bytes, hash); + var digest = Convert.ToHexString(hash).ToLowerInvariant(); + return new VexContentAddress("sha256", digest); + } +} + +internal sealed record CycloneDxExportDocument( + [property: JsonPropertyName("bomFormat")] string BomFormat, + [property: JsonPropertyName("specVersion")] string SpecVersion, + [property: JsonPropertyName("serialNumber")] string SerialNumber, + [property: JsonPropertyName("version")] int Version, + [property: JsonPropertyName("metadata")] CycloneDxMetadata Metadata, + [property: JsonPropertyName("components")] ImmutableArray Components, + [property: JsonPropertyName("vulnerabilities")] ImmutableArray Vulnerabilities, + [property: JsonPropertyName("properties"), JsonIgnore(Condition = JsonIgnoreCondition.WhenWritingNull)] ImmutableArray? Properties); + +internal sealed record CycloneDxMetadata( + [property: JsonPropertyName("timestamp")] string Timestamp); + +internal sealed record CycloneDxVulnerabilityEntry( + [property: JsonPropertyName("id")] string Id, + [property: JsonPropertyName("bom-ref")] string BomRef, + [property: JsonPropertyName("description"), JsonIgnore(Condition = JsonIgnoreCondition.WhenWritingNull)] string? Description, + [property: JsonPropertyName("analysis")] CycloneDxAnalysis Analysis, + [property: JsonPropertyName("affects")] ImmutableArray Affects, + [property: JsonPropertyName("properties")] ImmutableArray Properties); + +internal sealed record CycloneDxAnalysis( + [property: JsonPropertyName("state")] string State, + [property: JsonPropertyName("justification"), JsonIgnore(Condition = JsonIgnoreCondition.WhenWritingNull)] string? Justification, + [property: JsonPropertyName("response"), JsonIgnore(Condition = JsonIgnoreCondition.WhenWritingNull)] ImmutableArray? Responses); + +internal sealed record CycloneDxAffectEntry( + [property: JsonPropertyName("ref")] string Reference); diff --git a/src/Excititor/__Libraries/StellaOps.Excititor.Formats.CycloneDX/ServiceCollectionExtensions.cs b/src/Excititor/__Libraries/StellaOps.Excititor.Formats.CycloneDX/ServiceCollectionExtensions.cs index 518d7223..ffdc4fc6 100644 --- a/src/Excititor/__Libraries/StellaOps.Excititor.Formats.CycloneDX/ServiceCollectionExtensions.cs +++ b/src/Excititor/__Libraries/StellaOps.Excititor.Formats.CycloneDX/ServiceCollectionExtensions.cs @@ -5,10 +5,11 @@ namespace StellaOps.Excititor.Formats.CycloneDX; public static class CycloneDxFormatsServiceCollectionExtensions { - public static IServiceCollection AddCycloneDxNormalizer(this IServiceCollection services) - { - ArgumentNullException.ThrowIfNull(services); - services.AddSingleton(); - return services; - } -} + public static IServiceCollection AddCycloneDxNormalizer(this IServiceCollection services) + { + ArgumentNullException.ThrowIfNull(services); + services.AddSingleton(); + services.AddSingleton(); + return services; + } +} diff --git a/src/Excititor/__Libraries/StellaOps.Excititor.Formats.CycloneDX/TASKS.md b/src/Excititor/__Libraries/StellaOps.Excititor.Formats.CycloneDX/TASKS.md index 76bfacd2..7b148289 100644 --- a/src/Excititor/__Libraries/StellaOps.Excititor.Formats.CycloneDX/TASKS.md +++ b/src/Excititor/__Libraries/StellaOps.Excititor.Formats.CycloneDX/TASKS.md @@ -3,5 +3,5 @@ If you are working on this file you need to read docs/modules/excititor/ARCHITEC | Task | Owner(s) | Depends on | Notes | |---|---|---|---| |EXCITITOR-FMT-CYCLONE-01-001 – CycloneDX VEX normalizer|Team Excititor Formats|EXCITITOR-CORE-01-001|**DONE (2025-10-17)** – CycloneDX normalizer parses `analysis` data, resolves component references, and emits canonical `VexClaim`s; regression lives in `CycloneDxNormalizerTests`.| -|EXCITITOR-FMT-CYCLONE-01-002 – Component reference reconciliation|Team Excititor Formats|EXCITITOR-FMT-CYCLONE-01-001|**DOING (2025-10-19)** – Prereq EXCITITOR-FMT-CYCLONE-01-001 confirmed DONE; proceeding with reference reconciliation helpers and diagnostics for missing SBOM links.| -|EXCITITOR-FMT-CYCLONE-01-003 – CycloneDX export serializer|Team Excititor Formats|EXCITITOR-EXPORT-01-001, EXCITITOR-FMT-CYCLONE-01-001|**DOING (2025-10-19)** – Prereqs EXCITITOR-EXPORT-01-001 & EXCITITOR-FMT-CYCLONE-01-001 verified DONE; initiating deterministic CycloneDX VEX exporter work.| +|EXCITITOR-FMT-CYCLONE-01-002 – Component reference reconciliation|Team Excititor Formats|EXCITITOR-FMT-CYCLONE-01-001|**DONE (2025-10-29)** – Added reconciler producing stable bom-refs, aggregating component metadata, and reporting missing PURL diagnostics for policy gating.| +|EXCITITOR-FMT-CYCLONE-01-003 – CycloneDX export serializer|Team Excititor Formats|EXCITITOR-EXPORT-01-001, EXCITITOR-FMT-CYCLONE-01-001|**DONE (2025-10-29)** – Implemented CycloneDX VEX exporter emitting reconciled components, vulnerability analysis blocks, and canonical metadata.| diff --git a/src/Excititor/__Libraries/StellaOps.Excititor.Formats.OpenVEX/OpenVexExporter.cs b/src/Excititor/__Libraries/StellaOps.Excititor.Formats.OpenVEX/OpenVexExporter.cs new file mode 100644 index 00000000..d1915e65 --- /dev/null +++ b/src/Excititor/__Libraries/StellaOps.Excititor.Formats.OpenVEX/OpenVexExporter.cs @@ -0,0 +1,217 @@ +using System.Collections.Generic; +using System.Collections.Immutable; +using System.Globalization; +using System.IO; +using System.Linq; +using System.Security.Cryptography; +using System.Text; +using System.Text.Json.Serialization; +using System.Threading; +using System.Threading.Tasks; +using StellaOps.Excititor.Core; + +namespace StellaOps.Excititor.Formats.OpenVEX; + +/// +/// Serializes merged VEX statements into canonical OpenVEX export documents. +/// +public sealed class OpenVexExporter : IVexExporter +{ + public OpenVexExporter() + { + } + + public VexExportFormat Format => VexExportFormat.OpenVex; + + public VexContentAddress Digest(VexExportRequest request) + { + ArgumentNullException.ThrowIfNull(request); + var document = BuildDocument(request, out _); + var json = VexCanonicalJsonSerializer.Serialize(document); + return ComputeDigest(json); + } + + public async ValueTask SerializeAsync( + VexExportRequest request, + Stream output, + CancellationToken cancellationToken) + { + ArgumentNullException.ThrowIfNull(request); + ArgumentNullException.ThrowIfNull(output); + + var metadata = BuildDocument(request, out var exportMetadata); + var json = VexCanonicalJsonSerializer.Serialize(metadata); + var digest = ComputeDigest(json); + var buffer = Encoding.UTF8.GetBytes(json); + await output.WriteAsync(buffer, 0, buffer.Length, cancellationToken).ConfigureAwait(false); + return new VexExportResult(digest, buffer.LongLength, exportMetadata); + } + + private OpenVexExportDocument BuildDocument(VexExportRequest request, out ImmutableDictionary metadata) + { + var mergeResult = OpenVexStatementMerger.Merge(request.Claims); + var signature = VexQuerySignature.FromQuery(request.Query); + var signatureHash = signature.ComputeHash(); + var generatedAt = request.GeneratedAt.UtcDateTime.ToString("O", CultureInfo.InvariantCulture); + var sourceProviders = request.Claims + .Select(static claim => claim.ProviderId) + .Distinct(StringComparer.Ordinal) + .OrderBy(static provider => provider, StringComparer.Ordinal) + .ToImmutableArray(); + + var statements = mergeResult.Statements + .Select(statement => MapStatement(statement)) + .ToImmutableArray(); + + var document = new OpenVexDocumentSection( + Id: FormattableString.Invariant($"openvex:export:{signatureHash.Digest}"), + Author: "StellaOps Excititor", + Version: "1", + Created: generatedAt, + LastUpdated: generatedAt, + Profile: "stellaops-export/v1"); + + var metadataSection = new OpenVexExportMetadata( + generatedAt, + signature.Value, + sourceProviders, + mergeResult.Diagnostics); + + metadata = BuildMetadata(signature, mergeResult, sourceProviders, generatedAt); + + return new OpenVexExportDocument(document, statements, metadataSection); + } + + private static ImmutableDictionary BuildMetadata( + VexQuerySignature signature, + OpenVexMergeResult mergeResult, + ImmutableArray sourceProviders, + string generatedAt) + { + var metadataBuilder = ImmutableDictionary.CreateBuilder(StringComparer.Ordinal); + metadataBuilder["openvex.querySignature"] = signature.Value; + metadataBuilder["openvex.generatedAt"] = generatedAt; + metadataBuilder["openvex.statementCount"] = mergeResult.Statements.Length.ToString(CultureInfo.InvariantCulture); + metadataBuilder["openvex.providerCount"] = sourceProviders.Length.ToString(CultureInfo.InvariantCulture); + + var sourceCount = mergeResult.Statements.Sum(static statement => statement.Sources.Length); + metadataBuilder["openvex.sourceCount"] = sourceCount.ToString(CultureInfo.InvariantCulture); + + foreach (var diagnostic in mergeResult.Diagnostics.OrderBy(static pair => pair.Key, StringComparer.Ordinal)) + { + metadataBuilder[$"openvex.diagnostic.{diagnostic.Key}"] = diagnostic.Value; + } + + return metadataBuilder.ToImmutable(); + } + + private static OpenVexExportStatement MapStatement(OpenVexMergedStatement statement) + { + var products = ImmutableArray.Create( + new OpenVexExportProduct( + Id: statement.Product.Key, + Name: statement.Product.Name ?? statement.Product.Key, + Version: statement.Product.Version, + Purl: statement.Product.Purl, + Cpe: statement.Product.Cpe)); + + var sources = statement.Sources + .Select(source => new OpenVexExportSource( + Provider: source.ProviderId, + Status: source.Status.ToString().ToLowerInvariant(), + Justification: source.Justification?.ToString().ToLowerInvariant(), + DocumentDigest: source.DocumentDigest, + SourceUri: source.DocumentSource.ToString(), + Detail: source.Detail, + FirstObserved: source.FirstSeen.UtcDateTime.ToString("O", CultureInfo.InvariantCulture), + LastObserved: source.LastSeen.UtcDateTime.ToString("O", CultureInfo.InvariantCulture))) + .ToImmutableArray(); + + var statementId = FormattableString.Invariant($"{statement.VulnerabilityId}#{NormalizeProductKey(statement.Product.Key)}"); + + return new OpenVexExportStatement( + Id: statementId, + Vulnerability: statement.VulnerabilityId, + Status: statement.Status.ToString().ToLowerInvariant(), + Justification: statement.Justification?.ToString().ToLowerInvariant(), + Timestamp: statement.FirstObserved.UtcDateTime.ToString("O", CultureInfo.InvariantCulture), + LastUpdated: statement.LastObserved.UtcDateTime.ToString("O", CultureInfo.InvariantCulture), + Products: products, + Statement: statement.Detail, + Sources: sources); + } + + private static string NormalizeProductKey(string key) + { + if (string.IsNullOrWhiteSpace(key)) + { + return "unknown"; + } + + var builder = new StringBuilder(key.Length); + foreach (var ch in key) + { + builder.Append(char.IsLetterOrDigit(ch) ? char.ToLowerInvariant(ch) : '-'); + } + + var normalized = builder.ToString().Trim('-'); + return string.IsNullOrEmpty(normalized) ? "unknown" : normalized; + } + + private static VexContentAddress ComputeDigest(string json) + { + var bytes = Encoding.UTF8.GetBytes(json); + Span hash = stackalloc byte[SHA256.HashSizeInBytes]; + SHA256.HashData(bytes, hash); + var digest = Convert.ToHexString(hash).ToLowerInvariant(); + return new VexContentAddress("sha256", digest); + } +} + +internal sealed record OpenVexExportDocument( + OpenVexDocumentSection Document, + ImmutableArray Statements, + OpenVexExportMetadata Metadata); + +internal sealed record OpenVexDocumentSection( + [property: JsonPropertyName("@context")] string Context = "https://openvex.dev/ns/v0.2", + [property: JsonPropertyName("id")] string Id = "", + [property: JsonPropertyName("author")] string Author = "", + [property: JsonPropertyName("version")] string Version = "1", + [property: JsonPropertyName("created")] string Created = "", + [property: JsonPropertyName("last_updated")] string LastUpdated = "", + [property: JsonPropertyName("profile")] string Profile = ""); + +internal sealed record OpenVexExportStatement( + [property: JsonPropertyName("id")] string Id, + [property: JsonPropertyName("vulnerability")] string Vulnerability, + [property: JsonPropertyName("status")] string Status, + [property: JsonPropertyName("justification"), JsonIgnore(Condition = JsonIgnoreCondition.WhenWritingNull)] string? Justification, + [property: JsonPropertyName("timestamp")] string Timestamp, + [property: JsonPropertyName("last_updated")] string LastUpdated, + [property: JsonPropertyName("products")] ImmutableArray Products, + [property: JsonPropertyName("statement"), JsonIgnore(Condition = JsonIgnoreCondition.WhenWritingNull)] string? Statement, + [property: JsonPropertyName("sources")] ImmutableArray Sources); + +internal sealed record OpenVexExportProduct( + [property: JsonPropertyName("id")] string Id, + [property: JsonPropertyName("name")] string Name, + [property: JsonPropertyName("version"), JsonIgnore(Condition = JsonIgnoreCondition.WhenWritingNull)] string? Version, + [property: JsonPropertyName("purl"), JsonIgnore(Condition = JsonIgnoreCondition.WhenWritingNull)] string? Purl, + [property: JsonPropertyName("cpe"), JsonIgnore(Condition = JsonIgnoreCondition.WhenWritingNull)] string? Cpe); + +internal sealed record OpenVexExportSource( + [property: JsonPropertyName("provider")] string Provider, + [property: JsonPropertyName("status")] string Status, + [property: JsonPropertyName("justification"), JsonIgnore(Condition = JsonIgnoreCondition.WhenWritingNull)] string? Justification, + [property: JsonPropertyName("document_digest")] string DocumentDigest, + [property: JsonPropertyName("source_uri")] string SourceUri, + [property: JsonPropertyName("detail"), JsonIgnore(Condition = JsonIgnoreCondition.WhenWritingNull)] string? Detail, + [property: JsonPropertyName("first_observed")] string FirstObserved, + [property: JsonPropertyName("last_observed")] string LastObserved); + +internal sealed record OpenVexExportMetadata( + [property: JsonPropertyName("generated_at")] string GeneratedAt, + [property: JsonPropertyName("query_signature")] string QuerySignature, + [property: JsonPropertyName("source_providers")] ImmutableArray SourceProviders, + [property: JsonPropertyName("diagnostics")] ImmutableDictionary Diagnostics); diff --git a/src/Excititor/__Libraries/StellaOps.Excititor.Formats.OpenVEX/OpenVexStatementMerger.cs b/src/Excititor/__Libraries/StellaOps.Excititor.Formats.OpenVEX/OpenVexStatementMerger.cs new file mode 100644 index 00000000..3c858b50 --- /dev/null +++ b/src/Excititor/__Libraries/StellaOps.Excititor.Formats.OpenVEX/OpenVexStatementMerger.cs @@ -0,0 +1,282 @@ +using System.Collections.Generic; +using System.Collections.Immutable; +using System.Globalization; +using System.Linq; +using StellaOps.Excititor.Core; + +namespace StellaOps.Excititor.Formats.OpenVEX; + +/// +/// Provides deterministic merging utilities for OpenVEX statements derived from normalized VEX claims. +/// +public static class OpenVexStatementMerger +{ + private static readonly ImmutableDictionary StatusRiskPrecedence = new Dictionary + { + [VexClaimStatus.Affected] = 3, + [VexClaimStatus.UnderInvestigation] = 2, + [VexClaimStatus.Fixed] = 1, + [VexClaimStatus.NotAffected] = 0, + }.ToImmutableDictionary(); + + public static OpenVexMergeResult Merge(IEnumerable claims) + { + ArgumentNullException.ThrowIfNull(claims); + + var statements = new List(); + var diagnostics = new Dictionary>(StringComparer.Ordinal); + + foreach (var group in claims + .Where(static claim => claim is not null) + .GroupBy(static claim => (claim.VulnerabilityId, claim.Product.Key))) + { + var orderedClaims = group + .OrderBy(static claim => claim.ProviderId, StringComparer.Ordinal) + .ThenBy(static claim => claim.Document.Digest, StringComparer.Ordinal) + .ToImmutableArray(); + + if (orderedClaims.IsDefaultOrEmpty) + { + continue; + } + + var mergedProduct = MergeProduct(orderedClaims); + var sources = BuildSources(orderedClaims); + var firstSeen = orderedClaims.Min(static claim => claim.FirstSeen); + var lastSeen = orderedClaims.Max(static claim => claim.LastSeen); + var statusSet = orderedClaims + .Select(static claim => claim.Status) + .Distinct() + .ToArray(); + + if (statusSet.Length > 1) + { + AddDiagnostic( + diagnostics, + "openvex.status_conflict", + FormattableString.Invariant($"{group.Key.VulnerabilityId}:{group.Key.Key}={string.Join('|', statusSet.Select(static status => status.ToString().ToLowerInvariant()))}")); + } + + var canonicalStatus = SelectCanonicalStatus(statusSet); + var justification = SelectJustification(canonicalStatus, orderedClaims, diagnostics, group.Key); + + if (canonicalStatus == VexClaimStatus.NotAffected && justification is null) + { + AddDiagnostic( + diagnostics, + "policy.justification_missing", + FormattableString.Invariant($"{group.Key.VulnerabilityId}:{group.Key.Key}")); + } + + var detail = BuildDetail(orderedClaims); + + statements.Add(new OpenVexMergedStatement( + group.Key.VulnerabilityId, + mergedProduct, + canonicalStatus, + justification, + detail, + sources, + firstSeen, + lastSeen)); + } + + var orderedStatements = statements + .OrderBy(static statement => statement.VulnerabilityId, StringComparer.Ordinal) + .ThenBy(static statement => statement.Product.Key, StringComparer.Ordinal) + .ToImmutableArray(); + + var orderedDiagnostics = diagnostics.Count == 0 + ? ImmutableDictionary.Empty + : diagnostics.ToImmutableDictionary( + static pair => pair.Key, + pair => string.Join(",", pair.Value.OrderBy(static entry => entry, StringComparer.Ordinal)), + StringComparer.Ordinal); + + return new OpenVexMergeResult(orderedStatements, orderedDiagnostics); + } + + private static VexClaimStatus SelectCanonicalStatus(IReadOnlyCollection statuses) + { + if (statuses.Count == 0) + { + return VexClaimStatus.UnderInvestigation; + } + + return statuses + .OrderByDescending(static status => StatusRiskPrecedence.GetValueOrDefault(status, -1)) + .ThenBy(static status => status.ToString(), StringComparer.Ordinal) + .First(); + } + + private static VexJustification? SelectJustification( + VexClaimStatus canonicalStatus, + ImmutableArray claims, + IDictionary> diagnostics, + (string Vulnerability, string ProductKey) groupKey) + { + var relevantClaims = claims + .Where(claim => claim.Status == canonicalStatus) + .ToArray(); + + if (relevantClaims.Length == 0) + { + relevantClaims = claims.ToArray(); + } + + var justifications = relevantClaims + .Select(static claim => claim.Justification) + .Where(static justification => justification is not null) + .Cast() + .Distinct() + .ToArray(); + + if (justifications.Length == 0) + { + return null; + } + + if (justifications.Length > 1) + { + AddDiagnostic( + diagnostics, + "openvex.justification_conflict", + FormattableString.Invariant($"{groupKey.Vulnerability}:{groupKey.ProductKey}={string.Join('|', justifications.Select(static justification => justification.ToString().ToLowerInvariant()))}")); + } + + return justifications + .OrderBy(static justification => justification.ToString(), StringComparer.Ordinal) + .First(); + } + + private static string? BuildDetail(ImmutableArray claims) + { + var details = claims + .Select(static claim => claim.Detail) + .Where(static detail => !string.IsNullOrWhiteSpace(detail)) + .Select(static detail => detail!.Trim()) + .Distinct(StringComparer.Ordinal) + .ToArray(); + + if (details.Length == 0) + { + return null; + } + + return string.Join("; ", details.OrderBy(static detail => detail, StringComparer.Ordinal)); + } + + private static ImmutableArray BuildSources(ImmutableArray claims) + { + var builder = ImmutableArray.CreateBuilder(claims.Length); + foreach (var claim in claims) + { + builder.Add(new OpenVexSourceEntry( + claim.ProviderId, + claim.Status, + claim.Justification, + claim.Document.Digest, + claim.Document.SourceUri, + claim.Detail, + claim.FirstSeen, + claim.LastSeen)); + } + + return builder + .ToImmutable() + .OrderBy(static source => source.ProviderId, StringComparer.Ordinal) + .ThenBy(static source => source.DocumentDigest, StringComparer.Ordinal) + .ToImmutableArray(); + } + + private static VexProduct MergeProduct(ImmutableArray claims) + { + var key = claims[0].Product.Key; + var names = claims + .Select(static claim => claim.Product.Name) + .Where(static name => !string.IsNullOrWhiteSpace(name)) + .Select(static name => name!) + .Distinct(StringComparer.Ordinal) + .ToArray(); + + var versions = claims + .Select(static claim => claim.Product.Version) + .Where(static version => !string.IsNullOrWhiteSpace(version)) + .Select(static version => version!) + .Distinct(StringComparer.Ordinal) + .ToArray(); + + var purls = claims + .Select(static claim => claim.Product.Purl) + .Where(static purl => !string.IsNullOrWhiteSpace(purl)) + .Select(static purl => purl!) + .Distinct(StringComparer.OrdinalIgnoreCase) + .ToArray(); + + var cpes = claims + .Select(static claim => claim.Product.Cpe) + .Where(static cpe => !string.IsNullOrWhiteSpace(cpe)) + .Select(static cpe => cpe!) + .Distinct(StringComparer.OrdinalIgnoreCase) + .ToArray(); + + var identifiers = claims + .SelectMany(static claim => claim.Product.ComponentIdentifiers) + .Where(static identifier => !string.IsNullOrWhiteSpace(identifier)) + .Select(static identifier => identifier!) + .Distinct(StringComparer.OrdinalIgnoreCase) + .OrderBy(static identifier => identifier, StringComparer.OrdinalIgnoreCase) + .ToImmutableArray(); + + return new VexProduct( + key, + names.Length == 0 ? claims[0].Product.Name : names.OrderByDescending(static name => name.Length).ThenBy(static name => name, StringComparer.Ordinal).First(), + versions.Length == 0 ? claims[0].Product.Version : versions.OrderByDescending(static version => version.Length).ThenBy(static version => version, StringComparer.Ordinal).First(), + purls.Length == 0 ? claims[0].Product.Purl : purls.OrderBy(static purl => purl, StringComparer.OrdinalIgnoreCase).First(), + cpes.Length == 0 ? claims[0].Product.Cpe : cpes.OrderBy(static cpe => cpe, StringComparer.OrdinalIgnoreCase).First(), + identifiers); + } + + private static void AddDiagnostic( + IDictionary> diagnostics, + string code, + string value) + { + if (!diagnostics.TryGetValue(code, out var entries)) + { + entries = new SortedSet(StringComparer.Ordinal); + diagnostics[code] = entries; + } + + entries.Add(value); + } +} + +public sealed record OpenVexMergeResult( + ImmutableArray Statements, + ImmutableDictionary Diagnostics); + +public sealed record OpenVexMergedStatement( + string VulnerabilityId, + VexProduct Product, + VexClaimStatus Status, + VexJustification? Justification, + string? Detail, + ImmutableArray Sources, + DateTimeOffset FirstObserved, + DateTimeOffset LastObserved); + +public sealed record OpenVexSourceEntry( + string ProviderId, + VexClaimStatus Status, + VexJustification? Justification, + string DocumentDigest, + Uri DocumentSource, + string? Detail, + DateTimeOffset FirstSeen, + DateTimeOffset LastSeen) +{ + public string DocumentDigest { get; } = string.IsNullOrWhiteSpace(DocumentDigest) + ? throw new ArgumentException("Document digest must be provided.", nameof(DocumentDigest)) + : DocumentDigest.Trim(); +} diff --git a/src/Excititor/__Libraries/StellaOps.Excititor.Formats.OpenVEX/ServiceCollectionExtensions.cs b/src/Excititor/__Libraries/StellaOps.Excititor.Formats.OpenVEX/ServiceCollectionExtensions.cs index 392392fb..b830cd29 100644 --- a/src/Excititor/__Libraries/StellaOps.Excititor.Formats.OpenVEX/ServiceCollectionExtensions.cs +++ b/src/Excititor/__Libraries/StellaOps.Excititor.Formats.OpenVEX/ServiceCollectionExtensions.cs @@ -5,10 +5,11 @@ namespace StellaOps.Excititor.Formats.OpenVEX; public static class OpenVexFormatsServiceCollectionExtensions { - public static IServiceCollection AddOpenVexNormalizer(this IServiceCollection services) - { - ArgumentNullException.ThrowIfNull(services); - services.AddSingleton(); - return services; - } -} + public static IServiceCollection AddOpenVexNormalizer(this IServiceCollection services) + { + ArgumentNullException.ThrowIfNull(services); + services.AddSingleton(); + services.AddSingleton(); + return services; + } +} diff --git a/src/Excititor/__Libraries/StellaOps.Excititor.Formats.OpenVEX/TASKS.md b/src/Excititor/__Libraries/StellaOps.Excititor.Formats.OpenVEX/TASKS.md index d736ca08..6a84726a 100644 --- a/src/Excititor/__Libraries/StellaOps.Excititor.Formats.OpenVEX/TASKS.md +++ b/src/Excititor/__Libraries/StellaOps.Excititor.Formats.OpenVEX/TASKS.md @@ -3,5 +3,5 @@ If you are working on this file you need to read docs/modules/excititor/ARCHITEC | Task | Owner(s) | Depends on | Notes | |---|---|---|---| |EXCITITOR-FMT-OPENVEX-01-001 – OpenVEX normalizer|Team Excititor Formats|EXCITITOR-CORE-01-001|**DONE (2025-10-17)** – OpenVEX normalizer parses statements/products, maps status/justification, and surfaces provenance metadata; coverage in `OpenVexNormalizerTests`.| -|EXCITITOR-FMT-OPENVEX-01-002 – Statement merge utilities|Team Excititor Formats|EXCITITOR-FMT-OPENVEX-01-001|**DOING (2025-10-19)** – Prereq EXCITITOR-FMT-OPENVEX-01-001 confirmed DONE; building deterministic merge reducers with policy diagnostics.| -|EXCITITOR-FMT-OPENVEX-01-003 – OpenVEX export writer|Team Excititor Formats|EXCITITOR-EXPORT-01-001, EXCITITOR-FMT-OPENVEX-01-001|**DOING (2025-10-19)** – Prereqs EXCITITOR-EXPORT-01-001 & EXCITITOR-FMT-OPENVEX-01-001 verified DONE; starting canonical OpenVEX exporter with stable ordering/SBOM references.| +|EXCITITOR-FMT-OPENVEX-01-002 – Statement merge utilities|Team Excititor Formats|EXCITITOR-FMT-OPENVEX-01-001|**DONE (2025-10-29)** – Delivered deterministic statement merger prioritising risk status, preserving source provenance, and surfacing conflict diagnostics.| +|EXCITITOR-FMT-OPENVEX-01-003 – OpenVEX export writer|Team Excititor Formats|EXCITITOR-EXPORT-01-001, EXCITITOR-FMT-OPENVEX-01-001|**DONE (2025-10-29)** – Shipped canonical OpenVEX exporter emitting merged statements, metadata, and stable digests for attested distribution.| diff --git a/src/Excititor/__Tests/StellaOps.Excititor.Attestation.Tests/StellaOps.Excititor.Attestation.Tests.csproj b/src/Excititor/__Tests/StellaOps.Excititor.Attestation.Tests/StellaOps.Excititor.Attestation.Tests.csproj index c61c75b4..99ba9599 100644 --- a/src/Excititor/__Tests/StellaOps.Excititor.Attestation.Tests/StellaOps.Excititor.Attestation.Tests.csproj +++ b/src/Excititor/__Tests/StellaOps.Excititor.Attestation.Tests/StellaOps.Excititor.Attestation.Tests.csproj @@ -6,9 +6,14 @@ enable enable true + false + + + + - \ No newline at end of file + diff --git a/src/Excititor/__Tests/StellaOps.Excititor.Attestation.Tests/VexAttestationVerifierTests.cs b/src/Excititor/__Tests/StellaOps.Excititor.Attestation.Tests/VexAttestationVerifierTests.cs index f67911b4..cdf66af5 100644 --- a/src/Excititor/__Tests/StellaOps.Excititor.Attestation.Tests/VexAttestationVerifierTests.cs +++ b/src/Excititor/__Tests/StellaOps.Excititor.Attestation.Tests/VexAttestationVerifierTests.cs @@ -67,6 +67,45 @@ public sealed class VexAttestationVerifierTests : IDisposable Assert.Equal("offline", verification.Diagnostics["rekor.state"]); } + [Fact] + public async Task VerifyAsync_ReturnsInvalid_WhenTransparencyRequiredAndMissing() + { + var (request, metadata, envelope) = await CreateSignedAttestationAsync(includeRekor: false); + var verifier = CreateVerifier(options => + { + options.RequireTransparencyLog = true; + options.AllowOfflineTransparency = false; + }); + + var verification = await verifier.VerifyAsync( + new VexAttestationVerificationRequest(request, metadata, envelope), + CancellationToken.None); + + Assert.False(verification.IsValid); + Assert.Equal("missing", verification.Diagnostics["rekor.state"]); + Assert.Equal("invalid", verification.Diagnostics["result"]); + } + + [Fact] + public async Task VerifyAsync_ReturnsInvalid_WhenTransparencyUnavailableAndOfflineDisallowed() + { + var (request, metadata, envelope) = await CreateSignedAttestationAsync(includeRekor: true); + var transparency = new ThrowingTransparencyLogClient(); + var verifier = CreateVerifier(options => + { + options.RequireTransparencyLog = true; + options.AllowOfflineTransparency = false; + }, transparency); + + var verification = await verifier.VerifyAsync( + new VexAttestationVerificationRequest(request, metadata, envelope), + CancellationToken.None); + + Assert.False(verification.IsValid); + Assert.Equal("unreachable", verification.Diagnostics["rekor.state"]); + Assert.Equal("invalid", verification.Diagnostics["result"]); + } + private async Task<(VexAttestationRequest Request, VexAttestationMetadata Metadata, string Envelope)> CreateSignedAttestationAsync(bool includeRekor = false) { var signer = new FakeSigner(); diff --git a/src/Excititor/__Tests/StellaOps.Excititor.Connectors.Cisco.CSAF.Tests/Connectors/CiscoCsafConnectorTests.cs b/src/Excititor/__Tests/StellaOps.Excititor.Connectors.Cisco.CSAF.Tests/Connectors/CiscoCsafConnectorTests.cs index bbc581c6..f5f88e0e 100644 --- a/src/Excititor/__Tests/StellaOps.Excititor.Connectors.Cisco.CSAF.Tests/Connectors/CiscoCsafConnectorTests.cs +++ b/src/Excititor/__Tests/StellaOps.Excititor.Connectors.Cisco.CSAF.Tests/Connectors/CiscoCsafConnectorTests.cs @@ -107,9 +107,124 @@ public sealed class CiscoCsafConnectorTests documents.Add(doc); } - documents.Should().BeEmpty(); - sink.Documents.Should().BeEmpty(); - } + documents.Should().BeEmpty(); + sink.Documents.Should().BeEmpty(); + } + + [Fact] + public async Task FetchAsync_EmitsTrustMetadataAndUpsertsProvider() + { + var metadataResponse = """ + { + "metadata": { + "publisher": { + "name": "Cisco", + "category": "vendor", + "contact_details": { "id": "excititor:cisco" } + } + }, + "trust": { + "weight": 0.75, + "cosign": { + "issuer": "https://issuer.example.com", + "identity_pattern": "https://sig.example.com/*" + }, + "pgp_fingerprints": [ + "0123456789ABCDEF", + "FEDCBA9876543210" + ] + }, + "distributions": { + "directories": [ "https://api.cisco.test/csaf/" ] + } + } + """; + + var responses = new Dictionary> + { + [new Uri("https://api.cisco.test/.well-known/csaf/provider-metadata.json")] = QueueResponses(metadataResponse), + [new Uri("https://api.cisco.test/csaf/index.json")] = QueueResponses(""" + { + "advisories": [ + { + "id": "cisco-sa-2025", + "url": "https://api.cisco.test/csaf/cisco-sa-2025.json", + "published": "2025-10-01T00:00:00Z", + "lastModified": "2025-10-02T00:00:00Z", + "sha256": "cafebabe" + } + ] + } + """), + [new Uri("https://api.cisco.test/csaf/cisco-sa-2025.json")] = QueueResponses("{ \"document\": \"payload\" }") + }; + + var handler = new RoutingHttpMessageHandler(responses); + var httpClient = new HttpClient(handler); + var factory = new SingleHttpClientFactory(httpClient); + var connectorOptions = new CiscoConnectorOptions + { + MetadataUri = "https://api.cisco.test/.well-known/csaf/provider-metadata.json", + PersistOfflineSnapshot = false, + }; + + var metadataLoader = new CiscoProviderMetadataLoader( + factory, + new MemoryCache(new MemoryCacheOptions()), + Options.Create(connectorOptions), + NullLogger.Instance, + new MockFileSystem()); + + var stateRepository = new InMemoryConnectorStateRepository(); + var connector = new CiscoCsafConnector( + metadataLoader, + factory, + stateRepository, + new[] { new CiscoConnectorOptionsValidator() }, + NullLogger.Instance, + TimeProvider.System); + + await connector.ValidateAsync(VexConnectorSettings.Empty, CancellationToken.None); + + var providerStore = new StubProviderStore(); + var services = new ServiceCollection() + .AddSingleton(providerStore) + .BuildServiceProvider(); + + var sink = new InMemoryRawSink(); + var context = new VexConnectorContext( + null, + VexConnectorSettings.Empty, + sink, + new NoopSignatureVerifier(), + new NoopNormalizerRouter(), + services, + ImmutableDictionary.Empty); + + var documents = new List(); + await foreach (var doc in connector.FetchAsync(context, CancellationToken.None)) + { + documents.Add(doc); + } + + documents.Should().HaveCount(1); + var metadata = documents[0].Metadata; + metadata.Should().Contain("vex.provenance.provider", "excititor:cisco"); + metadata.Should().Contain("vex.provenance.providerName", "Cisco"); + metadata.Should().Contain("vex.provenance.trust.weight", "0.75"); + metadata.Should().Contain("vex.provenance.cosign.issuer", "https://issuer.example.com"); + metadata.Should().Contain("vex.provenance.cosign.identityPattern", "https://sig.example.com/*"); + metadata.Should().Contain("vex.provenance.pgp.fingerprints", "0123456789ABCDEF,FEDCBA9876543210"); + + providerStore.SavedProviders.Should().HaveCount(1); + var savedProvider = providerStore.SavedProviders[0]; + savedProvider.Id.Should().Be("excititor:cisco"); + savedProvider.Trust.Weight.Should().Be(0.75); + savedProvider.Trust.Cosign.Should().NotBeNull(); + savedProvider.Trust.Cosign!.Issuer.Should().Be("https://issuer.example.com"); + savedProvider.Trust.Cosign.IdentityPattern.Should().Be("https://sig.example.com/*"); + savedProvider.Trust.PgpFingerprints.Should().Contain(new[] { "0123456789ABCDEF", "FEDCBA9876543210" }); + } private static Queue QueueResponses(string payload) => new(new[] @@ -156,19 +271,36 @@ public sealed class CiscoCsafConnectorTests public HttpClient CreateClient(string name) => _client; } - private sealed class InMemoryConnectorStateRepository : IVexConnectorStateRepository - { - public VexConnectorState? CurrentState { get; private set; } - - public ValueTask GetAsync(string connectorId, CancellationToken cancellationToken, IClientSessionHandle? session = null) - => ValueTask.FromResult(CurrentState); - - public ValueTask SaveAsync(VexConnectorState state, CancellationToken cancellationToken, IClientSessionHandle? session = null) - { - CurrentState = state; - return ValueTask.CompletedTask; - } - } + private sealed class InMemoryConnectorStateRepository : IVexConnectorStateRepository + { + public VexConnectorState? CurrentState { get; private set; } + + public ValueTask GetAsync(string connectorId, CancellationToken cancellationToken, IClientSessionHandle? session = null) + => ValueTask.FromResult(CurrentState); + + public ValueTask SaveAsync(VexConnectorState state, CancellationToken cancellationToken, IClientSessionHandle? session = null) + { + CurrentState = state; + return ValueTask.CompletedTask; + } + } + + private sealed class StubProviderStore : IVexProviderStore + { + public List SavedProviders { get; } = new(); + + public ValueTask FindAsync(string id, CancellationToken cancellationToken, IClientSessionHandle? session = null) + => ValueTask.FromResult(null); + + public ValueTask> ListAsync(CancellationToken cancellationToken, IClientSessionHandle? session = null) + => ValueTask.FromResult>(Array.Empty()); + + public ValueTask SaveAsync(VexProvider provider, CancellationToken cancellationToken, IClientSessionHandle? session = null) + { + SavedProviders.Add(provider); + return ValueTask.CompletedTask; + } + } private sealed class InMemoryRawSink : IVexRawDocumentSink { diff --git a/src/Excititor/__Tests/StellaOps.Excititor.Connectors.Cisco.CSAF.Tests/StellaOps.Excititor.Connectors.Cisco.CSAF.Tests.csproj b/src/Excititor/__Tests/StellaOps.Excititor.Connectors.Cisco.CSAF.Tests/StellaOps.Excititor.Connectors.Cisco.CSAF.Tests.csproj index dd1a39fd..b3d04089 100644 --- a/src/Excititor/__Tests/StellaOps.Excititor.Connectors.Cisco.CSAF.Tests/StellaOps.Excititor.Connectors.Cisco.CSAF.Tests.csproj +++ b/src/Excititor/__Tests/StellaOps.Excititor.Connectors.Cisco.CSAF.Tests/StellaOps.Excititor.Connectors.Cisco.CSAF.Tests.csproj @@ -6,12 +6,17 @@ enable enable true + false + + + + - \ No newline at end of file + diff --git a/src/Excititor/__Tests/StellaOps.Excititor.Connectors.SUSE.RancherVEXHub.Tests/Connectors/RancherHubConnectorTests.cs b/src/Excititor/__Tests/StellaOps.Excititor.Connectors.SUSE.RancherVEXHub.Tests/Connectors/RancherHubConnectorTests.cs new file mode 100644 index 00000000..5fa3dca3 --- /dev/null +++ b/src/Excititor/__Tests/StellaOps.Excititor.Connectors.SUSE.RancherVEXHub.Tests/Connectors/RancherHubConnectorTests.cs @@ -0,0 +1,429 @@ +using System.Collections.Immutable; +using System.Globalization; +using System.Net; +using System.Net.Http; +using System.Security.Cryptography; +using System.Text; +using FluentAssertions; +using Microsoft.Extensions.Caching.Memory; +using Microsoft.Extensions.DependencyInjection; +using Microsoft.Extensions.Logging.Abstractions; +using StellaOps.Excititor.Connectors.Abstractions; +using StellaOps.Excititor.Connectors.SUSE.RancherVEXHub; +using StellaOps.Excititor.Connectors.SUSE.RancherVEXHub.Configuration; +using StellaOps.Excititor.Connectors.SUSE.RancherVEXHub.Events; +using StellaOps.Excititor.Connectors.SUSE.RancherVEXHub.Metadata; +using StellaOps.Excititor.Connectors.SUSE.RancherVEXHub.State; +using StellaOps.Excititor.Core; +using StellaOps.Excititor.Storage.Mongo; +using Xunit; + +namespace StellaOps.Excititor.Connectors.SUSE.RancherVEXHub.Tests.Connectors; + +public sealed class RancherHubConnectorTests +{ + [Fact] + public async Task FetchAsync_OfflineSnapshot_StoresDocumentAndUpdatesCheckpoint() + { + using var fixture = await ConnectorFixture.CreateAsync(); + + var sink = new InMemoryRawSink(); + var context = fixture.CreateContext(sink); + + var documents = await CollectAsync(fixture.Connector.FetchAsync(context, CancellationToken.None)); + + documents.Should().HaveCount(1); + var document = documents[0]; + document.Digest.Should().Be(fixture.ExpectedDocumentDigest); + document.Metadata.Should().ContainKey("rancher.event.id").WhoseValue.Should().Be("evt-1"); + document.Metadata.Should().ContainKey("rancher.event.cursor").WhoseValue.Should().Be("cursor-2"); + sink.Documents.Should().HaveCount(1); + + var state = fixture.StateRepository.State; + state.Should().NotBeNull(); + state!.LastUpdated.Should().Be(DateTimeOffset.Parse("2025-10-19T12:00:00Z", CultureInfo.InvariantCulture, DateTimeStyles.AssumeUniversal)); + state.DocumentDigests.Should().Contain(fixture.ExpectedDocumentDigest); + state.DocumentDigests.Should().Contain("checkpoint:cursor-2"); + state.DocumentDigests.Count.Should().BeLessOrEqualTo(ConnectorFixture.MaxDigestHistory + 1); + } + + [Fact] + public async Task FetchAsync_WhenDocumentDownloadFails_QuarantinesEvent() + { + using var fixture = await ConnectorFixture.CreateAsync(); + + fixture.Handler.SetRoute(fixture.DocumentUri, () => new HttpResponseMessage(HttpStatusCode.InternalServerError)); + + var sink = new InMemoryRawSink(); + var context = fixture.CreateContext(sink); + + var documents = await CollectAsync(fixture.Connector.FetchAsync(context, CancellationToken.None)); + + documents.Should().BeEmpty(); + sink.Documents.Should().HaveCount(1); + var quarantined = sink.Documents[0]; + quarantined.Metadata.Should().Contain("rancher.event.quarantine", "true"); + quarantined.Metadata.Should().ContainKey("rancher.event.error").WhoseValue.Should().Contain("document fetch failed"); + + var state = fixture.StateRepository.State; + state.Should().NotBeNull(); + state!.DocumentDigests.Should().Contain(d => d.StartsWith("quarantine:", StringComparison.Ordinal)); + } + + [Fact] + public async Task FetchAsync_ReplayingSnapshot_SkipsDuplicateDocuments() + { + using var fixture = await ConnectorFixture.CreateAsync(); + + var firstSink = new InMemoryRawSink(); + var firstContext = fixture.CreateContext(firstSink); + await CollectAsync(fixture.Connector.FetchAsync(firstContext, CancellationToken.None)); + + var secondSink = new InMemoryRawSink(); + var secondContext = fixture.CreateContext(secondSink); + var secondRunDocuments = await CollectAsync(fixture.Connector.FetchAsync(secondContext, CancellationToken.None)); + + secondRunDocuments.Should().BeEmpty(); + secondSink.Documents.Should().BeEmpty(); + + var state = fixture.StateRepository.State; + state.Should().NotBeNull(); + state!.DocumentDigests.Should().Contain(fixture.ExpectedDocumentDigest); + } + + [Fact] + public async Task FetchAsync_TrimsPersistedDigestHistory() + { + var existingDigests = Enumerable.Range(0, ConnectorFixture.MaxDigestHistory + 5) + .Select(i => $"sha256:{i:X32}") + .ToImmutableArray(); + var initialState = new VexConnectorState( + "excititor:suse.rancher", + DateTimeOffset.Parse("2025-10-18T00:00:00Z", CultureInfo.InvariantCulture, DateTimeStyles.AssumeUniversal), + ImmutableArray.CreateBuilder() + .Add("checkpoint:cursor-old") + .AddRange(existingDigests) + .ToImmutable()); + + using var fixture = await ConnectorFixture.CreateAsync(initialState); + + var sink = new InMemoryRawSink(); + var context = fixture.CreateContext(sink); + await CollectAsync(fixture.Connector.FetchAsync(context, CancellationToken.None)); + + var state = fixture.StateRepository.State; + state.Should().NotBeNull(); + state!.DocumentDigests.Should().Contain(d => d.StartsWith("checkpoint:", StringComparison.Ordinal)); + state.DocumentDigests.Count.Should().Be(ConnectorFixture.MaxDigestHistory + 1); + } + + private static async Task> CollectAsync(IAsyncEnumerable source) + { + var list = new List(); + await foreach (var document in source.ConfigureAwait(false)) + { + list.Add(document); + } + + return list; + } + + #region helpers + + private sealed class ConnectorFixture : IDisposable + { + public const int MaxDigestHistory = 200; + + private readonly IServiceProvider _serviceProvider; + private readonly TempDirectory _tempDirectory; + private readonly HttpClient _httpClient; + + private ConnectorFixture( + RancherHubConnector connector, + InMemoryConnectorStateRepository stateRepository, + RoutingHttpMessageHandler handler, + IServiceProvider serviceProvider, + TempDirectory tempDirectory, + HttpClient httpClient, + Uri documentUri, + string documentDigest) + { + Connector = connector; + StateRepository = stateRepository; + Handler = handler; + _serviceProvider = serviceProvider; + _tempDirectory = tempDirectory; + _httpClient = httpClient; + DocumentUri = documentUri; + ExpectedDocumentDigest = $"sha256:{documentDigest}"; + } + + public RancherHubConnector Connector { get; } + + public InMemoryConnectorStateRepository StateRepository { get; } + + public RoutingHttpMessageHandler Handler { get; } + + public Uri DocumentUri { get; } + + public string ExpectedDocumentDigest { get; } + + public VexConnectorContext CreateContext(InMemoryRawSink sink, DateTimeOffset? since = null) + => new( + since, + VexConnectorSettings.Empty, + sink, + new NoopSignatureVerifier(), + new NoopNormalizerRouter(), + _serviceProvider, + ImmutableDictionary.Empty); + + public void Dispose() + { + _httpClient.Dispose(); + _tempDirectory.Dispose(); + } + + public static async Task CreateAsync(VexConnectorState? initialState = null) + { + var tempDirectory = new TempDirectory(); + var documentPayload = "{\"document\":\"payload\"}"; + var documentDigest = ComputeSha256Hex(documentPayload); + + var documentUri = new Uri("https://hub.test/events/evt-1.json"); + var eventsPayload = """ + { + "cursor": "cursor-1", + "nextCursor": "cursor-2", + "events": [ + { + "id": "evt-1", + "type": "vex.statement.published", + "channel": "rancher/rke2", + "publishedAt": "2025-10-19T12:00:00Z", + "document": { + "uri": "https://hub.test/events/evt-1.json", + "sha256": "DOC_DIGEST", + "format": "csaf" + } + } + ] + } + """.Replace("DOC_DIGEST", documentDigest, StringComparison.Ordinal); + + var eventsPath = tempDirectory.Combine("events.json"); + await File.WriteAllTextAsync(eventsPath, eventsPayload, Encoding.UTF8).ConfigureAwait(false); + var eventsChecksum = ComputeSha256Hex(eventsPayload); + + var discoveryPayload = """ + { + "hubId": "excititor:suse.rancher", + "title": "SUSE Rancher VEX Hub", + "subscription": { + "eventsUri": "https://hub.test/events", + "checkpointUri": "https://hub.test/checkpoint", + "channels": [ "rancher/rke2" ], + "requiresAuthentication": false + }, + "offline": { + "snapshotUri": "EVENTS_URI", + "sha256": "EVENTS_DIGEST" + } + } + """ + .Replace("EVENTS_URI", new Uri(eventsPath).ToString(), StringComparison.Ordinal) + .Replace("EVENTS_DIGEST", eventsChecksum, StringComparison.Ordinal); + + var discoveryPath = tempDirectory.Combine("discovery.json"); + await File.WriteAllTextAsync(discoveryPath, discoveryPayload, Encoding.UTF8).ConfigureAwait(false); + + var handler = new RoutingHttpMessageHandler(); + handler.SetRoute(documentUri, () => JsonResponse(documentPayload)); + var httpClient = new HttpClient(handler) + { + Timeout = TimeSpan.FromSeconds(10), + }; + var httpFactory = new SingletonHttpClientFactory(httpClient); + + var memoryCache = new MemoryCache(new MemoryCacheOptions()); + var fileSystem = new System.IO.Abstractions.FileSystem(); + var tokenProvider = new RancherHubTokenProvider(httpFactory, memoryCache, NullLogger.Instance); + var metadataLoader = new RancherHubMetadataLoader(httpFactory, memoryCache, tokenProvider, fileSystem, NullLogger.Instance); + var eventClient = new RancherHubEventClient(httpFactory, tokenProvider, fileSystem, NullLogger.Instance); + + var stateRepository = new InMemoryConnectorStateRepository(initialState); + var checkpointManager = new RancherHubCheckpointManager(stateRepository); + + var validators = new[] { new RancherHubConnectorOptionsValidator(fileSystem) }; + var connector = new RancherHubConnector( + metadataLoader, + eventClient, + checkpointManager, + tokenProvider, + httpFactory, + NullLogger.Instance, + TimeProvider.System, + validators); + + var settingsValues = ImmutableDictionary.CreateBuilder(StringComparer.OrdinalIgnoreCase); + settingsValues["DiscoveryUri"] = "https://hub.test/.well-known/rancher-hub.json"; + settingsValues["OfflineSnapshotPath"] = discoveryPath; + settingsValues["PreferOfflineSnapshot"] = "true"; + var settings = new VexConnectorSettings(settingsValues.ToImmutable()); + await connector.ValidateAsync(settings, CancellationToken.None).ConfigureAwait(false); + + var services = new ServiceCollection().BuildServiceProvider(); + + return new ConnectorFixture( + connector, + stateRepository, + handler, + services, + tempDirectory, + httpClient, + documentUri, + documentDigest); + } + + private static HttpResponseMessage JsonResponse(string payload) + { + var response = new HttpResponseMessage(HttpStatusCode.OK) + { + Content = new StringContent(payload, Encoding.UTF8, "application/json"), + }; + return response; + } + } + + private sealed class SingletonHttpClientFactory : IHttpClientFactory + { + private readonly HttpClient _client; + + public SingletonHttpClientFactory(HttpClient client) + { + _client = client; + } + + public HttpClient CreateClient(string name) => _client; + } + + private sealed class RoutingHttpMessageHandler : HttpMessageHandler + { + private readonly Dictionary>> _routes = new(); + + public void SetRoute(Uri uri, params Func[] responders) + { + ArgumentNullException.ThrowIfNull(uri); + if (responders is null || responders.Length == 0) + { + _routes.Remove(uri); + return; + } + + _routes[uri] = new Queue>(responders); + } + + protected override Task SendAsync(HttpRequestMessage request, CancellationToken cancellationToken) + { + if (request.RequestUri is not null && + _routes.TryGetValue(request.RequestUri, out var queue) && + queue.Count > 0) + { + var responder = queue.Count > 1 ? queue.Dequeue() : queue.Peek(); + var response = responder(); + response.RequestMessage = request; + return Task.FromResult(response); + } + + return Task.FromResult(new HttpResponseMessage(HttpStatusCode.NotFound) + { + Content = new StringContent($"No response configured for {request.RequestUri}", Encoding.UTF8, "text/plain"), + }); + } + } + + private sealed class InMemoryConnectorStateRepository : IVexConnectorStateRepository + { + public InMemoryConnectorStateRepository(VexConnectorState? initialState = null) + { + State = initialState; + } + + public VexConnectorState? State { get; private set; } + + public ValueTask GetAsync(string connectorId, CancellationToken cancellationToken, MongoDB.Driver.IClientSessionHandle? session = null) + => ValueTask.FromResult(State); + + public ValueTask SaveAsync(VexConnectorState state, CancellationToken cancellationToken, MongoDB.Driver.IClientSessionHandle? session = null) + { + State = state; + return ValueTask.CompletedTask; + } + } + + private sealed class InMemoryRawSink : IVexRawDocumentSink + { + public List Documents { get; } = new(); + + public ValueTask StoreAsync(VexRawDocument document, CancellationToken cancellationToken) + { + Documents.Add(document); + return ValueTask.CompletedTask; + } + } + + private sealed class NoopSignatureVerifier : IVexSignatureVerifier + { + public ValueTask VerifyAsync(VexRawDocument document, CancellationToken cancellationToken) + => ValueTask.FromResult(null); + } + + private sealed class NoopNormalizerRouter : IVexNormalizerRouter + { + public ValueTask NormalizeAsync(VexRawDocument document, CancellationToken cancellationToken) + => ValueTask.FromResult(new VexClaimBatch(document, ImmutableArray.Empty, ImmutableDictionary.Empty)); + } + + private sealed class TempDirectory : IDisposable + { + private readonly string _path; + + public TempDirectory() + { + _path = Path.Combine(Path.GetTempPath(), "stellaops-excititor-tests", Guid.NewGuid().ToString("n")); + Directory.CreateDirectory(_path); + } + + public string Combine(string relative) => Path.Combine(_path, relative); + + public void Dispose() + { + try + { + if (Directory.Exists(_path)) + { + Directory.Delete(_path, recursive: true); + } + } + catch + { + // Best-effort cleanup. + } + } + } + + private static string ComputeSha256Hex(string payload) + { + var bytes = Encoding.UTF8.GetBytes(payload); + return ComputeSha256Hex(bytes); + } + + private static string ComputeSha256Hex(ReadOnlySpan payload) + { + Span buffer = stackalloc byte[32]; + SHA256.HashData(payload, buffer); + return Convert.ToHexString(buffer).ToLowerInvariant(); + } + + #endregion +} diff --git a/src/Excititor/__Tests/StellaOps.Excititor.Connectors.SUSE.RancherVEXHub.Tests/StellaOps.Excititor.Connectors.SUSE.RancherVEXHub.Tests.csproj b/src/Excititor/__Tests/StellaOps.Excititor.Connectors.SUSE.RancherVEXHub.Tests/StellaOps.Excititor.Connectors.SUSE.RancherVEXHub.Tests.csproj index 0c61596e..064d0661 100644 --- a/src/Excititor/__Tests/StellaOps.Excititor.Connectors.SUSE.RancherVEXHub.Tests/StellaOps.Excititor.Connectors.SUSE.RancherVEXHub.Tests.csproj +++ b/src/Excititor/__Tests/StellaOps.Excititor.Connectors.SUSE.RancherVEXHub.Tests/StellaOps.Excititor.Connectors.SUSE.RancherVEXHub.Tests.csproj @@ -6,13 +6,18 @@ enable enable true + false + + + + - \ No newline at end of file + diff --git a/src/Excititor/__Tests/StellaOps.Excititor.Formats.CSAF.Tests/CsafExporterTests.cs b/src/Excititor/__Tests/StellaOps.Excititor.Formats.CSAF.Tests/CsafExporterTests.cs new file mode 100644 index 00000000..bf7a3d86 --- /dev/null +++ b/src/Excititor/__Tests/StellaOps.Excititor.Formats.CSAF.Tests/CsafExporterTests.cs @@ -0,0 +1,73 @@ +using System.Collections.Immutable; +using System.Text.Json; +using FluentAssertions; +using StellaOps.Excititor.Core; +using StellaOps.Excititor.Formats.CSAF; + +namespace StellaOps.Excititor.Formats.CSAF.Tests; + +public sealed class CsafExporterTests +{ + [Fact] + public async Task SerializeAsync_WritesDeterministicCsafDocument() + { + var claims = ImmutableArray.Create( + new VexClaim( + "CVE-2025-3000", + "vendor:example", + new VexProduct("pkg:example/app@1.0.0", "Example App", "1.0.0", "pkg:example/app@1.0.0"), + VexClaimStatus.Affected, + new VexClaimDocument(VexDocumentFormat.Csaf, "sha256:doc1", new Uri("https://example.com/csaf/advisory1.json")), + new DateTimeOffset(2025, 10, 10, 0, 0, 0, TimeSpan.Zero), + new DateTimeOffset(2025, 10, 11, 0, 0, 0, TimeSpan.Zero), + detail: "Impact on Example App 1.0.0"), + new VexClaim( + "CVE-2025-3000", + "vendor:example", + new VexProduct("pkg:example/app@1.0.0", "Example App", "1.0.0", "pkg:example/app@1.0.0"), + VexClaimStatus.NotAffected, + new VexClaimDocument(VexDocumentFormat.Csaf, "sha256:doc2", new Uri("https://example.com/csaf/advisory2.json")), + new DateTimeOffset(2025, 10, 12, 0, 0, 0, TimeSpan.Zero), + new DateTimeOffset(2025, 10, 12, 0, 0, 0, TimeSpan.Zero), + justification: VexJustification.ComponentNotPresent), + new VexClaim( + "ADVISORY-1", + "vendor:example", + new VexProduct("pkg:example/lib@2.0.0", "Example Lib", "2.0.0"), + VexClaimStatus.NotAffected, + new VexClaimDocument(VexDocumentFormat.Csaf, "sha256:doc3", new Uri("https://example.com/csaf/advisory3.json")), + new DateTimeOffset(2025, 10, 12, 0, 0, 0, TimeSpan.Zero), + new DateTimeOffset(2025, 10, 12, 0, 0, 0, TimeSpan.Zero), + justification: null)); + + var request = new VexExportRequest( + VexQuery.Empty, + ImmutableArray.Empty, + claims, + new DateTimeOffset(2025, 10, 13, 0, 0, 0, TimeSpan.Zero)); + + var exporter = new CsafExporter(); + var digest = exporter.Digest(request); + + await using var stream = new MemoryStream(); + var result = await exporter.SerializeAsync(request, stream, CancellationToken.None); + + digest.Should().NotBeNull(); + digest.Should().Be(result.Digest); + + stream.Position = 0; + using var document = JsonDocument.Parse(stream); + var root = document.RootElement; + + root.GetProperty("document").GetProperty("tracking").GetProperty("id").GetString()!.Should().StartWith("stellaops:csaf"); + root.GetProperty("product_tree").GetProperty("full_product_names").GetArrayLength().Should().Be(2); + root.GetProperty("vulnerabilities").EnumerateArray().Should().HaveCount(2); + + var metadata = root.GetProperty("metadata"); + metadata.GetProperty("query_signature").GetString().Should().NotBeNull(); + metadata.GetProperty("diagnostics").EnumerateObject().Select(p => p.Name).Should().Contain("policy.justification_missing"); + + result.Metadata.Should().ContainKey("csaf.vulnerabilityCount"); + result.Metadata["csaf.productCount"].Should().Be("2"); + } +} diff --git a/src/Excititor/__Tests/StellaOps.Excititor.Formats.CSAF.Tests/CsafNormalizerTests.cs b/src/Excititor/__Tests/StellaOps.Excititor.Formats.CSAF.Tests/CsafNormalizerTests.cs index 3f722c8f..0c60a570 100644 --- a/src/Excititor/__Tests/StellaOps.Excititor.Formats.CSAF.Tests/CsafNormalizerTests.cs +++ b/src/Excititor/__Tests/StellaOps.Excititor.Formats.CSAF.Tests/CsafNormalizerTests.cs @@ -127,5 +127,53 @@ public sealed class CsafNormalizerTests claim.AdditionalMetadata["csaf.tracking.id"].Should().Be("RHSA-2025:1001"); claim.AdditionalMetadata["csaf.tracking.status"].Should().Be("final"); claim.AdditionalMetadata["csaf.publisher.name"].Should().Be("Red Hat Product Security"); - } -} + } + + [Fact] + public async Task NormalizeAsync_MissingJustification_AddsPolicyDiagnostic() + { + var json = """ + { + "document": { + "tracking": { + "initial_release_date": "2025-10-02T00:00:00Z", + "current_release_date": "2025-10-03T00:00:00Z" + } + }, + "product_tree": { + "full_product_names": [ + { + "product_id": "pkg:example/app@1.0.0", + "name": "Example App" + } + ] + }, + "vulnerabilities": [ + { + "id": "VULN-1", + "product_status": { + "known_not_affected": [ "pkg:example/app@1.0.0" ] + } + } + ] + } + """; + + var rawDocument = new VexRawDocument( + "excititor:example", + VexDocumentFormat.Csaf, + new Uri("https://example.com/csaf.json"), + new DateTimeOffset(2025, 10, 4, 0, 0, 0, TimeSpan.Zero), + "sha256:digest", + Encoding.UTF8.GetBytes(json), + ImmutableDictionary.Empty); + + var provider = new VexProvider("excititor:example", "Example CSAF", VexProviderKind.Vendor); + var normalizer = new CsafNormalizer(NullLogger.Instance); + + var batch = await normalizer.NormalizeAsync(rawDocument, provider, CancellationToken.None); + + batch.Diagnostics.Should().ContainKey("policy.justification_missing"); + batch.Diagnostics["policy.justification_missing"].Should().Contain("VULN-1:pkg:example/app@1.0.0"); + } +} diff --git a/src/Excititor/__Tests/StellaOps.Excititor.Formats.CycloneDX.Tests/CycloneDxComponentReconcilerTests.cs b/src/Excititor/__Tests/StellaOps.Excititor.Formats.CycloneDX.Tests/CycloneDxComponentReconcilerTests.cs new file mode 100644 index 00000000..70431e07 --- /dev/null +++ b/src/Excititor/__Tests/StellaOps.Excititor.Formats.CycloneDX.Tests/CycloneDxComponentReconcilerTests.cs @@ -0,0 +1,37 @@ +using System.Collections.Immutable; +using FluentAssertions; +using StellaOps.Excititor.Core; +using StellaOps.Excititor.Formats.CycloneDX; + +namespace StellaOps.Excititor.Formats.CycloneDX.Tests; + +public sealed class CycloneDxComponentReconcilerTests +{ + [Fact] + public void Reconcile_AssignsBomRefsAndDiagnostics() + { + var claims = ImmutableArray.Create( + new VexClaim( + "CVE-2025-7000", + "vendor:one", + new VexProduct("pkg:demo/component@1.0.0", "Demo Component", "1.0.0", "pkg:demo/component@1.0.0"), + VexClaimStatus.Affected, + new VexClaimDocument(VexDocumentFormat.CycloneDx, "sha256:doc1", new Uri("https://example.com/vex/1")), + DateTimeOffset.UtcNow, + DateTimeOffset.UtcNow), + new VexClaim( + "CVE-2025-7000", + "vendor:two", + new VexProduct("component-key", "Component Key"), + VexClaimStatus.NotAffected, + new VexClaimDocument(VexDocumentFormat.CycloneDx, "sha256:doc2", new Uri("https://example.com/vex/2")), + DateTimeOffset.UtcNow, + DateTimeOffset.UtcNow)); + + var result = CycloneDxComponentReconciler.Reconcile(claims); + + result.Components.Should().HaveCount(2); + result.ComponentRefs.Should().ContainKey(("CVE-2025-7000", "component-key")); + result.Diagnostics.Keys.Should().Contain("missing_purl"); + } +} diff --git a/src/Excititor/__Tests/StellaOps.Excititor.Formats.CycloneDX.Tests/CycloneDxExporterTests.cs b/src/Excititor/__Tests/StellaOps.Excititor.Formats.CycloneDX.Tests/CycloneDxExporterTests.cs new file mode 100644 index 00000000..99e33f69 --- /dev/null +++ b/src/Excititor/__Tests/StellaOps.Excititor.Formats.CycloneDX.Tests/CycloneDxExporterTests.cs @@ -0,0 +1,47 @@ +using System.Collections.Immutable; +using System.Text.Json; +using FluentAssertions; +using StellaOps.Excititor.Core; +using StellaOps.Excititor.Formats.CycloneDX; + +namespace StellaOps.Excititor.Formats.CycloneDX.Tests; + +public sealed class CycloneDxExporterTests +{ + [Fact] + public async Task SerializeAsync_WritesCycloneDxVexDocument() + { + var claims = ImmutableArray.Create( + new VexClaim( + "CVE-2025-6000", + "vendor:demo", + new VexProduct("pkg:demo/component@1.2.3", "Demo Component", "1.2.3", "pkg:demo/component@1.2.3"), + VexClaimStatus.Fixed, + new VexClaimDocument(VexDocumentFormat.CycloneDx, "sha256:doc1", new Uri("https://example.com/cyclonedx/1")), + new DateTimeOffset(2025, 10, 10, 0, 0, 0, TimeSpan.Zero), + new DateTimeOffset(2025, 10, 11, 0, 0, 0, TimeSpan.Zero), + detail: "Issue resolved in 1.2.3")); + + var request = new VexExportRequest( + VexQuery.Empty, + ImmutableArray.Empty, + claims, + new DateTimeOffset(2025, 10, 12, 0, 0, 0, TimeSpan.Zero)); + + var exporter = new CycloneDxExporter(); + await using var stream = new MemoryStream(); + var result = await exporter.SerializeAsync(request, stream, CancellationToken.None); + + stream.Position = 0; + using var document = JsonDocument.Parse(stream); + var root = document.RootElement; + + root.GetProperty("bomFormat").GetString().Should().Be("CycloneDX"); + root.GetProperty("components").EnumerateArray().Should().HaveCount(1); + root.GetProperty("vulnerabilities").EnumerateArray().Should().HaveCount(1); + + result.Metadata.Should().ContainKey("cyclonedx.vulnerabilityCount"); + result.Metadata["cyclonedx.componentCount"].Should().Be("1"); + result.Digest.Algorithm.Should().Be("sha256"); + } +} diff --git a/src/Excititor/__Tests/StellaOps.Excititor.Formats.OpenVEX.Tests/OpenVexExporterTests.cs b/src/Excititor/__Tests/StellaOps.Excititor.Formats.OpenVEX.Tests/OpenVexExporterTests.cs new file mode 100644 index 00000000..00bc2311 --- /dev/null +++ b/src/Excititor/__Tests/StellaOps.Excititor.Formats.OpenVEX.Tests/OpenVexExporterTests.cs @@ -0,0 +1,49 @@ +using System.Collections.Immutable; +using System.Text.Json; +using FluentAssertions; +using StellaOps.Excititor.Core; +using StellaOps.Excititor.Formats.OpenVEX; + +namespace StellaOps.Excititor.Formats.OpenVEX.Tests; + +public sealed class OpenVexExporterTests +{ + [Fact] + public async Task SerializeAsync_ProducesCanonicalOpenVexDocument() + { + var claims = ImmutableArray.Create( + new VexClaim( + "CVE-2025-5000", + "vendor:alpha", + new VexProduct("pkg:alpha/app@2.0.0", "Alpha App", "2.0.0", "pkg:alpha/app@2.0.0"), + VexClaimStatus.NotAffected, + new VexClaimDocument(VexDocumentFormat.OpenVex, "sha256:doc1", new Uri("https://example.com/openvex/alpha")), + new DateTimeOffset(2025, 10, 10, 0, 0, 0, TimeSpan.Zero), + new DateTimeOffset(2025, 10, 11, 0, 0, 0, TimeSpan.Zero), + justification: VexJustification.ComponentNotPresent, + detail: "Component not shipped.")); + + var request = new VexExportRequest( + VexQuery.Empty, + ImmutableArray.Empty, + claims, + new DateTimeOffset(2025, 10, 12, 0, 0, 0, TimeSpan.Zero)); + + var exporter = new OpenVexExporter(); + await using var stream = new MemoryStream(); + var result = await exporter.SerializeAsync(request, stream, CancellationToken.None); + + stream.Position = 0; + using var document = JsonDocument.Parse(stream); + var root = document.RootElement; + root.GetProperty("document").GetProperty("author").GetString().Should().Be("StellaOps Excititor"); + root.GetProperty("statements").GetArrayLength().Should().Be(1); + var statement = root.GetProperty("statements")[0]; + statement.GetProperty("status").GetString().Should().Be("not_affected"); + statement.GetProperty("products")[0].GetProperty("id").GetString().Should().Be("pkg:alpha/app@2.0.0"); + + result.Metadata.Should().ContainKey("openvex.statementCount"); + result.Metadata["openvex.statementCount"].Should().Be("1"); + result.Digest.Algorithm.Should().Be("sha256"); + } +} diff --git a/src/Excititor/__Tests/StellaOps.Excititor.Formats.OpenVEX.Tests/OpenVexStatementMergerTests.cs b/src/Excititor/__Tests/StellaOps.Excititor.Formats.OpenVEX.Tests/OpenVexStatementMergerTests.cs new file mode 100644 index 00000000..199c9f3f --- /dev/null +++ b/src/Excititor/__Tests/StellaOps.Excititor.Formats.OpenVEX.Tests/OpenVexStatementMergerTests.cs @@ -0,0 +1,39 @@ +using System.Collections.Immutable; +using FluentAssertions; +using StellaOps.Excititor.Core; +using StellaOps.Excititor.Formats.OpenVEX; + +namespace StellaOps.Excititor.Formats.OpenVEX.Tests; + +public sealed class OpenVexStatementMergerTests +{ + [Fact] + public void Merge_DetectsConflictsAndSelectsCanonicalStatus() + { + var claims = ImmutableArray.Create( + new VexClaim( + "CVE-2025-4000", + "vendor:one", + new VexProduct("pkg:demo/app@1.0.0", "Demo App", "1.0.0"), + VexClaimStatus.NotAffected, + new VexClaimDocument(VexDocumentFormat.OpenVex, "sha256:doc1", new Uri("https://example.com/openvex/1")), + DateTimeOffset.UtcNow, + DateTimeOffset.UtcNow, + justification: VexJustification.ComponentNotPresent), + new VexClaim( + "CVE-2025-4000", + "vendor:two", + new VexProduct("pkg:demo/app@1.0.0", "Demo App", "1.0.0"), + VexClaimStatus.Affected, + new VexClaimDocument(VexDocumentFormat.OpenVex, "sha256:doc2", new Uri("https://example.com/openvex/2")), + DateTimeOffset.UtcNow, + DateTimeOffset.UtcNow)); + + var result = OpenVexStatementMerger.Merge(claims); + + result.Statements.Should().HaveCount(1); + var statement = result.Statements[0]; + statement.Status.Should().Be(VexClaimStatus.Affected); + result.Diagnostics.Should().ContainKey("openvex.status_conflict"); + } +} diff --git a/src/Scanner/__Libraries/StellaOps.Scanner.Analyzers.Lang.Java/Internal/Reflection/JavaReflectionAnalyzer.cs b/src/Scanner/__Libraries/StellaOps.Scanner.Analyzers.Lang.Java/Internal/Reflection/JavaReflectionAnalyzer.cs index a478ae1d..a6d38de5 100644 --- a/src/Scanner/__Libraries/StellaOps.Scanner.Analyzers.Lang.Java/Internal/Reflection/JavaReflectionAnalyzer.cs +++ b/src/Scanner/__Libraries/StellaOps.Scanner.Analyzers.Lang.Java/Internal/Reflection/JavaReflectionAnalyzer.cs @@ -241,25 +241,40 @@ internal static class JavaReflectionAnalyzer instructionOffset, null)); } - else if (normalizedOwner == "java/lang/ClassLoader" && (name == "getResource" || name == "getResourceAsStream" || name == "getResources")) - { - var target = pendingString; - var confidence = pendingString is null ? JavaReflectionConfidence.Low : JavaReflectionConfidence.High; - edges.Add(new JavaReflectionEdge( - normalizedSource, - segmentIdentifier, - target, - JavaReflectionReason.ResourceLookup, - confidence, - method.Name, - method.Descriptor, - instructionOffset, - null)); - } - else if (normalizedOwner == "java/lang/Thread" && name == "currentThread") - { - sawCurrentThread = true; - } + else if (normalizedOwner == "java/lang/ClassLoader" && (name == "getResource" || name == "getResourceAsStream" || name == "getResources")) + { + var target = pendingString; + var confidence = pendingString is null ? JavaReflectionConfidence.Low : JavaReflectionConfidence.High; + edges.Add(new JavaReflectionEdge( + normalizedSource, + segmentIdentifier, + target, + JavaReflectionReason.ResourceLookup, + confidence, + method.Name, + method.Descriptor, + instructionOffset, + null)); + } + else if (normalizedOwner == "java/lang/Class" && (name == "getResource" || name == "getResourceAsStream")) + { + var target = pendingString; + var confidence = pendingString is null ? JavaReflectionConfidence.Low : JavaReflectionConfidence.High; + edges.Add(new JavaReflectionEdge( + normalizedSource, + segmentIdentifier, + target, + JavaReflectionReason.ResourceLookup, + confidence, + method.Name, + method.Descriptor, + instructionOffset, + null)); + } + else if (normalizedOwner == "java/lang/Thread" && name == "currentThread") + { + sawCurrentThread = true; + } else if (normalizedOwner == "java/lang/Thread" && name == "getContextClassLoader") { if (sawCurrentThread && !emittedTcclWarning) diff --git a/src/Scanner/__Libraries/StellaOps.Scanner.Analyzers.Lang.Java/TASKS.md b/src/Scanner/__Libraries/StellaOps.Scanner.Analyzers.Lang.Java/TASKS.md index 2215c69c..d81a25ea 100644 --- a/src/Scanner/__Libraries/StellaOps.Scanner.Analyzers.Lang.Java/TASKS.md +++ b/src/Scanner/__Libraries/StellaOps.Scanner.Analyzers.Lang.Java/TASKS.md @@ -7,7 +7,7 @@ | SCANNER-ANALYZERS-JAVA-21-001 | DONE (2025-10-27) | Java Analyzer Guild | SCANNER-CORE-09-501 | Build input normalizer and virtual file system for JAR/WAR/EAR/fat-jar/JMOD/jimage/container roots. Detect packaging type, layered dirs (BOOT-INF/WEB-INF), multi-release overlays, and jlink runtime metadata. | Normalizer walks fixtures without extraction, classifies packaging, selects MR overlays deterministically, records java version + vendor from runtime images. | | SCANNER-ANALYZERS-JAVA-21-002 | DONE (2025-10-27) | Java Analyzer Guild | SCANNER-ANALYZERS-JAVA-21-001 | Implement module/classpath builder: JPMS graph parser (`module-info.class`), classpath order rules (fat jar, war, ear), duplicate & split-package detection, package fingerprinting. | Classpath order reproduced for fixtures; module graph serialized; duplicate provider + split-package warnings emitted deterministically. | | SCANNER-ANALYZERS-JAVA-21-003 | DONE (2025-10-27) | Java Analyzer Guild | SCANNER-ANALYZERS-JAVA-21-002 | SPI scanner covering META-INF/services, provider selection, and warning generation. Include configurable SPI corpus (JDK, Spring, logging, Jackson, MicroProfile). | SPI tables produced with selected provider + candidates; fixtures show first-wins behaviour; warnings recorded for duplicate providers. | -| SCANNER-ANALYZERS-JAVA-21-004 | DOING (2025-10-27) | Java Analyzer Guild | SCANNER-ANALYZERS-JAVA-21-002 | Reflection/dynamic loader heuristics: scan constant pools, bytecode sites (Class.forName, loadClass, TCCL usage), resource-based plugin hints, manifest loader hints. Emit edges with reason codes + confidence. | Reflection edges generated for fixtures (classpath, boot, war); includes call site metadata and confidence scoring; TCCL warning emitted where detected. | +| SCANNER-ANALYZERS-JAVA-21-004 | DONE (2025-10-29) | Java Analyzer Guild | SCANNER-ANALYZERS-JAVA-21-002 | Reflection/dynamic loader heuristics: scan constant pools, bytecode sites (Class.forName, loadClass, TCCL usage), resource-based plugin hints, manifest loader hints. Emit edges with reason codes + confidence. | Reflection edges generated for fixtures (classpath, boot, war); includes call site metadata and confidence scoring; TCCL warning emitted where detected. | | SCANNER-ANALYZERS-JAVA-21-005 | TODO | Java Analyzer Guild | SCANNER-ANALYZERS-JAVA-21-002 | Framework config extraction: Spring Boot imports, spring.factories, application properties/yaml, Jakarta web.xml & fragments, JAX-RS/JPA/CDI/JAXB configs, logging files, Graal native-image configs. | Framework fixtures parsed; relevant class FQCNs surfaced with reasons (`config-spring`, `config-jaxrs`, etc.); non-class config ignored; determinism guard passes. | | SCANNER-ANALYZERS-JAVA-21-006 | TODO | Java Analyzer Guild | SCANNER-ANALYZERS-JAVA-21-002 | JNI/native hint scanner: detect native methods, System.load/Library literals, bundled native libs, Graal JNI configs; emit `jni-load` edges for native analyzer correlation. | JNI fixtures produce hint edges pointing at embedded libs; metadata includes candidate paths and reason `jni`. | | SCANNER-ANALYZERS-JAVA-21-007 | TODO | Java Analyzer Guild | SCANNER-ANALYZERS-JAVA-21-003 | Signature and manifest metadata collector: verify JAR signature structure, capture signers, manifest loader attributes (Main-Class, Agent-Class, Start-Class, Class-Path). | Signed jar fixture reports signer info and structural validation result; manifest metadata attached to entrypoints. | diff --git a/src/Scanner/__Libraries/StellaOps.Scanner.Analyzers.Lang.Rust/Internal/RustAnalyzerCollector.cs b/src/Scanner/__Libraries/StellaOps.Scanner.Analyzers.Lang.Rust/Internal/RustAnalyzerCollector.cs index 42c4800a..92f28088 100644 --- a/src/Scanner/__Libraries/StellaOps.Scanner.Analyzers.Lang.Rust/Internal/RustAnalyzerCollector.cs +++ b/src/Scanner/__Libraries/StellaOps.Scanner.Analyzers.Lang.Rust/Internal/RustAnalyzerCollector.cs @@ -1,6 +1,5 @@ using System.Collections.Immutable; using System.Linq; -using System.Security.Cryptography; namespace StellaOps.Scanner.Analyzers.Lang.Rust.Internal; @@ -30,6 +29,7 @@ internal static class RustAnalyzerCollector private readonly Dictionary> _cratesByName = new(StringComparer.Ordinal); private readonly Dictionary _heuristics = new(StringComparer.Ordinal); private readonly Dictionary _binaries = new(StringComparer.Ordinal); + private RustLicenseIndex _licenseIndex = RustLicenseIndex.Empty; public Collector(LanguageAnalyzerContext context) { @@ -38,6 +38,7 @@ internal static class RustAnalyzerCollector public void Execute(CancellationToken cancellationToken) { + _licenseIndex = RustLicenseScanner.GetOrCreate(_context.RootPath, cancellationToken); CollectCargoLocks(cancellationToken); CollectFingerprints(cancellationToken); CollectBinaries(cancellationToken); @@ -81,6 +82,7 @@ internal static class RustAnalyzerCollector { var builder = GetOrCreateCrate(package.Name, package.Version); builder.ApplyCargoPackage(package, relativePath); + TryApplyLicense(builder); } } } @@ -95,6 +97,7 @@ internal static class RustAnalyzerCollector var builder = GetOrCreateCrate(record.Name, record.Version); var relative = NormalizeRelative(_context.GetRelativePath(record.AbsolutePath)); builder.ApplyFingerprint(record, relative); + TryApplyLicense(builder); } } @@ -149,25 +152,30 @@ internal static class RustAnalyzerCollector private RustCrateBuilder GetOrCreateCrate(string name, string? version) { - var key = new RustCrateKey(name, version); - if (_crates.TryGetValue(key, out var existing)) - { - existing.EnsureVersion(version); - return existing; - } + var key = new RustCrateKey(name, version); + if (_crates.TryGetValue(key, out var existing)) + { + existing.EnsureVersion(version); + if (!existing.HasLicenseMetadata) + { + TryApplyLicense(existing); + } + return existing; + } - var builder = new RustCrateBuilder(name, version); - _crates[key] = builder; + var builder = new RustCrateBuilder(name, version); + _crates[key] = builder; if (!_cratesByName.TryGetValue(builder.Name, out var list)) - { - list = new List(); - _cratesByName[builder.Name] = list; - } + { + list = new List(); + _cratesByName[builder.Name] = list; + } - list.Add(builder); - return builder; - } + list.Add(builder); + TryApplyLicense(builder); + return builder; + } private RustCrateBuilder? FindCrateByName(string candidate) { @@ -250,6 +258,15 @@ internal static class RustAnalyzerCollector return relativePath.Replace('\\', '/'); } + + private void TryApplyLicense(RustCrateBuilder builder) + { + var info = _licenseIndex.Find(builder.Name, builder.Version); + if (info is not null) + { + builder.ApplyLicense(info); + } + } } } @@ -274,6 +291,8 @@ internal sealed class RustCrateBuilder private readonly HashSet _evidence = new(new LanguageComponentEvidenceComparer()); private readonly SortedSet _binaryPaths = new(StringComparer.Ordinal); private readonly SortedSet _binaryHashes = new(StringComparer.Ordinal); + private readonly SortedSet _licenseExpressions = new(StringComparer.OrdinalIgnoreCase); + private readonly SortedDictionary _licenseFiles = new(StringComparer.Ordinal); private string? _version; private string? _source; @@ -290,6 +309,8 @@ internal sealed class RustCrateBuilder public string? Version => _version; + public bool HasLicenseMetadata => _licenseExpressions.Count > 0 || _licenseFiles.Count > 0; + public static string NormalizeName(string value) { if (string.IsNullOrWhiteSpace(value)) @@ -399,9 +420,40 @@ internal sealed class RustCrateBuilder var metadata = _metadata .Select(static pair => new KeyValuePair(pair.Key, pair.Value)) - .OrderBy(static pair => pair.Key, StringComparer.Ordinal) .ToList(); + if (_licenseExpressions.Count > 0) + { + var index = 0; + foreach (var expression in _licenseExpressions) + { + if (string.IsNullOrWhiteSpace(expression)) + { + continue; + } + + metadata.Add(new KeyValuePair($"license.expression[{index}]", expression)); + index++; + } + } + + if (_licenseFiles.Count > 0) + { + var index = 0; + foreach (var pair in _licenseFiles) + { + metadata.Add(new KeyValuePair($"license.file[{index}]", pair.Key)); + if (!string.IsNullOrWhiteSpace(pair.Value)) + { + metadata.Add(new KeyValuePair($"license.file.sha256[{index}]", pair.Value)); + } + + index++; + } + } + + metadata.Sort(static (left, right) => string.CompareOrdinal(left.Key, right.Key)); + var evidence = _evidence .OrderBy(static item => item.ComparisonKey, StringComparer.Ordinal) .ToImmutableArray(); @@ -422,6 +474,45 @@ internal sealed class RustCrateBuilder UsedByEntrypoint: _usedByEntrypoint); } + public void ApplyLicense(RustLicenseInfo info) + { + if (info is null) + { + return; + } + + foreach (var expression in info.Expressions) + { + if (!string.IsNullOrWhiteSpace(expression)) + { + _licenseExpressions.Add(expression.Trim()); + } + } + + foreach (var file in info.Files) + { + if (string.IsNullOrWhiteSpace(file.RelativePath)) + { + continue; + } + + var normalized = file.RelativePath.Replace('\\', '/'); + if (_licenseFiles.ContainsKey(normalized)) + { + continue; + } + + if (string.IsNullOrWhiteSpace(file.Sha256)) + { + _licenseFiles[normalized] = null; + } + else + { + _licenseFiles[normalized] = file.Sha256!.Trim(); + } + } + } + private void AddMetadataIfEmpty(string key, string? value) { if (string.IsNullOrWhiteSpace(key) || string.IsNullOrWhiteSpace(value)) @@ -577,28 +668,14 @@ internal sealed class RustBinaryRecord _hash ??= hash; } - if (_hash is null) + if (!string.IsNullOrEmpty(_hash)) { - _hash = ComputeHashSafely(); + return; } - } - private string? ComputeHashSafely() - { - try + if (RustFileHashCache.TryGetSha256(AbsolutePath, out var computed) && !string.IsNullOrEmpty(computed)) { - using var stream = new FileStream(AbsolutePath, FileMode.Open, FileAccess.Read, FileShare.Read); - using var sha = SHA256.Create(); - var hash = sha.ComputeHash(stream); - return Convert.ToHexString(hash).ToLowerInvariant(); - } - catch (IOException) - { - return null; - } - catch (UnauthorizedAccessException) - { - return null; + _hash = computed; } } } diff --git a/src/Scanner/__Libraries/StellaOps.Scanner.Analyzers.Lang.Rust/Internal/RustBinaryClassifier.cs b/src/Scanner/__Libraries/StellaOps.Scanner.Analyzers.Lang.Rust/Internal/RustBinaryClassifier.cs index 3d6d31ce..a5a88a5a 100644 --- a/src/Scanner/__Libraries/StellaOps.Scanner.Analyzers.Lang.Rust/Internal/RustBinaryClassifier.cs +++ b/src/Scanner/__Libraries/StellaOps.Scanner.Analyzers.Lang.Rust/Internal/RustBinaryClassifier.cs @@ -1,7 +1,7 @@ using System.Buffers; +using System.Collections.Concurrent; using System.Collections.Immutable; using System.Linq; -using System.Security.Cryptography; using System.Text; namespace StellaOps.Scanner.Analyzers.Lang.Rust.Internal; @@ -32,6 +32,8 @@ internal static class RustBinaryClassifier AttributesToSkip = FileAttributes.Device | FileAttributes.ReparsePoint, }; + private static readonly ConcurrentDictionary> CandidateCache = new(); + public static IReadOnlyList Scan(string rootPath, CancellationToken cancellationToken) { if (string.IsNullOrWhiteSpace(rootPath)) @@ -49,7 +51,16 @@ internal static class RustBinaryClassifier continue; } - var candidates = ExtractCrateNames(path, cancellationToken); + if (!RustFileCacheKey.TryCreate(path, out var key)) + { + continue; + } + + var candidates = CandidateCache.GetOrAdd( + key, + static (_, state) => ExtractCrateNames(state.Path, state.CancellationToken), + (Path: path, CancellationToken: cancellationToken)); + binaries.Add(new RustBinaryInfo(path, candidates)); } @@ -220,31 +231,13 @@ internal static class RustBinaryClassifier internal sealed record RustBinaryInfo(string AbsolutePath, ImmutableArray CrateCandidates) { - private string? _sha256; - public string ComputeSha256() { - if (_sha256 is not null) + if (RustFileHashCache.TryGetSha256(AbsolutePath, out var sha256) && !string.IsNullOrEmpty(sha256)) { - return _sha256; + return sha256; } - try - { - using var stream = new FileStream(AbsolutePath, FileMode.Open, FileAccess.Read, FileShare.Read); - using var sha = SHA256.Create(); - var hash = sha.ComputeHash(stream); - _sha256 = Convert.ToHexString(hash).ToLowerInvariant(); - } - catch (IOException) - { - _sha256 = string.Empty; - } - catch (UnauthorizedAccessException) - { - _sha256 = string.Empty; - } - - return _sha256 ?? string.Empty; + return string.Empty; } } diff --git a/src/Scanner/__Libraries/StellaOps.Scanner.Analyzers.Lang.Rust/Internal/RustCargoLockParser.cs b/src/Scanner/__Libraries/StellaOps.Scanner.Analyzers.Lang.Rust/Internal/RustCargoLockParser.cs index 137a6985..7db9d584 100644 --- a/src/Scanner/__Libraries/StellaOps.Scanner.Analyzers.Lang.Rust/Internal/RustCargoLockParser.cs +++ b/src/Scanner/__Libraries/StellaOps.Scanner.Analyzers.Lang.Rust/Internal/RustCargoLockParser.cs @@ -1,7 +1,12 @@ +using System.Collections.Concurrent; +using System.Collections.Immutable; + namespace StellaOps.Scanner.Analyzers.Lang.Rust.Internal; internal static class RustCargoLockParser { + private static readonly ConcurrentDictionary> Cache = new(); + public static IReadOnlyList Parse(string path, CancellationToken cancellationToken) { if (string.IsNullOrWhiteSpace(path)) @@ -9,17 +14,26 @@ internal static class RustCargoLockParser throw new ArgumentException("Lock path is required", nameof(path)); } - var info = new FileInfo(path); - if (!info.Exists) + if (!RustFileCacheKey.TryCreate(path, out var key)) { return Array.Empty(); } - var packages = new List(); + var packages = Cache.GetOrAdd( + key, + static (_, state) => ParseInternal(state.Path, state.CancellationToken), + (Path: path, CancellationToken: cancellationToken)); + + return packages.IsDefaultOrEmpty ? Array.Empty() : packages; + } + + private static ImmutableArray ParseInternal(string path, CancellationToken cancellationToken) + { + var resultBuilder = ImmutableArray.CreateBuilder(); + using var stream = new FileStream(path, FileMode.Open, FileAccess.Read, FileShare.Read); using var reader = new StreamReader(stream); - - RustCargoPackageBuilder? builder = null; + RustCargoPackageBuilder? packageBuilder = null; string? currentArrayKey = null; var arrayValues = new List(); @@ -41,14 +55,14 @@ internal static class RustCargoLockParser if (IsPackageHeader(trimmed)) { - FlushCurrent(builder, packages); - builder = new RustCargoPackageBuilder(); + FlushCurrent(packageBuilder, resultBuilder); + packageBuilder = new RustCargoPackageBuilder(); currentArrayKey = null; arrayValues.Clear(); continue; } - if (builder is null) + if (packageBuilder is null) { continue; } @@ -94,12 +108,12 @@ internal static class RustCargoLockParser } if (valuePart[0] == '[') - { - currentArrayKey = key.ToString(); - arrayValues.Clear(); - - if (valuePart.Length > 1 && valuePart[^1] == ']') { + currentArrayKey = key.ToString(); + arrayValues.Clear(); + + if (valuePart.Length > 1 && valuePart[^1] == ']') + { var inline = valuePart[1..^1].Trim(); if (inline.Length > 0) { @@ -113,7 +127,7 @@ internal static class RustCargoLockParser } } - builder.SetArray(currentArrayKey, arrayValues); + packageBuilder.SetArray(currentArrayKey, arrayValues); currentArrayKey = null; arrayValues.Clear(); } @@ -124,17 +138,17 @@ internal static class RustCargoLockParser var parsed = ExtractString(valuePart); if (parsed is not null) { - builder.SetField(key, parsed); + packageBuilder.SetField(key, parsed); } } if (currentArrayKey is not null && arrayValues.Count > 0) { - builder?.SetArray(currentArrayKey, arrayValues); + packageBuilder?.SetArray(currentArrayKey, arrayValues); } - FlushCurrent(builder, packages); - return packages; + FlushCurrent(packageBuilder, resultBuilder); + return resultBuilder.ToImmutable(); } private static ReadOnlySpan TrimComments(ReadOnlySpan line) @@ -204,14 +218,14 @@ internal static class RustCargoLockParser return trimmed.Length == 0 ? null : trimmed.ToString(); } - private static void FlushCurrent(RustCargoPackageBuilder? builder, List packages) + private static void FlushCurrent(RustCargoPackageBuilder? packageBuilder, ImmutableArray.Builder packages) { - if (builder is null || !builder.HasData) + if (packageBuilder is null || !packageBuilder.HasData) { return; } - if (builder.TryBuild(out var package)) + if (packageBuilder.TryBuild(out var package)) { packages.Add(package); } diff --git a/src/Scanner/__Libraries/StellaOps.Scanner.Analyzers.Lang.Rust/Internal/RustFileCacheKey.cs b/src/Scanner/__Libraries/StellaOps.Scanner.Analyzers.Lang.Rust/Internal/RustFileCacheKey.cs new file mode 100644 index 00000000..2aacec1c --- /dev/null +++ b/src/Scanner/__Libraries/StellaOps.Scanner.Analyzers.Lang.Rust/Internal/RustFileCacheKey.cs @@ -0,0 +1,74 @@ +using System.Security; + +namespace StellaOps.Scanner.Analyzers.Lang.Rust.Internal; + +internal readonly struct RustFileCacheKey : IEquatable +{ + private readonly string _normalizedPath; + private readonly long _length; + private readonly long _lastWriteTicks; + + private RustFileCacheKey(string normalizedPath, long length, long lastWriteTicks) + { + _normalizedPath = normalizedPath; + _length = length; + _lastWriteTicks = lastWriteTicks; + } + + public static bool TryCreate(string path, out RustFileCacheKey key) + { + key = default; + + if (string.IsNullOrWhiteSpace(path)) + { + return false; + } + + try + { + var info = new FileInfo(path); + if (!info.Exists) + { + return false; + } + + var normalizedPath = OperatingSystem.IsWindows() + ? info.FullName.ToLowerInvariant() + : info.FullName; + + key = new RustFileCacheKey(normalizedPath, info.Length, info.LastWriteTimeUtc.Ticks); + return true; + } + catch (IOException) + { + return false; + } + catch (UnauthorizedAccessException) + { + return false; + } + catch (SecurityException) + { + return false; + } + catch (ArgumentException) + { + return false; + } + catch (NotSupportedException) + { + return false; + } + } + + public bool Equals(RustFileCacheKey other) + => _length == other._length + && _lastWriteTicks == other._lastWriteTicks + && string.Equals(_normalizedPath, other._normalizedPath, StringComparison.Ordinal); + + public override bool Equals(object? obj) + => obj is RustFileCacheKey other && Equals(other); + + public override int GetHashCode() + => HashCode.Combine(_normalizedPath, _length, _lastWriteTicks); +} diff --git a/src/Scanner/__Libraries/StellaOps.Scanner.Analyzers.Lang.Rust/Internal/RustFileHashCache.cs b/src/Scanner/__Libraries/StellaOps.Scanner.Analyzers.Lang.Rust/Internal/RustFileHashCache.cs new file mode 100644 index 00000000..78768eff --- /dev/null +++ b/src/Scanner/__Libraries/StellaOps.Scanner.Analyzers.Lang.Rust/Internal/RustFileHashCache.cs @@ -0,0 +1,45 @@ +using System.Collections.Concurrent; +using System.Security; + +namespace StellaOps.Scanner.Analyzers.Lang.Rust.Internal; + +internal static class RustFileHashCache +{ + private static readonly ConcurrentDictionary Sha256Cache = new(); + + public static bool TryGetSha256(string path, out string? sha256) + { + sha256 = null; + + if (!RustFileCacheKey.TryCreate(path, out var key)) + { + return false; + } + + try + { + sha256 = Sha256Cache.GetOrAdd(key, static (_, state) => ComputeSha256(state), path); + return !string.IsNullOrEmpty(sha256); + } + catch (IOException) + { + return false; + } + catch (UnauthorizedAccessException) + { + return false; + } + catch (SecurityException) + { + return false; + } + } + + private static string ComputeSha256(string path) + { + using var stream = File.Open(path, FileMode.Open, FileAccess.Read, FileShare.Read); + using var sha = System.Security.Cryptography.SHA256.Create(); + var hash = sha.ComputeHash(stream); + return Convert.ToHexString(hash).ToLowerInvariant(); + } +} diff --git a/src/Scanner/__Libraries/StellaOps.Scanner.Analyzers.Lang.Rust/Internal/RustFingerprintScanner.cs b/src/Scanner/__Libraries/StellaOps.Scanner.Analyzers.Lang.Rust/Internal/RustFingerprintScanner.cs index 6fd06017..f5246053 100644 --- a/src/Scanner/__Libraries/StellaOps.Scanner.Analyzers.Lang.Rust/Internal/RustFingerprintScanner.cs +++ b/src/Scanner/__Libraries/StellaOps.Scanner.Analyzers.Lang.Rust/Internal/RustFingerprintScanner.cs @@ -1,3 +1,4 @@ +using System.Collections.Concurrent; using System.Text.Json; namespace StellaOps.Scanner.Analyzers.Lang.Rust.Internal; @@ -13,6 +14,7 @@ internal static class RustFingerprintScanner }; private static readonly string FingerprintSegment = $"{Path.DirectorySeparatorChar}.fingerprint{Path.DirectorySeparatorChar}"; + private static readonly ConcurrentDictionary Cache = new(); public static IReadOnlyList Scan(string rootPath, CancellationToken cancellationToken) { @@ -31,7 +33,17 @@ internal static class RustFingerprintScanner continue; } - if (TryParse(path, out var record)) + if (!RustFileCacheKey.TryCreate(path, out var key)) + { + continue; + } + + var record = Cache.GetOrAdd( + key, + static (_, state) => ParseFingerprint(state), + path); + + if (record is not null) { results.Add(record); } @@ -40,10 +52,8 @@ internal static class RustFingerprintScanner return results; } - private static bool TryParse(string path, out RustFingerprintRecord record) + private static RustFingerprintRecord? ParseFingerprint(string path) { - record = default!; - try { using var stream = new FileStream(path, FileMode.Open, FileAccess.Read, FileShare.Read); @@ -57,33 +67,31 @@ internal static class RustFingerprintScanner var (name, version, source) = ParseIdentity(pkgId, path); if (string.IsNullOrWhiteSpace(name)) { - return false; + return null; } var profile = TryGetString(root, "profile"); var targetKind = TryGetKind(root); - record = new RustFingerprintRecord( + return new RustFingerprintRecord( Name: name!, Version: version, Source: source, TargetKind: targetKind, Profile: profile, AbsolutePath: path); - - return true; } catch (JsonException) { - return false; + return null; } catch (IOException) { - return false; + return null; } catch (UnauthorizedAccessException) { - return false; + return null; } } diff --git a/src/Scanner/__Libraries/StellaOps.Scanner.Analyzers.Lang.Rust/Internal/RustLicenseScanner.cs b/src/Scanner/__Libraries/StellaOps.Scanner.Analyzers.Lang.Rust/Internal/RustLicenseScanner.cs new file mode 100644 index 00000000..fc791086 --- /dev/null +++ b/src/Scanner/__Libraries/StellaOps.Scanner.Analyzers.Lang.Rust/Internal/RustLicenseScanner.cs @@ -0,0 +1,298 @@ +using System.Collections.Concurrent; +using System.Collections.Immutable; +using System.Security; + +namespace StellaOps.Scanner.Analyzers.Lang.Rust.Internal; + +internal static class RustLicenseScanner +{ + private static readonly ConcurrentDictionary IndexCache = new(StringComparer.Ordinal); + + public static RustLicenseIndex GetOrCreate(string rootPath, CancellationToken cancellationToken) + { + if (string.IsNullOrWhiteSpace(rootPath) || !Directory.Exists(rootPath)) + { + return RustLicenseIndex.Empty; + } + + var normalizedRoot = NormalizeRoot(rootPath); + return IndexCache.GetOrAdd( + normalizedRoot, + static (_, state) => BuildIndex(state.RootPath, state.CancellationToken), + (RootPath: rootPath, CancellationToken: cancellationToken)); + } + + private static RustLicenseIndex BuildIndex(string rootPath, CancellationToken cancellationToken) + { + var byName = new Dictionary>(StringComparer.Ordinal); + var enumeration = new EnumerationOptions + { + MatchCasing = MatchCasing.CaseSensitive, + IgnoreInaccessible = true, + RecurseSubdirectories = true, + AttributesToSkip = FileAttributes.Device | FileAttributes.ReparsePoint, + }; + + foreach (var cargoTomlPath in Directory.EnumerateFiles(rootPath, "Cargo.toml", enumeration)) + { + cancellationToken.ThrowIfCancellationRequested(); + + if (IsUnderTargetDirectory(cargoTomlPath)) + { + continue; + } + + if (!TryParseCargoToml(rootPath, cargoTomlPath, out var info)) + { + continue; + } + + var normalizedName = RustCrateBuilder.NormalizeName(info.Name); + if (!byName.TryGetValue(normalizedName, out var entries)) + { + entries = new List(); + byName[normalizedName] = entries; + } + + entries.Add(info); + } + + foreach (var entry in byName.Values) + { + entry.Sort(static (left, right) => + { + var versionCompare = string.Compare(left.Version, right.Version, StringComparison.OrdinalIgnoreCase); + if (versionCompare != 0) + { + return versionCompare; + } + + return string.Compare(left.CargoTomlRelativePath, right.CargoTomlRelativePath, StringComparison.Ordinal); + }); + } + + return new RustLicenseIndex(byName); + } + + private static bool TryParseCargoToml(string rootPath, string cargoTomlPath, out RustLicenseInfo info) + { + info = default!; + + try + { + using var stream = new FileStream(cargoTomlPath, FileMode.Open, FileAccess.Read, FileShare.Read); + using var reader = new StreamReader(stream, leaveOpen: false); + + string? name = null; + string? version = null; + string? licenseExpression = null; + string? licenseFile = null; + var inPackageSection = false; + + while (reader.ReadLine() is { } line) + { + line = StripComment(line).Trim(); + if (line.Length == 0) + { + continue; + } + + if (line.StartsWith("[", StringComparison.Ordinal)) + { + inPackageSection = string.Equals(line, "[package]", StringComparison.OrdinalIgnoreCase); + if (!inPackageSection && line.StartsWith("[dependency", StringComparison.OrdinalIgnoreCase)) + { + // Exiting package section. + break; + } + + continue; + } + + if (!inPackageSection) + { + continue; + } + + if (TryParseStringAssignment(line, "name", out var parsedName)) + { + name ??= parsedName; + continue; + } + + if (TryParseStringAssignment(line, "version", out var parsedVersion)) + { + version ??= parsedVersion; + continue; + } + + if (TryParseStringAssignment(line, "license", out var parsedLicense)) + { + licenseExpression ??= parsedLicense; + continue; + } + + if (TryParseStringAssignment(line, "license-file", out var parsedLicenseFile)) + { + licenseFile ??= parsedLicenseFile; + continue; + } + } + + if (string.IsNullOrWhiteSpace(name)) + { + return false; + } + + var expressions = ImmutableArray.Empty; + if (!string.IsNullOrWhiteSpace(licenseExpression)) + { + expressions = ImmutableArray.Create(licenseExpression!); + } + + var files = ImmutableArray.Empty; + if (!string.IsNullOrWhiteSpace(licenseFile)) + { + var directory = Path.GetDirectoryName(cargoTomlPath) ?? string.Empty; + var absolute = Path.GetFullPath(Path.Combine(directory, licenseFile!)); + if (File.Exists(absolute)) + { + var relative = NormalizeRelativePath(rootPath, absolute); + if (RustFileHashCache.TryGetSha256(absolute, out var sha256)) + { + files = ImmutableArray.Create(new RustLicenseFileReference(relative, sha256)); + } + else + { + files = ImmutableArray.Create(new RustLicenseFileReference(relative, null)); + } + } + } + + var cargoRelative = NormalizeRelativePath(rootPath, cargoTomlPath); + + info = new RustLicenseInfo( + name!.Trim(), + string.IsNullOrWhiteSpace(version) ? null : version!.Trim(), + expressions, + files, + cargoRelative); + + return true; + } + catch (IOException) + { + return false; + } + catch (UnauthorizedAccessException) + { + return false; + } + catch (SecurityException) + { + return false; + } + } + + private static string NormalizeRoot(string rootPath) + { + var full = Path.GetFullPath(rootPath); + return OperatingSystem.IsWindows() + ? full.ToLowerInvariant() + : full; + } + + private static bool TryParseStringAssignment(string line, string key, out string? value) + { + value = null; + + if (!line.StartsWith(key, StringComparison.Ordinal)) + { + return false; + } + + var remaining = line[key.Length..].TrimStart(); + if (remaining.Length == 0 || remaining[0] != '=') + { + return false; + } + + remaining = remaining[1..].TrimStart(); + if (remaining.Length < 2 || remaining[0] != '"' || remaining[^1] != '"') + { + return false; + } + + value = remaining[1..^1]; + return true; + } + + private static string StripComment(string line) + { + var index = line.IndexOf('#'); + return index < 0 ? line : line[..index]; + } + + private static bool IsUnderTargetDirectory(string path) + { + var segment = $"{Path.DirectorySeparatorChar}target{Path.DirectorySeparatorChar}"; + return path.Contains(segment, OperatingSystem.IsWindows() ? StringComparison.OrdinalIgnoreCase : StringComparison.Ordinal); + } + + private static string NormalizeRelativePath(string rootPath, string absolutePath) + { + var relative = Path.GetRelativePath(rootPath, absolutePath); + if (string.IsNullOrWhiteSpace(relative) || relative == ".") + { + return "."; + } + + return relative.Replace('\\', '/'); + } +} + +internal sealed class RustLicenseIndex +{ + private readonly Dictionary> _byName; + + public static readonly RustLicenseIndex Empty = new(new Dictionary>(StringComparer.Ordinal)); + + public RustLicenseIndex(Dictionary> byName) + { + _byName = byName ?? throw new ArgumentNullException(nameof(byName)); + } + + public RustLicenseInfo? Find(string crateName, string? version) + { + if (string.IsNullOrWhiteSpace(crateName)) + { + return null; + } + + var normalized = RustCrateBuilder.NormalizeName(crateName); + if (!_byName.TryGetValue(normalized, out var list) || list.Count == 0) + { + return null; + } + + if (!string.IsNullOrWhiteSpace(version)) + { + var match = list.FirstOrDefault(entry => string.Equals(entry.Version, version, StringComparison.OrdinalIgnoreCase)); + if (match is not null) + { + return match; + } + } + + return list[0]; + } +} + +internal sealed record RustLicenseInfo( + string Name, + string? Version, + ImmutableArray Expressions, + ImmutableArray Files, + string CargoTomlRelativePath); + +internal sealed record RustLicenseFileReference(string RelativePath, string? Sha256); diff --git a/src/Scanner/__Libraries/StellaOps.Scanner.Analyzers.Lang.Rust/TASKS.md b/src/Scanner/__Libraries/StellaOps.Scanner.Analyzers.Lang.Rust/TASKS.md index 02250523..655b043c 100644 --- a/src/Scanner/__Libraries/StellaOps.Scanner.Analyzers.Lang.Rust/TASKS.md +++ b/src/Scanner/__Libraries/StellaOps.Scanner.Analyzers.Lang.Rust/TASKS.md @@ -5,6 +5,6 @@ | 1 | SCANNER-ANALYZERS-LANG-10-306A | DONE (2025-10-22) | SCANNER-ANALYZERS-LANG-10-307 | Parse Cargo metadata (`Cargo.lock`, `.fingerprint`, `.metadata`) and map crates to components with evidence. | Fixtures confirm crate attribution ≥85 % coverage; metadata normalized; evidence includes path + hash. | | 2 | SCANNER-ANALYZERS-LANG-10-306B | DONE (2025-10-22) | SCANNER-ANALYZERS-LANG-10-306A | Implement heuristic classifier using ELF section names, symbol mangling, and `.comment` data for stripped binaries. | Heuristic output flagged as `heuristic`; regression tests ensure no false “observed” classifications. | | 3 | SCANNER-ANALYZERS-LANG-10-306C | DONE (2025-10-22) | SCANNER-ANALYZERS-LANG-10-306B | Integrate binary hash fallback (`bin:{sha256}`) and tie into shared quiet provenance helpers. | Fallback path deterministic; shared helpers reused; tests verify consistent hashing. | -| 4 | SCANNER-ANALYZERS-LANG-10-307R | DOING (2025-10-23) | SCANNER-ANALYZERS-LANG-10-306C | Finalize shared helper usage (license, usage flags) and concurrency-safe caches. | Analyzer uses shared utilities; concurrency tests pass; no race conditions. | +| 4 | SCANNER-ANALYZERS-LANG-10-307R | DONE (2025-10-29) | SCANNER-ANALYZERS-LANG-10-306C | Finalize shared helper usage (license, usage flags) and concurrency-safe caches. | Analyzer uses shared utilities; concurrency tests pass; no race conditions. | | 5 | SCANNER-ANALYZERS-LANG-10-308R | TODO | SCANNER-ANALYZERS-LANG-10-307R | Determinism fixtures + performance benchmarks; compare against competitor heuristic coverage. | Fixtures `Fixtures/lang/rust/` committed; determinism guard; benchmark shows ≥15 % better coverage vs competitor. | | 6 | SCANNER-ANALYZERS-LANG-10-309R | TODO | SCANNER-ANALYZERS-LANG-10-308R | Package plug-in manifest + Offline Kit documentation; ensure Worker integration. | Manifest copied; Worker loads analyzer; Offline Kit doc updated. | diff --git a/src/Scanner/__Tests/StellaOps.Scanner.Analyzers.Lang.Java.Tests/Java/JavaReflectionAnalyzerTests.cs b/src/Scanner/__Tests/StellaOps.Scanner.Analyzers.Lang.Java.Tests/Java/JavaReflectionAnalyzerTests.cs index eafe3c64..f6f13762 100644 --- a/src/Scanner/__Tests/StellaOps.Scanner.Analyzers.Lang.Java.Tests/Java/JavaReflectionAnalyzerTests.cs +++ b/src/Scanner/__Tests/StellaOps.Scanner.Analyzers.Lang.Java.Tests/Java/JavaReflectionAnalyzerTests.cs @@ -77,26 +77,59 @@ public sealed class JavaReflectionAnalyzerTests } [Fact] - public void Analyze_SpringBootFatJar_ScansEmbeddedAndBootSegments() - { - var root = TestPaths.CreateTemporaryDirectory(); - try - { - JavaFixtureBuilder.CreateSpringBootFatJar(root, "apps/app-fat.jar"); - - var cancellationToken = TestContext.Current.CancellationToken; - var context = new LanguageAnalyzerContext(root, TimeProvider.System); - var workspace = JavaWorkspaceNormalizer.Normalize(context, cancellationToken); - var classPath = JavaClassPathBuilder.Build(workspace, cancellationToken); - var analysis = JavaReflectionAnalyzer.Analyze(classPath, cancellationToken); - - // Expect at least one edge originating from BOOT-INF classes - Assert.Contains(analysis.Edges, edge => edge.SourceClass == "com.example.App" && edge.Reason == JavaReflectionReason.ClassForName); - Assert.Contains(analysis.Edges, edge => edge.SourceClass == "com.example.Lib" && edge.Reason == JavaReflectionReason.ClassForName); - } - finally - { - TestPaths.SafeDelete(root); - } - } -} + public void Analyze_SpringBootFatJar_ScansEmbeddedAndBootSegments() + { + var root = TestPaths.CreateTemporaryDirectory(); + try + { + JavaFixtureBuilder.CreateSpringBootFatJar(root, "apps/app-fat.jar"); + + var cancellationToken = TestContext.Current.CancellationToken; + var context = new LanguageAnalyzerContext(root, TimeProvider.System); + var workspace = JavaWorkspaceNormalizer.Normalize(context, cancellationToken); + var classPath = JavaClassPathBuilder.Build(workspace, cancellationToken); + var analysis = JavaReflectionAnalyzer.Analyze(classPath, cancellationToken); + + // Expect at least one edge originating from BOOT-INF classes + Assert.Contains(analysis.Edges, edge => edge.SourceClass == "com.example.App" && edge.Reason == JavaReflectionReason.ClassForName); + Assert.Contains(analysis.Edges, edge => edge.SourceClass == "com.example.Lib" && edge.Reason == JavaReflectionReason.ClassForName); + } + finally + { + TestPaths.SafeDelete(root); + } + } + + [Fact] + public void Analyze_ClassResourceLookup_ProducesResourceEdge() + { + var root = TestPaths.CreateTemporaryDirectory(); + try + { + var jarPath = Path.Combine(root, "libs", "resources.jar"); + Directory.CreateDirectory(Path.GetDirectoryName(jarPath)!); + using (var archive = new ZipArchive(new FileStream(jarPath, FileMode.Create, FileAccess.ReadWrite, FileShare.None), ZipArchiveMode.Create, leaveOpen: false)) + { + var entry = archive.CreateEntry("com/example/Resources.class"); + var bytes = JavaClassFileFactory.CreateClassResourceLookup("com/example/Resources", "/META-INF/plugin.properties"); + using var stream = entry.Open(); + stream.Write(bytes); + } + + var cancellationToken = TestContext.Current.CancellationToken; + var context = new LanguageAnalyzerContext(root, TimeProvider.System); + var workspace = JavaWorkspaceNormalizer.Normalize(context, cancellationToken); + var classPath = JavaClassPathBuilder.Build(workspace, cancellationToken); + var analysis = JavaReflectionAnalyzer.Analyze(classPath, cancellationToken); + + var edge = Assert.Single(analysis.Edges.Where(edge => edge.Reason == JavaReflectionReason.ResourceLookup)); + Assert.Equal("com.example.Resources", edge.SourceClass); + Assert.Equal("/META-INF/plugin.properties", edge.TargetType); + Assert.Equal(JavaReflectionConfidence.High, edge.Confidence); + } + finally + { + TestPaths.SafeDelete(root); + } + } +} diff --git a/src/Scanner/__Tests/StellaOps.Scanner.Analyzers.Lang.Tests/Fixtures/lang/rust/simple/Cargo.toml b/src/Scanner/__Tests/StellaOps.Scanner.Analyzers.Lang.Tests/Fixtures/lang/rust/simple/Cargo.toml new file mode 100644 index 00000000..2ef2bcc7 --- /dev/null +++ b/src/Scanner/__Tests/StellaOps.Scanner.Analyzers.Lang.Tests/Fixtures/lang/rust/simple/Cargo.toml @@ -0,0 +1,4 @@ +[package] +name = "my_app" +version = "0.1.0" +license = "MIT" diff --git a/src/Scanner/__Tests/StellaOps.Scanner.Analyzers.Lang.Tests/Fixtures/lang/rust/simple/LICENSE b/src/Scanner/__Tests/StellaOps.Scanner.Analyzers.Lang.Tests/Fixtures/lang/rust/simple/LICENSE new file mode 100644 index 00000000..d68d0427 --- /dev/null +++ b/src/Scanner/__Tests/StellaOps.Scanner.Analyzers.Lang.Tests/Fixtures/lang/rust/simple/LICENSE @@ -0,0 +1,16 @@ +MIT License + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. diff --git a/src/Scanner/__Tests/StellaOps.Scanner.Analyzers.Lang.Tests/Fixtures/lang/rust/simple/expected.json b/src/Scanner/__Tests/StellaOps.Scanner.Analyzers.Lang.Tests/Fixtures/lang/rust/simple/expected.json index fb2c2360..1c914534 100644 --- a/src/Scanner/__Tests/StellaOps.Scanner.Analyzers.Lang.Tests/Fixtures/lang/rust/simple/expected.json +++ b/src/Scanner/__Tests/StellaOps.Scanner.Analyzers.Lang.Tests/Fixtures/lang/rust/simple/expected.json @@ -7,12 +7,13 @@ "version": "0.1.0", "type": "cargo", "usedByEntrypoint": false, - "metadata": { - "cargo.lock.path": "Cargo.lock", - "fingerprint.profile": "debug", - "fingerprint.targetKind": "bin", - "source": "registry\u002Bhttps://github.com/rust-lang/crates.io-index" - }, + "metadata": { + "cargo.lock.path": "Cargo.lock", + "fingerprint.profile": "debug", + "fingerprint.targetKind": "bin", + "license.expression[0]": "MIT", + "source": "registry\u002Bhttps://github.com/rust-lang/crates.io-index" + }, "evidence": [ { "kind": "file", @@ -36,13 +37,14 @@ "version": "1.0.188", "type": "cargo", "usedByEntrypoint": false, - "metadata": { - "cargo.lock.path": "Cargo.lock", - "checksum": "abc123", - "fingerprint.profile": "release", - "fingerprint.targetKind": "lib", - "source": "registry\u002Bhttps://github.com/rust-lang/crates.io-index" - }, + "metadata": { + "cargo.lock.path": "Cargo.lock", + "checksum": "abc123", + "fingerprint.profile": "release", + "fingerprint.targetKind": "lib", + "license.expression[0]": "Apache-2.0", + "source": "registry\u002Bhttps://github.com/rust-lang/crates.io-index" + }, "evidence": [ { "kind": "file", @@ -59,4 +61,4 @@ } ] } -] \ No newline at end of file +] diff --git a/src/Scanner/__Tests/StellaOps.Scanner.Analyzers.Lang.Tests/Fixtures/lang/rust/simple/vendor/serde-1.0.188/Cargo.toml b/src/Scanner/__Tests/StellaOps.Scanner.Analyzers.Lang.Tests/Fixtures/lang/rust/simple/vendor/serde-1.0.188/Cargo.toml new file mode 100644 index 00000000..faa51d66 --- /dev/null +++ b/src/Scanner/__Tests/StellaOps.Scanner.Analyzers.Lang.Tests/Fixtures/lang/rust/simple/vendor/serde-1.0.188/Cargo.toml @@ -0,0 +1,4 @@ +[package] +name = "serde" +version = "1.0.188" +license = "Apache-2.0" diff --git a/src/Scanner/__Tests/StellaOps.Scanner.Analyzers.Lang.Tests/Rust/RustLanguageAnalyzerTests.cs b/src/Scanner/__Tests/StellaOps.Scanner.Analyzers.Lang.Tests/Rust/RustLanguageAnalyzerTests.cs index 75e773a2..080e0395 100644 --- a/src/Scanner/__Tests/StellaOps.Scanner.Analyzers.Lang.Tests/Rust/RustLanguageAnalyzerTests.cs +++ b/src/Scanner/__Tests/StellaOps.Scanner.Analyzers.Lang.Tests/Rust/RustLanguageAnalyzerTests.cs @@ -1,4 +1,6 @@ +using System; using System.IO; +using System.Linq; using StellaOps.Scanner.Analyzers.Lang.Rust; using StellaOps.Scanner.Analyzers.Lang.Tests.Harness; using StellaOps.Scanner.Analyzers.Lang.Tests.TestUtilities; @@ -31,4 +33,27 @@ public sealed class RustLanguageAnalyzerTests cancellationToken, usageHints); } + + [Fact] + public async Task AnalyzerIsThreadSafeUnderConcurrencyAsync() + { + var cancellationToken = TestContext.Current.CancellationToken; + var fixturePath = TestPaths.ResolveFixture("lang", "rust", "simple"); + + var analyzers = new ILanguageAnalyzer[] + { + new RustLanguageAnalyzer() + }; + + var workers = Math.Max(Environment.ProcessorCount, 4); + var tasks = Enumerable.Range(0, workers) + .Select(_ => LanguageAnalyzerTestHarness.RunToJsonAsync(fixturePath, analyzers, cancellationToken)); + + var results = await Task.WhenAll(tasks); + var baseline = results[0]; + foreach (var result in results) + { + Assert.Equal(baseline, result); + } + } } diff --git a/src/Scanner/__Tests/StellaOps.Scanner.Analyzers.Lang.Tests/TestUtilities/JavaClassFileFactory.cs b/src/Scanner/__Tests/StellaOps.Scanner.Analyzers.Lang.Tests/TestUtilities/JavaClassFileFactory.cs index 2f8fc10f..f91ad5b6 100644 --- a/src/Scanner/__Tests/StellaOps.Scanner.Analyzers.Lang.Tests/TestUtilities/JavaClassFileFactory.cs +++ b/src/Scanner/__Tests/StellaOps.Scanner.Analyzers.Lang.Tests/TestUtilities/JavaClassFileFactory.cs @@ -5,11 +5,11 @@ namespace StellaOps.Scanner.Analyzers.Lang.Tests.TestUtilities; public static class JavaClassFileFactory { - public static byte[] CreateClassForNameInvoker(string internalClassName, string targetClassName) - { - using var buffer = new MemoryStream(); - using var writer = new BigEndianWriter(buffer); - + public static byte[] CreateClassForNameInvoker(string internalClassName, string targetClassName) + { + using var buffer = new MemoryStream(); + using var writer = new BigEndianWriter(buffer); + WriteClassFileHeader(writer, constantPoolCount: 16); writer.WriteByte((byte)ConstantTag.Utf8); writer.WriteUtf8(internalClassName); // #1 @@ -40,8 +40,46 @@ public static class JavaClassFileFactory writer.WriteUInt16(0); // class attributes - return buffer.ToArray(); - } + return buffer.ToArray(); + } + + public static byte[] CreateClassResourceLookup(string internalClassName, string resourcePath) + { + using var buffer = new MemoryStream(); + using var writer = new BigEndianWriter(buffer); + + WriteClassFileHeader(writer, constantPoolCount: 18); + + writer.WriteByte((byte)ConstantTag.Utf8); writer.WriteUtf8(internalClassName); // #1 + writer.WriteByte((byte)ConstantTag.Class); writer.WriteUInt16(1); // #2 + writer.WriteByte((byte)ConstantTag.Utf8); writer.WriteUtf8("java/lang/Object"); // #3 + writer.WriteByte((byte)ConstantTag.Class); writer.WriteUInt16(3); // #4 + writer.WriteByte((byte)ConstantTag.Utf8); writer.WriteUtf8("load"); // #5 + writer.WriteByte((byte)ConstantTag.Utf8); writer.WriteUtf8("()V"); // #6 + writer.WriteByte((byte)ConstantTag.Utf8); writer.WriteUtf8("Code"); // #7 + writer.WriteByte((byte)ConstantTag.Utf8); writer.WriteUtf8(resourcePath); // #8 + writer.WriteByte((byte)ConstantTag.String); writer.WriteUInt16(8); // #9 + writer.WriteByte((byte)ConstantTag.Utf8); writer.WriteUtf8("java/lang/Class"); // #10 + writer.WriteByte((byte)ConstantTag.Class); writer.WriteUInt16(10); // #11 + writer.WriteByte((byte)ConstantTag.Utf8); writer.WriteUtf8("getResource"); // #12 + writer.WriteByte((byte)ConstantTag.Utf8); writer.WriteUtf8("(Ljava/lang/String;)Ljava/net/URL;"); // #13 + writer.WriteByte((byte)ConstantTag.NameAndType); writer.WriteUInt16(12); writer.WriteUInt16(13); // #14 + writer.WriteByte((byte)ConstantTag.Methodref); writer.WriteUInt16(11); writer.WriteUInt16(14); // #15 + + writer.WriteUInt16(0x0001); // public + writer.WriteUInt16(2); // this class + writer.WriteUInt16(4); // super class + + writer.WriteUInt16(0); // interfaces + writer.WriteUInt16(0); // fields + writer.WriteUInt16(1); // methods + + WriteResourceLookupMethod(writer, methodNameIndex: 5, descriptorIndex: 6, classConstantIndex: 4, stringIndex: 9, methodRefIndex: 15); + + writer.WriteUInt16(0); // class attributes + + return buffer.ToArray(); + } public static byte[] CreateTcclChecker(string internalClassName) { @@ -119,11 +157,11 @@ public static class JavaClassFileFactory writer.WriteBytes(codeBytes); } - private static void WriteTcclMethod(BigEndianWriter writer, ushort methodNameIndex, ushort descriptorIndex, ushort currentThreadMethodRefIndex, ushort getContextMethodRefIndex) - { - writer.WriteUInt16(0x0009); - writer.WriteUInt16(methodNameIndex); - writer.WriteUInt16(descriptorIndex); + private static void WriteTcclMethod(BigEndianWriter writer, ushort methodNameIndex, ushort descriptorIndex, ushort currentThreadMethodRefIndex, ushort getContextMethodRefIndex) + { + writer.WriteUInt16(0x0009); + writer.WriteUInt16(methodNameIndex); + writer.WriteUInt16(descriptorIndex); writer.WriteUInt16(1); writer.WriteUInt16(7); @@ -144,9 +182,40 @@ public static class JavaClassFileFactory } var codeBytes = codeBuffer.ToArray(); - writer.WriteUInt32((uint)codeBytes.Length); - writer.WriteBytes(codeBytes); - } + writer.WriteUInt32((uint)codeBytes.Length); + writer.WriteBytes(codeBytes); + } + + private static void WriteResourceLookupMethod(BigEndianWriter writer, ushort methodNameIndex, ushort descriptorIndex, ushort classConstantIndex, ushort stringIndex, ushort methodRefIndex) + { + writer.WriteUInt16(0x0009); + writer.WriteUInt16(methodNameIndex); + writer.WriteUInt16(descriptorIndex); + writer.WriteUInt16(1); + + writer.WriteUInt16(7); + using var codeBuffer = new MemoryStream(); + using (var codeWriter = new BigEndianWriter(codeBuffer)) + { + codeWriter.WriteUInt16(2); + codeWriter.WriteUInt16(0); + codeWriter.WriteUInt32(8); + codeWriter.WriteByte(0x13); // ldc_w for class literal + codeWriter.WriteUInt16(classConstantIndex); + codeWriter.WriteByte(0x12); + codeWriter.WriteByte((byte)stringIndex); + codeWriter.WriteByte(0xB6); + codeWriter.WriteUInt16(methodRefIndex); + codeWriter.WriteByte(0x57); + codeWriter.WriteByte(0xB1); + codeWriter.WriteUInt16(0); + codeWriter.WriteUInt16(0); + } + + var codeBytes = codeBuffer.ToArray(); + writer.WriteUInt32((uint)codeBytes.Length); + writer.WriteBytes(codeBytes); + } private sealed class BigEndianWriter : IDisposable { diff --git a/src/__Libraries/StellaOps.Cryptography.Kms/FileKmsClient.cs b/src/__Libraries/StellaOps.Cryptography.Kms/FileKmsClient.cs index c8c14d4d..194cd791 100644 --- a/src/__Libraries/StellaOps.Cryptography.Kms/FileKmsClient.cs +++ b/src/__Libraries/StellaOps.Cryptography.Kms/FileKmsClient.cs @@ -18,6 +18,7 @@ public sealed class FileKmsClient : IKmsClient, IDisposable new JsonStringEnumConverter(), }, }; + private const int MinKeyDerivationIterations = 600_000; private readonly FileKmsOptions _options; private readonly SemaphoreSlim _mutex = new(1, 1); @@ -36,6 +37,13 @@ public sealed class FileKmsClient : IKmsClient, IDisposable } _options = options; + if (_options.KeyDerivationIterations < MinKeyDerivationIterations) + { + throw new ArgumentOutOfRangeException( + nameof(options.KeyDerivationIterations), + _options.KeyDerivationIterations, + $"PBKDF2 iterations must be at least {MinKeyDerivationIterations:N0} to satisfy cryptographic guidance."); + } Directory.CreateDirectory(_options.RootPath); } @@ -415,7 +423,7 @@ public sealed class FileKmsClient : IKmsClient, IDisposable using var ecdsa = ECDsa.Create(); ecdsa.ImportParameters(parameters); - return ecdsa.SignData(data.ToArray(), HashAlgorithmName.SHA256); + return ecdsa.SignData(data, HashAlgorithmName.SHA256); } private bool VerifyData(string curveName, string publicKeyBase64, ReadOnlySpan data, ReadOnlySpan signature) @@ -442,7 +450,7 @@ public sealed class FileKmsClient : IKmsClient, IDisposable using var ecdsa = ECDsa.Create(); ecdsa.ImportParameters(parameters); - return ecdsa.VerifyData(data.ToArray(), signature.ToArray(), HashAlgorithmName.SHA256); + return ecdsa.VerifyData(data, signature, HashAlgorithmName.SHA256); } private KeyEnvelope EncryptPrivateKey(ReadOnlySpan privateKey) @@ -457,9 +465,10 @@ public sealed class FileKmsClient : IKmsClient, IDisposable var tag = new byte[16]; var plaintextCopy = privateKey.ToArray(); + using var aesGcm = new AesGcm(key, tag.Length); try { - AesGcm.Encrypt(key, nonce, plaintextCopy, ciphertext, tag); + aesGcm.Encrypt(nonce, plaintextCopy, ciphertext, tag); } finally { @@ -489,7 +498,8 @@ public sealed class FileKmsClient : IKmsClient, IDisposable try { var plaintext = new byte[ciphertext.Length]; - AesGcm.Decrypt(key, nonce, ciphertext, tag, plaintext); + using var aesGcm = new AesGcm(key, tag.Length); + aesGcm.Decrypt(nonce, ciphertext, tag, plaintext); return plaintext; } diff --git a/src/__Libraries/StellaOps.Cryptography.Kms/FileKmsOptions.cs b/src/__Libraries/StellaOps.Cryptography.Kms/FileKmsOptions.cs index 2e142f52..7694ac9a 100644 --- a/src/__Libraries/StellaOps.Cryptography.Kms/FileKmsOptions.cs +++ b/src/__Libraries/StellaOps.Cryptography.Kms/FileKmsOptions.cs @@ -16,12 +16,12 @@ public sealed class FileKmsOptions public required string Password { get; set; } /// - /// Signing algorithm identifier (default ED25519). + /// Signing algorithm identifier (default ES256). /// public string Algorithm { get; set; } = KmsAlgorithms.Es256; /// /// PBKDF2 iteration count for envelope encryption. /// - public int KeyDerivationIterations { get; set; } = 100_000; + public int KeyDerivationIterations { get; set; } = 600_000; } diff --git a/src/__Libraries/StellaOps.Cryptography.Kms/TASKS.md b/src/__Libraries/StellaOps.Cryptography.Kms/TASKS.md index d1355b84..be7c14cc 100644 --- a/src/__Libraries/StellaOps.Cryptography.Kms/TASKS.md +++ b/src/__Libraries/StellaOps.Cryptography.Kms/TASKS.md @@ -3,7 +3,7 @@ ## Sprint 72 – Abstractions & File Driver | ID | Status | Owner(s) | Depends on | Description | Exit Criteria | |----|--------|----------|------------|-------------|---------------| -| KMS-72-001 | DOING (2025-10-29) | KMS Guild | — | Implement KMS interface (sign, verify, metadata, rotate, revoke) and file-based key driver with encrypted at-rest storage. | Interface + file driver operational; unit tests cover sign/verify/rotation; lint passes.
2025-10-29: `FileKmsClient` (ES256) file driver scaffolding committed under `StellaOps.Cryptography.Kms`; includes disk encryption + unit tests. Follow-up: address PBKDF2/AesGcm warnings and wire into Authority services. | +| KMS-72-001 | DOING (2025-10-29) | KMS Guild | — | Implement KMS interface (sign, verify, metadata, rotate, revoke) and file-based key driver with encrypted at-rest storage. | Interface + file driver operational; unit tests cover sign/verify/rotation; lint passes.
2025-10-29: `FileKmsClient` (ES256) file driver scaffolding committed under `StellaOps.Cryptography.Kms`; includes disk encryption + unit tests. Follow-up: address PBKDF2/AesGcm warnings and wire into Authority services.
2025-10-29 18:40Z: Hardened PBKDF2 iteration floor (≥600k), switched to tag-size explicit `AesGcm` usage, removed transient array allocations, and refreshed unit tests (`StellaOps.Cryptography.Kms.Tests`). | | KMS-72-002 | TODO | KMS Guild | KMS-72-001 | Add CLI support for importing/exporting file-based keys with password protection. | CLI commands functional; docs updated; integration tests pass. | ## Sprint 73 – Cloud & HSM Integration