feat: Implement ScannerSurfaceSecretConfigurator for web service options
Some checks failed
Docs CI / lint-and-preview (push) Has been cancelled
Some checks failed
Docs CI / lint-and-preview (push) Has been cancelled
- Added ScannerSurfaceSecretConfigurator to configure ScannerWebServiceOptions using surface secrets. - Integrated ISurfaceSecretProvider to fetch and apply secrets for artifact store configuration. - Enhanced logging for secret retrieval and application processes. feat: Implement ScannerStorageSurfaceSecretConfigurator for worker options - Introduced ScannerStorageSurfaceSecretConfigurator to configure ScannerStorageOptions with surface secrets. - Utilized ISurfaceSecretProvider to retrieve and apply secrets for object store settings. - Improved logging for secret handling and configuration. feat: Create SurfaceManifestPublisher for publishing surface manifests - Developed SurfaceManifestPublisher to handle the creation and storage of surface manifests. - Implemented methods for serializing manifest documents and storing payloads in the object store. - Added dual write functionality for mirror storage of manifests. feat: Add SurfaceManifestStageExecutor for processing scan stages - Created SurfaceManifestStageExecutor to execute the manifest publishing stage in scan jobs. - Integrated with SurfaceManifestPublisher to publish manifests based on collected payloads. - Enhanced logging for job processing and manifest storage. feat: Define SurfaceManifest models for manifest structure - Established SurfaceManifestDocument, SurfaceManifestSource, SurfaceManifestArtifact, and SurfaceManifestStorage records. - Implemented serialization attributes for JSON handling of manifest models. feat: Implement CasAccessSecret and SurfaceSecretParser for secret handling - Created CasAccessSecret record to represent surface access secrets. - Developed SurfaceSecretParser to parse and validate surface secrets from JSON payloads. test: Add unit tests for CasAccessSecretParser - Implemented tests for parsing CasAccessSecret from JSON payloads and metadata fallbacks. - Verified expected values and behavior for secret parsing logic. test: Add unit tests for ScannerSurfaceSecretConfigurator - Created tests for ScannerSurfaceSecretConfigurator to ensure correct application of surface secrets to web service options. - Validated artifact store settings after configuration. test: Add unit tests for ScannerStorageSurfaceSecretConfigurator - Implemented tests for ScannerStorageSurfaceSecretConfigurator to verify correct application of surface secrets to storage options. - Ensured accurate configuration of object store settings.
This commit is contained in:
master
@@ -43,7 +43,7 @@ Follow the sprint files below in order. Update task status in both `SPRINTS` and
|
||||
> 2025-11-03: MERGE-LNM-21-001 moved to DOING (BE-Merge, Architecture Guild) – drafting `no-merge` migration playbook outline and capturing rollout/backfill checkpoints.
|
||||
> 2025-11-03: MERGE-LNM-21-001 marked DONE – published `docs/migration/no-merge.md` with rollout, backfill, validation, and rollback guidance for the LNM cutover.
|
||||
> 2025-11-04: GRAPH-INDEX-28-011 marked DONE (Graph Indexer Guild) – SBOM ingest DI wiring now emits graph snapshots by default, snapshot root configurable via `STELLAOPS_GRAPH_SNAPSHOT_DIR`, and Graph Indexer tests exercised with Mongo URI guidance.
|
||||
> 2025-11-03: MERGE-LNM-21-002 moved to DOING (BE-Merge) – auditing `AdvisoryMergeService` call sites to scope removal and analyzer enforcement.
|
||||
> 2025-11-06: MERGE-LNM-21-002 remains DOING (BE-Merge) – default-off merge DI + job gating landed, but Concelier WebService ingest/mirror tests are failing; guard and migration fixes pending before completion.
|
||||
> 2025-11-03: DOCS-LNM-22-008 moved to DOING (Docs Guild, DevOps Guild) – aligning migration playbook structure and readiness checklist.
|
||||
> 2025-11-03: DOCS-LNM-22-008 marked DONE – `/docs/migration/no-merge.md` published for DevOps/Export Center planning with checklist for cutover readiness.
|
||||
> 2025-11-03: SCHED-CONSOLE-27-001 marked DONE (Scheduler WebService Guild, Policy Registry Guild) – policy simulation endpoints now emit SSE retry/heartbeat, enforce metadata normalization, support Mongo-backed integration, and ship auth/stream coverage.
|
||||
|
||||
@@ -210,7 +210,7 @@ Depends on: Sprint 110.B - Concelier.VI
|
||||
Summary: Ingestion & Evidence focus on Concelier (phase VII).
|
||||
Task ID | State | Task description | Owners (Source)
|
||||
--- | --- | --- | ---
|
||||
MERGE-LNM-21-002 | DOING (2025-11-03) | Refactor or retire `AdvisoryMergeService` and related pipelines, ensuring callers transition to observation/linkset APIs; add compile-time analyzer preventing merge service usage.<br>2025-11-03: Began dependency audit and call-site inventory ahead of deprecation plan; cataloging service registrations/tests referencing merge APIs.<br>2025-11-05 14:42Z: Drafting `concelier:features:noMergeEnabled` gating, merge job allowlist handling, and deprecation/telemetry changes prior to analyzer rollout.<br>2025-11-06 16:10Z: Landed analyzer project (`CONCELIER0002`), wired into Concelier WebService/tests, and updated docs to direct suppressions through explicit migration notes.<br>2025-11-06 23:45Z: Analyzer enforcement merged; DI removal + flag defaults pending. Analyzer test project blocked by offline feed (`Microsoft.Bcl.AsyncInterfaces >= 8.0` missing) — rerun once nuget mirror refreshed. | BE-Merge (src/Concelier/__Libraries/StellaOps.Concelier.Merge/TASKS.md)
|
||||
MERGE-LNM-21-002 | DOING (2025-11-06) | Refactor or retire `AdvisoryMergeService` and related pipelines, ensuring callers transition to observation/linkset APIs; add compile-time analyzer preventing merge service usage.<br>2025-11-03: Began dependency audit and call-site inventory ahead of deprecation plan; cataloging service registrations/tests referencing merge APIs.<br>2025-11-05 14:42Z: Drafted `concelier:features:noMergeEnabled` gating, merge job allowlist handling, and deprecation/telemetry changes prior to analyzer rollout.<br>2025-11-06 16:10Z: Landed analyzer project (`CONCELIER0002`), wired into Concelier WebService/tests, and updated docs to direct suppressions through explicit migration notes.<br>2025-11-07 03:25Z: Default-on toggle + job gating break existing Concelier WebService tests; guard/migration adjustments pending before closing the task. | BE-Merge (src/Concelier/__Libraries/StellaOps.Concelier.Merge/TASKS.md)
|
||||
MERGE-LNM-21-003 Determinism/test updates | QA Guild, BE-Merge | Replace merge determinism suites with observation/linkset regression tests verifying no data mutation and conflicts remain visible. Dependencies: MERGE-LNM-21-002. | MERGE-LNM-21-002 (src/Concelier/__Libraries/StellaOps.Concelier.Merge/TASKS.md)
|
||||
|
||||
|
||||
|
||||
@@ -142,8 +142,8 @@ SCANNER-EVENTS-16-302 | DONE (2025-11-06) | Extend orchestrator event links (rep
|
||||
SCANNER-GRAPH-21-001 | TODO | Provide webhook/REST endpoint for Cartographer to request policy overlays and runtime evidence for graph nodes, ensuring determinism and tenant scoping. | Scanner WebService Guild, Cartographer Guild (src/Scanner/StellaOps.Scanner.WebService/TASKS.md)
|
||||
SCANNER-LNM-21-001 | TODO | Update `/reports` and `/policy/runtime` payloads to consume advisory/vex linksets, exposing source severity arrays and conflict summaries alongside effective verdicts. | Scanner WebService Guild, Policy Guild (src/Scanner/StellaOps.Scanner.WebService/TASKS.md)
|
||||
SCANNER-LNM-21-002 | TODO | Add evidence endpoint for Console to fetch linkset summaries with policy overlay for a component/SBOM, including AOC references. Dependencies: SCANNER-LNM-21-001. | Scanner WebService Guild, UI Guild (src/Scanner/StellaOps.Scanner.WebService/TASKS.md)
|
||||
SCANNER-SECRETS-01 | DOING (2025-11-02) | Adopt `StellaOps.Scanner.Surface.Secrets` for registry/CAS credentials during scan execution.<br>2025-11-02: Worker integration tests added for CAS token retrieval via Surface.Secrets abstraction; refactor under review. | Scanner Worker Guild, Security Guild (src/Scanner/StellaOps.Scanner.Worker/TASKS.md)
|
||||
SCANNER-SECRETS-02 | DOING (2025-11-02) | Replace ad-hoc secret wiring with Surface.Secrets for report/export operations (registry and CAS tokens). Dependencies: SCANNER-SECRETS-01.<br>2025-11-02: WebService export path now resolves registry credentials via Surface.Secrets stub; CI pipeline hook in progress. | Scanner WebService Guild, Security Guild (src/Scanner/StellaOps.Scanner.WebService/TASKS.md)
|
||||
SCANNER-SECRETS-01 | DOING (2025-11-06) | Adopt `StellaOps.Scanner.Surface.Secrets` for registry/CAS credentials during scan execution.<br>2025-11-02: Worker integration tests added for CAS token retrieval via Surface.Secrets abstraction; refactor under review.<br>2025-11-06: Resumed to replace remaining registry credential plumbing and emit rotation-aware metrics.<br>2025-11-06 21:35Z: Surface secret configurator now hydrates `ScannerStorageOptions` from `cas-access` payloads; unit coverage added. | Scanner Worker Guild, Security Guild (src/Scanner/StellaOps.Scanner.Worker/TASKS.md)
|
||||
SCANNER-SECRETS-02 | DOING (2025-11-06) | Replace ad-hoc secret wiring with Surface.Secrets for report/export operations (registry and CAS tokens). Dependencies: SCANNER-SECRETS-01.<br>2025-11-02: WebService export path now resolves registry credentials via Surface.Secrets stub; CI pipeline hook in progress.<br>2025-11-06: Picking up Surface.Secrets provider usage across report/export flows and removing legacy secret file readers.<br>2025-11-06 21:40Z: WebService options now consume `cas-access` secrets via configurator; storage mirrors updated; targeted tests passing. | Scanner WebService Guild, Security Guild (src/Scanner/StellaOps.Scanner.WebService/TASKS.md)
|
||||
SCANNER-SECRETS-03 | TODO | Use Surface.Secrets to retrieve registry credentials when interacting with CAS/referrers. Dependencies: SCANNER-SECRETS-02. | BuildX Plugin Guild, Security Guild (src/Scanner/StellaOps.Scanner.Sbomer.BuildXPlugin/TASKS.md)
|
||||
SCANNER-ENG-0020 | TODO | Implement Homebrew collector & fragment mapper per `design/macos-analyzer.md` §3.1. | Scanner Guild (docs/modules/scanner/TASKS.md)
|
||||
SCANNER-ENG-0021 | TODO | Implement pkgutil receipt collector per `design/macos-analyzer.md` §3.2. | Scanner Guild (docs/modules/scanner/TASKS.md)
|
||||
@@ -153,9 +153,9 @@ SCANNER-ENG-0024 | TODO | Implement Windows MSI collector per `design/windows-an
|
||||
SCANNER-ENG-0025 | TODO | Implement WinSxS manifest collector per `design/windows-analyzer.md` §3.2. | Scanner Guild (docs/modules/scanner/TASKS.md)
|
||||
SCANNER-ENG-0026 | TODO | Implement Windows Chocolatey & registry collectors per `design/windows-analyzer.md` §3.3–3.4. | Scanner Guild (docs/modules/scanner/TASKS.md)
|
||||
SCANNER-ENG-0027 | TODO | Deliver Windows policy/offline integration per `design/windows-analyzer.md` §5–6. | Scanner Guild, Policy Guild, Offline Kit Guild (docs/modules/scanner/TASKS.md)
|
||||
SCANNER-SURFACE-01 | DOING (2025-11-02) | Persist Surface.FS manifests after analyzer stages, including layer CAS metadata and EntryTrace fragments.<br>2025-11-02: Worker pipeline emitting draft Surface.FS manifests for sample scans; determinism checks running. | Scanner Worker Guild (src/Scanner/StellaOps.Scanner.Worker/TASKS.md)
|
||||
SCANNER-SURFACE-01 | DOING (2025-11-06) | Persist Surface.FS manifests after analyzer stages, including layer CAS metadata and EntryTrace fragments.<br>2025-11-02: Worker pipeline emitting draft Surface.FS manifests for sample scans; determinism checks running.<br>2025-11-06: Continuing with manifest writer abstraction + telemetry wiring for Surface.FS persistence. | Scanner Worker Guild (src/Scanner/StellaOps.Scanner.Worker/TASKS.md)
|
||||
SCANNER-SURFACE-02 | DONE (2025-11-05) | Publish Surface.FS pointers (CAS URIs, manifests) via scan/report APIs and update attestation metadata. Dependencies: SCANNER-SURFACE-01.<br>2025-11-05: Surface pointer projection wired through WebService endpoints, orchestrator samples & DSSE fixtures refreshed with `surface` manifest block, and regression suite (platform events, report sample, ready check) updated. | Scanner WebService Guild (src/Scanner/StellaOps.Scanner.WebService/TASKS.md)
|
||||
SCANNER-SURFACE-03 | TODO | Push layer manifests and entry fragments into Surface.FS during build-time SBOM generation. Dependencies: SCANNER-SURFACE-02. | BuildX Plugin Guild (src/Scanner/StellaOps.Scanner.Sbomer.BuildXPlugin/TASKS.md)
|
||||
SCANNER-SURFACE-03 | DOING (2025-11-06) | Push layer manifests and entry fragments into Surface.FS during build-time SBOM generation. Dependencies: SCANNER-SURFACE-02.<br>2025-11-06: Starting BuildX manifest upload implementation with Surface.FS client abstraction and integration tests. | BuildX Plugin Guild (src/Scanner/StellaOps.Scanner.Sbomer.BuildXPlugin/TASKS.md)
|
||||
|
||||
[Scanner & Surface] 130.A) Scanner.VIII
|
||||
Depends on: Sprint 130.A - Scanner.VII
|
||||
|
||||
@@ -7,6 +7,7 @@ using Microsoft.AspNetCore.Routing;
|
||||
using Microsoft.Extensions.DependencyInjection;
|
||||
using Microsoft.Extensions.Options;
|
||||
using StellaOps.Aoc;
|
||||
using StellaOps.Aoc.AspNetCore.Results;
|
||||
|
||||
namespace StellaOps.Aoc.AspNetCore.Routing;
|
||||
|
||||
@@ -55,7 +56,14 @@ public sealed class AocGuardEndpointFilter<TRequest> : IEndpointFilter
|
||||
_ => JsonSerializer.SerializeToElement(payload, _serializerOptions)
|
||||
};
|
||||
|
||||
guard.ValidateOrThrow(element, options);
|
||||
try
|
||||
{
|
||||
guard.ValidateOrThrow(element, options);
|
||||
}
|
||||
catch (AocGuardException exception)
|
||||
{
|
||||
return AocHttpResults.Problem(context.HttpContext, exception);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -111,7 +111,7 @@ internal static class JobRegistrationExtensions
|
||||
|
||||
private static void ConfigureMergeJob(JobSchedulerOptions options, IConfiguration configuration)
|
||||
{
|
||||
var noMergeEnabled = configuration.GetValue("concelier:features:noMergeEnabled", true);
|
||||
var noMergeEnabled = configuration.GetValue<bool?>("concelier:features:noMergeEnabled") ?? true;
|
||||
if (noMergeEnabled)
|
||||
{
|
||||
options.Definitions.Remove(MergeReconcileBuiltInJob.Kind);
|
||||
|
||||
@@ -244,7 +244,7 @@ if (resolvedAuthority.Enabled && resolvedAuthority.AllowAnonymousFallback)
|
||||
|
||||
app.MapConcelierMirrorEndpoints(authorityConfigured, enforceAuthority);
|
||||
|
||||
app.MapGet("/.well-known/openapi", (OpenApiDiscoveryDocumentProvider provider, HttpContext context) =>
|
||||
app.MapGet("/.well-known/openapi", ([FromServices] OpenApiDiscoveryDocumentProvider provider, HttpContext context) =>
|
||||
{
|
||||
var (payload, etag) = provider.GetDocument();
|
||||
|
||||
@@ -299,7 +299,7 @@ var observationsEndpoint = app.MapGet("/concelier/observations", async (
|
||||
[FromQuery(Name = "cpe")] string[]? cpes,
|
||||
[FromQuery(Name = "limit")] int? limit,
|
||||
[FromQuery(Name = "cursor")] string? cursor,
|
||||
IAdvisoryObservationQueryService queryService,
|
||||
[FromServices] IAdvisoryObservationQueryService queryService,
|
||||
CancellationToken cancellationToken) =>
|
||||
{
|
||||
ApplyNoCache(context.Response);
|
||||
@@ -356,8 +356,8 @@ if (authorityConfigured)
|
||||
var advisoryIngestEndpoint = app.MapPost("/ingest/advisory", async (
|
||||
HttpContext context,
|
||||
AdvisoryIngestRequest request,
|
||||
IAdvisoryRawService rawService,
|
||||
TimeProvider timeProvider,
|
||||
[FromServices] IAdvisoryRawService rawService,
|
||||
[FromServices] TimeProvider timeProvider,
|
||||
CancellationToken cancellationToken) =>
|
||||
{
|
||||
ApplyNoCache(context.Response);
|
||||
@@ -470,7 +470,7 @@ if (authorityConfigured)
|
||||
|
||||
var advisoryRawListEndpoint = app.MapGet("/advisories/raw", async (
|
||||
HttpContext context,
|
||||
IAdvisoryRawService rawService,
|
||||
[FromServices] IAdvisoryRawService rawService,
|
||||
CancellationToken cancellationToken) =>
|
||||
{
|
||||
ApplyNoCache(context.Response);
|
||||
@@ -560,7 +560,7 @@ if (authorityConfigured)
|
||||
var advisoryRawGetEndpoint = app.MapGet("/advisories/raw/{id}", async (
|
||||
string id,
|
||||
HttpContext context,
|
||||
IAdvisoryRawService rawService,
|
||||
[FromServices] IAdvisoryRawService rawService,
|
||||
CancellationToken cancellationToken) =>
|
||||
{
|
||||
ApplyNoCache(context.Response);
|
||||
@@ -604,7 +604,7 @@ if (authorityConfigured)
|
||||
var advisoryRawProvenanceEndpoint = app.MapGet("/advisories/raw/{id}/provenance", async (
|
||||
string id,
|
||||
HttpContext context,
|
||||
IAdvisoryRawService rawService,
|
||||
[FromServices] IAdvisoryRawService rawService,
|
||||
CancellationToken cancellationToken) =>
|
||||
{
|
||||
ApplyNoCache(context.Response);
|
||||
@@ -650,8 +650,8 @@ if (authorityConfigured)
|
||||
var aocVerifyEndpoint = app.MapPost("/aoc/verify", async (
|
||||
HttpContext context,
|
||||
AocVerifyRequest request,
|
||||
IAdvisoryRawService rawService,
|
||||
TimeProvider timeProvider,
|
||||
[FromServices] IAdvisoryRawService rawService,
|
||||
[FromServices] TimeProvider timeProvider,
|
||||
CancellationToken cancellationToken) =>
|
||||
{
|
||||
ApplyNoCache(context.Response);
|
||||
@@ -734,7 +734,7 @@ if (authorityConfigured)
|
||||
app.MapGet("/concelier/advisories/{vulnerabilityKey}/replay", async (
|
||||
string vulnerabilityKey,
|
||||
DateTimeOffset? asOf,
|
||||
IAdvisoryEventLog eventLog,
|
||||
[FromServices] IAdvisoryEventLog eventLog,
|
||||
CancellationToken cancellationToken) =>
|
||||
{
|
||||
if (string.IsNullOrWhiteSpace(vulnerabilityKey))
|
||||
@@ -1010,7 +1010,7 @@ void ApplyNoCache(HttpResponse response)
|
||||
|
||||
await InitializeMongoAsync(app);
|
||||
|
||||
app.MapGet("/health", (IOptions<ConcelierOptions> opts, ServiceStatus status, HttpContext context) =>
|
||||
app.MapGet("/health", ([FromServices] IOptions<ConcelierOptions> opts, [FromServices] ServiceStatus status, HttpContext context) =>
|
||||
{
|
||||
ApplyNoCache(context.Response);
|
||||
|
||||
@@ -1039,7 +1039,7 @@ app.MapGet("/health", (IOptions<ConcelierOptions> opts, ServiceStatus status, Ht
|
||||
return JsonResult(response);
|
||||
});
|
||||
|
||||
app.MapGet("/ready", async (IMongoDatabase database, ServiceStatus status, HttpContext context, CancellationToken cancellationToken) =>
|
||||
app.MapGet("/ready", async ([FromServices] IMongoDatabase database, [FromServices] ServiceStatus status, HttpContext context, CancellationToken cancellationToken) =>
|
||||
{
|
||||
ApplyNoCache(context.Response);
|
||||
|
||||
@@ -1097,7 +1097,7 @@ app.MapGet("/ready", async (IMongoDatabase database, ServiceStatus status, HttpC
|
||||
}
|
||||
});
|
||||
|
||||
app.MapGet("/diagnostics/aliases/{seed}", async (string seed, AliasGraphResolver resolver, HttpContext context, CancellationToken cancellationToken) =>
|
||||
app.MapGet("/diagnostics/aliases/{seed}", async (string seed, [FromServices] AliasGraphResolver resolver, HttpContext context, CancellationToken cancellationToken) =>
|
||||
{
|
||||
ApplyNoCache(context.Response);
|
||||
|
||||
@@ -1137,7 +1137,7 @@ app.MapGet("/diagnostics/aliases/{seed}", async (string seed, AliasGraphResolver
|
||||
return JsonResult(response);
|
||||
});
|
||||
|
||||
var jobsListEndpoint = app.MapGet("/jobs", async (string? kind, int? limit, IJobCoordinator coordinator, HttpContext context, CancellationToken cancellationToken) =>
|
||||
var jobsListEndpoint = app.MapGet("/jobs", async (string? kind, int? limit, [FromServices] IJobCoordinator coordinator, HttpContext context, CancellationToken cancellationToken) =>
|
||||
{
|
||||
ApplyNoCache(context.Response);
|
||||
|
||||
@@ -1151,7 +1151,7 @@ if (enforceAuthority)
|
||||
jobsListEndpoint.RequireAuthorization(JobsPolicyName);
|
||||
}
|
||||
|
||||
var jobByIdEndpoint = app.MapGet("/jobs/{runId:guid}", async (Guid runId, IJobCoordinator coordinator, HttpContext context, CancellationToken cancellationToken) =>
|
||||
var jobByIdEndpoint = app.MapGet("/jobs/{runId:guid}", async (Guid runId, [FromServices] IJobCoordinator coordinator, HttpContext context, CancellationToken cancellationToken) =>
|
||||
{
|
||||
ApplyNoCache(context.Response);
|
||||
|
||||
@@ -1168,7 +1168,7 @@ if (enforceAuthority)
|
||||
jobByIdEndpoint.RequireAuthorization(JobsPolicyName);
|
||||
}
|
||||
|
||||
var jobDefinitionsEndpoint = app.MapGet("/jobs/definitions", async (IJobCoordinator coordinator, HttpContext context, CancellationToken cancellationToken) =>
|
||||
var jobDefinitionsEndpoint = app.MapGet("/jobs/definitions", async ([FromServices] IJobCoordinator coordinator, HttpContext context, CancellationToken cancellationToken) =>
|
||||
{
|
||||
ApplyNoCache(context.Response);
|
||||
|
||||
@@ -1195,7 +1195,7 @@ if (enforceAuthority)
|
||||
jobDefinitionsEndpoint.RequireAuthorization(JobsPolicyName);
|
||||
}
|
||||
|
||||
var jobDefinitionEndpoint = app.MapGet("/jobs/definitions/{kind}", async (string kind, IJobCoordinator coordinator, HttpContext context, CancellationToken cancellationToken) =>
|
||||
var jobDefinitionEndpoint = app.MapGet("/jobs/definitions/{kind}", async (string kind, [FromServices] IJobCoordinator coordinator, HttpContext context, CancellationToken cancellationToken) =>
|
||||
{
|
||||
ApplyNoCache(context.Response);
|
||||
|
||||
@@ -1218,7 +1218,7 @@ if (enforceAuthority)
|
||||
jobDefinitionEndpoint.RequireAuthorization(JobsPolicyName);
|
||||
}
|
||||
|
||||
var jobDefinitionRunsEndpoint = app.MapGet("/jobs/definitions/{kind}/runs", async (string kind, int? limit, IJobCoordinator coordinator, HttpContext context, CancellationToken cancellationToken) =>
|
||||
var jobDefinitionRunsEndpoint = app.MapGet("/jobs/definitions/{kind}/runs", async (string kind, int? limit, [FromServices] IJobCoordinator coordinator, HttpContext context, CancellationToken cancellationToken) =>
|
||||
{
|
||||
ApplyNoCache(context.Response);
|
||||
|
||||
@@ -1240,7 +1240,7 @@ if (enforceAuthority)
|
||||
jobDefinitionRunsEndpoint.RequireAuthorization(JobsPolicyName);
|
||||
}
|
||||
|
||||
var activeJobsEndpoint = app.MapGet("/jobs/active", async (IJobCoordinator coordinator, HttpContext context, CancellationToken cancellationToken) =>
|
||||
var activeJobsEndpoint = app.MapGet("/jobs/active", async ([FromServices] IJobCoordinator coordinator, HttpContext context, CancellationToken cancellationToken) =>
|
||||
{
|
||||
ApplyNoCache(context.Response);
|
||||
|
||||
@@ -1253,7 +1253,7 @@ if (enforceAuthority)
|
||||
activeJobsEndpoint.RequireAuthorization(JobsPolicyName);
|
||||
}
|
||||
|
||||
var triggerJobEndpoint = app.MapPost("/jobs/{*jobKind}", async (string jobKind, JobTriggerRequest request, IJobCoordinator coordinator, HttpContext context) =>
|
||||
var triggerJobEndpoint = app.MapPost("/jobs/{*jobKind}", async (string jobKind, JobTriggerRequest request, [FromServices] IJobCoordinator coordinator, HttpContext context) =>
|
||||
{
|
||||
ApplyNoCache(context.Response);
|
||||
|
||||
|
||||
@@ -18,7 +18,7 @@ public static class MergeServiceCollectionExtensions
|
||||
ArgumentNullException.ThrowIfNull(services);
|
||||
ArgumentNullException.ThrowIfNull(configuration);
|
||||
|
||||
var noMergeEnabled = configuration.GetValue<bool?>("concelier:features:noMergeEnabled");
|
||||
var noMergeEnabled = configuration.GetValue<bool?>("concelier:features:noMergeEnabled") ?? true;
|
||||
if (noMergeEnabled)
|
||||
{
|
||||
return services;
|
||||
|
||||
@@ -10,6 +10,6 @@
|
||||
| Task | Owner(s) | Depends on | Notes |
|
||||
|---|---|---|---|
|
||||
|MERGE-LNM-21-001 Migration plan authoring|BE-Merge, Architecture Guild|CONCELIER-LNM-21-101|**DONE (2025-11-03)** – Authored `docs/migration/no-merge.md` with rollout phases, backfill/validation checklists, rollback guidance, and ownership matrix for the Link-Not-Merge cutover.|
|
||||
|MERGE-LNM-21-002 Merge service deprecation|BE-Merge|MERGE-LNM-21-001|**DONE (2025-11-06)** – Audited service registrations, gated legacy bindings, and delivered analyzer coverage ahead of removal.<br>2025-11-05 14:42Z: Implemented `concelier:features:noMergeEnabled` gate, merge job allowlist checks, `[Obsolete]` markings, and analyzer scaffolding to steer consumers toward linkset APIs.<br>2025-11-06 16:10Z: Introduced Roslyn analyzer (`CONCELIER0002`) referenced by Concelier WebService + tests, documented suppression guidance, and updated migration playbook.<br>2025-11-06 23:58Z: Defaulted `concelier:features:noMergeEnabled` to `true`, removed the built-in `merge:reconcile` job unless explicitly allowlisted, refreshed WebService tests/docs, and verified analyzer suites restore against local feeds.|
|
||||
|MERGE-LNM-21-002 Merge service deprecation|BE-Merge|MERGE-LNM-21-001|**DOING (2025-11-06)** – Defaulted `concelier:features:noMergeEnabled` to `true`, added merge job allowlist gate, and began rewiring guard/tier tests; follow-up work required to restore Concelier WebService test suite before declaring completion.<br>2025-11-05 14:42Z: Implemented `concelier:features:noMergeEnabled` gate, merge job allowlist checks, `[Obsolete]` markings, and analyzer scaffolding to steer consumers toward linkset APIs.<br>2025-11-06 16:10Z: Introduced Roslyn analyzer (`CONCELIER0002`) referenced by Concelier WebService + tests, documented suppression guidance, and updated migration playbook.<br>2025-11-07 03:25Z: Default-on toggle + job gating break existing Concelier WebService tests; guard + seed fixes pending to unblock ingest/mirror suites.|
|
||||
> 2025-11-03: Catalogued call sites (WebService Program `AddMergeModule`, built-in job registration `merge:reconcile`, `MergeReconcileJob`) and confirmed unit tests are the only direct `MergeAsync` callers; next step is to define analyzer + replacement observability coverage.
|
||||
|MERGE-LNM-21-003 Determinism/test updates|QA Guild, BE-Merge|MERGE-LNM-21-002|Replace merge determinism suites with observation/linkset regression tests verifying no data mutation and conflicts remain visible.|
|
||||
|
||||
@@ -2,6 +2,6 @@
|
||||
|
||||
| ID | Status | Owner(s) | Depends on | Description | Exit Criteria |
|
||||
|----|--------|----------|------------|-------------|---------------|
|
||||
| SCANNER-SURFACE-03 | TODO | BuildX Plugin Guild | SURFACE-FS-02 | Push layer manifests and entry fragments into Surface.FS during build-time SBOM generation. | BuildX integration tests confirm cache population; CLI docs updated. |
|
||||
| SCANNER-SURFACE-03 | DOING (2025-11-06) | BuildX Plugin Guild | SURFACE-FS-02 | Push layer manifests and entry fragments into Surface.FS during build-time SBOM generation.<br>2025-11-06: Kicked off manifest emitter wiring within BuildX export pipeline and outlined test fixtures targeting Surface.FS client mock. | BuildX integration tests confirm cache population; CLI docs updated. |
|
||||
| SCANNER-ENV-03 | TODO | BuildX Plugin Guild | SURFACE-ENV-02 | Adopt Surface.Env helpers for plugin configuration (cache roots, CAS endpoints, feature toggles). | Plugin loads helper; misconfig errors logged; README updated. |
|
||||
| SCANNER-SECRETS-03 | TODO | BuildX Plugin Guild, Security Guild | SURFACE-SECRETS-02 | Use Surface.Secrets to retrieve registry credentials when interacting with CAS/referrers. | Secrets retrieved via shared library; e2e tests cover rotation; operations guide refreshed. |
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
using System;
|
||||
using System.Collections.Generic;
|
||||
using System.Text.Json.Serialization;
|
||||
using StellaOps.Scanner.Surface.FS;
|
||||
|
||||
namespace StellaOps.Scanner.WebService.Contracts;
|
||||
|
||||
@@ -28,60 +28,3 @@ public sealed record SurfacePointersDto
|
||||
[JsonPropertyOrder(4)]
|
||||
public SurfaceManifestDocument Manifest { get; init; } = new();
|
||||
}
|
||||
|
||||
public sealed record SurfaceManifestDocument
|
||||
{
|
||||
[JsonPropertyName("schema")]
|
||||
[JsonPropertyOrder(0)]
|
||||
public string Schema { get; init; } = "stellaops.surface.manifest@1";
|
||||
|
||||
[JsonPropertyName("tenant")]
|
||||
[JsonPropertyOrder(1)]
|
||||
public string Tenant { get; init; } = string.Empty;
|
||||
|
||||
[JsonPropertyName("imageDigest")]
|
||||
[JsonPropertyOrder(2)]
|
||||
public string ImageDigest { get; init; } = string.Empty;
|
||||
|
||||
[JsonPropertyName("generatedAt")]
|
||||
[JsonPropertyOrder(3)]
|
||||
public DateTimeOffset GeneratedAt { get; init; } = DateTimeOffset.UtcNow;
|
||||
|
||||
[JsonPropertyName("artifacts")]
|
||||
[JsonPropertyOrder(4)]
|
||||
public IReadOnlyList<SurfaceManifestArtifact> Artifacts { get; init; } = Array.Empty<SurfaceManifestArtifact>();
|
||||
}
|
||||
|
||||
public sealed record SurfaceManifestArtifact
|
||||
{
|
||||
[JsonPropertyName("kind")]
|
||||
[JsonPropertyOrder(0)]
|
||||
public string Kind { get; init; } = string.Empty;
|
||||
|
||||
[JsonPropertyName("uri")]
|
||||
[JsonPropertyOrder(1)]
|
||||
public string Uri { get; init; } = string.Empty;
|
||||
|
||||
[JsonPropertyName("digest")]
|
||||
[JsonPropertyOrder(2)]
|
||||
public string Digest { get; init; } = string.Empty;
|
||||
|
||||
[JsonPropertyName("mediaType")]
|
||||
[JsonPropertyOrder(3)]
|
||||
public string MediaType { get; init; } = string.Empty;
|
||||
|
||||
[JsonPropertyName("format")]
|
||||
[JsonPropertyOrder(4)]
|
||||
public string Format { get; init; } = string.Empty;
|
||||
|
||||
[JsonPropertyName("sizeBytes")]
|
||||
[JsonPropertyOrder(5)]
|
||||
public long SizeBytes { get; init; }
|
||||
= 0;
|
||||
|
||||
[JsonPropertyName("view")]
|
||||
[JsonPropertyOrder(6)]
|
||||
[JsonIgnore(Condition = JsonIgnoreCondition.WhenWritingNull)]
|
||||
public string? View { get; init; }
|
||||
= null;
|
||||
}
|
||||
|
||||
@@ -0,0 +1,118 @@
|
||||
using Microsoft.Extensions.Logging;
|
||||
using Microsoft.Extensions.Options;
|
||||
using StellaOps.Scanner.Storage;
|
||||
|
||||
namespace StellaOps.Scanner.WebService.Options;
|
||||
|
||||
internal sealed class ScannerStorageOptionsPostConfigurator : IPostConfigureOptions<ScannerStorageOptions>
|
||||
{
|
||||
private readonly IOptionsMonitor<ScannerWebServiceOptions> _webOptions;
|
||||
private readonly ILogger<ScannerStorageOptionsPostConfigurator> _logger;
|
||||
|
||||
public ScannerStorageOptionsPostConfigurator(
|
||||
IOptionsMonitor<ScannerWebServiceOptions> webOptions,
|
||||
ILogger<ScannerStorageOptionsPostConfigurator> logger)
|
||||
{
|
||||
_webOptions = webOptions ?? throw new ArgumentNullException(nameof(webOptions));
|
||||
_logger = logger ?? throw new ArgumentNullException(nameof(logger));
|
||||
}
|
||||
|
||||
public void PostConfigure(string? name, ScannerStorageOptions options)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(options);
|
||||
|
||||
var source = _webOptions.CurrentValue?.ArtifactStore;
|
||||
if (source is null)
|
||||
{
|
||||
return;
|
||||
}
|
||||
|
||||
var target = options.ObjectStore ??= new ObjectStoreOptions();
|
||||
|
||||
if (!string.IsNullOrWhiteSpace(source.Driver))
|
||||
{
|
||||
target.Driver = source.Driver;
|
||||
}
|
||||
|
||||
if (!string.IsNullOrWhiteSpace(source.Region))
|
||||
{
|
||||
target.Region = source.Region!;
|
||||
}
|
||||
|
||||
if (!string.IsNullOrWhiteSpace(source.Bucket))
|
||||
{
|
||||
target.BucketName = source.Bucket!;
|
||||
}
|
||||
|
||||
if (!string.IsNullOrWhiteSpace(source.RootPrefix))
|
||||
{
|
||||
target.RootPrefix = source.RootPrefix;
|
||||
}
|
||||
|
||||
if (!string.IsNullOrWhiteSpace(source.Endpoint))
|
||||
{
|
||||
if (target.IsRustFsDriver())
|
||||
{
|
||||
target.RustFs ??= new RustFsOptions();
|
||||
target.RustFs.BaseUrl = source.Endpoint;
|
||||
}
|
||||
else
|
||||
{
|
||||
target.ServiceUrl = source.Endpoint;
|
||||
}
|
||||
}
|
||||
|
||||
if (target.IsRustFsDriver())
|
||||
{
|
||||
if (target.RustFs is null)
|
||||
{
|
||||
target.RustFs = new RustFsOptions();
|
||||
}
|
||||
|
||||
target.RustFs.AllowInsecureTls = source.AllowInsecureTls;
|
||||
|
||||
if (!string.IsNullOrWhiteSpace(source.ApiKeyHeader))
|
||||
{
|
||||
target.RustFs.ApiKeyHeader = source.ApiKeyHeader!;
|
||||
}
|
||||
|
||||
if (!string.IsNullOrWhiteSpace(source.ApiKey))
|
||||
{
|
||||
target.RustFs.ApiKey = source.ApiKey;
|
||||
}
|
||||
|
||||
if (!string.IsNullOrWhiteSpace(source.Endpoint))
|
||||
{
|
||||
target.RustFs.BaseUrl = source.Endpoint!;
|
||||
}
|
||||
}
|
||||
|
||||
if (!string.IsNullOrWhiteSpace(source.AccessKey))
|
||||
{
|
||||
target.AccessKeyId = source.AccessKey;
|
||||
}
|
||||
|
||||
if (!string.IsNullOrWhiteSpace(source.SecretKey))
|
||||
{
|
||||
target.SecretAccessKey = source.SecretKey;
|
||||
}
|
||||
|
||||
if (source.Headers is { Count: > 0 })
|
||||
{
|
||||
foreach (var (key, value) in source.Headers)
|
||||
{
|
||||
if (string.IsNullOrWhiteSpace(key) || string.IsNullOrWhiteSpace(value))
|
||||
{
|
||||
continue;
|
||||
}
|
||||
|
||||
target.Headers[key] = value;
|
||||
}
|
||||
}
|
||||
|
||||
_logger.LogDebug(
|
||||
"Mirrored artifact store settings into scanner storage options (driver: {Driver}, bucket: {Bucket}).",
|
||||
target.Driver,
|
||||
target.BucketName);
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,124 @@
|
||||
using Microsoft.Extensions.Logging;
|
||||
using Microsoft.Extensions.Options;
|
||||
using StellaOps.Scanner.Surface.Env;
|
||||
using StellaOps.Scanner.Surface.Secrets;
|
||||
|
||||
namespace StellaOps.Scanner.WebService.Options;
|
||||
|
||||
internal sealed class ScannerSurfaceSecretConfigurator : IConfigureOptions<ScannerWebServiceOptions>
|
||||
{
|
||||
private const string ComponentName = "Scanner.WebService";
|
||||
|
||||
private readonly ISurfaceSecretProvider _secretProvider;
|
||||
private readonly ISurfaceEnvironment _surfaceEnvironment;
|
||||
private readonly ILogger<ScannerSurfaceSecretConfigurator> _logger;
|
||||
|
||||
public ScannerSurfaceSecretConfigurator(
|
||||
ISurfaceSecretProvider secretProvider,
|
||||
ISurfaceEnvironment surfaceEnvironment,
|
||||
ILogger<ScannerSurfaceSecretConfigurator> logger)
|
||||
{
|
||||
_secretProvider = secretProvider ?? throw new ArgumentNullException(nameof(secretProvider));
|
||||
_surfaceEnvironment = surfaceEnvironment ?? throw new ArgumentNullException(nameof(surfaceEnvironment));
|
||||
_logger = logger ?? throw new ArgumentNullException(nameof(logger));
|
||||
}
|
||||
|
||||
public void Configure(ScannerWebServiceOptions options)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(options);
|
||||
|
||||
var tenant = _surfaceEnvironment.Settings.Secrets.Tenant;
|
||||
var request = new SurfaceSecretRequest(
|
||||
Tenant: tenant,
|
||||
Component: ComponentName,
|
||||
SecretType: "cas-access");
|
||||
|
||||
CasAccessSecret? secret = null;
|
||||
try
|
||||
{
|
||||
using var handle = _secretProvider.GetAsync(request).AsTask().GetAwaiter().GetResult();
|
||||
secret = SurfaceSecretParser.ParseCasAccessSecret(handle);
|
||||
}
|
||||
catch (SurfaceSecretNotFoundException)
|
||||
{
|
||||
_logger.LogDebug("Surface secret 'cas-access' not found for {Component}; retaining configured artifact store settings.", ComponentName);
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
_logger.LogWarning(ex, "Failed to resolve surface secret 'cas-access' for {Component}.", ComponentName);
|
||||
}
|
||||
|
||||
if (secret is null)
|
||||
{
|
||||
return;
|
||||
}
|
||||
|
||||
ApplySecret(options.ArtifactStore ??= new ScannerWebServiceOptions.ArtifactStoreOptions(), secret);
|
||||
}
|
||||
|
||||
private void ApplySecret(ScannerWebServiceOptions.ArtifactStoreOptions artifactStore, CasAccessSecret secret)
|
||||
{
|
||||
if (!string.IsNullOrWhiteSpace(secret.Driver))
|
||||
{
|
||||
artifactStore.Driver = secret.Driver;
|
||||
}
|
||||
|
||||
if (!string.IsNullOrWhiteSpace(secret.Endpoint))
|
||||
{
|
||||
artifactStore.Endpoint = secret.Endpoint!;
|
||||
}
|
||||
|
||||
if (secret.AllowInsecureTls is { } insecure)
|
||||
{
|
||||
artifactStore.AllowInsecureTls = insecure;
|
||||
artifactStore.UseTls = !insecure;
|
||||
}
|
||||
|
||||
if (!string.IsNullOrWhiteSpace(secret.Region))
|
||||
{
|
||||
artifactStore.Region = secret.Region;
|
||||
}
|
||||
|
||||
if (!string.IsNullOrWhiteSpace(secret.Bucket))
|
||||
{
|
||||
artifactStore.Bucket = secret.Bucket!;
|
||||
}
|
||||
|
||||
if (!string.IsNullOrWhiteSpace(secret.RootPrefix))
|
||||
{
|
||||
artifactStore.RootPrefix = secret.RootPrefix!;
|
||||
}
|
||||
|
||||
if (!string.IsNullOrWhiteSpace(secret.ApiKeyHeader))
|
||||
{
|
||||
artifactStore.ApiKeyHeader = secret.ApiKeyHeader!;
|
||||
}
|
||||
|
||||
if (!string.IsNullOrWhiteSpace(secret.ApiKey))
|
||||
{
|
||||
artifactStore.ApiKey = secret.ApiKey;
|
||||
}
|
||||
|
||||
if (!string.IsNullOrWhiteSpace(secret.AccessKeyId) && !string.IsNullOrWhiteSpace(secret.SecretAccessKey))
|
||||
{
|
||||
artifactStore.AccessKey = secret.AccessKeyId!;
|
||||
artifactStore.SecretKey = secret.SecretAccessKey!;
|
||||
}
|
||||
|
||||
foreach (var header in secret.Headers)
|
||||
{
|
||||
if (string.IsNullOrWhiteSpace(header.Key) || string.IsNullOrWhiteSpace(header.Value))
|
||||
{
|
||||
continue;
|
||||
}
|
||||
|
||||
artifactStore.Headers[header.Key] = header.Value;
|
||||
}
|
||||
|
||||
_logger.LogInformation(
|
||||
"Surface secret 'cas-access' applied for {Component} (driver: {Driver}, bucket: {Bucket}).",
|
||||
ComponentName,
|
||||
artifactStore.Driver,
|
||||
artifactStore.Bucket);
|
||||
}
|
||||
}
|
||||
@@ -30,12 +30,6 @@ public static class ScannerWebServiceOptionsPostConfigure
|
||||
|
||||
options.ArtifactStore ??= new ScannerWebServiceOptions.ArtifactStoreOptions();
|
||||
var artifactStore = options.ArtifactStore;
|
||||
if (string.IsNullOrWhiteSpace(artifactStore.SecretKey)
|
||||
&& !string.IsNullOrWhiteSpace(artifactStore.SecretKeyFile))
|
||||
{
|
||||
artifactStore.SecretKey = ReadSecretFile(artifactStore.SecretKeyFile!, contentRootPath);
|
||||
}
|
||||
|
||||
options.Signing ??= new ScannerWebServiceOptions.SigningOptions();
|
||||
var signing = options.Signing;
|
||||
if (string.IsNullOrWhiteSpace(signing.KeyPem)
|
||||
|
||||
@@ -97,6 +97,7 @@ builder.Services.AddSurfaceEnvironment(options =>
|
||||
builder.Services.AddSurfaceValidation();
|
||||
builder.Services.AddSurfaceFileCache();
|
||||
builder.Services.AddSurfaceSecrets();
|
||||
builder.Services.AddSingleton<IConfigureOptions<ScannerWebServiceOptions>, ScannerSurfaceSecretConfigurator>();
|
||||
builder.Services.AddSingleton<IConfigureOptions<SurfaceCacheOptions>>(sp =>
|
||||
new SurfaceCacheOptionsConfigurator(sp.GetRequiredService<ISurfaceEnvironment>()));
|
||||
builder.Services.AddSingleton<ISurfacePointerService, SurfacePointerService>();
|
||||
@@ -179,6 +180,7 @@ builder.Services.AddScannerStorage(storageOptions =>
|
||||
storageOptions.ObjectStore.RustFs.BaseUrl = string.Empty;
|
||||
}
|
||||
});
|
||||
builder.Services.AddSingleton<IPostConfigureOptions<ScannerStorageOptions>, ScannerStorageOptionsPostConfigurator>();
|
||||
builder.Services.AddSingleton<RuntimeEventRateLimiter>();
|
||||
builder.Services.AddSingleton<IRuntimeEventIngestionService, RuntimeEventIngestionService>();
|
||||
builder.Services.AddSingleton<IRuntimeAttestationVerifier, RuntimeAttestationVerifier>();
|
||||
|
||||
@@ -6,7 +6,7 @@
|
||||
| SCANNER-SURFACE-02 | DONE (2025-11-05) | Scanner WebService Guild | SURFACE-FS-02 | Publish Surface.FS pointers (CAS URIs, manifests) via scan/report APIs and update attestation metadata.<br>2025-11-05: Surface pointers projected through scan/report endpoints, orchestrator samples + DSSE fixtures refreshed with manifest block, readiness tests updated to use validator stub. | OpenAPI updated; clients regenerated; integration tests validate pointer presence and tenancy. |
|
||||
| SCANNER-ENV-02 | TODO (2025-11-06) | Scanner WebService Guild, Ops Guild | SURFACE-ENV-02 | Wire Surface.Env helpers into WebService hosting (cache roots, feature flags) and document configuration.<br>2025-11-02: Cache root resolution switched to helper; feature flag bindings updated; Helm/Compose updates pending review.<br>2025-11-05 14:55Z: Aligning readiness checks, docs, and Helm/Compose templates with Surface.Env outputs and planning test coverage for configuration fallbacks.<br>2025-11-06 17:05Z: Surface.Env documentation/README refreshed; warning catalogue captured for ops handoff.<br>2025-11-06 07:45Z: Helm values (dev/stage/prod/airgap/mirror) and Compose examples updated with `SCANNER_SURFACE_*` defaults plus rollout warning note in `deploy/README.md`.<br>2025-11-06 07:55Z: Paused; follow-up automation captured under `DEVOPS-OPENSSL-11-001/002` and pending Surface.Env readiness tests. | Service uses helper; env table documented; helm/compose templates updated. |
|
||||
> 2025-11-05 19:18Z: Added configurator to project wiring and unit test ensuring Surface.Env cache root is honoured.
|
||||
| SCANNER-SECRETS-02 | DOING (2025-11-02) | Scanner WebService Guild, Security Guild | SURFACE-SECRETS-02 | Replace ad-hoc secret wiring with Surface.Secrets for report/export operations (registry and CAS tokens).<br>2025-11-02: Export/report flows now depend on Surface.Secrets stub; integration tests in progress. | Secrets fetched through shared provider; unit/integration tests cover rotation + failure cases. |
|
||||
| SCANNER-SECRETS-02 | DOING (2025-11-06) | Scanner WebService Guild, Security Guild | SURFACE-SECRETS-02 | Replace ad-hoc secret wiring with Surface.Secrets for report/export operations (registry and CAS tokens).<br>2025-11-02: Export/report flows now depend on Surface.Secrets stub; integration tests in progress.<br>2025-11-06: Restarting work to eliminate file-based secrets, plumb provider handles through report/export services, and extend failure/rotation tests.<br>2025-11-06 21:40Z: Added configurator + storage post-config to hydrate artifact/CAS credentials from `cas-access` secrets with unit coverage. | Secrets fetched through shared provider; unit/integration tests cover rotation + failure cases. |
|
||||
| SCANNER-EVENTS-16-301 | BLOCKED (2025-10-26) | Scanner WebService Guild | ORCH-SVC-38-101, NOTIFY-SVC-38-001 | Emit orchestrator-compatible envelopes (`scanner.event.*`) and update integration tests to verify Notifier ingestion (no Redis queue coupling). | Tests assert envelope schema + orchestrator publish; Notifier consumer harness passes; docs updated with new event contract. Blocked by .NET 10 preview OpenAPI/Auth dependency drift preventing `dotnet test` completion. |
|
||||
| SCANNER-EVENTS-16-302 | DONE (2025-11-06) | Scanner WebService Guild | SCANNER-EVENTS-16-301 | Extend orchestrator event links (report/policy/attestation) once endpoints are finalised across gateway + console.<br>2025-11-06 22:55Z: Dispatcher now honours configurable API/console base segments, JSON samples/docs refreshed, and `ReportEventDispatcherTests` extended. Tests: `StellaOps.Scanner.WebService.Tests` build until pre-existing `SurfaceCacheOptionsConfiguratorTests` ctor signature drift (tracked separately). | Links section covers UI/API targets; downstream consumers validated; docs/samples updated. |
|
||||
|
||||
|
||||
@@ -0,0 +1,141 @@
|
||||
using Microsoft.Extensions.Logging;
|
||||
using Microsoft.Extensions.Options;
|
||||
using StellaOps.Scanner.Storage;
|
||||
using StellaOps.Scanner.Surface.Env;
|
||||
using StellaOps.Scanner.Surface.Secrets;
|
||||
|
||||
namespace StellaOps.Scanner.Worker.Options;
|
||||
|
||||
internal sealed class ScannerStorageSurfaceSecretConfigurator : IConfigureOptions<ScannerStorageOptions>
|
||||
{
|
||||
private static readonly string ComponentName = "Scanner.Worker";
|
||||
|
||||
private readonly ISurfaceSecretProvider _secretProvider;
|
||||
private readonly ISurfaceEnvironment _surfaceEnvironment;
|
||||
private readonly ILogger<ScannerStorageSurfaceSecretConfigurator> _logger;
|
||||
|
||||
public ScannerStorageSurfaceSecretConfigurator(
|
||||
ISurfaceSecretProvider secretProvider,
|
||||
ISurfaceEnvironment surfaceEnvironment,
|
||||
ILogger<ScannerStorageSurfaceSecretConfigurator> logger)
|
||||
{
|
||||
_secretProvider = secretProvider ?? throw new ArgumentNullException(nameof(secretProvider));
|
||||
_surfaceEnvironment = surfaceEnvironment ?? throw new ArgumentNullException(nameof(surfaceEnvironment));
|
||||
_logger = logger ?? throw new ArgumentNullException(nameof(logger));
|
||||
}
|
||||
|
||||
public void Configure(ScannerStorageOptions options)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(options);
|
||||
|
||||
var tenant = _surfaceEnvironment.Settings.Secrets.Tenant;
|
||||
var request = new SurfaceSecretRequest(
|
||||
Tenant: tenant,
|
||||
Component: ComponentName,
|
||||
SecretType: "cas-access");
|
||||
|
||||
CasAccessSecret? secret = null;
|
||||
try
|
||||
{
|
||||
using var handle = _secretProvider.GetAsync(request).AsTask().GetAwaiter().GetResult();
|
||||
secret = SurfaceSecretParser.ParseCasAccessSecret(handle);
|
||||
}
|
||||
catch (SurfaceSecretNotFoundException)
|
||||
{
|
||||
_logger.LogDebug("Surface secret 'cas-access' not found for {Component}; using configured storage settings.", ComponentName);
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
_logger.LogWarning(ex, "Failed to resolve surface secret 'cas-access' for {Component}.", ComponentName);
|
||||
}
|
||||
|
||||
if (secret is null)
|
||||
{
|
||||
return;
|
||||
}
|
||||
|
||||
ApplySecret(options, secret);
|
||||
}
|
||||
|
||||
private void ApplySecret(ScannerStorageOptions options, CasAccessSecret secret)
|
||||
{
|
||||
var objectStore = options.ObjectStore ??= new ObjectStoreOptions();
|
||||
|
||||
if (!string.IsNullOrWhiteSpace(secret.Driver))
|
||||
{
|
||||
objectStore.Driver = secret.Driver;
|
||||
}
|
||||
|
||||
if (!string.IsNullOrWhiteSpace(secret.Region))
|
||||
{
|
||||
objectStore.Region = secret.Region;
|
||||
}
|
||||
|
||||
if (!string.IsNullOrWhiteSpace(secret.Bucket))
|
||||
{
|
||||
objectStore.BucketName = secret.Bucket;
|
||||
}
|
||||
|
||||
if (!string.IsNullOrWhiteSpace(secret.RootPrefix))
|
||||
{
|
||||
objectStore.RootPrefix = secret.RootPrefix;
|
||||
}
|
||||
|
||||
if (!string.IsNullOrWhiteSpace(secret.Endpoint))
|
||||
{
|
||||
if (objectStore.IsRustFsDriver())
|
||||
{
|
||||
objectStore.RustFs ??= new RustFsOptions();
|
||||
objectStore.RustFs.BaseUrl = secret.Endpoint!;
|
||||
}
|
||||
else
|
||||
{
|
||||
objectStore.ServiceUrl = secret.Endpoint;
|
||||
}
|
||||
}
|
||||
|
||||
if (objectStore.IsRustFsDriver())
|
||||
{
|
||||
objectStore.RustFs ??= new RustFsOptions();
|
||||
|
||||
if (!string.IsNullOrWhiteSpace(secret.ApiKeyHeader))
|
||||
{
|
||||
objectStore.RustFs.ApiKeyHeader = secret.ApiKeyHeader!;
|
||||
}
|
||||
|
||||
if (!string.IsNullOrWhiteSpace(secret.ApiKey))
|
||||
{
|
||||
objectStore.RustFs.ApiKey = secret.ApiKey;
|
||||
}
|
||||
|
||||
if (secret.AllowInsecureTls is { } insecure)
|
||||
{
|
||||
objectStore.RustFs.AllowInsecureTls = insecure;
|
||||
}
|
||||
}
|
||||
|
||||
if (!string.IsNullOrWhiteSpace(secret.AccessKeyId) && !string.IsNullOrWhiteSpace(secret.SecretAccessKey))
|
||||
{
|
||||
objectStore.AccessKeyId = secret.AccessKeyId;
|
||||
objectStore.SecretAccessKey = secret.SecretAccessKey;
|
||||
objectStore.SessionToken = secret.SessionToken;
|
||||
}
|
||||
|
||||
foreach (var kvp in secret.Headers)
|
||||
{
|
||||
if (string.IsNullOrWhiteSpace(kvp.Key) || string.IsNullOrWhiteSpace(kvp.Value))
|
||||
{
|
||||
continue;
|
||||
}
|
||||
|
||||
objectStore.Headers[kvp.Key] = kvp.Value;
|
||||
}
|
||||
|
||||
_logger.LogInformation(
|
||||
"Surface secret 'cas-access' applied for {Component} (driver: {Driver}, bucket: {Bucket}, region: {Region}).",
|
||||
ComponentName,
|
||||
objectStore.Driver,
|
||||
objectStore.BucketName,
|
||||
objectStore.Region);
|
||||
}
|
||||
}
|
||||
@@ -21,6 +21,7 @@ using StellaOps.Scanner.Surface.Env;
|
||||
using StellaOps.Scanner.Surface.FS;
|
||||
using StellaOps.Scanner.Surface.Validation;
|
||||
using StellaOps.Scanner.Worker.Options;
|
||||
using StellaOps.Scanner.Worker.Diagnostics;
|
||||
|
||||
namespace StellaOps.Scanner.Worker.Processing;
|
||||
|
||||
@@ -206,7 +207,7 @@ internal sealed class CompositeScanAnalyzerDispatcher : IScanAnalyzerDispatcher
|
||||
try
|
||||
{
|
||||
var engine = new LanguageAnalyzerEngine(new[] { analyzer });
|
||||
var cacheEntry = await cacheAdapter.GetOrCreateAsync(
|
||||
var cacheEntry = await cacheAdapter.GetOrCreateEntryAsync(
|
||||
_logger,
|
||||
analyzer.Id,
|
||||
workspaceFingerprint,
|
||||
|
||||
@@ -0,0 +1,264 @@
|
||||
using System.Collections.Generic;
|
||||
using System.Collections.Immutable;
|
||||
using System.IO;
|
||||
using System.Security.Cryptography;
|
||||
using System.Text;
|
||||
using System.Text.Json;
|
||||
using Microsoft.Extensions.Logging;
|
||||
using Microsoft.Extensions.Options;
|
||||
using StellaOps.Scanner.Core.Contracts;
|
||||
using StellaOps.Scanner.Surface.FS;
|
||||
using StellaOps.Scanner.Storage;
|
||||
using StellaOps.Scanner.Storage.Catalog;
|
||||
using StellaOps.Scanner.Storage.ObjectStore;
|
||||
using StellaOps.Scanner.Storage.Repositories;
|
||||
using StellaOps.Scanner.Storage.Services;
|
||||
using StellaOps.Scanner.Surface.Env;
|
||||
|
||||
namespace StellaOps.Scanner.Worker.Processing.Surface;
|
||||
|
||||
internal sealed record SurfaceManifestPayload(
|
||||
ArtifactDocumentType ArtifactType,
|
||||
ArtifactDocumentFormat ArtifactFormat,
|
||||
string Kind,
|
||||
string MediaType,
|
||||
ReadOnlyMemory<byte> Content,
|
||||
string? View = null,
|
||||
IReadOnlyDictionary<string, string>? Metadata = null,
|
||||
bool RegisterArtifact = false);
|
||||
|
||||
internal sealed record SurfaceManifestRequest(
|
||||
string ScanId,
|
||||
string ImageDigest,
|
||||
int Attempt,
|
||||
IReadOnlyDictionary<string, string> Metadata,
|
||||
IReadOnlyList<SurfaceManifestPayload> Payloads,
|
||||
string Component,
|
||||
string? Version,
|
||||
string? WorkerInstance);
|
||||
|
||||
internal sealed class SurfaceManifestPublisher
|
||||
{
|
||||
private static readonly JsonSerializerOptions SerializerOptions = new(JsonSerializerDefaults.Web)
|
||||
{
|
||||
WriteIndented = false,
|
||||
DefaultIgnoreCondition = JsonIgnoreCondition.WhenWritingNull
|
||||
};
|
||||
|
||||
private readonly IArtifactObjectStore _objectStore;
|
||||
private readonly ArtifactRepository _artifactRepository;
|
||||
private readonly LinkRepository _linkRepository;
|
||||
private readonly ScannerStorageOptions _storageOptions;
|
||||
private readonly ISurfaceEnvironment _surfaceEnvironment;
|
||||
private readonly TimeProvider _timeProvider;
|
||||
private readonly ILogger<SurfaceManifestPublisher> _logger;
|
||||
|
||||
public SurfaceManifestPublisher(
|
||||
IArtifactObjectStore objectStore,
|
||||
ArtifactRepository artifactRepository,
|
||||
LinkRepository linkRepository,
|
||||
IOptions<ScannerStorageOptions> storageOptions,
|
||||
ISurfaceEnvironment surfaceEnvironment,
|
||||
TimeProvider timeProvider,
|
||||
ILogger<SurfaceManifestPublisher> logger)
|
||||
{
|
||||
_objectStore = objectStore ?? throw new ArgumentNullException(nameof(objectStore));
|
||||
_artifactRepository = artifactRepository ?? throw new ArgumentNullException(nameof(artifactRepository));
|
||||
_linkRepository = linkRepository ?? throw new ArgumentNullException(nameof(linkRepository));
|
||||
_storageOptions = (storageOptions ?? throw new ArgumentNullException(nameof(storageOptions))).Value;
|
||||
_surfaceEnvironment = surfaceEnvironment ?? throw new ArgumentNullException(nameof(surfaceEnvironment));
|
||||
_timeProvider = timeProvider ?? throw new ArgumentNullException(nameof(timeProvider));
|
||||
_logger = logger ?? throw new ArgumentNullException(nameof(logger));
|
||||
}
|
||||
|
||||
public async Task<SurfaceManifestPublishResult> PublishAsync(SurfaceManifestRequest request, CancellationToken cancellationToken)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(request);
|
||||
|
||||
if (request.Payloads.Count == 0)
|
||||
{
|
||||
throw new ArgumentException("At least one payload must be provided.", nameof(request));
|
||||
}
|
||||
|
||||
var tenant = _surfaceEnvironment.Settings.Tenant;
|
||||
var generatedAt = _timeProvider.GetUtcNow();
|
||||
var artifacts = new List<SurfaceManifestArtifact>(request.Payloads.Count);
|
||||
|
||||
foreach (var payload in request.Payloads)
|
||||
{
|
||||
var artifact = await StorePayloadAsync(payload, tenant, cancellationToken).ConfigureAwait(false);
|
||||
artifacts.Add(artifact);
|
||||
}
|
||||
|
||||
var manifestDocument = new SurfaceManifestDocument
|
||||
{
|
||||
Tenant = tenant,
|
||||
ImageDigest = NormalizeDigest(request.ImageDigest),
|
||||
ScanId = request.ScanId,
|
||||
GeneratedAt = generatedAt,
|
||||
Source = new SurfaceManifestSource
|
||||
{
|
||||
Component = request.Component,
|
||||
Version = request.Version,
|
||||
WorkerInstance = request.WorkerInstance,
|
||||
Attempt = request.Attempt
|
||||
},
|
||||
Artifacts = artifacts.ToImmutableArray()
|
||||
};
|
||||
|
||||
var manifestBytes = JsonSerializer.SerializeToUtf8Bytes(manifestDocument, SerializerOptions);
|
||||
var manifestDigest = ComputeDigest(manifestBytes);
|
||||
var manifestKey = ArtifactObjectKeyBuilder.Build(
|
||||
ArtifactDocumentType.SurfaceManifest,
|
||||
ArtifactDocumentFormat.SurfaceManifestJson,
|
||||
manifestDigest,
|
||||
_storageOptions.ObjectStore.RootPrefix);
|
||||
var manifestDescriptor = new ArtifactObjectDescriptor(
|
||||
_storageOptions.ObjectStore.BucketName,
|
||||
manifestKey,
|
||||
Immutable: true,
|
||||
RetainFor: _storageOptions.ObjectStore.ComplianceRetention);
|
||||
|
||||
await using (var stream = new MemoryStream(manifestBytes, writable: false))
|
||||
{
|
||||
await _objectStore.PutAsync(manifestDescriptor, stream, cancellationToken).ConfigureAwait(false);
|
||||
}
|
||||
|
||||
if (_storageOptions.DualWrite.Enabled && !string.IsNullOrWhiteSpace(_storageOptions.DualWrite.MirrorBucket))
|
||||
{
|
||||
await using var mirrorStream = new MemoryStream(manifestBytes, writable: false);
|
||||
var mirrorDescriptor = manifestDescriptor with { Bucket = _storageOptions.DualWrite.MirrorBucket! };
|
||||
await _objectStore.PutAsync(mirrorDescriptor, mirrorStream, cancellationToken).ConfigureAwait(false);
|
||||
}
|
||||
|
||||
var nowUtc = generatedAt.UtcDateTime;
|
||||
var artifactId = CatalogIdFactory.CreateArtifactId(ArtifactDocumentType.SurfaceManifest, manifestDigest);
|
||||
var manifestDocumentRecord = new ArtifactDocument
|
||||
{
|
||||
Id = artifactId,
|
||||
Type = ArtifactDocumentType.SurfaceManifest,
|
||||
Format = ArtifactDocumentFormat.SurfaceManifestJson,
|
||||
MediaType = "application/vnd.stellaops.surface.manifest+json",
|
||||
BytesSha256 = manifestDigest,
|
||||
SizeBytes = manifestBytes.Length,
|
||||
Immutable = true,
|
||||
RefCount = 1,
|
||||
CreatedAtUtc = nowUtc,
|
||||
UpdatedAtUtc = nowUtc,
|
||||
TtlClass = "surface.manifest"
|
||||
};
|
||||
|
||||
await _artifactRepository.UpsertAsync(manifestDocumentRecord, cancellationToken).ConfigureAwait(false);
|
||||
|
||||
var link = new LinkDocument
|
||||
{
|
||||
Id = CatalogIdFactory.CreateLinkId(LinkSourceType.Image, manifestDocument.ImageDigest ?? request.ScanId, artifactId),
|
||||
FromType = LinkSourceType.Image,
|
||||
FromDigest = manifestDocument.ImageDigest ?? request.ScanId,
|
||||
ArtifactId = artifactId,
|
||||
CreatedAtUtc = nowUtc
|
||||
};
|
||||
|
||||
await _linkRepository.UpsertAsync(link, cancellationToken).ConfigureAwait(false);
|
||||
|
||||
var manifestUri = BuildCasUri(_storageOptions.ObjectStore.BucketName, manifestKey);
|
||||
_logger.LogInformation("Published surface manifest {Manifest} for image {ImageDigest}.", artifactId, manifestDocument.ImageDigest);
|
||||
|
||||
return new SurfaceManifestPublishResult(
|
||||
ManifestDigest: manifestDigest,
|
||||
ManifestUri: manifestUri,
|
||||
ArtifactId: artifactId,
|
||||
Document: manifestDocument);
|
||||
}
|
||||
|
||||
private async Task<SurfaceManifestArtifact> StorePayloadAsync(SurfaceManifestPayload payload, string tenant, CancellationToken cancellationToken)
|
||||
{
|
||||
var digest = ComputeDigest(payload.Content.Span);
|
||||
var key = ArtifactObjectKeyBuilder.Build(
|
||||
payload.ArtifactType,
|
||||
payload.ArtifactFormat,
|
||||
digest,
|
||||
_storageOptions.ObjectStore.RootPrefix);
|
||||
|
||||
await using (var stream = new MemoryStream(payload.Content.ToArray(), writable: false))
|
||||
{
|
||||
var descriptor = new ArtifactObjectDescriptor(
|
||||
_storageOptions.ObjectStore.BucketName,
|
||||
key,
|
||||
Immutable: true,
|
||||
RetainFor: _storageOptions.ObjectStore.ComplianceRetention);
|
||||
|
||||
await _objectStore.PutAsync(descriptor, stream, cancellationToken).ConfigureAwait(false);
|
||||
|
||||
if (_storageOptions.DualWrite.Enabled && !string.IsNullOrWhiteSpace(_storageOptions.DualWrite.MirrorBucket))
|
||||
{
|
||||
await using var mirrorStream = new MemoryStream(payload.Content.ToArray(), writable: false);
|
||||
var mirrorDescriptor = descriptor with { Bucket = _storageOptions.DualWrite.MirrorBucket! };
|
||||
await _objectStore.PutAsync(mirrorDescriptor, mirrorStream, cancellationToken).ConfigureAwait(false);
|
||||
}
|
||||
}
|
||||
|
||||
return new SurfaceManifestArtifact
|
||||
{
|
||||
Kind = payload.Kind,
|
||||
Uri = BuildCasUri(_storageOptions.ObjectStore.BucketName, key),
|
||||
Digest = digest,
|
||||
MediaType = payload.MediaType,
|
||||
Format = MapFormat(payload.ArtifactFormat),
|
||||
SizeBytes = payload.Content.Length,
|
||||
View = payload.View,
|
||||
Storage = new SurfaceManifestStorage
|
||||
{
|
||||
Bucket = _storageOptions.ObjectStore.BucketName,
|
||||
ObjectKey = key,
|
||||
SizeBytes = payload.Content.Length,
|
||||
ContentType = payload.MediaType
|
||||
},
|
||||
Metadata = payload.Metadata
|
||||
};
|
||||
}
|
||||
|
||||
private static string BuildCasUri(string bucket, string key)
|
||||
{
|
||||
var normalizedKey = string.IsNullOrWhiteSpace(key) ? string.Empty : key.Trim().TrimStart('/');
|
||||
return $"cas://{bucket}/{normalizedKey}";
|
||||
}
|
||||
|
||||
private static string MapFormat(ArtifactDocumentFormat format)
|
||||
=> format switch
|
||||
{
|
||||
ArtifactDocumentFormat.EntryTraceNdjson => "entrytrace.ndjson",
|
||||
ArtifactDocumentFormat.EntryTraceGraphJson => "entrytrace.graph",
|
||||
ArtifactDocumentFormat.ComponentFragmentJson => "layer.fragments",
|
||||
ArtifactDocumentFormat.SurfaceManifestJson => "surface.manifest",
|
||||
ArtifactDocumentFormat.CycloneDxJson => "cdx-json",
|
||||
ArtifactDocumentFormat.CycloneDxProtobuf => "cdx-protobuf",
|
||||
ArtifactDocumentFormat.SpdxJson => "spdx-json",
|
||||
ArtifactDocumentFormat.BomIndex => "bom-index",
|
||||
ArtifactDocumentFormat.DsseJson => "dsse-json",
|
||||
_ => format.ToString().ToLowerInvariant()
|
||||
};
|
||||
|
||||
private static string ComputeDigest(ReadOnlySpan<byte> content)
|
||||
{
|
||||
Span<byte> hash = stackalloc byte[32];
|
||||
SHA256.HashData(content, hash);
|
||||
return $"sha256:{Convert.ToHexString(hash).ToLowerInvariant()}";
|
||||
}
|
||||
|
||||
private static string ComputeDigest(byte[] content)
|
||||
=> ComputeDigest(content.AsSpan());
|
||||
|
||||
private static string NormalizeDigest(string digest)
|
||||
{
|
||||
if (string.IsNullOrWhiteSpace(digest))
|
||||
{
|
||||
return string.Empty;
|
||||
}
|
||||
|
||||
var trimmed = digest.Trim();
|
||||
return trimmed.Contains(':', StringComparison.Ordinal)
|
||||
? trimmed
|
||||
: $"sha256:{trimmed}";
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,147 @@
|
||||
using System.Collections.Generic;
|
||||
using System.Collections.Immutable;
|
||||
using System.Globalization;
|
||||
using System.Reflection;
|
||||
using System.Text;
|
||||
using System.Text.Json;
|
||||
using Microsoft.Extensions.Logging;
|
||||
using StellaOps.Scanner.Core.Contracts;
|
||||
using StellaOps.Scanner.EntryTrace;
|
||||
using StellaOps.Scanner.EntryTrace.Serialization;
|
||||
using StellaOps.Scanner.Surface.FS;
|
||||
|
||||
namespace StellaOps.Scanner.Worker.Processing.Surface;
|
||||
|
||||
internal sealed class SurfaceManifestStageExecutor : IScanStageExecutor
|
||||
{
|
||||
private static readonly JsonSerializerOptions JsonOptions = new(JsonSerializerDefaults.Web)
|
||||
{
|
||||
WriteIndented = false,
|
||||
DefaultIgnoreCondition = JsonIgnoreCondition.WhenWritingNull
|
||||
};
|
||||
|
||||
private readonly SurfaceManifestPublisher _publisher;
|
||||
private readonly ILogger<SurfaceManifestStageExecutor> _logger;
|
||||
private readonly string _componentVersion;
|
||||
|
||||
public SurfaceManifestStageExecutor(
|
||||
SurfaceManifestPublisher publisher,
|
||||
ILogger<SurfaceManifestStageExecutor> logger)
|
||||
{
|
||||
_publisher = publisher ?? throw new ArgumentNullException(nameof(publisher));
|
||||
_logger = logger ?? throw new ArgumentNullException(nameof(logger));
|
||||
_componentVersion = Assembly.GetExecutingAssembly().GetName().Version?.ToString() ?? "unknown";
|
||||
}
|
||||
|
||||
public string StageName => ScanStageNames.ComposeArtifacts;
|
||||
|
||||
public async ValueTask ExecuteAsync(ScanJobContext context, CancellationToken cancellationToken)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(context);
|
||||
|
||||
var payloads = CollectPayloads(context);
|
||||
if (payloads.Count == 0)
|
||||
{
|
||||
_logger.LogDebug("No surface payloads available for job {JobId}; skipping manifest publish.", context.JobId);
|
||||
return;
|
||||
}
|
||||
|
||||
var request = new SurfaceManifestRequest(
|
||||
ScanId: context.ScanId,
|
||||
ImageDigest: ResolveImageDigest(context),
|
||||
Attempt: context.Lease.Attempt,
|
||||
Metadata: context.Lease.Metadata,
|
||||
Payloads: payloads,
|
||||
Component: "scanner.worker",
|
||||
Version: _componentVersion,
|
||||
WorkerInstance: Environment.MachineName);
|
||||
|
||||
var result = await _publisher.PublishAsync(request, cancellationToken).ConfigureAwait(false);
|
||||
context.Analysis.Set(ScanAnalysisKeys.SurfaceManifest, result);
|
||||
_logger.LogInformation("Surface manifest stored for job {JobId} with digest {Digest}.", context.JobId, result.ManifestDigest);
|
||||
}
|
||||
|
||||
private List<SurfaceManifestPayload> CollectPayloads(ScanJobContext context)
|
||||
{
|
||||
var payloads = new List<SurfaceManifestPayload>();
|
||||
|
||||
if (context.Analysis.TryGet<EntryTraceGraph>(ScanAnalysisKeys.EntryTraceGraph, out var graph) && graph is not null)
|
||||
{
|
||||
var graphJson = EntryTraceGraphSerializer.Serialize(graph);
|
||||
payloads.Add(new SurfaceManifestPayload(
|
||||
ArtifactDocumentType.SurfaceEntryTrace,
|
||||
ArtifactDocumentFormat.EntryTraceGraphJson,
|
||||
Kind: "entrytrace.graph",
|
||||
MediaType: "application/json",
|
||||
Content: Encoding.UTF8.GetBytes(graphJson),
|
||||
Metadata: new Dictionary<string, string>
|
||||
{
|
||||
["artifact"] = "entrytrace.graph",
|
||||
["nodes"] = graph.Nodes.Length.ToString(CultureInfoInvariant),
|
||||
["edges"] = graph.Edges.Length.ToString(CultureInfoInvariant)
|
||||
}));
|
||||
}
|
||||
|
||||
if (context.Analysis.TryGet(ScanAnalysisKeys.EntryTraceNdjson, out ImmutableArray<string> ndjson) && !ndjson.IsDefaultOrEmpty)
|
||||
{
|
||||
var builder = new StringBuilder();
|
||||
for (var i = 0; i < ndjson.Length; i++)
|
||||
{
|
||||
builder.Append(ndjson[i]);
|
||||
if (!ndjson[i].EndsWith('\n'))
|
||||
{
|
||||
builder.Append('\n');
|
||||
}
|
||||
}
|
||||
|
||||
payloads.Add(new SurfaceManifestPayload(
|
||||
ArtifactDocumentType.SurfaceEntryTrace,
|
||||
ArtifactDocumentFormat.EntryTraceNdjson,
|
||||
Kind: "entrytrace.ndjson",
|
||||
MediaType: "application/x-ndjson",
|
||||
Content: Encoding.UTF8.GetBytes(builder.ToString())));
|
||||
}
|
||||
|
||||
var fragments = context.Analysis.GetLayerFragments();
|
||||
if (!fragments.IsDefaultOrEmpty && fragments.Length > 0)
|
||||
{
|
||||
var fragmentsJson = JsonSerializer.Serialize(fragments, JsonOptions);
|
||||
payloads.Add(new SurfaceManifestPayload(
|
||||
ArtifactDocumentType.SurfaceLayerFragment,
|
||||
ArtifactDocumentFormat.ComponentFragmentJson,
|
||||
Kind: "layer.fragments",
|
||||
MediaType: "application/json",
|
||||
Content: Encoding.UTF8.GetBytes(fragmentsJson),
|
||||
View: "inventory"));
|
||||
}
|
||||
|
||||
return payloads;
|
||||
}
|
||||
|
||||
private static string ResolveImageDigest(ScanJobContext context)
|
||||
{
|
||||
static bool TryGet(IReadOnlyDictionary<string, string> metadata, string key, out string value)
|
||||
{
|
||||
if (metadata.TryGetValue(key, out var found) && !string.IsNullOrWhiteSpace(found))
|
||||
{
|
||||
value = found.Trim();
|
||||
return true;
|
||||
}
|
||||
|
||||
value = string.Empty;
|
||||
return false;
|
||||
}
|
||||
|
||||
var metadata = context.Lease.Metadata;
|
||||
if (TryGet(metadata, "image.digest", out var digest) ||
|
||||
TryGet(metadata, "imageDigest", out digest) ||
|
||||
TryGet(metadata, "scanner.image.digest", out digest))
|
||||
{
|
||||
return digest;
|
||||
}
|
||||
|
||||
return context.ScanId;
|
||||
}
|
||||
|
||||
private static readonly IFormatProvider CultureInfoInvariant = System.Globalization.CultureInfo.InvariantCulture;
|
||||
}
|
||||
@@ -18,7 +18,9 @@ using StellaOps.Scanner.Worker.Diagnostics;
|
||||
using StellaOps.Scanner.Worker.Hosting;
|
||||
using StellaOps.Scanner.Worker.Options;
|
||||
using StellaOps.Scanner.Worker.Processing;
|
||||
using StellaOps.Scanner.Worker.Processing.Surface;
|
||||
using StellaOps.Scanner.Storage.Extensions;
|
||||
using StellaOps.Scanner.Storage;
|
||||
|
||||
var builder = Host.CreateApplicationBuilder(args);
|
||||
|
||||
@@ -52,6 +54,9 @@ var connectionString = storageSection.GetValue<string>("Mongo:ConnectionString")
|
||||
if (!string.IsNullOrWhiteSpace(connectionString))
|
||||
{
|
||||
builder.Services.AddScannerStorage(storageSection);
|
||||
builder.Services.AddSingleton<IConfigureOptions<ScannerStorageOptions>, ScannerStorageSurfaceSecretConfigurator>();
|
||||
builder.Services.AddSingleton<SurfaceManifestPublisher>();
|
||||
builder.Services.AddSingleton<IScanStageExecutor, SurfaceManifestStageExecutor>();
|
||||
}
|
||||
|
||||
builder.Services.TryAddSingleton<IScanJobSource, NullScanJobSource>();
|
||||
|
||||
@@ -3,7 +3,7 @@
|
||||
| ID | Status | Owner(s) | Depends on | Description | Exit Criteria |
|
||||
|----|--------|----------|------------|-------------|---------------|
|
||||
| SCAN-REPLAY-186-002 | TODO | Scanner Worker Guild | REPLAY-CORE-185-001 | Enforce deterministic analyzer execution when consuming replay input bundles, emit layer Merkle metadata, and author `docs/modules/scanner/deterministic-execution.md` summarising invariants from `docs/replay/DETERMINISTIC_REPLAY.md` Section 4. | Replay mode analyzers pass determinism tests; new doc merged; integration fixtures updated. |
|
||||
| SCANNER-SURFACE-01 | DOING (2025-11-02) | Scanner Worker Guild | SURFACE-FS-02 | Persist Surface.FS manifests after analyzer stages, including layer CAS metadata and EntryTrace fragments.<br>2025-11-02: Draft Surface.FS manifests emitted for sample scans; telemetry counters under review. | Integration tests prove cache entries exist; telemetry counters exported. |
|
||||
| SCANNER-SURFACE-01 | DOING (2025-11-06) | Scanner Worker Guild | SURFACE-FS-02 | Persist Surface.FS manifests after analyzer stages, including layer CAS metadata and EntryTrace fragments.<br>2025-11-02: Draft Surface.FS manifests emitted for sample scans; telemetry counters under review.<br>2025-11-06: Resuming with manifest writer abstraction, rotation metadata, and telemetry counters for Surface.FS persistence. | Integration tests prove cache entries exist; telemetry counters exported. |
|
||||
| SCANNER-ENV-01 | TODO (2025-11-06) | Scanner Worker Guild | SURFACE-ENV-02 | Replace ad-hoc environment reads with `StellaOps.Scanner.Surface.Env` helpers for cache roots and CAS endpoints.<br>2025-11-02: Worker bootstrap now resolves cache roots via helper; warning path documented; smoke tests running.<br>2025-11-05 14:55Z: Extending helper usage into cache/secrets configuration, updating worker validator wiring, and drafting docs/tests for new Surface.Env outputs.<br>2025-11-06 17:05Z: README/design docs updated with warning catalogue; startup logging guidance captured for ops runbooks.<br>2025-11-06 07:45Z: Helm/Compose env profiles (dev/stage/prod/airgap/mirror) now seed `SCANNER_SURFACE_*` defaults to keep worker cache roots aligned with Surface.Env helpers.<br>2025-11-06 07:55Z: Paused; pending automation tracked via `DEVOPS-OPENSSL-11-001/002` and Surface.Env test fixtures. | Worker boots with helper; misconfiguration warnings documented; smoke tests updated. |
|
||||
> 2025-11-05 19:18Z: Bound `SurfaceCacheOptions` root directory to resolved Surface.Env settings and added unit coverage around the configurator.
|
||||
| SCANNER-SECRETS-01 | DOING (2025-11-02) | Scanner Worker Guild, Security Guild | SURFACE-SECRETS-02 | Adopt `StellaOps.Scanner.Surface.Secrets` for registry/CAS credentials during scan execution.<br>2025-11-02: Surface.Secrets provider wired for CAS token retrieval; integration tests added. | Secrets fetched via shared provider; legacy secret code removed; integration tests cover rotation. |
|
||||
| SCANNER-SECRETS-01 | DOING (2025-11-06) | Scanner Worker Guild, Security Guild | SURFACE-SECRETS-02 | Adopt `StellaOps.Scanner.Surface.Secrets` for registry/CAS credentials during scan execution.<br>2025-11-02: Surface.Secrets provider wired for CAS token retrieval; integration tests added.<br>2025-11-06: Continuing to replace legacy registry credential plumbing and extend rotation metrics/fixtures.<br>2025-11-06 21:35Z: Introduced `ScannerStorageSurfaceSecretConfigurator` mapping `cas-access` secrets into storage options plus unit coverage. | Secrets fetched via shared provider; legacy secret code removed; integration tests cover rotation. |
|
||||
|
||||
@@ -4,6 +4,8 @@ using System.Text;
|
||||
using Microsoft.Extensions.Logging;
|
||||
using StellaOps.Scanner.Surface.FS;
|
||||
|
||||
public readonly record struct LanguageAnalyzerSurfaceCacheEntry(LanguageAnalyzerResult Result, bool IsHit);
|
||||
|
||||
public sealed class LanguageAnalyzerSurfaceCache
|
||||
{
|
||||
private const string CacheNamespace = "scanner/lang/analyzers";
|
||||
@@ -24,6 +26,17 @@ public sealed class LanguageAnalyzerSurfaceCache
|
||||
string fingerprint,
|
||||
Func<CancellationToken, ValueTask<LanguageAnalyzerResult>> factory,
|
||||
CancellationToken cancellationToken)
|
||||
{
|
||||
var entry = await GetOrCreateEntryAsync(logger, analyzerId, fingerprint, factory, cancellationToken).ConfigureAwait(false);
|
||||
return entry.Result;
|
||||
}
|
||||
|
||||
public async ValueTask<LanguageAnalyzerSurfaceCacheEntry> GetOrCreateEntryAsync(
|
||||
ILogger logger,
|
||||
string analyzerId,
|
||||
string fingerprint,
|
||||
Func<CancellationToken, ValueTask<LanguageAnalyzerResult>> factory,
|
||||
CancellationToken cancellationToken)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(logger);
|
||||
ArgumentNullException.ThrowIfNull(factory);
|
||||
@@ -62,7 +75,7 @@ public sealed class LanguageAnalyzerSurfaceCache
|
||||
fingerprint);
|
||||
|
||||
result = await factory(cancellationToken).ConfigureAwait(false);
|
||||
return result;
|
||||
return new LanguageAnalyzerSurfaceCacheEntry(result, false);
|
||||
}
|
||||
|
||||
if (cacheHit)
|
||||
@@ -82,7 +95,7 @@ public sealed class LanguageAnalyzerSurfaceCache
|
||||
fingerprint);
|
||||
}
|
||||
|
||||
return result;
|
||||
return new LanguageAnalyzerSurfaceCacheEntry(result, cacheHit);
|
||||
}
|
||||
|
||||
private static ReadOnlyMemory<byte> Serialize(LanguageAnalyzerResult result)
|
||||
|
||||
@@ -15,4 +15,6 @@ public static class ScanAnalysisKeys
|
||||
public const string EntryTraceGraph = "analysis.entrytrace.graph";
|
||||
|
||||
public const string EntryTraceNdjson = "analysis.entrytrace.ndjson";
|
||||
|
||||
public const string SurfaceManifest = "analysis.surface.manifest";
|
||||
}
|
||||
|
||||
@@ -9,6 +9,9 @@ public enum ArtifactDocumentType
|
||||
Diff,
|
||||
Index,
|
||||
Attestation,
|
||||
SurfaceManifest,
|
||||
SurfaceEntryTrace,
|
||||
SurfaceLayerFragment,
|
||||
}
|
||||
|
||||
public enum ArtifactDocumentFormat
|
||||
@@ -18,6 +21,10 @@ public enum ArtifactDocumentFormat
|
||||
SpdxJson,
|
||||
BomIndex,
|
||||
DsseJson,
|
||||
SurfaceManifestJson,
|
||||
EntryTraceNdjson,
|
||||
EntryTraceGraphJson,
|
||||
ComponentFragmentJson,
|
||||
}
|
||||
|
||||
[BsonIgnoreExtraElements]
|
||||
|
||||
@@ -2,6 +2,7 @@ using System;
|
||||
using System.Net.Http;
|
||||
using Amazon;
|
||||
using Amazon.S3;
|
||||
using Amazon.Runtime;
|
||||
using Microsoft.Extensions.Configuration;
|
||||
using Microsoft.Extensions.DependencyInjection;
|
||||
using Microsoft.Extensions.DependencyInjection.Extensions;
|
||||
@@ -159,6 +160,14 @@ public static class ServiceCollectionExtensions
|
||||
config.ServiceURL = options.ServiceUrl;
|
||||
}
|
||||
|
||||
if (!string.IsNullOrWhiteSpace(options.AccessKeyId) && !string.IsNullOrWhiteSpace(options.SecretAccessKey))
|
||||
{
|
||||
AWSCredentials credentials = string.IsNullOrWhiteSpace(options.SessionToken)
|
||||
? new BasicAWSCredentials(options.AccessKeyId, options.SecretAccessKey)
|
||||
: new SessionAWSCredentials(options.AccessKeyId, options.SecretAccessKey, options.SessionToken);
|
||||
return new AmazonS3Client(credentials, config);
|
||||
}
|
||||
|
||||
return new AmazonS3Client(config);
|
||||
}
|
||||
|
||||
|
||||
@@ -33,6 +33,9 @@ public static class ArtifactObjectKeyBuilder
|
||||
ArtifactDocumentType.ImageBom => ScannerStorageDefaults.ObjectPrefixes.Images,
|
||||
ArtifactDocumentType.Index => ScannerStorageDefaults.ObjectPrefixes.Indexes,
|
||||
ArtifactDocumentType.Attestation => ScannerStorageDefaults.ObjectPrefixes.Attestations,
|
||||
ArtifactDocumentType.SurfaceManifest => ScannerStorageDefaults.ObjectPrefixes.SurfaceManifests,
|
||||
ArtifactDocumentType.SurfaceEntryTrace => ScannerStorageDefaults.ObjectPrefixes.SurfaceEntryTrace,
|
||||
ArtifactDocumentType.SurfaceLayerFragment => ScannerStorageDefaults.ObjectPrefixes.SurfaceLayerFragments,
|
||||
ArtifactDocumentType.Diff => "diffs",
|
||||
_ => ScannerStorageDefaults.ObjectPrefixes.Images,
|
||||
};
|
||||
@@ -44,6 +47,10 @@ public static class ArtifactObjectKeyBuilder
|
||||
ArtifactDocumentFormat.SpdxJson => "sbom.spdx.json",
|
||||
ArtifactDocumentFormat.BomIndex => "bom-index.bin",
|
||||
ArtifactDocumentFormat.DsseJson => "artifact.dsse.json",
|
||||
ArtifactDocumentFormat.SurfaceManifestJson => "surface.manifest.json",
|
||||
ArtifactDocumentFormat.EntryTraceNdjson => "entrytrace.ndjson",
|
||||
ArtifactDocumentFormat.EntryTraceGraphJson => "entrytrace.graph.json",
|
||||
ArtifactDocumentFormat.ComponentFragmentJson => "layer-fragments.json",
|
||||
_ => "artifact.bin",
|
||||
};
|
||||
|
||||
|
||||
@@ -32,5 +32,8 @@ public static class ScannerStorageDefaults
|
||||
public const string Images = "images";
|
||||
public const string Indexes = "indexes";
|
||||
public const string Attestations = "attest";
|
||||
public const string SurfaceManifests = "surface/manifests";
|
||||
public const string SurfaceEntryTrace = "surface/payloads/entrytrace";
|
||||
public const string SurfaceLayerFragments = "surface/payloads/layer-fragments";
|
||||
}
|
||||
}
|
||||
|
||||
@@ -102,6 +102,15 @@ public sealed class ObjectStoreOptions
|
||||
public TimeSpan? ComplianceRetention { get; set; }
|
||||
= TimeSpan.FromDays(90);
|
||||
|
||||
public string? AccessKeyId { get; set; }
|
||||
= null;
|
||||
|
||||
public string? SecretAccessKey { get; set; }
|
||||
= null;
|
||||
|
||||
public string? SessionToken { get; set; }
|
||||
= null;
|
||||
|
||||
public IDictionary<string, string> Headers { get; } = new Dictionary<string, string>(StringComparer.OrdinalIgnoreCase);
|
||||
|
||||
public RustFsOptions RustFs { get; set; } = new();
|
||||
|
||||
@@ -0,0 +1,142 @@
|
||||
using System;
|
||||
using System.Collections.Generic;
|
||||
using System.Collections.Immutable;
|
||||
using System.Text.Json.Serialization;
|
||||
|
||||
namespace StellaOps.Scanner.Surface.FS;
|
||||
|
||||
/// <summary>
|
||||
/// Canonical manifest describing surface artefacts produced for a scan.
|
||||
/// </summary>
|
||||
public sealed record SurfaceManifestDocument
|
||||
{
|
||||
public const string DefaultSchema = "stellaops.surface.manifest@1";
|
||||
|
||||
[JsonPropertyName("schema")]
|
||||
public string Schema { get; init; } = DefaultSchema;
|
||||
|
||||
[JsonPropertyName("tenant")]
|
||||
public string Tenant { get; init; } = string.Empty;
|
||||
|
||||
[JsonPropertyName("imageDigest")]
|
||||
[JsonIgnore(Condition = JsonIgnoreCondition.WhenWritingNull)]
|
||||
public string? ImageDigest { get; init; }
|
||||
= null;
|
||||
|
||||
[JsonPropertyName("scanId")]
|
||||
[JsonIgnore(Condition = JsonIgnoreCondition.WhenWritingNull)]
|
||||
public string? ScanId { get; init; }
|
||||
= null;
|
||||
|
||||
[JsonPropertyName("generatedAt")]
|
||||
public DateTimeOffset GeneratedAt { get; init; }
|
||||
= DateTimeOffset.UtcNow;
|
||||
|
||||
[JsonPropertyName("source")]
|
||||
[JsonIgnore(Condition = JsonIgnoreCondition.WhenWritingNull)]
|
||||
public SurfaceManifestSource? Source { get; init; }
|
||||
= null;
|
||||
|
||||
[JsonPropertyName("artifacts")]
|
||||
public IReadOnlyList<SurfaceManifestArtifact> Artifacts { get; init; }
|
||||
= ImmutableArray<SurfaceManifestArtifact>.Empty;
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Identifies the producer of the manifest.
|
||||
/// </summary>
|
||||
public sealed record SurfaceManifestSource
|
||||
{
|
||||
[JsonPropertyName("component")]
|
||||
public string Component { get; init; } = "scanner.worker";
|
||||
|
||||
[JsonPropertyName("version")]
|
||||
[JsonIgnore(Condition = JsonIgnoreCondition.WhenWritingNull)]
|
||||
public string? Version { get; init; }
|
||||
= null;
|
||||
|
||||
[JsonPropertyName("workerInstance")]
|
||||
[JsonIgnore(Condition = JsonIgnoreCondition.WhenWritingNull)]
|
||||
public string? WorkerInstance { get; init; }
|
||||
= null;
|
||||
|
||||
[JsonPropertyName("attempt")]
|
||||
[JsonIgnore(Condition = JsonIgnoreCondition.WhenWritingNull)]
|
||||
public int? Attempt { get; init; }
|
||||
= null;
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Describes a surface artefact referenced by the manifest.
|
||||
/// </summary>
|
||||
public sealed record SurfaceManifestArtifact
|
||||
{
|
||||
[JsonPropertyName("kind")]
|
||||
public string Kind { get; init; } = string.Empty;
|
||||
|
||||
[JsonPropertyName("uri")]
|
||||
public string Uri { get; init; } = string.Empty;
|
||||
|
||||
[JsonPropertyName("digest")]
|
||||
public string Digest { get; init; } = string.Empty;
|
||||
|
||||
[JsonPropertyName("mediaType")]
|
||||
public string MediaType { get; init; } = string.Empty;
|
||||
|
||||
[JsonPropertyName("format")]
|
||||
public string Format { get; init; } = string.Empty;
|
||||
|
||||
[JsonPropertyName("sizeBytes")]
|
||||
public long SizeBytes { get; init; }
|
||||
= 0;
|
||||
|
||||
[JsonPropertyName("view")]
|
||||
[JsonIgnore(Condition = JsonIgnoreCondition.WhenWritingNull)]
|
||||
public string? View { get; init; }
|
||||
= null;
|
||||
|
||||
[JsonPropertyName("storage")]
|
||||
[JsonIgnore(Condition = JsonIgnoreCondition.WhenWritingNull)]
|
||||
public SurfaceManifestStorage? Storage { get; init; }
|
||||
= null;
|
||||
|
||||
[JsonPropertyName("metadata")]
|
||||
[JsonIgnore(Condition = JsonIgnoreCondition.WhenWritingNull)]
|
||||
public IReadOnlyDictionary<string, string>? Metadata { get; init; }
|
||||
= null;
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Storage descriptor for an artefact.
|
||||
/// </summary>
|
||||
public sealed record SurfaceManifestStorage
|
||||
{
|
||||
[JsonPropertyName("bucket")]
|
||||
[JsonIgnore(Condition = JsonIgnoreCondition.WhenWritingNull)]
|
||||
public string? Bucket { get; init; }
|
||||
= null;
|
||||
|
||||
[JsonPropertyName("objectKey")]
|
||||
[JsonIgnore(Condition = JsonIgnoreCondition.WhenWritingNull)]
|
||||
public string? ObjectKey { get; init; }
|
||||
= null;
|
||||
|
||||
[JsonPropertyName("sizeBytes")]
|
||||
[JsonIgnore(Condition = JsonIgnoreCondition.WhenWritingNull)]
|
||||
public long? SizeBytes { get; init; }
|
||||
= null;
|
||||
|
||||
[JsonPropertyName("contentType")]
|
||||
[JsonIgnore(Condition = JsonIgnoreCondition.WhenWritingNull)]
|
||||
public string? ContentType { get; init; }
|
||||
= null;
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Result from publishing a surface manifest.
|
||||
/// </summary>
|
||||
public sealed record SurfaceManifestPublishResult(
|
||||
string ManifestDigest,
|
||||
string ManifestUri,
|
||||
string ArtifactId,
|
||||
SurfaceManifestDocument Document);
|
||||
@@ -0,0 +1,207 @@
|
||||
using System.Collections.Generic;
|
||||
using System.Collections.ObjectModel;
|
||||
using System.Text;
|
||||
using System.Text.Json;
|
||||
|
||||
namespace StellaOps.Scanner.Surface.Secrets;
|
||||
|
||||
public sealed record CasAccessSecret(
|
||||
string Driver,
|
||||
string? Endpoint,
|
||||
string? Region,
|
||||
string? Bucket,
|
||||
string? RootPrefix,
|
||||
string? ApiKey,
|
||||
string? ApiKeyHeader,
|
||||
IReadOnlyDictionary<string, string> Headers,
|
||||
string? AccessKeyId,
|
||||
string? SecretAccessKey,
|
||||
string? SessionToken,
|
||||
bool? AllowInsecureTls);
|
||||
|
||||
public static class SurfaceSecretParser
|
||||
{
|
||||
public static CasAccessSecret ParseCasAccessSecret(SurfaceSecretHandle handle)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(handle);
|
||||
var payload = handle.AsBytes();
|
||||
if (payload.IsEmpty)
|
||||
{
|
||||
throw new InvalidOperationException("Surface secret payload is empty.");
|
||||
}
|
||||
|
||||
var jsonText = DecodeUtf8(payload);
|
||||
using var document = JsonDocument.Parse(jsonText);
|
||||
var root = document.RootElement;
|
||||
|
||||
string driver = GetString(root, "driver") ?? GetMetadataValue(handle.Metadata, "driver") ?? "s3";
|
||||
string? endpoint = GetString(root, "endpoint") ?? GetMetadataValue(handle.Metadata, "endpoint");
|
||||
string? region = GetString(root, "region") ?? GetMetadataValue(handle.Metadata, "region");
|
||||
string? bucket = GetString(root, "bucket") ?? GetMetadataValue(handle.Metadata, "bucket");
|
||||
string? rootPrefix = GetString(root, "rootPrefix") ?? GetMetadataValue(handle.Metadata, "rootPrefix");
|
||||
string? apiKey = GetString(root, "apiKey") ?? GetMetadataValue(handle.Metadata, "apiKey");
|
||||
string? apiKeyHeader = GetString(root, "apiKeyHeader") ?? GetMetadataValue(handle.Metadata, "apiKeyHeader");
|
||||
string? accessKeyId = GetString(root, "accessKeyId") ?? GetMetadataValue(handle.Metadata, "accessKeyId");
|
||||
string? secretAccessKey = GetString(root, "secretAccessKey") ?? GetMetadataValue(handle.Metadata, "secretAccessKey");
|
||||
string? sessionToken = GetString(root, "sessionToken") ?? GetMetadataValue(handle.Metadata, "sessionToken");
|
||||
bool? allowInsecureTls = GetBoolean(root, "allowInsecureTls");
|
||||
|
||||
var headers = new Dictionary<string, string>(StringComparer.OrdinalIgnoreCase);
|
||||
PopulateHeaders(root, headers);
|
||||
PopulateMetadataHeaders(handle.Metadata, headers);
|
||||
|
||||
return new CasAccessSecret(
|
||||
driver.Trim(),
|
||||
endpoint?.Trim(),
|
||||
region?.Trim(),
|
||||
bucket?.Trim(),
|
||||
rootPrefix?.Trim(),
|
||||
apiKey?.Trim(),
|
||||
apiKeyHeader?.Trim(),
|
||||
new ReadOnlyDictionary<string, string>(headers),
|
||||
accessKeyId?.Trim(),
|
||||
secretAccessKey?.Trim(),
|
||||
sessionToken?.Trim(),
|
||||
allowInsecureTls);
|
||||
}
|
||||
|
||||
private static string DecodeUtf8(ReadOnlyMemory<byte> payload)
|
||||
{
|
||||
if (payload.IsEmpty)
|
||||
{
|
||||
return string.Empty;
|
||||
}
|
||||
|
||||
try
|
||||
{
|
||||
return Encoding.UTF8.GetString(payload.Span);
|
||||
}
|
||||
catch (DecoderFallbackException ex)
|
||||
{
|
||||
throw new InvalidOperationException("Surface secret payload is not valid UTF-8 JSON.", ex);
|
||||
}
|
||||
}
|
||||
|
||||
private static string? GetString(JsonElement element, string propertyName)
|
||||
{
|
||||
if (TryGetPropertyIgnoreCase(element, propertyName, out var property))
|
||||
{
|
||||
return property.ValueKind switch
|
||||
{
|
||||
JsonValueKind.String => property.GetString(),
|
||||
JsonValueKind.Number => property.GetRawText(),
|
||||
JsonValueKind.True => bool.TrueString,
|
||||
JsonValueKind.False => bool.FalseString,
|
||||
_ => null
|
||||
};
|
||||
}
|
||||
|
||||
return null;
|
||||
}
|
||||
|
||||
private static bool? GetBoolean(JsonElement element, string propertyName)
|
||||
{
|
||||
if (TryGetPropertyIgnoreCase(element, propertyName, out var property))
|
||||
{
|
||||
return property.ValueKind switch
|
||||
{
|
||||
JsonValueKind.True => true,
|
||||
JsonValueKind.False => false,
|
||||
JsonValueKind.String when bool.TryParse(property.GetString(), out var parsed) => parsed,
|
||||
_ => null
|
||||
};
|
||||
}
|
||||
|
||||
return null;
|
||||
}
|
||||
|
||||
private static bool TryGetPropertyIgnoreCase(JsonElement element, string propertyName, out JsonElement property)
|
||||
{
|
||||
if (element.ValueKind != JsonValueKind.Object)
|
||||
{
|
||||
property = default;
|
||||
return false;
|
||||
}
|
||||
|
||||
if (element.TryGetProperty(propertyName, out property))
|
||||
{
|
||||
return true;
|
||||
}
|
||||
|
||||
foreach (var candidate in element.EnumerateObject())
|
||||
{
|
||||
if (string.Equals(candidate.Name, propertyName, StringComparison.OrdinalIgnoreCase))
|
||||
{
|
||||
property = candidate.Value;
|
||||
return true;
|
||||
}
|
||||
}
|
||||
|
||||
property = default;
|
||||
return false;
|
||||
}
|
||||
|
||||
private static void PopulateHeaders(JsonElement element, IDictionary<string, string> headers)
|
||||
{
|
||||
if (!TryGetPropertyIgnoreCase(element, "headers", out var headersElement))
|
||||
{
|
||||
return;
|
||||
}
|
||||
|
||||
if (headersElement.ValueKind != JsonValueKind.Object)
|
||||
{
|
||||
return;
|
||||
}
|
||||
|
||||
foreach (var property in headersElement.EnumerateObject())
|
||||
{
|
||||
var value = property.Value.ValueKind switch
|
||||
{
|
||||
JsonValueKind.String => property.Value.GetString(),
|
||||
JsonValueKind.Number => property.Value.GetRawText(),
|
||||
JsonValueKind.True => bool.TrueString,
|
||||
JsonValueKind.False => bool.FalseString,
|
||||
_ => null
|
||||
};
|
||||
|
||||
if (string.IsNullOrWhiteSpace(value))
|
||||
{
|
||||
continue;
|
||||
}
|
||||
|
||||
headers[property.Name] = value.Trim();
|
||||
}
|
||||
}
|
||||
|
||||
private static void PopulateMetadataHeaders(IReadOnlyDictionary<string, string> metadata, IDictionary<string, string> headers)
|
||||
{
|
||||
foreach (var (key, value) in metadata)
|
||||
{
|
||||
if (!key.StartsWith("header:", StringComparison.OrdinalIgnoreCase))
|
||||
{
|
||||
continue;
|
||||
}
|
||||
|
||||
var headerName = key["header:".Length..];
|
||||
if (string.IsNullOrWhiteSpace(headerName) || string.IsNullOrWhiteSpace(value))
|
||||
{
|
||||
continue;
|
||||
}
|
||||
|
||||
headers[headerName] = value;
|
||||
}
|
||||
}
|
||||
|
||||
private static string? GetMetadataValue(IReadOnlyDictionary<string, string> metadata, string key)
|
||||
{
|
||||
foreach (var (metadataKey, metadataValue) in metadata)
|
||||
{
|
||||
if (string.Equals(metadataKey, key, StringComparison.OrdinalIgnoreCase))
|
||||
{
|
||||
return metadataValue;
|
||||
}
|
||||
}
|
||||
|
||||
return null;
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,65 @@
|
||||
using System.Collections.Generic;
|
||||
using System.Text;
|
||||
using StellaOps.Scanner.Surface.Secrets;
|
||||
using Xunit;
|
||||
|
||||
namespace StellaOps.Scanner.Surface.Secrets.Tests;
|
||||
|
||||
public sealed class CasAccessSecretParserTests
|
||||
{
|
||||
[Fact]
|
||||
public void ParseCasAccessSecret_WithRustFsPayload_ReturnsExpectedValues()
|
||||
{
|
||||
const string json = """
|
||||
{
|
||||
"driver": "rustfs",
|
||||
"endpoint": "https://surface.test.local",
|
||||
"region": "us-gov-west-1",
|
||||
"bucket": "stellaops-surface",
|
||||
"rootPrefix": "scanner",
|
||||
"apiKey": "secret-api-key",
|
||||
"apiKeyHeader": "X-Api-Key",
|
||||
"allowInsecureTls": true,
|
||||
"headers": {
|
||||
"X-Surface-Tenant": "tenant-a"
|
||||
}
|
||||
}
|
||||
""";
|
||||
|
||||
using var handle = SurfaceSecretHandle.FromBytes(Encoding.UTF8.GetBytes(json));
|
||||
var secret = SurfaceSecretParser.ParseCasAccessSecret(handle);
|
||||
|
||||
Assert.Equal("rustfs", secret.Driver);
|
||||
Assert.Equal("https://surface.test.local", secret.Endpoint);
|
||||
Assert.Equal("us-gov-west-1", secret.Region);
|
||||
Assert.Equal("stellaops-surface", secret.Bucket);
|
||||
Assert.Equal("scanner", secret.RootPrefix);
|
||||
Assert.Equal("secret-api-key", secret.ApiKey);
|
||||
Assert.Equal("X-Api-Key", secret.ApiKeyHeader);
|
||||
Assert.True(secret.AllowInsecureTls);
|
||||
Assert.Single(secret.Headers);
|
||||
Assert.Equal("tenant-a", secret.Headers["X-Surface-Tenant"]);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void ParseCasAccessSecret_UsesMetadataFallback_WhenFieldsMissing()
|
||||
{
|
||||
const string json = @"{ ""driver"": ""s3"" }";
|
||||
var metadata = new Dictionary<string, string>(StringComparer.OrdinalIgnoreCase)
|
||||
{
|
||||
["endpoint"] = "https://s3.test.local",
|
||||
["accessKeyId"] = "AKIA123",
|
||||
["secretAccessKey"] = "s3-secret",
|
||||
["header:X-Custom"] = "value"
|
||||
};
|
||||
|
||||
using var handle = SurfaceSecretHandle.FromBytes(Encoding.UTF8.GetBytes(json), metadata);
|
||||
var secret = SurfaceSecretParser.ParseCasAccessSecret(handle);
|
||||
|
||||
Assert.Equal("s3", secret.Driver);
|
||||
Assert.Equal("https://s3.test.local", secret.Endpoint);
|
||||
Assert.Equal("AKIA123", secret.AccessKeyId);
|
||||
Assert.Equal("s3-secret", secret.SecretAccessKey);
|
||||
Assert.Equal("value", secret.Headers["X-Custom"]);
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,142 @@
|
||||
using System;
|
||||
using System.Collections.Generic;
|
||||
using System.IO;
|
||||
using System.Text;
|
||||
using System.Threading;
|
||||
using Microsoft.Extensions.Logging.Abstractions;
|
||||
using Microsoft.Extensions.Options;
|
||||
using StellaOps.Scanner.Surface.Env;
|
||||
using StellaOps.Scanner.Surface.Secrets;
|
||||
using StellaOps.Scanner.WebService.Options;
|
||||
using StellaOps.Scanner.Storage;
|
||||
using Xunit;
|
||||
|
||||
namespace StellaOps.Scanner.WebService.Tests;
|
||||
|
||||
public sealed class ScannerSurfaceSecretConfiguratorTests
|
||||
{
|
||||
[Fact]
|
||||
public void Configure_AppliesCasAccessSecretToArtifactStore()
|
||||
{
|
||||
const string json = """
|
||||
{
|
||||
"driver": "rustfs",
|
||||
"endpoint": "https://surface.api",
|
||||
"bucket": "surface-artifacts",
|
||||
"apiKey": "rust-key",
|
||||
"apiKeyHeader": "X-Surface-Api-Key",
|
||||
"region": "ap-southeast-2"
|
||||
}
|
||||
""";
|
||||
|
||||
using var handle = SurfaceSecretHandle.FromBytes(Encoding.UTF8.GetBytes(json));
|
||||
var secretProvider = new StubSecretProvider(handle);
|
||||
var environment = new StubSurfaceEnvironment();
|
||||
var options = new ScannerWebServiceOptions();
|
||||
|
||||
var configurator = new ScannerSurfaceSecretConfigurator(
|
||||
secretProvider,
|
||||
environment,
|
||||
NullLogger<ScannerSurfaceSecretConfigurator>.Instance);
|
||||
|
||||
configurator.Configure(options);
|
||||
|
||||
Assert.Equal("rustfs", options.ArtifactStore.Driver);
|
||||
Assert.Equal("https://surface.api", options.ArtifactStore.Endpoint);
|
||||
Assert.Equal("surface-artifacts", options.ArtifactStore.Bucket);
|
||||
Assert.Equal("rust-key", options.ArtifactStore.ApiKey);
|
||||
Assert.Equal("X-Surface-Api-Key", options.ArtifactStore.ApiKeyHeader);
|
||||
Assert.Equal("ap-southeast-2", options.ArtifactStore.Region);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void PostConfigure_SynchronizesArtifactStoreToScannerStorageOptions()
|
||||
{
|
||||
var webOptions = Microsoft.Extensions.Options.Options.Create(new ScannerWebServiceOptions
|
||||
{
|
||||
ArtifactStore = new ScannerWebServiceOptions.ArtifactStoreOptions
|
||||
{
|
||||
Driver = "rustfs",
|
||||
Endpoint = "https://surface.sync",
|
||||
ApiKey = "sync-key",
|
||||
ApiKeyHeader = "X-Sync",
|
||||
Bucket = "sync-bucket",
|
||||
Region = "us-west-2",
|
||||
RootPrefix = "sync"
|
||||
}
|
||||
});
|
||||
|
||||
var configurator = new ScannerStorageOptionsPostConfigurator(
|
||||
new OptionsMonitorStub<ScannerWebServiceOptions>(webOptions),
|
||||
NullLogger<ScannerStorageOptionsPostConfigurator>.Instance);
|
||||
|
||||
var storageOptions = new ScannerStorageOptions();
|
||||
configurator.PostConfigure(Microsoft.Extensions.Options.Options.DefaultName, storageOptions);
|
||||
|
||||
Assert.Equal("rustfs", storageOptions.ObjectStore.Driver);
|
||||
Assert.Equal("https://surface.sync", storageOptions.ObjectStore.RustFs.BaseUrl);
|
||||
Assert.Equal("sync-bucket", storageOptions.ObjectStore.BucketName);
|
||||
Assert.Equal("sync", storageOptions.ObjectStore.RootPrefix);
|
||||
Assert.Equal("us-west-2", storageOptions.ObjectStore.Region);
|
||||
Assert.Equal("sync-key", storageOptions.ObjectStore.RustFs.ApiKey);
|
||||
Assert.Equal("X-Sync", storageOptions.ObjectStore.RustFs.ApiKeyHeader);
|
||||
}
|
||||
|
||||
private sealed class StubSecretProvider : ISurfaceSecretProvider
|
||||
{
|
||||
private readonly SurfaceSecretHandle _handle;
|
||||
|
||||
public StubSecretProvider(SurfaceSecretHandle handle)
|
||||
{
|
||||
_handle = handle;
|
||||
}
|
||||
|
||||
public ValueTask<SurfaceSecretHandle> GetAsync(SurfaceSecretRequest request, CancellationToken cancellationToken = default)
|
||||
=> ValueTask.FromResult(_handle);
|
||||
}
|
||||
|
||||
private sealed class StubSurfaceEnvironment : ISurfaceEnvironment
|
||||
{
|
||||
public StubSurfaceEnvironment()
|
||||
{
|
||||
Settings = new SurfaceEnvironmentSettings(
|
||||
new Uri("https://surface"),
|
||||
"bucket",
|
||||
"region",
|
||||
new DirectoryInfo(Path.GetTempPath()),
|
||||
256,
|
||||
false,
|
||||
Array.Empty<string>(),
|
||||
new SurfaceSecretsConfiguration("inline", "tenant", null, null, null, true),
|
||||
"tenant",
|
||||
new SurfaceTlsConfiguration(null, null, null));
|
||||
RawVariables = new Dictionary<string, string>(StringComparer.OrdinalIgnoreCase);
|
||||
}
|
||||
|
||||
public SurfaceEnvironmentSettings Settings { get; }
|
||||
|
||||
public IReadOnlyDictionary<string, string> RawVariables { get; }
|
||||
}
|
||||
|
||||
private sealed class OptionsMonitorStub<T> : IOptionsMonitor<T> where T : class
|
||||
{
|
||||
private readonly IOptions<T> _options;
|
||||
|
||||
public OptionsMonitorStub(IOptions<T> options)
|
||||
{
|
||||
_options = options;
|
||||
}
|
||||
|
||||
public T CurrentValue => _options.Value;
|
||||
|
||||
public T Get(string? name) => _options.Value;
|
||||
|
||||
public IDisposable OnChange(Action<T, string?> listener) => NullDisposable.Instance;
|
||||
|
||||
private sealed class NullDisposable : IDisposable
|
||||
{
|
||||
public static readonly NullDisposable Instance = new();
|
||||
public void Dispose() { }
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,92 @@
|
||||
using System;
|
||||
using System.Collections.Generic;
|
||||
using System.IO;
|
||||
using System.Text;
|
||||
using System.Threading;
|
||||
using Microsoft.Extensions.Logging.Abstractions;
|
||||
using StellaOps.Scanner.Storage;
|
||||
using StellaOps.Scanner.Surface.Env;
|
||||
using StellaOps.Scanner.Surface.Secrets;
|
||||
using StellaOps.Scanner.Worker.Options;
|
||||
using Xunit;
|
||||
|
||||
namespace StellaOps.Scanner.Worker.Tests;
|
||||
|
||||
public sealed class ScannerStorageSurfaceSecretConfiguratorTests
|
||||
{
|
||||
[Fact]
|
||||
public void Configure_WithCasAccessSecret_AppliesSettings()
|
||||
{
|
||||
const string json = """
|
||||
{
|
||||
"driver": "rustfs",
|
||||
"endpoint": "https://surface.example",
|
||||
"region": "eu-central-1",
|
||||
"bucket": "surface-bucket",
|
||||
"rootPrefix": "scanner",
|
||||
"apiKey": "rustfs-api",
|
||||
"apiKeyHeader": "X-Rustfs-Key",
|
||||
"allowInsecureTls": false
|
||||
}
|
||||
""";
|
||||
|
||||
using var handle = SurfaceSecretHandle.FromBytes(Encoding.UTF8.GetBytes(json));
|
||||
var secretProvider = new StubSecretProvider(handle);
|
||||
var environment = new StubSurfaceEnvironment("tenant-eu");
|
||||
|
||||
var configurator = new ScannerStorageSurfaceSecretConfigurator(
|
||||
secretProvider,
|
||||
environment,
|
||||
NullLogger<ScannerStorageSurfaceSecretConfigurator>.Instance);
|
||||
|
||||
var options = new ScannerStorageOptions();
|
||||
configurator.Configure(options);
|
||||
|
||||
Assert.Equal("rustfs", options.ObjectStore.Driver);
|
||||
Assert.Equal("https://surface.example", options.ObjectStore.RustFs.BaseUrl);
|
||||
Assert.Equal("eu-central-1", options.ObjectStore.Region);
|
||||
Assert.Equal("surface-bucket", options.ObjectStore.BucketName);
|
||||
Assert.Equal("scanner", options.ObjectStore.RootPrefix);
|
||||
Assert.Equal("rustfs-api", options.ObjectStore.RustFs.ApiKey);
|
||||
Assert.Equal("X-Rustfs-Key", options.ObjectStore.RustFs.ApiKeyHeader);
|
||||
}
|
||||
|
||||
private sealed class StubSecretProvider : ISurfaceSecretProvider
|
||||
{
|
||||
private readonly SurfaceSecretHandle _handle;
|
||||
|
||||
public StubSecretProvider(SurfaceSecretHandle handle)
|
||||
{
|
||||
_handle = handle;
|
||||
}
|
||||
|
||||
public ValueTask<SurfaceSecretHandle> GetAsync(SurfaceSecretRequest request, CancellationToken cancellationToken = default)
|
||||
=> ValueTask.FromResult(_handle);
|
||||
}
|
||||
|
||||
private sealed class StubSurfaceEnvironment : ISurfaceEnvironment
|
||||
{
|
||||
public StubSurfaceEnvironment(string tenant)
|
||||
{
|
||||
Settings = new SurfaceEnvironmentSettings(
|
||||
new Uri("https://surface"),
|
||||
"bucket",
|
||||
"region-1",
|
||||
new DirectoryInfo(Path.GetTempPath()),
|
||||
1024,
|
||||
false,
|
||||
Array.Empty<string>(),
|
||||
new SurfaceSecretsConfiguration("inline", tenant, null, null, null, true),
|
||||
tenant,
|
||||
new SurfaceTlsConfiguration(null, null, null))
|
||||
{
|
||||
CreatedAtUtc = DateTimeOffset.UtcNow
|
||||
};
|
||||
RawVariables = new Dictionary<string, string>(StringComparer.OrdinalIgnoreCase);
|
||||
}
|
||||
|
||||
public SurfaceEnvironmentSettings Settings { get; }
|
||||
|
||||
public IReadOnlyDictionary<string, string> RawVariables { get; }
|
||||
}
|
||||
}
|
||||
Reference in New Issue
Block a user