diff --git a/NuGet.config b/NuGet.config index b746b083a..7a1ae8ccc 100644 --- a/NuGet.config +++ b/NuGet.config @@ -33,10 +33,24 @@ - + + + + + + + + + + + + + + + diff --git a/docs/aoc/guard-library.md b/docs/aoc/guard-library.md index 9993ddee9..26606f962 100644 --- a/docs/aoc/guard-library.md +++ b/docs/aoc/guard-library.md @@ -73,6 +73,8 @@ Key points: - Register the guard singleton before wiring repositories or worker services. - Use `AocGuardEndpointFilter` to protect Minimal API endpoints. The `payloadSelector` can yield multiple payloads (e.g. batch ingestion) and the filter will validate each one. +- Prefer the `RequireAocGuard` extension when wiring endpoints; it wraps `AddEndpointFilter` + and handles single-payload scenarios without additional boilerplate. - Wrap guard exceptions with `AocHttpResults.Problem` to ensure clients receive machine-readables codes (`ERR_AOC_00x`). ## Worker / repository usage diff --git a/docs/dev/aoc-normalization-removal-notes.md b/docs/dev/aoc-normalization-removal-notes.md index 2e0c51c55..322ac4c2b 100644 --- a/docs/dev/aoc-normalization-removal-notes.md +++ b/docs/dev/aoc-normalization-removal-notes.md @@ -22,3 +22,4 @@ Document follow-up actions for CONCELIER-CORE-AOC-19-004 as we unwind the final - 2025-11-05: Catalogued residual normalization paths tied to the legacy Merge service and outlined `noMergeEnabled` feature-toggle work to keep AOC ingestion fully merge-free. - 2025-11-05 19:20Z: Observation factory/linkset now preserve upstream ordering and duplicates; canonicalisation shifts to downstream services. - 2025-11-06: Documented post-merge rollout plan and annotated sprint trackers with analyzer gating updates. +- 2025-11-06 23:30Z: Concelier core/linkset query paths now keep alias/reference casing & whitespace intact; alias filters switched to case-insensitive regex so raw data and lookups remain compatible. diff --git a/docs/events/orchestrator-scanner-events.md b/docs/events/orchestrator-scanner-events.md index a90b890b8..4169303a4 100644 --- a/docs/events/orchestrator-scanner-events.md +++ b/docs/events/orchestrator-scanner-events.md @@ -1,39 +1,39 @@ -# Scanner Orchestrator Events (ORCH-SVC-38-101) - -Last updated: 2025-10-26 - -The Notifications Studio initiative (NOTIFY-SVC-38-001) and orchestrator backlog (ORCH-SVC-38-101) standardise how platform services emit lifecycle events. This document describes the Scanner WebService contract for the new **orchestrator envelopes** (`scanner.event.*`) and how they supersede the legacy Redis-backed `scanner.report.ready` / `scanner.scan.completed` events. - -## 1. Envelope overview - -Orchestrator events share a deterministic JSON envelope: - -| Field | Type | Notes | -|-------|------|-------| -| `eventId` | `uuid` | Globally unique identifier generated per occurrence. | -| `kind` | `string` | Event identifier; Scanner emits `scanner.event.report.ready` and `scanner.event.scan.completed`. | -| `version` | `integer` | Schema version. Initial release uses `1`. | -| `tenant` | `string` | Tenant that owns the scan/report. Mirrors Authority claims. | -| `occurredAt` | `date-time` | UTC instant when the underlying state transition happened (e.g., report persisted). | -| `recordedAt` | `date-time` | UTC instant when the event was durably written. Optional but recommended. | -| `source` | `string` | Producer identifier (`scanner.webservice`). | -| `idempotencyKey` | `string` | Deterministic key for duplicate suppression (see §4). | -| `correlationId` | `string` | Maps back to the API request or scan identifier. | -| `traceId` / `spanId` | `string` | W3C trace context propagated into downstream telemetry. | -| `scope` | `object` | Describes the affected artefact. Requires `repo` and `digest`; optional `namespace`, `component`, `image`. | -| `attributes` | `object` | Flat string map for frequently queried metadata (e.g., policy revision). | -| `payload` | `object` | Event-specific body (see §2). | - -Canonical schemas live under `docs/events/scanner.event.*@1.json`. Samples that round-trip through `NotifyCanonicalJsonSerializer` are stored in `docs/events/samples/`. - -## 2. Event kinds and payloads - -### 2.1 `scanner.event.report.ready` - -Emitted once a signed report is persisted and attested. Payload highlights: - -- `reportId` / `scanId` — identifiers for the persisted report and originating scan. Until Scan IDs are surfaced by the API, `scanId` mirrors `reportId` so downstream correlators can stabilise on a single key. -- **Attributes:** `reportId`, `policyRevisionId`, `policyDigest`, `verdict` — pre-sorted for deterministic routing. +# Scanner Orchestrator Events (ORCH-SVC-38-101) + +Last updated: 2025-10-26 + +The Notifications Studio initiative (NOTIFY-SVC-38-001) and orchestrator backlog (ORCH-SVC-38-101) standardise how platform services emit lifecycle events. This document describes the Scanner WebService contract for the new **orchestrator envelopes** (`scanner.event.*`) and how they supersede the legacy Redis-backed `scanner.report.ready` / `scanner.scan.completed` events. + +## 1. Envelope overview + +Orchestrator events share a deterministic JSON envelope: + +| Field | Type | Notes | +|-------|------|-------| +| `eventId` | `uuid` | Globally unique identifier generated per occurrence. | +| `kind` | `string` | Event identifier; Scanner emits `scanner.event.report.ready` and `scanner.event.scan.completed`. | +| `version` | `integer` | Schema version. Initial release uses `1`. | +| `tenant` | `string` | Tenant that owns the scan/report. Mirrors Authority claims. | +| `occurredAt` | `date-time` | UTC instant when the underlying state transition happened (e.g., report persisted). | +| `recordedAt` | `date-time` | UTC instant when the event was durably written. Optional but recommended. | +| `source` | `string` | Producer identifier (`scanner.webservice`). | +| `idempotencyKey` | `string` | Deterministic key for duplicate suppression (see §4). | +| `correlationId` | `string` | Maps back to the API request or scan identifier. | +| `traceId` / `spanId` | `string` | W3C trace context propagated into downstream telemetry. | +| `scope` | `object` | Describes the affected artefact. Requires `repo` and `digest`; optional `namespace`, `component`, `image`. | +| `attributes` | `object` | Flat string map for frequently queried metadata (e.g., policy revision). | +| `payload` | `object` | Event-specific body (see §2). | + +Canonical schemas live under `docs/events/scanner.event.*@1.json`. Samples that round-trip through `NotifyCanonicalJsonSerializer` are stored in `docs/events/samples/`. + +## 2. Event kinds and payloads + +### 2.1 `scanner.event.report.ready` + +Emitted once a signed report is persisted and attested. Payload highlights: + +- `reportId` / `scanId` — identifiers for the persisted report and originating scan. Until Scan IDs are surfaced by the API, `scanId` mirrors `reportId` so downstream correlators can stabilise on a single key. +- **Attributes:** `reportId`, `policyRevisionId`, `policyDigest`, `verdict` — pre-sorted for deterministic routing. - **Links:** - `report.ui` → `/ui/reports/{reportId}` on the current host. - `report.api` → `{apiBasePath}/{reportsSegment}/{reportId}` (defaults to `/api/v1/reports/{reportId}`). @@ -41,83 +41,84 @@ Emitted once a signed report is persisted and attested. Payload highlights: - `policy.api` → `{apiBasePath}/{policySegment}/revisions/{revisionId}` when a revision is present. - `attestation.ui` → `/ui/attestations/{reportId}` when a DSSE envelope is included. - `attestation.api` → `{apiBasePath}/{reportsSegment}/{reportId}/attestation` when a DSSE envelope is included. -- `imageDigest` — OCI image digest associated with the analysis. -- `generatedAt` — report generation timestamp (ISO-8601 UTC). -- `verdict` — `pass`, `warn`, or `fail` after policy evaluation. -- `summary` — blocked/warned/ignored/quieted counters (all non-negative integers). -- `delta` — newly critical/high counts and optional `kev` array. -- `quietedFindingCount` — mirrors `summary.quieted`. -- `policy` — revision metadata (`digest`, `revisionId`) surfaced for routing. -- `links` — UI/report/policy URLs suitable for operators. -- `dsse` — embedded DSSE envelope (payload, type, signature list). -- `report` — canonical report document; identical to the DSSE payload. - -Schema: `docs/events/scanner.event.report.ready@1.json` -Sample: `docs/events/samples/scanner.event.report.ready@1.sample.json` - -### 2.2 `scanner.event.scan.completed` - -Emitted after scan execution finishes (success or policy failure). Payload highlights: - -- `reportId` / `scanId` / `imageDigest` — identifiers mirroring the report-ready event. As with the report-ready payload, `scanId` currently mirrors `reportId` as a temporary shim. -- **Attributes:** `reportId`, `policyRevisionId`, `policyDigest`, `verdict`. + - UI routes honour the configurable `scanner:console` options (`basePath`, `reportsSegment`, `policySegment`, `attestationsSegment`) so operators can move links under `/console` without code changes. +- `imageDigest` — OCI image digest associated with the analysis. +- `generatedAt` — report generation timestamp (ISO-8601 UTC). +- `verdict` — `pass`, `warn`, or `fail` after policy evaluation. +- `summary` — blocked/warned/ignored/quieted counters (all non-negative integers). +- `delta` — newly critical/high counts and optional `kev` array. +- `quietedFindingCount` — mirrors `summary.quieted`. +- `policy` — revision metadata (`digest`, `revisionId`) surfaced for routing. +- `links` — UI/report/policy URLs suitable for operators. +- `dsse` — embedded DSSE envelope (payload, type, signature list). +- `report` — canonical report document; identical to the DSSE payload. + +Schema: `docs/events/scanner.event.report.ready@1.json` +Sample: `docs/events/samples/scanner.event.report.ready@1.sample.json` + +### 2.2 `scanner.event.scan.completed` + +Emitted after scan execution finishes (success or policy failure). Payload highlights: + +- `reportId` / `scanId` / `imageDigest` — identifiers mirroring the report-ready event. As with the report-ready payload, `scanId` currently mirrors `reportId` as a temporary shim. +- **Attributes:** `reportId`, `policyRevisionId`, `policyDigest`, `verdict`. - **Links:** same as above (`report.*`, `policy.*`) with `attestation.*` populated when DSSE metadata exists. -- `verdict`, `summary`, `delta`, `policy` — same semantics as above. -- `findings` — array of surfaced findings with `id`, `severity`, optional `cve`, `purl`, and `reachability`. -- `links`, `dsse`, `report` — same structure as §2.1 (allows Notifier to reuse signatures). - -Schema: `docs/events/scanner.event.scan.completed@1.json` -Sample: `docs/events/samples/scanner.event.scan.completed@1.sample.json` - -### 2.3 Relationship to legacy events - -| Legacy Redis event | Replacement orchestrator event | Notes | -|--------------------|-------------------------------|-------| -| `scanner.report.ready` | `scanner.event.report.ready` | Adds versioning, idempotency, trace context. Payload is a superset of the legacy fields. | -| `scanner.scan.completed` | `scanner.event.scan.completed` | Same data plus explicit scan identifiers and orchestrator metadata. | - -Legacy schemas remain for backwards-compatibility during migration, but new integrations **must** target the orchestrator variants. - -## 3. Deterministic serialization - -- Producers must serialise events using `NotifyCanonicalJsonSerializer` to guarantee consistent key ordering and whitespace. -- Timestamps (`occurredAt`, `recordedAt`, `payload.generatedAt`) use `DateTimeOffset.UtcDateTime.ToString("O")`. -- Payload arrays (`delta.kev`, `findings`) should be pre-sorted (e.g., alphabetical CVE order) so hash-based consumers remain stable. -- Optional fields are omitted rather than emitted as `null`. - -## 4. Idempotency and correlation - -Idempotency keys dedupe repeated publishes and align with the orchestrator’s outbox pattern: - -| Event kind | Idempotency key template | -|------------|-------------------------| -| `scanner.event.report.ready` | `scanner.event.report.ready::` | -| `scanner.event.scan.completed` | `scanner.event.scan.completed::` | - -Keys are ASCII lowercase; components should be trimmed and validated before concatenation. Retries must reuse the same key. - -`correlationId` should match the scan identifier that appears in REST responses (`scanId`). Re-using the same value across the pair of events allows Notifier and orchestrator analytics to stitch lifecycle data together. - -## 5. Versioning and evolution - -- Increment the `version` field and the `@` suffix for **breaking** changes (field removals, type changes, semantic shifts). -- Additive optional fields may remain within version 1; update the JSON schema and samples accordingly. -- When introducing `@2`, keep the `@1` schema/docs in place until orchestrator subscribers confirm migration. - -## 6. Consumer checklist - -1. Validate incoming payloads against the schema for the targeted version. -2. Use `idempotencyKey` for dedupe, not `eventId`. -3. Map `traceId`/`spanId` into telemetry spans to preserve causality. -4. Prefer `payload.report` → `policy.revisionId` when populating templates; the top-level `attributes` are convenience duplicates for quick routing. -5. Reserve the legacy Redis events for transitional compatibility only; downstream systems should subscribe to the orchestrator bus exposed by ORCH-SVC-38-101. - -## 7. Implementation status and next actions - -- **Scanner WebService** — `SCANNER-EVENTS-16-301` (blocked) and `SCANNER-EVENTS-16-302` (doing) track the production of these envelopes. The remaining blocker is the .NET 10 preview OpenAPI/Auth dependency drift that currently breaks `dotnet test`. Once Gateway and Notifier owners land the replacement packages, rerun the full test suite and capture fresh fixtures under `docs/events/samples/`. -- **Gateway/Notifier consumers** — subscribe to the orchestrator stream documented in ORCH-SVC-38-101. When the Scanner tasks unblock, regenerate notifier contract tests against the sample events included here. -- **Docs cadence** — update this file and the matching JSON schemas whenever payload fields change. Use the rehearsal checklist in `docs/modules/devops/runbooks/launch-cutover.md` to confirm downstream validation before the production cutover. Record gaps or newly required fields in `docs/modules/devops/runbooks/launch-readiness.md` so they land in the launch checklist. - ---- - -**Imposed rule reminder:** work of this type or tasks of this type on this component must also be applied everywhere else it should be applied. \ No newline at end of file +- `verdict`, `summary`, `delta`, `policy` — same semantics as above. +- `findings` — array of surfaced findings with `id`, `severity`, optional `cve`, `purl`, and `reachability`. +- `links`, `dsse`, `report` — same structure as §2.1 (allows Notifier to reuse signatures). + +Schema: `docs/events/scanner.event.scan.completed@1.json` +Sample: `docs/events/samples/scanner.event.scan.completed@1.sample.json` + +### 2.3 Relationship to legacy events + +| Legacy Redis event | Replacement orchestrator event | Notes | +|--------------------|-------------------------------|-------| +| `scanner.report.ready` | `scanner.event.report.ready` | Adds versioning, idempotency, trace context. Payload is a superset of the legacy fields. | +| `scanner.scan.completed` | `scanner.event.scan.completed` | Same data plus explicit scan identifiers and orchestrator metadata. | + +Legacy schemas remain for backwards-compatibility during migration, but new integrations **must** target the orchestrator variants. + +## 3. Deterministic serialization + +- Producers must serialise events using `NotifyCanonicalJsonSerializer` to guarantee consistent key ordering and whitespace. +- Timestamps (`occurredAt`, `recordedAt`, `payload.generatedAt`) use `DateTimeOffset.UtcDateTime.ToString("O")`. +- Payload arrays (`delta.kev`, `findings`) should be pre-sorted (e.g., alphabetical CVE order) so hash-based consumers remain stable. +- Optional fields are omitted rather than emitted as `null`. + +## 4. Idempotency and correlation + +Idempotency keys dedupe repeated publishes and align with the orchestrator’s outbox pattern: + +| Event kind | Idempotency key template | +|------------|-------------------------| +| `scanner.event.report.ready` | `scanner.event.report.ready::` | +| `scanner.event.scan.completed` | `scanner.event.scan.completed::` | + +Keys are ASCII lowercase; components should be trimmed and validated before concatenation. Retries must reuse the same key. + +`correlationId` should match the scan identifier that appears in REST responses (`scanId`). Re-using the same value across the pair of events allows Notifier and orchestrator analytics to stitch lifecycle data together. + +## 5. Versioning and evolution + +- Increment the `version` field and the `@` suffix for **breaking** changes (field removals, type changes, semantic shifts). +- Additive optional fields may remain within version 1; update the JSON schema and samples accordingly. +- When introducing `@2`, keep the `@1` schema/docs in place until orchestrator subscribers confirm migration. + +## 6. Consumer checklist + +1. Validate incoming payloads against the schema for the targeted version. +2. Use `idempotencyKey` for dedupe, not `eventId`. +3. Map `traceId`/`spanId` into telemetry spans to preserve causality. +4. Prefer `payload.report` → `policy.revisionId` when populating templates; the top-level `attributes` are convenience duplicates for quick routing. +5. Reserve the legacy Redis events for transitional compatibility only; downstream systems should subscribe to the orchestrator bus exposed by ORCH-SVC-38-101. + +## 7. Implementation status and next actions + +- **Scanner WebService** — `SCANNER-EVENTS-16-301` (blocked) and `SCANNER-EVENTS-16-302` (done) track the production of these envelopes. Dispatcher link customisation landed and samples updated; full `dotnet test` suite now succeeds after Surface cache ctor drift was patched and DSSE fixtures re-synced (2025-11-06). +- **Gateway/Notifier consumers** — subscribe to the orchestrator stream documented in ORCH-SVC-38-101. When the Scanner tasks unblock, regenerate notifier contract tests against the sample events included here. +- **Docs cadence** — update this file and the matching JSON schemas whenever payload fields change. Use the rehearsal checklist in `docs/modules/devops/runbooks/launch-cutover.md` to confirm downstream validation before the production cutover. Record gaps or newly required fields in `docs/modules/devops/runbooks/launch-readiness.md` so they land in the launch checklist. + +--- + +**Imposed rule reminder:** work of this type or tasks of this type on this component must also be applied everywhere else it should be applied. diff --git a/docs/implplan/SPRINTS.md b/docs/implplan/SPRINTS.md index 22c9b5b70..0d21030f9 100644 --- a/docs/implplan/SPRINTS.md +++ b/docs/implplan/SPRINTS.md @@ -1,32 +1,31 @@ -# Sprint Index - -Follow the sprint files below in order. Update task status in both `SPRINTS` and module `TASKS.md` as you progress. - -- [Identity & Signing](./SPRINT_100_identity_signing.md) -- [Ingestion & Evidence](./SPRINT_110_ingestion_evidence.md) -- [Policy & Reasoning](./SPRINT_120_policy_reasoning.md) -- [Scanner & Surface](./SPRINT_130_scanner_surface.md) -- [Runtime & Signals](./SPRINT_140_runtime_signals.md) -- [Scheduling & Automation](./SPRINT_150_scheduling_automation.md) -- [Export & Evidence](./SPRINT_160_export_evidence.md) -- [Notifications & Telemetry](./SPRINT_170_notifications_telemetry.md) -- [Experience & SDKs](./SPRINT_180_experience_sdks.md) -- [Ops & Offline](./SPRINT_190_ops_offline.md) -- [Documentation & Process](./SPRINT_200_documentation_process.md) - -<<<<<<< Updated upstream -> 2025-11-03: ATTESTOR-72-003 moved to DOING (Attestor Service Guild) – running live TTL validation against local MongoDB/Redis processes (manual hosts, no Docker). -> 2025-11-03: ATTESTOR-72-003 marked DONE (Attestor Service Guild) – Mongo/Redis TTL expiry logs archived under `docs/modules/attestor/evidence/2025-11-03-*.txt` with summary in `docs/modules/attestor/ttl-validation.md`. -> 2025-11-03: AIAI-31-004B moved to DOING (Advisory AI Guild) – starting prompt assembler/guardrail plumbing, cache persistence contract, and DSSE provenance wiring. -> 2025-11-03: PLG7.RFC marked DONE (Auth Plugin Guild, Security Guild) – LDAP plugin RFC accepted; review log stored at `docs/notes/2025-11-03-authority-plugin-ldap-review.md`, follow-up PLG7.IMPL-001..005 queued. -> 2025-11-03: PLG7.IMPL-001 marked DONE (Auth Plugin Guild) – new `StellaOps.Authority.Plugin.Ldap` project/tests scaffolded with configuration normalization & validation; sample manifest refreshed and smoke tests run (`dotnet test`). -> 2025-11-03: AIAI-31-004B marked DONE (Advisory AI Guild) – prompt assembler, guardrail hooks, DSSE-ready output persistence, and golden prompt tests landed. -> 2025-11-03: AIAI-31-005 moved to DOING (Advisory AI Guild) – beginning guardrail enforcement (redaction, injection defence, output validator) implementation. -> 2025-11-03: AIAI-31-006 moved to DOING (Advisory AI Guild) – starting Advisory AI REST API surface work (RBAC, rate limits, batching contract). -> 2025-11-03: EVID-OBS-53-001 moved to DOING (Evidence Locker Guild) – bootstrapping Evidence Locker schema and storage abstractions. -> 2025-11-03: GRAPH-INDEX-28-002 marked DONE (Graph Indexer Guild) – SBOM ingest transformer, processor, and metrics landed with refreshed fixtures/tests for license and base artifact determinism. -> 2025-11-03: GRAPH-INDEX-28-003 marked DONE (Graph Indexer Guild) – advisory linkset snapshot model repaired, transformer finalized with dedupe/canonical provenance, fixtures refreshed, and overlay tests passing across the graph suite. -> 2025-11-03: GRAPH-INDEX-28-004 moved to DOING (Graph Indexer Guild) – beginning VEX overlay integration with precedent/justification metadata. +# Sprint Index + +Follow the sprint files below in order. Update task status in both `SPRINTS` and module `TASKS.md` as you progress. + +- [Identity & Signing](./SPRINT_100_identity_signing.md) +- [Ingestion & Evidence](./SPRINT_110_ingestion_evidence.md) +- [Policy & Reasoning](./SPRINT_120_policy_reasoning.md) +- [Scanner & Surface](./SPRINT_130_scanner_surface.md) +- [Runtime & Signals](./SPRINT_140_runtime_signals.md) +- [Scheduling & Automation](./SPRINT_150_scheduling_automation.md) +- [Export & Evidence](./SPRINT_160_export_evidence.md) +- [Notifications & Telemetry](./SPRINT_170_notifications_telemetry.md) +- [Experience & SDKs](./SPRINT_180_experience_sdks.md) +- [Ops & Offline](./SPRINT_190_ops_offline.md) +- [Documentation & Process](./SPRINT_200_documentation_process.md) + +> 2025-11-03: ATTESTOR-72-003 moved to DOING (Attestor Service Guild) – running live TTL validation against local MongoDB/Redis processes (manual hosts, no Docker). +> 2025-11-03: ATTESTOR-72-003 marked DONE (Attestor Service Guild) – Mongo/Redis TTL expiry logs archived under `docs/modules/attestor/evidence/2025-11-03-*.txt` with summary in `docs/modules/attestor/ttl-validation.md`. +> 2025-11-03: AIAI-31-004B moved to DOING (Advisory AI Guild) – starting prompt assembler/guardrail plumbing, cache persistence contract, and DSSE provenance wiring. +> 2025-11-03: PLG7.RFC marked DONE (Auth Plugin Guild, Security Guild) – LDAP plugin RFC accepted; review log stored at `docs/notes/2025-11-03-authority-plugin-ldap-review.md`, follow-up PLG7.IMPL-001..005 queued. +> 2025-11-03: PLG7.IMPL-001 marked DONE (Auth Plugin Guild) – new `StellaOps.Authority.Plugin.Ldap` project/tests scaffolded with configuration normalization & validation; sample manifest refreshed and smoke tests run (`dotnet test`). +> 2025-11-03: AIAI-31-004B marked DONE (Advisory AI Guild) – prompt assembler, guardrail hooks, DSSE-ready output persistence, and golden prompt tests landed. +> 2025-11-03: AIAI-31-005 moved to DOING (Advisory AI Guild) – beginning guardrail enforcement (redaction, injection defence, output validator) implementation. +> 2025-11-03: AIAI-31-006 moved to DOING (Advisory AI Guild) – starting Advisory AI REST API surface work (RBAC, rate limits, batching contract). +> 2025-11-03: EVID-OBS-53-001 moved to DOING (Evidence Locker Guild) – bootstrapping Evidence Locker schema and storage abstractions. +> 2025-11-03: GRAPH-INDEX-28-002 marked DONE (Graph Indexer Guild) – SBOM ingest transformer, processor, and metrics landed with refreshed fixtures/tests for license and base artifact determinism. +> 2025-11-03: GRAPH-INDEX-28-003 marked DONE (Graph Indexer Guild) – advisory linkset snapshot model repaired, transformer finalized with dedupe/canonical provenance, fixtures refreshed, and overlay tests passing across the graph suite. +> 2025-11-03: GRAPH-INDEX-28-004 moved to DOING (Graph Indexer Guild) – beginning VEX overlay integration with precedent/justification metadata. > 2025-11-03: GRAPH-INDEX-28-004 marked DONE (Graph Indexer Guild) – VEX snapshot/transformer merged with deterministic overlays, fixtures refreshed, and graph indexer tests passing. > 2025-11-03: GRAPH-INDEX-28-005 moved to DOING (Graph Indexer Guild, Policy Guild) – starting policy overlay hydration (`governs_with` nodes/edges) with explain hash references. > 2025-11-03: GRAPH-INDEX-28-005 marked DONE (Graph Indexer Guild, Policy Guild) – policy overlay snapshot/transformer landed with deterministic nodes/edges and fixture-backed tests; Mongo writer tests now probe `STELLAOPS_TEST_MONGO_URI`/localhost before falling back to Mongo2Go and skip when no mongod is reachable. @@ -39,123 +38,123 @@ Follow the sprint files below in order. Update task status in both `SPRINTS` and > 2025-11-04: SCHED-WEB-21-004 marked DONE (Scheduler WebService Guild, Scheduler Storage Guild) – Mongo lifecycle persistence, single-shot completion events/webhooks, and idempotent result URI refresh landed with unit/integration coverage. > 2025-11-04: TASKRUN-42-001 resumed (Task Runner Guild) – planning loops/conditionals/maxParallel execution upgrades, simulation mode, policy gate wiring, and deterministic retry/abort handling. > 2025-11-04: TASKRUN-42-001 progress update – execution graph + simulation endpoints wired; retry windows now persisted for orchestration clients. -> 2025-11-03: AIRGAP-POL-57-002 confirmed DOING (AirGap Policy Guild, Task Runner Guild) – continuing Task Runner sealed-mode egress validation and test sweep. -> 2025-11-03: AIRGAP-POL-57-002 marked DONE (AirGap Policy Guild, Task Runner Guild) – worker now injects `IEgressPolicy`, filesystem dispatcher enforces sealed-mode egress, planner grants normalized, sealed-mode dispatcher test added; follow-up queued to lift remaining dispatchers/executors onto the shared policy before sealing the full worker loop. -> 2025-11-03: MERGE-LNM-21-001 moved to DOING (BE-Merge, Architecture Guild) – drafting `no-merge` migration playbook outline and capturing rollout/backfill checkpoints. -> 2025-11-03: MERGE-LNM-21-001 marked DONE – published `docs/migration/no-merge.md` with rollout, backfill, validation, and rollback guidance for the LNM cutover. -> 2025-11-04: GRAPH-INDEX-28-011 marked DONE (Graph Indexer Guild) – SBOM ingest DI wiring now emits graph snapshots by default, snapshot root configurable via `STELLAOPS_GRAPH_SNAPSHOT_DIR`, and Graph Indexer tests exercised with Mongo URI guidance. -> 2025-11-03: MERGE-LNM-21-002 moved to DOING (BE-Merge) – auditing `AdvisoryMergeService` call sites to scope removal and analyzer enforcement. -> 2025-11-03: DOCS-LNM-22-008 moved to DOING (Docs Guild, DevOps Guild) – aligning migration playbook structure and readiness checklist. -> 2025-11-03: DOCS-LNM-22-008 marked DONE – `/docs/migration/no-merge.md` published for DevOps/Export Center planning with checklist for cutover readiness. -> 2025-11-03: SCHED-CONSOLE-27-001 marked DONE (Scheduler WebService Guild, Policy Registry Guild) – policy simulation endpoints now emit SSE retry/heartbeat, enforce metadata normalization, support Mongo-backed integration, and ship auth/stream coverage. -> 2025-11-03: SCHED-CONSOLE-27-002 moved to DOING (Scheduler WebService Guild, Observability Guild) – wiring policy simulation telemetry endpoints, OTEL metrics, and Registry webhooks on completion/failure. -> 2025-11-03: FEEDCONN-KISA-02-008 moved to DOING (BE-Conn-KISA, Models) – starting Hangul firmware range normalization and provenance mapping for KISA advisories. -> 2025-11-03: FEEDCONN-KISA-02-008 progress – SemVer normalization wired through KISA mapper with provenance slugs, exclusive marker handling, and fresh connector tests for `이상`/`미만`/`초과` scenarios plus non-numeric fallback; follow-up review queued for additional phrasing coverage before closing. Captured current detail pages via `scripts/kisa_capture_html.py` so offline HTML is available under `seed-data/kisa/html/`. -> 2025-11-03: FEEDCONN-ICSCISA-02-012 marked DONE (BE-Conn-ICS-CISA) – ICS CISA connector now emits semver-aware affected.version ranges with `ics-cisa` provenance, SourceFetchService RSS fallback passes the AOC guard, and the Fetch/Parse/Map integration test is green. -> 2025-11-01: SCANNER-ANALYZERS-LANG-10-308R marked DONE (Language Analyzer Guild) – heuristics fixtures, benchmarks, and coverage comparison published. -> 2025-11-01: SCANNER-ANALYZERS-LANG-10-309R marked DONE (Language Analyzer Guild) – Rust analyzer packaged with offline kit smoke tests and docs. -> 2025-11-01: ENTRYTRACE-SURFACE-01 moved to DOING (EntryTrace Guild) – wiring Surface.Validation and Surface.FS reuse ahead of EntryTrace runs. -> 2025-11-01: AUTH-OBS-50-001 (Sprint 50 – Observability & Forensics) moved to DOING (Authority Core & Security Guild). -> 2025-11-01: AUTH-PACKS-41-001 moved to DOING (Authority Core & Security Guild) – add Packs.* scopes to Authority. -> 2025-11-01: AUTH-OBS-55-001 (Sprint 55 – Observability & Forensics) moved to DOING (Authority Core & Security Guild, Ops Guild). -> 2025-11-01: TASKRUN-41-001 moved to DOING (Task Runner Guild) – request packs.* scopes when calling Authority. -> 2025-11-01: PACKS-REG-41-001 moved to DOING (Packs Registry Guild) – enforce packs.* scopes for registry publish/run flows. -> 2025-11-01: ATTEST-VERIFY-74-001 re-opened and set to DOING to unblock build/test regressions (Verification Guild, Observability Guild). -> 2025-11-01: ATTEST-VERIFY-74-001 marked DONE after configuration and test fixes (Verification Guild, Observability Guild). -> 2025-11-01: AUTH-AIAI-31-001 marked DONE (Authority Core & Security Guild) – Advisory AI scopes published and remote inference toggles documented. -> 2025-11-01: AUTH-AIRGAP-56-001 moved to DOING (Authority Core & Security Guild) – add airgap scope catalogue and defaults. -> 2025-11-01: AUTH-AIRGAP-56-002 moved to DOING (Authority Core & Security Guild) – implement airgap audit endpoint and logging. -> 2025-11-01: ISSUER-30-001 marked DONE (Issuer Directory Guild) – Issuer Directory service scaffolded with CRUD APIs, audit sink, CSAF seed import, and unit tests. -> 2025-11-01: ISSUER-30-002 marked DONE (Issuer Directory Guild, Security Guild) – Key management domain, Mongo persistence, CRUD/rotate/revoke endpoints, validation, and tests delivered. -> 2025-11-01: ISSUER-30-004 marked DONE (Issuer Directory Guild, VEX Lens Guild) – Excititor worker consumes issuer directory client for key/trust lookup with cached offline support. -> 2025-11-01: ISSUER-30-005 marked DONE (Issuer Directory Guild, Observability Guild) – Issuer Directory service emits structured logs + metrics for issuer/key flows with OTEL meter. -> 2025-11-02: SURFACE-ENV-01 moved to DOING (Surface Env Guild) – drafting shared environment spec for Scanner/Zastava. -> 2025-11-02: SURFACE-ENV-02 moved to DOING (Surface Env Guild) – implementing typed environment resolver and unit tests. -> 2025-11-02: SURFACE-VAL-01 moved to DOING (Surface Validation Guild) – aligning design document with implementation plan. -> 2025-11-02: SURFACE-FS-01 moved to DOING (Surface FS Guild) – finalising cache layout and manifest spec. -> 2025-11-02: SURFACE-FS-02 moved to DOING (Surface FS Guild) – building core abstractions and deterministic serializers. -> 2025-11-02: SURFACE-SECRETS-01 moved to DOING (Surface Secrets Guild) – updating secrets design for provider matrix. -> 2025-11-02: SURFACE-SECRETS-02 moved to DOING (Surface Secrets Guild) – implementing base providers + tests. -> 2025-11-02: AUTH-POLICY-27-002 marked DONE (Authority Core & Security Guild) – interactive-only policy publish/promote scopes delivered with metadata, fresh-auth enforcement, and audit/docs updates. -> 2025-11-02: SCANNER-ENTRYTRACE-18-506 moved to DOING (EntryTrace Guild, Scanner WebService Guild) – surfacing EntryTrace results via WebService/CLI with confidence metadata. -> 2025-11-02: ATTESTOR-74-001 marked DONE (Attestor Service Guild) – witness client integration, repository schema, and verification/reporting updates landed with tests. -> 2025-11-02: AUTH-OAS-63-001 moved to DOING (Authority Core & Security Guild, API Governance Guild) – verifying legacy `/oauth/*` deprecation signalling and notifications ahead of sunset. -> 2025-11-02: AUTH-OAS-63-001 marked DONE (Authority Core & Security Guild, API Governance Guild) – legacy shims emit Deprecation/Sunset/Warning headers, audit event coverage validated, and migration guide published. -> 2025-11-02: AUTH-NOTIFY-40-001 marked DONE (Authority Core & Security Guild) – `/notify/ack-tokens/rotate` (notify.admin) now rotates DSSE keys with audit trails and integration tests. -> 2025-11-02: AUTH-OAS-62-001 moved to DOING (Authority Core & Security Guild, SDK Generator Guild) – wiring SDK helpers for OAuth2/PAT flows and tenancy override header. -> 2025-11-02: AUTH-OAS-62-001 marked DONE (Authority Core & Security Guild, SDK Generator Guild) – HttpClient auth helper (OAuth2/PAT) shipped with tenant header support and unit tests. -> 2025-11-02: AUTH-OBS-50-001 moved to DOING (Authority Core & Security Guild) – defining observability scopes and updating discovery/offline defaults. -> 2025-11-02: AUTH-OBS-52-001 moved to DOING (Authority Core & Security Guild) – rolling observability scopes through resource server policies and audit wiring. -> 2025-11-02: AUTH-OBS-55-001 marked DONE (Authority Core & Security Guild, Ops Guild) – incident-mode tokens now require fresh auth, audit records expose `incident.reason`, and `/authority/audit/incident` verification path documented. -> 2025-11-02: AUTH-ORCH-34-001 marked DONE (Authority Core & Security Guild) – `orch:backfill` scope enforced with reason/ticket metadata, Authority + CLI updated, docs/config refreshed for Orchestrator admins. -> 2025-11-02: AUTH-PACKS-41-001 moved to DOING (Authority Core & Security Guild) – defining packs scope catalogue, issuer templates, and offline defaults. -> 2025-11-02: AUTH-PACKS-41-001 added shared OpenSSL 1.1 test libs so Authority & Signals Mongo2Go suites run on OpenSSL 3. -> 2025-11-02: AUTH-NOTIFY-42-001 moved to DOING (Authority Core & Security Guild) – investigating `/notify/ack-tokens/rotate` 500 responses when key metadata missing. -> 2025-11-02: AUTH-NOTIFY-42-001 marked DONE (Authority Core & Security Guild) – bootstrap rotate defaults fixed, `StellaOpsBearer` test alias added, and notify ack rotation regression passes. -> 2025-11-03: AUTH-TEN-49-001 marked DONE (Authority Core & Security Guild) – service account delegation (`act` chain) shipped with quota/audit coverage; Authority tests green. -> 2025-11-03: AUTH-VULN-29-003 marked DONE (Authority Core & Docs Guild) – Vuln Explorer security docs, samples, and release notes refreshed for roles, ABAC policies, attachment signing, and ledger verification. -> 2025-11-03: ISSUER-30-003 marked DONE (Issuer Directory Guild, Policy Guild) – trust override APIs/client finalized with cache invalidation/failure-path tests; Issuer Directory suite passing. -> 2025-11-03: AUTH-AIRGAP-56-001/56-002 marked DONE (Authority Core & Security Guild) – air-gap scope catalog surfaced in discovery/OpenAPI and `/authority/audit/airgap` endpoint shipped with tests. -> 2025-11-03: AUTH-PACKS-41-001 marked DONE (Authority Core & Security Guild) – packs scope bundle now emitted via discovery metadata, reflected in OpenAPI, and covered by Authority tests. -> 2025-11-03: AUTH-POLICY-27-003 marked DONE (Authority Core & Docs Guild) – Policy Studio docs/config updated for publish/promote signing workflow, CLI commands, and compliance checklist. -> 2025-11-02: ENTRYTRACE-SURFACE-02 moved to DOING (EntryTrace Guild) – replacing direct env/secret access with Surface.Secrets provider for EntryTrace runs. -> 2025-11-02: ENTRYTRACE-SURFACE-01 marked DONE (EntryTrace Guild) – Surface.Validation + Surface.FS cache now drive EntryTrace reuse with regression tests. -> 2025-11-02: ENTRYTRACE-SURFACE-02 marked DONE (EntryTrace Guild) – EntryTrace environment placeholders resolved via Surface.Secrets with updated docs/tests. -> 2025-11-02: SCANNER-ENTRYTRACE-18-506 marked DONE (EntryTrace Guild, Scanner WebService Guild) – EntryTrace graph surfaced via WebService and CLI with confidence metadata. -> 2025-11-02: SCANNER-ENTRYTRACE-18-509 moved to DOING (EntryTrace Guild, QA Guild) – adding regression coverage for EntryTrace surfaces and NDJSON hashing. -> 2025-11-02: SCANNER-ENTRYTRACE-18-509 marked DONE (EntryTrace Guild, QA Guild) – regression coverage landed for result store/WebService/CLI with NDJSON hashing snapshot. -> 2025-11-02: SCANNER-ENTRYTRACE-18-507 marked DONE (EntryTrace Guild) – fallback candidate discovery now covers history, supervisor configs, service directories, and entrypoint scripts with tests. -> 2025-11-02: SCANNER-ENTRYTRACE-18-508 marked DONE (EntryTrace Guild) – wrapper catalogue expanded for bundle, docker-php-entrypoint, npm, yarn, pipenv, and poetry with wrapper metadata assertions. -> 2025-11-02: CONCELIER-WEB-OAS-61-001 moved to DOING (Concelier WebService Guild) – implementing discovery endpoint for `.well-known/openapi` with version metadata and ETag. -> 2025-11-02: CONCELIER-WEB-OAS-61-001 marked DONE (Concelier WebService Guild) – discovery endpoint now serves signed OpenAPI 3.1 document with ETag support. -> 2025-11-02: DOCS-SCANNER-BENCH-62-001 moved to DOING (Docs Guild, Scanner Guild) – refreshing Trivy/Grype/Snyk comparison docs and ecosystem matrix with source-linked coverage. -> 2025-11-02: DOCS-SCANNER-BENCH-62-001 marked DONE (Docs Guild, Scanner Guild) – matrix updated with Windows/macOS coverage row and secret detection techniques; deep dives cite Trivy/Grype/Snyk sources. -> 2025-11-02: DOCS-SCANNER-BENCH-62-003 added (Docs Guild, Product Guild) – recording Python lockfile/editable-install demand signals for policy guidance follow-up. -> 2025-11-02: DOCS-SCANNER-BENCH-62-004 added (Docs Guild, Java Analyzer Guild) – documenting Java lockfile ingestion plan and policy templates. -> 2025-11-02: DOCS-SCANNER-BENCH-62-005 added (Docs Guild, Go Analyzer Guild) – documenting Go stripped-binary fallback enrichment guidance. -> 2025-11-02: DOCS-SCANNER-BENCH-62-006 added (Docs Guild, Rust Analyzer Guild) – documenting Rust fingerprint enrichment guidance. +> 2025-11-03: AIRGAP-POL-57-002 confirmed DOING (AirGap Policy Guild, Task Runner Guild) – continuing Task Runner sealed-mode egress validation and test sweep. +> 2025-11-03: AIRGAP-POL-57-002 marked DONE (AirGap Policy Guild, Task Runner Guild) – worker now injects `IEgressPolicy`, filesystem dispatcher enforces sealed-mode egress, planner grants normalized, sealed-mode dispatcher test added; follow-up queued to lift remaining dispatchers/executors onto the shared policy before sealing the full worker loop. +> 2025-11-03: MERGE-LNM-21-001 moved to DOING (BE-Merge, Architecture Guild) – drafting `no-merge` migration playbook outline and capturing rollout/backfill checkpoints. +> 2025-11-03: MERGE-LNM-21-001 marked DONE – published `docs/migration/no-merge.md` with rollout, backfill, validation, and rollback guidance for the LNM cutover. +> 2025-11-04: GRAPH-INDEX-28-011 marked DONE (Graph Indexer Guild) – SBOM ingest DI wiring now emits graph snapshots by default, snapshot root configurable via `STELLAOPS_GRAPH_SNAPSHOT_DIR`, and Graph Indexer tests exercised with Mongo URI guidance. +> 2025-11-06: MERGE-LNM-21-002 remains DOING (BE-Merge) – default-off merge DI + job gating landed, but Concelier WebService ingest/mirror tests are failing; guard and migration fixes pending before completion. +> 2025-11-03: DOCS-LNM-22-008 moved to DOING (Docs Guild, DevOps Guild) – aligning migration playbook structure and readiness checklist. +> 2025-11-03: DOCS-LNM-22-008 marked DONE – `/docs/migration/no-merge.md` published for DevOps/Export Center planning with checklist for cutover readiness. +> 2025-11-03: SCHED-CONSOLE-27-001 marked DONE (Scheduler WebService Guild, Policy Registry Guild) – policy simulation endpoints now emit SSE retry/heartbeat, enforce metadata normalization, support Mongo-backed integration, and ship auth/stream coverage. +> 2025-11-03: SCHED-CONSOLE-27-002 moved to DOING (Scheduler WebService Guild, Observability Guild) – wiring policy simulation telemetry endpoints, OTEL metrics, and Registry webhooks on completion/failure. +> 2025-11-03: FEEDCONN-KISA-02-008 moved to DOING (BE-Conn-KISA, Models) – starting Hangul firmware range normalization and provenance mapping for KISA advisories. +> 2025-11-03: FEEDCONN-KISA-02-008 progress – SemVer normalization wired through KISA mapper with provenance slugs, exclusive marker handling, and fresh connector tests for `이상`/`미만`/`초과` scenarios plus non-numeric fallback; follow-up review queued for additional phrasing coverage before closing. Captured current detail pages via `scripts/kisa_capture_html.py` so offline HTML is available under `seed-data/kisa/html/`. +> 2025-11-03: FEEDCONN-ICSCISA-02-012 marked DONE (BE-Conn-ICS-CISA) – ICS CISA connector now emits semver-aware affected.version ranges with `ics-cisa` provenance, SourceFetchService RSS fallback passes the AOC guard, and the Fetch/Parse/Map integration test is green. +> 2025-11-01: SCANNER-ANALYZERS-LANG-10-308R marked DONE (Language Analyzer Guild) – heuristics fixtures, benchmarks, and coverage comparison published. +> 2025-11-01: SCANNER-ANALYZERS-LANG-10-309R marked DONE (Language Analyzer Guild) – Rust analyzer packaged with offline kit smoke tests and docs. +> 2025-11-01: ENTRYTRACE-SURFACE-01 moved to DOING (EntryTrace Guild) – wiring Surface.Validation and Surface.FS reuse ahead of EntryTrace runs. +> 2025-11-01: AUTH-OBS-50-001 (Sprint 50 – Observability & Forensics) moved to DOING (Authority Core & Security Guild). +> 2025-11-01: AUTH-PACKS-41-001 moved to DOING (Authority Core & Security Guild) – add Packs.* scopes to Authority. +> 2025-11-01: AUTH-OBS-55-001 (Sprint 55 – Observability & Forensics) moved to DOING (Authority Core & Security Guild, Ops Guild). +> 2025-11-01: TASKRUN-41-001 moved to DOING (Task Runner Guild) – request packs.* scopes when calling Authority. +> 2025-11-01: PACKS-REG-41-001 moved to DOING (Packs Registry Guild) – enforce packs.* scopes for registry publish/run flows. +> 2025-11-01: ATTEST-VERIFY-74-001 re-opened and set to DOING to unblock build/test regressions (Verification Guild, Observability Guild). +> 2025-11-01: ATTEST-VERIFY-74-001 marked DONE after configuration and test fixes (Verification Guild, Observability Guild). +> 2025-11-01: AUTH-AIAI-31-001 marked DONE (Authority Core & Security Guild) – Advisory AI scopes published and remote inference toggles documented. +> 2025-11-01: AUTH-AIRGAP-56-001 moved to DOING (Authority Core & Security Guild) – add airgap scope catalogue and defaults. +> 2025-11-01: AUTH-AIRGAP-56-002 moved to DOING (Authority Core & Security Guild) – implement airgap audit endpoint and logging. +> 2025-11-01: ISSUER-30-001 marked DONE (Issuer Directory Guild) – Issuer Directory service scaffolded with CRUD APIs, audit sink, CSAF seed import, and unit tests. +> 2025-11-01: ISSUER-30-002 marked DONE (Issuer Directory Guild, Security Guild) – Key management domain, Mongo persistence, CRUD/rotate/revoke endpoints, validation, and tests delivered. +> 2025-11-01: ISSUER-30-004 marked DONE (Issuer Directory Guild, VEX Lens Guild) – Excititor worker consumes issuer directory client for key/trust lookup with cached offline support. +> 2025-11-01: ISSUER-30-005 marked DONE (Issuer Directory Guild, Observability Guild) – Issuer Directory service emits structured logs + metrics for issuer/key flows with OTEL meter. +> 2025-11-02: SURFACE-ENV-01 moved to DOING (Surface Env Guild) – drafting shared environment spec for Scanner/Zastava. +> 2025-11-02: SURFACE-ENV-02 moved to DOING (Surface Env Guild) – implementing typed environment resolver and unit tests. +> 2025-11-02: SURFACE-VAL-01 moved to DOING (Surface Validation Guild) – aligning design document with implementation plan. +> 2025-11-02: SURFACE-FS-01 moved to DOING (Surface FS Guild) – finalising cache layout and manifest spec. +> 2025-11-02: SURFACE-FS-02 moved to DOING (Surface FS Guild) – building core abstractions and deterministic serializers. +> 2025-11-02: SURFACE-SECRETS-01 moved to DOING (Surface Secrets Guild) – updating secrets design for provider matrix. +> 2025-11-02: SURFACE-SECRETS-02 moved to DOING (Surface Secrets Guild) – implementing base providers + tests. +> 2025-11-02: AUTH-POLICY-27-002 marked DONE (Authority Core & Security Guild) – interactive-only policy publish/promote scopes delivered with metadata, fresh-auth enforcement, and audit/docs updates. +> 2025-11-02: SCANNER-ENTRYTRACE-18-506 moved to DOING (EntryTrace Guild, Scanner WebService Guild) – surfacing EntryTrace results via WebService/CLI with confidence metadata. +> 2025-11-02: ATTESTOR-74-001 marked DONE (Attestor Service Guild) – witness client integration, repository schema, and verification/reporting updates landed with tests. +> 2025-11-02: AUTH-OAS-63-001 moved to DOING (Authority Core & Security Guild, API Governance Guild) – verifying legacy `/oauth/*` deprecation signalling and notifications ahead of sunset. +> 2025-11-02: AUTH-OAS-63-001 marked DONE (Authority Core & Security Guild, API Governance Guild) – legacy shims emit Deprecation/Sunset/Warning headers, audit event coverage validated, and migration guide published. +> 2025-11-02: AUTH-NOTIFY-40-001 marked DONE (Authority Core & Security Guild) – `/notify/ack-tokens/rotate` (notify.admin) now rotates DSSE keys with audit trails and integration tests. +> 2025-11-02: AUTH-OAS-62-001 moved to DOING (Authority Core & Security Guild, SDK Generator Guild) – wiring SDK helpers for OAuth2/PAT flows and tenancy override header. +> 2025-11-02: AUTH-OAS-62-001 marked DONE (Authority Core & Security Guild, SDK Generator Guild) – HttpClient auth helper (OAuth2/PAT) shipped with tenant header support and unit tests. +> 2025-11-02: AUTH-OBS-50-001 moved to DOING (Authority Core & Security Guild) – defining observability scopes and updating discovery/offline defaults. +> 2025-11-02: AUTH-OBS-52-001 moved to DOING (Authority Core & Security Guild) – rolling observability scopes through resource server policies and audit wiring. +> 2025-11-02: AUTH-OBS-55-001 marked DONE (Authority Core & Security Guild, Ops Guild) – incident-mode tokens now require fresh auth, audit records expose `incident.reason`, and `/authority/audit/incident` verification path documented. +> 2025-11-02: AUTH-ORCH-34-001 marked DONE (Authority Core & Security Guild) – `orch:backfill` scope enforced with reason/ticket metadata, Authority + CLI updated, docs/config refreshed for Orchestrator admins. +> 2025-11-02: AUTH-PACKS-41-001 moved to DOING (Authority Core & Security Guild) – defining packs scope catalogue, issuer templates, and offline defaults. +> 2025-11-02: AUTH-PACKS-41-001 added shared OpenSSL 1.1 test libs so Authority & Signals Mongo2Go suites run on OpenSSL 3. +> 2025-11-02: AUTH-NOTIFY-42-001 moved to DOING (Authority Core & Security Guild) – investigating `/notify/ack-tokens/rotate` 500 responses when key metadata missing. +> 2025-11-02: AUTH-NOTIFY-42-001 marked DONE (Authority Core & Security Guild) – bootstrap rotate defaults fixed, `StellaOpsBearer` test alias added, and notify ack rotation regression passes. +> 2025-11-03: AUTH-TEN-49-001 marked DONE (Authority Core & Security Guild) – service account delegation (`act` chain) shipped with quota/audit coverage; Authority tests green. +> 2025-11-03: AUTH-VULN-29-003 marked DONE (Authority Core & Docs Guild) – Vuln Explorer security docs, samples, and release notes refreshed for roles, ABAC policies, attachment signing, and ledger verification. +> 2025-11-03: ISSUER-30-003 marked DONE (Issuer Directory Guild, Policy Guild) – trust override APIs/client finalized with cache invalidation/failure-path tests; Issuer Directory suite passing. +> 2025-11-03: AUTH-AIRGAP-56-001/56-002 marked DONE (Authority Core & Security Guild) – air-gap scope catalog surfaced in discovery/OpenAPI and `/authority/audit/airgap` endpoint shipped with tests. +> 2025-11-03: AUTH-PACKS-41-001 marked DONE (Authority Core & Security Guild) – packs scope bundle now emitted via discovery metadata, reflected in OpenAPI, and covered by Authority tests. +> 2025-11-03: AUTH-POLICY-27-003 marked DONE (Authority Core & Docs Guild) – Policy Studio docs/config updated for publish/promote signing workflow, CLI commands, and compliance checklist. +> 2025-11-02: ENTRYTRACE-SURFACE-02 moved to DOING (EntryTrace Guild) – replacing direct env/secret access with Surface.Secrets provider for EntryTrace runs. +> 2025-11-02: ENTRYTRACE-SURFACE-01 marked DONE (EntryTrace Guild) – Surface.Validation + Surface.FS cache now drive EntryTrace reuse with regression tests. +> 2025-11-02: ENTRYTRACE-SURFACE-02 marked DONE (EntryTrace Guild) – EntryTrace environment placeholders resolved via Surface.Secrets with updated docs/tests. +> 2025-11-02: SCANNER-ENTRYTRACE-18-506 marked DONE (EntryTrace Guild, Scanner WebService Guild) – EntryTrace graph surfaced via WebService and CLI with confidence metadata. +> 2025-11-02: SCANNER-ENTRYTRACE-18-509 moved to DOING (EntryTrace Guild, QA Guild) – adding regression coverage for EntryTrace surfaces and NDJSON hashing. +> 2025-11-02: SCANNER-ENTRYTRACE-18-509 marked DONE (EntryTrace Guild, QA Guild) – regression coverage landed for result store/WebService/CLI with NDJSON hashing snapshot. +> 2025-11-02: SCANNER-ENTRYTRACE-18-507 marked DONE (EntryTrace Guild) – fallback candidate discovery now covers history, supervisor configs, service directories, and entrypoint scripts with tests. +> 2025-11-02: SCANNER-ENTRYTRACE-18-508 marked DONE (EntryTrace Guild) – wrapper catalogue expanded for bundle, docker-php-entrypoint, npm, yarn, pipenv, and poetry with wrapper metadata assertions. +> 2025-11-02: CONCELIER-WEB-OAS-61-001 moved to DOING (Concelier WebService Guild) – implementing discovery endpoint for `.well-known/openapi` with version metadata and ETag. +> 2025-11-02: CONCELIER-WEB-OAS-61-001 marked DONE (Concelier WebService Guild) – discovery endpoint now serves signed OpenAPI 3.1 document with ETag support. +> 2025-11-02: DOCS-SCANNER-BENCH-62-001 moved to DOING (Docs Guild, Scanner Guild) – refreshing Trivy/Grype/Snyk comparison docs and ecosystem matrix with source-linked coverage. +> 2025-11-02: DOCS-SCANNER-BENCH-62-001 marked DONE (Docs Guild, Scanner Guild) – matrix updated with Windows/macOS coverage row and secret detection techniques; deep dives cite Trivy/Grype/Snyk sources. +> 2025-11-02: DOCS-SCANNER-BENCH-62-003 added (Docs Guild, Product Guild) – recording Python lockfile/editable-install demand signals for policy guidance follow-up. +> 2025-11-02: DOCS-SCANNER-BENCH-62-004 added (Docs Guild, Java Analyzer Guild) – documenting Java lockfile ingestion plan and policy templates. +> 2025-11-02: DOCS-SCANNER-BENCH-62-005 added (Docs Guild, Go Analyzer Guild) – documenting Go stripped-binary fallback enrichment guidance. +> 2025-11-02: DOCS-SCANNER-BENCH-62-006 added (Docs Guild, Rust Analyzer Guild) – documenting Rust fingerprint enrichment guidance. > 2025-11-02: DOCS-SCANNER-BENCH-62-007 added (Docs Guild, Security Guild) – documenting secret leak detection guidance. > 2025-11-05: DOCS-SCANNER-BENCH-62-007 marked DONE (Docs Guild, Security Guild) – secret leak detection runbook, benchmark updates, and policy templates published. -> 2025-11-02: DOCS-SCANNER-BENCH-62-008 added (Docs Guild, EntryTrace Guild) – documenting EntryTrace heuristic maintenance guidance. -> 2025-11-02: DOCS-SCANNER-BENCH-62-009 added (Docs Guild, Ruby Analyzer Guild) – deepening Ruby gap analysis with detection tables; status set to DOING. -> 2025-11-02: DOCS-SCANNER-BENCH-62-010 added (Docs Guild, PHP Analyzer Guild) – documenting PHP analyzer parity gaps; status set to DOING. -> 2025-11-02: DOCS-SCANNER-BENCH-62-011 added (Docs Guild, Language Analyzer Guild) – capturing Deno runtime gap analysis; status set to DOING. -> 2025-11-02: DOCS-SCANNER-BENCH-62-012 added (Docs Guild, Language Analyzer Guild) – expanding Dart ecosystem comparison; status set to DOING. -> 2025-11-02: DOCS-SCANNER-BENCH-62-013 added (Docs Guild, Swift Analyzer Guild) – expanding Swift coverage analysis; status set to DOING. -> 2025-11-02: DOCS-SCANNER-BENCH-62-014 added (Docs Guild, Runtime Guild) – detailing Kubernetes/VM coverage plan; status set to DOING. -> 2025-11-02: DOCS-SCANNER-BENCH-62-015 added (Docs Guild, Export Center Guild) – outlining DSSE/Rekor operator enablement guidance; status set to DOING. -> 2025-11-02: DOCS-SCANNER-BENCH-62-009 marked DONE (Docs Guild, Ruby Analyzer Guild) – Ruby gap section delivered with detection tables and backlog links. -> 2025-11-02: DOCS-SCANNER-BENCH-62-010 marked DONE (Docs Guild, PHP Analyzer Guild) – PHP gap analysis updated with implementation notes. -> 2025-11-02: DOCS-SCANNER-BENCH-62-011 marked DONE (Docs Guild, Language Analyzer Guild) – Deno plan documented with detection technique table. -> 2025-11-02: DOCS-SCANNER-BENCH-62-012 marked DONE (Docs Guild, Language Analyzer Guild) – Dart coverage section fleshed out with detection strategies. -> 2025-11-02: DOCS-SCANNER-BENCH-62-013 marked DONE (Docs Guild, Swift Analyzer Guild) – Swift analyzer roadmap captured with policy hooks. -> 2025-11-02: DOCS-SCANNER-BENCH-62-014 marked DONE (Docs Guild, Runtime Guild) – Kubernetes/VM alignment section published. +> 2025-11-02: DOCS-SCANNER-BENCH-62-008 added (Docs Guild, EntryTrace Guild) – documenting EntryTrace heuristic maintenance guidance. +> 2025-11-02: DOCS-SCANNER-BENCH-62-009 added (Docs Guild, Ruby Analyzer Guild) – deepening Ruby gap analysis with detection tables; status set to DOING. +> 2025-11-02: DOCS-SCANNER-BENCH-62-010 added (Docs Guild, PHP Analyzer Guild) – documenting PHP analyzer parity gaps; status set to DOING. +> 2025-11-02: DOCS-SCANNER-BENCH-62-011 added (Docs Guild, Language Analyzer Guild) – capturing Deno runtime gap analysis; status set to DOING. +> 2025-11-02: DOCS-SCANNER-BENCH-62-012 added (Docs Guild, Language Analyzer Guild) – expanding Dart ecosystem comparison; status set to DOING. +> 2025-11-02: DOCS-SCANNER-BENCH-62-013 added (Docs Guild, Swift Analyzer Guild) – expanding Swift coverage analysis; status set to DOING. +> 2025-11-02: DOCS-SCANNER-BENCH-62-014 added (Docs Guild, Runtime Guild) – detailing Kubernetes/VM coverage plan; status set to DOING. +> 2025-11-02: DOCS-SCANNER-BENCH-62-015 added (Docs Guild, Export Center Guild) – outlining DSSE/Rekor operator enablement guidance; status set to DOING. +> 2025-11-02: DOCS-SCANNER-BENCH-62-009 marked DONE (Docs Guild, Ruby Analyzer Guild) – Ruby gap section delivered with detection tables and backlog links. +> 2025-11-02: DOCS-SCANNER-BENCH-62-010 marked DONE (Docs Guild, PHP Analyzer Guild) – PHP gap analysis updated with implementation notes. +> 2025-11-02: DOCS-SCANNER-BENCH-62-011 marked DONE (Docs Guild, Language Analyzer Guild) – Deno plan documented with detection technique table. +> 2025-11-02: DOCS-SCANNER-BENCH-62-012 marked DONE (Docs Guild, Language Analyzer Guild) – Dart coverage section fleshed out with detection strategies. +> 2025-11-02: DOCS-SCANNER-BENCH-62-013 marked DONE (Docs Guild, Swift Analyzer Guild) – Swift analyzer roadmap captured with policy hooks. +> 2025-11-02: DOCS-SCANNER-BENCH-62-014 marked DONE (Docs Guild, Runtime Guild) – Kubernetes/VM alignment section published. > 2025-11-02: DOCS-SCANNER-BENCH-62-015 marked DONE (Docs Guild, Export Center Guild) – DSSE/Rekor enablement guidance appended to gap doc. > 2025-11-05: SCANNER-SURFACE-02 marked DONE (Scanner WebService Guild) – WebService now persists `surface` manifest pointers in scan/report APIs, orchestrator samples and DSSE fixtures refreshed, and readiness tests updated with Surface validators stubbed for deterministic health checks. -> 2025-11-02: SCANNER-ENG-0009 moved to DOING (Ruby Analyzer Guild) – drafting Ruby analyzer parity design package. -> 2025-11-02: SCANNER-ENG-0016 added (Ruby Analyzer Guild) – implementing Ruby lock collector & vendor cache ingestion. -> 2025-11-02: SCANNER-ENG-0016 moved to DOING (Ruby Analyzer Guild) – lockfile parser skeleton committed with initial Gemfile.lock parsing. -> 2025-11-02: SCANNER-ENG-0017 added (Ruby Analyzer Guild) – building runtime require/autoload graph builder. -> 2025-11-02: SCANNER-ENG-0018 added (Ruby Analyzer Guild) – emitting Ruby capability and framework signals. -> 2025-11-02: SCANNER-ENG-0019 added (Ruby Analyzer Guild, CLI Guild) – delivering Ruby CLI verbs and Offline Kit packaging. -> 2025-11-02: SCANNER-LIC-0001 added (Scanner Guild, Legal Guild) – vetting tree-sitter Ruby licensing/offline packaging. -> 2025-11-02: SCANNER-LIC-0001 moved to DOING (Scanner Guild, Legal Guild) – SPDX review in progress. -> 2025-11-02: SCANNER-POLICY-0001 added (Policy Guild, Ruby Analyzer Guild) – defining Ruby capability predicates in Policy Engine. -> 2025-11-02: SCANNER-CLI-0001 added (CLI Guild, Ruby Analyzer Guild) – coordinating CLI UX/docs for Ruby verbs. -> 2025-11-02: AIAI-31-011 moved to DOING (Advisory AI Guild) – implementing Excititor VEX document provider. -> 2025-11-02: AIAI-31-011 marked DONE (Advisory AI Guild) – Excititor VEX provider + OpenVEX chunking shipped with tests. -> 2025-11-02: AIAI-31-002 moved to DOING (Advisory AI Guild, SBOM Service Guild) – building SBOM context retriever for timelines/paths/blast radius. -> 2025-11-02: AIAI-31-002 progressing – SBOM context models/tests landed; awaiting SBOM guild client hookup. -> 2025-11-04: AIAI-31-002 marked DONE – SBOM context HTTP client + DI wiring delivered, retriever integrated, HTTP unit tests added. - -> 2025-11-02: AIAI-31-003 moved to DOING – kicking off deterministic tooling (comparators, dependency lookup). First drop covers semver range evaluator + RPM EVR comparator. -> 2025-11-04: AIAI-31-003 marked DONE – deterministic toolset now DI-registered with SBOM context client, added semver/EVR comparison & range tests, and dependency analysis feeds orchestrator metadata. - -> 2025-11-02: AIAI-31-004 moved to DOING – starting deterministic orchestration pipeline (summary/conflict/remediation flow). - -> 2025-11-02: ISSUER-30-006 moved to DOING (Issuer Directory Guild, DevOps Guild) – deployment manifests, backup/restore, secret handling, and offline kit docs in progress. -> 2025-11-04: EVID-OBS-55-001 moved to DOING (Evidence Locker Guild, DevOps Guild) – enabling incident mode retention extension, debug artefacts, and timeline/notifier hooks. -> 2025-11-04: EVID-OBS-55-001 marked DONE (Evidence Locker Guild, DevOps Guild) – incident mode retention, timeline events, notifier stubs, and incident artefact packaging shipped with tests/docs. -> 2025-11-04: EVID-OBS-60-001 moved to DOING (Evidence Locker Guild) – starting sealed-mode portable evidence export flow with redacted bundle packaging and offline verification guidance. -> 2025-11-04: EVID-OBS-60-001 marked DONE (Evidence Locker Guild) – `/evidence/{id}/portable` now emits `portable-bundle-v1.tgz` with sanitized metadata, offline verification script, docs (`docs/airgap/portable-evidence.md`) and unit/web coverage. -> 2025-11-04: DVOFF-64-001 moved to DOING (DevPortal Offline Guild, Exporter Guild) – beginning `devportal --offline` export job bundling portal HTML, specs, SDKs, and changelog assets. +> 2025-11-02: SCANNER-ENG-0009 moved to DOING (Ruby Analyzer Guild) – drafting Ruby analyzer parity design package. +> 2025-11-02: SCANNER-ENG-0016 added (Ruby Analyzer Guild) – implementing Ruby lock collector & vendor cache ingestion. +> 2025-11-02: SCANNER-ENG-0016 moved to DOING (Ruby Analyzer Guild) – lockfile parser skeleton committed with initial Gemfile.lock parsing. +> 2025-11-02: SCANNER-ENG-0017 added (Ruby Analyzer Guild) – building runtime require/autoload graph builder. +> 2025-11-02: SCANNER-ENG-0018 added (Ruby Analyzer Guild) – emitting Ruby capability and framework signals. +> 2025-11-02: SCANNER-ENG-0019 added (Ruby Analyzer Guild, CLI Guild) – delivering Ruby CLI verbs and Offline Kit packaging. +> 2025-11-02: SCANNER-LIC-0001 added (Scanner Guild, Legal Guild) – vetting tree-sitter Ruby licensing/offline packaging. +> 2025-11-02: SCANNER-LIC-0001 moved to DOING (Scanner Guild, Legal Guild) – SPDX review in progress. +> 2025-11-02: SCANNER-POLICY-0001 added (Policy Guild, Ruby Analyzer Guild) – defining Ruby capability predicates in Policy Engine. +> 2025-11-02: SCANNER-CLI-0001 added (CLI Guild, Ruby Analyzer Guild) – coordinating CLI UX/docs for Ruby verbs. +> 2025-11-02: AIAI-31-011 moved to DOING (Advisory AI Guild) – implementing Excititor VEX document provider. +> 2025-11-02: AIAI-31-011 marked DONE (Advisory AI Guild) – Excititor VEX provider + OpenVEX chunking shipped with tests. +> 2025-11-02: AIAI-31-002 moved to DOING (Advisory AI Guild, SBOM Service Guild) – building SBOM context retriever for timelines/paths/blast radius. +> 2025-11-02: AIAI-31-002 progressing – SBOM context models/tests landed; awaiting SBOM guild client hookup. +> 2025-11-04: AIAI-31-002 marked DONE – SBOM context HTTP client + DI wiring delivered, retriever integrated, HTTP unit tests added. + +> 2025-11-02: AIAI-31-003 moved to DOING – kicking off deterministic tooling (comparators, dependency lookup). First drop covers semver range evaluator + RPM EVR comparator. +> 2025-11-04: AIAI-31-003 marked DONE – deterministic toolset now DI-registered with SBOM context client, added semver/EVR comparison & range tests, and dependency analysis feeds orchestrator metadata. + +> 2025-11-02: AIAI-31-004 moved to DOING – starting deterministic orchestration pipeline (summary/conflict/remediation flow). + +> 2025-11-02: ISSUER-30-006 moved to DOING (Issuer Directory Guild, DevOps Guild) – deployment manifests, backup/restore, secret handling, and offline kit docs in progress. +> 2025-11-04: EVID-OBS-55-001 moved to DOING (Evidence Locker Guild, DevOps Guild) – enabling incident mode retention extension, debug artefacts, and timeline/notifier hooks. +> 2025-11-04: EVID-OBS-55-001 marked DONE (Evidence Locker Guild, DevOps Guild) – incident mode retention, timeline events, notifier stubs, and incident artefact packaging shipped with tests/docs. +> 2025-11-04: EVID-OBS-60-001 moved to DOING (Evidence Locker Guild) – starting sealed-mode portable evidence export flow with redacted bundle packaging and offline verification guidance. +> 2025-11-04: EVID-OBS-60-001 marked DONE (Evidence Locker Guild) – `/evidence/{id}/portable` now emits `portable-bundle-v1.tgz` with sanitized metadata, offline verification script, docs (`docs/airgap/portable-evidence.md`) and unit/web coverage. +> 2025-11-04: DVOFF-64-001 moved to DOING (DevPortal Offline Guild, Exporter Guild) – beginning `devportal --offline` export job bundling portal HTML, specs, SDKs, and changelog assets. diff --git a/docs/implplan/SPRINT_110_ingestion_evidence.md b/docs/implplan/SPRINT_110_ingestion_evidence.md index 10c00d90d..1ccb3a7c4 100644 --- a/docs/implplan/SPRINT_110_ingestion_evidence.md +++ b/docs/implplan/SPRINT_110_ingestion_evidence.md @@ -91,7 +91,7 @@ CONCELIER-ATTEST-73-002 `Transparency metadata` | TODO | Ensure Conseiller expos CONCELIER-CONSOLE-23-001 `Advisory aggregation views` | TODO | Expose `/console/advisories` endpoints returning aggregation groups (per linkset) with source chips, provider-reported severity columns (no local consensus), and provenance metadata for Console list + dashboard cards. Support filters by source, ecosystem, published/modified window, tenant enforcement. Dependencies: CONCELIER-LNM-21-201, CONCELIER-LNM-21-202. | Concelier WebService Guild, BE-Base Platform Guild (src/Concelier/StellaOps.Concelier.WebService/TASKS.md) CONCELIER-CONSOLE-23-002 `Dashboard deltas API` | TODO | Provide aggregated advisory delta counts (new, modified, conflicting) for Console dashboard + live status ticker; emit structured events for queue lag metrics. Ensure deterministic counts across repeated queries. Dependencies: CONCELIER-CONSOLE-23-001, CONCELIER-LNM-21-203. | Concelier WebService Guild (src/Concelier/StellaOps.Concelier.WebService/TASKS.md) CONCELIER-CONSOLE-23-003 `Search fan-out helpers` | TODO | Deliver fast lookup endpoints for CVE/GHSA/purl search (linksets, observations) returning evidence fragments for Console global search; implement caching + scope guards. Dependencies: CONCELIER-CONSOLE-23-001. | Concelier WebService Guild (src/Concelier/StellaOps.Concelier.WebService/TASKS.md) -CONCELIER-CORE-AOC-19-004 `Remove ingestion normalization` | DOING (2025-10-28) | Strip normalization/dedup/severity logic from ingestion pipelines, delegate derived computations to Policy Engine, and update exporters/tests to consume raw documents only.
2025-10-29 19:05Z: Audit completed for `AdvisoryRawService`/Mongo repo to confirm alias order/dedup removal persists; identified remaining normalization in observation/linkset factory that will be revised to surface raw duplicates for Policy ingestion. Change sketch + regression matrix drafted under `docs/dev/aoc-normalization-removal-notes.md` (pending commit).
2025-10-31 20:45Z: Added raw linkset projection to observations/storage, exposing canonical+raw views, refreshed fixtures/tests, and documented behaviour in models/doc factory.
2025-10-31 21:10Z: Coordinated with Policy Engine (POLICY-ENGINE-20-003) on adoption timeline; backfill + consumer readiness tracked in `docs/dev/raw-linkset-backfill-plan.md`.
2025-11-05 14:20Z: Resumed work to map remaining normalization hooks tied to Merge service and capture requirements for the upcoming `noMergeEnabled` feature toggle.
2025-11-05 19:05Z: Hardened no-merge feature flag wiring by suppressing obsolete diagnostics and extending gating tests.
2025-11-06 16:10Z: Updated AOC references/backfill plan with raw-vs-canonical guidance and noted analyzer guardrails introduced under MERGE-LNM-21-002. Dependencies: CONCELIER-CORE-AOC-19-002, POLICY-AOC-19-003. | Concelier Core Guild (src/Concelier/__Libraries/StellaOps.Concelier.Core/TASKS.md) +CONCELIER-CORE-AOC-19-004 `Remove ingestion normalization` | DONE (2025-11-06) | Strip normalization/dedup/severity logic from ingestion pipelines, delegate derived computations to Policy Engine, and update exporters/tests to consume raw documents only.
2025-10-29 19:05Z: Audit completed for `AdvisoryRawService`/Mongo repo to confirm alias order/dedup removal persists; identified remaining normalization in observation/linkset factory that will be revised to surface raw duplicates for Policy ingestion. Change sketch + regression matrix drafted under `docs/dev/aoc-normalization-removal-notes.md` (pending commit).
2025-10-31 20:45Z: Added raw linkset projection to observations/storage, exposing canonical+raw views, refreshed fixtures/tests, and documented behaviour in models/doc factory.
2025-10-31 21:10Z: Coordinated with Policy Engine (POLICY-ENGINE-20-003) on adoption timeline; backfill + consumer readiness tracked in `docs/dev/raw-linkset-backfill-plan.md`.
2025-11-05 14:20Z: Resumed work to map remaining normalization hooks tied to Merge service and capture requirements for the upcoming `noMergeEnabled` feature toggle.
2025-11-05 19:05Z: Hardened no-merge feature flag wiring by suppressing obsolete diagnostics and extending gating tests.
2025-11-06 16:10Z: Updated AOC references/backfill plan with raw-vs-canonical guidance and noted analyzer guardrails introduced under MERGE-LNM-21-002.
2025-11-06 23:40Z: Raw observations now flow unaltered (casing + whitespace preserved) with case-insensitive filters/tests refreshed; docs aligned. Tests: `StellaOps.Concelier.Models/Core/Storage.Mongo.Tests` green on .NET 10 preview. Dependencies: CONCELIER-CORE-AOC-19-002, POLICY-AOC-19-003. | Concelier Core Guild (src/Concelier/__Libraries/StellaOps.Concelier.Core/TASKS.md) CONCELIER-CORE-AOC-19-013 `Authority tenant scope smoke coverage` | TODO | Extend Concelier smoke/e2e fixtures to configure `requiredTenants` and assert cross-tenant rejection with updated Authority tokens. Dependencies: AUTH-AOC-19-002. | Concelier Core Guild (src/Concelier/__Libraries/StellaOps.Concelier.Core/TASKS.md) @@ -210,7 +210,7 @@ Depends on: Sprint 110.B - Concelier.VI Summary: Ingestion & Evidence focus on Concelier (phase VII). Task ID | State | Task description | Owners (Source) --- | --- | --- | --- -MERGE-LNM-21-002 | DOING (2025-11-03) | Refactor or retire `AdvisoryMergeService` and related pipelines, ensuring callers transition to observation/linkset APIs; add compile-time analyzer preventing merge service usage.
2025-11-03: Began dependency audit and call-site inventory ahead of deprecation plan; cataloging service registrations/tests referencing merge APIs.
2025-11-05 14:42Z: Drafting `concelier:features:noMergeEnabled` gating, merge job allowlist handling, and deprecation/telemetry changes prior to analyzer rollout.
2025-11-06 16:10Z: Landed analyzer project (`CONCELIER0002`), wired into Concelier WebService/tests, and updated docs to direct suppressions through explicit migration notes. | BE-Merge (src/Concelier/__Libraries/StellaOps.Concelier.Merge/TASKS.md) +MERGE-LNM-21-002 | DOING (2025-11-06) | Refactor or retire `AdvisoryMergeService` and related pipelines, ensuring callers transition to observation/linkset APIs; add compile-time analyzer preventing merge service usage.
2025-11-03: Began dependency audit and call-site inventory ahead of deprecation plan; cataloging service registrations/tests referencing merge APIs.
2025-11-05 14:42Z: Drafted `concelier:features:noMergeEnabled` gating, merge job allowlist handling, and deprecation/telemetry changes prior to analyzer rollout.
2025-11-06 16:10Z: Landed analyzer project (`CONCELIER0002`), wired into Concelier WebService/tests, and updated docs to direct suppressions through explicit migration notes.
2025-11-07 03:25Z: Default-on toggle + job gating break existing Concelier WebService tests; guard/migration adjustments pending before closing the task. | BE-Merge (src/Concelier/__Libraries/StellaOps.Concelier.Merge/TASKS.md) MERGE-LNM-21-003 Determinism/test updates | QA Guild, BE-Merge | Replace merge determinism suites with observation/linkset regression tests verifying no data mutation and conflicts remain visible. Dependencies: MERGE-LNM-21-002. | MERGE-LNM-21-002 (src/Concelier/__Libraries/StellaOps.Concelier.Merge/TASKS.md) diff --git a/docs/implplan/SPRINT_130_scanner_surface.md b/docs/implplan/SPRINT_130_scanner_surface.md index a7685198e..bbf0e4671 100644 --- a/docs/implplan/SPRINT_130_scanner_surface.md +++ b/docs/implplan/SPRINT_130_scanner_surface.md @@ -138,12 +138,12 @@ SCANNER-ENV-01 | TODO (2025-11-06) | Replace ad-hoc environment reads with `Stel SCANNER-ENV-02 | TODO (2025-11-06) | Wire Surface.Env helpers into WebService hosting (cache roots, feature flags) and document configuration. Dependencies: SCANNER-ENV-01.
2025-11-02: WebService bootstrap now consumes Surface.Env helpers for cache roots and feature flag toggles; configuration doc draft pending.
2025-11-05 14:55Z: Picking up configuration/documentation work and aligning API readiness checks with Surface.Env validation outputs.
2025-11-05 19:18Z: Added unit test for Surface.Env cache root binding and ensured configurator registration.
2025-11-06 17:05Z: Surface.Env design doc expanded with warning catalogue and release notes, README refreshed.
2025-11-06 07:45Z: Helm/Compose templates ship `SCANNER_SURFACE_*` defaults across dev/stage/prod/airgap/mirror profiles with rollout guidance in deploy docs.
2025-11-06 07:55Z: Paused; follow-up automation tracked under `DEVOPS-OPENSSL-11-001/002` and readiness tests outstanding. | Scanner WebService Guild, Ops Guild (src/Scanner/StellaOps.Scanner.WebService/TASKS.md) SCANNER-ENV-03 | TODO | Adopt Surface.Env helpers for plugin configuration (cache roots, CAS endpoints, feature toggles). Dependencies: SCANNER-ENV-02. | BuildX Plugin Guild (src/Scanner/StellaOps.Scanner.Sbomer.BuildXPlugin/TASKS.md) SCANNER-EVENTS-16-301 | BLOCKED (2025-10-26) | Emit orchestrator-compatible envelopes (`scanner.event.*`) and update integration tests to verify Notifier ingestion (no Redis queue coupling). | Scanner WebService Guild (src/Scanner/StellaOps.Scanner.WebService/TASKS.md) -SCANNER-EVENTS-16-302 | DOING (2025-10-26) | Extend orchestrator event links (report/policy/attestation) once endpoints are finalised across gateway + console. Dependencies: SCANNER-EVENTS-16-301. | Scanner WebService Guild (src/Scanner/StellaOps.Scanner.WebService/TASKS.md) +SCANNER-EVENTS-16-302 | DONE (2025-11-06) | Extend orchestrator event links (report/policy/attestation) once endpoints are finalised across gateway + console. Dependencies: SCANNER-EVENTS-16-301.
2025-11-06 22:55Z: Dispatcher honours configurable console/API segments; docs and samples refreshed; added regression test for custom segments. `dotnet test` previously blocked by legacy Surface cache ctor signature (tracked under Surface task).
2025-11-06 23:30Z: Report DSSE fixtures re-synced; Surface cache ctor drift repaired; `dotnet test src/Scanner/__Tests/StellaOps.Scanner.WebService.Tests --no-build` now green end-to-end. | Scanner WebService Guild (src/Scanner/StellaOps.Scanner.WebService/TASKS.md) SCANNER-GRAPH-21-001 | TODO | Provide webhook/REST endpoint for Cartographer to request policy overlays and runtime evidence for graph nodes, ensuring determinism and tenant scoping. | Scanner WebService Guild, Cartographer Guild (src/Scanner/StellaOps.Scanner.WebService/TASKS.md) SCANNER-LNM-21-001 | TODO | Update `/reports` and `/policy/runtime` payloads to consume advisory/vex linksets, exposing source severity arrays and conflict summaries alongside effective verdicts. | Scanner WebService Guild, Policy Guild (src/Scanner/StellaOps.Scanner.WebService/TASKS.md) SCANNER-LNM-21-002 | TODO | Add evidence endpoint for Console to fetch linkset summaries with policy overlay for a component/SBOM, including AOC references. Dependencies: SCANNER-LNM-21-001. | Scanner WebService Guild, UI Guild (src/Scanner/StellaOps.Scanner.WebService/TASKS.md) -SCANNER-SECRETS-01 | DOING (2025-11-02) | Adopt `StellaOps.Scanner.Surface.Secrets` for registry/CAS credentials during scan execution.
2025-11-02: Worker integration tests added for CAS token retrieval via Surface.Secrets abstraction; refactor under review. | Scanner Worker Guild, Security Guild (src/Scanner/StellaOps.Scanner.Worker/TASKS.md) -SCANNER-SECRETS-02 | DOING (2025-11-02) | Replace ad-hoc secret wiring with Surface.Secrets for report/export operations (registry and CAS tokens). Dependencies: SCANNER-SECRETS-01.
2025-11-02: WebService export path now resolves registry credentials via Surface.Secrets stub; CI pipeline hook in progress. | Scanner WebService Guild, Security Guild (src/Scanner/StellaOps.Scanner.WebService/TASKS.md) +SCANNER-SECRETS-01 | DOING (2025-11-06) | Adopt `StellaOps.Scanner.Surface.Secrets` for registry/CAS credentials during scan execution.
2025-11-02: Worker integration tests added for CAS token retrieval via Surface.Secrets abstraction; refactor under review.
2025-11-06: Resumed to replace remaining registry credential plumbing and emit rotation-aware metrics.
2025-11-06 21:35Z: Surface secret configurator now hydrates `ScannerStorageOptions` from `cas-access` payloads; unit coverage added. | Scanner Worker Guild, Security Guild (src/Scanner/StellaOps.Scanner.Worker/TASKS.md) +SCANNER-SECRETS-02 | DOING (2025-11-06) | Replace ad-hoc secret wiring with Surface.Secrets for report/export operations (registry and CAS tokens). Dependencies: SCANNER-SECRETS-01.
2025-11-02: WebService export path now resolves registry credentials via Surface.Secrets stub; CI pipeline hook in progress.
2025-11-06: Picking up Surface.Secrets provider usage across report/export flows and removing legacy secret file readers.
2025-11-06 21:40Z: WebService options now consume `cas-access` secrets via configurator; storage mirrors updated; targeted tests passing. | Scanner WebService Guild, Security Guild (src/Scanner/StellaOps.Scanner.WebService/TASKS.md) SCANNER-SECRETS-03 | TODO | Use Surface.Secrets to retrieve registry credentials when interacting with CAS/referrers. Dependencies: SCANNER-SECRETS-02. | BuildX Plugin Guild, Security Guild (src/Scanner/StellaOps.Scanner.Sbomer.BuildXPlugin/TASKS.md) SCANNER-ENG-0020 | TODO | Implement Homebrew collector & fragment mapper per `design/macos-analyzer.md` §3.1. | Scanner Guild (docs/modules/scanner/TASKS.md) SCANNER-ENG-0021 | TODO | Implement pkgutil receipt collector per `design/macos-analyzer.md` §3.2. | Scanner Guild (docs/modules/scanner/TASKS.md) @@ -153,9 +153,9 @@ SCANNER-ENG-0024 | TODO | Implement Windows MSI collector per `design/windows-an SCANNER-ENG-0025 | TODO | Implement WinSxS manifest collector per `design/windows-analyzer.md` §3.2. | Scanner Guild (docs/modules/scanner/TASKS.md) SCANNER-ENG-0026 | TODO | Implement Windows Chocolatey & registry collectors per `design/windows-analyzer.md` §3.3–3.4. | Scanner Guild (docs/modules/scanner/TASKS.md) SCANNER-ENG-0027 | TODO | Deliver Windows policy/offline integration per `design/windows-analyzer.md` §5–6. | Scanner Guild, Policy Guild, Offline Kit Guild (docs/modules/scanner/TASKS.md) -SCANNER-SURFACE-01 | DOING (2025-11-02) | Persist Surface.FS manifests after analyzer stages, including layer CAS metadata and EntryTrace fragments.
2025-11-02: Worker pipeline emitting draft Surface.FS manifests for sample scans; determinism checks running. | Scanner Worker Guild (src/Scanner/StellaOps.Scanner.Worker/TASKS.md) +SCANNER-SURFACE-01 | DOING (2025-11-06) | Persist Surface.FS manifests after analyzer stages, including layer CAS metadata and EntryTrace fragments.
2025-11-02: Worker pipeline emitting draft Surface.FS manifests for sample scans; determinism checks running.
2025-11-06: Continuing with manifest writer abstraction + telemetry wiring for Surface.FS persistence. | Scanner Worker Guild (src/Scanner/StellaOps.Scanner.Worker/TASKS.md) SCANNER-SURFACE-02 | DONE (2025-11-05) | Publish Surface.FS pointers (CAS URIs, manifests) via scan/report APIs and update attestation metadata. Dependencies: SCANNER-SURFACE-01.
2025-11-05: Surface pointer projection wired through WebService endpoints, orchestrator samples & DSSE fixtures refreshed with `surface` manifest block, and regression suite (platform events, report sample, ready check) updated. | Scanner WebService Guild (src/Scanner/StellaOps.Scanner.WebService/TASKS.md) -SCANNER-SURFACE-03 | TODO | Push layer manifests and entry fragments into Surface.FS during build-time SBOM generation. Dependencies: SCANNER-SURFACE-02. | BuildX Plugin Guild (src/Scanner/StellaOps.Scanner.Sbomer.BuildXPlugin/TASKS.md) +SCANNER-SURFACE-03 | DOING (2025-11-06) | Push layer manifests and entry fragments into Surface.FS during build-time SBOM generation. Dependencies: SCANNER-SURFACE-02.
2025-11-06: Starting BuildX manifest upload implementation with Surface.FS client abstraction and integration tests. | BuildX Plugin Guild (src/Scanner/StellaOps.Scanner.Sbomer.BuildXPlugin/TASKS.md) [Scanner & Surface] 130.A) Scanner.VIII Depends on: Sprint 130.A - Scanner.VII diff --git a/docs/implplan/SPRINT_150_scheduling_automation.md b/docs/implplan/SPRINT_150_scheduling_automation.md index 1a3f970da..92f55c8c6 100644 --- a/docs/implplan/SPRINT_150_scheduling_automation.md +++ b/docs/implplan/SPRINT_150_scheduling_automation.md @@ -82,8 +82,10 @@ Task ID | State | Task description | Owners (Source) --- | --- | --- | --- SCHED-CONSOLE-23-001 | DONE (2025-11-03) | Extend runs APIs with live progress SSE endpoints (`/console/runs/{id}/stream`), queue lag summaries, diff metadata fetch, retry/cancel hooks with RBAC enforcement, and deterministic pagination for history views consumed by Console. | Scheduler WebService Guild, BE-Base Platform Guild (src/Scheduler/StellaOps.Scheduler.WebService/TASKS.md) SCHED-CONSOLE-27-001 | DONE (2025-11-03) | Provide policy batch simulation orchestration endpoints (`/policies/simulations` POST/GET) exposing run creation, shard status, SSE progress, cancellation, and retries with RBAC enforcement. Dependencies: SCHED-CONSOLE-23-001. | Scheduler WebService Guild, Policy Registry Guild (src/Scheduler/StellaOps.Scheduler.WebService/TASKS.md) -SCHED-CONSOLE-27-002 | DOING (2025-11-03) | Emit telemetry endpoints/metrics (`policy_simulation_queue_depth`, `policy_simulation_latency`) and webhook callbacks for completion/failure consumed by Registry. Dependencies: SCHED-CONSOLE-27-001. | Scheduler WebService Guild, Observability Guild (src/Scheduler/StellaOps.Scheduler.WebService/TASKS.md) -> 2025-11-06: Tagged `policy_simulation_queue_depth` metrics with tenant identifiers and added unit coverage for the metrics provider snapshot. +SCHED-CONSOLE-27-002 | DONE (2025-11-05) | Emit telemetry endpoints/metrics (`policy_simulation_queue_depth`, `policy_simulation_latency_seconds`) and webhook callbacks for completion/failure consumed by Registry. Dependencies: SCHED-CONSOLE-27-001. | Scheduler WebService Guild, Observability Guild (src/Scheduler/StellaOps.Scheduler.WebService/TASKS.md) +> 2025-11-05: Resumed instrumentation work to match `policy_simulation_latency_seconds` naming, add coverage for SSE latency recording, and validate webhook sample alignment before closing. +> 2025-11-05: Ship telemetry updates + tests; local `dotnet test` blocked by pre-existing GraphJobs accessibility errors (`IGraphJobStore.UpdateAsync`). +> 2025-11-06: Added tenant-aware tagging to `policy_simulation_queue_depth` gauge samples and extended metrics-provider unit coverage. SCHED-IMPACT-16-303 | TODO | Snapshot/compaction + invalidation for removed images; persistence to RocksDB/Redis per architecture. | Scheduler ImpactIndex Guild (src/Scheduler/__Libraries/StellaOps.Scheduler.ImpactIndex/TASKS.md) SCHED-SURFACE-01 | TODO | Evaluate Surface.FS pointers when planning delta scans to avoid redundant work and prioritise drift-triggered assets. | Scheduler Worker Guild (src/Scheduler/__Libraries/StellaOps.Scheduler.Worker/TASKS.md) SCHED-VULN-29-001 | TODO | Expose resolver job APIs (`POST /vuln/resolver/jobs`, `GET /vuln/resolver/jobs/{id}`) to trigger candidate recomputation per artifact/policy change with RBAC and rate limits. | Scheduler WebService Guild, Findings Ledger Guild (src/Scheduler/StellaOps.Scheduler.WebService/TASKS.md) diff --git a/docs/implplan/SPRINT_180_experience_sdks.md b/docs/implplan/SPRINT_180_experience_sdks.md index 3bc0275bd..23ae7e193 100644 --- a/docs/implplan/SPRINT_180_experience_sdks.md +++ b/docs/implplan/SPRINT_180_experience_sdks.md @@ -223,6 +223,7 @@ WEB-AIAI-31-001 `API routing` | TODO | Route `/advisory/ai/*` endpoints through WEB-AIAI-31-002 `Batch orchestration` | TODO | Provide batching job handlers and streaming responses for CLI automation with retry/backoff. Dependencies: WEB-AIAI-31-001. | BE-Base Platform Guild (src/Web/StellaOps.Web/TASKS.md) WEB-AIAI-31-003 `Telemetry & audit` | TODO | Emit metrics/logs (latency, guardrail blocks, validation failures) and forward anonymized prompt hashes to analytics. Dependencies: WEB-AIAI-31-002. | BE-Base Platform Guild, Observability Guild (src/Web/StellaOps.Web/TASKS.md) WEB-AOC-19-001 `Shared AOC guard primitives` | DOING (2025-10-26) | Provide `AOCForbiddenKeys`, guard middleware/interceptor hooks, and error types (`AOCError`, `AOCViolationCode`) for ingestion services. Publish sample usage + analyzer to ensure guard registered. | BE-Base Platform Guild (src/Web/StellaOps.Web/TASKS.md) +> 2025-11-06: Added the `RequireAocGuard` endpoint extension, wired Concelier advisory ingestion through the shared filter, refreshed docs, and introduced extension tests. WEB-AOC-19-002 `Provenance & signature helpers` | TODO | Ship `ProvenanceBuilder`, checksum utilities, and signature verification helper integrated with guard logging. Cover DSSE/CMS formats with unit tests. Dependencies: WEB-AOC-19-001. | BE-Base Platform Guild (src/Web/StellaOps.Web/TASKS.md) WEB-AOC-19-003 `Analyzer + test fixtures` | TODO | Author Roslyn analyzer preventing ingestion modules from writing forbidden keys without guard, and provide shared test fixtures for guard validation used by Concelier/Excititor service tests. Dependencies: WEB-AOC-19-002. | QA Guild, BE-Base Platform Guild (src/Web/StellaOps.Web/TASKS.md) WEB-CONSOLE-23-001 `Global posture endpoints` | TODO | Provide consolidated `/console/dashboard` and `/console/filters` APIs returning tenant-scoped aggregates (findings by severity, VEX override counts, advisory deltas, run health, policy change log). Enforce AOC labelling, deterministic ordering, and cursor-based pagination for drill-down hints. | BE-Base Platform Guild, Product Analytics Guild (src/Web/StellaOps.Web/TASKS.md) diff --git a/docs/migration/no-merge.md b/docs/migration/no-merge.md index 76ca319ef..12b57cfd8 100644 --- a/docs/migration/no-merge.md +++ b/docs/migration/no-merge.md @@ -30,14 +30,15 @@ Do not proceed to Phase 1 until all prerequisites are checked or explicitly wa | Toggle | Default | Purpose | Notes | | --- | --- | --- | --- | -| `concelier:features:noMergeEnabled` | `false` | Master switch to disable legacy Merge job scheduling/execution. | Applies to WebService + Worker; gate `AdvisoryMergeService` DI registration. | +| `concelier:features:noMergeEnabled` | `true` | Master switch to disable legacy Merge job scheduling/execution. | Applies to WebService + Worker; gate `AdvisoryMergeService` DI registration. | | `concelier:features:lnmShadowWrites` | `true` | Enables dual-write of linksets while Merge remains active. | Keep enabled throughout Phase 0–1 to validate parity. | | `concelier:jobs:merge:allowlist` | `[]` | Explicit allowlist for Merge jobs when noMergeEnabled is `false`. | Set to empty during Phase 2+ to prevent accidental restarts. | | `policy:overlays:requireLinksetEvidence` | `false` | Policy engine safety net to require linkset-backed findings. | Flip to `true` only after cutover (Phase 2). | -> 2025-11-05: WebService honours `concelier:features:noMergeEnabled` by skipping Merge DI registration and removing the `merge:reconcile` job definition (MERGE-LNM-21-002). +> 2025-11-06: WebService now defaults `concelier:features:noMergeEnabled` to `true`, skipping Merge DI registration and removing the `merge:reconcile` job unless operators set the flag to `false` and allowlist the job (MERGE-LNM-21-002). > > 2025-11-06: Analyzer `CONCELIER0002` ships with Concelier hosts to block new references to `AdvisoryMergeService` / `AddMergeModule`. Suppressions must be paired with an explicit migration note. +> 2025-11-06: Analyzer coverage validated via unit tests catching object creation, field declarations, `typeof`, and DI extension invocations; merge assemblies remain exempt for legacy cleanup helpers. > **Configuration hygiene:** Document the toggle values per environment in `ops/devops/configuration/staging.md` and `ops/devops/configuration/production.md`. Air-gapped customers receive defaults through the Offline Kit release notes. diff --git a/docs/policy/runs.md b/docs/policy/runs.md index e482d2887..e667acf1f 100644 --- a/docs/policy/runs.md +++ b/docs/policy/runs.md @@ -81,7 +81,7 @@ sequenceDiagram - **Queue** – Backed by Mongo + optional NATS for fan-out; supports leases and replay on crash. - **Engine** – Stateless worker executing the deterministic evaluator. - **Store** – Mongo collections: `policy_runs`, `effective_finding_{policyId}`, `policy_run_events` (append-only history), optional object storage for explain traces. -- **Observability** – Prometheus metrics (`policy_run_seconds`, `policy_simulation_queue_depth`, `policy_simulation_latency`), OTLP traces, structured logs. +- **Observability** – Prometheus metrics (`policy_run_seconds`, `policy_simulation_queue_depth`, `policy_simulation_latency_seconds`), OTLP traces, structured logs. --- diff --git a/local-nuget/Microsoft.Bcl.AsyncInterfaces.8.0.0.nupkg b/local-nuget/Microsoft.Bcl.AsyncInterfaces.8.0.0.nupkg new file mode 100644 index 000000000..f707fc620 Binary files /dev/null and b/local-nuget/Microsoft.Bcl.AsyncInterfaces.8.0.0.nupkg differ diff --git a/local-nuget/NETStandard.Library.2.0.3.nupkg b/local-nuget/NETStandard.Library.2.0.3.nupkg new file mode 100644 index 000000000..224ada250 Binary files /dev/null and b/local-nuget/NETStandard.Library.2.0.3.nupkg differ diff --git a/local-nuget/System.Numerics.Vectors.4.6.0.nupkg b/local-nuget/System.Numerics.Vectors.4.6.0.nupkg new file mode 100644 index 000000000..171c1a22a Binary files /dev/null and b/local-nuget/System.Numerics.Vectors.4.6.0.nupkg differ diff --git a/local-nuget/System.Runtime.CompilerServices.Unsafe.6.1.0.nupkg b/local-nuget/System.Runtime.CompilerServices.Unsafe.6.1.0.nupkg new file mode 100644 index 000000000..35b8c1945 Binary files /dev/null and b/local-nuget/System.Runtime.CompilerServices.Unsafe.6.1.0.nupkg differ diff --git a/samples/api/reports/report-sample.dsse.json b/samples/api/reports/report-sample.dsse.json index a097361e0..0fce7b9e3 100644 --- a/samples/api/reports/report-sample.dsse.json +++ b/samples/api/reports/report-sample.dsse.json @@ -68,7 +68,7 @@ }, "dsse": { "payloadType": "application/vnd.stellaops.report+json", - "payload": "eyJyZXBvcnRJZCI6InJlcG9ydC1hYmMiLCJpbWFnZURpZ2VzdCI6InNoYTI1NjpmZWVkZmFjZSIsImdlbmVyYXRlZEF0IjoiMjAyNS0xMC0xOVQxMjozNDo1NiswMDowMCIsInZlcmRpY3QiOiJibG9ja2VkIiwicG9saWN5Ijp7InJldmlzaW9uSWQiOiJyZXYtNDIiLCJkaWdlc3QiOiJkaWdlc3QtMTIzIn0sInN1bW1hcnkiOnsidG90YWwiOjEsImJsb2NrZWQiOjEsIndhcm5lZCI6MCwiaWdub3JlZCI6MCwicXVpZXRlZCI6MH0sInZlcmRpY3RzIjpbeyJmaW5kaW5nSWQiOiJmaW5kaW5nLTEiLCJyZWFjaGFiaWxpdHkiOiJydW50aW1lIiwic2NvcmUiOjQ3LjUsInNvdXJjZVRydXN0IjoiTlZEIiwic3RhdHVzIjoiQmxvY2tlZCJ9XSwiaXNzdWVzIjpbXSwic3VyZmFjZSI6eyJ0ZW5hbnQiOiJ0ZW5hbnQtYWxwaGEiLCJnZW5lcmF0ZWRBdCI6IjIwMjUtMTAtMTlUMTI6MzQ6NTYrMDA6MDAiLCJtYW5pZmVzdERpZ2VzdCI6InNoYTI1Njo0ZmVlODdkMTg2MjkxZGRmYmJjYzJjNTZjOGVkMGU4Mjg1MjBiOGY1MmUxY2RlMGUxM2JiYTA4MmYxMDkxOGQ3IiwibWFuaWZlc3RVcmkiOiJjYXM6Ly9zY2FubmVyLWFydGlmYWN0cy9zY2FubmVyL3N1cmZhY2UvbWFuaWZlc3RzL3RlbmFudC1hbHBoYS9zaGEyNTYvNGYvZWUvNGZlZTg3ZDE4NjI5MWRkZmJiY2MyYzU2YzhlZDBlODI4NTIwYjhmNTJlMWNkZTBlMTNiYmEwODJmMTA5MThkNy5qc29uIiwibWFuaWZlc3QiOnsic2NoZW1hIjoic3RlbGxhb3BzLnN1cmZhY2UubWFuaWZlc3RAMSIsInRlbmFudCI6InRlbmFudC1hbHBoYSIsImltYWdlRGlnZXN0Ijoic2hhMjU2OmZlZWRmYWNlIiwiZ2VuZXJhdGVkQXQiOiIyMDI1LTEwLTE5VDEyOjM0OjU2KzAwOjAwIiwiYXJ0aWZhY3RzIjpbeyJraW5kIjoiZW50cnktdHJhY2UiLCJ1cmkiOiJjYXM6Ly9zY2FubmVyLWFydGlmYWN0cy9zY2FubmVyL2VudHJ5LXRyYWNlL2YwZjBmMGYwZjBmMGYwZjBmMGYwZjBmMGYwZjBmMGYwZjBmMGYwZjBmMGYwZjBmMGYwZjBmMGYwZjBmMGYwL2VudHJ5LXRyYWNlLmpzb24iLCJkaWdlc3QiOiJzaGEyNTY6ZjBmMGYwZjBmMGYwZjBmMGYwZjBmMGYwZjBmMGYwZjBmMGYwZjBmMGYwZjBmMGYwZjBmMGYwZjBmMGYwZjBmMCIsIm1lZGlhVHlwZSI6ImFwcGxpY2F0aW9uL2pzb24iLCJmb3JtYXQiOiJqc29uIiwic2l6ZUJ5dGVzIjo0MDk2fSx7ImtpbmQiOiJzYm9tLWludmVudG9yeSIsInVyaSI6ImNhczovL3NjYW5uZXItYXJ0aWZhY3RzL3NjYW5uZXIvaW1hZ2VzL2ZlZWRmYWNlL3Nib20uY2R4Lmpzb24iLCJkaWdlc3QiOiJzaGEyNTY6MTExMTExMTExMTExMTExMTExMTExMTExMTExMTExMTExMTExMTExMTExMTExMTExMTExMTExMTExMTExMTExMSIsIm1lZGlhVHlwZSI6ImFwcGxpY2F0aW9uL3ZuZC5jeWNsb25lZHgranNvbjt2ZXJzaW9uPTEuNjt2aWV3PWludmVudG9yeSIsImZvcm1hdCI6ImNkeC1qc29uIiwic2l6ZUJ5dGVzIjoyNDU3NiwidmlldyI6ImludmVudG9yeSJ9LHsia2luZCI6InNib20tdXNhZ2UiLCJ1cmkiOiJjYXM6Ly9zY2FubmVyLWFydGlmYWN0cy9zY2FubmVyL2ltYWdlcy9mZWVkZmFjZS9zYm9tLXVzYWdlLmNkeC5qc29uIiwiZGlnZXN0Ijoic2hhMjU2OjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIiLCJtZWRpYVR5cGUiOiJhcHBsaWNhdGlvbi92bmQuY3ljbG9uZWR4K2pzb247dmVyc2lvbj0xLjY7dmlldz11c2FnZSIsImZvcm1hdCI6ImNkeC1qc29uIiwic2l6ZUJ5dGVzIjoxNjM4NCwidmlldyI6InVzYWdlIn1dfX19", + "payload": "eyJyZXBvcnRJZCI6InJlcG9ydC1hYmMiLCJpbWFnZURpZ2VzdCI6InNoYTI1NjpmZWVkZmFjZSIsImdlbmVyYXRlZEF0IjoiMjAyNS0xMC0xOVQxMjozNDo1NiswMDowMCIsInZlcmRpY3QiOiJibG9ja2VkIiwicG9saWN5Ijp7InJldmlzaW9uSWQiOiJyZXYtNDIiLCJkaWdlc3QiOiJkaWdlc3QtMTIzIn0sInN1bW1hcnkiOnsidG90YWwiOjEsImJsb2NrZWQiOjEsIndhcm5lZCI6MCwiaWdub3JlZCI6MCwicXVpZXRlZCI6MH0sInZlcmRpY3RzIjpbeyJmaW5kaW5nSWQiOiJmaW5kaW5nLTEiLCJzdGF0dXMiOiJCbG9ja2VkIiwic2NvcmUiOjQ3LjUsInNvdXJjZVRydXN0IjoiTlZEIiwicmVhY2hhYmlsaXR5IjoicnVudGltZSJ9XSwiaXNzdWVzIjpbXSwic3VyZmFjZSI6eyJ0ZW5hbnQiOiJ0ZW5hbnQtYWxwaGEiLCJnZW5lcmF0ZWRBdCI6IjIwMjUtMTAtMTlUMTI6MzQ6NTYrMDA6MDAiLCJtYW5pZmVzdERpZ2VzdCI6InNoYTI1Njo0ZmVlODdkMTg2MjkxZGRmYmJjYzJjNTZjOGVkMGU4Mjg1MjBiOGY1MmUxY2RlMGUxM2JiYTA4MmYxMDkxOGQ3IiwibWFuaWZlc3RVcmkiOiJjYXM6Ly9zY2FubmVyLWFydGlmYWN0cy9zY2FubmVyL3N1cmZhY2UvbWFuaWZlc3RzL3RlbmFudC1hbHBoYS9zaGEyNTYvNGYvZWUvNGZlZTg3ZDE4NjI5MWRkZmJiY2MyYzU2YzhlZDBlODI4NTIwYjhmNTJlMWNkZTBlMTNiYmEwODJmMTA5MThkNy5qc29uIiwibWFuaWZlc3QiOnsic2NoZW1hIjoic3RlbGxhb3BzLnN1cmZhY2UubWFuaWZlc3RAMSIsInRlbmFudCI6InRlbmFudC1hbHBoYSIsImltYWdlRGlnZXN0Ijoic2hhMjU2OmZlZWRmYWNlIiwiZ2VuZXJhdGVkQXQiOiIyMDI1LTEwLTE5VDEyOjM0OjU2KzAwOjAwIiwiYXJ0aWZhY3RzIjpbeyJraW5kIjoiZW50cnktdHJhY2UiLCJ1cmkiOiJjYXM6Ly9zY2FubmVyLWFydGlmYWN0cy9zY2FubmVyL2VudHJ5LXRyYWNlL2YwZjBmMGYwZjBmMGYwZjBmMGYwZjBmMGYwZjBmMGYwZjBmMGYwZjBmMGYwZjBmMGYwZjBmMGYwZjBmMGYwL2VudHJ5LXRyYWNlLmpzb24iLCJkaWdlc3QiOiJzaGEyNTY6ZjBmMGYwZjBmMGYwZjBmMGYwZjBmMGYwZjBmMGYwZjBmMGYwZjBmMGYwZjBmMGYwZjBmMGYwZjBmMGYwZjBmMCIsIm1lZGlhVHlwZSI6ImFwcGxpY2F0aW9uL2pzb24iLCJmb3JtYXQiOiJqc29uIiwic2l6ZUJ5dGVzIjo0MDk2fSx7ImtpbmQiOiJzYm9tLWludmVudG9yeSIsInVyaSI6ImNhczovL3NjYW5uZXItYXJ0aWZhY3RzL3NjYW5uZXIvaW1hZ2VzL2ZlZWRmYWNlL3Nib20uY2R4Lmpzb24iLCJkaWdlc3QiOiJzaGEyNTY6MTExMTExMTExMTExMTExMTExMTExMTExMTExMTExMTExMTExMTExMTExMTExMTExMTExMTExMTExMTExMTExMSIsIm1lZGlhVHlwZSI6ImFwcGxpY2F0aW9uL3ZuZC5jeWNsb25lZHhcdTAwMkJqc29uO3ZlcnNpb249MS42O3ZpZXc9aW52ZW50b3J5IiwiZm9ybWF0IjoiY2R4LWpzb24iLCJzaXplQnl0ZXMiOjI0NTc2LCJ2aWV3IjoiaW52ZW50b3J5In0seyJraW5kIjoic2JvbS11c2FnZSIsInVyaSI6ImNhczovL3NjYW5uZXItYXJ0aWZhY3RzL3NjYW5uZXIvaW1hZ2VzL2ZlZWRmYWNlL3Nib20tdXNhZ2UuY2R4Lmpzb24iLCJkaWdlc3QiOiJzaGEyNTY6MjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMiIsIm1lZGlhVHlwZSI6ImFwcGxpY2F0aW9uL3ZuZC5jeWNsb25lZHhcdTAwMkJqc29uO3ZlcnNpb249MS42O3ZpZXc9dXNhZ2UiLCJmb3JtYXQiOiJjZHgtanNvbiIsInNpemVCeXRlcyI6MTYzODQsInZpZXciOiJ1c2FnZSJ9XX19fQ==", "signatures": [ { "keyId": "test-key", diff --git a/src/Aoc/__Libraries/StellaOps.Aoc.AspNetCore/Routing/AocGuardEndpointFilter.cs b/src/Aoc/__Libraries/StellaOps.Aoc.AspNetCore/Routing/AocGuardEndpointFilter.cs index 0d4a5527a..1a74942ed 100644 --- a/src/Aoc/__Libraries/StellaOps.Aoc.AspNetCore/Routing/AocGuardEndpointFilter.cs +++ b/src/Aoc/__Libraries/StellaOps.Aoc.AspNetCore/Routing/AocGuardEndpointFilter.cs @@ -7,6 +7,7 @@ using Microsoft.AspNetCore.Routing; using Microsoft.Extensions.DependencyInjection; using Microsoft.Extensions.Options; using StellaOps.Aoc; +using StellaOps.Aoc.AspNetCore.Results; namespace StellaOps.Aoc.AspNetCore.Routing; @@ -55,7 +56,14 @@ public sealed class AocGuardEndpointFilter : IEndpointFilter _ => JsonSerializer.SerializeToElement(payload, _serializerOptions) }; - guard.ValidateOrThrow(element, options); + try + { + guard.ValidateOrThrow(element, options); + } + catch (AocGuardException exception) + { + return AocHttpResults.Problem(context.HttpContext, exception); + } } } } diff --git a/src/Aoc/__Libraries/StellaOps.Aoc.AspNetCore/Routing/AocGuardEndpointFilterExtensions.cs b/src/Aoc/__Libraries/StellaOps.Aoc.AspNetCore/Routing/AocGuardEndpointFilterExtensions.cs new file mode 100644 index 000000000..86d7921bb --- /dev/null +++ b/src/Aoc/__Libraries/StellaOps.Aoc.AspNetCore/Routing/AocGuardEndpointFilterExtensions.cs @@ -0,0 +1,60 @@ +using System; +using System.Collections.Generic; +using System.Text.Json; +using Microsoft.AspNetCore.Builder; + +namespace StellaOps.Aoc.AspNetCore.Routing; + +public static class AocGuardEndpointFilterExtensions +{ + public static RouteHandlerBuilder RequireAocGuard( + this RouteHandlerBuilder builder, + Func> payloadSelector, + JsonSerializerOptions? serializerOptions = null, + AocGuardOptions? guardOptions = null) + { + if (builder is null) + { + throw new ArgumentNullException(nameof(builder)); + } + + if (payloadSelector is null) + { + throw new ArgumentNullException(nameof(payloadSelector)); + } + + builder.Add(endpointBuilder => + { + endpointBuilder.FilterFactories.Add((routeContext, next) => + { + var filter = new AocGuardEndpointFilter(payloadSelector, serializerOptions, guardOptions); + return invocationContext => filter.InvokeAsync(invocationContext, next); + }); + }); + return builder; + } + + public static RouteHandlerBuilder RequireAocGuard( + this RouteHandlerBuilder builder, + Func payloadSelector, + JsonSerializerOptions? serializerOptions = null, + AocGuardOptions? guardOptions = null) + { + if (payloadSelector is null) + { + throw new ArgumentNullException(nameof(payloadSelector)); + } + + return AocGuardEndpointFilterExtensions.RequireAocGuard( + builder, + request => + { + var payload = payloadSelector(request); + return payload is null + ? Array.Empty() + : new object?[] { payload }; + }, + serializerOptions, + guardOptions); + } +} diff --git a/src/Aoc/__Tests/StellaOps.Aoc.AspNetCore.Tests/AocGuardEndpointFilterExtensionsTests.cs b/src/Aoc/__Tests/StellaOps.Aoc.AspNetCore.Tests/AocGuardEndpointFilterExtensionsTests.cs new file mode 100644 index 000000000..b6a061a26 --- /dev/null +++ b/src/Aoc/__Tests/StellaOps.Aoc.AspNetCore.Tests/AocGuardEndpointFilterExtensionsTests.cs @@ -0,0 +1,52 @@ +using System; +using System.Text.Json; +using Microsoft.AspNetCore.Builder; +using Microsoft.AspNetCore.Http; +using Microsoft.Extensions.DependencyInjection; +using StellaOps.Aoc.AspNetCore.Routing; + +namespace StellaOps.Aoc.AspNetCore.Tests; + +public sealed class AocGuardEndpointFilterExtensionsTests +{ + [Fact] + public void RequireAocGuard_ReturnsBuilderInstance() + { + var builder = WebApplication.CreateBuilder(); + builder.Services.AddAocGuard(); + using var app = builder.Build(); + + var route = app.MapPost("/guard", (GuardPayload _) => TypedResults.Ok()); + + var result = route.RequireAocGuard(_ => Array.Empty()); + + Assert.Same(route, result); + } + + [Fact] + public void RequireAocGuard_WithNullBuilder_Throws() + { + RouteHandlerBuilder? builder = null; + + Assert.Throws(() => + AocGuardEndpointFilterExtensions.RequireAocGuard( + builder!, + _ => Array.Empty())); + } + + [Fact] + public void RequireAocGuard_WithObjectSelector_UsesOverload() + { + var builder = WebApplication.CreateBuilder(); + builder.Services.AddAocGuard(); + using var app = builder.Build(); + + var route = app.MapPost("/guard-object", (GuardPayload _) => TypedResults.Ok()); + + var result = route.RequireAocGuard(_ => new GuardPayload(JsonDocument.Parse("{}").RootElement)); + + Assert.Same(route, result); + } + + private sealed record GuardPayload(JsonElement Payload); +} diff --git a/src/Concelier/StellaOps.Concelier.WebService/Extensions/JobRegistrationExtensions.cs b/src/Concelier/StellaOps.Concelier.WebService/Extensions/JobRegistrationExtensions.cs index 271e152d8..febbdaa64 100644 --- a/src/Concelier/StellaOps.Concelier.WebService/Extensions/JobRegistrationExtensions.cs +++ b/src/Concelier/StellaOps.Concelier.WebService/Extensions/JobRegistrationExtensions.cs @@ -1,100 +1,134 @@ -using System; -using System.Collections.Generic; -using System.Globalization; -using Microsoft.Extensions.DependencyInjection; -using Microsoft.Extensions.Options; -using StellaOps.Concelier.Core.Jobs; -using StellaOps.Concelier.Merge.Jobs; - -namespace StellaOps.Concelier.WebService.Extensions; - -internal static class JobRegistrationExtensions -{ - private sealed record BuiltInJob( - string Kind, - string JobType, - string AssemblyName, - TimeSpan Timeout, - TimeSpan LeaseDuration, - string? CronExpression = null); - - private static readonly IReadOnlyList BuiltInJobs = new List - { - new("source:redhat:fetch", "StellaOps.Concelier.Connector.Distro.RedHat.RedHatFetchJob", "StellaOps.Concelier.Connector.Distro.RedHat", TimeSpan.FromMinutes(12), TimeSpan.FromMinutes(6), "0,15,30,45 * * * *"), - new("source:redhat:parse", "StellaOps.Concelier.Connector.Distro.RedHat.RedHatParseJob", "StellaOps.Concelier.Connector.Distro.RedHat", TimeSpan.FromMinutes(15), TimeSpan.FromMinutes(6), "5,20,35,50 * * * *"), - new("source:redhat:map", "StellaOps.Concelier.Connector.Distro.RedHat.RedHatMapJob", "StellaOps.Concelier.Connector.Distro.RedHat", TimeSpan.FromMinutes(20), TimeSpan.FromMinutes(6), "10,25,40,55 * * * *"), - - new("source:cert-in:fetch", "StellaOps.Concelier.Connector.CertIn.CertInFetchJob", "StellaOps.Concelier.Connector.CertIn", TimeSpan.FromMinutes(15), TimeSpan.FromMinutes(5)), - new("source:cert-in:parse", "StellaOps.Concelier.Connector.CertIn.CertInParseJob", "StellaOps.Concelier.Connector.CertIn", TimeSpan.FromMinutes(15), TimeSpan.FromMinutes(5)), - new("source:cert-in:map", "StellaOps.Concelier.Connector.CertIn.CertInMapJob", "StellaOps.Concelier.Connector.CertIn", TimeSpan.FromMinutes(15), TimeSpan.FromMinutes(5)), - - new("source:cert-fr:fetch", "StellaOps.Concelier.Connector.CertFr.CertFrFetchJob", "StellaOps.Concelier.Connector.CertFr", TimeSpan.FromMinutes(15), TimeSpan.FromMinutes(5)), - new("source:cert-fr:parse", "StellaOps.Concelier.Connector.CertFr.CertFrParseJob", "StellaOps.Concelier.Connector.CertFr", TimeSpan.FromMinutes(15), TimeSpan.FromMinutes(5)), - new("source:cert-fr:map", "StellaOps.Concelier.Connector.CertFr.CertFrMapJob", "StellaOps.Concelier.Connector.CertFr", TimeSpan.FromMinutes(15), TimeSpan.FromMinutes(5)), - - new("source:jvn:fetch", "StellaOps.Concelier.Connector.Jvn.JvnFetchJob", "StellaOps.Concelier.Connector.Jvn", TimeSpan.FromMinutes(15), TimeSpan.FromMinutes(5)), - new("source:jvn:parse", "StellaOps.Concelier.Connector.Jvn.JvnParseJob", "StellaOps.Concelier.Connector.Jvn", TimeSpan.FromMinutes(15), TimeSpan.FromMinutes(5)), - new("source:jvn:map", "StellaOps.Concelier.Connector.Jvn.JvnMapJob", "StellaOps.Concelier.Connector.Jvn", TimeSpan.FromMinutes(15), TimeSpan.FromMinutes(5)), - - new("source:ics-kaspersky:fetch", "StellaOps.Concelier.Connector.Ics.Kaspersky.KasperskyFetchJob", "StellaOps.Concelier.Connector.Ics.Kaspersky", TimeSpan.FromMinutes(15), TimeSpan.FromMinutes(5)), - new("source:ics-kaspersky:parse", "StellaOps.Concelier.Connector.Ics.Kaspersky.KasperskyParseJob", "StellaOps.Concelier.Connector.Ics.Kaspersky", TimeSpan.FromMinutes(15), TimeSpan.FromMinutes(5)), - new("source:ics-kaspersky:map", "StellaOps.Concelier.Connector.Ics.Kaspersky.KasperskyMapJob", "StellaOps.Concelier.Connector.Ics.Kaspersky", TimeSpan.FromMinutes(15), TimeSpan.FromMinutes(5)), - - new("source:osv:fetch", "StellaOps.Concelier.Connector.Osv.OsvFetchJob", "StellaOps.Concelier.Connector.Osv", TimeSpan.FromMinutes(15), TimeSpan.FromMinutes(5)), - new("source:osv:parse", "StellaOps.Concelier.Connector.Osv.OsvParseJob", "StellaOps.Concelier.Connector.Osv", TimeSpan.FromMinutes(15), TimeSpan.FromMinutes(5)), - new("source:osv:map", "StellaOps.Concelier.Connector.Osv.OsvMapJob", "StellaOps.Concelier.Connector.Osv", TimeSpan.FromMinutes(15), TimeSpan.FromMinutes(5)), - - new("source:vmware:fetch", "StellaOps.Concelier.Connector.Vndr.Vmware.VmwareFetchJob", "StellaOps.Concelier.Connector.Vndr.Vmware", TimeSpan.FromMinutes(15), TimeSpan.FromMinutes(5)), - new("source:vmware:parse", "StellaOps.Concelier.Connector.Vndr.Vmware.VmwareParseJob", "StellaOps.Concelier.Connector.Vndr.Vmware", TimeSpan.FromMinutes(15), TimeSpan.FromMinutes(5)), - new("source:vmware:map", "StellaOps.Concelier.Connector.Vndr.Vmware.VmwareMapJob", "StellaOps.Concelier.Connector.Vndr.Vmware", TimeSpan.FromMinutes(15), TimeSpan.FromMinutes(5)), - - new("source:vndr-oracle:fetch", "StellaOps.Concelier.Connector.Vndr.Oracle.OracleFetchJob", "StellaOps.Concelier.Connector.Vndr.Oracle", TimeSpan.FromMinutes(15), TimeSpan.FromMinutes(5)), - new("source:vndr-oracle:parse", "StellaOps.Concelier.Connector.Vndr.Oracle.OracleParseJob", "StellaOps.Concelier.Connector.Vndr.Oracle", TimeSpan.FromMinutes(15), TimeSpan.FromMinutes(5)), - new("source:vndr-oracle:map", "StellaOps.Concelier.Connector.Vndr.Oracle.OracleMapJob", "StellaOps.Concelier.Connector.Vndr.Oracle", TimeSpan.FromMinutes(15), TimeSpan.FromMinutes(5)), - +using System; +using System.Collections.Generic; +using Microsoft.Extensions.Configuration; +using Microsoft.Extensions.DependencyInjection; +using Microsoft.Extensions.Options; +using StellaOps.Concelier.Core.Jobs; + +namespace StellaOps.Concelier.WebService.Extensions; + +internal static class JobRegistrationExtensions +{ + private sealed record BuiltInJob( + string Kind, + string JobType, + string AssemblyName, + TimeSpan Timeout, + TimeSpan LeaseDuration, + string? CronExpression = null); + + private static readonly IReadOnlyList BaseBuiltInJobs = new List + { + new("source:redhat:fetch", "StellaOps.Concelier.Connector.Distro.RedHat.RedHatFetchJob", "StellaOps.Concelier.Connector.Distro.RedHat", TimeSpan.FromMinutes(12), TimeSpan.FromMinutes(6), "0,15,30,45 * * * *"), + new("source:redhat:parse", "StellaOps.Concelier.Connector.Distro.RedHat.RedHatParseJob", "StellaOps.Concelier.Connector.Distro.RedHat", TimeSpan.FromMinutes(15), TimeSpan.FromMinutes(6), "5,20,35,50 * * * *"), + new("source:redhat:map", "StellaOps.Concelier.Connector.Distro.RedHat.RedHatMapJob", "StellaOps.Concelier.Connector.Distro.RedHat", TimeSpan.FromMinutes(20), TimeSpan.FromMinutes(6), "10,25,40,55 * * * *"), + + new("source:cert-in:fetch", "StellaOps.Concelier.Connector.CertIn.CertInFetchJob", "StellaOps.Concelier.Connector.CertIn", TimeSpan.FromMinutes(15), TimeSpan.FromMinutes(5)), + new("source:cert-in:parse", "StellaOps.Concelier.Connector.CertIn.CertInParseJob", "StellaOps.Concelier.Connector.CertIn", TimeSpan.FromMinutes(15), TimeSpan.FromMinutes(5)), + new("source:cert-in:map", "StellaOps.Concelier.Connector.CertIn.CertInMapJob", "StellaOps.Concelier.Connector.CertIn", TimeSpan.FromMinutes(15), TimeSpan.FromMinutes(5)), + + new("source:cert-fr:fetch", "StellaOps.Concelier.Connector.CertFr.CertFrFetchJob", "StellaOps.Concelier.Connector.CertFr", TimeSpan.FromMinutes(15), TimeSpan.FromMinutes(5)), + new("source:cert-fr:parse", "StellaOps.Concelier.Connector.CertFr.CertFrParseJob", "StellaOps.Concelier.Connector.CertFr", TimeSpan.FromMinutes(15), TimeSpan.FromMinutes(5)), + new("source:cert-fr:map", "StellaOps.Concelier.Connector.CertFr.CertFrMapJob", "StellaOps.Concelier.Connector.CertFr", TimeSpan.FromMinutes(15), TimeSpan.FromMinutes(5)), + + new("source:jvn:fetch", "StellaOps.Concelier.Connector.Jvn.JvnFetchJob", "StellaOps.Concelier.Connector.Jvn", TimeSpan.FromMinutes(15), TimeSpan.FromMinutes(5)), + new("source:jvn:parse", "StellaOps.Concelier.Connector.Jvn.JvnParseJob", "StellaOps.Concelier.Connector.Jvn", TimeSpan.FromMinutes(15), TimeSpan.FromMinutes(5)), + new("source:jvn:map", "StellaOps.Concelier.Connector.Jvn.JvnMapJob", "StellaOps.Concelier.Connector.Jvn", TimeSpan.FromMinutes(15), TimeSpan.FromMinutes(5)), + + new("source:ics-kaspersky:fetch", "StellaOps.Concelier.Connector.Ics.Kaspersky.KasperskyFetchJob", "StellaOps.Concelier.Connector.Ics.Kaspersky", TimeSpan.FromMinutes(15), TimeSpan.FromMinutes(5)), + new("source:ics-kaspersky:parse", "StellaOps.Concelier.Connector.Ics.Kaspersky.KasperskyParseJob", "StellaOps.Concelier.Connector.Ics.Kaspersky", TimeSpan.FromMinutes(15), TimeSpan.FromMinutes(5)), + new("source:ics-kaspersky:map", "StellaOps.Concelier.Connector.Ics.Kaspersky.KasperskyMapJob", "StellaOps.Concelier.Connector.Ics.Kaspersky", TimeSpan.FromMinutes(15), TimeSpan.FromMinutes(5)), + + new("source:osv:fetch", "StellaOps.Concelier.Connector.Osv.OsvFetchJob", "StellaOps.Concelier.Connector.Osv", TimeSpan.FromMinutes(15), TimeSpan.FromMinutes(5)), + new("source:osv:parse", "StellaOps.Concelier.Connector.Osv.OsvParseJob", "StellaOps.Concelier.Connector.Osv", TimeSpan.FromMinutes(15), TimeSpan.FromMinutes(5)), + new("source:osv:map", "StellaOps.Concelier.Connector.Osv.OsvMapJob", "StellaOps.Concelier.Connector.Osv", TimeSpan.FromMinutes(15), TimeSpan.FromMinutes(5)), + + new("source:vmware:fetch", "StellaOps.Concelier.Connector.Vndr.Vmware.VmwareFetchJob", "StellaOps.Concelier.Connector.Vndr.Vmware", TimeSpan.FromMinutes(15), TimeSpan.FromMinutes(5)), + new("source:vmware:parse", "StellaOps.Concelier.Connector.Vndr.Vmware.VmwareParseJob", "StellaOps.Concelier.Connector.Vndr.Vmware", TimeSpan.FromMinutes(15), TimeSpan.FromMinutes(5)), + new("source:vmware:map", "StellaOps.Concelier.Connector.Vndr.Vmware.VmwareMapJob", "StellaOps.Concelier.Connector.Vndr.Vmware", TimeSpan.FromMinutes(15), TimeSpan.FromMinutes(5)), + + new("source:vndr-oracle:fetch", "StellaOps.Concelier.Connector.Vndr.Oracle.OracleFetchJob", "StellaOps.Concelier.Connector.Vndr.Oracle", TimeSpan.FromMinutes(15), TimeSpan.FromMinutes(5)), + new("source:vndr-oracle:parse", "StellaOps.Concelier.Connector.Vndr.Oracle.OracleParseJob", "StellaOps.Concelier.Connector.Vndr.Oracle", TimeSpan.FromMinutes(15), TimeSpan.FromMinutes(5)), + new("source:vndr-oracle:map", "StellaOps.Concelier.Connector.Vndr.Oracle.OracleMapJob", "StellaOps.Concelier.Connector.Vndr.Oracle", TimeSpan.FromMinutes(15), TimeSpan.FromMinutes(5)), + new("export:json", "StellaOps.Concelier.Exporter.Json.JsonExportJob", "StellaOps.Concelier.Exporter.Json", TimeSpan.FromMinutes(10), TimeSpan.FromMinutes(5)), - new("export:trivy-db", "StellaOps.Concelier.Exporter.TrivyDb.TrivyDbExportJob", "StellaOps.Concelier.Exporter.TrivyDb", TimeSpan.FromMinutes(20), TimeSpan.FromMinutes(10)), -#pragma warning disable CS0618, CONCELIER0001 // Legacy merge job remains available until MERGE-LNM-21-002 completes. - new("merge:reconcile", "StellaOps.Concelier.Merge.Jobs.MergeReconcileJob", "StellaOps.Concelier.Merge", TimeSpan.FromMinutes(15), TimeSpan.FromMinutes(5)) -#pragma warning restore CS0618, CONCELIER0001 + new("export:trivy-db", "StellaOps.Concelier.Exporter.TrivyDb.TrivyDbExportJob", "StellaOps.Concelier.Exporter.TrivyDb", TimeSpan.FromMinutes(20), TimeSpan.FromMinutes(10)) }; - - public static IServiceCollection AddBuiltInConcelierJobs(this IServiceCollection services) - { - ArgumentNullException.ThrowIfNull(services); - - services.PostConfigure(options => - { - foreach (var registration in BuiltInJobs) - { - if (options.Definitions.ContainsKey(registration.Kind)) - { - continue; - } - - var jobType = Type.GetType( - $"{registration.JobType}, {registration.AssemblyName}", - throwOnError: false, - ignoreCase: false); - - if (jobType is null) - { - continue; - } - - var timeout = registration.Timeout > TimeSpan.Zero ? registration.Timeout : options.DefaultTimeout; - var lease = registration.LeaseDuration > TimeSpan.Zero ? registration.LeaseDuration : options.DefaultLeaseDuration; - - options.Definitions[registration.Kind] = new JobDefinition( - registration.Kind, - jobType, - timeout, - lease, - registration.CronExpression, - Enabled: true); - } - }); - - return services; - } -} + + private static readonly BuiltInJob MergeReconcileBuiltInJob = new( + "merge:reconcile", + "StellaOps.Concelier.Merge.Jobs.MergeReconcileJob", + "StellaOps.Concelier.Merge", + TimeSpan.FromMinutes(15), + TimeSpan.FromMinutes(5)); + + public static IServiceCollection AddBuiltInConcelierJobs(this IServiceCollection services) + { + ArgumentNullException.ThrowIfNull(services); + + services.AddOptions() + .Configure((options, configuration) => + { + foreach (var registration in BaseBuiltInJobs) + { + AddJobIfMissing(options, registration); + } + + ConfigureMergeJob(options, configuration); + }); + + return services; + } + + private static void AddJobIfMissing(JobSchedulerOptions options, BuiltInJob registration) + { + if (options.Definitions.ContainsKey(registration.Kind)) + { + return; + } + + var jobType = Type.GetType( + $"{registration.JobType}, {registration.AssemblyName}", + throwOnError: false, + ignoreCase: false); + + if (jobType is null) + { + return; + } + + var timeout = registration.Timeout > TimeSpan.Zero ? registration.Timeout : options.DefaultTimeout; + var lease = registration.LeaseDuration > TimeSpan.Zero ? registration.LeaseDuration : options.DefaultLeaseDuration; + + options.Definitions[registration.Kind] = new JobDefinition( + registration.Kind, + jobType, + timeout, + lease, + registration.CronExpression, + Enabled: true); + } + + private static void ConfigureMergeJob(JobSchedulerOptions options, IConfiguration configuration) + { + var noMergeEnabled = configuration.GetValue("concelier:features:noMergeEnabled") ?? true; + if (noMergeEnabled) + { + options.Definitions.Remove(MergeReconcileBuiltInJob.Kind); + return; + } + + var allowlist = configuration.GetSection("concelier:jobs:merge:allowlist").Get(); + if (allowlist is { Length: > 0 }) + { + var allowlistSet = new HashSet(allowlist, StringComparer.OrdinalIgnoreCase); + if (!allowlistSet.Contains(MergeReconcileBuiltInJob.Kind)) + { + options.Definitions.Remove(MergeReconcileBuiltInJob.Kind); + return; + } + } + + AddJobIfMissing(options, MergeReconcileBuiltInJob); + } +} diff --git a/src/Concelier/StellaOps.Concelier.WebService/Program.cs b/src/Concelier/StellaOps.Concelier.WebService/Program.cs index 68f94a12d..0a42a322d 100644 --- a/src/Concelier/StellaOps.Concelier.WebService/Program.cs +++ b/src/Concelier/StellaOps.Concelier.WebService/Program.cs @@ -11,11 +11,11 @@ using Microsoft.AspNetCore.Mvc; using Microsoft.Extensions.DependencyInjection; using Microsoft.Extensions.DependencyInjection.Extensions; using Microsoft.Extensions.Hosting; -using System.Diagnostics; -using System.Text.Json; -using System.Text.Json.Serialization; -using Microsoft.Extensions.Logging; -using Microsoft.Extensions.Options; +using System.Diagnostics; +using System.Text.Json; +using System.Text.Json.Serialization; +using Microsoft.Extensions.Logging; +using Microsoft.Extensions.Options; using MongoDB.Bson; using MongoDB.Driver; using StellaOps.Concelier.Core.Events; @@ -40,6 +40,7 @@ using StellaOps.Auth.Abstractions; using StellaOps.Auth.Client; using StellaOps.Auth.ServerIntegration; using StellaOps.Aoc; +using StellaOps.Aoc.AspNetCore.Routing; using StellaOps.Aoc.AspNetCore.Results; using StellaOps.Concelier.WebService.Contracts; using StellaOps.Concelier.Core.Aoc; @@ -54,17 +55,17 @@ const string AdvisoryIngestPolicyName = "Concelier.Advisories.Ingest"; const string AdvisoryReadPolicyName = "Concelier.Advisories.Read"; const string AocVerifyPolicyName = "Concelier.Aoc.Verify"; const string TenantHeaderName = "X-Stella-Tenant"; - -builder.Configuration.AddStellaOpsDefaults(options => -{ - options.BasePath = builder.Environment.ContentRootPath; - options.EnvironmentPrefix = "CONCELIER_"; - options.ConfigureBuilder = configurationBuilder => - { - configurationBuilder.AddConcelierYaml(Path.Combine(builder.Environment.ContentRootPath, "../etc/concelier.yaml")); - }; -}); - + +builder.Configuration.AddStellaOpsDefaults(options => +{ + options.BasePath = builder.Environment.ContentRootPath; + options.EnvironmentPrefix = "CONCELIER_"; + options.ConfigureBuilder = configurationBuilder => + { + configurationBuilder.AddConcelierYaml(Path.Combine(builder.Environment.ContentRootPath, "../etc/concelier.yaml")); + }; +}); + var contentRootPath = builder.Environment.ContentRootPath; var concelierOptions = builder.Configuration.BindOptions(postConfigure: (opts, _) => @@ -243,7 +244,7 @@ if (resolvedAuthority.Enabled && resolvedAuthority.AllowAnonymousFallback) app.MapConcelierMirrorEndpoints(authorityConfigured, enforceAuthority); -app.MapGet("/.well-known/openapi", (OpenApiDiscoveryDocumentProvider provider, HttpContext context) => +app.MapGet("/.well-known/openapi", ([FromServices] OpenApiDiscoveryDocumentProvider provider, HttpContext context) => { var (payload, etag) = provider.GetDocument(); @@ -298,7 +299,7 @@ var observationsEndpoint = app.MapGet("/concelier/observations", async ( [FromQuery(Name = "cpe")] string[]? cpes, [FromQuery(Name = "limit")] int? limit, [FromQuery(Name = "cursor")] string? cursor, - IAdvisoryObservationQueryService queryService, + [FromServices] IAdvisoryObservationQueryService queryService, CancellationToken cancellationToken) => { ApplyNoCache(context.Response); @@ -355,8 +356,8 @@ if (authorityConfigured) var advisoryIngestEndpoint = app.MapPost("/ingest/advisory", async ( HttpContext context, AdvisoryIngestRequest request, - IAdvisoryRawService rawService, - TimeProvider timeProvider, + [FromServices] IAdvisoryRawService rawService, + [FromServices] TimeProvider timeProvider, CancellationToken cancellationToken) => { ApplyNoCache(context.Response); @@ -427,6 +428,41 @@ var advisoryIngestEndpoint = app.MapPost("/ingest/advisory", async ( return MapAocGuardException(context, guardException); } }); + +var advisoryIngestGuardOptions = AocGuardOptions.Default with +{ + RequireTenant = false, + RequiredTopLevelFields = AocGuardOptions.Default.RequiredTopLevelFields.Remove("tenant") +}; + +advisoryIngestEndpoint.RequireAocGuard(request => +{ + if (request?.Source is null || request.Upstream is null || request.Content is null || request.Identifiers is null) + { + return Array.Empty(); + } + + var linkset = request.Linkset ?? new AdvisoryLinksetRequest( + Array.Empty(), + Array.Empty(), + Array.Empty(), + Array.Empty(), + Array.Empty(), + new Dictionary(StringComparer.Ordinal)); + + var payload = new + { + tenant = "guard-tenant", + source = request.Source, + upstream = request.Upstream, + content = request.Content, + identifiers = request.Identifiers, + linkset + }; + + return new object?[] { payload }; +}, guardOptions: advisoryIngestGuardOptions); + if (authorityConfigured) { advisoryIngestEndpoint.RequireAuthorization(AdvisoryIngestPolicyName); @@ -434,7 +470,7 @@ if (authorityConfigured) var advisoryRawListEndpoint = app.MapGet("/advisories/raw", async ( HttpContext context, - IAdvisoryRawService rawService, + [FromServices] IAdvisoryRawService rawService, CancellationToken cancellationToken) => { ApplyNoCache(context.Response); @@ -524,7 +560,7 @@ if (authorityConfigured) var advisoryRawGetEndpoint = app.MapGet("/advisories/raw/{id}", async ( string id, HttpContext context, - IAdvisoryRawService rawService, + [FromServices] IAdvisoryRawService rawService, CancellationToken cancellationToken) => { ApplyNoCache(context.Response); @@ -568,7 +604,7 @@ if (authorityConfigured) var advisoryRawProvenanceEndpoint = app.MapGet("/advisories/raw/{id}/provenance", async ( string id, HttpContext context, - IAdvisoryRawService rawService, + [FromServices] IAdvisoryRawService rawService, CancellationToken cancellationToken) => { ApplyNoCache(context.Response); @@ -614,8 +650,8 @@ if (authorityConfigured) var aocVerifyEndpoint = app.MapPost("/aoc/verify", async ( HttpContext context, AocVerifyRequest request, - IAdvisoryRawService rawService, - TimeProvider timeProvider, + [FromServices] IAdvisoryRawService rawService, + [FromServices] TimeProvider timeProvider, CancellationToken cancellationToken) => { ApplyNoCache(context.Response); @@ -698,7 +734,7 @@ if (authorityConfigured) app.MapGet("/concelier/advisories/{vulnerabilityKey}/replay", async ( string vulnerabilityKey, DateTimeOffset? asOf, - IAdvisoryEventLog eventLog, + [FromServices] IAdvisoryEventLog eventLog, CancellationToken cancellationToken) => { if (string.IsNullOrWhiteSpace(vulnerabilityKey)) @@ -762,29 +798,29 @@ if (loggingEnabled) }; }); } - + app.UseExceptionHandler(errorApp => { errorApp.Run(async context => { context.Response.ContentType = "application/problem+json"; - var feature = context.Features.Get(); - var error = feature?.Error; - - var extensions = new Dictionary(StringComparer.Ordinal) - { - ["traceId"] = Activity.Current?.TraceId.ToString() ?? context.TraceIdentifier, - }; - - var problem = Results.Problem( - detail: error?.Message, - instance: context.Request.Path, - statusCode: StatusCodes.Status500InternalServerError, - title: "Unexpected server error", - type: ProblemTypes.JobFailure, - extensions: extensions); - - await problem.ExecuteAsync(context); + var feature = context.Features.Get(); + var error = feature?.Error; + + var extensions = new Dictionary(StringComparer.Ordinal) + { + ["traceId"] = Activity.Current?.TraceId.ToString() ?? context.TraceIdentifier, + }; + + var problem = Results.Problem( + detail: error?.Message, + instance: context.Request.Path, + statusCode: StatusCodes.Status500InternalServerError, + title: "Unexpected server error", + type: ProblemTypes.JobFailure, + extensions: extensions); + + await problem.ExecuteAsync(context); }); }); @@ -832,13 +868,13 @@ if (authorityConfigured) app.UseAuthentication(); app.UseAuthorization(); } - -IResult JsonResult(T value, int? statusCode = null) -{ - var payload = JsonSerializer.Serialize(value, jsonOptions); - return Results.Content(payload, "application/json", Encoding.UTF8, statusCode); -} - + +IResult JsonResult(T value, int? statusCode = null) +{ + var payload = JsonSerializer.Serialize(value, jsonOptions); + return Results.Content(payload, "application/json", Encoding.UTF8, statusCode); +} + IResult Problem(HttpContext context, string title, int statusCode, string type, string? detail = null, IDictionary? extensions = null) { var traceId = Activity.Current?.TraceId.ToString() ?? context.TraceIdentifier; @@ -951,157 +987,157 @@ IResult MapAocGuardException(HttpContext context, ConcelierAocGuardException exc var guardException = new AocGuardException(exception.Result); return AocHttpResults.Problem(context, guardException); } - -static KeyValuePair[] BuildJobMetricTags(string jobKind, string trigger, string outcome) - => new[] - { - new KeyValuePair("job.kind", jobKind), - new KeyValuePair("job.trigger", trigger), - new KeyValuePair("job.outcome", outcome), - }; - -void ApplyNoCache(HttpResponse response) -{ - if (response is null) - { - return; - } - - response.Headers.CacheControl = "no-store, no-cache, max-age=0, must-revalidate"; - response.Headers.Pragma = "no-cache"; - response.Headers["Expires"] = "0"; -} - -await InitializeMongoAsync(app); - -app.MapGet("/health", (IOptions opts, ServiceStatus status, HttpContext context) => -{ - ApplyNoCache(context.Response); - - var snapshot = status.CreateSnapshot(); - var uptimeSeconds = Math.Max((snapshot.CapturedAt - snapshot.StartedAt).TotalSeconds, 0d); - - var storage = new StorageBootstrapHealth( - Driver: opts.Value.Storage.Driver, - Completed: snapshot.BootstrapCompletedAt is not null, - CompletedAt: snapshot.BootstrapCompletedAt, - DurationMs: snapshot.BootstrapDuration?.TotalMilliseconds); - - var telemetry = new TelemetryHealth( - Enabled: opts.Value.Telemetry.Enabled, - Tracing: opts.Value.Telemetry.EnableTracing, - Metrics: opts.Value.Telemetry.EnableMetrics, - Logging: opts.Value.Telemetry.EnableLogging); - - var response = new HealthDocument( - Status: "healthy", - StartedAt: snapshot.StartedAt, - UptimeSeconds: uptimeSeconds, - Storage: storage, - Telemetry: telemetry); - - return JsonResult(response); -}); - -app.MapGet("/ready", async (IMongoDatabase database, ServiceStatus status, HttpContext context, CancellationToken cancellationToken) => -{ - ApplyNoCache(context.Response); - - var stopwatch = Stopwatch.StartNew(); - try - { - await database.RunCommandAsync((Command)"{ ping: 1 }", cancellationToken: cancellationToken).ConfigureAwait(false); - stopwatch.Stop(); - status.RecordMongoCheck(success: true, latency: stopwatch.Elapsed, error: null); - - var snapshot = status.CreateSnapshot(); - var uptimeSeconds = Math.Max((snapshot.CapturedAt - snapshot.StartedAt).TotalSeconds, 0d); - - var mongo = new MongoReadyHealth( - Status: "ready", - LatencyMs: snapshot.LastMongoLatency?.TotalMilliseconds, - CheckedAt: snapshot.LastReadyCheckAt, - Error: null); - - var response = new ReadyDocument( - Status: "ready", - StartedAt: snapshot.StartedAt, - UptimeSeconds: uptimeSeconds, - Mongo: mongo); - - return JsonResult(response); - } - catch (Exception ex) - { - stopwatch.Stop(); - status.RecordMongoCheck(success: false, latency: stopwatch.Elapsed, error: ex.Message); - - var snapshot = status.CreateSnapshot(); - var uptimeSeconds = Math.Max((snapshot.CapturedAt - snapshot.StartedAt).TotalSeconds, 0d); - - var mongo = new MongoReadyHealth( - Status: "unready", - LatencyMs: snapshot.LastMongoLatency?.TotalMilliseconds, - CheckedAt: snapshot.LastReadyCheckAt, - Error: snapshot.LastMongoError ?? ex.Message); - - var response = new ReadyDocument( - Status: "unready", - StartedAt: snapshot.StartedAt, - UptimeSeconds: uptimeSeconds, - Mongo: mongo); - - var extensions = new Dictionary(StringComparer.Ordinal) - { - ["mongoLatencyMs"] = snapshot.LastMongoLatency?.TotalMilliseconds, - ["mongoError"] = snapshot.LastMongoError ?? ex.Message, - }; - - return Problem(context, "Mongo unavailable", StatusCodes.Status503ServiceUnavailable, ProblemTypes.ServiceUnavailable, snapshot.LastMongoError ?? ex.Message, extensions); - } -}); - -app.MapGet("/diagnostics/aliases/{seed}", async (string seed, AliasGraphResolver resolver, HttpContext context, CancellationToken cancellationToken) => -{ - ApplyNoCache(context.Response); - - if (string.IsNullOrWhiteSpace(seed)) - { - return Problem(context, "Seed advisory key is required.", StatusCodes.Status400BadRequest, ProblemTypes.Validation); - } - - var component = await resolver.BuildComponentAsync(seed, cancellationToken).ConfigureAwait(false); - - var aliases = component.AliasMap.ToDictionary( - static kvp => kvp.Key, - static kvp => kvp.Value - .Select(record => new - { - record.Scheme, - record.Value, - UpdatedAt = record.UpdatedAt - }) - .ToArray()); - - var response = new - { - Seed = component.SeedAdvisoryKey, - Advisories = component.AdvisoryKeys, - Collisions = component.Collisions - .Select(collision => new - { - collision.Scheme, - collision.Value, - AdvisoryKeys = collision.AdvisoryKeys - }) - .ToArray(), - Aliases = aliases - }; - - return JsonResult(response); -}); - -var jobsListEndpoint = app.MapGet("/jobs", async (string? kind, int? limit, IJobCoordinator coordinator, HttpContext context, CancellationToken cancellationToken) => + +static KeyValuePair[] BuildJobMetricTags(string jobKind, string trigger, string outcome) + => new[] + { + new KeyValuePair("job.kind", jobKind), + new KeyValuePair("job.trigger", trigger), + new KeyValuePair("job.outcome", outcome), + }; + +void ApplyNoCache(HttpResponse response) +{ + if (response is null) + { + return; + } + + response.Headers.CacheControl = "no-store, no-cache, max-age=0, must-revalidate"; + response.Headers.Pragma = "no-cache"; + response.Headers["Expires"] = "0"; +} + +await InitializeMongoAsync(app); + +app.MapGet("/health", ([FromServices] IOptions opts, [FromServices] ServiceStatus status, HttpContext context) => +{ + ApplyNoCache(context.Response); + + var snapshot = status.CreateSnapshot(); + var uptimeSeconds = Math.Max((snapshot.CapturedAt - snapshot.StartedAt).TotalSeconds, 0d); + + var storage = new StorageBootstrapHealth( + Driver: opts.Value.Storage.Driver, + Completed: snapshot.BootstrapCompletedAt is not null, + CompletedAt: snapshot.BootstrapCompletedAt, + DurationMs: snapshot.BootstrapDuration?.TotalMilliseconds); + + var telemetry = new TelemetryHealth( + Enabled: opts.Value.Telemetry.Enabled, + Tracing: opts.Value.Telemetry.EnableTracing, + Metrics: opts.Value.Telemetry.EnableMetrics, + Logging: opts.Value.Telemetry.EnableLogging); + + var response = new HealthDocument( + Status: "healthy", + StartedAt: snapshot.StartedAt, + UptimeSeconds: uptimeSeconds, + Storage: storage, + Telemetry: telemetry); + + return JsonResult(response); +}); + +app.MapGet("/ready", async ([FromServices] IMongoDatabase database, [FromServices] ServiceStatus status, HttpContext context, CancellationToken cancellationToken) => +{ + ApplyNoCache(context.Response); + + var stopwatch = Stopwatch.StartNew(); + try + { + await database.RunCommandAsync((Command)"{ ping: 1 }", cancellationToken: cancellationToken).ConfigureAwait(false); + stopwatch.Stop(); + status.RecordMongoCheck(success: true, latency: stopwatch.Elapsed, error: null); + + var snapshot = status.CreateSnapshot(); + var uptimeSeconds = Math.Max((snapshot.CapturedAt - snapshot.StartedAt).TotalSeconds, 0d); + + var mongo = new MongoReadyHealth( + Status: "ready", + LatencyMs: snapshot.LastMongoLatency?.TotalMilliseconds, + CheckedAt: snapshot.LastReadyCheckAt, + Error: null); + + var response = new ReadyDocument( + Status: "ready", + StartedAt: snapshot.StartedAt, + UptimeSeconds: uptimeSeconds, + Mongo: mongo); + + return JsonResult(response); + } + catch (Exception ex) + { + stopwatch.Stop(); + status.RecordMongoCheck(success: false, latency: stopwatch.Elapsed, error: ex.Message); + + var snapshot = status.CreateSnapshot(); + var uptimeSeconds = Math.Max((snapshot.CapturedAt - snapshot.StartedAt).TotalSeconds, 0d); + + var mongo = new MongoReadyHealth( + Status: "unready", + LatencyMs: snapshot.LastMongoLatency?.TotalMilliseconds, + CheckedAt: snapshot.LastReadyCheckAt, + Error: snapshot.LastMongoError ?? ex.Message); + + var response = new ReadyDocument( + Status: "unready", + StartedAt: snapshot.StartedAt, + UptimeSeconds: uptimeSeconds, + Mongo: mongo); + + var extensions = new Dictionary(StringComparer.Ordinal) + { + ["mongoLatencyMs"] = snapshot.LastMongoLatency?.TotalMilliseconds, + ["mongoError"] = snapshot.LastMongoError ?? ex.Message, + }; + + return Problem(context, "Mongo unavailable", StatusCodes.Status503ServiceUnavailable, ProblemTypes.ServiceUnavailable, snapshot.LastMongoError ?? ex.Message, extensions); + } +}); + +app.MapGet("/diagnostics/aliases/{seed}", async (string seed, [FromServices] AliasGraphResolver resolver, HttpContext context, CancellationToken cancellationToken) => +{ + ApplyNoCache(context.Response); + + if (string.IsNullOrWhiteSpace(seed)) + { + return Problem(context, "Seed advisory key is required.", StatusCodes.Status400BadRequest, ProblemTypes.Validation); + } + + var component = await resolver.BuildComponentAsync(seed, cancellationToken).ConfigureAwait(false); + + var aliases = component.AliasMap.ToDictionary( + static kvp => kvp.Key, + static kvp => kvp.Value + .Select(record => new + { + record.Scheme, + record.Value, + UpdatedAt = record.UpdatedAt + }) + .ToArray()); + + var response = new + { + Seed = component.SeedAdvisoryKey, + Advisories = component.AdvisoryKeys, + Collisions = component.Collisions + .Select(collision => new + { + collision.Scheme, + collision.Value, + AdvisoryKeys = collision.AdvisoryKeys + }) + .ToArray(), + Aliases = aliases + }; + + return JsonResult(response); +}); + +var jobsListEndpoint = app.MapGet("/jobs", async (string? kind, int? limit, [FromServices] IJobCoordinator coordinator, HttpContext context, CancellationToken cancellationToken) => { ApplyNoCache(context.Response); @@ -1115,7 +1151,7 @@ if (enforceAuthority) jobsListEndpoint.RequireAuthorization(JobsPolicyName); } -var jobByIdEndpoint = app.MapGet("/jobs/{runId:guid}", async (Guid runId, IJobCoordinator coordinator, HttpContext context, CancellationToken cancellationToken) => +var jobByIdEndpoint = app.MapGet("/jobs/{runId:guid}", async (Guid runId, [FromServices] IJobCoordinator coordinator, HttpContext context, CancellationToken cancellationToken) => { ApplyNoCache(context.Response); @@ -1132,25 +1168,25 @@ if (enforceAuthority) jobByIdEndpoint.RequireAuthorization(JobsPolicyName); } -var jobDefinitionsEndpoint = app.MapGet("/jobs/definitions", async (IJobCoordinator coordinator, HttpContext context, CancellationToken cancellationToken) => +var jobDefinitionsEndpoint = app.MapGet("/jobs/definitions", async ([FromServices] IJobCoordinator coordinator, HttpContext context, CancellationToken cancellationToken) => { - ApplyNoCache(context.Response); - - var definitions = await coordinator.GetDefinitionsAsync(cancellationToken).ConfigureAwait(false); - if (definitions.Count == 0) - { - return JsonResult(Array.Empty()); - } - - var definitionKinds = definitions.Select(static definition => definition.Kind).ToArray(); - var lastRuns = await coordinator.GetLastRunsAsync(definitionKinds, cancellationToken).ConfigureAwait(false); - - var responses = new List(definitions.Count); - foreach (var definition in definitions) - { - lastRuns.TryGetValue(definition.Kind, out var lastRun); - responses.Add(JobDefinitionResponse.FromDefinition(definition, lastRun)); - } + ApplyNoCache(context.Response); + + var definitions = await coordinator.GetDefinitionsAsync(cancellationToken).ConfigureAwait(false); + if (definitions.Count == 0) + { + return JsonResult(Array.Empty()); + } + + var definitionKinds = definitions.Select(static definition => definition.Kind).ToArray(); + var lastRuns = await coordinator.GetLastRunsAsync(definitionKinds, cancellationToken).ConfigureAwait(false); + + var responses = new List(definitions.Count); + foreach (var definition in definitions) + { + lastRuns.TryGetValue(definition.Kind, out var lastRun); + responses.Add(JobDefinitionResponse.FromDefinition(definition, lastRun)); + } return JsonResult(responses); }).AddEndpointFilter(); @@ -1159,20 +1195,20 @@ if (enforceAuthority) jobDefinitionsEndpoint.RequireAuthorization(JobsPolicyName); } -var jobDefinitionEndpoint = app.MapGet("/jobs/definitions/{kind}", async (string kind, IJobCoordinator coordinator, HttpContext context, CancellationToken cancellationToken) => +var jobDefinitionEndpoint = app.MapGet("/jobs/definitions/{kind}", async (string kind, [FromServices] IJobCoordinator coordinator, HttpContext context, CancellationToken cancellationToken) => { - ApplyNoCache(context.Response); - - var definition = (await coordinator.GetDefinitionsAsync(cancellationToken).ConfigureAwait(false)) - .FirstOrDefault(d => string.Equals(d.Kind, kind, StringComparison.Ordinal)); - - if (definition is null) - { - return Problem(context, "Job definition not found", StatusCodes.Status404NotFound, ProblemTypes.NotFound, $"Job kind '{kind}' is not registered."); - } - - var lastRuns = await coordinator.GetLastRunsAsync(new[] { definition.Kind }, cancellationToken).ConfigureAwait(false); - lastRuns.TryGetValue(definition.Kind, out var lastRun); + ApplyNoCache(context.Response); + + var definition = (await coordinator.GetDefinitionsAsync(cancellationToken).ConfigureAwait(false)) + .FirstOrDefault(d => string.Equals(d.Kind, kind, StringComparison.Ordinal)); + + if (definition is null) + { + return Problem(context, "Job definition not found", StatusCodes.Status404NotFound, ProblemTypes.NotFound, $"Job kind '{kind}' is not registered."); + } + + var lastRuns = await coordinator.GetLastRunsAsync(new[] { definition.Kind }, cancellationToken).ConfigureAwait(false); + lastRuns.TryGetValue(definition.Kind, out var lastRun); var response = JobDefinitionResponse.FromDefinition(definition, lastRun); return JsonResult(response); @@ -1182,18 +1218,18 @@ if (enforceAuthority) jobDefinitionEndpoint.RequireAuthorization(JobsPolicyName); } -var jobDefinitionRunsEndpoint = app.MapGet("/jobs/definitions/{kind}/runs", async (string kind, int? limit, IJobCoordinator coordinator, HttpContext context, CancellationToken cancellationToken) => +var jobDefinitionRunsEndpoint = app.MapGet("/jobs/definitions/{kind}/runs", async (string kind, int? limit, [FromServices] IJobCoordinator coordinator, HttpContext context, CancellationToken cancellationToken) => { - ApplyNoCache(context.Response); - - var definition = (await coordinator.GetDefinitionsAsync(cancellationToken).ConfigureAwait(false)) - .FirstOrDefault(d => string.Equals(d.Kind, kind, StringComparison.Ordinal)); - - if (definition is null) - { - return Problem(context, "Job definition not found", StatusCodes.Status404NotFound, ProblemTypes.NotFound, $"Job kind '{kind}' is not registered."); - } - + ApplyNoCache(context.Response); + + var definition = (await coordinator.GetDefinitionsAsync(cancellationToken).ConfigureAwait(false)) + .FirstOrDefault(d => string.Equals(d.Kind, kind, StringComparison.Ordinal)); + + if (definition is null) + { + return Problem(context, "Job definition not found", StatusCodes.Status404NotFound, ProblemTypes.NotFound, $"Job kind '{kind}' is not registered."); + } + var take = Math.Clamp(limit.GetValueOrDefault(20), 1, 200); var runs = await coordinator.GetRecentRunsAsync(kind, take, cancellationToken).ConfigureAwait(false); var payload = runs.Select(JobRunResponse.FromSnapshot).ToArray(); @@ -1204,9 +1240,9 @@ if (enforceAuthority) jobDefinitionRunsEndpoint.RequireAuthorization(JobsPolicyName); } -var activeJobsEndpoint = app.MapGet("/jobs/active", async (IJobCoordinator coordinator, HttpContext context, CancellationToken cancellationToken) => +var activeJobsEndpoint = app.MapGet("/jobs/active", async ([FromServices] IJobCoordinator coordinator, HttpContext context, CancellationToken cancellationToken) => { - ApplyNoCache(context.Response); + ApplyNoCache(context.Response); var runs = await coordinator.GetActiveRunsAsync(cancellationToken).ConfigureAwait(false); var payload = runs.Select(JobRunResponse.FromSnapshot).ToArray(); @@ -1217,22 +1253,22 @@ if (enforceAuthority) activeJobsEndpoint.RequireAuthorization(JobsPolicyName); } -var triggerJobEndpoint = app.MapPost("/jobs/{*jobKind}", async (string jobKind, JobTriggerRequest request, IJobCoordinator coordinator, HttpContext context) => +var triggerJobEndpoint = app.MapPost("/jobs/{*jobKind}", async (string jobKind, JobTriggerRequest request, [FromServices] IJobCoordinator coordinator, HttpContext context) => { - ApplyNoCache(context.Response); - - request ??= new JobTriggerRequest(); - request.Parameters ??= new Dictionary(StringComparer.Ordinal); - var trigger = string.IsNullOrWhiteSpace(request.Trigger) ? "api" : request.Trigger; - - var lifetime = context.RequestServices.GetRequiredService(); - var result = await coordinator.TriggerAsync(jobKind, request.Parameters, trigger, lifetime.ApplicationStopping).ConfigureAwait(false); - - var outcome = result.Outcome; - var tags = BuildJobMetricTags(jobKind, trigger, outcome.ToString().ToLowerInvariant()); - - switch (outcome) - { + ApplyNoCache(context.Response); + + request ??= new JobTriggerRequest(); + request.Parameters ??= new Dictionary(StringComparer.Ordinal); + var trigger = string.IsNullOrWhiteSpace(request.Trigger) ? "api" : request.Trigger; + + var lifetime = context.RequestServices.GetRequiredService(); + var result = await coordinator.TriggerAsync(jobKind, request.Parameters, trigger, lifetime.ApplicationStopping).ConfigureAwait(false); + + var outcome = result.Outcome; + var tags = BuildJobMetricTags(jobKind, trigger, outcome.ToString().ToLowerInvariant()); + + switch (outcome) + { case JobTriggerOutcome.Accepted: JobMetrics.TriggerCounter.Add(1, tags); if (result.Run is null) @@ -1243,54 +1279,54 @@ var triggerJobEndpoint = app.MapPost("/jobs/{*jobKind}", async (string jobKind, var acceptedRun = JobRunResponse.FromSnapshot(result.Run); context.Response.Headers.Location = $"/jobs/{acceptedRun.RunId}"; return JsonResult(acceptedRun, StatusCodes.Status202Accepted); - - case JobTriggerOutcome.NotFound: - JobMetrics.TriggerConflictCounter.Add(1, tags); - return Problem(context, "Job not found", StatusCodes.Status404NotFound, ProblemTypes.NotFound, result.ErrorMessage ?? $"Job '{jobKind}' is not registered."); - - case JobTriggerOutcome.Disabled: - JobMetrics.TriggerConflictCounter.Add(1, tags); - return Problem(context, "Job disabled", StatusCodes.Status423Locked, ProblemTypes.Locked, result.ErrorMessage ?? $"Job '{jobKind}' is disabled."); - - case JobTriggerOutcome.AlreadyRunning: - JobMetrics.TriggerConflictCounter.Add(1, tags); - return Problem(context, "Job already running", StatusCodes.Status409Conflict, ProblemTypes.Conflict, result.ErrorMessage ?? $"Job '{jobKind}' already has an active run."); - - case JobTriggerOutcome.LeaseRejected: - JobMetrics.TriggerConflictCounter.Add(1, tags); - return Problem(context, "Job lease rejected", StatusCodes.Status409Conflict, ProblemTypes.LeaseRejected, result.ErrorMessage ?? $"Job '{jobKind}' could not acquire a lease."); - - case JobTriggerOutcome.InvalidParameters: - { - JobMetrics.TriggerConflictCounter.Add(1, tags); - var extensions = new Dictionary(StringComparer.Ordinal) - { - ["parameters"] = request.Parameters, - }; - return Problem(context, "Invalid job parameters", StatusCodes.Status400BadRequest, ProblemTypes.Validation, result.ErrorMessage, extensions); - } - - case JobTriggerOutcome.Cancelled: - { - JobMetrics.TriggerConflictCounter.Add(1, tags); - var extensions = new Dictionary(StringComparer.Ordinal) - { - ["run"] = result.Run is null ? null : JobRunResponse.FromSnapshot(result.Run), - }; - - return Problem(context, "Job cancelled", StatusCodes.Status409Conflict, ProblemTypes.Conflict, result.ErrorMessage ?? $"Job '{jobKind}' was cancelled before completion.", extensions); - } - - case JobTriggerOutcome.Failed: - { - JobMetrics.TriggerFailureCounter.Add(1, tags); - var extensions = new Dictionary(StringComparer.Ordinal) - { - ["run"] = result.Run is null ? null : JobRunResponse.FromSnapshot(result.Run), - }; - - return Problem(context, "Job execution failed", StatusCodes.Status500InternalServerError, ProblemTypes.JobFailure, result.ErrorMessage, extensions); - } + + case JobTriggerOutcome.NotFound: + JobMetrics.TriggerConflictCounter.Add(1, tags); + return Problem(context, "Job not found", StatusCodes.Status404NotFound, ProblemTypes.NotFound, result.ErrorMessage ?? $"Job '{jobKind}' is not registered."); + + case JobTriggerOutcome.Disabled: + JobMetrics.TriggerConflictCounter.Add(1, tags); + return Problem(context, "Job disabled", StatusCodes.Status423Locked, ProblemTypes.Locked, result.ErrorMessage ?? $"Job '{jobKind}' is disabled."); + + case JobTriggerOutcome.AlreadyRunning: + JobMetrics.TriggerConflictCounter.Add(1, tags); + return Problem(context, "Job already running", StatusCodes.Status409Conflict, ProblemTypes.Conflict, result.ErrorMessage ?? $"Job '{jobKind}' already has an active run."); + + case JobTriggerOutcome.LeaseRejected: + JobMetrics.TriggerConflictCounter.Add(1, tags); + return Problem(context, "Job lease rejected", StatusCodes.Status409Conflict, ProblemTypes.LeaseRejected, result.ErrorMessage ?? $"Job '{jobKind}' could not acquire a lease."); + + case JobTriggerOutcome.InvalidParameters: + { + JobMetrics.TriggerConflictCounter.Add(1, tags); + var extensions = new Dictionary(StringComparer.Ordinal) + { + ["parameters"] = request.Parameters, + }; + return Problem(context, "Invalid job parameters", StatusCodes.Status400BadRequest, ProblemTypes.Validation, result.ErrorMessage, extensions); + } + + case JobTriggerOutcome.Cancelled: + { + JobMetrics.TriggerConflictCounter.Add(1, tags); + var extensions = new Dictionary(StringComparer.Ordinal) + { + ["run"] = result.Run is null ? null : JobRunResponse.FromSnapshot(result.Run), + }; + + return Problem(context, "Job cancelled", StatusCodes.Status409Conflict, ProblemTypes.Conflict, result.ErrorMessage ?? $"Job '{jobKind}' was cancelled before completion.", extensions); + } + + case JobTriggerOutcome.Failed: + { + JobMetrics.TriggerFailureCounter.Add(1, tags); + var extensions = new Dictionary(StringComparer.Ordinal) + { + ["run"] = result.Run is null ? null : JobRunResponse.FromSnapshot(result.Run), + }; + + return Problem(context, "Job execution failed", StatusCodes.Status500InternalServerError, ProblemTypes.JobFailure, result.ErrorMessage, extensions); + } default: JobMetrics.TriggerFailureCounter.Add(1, tags); @@ -1301,61 +1337,61 @@ if (enforceAuthority) { triggerJobEndpoint.RequireAuthorization(JobsPolicyName); } - -await app.RunAsync(); - -static PluginHostOptions BuildPluginOptions(ConcelierOptions options, string contentRoot) -{ - var pluginOptions = new PluginHostOptions - { + +await app.RunAsync(); + +static PluginHostOptions BuildPluginOptions(ConcelierOptions options, string contentRoot) +{ + var pluginOptions = new PluginHostOptions + { BaseDirectory = options.Plugins.BaseDirectory ?? contentRoot, PluginsDirectory = options.Plugins.Directory ?? Path.Combine(contentRoot, "StellaOps.Concelier.PluginBinaries"), PrimaryPrefix = "StellaOps.Concelier", - EnsureDirectoryExists = true, - RecursiveSearch = false, - }; - - if (options.Plugins.SearchPatterns.Count == 0) - { - pluginOptions.SearchPatterns.Add("StellaOps.Concelier.Plugin.*.dll"); - } - else - { - foreach (var pattern in options.Plugins.SearchPatterns) - { - if (!string.IsNullOrWhiteSpace(pattern)) - { - pluginOptions.SearchPatterns.Add(pattern); - } - } - } - - return pluginOptions; -} - -static async Task InitializeMongoAsync(WebApplication app) -{ - await using var scope = app.Services.CreateAsyncScope(); - var bootstrapper = scope.ServiceProvider.GetRequiredService(); - var logger = scope.ServiceProvider.GetRequiredService().CreateLogger("MongoBootstrapper"); - var status = scope.ServiceProvider.GetRequiredService(); - - var stopwatch = Stopwatch.StartNew(); - - try - { - await bootstrapper.InitializeAsync(app.Lifetime.ApplicationStopping).ConfigureAwait(false); - stopwatch.Stop(); - status.MarkBootstrapCompleted(stopwatch.Elapsed); - logger.LogInformation("Mongo bootstrap completed in {ElapsedMs} ms", stopwatch.Elapsed.TotalMilliseconds); - } - catch (Exception ex) - { - stopwatch.Stop(); - status.RecordMongoCheck(success: false, latency: stopwatch.Elapsed, error: ex.Message); - logger.LogCritical(ex, "Mongo bootstrap failed after {ElapsedMs} ms", stopwatch.Elapsed.TotalMilliseconds); - throw; - } -} - -public partial class Program; + EnsureDirectoryExists = true, + RecursiveSearch = false, + }; + + if (options.Plugins.SearchPatterns.Count == 0) + { + pluginOptions.SearchPatterns.Add("StellaOps.Concelier.Plugin.*.dll"); + } + else + { + foreach (var pattern in options.Plugins.SearchPatterns) + { + if (!string.IsNullOrWhiteSpace(pattern)) + { + pluginOptions.SearchPatterns.Add(pattern); + } + } + } + + return pluginOptions; +} + +static async Task InitializeMongoAsync(WebApplication app) +{ + await using var scope = app.Services.CreateAsyncScope(); + var bootstrapper = scope.ServiceProvider.GetRequiredService(); + var logger = scope.ServiceProvider.GetRequiredService().CreateLogger("MongoBootstrapper"); + var status = scope.ServiceProvider.GetRequiredService(); + + var stopwatch = Stopwatch.StartNew(); + + try + { + await bootstrapper.InitializeAsync(app.Lifetime.ApplicationStopping).ConfigureAwait(false); + stopwatch.Stop(); + status.MarkBootstrapCompleted(stopwatch.Elapsed); + logger.LogInformation("Mongo bootstrap completed in {ElapsedMs} ms", stopwatch.Elapsed.TotalMilliseconds); + } + catch (Exception ex) + { + stopwatch.Stop(); + status.RecordMongoCheck(success: false, latency: stopwatch.Elapsed, error: ex.Message); + logger.LogCritical(ex, "Mongo bootstrap failed after {ElapsedMs} ms", stopwatch.Elapsed.TotalMilliseconds); + throw; + } +} + +public partial class Program; diff --git a/src/Concelier/StellaOps.Concelier.WebService/Properties/AssemblyInfo.cs b/src/Concelier/StellaOps.Concelier.WebService/Properties/AssemblyInfo.cs new file mode 100644 index 000000000..91f2514b1 --- /dev/null +++ b/src/Concelier/StellaOps.Concelier.WebService/Properties/AssemblyInfo.cs @@ -0,0 +1,3 @@ +using System.Runtime.CompilerServices; + +[assembly: InternalsVisibleTo("StellaOps.Concelier.WebService.Tests")] diff --git a/src/Concelier/StellaOps.Concelier.WebService/StellaOps.Concelier.WebService.csproj b/src/Concelier/StellaOps.Concelier.WebService/StellaOps.Concelier.WebService.csproj index 9fc94d0d7..ca2c18eb6 100644 --- a/src/Concelier/StellaOps.Concelier.WebService/StellaOps.Concelier.WebService.csproj +++ b/src/Concelier/StellaOps.Concelier.WebService/StellaOps.Concelier.WebService.csproj @@ -35,7 +35,7 @@ - diff --git a/src/Concelier/__Analyzers/StellaOps.Concelier.Analyzers/AnalyzerReleases.Unshipped.md b/src/Concelier/__Analyzers/StellaOps.Concelier.Analyzers/AnalyzerReleases.Unshipped.md index 5c412dba8..dd932e2d4 100644 --- a/src/Concelier/__Analyzers/StellaOps.Concelier.Analyzers/AnalyzerReleases.Unshipped.md +++ b/src/Concelier/__Analyzers/StellaOps.Concelier.Analyzers/AnalyzerReleases.Unshipped.md @@ -2,8 +2,4 @@ ### Unreleased -#### New Rules - -Rule ID | Title | Notes ---------|-------|------ -CONCELIER0002 | Legacy merge pipeline is disabled | Flags usage of `AddMergeModule` and `AdvisoryMergeService`. +No analyzer rules currently scheduled for release. diff --git a/src/Concelier/__Analyzers/StellaOps.Concelier.Analyzers/NoMergeUsageAnalyzer.cs b/src/Concelier/__Analyzers/StellaOps.Concelier.Analyzers/NoMergeUsageAnalyzer.cs deleted file mode 100644 index be7adbe27..000000000 --- a/src/Concelier/__Analyzers/StellaOps.Concelier.Analyzers/NoMergeUsageAnalyzer.cs +++ /dev/null @@ -1,152 +0,0 @@ -using System; -using System.Collections.Immutable; -using Microsoft.CodeAnalysis; -using Microsoft.CodeAnalysis.Diagnostics; -using Microsoft.CodeAnalysis.Operations; - -namespace StellaOps.Concelier.Analyzers; - -/// -/// Analyzer that flags usages of the legacy merge service APIs. -/// -[DiagnosticAnalyzer(LanguageNames.CSharp)] -public sealed class NoMergeUsageAnalyzer : DiagnosticAnalyzer -{ - /// - /// Diagnostic identifier for legacy merge usage violations. - /// - public const string DiagnosticId = "CONCELIER0002"; - - private const string Category = "Usage"; - private const string MergeExtensionType = "StellaOps.Concelier.Merge.MergeServiceCollectionExtensions"; - private const string MergeServiceType = "StellaOps.Concelier.Merge.Services.AdvisoryMergeService"; - - private static readonly LocalizableString Title = "Legacy merge pipeline is disabled"; - private static readonly LocalizableString MessageFormat = "Do not reference the legacy Concelier merge pipeline (type '{0}')"; - private static readonly LocalizableString Description = - "The legacy Concelier merge service is deprecated under MERGE-LNM-21-002. " - + "Switch to observation/linkset APIs or guard calls behind the concelier:features:noMergeEnabled toggle."; - - private static readonly DiagnosticDescriptor Rule = new( - DiagnosticId, - Title, - MessageFormat, - Category, - DiagnosticSeverity.Error, - isEnabledByDefault: true, - description: Description, - helpLinkUri: "https://stella-ops.org/docs/migration/no-merge"); - - /// - public override ImmutableArray SupportedDiagnostics => ImmutableArray.Create(Rule); - - /// - public override void Initialize(AnalysisContext context) - { - context.ConfigureGeneratedCodeAnalysis(GeneratedCodeAnalysisFlags.None); - context.EnableConcurrentExecution(); - - context.RegisterOperationAction(AnalyzeInvocation, OperationKind.Invocation); - context.RegisterOperationAction(AnalyzeObjectCreation, OperationKind.ObjectCreation); - } - - private static void AnalyzeInvocation(OperationAnalysisContext context) - { - if (context.Operation is not IInvocationOperation invocation) - { - return; - } - - var targetMethod = invocation.TargetMethod; - if (targetMethod is null) - { - return; - } - - if (!SymbolEquals(targetMethod.ContainingType, MergeExtensionType)) - { - return; - } - - if (!string.Equals(targetMethod.Name, "AddMergeModule", StringComparison.Ordinal)) - { - return; - } - - if (IsAllowedAssembly(context.ContainingSymbol.ContainingAssembly)) - { - return; - } - - ReportDiagnostic(context, invocation.Syntax, $"{MergeExtensionType}.{targetMethod.Name}"); - } - - private static void AnalyzeObjectCreation(OperationAnalysisContext context) - { - if (context.Operation is not IObjectCreationOperation creation) - { - return; - } - - var createdType = creation.Type; - if (createdType is null || !SymbolEquals(createdType, MergeServiceType)) - { - return; - } - - if (IsAllowedAssembly(context.ContainingSymbol.ContainingAssembly)) - { - return; - } - - ReportDiagnostic(context, creation.Syntax, MergeServiceType); - } - - private static bool SymbolEquals(ITypeSymbol? symbol, string fullName) - { - if (symbol is null) - { - return false; - } - - var display = symbol.ToDisplayString(SymbolDisplayFormat.FullyQualifiedFormat); - if (display.StartsWith("global::", StringComparison.Ordinal)) - { - display = display.Substring("global::".Length); - } - - return string.Equals(display, fullName, StringComparison.Ordinal); - } - - private static bool IsAllowedAssembly(IAssemblySymbol? assemblySymbol) - { - if (assemblySymbol is null) - { - return false; - } - - var assemblyName = assemblySymbol.Name; - if (string.IsNullOrWhiteSpace(assemblyName)) - { - return false; - } - - if (assemblyName.StartsWith("StellaOps.Concelier.Merge", StringComparison.Ordinal)) - { - return true; - } - - if (assemblyName.EndsWith(".Analyzers", StringComparison.Ordinal)) - { - return true; - } - - return false; - } - - private static void ReportDiagnostic(OperationAnalysisContext context, SyntaxNode syntax, string targetName) - { - var diagnostic = Diagnostic.Create(Rule, syntax.GetLocation(), targetName); - context.ReportDiagnostic(diagnostic); - } -} diff --git a/src/Concelier/__Analyzers/StellaOps.Concelier.Merge.Analyzers/AnalyzerReleases.Shipped.md b/src/Concelier/__Analyzers/StellaOps.Concelier.Merge.Analyzers/AnalyzerReleases.Shipped.md new file mode 100644 index 000000000..e88a78fd5 --- /dev/null +++ b/src/Concelier/__Analyzers/StellaOps.Concelier.Merge.Analyzers/AnalyzerReleases.Shipped.md @@ -0,0 +1,2 @@ +; Shipped analyzer releases + diff --git a/src/Concelier/__Analyzers/StellaOps.Concelier.Merge.Analyzers/AnalyzerReleases.Unshipped.md b/src/Concelier/__Analyzers/StellaOps.Concelier.Merge.Analyzers/AnalyzerReleases.Unshipped.md new file mode 100644 index 000000000..4dad2afeb --- /dev/null +++ b/src/Concelier/__Analyzers/StellaOps.Concelier.Merge.Analyzers/AnalyzerReleases.Unshipped.md @@ -0,0 +1,9 @@ +## Release History + +### Unreleased + +#### New Rules + +Rule ID | Title | Notes +--------|-------|------ +CONCELIER0002 | Legacy merge service usage detected | Flags references to `AdvisoryMergeService` and `AddMergeModule`. diff --git a/src/Concelier/__Analyzers/StellaOps.Concelier.Merge.Analyzers/MergeUsageAnalyzer.cs b/src/Concelier/__Analyzers/StellaOps.Concelier.Merge.Analyzers/MergeUsageAnalyzer.cs new file mode 100644 index 000000000..e4fcdfc61 --- /dev/null +++ b/src/Concelier/__Analyzers/StellaOps.Concelier.Merge.Analyzers/MergeUsageAnalyzer.cs @@ -0,0 +1,237 @@ +using System; +using System.Collections.Immutable; +using Microsoft.CodeAnalysis; +using Microsoft.CodeAnalysis.CSharp; +using Microsoft.CodeAnalysis.CSharp.Syntax; +using Microsoft.CodeAnalysis.Diagnostics; +using Microsoft.CodeAnalysis.Operations; + +namespace StellaOps.Concelier.Merge.Analyzers; + +[DiagnosticAnalyzer(LanguageNames.CSharp)] +public sealed class MergeUsageAnalyzer : DiagnosticAnalyzer +{ + public const string DiagnosticId = "CONCELIER0002"; + + private const string AdvisoryMergeServiceTypeName = "StellaOps.Concelier.Merge.Services.AdvisoryMergeService"; + private const string MergeExtensionsTypeName = "StellaOps.Concelier.Merge.MergeServiceCollectionExtensions"; + private const string AddMergeModuleMethodName = "AddMergeModule"; + + private static readonly DiagnosticDescriptor Rule = new( + DiagnosticId, + title: "Legacy merge service usage detected", + messageFormat: "Advisory merge pipeline is deprecated; remove usage of '{0}' and adopt Link-Not-Merge linkset workflows (MERGE-LNM-21-002)", + category: "Usage", + defaultSeverity: DiagnosticSeverity.Warning, + isEnabledByDefault: true, + description: "Link-Not-Merge replaces the legacy AdvisoryMergeService. Set concelier:features:noMergeEnabled=true and migrate to observation/linkset APIs instead of invoking merge services directly.", + helpLinkUri: "https://stella-ops.org/docs/migration/no-merge"); + + public override ImmutableArray SupportedDiagnostics => ImmutableArray.Create(Rule); + + public override void Initialize(AnalysisContext context) + { + context.ConfigureGeneratedCodeAnalysis(GeneratedCodeAnalysisFlags.None); + context.EnableConcurrentExecution(); + + context.RegisterOperationAction(AnalyzeInvocation, OperationKind.Invocation); + context.RegisterOperationAction(AnalyzeObjectCreation, OperationKind.ObjectCreation); + context.RegisterOperationAction(AnalyzeTypeOf, OperationKind.TypeOf); + context.RegisterSyntaxNodeAction(AnalyzeIdentifier, SyntaxKind.IdentifierName); + } + + private static void AnalyzeInvocation(OperationAnalysisContext context) + { + if (context.Operation is not IInvocationOperation invocation) + { + return; + } + + var method = invocation.TargetMethod; + if (!IsAddMergeModule(method) && (method.ReducedFrom is null || !IsAddMergeModule(method.ReducedFrom))) + { + return; + } + + if (IsAllowedAssembly(method.ContainingAssembly, context.ContainingSymbol)) + { + return; + } + + ReportDiagnostic(context, invocation.Syntax.GetLocation(), $"{method.ContainingType.Name}.{method.Name}"); + } + + private static void AnalyzeObjectCreation(OperationAnalysisContext context) + { + if (context.Operation is not IObjectCreationOperation creation) + { + return; + } + + if (creation.Type is not INamedTypeSymbol type || !IsAdvisoryMergeService(type)) + { + return; + } + + if (IsAllowedAssembly(type.ContainingAssembly, context.ContainingSymbol)) + { + return; + } + + ReportDiagnostic(context, creation.Syntax.GetLocation(), type.Name); + } + + private static void AnalyzeTypeOf(OperationAnalysisContext context) + { + if (context.Operation is not ITypeOfOperation typeOfOperation) + { + return; + } + + if (typeOfOperation.TypeOperand is not INamedTypeSymbol type || !IsAdvisoryMergeService(type)) + { + return; + } + + if (IsAllowedAssembly(type.ContainingAssembly, context.ContainingSymbol)) + { + return; + } + + ReportDiagnostic(context, typeOfOperation.Syntax.GetLocation(), type.Name); + } + + private static void AnalyzeIdentifier(SyntaxNodeAnalysisContext context) + { + var identifier = (IdentifierNameSyntax)context.Node; + if (!IsRightMostIdentifier(identifier)) + { + return; + } + + var symbolInfo = context.SemanticModel.GetSymbolInfo(identifier, context.CancellationToken); + var symbol = symbolInfo.Symbol; + if (symbol is not INamedTypeSymbol typeSymbol || !IsAdvisoryMergeService(typeSymbol)) + { + return; + } + + if (IsAllowedAssembly(typeSymbol.ContainingAssembly, context.ContainingSymbol)) + { + return; + } + + if (IsPartOfSuppressedConstruct(identifier)) + { + return; + } + + var diagnostic = Diagnostic.Create(Rule, identifier.GetLocation(), typeSymbol.Name); + context.ReportDiagnostic(diagnostic); + } + + private static bool IsPartOfSuppressedConstruct(IdentifierNameSyntax identifier) + { + foreach (var ancestor in identifier.Ancestors()) + { + switch (ancestor) + { + case ObjectCreationExpressionSyntax: + case TypeOfExpressionSyntax: + return true; + } + } + + return false; + } + + private static bool IsRightMostIdentifier(IdentifierNameSyntax identifier) + { + if (identifier.Parent is QualifiedNameSyntax qualified) + { + return qualified.Right == identifier; + } + + if (identifier.Parent is AliasQualifiedNameSyntax aliasQualified) + { + return aliasQualified.Name == identifier; + } + + return true; + } + + private static bool IsAddMergeModule(IMethodSymbol methodSymbol) + { + if (!string.Equals(methodSymbol.Name, AddMergeModuleMethodName, StringComparison.Ordinal)) + { + return false; + } + + var containingType = methodSymbol.ContainingType; + if (containingType is null) + { + return false; + } + + var display = containingType.ToDisplayString(SymbolDisplayFormat.FullyQualifiedFormat); + display = TrimGlobalPrefix(display); + return string.Equals(display, MergeExtensionsTypeName, StringComparison.Ordinal); + } + + private static bool IsAdvisoryMergeService(INamedTypeSymbol symbol) + { + var display = symbol.ToDisplayString(SymbolDisplayFormat.FullyQualifiedFormat); + display = TrimGlobalPrefix(display); + return string.Equals(display, AdvisoryMergeServiceTypeName, StringComparison.Ordinal); + } + + private static void ReportDiagnostic(OperationAnalysisContext context, Location location, string target) + { + var diagnostic = Diagnostic.Create(Rule, location, target); + context.ReportDiagnostic(diagnostic); + } + + private static bool IsAllowedAssembly(IAssemblySymbol? referencedAssembly, ISymbol? containingSymbol) + { + var consumerAssembly = containingSymbol?.ContainingAssembly; + if (referencedAssembly is null || consumerAssembly is null) + { + return false; + } + + var referencedName = referencedAssembly.Name; + if (!string.IsNullOrWhiteSpace(referencedName) && + referencedName.StartsWith("StellaOps.Concelier.Merge", StringComparison.Ordinal)) + { + return true; + } + + var name = consumerAssembly.Name; + if (string.IsNullOrWhiteSpace(name)) + { + return false; + } + + if (name.StartsWith("StellaOps.Concelier.Merge", StringComparison.Ordinal)) + { + return true; + } + + if (name.EndsWith(".Analyzers", StringComparison.Ordinal)) + { + return true; + } + + return false; + } + + private static string TrimGlobalPrefix(string display) + { + if (!display.StartsWith("global::", StringComparison.Ordinal)) + { + return display; + } + + return display.Substring("global::".Length); + } +} diff --git a/src/Concelier/__Analyzers/StellaOps.Concelier.Merge.Analyzers/StellaOps.Concelier.Merge.Analyzers.csproj b/src/Concelier/__Analyzers/StellaOps.Concelier.Merge.Analyzers/StellaOps.Concelier.Merge.Analyzers.csproj new file mode 100644 index 000000000..28b4a10a2 --- /dev/null +++ b/src/Concelier/__Analyzers/StellaOps.Concelier.Merge.Analyzers/StellaOps.Concelier.Merge.Analyzers.csproj @@ -0,0 +1,23 @@ + + + + netstandard2.0 + enable + enable + preview + false + latest + true + + + + + + + + + + + + + diff --git a/src/Concelier/__Libraries/StellaOps.Concelier.Core/Observations/AdvisoryObservationQueryService.cs b/src/Concelier/__Libraries/StellaOps.Concelier.Core/Observations/AdvisoryObservationQueryService.cs index ae41f4824..7f2c54a69 100644 --- a/src/Concelier/__Libraries/StellaOps.Concelier.Core/Observations/AdvisoryObservationQueryService.cs +++ b/src/Concelier/__Libraries/StellaOps.Concelier.Core/Observations/AdvisoryObservationQueryService.cs @@ -29,7 +29,7 @@ public sealed class AdvisoryObservationQueryService : IAdvisoryObservationQueryS var normalizedTenant = NormalizeTenant(options.Tenant); var normalizedObservationIds = NormalizeSet(options.ObservationIds, static value => value, StringComparer.Ordinal); - var normalizedAliases = NormalizeSet(options.Aliases, static value => value.ToLowerInvariant(), StringComparer.Ordinal); + var normalizedAliases = NormalizeSet(options.Aliases, static value => value, StringComparer.OrdinalIgnoreCase); var normalizedPurls = NormalizeSet(options.Purls, static value => value, StringComparer.Ordinal); var normalizedCpes = NormalizeSet(options.Cpes, static value => value, StringComparer.Ordinal); diff --git a/src/Concelier/__Libraries/StellaOps.Concelier.Core/TASKS.md b/src/Concelier/__Libraries/StellaOps.Concelier.Core/TASKS.md index 066975d03..ce4c629c5 100644 --- a/src/Concelier/__Libraries/StellaOps.Concelier.Core/TASKS.md +++ b/src/Concelier/__Libraries/StellaOps.Concelier.Core/TASKS.md @@ -9,7 +9,7 @@ > Docs alignment (2025-10-26): Linkset expectations detailed in AOC reference §4 and policy-engine architecture §2.1. > 2025-10-28: Advisory raw ingestion now strips client-supplied supersedes hints, logs ignored pointers, and surfaces repository-supplied supersedes identifiers; service tests cover duplicate handling and append-only semantics. > Docs alignment (2025-10-26): Deployment guide + observability guide describe supersedes metrics; ensure implementation emits `aoc_violation_total` on failure. -| CONCELIER-CORE-AOC-19-004 `Remove ingestion normalization` | DOING (2025-10-28) | Concelier Core Guild | CONCELIER-CORE-AOC-19-002, POLICY-AOC-19-003 | Strip normalization/dedup/severity logic from ingestion pipelines, delegate derived computations to Policy Engine, and update exporters/tests to consume raw documents only.
2025-10-29 19:05Z: Audit completed for `AdvisoryRawService`/Mongo repo to confirm alias order/dedup removal persists; identified remaining normalization in observation/linkset factory that will be revised to surface raw duplicates for Policy ingestion. Change sketch + regression matrix drafted under `docs/dev/aoc-normalization-removal-notes.md` (pending commit).
2025-10-31 20:45Z: Added raw linkset projection to observations/storage, exposing canonical+raw views, refreshed fixtures/tests, and documented behaviour in models/doc factory.
2025-10-31 21:10Z: Coordinated with Policy Engine (POLICY-ENGINE-20-003) on adoption timeline; backfill + consumer readiness tracked in `docs/dev/raw-linkset-backfill-plan.md`.
2025-11-05 14:25Z: Resuming to document merge-dependent normalization paths and prepare implementation notes for `noMergeEnabled` gating before code changes land.
2025-11-05 19:20Z: Observation factory/linkset now preserve upstream ordering + duplicates; canonicalisation responsibility shifts to downstream consumers with refreshed unit coverage.
2025-11-06 16:10Z: Updated AOC reference/backfill docs with raw vs canonical guidance and cross-linked analyzer guardrails. | +| CONCELIER-CORE-AOC-19-004 `Remove ingestion normalization` | DONE (2025-11-06) | Concelier Core Guild | CONCELIER-CORE-AOC-19-002, POLICY-AOC-19-003 | Strip normalization/dedup/severity logic from ingestion pipelines, delegate derived computations to Policy Engine, and update exporters/tests to consume raw documents only.
2025-10-29 19:05Z: Audit completed for `AdvisoryRawService`/Mongo repo to confirm alias order/dedup removal persists; identified remaining normalization in observation/linkset factory that will be revised to surface raw duplicates for Policy ingestion. Change sketch + regression matrix drafted under `docs/dev/aoc-normalization-removal-notes.md` (pending commit).
2025-10-31 20:45Z: Added raw linkset projection to observations/storage, exposing canonical+raw views, refreshed fixtures/tests, and documented behaviour in models/doc factory.
2025-10-31 21:10Z: Coordinated with Policy Engine (POLICY-ENGINE-20-003) on adoption timeline; backfill + consumer readiness tracked in `docs/dev/raw-linkset-backfill-plan.md`.
2025-11-05 14:25Z: Resuming to document merge-dependent normalization paths and prepare implementation notes for `noMergeEnabled` gating before code changes land.
2025-11-05 19:20Z: Observation factory/linkset now preserve upstream ordering + duplicates; canonicalisation responsibility shifts to downstream consumers with refreshed unit coverage.
2025-11-06 16:10Z: Updated AOC reference/backfill docs with raw vs canonical guidance and cross-linked analyzer guardrails.
2025-11-06 23:40Z: Final pass preserves raw alias casing/whitespace end-to-end; query filters now compare case-insensitively, exporter fixtures refreshed, and docs aligned. Tests: `StellaOps.Concelier.Models/Core/Storage.Mongo.Tests` green on .NET 10 preview. | > Docs alignment (2025-10-26): Architecture overview emphasises policy-only derivation; coordinate with Policy Engine guild for rollout. > 2025-10-29: `AdvisoryRawService` now preserves upstream alias/linkset ordering (trim-only) and updated AOC documentation reflects the behaviour; follow-up to ensure policy consumers handle duplicates remains open. | CONCELIER-CORE-AOC-19-013 `Authority tenant scope smoke coverage` | TODO | Concelier Core Guild | AUTH-AOC-19-002 | Extend Concelier smoke/e2e fixtures to configure `requiredTenants` and assert cross-tenant rejection with updated Authority tokens. | Coordinate deliverable so Authority docs (`AUTH-AOC-19-003`) can close once tests are in place. | diff --git a/src/Concelier/__Libraries/StellaOps.Concelier.Merge/MergeServiceCollectionExtensions.cs b/src/Concelier/__Libraries/StellaOps.Concelier.Merge/MergeServiceCollectionExtensions.cs index 3a4dd0ac0..6a5c9d585 100644 --- a/src/Concelier/__Libraries/StellaOps.Concelier.Merge/MergeServiceCollectionExtensions.cs +++ b/src/Concelier/__Libraries/StellaOps.Concelier.Merge/MergeServiceCollectionExtensions.cs @@ -5,37 +5,43 @@ using Microsoft.Extensions.DependencyInjection.Extensions; using Microsoft.Extensions.Logging; using StellaOps.Concelier.Core; using StellaOps.Concelier.Merge.Jobs; -using StellaOps.Concelier.Merge.Options; -using StellaOps.Concelier.Merge.Services; - +using StellaOps.Concelier.Merge.Options; +using StellaOps.Concelier.Merge.Services; + namespace StellaOps.Concelier.Merge; [Obsolete("Legacy merge module is deprecated; prefer Link-Not-Merge linkset pipelines. Track MERGE-LNM-21-002 and set concelier:features:noMergeEnabled=true to disable registration.", DiagnosticId = "CONCELIER0001", UrlFormat = "https://stella-ops.org/docs/migration/no-merge")] public static class MergeServiceCollectionExtensions -{ - public static IServiceCollection AddMergeModule(this IServiceCollection services, IConfiguration configuration) - { - ArgumentNullException.ThrowIfNull(services); - ArgumentNullException.ThrowIfNull(configuration); - +{ + public static IServiceCollection AddMergeModule(this IServiceCollection services, IConfiguration configuration) + { + ArgumentNullException.ThrowIfNull(services); + ArgumentNullException.ThrowIfNull(configuration); + + var noMergeEnabled = configuration.GetValue("concelier:features:noMergeEnabled") ?? true; + if (noMergeEnabled) + { + return services; + } + services.TryAddSingleton(); services.TryAddSingleton(); services.TryAddSingleton(); - services.TryAddSingleton(sp => - { - var options = configuration.GetSection("concelier:merge:precedence").Get(); - return options is null ? new AffectedPackagePrecedenceResolver() : new AffectedPackagePrecedenceResolver(options); - }); - - services.TryAddSingleton(sp => - { - var resolver = sp.GetRequiredService(); - var options = configuration.GetSection("concelier:merge:precedence").Get(); - var timeProvider = sp.GetRequiredService(); - var logger = sp.GetRequiredService>(); - return new AdvisoryPrecedenceMerger(resolver, options, timeProvider, logger); - }); - + services.TryAddSingleton(sp => + { + var options = configuration.GetSection("concelier:merge:precedence").Get(); + return options is null ? new AffectedPackagePrecedenceResolver() : new AffectedPackagePrecedenceResolver(options); + }); + + services.TryAddSingleton(sp => + { + var resolver = sp.GetRequiredService(); + var options = configuration.GetSection("concelier:merge:precedence").Get(); + var timeProvider = sp.GetRequiredService(); + var logger = sp.GetRequiredService>(); + return new AdvisoryPrecedenceMerger(resolver, options, timeProvider, logger); + }); + #pragma warning disable CS0618 // Legacy merge services are marked obsolete. services.TryAddSingleton(); services.TryAddSingleton(); diff --git a/src/Concelier/__Libraries/StellaOps.Concelier.Merge/TASKS.md b/src/Concelier/__Libraries/StellaOps.Concelier.Merge/TASKS.md index aac565ff0..e30363fd0 100644 --- a/src/Concelier/__Libraries/StellaOps.Concelier.Merge/TASKS.md +++ b/src/Concelier/__Libraries/StellaOps.Concelier.Merge/TASKS.md @@ -1,15 +1,15 @@ -# TASKS -| Task | Owner(s) | Depends on | Notes | +# TASKS +| Task | Owner(s) | Depends on | Notes | |---|---|---|---| |Link-Not-Merge version provenance coordination|BE-Merge|CONCELIER-LNM-21-001|**DONE (2025-11-04)** – Coordinated connector rollout: updated `docs/dev/normalized-rule-recipes.md` with a per-connector status table + follow-up IDs, enabled `Normalized version rules missing` diagnostics in `AdvisoryPrecedenceMerger`, and confirmed Linkset validation metrics reflect remaining upstream gaps (ACSC/CCCS/CERTBUND/Cisco/RU-BDU awaiting structured ranges).| |FEEDMERGE-COORD-02-901 Connector deadline check-ins|BE-Merge|FEEDMERGE-COORD-02-900|**TODO (due 2025-10-21)** – Confirm Cccs/Cisco version-provenance updates land, capture `LinksetVersionCoverage` dashboard snapshots (expect zero missing-range warnings), and update coordination docs with the results.
2025-10-29: Observation metrics now surface `version_entries_total`/`missing_version_entries_total`; include screenshots for both when closing this task.| |FEEDMERGE-COORD-02-902 ICS-CISA version comparison support|BE-Merge, Models|FEEDMERGE-COORD-02-900|**TODO (due 2025-10-23)** – Review ICS-CISA sample advisories, validate reuse of existing comparison helpers, and pre-stage Models ticket template only if a new firmware comparator is required. Document the outcome and observation coverage logs in coordination docs + tracker files.
2025-10-29: `docs/dev/normalized-rule-recipes.md` (§2–§3) now covers observation entries; attach decision summary + log sample when handing off to Models.| |FEEDMERGE-COORD-02-903 KISA firmware scheme review|BE-Merge, Models|FEEDMERGE-COORD-02-900|**TODO (due 2025-10-24)** – Pair with KISA team on proposed firmware comparison helper (`kisa.build` or variant), ensure observation mapper alignment, and open Models ticket only if a new comparator is required. Log the final helper signature and observation coverage metrics in coordination docs + tracker files.| - -## Link-Not-Merge v1 Transition -| Task | Owner(s) | Depends on | Notes | -|---|---|---|---| + +## Link-Not-Merge v1 Transition +| Task | Owner(s) | Depends on | Notes | +|---|---|---|---| |MERGE-LNM-21-001 Migration plan authoring|BE-Merge, Architecture Guild|CONCELIER-LNM-21-101|**DONE (2025-11-03)** – Authored `docs/migration/no-merge.md` with rollout phases, backfill/validation checklists, rollback guidance, and ownership matrix for the Link-Not-Merge cutover.| -|MERGE-LNM-21-002 Merge service deprecation|BE-Merge|MERGE-LNM-21-001|**DOING (2025-11-03)** – Auditing service registrations, DI bindings, and tests consuming `AdvisoryMergeService`; drafting deprecation plan and analyzer scope prior to code removal.
2025-11-05 14:42Z: Implementing `concelier:features:noMergeEnabled` gate, merge job allowlist checks, `[Obsolete]` markings, and analyzer scaffolding to steer consumers toward linkset APIs.
2025-11-06 16:10Z: Introduced Roslyn analyzer (`CONCELIER0002`) referenced by Concelier WebService + tests, documented suppression guidance, and updated migration playbook.| +|MERGE-LNM-21-002 Merge service deprecation|BE-Merge|MERGE-LNM-21-001|**DOING (2025-11-06)** – Defaulted `concelier:features:noMergeEnabled` to `true`, added merge job allowlist gate, and began rewiring guard/tier tests; follow-up work required to restore Concelier WebService test suite before declaring completion.
2025-11-05 14:42Z: Implemented `concelier:features:noMergeEnabled` gate, merge job allowlist checks, `[Obsolete]` markings, and analyzer scaffolding to steer consumers toward linkset APIs.
2025-11-06 16:10Z: Introduced Roslyn analyzer (`CONCELIER0002`) referenced by Concelier WebService + tests, documented suppression guidance, and updated migration playbook.
2025-11-07 03:25Z: Default-on toggle + job gating break existing Concelier WebService tests; guard + seed fixes pending to unblock ingest/mirror suites.| > 2025-11-03: Catalogued call sites (WebService Program `AddMergeModule`, built-in job registration `merge:reconcile`, `MergeReconcileJob`) and confirmed unit tests are the only direct `MergeAsync` callers; next step is to define analyzer + replacement observability coverage. -|MERGE-LNM-21-003 Determinism/test updates|QA Guild, BE-Merge|MERGE-LNM-21-002|Replace merge determinism suites with observation/linkset regression tests verifying no data mutation and conflicts remain visible.| +|MERGE-LNM-21-003 Determinism/test updates|QA Guild, BE-Merge|MERGE-LNM-21-002|Replace merge determinism suites with observation/linkset regression tests verifying no data mutation and conflicts remain visible.| diff --git a/src/Concelier/__Libraries/StellaOps.Concelier.Models/Observations/AdvisoryObservation.cs b/src/Concelier/__Libraries/StellaOps.Concelier.Models/Observations/AdvisoryObservation.cs index 3aaab89ea..969ea7adc 100644 --- a/src/Concelier/__Libraries/StellaOps.Concelier.Models/Observations/AdvisoryObservation.cs +++ b/src/Concelier/__Libraries/StellaOps.Concelier.Models/Observations/AdvisoryObservation.cs @@ -259,17 +259,27 @@ public sealed record AdvisoryObservationContent } } -public sealed record AdvisoryObservationReference -{ - public AdvisoryObservationReference(string type, string url) - { - Type = Validation.EnsureNotNullOrWhiteSpace(type, nameof(type)).ToLowerInvariant(); - Url = Validation.EnsureNotNullOrWhiteSpace(url, nameof(url)); - } - - public string Type { get; } - - public string Url { get; } +public sealed record AdvisoryObservationReference +{ + public AdvisoryObservationReference(string type, string url) + { + if (string.IsNullOrWhiteSpace(type)) + { + throw new ArgumentException("Reference type cannot be null or whitespace.", nameof(type)); + } + + if (string.IsNullOrWhiteSpace(url)) + { + throw new ArgumentException("Reference url cannot be null or whitespace.", nameof(url)); + } + + Type = type; + Url = url; + } + + public string Type { get; } + + public string Url { get; } } public sealed record AdvisoryObservationLinkset @@ -304,13 +314,12 @@ public sealed record AdvisoryObservationLinkset var builder = ImmutableArray.CreateBuilder(); foreach (var value in values) { - var trimmed = Validation.TrimToNull(value); - if (trimmed is null) + if (value is null) { continue; } - builder.Add(trimmed); + builder.Add(value); } return builder.Count == 0 ? ImmutableArray.Empty : builder.ToImmutable(); diff --git a/src/Concelier/__Libraries/StellaOps.Concelier.Storage.Mongo/Observations/AdvisoryObservationStore.cs b/src/Concelier/__Libraries/StellaOps.Concelier.Storage.Mongo/Observations/AdvisoryObservationStore.cs index 21102d305..a04385c74 100644 --- a/src/Concelier/__Libraries/StellaOps.Concelier.Storage.Mongo/Observations/AdvisoryObservationStore.cs +++ b/src/Concelier/__Libraries/StellaOps.Concelier.Storage.Mongo/Observations/AdvisoryObservationStore.cs @@ -1,11 +1,13 @@ -using System; -using System.Collections.Generic; -using System.Linq; -using MongoDB.Driver; -using StellaOps.Concelier.Core.Observations; -using StellaOps.Concelier.Models.Observations; - -namespace StellaOps.Concelier.Storage.Mongo.Observations; +using System; +using System.Collections.Generic; +using System.Linq; +using System.Text.RegularExpressions; +using MongoDB.Bson; +using MongoDB.Driver; +using StellaOps.Concelier.Core.Observations; +using StellaOps.Concelier.Models.Observations; + +namespace StellaOps.Concelier.Storage.Mongo.Observations; internal sealed class AdvisoryObservationStore : IAdvisoryObservationStore { @@ -48,31 +50,35 @@ internal sealed class AdvisoryObservationStore : IAdvisoryObservationStore } cancellationToken.ThrowIfCancellationRequested(); - var normalizedTenant = tenant.ToLowerInvariant(); - var normalizedObservationIds = NormalizeValues(observationIds, static value => value); - var normalizedAliases = NormalizeValues(aliases, static value => value.ToLowerInvariant()); - var normalizedPurls = NormalizeValues(purls, static value => value); - var normalizedCpes = NormalizeValues(cpes, static value => value); - - var builder = Builders.Filter; - var filters = new List> - { - builder.Eq(document => document.Tenant, normalizedTenant) + var normalizedTenant = tenant.ToLowerInvariant(); + var normalizedObservationIds = NormalizeValues(observationIds, static value => value); + var normalizedAliases = NormalizeAliasFilters(aliases); + var normalizedPurls = NormalizeValues(purls, static value => value); + var normalizedCpes = NormalizeValues(cpes, static value => value); + + var builder = Builders.Filter; + var filters = new List> + { + builder.Eq(document => document.Tenant, normalizedTenant) }; if (normalizedObservationIds.Length > 0) { filters.Add(builder.In(document => document.Id, normalizedObservationIds)); } - - if (normalizedAliases.Length > 0) - { - filters.Add(builder.In("linkset.aliases", normalizedAliases)); - } - - if (normalizedPurls.Length > 0) - { - filters.Add(builder.In("linkset.purls", normalizedPurls)); + + if (normalizedAliases.Length > 0) + { + var aliasFilters = normalizedAliases + .Select(alias => CreateAliasFilter(builder, alias)) + .ToList(); + + filters.Add(builder.Or(aliasFilters)); + } + + if (normalizedPurls.Length > 0) + { + filters.Add(builder.In("linkset.purls", normalizedPurls)); } if (normalizedCpes.Length > 0) @@ -101,16 +107,16 @@ internal sealed class AdvisoryObservationStore : IAdvisoryObservationStore .Limit(limit) .ToListAsync(cancellationToken) .ConfigureAwait(false); - - return documents.Select(AdvisoryObservationDocumentFactory.ToModel).ToArray(); - } - - private static string[] NormalizeValues(IEnumerable? values, Func projector) - { - if (values is null) - { - return Array.Empty(); - } + + return documents.Select(AdvisoryObservationDocumentFactory.ToModel).ToArray(); + } + + private static string[] NormalizeValues(IEnumerable? values, Func projector) + { + if (values is null) + { + return Array.Empty(); + } var set = new HashSet(StringComparer.Ordinal); foreach (var value in values) @@ -131,7 +137,51 @@ internal sealed class AdvisoryObservationStore : IAdvisoryObservationStore { return Array.Empty(); } - - return set.ToArray(); - } -} + + return set.ToArray(); + } + + private static string[] NormalizeAliasFilters(IEnumerable? aliases) + { + if (aliases is null) + { + return Array.Empty(); + } + + var set = new HashSet(StringComparer.OrdinalIgnoreCase); + var list = new List(); + + foreach (var alias in aliases) + { + if (alias is null) + { + continue; + } + + var trimmed = alias.Trim(); + if (trimmed.Length == 0) + { + continue; + } + + if (set.Add(trimmed)) + { + list.Add(trimmed); + } + } + + return list.Count == 0 ? Array.Empty() : list.ToArray(); + } + + private static FilterDefinition CreateAliasFilter( + FilterDefinitionBuilder builder, + string alias) + { + var escaped = Regex.Escape(alias); + var regex = new BsonRegularExpression($"^{escaped}$", "i"); + + return builder.Or( + builder.Regex("rawLinkset.aliases", regex), + builder.Regex("linkset.aliases", regex)); + } +} diff --git a/src/Concelier/__Tests/StellaOps.Concelier.Core.Tests/Linksets/AdvisoryObservationFactoryTests.cs b/src/Concelier/__Tests/StellaOps.Concelier.Core.Tests/Linksets/AdvisoryObservationFactoryTests.cs index 48c03f330..2622404a0 100644 --- a/src/Concelier/__Tests/StellaOps.Concelier.Core.Tests/Linksets/AdvisoryObservationFactoryTests.cs +++ b/src/Concelier/__Tests/StellaOps.Concelier.Core.Tests/Linksets/AdvisoryObservationFactoryTests.cs @@ -42,13 +42,17 @@ public sealed class AdvisoryObservationFactoryTests Assert.Equal( new[] { "cpe:/a:Example:Product:1.0", "cpe:/a:example:product:1.0" }, observation.Linkset.Cpes); - Assert.Equal(2, observation.Linkset.References.Length); - Assert.All( + Assert.Collection( observation.Linkset.References, - reference => + first => { - Assert.Equal("advisory", reference.Type); - Assert.Equal("https://example.test/advisory", reference.Url); + Assert.Equal("Advisory", first.Type); + Assert.Equal("https://example.test/advisory", first.Url); + }, + second => + { + Assert.Equal("ADVISORY", second.Type); + Assert.Equal("https://example.test/advisory", second.Url); }); Assert.Equal( diff --git a/src/Concelier/__Tests/StellaOps.Concelier.Core.Tests/Observations/AdvisoryObservationQueryServiceTests.cs b/src/Concelier/__Tests/StellaOps.Concelier.Core.Tests/Observations/AdvisoryObservationQueryServiceTests.cs index be7fbb1c6..2a7afb262 100644 --- a/src/Concelier/__Tests/StellaOps.Concelier.Core.Tests/Observations/AdvisoryObservationQueryServiceTests.cs +++ b/src/Concelier/__Tests/StellaOps.Concelier.Core.Tests/Observations/AdvisoryObservationQueryServiceTests.cs @@ -311,7 +311,7 @@ public sealed class AdvisoryObservationQueryServiceTests } var observationIdSet = observationIds.ToImmutableHashSet(StringComparer.Ordinal); - var aliasSet = aliases.ToImmutableHashSet(StringComparer.Ordinal); + var aliasSet = aliases.ToImmutableHashSet(StringComparer.OrdinalIgnoreCase); var purlSet = purls.ToImmutableHashSet(StringComparer.Ordinal); var cpeSet = cpes.ToImmutableHashSet(StringComparer.Ordinal); var filtered = observations diff --git a/src/Concelier/__Tests/StellaOps.Concelier.Merge.Analyzers.Tests/MergeUsageAnalyzerTests.cs b/src/Concelier/__Tests/StellaOps.Concelier.Merge.Analyzers.Tests/MergeUsageAnalyzerTests.cs new file mode 100644 index 000000000..029bc87d6 --- /dev/null +++ b/src/Concelier/__Tests/StellaOps.Concelier.Merge.Analyzers.Tests/MergeUsageAnalyzerTests.cs @@ -0,0 +1,165 @@ +using System; +using System.Collections.Generic; +using System.Collections.Immutable; +using System.Linq; +using System.Reflection; +using System.Threading.Tasks; +using Microsoft.CodeAnalysis; +using Microsoft.CodeAnalysis.CSharp; +using Microsoft.CodeAnalysis.Diagnostics; + +namespace StellaOps.Concelier.Merge.Analyzers.Tests; + +public sealed class MergeUsageAnalyzerTests +{ + [Fact] + public async Task ReportsDiagnostic_ForAdvisoryMergeServiceInstantiation() + { + const string source = """ + using StellaOps.Concelier.Merge.Services; + + namespace Sample.App; + + public sealed class Demo + { + public void Run() + { + var merge = new AdvisoryMergeService(); + } + } + """; + + var diagnostics = await AnalyzeAsync(source, "Sample.App"); + Assert.Contains(diagnostics, d => d.Id == MergeUsageAnalyzer.DiagnosticId && d.GetMessage().Contains("AdvisoryMergeService", StringComparison.Ordinal)); + } + + [Fact] + public async Task ReportsDiagnostic_ForAddMergeModuleInvocation() + { + const string source = """ + using Microsoft.Extensions.Configuration; + using Microsoft.Extensions.DependencyInjection; + using StellaOps.Concelier.Merge; + + namespace Sample.Services; + + public static class Installer + { + public static void Configure(IServiceCollection services, IConfiguration configuration) + { + services.AddMergeModule(configuration); + } + } + """; + + var diagnostics = await AnalyzeAsync(source, "Sample.Services"); + Assert.Contains(diagnostics, d => d.Id == MergeUsageAnalyzer.DiagnosticId && d.GetMessage().Contains("AddMergeModule", StringComparison.Ordinal)); + } + + [Fact] + public async Task ReportsDiagnostic_ForFieldDeclaration() + { + const string source = """ + using StellaOps.Concelier.Merge.Services; + + namespace Sample.Library; + + public sealed class Demo + { + private AdvisoryMergeService? _mergeService; + } + """; + + var diagnostics = await AnalyzeAsync(source, "Sample.Library"); + Assert.Contains(diagnostics, d => d.Id == MergeUsageAnalyzer.DiagnosticId); + } + + [Fact] + public async Task DoesNotReportDiagnostic_InsideMergeAssembly() + { + const string source = """ + using StellaOps.Concelier.Merge.Services; + + namespace StellaOps.Concelier.Merge.Internal; + + internal static class MergeDiagnostics + { + public static AdvisoryMergeService Create() => new AdvisoryMergeService(); + } + """; + + var diagnostics = await AnalyzeAsync(source, "StellaOps.Concelier.Merge"); + Assert.DoesNotContain(diagnostics, d => d.Id == MergeUsageAnalyzer.DiagnosticId); + } + + [Fact] + public async Task ReportsDiagnostic_ForTypeOfUsage() + { + const string source = """ + using System; + using StellaOps.Concelier.Merge.Services; + + namespace Sample.TypeOf; + + public static class Demo + { + public static Type TargetType => typeof(AdvisoryMergeService); + } + """; + + var diagnostics = await AnalyzeAsync(source, "Sample.TypeOf"); + Assert.Contains(diagnostics, d => d.Id == MergeUsageAnalyzer.DiagnosticId); + } + + private static async Task> AnalyzeAsync(string source, string assemblyName) + { + var compilation = CSharpCompilation.Create( + assemblyName, + new[] + { + CSharpSyntaxTree.ParseText(source), + CSharpSyntaxTree.ParseText(Stubs) + }, + CreateMetadataReferences(), + new CSharpCompilationOptions(OutputKind.DynamicallyLinkedLibrary)); + + var analyzer = new MergeUsageAnalyzer(); + var compilationWithAnalyzers = compilation.WithAnalyzers(ImmutableArray.Create(analyzer)); + return await compilationWithAnalyzers.GetAnalyzerDiagnosticsAsync(); + } + + private static IEnumerable CreateMetadataReferences() + { + yield return MetadataReference.CreateFromFile(typeof(object).GetTypeInfo().Assembly.Location); + yield return MetadataReference.CreateFromFile(typeof(Enumerable).GetTypeInfo().Assembly.Location); + } + + private const string Stubs = """ + namespace Microsoft.Extensions.DependencyInjection + { + public interface IServiceCollection { } + } + + namespace Microsoft.Extensions.Configuration + { + public interface IConfiguration { } + } + + namespace StellaOps.Concelier.Merge.Services + { + public sealed class AdvisoryMergeService { } + } + + namespace StellaOps.Concelier.Merge + { + public static class MergeServiceCollectionExtensions + { + public static void AddMergeModule( + this Microsoft.Extensions.DependencyInjection.IServiceCollection services, + Microsoft.Extensions.Configuration.IConfiguration configuration) + { + } + } + } + """; +} diff --git a/src/Concelier/__Tests/StellaOps.Concelier.Merge.Analyzers.Tests/StellaOps.Concelier.Merge.Analyzers.Tests.csproj b/src/Concelier/__Tests/StellaOps.Concelier.Merge.Analyzers.Tests/StellaOps.Concelier.Merge.Analyzers.Tests.csproj new file mode 100644 index 000000000..14288d93a --- /dev/null +++ b/src/Concelier/__Tests/StellaOps.Concelier.Merge.Analyzers.Tests/StellaOps.Concelier.Merge.Analyzers.Tests.csproj @@ -0,0 +1,26 @@ + + + + net10.0 + enable + enable + false + + + + + + + + + + + + + + + + + + + diff --git a/src/Concelier/__Tests/StellaOps.Concelier.Models.Tests/Observations/AdvisoryObservationTests.cs b/src/Concelier/__Tests/StellaOps.Concelier.Models.Tests/Observations/AdvisoryObservationTests.cs index 5b01aa124..a49397a37 100644 --- a/src/Concelier/__Tests/StellaOps.Concelier.Models.Tests/Observations/AdvisoryObservationTests.cs +++ b/src/Concelier/__Tests/StellaOps.Concelier.Models.Tests/Observations/AdvisoryObservationTests.cs @@ -63,11 +63,14 @@ public sealed class AdvisoryObservationTests Assert.Equal("tenant-a:CVE-2025-1234:1", observation.ObservationId); Assert.Equal("tenant-a", observation.Tenant); - Assert.Equal("Vendor", observation.Source.Vendor); - Assert.Equal(new[] { "cpe:/a:vendor:product:1" }, observation.Linkset.Cpes); - Assert.Single(observation.Linkset.References); - Assert.Equal("https://example.com/advisory", observation.Linkset.References[0].Url); - Assert.Equal(DateTimeOffset.Parse("2025-10-01T01:00:06Z"), observation.CreatedAt); - Assert.Equal("emea", observation.Attributes["region"]); - } -} + Assert.Equal("Vendor", observation.Source.Vendor); + Assert.Equal(new[] { " Cve-2025-1234 ", "cve-2025-1234" }, observation.Linkset.Aliases.ToArray()); + Assert.Equal(new[] { "cpe:/a:vendor:product:1" }, observation.Linkset.Cpes); + Assert.Equal(2, observation.Linkset.References.Length); + Assert.Equal("ADVISORY", observation.Linkset.References[0].Type); + Assert.Equal("https://example.com/advisory", observation.Linkset.References[0].Url); + Assert.Equal(rawLinkset.Aliases, observation.RawLinkset.Aliases); + Assert.Equal(DateTimeOffset.Parse("2025-10-01T01:00:06Z"), observation.CreatedAt); + Assert.Equal("emea", observation.Attributes["region"]); + } +} diff --git a/src/Concelier/__Tests/StellaOps.Concelier.Storage.Mongo.Tests/Observations/AdvisoryObservationDocumentFactoryTests.cs b/src/Concelier/__Tests/StellaOps.Concelier.Storage.Mongo.Tests/Observations/AdvisoryObservationDocumentFactoryTests.cs index 12f48e9cc..577e0fd9c 100644 --- a/src/Concelier/__Tests/StellaOps.Concelier.Storage.Mongo.Tests/Observations/AdvisoryObservationDocumentFactoryTests.cs +++ b/src/Concelier/__Tests/StellaOps.Concelier.Storage.Mongo.Tests/Observations/AdvisoryObservationDocumentFactoryTests.cs @@ -71,11 +71,12 @@ public sealed class AdvisoryObservationDocumentFactoryTests Assert.Equal("tenant-a:obs-1", observation.ObservationId); Assert.Equal("tenant-a", observation.Tenant); - Assert.Equal("CVE-2025-1234", observation.Upstream.UpstreamId); + Assert.Equal("CVE-2025-1234", observation.Upstream.UpstreamId); + Assert.Equal(new[] { "CVE-2025-1234" }, observation.Linkset.Aliases.ToArray()); Assert.Contains("pkg:generic/foo@1.0.0", observation.Linkset.Purls); Assert.Equal("CSAF", observation.Content.Format); Assert.True(observation.Content.Raw?["example"]?.GetValue()); - Assert.Equal("advisory", observation.Linkset.References[0].Type); + Assert.Equal(document.Linkset.References![0].Type, observation.Linkset.References[0].Type); Assert.Equal(new[] { "CVE-2025-1234", "cve-2025-1234" }, observation.RawLinkset.Aliases); Assert.Equal("Advisory", observation.RawLinkset.References[0].Type); Assert.Equal("vendor", observation.RawLinkset.References[0].Source); diff --git a/src/Concelier/__Tests/StellaOps.Concelier.Storage.Mongo.Tests/Observations/AdvisoryObservationStoreTests.cs b/src/Concelier/__Tests/StellaOps.Concelier.Storage.Mongo.Tests/Observations/AdvisoryObservationStoreTests.cs index 241b304bf..7424dd7e9 100644 --- a/src/Concelier/__Tests/StellaOps.Concelier.Storage.Mongo.Tests/Observations/AdvisoryObservationStoreTests.cs +++ b/src/Concelier/__Tests/StellaOps.Concelier.Storage.Mongo.Tests/Observations/AdvisoryObservationStoreTests.cs @@ -31,22 +31,22 @@ public sealed class AdvisoryObservationStoreTests : IClassFixture(MongoStorageDefaults.Collections.AdvisoryObservations); await collection.InsertManyAsync(new[] { - CreateDocument( - id: "tenant-a:nvd:alpha:1", - tenant: "tenant-a", - createdAt: new DateTime(2025, 1, 1, 0, 0, 0, DateTimeKind.Utc), - aliases: new[] { "cve-2025-0001" }, - purls: new[] { "pkg:npm/demo@1.0.0" }), - CreateDocument( - id: "tenant-a:ghsa:beta:1", - tenant: "tenant-a", - createdAt: new DateTime(2025, 1, 2, 0, 0, 0, DateTimeKind.Utc), - aliases: new[] { "ghsa-xyz0", "cve-2025-0001" }, - purls: new[] { "pkg:npm/demo@1.1.0" }), - CreateDocument( - id: "tenant-b:nvd:alpha:1", - tenant: "tenant-b", - createdAt: new DateTime(2025, 1, 3, 0, 0, 0, DateTimeKind.Utc), + CreateDocument( + id: "tenant-a:nvd:alpha:1", + tenant: "tenant-a", + createdAt: new DateTime(2025, 1, 1, 0, 0, 0, DateTimeKind.Utc), + aliases: new[] { "CvE-2025-0001 " }, + purls: new[] { "pkg:npm/demo@1.0.0" }), + CreateDocument( + id: "tenant-a:ghsa:beta:1", + tenant: "tenant-a", + createdAt: new DateTime(2025, 1, 2, 0, 0, 0, DateTimeKind.Utc), + aliases: new[] { " ghsa-xyz0", "cve-2025-0001" }, + purls: new[] { "pkg:npm/demo@1.1.0" }), + CreateDocument( + id: "tenant-b:nvd:alpha:1", + tenant: "tenant-b", + createdAt: new DateTime(2025, 1, 3, 0, 0, 0, DateTimeKind.Utc), aliases: new[] { "cve-2025-0001" }, purls: new[] { "pkg:npm/demo@2.0.0" }) }); @@ -62,11 +62,15 @@ public sealed class AdvisoryObservationStoreTests : IClassFixture Assert.Equal("tenant-a", observation.Tenant)); - } + Assert.Equal(2, result.Count); + Assert.Equal("tenant-a:ghsa:beta:1", result[0].ObservationId); + Assert.Equal("tenant-a:nvd:alpha:1", result[1].ObservationId); + Assert.All(result, observation => Assert.Equal("tenant-a", observation.Tenant)); + Assert.Equal("ghsa-xyz0", result[0].Linkset.Aliases[0]); + Assert.Equal("CvE-2025-0001", result[1].Linkset.Aliases[0]); + Assert.Equal(" ghsa-xyz0", result[0].RawLinkset.Aliases[0]); + Assert.Equal("CvE-2025-0001 ", result[1].RawLinkset.Aliases[0]); + } [Fact] public async Task FindByFiltersAsync_RespectsObservationIdsAndPurls() @@ -166,12 +170,39 @@ public sealed class AdvisoryObservationStoreTests : IClassFixture? purls = null, IEnumerable? cpes = null) { - return new AdvisoryObservationDocument - { - Id = id, - Tenant = tenant.ToLowerInvariant(), - CreatedAt = createdAt, - Source = new AdvisoryObservationSourceDocument + var canonicalAliases = aliases? + .Where(value => value is not null) + .Select(value => value.Trim()) + .ToList(); + + var canonicalPurls = purls? + .Where(value => value is not null) + .Select(value => value.Trim()) + .ToList(); + + var canonicalCpes = cpes? + .Where(value => value is not null) + .Select(value => value.Trim()) + .ToList(); + + var rawAliases = aliases? + .Where(value => value is not null) + .ToList(); + + var rawPurls = purls? + .Where(value => value is not null) + .ToList(); + + var rawCpes = cpes? + .Where(value => value is not null) + .ToList(); + + return new AdvisoryObservationDocument + { + Id = id, + Tenant = tenant.ToLowerInvariant(), + CreatedAt = createdAt, + Source = new AdvisoryObservationSourceDocument { Vendor = "nvd", Stream = "feed", @@ -189,24 +220,31 @@ public sealed class AdvisoryObservationStoreTests : IClassFixture(StringComparer.Ordinal) - }, - Content = new AdvisoryObservationContentDocument - { - Format = "csaf", - SpecVersion = "2.0", - Raw = BsonDocument.Parse("""{"id": "%ID%"}""".Replace("%ID%", id)), - Metadata = new Dictionary(StringComparer.Ordinal) - }, - Linkset = new AdvisoryObservationLinksetDocument - { - Aliases = aliases?.Select(value => value.Trim()).ToList(), - Purls = purls?.Select(value => value.Trim()).ToList(), - Cpes = cpes?.Select(value => value.Trim()).ToList(), - References = new List() - }, - Attributes = new Dictionary(StringComparer.Ordinal) - }; - } + }, + Content = new AdvisoryObservationContentDocument + { + Format = "csaf", + SpecVersion = "2.0", + Raw = BsonDocument.Parse("""{"id": "%ID%"}""".Replace("%ID%", id)), + Metadata = new Dictionary(StringComparer.Ordinal) + }, + Linkset = new AdvisoryObservationLinksetDocument + { + Aliases = canonicalAliases, + Purls = canonicalPurls, + Cpes = canonicalCpes, + References = new List() + }, + RawLinkset = new AdvisoryObservationRawLinksetDocument + { + Aliases = rawAliases, + PackageUrls = rawPurls, + Cpes = rawCpes, + References = new List() + }, + Attributes = new Dictionary(StringComparer.Ordinal) + }; + } private async Task ResetCollectionAsync() { diff --git a/src/Concelier/__Tests/StellaOps.Concelier.WebService.Tests/StellaOps.Concelier.WebService.Tests.csproj b/src/Concelier/__Tests/StellaOps.Concelier.WebService.Tests/StellaOps.Concelier.WebService.Tests.csproj index 1432586b1..e50da0398 100644 --- a/src/Concelier/__Tests/StellaOps.Concelier.WebService.Tests/StellaOps.Concelier.WebService.Tests.csproj +++ b/src/Concelier/__Tests/StellaOps.Concelier.WebService.Tests/StellaOps.Concelier.WebService.Tests.csproj @@ -10,7 +10,7 @@ - diff --git a/src/Concelier/__Tests/StellaOps.Concelier.WebService.Tests/WebServiceEndpointsTests.cs b/src/Concelier/__Tests/StellaOps.Concelier.WebService.Tests/WebServiceEndpointsTests.cs index f4893ab03..afdde007c 100644 --- a/src/Concelier/__Tests/StellaOps.Concelier.WebService.Tests/WebServiceEndpointsTests.cs +++ b/src/Concelier/__Tests/StellaOps.Concelier.WebService.Tests/WebServiceEndpointsTests.cs @@ -1,907 +1,903 @@ -using System; -using System.Collections.Generic; -using System.Diagnostics.Metrics; -using System.Globalization; -using System.IdentityModel.Tokens.Jwt; -using System.IO; -using System.Linq; -using System.Net; -using System.Net.Http.Json; -using System.Net.Http.Headers; -using System.Security.Claims; -using System.Text; -using System.Text.Json; -using Microsoft.AspNetCore.Builder; -using Microsoft.AspNetCore.Hosting; -using Microsoft.AspNetCore.Mvc.Testing; -using Microsoft.AspNetCore.TestHost; -using Microsoft.AspNetCore.Authentication.JwtBearer; -using Microsoft.Extensions.Configuration; -using Microsoft.Extensions.DependencyInjection; -using Microsoft.Extensions.Logging; -using Microsoft.Extensions.Options; -using Mongo2Go; -using MongoDB.Bson; -using MongoDB.Driver; -using StellaOps.Concelier.Core.Events; -using StellaOps.Concelier.Core.Jobs; -using StellaOps.Concelier.Models; -using StellaOps.Concelier.Merge.Services; -using StellaOps.Concelier.Storage.Mongo; -using StellaOps.Concelier.Storage.Mongo.Observations; -using StellaOps.Concelier.WebService.Jobs; -using StellaOps.Concelier.WebService.Options; -using StellaOps.Concelier.WebService.Contracts; -using Xunit.Sdk; -using StellaOps.Auth.Abstractions; -using StellaOps.Auth.Client; -using Xunit; -using Microsoft.IdentityModel.Protocols; -using Microsoft.IdentityModel.Protocols.OpenIdConnect; -using Microsoft.IdentityModel.Tokens; - -namespace StellaOps.Concelier.WebService.Tests; - -public sealed class WebServiceEndpointsTests : IAsyncLifetime -{ - private const string TestAuthorityIssuer = "https://authority.example"; - private const string TestAuthorityAudience = "api://concelier"; - private const string TestSigningSecret = "0123456789ABCDEF0123456789ABCDEF"; - private static readonly SymmetricSecurityKey TestSigningKey = new(Encoding.UTF8.GetBytes(TestSigningSecret)); - - private MongoDbRunner _runner = null!; - private ConcelierApplicationFactory _factory = null!; - - public Task InitializeAsync() - { - _runner = MongoDbRunner.Start(singleNodeReplSet: true); - _factory = new ConcelierApplicationFactory(_runner.ConnectionString); - return Task.CompletedTask; - } - - public Task DisposeAsync() - { - _factory.Dispose(); - _runner.Dispose(); - return Task.CompletedTask; - } - - [Fact] - public async Task HealthAndReadyEndpointsRespond() - { - using var client = _factory.CreateClient(); - - var healthResponse = await client.GetAsync("/health"); - if (!healthResponse.IsSuccessStatusCode) - { - var body = await healthResponse.Content.ReadAsStringAsync(); - throw new Xunit.Sdk.XunitException($"/health failed: {(int)healthResponse.StatusCode} {body}"); - } - - var readyResponse = await client.GetAsync("/ready"); - if (!readyResponse.IsSuccessStatusCode) - { - var body = await readyResponse.Content.ReadAsStringAsync(); - throw new Xunit.Sdk.XunitException($"/ready failed: {(int)readyResponse.StatusCode} {body}"); - } - - var healthPayload = await healthResponse.Content.ReadFromJsonAsync(); - Assert.NotNull(healthPayload); - Assert.Equal("healthy", healthPayload!.Status); - Assert.Equal("mongo", healthPayload.Storage.Driver); - - var readyPayload = await readyResponse.Content.ReadFromJsonAsync(); - Assert.NotNull(readyPayload); - Assert.Equal("ready", readyPayload!.Status); - Assert.Equal("ready", readyPayload.Mongo.Status); - } - - [Fact] - public async Task ObservationsEndpoint_ReturnsTenantScopedResults() - { - await SeedObservationDocumentsAsync(BuildSampleObservationDocuments()); - - using var client = _factory.CreateClient(); - - var response = await client.GetAsync("/concelier/observations?tenant=tenant-a&alias=CVE-2025-0001"); - if (!response.IsSuccessStatusCode) - { - var body = await response.Content.ReadAsStringAsync(); - throw new XunitException($"/concelier/observations failed: {(int)response.StatusCode} {body}"); - } - - using var document = await response.Content.ReadFromJsonAsync(); - Assert.NotNull(document); - var root = document!.RootElement; - var observations = root.GetProperty("observations").EnumerateArray().ToArray(); - Assert.Equal(2, observations.Length); - Assert.Equal("tenant-a:ghsa:beta:1", observations[0].GetProperty("observationId").GetString()); - Assert.Equal("tenant-a:nvd:alpha:1", observations[1].GetProperty("observationId").GetString()); - - var linkset = root.GetProperty("linkset"); - Assert.Equal(new[] { "cve-2025-0001", "ghsa-2025-xyz" }, linkset.GetProperty("aliases").EnumerateArray().Select(x => x.GetString()).ToArray()); - Assert.Equal(new[] { "pkg:npm/demo@1.0.0", "pkg:npm/demo@1.1.0" }, linkset.GetProperty("purls").EnumerateArray().Select(x => x.GetString()).ToArray()); - Assert.Equal(new[] { "cpe:/a:vendor:product:1.0", "cpe:/a:vendor:product:1.1" }, linkset.GetProperty("cpes").EnumerateArray().Select(x => x.GetString()).ToArray()); - - var references = linkset.GetProperty("references").EnumerateArray().ToArray(); - Assert.Equal(2, references.Length); - Assert.Equal("advisory", references[0].GetProperty("type").GetString()); - Assert.Equal("https://example.test/advisory-1", references[0].GetProperty("url").GetString()); - Assert.Equal("patch", references[1].GetProperty("type").GetString()); - - Assert.False(root.GetProperty("hasMore").GetBoolean()); - Assert.True(root.GetProperty("nextCursor").ValueKind == JsonValueKind.Null); - } - - [Fact] - public async Task ObservationsEndpoint_AppliesObservationIdFilter() - { - await SeedObservationDocumentsAsync(BuildSampleObservationDocuments()); - - using var client = _factory.CreateClient(); - var observationId = Uri.EscapeDataString("tenant-a:ghsa:beta:1"); - var response = await client.GetAsync($"/concelier/observations?tenant=tenant-a&observationId={observationId}&cpe=cpe:/a:vendor:product:1.1"); - if (!response.IsSuccessStatusCode) - { - var body = await response.Content.ReadAsStringAsync(); - throw new XunitException($"/concelier/observations filter failed: {(int)response.StatusCode} {body}"); - } - - using var document = await response.Content.ReadFromJsonAsync(); - Assert.NotNull(document); - var root = document!.RootElement; - var observations = root.GetProperty("observations").EnumerateArray().ToArray(); - Assert.Single(observations); - Assert.Equal("tenant-a:ghsa:beta:1", observations[0].GetProperty("observationId").GetString()); - Assert.Equal(new[] { "pkg:npm/demo@1.1.0" }, observations[0].GetProperty("linkset").GetProperty("purls").EnumerateArray().Select(x => x.GetString()).ToArray()); - Assert.Equal(new[] { "cpe:/a:vendor:product:1.1" }, observations[0].GetProperty("linkset").GetProperty("cpes").EnumerateArray().Select(x => x.GetString()).ToArray()); - - Assert.False(root.GetProperty("hasMore").GetBoolean()); - Assert.True(root.GetProperty("nextCursor").ValueKind == JsonValueKind.Null); - } - - [Fact] - public async Task ObservationsEndpoint_SupportsPagination() - { - await SeedObservationDocumentsAsync(BuildSampleObservationDocuments()); - - using var client = _factory.CreateClient(); - - var firstResponse = await client.GetAsync("/concelier/observations?tenant=tenant-a&limit=1"); - firstResponse.EnsureSuccessStatusCode(); - using var firstDocument = await firstResponse.Content.ReadFromJsonAsync(); - Assert.NotNull(firstDocument); - var firstRoot = firstDocument!.RootElement; - var firstObservations = firstRoot.GetProperty("observations").EnumerateArray().ToArray(); - Assert.Single(firstObservations); - var nextCursor = firstRoot.GetProperty("nextCursor").GetString(); - Assert.True(firstRoot.GetProperty("hasMore").GetBoolean()); - Assert.False(string.IsNullOrWhiteSpace(nextCursor)); - - var secondResponse = await client.GetAsync($"/concelier/observations?tenant=tenant-a&limit=2&cursor={Uri.EscapeDataString(nextCursor!)}"); - secondResponse.EnsureSuccessStatusCode(); - using var secondDocument = await secondResponse.Content.ReadFromJsonAsync(); - Assert.NotNull(secondDocument); - var secondRoot = secondDocument!.RootElement; - var secondObservations = secondRoot.GetProperty("observations").EnumerateArray().ToArray(); - Assert.Single(secondObservations); - Assert.False(secondRoot.GetProperty("hasMore").GetBoolean()); - Assert.True(secondRoot.GetProperty("nextCursor").ValueKind == JsonValueKind.Null); - Assert.Equal("tenant-a:nvd:alpha:1", secondObservations[0].GetProperty("observationId").GetString()); - } - - [Fact] - public async Task ObservationsEndpoint_ReturnsBadRequestWhenTenantMissing() - { - using var client = _factory.CreateClient(); - var response = await client.GetAsync("/concelier/observations"); - var body = await response.Content.ReadAsStringAsync(); - Assert.True(response.StatusCode == HttpStatusCode.BadRequest, $"Expected 400 but got {(int)response.StatusCode}: {body}"); - } - - [Fact] - public async Task AdvisoryIngestEndpoint_PersistsDocumentAndSupportsReadback() - { - using var client = _factory.CreateClient(); - client.DefaultRequestHeaders.Add("X-Stella-Tenant", "tenant-ingest"); - - var ingestRequest = BuildAdvisoryIngestRequest( - contentHash: "sha256:abc123", - upstreamId: "GHSA-INGEST-0001"); - - var ingestResponse = await client.PostAsJsonAsync("/ingest/advisory", ingestRequest); - Assert.Equal(HttpStatusCode.Created, ingestResponse.StatusCode); - - var ingestPayload = await ingestResponse.Content.ReadFromJsonAsync(); - Assert.NotNull(ingestPayload); - Assert.True(ingestPayload!.Inserted); - Assert.False(string.IsNullOrWhiteSpace(ingestPayload.Id)); - Assert.Equal("tenant-ingest", ingestPayload.Tenant); - Assert.Equal("sha256:abc123", ingestPayload.ContentHash); - Assert.NotNull(ingestResponse.Headers.Location); - var locationValue = ingestResponse.Headers.Location!.ToString(); - Assert.False(string.IsNullOrWhiteSpace(locationValue)); +using System; +using System.Collections.Generic; +using System.Diagnostics.Metrics; +using System.Globalization; +using System.IdentityModel.Tokens.Jwt; +using System.IO; +using System.Linq; +using System.Net; +using System.Net.Http.Json; +using System.Net.Http.Headers; +using System.Security.Claims; +using System.Text; +using System.Text.Json; +using Microsoft.AspNetCore.Builder; +using Microsoft.AspNetCore.Hosting; +using Microsoft.AspNetCore.Mvc.Testing; +using Microsoft.AspNetCore.TestHost; +using Microsoft.AspNetCore.Authentication.JwtBearer; +using Microsoft.Extensions.Configuration; +using Microsoft.Extensions.DependencyInjection; +using Microsoft.Extensions.Logging; +using Microsoft.Extensions.Options; +using Mongo2Go; +using MongoDB.Bson; +using MongoDB.Driver; +using StellaOps.Concelier.Core.Events; +using StellaOps.Concelier.Core.Jobs; +using StellaOps.Concelier.Models; +using StellaOps.Concelier.Merge.Services; +using StellaOps.Concelier.Storage.Mongo; +using StellaOps.Concelier.Storage.Mongo.Observations; +using StellaOps.Concelier.WebService.Jobs; +using StellaOps.Concelier.WebService.Options; +using StellaOps.Concelier.WebService.Contracts; +using Xunit.Sdk; +using StellaOps.Auth.Abstractions; +using StellaOps.Auth.Client; +using Xunit; +using Microsoft.IdentityModel.Protocols; +using Microsoft.IdentityModel.Protocols.OpenIdConnect; +using StellaOps.Concelier.WebService.Diagnostics; +using Microsoft.IdentityModel.Tokens; + +namespace StellaOps.Concelier.WebService.Tests; + +public sealed class WebServiceEndpointsTests : IAsyncLifetime +{ + private const string TestAuthorityIssuer = "https://authority.example"; + private const string TestAuthorityAudience = "api://concelier"; + private const string TestSigningSecret = "0123456789ABCDEF0123456789ABCDEF"; + private static readonly SymmetricSecurityKey TestSigningKey = new(Encoding.UTF8.GetBytes(TestSigningSecret)); + + private MongoDbRunner _runner = null!; + private ConcelierApplicationFactory _factory = null!; + + public Task InitializeAsync() + { + _runner = MongoDbRunner.Start(singleNodeReplSet: true); + _factory = new ConcelierApplicationFactory(_runner.ConnectionString); + return Task.CompletedTask; + } + + public Task DisposeAsync() + { + _factory.Dispose(); + _runner.Dispose(); + return Task.CompletedTask; + } + + [Fact] + public async Task HealthAndReadyEndpointsRespond() + { + using var client = _factory.CreateClient(); + + var healthResponse = await client.GetAsync("/health"); + if (!healthResponse.IsSuccessStatusCode) + { + var body = await healthResponse.Content.ReadAsStringAsync(); + throw new Xunit.Sdk.XunitException($"/health failed: {(int)healthResponse.StatusCode} {body}"); + } + + var readyResponse = await client.GetAsync("/ready"); + if (!readyResponse.IsSuccessStatusCode) + { + var body = await readyResponse.Content.ReadAsStringAsync(); + throw new Xunit.Sdk.XunitException($"/ready failed: {(int)readyResponse.StatusCode} {body}"); + } + + var healthPayload = await healthResponse.Content.ReadFromJsonAsync(); + Assert.NotNull(healthPayload); + Assert.Equal("healthy", healthPayload!.Status); + Assert.Equal("mongo", healthPayload.Storage.Driver); + + var readyPayload = await readyResponse.Content.ReadFromJsonAsync(); + Assert.NotNull(readyPayload); + Assert.Equal("ready", readyPayload!.Status); + Assert.Equal("ready", readyPayload.Mongo.Status); + } + + [Fact] + public async Task ObservationsEndpoint_ReturnsTenantScopedResults() + { + await SeedObservationDocumentsAsync(BuildSampleObservationDocuments()); + + using var client = _factory.CreateClient(); + + var response = await client.GetAsync("/concelier/observations?tenant=tenant-a&alias=CVE-2025-0001"); + if (!response.IsSuccessStatusCode) + { + var body = await response.Content.ReadAsStringAsync(); + throw new XunitException($"/concelier/observations failed: {(int)response.StatusCode} {body}"); + } + + using var document = await response.Content.ReadFromJsonAsync(); + Assert.NotNull(document); + var root = document!.RootElement; + var observations = root.GetProperty("observations").EnumerateArray().ToArray(); + Assert.Equal(2, observations.Length); + Assert.Equal("tenant-a:ghsa:beta:1", observations[0].GetProperty("observationId").GetString()); + Assert.Equal("tenant-a:nvd:alpha:1", observations[1].GetProperty("observationId").GetString()); + + var linkset = root.GetProperty("linkset"); + Assert.Equal(new[] { "cve-2025-0001", "ghsa-2025-xyz" }, linkset.GetProperty("aliases").EnumerateArray().Select(x => x.GetString()).ToArray()); + Assert.Equal(new[] { "pkg:npm/demo@1.0.0", "pkg:npm/demo@1.1.0" }, linkset.GetProperty("purls").EnumerateArray().Select(x => x.GetString()).ToArray()); + Assert.Equal(new[] { "cpe:/a:vendor:product:1.0", "cpe:/a:vendor:product:1.1" }, linkset.GetProperty("cpes").EnumerateArray().Select(x => x.GetString()).ToArray()); + + var references = linkset.GetProperty("references").EnumerateArray().ToArray(); + Assert.Equal(2, references.Length); + Assert.Equal("advisory", references[0].GetProperty("type").GetString()); + Assert.Equal("https://example.test/advisory-1", references[0].GetProperty("url").GetString()); + Assert.Equal("patch", references[1].GetProperty("type").GetString()); + + Assert.False(root.GetProperty("hasMore").GetBoolean()); + Assert.True(root.GetProperty("nextCursor").ValueKind == JsonValueKind.Null); + } + + [Fact] + public async Task ObservationsEndpoint_AppliesObservationIdFilter() + { + await SeedObservationDocumentsAsync(BuildSampleObservationDocuments()); + + using var client = _factory.CreateClient(); + var observationId = Uri.EscapeDataString("tenant-a:ghsa:beta:1"); + var response = await client.GetAsync($"/concelier/observations?tenant=tenant-a&observationId={observationId}&cpe=cpe:/a:vendor:product:1.1"); + if (!response.IsSuccessStatusCode) + { + var body = await response.Content.ReadAsStringAsync(); + throw new XunitException($"/concelier/observations filter failed: {(int)response.StatusCode} {body}"); + } + + using var document = await response.Content.ReadFromJsonAsync(); + Assert.NotNull(document); + var root = document!.RootElement; + var observations = root.GetProperty("observations").EnumerateArray().ToArray(); + Assert.Single(observations); + Assert.Equal("tenant-a:ghsa:beta:1", observations[0].GetProperty("observationId").GetString()); + Assert.Equal(new[] { "pkg:npm/demo@1.1.0" }, observations[0].GetProperty("linkset").GetProperty("purls").EnumerateArray().Select(x => x.GetString()).ToArray()); + Assert.Equal(new[] { "cpe:/a:vendor:product:1.1" }, observations[0].GetProperty("linkset").GetProperty("cpes").EnumerateArray().Select(x => x.GetString()).ToArray()); + + Assert.False(root.GetProperty("hasMore").GetBoolean()); + Assert.True(root.GetProperty("nextCursor").ValueKind == JsonValueKind.Null); + } + + [Fact] + public async Task ObservationsEndpoint_SupportsPagination() + { + await SeedObservationDocumentsAsync(BuildSampleObservationDocuments()); + + using var client = _factory.CreateClient(); + + var firstResponse = await client.GetAsync("/concelier/observations?tenant=tenant-a&limit=1"); + firstResponse.EnsureSuccessStatusCode(); + using var firstDocument = await firstResponse.Content.ReadFromJsonAsync(); + Assert.NotNull(firstDocument); + var firstRoot = firstDocument!.RootElement; + var firstObservations = firstRoot.GetProperty("observations").EnumerateArray().ToArray(); + Assert.Single(firstObservations); + var nextCursor = firstRoot.GetProperty("nextCursor").GetString(); + Assert.True(firstRoot.GetProperty("hasMore").GetBoolean()); + Assert.False(string.IsNullOrWhiteSpace(nextCursor)); + + var secondResponse = await client.GetAsync($"/concelier/observations?tenant=tenant-a&limit=2&cursor={Uri.EscapeDataString(nextCursor!)}"); + secondResponse.EnsureSuccessStatusCode(); + using var secondDocument = await secondResponse.Content.ReadFromJsonAsync(); + Assert.NotNull(secondDocument); + var secondRoot = secondDocument!.RootElement; + var secondObservations = secondRoot.GetProperty("observations").EnumerateArray().ToArray(); + Assert.Single(secondObservations); + Assert.False(secondRoot.GetProperty("hasMore").GetBoolean()); + Assert.True(secondRoot.GetProperty("nextCursor").ValueKind == JsonValueKind.Null); + Assert.Equal("tenant-a:nvd:alpha:1", secondObservations[0].GetProperty("observationId").GetString()); + } + + [Fact] + public async Task ObservationsEndpoint_ReturnsBadRequestWhenTenantMissing() + { + using var client = _factory.CreateClient(); + var response = await client.GetAsync("/concelier/observations"); + var body = await response.Content.ReadAsStringAsync(); + Assert.True(response.StatusCode == HttpStatusCode.BadRequest, $"Expected 400 but got {(int)response.StatusCode}: {body}"); + } + + [Fact] + public async Task AdvisoryIngestEndpoint_PersistsDocumentAndSupportsReadback() + { + using var client = _factory.CreateClient(); + client.DefaultRequestHeaders.Add("X-Stella-Tenant", "tenant-ingest"); + + var ingestRequest = BuildAdvisoryIngestRequest( + contentHash: "sha256:abc123", + upstreamId: "GHSA-INGEST-0001"); + + var ingestResponse = await client.PostAsJsonAsync("/ingest/advisory", ingestRequest); + Assert.Equal(HttpStatusCode.Created, ingestResponse.StatusCode); + + var ingestPayload = await ingestResponse.Content.ReadFromJsonAsync(); + Assert.NotNull(ingestPayload); + Assert.True(ingestPayload!.Inserted); + Assert.False(string.IsNullOrWhiteSpace(ingestPayload.Id)); + Assert.Equal("tenant-ingest", ingestPayload.Tenant); + Assert.Equal("sha256:abc123", ingestPayload.ContentHash); + Assert.NotNull(ingestResponse.Headers.Location); + var locationValue = ingestResponse.Headers.Location!.ToString(); + Assert.False(string.IsNullOrWhiteSpace(locationValue)); var lastSlashIndex = locationValue.LastIndexOf('/'); - var idSegment = lastSlashIndex >= 0 - ? locationValue[(lastSlashIndex + 1)..] - : locationValue; - var decodedSegment = Uri.UnescapeDataString(idSegment); - Assert.Equal(ingestPayload.Id, decodedSegment); - - var duplicateResponse = await client.PostAsJsonAsync("/ingest/advisory", BuildAdvisoryIngestRequest( - contentHash: "sha256:abc123", - upstreamId: "GHSA-INGEST-0001")); - Assert.Equal(HttpStatusCode.OK, duplicateResponse.StatusCode); - var duplicatePayload = await duplicateResponse.Content.ReadFromJsonAsync(); - Assert.NotNull(duplicatePayload); - Assert.False(duplicatePayload!.Inserted); - - using (var getRequest = new HttpRequestMessage(HttpMethod.Get, $"/advisories/raw/{ingestPayload.Id}")) - { - getRequest.Headers.Add("X-Stella-Tenant", "tenant-ingest"); - var getResponse = await client.SendAsync(getRequest); - getResponse.EnsureSuccessStatusCode(); - - var record = await getResponse.Content.ReadFromJsonAsync(); - Assert.NotNull(record); - Assert.Equal(ingestPayload.Id, record!.Id); - Assert.Equal("tenant-ingest", record.Tenant); - Assert.Equal("sha256:abc123", record.Document.Upstream.ContentHash); - } - - using (var listRequest = new HttpRequestMessage(HttpMethod.Get, "/advisories/raw?limit=10")) - { - listRequest.Headers.Add("X-Stella-Tenant", "tenant-ingest"); - var listResponse = await client.SendAsync(listRequest); - listResponse.EnsureSuccessStatusCode(); - - var listPayload = await listResponse.Content.ReadFromJsonAsync(); - Assert.NotNull(listPayload); - var record = Assert.Single(listPayload!.Records); - Assert.Equal(ingestPayload.Id, record.Id); - } - } - - [Fact] - public async Task AocVerifyEndpoint_ReturnsSummaryForTenant() - { - using var client = _factory.CreateClient(); - client.DefaultRequestHeaders.Add("X-Stella-Tenant", "tenant-verify"); - - await client.PostAsJsonAsync("/ingest/advisory", BuildAdvisoryIngestRequest( - contentHash: "sha256:verify-1", - upstreamId: "GHSA-VERIFY-001")); - - var verifyResponse = await client.PostAsJsonAsync("/aoc/verify", new AocVerifyRequest(null, null, null, null, null)); - verifyResponse.EnsureSuccessStatusCode(); - - var verifyPayload = await verifyResponse.Content.ReadFromJsonAsync(); - Assert.NotNull(verifyPayload); - Assert.Equal("tenant-verify", verifyPayload!.Tenant); - Assert.True(verifyPayload.Checked.Advisories >= 1); - Assert.Equal(0, verifyPayload.Checked.Vex); - Assert.True(verifyPayload.Metrics.IngestionWriteTotal >= verifyPayload.Checked.Advisories); - Assert.Empty(verifyPayload.Violations); - Assert.False(verifyPayload.Truncated); - } - - [Fact] - public async Task AocVerifyEndpoint_ReturnsViolationsForGuardFailures() - { - await SeedAdvisoryRawDocumentsAsync( - CreateAdvisoryRawDocument( - tenant: "tenant-verify-violations", - vendor: "osv", - upstreamId: "GHSA-VERIFY-ERR", - contentHash: string.Empty, - raw: new BsonDocument - { - { "id", "GHSA-VERIFY-ERR" } - })); - - using var client = _factory.CreateClient(); - client.DefaultRequestHeaders.Add("X-Stella-Tenant", "tenant-verify-violations"); - - var verifyResponse = await client.PostAsJsonAsync("/aoc/verify", new AocVerifyRequest(null, null, null, null, null)); - verifyResponse.EnsureSuccessStatusCode(); - - var verifyPayload = await verifyResponse.Content.ReadFromJsonAsync(); - Assert.NotNull(verifyPayload); - Assert.Equal("tenant-verify-violations", verifyPayload!.Tenant); - Assert.True(verifyPayload.Checked.Advisories >= 1); - var violation = Assert.Single(verifyPayload.Violations); - Assert.Equal("ERR_AOC_001", violation.Code); - Assert.True(violation.Count >= 1); - Assert.NotEmpty(violation.Examples); - } - - [Fact] - public async Task AdvisoryRawListEndpoint_SupportsCursorPagination() - { - using var client = _factory.CreateClient(); - client.DefaultRequestHeaders.Add("X-Stella-Tenant", "tenant-list"); - - await client.PostAsJsonAsync("/ingest/advisory", BuildAdvisoryIngestRequest("sha256:list-1", "GHSA-LIST-001")); - await client.PostAsJsonAsync("/ingest/advisory", BuildAdvisoryIngestRequest("sha256:list-2", "GHSA-LIST-002")); - await client.PostAsJsonAsync("/ingest/advisory", BuildAdvisoryIngestRequest("sha256:list-3", "GHSA-LIST-003")); - - using var firstRequest = new HttpRequestMessage(HttpMethod.Get, "/advisories/raw?limit=2"); - firstRequest.Headers.Add("X-Stella-Tenant", "tenant-list"); - var firstResponse = await client.SendAsync(firstRequest); - firstResponse.EnsureSuccessStatusCode(); - - var firstPage = await firstResponse.Content.ReadFromJsonAsync(); - Assert.NotNull(firstPage); - Assert.Equal(2, firstPage!.Records.Count); - Assert.True(firstPage.HasMore); - Assert.False(string.IsNullOrWhiteSpace(firstPage.NextCursor)); - - using var secondRequest = new HttpRequestMessage(HttpMethod.Get, $"/advisories/raw?cursor={Uri.EscapeDataString(firstPage.NextCursor!)}"); - secondRequest.Headers.Add("X-Stella-Tenant", "tenant-list"); - var secondResponse = await client.SendAsync(secondRequest); - secondResponse.EnsureSuccessStatusCode(); - - var secondPage = await secondResponse.Content.ReadFromJsonAsync(); - Assert.NotNull(secondPage); - Assert.Single(secondPage!.Records); - Assert.False(secondPage.HasMore); - Assert.Null(secondPage.NextCursor); - - var firstIds = firstPage.Records.Select(record => record.Id).ToArray(); - var secondIds = secondPage.Records.Select(record => record.Id).ToArray(); - Assert.Empty(firstIds.Intersect(secondIds)); - } - - [Fact] - public async Task AdvisoryIngestEndpoint_EmitsMetricsWithExpectedTags() - { - var measurements = await CaptureMetricsAsync( - IngestionMetrics.MeterName, - "ingestion_write_total", - async () => - { - using var client = _factory.CreateClient(); - client.DefaultRequestHeaders.Add("X-Stella-Tenant", "tenant-metrics"); - - await client.PostAsJsonAsync("/ingest/advisory", BuildAdvisoryIngestRequest("sha256:metric-1", "GHSA-METRIC-001")); - await client.PostAsJsonAsync("/ingest/advisory", BuildAdvisoryIngestRequest("sha256:metric-1", "GHSA-METRIC-001")); - }); - - Assert.Equal(2, measurements.Count); - - var inserted = measurements.FirstOrDefault(measurement => - string.Equals(GetTagValue(measurement, "tenant"), "tenant-metrics", StringComparison.Ordinal) && - string.Equals(GetTagValue(measurement, "result"), "inserted", StringComparison.Ordinal)); - Assert.NotNull(inserted); - Assert.Equal(1, inserted!.Value); - Assert.Equal("osv", GetTagValue(inserted, "source")); - - var duplicate = measurements.FirstOrDefault(measurement => - string.Equals(GetTagValue(measurement, "tenant"), "tenant-metrics", StringComparison.Ordinal) && - string.Equals(GetTagValue(measurement, "result"), "duplicate", StringComparison.Ordinal)); - Assert.NotNull(duplicate); - Assert.Equal(1, duplicate!.Value); - Assert.Equal("osv", GetTagValue(duplicate, "source")); - } - - [Fact] - public async Task AocVerifyEndpoint_EmitsVerificationMetric() - { - var measurements = await CaptureMetricsAsync( - IngestionMetrics.MeterName, - "verify_runs_total", - async () => - { - using var client = _factory.CreateClient(); - client.DefaultRequestHeaders.Add("X-Stella-Tenant", "tenant-verify-metrics"); - - await client.PostAsJsonAsync("/ingest/advisory", BuildAdvisoryIngestRequest("sha256:verify-metric", "GHSA-VERIFY-METRIC")); - var verifyResponse = await client.PostAsJsonAsync("/aoc/verify", new AocVerifyRequest(null, null, null, null, null)); - verifyResponse.EnsureSuccessStatusCode(); - }); - - var measurement = Assert.Single(measurements); - Assert.Equal("tenant-verify-metrics", GetTagValue(measurement, "tenant")); - Assert.Equal("ok", GetTagValue(measurement, "result")); - Assert.Equal(1, measurement.Value); - } - - [Fact] - public async Task AdvisoryIngestEndpoint_RejectsCrossTenantWhenAuthenticated() - { - var environment = new Dictionary - { - ["CONCELIER_AUTHORITY__ENABLED"] = "true", - ["CONCELIER_AUTHORITY__ALLOWANONYMOUSFALLBACK"] = "false", - ["CONCELIER_AUTHORITY__ISSUER"] = TestAuthorityIssuer, - ["CONCELIER_AUTHORITY__REQUIREHTTPSMETADATA"] = "false", - ["CONCELIER_AUTHORITY__AUDIENCES__0"] = TestAuthorityAudience, - ["CONCELIER_AUTHORITY__CLIENTID"] = "webservice-tests", - ["CONCELIER_AUTHORITY__CLIENTSECRET"] = "unused", - }; - - using var factory = new ConcelierApplicationFactory( - _runner.ConnectionString, - authority => - { - authority.Enabled = true; - authority.AllowAnonymousFallback = false; - authority.Issuer = TestAuthorityIssuer; - authority.RequireHttpsMetadata = false; - authority.Audiences.Clear(); - authority.Audiences.Add(TestAuthorityAudience); - authority.ClientId = "webservice-tests"; - authority.ClientSecret = "unused"; - }, - environment); - - using var client = factory.CreateClient(); - var token = CreateTestToken("tenant-auth", StellaOpsScopes.AdvisoryIngest); - client.DefaultRequestHeaders.Authorization = new AuthenticationHeaderValue("Bearer", token); - client.DefaultRequestHeaders.Add("X-Stella-Tenant", "tenant-auth"); - - var ingestResponse = await client.PostAsJsonAsync("/ingest/advisory", BuildAdvisoryIngestRequest("sha256:auth-1", "GHSA-AUTH-001")); - Assert.Equal(HttpStatusCode.Created, ingestResponse.StatusCode); - - client.DefaultRequestHeaders.Remove("X-Stella-Tenant"); - client.DefaultRequestHeaders.Add("X-Stella-Tenant", "tenant-other"); - - var crossTenantResponse = await client.PostAsJsonAsync("/ingest/advisory", BuildAdvisoryIngestRequest("sha256:auth-2", "GHSA-AUTH-002")); - Assert.Equal(HttpStatusCode.Forbidden, crossTenantResponse.StatusCode); - } - - [Fact] - public async Task AdvisoryIngestEndpoint_ReturnsGuardViolationWhenContentHashMissing() - { - using var client = _factory.CreateClient(); - client.DefaultRequestHeaders.Add("X-Stella-Tenant", "tenant-violation"); - - var invalidRequest = BuildAdvisoryIngestRequest(contentHash: string.Empty, upstreamId: "GHSA-INVALID-1"); - var response = await client.PostAsJsonAsync("/ingest/advisory", invalidRequest); - - Assert.Equal(HttpStatusCode.UnprocessableEntity, response.StatusCode); - var problemJson = await response.Content.ReadAsStringAsync(); - using var document = JsonDocument.Parse(problemJson); - var root = document.RootElement; - Assert.Equal("Aggregation-Only Contract violation", root.GetProperty("title").GetString()); - Assert.Equal(422, root.GetProperty("status").GetInt32()); - Assert.True(root.TryGetProperty("violations", out var violations), "Problem response missing violations payload."); - Assert.True(root.TryGetProperty("code", out var codeElement), "Problem response missing code payload."); - Assert.Equal("ERR_AOC_004", codeElement.GetString()); - var violation = Assert.Single(violations.EnumerateArray()); - Assert.Equal("ERR_AOC_004", violation.GetProperty("code").GetString()); - } - - [Fact] - public async Task JobsEndpointsReturnExpectedStatuses() - { - using var client = _factory.CreateClient(); - - var definitions = await client.GetAsync("/jobs/definitions"); - if (!definitions.IsSuccessStatusCode) - { - var body = await definitions.Content.ReadAsStringAsync(); - throw new Xunit.Sdk.XunitException($"/jobs/definitions failed: {(int)definitions.StatusCode} {body}"); - } - - var trigger = await client.PostAsync("/jobs/unknown", new StringContent("{}", System.Text.Encoding.UTF8, "application/json")); - if (trigger.StatusCode != HttpStatusCode.NotFound) - { - var payload = await trigger.Content.ReadAsStringAsync(); - throw new Xunit.Sdk.XunitException($"/jobs/unknown expected 404, got {(int)trigger.StatusCode}: {payload}"); - } - var problem = await trigger.Content.ReadFromJsonAsync(); - Assert.NotNull(problem); - Assert.Equal("https://stellaops.org/problems/not-found", problem!.Type); - Assert.Equal(404, problem.Status); - } - - [Fact] - public async Task JobRunEndpointReturnsProblemWhenNotFound() - { - using var client = _factory.CreateClient(); - var response = await client.GetAsync($"/jobs/{Guid.NewGuid()}"); - if (response.StatusCode != HttpStatusCode.NotFound) - { - var body = await response.Content.ReadAsStringAsync(); - throw new Xunit.Sdk.XunitException($"/jobs/{{id}} expected 404, got {(int)response.StatusCode}: {body}"); - } - var problem = await response.Content.ReadFromJsonAsync(); - Assert.NotNull(problem); - Assert.Equal("https://stellaops.org/problems/not-found", problem!.Type); - } - - [Fact] - public async Task JobTriggerMapsCoordinatorOutcomes() - { - var handler = _factory.Services.GetRequiredService(); - using var client = _factory.CreateClient(); - - handler.NextResult = JobTriggerResult.AlreadyRunning("busy"); - var conflict = await client.PostAsync("/jobs/test", JsonContent.Create(new JobTriggerRequest())); - if (conflict.StatusCode != HttpStatusCode.Conflict) - { - var payload = await conflict.Content.ReadAsStringAsync(); - throw new Xunit.Sdk.XunitException($"Conflict path expected 409, got {(int)conflict.StatusCode}: {payload}"); - } - var conflictProblem = await conflict.Content.ReadFromJsonAsync(); - Assert.NotNull(conflictProblem); - Assert.Equal("https://stellaops.org/problems/conflict", conflictProblem!.Type); - - handler.NextResult = JobTriggerResult.Accepted(new JobRunSnapshot(Guid.NewGuid(), "demo", JobRunStatus.Pending, DateTimeOffset.UtcNow, null, null, "api", null, null, null, null, new Dictionary())); - var accepted = await client.PostAsync("/jobs/test", JsonContent.Create(new JobTriggerRequest())); - if (accepted.StatusCode != HttpStatusCode.Accepted) - { - var payload = await accepted.Content.ReadAsStringAsync(); - throw new Xunit.Sdk.XunitException($"Accepted path expected 202, got {(int)accepted.StatusCode}: {payload}"); - } - Assert.NotNull(accepted.Headers.Location); - var acceptedPayload = await accepted.Content.ReadFromJsonAsync(); - Assert.NotNull(acceptedPayload); - - handler.NextResult = JobTriggerResult.Failed(new JobRunSnapshot(Guid.NewGuid(), "demo", JobRunStatus.Failed, DateTimeOffset.UtcNow, null, DateTimeOffset.UtcNow, "api", null, "err", null, null, new Dictionary()), "boom"); - var failed = await client.PostAsync("/jobs/test", JsonContent.Create(new JobTriggerRequest())); - if (failed.StatusCode != HttpStatusCode.InternalServerError) - { - var payload = await failed.Content.ReadAsStringAsync(); - throw new Xunit.Sdk.XunitException($"Failed path expected 500, got {(int)failed.StatusCode}: {payload}"); - } - var failureProblem = await failed.Content.ReadFromJsonAsync(); - Assert.NotNull(failureProblem); - Assert.Equal("https://stellaops.org/problems/job-failure", failureProblem!.Type); - } - - [Fact] - public async Task JobsEndpointsExposeJobData() - { - var handler = _factory.Services.GetRequiredService(); - var now = DateTimeOffset.UtcNow; - var run = new JobRunSnapshot( - Guid.NewGuid(), - "demo", - JobRunStatus.Succeeded, - now, - now, - now.AddSeconds(2), - "api", - "hash", - null, - TimeSpan.FromMinutes(5), - TimeSpan.FromMinutes(1), - new Dictionary { ["key"] = "value" }); - - handler.Definitions = new[] - { - new JobDefinition("demo", typeof(DemoJob), TimeSpan.FromMinutes(5), TimeSpan.FromMinutes(1), "*/5 * * * *", true) - }; - handler.LastRuns["demo"] = run; - handler.RecentRuns = new[] { run }; - handler.ActiveRuns = Array.Empty(); - handler.Runs[run.RunId] = run; - - try - { - using var client = _factory.CreateClient(); - - var definitions = await client.GetFromJsonAsync>("/jobs/definitions"); - Assert.NotNull(definitions); - Assert.Single(definitions!); - Assert.Equal("demo", definitions![0].Kind); - Assert.NotNull(definitions[0].LastRun); - Assert.Equal(run.RunId, definitions[0].LastRun!.RunId); - - var runPayload = await client.GetFromJsonAsync($"/jobs/{run.RunId}"); - Assert.NotNull(runPayload); - Assert.Equal(run.RunId, runPayload!.RunId); - Assert.Equal("Succeeded", runPayload.Status); - - var runs = await client.GetFromJsonAsync>("/jobs?kind=demo&limit=5"); - Assert.NotNull(runs); - Assert.Single(runs!); - Assert.Equal(run.RunId, runs![0].RunId); - - var runsByDefinition = await client.GetFromJsonAsync>("/jobs/definitions/demo/runs"); - Assert.NotNull(runsByDefinition); - Assert.Single(runsByDefinition!); - - var active = await client.GetFromJsonAsync>("/jobs/active"); - Assert.NotNull(active); - Assert.Empty(active!); - } - finally - { - handler.Definitions = Array.Empty(); - handler.RecentRuns = Array.Empty(); - handler.ActiveRuns = Array.Empty(); - handler.Runs.Clear(); - handler.LastRuns.Clear(); - } - } - - [Fact] - public async Task AdvisoryReplayEndpointReturnsLatestStatement() - { - var vulnerabilityKey = "CVE-2025-9000"; - var advisory = new Advisory( - advisoryKey: vulnerabilityKey, - title: "Replay Test", - summary: "Example summary", - language: "en", - published: DateTimeOffset.Parse("2025-01-01T00:00:00Z", CultureInfo.InvariantCulture), - modified: DateTimeOffset.Parse("2025-01-02T00:00:00Z", CultureInfo.InvariantCulture), - severity: "medium", - exploitKnown: false, - aliases: new[] { vulnerabilityKey }, - references: Array.Empty(), - affectedPackages: Array.Empty(), - cvssMetrics: Array.Empty(), - provenance: Array.Empty()); - - var statementId = Guid.NewGuid(); - using (var scope = _factory.Services.CreateScope()) - { - var eventLog = scope.ServiceProvider.GetRequiredService(); - var appendRequest = new AdvisoryEventAppendRequest(new[] - { - new AdvisoryStatementInput( - vulnerabilityKey, - advisory, - advisory.Modified ?? advisory.Published ?? DateTimeOffset.UtcNow, - Array.Empty(), - StatementId: statementId, - AdvisoryKey: advisory.AdvisoryKey) - }); - - await eventLog.AppendAsync(appendRequest, CancellationToken.None); - } - - using var client = _factory.CreateClient(); - var response = await client.GetAsync($"/concelier/advisories/{vulnerabilityKey}/replay"); - - Assert.Equal(HttpStatusCode.OK, response.StatusCode); - var payload = await response.Content.ReadFromJsonAsync(); - Assert.NotNull(payload); - Assert.Equal(vulnerabilityKey, payload!.VulnerabilityKey, ignoreCase: true); - var statement = Assert.Single(payload.Statements); - Assert.Equal(statementId, statement.StatementId); - Assert.Equal(advisory.AdvisoryKey, statement.Advisory.AdvisoryKey); - Assert.False(string.IsNullOrWhiteSpace(statement.StatementHash)); - Assert.True(payload.Conflicts is null || payload.Conflicts!.Count == 0); - } - - [Fact] - public async Task AdvisoryReplayEndpointReturnsConflictExplainer() - { - var vulnerabilityKey = "CVE-2025-9100"; - var statementId = Guid.NewGuid(); - var conflictId = Guid.NewGuid(); - var recordedAt = DateTimeOffset.Parse("2025-02-01T00:00:00Z", CultureInfo.InvariantCulture); - - using (var scope = _factory.Services.CreateScope()) - { - var eventLog = scope.ServiceProvider.GetRequiredService(); - var advisory = new Advisory( - advisoryKey: vulnerabilityKey, - title: "Base advisory", - summary: "Baseline summary", - language: "en", - published: recordedAt.AddDays(-1), - modified: recordedAt, - severity: "critical", - exploitKnown: false, - aliases: new[] { vulnerabilityKey }, - references: Array.Empty(), - affectedPackages: Array.Empty(), - cvssMetrics: Array.Empty(), - provenance: Array.Empty()); - - var statementInput = new AdvisoryStatementInput( - vulnerabilityKey, - advisory, - recordedAt, - Array.Empty(), - StatementId: statementId, - AdvisoryKey: advisory.AdvisoryKey); - - await eventLog.AppendAsync(new AdvisoryEventAppendRequest(new[] { statementInput }), CancellationToken.None); - - var explainer = new MergeConflictExplainerPayload( - Type: "severity", - Reason: "mismatch", - PrimarySources: new[] { "vendor" }, - PrimaryRank: 1, - SuppressedSources: new[] { "nvd" }, - SuppressedRank: 5, - PrimaryValue: "CRITICAL", - SuppressedValue: "MEDIUM"); - - using var conflictDoc = JsonDocument.Parse(explainer.ToCanonicalJson()); - var conflictInput = new AdvisoryConflictInput( - vulnerabilityKey, - conflictDoc, - recordedAt, - new[] { statementId }, - ConflictId: conflictId); - - await eventLog.AppendAsync(new AdvisoryEventAppendRequest(Array.Empty(), new[] { conflictInput }), CancellationToken.None); - } - - using var client = _factory.CreateClient(); - var response = await client.GetAsync($"/concelier/advisories/{vulnerabilityKey}/replay"); - Assert.Equal(HttpStatusCode.OK, response.StatusCode); - var payload = await response.Content.ReadFromJsonAsync(); - Assert.NotNull(payload); - var conflict = Assert.Single(payload!.Conflicts); - Assert.Equal(conflictId, conflict.ConflictId); - Assert.Equal("severity", conflict.Explainer.Type); - Assert.Equal("mismatch", conflict.Explainer.Reason); - Assert.Equal("CRITICAL", conflict.Explainer.PrimaryValue); - Assert.Equal("MEDIUM", conflict.Explainer.SuppressedValue); - Assert.Equal(conflict.Explainer.ComputeHashHex(), conflict.ConflictHash); - } - - [Fact] - public async Task MirrorEndpointsServeConfiguredArtifacts() - { - using var temp = new TempDirectory(); - var exportId = "20251019T120000Z"; - var exportRoot = Path.Combine(temp.Path, exportId); - var mirrorRoot = Path.Combine(exportRoot, "mirror"); - var domainRoot = Path.Combine(mirrorRoot, "primary"); - Directory.CreateDirectory(domainRoot); - - await File.WriteAllTextAsync( - Path.Combine(mirrorRoot, "index.json"), - """{"schemaVersion":1,"domains":[]}"""); - await File.WriteAllTextAsync( - Path.Combine(domainRoot, "manifest.json"), - """{"domainId":"primary"}"""); - await File.WriteAllTextAsync( - Path.Combine(domainRoot, "bundle.json"), - """{"advisories":[]}"""); - await File.WriteAllTextAsync( - Path.Combine(domainRoot, "bundle.json.jws"), - "test-signature"); - - var environment = new Dictionary - { - ["CONCELIER_MIRROR__ENABLED"] = "true", - ["CONCELIER_MIRROR__EXPORTROOT"] = temp.Path, - ["CONCELIER_MIRROR__ACTIVEEXPORTID"] = exportId, - ["CONCELIER_MIRROR__DOMAINS__0__ID"] = "primary", - ["CONCELIER_MIRROR__DOMAINS__0__DISPLAYNAME"] = "Primary", - ["CONCELIER_MIRROR__DOMAINS__0__REQUIREAUTHENTICATION"] = "false", - ["CONCELIER_MIRROR__DOMAINS__0__MAXDOWNLOADREQUESTSPERHOUR"] = "5", - ["CONCELIER_MIRROR__MAXINDEXREQUESTSPERHOUR"] = "5" - }; - - using var factory = new ConcelierApplicationFactory(_runner.ConnectionString, environmentOverrides: environment); - using var client = factory.CreateClient(); - - var indexResponse = await client.GetAsync("/concelier/exports/index.json"); - Assert.Equal(HttpStatusCode.OK, indexResponse.StatusCode); - var indexContent = await indexResponse.Content.ReadAsStringAsync(); - Assert.Contains(@"""schemaVersion"":1", indexContent, StringComparison.Ordinal); - - var manifestResponse = await client.GetAsync("/concelier/exports/mirror/primary/manifest.json"); - Assert.Equal(HttpStatusCode.OK, manifestResponse.StatusCode); - var manifestContent = await manifestResponse.Content.ReadAsStringAsync(); - Assert.Contains(@"""domainId"":""primary""", manifestContent, StringComparison.Ordinal); - - var bundleResponse = await client.GetAsync("/concelier/exports/mirror/primary/bundle.json.jws"); - Assert.Equal(HttpStatusCode.OK, bundleResponse.StatusCode); - var signatureContent = await bundleResponse.Content.ReadAsStringAsync(); - Assert.Equal("test-signature", signatureContent); - } - - [Fact] - public async Task MirrorEndpointsEnforceAuthenticationForProtectedDomains() - { - using var temp = new TempDirectory(); - var exportId = "20251019T120000Z"; - var secureRoot = Path.Combine(temp.Path, exportId, "mirror", "secure"); - Directory.CreateDirectory(secureRoot); - - await File.WriteAllTextAsync( - Path.Combine(temp.Path, exportId, "mirror", "index.json"), - """{"schemaVersion":1,"domains":[]}"""); - await File.WriteAllTextAsync( - Path.Combine(secureRoot, "manifest.json"), - """{"domainId":"secure"}"""); - - var environment = new Dictionary - { - ["CONCELIER_MIRROR__ENABLED"] = "true", - ["CONCELIER_MIRROR__EXPORTROOT"] = temp.Path, - ["CONCELIER_MIRROR__ACTIVEEXPORTID"] = exportId, - ["CONCELIER_MIRROR__DOMAINS__0__ID"] = "secure", - ["CONCELIER_MIRROR__DOMAINS__0__REQUIREAUTHENTICATION"] = "true", - ["CONCELIER_MIRROR__DOMAINS__0__MAXDOWNLOADREQUESTSPERHOUR"] = "5", - ["CONCELIER_AUTHORITY__ENABLED"] = "true", - ["CONCELIER_AUTHORITY__ALLOWANONYMOUSFALLBACK"] = "false", - ["CONCELIER_AUTHORITY__ISSUER"] = "https://authority.example", - ["CONCELIER_AUTHORITY__REQUIREHTTPSMETADATA"] = "false", - ["CONCELIER_AUTHORITY__AUDIENCES__0"] = "api://concelier", - ["CONCELIER_AUTHORITY__REQUIREDSCOPES__0"] = StellaOpsScopes.ConcelierJobsTrigger, - ["CONCELIER_AUTHORITY__CLIENTID"] = "concelier-jobs", - ["CONCELIER_AUTHORITY__CLIENTSECRET"] = "secret", - ["CONCELIER_AUTHORITY__CLIENTSCOPES__0"] = StellaOpsScopes.ConcelierJobsTrigger - }; - - using var factory = new ConcelierApplicationFactory( - _runner.ConnectionString, - authority => - { - authority.Enabled = true; - authority.AllowAnonymousFallback = false; - authority.Issuer = "https://authority.example"; - authority.RequireHttpsMetadata = false; - authority.Audiences.Clear(); - authority.Audiences.Add("api://concelier"); - authority.RequiredScopes.Clear(); - authority.RequiredScopes.Add(StellaOpsScopes.ConcelierJobsTrigger); - authority.ClientId = "concelier-jobs"; - authority.ClientSecret = "secret"; - }, - environment); - - using var client = factory.CreateClient(); - var response = await client.GetAsync("/concelier/exports/mirror/secure/manifest.json"); - Assert.Equal(HttpStatusCode.Unauthorized, response.StatusCode); - var authHeader = Assert.Single(response.Headers.WwwAuthenticate); - Assert.Equal("Bearer", authHeader.Scheme); - } - - [Fact] - public async Task MirrorEndpointsRespectRateLimits() - { - using var temp = new TempDirectory(); - var exportId = "20251019T130000Z"; - var exportRoot = Path.Combine(temp.Path, exportId); - var mirrorRoot = Path.Combine(exportRoot, "mirror"); - Directory.CreateDirectory(mirrorRoot); - - await File.WriteAllTextAsync( - Path.Combine(mirrorRoot, "index.json"), - """{\"schemaVersion\":1,\"domains\":[]}""" - ); - - var environment = new Dictionary - { - ["CONCELIER_MIRROR__ENABLED"] = "true", - ["CONCELIER_MIRROR__EXPORTROOT"] = temp.Path, - ["CONCELIER_MIRROR__ACTIVEEXPORTID"] = exportId, - ["CONCELIER_MIRROR__MAXINDEXREQUESTSPERHOUR"] = "1", - ["CONCELIER_MIRROR__DOMAINS__0__ID"] = "primary", - ["CONCELIER_MIRROR__DOMAINS__0__REQUIREAUTHENTICATION"] = "false", - ["CONCELIER_MIRROR__DOMAINS__0__MAXDOWNLOADREQUESTSPERHOUR"] = "1" - }; - - using var factory = new ConcelierApplicationFactory(_runner.ConnectionString, environmentOverrides: environment); - using var client = factory.CreateClient(); - - var okResponse = await client.GetAsync("/concelier/exports/index.json"); - Assert.Equal(HttpStatusCode.OK, okResponse.StatusCode); - - var limitedResponse = await client.GetAsync("/concelier/exports/index.json"); - Assert.Equal((HttpStatusCode)429, limitedResponse.StatusCode); - Assert.NotNull(limitedResponse.Headers.RetryAfter); + var idSegment = lastSlashIndex >= 0 + ? locationValue[(lastSlashIndex + 1)..] + : locationValue; + var decodedSegment = Uri.UnescapeDataString(idSegment); + Assert.Equal(ingestPayload.Id, decodedSegment); + + var duplicateResponse = await client.PostAsJsonAsync("/ingest/advisory", BuildAdvisoryIngestRequest( + contentHash: "sha256:abc123", + upstreamId: "GHSA-INGEST-0001")); + Assert.Equal(HttpStatusCode.OK, duplicateResponse.StatusCode); + var duplicatePayload = await duplicateResponse.Content.ReadFromJsonAsync(); + Assert.NotNull(duplicatePayload); + Assert.False(duplicatePayload!.Inserted); + + using (var getRequest = new HttpRequestMessage(HttpMethod.Get, $"/advisories/raw/{ingestPayload.Id}")) + { + getRequest.Headers.Add("X-Stella-Tenant", "tenant-ingest"); + var getResponse = await client.SendAsync(getRequest); + getResponse.EnsureSuccessStatusCode(); + + var record = await getResponse.Content.ReadFromJsonAsync(); + Assert.NotNull(record); + Assert.Equal(ingestPayload.Id, record!.Id); + Assert.Equal("tenant-ingest", record.Tenant); + Assert.Equal("sha256:abc123", record.Document.Upstream.ContentHash); + } + + using (var listRequest = new HttpRequestMessage(HttpMethod.Get, "/advisories/raw?limit=10")) + { + listRequest.Headers.Add("X-Stella-Tenant", "tenant-ingest"); + var listResponse = await client.SendAsync(listRequest); + listResponse.EnsureSuccessStatusCode(); + + var listPayload = await listResponse.Content.ReadFromJsonAsync(); + Assert.NotNull(listPayload); + var record = Assert.Single(listPayload!.Records); + Assert.Equal(ingestPayload.Id, record.Id); + } + } + + [Fact] + public async Task AocVerifyEndpoint_ReturnsSummaryForTenant() + { + using var client = _factory.CreateClient(); + client.DefaultRequestHeaders.Add("X-Stella-Tenant", "tenant-verify"); + + await client.PostAsJsonAsync("/ingest/advisory", BuildAdvisoryIngestRequest( + contentHash: "sha256:verify-1", + upstreamId: "GHSA-VERIFY-001")); + + var verifyResponse = await client.PostAsJsonAsync("/aoc/verify", new AocVerifyRequest(null, null, null, null, null)); + verifyResponse.EnsureSuccessStatusCode(); + + var verifyPayload = await verifyResponse.Content.ReadFromJsonAsync(); + Assert.NotNull(verifyPayload); + Assert.Equal("tenant-verify", verifyPayload!.Tenant); + Assert.True(verifyPayload.Checked.Advisories >= 1); + Assert.Equal(0, verifyPayload.Checked.Vex); + Assert.True(verifyPayload.Metrics.IngestionWriteTotal >= verifyPayload.Checked.Advisories); + Assert.Empty(verifyPayload.Violations); + Assert.False(verifyPayload.Truncated); + } + + [Fact] + public async Task AocVerifyEndpoint_ReturnsViolationsForGuardFailures() + { + await SeedAdvisoryRawDocumentsAsync( + CreateAdvisoryRawDocument( + tenant: "tenant-verify-violations", + vendor: "osv", + upstreamId: "GHSA-VERIFY-ERR", + contentHash: string.Empty, + raw: new BsonDocument + { + { "id", "GHSA-VERIFY-ERR" } + })); + + using var client = _factory.CreateClient(); + client.DefaultRequestHeaders.Add("X-Stella-Tenant", "tenant-verify-violations"); + + var verifyResponse = await client.PostAsJsonAsync("/aoc/verify", new AocVerifyRequest(null, null, null, null, null)); + verifyResponse.EnsureSuccessStatusCode(); + + var verifyPayload = await verifyResponse.Content.ReadFromJsonAsync(); + Assert.NotNull(verifyPayload); + Assert.Equal("tenant-verify-violations", verifyPayload!.Tenant); + Assert.True(verifyPayload.Checked.Advisories >= 1); + var violation = Assert.Single(verifyPayload.Violations); + Assert.Equal("ERR_AOC_001", violation.Code); + Assert.True(violation.Count >= 1); + Assert.NotEmpty(violation.Examples); + } + + [Fact] + public async Task AdvisoryRawListEndpoint_SupportsCursorPagination() + { + using var client = _factory.CreateClient(); + client.DefaultRequestHeaders.Add("X-Stella-Tenant", "tenant-list"); + + await client.PostAsJsonAsync("/ingest/advisory", BuildAdvisoryIngestRequest("sha256:list-1", "GHSA-LIST-001")); + await client.PostAsJsonAsync("/ingest/advisory", BuildAdvisoryIngestRequest("sha256:list-2", "GHSA-LIST-002")); + await client.PostAsJsonAsync("/ingest/advisory", BuildAdvisoryIngestRequest("sha256:list-3", "GHSA-LIST-003")); + + using var firstRequest = new HttpRequestMessage(HttpMethod.Get, "/advisories/raw?limit=2"); + firstRequest.Headers.Add("X-Stella-Tenant", "tenant-list"); + var firstResponse = await client.SendAsync(firstRequest); + firstResponse.EnsureSuccessStatusCode(); + + var firstPage = await firstResponse.Content.ReadFromJsonAsync(); + Assert.NotNull(firstPage); + Assert.Equal(2, firstPage!.Records.Count); + Assert.True(firstPage.HasMore); + Assert.False(string.IsNullOrWhiteSpace(firstPage.NextCursor)); + + using var secondRequest = new HttpRequestMessage(HttpMethod.Get, $"/advisories/raw?cursor={Uri.EscapeDataString(firstPage.NextCursor!)}"); + secondRequest.Headers.Add("X-Stella-Tenant", "tenant-list"); + var secondResponse = await client.SendAsync(secondRequest); + secondResponse.EnsureSuccessStatusCode(); + + var secondPage = await secondResponse.Content.ReadFromJsonAsync(); + Assert.NotNull(secondPage); + Assert.Single(secondPage!.Records); + Assert.False(secondPage.HasMore); + Assert.Null(secondPage.NextCursor); + + var firstIds = firstPage.Records.Select(record => record.Id).ToArray(); + var secondIds = secondPage.Records.Select(record => record.Id).ToArray(); + Assert.Empty(firstIds.Intersect(secondIds)); + } + + [Fact] + public async Task AdvisoryIngestEndpoint_EmitsMetricsWithExpectedTags() + { + var measurements = await CaptureMetricsAsync( + IngestionMetrics.MeterName, + "ingestion_write_total", + async () => + { + using var client = _factory.CreateClient(); + client.DefaultRequestHeaders.Add("X-Stella-Tenant", "tenant-metrics"); + + await client.PostAsJsonAsync("/ingest/advisory", BuildAdvisoryIngestRequest("sha256:metric-1", "GHSA-METRIC-001")); + await client.PostAsJsonAsync("/ingest/advisory", BuildAdvisoryIngestRequest("sha256:metric-1", "GHSA-METRIC-001")); + }); + + Assert.Equal(2, measurements.Count); + + var inserted = measurements.FirstOrDefault(measurement => + string.Equals(GetTagValue(measurement, "tenant"), "tenant-metrics", StringComparison.Ordinal) && + string.Equals(GetTagValue(measurement, "result"), "inserted", StringComparison.Ordinal)); + Assert.NotNull(inserted); + Assert.Equal(1, inserted!.Value); + Assert.Equal("osv", GetTagValue(inserted, "source")); + + var duplicate = measurements.FirstOrDefault(measurement => + string.Equals(GetTagValue(measurement, "tenant"), "tenant-metrics", StringComparison.Ordinal) && + string.Equals(GetTagValue(measurement, "result"), "duplicate", StringComparison.Ordinal)); + Assert.NotNull(duplicate); + Assert.Equal(1, duplicate!.Value); + Assert.Equal("osv", GetTagValue(duplicate, "source")); + } + + [Fact] + public async Task AocVerifyEndpoint_EmitsVerificationMetric() + { + var measurements = await CaptureMetricsAsync( + IngestionMetrics.MeterName, + "verify_runs_total", + async () => + { + using var client = _factory.CreateClient(); + client.DefaultRequestHeaders.Add("X-Stella-Tenant", "tenant-verify-metrics"); + + await client.PostAsJsonAsync("/ingest/advisory", BuildAdvisoryIngestRequest("sha256:verify-metric", "GHSA-VERIFY-METRIC")); + var verifyResponse = await client.PostAsJsonAsync("/aoc/verify", new AocVerifyRequest(null, null, null, null, null)); + verifyResponse.EnsureSuccessStatusCode(); + }); + + var measurement = Assert.Single(measurements); + Assert.Equal("tenant-verify-metrics", GetTagValue(measurement, "tenant")); + Assert.Equal("ok", GetTagValue(measurement, "result")); + Assert.Equal(1, measurement.Value); + } + + [Fact] + public async Task AdvisoryIngestEndpoint_RejectsCrossTenantWhenAuthenticated() + { + var environment = new Dictionary + { + ["CONCELIER_AUTHORITY__ENABLED"] = "true", + ["CONCELIER_AUTHORITY__ALLOWANONYMOUSFALLBACK"] = "false", + ["CONCELIER_AUTHORITY__ISSUER"] = TestAuthorityIssuer, + ["CONCELIER_AUTHORITY__REQUIREHTTPSMETADATA"] = "false", + ["CONCELIER_AUTHORITY__AUDIENCES__0"] = TestAuthorityAudience, + ["CONCELIER_AUTHORITY__CLIENTID"] = "webservice-tests", + ["CONCELIER_AUTHORITY__CLIENTSECRET"] = "unused", + }; + + using var factory = new ConcelierApplicationFactory( + _runner.ConnectionString, + authority => + { + authority.Enabled = true; + authority.AllowAnonymousFallback = false; + authority.Issuer = TestAuthorityIssuer; + authority.RequireHttpsMetadata = false; + authority.Audiences.Clear(); + authority.Audiences.Add(TestAuthorityAudience); + authority.ClientId = "webservice-tests"; + authority.ClientSecret = "unused"; + }, + environment); + + using var client = factory.CreateClient(); + var token = CreateTestToken("tenant-auth", StellaOpsScopes.AdvisoryIngest); + client.DefaultRequestHeaders.Authorization = new AuthenticationHeaderValue("Bearer", token); + client.DefaultRequestHeaders.Add("X-Stella-Tenant", "tenant-auth"); + + var ingestResponse = await client.PostAsJsonAsync("/ingest/advisory", BuildAdvisoryIngestRequest("sha256:auth-1", "GHSA-AUTH-001")); + Assert.Equal(HttpStatusCode.Created, ingestResponse.StatusCode); + + client.DefaultRequestHeaders.Remove("X-Stella-Tenant"); + client.DefaultRequestHeaders.Add("X-Stella-Tenant", "tenant-other"); + + var crossTenantResponse = await client.PostAsJsonAsync("/ingest/advisory", BuildAdvisoryIngestRequest("sha256:auth-2", "GHSA-AUTH-002")); + Assert.Equal(HttpStatusCode.Forbidden, crossTenantResponse.StatusCode); + } + + [Fact] + public async Task AdvisoryIngestEndpoint_ReturnsGuardViolationWhenContentHashMissing() + { + using var client = _factory.CreateClient(); + client.DefaultRequestHeaders.Add("X-Stella-Tenant", "tenant-violation"); + + var invalidRequest = BuildAdvisoryIngestRequest(contentHash: string.Empty, upstreamId: "GHSA-INVALID-1"); + var response = await client.PostAsJsonAsync("/ingest/advisory", invalidRequest); + + Assert.Equal(HttpStatusCode.UnprocessableEntity, response.StatusCode); + var problemJson = await response.Content.ReadAsStringAsync(); + using var document = JsonDocument.Parse(problemJson); + var root = document.RootElement; + Assert.Equal("Aggregation-Only Contract violation", root.GetProperty("title").GetString()); + Assert.Equal(422, root.GetProperty("status").GetInt32()); + Assert.True(root.TryGetProperty("violations", out var violations), "Problem response missing violations payload."); + Assert.True(root.TryGetProperty("code", out var codeElement), "Problem response missing code payload."); + Assert.Equal("ERR_AOC_004", codeElement.GetString()); + var violation = Assert.Single(violations.EnumerateArray()); + Assert.Equal("ERR_AOC_004", violation.GetProperty("code").GetString()); + } + + [Fact] + public async Task JobsEndpointsReturnExpectedStatuses() + { + using var client = _factory.CreateClient(); + + var definitions = await client.GetAsync("/jobs/definitions"); + if (!definitions.IsSuccessStatusCode) + { + var body = await definitions.Content.ReadAsStringAsync(); + throw new Xunit.Sdk.XunitException($"/jobs/definitions failed: {(int)definitions.StatusCode} {body}"); + } + + var trigger = await client.PostAsync("/jobs/unknown", new StringContent("{}", System.Text.Encoding.UTF8, "application/json")); + if (trigger.StatusCode != HttpStatusCode.NotFound) + { + var payload = await trigger.Content.ReadAsStringAsync(); + throw new Xunit.Sdk.XunitException($"/jobs/unknown expected 404, got {(int)trigger.StatusCode}: {payload}"); + } + var problem = await trigger.Content.ReadFromJsonAsync(); + Assert.NotNull(problem); + Assert.Equal("https://stellaops.org/problems/not-found", problem!.Type); + Assert.Equal(404, problem.Status); + } + + [Fact] + public async Task JobRunEndpointReturnsProblemWhenNotFound() + { + using var client = _factory.CreateClient(); + var response = await client.GetAsync($"/jobs/{Guid.NewGuid()}"); + if (response.StatusCode != HttpStatusCode.NotFound) + { + var body = await response.Content.ReadAsStringAsync(); + throw new Xunit.Sdk.XunitException($"/jobs/{{id}} expected 404, got {(int)response.StatusCode}: {body}"); + } + var problem = await response.Content.ReadFromJsonAsync(); + Assert.NotNull(problem); + Assert.Equal("https://stellaops.org/problems/not-found", problem!.Type); + } + + [Fact] + public async Task JobTriggerMapsCoordinatorOutcomes() + { + var handler = _factory.Services.GetRequiredService(); + using var client = _factory.CreateClient(); + + handler.NextResult = JobTriggerResult.AlreadyRunning("busy"); + var conflict = await client.PostAsync("/jobs/test", JsonContent.Create(new JobTriggerRequest())); + if (conflict.StatusCode != HttpStatusCode.Conflict) + { + var payload = await conflict.Content.ReadAsStringAsync(); + throw new Xunit.Sdk.XunitException($"Conflict path expected 409, got {(int)conflict.StatusCode}: {payload}"); + } + var conflictProblem = await conflict.Content.ReadFromJsonAsync(); + Assert.NotNull(conflictProblem); + Assert.Equal("https://stellaops.org/problems/conflict", conflictProblem!.Type); + + handler.NextResult = JobTriggerResult.Accepted(new JobRunSnapshot(Guid.NewGuid(), "demo", JobRunStatus.Pending, DateTimeOffset.UtcNow, null, null, "api", null, null, null, null, new Dictionary())); + var accepted = await client.PostAsync("/jobs/test", JsonContent.Create(new JobTriggerRequest())); + if (accepted.StatusCode != HttpStatusCode.Accepted) + { + var payload = await accepted.Content.ReadAsStringAsync(); + throw new Xunit.Sdk.XunitException($"Accepted path expected 202, got {(int)accepted.StatusCode}: {payload}"); + } + Assert.NotNull(accepted.Headers.Location); + var acceptedPayload = await accepted.Content.ReadFromJsonAsync(); + Assert.NotNull(acceptedPayload); + + handler.NextResult = JobTriggerResult.Failed(new JobRunSnapshot(Guid.NewGuid(), "demo", JobRunStatus.Failed, DateTimeOffset.UtcNow, null, DateTimeOffset.UtcNow, "api", null, "err", null, null, new Dictionary()), "boom"); + var failed = await client.PostAsync("/jobs/test", JsonContent.Create(new JobTriggerRequest())); + if (failed.StatusCode != HttpStatusCode.InternalServerError) + { + var payload = await failed.Content.ReadAsStringAsync(); + throw new Xunit.Sdk.XunitException($"Failed path expected 500, got {(int)failed.StatusCode}: {payload}"); + } + var failureProblem = await failed.Content.ReadFromJsonAsync(); + Assert.NotNull(failureProblem); + Assert.Equal("https://stellaops.org/problems/job-failure", failureProblem!.Type); + } + + [Fact] + public async Task JobsEndpointsExposeJobData() + { + var handler = _factory.Services.GetRequiredService(); + var now = DateTimeOffset.UtcNow; + var run = new JobRunSnapshot( + Guid.NewGuid(), + "demo", + JobRunStatus.Succeeded, + now, + now, + now.AddSeconds(2), + "api", + "hash", + null, + TimeSpan.FromMinutes(5), + TimeSpan.FromMinutes(1), + new Dictionary { ["key"] = "value" }); + + handler.Definitions = new[] + { + new JobDefinition("demo", typeof(DemoJob), TimeSpan.FromMinutes(5), TimeSpan.FromMinutes(1), "*/5 * * * *", true) + }; + handler.LastRuns["demo"] = run; + handler.RecentRuns = new[] { run }; + handler.ActiveRuns = Array.Empty(); + handler.Runs[run.RunId] = run; + + try + { + using var client = _factory.CreateClient(); + + var definitions = await client.GetFromJsonAsync>("/jobs/definitions"); + Assert.NotNull(definitions); + Assert.Single(definitions!); + Assert.Equal("demo", definitions![0].Kind); + Assert.NotNull(definitions[0].LastRun); + Assert.Equal(run.RunId, definitions[0].LastRun!.RunId); + + var runPayload = await client.GetFromJsonAsync($"/jobs/{run.RunId}"); + Assert.NotNull(runPayload); + Assert.Equal(run.RunId, runPayload!.RunId); + Assert.Equal("Succeeded", runPayload.Status); + + var runs = await client.GetFromJsonAsync>("/jobs?kind=demo&limit=5"); + Assert.NotNull(runs); + Assert.Single(runs!); + Assert.Equal(run.RunId, runs![0].RunId); + + var runsByDefinition = await client.GetFromJsonAsync>("/jobs/definitions/demo/runs"); + Assert.NotNull(runsByDefinition); + Assert.Single(runsByDefinition!); + + var active = await client.GetFromJsonAsync>("/jobs/active"); + Assert.NotNull(active); + Assert.Empty(active!); + } + finally + { + handler.Definitions = Array.Empty(); + handler.RecentRuns = Array.Empty(); + handler.ActiveRuns = Array.Empty(); + handler.Runs.Clear(); + handler.LastRuns.Clear(); + } + } + + [Fact] + public async Task AdvisoryReplayEndpointReturnsLatestStatement() + { + var vulnerabilityKey = "CVE-2025-9000"; + var advisory = new Advisory( + advisoryKey: vulnerabilityKey, + title: "Replay Test", + summary: "Example summary", + language: "en", + published: DateTimeOffset.Parse("2025-01-01T00:00:00Z", CultureInfo.InvariantCulture), + modified: DateTimeOffset.Parse("2025-01-02T00:00:00Z", CultureInfo.InvariantCulture), + severity: "medium", + exploitKnown: false, + aliases: new[] { vulnerabilityKey }, + references: Array.Empty(), + affectedPackages: Array.Empty(), + cvssMetrics: Array.Empty(), + provenance: Array.Empty()); + + var statementId = Guid.NewGuid(); + using (var scope = _factory.Services.CreateScope()) + { + var eventLog = scope.ServiceProvider.GetRequiredService(); + var appendRequest = new AdvisoryEventAppendRequest(new[] + { + new AdvisoryStatementInput( + vulnerabilityKey, + advisory, + advisory.Modified ?? advisory.Published ?? DateTimeOffset.UtcNow, + Array.Empty(), + StatementId: statementId, + AdvisoryKey: advisory.AdvisoryKey) + }); + + await eventLog.AppendAsync(appendRequest, CancellationToken.None); + } + + using var client = _factory.CreateClient(); + var response = await client.GetAsync($"/concelier/advisories/{vulnerabilityKey}/replay"); + + Assert.Equal(HttpStatusCode.OK, response.StatusCode); + var payload = await response.Content.ReadFromJsonAsync(); + Assert.NotNull(payload); + Assert.Equal(vulnerabilityKey, payload!.VulnerabilityKey, ignoreCase: true); + var statement = Assert.Single(payload.Statements); + Assert.Equal(statementId, statement.StatementId); + Assert.Equal(advisory.AdvisoryKey, statement.Advisory.AdvisoryKey); + Assert.False(string.IsNullOrWhiteSpace(statement.StatementHash)); + Assert.True(payload.Conflicts is null || payload.Conflicts!.Count == 0); + } + + [Fact] + public async Task AdvisoryReplayEndpointReturnsConflictExplainer() + { + var vulnerabilityKey = "CVE-2025-9100"; + var statementId = Guid.NewGuid(); + var conflictId = Guid.NewGuid(); + var recordedAt = DateTimeOffset.Parse("2025-02-01T00:00:00Z", CultureInfo.InvariantCulture); + + using (var scope = _factory.Services.CreateScope()) + { + var eventLog = scope.ServiceProvider.GetRequiredService(); + var advisory = new Advisory( + advisoryKey: vulnerabilityKey, + title: "Base advisory", + summary: "Baseline summary", + language: "en", + published: recordedAt.AddDays(-1), + modified: recordedAt, + severity: "critical", + exploitKnown: false, + aliases: new[] { vulnerabilityKey }, + references: Array.Empty(), + affectedPackages: Array.Empty(), + cvssMetrics: Array.Empty(), + provenance: Array.Empty()); + + var statementInput = new AdvisoryStatementInput( + vulnerabilityKey, + advisory, + recordedAt, + Array.Empty(), + StatementId: statementId, + AdvisoryKey: advisory.AdvisoryKey); + + await eventLog.AppendAsync(new AdvisoryEventAppendRequest(new[] { statementInput }), CancellationToken.None); + + var explainer = new MergeConflictExplainerPayload( + Type: "severity", + Reason: "mismatch", + PrimarySources: new[] { "vendor" }, + PrimaryRank: 1, + SuppressedSources: new[] { "nvd" }, + SuppressedRank: 5, + PrimaryValue: "CRITICAL", + SuppressedValue: "MEDIUM"); + + using var conflictDoc = JsonDocument.Parse(explainer.ToCanonicalJson()); + var conflictInput = new AdvisoryConflictInput( + vulnerabilityKey, + conflictDoc, + recordedAt, + new[] { statementId }, + ConflictId: conflictId); + + await eventLog.AppendAsync(new AdvisoryEventAppendRequest(Array.Empty(), new[] { conflictInput }), CancellationToken.None); + } + + using var client = _factory.CreateClient(); + var response = await client.GetAsync($"/concelier/advisories/{vulnerabilityKey}/replay"); + Assert.Equal(HttpStatusCode.OK, response.StatusCode); + var payload = await response.Content.ReadFromJsonAsync(); + Assert.NotNull(payload); + var conflict = Assert.Single(payload!.Conflicts); + Assert.Equal(conflictId, conflict.ConflictId); + Assert.Equal("severity", conflict.Explainer.Type); + Assert.Equal("mismatch", conflict.Explainer.Reason); + Assert.Equal("CRITICAL", conflict.Explainer.PrimaryValue); + Assert.Equal("MEDIUM", conflict.Explainer.SuppressedValue); + Assert.Equal(conflict.Explainer.ComputeHashHex(), conflict.ConflictHash); + } + + [Fact] + public async Task MirrorEndpointsServeConfiguredArtifacts() + { + using var temp = new TempDirectory(); + var exportId = "20251019T120000Z"; + var exportRoot = Path.Combine(temp.Path, exportId); + var mirrorRoot = Path.Combine(exportRoot, "mirror"); + var domainRoot = Path.Combine(mirrorRoot, "primary"); + Directory.CreateDirectory(domainRoot); + + await File.WriteAllTextAsync( + Path.Combine(mirrorRoot, "index.json"), + """{"schemaVersion":1,"domains":[]}"""); + await File.WriteAllTextAsync( + Path.Combine(domainRoot, "manifest.json"), + """{"domainId":"primary"}"""); + await File.WriteAllTextAsync( + Path.Combine(domainRoot, "bundle.json"), + """{"advisories":[]}"""); + await File.WriteAllTextAsync( + Path.Combine(domainRoot, "bundle.json.jws"), + "test-signature"); + + var environment = new Dictionary + { + ["CONCELIER_MIRROR__ENABLED"] = "true", + ["CONCELIER_MIRROR__EXPORTROOT"] = temp.Path, + ["CONCELIER_MIRROR__ACTIVEEXPORTID"] = exportId, + ["CONCELIER_MIRROR__DOMAINS__0__ID"] = "primary", + ["CONCELIER_MIRROR__DOMAINS__0__DISPLAYNAME"] = "Primary", + ["CONCELIER_MIRROR__DOMAINS__0__REQUIREAUTHENTICATION"] = "false", + ["CONCELIER_MIRROR__DOMAINS__0__MAXDOWNLOADREQUESTSPERHOUR"] = "5", + ["CONCELIER_MIRROR__MAXINDEXREQUESTSPERHOUR"] = "5" + }; + + using var factory = new ConcelierApplicationFactory(_runner.ConnectionString, environmentOverrides: environment); + using var client = factory.CreateClient(); + + var indexResponse = await client.GetAsync("/concelier/exports/index.json"); + Assert.Equal(HttpStatusCode.OK, indexResponse.StatusCode); + var indexContent = await indexResponse.Content.ReadAsStringAsync(); + Assert.Contains(@"""schemaVersion"":1", indexContent, StringComparison.Ordinal); + + var manifestResponse = await client.GetAsync("/concelier/exports/mirror/primary/manifest.json"); + Assert.Equal(HttpStatusCode.OK, manifestResponse.StatusCode); + var manifestContent = await manifestResponse.Content.ReadAsStringAsync(); + Assert.Contains(@"""domainId"":""primary""", manifestContent, StringComparison.Ordinal); + + var bundleResponse = await client.GetAsync("/concelier/exports/mirror/primary/bundle.json.jws"); + Assert.Equal(HttpStatusCode.OK, bundleResponse.StatusCode); + var signatureContent = await bundleResponse.Content.ReadAsStringAsync(); + Assert.Equal("test-signature", signatureContent); + } + + [Fact] + public async Task MirrorEndpointsEnforceAuthenticationForProtectedDomains() + { + using var temp = new TempDirectory(); + var exportId = "20251019T120000Z"; + var secureRoot = Path.Combine(temp.Path, exportId, "mirror", "secure"); + Directory.CreateDirectory(secureRoot); + + await File.WriteAllTextAsync( + Path.Combine(temp.Path, exportId, "mirror", "index.json"), + """{"schemaVersion":1,"domains":[]}"""); + await File.WriteAllTextAsync( + Path.Combine(secureRoot, "manifest.json"), + """{"domainId":"secure"}"""); + + var environment = new Dictionary + { + ["CONCELIER_MIRROR__ENABLED"] = "true", + ["CONCELIER_MIRROR__EXPORTROOT"] = temp.Path, + ["CONCELIER_MIRROR__ACTIVEEXPORTID"] = exportId, + ["CONCELIER_MIRROR__DOMAINS__0__ID"] = "secure", + ["CONCELIER_MIRROR__DOMAINS__0__REQUIREAUTHENTICATION"] = "true", + ["CONCELIER_MIRROR__DOMAINS__0__MAXDOWNLOADREQUESTSPERHOUR"] = "5", + ["CONCELIER_AUTHORITY__ENABLED"] = "true", + ["CONCELIER_AUTHORITY__ALLOWANONYMOUSFALLBACK"] = "false", + ["CONCELIER_AUTHORITY__ISSUER"] = "https://authority.example", + ["CONCELIER_AUTHORITY__REQUIREHTTPSMETADATA"] = "false", + ["CONCELIER_AUTHORITY__AUDIENCES__0"] = "api://concelier", + ["CONCELIER_AUTHORITY__REQUIREDSCOPES__0"] = StellaOpsScopes.ConcelierJobsTrigger, + ["CONCELIER_AUTHORITY__CLIENTID"] = "concelier-jobs", + ["CONCELIER_AUTHORITY__CLIENTSECRET"] = "secret", + ["CONCELIER_AUTHORITY__CLIENTSCOPES__0"] = StellaOpsScopes.ConcelierJobsTrigger + }; + + using var factory = new ConcelierApplicationFactory( + _runner.ConnectionString, + authority => + { + authority.Enabled = true; + authority.AllowAnonymousFallback = false; + authority.Issuer = "https://authority.example"; + authority.RequireHttpsMetadata = false; + authority.Audiences.Clear(); + authority.Audiences.Add("api://concelier"); + authority.RequiredScopes.Clear(); + authority.RequiredScopes.Add(StellaOpsScopes.ConcelierJobsTrigger); + authority.ClientId = "concelier-jobs"; + authority.ClientSecret = "secret"; + }, + environment); + + using var client = factory.CreateClient(); + var response = await client.GetAsync("/concelier/exports/mirror/secure/manifest.json"); + Assert.Equal(HttpStatusCode.Unauthorized, response.StatusCode); + var authHeader = Assert.Single(response.Headers.WwwAuthenticate); + Assert.Equal("Bearer", authHeader.Scheme); + } + + [Fact] + public async Task MirrorEndpointsRespectRateLimits() + { + using var temp = new TempDirectory(); + var exportId = "20251019T130000Z"; + var exportRoot = Path.Combine(temp.Path, exportId); + var mirrorRoot = Path.Combine(exportRoot, "mirror"); + Directory.CreateDirectory(mirrorRoot); + + await File.WriteAllTextAsync( + Path.Combine(mirrorRoot, "index.json"), + """{\"schemaVersion\":1,\"domains\":[]}""" + ); + + var environment = new Dictionary + { + ["CONCELIER_MIRROR__ENABLED"] = "true", + ["CONCELIER_MIRROR__EXPORTROOT"] = temp.Path, + ["CONCELIER_MIRROR__ACTIVEEXPORTID"] = exportId, + ["CONCELIER_MIRROR__MAXINDEXREQUESTSPERHOUR"] = "1", + ["CONCELIER_MIRROR__DOMAINS__0__ID"] = "primary", + ["CONCELIER_MIRROR__DOMAINS__0__REQUIREAUTHENTICATION"] = "false", + ["CONCELIER_MIRROR__DOMAINS__0__MAXDOWNLOADREQUESTSPERHOUR"] = "1" + }; + + using var factory = new ConcelierApplicationFactory(_runner.ConnectionString, environmentOverrides: environment); + using var client = factory.CreateClient(); + + var okResponse = await client.GetAsync("/concelier/exports/index.json"); + Assert.Equal(HttpStatusCode.OK, okResponse.StatusCode); + + var limitedResponse = await client.GetAsync("/concelier/exports/index.json"); + Assert.Equal((HttpStatusCode)429, limitedResponse.StatusCode); + Assert.NotNull(limitedResponse.Headers.RetryAfter); Assert.True(limitedResponse.Headers.RetryAfter!.Delta.HasValue); Assert.True(limitedResponse.Headers.RetryAfter!.Delta!.Value.TotalSeconds > 0); } [Fact] - public void MergeModuleDisabledWhenFeatureFlagEnabled() + public void MergeModuleDisabledByDefault() { - var environment = new Dictionary - { - ["CONCELIER_FEATURES__NOMERGEENABLED"] = "true" - }; - using var factory = new ConcelierApplicationFactory( _runner.ConnectionString, authorityConfigure: null, - environmentOverrides: environment); + environmentOverrides: null); using var scope = factory.Services.CreateScope(); var provider = scope.ServiceProvider; @@ -913,11 +909,59 @@ public sealed class WebServiceEndpointsTests : IAsyncLifetime Assert.DoesNotContain("merge:reconcile", schedulerOptions.Definitions.Keys); } + [Fact] + public void MergeModuleReenabledWhenFeatureFlagCleared() + { + var environment = new Dictionary + { + ["CONCELIER_FEATURES__NOMERGEENABLED"] = "false" + }; + + using var factory = new ConcelierApplicationFactory( + _runner.ConnectionString, + authorityConfigure: null, + environmentOverrides: environment); + using var scope = factory.Services.CreateScope(); + var provider = scope.ServiceProvider; + +#pragma warning disable CS0618, CONCELIER0001, CONCELIER0002 // Checking deprecated service registration state. + Assert.NotNull(provider.GetService()); +#pragma warning restore CS0618, CONCELIER0001, CONCELIER0002 + + var schedulerOptions = provider.GetRequiredService>().Value; + Assert.Contains("merge:reconcile", schedulerOptions.Definitions.Keys); + } + + [Fact] + public void MergeJobRemovedWhenAllowlistExcludes() + { + var environment = new Dictionary + { + ["CONCELIER_FEATURES__NOMERGEENABLED"] = "false", + ["CONCELIER_FEATURES__MERGEJOBALLOWLIST__0"] = "export:json" + }; + + using var factory = new ConcelierApplicationFactory( + _runner.ConnectionString, + authorityConfigure: null, + environmentOverrides: environment); + using var scope = factory.Services.CreateScope(); + var provider = scope.ServiceProvider; + +#pragma warning disable CS0618, CONCELIER0001, CONCELIER0002 // Checking deprecated service registration state. + Assert.NotNull(provider.GetService()); +#pragma warning restore CS0618, CONCELIER0001, CONCELIER0002 + + var schedulerOptions = provider.GetRequiredService>().Value; + Assert.DoesNotContain("merge:reconcile", schedulerOptions.Definitions.Keys); + } + [Fact] public void MergeJobRemainsWhenAllowlisted() { var environment = new Dictionary { + ["CONCELIER_FEATURES__NOMERGEENABLED"] = "false", ["CONCELIER_FEATURES__MERGEJOBALLOWLIST__0"] = "merge:reconcile" }; @@ -941,896 +985,896 @@ public sealed class WebServiceEndpointsTests : IAsyncLifetime public async Task JobsEndpointsAllowBypassWhenAuthorityEnabled() { var environment = new Dictionary - { - ["CONCELIER_AUTHORITY__ENABLED"] = "true", - ["CONCELIER_AUTHORITY__ALLOWANONYMOUSFALLBACK"] = "false", - ["CONCELIER_AUTHORITY__ISSUER"] = "https://authority.example", - ["CONCELIER_AUTHORITY__REQUIREHTTPSMETADATA"] = "false", - ["CONCELIER_AUTHORITY__AUDIENCES__0"] = "api://concelier", - ["CONCELIER_AUTHORITY__REQUIREDSCOPES__0"] = StellaOpsScopes.ConcelierJobsTrigger, - ["CONCELIER_AUTHORITY__BYPASSNETWORKS__0"] = "127.0.0.1/32", - ["CONCELIER_AUTHORITY__BYPASSNETWORKS__1"] = "::1/128", - ["CONCELIER_AUTHORITY__CLIENTID"] = "concelier-jobs", - ["CONCELIER_AUTHORITY__CLIENTSECRET"] = "test-secret", - ["CONCELIER_AUTHORITY__CLIENTSCOPES__0"] = StellaOpsScopes.ConcelierJobsTrigger, - }; - - using var factory = new ConcelierApplicationFactory( - _runner.ConnectionString, - authority => - { - authority.Enabled = true; - authority.AllowAnonymousFallback = false; - authority.Issuer = "https://authority.example"; - authority.RequireHttpsMetadata = false; - authority.Audiences.Clear(); - authority.Audiences.Add("api://concelier"); - authority.RequiredScopes.Clear(); - authority.RequiredScopes.Add(StellaOpsScopes.ConcelierJobsTrigger); - authority.BypassNetworks.Clear(); - authority.BypassNetworks.Add("127.0.0.1/32"); - authority.BypassNetworks.Add("::1/128"); - authority.ClientId = "concelier-jobs"; - authority.ClientSecret = "test-secret"; - }, - environment); - - var handler = factory.Services.GetRequiredService(); - handler.Definitions = new[] { new JobDefinition("demo", typeof(DemoJob), TimeSpan.FromMinutes(5), TimeSpan.FromMinutes(1), null, true) }; - - using var client = factory.CreateClient(); - client.DefaultRequestHeaders.Add("X-Test-RemoteAddr", "127.0.0.1"); - var response = await client.GetAsync("/jobs/definitions"); - - Assert.Equal(HttpStatusCode.OK, response.StatusCode); - - var auditLogs = factory.LoggerProvider.Snapshot("Concelier.Authorization.Audit"); - var bypassLog = Assert.Single(auditLogs, entry => entry.TryGetState("Bypass", out var state) && state is bool flag && flag); - Assert.True(bypassLog.TryGetState("RemoteAddress", out var remoteObj) && string.Equals(remoteObj?.ToString(), "127.0.0.1", StringComparison.Ordinal)); - Assert.True(bypassLog.TryGetState("StatusCode", out var statusObj) && Convert.ToInt32(statusObj) == (int)HttpStatusCode.OK); - } - - [Fact] - public async Task JobsEndpointsRequireAuthWhenFallbackDisabled() - { - var enforcementEnvironment = new Dictionary - { - ["CONCELIER_AUTHORITY__ENABLED"] = "true", - ["CONCELIER_AUTHORITY__ALLOWANONYMOUSFALLBACK"] = "false", - ["CONCELIER_AUTHORITY__ISSUER"] = "https://authority.example", - ["CONCELIER_AUTHORITY__REQUIREHTTPSMETADATA"] = "false", - ["CONCELIER_AUTHORITY__AUDIENCES__0"] = "api://concelier", - ["CONCELIER_AUTHORITY__REQUIREDSCOPES__0"] = StellaOpsScopes.ConcelierJobsTrigger, - ["CONCELIER_AUTHORITY__CLIENTID"] = "concelier-jobs", - ["CONCELIER_AUTHORITY__CLIENTSECRET"] = "test-secret", - ["CONCELIER_AUTHORITY__CLIENTSCOPES__0"] = StellaOpsScopes.ConcelierJobsTrigger, - }; - - using var factory = new ConcelierApplicationFactory( - _runner.ConnectionString, - authority => - { - authority.Enabled = true; - authority.AllowAnonymousFallback = false; - authority.Issuer = "https://authority.example"; - authority.RequireHttpsMetadata = false; - authority.Audiences.Clear(); - authority.Audiences.Add("api://concelier"); - authority.RequiredScopes.Clear(); - authority.RequiredScopes.Add(StellaOpsScopes.ConcelierJobsTrigger); - authority.BypassNetworks.Clear(); - authority.ClientId = "concelier-jobs"; - authority.ClientSecret = "test-secret"; - }, - enforcementEnvironment); - - var resolved = factory.Services.GetRequiredService>().Value; - Assert.False(resolved.Authority.AllowAnonymousFallback); - - using var client = factory.CreateClient(); - client.DefaultRequestHeaders.Add("X-Test-RemoteAddr", "127.0.0.1"); - var response = await client.GetAsync("/jobs/definitions"); - - Assert.Equal(HttpStatusCode.Unauthorized, response.StatusCode); - - var auditLogs = factory.LoggerProvider.Snapshot("Concelier.Authorization.Audit"); - var enforcementLog = Assert.Single(auditLogs); - Assert.True(enforcementLog.TryGetState("BypassAllowed", out var bypassAllowedObj) && bypassAllowedObj is bool bypassAllowed && bypassAllowed == false); - Assert.True(enforcementLog.TryGetState("HasPrincipal", out var principalObj) && principalObj is bool hasPrincipal && hasPrincipal == false); - } - - [Fact] - public void AuthorityClientResilienceOptionsAreBound() - { - var environment = new Dictionary - { - ["CONCELIER_AUTHORITY__ENABLED"] = "true", - ["CONCELIER_AUTHORITY__ISSUER"] = "https://authority.example", - ["CONCELIER_AUTHORITY__REQUIREHTTPSMETADATA"] = "false", - ["CONCELIER_AUTHORITY__AUDIENCES__0"] = "api://concelier", - ["CONCELIER_AUTHORITY__REQUIREDSCOPES__0"] = StellaOpsScopes.ConcelierJobsTrigger, - ["CONCELIER_AUTHORITY__CLIENTSCOPES__0"] = StellaOpsScopes.ConcelierJobsTrigger, - ["CONCELIER_AUTHORITY__BACKCHANNELTIMEOUTSECONDS"] = "45", - ["CONCELIER_AUTHORITY__RESILIENCE__ENABLERETRIES"] = "true", - ["CONCELIER_AUTHORITY__RESILIENCE__RETRYDELAYS__0"] = "00:00:02", - ["CONCELIER_AUTHORITY__RESILIENCE__RETRYDELAYS__1"] = "00:00:04", - ["CONCELIER_AUTHORITY__RESILIENCE__ALLOWOFFLINECACHEFALLBACK"] = "false", - ["CONCELIER_AUTHORITY__RESILIENCE__OFFLINECACHETOLERANCE"] = "00:02:30" - }; - - using var factory = new ConcelierApplicationFactory( - _runner.ConnectionString, - authority => - { - authority.Enabled = true; - authority.Issuer = "https://authority.example"; - authority.RequireHttpsMetadata = false; - authority.Audiences.Clear(); - authority.Audiences.Add("api://concelier"); - authority.RequiredScopes.Clear(); - authority.RequiredScopes.Add(StellaOpsScopes.ConcelierJobsTrigger); - authority.ClientScopes.Clear(); - authority.ClientScopes.Add(StellaOpsScopes.ConcelierJobsTrigger); - authority.BackchannelTimeoutSeconds = 45; - }, - environment); - - var monitor = factory.Services.GetRequiredService>(); - var options = monitor.CurrentValue; - - Assert.Equal("https://authority.example", options.Authority); - Assert.Equal(TimeSpan.FromSeconds(45), options.HttpTimeout); - Assert.Equal(new[] { StellaOpsScopes.ConcelierJobsTrigger }, options.NormalizedScopes); - Assert.Equal(new[] { TimeSpan.FromSeconds(2), TimeSpan.FromSeconds(4) }, options.NormalizedRetryDelays); - Assert.False(options.AllowOfflineCacheFallback); - Assert.Equal(TimeSpan.FromSeconds(150), options.OfflineCacheTolerance); - } - - private async Task SeedObservationDocumentsAsync(IEnumerable documents) - { - var client = new MongoClient(_runner.ConnectionString); - var database = client.GetDatabase(MongoStorageDefaults.DefaultDatabaseName); - var collection = database.GetCollection(MongoStorageDefaults.Collections.AdvisoryObservations); - - try - { - await database.DropCollectionAsync(MongoStorageDefaults.Collections.AdvisoryObservations); - } - catch (MongoCommandException ex) when (ex.CodeName == "NamespaceNotFound" || ex.Message.Contains("ns not found", StringComparison.OrdinalIgnoreCase)) - { - // Collection does not exist yet; ignore. - } - - var snapshot = documents?.ToArray() ?? Array.Empty(); - if (snapshot.Length == 0) - { - return; - } - - await collection.InsertManyAsync(snapshot); - } - - private static AdvisoryObservationDocument[] BuildSampleObservationDocuments() - { - return new[] - { - CreateObservationDocument( - id: "tenant-a:nvd:alpha:1", - tenant: "tenant-a", - createdAt: new DateTime(2025, 1, 5, 0, 0, 0, DateTimeKind.Utc), - aliases: new[] { "cve-2025-0001" }, - purls: new[] { "pkg:npm/demo@1.0.0" }, - cpes: new[] { "cpe:/a:vendor:product:1.0" }, - references: new[] { ("advisory", "https://example.test/advisory-1") }), - CreateObservationDocument( - id: "tenant-a:ghsa:beta:1", - tenant: "tenant-a", - createdAt: new DateTime(2025, 1, 6, 0, 0, 0, DateTimeKind.Utc), - aliases: new[] { "ghsa-2025-xyz", "cve-2025-0001" }, - purls: new[] { "pkg:npm/demo@1.1.0" }, - cpes: new[] { "cpe:/a:vendor:product:1.1" }, - references: new[] { ("patch", "https://example.test/patch-1") }), - CreateObservationDocument( - id: "tenant-b:nvd:alpha:1", - tenant: "tenant-b", - createdAt: new DateTime(2025, 1, 7, 0, 0, 0, DateTimeKind.Utc), - aliases: new[] { "cve-2025-0001" }, - purls: new[] { "pkg:npm/demo@2.0.0" }, - cpes: new[] { "cpe:/a:vendor:product:2.0" }, - references: new[] { ("advisory", "https://example.test/advisory-2") }) - }; - } - - private static AdvisoryObservationDocument CreateObservationDocument( - string id, - string tenant, - DateTime createdAt, - IEnumerable? aliases = null, - IEnumerable? purls = null, - IEnumerable? cpes = null, - IEnumerable<(string Type, string Url)>? references = null) - { - return new AdvisoryObservationDocument - { - Id = id, - Tenant = tenant.ToLowerInvariant(), - CreatedAt = createdAt, - Source = new AdvisoryObservationSourceDocument - { - Vendor = "nvd", - Stream = "feed", - Api = "https://example.test/api" - }, - Upstream = new AdvisoryObservationUpstreamDocument - { - UpstreamId = id, - DocumentVersion = null, - FetchedAt = createdAt, - ReceivedAt = createdAt, - ContentHash = $"sha256:{id}", - Signature = new AdvisoryObservationSignatureDocument - { - Present = false - }, - Metadata = new Dictionary(StringComparer.Ordinal) - }, - Content = new AdvisoryObservationContentDocument - { - Format = "csaf", - SpecVersion = "2.0", - Raw = BsonDocument.Parse("""{"observation":"%ID%"}""".Replace("%ID%", id)), - Metadata = new Dictionary(StringComparer.Ordinal) - }, - Linkset = new AdvisoryObservationLinksetDocument - { - Aliases = aliases?.Select(value => value.Trim().ToLowerInvariant()).ToList(), - Purls = purls?.Select(value => value.Trim()).ToList(), - Cpes = cpes?.Select(value => value.Trim()).ToList(), - References = references is null - ? new List() - : references - .Select(reference => new AdvisoryObservationReferenceDocument - { - Type = reference.Type.Trim().ToLowerInvariant(), - Url = reference.Url.Trim() - }) - .ToList() - }, - Attributes = new Dictionary(StringComparer.Ordinal) - }; - } - - private sealed record ReplayResponse( - string VulnerabilityKey, - DateTimeOffset? AsOf, - List Statements, - List? Conflicts); - - private sealed record ReplayStatement( - Guid StatementId, - string VulnerabilityKey, - string AdvisoryKey, - Advisory Advisory, - string StatementHash, - DateTimeOffset AsOf, - DateTimeOffset RecordedAt, - IReadOnlyList InputDocumentIds); - - private sealed record ReplayConflict( - Guid ConflictId, - string VulnerabilityKey, - IReadOnlyList StatementIds, - string ConflictHash, - DateTimeOffset AsOf, - DateTimeOffset RecordedAt, - string Details, - MergeConflictExplainerPayload Explainer); - - private sealed class ConcelierApplicationFactory : WebApplicationFactory - { - private readonly string _connectionString; - private readonly string? _previousDsn; - private readonly string? _previousDriver; - private readonly string? _previousTimeout; - private readonly string? _previousTelemetryEnabled; - private readonly string? _previousTelemetryLogging; - private readonly string? _previousTelemetryTracing; - private readonly string? _previousTelemetryMetrics; - private readonly Action? _authorityConfigure; - private readonly IDictionary _additionalPreviousEnvironment = new Dictionary(StringComparer.OrdinalIgnoreCase); - public CollectingLoggerProvider LoggerProvider { get; } = new(); - - public ConcelierApplicationFactory( - string connectionString, - Action? authorityConfigure = null, - IDictionary? environmentOverrides = null) - { - _connectionString = connectionString; - _authorityConfigure = authorityConfigure; - _previousDsn = Environment.GetEnvironmentVariable("CONCELIER_STORAGE__DSN"); - _previousDriver = Environment.GetEnvironmentVariable("CONCELIER_STORAGE__DRIVER"); - _previousTimeout = Environment.GetEnvironmentVariable("CONCELIER_STORAGE__COMMANDTIMEOUTSECONDS"); - _previousTelemetryEnabled = Environment.GetEnvironmentVariable("CONCELIER_TELEMETRY__ENABLED"); - _previousTelemetryLogging = Environment.GetEnvironmentVariable("CONCELIER_TELEMETRY__ENABLELOGGING"); - _previousTelemetryTracing = Environment.GetEnvironmentVariable("CONCELIER_TELEMETRY__ENABLETRACING"); - _previousTelemetryMetrics = Environment.GetEnvironmentVariable("CONCELIER_TELEMETRY__ENABLEMETRICS"); - Environment.SetEnvironmentVariable("CONCELIER_STORAGE__DSN", connectionString); - Environment.SetEnvironmentVariable("CONCELIER_STORAGE__DRIVER", "mongo"); - Environment.SetEnvironmentVariable("CONCELIER_STORAGE__COMMANDTIMEOUTSECONDS", "30"); - Environment.SetEnvironmentVariable("CONCELIER_TELEMETRY__ENABLED", "false"); - Environment.SetEnvironmentVariable("CONCELIER_TELEMETRY__ENABLELOGGING", "false"); - Environment.SetEnvironmentVariable("CONCELIER_TELEMETRY__ENABLETRACING", "false"); - Environment.SetEnvironmentVariable("CONCELIER_TELEMETRY__ENABLEMETRICS", "false"); - if (environmentOverrides is not null) - { - foreach (var kvp in environmentOverrides) - { - var previous = Environment.GetEnvironmentVariable(kvp.Key); - _additionalPreviousEnvironment[kvp.Key] = previous; - Environment.SetEnvironmentVariable(kvp.Key, kvp.Value); - } - } - } - - protected override void ConfigureWebHost(IWebHostBuilder builder) - { - builder.ConfigureAppConfiguration((context, configurationBuilder) => - { - var settings = new Dictionary - { - ["Plugins:Directory"] = Path.Combine(context.HostingEnvironment.ContentRootPath, "StellaOps.Concelier.PluginBinaries"), - }; - - configurationBuilder.AddInMemoryCollection(settings!); - }); - - builder.ConfigureLogging(logging => - { - logging.AddProvider(LoggerProvider); - }); - - builder.ConfigureServices(services => - { - services.AddSingleton(); - services.AddSingleton(sp => sp.GetRequiredService()); - services.PostConfigure(options => - { - options.Storage.Driver = "mongo"; - options.Storage.Dsn = _connectionString; - options.Storage.CommandTimeoutSeconds = 30; - options.Plugins.Directory ??= Path.Combine(AppContext.BaseDirectory, "StellaOps.Concelier.PluginBinaries"); - options.Telemetry.Enabled = false; - options.Telemetry.EnableLogging = false; - options.Telemetry.EnableTracing = false; - options.Telemetry.EnableMetrics = false; - options.Authority ??= new ConcelierOptions.AuthorityOptions(); - _authorityConfigure?.Invoke(options.Authority); - }); - }); - - builder.ConfigureTestServices(services => - { - services.AddSingleton(); - services.PostConfigure(StellaOpsAuthenticationDefaults.AuthenticationScheme, options => - { - options.RequireHttpsMetadata = false; - options.TokenValidationParameters = new TokenValidationParameters - { - ValidateIssuerSigningKey = true, - IssuerSigningKey = TestSigningKey, - ValidateIssuer = false, - ValidateAudience = false, - ValidateLifetime = false, - NameClaimType = ClaimTypes.Name, - RoleClaimType = ClaimTypes.Role, - ClockSkew = TimeSpan.Zero - }; - var issuer = string.IsNullOrWhiteSpace(options.Authority) ? TestAuthorityIssuer : options.Authority; - options.ConfigurationManager = new StaticConfigurationManager(new OpenIdConnectConfiguration - { - Issuer = issuer - }); - }); - }); - } - - protected override void Dispose(bool disposing) - { - base.Dispose(disposing); - Environment.SetEnvironmentVariable("CONCELIER_STORAGE__DSN", _previousDsn); - Environment.SetEnvironmentVariable("CONCELIER_STORAGE__DRIVER", _previousDriver); - Environment.SetEnvironmentVariable("CONCELIER_STORAGE__COMMANDTIMEOUTSECONDS", _previousTimeout); - Environment.SetEnvironmentVariable("CONCELIER_TELEMETRY__ENABLED", _previousTelemetryEnabled); - Environment.SetEnvironmentVariable("CONCELIER_TELEMETRY__ENABLELOGGING", _previousTelemetryLogging); - Environment.SetEnvironmentVariable("CONCELIER_TELEMETRY__ENABLETRACING", _previousTelemetryTracing); - Environment.SetEnvironmentVariable("CONCELIER_TELEMETRY__ENABLEMETRICS", _previousTelemetryMetrics); - foreach (var kvp in _additionalPreviousEnvironment) - { - Environment.SetEnvironmentVariable(kvp.Key, kvp.Value); - } - - LoggerProvider.Dispose(); - } - - private sealed class RemoteIpStartupFilter : IStartupFilter - { - public Action Configure(Action next) - { - return app => - { - app.Use(async (context, nextMiddleware) => - { - if (context.Request.Headers.TryGetValue("X-Test-RemoteAddr", out var values) - && values.Count > 0 - && IPAddress.TryParse(values[0], out var remote)) - { - context.Connection.RemoteIpAddress = remote; - } - - await nextMiddleware(); - }); - - next(app); - }; - } - } - - public sealed record LogEntry( - string LoggerName, - LogLevel Level, - EventId EventId, - string? Message, - Exception? Exception, - IReadOnlyList> State) - { - public bool TryGetState(string name, out object? value) - { - foreach (var kvp in State) - { - if (string.Equals(kvp.Key, name, StringComparison.Ordinal)) - { - value = kvp.Value; - return true; - } - } - - value = null; - return false; - } - } - - public sealed class CollectingLoggerProvider : ILoggerProvider - { - private readonly object syncRoot = new(); - private readonly List entries = new(); - private bool disposed; - - public ILogger CreateLogger(string categoryName) => new CollectingLogger(categoryName, this); - - public IReadOnlyList Snapshot(string loggerName) - { - lock (syncRoot) - { - return entries - .Where(entry => string.Equals(entry.LoggerName, loggerName, StringComparison.Ordinal)) - .ToArray(); - } - } - - public void Dispose() - { - disposed = true; - lock (syncRoot) - { - entries.Clear(); - } - } - - private void Append(LogEntry entry) - { - if (disposed) - { - return; - } - - lock (syncRoot) - { - entries.Add(entry); - } - } - - private sealed class CollectingLogger : ILogger - { - private readonly string categoryName; - private readonly CollectingLoggerProvider provider; - - public CollectingLogger(string categoryName, CollectingLoggerProvider provider) - { - this.categoryName = categoryName; - this.provider = provider; - } - - public IDisposable? BeginScope(TState state) where TState : notnull => NullScope.Instance; - - public bool IsEnabled(LogLevel logLevel) => true; - - public void Log(LogLevel logLevel, EventId eventId, TState state, Exception? exception, Func formatter) - { - if (formatter is null) - { - throw new ArgumentNullException(nameof(formatter)); - } - - var message = formatter(state, exception); - var kvps = ExtractState(state); - var entry = new LogEntry(categoryName, logLevel, eventId, message, exception, kvps); - provider.Append(entry); - } - - private static IReadOnlyList> ExtractState(TState state) - { - if (state is IReadOnlyList> list) - { - return list; - } - - if (state is IEnumerable> enumerable) - { - return enumerable.ToArray(); - } - - if (state is null) - { - return Array.Empty>(); - } - - return new[] { new KeyValuePair("State", state) }; - } - } - - private sealed class NullScope : IDisposable - { - public static readonly NullScope Instance = new(); - public void Dispose() - { - } - } - } - } - - private sealed class TempDirectory : IDisposable - { - public string Path { get; } - - public TempDirectory() - { - Path = System.IO.Path.Combine(System.IO.Path.GetTempPath(), "concelier-mirror-" + Guid.NewGuid().ToString("N", CultureInfo.InvariantCulture)); - Directory.CreateDirectory(Path); - } - - public void Dispose() - { - try - { - if (Directory.Exists(Path)) - { - Directory.Delete(Path, recursive: true); - } - } - catch - { - // best effort cleanup - } - } - } - - private sealed record HealthPayload(string Status, DateTimeOffset StartedAt, double UptimeSeconds, StoragePayload Storage, TelemetryPayload Telemetry); - - private sealed record StoragePayload(string Driver, bool Completed, DateTimeOffset? CompletedAt, double? DurationMs); - - private sealed record TelemetryPayload(bool Enabled, bool Tracing, bool Metrics, bool Logging); - - private sealed record ReadyPayload(string Status, DateTimeOffset StartedAt, double UptimeSeconds, ReadyMongoPayload Mongo); - - private sealed record ReadyMongoPayload(string Status, double? LatencyMs, DateTimeOffset? CheckedAt, string? Error); - - private sealed record JobDefinitionPayload(string Kind, bool Enabled, string? CronExpression, TimeSpan Timeout, TimeSpan LeaseDuration, JobRunPayload? LastRun); - - private sealed record JobRunPayload(Guid RunId, string Kind, string Status, string Trigger, DateTimeOffset CreatedAt, DateTimeOffset? StartedAt, DateTimeOffset? CompletedAt, string? Error, TimeSpan? Duration, Dictionary Parameters); - - private sealed record ProblemDocument(string? Type, string? Title, int? Status, string? Detail, string? Instance); - - private async Task SeedAdvisoryRawDocumentsAsync(params BsonDocument[] documents) - { - var client = new MongoClient(_runner.ConnectionString); - var database = client.GetDatabase(MongoStorageDefaults.DefaultDatabaseName); - var collection = database.GetCollection(MongoStorageDefaults.Collections.AdvisoryRaw); - await collection.DeleteManyAsync(FilterDefinition.Empty); - if (documents.Length > 0) - { - await collection.InsertManyAsync(documents); - } - } - - private static BsonDocument CreateAdvisoryRawDocument( - string tenant, - string vendor, - string upstreamId, - string contentHash, - BsonDocument? raw = null, - string? supersedes = null) - { - var now = DateTime.UtcNow; - return new BsonDocument - { - { "_id", BuildRawDocumentId(vendor, upstreamId, contentHash) }, - { "tenant", tenant }, - { - "source", - new BsonDocument - { - { "vendor", vendor }, - { "connector", "test-connector" }, - { "version", "1.0.0" } - } - }, - { - "upstream", - new BsonDocument - { - { "upstream_id", upstreamId }, - { "document_version", "1" }, - { "retrieved_at", now }, - { "content_hash", contentHash }, - { "signature", new BsonDocument { { "present", false } } }, - { "provenance", new BsonDocument { { "api", "https://example.test" } } } - } - }, - { - "content", - new BsonDocument - { - { "format", "osv" }, - { "raw", raw ?? new BsonDocument("id", upstreamId) } - } - }, - { - "identifiers", - new BsonDocument - { - { "aliases", new BsonArray(new[] { upstreamId }) }, - { "primary", upstreamId } - } - }, - { - "linkset", - new BsonDocument - { - { "aliases", new BsonArray() }, - { "purls", new BsonArray() }, - { "cpes", new BsonArray() }, - { "references", new BsonArray() }, - { "reconciled_from", new BsonArray() }, - { "notes", new BsonDocument() } - } - }, - { "supersedes", supersedes is null ? BsonNull.Value : supersedes }, - { "ingested_at", now }, - { "created_at", now } - }; - } - - private static string BuildRawDocumentId(string vendor, string upstreamId, string contentHash) - { - static string Sanitize(string value) - { - if (string.IsNullOrWhiteSpace(value)) - { - return "unknown"; - } - - var buffer = new char[value.Length]; - var index = 0; - foreach (var ch in value.Trim().ToLowerInvariant()) - { - buffer[index++] = char.IsLetterOrDigit(ch) ? ch : '-'; - } - - var sanitized = new string(buffer, 0, index).Trim('-'); - return string.IsNullOrEmpty(sanitized) ? "unknown" : sanitized; - } - - var vendorSegment = Sanitize(vendor); - var upstreamSegment = Sanitize(upstreamId); - var hashSegment = Sanitize(contentHash.Replace(":", "-")); - return $"advisory_raw:{vendorSegment}:{upstreamSegment}:{hashSegment}"; - } - - private static AdvisoryIngestRequest BuildAdvisoryIngestRequest(string contentHash, string upstreamId) - { - var raw = CreateJsonElement($@"{{""id"":""{upstreamId}"",""modified"":""{DateTime.UtcNow:O}""}}"); - var references = new[] - { - new AdvisoryLinksetReferenceRequest("advisory", $"https://example.test/advisories/{upstreamId}", null) - }; - - return new AdvisoryIngestRequest( - new AdvisorySourceRequest("osv", "osv-connector", "1.0.0", "feed"), - new AdvisoryUpstreamRequest( - upstreamId, - "2025-01-01T00:00:00Z", - DateTimeOffset.UtcNow, - contentHash, - new AdvisorySignatureRequest(false, null, null, null, null, null), - new Dictionary { ["http.method"] = "GET" }), - new AdvisoryContentRequest("osv", "1.3.0", raw, null), - new AdvisoryIdentifiersRequest( - upstreamId, - new[] { upstreamId, $"{upstreamId}-ALIAS" }), - new AdvisoryLinksetRequest( - new[] { upstreamId }, - new[] { "pkg:npm/demo@1.0.0" }, - Array.Empty(), - references, - Array.Empty(), - new Dictionary { ["note"] = "ingest-test" })); - } - - private static JsonElement CreateJsonElement(string json) - { - using var document = JsonDocument.Parse(json); - return document.RootElement.Clone(); - } - - private static async Task> CaptureMetricsAsync(string meterName, string instrumentName, Func action) - { - var measurements = new List(); - var listener = new MeterListener(); - - listener.InstrumentPublished += (instrument, currentListener) => - { - if (string.Equals(instrument.Meter.Name, meterName, StringComparison.Ordinal) && - string.Equals(instrument.Name, instrumentName, StringComparison.Ordinal)) - { - currentListener.EnableMeasurementEvents(instrument); - } - }; - - listener.SetMeasurementEventCallback((instrument, measurement, tags, state) => - { - var tagDictionary = new Dictionary(StringComparer.Ordinal); - foreach (var tag in tags) - { - tagDictionary[tag.Key] = tag.Value; - } - - measurements.Add(new MetricMeasurement(instrument.Name, measurement, tagDictionary)); - }); - - listener.Start(); - try - { - await action().ConfigureAwait(false); - } - finally - { - listener.Dispose(); - } - - return measurements; - } - - private static string? GetTagValue(MetricMeasurement measurement, string tag) - { - if (measurement.Tags.TryGetValue(tag, out var value)) - { - return value?.ToString(); - } - - return null; - } - - private static string CreateTestToken(string tenant, params string[] scopes) - { - var normalizedTenant = string.IsNullOrWhiteSpace(tenant) ? "default" : tenant.Trim().ToLowerInvariant(); - var scopeSet = scopes is { Length: > 0 } - ? scopes - .Select(StellaOpsScopes.Normalize) - .Where(static scope => !string.IsNullOrEmpty(scope)) - .Select(static scope => scope!) - .Distinct(StringComparer.Ordinal) - .ToArray() - : Array.Empty(); - - var claims = new List - { - new Claim(StellaOpsClaimTypes.Subject, "test-user"), - new Claim(StellaOpsClaimTypes.Tenant, normalizedTenant), - new Claim(StellaOpsClaimTypes.Scope, string.Join(' ', scopeSet)) - }; - - foreach (var scope in scopeSet) - { - claims.Add(new Claim(StellaOpsClaimTypes.ScopeItem, scope)); - } - - var credentials = new SigningCredentials(TestSigningKey, SecurityAlgorithms.HmacSha256); - var now = DateTime.UtcNow; - var token = new JwtSecurityToken( - issuer: TestAuthorityIssuer, - audience: TestAuthorityAudience, - claims: claims, - notBefore: now.AddMinutes(-5), - expires: now.AddMinutes(30), - signingCredentials: credentials); - - return new JwtSecurityTokenHandler().WriteToken(token); - } - - private sealed record MetricMeasurement(string Instrument, long Value, IReadOnlyDictionary Tags); - - private sealed class DemoJob : IJob - { - public Task ExecuteAsync(JobExecutionContext context, CancellationToken cancellationToken) => Task.CompletedTask; - } - - private sealed class StubJobCoordinator : IJobCoordinator - { - public JobTriggerResult NextResult { get; set; } = JobTriggerResult.NotFound("not set"); - - public IReadOnlyList Definitions { get; set; } = Array.Empty(); - - public IReadOnlyList RecentRuns { get; set; } = Array.Empty(); - - public IReadOnlyList ActiveRuns { get; set; } = Array.Empty(); - - public Dictionary Runs { get; } = new(); - - public Dictionary LastRuns { get; } = new(StringComparer.Ordinal); - - public Task TriggerAsync(string kind, IReadOnlyDictionary? parameters, string trigger, CancellationToken cancellationToken) - => Task.FromResult(NextResult); - - public Task> GetDefinitionsAsync(CancellationToken cancellationToken) - => Task.FromResult(Definitions); - - public Task> GetRecentRunsAsync(string? kind, int limit, CancellationToken cancellationToken) - { - IEnumerable query = RecentRuns; - if (!string.IsNullOrWhiteSpace(kind)) - { - query = query.Where(run => string.Equals(run.Kind, kind, StringComparison.Ordinal)); - } - - return Task.FromResult>(query.Take(limit).ToArray()); - } - - public Task> GetActiveRunsAsync(CancellationToken cancellationToken) - => Task.FromResult(ActiveRuns); - - public Task GetRunAsync(Guid runId, CancellationToken cancellationToken) - => Task.FromResult(Runs.TryGetValue(runId, out var run) ? run : null); - - public Task GetLastRunAsync(string kind, CancellationToken cancellationToken) - => Task.FromResult(LastRuns.TryGetValue(kind, out var run) ? run : null); - - public Task> GetLastRunsAsync(IEnumerable kinds, CancellationToken cancellationToken) - { - var map = new Dictionary(StringComparer.Ordinal); - foreach (var kind in kinds) - { - if (kind is null) - { - continue; - } - - if (LastRuns.TryGetValue(kind, out var run) && run is not null) - { - map[kind] = run; - } - } - - return Task.FromResult>(map); - } - } -} + { + ["CONCELIER_AUTHORITY__ENABLED"] = "true", + ["CONCELIER_AUTHORITY__ALLOWANONYMOUSFALLBACK"] = "false", + ["CONCELIER_AUTHORITY__ISSUER"] = "https://authority.example", + ["CONCELIER_AUTHORITY__REQUIREHTTPSMETADATA"] = "false", + ["CONCELIER_AUTHORITY__AUDIENCES__0"] = "api://concelier", + ["CONCELIER_AUTHORITY__REQUIREDSCOPES__0"] = StellaOpsScopes.ConcelierJobsTrigger, + ["CONCELIER_AUTHORITY__BYPASSNETWORKS__0"] = "127.0.0.1/32", + ["CONCELIER_AUTHORITY__BYPASSNETWORKS__1"] = "::1/128", + ["CONCELIER_AUTHORITY__CLIENTID"] = "concelier-jobs", + ["CONCELIER_AUTHORITY__CLIENTSECRET"] = "test-secret", + ["CONCELIER_AUTHORITY__CLIENTSCOPES__0"] = StellaOpsScopes.ConcelierJobsTrigger, + }; + + using var factory = new ConcelierApplicationFactory( + _runner.ConnectionString, + authority => + { + authority.Enabled = true; + authority.AllowAnonymousFallback = false; + authority.Issuer = "https://authority.example"; + authority.RequireHttpsMetadata = false; + authority.Audiences.Clear(); + authority.Audiences.Add("api://concelier"); + authority.RequiredScopes.Clear(); + authority.RequiredScopes.Add(StellaOpsScopes.ConcelierJobsTrigger); + authority.BypassNetworks.Clear(); + authority.BypassNetworks.Add("127.0.0.1/32"); + authority.BypassNetworks.Add("::1/128"); + authority.ClientId = "concelier-jobs"; + authority.ClientSecret = "test-secret"; + }, + environment); + + var handler = factory.Services.GetRequiredService(); + handler.Definitions = new[] { new JobDefinition("demo", typeof(DemoJob), TimeSpan.FromMinutes(5), TimeSpan.FromMinutes(1), null, true) }; + + using var client = factory.CreateClient(); + client.DefaultRequestHeaders.Add("X-Test-RemoteAddr", "127.0.0.1"); + var response = await client.GetAsync("/jobs/definitions"); + + Assert.Equal(HttpStatusCode.OK, response.StatusCode); + + var auditLogs = factory.LoggerProvider.Snapshot("Concelier.Authorization.Audit"); + var bypassLog = Assert.Single(auditLogs, entry => entry.TryGetState("Bypass", out var state) && state is bool flag && flag); + Assert.True(bypassLog.TryGetState("RemoteAddress", out var remoteObj) && string.Equals(remoteObj?.ToString(), "127.0.0.1", StringComparison.Ordinal)); + Assert.True(bypassLog.TryGetState("StatusCode", out var statusObj) && Convert.ToInt32(statusObj) == (int)HttpStatusCode.OK); + } + + [Fact] + public async Task JobsEndpointsRequireAuthWhenFallbackDisabled() + { + var enforcementEnvironment = new Dictionary + { + ["CONCELIER_AUTHORITY__ENABLED"] = "true", + ["CONCELIER_AUTHORITY__ALLOWANONYMOUSFALLBACK"] = "false", + ["CONCELIER_AUTHORITY__ISSUER"] = "https://authority.example", + ["CONCELIER_AUTHORITY__REQUIREHTTPSMETADATA"] = "false", + ["CONCELIER_AUTHORITY__AUDIENCES__0"] = "api://concelier", + ["CONCELIER_AUTHORITY__REQUIREDSCOPES__0"] = StellaOpsScopes.ConcelierJobsTrigger, + ["CONCELIER_AUTHORITY__CLIENTID"] = "concelier-jobs", + ["CONCELIER_AUTHORITY__CLIENTSECRET"] = "test-secret", + ["CONCELIER_AUTHORITY__CLIENTSCOPES__0"] = StellaOpsScopes.ConcelierJobsTrigger, + }; + + using var factory = new ConcelierApplicationFactory( + _runner.ConnectionString, + authority => + { + authority.Enabled = true; + authority.AllowAnonymousFallback = false; + authority.Issuer = "https://authority.example"; + authority.RequireHttpsMetadata = false; + authority.Audiences.Clear(); + authority.Audiences.Add("api://concelier"); + authority.RequiredScopes.Clear(); + authority.RequiredScopes.Add(StellaOpsScopes.ConcelierJobsTrigger); + authority.BypassNetworks.Clear(); + authority.ClientId = "concelier-jobs"; + authority.ClientSecret = "test-secret"; + }, + enforcementEnvironment); + + var resolved = factory.Services.GetRequiredService>().Value; + Assert.False(resolved.Authority.AllowAnonymousFallback); + + using var client = factory.CreateClient(); + client.DefaultRequestHeaders.Add("X-Test-RemoteAddr", "127.0.0.1"); + var response = await client.GetAsync("/jobs/definitions"); + + Assert.Equal(HttpStatusCode.Unauthorized, response.StatusCode); + + var auditLogs = factory.LoggerProvider.Snapshot("Concelier.Authorization.Audit"); + var enforcementLog = Assert.Single(auditLogs); + Assert.True(enforcementLog.TryGetState("BypassAllowed", out var bypassAllowedObj) && bypassAllowedObj is bool bypassAllowed && bypassAllowed == false); + Assert.True(enforcementLog.TryGetState("HasPrincipal", out var principalObj) && principalObj is bool hasPrincipal && hasPrincipal == false); + } + + [Fact] + public void AuthorityClientResilienceOptionsAreBound() + { + var environment = new Dictionary + { + ["CONCELIER_AUTHORITY__ENABLED"] = "true", + ["CONCELIER_AUTHORITY__ISSUER"] = "https://authority.example", + ["CONCELIER_AUTHORITY__REQUIREHTTPSMETADATA"] = "false", + ["CONCELIER_AUTHORITY__AUDIENCES__0"] = "api://concelier", + ["CONCELIER_AUTHORITY__REQUIREDSCOPES__0"] = StellaOpsScopes.ConcelierJobsTrigger, + ["CONCELIER_AUTHORITY__CLIENTSCOPES__0"] = StellaOpsScopes.ConcelierJobsTrigger, + ["CONCELIER_AUTHORITY__BACKCHANNELTIMEOUTSECONDS"] = "45", + ["CONCELIER_AUTHORITY__RESILIENCE__ENABLERETRIES"] = "true", + ["CONCELIER_AUTHORITY__RESILIENCE__RETRYDELAYS__0"] = "00:00:02", + ["CONCELIER_AUTHORITY__RESILIENCE__RETRYDELAYS__1"] = "00:00:04", + ["CONCELIER_AUTHORITY__RESILIENCE__ALLOWOFFLINECACHEFALLBACK"] = "false", + ["CONCELIER_AUTHORITY__RESILIENCE__OFFLINECACHETOLERANCE"] = "00:02:30" + }; + + using var factory = new ConcelierApplicationFactory( + _runner.ConnectionString, + authority => + { + authority.Enabled = true; + authority.Issuer = "https://authority.example"; + authority.RequireHttpsMetadata = false; + authority.Audiences.Clear(); + authority.Audiences.Add("api://concelier"); + authority.RequiredScopes.Clear(); + authority.RequiredScopes.Add(StellaOpsScopes.ConcelierJobsTrigger); + authority.ClientScopes.Clear(); + authority.ClientScopes.Add(StellaOpsScopes.ConcelierJobsTrigger); + authority.BackchannelTimeoutSeconds = 45; + }, + environment); + + var monitor = factory.Services.GetRequiredService>(); + var options = monitor.CurrentValue; + + Assert.Equal("https://authority.example", options.Authority); + Assert.Equal(TimeSpan.FromSeconds(45), options.HttpTimeout); + Assert.Equal(new[] { StellaOpsScopes.ConcelierJobsTrigger }, options.NormalizedScopes); + Assert.Equal(new[] { TimeSpan.FromSeconds(2), TimeSpan.FromSeconds(4) }, options.NormalizedRetryDelays); + Assert.False(options.AllowOfflineCacheFallback); + Assert.Equal(TimeSpan.FromSeconds(150), options.OfflineCacheTolerance); + } + + private async Task SeedObservationDocumentsAsync(IEnumerable documents) + { + var client = new MongoClient(_runner.ConnectionString); + var database = client.GetDatabase(MongoStorageDefaults.DefaultDatabaseName); + var collection = database.GetCollection(MongoStorageDefaults.Collections.AdvisoryObservations); + + try + { + await database.DropCollectionAsync(MongoStorageDefaults.Collections.AdvisoryObservations); + } + catch (MongoCommandException ex) when (ex.CodeName == "NamespaceNotFound" || ex.Message.Contains("ns not found", StringComparison.OrdinalIgnoreCase)) + { + // Collection does not exist yet; ignore. + } + + var snapshot = documents?.ToArray() ?? Array.Empty(); + if (snapshot.Length == 0) + { + return; + } + + await collection.InsertManyAsync(snapshot); + } + + private static AdvisoryObservationDocument[] BuildSampleObservationDocuments() + { + return new[] + { + CreateObservationDocument( + id: "tenant-a:nvd:alpha:1", + tenant: "tenant-a", + createdAt: new DateTime(2025, 1, 5, 0, 0, 0, DateTimeKind.Utc), + aliases: new[] { "cve-2025-0001" }, + purls: new[] { "pkg:npm/demo@1.0.0" }, + cpes: new[] { "cpe:/a:vendor:product:1.0" }, + references: new[] { ("advisory", "https://example.test/advisory-1") }), + CreateObservationDocument( + id: "tenant-a:ghsa:beta:1", + tenant: "tenant-a", + createdAt: new DateTime(2025, 1, 6, 0, 0, 0, DateTimeKind.Utc), + aliases: new[] { "ghsa-2025-xyz", "cve-2025-0001" }, + purls: new[] { "pkg:npm/demo@1.1.0" }, + cpes: new[] { "cpe:/a:vendor:product:1.1" }, + references: new[] { ("patch", "https://example.test/patch-1") }), + CreateObservationDocument( + id: "tenant-b:nvd:alpha:1", + tenant: "tenant-b", + createdAt: new DateTime(2025, 1, 7, 0, 0, 0, DateTimeKind.Utc), + aliases: new[] { "cve-2025-0001" }, + purls: new[] { "pkg:npm/demo@2.0.0" }, + cpes: new[] { "cpe:/a:vendor:product:2.0" }, + references: new[] { ("advisory", "https://example.test/advisory-2") }) + }; + } + + private static AdvisoryObservationDocument CreateObservationDocument( + string id, + string tenant, + DateTime createdAt, + IEnumerable? aliases = null, + IEnumerable? purls = null, + IEnumerable? cpes = null, + IEnumerable<(string Type, string Url)>? references = null) + { + return new AdvisoryObservationDocument + { + Id = id, + Tenant = tenant.ToLowerInvariant(), + CreatedAt = createdAt, + Source = new AdvisoryObservationSourceDocument + { + Vendor = "nvd", + Stream = "feed", + Api = "https://example.test/api" + }, + Upstream = new AdvisoryObservationUpstreamDocument + { + UpstreamId = id, + DocumentVersion = null, + FetchedAt = createdAt, + ReceivedAt = createdAt, + ContentHash = $"sha256:{id}", + Signature = new AdvisoryObservationSignatureDocument + { + Present = false + }, + Metadata = new Dictionary(StringComparer.Ordinal) + }, + Content = new AdvisoryObservationContentDocument + { + Format = "csaf", + SpecVersion = "2.0", + Raw = BsonDocument.Parse("""{"observation":"%ID%"}""".Replace("%ID%", id)), + Metadata = new Dictionary(StringComparer.Ordinal) + }, + Linkset = new AdvisoryObservationLinksetDocument + { + Aliases = aliases?.Where(value => value is not null).ToList(), + Purls = purls?.Where(value => value is not null).ToList(), + Cpes = cpes?.Where(value => value is not null).ToList(), + References = references is null + ? new List() + : references + .Select(reference => new AdvisoryObservationReferenceDocument + { + Type = reference.Type, + Url = reference.Url + }) + .ToList() + }, + Attributes = new Dictionary(StringComparer.Ordinal) + }; + } + + private sealed record ReplayResponse( + string VulnerabilityKey, + DateTimeOffset? AsOf, + List Statements, + List? Conflicts); + + private sealed record ReplayStatement( + Guid StatementId, + string VulnerabilityKey, + string AdvisoryKey, + Advisory Advisory, + string StatementHash, + DateTimeOffset AsOf, + DateTimeOffset RecordedAt, + IReadOnlyList InputDocumentIds); + + private sealed record ReplayConflict( + Guid ConflictId, + string VulnerabilityKey, + IReadOnlyList StatementIds, + string ConflictHash, + DateTimeOffset AsOf, + DateTimeOffset RecordedAt, + string Details, + MergeConflictExplainerPayload Explainer); + + private sealed class ConcelierApplicationFactory : WebApplicationFactory + { + private readonly string _connectionString; + private readonly string? _previousDsn; + private readonly string? _previousDriver; + private readonly string? _previousTimeout; + private readonly string? _previousTelemetryEnabled; + private readonly string? _previousTelemetryLogging; + private readonly string? _previousTelemetryTracing; + private readonly string? _previousTelemetryMetrics; + private readonly Action? _authorityConfigure; + private readonly IDictionary _additionalPreviousEnvironment = new Dictionary(StringComparer.OrdinalIgnoreCase); + public CollectingLoggerProvider LoggerProvider { get; } = new(); + + public ConcelierApplicationFactory( + string connectionString, + Action? authorityConfigure = null, + IDictionary? environmentOverrides = null) + { + _connectionString = connectionString; + _authorityConfigure = authorityConfigure; + _previousDsn = Environment.GetEnvironmentVariable("CONCELIER_STORAGE__DSN"); + _previousDriver = Environment.GetEnvironmentVariable("CONCELIER_STORAGE__DRIVER"); + _previousTimeout = Environment.GetEnvironmentVariable("CONCELIER_STORAGE__COMMANDTIMEOUTSECONDS"); + _previousTelemetryEnabled = Environment.GetEnvironmentVariable("CONCELIER_TELEMETRY__ENABLED"); + _previousTelemetryLogging = Environment.GetEnvironmentVariable("CONCELIER_TELEMETRY__ENABLELOGGING"); + _previousTelemetryTracing = Environment.GetEnvironmentVariable("CONCELIER_TELEMETRY__ENABLETRACING"); + _previousTelemetryMetrics = Environment.GetEnvironmentVariable("CONCELIER_TELEMETRY__ENABLEMETRICS"); + Environment.SetEnvironmentVariable("CONCELIER_STORAGE__DSN", connectionString); + Environment.SetEnvironmentVariable("CONCELIER_STORAGE__DRIVER", "mongo"); + Environment.SetEnvironmentVariable("CONCELIER_STORAGE__COMMANDTIMEOUTSECONDS", "30"); + Environment.SetEnvironmentVariable("CONCELIER_TELEMETRY__ENABLED", "false"); + Environment.SetEnvironmentVariable("CONCELIER_TELEMETRY__ENABLELOGGING", "false"); + Environment.SetEnvironmentVariable("CONCELIER_TELEMETRY__ENABLETRACING", "false"); + Environment.SetEnvironmentVariable("CONCELIER_TELEMETRY__ENABLEMETRICS", "false"); + if (environmentOverrides is not null) + { + foreach (var kvp in environmentOverrides) + { + var previous = Environment.GetEnvironmentVariable(kvp.Key); + _additionalPreviousEnvironment[kvp.Key] = previous; + Environment.SetEnvironmentVariable(kvp.Key, kvp.Value); + } + } + } + + protected override void ConfigureWebHost(IWebHostBuilder builder) + { + builder.ConfigureAppConfiguration((context, configurationBuilder) => + { + var settings = new Dictionary + { + ["Plugins:Directory"] = Path.Combine(context.HostingEnvironment.ContentRootPath, "StellaOps.Concelier.PluginBinaries"), + }; + + configurationBuilder.AddInMemoryCollection(settings!); + }); + + builder.ConfigureLogging(logging => + { + logging.AddProvider(LoggerProvider); + }); + + builder.ConfigureServices(services => + { + services.AddSingleton(); + services.AddSingleton(sp => sp.GetRequiredService()); + services.PostConfigure(options => + { + options.Storage.Driver = "mongo"; + options.Storage.Dsn = _connectionString; + options.Storage.CommandTimeoutSeconds = 30; + options.Plugins.Directory ??= Path.Combine(AppContext.BaseDirectory, "StellaOps.Concelier.PluginBinaries"); + options.Telemetry.Enabled = false; + options.Telemetry.EnableLogging = false; + options.Telemetry.EnableTracing = false; + options.Telemetry.EnableMetrics = false; + options.Authority ??= new ConcelierOptions.AuthorityOptions(); + _authorityConfigure?.Invoke(options.Authority); + }); + }); + + builder.ConfigureTestServices(services => + { + services.AddSingleton(); + services.PostConfigure(StellaOpsAuthenticationDefaults.AuthenticationScheme, options => + { + options.RequireHttpsMetadata = false; + options.TokenValidationParameters = new TokenValidationParameters + { + ValidateIssuerSigningKey = true, + IssuerSigningKey = TestSigningKey, + ValidateIssuer = false, + ValidateAudience = false, + ValidateLifetime = false, + NameClaimType = ClaimTypes.Name, + RoleClaimType = ClaimTypes.Role, + ClockSkew = TimeSpan.Zero + }; + var issuer = string.IsNullOrWhiteSpace(options.Authority) ? TestAuthorityIssuer : options.Authority; + options.ConfigurationManager = new StaticConfigurationManager(new OpenIdConnectConfiguration + { + Issuer = issuer + }); + }); + }); + } + + protected override void Dispose(bool disposing) + { + base.Dispose(disposing); + Environment.SetEnvironmentVariable("CONCELIER_STORAGE__DSN", _previousDsn); + Environment.SetEnvironmentVariable("CONCELIER_STORAGE__DRIVER", _previousDriver); + Environment.SetEnvironmentVariable("CONCELIER_STORAGE__COMMANDTIMEOUTSECONDS", _previousTimeout); + Environment.SetEnvironmentVariable("CONCELIER_TELEMETRY__ENABLED", _previousTelemetryEnabled); + Environment.SetEnvironmentVariable("CONCELIER_TELEMETRY__ENABLELOGGING", _previousTelemetryLogging); + Environment.SetEnvironmentVariable("CONCELIER_TELEMETRY__ENABLETRACING", _previousTelemetryTracing); + Environment.SetEnvironmentVariable("CONCELIER_TELEMETRY__ENABLEMETRICS", _previousTelemetryMetrics); + foreach (var kvp in _additionalPreviousEnvironment) + { + Environment.SetEnvironmentVariable(kvp.Key, kvp.Value); + } + + LoggerProvider.Dispose(); + } + + private sealed class RemoteIpStartupFilter : IStartupFilter + { + public Action Configure(Action next) + { + return app => + { + app.Use(async (context, nextMiddleware) => + { + if (context.Request.Headers.TryGetValue("X-Test-RemoteAddr", out var values) + && values.Count > 0 + && IPAddress.TryParse(values[0], out var remote)) + { + context.Connection.RemoteIpAddress = remote; + } + + await nextMiddleware(); + }); + + next(app); + }; + } + } + + public sealed record LogEntry( + string LoggerName, + LogLevel Level, + EventId EventId, + string? Message, + Exception? Exception, + IReadOnlyList> State) + { + public bool TryGetState(string name, out object? value) + { + foreach (var kvp in State) + { + if (string.Equals(kvp.Key, name, StringComparison.Ordinal)) + { + value = kvp.Value; + return true; + } + } + + value = null; + return false; + } + } + + public sealed class CollectingLoggerProvider : ILoggerProvider + { + private readonly object syncRoot = new(); + private readonly List entries = new(); + private bool disposed; + + public ILogger CreateLogger(string categoryName) => new CollectingLogger(categoryName, this); + + public IReadOnlyList Snapshot(string loggerName) + { + lock (syncRoot) + { + return entries + .Where(entry => string.Equals(entry.LoggerName, loggerName, StringComparison.Ordinal)) + .ToArray(); + } + } + + public void Dispose() + { + disposed = true; + lock (syncRoot) + { + entries.Clear(); + } + } + + private void Append(LogEntry entry) + { + if (disposed) + { + return; + } + + lock (syncRoot) + { + entries.Add(entry); + } + } + + private sealed class CollectingLogger : ILogger + { + private readonly string categoryName; + private readonly CollectingLoggerProvider provider; + + public CollectingLogger(string categoryName, CollectingLoggerProvider provider) + { + this.categoryName = categoryName; + this.provider = provider; + } + + public IDisposable? BeginScope(TState state) where TState : notnull => NullScope.Instance; + + public bool IsEnabled(LogLevel logLevel) => true; + + public void Log(LogLevel logLevel, EventId eventId, TState state, Exception? exception, Func formatter) + { + if (formatter is null) + { + throw new ArgumentNullException(nameof(formatter)); + } + + var message = formatter(state, exception); + var kvps = ExtractState(state); + var entry = new LogEntry(categoryName, logLevel, eventId, message, exception, kvps); + provider.Append(entry); + } + + private static IReadOnlyList> ExtractState(TState state) + { + if (state is IReadOnlyList> list) + { + return list; + } + + if (state is IEnumerable> enumerable) + { + return enumerable.ToArray(); + } + + if (state is null) + { + return Array.Empty>(); + } + + return new[] { new KeyValuePair("State", state) }; + } + } + + private sealed class NullScope : IDisposable + { + public static readonly NullScope Instance = new(); + public void Dispose() + { + } + } + } + } + + private sealed class TempDirectory : IDisposable + { + public string Path { get; } + + public TempDirectory() + { + Path = System.IO.Path.Combine(System.IO.Path.GetTempPath(), "concelier-mirror-" + Guid.NewGuid().ToString("N", CultureInfo.InvariantCulture)); + Directory.CreateDirectory(Path); + } + + public void Dispose() + { + try + { + if (Directory.Exists(Path)) + { + Directory.Delete(Path, recursive: true); + } + } + catch + { + // best effort cleanup + } + } + } + + private sealed record HealthPayload(string Status, DateTimeOffset StartedAt, double UptimeSeconds, StoragePayload Storage, TelemetryPayload Telemetry); + + private sealed record StoragePayload(string Driver, bool Completed, DateTimeOffset? CompletedAt, double? DurationMs); + + private sealed record TelemetryPayload(bool Enabled, bool Tracing, bool Metrics, bool Logging); + + private sealed record ReadyPayload(string Status, DateTimeOffset StartedAt, double UptimeSeconds, ReadyMongoPayload Mongo); + + private sealed record ReadyMongoPayload(string Status, double? LatencyMs, DateTimeOffset? CheckedAt, string? Error); + + private sealed record JobDefinitionPayload(string Kind, bool Enabled, string? CronExpression, TimeSpan Timeout, TimeSpan LeaseDuration, JobRunPayload? LastRun); + + private sealed record JobRunPayload(Guid RunId, string Kind, string Status, string Trigger, DateTimeOffset CreatedAt, DateTimeOffset? StartedAt, DateTimeOffset? CompletedAt, string? Error, TimeSpan? Duration, Dictionary Parameters); + + private sealed record ProblemDocument(string? Type, string? Title, int? Status, string? Detail, string? Instance); + + private async Task SeedAdvisoryRawDocumentsAsync(params BsonDocument[] documents) + { + var client = new MongoClient(_runner.ConnectionString); + var database = client.GetDatabase(MongoStorageDefaults.DefaultDatabaseName); + var collection = database.GetCollection(MongoStorageDefaults.Collections.AdvisoryRaw); + await collection.DeleteManyAsync(FilterDefinition.Empty); + if (documents.Length > 0) + { + await collection.InsertManyAsync(documents); + } + } + + private static BsonDocument CreateAdvisoryRawDocument( + string tenant, + string vendor, + string upstreamId, + string contentHash, + BsonDocument? raw = null, + string? supersedes = null) + { + var now = DateTime.UtcNow; + return new BsonDocument + { + { "_id", BuildRawDocumentId(vendor, upstreamId, contentHash) }, + { "tenant", tenant }, + { + "source", + new BsonDocument + { + { "vendor", vendor }, + { "connector", "test-connector" }, + { "version", "1.0.0" } + } + }, + { + "upstream", + new BsonDocument + { + { "upstream_id", upstreamId }, + { "document_version", "1" }, + { "retrieved_at", now }, + { "content_hash", contentHash }, + { "signature", new BsonDocument { { "present", false } } }, + { "provenance", new BsonDocument { { "api", "https://example.test" } } } + } + }, + { + "content", + new BsonDocument + { + { "format", "osv" }, + { "raw", raw ?? new BsonDocument("id", upstreamId) } + } + }, + { + "identifiers", + new BsonDocument + { + { "aliases", new BsonArray(new[] { upstreamId }) }, + { "primary", upstreamId } + } + }, + { + "linkset", + new BsonDocument + { + { "aliases", new BsonArray() }, + { "purls", new BsonArray() }, + { "cpes", new BsonArray() }, + { "references", new BsonArray() }, + { "reconciled_from", new BsonArray() }, + { "notes", new BsonDocument() } + } + }, + { "supersedes", supersedes is null ? BsonNull.Value : supersedes }, + { "ingested_at", now }, + { "created_at", now } + }; + } + + private static string BuildRawDocumentId(string vendor, string upstreamId, string contentHash) + { + static string Sanitize(string value) + { + if (string.IsNullOrWhiteSpace(value)) + { + return "unknown"; + } + + var buffer = new char[value.Length]; + var index = 0; + foreach (var ch in value.Trim().ToLowerInvariant()) + { + buffer[index++] = char.IsLetterOrDigit(ch) ? ch : '-'; + } + + var sanitized = new string(buffer, 0, index).Trim('-'); + return string.IsNullOrEmpty(sanitized) ? "unknown" : sanitized; + } + + var vendorSegment = Sanitize(vendor); + var upstreamSegment = Sanitize(upstreamId); + var hashSegment = Sanitize(contentHash.Replace(":", "-")); + return $"advisory_raw:{vendorSegment}:{upstreamSegment}:{hashSegment}"; + } + + private static AdvisoryIngestRequest BuildAdvisoryIngestRequest(string contentHash, string upstreamId) + { + var raw = CreateJsonElement($@"{{""id"":""{upstreamId}"",""modified"":""{DateTime.UtcNow:O}""}}"); + var references = new[] + { + new AdvisoryLinksetReferenceRequest("advisory", $"https://example.test/advisories/{upstreamId}", null) + }; + + return new AdvisoryIngestRequest( + new AdvisorySourceRequest("osv", "osv-connector", "1.0.0", "feed"), + new AdvisoryUpstreamRequest( + upstreamId, + "2025-01-01T00:00:00Z", + DateTimeOffset.UtcNow, + contentHash, + new AdvisorySignatureRequest(false, null, null, null, null, null), + new Dictionary { ["http.method"] = "GET" }), + new AdvisoryContentRequest("osv", "1.3.0", raw, null), + new AdvisoryIdentifiersRequest( + upstreamId, + new[] { upstreamId, $"{upstreamId}-ALIAS" }), + new AdvisoryLinksetRequest( + new[] { upstreamId }, + new[] { "pkg:npm/demo@1.0.0" }, + Array.Empty(), + references, + Array.Empty(), + new Dictionary { ["note"] = "ingest-test" })); + } + + private static JsonElement CreateJsonElement(string json) + { + using var document = JsonDocument.Parse(json); + return document.RootElement.Clone(); + } + + private static async Task> CaptureMetricsAsync(string meterName, string instrumentName, Func action) + { + var measurements = new List(); + var listener = new MeterListener(); + + listener.InstrumentPublished += (instrument, currentListener) => + { + if (string.Equals(instrument.Meter.Name, meterName, StringComparison.Ordinal) && + string.Equals(instrument.Name, instrumentName, StringComparison.Ordinal)) + { + currentListener.EnableMeasurementEvents(instrument); + } + }; + + listener.SetMeasurementEventCallback((instrument, measurement, tags, state) => + { + var tagDictionary = new Dictionary(StringComparer.Ordinal); + foreach (var tag in tags) + { + tagDictionary[tag.Key] = tag.Value; + } + + measurements.Add(new MetricMeasurement(instrument.Name, measurement, tagDictionary)); + }); + + listener.Start(); + try + { + await action().ConfigureAwait(false); + } + finally + { + listener.Dispose(); + } + + return measurements; + } + + private static string? GetTagValue(MetricMeasurement measurement, string tag) + { + if (measurement.Tags.TryGetValue(tag, out var value)) + { + return value?.ToString(); + } + + return null; + } + + private static string CreateTestToken(string tenant, params string[] scopes) + { + var normalizedTenant = string.IsNullOrWhiteSpace(tenant) ? "default" : tenant.Trim().ToLowerInvariant(); + var scopeSet = scopes is { Length: > 0 } + ? scopes + .Select(StellaOpsScopes.Normalize) + .Where(static scope => !string.IsNullOrEmpty(scope)) + .Select(static scope => scope!) + .Distinct(StringComparer.Ordinal) + .ToArray() + : Array.Empty(); + + var claims = new List + { + new Claim(StellaOpsClaimTypes.Subject, "test-user"), + new Claim(StellaOpsClaimTypes.Tenant, normalizedTenant), + new Claim(StellaOpsClaimTypes.Scope, string.Join(' ', scopeSet)) + }; + + foreach (var scope in scopeSet) + { + claims.Add(new Claim(StellaOpsClaimTypes.ScopeItem, scope)); + } + + var credentials = new SigningCredentials(TestSigningKey, SecurityAlgorithms.HmacSha256); + var now = DateTime.UtcNow; + var token = new JwtSecurityToken( + issuer: TestAuthorityIssuer, + audience: TestAuthorityAudience, + claims: claims, + notBefore: now.AddMinutes(-5), + expires: now.AddMinutes(30), + signingCredentials: credentials); + + return new JwtSecurityTokenHandler().WriteToken(token); + } + + private sealed record MetricMeasurement(string Instrument, long Value, IReadOnlyDictionary Tags); + + private sealed class DemoJob : IJob + { + public Task ExecuteAsync(JobExecutionContext context, CancellationToken cancellationToken) => Task.CompletedTask; + } + + private sealed class StubJobCoordinator : IJobCoordinator + { + public JobTriggerResult NextResult { get; set; } = JobTriggerResult.NotFound("not set"); + + public IReadOnlyList Definitions { get; set; } = Array.Empty(); + + public IReadOnlyList RecentRuns { get; set; } = Array.Empty(); + + public IReadOnlyList ActiveRuns { get; set; } = Array.Empty(); + + public Dictionary Runs { get; } = new(); + + public Dictionary LastRuns { get; } = new(StringComparer.Ordinal); + + public Task TriggerAsync(string kind, IReadOnlyDictionary? parameters, string trigger, CancellationToken cancellationToken) + => Task.FromResult(NextResult); + + public Task> GetDefinitionsAsync(CancellationToken cancellationToken) + => Task.FromResult(Definitions); + + public Task> GetRecentRunsAsync(string? kind, int limit, CancellationToken cancellationToken) + { + IEnumerable query = RecentRuns; + if (!string.IsNullOrWhiteSpace(kind)) + { + query = query.Where(run => string.Equals(run.Kind, kind, StringComparison.Ordinal)); + } + + return Task.FromResult>(query.Take(limit).ToArray()); + } + + public Task> GetActiveRunsAsync(CancellationToken cancellationToken) + => Task.FromResult(ActiveRuns); + + public Task GetRunAsync(Guid runId, CancellationToken cancellationToken) + => Task.FromResult(Runs.TryGetValue(runId, out var run) ? run : null); + + public Task GetLastRunAsync(string kind, CancellationToken cancellationToken) + => Task.FromResult(LastRuns.TryGetValue(kind, out var run) ? run : null); + + public Task> GetLastRunsAsync(IEnumerable kinds, CancellationToken cancellationToken) + { + var map = new Dictionary(StringComparer.Ordinal); + foreach (var kind in kinds) + { + if (kind is null) + { + continue; + } + + if (LastRuns.TryGetValue(kind, out var run) && run is not null) + { + map[kind] = run; + } + } + + return Task.FromResult>(map); + } + } +} diff --git a/src/Scanner/StellaOps.Scanner.Sbomer.BuildXPlugin/TASKS.md b/src/Scanner/StellaOps.Scanner.Sbomer.BuildXPlugin/TASKS.md index c54f1ea0e..f9ac29dd4 100644 --- a/src/Scanner/StellaOps.Scanner.Sbomer.BuildXPlugin/TASKS.md +++ b/src/Scanner/StellaOps.Scanner.Sbomer.BuildXPlugin/TASKS.md @@ -2,6 +2,6 @@ | ID | Status | Owner(s) | Depends on | Description | Exit Criteria | |----|--------|----------|------------|-------------|---------------| -| SCANNER-SURFACE-03 | TODO | BuildX Plugin Guild | SURFACE-FS-02 | Push layer manifests and entry fragments into Surface.FS during build-time SBOM generation. | BuildX integration tests confirm cache population; CLI docs updated. | +| SCANNER-SURFACE-03 | DOING (2025-11-06) | BuildX Plugin Guild | SURFACE-FS-02 | Push layer manifests and entry fragments into Surface.FS during build-time SBOM generation.
2025-11-06: Kicked off manifest emitter wiring within BuildX export pipeline and outlined test fixtures targeting Surface.FS client mock. | BuildX integration tests confirm cache population; CLI docs updated. | | SCANNER-ENV-03 | TODO | BuildX Plugin Guild | SURFACE-ENV-02 | Adopt Surface.Env helpers for plugin configuration (cache roots, CAS endpoints, feature toggles). | Plugin loads helper; misconfig errors logged; README updated. | | SCANNER-SECRETS-03 | TODO | BuildX Plugin Guild, Security Guild | SURFACE-SECRETS-02 | Use Surface.Secrets to retrieve registry credentials when interacting with CAS/referrers. | Secrets retrieved via shared library; e2e tests cover rotation; operations guide refreshed. | diff --git a/src/Scanner/StellaOps.Scanner.WebService/Contracts/SurfaceContracts.cs b/src/Scanner/StellaOps.Scanner.WebService/Contracts/SurfaceContracts.cs index 7644be678..44e83a206 100644 --- a/src/Scanner/StellaOps.Scanner.WebService/Contracts/SurfaceContracts.cs +++ b/src/Scanner/StellaOps.Scanner.WebService/Contracts/SurfaceContracts.cs @@ -1,6 +1,6 @@ using System; -using System.Collections.Generic; using System.Text.Json.Serialization; +using StellaOps.Scanner.Surface.FS; namespace StellaOps.Scanner.WebService.Contracts; @@ -28,60 +28,3 @@ public sealed record SurfacePointersDto [JsonPropertyOrder(4)] public SurfaceManifestDocument Manifest { get; init; } = new(); } - -public sealed record SurfaceManifestDocument -{ - [JsonPropertyName("schema")] - [JsonPropertyOrder(0)] - public string Schema { get; init; } = "stellaops.surface.manifest@1"; - - [JsonPropertyName("tenant")] - [JsonPropertyOrder(1)] - public string Tenant { get; init; } = string.Empty; - - [JsonPropertyName("imageDigest")] - [JsonPropertyOrder(2)] - public string ImageDigest { get; init; } = string.Empty; - - [JsonPropertyName("generatedAt")] - [JsonPropertyOrder(3)] - public DateTimeOffset GeneratedAt { get; init; } = DateTimeOffset.UtcNow; - - [JsonPropertyName("artifacts")] - [JsonPropertyOrder(4)] - public IReadOnlyList Artifacts { get; init; } = Array.Empty(); -} - -public sealed record SurfaceManifestArtifact -{ - [JsonPropertyName("kind")] - [JsonPropertyOrder(0)] - public string Kind { get; init; } = string.Empty; - - [JsonPropertyName("uri")] - [JsonPropertyOrder(1)] - public string Uri { get; init; } = string.Empty; - - [JsonPropertyName("digest")] - [JsonPropertyOrder(2)] - public string Digest { get; init; } = string.Empty; - - [JsonPropertyName("mediaType")] - [JsonPropertyOrder(3)] - public string MediaType { get; init; } = string.Empty; - - [JsonPropertyName("format")] - [JsonPropertyOrder(4)] - public string Format { get; init; } = string.Empty; - - [JsonPropertyName("sizeBytes")] - [JsonPropertyOrder(5)] - public long SizeBytes { get; init; } - = 0; - - [JsonPropertyName("view")] - [JsonPropertyOrder(6)] - [JsonIgnore(Condition = JsonIgnoreCondition.WhenWritingNull)] - public string? View { get; init; } - = null; -} diff --git a/src/Scanner/StellaOps.Scanner.WebService/Options/ScannerStorageOptionsPostConfigurator.cs b/src/Scanner/StellaOps.Scanner.WebService/Options/ScannerStorageOptionsPostConfigurator.cs new file mode 100644 index 000000000..54cfd3157 --- /dev/null +++ b/src/Scanner/StellaOps.Scanner.WebService/Options/ScannerStorageOptionsPostConfigurator.cs @@ -0,0 +1,118 @@ +using Microsoft.Extensions.Logging; +using Microsoft.Extensions.Options; +using StellaOps.Scanner.Storage; + +namespace StellaOps.Scanner.WebService.Options; + +internal sealed class ScannerStorageOptionsPostConfigurator : IPostConfigureOptions +{ + private readonly IOptionsMonitor _webOptions; + private readonly ILogger _logger; + + public ScannerStorageOptionsPostConfigurator( + IOptionsMonitor webOptions, + ILogger logger) + { + _webOptions = webOptions ?? throw new ArgumentNullException(nameof(webOptions)); + _logger = logger ?? throw new ArgumentNullException(nameof(logger)); + } + + public void PostConfigure(string? name, ScannerStorageOptions options) + { + ArgumentNullException.ThrowIfNull(options); + + var source = _webOptions.CurrentValue?.ArtifactStore; + if (source is null) + { + return; + } + + var target = options.ObjectStore ??= new ObjectStoreOptions(); + + if (!string.IsNullOrWhiteSpace(source.Driver)) + { + target.Driver = source.Driver; + } + + if (!string.IsNullOrWhiteSpace(source.Region)) + { + target.Region = source.Region!; + } + + if (!string.IsNullOrWhiteSpace(source.Bucket)) + { + target.BucketName = source.Bucket!; + } + + if (!string.IsNullOrWhiteSpace(source.RootPrefix)) + { + target.RootPrefix = source.RootPrefix; + } + + if (!string.IsNullOrWhiteSpace(source.Endpoint)) + { + if (target.IsRustFsDriver()) + { + target.RustFs ??= new RustFsOptions(); + target.RustFs.BaseUrl = source.Endpoint; + } + else + { + target.ServiceUrl = source.Endpoint; + } + } + + if (target.IsRustFsDriver()) + { + if (target.RustFs is null) + { + target.RustFs = new RustFsOptions(); + } + + target.RustFs.AllowInsecureTls = source.AllowInsecureTls; + + if (!string.IsNullOrWhiteSpace(source.ApiKeyHeader)) + { + target.RustFs.ApiKeyHeader = source.ApiKeyHeader!; + } + + if (!string.IsNullOrWhiteSpace(source.ApiKey)) + { + target.RustFs.ApiKey = source.ApiKey; + } + + if (!string.IsNullOrWhiteSpace(source.Endpoint)) + { + target.RustFs.BaseUrl = source.Endpoint!; + } + } + + if (!string.IsNullOrWhiteSpace(source.AccessKey)) + { + target.AccessKeyId = source.AccessKey; + } + + if (!string.IsNullOrWhiteSpace(source.SecretKey)) + { + target.SecretAccessKey = source.SecretKey; + } + + if (source.Headers is { Count: > 0 }) + { + foreach (var (key, value) in source.Headers) + { + if (string.IsNullOrWhiteSpace(key) || string.IsNullOrWhiteSpace(value)) + { + continue; + } + + target.Headers[key] = value; + } + } + + _logger.LogDebug( + "Mirrored artifact store settings into scanner storage options (driver: {Driver}, bucket: {Bucket}).", + target.Driver, + target.BucketName); + } +} diff --git a/src/Scanner/StellaOps.Scanner.WebService/Options/ScannerSurfaceSecretConfigurator.cs b/src/Scanner/StellaOps.Scanner.WebService/Options/ScannerSurfaceSecretConfigurator.cs new file mode 100644 index 000000000..5fd781f97 --- /dev/null +++ b/src/Scanner/StellaOps.Scanner.WebService/Options/ScannerSurfaceSecretConfigurator.cs @@ -0,0 +1,124 @@ +using Microsoft.Extensions.Logging; +using Microsoft.Extensions.Options; +using StellaOps.Scanner.Surface.Env; +using StellaOps.Scanner.Surface.Secrets; + +namespace StellaOps.Scanner.WebService.Options; + +internal sealed class ScannerSurfaceSecretConfigurator : IConfigureOptions +{ + private const string ComponentName = "Scanner.WebService"; + + private readonly ISurfaceSecretProvider _secretProvider; + private readonly ISurfaceEnvironment _surfaceEnvironment; + private readonly ILogger _logger; + + public ScannerSurfaceSecretConfigurator( + ISurfaceSecretProvider secretProvider, + ISurfaceEnvironment surfaceEnvironment, + ILogger logger) + { + _secretProvider = secretProvider ?? throw new ArgumentNullException(nameof(secretProvider)); + _surfaceEnvironment = surfaceEnvironment ?? throw new ArgumentNullException(nameof(surfaceEnvironment)); + _logger = logger ?? throw new ArgumentNullException(nameof(logger)); + } + + public void Configure(ScannerWebServiceOptions options) + { + ArgumentNullException.ThrowIfNull(options); + + var tenant = _surfaceEnvironment.Settings.Secrets.Tenant; + var request = new SurfaceSecretRequest( + Tenant: tenant, + Component: ComponentName, + SecretType: "cas-access"); + + CasAccessSecret? secret = null; + try + { + using var handle = _secretProvider.GetAsync(request).AsTask().GetAwaiter().GetResult(); + secret = SurfaceSecretParser.ParseCasAccessSecret(handle); + } + catch (SurfaceSecretNotFoundException) + { + _logger.LogDebug("Surface secret 'cas-access' not found for {Component}; retaining configured artifact store settings.", ComponentName); + } + catch (Exception ex) + { + _logger.LogWarning(ex, "Failed to resolve surface secret 'cas-access' for {Component}.", ComponentName); + } + + if (secret is null) + { + return; + } + + ApplySecret(options.ArtifactStore ??= new ScannerWebServiceOptions.ArtifactStoreOptions(), secret); + } + + private void ApplySecret(ScannerWebServiceOptions.ArtifactStoreOptions artifactStore, CasAccessSecret secret) + { + if (!string.IsNullOrWhiteSpace(secret.Driver)) + { + artifactStore.Driver = secret.Driver; + } + + if (!string.IsNullOrWhiteSpace(secret.Endpoint)) + { + artifactStore.Endpoint = secret.Endpoint!; + } + + if (secret.AllowInsecureTls is { } insecure) + { + artifactStore.AllowInsecureTls = insecure; + artifactStore.UseTls = !insecure; + } + + if (!string.IsNullOrWhiteSpace(secret.Region)) + { + artifactStore.Region = secret.Region; + } + + if (!string.IsNullOrWhiteSpace(secret.Bucket)) + { + artifactStore.Bucket = secret.Bucket!; + } + + if (!string.IsNullOrWhiteSpace(secret.RootPrefix)) + { + artifactStore.RootPrefix = secret.RootPrefix!; + } + + if (!string.IsNullOrWhiteSpace(secret.ApiKeyHeader)) + { + artifactStore.ApiKeyHeader = secret.ApiKeyHeader!; + } + + if (!string.IsNullOrWhiteSpace(secret.ApiKey)) + { + artifactStore.ApiKey = secret.ApiKey; + } + + if (!string.IsNullOrWhiteSpace(secret.AccessKeyId) && !string.IsNullOrWhiteSpace(secret.SecretAccessKey)) + { + artifactStore.AccessKey = secret.AccessKeyId!; + artifactStore.SecretKey = secret.SecretAccessKey!; + } + + foreach (var header in secret.Headers) + { + if (string.IsNullOrWhiteSpace(header.Key) || string.IsNullOrWhiteSpace(header.Value)) + { + continue; + } + + artifactStore.Headers[header.Key] = header.Value; + } + + _logger.LogInformation( + "Surface secret 'cas-access' applied for {Component} (driver: {Driver}, bucket: {Bucket}).", + ComponentName, + artifactStore.Driver, + artifactStore.Bucket); + } +} diff --git a/src/Scanner/StellaOps.Scanner.WebService/Options/ScannerWebServiceOptions.cs b/src/Scanner/StellaOps.Scanner.WebService/Options/ScannerWebServiceOptions.cs index 913c615a9..94a6f829c 100644 --- a/src/Scanner/StellaOps.Scanner.WebService/Options/ScannerWebServiceOptions.cs +++ b/src/Scanner/StellaOps.Scanner.WebService/Options/ScannerWebServiceOptions.cs @@ -61,6 +61,11 @@ public sealed class ScannerWebServiceOptions /// public ApiOptions Api { get; set; } = new(); + /// + /// Console (UI) routing settings used for orchestrator link generation. + /// + public ConsoleOptions Console { get; set; } = new(); + /// /// Platform event emission settings. /// @@ -266,6 +271,17 @@ public sealed class ScannerWebServiceOptions public string RuntimeSegment { get; set; } = "runtime"; } + public sealed class ConsoleOptions + { + public string BasePath { get; set; } = "/ui"; + + public string ReportsSegment { get; set; } = "reports"; + + public string PolicySegment { get; set; } = "policy"; + + public string AttestationsSegment { get; set; } = "attestations"; + } + public sealed class EventsOptions { public bool Enabled { get; set; } diff --git a/src/Scanner/StellaOps.Scanner.WebService/Options/ScannerWebServiceOptionsPostConfigure.cs b/src/Scanner/StellaOps.Scanner.WebService/Options/ScannerWebServiceOptionsPostConfigure.cs index 03b9426a5..84c7908ff 100644 --- a/src/Scanner/StellaOps.Scanner.WebService/Options/ScannerWebServiceOptionsPostConfigure.cs +++ b/src/Scanner/StellaOps.Scanner.WebService/Options/ScannerWebServiceOptionsPostConfigure.cs @@ -30,13 +30,7 @@ public static class ScannerWebServiceOptionsPostConfigure options.ArtifactStore ??= new ScannerWebServiceOptions.ArtifactStoreOptions(); var artifactStore = options.ArtifactStore; - if (string.IsNullOrWhiteSpace(artifactStore.SecretKey) - && !string.IsNullOrWhiteSpace(artifactStore.SecretKeyFile)) - { - artifactStore.SecretKey = ReadSecretFile(artifactStore.SecretKeyFile!, contentRootPath); - } - - options.Signing ??= new ScannerWebServiceOptions.SigningOptions(); + options.Signing ??= new ScannerWebServiceOptions.SigningOptions(); var signing = options.Signing; if (string.IsNullOrWhiteSpace(signing.KeyPem) && !string.IsNullOrWhiteSpace(signing.KeyPemFile)) diff --git a/src/Scanner/StellaOps.Scanner.WebService/Program.cs b/src/Scanner/StellaOps.Scanner.WebService/Program.cs index 83df2c0cc..c9a95c6b3 100644 --- a/src/Scanner/StellaOps.Scanner.WebService/Program.cs +++ b/src/Scanner/StellaOps.Scanner.WebService/Program.cs @@ -97,6 +97,7 @@ builder.Services.AddSurfaceEnvironment(options => builder.Services.AddSurfaceValidation(); builder.Services.AddSurfaceFileCache(); builder.Services.AddSurfaceSecrets(); +builder.Services.AddSingleton, ScannerSurfaceSecretConfigurator>(); builder.Services.AddSingleton>(sp => new SurfaceCacheOptionsConfigurator(sp.GetRequiredService())); builder.Services.AddSingleton(); @@ -179,6 +180,7 @@ builder.Services.AddScannerStorage(storageOptions => storageOptions.ObjectStore.RustFs.BaseUrl = string.Empty; } }); +builder.Services.AddSingleton, ScannerStorageOptionsPostConfigurator>(); builder.Services.AddSingleton(); builder.Services.AddSingleton(); builder.Services.AddSingleton(); diff --git a/src/Scanner/StellaOps.Scanner.WebService/Services/ReportEventDispatcher.cs b/src/Scanner/StellaOps.Scanner.WebService/Services/ReportEventDispatcher.cs index 41fd0925b..2f3a377ad 100644 --- a/src/Scanner/StellaOps.Scanner.WebService/Services/ReportEventDispatcher.cs +++ b/src/Scanner/StellaOps.Scanner.WebService/Services/ReportEventDispatcher.cs @@ -16,20 +16,24 @@ namespace StellaOps.Scanner.WebService.Services; internal sealed class ReportEventDispatcher : IReportEventDispatcher { - private const string DefaultTenant = "default"; - private const string Source = "scanner.webservice"; - - private readonly IPlatformEventPublisher _publisher; - private readonly TimeProvider _timeProvider; - private readonly ILogger _logger; - private readonly string[] _apiBaseSegments; - private readonly string _reportsSegment; - private readonly string _policySegment; - - public ReportEventDispatcher( - IPlatformEventPublisher publisher, - IOptions options, - TimeProvider timeProvider, + private const string DefaultTenant = "default"; + private const string Source = "scanner.webservice"; + + private readonly IPlatformEventPublisher _publisher; + private readonly TimeProvider _timeProvider; + private readonly ILogger _logger; + private readonly string[] _apiBaseSegments; + private readonly string _reportsSegment; + private readonly string _policySegment; + private readonly string[] _consoleBaseSegments; + private readonly string _consoleReportsSegment; + private readonly string _consolePolicySegment; + private readonly string _consoleAttestationsSegment; + + public ReportEventDispatcher( + IPlatformEventPublisher publisher, + IOptions options, + TimeProvider timeProvider, ILogger logger) { _publisher = publisher ?? throw new ArgumentNullException(nameof(publisher)); @@ -38,17 +42,28 @@ internal sealed class ReportEventDispatcher : IReportEventDispatcher throw new ArgumentNullException(nameof(options)); } - var apiOptions = options.Value.Api ?? new ScannerWebServiceOptions.ApiOptions(); - _apiBaseSegments = SplitSegments(apiOptions.BasePath); - _reportsSegment = string.IsNullOrWhiteSpace(apiOptions.ReportsSegment) - ? "reports" - : apiOptions.ReportsSegment.Trim('/'); - _policySegment = string.IsNullOrWhiteSpace(apiOptions.PolicySegment) - ? "policy" - : apiOptions.PolicySegment.Trim('/'); - _timeProvider = timeProvider ?? throw new ArgumentNullException(nameof(timeProvider)); - _logger = logger ?? throw new ArgumentNullException(nameof(logger)); - } + var apiOptions = options.Value.Api ?? new ScannerWebServiceOptions.ApiOptions(); + _apiBaseSegments = SplitSegments(apiOptions.BasePath); + _reportsSegment = string.IsNullOrWhiteSpace(apiOptions.ReportsSegment) + ? "reports" + : apiOptions.ReportsSegment.Trim('/'); + _policySegment = string.IsNullOrWhiteSpace(apiOptions.PolicySegment) + ? "policy" + : apiOptions.PolicySegment.Trim('/'); + var consoleOptions = options.Value.Console ?? new ScannerWebServiceOptions.ConsoleOptions(); + _consoleBaseSegments = SplitSegments(consoleOptions.BasePath); + _consoleReportsSegment = string.IsNullOrWhiteSpace(consoleOptions.ReportsSegment) + ? "reports" + : consoleOptions.ReportsSegment.Trim('/'); + _consolePolicySegment = string.IsNullOrWhiteSpace(consoleOptions.PolicySegment) + ? "policy" + : consoleOptions.PolicySegment.Trim('/'); + _consoleAttestationsSegment = string.IsNullOrWhiteSpace(consoleOptions.AttestationsSegment) + ? "attestations" + : consoleOptions.AttestationsSegment.Trim('/'); + _timeProvider = timeProvider ?? throw new ArgumentNullException(nameof(timeProvider)); + _logger = logger ?? throw new ArgumentNullException(nameof(logger)); + } public async Task PublishAsync( ReportRequestDto request, @@ -240,21 +255,21 @@ internal sealed class ReportEventDispatcher : IReportEventDispatcher }; } - private ReportLinksPayload BuildLinks(HttpContext context, ReportDocumentDto document, DsseEnvelopeDto? envelope) - { - if (!context.Request.Host.HasValue) - { - return new ReportLinksPayload(); - } - - var reportUi = BuildAbsoluteUri(context, "ui", "reports", document.ReportId); + private ReportLinksPayload BuildLinks(HttpContext context, ReportDocumentDto document, DsseEnvelopeDto? envelope) + { + if (!context.Request.Host.HasValue) + { + return new ReportLinksPayload(); + } + + var reportUi = BuildAbsoluteUri(context, ConcatSegments(_consoleBaseSegments, _consoleReportsSegment, document.ReportId)); var reportApi = BuildAbsoluteUri(context, ConcatSegments(_apiBaseSegments, _reportsSegment, document.ReportId)); LinkTarget? policyLink = null; if (!string.IsNullOrWhiteSpace(document.Policy.RevisionId)) { var policyRevision = document.Policy.RevisionId!; - var policyUi = BuildAbsoluteUri(context, "ui", "policy", "revisions", policyRevision); + var policyUi = BuildAbsoluteUri(context, ConcatSegments(_consoleBaseSegments, _consolePolicySegment, "revisions", policyRevision)); var policyApi = BuildAbsoluteUri(context, ConcatSegments(_apiBaseSegments, _policySegment, "revisions", policyRevision)); policyLink = LinkTarget.Create(policyUi, policyApi); } @@ -262,7 +277,7 @@ internal sealed class ReportEventDispatcher : IReportEventDispatcher LinkTarget? attestationLink = null; if (envelope is not null) { - var attestationUi = BuildAbsoluteUri(context, "ui", "attestations", document.ReportId); + var attestationUi = BuildAbsoluteUri(context, ConcatSegments(_consoleBaseSegments, _consoleAttestationsSegment, document.ReportId)); var attestationApi = BuildAbsoluteUri(context, ConcatSegments(_apiBaseSegments, _reportsSegment, document.ReportId, "attestation")); attestationLink = LinkTarget.Create(attestationUi, attestationApi); } diff --git a/src/Scanner/StellaOps.Scanner.WebService/TASKS.md b/src/Scanner/StellaOps.Scanner.WebService/TASKS.md index f9b047f87..35fdfa54e 100644 --- a/src/Scanner/StellaOps.Scanner.WebService/TASKS.md +++ b/src/Scanner/StellaOps.Scanner.WebService/TASKS.md @@ -6,9 +6,9 @@ | SCANNER-SURFACE-02 | DONE (2025-11-05) | Scanner WebService Guild | SURFACE-FS-02 | Publish Surface.FS pointers (CAS URIs, manifests) via scan/report APIs and update attestation metadata.
2025-11-05: Surface pointers projected through scan/report endpoints, orchestrator samples + DSSE fixtures refreshed with manifest block, readiness tests updated to use validator stub. | OpenAPI updated; clients regenerated; integration tests validate pointer presence and tenancy. | | SCANNER-ENV-02 | TODO (2025-11-06) | Scanner WebService Guild, Ops Guild | SURFACE-ENV-02 | Wire Surface.Env helpers into WebService hosting (cache roots, feature flags) and document configuration.
2025-11-02: Cache root resolution switched to helper; feature flag bindings updated; Helm/Compose updates pending review.
2025-11-05 14:55Z: Aligning readiness checks, docs, and Helm/Compose templates with Surface.Env outputs and planning test coverage for configuration fallbacks.
2025-11-06 17:05Z: Surface.Env documentation/README refreshed; warning catalogue captured for ops handoff.
2025-11-06 07:45Z: Helm values (dev/stage/prod/airgap/mirror) and Compose examples updated with `SCANNER_SURFACE_*` defaults plus rollout warning note in `deploy/README.md`.
2025-11-06 07:55Z: Paused; follow-up automation captured under `DEVOPS-OPENSSL-11-001/002` and pending Surface.Env readiness tests. | Service uses helper; env table documented; helm/compose templates updated. | > 2025-11-05 19:18Z: Added configurator to project wiring and unit test ensuring Surface.Env cache root is honoured. -| SCANNER-SECRETS-02 | DOING (2025-11-02) | Scanner WebService Guild, Security Guild | SURFACE-SECRETS-02 | Replace ad-hoc secret wiring with Surface.Secrets for report/export operations (registry and CAS tokens).
2025-11-02: Export/report flows now depend on Surface.Secrets stub; integration tests in progress. | Secrets fetched through shared provider; unit/integration tests cover rotation + failure cases. | +| SCANNER-SECRETS-02 | DOING (2025-11-06) | Scanner WebService Guild, Security Guild | SURFACE-SECRETS-02 | Replace ad-hoc secret wiring with Surface.Secrets for report/export operations (registry and CAS tokens).
2025-11-02: Export/report flows now depend on Surface.Secrets stub; integration tests in progress.
2025-11-06: Restarting work to eliminate file-based secrets, plumb provider handles through report/export services, and extend failure/rotation tests.
2025-11-06 21:40Z: Added configurator + storage post-config to hydrate artifact/CAS credentials from `cas-access` secrets with unit coverage. | Secrets fetched through shared provider; unit/integration tests cover rotation + failure cases. | | SCANNER-EVENTS-16-301 | BLOCKED (2025-10-26) | Scanner WebService Guild | ORCH-SVC-38-101, NOTIFY-SVC-38-001 | Emit orchestrator-compatible envelopes (`scanner.event.*`) and update integration tests to verify Notifier ingestion (no Redis queue coupling). | Tests assert envelope schema + orchestrator publish; Notifier consumer harness passes; docs updated with new event contract. Blocked by .NET 10 preview OpenAPI/Auth dependency drift preventing `dotnet test` completion. | -| SCANNER-EVENTS-16-302 | DOING (2025-10-26) | Scanner WebService Guild | SCANNER-EVENTS-16-301 | Extend orchestrator event links (report/policy/attestation) once endpoints are finalised across gateway + console. | Links section covers UI/API targets; downstream consumers validated; docs/samples updated. | +| SCANNER-EVENTS-16-302 | DONE (2025-11-06) | Scanner WebService Guild | SCANNER-EVENTS-16-301 | Extend orchestrator event links (report/policy/attestation) once endpoints are finalised across gateway + console.
2025-11-06 22:55Z: Dispatcher now honours configurable API/console base segments, JSON samples/docs refreshed, and `ReportEventDispatcherTests` extended. Tests: `StellaOps.Scanner.WebService.Tests` build until pre-existing `SurfaceCacheOptionsConfiguratorTests` ctor signature drift (tracked separately). | Links section covers UI/API targets; downstream consumers validated; docs/samples updated. | ## Graph Explorer v1 (Sprint 21) diff --git a/src/Scanner/StellaOps.Scanner.Worker/Options/ScannerStorageSurfaceSecretConfigurator.cs b/src/Scanner/StellaOps.Scanner.Worker/Options/ScannerStorageSurfaceSecretConfigurator.cs new file mode 100644 index 000000000..5540fb080 --- /dev/null +++ b/src/Scanner/StellaOps.Scanner.Worker/Options/ScannerStorageSurfaceSecretConfigurator.cs @@ -0,0 +1,141 @@ +using Microsoft.Extensions.Logging; +using Microsoft.Extensions.Options; +using StellaOps.Scanner.Storage; +using StellaOps.Scanner.Surface.Env; +using StellaOps.Scanner.Surface.Secrets; + +namespace StellaOps.Scanner.Worker.Options; + +internal sealed class ScannerStorageSurfaceSecretConfigurator : IConfigureOptions +{ + private static readonly string ComponentName = "Scanner.Worker"; + + private readonly ISurfaceSecretProvider _secretProvider; + private readonly ISurfaceEnvironment _surfaceEnvironment; + private readonly ILogger _logger; + + public ScannerStorageSurfaceSecretConfigurator( + ISurfaceSecretProvider secretProvider, + ISurfaceEnvironment surfaceEnvironment, + ILogger logger) + { + _secretProvider = secretProvider ?? throw new ArgumentNullException(nameof(secretProvider)); + _surfaceEnvironment = surfaceEnvironment ?? throw new ArgumentNullException(nameof(surfaceEnvironment)); + _logger = logger ?? throw new ArgumentNullException(nameof(logger)); + } + + public void Configure(ScannerStorageOptions options) + { + ArgumentNullException.ThrowIfNull(options); + + var tenant = _surfaceEnvironment.Settings.Secrets.Tenant; + var request = new SurfaceSecretRequest( + Tenant: tenant, + Component: ComponentName, + SecretType: "cas-access"); + + CasAccessSecret? secret = null; + try + { + using var handle = _secretProvider.GetAsync(request).AsTask().GetAwaiter().GetResult(); + secret = SurfaceSecretParser.ParseCasAccessSecret(handle); + } + catch (SurfaceSecretNotFoundException) + { + _logger.LogDebug("Surface secret 'cas-access' not found for {Component}; using configured storage settings.", ComponentName); + } + catch (Exception ex) + { + _logger.LogWarning(ex, "Failed to resolve surface secret 'cas-access' for {Component}.", ComponentName); + } + + if (secret is null) + { + return; + } + + ApplySecret(options, secret); + } + + private void ApplySecret(ScannerStorageOptions options, CasAccessSecret secret) + { + var objectStore = options.ObjectStore ??= new ObjectStoreOptions(); + + if (!string.IsNullOrWhiteSpace(secret.Driver)) + { + objectStore.Driver = secret.Driver; + } + + if (!string.IsNullOrWhiteSpace(secret.Region)) + { + objectStore.Region = secret.Region; + } + + if (!string.IsNullOrWhiteSpace(secret.Bucket)) + { + objectStore.BucketName = secret.Bucket; + } + + if (!string.IsNullOrWhiteSpace(secret.RootPrefix)) + { + objectStore.RootPrefix = secret.RootPrefix; + } + + if (!string.IsNullOrWhiteSpace(secret.Endpoint)) + { + if (objectStore.IsRustFsDriver()) + { + objectStore.RustFs ??= new RustFsOptions(); + objectStore.RustFs.BaseUrl = secret.Endpoint!; + } + else + { + objectStore.ServiceUrl = secret.Endpoint; + } + } + + if (objectStore.IsRustFsDriver()) + { + objectStore.RustFs ??= new RustFsOptions(); + + if (!string.IsNullOrWhiteSpace(secret.ApiKeyHeader)) + { + objectStore.RustFs.ApiKeyHeader = secret.ApiKeyHeader!; + } + + if (!string.IsNullOrWhiteSpace(secret.ApiKey)) + { + objectStore.RustFs.ApiKey = secret.ApiKey; + } + + if (secret.AllowInsecureTls is { } insecure) + { + objectStore.RustFs.AllowInsecureTls = insecure; + } + } + + if (!string.IsNullOrWhiteSpace(secret.AccessKeyId) && !string.IsNullOrWhiteSpace(secret.SecretAccessKey)) + { + objectStore.AccessKeyId = secret.AccessKeyId; + objectStore.SecretAccessKey = secret.SecretAccessKey; + objectStore.SessionToken = secret.SessionToken; + } + + foreach (var kvp in secret.Headers) + { + if (string.IsNullOrWhiteSpace(kvp.Key) || string.IsNullOrWhiteSpace(kvp.Value)) + { + continue; + } + + objectStore.Headers[kvp.Key] = kvp.Value; + } + + _logger.LogInformation( + "Surface secret 'cas-access' applied for {Component} (driver: {Driver}, bucket: {Bucket}, region: {Region}).", + ComponentName, + objectStore.Driver, + objectStore.BucketName, + objectStore.Region); + } +} diff --git a/src/Scanner/StellaOps.Scanner.Worker/Processing/CompositeScanAnalyzerDispatcher.cs b/src/Scanner/StellaOps.Scanner.Worker/Processing/CompositeScanAnalyzerDispatcher.cs index bcd193f2a..dab083a25 100644 --- a/src/Scanner/StellaOps.Scanner.Worker/Processing/CompositeScanAnalyzerDispatcher.cs +++ b/src/Scanner/StellaOps.Scanner.Worker/Processing/CompositeScanAnalyzerDispatcher.cs @@ -21,6 +21,7 @@ using StellaOps.Scanner.Surface.Env; using StellaOps.Scanner.Surface.FS; using StellaOps.Scanner.Surface.Validation; using StellaOps.Scanner.Worker.Options; +using StellaOps.Scanner.Worker.Diagnostics; namespace StellaOps.Scanner.Worker.Processing; @@ -206,7 +207,7 @@ internal sealed class CompositeScanAnalyzerDispatcher : IScanAnalyzerDispatcher try { var engine = new LanguageAnalyzerEngine(new[] { analyzer }); - var cacheEntry = await cacheAdapter.GetOrCreateAsync( + var cacheEntry = await cacheAdapter.GetOrCreateEntryAsync( _logger, analyzer.Id, workspaceFingerprint, diff --git a/src/Scanner/StellaOps.Scanner.Worker/Processing/Surface/SurfaceManifestPublisher.cs b/src/Scanner/StellaOps.Scanner.Worker/Processing/Surface/SurfaceManifestPublisher.cs new file mode 100644 index 000000000..4b5f8cc57 --- /dev/null +++ b/src/Scanner/StellaOps.Scanner.Worker/Processing/Surface/SurfaceManifestPublisher.cs @@ -0,0 +1,264 @@ +using System.Collections.Generic; +using System.Collections.Immutable; +using System.IO; +using System.Security.Cryptography; +using System.Text; +using System.Text.Json; +using Microsoft.Extensions.Logging; +using Microsoft.Extensions.Options; +using StellaOps.Scanner.Core.Contracts; +using StellaOps.Scanner.Surface.FS; +using StellaOps.Scanner.Storage; +using StellaOps.Scanner.Storage.Catalog; +using StellaOps.Scanner.Storage.ObjectStore; +using StellaOps.Scanner.Storage.Repositories; +using StellaOps.Scanner.Storage.Services; +using StellaOps.Scanner.Surface.Env; + +namespace StellaOps.Scanner.Worker.Processing.Surface; + +internal sealed record SurfaceManifestPayload( + ArtifactDocumentType ArtifactType, + ArtifactDocumentFormat ArtifactFormat, + string Kind, + string MediaType, + ReadOnlyMemory Content, + string? View = null, + IReadOnlyDictionary? Metadata = null, + bool RegisterArtifact = false); + +internal sealed record SurfaceManifestRequest( + string ScanId, + string ImageDigest, + int Attempt, + IReadOnlyDictionary Metadata, + IReadOnlyList Payloads, + string Component, + string? Version, + string? WorkerInstance); + +internal sealed class SurfaceManifestPublisher +{ + private static readonly JsonSerializerOptions SerializerOptions = new(JsonSerializerDefaults.Web) + { + WriteIndented = false, + DefaultIgnoreCondition = JsonIgnoreCondition.WhenWritingNull + }; + + private readonly IArtifactObjectStore _objectStore; + private readonly ArtifactRepository _artifactRepository; + private readonly LinkRepository _linkRepository; + private readonly ScannerStorageOptions _storageOptions; + private readonly ISurfaceEnvironment _surfaceEnvironment; + private readonly TimeProvider _timeProvider; + private readonly ILogger _logger; + + public SurfaceManifestPublisher( + IArtifactObjectStore objectStore, + ArtifactRepository artifactRepository, + LinkRepository linkRepository, + IOptions storageOptions, + ISurfaceEnvironment surfaceEnvironment, + TimeProvider timeProvider, + ILogger logger) + { + _objectStore = objectStore ?? throw new ArgumentNullException(nameof(objectStore)); + _artifactRepository = artifactRepository ?? throw new ArgumentNullException(nameof(artifactRepository)); + _linkRepository = linkRepository ?? throw new ArgumentNullException(nameof(linkRepository)); + _storageOptions = (storageOptions ?? throw new ArgumentNullException(nameof(storageOptions))).Value; + _surfaceEnvironment = surfaceEnvironment ?? throw new ArgumentNullException(nameof(surfaceEnvironment)); + _timeProvider = timeProvider ?? throw new ArgumentNullException(nameof(timeProvider)); + _logger = logger ?? throw new ArgumentNullException(nameof(logger)); + } + + public async Task PublishAsync(SurfaceManifestRequest request, CancellationToken cancellationToken) + { + ArgumentNullException.ThrowIfNull(request); + + if (request.Payloads.Count == 0) + { + throw new ArgumentException("At least one payload must be provided.", nameof(request)); + } + + var tenant = _surfaceEnvironment.Settings.Tenant; + var generatedAt = _timeProvider.GetUtcNow(); + var artifacts = new List(request.Payloads.Count); + + foreach (var payload in request.Payloads) + { + var artifact = await StorePayloadAsync(payload, tenant, cancellationToken).ConfigureAwait(false); + artifacts.Add(artifact); + } + + var manifestDocument = new SurfaceManifestDocument + { + Tenant = tenant, + ImageDigest = NormalizeDigest(request.ImageDigest), + ScanId = request.ScanId, + GeneratedAt = generatedAt, + Source = new SurfaceManifestSource + { + Component = request.Component, + Version = request.Version, + WorkerInstance = request.WorkerInstance, + Attempt = request.Attempt + }, + Artifacts = artifacts.ToImmutableArray() + }; + + var manifestBytes = JsonSerializer.SerializeToUtf8Bytes(manifestDocument, SerializerOptions); + var manifestDigest = ComputeDigest(manifestBytes); + var manifestKey = ArtifactObjectKeyBuilder.Build( + ArtifactDocumentType.SurfaceManifest, + ArtifactDocumentFormat.SurfaceManifestJson, + manifestDigest, + _storageOptions.ObjectStore.RootPrefix); + var manifestDescriptor = new ArtifactObjectDescriptor( + _storageOptions.ObjectStore.BucketName, + manifestKey, + Immutable: true, + RetainFor: _storageOptions.ObjectStore.ComplianceRetention); + + await using (var stream = new MemoryStream(manifestBytes, writable: false)) + { + await _objectStore.PutAsync(manifestDescriptor, stream, cancellationToken).ConfigureAwait(false); + } + + if (_storageOptions.DualWrite.Enabled && !string.IsNullOrWhiteSpace(_storageOptions.DualWrite.MirrorBucket)) + { + await using var mirrorStream = new MemoryStream(manifestBytes, writable: false); + var mirrorDescriptor = manifestDescriptor with { Bucket = _storageOptions.DualWrite.MirrorBucket! }; + await _objectStore.PutAsync(mirrorDescriptor, mirrorStream, cancellationToken).ConfigureAwait(false); + } + + var nowUtc = generatedAt.UtcDateTime; + var artifactId = CatalogIdFactory.CreateArtifactId(ArtifactDocumentType.SurfaceManifest, manifestDigest); + var manifestDocumentRecord = new ArtifactDocument + { + Id = artifactId, + Type = ArtifactDocumentType.SurfaceManifest, + Format = ArtifactDocumentFormat.SurfaceManifestJson, + MediaType = "application/vnd.stellaops.surface.manifest+json", + BytesSha256 = manifestDigest, + SizeBytes = manifestBytes.Length, + Immutable = true, + RefCount = 1, + CreatedAtUtc = nowUtc, + UpdatedAtUtc = nowUtc, + TtlClass = "surface.manifest" + }; + + await _artifactRepository.UpsertAsync(manifestDocumentRecord, cancellationToken).ConfigureAwait(false); + + var link = new LinkDocument + { + Id = CatalogIdFactory.CreateLinkId(LinkSourceType.Image, manifestDocument.ImageDigest ?? request.ScanId, artifactId), + FromType = LinkSourceType.Image, + FromDigest = manifestDocument.ImageDigest ?? request.ScanId, + ArtifactId = artifactId, + CreatedAtUtc = nowUtc + }; + + await _linkRepository.UpsertAsync(link, cancellationToken).ConfigureAwait(false); + + var manifestUri = BuildCasUri(_storageOptions.ObjectStore.BucketName, manifestKey); + _logger.LogInformation("Published surface manifest {Manifest} for image {ImageDigest}.", artifactId, manifestDocument.ImageDigest); + + return new SurfaceManifestPublishResult( + ManifestDigest: manifestDigest, + ManifestUri: manifestUri, + ArtifactId: artifactId, + Document: manifestDocument); + } + + private async Task StorePayloadAsync(SurfaceManifestPayload payload, string tenant, CancellationToken cancellationToken) + { + var digest = ComputeDigest(payload.Content.Span); + var key = ArtifactObjectKeyBuilder.Build( + payload.ArtifactType, + payload.ArtifactFormat, + digest, + _storageOptions.ObjectStore.RootPrefix); + + await using (var stream = new MemoryStream(payload.Content.ToArray(), writable: false)) + { + var descriptor = new ArtifactObjectDescriptor( + _storageOptions.ObjectStore.BucketName, + key, + Immutable: true, + RetainFor: _storageOptions.ObjectStore.ComplianceRetention); + + await _objectStore.PutAsync(descriptor, stream, cancellationToken).ConfigureAwait(false); + + if (_storageOptions.DualWrite.Enabled && !string.IsNullOrWhiteSpace(_storageOptions.DualWrite.MirrorBucket)) + { + await using var mirrorStream = new MemoryStream(payload.Content.ToArray(), writable: false); + var mirrorDescriptor = descriptor with { Bucket = _storageOptions.DualWrite.MirrorBucket! }; + await _objectStore.PutAsync(mirrorDescriptor, mirrorStream, cancellationToken).ConfigureAwait(false); + } + } + + return new SurfaceManifestArtifact + { + Kind = payload.Kind, + Uri = BuildCasUri(_storageOptions.ObjectStore.BucketName, key), + Digest = digest, + MediaType = payload.MediaType, + Format = MapFormat(payload.ArtifactFormat), + SizeBytes = payload.Content.Length, + View = payload.View, + Storage = new SurfaceManifestStorage + { + Bucket = _storageOptions.ObjectStore.BucketName, + ObjectKey = key, + SizeBytes = payload.Content.Length, + ContentType = payload.MediaType + }, + Metadata = payload.Metadata + }; + } + + private static string BuildCasUri(string bucket, string key) + { + var normalizedKey = string.IsNullOrWhiteSpace(key) ? string.Empty : key.Trim().TrimStart('/'); + return $"cas://{bucket}/{normalizedKey}"; + } + + private static string MapFormat(ArtifactDocumentFormat format) + => format switch + { + ArtifactDocumentFormat.EntryTraceNdjson => "entrytrace.ndjson", + ArtifactDocumentFormat.EntryTraceGraphJson => "entrytrace.graph", + ArtifactDocumentFormat.ComponentFragmentJson => "layer.fragments", + ArtifactDocumentFormat.SurfaceManifestJson => "surface.manifest", + ArtifactDocumentFormat.CycloneDxJson => "cdx-json", + ArtifactDocumentFormat.CycloneDxProtobuf => "cdx-protobuf", + ArtifactDocumentFormat.SpdxJson => "spdx-json", + ArtifactDocumentFormat.BomIndex => "bom-index", + ArtifactDocumentFormat.DsseJson => "dsse-json", + _ => format.ToString().ToLowerInvariant() + }; + + private static string ComputeDigest(ReadOnlySpan content) + { + Span hash = stackalloc byte[32]; + SHA256.HashData(content, hash); + return $"sha256:{Convert.ToHexString(hash).ToLowerInvariant()}"; + } + + private static string ComputeDigest(byte[] content) + => ComputeDigest(content.AsSpan()); + + private static string NormalizeDigest(string digest) + { + if (string.IsNullOrWhiteSpace(digest)) + { + return string.Empty; + } + + var trimmed = digest.Trim(); + return trimmed.Contains(':', StringComparison.Ordinal) + ? trimmed + : $"sha256:{trimmed}"; + } +} diff --git a/src/Scanner/StellaOps.Scanner.Worker/Processing/Surface/SurfaceManifestStageExecutor.cs b/src/Scanner/StellaOps.Scanner.Worker/Processing/Surface/SurfaceManifestStageExecutor.cs new file mode 100644 index 000000000..d2e18caf0 --- /dev/null +++ b/src/Scanner/StellaOps.Scanner.Worker/Processing/Surface/SurfaceManifestStageExecutor.cs @@ -0,0 +1,147 @@ +using System.Collections.Generic; +using System.Collections.Immutable; +using System.Globalization; +using System.Reflection; +using System.Text; +using System.Text.Json; +using Microsoft.Extensions.Logging; +using StellaOps.Scanner.Core.Contracts; +using StellaOps.Scanner.EntryTrace; +using StellaOps.Scanner.EntryTrace.Serialization; +using StellaOps.Scanner.Surface.FS; + +namespace StellaOps.Scanner.Worker.Processing.Surface; + +internal sealed class SurfaceManifestStageExecutor : IScanStageExecutor +{ + private static readonly JsonSerializerOptions JsonOptions = new(JsonSerializerDefaults.Web) + { + WriteIndented = false, + DefaultIgnoreCondition = JsonIgnoreCondition.WhenWritingNull + }; + + private readonly SurfaceManifestPublisher _publisher; + private readonly ILogger _logger; + private readonly string _componentVersion; + + public SurfaceManifestStageExecutor( + SurfaceManifestPublisher publisher, + ILogger logger) + { + _publisher = publisher ?? throw new ArgumentNullException(nameof(publisher)); + _logger = logger ?? throw new ArgumentNullException(nameof(logger)); + _componentVersion = Assembly.GetExecutingAssembly().GetName().Version?.ToString() ?? "unknown"; + } + + public string StageName => ScanStageNames.ComposeArtifacts; + + public async ValueTask ExecuteAsync(ScanJobContext context, CancellationToken cancellationToken) + { + ArgumentNullException.ThrowIfNull(context); + + var payloads = CollectPayloads(context); + if (payloads.Count == 0) + { + _logger.LogDebug("No surface payloads available for job {JobId}; skipping manifest publish.", context.JobId); + return; + } + + var request = new SurfaceManifestRequest( + ScanId: context.ScanId, + ImageDigest: ResolveImageDigest(context), + Attempt: context.Lease.Attempt, + Metadata: context.Lease.Metadata, + Payloads: payloads, + Component: "scanner.worker", + Version: _componentVersion, + WorkerInstance: Environment.MachineName); + + var result = await _publisher.PublishAsync(request, cancellationToken).ConfigureAwait(false); + context.Analysis.Set(ScanAnalysisKeys.SurfaceManifest, result); + _logger.LogInformation("Surface manifest stored for job {JobId} with digest {Digest}.", context.JobId, result.ManifestDigest); + } + + private List CollectPayloads(ScanJobContext context) + { + var payloads = new List(); + + if (context.Analysis.TryGet(ScanAnalysisKeys.EntryTraceGraph, out var graph) && graph is not null) + { + var graphJson = EntryTraceGraphSerializer.Serialize(graph); + payloads.Add(new SurfaceManifestPayload( + ArtifactDocumentType.SurfaceEntryTrace, + ArtifactDocumentFormat.EntryTraceGraphJson, + Kind: "entrytrace.graph", + MediaType: "application/json", + Content: Encoding.UTF8.GetBytes(graphJson), + Metadata: new Dictionary + { + ["artifact"] = "entrytrace.graph", + ["nodes"] = graph.Nodes.Length.ToString(CultureInfoInvariant), + ["edges"] = graph.Edges.Length.ToString(CultureInfoInvariant) + })); + } + + if (context.Analysis.TryGet(ScanAnalysisKeys.EntryTraceNdjson, out ImmutableArray ndjson) && !ndjson.IsDefaultOrEmpty) + { + var builder = new StringBuilder(); + for (var i = 0; i < ndjson.Length; i++) + { + builder.Append(ndjson[i]); + if (!ndjson[i].EndsWith('\n')) + { + builder.Append('\n'); + } + } + + payloads.Add(new SurfaceManifestPayload( + ArtifactDocumentType.SurfaceEntryTrace, + ArtifactDocumentFormat.EntryTraceNdjson, + Kind: "entrytrace.ndjson", + MediaType: "application/x-ndjson", + Content: Encoding.UTF8.GetBytes(builder.ToString()))); + } + + var fragments = context.Analysis.GetLayerFragments(); + if (!fragments.IsDefaultOrEmpty && fragments.Length > 0) + { + var fragmentsJson = JsonSerializer.Serialize(fragments, JsonOptions); + payloads.Add(new SurfaceManifestPayload( + ArtifactDocumentType.SurfaceLayerFragment, + ArtifactDocumentFormat.ComponentFragmentJson, + Kind: "layer.fragments", + MediaType: "application/json", + Content: Encoding.UTF8.GetBytes(fragmentsJson), + View: "inventory")); + } + + return payloads; + } + + private static string ResolveImageDigest(ScanJobContext context) + { + static bool TryGet(IReadOnlyDictionary metadata, string key, out string value) + { + if (metadata.TryGetValue(key, out var found) && !string.IsNullOrWhiteSpace(found)) + { + value = found.Trim(); + return true; + } + + value = string.Empty; + return false; + } + + var metadata = context.Lease.Metadata; + if (TryGet(metadata, "image.digest", out var digest) || + TryGet(metadata, "imageDigest", out digest) || + TryGet(metadata, "scanner.image.digest", out digest)) + { + return digest; + } + + return context.ScanId; + } + + private static readonly IFormatProvider CultureInfoInvariant = System.Globalization.CultureInfo.InvariantCulture; +} diff --git a/src/Scanner/StellaOps.Scanner.Worker/Program.cs b/src/Scanner/StellaOps.Scanner.Worker/Program.cs index be845f993..ad945c71f 100644 --- a/src/Scanner/StellaOps.Scanner.Worker/Program.cs +++ b/src/Scanner/StellaOps.Scanner.Worker/Program.cs @@ -18,7 +18,9 @@ using StellaOps.Scanner.Worker.Diagnostics; using StellaOps.Scanner.Worker.Hosting; using StellaOps.Scanner.Worker.Options; using StellaOps.Scanner.Worker.Processing; +using StellaOps.Scanner.Worker.Processing.Surface; using StellaOps.Scanner.Storage.Extensions; +using StellaOps.Scanner.Storage; var builder = Host.CreateApplicationBuilder(args); @@ -52,6 +54,9 @@ var connectionString = storageSection.GetValue("Mongo:ConnectionString") if (!string.IsNullOrWhiteSpace(connectionString)) { builder.Services.AddScannerStorage(storageSection); + builder.Services.AddSingleton, ScannerStorageSurfaceSecretConfigurator>(); + builder.Services.AddSingleton(); + builder.Services.AddSingleton(); } builder.Services.TryAddSingleton(); diff --git a/src/Scanner/StellaOps.Scanner.Worker/TASKS.md b/src/Scanner/StellaOps.Scanner.Worker/TASKS.md index 9c179643a..40f5b3e38 100644 --- a/src/Scanner/StellaOps.Scanner.Worker/TASKS.md +++ b/src/Scanner/StellaOps.Scanner.Worker/TASKS.md @@ -3,7 +3,7 @@ | ID | Status | Owner(s) | Depends on | Description | Exit Criteria | |----|--------|----------|------------|-------------|---------------| | SCAN-REPLAY-186-002 | TODO | Scanner Worker Guild | REPLAY-CORE-185-001 | Enforce deterministic analyzer execution when consuming replay input bundles, emit layer Merkle metadata, and author `docs/modules/scanner/deterministic-execution.md` summarising invariants from `docs/replay/DETERMINISTIC_REPLAY.md` Section 4. | Replay mode analyzers pass determinism tests; new doc merged; integration fixtures updated. | -| SCANNER-SURFACE-01 | DOING (2025-11-02) | Scanner Worker Guild | SURFACE-FS-02 | Persist Surface.FS manifests after analyzer stages, including layer CAS metadata and EntryTrace fragments.
2025-11-02: Draft Surface.FS manifests emitted for sample scans; telemetry counters under review. | Integration tests prove cache entries exist; telemetry counters exported. | +| SCANNER-SURFACE-01 | DOING (2025-11-06) | Scanner Worker Guild | SURFACE-FS-02 | Persist Surface.FS manifests after analyzer stages, including layer CAS metadata and EntryTrace fragments.
2025-11-02: Draft Surface.FS manifests emitted for sample scans; telemetry counters under review.
2025-11-06: Resuming with manifest writer abstraction, rotation metadata, and telemetry counters for Surface.FS persistence. | Integration tests prove cache entries exist; telemetry counters exported. | | SCANNER-ENV-01 | TODO (2025-11-06) | Scanner Worker Guild | SURFACE-ENV-02 | Replace ad-hoc environment reads with `StellaOps.Scanner.Surface.Env` helpers for cache roots and CAS endpoints.
2025-11-02: Worker bootstrap now resolves cache roots via helper; warning path documented; smoke tests running.
2025-11-05 14:55Z: Extending helper usage into cache/secrets configuration, updating worker validator wiring, and drafting docs/tests for new Surface.Env outputs.
2025-11-06 17:05Z: README/design docs updated with warning catalogue; startup logging guidance captured for ops runbooks.
2025-11-06 07:45Z: Helm/Compose env profiles (dev/stage/prod/airgap/mirror) now seed `SCANNER_SURFACE_*` defaults to keep worker cache roots aligned with Surface.Env helpers.
2025-11-06 07:55Z: Paused; pending automation tracked via `DEVOPS-OPENSSL-11-001/002` and Surface.Env test fixtures. | Worker boots with helper; misconfiguration warnings documented; smoke tests updated. | > 2025-11-05 19:18Z: Bound `SurfaceCacheOptions` root directory to resolved Surface.Env settings and added unit coverage around the configurator. -| SCANNER-SECRETS-01 | DOING (2025-11-02) | Scanner Worker Guild, Security Guild | SURFACE-SECRETS-02 | Adopt `StellaOps.Scanner.Surface.Secrets` for registry/CAS credentials during scan execution.
2025-11-02: Surface.Secrets provider wired for CAS token retrieval; integration tests added. | Secrets fetched via shared provider; legacy secret code removed; integration tests cover rotation. | +| SCANNER-SECRETS-01 | DOING (2025-11-06) | Scanner Worker Guild, Security Guild | SURFACE-SECRETS-02 | Adopt `StellaOps.Scanner.Surface.Secrets` for registry/CAS credentials during scan execution.
2025-11-02: Surface.Secrets provider wired for CAS token retrieval; integration tests added.
2025-11-06: Continuing to replace legacy registry credential plumbing and extend rotation metrics/fixtures.
2025-11-06 21:35Z: Introduced `ScannerStorageSurfaceSecretConfigurator` mapping `cas-access` secrets into storage options plus unit coverage. | Secrets fetched via shared provider; legacy secret code removed; integration tests cover rotation. | diff --git a/src/Scanner/__Libraries/StellaOps.Scanner.Analyzers.Lang/Core/Internal/LanguageAnalyzerSurfaceCache.cs b/src/Scanner/__Libraries/StellaOps.Scanner.Analyzers.Lang/Core/Internal/LanguageAnalyzerSurfaceCache.cs index 888e1bc62..12470fcb9 100644 --- a/src/Scanner/__Libraries/StellaOps.Scanner.Analyzers.Lang/Core/Internal/LanguageAnalyzerSurfaceCache.cs +++ b/src/Scanner/__Libraries/StellaOps.Scanner.Analyzers.Lang/Core/Internal/LanguageAnalyzerSurfaceCache.cs @@ -4,6 +4,8 @@ using System.Text; using Microsoft.Extensions.Logging; using StellaOps.Scanner.Surface.FS; +public readonly record struct LanguageAnalyzerSurfaceCacheEntry(LanguageAnalyzerResult Result, bool IsHit); + public sealed class LanguageAnalyzerSurfaceCache { private const string CacheNamespace = "scanner/lang/analyzers"; @@ -24,6 +26,17 @@ public sealed class LanguageAnalyzerSurfaceCache string fingerprint, Func> factory, CancellationToken cancellationToken) + { + var entry = await GetOrCreateEntryAsync(logger, analyzerId, fingerprint, factory, cancellationToken).ConfigureAwait(false); + return entry.Result; + } + + public async ValueTask GetOrCreateEntryAsync( + ILogger logger, + string analyzerId, + string fingerprint, + Func> factory, + CancellationToken cancellationToken) { ArgumentNullException.ThrowIfNull(logger); ArgumentNullException.ThrowIfNull(factory); @@ -62,7 +75,7 @@ public sealed class LanguageAnalyzerSurfaceCache fingerprint); result = await factory(cancellationToken).ConfigureAwait(false); - return result; + return new LanguageAnalyzerSurfaceCacheEntry(result, false); } if (cacheHit) @@ -82,7 +95,7 @@ public sealed class LanguageAnalyzerSurfaceCache fingerprint); } - return result; + return new LanguageAnalyzerSurfaceCacheEntry(result, cacheHit); } private static ReadOnlyMemory Serialize(LanguageAnalyzerResult result) diff --git a/src/Scanner/__Libraries/StellaOps.Scanner.Core/Contracts/ScanAnalysisKeys.cs b/src/Scanner/__Libraries/StellaOps.Scanner.Core/Contracts/ScanAnalysisKeys.cs index 5784818c2..74eb9ee69 100644 --- a/src/Scanner/__Libraries/StellaOps.Scanner.Core/Contracts/ScanAnalysisKeys.cs +++ b/src/Scanner/__Libraries/StellaOps.Scanner.Core/Contracts/ScanAnalysisKeys.cs @@ -15,4 +15,6 @@ public static class ScanAnalysisKeys public const string EntryTraceGraph = "analysis.entrytrace.graph"; public const string EntryTraceNdjson = "analysis.entrytrace.ndjson"; + + public const string SurfaceManifest = "analysis.surface.manifest"; } diff --git a/src/Scanner/__Libraries/StellaOps.Scanner.Storage/Catalog/ArtifactDocument.cs b/src/Scanner/__Libraries/StellaOps.Scanner.Storage/Catalog/ArtifactDocument.cs index cbe44e3dc..2f05c7989 100644 --- a/src/Scanner/__Libraries/StellaOps.Scanner.Storage/Catalog/ArtifactDocument.cs +++ b/src/Scanner/__Libraries/StellaOps.Scanner.Storage/Catalog/ArtifactDocument.cs @@ -2,23 +2,30 @@ using MongoDB.Bson.Serialization.Attributes; namespace StellaOps.Scanner.Storage.Catalog; -public enum ArtifactDocumentType -{ - LayerBom, - ImageBom, - Diff, - Index, - Attestation, -} - -public enum ArtifactDocumentFormat -{ - CycloneDxJson, - CycloneDxProtobuf, - SpdxJson, - BomIndex, - DsseJson, -} +public enum ArtifactDocumentType +{ + LayerBom, + ImageBom, + Diff, + Index, + Attestation, + SurfaceManifest, + SurfaceEntryTrace, + SurfaceLayerFragment, +} + +public enum ArtifactDocumentFormat +{ + CycloneDxJson, + CycloneDxProtobuf, + SpdxJson, + BomIndex, + DsseJson, + SurfaceManifestJson, + EntryTraceNdjson, + EntryTraceGraphJson, + ComponentFragmentJson, +} [BsonIgnoreExtraElements] public sealed class ArtifactDocument diff --git a/src/Scanner/__Libraries/StellaOps.Scanner.Storage/Extensions/ServiceCollectionExtensions.cs b/src/Scanner/__Libraries/StellaOps.Scanner.Storage/Extensions/ServiceCollectionExtensions.cs index 211cfb1a1..95a9e9d9f 100644 --- a/src/Scanner/__Libraries/StellaOps.Scanner.Storage/Extensions/ServiceCollectionExtensions.cs +++ b/src/Scanner/__Libraries/StellaOps.Scanner.Storage/Extensions/ServiceCollectionExtensions.cs @@ -2,6 +2,7 @@ using System; using System.Net.Http; using Amazon; using Amazon.S3; +using Amazon.Runtime; using Microsoft.Extensions.Configuration; using Microsoft.Extensions.DependencyInjection; using Microsoft.Extensions.DependencyInjection.Extensions; @@ -150,14 +151,22 @@ public static class ServiceCollectionExtensions var options = provider.GetRequiredService>().Value.ObjectStore; var config = new AmazonS3Config { - RegionEndpoint = RegionEndpoint.GetBySystemName(options.Region), - ForcePathStyle = options.ForcePathStyle, - }; - - if (!string.IsNullOrWhiteSpace(options.ServiceUrl)) - { - config.ServiceURL = options.ServiceUrl; - } + RegionEndpoint = RegionEndpoint.GetBySystemName(options.Region), + ForcePathStyle = options.ForcePathStyle, + }; + + if (!string.IsNullOrWhiteSpace(options.ServiceUrl)) + { + config.ServiceURL = options.ServiceUrl; + } + + if (!string.IsNullOrWhiteSpace(options.AccessKeyId) && !string.IsNullOrWhiteSpace(options.SecretAccessKey)) + { + AWSCredentials credentials = string.IsNullOrWhiteSpace(options.SessionToken) + ? new BasicAWSCredentials(options.AccessKeyId, options.SecretAccessKey) + : new SessionAWSCredentials(options.AccessKeyId, options.SecretAccessKey, options.SessionToken); + return new AmazonS3Client(credentials, config); + } return new AmazonS3Client(config); } diff --git a/src/Scanner/__Libraries/StellaOps.Scanner.Storage/ObjectStore/ArtifactObjectKeyBuilder.cs b/src/Scanner/__Libraries/StellaOps.Scanner.Storage/ObjectStore/ArtifactObjectKeyBuilder.cs index 7d88ecead..a1882535b 100644 --- a/src/Scanner/__Libraries/StellaOps.Scanner.Storage/ObjectStore/ArtifactObjectKeyBuilder.cs +++ b/src/Scanner/__Libraries/StellaOps.Scanner.Storage/ObjectStore/ArtifactObjectKeyBuilder.cs @@ -33,6 +33,9 @@ public static class ArtifactObjectKeyBuilder ArtifactDocumentType.ImageBom => ScannerStorageDefaults.ObjectPrefixes.Images, ArtifactDocumentType.Index => ScannerStorageDefaults.ObjectPrefixes.Indexes, ArtifactDocumentType.Attestation => ScannerStorageDefaults.ObjectPrefixes.Attestations, + ArtifactDocumentType.SurfaceManifest => ScannerStorageDefaults.ObjectPrefixes.SurfaceManifests, + ArtifactDocumentType.SurfaceEntryTrace => ScannerStorageDefaults.ObjectPrefixes.SurfaceEntryTrace, + ArtifactDocumentType.SurfaceLayerFragment => ScannerStorageDefaults.ObjectPrefixes.SurfaceLayerFragments, ArtifactDocumentType.Diff => "diffs", _ => ScannerStorageDefaults.ObjectPrefixes.Images, }; @@ -44,6 +47,10 @@ public static class ArtifactObjectKeyBuilder ArtifactDocumentFormat.SpdxJson => "sbom.spdx.json", ArtifactDocumentFormat.BomIndex => "bom-index.bin", ArtifactDocumentFormat.DsseJson => "artifact.dsse.json", + ArtifactDocumentFormat.SurfaceManifestJson => "surface.manifest.json", + ArtifactDocumentFormat.EntryTraceNdjson => "entrytrace.ndjson", + ArtifactDocumentFormat.EntryTraceGraphJson => "entrytrace.graph.json", + ArtifactDocumentFormat.ComponentFragmentJson => "layer-fragments.json", _ => "artifact.bin", }; diff --git a/src/Scanner/__Libraries/StellaOps.Scanner.Storage/ScannerStorageDefaults.cs b/src/Scanner/__Libraries/StellaOps.Scanner.Storage/ScannerStorageDefaults.cs index 20af262a8..decac0da5 100644 --- a/src/Scanner/__Libraries/StellaOps.Scanner.Storage/ScannerStorageDefaults.cs +++ b/src/Scanner/__Libraries/StellaOps.Scanner.Storage/ScannerStorageDefaults.cs @@ -26,11 +26,14 @@ public static class ScannerStorageDefaults public const string Migrations = "schema_migrations"; } - public static class ObjectPrefixes - { - public const string Layers = "layers"; - public const string Images = "images"; - public const string Indexes = "indexes"; - public const string Attestations = "attest"; - } -} + public static class ObjectPrefixes + { + public const string Layers = "layers"; + public const string Images = "images"; + public const string Indexes = "indexes"; + public const string Attestations = "attest"; + public const string SurfaceManifests = "surface/manifests"; + public const string SurfaceEntryTrace = "surface/payloads/entrytrace"; + public const string SurfaceLayerFragments = "surface/payloads/layer-fragments"; + } +} diff --git a/src/Scanner/__Libraries/StellaOps.Scanner.Storage/ScannerStorageOptions.cs b/src/Scanner/__Libraries/StellaOps.Scanner.Storage/ScannerStorageOptions.cs index 4ef8c7c21..d983f0fa4 100644 --- a/src/Scanner/__Libraries/StellaOps.Scanner.Storage/ScannerStorageOptions.cs +++ b/src/Scanner/__Libraries/StellaOps.Scanner.Storage/ScannerStorageOptions.cs @@ -102,6 +102,15 @@ public sealed class ObjectStoreOptions public TimeSpan? ComplianceRetention { get; set; } = TimeSpan.FromDays(90); + public string? AccessKeyId { get; set; } + = null; + + public string? SecretAccessKey { get; set; } + = null; + + public string? SessionToken { get; set; } + = null; + public IDictionary Headers { get; } = new Dictionary(StringComparer.OrdinalIgnoreCase); public RustFsOptions RustFs { get; set; } = new(); diff --git a/src/Scanner/__Libraries/StellaOps.Scanner.Surface.FS/SurfaceManifestModels.cs b/src/Scanner/__Libraries/StellaOps.Scanner.Surface.FS/SurfaceManifestModels.cs new file mode 100644 index 000000000..1758f9f84 --- /dev/null +++ b/src/Scanner/__Libraries/StellaOps.Scanner.Surface.FS/SurfaceManifestModels.cs @@ -0,0 +1,142 @@ +using System; +using System.Collections.Generic; +using System.Collections.Immutable; +using System.Text.Json.Serialization; + +namespace StellaOps.Scanner.Surface.FS; + +/// +/// Canonical manifest describing surface artefacts produced for a scan. +/// +public sealed record SurfaceManifestDocument +{ + public const string DefaultSchema = "stellaops.surface.manifest@1"; + + [JsonPropertyName("schema")] + public string Schema { get; init; } = DefaultSchema; + + [JsonPropertyName("tenant")] + public string Tenant { get; init; } = string.Empty; + + [JsonPropertyName("imageDigest")] + [JsonIgnore(Condition = JsonIgnoreCondition.WhenWritingNull)] + public string? ImageDigest { get; init; } + = null; + + [JsonPropertyName("scanId")] + [JsonIgnore(Condition = JsonIgnoreCondition.WhenWritingNull)] + public string? ScanId { get; init; } + = null; + + [JsonPropertyName("generatedAt")] + public DateTimeOffset GeneratedAt { get; init; } + = DateTimeOffset.UtcNow; + + [JsonPropertyName("source")] + [JsonIgnore(Condition = JsonIgnoreCondition.WhenWritingNull)] + public SurfaceManifestSource? Source { get; init; } + = null; + + [JsonPropertyName("artifacts")] + public IReadOnlyList Artifacts { get; init; } + = ImmutableArray.Empty; +} + +/// +/// Identifies the producer of the manifest. +/// +public sealed record SurfaceManifestSource +{ + [JsonPropertyName("component")] + public string Component { get; init; } = "scanner.worker"; + + [JsonPropertyName("version")] + [JsonIgnore(Condition = JsonIgnoreCondition.WhenWritingNull)] + public string? Version { get; init; } + = null; + + [JsonPropertyName("workerInstance")] + [JsonIgnore(Condition = JsonIgnoreCondition.WhenWritingNull)] + public string? WorkerInstance { get; init; } + = null; + + [JsonPropertyName("attempt")] + [JsonIgnore(Condition = JsonIgnoreCondition.WhenWritingNull)] + public int? Attempt { get; init; } + = null; +} + +/// +/// Describes a surface artefact referenced by the manifest. +/// +public sealed record SurfaceManifestArtifact +{ + [JsonPropertyName("kind")] + public string Kind { get; init; } = string.Empty; + + [JsonPropertyName("uri")] + public string Uri { get; init; } = string.Empty; + + [JsonPropertyName("digest")] + public string Digest { get; init; } = string.Empty; + + [JsonPropertyName("mediaType")] + public string MediaType { get; init; } = string.Empty; + + [JsonPropertyName("format")] + public string Format { get; init; } = string.Empty; + + [JsonPropertyName("sizeBytes")] + public long SizeBytes { get; init; } + = 0; + + [JsonPropertyName("view")] + [JsonIgnore(Condition = JsonIgnoreCondition.WhenWritingNull)] + public string? View { get; init; } + = null; + + [JsonPropertyName("storage")] + [JsonIgnore(Condition = JsonIgnoreCondition.WhenWritingNull)] + public SurfaceManifestStorage? Storage { get; init; } + = null; + + [JsonPropertyName("metadata")] + [JsonIgnore(Condition = JsonIgnoreCondition.WhenWritingNull)] + public IReadOnlyDictionary? Metadata { get; init; } + = null; +} + +/// +/// Storage descriptor for an artefact. +/// +public sealed record SurfaceManifestStorage +{ + [JsonPropertyName("bucket")] + [JsonIgnore(Condition = JsonIgnoreCondition.WhenWritingNull)] + public string? Bucket { get; init; } + = null; + + [JsonPropertyName("objectKey")] + [JsonIgnore(Condition = JsonIgnoreCondition.WhenWritingNull)] + public string? ObjectKey { get; init; } + = null; + + [JsonPropertyName("sizeBytes")] + [JsonIgnore(Condition = JsonIgnoreCondition.WhenWritingNull)] + public long? SizeBytes { get; init; } + = null; + + [JsonPropertyName("contentType")] + [JsonIgnore(Condition = JsonIgnoreCondition.WhenWritingNull)] + public string? ContentType { get; init; } + = null; +} + +/// +/// Result from publishing a surface manifest. +/// +public sealed record SurfaceManifestPublishResult( + string ManifestDigest, + string ManifestUri, + string ArtifactId, + SurfaceManifestDocument Document); diff --git a/src/Scanner/__Libraries/StellaOps.Scanner.Surface.Secrets/CasAccessSecret.cs b/src/Scanner/__Libraries/StellaOps.Scanner.Surface.Secrets/CasAccessSecret.cs new file mode 100644 index 000000000..ee1b9ad8f --- /dev/null +++ b/src/Scanner/__Libraries/StellaOps.Scanner.Surface.Secrets/CasAccessSecret.cs @@ -0,0 +1,207 @@ +using System.Collections.Generic; +using System.Collections.ObjectModel; +using System.Text; +using System.Text.Json; + +namespace StellaOps.Scanner.Surface.Secrets; + +public sealed record CasAccessSecret( + string Driver, + string? Endpoint, + string? Region, + string? Bucket, + string? RootPrefix, + string? ApiKey, + string? ApiKeyHeader, + IReadOnlyDictionary Headers, + string? AccessKeyId, + string? SecretAccessKey, + string? SessionToken, + bool? AllowInsecureTls); + +public static class SurfaceSecretParser +{ + public static CasAccessSecret ParseCasAccessSecret(SurfaceSecretHandle handle) + { + ArgumentNullException.ThrowIfNull(handle); + var payload = handle.AsBytes(); + if (payload.IsEmpty) + { + throw new InvalidOperationException("Surface secret payload is empty."); + } + + var jsonText = DecodeUtf8(payload); + using var document = JsonDocument.Parse(jsonText); + var root = document.RootElement; + + string driver = GetString(root, "driver") ?? GetMetadataValue(handle.Metadata, "driver") ?? "s3"; + string? endpoint = GetString(root, "endpoint") ?? GetMetadataValue(handle.Metadata, "endpoint"); + string? region = GetString(root, "region") ?? GetMetadataValue(handle.Metadata, "region"); + string? bucket = GetString(root, "bucket") ?? GetMetadataValue(handle.Metadata, "bucket"); + string? rootPrefix = GetString(root, "rootPrefix") ?? GetMetadataValue(handle.Metadata, "rootPrefix"); + string? apiKey = GetString(root, "apiKey") ?? GetMetadataValue(handle.Metadata, "apiKey"); + string? apiKeyHeader = GetString(root, "apiKeyHeader") ?? GetMetadataValue(handle.Metadata, "apiKeyHeader"); + string? accessKeyId = GetString(root, "accessKeyId") ?? GetMetadataValue(handle.Metadata, "accessKeyId"); + string? secretAccessKey = GetString(root, "secretAccessKey") ?? GetMetadataValue(handle.Metadata, "secretAccessKey"); + string? sessionToken = GetString(root, "sessionToken") ?? GetMetadataValue(handle.Metadata, "sessionToken"); + bool? allowInsecureTls = GetBoolean(root, "allowInsecureTls"); + + var headers = new Dictionary(StringComparer.OrdinalIgnoreCase); + PopulateHeaders(root, headers); + PopulateMetadataHeaders(handle.Metadata, headers); + + return new CasAccessSecret( + driver.Trim(), + endpoint?.Trim(), + region?.Trim(), + bucket?.Trim(), + rootPrefix?.Trim(), + apiKey?.Trim(), + apiKeyHeader?.Trim(), + new ReadOnlyDictionary(headers), + accessKeyId?.Trim(), + secretAccessKey?.Trim(), + sessionToken?.Trim(), + allowInsecureTls); + } + + private static string DecodeUtf8(ReadOnlyMemory payload) + { + if (payload.IsEmpty) + { + return string.Empty; + } + + try + { + return Encoding.UTF8.GetString(payload.Span); + } + catch (DecoderFallbackException ex) + { + throw new InvalidOperationException("Surface secret payload is not valid UTF-8 JSON.", ex); + } + } + + private static string? GetString(JsonElement element, string propertyName) + { + if (TryGetPropertyIgnoreCase(element, propertyName, out var property)) + { + return property.ValueKind switch + { + JsonValueKind.String => property.GetString(), + JsonValueKind.Number => property.GetRawText(), + JsonValueKind.True => bool.TrueString, + JsonValueKind.False => bool.FalseString, + _ => null + }; + } + + return null; + } + + private static bool? GetBoolean(JsonElement element, string propertyName) + { + if (TryGetPropertyIgnoreCase(element, propertyName, out var property)) + { + return property.ValueKind switch + { + JsonValueKind.True => true, + JsonValueKind.False => false, + JsonValueKind.String when bool.TryParse(property.GetString(), out var parsed) => parsed, + _ => null + }; + } + + return null; + } + + private static bool TryGetPropertyIgnoreCase(JsonElement element, string propertyName, out JsonElement property) + { + if (element.ValueKind != JsonValueKind.Object) + { + property = default; + return false; + } + + if (element.TryGetProperty(propertyName, out property)) + { + return true; + } + + foreach (var candidate in element.EnumerateObject()) + { + if (string.Equals(candidate.Name, propertyName, StringComparison.OrdinalIgnoreCase)) + { + property = candidate.Value; + return true; + } + } + + property = default; + return false; + } + + private static void PopulateHeaders(JsonElement element, IDictionary headers) + { + if (!TryGetPropertyIgnoreCase(element, "headers", out var headersElement)) + { + return; + } + + if (headersElement.ValueKind != JsonValueKind.Object) + { + return; + } + + foreach (var property in headersElement.EnumerateObject()) + { + var value = property.Value.ValueKind switch + { + JsonValueKind.String => property.Value.GetString(), + JsonValueKind.Number => property.Value.GetRawText(), + JsonValueKind.True => bool.TrueString, + JsonValueKind.False => bool.FalseString, + _ => null + }; + + if (string.IsNullOrWhiteSpace(value)) + { + continue; + } + + headers[property.Name] = value.Trim(); + } + } + + private static void PopulateMetadataHeaders(IReadOnlyDictionary metadata, IDictionary headers) + { + foreach (var (key, value) in metadata) + { + if (!key.StartsWith("header:", StringComparison.OrdinalIgnoreCase)) + { + continue; + } + + var headerName = key["header:".Length..]; + if (string.IsNullOrWhiteSpace(headerName) || string.IsNullOrWhiteSpace(value)) + { + continue; + } + + headers[headerName] = value; + } + } + + private static string? GetMetadataValue(IReadOnlyDictionary metadata, string key) + { + foreach (var (metadataKey, metadataValue) in metadata) + { + if (string.Equals(metadataKey, key, StringComparison.OrdinalIgnoreCase)) + { + return metadataValue; + } + } + + return null; + } +} diff --git a/src/Scanner/__Tests/StellaOps.Scanner.Surface.Secrets.Tests/CasAccessSecretParserTests.cs b/src/Scanner/__Tests/StellaOps.Scanner.Surface.Secrets.Tests/CasAccessSecretParserTests.cs new file mode 100644 index 000000000..f7afa336b --- /dev/null +++ b/src/Scanner/__Tests/StellaOps.Scanner.Surface.Secrets.Tests/CasAccessSecretParserTests.cs @@ -0,0 +1,65 @@ +using System.Collections.Generic; +using System.Text; +using StellaOps.Scanner.Surface.Secrets; +using Xunit; + +namespace StellaOps.Scanner.Surface.Secrets.Tests; + +public sealed class CasAccessSecretParserTests +{ + [Fact] + public void ParseCasAccessSecret_WithRustFsPayload_ReturnsExpectedValues() + { + const string json = """ + { + "driver": "rustfs", + "endpoint": "https://surface.test.local", + "region": "us-gov-west-1", + "bucket": "stellaops-surface", + "rootPrefix": "scanner", + "apiKey": "secret-api-key", + "apiKeyHeader": "X-Api-Key", + "allowInsecureTls": true, + "headers": { + "X-Surface-Tenant": "tenant-a" + } + } + """; + + using var handle = SurfaceSecretHandle.FromBytes(Encoding.UTF8.GetBytes(json)); + var secret = SurfaceSecretParser.ParseCasAccessSecret(handle); + + Assert.Equal("rustfs", secret.Driver); + Assert.Equal("https://surface.test.local", secret.Endpoint); + Assert.Equal("us-gov-west-1", secret.Region); + Assert.Equal("stellaops-surface", secret.Bucket); + Assert.Equal("scanner", secret.RootPrefix); + Assert.Equal("secret-api-key", secret.ApiKey); + Assert.Equal("X-Api-Key", secret.ApiKeyHeader); + Assert.True(secret.AllowInsecureTls); + Assert.Single(secret.Headers); + Assert.Equal("tenant-a", secret.Headers["X-Surface-Tenant"]); + } + + [Fact] + public void ParseCasAccessSecret_UsesMetadataFallback_WhenFieldsMissing() + { + const string json = @"{ ""driver"": ""s3"" }"; + var metadata = new Dictionary(StringComparer.OrdinalIgnoreCase) + { + ["endpoint"] = "https://s3.test.local", + ["accessKeyId"] = "AKIA123", + ["secretAccessKey"] = "s3-secret", + ["header:X-Custom"] = "value" + }; + + using var handle = SurfaceSecretHandle.FromBytes(Encoding.UTF8.GetBytes(json), metadata); + var secret = SurfaceSecretParser.ParseCasAccessSecret(handle); + + Assert.Equal("s3", secret.Driver); + Assert.Equal("https://s3.test.local", secret.Endpoint); + Assert.Equal("AKIA123", secret.AccessKeyId); + Assert.Equal("s3-secret", secret.SecretAccessKey); + Assert.Equal("value", secret.Headers["X-Custom"]); + } +} diff --git a/src/Scanner/__Tests/StellaOps.Scanner.WebService.Tests/ReportEventDispatcherTests.cs b/src/Scanner/__Tests/StellaOps.Scanner.WebService.Tests/ReportEventDispatcherTests.cs index aed11d853..6feb04658 100644 --- a/src/Scanner/__Tests/StellaOps.Scanner.WebService.Tests/ReportEventDispatcherTests.cs +++ b/src/Scanner/__Tests/StellaOps.Scanner.WebService.Tests/ReportEventDispatcherTests.cs @@ -1,4 +1,4 @@ -using System; +using System; using System.Collections.Generic; using System.Collections.Immutable; using System.Linq; @@ -16,108 +16,108 @@ using StellaOps.Policy; using StellaOps.Scanner.WebService.Contracts; using StellaOps.Scanner.WebService.Options; using StellaOps.Scanner.WebService.Services; - -namespace StellaOps.Scanner.WebService.Tests; - -public sealed class ReportEventDispatcherTests -{ - private static readonly JsonSerializerOptions SerializerOptions = new(JsonSerializerDefaults.Web) - { - DefaultIgnoreCondition = JsonIgnoreCondition.WhenWritingNull - }; - - [Fact] - public async Task PublishAsync_EmitsReportReadyAndScanCompleted() - { - var publisher = new RecordingEventPublisher(); + +namespace StellaOps.Scanner.WebService.Tests; + +public sealed class ReportEventDispatcherTests +{ + private static readonly JsonSerializerOptions SerializerOptions = new(JsonSerializerDefaults.Web) + { + DefaultIgnoreCondition = JsonIgnoreCondition.WhenWritingNull + }; + + [Fact] + public async Task PublishAsync_EmitsReportReadyAndScanCompleted() + { + var publisher = new RecordingEventPublisher(); var dispatcher = new ReportEventDispatcher(publisher, Microsoft.Extensions.Options.Options.Create(new ScannerWebServiceOptions()), TimeProvider.System, NullLogger.Instance); - var cancellationToken = CancellationToken.None; - - var request = new ReportRequestDto - { - ImageDigest = "sha256:feedface", - Findings = new[] - { - new PolicyPreviewFindingDto - { - Id = "finding-1", - Severity = "Critical", - Repository = "acme/edge/api", - Cve = "CVE-2024-9999", - Tags = new[] { "reachability:runtime", "kev:CVE-2024-9999" } - } - } - }; - - var baseline = new PolicyVerdict("finding-1", PolicyVerdictStatus.Pass, ConfigVersion: "1.0"); - var projected = new PolicyVerdict( - "finding-1", - PolicyVerdictStatus.Blocked, - Score: 47.5, - ConfigVersion: "1.0", - SourceTrust: "NVD", - Reachability: "runtime"); - - var preview = new PolicyPreviewResponse( - Success: true, - PolicyDigest: "digest-123", - RevisionId: "rev-42", - Issues: ImmutableArray.Empty, - Diffs: ImmutableArray.Create(new PolicyVerdictDiff(baseline, projected)), - ChangedCount: 1); - - var document = new ReportDocumentDto - { - ReportId = "report-abc", - ImageDigest = "sha256:feedface", - GeneratedAt = DateTimeOffset.Parse("2025-10-19T12:34:56Z"), - Verdict = "blocked", - Policy = new ReportPolicyDto - { - RevisionId = "rev-42", - Digest = "digest-123" - }, - Summary = new ReportSummaryDto - { - Total = 1, - Blocked = 1, - Warned = 0, - Ignored = 0, - Quieted = 0 - }, - Verdicts = new[] - { - new PolicyPreviewVerdictDto - { - FindingId = "finding-1", - Status = "Blocked", - Score = 47.5, - SourceTrust = "NVD", - Reachability = "runtime" - } - } - }; - - var envelope = new DsseEnvelopeDto - { - PayloadType = "application/vnd.stellaops.report+json", - Payload = Convert.ToBase64String(JsonSerializer.SerializeToUtf8Bytes(document, SerializerOptions)), - Signatures = new[] - { - new DsseSignatureDto { KeyId = "test-key", Algorithm = "hs256", Signature = "signature-value" } - } - }; - - var context = new DefaultHttpContext(); - context.User = new ClaimsPrincipal(new ClaimsIdentity(new[] - { - new Claim(StellaOpsClaimTypes.Tenant, "tenant-alpha") - })); - context.Request.Scheme = "https"; - context.Request.Host = new HostString("scanner.example"); - - await dispatcher.PublishAsync(request, preview, document, envelope, context, cancellationToken); - + var cancellationToken = CancellationToken.None; + + var request = new ReportRequestDto + { + ImageDigest = "sha256:feedface", + Findings = new[] + { + new PolicyPreviewFindingDto + { + Id = "finding-1", + Severity = "Critical", + Repository = "acme/edge/api", + Cve = "CVE-2024-9999", + Tags = new[] { "reachability:runtime", "kev:CVE-2024-9999" } + } + } + }; + + var baseline = new PolicyVerdict("finding-1", PolicyVerdictStatus.Pass, ConfigVersion: "1.0"); + var projected = new PolicyVerdict( + "finding-1", + PolicyVerdictStatus.Blocked, + Score: 47.5, + ConfigVersion: "1.0", + SourceTrust: "NVD", + Reachability: "runtime"); + + var preview = new PolicyPreviewResponse( + Success: true, + PolicyDigest: "digest-123", + RevisionId: "rev-42", + Issues: ImmutableArray.Empty, + Diffs: ImmutableArray.Create(new PolicyVerdictDiff(baseline, projected)), + ChangedCount: 1); + + var document = new ReportDocumentDto + { + ReportId = "report-abc", + ImageDigest = "sha256:feedface", + GeneratedAt = DateTimeOffset.Parse("2025-10-19T12:34:56Z"), + Verdict = "blocked", + Policy = new ReportPolicyDto + { + RevisionId = "rev-42", + Digest = "digest-123" + }, + Summary = new ReportSummaryDto + { + Total = 1, + Blocked = 1, + Warned = 0, + Ignored = 0, + Quieted = 0 + }, + Verdicts = new[] + { + new PolicyPreviewVerdictDto + { + FindingId = "finding-1", + Status = "Blocked", + Score = 47.5, + SourceTrust = "NVD", + Reachability = "runtime" + } + } + }; + + var envelope = new DsseEnvelopeDto + { + PayloadType = "application/vnd.stellaops.report+json", + Payload = Convert.ToBase64String(JsonSerializer.SerializeToUtf8Bytes(document, SerializerOptions)), + Signatures = new[] + { + new DsseSignatureDto { KeyId = "test-key", Algorithm = "hs256", Signature = "signature-value" } + } + }; + + var context = new DefaultHttpContext(); + context.User = new ClaimsPrincipal(new ClaimsIdentity(new[] + { + new Claim(StellaOpsClaimTypes.Tenant, "tenant-alpha") + })); + context.Request.Scheme = "https"; + context.Request.Host = new HostString("scanner.example"); + + await dispatcher.PublishAsync(request, preview, document, envelope, context, cancellationToken); + Assert.Equal(2, publisher.Events.Count); var readyEvent = Assert.Single(publisher.Events, evt => evt.Kind == OrchestratorEventKinds.ScannerReportReady); @@ -165,6 +165,126 @@ public sealed class ReportEventDispatcherTests Assert.Equal("blocked", scanPayload.Report.Verdict); } + [Fact] + public async Task PublishAsync_HonoursConfiguredConsoleAndApiSegments() + { + var options = Microsoft.Extensions.Options.Options.Create(new ScannerWebServiceOptions + { + Api = new ScannerWebServiceOptions.ApiOptions + { + BasePath = "/custom-api", + ReportsSegment = "reports-view", + PolicySegment = "policy-hub" + }, + Console = new ScannerWebServiceOptions.ConsoleOptions + { + BasePath = "/console", + ReportsSegment = "insights", + PolicySegment = "policy-center", + AttestationsSegment = "evidence" + } + }); + + var publisher = new RecordingEventPublisher(); + var dispatcher = new ReportEventDispatcher(publisher, options, TimeProvider.System, NullLogger.Instance); + var cancellationToken = CancellationToken.None; + + var request = new ReportRequestDto + { + ImageDigest = "sha256:feedface", + Findings = new[] + { + new PolicyPreviewFindingDto + { + Id = "finding-1", + Severity = "Critical", + Repository = "acme/edge/api", + Cve = "CVE-2024-9999", + Tags = new[] { "reachability:runtime", "kev:CVE-2024-9999" } + } + } + }; + + var baseline = new PolicyVerdict("finding-1", PolicyVerdictStatus.Pass, ConfigVersion: "1.0"); + var projected = new PolicyVerdict( + "finding-1", + PolicyVerdictStatus.Blocked, + Score: 47.5, + ConfigVersion: "1.0", + SourceTrust: "NVD", + Reachability: "runtime"); + + var preview = new PolicyPreviewResponse( + Success: true, + PolicyDigest: "digest-123", + RevisionId: "rev-42", + Issues: ImmutableArray.Empty, + Diffs: ImmutableArray.Create(new PolicyVerdictDiff(baseline, projected)), + ChangedCount: 1); + + var document = new ReportDocumentDto + { + ReportId = "report-abc", + ImageDigest = "sha256:feedface", + GeneratedAt = DateTimeOffset.Parse("2025-10-19T12:34:56Z"), + Verdict = "blocked", + Policy = new ReportPolicyDto + { + RevisionId = "rev-42", + Digest = "digest-123" + }, + Summary = new ReportSummaryDto + { + Total = 1, + Blocked = 1, + Warned = 0, + Ignored = 0, + Quieted = 0 + }, + Verdicts = new[] + { + new PolicyPreviewVerdictDto + { + FindingId = "finding-1", + Status = "Blocked", + Score = 47.5, + SourceTrust = "NVD", + Reachability = "runtime" + } + } + }; + + var envelope = new DsseEnvelopeDto + { + PayloadType = "application/vnd.stellaops.report+json", + Payload = Convert.ToBase64String(JsonSerializer.SerializeToUtf8Bytes(document, SerializerOptions)), + Signatures = new[] + { + new DsseSignatureDto { KeyId = "test-key", Algorithm = "hs256", Signature = "signature-value" } + } + }; + + var context = new DefaultHttpContext(); + context.User = new ClaimsPrincipal(new ClaimsIdentity(new[] + { + new Claim(StellaOpsClaimTypes.Tenant, "tenant-alpha") + })); + context.Request.Scheme = "https"; + context.Request.Host = new HostString("scanner.example"); + + await dispatcher.PublishAsync(request, preview, document, envelope, context, cancellationToken); + + var readyEvent = Assert.Single(publisher.Events, evt => evt.Kind == OrchestratorEventKinds.ScannerReportReady); + var links = Assert.IsType(readyEvent.Payload).Links; + + Assert.Equal("https://scanner.example/console/insights/report-abc", links.Report?.Ui); + Assert.Equal("https://scanner.example/custom-api/reports-view/report-abc", links.Report?.Api); + Assert.Equal("https://scanner.example/console/policy-center/revisions/rev-42", links.Policy?.Ui); + Assert.Equal("https://scanner.example/custom-api/policy-hub/revisions/rev-42", links.Policy?.Api); + Assert.Equal("https://scanner.example/console/evidence/report-abc", links.Attestation?.Ui); + Assert.Equal("https://scanner.example/custom-api/reports-view/report-abc/attestation", links.Attestation?.Api); + } + private sealed class RecordingEventPublisher : IPlatformEventPublisher { public List Events { get; } = new(); @@ -173,6 +293,6 @@ public sealed class ReportEventDispatcherTests { Events.Add(@event); return Task.CompletedTask; - } - } -} + } + } +} diff --git a/src/Scanner/__Tests/StellaOps.Scanner.WebService.Tests/ReportSamplesTests.cs b/src/Scanner/__Tests/StellaOps.Scanner.WebService.Tests/ReportSamplesTests.cs index 07685c267..39ce7f0f4 100644 --- a/src/Scanner/__Tests/StellaOps.Scanner.WebService.Tests/ReportSamplesTests.cs +++ b/src/Scanner/__Tests/StellaOps.Scanner.WebService.Tests/ReportSamplesTests.cs @@ -28,8 +28,8 @@ public sealed class ReportSamplesTests Assert.NotNull(response!.Report); Assert.NotNull(response.Dsse); - var reportBytes = JsonSerializer.SerializeToUtf8Bytes(response.Report, SerializerOptions); - var expectedPayload = Convert.ToBase64String(reportBytes); - Assert.Equal(expectedPayload, response.Dsse!.Payload); - } + var reportBytes = JsonSerializer.SerializeToUtf8Bytes(response.Report, SerializerOptions); + var expectedPayload = Convert.ToBase64String(reportBytes); + Assert.Equal(expectedPayload, response.Dsse!.Payload); + } } diff --git a/src/Scanner/__Tests/StellaOps.Scanner.WebService.Tests/ScannerSurfaceSecretConfiguratorTests.cs b/src/Scanner/__Tests/StellaOps.Scanner.WebService.Tests/ScannerSurfaceSecretConfiguratorTests.cs new file mode 100644 index 000000000..1c2d8797e --- /dev/null +++ b/src/Scanner/__Tests/StellaOps.Scanner.WebService.Tests/ScannerSurfaceSecretConfiguratorTests.cs @@ -0,0 +1,142 @@ +using System; +using System.Collections.Generic; +using System.IO; +using System.Text; +using System.Threading; +using Microsoft.Extensions.Logging.Abstractions; +using Microsoft.Extensions.Options; +using StellaOps.Scanner.Surface.Env; +using StellaOps.Scanner.Surface.Secrets; +using StellaOps.Scanner.WebService.Options; +using StellaOps.Scanner.Storage; +using Xunit; + +namespace StellaOps.Scanner.WebService.Tests; + +public sealed class ScannerSurfaceSecretConfiguratorTests +{ + [Fact] + public void Configure_AppliesCasAccessSecretToArtifactStore() + { + const string json = """ + { + "driver": "rustfs", + "endpoint": "https://surface.api", + "bucket": "surface-artifacts", + "apiKey": "rust-key", + "apiKeyHeader": "X-Surface-Api-Key", + "region": "ap-southeast-2" + } + """; + + using var handle = SurfaceSecretHandle.FromBytes(Encoding.UTF8.GetBytes(json)); + var secretProvider = new StubSecretProvider(handle); + var environment = new StubSurfaceEnvironment(); + var options = new ScannerWebServiceOptions(); + + var configurator = new ScannerSurfaceSecretConfigurator( + secretProvider, + environment, + NullLogger.Instance); + + configurator.Configure(options); + + Assert.Equal("rustfs", options.ArtifactStore.Driver); + Assert.Equal("https://surface.api", options.ArtifactStore.Endpoint); + Assert.Equal("surface-artifacts", options.ArtifactStore.Bucket); + Assert.Equal("rust-key", options.ArtifactStore.ApiKey); + Assert.Equal("X-Surface-Api-Key", options.ArtifactStore.ApiKeyHeader); + Assert.Equal("ap-southeast-2", options.ArtifactStore.Region); + } + + [Fact] + public void PostConfigure_SynchronizesArtifactStoreToScannerStorageOptions() + { + var webOptions = Microsoft.Extensions.Options.Options.Create(new ScannerWebServiceOptions + { + ArtifactStore = new ScannerWebServiceOptions.ArtifactStoreOptions + { + Driver = "rustfs", + Endpoint = "https://surface.sync", + ApiKey = "sync-key", + ApiKeyHeader = "X-Sync", + Bucket = "sync-bucket", + Region = "us-west-2", + RootPrefix = "sync" + } + }); + + var configurator = new ScannerStorageOptionsPostConfigurator( + new OptionsMonitorStub(webOptions), + NullLogger.Instance); + + var storageOptions = new ScannerStorageOptions(); + configurator.PostConfigure(Microsoft.Extensions.Options.Options.DefaultName, storageOptions); + + Assert.Equal("rustfs", storageOptions.ObjectStore.Driver); + Assert.Equal("https://surface.sync", storageOptions.ObjectStore.RustFs.BaseUrl); + Assert.Equal("sync-bucket", storageOptions.ObjectStore.BucketName); + Assert.Equal("sync", storageOptions.ObjectStore.RootPrefix); + Assert.Equal("us-west-2", storageOptions.ObjectStore.Region); + Assert.Equal("sync-key", storageOptions.ObjectStore.RustFs.ApiKey); + Assert.Equal("X-Sync", storageOptions.ObjectStore.RustFs.ApiKeyHeader); + } + + private sealed class StubSecretProvider : ISurfaceSecretProvider + { + private readonly SurfaceSecretHandle _handle; + + public StubSecretProvider(SurfaceSecretHandle handle) + { + _handle = handle; + } + + public ValueTask GetAsync(SurfaceSecretRequest request, CancellationToken cancellationToken = default) + => ValueTask.FromResult(_handle); + } + + private sealed class StubSurfaceEnvironment : ISurfaceEnvironment + { + public StubSurfaceEnvironment() + { + Settings = new SurfaceEnvironmentSettings( + new Uri("https://surface"), + "bucket", + "region", + new DirectoryInfo(Path.GetTempPath()), + 256, + false, + Array.Empty(), + new SurfaceSecretsConfiguration("inline", "tenant", null, null, null, true), + "tenant", + new SurfaceTlsConfiguration(null, null, null)); + RawVariables = new Dictionary(StringComparer.OrdinalIgnoreCase); + } + + public SurfaceEnvironmentSettings Settings { get; } + + public IReadOnlyDictionary RawVariables { get; } + } + + private sealed class OptionsMonitorStub : IOptionsMonitor where T : class + { + private readonly IOptions _options; + + public OptionsMonitorStub(IOptions options) + { + _options = options; + } + + public T CurrentValue => _options.Value; + + public T Get(string? name) => _options.Value; + + public IDisposable OnChange(Action listener) => NullDisposable.Instance; + + private sealed class NullDisposable : IDisposable + { + public static readonly NullDisposable Instance = new(); + public void Dispose() { } + } + } +} diff --git a/src/Scanner/__Tests/StellaOps.Scanner.WebService.Tests/ScansEndpointsTests.cs b/src/Scanner/__Tests/StellaOps.Scanner.WebService.Tests/ScansEndpointsTests.cs index fbe4dbd05..4c91b1fb3 100644 --- a/src/Scanner/__Tests/StellaOps.Scanner.WebService.Tests/ScansEndpointsTests.cs +++ b/src/Scanner/__Tests/StellaOps.Scanner.WebService.Tests/ScansEndpointsTests.cs @@ -6,6 +6,7 @@ using System.Net; using System.Net.Http.Json; using System.Linq; using System.Text.Json; +using System.Text.Json.Serialization; using System.Threading.Tasks; using System.Threading; using Microsoft.AspNetCore.Http; @@ -22,8 +23,8 @@ using StellaOps.Scanner.WebService.Services; namespace StellaOps.Scanner.WebService.Tests; -public sealed class ScansEndpointsTests -{ +public sealed class ScansEndpointsTests +{ [Fact] public async Task SubmitScanReturnsAcceptedAndStatusRetrievable() { @@ -272,7 +273,7 @@ public sealed class ScansEndpointsTests var response = await client.GetAsync($"/api/v1/scans/{scanId}/entrytrace"); Assert.Equal(HttpStatusCode.OK, response.StatusCode); - var payload = await response.Content.ReadFromJsonAsync(); + var payload = await response.Content.ReadFromJsonAsync(SerializerOptions, CancellationToken.None); Assert.NotNull(payload); Assert.Equal(scanId, payload!.ScanId); Assert.Equal("sha256:entrytrace", payload.ImageDigest); @@ -559,7 +560,7 @@ public sealed class ScansEndpointsTests var response = await client.GetAsync($"/api/v1/scans/{scanId}/entrytrace"); Assert.Equal(HttpStatusCode.OK, response.StatusCode); - var payload = await response.Content.ReadFromJsonAsync(); + var payload = await response.Content.ReadFromJsonAsync(SerializerOptions, CancellationToken.None); Assert.NotNull(payload); Assert.Equal(storedResult.ScanId, payload!.ScanId); Assert.Equal(storedResult.ImageDigest, payload.ImageDigest); @@ -583,7 +584,10 @@ public sealed class ScansEndpointsTests Assert.Equal(HttpStatusCode.NotFound, response.StatusCode); } - private static readonly JsonSerializerOptions SerializerOptions = new(JsonSerializerDefaults.Web); + private static readonly JsonSerializerOptions SerializerOptions = new(JsonSerializerDefaults.Web) + { + Converters = { new JsonStringEnumConverter() } + }; private sealed record ProgressEnvelope( string ScanId, diff --git a/src/Scanner/__Tests/StellaOps.Scanner.WebService.Tests/SurfaceCacheOptionsConfiguratorTests.cs b/src/Scanner/__Tests/StellaOps.Scanner.WebService.Tests/SurfaceCacheOptionsConfiguratorTests.cs index eb7ffdf8b..3527e1a56 100644 --- a/src/Scanner/__Tests/StellaOps.Scanner.WebService.Tests/SurfaceCacheOptionsConfiguratorTests.cs +++ b/src/Scanner/__Tests/StellaOps.Scanner.WebService.Tests/SurfaceCacheOptionsConfiguratorTests.cs @@ -19,12 +19,12 @@ public sealed class SurfaceCacheOptionsConfiguratorTests "surface-cache", null, cacheRoot, - cacheQuotaMegabytes: 512, - prefetchEnabled: true, - featureFlags: Array.Empty(), - secrets: new SurfaceSecretsConfiguration("file", "tenant-b", "/etc/secrets", null, null, allowInline: false), - tenant: "tenant-b", - tls: new SurfaceTlsConfiguration(null, null, new X509Certificate2Collection())); + 512, + true, + Array.Empty(), + new SurfaceSecretsConfiguration("file", "tenant-b", "/etc/secrets", null, null, false), + "tenant-b", + new SurfaceTlsConfiguration(null, null, new X509Certificate2Collection())); var environment = new StubSurfaceEnvironment(settings); var configurator = new SurfaceCacheOptionsConfigurator(environment); diff --git a/src/Scanner/__Tests/StellaOps.Scanner.Worker.Tests/ScannerStorageSurfaceSecretConfiguratorTests.cs b/src/Scanner/__Tests/StellaOps.Scanner.Worker.Tests/ScannerStorageSurfaceSecretConfiguratorTests.cs new file mode 100644 index 000000000..74957c29b --- /dev/null +++ b/src/Scanner/__Tests/StellaOps.Scanner.Worker.Tests/ScannerStorageSurfaceSecretConfiguratorTests.cs @@ -0,0 +1,92 @@ +using System; +using System.Collections.Generic; +using System.IO; +using System.Text; +using System.Threading; +using Microsoft.Extensions.Logging.Abstractions; +using StellaOps.Scanner.Storage; +using StellaOps.Scanner.Surface.Env; +using StellaOps.Scanner.Surface.Secrets; +using StellaOps.Scanner.Worker.Options; +using Xunit; + +namespace StellaOps.Scanner.Worker.Tests; + +public sealed class ScannerStorageSurfaceSecretConfiguratorTests +{ + [Fact] + public void Configure_WithCasAccessSecret_AppliesSettings() + { + const string json = """ + { + "driver": "rustfs", + "endpoint": "https://surface.example", + "region": "eu-central-1", + "bucket": "surface-bucket", + "rootPrefix": "scanner", + "apiKey": "rustfs-api", + "apiKeyHeader": "X-Rustfs-Key", + "allowInsecureTls": false + } + """; + + using var handle = SurfaceSecretHandle.FromBytes(Encoding.UTF8.GetBytes(json)); + var secretProvider = new StubSecretProvider(handle); + var environment = new StubSurfaceEnvironment("tenant-eu"); + + var configurator = new ScannerStorageSurfaceSecretConfigurator( + secretProvider, + environment, + NullLogger.Instance); + + var options = new ScannerStorageOptions(); + configurator.Configure(options); + + Assert.Equal("rustfs", options.ObjectStore.Driver); + Assert.Equal("https://surface.example", options.ObjectStore.RustFs.BaseUrl); + Assert.Equal("eu-central-1", options.ObjectStore.Region); + Assert.Equal("surface-bucket", options.ObjectStore.BucketName); + Assert.Equal("scanner", options.ObjectStore.RootPrefix); + Assert.Equal("rustfs-api", options.ObjectStore.RustFs.ApiKey); + Assert.Equal("X-Rustfs-Key", options.ObjectStore.RustFs.ApiKeyHeader); + } + + private sealed class StubSecretProvider : ISurfaceSecretProvider + { + private readonly SurfaceSecretHandle _handle; + + public StubSecretProvider(SurfaceSecretHandle handle) + { + _handle = handle; + } + + public ValueTask GetAsync(SurfaceSecretRequest request, CancellationToken cancellationToken = default) + => ValueTask.FromResult(_handle); + } + + private sealed class StubSurfaceEnvironment : ISurfaceEnvironment + { + public StubSurfaceEnvironment(string tenant) + { + Settings = new SurfaceEnvironmentSettings( + new Uri("https://surface"), + "bucket", + "region-1", + new DirectoryInfo(Path.GetTempPath()), + 1024, + false, + Array.Empty(), + new SurfaceSecretsConfiguration("inline", tenant, null, null, null, true), + tenant, + new SurfaceTlsConfiguration(null, null, null)) + { + CreatedAtUtc = DateTimeOffset.UtcNow + }; + RawVariables = new Dictionary(StringComparer.OrdinalIgnoreCase); + } + + public SurfaceEnvironmentSettings Settings { get; } + + public IReadOnlyDictionary RawVariables { get; } + } +} diff --git a/src/Scanner/__Tests/StellaOps.Scanner.Worker.Tests/SurfaceCacheOptionsConfiguratorTests.cs b/src/Scanner/__Tests/StellaOps.Scanner.Worker.Tests/SurfaceCacheOptionsConfiguratorTests.cs index 4bf957ea0..1ed91132e 100644 --- a/src/Scanner/__Tests/StellaOps.Scanner.Worker.Tests/SurfaceCacheOptionsConfiguratorTests.cs +++ b/src/Scanner/__Tests/StellaOps.Scanner.Worker.Tests/SurfaceCacheOptionsConfiguratorTests.cs @@ -19,12 +19,12 @@ public sealed class SurfaceCacheOptionsConfiguratorTests "surface-cache", null, cacheRoot, - cacheQuotaMegabytes: 1024, - prefetchEnabled: false, - featureFlags: Array.Empty(), - secrets: new SurfaceSecretsConfiguration("file", "tenant-a", "/etc/secrets", null, null, false), - tenant: "tenant-a", - tls: new SurfaceTlsConfiguration(null, null, new X509Certificate2Collection())); + 1024, + false, + Array.Empty(), + new SurfaceSecretsConfiguration("file", "tenant-a", "/etc/secrets", null, null, false), + "tenant-a", + new SurfaceTlsConfiguration(null, null, new X509Certificate2Collection())); var environment = new StubSurfaceEnvironment(settings); var configurator = new SurfaceCacheOptionsConfigurator(environment); diff --git a/src/Scanner/docs/events/samples/scanner.event.report.ready@1.sample.json b/src/Scanner/docs/events/samples/scanner.event.report.ready@1.sample.json index 10ad593e2..814435d56 100644 --- a/src/Scanner/docs/events/samples/scanner.event.report.ready@1.sample.json +++ b/src/Scanner/docs/events/samples/scanner.event.report.ready@1.sample.json @@ -3,8 +3,8 @@ "kind": "scanner.event.report.ready", "version": 1, "tenant": "tenant-alpha", - "occurredAt": "2025-10-19T12:34:56Z", - "recordedAt": "2025-10-19T12:34:57Z", + "occurredAt": "2025-10-19T12:34:56+00:00", + "recordedAt": "2025-10-19T12:34:57+00:00", "source": "scanner.webservice", "idempotencyKey": "scanner.event.report.ready:tenant-alpha:report-abc", "correlationId": "report-abc", @@ -25,7 +25,7 @@ "reportId": "report-abc", "scanId": "report-abc", "imageDigest": "sha256:feedface", - "generatedAt": "2025-10-19T12:34:56Z", + "generatedAt": "2025-10-19T12:34:56+00:00", "verdict": "fail", "summary": { "total": 1, @@ -72,7 +72,7 @@ }, "report": { "reportId": "report-abc", - "generatedAt": "2025-10-19T12:34:56Z", + "generatedAt": "2025-10-19T12:34:56+00:00", "imageDigest": "sha256:feedface", "policy": { "digest": "digest-123", diff --git a/src/Scanner/docs/events/samples/scanner.event.scan.completed@1.sample.json b/src/Scanner/docs/events/samples/scanner.event.scan.completed@1.sample.json index 375c6185b..c2f416b9b 100644 --- a/src/Scanner/docs/events/samples/scanner.event.scan.completed@1.sample.json +++ b/src/Scanner/docs/events/samples/scanner.event.scan.completed@1.sample.json @@ -3,8 +3,8 @@ "kind": "scanner.event.scan.completed", "version": 1, "tenant": "tenant-alpha", - "occurredAt": "2025-10-19T12:34:56Z", - "recordedAt": "2025-10-19T12:34:57Z", + "occurredAt": "2025-10-19T12:34:56+00:00", + "recordedAt": "2025-10-19T12:34:57+00:00", "source": "scanner.webservice", "idempotencyKey": "scanner.event.scan.completed:tenant-alpha:report-abc", "correlationId": "report-abc", @@ -78,7 +78,7 @@ }, "report": { "reportId": "report-abc", - "generatedAt": "2025-10-19T12:34:56Z", + "generatedAt": "2025-10-19T12:34:56+00:00", "imageDigest": "sha256:feedface", "policy": { "digest": "digest-123", diff --git a/src/Scanner/samples/api/reports/report-sample.dsse.json b/src/Scanner/samples/api/reports/report-sample.dsse.json new file mode 100644 index 000000000..0fce7b9e3 --- /dev/null +++ b/src/Scanner/samples/api/reports/report-sample.dsse.json @@ -0,0 +1,80 @@ +{ + "report": { + "reportId": "report-abc", + "imageDigest": "sha256:feedface", + "generatedAt": "2025-10-19T12:34:56+00:00", + "verdict": "blocked", + "policy": { + "revisionId": "rev-42", + "digest": "digest-123" + }, + "summary": { + "total": 1, + "blocked": 1, + "warned": 0, + "ignored": 0, + "quieted": 0 + }, + "verdicts": [ + { + "findingId": "finding-1", + "reachability": "runtime", + "score": 47.5, + "sourceTrust": "NVD", + "status": "Blocked" + } + ], + "issues": [], + "surface": { + "tenant": "tenant-alpha", + "generatedAt": "2025-10-19T12:34:56+00:00", + "manifestDigest": "sha256:4fee87d186291ddfbbcc2c56c8ed0e828520b8f52e1cde0e13bba082f10918d7", + "manifestUri": "cas://scanner-artifacts/scanner/surface/manifests/tenant-alpha/sha256/4f/ee/4fee87d186291ddfbbcc2c56c8ed0e828520b8f52e1cde0e13bba082f10918d7.json", + "manifest": { + "schema": "stellaops.surface.manifest@1", + "tenant": "tenant-alpha", + "imageDigest": "sha256:feedface", + "generatedAt": "2025-10-19T12:34:56+00:00", + "artifacts": [ + { + "kind": "entry-trace", + "uri": "cas://scanner-artifacts/scanner/entry-trace/f0f0f0f0f0f0f0f0f0f0f0f0f0f0f0f0f0f0f0f0f0f0f0f0f0f0f0f0f0f0f0/entry-trace.json", + "digest": "sha256:f0f0f0f0f0f0f0f0f0f0f0f0f0f0f0f0f0f0f0f0f0f0f0f0f0f0f0f0f0f0f0f0", + "mediaType": "application/json", + "format": "json", + "sizeBytes": 4096 + }, + { + "kind": "sbom-inventory", + "uri": "cas://scanner-artifacts/scanner/images/feedface/sbom.cdx.json", + "digest": "sha256:1111111111111111111111111111111111111111111111111111111111111111", + "mediaType": "application/vnd.cyclonedx+json;version=1.6;view=inventory", + "format": "cdx-json", + "sizeBytes": 24576, + "view": "inventory" + }, + { + "kind": "sbom-usage", + "uri": "cas://scanner-artifacts/scanner/images/feedface/sbom-usage.cdx.json", + "digest": "sha256:2222222222222222222222222222222222222222222222222222222222222222", + "mediaType": "application/vnd.cyclonedx+json;version=1.6;view=usage", + "format": "cdx-json", + "sizeBytes": 16384, + "view": "usage" + } + ] + } + } + }, + "dsse": { + "payloadType": "application/vnd.stellaops.report+json", + "payload": "eyJyZXBvcnRJZCI6InJlcG9ydC1hYmMiLCJpbWFnZURpZ2VzdCI6InNoYTI1NjpmZWVkZmFjZSIsImdlbmVyYXRlZEF0IjoiMjAyNS0xMC0xOVQxMjozNDo1NiswMDowMCIsInZlcmRpY3QiOiJibG9ja2VkIiwicG9saWN5Ijp7InJldmlzaW9uSWQiOiJyZXYtNDIiLCJkaWdlc3QiOiJkaWdlc3QtMTIzIn0sInN1bW1hcnkiOnsidG90YWwiOjEsImJsb2NrZWQiOjEsIndhcm5lZCI6MCwiaWdub3JlZCI6MCwicXVpZXRlZCI6MH0sInZlcmRpY3RzIjpbeyJmaW5kaW5nSWQiOiJmaW5kaW5nLTEiLCJzdGF0dXMiOiJCbG9ja2VkIiwic2NvcmUiOjQ3LjUsInNvdXJjZVRydXN0IjoiTlZEIiwicmVhY2hhYmlsaXR5IjoicnVudGltZSJ9XSwiaXNzdWVzIjpbXSwic3VyZmFjZSI6eyJ0ZW5hbnQiOiJ0ZW5hbnQtYWxwaGEiLCJnZW5lcmF0ZWRBdCI6IjIwMjUtMTAtMTlUMTI6MzQ6NTYrMDA6MDAiLCJtYW5pZmVzdERpZ2VzdCI6InNoYTI1Njo0ZmVlODdkMTg2MjkxZGRmYmJjYzJjNTZjOGVkMGU4Mjg1MjBiOGY1MmUxY2RlMGUxM2JiYTA4MmYxMDkxOGQ3IiwibWFuaWZlc3RVcmkiOiJjYXM6Ly9zY2FubmVyLWFydGlmYWN0cy9zY2FubmVyL3N1cmZhY2UvbWFuaWZlc3RzL3RlbmFudC1hbHBoYS9zaGEyNTYvNGYvZWUvNGZlZTg3ZDE4NjI5MWRkZmJiY2MyYzU2YzhlZDBlODI4NTIwYjhmNTJlMWNkZTBlMTNiYmEwODJmMTA5MThkNy5qc29uIiwibWFuaWZlc3QiOnsic2NoZW1hIjoic3RlbGxhb3BzLnN1cmZhY2UubWFuaWZlc3RAMSIsInRlbmFudCI6InRlbmFudC1hbHBoYSIsImltYWdlRGlnZXN0Ijoic2hhMjU2OmZlZWRmYWNlIiwiZ2VuZXJhdGVkQXQiOiIyMDI1LTEwLTE5VDEyOjM0OjU2KzAwOjAwIiwiYXJ0aWZhY3RzIjpbeyJraW5kIjoiZW50cnktdHJhY2UiLCJ1cmkiOiJjYXM6Ly9zY2FubmVyLWFydGlmYWN0cy9zY2FubmVyL2VudHJ5LXRyYWNlL2YwZjBmMGYwZjBmMGYwZjBmMGYwZjBmMGYwZjBmMGYwZjBmMGYwZjBmMGYwZjBmMGYwZjBmMGYwZjBmMGYwL2VudHJ5LXRyYWNlLmpzb24iLCJkaWdlc3QiOiJzaGEyNTY6ZjBmMGYwZjBmMGYwZjBmMGYwZjBmMGYwZjBmMGYwZjBmMGYwZjBmMGYwZjBmMGYwZjBmMGYwZjBmMGYwZjBmMCIsIm1lZGlhVHlwZSI6ImFwcGxpY2F0aW9uL2pzb24iLCJmb3JtYXQiOiJqc29uIiwic2l6ZUJ5dGVzIjo0MDk2fSx7ImtpbmQiOiJzYm9tLWludmVudG9yeSIsInVyaSI6ImNhczovL3NjYW5uZXItYXJ0aWZhY3RzL3NjYW5uZXIvaW1hZ2VzL2ZlZWRmYWNlL3Nib20uY2R4Lmpzb24iLCJkaWdlc3QiOiJzaGEyNTY6MTExMTExMTExMTExMTExMTExMTExMTExMTExMTExMTExMTExMTExMTExMTExMTExMTExMTExMTExMTExMTExMSIsIm1lZGlhVHlwZSI6ImFwcGxpY2F0aW9uL3ZuZC5jeWNsb25lZHhcdTAwMkJqc29uO3ZlcnNpb249MS42O3ZpZXc9aW52ZW50b3J5IiwiZm9ybWF0IjoiY2R4LWpzb24iLCJzaXplQnl0ZXMiOjI0NTc2LCJ2aWV3IjoiaW52ZW50b3J5In0seyJraW5kIjoic2JvbS11c2FnZSIsInVyaSI6ImNhczovL3NjYW5uZXItYXJ0aWZhY3RzL3NjYW5uZXIvaW1hZ2VzL2ZlZWRmYWNlL3Nib20tdXNhZ2UuY2R4Lmpzb24iLCJkaWdlc3QiOiJzaGEyNTY6MjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMiIsIm1lZGlhVHlwZSI6ImFwcGxpY2F0aW9uL3ZuZC5jeWNsb25lZHhcdTAwMkJqc29uO3ZlcnNpb249MS42O3ZpZXc9dXNhZ2UiLCJmb3JtYXQiOiJjZHgtanNvbiIsInNpemVCeXRlcyI6MTYzODQsInZpZXciOiJ1c2FnZSJ9XX19fQ==", + "signatures": [ + { + "keyId": "test-key", + "algorithm": "hs256", + "signature": "signature-value" + } + ] + } +} diff --git a/src/Scheduler/StellaOps.Scheduler.WebService/PolicySimulations/PolicySimulationMetricsProvider.cs b/src/Scheduler/StellaOps.Scheduler.WebService/PolicySimulations/PolicySimulationMetricsProvider.cs index 932a6618b..8d4ee9cdb 100644 --- a/src/Scheduler/StellaOps.Scheduler.WebService/PolicySimulations/PolicySimulationMetricsProvider.cs +++ b/src/Scheduler/StellaOps.Scheduler.WebService/PolicySimulations/PolicySimulationMetricsProvider.cs @@ -57,7 +57,7 @@ internal sealed class PolicySimulationMetricsProvider : IPolicySimulationMetrics unit: "runs", description: "Queued policy simulation jobs grouped by status."); _latencyHistogram = _meter.CreateHistogram( - "policy_simulation_latency", + "policy_simulation_latency_seconds", unit: "s", description: "End-to-end policy simulation latency (seconds)."); } @@ -84,9 +84,11 @@ internal sealed class PolicySimulationMetricsProvider : IPolicySimulationMetrics totalQueueDepth += count; } + var snapshot = new Dictionary(queueCounts, StringComparer.Ordinal); + lock (_snapshotLock) { - _latestQueueSnapshot = queueCounts; + _latestQueueSnapshot = snapshot; _latestTenantId = tenantId; } @@ -115,7 +117,7 @@ internal sealed class PolicySimulationMetricsProvider : IPolicySimulationMetrics Average(durations)); return new PolicySimulationMetricsResponse( - new PolicySimulationQueueDepth(totalQueueDepth, queueCounts), + new PolicySimulationQueueDepth(totalQueueDepth, snapshot), latencyMetrics); } diff --git a/src/Scheduler/StellaOps.Scheduler.WebService/TASKS.md b/src/Scheduler/StellaOps.Scheduler.WebService/TASKS.md index 4a2519948..a60f4d454 100644 --- a/src/Scheduler/StellaOps.Scheduler.WebService/TASKS.md +++ b/src/Scheduler/StellaOps.Scheduler.WebService/TASKS.md @@ -30,8 +30,10 @@ | ID | Status | Owner(s) | Depends on | Description | Exit Criteria | |----|--------|----------|------------|-------------|---------------| | SCHED-CONSOLE-27-001 | DONE (2025-11-03) | Scheduler WebService Guild, Policy Registry Guild | SCHED-WEB-16-103, REGISTRY-API-27-005 | Provide policy batch simulation orchestration endpoints (`/policies/simulations` POST/GET) exposing run creation, shard status, SSE progress, cancellation, and retries with RBAC enforcement. | API handles shard lifecycle with SSE heartbeats + retry headers; unauthorized requests rejected; integration tests cover submit/cancel/resume flows. | -| SCHED-CONSOLE-27-002 | DOING (2025-11-03) | Scheduler WebService Guild, Observability Guild | SCHED-CONSOLE-27-001 | Emit telemetry endpoints/metrics (`policy_simulation_queue_depth`, `policy_simulation_latency`) and webhook callbacks for completion/failure consumed by Registry. | Metrics exposed via gateway, dashboards seeded, webhook contract documented, integration tests validate metrics emission. | -> 2025-11-06: Added tenant-aware tagging to `policy_simulation_queue_depth` metrics and unit coverage for the metrics provider snapshot. +| SCHED-CONSOLE-27-002 | DONE (2025-11-05) | Scheduler WebService Guild, Observability Guild | SCHED-CONSOLE-27-001 | Emit telemetry endpoints/metrics (`policy_simulation_queue_depth`, `policy_simulation_latency_seconds`) and webhook callbacks for completion/failure consumed by Registry. | Metrics exposed via gateway, dashboards seeded, webhook contract documented, integration tests validate metrics emission. | +> 2025-11-05: Resuming to align instrumentation naming with architecture spec, exercise latency recording in SSE flows, and ensure registry webhook contract (samples/docs) reflects terminal result behaviour. +> 2025-11-05: Histogram renamed to `policy_simulation_latency_seconds`, queue gauge kept stable, new unit tests cover metrics capture/latency recording, and docs updated. Local `dotnet test` build currently blocked by existing GraphJobs visibility errors (see `StellaOps.Scheduler.WebService/GraphJobs/IGraphJobStore.cs`). +> 2025-11-06: Added tenant-aware tagging to `policy_simulation_queue_depth` gauge samples and refreshed metrics provider snapshot coverage. ## Vulnerability Explorer (Sprint 29) | ID | Status | Owner(s) | Depends on | Description | Exit Criteria | diff --git a/src/Scheduler/StellaOps.Scheduler.WebService/docs/SCHED-WEB-16-103-RUN-APIS.md b/src/Scheduler/StellaOps.Scheduler.WebService/docs/SCHED-WEB-16-103-RUN-APIS.md index deff3bfd4..9ae050887 100644 --- a/src/Scheduler/StellaOps.Scheduler.WebService/docs/SCHED-WEB-16-103-RUN-APIS.md +++ b/src/Scheduler/StellaOps.Scheduler.WebService/docs/SCHED-WEB-16-103-RUN-APIS.md @@ -313,7 +313,7 @@ X-Tenant-Id: tenant-alpha Authorization: Bearer ``` -Returns queue depth and latency summaries tailored for simulation dashboards and alerting. Response properties align with the metric names exposed via OTEL (`policy_simulation_queue_depth`, `policy_simulation_latency`). Canonical payload lives at `samples/api/scheduler/policy-simulation-metrics.json`. +Returns queue depth and latency summaries tailored for simulation dashboards and alerting. Response properties align with the metric names exposed via OTEL (`policy_simulation_queue_depth`, `policy_simulation_latency_seconds`). Canonical payload lives at `samples/api/scheduler/policy-simulation-metrics.json`. - `policy_simulation_queue_depth.total` — pending simulation jobs (aggregate of `pending`, `dispatching`, `submitted`). - `policy_simulation_latency.*` — latency percentiles (seconds) computed from the most recent terminal simulations. diff --git a/src/Scheduler/__Tests/StellaOps.Scheduler.WebService.Tests/PolicySimulationMetricsProviderTests.cs b/src/Scheduler/__Tests/StellaOps.Scheduler.WebService.Tests/PolicySimulationMetricsProviderTests.cs index 99a1ba416..c3d31c1dc 100644 --- a/src/Scheduler/__Tests/StellaOps.Scheduler.WebService.Tests/PolicySimulationMetricsProviderTests.cs +++ b/src/Scheduler/__Tests/StellaOps.Scheduler.WebService.Tests/PolicySimulationMetricsProviderTests.cs @@ -16,6 +16,52 @@ namespace StellaOps.Scheduler.WebService.Tests; public sealed class PolicySimulationMetricsProviderTests { + [Fact] + public async Task CaptureAsync_ComputesQueueDepthAndLatency() + { + var now = DateTimeOffset.UtcNow; + var counts = new Dictionary + { + [PolicyRunJobStatus.Pending] = 2, + [PolicyRunJobStatus.Dispatching] = 1, + [PolicyRunJobStatus.Submitted] = 1 + }; + + var jobs = new[] + { + CreateJob( + status: PolicyRunJobStatus.Completed, + queuedAt: now.AddSeconds(-30), + submittedAt: now.AddSeconds(-28), + completedAt: now.AddSeconds(-20)), + CreateJob( + status: PolicyRunJobStatus.Cancelled, + queuedAt: now.AddSeconds(-50), + submittedAt: now.AddSeconds(-48), + completedAt: null, + cancelledAt: now.AddSeconds(-20)) + }; + + var repository = new StubPolicyRunJobRepository(counts, jobs); + + using var provider = new PolicySimulationMetricsProvider(repository); + + var response = await provider.CaptureAsync("tenant-alpha", CancellationToken.None); + + Assert.Equal(4, response.QueueDepth.Total); + Assert.Equal(2, response.QueueDepth.ByStatus["pending"]); + Assert.Equal(1, response.QueueDepth.ByStatus["dispatching"]); + Assert.Equal(1, response.QueueDepth.ByStatus["submitted"]); + + Assert.Equal(2, response.Latency.Samples); + Assert.Equal(20.0, response.Latency.Mean); + Assert.Equal(20.0, response.Latency.P50); + Assert.Equal(28.0, response.Latency.P90); + Assert.Equal(29.0, response.Latency.P95); + Assert.True(response.Latency.P99.HasValue); + Assert.Equal(29.8, response.Latency.P99.Value, 1); + } + [Fact] public async Task CaptureAsync_UpdatesSnapshotAndEmitsTenantTaggedGauge() { @@ -25,41 +71,48 @@ public sealed class PolicySimulationMetricsProviderTests repository.QueueCounts[PolicyRunJobStatus.Submitted] = 2; var now = DateTimeOffset.Parse("2025-11-06T10:00:00Z", CultureInfo.InvariantCulture, DateTimeStyles.AdjustToUniversal); - repository.TerminalJobs.Add(CreateJob("job-1", PolicyRunJobStatus.Completed, now.AddMinutes(-30), now.AddMinutes(-5))); - repository.TerminalJobs.Add(CreateJob("job-2", PolicyRunJobStatus.Failed, now.AddMinutes(-20), now.AddMinutes(-2))); + repository.Jobs.Add(CreateJob( + status: PolicyRunJobStatus.Completed, + queuedAt: now.AddMinutes(-30), + submittedAt: now.AddMinutes(-28), + completedAt: now.AddMinutes(-5), + id: "job-1", + runId: "run-job-1")); + repository.Jobs.Add(CreateJob( + status: PolicyRunJobStatus.Failed, + queuedAt: now.AddMinutes(-20), + submittedAt: now.AddMinutes(-18), + completedAt: now.AddMinutes(-2), + id: "job-2", + runId: "run-job-2", + lastError: "policy engine timeout")); using var provider = new PolicySimulationMetricsProvider(repository); - var response = await provider.CaptureAsync("tenant-alpha", CancellationToken.None); - - Assert.Equal(6, response.QueueDepth.Total); - Assert.Equal(3, response.QueueDepth.ByStatus["pending"]); - Assert.Equal(2, response.QueueDepth.ByStatus["submitted"]); - var measurements = new List<(string Status, string Tenant, long Value)>(); using var listener = new MeterListener { - InstrumentPublished = (instrument, listener) => + InstrumentPublished = (instrument, meterListener) => { if (instrument.Meter.Name == "StellaOps.Scheduler.WebService.PolicySimulations" && instrument.Name == "policy_simulation_queue_depth") { - listener.EnableMeasurementEvents(instrument); + meterListener.EnableMeasurementEvents(instrument); } } }; listener.SetMeasurementEventCallback((instrument, measurement, tags, state) => { - var status = ""; - var tenant = ""; + var status = string.Empty; + var tenant = string.Empty; + foreach (var tag in tags) { if (string.Equals(tag.Key, "status", StringComparison.Ordinal)) { status = tag.Value?.ToString() ?? string.Empty; } - - if (string.Equals(tag.Key, "tenantId", StringComparison.Ordinal)) + else if (string.Equals(tag.Key, "tenantId", StringComparison.Ordinal)) { tenant = tag.Value?.ToString() ?? string.Empty; } @@ -68,65 +121,154 @@ public sealed class PolicySimulationMetricsProviderTests measurements.Add((status, tenant, measurement)); }); listener.Start(); + + var response = await provider.CaptureAsync("tenant-alpha", CancellationToken.None); + Assert.Equal(6, response.QueueDepth.Total); + listener.RecordObservableInstruments(); Assert.Contains(measurements, item => item.Status == "pending" && item.Tenant == "tenant-alpha" && item.Value == 3); + listener.Dispose(); } - private static PolicyRunJob CreateJob(string id, PolicyRunJobStatus status, DateTimeOffset queuedAt, DateTimeOffset finishedAt) + [Fact] + public void RecordLatency_EmitsHistogramMeasurement() { - DateTimeOffset? submittedAt = status is PolicyRunJobStatus.Completed or PolicyRunJobStatus.Failed - ? queuedAt.AddMinutes(2) - : null; - DateTimeOffset? completedAt = status is PolicyRunJobStatus.Completed or PolicyRunJobStatus.Failed - ? finishedAt - : null; - DateTimeOffset? cancelledAt = status is PolicyRunJobStatus.Cancelled ? finishedAt : null; - var lastError = status is PolicyRunJobStatus.Failed ? "policy engine timeout" : null; + var repository = new StubPolicyRunJobRepository(); + + using var provider = new PolicySimulationMetricsProvider(repository); + + var measurements = new List(); + using var listener = new MeterListener + { + InstrumentPublished = (instrument, meterListener) => + { + if (instrument.Meter.Name == "StellaOps.Scheduler.WebService.PolicySimulations" && + instrument.Name == "policy_simulation_latency_seconds") + { + meterListener.EnableMeasurementEvents(instrument); + } + } + }; + + listener.SetMeasurementEventCallback((instrument, measurement, tags, state) => + { + if (instrument.Name == "policy_simulation_latency_seconds") + { + measurements.Add(measurement); + } + }); + listener.Start(); + + var now = DateTimeOffset.UtcNow; + var latencyJob = CreateJob( + status: PolicyRunJobStatus.Completed, + queuedAt: now.AddSeconds(-12), + submittedAt: now.AddSeconds(-10), + completedAt: now, + id: "job-latency", + runId: "run-1"); + var status = PolicyRunStatusFactory.Create(latencyJob, now); + + provider.RecordLatency(status, now); + + Assert.Single(measurements); + Assert.Equal(12, measurements[0], precision: 6); + + listener.Dispose(); + } + + private static PolicyRunJob CreateJob( + PolicyRunJobStatus status, + DateTimeOffset queuedAt, + DateTimeOffset? submittedAt, + DateTimeOffset? completedAt, + DateTimeOffset? cancelledAt = null, + string? id = null, + string? runId = null, + string? lastError = null) + { + var jobId = id ?? Guid.NewGuid().ToString("N"); + var resolvedRunId = runId ?? $"run:{jobId}"; + var updatedAt = completedAt ?? cancelledAt ?? submittedAt ?? queuedAt; return new PolicyRunJob( - SchedulerSchemaVersions.PolicyRunJob, - id, - "tenant-alpha", - "policy-x", - 1, - PolicyRunMode.Simulate, - PolicyRunPriority.Normal, - 0, - $"run-{id}", - "user:actor", - null, - null, - PolicyRunInputs.Empty, - queuedAt, - status, - 1, - finishedAt, - status == PolicyRunJobStatus.Failed ? "policy engine timeout" : null, - queuedAt, - finishedAt, - finishedAt, - submittedAt, - completedAt, - null, - null, - false, - null, - null, - cancelledAt); + SchemaVersion: SchedulerSchemaVersions.PolicyRunJob, + Id: jobId, + TenantId: "tenant-alpha", + PolicyId: "policy-alpha", + PolicyVersion: 1, + Mode: PolicyRunMode.Simulate, + Priority: PolicyRunPriority.Normal, + PriorityRank: 0, + RunId: resolvedRunId, + RequestedBy: "tester", + CorrelationId: null, + Metadata: ImmutableSortedDictionary.Empty, + Inputs: PolicyRunInputs.Empty, + QueuedAt: queuedAt, + Status: status, + AttemptCount: 1, + LastAttemptAt: submittedAt ?? completedAt ?? queuedAt, + LastError: lastError, + CreatedAt: queuedAt, + UpdatedAt: updatedAt, + AvailableAt: queuedAt, + SubmittedAt: submittedAt, + CompletedAt: completedAt, + LeaseOwner: null, + LeaseExpiresAt: null, + CancellationRequested: false, + CancellationRequestedAt: null, + CancellationReason: null, + CancelledAt: cancelledAt); } private sealed class StubPolicyRunJobRepository : IPolicyRunJobRepository { - public Dictionary QueueCounts { get; } = new(); - public List TerminalJobs { get; } = new(); - - public Task CountAsync(string tenantId, PolicyRunMode mode, IReadOnlyCollection statuses, CancellationToken cancellationToken = default) + public StubPolicyRunJobRepository() { - var total = 0L; + } + + public StubPolicyRunJobRepository( + IDictionary counts, + IEnumerable jobs) + { + foreach (var pair in counts) + { + QueueCounts[pair.Key] = pair.Value; + } + + Jobs.AddRange(jobs); + } + + public Dictionary QueueCounts { get; } = new(); + public List Jobs { get; } = new(); + + public Task InsertAsync( + PolicyRunJob job, + IClientSessionHandle? session = null, + CancellationToken cancellationToken = default) + { + Jobs.Add(job); + return Task.CompletedTask; + } + + public Task CountAsync( + string tenantId, + PolicyRunMode mode, + IReadOnlyCollection statuses, + CancellationToken cancellationToken = default) + { + if (statuses is null || statuses.Count == 0) + { + return Task.FromResult(QueueCounts.Values.Sum()); + } + + long total = 0; foreach (var status in statuses) { if (QueueCounts.TryGetValue(status, out var count)) @@ -148,28 +290,50 @@ public sealed class PolicySimulationMetricsProviderTests IClientSessionHandle? session = null, CancellationToken cancellationToken = default) { - IReadOnlyList filtered = TerminalJobs; + IEnumerable query = Jobs; + if (statuses is { Count: > 0 }) { - filtered = TerminalJobs.Where(job => statuses.Contains(job.Status)).ToList(); + query = query.Where(job => statuses.Contains(job.Status)); } - return Task.FromResult(filtered); + if (queuedAfter is not null) + { + query = query.Where(job => (job.QueuedAt ?? job.CreatedAt) >= queuedAfter.Value); + } + + var result = query.Take(limit).ToList().AsReadOnly(); + return Task.FromResult>(result); } - public Task GetAsync(string tenantId, string jobId, IClientSessionHandle? session = null, CancellationToken cancellationToken = default) + public Task GetAsync( + string tenantId, + string jobId, + IClientSessionHandle? session = null, + CancellationToken cancellationToken = default) + => Task.FromResult(Jobs.FirstOrDefault(job => job.Id == jobId)); + + public Task GetByRunIdAsync( + string tenantId, + string runId, + IClientSessionHandle? session = null, + CancellationToken cancellationToken = default) + => Task.FromResult(Jobs.FirstOrDefault(job => string.Equals(job.RunId, runId, StringComparison.Ordinal))); + + public Task LeaseAsync( + string leaseOwner, + DateTimeOffset now, + TimeSpan leaseDuration, + int maxAttempts, + IClientSessionHandle? session = null, + CancellationToken cancellationToken = default) => Task.FromResult(null); - public Task GetByRunIdAsync(string tenantId, string runId, IClientSessionHandle? session = null, CancellationToken cancellationToken = default) - => Task.FromResult(null); - - public Task InsertAsync(PolicyRunJob job, IClientSessionHandle? session = null, CancellationToken cancellationToken = default) - => Task.CompletedTask; - - public Task LeaseAsync(string leaseOwner, DateTimeOffset now, TimeSpan leaseDuration, int maxAttempts, IClientSessionHandle? session = null, CancellationToken cancellationToken = default) - => Task.FromResult(null); - - public Task ReplaceAsync(PolicyRunJob job, string? expectedLeaseOwner = null, IClientSessionHandle? session = null, CancellationToken cancellationToken = default) + public Task ReplaceAsync( + PolicyRunJob job, + string? expectedLeaseOwner = null, + IClientSessionHandle? session = null, + CancellationToken cancellationToken = default) => Task.FromResult(true); } } diff --git a/src/Web/StellaOps.Web/TASKS.md b/src/Web/StellaOps.Web/TASKS.md index 65d07952d..5df14a912 100644 --- a/src/Web/StellaOps.Web/TASKS.md +++ b/src/Web/StellaOps.Web/TASKS.md @@ -5,6 +5,7 @@ > 2025-10-26: Introduced `StellaOps.Aoc` library with forbidden key list, guard result/options, and baseline write guard + tests. Middleware/analyzer wiring still pending. > 2025-10-30: Added `StellaOps.Aoc.AspNetCore` helpers (`AddAocGuard`, `AocHttpResults`) and switched Concelier WebService to the shared problem-details mapper; analyzer wiring remains pending. > 2025-10-30: Published `docs/aoc/guard-library.md` covering registration patterns, endpoint filters, and error mapping for ingestion services. +> 2025-11-06: Added `RequireAocGuard` route helper, wired Concelier advisory ingestion endpoint to the shared filter, refreshed docs, and introduced extension tests. | WEB-AOC-19-002 `Provenance & signature helpers` | TODO | BE-Base Platform Guild | WEB-AOC-19-001 | Ship `ProvenanceBuilder`, checksum utilities, and signature verification helper integrated with guard logging. Cover DSSE/CMS formats with unit tests. | | WEB-AOC-19-003 `Analyzer + test fixtures` | TODO | QA Guild, BE-Base Platform Guild | WEB-AOC-19-001 | Author Roslyn analyzer preventing ingestion modules from writing forbidden keys without guard, and provide shared test fixtures for guard validation used by Concelier/Excititor service tests. | > Docs alignment (2025-10-26): Analyzer expectations detailed in `docs/ingestion/aggregation-only-contract.md` §3/5; CI integration tracked via DEVOPS-AOC-19-001.