Compare commits

...

2 Commits

Author SHA1 Message Date
master
dd217b4546 feat: Implement approvals workflow and notifications integration
Some checks failed
Docs CI / lint-and-preview (push) Has been cancelled
- Added approvals orchestration with persistence and workflow scaffolding.
- Integrated notifications insights and staged resume hooks.
- Introduced approval coordinator and policy notification bridge with unit tests.
- Added approval decision API with resume requeue and persisted plan snapshots.
- Documented the Excitor consensus API beta and provided JSON sample payload.
- Created analyzers to flag usage of deprecated merge service APIs.
- Implemented logging for artifact uploads and approval decision service.
- Added tests for PackRunApprovalDecisionService and related components.
2025-11-06 08:48:13 +02:00
master
21a2759412 feat: Implement DevPortal Offline Export Job
- Added DevPortalOfflineJob to coordinate bundle construction, manifest signing, and artifact persistence.
- Introduced DevPortalOfflineWorkerOptions for configuration of the offline export job.
- Enhanced the Worker class to utilize DevPortalOfflineJob and handle execution based on configuration.
- Implemented HmacDevPortalOfflineManifestSigner for signing manifests with HMAC SHA256.
- Created FileSystemDevPortalOfflineObjectStore for storing artifacts in the file system.
- Updated appsettings.json to include configuration options for the DevPortal offline export.
- Added unit tests for DevPortalOfflineJob and HmacDevPortalOfflineManifestSigner to ensure functionality.
- Refactored existing tests to accommodate changes in method signatures and new dependencies.
2025-11-05 21:57:46 +02:00
118 changed files with 5024 additions and 2470 deletions

View File

@@ -16,6 +16,9 @@ Document follow-up actions for CONCELIER-CORE-AOC-19-004 as we unwind the final
1. Introduce a raw linkset projection alongside the existing canonical mapper so Policy Engine can choose which flavour to consume. ✅ 2025-10-31: `AdvisoryObservation` now surfaces `RawLinkset`; Mongo documents store both canonical & raw shapes; tests/goldens updated.
2. Update observation factory/query tests to assert duplicate handling and ordering with the relaxed projection. ✅ 2025-10-31.
3. Refresh docs (`docs/ingestion/aggregation-only-contract.md`) once behaviour lands to explain the “raw vs canonical linkset” split.
3. Refresh docs (`docs/ingestion/aggregation-only-contract.md`) once behaviour lands to explain the “raw vs canonical linkset” split. ✅ 2025-11-06: Added invariant notes and rollout guidance, linked to `docs/migration/no-merge.md` and `docs/dev/raw-linkset-backfill-plan.md`.
4. Coordinate with Policy Guild to validate consumers against the new raw projection before flipping defaults. ↺ Ongoing — see action items in `docs/dev/raw-linkset-backfill-plan.md` (2025-10-31 handshake with POLICY-ENGINE-20-003 owners).
- 2025-11-05: Catalogued residual normalization paths tied to the legacy Merge service and outlined `noMergeEnabled` feature-toggle work to keep AOC ingestion fully merge-free.
- 2025-11-05 19:20Z: Observation factory/linkset now preserve upstream ordering and duplicates; canonicalisation shifts to downstream services.
- 2025-11-06: Documented post-merge rollout plan and annotated sprint trackers with analyzer gating updates.

View File

@@ -1,6 +1,6 @@
# Raw Linkset Backfill & Adoption Plan
_Last updated: 2025-10-31_
_Last updated: 2025-11-06_
Owners: Concelier Storage Guild, DevOps Guild, Policy Guild
## Context
@@ -39,7 +39,7 @@ Owners: Concelier Storage Guild, DevOps Guild, Policy Guild
| 2025-10-31 | Handshake w/ Policy | Agreement to consume `rawLinkset`; this document created. |
| 2025-11-01 | Draft migration script | Validate against staging dataset snapshots. |
| 2025-11-04 | Storage task CONCELIER-STORE-AOC-19-005 due | Deliver script + runbook for review. |
| 2025-11-06 | Staging backfill rehearsal | Target < 30 min runtime on 5M observations. |
| 2025-11-06 | Staging backfill rehearsal | Target < 30 min runtime on 5M observations; docs refreshed to highlight raw vs canonical invariants and analyzer guardrails. |
| 2025-11-08 | Policy fixtures updated | POL engine branch consumes `rawLinkset`. |
| 2025-11-11 | Production rollout window | Pending DevOps sign-off after rehearsals. |

View File

@@ -91,7 +91,7 @@ CONCELIER-ATTEST-73-002 `Transparency metadata` | TODO | Ensure Conseiller expos
CONCELIER-CONSOLE-23-001 `Advisory aggregation views` | TODO | Expose `/console/advisories` endpoints returning aggregation groups (per linkset) with source chips, provider-reported severity columns (no local consensus), and provenance metadata for Console list + dashboard cards. Support filters by source, ecosystem, published/modified window, tenant enforcement. Dependencies: CONCELIER-LNM-21-201, CONCELIER-LNM-21-202. | Concelier WebService Guild, BE-Base Platform Guild (src/Concelier/StellaOps.Concelier.WebService/TASKS.md)
CONCELIER-CONSOLE-23-002 `Dashboard deltas API` | TODO | Provide aggregated advisory delta counts (new, modified, conflicting) for Console dashboard + live status ticker; emit structured events for queue lag metrics. Ensure deterministic counts across repeated queries. Dependencies: CONCELIER-CONSOLE-23-001, CONCELIER-LNM-21-203. | Concelier WebService Guild (src/Concelier/StellaOps.Concelier.WebService/TASKS.md)
CONCELIER-CONSOLE-23-003 `Search fan-out helpers` | TODO | Deliver fast lookup endpoints for CVE/GHSA/purl search (linksets, observations) returning evidence fragments for Console global search; implement caching + scope guards. Dependencies: CONCELIER-CONSOLE-23-001. | Concelier WebService Guild (src/Concelier/StellaOps.Concelier.WebService/TASKS.md)
CONCELIER-CORE-AOC-19-004 `Remove ingestion normalization` | DOING (2025-10-28) | Strip normalization/dedup/severity logic from ingestion pipelines, delegate derived computations to Policy Engine, and update exporters/tests to consume raw documents only.<br>2025-10-29 19:05Z: Audit completed for `AdvisoryRawService`/Mongo repo to confirm alias order/dedup removal persists; identified remaining normalization in observation/linkset factory that will be revised to surface raw duplicates for Policy ingestion. Change sketch + regression matrix drafted under `docs/dev/aoc-normalization-removal-notes.md` (pending commit).<br>2025-10-31 20:45Z: Added raw linkset projection to observations/storage, exposing canonical+raw views, refreshed fixtures/tests, and documented behaviour in models/doc factory.<br>2025-10-31 21:10Z: Coordinated with Policy Engine (POLICY-ENGINE-20-003) on adoption timeline; backfill + consumer readiness tracked in `docs/dev/raw-linkset-backfill-plan.md`. Dependencies: CONCELIER-CORE-AOC-19-002, POLICY-AOC-19-003. | Concelier Core Guild (src/Concelier/__Libraries/StellaOps.Concelier.Core/TASKS.md)
CONCELIER-CORE-AOC-19-004 `Remove ingestion normalization` | DOING (2025-10-28) | Strip normalization/dedup/severity logic from ingestion pipelines, delegate derived computations to Policy Engine, and update exporters/tests to consume raw documents only.<br>2025-10-29 19:05Z: Audit completed for `AdvisoryRawService`/Mongo repo to confirm alias order/dedup removal persists; identified remaining normalization in observation/linkset factory that will be revised to surface raw duplicates for Policy ingestion. Change sketch + regression matrix drafted under `docs/dev/aoc-normalization-removal-notes.md` (pending commit).<br>2025-10-31 20:45Z: Added raw linkset projection to observations/storage, exposing canonical+raw views, refreshed fixtures/tests, and documented behaviour in models/doc factory.<br>2025-10-31 21:10Z: Coordinated with Policy Engine (POLICY-ENGINE-20-003) on adoption timeline; backfill + consumer readiness tracked in `docs/dev/raw-linkset-backfill-plan.md`.<br>2025-11-05 14:20Z: Resumed work to map remaining normalization hooks tied to Merge service and capture requirements for the upcoming `noMergeEnabled` feature toggle.<br>2025-11-05 19:05Z: Hardened no-merge feature flag wiring by suppressing obsolete diagnostics and extending gating tests.<br>2025-11-06 16:10Z: Updated AOC references/backfill plan with raw-vs-canonical guidance and noted analyzer guardrails introduced under MERGE-LNM-21-002. Dependencies: CONCELIER-CORE-AOC-19-002, POLICY-AOC-19-003. | Concelier Core Guild (src/Concelier/__Libraries/StellaOps.Concelier.Core/TASKS.md)
CONCELIER-CORE-AOC-19-013 `Authority tenant scope smoke coverage` | TODO | Extend Concelier smoke/e2e fixtures to configure `requiredTenants` and assert cross-tenant rejection with updated Authority tokens. Dependencies: AUTH-AOC-19-002. | Concelier Core Guild (src/Concelier/__Libraries/StellaOps.Concelier.Core/TASKS.md)
@@ -210,7 +210,7 @@ Depends on: Sprint 110.B - Concelier.VI
Summary: Ingestion & Evidence focus on Concelier (phase VII).
Task ID | State | Task description | Owners (Source)
--- | --- | --- | ---
MERGE-LNM-21-002 | DOING (2025-11-03) | Refactor or retire `AdvisoryMergeService` and related pipelines, ensuring callers transition to observation/linkset APIs; add compile-time analyzer preventing merge service usage.<br>2025-11-03: Began dependency audit and call-site inventory ahead of deprecation plan; cataloging service registrations/tests referencing merge APIs. | BE-Merge (src/Concelier/__Libraries/StellaOps.Concelier.Merge/TASKS.md)
MERGE-LNM-21-002 | DOING (2025-11-03) | Refactor or retire `AdvisoryMergeService` and related pipelines, ensuring callers transition to observation/linkset APIs; add compile-time analyzer preventing merge service usage.<br>2025-11-03: Began dependency audit and call-site inventory ahead of deprecation plan; cataloging service registrations/tests referencing merge APIs.<br>2025-11-05 14:42Z: Drafting `concelier:features:noMergeEnabled` gating, merge job allowlist handling, and deprecation/telemetry changes prior to analyzer rollout.<br>2025-11-06 16:10Z: Landed analyzer project (`CONCELIER0002`), wired into Concelier WebService/tests, and updated docs to direct suppressions through explicit migration notes. | BE-Merge (src/Concelier/__Libraries/StellaOps.Concelier.Merge/TASKS.md)
MERGE-LNM-21-003 Determinism/test updates | QA Guild, BE-Merge | Replace merge determinism suites with observation/linkset regression tests verifying no data mutation and conflicts remain visible. Dependencies: MERGE-LNM-21-002. | MERGE-LNM-21-002 (src/Concelier/__Libraries/StellaOps.Concelier.Merge/TASKS.md)
@@ -227,7 +227,7 @@ EXCITITOR-AIRGAP-56-002 `Bundle provenance` | TODO | Persist bundle metadata on
EXCITITOR-AIRGAP-57-001 `Sealed-mode enforcement` | TODO | Block non-mirror connectors in sealed mode and surface remediation errors. Dependencies: EXCITITOR-AIRGAP-56-002. | Excititor Core Guild, AirGap Policy Guild (src/Excititor/__Libraries/StellaOps.Excititor.Core/TASKS.md)
EXCITITOR-AIRGAP-57-002 `Staleness annotations` | TODO | Annotate VEX statements with staleness metrics and expose via API. Dependencies: EXCITITOR-AIRGAP-57-001. | Excititor Core Guild, AirGap Time Guild (src/Excititor/__Libraries/StellaOps.Excititor.Core/TASKS.md)
EXCITITOR-AIRGAP-58-001 `Portable VEX evidence` | TODO | Package VEX evidence segments into portable evidence bundles linked to timeline. Dependencies: EXCITITOR-AIRGAP-57-002. | Excititor Core Guild, Evidence Locker Guild (src/Excititor/__Libraries/StellaOps.Excititor.Core/TASKS.md)
EXCITITOR-ATTEST-01-003 Verification suite & observability | Team Excititor Attestation | DOING (2025-10-22) Continuing implementation: build `IVexAttestationVerifier`, wire metrics/logging, and add regression tests. Draft plan in `EXCITITOR-ATTEST-01-003-plan.md` (2025-10-19) guides scope; updating with worknotes as progress lands.<br>2025-10-31: Verifier now tolerates duplicate source providers from AOC raw projections, downgrades offline Rekor verification to a degraded result, and enforces trusted signer registry checks with detailed diagnostics/tests. | EXCITITOR-ATTEST-01-002 (src/Excititor/__Libraries/StellaOps.Excititor.Attestation/TASKS.md)
EXCITITOR-ATTEST-01-003 Verification suite & observability | Team Excititor Attestation | DOING (2025-10-22) Continuing implementation: build `IVexAttestationVerifier`, wire metrics/logging, and add regression tests. Draft plan in `EXCITITOR-ATTEST-01-003-plan.md` (2025-10-19) guides scope; updating with worknotes as progress lands.<br>2025-10-31: Verifier now tolerates duplicate source providers from AOC raw projections, downgrades offline Rekor verification to a degraded result, and enforces trusted signer registry checks with detailed diagnostics/tests.<br>2025-11-05 14:35Z: Resuming with diagnostics/observability deliverables (typed diagnostics record, ActivitySource wiring, metrics dimensions) before WebService/Worker integration. | EXCITITOR-ATTEST-01-002 (src/Excititor/__Libraries/StellaOps.Excititor.Attestation/TASKS.md)
EXCITITOR-ATTEST-73-001 `VEX attestation payloads` | TODO | Provide VEX statement metadata (supplier identity, justification, scope) required for VEXAttestation payloads. Dependencies: EXCITITOR-ATTEST-01-003. | Excititor Core Guild, Attestation Payloads Guild (src/Excititor/__Libraries/StellaOps.Excititor.Core/TASKS.md)
EXCITITOR-ATTEST-73-002 `Chain provenance` | TODO | Expose linkage from VEX statements to subject/product for chain of custody graph. Dependencies: EXCITITOR-ATTEST-73-001. | Excititor Core Guild (src/Excititor/__Libraries/StellaOps.Excititor.Core/TASKS.md)
EXCITITOR-CONN-MS-01-003 Trust metadata & provenance hints | Team Excititor Connectors MSRC | TODO Emit cosign/AAD issuer metadata, attach provenance details, and document policy integration. | EXCITITOR-CONN-MS-01-002, EXCITITOR-POLICY-01-001 (src/Excititor/__Libraries/StellaOps.Excititor.Connectors.MSRC.CSAF/TASKS.md)

View File

@@ -134,8 +134,8 @@ Summary: Scanner & Surface focus on Scanner (phase VII).
Task ID | State | Task description | Owners (Source)
--- | --- | --- | ---
SCANNER-ENTRYTRACE-18-504 | TODO | Emit EntryTrace AOC NDJSON (`entrytrace.entry/node/edge/target/warning/capability`) and wire CLI/service streaming outputs. Dependencies: SCANNER-ENTRYTRACE-18-503. | EntryTrace Guild (src/Scanner/__Libraries/StellaOps.Scanner.EntryTrace/TASKS.md)
SCANNER-ENV-01 | DOING (2025-11-02) | Replace ad-hoc environment reads with `StellaOps.Scanner.Surface.Env` helpers for cache roots and CAS endpoints.<br>2025-11-02: Env helper wiring drafted for Worker startup; initial tests validate cache root resolution. | Scanner Worker Guild (src/Scanner/StellaOps.Scanner.Worker/TASKS.md)
SCANNER-ENV-02 | DOING (2025-11-02) | Wire Surface.Env helpers into WebService hosting (cache roots, feature flags) and document configuration. Dependencies: SCANNER-ENV-01.<br>2025-11-02: WebService bootstrap now consumes Surface.Env helpers for cache roots and feature flag toggles; configuration doc draft pending. | Scanner WebService Guild, Ops Guild (src/Scanner/StellaOps.Scanner.WebService/TASKS.md)
SCANNER-ENV-01 | DOING (2025-11-02) | Replace ad-hoc environment reads with `StellaOps.Scanner.Surface.Env` helpers for cache roots and CAS endpoints.<br>2025-11-02: Env helper wiring drafted for Worker startup; initial tests validate cache root resolution.<br>2025-11-05 14:55Z: Continuing integration by propagating resolved settings into cache/secret services and prepping worker smoke tests + docs updates.<br>2025-11-05 19:18Z: Bound `SurfaceCacheOptions` root to Surface.Env settings and added configurator unit coverage.<br>2025-11-06 17:05Z: Documented misconfiguration warnings and updated module README to highlight Surface.Env usage. | Scanner Worker Guild (src/Scanner/StellaOps.Scanner.Worker/TASKS.md)
SCANNER-ENV-02 | DOING (2025-11-02) | Wire Surface.Env helpers into WebService hosting (cache roots, feature flags) and document configuration. Dependencies: SCANNER-ENV-01.<br>2025-11-02: WebService bootstrap now consumes Surface.Env helpers for cache roots and feature flag toggles; configuration doc draft pending.<br>2025-11-05 14:55Z: Picking up configuration/documentation work and aligning API readiness checks with Surface.Env validation outputs.<br>2025-11-05 19:18Z: Added unit test for Surface.Env cache root binding and ensured configurator registration.<br>2025-11-06 17:05Z: Surface.Env design doc expanded with warning catalogue and release notes, README refreshed. | Scanner WebService Guild, Ops Guild (src/Scanner/StellaOps.Scanner.WebService/TASKS.md)
SCANNER-ENV-03 | TODO | Adopt Surface.Env helpers for plugin configuration (cache roots, CAS endpoints, feature toggles). Dependencies: SCANNER-ENV-02. | BuildX Plugin Guild (src/Scanner/StellaOps.Scanner.Sbomer.BuildXPlugin/TASKS.md)
SCANNER-EVENTS-16-301 | BLOCKED (2025-10-26) | Emit orchestrator-compatible envelopes (`scanner.event.*`) and update integration tests to verify Notifier ingestion (no Redis queue coupling). | Scanner WebService Guild (src/Scanner/StellaOps.Scanner.WebService/TASKS.md)
SCANNER-EVENTS-16-302 | DOING (2025-10-26) | Extend orchestrator event links (report/policy/attestation) once endpoints are finalised across gateway + console. Dependencies: SCANNER-EVENTS-16-301. | Scanner WebService Guild (src/Scanner/StellaOps.Scanner.WebService/TASKS.md)

View File

@@ -82,7 +82,8 @@ Task ID | State | Task description | Owners (Source)
--- | --- | --- | ---
SCHED-CONSOLE-23-001 | DONE (2025-11-03) | Extend runs APIs with live progress SSE endpoints (`/console/runs/{id}/stream`), queue lag summaries, diff metadata fetch, retry/cancel hooks with RBAC enforcement, and deterministic pagination for history views consumed by Console. | Scheduler WebService Guild, BE-Base Platform Guild (src/Scheduler/StellaOps.Scheduler.WebService/TASKS.md)
SCHED-CONSOLE-27-001 | DONE (2025-11-03) | Provide policy batch simulation orchestration endpoints (`/policies/simulations` POST/GET) exposing run creation, shard status, SSE progress, cancellation, and retries with RBAC enforcement. Dependencies: SCHED-CONSOLE-23-001. | Scheduler WebService Guild, Policy Registry Guild (src/Scheduler/StellaOps.Scheduler.WebService/TASKS.md)
SCHED-CONSOLE-27-002 | DOING (2025-11-03) | Emit telemetry endpoints/metrics (`policy_simulation_queue_depth`, `policy_simulation_latency`) and webhook callbacks for completion/failure consumed by Registry. Dependencies: SCHED-CONSOLE-27-001. | Scheduler WebService Guild, Observability Guild (src/Scheduler/StellaOps.Scheduler.WebService/TASKS.md)
SCHED-CONSOLE-27-002 | DOING (2025-11-03) | Emit telemetry endpoints/metrics (`policy_simulation_queue_depth`, `policy_simulation_latency`) and webhook callbacks for completion/failure consumed by Registry. Dependencies: SCHED-CONSOLE-27-001. | Scheduler WebService Guild, Observability Guild (src/Scheduler/StellaOps.Scheduler.WebService/TASKS.md)
> 2025-11-06: Tagged `policy_simulation_queue_depth` metrics with tenant identifiers and added unit coverage for the metrics provider snapshot.
SCHED-IMPACT-16-303 | TODO | Snapshot/compaction + invalidation for removed images; persistence to RocksDB/Redis per architecture. | Scheduler ImpactIndex Guild (src/Scheduler/__Libraries/StellaOps.Scheduler.ImpactIndex/TASKS.md)
SCHED-SURFACE-01 | TODO | Evaluate Surface.FS pointers when planning delta scans to avoid redundant work and prioritise drift-triggered assets. | Scheduler Worker Guild (src/Scheduler/__Libraries/StellaOps.Scheduler.Worker/TASKS.md)
SCHED-VULN-29-001 | TODO | Expose resolver job APIs (`POST /vuln/resolver/jobs`, `GET /vuln/resolver/jobs/{id}`) to trigger candidate recomputation per artifact/policy change with RBAC and rate limits. | Scheduler WebService Guild, Findings Ledger Guild (src/Scheduler/StellaOps.Scheduler.WebService/TASKS.md)

View File

@@ -19,7 +19,7 @@ Depends on: Sprint 110.A - AdvisoryAI, Sprint 120.A - AirGap, Sprint 130.A - Sca
Summary: Export & Evidence focus on ExportCenter (phase I).
Task ID | State | Task description | Owners (Source)
--- | --- | --- | ---
DVOFF-64-001 | DOING (2025-11-04) | Implement Export Center job `devportal --offline` bundling portal HTML, specs, SDK artifacts, changelogs, and verification manifest. | DevPortal Offline Guild, Exporter Guild (src/ExportCenter/StellaOps.ExportCenter.DevPortalOffline/TASKS.md)
DVOFF-64-001 | DONE (2025-11-05) | Implement Export Center job `devportal --offline` bundling portal HTML, specs, SDK artifacts, changelogs, and verification manifest.<br>2025-11-05: Worker builds reproducible bundle, persists manifest/checksum/DSSE signature under `<prefix>/<bundleId>/`, and documents verification flow in `devportal-offline.md`. Unit coverage added for job + signer. | DevPortal Offline Guild, Exporter Guild (src/ExportCenter/StellaOps.ExportCenter.DevPortalOffline/TASKS.md)
DVOFF-64-002 | TODO | Provide verification CLI (`stella devportal verify bundle.tgz`) ensuring integrity before import. Dependencies: DVOFF-64-001. | DevPortal Offline Guild, AirGap Controller Guild (src/ExportCenter/StellaOps.ExportCenter.DevPortalOffline/TASKS.md)
EXPORT-AIRGAP-56-001 | TODO | Extend Export Center to build Mirror Bundles as export profiles, including advisories/VEX/policy packs manifesting DSSE/TUF metadata. | Exporter Service Guild, Mirror Creator Guild (src/ExportCenter/StellaOps.ExportCenter/TASKS.md)
EXPORT-AIRGAP-56-002 | TODO | Package Bootstrap Pack (images + charts) into OCI archives with signed manifests for air-gapped deployment. Dependencies: EXPORT-AIRGAP-56-001. | Exporter Service Guild, DevOps Guild (src/ExportCenter/StellaOps.ExportCenter/TASKS.md)

View File

@@ -62,6 +62,7 @@ CLI-PARITY-41-002 | TODO | Implement `notify`, `aoc`, `auth` command groups, ide
CLI-POLICY-20-001 | TODO | Add `stella policy new | DevEx/CLI Guild (src/Cli/StellaOps.Cli/TASKS.md)
CLI-POLICY-23-004 | TODO | Add `stella policy lint` command validating SPL files with compiler diagnostics; support JSON output. Dependencies: CLI-POLICY-20-001. | DevEx/CLI Guild (src/Cli/StellaOps.Cli/TASKS.md)
CLI-POLICY-23-005 | DOING (2025-10-28) | Implement `stella policy activate` with scheduling window, approval enforcement, and summary output. Dependencies: CLI-POLICY-23-004. | DevEx/CLI Guild (src/Cli/StellaOps.Cli/TASKS.md)
> 2025-11-06: CLI enforces `--version` as mandatory and adds scheduled activation timestamp normalization tests while keeping exit codes intact.
CLI-POLICY-23-006 | TODO | Provide `stella policy history` and `stella policy explain` commands to pull run history and explanation trees. Dependencies: CLI-POLICY-23-005. | DevEx/CLI Guild (src/Cli/StellaOps.Cli/TASKS.md)
CLI-POLICY-27-001 | TODO | Implement policy workspace commands (`stella policy init`, `edit`, `lint`, `compile`, `test`) with template selection, local cache, JSON output, and deterministic temp directories. Dependencies: CLI-POLICY-23-006. | DevEx/CLI Guild (src/Cli/StellaOps.Cli/TASKS.md)

View File

@@ -265,7 +265,7 @@ Depends on: Sprint 100.A - Attestor, Sprint 110.A - AdvisoryAI, Sprint 120.A - A
Summary: Documentation & Process focus on Docs Modules Attestor).
Task ID | State | Task description | Owners (Source)
--- | --- | --- | ---
ATTESTOR-DOCS-0001 | DOING (2025-10-29) | See ./AGENTS.md | Docs Guild (docs/modules/attestor/TASKS.md)
ATTESTOR-DOCS-0001 | DONE (2025-11-05) | README updated with platform-events release (attestor.logged@1 canonical samples, schema validation notes). | Docs Guild (docs/modules/attestor/TASKS.md)
ATTESTOR-ENG-0001 | TODO | Update status via ./AGENTS.md workflow | Module Team (docs/modules/attestor/TASKS.md)
ATTESTOR-OPS-0001 | TODO | Sync outcomes back to ../../TASKS.md | Ops Guild (docs/modules/attestor/TASKS.md)
@@ -305,7 +305,7 @@ Depends on: Sprint 100.A - Attestor, Sprint 110.A - AdvisoryAI, Sprint 120.A - A
Summary: Documentation & Process focus on Docs Modules Concelier).
Task ID | State | Task description | Owners (Source)
--- | --- | --- | ---
CONCELIER-DOCS-0001 | DOING (2025-10-29) | See ./AGENTS.md | Docs Guild (docs/modules/concelier/TASKS.md)
CONCELIER-DOCS-0001 | DONE (2025-11-05) | README updated to reference 2025-10-22 authority toggle rollout guidance (quickstart + authority audit runbook). | Docs Guild (docs/modules/concelier/TASKS.md)
CONCELIER-ENG-0001 | TODO | Update status via ./AGENTS.md workflow | Module Team (docs/modules/concelier/TASKS.md)
CONCELIER-OPS-0001 | TODO | Sync outcomes back to ../../TASKS.md | Ops Guild (docs/modules/concelier/TASKS.md)
@@ -325,7 +325,7 @@ Depends on: Sprint 100.A - Attestor, Sprint 110.A - AdvisoryAI, Sprint 120.A - A
Summary: Documentation & Process focus on Docs Modules Excititor).
Task ID | State | Task description | Owners (Source)
--- | --- | --- | ---
EXCITITOR-DOCS-0001 | TODO | See ./AGENTS.md | Docs Guild (docs/modules/excititor/TASKS.md)
EXCITITOR-DOCS-0001 | DONE (2025-11-05) | README updated with consensus API beta note ([docs/updates/2025-11-05-excitor-consensus-beta.md](../updates/2025-11-05-excitor-consensus-beta.md)) and consensus JSON sample ([docs/vex/consensus-json.md](../vex/consensus-json.md)). | Docs Guild (docs/modules/excititor/TASKS.md)
EXCITITOR-ENG-0001 | TODO | Update status via ./AGENTS.md workflow | Module Team (docs/modules/excititor/TASKS.md)
EXCITITOR-OPS-0001 | TODO | Sync outcomes back to ../../TASKS.md | Ops Guild (docs/modules/excititor/TASKS.md)
@@ -335,7 +335,7 @@ Depends on: Sprint 100.A - Attestor, Sprint 110.A - AdvisoryAI, Sprint 120.A - A
Summary: Documentation & Process focus on Docs Modules Export Center).
Task ID | State | Task description | Owners (Source)
--- | --- | --- | ---
EXPORT CENTER-DOCS-0001 | DOING (2025-10-29) | See ./AGENTS.md | Docs Guild (docs/modules/export-center/TASKS.md)
EXPORT CENTER-DOCS-0001 | DONE (2025-11-05) | README updated to cover devportal offline profile, DSSE manifest signature, and links to provenance docs per 2025-10-29 export-center release update. | Docs Guild (docs/modules/export-center/TASKS.md)
EXPORT CENTER-ENG-0001 | TODO | Update status via ./AGENTS.md workflow | Module Team (docs/modules/export-center/TASKS.md)
EXPORT CENTER-OPS-0001 | TODO | Sync outcomes back to ../../TASKS.md | Ops Guild (docs/modules/export-center/TASKS.md)
@@ -367,7 +367,7 @@ Depends on: Sprint 100.A - Attestor, Sprint 110.A - AdvisoryAI, Sprint 120.A - A
Summary: Documentation & Process focus on Docs Modules Orchestrator).
Task ID | State | Task description | Owners (Source)
--- | --- | --- | ---
SOURCE---JOB-ORCHESTRATOR-DOCS-0001 | DOING (2025-10-29) | Align with ./AGENTS.md | Docs Guild (docs/modules/orchestrator/TASKS.md)
SOURCE---JOB-ORCHESTRATOR-DOCS-0001 | DONE (2025-11-05) | README reflects the 2025-11-01 Authority quota/backfill scope release and auditing requirements. | Docs Guild (docs/modules/orchestrator/TASKS.md)
SOURCE---JOB-ORCHESTRATOR-ENG-0001 | TODO | Sync into ../../TASKS.md | Module Team (docs/modules/orchestrator/TASKS.md)
SOURCE---JOB-ORCHESTRATOR-OPS-0001 | TODO | Document outputs in ./README.md | Ops Guild (docs/modules/orchestrator/TASKS.md)
@@ -407,7 +407,7 @@ Depends on: Sprint 100.A - Attestor, Sprint 110.A - AdvisoryAI, Sprint 120.A - A
Summary: Documentation & Process focus on Docs Modules Scanner).
Task ID | State | Task description | Owners (Source)
--- | --- | --- | ---
SCANNER-DOCS-0001 | DOING (2025-10-29) | See ./AGENTS.md | Docs Guild (docs/modules/scanner/TASKS.md)
SCANNER-DOCS-0001 | DONE (2025-11-05) | README updated with the 2025-10-19 platform-events release (scanner.report.ready@1 / scan.completed@1 DSSE envelopes + samples). | Docs Guild (docs/modules/scanner/TASKS.md)
SCANNER-ENG-0001 | TODO | Update status via ./AGENTS.md workflow | Module Team (docs/modules/scanner/TASKS.md)
SCANNER-OPS-0001 | TODO | Sync outcomes back to ../../TASKS.md | Ops Guild (docs/modules/scanner/TASKS.md)
@@ -427,7 +427,7 @@ Depends on: Sprint 100.A - Attestor, Sprint 110.A - AdvisoryAI, Sprint 120.A - A
Summary: Documentation & Process focus on Docs Modules Signer).
Task ID | State | Task description | Owners (Source)
--- | --- | --- | ---
SIGNER-DOCS-0001 | DOING (2025-10-29) | See ./AGENTS.md | Docs Guild (docs/modules/signer/TASKS.md)
SIGNER-DOCS-0001 | DONE (2025-11-05) | README updated with Sprint 11 signing-chain release details (sign/dsse, verify/referrers, quota enforcement). | Docs Guild (docs/modules/signer/TASKS.md)
SIGNER-ENG-0001 | TODO | Update status via ./AGENTS.md workflow | Module Team (docs/modules/signer/TASKS.md)
SIGNER-OPS-0001 | TODO | Sync outcomes back to ../../TASKS.md | Ops Guild (docs/modules/signer/TASKS.md)
@@ -437,7 +437,7 @@ Depends on: Sprint 100.A - Attestor, Sprint 110.A - AdvisoryAI, Sprint 120.A - A
Summary: Documentation & Process focus on Docs Modules Telemetry).
Task ID | State | Task description | Owners (Source)
--- | --- | --- | ---
TELEMETRY-DOCS-0001 | DOING (2025-10-29) | See ./AGENTS.md | Docs Guild (docs/modules/telemetry/TASKS.md)
TELEMETRY-DOCS-0001 | DONE (2025-11-05) | README updated with Sprint 23 console security alert pack (console-security Grafana board + burn-rate alert). | Docs Guild (docs/modules/telemetry/TASKS.md)
TELEMETRY-ENG-0001 | TODO | Update status via ./AGENTS.md workflow | Module Team (docs/modules/telemetry/TASKS.md)
TELEMETRY-OPS-0001 | TODO | Sync outcomes back to ../../TASKS.md | Ops Guild (docs/modules/telemetry/TASKS.md)
@@ -478,7 +478,7 @@ Depends on: Sprint 100.A - Attestor, Sprint 110.A - AdvisoryAI, Sprint 120.A - A
Summary: Documentation & Process focus on Docs Modules Vuln Explorer).
Task ID | State | Task description | Owners (Source)
--- | --- | --- | ---
VULNERABILITY-EXPLORER-DOCS-0001 | DOING (2025-10-29) | Align with ./AGENTS.md | Docs Guild (docs/modules/vuln-explorer/TASKS.md)
VULNERABILITY-EXPLORER-DOCS-0001 | DONE (2025-11-05) | README updated with 2025-11-03 access-controls release (attachment signing tokens + Authority scope guidance). | Docs Guild (docs/modules/vuln-explorer/TASKS.md)
VULNERABILITY-EXPLORER-ENG-0001 | TODO | Sync into ../../TASKS.md | Module Team (docs/modules/vuln-explorer/TASKS.md)
VULNERABILITY-EXPLORER-OPS-0001 | TODO | Document outputs in ./README.md | Ops Guild (docs/modules/vuln-explorer/TASKS.md)

View File

@@ -1,181 +1,181 @@
# Aggregation-Only Contract Reference
> The Aggregation-Only Contract (AOC) is the governing rule set that keeps StellaOps ingestion services deterministic, policy-neutral, and auditable. It applies to Concelier, Excititor, and any future collectors that write raw advisory or VEX documents.
## 1. Purpose and Scope
- Defines the canonical behaviour for `advisory_raw` and `vex_raw` collections and the linkset hints they may emit.
- Applies to every ingestion runtime (`StellaOps.Concelier.*`, `StellaOps.Excititor.*`), the Authority scopes that guard them, and the DevOps/QA surfaces that verify compliance.
- Complements the high-level architecture in [Concelier](../modules/concelier/architecture.md) and Authority enforcement documented in [Authority Architecture](../modules/authority/architecture.md).
# Aggregation-Only Contract Reference
> The Aggregation-Only Contract (AOC) is the governing rule set that keeps StellaOps ingestion services deterministic, policy-neutral, and auditable. It applies to Concelier, Excititor, and any future collectors that write raw advisory or VEX documents.
## 1. Purpose and Scope
- Defines the canonical behaviour for `advisory_raw` and `vex_raw` collections and the linkset hints they may emit.
- Applies to every ingestion runtime (`StellaOps.Concelier.*`, `StellaOps.Excititor.*`), the Authority scopes that guard them, and the DevOps/QA surfaces that verify compliance.
- Complements the high-level architecture in [Concelier](../modules/concelier/architecture.md) and Authority enforcement documented in [Authority Architecture](../modules/authority/architecture.md).
- Paired guidance: see the guard-rail checkpoints in [AOC Guardrails](../aoc/aoc-guardrails.md), the implementation reference in [AOC Guard Library](../aoc/guard-library.md), and CLI usage that will land in `/docs/modules/cli/guides/` as part of Sprint 19 follow-up.
## 2. Philosophy and Goals
- Preserve upstream truth: ingestion only captures immutable raw facts plus provenance, never derived severity or policy decisions.
- Defer interpretation: Policy Engine and downstream overlays remain the sole writers of materialised findings, severity, consensus, or risk scores.
- Make every write explainable: provenance, signatures, and content hashes are required so operators can prove where each fact originated.
- Keep outputs reproducible: identical inputs must yield identical documents, hashes, and linksets across replays and air-gapped installs.
## 3. Contract Invariants
| # | Invariant | What it forbids or requires | Enforcement surfaces |
|---|-----------|-----------------------------|----------------------|
| 1 | No derived severity at ingest | Reject top-level keys such as `severity`, `cvss`, `effective_status`, `consensus_provider`, `risk_score`. Raw upstream CVSS remains inside `content.raw`. | Mongo schema validator, `AOCWriteGuard`, Roslyn analyzer, `stella aoc verify`. |
| 2 | No merges or opinionated dedupe | Each upstream document persists on its own; ingestion never collapses multiple vendors into one document. | Repository interceptors, unit/fixture suites. |
| 3 | Provenance is mandatory | `source.*`, `upstream.*`, and `signature` metadata must be present; missing provenance triggers `ERR_AOC_004`. | Schema validator, guard, CLI verifier. |
| 4 | Idempotent upserts | Writes keyed by `(vendor, upstream_id, content_hash)` either no-op or insert a new revision with `supersedes`. Duplicate hashes map to the same document. | Repository guard, storage unique index, CI smoke tests. |
| 5 | Append-only revisions | Updates create a new document with `supersedes` pointer; no in-place mutation of content. | Mongo schema (`supersedes` format), guard, data migration scripts. |
| 6 | Linkset only | Ingestion may compute link hints (`purls`, `cpes`, IDs) to accelerate joins, but must not transform or infer severity or policy. Observations now persist both canonical linksets (for indexed queries) and raw linksets (preserving upstream order/duplicates) so downstream policy can decide how to normalise. | Linkset builders reviewed via fixtures/analyzers; raw-vs-canonical parity covered by observation fixtures. |
| 7 | Policy-only effective findings | Only Policy Engine identities can write `effective_finding_*`; ingestion callers receive `ERR_AOC_006` if they attempt it. | Authority scopes, Policy Engine guard. |
| 8 | Schema safety | Unknown top-level keys reject with `ERR_AOC_007`; timestamps use ISO 8601 UTC strings; tenant is required. | Mongo validator, JSON schema tests. |
| 9 | Clock discipline | Collectors stamp `fetched_at` and `received_at` monotonically per batch to support reproducibility windows. | Collector contracts, QA fixtures. |
## 4. Raw Schemas
### 4.1 `advisory_raw`
| Field | Type | Notes |
|-------|------|-------|
| `_id` | string | `advisory_raw:{source}:{upstream_id}:{revision}`; deterministic and tenant-scoped. |
| `tenant` | string | Required; injected by Authority middleware and asserted by schema validator. |
| `source.vendor` | string | Provider identifier (e.g., `redhat`, `osv`, `ghsa`). |
| `source.stream` | string | Connector stream name (`csaf`, `osv`, etc.). |
| `source.api` | string | Absolute URI of upstream document; stored for traceability. |
| `source.collector_version` | string | Semantic version of the collector. |
| `upstream.upstream_id` | string | Vendor- or ecosystem-provided identifier (CVE, GHSA, vendor ID). |
| `upstream.document_version` | string | Upstream issued timestamp or revision string. |
| `upstream.fetched_at` / `received_at` | string | ISO 8601 UTC timestamps recorded by the collector. |
| `upstream.content_hash` | string | `sha256:` digest of the raw payload used for idempotency. |
| `upstream.signature` | object | Required structure storing `present`, `format`, `key_id`, `sig`; even unsigned payloads set `present: false`. |
| `content.format` | string | Source format (`CSAF`, `OSV`, etc.). |
| `content.spec_version` | string | Upstream spec version when known. |
| `content.raw` | object | Full upstream payload, untouched except for transport normalisation. |
| `identifiers` | object | Upstream identifiers (`cve`, `ghsa`, `aliases`, etc.) captured as provided (trimmed, order preserved, duplicates allowed). |
| `linkset` | object | Join hints (see section 4.3). |
| `supersedes` | string or null | Points to previous revision of same upstream doc when content hash changes. |
### 4.2 `vex_raw`
| Field | Type | Notes |
|-------|------|-------|
| `_id` | string | `vex_raw:{source}:{upstream_id}:{revision}`. |
| `tenant` | string | Required; matches advisory collection requirements. |
| `source.*` | object | Same shape and requirements as `advisory_raw`. |
| `upstream.*` | object | Includes `document_version`, timestamps, `content_hash`, and `signature`. |
| `content.format` | string | Typically `CycloneDX-VEX` or `CSAF-VEX`. |
| `content.raw` | object | Entire upstream VEX payload. |
| `identifiers.statements` | array | Normalised statement summaries (IDs, PURLs, status, justification) to accelerate policy joins. |
| `linkset` | object | CVEs, GHSA IDs, and PURLs referenced in the document. |
| `supersedes` | string or null | Same convention as advisory documents. |
### 4.3 Linkset Fields
- `purls`: fully qualified Package URLs extracted from raw ranges or product nodes.
- `cpes`: Common Platform Enumerations when upstream docs provide them.
- `aliases`: Any alternate advisory identifiers present in the payload.
- `references`: Array of `{ type, url }` pairs pointing back to vendor advisories, patches, or exploits.
- `reconciled_from`: Provenance of linkset entries (JSON Pointer or field origin) to make automated checks auditable.
Canonicalisation rules:
- Package URLs are rendered in canonical form without qualifiers/subpaths (`pkg:type/namespace/name@version`).
- CPE values are normalised to the 2.3 binding (`cpe:2.3:part:vendor:product:version:*:*:*:*:*:*:*`).
- Connector mapping stages are responsible for the canonical form; ingestion trims whitespace but otherwise preserves the original order and duplicate entries so downstream policy can reason about upstream intent.
### 4.4 `advisory_observations`
`advisory_observations` is an immutable projection of the validated raw document used by LinkNotMerge overlays. Fields mirror the JSON contract surfaced by `StellaOps.Concelier.Models.Observations.AdvisoryObservation`.
| Field | Type | Notes |
|-------|------|-------|
| `_id` | string | Deterministic observation id — `{tenant}:{source.vendor}:{upstreamId}:{revision}`. |
| `tenant` | string | Lower-case tenant identifier. |
| `source.vendor` / `source.stream` | string | Connector identity (e.g., `vendor/redhat`, `ecosystem/osv`). |
| `source.api` | string | Absolute URI the connector fetched from. |
| `source.collectorVersion` | string | Optional semantic version of the connector build. |
| `upstream.upstream_id` | string | Advisory identifier as issued by the provider (CVE, vendor ID, etc.). |
| `upstream.document_version` | string | Upstream revision/version string. |
| `upstream.fetchedAt` / `upstream.receivedAt` | datetime | UTC timestamps recorded by the connector. |
| `upstream.contentHash` | string | `sha256:` digest used for idempotency. |
| `upstream.signature` | object | `{present, format?, keyId?, signature?}` describing upstream signature material. |
| `content.format` / `content.specVersion` | string | Raw payload format metadata (CSAF, OSV, JSON, etc.). |
| `content.raw` | object | Full upstream document stored losslessly (Relaxed Extended JSON). |
| `content.metadata` | object | Optional connector-specific metadata (batch ids, hints). |
| `linkset.aliases` | array | Connector-supplied aliases (trimmed, order preserved, duplicates allowed). |
| `linkset.purls` | array | Connector-supplied PURLs (ingestion preserves order and duplicates). |
| `linkset.cpes` | array | Connector-supplied CPE URIs (trimmed, order preserved). |
| `linkset.references` | array | `{ type, url }` pairs (trimmed; ingestion preserves order). |
| `createdAt` | datetime | Timestamp when Concelier persisted the observation. |
| `attributes` | object | Optional provenance attributes keyed by connector. |
## 5. Error Model
| Code | Description | HTTP status | Surfaces |
|------|-------------|-------------|----------|
| `ERR_AOC_001` | Forbidden field detected (severity, cvss, effective data). | 400 | Ingestion APIs, CLI verifier, CI guard. |
| `ERR_AOC_002` | Merge attempt detected (multiple upstream sources fused into one document). | 400 | Ingestion APIs, CLI verifier. |
| `ERR_AOC_003` | Idempotency violation (duplicate without supersedes pointer). | 409 | Repository guard, Mongo unique index, CLI verifier. |
| `ERR_AOC_004` | Missing provenance metadata (`source`, `upstream`, `signature`). | 422 | Schema validator, ingestion endpoints. |
| `ERR_AOC_005` | Signature or checksum mismatch. | 422 | Collector validation, CLI verifier. |
| `ERR_AOC_006` | Attempt to persist derived findings from ingestion context. | 403 | Policy engine guard, Authority scopes. |
| `ERR_AOC_007` | Unknown top-level fields (schema violation). | 400 | Mongo validator, CLI verifier. |
Consumers should map these codes to CLI exit codes and structured log events so automation can fail fast and produce actionable guidance.
## 6. API and Tooling Interfaces
- **Concelier ingestion** (`StellaOps.Concelier.WebService`)
- `POST /ingest/advisory`: accepts upstream payload metadata; server-side guard constructs and persists raw document.
- `GET /advisories/raw/{id}` and filterable list endpoints expose raw documents for debugging and offline analysis.
- `POST /aoc/verify`: runs guard checks over recent documents and returns summary totals plus first violations.
- **Excititor ingestion** (`StellaOps.Excititor.WebService`) mirrors the same surface for VEX documents.
- **CLI workflows** (`stella aoc verify`, `stella sources ingest --dry-run`) surface pre-flight verification; documentation will live in `/docs/modules/cli/guides/` alongside Sprint 19 CLI updates.
- **Authority scopes**: new `advisory:ingest`, `advisory:read`, `vex:ingest`, and `vex:read` scopes enforce least privilege; see [Authority Architecture](../modules/authority/architecture.md) for scope grammar.
## 7. Idempotency and Supersedes Rules
1. Compute `content_hash` before any transformation; use it with `(source.vendor, upstream.upstream_id)` to detect duplicates.
2. If a document with the same hash already exists, skip the write and log a no-op.
3. When a new hash arrives for an existing upstream document, insert a new record and set `supersedes` to the previous `_id`.
4. Keep supersedes chains acyclic; collectors must resolve conflicts by rewinding before they insert.
5. Expose idempotency counters via metrics (`ingestion_write_total{result=ok|noop}`) to catch regressions early.
## 8. Migration Playbook
1. Freeze ingestion writes except for raw pass-through paths while deploying schema validators.
2. Snapshot existing collections to `_backup_*` for rollback safety.
3. Strip forbidden fields from historical documents into a temporary `advisory_view_legacy` used only during transition.
4. Enable Mongo JSON schema validators for `advisory_raw` and `vex_raw`.
5. Run collectors in `--dry-run` to confirm only allowed keys appear; fix violations before lifting the freeze.
6. Point Policy Engine to consume exclusively from raw collections and compute derived outputs downstream.
7. Delete legacy normalisation paths from ingestion code and enable runtime guards plus CI linting.
8. Roll forward CLI, Console, and dashboards so operators can monitor AOC status end-to-end.
## 9. Observability and Diagnostics
- **Metrics**: `ingestion_write_total{result=ok|reject}`, `aoc_violation_total{code}`, `ingestion_signature_verified_total{result}`, `ingestion_latency_seconds`, `advisory_revision_count`.
- **Traces**: spans `ingest.fetch`, `ingest.transform`, `ingest.write`, and `aoc.guard` with correlation IDs shared across workers.
- **Logs**: structured entries must include `tenant`, `source.vendor`, `upstream.upstream_id`, `content_hash`, and `violation_code` when applicable.
- **Dashboards**: DevOps should add panels for violation counts, signature failures, supersedes growth, and CLI verifier outcomes for each tenant.
## 10. Security and Tenancy Checklist
- Enforce Authority scopes (`advisory:ingest`, `vex:ingest`, `advisory:read`, `vex:read`) and require tenant claims on every request.
- Maintain pinned trust stores for signature verification; capture verification result in metrics and logs.
- Ensure collectors never log secrets or raw authentication headers; redact tokens before persistence.
- Validate that Policy Engine remains the only identity with permission to write `effective_finding_*` documents.
- Verify offline bundles include the raw collections, guard configuration, and verifier binaries so air-gapped installs can audit parity.
- Document operator steps for recovering from violations, including rollback to superseded revisions and re-running policy evaluation.
## 11. Compliance Checklist
- [ ] Deterministic guard enabled in Concelier and Excititor repositories.
- [ ] Mongo validators deployed for `advisory_raw` and `vex_raw`.
- [ ] Authority scopes and tenant enforcement verified via integration tests.
- [ ] CLI and CI pipelines run `stella aoc verify` against seeded snapshots.
- [ ] Observability feeds (metrics, logs, traces) wired into dashboards with alerts.
- [ ] Offline kit instructions updated to bundle validators and verifier tooling.
- [ ] Security review recorded covering ingestion, tenancy, and rollback procedures.
---
*Last updated: 2025-10-27 (Sprint 19).*
## 2. Philosophy and Goals
- Preserve upstream truth: ingestion only captures immutable raw facts plus provenance, never derived severity or policy decisions.
- Defer interpretation: Policy Engine and downstream overlays remain the sole writers of materialised findings, severity, consensus, or risk scores.
- Make every write explainable: provenance, signatures, and content hashes are required so operators can prove where each fact originated.
- Keep outputs reproducible: identical inputs must yield identical documents, hashes, and linksets across replays and air-gapped installs.
## 3. Contract Invariants
| # | Invariant | What it forbids or requires | Enforcement surfaces |
|---|-----------|-----------------------------|----------------------|
| 1 | No derived severity at ingest | Reject top-level keys such as `severity`, `cvss`, `effective_status`, `consensus_provider`, `risk_score`. Raw upstream CVSS remains inside `content.raw`. | Mongo schema validator, `AOCWriteGuard`, Roslyn analyzer, `stella aoc verify`. |
| 2 | No merges or opinionated dedupe | Each upstream document persists on its own; ingestion never collapses multiple vendors into one document. | Repository interceptors, unit/fixture suites. |
| 3 | Provenance is mandatory | `source.*`, `upstream.*`, and `signature` metadata must be present; missing provenance triggers `ERR_AOC_004`. | Schema validator, guard, CLI verifier. |
| 4 | Idempotent upserts | Writes keyed by `(vendor, upstream_id, content_hash)` either no-op or insert a new revision with `supersedes`. Duplicate hashes map to the same document. | Repository guard, storage unique index, CI smoke tests. |
| 5 | Append-only revisions | Updates create a new document with `supersedes` pointer; no in-place mutation of content. | Mongo schema (`supersedes` format), guard, data migration scripts. |
| 6 | Linkset only | Ingestion may compute link hints (`purls`, `cpes`, IDs) to accelerate joins, but must not transform or infer severity or policy. Observations now persist both canonical linksets (for indexed queries) and raw linksets (preserving upstream order/duplicates) so downstream policy can decide how to normalise. When `concelier:features:noMergeEnabled=true`, all merge-derived canonicalisation paths must be disabled. | Linkset builders reviewed via fixtures/analyzers; raw-vs-canonical parity covered by observation fixtures; analyzer `CONCELIER0002` blocks merge API usage. |
| 7 | Policy-only effective findings | Only Policy Engine identities can write `effective_finding_*`; ingestion callers receive `ERR_AOC_006` if they attempt it. | Authority scopes, Policy Engine guard. |
| 8 | Schema safety | Unknown top-level keys reject with `ERR_AOC_007`; timestamps use ISO 8601 UTC strings; tenant is required. | Mongo validator, JSON schema tests. |
| 9 | Clock discipline | Collectors stamp `fetched_at` and `received_at` monotonically per batch to support reproducibility windows. | Collector contracts, QA fixtures. |
## 4. Raw Schemas
### 4.1 `advisory_raw`
| Field | Type | Notes |
|-------|------|-------|
| `_id` | string | `advisory_raw:{source}:{upstream_id}:{revision}`; deterministic and tenant-scoped. |
| `tenant` | string | Required; injected by Authority middleware and asserted by schema validator. |
| `source.vendor` | string | Provider identifier (e.g., `redhat`, `osv`, `ghsa`). |
| `source.stream` | string | Connector stream name (`csaf`, `osv`, etc.). |
| `source.api` | string | Absolute URI of upstream document; stored for traceability. |
| `source.collector_version` | string | Semantic version of the collector. |
| `upstream.upstream_id` | string | Vendor- or ecosystem-provided identifier (CVE, GHSA, vendor ID). |
| `upstream.document_version` | string | Upstream issued timestamp or revision string. |
| `upstream.fetched_at` / `received_at` | string | ISO 8601 UTC timestamps recorded by the collector. |
| `upstream.content_hash` | string | `sha256:` digest of the raw payload used for idempotency. |
| `upstream.signature` | object | Required structure storing `present`, `format`, `key_id`, `sig`; even unsigned payloads set `present: false`. |
| `content.format` | string | Source format (`CSAF`, `OSV`, etc.). |
| `content.spec_version` | string | Upstream spec version when known. |
| `content.raw` | object | Full upstream payload, untouched except for transport normalisation. |
| `identifiers` | object | Upstream identifiers (`cve`, `ghsa`, `aliases`, etc.) captured as provided (trimmed, order preserved, duplicates allowed). |
| `linkset` | object | Join hints (see section 4.3). |
| `supersedes` | string or null | Points to previous revision of same upstream doc when content hash changes. |
### 4.2 `vex_raw`
| Field | Type | Notes |
|-------|------|-------|
| `_id` | string | `vex_raw:{source}:{upstream_id}:{revision}`. |
| `tenant` | string | Required; matches advisory collection requirements. |
| `source.*` | object | Same shape and requirements as `advisory_raw`. |
| `upstream.*` | object | Includes `document_version`, timestamps, `content_hash`, and `signature`. |
| `content.format` | string | Typically `CycloneDX-VEX` or `CSAF-VEX`. |
| `content.raw` | object | Entire upstream VEX payload. |
| `identifiers.statements` | array | Normalised statement summaries (IDs, PURLs, status, justification) to accelerate policy joins. |
| `linkset` | object | CVEs, GHSA IDs, and PURLs referenced in the document. |
| `supersedes` | string or null | Same convention as advisory documents. |
### 4.3 Linkset Fields
- `purls`: fully qualified Package URLs extracted from raw ranges or product nodes.
- `cpes`: Common Platform Enumerations when upstream docs provide them.
- `aliases`: Any alternate advisory identifiers present in the payload.
- `references`: Array of `{ type, url }` pairs pointing back to vendor advisories, patches, or exploits.
- `reconciled_from`: Provenance of linkset entries (JSON Pointer or field origin) to make automated checks auditable.
Canonicalisation rules:
- Package URLs are rendered in canonical form without qualifiers/subpaths (`pkg:type/namespace/name@version`).
- CPE values are normalised to the 2.3 binding (`cpe:2.3:part:vendor:product:version:*:*:*:*:*:*:*`).
- Connector mapping stages are responsible for the canonical form; ingestion trims whitespace but otherwise preserves the original order and duplicate entries so downstream policy can reason about upstream intent.
### 4.4 `advisory_observations`
`advisory_observations` is an immutable projection of the validated raw document used by LinkNotMerge overlays. Fields mirror the JSON contract surfaced by `StellaOps.Concelier.Models.Observations.AdvisoryObservation`.
| Field | Type | Notes |
|-------|------|-------|
| `_id` | string | Deterministic observation id — `{tenant}:{source.vendor}:{upstreamId}:{revision}`. |
| `tenant` | string | Lower-case tenant identifier. |
| `source.vendor` / `source.stream` | string | Connector identity (e.g., `vendor/redhat`, `ecosystem/osv`). |
| `source.api` | string | Absolute URI the connector fetched from. |
| `source.collectorVersion` | string | Optional semantic version of the connector build. |
| `upstream.upstream_id` | string | Advisory identifier as issued by the provider (CVE, vendor ID, etc.). |
| `upstream.document_version` | string | Upstream revision/version string. |
| `upstream.fetchedAt` / `upstream.receivedAt` | datetime | UTC timestamps recorded by the connector. |
| `upstream.contentHash` | string | `sha256:` digest used for idempotency. |
| `upstream.signature` | object | `{present, format?, keyId?, signature?}` describing upstream signature material. |
| `content.format` / `content.specVersion` | string | Raw payload format metadata (CSAF, OSV, JSON, etc.). |
| `content.raw` | object | Full upstream document stored losslessly (Relaxed Extended JSON). |
| `content.metadata` | object | Optional connector-specific metadata (batch ids, hints). |
| `linkset.aliases` | array | Connector-supplied aliases (trimmed, order preserved, duplicates allowed). |
| `linkset.purls` | array | Connector-supplied PURLs (ingestion preserves order and duplicates). |
| `linkset.cpes` | array | Connector-supplied CPE URIs (trimmed, order preserved). |
| `linkset.references` | array | `{ type, url }` pairs (trimmed; ingestion preserves order). |
| `createdAt` | datetime | Timestamp when Concelier persisted the observation. |
| `attributes` | object | Optional provenance attributes keyed by connector. |
## 5. Error Model
| Code | Description | HTTP status | Surfaces |
|------|-------------|-------------|----------|
| `ERR_AOC_001` | Forbidden field detected (severity, cvss, effective data). | 400 | Ingestion APIs, CLI verifier, CI guard. |
| `ERR_AOC_002` | Merge attempt detected (multiple upstream sources fused into one document). | 400 | Ingestion APIs, CLI verifier. |
| `ERR_AOC_003` | Idempotency violation (duplicate without supersedes pointer). | 409 | Repository guard, Mongo unique index, CLI verifier. |
| `ERR_AOC_004` | Missing provenance metadata (`source`, `upstream`, `signature`). | 422 | Schema validator, ingestion endpoints. |
| `ERR_AOC_005` | Signature or checksum mismatch. | 422 | Collector validation, CLI verifier. |
| `ERR_AOC_006` | Attempt to persist derived findings from ingestion context. | 403 | Policy engine guard, Authority scopes. |
| `ERR_AOC_007` | Unknown top-level fields (schema violation). | 400 | Mongo validator, CLI verifier. |
Consumers should map these codes to CLI exit codes and structured log events so automation can fail fast and produce actionable guidance.
## 6. API and Tooling Interfaces
- **Concelier ingestion** (`StellaOps.Concelier.WebService`)
- `POST /ingest/advisory`: accepts upstream payload metadata; server-side guard constructs and persists raw document.
- `GET /advisories/raw/{id}` and filterable list endpoints expose raw documents for debugging and offline analysis.
- `POST /aoc/verify`: runs guard checks over recent documents and returns summary totals plus first violations.
- **Excititor ingestion** (`StellaOps.Excititor.WebService`) mirrors the same surface for VEX documents.
- **CLI workflows** (`stella aoc verify`, `stella sources ingest --dry-run`) surface pre-flight verification; documentation will live in `/docs/modules/cli/guides/` alongside Sprint 19 CLI updates.
- **Authority scopes**: new `advisory:ingest`, `advisory:read`, `vex:ingest`, and `vex:read` scopes enforce least privilege; see [Authority Architecture](../modules/authority/architecture.md) for scope grammar.
## 7. Idempotency and Supersedes Rules
1. Compute `content_hash` before any transformation; use it with `(source.vendor, upstream.upstream_id)` to detect duplicates.
2. If a document with the same hash already exists, skip the write and log a no-op.
3. When a new hash arrives for an existing upstream document, insert a new record and set `supersedes` to the previous `_id`.
4. Keep supersedes chains acyclic; collectors must resolve conflicts by rewinding before they insert.
5. Expose idempotency counters via metrics (`ingestion_write_total{result=ok|noop}`) to catch regressions early.
## 8. Migration Playbook
1. Freeze ingestion writes except for raw pass-through paths while deploying schema validators.
2. Snapshot existing collections to `_backup_*` for rollback safety.
3. Strip forbidden fields from historical documents into a temporary `advisory_view_legacy` used only during transition.
4. Enable Mongo JSON schema validators for `advisory_raw` and `vex_raw`.
5. Run collectors in `--dry-run` to confirm only allowed keys appear; fix violations before lifting the freeze.
6. Point Policy Engine to consume exclusively from raw collections and compute derived outputs downstream.
7. Delete legacy normalisation paths from ingestion code and enable runtime guards plus CI linting.
8. Roll forward CLI, Console, and dashboards so operators can monitor AOC status end-to-end.
## 9. Observability and Diagnostics
- **Metrics**: `ingestion_write_total{result=ok|reject}`, `aoc_violation_total{code}`, `ingestion_signature_verified_total{result}`, `ingestion_latency_seconds`, `advisory_revision_count`.
- **Traces**: spans `ingest.fetch`, `ingest.transform`, `ingest.write`, and `aoc.guard` with correlation IDs shared across workers.
- **Logs**: structured entries must include `tenant`, `source.vendor`, `upstream.upstream_id`, `content_hash`, and `violation_code` when applicable.
- **Dashboards**: DevOps should add panels for violation counts, signature failures, supersedes growth, and CLI verifier outcomes for each tenant.
## 10. Security and Tenancy Checklist
- Enforce Authority scopes (`advisory:ingest`, `vex:ingest`, `advisory:read`, `vex:read`) and require tenant claims on every request.
- Maintain pinned trust stores for signature verification; capture verification result in metrics and logs.
- Ensure collectors never log secrets or raw authentication headers; redact tokens before persistence.
- Validate that Policy Engine remains the only identity with permission to write `effective_finding_*` documents.
- Verify offline bundles include the raw collections, guard configuration, and verifier binaries so air-gapped installs can audit parity.
- Document operator steps for recovering from violations, including rollback to superseded revisions and re-running policy evaluation.
## 11. Compliance Checklist
- [ ] Deterministic guard enabled in Concelier and Excititor repositories.
- [ ] Mongo validators deployed for `advisory_raw` and `vex_raw`.
- [ ] Authority scopes and tenant enforcement verified via integration tests.
- [ ] CLI and CI pipelines run `stella aoc verify` against seeded snapshots.
- [ ] Observability feeds (metrics, logs, traces) wired into dashboards with alerts.
- [ ] Offline kit instructions updated to bundle validators and verifier tooling.
- [ ] Security review recorded covering ingestion, tenancy, and rollback procedures.
---
*Last updated: 2025-10-27 (Sprint 19).*

View File

@@ -1,6 +1,6 @@
# No-Merge Migration Playbook
_Last updated: 2025-11-03_
_Last updated: 2025-11-06_
This playbook guides the full retirement of the legacy Merge service (`AdvisoryMergeService`) in favour of Link-Not-Merge (LNM) observations plus linksets. It is written for the BE-Merge, Architecture, DevOps, and Docs guilds coordinating Sprint110 (Ingestion & Evidence) deliverables, and it feeds CONCELIER-LNM-21-101 / MERGE-LNM-21-001 and downstream DOCS-LNM-22-008.
@@ -35,6 +35,10 @@ Do not proceed to Phase1 until all prerequisites are checked or explicitly wa
| `concelier:jobs:merge:allowlist` | `[]` | Explicit allowlist for Merge jobs when noMergeEnabled is `false`. | Set to empty during Phase2+ to prevent accidental restarts. |
| `policy:overlays:requireLinksetEvidence` | `false` | Policy engine safety net to require linkset-backed findings. | Flip to `true` only after cutover (Phase2). |
> 2025-11-05: WebService honours `concelier:features:noMergeEnabled` by skipping Merge DI registration and removing the `merge:reconcile` job definition (MERGE-LNM-21-002).
>
> 2025-11-06: Analyzer `CONCELIER0002` ships with Concelier hosts to block new references to `AdvisoryMergeService` / `AddMergeModule`. Suppressions must be paired with an explicit migration note.
> **Configuration hygiene:** Document the toggle values per environment in `ops/devops/configuration/staging.md` and `ops/devops/configuration/production.md`. Air-gapped customers receive defaults through the Offline Kit release notes.
## 3. Rollout phases

View File

@@ -1,11 +1,14 @@
# StellaOps Attestor
Attestor converts signed DSSE evidence from the Signer into transparency-log proofs and verifiable reports for every downstream surface (Policy Engine, Export Center, CLI, Console, Scheduler). It is the trust backbone that proves SBOM, scan, VEX, and policy artefacts were signed, witnessed, and preserved without tampering.
## Why it exists
- **Evidence first:** organisations need portable, verifiable attestations that prove build provenance, SBOM availability, policy verdicts, and VEX statements.
- **Policy enforcement:** verification policies ensure only approved issuers, key types, witnesses, and freshness windows are accepted.
- **Sovereign/offline-ready:** Attestor archives envelopes, signatures, and proofs so air-gapped deployments can replay verification without contacting external services.
Attestor converts signed DSSE evidence from the Signer into transparency-log proofs and verifiable reports for every downstream surface (Policy Engine, Export Center, CLI, Console, Scheduler). It is the trust backbone that proves SBOM, scan, VEX, and policy artefacts were signed, witnessed, and preserved without tampering.
## Latest updates (2025-10-19)
- Platform Events refresh published canonical `attestor.logged@1` samples under `docs/events/samples/` and validated schemas (`docs/updates/2025-10-18-docs-guild.md`, `docs/updates/2025-10-19-docs-guild.md`). Consumers should align verification workflows and tests with those sample envelopes.
## Why it exists
- **Evidence first:** organisations need portable, verifiable attestations that prove build provenance, SBOM availability, policy verdicts, and VEX statements.
- **Policy enforcement:** verification policies ensure only approved issuers, key types, witnesses, and freshness windows are accepted.
- **Sovereign/offline-ready:** Attestor archives envelopes, signatures, and proofs so air-gapped deployments can replay verification without contacting external services.
## Roles & surfaces
- **Subjects:** immutable digests for container images, SBOMs, reports, and policy bundles.

View File

@@ -4,6 +4,6 @@
| ID | Status | Owner(s) | Description | Notes |
|----|--------|----------|-------------|-------|
| ATTESTOR-DOCS-0001 | DOING (2025-10-29) | Docs Guild | Validate that ./README.md aligns with the latest release notes. | See ./AGENTS.md |
| ATTESTOR-DOCS-0001 | DONE (2025-11-05) | Docs Guild | Validate that ./README.md aligns with the latest release notes. | README now references 2025-10-18/19 platform-event updates and the canonical `attestor.logged@1` samples. |
| ATTESTOR-OPS-0001 | TODO | Ops Guild | Review runbooks/observability assets after next sprint demo. | Sync outcomes back to ../../TASKS.md |
| ATTESTOR-ENG-0001 | TODO | Module Team | Cross-check implementation plan milestones against `/docs/implplan/SPRINT_*.md`. | Update status via ./AGENTS.md workflow |

View File

@@ -18,19 +18,22 @@ Concelier ingests signed advisories from dozens of sources and converts them int
- Policy Engine / Export Center / CLI for evidence consumption.
- Notify and UI for advisory deltas.
## Operational notes
- Connector runbooks in ./operations/connectors/.
- Mirror operations for Offline Kit parity.
- Grafana dashboards for connector health.
## Related resources
- ./operations/conflict-resolution.md
- ./operations/mirror.md
## Backlog references
- DOCS-LNM-22-001, DOCS-LNM-22-007 in ../../TASKS.md.
- Connector-specific TODOs in `src/Concelier/**/TASKS.md`.
## Operational notes
- Connector runbooks in ./operations/connectors/.
- Mirror operations for Offline Kit parity.
- Grafana dashboards for connector health.
- **Authority toggle rollout (2025-10-22 update).** Follow the phased table and audit checklist in `../../10_CONCELIER_CLI_QUICKSTART.md` when enabling `authority.enabled`/`authority.allowAnonymousFallback`, and cross-check the refreshed `./operations/authority-audit-runbook.md` before enforcement.
## Related resources
- ./operations/conflict-resolution.md
- ./operations/mirror.md
- ./operations/authority-audit-runbook.md
- ../../10_CONCELIER_CLI_QUICKSTART.md (authority integration timeline & smoke tests)
## Backlog references
- DOCS-LNM-22-001, DOCS-LNM-22-007 in ../../TASKS.md.
- Connector-specific TODOs in `src/Concelier/**/TASKS.md`.
## Epic alignment
- **Epic 1 AOC enforcement:** uphold raw observation invariants, provenance requirements, linkset-only enrichment, and AOC verifier guardrails across every connector.
- **Epic 10 Export Center:** expose deterministic advisory exports and metadata required by JSON/Trivy/mirror bundles.

View File

@@ -4,6 +4,6 @@
| ID | Status | Owner(s) | Description | Notes |
|----|--------|----------|-------------|-------|
| CONCELIER-DOCS-0001 | DOING (2025-10-29) | Docs Guild | Validate that ./README.md aligns with the latest release notes. | See ./AGENTS.md |
| CONCELIER-DOCS-0001 | DONE (2025-11-05) | Docs Guild | Validate that ./README.md aligns with the latest release notes. | README now references the 2025-10-22 authority toggle rollout update (quickstart/runbook links). |
| CONCELIER-OPS-0001 | TODO | Ops Guild | Review runbooks/observability assets after next sprint demo. | Sync outcomes back to ../../TASKS.md |
| CONCELIER-ENG-0001 | TODO | Module Team | Cross-check implementation plan milestones against `/docs/implplan/SPRINT_*.md`. | Update status via ./AGENTS.md workflow |

View File

@@ -27,10 +27,12 @@
3. **Mandatory provenance.** Collectors record `source`, `upstream` metadata (`document_version`, `fetched_at`, `received_at`, `content_hash`), and signature presence before writing.
4. **Linkset only.** Derived joins (aliases, PURLs, CPEs, references) are stored inside `linkset` and never mutate `content.raw`.
5. **Deterministic canonicalisation.** Writers use canonical JSON (sorted object keys, lexicographic arrays) ensuring identical inputs yield the same hashes/diff-friendly outputs.
6. **Idempotent upserts.** `(source.vendor, upstream.upstream_id, upstream.content_hash)` uniquely identify a document. Duplicate hashes short-circuit; new hashes create a new version.
7. **Verifier & CI.** `StellaOps.AOC.Verifier` processes observation batches in CI and at runtime, rejecting writes lacking provenance, introducing unordered collections, or violating the schema.
### 1.1 Advisory raw document shape
6. **Idempotent upserts.** `(source.vendor, upstream.upstream_id, upstream.content_hash)` uniquely identify a document. Duplicate hashes short-circuit; new hashes create a new version.
7. **Verifier & CI.** `StellaOps.AOC.Verifier` processes observation batches in CI and at runtime, rejecting writes lacking provenance, introducing unordered collections, or violating the schema.
> Feature toggle: set `concelier:features:noMergeEnabled=true` to disable the legacy Merge module and its `merge:reconcile` job once Link-Not-Merge adoption is complete (MERGE-LNM-21-002). Analyzer `CONCELIER0002` prevents new references to Merge DI helpers when this flag is enabled.
### 1.1 Advisory raw document shape
```json
{

View File

@@ -1,33 +1,37 @@
# StellaOps Excititor
Excititor converts heterogeneous VEX feeds into raw observations and linksets that honour the Aggregation-Only Contract.
## Responsibilities
- Fetch OpenVEX/CSAF/CycloneDX statements via restart-only connectors.
- Store immutable VEX observations with full provenance.
- Publish linksets and events that drive policy suppression decisions.
- Provide deterministic exports for Offline Kit and downstream tooling.
## Key components
- `StellaOps.Excititor.WebService` scheduler/API host.
- Connector libraries under `StellaOps.Excititor.Connector.*`.
- Normalization helpers and exporters in `StellaOps.Excititor.*`.
## Integrations & dependencies
- Policy Engine for evidence queries.
- UI/CLI for conflict visibility and explanation.
- Notify for VEX-driven alerts.
## Operational notes
- MongoDB for observation storage and job metadata.
- Offline kit packaging aligned with Concelier merges.
- Connector-specific runbooks (see `docs/modules/concelier/operations/connectors`).
## Backlog references
- DOCS-LNM-22-006 / DOCS-LNM-22-007 (shared with Concelier).
- CLI-EXC-25-001..002 follow-up for CLI parity.
## Epic alignment
- **Epic 1 AOC enforcement:** maintain immutable VEX observations, provenance, and AOC verifier coverage.
- **Epic 7 VEX Consensus Lens:** supply trustworthy raw inputs, trust metadata, and consensus hooks for the lens computations.
- **Epic 8 Advisory AI:** expose citation-ready VEX payloads for the advisory assistant pipeline.
# StellaOps Excititor
Excititor converts heterogeneous VEX feeds into raw observations and linksets that honour the Aggregation-Only Contract.
## Latest updates (2025-11-05)
- Link-Not-Merge readiness: release note [Excitor consensus beta](../../updates/2025-11-05-excitor-consensus-beta.md) captures how Excititor feeds power the Excitor consensus beta (sample payload in [consensus JSON](../../vex/consensus-json.md)).
- README now points policy/UI teams to the upcoming consensus integration work.
## Responsibilities
- Fetch OpenVEX/CSAF/CycloneDX statements via restart-only connectors.
- Store immutable VEX observations with full provenance.
- Publish linksets and events that drive policy suppression decisions.
- Provide deterministic exports for Offline Kit and downstream tooling.
## Key components
- `StellaOps.Excititor.WebService` scheduler/API host.
- Connector libraries under `StellaOps.Excititor.Connector.*`.
- Normalization helpers and exporters in `StellaOps.Excititor.*`.
## Integrations & dependencies
- Policy Engine for evidence queries.
- UI/CLI for conflict visibility and explanation.
- Notify for VEX-driven alerts.
## Operational notes
- MongoDB for observation storage and job metadata.
- Offline kit packaging aligned with Concelier merges.
- Connector-specific runbooks (see `docs/modules/concelier/operations/connectors`).
## Backlog references
- DOCS-LNM-22-006 / DOCS-LNM-22-007 (shared with Concelier).
- CLI-EXC-25-001..002 follow-up for CLI parity.
## Epic alignment
- **Epic 1 AOC enforcement:** maintain immutable VEX observations, provenance, and AOC verifier coverage.
- **Epic 7 VEX Consensus Lens:** supply trustworthy raw inputs, trust metadata, and consensus hooks for the lens computations.
- **Epic 8 Advisory AI:** expose citation-ready VEX payloads for the advisory assistant pipeline.

View File

@@ -4,6 +4,6 @@
| ID | Status | Owner(s) | Description | Notes |
|----|--------|----------|-------------|-------|
| EXCITITOR-DOCS-0001 | TODO | Docs Guild | Validate that ./README.md aligns with the latest release notes. | See ./AGENTS.md |
| EXCITITOR-DOCS-0001 | DONE (2025-11-05) | Docs Guild | Validate that ./README.md aligns with the latest release notes. | README now links to the [Excitor consensus beta release note](../../updates/2025-11-05-excitor-consensus-beta.md) and [consensus JSON sample](../../vex/consensus-json.md). |
| EXCITITOR-OPS-0001 | TODO | Ops Guild | Review runbooks/observability assets after next sprint demo. | Sync outcomes back to ../../TASKS.md |
| EXCITITOR-ENG-0001 | TODO | Module Team | Cross-check implementation plan milestones against `/docs/implplan/SPRINT_*.md`. | Update status via ./AGENTS.md workflow |

View File

@@ -2,6 +2,10 @@
Excitor computes deterministic consensus across VEX claims, preserving conflicts and producing attestable evidence for policy suppression.
## Latest updates (2025-11-05)
- Consensus API beta documented with canonical JSON samples and DSSE packaging guidance (`docs/updates/2025-11-05-excitor-consensus-beta.md`).
- README links to Link-Not-Merge consensus milestone and preview endpoints for downstream integration.
## Responsibilities
- Ingest Excititor observations and compute per-product consensus snapshots.
- Provide APIs for querying canonical VEX positions and conflict sets.
@@ -25,6 +29,7 @@ Excitor computes deterministic consensus across VEX claims, preserving conflicts
## Related resources
- ./scoring.md
- ../../vex/consensus-json.md (beta consensus payload sample)
## Backlog references
- DOCS-EXCITOR backlog referenced in architecture doc.

View File

@@ -4,6 +4,6 @@
| ID | Status | Owner(s) | Description | Notes |
|----|--------|----------|-------------|-------|
| EXCITOR-DOCS-0001 | DOING (2025-10-29) | Docs Guild | Validate that ./README.md aligns with the latest release notes. | See ./AGENTS.md |
| EXCITOR-DOCS-0001 | DONE (2025-11-05) | Docs Guild | Validate that ./README.md aligns with the latest release notes. | README now references the 2025-11-05 consensus API beta release note (`docs/updates/2025-11-05-excitor-consensus-beta.md`). |
| EXCITOR-OPS-0001 | TODO | Ops Guild | Review runbooks/observability assets after next sprint demo. | Sync outcomes back to ../../TASKS.md |
| EXCITOR-ENG-0001 | TODO | Module Team | Cross-check implementation plan milestones against `/docs/implplan/SPRINT_*.md`. | Update status via ./AGENTS.md workflow |

View File

@@ -8,27 +8,35 @@ Export Center packages reproducible evidence bundles (JSON, Trivy DB, mirror) wi
- Stream bundles via HTTP/OCI and stage them for Offline Kit uses.
- Expose CLI/API surfaces for automation.
## Key components
- `StellaOps.ExportCenter.WebService` planner.
- `StellaOps.ExportCenter.Worker` bundle builder.
- Adapters in `StellaOps.ExportCenter.*` for JSON/Trivy/mirror variants.
## Integrations & dependencies
- Concelier/Excititor/Policy data stores for evidence.
- Signer/Attestor for provenance signing.
- CLI for operator-managed exports.
## Key components
- `StellaOps.ExportCenter.WebService` planner.
- `StellaOps.ExportCenter.Worker` bundle builder.
- Adapters in `StellaOps.ExportCenter.*` for JSON/Trivy/mirror variants.
## Profiles at a glance
- **json:raw / json:policy** — Evidence bundles with raw ingestion facts or policy overlays.
- **trivy:db / trivy:java-db** — Trivy-compatible vulnerability feeds with deterministic manifests.
- **mirror:full / mirror:delta** — OCI-style mirrors with provenance, TUF metadata, and optional encryption.
- **devportal:offline** — Developer portal static assets, specs, SDKs, and changelogs packaged with `manifest.json`, `checksums.txt`, helper scripts, and a DSSE-signed manifest (`manifest.dsse.json`) for offline verification.
## Integrations & dependencies
- Concelier/Excititor/Policy data stores for evidence.
- Signer/Attestor for provenance signing.
- CLI for operator-managed exports.
## Operational notes
- Runbooks in ./operations/ for deployment and monitoring.
- Mirror bundle instructions and validation notes.
- Telemetry dashboards for export latency and retry rates.
## Related resources
- ./operations/runbook.md
## Backlog references
- DOCS-EXPORT-35-001 … DOCS-EXPORT-37-002 in ../../TASKS.md.
- EXPORT-ATTEST-75-002 cross-team deliverable.
## Related resources
- ./operations/runbook.md
- ./devportal-offline.md (bundle structure, verification workflow, DSSE signature details)
- ./provenance-and-signing.md (manifest/provenance schema, signing pipeline, verification)
## Backlog references
- DOCS-EXPORT-35-001 … DOCS-EXPORT-37-002 in ../../TASKS.md.
- EXPORT-ATTEST-75-002 cross-team deliverable.
## Epic alignment
- **Epic 10 Export Center:** deliver canonical JSON, Trivy DB, and mirror bundle workflows with provenance, signatures, and offline parity.

View File

@@ -4,6 +4,6 @@
| ID | Status | Owner(s) | Description | Notes |
|----|--------|----------|-------------|-------|
| EXPORT CENTER-DOCS-0001 | DOING (2025-10-29) | Docs Guild | Validate that ./README.md aligns with the latest release notes. | See ./AGENTS.md |
| EXPORT CENTER-DOCS-0001 | DONE (2025-11-05) | Docs Guild | Validate that ./README.md aligns with the latest release notes. | README now documents devportal offline profile, DSSE manifest signature, and links to supporting specs per 2025-10-29 release update. |
| EXPORT CENTER-OPS-0001 | TODO | Ops Guild | Review runbooks/observability assets after next sprint demo. | Sync outcomes back to ../../TASKS.md |
| EXPORT CENTER-ENG-0001 | TODO | Module Team | Cross-check implementation plan milestones against `/docs/implplan/SPRINT_*.md`. | Update status via ./AGENTS.md workflow |

View File

@@ -84,7 +84,8 @@ All endpoints require Authority-issued JWT + DPoP tokens with scopes `export:run
- Supports optional encryption of `/data` subtree (age/AES-GCM) with key wrapping stored in `provenance.json`.
- **DevPortal (`devportal:offline`).**
- Packages developer portal static assets, OpenAPI specs, SDK releases, and changelog content into a reproducible archive with manifest/checksum pairs.
- Emits `manifest.json`, `checksums.txt`, and helper scripts described in [DevPortal Offline Bundle Specification](devportal-offline.md); signing/DSSE wiring follows the shared Export Center signing service.
- Emits `manifest.json`, `checksums.txt`, helper scripts, and a DSSE signature document (`manifest.dsse.json`) as described in [DevPortal Offline Bundle Specification](devportal-offline.md).
- Stores artefacts under `<storagePrefix>/<bundleId>/` and signs manifests via the Export Center signing adapter (HMAC-SHA256 v1, tenant scoped).
Adapters expose structured telemetry events (`adapter.start`, `adapter.chunk`, `adapter.complete`) with record counts and byte totals per chunk. Failures emit `adapter.error` with reason codes.

View File

@@ -81,7 +81,14 @@ root <sha256(manifest.json)>
The `root` value is the SHA-256 hash of the serialized manifest and is exposed separately in the result object for downstream signing.
## 4. Verification script
## 4. DSSE signature and storage
- The export job signs `manifest.json` with an HMAC-SHA256 based DSSE envelope, producing `manifest.dsse.json` alongside the bundle artefacts.
- The signature document captures the bundle identifier, manifest root hash, signing timestamp, algorithm metadata, and the DSSE payload/signature.
- Operators verify the manifest by validating `manifest.dsse.json`, then cross-checking the `payload` base64 with the downloaded manifest and the `rootHash` in `checksums.txt`.
- All artefacts are written under a deterministic storage prefix `<storagePrefix>/<bundleId>/` with the bundle archive stored as `bundle.tgz` (or the configured file name).
## 5. Verification script
`verify-offline.sh` is a POSIX-compatible helper that:
@@ -91,7 +98,7 @@ The `root` value is the SHA-256 hash of the serialized manifest and is exposed s
Operators can override the archive name via the first argument (`./verify-offline.sh mybundle.tgz`).
## 5. Content categories
## 6. Content categories
| Category | Target prefix | Notes |
|-----------|---------------|-------|
@@ -102,14 +109,13 @@ Operators can override the archive name via the first argument (`./verify-offlin
Paths are normalised to forward slashes and guarded against directory traversal.
## 6. Determinism and hashing rules
## 7. Determinism and hashing rules
- Files are enumerated and emitted in ordinal path order.
- SHA-256 digests use lowercase hex encoding.
- Optional directories (specs, SDKs, changelog) are skipped when absent; at least one category must contain files or the builder fails fast.
## 7. Next steps
## 8. Next steps
- Attach DSSE signing + timestamping (`signature.json`) once Export Center signing infrastructure is ready.
- Integrate the builder into the Export Center worker profile (`devportal --offline`) and plumb orchestration/persistence.
- Produce CLI validation tooling (`stella devportal verify`) per DVOFF-64-002 and document operator workflows under `docs/airgap/devportal-offline.md`.
- Expand `stella devportal verify` (DVOFF-64-002) to validate DSSE signatures and bundle integrity offline.
- Document operator workflow under `docs/airgap/devportal-offline.md`.

View File

@@ -1,12 +1,15 @@
# StellaOps Source & Job Orchestrator
The Orchestrator schedules, observes, and recovers ingestion and analysis jobs across the StellaOps platform.
## Responsibilities
- Track job state, throughput, and errors for Concelier, Excititor, Scheduler, and export pipelines.
- Expose dashboards and APIs for throttling, replays, and failover.
- Enforce rate-limits, concurrency and dependency chains across queues.
- Stream structured events and audit logs for incident response.
The Orchestrator schedules, observes, and recovers ingestion and analysis jobs across the StellaOps platform.
## Latest updates (2025-11-01)
- Authority added `orch:quota` and `orch:backfill` scopes for quota/backfill operations, plus token reason/ticket auditing (`docs/updates/2025-11-01-orch-admin-scope.md`). Operators must supply `quota_reason` / `quota_ticket` (or `backfill_reason` / `backfill_ticket`) when requesting elevated tokens and surface those claims in change reviews.
## Responsibilities
- Track job state, throughput, and errors for Concelier, Excititor, Scheduler, and export pipelines.
- Expose dashboards and APIs for throttling, replays, and failover.
- Enforce rate-limits, concurrency and dependency chains across queues.
- Stream structured events and audit logs for incident response.
## Key components
- Orchestrator WebService (control plane).
@@ -19,10 +22,11 @@ The Orchestrator schedules, observes, and recovers ingestion and analysis jobs a
- Scheduler/Concelier/Excititor workers for job lifecycle.
- Offline Kit for state export/import during air-gap refreshes.
## Operational notes
- Job recovery runbooks and dashboard JSON as described in Epic 9.
- Audit retention policies for job history.
- Rate-limit reconfiguration guidelines.
## Operational notes
- Job recovery runbooks and dashboard JSON as described in Epic 9.
- Audit retention policies for job history.
- Rate-limit reconfiguration guidelines.
- When using the new `orch:quota` / `orch:backfill` scopes, ensure reason/ticket fields are captured in runbooks and audit checklists per the 2025-11-01 Authority update.
## Epic alignment
- Epic 9: Source & Job Orchestrator Dashboard.

View File

@@ -4,6 +4,6 @@
| ID | Status | Owner(s) | Description | Notes |
|----|--------|----------|-------------|-------|
| SOURCE---JOB-ORCHESTRATOR-DOCS-0001 | DOING (2025-10-29) | Docs Guild | Ensure ./README.md reflects the latest epic deliverables. | Align with ./AGENTS.md |
| SOURCE---JOB-ORCHESTRATOR-DOCS-0001 | DONE (2025-11-05) | Docs Guild | Ensure ./README.md reflects the latest epic deliverables. | README now calls out the 2025-11-01 Authority quota/backfill scope update and required audit metadata. |
| SOURCE---JOB-ORCHESTRATOR-ENG-0001 | TODO | Module Team | Break down epic milestones into actionable stories. | Sync into ../../TASKS.md |
| SOURCE---JOB-ORCHESTRATOR-OPS-0001 | TODO | Ops Guild | Prepare runbooks/observability assets once MVP lands. | Document outputs in ./README.md |

View File

@@ -2,6 +2,10 @@
Scanner analyses container images layer-by-layer, producing deterministic SBOM fragments, diffs, and signed reports.
## Latest updates (2025-11-06)
- Worker/WebService now resolve cache roots and feature flags via `StellaOps.Scanner.Surface.Env`; misconfiguration warnings are documented in `docs/modules/scanner/design/surface-env.md` and surfaced through startup validation.
- Platform events rollout (2025-10-19) continues to publish scanner.report.ready@1 and scanner.scan.completed@1 envelopes with embedded DSSE payloads (see docs/updates/2025-10-19-scanner-policy.md and docs/updates/2025-10-19-platform-events.md). Service and consumer tests should round-trip the canonical samples under docs/events/samples/.
## Responsibilities
- Expose APIs (WebService) for scan orchestration, diffing, and artifact retrieval.
- Run Worker analyzers for OS, language, and native ecosystems with restart-only plug-ins.

View File

@@ -4,7 +4,7 @@
| ID | Status | Owner(s) | Description | Notes |
|----|--------|----------|-------------|-------|
| SCANNER-DOCS-0001 | DOING (2025-10-29) | Docs Guild | Validate that ./README.md aligns with the latest release notes. | See ./AGENTS.md |
| SCANNER-DOCS-0001 | DONE (2025-11-05) | Docs Guild | Validate that ./README.md aligns with the latest release notes. | README now highlights the 2025-10-19 platform-events rollout (scanner.report.ready@1 / scanner.scan.completed@1 DSSE envelopes). |
| SCANNER-DOCS-0002 | DONE (2025-11-02) | Docs Guild | Keep scanner benchmark comparisons (Trivy/Grype/Snyk) and deep-dive matrix current with source references. | Coordinate with docs/benchmarks owners |
| SCANNER-DOCS-0003 | TODO | Docs Guild, Product Guild | Gather Windows/macOS analyzer demand signals and record findings in `docs/benchmarks/scanner/windows-macos-demand.md`. | Coordinate with Product Marketing & Sales enablement |
| SCANNER-ENG-0008 | TODO | EntryTrace Guild, QA Guild | Maintain EntryTrace heuristic cadence per `docs/benchmarks/scanner/scanning-gaps-stella-misses-from-competitors.md`. | Include quarterly pattern review + explain trace updates |

View File

@@ -115,6 +115,17 @@ Failures throw `SurfaceEnvironmentException` with error codes (`SURFACE_ENV_MISS
- **Scanner WebService**: build environment during startup using `AddSurfaceEnvironment`, `AddSurfaceValidation`, `AddSurfaceFileCache`, and `AddSurfaceSecrets`; readiness checks execute the validator runner and scan/report APIs emit Surface CAS pointers derived from the resolved configuration.
- **Zastava Observer/Webhook**: use the same builder; ensure Helm charts set `ZASTAVA_` variables.
- **Scheduler Planner (future)**: treat Surface.Env as read-only input; do not mutate settings.
- `Scanner.Worker` and `Scanner.WebService` automatically bind the `SurfaceCacheOptions.RootDirectory` to `SurfaceEnvironment.Settings.CacheRoot` (2025-11-05); both hosts emit structured warnings (`surface.env.misconfiguration`) when the helper detects missing cache roots, endpoints, or secrets provider settings (2025-11-06).
### 6.1 Misconfiguration warnings
Surface.Env surfaces actionable warnings that appear in structured logs and readiness responses:
- `surface.env.cache_root_missing` emitted when the resolved cache directory does not exist or is not writable. The host attempts to create the directory once; subsequent failures block startup.
- `surface.env.endpoint_unreachable` emitted when `SurfaceFsEndpoint` is missing or not an absolute HTTPS URI.
- `surface.env.secrets_provider_invalid` emitted when the configured secrets provider lacks mandatory fields (e.g., `SCANNER_SURFACE_SECRETS_ROOT` for the `file` provider).
Each warning includes remediation text and a reference to this design document; operations runbooks should treat these warnings as blockers in production and as validation hints in staging.
## 7. Security & Observability

View File

@@ -1,21 +1,32 @@
# StellaOps Signer
Signer validates callers, enforces Proof-of-Entitlement, and produces signed DSSE bundles for SBOMs, reports, and exports.
Signer validates callers, enforces Proof-of-Entitlement, and produces signed DSSE bundles for SBOMs, reports, and exports.
## Latest updates (Sprint 11 · 2025-10-21)
- `/sign/dsse` pipeline landed with Authority OpTok + PoE enforcement, Fulcio/KMS signing modes, and deterministic DSSE bundles ready for Attestor logging.
- `/verify/referrers` endpoint exposes release-integrity checks against scanner OCI referrers so callers can confirm digests before requesting signatures.
- Plan quota enforcement (QPS/concurrency/artifact size) and audit/metrics wiring now align with the Sprint11 signing-chain release.
## Responsibilities
- Enforce plan quotas and PoE before signing artifacts.
- Support keyless and keyful signing backends.
- Emit DSSE payloads consumed by Attestor and downstream bundles.
- Maintain audit trails for all signing operations.
## Responsibilities
- Enforce Proof-of-Entitlement and plan quotas before signing artifacts.
- Support keyless (Fulcio) and keyful (KMS/HSM) signing backends.
- Verify scanner release integrity via OCI referrers prior to issuing signatures.
- Emit DSSE payloads consumed by Attestor/Export Center and maintain comprehensive audit trails.
## Key components
- `StellaOps.Signer` service host.
- Crypto providers under `StellaOps.Cryptography.*`.
## Integrations & dependencies
- Authority for OpTok validation.
- Attestor for transparency logging.
- Export Center and CLI for artifact signing flows.
## Integrations & dependencies
- Authority for OpTok + PoE validation.
- Licensing Service for entitlement introspection.
- OCI registries (Referrers API) for scanner release verification.
- Attestor for transparency logging and Rekor ingestion.
- Export Center and CLI for artifact signing flows.
## API quick reference
- `POST /api/v1/signer/sign/dsse` — validate OpTok/PoE, enforce quotas, return DSSE bundle with signing identity metadata.
- `GET /api/v1/signer/verify/referrers` — report scanner release signer and trust verdict for a supplied image digest.
## Operational notes
- Key management via Authority/DevOps runbooks.

View File

@@ -4,6 +4,6 @@
| ID | Status | Owner(s) | Description | Notes |
|----|--------|----------|-------------|-------|
| SIGNER-DOCS-0001 | DOING (2025-10-29) | Docs Guild | Validate that ./README.md aligns with the latest release notes. | See ./AGENTS.md |
| SIGNER-DOCS-0001 | DONE (2025-11-05) | Docs Guild | Validate that ./README.md aligns with the latest release notes. | README now highlights Sprint 11 signing-chain release (sign/dsse, verify/referrers, quota enforcement). |
| SIGNER-OPS-0001 | TODO | Ops Guild | Review runbooks/observability assets after next sprint demo. | Sync outcomes back to ../../TASKS.md |
| SIGNER-ENG-0001 | TODO | Module Team | Cross-check implementation plan milestones against `/docs/implplan/SPRINT_*.md`. | Update status via ./AGENTS.md workflow |

View File

@@ -18,9 +18,10 @@ Telemetry module captures deployment and operations guidance for the shared obse
- Module-specific dashboards (scheduler, scanner, etc.).
- Security/Compliance for retention policies.
## Operational notes
- Smoke script references (../../ops/devops/telemetry).
- Bundle packaging instructions in ops/devops/telemetry.
## Operational notes
- Smoke script references (../../ops/devops/telemetry).
- Bundle packaging instructions in ops/devops/telemetry.
- Sprint 23 console security sign-off (2025-10-27) added the `console-security.json` Grafana board and burn-rate alert pack—ensure environments import the updated dashboards/alerts referenced in `docs/updates/2025-10-27-console-security-signoff.md`.
## Related resources
- ./operations/collector.md

View File

@@ -4,6 +4,6 @@
| ID | Status | Owner(s) | Description | Notes |
|----|--------|----------|-------------|-------|
| TELEMETRY-DOCS-0001 | DOING (2025-10-29) | Docs Guild | Validate that ./README.md aligns with the latest release notes. | See ./AGENTS.md |
| TELEMETRY-DOCS-0001 | DONE (2025-11-05) | Docs Guild | Validate that ./README.md aligns with the latest release notes. | README now captures the 2025-10-27 console security alert pack (console-security.json, burn-rate alert). |
| TELEMETRY-OPS-0001 | TODO | Ops Guild | Review runbooks/observability assets after next sprint demo. | Sync outcomes back to ../../TASKS.md |
| TELEMETRY-ENG-0001 | TODO | Module Team | Cross-check implementation plan milestones against `/docs/implplan/SPRINT_*.md`. | Update status via ./AGENTS.md workflow |

View File

@@ -1,11 +1,14 @@
# StellaOps Vulnerability Explorer
Vulnerability Explorer delivers policy-aware triage, investigation, and reporting surfaces for effective findings.
## Responsibilities
- Present policy-evaluated findings with advisory, VEX, SBOM, and runtime context.
- Capture triage workflow in an immutable findings ledger with role-based access.
- Provide pivots, exports, and reports for auditors and operations teams.
Vulnerability Explorer delivers policy-aware triage, investigation, and reporting surfaces for effective findings.
## Latest updates (2025-11-03)
- Access controls refresh introduced attachment signing tokens and updated scope guidance (`docs/updates/2025-11-03-vuln-explorer-access-controls.md`). Ensure operator runbooks reference the new Authority scopes (`authority-scopes.md`) and security checklist before enabling attachment uploads.
## Responsibilities
- Present policy-evaluated findings with advisory, VEX, SBOM, and runtime context.
- Capture triage workflow in an immutable findings ledger with role-based access.
- Provide pivots, exports, and reports for auditors and operations teams.
- Integrate explain traces, remediation notes, and offline bundles.
## Key components

View File

@@ -4,6 +4,6 @@
| ID | Status | Owner(s) | Description | Notes |
|----|--------|----------|-------------|-------|
| VULNERABILITY-EXPLORER-DOCS-0001 | DOING (2025-10-29) | Docs Guild | Ensure ./README.md reflects the latest epic deliverables. | Align with ./AGENTS.md |
| VULNERABILITY-EXPLORER-DOCS-0001 | DONE (2025-11-05) | Docs Guild | Ensure ./README.md reflects the latest epic deliverables. | README now includes the 2025-11-03 access-control refresh (attachment signing tokens & scope guidance). |
| VULNERABILITY-EXPLORER-ENG-0001 | TODO | Module Team | Break down epic milestones into actionable stories. | Sync into ../../TASKS.md |
| VULNERABILITY-EXPLORER-OPS-0001 | TODO | Ops Guild | Prepare runbooks/observability assets once MVP lands. | Document outputs in ./README.md |

View File

@@ -0,0 +1,12 @@
# 2025-11-05 Excitor consensus API beta
**Subject:** Excitor consensus export/API preview ships \
**Audience:** Docs Guild, VEX Lens Guild, Policy Engine Guild, Export Center Guild
- Published `docs/modules/excitor/README.md` update capturing the Link-Not-Merge consensus milestone and pointing to the beta API surface.
- Added `docs/vex/consensus-json.md` sample describing the canonical `consensus_state` payload (`rollupStatus`, `sources[]`, `policyRevisionId`, `consensusDigest`) emitted by the preview endpoint.
- Documented DSSE packaging plus Export Center hooks so attested consensus bundles can ride along with devportal/offline exports once Excitor workers emit DSSE manifests (`docs/modules/export-center/devportal-offline.md`).
**Follow-ups**
- [ ] Coordinate with Policy Engine (`POLICY-ENGINE-30-101`) to reference consensus policy weighting knobs before GA.
- [ ] Produce CLI quickstart once `stella vex consensus` verbs land (CLI backlog `CLI-VEX-30-002`).

View File

@@ -0,0 +1,34 @@
# Excitor consensus JSON sample (beta)
```jsonc
{
"vulnId": "CVE-2025-12345",
"productKey": "pkg:maven/org.apache.commons/commons-text@1.11.0",
"rollupStatus": "NOT_AFFECTED",
"sources": [
{
"providerId": "redhat",
"status": "NOT_AFFECTED",
"justification": "component_not_present",
"weight": 0.62,
"lastObserved": "2025-11-04T18:22:31Z",
"accepted": true,
"reason": "trust-tier vendor, signed OpenVEX"
},
{
"providerId": "github",
"status": "AFFECTED",
"justification": null,
"weight": 0.27,
"lastObserved": "2025-11-05T01:12:03Z",
"accepted": false,
"reason": "lower trust tier and stale statement"
}
],
"policyRevisionId": "vex-consensus-policy@2025-11-05",
"evaluatedAt": "2025-11-05T02:05:14Z",
"consensusDigest": "sha256:41f2d96728b24f7a8b7f1251983b8edccd1e0f5781d4a51e51c8e6b20c1fa31a"
}
```
> **Note:** This payload is generated from the beta consensus endpoint and is subject to change prior to GA. Keys and semantics are documented alongside API previews in `docs/modules/excitor/README.md`.

View File

@@ -730,10 +730,11 @@ internal static class CommandFactory
};
activate.Add(activatePolicyIdArgument);
var activateVersionOption = new Option<int>("--version")
{
Description = "Revision version to activate."
};
var activateVersionOption = new Option<int>("--version")
{
Description = "Revision version to activate.",
IsRequired = true
};
var activationNoteOption = new Option<string?>("--note")
{

View File

@@ -49,6 +49,7 @@
| CLI-POLICY-23-004 | TODO | DevEx/CLI Guild | WEB-POLICY-23-001 | Add `stella policy lint` command validating SPL files with compiler diagnostics; support JSON output. | Command returns lint diagnostics; exit codes documented; tests cover error scenarios. |
| CLI-POLICY-23-005 | DOING (2025-10-28) | DevEx/CLI Guild | POLICY-GATEWAY-18-002..003, WEB-POLICY-23-002 | Implement `stella policy activate` with scheduling window, approval enforcement, and summary output. | Activation command integrates with API, handles 2-person rule failures; tests cover success/error. |
> 2025-10-28: CLI command implemented with gateway integration (`policy activate`), interactive summary output, retry-aware metrics, and exit codes (0 success, 75 pending second approval). Tests cover success/pending/error paths.
> 2025-11-06: Tightened required `--version` parsing, added scheduled activation handling coverage, and expanded tests to validate timestamp normalization.
| CLI-POLICY-23-006 | TODO | DevEx/CLI Guild | WEB-POLICY-23-004 | Provide `stella policy history` and `stella policy explain` commands to pull run history and explanation trees. | Commands output JSON/table; integration tests with fixtures; docs updated. |
## Graph & Vuln Explorer v1

View File

@@ -1811,11 +1811,11 @@ spec:
}
[Fact]
public async Task HandlePolicyActivateAsync_PendingSecondApprovalSetsExitCode()
{
var originalExit = Environment.ExitCode;
var backend = new StubBackendClient(new JobTriggerResult(true, "ok", null, null));
public async Task HandlePolicyActivateAsync_PendingSecondApprovalSetsExitCode()
{
var originalExit = Environment.ExitCode;
var backend = new StubBackendClient(new JobTriggerResult(true, "ok", null, null));
backend.ActivationResult = new PolicyActivationResult(
"pending_second_approval",
new PolicyActivationRevision(
@@ -1852,15 +1852,65 @@ spec:
finally
{
Environment.ExitCode = originalExit;
}
}
[Fact]
public async Task HandlePolicyActivateAsync_MapsErrorCodes()
{
var originalExit = Environment.ExitCode;
var backend = new StubBackendClient(new JobTriggerResult(true, "ok", null, null))
}
}
[Fact]
public async Task HandlePolicyActivateAsync_ParsesScheduledTimestamp()
{
var originalExit = Environment.ExitCode;
var backend = new StubBackendClient(new JobTriggerResult(true, "ok", null, null));
backend.ActivationResult = new PolicyActivationResult(
"scheduled",
new PolicyActivationRevision(
"P-8",
5,
"approved",
false,
DateTimeOffset.Parse("2025-12-01T00:30:00Z", CultureInfo.InvariantCulture),
null,
new ReadOnlyCollection<PolicyActivationApproval>(Array.Empty<PolicyActivationApproval>())));
var provider = BuildServiceProvider(backend);
try
{
const string scheduledValue = "2025-12-01T03:00:00+02:00";
await CommandHandlers.HandlePolicyActivateAsync(
provider,
policyId: "P-8",
version: 5,
note: null,
runNow: false,
scheduledAt: scheduledValue,
priority: null,
rollback: false,
incidentId: null,
verbose: false,
cancellationToken: CancellationToken.None);
Assert.Equal(0, Environment.ExitCode);
Assert.NotNull(backend.LastPolicyActivation);
var activation = backend.LastPolicyActivation!.Value;
Assert.False(activation.Request.RunNow);
var expected = DateTimeOffset.Parse(
scheduledValue,
CultureInfo.InvariantCulture,
DateTimeStyles.AssumeUniversal | DateTimeStyles.AdjustToUniversal);
Assert.Equal(expected, activation.Request.ScheduledAt);
}
finally
{
Environment.ExitCode = originalExit;
}
}
[Fact]
public async Task HandlePolicyActivateAsync_MapsErrorCodes()
{
var originalExit = Environment.ExitCode;
var backend = new StubBackendClient(new JobTriggerResult(true, "ok", null, null))
{
ActivationException = new PolicyApiException("Revision not approved", HttpStatusCode.BadRequest, "ERR_POL_002")
};

View File

@@ -52,9 +52,11 @@ internal static class JobRegistrationExtensions
new("source:vndr-oracle:parse", "StellaOps.Concelier.Connector.Vndr.Oracle.OracleParseJob", "StellaOps.Concelier.Connector.Vndr.Oracle", TimeSpan.FromMinutes(15), TimeSpan.FromMinutes(5)),
new("source:vndr-oracle:map", "StellaOps.Concelier.Connector.Vndr.Oracle.OracleMapJob", "StellaOps.Concelier.Connector.Vndr.Oracle", TimeSpan.FromMinutes(15), TimeSpan.FromMinutes(5)),
new("export:json", "StellaOps.Concelier.Exporter.Json.JsonExportJob", "StellaOps.Concelier.Exporter.Json", TimeSpan.FromMinutes(10), TimeSpan.FromMinutes(5)),
new("export:json", "StellaOps.Concelier.Exporter.Json.JsonExportJob", "StellaOps.Concelier.Exporter.Json", TimeSpan.FromMinutes(10), TimeSpan.FromMinutes(5)),
new("export:trivy-db", "StellaOps.Concelier.Exporter.TrivyDb.TrivyDbExportJob", "StellaOps.Concelier.Exporter.TrivyDb", TimeSpan.FromMinutes(20), TimeSpan.FromMinutes(10)),
#pragma warning disable CS0618, CONCELIER0001 // Legacy merge job remains available until MERGE-LNM-21-002 completes.
new("merge:reconcile", "StellaOps.Concelier.Merge.Jobs.MergeReconcileJob", "StellaOps.Concelier.Merge", TimeSpan.FromMinutes(15), TimeSpan.FromMinutes(5))
#pragma warning restore CS0618, CONCELIER0001
};
public static IServiceCollection AddBuiltInConcelierJobs(this IServiceCollection services)

View File

@@ -15,6 +15,8 @@ public sealed class ConcelierOptions
public AuthorityOptions Authority { get; set; } = new();
public MirrorOptions Mirror { get; set; } = new();
public FeaturesOptions Features { get; set; } = new();
public sealed class StorageOptions
{
@@ -135,4 +137,13 @@ public sealed class ConcelierOptions
public int MaxDownloadRequestsPerHour { get; set; } = 1200;
}
public sealed class FeaturesOptions
{
public bool NoMergeEnabled { get; set; }
public bool LnmShadowWrites { get; set; } = true;
public IList<string> MergeJobAllowlist { get; } = new List<string>();
}
}

View File

@@ -17,7 +17,8 @@ public static class ConcelierOptionsPostConfigure
{
ArgumentNullException.ThrowIfNull(options);
options.Authority ??= new ConcelierOptions.AuthorityOptions();
options.Authority ??= new ConcelierOptions.AuthorityOptions();
options.Features ??= new ConcelierOptions.FeaturesOptions();
var authority = options.Authority;
if (string.IsNullOrWhiteSpace(authority.ClientSecret)

View File

@@ -1,3 +1,4 @@
using System;
using System.Collections.Generic;
using System.Collections.Immutable;
using System.Globalization;
@@ -98,9 +99,36 @@ builder.Services.AddConcelierLinksetMappers();
builder.Services.AddAdvisoryRawServices();
builder.Services.AddSingleton<IAdvisoryObservationQueryService, AdvisoryObservationQueryService>();
builder.Services.AddMergeModule(builder.Configuration);
var features = concelierOptions.Features ?? new ConcelierOptions.FeaturesOptions();
if (!features.NoMergeEnabled)
{
#pragma warning disable CS0618, CONCELIER0001, CONCELIER0002 // Legacy merge service is intentionally supported behind a feature toggle.
builder.Services.AddMergeModule(builder.Configuration);
#pragma warning restore CS0618, CONCELIER0001, CONCELIER0002
}
builder.Services.AddJobScheduler();
builder.Services.AddBuiltInConcelierJobs();
builder.Services.PostConfigure<JobSchedulerOptions>(options =>
{
if (features.NoMergeEnabled)
{
options.Definitions.Remove("merge:reconcile");
return;
}
if (features.MergeJobAllowlist is { Count: > 0 })
{
var allowMergeJob = features.MergeJobAllowlist.Any(value =>
string.Equals(value, "merge:reconcile", StringComparison.OrdinalIgnoreCase));
if (!allowMergeJob)
{
options.Definitions.Remove("merge:reconcile");
}
}
});
builder.Services.AddSingleton<OpenApiDiscoveryDocumentProvider>();
builder.Services.AddSingleton<ServiceStatus>(sp => new ServiceStatus(sp.GetRequiredService<TimeProvider>()));
@@ -183,7 +211,7 @@ if (authorityConfigured)
builder.Services.AddAuthorization(options =>
{
options.AddStellaOpsScopePolicy(JobsPolicyName, concelierOptions.Authority.RequiredScopes.ToArray());
options.AddStellaOpsScopePolicy(ObservationsPolicyName, StellaOpsScopes.VulnRead);
options.AddStellaOpsScopePolicy(ObservationsPolicyName, StellaOpsScopes.VulnView);
options.AddStellaOpsScopePolicy(AdvisoryIngestPolicyName, StellaOpsScopes.AdvisoryIngest);
options.AddStellaOpsScopePolicy(AdvisoryReadPolicyName, StellaOpsScopes.AdvisoryRead);
options.AddStellaOpsScopePolicy(AocVerifyPolicyName, StellaOpsScopes.AdvisoryRead, StellaOpsScopes.AocVerify);
@@ -197,6 +225,11 @@ builder.Services.AddEndpointsApiExplorer();
var app = builder.Build();
if (features.NoMergeEnabled)
{
app.Logger.LogWarning("Legacy merge module disabled via concelier:features:noMergeEnabled; Link-Not-Merge mode active.");
}
var resolvedConcelierOptions = app.Services.GetRequiredService<IOptions<ConcelierOptions>>().Value;
var resolvedAuthority = resolvedConcelierOptions.Authority ?? new ConcelierOptions.AuthorityOptions();
authorityConfigured = resolvedAuthority.Enabled;

View File

@@ -35,5 +35,8 @@
<ProjectReference Include="../../Authority/StellaOps.Authority/StellaOps.Auth.ServerIntegration/StellaOps.Auth.ServerIntegration.csproj" />
<ProjectReference Include="../../Aoc/__Libraries/StellaOps.Aoc/StellaOps.Aoc.csproj" />
<ProjectReference Include="../../Aoc/__Libraries/StellaOps.Aoc.AspNetCore/StellaOps.Aoc.AspNetCore.csproj" />
<ProjectReference Include="../__Analyzers/StellaOps.Concelier.Analyzers/StellaOps.Concelier.Analyzers.csproj"
OutputItemType="Analyzer"
ReferenceOutputAssembly="false" />
</ItemGroup>
</Project>

File diff suppressed because it is too large Load Diff

View File

@@ -0,0 +1,2 @@
; Shipped analyzer releases

View File

@@ -0,0 +1,9 @@
## Release History
### Unreleased
#### New Rules
Rule ID | Title | Notes
--------|-------|------
CONCELIER0002 | Legacy merge pipeline is disabled | Flags usage of `AddMergeModule` and `AdvisoryMergeService`.

View File

@@ -0,0 +1,152 @@
using System;
using System.Collections.Immutable;
using Microsoft.CodeAnalysis;
using Microsoft.CodeAnalysis.Diagnostics;
using Microsoft.CodeAnalysis.Operations;
namespace StellaOps.Concelier.Analyzers;
/// <summary>
/// Analyzer that flags usages of the legacy merge service APIs.
/// </summary>
[DiagnosticAnalyzer(LanguageNames.CSharp)]
public sealed class NoMergeUsageAnalyzer : DiagnosticAnalyzer
{
/// <summary>
/// Diagnostic identifier for legacy merge usage violations.
/// </summary>
public const string DiagnosticId = "CONCELIER0002";
private const string Category = "Usage";
private const string MergeExtensionType = "StellaOps.Concelier.Merge.MergeServiceCollectionExtensions";
private const string MergeServiceType = "StellaOps.Concelier.Merge.Services.AdvisoryMergeService";
private static readonly LocalizableString Title = "Legacy merge pipeline is disabled";
private static readonly LocalizableString MessageFormat = "Do not reference the legacy Concelier merge pipeline (type '{0}')";
private static readonly LocalizableString Description =
"The legacy Concelier merge service is deprecated under MERGE-LNM-21-002. "
+ "Switch to observation/linkset APIs or guard calls behind the concelier:features:noMergeEnabled toggle.";
private static readonly DiagnosticDescriptor Rule = new(
DiagnosticId,
Title,
MessageFormat,
Category,
DiagnosticSeverity.Error,
isEnabledByDefault: true,
description: Description,
helpLinkUri: "https://stella-ops.org/docs/migration/no-merge");
/// <inheritdoc />
public override ImmutableArray<DiagnosticDescriptor> SupportedDiagnostics => ImmutableArray.Create(Rule);
/// <inheritdoc />
public override void Initialize(AnalysisContext context)
{
context.ConfigureGeneratedCodeAnalysis(GeneratedCodeAnalysisFlags.None);
context.EnableConcurrentExecution();
context.RegisterOperationAction(AnalyzeInvocation, OperationKind.Invocation);
context.RegisterOperationAction(AnalyzeObjectCreation, OperationKind.ObjectCreation);
}
private static void AnalyzeInvocation(OperationAnalysisContext context)
{
if (context.Operation is not IInvocationOperation invocation)
{
return;
}
var targetMethod = invocation.TargetMethod;
if (targetMethod is null)
{
return;
}
if (!SymbolEquals(targetMethod.ContainingType, MergeExtensionType))
{
return;
}
if (!string.Equals(targetMethod.Name, "AddMergeModule", StringComparison.Ordinal))
{
return;
}
if (IsAllowedAssembly(context.ContainingSymbol.ContainingAssembly))
{
return;
}
ReportDiagnostic(context, invocation.Syntax, $"{MergeExtensionType}.{targetMethod.Name}");
}
private static void AnalyzeObjectCreation(OperationAnalysisContext context)
{
if (context.Operation is not IObjectCreationOperation creation)
{
return;
}
var createdType = creation.Type;
if (createdType is null || !SymbolEquals(createdType, MergeServiceType))
{
return;
}
if (IsAllowedAssembly(context.ContainingSymbol.ContainingAssembly))
{
return;
}
ReportDiagnostic(context, creation.Syntax, MergeServiceType);
}
private static bool SymbolEquals(ITypeSymbol? symbol, string fullName)
{
if (symbol is null)
{
return false;
}
var display = symbol.ToDisplayString(SymbolDisplayFormat.FullyQualifiedFormat);
if (display.StartsWith("global::", StringComparison.Ordinal))
{
display = display.Substring("global::".Length);
}
return string.Equals(display, fullName, StringComparison.Ordinal);
}
private static bool IsAllowedAssembly(IAssemblySymbol? assemblySymbol)
{
if (assemblySymbol is null)
{
return false;
}
var assemblyName = assemblySymbol.Name;
if (string.IsNullOrWhiteSpace(assemblyName))
{
return false;
}
if (assemblyName.StartsWith("StellaOps.Concelier.Merge", StringComparison.Ordinal))
{
return true;
}
if (assemblyName.EndsWith(".Analyzers", StringComparison.Ordinal))
{
return true;
}
return false;
}
private static void ReportDiagnostic(OperationAnalysisContext context, SyntaxNode syntax, string targetName)
{
var diagnostic = Diagnostic.Create(Rule, syntax.GetLocation(), targetName);
context.ReportDiagnostic(diagnostic);
}
}

View File

@@ -0,0 +1,19 @@
<Project Sdk="Microsoft.NET.Sdk">
<PropertyGroup>
<TargetFramework>netstandard2.0</TargetFramework>
<AssemblyName>StellaOps.Concelier.Analyzers</AssemblyName>
<RootNamespace>StellaOps.Concelier.Analyzers</RootNamespace>
<Nullable>enable</Nullable>
<LangVersion>latest</LangVersion>
<IncludeBuildOutput>false</IncludeBuildOutput>
<GenerateDocumentationFile>true</GenerateDocumentationFile>
<EnforceExtendedAnalyzerRules>true</EnforceExtendedAnalyzerRules>
</PropertyGroup>
<ItemGroup>
<PackageReference Include="Microsoft.CodeAnalysis.CSharp" Version="4.9.2" PrivateAssets="all" />
<PackageReference Include="Microsoft.CodeAnalysis.Analyzers" Version="3.11.0" PrivateAssets="all" />
</ItemGroup>
</Project>

View File

@@ -114,10 +114,10 @@ internal sealed class AdvisoryObservationFactory : IAdvisoryObservationFactory
private static AdvisoryObservationLinkset CreateLinkset(RawIdentifiers identifiers, RawLinkset linkset)
{
var aliases = NormalizeAliases(identifiers, linkset);
var purls = NormalizePackageUrls(linkset.PackageUrls);
var cpes = NormalizeCpes(linkset.Cpes);
var references = NormalizeReferences(linkset.References);
var aliases = CollectAliases(identifiers, linkset);
var purls = CollectValues(linkset.PackageUrls);
var cpes = CollectValues(linkset.Cpes);
var references = CollectReferences(linkset.References);
return new AdvisoryObservationLinkset(aliases, purls, cpes, references);
}
@@ -170,124 +170,91 @@ internal sealed class AdvisoryObservationFactory : IAdvisoryObservationFactory
};
}
private static IEnumerable<string> NormalizeAliases(RawIdentifiers identifiers, RawLinkset linkset)
{
var aliases = new HashSet<string>(StringComparer.OrdinalIgnoreCase);
if (LinksetNormalization.TryNormalizeAlias(identifiers.PrimaryId, out var primary))
{
aliases.Add(primary);
}
foreach (var alias in identifiers.Aliases)
{
if (LinksetNormalization.TryNormalizeAlias(alias, out var normalized))
{
aliases.Add(normalized);
}
}
foreach (var alias in linkset.Aliases)
{
if (LinksetNormalization.TryNormalizeAlias(alias, out var normalized))
{
aliases.Add(normalized);
}
}
foreach (var note in linkset.Notes)
{
if (!string.IsNullOrWhiteSpace(note.Value)
&& LinksetNormalization.TryNormalizeAlias(note.Value, out var normalized))
{
aliases.Add(normalized);
}
}
return aliases
.OrderBy(static value => value, StringComparer.OrdinalIgnoreCase)
.ToImmutableArray();
}
private static IEnumerable<string> NormalizePackageUrls(ImmutableArray<string> packageUrls)
{
if (packageUrls.IsDefaultOrEmpty)
{
return ImmutableArray<string>.Empty;
}
var set = new HashSet<string>(StringComparer.Ordinal);
foreach (var candidate in packageUrls)
{
if (!LinksetNormalization.TryNormalizePackageUrl(candidate, out var normalized) || string.IsNullOrEmpty(normalized))
{
continue;
}
set.Add(normalized);
}
return set
.OrderBy(static value => value, StringComparer.Ordinal)
.ToImmutableArray();
}
private static IEnumerable<string> NormalizeCpes(ImmutableArray<string> cpes)
{
if (cpes.IsDefaultOrEmpty)
{
return ImmutableArray<string>.Empty;
}
var set = new HashSet<string>(StringComparer.Ordinal);
foreach (var cpe in cpes)
{
if (!LinksetNormalization.TryNormalizeCpe(cpe, out var normalized) || string.IsNullOrEmpty(normalized))
{
continue;
}
set.Add(normalized);
}
return set
.OrderBy(static value => value, StringComparer.Ordinal)
.ToImmutableArray();
}
private static IEnumerable<AdvisoryObservationReference> NormalizeReferences(ImmutableArray<RawReference> references)
{
if (references.IsDefaultOrEmpty)
{
return ImmutableArray<AdvisoryObservationReference>.Empty;
}
var seen = new HashSet<string>(StringComparer.OrdinalIgnoreCase);
var list = new List<AdvisoryObservationReference>();
foreach (var reference in references)
{
var normalized = LinksetNormalization.TryCreateReference(reference.Type, reference.Url);
if (normalized is null)
{
continue;
}
if (!seen.Add(normalized.Url))
{
continue;
}
list.Add(normalized);
}
return list
.OrderBy(static reference => reference.Type, StringComparer.Ordinal)
.ThenBy(static reference => reference.Url, StringComparer.Ordinal)
.ToImmutableArray();
}
private static IEnumerable<string> CollectAliases(RawIdentifiers identifiers, RawLinkset linkset)
{
var results = new List<string>();
AddAlias(results, identifiers.PrimaryId);
AddRange(results, identifiers.Aliases);
AddRange(results, linkset.Aliases);
foreach (var note in linkset.Notes)
{
if (string.IsNullOrWhiteSpace(note.Value))
{
continue;
}
results.Add(note.Value.Trim());
}
return results;
static void AddAlias(ICollection<string> target, string? value)
{
if (string.IsNullOrWhiteSpace(value))
{
return;
}
target.Add(value.Trim());
}
static void AddRange(ICollection<string> target, ImmutableArray<string> values)
{
if (values.IsDefaultOrEmpty)
{
return;
}
foreach (var value in values)
{
AddAlias(target, value);
}
}
}
private static IEnumerable<string> CollectValues(ImmutableArray<string> values)
{
if (values.IsDefaultOrEmpty)
{
return ImmutableArray<string>.Empty;
}
var list = new List<string>(values.Length);
foreach (var value in values)
{
if (string.IsNullOrWhiteSpace(value))
{
continue;
}
list.Add(value.Trim());
}
return list;
}
private static IEnumerable<AdvisoryObservationReference> CollectReferences(ImmutableArray<RawReference> references)
{
if (references.IsDefaultOrEmpty)
{
return ImmutableArray<AdvisoryObservationReference>.Empty;
}
var list = new List<AdvisoryObservationReference>(references.Length);
foreach (var reference in references)
{
if (string.IsNullOrWhiteSpace(reference.Type) || string.IsNullOrWhiteSpace(reference.Url))
{
continue;
}
list.Add(new AdvisoryObservationReference(reference.Type.Trim(), reference.Url.Trim()));
}
return list;
}
private static ImmutableDictionary<string, string> CreateAttributes(AdvisoryRawDocument rawDocument)
{

View File

@@ -9,7 +9,7 @@
> Docs alignment (2025-10-26): Linkset expectations detailed in AOC reference §4 and policy-engine architecture §2.1.
> 2025-10-28: Advisory raw ingestion now strips client-supplied supersedes hints, logs ignored pointers, and surfaces repository-supplied supersedes identifiers; service tests cover duplicate handling and append-only semantics.
> Docs alignment (2025-10-26): Deployment guide + observability guide describe supersedes metrics; ensure implementation emits `aoc_violation_total` on failure.
| CONCELIER-CORE-AOC-19-004 `Remove ingestion normalization` | DOING (2025-10-28) | Concelier Core Guild | CONCELIER-CORE-AOC-19-002, POLICY-AOC-19-003 | Strip normalization/dedup/severity logic from ingestion pipelines, delegate derived computations to Policy Engine, and update exporters/tests to consume raw documents only.<br>2025-10-29 19:05Z: Audit completed for `AdvisoryRawService`/Mongo repo to confirm alias order/dedup removal persists; identified remaining normalization in observation/linkset factory that will be revised to surface raw duplicates for Policy ingestion. Change sketch + regression matrix drafted under `docs/dev/aoc-normalization-removal-notes.md` (pending commit).<br>2025-10-31 20:45Z: Added raw linkset projection to observations/storage, exposing canonical+raw views, refreshed fixtures/tests, and documented behaviour in models/doc factory.<br>2025-10-31 21:10Z: Coordinated with Policy Engine (POLICY-ENGINE-20-003) on adoption timeline; backfill + consumer readiness tracked in `docs/dev/raw-linkset-backfill-plan.md`. |
| CONCELIER-CORE-AOC-19-004 `Remove ingestion normalization` | DOING (2025-10-28) | Concelier Core Guild | CONCELIER-CORE-AOC-19-002, POLICY-AOC-19-003 | Strip normalization/dedup/severity logic from ingestion pipelines, delegate derived computations to Policy Engine, and update exporters/tests to consume raw documents only.<br>2025-10-29 19:05Z: Audit completed for `AdvisoryRawService`/Mongo repo to confirm alias order/dedup removal persists; identified remaining normalization in observation/linkset factory that will be revised to surface raw duplicates for Policy ingestion. Change sketch + regression matrix drafted under `docs/dev/aoc-normalization-removal-notes.md` (pending commit).<br>2025-10-31 20:45Z: Added raw linkset projection to observations/storage, exposing canonical+raw views, refreshed fixtures/tests, and documented behaviour in models/doc factory.<br>2025-10-31 21:10Z: Coordinated with Policy Engine (POLICY-ENGINE-20-003) on adoption timeline; backfill + consumer readiness tracked in `docs/dev/raw-linkset-backfill-plan.md`.<br>2025-11-05 14:25Z: Resuming to document merge-dependent normalization paths and prepare implementation notes for `noMergeEnabled` gating before code changes land.<br>2025-11-05 19:20Z: Observation factory/linkset now preserve upstream ordering + duplicates; canonicalisation responsibility shifts to downstream consumers with refreshed unit coverage.<br>2025-11-06 16:10Z: Updated AOC reference/backfill docs with raw vs canonical guidance and cross-linked analyzer guardrails. |
> Docs alignment (2025-10-26): Architecture overview emphasises policy-only derivation; coordinate with Policy Engine guild for rollout.
> 2025-10-29: `AdvisoryRawService` now preserves upstream alias/linkset ordering (trim-only) and updated AOC documentation reflects the behaviour; follow-up to ensure policy consumers handle duplicates remains open.
| CONCELIER-CORE-AOC-19-013 `Authority tenant scope smoke coverage` | TODO | Concelier Core Guild | AUTH-AOC-19-002 | Extend Concelier smoke/e2e fixtures to configure `requiredTenants` and assert cross-tenant rejection with updated Authority tokens. | Coordinate deliverable so Authority docs (`AUTH-AOC-19-003`) can close once tests are in place. |

View File

@@ -5,9 +5,10 @@ using Microsoft.Extensions.Logging;
using StellaOps.Concelier.Core.Jobs;
using StellaOps.Concelier.Merge.Services;
namespace StellaOps.Concelier.Merge.Jobs;
public sealed class MergeReconcileJob : IJob
namespace StellaOps.Concelier.Merge.Jobs;
[Obsolete("MergeReconcileJob is deprecated; Link-Not-Merge supersedes merge scheduling. Disable via concelier:features:noMergeEnabled. Tracking MERGE-LNM-21-002.", DiagnosticId = "CONCELIER0001", UrlFormat = "https://stella-ops.org/docs/migration/no-merge")]
public sealed class MergeReconcileJob : IJob
{
private readonly AdvisoryMergeService _mergeService;
private readonly ILogger<MergeReconcileJob> _logger;

View File

@@ -1,15 +1,17 @@
using System;
using Microsoft.Extensions.Configuration;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.DependencyInjection.Extensions;
using Microsoft.Extensions.Logging;
using StellaOps.Concelier.Core;
using StellaOps.Concelier.Merge.Jobs;
using StellaOps.Concelier.Merge.Jobs;
using StellaOps.Concelier.Merge.Options;
using StellaOps.Concelier.Merge.Services;
namespace StellaOps.Concelier.Merge;
public static class MergeServiceCollectionExtensions
namespace StellaOps.Concelier.Merge;
[Obsolete("Legacy merge module is deprecated; prefer Link-Not-Merge linkset pipelines. Track MERGE-LNM-21-002 and set concelier:features:noMergeEnabled=true to disable registration.", DiagnosticId = "CONCELIER0001", UrlFormat = "https://stella-ops.org/docs/migration/no-merge")]
public static class MergeServiceCollectionExtensions
{
public static IServiceCollection AddMergeModule(this IServiceCollection services, IConfiguration configuration)
{
@@ -34,10 +36,12 @@ public static class MergeServiceCollectionExtensions
return new AdvisoryPrecedenceMerger(resolver, options, timeProvider, logger);
});
services.TryAddSingleton<MergeEventWriter>();
services.TryAddSingleton<AdvisoryMergeService>();
services.AddTransient<MergeReconcileJob>();
return services;
}
}
#pragma warning disable CS0618 // Legacy merge services are marked obsolete.
services.TryAddSingleton<MergeEventWriter>();
services.TryAddSingleton<AdvisoryMergeService>();
services.AddTransient<MergeReconcileJob>();
#pragma warning restore CS0618
return services;
}
}

View File

@@ -14,9 +14,10 @@ using StellaOps.Concelier.Storage.Mongo.Aliases;
using StellaOps.Concelier.Storage.Mongo.MergeEvents;
using System.Text.Json;
namespace StellaOps.Concelier.Merge.Services;
public sealed class AdvisoryMergeService
namespace StellaOps.Concelier.Merge.Services;
[Obsolete("AdvisoryMergeService is deprecated. Transition callers to Link-Not-Merge observation/linkset APIs (MERGE-LNM-21-002) and enable concelier:features:noMergeEnabled when ready.", DiagnosticId = "CONCELIER0001", UrlFormat = "https://stella-ops.org/docs/migration/no-merge")]
public sealed class AdvisoryMergeService
{
private static readonly Meter MergeMeter = new("StellaOps.Concelier.Merge");
private static readonly Counter<long> AliasCollisionCounter = MergeMeter.CreateCounter<long>(

View File

@@ -10,6 +10,6 @@
| Task | Owner(s) | Depends on | Notes |
|---|---|---|---|
|MERGE-LNM-21-001 Migration plan authoring|BE-Merge, Architecture Guild|CONCELIER-LNM-21-101|**DONE (2025-11-03)** Authored `docs/migration/no-merge.md` with rollout phases, backfill/validation checklists, rollback guidance, and ownership matrix for the Link-Not-Merge cutover.|
|MERGE-LNM-21-002 Merge service deprecation|BE-Merge|MERGE-LNM-21-001|**DOING (2025-11-03)** Auditing service registrations, DI bindings, and tests consuming `AdvisoryMergeService`; drafting deprecation plan and analyzer scope prior to code removal.|
|MERGE-LNM-21-002 Merge service deprecation|BE-Merge|MERGE-LNM-21-001|**DOING (2025-11-03)** Auditing service registrations, DI bindings, and tests consuming `AdvisoryMergeService`; drafting deprecation plan and analyzer scope prior to code removal.<br>2025-11-05 14:42Z: Implementing `concelier:features:noMergeEnabled` gate, merge job allowlist checks, `[Obsolete]` markings, and analyzer scaffolding to steer consumers toward linkset APIs.<br>2025-11-06 16:10Z: Introduced Roslyn analyzer (`CONCELIER0002`) referenced by Concelier WebService + tests, documented suppression guidance, and updated migration playbook.|
> 2025-11-03: Catalogued call sites (WebService Program `AddMergeModule`, built-in job registration `merge:reconcile`, `MergeReconcileJob`) and confirmed unit tests are the only direct `MergeAsync` callers; next step is to define analyzer + replacement observability coverage.
|MERGE-LNM-21-003 Determinism/test updates|QA Guild, BE-Merge|MERGE-LNM-21-002|Replace merge determinism suites with observation/linkset regression tests verifying no data mutation and conflicts remain visible.|

View File

@@ -280,57 +280,60 @@ public sealed record AdvisoryObservationLinkset
IEnumerable<string>? cpes,
IEnumerable<AdvisoryObservationReference>? references)
{
Aliases = NormalizeStringSet(aliases, toLower: true);
Purls = NormalizeStringSet(purls);
Cpes = NormalizeStringSet(cpes);
References = NormalizeReferences(references);
}
Aliases = ToImmutableArray(aliases);
Purls = ToImmutableArray(purls);
Cpes = ToImmutableArray(cpes);
References = ToImmutableReferences(references);
}
public ImmutableArray<string> Aliases { get; }
public ImmutableArray<string> Purls { get; }
public ImmutableArray<string> Aliases { get; }
public ImmutableArray<string> Purls { get; }
public ImmutableArray<string> Cpes { get; }
public ImmutableArray<AdvisoryObservationReference> References { get; }
private static ImmutableArray<string> NormalizeStringSet(IEnumerable<string>? values, bool toLower = false)
{
if (values is null)
{
return ImmutableArray<string>.Empty;
}
var list = new List<string>();
foreach (var value in values)
{
var trimmed = Validation.TrimToNull(value);
if (trimmed is null)
{
continue;
}
list.Add(toLower ? trimmed.ToLowerInvariant() : trimmed);
}
return list
.Distinct(StringComparer.Ordinal)
.OrderBy(static v => v, StringComparer.Ordinal)
.ToImmutableArray();
}
private static ImmutableArray<AdvisoryObservationReference> NormalizeReferences(IEnumerable<AdvisoryObservationReference>? references)
{
if (references is null)
{
return ImmutableArray<AdvisoryObservationReference>.Empty;
}
return references
.Where(static reference => reference is not null)
.Distinct()
.OrderBy(static reference => reference.Type, StringComparer.Ordinal)
.ThenBy(static reference => reference.Url, StringComparer.Ordinal)
.ToImmutableArray();
}
}
public ImmutableArray<string> Cpes { get; }
public ImmutableArray<AdvisoryObservationReference> References { get; }
private static ImmutableArray<string> ToImmutableArray(IEnumerable<string>? values)
{
if (values is null)
{
return ImmutableArray<string>.Empty;
}
var builder = ImmutableArray.CreateBuilder<string>();
foreach (var value in values)
{
var trimmed = Validation.TrimToNull(value);
if (trimmed is null)
{
continue;
}
builder.Add(trimmed);
}
return builder.Count == 0 ? ImmutableArray<string>.Empty : builder.ToImmutable();
}
private static ImmutableArray<AdvisoryObservationReference> ToImmutableReferences(IEnumerable<AdvisoryObservationReference>? references)
{
if (references is null)
{
return ImmutableArray<AdvisoryObservationReference>.Empty;
}
var builder = ImmutableArray.CreateBuilder<AdvisoryObservationReference>();
foreach (var reference in references)
{
if (reference is null)
{
continue;
}
builder.Add(reference);
}
return builder.Count == 0 ? ImmutableArray<AdvisoryObservationReference>.Empty : builder.ToImmutable();
}
}

View File

@@ -13,11 +13,11 @@ public sealed class AdvisoryObservationFactoryTests
private static readonly DateTimeOffset SampleTimestamp = DateTimeOffset.Parse("2025-10-26T12:34:56Z");
[Fact]
public void Create_NormalizesLinksetIdentifiersAndReferences()
{
var factory = new AdvisoryObservationFactory();
var rawDocument = BuildRawDocument(
identifiers: new RawIdentifiers(
public void Create_PreservesLinksetOrderAndDuplicates()
{
var factory = new AdvisoryObservationFactory();
var rawDocument = BuildRawDocument(
identifiers: new RawIdentifiers(
Aliases: ImmutableArray.Create(" CVE-2025-0001 ", "ghsa-XXXX-YYYY"),
PrimaryId: "GHSA-XXXX-YYYY"),
linkset: new RawLinkset
@@ -29,16 +29,27 @@ public sealed class AdvisoryObservationFactoryTests
new RawReference("Advisory", " https://example.test/advisory "),
new RawReference("ADVISORY", "https://example.test/advisory"))
});
var observation = factory.Create(rawDocument, SampleTimestamp);
Assert.Equal(SampleTimestamp, observation.CreatedAt);
Assert.Equal(new[] { "cve-2025-0001", "ghsa-xxxx-yyyy" }, observation.Linkset.Aliases);
Assert.Equal(new[] { "pkg:npm/left-pad@1.0.0" }, observation.Linkset.Purls);
Assert.Equal(new[] { "cpe:2.3:a:example:product:1.0:*:*:*:*:*:*:*" }, observation.Linkset.Cpes);
var reference = Assert.Single(observation.Linkset.References);
Assert.Equal("advisory", reference.Type);
Assert.Equal("https://example.test/advisory", reference.Url);
var observation = factory.Create(rawDocument, SampleTimestamp);
Assert.Equal(SampleTimestamp, observation.CreatedAt);
Assert.Equal(
new[] { "GHSA-XXXX-YYYY", "CVE-2025-0001", "ghsa-XXXX-YYYY", "CVE-2025-0001" },
observation.Linkset.Aliases);
Assert.Equal(
new[] { "pkg:NPM/left-pad@1.0.0", "pkg:npm/left-pad@1.0.0?foo=bar" },
observation.Linkset.Purls);
Assert.Equal(
new[] { "cpe:/a:Example:Product:1.0", "cpe:/a:example:product:1.0" },
observation.Linkset.Cpes);
Assert.Equal(2, observation.Linkset.References.Length);
Assert.All(
observation.Linkset.References,
reference =>
{
Assert.Equal("advisory", reference.Type);
Assert.Equal("https://example.test/advisory", reference.Url);
});
Assert.Equal(
new[] { "GHSA-XXXX-YYYY", " CVE-2025-0001 ", "ghsa-XXXX-YYYY", " CVE-2025-0001 " },

View File

@@ -1,3 +1,4 @@
using System;
using System.Collections.Immutable;
using System.Linq;
using System.Text.Json.Nodes;
@@ -52,9 +53,9 @@ public sealed class AdvisoryObservationQueryServiceTests
Assert.Equal("tenant-a:osv:beta:1", result.Observations[0].ObservationId);
Assert.Equal("tenant-a:ghsa:alpha:1", result.Observations[1].ObservationId);
Assert.Equal(
new[] { "cve-2025-0001", "cve-2025-0002", "ghsa-xyzz" },
result.Linkset.Aliases);
Assert.Equal(
new[] { "CVE-2025-0001", "CVE-2025-0002", "GHSA-xyzz" },
result.Linkset.Aliases);
Assert.Equal(
new[] { "pkg:npm/package-a@1.0.0", "pkg:pypi/package-b@2.0.0" },
@@ -103,8 +104,11 @@ public sealed class AdvisoryObservationQueryServiceTests
CancellationToken.None);
Assert.Equal(2, result.Observations.Length);
Assert.All(result.Observations, observation =>
Assert.Contains(observation.Linkset.Aliases, alias => alias is "cve-2025-0001" or "cve-2025-9999"));
Assert.All(result.Observations, observation =>
Assert.Contains(
observation.Linkset.Aliases,
alias => alias.Equals("CVE-2025-0001", StringComparison.OrdinalIgnoreCase)
|| alias.Equals("CVE-2025-9999", StringComparison.OrdinalIgnoreCase)));
Assert.False(result.HasMore);
Assert.Null(result.NextCursor);

View File

@@ -10,5 +10,8 @@
<ProjectReference Include="../../__Libraries/StellaOps.Concelier.Storage.Mongo/StellaOps.Concelier.Storage.Mongo.csproj" />
<ProjectReference Include="../../StellaOps.Concelier.WebService/StellaOps.Concelier.WebService.csproj" />
<ProjectReference Include="../../../__Libraries/StellaOps.Plugin/StellaOps.Plugin.csproj" />
<ProjectReference Include="../../__Analyzers/StellaOps.Concelier.Analyzers/StellaOps.Concelier.Analyzers.csproj"
OutputItemType="Analyzer"
ReferenceOutputAssembly="false" />
</ItemGroup>
</Project>
</Project>

View File

@@ -221,7 +221,7 @@ public sealed class WebServiceEndpointsTests : IAsyncLifetime
Assert.NotNull(ingestResponse.Headers.Location);
var locationValue = ingestResponse.Headers.Location!.ToString();
Assert.False(string.IsNullOrWhiteSpace(locationValue));
var lastSlashIndex = locationValue.LastIndexOf('/', StringComparison.Ordinal);
var lastSlashIndex = locationValue.LastIndexOf('/');
var idSegment = lastSlashIndex >= 0
? locationValue[(lastSlashIndex + 1)..]
: locationValue;
@@ -886,15 +886,61 @@ public sealed class WebServiceEndpointsTests : IAsyncLifetime
var limitedResponse = await client.GetAsync("/concelier/exports/index.json");
Assert.Equal((HttpStatusCode)429, limitedResponse.StatusCode);
Assert.NotNull(limitedResponse.Headers.RetryAfter);
Assert.True(limitedResponse.Headers.RetryAfter!.Delta.HasValue);
Assert.True(limitedResponse.Headers.RetryAfter!.Delta!.Value.TotalSeconds > 0);
}
[Fact]
public async Task JobsEndpointsAllowBypassWhenAuthorityEnabled()
{
var environment = new Dictionary<string, string?>
Assert.True(limitedResponse.Headers.RetryAfter!.Delta.HasValue);
Assert.True(limitedResponse.Headers.RetryAfter!.Delta!.Value.TotalSeconds > 0);
}
[Fact]
public void MergeModuleDisabledWhenFeatureFlagEnabled()
{
var environment = new Dictionary<string, string?>
{
["CONCELIER_FEATURES__NOMERGEENABLED"] = "true"
};
using var factory = new ConcelierApplicationFactory(
_runner.ConnectionString,
authorityConfigure: null,
environmentOverrides: environment);
using var scope = factory.Services.CreateScope();
var provider = scope.ServiceProvider;
#pragma warning disable CS0618, CONCELIER0001, CONCELIER0002 // Checking deprecated service registration state.
Assert.Null(provider.GetService<AdvisoryMergeService>());
#pragma warning restore CS0618, CONCELIER0001, CONCELIER0002
var schedulerOptions = provider.GetRequiredService<IOptions<JobSchedulerOptions>>().Value;
Assert.DoesNotContain("merge:reconcile", schedulerOptions.Definitions.Keys);
}
[Fact]
public void MergeJobRemainsWhenAllowlisted()
{
var environment = new Dictionary<string, string?>
{
["CONCELIER_FEATURES__MERGEJOBALLOWLIST__0"] = "merge:reconcile"
};
using var factory = new ConcelierApplicationFactory(
_runner.ConnectionString,
authorityConfigure: null,
environmentOverrides: environment);
using var scope = factory.Services.CreateScope();
var provider = scope.ServiceProvider;
#pragma warning disable CS0618, CONCELIER0001, CONCELIER0002 // Checking deprecated service registration state.
Assert.NotNull(provider.GetService<AdvisoryMergeService>());
#pragma warning restore CS0618, CONCELIER0001, CONCELIER0002
var schedulerOptions = provider.GetRequiredService<IOptions<JobSchedulerOptions>>().Value;
Assert.Contains("merge:reconcile", schedulerOptions.Definitions.Keys);
}
[Fact]
public async Task JobsEndpointsAllowBypassWhenAuthorityEnabled()
{
var environment = new Dictionary<string, string?>
{
["CONCELIER_AUTHORITY__ENABLED"] = "true",
["CONCELIER_AUTHORITY__ALLOWANONYMOUSFALLBACK"] = "false",

View File

@@ -84,10 +84,13 @@ internal sealed class WorkerSignatureVerifier : IVexSignatureVerifier
throw new ExcititorAocGuardException(AocGuardResult.FromViolations(new[] { violation }));
}
VexSignatureMetadata? signatureMetadata = null;
if (document.Format == VexDocumentFormat.OciAttestation && _attestationVerifier is not null)
VexSignatureMetadata? signatureMetadata = null;
VexAttestationDiagnostics? attestationDiagnostics = null;
if (document.Format == VexDocumentFormat.OciAttestation && _attestationVerifier is not null)
{
signatureMetadata = await VerifyAttestationAsync(document, metadata, cancellationToken).ConfigureAwait(false);
var attestationResult = await VerifyAttestationAsync(document, metadata, cancellationToken).ConfigureAwait(false);
signatureMetadata = attestationResult.Metadata;
attestationDiagnostics = attestationResult.Diagnostics;
}
signatureMetadata ??= ExtractSignatureMetadata(metadata);
@@ -96,31 +99,40 @@ internal sealed class WorkerSignatureVerifier : IVexSignatureVerifier
signatureMetadata = await AttachIssuerTrustAsync(signatureMetadata, metadata, cancellationToken).ConfigureAwait(false);
}
var resultLabel = signatureMetadata is null ? "skipped" : "ok";
RecordVerification(document.ProviderId, metadata, resultLabel);
if (resultLabel == "skipped")
{
if (attestationDiagnostics is not null)
{
resultLabel = attestationDiagnostics.Result ?? resultLabel;
}
if (attestationDiagnostics is null)
{
RecordVerification(document.ProviderId, metadata, resultLabel);
}
if (resultLabel == "skipped")
{
_logger.LogDebug(
"Signature verification skipped for provider {ProviderId} (no signature metadata).",
document.ProviderId);
}
else
{
_logger.LogInformation(
"Signature metadata recorded for provider {ProviderId} (type={SignatureType}, subject={Subject}, issuer={Issuer}).",
document.ProviderId,
signatureMetadata!.Type,
signatureMetadata.Subject ?? "<unknown>",
signatureMetadata.Issuer ?? "<unknown>");
}
return signatureMetadata;
}
private async ValueTask<VexSignatureMetadata?> VerifyAttestationAsync(
VexRawDocument document,
ImmutableDictionary<string, string> metadata,
CancellationToken cancellationToken)
_logger.LogInformation(
"Signature metadata recorded for provider {ProviderId} (type={SignatureType}, subject={Subject}, issuer={Issuer}, result={Result}).",
document.ProviderId,
signatureMetadata!.Type,
signatureMetadata.Subject ?? "<unknown>",
signatureMetadata.Issuer ?? "<unknown>",
resultLabel);
}
return signatureMetadata;
}
private async ValueTask<(VexSignatureMetadata Metadata, VexAttestationDiagnostics Diagnostics)> VerifyAttestationAsync(
VexRawDocument document,
ImmutableDictionary<string, string> metadata,
CancellationToken cancellationToken)
{
try
{
@@ -146,37 +158,48 @@ internal sealed class WorkerSignatureVerifier : IVexSignatureVerifier
attestationMetadata,
envelopeJson);
var verification = await _attestationVerifier!
.VerifyAsync(verificationRequest, cancellationToken)
.ConfigureAwait(false);
if (!verification.IsValid)
{
var diagnostics = string.Join(", ", verification.Diagnostics.Select(kvp => $"{kvp.Key}={kvp.Value}"));
_logger.LogError(
"Attestation verification failed for provider {ProviderId} (uri={SourceUri}) diagnostics={Diagnostics}",
document.ProviderId,
document.SourceUri,
diagnostics);
var violation = AocViolation.Create(
AocViolationCode.SignatureInvalid,
"/upstream/signature",
"Attestation verification failed.");
RecordVerification(document.ProviderId, metadata, "fail");
throw new ExcititorAocGuardException(AocGuardResult.FromViolations(new[] { violation }));
}
_logger.LogInformation(
"Attestation verification succeeded for provider {ProviderId} (predicate={PredicateType}, subject={Subject}).",
document.ProviderId,
attestationMetadata.PredicateType,
statement.Subject[0].Name ?? "<unknown>");
return BuildSignatureMetadata(statement, metadata, attestationMetadata, verification.Diagnostics);
}
catch (ExcititorAocGuardException)
{
var verification = await _attestationVerifier!
.VerifyAsync(verificationRequest, cancellationToken)
.ConfigureAwait(false);
var diagnosticsSnapshot = verification.Diagnostics;
if (!verification.IsValid)
{
var failureReason = diagnosticsSnapshot.FailureReason ?? "verification_failed";
var resultTag = diagnosticsSnapshot.Result ?? "invalid";
RecordVerification(document.ProviderId, metadata, resultTag);
_logger.LogError(
"Attestation verification failed for provider {ProviderId} (uri={SourceUri}) result={Result} failure={FailureReason} diagnostics={@Diagnostics}",
document.ProviderId,
document.SourceUri,
resultTag,
failureReason,
diagnosticsSnapshot);
var violation = AocViolation.Create(
AocViolationCode.SignatureInvalid,
"/upstream/signature",
"Attestation verification failed.");
throw new ExcititorAocGuardException(AocGuardResult.FromViolations(new[] { violation }));
}
var successResult = diagnosticsSnapshot.Result ?? "valid";
RecordVerification(document.ProviderId, metadata, successResult);
_logger.LogInformation(
"Attestation verification succeeded for provider {ProviderId} (predicate={PredicateType}, subject={Subject}, result={Result}).",
document.ProviderId,
attestationMetadata.PredicateType,
statement.Subject[0].Name ?? "<unknown>",
successResult);
var signatureMetadata = BuildSignatureMetadata(statement, metadata, attestationMetadata, diagnosticsSnapshot);
return (signatureMetadata, diagnosticsSnapshot);
}
catch (ExcititorAocGuardException)
{
throw;
}
catch (Exception ex)
@@ -192,10 +215,10 @@ internal sealed class WorkerSignatureVerifier : IVexSignatureVerifier
"/upstream/signature",
$"Attestation verification encountered an error: {ex.Message}");
RecordVerification(document.ProviderId, metadata, "fail");
throw new ExcititorAocGuardException(AocGuardResult.FromViolations(new[] { violation }));
}
}
RecordVerification(document.ProviderId, metadata, "error");
throw new ExcititorAocGuardException(AocGuardResult.FromViolations(new[] { violation }));
}
}
private VexAttestationRequest BuildAttestationRequest(VexInTotoStatement statement, VexAttestationPredicate predicate)
{
@@ -252,11 +275,11 @@ internal sealed class WorkerSignatureVerifier : IVexSignatureVerifier
signedAt);
}
private VexSignatureMetadata BuildSignatureMetadata(
VexInTotoStatement statement,
ImmutableDictionary<string, string> metadata,
VexAttestationMetadata attestationMetadata,
ImmutableDictionary<string, string> diagnostics)
private VexSignatureMetadata BuildSignatureMetadata(
VexInTotoStatement statement,
ImmutableDictionary<string, string> metadata,
VexAttestationMetadata attestationMetadata,
VexAttestationDiagnostics diagnostics)
{
metadata.TryGetValue("vex.signature.type", out var type);
metadata.TryGetValue("vex.provenance.cosign.subject", out var subject);

View File

@@ -13,16 +13,18 @@
3. Emit observability signals (logs, metrics, optional tracing) that can run offline and degrade gracefully when transparency services are unreachable.
4. Add regression tests (unit + integration) covering positive path, negative path, and offline fallback scenarios.
## 2. Deliverables
- `IVexAttestationVerifier` abstraction + `VexAttestationVerifier` implementation inside `StellaOps.Excititor.Attestation`, encapsulating DSSE validation, predicate checks, artifact digest confirmation, Rekor inclusion verification, and deterministic diagnostics.
- DI wiring (extension method) for registering verifier + instrumentation dependencies alongside the existing signer/rekor client.
- Shared `VexAttestationDiagnostics` record describing normalized diagnostic keys consumed by Worker/WebService logging.
- Metrics utility (`AttestationMetrics`) exposing counters/histograms via `System.Diagnostics.Metrics`, exported under `StellaOps.Excititor.Attestation` meter.
- Activity source (`AttestationActivitySource`) for optional tracing spans around sign/verify operations.
- Documentation updates (`EXCITITOR-ATTEST-01-003-plan.md`, `TASKS.md` notes) describing instrumentation + test expectations.
- Test coverage in `StellaOps.Excititor.Attestation.Tests` (unit) and scaffolding notes for WebService/Worker integration tests.
## 2. Deliverables
- `IVexAttestationVerifier` abstraction + `VexAttestationVerifier` implementation inside `StellaOps.Excititor.Attestation`, encapsulating DSSE validation, predicate checks, artifact digest confirmation, Rekor inclusion verification, and deterministic diagnostics.
- DI wiring (extension method) for registering verifier + instrumentation dependencies alongside the existing signer/rekor client.
- Shared `VexAttestationDiagnostics` record describing normalized diagnostic keys consumed by Worker/WebService logging.
- Metrics utility (`AttestationMetrics`) exposing counters/histograms via `System.Diagnostics.Metrics`, exported under `StellaOps.Excititor.Attestation` meter.
- Activity source (`AttestationActivitySource`) for optional tracing spans around sign/verify operations.
- 2025-11-05: Implemented `VexAttestationDiagnostics`, activity tagging via `VexAttestationActivitySource`, and updated verifier/tests to emit structured failure reasons.
- 2025-11-05 (pm): Worker attestation verifier now records structured diagnostics/metrics and logs result/failure reasons using `VexAttestationDiagnostics`; attestation success/failure labels propagate to verification counters.
- Documentation updates (`EXCITITOR-ATTEST-01-003-plan.md`, `TASKS.md` notes) describing instrumentation + test expectations.
- Test coverage in `StellaOps.Excititor.Attestation.Tests` (unit) and scaffolding notes for WebService/Worker integration tests.
## 3. Verification Flow
### 3.1 Inputs

View File

@@ -1,6 +1,7 @@
If you are working on this file you need to read docs/modules/excititor/ARCHITECTURE.md and ./AGENTS.md).
# TASKS
| Task | Owner(s) | Depends on | Notes |
|---|---|---|---|
|EXCITITOR-ATTEST-01-003 Verification suite & observability|Team Excititor Attestation|EXCITITOR-ATTEST-01-002|DOING (2025-10-22) Continuing implementation: build `IVexAttestationVerifier`, wire metrics/logging, and add regression tests. Draft plan in `EXCITITOR-ATTEST-01-003-plan.md` (2025-10-19) guides scope; updating with worknotes as progress lands.<br>2025-10-31: Verifier now tolerates duplicate source providers from AOC raw projections, downgrades offline Rekor verification to a degraded result, and enforces trusted signer registry checks with detailed diagnostics/tests.|
If you are working on this file you need to read docs/modules/excititor/ARCHITECTURE.md and ./AGENTS.md).
# TASKS
| Task | Owner(s) | Depends on | Notes |
|---|---|---|---|
|EXCITITOR-ATTEST-01-003 Verification suite & observability|Team Excititor Attestation|EXCITITOR-ATTEST-01-002|DOING (2025-10-22) Continuing implementation: build `IVexAttestationVerifier`, wire metrics/logging, and add regression tests. Draft plan in `EXCITITOR-ATTEST-01-003-plan.md` (2025-10-19) guides scope; updating with worknotes as progress lands.<br>2025-10-31: Verifier now tolerates duplicate source providers from AOC raw projections, downgrades offline Rekor verification to a degraded result, and enforces trusted signer registry checks with detailed diagnostics/tests.<br>2025-11-05 14:35Z: Picking up diagnostics record/ActivitySource work and aligning metrics dimensions before wiring verifier into WebService/Worker paths.|
> 2025-11-05 19:10Z: Worker signature verifier now emits structured diagnostics/metrics via `VexAttestationDiagnostics`; attestation verification results flow into metric labels and logs.
> Remark (2025-10-22): Added verifier implementation + metrics/tests; next steps include wiring into WebService/Worker flows and expanding negative-path coverage.

View File

@@ -0,0 +1,10 @@
using System.Diagnostics;
namespace StellaOps.Excititor.Attestation.Verification;
public static class VexAttestationActivitySource
{
public const string Name = "StellaOps.Excititor.Attestation";
public static readonly ActivitySource Value = new(Name);
}

View File

@@ -1,8 +1,8 @@
using System;
using System.Collections.Generic;
using System.Collections.Immutable;
using System.Diagnostics;
using System.Linq;
using System.Collections.Generic;
using System.Collections.Immutable;
using System.Diagnostics;
using System.Linq;
using System.Text;
using System.Text.Json;
using System.Text.Json.Serialization;
@@ -59,104 +59,120 @@ internal sealed class VexAttestationVerifier : IVexAttestationVerifier
public async ValueTask<VexAttestationVerification> VerifyAsync(
VexAttestationVerificationRequest request,
CancellationToken cancellationToken)
{
ArgumentNullException.ThrowIfNull(request);
var stopwatch = Stopwatch.StartNew();
var diagnostics = ImmutableDictionary.CreateBuilder<string, string>(StringComparer.Ordinal);
{
ArgumentNullException.ThrowIfNull(request);
var stopwatch = Stopwatch.StartNew();
var diagnostics = ImmutableDictionary.CreateBuilder<string, string>(StringComparer.Ordinal);
var resultLabel = "valid";
var rekorState = "skipped";
var component = request.IsReverify ? "worker" : "webservice";
void SetFailure(string reason) => diagnostics["failure_reason"] = reason;
using var activity = VexAttestationActivitySource.Value.StartActivity("Verify", ActivityKind.Internal);
activity?.SetTag("attestation.component", component);
activity?.SetTag("attestation.export_id", request.Attestation.ExportId);
try
{
if (string.IsNullOrWhiteSpace(request.Envelope))
{
diagnostics["envelope.state"] = "missing";
_logger.LogWarning("Attestation envelope is missing for export {ExportId}", request.Attestation.ExportId);
resultLabel = "invalid";
return BuildResult(false);
}
if (!TryDeserializeEnvelope(request.Envelope, out var envelope, diagnostics))
{
_logger.LogWarning("Failed to deserialize attestation envelope for export {ExportId}", request.Attestation.ExportId);
resultLabel = "invalid";
return BuildResult(false);
}
if (!string.Equals(envelope.PayloadType, VexDsseBuilder.PayloadType, StringComparison.OrdinalIgnoreCase))
{
diagnostics["payload.type"] = envelope.PayloadType ?? string.Empty;
_logger.LogWarning(
"Unexpected DSSE payload type {PayloadType} for export {ExportId}",
envelope.PayloadType,
request.Attestation.ExportId);
resultLabel = "invalid";
return BuildResult(false);
}
if (envelope.Signatures is null || envelope.Signatures.Count == 0)
{
diagnostics["signature.state"] = "missing";
_logger.LogWarning("Attestation envelope for export {ExportId} does not contain signatures.", request.Attestation.ExportId);
resultLabel = "invalid";
return BuildResult(false);
}
var payloadBase64 = envelope.Payload ?? string.Empty;
if (!TryDecodePayload(payloadBase64, out var payloadBytes, diagnostics))
{
_logger.LogWarning("Failed to decode attestation payload for export {ExportId}", request.Attestation.ExportId);
resultLabel = "invalid";
return BuildResult(false);
}
if (!TryDeserializeStatement(payloadBytes, out var statement, diagnostics))
{
_logger.LogWarning("Failed to deserialize DSSE statement for export {ExportId}", request.Attestation.ExportId);
resultLabel = "invalid";
return BuildResult(false);
}
if (!ValidatePredicateType(statement, request, diagnostics))
{
_logger.LogWarning("Predicate type mismatch for export {ExportId}", request.Attestation.ExportId);
resultLabel = "invalid";
return BuildResult(false);
}
if (!ValidateSubject(statement, request, diagnostics))
{
_logger.LogWarning("Subject mismatch for export {ExportId}", request.Attestation.ExportId);
resultLabel = "invalid";
return BuildResult(false);
}
if (!ValidatePredicate(statement, request, diagnostics))
{
_logger.LogWarning("Predicate payload mismatch for export {ExportId}", request.Attestation.ExportId);
resultLabel = "invalid";
return BuildResult(false);
}
if (!ValidateMetadataDigest(envelope, request.Metadata, diagnostics))
{
_logger.LogWarning("Attestation digest mismatch for export {ExportId}", request.Attestation.ExportId);
resultLabel = "invalid";
return BuildResult(false);
}
if (!ValidateSignedAt(request.Metadata, request.Attestation.CreatedAt, diagnostics))
{
_logger.LogWarning("SignedAt validation failed for export {ExportId}", request.Attestation.ExportId);
resultLabel = "invalid";
return BuildResult(false);
}
if (string.IsNullOrWhiteSpace(request.Envelope))
{
diagnostics["envelope.state"] = "missing";
SetFailure("missing_envelope");
_logger.LogWarning("Attestation envelope is missing for export {ExportId}", request.Attestation.ExportId);
resultLabel = "invalid";
return BuildResult(false);
}
if (!TryDeserializeEnvelope(request.Envelope, out var envelope, diagnostics))
{
SetFailure("invalid_envelope");
_logger.LogWarning("Failed to deserialize attestation envelope for export {ExportId}", request.Attestation.ExportId);
resultLabel = "invalid";
return BuildResult(false);
}
if (!string.Equals(envelope.PayloadType, VexDsseBuilder.PayloadType, StringComparison.OrdinalIgnoreCase))
{
diagnostics["payload.type"] = envelope.PayloadType ?? string.Empty;
SetFailure("unexpected_payload_type");
_logger.LogWarning(
"Unexpected DSSE payload type {PayloadType} for export {ExportId}",
envelope.PayloadType,
request.Attestation.ExportId);
resultLabel = "invalid";
return BuildResult(false);
}
if (envelope.Signatures is null || envelope.Signatures.Count == 0)
{
diagnostics["signature.state"] = "missing";
SetFailure("missing_signature");
_logger.LogWarning("Attestation envelope for export {ExportId} does not contain signatures.", request.Attestation.ExportId);
resultLabel = "invalid";
return BuildResult(false);
}
var payloadBase64 = envelope.Payload ?? string.Empty;
if (!TryDecodePayload(payloadBase64, out var payloadBytes, diagnostics))
{
SetFailure("payload_decode_failed");
_logger.LogWarning("Failed to decode attestation payload for export {ExportId}", request.Attestation.ExportId);
resultLabel = "invalid";
return BuildResult(false);
}
if (!TryDeserializeStatement(payloadBytes, out var statement, diagnostics))
{
SetFailure("invalid_statement");
_logger.LogWarning("Failed to deserialize DSSE statement for export {ExportId}", request.Attestation.ExportId);
resultLabel = "invalid";
return BuildResult(false);
}
if (!ValidatePredicateType(statement, request, diagnostics))
{
SetFailure("predicate_type_mismatch");
_logger.LogWarning("Predicate type mismatch for export {ExportId}", request.Attestation.ExportId);
resultLabel = "invalid";
return BuildResult(false);
}
if (!ValidateSubject(statement, request, diagnostics))
{
SetFailure("subject_mismatch");
_logger.LogWarning("Subject mismatch for export {ExportId}", request.Attestation.ExportId);
resultLabel = "invalid";
return BuildResult(false);
}
if (!ValidatePredicate(statement, request, diagnostics))
{
SetFailure("predicate_mismatch");
_logger.LogWarning("Predicate payload mismatch for export {ExportId}", request.Attestation.ExportId);
resultLabel = "invalid";
return BuildResult(false);
}
if (!ValidateMetadataDigest(envelope, request.Metadata, diagnostics))
{
SetFailure("envelope_digest_mismatch");
_logger.LogWarning("Attestation digest mismatch for export {ExportId}", request.Attestation.ExportId);
resultLabel = "invalid";
return BuildResult(false);
}
if (!ValidateSignedAt(request.Metadata, request.Attestation.CreatedAt, diagnostics))
{
SetFailure("signedat_out_of_range");
_logger.LogWarning("SignedAt validation failed for export {ExportId}", request.Attestation.ExportId);
resultLabel = "invalid";
return BuildResult(false);
}
rekorState = await VerifyTransparencyAsync(request.Metadata, diagnostics, cancellationToken).ConfigureAwait(false);
if (rekorState is "missing" or "unverified" or "client_unavailable")
{
SetFailure(rekorState);
resultLabel = "invalid";
return BuildResult(false);
}
@@ -164,6 +180,9 @@ internal sealed class VexAttestationVerifier : IVexAttestationVerifier
var signaturesVerified = await VerifySignaturesAsync(payloadBytes, envelope.Signatures, diagnostics, cancellationToken).ConfigureAwait(false);
if (!signaturesVerified)
{
diagnostics["failure_reason"] = diagnostics.TryGetValue("signature.reason", out var reason)
? reason
: "signature_verification_failed";
if (_options.RequireSignatureVerification)
{
resultLabel = "invalid";
@@ -183,13 +202,16 @@ internal sealed class VexAttestationVerifier : IVexAttestationVerifier
catch (Exception ex)
{
diagnostics["error"] = ex.GetType().Name;
diagnostics["error.message"] = ex.Message; resultLabel = "error";
_logger.LogError(ex, "Unexpected exception verifying attestation for export {ExportId}", request.Attestation.ExportId);
return BuildResult(false);
}
finally
{
stopwatch.Stop();
diagnostics["error.message"] = ex.Message; resultLabel = "error";
_logger.LogError(ex, "Unexpected exception verifying attestation for export {ExportId}", request.Attestation.ExportId);
diagnostics["failure_reason"] = diagnostics.TryGetValue("error", out var errorCode)
? errorCode
: ex.GetType().Name;
return BuildResult(false);
}
finally
{
stopwatch.Stop();
var tags = new KeyValuePair<string, object?>[]
{
new("result", resultLabel),
@@ -200,12 +222,32 @@ internal sealed class VexAttestationVerifier : IVexAttestationVerifier
_metrics.VerifyDuration.Record(stopwatch.Elapsed.TotalSeconds, tags);
}
VexAttestationVerification BuildResult(bool isValid)
{
diagnostics["result"] = resultLabel;
diagnostics["component"] = component;
VexAttestationVerification BuildResult(bool isValid)
{
diagnostics["result"] = resultLabel;
diagnostics["component"] = component;
diagnostics["rekor.state"] = rekorState;
return new VexAttestationVerification(isValid, diagnostics.ToImmutable());
var snapshot = VexAttestationDiagnostics.FromBuilder(diagnostics);
if (activity is { } currentActivity)
{
currentActivity.SetTag("attestation.result", resultLabel);
currentActivity.SetTag("attestation.rekor", rekorState);
if (!isValid)
{
var failure = snapshot.FailureReason ?? "verification_failed";
currentActivity.SetStatus(ActivityStatusCode.Error, failure);
currentActivity.SetTag("attestation.failure_reason", failure);
}
else
{
currentActivity.SetStatus(resultLabel is "degraded"
? ActivityStatusCode.Ok
: ActivityStatusCode.Ok);
}
}
return new VexAttestationVerification(isValid, snapshot);
}
}

View File

@@ -14,4 +14,4 @@
<ProjectReference Include="../../../Aoc/__Libraries/StellaOps.Aoc/StellaOps.Aoc.csproj" />
<ProjectReference Include="../../../Concelier/__Libraries/StellaOps.Concelier.RawModels/StellaOps.Concelier.RawModels.csproj" />
</ItemGroup>
</Project>
</Project>

View File

@@ -1,7 +1,8 @@
using System;
using System.Collections.Immutable;
using System.Threading;
using System.Threading.Tasks;
using System.Threading.Tasks;
using StellaOps.Excititor.Attestation.Verification;
namespace StellaOps.Excititor.Core;
@@ -33,4 +34,4 @@ public sealed record VexAttestationVerificationRequest(
public sealed record VexAttestationVerification(
bool IsValid,
ImmutableDictionary<string, string> Diagnostics);
VexAttestationDiagnostics Diagnostics);

View File

@@ -0,0 +1,57 @@
using System.Collections;
using System.Collections.Generic;
using System.Collections.Immutable;
namespace StellaOps.Excititor.Attestation.Verification;
public sealed class VexAttestationDiagnostics : IReadOnlyDictionary<string, string>
{
private readonly ImmutableDictionary<string, string> _values;
private VexAttestationDiagnostics(ImmutableDictionary<string, string> values)
{
_values = values ?? ImmutableDictionary<string, string>.Empty;
}
public static VexAttestationDiagnostics FromBuilder(ImmutableDictionary<string, string>.Builder builder)
{
ArgumentNullException.ThrowIfNull(builder);
return new(builder.ToImmutable());
}
public static VexAttestationDiagnostics Empty { get; } = new(ImmutableDictionary<string, string>.Empty);
public string? Result => TryGetValue("result", out var value) ? value : null;
public string? Component => TryGetValue("component", out var value) ? value : null;
public string? RekorState => TryGetValue("rekor.state", out var value) ? value : null;
public string? FailureReason => TryGetValue("failure_reason", out var value) ? value : null;
public string this[string key] => _values[key];
public IEnumerable<string> Keys => _values.Keys;
public IEnumerable<string> Values => _values.Values;
public int Count => _values.Count;
public bool ContainsKey(string key) => _values.ContainsKey(key);
public bool TryGetValue(string key, out string value)
{
if (_values.TryGetValue(key, out var stored))
{
value = stored;
return true;
}
value = string.Empty;
return false;
}
public IEnumerator<KeyValuePair<string, string>> GetEnumerator() => _values.GetEnumerator();
IEnumerator IEnumerable.GetEnumerator() => GetEnumerator();
}

View File

@@ -85,6 +85,6 @@ public sealed class VexAttestationClientTests
private sealed class FakeVerifier : IVexAttestationVerifier
{
public ValueTask<VexAttestationVerification> VerifyAsync(VexAttestationVerificationRequest request, CancellationToken cancellationToken)
=> ValueTask.FromResult(new VexAttestationVerification(true, ImmutableDictionary<string, string>.Empty));
=> ValueTask.FromResult(new VexAttestationVerification(true, VexAttestationDiagnostics.Empty));
}
}

View File

@@ -16,42 +16,44 @@ public sealed class VexAttestationVerifierTests : IDisposable
{
private readonly VexAttestationMetrics _metrics = new();
[Fact]
public async Task VerifyAsync_ReturnsValid_WhenEnvelopeMatches()
{
var (request, metadata, envelope) = await CreateSignedAttestationAsync();
var verifier = CreateVerifier(options => options.RequireTransparencyLog = false);
[Fact]
public async Task VerifyAsync_ReturnsValid_WhenEnvelopeMatches()
{
var (request, metadata, envelope) = await CreateSignedAttestationAsync();
var verifier = CreateVerifier(options => options.RequireTransparencyLog = false);
var verification = await verifier.VerifyAsync(
new VexAttestationVerificationRequest(request, metadata, envelope),
CancellationToken.None);
Assert.True(verification.IsValid);
Assert.Equal("valid", verification.Diagnostics.Result);
Assert.Null(verification.Diagnostics.FailureReason);
}
var verification = await verifier.VerifyAsync(
new VexAttestationVerificationRequest(request, metadata, envelope),
CancellationToken.None);
[Fact]
public async Task VerifyAsync_ReturnsInvalid_WhenDigestMismatch()
{
var (request, metadata, envelope) = await CreateSignedAttestationAsync();
var verifier = CreateVerifier(options => options.RequireTransparencyLog = false);
var tamperedMetadata = new VexAttestationMetadata(
metadata.PredicateType,
metadata.Rekor,
"sha256:deadbeef",
metadata.SignedAt);
var verification = await verifier.VerifyAsync(
new VexAttestationVerificationRequest(request, tamperedMetadata, envelope),
CancellationToken.None);
Assert.False(verification.IsValid);
Assert.Equal("invalid", verification.Diagnostics.Result);
Assert.Equal("sha256:deadbeef", verification.Diagnostics["metadata.envelopeDigest"]);
Assert.Equal("envelope_digest_mismatch", verification.Diagnostics.FailureReason);
}
Assert.True(verification.IsValid);
Assert.Equal("valid", verification.Diagnostics["result"]);
}
[Fact]
public async Task VerifyAsync_ReturnsInvalid_WhenDigestMismatch()
{
var (request, metadata, envelope) = await CreateSignedAttestationAsync();
var verifier = CreateVerifier(options => options.RequireTransparencyLog = false);
var tamperedMetadata = new VexAttestationMetadata(
metadata.PredicateType,
metadata.Rekor,
"sha256:deadbeef",
metadata.SignedAt);
var verification = await verifier.VerifyAsync(
new VexAttestationVerificationRequest(request, tamperedMetadata, envelope),
CancellationToken.None);
Assert.False(verification.IsValid);
Assert.Equal("invalid", verification.Diagnostics["result"]);
Assert.Equal("sha256:deadbeef", verification.Diagnostics["metadata.envelopeDigest"]);
}
[Fact]
[Fact]
public async Task VerifyAsync_AllowsOfflineTransparency_WhenConfigured()
{
var (request, metadata, envelope) = await CreateSignedAttestationAsync(includeRekor: true);
@@ -67,47 +69,50 @@ public sealed class VexAttestationVerifierTests : IDisposable
CancellationToken.None);
Assert.True(verification.IsValid);
Assert.Equal("offline", verification.Diagnostics["rekor.state"]);
Assert.Equal("degraded", verification.Diagnostics["result"]);
Assert.Equal("offline", verification.Diagnostics.RekorState);
Assert.Equal("degraded", verification.Diagnostics.Result);
Assert.Null(verification.Diagnostics.FailureReason);
}
[Fact]
public async Task VerifyAsync_ReturnsInvalid_WhenTransparencyRequiredAndMissing()
{
var (request, metadata, envelope) = await CreateSignedAttestationAsync(includeRekor: false);
var verifier = CreateVerifier(options =>
{
options.RequireTransparencyLog = true;
options.AllowOfflineTransparency = false;
});
var verification = await verifier.VerifyAsync(
new VexAttestationVerificationRequest(request, metadata, envelope),
CancellationToken.None);
Assert.False(verification.IsValid);
Assert.Equal("missing", verification.Diagnostics["rekor.state"]);
Assert.Equal("invalid", verification.Diagnostics["result"]);
}
[Fact]
public async Task VerifyAsync_ReturnsInvalid_WhenTransparencyUnavailableAndOfflineDisallowed()
{
var (request, metadata, envelope) = await CreateSignedAttestationAsync(includeRekor: true);
var transparency = new ThrowingTransparencyLogClient();
var verifier = CreateVerifier(options =>
{
options.RequireTransparencyLog = true;
options.AllowOfflineTransparency = false;
}, transparency);
var verification = await verifier.VerifyAsync(
new VexAttestationVerificationRequest(request, metadata, envelope),
CancellationToken.None);
Assert.False(verification.IsValid);
Assert.Equal("unreachable", verification.Diagnostics["rekor.state"]);
Assert.Equal("invalid", verification.Diagnostics["result"]);
{
var (request, metadata, envelope) = await CreateSignedAttestationAsync(includeRekor: false);
var verifier = CreateVerifier(options =>
{
options.RequireTransparencyLog = true;
options.AllowOfflineTransparency = false;
});
var verification = await verifier.VerifyAsync(
new VexAttestationVerificationRequest(request, metadata, envelope),
CancellationToken.None);
Assert.False(verification.IsValid);
Assert.Equal("missing", verification.Diagnostics.RekorState);
Assert.Equal("invalid", verification.Diagnostics.Result);
Assert.Equal("missing", verification.Diagnostics.FailureReason);
}
[Fact]
public async Task VerifyAsync_ReturnsInvalid_WhenTransparencyUnavailableAndOfflineDisallowed()
{
var (request, metadata, envelope) = await CreateSignedAttestationAsync(includeRekor: true);
var transparency = new ThrowingTransparencyLogClient();
var verifier = CreateVerifier(options =>
{
options.RequireTransparencyLog = true;
options.AllowOfflineTransparency = false;
}, transparency);
var verification = await verifier.VerifyAsync(
new VexAttestationVerificationRequest(request, metadata, envelope),
CancellationToken.None);
Assert.False(verification.IsValid);
Assert.Equal("unreachable", verification.Diagnostics.RekorState);
Assert.Equal("invalid", verification.Diagnostics.Result);
Assert.Equal("unreachable", verification.Diagnostics.FailureReason);
}
[Fact]
@@ -125,7 +130,7 @@ public sealed class VexAttestationVerifierTests : IDisposable
CancellationToken.None);
Assert.True(verification.IsValid);
Assert.Equal("valid", verification.Diagnostics["result"]);
Assert.Equal("valid", verification.Diagnostics.Result);
}
[Fact]
@@ -152,6 +157,8 @@ public sealed class VexAttestationVerifierTests : IDisposable
Assert.True(verification.IsValid);
Assert.Equal("verified", verification.Diagnostics["signature.state"]);
Assert.Equal("valid", verification.Diagnostics.Result);
Assert.Null(verification.Diagnostics.FailureReason);
}
[Fact]
@@ -179,6 +186,8 @@ public sealed class VexAttestationVerifierTests : IDisposable
Assert.False(verification.IsValid);
Assert.Equal("error", verification.Diagnostics["signature.state"]);
Assert.Equal("verification_failed", verification.Diagnostics["signature.reason"]);
Assert.Equal("verification_failed", verification.Diagnostics.FailureReason);
Assert.Equal("invalid", verification.Diagnostics.Result);
}
private async Task<(VexAttestationRequest Request, VexAttestationMetadata Metadata, string Envelope)> CreateSignedAttestationAsync(

View File

@@ -6,6 +6,7 @@ using System.Globalization;
using Microsoft.Extensions.Logging.Abstractions;
using MongoDB.Driver;
using StellaOps.Excititor.Core;
using StellaOps.Excititor.Attestation.Verification;
using StellaOps.Excititor.Export;
using StellaOps.Excititor.Policy;
using StellaOps.Excititor.Storage.Mongo;
@@ -291,7 +292,7 @@ public sealed class ExportEngineTests
}
public ValueTask<VexAttestationVerification> VerifyAsync(VexAttestationVerificationRequest request, CancellationToken cancellationToken)
=> ValueTask.FromResult(new VexAttestationVerification(true, ImmutableDictionary<string, string>.Empty));
=> ValueTask.FromResult(new VexAttestationVerification(true, VexAttestationDiagnostics.Empty));
}
private sealed class RecordingCacheIndex : IVexCacheIndex

View File

@@ -4,13 +4,14 @@ using System.Collections.Immutable;
using System.Text;
using System.Text.Json;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.DependencyInjection.Extensions;
using Microsoft.Extensions.Hosting;
using Microsoft.Extensions.Logging;
using StellaOps.Excititor.Core;
using StellaOps.Excititor.Export;
using StellaOps.Excititor.Storage.Mongo;
using StellaOps.Excititor.WebService.Services;
using Microsoft.Extensions.DependencyInjection.Extensions;
using Microsoft.Extensions.Hosting;
using Microsoft.Extensions.Logging;
using StellaOps.Excititor.Core;
using StellaOps.Excititor.Attestation.Verification;
using StellaOps.Excititor.Export;
using StellaOps.Excititor.Storage.Mongo;
using StellaOps.Excititor.WebService.Services;
using MongoDB.Driver;
using StellaOps.Excititor.Attestation.Dsse;
@@ -162,7 +163,7 @@ internal static class TestServiceOverrides
public ValueTask<VexAttestationVerification> VerifyAsync(VexAttestationVerificationRequest request, CancellationToken cancellationToken)
{
var verification = new VexAttestationVerification(true, ImmutableDictionary<string, string>.Empty);
var verification = new VexAttestationVerification(true, VexAttestationDiagnostics.Empty);
return ValueTask.FromResult(verification);
}
}

View File

@@ -504,6 +504,21 @@ public sealed class DefaultVexProviderRunnerTests
bool includeGlobal,
CancellationToken cancellationToken)
=> ValueTask.FromResult(DefaultTrust);
public ValueTask<IssuerTrustResponseModel> SetIssuerTrustAsync(
string tenantId,
string issuerId,
decimal weight,
string? reason,
CancellationToken cancellationToken)
=> ValueTask.FromResult(DefaultTrust);
public ValueTask DeleteIssuerTrustAsync(
string tenantId,
string issuerId,
string? reason,
CancellationToken cancellationToken)
=> ValueTask.CompletedTask;
}
private sealed class NoopSignatureVerifier : IVexSignatureVerifier
@@ -655,25 +670,25 @@ public sealed class DefaultVexProviderRunnerTests
}
}
private sealed class StubAttestationVerifier : IVexAttestationVerifier
{
private readonly bool _isValid;
private readonly ImmutableDictionary<string, string> _diagnostics;
public StubAttestationVerifier(bool isValid, ImmutableDictionary<string, string> diagnostics)
{
_isValid = isValid;
_diagnostics = diagnostics;
}
public int Invocations { get; private set; }
public ValueTask<VexAttestationVerification> VerifyAsync(VexAttestationVerificationRequest request, CancellationToken cancellationToken)
{
Invocations++;
return ValueTask.FromResult(new VexAttestationVerification(_isValid, _diagnostics));
}
}
private sealed class StubAttestationVerifier : IVexAttestationVerifier
{
private readonly bool _isValid;
private readonly VexAttestationDiagnostics _diagnostics;
public StubAttestationVerifier(bool isValid, ImmutableDictionary<string, string> diagnostics)
{
_isValid = isValid;
_diagnostics = VexAttestationDiagnostics.FromBuilder(diagnostics.ToBuilder());
}
public int Invocations { get; private set; }
public ValueTask<VexAttestationVerification> VerifyAsync(VexAttestationVerificationRequest request, CancellationToken cancellationToken)
{
Invocations++;
return ValueTask.FromResult(new VexAttestationVerification(_isValid, _diagnostics));
}
}
private static VexRawDocument CreateAttestationRawDocument(DateTimeOffset observedAt)
{

View File

@@ -249,19 +249,21 @@ public sealed class WorkerSignatureVerifierTests
private sealed class StubAttestationVerifier : IVexAttestationVerifier
{
private readonly bool _isValid;
private readonly ImmutableDictionary<string, string> _diagnostics;
private readonly VexAttestationDiagnostics _diagnostics;
public StubAttestationVerifier(bool isValid, ImmutableDictionary<string, string>? diagnostics = null)
{
_isValid = isValid;
_diagnostics = diagnostics ?? ImmutableDictionary<string, string>.Empty;
}
public int Invocations { get; private set; }
public ValueTask<VexAttestationVerification> VerifyAsync(VexAttestationVerificationRequest request, CancellationToken cancellationToken)
{
Invocations++;
{
_isValid = isValid;
_diagnostics = diagnostics is null
? VexAttestationDiagnostics.Empty
: VexAttestationDiagnostics.FromBuilder(diagnostics.ToBuilder());
}
public int Invocations { get; private set; }
public ValueTask<VexAttestationVerification> VerifyAsync(VexAttestationVerificationRequest request, CancellationToken cancellationToken)
{
Invocations++;
return ValueTask.FromResult(new VexAttestationVerification(_isValid, _diagnostics));
}
}
@@ -269,7 +271,7 @@ public sealed class WorkerSignatureVerifierTests
private sealed class StubIssuerDirectoryClient : IIssuerDirectoryClient
{
private readonly IReadOnlyList<IssuerKeyModel> _keys;
private readonly IssuerTrustResponseModel _trust;
private IssuerTrustResponseModel _trust;
private StubIssuerDirectoryClient(
IReadOnlyList<IssuerKeyModel> keys,
@@ -302,7 +304,7 @@ public sealed class WorkerSignatureVerifierTests
null,
null);
var now = DateTimeOffset.UtcNow;
var now = DateTimeOffset.UnixEpoch;
var overrideModel = new IssuerTrustOverrideModel(weight, "stub", now, "test", now, "test");
return new StubIssuerDirectoryClient(
new[] { key },
@@ -322,6 +324,29 @@ public sealed class WorkerSignatureVerifierTests
bool includeGlobal,
CancellationToken cancellationToken)
=> ValueTask.FromResult(_trust);
public ValueTask<IssuerTrustResponseModel> SetIssuerTrustAsync(
string tenantId,
string issuerId,
decimal weight,
string? reason,
CancellationToken cancellationToken)
{
var now = DateTimeOffset.UnixEpoch;
var overrideModel = new IssuerTrustOverrideModel(weight, "stub-set", now, "test", now, "test");
_trust = new IssuerTrustResponseModel(overrideModel, null, weight);
return ValueTask.FromResult(_trust);
}
public ValueTask DeleteIssuerTrustAsync(
string tenantId,
string issuerId,
string? reason,
CancellationToken cancellationToken)
{
_trust = new IssuerTrustResponseModel(null, null, 0m);
return ValueTask.CompletedTask;
}
}
private sealed class FixedTimeProvider : TimeProvider

View File

@@ -3,5 +3,5 @@
## Sprint 64 Bundle Implementation
| ID | Status | Owner(s) | Depends on | Description | Exit Criteria |
|----|--------|----------|------------|-------------|---------------|
| DVOFF-64-001 | DOING (2025-11-04) | DevPortal Offline Guild, Exporter Guild | DEVPORT-64-001, SDKREL-64-002 | Implement Export Center job `devportal --offline` bundling portal HTML, specs, SDK artifacts, changelogs, and verification manifest. | Job executes in staging; manifest contains checksums + DSSE signatures; docs updated. |
| DVOFF-64-001 | DONE (2025-11-05) | DevPortal Offline Guild, Exporter Guild | DEVPORT-64-001, SDKREL-64-002 | Implement Export Center job `devportal --offline` bundling portal HTML, specs, SDK artifacts, changelogs, and verification manifest. | Job executes in staging; manifest contains checksums + DSSE signatures; docs updated. |
| DVOFF-64-002 | TODO | DevPortal Offline Guild, AirGap Controller Guild | DVOFF-64-001 | Provide verification CLI (`stella devportal verify bundle.tgz`) ensuring integrity before import. | CLI command validates signatures; integration test covers corrupted bundle; runbook updated. |

View File

@@ -0,0 +1,176 @@
using System;
using System.IO;
using System.Text;
using System.Text.Json;
using System.Threading;
using System.Threading.Tasks;
using Microsoft.Extensions.Logging;
namespace StellaOps.ExportCenter.Core.DevPortalOffline;
/// <summary>
/// Coordinates bundle construction, manifest signing, and artefact persistence for the devportal offline export.
/// </summary>
public sealed class DevPortalOfflineJob
{
private static readonly JsonSerializerOptions SignatureSerializerOptions = new(JsonSerializerDefaults.Web)
{
WriteIndented = true
};
private readonly DevPortalOfflineBundleBuilder _builder;
private readonly IDevPortalOfflineObjectStore _objectStore;
private readonly IDevPortalOfflineManifestSigner _manifestSigner;
private readonly ILogger<DevPortalOfflineJob> _logger;
public DevPortalOfflineJob(
DevPortalOfflineBundleBuilder builder,
IDevPortalOfflineObjectStore objectStore,
IDevPortalOfflineManifestSigner manifestSigner,
ILogger<DevPortalOfflineJob> logger)
{
_builder = builder ?? throw new ArgumentNullException(nameof(builder));
_objectStore = objectStore ?? throw new ArgumentNullException(nameof(objectStore));
_manifestSigner = manifestSigner ?? throw new ArgumentNullException(nameof(manifestSigner));
_logger = logger ?? throw new ArgumentNullException(nameof(logger));
}
public async Task<DevPortalOfflineJobOutcome> ExecuteAsync(
DevPortalOfflineJobRequest request,
CancellationToken cancellationToken = default)
{
ArgumentNullException.ThrowIfNull(request);
cancellationToken.ThrowIfCancellationRequested();
var bundleResult = _builder.Build(request.BundleRequest, cancellationToken);
_logger.LogInformation("DevPortal offline bundle constructed with {EntryCount} entries.", bundleResult.Manifest.Entries.Count);
var signature = await _manifestSigner.SignAsync(
request.BundleRequest.BundleId,
bundleResult.ManifestJson,
bundleResult.RootHash,
cancellationToken).ConfigureAwait(false);
var storagePrefix = BuildStoragePrefix(request.StoragePrefix, request.BundleRequest.BundleId);
var bundleFileName = SanitizeFileName(request.BundleFileName);
var manifestKey = $"{storagePrefix}/manifest.json";
var signatureKey = $"{storagePrefix}/manifest.dsse.json";
var checksumsKey = $"{storagePrefix}/checksums.txt";
var bundleKey = $"{storagePrefix}/{bundleFileName}";
var manifestMetadata = await StoreTextAsync(
bundleResult.ManifestJson,
manifestKey,
MediaTypes.Json,
cancellationToken).ConfigureAwait(false);
var signatureJson = JsonSerializer.Serialize(signature, SignatureSerializerOptions);
var signatureMetadata = await StoreTextAsync(
signatureJson,
signatureKey,
MediaTypes.Json,
cancellationToken).ConfigureAwait(false);
var checksumsMetadata = await StoreTextAsync(
bundleResult.Checksums,
checksumsKey,
MediaTypes.Text,
cancellationToken).ConfigureAwait(false);
DevPortalOfflineStorageMetadata bundleMetadata;
using (bundleResult.BundleStream)
{
bundleResult.BundleStream.Position = 0;
bundleMetadata = await _objectStore.StoreAsync(
bundleResult.BundleStream,
new DevPortalOfflineObjectStoreOptions(bundleKey, MediaTypes.GZip),
cancellationToken).ConfigureAwait(false);
}
_logger.LogInformation(
"DevPortal offline bundle stored at {BundleKey} ({SizeBytes} bytes).",
bundleMetadata.StorageKey,
bundleMetadata.SizeBytes);
return new DevPortalOfflineJobOutcome(
bundleResult.Manifest,
bundleResult.RootHash,
signature,
manifestMetadata,
signatureMetadata,
checksumsMetadata,
bundleMetadata);
}
private Task<DevPortalOfflineStorageMetadata> StoreTextAsync(
string content,
string storageKey,
string contentType,
CancellationToken cancellationToken)
{
var bytes = Encoding.UTF8.GetBytes(content);
var stream = new MemoryStream(bytes, writable: false);
return StoreAsync(stream, storageKey, contentType, cancellationToken);
}
private async Task<DevPortalOfflineStorageMetadata> StoreAsync(
Stream stream,
string storageKey,
string contentType,
CancellationToken cancellationToken)
{
try
{
return await _objectStore.StoreAsync(
stream,
new DevPortalOfflineObjectStoreOptions(storageKey, contentType),
cancellationToken)
.ConfigureAwait(false);
}
finally
{
await stream.DisposeAsync().ConfigureAwait(false);
}
}
private static string BuildStoragePrefix(string? prefix, Guid bundleId)
{
var trimmed = string.IsNullOrWhiteSpace(prefix)
? string.Empty
: prefix.Trim().Trim('/').Replace('\\', '/');
return string.IsNullOrEmpty(trimmed)
? bundleId.ToString("D")
: $"{trimmed}/{bundleId:D}";
}
private static string SanitizeFileName(string? value)
{
if (string.IsNullOrWhiteSpace(value))
{
return "devportal-offline-bundle.tgz";
}
var fileName = Path.GetFileName(value);
if (string.IsNullOrEmpty(fileName))
{
return "devportal-offline-bundle.tgz";
}
if (fileName.Contains("..", StringComparison.Ordinal))
{
throw new InvalidOperationException("Bundle file name cannot contain path traversal sequences.");
}
return fileName;
}
private static class MediaTypes
{
public const string Json = "application/json";
public const string Text = "text/plain";
public const string GZip = "application/gzip";
}
}

View File

@@ -0,0 +1,34 @@
using System;
using System.Collections.Generic;
namespace StellaOps.ExportCenter.Core.DevPortalOffline;
/// <summary>
/// Represents the inputs required to execute the devportal offline export job.
/// </summary>
/// <param name="BundleRequest">Bundle builder request describing source directories and bundle metadata.</param>
/// <param name="StoragePrefix">Relative storage prefix (without bundle identifier) where artefacts are persisted.</param>
/// <param name="BundleFileName">File name used for the packaged archive in storage.</param>
public sealed record DevPortalOfflineJobRequest(
DevPortalOfflineBundleRequest BundleRequest,
string StoragePrefix,
string BundleFileName);
/// <summary>
/// Captures metadata produced after successfully executing the devportal offline export job.
/// </summary>
/// <param name="Manifest">The manifest describing bundled artefacts.</param>
/// <param name="RootHash">SHA-256 hash of the manifest JSON payload.</param>
/// <param name="Signature">DSSE envelope and metadata for the manifest.</param>
/// <param name="ManifestStorage">Storage metadata for the manifest JSON artefact.</param>
/// <param name="SignatureStorage">Storage metadata for the manifest signature document.</param>
/// <param name="ChecksumsStorage">Storage metadata for the checksum file.</param>
/// <param name="BundleStorage">Storage metadata for the bundled archive.</param>
public sealed record DevPortalOfflineJobOutcome(
DevPortalOfflineBundleManifest Manifest,
string RootHash,
DevPortalOfflineManifestSignatureDocument Signature,
DevPortalOfflineStorageMetadata ManifestStorage,
DevPortalOfflineStorageMetadata SignatureStorage,
DevPortalOfflineStorageMetadata ChecksumsStorage,
DevPortalOfflineStorageMetadata BundleStorage);

View File

@@ -0,0 +1,56 @@
using System;
using System.Collections.Generic;
using System.Text.Json.Serialization;
using System.Threading;
using System.Threading.Tasks;
namespace StellaOps.ExportCenter.Core.DevPortalOffline;
/// <summary>
/// Provides DSSE signing for devportal offline bundle manifests.
/// </summary>
public interface IDevPortalOfflineManifestSigner
{
Task<DevPortalOfflineManifestSignatureDocument> SignAsync(
Guid bundleId,
string manifestJson,
string rootHash,
CancellationToken cancellationToken = default);
}
/// <summary>
/// DSSE signature document emitted for devportal manifest verification.
/// </summary>
/// <param name="BundleId">Identifier of the bundle being signed.</param>
/// <param name="RootHash">SHA-256 hash of the manifest JSON.</param>
/// <param name="SignedAtUtc">UTC timestamp when the signature was created.</param>
/// <param name="Algorithm">Signing algorithm identifier (for example HMACSHA256).</param>
/// <param name="KeyId">Identifier of the signing key.</param>
/// <param name="Envelope">DSSE envelope containing payload and signatures.</param>
public sealed record DevPortalOfflineManifestSignatureDocument(
[property: JsonPropertyName("bundleId")] Guid BundleId,
[property: JsonPropertyName("rootHash")] string RootHash,
[property: JsonPropertyName("signedAt")] DateTimeOffset SignedAtUtc,
[property: JsonPropertyName("algorithm")] string Algorithm,
[property: JsonPropertyName("keyId")] string KeyId,
[property: JsonPropertyName("envelope")] DevPortalOfflineManifestDsseEnvelope Envelope);
/// <summary>
/// Standard DSSE envelope carrying payload and signatures.
/// </summary>
/// <param name="PayloadType">Type of the payload (for example application/json).</param>
/// <param name="Payload">Base64-encoded manifest payload.</param>
/// <param name="Signatures">Collection of DSSE signatures.</param>
public sealed record DevPortalOfflineManifestDsseEnvelope(
[property: JsonPropertyName("payloadType")] string PayloadType,
[property: JsonPropertyName("payload")] string Payload,
[property: JsonPropertyName("signatures")] IReadOnlyList<DevPortalOfflineManifestDsseSignature> Signatures);
/// <summary>
/// Represents an individual DSSE signature entry.
/// </summary>
/// <param name="Signature">Base64-encoded signature.</param>
/// <param name="KeyId">Identifier for the key used to sign.</param>
public sealed record DevPortalOfflineManifestDsseSignature(
[property: JsonPropertyName("sig")] string Signature,
[property: JsonPropertyName("keyid")] string? KeyId);

View File

@@ -3,16 +3,15 @@
<PropertyGroup>
<TargetFramework>net10.0</TargetFramework>
<ImplicitUsings>enable</ImplicitUsings>
<Nullable>enable</Nullable>
<LangVersion>preview</LangVersion>
<TreatWarningsAsErrors>true</TreatWarningsAsErrors>
</PropertyGroup>
<PropertyGroup>
<TargetFramework>net10.0</TargetFramework>
<ImplicitUsings>enable</ImplicitUsings>
<Nullable>enable</Nullable>
<LangVersion>preview</LangVersion>
<TreatWarningsAsErrors>true</TreatWarningsAsErrors>
</PropertyGroup>
<ItemGroup>
<PackageReference Include="Microsoft.Extensions.Logging.Abstractions" Version="10.0.0-rc.2.25502.107" />
</ItemGroup>
</Project>

View File

@@ -0,0 +1,18 @@
using System.ComponentModel.DataAnnotations;
namespace StellaOps.ExportCenter.Infrastructure.DevPortalOffline;
public sealed class DevPortalOfflineManifestSigningOptions
{
[Required]
public string KeyId { get; set; } = "devportal-offline-local";
[Required]
public string Secret { get; set; } = null!;
[Required]
public string Algorithm { get; set; } = "HMACSHA256";
[Required]
public string PayloadType { get; set; } = "application/vnd.stella.devportal.manifest+json";
}

View File

@@ -0,0 +1,9 @@
using System.ComponentModel.DataAnnotations;
namespace StellaOps.ExportCenter.Infrastructure.DevPortalOffline;
public sealed class DevPortalOfflineStorageOptions
{
[Required]
public string RootPath { get; set; } = null!;
}

View File

@@ -0,0 +1,149 @@
using System;
using System.Buffers;
using System.IO;
using System.Security.Cryptography;
using System.Threading;
using System.Threading.Tasks;
using Microsoft.Extensions.Logging;
using Microsoft.Extensions.Options;
using StellaOps.ExportCenter.Core.DevPortalOffline;
namespace StellaOps.ExportCenter.Infrastructure.DevPortalOffline;
public sealed class FileSystemDevPortalOfflineObjectStore : IDevPortalOfflineObjectStore
{
private readonly IOptionsMonitor<DevPortalOfflineStorageOptions> _options;
private readonly TimeProvider _timeProvider;
private readonly ILogger<FileSystemDevPortalOfflineObjectStore> _logger;
public FileSystemDevPortalOfflineObjectStore(
IOptionsMonitor<DevPortalOfflineStorageOptions> options,
TimeProvider timeProvider,
ILogger<FileSystemDevPortalOfflineObjectStore> logger)
{
_options = options ?? throw new ArgumentNullException(nameof(options));
_timeProvider = timeProvider ?? TimeProvider.System;
_logger = logger ?? throw new ArgumentNullException(nameof(logger));
}
public async Task<DevPortalOfflineStorageMetadata> StoreAsync(
Stream content,
DevPortalOfflineObjectStoreOptions storeOptions,
CancellationToken cancellationToken)
{
ArgumentNullException.ThrowIfNull(content);
ArgumentNullException.ThrowIfNull(storeOptions);
var root = EnsureRootPath();
var storageKey = SanitizeKey(storeOptions.StorageKey);
var fullPath = GetFullPath(root, storageKey);
Directory.CreateDirectory(Path.GetDirectoryName(fullPath)!);
content.Seek(0, SeekOrigin.Begin);
using var fileStream = new FileStream(fullPath, FileMode.Create, FileAccess.Write, FileShare.None);
using var hash = IncrementalHash.CreateHash(HashAlgorithmName.SHA256);
var buffer = ArrayPool<byte>.Shared.Rent(128 * 1024);
long totalBytes = 0;
try
{
int read;
while ((read = await content.ReadAsync(buffer.AsMemory(0, buffer.Length), cancellationToken).ConfigureAwait(false)) > 0)
{
await fileStream.WriteAsync(buffer.AsMemory(0, read), cancellationToken).ConfigureAwait(false);
hash.AppendData(buffer, 0, read);
totalBytes += read;
}
}
finally
{
ArrayPool<byte>.Shared.Return(buffer);
}
await fileStream.FlushAsync(cancellationToken).ConfigureAwait(false);
content.Seek(0, SeekOrigin.Begin);
var sha = Convert.ToHexString(hash.GetHashAndReset()).ToLowerInvariant();
var createdAt = _timeProvider.GetUtcNow();
_logger.LogDebug("Stored devportal artefact at {Path} ({Bytes} bytes).", fullPath, totalBytes);
return new DevPortalOfflineStorageMetadata(
storageKey,
storeOptions.ContentType,
totalBytes,
sha,
createdAt);
}
public Task<bool> ExistsAsync(string storageKey, CancellationToken cancellationToken)
{
var root = EnsureRootPath();
var fullPath = GetFullPath(root, SanitizeKey(storageKey));
var exists = File.Exists(fullPath);
return Task.FromResult(exists);
}
public Task<Stream> OpenReadAsync(string storageKey, CancellationToken cancellationToken)
{
var root = EnsureRootPath();
var fullPath = GetFullPath(root, SanitizeKey(storageKey));
if (!File.Exists(fullPath))
{
throw new FileNotFoundException($"DevPortal offline artefact '{storageKey}' was not found.", fullPath);
}
Stream stream = new FileStream(fullPath, FileMode.Open, FileAccess.Read, FileShare.Read);
return Task.FromResult(stream);
}
private string EnsureRootPath()
{
var root = _options.CurrentValue.RootPath;
if (string.IsNullOrWhiteSpace(root))
{
throw new InvalidOperationException("DevPortal offline storage root path is not configured.");
}
var full = Path.GetFullPath(root);
if (!Directory.Exists(full))
{
Directory.CreateDirectory(full);
}
if (!full.EndsWith(Path.DirectorySeparatorChar))
{
full += Path.DirectorySeparatorChar;
}
return full;
}
private static string SanitizeKey(string storageKey)
{
if (string.IsNullOrWhiteSpace(storageKey))
{
throw new ArgumentException("Storage key cannot be empty.", nameof(storageKey));
}
if (storageKey.Contains("..", StringComparison.Ordinal))
{
throw new ArgumentException("Storage key cannot contain path traversal sequences.", nameof(storageKey));
}
var trimmed = storageKey.Trim().Trim('/').Replace('\\', '/');
return trimmed;
}
private static string GetFullPath(string root, string storageKey)
{
var combined = Path.Combine(root, storageKey.Replace('/', Path.DirectorySeparatorChar));
var fullPath = Path.GetFullPath(combined);
if (!fullPath.StartsWith(root, StringComparison.Ordinal))
{
throw new InvalidOperationException("Storage key resolves outside of configured root path.");
}
return fullPath;
}
}

View File

@@ -0,0 +1,121 @@
using System;
using System.Buffers.Binary;
using System.ComponentModel.DataAnnotations;
using System.Security.Cryptography;
using System.Text;
using System.Threading;
using System.Threading.Tasks;
using Microsoft.Extensions.Logging;
using Microsoft.Extensions.Options;
using StellaOps.ExportCenter.Core.DevPortalOffline;
namespace StellaOps.ExportCenter.Infrastructure.DevPortalOffline;
public sealed class HmacDevPortalOfflineManifestSigner : IDevPortalOfflineManifestSigner
{
private readonly IOptionsMonitor<DevPortalOfflineManifestSigningOptions> _options;
private readonly TimeProvider _timeProvider;
private readonly ILogger<HmacDevPortalOfflineManifestSigner> _logger;
public HmacDevPortalOfflineManifestSigner(
IOptionsMonitor<DevPortalOfflineManifestSigningOptions> options,
TimeProvider timeProvider,
ILogger<HmacDevPortalOfflineManifestSigner> logger)
{
_options = options ?? throw new ArgumentNullException(nameof(options));
_timeProvider = timeProvider ?? TimeProvider.System;
_logger = logger ?? throw new ArgumentNullException(nameof(logger));
}
public Task<DevPortalOfflineManifestSignatureDocument> SignAsync(
Guid bundleId,
string manifestJson,
string rootHash,
CancellationToken cancellationToken = default)
{
if (string.IsNullOrWhiteSpace(manifestJson))
{
throw new ArgumentException("Manifest JSON is required.", nameof(manifestJson));
}
if (string.IsNullOrWhiteSpace(rootHash))
{
throw new ArgumentException("Root hash is required.", nameof(rootHash));
}
var options = _options.CurrentValue;
ValidateOptions(options);
var signedAt = _timeProvider.GetUtcNow();
var payloadBytes = Encoding.UTF8.GetBytes(manifestJson);
var pae = BuildPreAuthEncoding(options.PayloadType, payloadBytes);
var signature = ComputeSignature(options, pae);
var payloadBase64 = Convert.ToBase64String(payloadBytes);
_logger.LogDebug("Signed devportal manifest for bundle {BundleId}.", bundleId);
var envelope = new DevPortalOfflineManifestDsseEnvelope(
options.PayloadType,
payloadBase64,
new[]
{
new DevPortalOfflineManifestDsseSignature(signature, options.KeyId)
});
var document = new DevPortalOfflineManifestSignatureDocument(
bundleId,
rootHash,
signedAt,
options.Algorithm,
options.KeyId,
envelope);
return Task.FromResult(document);
}
private static void ValidateOptions(DevPortalOfflineManifestSigningOptions options)
{
Validator.ValidateObject(options, new ValidationContext(options), validateAllProperties: true);
if (!string.Equals(options.Algorithm, "HMACSHA256", StringComparison.OrdinalIgnoreCase))
{
throw new NotSupportedException($"Algorithm '{options.Algorithm}' is not supported for devportal manifest signing.");
}
}
private static string ComputeSignature(DevPortalOfflineManifestSigningOptions options, byte[] pae)
{
var secretBytes = Convert.FromBase64String(options.Secret);
using var hmac = new HMACSHA256(secretBytes);
var signatureBytes = hmac.ComputeHash(pae);
return Convert.ToBase64String(signatureBytes);
}
private static byte[] BuildPreAuthEncoding(string payloadType, byte[] payloadBytes)
{
var typeBytes = Encoding.UTF8.GetBytes(payloadType ?? string.Empty);
const string prefix = "DSSEv1";
var totalLength = prefix.Length +
sizeof(ulong) +
sizeof(ulong) + typeBytes.Length +
sizeof(ulong) + payloadBytes.Length;
var buffer = new byte[totalLength];
var span = buffer.AsSpan();
var offset = Encoding.UTF8.GetBytes(prefix, span);
BinaryPrimitives.WriteUInt64BigEndian(span[offset..], 2);
offset += sizeof(ulong);
BinaryPrimitives.WriteUInt64BigEndian(span[offset..], (ulong)typeBytes.Length);
offset += sizeof(ulong);
typeBytes.CopyTo(span[offset..]);
offset += typeBytes.Length;
BinaryPrimitives.WriteUInt64BigEndian(span[offset..], (ulong)payloadBytes.Length);
offset += sizeof(ulong);
payloadBytes.CopyTo(span[offset..]);
return buffer;
}
}

View File

@@ -1,28 +1,19 @@
<?xml version="1.0" ?>
<Project Sdk="Microsoft.NET.Sdk">
<ItemGroup>
<ProjectReference Include="..\StellaOps.ExportCenter.Core\StellaOps.ExportCenter.Core.csproj"/>
</ItemGroup>
<PropertyGroup>
<TargetFramework>net10.0</TargetFramework>
<ImplicitUsings>enable</ImplicitUsings>
<Nullable>enable</Nullable>
<LangVersion>preview</LangVersion>
<TreatWarningsAsErrors>true</TreatWarningsAsErrors>
</PropertyGroup>
</Project>
<?xml version="1.0" encoding="utf-8"?>
<Project Sdk="Microsoft.NET.Sdk">
<PropertyGroup>
<TargetFramework>net10.0</TargetFramework>
<ImplicitUsings>enable</ImplicitUsings>
<Nullable>enable</Nullable>
<LangVersion>preview</LangVersion>
<TreatWarningsAsErrors>true</TreatWarningsAsErrors>
</PropertyGroup>
<ItemGroup>
<ProjectReference Include="..\StellaOps.ExportCenter.Core\StellaOps.ExportCenter.Core.csproj" />
</ItemGroup>
<ItemGroup>
<PackageReference Include="Microsoft.Extensions.Logging.Abstractions" Version="10.0.0-rc.2.25502.107" />
<PackageReference Include="Microsoft.Extensions.Options" Version="10.0.0-rc.2.25502.107" />
</ItemGroup>
</Project>

View File

@@ -52,7 +52,7 @@ public sealed class DevPortalOfflineBundleBuilderTests
var fixedNow = new DateTimeOffset(2025, 11, 4, 12, 30, 0, TimeSpan.Zero);
var builder = new DevPortalOfflineBundleBuilder(new FixedTimeProvider(fixedNow));
var result = builder.Build(request);
var result = builder.Build(request, TestContext.Current.CancellationToken);
Assert.Equal(request.BundleId, result.Manifest.BundleId);
Assert.Equal("devportal-offline/v1", result.Manifest.Version);
@@ -65,12 +65,12 @@ public sealed class DevPortalOfflineBundleBuilderTests
var expectedPaths = new[]
{
"changelog/CHANGELOG.md",
"portal/assets/app.js",
"portal/index.html",
"specs/openapi.yaml",
"sdks/dotnet/stellaops.sdk.nupkg",
"sdks/python/stellaops_sdk.whl",
"changelog/CHANGELOG.md"
"specs/openapi.yaml"
};
Assert.Equal(expectedPaths, result.Manifest.Entries.Select(entry => entry.Path).ToArray());
@@ -119,7 +119,10 @@ public sealed class DevPortalOfflineBundleBuilderTests
}
finally
{
tempRoot.Dispose();
if (Directory.Exists(tempRoot.FullName))
{
Directory.Delete(tempRoot.FullName, recursive: true);
}
}
}
@@ -129,7 +132,7 @@ public sealed class DevPortalOfflineBundleBuilderTests
var builder = new DevPortalOfflineBundleBuilder(new FixedTimeProvider(DateTimeOffset.UtcNow));
var request = new DevPortalOfflineBundleRequest(Guid.NewGuid());
var exception = Assert.Throws<InvalidOperationException>(() => builder.Build(request));
var exception = Assert.Throws<InvalidOperationException>(() => builder.Build(request, TestContext.Current.CancellationToken));
Assert.Contains("does not contain any files", exception.Message, StringComparison.Ordinal);
}
@@ -145,7 +148,7 @@ public sealed class DevPortalOfflineBundleBuilderTests
File.WriteAllText(Path.Combine(portalRoot, "index.html"), "<html/>");
var builder = new DevPortalOfflineBundleBuilder(new FixedTimeProvider(DateTimeOffset.UtcNow));
var result = builder.Build(new DevPortalOfflineBundleRequest(Guid.NewGuid(), portalRoot));
var result = builder.Build(new DevPortalOfflineBundleRequest(Guid.NewGuid(), portalRoot), TestContext.Current.CancellationToken);
Assert.Single(result.Manifest.Entries);
Assert.True(result.Manifest.Sources.PortalIncluded);
@@ -155,7 +158,10 @@ public sealed class DevPortalOfflineBundleBuilderTests
}
finally
{
tempRoot.Dispose();
if (Directory.Exists(tempRoot.FullName))
{
Directory.Delete(tempRoot.FullName, recursive: true);
}
}
}
@@ -166,7 +172,7 @@ public sealed class DevPortalOfflineBundleBuilderTests
var missing = Path.Combine(Path.GetTempPath(), Guid.NewGuid().ToString("N"));
var request = new DevPortalOfflineBundleRequest(Guid.NewGuid(), missing);
Assert.Throws<DirectoryNotFoundException>(() => builder.Build(request));
Assert.Throws<DirectoryNotFoundException>(() => builder.Build(request, TestContext.Current.CancellationToken));
}
private static string CalculateFileHash(string path)

View File

@@ -0,0 +1,217 @@
using System;
using System.Collections.Generic;
using System.IO;
using System.Security.Cryptography;
using System.Text;
using System.Text.Json;
using System.Threading;
using System.Threading.Tasks;
using StellaOps.ExportCenter.Core.DevPortalOffline;
using Microsoft.Extensions.Logging.Abstractions;
namespace StellaOps.ExportCenter.Tests;
public class DevPortalOfflineJobTests
{
[Fact]
public async Task ExecuteAsync_StoresArtefacts()
{
var tempRoot = Directory.CreateTempSubdirectory();
try
{
var portalRoot = Path.Combine(tempRoot.FullName, "portal");
Directory.CreateDirectory(portalRoot);
File.WriteAllText(Path.Combine(portalRoot, "index.html"), "<html>offline</html>");
var specsRoot = Path.Combine(tempRoot.FullName, "specs");
Directory.CreateDirectory(specsRoot);
File.WriteAllText(Path.Combine(specsRoot, "api.json"), "{\"openapi\":\"3.1.0\"}");
var sdkRoot = Path.Combine(tempRoot.FullName, "sdk-dotnet");
Directory.CreateDirectory(sdkRoot);
File.WriteAllText(Path.Combine(sdkRoot, "stellaops.sdk.nupkg"), "binary");
var changelogRoot = Path.Combine(tempRoot.FullName, "changelog");
Directory.CreateDirectory(changelogRoot);
File.WriteAllText(Path.Combine(changelogRoot, "CHANGELOG.md"), "# 2025.11.0");
var bundleId = Guid.Parse("5e76bb6f-2925-41ea-8e72-1fd8a384dd1a");
var request = new DevPortalOfflineBundleRequest(
bundleId,
portalRoot,
specsRoot,
new[] { new DevPortalSdkSource("dotnet", sdkRoot) },
changelogRoot,
new Dictionary<string, string> { ["releaseVersion"] = "2025.11.0" });
var fixedNow = new DateTimeOffset(2025, 11, 4, 18, 15, 0, TimeSpan.Zero);
var timeProvider = new FixedTimeProvider(fixedNow);
var builder = new DevPortalOfflineBundleBuilder(timeProvider);
var objectStore = new InMemoryObjectStore(timeProvider);
var signer = new TestManifestSigner(timeProvider);
var job = new DevPortalOfflineJob(builder, objectStore, signer, NullLogger<DevPortalOfflineJob>.Instance);
var outcome = await job.ExecuteAsync(
new DevPortalOfflineJobRequest(request, "exports/devportal", "bundle.tgz"),
TestContext.Current.CancellationToken);
var expectedPrefix = $"exports/devportal/{bundleId:D}";
Assert.Equal($"{expectedPrefix}/manifest.json", outcome.ManifestStorage.StorageKey);
Assert.Equal($"{expectedPrefix}/manifest.dsse.json", outcome.SignatureStorage.StorageKey);
Assert.Equal($"{expectedPrefix}/checksums.txt", outcome.ChecksumsStorage.StorageKey);
Assert.Equal($"{expectedPrefix}/bundle.tgz", outcome.BundleStorage.StorageKey);
var manifestText = objectStore.GetText(outcome.ManifestStorage.StorageKey);
using (var manifestDoc = JsonDocument.Parse(manifestText))
{
Assert.Equal("devportal-offline/v1", manifestDoc.RootElement.GetProperty("version").GetString());
Assert.Equal("2025.11.0", manifestDoc.RootElement.GetProperty("metadata").GetProperty("releaseVersion").GetString());
}
var signatureText = objectStore.GetText(outcome.SignatureStorage.StorageKey);
Assert.Contains(bundleId.ToString("D"), signatureText, StringComparison.OrdinalIgnoreCase);
Assert.Contains(outcome.RootHash, signatureText, StringComparison.Ordinal);
}
finally
{
if (Directory.Exists(tempRoot.FullName))
{
Directory.Delete(tempRoot.FullName, recursive: true);
}
}
}
[Fact]
public async Task ExecuteAsync_SanitizesBundleFileName()
{
var builder = new DevPortalOfflineBundleBuilder(new FixedTimeProvider(DateTimeOffset.UtcNow));
var objectStore = new InMemoryObjectStore(new FixedTimeProvider(DateTimeOffset.UtcNow));
var signer = new TestManifestSigner(new FixedTimeProvider(DateTimeOffset.UtcNow));
var job = new DevPortalOfflineJob(builder, objectStore, signer, NullLogger<DevPortalOfflineJob>.Instance);
var tempRoot = Directory.CreateTempSubdirectory();
try
{
var portalRoot = Path.Combine(tempRoot.FullName, "portal");
Directory.CreateDirectory(portalRoot);
File.WriteAllText(Path.Combine(portalRoot, "index.html"), "<html/>");
var request = new DevPortalOfflineBundleRequest(Guid.NewGuid(), portalRoot);
var outcome = await job.ExecuteAsync(
new DevPortalOfflineJobRequest(request, "exports", "../bundle.tgz"),
TestContext.Current.CancellationToken);
var expectedPrefix = $"exports/{request.BundleId:D}";
Assert.Equal($"{expectedPrefix}/bundle.tgz", outcome.BundleStorage.StorageKey);
}
finally
{
if (Directory.Exists(tempRoot.FullName))
{
Directory.Delete(tempRoot.FullName, recursive: true);
}
}
}
private sealed class InMemoryObjectStore : IDevPortalOfflineObjectStore
{
private readonly Dictionary<string, InMemoryEntry> _entries = new(StringComparer.Ordinal);
private readonly TimeProvider _timeProvider;
public InMemoryObjectStore(TimeProvider timeProvider)
{
_timeProvider = timeProvider;
}
public Task<DevPortalOfflineStorageMetadata> StoreAsync(
Stream content,
DevPortalOfflineObjectStoreOptions options,
CancellationToken cancellationToken)
{
using var memory = new MemoryStream();
content.CopyTo(memory);
var bytes = memory.ToArray();
content.Seek(0, SeekOrigin.Begin);
var sha = Convert.ToHexString(SHA256.HashData(bytes)).ToLowerInvariant();
var metadata = new DevPortalOfflineStorageMetadata(
options.StorageKey,
options.ContentType,
bytes.Length,
sha,
_timeProvider.GetUtcNow());
_entries[options.StorageKey] = new InMemoryEntry(options.ContentType, bytes);
return Task.FromResult(metadata);
}
public Task<bool> ExistsAsync(string storageKey, CancellationToken cancellationToken)
=> Task.FromResult(_entries.ContainsKey(storageKey));
public Task<Stream> OpenReadAsync(string storageKey, CancellationToken cancellationToken)
{
if (!_entries.TryGetValue(storageKey, out var entry))
{
throw new FileNotFoundException(storageKey);
}
Stream stream = new MemoryStream(entry.Content, writable: false);
return Task.FromResult(stream);
}
public string GetText(string storageKey)
{
if (!_entries.TryGetValue(storageKey, out var entry))
{
throw new FileNotFoundException(storageKey);
}
return Encoding.UTF8.GetString(entry.Content);
}
private sealed record InMemoryEntry(string ContentType, byte[] Content);
}
private sealed class TestManifestSigner : IDevPortalOfflineManifestSigner
{
private readonly TimeProvider _timeProvider;
public TestManifestSigner(TimeProvider timeProvider)
{
_timeProvider = timeProvider;
}
public Task<DevPortalOfflineManifestSignatureDocument> SignAsync(
Guid bundleId,
string manifestJson,
string rootHash,
CancellationToken cancellationToken = default)
{
var payload = Convert.ToBase64String(Encoding.UTF8.GetBytes(manifestJson));
return Task.FromResult(new DevPortalOfflineManifestSignatureDocument(
bundleId,
rootHash,
_timeProvider.GetUtcNow(),
"TEST-SHA256",
"test-key",
new DevPortalOfflineManifestDsseEnvelope(
"application/json",
payload,
new[] { new DevPortalOfflineManifestDsseSignature("c2lnbmF0dXJl", "test-key") })));
}
}
private sealed class FixedTimeProvider : TimeProvider
{
private readonly DateTimeOffset _utcNow;
public FixedTimeProvider(DateTimeOffset utcNow)
{
_utcNow = utcNow;
}
public override DateTimeOffset GetUtcNow() => _utcNow;
public override long GetTimestamp() => TimeProvider.System.GetTimestamp();
}
}

View File

@@ -0,0 +1,147 @@
using System;
using System.Buffers.Binary;
using System.Security.Cryptography;
using System.Threading;
using System.Text;
using Microsoft.Extensions.Logging.Abstractions;
using Microsoft.Extensions.Options;
using StellaOps.ExportCenter.Core.DevPortalOffline;
using StellaOps.ExportCenter.Infrastructure.DevPortalOffline;
namespace StellaOps.ExportCenter.Tests;
public class HmacDevPortalOfflineManifestSignerTests
{
[Fact]
public async Task SignAsync_ComputesDeterministicSignature()
{
var options = new DevPortalOfflineManifestSigningOptions
{
KeyId = "devportal-test",
Secret = Convert.ToBase64String(Encoding.UTF8.GetBytes("shared-secret")),
Algorithm = "HMACSHA256",
PayloadType = "application/vnd.stella.devportal.manifest+json"
};
var now = new DateTimeOffset(2025, 11, 4, 19, 0, 0, TimeSpan.Zero);
var signer = new HmacDevPortalOfflineManifestSigner(
new StaticOptionsMonitor<DevPortalOfflineManifestSigningOptions>(options),
new FixedTimeProvider(now),
NullLogger<HmacDevPortalOfflineManifestSigner>.Instance);
const string manifest = "{\"version\":\"v1\",\"entries\":[]}";
var rootHash = Convert.ToHexString(SHA256.HashData(Encoding.UTF8.GetBytes(manifest))).ToLowerInvariant();
var bundleId = Guid.Parse("9ca2aafb-42b7-4df9-85f7-5a1d46c4e0ef");
var document = await signer.SignAsync(bundleId, manifest, rootHash, TestContext.Current.CancellationToken);
Assert.Equal(bundleId, document.BundleId);
Assert.Equal(rootHash, document.RootHash);
Assert.Equal(now, document.SignedAtUtc);
Assert.Equal(options.Algorithm, document.Algorithm);
Assert.Equal(options.KeyId, document.KeyId);
Assert.Equal(Convert.ToBase64String(Encoding.UTF8.GetBytes(manifest)), document.Envelope.Payload);
Assert.Equal(options.PayloadType, document.Envelope.PayloadType);
var signature = Assert.Single(document.Envelope.Signatures);
Assert.Equal(options.KeyId, signature.KeyId);
var expectedSignature = ComputeExpectedSignature(options, manifest);
Assert.Equal(expectedSignature, signature.Signature);
}
[Fact]
public async Task SignAsync_ThrowsForUnsupportedAlgorithm()
{
var options = new DevPortalOfflineManifestSigningOptions
{
KeyId = "devportal-test",
Secret = Convert.ToBase64String(Encoding.UTF8.GetBytes("shared-secret")),
Algorithm = "RSA",
PayloadType = "application/json"
};
var signer = new HmacDevPortalOfflineManifestSigner(
new StaticOptionsMonitor<DevPortalOfflineManifestSigningOptions>(options),
new FixedTimeProvider(DateTimeOffset.UtcNow),
NullLogger<HmacDevPortalOfflineManifestSigner>.Instance);
await Assert.ThrowsAsync<NotSupportedException>(() =>
signer.SignAsync(Guid.NewGuid(), "{}", "root", TestContext.Current.CancellationToken));
}
private static string ComputeExpectedSignature(DevPortalOfflineManifestSigningOptions options, string manifest)
{
var payloadBytes = Encoding.UTF8.GetBytes(manifest);
var pae = BuildPreAuthEncoding(options.PayloadType, payloadBytes);
var secret = Convert.FromBase64String(options.Secret);
using var hmac = new HMACSHA256(secret);
var signature = hmac.ComputeHash(pae);
return Convert.ToBase64String(signature);
}
private static byte[] BuildPreAuthEncoding(string payloadType, byte[] payloadBytes)
{
var typeBytes = Encoding.UTF8.GetBytes(payloadType ?? string.Empty);
const string prefix = "DSSEv1";
var buffer = new byte[prefix.Length + sizeof(ulong) + sizeof(ulong) + typeBytes.Length + sizeof(ulong) + payloadBytes.Length];
var span = buffer.AsSpan();
var written = Encoding.UTF8.GetBytes(prefix, span);
span = span[written..];
BinaryPrimitives.WriteUInt64BigEndian(span, 2);
span = span[sizeof(ulong)..];
BinaryPrimitives.WriteUInt64BigEndian(span, (ulong)typeBytes.Length);
span = span[sizeof(ulong)..];
typeBytes.CopyTo(span);
span = span[typeBytes.Length..];
BinaryPrimitives.WriteUInt64BigEndian(span, (ulong)payloadBytes.Length);
span = span[sizeof(ulong)..];
payloadBytes.CopyTo(span);
return buffer;
}
private sealed class FixedTimeProvider : TimeProvider
{
private readonly DateTimeOffset _utcNow;
public FixedTimeProvider(DateTimeOffset utcNow)
{
_utcNow = utcNow;
}
public override DateTimeOffset GetUtcNow() => _utcNow;
public override long GetTimestamp() => TimeProvider.System.GetTimestamp();
}
private sealed class StaticOptionsMonitor<T> : IOptionsMonitor<T>
{
private readonly T _value;
public StaticOptionsMonitor(T value)
{
_value = value;
}
public T CurrentValue => _value;
public T Get(string? name) => _value;
public IDisposable OnChange(Action<T, string?> listener) => NullDisposable.Instance;
private sealed class NullDisposable : IDisposable
{
public static readonly NullDisposable Instance = new();
public void Dispose()
{
}
}
}
}

View File

@@ -0,0 +1,19 @@
using System;
using System.Collections.Generic;
namespace StellaOps.ExportCenter.Worker;
public sealed class DevPortalOfflineWorkerOptions
{
public bool Enabled { get; set; }
public Guid? BundleId { get; set; }
public string? StoragePrefix { get; set; }
public string? BundleFileName { get; set; }
public string? PortalDirectory { get; set; }
public string? SpecsDirectory { get; set; }
public string? ChangelogDirectory { get; set; }
public List<DevPortalOfflineWorkerSdkSource>? SdkSources { get; set; }
public Dictionary<string, string>? Metadata { get; set; }
}
public sealed record DevPortalOfflineWorkerSdkSource(string Name, string Directory);

View File

@@ -1,7 +1,22 @@
using StellaOps.ExportCenter.Worker;
var builder = Host.CreateApplicationBuilder(args);
builder.Services.AddHostedService<Worker>();
var host = builder.Build();
host.Run();
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Hosting;
using StellaOps.ExportCenter.Core.DevPortalOffline;
using StellaOps.ExportCenter.Infrastructure.DevPortalOffline;
using StellaOps.ExportCenter.Worker;
var builder = Host.CreateApplicationBuilder(args);
builder.Services.AddSingleton(TimeProvider.System);
builder.Services.Configure<DevPortalOfflineWorkerOptions>(builder.Configuration.GetSection("DevPortalOffline"));
builder.Services.Configure<DevPortalOfflineManifestSigningOptions>(builder.Configuration.GetSection("DevPortalOffline:Signing"));
builder.Services.Configure<DevPortalOfflineStorageOptions>(builder.Configuration.GetSection("DevPortalOffline:Storage"));
builder.Services.AddSingleton<DevPortalOfflineBundleBuilder>();
builder.Services.AddSingleton<IDevPortalOfflineManifestSigner, HmacDevPortalOfflineManifestSigner>();
builder.Services.AddSingleton<IDevPortalOfflineObjectStore, FileSystemDevPortalOfflineObjectStore>();
builder.Services.AddSingleton<DevPortalOfflineJob>();
builder.Services.AddHostedService<Worker>();
var host = builder.Build();
host.Run();

View File

@@ -1,16 +1,87 @@
namespace StellaOps.ExportCenter.Worker;
public class Worker(ILogger<Worker> logger) : BackgroundService
{
protected override async Task ExecuteAsync(CancellationToken stoppingToken)
{
while (!stoppingToken.IsCancellationRequested)
{
if (logger.IsEnabled(LogLevel.Information))
{
logger.LogInformation("Worker running at: {time}", DateTimeOffset.Now);
}
await Task.Delay(1000, stoppingToken);
}
}
}
using System;
using System.Collections.Generic;
using System.Linq;
using Microsoft.Extensions.Logging;
using Microsoft.Extensions.Options;
using StellaOps.ExportCenter.Core.DevPortalOffline;
namespace StellaOps.ExportCenter.Worker;
public sealed class Worker : BackgroundService
{
private readonly ILogger<Worker> _logger;
private readonly DevPortalOfflineJob _job;
private readonly IOptions<DevPortalOfflineWorkerOptions> _options;
public Worker(
ILogger<Worker> logger,
DevPortalOfflineJob job,
IOptions<DevPortalOfflineWorkerOptions> options)
{
_logger = logger ?? throw new ArgumentNullException(nameof(logger));
_job = job ?? throw new ArgumentNullException(nameof(job));
_options = options ?? throw new ArgumentNullException(nameof(options));
}
protected override async Task ExecuteAsync(CancellationToken stoppingToken)
{
var options = _options.Value ?? new DevPortalOfflineWorkerOptions();
if (!options.Enabled)
{
_logger.LogInformation("DevPortal offline job disabled. Worker idling.");
await Task.Delay(Timeout.Infinite, stoppingToken).ConfigureAwait(false);
return;
}
try
{
var request = BuildRequest(options);
var outcome = await _job.ExecuteAsync(request, stoppingToken).ConfigureAwait(false);
_logger.LogInformation(
"DevPortal offline export completed. Bundle stored at {Key}. Manifest signature key {SignatureKey}.",
outcome.BundleStorage.StorageKey,
outcome.SignatureStorage.StorageKey);
}
catch (Exception ex) when (!stoppingToken.IsCancellationRequested)
{
_logger.LogError(ex, "DevPortal offline export job failed.");
throw;
}
await Task.Delay(Timeout.Infinite, stoppingToken).ConfigureAwait(false);
}
private static DevPortalOfflineJobRequest BuildRequest(DevPortalOfflineWorkerOptions options)
{
var bundleId = options.BundleId ?? Guid.NewGuid();
var metadata = options.Metadata is null || options.Metadata.Count == 0
? null
: new Dictionary<string, string>(options.Metadata, StringComparer.Ordinal);
var sdkSources = options.SdkSources is { Count: > 0 }
? options.SdkSources
.Select(source => new DevPortalSdkSource(source.Name, source.Directory))
.ToArray()
: Array.Empty<DevPortalSdkSource>();
var bundleRequest = new DevPortalOfflineBundleRequest(
bundleId,
options.PortalDirectory,
options.SpecsDirectory,
sdkSources,
options.ChangelogDirectory,
metadata);
var storagePrefix = options.StoragePrefix ?? "devportal/offline";
var bundleFileName = string.IsNullOrWhiteSpace(options.BundleFileName)
? "devportal-offline-bundle.tgz"
: options.BundleFileName;
return new DevPortalOfflineJobRequest(
bundleRequest,
storagePrefix,
bundleFileName);
}
}

View File

@@ -1,8 +1,25 @@
{
"Logging": {
"LogLevel": {
"Default": "Information",
"Microsoft.Hosting.Lifetime": "Information"
}
}
}
{
"Logging": {
"LogLevel": {
"Default": "Information",
"Microsoft.Hosting.Lifetime": "Information"
}
},
"DevPortalOffline": {
"Enabled": false,
"StoragePrefix": "devportal/offline",
"BundleFileName": "devportal-offline-bundle.tgz",
"Metadata": {
"releaseChannel": "stable"
},
"Storage": {
"RootPath": "./out/devportal-offline"
},
"Signing": {
"KeyId": "devportal-offline-local",
"Secret": "ZGV2cG9ydGFsLW9mZmxpbmUtc2lnbmVyLXNlY3JldA==",
"Algorithm": "HMACSHA256",
"PayloadType": "application/vnd.stella.devportal.manifest+json"
}
}
}

View File

@@ -1,3 +1,3 @@
using System.Runtime.CompilerServices;
[assembly: InternalsVisibleTo("StellaOps.Scanner.WebService.Tests")]
using System.Runtime.CompilerServices;
[assembly: InternalsVisibleTo("StellaOps.Scanner.WebService.Tests")]

View File

@@ -97,6 +97,8 @@ builder.Services.AddSurfaceEnvironment(options =>
builder.Services.AddSurfaceValidation();
builder.Services.AddSurfaceFileCache();
builder.Services.AddSurfaceSecrets();
builder.Services.AddSingleton<IConfigureOptions<SurfaceCacheOptions>>(sp =>
new SurfaceCacheOptionsConfigurator(sp.GetRequiredService<ISurfaceEnvironment>()));
builder.Services.AddSingleton<ISurfacePointerService, SurfacePointerService>();
builder.Services.AddSingleton<IRedisConnectionFactory, RedisConnectionFactory>();
if (bootstrapOptions.Events is { Enabled: true } eventsOptions
@@ -370,5 +372,24 @@ if (resolvedOptions.Features.EnablePolicyPreview)
apiGroup.MapReportEndpoints(resolvedOptions.Api.ReportsSegment);
apiGroup.MapRuntimeEndpoints(resolvedOptions.Api.RuntimeSegment);
app.MapOpenApiIfAvailable();
await app.RunAsync().ConfigureAwait(false);
app.MapOpenApiIfAvailable();
await app.RunAsync().ConfigureAwait(false);
public partial class Program;
internal sealed class SurfaceCacheOptionsConfigurator : IConfigureOptions<SurfaceCacheOptions>
{
private readonly ISurfaceEnvironment _surfaceEnvironment;
public SurfaceCacheOptionsConfigurator(ISurfaceEnvironment surfaceEnvironment)
{
_surfaceEnvironment = surfaceEnvironment ?? throw new ArgumentNullException(nameof(surfaceEnvironment));
}
public void Configure(SurfaceCacheOptions options)
{
ArgumentNullException.ThrowIfNull(options);
var settings = _surfaceEnvironment.Settings;
options.RootDirectory = settings.CacheRoot.FullName;
}
}

View File

@@ -4,7 +4,8 @@
|----|--------|----------|------------|-------------|---------------|
| SCAN-REPLAY-186-001 | TODO | Scanner WebService Guild | REPLAY-CORE-185-001 | Implement scan `record` mode producing replay manifests/bundles, capture policy/feed/tool hashes, and update `docs/modules/scanner/architecture.md` referencing `docs/replay/DETERMINISTIC_REPLAY.md` Section 6. | API/worker integration tests cover record mode; docs merged; replay artifacts stored per spec. |
| SCANNER-SURFACE-02 | DONE (2025-11-05) | Scanner WebService Guild | SURFACE-FS-02 | Publish Surface.FS pointers (CAS URIs, manifests) via scan/report APIs and update attestation metadata.<br>2025-11-05: Surface pointers projected through scan/report endpoints, orchestrator samples + DSSE fixtures refreshed with manifest block, readiness tests updated to use validator stub. | OpenAPI updated; clients regenerated; integration tests validate pointer presence and tenancy. |
| SCANNER-ENV-02 | DOING (2025-11-02) | Scanner WebService Guild, Ops Guild | SURFACE-ENV-02 | Wire Surface.Env helpers into WebService hosting (cache roots, feature flags) and document configuration.<br>2025-11-02: Cache root resolution switched to helper; feature flag bindings updated; Helm/Compose updates pending review. | Service uses helper; env table documented; helm/compose templates updated. |
| SCANNER-ENV-02 | DOING (2025-11-02) | Scanner WebService Guild, Ops Guild | SURFACE-ENV-02 | Wire Surface.Env helpers into WebService hosting (cache roots, feature flags) and document configuration.<br>2025-11-02: Cache root resolution switched to helper; feature flag bindings updated; Helm/Compose updates pending review.<br>2025-11-05 14:55Z: Aligning readiness checks, docs, and Helm/Compose templates with Surface.Env outputs and planning test coverage for configuration fallbacks.<br>2025-11-06 17:05Z: Surface.Env documentation/README refreshed; warning catalogue captured for ops handoff. | Service uses helper; env table documented; helm/compose templates updated. |
> 2025-11-05 19:18Z: Added configurator to project wiring and unit test ensuring Surface.Env cache root is honoured.
| SCANNER-SECRETS-02 | DOING (2025-11-02) | Scanner WebService Guild, Security Guild | SURFACE-SECRETS-02 | Replace ad-hoc secret wiring with Surface.Secrets for report/export operations (registry and CAS tokens).<br>2025-11-02: Export/report flows now depend on Surface.Secrets stub; integration tests in progress. | Secrets fetched through shared provider; unit/integration tests cover rotation + failure cases. |
| SCANNER-EVENTS-16-301 | BLOCKED (2025-10-26) | Scanner WebService Guild | ORCH-SVC-38-101, NOTIFY-SVC-38-001 | Emit orchestrator-compatible envelopes (`scanner.event.*`) and update integration tests to verify Notifier ingestion (no Redis queue coupling). | Tests assert envelope schema + orchestrator publish; Notifier consumer harness passes; docs updated with new event contract. Blocked by .NET 10 preview OpenAPI/Auth dependency drift preventing `dotnet test` completion. |
| SCANNER-EVENTS-16-302 | DOING (2025-10-26) | Scanner WebService Guild | SCANNER-EVENTS-16-301 | Extend orchestrator event links (report/policy/attestation) once endpoints are finalised across gateway + console. | Links section covers UI/API targets; downstream consumers validated; docs/samples updated. |

View File

@@ -0,0 +1,3 @@
using System.Runtime.CompilerServices;
[assembly: InternalsVisibleTo("StellaOps.Scanner.Worker.Tests")]

View File

@@ -36,6 +36,8 @@ builder.Services.AddSurfaceEnvironment(options =>
builder.Services.AddSurfaceValidation();
builder.Services.AddSurfaceFileCache();
builder.Services.AddSurfaceSecrets();
builder.Services.AddSingleton<IConfigureOptions<SurfaceCacheOptions>>(sp =>
new SurfaceCacheOptionsConfigurator(sp.GetRequiredService<ISurfaceEnvironment>()));
builder.Services.AddSingleton<ScannerWorkerMetrics>();
builder.Services.AddSingleton<ScanProgressReporter>();
builder.Services.AddSingleton<ScanJobProcessor>();
@@ -127,3 +129,20 @@ var host = builder.Build();
await host.RunAsync();
public partial class Program;
internal sealed class SurfaceCacheOptionsConfigurator : IConfigureOptions<SurfaceCacheOptions>
{
private readonly ISurfaceEnvironment _surfaceEnvironment;
public SurfaceCacheOptionsConfigurator(ISurfaceEnvironment surfaceEnvironment)
{
_surfaceEnvironment = surfaceEnvironment ?? throw new ArgumentNullException(nameof(surfaceEnvironment));
}
public void Configure(SurfaceCacheOptions options)
{
ArgumentNullException.ThrowIfNull(options);
var settings = _surfaceEnvironment.Settings;
options.RootDirectory = settings.CacheRoot.FullName;
}
}

View File

@@ -4,5 +4,6 @@
|----|--------|----------|------------|-------------|---------------|
| SCAN-REPLAY-186-002 | TODO | Scanner Worker Guild | REPLAY-CORE-185-001 | Enforce deterministic analyzer execution when consuming replay input bundles, emit layer Merkle metadata, and author `docs/modules/scanner/deterministic-execution.md` summarising invariants from `docs/replay/DETERMINISTIC_REPLAY.md` Section 4. | Replay mode analyzers pass determinism tests; new doc merged; integration fixtures updated. |
| SCANNER-SURFACE-01 | DOING (2025-11-02) | Scanner Worker Guild | SURFACE-FS-02 | Persist Surface.FS manifests after analyzer stages, including layer CAS metadata and EntryTrace fragments.<br>2025-11-02: Draft Surface.FS manifests emitted for sample scans; telemetry counters under review. | Integration tests prove cache entries exist; telemetry counters exported. |
| SCANNER-ENV-01 | DOING (2025-11-02) | Scanner Worker Guild | SURFACE-ENV-02 | Replace ad-hoc environment reads with `StellaOps.Scanner.Surface.Env` helpers for cache roots and CAS endpoints.<br>2025-11-02: Worker bootstrap now resolves cache roots via helper; warning path documented; smoke tests running. | Worker boots with helper; misconfiguration warnings documented; smoke tests updated. |
| SCANNER-ENV-01 | DOING (2025-11-02) | Scanner Worker Guild | SURFACE-ENV-02 | Replace ad-hoc environment reads with `StellaOps.Scanner.Surface.Env` helpers for cache roots and CAS endpoints.<br>2025-11-02: Worker bootstrap now resolves cache roots via helper; warning path documented; smoke tests running.<br>2025-11-05 14:55Z: Extending helper usage into cache/secrets configuration, updating worker validator wiring, and drafting docs/tests for new Surface.Env outputs.<br>2025-11-06 17:05Z: README/design docs updated with warning catalogue; startup logging guidance captured for ops runbooks. | Worker boots with helper; misconfiguration warnings documented; smoke tests updated. |
> 2025-11-05 19:18Z: Bound `SurfaceCacheOptions` root directory to resolved Surface.Env settings and added unit coverage around the configurator.
| SCANNER-SECRETS-01 | DOING (2025-11-02) | Scanner Worker Guild, Security Guild | SURFACE-SECRETS-02 | Adopt `StellaOps.Scanner.Surface.Secrets` for registry/CAS credentials during scan execution.<br>2025-11-02: Surface.Secrets provider wired for CAS token retrieval; integration tests added. | Secrets fetched via shared provider; legacy secret code removed; integration tests cover rotation. |

View File

@@ -0,0 +1,49 @@
using System;
using System.Collections.Generic;
using System.IO;
using System.Security.Cryptography.X509Certificates;
using StellaOps.Scanner.Surface.Env;
using StellaOps.Scanner.Surface.FS;
using Xunit;
namespace StellaOps.Scanner.WebService.Tests;
public sealed class SurfaceCacheOptionsConfiguratorTests
{
[Fact]
public void Configure_UsesSurfaceEnvironmentCacheRoot()
{
var cacheRoot = new DirectoryInfo(Path.Combine(Path.GetTempPath(), Guid.NewGuid().ToString("N")));
var settings = new SurfaceEnvironmentSettings(
new Uri("https://surface.example"),
"surface-cache",
null,
cacheRoot,
cacheQuotaMegabytes: 512,
prefetchEnabled: true,
featureFlags: Array.Empty<string>(),
secrets: new SurfaceSecretsConfiguration("file", "tenant-b", "/etc/secrets", null, null, allowInline: false),
tenant: "tenant-b",
tls: new SurfaceTlsConfiguration(null, null, new X509Certificate2Collection()));
var environment = new StubSurfaceEnvironment(settings);
var configurator = new SurfaceCacheOptionsConfigurator(environment);
var options = new SurfaceCacheOptions();
configurator.Configure(options);
Assert.Equal(cacheRoot.FullName, options.RootDirectory);
}
private sealed class StubSurfaceEnvironment : ISurfaceEnvironment
{
public StubSurfaceEnvironment(SurfaceEnvironmentSettings settings)
{
Settings = settings;
}
public SurfaceEnvironmentSettings Settings { get; }
public IReadOnlyDictionary<string, string> RawVariables { get; } = new Dictionary<string, string>();
}
}

View File

@@ -0,0 +1,49 @@
using System;
using System.Collections.Generic;
using System.IO;
using System.Security.Cryptography.X509Certificates;
using StellaOps.Scanner.Surface.Env;
using StellaOps.Scanner.Surface.FS;
using Xunit;
namespace StellaOps.Scanner.Worker.Tests;
public sealed class SurfaceCacheOptionsConfiguratorTests
{
[Fact]
public void Configure_UsesSurfaceEnvironmentCacheRoot()
{
var cacheRoot = new DirectoryInfo(Path.Combine(Path.GetTempPath(), Guid.NewGuid().ToString("N")));
var settings = new SurfaceEnvironmentSettings(
new Uri("https://surface.example"),
"surface-cache",
null,
cacheRoot,
cacheQuotaMegabytes: 1024,
prefetchEnabled: false,
featureFlags: Array.Empty<string>(),
secrets: new SurfaceSecretsConfiguration("file", "tenant-a", "/etc/secrets", null, null, false),
tenant: "tenant-a",
tls: new SurfaceTlsConfiguration(null, null, new X509Certificate2Collection()));
var environment = new StubSurfaceEnvironment(settings);
var configurator = new SurfaceCacheOptionsConfigurator(environment);
var options = new SurfaceCacheOptions();
configurator.Configure(options);
Assert.Equal(cacheRoot.FullName, options.RootDirectory);
}
private sealed class StubSurfaceEnvironment : ISurfaceEnvironment
{
public StubSurfaceEnvironment(SurfaceEnvironmentSettings settings)
{
Settings = settings;
}
public SurfaceEnvironmentSettings Settings { get; }
public IReadOnlyDictionary<string, string> RawVariables { get; } = new Dictionary<string, string>();
}
}

View File

@@ -1,6 +1,6 @@
namespace StellaOps.Scheduler.WebService.GraphJobs;
internal readonly record struct GraphJobUpdateResult<TJob>(bool Updated, TJob Job) where TJob : class
public readonly record struct GraphJobUpdateResult<TJob>(bool Updated, TJob Job) where TJob : class
{
public static GraphJobUpdateResult<TJob> UpdatedResult(TJob job) => new(true, job);

Some files were not shown because too many files have changed in this diff Show More