diff --git a/AGENTS.md b/AGENTS.md index dcd1d5d6..e61f0e77 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -48,24 +48,7 @@ StellaOps is contained by different modules installable via docker containers - Agent. Installable daemon that does the scanning - Zastava. Realtime monitor for allowed (verified) installations. -## 4.1) Concelier -It is webservice based module that is responsible for aggregating vulnerabilities information from various sources, parsing and normalizing them into a canonical shape, merging and deduplicating the results in one place, with export capabilities to Json and TrivyDb. It supports init and resume for all of the sources, parse/normalize and merge/deduplication operations, plus export. Export supports delta exports—similarly to full and incremential database backups. - -### 4.1.1) Usage -It supports operations to be started by cmd line: -# stella db [fetch|merge|export] [init|resume ] -or -api available on https://db.stella-ops.org - -### 4.1.2) Data flow (end‑to‑end) - -1. **Fetch**: connectors request source windows with retries/backoff, persist raw documents with SHA256/ETag metadata. -2. **Parse & Normalize**: validate to DTOs (schema-checked), quarantine failures, normalize to canonical advisories (aliases, affected ranges with NEVRA/EVR/SemVer, references, provenance). -3. **Merge & Deduplicate**: enforce precedence, build/maintain alias graphs, compute deterministic hashes, and eliminate duplicates before persisting to MongoDB. -4. **Export**: JSON tree and/or Trivy DB; package and (optionally) push; write export state. - -### 4.1.3) Architecture -For more information of the architecture see `./docs/ARCHITECTURE_CONCELIER.md`. +For more information of the architecture see `./docs/*ARCHITECTURE*.md` files. --- @@ -118,9 +101,9 @@ You main characteristics: - **Directory ownership**: Each agent works **only inside its module directory**. Cross‑module edits require a brief handshake in issues/PR description. - **Scoping**: Use each module’s `AGENTS.md` and `TASKS.md` to plan; autonomous agents must read `src/AGENTS.md` and the module docs before acting. - **Determinism**: Sort keys, normalize timestamps to UTC ISO‑8601, avoid non‑deterministic data in exports and tests. -- **Status tracking**: Update your module’s `TASKS.md` as you progress (TODO → DOING → DONE/BLOCKED). Before starting of actual work - ensure you have set the task to DOING. When complete or stop update the status in corresponding TASKS.md or in ./SPRINTS.md file. +- **Status tracking**: Update your module’s `TASKS.md` as you progress (TODO → DOING → DONE/BLOCKED). Before starting of actual work - ensure you have set the task to DOING. When complete or stop update the status in corresponding TASKS.md and in ./SPRINTS.md and ./EXECPLAN.md file. - **Coordination**: In case task is discovered as blocked on other team or task, according TASKS.md files that dependency is on needs to be changed by adding new tasks describing the requirement. the current task must be updated as completed. In case task changes, scope or requirements or rules - other documentations needs be updated accordingly. -- **Sprint synchronization**: When given task seek for relevant directory to work on from SPRINTS.md. Confirm its state on both SPRINTS.md and the relevant TODOS.md file. Always check the AGENTS.md in the relevant TODOS.md directory. +- **Sprint synchronization**: When given task seek for relevant directory to work on from SPRINTS.md. Confirm its state on both SPRINTS.md and EXECPLAN.md and the relevant TASKS.md file. Always check the AGENTS.md in the relevant TASKS.md directory. - **Tests**: Add/extend fixtures and unit tests per change; never regress determinism or precedence. - **Test layout**: Use module-specific projects in `StellaOps.Concelier..Tests`; shared fixtures/harnesses live in `StellaOps.Concelier.Testing`. - **Execution autonomous**: In case you need to continue with more than one options just continue sequentially, unless the continue requires design decision. diff --git a/EXECPLAN.md b/EXECPLAN.md index 35388cce..baab0178 100644 --- a/EXECPLAN.md +++ b/EXECPLAN.md @@ -4,10 +4,10 @@ Generated from SPRINTS.md and module TASKS.md files on 2025-10-19. Waves cluster ## Wave Instructions ### Wave 0 - Team Attestor Guild: read EXECPLAN.md Wave 0 and SPRINTS.md rows for `src/StellaOps.Attestor/TASKS.md`. Focus on ATTESTOR-API-11-201 (TODO), ATTESTOR-VERIFY-11-202 (TODO), ATTESTOR-OBS-11-203 (TODO). Confirm prerequisites (none) before starting and report status in module TASKS.md. -- Team Authority Core & Security Guild: read EXECPLAN.md Wave 0 and SPRINTS.md rows for `src/StellaOps.Authority/TASKS.md`. Focus on AUTH-DPOP-11-001 (TODO), AUTH-MTLS-11-002 (TODO). Confirm prerequisites (none) before starting and report status in module TASKS.md. -- Team Authority Core & Storage Guild: read EXECPLAN.md Wave 0 and SPRINTS.md rows for `src/StellaOps.Authority/TASKS.md`. Focus on AUTHSTORAGE-MONGO-08-001 (BLOCKED). Confirm prerequisites (none) before starting and report status in module TASKS.md. +- Team Authority Core & Security Guild: read EXECPLAN.md Wave 0 and SPRINTS.md rows for `src/StellaOps.Authority/TASKS.md`. Focus on AUTH-DPOP-11-001 (DOING 2025-10-19), AUTH-MTLS-11-002 (DOING 2025-10-19). Confirm prerequisites (none) before starting and report status in module TASKS.md. +- Team Authority Core & Storage Guild: read EXECPLAN.md Wave 0 and SPRINTS.md rows for `src/StellaOps.Authority/TASKS.md`. Focus on AUTHSTORAGE-MONGO-08-001 (DONE 2025-10-19). Confirm prerequisites (none) before starting and report status in module TASKS.md. - Team DevEx/CLI: read EXECPLAN.md Wave 0 and SPRINTS.md rows for `src/StellaOps.Cli/TASKS.md`. Focus on EXCITITOR-CLI-01-002 (TODO), CLI-RUNTIME-13-005 (TODO). Confirm prerequisites (external: EXCITITOR-CLI-01-001, EXCITITOR-EXPORT-01-001) before starting and report status in module TASKS.md. -- Team DevOps Guild: read EXECPLAN.md Wave 0 and SPRINTS.md rows for `ops/devops/TASKS.md`. Focus on DEVOPS-SEC-10-301 (TODO). Confirm prerequisites (none) before starting and report status in module TASKS.md. +- Team DevOps Guild: read EXECPLAN.md Wave 0 and SPRINTS.md rows for `ops/devops/TASKS.md`. Focus on DEVOPS-SEC-10-301 (DOING 2025-10-19); Wave 0A prerequisites reconfirmed so remediation work may proceed. Keep module TASKS.md/Sprints in sync as patches land. - Team Diff Guild: read EXECPLAN.md Wave 0 and SPRINTS.md rows for `src/StellaOps.Scanner.Diff/TASKS.md`. Focus on SCANNER-DIFF-10-501 (TODO), SCANNER-DIFF-10-502 (TODO), SCANNER-DIFF-10-503 (TODO). Confirm prerequisites (none) before starting and report status in module TASKS.md. - Team Docs Guild, Plugin Team: read EXECPLAN.md Wave 0 and SPRINTS.md rows for `docs/TASKS.md`. Focus on DOC4.AUTH-PDG (REVIEW). Confirm prerequisites (none) before starting and report status in module TASKS.md. - Team Docs/CLI: read EXECPLAN.md Wave 0 and SPRINTS.md rows for `src/StellaOps.Cli/TASKS.md`. Focus on EXCITITOR-CLI-01-003 (TODO). Confirm prerequisites (external: EXCITITOR-CLI-01-001) before starting and report status in module TASKS.md. @@ -19,7 +19,7 @@ Generated from SPRINTS.md and module TASKS.md files on 2025-10-19. Waves cluster - Team Notify WebService Guild: read EXECPLAN.md Wave 0 and SPRINTS.md rows for `src/StellaOps.Notify.WebService/TASKS.md`. Focus on NOTIFY-WEB-15-101 (TODO), NOTIFY-WEB-15-102 (TODO). Confirm prerequisites (none) before starting and report status in module TASKS.md. - Team Platform Events Guild: read EXECPLAN.md Wave 0 and SPRINTS.md rows for `docs/TASKS.md`. Focus on PLATFORM-EVENTS-09-401 (TODO). Confirm prerequisites (external: DOCS-EVENTS-09-003) before starting and report status in module TASKS.md. - Team Plugin Platform Guild: read EXECPLAN.md Wave 0 and SPRINTS.md rows for `src/StellaOps.Plugin/TASKS.md`. Focus on PLUGIN-DI-08-001 (TODO). Confirm prerequisites (none) before starting and report status in module TASKS.md. -- Team Plugin Platform Guild, Authority Core: read EXECPLAN.md Wave 0 and SPRINTS.md rows for `src/StellaOps.Plugin/TASKS.md`. Focus on PLUGIN-DI-08-002 (TODO). Confirm prerequisites (none) before starting and report status in module TASKS.md. +- Team Plugin Platform Guild, Authority Core: read EXECPLAN.md Wave 0 and SPRINTS.md rows for `src/StellaOps.Plugin/TASKS.md`. Focus on PLUGIN-DI-08-002 (TODO); coordination session booked for 2025-10-20 to unblock implementation. Confirm prerequisites (none) before starting and report status in module TASKS.md. - Team Policy Guild: read EXECPLAN.md Wave 0 and SPRINTS.md rows for `src/StellaOps.Policy/TASKS.md`. Focus on POLICY-CORE-09-004 (TODO), POLICY-CORE-09-005 (TODO), POLICY-CORE-09-006 (TODO). Confirm prerequisites (none) before starting and report status in module TASKS.md. - Team Runtime Guild: read EXECPLAN.md Wave 0 and SPRINTS.md rows for `docs/TASKS.md`. Focus on RUNTIME-GUILD-09-402 (TODO). Confirm prerequisites (external: SCANNER-POLICY-09-107) before starting and report status in module TASKS.md. - Team Scanner WebService Guild: read EXECPLAN.md Wave 0 and SPRINTS.md rows for `src/StellaOps.Scanner.WebService/TASKS.md`. Focus on SCANNER-EVENTS-15-201 (TODO). Confirm prerequisites (none) before starting and report status in module TASKS.md. @@ -39,11 +39,11 @@ Generated from SPRINTS.md and module TASKS.md files on 2025-10-19. Waves cluster - Team Team Excititor Connectors – Ubuntu: read EXECPLAN.md Wave 0 and SPRINTS.md rows for `src/StellaOps.Excititor.Connectors.Ubuntu.CSAF/TASKS.md`. Focus on EXCITITOR-CONN-UBUNTU-01-002 (TODO). Confirm prerequisites (external: EXCITITOR-CONN-UBUNTU-01-001, EXCITITOR-STORAGE-01-003) before starting and report status in module TASKS.md. - Team Team Excititor Export: read EXECPLAN.md Wave 0 and SPRINTS.md rows for `src/StellaOps.Excititor.Export/TASKS.md`. Focus on EXCITITOR-EXPORT-01-005 (TODO). Confirm prerequisites (external: EXCITITOR-CORE-02-001, EXCITITOR-EXPORT-01-004) before starting and report status in module TASKS.md. - Team Team Excititor Formats: read EXECPLAN.md Wave 0 and SPRINTS.md rows for `src/StellaOps.Excititor.Formats.CSAF/TASKS.md`, `src/StellaOps.Excititor.Formats.CycloneDX/TASKS.md`, `src/StellaOps.Excititor.Formats.OpenVEX/TASKS.md`. Focus on EXCITITOR-FMT-CSAF-01-002 (TODO), EXCITITOR-FMT-CSAF-01-003 (TODO), EXCITITOR-FMT-CYCLONE-01-002 (TODO), EXCITITOR-FMT-CYCLONE-01-003 (TODO), EXCITITOR-FMT-OPENVEX-01-002 (TODO), EXCITITOR-FMT-OPENVEX-01-003 (TODO). Confirm prerequisites (external: EXCITITOR-EXPORT-01-001, EXCITITOR-FMT-CSAF-01-001, EXCITITOR-FMT-CYCLONE-01-001, EXCITITOR-FMT-OPENVEX-01-001, EXCITITOR-POLICY-01-001) before starting and report status in module TASKS.md. -- Team Team Excititor Storage: read EXECPLAN.md Wave 0 and SPRINTS.md rows for `src/StellaOps.Excititor.Storage.Mongo/TASKS.md`. Focus on EXCITITOR-STORAGE-MONGO-08-001 (TODO), EXCITITOR-STORAGE-03-001 (TODO). Confirm prerequisites (external: EXCITITOR-STORAGE-01-003, EXCITITOR-STORAGE-02-001) before starting and report status in module TASKS.md. +- Team Team Excititor Storage: read EXECPLAN.md Wave 0 and SPRINTS.md rows for `src/StellaOps.Excititor.Storage.Mongo/TASKS.md`. Focus on EXCITITOR-STORAGE-MONGO-08-001 (DONE 2025-10-19), EXCITITOR-STORAGE-03-001 (TODO). Confirm prerequisites (external: EXCITITOR-STORAGE-01-003, EXCITITOR-STORAGE-02-001) before starting and report status in module TASKS.md. - Team Team Excititor WebService: read EXECPLAN.md Wave 0 and SPRINTS.md rows for `src/StellaOps.Excititor.WebService/TASKS.md`. Focus on EXCITITOR-WEB-01-002 (TODO), EXCITITOR-WEB-01-003 (TODO), EXCITITOR-WEB-01-004 (TODO). Confirm prerequisites (external: EXCITITOR-ATTEST-01-001, EXCITITOR-EXPORT-01-001, EXCITITOR-WEB-01-001) before starting and report status in module TASKS.md. - Team Team Excititor Worker: read EXECPLAN.md Wave 0 and SPRINTS.md rows for `src/StellaOps.Excititor.Worker/TASKS.md`. Focus on EXCITITOR-WORKER-01-002 (TODO), EXCITITOR-WORKER-01-004 (TODO), EXCITITOR-WORKER-02-001 (TODO). Confirm prerequisites (external: EXCITITOR-CORE-02-001, EXCITITOR-WORKER-01-001) before starting and report status in module TASKS.md. -- Team Team Merge & QA Enforcement: read EXECPLAN.md Wave 0 and SPRINTS.md rows for `src/StellaOps.Concelier.Merge/TASKS.md`. Focus on FEEDMERGE-COORD-02-900 (DOING). Confirm prerequisites (none) before starting and report status in module TASKS.md. -- Team Team Normalization & Storage Backbone: read EXECPLAN.md Wave 0 and SPRINTS.md rows for `src/StellaOps.Concelier.Storage.Mongo/TASKS.md`. Focus on FEEDSTORAGE-MONGO-08-001 (TODO). Confirm prerequisites (none) before starting and report status in module TASKS.md. +- Team Team Merge & QA Enforcement: read EXECPLAN.md Wave 0 and SPRINTS.md rows for `src/StellaOps.Concelier.Merge/TASKS.md`. Focus on FEEDMERGE-COORD-02-900 (DOING). Confirm prerequisites (none) before starting and report status in module TASKS.md. **2025-10-19:** Coordination refreshed; connector owners notified and TASKS.md entries updated. +- Team Team Normalization & Storage Backbone: read EXECPLAN.md Wave 0 and SPRINTS.md rows for `src/StellaOps.Concelier.Storage.Mongo/TASKS.md`. Focus on FEEDSTORAGE-MONGO-08-001 (DONE 2025-10-19). Confirm prerequisites (none) before starting and report status in module TASKS.md. - Team Team WebService & Authority: read EXECPLAN.md Wave 0 and SPRINTS.md rows for `src/StellaOps.Authority/StellaOps.Authority.Plugin.Standard/TASKS.md`, `src/StellaOps.Concelier.WebService/TASKS.md`. Focus on SEC2.PLG (DOING), SEC3.PLG (DOING), SEC5.PLG (DOING), PLG4-6.CAPABILITIES (BLOCKED), PLG6.DIAGRAM (TODO), PLG7.RFC (REVIEW), FEEDWEB-DOCS-01-001 (DOING), FEEDWEB-OPS-01-006 (TODO), FEEDWEB-OPS-01-007 (BLOCKED). Confirm prerequisites (none) before starting and report status in module TASKS.md. - Team Tools Guild, BE-Conn-MSRC: read EXECPLAN.md Wave 0 and SPRINTS.md rows for `src/StellaOps.Concelier.Connector.Common/TASKS.md`. Focus on FEEDCONN-SHARED-STATE-003 (**TODO). Confirm prerequisites (none) before starting and report status in module TASKS.md. - Team UX Specialist, Angular Eng: read EXECPLAN.md Wave 0 and SPRINTS.md rows for `src/StellaOps.Web/TASKS.md`. Focus on WEB1.TRIVY-SETTINGS (TODO). Confirm prerequisites (none) before starting and report status in module TASKS.md. @@ -60,7 +60,7 @@ Generated from SPRINTS.md and module TASKS.md files on 2025-10-19. Waves cluster - Team Licensing Guild: read EXECPLAN.md Wave 1 and SPRINTS.md rows for `ops/licensing/TASKS.md`. Focus on DEVOPS-LIC-14-004 (TODO). Confirm prerequisites (internal: AUTH-MTLS-11-002 (Wave 0)) before starting and report status in module TASKS.md. - Team Notify Engine Guild: read EXECPLAN.md Wave 1 and SPRINTS.md rows for `src/StellaOps.Notify.Engine/TASKS.md`. Focus on NOTIFY-ENGINE-15-301 (TODO). Confirm prerequisites (internal: NOTIFY-MODELS-15-101 (Wave 0)) before starting and report status in module TASKS.md. - Team Notify Queue Guild: read EXECPLAN.md Wave 1 and SPRINTS.md rows for `src/StellaOps.Notify.Queue/TASKS.md`. Focus on NOTIFY-QUEUE-15-401 (TODO). Confirm prerequisites (internal: NOTIFY-MODELS-15-101 (Wave 0)) before starting and report status in module TASKS.md. -- Team Notify WebService Guild: read EXECPLAN.md Wave 1 and SPRINTS.md rows for `src/StellaOps.Notify.WebService/TASKS.md`. Focus on NOTIFY-WEB-15-103 (TODO). Confirm prerequisites (internal: NOTIFY-WEB-15-102 (Wave 0)) before starting and report status in module TASKS.md. +- Team Notify WebService Guild: read EXECPLAN.md Wave 1 and SPRINTS.md rows for `src/StellaOps.Notify.WebService/TASKS.md`. Focus on NOTIFY-WEB-15-103 (DONE). Confirm prerequisites (internal: NOTIFY-WEB-15-102 (Wave 0)) before starting and report status in module TASKS.md. - Team Scanner WebService Guild: read EXECPLAN.md Wave 1 and SPRINTS.md rows for `src/StellaOps.Scanner.WebService/TASKS.md`. Focus on SCANNER-RUNTIME-12-301 (TODO). Confirm prerequisites (internal: ZASTAVA-CORE-12-201 (Wave 0)) before starting and report status in module TASKS.md. - Team Scheduler ImpactIndex Guild: read EXECPLAN.md Wave 1 and SPRINTS.md rows for `src/StellaOps.Scheduler.ImpactIndex/TASKS.md`. Focus on SCHED-IMPACT-16-301 (TODO). Confirm prerequisites (internal: SCANNER-EMIT-10-605 (Wave 0)) before starting and report status in module TASKS.md. - Team Scheduler Queue Guild: read EXECPLAN.md Wave 1 and SPRINTS.md rows for `src/StellaOps.Scheduler.Queue/TASKS.md`. Focus on SCHED-QUEUE-16-402 (TODO), SCHED-QUEUE-16-403 (TODO). Confirm prerequisites (internal: SCHED-QUEUE-16-401 (Wave 0)) before starting and report status in module TASKS.md. @@ -74,14 +74,14 @@ Generated from SPRINTS.md and module TASKS.md files on 2025-10-19. Waves cluster - Team Team Excititor Connectors – Ubuntu: read EXECPLAN.md Wave 1 and SPRINTS.md rows for `src/StellaOps.Excititor.Connectors.Ubuntu.CSAF/TASKS.md`. Focus on EXCITITOR-CONN-UBUNTU-01-003 (TODO). Confirm prerequisites (internal: EXCITITOR-CONN-UBUNTU-01-002 (Wave 0); external: EXCITITOR-POLICY-01-001) before starting and report status in module TASKS.md. - Team Team Excititor Export: read EXECPLAN.md Wave 1 and SPRINTS.md rows for `src/StellaOps.Excititor.Export/TASKS.md`. Focus on EXCITITOR-EXPORT-01-006 (TODO). Confirm prerequisites (internal: EXCITITOR-EXPORT-01-005 (Wave 0), POLICY-CORE-09-005 (Wave 0)) before starting and report status in module TASKS.md. - Team Team Excititor Worker: read EXECPLAN.md Wave 1 and SPRINTS.md rows for `src/StellaOps.Excititor.Worker/TASKS.md`. Focus on EXCITITOR-WORKER-01-003 (TODO). Confirm prerequisites (internal: EXCITITOR-ATTEST-01-003 (Wave 0); external: EXCITITOR-EXPORT-01-002, EXCITITOR-WORKER-01-001) before starting and report status in module TASKS.md. -- Team UI Guild: read EXECPLAN.md Wave 1 and SPRINTS.md rows for `src/StellaOps.UI/TASKS.md`. Focus on UI-ATTEST-11-005 (TODO), UI-VEX-13-003 (TODO), UI-POLICY-13-007 (TODO), UI-ADMIN-13-004 (TODO), UI-AUTH-13-001 (TODO), UI-SCANS-13-002 (TODO), UI-NOTIFY-13-006 (TODO), UI-SCHED-13-005 (TODO). Confirm prerequisites (internal: ATTESTOR-API-11-201 (Wave 0), AUTH-DPOP-11-001 (Wave 0), AUTH-MTLS-11-002 (Wave 0), EXCITITOR-EXPORT-01-005 (Wave 0), NOTIFY-WEB-15-101 (Wave 0), POLICY-CORE-09-006 (Wave 0), SCHED-WEB-16-101 (Wave 0), SIGNER-API-11-101 (Wave 0); external: EXCITITOR-CORE-02-001, SCANNER-WEB-09-102, SCANNER-WEB-09-103) before starting and report status in module TASKS.md. +- Team UI Guild: read EXECPLAN.md Wave 1 and SPRINTS.md rows for `src/StellaOps.UI/TASKS.md`. Focus on UI-ATTEST-11-005 (TODO), UI-VEX-13-003 (TODO), UI-POLICY-13-007 (TODO), UI-ADMIN-13-004 (TODO), UI-AUTH-13-001 (TODO), UI-SCANS-13-002 (TODO), UI-NOTIFY-13-006 (DOING), UI-SCHED-13-005 (TODO). Confirm prerequisites (internal: ATTESTOR-API-11-201 (Wave 0), AUTH-DPOP-11-001 (Wave 0), AUTH-MTLS-11-002 (Wave 0), EXCITITOR-EXPORT-01-005 (Wave 0), NOTIFY-WEB-15-101 (Wave 0), POLICY-CORE-09-006 (Wave 0), SCHED-WEB-16-101 (Wave 0), SIGNER-API-11-101 (Wave 0); external: EXCITITOR-CORE-02-001, SCANNER-WEB-09-102, SCANNER-WEB-09-103) before starting and report status in module TASKS.md. - Team Zastava Observer Guild: read EXECPLAN.md Wave 1 and SPRINTS.md rows for `src/StellaOps.Zastava.Observer/TASKS.md`. Focus on ZASTAVA-OBS-12-001 (TODO). Confirm prerequisites (internal: ZASTAVA-CORE-12-201 (Wave 0)) before starting and report status in module TASKS.md. ### Wave 2 - Team Bench Guild, Notify Team: read EXECPLAN.md Wave 2 and SPRINTS.md rows for `bench/TASKS.md`. Focus on BENCH-NOTIFY-15-001 (TODO). Confirm prerequisites (internal: NOTIFY-ENGINE-15-301 (Wave 1)) before starting and report status in module TASKS.md. - Team Bench Guild, Scheduler Team: read EXECPLAN.md Wave 2 and SPRINTS.md rows for `bench/TASKS.md`. Focus on BENCH-IMPACT-16-001 (TODO). Confirm prerequisites (internal: SCHED-IMPACT-16-301 (Wave 1)) before starting and report status in module TASKS.md. - Team Deployment Guild: read EXECPLAN.md Wave 2 and SPRINTS.md rows for `ops/deployment/TASKS.md`. Focus on DEVOPS-OPS-14-003 (TODO). Confirm prerequisites (internal: DEVOPS-REL-14-001 (Wave 1)) before starting and report status in module TASKS.md. -- Team DevOps Guild: read EXECPLAN.md Wave 2 and SPRINTS.md rows for `ops/devops/TASKS.md`. Focus on DEVOPS-MIRROR-08-001 (TODO), DEVOPS-PERF-10-002 (TODO), DEVOPS-REL-17-002 (TODO). Confirm prerequisites (internal: BENCH-SCANNER-10-002 (Wave 1), DEVOPS-REL-14-001 (Wave 1), SCANNER-EMIT-17-701 (Wave 1)) before starting and report status in module TASKS.md. +- Team DevOps Guild: read EXECPLAN.md Wave 2 and SPRINTS.md rows for `ops/devops/TASKS.md`. Focus on DEVOPS-MIRROR-08-001 (DONE 2025-10-19), DEVOPS-PERF-10-002 (TODO), DEVOPS-REL-17-002 (TODO). Confirm prerequisites (internal: BENCH-SCANNER-10-002 (Wave 1), DEVOPS-REL-14-001 (Wave 1), SCANNER-EMIT-17-701 (Wave 1)) before starting and report status in module TASKS.md. - Team DevOps Guild, Notify Guild: read EXECPLAN.md Wave 2 and SPRINTS.md rows for `ops/devops/TASKS.md`. Focus on DEVOPS-SCANNER-09-205 (TODO). Confirm prerequisites (internal: DEVOPS-SCANNER-09-204 (Wave 1)) before starting and report status in module TASKS.md. - Team Notify Engine Guild: read EXECPLAN.md Wave 2 and SPRINTS.md rows for `src/StellaOps.Notify.Engine/TASKS.md`. Focus on NOTIFY-ENGINE-15-302 (TODO). Confirm prerequisites (internal: NOTIFY-ENGINE-15-301 (Wave 1)) before starting and report status in module TASKS.md. - Team Notify Queue Guild: read EXECPLAN.md Wave 2 and SPRINTS.md rows for `src/StellaOps.Notify.Queue/TASKS.md`. Focus on NOTIFY-QUEUE-15-403 (TODO), NOTIFY-QUEUE-15-402 (TODO). Confirm prerequisites (internal: NOTIFY-QUEUE-15-401 (Wave 1)) before starting and report status in module TASKS.md. @@ -121,7 +121,7 @@ Generated from SPRINTS.md and module TASKS.md files on 2025-10-19. Waves cluster ### Wave 5 - Team Excititor Connectors – Stella: read EXECPLAN.md Wave 5 and SPRINTS.md rows for `src/StellaOps.Excititor.Connectors.StellaOpsMirror/TASKS.md`. Focus on EXCITITOR-CONN-STELLA-07-003 (TODO). Confirm prerequisites (internal: EXCITITOR-CONN-STELLA-07-002 (Wave 4)) before starting and report status in module TASKS.md. -- Team Notify Connectors Guild: read EXECPLAN.md Wave 5 and SPRINTS.md rows for `src/StellaOps.Notify.Connectors.Email/TASKS.md`, `src/StellaOps.Notify.Connectors.Slack/TASKS.md`, `src/StellaOps.Notify.Connectors.Teams/TASKS.md`, `src/StellaOps.Notify.Connectors.Webhook/TASKS.md`. Focus on NOTIFY-CONN-SLACK-15-502 (TODO), NOTIFY-CONN-TEAMS-15-602 (TODO), NOTIFY-CONN-EMAIL-15-702 (TODO), NOTIFY-CONN-WEBHOOK-15-802 (TODO). Confirm prerequisites (internal: NOTIFY-CONN-EMAIL-15-701 (Wave 4), NOTIFY-CONN-SLACK-15-501 (Wave 4), NOTIFY-CONN-TEAMS-15-601 (Wave 4), NOTIFY-CONN-WEBHOOK-15-801 (Wave 4)) before starting and report status in module TASKS.md. +- Team Notify Connectors Guild: read EXECPLAN.md Wave 5 and SPRINTS.md rows for `src/StellaOps.Notify.Connectors.Email/TASKS.md`, `src/StellaOps.Notify.Connectors.Slack/TASKS.md`, `src/StellaOps.Notify.Connectors.Teams/TASKS.md`, `src/StellaOps.Notify.Connectors.Webhook/TASKS.md`. Focus on NOTIFY-CONN-SLACK-15-502 (DOING), NOTIFY-CONN-TEAMS-15-602 (DOING), NOTIFY-CONN-EMAIL-15-702 (DOING), NOTIFY-CONN-WEBHOOK-15-802 (DOING). Confirm prerequisites (internal: NOTIFY-CONN-EMAIL-15-701 (Wave 4), NOTIFY-CONN-SLACK-15-501 (Wave 4), NOTIFY-CONN-TEAMS-15-601 (Wave 4), NOTIFY-CONN-WEBHOOK-15-801 (Wave 4)) before starting and report status in module TASKS.md. - Team Scanner WebService Guild: read EXECPLAN.md Wave 5 and SPRINTS.md rows for `src/StellaOps.Scanner.WebService/TASKS.md`. Focus on SCANNER-RUNTIME-17-401 (TODO). Confirm prerequisites (internal: POLICY-RUNTIME-17-201 (Wave 4), SCANNER-EMIT-17-701 (Wave 1), SCANNER-RUNTIME-12-301 (Wave 1), ZASTAVA-OBS-17-005 (Wave 3)) before starting and report status in module TASKS.md. - Team TBD: read EXECPLAN.md Wave 5 and SPRINTS.md rows for `src/StellaOps.Scanner.Analyzers.Lang.DotNet/TASKS.md`, `src/StellaOps.Scanner.Analyzers.Lang.Go/TASKS.md`, `src/StellaOps.Scanner.Analyzers.Lang.Python/TASKS.md`, `src/StellaOps.Scanner.Analyzers.Lang.Rust/TASKS.md`. Focus on SCANNER-ANALYZERS-LANG-10-308D (TODO), SCANNER-ANALYZERS-LANG-10-308G (TODO), SCANNER-ANALYZERS-LANG-10-308P (TODO), SCANNER-ANALYZERS-LANG-10-308R (TODO). Confirm prerequisites (internal: SCANNER-ANALYZERS-LANG-10-307D (Wave 4), SCANNER-ANALYZERS-LANG-10-307G (Wave 4), SCANNER-ANALYZERS-LANG-10-307P (Wave 4), SCANNER-ANALYZERS-LANG-10-307R (Wave 4)) before starting and report status in module TASKS.md. @@ -130,7 +130,7 @@ Generated from SPRINTS.md and module TASKS.md files on 2025-10-19. Waves cluster - Team TBD: read EXECPLAN.md Wave 6 and SPRINTS.md rows for `src/StellaOps.Scanner.Analyzers.Lang.DotNet/TASKS.md`, `src/StellaOps.Scanner.Analyzers.Lang.Go/TASKS.md`, `src/StellaOps.Scanner.Analyzers.Lang.Python/TASKS.md`, `src/StellaOps.Scanner.Analyzers.Lang.Rust/TASKS.md`. Focus on SCANNER-ANALYZERS-LANG-10-309D (TODO), SCANNER-ANALYZERS-LANG-10-309G (TODO), SCANNER-ANALYZERS-LANG-10-309P (TODO), SCANNER-ANALYZERS-LANG-10-309R (TODO). Confirm prerequisites (internal: SCANNER-ANALYZERS-LANG-10-308D (Wave 5), SCANNER-ANALYZERS-LANG-10-308G (Wave 5), SCANNER-ANALYZERS-LANG-10-308P (Wave 5), SCANNER-ANALYZERS-LANG-10-308R (Wave 5)) before starting and report status in module TASKS.md. ### Wave 7 -- Team Team Core Engine & Storage Analytics: read EXECPLAN.md Wave 7 and SPRINTS.md rows for `src/StellaOps.Concelier.Core/TASKS.md`. Focus on FEEDCORE-ENGINE-07-001 (TODO). Confirm prerequisites (internal: FEEDSTORAGE-DATA-07-001 (Wave 10)) before starting and report status in module TASKS.md. +- Team Team Core Engine & Storage Analytics: read EXECPLAN.md Wave 7 and SPRINTS.md rows for `src/StellaOps.Concelier.Core/TASKS.md`. Focus on FEEDCORE-ENGINE-07-001 (DONE 2025-10-19). Confirm prerequisites (internal: FEEDSTORAGE-DATA-07-001 (Wave 10)) before starting and report status in module TASKS.md. ### Wave 8 - Team Team Core Engine & Data Science: read EXECPLAN.md Wave 8 and SPRINTS.md rows for `src/StellaOps.Concelier.Core/TASKS.md`. Focus on FEEDCORE-ENGINE-07-002 (TODO). Confirm prerequisites (internal: FEEDCORE-ENGINE-07-001 (Wave 7)) before starting and report status in module TASKS.md. @@ -148,7 +148,7 @@ Generated from SPRINTS.md and module TASKS.md files on 2025-10-19. Waves cluster - Team Concelier Export Guild: read EXECPLAN.md Wave 12 and SPRINTS.md rows for `src/StellaOps.Concelier.Exporter.Json/TASKS.md`. Focus on CONCELIER-EXPORT-08-201 (TODO). Confirm prerequisites (internal: FEEDCORE-ENGINE-07-001 (Wave 7)) before starting and report status in module TASKS.md. ### Wave 13 -- Team Concelier Export Guild: read EXECPLAN.md Wave 13 and SPRINTS.md rows for `src/StellaOps.Concelier.Exporter.TrivyDb/TASKS.md`. Focus on CONCELIER-EXPORT-08-202 (TODO). Confirm prerequisites (internal: CONCELIER-EXPORT-08-201 (Wave 12)) before starting and report status in module TASKS.md. +- Team Concelier Export Guild: read EXECPLAN.md Wave 13 and SPRINTS.md rows for `src/StellaOps.Concelier.Exporter.TrivyDb/TASKS.md`. Focus on CONCELIER-EXPORT-08-202 (DONE 2025-10-19). Confirm prerequisites (internal: CONCELIER-EXPORT-08-201 (Wave 12)) before starting and report status in module TASKS.md. ### Wave 14 - Team Concelier WebService Guild: read EXECPLAN.md Wave 14 and SPRINTS.md rows for `src/StellaOps.Concelier.WebService/TASKS.md`. Focus on CONCELIER-WEB-08-201 (TODO). Confirm prerequisites (internal: CONCELIER-EXPORT-08-201 (Wave 12), DEVOPS-MIRROR-08-001 (Wave 2)) before starting and report status in module TASKS.md. @@ -210,7 +210,7 @@ Generated from SPRINTS.md and module TASKS.md files on 2025-10-19. Waves cluster 1. [DOING] FEEDWEB-DOCS-01-001 — Document authority toggle & scope requirements — Quickstart updates are staged; awaiting Docs guild review before publishing operator guide refresh. • Prereqs: — • Current: DOING (2025-10-10) - 2. [TODO] FEEDWEB-OPS-01-006 — Rename plugin drop directory to namespaced path — Repoint build outputs to `StellaOps.Concelier.PluginBinaries`/`StellaOps.Authority.PluginBinaries`, update PluginHost defaults, Offline Kit packaging, and operator docs. + 2. [DONE] FEEDWEB-OPS-01-006 — Rename plugin drop directory to namespaced path — Build outputs now target `StellaOps.Concelier.PluginBinaries`/`StellaOps.Authority.PluginBinaries`, plugin host defaults updated, and docs/tests refreshed (see `dotnet test src/StellaOps.Concelier.WebService.Tests/StellaOps.Concelier.WebService.Tests.csproj --no-restore`). • Prereqs: — • Current: TODO 3. [BLOCKED] FEEDWEB-OPS-01-007 — Authority resilience adoption — Roll out retry/offline knobs to deployment docs and align CLI parity once LIB5 resilience options land; unblock when library release is available and docs review completes. @@ -319,25 +319,25 @@ Generated from SPRINTS.md and module TASKS.md files on 2025-10-19. Waves cluster - **Sprint 8** · Mongo strengthening - Team: Authority Core & Storage Guild - Path: `src/StellaOps.Authority/TASKS.md` - 1. [BLOCKED] AUTHSTORAGE-MONGO-08-001 — Harden Authority Mongo usage — Scoped sessions with causal consistency pending rate-limiter stream updates; resume once plugin lockout telemetry stabilises. + 1. [DONE] AUTHSTORAGE-MONGO-08-001 — Harden Authority Mongo usage — Scoped Mongo sessions with majority read/write concerns wired through stores and GraphQL/HTTP pipelines; replica-set election regression validated. • Prereqs: — • Current: BLOCKED (2025-10-19) - Team: Team Excititor Storage - Path: `src/StellaOps.Excititor.Storage.Mongo/TASKS.md` - 1. [TODO] EXCITITOR-STORAGE-MONGO-08-001 — EXCITITOR-STORAGE-MONGO-08-001 – Session + causal consistency hardening + 1. [DONE 2025-10-19] EXCITITOR-STORAGE-MONGO-08-001 — Session + causal consistency hardening shipped with scoped session provider, repository updates, and replica-set consistency tests (`dotnet test src/StellaOps.Excititor.Storage.Mongo.Tests/StellaOps.Excititor.Storage.Mongo.Tests.csproj`) • Prereqs: EXCITITOR-STORAGE-01-003 (external/completed) - • Current: TODO – Register Mongo client/database with majority read/write concerns, expose scoped session helper enabling causal consistency, thread session handles through raw/export/consensus/cache stores (including GridFS reads), and extend integration tests to verify read-your-write semantics during replica-set failover. + • Current: DONE – Scoped sessions with causal consistency in place; repositories/tests updated for deterministic read-your-write semantics. - Team: Team Normalization & Storage Backbone - Path: `src/StellaOps.Concelier.Storage.Mongo/TASKS.md` - 1. [TODO] FEEDSTORAGE-MONGO-08-001 — Causal-consistent Concelier storage sessions — Ensure `AddMongoStorage` registers a scoped session facilitator (causal consistency + majority concerns), update repositories to accept optional session handles, and add integration coverage proving read-your-write and monotonic reads across a replica set/election scenario. + 1. [DONE] FEEDSTORAGE-MONGO-08-001 — Causal-consistent Concelier storage sessions — Scoped session facilitator registered, repositories accept optional session handles, and replica-set failover tests verify read-your-write + monotonic reads. • Prereqs: — • Current: TODO - **Sprint 8** · Platform Maintenance - Team: Team Excititor Storage - Path: `src/StellaOps.Excititor.Storage.Mongo/TASKS.md` - 1. [TODO] EXCITITOR-STORAGE-03-001 — EXCITITOR-STORAGE-03-001 – Statement backfill tooling + 1. [DONE 2025-10-19] EXCITITOR-STORAGE-03-001 — Statement backfill tooling • Prereqs: EXCITITOR-STORAGE-02-001 (external/completed) - • Current: TODO – Provide CLI/scripted tooling to replay historical statements into `vex.statements` (leveraging `/excititor/statements`), document operational runbook, and add smoke test verifying replayed data includes severity/KEV/EPSS signals. + • Current: DONE – Admin backfill endpoint, CLI command (`stellaops excititor backfill-statements`), integration coverage, and operator runbook published; further automation tracked separately if needed. - Team: Team Excititor Worker - Path: `src/StellaOps.Excititor.Worker/TASKS.md` 1. [TODO] EXCITITOR-WORKER-02-001 — EXCITITOR-WORKER-02-001 – Resolve Microsoft.Extensions.Caching.Memory advisory @@ -351,7 +351,7 @@ Generated from SPRINTS.md and module TASKS.md files on 2025-10-19. Waves cluster • Current: TODO - Team: Plugin Platform Guild, Authority Core - Path: `src/StellaOps.Plugin/TASKS.md` - 1. [TODO] PLUGIN-DI-08-002 — Update Authority plugin integration — Flow scoped services through identity-provider registrars, bootstrap flows, and background jobs; add regression coverage around scoped lifetimes. + 1. [TODO] PLUGIN-DI-08-002 — Update Authority plugin integration — Flow scoped services through identity-provider registrars, bootstrap flows, and background jobs; add regression coverage around scoped lifetimes. (Coordination session set for 2025-10-20 15:00–16:00 UTC; document outcomes before implementation.) • Prereqs: — • Current: TODO - **Sprint 9** · Docs & Governance @@ -386,7 +386,7 @@ Generated from SPRINTS.md and module TASKS.md files on 2025-10-19. Waves cluster - **Sprint 10** · DevOps Perf - Team: DevOps Guild - Path: `ops/devops/TASKS.md` - 1. [TODO] DEVOPS-SEC-10-301 — Address NU1902/NU1903 advisories for `MongoDB.Driver` 2.12.0 and `SharpCompress` 0.23.0 surfaced during scanner cache and worker test runs. + 1. [DOING] DEVOPS-SEC-10-301 — Address NU1902/NU1903 advisories for `MongoDB.Driver` 2.12.0 and `SharpCompress` 0.23.0 surfaced during scanner cache and worker test runs (Wave 0A prerequisites cleared; remediation in progress). • Prereqs: — • Current: TODO - **Sprint 10** · Scanner Analyzers & SBOM @@ -473,12 +473,12 @@ Generated from SPRINTS.md and module TASKS.md files on 2025-10-19. Waves cluster • Current: TODO - Team: Authority Core & Security Guild - Path: `src/StellaOps.Authority/TASKS.md` - 1. [TODO] AUTH-DPOP-11-001 — Implement DPoP proof validation + nonce handling for high-value audiences per architecture. + 1. [DOING] AUTH-DPOP-11-001 — Implement DPoP proof validation + nonce handling for high-value audiences per architecture. • Prereqs: — - • Current: TODO - 2. [TODO] AUTH-MTLS-11-002 — Add OAuth mTLS client credential support with certificate-bound tokens and introspection updates. + • Current: DOING (2025-10-19) + 2. [DOING] AUTH-MTLS-11-002 — Add OAuth mTLS client credential support with certificate-bound tokens and introspection updates. • Prereqs: — - • Current: TODO + • Current: DOING (2025-10-19) - Team: Signer Guild - Path: `src/StellaOps.Signer/TASKS.md` 1. [TODO] SIGNER-API-11-101 — `/sign/dsse` pipeline with Authority auth, PoE introspection, release verification, DSSE signing. @@ -718,7 +718,7 @@ Generated from SPRINTS.md and module TASKS.md files on 2025-10-19. Waves cluster 5. [TODO] UI-SCANS-13-002 — Build scans module (list/detail/SBOM/diff/attestation) with performance + accessibility targets. • Prereqs: SCANNER-WEB-09-102 (external/completed), SIGNER-API-11-101 (Wave 0) • Current: TODO - 6. [TODO] UI-NOTIFY-13-006 — Notify panel: channels/rules CRUD, deliveries view, test send integration. + 6. [DOING] UI-NOTIFY-13-006 — Notify panel: channels/rules CRUD, deliveries view, test send integration. • Prereqs: NOTIFY-WEB-15-101 (Wave 0) • Current: TODO 7. [TODO] UI-SCHED-13-005 — Scheduler panel: schedules CRUD, run history, dry-run preview using API/mocks. @@ -748,7 +748,7 @@ Generated from SPRINTS.md and module TASKS.md files on 2025-10-19. Waves cluster • Current: TODO - Team: Notify WebService Guild - Path: `src/StellaOps.Notify.WebService/TASKS.md` - 1. [TODO] NOTIFY-WEB-15-103 — Delivery history + test-send endpoints with rate limits. + 1. [DONE] NOTIFY-WEB-15-103 — Delivery history + test-send endpoints with rate limits. • Prereqs: NOTIFY-WEB-15-102 (Wave 0) • Current: TODO - **Sprint 16** · Scheduler Intelligence @@ -809,9 +809,9 @@ Generated from SPRINTS.md and module TASKS.md files on 2025-10-19. Waves cluster - **Sprint 8** · Mirror Distribution - Team: DevOps Guild - Path: `ops/devops/TASKS.md` - 1. [TODO] DEVOPS-MIRROR-08-001 — Stand up managed mirror profiles for `*.stella-ops.org` (Concelier/Excititor), including Helm/Compose overlays, multi-tenant secrets, CDN caching, and sync documentation. + 1. [DONE] DEVOPS-MIRROR-08-001 — Stand up managed mirror profiles for `*.stella-ops.org` (Concelier/Excititor), including Helm/Compose overlays, multi-tenant secrets, CDN caching, and sync documentation. • Prereqs: DEVOPS-REL-14-001 (Wave 1) - • Current: TODO + • Current: DONE (2025-10-19) - **Sprint 9** · DevOps Foundations - Team: DevOps Guild, Notify Guild - Path: `ops/devops/TASKS.md` @@ -1121,19 +1121,19 @@ Generated from SPRINTS.md and module TASKS.md files on 2025-10-19. Waves cluster - **Sprint 15** · Notify Foundations - Team: Notify Connectors Guild - Path: `src/StellaOps.Notify.Connectors.Email/TASKS.md` - 1. [TODO] NOTIFY-CONN-EMAIL-15-702 — Add DKIM signing optional support and health/test-send flows. + 1. [DOING] NOTIFY-CONN-EMAIL-15-702 — Add DKIM signing optional support and health/test-send flows. • Prereqs: NOTIFY-CONN-EMAIL-15-701 (Wave 4) • Current: TODO - Path: `src/StellaOps.Notify.Connectors.Slack/TASKS.md` - 1. [TODO] NOTIFY-CONN-SLACK-15-502 — Health check & test-send support with minimal scopes and redacted tokens. + 1. [DOING] NOTIFY-CONN-SLACK-15-502 — Health check & test-send support with minimal scopes and redacted tokens. • Prereqs: NOTIFY-CONN-SLACK-15-501 (Wave 4) • Current: TODO - Path: `src/StellaOps.Notify.Connectors.Teams/TASKS.md` - 1. [TODO] NOTIFY-CONN-TEAMS-15-602 — Provide health/test-send support with fallback text for legacy clients. + 1. [DOING] NOTIFY-CONN-TEAMS-15-602 — Provide health/test-send support with fallback text for legacy clients. • Prereqs: NOTIFY-CONN-TEAMS-15-601 (Wave 4) • Current: TODO - Path: `src/StellaOps.Notify.Connectors.Webhook/TASKS.md` - 1. [TODO] NOTIFY-CONN-WEBHOOK-15-802 — Health/test-send support with signature validation hints and secret management. + 1. [DOING] NOTIFY-CONN-WEBHOOK-15-802 — Health/test-send support with signature validation hints and secret management. • Prereqs: NOTIFY-CONN-WEBHOOK-15-801 (Wave 4) • Current: TODO - **Sprint 17** · Symbol Intelligence & Forensics @@ -1185,9 +1185,9 @@ Generated from SPRINTS.md and module TASKS.md files on 2025-10-19. Waves cluster - **Sprint 7** · Contextual Truth Foundations - Team: Team Core Engine & Storage Analytics - Path: `src/StellaOps.Concelier.Core/TASKS.md` - 1. [TODO] FEEDCORE-ENGINE-07-001 — FEEDCORE-ENGINE-07-001 – Advisory event log & asOf queries + 1. [DONE] FEEDCORE-ENGINE-07-001 — FEEDCORE-ENGINE-07-001 – Advisory event log & asOf queries • Prereqs: FEEDSTORAGE-DATA-07-001 (Wave 10) - • Current: TODO – Introduce immutable advisory statement events, expose `asOf` query surface for merge/export pipelines, and document determinism guarantees for replay. + • Current: DONE (2025-10-19) – `AdvisoryEventLog` service and repository abstractions landed with canonical hashing, lower-cased keys, replay API, and doc updates. Tests: `dotnet test src/StellaOps.Concelier.Core.Tests/StellaOps.Concelier.Core.Tests.csproj`. ## Wave 8 — 1 task(s) ready after Wave 7 - **Sprint 7** · Contextual Truth Foundations @@ -1225,33 +1225,33 @@ Generated from SPRINTS.md and module TASKS.md files on 2025-10-19. Waves cluster - **Sprint 8** · Mirror Distribution - Team: Concelier Export Guild - Path: `src/StellaOps.Concelier.Exporter.Json/TASKS.md` - 1. [TODO] CONCELIER-EXPORT-08-201 — CONCELIER-EXPORT-08-201 – Mirror bundle + domain manifest + 1. [DONE] CONCELIER-EXPORT-08-201 — CONCELIER-EXPORT-08-201 – Mirror bundle + domain manifest • Prereqs: FEEDCORE-ENGINE-07-001 (Wave 7) - • Current: TODO – Produce per-domain aggregate bundles (JSON + manifest) with deterministic digests, include upstream source metadata, and publish index consumed by mirror endpoints/tests. + • Current: DONE (2025-10-19) – Mirror bundles + manifests + signed index shipped; regression coverage via `dotnet test src/StellaOps.Concelier.Exporter.Json.Tests/StellaOps.Concelier.Exporter.Json.Tests.csproj` (2025-10-19). ## Wave 13 — 1 task(s) ready after Wave 12 - **Sprint 8** · Mirror Distribution - Team: Concelier Export Guild - Path: `src/StellaOps.Concelier.Exporter.TrivyDb/TASKS.md` - 1. [TODO] CONCELIER-EXPORT-08-202 — CONCELIER-EXPORT-08-202 – Mirror-ready Trivy DB bundles + 1. [DONE] CONCELIER-EXPORT-08-202 — CONCELIER-EXPORT-08-202 – Mirror-ready Trivy DB bundles • Prereqs: CONCELIER-EXPORT-08-201 (Wave 12) - • Current: TODO – Generate domain-specific Trivy DB archives + metadata manifest, ensure deterministic digests, and document sync process for downstream Concelier nodes. + • Current: DONE (2025-10-19) – Trivy exporter mirror options produce `mirror/index.json` plus per-domain manifest/metadata/db files with reproducible SHA-256 digests; validated via `dotnet test src/StellaOps.Concelier.Exporter.TrivyDb.Tests/StellaOps.Concelier.Exporter.TrivyDb.Tests.csproj`. ## Wave 14 — 1 task(s) ready after Wave 13 - **Sprint 8** · Mirror Distribution - Team: Concelier WebService Guild - Path: `src/StellaOps.Concelier.WebService/TASKS.md` - 1. [TODO] CONCELIER-WEB-08-201 — CONCELIER-WEB-08-201 – Mirror distribution endpoints + 1. [DOING] CONCELIER-WEB-08-201 — CONCELIER-WEB-08-201 – Mirror distribution endpoints • Prereqs: CONCELIER-EXPORT-08-201 (Wave 12), DEVOPS-MIRROR-08-001 (Wave 2) - • Current: TODO – Add domain-scoped mirror configuration (`*.stella-ops.org`), expose signed export index/download APIs with quota and auth, and document sync workflow for downstream Concelier instances. + • Current: DOING (2025-10-19) – Wiring API surface against exporter-delivered `mirror/index.json` + signed bundles, layering quota/auth and updating docs/test fixtures for downstream sync. ## Wave 15 — 1 task(s) ready after Wave 14 - **Sprint 8** · Mirror Distribution - Team: BE-Conn-Stella - Path: `src/StellaOps.Concelier.Connector.StellaOpsMirror/TASKS.md` - 1. [TODO] FEEDCONN-STELLA-08-001 — Implement Concelier mirror fetcher hitting `https://.stella-ops.org/concelier/exports/index.json`, verify signatures/digests, and persist raw documents with provenance. + 1. [DOING] FEEDCONN-STELLA-08-001 — Implement Concelier mirror fetcher hitting `https://.stella-ops.org/concelier/exports/index.json`, verify signatures/digests, and persist raw documents with provenance. • Prereqs: CONCELIER-EXPORT-08-201 (Wave 12) - • Current: TODO + • Current: DOING (2025-10-19) – Client consuming new signed mirror bundles/index, standing up verification + storage plumbing ahead of DTO mapping. ## Wave 16 — 1 task(s) ready after Wave 15 - **Sprint 8** · Mirror Distribution diff --git a/Mongo2Go-4.1.0/src/Mongo2Go/Mongo2Go.csproj b/Mongo2Go-4.1.0/src/Mongo2Go/Mongo2Go.csproj index e4a19148..c83d13ce 100644 --- a/Mongo2Go-4.1.0/src/Mongo2Go/Mongo2Go.csproj +++ b/Mongo2Go-4.1.0/src/Mongo2Go/Mongo2Go.csproj @@ -76,6 +76,7 @@ Mongo2Go has two use cases: + diff --git a/Mongo2Go-4.1.0/src/Mongo2Go/packages.lock.json b/Mongo2Go-4.1.0/src/Mongo2Go/packages.lock.json index 668ae79d..4d612a99 100644 --- a/Mongo2Go-4.1.0/src/Mongo2Go/packages.lock.json +++ b/Mongo2Go-4.1.0/src/Mongo2Go/packages.lock.json @@ -39,13 +39,13 @@ }, "MongoDB.Driver": { "type": "Direct", - "requested": "[3.1.0, )", - "resolved": "3.1.0", - "contentHash": "+O7lKaIl7VUHptE0hqTd7UY1G5KDp/o8S4upG7YL4uChMNKD/U6tz9i17nMGHaD/L2AiPLgaJcaDe2XACsegGA==", + "requested": "[3.5.0, )", + "resolved": "3.5.0", + "contentHash": "ST90u7psyMkNNOWFgSkexsrB3kPn7Ynl2DlMFj2rJyYuc6SIxjmzu4ufy51yzM+cPVE1SvVcdb5UFobrRw6cMg==", "dependencies": { "DnsClient": "1.6.1", "Microsoft.Extensions.Logging.Abstractions": "2.0.0", - "MongoDB.Bson": "3.1.0", + "MongoDB.Bson": "3.5.0", "SharpCompress": "0.30.1", "Snappier": "1.0.0", "System.Buffers": "4.5.1", @@ -54,6 +54,19 @@ "ZstdSharp.Port": "0.7.3" } }, + "SharpCompress": { + "type": "Direct", + "requested": "[0.41.0, )", + "resolved": "0.41.0", + "contentHash": "z04dBVdTIAFTRKi38f0LkajaKA++bR+M8kYCbasXePILD2H+qs7CkLpyiippB24CSbTrWIgpBKm6BenZqkUwvw==", + "dependencies": { + "Microsoft.Bcl.AsyncInterfaces": "8.0.0", + "System.Buffers": "4.6.0", + "System.Memory": "4.6.0", + "System.Text.Encoding.CodePages": "8.0.0", + "ZstdSharp.Port": "0.8.6" + } + }, "System.Text.Json": { "type": "Direct", "requested": "[6.0.10, )", @@ -81,8 +94,8 @@ }, "Microsoft.Bcl.AsyncInterfaces": { "type": "Transitive", - "resolved": "6.0.0", - "contentHash": "UcSjPsst+DfAdJGVDsu346FX0ci0ah+lw3WRtn18NUwEqRt70HaOQ7lI72vy3+1LxtqI3T5GWwV39rQSrCzAeg==", + "resolved": "8.0.0", + "contentHash": "3WA9q9yVqJp222P3x1wYIGDAkpjAku0TMUaaQV22g6L67AI0LdOIrVS7Ht2vJfLHGSPVuqN94vIr15qn+HEkHw==", "dependencies": { "System.Threading.Tasks.Extensions": "4.5.4" } @@ -113,22 +126,13 @@ }, "MongoDB.Bson": { "type": "Transitive", - "resolved": "3.1.0", - "contentHash": "3dhaZhz18B5vUoEP13o2j8A6zQfkHdZhwBvLZEjDJum4BTLLv1/Z8bt25UQEtpqvYwLgde4R6ekWZ7XAYUMxuw==", + "resolved": "3.5.0", + "contentHash": "JGNK6BanLDEifgkvPLqVFCPus5EDCy416pxf1dxUBRSVd3D9+NB3AvMVX190eXlk5/UXuCxpsQv7jWfNKvppBQ==", "dependencies": { "System.Memory": "4.5.5", "System.Runtime.CompilerServices.Unsafe": "5.0.0" } }, - "SharpCompress": { - "type": "Transitive", - "resolved": "0.30.1", - "contentHash": "XqD4TpfyYGa7QTPzaGlMVbcecKnXy4YmYLDWrU+JIj7IuRNl7DH2END+Ll7ekWIY8o3dAMWLFDE1xdhfIWD1nw==", - "dependencies": { - "System.Memory": "4.5.4", - "System.Text.Encoding.CodePages": "5.0.0" - } - }, "Snappier": { "type": "Transitive", "resolved": "1.0.0", @@ -141,8 +145,8 @@ }, "System.Buffers": { "type": "Transitive", - "resolved": "4.5.1", - "contentHash": "Rw7ijyl1qqRS0YQD/WycNst8hUUMgrMH4FCn1nNm27M4VxchZ1js3fVjQaANHO5f3sN4isvP4a+Met9Y4YomAg==" + "resolved": "4.6.0", + "contentHash": "lN6tZi7Q46zFzAbRYXTIvfXcyvQQgxnY7Xm6C6xQ9784dEL1amjM6S6Iw4ZpsvesAKnRVsM4scrDQaDqSClkjA==" }, "System.IO": { "type": "Transitive", @@ -151,12 +155,12 @@ }, "System.Memory": { "type": "Transitive", - "resolved": "4.5.5", - "contentHash": "XIWiDvKPXaTveaB7HVganDlOCRoj03l+jrwNvcge/t8vhGYKvqV+dMv6G4SAX2NoNmN0wZfVPTAlFwZcZvVOUw==", + "resolved": "4.6.0", + "contentHash": "OEkbBQoklHngJ8UD8ez2AERSk2g+/qpAaSWWCBFbpH727HxDq5ydVkuncBaKcKfwRqXGWx64dS6G1SUScMsitg==", "dependencies": { - "System.Buffers": "4.5.1", - "System.Numerics.Vectors": "4.5.0", - "System.Runtime.CompilerServices.Unsafe": "4.5.3" + "System.Buffers": "4.6.0", + "System.Numerics.Vectors": "4.6.0", + "System.Runtime.CompilerServices.Unsafe": "6.1.0" } }, "System.Net.Http": { @@ -169,8 +173,8 @@ }, "System.Numerics.Vectors": { "type": "Transitive", - "resolved": "4.5.0", - "contentHash": "QQTlPTl06J/iiDbJCiepZ4H//BVraReU4O4EoRw1U02H5TLUIT7xn3GnDp9AXPSlJUDyFs4uWjWafNX6WrAojQ==" + "resolved": "4.6.0", + "contentHash": "t+SoieZsRuEyiw/J+qXUbolyO219tKQQI0+2/YI+Qv7YdGValA6WiuokrNKqjrTNsy5ABWU11bdKOzUdheteXg==" }, "System.Runtime": { "type": "Transitive", @@ -179,8 +183,8 @@ }, "System.Runtime.CompilerServices.Unsafe": { "type": "Transitive", - "resolved": "6.0.0", - "contentHash": "/iUeP3tq1S0XdNNoMz5C9twLSrM/TH+qElHkXWaPvuNOt+99G75NrV0OS2EqHx5wMN7popYjpc8oTjC1y16DLg==" + "resolved": "6.1.0", + "contentHash": "5o/HZxx6RVqYlhKSq8/zronDkALJZUT2Vz0hx43f0gwe8mwlM0y2nYlqdBwLMzr262Bwvpikeb/yEwkAa5PADg==" }, "System.Runtime.InteropServices.RuntimeInformation": { "type": "Transitive", @@ -232,10 +236,11 @@ }, "System.Text.Encoding.CodePages": { "type": "Transitive", - "resolved": "5.0.0", - "contentHash": "NyscU59xX6Uo91qvhOs2Ccho3AR2TnZPomo1Z0K6YpyztBPM/A5VbkzOO19sy3A3i1TtEnTxA7bCe3Us+r5MWg==", + "resolved": "8.0.0", + "contentHash": "OZIsVplFGaVY90G2SbpgU7EnCoOO5pw1t4ic21dBF3/1omrJFpAGoNAVpPyMVOC90/hvgkGG3VFqR13YgZMQfg==", "dependencies": { - "System.Runtime.CompilerServices.Unsafe": "5.0.0" + "System.Memory": "4.5.5", + "System.Runtime.CompilerServices.Unsafe": "6.0.0" } }, "System.Text.Encodings.Web": { @@ -263,11 +268,12 @@ }, "ZstdSharp.Port": { "type": "Transitive", - "resolved": "0.7.3", - "contentHash": "U9Ix4l4cl58Kzz1rJzj5hoVTjmbx1qGMwzAcbv1j/d3NzrFaESIurQyg+ow4mivCgkE3S413y+U9k4WdnEIkRA==", + "resolved": "0.8.6", + "contentHash": "iP4jVLQoQmUjMU88g1WObiNr6YKZGvh4aOXn3yOJsHqZsflwRsxZPcIBvNXgjXO3vQKSLctXGLTpcBPLnWPS8A==", "dependencies": { "Microsoft.Bcl.AsyncInterfaces": "5.0.0", - "System.Memory": "4.5.5" + "System.Memory": "4.5.5", + "System.Runtime.CompilerServices.Unsafe": "6.0.0" } } }, @@ -309,19 +315,32 @@ }, "MongoDB.Driver": { "type": "Direct", - "requested": "[3.1.0, )", - "resolved": "3.1.0", - "contentHash": "+O7lKaIl7VUHptE0hqTd7UY1G5KDp/o8S4upG7YL4uChMNKD/U6tz9i17nMGHaD/L2AiPLgaJcaDe2XACsegGA==", + "requested": "[3.5.0, )", + "resolved": "3.5.0", + "contentHash": "ST90u7psyMkNNOWFgSkexsrB3kPn7Ynl2DlMFj2rJyYuc6SIxjmzu4ufy51yzM+cPVE1SvVcdb5UFobrRw6cMg==", "dependencies": { "DnsClient": "1.6.1", "Microsoft.Extensions.Logging.Abstractions": "2.0.0", - "MongoDB.Bson": "3.1.0", + "MongoDB.Bson": "3.5.0", "SharpCompress": "0.30.1", "Snappier": "1.0.0", "System.Buffers": "4.5.1", "ZstdSharp.Port": "0.7.3" } }, + "SharpCompress": { + "type": "Direct", + "requested": "[0.41.0, )", + "resolved": "0.41.0", + "contentHash": "z04dBVdTIAFTRKi38f0LkajaKA++bR+M8kYCbasXePILD2H+qs7CkLpyiippB24CSbTrWIgpBKm6BenZqkUwvw==", + "dependencies": { + "Microsoft.Bcl.AsyncInterfaces": "8.0.0", + "System.Buffers": "4.6.0", + "System.Memory": "4.6.0", + "System.Text.Encoding.CodePages": "8.0.0", + "ZstdSharp.Port": "0.8.6" + } + }, "System.Text.Json": { "type": "Direct", "requested": "[6.0.10, )", @@ -347,8 +366,8 @@ }, "Microsoft.Bcl.AsyncInterfaces": { "type": "Transitive", - "resolved": "6.0.0", - "contentHash": "UcSjPsst+DfAdJGVDsu346FX0ci0ah+lw3WRtn18NUwEqRt70HaOQ7lI72vy3+1LxtqI3T5GWwV39rQSrCzAeg==" + "resolved": "8.0.0", + "contentHash": "3WA9q9yVqJp222P3x1wYIGDAkpjAku0TMUaaQV22g6L67AI0LdOIrVS7Ht2vJfLHGSPVuqN94vIr15qn+HEkHw==" }, "Microsoft.Build.Tasks.Git": { "type": "Transitive", @@ -378,21 +397,13 @@ }, "MongoDB.Bson": { "type": "Transitive", - "resolved": "3.1.0", - "contentHash": "3dhaZhz18B5vUoEP13o2j8A6zQfkHdZhwBvLZEjDJum4BTLLv1/Z8bt25UQEtpqvYwLgde4R6ekWZ7XAYUMxuw==", + "resolved": "3.5.0", + "contentHash": "JGNK6BanLDEifgkvPLqVFCPus5EDCy416pxf1dxUBRSVd3D9+NB3AvMVX190eXlk5/UXuCxpsQv7jWfNKvppBQ==", "dependencies": { "System.Memory": "4.5.5", "System.Runtime.CompilerServices.Unsafe": "5.0.0" } }, - "SharpCompress": { - "type": "Transitive", - "resolved": "0.30.1", - "contentHash": "XqD4TpfyYGa7QTPzaGlMVbcecKnXy4YmYLDWrU+JIj7IuRNl7DH2END+Ll7ekWIY8o3dAMWLFDE1xdhfIWD1nw==", - "dependencies": { - "System.Text.Encoding.CodePages": "5.0.0" - } - }, "Snappier": { "type": "Transitive", "resolved": "1.0.0", @@ -403,28 +414,28 @@ }, "System.Buffers": { "type": "Transitive", - "resolved": "4.5.1", - "contentHash": "Rw7ijyl1qqRS0YQD/WycNst8hUUMgrMH4FCn1nNm27M4VxchZ1js3fVjQaANHO5f3sN4isvP4a+Met9Y4YomAg==" + "resolved": "4.6.0", + "contentHash": "lN6tZi7Q46zFzAbRYXTIvfXcyvQQgxnY7Xm6C6xQ9784dEL1amjM6S6Iw4ZpsvesAKnRVsM4scrDQaDqSClkjA==" }, "System.Memory": { "type": "Transitive", - "resolved": "4.5.5", - "contentHash": "XIWiDvKPXaTveaB7HVganDlOCRoj03l+jrwNvcge/t8vhGYKvqV+dMv6G4SAX2NoNmN0wZfVPTAlFwZcZvVOUw==", + "resolved": "4.6.0", + "contentHash": "OEkbBQoklHngJ8UD8ez2AERSk2g+/qpAaSWWCBFbpH727HxDq5ydVkuncBaKcKfwRqXGWx64dS6G1SUScMsitg==", "dependencies": { - "System.Buffers": "4.5.1", - "System.Numerics.Vectors": "4.4.0", - "System.Runtime.CompilerServices.Unsafe": "4.5.3" + "System.Buffers": "4.6.0", + "System.Numerics.Vectors": "4.6.0", + "System.Runtime.CompilerServices.Unsafe": "6.1.0" } }, "System.Numerics.Vectors": { "type": "Transitive", - "resolved": "4.5.0", - "contentHash": "QQTlPTl06J/iiDbJCiepZ4H//BVraReU4O4EoRw1U02H5TLUIT7xn3GnDp9AXPSlJUDyFs4uWjWafNX6WrAojQ==" + "resolved": "4.6.0", + "contentHash": "t+SoieZsRuEyiw/J+qXUbolyO219tKQQI0+2/YI+Qv7YdGValA6WiuokrNKqjrTNsy5ABWU11bdKOzUdheteXg==" }, "System.Runtime.CompilerServices.Unsafe": { "type": "Transitive", - "resolved": "6.0.0", - "contentHash": "/iUeP3tq1S0XdNNoMz5C9twLSrM/TH+qElHkXWaPvuNOt+99G75NrV0OS2EqHx5wMN7popYjpc8oTjC1y16DLg==" + "resolved": "6.1.0", + "contentHash": "5o/HZxx6RVqYlhKSq8/zronDkALJZUT2Vz0hx43f0gwe8mwlM0y2nYlqdBwLMzr262Bwvpikeb/yEwkAa5PADg==" }, "System.Security.AccessControl": { "type": "Transitive", @@ -441,10 +452,11 @@ }, "System.Text.Encoding.CodePages": { "type": "Transitive", - "resolved": "5.0.0", - "contentHash": "NyscU59xX6Uo91qvhOs2Ccho3AR2TnZPomo1Z0K6YpyztBPM/A5VbkzOO19sy3A3i1TtEnTxA7bCe3Us+r5MWg==", + "resolved": "8.0.0", + "contentHash": "OZIsVplFGaVY90G2SbpgU7EnCoOO5pw1t4ic21dBF3/1omrJFpAGoNAVpPyMVOC90/hvgkGG3VFqR13YgZMQfg==", "dependencies": { - "System.Runtime.CompilerServices.Unsafe": "5.0.0" + "System.Memory": "4.5.5", + "System.Runtime.CompilerServices.Unsafe": "6.0.0" } }, "System.Text.Encodings.Web": { @@ -467,8 +479,8 @@ }, "ZstdSharp.Port": { "type": "Transitive", - "resolved": "0.7.3", - "contentHash": "U9Ix4l4cl58Kzz1rJzj5hoVTjmbx1qGMwzAcbv1j/d3NzrFaESIurQyg+ow4mivCgkE3S413y+U9k4WdnEIkRA==", + "resolved": "0.8.6", + "contentHash": "iP4jVLQoQmUjMU88g1WObiNr6YKZGvh4aOXn3yOJsHqZsflwRsxZPcIBvNXgjXO3vQKSLctXGLTpcBPLnWPS8A==", "dependencies": { "System.Runtime.CompilerServices.Unsafe": "6.0.0" } diff --git a/SPRINTS.md b/SPRINTS.md index 30e89075..2e91487e 100644 --- a/SPRINTS.md +++ b/SPRINTS.md @@ -1,343 +1,348 @@ -This file describe implementation of Stella Ops (docs/README.md). Implementation must respect rules from AGENTS.md (read if you have not). - -| Sprint | Theme | Tasks File Path | Status | Type of Specialist | Task ID | Task Description | -| --- | --- | --- | --- | --- | --- | --- | -| Sprint 1 | Stabilize In-Progress Foundations | src/StellaOps.Concelier.Models/TASKS.md | DONE (2025-10-12) | Team Models & Merge Leads | FEEDMODELS-SCHEMA-01-001 | SemVer primitive range-style metadata
Instructions to work:
DONE Read ./AGENTS.md and src/StellaOps.Concelier.Models/AGENTS.md. This task lays the groundwork—complete the SemVer helper updates before teammates pick up FEEDMODELS-SCHEMA-01-002/003 and FEEDMODELS-SCHEMA-02-900. Use ./src/FASTER_MODELING_AND_NORMALIZATION.md for the target rule structure. | -| Sprint 1 | Stabilize In-Progress Foundations | src/StellaOps.Concelier.Models/TASKS.md | DONE (2025-10-11) | Team Models & Merge Leads | FEEDMODELS-SCHEMA-01-002 | Provenance decision rationale field
Instructions to work:
AdvisoryProvenance now carries `decisionReason` and docs/tests were updated. Connectors and merge tasks should populate the field when applying precedence/freshness/tie-breaker logic; see src/StellaOps.Concelier.Models/PROVENANCE_GUIDELINES.md for usage guidance. | -| Sprint 1 | Stabilize In-Progress Foundations | src/StellaOps.Concelier.Models/TASKS.md | DONE (2025-10-11) | Team Models & Merge Leads | FEEDMODELS-SCHEMA-01-003 | Normalized version rules collection
Instructions to work:
`AffectedPackage.NormalizedVersions` and supporting comparer/docs/tests shipped. Connector owners must emit rule arrays per ./src/FASTER_MODELING_AND_NORMALIZATION.md and report progress via FEEDMERGE-COORD-02-900 so merge/storage backfills can proceed. | -| Sprint 1 | Stabilize In-Progress Foundations | src/StellaOps.Concelier.Models/TASKS.md | DONE (2025-10-12) | Team Models & Merge Leads | FEEDMODELS-SCHEMA-02-900 | Range primitives for SemVer/EVR/NEVRA metadata
Instructions to work:
DONE Read ./AGENTS.md and src/StellaOps.Concelier.Models/AGENTS.md before resuming this stalled effort. Confirm helpers align with the new `NormalizedVersions` representation so connectors finishing in Sprint 2 can emit consistent metadata. | -| Sprint 1 | Stabilize In-Progress Foundations | src/StellaOps.Concelier.Normalization/TASKS.md | DONE (2025-10-11) | Team Normalization & Storage Backbone | FEEDNORM-NORM-02-001 | SemVer normalized rule emitter
Shared `SemVerRangeRuleBuilder` now outputs primitives + normalized rules per `FASTER_MODELING_AND_NORMALIZATION.md`; CVE/GHSA connectors consuming the API have verified fixtures. | -| Sprint 1 | Stabilize In-Progress Foundations | src/StellaOps.Concelier.Storage.Mongo/TASKS.md | DONE (2025-10-11) | Team Normalization & Storage Backbone | FEEDSTORAGE-DATA-02-001 | Normalized range dual-write + backfill
AdvisoryStore dual-writes flattened `normalizedVersions` when `concelier.storage.enableSemVerStyle` is set; migration `20251011-semver-style-backfill` updates historical records and docs outline the rollout. | -| Sprint 1 | Stabilize In-Progress Foundations | src/StellaOps.Concelier.Storage.Mongo/TASKS.md | DONE (2025-10-11) | Team Normalization & Storage Backbone | FEEDSTORAGE-DATA-02-002 | Provenance decision reason persistence
Storage now persists `provenance.decisionReason` for advisories and merge events; tests cover round-trips. | -| Sprint 1 | Stabilize In-Progress Foundations | src/StellaOps.Concelier.Storage.Mongo/TASKS.md | DONE (2025-10-11) | Team Normalization & Storage Backbone | FEEDSTORAGE-DATA-02-003 | Normalized versions indexing
Bootstrapper seeds compound/sparse indexes for flattened normalized rules and `docs/dev/mongo_indices.md` documents query guidance. | -| Sprint 1 | Stabilize In-Progress Foundations | src/StellaOps.Concelier.Storage.Mongo/TASKS.md | DONE (2025-10-11) | Team Normalization & Storage Backbone | FEEDSTORAGE-TESTS-02-004 | Restore AdvisoryStore build after normalized versions refactor
Updated constructors/tests keep storage suites passing with the new feature flag defaults. | -| Sprint 1 | Stabilize In-Progress Foundations | src/StellaOps.Concelier.WebService/TASKS.md | DONE (2025-10-12) | Team WebService & Authority | FEEDWEB-ENGINE-01-002 | Plumb Authority client resilience options
WebService wires `authority.resilience.*` into `AddStellaOpsAuthClient` and adds binding coverage via `AuthorityClientResilienceOptionsAreBound`. | -| Sprint 1 | Stabilize In-Progress Foundations | src/StellaOps.Concelier.WebService/TASKS.md | DONE (2025-10-12) | Team WebService & Authority | FEEDWEB-DOCS-01-003 | Author ops guidance for resilience tuning
Install/runbooks document connected vs air-gapped resilience profiles and monitoring hooks. | -| Sprint 1 | Stabilize In-Progress Foundations | src/StellaOps.Concelier.WebService/TASKS.md | DONE (2025-10-12) | Team WebService & Authority | FEEDWEB-DOCS-01-004 | Document authority bypass logging patterns
Operator guides now call out `route/status/subject/clientId/scopes/bypass/remote` audit fields and SIEM triggers. | -| Sprint 1 | Stabilize In-Progress Foundations | src/StellaOps.Concelier.WebService/TASKS.md | DONE (2025-10-12) | Team WebService & Authority | FEEDWEB-DOCS-01-005 | Update Concelier operator guide for enforcement cutoff
Install guide reiterates the 2025-12-31 cutoff and links audit signals to the rollout checklist. | -| Sprint 1 | Stabilize In-Progress Foundations | src/StellaOps.Authority/TASKS.md | DONE (2025-10-11) | Team WebService & Authority | SEC3.HOST | Rate limiter policy binding
Authority host now applies configuration-driven fixed windows to `/token`, `/authorize`, and `/internal/*`; integration tests assert 429 + `Retry-After` headers; docs/config samples refreshed for Docs guild diagrams. | -| Sprint 1 | Stabilize In-Progress Foundations | src/StellaOps.Authority/TASKS.md | DONE (2025-10-11) | Team WebService & Authority | SEC3.BUILD | Authority rate-limiter follow-through
`Security.RateLimiting` now fronts token/authorize/internal limiters; Authority + Configuration matrices (`dotnet test src/StellaOps.Authority/StellaOps.Authority.sln`, `dotnet test src/StellaOps.Configuration.Tests/StellaOps.Configuration.Tests.csproj`) passed on 2025-10-11; awaiting #authority-core broadcast. | -| Sprint 1 | Stabilize In-Progress Foundations | src/StellaOps.Authority/TASKS.md | DONE (2025-10-14) | Team Authority Platform & Security Guild | AUTHCORE-BUILD-OPENIDDICT / AUTHCORE-STORAGE-DEVICE-TOKENS / AUTHCORE-BOOTSTRAP-INVITES | Address remaining Authority compile blockers (OpenIddict transaction shim, token device document, bootstrap invite cleanup) so `dotnet build src/StellaOps.Authority.sln` returns success. | -| Sprint 1 | Stabilize In-Progress Foundations | src/StellaOps.Authority/StellaOps.Authority.Plugin.Standard/TASKS.md | DONE (2025-10-11) | Team WebService & Authority | PLG6.DOC | Plugin developer guide polish
Section 9 now documents rate limiter metadata, config keys, and lockout interplay; YAML samples updated alongside Authority config templates. | -| Sprint 1 | Stabilize In-Progress Foundations | src/StellaOps.Concelier.Connector.CertCc/TASKS.md | DONE (2025-10-11) | Team Connector Resumption – CERT/RedHat | FEEDCONN-CERTCC-02-001 | Fetch pipeline & state tracking
Summary planner now drives monthly/yearly VINCE fetches, persists pending summaries/notes, and hydrates VINCE detail queue with telemetry.
Team instructions: Read ./AGENTS.md and src/StellaOps.Concelier.Connector.CertCc/AGENTS.md. Coordinate daily with Models/Merge leads so new normalizedVersions output and provenance tags stay aligned with ./src/FASTER_MODELING_AND_NORMALIZATION.md. | -| Sprint 1 | Stabilize In-Progress Foundations | src/StellaOps.Concelier.Connector.CertCc/TASKS.md | DONE (2025-10-11) | Team Connector Resumption – CERT/RedHat | FEEDCONN-CERTCC-02-002 | VINCE note detail fetcher
Summary planner queues VINCE note detail endpoints, persists raw JSON with SHA/ETag metadata, and records retry/backoff metrics. | -| Sprint 1 | Stabilize In-Progress Foundations | src/StellaOps.Concelier.Connector.CertCc/TASKS.md | DONE (2025-10-11) | Team Connector Resumption – CERT/RedHat | FEEDCONN-CERTCC-02-003 | DTO & parser implementation
Added VINCE DTO aggregate, Markdown→text sanitizer, vendor/status/vulnerability parsers, and parser regression fixture. | -| Sprint 1 | Stabilize In-Progress Foundations | src/StellaOps.Concelier.Connector.CertCc/TASKS.md | DONE (2025-10-11) | Team Connector Resumption – CERT/RedHat | FEEDCONN-CERTCC-02-004 | Canonical mapping & range primitives
VINCE DTO aggregate flows through `CertCcMapper`, emitting vendor range primitives + normalized version rules that persist via `_advisoryStore`. | -| Sprint 1 | Stabilize In-Progress Foundations | src/StellaOps.Concelier.Connector.CertCc/TASKS.md | DONE (2025-10-12) | Team Connector Resumption – CERT/RedHat | FEEDCONN-CERTCC-02-005 | Deterministic fixtures/tests
Snapshot harness refreshed 2025-10-12; `certcc-*.snapshot.json` regenerated and regression suite green without UPDATE flag drift. | -| Sprint 1 | Stabilize In-Progress Foundations | src/StellaOps.Concelier.Connector.CertCc/TASKS.md | DONE (2025-10-12) | Team Connector Resumption – CERT/RedHat | FEEDCONN-CERTCC-02-006 | Telemetry & documentation
`CertCcDiagnostics` publishes summary/detail/parse/map metrics (meter `StellaOps.Concelier.Connector.CertCc`), README documents instruments, and log guidance captured for Ops on 2025-10-12. | -| Sprint 1 | Stabilize In-Progress Foundations | src/StellaOps.Concelier.Connector.CertCc/TASKS.md | DONE (2025-10-12) | Team Connector Resumption – CERT/RedHat | FEEDCONN-CERTCC-02-007 | Connector test harness remediation
Harness now wires `AddSourceCommon`, resets `FakeTimeProvider`, and passes canned-response regression run dated 2025-10-12. | -| Sprint 1 | Stabilize In-Progress Foundations | src/StellaOps.Concelier.Connector.CertCc/TASKS.md | DONE (2025-10-11) | Team Connector Resumption – CERT/RedHat | FEEDCONN-CERTCC-02-008 | Snapshot coverage handoff
Fixtures regenerated with normalized ranges + provenance fields on 2025-10-11; QA handoff notes published and merge backfill unblocked. | -| Sprint 1 | Stabilize In-Progress Foundations | src/StellaOps.Concelier.Connector.CertCc/TASKS.md | DONE (2025-10-12) | Team Connector Resumption – CERT/RedHat | FEEDCONN-CERTCC-02-012 | Schema sync & snapshot regen follow-up
Fixtures regenerated with normalizedVersions + provenance decision reasons; handoff notes updated for Merge backfill 2025-10-12. | -| Sprint 1 | Stabilize In-Progress Foundations | src/StellaOps.Concelier.Connector.CertCc/TASKS.md | DONE (2025-10-11) | Team Connector Resumption – CERT/RedHat | FEEDCONN-CERTCC-02-009 | Detail/map reintegration plan
Staged reintegration plan published in `src/StellaOps.Concelier.Connector.CertCc/FEEDCONN-CERTCC-02-009_PLAN.md`; coordinates enablement with FEEDCONN-CERTCC-02-004. | -| Sprint 1 | Stabilize In-Progress Foundations | src/StellaOps.Concelier.Connector.CertCc/TASKS.md | DONE (2025-10-12) | Team Connector Resumption – CERT/RedHat | FEEDCONN-CERTCC-02-010 | Partial-detail graceful degradation
Detail fetch now tolerates 404/403/410 responses and regression tests cover mixed endpoint availability. | -| Sprint 1 | Stabilize In-Progress Foundations | src/StellaOps.Concelier.Connector.Distro.RedHat/TASKS.md | DONE (2025-10-11) | Team Connector Resumption – CERT/RedHat | FEEDCONN-REDHAT-02-001 | Fixture validation sweep
Instructions to work:
Fixtures regenerated post-model-helper rollout; provenance ordering and normalizedVersions scaffolding verified via tests. Conflict resolver deltas logged in src/StellaOps.Concelier.Connector.Distro.RedHat/CONFLICT_RESOLVER_NOTES.md for Sprint 3 consumers. | -| Sprint 1 | Stabilize In-Progress Foundations | src/StellaOps.Concelier.Connector.Vndr.Apple/TASKS.md | DONE (2025-10-12) | Team Vendor Apple Specialists | FEEDCONN-APPLE-02-001 | Canonical mapping & range primitives
Mapper emits SemVer rules (`scheme=apple:*`); fixtures regenerated with trimmed references + new RSR coverage, update tooling finalized. | -| Sprint 1 | Stabilize In-Progress Foundations | src/StellaOps.Concelier.Connector.Vndr.Apple/TASKS.md | DONE (2025-10-11) | Team Vendor Apple Specialists | FEEDCONN-APPLE-02-002 | Deterministic fixtures/tests
Sanitized live fixtures + regression snapshots wired into tests; normalized rule coverage asserted. | -| Sprint 1 | Stabilize In-Progress Foundations | src/StellaOps.Concelier.Connector.Vndr.Apple/TASKS.md | DONE (2025-10-11) | Team Vendor Apple Specialists | FEEDCONN-APPLE-02-003 | Telemetry & documentation
Apple meter metrics wired into Concelier WebService OpenTelemetry configuration; README and fixtures document normalizedVersions coverage. | -| Sprint 1 | Stabilize In-Progress Foundations | src/StellaOps.Concelier.Connector.Vndr.Apple/TASKS.md | DONE (2025-10-12) | Team Vendor Apple Specialists | FEEDCONN-APPLE-02-004 | Live HTML regression sweep
Sanitised HT125326/HT125328/HT106355/HT214108/HT215500 fixtures recorded and regression tests green on 2025-10-12. | -| Sprint 1 | Stabilize In-Progress Foundations | src/StellaOps.Concelier.Connector.Vndr.Apple/TASKS.md | DONE (2025-10-11) | Team Vendor Apple Specialists | FEEDCONN-APPLE-02-005 | Fixture regeneration tooling
`UPDATE_APPLE_FIXTURES=1` flow fetches & rewrites fixtures; README documents usage.
Instructions to work:
DONE Read ./AGENTS.md and src/StellaOps.Concelier.Connector.Vndr.Apple/AGENTS.md. Resume stalled tasks, ensuring normalizedVersions output and fixtures align with ./src/FASTER_MODELING_AND_NORMALIZATION.md before handing data to the conflict sprint. | -| Sprint 1 | Stabilize In-Progress Foundations | src/StellaOps.Concelier.Connector.Ghsa/TASKS.md | DONE (2025-10-12) | Team Connector Normalized Versions Rollout | FEEDCONN-GHSA-02-001 | GHSA normalized versions & provenance
Team instructions: Read ./AGENTS.md and each module's AGENTS file. Adopt the `NormalizedVersions` array emitted by the models sprint, wiring provenance `decisionReason` where merge overrides occur. Follow ./src/FASTER_MODELING_AND_NORMALIZATION.md; report via src/StellaOps.Concelier.Merge/TASKS.md (FEEDMERGE-COORD-02-900). Progress 2025-10-11: GHSA/OSV emit normalized arrays with refreshed fixtures; CVE mapper now surfaces SemVer normalized ranges; NVD/KEV adoption pending; outstanding follow-ups include FEEDSTORAGE-DATA-02-001, FEEDMERGE-ENGINE-02-002, and rolling `tools/FixtureUpdater` updates across connectors. | -| Sprint 1 | Stabilize In-Progress Foundations | src/StellaOps.Concelier.Connector.Osv/TASKS.md | DONE (2025-10-12) | Team Connector Normalized Versions Rollout | FEEDCONN-OSV-02-003 | OSV normalized versions & freshness | -| Sprint 1 | Stabilize In-Progress Foundations | src/StellaOps.Concelier.Connector.Nvd/TASKS.md | DONE (2025-10-12) | Team Connector Normalized Versions Rollout | FEEDCONN-NVD-02-002 | NVD normalized versions & timestamps | -| Sprint 1 | Stabilize In-Progress Foundations | src/StellaOps.Concelier.Connector.Cve/TASKS.md | DONE (2025-10-12) | Team Connector Normalized Versions Rollout | FEEDCONN-CVE-02-003 | CVE normalized versions uplift | -| Sprint 1 | Stabilize In-Progress Foundations | src/StellaOps.Concelier.Connector.Kev/TASKS.md | DONE (2025-10-12) | Team Connector Normalized Versions Rollout | FEEDCONN-KEV-02-003 | KEV normalized versions propagation | -| Sprint 1 | Stabilize In-Progress Foundations | src/StellaOps.Concelier.Connector.Osv/TASKS.md | DONE (2025-10-12) | Team Connector Normalized Versions Rollout | FEEDCONN-OSV-04-003 | OSV parity fixture refresh | -| Sprint 1 | Stabilize In-Progress Foundations | src/StellaOps.Concelier.WebService/TASKS.md | DONE (2025-10-10) | Team WebService & Authority | FEEDWEB-DOCS-01-001 | Document authority toggle & scope requirements
Quickstart carries toggle/scope guidance pending docs guild review (no change this sprint). | -| Sprint 1 | Stabilize In-Progress Foundations | src/StellaOps.Concelier.WebService/TASKS.md | DONE (2025-10-12) | Team WebService & Authority | FEEDWEB-ENGINE-01-002 | Plumb Authority client resilience options
WebService wires `authority.resilience.*` into `AddStellaOpsAuthClient` and adds binding coverage via `AuthorityClientResilienceOptionsAreBound`. | -| Sprint 1 | Stabilize In-Progress Foundations | src/StellaOps.Concelier.WebService/TASKS.md | DONE (2025-10-12) | Team WebService & Authority | FEEDWEB-DOCS-01-003 | Author ops guidance for resilience tuning
Operator docs now outline connected vs air-gapped resilience profiles and monitoring cues. | -| Sprint 1 | Stabilize In-Progress Foundations | src/StellaOps.Concelier.WebService/TASKS.md | DONE (2025-10-12) | Team WebService & Authority | FEEDWEB-DOCS-01-004 | Document authority bypass logging patterns
Audit logging guidance highlights `route/status/subject/clientId/scopes/bypass/remote` fields and SIEM alerts. | -| Sprint 1 | Stabilize In-Progress Foundations | src/StellaOps.Concelier.WebService/TASKS.md | DONE (2025-10-12) | Team WebService & Authority | FEEDWEB-DOCS-01-005 | Update Concelier operator guide for enforcement cutoff
Install guide reiterates the 2025-12-31 cutoff and ties audit signals to rollout checks. | -| Sprint 1 | Stabilize In-Progress Foundations | src/StellaOps.Concelier.WebService/TASKS.md | DONE (2025-10-11) | Team WebService & Authority | FEEDWEB-OPS-01-006 | Rename plugin drop directory to namespaced path
Build outputs, tests, and docs now target `StellaOps.Concelier.PluginBinaries`/`StellaOps.Authority.PluginBinaries`. | -| Sprint 1 | Stabilize In-Progress Foundations | src/StellaOps.Concelier.WebService/TASKS.md | DONE (2025-10-11) | Team WebService & Authority | FEEDWEB-OPS-01-007 | Authority resilience adoption
Deployment docs and CLI notes explain the LIB5 resilience knobs for rollout.
Instructions to work:
DONE Read ./AGENTS.md and src/StellaOps.Concelier.WebService/AGENTS.md. These items were mid-flight; resume implementation ensuring docs/operators receive timely updates. | -| Sprint 1 | Stabilize In-Progress Foundations | src/StellaOps.Authority/TASKS.md | DONE (2025-10-11) | Team Authority Platform & Security Guild | AUTHCORE-ENGINE-01-001 | CORE8.RL — Rate limiter plumbing validated; integration tests green and docs handoff recorded for middleware ordering + Retry-After headers (see `docs/dev/authority-rate-limit-tuning-outline.md` for continuing guidance). | -| Sprint 1 | Stabilize In-Progress Foundations | src/StellaOps.Cryptography/TASKS.md | DONE (2025-10-11) | Team Authority Platform & Security Guild | AUTHCRYPTO-ENGINE-01-001 | SEC3.A — Shared metadata resolver confirmed via host test run; SEC3.B now unblocked for tuning guidance (outline captured in `docs/dev/authority-rate-limit-tuning-outline.md`). | -| Sprint 1 | Stabilize In-Progress Foundations | src/StellaOps.Cryptography/TASKS.md | DONE (2025-10-13) | Team Authority Platform & Security Guild | AUTHSEC-DOCS-01-002 | SEC3.B — Published `docs/security/rate-limits.md` with tuning matrix, alert thresholds, and lockout interplay guidance; Docs guild can lift copy into plugin guide. | -| Sprint 1 | Stabilize In-Progress Foundations | src/StellaOps.Cryptography/TASKS.md | DONE (2025-10-14) | Team Authority Platform & Security Guild | AUTHSEC-CRYPTO-02-001 | SEC5.B1 — Introduce libsodium signing provider and parity tests to unblock CLI verification enhancements. | -| Sprint 1 | Bootstrap & Replay Hardening | src/StellaOps.Cryptography/TASKS.md | DONE (2025-10-14) | Security Guild | AUTHSEC-CRYPTO-02-004 | SEC5.D/E — Finish bootstrap invite lifecycle (API/store/cleanup) and token device heuristics; build currently red due to pending handler integration. | -| Sprint 1 | Developer Tooling | src/StellaOps.Cli/TASKS.md | DONE (2025-10-15) | DevEx/CLI | AUTHCLI-DIAG-01-001 | Surface password policy diagnostics in CLI startup/output so operators see weakened overrides immediately.
CLI now loads Authority plug-ins at startup, logs weakened password policies (length/complexity), and regression coverage lives in `StellaOps.Cli.Tests/Services/AuthorityDiagnosticsReporterTests`. | -| Sprint 1 | Stabilize In-Progress Foundations | src/StellaOps.Authority/StellaOps.Authority.Plugin.Standard/TASKS.md | DONE (2025-10-11) | Team Authority Platform & Security Guild | AUTHPLUG-DOCS-01-001 | PLG6.DOC — Developer guide copy + diagrams merged 2025-10-11; limiter guidance incorporated and handed to Docs guild for asset export. | -| Sprint 2 | Connector & Data Implementation Wave | src/StellaOps.Concelier.Normalization/TASKS.md | DONE (2025-10-12) | Team Normalization & Storage Backbone | FEEDNORM-NORM-02-001 | SemVer normalized rule emitter
`SemVerRangeRuleBuilder` shipped 2025-10-12 with comparator/`||` support and fixtures aligning to `FASTER_MODELING_AND_NORMALIZATION.md`. | -| Sprint 2 | Connector & Data Implementation Wave | src/StellaOps.Concelier.Storage.Mongo/TASKS.md | DONE (2025-10-11) | Team Normalization & Storage Backbone | FEEDSTORAGE-DATA-02-001 | Normalized range dual-write + backfill | -| Sprint 2 | Connector & Data Implementation Wave | src/StellaOps.Concelier.Storage.Mongo/TASKS.md | DONE (2025-10-11) | Team Normalization & Storage Backbone | FEEDSTORAGE-DATA-02-002 | Provenance decision reason persistence | -| Sprint 2 | Connector & Data Implementation Wave | src/StellaOps.Concelier.Storage.Mongo/TASKS.md | DONE (2025-10-11) | Team Normalization & Storage Backbone | FEEDSTORAGE-DATA-02-003 | Normalized versions indexing
Indexes seeded + docs updated 2025-10-11 to cover flattened normalized rules for connector adoption. | -| Sprint 2 | Connector & Data Implementation Wave | src/StellaOps.Concelier.Merge/TASKS.md | DONE (2025-10-11) | Team Normalization & Storage Backbone | FEEDMERGE-ENGINE-02-002 | Normalized versions union & dedupe
Affected package resolver unions/dedupes normalized rules, stamps merge provenance with `decisionReason`, and tests cover the rollout. | -| Sprint 2 | Connector & Data Implementation Wave | src/StellaOps.Concelier.Connector.Ghsa/TASKS.md | DONE (2025-10-11) | Team Connector Expansion – GHSA/NVD/OSV | FEEDCONN-GHSA-02-001 | GHSA normalized versions & provenance | -| Sprint 2 | Connector & Data Implementation Wave | src/StellaOps.Concelier.Connector.Ghsa/TASKS.md | DONE (2025-10-11) | Team Connector Expansion – GHSA/NVD/OSV | FEEDCONN-GHSA-02-004 | GHSA credits & ecosystem severity mapping | -| Sprint 2 | Connector & Data Implementation Wave | src/StellaOps.Concelier.Connector.Ghsa/TASKS.md | DONE (2025-10-12) | Team Connector Expansion – GHSA/NVD/OSV | FEEDCONN-GHSA-02-005 | GitHub quota monitoring & retries | -| Sprint 2 | Connector & Data Implementation Wave | src/StellaOps.Concelier.Connector.Ghsa/TASKS.md | DONE (2025-10-12) | Team Connector Expansion – GHSA/NVD/OSV | FEEDCONN-GHSA-02-006 | Production credential & scheduler rollout | -| Sprint 2 | Connector & Data Implementation Wave | src/StellaOps.Concelier.Connector.Ghsa/TASKS.md | DONE (2025-10-12) | Team Connector Expansion – GHSA/NVD/OSV | FEEDCONN-GHSA-02-007 | Credit parity regression fixtures | -| Sprint 2 | Connector & Data Implementation Wave | src/StellaOps.Concelier.Connector.Nvd/TASKS.md | DONE (2025-10-11) | Team Connector Expansion – GHSA/NVD/OSV | FEEDCONN-NVD-02-002 | NVD normalized versions & timestamps | -| Sprint 2 | Connector & Data Implementation Wave | src/StellaOps.Concelier.Connector.Nvd/TASKS.md | DONE (2025-10-11) | Team Connector Expansion – GHSA/NVD/OSV | FEEDCONN-NVD-02-004 | NVD CVSS & CWE precedence payloads | -| Sprint 2 | Connector & Data Implementation Wave | src/StellaOps.Concelier.Connector.Nvd/TASKS.md | DONE (2025-10-12) | Team Connector Expansion – GHSA/NVD/OSV | FEEDCONN-NVD-02-005 | NVD merge/export parity regression | -| Sprint 2 | Connector & Data Implementation Wave | src/StellaOps.Concelier.Connector.Osv/TASKS.md | DONE (2025-10-11) | Team Connector Expansion – GHSA/NVD/OSV | FEEDCONN-OSV-02-003 | OSV normalized versions & freshness | -| Sprint 2 | Connector & Data Implementation Wave | src/StellaOps.Concelier.Connector.Osv/TASKS.md | DONE (2025-10-11) | Team Connector Expansion – GHSA/NVD/OSV | FEEDCONN-OSV-02-004 | OSV references & credits alignment | -| Sprint 2 | Connector & Data Implementation Wave | src/StellaOps.Concelier.Connector.Osv/TASKS.md | DONE (2025-10-12) | Team Connector Expansion – GHSA/NVD/OSV | FEEDCONN-OSV-02-005 | Fixture updater workflow
Resolved 2025-10-12: OSV mapper now derives canonical PURLs for Go + scoped npm packages when raw payloads omit `purl`; conflict fixtures unchanged for invalid npm names. Verified via `dotnet test src/StellaOps.Concelier.Connector.Osv.Tests`, `src/StellaOps.Concelier.Connector.Ghsa.Tests`, `src/StellaOps.Concelier.Connector.Nvd.Tests`, and backbone normalization/storage suites. | -| Sprint 2 | Connector & Data Implementation Wave | src/StellaOps.Concelier.Connector.Acsc/TASKS.md | DONE (2025-10-12) | Team Connector Expansion – Regional & Vendor Feeds | FEEDCONN-ACSC-02-001 … 02-008 | Fetch→parse→map pipeline, fixtures, diagnostics, and README finished 2025-10-12; downstream export parity captured via FEEDEXPORT-JSON-04-001 / FEEDEXPORT-TRIVY-04-001 (completed). | -| Sprint 2 | Connector & Data Implementation Wave | src/StellaOps.Concelier.Connector.Cccs/TASKS.md | DONE (2025-10-16) | Team Connector Expansion – Regional & Vendor Feeds | FEEDCONN-CCCS-02-001 … 02-008 | Observability meter, historical harvest plan, and DOM sanitizer refinements wrapped; ops notes live under `docs/ops/concelier-cccs-operations.md` with fixtures validating EN/FR list handling. | -| Sprint 2 | Connector & Data Implementation Wave | src/StellaOps.Concelier.Connector.CertBund/TASKS.md | DONE (2025-10-15) | Team Connector Expansion – Regional & Vendor Feeds | FEEDCONN-CERTBUND-02-001 … 02-008 | Telemetry/docs (02-006) and history/locale sweep (02-007) completed alongside pipeline; runbook `docs/ops/concelier-certbund-operations.md` captures locale guidance and offline packaging. | -| Sprint 2 | Connector & Data Implementation Wave | src/StellaOps.Concelier.Connector.Kisa/TASKS.md | DONE (2025-10-14) | Team Connector Expansion – Regional & Vendor Feeds | FEEDCONN-KISA-02-001 … 02-007 | Connector, tests, and telemetry/docs (02-006) finalized; localisation notes in `docs/dev/kisa_connector_notes.md` complete rollout. | -| Sprint 2 | Connector & Data Implementation Wave | src/StellaOps.Concelier.Connector.Ru.Bdu/TASKS.md | DONE (2025-10-14) | Team Connector Expansion – Regional & Vendor Feeds | FEEDCONN-RUBDU-02-001 … 02-008 | Fetch/parser/mapper refinements, regression fixtures, telemetry/docs, access options, and trusted root packaging all landed; README documents offline access strategy. | -| Sprint 2 | Connector & Data Implementation Wave | src/StellaOps.Concelier.Connector.Ru.Nkcki/TASKS.md | DONE (2025-10-13) | Team Connector Expansion – Regional & Vendor Feeds | FEEDCONN-NKCKI-02-001 … 02-008 | Listing fetch, parser, mapper, fixtures, telemetry/docs, and archive plan finished; Mongo2Go/libcrypto dependency resolved via bundled OpenSSL noted in ops guide. | -| Sprint 2 | Connector & Data Implementation Wave | src/StellaOps.Concelier.Connector.Ics.Cisa/TASKS.md | DONE (2025-10-16) | Team Connector Expansion – Regional & Vendor Feeds | FEEDCONN-ICSCISA-02-001 … 02-011 | Feed parser attachment fixes, SemVer exact values, regression suites, telemetry/docs updates, and handover complete; ops runbook now details attachment verification + proxy usage. | -| Sprint 2 | Connector & Data Implementation Wave | src/StellaOps.Concelier.Connector.Vndr.Cisco/TASKS.md | DONE (2025-10-14) | Team Connector Expansion – Regional & Vendor Feeds | FEEDCONN-CISCO-02-001 … 02-007 | OAuth fetch pipeline, DTO/mapping, tests, and telemetry/docs shipped; monitoring/export integration follow-ups recorded in Ops docs and exporter backlog (completed). | -| Sprint 2 | Connector & Data Implementation Wave | src/StellaOps.Concelier.Connector.Vndr.Msrc/TASKS.md | DONE (2025-10-15) | Team Connector Expansion – Regional & Vendor Feeds | FEEDCONN-MSRC-02-001 … 02-008 | Azure AD onboarding (02-008) unblocked fetch/parse/map pipeline; fixtures, telemetry/docs, and Offline Kit guidance published in `docs/ops/concelier-msrc-operations.md`. | -| Sprint 2 | Connector & Data Implementation Wave | src/StellaOps.Concelier.Connector.Cve/TASKS.md | DONE (2025-10-15) | Team Connector Support & Monitoring | FEEDCONN-CVE-02-001 … 02-002 | CVE data-source selection, fetch pipeline, and docs landed 2025-10-10. 2025-10-15: smoke verified using the seeded mirror fallback; connector now logs a warning and pulls from `seed-data/cve/` until live CVE Services credentials arrive. | -| Sprint 2 | Connector & Data Implementation Wave | src/StellaOps.Concelier.Connector.Kev/TASKS.md | DONE (2025-10-12) | Team Connector Support & Monitoring | FEEDCONN-KEV-02-001 … 02-002 | KEV catalog ingestion, fixtures, telemetry, and schema validation completed 2025-10-12; ops dashboard published. | -| Sprint 2 | Connector & Data Implementation Wave | docs/TASKS.md | DONE (2025-10-11) | Team Docs & Knowledge Base | FEEDDOCS-DOCS-01-001 | Canonical schema docs refresh
Updated canonical schema + provenance guides with SemVer style, normalized version rules, decision reason change log, and migration notes. | -| Sprint 2 | Connector & Data Implementation Wave | docs/TASKS.md | DONE (2025-10-11) | Team Docs & Knowledge Base | FEEDDOCS-DOCS-02-001 | Concelier-SemVer Playbook
Published merge playbook covering mapper patterns, dedupe flow, indexes, and rollout checklist. | -| Sprint 2 | Connector & Data Implementation Wave | docs/TASKS.md | DONE (2025-10-11) | Team Docs & Knowledge Base | FEEDDOCS-DOCS-02-002 | Normalized versions query guide
Delivered Mongo index/query addendum with `$unwind` recipes, dedupe checks, and operational checklist.
Instructions to work:
DONE Read ./AGENTS.md and docs/AGENTS.md. Document every schema/index/query change produced in Sprint 1-2 leveraging ./src/FASTER_MODELING_AND_NORMALIZATION.md. | -| Sprint 3 | Conflict Resolution Integration & Communications | src/StellaOps.Concelier.Core/TASKS.md | DONE (2025-10-11) | Team Core Engine & Storage Analytics | FEEDCORE-ENGINE-03-001 | Canonical merger implementation
`CanonicalMerger` ships with freshness/tie-breaker logic, provenance, and unit coverage feeding Merge. | -| Sprint 3 | Conflict Resolution Integration & Communications | src/StellaOps.Concelier.Core/TASKS.md | DONE (2025-10-11) | Team Core Engine & Storage Analytics | FEEDCORE-ENGINE-03-002 | Field precedence and tie-breaker map
Field precedence tables and tie-breaker metrics wired into the canonical merge flow; docs/tests updated.
Instructions to work:
Read ./AGENTS.md and core AGENTS. Implement the conflict resolver exactly as specified in ./src/DEDUP_CONFLICTS_RESOLUTION_ALGO.md, coordinating with Merge and Storage teammates. | -| Sprint 3 | Conflict Resolution Integration & Communications | src/StellaOps.Concelier.Storage.Mongo/TASKS.md | DONE (2025-10-11) | Team Core Engine & Storage Analytics | FEEDSTORAGE-DATA-03-001 | Merge event provenance audit prep
Merge events now persist `fieldDecisions` and analytics-ready provenance snapshots. | -| Sprint 3 | Conflict Resolution Integration & Communications | src/StellaOps.Concelier.Storage.Mongo/TASKS.md | DONE (2025-10-11) | Team Core Engine & Storage Analytics | FEEDSTORAGE-DATA-02-001 | Normalized range dual-write + backfill
Dual-write/backfill flag delivered; migration + options validated in tests. | -| Sprint 3 | Conflict Resolution Integration & Communications | src/StellaOps.Concelier.Storage.Mongo/TASKS.md | DONE (2025-10-11) | Team Core Engine & Storage Analytics | FEEDSTORAGE-TESTS-02-004 | Restore AdvisoryStore build after normalized versions refactor
Storage tests adjusted for normalized versions/decision reasons.
Instructions to work:
Read ./AGENTS.md and storage AGENTS. Extend merge events with decision reasons and analytics views to support the conflict rules, and deliver the dual-write/backfill for `NormalizedVersions` + `decisionReason` so connectors can roll out safely. | -| Sprint 3 | Conflict Resolution Integration & Communications | src/StellaOps.Concelier.Merge/TASKS.md | DONE (2025-10-11) | Team Merge & QA Enforcement | FEEDMERGE-ENGINE-04-001 | GHSA/NVD/OSV conflict rules
Merge pipeline consumes `CanonicalMerger` output prior to precedence merge. | -| Sprint 3 | Conflict Resolution Integration & Communications | src/StellaOps.Concelier.Merge/TASKS.md | DONE (2025-10-11) | Team Merge & QA Enforcement | FEEDMERGE-ENGINE-04-002 | Override metrics instrumentation
Merge events capture per-field decisions; counters/logs align with conflict rules. | -| Sprint 3 | Conflict Resolution Integration & Communications | src/StellaOps.Concelier.Merge/TASKS.md | DONE (2025-10-11) | Team Merge & QA Enforcement | FEEDMERGE-ENGINE-04-003 | Reference & credit union pipeline
Canonical merge preserves unions with updated tests. | -| Sprint 3 | Conflict Resolution Integration & Communications | src/StellaOps.Concelier.Merge/TASKS.md | DONE (2025-10-11) | Team Merge & QA Enforcement | FEEDMERGE-QA-04-001 | End-to-end conflict regression suite
Added regression tests (`AdvisoryMergeServiceTests`) covering canonical + precedence flow.
Instructions to work:
Read ./AGENTS.md and merge AGENTS. Integrate the canonical merger, instrument metrics, and deliver comprehensive regression tests following ./src/DEDUP_CONFLICTS_RESOLUTION_ALGO.md. | -| Sprint 3 | Conflict Resolution Integration & Communications | src/StellaOps.Concelier.Connector.Ghsa/TASKS.md | DONE (2025-10-12) | Team Connector Regression Fixtures | FEEDCONN-GHSA-04-002 | GHSA conflict regression fixtures | -| Sprint 3 | Conflict Resolution Integration & Communications | src/StellaOps.Concelier.Connector.Nvd/TASKS.md | DONE (2025-10-12) | Team Connector Regression Fixtures | FEEDCONN-NVD-04-002 | NVD conflict regression fixtures | -| Sprint 3 | Conflict Resolution Integration & Communications | src/StellaOps.Concelier.Connector.Osv/TASKS.md | DONE (2025-10-12) | Team Connector Regression Fixtures | FEEDCONN-OSV-04-002 | OSV conflict regression fixtures
Instructions to work:
Read ./AGENTS.md and module AGENTS. Produce fixture triples supporting the precedence/tie-breaker paths defined in ./src/DEDUP_CONFLICTS_RESOLUTION_ALGO.md and hand them to Merge QA. | -| Sprint 3 | Conflict Resolution Integration & Communications | docs/TASKS.md | DONE (2025-10-11) | Team Documentation Guild – Conflict Guidance | FEEDDOCS-DOCS-05-001 | Concelier Conflict Rules
Runbook published at `docs/ops/concelier-conflict-resolution.md`; metrics/log guidance aligned with Sprint 3 merge counters. | -| Sprint 3 | Conflict Resolution Integration & Communications | docs/TASKS.md | DONE (2025-10-16) | Team Documentation Guild – Conflict Guidance | FEEDDOCS-DOCS-05-002 | Conflict runbook ops rollout
Ops review completed, alert thresholds applied, and change log appended in `docs/ops/concelier-conflict-resolution.md`; task closed after connector signals verified. | -| Sprint 4 | Schema Parity & Freshness Alignment | src/StellaOps.Concelier.Models/TASKS.md | DONE (2025-10-15) | Team Models & Merge Leads | FEEDMODELS-SCHEMA-04-001 | Advisory schema parity (description/CWE/canonical metric)
Extend `Advisory` and related records with description text, CWE collection, and canonical metric pointer; refresh validation + serializer determinism tests. | -| Sprint 4 | Schema Parity & Freshness Alignment | src/StellaOps.Concelier.Core/TASKS.md | DONE (2025-10-15) | Team Core Engine & Storage Analytics | FEEDCORE-ENGINE-04-003 | Canonical merger parity for new fields
Teach `CanonicalMerger` to populate description, CWEResults, and canonical metric pointer with provenance + regression coverage. | -| Sprint 4 | Schema Parity & Freshness Alignment | src/StellaOps.Concelier.Core/TASKS.md | DONE (2025-10-15) | Team Core Engine & Storage Analytics | FEEDCORE-ENGINE-04-004 | Reference normalization & freshness instrumentation cleanup
Implement URL normalization for reference dedupe, align freshness-sensitive instrumentation, and add analytics tests. | -| Sprint 4 | Schema Parity & Freshness Alignment | src/StellaOps.Concelier.Merge/TASKS.md | DONE (2025-10-15) | Team Merge & QA Enforcement | FEEDMERGE-ENGINE-04-004 | Merge pipeline parity for new advisory fields
Ensure merge service + merge events surface description/CWE/canonical metric decisions with updated metrics/tests. | -| Sprint 4 | Schema Parity & Freshness Alignment | src/StellaOps.Concelier.Merge/TASKS.md | DONE (2025-10-15) | Team Merge & QA Enforcement | FEEDMERGE-ENGINE-04-005 | Connector coordination for new advisory fields
GHSA/NVD/OSV connectors now ship description, CWE, and canonical metric data with refreshed fixtures; merge coordination log updated and exporters notified. | -| Sprint 4 | Schema Parity & Freshness Alignment | src/StellaOps.Concelier.Exporter.Json/TASKS.md | DONE (2025-10-15) | Team Exporters – JSON | FEEDEXPORT-JSON-04-001 | Surface new advisory fields in JSON exporter
Update schemas/offline bundle + fixtures once model/core parity lands.
2025-10-15: `dotnet test src/StellaOps.Concelier.Exporter.Json.Tests` validated canonical metric/CWE emission. | -| Sprint 4 | Schema Parity & Freshness Alignment | src/StellaOps.Concelier.Exporter.TrivyDb/TASKS.md | DONE (2025-10-15) | Team Exporters – Trivy DB | FEEDEXPORT-TRIVY-04-001 | Propagate new advisory fields into Trivy DB package
Extend Bolt builder, metadata, and regression tests for the expanded schema.
2025-10-15: `dotnet test src/StellaOps.Concelier.Exporter.TrivyDb.Tests` confirmed canonical metric/CWE propagation. | -| Sprint 4 | Schema Parity & Freshness Alignment | src/StellaOps.Concelier.Connector.Ghsa/TASKS.md | DONE (2025-10-16) | Team Connector Regression Fixtures | FEEDCONN-GHSA-04-004 | Harden CVSS fallback so canonical metric ids persist when GitHub omits vectors; extend fixtures and document severity precedence hand-off to Merge. | -| Sprint 4 | Schema Parity & Freshness Alignment | src/StellaOps.Concelier.Connector.Osv/TASKS.md | DONE (2025-10-16) | Team Connector Expansion – GHSA/NVD/OSV | FEEDCONN-OSV-04-005 | Map OSV advisories lacking CVSS vectors to canonical metric ids/notes and document CWE provenance quirks; schedule parity fixture updates. | -| Sprint 5 | Excititor Core Foundations | src/StellaOps.Excititor.Core/TASKS.md | DONE (2025-10-15) | Team Excititor Core & Policy | EXCITITOR-CORE-01-001 | Stand up canonical VEX claim/consensus records with deterministic serializers so Storage/Exports share a stable contract. | -| Sprint 5 | Excititor Core Foundations | src/StellaOps.Excititor.Core/TASKS.md | DONE (2025-10-15) | Team Excititor Core & Policy | EXCITITOR-CORE-01-002 | Implement trust-weighted consensus resolver with baseline policy weights, justification gates, telemetry output, and majority/tie handling. | -| Sprint 5 | Excititor Core Foundations | src/StellaOps.Excititor.Core/TASKS.md | DONE (2025-10-15) | Team Excititor Core & Policy | EXCITITOR-CORE-01-003 | Publish shared connector/exporter/attestation abstractions and deterministic query signature utilities for cache/attestation workflows. | -| Sprint 5 | Excititor Core Foundations | src/StellaOps.Excititor.Policy/TASKS.md | DONE (2025-10-15) | Team Excititor Policy | EXCITITOR-POLICY-01-001 | Established policy options & snapshot provider covering baseline weights/overrides. | -| Sprint 5 | Excititor Core Foundations | src/StellaOps.Excititor.Policy/TASKS.md | DONE (2025-10-15) | Team Excititor Policy | EXCITITOR-POLICY-01-002 | Policy evaluator now feeds consensus resolver with immutable snapshots. | -| Sprint 5 | Excititor Core Foundations | src/StellaOps.Excititor.Policy/TASKS.md | DONE (2025-10-16) | Team Excititor Policy | EXCITITOR-POLICY-01-003 | Author policy diagnostics, CLI/WebService surfacing, and documentation updates. | -| Sprint 5 | Excititor Core Foundations | src/StellaOps.Excititor.Policy/TASKS.md | DONE (2025-10-16) | Team Excititor Policy | EXCITITOR-POLICY-01-004 | Implement YAML/JSON schema validation and deterministic diagnostics for operator bundles. | -| Sprint 5 | Excititor Core Foundations | src/StellaOps.Excititor.Policy/TASKS.md | DONE (2025-10-16) | Team Excititor Policy | EXCITITOR-POLICY-01-005 | Add policy change tracking, snapshot digests, and telemetry/logging hooks. | -| Sprint 5 | Excititor Core Foundations | src/StellaOps.Excititor.Storage.Mongo/TASKS.md | DONE (2025-10-15) | Team Excititor Storage | EXCITITOR-STORAGE-01-001 | Mongo mapping registry plus raw/export entities and DI extensions in place. | -| Sprint 5 | Excititor Core Foundations | src/StellaOps.Excititor.Storage.Mongo/TASKS.md | DONE (2025-10-16) | Team Excititor Storage | EXCITITOR-STORAGE-01-004 | Build provider/consensus/cache class maps and related collections. | -| Sprint 5 | Excititor Core Foundations | src/StellaOps.Excititor.Export/TASKS.md | DONE (2025-10-15) | Team Excititor Export | EXCITITOR-EXPORT-01-001 | Export engine delivers cache lookup, manifest creation, and policy integration. | -| Sprint 5 | Excititor Core Foundations | src/StellaOps.Excititor.Export/TASKS.md | DONE (2025-10-17) | Team Excititor Export | EXCITITOR-EXPORT-01-004 | Connect export engine to attestation client and persist Rekor metadata. | -| Sprint 5 | Excititor Core Foundations | src/StellaOps.Excititor.Attestation/TASKS.md | DONE (2025-10-16) | Team Excititor Attestation | EXCITITOR-ATTEST-01-001 | Implement in-toto predicate + DSSE builder providing envelopes for export attestation. | -| Sprint 5 | Excititor Core Foundations | src/StellaOps.Excititor.Connectors.Abstractions/TASKS.md | DONE (2025-10-17) | Team Excititor Connectors | EXCITITOR-CONN-ABS-01-001 | Deliver shared connector context/base classes so provider plug-ins can be activated via WebService/Worker. | -| Sprint 5 | Excititor Core Foundations | src/StellaOps.Excititor.WebService/TASKS.md | DONE (2025-10-17) | Team Excititor WebService | EXCITITOR-WEB-01-001 | Scaffold minimal API host, DI, and `/excititor/status` endpoint integrating policy, storage, export, and attestation services. | -| Sprint 6 | Excititor Ingest & Formats | src/StellaOps.Excititor.Worker/TASKS.md | DONE (2025-10-17) | Team Excititor Worker | EXCITITOR-WORKER-01-001 | Create Worker host with provider scheduling and logging to drive recurring pulls/reconciliation. | -| Sprint 6 | Excititor Ingest & Formats | src/StellaOps.Excititor.Formats.CSAF/TASKS.md | DONE (2025-10-17) | Team Excititor Formats | EXCITITOR-FMT-CSAF-01-001 | Implement CSAF normalizer foundation translating provider documents into `VexClaim` entries. | -| Sprint 6 | Excititor Ingest & Formats | src/StellaOps.Excititor.Formats.CycloneDX/TASKS.md | DONE (2025-10-17) | Team Excititor Formats | EXCITITOR-FMT-CYCLONE-01-001 | Implement CycloneDX VEX normalizer capturing `analysis` state and component references. | -| Sprint 6 | Excititor Ingest & Formats | src/StellaOps.Excititor.Formats.OpenVEX/TASKS.md | DONE (2025-10-17) | Team Excititor Formats | EXCITITOR-FMT-OPENVEX-01-001 | Implement OpenVEX normalizer to ingest attestations into canonical claims with provenance. | -| Sprint 6 | Excititor Ingest & Formats | src/StellaOps.Excititor.Connectors.RedHat.CSAF/TASKS.md | DONE (2025-10-17) | Team Excititor Connectors – Red Hat | EXCITITOR-CONN-RH-01-001 | Ship Red Hat CSAF provider metadata discovery enabling incremental pulls. | -| Sprint 6 | Excititor Ingest & Formats | src/StellaOps.Excititor.Connectors.RedHat.CSAF/TASKS.md | DONE (2025-10-17) | Team Excititor Connectors – Red Hat | EXCITITOR-CONN-RH-01-002 | Fetch CSAF windows with ETag handling, resume tokens, quarantine on schema errors, and persist raw docs. | -| Sprint 6 | Excititor Ingest & Formats | src/StellaOps.Excititor.Connectors.RedHat.CSAF/TASKS.md | DONE (2025-10-17) | Team Excititor Connectors – Red Hat | EXCITITOR-CONN-RH-01-003 | Populate provider trust overrides (cosign issuer, identity regex) and provenance hints for policy evaluation/logging. | -| Sprint 6 | Excititor Ingest & Formats | src/StellaOps.Excititor.Connectors.RedHat.CSAF/TASKS.md | DONE (2025-10-17) | Team Excititor Connectors – Red Hat | EXCITITOR-CONN-RH-01-004 | Persist resume cursors (last updated timestamp/document hashes) in storage and reload during fetch to avoid duplicates. | -| Sprint 6 | Excititor Ingest & Formats | src/StellaOps.Excititor.Connectors.RedHat.CSAF/TASKS.md | DONE (2025-10-17) | Team Excititor Connectors – Red Hat | EXCITITOR-CONN-RH-01-005 | Register connector in Worker/WebService DI, add scheduled jobs, and document CLI triggers for Red Hat CSAF pulls. | -| Sprint 6 | Excititor Ingest & Formats | src/StellaOps.Excititor.Connectors.RedHat.CSAF/TASKS.md | DONE (2025-10-17) | Team Excititor Connectors – Red Hat | EXCITITOR-CONN-RH-01-006 | Add CSAF normalization parity fixtures ensuring RHSA-specific metadata is preserved. | -| Sprint 6 | Excititor Ingest & Formats | src/StellaOps.Excititor.Connectors.Cisco.CSAF/TASKS.md | DONE (2025-10-17) | Team Excititor Connectors – Cisco | EXCITITOR-CONN-CISCO-01-001 | Implement Cisco CSAF endpoint discovery/auth to unlock paginated pulls. | -| Sprint 6 | Excititor Ingest & Formats | src/StellaOps.Excititor.Connectors.Cisco.CSAF/TASKS.md | DONE (2025-10-17) | Team Excititor Connectors – Cisco | EXCITITOR-CONN-CISCO-01-002 | Implement Cisco CSAF paginated fetch loop with dedupe and raw persistence support. | -| Sprint 6 | Excititor Ingest & Formats | src/StellaOps.Excititor.Connectors.SUSE.RancherVEXHub/TASKS.md | DONE (2025-10-17) | Team Excititor Connectors – SUSE | EXCITITOR-CONN-SUSE-01-001 | Build Rancher VEX Hub discovery/subscription path with offline snapshot support. | -| Sprint 6 | Excititor Ingest & Formats | src/StellaOps.Excititor.Connectors.MSRC.CSAF/TASKS.md | DONE (2025-10-17) | Team Excititor Connectors – MSRC | EXCITITOR-CONN-MS-01-001 | Deliver AAD onboarding/token cache for MSRC CSAF ingestion. | -| Sprint 6 | Excititor Ingest & Formats | src/StellaOps.Excititor.Connectors.Oracle.CSAF/TASKS.md | DONE (2025-10-17) | Team Excititor Connectors – Oracle | EXCITITOR-CONN-ORACLE-01-001 | Implement Oracle CSAF catalogue discovery with CPU calendar awareness. | -| Sprint 6 | Excititor Ingest & Formats | src/StellaOps.Excititor.Connectors.Ubuntu.CSAF/TASKS.md | DONE (2025-10-17) | Team Excititor Connectors – Ubuntu | EXCITITOR-CONN-UBUNTU-01-001 | Implement Ubuntu CSAF discovery and channel selection for USN ingestion. | -| Sprint 6 | Excititor Ingest & Formats | src/StellaOps.Excititor.Connectors.OCI.OpenVEX.Attest/TASKS.md | DONE (2025-10-18) | Team Excititor Connectors – OCI | EXCITITOR-CONN-OCI-01-001 | Wire OCI discovery/auth to fetch OpenVEX attestations for configured images. | -| Sprint 6 | Excititor Ingest & Formats | src/StellaOps.Excititor.Connectors.OCI.OpenVEX.Attest/TASKS.md | DONE (2025-10-18) | Team Excititor Connectors – OCI | EXCITITOR-CONN-OCI-01-002 | Attestation fetch & verify loop – download DSSE attestations, trigger verification, handle retries/backoff, persist raw statements. | -| Sprint 6 | Excititor Ingest & Formats | src/StellaOps.Excititor.Connectors.OCI.OpenVEX.Attest/TASKS.md | DONE (2025-10-18) | Team Excititor Connectors – OCI | EXCITITOR-CONN-OCI-01-003 | Provenance metadata & policy hooks – emit image, subject digest, issuer, and trust metadata for policy weighting/logging. | -| Sprint 6 | Excititor Ingest & Formats | src/StellaOps.Cli/TASKS.md | DONE (2025-10-18) | DevEx/CLI | EXCITITOR-CLI-01-001 | Add `excititor` CLI verbs bridging to WebService with consistent auth and offline UX. | +This file describe implementation of Stella Ops (docs/README.md). Implementation must respect rules from AGENTS.md (read if you have not). + +| Sprint | Theme | Tasks File Path | Status | Type of Specialist | Task ID | Task Description | +| --- | --- | --- | --- | --- | --- | --- | +| Sprint 1 | Stabilize In-Progress Foundations | src/StellaOps.Concelier.Models/TASKS.md | DONE (2025-10-12) | Team Models & Merge Leads | FEEDMODELS-SCHEMA-01-001 | SemVer primitive range-style metadata
Instructions to work:
DONE Read ./AGENTS.md and src/StellaOps.Concelier.Models/AGENTS.md. This task lays the groundwork—complete the SemVer helper updates before teammates pick up FEEDMODELS-SCHEMA-01-002/003 and FEEDMODELS-SCHEMA-02-900. Use ./src/FASTER_MODELING_AND_NORMALIZATION.md for the target rule structure. | +| Sprint 1 | Stabilize In-Progress Foundations | src/StellaOps.Concelier.Models/TASKS.md | DONE (2025-10-11) | Team Models & Merge Leads | FEEDMODELS-SCHEMA-01-002 | Provenance decision rationale field
Instructions to work:
AdvisoryProvenance now carries `decisionReason` and docs/tests were updated. Connectors and merge tasks should populate the field when applying precedence/freshness/tie-breaker logic; see src/StellaOps.Concelier.Models/PROVENANCE_GUIDELINES.md for usage guidance. | +| Sprint 1 | Stabilize In-Progress Foundations | src/StellaOps.Concelier.Models/TASKS.md | DONE (2025-10-11) | Team Models & Merge Leads | FEEDMODELS-SCHEMA-01-003 | Normalized version rules collection
Instructions to work:
`AffectedPackage.NormalizedVersions` and supporting comparer/docs/tests shipped. Connector owners must emit rule arrays per ./src/FASTER_MODELING_AND_NORMALIZATION.md and report progress via FEEDMERGE-COORD-02-900 so merge/storage backfills can proceed. | +| Sprint 1 | Stabilize In-Progress Foundations | src/StellaOps.Concelier.Models/TASKS.md | DONE (2025-10-12) | Team Models & Merge Leads | FEEDMODELS-SCHEMA-02-900 | Range primitives for SemVer/EVR/NEVRA metadata
Instructions to work:
DONE Read ./AGENTS.md and src/StellaOps.Concelier.Models/AGENTS.md before resuming this stalled effort. Confirm helpers align with the new `NormalizedVersions` representation so connectors finishing in Sprint 2 can emit consistent metadata. | +| Sprint 1 | Stabilize In-Progress Foundations | src/StellaOps.Concelier.Normalization/TASKS.md | DONE (2025-10-11) | Team Normalization & Storage Backbone | FEEDNORM-NORM-02-001 | SemVer normalized rule emitter
Shared `SemVerRangeRuleBuilder` now outputs primitives + normalized rules per `FASTER_MODELING_AND_NORMALIZATION.md`; CVE/GHSA connectors consuming the API have verified fixtures. | +| Sprint 1 | Stabilize In-Progress Foundations | src/StellaOps.Concelier.Storage.Mongo/TASKS.md | DONE (2025-10-11) | Team Normalization & Storage Backbone | FEEDSTORAGE-DATA-02-001 | Normalized range dual-write + backfill
AdvisoryStore dual-writes flattened `normalizedVersions` when `concelier.storage.enableSemVerStyle` is set; migration `20251011-semver-style-backfill` updates historical records and docs outline the rollout. | +| Sprint 1 | Stabilize In-Progress Foundations | src/StellaOps.Concelier.Storage.Mongo/TASKS.md | DONE (2025-10-11) | Team Normalization & Storage Backbone | FEEDSTORAGE-DATA-02-002 | Provenance decision reason persistence
Storage now persists `provenance.decisionReason` for advisories and merge events; tests cover round-trips. | +| Sprint 1 | Stabilize In-Progress Foundations | src/StellaOps.Concelier.Storage.Mongo/TASKS.md | DONE (2025-10-11) | Team Normalization & Storage Backbone | FEEDSTORAGE-DATA-02-003 | Normalized versions indexing
Bootstrapper seeds compound/sparse indexes for flattened normalized rules and `docs/dev/mongo_indices.md` documents query guidance. | +| Sprint 1 | Stabilize In-Progress Foundations | src/StellaOps.Concelier.Storage.Mongo/TASKS.md | DONE (2025-10-11) | Team Normalization & Storage Backbone | FEEDSTORAGE-TESTS-02-004 | Restore AdvisoryStore build after normalized versions refactor
Updated constructors/tests keep storage suites passing with the new feature flag defaults. | +| Sprint 1 | Stabilize In-Progress Foundations | src/StellaOps.Concelier.WebService/TASKS.md | DONE (2025-10-12) | Team WebService & Authority | FEEDWEB-ENGINE-01-002 | Plumb Authority client resilience options
WebService wires `authority.resilience.*` into `AddStellaOpsAuthClient` and adds binding coverage via `AuthorityClientResilienceOptionsAreBound`. | +| Sprint 1 | Stabilize In-Progress Foundations | src/StellaOps.Concelier.WebService/TASKS.md | DONE (2025-10-12) | Team WebService & Authority | FEEDWEB-DOCS-01-003 | Author ops guidance for resilience tuning
Install/runbooks document connected vs air-gapped resilience profiles and monitoring hooks. | +| Sprint 1 | Stabilize In-Progress Foundations | src/StellaOps.Concelier.WebService/TASKS.md | DONE (2025-10-12) | Team WebService & Authority | FEEDWEB-DOCS-01-004 | Document authority bypass logging patterns
Operator guides now call out `route/status/subject/clientId/scopes/bypass/remote` audit fields and SIEM triggers. | +| Sprint 1 | Stabilize In-Progress Foundations | src/StellaOps.Concelier.WebService/TASKS.md | DONE (2025-10-12) | Team WebService & Authority | FEEDWEB-DOCS-01-005 | Update Concelier operator guide for enforcement cutoff
Install guide reiterates the 2025-12-31 cutoff and links audit signals to the rollout checklist. | +| Sprint 1 | Stabilize In-Progress Foundations | src/StellaOps.Authority/TASKS.md | DONE (2025-10-11) | Team WebService & Authority | SEC3.HOST | Rate limiter policy binding
Authority host now applies configuration-driven fixed windows to `/token`, `/authorize`, and `/internal/*`; integration tests assert 429 + `Retry-After` headers; docs/config samples refreshed for Docs guild diagrams. | +| Sprint 1 | Stabilize In-Progress Foundations | src/StellaOps.Authority/TASKS.md | DONE (2025-10-11) | Team WebService & Authority | SEC3.BUILD | Authority rate-limiter follow-through
`Security.RateLimiting` now fronts token/authorize/internal limiters; Authority + Configuration matrices (`dotnet test src/StellaOps.Authority/StellaOps.Authority.sln`, `dotnet test src/StellaOps.Configuration.Tests/StellaOps.Configuration.Tests.csproj`) passed on 2025-10-11; awaiting #authority-core broadcast. | +| Sprint 1 | Stabilize In-Progress Foundations | src/StellaOps.Authority/TASKS.md | DONE (2025-10-14) | Team Authority Platform & Security Guild | AUTHCORE-BUILD-OPENIDDICT / AUTHCORE-STORAGE-DEVICE-TOKENS / AUTHCORE-BOOTSTRAP-INVITES | Address remaining Authority compile blockers (OpenIddict transaction shim, token device document, bootstrap invite cleanup) so `dotnet build src/StellaOps.Authority.sln` returns success. | +| Sprint 1 | Stabilize In-Progress Foundations | src/StellaOps.Authority/StellaOps.Authority.Plugin.Standard/TASKS.md | DONE (2025-10-11) | Team WebService & Authority | PLG6.DOC | Plugin developer guide polish
Section 9 now documents rate limiter metadata, config keys, and lockout interplay; YAML samples updated alongside Authority config templates. | +| Sprint 1 | Stabilize In-Progress Foundations | src/StellaOps.Concelier.Connector.CertCc/TASKS.md | DONE (2025-10-11) | Team Connector Resumption – CERT/RedHat | FEEDCONN-CERTCC-02-001 | Fetch pipeline & state tracking
Summary planner now drives monthly/yearly VINCE fetches, persists pending summaries/notes, and hydrates VINCE detail queue with telemetry.
Team instructions: Read ./AGENTS.md and src/StellaOps.Concelier.Connector.CertCc/AGENTS.md. Coordinate daily with Models/Merge leads so new normalizedVersions output and provenance tags stay aligned with ./src/FASTER_MODELING_AND_NORMALIZATION.md. | +| Sprint 1 | Stabilize In-Progress Foundations | src/StellaOps.Concelier.Connector.CertCc/TASKS.md | DONE (2025-10-11) | Team Connector Resumption – CERT/RedHat | FEEDCONN-CERTCC-02-002 | VINCE note detail fetcher
Summary planner queues VINCE note detail endpoints, persists raw JSON with SHA/ETag metadata, and records retry/backoff metrics. | +| Sprint 1 | Stabilize In-Progress Foundations | src/StellaOps.Concelier.Connector.CertCc/TASKS.md | DONE (2025-10-11) | Team Connector Resumption – CERT/RedHat | FEEDCONN-CERTCC-02-003 | DTO & parser implementation
Added VINCE DTO aggregate, Markdown→text sanitizer, vendor/status/vulnerability parsers, and parser regression fixture. | +| Sprint 1 | Stabilize In-Progress Foundations | src/StellaOps.Concelier.Connector.CertCc/TASKS.md | DONE (2025-10-11) | Team Connector Resumption – CERT/RedHat | FEEDCONN-CERTCC-02-004 | Canonical mapping & range primitives
VINCE DTO aggregate flows through `CertCcMapper`, emitting vendor range primitives + normalized version rules that persist via `_advisoryStore`. | +| Sprint 1 | Stabilize In-Progress Foundations | src/StellaOps.Concelier.Connector.CertCc/TASKS.md | DONE (2025-10-12) | Team Connector Resumption – CERT/RedHat | FEEDCONN-CERTCC-02-005 | Deterministic fixtures/tests
Snapshot harness refreshed 2025-10-12; `certcc-*.snapshot.json` regenerated and regression suite green without UPDATE flag drift. | +| Sprint 1 | Stabilize In-Progress Foundations | src/StellaOps.Concelier.Connector.CertCc/TASKS.md | DONE (2025-10-12) | Team Connector Resumption – CERT/RedHat | FEEDCONN-CERTCC-02-006 | Telemetry & documentation
`CertCcDiagnostics` publishes summary/detail/parse/map metrics (meter `StellaOps.Concelier.Connector.CertCc`), README documents instruments, and log guidance captured for Ops on 2025-10-12. | +| Sprint 1 | Stabilize In-Progress Foundations | src/StellaOps.Concelier.Connector.CertCc/TASKS.md | DONE (2025-10-12) | Team Connector Resumption – CERT/RedHat | FEEDCONN-CERTCC-02-007 | Connector test harness remediation
Harness now wires `AddSourceCommon`, resets `FakeTimeProvider`, and passes canned-response regression run dated 2025-10-12. | +| Sprint 1 | Stabilize In-Progress Foundations | src/StellaOps.Concelier.Connector.CertCc/TASKS.md | DONE (2025-10-11) | Team Connector Resumption – CERT/RedHat | FEEDCONN-CERTCC-02-008 | Snapshot coverage handoff
Fixtures regenerated with normalized ranges + provenance fields on 2025-10-11; QA handoff notes published and merge backfill unblocked. | +| Sprint 1 | Stabilize In-Progress Foundations | src/StellaOps.Concelier.Connector.CertCc/TASKS.md | DONE (2025-10-12) | Team Connector Resumption – CERT/RedHat | FEEDCONN-CERTCC-02-012 | Schema sync & snapshot regen follow-up
Fixtures regenerated with normalizedVersions + provenance decision reasons; handoff notes updated for Merge backfill 2025-10-12. | +| Sprint 1 | Stabilize In-Progress Foundations | src/StellaOps.Concelier.Connector.CertCc/TASKS.md | DONE (2025-10-11) | Team Connector Resumption – CERT/RedHat | FEEDCONN-CERTCC-02-009 | Detail/map reintegration plan
Staged reintegration plan published in `src/StellaOps.Concelier.Connector.CertCc/FEEDCONN-CERTCC-02-009_PLAN.md`; coordinates enablement with FEEDCONN-CERTCC-02-004. | +| Sprint 1 | Stabilize In-Progress Foundations | src/StellaOps.Concelier.Connector.CertCc/TASKS.md | DONE (2025-10-12) | Team Connector Resumption – CERT/RedHat | FEEDCONN-CERTCC-02-010 | Partial-detail graceful degradation
Detail fetch now tolerates 404/403/410 responses and regression tests cover mixed endpoint availability. | +| Sprint 1 | Stabilize In-Progress Foundations | src/StellaOps.Concelier.Connector.Distro.RedHat/TASKS.md | DONE (2025-10-11) | Team Connector Resumption – CERT/RedHat | FEEDCONN-REDHAT-02-001 | Fixture validation sweep
Instructions to work:
Fixtures regenerated post-model-helper rollout; provenance ordering and normalizedVersions scaffolding verified via tests. Conflict resolver deltas logged in src/StellaOps.Concelier.Connector.Distro.RedHat/CONFLICT_RESOLVER_NOTES.md for Sprint 3 consumers. | +| Sprint 1 | Stabilize In-Progress Foundations | src/StellaOps.Concelier.Connector.Vndr.Apple/TASKS.md | DONE (2025-10-12) | Team Vendor Apple Specialists | FEEDCONN-APPLE-02-001 | Canonical mapping & range primitives
Mapper emits SemVer rules (`scheme=apple:*`); fixtures regenerated with trimmed references + new RSR coverage, update tooling finalized. | +| Sprint 1 | Stabilize In-Progress Foundations | src/StellaOps.Concelier.Connector.Vndr.Apple/TASKS.md | DONE (2025-10-11) | Team Vendor Apple Specialists | FEEDCONN-APPLE-02-002 | Deterministic fixtures/tests
Sanitized live fixtures + regression snapshots wired into tests; normalized rule coverage asserted. | +| Sprint 1 | Stabilize In-Progress Foundations | src/StellaOps.Concelier.Connector.Vndr.Apple/TASKS.md | DONE (2025-10-11) | Team Vendor Apple Specialists | FEEDCONN-APPLE-02-003 | Telemetry & documentation
Apple meter metrics wired into Concelier WebService OpenTelemetry configuration; README and fixtures document normalizedVersions coverage. | +| Sprint 1 | Stabilize In-Progress Foundations | src/StellaOps.Concelier.Connector.Vndr.Apple/TASKS.md | DONE (2025-10-12) | Team Vendor Apple Specialists | FEEDCONN-APPLE-02-004 | Live HTML regression sweep
Sanitised HT125326/HT125328/HT106355/HT214108/HT215500 fixtures recorded and regression tests green on 2025-10-12. | +| Sprint 1 | Stabilize In-Progress Foundations | src/StellaOps.Concelier.Connector.Vndr.Apple/TASKS.md | DONE (2025-10-11) | Team Vendor Apple Specialists | FEEDCONN-APPLE-02-005 | Fixture regeneration tooling
`UPDATE_APPLE_FIXTURES=1` flow fetches & rewrites fixtures; README documents usage.
Instructions to work:
DONE Read ./AGENTS.md and src/StellaOps.Concelier.Connector.Vndr.Apple/AGENTS.md. Resume stalled tasks, ensuring normalizedVersions output and fixtures align with ./src/FASTER_MODELING_AND_NORMALIZATION.md before handing data to the conflict sprint. | +| Sprint 1 | Stabilize In-Progress Foundations | src/StellaOps.Concelier.Connector.Ghsa/TASKS.md | DONE (2025-10-12) | Team Connector Normalized Versions Rollout | FEEDCONN-GHSA-02-001 | GHSA normalized versions & provenance
Team instructions: Read ./AGENTS.md and each module's AGENTS file. Adopt the `NormalizedVersions` array emitted by the models sprint, wiring provenance `decisionReason` where merge overrides occur. Follow ./src/FASTER_MODELING_AND_NORMALIZATION.md; report via src/StellaOps.Concelier.Merge/TASKS.md (FEEDMERGE-COORD-02-900). Progress 2025-10-11: GHSA/OSV emit normalized arrays with refreshed fixtures; CVE mapper now surfaces SemVer normalized ranges; NVD/KEV adoption pending; outstanding follow-ups include FEEDSTORAGE-DATA-02-001, FEEDMERGE-ENGINE-02-002, and rolling `tools/FixtureUpdater` updates across connectors. | +| Sprint 1 | Stabilize In-Progress Foundations | src/StellaOps.Concelier.Connector.Osv/TASKS.md | DONE (2025-10-12) | Team Connector Normalized Versions Rollout | FEEDCONN-OSV-02-003 | OSV normalized versions & freshness | +| Sprint 1 | Stabilize In-Progress Foundations | src/StellaOps.Concelier.Connector.Nvd/TASKS.md | DONE (2025-10-12) | Team Connector Normalized Versions Rollout | FEEDCONN-NVD-02-002 | NVD normalized versions & timestamps | +| Sprint 1 | Stabilize In-Progress Foundations | src/StellaOps.Concelier.Connector.Cve/TASKS.md | DONE (2025-10-12) | Team Connector Normalized Versions Rollout | FEEDCONN-CVE-02-003 | CVE normalized versions uplift | +| Sprint 1 | Stabilize In-Progress Foundations | src/StellaOps.Concelier.Connector.Kev/TASKS.md | DONE (2025-10-12) | Team Connector Normalized Versions Rollout | FEEDCONN-KEV-02-003 | KEV normalized versions propagation | +| Sprint 1 | Stabilize In-Progress Foundations | src/StellaOps.Concelier.Connector.Osv/TASKS.md | DONE (2025-10-12) | Team Connector Normalized Versions Rollout | FEEDCONN-OSV-04-003 | OSV parity fixture refresh | +| Sprint 1 | Stabilize In-Progress Foundations | src/StellaOps.Concelier.WebService/TASKS.md | DONE (2025-10-10) | Team WebService & Authority | FEEDWEB-DOCS-01-001 | Document authority toggle & scope requirements
Quickstart carries toggle/scope guidance pending docs guild review (no change this sprint). | +| Sprint 1 | Stabilize In-Progress Foundations | src/StellaOps.Concelier.WebService/TASKS.md | DONE (2025-10-12) | Team WebService & Authority | FEEDWEB-ENGINE-01-002 | Plumb Authority client resilience options
WebService wires `authority.resilience.*` into `AddStellaOpsAuthClient` and adds binding coverage via `AuthorityClientResilienceOptionsAreBound`. | +| Sprint 1 | Stabilize In-Progress Foundations | src/StellaOps.Concelier.WebService/TASKS.md | DONE (2025-10-12) | Team WebService & Authority | FEEDWEB-DOCS-01-003 | Author ops guidance for resilience tuning
Operator docs now outline connected vs air-gapped resilience profiles and monitoring cues. | +| Sprint 1 | Stabilize In-Progress Foundations | src/StellaOps.Concelier.WebService/TASKS.md | DONE (2025-10-12) | Team WebService & Authority | FEEDWEB-DOCS-01-004 | Document authority bypass logging patterns
Audit logging guidance highlights `route/status/subject/clientId/scopes/bypass/remote` fields and SIEM alerts. | +| Sprint 1 | Stabilize In-Progress Foundations | src/StellaOps.Concelier.WebService/TASKS.md | DONE (2025-10-12) | Team WebService & Authority | FEEDWEB-DOCS-01-005 | Update Concelier operator guide for enforcement cutoff
Install guide reiterates the 2025-12-31 cutoff and ties audit signals to rollout checks. | +| Sprint 1 | Stabilize In-Progress Foundations | src/StellaOps.Concelier.WebService/TASKS.md | DONE (2025-10-19) | Team WebService & Authority | FEEDWEB-OPS-01-006 | Rename plugin drop directory to namespaced path
Build outputs now point at `StellaOps.Concelier.PluginBinaries`/`StellaOps.Authority.PluginBinaries`; defaults/docs/tests updated to reflect the new layout. | +| Sprint 1 | Stabilize In-Progress Foundations | src/StellaOps.Concelier.WebService/TASKS.md | DONE (2025-10-11) | Team WebService & Authority | FEEDWEB-OPS-01-007 | Authority resilience adoption
Deployment docs and CLI notes explain the LIB5 resilience knobs for rollout.
Instructions to work:
DONE Read ./AGENTS.md and src/StellaOps.Concelier.WebService/AGENTS.md. These items were mid-flight; resume implementation ensuring docs/operators receive timely updates. | +| Sprint 1 | Stabilize In-Progress Foundations | src/StellaOps.Authority/TASKS.md | DONE (2025-10-11) | Team Authority Platform & Security Guild | AUTHCORE-ENGINE-01-001 | CORE8.RL — Rate limiter plumbing validated; integration tests green and docs handoff recorded for middleware ordering + Retry-After headers (see `docs/dev/authority-rate-limit-tuning-outline.md` for continuing guidance). | +| Sprint 1 | Stabilize In-Progress Foundations | src/StellaOps.Cryptography/TASKS.md | DONE (2025-10-11) | Team Authority Platform & Security Guild | AUTHCRYPTO-ENGINE-01-001 | SEC3.A — Shared metadata resolver confirmed via host test run; SEC3.B now unblocked for tuning guidance (outline captured in `docs/dev/authority-rate-limit-tuning-outline.md`). | +| Sprint 1 | Stabilize In-Progress Foundations | src/StellaOps.Cryptography/TASKS.md | DONE (2025-10-13) | Team Authority Platform & Security Guild | AUTHSEC-DOCS-01-002 | SEC3.B — Published `docs/security/rate-limits.md` with tuning matrix, alert thresholds, and lockout interplay guidance; Docs guild can lift copy into plugin guide. | +| Sprint 1 | Stabilize In-Progress Foundations | src/StellaOps.Cryptography/TASKS.md | DONE (2025-10-14) | Team Authority Platform & Security Guild | AUTHSEC-CRYPTO-02-001 | SEC5.B1 — Introduce libsodium signing provider and parity tests to unblock CLI verification enhancements. | +| Sprint 1 | Bootstrap & Replay Hardening | src/StellaOps.Cryptography/TASKS.md | DONE (2025-10-14) | Security Guild | AUTHSEC-CRYPTO-02-004 | SEC5.D/E — Finish bootstrap invite lifecycle (API/store/cleanup) and token device heuristics; build currently red due to pending handler integration. | +| Sprint 1 | Developer Tooling | src/StellaOps.Cli/TASKS.md | DONE (2025-10-15) | DevEx/CLI | AUTHCLI-DIAG-01-001 | Surface password policy diagnostics in CLI startup/output so operators see weakened overrides immediately.
CLI now loads Authority plug-ins at startup, logs weakened password policies (length/complexity), and regression coverage lives in `StellaOps.Cli.Tests/Services/AuthorityDiagnosticsReporterTests`. | +| Sprint 1 | Stabilize In-Progress Foundations | src/StellaOps.Authority/StellaOps.Authority.Plugin.Standard/TASKS.md | DONE (2025-10-11) | Team Authority Platform & Security Guild | AUTHPLUG-DOCS-01-001 | PLG6.DOC — Developer guide copy + diagrams merged 2025-10-11; limiter guidance incorporated and handed to Docs guild for asset export. | +| Sprint 2 | Connector & Data Implementation Wave | src/StellaOps.Concelier.Normalization/TASKS.md | DONE (2025-10-12) | Team Normalization & Storage Backbone | FEEDNORM-NORM-02-001 | SemVer normalized rule emitter
`SemVerRangeRuleBuilder` shipped 2025-10-12 with comparator/`||` support and fixtures aligning to `FASTER_MODELING_AND_NORMALIZATION.md`. | +| Sprint 2 | Connector & Data Implementation Wave | src/StellaOps.Concelier.Storage.Mongo/TASKS.md | DONE (2025-10-11) | Team Normalization & Storage Backbone | FEEDSTORAGE-DATA-02-001 | Normalized range dual-write + backfill | +| Sprint 2 | Connector & Data Implementation Wave | src/StellaOps.Concelier.Storage.Mongo/TASKS.md | DONE (2025-10-11) | Team Normalization & Storage Backbone | FEEDSTORAGE-DATA-02-002 | Provenance decision reason persistence | +| Sprint 2 | Connector & Data Implementation Wave | src/StellaOps.Concelier.Storage.Mongo/TASKS.md | DONE (2025-10-11) | Team Normalization & Storage Backbone | FEEDSTORAGE-DATA-02-003 | Normalized versions indexing
Indexes seeded + docs updated 2025-10-11 to cover flattened normalized rules for connector adoption. | +| Sprint 2 | Connector & Data Implementation Wave | src/StellaOps.Concelier.Merge/TASKS.md | DONE (2025-10-11) | Team Normalization & Storage Backbone | FEEDMERGE-ENGINE-02-002 | Normalized versions union & dedupe
Affected package resolver unions/dedupes normalized rules, stamps merge provenance with `decisionReason`, and tests cover the rollout. | +| Sprint 2 | Connector & Data Implementation Wave | src/StellaOps.Concelier.Connector.Ghsa/TASKS.md | DONE (2025-10-11) | Team Connector Expansion – GHSA/NVD/OSV | FEEDCONN-GHSA-02-001 | GHSA normalized versions & provenance | +| Sprint 2 | Connector & Data Implementation Wave | src/StellaOps.Concelier.Connector.Ghsa/TASKS.md | DONE (2025-10-11) | Team Connector Expansion – GHSA/NVD/OSV | FEEDCONN-GHSA-02-004 | GHSA credits & ecosystem severity mapping | +| Sprint 2 | Connector & Data Implementation Wave | src/StellaOps.Concelier.Connector.Ghsa/TASKS.md | DONE (2025-10-12) | Team Connector Expansion – GHSA/NVD/OSV | FEEDCONN-GHSA-02-005 | GitHub quota monitoring & retries | +| Sprint 2 | Connector & Data Implementation Wave | src/StellaOps.Concelier.Connector.Ghsa/TASKS.md | DONE (2025-10-12) | Team Connector Expansion – GHSA/NVD/OSV | FEEDCONN-GHSA-02-006 | Production credential & scheduler rollout | +| Sprint 2 | Connector & Data Implementation Wave | src/StellaOps.Concelier.Connector.Ghsa/TASKS.md | DONE (2025-10-12) | Team Connector Expansion – GHSA/NVD/OSV | FEEDCONN-GHSA-02-007 | Credit parity regression fixtures | +| Sprint 2 | Connector & Data Implementation Wave | src/StellaOps.Concelier.Connector.Nvd/TASKS.md | DONE (2025-10-11) | Team Connector Expansion – GHSA/NVD/OSV | FEEDCONN-NVD-02-002 | NVD normalized versions & timestamps | +| Sprint 2 | Connector & Data Implementation Wave | src/StellaOps.Concelier.Connector.Nvd/TASKS.md | DONE (2025-10-11) | Team Connector Expansion – GHSA/NVD/OSV | FEEDCONN-NVD-02-004 | NVD CVSS & CWE precedence payloads | +| Sprint 2 | Connector & Data Implementation Wave | src/StellaOps.Concelier.Connector.Nvd/TASKS.md | DONE (2025-10-12) | Team Connector Expansion – GHSA/NVD/OSV | FEEDCONN-NVD-02-005 | NVD merge/export parity regression | +| Sprint 2 | Connector & Data Implementation Wave | src/StellaOps.Concelier.Connector.Osv/TASKS.md | DONE (2025-10-11) | Team Connector Expansion – GHSA/NVD/OSV | FEEDCONN-OSV-02-003 | OSV normalized versions & freshness | +| Sprint 2 | Connector & Data Implementation Wave | src/StellaOps.Concelier.Connector.Osv/TASKS.md | DONE (2025-10-11) | Team Connector Expansion – GHSA/NVD/OSV | FEEDCONN-OSV-02-004 | OSV references & credits alignment | +| Sprint 2 | Connector & Data Implementation Wave | src/StellaOps.Concelier.Connector.Osv/TASKS.md | DONE (2025-10-12) | Team Connector Expansion – GHSA/NVD/OSV | FEEDCONN-OSV-02-005 | Fixture updater workflow
Resolved 2025-10-12: OSV mapper now derives canonical PURLs for Go + scoped npm packages when raw payloads omit `purl`; conflict fixtures unchanged for invalid npm names. Verified via `dotnet test src/StellaOps.Concelier.Connector.Osv.Tests`, `src/StellaOps.Concelier.Connector.Ghsa.Tests`, `src/StellaOps.Concelier.Connector.Nvd.Tests`, and backbone normalization/storage suites. | +| Sprint 2 | Connector & Data Implementation Wave | src/StellaOps.Concelier.Connector.Acsc/TASKS.md | DONE (2025-10-12) | Team Connector Expansion – Regional & Vendor Feeds | FEEDCONN-ACSC-02-001 … 02-008 | Fetch→parse→map pipeline, fixtures, diagnostics, and README finished 2025-10-12; downstream export parity captured via FEEDEXPORT-JSON-04-001 / FEEDEXPORT-TRIVY-04-001 (completed). | +| Sprint 2 | Connector & Data Implementation Wave | src/StellaOps.Concelier.Connector.Cccs/TASKS.md | DONE (2025-10-16) | Team Connector Expansion – Regional & Vendor Feeds | FEEDCONN-CCCS-02-001 … 02-008 | Observability meter, historical harvest plan, and DOM sanitizer refinements wrapped; ops notes live under `docs/ops/concelier-cccs-operations.md` with fixtures validating EN/FR list handling. | +| Sprint 2 | Connector & Data Implementation Wave | src/StellaOps.Concelier.Connector.CertBund/TASKS.md | DONE (2025-10-15) | Team Connector Expansion – Regional & Vendor Feeds | FEEDCONN-CERTBUND-02-001 … 02-008 | Telemetry/docs (02-006) and history/locale sweep (02-007) completed alongside pipeline; runbook `docs/ops/concelier-certbund-operations.md` captures locale guidance and offline packaging. | +| Sprint 2 | Connector & Data Implementation Wave | src/StellaOps.Concelier.Connector.Kisa/TASKS.md | DONE (2025-10-14) | Team Connector Expansion – Regional & Vendor Feeds | FEEDCONN-KISA-02-001 … 02-007 | Connector, tests, and telemetry/docs (02-006) finalized; localisation notes in `docs/dev/kisa_connector_notes.md` complete rollout. | +| Sprint 2 | Connector & Data Implementation Wave | src/StellaOps.Concelier.Connector.Ru.Bdu/TASKS.md | DONE (2025-10-14) | Team Connector Expansion – Regional & Vendor Feeds | FEEDCONN-RUBDU-02-001 … 02-008 | Fetch/parser/mapper refinements, regression fixtures, telemetry/docs, access options, and trusted root packaging all landed; README documents offline access strategy. | +| Sprint 2 | Connector & Data Implementation Wave | src/StellaOps.Concelier.Connector.Ru.Nkcki/TASKS.md | DONE (2025-10-13) | Team Connector Expansion – Regional & Vendor Feeds | FEEDCONN-NKCKI-02-001 … 02-008 | Listing fetch, parser, mapper, fixtures, telemetry/docs, and archive plan finished; Mongo2Go/libcrypto dependency resolved via bundled OpenSSL noted in ops guide. | +| Sprint 2 | Connector & Data Implementation Wave | src/StellaOps.Concelier.Connector.Ics.Cisa/TASKS.md | DONE (2025-10-16) | Team Connector Expansion – Regional & Vendor Feeds | FEEDCONN-ICSCISA-02-001 … 02-011 | Feed parser attachment fixes, SemVer exact values, regression suites, telemetry/docs updates, and handover complete; ops runbook now details attachment verification + proxy usage. | +| Sprint 2 | Connector & Data Implementation Wave | src/StellaOps.Concelier.Connector.Vndr.Cisco/TASKS.md | DONE (2025-10-14) | Team Connector Expansion – Regional & Vendor Feeds | FEEDCONN-CISCO-02-001 … 02-007 | OAuth fetch pipeline, DTO/mapping, tests, and telemetry/docs shipped; monitoring/export integration follow-ups recorded in Ops docs and exporter backlog (completed). | +| Sprint 2 | Connector & Data Implementation Wave | src/StellaOps.Concelier.Connector.Vndr.Msrc/TASKS.md | DONE (2025-10-15) | Team Connector Expansion – Regional & Vendor Feeds | FEEDCONN-MSRC-02-001 … 02-008 | Azure AD onboarding (02-008) unblocked fetch/parse/map pipeline; fixtures, telemetry/docs, and Offline Kit guidance published in `docs/ops/concelier-msrc-operations.md`. | +| Sprint 2 | Connector & Data Implementation Wave | src/StellaOps.Concelier.Connector.Cve/TASKS.md | DONE (2025-10-15) | Team Connector Support & Monitoring | FEEDCONN-CVE-02-001 … 02-002 | CVE data-source selection, fetch pipeline, and docs landed 2025-10-10. 2025-10-15: smoke verified using the seeded mirror fallback; connector now logs a warning and pulls from `seed-data/cve/` until live CVE Services credentials arrive. | +| Sprint 2 | Connector & Data Implementation Wave | src/StellaOps.Concelier.Connector.Kev/TASKS.md | DONE (2025-10-12) | Team Connector Support & Monitoring | FEEDCONN-KEV-02-001 … 02-002 | KEV catalog ingestion, fixtures, telemetry, and schema validation completed 2025-10-12; ops dashboard published. | +| Sprint 2 | Connector & Data Implementation Wave | docs/TASKS.md | DONE (2025-10-11) | Team Docs & Knowledge Base | FEEDDOCS-DOCS-01-001 | Canonical schema docs refresh
Updated canonical schema + provenance guides with SemVer style, normalized version rules, decision reason change log, and migration notes. | +| Sprint 2 | Connector & Data Implementation Wave | docs/TASKS.md | DONE (2025-10-11) | Team Docs & Knowledge Base | FEEDDOCS-DOCS-02-001 | Concelier-SemVer Playbook
Published merge playbook covering mapper patterns, dedupe flow, indexes, and rollout checklist. | +| Sprint 2 | Connector & Data Implementation Wave | docs/TASKS.md | DONE (2025-10-11) | Team Docs & Knowledge Base | FEEDDOCS-DOCS-02-002 | Normalized versions query guide
Delivered Mongo index/query addendum with `$unwind` recipes, dedupe checks, and operational checklist.
Instructions to work:
DONE Read ./AGENTS.md and docs/AGENTS.md. Document every schema/index/query change produced in Sprint 1-2 leveraging ./src/FASTER_MODELING_AND_NORMALIZATION.md. | +| Sprint 3 | Conflict Resolution Integration & Communications | src/StellaOps.Concelier.Core/TASKS.md | DONE (2025-10-11) | Team Core Engine & Storage Analytics | FEEDCORE-ENGINE-03-001 | Canonical merger implementation
`CanonicalMerger` ships with freshness/tie-breaker logic, provenance, and unit coverage feeding Merge. | +| Sprint 3 | Conflict Resolution Integration & Communications | src/StellaOps.Concelier.Core/TASKS.md | DONE (2025-10-11) | Team Core Engine & Storage Analytics | FEEDCORE-ENGINE-03-002 | Field precedence and tie-breaker map
Field precedence tables and tie-breaker metrics wired into the canonical merge flow; docs/tests updated.
Instructions to work:
Read ./AGENTS.md and core AGENTS. Implement the conflict resolver exactly as specified in ./src/DEDUP_CONFLICTS_RESOLUTION_ALGO.md, coordinating with Merge and Storage teammates. | +| Sprint 3 | Conflict Resolution Integration & Communications | src/StellaOps.Concelier.Storage.Mongo/TASKS.md | DONE (2025-10-11) | Team Core Engine & Storage Analytics | FEEDSTORAGE-DATA-03-001 | Merge event provenance audit prep
Merge events now persist `fieldDecisions` and analytics-ready provenance snapshots. | +| Sprint 3 | Conflict Resolution Integration & Communications | src/StellaOps.Concelier.Storage.Mongo/TASKS.md | DONE (2025-10-11) | Team Core Engine & Storage Analytics | FEEDSTORAGE-DATA-02-001 | Normalized range dual-write + backfill
Dual-write/backfill flag delivered; migration + options validated in tests. | +| Sprint 3 | Conflict Resolution Integration & Communications | src/StellaOps.Concelier.Storage.Mongo/TASKS.md | DONE (2025-10-11) | Team Core Engine & Storage Analytics | FEEDSTORAGE-TESTS-02-004 | Restore AdvisoryStore build after normalized versions refactor
Storage tests adjusted for normalized versions/decision reasons.
Instructions to work:
Read ./AGENTS.md and storage AGENTS. Extend merge events with decision reasons and analytics views to support the conflict rules, and deliver the dual-write/backfill for `NormalizedVersions` + `decisionReason` so connectors can roll out safely. | +| Sprint 3 | Conflict Resolution Integration & Communications | src/StellaOps.Concelier.Merge/TASKS.md | DONE (2025-10-11) | Team Merge & QA Enforcement | FEEDMERGE-ENGINE-04-001 | GHSA/NVD/OSV conflict rules
Merge pipeline consumes `CanonicalMerger` output prior to precedence merge. | +| Sprint 3 | Conflict Resolution Integration & Communications | src/StellaOps.Concelier.Merge/TASKS.md | DONE (2025-10-11) | Team Merge & QA Enforcement | FEEDMERGE-ENGINE-04-002 | Override metrics instrumentation
Merge events capture per-field decisions; counters/logs align with conflict rules. | +| Sprint 3 | Conflict Resolution Integration & Communications | src/StellaOps.Concelier.Merge/TASKS.md | DONE (2025-10-11) | Team Merge & QA Enforcement | FEEDMERGE-ENGINE-04-003 | Reference & credit union pipeline
Canonical merge preserves unions with updated tests. | +| Sprint 3 | Conflict Resolution Integration & Communications | src/StellaOps.Concelier.Merge/TASKS.md | DONE (2025-10-11) | Team Merge & QA Enforcement | FEEDMERGE-QA-04-001 | End-to-end conflict regression suite
Added regression tests (`AdvisoryMergeServiceTests`) covering canonical + precedence flow.
Instructions to work:
Read ./AGENTS.md and merge AGENTS. Integrate the canonical merger, instrument metrics, and deliver comprehensive regression tests following ./src/DEDUP_CONFLICTS_RESOLUTION_ALGO.md. | +| Sprint 3 | Conflict Resolution Integration & Communications | src/StellaOps.Concelier.Connector.Ghsa/TASKS.md | DONE (2025-10-12) | Team Connector Regression Fixtures | FEEDCONN-GHSA-04-002 | GHSA conflict regression fixtures | +| Sprint 3 | Conflict Resolution Integration & Communications | src/StellaOps.Concelier.Connector.Nvd/TASKS.md | DONE (2025-10-12) | Team Connector Regression Fixtures | FEEDCONN-NVD-04-002 | NVD conflict regression fixtures | +| Sprint 3 | Conflict Resolution Integration & Communications | src/StellaOps.Concelier.Connector.Osv/TASKS.md | DONE (2025-10-12) | Team Connector Regression Fixtures | FEEDCONN-OSV-04-002 | OSV conflict regression fixtures
Instructions to work:
Read ./AGENTS.md and module AGENTS. Produce fixture triples supporting the precedence/tie-breaker paths defined in ./src/DEDUP_CONFLICTS_RESOLUTION_ALGO.md and hand them to Merge QA. | +| Sprint 3 | Conflict Resolution Integration & Communications | docs/TASKS.md | DONE (2025-10-11) | Team Documentation Guild – Conflict Guidance | FEEDDOCS-DOCS-05-001 | Concelier Conflict Rules
Runbook published at `docs/ops/concelier-conflict-resolution.md`; metrics/log guidance aligned with Sprint 3 merge counters. | +| Sprint 3 | Conflict Resolution Integration & Communications | docs/TASKS.md | DONE (2025-10-16) | Team Documentation Guild – Conflict Guidance | FEEDDOCS-DOCS-05-002 | Conflict runbook ops rollout
Ops review completed, alert thresholds applied, and change log appended in `docs/ops/concelier-conflict-resolution.md`; task closed after connector signals verified. | +| Sprint 4 | Schema Parity & Freshness Alignment | src/StellaOps.Concelier.Models/TASKS.md | DONE (2025-10-15) | Team Models & Merge Leads | FEEDMODELS-SCHEMA-04-001 | Advisory schema parity (description/CWE/canonical metric)
Extend `Advisory` and related records with description text, CWE collection, and canonical metric pointer; refresh validation + serializer determinism tests. | +| Sprint 4 | Schema Parity & Freshness Alignment | src/StellaOps.Concelier.Core/TASKS.md | DONE (2025-10-15) | Team Core Engine & Storage Analytics | FEEDCORE-ENGINE-04-003 | Canonical merger parity for new fields
Teach `CanonicalMerger` to populate description, CWEResults, and canonical metric pointer with provenance + regression coverage. | +| Sprint 4 | Schema Parity & Freshness Alignment | src/StellaOps.Concelier.Core/TASKS.md | DONE (2025-10-15) | Team Core Engine & Storage Analytics | FEEDCORE-ENGINE-04-004 | Reference normalization & freshness instrumentation cleanup
Implement URL normalization for reference dedupe, align freshness-sensitive instrumentation, and add analytics tests. | +| Sprint 4 | Schema Parity & Freshness Alignment | src/StellaOps.Concelier.Merge/TASKS.md | DONE (2025-10-15) | Team Merge & QA Enforcement | FEEDMERGE-ENGINE-04-004 | Merge pipeline parity for new advisory fields
Ensure merge service + merge events surface description/CWE/canonical metric decisions with updated metrics/tests. | +| Sprint 4 | Schema Parity & Freshness Alignment | src/StellaOps.Concelier.Merge/TASKS.md | DONE (2025-10-15) | Team Merge & QA Enforcement | FEEDMERGE-ENGINE-04-005 | Connector coordination for new advisory fields
GHSA/NVD/OSV connectors now ship description, CWE, and canonical metric data with refreshed fixtures; merge coordination log updated and exporters notified. | +| Sprint 4 | Schema Parity & Freshness Alignment | src/StellaOps.Concelier.Exporter.Json/TASKS.md | DONE (2025-10-15) | Team Exporters – JSON | FEEDEXPORT-JSON-04-001 | Surface new advisory fields in JSON exporter
Update schemas/offline bundle + fixtures once model/core parity lands.
2025-10-15: `dotnet test src/StellaOps.Concelier.Exporter.Json.Tests` validated canonical metric/CWE emission. | +| Sprint 4 | Schema Parity & Freshness Alignment | src/StellaOps.Concelier.Exporter.TrivyDb/TASKS.md | DONE (2025-10-15) | Team Exporters – Trivy DB | FEEDEXPORT-TRIVY-04-001 | Propagate new advisory fields into Trivy DB package
Extend Bolt builder, metadata, and regression tests for the expanded schema.
2025-10-15: `dotnet test src/StellaOps.Concelier.Exporter.TrivyDb.Tests` confirmed canonical metric/CWE propagation. | +| Sprint 4 | Schema Parity & Freshness Alignment | src/StellaOps.Concelier.Connector.Ghsa/TASKS.md | DONE (2025-10-16) | Team Connector Regression Fixtures | FEEDCONN-GHSA-04-004 | Harden CVSS fallback so canonical metric ids persist when GitHub omits vectors; extend fixtures and document severity precedence hand-off to Merge. | +| Sprint 4 | Schema Parity & Freshness Alignment | src/StellaOps.Concelier.Connector.Osv/TASKS.md | DONE (2025-10-16) | Team Connector Expansion – GHSA/NVD/OSV | FEEDCONN-OSV-04-005 | Map OSV advisories lacking CVSS vectors to canonical metric ids/notes and document CWE provenance quirks; schedule parity fixture updates. | +| Sprint 5 | Excititor Core Foundations | src/StellaOps.Excititor.Core/TASKS.md | DONE (2025-10-15) | Team Excititor Core & Policy | EXCITITOR-CORE-01-001 | Stand up canonical VEX claim/consensus records with deterministic serializers so Storage/Exports share a stable contract. | +| Sprint 5 | Excititor Core Foundations | src/StellaOps.Excititor.Core/TASKS.md | DONE (2025-10-15) | Team Excititor Core & Policy | EXCITITOR-CORE-01-002 | Implement trust-weighted consensus resolver with baseline policy weights, justification gates, telemetry output, and majority/tie handling. | +| Sprint 5 | Excititor Core Foundations | src/StellaOps.Excititor.Core/TASKS.md | DONE (2025-10-15) | Team Excititor Core & Policy | EXCITITOR-CORE-01-003 | Publish shared connector/exporter/attestation abstractions and deterministic query signature utilities for cache/attestation workflows. | +| Sprint 5 | Excititor Core Foundations | src/StellaOps.Excititor.Policy/TASKS.md | DONE (2025-10-15) | Team Excititor Policy | EXCITITOR-POLICY-01-001 | Established policy options & snapshot provider covering baseline weights/overrides. | +| Sprint 5 | Excititor Core Foundations | src/StellaOps.Excititor.Policy/TASKS.md | DONE (2025-10-15) | Team Excititor Policy | EXCITITOR-POLICY-01-002 | Policy evaluator now feeds consensus resolver with immutable snapshots. | +| Sprint 5 | Excititor Core Foundations | src/StellaOps.Excititor.Policy/TASKS.md | DONE (2025-10-16) | Team Excititor Policy | EXCITITOR-POLICY-01-003 | Author policy diagnostics, CLI/WebService surfacing, and documentation updates. | +| Sprint 5 | Excititor Core Foundations | src/StellaOps.Excititor.Policy/TASKS.md | DONE (2025-10-16) | Team Excititor Policy | EXCITITOR-POLICY-01-004 | Implement YAML/JSON schema validation and deterministic diagnostics for operator bundles. | +| Sprint 5 | Excititor Core Foundations | src/StellaOps.Excititor.Policy/TASKS.md | DONE (2025-10-16) | Team Excititor Policy | EXCITITOR-POLICY-01-005 | Add policy change tracking, snapshot digests, and telemetry/logging hooks. | +| Sprint 5 | Excititor Core Foundations | src/StellaOps.Excititor.Storage.Mongo/TASKS.md | DONE (2025-10-15) | Team Excititor Storage | EXCITITOR-STORAGE-01-001 | Mongo mapping registry plus raw/export entities and DI extensions in place. | +| Sprint 5 | Excititor Core Foundations | src/StellaOps.Excititor.Storage.Mongo/TASKS.md | DONE (2025-10-16) | Team Excititor Storage | EXCITITOR-STORAGE-01-004 | Build provider/consensus/cache class maps and related collections. | +| Sprint 5 | Excititor Core Foundations | src/StellaOps.Excititor.Export/TASKS.md | DONE (2025-10-15) | Team Excititor Export | EXCITITOR-EXPORT-01-001 | Export engine delivers cache lookup, manifest creation, and policy integration. | +| Sprint 5 | Excititor Core Foundations | src/StellaOps.Excititor.Export/TASKS.md | DONE (2025-10-17) | Team Excititor Export | EXCITITOR-EXPORT-01-004 | Connect export engine to attestation client and persist Rekor metadata. | +| Sprint 5 | Excititor Core Foundations | src/StellaOps.Excititor.Attestation/TASKS.md | DONE (2025-10-16) | Team Excititor Attestation | EXCITITOR-ATTEST-01-001 | Implement in-toto predicate + DSSE builder providing envelopes for export attestation. | +| Sprint 5 | Excititor Core Foundations | src/StellaOps.Excititor.Connectors.Abstractions/TASKS.md | DONE (2025-10-17) | Team Excititor Connectors | EXCITITOR-CONN-ABS-01-001 | Deliver shared connector context/base classes so provider plug-ins can be activated via WebService/Worker. | +| Sprint 5 | Excititor Core Foundations | src/StellaOps.Excititor.WebService/TASKS.md | DONE (2025-10-17) | Team Excititor WebService | EXCITITOR-WEB-01-001 | Scaffold minimal API host, DI, and `/excititor/status` endpoint integrating policy, storage, export, and attestation services. | +| Sprint 6 | Excititor Ingest & Formats | src/StellaOps.Excititor.Worker/TASKS.md | DONE (2025-10-17) | Team Excititor Worker | EXCITITOR-WORKER-01-001 | Create Worker host with provider scheduling and logging to drive recurring pulls/reconciliation. | +| Sprint 6 | Excititor Ingest & Formats | src/StellaOps.Excititor.Formats.CSAF/TASKS.md | DONE (2025-10-17) | Team Excititor Formats | EXCITITOR-FMT-CSAF-01-001 | Implement CSAF normalizer foundation translating provider documents into `VexClaim` entries. | +| Sprint 6 | Excititor Ingest & Formats | src/StellaOps.Excititor.Formats.CycloneDX/TASKS.md | DONE (2025-10-17) | Team Excititor Formats | EXCITITOR-FMT-CYCLONE-01-001 | Implement CycloneDX VEX normalizer capturing `analysis` state and component references. | +| Sprint 6 | Excititor Ingest & Formats | src/StellaOps.Excititor.Formats.OpenVEX/TASKS.md | DONE (2025-10-17) | Team Excititor Formats | EXCITITOR-FMT-OPENVEX-01-001 | Implement OpenVEX normalizer to ingest attestations into canonical claims with provenance. | +| Sprint 6 | Excititor Ingest & Formats | src/StellaOps.Excititor.Connectors.RedHat.CSAF/TASKS.md | DONE (2025-10-17) | Team Excititor Connectors – Red Hat | EXCITITOR-CONN-RH-01-001 | Ship Red Hat CSAF provider metadata discovery enabling incremental pulls. | +| Sprint 6 | Excititor Ingest & Formats | src/StellaOps.Excititor.Connectors.RedHat.CSAF/TASKS.md | DONE (2025-10-17) | Team Excititor Connectors – Red Hat | EXCITITOR-CONN-RH-01-002 | Fetch CSAF windows with ETag handling, resume tokens, quarantine on schema errors, and persist raw docs. | +| Sprint 6 | Excititor Ingest & Formats | src/StellaOps.Excititor.Connectors.RedHat.CSAF/TASKS.md | DONE (2025-10-17) | Team Excititor Connectors – Red Hat | EXCITITOR-CONN-RH-01-003 | Populate provider trust overrides (cosign issuer, identity regex) and provenance hints for policy evaluation/logging. | +| Sprint 6 | Excititor Ingest & Formats | src/StellaOps.Excititor.Connectors.RedHat.CSAF/TASKS.md | DONE (2025-10-17) | Team Excititor Connectors – Red Hat | EXCITITOR-CONN-RH-01-004 | Persist resume cursors (last updated timestamp/document hashes) in storage and reload during fetch to avoid duplicates. | +| Sprint 6 | Excititor Ingest & Formats | src/StellaOps.Excititor.Connectors.RedHat.CSAF/TASKS.md | DONE (2025-10-17) | Team Excititor Connectors – Red Hat | EXCITITOR-CONN-RH-01-005 | Register connector in Worker/WebService DI, add scheduled jobs, and document CLI triggers for Red Hat CSAF pulls. | +| Sprint 6 | Excititor Ingest & Formats | src/StellaOps.Excititor.Connectors.RedHat.CSAF/TASKS.md | DONE (2025-10-17) | Team Excititor Connectors – Red Hat | EXCITITOR-CONN-RH-01-006 | Add CSAF normalization parity fixtures ensuring RHSA-specific metadata is preserved. | +| Sprint 6 | Excititor Ingest & Formats | src/StellaOps.Excititor.Connectors.Cisco.CSAF/TASKS.md | DONE (2025-10-17) | Team Excititor Connectors – Cisco | EXCITITOR-CONN-CISCO-01-001 | Implement Cisco CSAF endpoint discovery/auth to unlock paginated pulls. | +| Sprint 6 | Excititor Ingest & Formats | src/StellaOps.Excititor.Connectors.Cisco.CSAF/TASKS.md | DONE (2025-10-17) | Team Excititor Connectors – Cisco | EXCITITOR-CONN-CISCO-01-002 | Implement Cisco CSAF paginated fetch loop with dedupe and raw persistence support. | +| Sprint 6 | Excititor Ingest & Formats | src/StellaOps.Excititor.Connectors.SUSE.RancherVEXHub/TASKS.md | DONE (2025-10-17) | Team Excititor Connectors – SUSE | EXCITITOR-CONN-SUSE-01-001 | Build Rancher VEX Hub discovery/subscription path with offline snapshot support. | +| Sprint 6 | Excititor Ingest & Formats | src/StellaOps.Excititor.Connectors.MSRC.CSAF/TASKS.md | DONE (2025-10-17) | Team Excititor Connectors – MSRC | EXCITITOR-CONN-MS-01-001 | Deliver AAD onboarding/token cache for MSRC CSAF ingestion. | +| Sprint 6 | Excititor Ingest & Formats | src/StellaOps.Excititor.Connectors.Oracle.CSAF/TASKS.md | DONE (2025-10-17) | Team Excititor Connectors – Oracle | EXCITITOR-CONN-ORACLE-01-001 | Implement Oracle CSAF catalogue discovery with CPU calendar awareness. | +| Sprint 6 | Excititor Ingest & Formats | src/StellaOps.Excititor.Connectors.Ubuntu.CSAF/TASKS.md | DONE (2025-10-17) | Team Excititor Connectors – Ubuntu | EXCITITOR-CONN-UBUNTU-01-001 | Implement Ubuntu CSAF discovery and channel selection for USN ingestion. | +| Sprint 6 | Excititor Ingest & Formats | src/StellaOps.Excititor.Connectors.OCI.OpenVEX.Attest/TASKS.md | DONE (2025-10-18) | Team Excititor Connectors – OCI | EXCITITOR-CONN-OCI-01-001 | Wire OCI discovery/auth to fetch OpenVEX attestations for configured images. | +| Sprint 6 | Excititor Ingest & Formats | src/StellaOps.Excititor.Connectors.OCI.OpenVEX.Attest/TASKS.md | DONE (2025-10-18) | Team Excititor Connectors – OCI | EXCITITOR-CONN-OCI-01-002 | Attestation fetch & verify loop – download DSSE attestations, trigger verification, handle retries/backoff, persist raw statements. | +| Sprint 6 | Excititor Ingest & Formats | src/StellaOps.Excititor.Connectors.OCI.OpenVEX.Attest/TASKS.md | DONE (2025-10-18) | Team Excititor Connectors – OCI | EXCITITOR-CONN-OCI-01-003 | Provenance metadata & policy hooks – emit image, subject digest, issuer, and trust metadata for policy weighting/logging. | +| Sprint 6 | Excititor Ingest & Formats | src/StellaOps.Cli/TASKS.md | DONE (2025-10-18) | DevEx/CLI | EXCITITOR-CLI-01-001 | Add `excititor` CLI verbs bridging to WebService with consistent auth and offline UX. | | Sprint 7 | Contextual Truth Foundations | src/StellaOps.Excititor.Core/TASKS.md | DONE (2025-10-19) | Team Excititor Core & Policy | EXCITITOR-CORE-02-001 | Context signal schema prep – extend consensus models with severity/KEV/EPSS fields and update canonical serializers. | | Sprint 7 | Contextual Truth Foundations | src/StellaOps.Excititor.Policy/TASKS.md | DONE (2025-10-19) | Team Excititor Policy | EXCITITOR-POLICY-02-001 | Scoring coefficients & weight ceilings – add α/β options, weight boosts, and validation guidance. | -| Sprint 7 | Contextual Truth Foundations | src/StellaOps.Excititor.Storage.Mongo/TASKS.md | TODO | Team Excititor Storage | EXCITITOR-STORAGE-02-001 | Statement events & scoring signals – create immutable VEX statement store plus consensus extensions with indexes/migrations. | -| Sprint 7 | Contextual Truth Foundations | src/StellaOps.Excititor.WebService/TASKS.md | TODO | Team Excititor WebService | EXCITITOR-WEB-01-004 | Resolve API & signed responses – expose `/excititor/resolve`, return signed consensus/score envelopes, document auth. | -| Sprint 7 | Contextual Truth Foundations | src/StellaOps.Excititor.WebService/TASKS.md | TODO | Team Excititor WebService | EXCITITOR-WEB-01-005 | Mirror distribution endpoints – expose download APIs for downstream Excititor instances. | -| Sprint 7 | Contextual Truth Foundations | src/StellaOps.Excititor.Attestation/TASKS.md | DONE (2025-10-16) | Team Excititor Attestation | EXCITITOR-ATTEST-01-002 | Rekor v2 client integration – ship transparency log client with retries and offline queue. | -| Sprint 7 | Contextual Truth Foundations | src/StellaOps.Excititor.Worker/TASKS.md | TODO | Team Excititor Worker | EXCITITOR-WORKER-01-004 | TTL refresh & stability damper – schedule re-resolve loops and guard against status flapping. | -| Sprint 7 | Contextual Truth Foundations | src/StellaOps.Excititor.Export/TASKS.md | TODO | Team Excititor Export | EXCITITOR-EXPORT-01-005 | Score & resolve envelope surfaces – include signed consensus/score artifacts in exports. | -| Sprint 7 | Contextual Truth Foundations | src/StellaOps.Excititor.Export/TASKS.md | TODO | Team Excititor Export | EXCITITOR-EXPORT-01-006 | Quiet provenance packaging – attach quieted-by statement IDs, signers, justification codes to exports and attestations. | -| Sprint 7 | Contextual Truth Foundations | src/StellaOps.Excititor.Export/TASKS.md | TODO | Team Excititor Export | EXCITITOR-EXPORT-01-007 | Mirror bundle + domain manifest – publish signed consensus bundles for mirrors. | -| Sprint 7 | Contextual Truth Foundations | src/StellaOps.Excititor.Connectors.StellaOpsMirror/TASKS.md | TODO | Excititor Connectors – Stella | EXCITITOR-CONN-STELLA-07-001 | Excititor mirror connector – ingest signed mirror bundles and map to VexClaims with resume handling. | -| Sprint 7 | Contextual Truth Foundations | src/StellaOps.Concelier.Core/TASKS.md | TODO | Team Core Engine & Storage Analytics | FEEDCORE-ENGINE-07-001 | Advisory event log & asOf queries – surface immutable statements and replay capability. | -| Sprint 7 | Contextual Truth Foundations | src/StellaOps.Concelier.Core/TASKS.md | TODO | Team Core Engine & Data Science | FEEDCORE-ENGINE-07-002 | Noise prior computation service – learn false-positive priors and expose deterministic summaries. | -| Sprint 7 | Contextual Truth Foundations | src/StellaOps.Concelier.Core/TASKS.md | TODO | Team Core Engine & Storage Analytics | FEEDCORE-ENGINE-07-003 | Unknown state ledger & confidence seeding – persist unknown flags, seed confidence bands, expose query surface. | -| Sprint 7 | Contextual Truth Foundations | src/StellaOps.Concelier.Storage.Mongo/TASKS.md | TODO | Team Normalization & Storage Backbone | FEEDSTORAGE-DATA-07-001 | Advisory statement & conflict collections – provision Mongo schema/indexes for event-sourced merge. | -| Sprint 7 | Contextual Truth Foundations | src/StellaOps.Concelier.Merge/TASKS.md | TODO | BE-Merge | FEEDMERGE-ENGINE-07-001 | Conflict sets & explainers – persist conflict materialization and replay hashes for merge decisions. | -| Sprint 8 | Mongo strengthening | src/StellaOps.Concelier.Storage.Mongo/TASKS.md | TODO | Team Normalization & Storage Backbone | FEEDSTORAGE-MONGO-08-001 | Causal-consistent Concelier storage sessions
Ensure `AddMongoStorage` registers a scoped session facilitator (causal consistency + majority concerns), update repositories to accept optional session handles, and add integration coverage proving read-your-write and monotonic reads across a replica set/election scenario. | -| Sprint 8 | Mongo strengthening | src/StellaOps.Authority/TASKS.md | TODO | Authority Core & Storage Guild | AUTHSTORAGE-MONGO-08-001 | Harden Authority Mongo usage
Introduce scoped MongoDB sessions with `writeConcern`/`readConcern` majority defaults, flow the session through stores used in mutations + follow-up reads, and document middleware pattern for web/API & GraphQL layers. | -| Sprint 8 | Mongo strengthening | src/StellaOps.Excititor.Storage.Mongo/TASKS.md | TODO | Team Excititor Storage | EXCITITOR-STORAGE-MONGO-08-001 | Causal consistency for Excititor repositories
Register Mongo options with majority defaults, push session-aware overloads through raw/export/consensus/cache stores, and extend migration/tests to validate causal reads after writes (including GridFS-backed content) under replica-set failover. | -| Sprint 8 | Mirror Distribution | src/StellaOps.Concelier.Exporter.Json/TASKS.md | TODO | Concelier Export Guild | CONCELIER-EXPORT-08-201 | Mirror bundle + domain manifest – produce signed JSON aggregates for `*.stella-ops.org` mirrors. | -| Sprint 8 | Mirror Distribution | src/StellaOps.Concelier.Exporter.TrivyDb/TASKS.md | TODO | Concelier Export Guild | CONCELIER-EXPORT-08-202 | Mirror-ready Trivy DB bundles – ship domain-specific archives + metadata for downstream sync. | -| Sprint 8 | Mirror Distribution | src/StellaOps.Concelier.WebService/TASKS.md | TODO | Concelier WebService Guild | CONCELIER-WEB-08-201 | Mirror distribution endpoints – expose domain-scoped index/download APIs with auth/quota. | -| Sprint 8 | Mirror Distribution | src/StellaOps.Concelier.Connector.StellaOpsMirror/TASKS.md | TODO | BE-Conn-Stella | FEEDCONN-STELLA-08-001 | Concelier mirror connector – fetch mirror manifest, verify signatures, and hydrate canonical DTOs with resume support. | -| Sprint 8 | Mirror Distribution | ops/devops/TASKS.md | TODO | DevOps Guild | DEVOPS-MIRROR-08-001 | Managed mirror deployments for `*.stella-ops.org` – Helm/Compose overlays, CDN, runbooks. | -| Sprint 9 | Scanner Core Foundations | src/StellaOps.Scanner.Core/TASKS.md | DONE (2025-10-18) | Team Scanner Core | SCANNER-CORE-09-501 | Define shared DTOs (ScanJob, ProgressEvent), error taxonomy, and deterministic ID/timestamp helpers aligning with `ARCHITECTURE_SCANNER.md` §3–§4. | -| Sprint 9 | Scanner Core Foundations | src/StellaOps.Scanner.Core/TASKS.md | DONE (2025-10-18) | Team Scanner Core | SCANNER-CORE-09-502 | Observability helpers (correlation IDs, logging scopes, metric namespacing, deterministic hashes) consumed by WebService/Worker. | -| Sprint 9 | Scanner Core Foundations | src/StellaOps.Scanner.Core/TASKS.md | DONE (2025-10-18) | Team Scanner Core | SCANNER-CORE-09-503 | Security utilities: Authority client factory, OpTok caching, DPoP verifier, restart-time plug-in guardrails for scanner components. | +| Sprint 7 | Contextual Truth Foundations | src/StellaOps.Excititor.Storage.Mongo/TASKS.md | DONE (2025-10-19) | Team Excititor Storage | EXCITITOR-STORAGE-02-001 | Statement events & scoring signals – immutable VEX statements store, consensus signal fields, and migration `20251019-consensus-signals-statements` with tests (`dotnet test src/StellaOps.Excititor.Core.Tests/StellaOps.Excititor.Core.Tests.csproj`, `dotnet test src/StellaOps.Excititor.Storage.Mongo.Tests/StellaOps.Excititor.Storage.Mongo.Tests.csproj`). | +| Sprint 7 | Contextual Truth Foundations | src/StellaOps.Excititor.WebService/TASKS.md | TODO | Team Excititor WebService | EXCITITOR-WEB-01-004 | Resolve API & signed responses – expose `/excititor/resolve`, return signed consensus/score envelopes, document auth. | +| Sprint 7 | Contextual Truth Foundations | src/StellaOps.Excititor.WebService/TASKS.md | TODO | Team Excititor WebService | EXCITITOR-WEB-01-005 | Mirror distribution endpoints – expose download APIs for downstream Excititor instances. | +| Sprint 7 | Contextual Truth Foundations | src/StellaOps.Excititor.Attestation/TASKS.md | DONE (2025-10-16) | Team Excititor Attestation | EXCITITOR-ATTEST-01-002 | Rekor v2 client integration – ship transparency log client with retries and offline queue. | +| Sprint 7 | Contextual Truth Foundations | src/StellaOps.Excititor.Worker/TASKS.md | TODO | Team Excititor Worker | EXCITITOR-WORKER-01-004 | TTL refresh & stability damper – schedule re-resolve loops and guard against status flapping. | +| Sprint 7 | Contextual Truth Foundations | src/StellaOps.Excititor.Export/TASKS.md | TODO | Team Excititor Export | EXCITITOR-EXPORT-01-005 | Score & resolve envelope surfaces – include signed consensus/score artifacts in exports. | +| Sprint 7 | Contextual Truth Foundations | src/StellaOps.Excititor.Export/TASKS.md | TODO | Team Excititor Export | EXCITITOR-EXPORT-01-006 | Quiet provenance packaging – attach quieted-by statement IDs, signers, justification codes to exports and attestations. | +| Sprint 7 | Contextual Truth Foundations | src/StellaOps.Excititor.Export/TASKS.md | TODO | Team Excititor Export | EXCITITOR-EXPORT-01-007 | Mirror bundle + domain manifest – publish signed consensus bundles for mirrors. | +| Sprint 7 | Contextual Truth Foundations | src/StellaOps.Excititor.Connectors.StellaOpsMirror/TASKS.md | TODO | Excititor Connectors – Stella | EXCITITOR-CONN-STELLA-07-001 | Excititor mirror connector – ingest signed mirror bundles and map to VexClaims with resume handling. | +| Sprint 7 | Contextual Truth Foundations | src/StellaOps.Concelier.Core/TASKS.md | DONE (2025-10-19) | Team Core Engine & Storage Analytics | FEEDCORE-ENGINE-07-001 | Advisory event log & asOf queries – surface immutable statements and replay capability. | +| Sprint 7 | Contextual Truth Foundations | src/StellaOps.Concelier.WebService/TASKS.md | DONE (2025-10-19) | Concelier WebService Guild | FEEDWEB-EVENTS-07-001 | Advisory event replay API – expose `/concelier/advisories/{key}/replay` with `asOf` filter, hex hashes, and conflict data. | +| Sprint 7 | Contextual Truth Foundations | src/StellaOps.Concelier.Core/TASKS.md | TODO | Team Core Engine & Data Science | FEEDCORE-ENGINE-07-002 | Noise prior computation service – learn false-positive priors and expose deterministic summaries. | +| Sprint 7 | Contextual Truth Foundations | src/StellaOps.Concelier.Core/TASKS.md | TODO | Team Core Engine & Storage Analytics | FEEDCORE-ENGINE-07-003 | Unknown state ledger & confidence seeding – persist unknown flags, seed confidence bands, expose query surface. | +| Sprint 7 | Contextual Truth Foundations | src/StellaOps.Concelier.Storage.Mongo/TASKS.md | TODO | Team Normalization & Storage Backbone | FEEDSTORAGE-DATA-07-001 | Advisory statement & conflict collections – provision Mongo schema/indexes for event-sourced merge. | +| Sprint 7 | Contextual Truth Foundations | src/StellaOps.Concelier.Merge/TASKS.md | DOING | BE-Merge | FEEDMERGE-ENGINE-07-001 | Conflict sets & explainers – persist conflict materialization and replay hashes for merge decisions. | +| Sprint 8 | Mongo strengthening | src/StellaOps.Concelier.Storage.Mongo/TASKS.md | DONE (2025-10-19) | Team Normalization & Storage Backbone | FEEDSTORAGE-MONGO-08-001 | Causal-consistent Concelier storage sessions
Scoped session facilitator registered, repositories accept optional session handles, and replica-set failover tests verify read-your-write + monotonic reads. | +| Sprint 8 | Mongo strengthening | src/StellaOps.Authority/TASKS.md | DONE (2025-10-19) | Authority Core & Storage Guild | AUTHSTORAGE-MONGO-08-001 | Harden Authority Mongo usage
Scoped Mongo sessions with majority read/write concerns wired through stores and GraphQL/HTTP pipelines; replica-set election regression validated. | +| Sprint 8 | Mongo strengthening | src/StellaOps.Excititor.Storage.Mongo/TASKS.md | DONE (2025-10-19) | Team Excititor Storage | EXCITITOR-STORAGE-MONGO-08-001 | Causal consistency for Excititor repositories
Session-scoped repositories shipped with new Mongo records, orchestrators/workers now share scoped sessions, and replica-set failover coverage added via `dotnet test src/StellaOps.Excititor.Storage.Mongo.Tests/StellaOps.Excititor.Storage.Mongo.Tests.csproj`. | +| Sprint 8 | Platform Maintenance | src/StellaOps.Excititor.Storage.Mongo/TASKS.md | DONE (2025-10-19) | Team Excititor Storage | EXCITITOR-STORAGE-03-001 | Statement backfill tooling – shipped admin backfill endpoint, CLI hook (`stellaops excititor backfill-statements`), integration tests, and operator runbook (`docs/dev/EXCITITOR_STATEMENT_BACKFILL.md`). | +| Sprint 8 | Mirror Distribution | src/StellaOps.Concelier.Exporter.Json/TASKS.md | DONE (2025-10-19) | Concelier Export Guild | CONCELIER-EXPORT-08-201 | Mirror bundle + domain manifest – produce signed JSON aggregates for `*.stella-ops.org` mirrors. | +| Sprint 8 | Mirror Distribution | src/StellaOps.Concelier.Exporter.TrivyDb/TASKS.md | DONE (2025-10-19) | Concelier Export Guild | CONCELIER-EXPORT-08-202 | Mirror-ready Trivy DB bundles – mirror options emit per-domain manifests/metadata/db archives with deterministic digests for downstream sync. | +| Sprint 8 | Mirror Distribution | src/StellaOps.Concelier.WebService/TASKS.md | DOING (2025-10-19) | Concelier WebService Guild | CONCELIER-WEB-08-201 | Mirror distribution endpoints – expose domain-scoped index/download APIs with auth/quota. | +| Sprint 8 | Mirror Distribution | src/StellaOps.Concelier.Connector.StellaOpsMirror/TASKS.md | DOING (2025-10-19) | BE-Conn-Stella | FEEDCONN-STELLA-08-001 | Concelier mirror connector – fetch mirror manifest, verify signatures, and hydrate canonical DTOs with resume support. | +| Sprint 8 | Mirror Distribution | ops/devops/TASKS.md | DONE (2025-10-19) | DevOps Guild | DEVOPS-MIRROR-08-001 | Managed mirror deployments for `*.stella-ops.org` – Helm/Compose overlays, CDN, runbooks. | +| Sprint 8 | Plugin Infrastructure | src/StellaOps.Plugin/TASKS.md | DOING | Plugin Platform Guild, Authority Core | PLUGIN-DI-08-002.COORD | Authority scoped-service integration handshake
Session scheduled for 2025-10-20 15:00–16:00 UTC; agenda + attendees logged in `docs/dev/authority-plugin-di-coordination.md`. | +| Sprint 8 | Plugin Infrastructure | src/StellaOps.Authority/TASKS.md | DOING | Authority Core, Plugin Platform Guild | AUTH-PLUGIN-COORD-08-002 | Coordinate scoped-service adoption for Authority plug-in registrars
Workshop locked for 2025-10-20 15:00–16:00 UTC; pre-read checklist tracked in `docs/dev/authority-plugin-di-coordination.md`. | +| Sprint 9 | Scanner Core Foundations | src/StellaOps.Scanner.Core/TASKS.md | DONE (2025-10-18) | Team Scanner Core | SCANNER-CORE-09-501 | Define shared DTOs (ScanJob, ProgressEvent), error taxonomy, and deterministic ID/timestamp helpers aligning with `ARCHITECTURE_SCANNER.md` §3–§4. | +| Sprint 9 | Scanner Core Foundations | src/StellaOps.Scanner.Core/TASKS.md | DONE (2025-10-18) | Team Scanner Core | SCANNER-CORE-09-502 | Observability helpers (correlation IDs, logging scopes, metric namespacing, deterministic hashes) consumed by WebService/Worker. | +| Sprint 9 | Scanner Core Foundations | src/StellaOps.Scanner.Core/TASKS.md | DONE (2025-10-18) | Team Scanner Core | SCANNER-CORE-09-503 | Security utilities: Authority client factory, OpTok caching, DPoP verifier, restart-time plug-in guardrails for scanner components. | | Sprint 9 | Scanner Build-time | src/StellaOps.Scanner.Sbomer.BuildXPlugin/TASKS.md | DONE (2025-10-19) | BuildX Guild | SP9-BLDX-09-001 | Buildx driver scaffold + handshake with Scanner.Emit (local CAS). | | Sprint 9 | Scanner Build-time | src/StellaOps.Scanner.Sbomer.BuildXPlugin/TASKS.md | DONE (2025-10-19) | BuildX Guild | SP9-BLDX-09-002 | OCI annotations + provenance hand-off to Attestor. | | Sprint 9 | Scanner Build-time | src/StellaOps.Scanner.Sbomer.BuildXPlugin/TASKS.md | DONE (2025-10-19) | BuildX Guild | SP9-BLDX-09-003 | CI demo: minimal SBOM push & backend report wiring. | | Sprint 9 | Scanner Build-time | src/StellaOps.Scanner.Sbomer.BuildXPlugin/TASKS.md | DONE (2025-10-19) | BuildX Guild | SP9-BLDX-09-004 | Stabilize descriptor nonce derivation so repeated builds emit deterministic placeholders. | | Sprint 9 | Scanner Build-time | src/StellaOps.Scanner.Sbomer.BuildXPlugin/TASKS.md | DONE (2025-10-19) | BuildX Guild | SP9-BLDX-09-005 | Integrate determinism guard into GitHub/Gitea workflows and archive proof artifacts. | -| Sprint 9 | Scanner Core Foundations | src/StellaOps.Scanner.WebService/TASKS.md | DONE (2025-10-18) | Team Scanner WebService | SCANNER-WEB-09-101 | Minimal API host with Authority enforcement, health/ready endpoints, and restart-time plug-in loader per architecture §1, §4. | -| Sprint 9 | Scanner Core Foundations | src/StellaOps.Scanner.WebService/TASKS.md | DONE (2025-10-18) | Team Scanner WebService | SCANNER-WEB-09-102 | `/api/v1/scans` submission/status endpoints with deterministic IDs, validation, and cancellation support. | -| Sprint 9 | Scanner Core Foundations | src/StellaOps.Scanner.WebService/TASKS.md | TODO | Team Scanner WebService | SCANNER-WEB-09-103 | Progress streaming (SSE/JSONL) with correlation IDs and ISO-8601 UTC timestamps, documented in API reference. | -| Sprint 9 | Scanner Core Foundations | src/StellaOps.Scanner.WebService/TASKS.md | DONE (2025-10-19) | Team Scanner WebService | SCANNER-WEB-09-104 | Configuration binding for Mongo, MinIO, queue, feature flags; startup diagnostics and fail-fast policy. | -| Sprint 9 | Scanner Core Foundations | src/StellaOps.Scanner.WebService/TASKS.md | TODO | Team Scanner WebService | SCANNER-POLICY-09-105 | Policy snapshot loader + schema + OpenAPI (YAML ignore rules, VEX include/exclude, vendor precedence). | -| Sprint 9 | Scanner Core Foundations | src/StellaOps.Scanner.WebService/TASKS.md | TODO | Team Scanner WebService | SCANNER-POLICY-09-106 | `/reports` verdict assembly (Feedser+Vexer+Policy) + signed response envelope. | -| Sprint 9 | Scanner Core Foundations | src/StellaOps.Scanner.WebService/TASKS.md | TODO | Team Scanner WebService | SCANNER-POLICY-09-107 | Expose score inputs, config version, and quiet provenance in `/reports` JSON and signed payload. | -| Sprint 9 | Scanner Core Foundations | src/StellaOps.Scanner.Worker/TASKS.md | DONE (2025-10-19) | Team Scanner Worker | SCANNER-WORKER-09-201 | Worker host bootstrap with Authority auth, hosted services, and graceful shutdown semantics. | -| Sprint 9 | Scanner Core Foundations | src/StellaOps.Scanner.Worker/TASKS.md | DONE (2025-10-19) | Team Scanner Worker | SCANNER-WORKER-09-202 | Lease/heartbeat loop with retry+jitter, poison-job quarantine, structured logging. | +| Sprint 9 | Scanner Core Foundations | src/StellaOps.Scanner.WebService/TASKS.md | DONE (2025-10-18) | Team Scanner WebService | SCANNER-WEB-09-101 | Minimal API host with Authority enforcement, health/ready endpoints, and restart-time plug-in loader per architecture §1, §4. | +| Sprint 9 | Scanner Core Foundations | src/StellaOps.Scanner.WebService/TASKS.md | DONE (2025-10-18) | Team Scanner WebService | SCANNER-WEB-09-102 | `/api/v1/scans` submission/status endpoints with deterministic IDs, validation, and cancellation support. | +| Sprint 9 | Scanner Core Foundations | src/StellaOps.Scanner.WebService/TASKS.md | TODO | Team Scanner WebService | SCANNER-WEB-09-103 | Progress streaming (SSE/JSONL) with correlation IDs and ISO-8601 UTC timestamps, documented in API reference. | +| Sprint 9 | Scanner Core Foundations | src/StellaOps.Scanner.WebService/TASKS.md | DONE (2025-10-19) | Team Scanner WebService | SCANNER-WEB-09-104 | Configuration binding for Mongo, MinIO, queue, feature flags; startup diagnostics and fail-fast policy. | +| Sprint 9 | Scanner Core Foundations | src/StellaOps.Scanner.WebService/TASKS.md | TODO | Team Scanner WebService | SCANNER-POLICY-09-105 | Policy snapshot loader + schema + OpenAPI (YAML ignore rules, VEX include/exclude, vendor precedence). | +| Sprint 9 | Scanner Core Foundations | src/StellaOps.Scanner.WebService/TASKS.md | TODO | Team Scanner WebService | SCANNER-POLICY-09-106 | `/reports` verdict assembly (Feedser+Vexer+Policy) + signed response envelope. | +| Sprint 9 | Scanner Core Foundations | src/StellaOps.Scanner.WebService/TASKS.md | TODO | Team Scanner WebService | SCANNER-POLICY-09-107 | Expose score inputs, config version, and quiet provenance in `/reports` JSON and signed payload. | +| Sprint 9 | Scanner Core Foundations | src/StellaOps.Scanner.Worker/TASKS.md | DONE (2025-10-19) | Team Scanner Worker | SCANNER-WORKER-09-201 | Worker host bootstrap with Authority auth, hosted services, and graceful shutdown semantics. | +| Sprint 9 | Scanner Core Foundations | src/StellaOps.Scanner.Worker/TASKS.md | DONE (2025-10-19) | Team Scanner Worker | SCANNER-WORKER-09-202 | Lease/heartbeat loop with retry+jitter, poison-job quarantine, structured logging. | | Sprint 9 | Scanner Core Foundations | src/StellaOps.Scanner.Worker/TASKS.md | DONE (2025-10-19) | Team Scanner Worker | SCANNER-WORKER-09-203 | Analyzer dispatch skeleton emitting deterministic stage progress and honoring cancellation tokens. | | Sprint 9 | Scanner Core Foundations | src/StellaOps.Scanner.Worker/TASKS.md | DONE (2025-10-19) | Team Scanner Worker | SCANNER-WORKER-09-204 | Worker metrics (queue latency, stage duration, failure counts) with OpenTelemetry resource wiring. | | Sprint 9 | Scanner Core Foundations | src/StellaOps.Scanner.Worker/TASKS.md | DONE (2025-10-19) | Team Scanner Worker | SCANNER-WORKER-09-205 | Harden heartbeat jitter so lease safety margin stays ≥3× and cover with regression tests + optional live queue smoke run. | -| Sprint 9 | Policy Foundations | src/StellaOps.Policy/TASKS.md | DONE | Policy Guild | POLICY-CORE-09-001 | Policy schema + binder + diagnostics. | -| Sprint 9 | Policy Foundations | src/StellaOps.Policy/TASKS.md | DONE | Policy Guild | POLICY-CORE-09-002 | Policy snapshot store + revision digests. | -| Sprint 9 | Policy Foundations | src/StellaOps.Policy/TASKS.md | DONE | Policy Guild | POLICY-CORE-09-003 | `/policy/preview` API (image digest → projected verdict diff). | -| Sprint 9 | Policy Foundations | src/StellaOps.Policy/TASKS.md | TODO | Policy Guild | POLICY-CORE-09-004 | Versioned scoring config with schema validation, trust table, and golden fixtures. | -| Sprint 9 | Policy Foundations | src/StellaOps.Policy/TASKS.md | TODO | Policy Guild | POLICY-CORE-09-005 | Scoring/quiet engine – compute score, enforce VEX-only quiet rules, emit inputs and provenance. | -| Sprint 9 | Policy Foundations | src/StellaOps.Policy/TASKS.md | TODO | Policy Guild | POLICY-CORE-09-006 | Unknown state & confidence decay – deterministic bands surfaced in policy outputs. | -| Sprint 9 | DevOps Foundations | ops/devops/TASKS.md | DONE (2025-10-19) | DevOps Guild | DEVOPS-HELM-09-001 | Helm/Compose environment profiles (dev/staging/airgap) with deterministic digests. | -| Sprint 9 | Docs & Governance | docs/TASKS.md | DONE (2025-10-19) | Docs Guild, DevEx | DOCS-ADR-09-001 | Establish ADR process and template. | -| Sprint 9 | Docs & Governance | docs/TASKS.md | DONE (2025-10-19) | Docs Guild, Platform Events | DOCS-EVENTS-09-002 | Publish event schema catalog (`docs/events/`) for critical envelopes. | -| Sprint 9 | Scanner Core Foundations | src/StellaOps.Scanner.Storage/TASKS.md | DONE (2025-10-19) | Team Scanner Storage | SCANNER-STORAGE-09-301 | Mongo catalog schemas/indexes for images, layers, artifacts, jobs, lifecycle rules plus migrations. | -| Sprint 9 | Scanner Core Foundations | src/StellaOps.Scanner.Storage/TASKS.md | DONE (2025-10-19) | Team Scanner Storage | SCANNER-STORAGE-09-302 | MinIO layout, immutability policies, client abstraction, and configuration binding. | -| Sprint 9 | Scanner Core Foundations | src/StellaOps.Scanner.Storage/TASKS.md | DONE (2025-10-19) | Team Scanner Storage | SCANNER-STORAGE-09-303 | Repositories/services with dual-write feature flag, deterministic digests, TTL enforcement tests. | -| Sprint 9 | Scanner Core Foundations | src/StellaOps.Scanner.Queue/TASKS.md | DONE (2025-10-19) | Team Scanner Queue | SCANNER-QUEUE-09-401 | Queue abstraction + Redis Streams adapter with ack/claim APIs and idempotency tokens. | -| Sprint 9 | Scanner Core Foundations | src/StellaOps.Scanner.Queue/TASKS.md | DONE (2025-10-19) | Team Scanner Queue | SCANNER-QUEUE-09-402 | Pluggable backend support (Redis, NATS) with configuration binding, health probes, failover docs. | -| Sprint 9 | Scanner Core Foundations | src/StellaOps.Scanner.Queue/TASKS.md | DONE (2025-10-19) | Team Scanner Queue | SCANNER-QUEUE-09-403 | Retry + dead-letter strategy with structured logs/metrics for offline deployments. | -| Sprint 10 | Scanner Analyzers & SBOM | src/StellaOps.Scanner.Cache/TASKS.md | TODO | Scanner Cache Guild | SCANNER-CACHE-10-101 | Implement layer cache store keyed by layer digest with metadata retention per architecture §3.3. | -| Sprint 10 | Scanner Analyzers & SBOM | src/StellaOps.Scanner.Cache/TASKS.md | TODO | Scanner Cache Guild | SCANNER-CACHE-10-102 | Build file CAS with dedupe, TTL enforcement, and offline import/export hooks. | -| Sprint 10 | Scanner Analyzers & SBOM | src/StellaOps.Scanner.Cache/TASKS.md | TODO | Scanner Cache Guild | SCANNER-CACHE-10-103 | Expose cache metrics/logging and configuration toggles for warm/cold thresholds. | -| Sprint 10 | Scanner Analyzers & SBOM | src/StellaOps.Scanner.Cache/TASKS.md | TODO | Scanner Cache Guild | SCANNER-CACHE-10-104 | Implement cache invalidation workflows (layer delete, TTL expiry, diff invalidation). | -| Sprint 10 | Scanner Analyzers & SBOM | src/StellaOps.Scanner.Analyzers.OS/TASKS.md | TODO | OS Analyzer Guild | SCANNER-ANALYZERS-OS-10-201 | Alpine/apk analyzer emitting deterministic components with provenance. | -| Sprint 10 | Scanner Analyzers & SBOM | src/StellaOps.Scanner.Analyzers.OS/TASKS.md | TODO | OS Analyzer Guild | SCANNER-ANALYZERS-OS-10-202 | Debian/dpkg analyzer mapping packages to purl identity with evidence. | -| Sprint 10 | Scanner Analyzers & SBOM | src/StellaOps.Scanner.Analyzers.OS/TASKS.md | TODO | OS Analyzer Guild | SCANNER-ANALYZERS-OS-10-203 | RPM analyzer capturing EVR, file listings, provenance. | -| Sprint 10 | Scanner Analyzers & SBOM | src/StellaOps.Scanner.Analyzers.OS/TASKS.md | TODO | OS Analyzer Guild | SCANNER-ANALYZERS-OS-10-204 | Shared OS evidence helpers for package identity + provenance. | -| Sprint 10 | Scanner Analyzers & SBOM | src/StellaOps.Scanner.Analyzers.OS/TASKS.md | TODO | OS Analyzer Guild | SCANNER-ANALYZERS-OS-10-205 | Vendor metadata enrichment (source packages, license, CVE hints). | -| Sprint 10 | Scanner Analyzers & SBOM | src/StellaOps.Scanner.Analyzers.OS/TASKS.md | TODO | OS Analyzer Guild | SCANNER-ANALYZERS-OS-10-206 | Determinism harness + fixtures for OS analyzers. | -| Sprint 10 | Scanner Analyzers & SBOM | src/StellaOps.Scanner.Analyzers.OS/TASKS.md | TODO | OS Analyzer Guild | SCANNER-ANALYZERS-OS-10-207 | Package OS analyzers as restart-time plug-ins (manifest + host registration). | -| Sprint 10 | Scanner Analyzers & SBOM | src/StellaOps.Scanner.Analyzers.Lang/TASKS.md | TODO | Language Analyzer Guild | SCANNER-ANALYZERS-LANG-10-301 | Java analyzer emitting `pkg:maven` with provenance. | -| Sprint 10 | Scanner Analyzers & SBOM | src/StellaOps.Scanner.Analyzers.Lang/TASKS.md | TODO | Language Analyzer Guild | SCANNER-ANALYZERS-LANG-10-302 | Node analyzer handling workspaces/symlinks emitting `pkg:npm`. | -| Sprint 10 | Scanner Analyzers & SBOM | src/StellaOps.Scanner.Analyzers.Lang/TASKS.md | TODO | Language Analyzer Guild | SCANNER-ANALYZERS-LANG-10-303 | Python analyzer reading `*.dist-info`, RECORD hashes, entry points. | -| Sprint 10 | Scanner Analyzers & SBOM | src/StellaOps.Scanner.Analyzers.Lang/TASKS.md | TODO | Language Analyzer Guild | SCANNER-ANALYZERS-LANG-10-304 | Go analyzer leveraging buildinfo for `pkg:golang` components. | -| Sprint 10 | Scanner Analyzers & SBOM | src/StellaOps.Scanner.Analyzers.Lang/TASKS.md | TODO | Language Analyzer Guild | SCANNER-ANALYZERS-LANG-10-305 | .NET analyzer parsing `*.deps.json`, assembly metadata, RID variants. | -| Sprint 10 | Scanner Analyzers & SBOM | src/StellaOps.Scanner.Analyzers.Lang/TASKS.md | TODO | Language Analyzer Guild | SCANNER-ANALYZERS-LANG-10-306 | Rust analyzer detecting crates or falling back to `bin:{sha256}`. | -| Sprint 10 | Scanner Analyzers & SBOM | src/StellaOps.Scanner.Analyzers.Lang/TASKS.md | TODO | Language Analyzer Guild | SCANNER-ANALYZERS-LANG-10-307 | Shared language evidence helpers + usage flag propagation. | -| Sprint 10 | Scanner Analyzers & SBOM | src/StellaOps.Scanner.Analyzers.Lang/TASKS.md | TODO | Language Analyzer Guild | SCANNER-ANALYZERS-LANG-10-308 | Determinism + fixture harness for language analyzers. | -| Sprint 10 | Scanner Analyzers & SBOM | src/StellaOps.Scanner.Analyzers.Lang/TASKS.md | TODO | Language Analyzer Guild | SCANNER-ANALYZERS-LANG-10-309 | Package language analyzers as restart-time plug-ins (manifest + host registration). | -| Sprint 10 | Scanner Analyzers & SBOM | src/StellaOps.Scanner.EntryTrace/TASKS.md | TODO | EntryTrace Guild | SCANNER-ENTRYTRACE-10-401 | POSIX shell AST parser with deterministic output. | -| Sprint 10 | Scanner Analyzers & SBOM | src/StellaOps.Scanner.EntryTrace/TASKS.md | TODO | EntryTrace Guild | SCANNER-ENTRYTRACE-10-402 | Command resolution across layered rootfs with evidence attribution. | -| Sprint 10 | Scanner Analyzers & SBOM | src/StellaOps.Scanner.EntryTrace/TASKS.md | TODO | EntryTrace Guild | SCANNER-ENTRYTRACE-10-403 | Interpreter tracing for shell wrappers to Python/Node/Java launchers. | -| Sprint 10 | Scanner Analyzers & SBOM | src/StellaOps.Scanner.EntryTrace/TASKS.md | TODO | EntryTrace Guild | SCANNER-ENTRYTRACE-10-404 | Python entry analyzer (venv shebang, module invocation, usage flag). | -| Sprint 10 | Scanner Analyzers & SBOM | src/StellaOps.Scanner.EntryTrace/TASKS.md | TODO | EntryTrace Guild | SCANNER-ENTRYTRACE-10-405 | Node/Java launcher analyzer capturing script/jar targets. | -| Sprint 10 | Scanner Analyzers & SBOM | src/StellaOps.Scanner.EntryTrace/TASKS.md | TODO | EntryTrace Guild | SCANNER-ENTRYTRACE-10-406 | Explainability + diagnostics for unresolved constructs with metrics. | -| Sprint 10 | Scanner Analyzers & SBOM | src/StellaOps.Scanner.EntryTrace/TASKS.md | TODO | EntryTrace Guild | SCANNER-ENTRYTRACE-10-407 | Package EntryTrace analyzers as restart-time plug-ins (manifest + host registration). | -| Sprint 10 | Scanner Analyzers & SBOM | src/StellaOps.Scanner.Diff/TASKS.md | TODO | Diff Guild | SCANNER-DIFF-10-501 | Build component differ tracking add/remove/version changes with deterministic ordering. | -| Sprint 10 | Scanner Analyzers & SBOM | src/StellaOps.Scanner.Diff/TASKS.md | TODO | Diff Guild | SCANNER-DIFF-10-502 | Attribute diffs to introducing/removing layers including provenance evidence. | -| Sprint 10 | Scanner Analyzers & SBOM | src/StellaOps.Scanner.Diff/TASKS.md | TODO | Diff Guild | SCANNER-DIFF-10-503 | Produce JSON diff output for inventory vs usage views aligned with API contract. | -| Sprint 10 | Scanner Analyzers & SBOM | src/StellaOps.Scanner.Emit/TASKS.md | TODO | Emit Guild | SCANNER-EMIT-10-601 | Compose inventory SBOM (CycloneDX JSON/Protobuf) from layer fragments. | -| Sprint 10 | Scanner Analyzers & SBOM | src/StellaOps.Scanner.Emit/TASKS.md | TODO | Emit Guild | SCANNER-EMIT-10-602 | Compose usage SBOM leveraging EntryTrace to flag actual usage. | -| Sprint 10 | Scanner Analyzers & SBOM | src/StellaOps.Scanner.Emit/TASKS.md | TODO | Emit Guild | SCANNER-EMIT-10-603 | Generate BOM index sidecar (purl table + roaring bitmap + usage flag). | -| Sprint 10 | Scanner Analyzers & SBOM | src/StellaOps.Scanner.Emit/TASKS.md | TODO | Emit Guild | SCANNER-EMIT-10-604 | Package artifacts for export + attestation with deterministic manifests. | -| Sprint 10 | Scanner Analyzers & SBOM | src/StellaOps.Scanner.Emit/TASKS.md | TODO | Emit Guild | SCANNER-EMIT-10-605 | Emit BOM-Index sidecar schema/fixtures (CRITICAL PATH for SP16). | -| Sprint 10 | Scanner Analyzers & SBOM | src/StellaOps.Scanner.Emit/TASKS.md | TODO | Emit Guild | SCANNER-EMIT-10-606 | Usage view bit flags integrated with EntryTrace. | -| Sprint 10 | Scanner Analyzers & SBOM | src/StellaOps.Scanner.Emit/TASKS.md | TODO | Emit Guild | SCANNER-EMIT-10-607 | Embed scoring inputs, confidence band, and quiet provenance in CycloneDX/DSSE artifacts. | -| Sprint 10 | Benchmarks | bench/TASKS.md | TODO | Bench Guild, Scanner Team | BENCH-SCANNER-10-001 | Analyzer microbench harness + baseline CSV. | -| Sprint 10 | Samples | samples/TASKS.md | TODO | Samples Guild, Scanner Team | SAMPLES-10-001 | Sample images with SBOM/BOM-Index sidecars. | -| Sprint 10 | DevOps Perf | ops/devops/TASKS.md | TODO | DevOps Guild | DEVOPS-PERF-10-001 | Perf smoke job ensuring <5 s SBOM compose. | -| Sprint 11 | Signing Chain Bring-up | src/StellaOps.Authority/TASKS.md | TODO | Authority Core & Security Guild | AUTH-DPOP-11-001 | Implement DPoP proof validation + nonce handling for high-value audiences per architecture. | -| Sprint 11 | Signing Chain Bring-up | src/StellaOps.Authority/TASKS.md | TODO | Authority Core & Security Guild | AUTH-MTLS-11-002 | Add OAuth mTLS client credential support with certificate-bound tokens and introspection updates. | -| Sprint 11 | Signing Chain Bring-up | src/StellaOps.Signer/TASKS.md | TODO | Signer Guild | SIGNER-API-11-101 | `/sign/dsse` pipeline with Authority auth, PoE introspection, release verification, DSSE signing. | -| Sprint 11 | Signing Chain Bring-up | src/StellaOps.Signer/TASKS.md | TODO | Signer Guild | SIGNER-REF-11-102 | `/verify/referrers` endpoint with OCI lookup, caching, and policy enforcement. | -| Sprint 11 | Signing Chain Bring-up | src/StellaOps.Signer/TASKS.md | TODO | Signer Guild | SIGNER-QUOTA-11-103 | Enforce plan quotas, concurrency/QPS limits, artifact size caps with metrics/audit logs. | -| Sprint 11 | Signing Chain Bring-up | src/StellaOps.Attestor/TASKS.md | TODO | Attestor Guild | ATTESTOR-API-11-201 | `/rekor/entries` submission pipeline with dedupe, proof acquisition, and persistence. | -| Sprint 11 | Signing Chain Bring-up | src/StellaOps.Attestor/TASKS.md | TODO | Attestor Guild | ATTESTOR-VERIFY-11-202 | `/rekor/verify` + retrieval endpoints validating signatures and Merkle proofs. | -| Sprint 11 | Signing Chain Bring-up | src/StellaOps.Attestor/TASKS.md | TODO | Attestor Guild | ATTESTOR-OBS-11-203 | Telemetry, alerting, mTLS hardening, and archive workflow for Attestor. | -| Sprint 11 | UI Integration | src/StellaOps.UI/TASKS.md | TODO | UI Guild | UI-ATTEST-11-005 | Attestation visibility (Rekor id, status) on Scan Detail. | -| Sprint 12 | Runtime Guardrails | src/StellaOps.Zastava.Core/TASKS.md | TODO | Zastava Core Guild | ZASTAVA-CORE-12-201 | Define runtime event/admission DTOs, hashing helpers, and versioning strategy. | -| Sprint 12 | Runtime Guardrails | src/StellaOps.Zastava.Core/TASKS.md | TODO | Zastava Core Guild | ZASTAVA-CORE-12-202 | Provide configuration/logging/metrics utilities shared by Observer/Webhook. | -| Sprint 12 | Runtime Guardrails | src/StellaOps.Zastava.Core/TASKS.md | TODO | Zastava Core Guild | ZASTAVA-CORE-12-203 | Authority client helpers, OpTok caching, and security guardrails for runtime services. | -| Sprint 12 | Runtime Guardrails | src/StellaOps.Zastava.Core/TASKS.md | TODO | Zastava Core Guild | ZASTAVA-OPS-12-204 | Operational runbooks, alert rules, and dashboard exports for runtime plane. | -| Sprint 12 | Runtime Guardrails | src/StellaOps.Zastava.Observer/TASKS.md | TODO | Zastava Observer Guild | ZASTAVA-OBS-12-001 | Container lifecycle watcher emitting deterministic runtime events with buffering. | -| Sprint 12 | Runtime Guardrails | src/StellaOps.Zastava.Observer/TASKS.md | TODO | Zastava Observer Guild | ZASTAVA-OBS-12-002 | Capture entrypoint traces + loaded libraries, hashing binaries and linking to baseline SBOM. | -| Sprint 12 | Runtime Guardrails | src/StellaOps.Zastava.Observer/TASKS.md | TODO | Zastava Observer Guild | ZASTAVA-OBS-12-003 | Posture checks for signatures/SBOM/attestation with offline caching. | -| Sprint 12 | Runtime Guardrails | src/StellaOps.Zastava.Observer/TASKS.md | TODO | Zastava Observer Guild | ZASTAVA-OBS-12-004 | Batch `/runtime/events` submissions with disk-backed buffer and rate limits. | -| Sprint 12 | Runtime Guardrails | src/StellaOps.Zastava.Webhook/TASKS.md | TODO | Zastava Webhook Guild | ZASTAVA-WEBHOOK-12-101 | Admission controller host with TLS bootstrap and Authority auth. | -| Sprint 12 | Runtime Guardrails | src/StellaOps.Zastava.Webhook/TASKS.md | TODO | Zastava Webhook Guild | ZASTAVA-WEBHOOK-12-102 | Query Scanner `/policy/runtime`, resolve digests, enforce verdicts. | -| Sprint 12 | Runtime Guardrails | src/StellaOps.Zastava.Webhook/TASKS.md | TODO | Zastava Webhook Guild | ZASTAVA-WEBHOOK-12-103 | Caching, fail-open/closed toggles, metrics/logging for admission decisions. | -| Sprint 12 | Runtime Guardrails | src/StellaOps.Scanner.WebService/TASKS.md | TODO | Scanner WebService Guild | SCANNER-RUNTIME-12-301 | `/runtime/events` ingestion endpoint with validation, batching, storage hooks. | -| Sprint 12 | Runtime Guardrails | src/StellaOps.Scanner.WebService/TASKS.md | TODO | Scanner WebService Guild | SCANNER-RUNTIME-12-302 | `/policy/runtime` endpoint joining SBOM baseline + policy verdict with TTL guidance. | -| Sprint 13 | UX & CLI Experience | src/StellaOps.UI/TASKS.md | TODO | UI Guild | UI-AUTH-13-001 | Integrate Authority OIDC + DPoP flows with session management. | -| Sprint 13 | UX & CLI Experience | src/StellaOps.UI/TASKS.md | TODO | UI Guild | UI-SCANS-13-002 | Build scans module (list/detail/SBOM/diff/attestation) with performance + accessibility targets. | -| Sprint 13 | UX & CLI Experience | src/StellaOps.UI/TASKS.md | TODO | UI Guild | UI-VEX-13-003 | Implement VEX explorer + policy editor with preview integration. | -| Sprint 13 | UX & CLI Experience | src/StellaOps.UI/TASKS.md | TODO | UI Guild | UI-ADMIN-13-004 | Deliver admin area (tenants/clients/quotas/licensing) with RBAC + audit hooks. | -| Sprint 13 | UX & CLI Experience | src/StellaOps.UI/TASKS.md | TODO | UI Guild | UI-SCHED-13-005 | Scheduler panel: schedules CRUD, run history, dry-run preview. | -| Sprint 13 | UX & CLI Experience | src/StellaOps.UI/TASKS.md | TODO | UI Guild | UI-NOTIFY-13-006 | Notify panel: channels/rules CRUD, deliveries view, test send. | -| Sprint 13 | UX & CLI Experience | src/StellaOps.Cli/TASKS.md | TODO | DevEx/CLI | CLI-RUNTIME-13-005 | Add runtime policy test verbs that consume `/policy/runtime` and display verdicts. | -| Sprint 13 | UX & CLI Experience | src/StellaOps.Cli/TASKS.md | TODO | DevEx/CLI | CLI-OFFLINE-13-006 | Implement offline kit pull/import/status commands with integrity checks. | -| Sprint 13 | UX & CLI Experience | src/StellaOps.Cli/TASKS.md | TODO | DevEx/CLI | CLI-PLUGIN-13-007 | Package non-core CLI verbs as restart-time plug-ins (manifest + loader tests). | -| Sprint 14 | Release & Offline Ops | ops/devops/TASKS.md | TODO | DevOps Guild | DEVOPS-REL-14-001 | Deterministic build/release pipeline with SBOM/provenance, signing, and manifest generation. | -| Sprint 14 | Release & Offline Ops | ops/offline-kit/TASKS.md | TODO | Offline Kit Guild | DEVOPS-OFFLINE-14-002 | Offline kit packaging workflow with integrity verification and documentation. | -| Sprint 14 | Release & Offline Ops | ops/deployment/TASKS.md | TODO | Deployment Guild | DEVOPS-OPS-14-003 | Deployment/update/rollback automation and channel management documentation. | -| Sprint 14 | Release & Offline Ops | ops/licensing/TASKS.md | TODO | Licensing Guild | DEVOPS-LIC-14-004 | Registry token service tied to Authority, plan gating, revocation handling, monitoring. | -| Sprint 15 | Notify Foundations | src/StellaOps.Notify.Models/TASKS.md | TODO | Notify Models Guild | NOTIFY-MODELS-15-101 | Define core Notify DTOs, validation helpers, canonical serialization. | -| Sprint 15 | Notify Foundations | src/StellaOps.Notify.Models/TASKS.md | TODO | Notify Models Guild | NOTIFY-MODELS-15-102 | Publish schema docs and sample payloads for Notify. | -| Sprint 15 | Notify Foundations | src/StellaOps.Notify.Models/TASKS.md | TODO | Notify Models Guild | NOTIFY-MODELS-15-103 | Versioning/migration helpers for rules/templates/deliveries. | -| Sprint 15 | Notify Foundations | src/StellaOps.Notify.Storage.Mongo/TASKS.md | TODO | Notify Storage Guild | NOTIFY-STORAGE-15-201 | Mongo schemas/indexes for rules, channels, deliveries, digests, locks, audit. | -| Sprint 15 | Notify Foundations | src/StellaOps.Notify.Storage.Mongo/TASKS.md | TODO | Notify Storage Guild | NOTIFY-STORAGE-15-202 | Repositories with tenant scoping, soft delete, TTL, causal consistency options. | -| Sprint 15 | Notify Foundations | src/StellaOps.Notify.Storage.Mongo/TASKS.md | TODO | Notify Storage Guild | NOTIFY-STORAGE-15-203 | Delivery history retention and query APIs. | -| Sprint 15 | Notify Foundations | src/StellaOps.Notify.Queue/TASKS.md | TODO | Notify Queue Guild | NOTIFY-QUEUE-15-401 | Bus abstraction + Redis Streams adapter with ordering/idempotency. | -| Sprint 15 | Notify Foundations | src/StellaOps.Notify.Queue/TASKS.md | TODO | Notify Queue Guild | NOTIFY-QUEUE-15-402 | NATS JetStream adapter with health probes and failover. | -| Sprint 15 | Notify Foundations | src/StellaOps.Notify.Queue/TASKS.md | TODO | Notify Queue Guild | NOTIFY-QUEUE-15-403 | Delivery queue with retry/dead-letter + metrics. | -| Sprint 15 | Notify Foundations | src/StellaOps.Notify.Engine/TASKS.md | TODO | Notify Engine Guild | NOTIFY-ENGINE-15-301 | Rules evaluation core (filters, throttles, idempotency). | -| Sprint 15 | Notify Foundations | src/StellaOps.Notify.Engine/TASKS.md | TODO | Notify Engine Guild | NOTIFY-ENGINE-15-302 | Action planner + digest coalescer. | -| Sprint 15 | Notify Foundations | src/StellaOps.Notify.Engine/TASKS.md | TODO | Notify Engine Guild | NOTIFY-ENGINE-15-303 | Template rendering engine (Slack/Teams/Email/Webhook). | -| Sprint 15 | Notify Foundations | src/StellaOps.Notify.Engine/TASKS.md | TODO | Notify Engine Guild | NOTIFY-ENGINE-15-304 | Test-send sandbox + preview utilities. | -| Sprint 15 | Notify Foundations | src/StellaOps.Notify.WebService/TASKS.md | TODO | Notify WebService Guild | NOTIFY-WEB-15-101 | Minimal API host with Authority enforcement and plug-in loading. | -| Sprint 15 | Notify Foundations | src/StellaOps.Notify.WebService/TASKS.md | TODO | Notify WebService Guild | NOTIFY-WEB-15-102 | Rules/channel/template CRUD with audit logging. | -| Sprint 15 | Notify Foundations | src/StellaOps.Notify.WebService/TASKS.md | TODO | Notify WebService Guild | NOTIFY-WEB-15-103 | Delivery history & test-send endpoints. | -| Sprint 15 | Notify Foundations | src/StellaOps.Notify.WebService/TASKS.md | TODO | Notify WebService Guild | NOTIFY-WEB-15-104 | Configuration binding + startup diagnostics. | -| Sprint 15 | Notify Foundations | src/StellaOps.Notify.Worker/TASKS.md | TODO | Notify Worker Guild | NOTIFY-WORKER-15-201 | Bus subscription + leasing loop with backoff. | -| Sprint 15 | Notify Foundations | src/StellaOps.Notify.Worker/TASKS.md | TODO | Notify Worker Guild | NOTIFY-WORKER-15-202 | Rules evaluation pipeline integration. | -| Sprint 15 | Notify Foundations | src/StellaOps.Notify.Worker/TASKS.md | TODO | Notify Worker Guild | NOTIFY-WORKER-15-203 | Channel dispatch orchestration with retries. | -| Sprint 15 | Notify Foundations | src/StellaOps.Notify.Worker/TASKS.md | TODO | Notify Worker Guild | NOTIFY-WORKER-15-204 | Metrics/telemetry for Notify workers. | -| Sprint 15 | Notify Foundations | src/StellaOps.Notify.Connectors.Slack/TASKS.md | TODO | Notify Connectors Guild | NOTIFY-CONN-SLACK-15-501 | Slack connector with rate-limit aware delivery. | -| Sprint 15 | Notify Foundations | src/StellaOps.Notify.Connectors.Slack/TASKS.md | TODO | Notify Connectors Guild | NOTIFY-CONN-SLACK-15-502 | Slack health/test-send support. | -| Sprint 15 | Notify Foundations | src/StellaOps.Notify.Connectors.Teams/TASKS.md | TODO | Notify Connectors Guild | NOTIFY-CONN-TEAMS-15-601 | Teams connector with Adaptive Cards. | -| Sprint 15 | Notify Foundations | src/StellaOps.Notify.Connectors.Teams/TASKS.md | TODO | Notify Connectors Guild | NOTIFY-CONN-TEAMS-15-602 | Teams health/test-send support. | -| Sprint 15 | Notify Foundations | src/StellaOps.Notify.Connectors.Email/TASKS.md | TODO | Notify Connectors Guild | NOTIFY-CONN-EMAIL-15-701 | SMTP connector with TLS + rendering. | -| Sprint 15 | Notify Foundations | src/StellaOps.Notify.Connectors.Email/TASKS.md | TODO | Notify Connectors Guild | NOTIFY-CONN-EMAIL-15-702 | DKIM + health/test-send flows. | -| Sprint 15 | Notify Foundations | src/StellaOps.Notify.Connectors.Webhook/TASKS.md | TODO | Notify Connectors Guild | NOTIFY-CONN-WEBHOOK-15-801 | Webhook connector with signing/retries. | -| Sprint 15 | Notify Foundations | src/StellaOps.Notify.Connectors.Webhook/TASKS.md | TODO | Notify Connectors Guild | NOTIFY-CONN-WEBHOOK-15-802 | Webhook health/test-send support. | -| Sprint 15 | Notify Foundations | src/StellaOps.Notify.Connectors.Slack/TASKS.md | TODO | Notify Connectors Guild | NOTIFY-CONN-SLACK-15-503 | Package Slack connector as restart-time plug-in (manifest + host registration). | -| Sprint 15 | Notify Foundations | src/StellaOps.Notify.Connectors.Teams/TASKS.md | TODO | Notify Connectors Guild | NOTIFY-CONN-TEAMS-15-603 | Package Teams connector as restart-time plug-in (manifest + host registration). | -| Sprint 15 | Notify Foundations | src/StellaOps.Notify.Connectors.Email/TASKS.md | TODO | Notify Connectors Guild | NOTIFY-CONN-EMAIL-15-703 | Package Email connector as restart-time plug-in (manifest + host registration). | +| Sprint 9 | Policy Foundations | src/StellaOps.Policy/TASKS.md | DONE | Policy Guild | POLICY-CORE-09-001 | Policy schema + binder + diagnostics. | +| Sprint 9 | Policy Foundations | src/StellaOps.Policy/TASKS.md | DONE | Policy Guild | POLICY-CORE-09-002 | Policy snapshot store + revision digests. | +| Sprint 9 | Policy Foundations | src/StellaOps.Policy/TASKS.md | DONE | Policy Guild | POLICY-CORE-09-003 | `/policy/preview` API (image digest → projected verdict diff). | +| Sprint 9 | Policy Foundations | src/StellaOps.Policy/TASKS.md | TODO | Policy Guild | POLICY-CORE-09-004 | Versioned scoring config with schema validation, trust table, and golden fixtures. | +| Sprint 9 | Policy Foundations | src/StellaOps.Policy/TASKS.md | TODO | Policy Guild | POLICY-CORE-09-005 | Scoring/quiet engine – compute score, enforce VEX-only quiet rules, emit inputs and provenance. | +| Sprint 9 | Policy Foundations | src/StellaOps.Policy/TASKS.md | TODO | Policy Guild | POLICY-CORE-09-006 | Unknown state & confidence decay – deterministic bands surfaced in policy outputs. | +| Sprint 9 | DevOps Foundations | ops/devops/TASKS.md | DONE (2025-10-19) | DevOps Guild | DEVOPS-HELM-09-001 | Helm/Compose environment profiles (dev/staging/airgap) with deterministic digests. | +| Sprint 9 | Docs & Governance | docs/TASKS.md | DONE (2025-10-19) | Docs Guild, DevEx | DOCS-ADR-09-001 | Establish ADR process and template. | +| Sprint 9 | Docs & Governance | docs/TASKS.md | DONE (2025-10-19) | Docs Guild, Platform Events | DOCS-EVENTS-09-002 | Publish event schema catalog (`docs/events/`) for critical envelopes. | +| Sprint 9 | Scanner Core Foundations | src/StellaOps.Scanner.Storage/TASKS.md | DONE (2025-10-19) | Team Scanner Storage | SCANNER-STORAGE-09-301 | Mongo catalog schemas/indexes for images, layers, artifacts, jobs, lifecycle rules plus migrations. | +| Sprint 9 | Scanner Core Foundations | src/StellaOps.Scanner.Storage/TASKS.md | DONE (2025-10-19) | Team Scanner Storage | SCANNER-STORAGE-09-302 | MinIO layout, immutability policies, client abstraction, and configuration binding. | +| Sprint 9 | Scanner Core Foundations | src/StellaOps.Scanner.Storage/TASKS.md | DONE (2025-10-19) | Team Scanner Storage | SCANNER-STORAGE-09-303 | Repositories/services with dual-write feature flag, deterministic digests, TTL enforcement tests. | +| Sprint 9 | Scanner Core Foundations | src/StellaOps.Scanner.Queue/TASKS.md | DONE (2025-10-19) | Team Scanner Queue | SCANNER-QUEUE-09-401 | Queue abstraction + Redis Streams adapter with ack/claim APIs and idempotency tokens. | +| Sprint 9 | Scanner Core Foundations | src/StellaOps.Scanner.Queue/TASKS.md | DONE (2025-10-19) | Team Scanner Queue | SCANNER-QUEUE-09-402 | Pluggable backend support (Redis, NATS) with configuration binding, health probes, failover docs. | +| Sprint 9 | Scanner Core Foundations | src/StellaOps.Scanner.Queue/TASKS.md | DONE (2025-10-19) | Team Scanner Queue | SCANNER-QUEUE-09-403 | Retry + dead-letter strategy with structured logs/metrics for offline deployments. | +| Sprint 10 | Scanner Analyzers & SBOM | src/StellaOps.Scanner.Cache/TASKS.md | TODO | Scanner Cache Guild | SCANNER-CACHE-10-101 | Implement layer cache store keyed by layer digest with metadata retention per architecture §3.3. | +| Sprint 10 | Scanner Analyzers & SBOM | src/StellaOps.Scanner.Cache/TASKS.md | TODO | Scanner Cache Guild | SCANNER-CACHE-10-102 | Build file CAS with dedupe, TTL enforcement, and offline import/export hooks. | +| Sprint 10 | Scanner Analyzers & SBOM | src/StellaOps.Scanner.Cache/TASKS.md | TODO | Scanner Cache Guild | SCANNER-CACHE-10-103 | Expose cache metrics/logging and configuration toggles for warm/cold thresholds. | +| Sprint 10 | Scanner Analyzers & SBOM | src/StellaOps.Scanner.Cache/TASKS.md | TODO | Scanner Cache Guild | SCANNER-CACHE-10-104 | Implement cache invalidation workflows (layer delete, TTL expiry, diff invalidation). | +| Sprint 10 | Scanner Analyzers & SBOM | src/StellaOps.Scanner.Analyzers.OS/TASKS.md | TODO | OS Analyzer Guild | SCANNER-ANALYZERS-OS-10-201 | Alpine/apk analyzer emitting deterministic components with provenance. | +| Sprint 10 | Scanner Analyzers & SBOM | src/StellaOps.Scanner.Analyzers.OS/TASKS.md | TODO | OS Analyzer Guild | SCANNER-ANALYZERS-OS-10-202 | Debian/dpkg analyzer mapping packages to purl identity with evidence. | +| Sprint 10 | Scanner Analyzers & SBOM | src/StellaOps.Scanner.Analyzers.OS/TASKS.md | TODO | OS Analyzer Guild | SCANNER-ANALYZERS-OS-10-203 | RPM analyzer capturing EVR, file listings, provenance. | +| Sprint 10 | Scanner Analyzers & SBOM | src/StellaOps.Scanner.Analyzers.OS/TASKS.md | TODO | OS Analyzer Guild | SCANNER-ANALYZERS-OS-10-204 | Shared OS evidence helpers for package identity + provenance. | +| Sprint 10 | Scanner Analyzers & SBOM | src/StellaOps.Scanner.Analyzers.OS/TASKS.md | TODO | OS Analyzer Guild | SCANNER-ANALYZERS-OS-10-205 | Vendor metadata enrichment (source packages, license, CVE hints). | +| Sprint 10 | Scanner Analyzers & SBOM | src/StellaOps.Scanner.Analyzers.OS/TASKS.md | TODO | OS Analyzer Guild | SCANNER-ANALYZERS-OS-10-206 | Determinism harness + fixtures for OS analyzers. | +| Sprint 10 | Scanner Analyzers & SBOM | src/StellaOps.Scanner.Analyzers.OS/TASKS.md | TODO | OS Analyzer Guild | SCANNER-ANALYZERS-OS-10-207 | Package OS analyzers as restart-time plug-ins (manifest + host registration). | +| Sprint 10 | Scanner Analyzers & SBOM | src/StellaOps.Scanner.Analyzers.Lang/TASKS.md | TODO | Language Analyzer Guild | SCANNER-ANALYZERS-LANG-10-301 | Java analyzer emitting `pkg:maven` with provenance. | +| Sprint 10 | Scanner Analyzers & SBOM | src/StellaOps.Scanner.Analyzers.Lang/TASKS.md | TODO | Language Analyzer Guild | SCANNER-ANALYZERS-LANG-10-302 | Node analyzer handling workspaces/symlinks emitting `pkg:npm`. | +| Sprint 10 | Scanner Analyzers & SBOM | src/StellaOps.Scanner.Analyzers.Lang/TASKS.md | TODO | Language Analyzer Guild | SCANNER-ANALYZERS-LANG-10-303 | Python analyzer reading `*.dist-info`, RECORD hashes, entry points. | +| Sprint 10 | Scanner Analyzers & SBOM | src/StellaOps.Scanner.Analyzers.Lang/TASKS.md | TODO | Language Analyzer Guild | SCANNER-ANALYZERS-LANG-10-304 | Go analyzer leveraging buildinfo for `pkg:golang` components. | +| Sprint 10 | Scanner Analyzers & SBOM | src/StellaOps.Scanner.Analyzers.Lang/TASKS.md | TODO | Language Analyzer Guild | SCANNER-ANALYZERS-LANG-10-305 | .NET analyzer parsing `*.deps.json`, assembly metadata, RID variants. | +| Sprint 10 | Scanner Analyzers & SBOM | src/StellaOps.Scanner.Analyzers.Lang/TASKS.md | TODO | Language Analyzer Guild | SCANNER-ANALYZERS-LANG-10-306 | Rust analyzer detecting crates or falling back to `bin:{sha256}`. | +| Sprint 10 | Scanner Analyzers & SBOM | src/StellaOps.Scanner.Analyzers.Lang/TASKS.md | TODO | Language Analyzer Guild | SCANNER-ANALYZERS-LANG-10-307 | Shared language evidence helpers + usage flag propagation. | +| Sprint 10 | Scanner Analyzers & SBOM | src/StellaOps.Scanner.Analyzers.Lang/TASKS.md | TODO | Language Analyzer Guild | SCANNER-ANALYZERS-LANG-10-308 | Determinism + fixture harness for language analyzers. | +| Sprint 10 | Scanner Analyzers & SBOM | src/StellaOps.Scanner.Analyzers.Lang/TASKS.md | TODO | Language Analyzer Guild | SCANNER-ANALYZERS-LANG-10-309 | Package language analyzers as restart-time plug-ins (manifest + host registration). | +| Sprint 10 | Scanner Analyzers & SBOM | src/StellaOps.Scanner.EntryTrace/TASKS.md | TODO | EntryTrace Guild | SCANNER-ENTRYTRACE-10-401 | POSIX shell AST parser with deterministic output. | +| Sprint 10 | Scanner Analyzers & SBOM | src/StellaOps.Scanner.EntryTrace/TASKS.md | TODO | EntryTrace Guild | SCANNER-ENTRYTRACE-10-402 | Command resolution across layered rootfs with evidence attribution. | +| Sprint 10 | Scanner Analyzers & SBOM | src/StellaOps.Scanner.EntryTrace/TASKS.md | TODO | EntryTrace Guild | SCANNER-ENTRYTRACE-10-403 | Interpreter tracing for shell wrappers to Python/Node/Java launchers. | +| Sprint 10 | Scanner Analyzers & SBOM | src/StellaOps.Scanner.EntryTrace/TASKS.md | TODO | EntryTrace Guild | SCANNER-ENTRYTRACE-10-404 | Python entry analyzer (venv shebang, module invocation, usage flag). | +| Sprint 10 | Scanner Analyzers & SBOM | src/StellaOps.Scanner.EntryTrace/TASKS.md | TODO | EntryTrace Guild | SCANNER-ENTRYTRACE-10-405 | Node/Java launcher analyzer capturing script/jar targets. | +| Sprint 10 | Scanner Analyzers & SBOM | src/StellaOps.Scanner.EntryTrace/TASKS.md | TODO | EntryTrace Guild | SCANNER-ENTRYTRACE-10-406 | Explainability + diagnostics for unresolved constructs with metrics. | +| Sprint 10 | Scanner Analyzers & SBOM | src/StellaOps.Scanner.EntryTrace/TASKS.md | TODO | EntryTrace Guild | SCANNER-ENTRYTRACE-10-407 | Package EntryTrace analyzers as restart-time plug-ins (manifest + host registration). | +| Sprint 10 | Scanner Analyzers & SBOM | src/StellaOps.Scanner.Diff/TASKS.md | TODO | Diff Guild | SCANNER-DIFF-10-501 | Build component differ tracking add/remove/version changes with deterministic ordering. | +| Sprint 10 | Scanner Analyzers & SBOM | src/StellaOps.Scanner.Diff/TASKS.md | TODO | Diff Guild | SCANNER-DIFF-10-502 | Attribute diffs to introducing/removing layers including provenance evidence. | +| Sprint 10 | Scanner Analyzers & SBOM | src/StellaOps.Scanner.Diff/TASKS.md | TODO | Diff Guild | SCANNER-DIFF-10-503 | Produce JSON diff output for inventory vs usage views aligned with API contract. | +| Sprint 10 | Scanner Analyzers & SBOM | src/StellaOps.Scanner.Emit/TASKS.md | TODO | Emit Guild | SCANNER-EMIT-10-601 | Compose inventory SBOM (CycloneDX JSON/Protobuf) from layer fragments. | +| Sprint 10 | Scanner Analyzers & SBOM | src/StellaOps.Scanner.Emit/TASKS.md | TODO | Emit Guild | SCANNER-EMIT-10-602 | Compose usage SBOM leveraging EntryTrace to flag actual usage. | +| Sprint 10 | Scanner Analyzers & SBOM | src/StellaOps.Scanner.Emit/TASKS.md | TODO | Emit Guild | SCANNER-EMIT-10-603 | Generate BOM index sidecar (purl table + roaring bitmap + usage flag). | +| Sprint 10 | Scanner Analyzers & SBOM | src/StellaOps.Scanner.Emit/TASKS.md | TODO | Emit Guild | SCANNER-EMIT-10-604 | Package artifacts for export + attestation with deterministic manifests. | +| Sprint 10 | Scanner Analyzers & SBOM | src/StellaOps.Scanner.Emit/TASKS.md | TODO | Emit Guild | SCANNER-EMIT-10-605 | Emit BOM-Index sidecar schema/fixtures (CRITICAL PATH for SP16). | +| Sprint 10 | Scanner Analyzers & SBOM | src/StellaOps.Scanner.Emit/TASKS.md | TODO | Emit Guild | SCANNER-EMIT-10-606 | Usage view bit flags integrated with EntryTrace. | +| Sprint 10 | Scanner Analyzers & SBOM | src/StellaOps.Scanner.Emit/TASKS.md | TODO | Emit Guild | SCANNER-EMIT-10-607 | Embed scoring inputs, confidence band, and quiet provenance in CycloneDX/DSSE artifacts. | +| Sprint 10 | Benchmarks | bench/TASKS.md | TODO | Bench Guild, Scanner Team | BENCH-SCANNER-10-001 | Analyzer microbench harness + baseline CSV. | +| Sprint 10 | Samples | samples/TASKS.md | TODO | Samples Guild, Scanner Team | SAMPLES-10-001 | Sample images with SBOM/BOM-Index sidecars. | +| Sprint 10 | DevOps Security | ops/devops/TASKS.md | DOING | DevOps Guild | DEVOPS-SEC-10-301 | Address NU1902/NU1903 advisories for `MongoDB.Driver` 2.12.0 and `SharpCompress` 0.23.0; Wave 0A prerequisites confirmed complete before remediation work. | +| Sprint 10 | DevOps Perf | ops/devops/TASKS.md | TODO | DevOps Guild | DEVOPS-PERF-10-001 | Perf smoke job ensuring <5 s SBOM compose. | +| Sprint 11 | Signing Chain Bring-up | src/StellaOps.Authority/TASKS.md | DOING (2025-10-19) | Authority Core & Security Guild | AUTH-DPOP-11-001 | Implement DPoP proof validation + nonce handling for high-value audiences per architecture. | +| Sprint 11 | Signing Chain Bring-up | src/StellaOps.Authority/TASKS.md | DOING (2025-10-19) | Authority Core & Security Guild | AUTH-MTLS-11-002 | Add OAuth mTLS client credential support with certificate-bound tokens and introspection updates. | +| Sprint 11 | Signing Chain Bring-up | src/StellaOps.Signer/TASKS.md | TODO | Signer Guild | SIGNER-API-11-101 | `/sign/dsse` pipeline with Authority auth, PoE introspection, release verification, DSSE signing. | +| Sprint 11 | Signing Chain Bring-up | src/StellaOps.Signer/TASKS.md | TODO | Signer Guild | SIGNER-REF-11-102 | `/verify/referrers` endpoint with OCI lookup, caching, and policy enforcement. | +| Sprint 11 | Signing Chain Bring-up | src/StellaOps.Signer/TASKS.md | TODO | Signer Guild | SIGNER-QUOTA-11-103 | Enforce plan quotas, concurrency/QPS limits, artifact size caps with metrics/audit logs. | +| Sprint 11 | Signing Chain Bring-up | src/StellaOps.Attestor/TASKS.md | TODO | Attestor Guild | ATTESTOR-API-11-201 | `/rekor/entries` submission pipeline with dedupe, proof acquisition, and persistence. | +| Sprint 11 | Signing Chain Bring-up | src/StellaOps.Attestor/TASKS.md | TODO | Attestor Guild | ATTESTOR-VERIFY-11-202 | `/rekor/verify` + retrieval endpoints validating signatures and Merkle proofs. | +| Sprint 11 | Signing Chain Bring-up | src/StellaOps.Attestor/TASKS.md | TODO | Attestor Guild | ATTESTOR-OBS-11-203 | Telemetry, alerting, mTLS hardening, and archive workflow for Attestor. | +| Sprint 11 | UI Integration | src/StellaOps.UI/TASKS.md | TODO | UI Guild | UI-ATTEST-11-005 | Attestation visibility (Rekor id, status) on Scan Detail. | +| Sprint 12 | Runtime Guardrails | src/StellaOps.Zastava.Core/TASKS.md | TODO | Zastava Core Guild | ZASTAVA-CORE-12-201 | Define runtime event/admission DTOs, hashing helpers, and versioning strategy. | +| Sprint 12 | Runtime Guardrails | src/StellaOps.Zastava.Core/TASKS.md | TODO | Zastava Core Guild | ZASTAVA-CORE-12-202 | Provide configuration/logging/metrics utilities shared by Observer/Webhook. | +| Sprint 12 | Runtime Guardrails | src/StellaOps.Zastava.Core/TASKS.md | TODO | Zastava Core Guild | ZASTAVA-CORE-12-203 | Authority client helpers, OpTok caching, and security guardrails for runtime services. | +| Sprint 12 | Runtime Guardrails | src/StellaOps.Zastava.Core/TASKS.md | TODO | Zastava Core Guild | ZASTAVA-OPS-12-204 | Operational runbooks, alert rules, and dashboard exports for runtime plane. | +| Sprint 12 | Runtime Guardrails | src/StellaOps.Zastava.Observer/TASKS.md | TODO | Zastava Observer Guild | ZASTAVA-OBS-12-001 | Container lifecycle watcher emitting deterministic runtime events with buffering. | +| Sprint 12 | Runtime Guardrails | src/StellaOps.Zastava.Observer/TASKS.md | TODO | Zastava Observer Guild | ZASTAVA-OBS-12-002 | Capture entrypoint traces + loaded libraries, hashing binaries and linking to baseline SBOM. | +| Sprint 12 | Runtime Guardrails | src/StellaOps.Zastava.Observer/TASKS.md | TODO | Zastava Observer Guild | ZASTAVA-OBS-12-003 | Posture checks for signatures/SBOM/attestation with offline caching. | +| Sprint 12 | Runtime Guardrails | src/StellaOps.Zastava.Observer/TASKS.md | TODO | Zastava Observer Guild | ZASTAVA-OBS-12-004 | Batch `/runtime/events` submissions with disk-backed buffer and rate limits. | +| Sprint 12 | Runtime Guardrails | src/StellaOps.Zastava.Webhook/TASKS.md | TODO | Zastava Webhook Guild | ZASTAVA-WEBHOOK-12-101 | Admission controller host with TLS bootstrap and Authority auth. | +| Sprint 12 | Runtime Guardrails | src/StellaOps.Zastava.Webhook/TASKS.md | TODO | Zastava Webhook Guild | ZASTAVA-WEBHOOK-12-102 | Query Scanner `/policy/runtime`, resolve digests, enforce verdicts. | +| Sprint 12 | Runtime Guardrails | src/StellaOps.Zastava.Webhook/TASKS.md | TODO | Zastava Webhook Guild | ZASTAVA-WEBHOOK-12-103 | Caching, fail-open/closed toggles, metrics/logging for admission decisions. | +| Sprint 12 | Runtime Guardrails | src/StellaOps.Scanner.WebService/TASKS.md | TODO | Scanner WebService Guild | SCANNER-RUNTIME-12-301 | `/runtime/events` ingestion endpoint with validation, batching, storage hooks. | +| Sprint 12 | Runtime Guardrails | src/StellaOps.Scanner.WebService/TASKS.md | TODO | Scanner WebService Guild | SCANNER-RUNTIME-12-302 | `/policy/runtime` endpoint joining SBOM baseline + policy verdict with TTL guidance. | +| Sprint 13 | UX & CLI Experience | src/StellaOps.UI/TASKS.md | TODO | UI Guild | UI-AUTH-13-001 | Integrate Authority OIDC + DPoP flows with session management. | +| Sprint 13 | UX & CLI Experience | src/StellaOps.UI/TASKS.md | TODO | UI Guild | UI-SCANS-13-002 | Build scans module (list/detail/SBOM/diff/attestation) with performance + accessibility targets. | +| Sprint 13 | UX & CLI Experience | src/StellaOps.UI/TASKS.md | TODO | UI Guild | UI-VEX-13-003 | Implement VEX explorer + policy editor with preview integration. | +| Sprint 13 | UX & CLI Experience | src/StellaOps.UI/TASKS.md | TODO | UI Guild | UI-ADMIN-13-004 | Deliver admin area (tenants/clients/quotas/licensing) with RBAC + audit hooks. | +| Sprint 13 | UX & CLI Experience | src/StellaOps.UI/TASKS.md | TODO | UI Guild | UI-SCHED-13-005 | Scheduler panel: schedules CRUD, run history, dry-run preview. | +| Sprint 13 | UX & CLI Experience | src/StellaOps.UI/TASKS.md | DOING (2025-10-19) | UI Guild | UI-NOTIFY-13-006 | Notify panel: channels/rules CRUD, deliveries view, test send. | +| Sprint 13 | UX & CLI Experience | src/StellaOps.Cli/TASKS.md | TODO | DevEx/CLI | CLI-RUNTIME-13-005 | Add runtime policy test verbs that consume `/policy/runtime` and display verdicts. | +| Sprint 13 | UX & CLI Experience | src/StellaOps.Cli/TASKS.md | TODO | DevEx/CLI | CLI-OFFLINE-13-006 | Implement offline kit pull/import/status commands with integrity checks. | +| Sprint 13 | UX & CLI Experience | src/StellaOps.Cli/TASKS.md | TODO | DevEx/CLI | CLI-PLUGIN-13-007 | Package non-core CLI verbs as restart-time plug-ins (manifest + loader tests). | +| Sprint 14 | Release & Offline Ops | ops/devops/TASKS.md | TODO | DevOps Guild | DEVOPS-REL-14-001 | Deterministic build/release pipeline with SBOM/provenance, signing, and manifest generation. | +| Sprint 14 | Release & Offline Ops | ops/offline-kit/TASKS.md | TODO | Offline Kit Guild | DEVOPS-OFFLINE-14-002 | Offline kit packaging workflow with integrity verification and documentation. | +| Sprint 14 | Release & Offline Ops | ops/deployment/TASKS.md | TODO | Deployment Guild | DEVOPS-OPS-14-003 | Deployment/update/rollback automation and channel management documentation. | +| Sprint 14 | Release & Offline Ops | ops/licensing/TASKS.md | TODO | Licensing Guild | DEVOPS-LIC-14-004 | Registry token service tied to Authority, plan gating, revocation handling, monitoring. | +| Sprint 15 | Notify Foundations | src/StellaOps.Notify.Models/TASKS.md | TODO | Notify Models Guild | NOTIFY-MODELS-15-101 | Define core Notify DTOs, validation helpers, canonical serialization. | +| Sprint 15 | Notify Foundations | src/StellaOps.Notify.Models/TASKS.md | TODO | Notify Models Guild | NOTIFY-MODELS-15-102 | Publish schema docs and sample payloads for Notify. | +| Sprint 15 | Notify Foundations | src/StellaOps.Notify.Models/TASKS.md | TODO | Notify Models Guild | NOTIFY-MODELS-15-103 | Versioning/migration helpers for rules/templates/deliveries. | +| Sprint 15 | Notify Foundations | src/StellaOps.Notify.Storage.Mongo/TASKS.md | TODO | Notify Storage Guild | NOTIFY-STORAGE-15-201 | Mongo schemas/indexes for rules, channels, deliveries, digests, locks, audit. | +| Sprint 15 | Notify Foundations | src/StellaOps.Notify.Storage.Mongo/TASKS.md | TODO | Notify Storage Guild | NOTIFY-STORAGE-15-202 | Repositories with tenant scoping, soft delete, TTL, causal consistency options. | +| Sprint 15 | Notify Foundations | src/StellaOps.Notify.Storage.Mongo/TASKS.md | TODO | Notify Storage Guild | NOTIFY-STORAGE-15-203 | Delivery history retention and query APIs. | +| Sprint 15 | Notify Foundations | src/StellaOps.Notify.Queue/TASKS.md | TODO | Notify Queue Guild | NOTIFY-QUEUE-15-401 | Bus abstraction + Redis Streams adapter with ordering/idempotency. | +| Sprint 15 | Notify Foundations | src/StellaOps.Notify.Queue/TASKS.md | TODO | Notify Queue Guild | NOTIFY-QUEUE-15-402 | NATS JetStream adapter with health probes and failover. | +| Sprint 15 | Notify Foundations | src/StellaOps.Notify.Queue/TASKS.md | TODO | Notify Queue Guild | NOTIFY-QUEUE-15-403 | Delivery queue with retry/dead-letter + metrics. | +| Sprint 15 | Notify Foundations | src/StellaOps.Notify.Engine/TASKS.md | TODO | Notify Engine Guild | NOTIFY-ENGINE-15-301 | Rules evaluation core (filters, throttles, idempotency). | +| Sprint 15 | Notify Foundations | src/StellaOps.Notify.Engine/TASKS.md | TODO | Notify Engine Guild | NOTIFY-ENGINE-15-302 | Action planner + digest coalescer. | +| Sprint 15 | Notify Foundations | src/StellaOps.Notify.Engine/TASKS.md | TODO | Notify Engine Guild | NOTIFY-ENGINE-15-303 | Template rendering engine (Slack/Teams/Email/Webhook). | +| Sprint 15 | Notify Foundations | src/StellaOps.Notify.Engine/TASKS.md | TODO | Notify Engine Guild | NOTIFY-ENGINE-15-304 | Test-send sandbox + preview utilities. | +| Sprint 15 | Notify Foundations | src/StellaOps.Notify.WebService/TASKS.md | TODO | Notify WebService Guild | NOTIFY-WEB-15-101 | Minimal API host with Authority enforcement and plug-in loading. | +| Sprint 15 | Notify Foundations | src/StellaOps.Notify.WebService/TASKS.md | TODO | Notify WebService Guild | NOTIFY-WEB-15-102 | Rules/channel/template CRUD with audit logging. | +| Sprint 15 | Notify Foundations | src/StellaOps.Notify.WebService/TASKS.md | DONE (2025-10-19) | Notify WebService Guild | NOTIFY-WEB-15-103 | Delivery history & test-send endpoints. | +| Sprint 15 | Notify Foundations | src/StellaOps.Notify.WebService/TASKS.md | TODO | Notify WebService Guild | NOTIFY-WEB-15-104 | Configuration binding + startup diagnostics. | +| Sprint 15 | Notify Foundations | src/StellaOps.Notify.Worker/TASKS.md | TODO | Notify Worker Guild | NOTIFY-WORKER-15-201 | Bus subscription + leasing loop with backoff. | +| Sprint 15 | Notify Foundations | src/StellaOps.Notify.Worker/TASKS.md | TODO | Notify Worker Guild | NOTIFY-WORKER-15-202 | Rules evaluation pipeline integration. | +| Sprint 15 | Notify Foundations | src/StellaOps.Notify.Worker/TASKS.md | TODO | Notify Worker Guild | NOTIFY-WORKER-15-203 | Channel dispatch orchestration with retries. | +| Sprint 15 | Notify Foundations | src/StellaOps.Notify.Worker/TASKS.md | TODO | Notify Worker Guild | NOTIFY-WORKER-15-204 | Metrics/telemetry for Notify workers. | +| Sprint 15 | Notify Foundations | src/StellaOps.Notify.Connectors.Slack/TASKS.md | TODO | Notify Connectors Guild | NOTIFY-CONN-SLACK-15-501 | Slack connector with rate-limit aware delivery. | +| Sprint 15 | Notify Foundations | src/StellaOps.Notify.Connectors.Slack/TASKS.md | DOING (2025-10-19) | Notify Connectors Guild | NOTIFY-CONN-SLACK-15-502 | Slack health/test-send support. | +| Sprint 15 | Notify Foundations | src/StellaOps.Notify.Connectors.Teams/TASKS.md | TODO | Notify Connectors Guild | NOTIFY-CONN-TEAMS-15-601 | Teams connector with Adaptive Cards. | +| Sprint 15 | Notify Foundations | src/StellaOps.Notify.Connectors.Teams/TASKS.md | DOING (2025-10-19) | Notify Connectors Guild | NOTIFY-CONN-TEAMS-15-602 | Teams health/test-send support. | +| Sprint 15 | Notify Foundations | src/StellaOps.Notify.Connectors.Email/TASKS.md | TODO | Notify Connectors Guild | NOTIFY-CONN-EMAIL-15-701 | SMTP connector with TLS + rendering. | +| Sprint 15 | Notify Foundations | src/StellaOps.Notify.Connectors.Email/TASKS.md | DOING (2025-10-19) | Notify Connectors Guild | NOTIFY-CONN-EMAIL-15-702 | DKIM + health/test-send flows. | +| Sprint 15 | Notify Foundations | src/StellaOps.Notify.Connectors.Webhook/TASKS.md | TODO | Notify Connectors Guild | NOTIFY-CONN-WEBHOOK-15-801 | Webhook connector with signing/retries. | +| Sprint 15 | Notify Foundations | src/StellaOps.Notify.Connectors.Webhook/TASKS.md | DOING (2025-10-19) | Notify Connectors Guild | NOTIFY-CONN-WEBHOOK-15-802 | Webhook health/test-send support. | +| Sprint 15 | Notify Foundations | src/StellaOps.Notify.Connectors.Slack/TASKS.md | TODO | Notify Connectors Guild | NOTIFY-CONN-SLACK-15-503 | Package Slack connector as restart-time plug-in (manifest + host registration). | +| Sprint 15 | Notify Foundations | src/StellaOps.Notify.Connectors.Teams/TASKS.md | TODO | Notify Connectors Guild | NOTIFY-CONN-TEAMS-15-603 | Package Teams connector as restart-time plug-in (manifest + host registration). | +| Sprint 15 | Notify Foundations | src/StellaOps.Notify.Connectors.Email/TASKS.md | TODO | Notify Connectors Guild | NOTIFY-CONN-EMAIL-15-703 | Package Email connector as restart-time plug-in (manifest + host registration). | | Sprint 15 | Notify Foundations | src/StellaOps.Scanner.WebService/TASKS.md | DOING (2025-10-19) | Scanner WebService Guild | SCANNER-EVENTS-15-201 | Emit `scanner.report.ready` + `scanner.scan.completed` events. | -| Sprint 15 | Benchmarks | bench/TASKS.md | TODO | Bench Guild, Notify Team | BENCH-NOTIFY-15-001 | Notify dispatch throughput bench with results CSV. | -| Sprint 15 | Notify Foundations | src/StellaOps.Notify.Connectors.Webhook/TASKS.md | TODO | Notify Connectors Guild | NOTIFY-CONN-WEBHOOK-15-803 | Package Webhook connector as restart-time plug-in (manifest + host registration). | -| Sprint 16 | Scheduler Intelligence | src/StellaOps.Scheduler.Models/TASKS.md | TODO | Scheduler Models Guild | SCHED-MODELS-16-101 | Define Scheduler DTOs & validation. | -| Sprint 16 | Scheduler Intelligence | src/StellaOps.Scheduler.Models/TASKS.md | TODO | Scheduler Models Guild | SCHED-MODELS-16-102 | Publish schema docs/sample payloads. | -| Sprint 16 | Scheduler Intelligence | src/StellaOps.Scheduler.Models/TASKS.md | TODO | Scheduler Models Guild | SCHED-MODELS-16-103 | Versioning/migration helpers for schedules/runs. | -| Sprint 16 | Scheduler Intelligence | src/StellaOps.Scheduler.Storage.Mongo/TASKS.md | TODO | Scheduler Storage Guild | SCHED-STORAGE-16-201 | Mongo schemas/indexes for Scheduler state. | -| Sprint 16 | Scheduler Intelligence | src/StellaOps.Scheduler.Storage.Mongo/TASKS.md | TODO | Scheduler Storage Guild | SCHED-STORAGE-16-202 | Repositories with tenant scoping, TTL, causal consistency. | -| Sprint 16 | Scheduler Intelligence | src/StellaOps.Scheduler.Storage.Mongo/TASKS.md | TODO | Scheduler Storage Guild | SCHED-STORAGE-16-203 | Audit + stats materialization for UI. | -| Sprint 16 | Scheduler Intelligence | src/StellaOps.Scheduler.Queue/TASKS.md | TODO | Scheduler Queue Guild | SCHED-QUEUE-16-401 | Queue abstraction + Redis Streams adapter. | -| Sprint 16 | Scheduler Intelligence | src/StellaOps.Scheduler.Queue/TASKS.md | TODO | Scheduler Queue Guild | SCHED-QUEUE-16-402 | NATS JetStream adapter with health probes. | -| Sprint 16 | Scheduler Intelligence | src/StellaOps.Scheduler.Queue/TASKS.md | TODO | Scheduler Queue Guild | SCHED-QUEUE-16-403 | Dead-letter handling + metrics. | -| Sprint 16 | Scheduler Intelligence | src/StellaOps.Scheduler.ImpactIndex/TASKS.md | TODO | Scheduler ImpactIndex Guild | SCHED-IMPACT-16-301 | Ingest BOM-Index into roaring bitmap store. | -| Sprint 16 | Scheduler Intelligence | src/StellaOps.Scheduler.ImpactIndex/TASKS.md | TODO | Scheduler ImpactIndex Guild | SCHED-IMPACT-16-302 | Query APIs for ResolveByPurls/ResolveByVulns/ResolveAll. | -| Sprint 16 | Scheduler Intelligence | src/StellaOps.Scheduler.ImpactIndex/TASKS.md | TODO | Scheduler ImpactIndex Guild | SCHED-IMPACT-16-303 | Snapshot/compaction/invalidation workflow. | -| Sprint 16 | Scheduler Intelligence | src/StellaOps.Scheduler.ImpactIndex/TASKS.md | DOING | Scheduler ImpactIndex Guild | SCHED-IMPACT-16-300 | **STUB** ImpactIndex ingest/query using fixtures (to be removed by SP16 completion). | -| Sprint 16 | Scheduler Intelligence | src/StellaOps.Scheduler.WebService/TASKS.md | TODO | Scheduler WebService Guild | SCHED-WEB-16-101 | Minimal API host with Authority enforcement. | -| Sprint 16 | Scheduler Intelligence | src/StellaOps.Scheduler.WebService/TASKS.md | TODO | Scheduler WebService Guild | SCHED-WEB-16-102 | Schedules CRUD (cron validation, pause/resume, audit). | -| Sprint 16 | Scheduler Intelligence | src/StellaOps.Scheduler.WebService/TASKS.md | TODO | Scheduler WebService Guild | SCHED-WEB-16-103 | Runs API (list/detail/cancel) + impact previews. | -| Sprint 16 | Scheduler Intelligence | src/StellaOps.Scheduler.WebService/TASKS.md | TODO | Scheduler WebService Guild | SCHED-WEB-16-104 | Feedser/Vexer webhook handlers with security enforcement. | -| Sprint 16 | Scheduler Intelligence | src/StellaOps.Scheduler.Worker/TASKS.md | TODO | Scheduler Worker Guild | SCHED-WORKER-16-201 | Planner loop (cron/event triggers, leases, fairness). | -| Sprint 16 | Scheduler Intelligence | src/StellaOps.Scheduler.Worker/TASKS.md | TODO | Scheduler Worker Guild | SCHED-WORKER-16-202 | ImpactIndex targeting and shard planning. | -| Sprint 16 | Scheduler Intelligence | src/StellaOps.Scheduler.Worker/TASKS.md | TODO | Scheduler Worker Guild | SCHED-WORKER-16-203 | Runner execution invoking Scanner analysis/content refresh. | -| Sprint 16 | Scheduler Intelligence | src/StellaOps.Scheduler.Worker/TASKS.md | TODO | Scheduler Worker Guild | SCHED-WORKER-16-204 | Emit rescan/report events for Notify/UI. | -| Sprint 16 | Scheduler Intelligence | src/StellaOps.Scheduler.Worker/TASKS.md | TODO | Scheduler Worker Guild | SCHED-WORKER-16-205 | Metrics/telemetry for Scheduler planners/runners. | -| Sprint 16 | Benchmarks | bench/TASKS.md | TODO | Bench Guild, Scheduler Team | BENCH-IMPACT-16-001 | ImpactIndex throughput bench + RAM profile. | -| Sprint 17 | Symbol Intelligence & Forensics | src/StellaOps.Scanner.Emit/TASKS.md | TODO | Emit Guild | SCANNER-EMIT-17-701 | Record GNU build-id for ELF components and surface it in SBOM/diff outputs. | -| Sprint 17 | Symbol Intelligence & Forensics | src/StellaOps.Zastava.Observer/TASKS.md | TODO | Zastava Observer Guild | ZASTAVA-OBS-17-005 | Collect GNU build-id during runtime observation and attach it to emitted events. | -| Sprint 17 | Symbol Intelligence & Forensics | src/StellaOps.Scanner.WebService/TASKS.md | TODO | Scanner WebService Guild | SCANNER-RUNTIME-17-401 | Persist runtime build-id observations and expose them for debug-symbol correlation. | -| Sprint 17 | Symbol Intelligence & Forensics | ops/devops/TASKS.md | TODO | DevOps Guild | DEVOPS-REL-17-002 | Ship stripped debug artifacts organised by build-id within release/offline kits. | -| Sprint 17 | Symbol Intelligence & Forensics | docs/TASKS.md | TODO | Docs Guild | DOCS-RUNTIME-17-004 | Document build-id workflows for SBOMs, runtime events, and debug-store usage. | +| Sprint 15 | Benchmarks | bench/TASKS.md | TODO | Bench Guild, Notify Team | BENCH-NOTIFY-15-001 | Notify dispatch throughput bench with results CSV. | +| Sprint 15 | Notify Foundations | src/StellaOps.Notify.Connectors.Webhook/TASKS.md | TODO | Notify Connectors Guild | NOTIFY-CONN-WEBHOOK-15-803 | Package Webhook connector as restart-time plug-in (manifest + host registration). | +| Sprint 16 | Scheduler Intelligence | src/StellaOps.Scheduler.Models/TASKS.md | TODO | Scheduler Models Guild | SCHED-MODELS-16-101 | Define Scheduler DTOs & validation. | +| Sprint 16 | Scheduler Intelligence | src/StellaOps.Scheduler.Models/TASKS.md | TODO | Scheduler Models Guild | SCHED-MODELS-16-102 | Publish schema docs/sample payloads. | +| Sprint 16 | Scheduler Intelligence | src/StellaOps.Scheduler.Models/TASKS.md | TODO | Scheduler Models Guild | SCHED-MODELS-16-103 | Versioning/migration helpers for schedules/runs. | +| Sprint 16 | Scheduler Intelligence | src/StellaOps.Scheduler.Storage.Mongo/TASKS.md | TODO | Scheduler Storage Guild | SCHED-STORAGE-16-201 | Mongo schemas/indexes for Scheduler state. | +| Sprint 16 | Scheduler Intelligence | src/StellaOps.Scheduler.Storage.Mongo/TASKS.md | TODO | Scheduler Storage Guild | SCHED-STORAGE-16-202 | Repositories with tenant scoping, TTL, causal consistency. | +| Sprint 16 | Scheduler Intelligence | src/StellaOps.Scheduler.Storage.Mongo/TASKS.md | TODO | Scheduler Storage Guild | SCHED-STORAGE-16-203 | Audit + stats materialization for UI. | +| Sprint 16 | Scheduler Intelligence | src/StellaOps.Scheduler.Queue/TASKS.md | TODO | Scheduler Queue Guild | SCHED-QUEUE-16-401 | Queue abstraction + Redis Streams adapter. | +| Sprint 16 | Scheduler Intelligence | src/StellaOps.Scheduler.Queue/TASKS.md | TODO | Scheduler Queue Guild | SCHED-QUEUE-16-402 | NATS JetStream adapter with health probes. | +| Sprint 16 | Scheduler Intelligence | src/StellaOps.Scheduler.Queue/TASKS.md | TODO | Scheduler Queue Guild | SCHED-QUEUE-16-403 | Dead-letter handling + metrics. | +| Sprint 16 | Scheduler Intelligence | src/StellaOps.Scheduler.ImpactIndex/TASKS.md | TODO | Scheduler ImpactIndex Guild | SCHED-IMPACT-16-301 | Ingest BOM-Index into roaring bitmap store. | +| Sprint 16 | Scheduler Intelligence | src/StellaOps.Scheduler.ImpactIndex/TASKS.md | TODO | Scheduler ImpactIndex Guild | SCHED-IMPACT-16-302 | Query APIs for ResolveByPurls/ResolveByVulns/ResolveAll. | +| Sprint 16 | Scheduler Intelligence | src/StellaOps.Scheduler.ImpactIndex/TASKS.md | TODO | Scheduler ImpactIndex Guild | SCHED-IMPACT-16-303 | Snapshot/compaction/invalidation workflow. | +| Sprint 16 | Scheduler Intelligence | src/StellaOps.Scheduler.ImpactIndex/TASKS.md | DOING | Scheduler ImpactIndex Guild | SCHED-IMPACT-16-300 | **STUB** ImpactIndex ingest/query using fixtures (to be removed by SP16 completion). | +| Sprint 16 | Scheduler Intelligence | src/StellaOps.Scheduler.WebService/TASKS.md | TODO | Scheduler WebService Guild | SCHED-WEB-16-101 | Minimal API host with Authority enforcement. | +| Sprint 16 | Scheduler Intelligence | src/StellaOps.Scheduler.WebService/TASKS.md | TODO | Scheduler WebService Guild | SCHED-WEB-16-102 | Schedules CRUD (cron validation, pause/resume, audit). | +| Sprint 16 | Scheduler Intelligence | src/StellaOps.Scheduler.WebService/TASKS.md | TODO | Scheduler WebService Guild | SCHED-WEB-16-103 | Runs API (list/detail/cancel) + impact previews. | +| Sprint 16 | Scheduler Intelligence | src/StellaOps.Scheduler.WebService/TASKS.md | TODO | Scheduler WebService Guild | SCHED-WEB-16-104 | Feedser/Vexer webhook handlers with security enforcement. | +| Sprint 16 | Scheduler Intelligence | src/StellaOps.Scheduler.Worker/TASKS.md | TODO | Scheduler Worker Guild | SCHED-WORKER-16-201 | Planner loop (cron/event triggers, leases, fairness). | +| Sprint 16 | Scheduler Intelligence | src/StellaOps.Scheduler.Worker/TASKS.md | TODO | Scheduler Worker Guild | SCHED-WORKER-16-202 | ImpactIndex targeting and shard planning. | +| Sprint 16 | Scheduler Intelligence | src/StellaOps.Scheduler.Worker/TASKS.md | TODO | Scheduler Worker Guild | SCHED-WORKER-16-203 | Runner execution invoking Scanner analysis/content refresh. | +| Sprint 16 | Scheduler Intelligence | src/StellaOps.Scheduler.Worker/TASKS.md | TODO | Scheduler Worker Guild | SCHED-WORKER-16-204 | Emit rescan/report events for Notify/UI. | +| Sprint 16 | Scheduler Intelligence | src/StellaOps.Scheduler.Worker/TASKS.md | TODO | Scheduler Worker Guild | SCHED-WORKER-16-205 | Metrics/telemetry for Scheduler planners/runners. | +| Sprint 16 | Benchmarks | bench/TASKS.md | TODO | Bench Guild, Scheduler Team | BENCH-IMPACT-16-001 | ImpactIndex throughput bench + RAM profile. | +| Sprint 17 | Symbol Intelligence & Forensics | src/StellaOps.Scanner.Emit/TASKS.md | TODO | Emit Guild | SCANNER-EMIT-17-701 | Record GNU build-id for ELF components and surface it in SBOM/diff outputs. | +| Sprint 17 | Symbol Intelligence & Forensics | src/StellaOps.Zastava.Observer/TASKS.md | TODO | Zastava Observer Guild | ZASTAVA-OBS-17-005 | Collect GNU build-id during runtime observation and attach it to emitted events. | +| Sprint 17 | Symbol Intelligence & Forensics | src/StellaOps.Scanner.WebService/TASKS.md | TODO | Scanner WebService Guild | SCANNER-RUNTIME-17-401 | Persist runtime build-id observations and expose them for debug-symbol correlation. | +| Sprint 17 | Symbol Intelligence & Forensics | ops/devops/TASKS.md | TODO | DevOps Guild | DEVOPS-REL-17-002 | Ship stripped debug artifacts organised by build-id within release/offline kits. | +| Sprint 17 | Symbol Intelligence & Forensics | docs/TASKS.md | TODO | Docs Guild | DOCS-RUNTIME-17-004 | Document build-id workflows for SBOMs, runtime events, and debug-store usage. | diff --git a/SPRINTS.updated.tmp b/SPRINTS.updated.tmp index 7c385fa1..b2cb3923 100644 --- a/SPRINTS.updated.tmp +++ b/SPRINTS.updated.tmp @@ -53,7 +53,7 @@ This file describe implementation of Stella Ops (docs/README.md). Implementation | Sprint 1 | Stabilize In-Progress Foundations | src/StellaOps.Concelier.WebService/TASKS.md | DONE (2025-10-12) | Team WebService & Authority | FEEDWEB-DOCS-01-003 | Author ops guidance for resilience tuning
Operator docs now outline connected vs air-gapped resilience profiles and monitoring cues. | | Sprint 1 | Stabilize In-Progress Foundations | src/StellaOps.Concelier.WebService/TASKS.md | DONE (2025-10-12) | Team WebService & Authority | FEEDWEB-DOCS-01-004 | Document authority bypass logging patterns
Audit logging guidance highlights `route/status/subject/clientId/scopes/bypass/remote` fields and SIEM alerts. | | Sprint 1 | Stabilize In-Progress Foundations | src/StellaOps.Concelier.WebService/TASKS.md | DONE (2025-10-12) | Team WebService & Authority | FEEDWEB-DOCS-01-005 | Update Concelier operator guide for enforcement cutoff
Install guide reiterates the 2025-12-31 cutoff and ties audit signals to rollout checks. | -| Sprint 1 | Stabilize In-Progress Foundations | src/StellaOps.Concelier.WebService/TASKS.md | TODO | Team WebService & Authority | FEEDWEB-OPS-01-006 | Rename plugin drop directory to namespaced path
Repoint build outputs to `StellaOps.Concelier.PluginBinaries`/`StellaOps.Authority.PluginBinaries`, update PluginHost defaults, Offline Kit packaging, and operator docs. | +| Sprint 1 | Stabilize In-Progress Foundations | src/StellaOps.Concelier.WebService/TASKS.md | DONE (2025-10-19) | Team WebService & Authority | FEEDWEB-OPS-01-006 | Rename plugin drop directory to namespaced path
Build outputs now point at `StellaOps.Concelier.PluginBinaries`/`StellaOps.Authority.PluginBinaries`; defaults/docs/tests updated to reflect the new layout. | | Sprint 1 | Stabilize In-Progress Foundations | src/StellaOps.Concelier.WebService/TASKS.md | BLOCKED (2025-10-10) | Team WebService & Authority | FEEDWEB-OPS-01-007 | Authority resilience adoption
Roll out retry/offline knobs to deployment docs and align CLI parity once LIB5 resilience options land; unblock when library release is available and docs review completes. | | Sprint 1 | Stabilize In-Progress Foundations | src/StellaOps.Authority/TASKS.md | DONE (2025-10-11) | Team Authority Platform & Security Guild | AUTHCORE-ENGINE-01-001 | CORE8.RL — Rate limiter plumbing validated; integration tests green and docs handoff recorded for middleware ordering + Retry-After headers (see `docs/dev/authority-rate-limit-tuning-outline.md` for continuing guidance). | | Sprint 1 | Stabilize In-Progress Foundations | src/StellaOps.Cryptography/TASKS.md | DONE (2025-10-11) | Team Authority Platform & Security Guild | AUTHCRYPTO-ENGINE-01-001 | SEC3.A — Shared metadata resolver confirmed via host test run; SEC3.B now unblocked for tuning guidance (outline captured in `docs/dev/authority-rate-limit-tuning-outline.md`). | @@ -191,20 +191,21 @@ This file describe implementation of Stella Ops (docs/README.md). Implementation | Sprint 7 | Contextual Truth Foundations | src/StellaOps.Excititor.Connectors.StellaOpsMirror/TASKS.md | TODO | Excititor Connectors – Stella | EXCITITOR-CONN-STELLA-07-001 | Excititor mirror connector – ingest signed mirror bundles and map to VexClaims with resume handling. | | Sprint 7 | Contextual Truth Foundations | src/StellaOps.Excititor.Connectors.StellaOpsMirror/TASKS.md | TODO | Excititor Connectors – Stella | EXCITITOR-CONN-STELLA-07-002 | Normalize mirror bundles into VexClaim sets referencing original provider metadata and mirror provenance. | | Sprint 7 | Contextual Truth Foundations | src/StellaOps.Excititor.Connectors.StellaOpsMirror/TASKS.md | TODO | Excititor Connectors – Stella | EXCITITOR-CONN-STELLA-07-003 | Implement incremental cursor handling per-export digest, support resume, and document configuration for downstream Excititor mirrors. | -| Sprint 7 | Contextual Truth Foundations | src/StellaOps.Concelier.Core/TASKS.md | TODO | Team Core Engine & Storage Analytics | FEEDCORE-ENGINE-07-001 | Advisory event log & asOf queries – surface immutable statements and replay capability. | +| Sprint 7 | Contextual Truth Foundations | src/StellaOps.Concelier.Core/TASKS.md | DONE (2025-10-19) | Team Core Engine & Storage Analytics | FEEDCORE-ENGINE-07-001 | Advisory event log & asOf queries – surface immutable statements and replay capability. | +| Sprint 7 | Contextual Truth Foundations | src/StellaOps.Concelier.WebService/TASKS.md | DONE (2025-10-19) | Concelier WebService Guild | FEEDWEB-EVENTS-07-001 | Advisory event replay API – expose `/concelier/advisories/{key}/replay` with `asOf` filter, hex hashes, and conflict data. | | Sprint 7 | Contextual Truth Foundations | src/StellaOps.Concelier.Core/TASKS.md | TODO | Team Core Engine & Data Science | FEEDCORE-ENGINE-07-002 | Noise prior computation service – learn false-positive priors and expose deterministic summaries. | | Sprint 7 | Contextual Truth Foundations | src/StellaOps.Concelier.Core/TASKS.md | TODO | Team Core Engine & Storage Analytics | FEEDCORE-ENGINE-07-003 | Unknown state ledger & confidence seeding – persist unknown flags, seed confidence bands, expose query surface. | | Sprint 7 | Contextual Truth Foundations | src/StellaOps.Concelier.Storage.Mongo/TASKS.md | TODO | Team Normalization & Storage Backbone | FEEDSTORAGE-DATA-07-001 | Advisory statement & conflict collections – provision Mongo schema/indexes for event-sourced merge. | | Sprint 7 | Contextual Truth Foundations | src/StellaOps.Concelier.Merge/TASKS.md | TODO | BE-Merge | FEEDMERGE-ENGINE-07-001 | Conflict sets & explainers – persist conflict materialization and replay hashes for merge decisions. | | Sprint 8 | Plugin Infrastructure | src/StellaOps.Plugin/TASKS.md | TODO | Plugin Platform Guild | PLUGIN-DI-08-001 | Scoped service support in plugin bootstrap
Teach the plugin loader/registrar to surface services with scoped lifetimes, honour `StellaOps.DependencyInjection` metadata, and document the new contract. | | Sprint 8 | Plugin Infrastructure | src/StellaOps.Plugin/TASKS.md | TODO | Plugin Platform Guild, Authority Core | PLUGIN-DI-08-002 | Update Authority plugin integration
Flow scoped services through identity-provider registrars, bootstrap flows, and background jobs; add regression coverage around scoped lifetimes. | -| Sprint 8 | Mongo strengthening | src/StellaOps.Concelier.Storage.Mongo/TASKS.md | TODO | Team Normalization & Storage Backbone | FEEDSTORAGE-MONGO-08-001 | Causal-consistent Concelier storage sessions
Ensure `AddMongoStorage` registers a scoped session facilitator (causal consistency + majority concerns), update repositories to accept optional session handles, and add integration coverage proving read-your-write and monotonic reads across a replica set/election scenario. | -| Sprint 8 | Mongo strengthening | src/StellaOps.Authority/TASKS.md | BLOCKED (2025-10-19) | Authority Core & Storage Guild | AUTHSTORAGE-MONGO-08-001 | Harden Authority Mongo usage
Scoped sessions with causal consistency pending rate-limiter stream updates; resume once plugin lockout telemetry stabilises. | +| Sprint 8 | Mongo strengthening | src/StellaOps.Concelier.Storage.Mongo/TASKS.md | DONE (2025-10-19) | Team Normalization & Storage Backbone | FEEDSTORAGE-MONGO-08-001 | Causal-consistent Concelier storage sessions
Scoped session facilitator registered, repositories accept optional session handles, and replica-set failover tests verify read-your-write + monotonic reads. | +| Sprint 8 | Mongo strengthening | src/StellaOps.Authority/TASKS.md | DONE (2025-10-19) | Authority Core & Storage Guild | AUTHSTORAGE-MONGO-08-001 | Harden Authority Mongo usage
Scoped Mongo sessions with majority read/write concerns wired through stores and GraphQL/HTTP pipelines; replica-set election regression validated. | | Sprint 8 | Mongo strengthening | src/StellaOps.Excititor.Storage.Mongo/TASKS.md | TODO | Team Excititor Storage | EXCITITOR-STORAGE-MONGO-08-001 | Causal consistency for Excititor repositories
Register Mongo options with majority defaults, push session-aware overloads through raw/export/consensus/cache stores, and extend migration/tests to validate causal reads after writes (including GridFS-backed content) under replica-set failover. | | Sprint 8 | Platform Maintenance | src/StellaOps.Excititor.Worker/TASKS.md | TODO | Team Excititor Worker | EXCITITOR-WORKER-02-001 | Resolve Microsoft.Extensions.Caching.Memory advisory – bump to latest .NET 10 preview, regenerate lockfiles, and rerun worker/webservice tests to clear NU1903. | | Sprint 8 | Platform Maintenance | src/StellaOps.Excititor.Storage.Mongo/TASKS.md | TODO | Team Excititor Storage | EXCITITOR-STORAGE-03-001 | Statement backfill tooling – provide CLI/backfill scripts that populate the `vex.statements` log via WebService ingestion and validate severity/KEV/EPSS signal replay. | | Sprint 8 | Mirror Distribution | src/StellaOps.Concelier.Exporter.Json/TASKS.md | TODO | Concelier Export Guild | CONCELIER-EXPORT-08-201 | Mirror bundle + domain manifest – produce signed JSON aggregates for `*.stella-ops.org` mirrors. | -| Sprint 8 | Mirror Distribution | src/StellaOps.Concelier.Exporter.TrivyDb/TASKS.md | TODO | Concelier Export Guild | CONCELIER-EXPORT-08-202 | Mirror-ready Trivy DB bundles – ship domain-specific archives + metadata for downstream sync. | +| Sprint 8 | Mirror Distribution | src/StellaOps.Concelier.Exporter.TrivyDb/TASKS.md | DONE (2025-10-19) | Concelier Export Guild | CONCELIER-EXPORT-08-202 | Mirror-ready Trivy DB bundles – mirror options emit per-domain manifests/metadata/db archives with deterministic digests for downstream sync. | | Sprint 8 | Mirror Distribution | src/StellaOps.Concelier.WebService/TASKS.md | TODO | Concelier WebService Guild | CONCELIER-WEB-08-201 | Mirror distribution endpoints – expose domain-scoped index/download APIs with auth/quota. | | Sprint 8 | Mirror Distribution | src/StellaOps.Concelier.Connector.StellaOpsMirror/TASKS.md | TODO | BE-Conn-Stella | FEEDCONN-STELLA-08-001 | Concelier mirror connector – fetch mirror manifest, verify signatures, and hydrate canonical DTOs with resume support. | | Sprint 8 | Mirror Distribution | src/StellaOps.Concelier.Connector.StellaOpsMirror/TASKS.md | TODO | BE-Conn-Stella | FEEDCONN-STELLA-08-002 | Map mirror payloads into canonical advisory DTOs with provenance referencing mirror domain + original source metadata. | @@ -350,7 +351,7 @@ This file describe implementation of Stella Ops (docs/README.md). Implementation | Sprint 13 | UX & CLI Experience | src/StellaOps.UI/TASKS.md | TODO | UI Guild | UI-VEX-13-003 | Implement VEX explorer + policy editor with preview integration. | | Sprint 13 | UX & CLI Experience | src/StellaOps.UI/TASKS.md | TODO | UI Guild | UI-ADMIN-13-004 | Deliver admin area (tenants/clients/quotas/licensing) with RBAC + audit hooks. | | Sprint 13 | UX & CLI Experience | src/StellaOps.UI/TASKS.md | TODO | UI Guild | UI-SCHED-13-005 | Scheduler panel: schedules CRUD, run history, dry-run preview. | -| Sprint 13 | UX & CLI Experience | src/StellaOps.UI/TASKS.md | TODO | UI Guild | UI-NOTIFY-13-006 | Notify panel: channels/rules CRUD, deliveries view, test send. | +| Sprint 13 | UX & CLI Experience | src/StellaOps.UI/TASKS.md | DOING (2025-10-19) | UI Guild | UI-NOTIFY-13-006 | Notify panel: channels/rules CRUD, deliveries view, test send. | | Sprint 13 | UX & CLI Experience | src/StellaOps.UI/TASKS.md | TODO | UI Guild | UI-POLICY-13-007 | Surface policy confidence metadata (band, age, quiet provenance) on preview and report views. | | Sprint 13 | UX & CLI Experience | src/StellaOps.Cli/TASKS.md | TODO | DevEx/CLI | CLI-RUNTIME-13-005 | Add runtime policy test verbs that consume `/policy/runtime` and display verdicts. | | Sprint 13 | UX & CLI Experience | src/StellaOps.Cli/TASKS.md | TODO | DevEx/CLI | CLI-OFFLINE-13-006 | Implement offline kit pull/import/status commands with integrity checks. | @@ -376,20 +377,20 @@ This file describe implementation of Stella Ops (docs/README.md). Implementation | Sprint 15 | Notify Foundations | src/StellaOps.Notify.Engine/TASKS.md | TODO | Notify Engine Guild | NOTIFY-ENGINE-15-304 | Test-send sandbox + preview utilities. | | Sprint 15 | Notify Foundations | src/StellaOps.Notify.WebService/TASKS.md | TODO | Notify WebService Guild | NOTIFY-WEB-15-101 | Minimal API host with Authority enforcement and plug-in loading. | | Sprint 15 | Notify Foundations | src/StellaOps.Notify.WebService/TASKS.md | TODO | Notify WebService Guild | NOTIFY-WEB-15-102 | Rules/channel/template CRUD with audit logging. | -| Sprint 15 | Notify Foundations | src/StellaOps.Notify.WebService/TASKS.md | TODO | Notify WebService Guild | NOTIFY-WEB-15-103 | Delivery history & test-send endpoints. | +| Sprint 15 | Notify Foundations | src/StellaOps.Notify.WebService/TASKS.md | DONE (2025-10-19) | Notify WebService Guild | NOTIFY-WEB-15-103 | Delivery history & test-send endpoints. | | Sprint 15 | Notify Foundations | src/StellaOps.Notify.WebService/TASKS.md | TODO | Notify WebService Guild | NOTIFY-WEB-15-104 | Configuration binding + startup diagnostics. | | Sprint 15 | Notify Foundations | src/StellaOps.Notify.Worker/TASKS.md | TODO | Notify Worker Guild | NOTIFY-WORKER-15-201 | Bus subscription + leasing loop with backoff. | | Sprint 15 | Notify Foundations | src/StellaOps.Notify.Worker/TASKS.md | TODO | Notify Worker Guild | NOTIFY-WORKER-15-202 | Rules evaluation pipeline integration. | | Sprint 15 | Notify Foundations | src/StellaOps.Notify.Worker/TASKS.md | TODO | Notify Worker Guild | NOTIFY-WORKER-15-203 | Channel dispatch orchestration with retries. | | Sprint 15 | Notify Foundations | src/StellaOps.Notify.Worker/TASKS.md | TODO | Notify Worker Guild | NOTIFY-WORKER-15-204 | Metrics/telemetry for Notify workers. | | Sprint 15 | Notify Foundations | src/StellaOps.Notify.Connectors.Slack/TASKS.md | TODO | Notify Connectors Guild | NOTIFY-CONN-SLACK-15-501 | Slack connector with rate-limit aware delivery. | -| Sprint 15 | Notify Foundations | src/StellaOps.Notify.Connectors.Slack/TASKS.md | TODO | Notify Connectors Guild | NOTIFY-CONN-SLACK-15-502 | Slack health/test-send support. | +| Sprint 15 | Notify Foundations | src/StellaOps.Notify.Connectors.Slack/TASKS.md | DOING (2025-10-19) | Notify Connectors Guild | NOTIFY-CONN-SLACK-15-502 | Slack health/test-send support. | | Sprint 15 | Notify Foundations | src/StellaOps.Notify.Connectors.Teams/TASKS.md | TODO | Notify Connectors Guild | NOTIFY-CONN-TEAMS-15-601 | Teams connector with Adaptive Cards. | -| Sprint 15 | Notify Foundations | src/StellaOps.Notify.Connectors.Teams/TASKS.md | TODO | Notify Connectors Guild | NOTIFY-CONN-TEAMS-15-602 | Teams health/test-send support. | +| Sprint 15 | Notify Foundations | src/StellaOps.Notify.Connectors.Teams/TASKS.md | DOING (2025-10-19) | Notify Connectors Guild | NOTIFY-CONN-TEAMS-15-602 | Teams health/test-send support. | | Sprint 15 | Notify Foundations | src/StellaOps.Notify.Connectors.Email/TASKS.md | TODO | Notify Connectors Guild | NOTIFY-CONN-EMAIL-15-701 | SMTP connector with TLS + rendering. | -| Sprint 15 | Notify Foundations | src/StellaOps.Notify.Connectors.Email/TASKS.md | TODO | Notify Connectors Guild | NOTIFY-CONN-EMAIL-15-702 | DKIM + health/test-send flows. | +| Sprint 15 | Notify Foundations | src/StellaOps.Notify.Connectors.Email/TASKS.md | DOING (2025-10-19) | Notify Connectors Guild | NOTIFY-CONN-EMAIL-15-702 | DKIM + health/test-send flows. | | Sprint 15 | Notify Foundations | src/StellaOps.Notify.Connectors.Webhook/TASKS.md | TODO | Notify Connectors Guild | NOTIFY-CONN-WEBHOOK-15-801 | Webhook connector with signing/retries. | -| Sprint 15 | Notify Foundations | src/StellaOps.Notify.Connectors.Webhook/TASKS.md | TODO | Notify Connectors Guild | NOTIFY-CONN-WEBHOOK-15-802 | Webhook health/test-send support. | +| Sprint 15 | Notify Foundations | src/StellaOps.Notify.Connectors.Webhook/TASKS.md | DOING (2025-10-19) | Notify Connectors Guild | NOTIFY-CONN-WEBHOOK-15-802 | Webhook health/test-send support. | | Sprint 15 | Notify Foundations | src/StellaOps.Notify.Connectors.Slack/TASKS.md | TODO | Notify Connectors Guild | NOTIFY-CONN-SLACK-15-503 | Package Slack connector as restart-time plug-in (manifest + host registration). | | Sprint 15 | Notify Foundations | src/StellaOps.Notify.Connectors.Teams/TASKS.md | TODO | Notify Connectors Guild | NOTIFY-CONN-TEAMS-15-603 | Package Teams connector as restart-time plug-in (manifest + host registration). | | Sprint 15 | Notify Foundations | src/StellaOps.Notify.Connectors.Email/TASKS.md | TODO | Notify Connectors Guild | NOTIFY-CONN-EMAIL-15-703 | Package Email connector as restart-time plug-in (manifest + host registration). | diff --git a/SPRINTS_PRIOR_20251019.md b/SPRINTS_PRIOR_20251019.md new file mode 100644 index 00000000..3b0b87d7 --- /dev/null +++ b/SPRINTS_PRIOR_20251019.md @@ -0,0 +1,177 @@ +Closed sprint tasks archived from SPRINTS.md on 2025-10-19. + +| Sprint | Theme | Tasks File Path | Status | Type of Specialist | Task ID | Task Description | +| --- | --- | --- | --- | --- | --- | --- | +| Sprint 1 | Stabilize In-Progress Foundations | src/StellaOps.Concelier.Models/TASKS.md | DONE (2025-10-12) | Team Models & Merge Leads | FEEDMODELS-SCHEMA-01-001 | SemVer primitive range-style metadata
Instructions to work:
DONE Read ./AGENTS.md and src/StellaOps.Concelier.Models/AGENTS.md. This task lays the groundwork—complete the SemVer helper updates before teammates pick up FEEDMODELS-SCHEMA-01-002/003 and FEEDMODELS-SCHEMA-02-900. Use ./src/FASTER_MODELING_AND_NORMALIZATION.md for the target rule structure. | +| Sprint 1 | Stabilize In-Progress Foundations | src/StellaOps.Concelier.Models/TASKS.md | DONE (2025-10-11) | Team Models & Merge Leads | FEEDMODELS-SCHEMA-01-002 | Provenance decision rationale field
Instructions to work:
AdvisoryProvenance now carries `decisionReason` and docs/tests were updated. Connectors and merge tasks should populate the field when applying precedence/freshness/tie-breaker logic; see src/StellaOps.Concelier.Models/PROVENANCE_GUIDELINES.md for usage guidance. | +| Sprint 1 | Stabilize In-Progress Foundations | src/StellaOps.Concelier.Models/TASKS.md | DONE (2025-10-11) | Team Models & Merge Leads | FEEDMODELS-SCHEMA-01-003 | Normalized version rules collection
Instructions to work:
`AffectedPackage.NormalizedVersions` and supporting comparer/docs/tests shipped. Connector owners must emit rule arrays per ./src/FASTER_MODELING_AND_NORMALIZATION.md and report progress via FEEDMERGE-COORD-02-900 so merge/storage backfills can proceed. | +| Sprint 1 | Stabilize In-Progress Foundations | src/StellaOps.Concelier.Models/TASKS.md | DONE (2025-10-12) | Team Models & Merge Leads | FEEDMODELS-SCHEMA-02-900 | Range primitives for SemVer/EVR/NEVRA metadata
Instructions to work:
DONE Read ./AGENTS.md and src/StellaOps.Concelier.Models/AGENTS.md before resuming this stalled effort. Confirm helpers align with the new `NormalizedVersions` representation so connectors finishing in Sprint 2 can emit consistent metadata. | +| Sprint 1 | Stabilize In-Progress Foundations | src/StellaOps.Concelier.Normalization/TASKS.md | DONE (2025-10-11) | Team Normalization & Storage Backbone | FEEDNORM-NORM-02-001 | SemVer normalized rule emitter
Shared `SemVerRangeRuleBuilder` now outputs primitives + normalized rules per `FASTER_MODELING_AND_NORMALIZATION.md`; CVE/GHSA connectors consuming the API have verified fixtures. | +| Sprint 1 | Stabilize In-Progress Foundations | src/StellaOps.Concelier.Storage.Mongo/TASKS.md | DONE (2025-10-11) | Team Normalization & Storage Backbone | FEEDSTORAGE-DATA-02-001 | Normalized range dual-write + backfill
AdvisoryStore dual-writes flattened `normalizedVersions` when `concelier.storage.enableSemVerStyle` is set; migration `20251011-semver-style-backfill` updates historical records and docs outline the rollout. | +| Sprint 1 | Stabilize In-Progress Foundations | src/StellaOps.Concelier.Storage.Mongo/TASKS.md | DONE (2025-10-11) | Team Normalization & Storage Backbone | FEEDSTORAGE-DATA-02-002 | Provenance decision reason persistence
Storage now persists `provenance.decisionReason` for advisories and merge events; tests cover round-trips. | +| Sprint 1 | Stabilize In-Progress Foundations | src/StellaOps.Concelier.Storage.Mongo/TASKS.md | DONE (2025-10-11) | Team Normalization & Storage Backbone | FEEDSTORAGE-DATA-02-003 | Normalized versions indexing
Bootstrapper seeds compound/sparse indexes for flattened normalized rules and `docs/dev/mongo_indices.md` documents query guidance. | +| Sprint 1 | Stabilize In-Progress Foundations | src/StellaOps.Concelier.Storage.Mongo/TASKS.md | DONE (2025-10-11) | Team Normalization & Storage Backbone | FEEDSTORAGE-TESTS-02-004 | Restore AdvisoryStore build after normalized versions refactor
Updated constructors/tests keep storage suites passing with the new feature flag defaults. | +| Sprint 1 | Stabilize In-Progress Foundations | src/StellaOps.Concelier.WebService/TASKS.md | DONE (2025-10-12) | Team WebService & Authority | FEEDWEB-ENGINE-01-002 | Plumb Authority client resilience options
WebService wires `authority.resilience.*` into `AddStellaOpsAuthClient` and adds binding coverage via `AuthorityClientResilienceOptionsAreBound`. | +| Sprint 1 | Stabilize In-Progress Foundations | src/StellaOps.Concelier.WebService/TASKS.md | DONE (2025-10-12) | Team WebService & Authority | FEEDWEB-DOCS-01-003 | Author ops guidance for resilience tuning
Install/runbooks document connected vs air-gapped resilience profiles and monitoring hooks. | +| Sprint 1 | Stabilize In-Progress Foundations | src/StellaOps.Concelier.WebService/TASKS.md | DONE (2025-10-12) | Team WebService & Authority | FEEDWEB-DOCS-01-004 | Document authority bypass logging patterns
Operator guides now call out `route/status/subject/clientId/scopes/bypass/remote` audit fields and SIEM triggers. | +| Sprint 1 | Stabilize In-Progress Foundations | src/StellaOps.Concelier.WebService/TASKS.md | DONE (2025-10-12) | Team WebService & Authority | FEEDWEB-DOCS-01-005 | Update Concelier operator guide for enforcement cutoff
Install guide reiterates the 2025-12-31 cutoff and links audit signals to the rollout checklist. | +| Sprint 1 | Stabilize In-Progress Foundations | src/StellaOps.Authority/TASKS.md | DONE (2025-10-11) | Team WebService & Authority | SEC3.HOST | Rate limiter policy binding
Authority host now applies configuration-driven fixed windows to `/token`, `/authorize`, and `/internal/*`; integration tests assert 429 + `Retry-After` headers; docs/config samples refreshed for Docs guild diagrams. | +| Sprint 1 | Stabilize In-Progress Foundations | src/StellaOps.Authority/TASKS.md | DONE (2025-10-11) | Team WebService & Authority | SEC3.BUILD | Authority rate-limiter follow-through
`Security.RateLimiting` now fronts token/authorize/internal limiters; Authority + Configuration matrices (`dotnet test src/StellaOps.Authority/StellaOps.Authority.sln`, `dotnet test src/StellaOps.Configuration.Tests/StellaOps.Configuration.Tests.csproj`) passed on 2025-10-11; awaiting #authority-core broadcast. | +| Sprint 1 | Stabilize In-Progress Foundations | src/StellaOps.Authority/TASKS.md | DONE (2025-10-14) | Team Authority Platform & Security Guild | AUTHCORE-BUILD-OPENIDDICT / AUTHCORE-STORAGE-DEVICE-TOKENS / AUTHCORE-BOOTSTRAP-INVITES | Address remaining Authority compile blockers (OpenIddict transaction shim, token device document, bootstrap invite cleanup) so `dotnet build src/StellaOps.Authority.sln` returns success. | +| Sprint 1 | Stabilize In-Progress Foundations | src/StellaOps.Authority/StellaOps.Authority.Plugin.Standard/TASKS.md | DONE (2025-10-11) | Team WebService & Authority | PLG6.DOC | Plugin developer guide polish
Section 9 now documents rate limiter metadata, config keys, and lockout interplay; YAML samples updated alongside Authority config templates. | +| Sprint 1 | Stabilize In-Progress Foundations | src/StellaOps.Concelier.Connector.CertCc/TASKS.md | DONE (2025-10-11) | Team Connector Resumption – CERT/RedHat | FEEDCONN-CERTCC-02-001 | Fetch pipeline & state tracking
Summary planner now drives monthly/yearly VINCE fetches, persists pending summaries/notes, and hydrates VINCE detail queue with telemetry.
Team instructions: Read ./AGENTS.md and src/StellaOps.Concelier.Connector.CertCc/AGENTS.md. Coordinate daily with Models/Merge leads so new normalizedVersions output and provenance tags stay aligned with ./src/FASTER_MODELING_AND_NORMALIZATION.md. | +| Sprint 1 | Stabilize In-Progress Foundations | src/StellaOps.Concelier.Connector.CertCc/TASKS.md | DONE (2025-10-11) | Team Connector Resumption – CERT/RedHat | FEEDCONN-CERTCC-02-002 | VINCE note detail fetcher
Summary planner queues VINCE note detail endpoints, persists raw JSON with SHA/ETag metadata, and records retry/backoff metrics. | +| Sprint 1 | Stabilize In-Progress Foundations | src/StellaOps.Concelier.Connector.CertCc/TASKS.md | DONE (2025-10-11) | Team Connector Resumption – CERT/RedHat | FEEDCONN-CERTCC-02-003 | DTO & parser implementation
Added VINCE DTO aggregate, Markdown→text sanitizer, vendor/status/vulnerability parsers, and parser regression fixture. | +| Sprint 1 | Stabilize In-Progress Foundations | src/StellaOps.Concelier.Connector.CertCc/TASKS.md | DONE (2025-10-11) | Team Connector Resumption – CERT/RedHat | FEEDCONN-CERTCC-02-004 | Canonical mapping & range primitives
VINCE DTO aggregate flows through `CertCcMapper`, emitting vendor range primitives + normalized version rules that persist via `_advisoryStore`. | +| Sprint 1 | Stabilize In-Progress Foundations | src/StellaOps.Concelier.Connector.CertCc/TASKS.md | DONE (2025-10-12) | Team Connector Resumption – CERT/RedHat | FEEDCONN-CERTCC-02-005 | Deterministic fixtures/tests
Snapshot harness refreshed 2025-10-12; `certcc-*.snapshot.json` regenerated and regression suite green without UPDATE flag drift. | +| Sprint 1 | Stabilize In-Progress Foundations | src/StellaOps.Concelier.Connector.CertCc/TASKS.md | DONE (2025-10-12) | Team Connector Resumption – CERT/RedHat | FEEDCONN-CERTCC-02-006 | Telemetry & documentation
`CertCcDiagnostics` publishes summary/detail/parse/map metrics (meter `StellaOps.Concelier.Connector.CertCc`), README documents instruments, and log guidance captured for Ops on 2025-10-12. | +| Sprint 1 | Stabilize In-Progress Foundations | src/StellaOps.Concelier.Connector.CertCc/TASKS.md | DONE (2025-10-12) | Team Connector Resumption – CERT/RedHat | FEEDCONN-CERTCC-02-007 | Connector test harness remediation
Harness now wires `AddSourceCommon`, resets `FakeTimeProvider`, and passes canned-response regression run dated 2025-10-12. | +| Sprint 1 | Stabilize In-Progress Foundations | src/StellaOps.Concelier.Connector.CertCc/TASKS.md | DONE (2025-10-11) | Team Connector Resumption – CERT/RedHat | FEEDCONN-CERTCC-02-008 | Snapshot coverage handoff
Fixtures regenerated with normalized ranges + provenance fields on 2025-10-11; QA handoff notes published and merge backfill unblocked. | +| Sprint 1 | Stabilize In-Progress Foundations | src/StellaOps.Concelier.Connector.CertCc/TASKS.md | DONE (2025-10-12) | Team Connector Resumption – CERT/RedHat | FEEDCONN-CERTCC-02-012 | Schema sync & snapshot regen follow-up
Fixtures regenerated with normalizedVersions + provenance decision reasons; handoff notes updated for Merge backfill 2025-10-12. | +| Sprint 1 | Stabilize In-Progress Foundations | src/StellaOps.Concelier.Connector.CertCc/TASKS.md | DONE (2025-10-11) | Team Connector Resumption – CERT/RedHat | FEEDCONN-CERTCC-02-009 | Detail/map reintegration plan
Staged reintegration plan published in `src/StellaOps.Concelier.Connector.CertCc/FEEDCONN-CERTCC-02-009_PLAN.md`; coordinates enablement with FEEDCONN-CERTCC-02-004. | +| Sprint 1 | Stabilize In-Progress Foundations | src/StellaOps.Concelier.Connector.CertCc/TASKS.md | DONE (2025-10-12) | Team Connector Resumption – CERT/RedHat | FEEDCONN-CERTCC-02-010 | Partial-detail graceful degradation
Detail fetch now tolerates 404/403/410 responses and regression tests cover mixed endpoint availability. | +| Sprint 1 | Stabilize In-Progress Foundations | src/StellaOps.Concelier.Connector.Distro.RedHat/TASKS.md | DONE (2025-10-11) | Team Connector Resumption – CERT/RedHat | FEEDCONN-REDHAT-02-001 | Fixture validation sweep
Instructions to work:
Fixtures regenerated post-model-helper rollout; provenance ordering and normalizedVersions scaffolding verified via tests. Conflict resolver deltas logged in src/StellaOps.Concelier.Connector.Distro.RedHat/CONFLICT_RESOLVER_NOTES.md for Sprint 3 consumers. | +| Sprint 1 | Stabilize In-Progress Foundations | src/StellaOps.Concelier.Connector.Vndr.Apple/TASKS.md | DONE (2025-10-12) | Team Vendor Apple Specialists | FEEDCONN-APPLE-02-001 | Canonical mapping & range primitives
Mapper emits SemVer rules (`scheme=apple:*`); fixtures regenerated with trimmed references + new RSR coverage, update tooling finalized. | +| Sprint 1 | Stabilize In-Progress Foundations | src/StellaOps.Concelier.Connector.Vndr.Apple/TASKS.md | DONE (2025-10-11) | Team Vendor Apple Specialists | FEEDCONN-APPLE-02-002 | Deterministic fixtures/tests
Sanitized live fixtures + regression snapshots wired into tests; normalized rule coverage asserted. | +| Sprint 1 | Stabilize In-Progress Foundations | src/StellaOps.Concelier.Connector.Vndr.Apple/TASKS.md | DONE (2025-10-11) | Team Vendor Apple Specialists | FEEDCONN-APPLE-02-003 | Telemetry & documentation
Apple meter metrics wired into Concelier WebService OpenTelemetry configuration; README and fixtures document normalizedVersions coverage. | +| Sprint 1 | Stabilize In-Progress Foundations | src/StellaOps.Concelier.Connector.Vndr.Apple/TASKS.md | DONE (2025-10-12) | Team Vendor Apple Specialists | FEEDCONN-APPLE-02-004 | Live HTML regression sweep
Sanitised HT125326/HT125328/HT106355/HT214108/HT215500 fixtures recorded and regression tests green on 2025-10-12. | +| Sprint 1 | Stabilize In-Progress Foundations | src/StellaOps.Concelier.Connector.Vndr.Apple/TASKS.md | DONE (2025-10-11) | Team Vendor Apple Specialists | FEEDCONN-APPLE-02-005 | Fixture regeneration tooling
`UPDATE_APPLE_FIXTURES=1` flow fetches & rewrites fixtures; README documents usage.
Instructions to work:
DONE Read ./AGENTS.md and src/StellaOps.Concelier.Connector.Vndr.Apple/AGENTS.md. Resume stalled tasks, ensuring normalizedVersions output and fixtures align with ./src/FASTER_MODELING_AND_NORMALIZATION.md before handing data to the conflict sprint. | +| Sprint 1 | Stabilize In-Progress Foundations | src/StellaOps.Concelier.Connector.Ghsa/TASKS.md | DONE (2025-10-12) | Team Connector Normalized Versions Rollout | FEEDCONN-GHSA-02-001 | GHSA normalized versions & provenance
Team instructions: Read ./AGENTS.md and each module's AGENTS file. Adopt the `NormalizedVersions` array emitted by the models sprint, wiring provenance `decisionReason` where merge overrides occur. Follow ./src/FASTER_MODELING_AND_NORMALIZATION.md; report via src/StellaOps.Concelier.Merge/TASKS.md (FEEDMERGE-COORD-02-900). Progress 2025-10-11: GHSA/OSV emit normalized arrays with refreshed fixtures; CVE mapper now surfaces SemVer normalized ranges; NVD/KEV adoption pending; outstanding follow-ups include FEEDSTORAGE-DATA-02-001, FEEDMERGE-ENGINE-02-002, and rolling `tools/FixtureUpdater` updates across connectors. | +| Sprint 1 | Stabilize In-Progress Foundations | src/StellaOps.Concelier.Connector.Osv/TASKS.md | DONE (2025-10-12) | Team Connector Normalized Versions Rollout | FEEDCONN-OSV-02-003 | OSV normalized versions & freshness | +| Sprint 1 | Stabilize In-Progress Foundations | src/StellaOps.Concelier.Connector.Nvd/TASKS.md | DONE (2025-10-12) | Team Connector Normalized Versions Rollout | FEEDCONN-NVD-02-002 | NVD normalized versions & timestamps | +| Sprint 1 | Stabilize In-Progress Foundations | src/StellaOps.Concelier.Connector.Cve/TASKS.md | DONE (2025-10-12) | Team Connector Normalized Versions Rollout | FEEDCONN-CVE-02-003 | CVE normalized versions uplift | +| Sprint 1 | Stabilize In-Progress Foundations | src/StellaOps.Concelier.Connector.Kev/TASKS.md | DONE (2025-10-12) | Team Connector Normalized Versions Rollout | FEEDCONN-KEV-02-003 | KEV normalized versions propagation | +| Sprint 1 | Stabilize In-Progress Foundations | src/StellaOps.Concelier.Connector.Osv/TASKS.md | DONE (2025-10-12) | Team Connector Normalized Versions Rollout | FEEDCONN-OSV-04-003 | OSV parity fixture refresh | +| Sprint 1 | Stabilize In-Progress Foundations | src/StellaOps.Concelier.WebService/TASKS.md | DONE (2025-10-10) | Team WebService & Authority | FEEDWEB-DOCS-01-001 | Document authority toggle & scope requirements
Quickstart carries toggle/scope guidance pending docs guild review (no change this sprint). | +| Sprint 1 | Stabilize In-Progress Foundations | src/StellaOps.Concelier.WebService/TASKS.md | DONE (2025-10-12) | Team WebService & Authority | FEEDWEB-ENGINE-01-002 | Plumb Authority client resilience options
WebService wires `authority.resilience.*` into `AddStellaOpsAuthClient` and adds binding coverage via `AuthorityClientResilienceOptionsAreBound`. | +| Sprint 1 | Stabilize In-Progress Foundations | src/StellaOps.Concelier.WebService/TASKS.md | DONE (2025-10-12) | Team WebService & Authority | FEEDWEB-DOCS-01-003 | Author ops guidance for resilience tuning
Operator docs now outline connected vs air-gapped resilience profiles and monitoring cues. | +| Sprint 1 | Stabilize In-Progress Foundations | src/StellaOps.Concelier.WebService/TASKS.md | DONE (2025-10-12) | Team WebService & Authority | FEEDWEB-DOCS-01-004 | Document authority bypass logging patterns
Audit logging guidance highlights `route/status/subject/clientId/scopes/bypass/remote` fields and SIEM alerts. | +| Sprint 1 | Stabilize In-Progress Foundations | src/StellaOps.Concelier.WebService/TASKS.md | DONE (2025-10-12) | Team WebService & Authority | FEEDWEB-DOCS-01-005 | Update Concelier operator guide for enforcement cutoff
Install guide reiterates the 2025-12-31 cutoff and ties audit signals to rollout checks. | +| Sprint 1 | Stabilize In-Progress Foundations | src/StellaOps.Concelier.WebService/TASKS.md | DONE (2025-10-11) | Team WebService & Authority | FEEDWEB-OPS-01-006 | Rename plugin drop directory to namespaced path
Build outputs, tests, and docs now target `StellaOps.Concelier.PluginBinaries`/`StellaOps.Authority.PluginBinaries`. | +| Sprint 1 | Stabilize In-Progress Foundations | src/StellaOps.Concelier.WebService/TASKS.md | DONE (2025-10-11) | Team WebService & Authority | FEEDWEB-OPS-01-007 | Authority resilience adoption
Deployment docs and CLI notes explain the LIB5 resilience knobs for rollout.
Instructions to work:
DONE Read ./AGENTS.md and src/StellaOps.Concelier.WebService/AGENTS.md. These items were mid-flight; resume implementation ensuring docs/operators receive timely updates. | +| Sprint 1 | Stabilize In-Progress Foundations | src/StellaOps.Authority/TASKS.md | DONE (2025-10-11) | Team Authority Platform & Security Guild | AUTHCORE-ENGINE-01-001 | CORE8.RL — Rate limiter plumbing validated; integration tests green and docs handoff recorded for middleware ordering + Retry-After headers (see `docs/dev/authority-rate-limit-tuning-outline.md` for continuing guidance). | +| Sprint 1 | Stabilize In-Progress Foundations | src/StellaOps.Cryptography/TASKS.md | DONE (2025-10-11) | Team Authority Platform & Security Guild | AUTHCRYPTO-ENGINE-01-001 | SEC3.A — Shared metadata resolver confirmed via host test run; SEC3.B now unblocked for tuning guidance (outline captured in `docs/dev/authority-rate-limit-tuning-outline.md`). | +| Sprint 1 | Stabilize In-Progress Foundations | src/StellaOps.Cryptography/TASKS.md | DONE (2025-10-13) | Team Authority Platform & Security Guild | AUTHSEC-DOCS-01-002 | SEC3.B — Published `docs/security/rate-limits.md` with tuning matrix, alert thresholds, and lockout interplay guidance; Docs guild can lift copy into plugin guide. | +| Sprint 1 | Stabilize In-Progress Foundations | src/StellaOps.Cryptography/TASKS.md | DONE (2025-10-14) | Team Authority Platform & Security Guild | AUTHSEC-CRYPTO-02-001 | SEC5.B1 — Introduce libsodium signing provider and parity tests to unblock CLI verification enhancements. | +| Sprint 1 | Bootstrap & Replay Hardening | src/StellaOps.Cryptography/TASKS.md | DONE (2025-10-14) | Security Guild | AUTHSEC-CRYPTO-02-004 | SEC5.D/E — Finish bootstrap invite lifecycle (API/store/cleanup) and token device heuristics; build currently red due to pending handler integration. | +| Sprint 1 | Developer Tooling | src/StellaOps.Cli/TASKS.md | DONE (2025-10-15) | DevEx/CLI | AUTHCLI-DIAG-01-001 | Surface password policy diagnostics in CLI startup/output so operators see weakened overrides immediately.
CLI now loads Authority plug-ins at startup, logs weakened password policies (length/complexity), and regression coverage lives in `StellaOps.Cli.Tests/Services/AuthorityDiagnosticsReporterTests`. | +| Sprint 1 | Stabilize In-Progress Foundations | src/StellaOps.Authority/StellaOps.Authority.Plugin.Standard/TASKS.md | DONE (2025-10-11) | Team Authority Platform & Security Guild | AUTHPLUG-DOCS-01-001 | PLG6.DOC — Developer guide copy + diagrams merged 2025-10-11; limiter guidance incorporated and handed to Docs guild for asset export. | +| Sprint 2 | Connector & Data Implementation Wave | src/StellaOps.Concelier.Normalization/TASKS.md | DONE (2025-10-12) | Team Normalization & Storage Backbone | FEEDNORM-NORM-02-001 | SemVer normalized rule emitter
`SemVerRangeRuleBuilder` shipped 2025-10-12 with comparator/`||` support and fixtures aligning to `FASTER_MODELING_AND_NORMALIZATION.md`. | +| Sprint 2 | Connector & Data Implementation Wave | src/StellaOps.Concelier.Storage.Mongo/TASKS.md | DONE (2025-10-11) | Team Normalization & Storage Backbone | FEEDSTORAGE-DATA-02-001 | Normalized range dual-write + backfill | +| Sprint 2 | Connector & Data Implementation Wave | src/StellaOps.Concelier.Storage.Mongo/TASKS.md | DONE (2025-10-11) | Team Normalization & Storage Backbone | FEEDSTORAGE-DATA-02-002 | Provenance decision reason persistence | +| Sprint 2 | Connector & Data Implementation Wave | src/StellaOps.Concelier.Storage.Mongo/TASKS.md | DONE (2025-10-11) | Team Normalization & Storage Backbone | FEEDSTORAGE-DATA-02-003 | Normalized versions indexing
Indexes seeded + docs updated 2025-10-11 to cover flattened normalized rules for connector adoption. | +| Sprint 2 | Connector & Data Implementation Wave | src/StellaOps.Concelier.Merge/TASKS.md | DONE (2025-10-11) | Team Normalization & Storage Backbone | FEEDMERGE-ENGINE-02-002 | Normalized versions union & dedupe
Affected package resolver unions/dedupes normalized rules, stamps merge provenance with `decisionReason`, and tests cover the rollout. | +| Sprint 2 | Connector & Data Implementation Wave | src/StellaOps.Concelier.Connector.Ghsa/TASKS.md | DONE (2025-10-11) | Team Connector Expansion – GHSA/NVD/OSV | FEEDCONN-GHSA-02-001 | GHSA normalized versions & provenance | +| Sprint 2 | Connector & Data Implementation Wave | src/StellaOps.Concelier.Connector.Ghsa/TASKS.md | DONE (2025-10-11) | Team Connector Expansion – GHSA/NVD/OSV | FEEDCONN-GHSA-02-004 | GHSA credits & ecosystem severity mapping | +| Sprint 2 | Connector & Data Implementation Wave | src/StellaOps.Concelier.Connector.Ghsa/TASKS.md | DONE (2025-10-12) | Team Connector Expansion – GHSA/NVD/OSV | FEEDCONN-GHSA-02-005 | GitHub quota monitoring & retries | +| Sprint 2 | Connector & Data Implementation Wave | src/StellaOps.Concelier.Connector.Ghsa/TASKS.md | DONE (2025-10-12) | Team Connector Expansion – GHSA/NVD/OSV | FEEDCONN-GHSA-02-006 | Production credential & scheduler rollout | +| Sprint 2 | Connector & Data Implementation Wave | src/StellaOps.Concelier.Connector.Ghsa/TASKS.md | DONE (2025-10-12) | Team Connector Expansion – GHSA/NVD/OSV | FEEDCONN-GHSA-02-007 | Credit parity regression fixtures | +| Sprint 2 | Connector & Data Implementation Wave | src/StellaOps.Concelier.Connector.Nvd/TASKS.md | DONE (2025-10-11) | Team Connector Expansion – GHSA/NVD/OSV | FEEDCONN-NVD-02-002 | NVD normalized versions & timestamps | +| Sprint 2 | Connector & Data Implementation Wave | src/StellaOps.Concelier.Connector.Nvd/TASKS.md | DONE (2025-10-11) | Team Connector Expansion – GHSA/NVD/OSV | FEEDCONN-NVD-02-004 | NVD CVSS & CWE precedence payloads | +| Sprint 2 | Connector & Data Implementation Wave | src/StellaOps.Concelier.Connector.Nvd/TASKS.md | DONE (2025-10-12) | Team Connector Expansion – GHSA/NVD/OSV | FEEDCONN-NVD-02-005 | NVD merge/export parity regression | +| Sprint 2 | Connector & Data Implementation Wave | src/StellaOps.Concelier.Connector.Osv/TASKS.md | DONE (2025-10-11) | Team Connector Expansion – GHSA/NVD/OSV | FEEDCONN-OSV-02-003 | OSV normalized versions & freshness | +| Sprint 2 | Connector & Data Implementation Wave | src/StellaOps.Concelier.Connector.Osv/TASKS.md | DONE (2025-10-11) | Team Connector Expansion – GHSA/NVD/OSV | FEEDCONN-OSV-02-004 | OSV references & credits alignment | +| Sprint 2 | Connector & Data Implementation Wave | src/StellaOps.Concelier.Connector.Osv/TASKS.md | DONE (2025-10-12) | Team Connector Expansion – GHSA/NVD/OSV | FEEDCONN-OSV-02-005 | Fixture updater workflow
Resolved 2025-10-12: OSV mapper now derives canonical PURLs for Go + scoped npm packages when raw payloads omit `purl`; conflict fixtures unchanged for invalid npm names. Verified via `dotnet test src/StellaOps.Concelier.Connector.Osv.Tests`, `src/StellaOps.Concelier.Connector.Ghsa.Tests`, `src/StellaOps.Concelier.Connector.Nvd.Tests`, and backbone normalization/storage suites. | +| Sprint 2 | Connector & Data Implementation Wave | src/StellaOps.Concelier.Connector.Acsc/TASKS.md | DONE (2025-10-12) | Team Connector Expansion – Regional & Vendor Feeds | FEEDCONN-ACSC-02-001 … 02-008 | Fetch→parse→map pipeline, fixtures, diagnostics, and README finished 2025-10-12; downstream export parity captured via FEEDEXPORT-JSON-04-001 / FEEDEXPORT-TRIVY-04-001 (completed). | +| Sprint 2 | Connector & Data Implementation Wave | src/StellaOps.Concelier.Connector.Cccs/TASKS.md | DONE (2025-10-16) | Team Connector Expansion – Regional & Vendor Feeds | FEEDCONN-CCCS-02-001 … 02-008 | Observability meter, historical harvest plan, and DOM sanitizer refinements wrapped; ops notes live under `docs/ops/concelier-cccs-operations.md` with fixtures validating EN/FR list handling. | +| Sprint 2 | Connector & Data Implementation Wave | src/StellaOps.Concelier.Connector.CertBund/TASKS.md | DONE (2025-10-15) | Team Connector Expansion – Regional & Vendor Feeds | FEEDCONN-CERTBUND-02-001 … 02-008 | Telemetry/docs (02-006) and history/locale sweep (02-007) completed alongside pipeline; runbook `docs/ops/concelier-certbund-operations.md` captures locale guidance and offline packaging. | +| Sprint 2 | Connector & Data Implementation Wave | src/StellaOps.Concelier.Connector.Kisa/TASKS.md | DONE (2025-10-14) | Team Connector Expansion – Regional & Vendor Feeds | FEEDCONN-KISA-02-001 … 02-007 | Connector, tests, and telemetry/docs (02-006) finalized; localisation notes in `docs/dev/kisa_connector_notes.md` complete rollout. | +| Sprint 2 | Connector & Data Implementation Wave | src/StellaOps.Concelier.Connector.Ru.Bdu/TASKS.md | DONE (2025-10-14) | Team Connector Expansion – Regional & Vendor Feeds | FEEDCONN-RUBDU-02-001 … 02-008 | Fetch/parser/mapper refinements, regression fixtures, telemetry/docs, access options, and trusted root packaging all landed; README documents offline access strategy. | +| Sprint 2 | Connector & Data Implementation Wave | src/StellaOps.Concelier.Connector.Ru.Nkcki/TASKS.md | DONE (2025-10-13) | Team Connector Expansion – Regional & Vendor Feeds | FEEDCONN-NKCKI-02-001 … 02-008 | Listing fetch, parser, mapper, fixtures, telemetry/docs, and archive plan finished; Mongo2Go/libcrypto dependency resolved via bundled OpenSSL noted in ops guide. | +| Sprint 2 | Connector & Data Implementation Wave | src/StellaOps.Concelier.Connector.Ics.Cisa/TASKS.md | DONE (2025-10-16) | Team Connector Expansion – Regional & Vendor Feeds | FEEDCONN-ICSCISA-02-001 … 02-011 | Feed parser attachment fixes, SemVer exact values, regression suites, telemetry/docs updates, and handover complete; ops runbook now details attachment verification + proxy usage. | +| Sprint 2 | Connector & Data Implementation Wave | src/StellaOps.Concelier.Connector.Vndr.Cisco/TASKS.md | DONE (2025-10-14) | Team Connector Expansion – Regional & Vendor Feeds | FEEDCONN-CISCO-02-001 … 02-007 | OAuth fetch pipeline, DTO/mapping, tests, and telemetry/docs shipped; monitoring/export integration follow-ups recorded in Ops docs and exporter backlog (completed). | +| Sprint 2 | Connector & Data Implementation Wave | src/StellaOps.Concelier.Connector.Vndr.Msrc/TASKS.md | DONE (2025-10-15) | Team Connector Expansion – Regional & Vendor Feeds | FEEDCONN-MSRC-02-001 … 02-008 | Azure AD onboarding (02-008) unblocked fetch/parse/map pipeline; fixtures, telemetry/docs, and Offline Kit guidance published in `docs/ops/concelier-msrc-operations.md`. | +| Sprint 2 | Connector & Data Implementation Wave | src/StellaOps.Concelier.Connector.Cve/TASKS.md | DONE (2025-10-15) | Team Connector Support & Monitoring | FEEDCONN-CVE-02-001 … 02-002 | CVE data-source selection, fetch pipeline, and docs landed 2025-10-10. 2025-10-15: smoke verified using the seeded mirror fallback; connector now logs a warning and pulls from `seed-data/cve/` until live CVE Services credentials arrive. | +| Sprint 2 | Connector & Data Implementation Wave | src/StellaOps.Concelier.Connector.Kev/TASKS.md | DONE (2025-10-12) | Team Connector Support & Monitoring | FEEDCONN-KEV-02-001 … 02-002 | KEV catalog ingestion, fixtures, telemetry, and schema validation completed 2025-10-12; ops dashboard published. | +| Sprint 2 | Connector & Data Implementation Wave | docs/TASKS.md | DONE (2025-10-11) | Team Docs & Knowledge Base | FEEDDOCS-DOCS-01-001 | Canonical schema docs refresh
Updated canonical schema + provenance guides with SemVer style, normalized version rules, decision reason change log, and migration notes. | +| Sprint 2 | Connector & Data Implementation Wave | docs/TASKS.md | DONE (2025-10-11) | Team Docs & Knowledge Base | FEEDDOCS-DOCS-02-001 | Concelier-SemVer Playbook
Published merge playbook covering mapper patterns, dedupe flow, indexes, and rollout checklist. | +| Sprint 2 | Connector & Data Implementation Wave | docs/TASKS.md | DONE (2025-10-11) | Team Docs & Knowledge Base | FEEDDOCS-DOCS-02-002 | Normalized versions query guide
Delivered Mongo index/query addendum with `$unwind` recipes, dedupe checks, and operational checklist.
Instructions to work:
DONE Read ./AGENTS.md and docs/AGENTS.md. Document every schema/index/query change produced in Sprint 1-2 leveraging ./src/FASTER_MODELING_AND_NORMALIZATION.md. | +| Sprint 3 | Conflict Resolution Integration & Communications | src/StellaOps.Concelier.Core/TASKS.md | DONE (2025-10-11) | Team Core Engine & Storage Analytics | FEEDCORE-ENGINE-03-001 | Canonical merger implementation
`CanonicalMerger` ships with freshness/tie-breaker logic, provenance, and unit coverage feeding Merge. | +| Sprint 3 | Conflict Resolution Integration & Communications | src/StellaOps.Concelier.Core/TASKS.md | DONE (2025-10-11) | Team Core Engine & Storage Analytics | FEEDCORE-ENGINE-03-002 | Field precedence and tie-breaker map
Field precedence tables and tie-breaker metrics wired into the canonical merge flow; docs/tests updated.
Instructions to work:
Read ./AGENTS.md and core AGENTS. Implement the conflict resolver exactly as specified in ./src/DEDUP_CONFLICTS_RESOLUTION_ALGO.md, coordinating with Merge and Storage teammates. | +| Sprint 3 | Conflict Resolution Integration & Communications | src/StellaOps.Concelier.Storage.Mongo/TASKS.md | DONE (2025-10-11) | Team Core Engine & Storage Analytics | FEEDSTORAGE-DATA-03-001 | Merge event provenance audit prep
Merge events now persist `fieldDecisions` and analytics-ready provenance snapshots. | +| Sprint 3 | Conflict Resolution Integration & Communications | src/StellaOps.Concelier.Storage.Mongo/TASKS.md | DONE (2025-10-11) | Team Core Engine & Storage Analytics | FEEDSTORAGE-DATA-02-001 | Normalized range dual-write + backfill
Dual-write/backfill flag delivered; migration + options validated in tests. | +| Sprint 3 | Conflict Resolution Integration & Communications | src/StellaOps.Concelier.Storage.Mongo/TASKS.md | DONE (2025-10-11) | Team Core Engine & Storage Analytics | FEEDSTORAGE-TESTS-02-004 | Restore AdvisoryStore build after normalized versions refactor
Storage tests adjusted for normalized versions/decision reasons.
Instructions to work:
Read ./AGENTS.md and storage AGENTS. Extend merge events with decision reasons and analytics views to support the conflict rules, and deliver the dual-write/backfill for `NormalizedVersions` + `decisionReason` so connectors can roll out safely. | +| Sprint 3 | Conflict Resolution Integration & Communications | src/StellaOps.Concelier.Merge/TASKS.md | DONE (2025-10-11) | Team Merge & QA Enforcement | FEEDMERGE-ENGINE-04-001 | GHSA/NVD/OSV conflict rules
Merge pipeline consumes `CanonicalMerger` output prior to precedence merge. | +| Sprint 3 | Conflict Resolution Integration & Communications | src/StellaOps.Concelier.Merge/TASKS.md | DONE (2025-10-11) | Team Merge & QA Enforcement | FEEDMERGE-ENGINE-04-002 | Override metrics instrumentation
Merge events capture per-field decisions; counters/logs align with conflict rules. | +| Sprint 3 | Conflict Resolution Integration & Communications | src/StellaOps.Concelier.Merge/TASKS.md | DONE (2025-10-11) | Team Merge & QA Enforcement | FEEDMERGE-ENGINE-04-003 | Reference & credit union pipeline
Canonical merge preserves unions with updated tests. | +| Sprint 3 | Conflict Resolution Integration & Communications | src/StellaOps.Concelier.Merge/TASKS.md | DONE (2025-10-11) | Team Merge & QA Enforcement | FEEDMERGE-QA-04-001 | End-to-end conflict regression suite
Added regression tests (`AdvisoryMergeServiceTests`) covering canonical + precedence flow.
Instructions to work:
Read ./AGENTS.md and merge AGENTS. Integrate the canonical merger, instrument metrics, and deliver comprehensive regression tests following ./src/DEDUP_CONFLICTS_RESOLUTION_ALGO.md. | +| Sprint 3 | Conflict Resolution Integration & Communications | src/StellaOps.Concelier.Connector.Ghsa/TASKS.md | DONE (2025-10-12) | Team Connector Regression Fixtures | FEEDCONN-GHSA-04-002 | GHSA conflict regression fixtures | +| Sprint 3 | Conflict Resolution Integration & Communications | src/StellaOps.Concelier.Connector.Nvd/TASKS.md | DONE (2025-10-12) | Team Connector Regression Fixtures | FEEDCONN-NVD-04-002 | NVD conflict regression fixtures | +| Sprint 3 | Conflict Resolution Integration & Communications | src/StellaOps.Concelier.Connector.Osv/TASKS.md | DONE (2025-10-12) | Team Connector Regression Fixtures | FEEDCONN-OSV-04-002 | OSV conflict regression fixtures
Instructions to work:
Read ./AGENTS.md and module AGENTS. Produce fixture triples supporting the precedence/tie-breaker paths defined in ./src/DEDUP_CONFLICTS_RESOLUTION_ALGO.md and hand them to Merge QA. | +| Sprint 3 | Conflict Resolution Integration & Communications | docs/TASKS.md | DONE (2025-10-11) | Team Documentation Guild – Conflict Guidance | FEEDDOCS-DOCS-05-001 | Concelier Conflict Rules
Runbook published at `docs/ops/concelier-conflict-resolution.md`; metrics/log guidance aligned with Sprint 3 merge counters. | +| Sprint 3 | Conflict Resolution Integration & Communications | docs/TASKS.md | DONE (2025-10-16) | Team Documentation Guild – Conflict Guidance | FEEDDOCS-DOCS-05-002 | Conflict runbook ops rollout
Ops review completed, alert thresholds applied, and change log appended in `docs/ops/concelier-conflict-resolution.md`; task closed after connector signals verified. | +| Sprint 4 | Schema Parity & Freshness Alignment | src/StellaOps.Concelier.Models/TASKS.md | DONE (2025-10-15) | Team Models & Merge Leads | FEEDMODELS-SCHEMA-04-001 | Advisory schema parity (description/CWE/canonical metric)
Extend `Advisory` and related records with description text, CWE collection, and canonical metric pointer; refresh validation + serializer determinism tests. | +| Sprint 4 | Schema Parity & Freshness Alignment | src/StellaOps.Concelier.Core/TASKS.md | DONE (2025-10-15) | Team Core Engine & Storage Analytics | FEEDCORE-ENGINE-04-003 | Canonical merger parity for new fields
Teach `CanonicalMerger` to populate description, CWEResults, and canonical metric pointer with provenance + regression coverage. | +| Sprint 4 | Schema Parity & Freshness Alignment | src/StellaOps.Concelier.Core/TASKS.md | DONE (2025-10-15) | Team Core Engine & Storage Analytics | FEEDCORE-ENGINE-04-004 | Reference normalization & freshness instrumentation cleanup
Implement URL normalization for reference dedupe, align freshness-sensitive instrumentation, and add analytics tests. | +| Sprint 4 | Schema Parity & Freshness Alignment | src/StellaOps.Concelier.Merge/TASKS.md | DONE (2025-10-15) | Team Merge & QA Enforcement | FEEDMERGE-ENGINE-04-004 | Merge pipeline parity for new advisory fields
Ensure merge service + merge events surface description/CWE/canonical metric decisions with updated metrics/tests. | +| Sprint 4 | Schema Parity & Freshness Alignment | src/StellaOps.Concelier.Merge/TASKS.md | DONE (2025-10-15) | Team Merge & QA Enforcement | FEEDMERGE-ENGINE-04-005 | Connector coordination for new advisory fields
GHSA/NVD/OSV connectors now ship description, CWE, and canonical metric data with refreshed fixtures; merge coordination log updated and exporters notified. | +| Sprint 4 | Schema Parity & Freshness Alignment | src/StellaOps.Concelier.Exporter.Json/TASKS.md | DONE (2025-10-15) | Team Exporters – JSON | FEEDEXPORT-JSON-04-001 | Surface new advisory fields in JSON exporter
Update schemas/offline bundle + fixtures once model/core parity lands.
2025-10-15: `dotnet test src/StellaOps.Concelier.Exporter.Json.Tests` validated canonical metric/CWE emission. | +| Sprint 4 | Schema Parity & Freshness Alignment | src/StellaOps.Concelier.Exporter.TrivyDb/TASKS.md | DONE (2025-10-15) | Team Exporters – Trivy DB | FEEDEXPORT-TRIVY-04-001 | Propagate new advisory fields into Trivy DB package
Extend Bolt builder, metadata, and regression tests for the expanded schema.
2025-10-15: `dotnet test src/StellaOps.Concelier.Exporter.TrivyDb.Tests` confirmed canonical metric/CWE propagation. | +| Sprint 4 | Schema Parity & Freshness Alignment | src/StellaOps.Concelier.Connector.Ghsa/TASKS.md | DONE (2025-10-16) | Team Connector Regression Fixtures | FEEDCONN-GHSA-04-004 | Harden CVSS fallback so canonical metric ids persist when GitHub omits vectors; extend fixtures and document severity precedence hand-off to Merge. | +| Sprint 4 | Schema Parity & Freshness Alignment | src/StellaOps.Concelier.Connector.Osv/TASKS.md | DONE (2025-10-16) | Team Connector Expansion – GHSA/NVD/OSV | FEEDCONN-OSV-04-005 | Map OSV advisories lacking CVSS vectors to canonical metric ids/notes and document CWE provenance quirks; schedule parity fixture updates. | +| Sprint 5 | Excititor Core Foundations | src/StellaOps.Excititor.Core/TASKS.md | DONE (2025-10-15) | Team Excititor Core & Policy | EXCITITOR-CORE-01-001 | Stand up canonical VEX claim/consensus records with deterministic serializers so Storage/Exports share a stable contract. | +| Sprint 5 | Excititor Core Foundations | src/StellaOps.Excititor.Core/TASKS.md | DONE (2025-10-15) | Team Excititor Core & Policy | EXCITITOR-CORE-01-002 | Implement trust-weighted consensus resolver with baseline policy weights, justification gates, telemetry output, and majority/tie handling. | +| Sprint 5 | Excititor Core Foundations | src/StellaOps.Excititor.Core/TASKS.md | DONE (2025-10-15) | Team Excititor Core & Policy | EXCITITOR-CORE-01-003 | Publish shared connector/exporter/attestation abstractions and deterministic query signature utilities for cache/attestation workflows. | +| Sprint 5 | Excititor Core Foundations | src/StellaOps.Excititor.Policy/TASKS.md | DONE (2025-10-15) | Team Excititor Policy | EXCITITOR-POLICY-01-001 | Established policy options & snapshot provider covering baseline weights/overrides. | +| Sprint 5 | Excititor Core Foundations | src/StellaOps.Excititor.Policy/TASKS.md | DONE (2025-10-15) | Team Excititor Policy | EXCITITOR-POLICY-01-002 | Policy evaluator now feeds consensus resolver with immutable snapshots. | +| Sprint 5 | Excititor Core Foundations | src/StellaOps.Excititor.Policy/TASKS.md | DONE (2025-10-16) | Team Excititor Policy | EXCITITOR-POLICY-01-003 | Author policy diagnostics, CLI/WebService surfacing, and documentation updates. | +| Sprint 5 | Excititor Core Foundations | src/StellaOps.Excititor.Policy/TASKS.md | DONE (2025-10-16) | Team Excititor Policy | EXCITITOR-POLICY-01-004 | Implement YAML/JSON schema validation and deterministic diagnostics for operator bundles. | +| Sprint 5 | Excititor Core Foundations | src/StellaOps.Excititor.Policy/TASKS.md | DONE (2025-10-16) | Team Excititor Policy | EXCITITOR-POLICY-01-005 | Add policy change tracking, snapshot digests, and telemetry/logging hooks. | +| Sprint 5 | Excititor Core Foundations | src/StellaOps.Excititor.Storage.Mongo/TASKS.md | DONE (2025-10-15) | Team Excititor Storage | EXCITITOR-STORAGE-01-001 | Mongo mapping registry plus raw/export entities and DI extensions in place. | +| Sprint 5 | Excititor Core Foundations | src/StellaOps.Excititor.Storage.Mongo/TASKS.md | DONE (2025-10-16) | Team Excititor Storage | EXCITITOR-STORAGE-01-004 | Build provider/consensus/cache class maps and related collections. | +| Sprint 5 | Excititor Core Foundations | src/StellaOps.Excititor.Export/TASKS.md | DONE (2025-10-15) | Team Excititor Export | EXCITITOR-EXPORT-01-001 | Export engine delivers cache lookup, manifest creation, and policy integration. | +| Sprint 5 | Excititor Core Foundations | src/StellaOps.Excititor.Export/TASKS.md | DONE (2025-10-17) | Team Excititor Export | EXCITITOR-EXPORT-01-004 | Connect export engine to attestation client and persist Rekor metadata. | +| Sprint 5 | Excititor Core Foundations | src/StellaOps.Excititor.Attestation/TASKS.md | DONE (2025-10-16) | Team Excititor Attestation | EXCITITOR-ATTEST-01-001 | Implement in-toto predicate + DSSE builder providing envelopes for export attestation. | +| Sprint 5 | Excititor Core Foundations | src/StellaOps.Excititor.Connectors.Abstractions/TASKS.md | DONE (2025-10-17) | Team Excititor Connectors | EXCITITOR-CONN-ABS-01-001 | Deliver shared connector context/base classes so provider plug-ins can be activated via WebService/Worker. | +| Sprint 5 | Excititor Core Foundations | src/StellaOps.Excititor.WebService/TASKS.md | DONE (2025-10-17) | Team Excititor WebService | EXCITITOR-WEB-01-001 | Scaffold minimal API host, DI, and `/excititor/status` endpoint integrating policy, storage, export, and attestation services. | +| Sprint 6 | Excititor Ingest & Formats | src/StellaOps.Excititor.Worker/TASKS.md | DONE (2025-10-17) | Team Excititor Worker | EXCITITOR-WORKER-01-001 | Create Worker host with provider scheduling and logging to drive recurring pulls/reconciliation. | +| Sprint 6 | Excititor Ingest & Formats | src/StellaOps.Excititor.Formats.CSAF/TASKS.md | DONE (2025-10-17) | Team Excititor Formats | EXCITITOR-FMT-CSAF-01-001 | Implement CSAF normalizer foundation translating provider documents into `VexClaim` entries. | +| Sprint 6 | Excititor Ingest & Formats | src/StellaOps.Excititor.Formats.CycloneDX/TASKS.md | DONE (2025-10-17) | Team Excititor Formats | EXCITITOR-FMT-CYCLONE-01-001 | Implement CycloneDX VEX normalizer capturing `analysis` state and component references. | +| Sprint 6 | Excititor Ingest & Formats | src/StellaOps.Excititor.Formats.OpenVEX/TASKS.md | DONE (2025-10-17) | Team Excititor Formats | EXCITITOR-FMT-OPENVEX-01-001 | Implement OpenVEX normalizer to ingest attestations into canonical claims with provenance. | +| Sprint 6 | Excititor Ingest & Formats | src/StellaOps.Excititor.Connectors.RedHat.CSAF/TASKS.md | DONE (2025-10-17) | Team Excititor Connectors – Red Hat | EXCITITOR-CONN-RH-01-001 | Ship Red Hat CSAF provider metadata discovery enabling incremental pulls. | +| Sprint 6 | Excititor Ingest & Formats | src/StellaOps.Excititor.Connectors.RedHat.CSAF/TASKS.md | DONE (2025-10-17) | Team Excititor Connectors – Red Hat | EXCITITOR-CONN-RH-01-002 | Fetch CSAF windows with ETag handling, resume tokens, quarantine on schema errors, and persist raw docs. | +| Sprint 6 | Excititor Ingest & Formats | src/StellaOps.Excititor.Connectors.RedHat.CSAF/TASKS.md | DONE (2025-10-17) | Team Excititor Connectors – Red Hat | EXCITITOR-CONN-RH-01-003 | Populate provider trust overrides (cosign issuer, identity regex) and provenance hints for policy evaluation/logging. | +| Sprint 6 | Excititor Ingest & Formats | src/StellaOps.Excititor.Connectors.RedHat.CSAF/TASKS.md | DONE (2025-10-17) | Team Excititor Connectors – Red Hat | EXCITITOR-CONN-RH-01-004 | Persist resume cursors (last updated timestamp/document hashes) in storage and reload during fetch to avoid duplicates. | +| Sprint 6 | Excititor Ingest & Formats | src/StellaOps.Excititor.Connectors.RedHat.CSAF/TASKS.md | DONE (2025-10-17) | Team Excititor Connectors – Red Hat | EXCITITOR-CONN-RH-01-005 | Register connector in Worker/WebService DI, add scheduled jobs, and document CLI triggers for Red Hat CSAF pulls. | +| Sprint 6 | Excititor Ingest & Formats | src/StellaOps.Excititor.Connectors.RedHat.CSAF/TASKS.md | DONE (2025-10-17) | Team Excititor Connectors – Red Hat | EXCITITOR-CONN-RH-01-006 | Add CSAF normalization parity fixtures ensuring RHSA-specific metadata is preserved. | +| Sprint 6 | Excititor Ingest & Formats | src/StellaOps.Excititor.Connectors.Cisco.CSAF/TASKS.md | DONE (2025-10-17) | Team Excititor Connectors – Cisco | EXCITITOR-CONN-CISCO-01-001 | Implement Cisco CSAF endpoint discovery/auth to unlock paginated pulls. | +| Sprint 6 | Excititor Ingest & Formats | src/StellaOps.Excititor.Connectors.Cisco.CSAF/TASKS.md | DONE (2025-10-17) | Team Excititor Connectors – Cisco | EXCITITOR-CONN-CISCO-01-002 | Implement Cisco CSAF paginated fetch loop with dedupe and raw persistence support. | +| Sprint 6 | Excititor Ingest & Formats | src/StellaOps.Excititor.Connectors.SUSE.RancherVEXHub/TASKS.md | DONE (2025-10-17) | Team Excititor Connectors – SUSE | EXCITITOR-CONN-SUSE-01-001 | Build Rancher VEX Hub discovery/subscription path with offline snapshot support. | +| Sprint 6 | Excititor Ingest & Formats | src/StellaOps.Excititor.Connectors.MSRC.CSAF/TASKS.md | DONE (2025-10-17) | Team Excititor Connectors – MSRC | EXCITITOR-CONN-MS-01-001 | Deliver AAD onboarding/token cache for MSRC CSAF ingestion. | +| Sprint 6 | Excititor Ingest & Formats | src/StellaOps.Excititor.Connectors.Oracle.CSAF/TASKS.md | DONE (2025-10-17) | Team Excititor Connectors – Oracle | EXCITITOR-CONN-ORACLE-01-001 | Implement Oracle CSAF catalogue discovery with CPU calendar awareness. | +| Sprint 6 | Excititor Ingest & Formats | src/StellaOps.Excititor.Connectors.Ubuntu.CSAF/TASKS.md | DONE (2025-10-17) | Team Excititor Connectors – Ubuntu | EXCITITOR-CONN-UBUNTU-01-001 | Implement Ubuntu CSAF discovery and channel selection for USN ingestion. | +| Sprint 6 | Excititor Ingest & Formats | src/StellaOps.Excititor.Connectors.OCI.OpenVEX.Attest/TASKS.md | DONE (2025-10-18) | Team Excititor Connectors – OCI | EXCITITOR-CONN-OCI-01-001 | Wire OCI discovery/auth to fetch OpenVEX attestations for configured images. | +| Sprint 6 | Excititor Ingest & Formats | src/StellaOps.Excititor.Connectors.OCI.OpenVEX.Attest/TASKS.md | DONE (2025-10-18) | Team Excititor Connectors – OCI | EXCITITOR-CONN-OCI-01-002 | Attestation fetch & verify loop – download DSSE attestations, trigger verification, handle retries/backoff, persist raw statements. | +| Sprint 6 | Excititor Ingest & Formats | src/StellaOps.Excititor.Connectors.OCI.OpenVEX.Attest/TASKS.md | DONE (2025-10-18) | Team Excititor Connectors – OCI | EXCITITOR-CONN-OCI-01-003 | Provenance metadata & policy hooks – emit image, subject digest, issuer, and trust metadata for policy weighting/logging. | +| Sprint 6 | Excititor Ingest & Formats | src/StellaOps.Cli/TASKS.md | DONE (2025-10-18) | DevEx/CLI | EXCITITOR-CLI-01-001 | Add `excititor` CLI verbs bridging to WebService with consistent auth and offline UX. | +| Sprint 7 | Contextual Truth Foundations | src/StellaOps.Excititor.Core/TASKS.md | DONE (2025-10-19) | Team Excititor Core & Policy | EXCITITOR-CORE-02-001 | Context signal schema prep – extend consensus models with severity/KEV/EPSS fields and update canonical serializers. | +| Sprint 7 | Contextual Truth Foundations | src/StellaOps.Excititor.Policy/TASKS.md | DONE (2025-10-19) | Team Excititor Policy | EXCITITOR-POLICY-02-001 | Scoring coefficients & weight ceilings – add α/β options, weight boosts, and validation guidance. | +| Sprint 7 | Contextual Truth Foundations | src/StellaOps.Excititor.Attestation/TASKS.md | DONE (2025-10-16) | Team Excititor Attestation | EXCITITOR-ATTEST-01-002 | Rekor v2 client integration – ship transparency log client with retries and offline queue. | +| Sprint 9 | Scanner Core Foundations | src/StellaOps.Scanner.Core/TASKS.md | DONE (2025-10-18) | Team Scanner Core | SCANNER-CORE-09-501 | Define shared DTOs (ScanJob, ProgressEvent), error taxonomy, and deterministic ID/timestamp helpers aligning with `ARCHITECTURE_SCANNER.md` §3–§4. | +| Sprint 9 | Scanner Core Foundations | src/StellaOps.Scanner.Core/TASKS.md | DONE (2025-10-18) | Team Scanner Core | SCANNER-CORE-09-502 | Observability helpers (correlation IDs, logging scopes, metric namespacing, deterministic hashes) consumed by WebService/Worker. | +| Sprint 9 | Scanner Core Foundations | src/StellaOps.Scanner.Core/TASKS.md | DONE (2025-10-18) | Team Scanner Core | SCANNER-CORE-09-503 | Security utilities: Authority client factory, OpTok caching, DPoP verifier, restart-time plug-in guardrails for scanner components. | +| Sprint 9 | Scanner Build-time | src/StellaOps.Scanner.Sbomer.BuildXPlugin/TASKS.md | DONE (2025-10-19) | BuildX Guild | SP9-BLDX-09-001 | Buildx driver scaffold + handshake with Scanner.Emit (local CAS). | +| Sprint 9 | Scanner Build-time | src/StellaOps.Scanner.Sbomer.BuildXPlugin/TASKS.md | DONE (2025-10-19) | BuildX Guild | SP9-BLDX-09-002 | OCI annotations + provenance hand-off to Attestor. | +| Sprint 9 | Scanner Build-time | src/StellaOps.Scanner.Sbomer.BuildXPlugin/TASKS.md | DONE (2025-10-19) | BuildX Guild | SP9-BLDX-09-003 | CI demo: minimal SBOM push & backend report wiring. | +| Sprint 9 | Scanner Build-time | src/StellaOps.Scanner.Sbomer.BuildXPlugin/TASKS.md | DONE (2025-10-19) | BuildX Guild | SP9-BLDX-09-004 | Stabilize descriptor nonce derivation so repeated builds emit deterministic placeholders. | +| Sprint 9 | Scanner Build-time | src/StellaOps.Scanner.Sbomer.BuildXPlugin/TASKS.md | DONE (2025-10-19) | BuildX Guild | SP9-BLDX-09-005 | Integrate determinism guard into GitHub/Gitea workflows and archive proof artifacts. | +| Sprint 9 | Scanner Core Foundations | src/StellaOps.Scanner.WebService/TASKS.md | DONE (2025-10-18) | Team Scanner WebService | SCANNER-WEB-09-101 | Minimal API host with Authority enforcement, health/ready endpoints, and restart-time plug-in loader per architecture §1, §4. | +| Sprint 9 | Scanner Core Foundations | src/StellaOps.Scanner.WebService/TASKS.md | DONE (2025-10-18) | Team Scanner WebService | SCANNER-WEB-09-102 | `/api/v1/scans` submission/status endpoints with deterministic IDs, validation, and cancellation support. | +| Sprint 9 | Scanner Core Foundations | src/StellaOps.Scanner.WebService/TASKS.md | DONE (2025-10-19) | Team Scanner WebService | SCANNER-WEB-09-104 | Configuration binding for Mongo, MinIO, queue, feature flags; startup diagnostics and fail-fast policy. | +| Sprint 9 | Scanner Core Foundations | src/StellaOps.Scanner.Worker/TASKS.md | DONE (2025-10-19) | Team Scanner Worker | SCANNER-WORKER-09-201 | Worker host bootstrap with Authority auth, hosted services, and graceful shutdown semantics. | +| Sprint 9 | Scanner Core Foundations | src/StellaOps.Scanner.Worker/TASKS.md | DONE (2025-10-19) | Team Scanner Worker | SCANNER-WORKER-09-202 | Lease/heartbeat loop with retry+jitter, poison-job quarantine, structured logging. | +| Sprint 9 | Scanner Core Foundations | src/StellaOps.Scanner.Worker/TASKS.md | DONE (2025-10-19) | Team Scanner Worker | SCANNER-WORKER-09-203 | Analyzer dispatch skeleton emitting deterministic stage progress and honoring cancellation tokens. | +| Sprint 9 | Scanner Core Foundations | src/StellaOps.Scanner.Worker/TASKS.md | DONE (2025-10-19) | Team Scanner Worker | SCANNER-WORKER-09-204 | Worker metrics (queue latency, stage duration, failure counts) with OpenTelemetry resource wiring. | +| Sprint 9 | Scanner Core Foundations | src/StellaOps.Scanner.Worker/TASKS.md | DONE (2025-10-19) | Team Scanner Worker | SCANNER-WORKER-09-205 | Harden heartbeat jitter so lease safety margin stays ≥3× and cover with regression tests + optional live queue smoke run. | +| Sprint 9 | Policy Foundations | src/StellaOps.Policy/TASKS.md | DONE | Policy Guild | POLICY-CORE-09-001 | Policy schema + binder + diagnostics. | +| Sprint 9 | Policy Foundations | src/StellaOps.Policy/TASKS.md | DONE | Policy Guild | POLICY-CORE-09-002 | Policy snapshot store + revision digests. | +| Sprint 9 | Policy Foundations | src/StellaOps.Policy/TASKS.md | DONE | Policy Guild | POLICY-CORE-09-003 | `/policy/preview` API (image digest → projected verdict diff). | +| Sprint 9 | DevOps Foundations | ops/devops/TASKS.md | DONE (2025-10-19) | DevOps Guild | DEVOPS-HELM-09-001 | Helm/Compose environment profiles (dev/staging/airgap) with deterministic digests. | +| Sprint 9 | Docs & Governance | docs/TASKS.md | DONE (2025-10-19) | Docs Guild, DevEx | DOCS-ADR-09-001 | Establish ADR process and template. | +| Sprint 9 | Docs & Governance | docs/TASKS.md | DONE (2025-10-19) | Docs Guild, Platform Events | DOCS-EVENTS-09-002 | Publish event schema catalog (`docs/events/`) for critical envelopes. | +| Sprint 9 | Scanner Core Foundations | src/StellaOps.Scanner.Storage/TASKS.md | DONE (2025-10-19) | Team Scanner Storage | SCANNER-STORAGE-09-301 | Mongo catalog schemas/indexes for images, layers, artifacts, jobs, lifecycle rules plus migrations. | +| Sprint 9 | Scanner Core Foundations | src/StellaOps.Scanner.Storage/TASKS.md | DONE (2025-10-19) | Team Scanner Storage | SCANNER-STORAGE-09-302 | MinIO layout, immutability policies, client abstraction, and configuration binding. | +| Sprint 9 | Scanner Core Foundations | src/StellaOps.Scanner.Storage/TASKS.md | DONE (2025-10-19) | Team Scanner Storage | SCANNER-STORAGE-09-303 | Repositories/services with dual-write feature flag, deterministic digests, TTL enforcement tests. | +| Sprint 9 | Scanner Core Foundations | src/StellaOps.Scanner.Queue/TASKS.md | DONE (2025-10-19) | Team Scanner Queue | SCANNER-QUEUE-09-401 | Queue abstraction + Redis Streams adapter with ack/claim APIs and idempotency tokens. | +| Sprint 9 | Scanner Core Foundations | src/StellaOps.Scanner.Queue/TASKS.md | DONE (2025-10-19) | Team Scanner Queue | SCANNER-QUEUE-09-402 | Pluggable backend support (Redis, NATS) with configuration binding, health probes, failover docs. | +| Sprint 9 | Scanner Core Foundations | src/StellaOps.Scanner.Queue/TASKS.md | DONE (2025-10-19) | Team Scanner Queue | SCANNER-QUEUE-09-403 | Retry + dead-letter strategy with structured logs/metrics for offline deployments. | diff --git a/deploy/README.md b/deploy/README.md index 79554bfd..3e3dd92c 100644 --- a/deploy/README.md +++ b/deploy/README.md @@ -6,6 +6,7 @@ This directory contains deterministic deployment bundles for the core Stella Ops - `releases/` – canonical release manifests (edge, stable, airgap) used to source image digests. - `compose/` – Docker Compose bundles for dev/stage/airgap targets plus `.env` seed files. +- `compose/docker-compose.mirror.yaml` – managed mirror bundle for `*.stella-ops.org` with gateway cache and multi-tenant auth. - `helm/stellaops/` – multi-profile Helm chart with values files for dev/stage/airgap. - `tools/validate-profiles.sh` – helper that runs `docker compose config` and `helm lint/template` for every profile. diff --git a/deploy/compose/README.md b/deploy/compose/README.md index 9a1a250d..dfcfa470 100644 --- a/deploy/compose/README.md +++ b/deploy/compose/README.md @@ -9,6 +9,7 @@ These Compose bundles ship the minimum services required to exercise the scanner | `docker-compose.dev.yaml` | Edge/nightly stack tuned for laptops and iterative work. | | `docker-compose.stage.yaml` | Stable channel stack mirroring pre-production clusters. | | `docker-compose.airgap.yaml` | Stable stack with air-gapped defaults (no outbound hostnames). | +| `docker-compose.mirror.yaml` | Managed mirror topology for `*.stella-ops.org` distribution (Concelier + Excititor + CDN gateway). | | `env/*.env.example` | Seed `.env` files that document required secrets and ports per profile. | ## Usage diff --git a/deploy/compose/docker-compose.mirror.yaml b/deploy/compose/docker-compose.mirror.yaml new file mode 100644 index 00000000..5695673a --- /dev/null +++ b/deploy/compose/docker-compose.mirror.yaml @@ -0,0 +1,152 @@ +x-release-labels: &release-labels + com.stellaops.release.version: "2025.10.0-edge" + com.stellaops.release.channel: "edge" + com.stellaops.profile: "mirror-managed" + +networks: + mirror: + driver: bridge + +volumes: + mongo-data: + minio-data: + concelier-jobs: + concelier-exports: + excititor-exports: + nginx-cache: + +services: + mongo: + image: docker.io/library/mongo@sha256:c258b26dbb7774f97f52aff52231ca5f228273a84329c5f5e451c3739457db49 + command: ["mongod", "--bind_ip_all"] + restart: unless-stopped + environment: + MONGO_INITDB_ROOT_USERNAME: "${MONGO_INITDB_ROOT_USERNAME:-stellaops_mirror}" + MONGO_INITDB_ROOT_PASSWORD: "${MONGO_INITDB_ROOT_PASSWORD:-mirror-password}" + volumes: + - mongo-data:/data/db + networks: + - mirror + labels: *release-labels + + minio: + image: docker.io/minio/minio@sha256:14cea493d9a34af32f524e538b8346cf79f3321eff8e708c1e2960462bd8936e + command: ["server", "/data", "--console-address", ":9001"] + restart: unless-stopped + environment: + MINIO_ROOT_USER: "${MINIO_ROOT_USER:-stellaops-mirror}" + MINIO_ROOT_PASSWORD: "${MINIO_ROOT_PASSWORD:-mirror-minio-secret}" + volumes: + - minio-data:/data + networks: + - mirror + labels: *release-labels + + concelier: + image: registry.stella-ops.org/stellaops/concelier@sha256:dafef3954eb4b837e2c424dd2d23e1e4d60fa83794840fac9cd3dea1d43bd085 + restart: unless-stopped + depends_on: + - mongo + - minio + environment: + ASPNETCORE_URLS: "http://+:8445" + CONCELIER__STORAGE__MONGO__CONNECTIONSTRING: "mongodb://${MONGO_INITDB_ROOT_USERNAME:-stellaops_mirror}:${MONGO_INITDB_ROOT_PASSWORD:-mirror-password}@mongo:27017/concelier?authSource=admin" + CONCELIER__STORAGE__S3__ENDPOINT: "http://minio:9000" + CONCELIER__STORAGE__S3__ACCESSKEYID: "${MINIO_ROOT_USER:-stellaops-mirror}" + CONCELIER__STORAGE__S3__SECRETACCESSKEY: "${MINIO_ROOT_PASSWORD:-mirror-minio-secret}" + CONCELIER__TELEMETRY__SERVICENAME: "stellaops-concelier-mirror" + CONCELIER__MIRROR__ENABLED: "true" + CONCELIER__MIRROR__EXPORTROOT: "/exports/json" + CONCELIER__MIRROR__LATESTDIRECTORYNAME: "${CONCELIER_MIRROR_LATEST_SEGMENT:-latest}" + CONCELIER__MIRROR__MIRRORDIRECTORYNAME: "${CONCELIER_MIRROR_DIRECTORY_SEGMENT:-mirror}" + CONCELIER__MIRROR__REQUIREAUTHENTICATION: "${CONCELIER_MIRROR_REQUIRE_AUTH:-true}" + CONCELIER__MIRROR__MAXINDEXREQUESTSPERHOUR: "${CONCELIER_MIRROR_INDEX_BUDGET:-600}" + CONCELIER__MIRROR__DOMAINS__0__ID: "${CONCELIER_MIRROR_DOMAIN_PRIMARY_ID:-primary}" + CONCELIER__MIRROR__DOMAINS__0__DISPLAYNAME: "${CONCELIER_MIRROR_DOMAIN_PRIMARY_NAME:-Primary Mirror}" + CONCELIER__MIRROR__DOMAINS__0__REQUIREAUTHENTICATION: "${CONCELIER_MIRROR_DOMAIN_PRIMARY_AUTH:-true}" + CONCELIER__MIRROR__DOMAINS__0__MAXDOWNLOADREQUESTSPERHOUR: "${CONCELIER_MIRROR_DOMAIN_PRIMARY_DOWNLOAD_BUDGET:-3600}" + CONCELIER__MIRROR__DOMAINS__1__ID: "${CONCELIER_MIRROR_DOMAIN_SECONDARY_ID:-community}" + CONCELIER__MIRROR__DOMAINS__1__DISPLAYNAME: "${CONCELIER_MIRROR_DOMAIN_SECONDARY_NAME:-Community Mirror}" + CONCELIER__MIRROR__DOMAINS__1__REQUIREAUTHENTICATION: "${CONCELIER_MIRROR_DOMAIN_SECONDARY_AUTH:-false}" + CONCELIER__MIRROR__DOMAINS__1__MAXDOWNLOADREQUESTSPERHOUR: "${CONCELIER_MIRROR_DOMAIN_SECONDARY_DOWNLOAD_BUDGET:-1800}" + CONCELIER__AUTHORITY__ENABLED: "${CONCELIER_AUTHORITY_ENABLED:-true}" + CONCELIER__AUTHORITY__ALLOWANONYMOUSFALLBACK: "${CONCELIER_AUTHORITY_ALLOW_ANON:-false}" + CONCELIER__AUTHORITY__ISSUER: "${CONCELIER_AUTHORITY_ISSUER:-https://authority.stella-ops.org}" + CONCELIER__AUTHORITY__METADATAADDRESS: "${CONCELIER_AUTHORITY_METADATA:-}" + CONCELIER__AUTHORITY__CLIENTID: "${CONCELIER_AUTHORITY_CLIENT_ID:-stellaops-concelier-mirror}" + CONCELIER__AUTHORITY__CLIENTSECRETFILE: "/run/secrets/concelier-authority-client" + CONCELIER__AUTHORITY__CLIENTSCOPES__0: "${CONCELIER_AUTHORITY_SCOPE:-concelier.mirror.read}" + CONCELIER__AUTHORITY__AUDIENCES__0: "${CONCELIER_AUTHORITY_AUDIENCE:-api://concelier.mirror}" + CONCELIER__AUTHORITY__BYPASSNETWORKS__0: "10.0.0.0/8" + CONCELIER__AUTHORITY__BYPASSNETWORKS__1: "127.0.0.1/32" + CONCELIER__AUTHORITY__BYPASSNETWORKS__2: "::1/128" + CONCELIER__AUTHORITY__RESILIENCE__ENABLERETRIES: "true" + CONCELIER__AUTHORITY__RESILIENCE__RETRYDELAYS__0: "00:00:01" + CONCELIER__AUTHORITY__RESILIENCE__RETRYDELAYS__1: "00:00:02" + CONCELIER__AUTHORITY__RESILIENCE__RETRYDELAYS__2: "00:00:05" + CONCELIER__AUTHORITY__RESILIENCE__ALLOWOFFLINECACHEFALLBACK: "true" + CONCELIER__AUTHORITY__RESILIENCE__OFFLINECACHETOLERANCE: "00:10:00" + volumes: + - concelier-jobs:/var/lib/concelier/jobs + - concelier-exports:/exports/json + - ./mirror-secrets:/run/secrets:ro + networks: + - mirror + labels: *release-labels + + excititor: + image: registry.stella-ops.org/stellaops/excititor@sha256:d9bd5cadf1eab427447ce3df7302c30ded837239771cc6433b9befb895054285 + restart: unless-stopped + depends_on: + - mongo + environment: + ASPNETCORE_URLS: "http://+:8448" + EXCITITOR__STORAGE__MONGO__CONNECTIONSTRING: "mongodb://${MONGO_INITDB_ROOT_USERNAME:-stellaops_mirror}:${MONGO_INITDB_ROOT_PASSWORD:-mirror-password}@mongo:27017/excititor?authSource=admin" + EXCITITOR__STORAGE__MONGO__DATABASENAME: "${EXCITITOR_MONGO_DATABASE:-excititor}" + EXCITITOR__ARTIFACTS__FILESYSTEM__ROOT: "/exports" + EXCITITOR__ARTIFACTS__FILESYSTEM__OVERWRITEEXISTING: "${EXCITITOR_FILESYSTEM_OVERWRITE:-false}" + EXCITITOR__MIRROR__DOMAINS__0__ID: "${EXCITITOR_MIRROR_DOMAIN_PRIMARY_ID:-primary}" + EXCITITOR__MIRROR__DOMAINS__0__DISPLAYNAME: "${EXCITITOR_MIRROR_DOMAIN_PRIMARY_NAME:-Primary Mirror}" + EXCITITOR__MIRROR__DOMAINS__0__REQUIREAUTHENTICATION: "${EXCITITOR_MIRROR_DOMAIN_PRIMARY_AUTH:-true}" + EXCITITOR__MIRROR__DOMAINS__0__MAXINDEXREQUESTSPERHOUR: "${EXCITITOR_MIRROR_DOMAIN_PRIMARY_INDEX_BUDGET:-300}" + EXCITITOR__MIRROR__DOMAINS__0__MAXDOWNLOADREQUESTSPERHOUR: "${EXCITITOR_MIRROR_DOMAIN_PRIMARY_DOWNLOAD_BUDGET:-2400}" + EXCITITOR__MIRROR__DOMAINS__0__EXPORTS__0__KEY: "${EXCITITOR_MIRROR_PRIMARY_EXPORT_CONSENSUS_KEY:-consensus-json}" + EXCITITOR__MIRROR__DOMAINS__0__EXPORTS__0__FORMAT: "${EXCITITOR_MIRROR_PRIMARY_EXPORT_CONSENSUS_FORMAT:-json}" + EXCITITOR__MIRROR__DOMAINS__0__EXPORTS__0__VIEW: "${EXCITITOR_MIRROR_PRIMARY_EXPORT_CONSENSUS_VIEW:-consensus}" + EXCITITOR__MIRROR__DOMAINS__0__EXPORTS__1__KEY: "${EXCITITOR_MIRROR_PRIMARY_EXPORT_OPENVEX_KEY:-consensus-openvex}" + EXCITITOR__MIRROR__DOMAINS__0__EXPORTS__1__FORMAT: "${EXCITITOR_MIRROR_PRIMARY_EXPORT_OPENVEX_FORMAT:-openvex}" + EXCITITOR__MIRROR__DOMAINS__0__EXPORTS__1__VIEW: "${EXCITITOR_MIRROR_PRIMARY_EXPORT_OPENVEX_VIEW:-consensus}" + EXCITITOR__MIRROR__DOMAINS__1__ID: "${EXCITITOR_MIRROR_DOMAIN_SECONDARY_ID:-community}" + EXCITITOR__MIRROR__DOMAINS__1__DISPLAYNAME: "${EXCITITOR_MIRROR_DOMAIN_SECONDARY_NAME:-Community Mirror}" + EXCITITOR__MIRROR__DOMAINS__1__REQUIREAUTHENTICATION: "${EXCITITOR_MIRROR_DOMAIN_SECONDARY_AUTH:-false}" + EXCITITOR__MIRROR__DOMAINS__1__MAXINDEXREQUESTSPERHOUR: "${EXCITITOR_MIRROR_DOMAIN_SECONDARY_INDEX_BUDGET:-120}" + EXCITITOR__MIRROR__DOMAINS__1__MAXDOWNLOADREQUESTSPERHOUR: "${EXCITITOR_MIRROR_DOMAIN_SECONDARY_DOWNLOAD_BUDGET:-600}" + EXCITITOR__MIRROR__DOMAINS__1__EXPORTS__0__KEY: "${EXCITITOR_MIRROR_SECONDARY_EXPORT_KEY:-community-consensus}" + EXCITITOR__MIRROR__DOMAINS__1__EXPORTS__0__FORMAT: "${EXCITITOR_MIRROR_SECONDARY_EXPORT_FORMAT:-json}" + EXCITITOR__MIRROR__DOMAINS__1__EXPORTS__0__VIEW: "${EXCITITOR_MIRROR_SECONDARY_EXPORT_VIEW:-consensus}" + volumes: + - excititor-exports:/exports + - ./mirror-secrets:/run/secrets:ro + expose: + - "8448" + networks: + - mirror + labels: *release-labels + + mirror-gateway: + image: docker.io/library/nginx@sha256:208b70eefac13ee9be00e486f79c695b15cef861c680527171a27d253d834be9 + restart: unless-stopped + depends_on: + - concelier + - excititor + ports: + - "${MIRROR_GATEWAY_HTTP_PORT:-8080}:80" + - "${MIRROR_GATEWAY_HTTPS_PORT:-9443}:443" + volumes: + - nginx-cache:/var/cache/nginx + - ./mirror-gateway/conf.d:/etc/nginx/conf.d:ro + - ./mirror-gateway/tls:/etc/nginx/tls:ro + - ./mirror-gateway/secrets:/etc/nginx/secrets:ro + networks: + - mirror + labels: *release-labels diff --git a/deploy/compose/env/mirror.env.example b/deploy/compose/env/mirror.env.example new file mode 100644 index 00000000..cc78c8cc --- /dev/null +++ b/deploy/compose/env/mirror.env.example @@ -0,0 +1,57 @@ +# Managed mirror profile substitutions + +# Core infrastructure credentials +MONGO_INITDB_ROOT_USERNAME=stellaops_mirror +MONGO_INITDB_ROOT_PASSWORD=mirror-password +MINIO_ROOT_USER=stellaops-mirror +MINIO_ROOT_PASSWORD=mirror-minio-secret + +# Mirror HTTP listeners +MIRROR_GATEWAY_HTTP_PORT=8080 +MIRROR_GATEWAY_HTTPS_PORT=9443 + +# Concelier mirror configuration +CONCELIER_MIRROR_LATEST_SEGMENT=latest +CONCELIER_MIRROR_DIRECTORY_SEGMENT=mirror +CONCELIER_MIRROR_REQUIRE_AUTH=true +CONCELIER_MIRROR_INDEX_BUDGET=600 +CONCELIER_MIRROR_DOMAIN_PRIMARY_ID=primary +CONCELIER_MIRROR_DOMAIN_PRIMARY_NAME=Primary Mirror +CONCELIER_MIRROR_DOMAIN_PRIMARY_AUTH=true +CONCELIER_MIRROR_DOMAIN_PRIMARY_DOWNLOAD_BUDGET=3600 +CONCELIER_MIRROR_DOMAIN_SECONDARY_ID=community +CONCELIER_MIRROR_DOMAIN_SECONDARY_NAME=Community Mirror +CONCELIER_MIRROR_DOMAIN_SECONDARY_AUTH=false +CONCELIER_MIRROR_DOMAIN_SECONDARY_DOWNLOAD_BUDGET=1800 + +# Authority integration (tokens issued by production Authority) +CONCELIER_AUTHORITY_ENABLED=true +CONCELIER_AUTHORITY_ALLOW_ANON=false +CONCELIER_AUTHORITY_ISSUER=https://authority.stella-ops.org +CONCELIER_AUTHORITY_METADATA= +CONCELIER_AUTHORITY_CLIENT_ID=stellaops-concelier-mirror +CONCELIER_AUTHORITY_SCOPE=concelier.mirror.read +CONCELIER_AUTHORITY_AUDIENCE=api://concelier.mirror + +# Excititor mirror configuration +EXCITITOR_MONGO_DATABASE=excititor +EXCITITOR_FILESYSTEM_OVERWRITE=false +EXCITITOR_MIRROR_DOMAIN_PRIMARY_ID=primary +EXCITITOR_MIRROR_DOMAIN_PRIMARY_NAME=Primary Mirror +EXCITITOR_MIRROR_DOMAIN_PRIMARY_AUTH=true +EXCITITOR_MIRROR_DOMAIN_PRIMARY_INDEX_BUDGET=300 +EXCITITOR_MIRROR_DOMAIN_PRIMARY_DOWNLOAD_BUDGET=2400 +EXCITITOR_MIRROR_PRIMARY_EXPORT_CONSENSUS_KEY=consensus-json +EXCITITOR_MIRROR_PRIMARY_EXPORT_CONSENSUS_FORMAT=json +EXCITITOR_MIRROR_PRIMARY_EXPORT_CONSENSUS_VIEW=consensus +EXCITITOR_MIRROR_PRIMARY_EXPORT_OPENVEX_KEY=consensus-openvex +EXCITITOR_MIRROR_PRIMARY_EXPORT_OPENVEX_FORMAT=openvex +EXCITITOR_MIRROR_PRIMARY_EXPORT_OPENVEX_VIEW=consensus +EXCITITOR_MIRROR_DOMAIN_SECONDARY_ID=community +EXCITITOR_MIRROR_DOMAIN_SECONDARY_NAME=Community Mirror +EXCITITOR_MIRROR_DOMAIN_SECONDARY_AUTH=false +EXCITITOR_MIRROR_DOMAIN_SECONDARY_INDEX_BUDGET=120 +EXCITITOR_MIRROR_DOMAIN_SECONDARY_DOWNLOAD_BUDGET=600 +EXCITITOR_MIRROR_SECONDARY_EXPORT_KEY=community-consensus +EXCITITOR_MIRROR_SECONDARY_EXPORT_FORMAT=json +EXCITITOR_MIRROR_SECONDARY_EXPORT_VIEW=consensus diff --git a/deploy/compose/mirror-data/concelier/.gitkeep b/deploy/compose/mirror-data/concelier/.gitkeep new file mode 100644 index 00000000..e69de29b diff --git a/deploy/compose/mirror-data/excititor/.gitkeep b/deploy/compose/mirror-data/excititor/.gitkeep new file mode 100644 index 00000000..e69de29b diff --git a/deploy/compose/mirror-gateway/README.md b/deploy/compose/mirror-gateway/README.md new file mode 100644 index 00000000..380a53a9 --- /dev/null +++ b/deploy/compose/mirror-gateway/README.md @@ -0,0 +1,13 @@ +# Mirror Gateway Assets + +This directory holds the reverse-proxy configuration and TLS material for the managed +mirror profile: + +- `conf.d/*.conf` – nginx configuration shipped with the profile. +- `tls/` – place environment-specific certificates and private keys + (`mirror-primary.{crt,key}`, `mirror-community.{crt,key}`, etc.). +- `secrets/` – populate Basic Auth credential stores (`*.htpasswd`) that gate each + mirror domain. Generate with `htpasswd -B`. + +The Compose bundle mounts these paths read-only. Populate `tls/` with the actual +certificates before invoking `docker compose config` or `docker compose up`. diff --git a/deploy/compose/mirror-gateway/conf.d/mirror-locations.conf b/deploy/compose/mirror-gateway/conf.d/mirror-locations.conf new file mode 100644 index 00000000..b9932b7d --- /dev/null +++ b/deploy/compose/mirror-gateway/conf.d/mirror-locations.conf @@ -0,0 +1,44 @@ +proxy_set_header Host $host; +proxy_set_header X-Real-IP $remote_addr; +proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; +proxy_set_header X-Forwarded-Proto $scheme; +proxy_redirect off; + +add_header X-Cache-Status $upstream_cache_status always; + +location = /healthz { + default_type application/json; + return 200 '{"status":"ok"}'; +} + +location /concelier/exports/ { + proxy_pass http://concelier_backend/concelier/exports/; + proxy_cache mirror_cache; + proxy_cache_key $mirror_cache_key; + proxy_cache_valid 200 5m; + proxy_cache_valid 404 1m; + add_header Cache-Control "public, max-age=300, immutable" always; +} + +location /concelier/ { + proxy_pass http://concelier_backend/concelier/; + proxy_cache off; +} + +location /excititor/mirror/ { + proxy_pass http://excititor_backend/excititor/mirror/; + proxy_cache mirror_cache; + proxy_cache_key $mirror_cache_key; + proxy_cache_valid 200 5m; + proxy_cache_valid 404 1m; + add_header Cache-Control "public, max-age=300, immutable" always; +} + +location /excititor/ { + proxy_pass http://excititor_backend/excititor/; + proxy_cache off; +} + +location / { + return 404; +} diff --git a/deploy/compose/mirror-gateway/conf.d/mirror.conf b/deploy/compose/mirror-gateway/conf.d/mirror.conf new file mode 100644 index 00000000..b389eeb3 --- /dev/null +++ b/deploy/compose/mirror-gateway/conf.d/mirror.conf @@ -0,0 +1,51 @@ +proxy_cache_path /var/cache/nginx/mirror levels=1:2 keys_zone=mirror_cache:100m max_size=10g inactive=12h use_temp_path=off; + +map $request_uri $mirror_cache_key { + default $scheme$request_method$host$request_uri; +} + +upstream concelier_backend { + server concelier:8445; + keepalive 32; +} + +upstream excititor_backend { + server excititor:8448; + keepalive 32; +} + +server { + listen 80; + server_name _; + return 301 https://$host$request_uri; +} + +server { + listen 443 ssl http2; + server_name mirror-primary.stella-ops.org; + + ssl_certificate /etc/nginx/tls/mirror-primary.crt; + ssl_certificate_key /etc/nginx/tls/mirror-primary.key; + ssl_protocols TLSv1.2 TLSv1.3; + ssl_prefer_server_ciphers on; + + auth_basic "StellaOps Mirror – primary"; + auth_basic_user_file /etc/nginx/secrets/mirror-primary.htpasswd; + + include /etc/nginx/conf.d/mirror-locations.conf; +} + +server { + listen 443 ssl http2; + server_name mirror-community.stella-ops.org; + + ssl_certificate /etc/nginx/tls/mirror-community.crt; + ssl_certificate_key /etc/nginx/tls/mirror-community.key; + ssl_protocols TLSv1.2 TLSv1.3; + ssl_prefer_server_ciphers on; + + auth_basic "StellaOps Mirror – community"; + auth_basic_user_file /etc/nginx/secrets/mirror-community.htpasswd; + + include /etc/nginx/conf.d/mirror-locations.conf; +} diff --git a/deploy/compose/mirror-gateway/secrets/.gitkeep b/deploy/compose/mirror-gateway/secrets/.gitkeep new file mode 100644 index 00000000..e69de29b diff --git a/deploy/compose/mirror-gateway/tls/.gitkeep b/deploy/compose/mirror-gateway/tls/.gitkeep new file mode 100644 index 00000000..e69de29b diff --git a/deploy/compose/mirror-secrets/.gitkeep b/deploy/compose/mirror-secrets/.gitkeep new file mode 100644 index 00000000..e69de29b diff --git a/deploy/helm/stellaops/values-mirror.yaml b/deploy/helm/stellaops/values-mirror.yaml new file mode 100644 index 00000000..8a60441c --- /dev/null +++ b/deploy/helm/stellaops/values-mirror.yaml @@ -0,0 +1,282 @@ +global: + profile: mirror-managed + release: + version: "2025.10.0-edge" + channel: edge + manifestSha256: "822f82987529ea38d2321dbdd2ef6874a4062a117116a20861c26a8df1807beb" + image: + pullPolicy: IfNotPresent + labels: + stellaops.io/channel: edge + +configMaps: + mirror-gateway: + data: + mirror.conf: | + proxy_cache_path /var/cache/nginx/mirror levels=1:2 keys_zone=mirror_cache:100m max_size=10g inactive=12h use_temp_path=off; + + map $request_uri $mirror_cache_key { + default $scheme$request_method$host$request_uri; + } + + upstream concelier_backend { + server stellaops-concelier:8445; + keepalive 32; + } + + upstream excititor_backend { + server stellaops-excititor:8448; + keepalive 32; + } + + server { + listen 80; + server_name _; + return 301 https://$host$request_uri; + } + + server { + listen 443 ssl http2; + server_name mirror-primary.stella-ops.org; + + ssl_certificate /etc/nginx/tls/mirror-primary.crt; + ssl_certificate_key /etc/nginx/tls/mirror-primary.key; + ssl_protocols TLSv1.2 TLSv1.3; + ssl_prefer_server_ciphers on; + + auth_basic "StellaOps Mirror – primary"; + auth_basic_user_file /etc/nginx/secrets/mirror-primary.htpasswd; + + include /etc/nginx/conf.d/mirror-locations.conf; + } + + server { + listen 443 ssl http2; + server_name mirror-community.stella-ops.org; + + ssl_certificate /etc/nginx/tls/mirror-community.crt; + ssl_certificate_key /etc/nginx/tls/mirror-community.key; + ssl_protocols TLSv1.2 TLSv1.3; + ssl_prefer_server_ciphers on; + + auth_basic "StellaOps Mirror – community"; + auth_basic_user_file /etc/nginx/secrets/mirror-community.htpasswd; + + include /etc/nginx/conf.d/mirror-locations.conf; + } + mirror-locations.conf: | + proxy_set_header Host $host; + proxy_set_header X-Real-IP $remote_addr; + proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; + proxy_set_header X-Forwarded-Proto $scheme; + proxy_redirect off; + + add_header X-Cache-Status $upstream_cache_status always; + + location = /healthz { + default_type application/json; + return 200 '{"status":"ok"}'; + } + + location /concelier/exports/ { + proxy_pass http://concelier_backend/concelier/exports/; + proxy_cache mirror_cache; + proxy_cache_key $mirror_cache_key; + proxy_cache_valid 200 5m; + proxy_cache_valid 404 1m; + add_header Cache-Control "public, max-age=300, immutable" always; + } + + location /concelier/ { + proxy_pass http://concelier_backend/concelier/; + proxy_cache off; + } + + location /excititor/mirror/ { + proxy_pass http://excititor_backend/excititor/mirror/; + proxy_cache mirror_cache; + proxy_cache_key $mirror_cache_key; + proxy_cache_valid 200 5m; + proxy_cache_valid 404 1m; + add_header Cache-Control "public, max-age=300, immutable" always; + } + + location /excititor/ { + proxy_pass http://excititor_backend/excititor/; + proxy_cache off; + } + + location / { + return 404; + } + +services: + concelier: + image: registry.stella-ops.org/stellaops/concelier@sha256:dafef3954eb4b837e2c424dd2d23e1e4d60fa83794840fac9cd3dea1d43bd085 + service: + port: 8445 + env: + ASPNETCORE_URLS: "http://+:8445" + CONCELIER__STORAGE__MONGO__CONNECTIONSTRING: "mongodb://stellaops_mirror:mirror-password@stellaops-mongo:27017/concelier?authSource=admin" + CONCELIER__STORAGE__S3__ENDPOINT: "http://stellaops-minio:9000" + CONCELIER__STORAGE__S3__ACCESSKEYID: "stellaops-mirror" + CONCELIER__STORAGE__S3__SECRETACCESSKEY: "mirror-minio-secret" + CONCELIER__TELEMETRY__SERVICENAME: "stellaops-concelier-mirror" + CONCELIER__MIRROR__ENABLED: "true" + CONCELIER__MIRROR__EXPORTROOT: "/exports/json" + CONCELIER__MIRROR__LATESTDIRECTORYNAME: "latest" + CONCELIER__MIRROR__MIRRORDIRECTORYNAME: "mirror" + CONCELIER__MIRROR__REQUIREAUTHENTICATION: "true" + CONCELIER__MIRROR__MAXINDEXREQUESTSPERHOUR: "600" + CONCELIER__MIRROR__DOMAINS__0__ID: "primary" + CONCELIER__MIRROR__DOMAINS__0__DISPLAYNAME: "Primary Mirror" + CONCELIER__MIRROR__DOMAINS__0__REQUIREAUTHENTICATION: "true" + CONCELIER__MIRROR__DOMAINS__0__MAXDOWNLOADREQUESTSPERHOUR: "3600" + CONCELIER__MIRROR__DOMAINS__1__ID: "community" + CONCELIER__MIRROR__DOMAINS__1__DISPLAYNAME: "Community Mirror" + CONCELIER__MIRROR__DOMAINS__1__REQUIREAUTHENTICATION: "false" + CONCELIER__MIRROR__DOMAINS__1__MAXDOWNLOADREQUESTSPERHOUR: "1800" + CONCELIER__AUTHORITY__ENABLED: "true" + CONCELIER__AUTHORITY__ALLOWANONYMOUSFALLBACK: "false" + CONCELIER__AUTHORITY__ISSUER: "https://authority.stella-ops.org" + CONCELIER__AUTHORITY__METADATAADDRESS: "" + CONCELIER__AUTHORITY__CLIENTID: "stellaops-concelier-mirror" + CONCELIER__AUTHORITY__CLIENTSECRETFILE: "/run/secrets/concelier-authority-client" + CONCELIER__AUTHORITY__CLIENTSCOPES__0: "concelier.mirror.read" + CONCELIER__AUTHORITY__AUDIENCES__0: "api://concelier.mirror" + CONCELIER__AUTHORITY__BYPASSNETWORKS__0: "10.0.0.0/8" + CONCELIER__AUTHORITY__BYPASSNETWORKS__1: "127.0.0.1/32" + CONCELIER__AUTHORITY__BYPASSNETWORKS__2: "::1/128" + CONCELIER__AUTHORITY__RESILIENCE__ENABLERETRIES: "true" + CONCELIER__AUTHORITY__RESILIENCE__RETRYDELAYS__0: "00:00:01" + CONCELIER__AUTHORITY__RESILIENCE__RETRYDELAYS__1: "00:00:02" + CONCELIER__AUTHORITY__RESILIENCE__RETRYDELAYS__2: "00:00:05" + CONCELIER__AUTHORITY__RESILIENCE__ALLOWOFFLINECACHEFALLBACK: "true" + CONCELIER__AUTHORITY__RESILIENCE__OFFLINECACHETOLERANCE: "00:10:00" + volumeMounts: + - name: concelier-jobs + mountPath: /var/lib/concelier/jobs + - name: concelier-exports + mountPath: /exports/json + - name: concelier-secrets + mountPath: /run/secrets + readOnly: true + volumes: + - name: concelier-jobs + persistentVolumeClaim: + claimName: concelier-mirror-jobs + - name: concelier-exports + persistentVolumeClaim: + claimName: concelier-mirror-exports + - name: concelier-secrets + secret: + secretName: concelier-mirror-auth + + excititor: + image: registry.stella-ops.org/stellaops/excititor@sha256:d9bd5cadf1eab427447ce3df7302c30ded837239771cc6433b9befb895054285 + env: + ASPNETCORE_URLS: "http://+:8448" + EXCITITOR__STORAGE__MONGO__CONNECTIONSTRING: "mongodb://stellaops_mirror:mirror-password@stellaops-mongo:27017/excititor?authSource=admin" + EXCITITOR__STORAGE__MONGO__DATABASENAME: "excititor" + EXCITITOR__ARTIFACTS__FILESYSTEM__ROOT: "/exports" + EXCITITOR__ARTIFACTS__FILESYSTEM__OVERWRITEEXISTING: "false" + EXCITITOR__MIRROR__DOMAINS__0__ID: "primary" + EXCITITOR__MIRROR__DOMAINS__0__DISPLAYNAME: "Primary Mirror" + EXCITITOR__MIRROR__DOMAINS__0__REQUIREAUTHENTICATION: "true" + EXCITITOR__MIRROR__DOMAINS__0__MAXINDEXREQUESTSPERHOUR: "300" + EXCITITOR__MIRROR__DOMAINS__0__MAXDOWNLOADREQUESTSPERHOUR: "2400" + EXCITITOR__MIRROR__DOMAINS__0__EXPORTS__0__KEY: "consensus-json" + EXCITITOR__MIRROR__DOMAINS__0__EXPORTS__0__FORMAT: "json" + EXCITITOR__MIRROR__DOMAINS__0__EXPORTS__0__VIEW: "consensus" + EXCITITOR__MIRROR__DOMAINS__0__EXPORTS__1__KEY: "consensus-openvex" + EXCITITOR__MIRROR__DOMAINS__0__EXPORTS__1__FORMAT: "openvex" + EXCITITOR__MIRROR__DOMAINS__0__EXPORTS__1__VIEW: "consensus" + EXCITITOR__MIRROR__DOMAINS__1__ID: "community" + EXCITITOR__MIRROR__DOMAINS__1__DISPLAYNAME: "Community Mirror" + EXCITITOR__MIRROR__DOMAINS__1__REQUIREAUTHENTICATION: "false" + EXCITITOR__MIRROR__DOMAINS__1__MAXINDEXREQUESTSPERHOUR: "120" + EXCITITOR__MIRROR__DOMAINS__1__MAXDOWNLOADREQUESTSPERHOUR: "600" + EXCITITOR__MIRROR__DOMAINS__1__EXPORTS__0__KEY: "community-consensus" + EXCITITOR__MIRROR__DOMAINS__1__EXPORTS__0__FORMAT: "json" + EXCITITOR__MIRROR__DOMAINS__1__EXPORTS__0__VIEW: "consensus" + volumeMounts: + - name: excititor-exports + mountPath: /exports + - name: excititor-secrets + mountPath: /run/secrets + readOnly: true + volumes: + - name: excititor-exports + persistentVolumeClaim: + claimName: excititor-mirror-exports + - name: excititor-secrets + secret: + secretName: excititor-mirror-auth + + mongo: + class: infrastructure + image: docker.io/library/mongo@sha256:c258b26dbb7774f97f52aff52231ca5f228273a84329c5f5e451c3739457db49 + service: + port: 27017 + command: + - mongod + - --bind_ip_all + env: + MONGO_INITDB_ROOT_USERNAME: "stellaops_mirror" + MONGO_INITDB_ROOT_PASSWORD: "mirror-password" + volumeMounts: + - name: mongo-data + mountPath: /data/db + volumeClaims: + - name: mongo-data + claimName: mirror-mongo-data + + minio: + class: infrastructure + image: docker.io/minio/minio@sha256:14cea493d9a34af32f524e538b8346cf79f3321eff8e708c1e2960462bd8936e + service: + port: 9000 + command: + - server + - /data + - --console-address + - :9001 + env: + MINIO_ROOT_USER: "stellaops-mirror" + MINIO_ROOT_PASSWORD: "mirror-minio-secret" + volumeMounts: + - name: minio-data + mountPath: /data + volumeClaims: + - name: minio-data + claimName: mirror-minio-data + + mirror-gateway: + image: docker.io/library/nginx@sha256:208b70eefac13ee9be00e486f79c695b15cef861c680527171a27d253d834be9 + service: + type: LoadBalancer + port: 443 + portName: https + targetPort: 443 + configMounts: + - name: mirror-gateway-conf + mountPath: /etc/nginx/conf.d + configMap: mirror-gateway + volumeMounts: + - name: mirror-gateway-tls + mountPath: /etc/nginx/tls + readOnly: true + - name: mirror-gateway-secrets + mountPath: /etc/nginx/secrets + readOnly: true + - name: mirror-cache + mountPath: /var/cache/nginx + volumes: + - name: mirror-gateway-tls + secret: + secretName: mirror-gateway-tls + - name: mirror-gateway-secrets + secret: + secretName: mirror-gateway-htpasswd + - name: mirror-cache + emptyDir: {} diff --git a/deploy/tools/validate-profiles.sh b/deploy/tools/validate-profiles.sh index d7148385..f4b3a806 100644 --- a/deploy/tools/validate-profiles.sh +++ b/deploy/tools/validate-profiles.sh @@ -9,6 +9,7 @@ compose_profiles=( "docker-compose.dev.yaml:env/dev.env.example" "docker-compose.stage.yaml:env/stage.env.example" "docker-compose.airgap.yaml:env/airgap.env.example" + "docker-compose.mirror.yaml:env/mirror.env.example" ) docker_ready=false @@ -36,6 +37,7 @@ helm_values=( "$HELM_DIR/values-dev.yaml" "$HELM_DIR/values-stage.yaml" "$HELM_DIR/values-airgap.yaml" + "$HELM_DIR/values-mirror.yaml" ) if command -v helm >/dev/null 2>&1; then diff --git a/docs/09_API_CLI_REFERENCE.md b/docs/09_API_CLI_REFERENCE.md index f8a52ce8..94daac59 100755 --- a/docs/09_API_CLI_REFERENCE.md +++ b/docs/09_API_CLI_REFERENCE.md @@ -522,10 +522,12 @@ See `docs/dev/32_AUTH_CLIENT_GUIDE.md` for recommended profiles (online vs. air- | `stellaops-cli auth revoke export` | Export the Authority revocation bundle | `--output ` (defaults to CWD) | Writes `revocation-bundle.json`, `.json.jws`, and `.json.sha256`; verifies the digest locally and includes key metadata in the log summary. | | `stellaops-cli auth revoke verify` | Validate a revocation bundle offline | `--bundle ` `--signature ` `--key `
`--verbose` | Verifies detached JWS signatures, reports the computed SHA-256, and can fall back to cached JWKS when `--key` is omitted. | | `stellaops-cli config show` | Display resolved configuration | — | Masks secret values; helpful for air‑gapped installs | -| `stellaops-cli runtime policy test` | Ask Scanner.WebService for runtime verdicts (Webhook parity) | `--image/-i ` (repeatable, comma/space lists supported)
`--file/-f `
`--namespace/--ns `
`--label/-l key=value` (repeatable)
`--json` | Posts to `POST /api/v1/scanner/policy/runtime`, deduplicates image digests, and prints TTL + per-image verdict/signed/SBOM status. Accepts newline/whitespace-delimited stdin when piped; `--json` emits the raw response without additional logging. | +| `stellaops-cli runtime policy test` | Ask Scanner.WebService for runtime verdicts (Webhook parity) | `--image/-i ` (repeatable, comma/space lists supported)
`--file/-f `
`--namespace/--ns `
`--label/-l key=value` (repeatable)
`--json` | Posts to `POST /api/v1/scanner/policy/runtime`, deduplicates image digests, and prints TTL/policy revision plus per-image columns for signed state, SBOM referrers, quieted-by metadata, confidence, and Rekor attestation (uuid + verified flag). Accepts newline/whitespace-delimited stdin when piped; `--json` emits the raw response without additional logging. | When running on an interactive terminal without explicit override flags, the CLI uses Spectre.Console prompts to let you choose per-run ORAS/offline bundle behaviour. +Runtime verdict output reflects the SCANNER-RUNTIME-12-302 contract sign-off (quieted provenance, confidence band, attestation verification). CLI-RUNTIME-13-008 now mirrors those fields in both table and `--json` formats. + **Startup diagnostics** - `stellaops-cli` now loads Authority plug-in manifests during startup (respecting `Authority:Plugins:*`) and surfaces analyzer warnings when a plug-in weakens the baseline password policy (minimum length **12** and all character classes required). diff --git a/docs/10_CONCELIER_CLI_QUICKSTART.md b/docs/10_CONCELIER_CLI_QUICKSTART.md index 25cc2993..ce0f17ae 100644 --- a/docs/10_CONCELIER_CLI_QUICKSTART.md +++ b/docs/10_CONCELIER_CLI_QUICKSTART.md @@ -12,7 +12,7 @@ runtime wiring, CLI usage) and leaves connector/internal customization for later - .NET SDK **10.0.100-preview** (matches `global.json`) - MongoDB instance reachable from the host (local Docker or managed) - `trivy-db` binary on `PATH` for Trivy exports (and `oras` if publishing to OCI) -- Plugin assemblies present in `PluginBinaries/` (already included in the repo) +- Plugin assemblies present in `StellaOps.Concelier.PluginBinaries/` (already included in the repo) - Optional: Docker/Podman runtime if you plan to run scanners locally > **Tip** – air-gapped installs should preload `trivy-db` and `oras` binaries into the @@ -31,7 +31,7 @@ runtime wiring, CLI usage) and leaves connector/internal customization for later ``` 2. Edit `etc/concelier.yaml` and update the MongoDB DSN (and optional database name). - The default template configures plug-in discovery to look in `PluginBinaries/` + The default template configures plug-in discovery to look in `StellaOps.Concelier.PluginBinaries/` and disables remote telemetry exporters by default. 3. (Optional) Override settings via environment variables. All keys are prefixed with @@ -71,7 +71,7 @@ runtime wiring, CLI usage) and leaves connector/internal customization for later environment. Authority expects per-plugin manifests in `etc/authority.plugins/`; sample `standard.yaml` and `ldap.yaml` files are provided as starting points. For air-gapped installs keep the default plug-in binary directory - (`../PluginBinaries/Authority`) so packaged plug-ins load without outbound access. + (`../StellaOps.Authority.PluginBinaries`) so packaged plug-ins load without outbound access. 3. Environment variables prefixed with `STELLAOPS_AUTHORITY_` override individual fields. Example: diff --git a/docs/10_PLUGIN_SDK_GUIDE.md b/docs/10_PLUGIN_SDK_GUIDE.md index 0956c046..d5a02314 100755 --- a/docs/10_PLUGIN_SDK_GUIDE.md +++ b/docs/10_PLUGIN_SDK_GUIDE.md @@ -82,28 +82,53 @@ Add this to **`MyPlugin.Schedule.csproj`** so the signed DLL + `.sig` land in th --- -## 5 Dependency‑Injection Entry‑point - -Back‑end auto‑discovers the static method below: - -~~~csharp -namespace StellaOps.DependencyInjection; - -public static class IoCConfigurator -{ - public static IServiceCollection Configure(this IServiceCollection services, - IConfiguration cfg) - { - services.AddSingleton(); // schedule job - services.Configure(cfg.GetSection("Plugins:MyPlugin")); - return services; - } -} -~~~ - ---- - -## 6 Schedule Plug‑ins +## 5 Dependency‑Injection Entry‑point + +Back‑end auto‑discovers restart‑time bindings through two mechanisms: + +1. **Service binding metadata** for simple contracts. +2. **`IDependencyInjectionRoutine`** implementations when you need full control. + +### 5.1 Service binding metadata + +Annotate implementations with `[ServiceBinding]` to declare their lifetime and service contract. +The loader honours scoped lifetimes and will register the service before executing any custom DI routines. + +~~~csharp +using Microsoft.Extensions.DependencyInjection; +using StellaOps.DependencyInjection; + +[ServiceBinding(typeof(IJob), ServiceLifetime.Scoped, RegisterAsSelf = true)] +public sealed class MyJob : IJob +{ + // IJob dependencies can now use scoped services (Mongo sessions, etc.) +} +~~~ + +Use `RegisterAsSelf = true` when you also want to resolve the concrete type. +Set `ReplaceExisting = true` to override default descriptors if the host already provides one. + +### 5.2 Dependency injection routines + +For advanced scenarios continue to expose a routine: + +~~~csharp +namespace StellaOps.DependencyInjection; + +public sealed class IoCConfigurator : IDependencyInjectionRoutine +{ + public IServiceCollection Register(IServiceCollection services, IConfiguration cfg) + { + services.AddSingleton(); // schedule job + services.Configure(cfg.GetSection("Plugins:MyPlugin")); + return services; + } +} +~~~ + +--- + +## 6 Schedule Plug‑ins ### 6.1 Minimal Job @@ -191,4 +216,4 @@ On merge, the plug‑in shows up in the UI Marketplace. | NotDetected | .sig missing | cosign sign … | | VersionGateMismatch | Backend 2.1 vs plug‑in 2.0 | Re‑compile / bump attribute | | FileLoadException | Duplicate | StellaOps.Common Ensure PrivateAssets="all" | -| Redis | timeouts Large writes | Batch or use Mongo | \ No newline at end of file +| Redis | timeouts Large writes | Batch or use Mongo | diff --git a/docs/11_AUTHORITY.md b/docs/11_AUTHORITY.md index 9cd379b5..42f0b76c 100644 --- a/docs/11_AUTHORITY.md +++ b/docs/11_AUTHORITY.md @@ -147,6 +147,20 @@ All administrative calls emit `AuthEventRecord` entries enriched with correlatio | Security | `security.rateLimiting` | Fixed-window limits for `/token`, `/authorize`, `/internal/*`. | See `docs/security/rate-limits.md` for tuning. | | Bootstrap | `bootstrap.apiKey` | Shared secret required for `/internal/*`. | Only required when `bootstrap.enabled` is true. | +### 7.1 Sender-constrained clients (DPoP & mTLS) + +Authority now understands two flavours of sender-constrained OAuth clients: + +- **DPoP proof-of-possession** – clients sign a `DPoP` header for `/token` requests. Authority validates the JWK thumbprint, HTTP method/URI, and replay window, then stamps the resulting access token with `cnf.jkt` so downstream services can verify the same key is reused. + - Configure under `security.senderConstraints.dpop`. `allowedAlgorithms`, `proofLifetime`, and `replayWindow` are enforced at validation time. + - `security.senderConstraints.dpop.nonce.enabled` enables nonce challenges for high-value audiences (`requiredAudiences`, normalised to case-insensitive strings). When a nonce is required but missing or expired, `/token` replies with `WWW-Authenticate: DPoP error="use_dpop_nonce"` (and, when available, a fresh `DPoP-Nonce` header). Clients must retry with the issued nonce embedded in the proof. + - `security.senderConstraints.dpop.nonce.store` selects `memory` (default) or `redis`. When `redis` is configured, set `security.senderConstraints.dpop.nonce.redisConnectionString` so replicas share nonce issuance and high-value clients avoid replay gaps during failover. + - Declare client `audiences` in bootstrap manifests or plug-in provisioning metadata; Authority now defaults the token `aud` claim and `resource` indicator from this list, which is also used to trigger nonce enforcement for audiences such as `signer` and `attestor`. +- **Mutual TLS clients** – client registrations may declare an mTLS binding (`senderConstraint: mtls`). When enabled via `security.senderConstraints.mtls`, Authority validates the presented client certificate against stored bindings (`certificateBindings[]`), optional chain verification, and timing windows. Successful requests embed `cnf.x5t#S256` into the access token so resource servers can enforce the certificate thumbprint. + - Certificate bindings record the certificate thumbprint, optional SANs, subject/issuer metadata, and activation windows. Operators can enforce subject regexes, SAN type allow-lists (`dns`, `uri`, `ip`), trusted certificate authorities, and rotation grace via `security.senderConstraints.mtls.*`. + +Both modes persist additional metadata in `authority_tokens`: `senderConstraint` records the enforced policy, while `senderKeyThumbprint` stores the DPoP JWK thumbprint or mTLS certificate hash captured at issuance. Downstream services can rely on these fields (and the corresponding `cnf` claim) when auditing offline copies of the token store. + ## 8. Offline & Sovereign Operation - **No outbound dependencies:** Authority only contacts MongoDB and local plugins. Discovery and JWKS are cached by clients with offline tolerances (`AllowOfflineCacheFallback`, `OfflineCacheTolerance`). Operators should mirror these responses for air-gapped use. - **Structured logging:** Every revocation export, signing rotation, bootstrap action, and token issuance emits structured logs with `traceId`, `client_id`, `subjectId`, and `network.remoteIp` where applicable. Mirror logs to your SIEM to retain audit trails without central connectivity. diff --git a/docs/ARCHITECTURE_ATTESTOR.md b/docs/ARCHITECTURE_ATTESTOR.md index 5d9de7e8..9ea10b2c 100644 --- a/docs/ARCHITECTURE_ATTESTOR.md +++ b/docs/ARCHITECTURE_ATTESTOR.md @@ -200,7 +200,8 @@ Indexes: * Predicate `predicateType` must be on allowlist (sbom/report/vex-export). * `subject.digest.sha256` values must be present and well‑formed (hex). * **No public submission** path. **Never** accept bundles from untrusted clients. -* **Rate limits**: per mTLS thumbprint/license (from Signer‑forwarded claims) to avoid flooding the log. +* **Client certificate allowlists**: optional `security.mtls.allowedSubjects` / `allowedThumbprints` tighten peer identity checks beyond CA pinning. +* **Rate limits**: token-bucket per caller derived from `quotas.perCaller` (QPS/burst) returns `429` + `Retry-After` when exceeded. * **Redaction**: Attestor never logs secret material; DSSE payloads **should** be public by design (SBOMs/reports). If customers require redaction, enforce policy at Signer (predicate minimization) **before** Attestor. --- @@ -233,6 +234,10 @@ Indexes: * `attestor.dedupe_hits_total` * `attestor.errors_total{type}` +**Correlation**: + +* HTTP callers may supply `X-Correlation-Id`; Attestor will echo the header and push `CorrelationId` into the log scope for cross-service tracing. + **Tracing**: * Spans: `validate`, `rekor.submit`, `rekor.poll`, `persist`, `archive`, `verify`. diff --git a/docs/ARCHITECTURE_AUTHORITY.md b/docs/ARCHITECTURE_AUTHORITY.md index 5bad72b3..f4f92f4d 100644 --- a/docs/ARCHITECTURE_AUTHORITY.md +++ b/docs/ARCHITECTURE_AUTHORITY.md @@ -229,6 +229,8 @@ GET /admin/metrics # Prometheus exposition (token issue rates, GET /admin/healthz|readyz # health/readiness ``` +Declared client `audiences` flow through to the issued JWT `aud` claim and the token request's `resource` indicators. Authority relies on this metadata to enforce DPoP nonce challenges for `signer`, `attestor`, and other high-value services without requiring clients to repeat the audience parameter on every request. + --- ## 11) Integration hard lines (what resource servers must enforce) @@ -286,6 +288,8 @@ authority: nonce: enable: true ttlSeconds: 600 + store: redis + redisConnectionString: "redis://authority-redis:6379?ssl=false" mtls: enable: true caBundleFile: /etc/ssl/mtls/clients-ca.pem diff --git a/docs/ARCHITECTURE_CONCELIER.md b/docs/ARCHITECTURE_CONCELIER.md index 88c96a82..7ad36345 100644 --- a/docs/ARCHITECTURE_CONCELIER.md +++ b/docs/ARCHITECTURE_CONCELIER.md @@ -94,6 +94,34 @@ mergedAt inputs[] // source doc digests that contributed ``` +**AdvisoryStatement (event log)** + +``` +statementId // GUID (immutable) +vulnerabilityKey // canonical advisory key (e.g., CVE-2025-12345) +advisoryKey // merge snapshot advisory key (may reference variant) +statementHash // canonical hash of advisory payload +asOf // timestamp of snapshot (UTC) +recordedAt // persistence timestamp (UTC) +inputDocuments[] // document IDs contributing to the snapshot +payload // canonical advisory document (BSON / canonical JSON) +``` + +**AdvisoryConflict** + +``` +conflictId // GUID +vulnerabilityKey // canonical advisory key +conflictHash // deterministic hash of conflict payload +asOf // timestamp aligned with originating statement set +recordedAt // persistence timestamp +statementIds[] // related advisoryStatement identifiers +details // structured conflict explanation / merge reasoning +``` + +- `AdvisoryEventLog` (Concelier.Core) provides the public API for appending immutable statements/conflicts and querying replay history. Inputs are normalized by trimming and lower-casing `vulnerabilityKey`, serializing advisories with `CanonicalJsonSerializer`, and computing SHA-256 hashes (`statementHash`, `conflictHash`) over the canonical JSON payloads. Consumers can replay by key with an optional `asOf` filter to obtain deterministic snapshots ordered by `asOf` then `recordedAt`. +- Concelier.WebService exposes the immutable log via `GET /concelier/advisories/{vulnerabilityKey}/replay[?asOf=UTC_ISO8601]`, returning the latest statements (with hex-encoded hashes) and any conflict explanations for downstream exporters and APIs. + **ExportState** ``` @@ -252,6 +280,7 @@ public interface IFeedConnector { ``` * Optional ORAS push (OCI layout) for registries. * Offline kit bundles include Trivy DB + JSON tree + export manifest. +* Mirror-ready bundles: when `concelier.trivy.mirror` defines domains, the exporter emits `mirror/index.json` plus per-domain `manifest.json`, `metadata.json`, and `db.tar.gz` files with SHA-256 digests so Concelier mirrors can expose domain-scoped download endpoints. ### 7.3 Hand‑off to Signer/Attestor (optional) @@ -286,6 +315,10 @@ GET /jobs/{id} → job status POST /exports/json { full?:bool, force?:bool, attest?:bool } → { exportId, digest, rekor? } POST /exports/trivy { full?:bool, force?:bool, publish?:bool, attest?:bool } → { exportId, digest, rekor? } GET /exports/{id} → export metadata (kind, digest, createdAt, rekor?) +GET /concelier/exports/index.json → mirror index describing available domains/bundles +GET /concelier/exports/mirror/{domain}/manifest.json +GET /concelier/exports/mirror/{domain}/bundle.json +GET /concelier/exports/mirror/{domain}/bundle.json.jws ``` **Search (operator debugging)** diff --git a/docs/ARCHITECTURE_DEVOPS.md b/docs/ARCHITECTURE_DEVOPS.md index 897da00a..9654b9cf 100644 --- a/docs/ARCHITECTURE_DEVOPS.md +++ b/docs/ARCHITECTURE_DEVOPS.md @@ -86,6 +86,7 @@ At startup, services **self‑advertise** their semver & channel; the UI surface * **Primary**: `registry.stella-ops.org` (OCI v2, supports Referrers API). * **Mirrors**: GHCR (read‑only), regional mirrors for latency. + * Operational runbook: see `docs/ops/concelier-mirror-operations.md` for deployment profiles, CDN guidance, and sync automation. * **Pull by digest only** in Kubernetes/Compose manifests. **Gating policy**: @@ -335,7 +336,8 @@ Prometheus + OTLP; Grafana dashboards ship in the charts. * **Vulnerability response**: - * Concelier red‑flag advisories trigger accelerated **stable** patch rollout; UI/CLI “security patch available” notice. + * Concelier red-flag advisories trigger accelerated **stable** patch rollout; UI/CLI “security patch available” notice. + * 2025-10: Pinned `MongoDB.Driver` **3.5.0** and `SharpCompress` **0.41.0** across services (DEVOPS-SEC-10-301) to eliminate NU1902/NU1903 warnings surfaced during scanner cache/worker test runs; future dependency bumps follow the same central override pattern. * **Backups/DR**: diff --git a/docs/ARCHITECTURE_EXCITITOR.md b/docs/ARCHITECTURE_EXCITITOR.md index 057836aa..c1196b87 100644 --- a/docs/ARCHITECTURE_EXCITITOR.md +++ b/docs/ARCHITECTURE_EXCITITOR.md @@ -482,7 +482,13 @@ Run the ingestion endpoint once after applying migration `20251019-consensus-sig --- -## 17) Appendix — canonical JSON (stable ordering) +## 17) Operational runbooks + +* **Statement backfill** — see `docs/dev/EXCITITOR_STATEMENT_BACKFILL.md` for the CLI workflow, required permissions, observability guidance, and rollback steps. + +--- + +## 18) Appendix — canonical JSON (stable ordering) All exports and consensus entries are serialized via `VexCanonicalJsonSerializer`: diff --git a/docs/ARCHITECTURE_NOTIFY.md b/docs/ARCHITECTURE_NOTIFY.md index 0f86f847..a03d9915 100644 --- a/docs/ARCHITECTURE_NOTIFY.md +++ b/docs/ARCHITECTURE_NOTIFY.md @@ -37,6 +37,8 @@ src/ **Dependencies**: Authority (OpToks; DPoP/mTLS), MongoDB, Redis/NATS (bus), HTTP egress to Slack/Teams/Webhooks, SMTP relay for Email. > **Configuration.** Notify.WebService bootstraps from `notify.yaml` (see `etc/notify.yaml.sample`). Use `storage.driver: mongo` with a production connection string; the optional `memory` driver exists only for tests. Authority settings follow the platform defaults—when running locally without Authority, set `authority.enabled: false` and supply `developmentSigningKey` so JWTs can be validated offline. +> +> `api.rateLimits` exposes token-bucket controls for delivery history queries and test-send previews (`deliveryHistory`, `testSend`). Default values allow generous browsing while preventing accidental bursts; operators can relax/tighten the buckets per deployment. > **Plug-ins.** All channel connectors are packaged under `/plugins/notify`. The ordered load list must start with Slack/Teams before Email/Webhook so chat-first actions are registered deterministically for Offline Kit bundles: > @@ -204,6 +206,8 @@ public interface INotifyConnector { **DeliveryContext** includes **rendered content** and **raw event** for audit. +**Test-send previews.** Plug-ins can optionally implement `INotifyChannelTestProvider` to shape `/channels/{id}/test` responses. Providers receive a sanitised `ChannelTestPreviewContext` (channel, tenant, target, timestamp, trace) and return a `NotifyDeliveryRendered` preview + metadata. When no provider is present, the host falls back to a generic preview so the endpoint always responds. + **Secrets**: `ChannelConfig.secretRef` points to Authority‑managed secret handle or K8s Secret path; workers load at send-time; plug-in manifests (`notify-plugin.json`) declare capabilities and version. --- @@ -294,7 +298,7 @@ Internal tooling can hit `/internal/notify//normalize` to upgrade legacy * **Channels** * `POST /channels` | `GET /channels` | `GET /channels/{id}` | `PATCH /channels/{id}` | `DELETE /channels/{id}` - * `POST /channels/{id}/test` → send sample message (no rule evaluation) + * `POST /channels/{id}/test` → send sample message (no rule evaluation); returns `202 Accepted` with rendered preview + metadata (base keys: `channelType`, `target`, `previewProvider`, `traceId` + connector-specific entries); governed by `api.rateLimits:testSend`. * `GET /channels/{id}/health` → connector self‑check * **Rules** @@ -305,7 +309,7 @@ Internal tooling can hit `/internal/notify//normalize` to upgrade legacy * **Deliveries** * `POST /deliveries` → ingest worker delivery state (idempotent via `deliveryId`). - * `GET /deliveries?since=...&status=...&limit=...` → list (most recent first) + * `GET /deliveries?since=...&status=...&limit=...` → list envelope `{ items, count, continuationToken }` (most recent first); base metadata keys match the test-send response (`channelType`, `target`, `previewProvider`, `traceId`); rate-limited via `api.rateLimits.deliveryHistory`. See `docs/notify/samples/notify-delivery-list-response.sample.json`. * `GET /deliveries/{id}` → detail (redacted body + metadata) * `POST /deliveries/{id}/retry` → force retry (admin, future sprint) diff --git a/docs/ARCHITECTURE_UI.md b/docs/ARCHITECTURE_UI.md index 54880d3f..58c9c373 100644 --- a/docs/ARCHITECTURE_UI.md +++ b/docs/ARCHITECTURE_UI.md @@ -166,6 +166,28 @@ export interface VexConsensus { } ``` +*Upcoming:* `NotifyApi` consumes delivery history using the new paginated envelope returned by `/api/v1/notify/deliveries`. + +```ts +export interface NotifyDeliveryListResponse { + items: NotifyDelivery[]; + count: number; + continuationToken?: string; +} + +export interface NotifyDelivery { + deliveryId: string; + ruleId: string; + actionId: string; + status: 'pending'|'sent'|'failed'|'throttled'|'digested'|'dropped'; + rendered: NotifyDeliveryRendered; + metadata: Record; // includes channelType, target, previewProvider, traceId, and provider-specific entries + createdAt: string; + sentAt?: string; + completedAt?: string; +} +``` + --- ## 6) State, caching & real‑time diff --git a/docs/README.md b/docs/README.md index 04024f20..43c91e5d 100755 --- a/docs/README.md +++ b/docs/README.md @@ -72,16 +72,17 @@ Everything here is open‑source and versioned — when you check out a git ta - **21 – [Install Guide](21_INSTALL_GUIDE.md)** - **22 – [CI/CD Recipes Library](ci/20_CI_RECIPES.md)** - **23 – [FAQ](23_FAQ_MATRIX.md)** -- **24 – [Offline Update Kit Admin Guide](24_OFFLINE_KIT.md)** -- **25 – [Concelier Apple Connector Operations](ops/concelier-apple-operations.md)** -- **26 – [Authority Key Rotation Playbook](ops/authority-key-rotation.md)** -- **27 – [Concelier CCCS Connector Operations](ops/concelier-cccs-operations.md)** -- **28 – [Concelier CISA ICS Connector Operations](ops/concelier-icscisa-operations.md)** -- **29 – [Concelier CERT-Bund Connector Operations](ops/concelier-certbund-operations.md)** -- **30 – [Concelier MSRC Connector – AAD Onboarding](ops/concelier-msrc-operations.md)** - -### Legal & licence -- **31 – [Legal & Quota FAQ](29_LEGAL_FAQ_QUOTA.md)** +- **24 – [Offline Update Kit Admin Guide](24_OFFLINE_KIT.md)** +- **25 – [Mirror Operations Runbook](ops/concelier-mirror-operations.md)** +- **26 – [Concelier Apple Connector Operations](ops/concelier-apple-operations.md)** +- **27 – [Authority Key Rotation Playbook](ops/authority-key-rotation.md)** +- **28 – [Concelier CCCS Connector Operations](ops/concelier-cccs-operations.md)** +- **29 – [Concelier CISA ICS Connector Operations](ops/concelier-icscisa-operations.md)** +- **30 – [Concelier CERT-Bund Connector Operations](ops/concelier-certbund-operations.md)** +- **31 – [Concelier MSRC Connector – AAD Onboarding](ops/concelier-msrc-operations.md)** + +### Legal & licence +- **32 – [Legal & Quota FAQ](29_LEGAL_FAQ_QUOTA.md)** diff --git a/docs/dev/31_AUTHORITY_PLUGIN_DEVELOPER_GUIDE.md b/docs/dev/31_AUTHORITY_PLUGIN_DEVELOPER_GUIDE.md index ac276b6a..634286ed 100644 --- a/docs/dev/31_AUTHORITY_PLUGIN_DEVELOPER_GUIDE.md +++ b/docs/dev/31_AUTHORITY_PLUGIN_DEVELOPER_GUIDE.md @@ -13,7 +13,7 @@ Authority plug-ins extend the **StellaOps Authority** service with custom identi Authority hosts follow a deterministic plug-in lifecycle. The exported diagram (`docs/assets/authority/authority-plugin-lifecycle.svg`) mirrors the steps below; regenerate it from the Mermaid source if you update the flow. 1. **Configuration load** – `AuthorityPluginConfigurationLoader` resolves YAML manifests under `etc/authority.plugins/`. -2. **Assembly discovery** – the shared `PluginHost` scans `PluginBinaries/Authority` for `StellaOps.Authority.Plugin.*.dll` assemblies. +2. **Assembly discovery** – the shared `PluginHost` scans `StellaOps.Authority.PluginBinaries` for `StellaOps.Authority.Plugin.*.dll` assemblies. 3. **Registrar execution** – each assembly is searched for `IAuthorityPluginRegistrar` implementations. Registrars bind options, register services, and optionally queue bootstrap tasks. 4. **Runtime** – the host resolves `IIdentityProviderPlugin` instances, uses capability metadata to decide which OAuth grants to expose, and invokes health checks for readiness endpoints. @@ -177,7 +177,7 @@ _Source:_ `docs/assets/authority/authority-rate-limit-flow.mmd` ## 10. Testing & Tooling - Unit tests: use Mongo2Go (or similar) to exercise credential stores without hitting production infrastructure (`StandardUserCredentialStoreTests` is a template). - Determinism: fix timestamps to UTC and sort outputs consistently; avoid random GUIDs unless stable. -- Smoke tests: launch `dotnet run --project src/StellaOps.Authority/StellaOps.Authority` with your plug-in under `PluginBinaries/Authority` and verify `/ready`. +- Smoke tests: launch `dotnet run --project src/StellaOps.Authority/StellaOps.Authority` with your plug-in under `StellaOps.Authority.PluginBinaries` and verify `/ready`. - Example verification snippet: ```csharp [Fact] @@ -195,7 +195,7 @@ _Source:_ `docs/assets/authority/authority-rate-limit-flow.mmd` ## 11. Packaging & Delivery - Output assembly should follow `StellaOps.Authority.Plugin..dll` so the host’s search pattern picks it up. -- Place the compiled DLL plus dependencies under `PluginBinaries/Authority` for offline deployments; include hashes/signatures in release notes (Security Guild guidance forthcoming). +- Place the compiled DLL plus dependencies under `StellaOps.Authority.PluginBinaries` for offline deployments; include hashes/signatures in release notes (Security Guild guidance forthcoming). - Document any external prerequisites (e.g., CA cert bundle) in your plug-in README. - Update `etc/authority.plugins/.yaml` samples and include deterministic SHA256 hashes for optional bootstrap payloads when distributing Offline Kit artefacts. diff --git a/docs/dev/EXCITITOR_STATEMENT_BACKFILL.md b/docs/dev/EXCITITOR_STATEMENT_BACKFILL.md new file mode 100644 index 00000000..349106c5 --- /dev/null +++ b/docs/dev/EXCITITOR_STATEMENT_BACKFILL.md @@ -0,0 +1,86 @@ +# Excititor Statement Backfill Runbook + +Last updated: 2025-10-19 + +## Overview + +Use this runbook when you need to rebuild the `vex.statements` collection from historical raw documents. Typical scenarios: + +- Upgrading the statement schema (e.g., adding severity/KEV/EPSS signals). +- Recovering from a partial ingest outage where statements were never persisted. +- Seeding a freshly provisioned Excititor deployment from an existing raw archive. + +Backfill operates server-side via the Excititor WebService and reuses the same pipeline that powers the `/excititor/statements` ingestion endpoint. Each raw document is normalized, signed metadata is preserved, and duplicate statements are skipped unless the run is forced. + +## Prerequisites + +1. **Connectivity to Excititor WebService** – the CLI uses the backend URL configured in `stellaops.yml` or the `--backend-url` argument. +2. **Authority credentials** – the CLI honours the existing Authority client configuration; ensure the caller has permission to invoke admin endpoints. +3. **Mongo replica set** (recommended) – causal consistency guarantees rely on majority read/write concerns. Standalone deployment works but skips cross-document transactions. + +## CLI command + +``` +stellaops excititor backfill-statements \ + [--retrieved-since ] \ + [--force] \ + [--batch-size ] \ + [--max-documents ] +``` + +| Option | Description | +| ------ | ----------- | +| `--retrieved-since` | Only process raw documents fetched on or after the specified timestamp (UTC by default). | +| `--force` | Reprocess documents even if matching statements already exist (useful after schema upgrades). | +| `--batch-size` | Number of raw documents pulled per batch (default `100`). | +| `--max-documents` | Optional hard limit on the number of raw documents to evaluate. | + +Example – replay the last 48 hours of Red Hat ingest while keeping existing statements: + +``` +stellaops excititor backfill-statements \ + --retrieved-since "$(date -u -d '48 hours ago' +%Y-%m-%dT%H:%M:%SZ)" +``` + +Example – full replay with forced overwrites, capped at 2,000 documents: + +``` +stellaops excititor backfill-statements --force --max-documents 2000 +``` + +The command returns a summary similar to: + +``` +Backfill completed: evaluated 450, backfilled 180, claims written 320, skipped 270, failures 0. +``` + +## Behaviour + +- Raw documents are streamed in ascending `retrievedAt` order. +- Each document is normalized using the registered VEX normalizers (CSAF, CycloneDX, OpenVEX). +- Statements are appended through the same `IVexClaimStore.AppendAsync` path that powers `/excititor/statements`. +- Duplicate detection compares `Document.Digest`; duplicates are skipped unless `--force` is specified. +- Failures are logged with the offending digest and continue with the next document. + +## Observability + +- CLI logs aggregate counts and the backend logs per-digest warnings or errors. +- Mongo writes carry majority write concern; expect backfill throughput to match ingest baselines (≈5 seconds warm, 30 seconds cold). +- Monitor the `excititor.storage.backfill` log scope for detailed telemetry. + +## Post-run verification + +1. Inspect the `vex.statements` collection for the targeted window (check `InsertedAt`). +2. Re-run the Excititor storage test suite if possible: + ``` + dotnet test src/StellaOps.Excititor.Storage.Mongo.Tests/StellaOps.Excititor.Storage.Mongo.Tests.csproj + ``` +3. Optionally, call `/excititor/statements/{vulnerabilityId}/{productKey}` to confirm the expected statements exist. + +## Rollback + +If a forced run produced incorrect statements, use the standard Mongo rollback procedure: + +1. Identify the `InsertedAt` window for the backfill run. +2. Delete affected records from `vex.statements` (and any downstream exports if applicable). +3. Rerun the backfill command with corrected parameters. diff --git a/docs/dev/authority-dpop-mtls-plan.md b/docs/dev/authority-dpop-mtls-plan.md index 483978a4..f51de44f 100644 --- a/docs/dev/authority-dpop-mtls-plan.md +++ b/docs/dev/authority-dpop-mtls-plan.md @@ -11,6 +11,8 @@ - Operator-facing configuration, auditing, and observability. - Out of scope: PoE enforcement (Signer) and CLI/UI client UX; those teams consume the new capabilities. +> **Status update (2025-10-19):** `ValidateDpopProofHandler`, `AuthorityClientCertificateValidator`, and the supporting storage/audit plumbing now live in `src/StellaOps.Authority`. DPoP proofs populate `cnf.jkt`, mTLS bindings enforce certificate thumbprints via `cnf.x5t#S256`, and token documents persist the sender constraint metadata. In-memory nonce issuance is wired (Redis implementation to follow). Documentation and configuration references were updated (`docs/11_AUTHORITY.md`). Targeted unit/integration tests were added; running the broader test suite is currently blocked by pre-existing `StellaOps.Concelier.Storage.Mongo` build errors. + ## Design Summary - Extract the existing Scanner `DpopProofValidator` stack into a shared `StellaOps.Auth.Security` library used by Authority and resource servers. - Extend Authority configuration (`authority.yaml`) with strongly-typed `senderConstraints.dpop` and `senderConstraints.mtls` sections (map to sample already shown in architecture doc). diff --git a/docs/dev/authority-plugin-di-coordination.md b/docs/dev/authority-plugin-di-coordination.md new file mode 100644 index 00000000..bc41ae1a --- /dev/null +++ b/docs/dev/authority-plugin-di-coordination.md @@ -0,0 +1,52 @@ +# Authority Plug-in Scoped Service Coordination + +> Created: 2025-10-19 — Plugin Platform Guild & Authority Core +> Status: Scheduled (session confirmed for 2025-10-20 15:00–16:00 UTC) + +This document tracks preparation, agenda, and outcomes for the scoped-service workshop required before implementing PLUGIN-DI-08-002. + +## Objectives + +- Inventory Authority plug-in surfaces that need scoped service lifetimes. +- Confirm session/scope handling for identity-provider registrars and background jobs. +- Assign follow-up tasks/actions with owners and due dates. + +## Scheduling Snapshot + +- **Meeting time:** 2025-10-20 15:00–16:00 UTC (10:00–11:00 CDT / 08:00–09:00 PDT). +- **Facilitator:** Plugin Platform Guild — Alicia Rivera. +- **Attendees (confirmed):** Authority Core — Jasmin Patel; Authority Security Guild — Mohan Singh; Plugin Platform — Alicia Rivera, Leah Chen. +- **Optional invitees:** DevOps liaison — Sofia Ortega (accepted). +- **Logistics:** Invites sent via shared calendar on 2025-10-19 15:30 UTC with Teams bridge + offline dial-in. Meeting notes will be captured here. +- **Preparation deadline:** 2025-10-20 12:00 UTC — complete checklist below. + +## Pre-work Checklist + +- Review `ServiceBindingAttribute` contract introduced by PLUGIN-DI-08-001. +- Collect existing Authority plug-in registration code paths to evaluate. +- Audit background jobs that assume singleton lifetimes. +- Identify plug-in health checks/telemetry surfaces impacted by scoped lifetimes. + +### Pre-work References + +- _Add links, file paths, or notes here prior to the session._ + +## Draft Agenda + +1. Context recap (5 min) — why scoped DI is needed; summary of PLUGIN-DI-08-001 changes. +2. Authority plug-in surfaces (15 min) — registrars, background services, telemetry. +3. Session handling strategy (10 min) — scope creation semantics, cancellation propagation. +4. Action items & owners (10 min) — capture code/docs/test tasks with due dates. +5. Risks & follow-ups (5 min) — dependencies, rollout sequencing. + +## Notes + +- _Pending coordination session; populate afterwards._ + +## Action Item Log + +| Item | Owner | Due | Status | Notes | +|------|-------|-----|--------|-------| +| Confirm meeting time | Alicia Rivera | 2025-10-19 15:30 UTC | DONE | Calendar invite sent; all required attendees accepted | +| Compile Authority plug-in DI entry points | Jasmin Patel | 2025-10-20 | IN PROGRESS | Gather current Authority plug-in registrars, background jobs, and helper factories that assume singleton lifetimes; add the list with file paths to **Pre-work References** in this document before 2025-10-20 12:00 UTC. | +| Outline scoped-session pattern for background jobs | Leah Chen | Post-session | BLOCKED | Requires meeting outcomes | diff --git a/docs/dev/normalized_versions_rollout.md b/docs/dev/normalized_versions_rollout.md index dab2d619..2a5b6091 100644 --- a/docs/dev/normalized_versions_rollout.md +++ b/docs/dev/normalized_versions_rollout.md @@ -19,19 +19,19 @@ This dashboard tracks connector readiness for emitting `AffectedPackage.Normaliz | Connector | Owner team | Normalized versions status | Last update | Next action / link | |-----------|------------|---------------------------|-------------|--------------------| | Acsc | BE-Conn-ACSC | ❌ Not started – mapper pending | 2025-10-11 | Design DTOs + mapper with normalized rule array; see `src/StellaOps.Concelier.Connector.Acsc/TASKS.md`. | -| Cccs | BE-Conn-CCCS | ❌ Not started – mapper pending | 2025-10-11 | Add normalized SemVer array in canonical mapper; coordinate fixtures per `TASKS.md`. | -| CertBund | BE-Conn-CERTBUND | ✅ Canonical mapper emitting vendor ranges | 2025-10-14 | Normalized vendor range payloads landed alongside telemetry/docs updates; see `src/StellaOps.Concelier.Connector.CertBund/TASKS.md`. | +| Cccs | BE-Conn-CCCS | ⚠️ Scheduled – helper ready, implementation due 2025-10-21 | 2025-10-19 | Apply Merge-provided trailing-version helper to emit `NormalizedVersions`; update mapper/tests per `src/StellaOps.Concelier.Connector.Cccs/TASKS.md`. | +| CertBund | BE-Conn-CERTBUND | ⚠️ Follow-up – translate `versions` strings to normalized rules | 2025-10-19 | Build `bis`/`alle` translator + fixtures before 2025-10-22 per `src/StellaOps.Concelier.Connector.CertBund/TASKS.md`. | | CertCc | BE-Conn-CERTCC | ⚠️ In progress – fetch pipeline DOING | 2025-10-11 | Implement VINCE mapper with SemVer/NEVRA rules; unblock snapshot regeneration; `src/StellaOps.Concelier.Connector.CertCc/TASKS.md`. | | Kev | BE-Conn-KEV | ✅ Normalized catalog/due-date rules verified | 2025-10-12 | Fixtures reconfirmed via `dotnet test src/StellaOps.Concelier.Connector.Kev.Tests`; `src/StellaOps.Concelier.Connector.Kev/TASKS.md`. | | Cve | BE-Conn-CVE | ✅ Normalized SemVer rules verified | 2025-10-12 | Snapshot parity green (`dotnet test src/StellaOps.Concelier.Connector.Cve.Tests`); `src/StellaOps.Concelier.Connector.Cve/TASKS.md`. | | Ghsa | BE-Conn-GHSA | ⚠️ DOING – normalized rollout task active | 2025-10-11 18:45 UTC | Wire `SemVerRangeRuleBuilder` + refresh fixtures; `src/StellaOps.Concelier.Connector.Ghsa/TASKS.md`. | | Osv | BE-Conn-OSV | ✅ SemVer mapper & parity fixtures verified | 2025-10-12 | GHSA parity regression passing (`dotnet test src/StellaOps.Concelier.Connector.Osv.Tests`); `src/StellaOps.Concelier.Connector.Osv/TASKS.md`. | -| Ics.Cisa | BE-Conn-ICS-CISA | ❌ Not started – mapper TODO | 2025-10-11 | Plan SemVer/firmware scheme selection; `src/StellaOps.Concelier.Connector.Ics.Cisa/TASKS.md`. | -| Kisa | BE-Conn-KISA | ✅ Landed 2025-10-14 (mapper + telemetry) | 2025-10-11 | Hangul-aware mapper emits normalized rules; see `docs/dev/kisa_connector_notes.md` for localisation/metric details. | +| Ics.Cisa | BE-Conn-ICS-CISA | ⚠️ Decision pending – normalize SemVer exacts or escalate scheme | 2025-10-19 | Promote `SemVerPrimitive` outputs into `NormalizedVersions` or file Models ticket by 2025-10-23 (`src/StellaOps.Concelier.Connector.Ics.Cisa/TASKS.md`). | +| Kisa | BE-Conn-KISA | ⚠️ Proposal required – firmware scheme due 2025-10-24 | 2025-10-19 | Draft `kisa.build` (or equivalent) scheme with Models, then emit normalized rules; track in `src/StellaOps.Concelier.Connector.Kisa/TASKS.md`. | | Ru.Bdu | BE-Conn-BDU | ✅ Raw scheme emitted | 2025-10-14 | Mapper now writes `ru-bdu.raw` normalized rules with provenance + telemetry; `src/StellaOps.Concelier.Connector.Ru.Bdu/TASKS.md`. | | Ru.Nkcki | BE-Conn-Nkcki | ❌ Not started – mapper TODO | 2025-10-11 | Similar to BDU; ensure Cyrillic provenance preserved; `src/StellaOps.Concelier.Connector.Ru.Nkcki/TASKS.md`. | | Vndr.Apple | BE-Conn-Apple | ✅ Shipped – emitting normalized arrays | 2025-10-11 | Continue fixture/tooling work; `src/StellaOps.Concelier.Connector.Vndr.Apple/TASKS.md`. | -| Vndr.Cisco | BE-Conn-Cisco | ✅ SemVer + vendor extensions emitted | 2025-10-14 | Connector outputs SemVer primitives with `cisco.productId` notes; see `CiscoMapper` and fixtures for coverage. | +| Vndr.Cisco | BE-Conn-Cisco | ⚠️ Scheduled – normalized rule emission due 2025-10-21 | 2025-10-19 | Use Merge helper to persist `NormalizedVersions` alongside SemVer primitives; see `src/StellaOps.Concelier.Connector.Vndr.Cisco/TASKS.md`. | | Vndr.Msrc | BE-Conn-MSRC | ✅ Map + normalized build rules landed | 2025-10-15 | `MsrcMapper` emits `msrc.build` normalized rules with CVRF references; see `src/StellaOps.Concelier.Connector.Vndr.Msrc/TASKS.md`. | | Nvd | BE-Conn-NVD | ⚠️ Needs follow-up – mapper complete but normalized array MR pending | 2025-10-11 | Align CVE notes + normalized payload flag; `src/StellaOps.Concelier.Connector.Nvd/TASKS.md`. | diff --git a/docs/notify/samples/notify-delivery-list-response.sample.json b/docs/notify/samples/notify-delivery-list-response.sample.json new file mode 100644 index 00000000..af870771 --- /dev/null +++ b/docs/notify/samples/notify-delivery-list-response.sample.json @@ -0,0 +1,46 @@ +{ + "items": [ + { + "deliveryId": "delivery-7f3b6c51", + "tenantId": "tenant-acme", + "ruleId": "rule-critical-slack", + "actionId": "slack-secops", + "eventId": "4f6e9c09-01b4-4c2a-8a57-3d06de182d74", + "kind": "scanner.report.ready", + "status": "Sent", + "statusReason": null, + "rendered": { + "channelType": "Slack", + "format": "Slack", + "target": "#sec-alerts", + "title": "Critical findings detected", + "body": "{\"text\":\"Critical findings detected\",\"blocks\":[{\"type\":\"section\",\"text\":{\"type\":\"mrkdwn\",\"text\":\"*Critical findings detected*\\n1 new critical finding across 2 images.\"}},{\"type\":\"context\",\"elements\":[{\"type\":\"mrkdwn\",\"text\":\"Preview generated 2025-10-19T16:23:41.889Z · Trace `trace-58c212`\"}]}]}", + "summary": "1 new critical finding across 2 images.", + "textBody": "1 new critical finding across 2 images.\nTrace: trace-58c212", + "locale": "en-us", + "bodyHash": "febf9b2a630d862b07f4390edfbf31f5e8b836529f5232c491f4b3f6dba4a4b2", + "attachments": [] + }, + "attempts": [ + { + "timestamp": "2025-10-19T16:23:42.112Z", + "status": "Succeeded", + "statusCode": 200, + "reason": null + } + ], + "metadata": { + "channelType": "slack", + "target": "#sec-alerts", + "previewProvider": "fallback", + "traceId": "trace-58c212", + "slack.channel": "#sec-alerts" + }, + "createdAt": "2025-10-19T16:23:41.889Z", + "sentAt": "2025-10-19T16:23:42.101Z", + "completedAt": "2025-10-19T16:23:42.112Z" + } + ], + "count": 1, + "continuationToken": "2025-10-19T16:23:41.889Z|tenant-acme:delivery-7f3b6c51" +} diff --git a/docs/ops/concelier-mirror-operations.md b/docs/ops/concelier-mirror-operations.md new file mode 100644 index 00000000..e844054d --- /dev/null +++ b/docs/ops/concelier-mirror-operations.md @@ -0,0 +1,196 @@ +# Concelier & Excititor Mirror Operations + +This runbook describes how Stella Ops operates the managed mirrors under `*.stella-ops.org`. +It covers Docker Compose and Helm deployment overlays, secret handling for multi-tenant +authn, CDN fronting, and the recurring sync pipeline that keeps mirror bundles current. + +## 1. Prerequisites + +- **Authority access** – client credentials (`client_id` + secret) authorised for + `concelier.mirror.read` and `excititor.mirror.read` scopes. Secrets live outside git. +- **Signed TLS certificates** – wildcard or per-domain (`mirror-primary`, `mirror-community`). + Store them under `deploy/compose/mirror-gateway/tls/` or in Kubernetes secrets. +- **Mirror gateway credentials** – Basic Auth htpasswd files per domain. Generate with + `htpasswd -B`. Operators distribute credentials to downstream consumers. +- **Export artifact source** – read access to the canonical S3 buckets (or rsync share) + that hold `concelier` JSON bundles and `excititor` VEX exports. +- **Persistent volumes** – storage for Concelier job metadata and mirror export trees. + For Helm, provision PVCs (`concelier-mirror-jobs`, `concelier-mirror-exports`, + `excititor-mirror-exports`, `mirror-mongo-data`, `mirror-minio-data`) before rollout. + +## 2. Secret & certificate layout + +### Docker Compose (`deploy/compose/docker-compose.mirror.yaml`) + +- `deploy/compose/env/mirror.env.example` – copy to `.env` and adjust quotas or domain IDs. +- `deploy/compose/mirror-secrets/` – mount read-only into `/run/secrets`. Place: + - `concelier-authority-client` – Authority client secret. + - `excititor-authority-client` (optional) – reserve for future authn. +- `deploy/compose/mirror-gateway/tls/` – PEM-encoded cert/key pairs: + - `mirror-primary.crt`, `mirror-primary.key` + - `mirror-community.crt`, `mirror-community.key` +- `deploy/compose/mirror-gateway/secrets/` – htpasswd files: + - `mirror-primary.htpasswd` + - `mirror-community.htpasswd` + +### Helm (`deploy/helm/stellaops/values-mirror.yaml`) + +Create secrets in the target namespace: + +```bash +kubectl create secret generic concelier-mirror-auth \ + --from-file=concelier-authority-client=concelier-authority-client + +kubectl create secret generic excititor-mirror-auth \ + --from-file=excititor-authority-client=excititor-authority-client + +kubectl create secret tls mirror-gateway-tls \ + --cert=mirror-primary.crt --key=mirror-primary.key + +kubectl create secret generic mirror-gateway-htpasswd \ + --from-file=mirror-primary.htpasswd --from-file=mirror-community.htpasswd +``` + +> Keep Basic Auth lists short-lived (rotate quarterly) and document credential recipients. + +## 3. Deployment + +### 3.1 Docker Compose (edge mirrors, lab validation) + +1. `cp deploy/compose/env/mirror.env.example deploy/compose/env/mirror.env` +2. Populate secrets/tls directories as described above. +3. Sync mirror bundles (see §4) into `deploy/compose/mirror-data/…` and ensure they are mounted + on the host path backing the `concelier-exports` and `excititor-exports` volumes. +4. Run the profile validator: `deploy/tools/validate-profiles.sh`. +5. Launch: `docker compose --env-file env/mirror.env -f docker-compose.mirror.yaml up -d`. + +### 3.2 Helm (production mirrors) + +1. Provision PVCs sized for mirror bundles (baseline: 20 GiB per domain). +2. Create secrets/tls config maps (§2). +3. `helm upgrade --install mirror deploy/helm/stellaops -f deploy/helm/stellaops/values-mirror.yaml`. +4. Annotate the `stellaops-mirror-gateway` service with ingress/LoadBalancer metadata required by + your CDN (e.g., AWS load balancer scheme internal + NLB idle timeout). + +## 4. Artifact sync workflow + +Mirrors never generate exports—they ingest signed bundles produced by the Concelier and Excititor +export jobs. Recommended sync pattern: + +### 4.1 Compose host (systemd timer) + +`/usr/local/bin/mirror-sync.sh`: + +```bash +#!/usr/bin/env bash +set -euo pipefail +export AWS_ACCESS_KEY_ID=… +export AWS_SECRET_ACCESS_KEY=… + +aws s3 sync s3://mirror-stellaops/concelier/latest \ + /opt/stellaops/mirror-data/concelier --delete --size-only + +aws s3 sync s3://mirror-stellaops/excititor/latest \ + /opt/stellaops/mirror-data/excititor --delete --size-only +``` + +Schedule with a systemd timer every 5 minutes. The Compose volumes mount `/opt/stellaops/mirror-data/*` +into the containers read-only, matching `CONCELIER__MIRROR__EXPORTROOT=/exports/json` and +`EXCITITOR__ARTIFACTS__FILESYSTEM__ROOT=/exports`. + +### 4.2 Kubernetes (CronJob) + +Create a CronJob running the AWS CLI (or rclone) in the same namespace, writing into the PVCs: + +```yaml +apiVersion: batch/v1 +kind: CronJob +metadata: + name: mirror-sync +spec: + schedule: "*/5 * * * *" + jobTemplate: + spec: + template: + spec: + containers: + - name: sync + image: public.ecr.aws/aws-cli/aws-cli@sha256:5df5f52c29f5e3ba46d0ad9e0e3afc98701c4a0f879400b4c5f80d943b5fadea + command: + - /bin/sh + - -c + - > + aws s3 sync s3://mirror-stellaops/concelier/latest /exports/concelier --delete --size-only && + aws s3 sync s3://mirror-stellaops/excititor/latest /exports/excititor --delete --size-only + volumeMounts: + - name: concelier-exports + mountPath: /exports/concelier + - name: excititor-exports + mountPath: /exports/excititor + envFrom: + - secretRef: + name: mirror-sync-aws + restartPolicy: OnFailure + volumes: + - name: concelier-exports + persistentVolumeClaim: + claimName: concelier-mirror-exports + - name: excititor-exports + persistentVolumeClaim: + claimName: excititor-mirror-exports +``` + +## 5. CDN integration + +1. Point the CDN origin at the mirror gateway (Compose host or Kubernetes LoadBalancer). +2. Honour the response headers emitted by the gateway and Concelier/Excititor: + `Cache-Control: public, max-age=300, immutable` for mirror payloads. +3. Configure origin shields in the CDN to prevent cache stampedes. Recommended TTLs: + - Index (`/concelier/exports/index.json`, `/excititor/mirror/*/index`) → 60 s. + - Bundle/manifest payloads → 300 s. +4. Forward the `Authorization` header—Basic Auth terminates at the gateway. +5. Enforce per-domain rate limits at the CDN (matching gateway budgets) and enable logging + to SIEM for anomaly detection. + +## 6. Smoke tests + +After each deployment or sync cycle: + +```bash +# Index with Basic Auth +curl -u $PRIMARY_CREDS https://mirror-primary.stella-ops.org/concelier/exports/index.json | jq 'keys' + +# Mirror manifest signature +curl -u $PRIMARY_CREDS -I https://mirror-primary.stella-ops.org/concelier/exports/mirror/primary/manifest.json + +# Excititor consensus bundle metadata +curl -u $COMMUNITY_CREDS https://mirror-community.stella-ops.org/excititor/mirror/community/index \ + | jq '.exports[].exportKey' + +# Signed bundle + detached JWS (spot check digests) +curl -u $PRIMARY_CREDS https://mirror-primary.stella-ops.org/concelier/exports/mirror/primary/bundle.json.jws \ + -o bundle.json.jws +cosign verify-blob --signature bundle.json.jws --key mirror-key.pub bundle.json +``` + +Watch the gateway metrics (`nginx_vts` or access logs) for cache hits. In Kubernetes, `kubectl logs deploy/stellaops-mirror-gateway` +should show `X-Cache-Status: HIT/MISS`. + +## 7. Maintenance & rotation + +- **Bundle freshness** – alert if sync job lag exceeds 15 minutes or if `concelier` logs + `Mirror export root is not configured`. +- **Secret rotation** – change Authority client secrets and Basic Auth credentials quarterly. + Update the mounted secrets and restart deployments (`docker compose restart concelier` or + `kubectl rollout restart deploy/stellaops-concelier`). +- **TLS renewal** – reissue certificates, place new files, and reload gateway (`docker compose exec mirror-gateway nginx -s reload`). +- **Quota tuning** – adjust per-domain `MAXDOWNLOADREQUESTSPERHOUR` in `.env` or values file. + Align CDN rate limits and inform downstreams. + +## 8. References + +- Deployment profiles: `deploy/compose/docker-compose.mirror.yaml`, + `deploy/helm/stellaops/values-mirror.yaml` +- Mirror architecture dossiers: `docs/ARCHITECTURE_CONCELIER.md`, + `docs/ARCHITECTURE_EXCITITOR_MIRRORS.md` +- Export bundling: `docs/ARCHITECTURE_DEVOPS.md` §3, `docs/ARCHITECTURE_EXCITITOR.md` §7 diff --git a/etc/authority.yaml.sample b/etc/authority.yaml.sample index 002a189f..141d3315 100644 --- a/etc/authority.yaml.sample +++ b/etc/authority.yaml.sample @@ -52,7 +52,7 @@ bootstrap: # against the application content root, enabling air-gapped deployments # that package plug-ins alongside binaries. pluginDirectories: - - "../PluginBinaries/Authority" + - "../StellaOps.Authority.PluginBinaries" # "/var/lib/stellaops/authority/plugins" # Plug-in manifests live in descriptors below; per-plugin settings are stored @@ -73,7 +73,7 @@ plugins: metadata: defaultRole: "operators" # Example for an external identity provider plugin. Leave disabled unless - # the plug-in package exists under PluginBinaries/Authority. + # the plug-in package exists under StellaOps.Authority.PluginBinaries. ldap: type: "ldap" assemblyName: "StellaOps.Authority.Plugin.Ldap" diff --git a/etc/concelier.yaml.sample b/etc/concelier.yaml.sample index 98b3e9d1..9eae8a2e 100644 --- a/etc/concelier.yaml.sample +++ b/etc/concelier.yaml.sample @@ -15,7 +15,7 @@ storage: plugins: # Concelier resolves plug-ins relative to the content root; override as needed. baseDirectory: ".." - directory: "PluginBinaries" + directory: "StellaOps.Concelier.PluginBinaries" searchPatterns: - "StellaOps.Concelier.Plugin.*.dll" @@ -74,6 +74,22 @@ authority: - "127.0.0.1/32" - "::1/128" +mirror: + enabled: false + # Directory containing JSON exporter outputs (absolute or relative to content root). + exportRoot: "exports/json" + # Optional explicit export identifier; defaults to `latest` symlink or most recent export. + activeExportId: "" + latestDirectoryName: "latest" + mirrorDirectoryName: "mirror" + requireAuthentication: false + maxIndexRequestsPerHour: 600 + domains: + - id: "primary" + displayName: "Primary Mirror" + requireAuthentication: false + maxDownloadRequestsPerHour: 1200 + sources: ghsa: apiToken: "${GITHUB_PAT}" diff --git a/etc/notify.yaml.sample b/etc/notify.yaml.sample index c8615ce4..2ab9e14a 100644 --- a/etc/notify.yaml.sample +++ b/etc/notify.yaml.sample @@ -24,6 +24,19 @@ api: basePath: "/api/v1/notify" internalBasePath: "/internal/notify" tenantHeader: "X-StellaOps-Tenant" + rateLimits: + deliveryHistory: + enabled: true + tokenLimit: 60 + tokensPerPeriod: 30 + replenishmentPeriodSeconds: 60 + queueLimit: 20 + testSend: + enabled: true + tokenLimit: 5 + tokensPerPeriod: 5 + replenishmentPeriodSeconds: 60 + queueLimit: 2 plugins: baseDirectory: "../" diff --git a/ops/devops/TASKS.md b/ops/devops/TASKS.md index 9ca62b13..0320e1a7 100644 --- a/ops/devops/TASKS.md +++ b/ops/devops/TASKS.md @@ -9,5 +9,5 @@ | DEVOPS-PERF-10-002 | TODO | DevOps Guild | BENCH-SCANNER-10-002 | Publish analyzer bench metrics to Grafana/perf workbook and alarm on ≥20 % regressions. | CI exports JSON for dashboards; Grafana panel wired; Ops on-call doc updated with alert hook. | | DEVOPS-REL-14-001 | TODO | DevOps Guild | SIGNER-API-11-101, ATTESTOR-API-11-201 | Deterministic build/release pipeline with SBOM/provenance, signing, manifest generation. | CI pipeline produces signed images + SBOM/attestations, manifests published with verified hashes, docs updated. | | DEVOPS-REL-17-002 | TODO | DevOps Guild | DEVOPS-REL-14-001, SCANNER-EMIT-17-701 | Persist stripped-debug artifacts organised by GNU build-id and bundle them into release/offline kits with checksum manifests. | CI job writes `.debug` files under `artifacts/debug/.build-id/`, manifest + checksums published, offline kit includes cache, smoke job proves symbol lookup via build-id. | -| DEVOPS-MIRROR-08-001 | TODO | DevOps Guild | DEVOPS-REL-14-001 | Stand up managed mirror profiles for `*.stella-ops.org` (Concelier/Excititor), including Helm/Compose overlays, multi-tenant secrets, CDN caching, and sync documentation. | Infra overlays committed, CI smoke deploy hits mirror endpoints, runbooks published for downstream sync and quota management. | -| DEVOPS-SEC-10-301 | DOING | DevOps Guild | — | Address NU1902/NU1903 advisories for `MongoDB.Driver` 2.12.0 and `SharpCompress` 0.23.0 surfaced during scanner cache and worker test runs. | Dependencies bumped to patched releases, audit logs free of NU1902/NU1903 warnings, regression tests green, change log documents upgrade guidance. | +| DEVOPS-MIRROR-08-001 | DONE (2025-10-19) | DevOps Guild | DEVOPS-REL-14-001 | Stand up managed mirror profiles for `*.stella-ops.org` (Concelier/Excititor), including Helm/Compose overlays, multi-tenant secrets, CDN caching, and sync documentation. | Infra overlays committed, CI smoke deploy hits mirror endpoints, runbooks published for downstream sync and quota management. | +| DEVOPS-SEC-10-301 | DOING (2025-10-19) | DevOps Guild | Wave 0A complete | Address NU1902/NU1903 advisories for `MongoDB.Driver` 2.12.0 and `SharpCompress` 0.23.0 surfaced during scanner cache and worker test runs. | Dependencies bumped to patched releases, audit logs free of NU1902/NU1903 warnings, regression tests green, change log documents upgrade guidance. | diff --git a/src/Directory.Build.props b/src/Directory.Build.props index 1f61b4a9..64b50dfa 100644 --- a/src/Directory.Build.props +++ b/src/Directory.Build.props @@ -1,9 +1,9 @@ - $(SolutionDir)PluginBinaries - $(MSBuildThisFileDirectory)PluginBinaries - $(SolutionDir)PluginBinaries\Authority - $(MSBuildThisFileDirectory)PluginBinaries\Authority + $(SolutionDir)StellaOps.Concelier.PluginBinaries + $(MSBuildThisFileDirectory)StellaOps.Concelier.PluginBinaries + $(SolutionDir)StellaOps.Authority.PluginBinaries + $(MSBuildThisFileDirectory)StellaOps.Authority.PluginBinaries true true true @@ -20,12 +20,17 @@ false runtime - - - + + + + + + + + - - + + diff --git a/src/StellaOps.Attestor/StellaOps.Attestor.Core/Options/AttestorOptions.cs b/src/StellaOps.Attestor/StellaOps.Attestor.Core/Options/AttestorOptions.cs index 9001b00f..1d212c27 100644 --- a/src/StellaOps.Attestor/StellaOps.Attestor.Core/Options/AttestorOptions.cs +++ b/src/StellaOps.Attestor/StellaOps.Attestor.Core/Options/AttestorOptions.cs @@ -37,6 +37,10 @@ public sealed class AttestorOptions public bool RequireClientCertificate { get; set; } = true; public string? CaBundle { get; set; } + + public IList AllowedSubjects { get; set; } = new List(); + + public IList AllowedThumbprints { get; set; } = new List(); } public sealed class AuthorityOptions diff --git a/src/StellaOps.Attestor/StellaOps.Attestor.Core/Storage/AttestorEntry.cs b/src/StellaOps.Attestor/StellaOps.Attestor.Core/Storage/AttestorEntry.cs index 94672075..e90abdcc 100644 --- a/src/StellaOps.Attestor/StellaOps.Attestor.Core/Storage/AttestorEntry.cs +++ b/src/StellaOps.Attestor/StellaOps.Attestor.Core/Storage/AttestorEntry.cs @@ -26,6 +26,8 @@ public sealed class AttestorEntry public SignerIdentityDescriptor SignerIdentity { get; init; } = new(); + public LogReplicaDescriptor? Mirror { get; init; } + public sealed class ArtifactDescriptor { public string Sha256 { get; init; } = string.Empty; @@ -64,6 +66,8 @@ public sealed class AttestorEntry public sealed class LogDescriptor { + public string Backend { get; init; } = "primary"; + public string Url { get; init; } = string.Empty; public string? LogId { get; init; } @@ -79,4 +83,23 @@ public sealed class AttestorEntry public string? KeyId { get; init; } } + + public sealed class LogReplicaDescriptor + { + public string Backend { get; init; } = string.Empty; + + public string Url { get; init; } = string.Empty; + + public string? Uuid { get; init; } + + public long? Index { get; init; } + + public string Status { get; init; } = "pending"; + + public ProofDescriptor? Proof { get; init; } + + public string? LogId { get; init; } + + public string? Error { get; init; } + } } diff --git a/src/StellaOps.Attestor/StellaOps.Attestor.Core/Submission/AttestorSubmissionResult.cs b/src/StellaOps.Attestor/StellaOps.Attestor.Core/Submission/AttestorSubmissionResult.cs index 85399b83..b34ae890 100644 --- a/src/StellaOps.Attestor/StellaOps.Attestor.Core/Submission/AttestorSubmissionResult.cs +++ b/src/StellaOps.Attestor/StellaOps.Attestor.Core/Submission/AttestorSubmissionResult.cs @@ -24,6 +24,9 @@ public sealed class AttestorSubmissionResult [JsonPropertyName("status")] public string Status { get; set; } = "pending"; + [JsonPropertyName("mirror")] + public MirrorLog? Mirror { get; set; } + public sealed class RekorProof { [JsonPropertyName("checkpoint")] @@ -56,4 +59,25 @@ public sealed class AttestorSubmissionResult [JsonPropertyName("path")] public IReadOnlyList Path { get; init; } = Array.Empty(); } + + public sealed class MirrorLog + { + [JsonPropertyName("uuid")] + public string? Uuid { get; set; } + + [JsonPropertyName("index")] + public long? Index { get; set; } + + [JsonPropertyName("logURL")] + public string? LogUrl { get; set; } + + [JsonPropertyName("status")] + public string Status { get; set; } = "pending"; + + [JsonPropertyName("proof")] + public RekorProof? Proof { get; set; } + + [JsonPropertyName("error")] + public string? Error { get; set; } + } } diff --git a/src/StellaOps.Attestor/StellaOps.Attestor.Core/Submission/AttestorSubmissionValidator.cs b/src/StellaOps.Attestor/StellaOps.Attestor.Core/Submission/AttestorSubmissionValidator.cs index 00798027..c8788494 100644 --- a/src/StellaOps.Attestor/StellaOps.Attestor.Core/Submission/AttestorSubmissionValidator.cs +++ b/src/StellaOps.Attestor/StellaOps.Attestor.Core/Submission/AttestorSubmissionValidator.cs @@ -12,10 +12,14 @@ public sealed class AttestorSubmissionValidator private static readonly string[] AllowedKinds = ["sbom", "report", "vex-export"]; private readonly IDsseCanonicalizer _canonicalizer; + private readonly HashSet _allowedModes; - public AttestorSubmissionValidator(IDsseCanonicalizer canonicalizer) + public AttestorSubmissionValidator(IDsseCanonicalizer canonicalizer, IEnumerable? allowedModes = null) { _canonicalizer = canonicalizer ?? throw new ArgumentNullException(nameof(canonicalizer)); + _allowedModes = allowedModes is null + ? new HashSet(StringComparer.OrdinalIgnoreCase) + : new HashSet(allowedModes, StringComparer.OrdinalIgnoreCase); } public async Task ValidateAsync(AttestorSubmissionRequest request, CancellationToken cancellationToken = default) @@ -47,6 +51,11 @@ public sealed class AttestorSubmissionValidator throw new AttestorValidationException("signature_missing", "At least one DSSE signature is required."); } + if (_allowedModes.Count > 0 && !string.IsNullOrWhiteSpace(request.Bundle.Mode) && !_allowedModes.Contains(request.Bundle.Mode)) + { + throw new AttestorValidationException("mode_not_allowed", $"Submission mode '{request.Bundle.Mode}' is not permitted."); + } + if (request.Meta is null) { throw new AttestorValidationException("meta_missing", "Submission metadata is required."); diff --git a/src/StellaOps.Attestor/StellaOps.Attestor.Infrastructure/ServiceCollectionExtensions.cs b/src/StellaOps.Attestor/StellaOps.Attestor.Infrastructure/ServiceCollectionExtensions.cs index cc845da9..cd8118e3 100644 --- a/src/StellaOps.Attestor/StellaOps.Attestor.Infrastructure/ServiceCollectionExtensions.cs +++ b/src/StellaOps.Attestor/StellaOps.Attestor.Infrastructure/ServiceCollectionExtensions.cs @@ -24,7 +24,12 @@ public static class ServiceCollectionExtensions public static IServiceCollection AddAttestorInfrastructure(this IServiceCollection services) { services.AddSingleton(); - services.AddSingleton(); + services.AddSingleton(sp => + { + var canonicalizer = sp.GetRequiredService(); + var options = sp.GetRequiredService>().Value; + return new AttestorSubmissionValidator(canonicalizer, options.Security.SignerIdentity.Mode); + }); services.AddSingleton(); services.AddSingleton(); services.AddSingleton(); diff --git a/src/StellaOps.Attestor/StellaOps.Attestor.Infrastructure/Storage/MongoAttestorEntryRepository.cs b/src/StellaOps.Attestor/StellaOps.Attestor.Infrastructure/Storage/MongoAttestorEntryRepository.cs index d5cf7127..179a258a 100644 --- a/src/StellaOps.Attestor/StellaOps.Attestor.Infrastructure/Storage/MongoAttestorEntryRepository.cs +++ b/src/StellaOps.Attestor/StellaOps.Attestor.Infrastructure/Storage/MongoAttestorEntryRepository.cs @@ -76,6 +76,9 @@ internal sealed class MongoAttestorEntryRepository : IAttestorEntryRepository [BsonElement("signerIdentity")] public SignerIdentityDocument SignerIdentity { get; set; } = new(); + [BsonElement("mirror")] + public MirrorDocument? Mirror { get; set; } + public static AttestorEntryDocument FromDomain(AttestorEntry entry) { return new AttestorEntryDocument @@ -109,6 +112,7 @@ internal sealed class MongoAttestorEntryRepository : IAttestorEntryRepository }, Log = new LogDocument { + Backend = entry.Log.Backend, Url = entry.Log.Url, LogId = entry.Log.LogId }, @@ -120,7 +124,8 @@ internal sealed class MongoAttestorEntryRepository : IAttestorEntryRepository Issuer = entry.SignerIdentity.Issuer, SubjectAlternativeName = entry.SignerIdentity.SubjectAlternativeName, KeyId = entry.SignerIdentity.KeyId - } + }, + Mirror = entry.Mirror is null ? null : MirrorDocument.FromDomain(entry.Mirror) }; } @@ -155,6 +160,7 @@ internal sealed class MongoAttestorEntryRepository : IAttestorEntryRepository }, Log = new AttestorEntry.LogDescriptor { + Backend = Log.Backend, Url = Log.Url, LogId = Log.LogId }, @@ -166,7 +172,8 @@ internal sealed class MongoAttestorEntryRepository : IAttestorEntryRepository Issuer = SignerIdentity.Issuer, SubjectAlternativeName = SignerIdentity.SubjectAlternativeName, KeyId = SignerIdentity.KeyId - } + }, + Mirror = Mirror?.ToDomain() }; } @@ -220,6 +227,9 @@ internal sealed class MongoAttestorEntryRepository : IAttestorEntryRepository internal sealed class LogDocument { + [BsonElement("backend")] + public string Backend { get; set; } = "primary"; + [BsonElement("url")] public string Url { get; set; } = string.Empty; @@ -241,5 +251,92 @@ internal sealed class MongoAttestorEntryRepository : IAttestorEntryRepository [BsonElement("kid")] public string? KeyId { get; set; } } + + internal sealed class MirrorDocument + { + [BsonElement("backend")] + public string Backend { get; set; } = string.Empty; + + [BsonElement("url")] + public string Url { get; set; } = string.Empty; + + [BsonElement("uuid")] + public string? Uuid { get; set; } + + [BsonElement("index")] + public long? Index { get; set; } + + [BsonElement("status")] + public string Status { get; set; } = "pending"; + + [BsonElement("proof")] + public ProofDocument? Proof { get; set; } + + [BsonElement("logId")] + public string? LogId { get; set; } + + [BsonElement("error")] + public string? Error { get; set; } + + public static MirrorDocument FromDomain(AttestorEntry.LogReplicaDescriptor mirror) + { + return new MirrorDocument + { + Backend = mirror.Backend, + Url = mirror.Url, + Uuid = mirror.Uuid, + Index = mirror.Index, + Status = mirror.Status, + Proof = mirror.Proof is null ? null : new ProofDocument + { + Checkpoint = mirror.Proof.Checkpoint is null ? null : new CheckpointDocument + { + Origin = mirror.Proof.Checkpoint.Origin, + Size = mirror.Proof.Checkpoint.Size, + RootHash = mirror.Proof.Checkpoint.RootHash, + Timestamp = mirror.Proof.Checkpoint.Timestamp is null + ? null + : BsonDateTime.Create(mirror.Proof.Checkpoint.Timestamp.Value) + }, + Inclusion = mirror.Proof.Inclusion is null ? null : new InclusionDocument + { + LeafHash = mirror.Proof.Inclusion.LeafHash, + Path = mirror.Proof.Inclusion.Path + } + }, + LogId = mirror.LogId, + Error = mirror.Error + }; + } + + public AttestorEntry.LogReplicaDescriptor ToDomain() + { + return new AttestorEntry.LogReplicaDescriptor + { + Backend = Backend, + Url = Url, + Uuid = Uuid, + Index = Index, + Status = Status, + Proof = Proof is null ? null : new AttestorEntry.ProofDescriptor + { + Checkpoint = Proof.Checkpoint is null ? null : new AttestorEntry.CheckpointDescriptor + { + Origin = Proof.Checkpoint.Origin, + Size = Proof.Checkpoint.Size, + RootHash = Proof.Checkpoint.RootHash, + Timestamp = Proof.Checkpoint.Timestamp?.ToUniversalTime() + }, + Inclusion = Proof.Inclusion is null ? null : new AttestorEntry.InclusionDescriptor + { + LeafHash = Proof.Inclusion.LeafHash, + Path = Proof.Inclusion.Path + } + }, + LogId = LogId, + Error = Error + }; + } + } } } diff --git a/src/StellaOps.Attestor/StellaOps.Attestor.Infrastructure/Submission/AttestorSubmissionService.cs b/src/StellaOps.Attestor/StellaOps.Attestor.Infrastructure/Submission/AttestorSubmissionService.cs index ee36ef73..5cb997ca 100644 --- a/src/StellaOps.Attestor/StellaOps.Attestor.Infrastructure/Submission/AttestorSubmissionService.cs +++ b/src/StellaOps.Attestor/StellaOps.Attestor.Infrastructure/Submission/AttestorSubmissionService.cs @@ -1,5 +1,6 @@ using System; using System.Collections.Generic; +using System.Diagnostics; using System.Text.Json; using System.Threading; using System.Threading.Tasks; @@ -58,137 +59,136 @@ internal sealed class AttestorSubmissionService : IAttestorSubmissionService SubmissionContext context, CancellationToken cancellationToken = default) { - var start = System.Diagnostics.Stopwatch.GetTimestamp(); + ArgumentNullException.ThrowIfNull(request); + ArgumentNullException.ThrowIfNull(context); var validation = await _validator.ValidateAsync(request, cancellationToken).ConfigureAwait(false); - var canonicalBundle = validation.CanonicalBundle; - var dedupeUuid = await _dedupeStore.TryGetExistingAsync(request.Meta.BundleSha256, cancellationToken).ConfigureAwait(false); - if (!string.IsNullOrEmpty(dedupeUuid)) + var preference = NormalizeLogPreference(request.Meta.LogPreference); + var requiresPrimary = preference is "primary" or "both"; + var requiresMirror = preference is "mirror" or "both"; + + if (!requiresPrimary && !requiresMirror) + { + requiresPrimary = true; + } + + if (requiresMirror && !_options.Rekor.Mirror.Enabled) + { + throw new AttestorValidationException("mirror_disabled", "Mirror log requested but not configured."); + } + + var existing = await TryGetExistingEntryAsync(request.Meta.BundleSha256, cancellationToken).ConfigureAwait(false); + if (existing is not null) { - _logger.LogInformation("Dedupe hit for bundle {BundleSha256} -> {RekorUuid}", request.Meta.BundleSha256, dedupeUuid); _metrics.DedupeHitsTotal.Add(1, new KeyValuePair("result", "hit")); - var existing = await _repository.GetByUuidAsync(dedupeUuid, cancellationToken).ConfigureAwait(false) - ?? await _repository.GetByBundleShaAsync(request.Meta.BundleSha256, cancellationToken).ConfigureAwait(false); - - if (existing is not null) - { - _metrics.SubmitTotal.Add(1, - new KeyValuePair("result", "dedupe"), - new KeyValuePair("backend", "cache")); - return ToResult(existing); - } - } - else - { - _metrics.DedupeHitsTotal.Add(1, new KeyValuePair("result", "miss")); + var updated = await EnsureBackendsAsync(existing, request, context, requiresPrimary, requiresMirror, cancellationToken).ConfigureAwait(false); + return ToResult(updated); } - var primaryBackend = BuildBackend("primary", _options.Rekor.Primary); - RekorSubmissionResponse submissionResponse; - try + _metrics.DedupeHitsTotal.Add(1, new KeyValuePair("result", "miss")); + + SubmissionOutcome? canonicalOutcome = null; + SubmissionOutcome? mirrorOutcome = null; + + if (requiresPrimary) { - submissionResponse = await _rekorClient.SubmitAsync(request, primaryBackend, cancellationToken).ConfigureAwait(false); - } - catch (Exception ex) - { - _metrics.ErrorTotal.Add(1, new KeyValuePair("type", "submit")); - _logger.LogError(ex, "Failed to submit bundle {BundleSha} to Rekor backend {Backend}", request.Meta.BundleSha256, primaryBackend.Name); - throw; + canonicalOutcome = await SubmitToBackendAsync(request, "primary", _options.Rekor.Primary, cancellationToken).ConfigureAwait(false); } - var proof = submissionResponse.Proof; - if (proof is null && string.Equals(submissionResponse.Status, "included", StringComparison.OrdinalIgnoreCase)) + if (requiresMirror) { try { - proof = await _rekorClient.GetProofAsync(submissionResponse.Uuid, primaryBackend, cancellationToken).ConfigureAwait(false); - _metrics.ProofFetchTotal.Add(1, - new KeyValuePair("result", proof is null ? "missing" : "ok")); + var mirror = await SubmitToBackendAsync(request, "mirror", _options.Rekor.Mirror, cancellationToken).ConfigureAwait(false); + if (canonicalOutcome is null) + { + canonicalOutcome = mirror; + } + else + { + mirrorOutcome = mirror; + } } catch (Exception ex) { - _metrics.ErrorTotal.Add(1, new KeyValuePair("type", "proof_fetch")); - _logger.LogWarning(ex, "Proof fetch failed for {Uuid} on backend {Backend}", submissionResponse.Uuid, primaryBackend.Name); + if (canonicalOutcome is null) + { + throw; + } + + _metrics.ErrorTotal.Add(1, new KeyValuePair("type", "submit_mirror")); + _logger.LogWarning(ex, "Mirror submission failed for bundle {BundleSha}", request.Meta.BundleSha256); + mirrorOutcome = SubmissionOutcome.Failure("mirror", _options.Rekor.Mirror.Url, ex, TimeSpan.Zero); + RecordSubmissionMetrics(mirrorOutcome); } } - var entry = CreateEntry(request, submissionResponse, proof, context, canonicalBundle); + if (canonicalOutcome is null) + { + throw new InvalidOperationException("No Rekor submission outcome was produced."); + } + + var entry = CreateEntry(request, context, canonicalOutcome, mirrorOutcome); await _repository.SaveAsync(entry, cancellationToken).ConfigureAwait(false); await _dedupeStore.SetAsync(request.Meta.BundleSha256, entry.RekorUuid, DedupeTtl, cancellationToken).ConfigureAwait(false); if (request.Meta.Archive) { - var archiveBundle = new AttestorArchiveBundle - { - RekorUuid = entry.RekorUuid, - ArtifactSha256 = entry.Artifact.Sha256, - BundleSha256 = entry.BundleSha256, - CanonicalBundleJson = canonicalBundle, - ProofJson = proof is null ? Array.Empty() : JsonSerializer.SerializeToUtf8Bytes(proof, JsonSerializerOptions.Default), - Metadata = new Dictionary - { - ["logUrl"] = entry.Log.Url, - ["status"] = entry.Status - } - }; - - try - { - await _archiveStore.ArchiveBundleAsync(archiveBundle, cancellationToken).ConfigureAwait(false); - } - catch (Exception ex) - { - _logger.LogWarning(ex, "Failed to archive bundle {BundleSha}", entry.BundleSha256); - _metrics.ErrorTotal.Add(1, new KeyValuePair("type", "archive")); - } + await ArchiveAsync(entry, canonicalBundle, canonicalOutcome.Proof, cancellationToken).ConfigureAwait(false); } - var elapsed = System.Diagnostics.Stopwatch.GetElapsedTime(start, System.Diagnostics.Stopwatch.GetTimestamp()); - _metrics.SubmitTotal.Add(1, - new KeyValuePair("result", submissionResponse.Status ?? "unknown"), - new KeyValuePair("backend", primaryBackend.Name)); - _metrics.SubmitLatency.Record(elapsed.TotalSeconds, new KeyValuePair("backend", primaryBackend.Name)); - await WriteAuditAsync(request, context, entry, submissionResponse, (long)elapsed.TotalMilliseconds, cancellationToken).ConfigureAwait(false); + await WriteAuditAsync(request, context, entry, canonicalOutcome, cancellationToken).ConfigureAwait(false); + if (mirrorOutcome is not null) + { + await WriteAuditAsync(request, context, entry, mirrorOutcome, cancellationToken).ConfigureAwait(false); + } return ToResult(entry); } private static AttestorSubmissionResult ToResult(AttestorEntry entry) { - return new AttestorSubmissionResult + var result = new AttestorSubmissionResult { Uuid = entry.RekorUuid, Index = entry.Index, LogUrl = entry.Log.Url, Status = entry.Status, - Proof = entry.Proof is null ? null : new AttestorSubmissionResult.RekorProof - { - Checkpoint = entry.Proof.Checkpoint is null ? null : new AttestorSubmissionResult.Checkpoint - { - Origin = entry.Proof.Checkpoint.Origin, - Size = entry.Proof.Checkpoint.Size, - RootHash = entry.Proof.Checkpoint.RootHash, - Timestamp = entry.Proof.Checkpoint.Timestamp?.ToString("O") - }, - Inclusion = entry.Proof.Inclusion is null ? null : new AttestorSubmissionResult.InclusionProof - { - LeafHash = entry.Proof.Inclusion.LeafHash, - Path = entry.Proof.Inclusion.Path - } - } + Proof = ToResultProof(entry.Proof) }; + + if (entry.Mirror is not null) + { + result.Mirror = new AttestorSubmissionResult.MirrorLog + { + Uuid = entry.Mirror.Uuid, + Index = entry.Mirror.Index, + LogUrl = entry.Mirror.Url, + Status = entry.Mirror.Status, + Proof = ToResultProof(entry.Mirror.Proof), + Error = entry.Mirror.Error + }; + } + + return result; } private AttestorEntry CreateEntry( AttestorSubmissionRequest request, - RekorSubmissionResponse submission, - RekorProofResponse? proof, SubmissionContext context, - byte[] canonicalBundle) + SubmissionOutcome canonicalOutcome, + SubmissionOutcome? mirrorOutcome) { + if (canonicalOutcome.Submission is null) + { + throw new InvalidOperationException("Canonical submission outcome must include a Rekor response."); + } + + var submission = canonicalOutcome.Submission; var now = _timeProvider.GetUtcNow(); + return new AttestorEntry { RekorUuid = submission.Uuid, @@ -201,24 +201,11 @@ internal sealed class AttestorSubmissionService : IAttestorSubmissionService }, BundleSha256 = request.Meta.BundleSha256, Index = submission.Index, - Proof = proof is null ? null : new AttestorEntry.ProofDescriptor - { - Checkpoint = proof.Checkpoint is null ? null : new AttestorEntry.CheckpointDescriptor - { - Origin = proof.Checkpoint.Origin, - Size = proof.Checkpoint.Size, - RootHash = proof.Checkpoint.RootHash, - Timestamp = proof.Checkpoint.Timestamp - }, - Inclusion = proof.Inclusion is null ? null : new AttestorEntry.InclusionDescriptor - { - LeafHash = proof.Inclusion.LeafHash, - Path = proof.Inclusion.Path - } - }, + Proof = ConvertProof(canonicalOutcome.Proof), Log = new AttestorEntry.LogDescriptor { - Url = submission.LogUrl ?? string.Empty, + Backend = canonicalOutcome.Backend, + Url = submission.LogUrl ?? canonicalOutcome.Url, LogId = null }, CreatedAt = now, @@ -229,28 +216,233 @@ internal sealed class AttestorSubmissionService : IAttestorSubmissionService Issuer = context.CallerAudience, SubjectAlternativeName = context.CallerSubject, KeyId = context.CallerClientId - } + }, + Mirror = mirrorOutcome is null ? null : CreateMirrorDescriptor(mirrorOutcome) }; } + private static string NormalizeLogPreference(string? value) + { + if (string.IsNullOrWhiteSpace(value)) + { + return "primary"; + } + + var normalized = value.Trim().ToLowerInvariant(); + return normalized switch + { + "primary" => "primary", + "mirror" => "mirror", + "both" => "both", + _ => "primary" + }; + } + + private async Task TryGetExistingEntryAsync(string bundleSha256, CancellationToken cancellationToken) + { + var dedupeUuid = await _dedupeStore.TryGetExistingAsync(bundleSha256, cancellationToken).ConfigureAwait(false); + if (string.IsNullOrWhiteSpace(dedupeUuid)) + { + return null; + } + + return await _repository.GetByUuidAsync(dedupeUuid, cancellationToken).ConfigureAwait(false) + ?? await _repository.GetByBundleShaAsync(bundleSha256, cancellationToken).ConfigureAwait(false); + } + + private async Task EnsureBackendsAsync( + AttestorEntry existing, + AttestorSubmissionRequest request, + SubmissionContext context, + bool requiresPrimary, + bool requiresMirror, + CancellationToken cancellationToken) + { + var entry = existing; + var updated = false; + + if (requiresPrimary && !IsPrimary(entry)) + { + var outcome = await SubmitToBackendAsync(request, "primary", _options.Rekor.Primary, cancellationToken).ConfigureAwait(false); + entry = PromoteToPrimary(entry, outcome); + await _repository.SaveAsync(entry, cancellationToken).ConfigureAwait(false); + await _dedupeStore.SetAsync(request.Meta.BundleSha256, entry.RekorUuid, DedupeTtl, cancellationToken).ConfigureAwait(false); + await WriteAuditAsync(request, context, entry, outcome, cancellationToken).ConfigureAwait(false); + updated = true; + } + + if (requiresMirror) + { + var mirrorSatisfied = entry.Mirror is not null + && entry.Mirror.Error is null + && string.Equals(entry.Mirror.Status, "included", StringComparison.OrdinalIgnoreCase) + && !string.IsNullOrEmpty(entry.Mirror.Uuid); + + if (!mirrorSatisfied) + { + try + { + var mirrorOutcome = await SubmitToBackendAsync(request, "mirror", _options.Rekor.Mirror, cancellationToken).ConfigureAwait(false); + entry = WithMirror(entry, mirrorOutcome); + await _repository.SaveAsync(entry, cancellationToken).ConfigureAwait(false); + await WriteAuditAsync(request, context, entry, mirrorOutcome, cancellationToken).ConfigureAwait(false); + updated = true; + } + catch (Exception ex) + { + _metrics.ErrorTotal.Add(1, new KeyValuePair("type", "submit_mirror")); + _logger.LogWarning(ex, "Mirror submission failed for deduplicated bundle {BundleSha}", request.Meta.BundleSha256); + var failure = SubmissionOutcome.Failure("mirror", _options.Rekor.Mirror.Url, ex, TimeSpan.Zero); + RecordSubmissionMetrics(failure); + entry = WithMirror(entry, failure); + await _repository.SaveAsync(entry, cancellationToken).ConfigureAwait(false); + await WriteAuditAsync(request, context, entry, failure, cancellationToken).ConfigureAwait(false); + updated = true; + } + } + } + + if (!updated) + { + _metrics.SubmitTotal.Add(1, + new KeyValuePair("result", "dedupe"), + new KeyValuePair("backend", "cache")); + } + + return entry; + } + + private static bool IsPrimary(AttestorEntry entry) => + string.Equals(entry.Log.Backend, "primary", StringComparison.OrdinalIgnoreCase); + + private async Task SubmitToBackendAsync( + AttestorSubmissionRequest request, + string backendName, + AttestorOptions.RekorBackendOptions backendOptions, + CancellationToken cancellationToken) + { + var backend = BuildBackend(backendName, backendOptions); + var stopwatch = Stopwatch.StartNew(); + try + { + var submission = await _rekorClient.SubmitAsync(request, backend, cancellationToken).ConfigureAwait(false); + stopwatch.Stop(); + + var proof = submission.Proof; + if (proof is null && string.Equals(submission.Status, "included", StringComparison.OrdinalIgnoreCase)) + { + try + { + proof = await _rekorClient.GetProofAsync(submission.Uuid, backend, cancellationToken).ConfigureAwait(false); + _metrics.ProofFetchTotal.Add(1, + new KeyValuePair("result", proof is null ? "missing" : "ok")); + } + catch (Exception ex) + { + _metrics.ErrorTotal.Add(1, new KeyValuePair("type", "proof_fetch")); + _logger.LogWarning(ex, "Proof fetch failed for {Uuid} on backend {Backend}", submission.Uuid, backendName); + } + } + + var outcome = SubmissionOutcome.Success(backendName, backend.Url, submission, proof, stopwatch.Elapsed); + RecordSubmissionMetrics(outcome); + return outcome; + } + catch (Exception ex) + { + stopwatch.Stop(); + _metrics.ErrorTotal.Add(1, new KeyValuePair("type", $"submit_{backendName}")); + _logger.LogError(ex, "Failed to submit bundle {BundleSha} to Rekor backend {Backend}", request.Meta.BundleSha256, backendName); + throw; + } + } + + private void RecordSubmissionMetrics(SubmissionOutcome outcome) + { + var result = outcome.IsSuccess + ? outcome.Submission!.Status ?? "unknown" + : "failed"; + + _metrics.SubmitTotal.Add(1, + new KeyValuePair("result", result), + new KeyValuePair("backend", outcome.Backend)); + + if (outcome.Latency > TimeSpan.Zero) + { + _metrics.SubmitLatency.Record(outcome.Latency.TotalSeconds, + new KeyValuePair("backend", outcome.Backend)); + } + } + + private async Task ArchiveAsync( + AttestorEntry entry, + byte[] canonicalBundle, + RekorProofResponse? proof, + CancellationToken cancellationToken) + { + var metadata = new Dictionary + { + ["logUrl"] = entry.Log.Url, + ["status"] = entry.Status + }; + + if (entry.Mirror is not null) + { + metadata["mirror.backend"] = entry.Mirror.Backend; + metadata["mirror.uuid"] = entry.Mirror.Uuid ?? string.Empty; + metadata["mirror.status"] = entry.Mirror.Status; + } + + var archiveBundle = new AttestorArchiveBundle + { + RekorUuid = entry.RekorUuid, + ArtifactSha256 = entry.Artifact.Sha256, + BundleSha256 = entry.BundleSha256, + CanonicalBundleJson = canonicalBundle, + ProofJson = proof is null ? Array.Empty() : JsonSerializer.SerializeToUtf8Bytes(proof, JsonSerializerOptions.Default), + Metadata = metadata + }; + + try + { + await _archiveStore.ArchiveBundleAsync(archiveBundle, cancellationToken).ConfigureAwait(false); + } + catch (Exception ex) + { + _logger.LogWarning(ex, "Failed to archive bundle {BundleSha}", entry.BundleSha256); + _metrics.ErrorTotal.Add(1, new KeyValuePair("type", "archive")); + } + } + private Task WriteAuditAsync( AttestorSubmissionRequest request, SubmissionContext context, AttestorEntry entry, - RekorSubmissionResponse submission, - long latencyMs, + SubmissionOutcome outcome, CancellationToken cancellationToken) { + var metadata = new Dictionary(); + if (!outcome.IsSuccess && outcome.Error is not null) + { + metadata["error"] = outcome.Error.Message; + } + var record = new AttestorAuditRecord { Action = "submit", - Result = submission.Status ?? "included", - RekorUuid = submission.Uuid, - Index = submission.Index, + Result = outcome.IsSuccess + ? outcome.Submission!.Status ?? "included" + : "failed", + RekorUuid = outcome.IsSuccess + ? outcome.Submission!.Uuid + : string.Equals(outcome.Backend, "primary", StringComparison.OrdinalIgnoreCase) + ? entry.RekorUuid + : entry.Mirror?.Uuid, + Index = outcome.Submission?.Index, ArtifactSha256 = request.Meta.Artifact.Sha256, BundleSha256 = request.Meta.BundleSha256, - Backend = "primary", - LatencyMs = latencyMs, + Backend = outcome.Backend, + LatencyMs = (long)outcome.Latency.TotalMilliseconds, Timestamp = _timeProvider.GetUtcNow(), Caller = new AttestorAuditRecord.CallerDescriptor { @@ -259,12 +451,160 @@ internal sealed class AttestorSubmissionService : IAttestorSubmissionService ClientId = context.CallerClientId, MtlsThumbprint = context.MtlsThumbprint, Tenant = context.CallerTenant - } + }, + Metadata = metadata }; return _auditSink.WriteAsync(record, cancellationToken); } + private static AttestorEntry.ProofDescriptor? ConvertProof(RekorProofResponse? proof) + { + if (proof is null) + { + return null; + } + + return new AttestorEntry.ProofDescriptor + { + Checkpoint = proof.Checkpoint is null ? null : new AttestorEntry.CheckpointDescriptor + { + Origin = proof.Checkpoint.Origin, + Size = proof.Checkpoint.Size, + RootHash = proof.Checkpoint.RootHash, + Timestamp = proof.Checkpoint.Timestamp + }, + Inclusion = proof.Inclusion is null ? null : new AttestorEntry.InclusionDescriptor + { + LeafHash = proof.Inclusion.LeafHash, + Path = proof.Inclusion.Path + } + }; + } + + private static AttestorSubmissionResult.RekorProof? ToResultProof(AttestorEntry.ProofDescriptor? proof) + { + if (proof is null) + { + return null; + } + + return new AttestorSubmissionResult.RekorProof + { + Checkpoint = proof.Checkpoint is null ? null : new AttestorSubmissionResult.Checkpoint + { + Origin = proof.Checkpoint.Origin, + Size = proof.Checkpoint.Size, + RootHash = proof.Checkpoint.RootHash, + Timestamp = proof.Checkpoint.Timestamp?.ToString("O") + }, + Inclusion = proof.Inclusion is null ? null : new AttestorSubmissionResult.InclusionProof + { + LeafHash = proof.Inclusion.LeafHash, + Path = proof.Inclusion.Path + } + }; + } + + private static AttestorEntry.LogReplicaDescriptor CreateMirrorDescriptor(SubmissionOutcome outcome) + { + return new AttestorEntry.LogReplicaDescriptor + { + Backend = outcome.Backend, + Url = outcome.IsSuccess + ? outcome.Submission!.LogUrl ?? outcome.Url + : outcome.Url, + Uuid = outcome.Submission?.Uuid, + Index = outcome.Submission?.Index, + Status = outcome.IsSuccess + ? outcome.Submission!.Status ?? "included" + : "failed", + Proof = outcome.IsSuccess ? ConvertProof(outcome.Proof) : null, + Error = outcome.Error?.Message + }; + } + + private static AttestorEntry WithMirror(AttestorEntry entry, SubmissionOutcome outcome) + { + return new AttestorEntry + { + RekorUuid = entry.RekorUuid, + Artifact = entry.Artifact, + BundleSha256 = entry.BundleSha256, + Index = entry.Index, + Proof = entry.Proof, + Log = entry.Log, + CreatedAt = entry.CreatedAt, + Status = entry.Status, + SignerIdentity = entry.SignerIdentity, + Mirror = CreateMirrorDescriptor(outcome) + }; + } + + private AttestorEntry PromoteToPrimary(AttestorEntry existing, SubmissionOutcome outcome) + { + if (outcome.Submission is null) + { + throw new InvalidOperationException("Cannot promote to primary without a successful submission."); + } + + var mirrorDescriptor = existing.Mirror; + if (mirrorDescriptor is null && !string.Equals(existing.Log.Backend, outcome.Backend, StringComparison.OrdinalIgnoreCase)) + { + mirrorDescriptor = CreateMirrorDescriptorFromEntry(existing); + } + + return new AttestorEntry + { + RekorUuid = outcome.Submission.Uuid, + Artifact = existing.Artifact, + BundleSha256 = existing.BundleSha256, + Index = outcome.Submission.Index, + Proof = ConvertProof(outcome.Proof), + Log = new AttestorEntry.LogDescriptor + { + Backend = outcome.Backend, + Url = outcome.Submission.LogUrl ?? outcome.Url, + LogId = existing.Log.LogId + }, + CreatedAt = existing.CreatedAt, + Status = outcome.Submission.Status ?? "included", + SignerIdentity = existing.SignerIdentity, + Mirror = mirrorDescriptor + }; + } + + private static AttestorEntry.LogReplicaDescriptor CreateMirrorDescriptorFromEntry(AttestorEntry entry) + { + return new AttestorEntry.LogReplicaDescriptor + { + Backend = entry.Log.Backend, + Url = entry.Log.Url, + Uuid = entry.RekorUuid, + Index = entry.Index, + Status = entry.Status, + Proof = entry.Proof, + LogId = entry.Log.LogId + }; + } + + private sealed record SubmissionOutcome( + string Backend, + string Url, + RekorSubmissionResponse? Submission, + RekorProofResponse? Proof, + TimeSpan Latency, + Exception? Error) + { + public bool IsSuccess => Submission is not null && Error is null; + + public static SubmissionOutcome Success(string backend, Uri backendUrl, RekorSubmissionResponse submission, RekorProofResponse? proof, TimeSpan latency) => + new SubmissionOutcome(backend, backendUrl.ToString(), submission, proof, latency, null); + + public static SubmissionOutcome Failure(string backend, string? url, Exception error, TimeSpan latency) => + new SubmissionOutcome(backend, url ?? string.Empty, null, null, latency, error); + } + private static RekorBackend BuildBackend(string name, AttestorOptions.RekorBackendOptions options) { if (string.IsNullOrWhiteSpace(options.Url)) diff --git a/src/StellaOps.Attestor/StellaOps.Attestor.Infrastructure/Verification/AttestorVerificationService.cs b/src/StellaOps.Attestor/StellaOps.Attestor.Infrastructure/Verification/AttestorVerificationService.cs index 28251220..5ab574a1 100644 --- a/src/StellaOps.Attestor/StellaOps.Attestor.Infrastructure/Verification/AttestorVerificationService.cs +++ b/src/StellaOps.Attestor/StellaOps.Attestor.Infrastructure/Verification/AttestorVerificationService.cs @@ -1,6 +1,12 @@ using System; +using System.Buffers.Binary; using System.Collections.Generic; +using System.Globalization; +using System.IO; using System.Linq; +using System.Security.Cryptography; +using System.Security.Cryptography.X509Certificates; +using System.Text; using System.Threading; using System.Threading.Tasks; using Microsoft.Extensions.Logging; @@ -10,7 +16,7 @@ using StellaOps.Attestor.Core.Rekor; using StellaOps.Attestor.Core.Storage; using StellaOps.Attestor.Core.Submission; using StellaOps.Attestor.Core.Verification; -using System.Security.Cryptography; +using StellaOps.Attestor.Core.Observability; namespace StellaOps.Attestor.Infrastructure.Verification; @@ -21,19 +27,22 @@ internal sealed class AttestorVerificationService : IAttestorVerificationService private readonly IRekorClient _rekorClient; private readonly ILogger _logger; private readonly AttestorOptions _options; + private readonly AttestorMetrics _metrics; public AttestorVerificationService( IAttestorEntryRepository repository, IDsseCanonicalizer canonicalizer, IRekorClient rekorClient, IOptions options, - ILogger logger) + ILogger logger, + AttestorMetrics metrics) { _repository = repository; _canonicalizer = canonicalizer; _rekorClient = rekorClient; _logger = logger; _options = options.Value; + _metrics = metrics; } public async Task VerifyAsync(AttestorVerificationRequest request, CancellationToken cancellationToken = default) @@ -67,11 +76,25 @@ internal sealed class AttestorVerificationService : IAttestorVerificationService } }, cancellationToken).ConfigureAwait(false); - var computedHash = Convert.ToHexString(System.Security.Cryptography.SHA256.HashData(canonicalBundle)).ToLowerInvariant(); + var computedHash = Convert.ToHexString(SHA256.HashData(canonicalBundle)).ToLowerInvariant(); if (!string.Equals(computedHash, entry.BundleSha256, StringComparison.OrdinalIgnoreCase)) { - issues.Add("Bundle hash does not match stored canonical hash."); + issues.Add("bundle_hash_mismatch"); } + + if (!TryDecodeBase64(request.Bundle.Dsse.PayloadBase64, out var payloadBytes)) + { + issues.Add("bundle_payload_invalid_base64"); + } + else + { + var preAuth = ComputePreAuthEncoding(request.Bundle.Dsse.PayloadType, payloadBytes); + VerifySignatures(entry, request.Bundle, preAuth, issues); + } + } + else + { + _logger.LogDebug("No DSSE bundle supplied for verification of {Uuid}; signature checks skipped.", entry.RekorUuid); } if (request.RefreshProof || entry.Proof is null) @@ -94,8 +117,12 @@ internal sealed class AttestorVerificationService : IAttestorVerificationService } } + VerifyMerkleProof(entry, issues); + var ok = issues.Count == 0 && string.Equals(entry.Status, "included", StringComparison.OrdinalIgnoreCase); + _metrics.VerifyTotal.Add(1, new KeyValuePair("result", ok ? "ok" : "failed")); + return new AttestorVerificationResult { Ok = ok, @@ -204,6 +231,472 @@ internal sealed class AttestorVerificationService : IAttestorVerificationService : entry; } + private void VerifySignatures(AttestorEntry entry, AttestorSubmissionRequest.SubmissionBundle bundle, byte[] preAuthEncoding, IList issues) + { + var mode = (entry.SignerIdentity.Mode ?? bundle.Mode ?? string.Empty).ToLowerInvariant(); + + if (mode == "kms") + { + if (!VerifyKmsSignature(bundle, preAuthEncoding, issues)) + { + issues.Add("signature_invalid_kms"); + } + + return; + } + + if (mode == "keyless") + { + VerifyKeylessSignature(entry, bundle, preAuthEncoding, issues); + return; + } + + issues.Add(string.IsNullOrEmpty(mode) + ? "signer_mode_unknown" + : $"signer_mode_unsupported:{mode}"); + } + + private bool VerifyKmsSignature(AttestorSubmissionRequest.SubmissionBundle bundle, byte[] preAuthEncoding, IList issues) + { + if (_options.Security.SignerIdentity.KmsKeys.Count == 0) + { + issues.Add("kms_key_missing"); + return false; + } + + var signatures = new List(); + foreach (var signature in bundle.Dsse.Signatures) + { + if (!TryDecodeBase64(signature.Signature, out var signatureBytes)) + { + issues.Add("signature_invalid_base64"); + return false; + } + + signatures.Add(signatureBytes); + } + + foreach (var secret in _options.Security.SignerIdentity.KmsKeys) + { + if (!TryDecodeSecret(secret, out var secretBytes)) + { + continue; + } + + using var hmac = new HMACSHA256(secretBytes); + var computed = hmac.ComputeHash(preAuthEncoding); + + foreach (var signatureBytes in signatures) + { + if (CryptographicOperations.FixedTimeEquals(computed, signatureBytes)) + { + return true; + } + } + } + + return false; + } + + private void VerifyKeylessSignature(AttestorEntry entry, AttestorSubmissionRequest.SubmissionBundle bundle, byte[] preAuthEncoding, IList issues) + { + if (bundle.CertificateChain.Count == 0) + { + issues.Add("certificate_chain_missing"); + return; + } + + var certificates = new List(); + try + { + foreach (var pem in bundle.CertificateChain) + { + certificates.Add(X509Certificate2.CreateFromPem(pem)); + } + } + catch (Exception ex) when (ex is CryptographicException or ArgumentException) + { + issues.Add("certificate_chain_invalid"); + _logger.LogWarning(ex, "Failed to parse certificate chain for {Uuid}", entry.RekorUuid); + return; + } + + var leafCertificate = certificates[0]; + + if (_options.Security.SignerIdentity.FulcioRoots.Count > 0) + { + using var chain = new X509Chain + { + ChainPolicy = + { + RevocationMode = X509RevocationMode.NoCheck, + VerificationFlags = X509VerificationFlags.NoFlag, + TrustMode = X509ChainTrustMode.CustomRootTrust + } + }; + + foreach (var rootPath in _options.Security.SignerIdentity.FulcioRoots) + { + try + { + if (File.Exists(rootPath)) + { + var rootCertificate = X509CertificateLoader.LoadCertificateFromFile(rootPath); + chain.ChainPolicy.CustomTrustStore.Add(rootCertificate); + } + } + catch (Exception ex) + { + _logger.LogWarning(ex, "Failed to load Fulcio root {Root}", rootPath); + } + } + + if (!chain.Build(leafCertificate)) + { + var status = string.Join(";", chain.ChainStatus.Select(s => s.StatusInformation.Trim())) + .Trim(';'); + issues.Add(string.IsNullOrEmpty(status) ? "certificate_chain_untrusted" : $"certificate_chain_untrusted:{status}"); + } + } + + if (_options.Security.SignerIdentity.AllowedSans.Count > 0) + { + var sans = GetSubjectAlternativeNames(leafCertificate); + if (!sans.Any(san => _options.Security.SignerIdentity.AllowedSans.Contains(san, StringComparer.OrdinalIgnoreCase))) + { + issues.Add("certificate_san_untrusted"); + } + } + + var signatureVerified = false; + foreach (var signature in bundle.Dsse.Signatures) + { + if (!TryDecodeBase64(signature.Signature, out var signatureBytes)) + { + issues.Add("signature_invalid_base64"); + return; + } + + if (TryVerifyWithCertificate(leafCertificate, preAuthEncoding, signatureBytes)) + { + signatureVerified = true; + break; + } + } + + if (!signatureVerified) + { + issues.Add("signature_invalid"); + } + } + + private static bool TryVerifyWithCertificate(X509Certificate2 certificate, byte[] preAuthEncoding, byte[] signature) + { + try + { + var ecdsa = certificate.GetECDsaPublicKey(); + if (ecdsa is not null) + { + using (ecdsa) + { + return ecdsa.VerifyData(preAuthEncoding, signature, HashAlgorithmName.SHA256); + } + } + + var rsa = certificate.GetRSAPublicKey(); + if (rsa is not null) + { + using (rsa) + { + return rsa.VerifyData(preAuthEncoding, signature, HashAlgorithmName.SHA256, RSASignaturePadding.Pkcs1); + } + } + } + catch (CryptographicException) + { + return false; + } + + return false; + } + + private static IEnumerable GetSubjectAlternativeNames(X509Certificate2 certificate) + { + foreach (var extension in certificate.Extensions) + { + if (!string.Equals(extension.Oid?.Value, "2.5.29.17", StringComparison.Ordinal)) + { + continue; + } + + var formatted = extension.Format(true); + var lines = formatted.Split(new[] { '\r', '\n' }, StringSplitOptions.RemoveEmptyEntries); + foreach (var line in lines) + { + var parts = line.Split('='); + if (parts.Length == 2) + { + yield return parts[1].Trim(); + } + } + } + } + + private static byte[] ComputePreAuthEncoding(string payloadType, byte[] payload) + { + var headerBytes = Encoding.UTF8.GetBytes(payloadType ?? string.Empty); + var buffer = new byte[6 + 8 + headerBytes.Length + 8 + payload.Length]; + var offset = 0; + + Encoding.ASCII.GetBytes("DSSEv1", 0, 6, buffer, offset); + offset += 6; + + BinaryPrimitives.WriteUInt64BigEndian(buffer.AsSpan(offset, 8), (ulong)headerBytes.Length); + offset += 8; + Buffer.BlockCopy(headerBytes, 0, buffer, offset, headerBytes.Length); + offset += headerBytes.Length; + + BinaryPrimitives.WriteUInt64BigEndian(buffer.AsSpan(offset, 8), (ulong)payload.Length); + offset += 8; + Buffer.BlockCopy(payload, 0, buffer, offset, payload.Length); + + return buffer; + } + + private void VerifyMerkleProof(AttestorEntry entry, IList issues) + { + if (entry.Proof is null) + { + issues.Add("proof_missing"); + return; + } + + if (!TryDecodeHash(entry.BundleSha256, out var bundleHash)) + { + issues.Add("bundle_hash_decode_failed"); + return; + } + + if (entry.Proof.Inclusion is null) + { + issues.Add("proof_inclusion_missing"); + return; + } + + if (entry.Proof.Inclusion.LeafHash is not null) + { + if (!TryDecodeHash(entry.Proof.Inclusion.LeafHash, out var proofLeaf)) + { + issues.Add("proof_leafhash_decode_failed"); + return; + } + + if (!CryptographicOperations.FixedTimeEquals(bundleHash, proofLeaf)) + { + issues.Add("proof_leafhash_mismatch"); + } + } + + var current = bundleHash; + + if (entry.Proof.Inclusion.Path.Count > 0) + { + var nodes = new List(); + foreach (var element in entry.Proof.Inclusion.Path) + { + if (!ProofPathNode.TryParse(element, out var node)) + { + issues.Add("proof_path_decode_failed"); + return; + } + + if (!node.HasOrientation) + { + issues.Add("proof_path_orientation_missing"); + return; + } + + nodes.Add(node); + } + + foreach (var node in nodes) + { + current = node.Left + ? HashInternal(node.Hash, current) + : HashInternal(current, node.Hash); + } + } + + if (entry.Proof.Checkpoint is null) + { + issues.Add("checkpoint_missing"); + return; + } + + if (!TryDecodeHash(entry.Proof.Checkpoint.RootHash, out var rootHash)) + { + issues.Add("checkpoint_root_decode_failed"); + return; + } + + if (!CryptographicOperations.FixedTimeEquals(current, rootHash)) + { + issues.Add("proof_root_mismatch"); + } + } + + private static byte[] HashInternal(byte[] left, byte[] right) + { + using var sha = SHA256.Create(); + var buffer = new byte[1 + left.Length + right.Length]; + buffer[0] = 0x01; + Buffer.BlockCopy(left, 0, buffer, 1, left.Length); + Buffer.BlockCopy(right, 0, buffer, 1 + left.Length, right.Length); + return sha.ComputeHash(buffer); + } + + private static bool TryDecodeSecret(string value, out byte[] bytes) + { + if (string.IsNullOrWhiteSpace(value)) + { + bytes = Array.Empty(); + return false; + } + + value = value.Trim(); + + if (value.StartsWith("base64:", StringComparison.OrdinalIgnoreCase)) + { + return TryDecodeBase64(value[7..], out bytes); + } + + if (value.StartsWith("hex:", StringComparison.OrdinalIgnoreCase)) + { + return TryDecodeHex(value[4..], out bytes); + } + + if (TryDecodeBase64(value, out bytes)) + { + return true; + } + + if (TryDecodeHex(value, out bytes)) + { + return true; + } + + bytes = Array.Empty(); + return false; + } + + private static bool TryDecodeBase64(string value, out byte[] bytes) + { + try + { + bytes = Convert.FromBase64String(value); + return true; + } + catch (FormatException) + { + bytes = Array.Empty(); + return false; + } + } + + private static bool TryDecodeHex(string value, out byte[] bytes) + { + try + { + bytes = Convert.FromHexString(value); + return true; + } + catch (FormatException) + { + bytes = Array.Empty(); + return false; + } + } + + private static bool TryDecodeHash(string? value, out byte[] bytes) + { + bytes = Array.Empty(); + if (string.IsNullOrWhiteSpace(value)) + { + return false; + } + + var trimmed = value.Trim(); + + if (TryDecodeHex(trimmed, out bytes)) + { + return true; + } + + if (TryDecodeBase64(trimmed, out bytes)) + { + return true; + } + + bytes = Array.Empty(); + return false; + } + + private readonly struct ProofPathNode + { + private ProofPathNode(bool hasOrientation, bool left, byte[] hash) + { + HasOrientation = hasOrientation; + Left = left; + Hash = hash; + } + + public bool HasOrientation { get; } + + public bool Left { get; } + + public byte[] Hash { get; } + + public static bool TryParse(string value, out ProofPathNode node) + { + node = default; + if (string.IsNullOrWhiteSpace(value)) + { + return false; + } + + var trimmed = value.Trim(); + var parts = trimmed.Split(':', 2); + bool hasOrientation = false; + bool left = false; + string hashPart = trimmed; + + if (parts.Length == 2) + { + var prefix = parts[0].Trim().ToLowerInvariant(); + if (prefix is "l" or "left") + { + hasOrientation = true; + left = true; + } + else if (prefix is "r" or "right") + { + hasOrientation = true; + left = false; + } + + hashPart = parts[1].Trim(); + } + + if (!TryDecodeHash(hashPart, out var hash)) + { + return false; + } + + node = new ProofPathNode(hasOrientation, left, hash); + return true; + } + } + private static AttestorEntry CloneWithProof(AttestorEntry entry, AttestorEntry.ProofDescriptor? proof) { return new AttestorEntry diff --git a/src/StellaOps.Attestor/StellaOps.Attestor.Tests/AttestorSubmissionServiceTests.cs b/src/StellaOps.Attestor/StellaOps.Attestor.Tests/AttestorSubmissionServiceTests.cs index c0c706f6..6ad8ab13 100644 --- a/src/StellaOps.Attestor/StellaOps.Attestor.Tests/AttestorSubmissionServiceTests.cs +++ b/src/StellaOps.Attestor/StellaOps.Attestor.Tests/AttestorSubmissionServiceTests.cs @@ -80,6 +80,207 @@ public sealed class AttestorSubmissionServiceTests Assert.Equal(first.Uuid, stored!.RekorUuid); } + [Fact] + public async Task Validator_ThrowsWhenModeNotAllowed() + { + var canonicalizer = new DefaultDsseCanonicalizer(); + var validator = new AttestorSubmissionValidator(canonicalizer, new[] { "kms" }); + + var request = CreateValidRequest(canonicalizer); + request.Bundle.Mode = "keyless"; + + await Assert.ThrowsAsync(() => validator.ValidateAsync(request)); + } + + [Fact] + public async Task SubmitAsync_Throws_WhenMirrorDisabledButRequested() + { + var options = Options.Create(new AttestorOptions + { + Redis = new AttestorOptions.RedisOptions { Url = string.Empty }, + Rekor = new AttestorOptions.RekorOptions + { + Primary = new AttestorOptions.RekorBackendOptions + { + Url = "https://rekor.primary.test", + ProofTimeoutMs = 1000, + PollIntervalMs = 50, + MaxAttempts = 2 + } + } + }); + + var canonicalizer = new DefaultDsseCanonicalizer(); + var validator = new AttestorSubmissionValidator(canonicalizer); + var repository = new InMemoryAttestorEntryRepository(); + var dedupeStore = new InMemoryAttestorDedupeStore(); + var rekorClient = new StubRekorClient(new NullLogger()); + var archiveStore = new NullAttestorArchiveStore(new NullLogger()); + var auditSink = new InMemoryAttestorAuditSink(); + var logger = new NullLogger(); + using var metrics = new AttestorMetrics(); + + var service = new AttestorSubmissionService( + validator, + repository, + dedupeStore, + rekorClient, + archiveStore, + auditSink, + options, + logger, + TimeProvider.System, + metrics); + + var request = CreateValidRequest(canonicalizer); + request.Meta.LogPreference = "mirror"; + + var context = new SubmissionContext + { + CallerSubject = "urn:stellaops:signer", + CallerAudience = "attestor", + CallerClientId = "signer-service", + CallerTenant = "default" + }; + + var ex = await Assert.ThrowsAsync(() => service.SubmitAsync(request, context)); + Assert.Equal("mirror_disabled", ex.Code); + } + + [Fact] + public async Task SubmitAsync_ReturnsMirrorMetadata_WhenPreferenceBoth() + { + var options = Options.Create(new AttestorOptions + { + Redis = new AttestorOptions.RedisOptions { Url = string.Empty }, + Rekor = new AttestorOptions.RekorOptions + { + Primary = new AttestorOptions.RekorBackendOptions + { + Url = "https://rekor.primary.test", + ProofTimeoutMs = 1000, + PollIntervalMs = 50, + MaxAttempts = 2 + }, + Mirror = new AttestorOptions.RekorMirrorOptions + { + Enabled = true, + Url = "https://rekor.mirror.test", + ProofTimeoutMs = 1000, + PollIntervalMs = 50, + MaxAttempts = 2 + } + } + }); + + var canonicalizer = new DefaultDsseCanonicalizer(); + var validator = new AttestorSubmissionValidator(canonicalizer); + var repository = new InMemoryAttestorEntryRepository(); + var dedupeStore = new InMemoryAttestorDedupeStore(); + var rekorClient = new StubRekorClient(new NullLogger()); + var archiveStore = new NullAttestorArchiveStore(new NullLogger()); + var auditSink = new InMemoryAttestorAuditSink(); + var logger = new NullLogger(); + using var metrics = new AttestorMetrics(); + + var service = new AttestorSubmissionService( + validator, + repository, + dedupeStore, + rekorClient, + archiveStore, + auditSink, + options, + logger, + TimeProvider.System, + metrics); + + var request = CreateValidRequest(canonicalizer); + request.Meta.LogPreference = "both"; + + var context = new SubmissionContext + { + CallerSubject = "urn:stellaops:signer", + CallerAudience = "attestor", + CallerClientId = "signer-service", + CallerTenant = "default" + }; + + var result = await service.SubmitAsync(request, context); + + Assert.NotNull(result.Mirror); + Assert.False(string.IsNullOrEmpty(result.Mirror!.Uuid)); + Assert.Equal("included", result.Mirror.Status); + } + + [Fact] + public async Task SubmitAsync_UsesMirrorAsCanonical_WhenPreferenceMirror() + { + var options = Options.Create(new AttestorOptions + { + Redis = new AttestorOptions.RedisOptions { Url = string.Empty }, + Rekor = new AttestorOptions.RekorOptions + { + Primary = new AttestorOptions.RekorBackendOptions + { + Url = "https://rekor.primary.test", + ProofTimeoutMs = 1000, + PollIntervalMs = 50, + MaxAttempts = 2 + }, + Mirror = new AttestorOptions.RekorMirrorOptions + { + Enabled = true, + Url = "https://rekor.mirror.test", + ProofTimeoutMs = 1000, + PollIntervalMs = 50, + MaxAttempts = 2 + } + } + }); + + var canonicalizer = new DefaultDsseCanonicalizer(); + var validator = new AttestorSubmissionValidator(canonicalizer); + var repository = new InMemoryAttestorEntryRepository(); + var dedupeStore = new InMemoryAttestorDedupeStore(); + var rekorClient = new StubRekorClient(new NullLogger()); + var archiveStore = new NullAttestorArchiveStore(new NullLogger()); + var auditSink = new InMemoryAttestorAuditSink(); + var logger = new NullLogger(); + using var metrics = new AttestorMetrics(); + + var service = new AttestorSubmissionService( + validator, + repository, + dedupeStore, + rekorClient, + archiveStore, + auditSink, + options, + logger, + TimeProvider.System, + metrics); + + var request = CreateValidRequest(canonicalizer); + request.Meta.LogPreference = "mirror"; + + var context = new SubmissionContext + { + CallerSubject = "urn:stellaops:signer", + CallerAudience = "attestor", + CallerClientId = "signer-service", + CallerTenant = "default" + }; + + var result = await service.SubmitAsync(request, context); + + Assert.NotNull(result.Uuid); + var stored = await repository.GetByBundleShaAsync(request.Meta.BundleSha256); + Assert.NotNull(stored); + Assert.Equal("mirror", stored!.Log.Backend); + Assert.Null(result.Mirror); + } + private static AttestorSubmissionRequest CreateValidRequest(DefaultDsseCanonicalizer canonicalizer) { var request = new AttestorSubmissionRequest diff --git a/src/StellaOps.Attestor/StellaOps.Attestor.Tests/AttestorVerificationServiceTests.cs b/src/StellaOps.Attestor/StellaOps.Attestor.Tests/AttestorVerificationServiceTests.cs index 840c1cd3..0e307130 100644 --- a/src/StellaOps.Attestor/StellaOps.Attestor.Tests/AttestorVerificationServiceTests.cs +++ b/src/StellaOps.Attestor/StellaOps.Attestor.Tests/AttestorVerificationServiceTests.cs @@ -1,3 +1,5 @@ +using System.Buffers.Binary; +using System.Collections.Generic; using System.Security.Cryptography; using System.Text; using System.Threading.Tasks; @@ -17,6 +19,9 @@ namespace StellaOps.Attestor.Tests; public sealed class AttestorVerificationServiceTests { + private static readonly byte[] HmacSecret = Encoding.UTF8.GetBytes("attestor-hmac-secret"); + private static readonly string HmacSecretBase64 = Convert.ToBase64String(HmacSecret); + [Fact] public async Task VerifyAsync_ReturnsOk_ForExistingUuid() { @@ -35,6 +40,14 @@ public sealed class AttestorVerificationServiceTests PollIntervalMs = 50, MaxAttempts = 2 } + }, + Security = new AttestorOptions.SecurityOptions + { + SignerIdentity = new AttestorOptions.SignerIdentityOptions + { + Mode = { "kms" }, + KmsKeys = { HmacSecretBase64 } + } } }); @@ -57,7 +70,7 @@ public sealed class AttestorVerificationServiceTests TimeProvider.System, metrics); - var submission = CreateSubmissionRequest(canonicalizer); + var submission = CreateSubmissionRequest(canonicalizer, HmacSecret); var context = new SubmissionContext { CallerSubject = "urn:stellaops:signer", @@ -73,11 +86,13 @@ public sealed class AttestorVerificationServiceTests canonicalizer, rekorClient, options, - new NullLogger()); + new NullLogger(), + metrics); var verifyResult = await verificationService.VerifyAsync(new AttestorVerificationRequest { - Uuid = response.Uuid + Uuid = response.Uuid, + Bundle = submission.Bundle }); Assert.True(verifyResult.Ok); @@ -100,6 +115,14 @@ public sealed class AttestorVerificationServiceTests PollIntervalMs = 50, MaxAttempts = 2 } + }, + Security = new AttestorOptions.SecurityOptions + { + SignerIdentity = new AttestorOptions.SignerIdentityOptions + { + Mode = { "kms" }, + KmsKeys = { HmacSecretBase64 } + } } }); @@ -122,7 +145,7 @@ public sealed class AttestorVerificationServiceTests TimeProvider.System, metrics); - var submission = CreateSubmissionRequest(canonicalizer); + var submission = CreateSubmissionRequest(canonicalizer, HmacSecret); var context = new SubmissionContext { CallerSubject = "urn:stellaops:signer", @@ -138,9 +161,10 @@ public sealed class AttestorVerificationServiceTests canonicalizer, rekorClient, options, - new NullLogger()); + new NullLogger(), + metrics); - var tamperedBundle = submission.Bundle; + var tamperedBundle = CloneBundle(submission.Bundle); tamperedBundle.Dsse.PayloadBase64 = Convert.ToBase64String(Encoding.UTF8.GetBytes("{\"tampered\":true}")); var result = await verificationService.VerifyAsync(new AttestorVerificationRequest @@ -150,29 +174,21 @@ public sealed class AttestorVerificationServiceTests }); Assert.False(result.Ok); - Assert.Contains(result.Issues, issue => issue.Contains("Bundle hash", StringComparison.OrdinalIgnoreCase)); + Assert.Contains(result.Issues, issue => issue.Contains("signature_invalid", StringComparison.OrdinalIgnoreCase)); } - private static AttestorSubmissionRequest CreateSubmissionRequest(DefaultDsseCanonicalizer canonicalizer) + private static AttestorSubmissionRequest CreateSubmissionRequest(DefaultDsseCanonicalizer canonicalizer, byte[] hmacSecret) { var payload = Encoding.UTF8.GetBytes("{}"); var request = new AttestorSubmissionRequest { Bundle = new AttestorSubmissionRequest.SubmissionBundle { - Mode = "keyless", + Mode = "kms", Dsse = new AttestorSubmissionRequest.DsseEnvelope { PayloadType = "application/vnd.in-toto+json", - PayloadBase64 = Convert.ToBase64String(payload), - Signatures = - { - new AttestorSubmissionRequest.DsseSignature - { - KeyId = "test", - Signature = Convert.ToBase64String(RandomNumberGenerator.GetBytes(32)) - } - } + PayloadBase64 = Convert.ToBase64String(payload) } }, Meta = new AttestorSubmissionRequest.SubmissionMeta @@ -187,8 +203,65 @@ public sealed class AttestorVerificationServiceTests } }; + var preAuth = ComputePreAuthEncodingForTests(request.Bundle.Dsse.PayloadType, payload); + using (var hmac = new HMACSHA256(hmacSecret)) + { + var signature = hmac.ComputeHash(preAuth); + request.Bundle.Dsse.Signatures.Add(new AttestorSubmissionRequest.DsseSignature + { + KeyId = "kms-test", + Signature = Convert.ToBase64String(signature) + }); + } + var canonical = canonicalizer.CanonicalizeAsync(request).GetAwaiter().GetResult(); request.Meta.BundleSha256 = Convert.ToHexString(SHA256.HashData(canonical)).ToLowerInvariant(); return request; } + + private static AttestorSubmissionRequest.SubmissionBundle CloneBundle(AttestorSubmissionRequest.SubmissionBundle source) + { + var clone = new AttestorSubmissionRequest.SubmissionBundle + { + Mode = source.Mode, + Dsse = new AttestorSubmissionRequest.DsseEnvelope + { + PayloadType = source.Dsse.PayloadType, + PayloadBase64 = source.Dsse.PayloadBase64 + } + }; + + foreach (var certificate in source.CertificateChain) + { + clone.CertificateChain.Add(certificate); + } + + foreach (var signature in source.Dsse.Signatures) + { + clone.Dsse.Signatures.Add(new AttestorSubmissionRequest.DsseSignature + { + KeyId = signature.KeyId, + Signature = signature.Signature + }); + } + + return clone; + } + + private static byte[] ComputePreAuthEncodingForTests(string payloadType, byte[] payload) + { + var headerBytes = Encoding.UTF8.GetBytes(payloadType ?? string.Empty); + var buffer = new byte[6 + 8 + headerBytes.Length + 8 + payload.Length]; + var offset = 0; + Encoding.ASCII.GetBytes("DSSEv1", 0, 6, buffer, offset); + offset += 6; + BinaryPrimitives.WriteUInt64BigEndian(buffer.AsSpan(offset, 8), (ulong)headerBytes.Length); + offset += 8; + Buffer.BlockCopy(headerBytes, 0, buffer, offset, headerBytes.Length); + offset += headerBytes.Length; + BinaryPrimitives.WriteUInt64BigEndian(buffer.AsSpan(offset, 8), (ulong)payload.Length); + offset += 8; + Buffer.BlockCopy(payload, 0, buffer, offset, payload.Length); + return buffer; + } } diff --git a/src/StellaOps.Attestor/StellaOps.Attestor.WebService/Program.cs b/src/StellaOps.Attestor/StellaOps.Attestor.WebService/Program.cs index a577ee58..f30741f2 100644 --- a/src/StellaOps.Attestor/StellaOps.Attestor.WebService/Program.cs +++ b/src/StellaOps.Attestor/StellaOps.Attestor.WebService/Program.cs @@ -1,6 +1,11 @@ using System.Collections.Generic; +using System.IO; +using System.Linq; +using System.Security.Authentication; +using System.Security.Cryptography; using System.Security.Claims; using System.Security.Cryptography.X509Certificates; +using System.Threading.RateLimiting; using Serilog; using Serilog.Events; using StellaOps.Attestor.Core.Options; @@ -13,6 +18,7 @@ using OpenTelemetry.Metrics; using StellaOps.Attestor.Core.Observability; using StellaOps.Attestor.Core.Verification; using Microsoft.AspNetCore.Server.Kestrel.Https; +using Serilog.Context; const string ConfigurationSection = "attestor"; @@ -36,9 +42,45 @@ builder.Host.UseSerilog((context, services, loggerConfiguration) => var attestorOptions = builder.Configuration.BindOptions(ConfigurationSection); +var clientCertificateAuthorities = LoadClientCertificateAuthorities(attestorOptions.Security.Mtls.CaBundle); + builder.Services.AddSingleton(TimeProvider.System); builder.Services.AddSingleton(attestorOptions); +builder.Services.AddRateLimiter(options => +{ + options.RejectionStatusCode = StatusCodes.Status429TooManyRequests; + options.OnRejected = static (context, _) => + { + context.HttpContext.Response.Headers.TryAdd("Retry-After", "1"); + return ValueTask.CompletedTask; + }; + + options.AddPolicy("attestor-submissions", httpContext => + { + var identity = httpContext.Connection.ClientCertificate?.Thumbprint + ?? httpContext.User.FindFirst("sub")?.Value + ?? httpContext.User.FindFirst("client_id")?.Value + ?? httpContext.Connection.RemoteIpAddress?.ToString() + ?? "anonymous"; + + var quota = attestorOptions.Quotas.PerCaller; + var tokensPerPeriod = Math.Max(1, quota.Qps); + var tokenLimit = Math.Max(tokensPerPeriod, quota.Burst); + var queueLimit = Math.Max(quota.Burst, tokensPerPeriod); + + return RateLimitPartition.GetTokenBucketLimiter(identity, _ => new TokenBucketRateLimiterOptions + { + TokenLimit = tokenLimit, + TokensPerPeriod = tokensPerPeriod, + ReplenishmentPeriod = TimeSpan.FromSeconds(1), + QueueLimit = queueLimit, + QueueProcessingOrder = QueueProcessingOrder.OldestFirst, + AutoReplenishment = true + }); + }); +}); + builder.Services.AddOptions() .Bind(builder.Configuration.GetSection(ConfigurationSection)) .ValidateOnStart(); @@ -105,6 +147,61 @@ builder.WebHost.ConfigureKestrel(kestrel => { https.ClientCertificateMode = ClientCertificateMode.RequireCertificate; } + + https.SslProtocols = SslProtocols.Tls13 | SslProtocols.Tls12; + + https.ClientCertificateValidation = (certificate, _, _) => + { + if (!attestorOptions.Security.Mtls.RequireClientCertificate) + { + return true; + } + + if (certificate is null) + { + Log.Warning("Client certificate missing"); + return false; + } + + if (clientCertificateAuthorities.Count > 0) + { + using var chain = new X509Chain + { + ChainPolicy = + { + RevocationMode = X509RevocationMode.NoCheck, + TrustMode = X509ChainTrustMode.CustomRootTrust + } + }; + + foreach (var authority in clientCertificateAuthorities) + { + chain.ChainPolicy.CustomTrustStore.Add(authority); + } + + if (!chain.Build(certificate)) + { + Log.Warning("Client certificate chain validation failed for {Subject}", certificate.Subject); + return false; + } + } + + if (attestorOptions.Security.Mtls.AllowedThumbprints.Count > 0 && + !attestorOptions.Security.Mtls.AllowedThumbprints.Contains(certificate.Thumbprint ?? string.Empty, StringComparer.OrdinalIgnoreCase)) + { + Log.Warning("Client certificate thumbprint {Thumbprint} rejected", certificate.Thumbprint); + return false; + } + + if (attestorOptions.Security.Mtls.AllowedSubjects.Count > 0 && + !attestorOptions.Security.Mtls.AllowedSubjects.Contains(certificate.Subject, StringComparer.OrdinalIgnoreCase)) + { + Log.Warning("Client certificate subject {Subject} rejected", certificate.Subject); + return false; + } + + return true; + }; }); }); @@ -112,6 +209,22 @@ var app = builder.Build(); app.UseSerilogRequestLogging(); +app.Use(async (context, next) => +{ + var correlationId = context.Request.Headers["X-Correlation-Id"].FirstOrDefault(); + if (string.IsNullOrWhiteSpace(correlationId)) + { + correlationId = Guid.NewGuid().ToString("N"); + } + + context.Response.Headers["X-Correlation-Id"] = correlationId; + + using (LogContext.PushProperty("CorrelationId", correlationId)) + { + await next().ConfigureAwait(false); + } +}); + app.UseExceptionHandler(static handler => { handler.Run(async context => @@ -121,6 +234,8 @@ app.UseExceptionHandler(static handler => }); }); +app.UseRateLimiter(); + app.UseAuthentication(); app.UseAuthorization(); @@ -156,7 +271,8 @@ app.MapPost("/api/v1/rekor/entries", async (AttestorSubmissionRequest request, H }); } }) -.RequireAuthorization("attestor:write"); +.RequireAuthorization("attestor:write") +.RequireRateLimiting("attestor-submissions"); app.MapGet("/api/v1/rekor/entries/{uuid}", async (string uuid, bool? refresh, IAttestorVerificationService verificationService, CancellationToken cancellationToken) => { @@ -170,6 +286,7 @@ app.MapGet("/api/v1/rekor/entries/{uuid}", async (string uuid, bool? refresh, IA { uuid = entry.RekorUuid, index = entry.Index, + backend = entry.Log.Backend, proof = entry.Proof is null ? null : new { checkpoint = entry.Proof.Checkpoint is null ? null : new @@ -187,6 +304,30 @@ app.MapGet("/api/v1/rekor/entries/{uuid}", async (string uuid, bool? refresh, IA }, logURL = entry.Log.Url, status = entry.Status, + mirror = entry.Mirror is null ? null : new + { + backend = entry.Mirror.Backend, + uuid = entry.Mirror.Uuid, + index = entry.Mirror.Index, + logURL = entry.Mirror.Url, + status = entry.Mirror.Status, + proof = entry.Mirror.Proof is null ? null : new + { + checkpoint = entry.Mirror.Proof.Checkpoint is null ? null : new + { + origin = entry.Mirror.Proof.Checkpoint.Origin, + size = entry.Mirror.Proof.Checkpoint.Size, + rootHash = entry.Mirror.Proof.Checkpoint.RootHash, + timestamp = entry.Mirror.Proof.Checkpoint.Timestamp?.ToString("O") + }, + inclusion = entry.Mirror.Proof.Inclusion is null ? null : new + { + leafHash = entry.Mirror.Proof.Inclusion.LeafHash, + path = entry.Mirror.Proof.Inclusion.Path + } + }, + error = entry.Mirror.Error + }, artifact = new { sha256 = entry.Artifact.Sha256, @@ -232,3 +373,33 @@ static SubmissionContext BuildSubmissionContext(ClaimsPrincipal user, X509Certif MtlsThumbprint = certificate.Thumbprint }; } + +static List LoadClientCertificateAuthorities(string? path) +{ + var certificates = new List(); + + if (string.IsNullOrWhiteSpace(path)) + { + return certificates; + } + + try + { + if (!File.Exists(path)) + { + Log.Warning("Client CA bundle '{Path}' not found", path); + return certificates; + } + + var collection = new X509Certificate2Collection(); + collection.ImportFromPemFile(path); + + certificates.AddRange(collection.Cast()); + } + catch (Exception ex) when (ex is IOException or CryptographicException) + { + Log.Warning(ex, "Failed to load client CA bundle from {Path}", path); + } + + return certificates; +} diff --git a/src/StellaOps.Attestor/TASKS.md b/src/StellaOps.Attestor/TASKS.md index 6dc39983..5ed09250 100644 --- a/src/StellaOps.Attestor/TASKS.md +++ b/src/StellaOps.Attestor/TASKS.md @@ -6,6 +6,5 @@ | ATTESTOR-VERIFY-11-202 | DONE (2025-10-19) | Attestor Guild | — | `/rekor/verify` + retrieval endpoints validating signatures and Merkle proofs. | ✅ `GET /api/v1/rekor/entries/{uuid}` surfaces cached entries with optional backend refresh and handles not-found/refresh flows.
✅ `POST /api/v1/rekor/verify` accepts UUID, bundle, or artifact hash inputs; verifies DSSE signatures, Merkle proofs, and checkpoint anchors.
✅ Verification output returns `{ok, uuid, index, logURL, checkedAt}` with failure diagnostics for invalid proofs.
✅ Unit/integration tests exercise cache hits, backend refresh, invalid bundle/proof scenarios, and checkpoint trust anchor enforcement. | | ATTESTOR-OBS-11-203 | DONE (2025-10-19) | Attestor Guild | — | Telemetry, alerting, mTLS hardening, and archive workflow for Attestor. | ✅ Structured logs, metrics, and optional traces record submission latency, proof fetch outcomes, verification results, and Rekor error buckets with correlation IDs.
✅ mTLS enforcement hardened (peer allowlist, SAN checks, rate limiting) and documented; TLS settings audited for modern ciphers only.
✅ Alerting/dashboard pack covers error rates, proof backlog, Redis/Mongo health, and archive job failures; runbook updated.
✅ Archive workflow includes retention policy jobs, failure alerts, and periodic verification of stored bundles and proofs. | -> Remark (2025-10-19): Wave 0 prerequisites reviewed (none outstanding); Attestor Guild tasks moved to DOING for execution. -> Remark (2025-10-19): `/rekor/entries` submission service implemented with Mongo/Redis persistence, optional S3 archival, Rekor HTTP client, and OpenTelemetry metrics; verification APIs (`/rekor/entries/{uuid}`, `/rekor/verify`) added with proof refresh and canonical hash checks. Remaining: integrate real Rekor endpoints in staging and expand failure-mode tests. -> Remark (2025-10-19): Added Rekor mock client + integration harness to unblock attestor verification testing without external connectivity. Follow-up tasks to wire staging Rekor and record retry/error behavior still pending. +> Remark (2025-10-19): Wave 0 prerequisites reviewed (none outstanding); ATTESTOR-API-11-201, ATTESTOR-VERIFY-11-202, and ATTESTOR-OBS-11-203 tracked as DOING per Wave 0A kickoff. +> Remark (2025-10-19): Dual-log submissions, signature/proof verification, and observability hardening landed; attestor endpoints now rate-limited per client with correlation-ID logging and updated docs/tests. diff --git a/src/StellaOps.Auth.Security/Dpop/DpopNonceConsumeResult.cs b/src/StellaOps.Auth.Security/Dpop/DpopNonceConsumeResult.cs new file mode 100644 index 00000000..c0ba29bd --- /dev/null +++ b/src/StellaOps.Auth.Security/Dpop/DpopNonceConsumeResult.cs @@ -0,0 +1,50 @@ +using System; + +namespace StellaOps.Auth.Security.Dpop; + +/// +/// Represents the outcome of attempting to consume a DPoP nonce. +/// +public sealed class DpopNonceConsumeResult +{ + private DpopNonceConsumeResult(DpopNonceConsumeStatus status, DateTimeOffset? issuedAt, DateTimeOffset? expiresAt) + { + Status = status; + IssuedAt = issuedAt; + ExpiresAt = expiresAt; + } + + /// + /// Consumption status. + /// + public DpopNonceConsumeStatus Status { get; } + + /// + /// Timestamp the nonce was originally issued (when available). + /// + public DateTimeOffset? IssuedAt { get; } + + /// + /// Expiry timestamp for the nonce (when available). + /// + public DateTimeOffset? ExpiresAt { get; } + + public static DpopNonceConsumeResult Success(DateTimeOffset issuedAt, DateTimeOffset expiresAt) + => new(DpopNonceConsumeStatus.Success, issuedAt, expiresAt); + + public static DpopNonceConsumeResult Expired(DateTimeOffset? issuedAt, DateTimeOffset expiresAt) + => new(DpopNonceConsumeStatus.Expired, issuedAt, expiresAt); + + public static DpopNonceConsumeResult NotFound() + => new(DpopNonceConsumeStatus.NotFound, null, null); +} + +/// +/// Known statuses for nonce consumption attempts. +/// +public enum DpopNonceConsumeStatus +{ + Success, + Expired, + NotFound +} diff --git a/src/StellaOps.Auth.Security/Dpop/DpopNonceIssueResult.cs b/src/StellaOps.Auth.Security/Dpop/DpopNonceIssueResult.cs new file mode 100644 index 00000000..2546725b --- /dev/null +++ b/src/StellaOps.Auth.Security/Dpop/DpopNonceIssueResult.cs @@ -0,0 +1,56 @@ +using System; + +namespace StellaOps.Auth.Security.Dpop; + +/// +/// Represents the result of issuing a DPoP nonce. +/// +public sealed class DpopNonceIssueResult +{ + private DpopNonceIssueResult(DpopNonceIssueStatus status, string? nonce, DateTimeOffset? expiresAt, string? error) + { + Status = status; + Nonce = nonce; + ExpiresAt = expiresAt; + Error = error; + } + + /// + /// Issue status. + /// + public DpopNonceIssueStatus Status { get; } + + /// + /// Issued nonce when is . + /// + public string? Nonce { get; } + + /// + /// Expiry timestamp for the issued nonce (UTC). + /// + public DateTimeOffset? ExpiresAt { get; } + + /// + /// Additional failure information, where applicable. + /// + public string? Error { get; } + + public static DpopNonceIssueResult Success(string nonce, DateTimeOffset expiresAt) + => new(DpopNonceIssueStatus.Success, nonce, expiresAt, null); + + public static DpopNonceIssueResult RateLimited(string? error = null) + => new(DpopNonceIssueStatus.RateLimited, null, null, error); + + public static DpopNonceIssueResult Failure(string? error = null) + => new(DpopNonceIssueStatus.Failure, null, null, error); +} + +/// +/// Known statuses for nonce issuance. +/// +public enum DpopNonceIssueStatus +{ + Success, + RateLimited, + Failure +} diff --git a/src/StellaOps.Auth.Security/Dpop/DpopNonceUtilities.cs b/src/StellaOps.Auth.Security/Dpop/DpopNonceUtilities.cs new file mode 100644 index 00000000..917b4ef1 --- /dev/null +++ b/src/StellaOps.Auth.Security/Dpop/DpopNonceUtilities.cs @@ -0,0 +1,66 @@ +using System; +using System.Security.Cryptography; +using System.Text; + +namespace StellaOps.Auth.Security.Dpop; + +internal static class DpopNonceUtilities +{ + private static readonly char[] Base64Padding = { '=' }; + + internal static string GenerateNonce() + { + Span buffer = stackalloc byte[32]; + RandomNumberGenerator.Fill(buffer); + + return Convert.ToBase64String(buffer) + .TrimEnd(Base64Padding) + .Replace('+', '-') + .Replace('/', '_'); + } + + internal static byte[] ComputeNonceHash(string nonce) + { + ArgumentException.ThrowIfNullOrWhiteSpace(nonce); + var bytes = Encoding.UTF8.GetBytes(nonce); + return SHA256.HashData(bytes); + } + + internal static string EncodeHash(ReadOnlySpan hash) + => Convert.ToHexString(hash); + + internal static string ComputeStorageKey(string audience, string clientId, string keyThumbprint) + { + ArgumentException.ThrowIfNullOrWhiteSpace(audience); + ArgumentException.ThrowIfNullOrWhiteSpace(clientId); + ArgumentException.ThrowIfNullOrWhiteSpace(keyThumbprint); + + return string.Create( + "dpop-nonce:".Length + audience.Length + clientId.Length + keyThumbprint.Length + 2, + (audience.Trim(), clientId.Trim(), keyThumbprint.Trim()), + static (span, parts) => + { + var index = 0; + const string Prefix = "dpop-nonce:"; + Prefix.CopyTo(span); + index += Prefix.Length; + + index = Append(span, index, parts.Item1); + span[index++] = ':'; + index = Append(span, index, parts.Item2); + span[index++] = ':'; + _ = Append(span, index, parts.Item3); + }); + + static int Append(Span span, int index, string value) + { + if (value.Length == 0) + { + throw new ArgumentException("Value must not be empty after trimming."); + } + + value.AsSpan().CopyTo(span[index..]); + return index + value.Length; + } + } +} diff --git a/src/StellaOps.Auth.Security/Dpop/IDpopNonceStore.cs b/src/StellaOps.Auth.Security/Dpop/IDpopNonceStore.cs new file mode 100644 index 00000000..600296bc --- /dev/null +++ b/src/StellaOps.Auth.Security/Dpop/IDpopNonceStore.cs @@ -0,0 +1,45 @@ +using System; +using System.Threading; +using System.Threading.Tasks; + +namespace StellaOps.Auth.Security.Dpop; + +/// +/// Provides persistence and validation for DPoP nonces. +/// +public interface IDpopNonceStore +{ + /// + /// Issues a nonce tied to the specified audience, client, and DPoP key thumbprint. + /// + /// Audience the nonce applies to. + /// Client identifier requesting the nonce. + /// Thumbprint of the DPoP public key. + /// Time-to-live for the nonce. + /// Maximum number of nonces that can be issued within a one-minute window for the tuple. + /// Cancellation token. + /// Outcome describing the issued nonce. + ValueTask IssueAsync( + string audience, + string clientId, + string keyThumbprint, + TimeSpan ttl, + int maxIssuancePerMinute, + CancellationToken cancellationToken = default); + + /// + /// Attempts to consume a nonce previously issued for the tuple. + /// + /// Nonce supplied by the client. + /// Audience the nonce should match. + /// Client identifier. + /// Thumbprint of the DPoP public key. + /// Cancellation token. + /// Outcome describing whether the nonce was accepted. + ValueTask TryConsumeAsync( + string nonce, + string audience, + string clientId, + string keyThumbprint, + CancellationToken cancellationToken = default); +} diff --git a/src/StellaOps.Auth.Security/Dpop/InMemoryDpopNonceStore.cs b/src/StellaOps.Auth.Security/Dpop/InMemoryDpopNonceStore.cs new file mode 100644 index 00000000..37b1c77f --- /dev/null +++ b/src/StellaOps.Auth.Security/Dpop/InMemoryDpopNonceStore.cs @@ -0,0 +1,176 @@ +using System; +using System.Collections.Concurrent; +using System.Collections.Generic; +using System.Security.Cryptography; +using System.Text; +using System.Threading; +using Microsoft.Extensions.Logging; +using System.Threading.Tasks; + +namespace StellaOps.Auth.Security.Dpop; + +/// +/// In-memory implementation of suitable for single-host or test environments. +/// +public sealed class InMemoryDpopNonceStore : IDpopNonceStore +{ + private static readonly TimeSpan IssuanceWindow = TimeSpan.FromMinutes(1); + private readonly ConcurrentDictionary nonces = new(StringComparer.Ordinal); + private readonly ConcurrentDictionary issuanceBuckets = new(StringComparer.Ordinal); + private readonly TimeProvider timeProvider; + private readonly ILogger? logger; + + public InMemoryDpopNonceStore(TimeProvider? timeProvider = null, ILogger? logger = null) + { + this.timeProvider = timeProvider ?? TimeProvider.System; + this.logger = logger; + } + + public ValueTask IssueAsync( + string audience, + string clientId, + string keyThumbprint, + TimeSpan ttl, + int maxIssuancePerMinute, + CancellationToken cancellationToken = default) + { + ArgumentException.ThrowIfNullOrWhiteSpace(audience); + ArgumentException.ThrowIfNullOrWhiteSpace(clientId); + ArgumentException.ThrowIfNullOrWhiteSpace(keyThumbprint); + + if (ttl <= TimeSpan.Zero) + { + throw new ArgumentOutOfRangeException(nameof(ttl), "Nonce TTL must be greater than zero."); + } + + if (maxIssuancePerMinute < 1) + { + throw new ArgumentOutOfRangeException(nameof(maxIssuancePerMinute), "Max issuance per minute must be at least 1."); + } + + cancellationToken.ThrowIfCancellationRequested(); + + var now = timeProvider.GetUtcNow(); + var bucketKey = BuildBucketKey(audience, clientId, keyThumbprint); + var bucket = issuanceBuckets.GetOrAdd(bucketKey, static _ => new IssuanceBucket()); + + bool allowed; + lock (bucket.SyncRoot) + { + bucket.Prune(now - IssuanceWindow); + + if (bucket.IssuanceTimes.Count >= maxIssuancePerMinute) + { + allowed = false; + } + else + { + bucket.IssuanceTimes.Enqueue(now); + allowed = true; + } + } + + if (!allowed) + { + logger?.LogDebug("DPoP nonce issuance throttled for {BucketKey}.", bucketKey); + return ValueTask.FromResult(DpopNonceIssueResult.RateLimited("rate_limited")); + } + + var nonce = GenerateNonce(); + var nonceKey = BuildNonceKey(audience, clientId, keyThumbprint, nonce); + var expiresAt = now + ttl; + nonces[nonceKey] = new StoredNonce(now, expiresAt); + return ValueTask.FromResult(DpopNonceIssueResult.Success(nonce, expiresAt)); + } + + public ValueTask TryConsumeAsync( + string nonce, + string audience, + string clientId, + string keyThumbprint, + CancellationToken cancellationToken = default) + { + ArgumentException.ThrowIfNullOrWhiteSpace(nonce); + ArgumentException.ThrowIfNullOrWhiteSpace(audience); + ArgumentException.ThrowIfNullOrWhiteSpace(clientId); + ArgumentException.ThrowIfNullOrWhiteSpace(keyThumbprint); + + cancellationToken.ThrowIfCancellationRequested(); + + var now = timeProvider.GetUtcNow(); + var nonceKey = BuildNonceKey(audience, clientId, keyThumbprint, nonce); + + if (!nonces.TryRemove(nonceKey, out var stored)) + { + logger?.LogDebug("DPoP nonce {NonceKey} not found during consumption.", nonceKey); + return ValueTask.FromResult(DpopNonceConsumeResult.NotFound()); + } + + if (stored.ExpiresAt <= now) + { + logger?.LogDebug("DPoP nonce {NonceKey} expired at {ExpiresAt:o}.", nonceKey, stored.ExpiresAt); + return ValueTask.FromResult(DpopNonceConsumeResult.Expired(stored.IssuedAt, stored.ExpiresAt)); + } + + return ValueTask.FromResult(DpopNonceConsumeResult.Success(stored.IssuedAt, stored.ExpiresAt)); + } + + private static string BuildBucketKey(string audience, string clientId, string keyThumbprint) + => $"{audience.Trim().ToLowerInvariant()}::{clientId.Trim().ToLowerInvariant()}::{keyThumbprint.Trim().ToLowerInvariant()}"; + + private static string BuildNonceKey(string audience, string clientId, string keyThumbprint, string nonce) + { + var bucketKey = BuildBucketKey(audience, clientId, keyThumbprint); + var digest = ComputeSha256(nonce); + return $"{bucketKey}::{digest}"; + } + + private static string ComputeSha256(string value) + { + var bytes = Encoding.UTF8.GetBytes(value); + var hash = SHA256.HashData(bytes); + return Base64UrlEncode(hash); + } + + private static string Base64UrlEncode(ReadOnlySpan bytes) + { + return Convert.ToBase64String(bytes) + .TrimEnd('=') + .Replace('+', '-') + .Replace('/', '_'); + } + + private static string GenerateNonce() + { + Span buffer = stackalloc byte[32]; + RandomNumberGenerator.Fill(buffer); + return Base64UrlEncode(buffer); + } + + private sealed class StoredNonce + { + internal StoredNonce(DateTimeOffset issuedAt, DateTimeOffset expiresAt) + { + IssuedAt = issuedAt; + ExpiresAt = expiresAt; + } + + internal DateTimeOffset IssuedAt { get; } + + internal DateTimeOffset ExpiresAt { get; } + } + + private sealed class IssuanceBucket + { + internal object SyncRoot { get; } = new(); + internal Queue IssuanceTimes { get; } = new(); + + internal void Prune(DateTimeOffset threshold) + { + while (IssuanceTimes.Count > 0 && IssuanceTimes.Peek() < threshold) + { + IssuanceTimes.Dequeue(); + } + } + } +} diff --git a/src/StellaOps.Auth.Security/Dpop/RedisDpopNonceStore.cs b/src/StellaOps.Auth.Security/Dpop/RedisDpopNonceStore.cs new file mode 100644 index 00000000..66b7c99b --- /dev/null +++ b/src/StellaOps.Auth.Security/Dpop/RedisDpopNonceStore.cs @@ -0,0 +1,138 @@ +using System; +using System.Globalization; +using System.Threading; +using System.Threading.Tasks; +using StackExchange.Redis; + +namespace StellaOps.Auth.Security.Dpop; + +/// +/// Redis-backed implementation of that supports multi-node deployments. +/// +public sealed class RedisDpopNonceStore : IDpopNonceStore +{ + private const string ConsumeScript = @" +local value = redis.call('GET', KEYS[1]) +if value ~= false and value == ARGV[1] then + redis.call('DEL', KEYS[1]) + return 1 +end +return 0"; + + private readonly IConnectionMultiplexer connection; + private readonly TimeProvider timeProvider; + + public RedisDpopNonceStore(IConnectionMultiplexer connection, TimeProvider? timeProvider = null) + { + this.connection = connection ?? throw new ArgumentNullException(nameof(connection)); + this.timeProvider = timeProvider ?? TimeProvider.System; + } + + public async ValueTask IssueAsync( + string audience, + string clientId, + string keyThumbprint, + TimeSpan ttl, + int maxIssuancePerMinute, + CancellationToken cancellationToken = default) + { + ArgumentException.ThrowIfNullOrWhiteSpace(audience); + ArgumentException.ThrowIfNullOrWhiteSpace(clientId); + ArgumentException.ThrowIfNullOrWhiteSpace(keyThumbprint); + + if (ttl <= TimeSpan.Zero) + { + throw new ArgumentOutOfRangeException(nameof(ttl), "Nonce TTL must be greater than zero."); + } + + if (maxIssuancePerMinute < 1) + { + throw new ArgumentOutOfRangeException(nameof(maxIssuancePerMinute), "Max issuance per minute must be at least 1."); + } + + cancellationToken.ThrowIfCancellationRequested(); + + var database = connection.GetDatabase(); + var issuedAt = timeProvider.GetUtcNow(); + + var baseKey = DpopNonceUtilities.ComputeStorageKey(audience, clientId, keyThumbprint); + var nonceKey = (RedisKey)baseKey; + var metadataKey = (RedisKey)(baseKey + ":meta"); + var rateKey = (RedisKey)(baseKey + ":rate"); + + var rateCount = await database.StringIncrementAsync(rateKey, flags: CommandFlags.DemandMaster).ConfigureAwait(false); + if (rateCount == 1) + { + await database.KeyExpireAsync(rateKey, TimeSpan.FromMinutes(1), CommandFlags.DemandMaster).ConfigureAwait(false); + } + + if (rateCount > maxIssuancePerMinute) + { + return DpopNonceIssueResult.RateLimited("rate_limited"); + } + + var nonce = DpopNonceUtilities.GenerateNonce(); + var hash = (RedisValue)DpopNonceUtilities.EncodeHash(DpopNonceUtilities.ComputeNonceHash(nonce)); + var expiresAt = issuedAt + ttl; + + await database.StringSetAsync(nonceKey, hash, ttl, When.Always, CommandFlags.DemandMaster).ConfigureAwait(false); + var metadataValue = FormattableString.Invariant($"{issuedAt.UtcTicks}|{ttl.Ticks}"); + await database.StringSetAsync(metadataKey, metadataValue, ttl, When.Always, CommandFlags.DemandMaster).ConfigureAwait(false); + + return DpopNonceIssueResult.Success(nonce, expiresAt); + } + + public async ValueTask TryConsumeAsync( + string nonce, + string audience, + string clientId, + string keyThumbprint, + CancellationToken cancellationToken = default) + { + ArgumentException.ThrowIfNullOrWhiteSpace(nonce); + ArgumentException.ThrowIfNullOrWhiteSpace(audience); + ArgumentException.ThrowIfNullOrWhiteSpace(clientId); + ArgumentException.ThrowIfNullOrWhiteSpace(keyThumbprint); + + cancellationToken.ThrowIfCancellationRequested(); + + var database = connection.GetDatabase(); + + var baseKey = DpopNonceUtilities.ComputeStorageKey(audience, clientId, keyThumbprint); + var nonceKey = (RedisKey)baseKey; + var metadataKey = (RedisKey)(baseKey + ":meta"); + var hash = (RedisValue)DpopNonceUtilities.EncodeHash(DpopNonceUtilities.ComputeNonceHash(nonce)); + + var rawResult = await database.ScriptEvaluateAsync( + ConsumeScript, + new[] { nonceKey }, + new RedisValue[] { hash }).ConfigureAwait(false); + + if (rawResult.IsNull || (long)rawResult != 1) + { + return DpopNonceConsumeResult.NotFound(); + } + + var metadata = await database.StringGetAsync(metadataKey).ConfigureAwait(false); + await database.KeyDeleteAsync(metadataKey, CommandFlags.DemandMaster).ConfigureAwait(false); + + if (!metadata.IsNull) + { + var parts = metadata.ToString() + .Split('|', StringSplitOptions.RemoveEmptyEntries | StringSplitOptions.TrimEntries); + + if (parts.Length == 2 && + long.TryParse(parts[0], NumberStyles.Integer, CultureInfo.InvariantCulture, out var issuedTicks) && + long.TryParse(parts[1], NumberStyles.Integer, CultureInfo.InvariantCulture, out var ttlTicks)) + { + var issuedAt = new DateTimeOffset(issuedTicks, TimeSpan.Zero); + var expiresAt = issuedAt + TimeSpan.FromTicks(ttlTicks); + return expiresAt <= timeProvider.GetUtcNow() + ? DpopNonceConsumeResult.Expired(issuedAt, expiresAt) + : DpopNonceConsumeResult.Success(issuedAt, expiresAt); + } + } + + return DpopNonceConsumeResult.Success(timeProvider.GetUtcNow(), timeProvider.GetUtcNow()); + } +} diff --git a/src/StellaOps.Auth.Security/StellaOps.Auth.Security.csproj b/src/StellaOps.Auth.Security/StellaOps.Auth.Security.csproj index bb6f8314..990003f8 100644 --- a/src/StellaOps.Auth.Security/StellaOps.Auth.Security.csproj +++ b/src/StellaOps.Auth.Security/StellaOps.Auth.Security.csproj @@ -29,6 +29,7 @@ + diff --git a/src/StellaOps.Authority/StellaOps.Authority.Plugin.Standard.Tests/StandardClientProvisioningStoreTests.cs b/src/StellaOps.Authority/StellaOps.Authority.Plugin.Standard.Tests/StandardClientProvisioningStoreTests.cs index ec441ebc..a7017fef 100644 --- a/src/StellaOps.Authority/StellaOps.Authority.Plugin.Standard.Tests/StandardClientProvisioningStoreTests.cs +++ b/src/StellaOps.Authority/StellaOps.Authority.Plugin.Standard.Tests/StandardClientProvisioningStoreTests.cs @@ -1,4 +1,6 @@ +using System; using System.Collections.Generic; +using System.Linq; using System.Threading; using System.Threading.Tasks; using MongoDB.Driver; @@ -43,6 +45,74 @@ public class StandardClientProvisioningStoreTests Assert.Contains("scopeA", descriptor.AllowedScopes); } + [Fact] + public async Task CreateOrUpdateAsync_StoresAudiences() + { + var store = new TrackingClientStore(); + var revocations = new TrackingRevocationStore(); + var provisioning = new StandardClientProvisioningStore("standard", store, revocations, TimeProvider.System); + + var registration = new AuthorityClientRegistration( + clientId: "signer", + confidential: false, + displayName: "Signer", + clientSecret: null, + allowedGrantTypes: new[] { "client_credentials" }, + allowedScopes: new[] { "signer.sign" }, + allowedAudiences: new[] { "attestor", "signer" }); + + var result = await provisioning.CreateOrUpdateAsync(registration, CancellationToken.None); + + Assert.True(result.Succeeded); + var document = Assert.Contains("signer", store.Documents); + Assert.Equal("attestor signer", document.Value.Properties[AuthorityClientMetadataKeys.Audiences]); + + var descriptor = await provisioning.FindByClientIdAsync("signer", CancellationToken.None); + Assert.NotNull(descriptor); + Assert.Equal(new[] { "attestor", "signer" }, descriptor!.Audiences.OrderBy(value => value, StringComparer.Ordinal)); + } + + [Fact] + public async Task CreateOrUpdateAsync_MapsCertificateBindings() + { + var store = new TrackingClientStore(); + var revocations = new TrackingRevocationStore(); + var provisioning = new StandardClientProvisioningStore("standard", store, revocations, TimeProvider.System); + + var bindingRegistration = new AuthorityClientCertificateBindingRegistration( + thumbprint: "aa:bb:cc:dd", + serialNumber: "01ff", + subject: "CN=mtls-client", + issuer: "CN=test-ca", + subjectAlternativeNames: new[] { "client.mtls.test", "spiffe://client" }, + notBefore: DateTimeOffset.UtcNow.AddMinutes(-5), + notAfter: DateTimeOffset.UtcNow.AddHours(1), + label: "primary"); + + var registration = new AuthorityClientRegistration( + clientId: "mtls-client", + confidential: true, + displayName: "MTLS Client", + clientSecret: "secret", + allowedGrantTypes: new[] { "client_credentials" }, + allowedScopes: new[] { "signer.sign" }, + allowedAudiences: new[] { "signer" }, + certificateBindings: new[] { bindingRegistration }); + + await provisioning.CreateOrUpdateAsync(registration, CancellationToken.None); + + var document = Assert.Contains("mtls-client", store.Documents).Value; + var binding = Assert.Single(document.CertificateBindings); + Assert.Equal("AABBCCDD", binding.Thumbprint); + Assert.Equal("01ff", binding.SerialNumber); + Assert.Equal("CN=mtls-client", binding.Subject); + Assert.Equal("CN=test-ca", binding.Issuer); + Assert.Equal(new[] { "client.mtls.test", "spiffe://client" }, binding.SubjectAlternativeNames); + Assert.Equal(bindingRegistration.NotBefore, binding.NotBefore); + Assert.Equal(bindingRegistration.NotAfter, binding.NotAfter); + Assert.Equal("primary", binding.Label); + } + private sealed class TrackingClientStore : IAuthorityClientStore { public Dictionary Documents { get; } = new(StringComparer.OrdinalIgnoreCase); diff --git a/src/StellaOps.Authority/StellaOps.Authority.Plugin.Standard/Storage/StandardClientProvisioningStore.cs b/src/StellaOps.Authority/StellaOps.Authority.Plugin.Standard/Storage/StandardClientProvisioningStore.cs index 6765cae6..ea7b4ccc 100644 --- a/src/StellaOps.Authority/StellaOps.Authority.Plugin.Standard/Storage/StandardClientProvisioningStore.cs +++ b/src/StellaOps.Authority/StellaOps.Authority.Plugin.Standard/Storage/StandardClientProvisioningStore.cs @@ -50,11 +50,21 @@ internal sealed class StandardClientProvisioningStore : IClientProvisioningStore document.RedirectUris = registration.RedirectUris.Select(static uri => uri.ToString()).ToList(); document.PostLogoutRedirectUris = registration.PostLogoutRedirectUris.Select(static uri => uri.ToString()).ToList(); - document.Properties[AuthorityClientMetadataKeys.AllowedGrantTypes] = string.Join(" ", registration.AllowedGrantTypes); - document.Properties[AuthorityClientMetadataKeys.AllowedScopes] = string.Join(" ", registration.AllowedScopes); + document.Properties[AuthorityClientMetadataKeys.AllowedGrantTypes] = JoinValues(registration.AllowedGrantTypes); + document.Properties[AuthorityClientMetadataKeys.AllowedScopes] = JoinValues(registration.AllowedScopes); + document.Properties[AuthorityClientMetadataKeys.Audiences] = JoinValues(registration.AllowedAudiences); document.Properties[AuthorityClientMetadataKeys.RedirectUris] = string.Join(" ", document.RedirectUris); document.Properties[AuthorityClientMetadataKeys.PostLogoutRedirectUris] = string.Join(" ", document.PostLogoutRedirectUris); + if (registration.CertificateBindings is not null) + { + var now = clock.GetUtcNow(); + document.CertificateBindings = registration.CertificateBindings + .Select(binding => MapCertificateBinding(binding, now)) + .OrderBy(binding => binding.Thumbprint, StringComparer.Ordinal) + .ToList(); + } + foreach (var (key, value) in registration.Properties) { document.Properties[key] = value; @@ -142,12 +152,15 @@ internal sealed class StandardClientProvisioningStore : IClientProvisioningStore .Cast() .ToArray(); + var audiences = Split(document.Properties, AuthorityClientMetadataKeys.Audiences); + return new AuthorityClientDescriptor( document.ClientId, document.DisplayName, string.Equals(document.ClientType, "confidential", StringComparison.OrdinalIgnoreCase), allowedGrantTypes, allowedScopes, + audiences, redirectUris, postLogoutUris, document.Properties); @@ -163,6 +176,47 @@ internal sealed class StandardClientProvisioningStore : IClientProvisioningStore return value.Split(' ', StringSplitOptions.RemoveEmptyEntries | StringSplitOptions.TrimEntries); } + private static string JoinValues(IReadOnlyCollection values) + { + if (values is null || values.Count == 0) + { + return string.Empty; + } + + return string.Join( + " ", + values + .Where(static value => !string.IsNullOrWhiteSpace(value)) + .Select(static value => value.Trim()) + .OrderBy(static value => value, StringComparer.Ordinal)); + } + + private static AuthorityClientCertificateBinding MapCertificateBinding( + AuthorityClientCertificateBindingRegistration registration, + DateTimeOffset now) + { + var subjectAlternativeNames = registration.SubjectAlternativeNames.Count == 0 + ? new List() + : registration.SubjectAlternativeNames + .Select(name => name.Trim()) + .OrderBy(name => name, StringComparer.OrdinalIgnoreCase) + .ToList(); + + return new AuthorityClientCertificateBinding + { + Thumbprint = registration.Thumbprint, + SerialNumber = registration.SerialNumber, + Subject = registration.Subject, + Issuer = registration.Issuer, + SubjectAlternativeNames = subjectAlternativeNames, + NotBefore = registration.NotBefore, + NotAfter = registration.NotAfter, + Label = registration.Label, + CreatedAt = now, + UpdatedAt = now + }; + } + private static string? NormalizeSenderConstraint(string? value) { if (string.IsNullOrWhiteSpace(value)) diff --git a/src/StellaOps.Authority/StellaOps.Authority.Plugin.Standard/TASKS.md b/src/StellaOps.Authority/StellaOps.Authority.Plugin.Standard/TASKS.md index 6f7190fa..a4caeba9 100644 --- a/src/StellaOps.Authority/StellaOps.Authority.Plugin.Standard/TASKS.md +++ b/src/StellaOps.Authority/StellaOps.Authority.Plugin.Standard/TASKS.md @@ -5,10 +5,10 @@ | PLG6.DOC | DONE (2025-10-11) | BE-Auth Plugin, Docs Guild | PLG1–PLG5 | Final polish + diagrams for plugin developer guide (AUTHPLUG-DOCS-01-001). | Docs team delivers copy-edit + exported diagrams; PR merged. | | SEC1.PLG | DONE (2025-10-11) | Security Guild, BE-Auth Plugin | SEC1.A (StellaOps.Cryptography) | Swap Standard plugin hashing to Argon2id via `StellaOps.Cryptography` abstractions; keep PBKDF2 verification for legacy. | ✅ `StandardUserCredentialStore` uses `ICryptoProvider` to hash/check; ✅ Transparent rehash on success; ✅ Unit tests cover tamper + legacy rehash. | | SEC1.OPT | DONE (2025-10-11) | Security Guild | SEC1.PLG | Expose password hashing knobs in `StandardPluginOptions` (`memoryKiB`, `iterations`, `parallelism`, `algorithm`) with validation. | ✅ Options bound from YAML; ✅ Invalid configs throw; ✅ Docs include tuning guidance. | -| SEC2.PLG | DOING (2025-10-14) | Security Guild, Storage Guild | SEC2.A (audit contract) | Emit audit events from password verification outcomes and persist via `IAuthorityLoginAttemptStore`. | ✅ Serilog events enriched with subject/client/IP/outcome; ✅ Mongo records written per attempt; ✅ Tests assert success/lockout/failure cases. | -| SEC3.PLG | DOING (2025-10-14) | Security Guild, BE-Auth Plugin | CORE8, SEC3.A (rate limiter) | Ensure lockout responses and rate-limit metadata flow through plugin logs/events (include retry-after). | ✅ Audit record includes retry-after; ✅ Tests confirm lockout + limiter interplay. | +| SEC2.PLG | DOING (2025-10-14) | Security Guild, Storage Guild | SEC2.A (audit contract) | Emit audit events from password verification outcomes and persist via `IAuthorityLoginAttemptStore`.
⏳ Awaiting AUTH-DPOP-11-001 / AUTH-MTLS-11-002 / PLUGIN-DI-08-001 completion to unlock Wave 0B verification paths. | ✅ Serilog events enriched with subject/client/IP/outcome; ✅ Mongo records written per attempt; ✅ Tests assert success/lockout/failure cases. | +| SEC3.PLG | DOING (2025-10-14) | Security Guild, BE-Auth Plugin | CORE8, SEC3.A (rate limiter) | Ensure lockout responses and rate-limit metadata flow through plugin logs/events (include retry-after).
⏳ Pending AUTH-DPOP-11-001 / AUTH-MTLS-11-002 / PLUGIN-DI-08-001 so limiter telemetry contract matches final authority surface. | ✅ Audit record includes retry-after; ✅ Tests confirm lockout + limiter interplay. | | SEC4.PLG | DONE (2025-10-12) | Security Guild | SEC4.A (revocation schema) | Provide plugin hooks so revoked users/clients write reasons for revocation bundle export. | ✅ Revocation exporter consumes plugin data; ✅ Tests cover revoked user/client output. | -| SEC5.PLG | DOING (2025-10-14) | Security Guild | SEC5.A (threat model) | Address plugin-specific mitigations (bootstrap user handling, password policy docs) in threat model backlog. | ✅ Threat model lists plugin attack surfaces; ✅ Mitigation items filed. | +| SEC5.PLG | DOING (2025-10-14) | Security Guild | SEC5.A (threat model) | Address plugin-specific mitigations (bootstrap user handling, password policy docs) in threat model backlog.
⏳ Final documentation depends on AUTH-DPOP-11-001 / AUTH-MTLS-11-002 / PLUGIN-DI-08-001 outcomes. | ✅ Threat model lists plugin attack surfaces; ✅ Mitigation items filed. | | PLG4-6.CAPABILITIES | BLOCKED (2025-10-12) | BE-Auth Plugin, Docs Guild | PLG1–PLG3 | Finalise capability metadata exposure, config validation, and developer guide updates; remaining action is Docs polish/diagram export. | ✅ Capability metadata + validation merged; ✅ Plugin guide updated with final copy & diagrams; ✅ Release notes mention new toggles.
⛔ Blocked awaiting Authority rate-limiter stream (CORE8/SEC3) to resume so doc updates reflect final limiter behaviour. | | PLG7.RFC | REVIEW | BE-Auth Plugin, Security Guild | PLG4 | Socialize LDAP plugin RFC (`docs/rfcs/authority-plugin-ldap.md`) and capture guild feedback. | ✅ Guild review sign-off recorded; ✅ Follow-up issues filed in module boards. | | PLG6.DIAGRAM | TODO | Docs Guild | PLG6.DOC | Export final sequence/component diagrams for the developer guide and add offline-friendly assets under `docs/assets/authority`. | ✅ Mermaid sources committed; ✅ Rendered SVG/PNG linked from Section 2 + Section 9; ✅ Docs build preview shared with Plugin + Docs guilds. | @@ -16,3 +16,5 @@ > Update statuses to DOING/DONE/BLOCKED as you make progress. Always run `dotnet test` for touched projects before marking DONE. > Remark (2025-10-13, PLG6.DOC/PLG6.DIAGRAM): Security Guild delivered `docs/security/rate-limits.md`; Docs team can lift Section 3 (tuning table + alerts) into the developer guide diagrams when rendering assets. + +> Check-in (2025-10-19): Wave 0A dependencies (AUTH-DPOP-11-001, AUTH-MTLS-11-002, PLUGIN-DI-08-001) still open, so SEC2/SEC3/SEC5 remain in progress without new scope until upstream limiter updates land. diff --git a/src/StellaOps.Authority/StellaOps.Authority.Plugins.Abstractions/AuthorityClientMetadataKeys.cs b/src/StellaOps.Authority/StellaOps.Authority.Plugins.Abstractions/AuthorityClientMetadataKeys.cs index 69e7660d..b29b7c77 100644 --- a/src/StellaOps.Authority/StellaOps.Authority.Plugins.Abstractions/AuthorityClientMetadataKeys.cs +++ b/src/StellaOps.Authority/StellaOps.Authority.Plugins.Abstractions/AuthorityClientMetadataKeys.cs @@ -7,6 +7,7 @@ public static class AuthorityClientMetadataKeys { public const string AllowedGrantTypes = "allowedGrantTypes"; public const string AllowedScopes = "allowedScopes"; + public const string Audiences = "audiences"; public const string RedirectUris = "redirectUris"; public const string PostLogoutRedirectUris = "postLogoutRedirectUris"; public const string SenderConstraint = "senderConstraint"; diff --git a/src/StellaOps.Authority/StellaOps.Authority.Plugins.Abstractions/IdentityProviderContracts.cs b/src/StellaOps.Authority/StellaOps.Authority.Plugins.Abstractions/IdentityProviderContracts.cs index ae35bdc2..3f387e6f 100644 --- a/src/StellaOps.Authority/StellaOps.Authority.Plugins.Abstractions/IdentityProviderContracts.cs +++ b/src/StellaOps.Authority/StellaOps.Authority.Plugins.Abstractions/IdentityProviderContracts.cs @@ -632,15 +632,13 @@ public sealed class AuthorityClaimsEnrichmentContext /// public sealed record AuthorityClientDescriptor { - /// - /// Initialises a new client descriptor. - /// public AuthorityClientDescriptor( string clientId, string? displayName, bool confidential, IReadOnlyCollection? allowedGrantTypes = null, IReadOnlyCollection? allowedScopes = null, + IReadOnlyCollection? allowedAudiences = null, IReadOnlyCollection? redirectUris = null, IReadOnlyCollection? postLogoutRedirectUris = null, IReadOnlyDictionary? properties = null) @@ -648,8 +646,9 @@ public sealed record AuthorityClientDescriptor ClientId = ValidateRequired(clientId, nameof(clientId)); DisplayName = displayName; Confidential = confidential; - AllowedGrantTypes = allowedGrantTypes is null ? Array.Empty() : allowedGrantTypes.ToArray(); - AllowedScopes = allowedScopes is null ? Array.Empty() : allowedScopes.ToArray(); + AllowedGrantTypes = Normalize(allowedGrantTypes); + AllowedScopes = Normalize(allowedScopes); + AllowedAudiences = Normalize(allowedAudiences); RedirectUris = redirectUris is null ? Array.Empty() : redirectUris.ToArray(); PostLogoutRedirectUris = postLogoutRedirectUris is null ? Array.Empty() : postLogoutRedirectUris.ToArray(); Properties = properties is null @@ -657,60 +656,87 @@ public sealed record AuthorityClientDescriptor : new Dictionary(properties, StringComparer.OrdinalIgnoreCase); } - /// - /// Unique client identifier. - /// public string ClientId { get; } - - /// - /// Optional display name. - /// public string? DisplayName { get; } - - /// - /// Indicates whether the client is confidential (requires secret). - /// public bool Confidential { get; } - - /// - /// Permitted OAuth grant types. - /// public IReadOnlyCollection AllowedGrantTypes { get; } - - /// - /// Permitted scopes. - /// public IReadOnlyCollection AllowedScopes { get; } - - /// - /// Registered redirect URIs. - /// + public IReadOnlyCollection AllowedAudiences { get; } public IReadOnlyCollection RedirectUris { get; } - - /// - /// Registered post-logout redirect URIs. - /// public IReadOnlyCollection PostLogoutRedirectUris { get; } - - /// - /// Additional plugin-defined metadata. - /// public IReadOnlyDictionary Properties { get; } + private static IReadOnlyCollection Normalize(IReadOnlyCollection? values) + => values is null || values.Count == 0 + ? Array.Empty() + : values + .Where(value => !string.IsNullOrWhiteSpace(value)) + .Select(value => value.Trim()) + .Distinct(StringComparer.Ordinal) + .ToArray(); + private static string ValidateRequired(string value, string paramName) => string.IsNullOrWhiteSpace(value) ? throw new ArgumentException("Value cannot be null or whitespace.", paramName) : value; } -/// -/// Client registration payload used when provisioning clients through plugins. -/// +public sealed record AuthorityClientCertificateBindingRegistration +{ + public AuthorityClientCertificateBindingRegistration( + string thumbprint, + string? serialNumber = null, + string? subject = null, + string? issuer = null, + IReadOnlyCollection? subjectAlternativeNames = null, + DateTimeOffset? notBefore = null, + DateTimeOffset? notAfter = null, + string? label = null) + { + Thumbprint = NormalizeThumbprint(thumbprint); + SerialNumber = Normalize(serialNumber); + Subject = Normalize(subject); + Issuer = Normalize(issuer); + SubjectAlternativeNames = subjectAlternativeNames is null || subjectAlternativeNames.Count == 0 + ? Array.Empty() + : subjectAlternativeNames + .Where(value => !string.IsNullOrWhiteSpace(value)) + .Select(value => value.Trim()) + .Distinct(StringComparer.OrdinalIgnoreCase) + .ToArray(); + NotBefore = notBefore; + NotAfter = notAfter; + Label = Normalize(label); + } + + public string Thumbprint { get; } + public string? SerialNumber { get; } + public string? Subject { get; } + public string? Issuer { get; } + public IReadOnlyCollection SubjectAlternativeNames { get; } + public DateTimeOffset? NotBefore { get; } + public DateTimeOffset? NotAfter { get; } + public string? Label { get; } + + private static string NormalizeThumbprint(string value) + { + if (string.IsNullOrWhiteSpace(value)) + { + throw new ArgumentException("Thumbprint is required.", nameof(value)); + } + + return value + .Replace(":", string.Empty, StringComparison.Ordinal) + .Replace(" ", string.Empty, StringComparison.Ordinal) + .ToUpperInvariant(); + } + + private static string? Normalize(string? value) + => string.IsNullOrWhiteSpace(value) ? null : value.Trim(); +} + public sealed record AuthorityClientRegistration { - /// - /// Initialises a new registration. - /// public AuthorityClientRegistration( string clientId, bool confidential, @@ -718,9 +744,11 @@ public sealed record AuthorityClientRegistration string? clientSecret, IReadOnlyCollection? allowedGrantTypes = null, IReadOnlyCollection? allowedScopes = null, + IReadOnlyCollection? allowedAudiences = null, IReadOnlyCollection? redirectUris = null, IReadOnlyCollection? postLogoutRedirectUris = null, - IReadOnlyDictionary? properties = null) + IReadOnlyDictionary? properties = null, + IReadOnlyCollection? certificateBindings = null) { ClientId = ValidateRequired(clientId, nameof(clientId)); Confidential = confidential; @@ -728,65 +756,42 @@ public sealed record AuthorityClientRegistration ClientSecret = confidential ? ValidateRequired(clientSecret ?? string.Empty, nameof(clientSecret)) : clientSecret; - AllowedGrantTypes = allowedGrantTypes is null ? Array.Empty() : allowedGrantTypes.ToArray(); - AllowedScopes = allowedScopes is null ? Array.Empty() : allowedScopes.ToArray(); + AllowedGrantTypes = Normalize(allowedGrantTypes); + AllowedScopes = Normalize(allowedScopes); + AllowedAudiences = Normalize(allowedAudiences); RedirectUris = redirectUris is null ? Array.Empty() : redirectUris.ToArray(); PostLogoutRedirectUris = postLogoutRedirectUris is null ? Array.Empty() : postLogoutRedirectUris.ToArray(); Properties = properties is null ? new Dictionary(StringComparer.OrdinalIgnoreCase) : new Dictionary(properties, StringComparer.OrdinalIgnoreCase); + CertificateBindings = certificateBindings is null + ? Array.Empty() + : certificateBindings.ToArray(); } - /// - /// Unique client identifier. - /// public string ClientId { get; } - - /// - /// Indicates whether the client is confidential (requires secret handling). - /// public bool Confidential { get; } - - /// - /// Optional display name. - /// public string? DisplayName { get; } - - /// - /// Optional raw client secret (hashed by the plugin for storage). - /// public string? ClientSecret { get; init; } - - /// - /// Grant types to enable. - /// public IReadOnlyCollection AllowedGrantTypes { get; } - - /// - /// Scopes assigned to the client. - /// public IReadOnlyCollection AllowedScopes { get; } - - /// - /// Redirect URIs permitted for the client. - /// + public IReadOnlyCollection AllowedAudiences { get; } public IReadOnlyCollection RedirectUris { get; } - - /// - /// Post-logout redirect URIs. - /// public IReadOnlyCollection PostLogoutRedirectUris { get; } - - /// - /// Additional metadata for the plugin. - /// public IReadOnlyDictionary Properties { get; } + public IReadOnlyCollection CertificateBindings { get; } - /// - /// Creates a copy of the registration with the provided client secret. - /// public AuthorityClientRegistration WithClientSecret(string? clientSecret) - => new(ClientId, Confidential, DisplayName, clientSecret, AllowedGrantTypes, AllowedScopes, RedirectUris, PostLogoutRedirectUris, Properties); + => new(ClientId, Confidential, DisplayName, clientSecret, AllowedGrantTypes, AllowedScopes, AllowedAudiences, RedirectUris, PostLogoutRedirectUris, Properties, CertificateBindings); + + private static IReadOnlyCollection Normalize(IReadOnlyCollection? values) + => values is null || values.Count == 0 + ? Array.Empty() + : values + .Where(value => !string.IsNullOrWhiteSpace(value)) + .Select(value => value.Trim()) + .Distinct(StringComparer.Ordinal) + .ToArray(); private static string ValidateRequired(string value, string paramName) => string.IsNullOrWhiteSpace(value) diff --git a/src/StellaOps.Authority/StellaOps.Authority.Storage.Mongo/Documents/AuthorityTokenDocument.cs b/src/StellaOps.Authority/StellaOps.Authority.Storage.Mongo/Documents/AuthorityTokenDocument.cs index fd5156b5..79df9ac0 100644 --- a/src/StellaOps.Authority/StellaOps.Authority.Storage.Mongo/Documents/AuthorityTokenDocument.cs +++ b/src/StellaOps.Authority/StellaOps.Authority.Storage.Mongo/Documents/AuthorityTokenDocument.cs @@ -62,6 +62,18 @@ public sealed class AuthorityTokenDocument [BsonIgnoreIfNull] public string? RevokedReasonDescription { get; set; } + [BsonElement("senderConstraint")] + [BsonIgnoreIfNull] + public string? SenderConstraint { get; set; } + + [BsonElement("senderKeyThumbprint")] + [BsonIgnoreIfNull] + public string? SenderKeyThumbprint { get; set; } + + [BsonElement("senderNonce")] + [BsonIgnoreIfNull] + public string? SenderNonce { get; set; } + [BsonElement("devices")] [BsonIgnoreIfNull] diff --git a/src/StellaOps.Authority/StellaOps.Authority.Storage.Mongo/Initialization/AuthorityTokenCollectionInitializer.cs b/src/StellaOps.Authority/StellaOps.Authority.Storage.Mongo/Initialization/AuthorityTokenCollectionInitializer.cs index eca61f40..a9367593 100644 --- a/src/StellaOps.Authority/StellaOps.Authority.Storage.Mongo/Initialization/AuthorityTokenCollectionInitializer.cs +++ b/src/StellaOps.Authority/StellaOps.Authority.Storage.Mongo/Initialization/AuthorityTokenCollectionInitializer.cs @@ -27,7 +27,13 @@ internal sealed class AuthorityTokenCollectionInitializer : IAuthorityCollection Builders.IndexKeys .Ascending(t => t.Status) .Ascending(t => t.RevokedAt), - new CreateIndexOptions { Name = "token_status_revokedAt" }) + new CreateIndexOptions { Name = "token_status_revokedAt" }), + new( + Builders.IndexKeys.Ascending(t => t.SenderConstraint), + new CreateIndexOptions { Name = "token_sender_constraint", Sparse = true }), + new( + Builders.IndexKeys.Ascending(t => t.SenderKeyThumbprint), + new CreateIndexOptions { Name = "token_sender_thumbprint", Sparse = true }) }; var expirationFilter = Builders.Filter.Exists(t => t.ExpiresAt, true); diff --git a/src/StellaOps.Authority/StellaOps.Authority.Tests/OpenIddict/ClientCredentialsAndTokenHandlersTests.cs b/src/StellaOps.Authority/StellaOps.Authority.Tests/OpenIddict/ClientCredentialsAndTokenHandlersTests.cs index dbd51bcf..03dd02f9 100644 --- a/src/StellaOps.Authority/StellaOps.Authority.Tests/OpenIddict/ClientCredentialsAndTokenHandlersTests.cs +++ b/src/StellaOps.Authority/StellaOps.Authority.Tests/OpenIddict/ClientCredentialsAndTokenHandlersTests.cs @@ -1,9 +1,21 @@ using System; using System.Collections.Generic; using System.Diagnostics; +using System.IdentityModel.Tokens.Jwt; using System.Security.Claims; +using System.Security.Cryptography; +using System.Security.Cryptography.X509Certificates; +using System.Text.Json; +using Microsoft.AspNetCore.Http; +using Microsoft.AspNetCore.Http.Extensions; using Microsoft.Extensions.Configuration; using Microsoft.Extensions.Logging.Abstractions; +using Microsoft.Extensions.Options; +using Microsoft.Extensions.Primitives; +using Microsoft.IdentityModel.Tokens; +using StellaOps.Configuration; +using StellaOps.Authority.Security; +using StellaOps.Auth.Security.Dpop; using OpenIddict.Abstractions; using OpenIddict.Extensions; using OpenIddict.Server; @@ -44,6 +56,8 @@ public class ClientCredentialsHandlersTests new TestAuthEventSink(), new TestRateLimiterMetadataAccessor(), TimeProvider.System, + new NoopCertificateValidator(), + new HttpContextAccessor(), NullLogger.Instance); var transaction = CreateTokenTransaction(clientDocument.ClientId, "s3cr3t!", scope: "jobs:write"); @@ -72,6 +86,8 @@ public class ClientCredentialsHandlersTests new TestAuthEventSink(), new TestRateLimiterMetadataAccessor(), TimeProvider.System, + new NoopCertificateValidator(), + new HttpContextAccessor(), NullLogger.Instance); var transaction = CreateTokenTransaction(clientDocument.ClientId, "s3cr3t!", scope: "jobs:read"); @@ -104,6 +120,8 @@ public class ClientCredentialsHandlersTests sink, new TestRateLimiterMetadataAccessor(), TimeProvider.System, + new NoopCertificateValidator(), + new HttpContextAccessor(), NullLogger.Instance); var transaction = CreateTokenTransaction(clientDocument.ClientId, "s3cr3t!", scope: "jobs:read"); @@ -117,6 +135,315 @@ public class ClientCredentialsHandlersTests string.Equals(property.Value.Value, "unexpected_param", StringComparison.OrdinalIgnoreCase)); } + [Fact] + public async Task ValidateDpopProof_AllowsSenderConstrainedClient() + { + var options = new StellaOpsAuthorityOptions + { + Issuer = new Uri("https://authority.test") + }; + options.Security.SenderConstraints.Dpop.Enabled = true; + options.Security.SenderConstraints.Dpop.Nonce.Enabled = false; + + var clientDocument = CreateClient( + secret: "s3cr3t!", + allowedGrantTypes: "client_credentials", + allowedScopes: "jobs:read"); + clientDocument.SenderConstraint = AuthoritySenderConstraintKinds.Dpop; + clientDocument.Properties[AuthorityClientMetadataKeys.SenderConstraint] = AuthoritySenderConstraintKinds.Dpop; + + using var ecdsa = ECDsa.Create(ECCurve.NamedCurves.nistP256); + var securityKey = new ECDsaSecurityKey(ecdsa) + { + KeyId = Guid.NewGuid().ToString("N") + }; + var jwk = JsonWebKeyConverter.ConvertFromECDsaSecurityKey(securityKey); + var expectedThumbprint = ConvertThumbprintToString(jwk.ComputeJwkThumbprint()); + + var clientStore = new TestClientStore(clientDocument); + var auditSink = new TestAuthEventSink(); + var rateMetadata = new TestRateLimiterMetadataAccessor(); + + var dpopValidator = new DpopProofValidator( + Options.Create(new DpopValidationOptions()), + new InMemoryDpopReplayCache(TimeProvider.System), + TimeProvider.System, + NullLogger.Instance); + + var nonceStore = new InMemoryDpopNonceStore(TimeProvider.System, NullLogger.Instance); + + var dpopHandler = new ValidateDpopProofHandler( + options, + clientStore, + dpopValidator, + nonceStore, + rateMetadata, + auditSink, + TimeProvider.System, + TestActivitySource, + NullLogger.Instance); + + var transaction = CreateTokenTransaction(clientDocument.ClientId, "s3cr3t!", scope: "jobs:read"); + transaction.Options = new OpenIddictServerOptions(); + + var httpContext = new DefaultHttpContext(); + httpContext.Request.Method = "POST"; + httpContext.Request.Scheme = "https"; + httpContext.Request.Host = new HostString("authority.test"); + httpContext.Request.Path = "/token"; + + var now = TimeProvider.System.GetUtcNow(); + var proof = TestHelpers.CreateDpopProof(securityKey, httpContext.Request.Method, httpContext.Request.GetDisplayUrl(), now.ToUnixTimeSeconds()); + httpContext.Request.Headers["DPoP"] = proof; + + transaction.Properties[typeof(HttpContext).FullName!] = httpContext; + + var validateContext = new OpenIddictServerEvents.ValidateTokenRequestContext(transaction); + await dpopHandler.HandleAsync(validateContext); + + Assert.False(validateContext.IsRejected); + + var registry = CreateRegistry(withClientProvisioning: true, clientDescriptor: CreateDescriptor(clientDocument)); + var validateHandler = new ValidateClientCredentialsHandler( + clientStore, + registry, + TestActivitySource, + auditSink, + rateMetadata, + TimeProvider.System, + new NoopCertificateValidator(), + new HttpContextAccessor(), + NullLogger.Instance); + + await validateHandler.HandleAsync(validateContext); + Assert.False(validateContext.IsRejected); + + var tokenStore = new TestTokenStore(); + var sessionAccessor = new NullMongoSessionAccessor(); + var handleHandler = new HandleClientCredentialsHandler( + registry, + tokenStore, + sessionAccessor, + TimeProvider.System, + TestActivitySource, + NullLogger.Instance); + + var handleContext = new OpenIddictServerEvents.HandleTokenRequestContext(transaction); + await handleHandler.HandleAsync(handleContext); + Assert.True(handleContext.IsRequestHandled); + + var persistHandler = new PersistTokensHandler( + tokenStore, + sessionAccessor, + TimeProvider.System, + TestActivitySource, + NullLogger.Instance); + + var signInContext = new OpenIddictServerEvents.ProcessSignInContext(transaction) + { + Principal = handleContext.Principal, + AccessTokenPrincipal = handleContext.Principal + }; + + await persistHandler.HandleAsync(signInContext); + + var confirmationClaim = handleContext.Principal?.GetClaim(AuthorityOpenIddictConstants.ConfirmationClaimType); + Assert.False(string.IsNullOrWhiteSpace(confirmationClaim)); + + using (var confirmationJson = JsonDocument.Parse(confirmationClaim!)) + { + Assert.Equal(expectedThumbprint, confirmationJson.RootElement.GetProperty("jkt").GetString()); + } + + Assert.NotNull(tokenStore.Inserted); + Assert.Equal(AuthoritySenderConstraintKinds.Dpop, tokenStore.Inserted!.SenderConstraint); + Assert.Equal(expectedThumbprint, tokenStore.Inserted!.SenderKeyThumbprint); + } + + [Fact] + public async Task ValidateDpopProof_IssuesNonceChallenge_WhenNonceMissing() + { + var options = new StellaOpsAuthorityOptions + { + Issuer = new Uri("https://authority.test") + }; + options.Security.SenderConstraints.Dpop.Enabled = true; + options.Security.SenderConstraints.Dpop.Nonce.Enabled = true; + options.Security.SenderConstraints.Dpop.Nonce.RequiredAudiences.Clear(); + options.Security.SenderConstraints.Dpop.Nonce.RequiredAudiences.Add("signer"); + options.Signing.ActiveKeyId = "test-key"; + options.Signing.KeyPath = "/tmp/test-key.pem"; + options.Storage.ConnectionString = "mongodb://localhost/test"; + Assert.Contains("signer", options.Security.SenderConstraints.Dpop.Nonce.RequiredAudiences); + + var clientDocument = CreateClient( + secret: "s3cr3t!", + allowedGrantTypes: "client_credentials", + allowedScopes: "jobs:read", + allowedAudiences: "signer"); + clientDocument.SenderConstraint = AuthoritySenderConstraintKinds.Dpop; + clientDocument.Properties[AuthorityClientMetadataKeys.SenderConstraint] = AuthoritySenderConstraintKinds.Dpop; + + using var ecdsa = ECDsa.Create(ECCurve.NamedCurves.nistP256); + var securityKey = new ECDsaSecurityKey(ecdsa) + { + KeyId = Guid.NewGuid().ToString("N") + }; + + var clientStore = new TestClientStore(clientDocument); + var auditSink = new TestAuthEventSink(); + var rateMetadata = new TestRateLimiterMetadataAccessor(); + + var dpopValidator = new DpopProofValidator( + Options.Create(new DpopValidationOptions()), + new InMemoryDpopReplayCache(TimeProvider.System), + TimeProvider.System, + NullLogger.Instance); + + var nonceStore = new InMemoryDpopNonceStore(TimeProvider.System, NullLogger.Instance); + + var dpopHandler = new ValidateDpopProofHandler( + options, + clientStore, + dpopValidator, + nonceStore, + rateMetadata, + auditSink, + TimeProvider.System, + TestActivitySource, + NullLogger.Instance); + + var transaction = CreateTokenTransaction(clientDocument.ClientId, "s3cr3t!", scope: "jobs:read"); + transaction.Options = new OpenIddictServerOptions(); + + var httpContext = new DefaultHttpContext(); + httpContext.Request.Method = "POST"; + httpContext.Request.Scheme = "https"; + httpContext.Request.Host = new HostString("authority.test"); + httpContext.Request.Path = "/token"; + + var now = TimeProvider.System.GetUtcNow(); + var proof = TestHelpers.CreateDpopProof(securityKey, httpContext.Request.Method, httpContext.Request.GetDisplayUrl(), now.ToUnixTimeSeconds()); + httpContext.Request.Headers["DPoP"] = proof; + + transaction.Properties[typeof(HttpContext).FullName!] = httpContext; + + var validateContext = new OpenIddictServerEvents.ValidateTokenRequestContext(transaction); + await dpopHandler.HandleAsync(validateContext); + + Assert.True(validateContext.IsRejected); + Assert.Equal(OpenIddictConstants.Errors.InvalidClient, validateContext.Error); + var authenticateHeader = Assert.Single(httpContext.Response.Headers.Select(header => header) + .Where(header => string.Equals(header.Key, "WWW-Authenticate", StringComparison.OrdinalIgnoreCase))).Value; + Assert.Contains("use_dpop_nonce", authenticateHeader.ToString()); + Assert.True(httpContext.Response.Headers.TryGetValue("DPoP-Nonce", out var nonceValues)); + Assert.False(StringValues.IsNullOrEmpty(nonceValues)); + Assert.Contains(auditSink.Events, record => record.EventType == "authority.dpop.proof.challenge"); + } + + [Fact] + public async Task ValidateClientCredentials_AllowsMtlsClient_WithValidCertificate() + { + var options = new StellaOpsAuthorityOptions + { + Issuer = new Uri("https://authority.test") + }; + options.Security.SenderConstraints.Mtls.Enabled = true; + options.Security.SenderConstraints.Mtls.RequireChainValidation = false; + options.Signing.ActiveKeyId = "test-key"; + options.Signing.KeyPath = "/tmp/test-key.pem"; + options.Storage.ConnectionString = "mongodb://localhost/test"; + + var clientDocument = CreateClient( + secret: "s3cr3t!", + allowedGrantTypes: "client_credentials", + allowedScopes: "jobs:read"); + clientDocument.SenderConstraint = AuthoritySenderConstraintKinds.Mtls; + clientDocument.Properties[AuthorityClientMetadataKeys.SenderConstraint] = AuthoritySenderConstraintKinds.Mtls; + + using var rsa = RSA.Create(2048); + var certificateRequest = new CertificateRequest("CN=mtls-client", rsa, HashAlgorithmName.SHA256, RSASignaturePadding.Pkcs1); + using var certificate = certificateRequest.CreateSelfSigned(DateTimeOffset.UtcNow.AddMinutes(-5), DateTimeOffset.UtcNow.AddHours(1)); + var hexThumbprint = Convert.ToHexString(certificate.GetCertHash(HashAlgorithmName.SHA256)); + clientDocument.CertificateBindings.Add(new AuthorityClientCertificateBinding + { + Thumbprint = hexThumbprint + }); + + var registry = CreateRegistry(withClientProvisioning: true, clientDescriptor: CreateDescriptor(clientDocument)); + var auditSink = new TestAuthEventSink(); + var metadataAccessor = new TestRateLimiterMetadataAccessor(); + var httpContextAccessor = new HttpContextAccessor { HttpContext = new DefaultHttpContext() }; + httpContextAccessor.HttpContext!.Connection.ClientCertificate = certificate; + + var validator = new AuthorityClientCertificateValidator(options, TimeProvider.System, NullLogger.Instance); + + var handler = new ValidateClientCredentialsHandler( + new TestClientStore(clientDocument), + registry, + TestActivitySource, + auditSink, + metadataAccessor, + TimeProvider.System, + validator, + httpContextAccessor, + NullLogger.Instance); + + var transaction = CreateTokenTransaction(clientDocument.ClientId, "s3cr3t!", scope: "jobs:read"); + var context = new OpenIddictServerEvents.ValidateTokenRequestContext(transaction); + + await handler.HandleAsync(context); + + Assert.False(context.IsRejected); + Assert.Equal(AuthoritySenderConstraintKinds.Mtls, context.Transaction.Properties[AuthorityOpenIddictConstants.SenderConstraintProperty]); + + var expectedBase64 = Base64UrlEncoder.Encode(certificate.GetCertHash(HashAlgorithmName.SHA256)); + Assert.Equal(expectedBase64, context.Transaction.Properties[AuthorityOpenIddictConstants.MtlsCertificateThumbprintProperty]); + } + + [Fact] + public async Task ValidateClientCredentials_RejectsMtlsClient_WhenCertificateMissing() + { + var options = new StellaOpsAuthorityOptions + { + Issuer = new Uri("https://authority.test") + }; + options.Security.SenderConstraints.Mtls.Enabled = true; + options.Signing.ActiveKeyId = "test-key"; + options.Signing.KeyPath = "/tmp/test-key.pem"; + options.Storage.ConnectionString = "mongodb://localhost/test"; + + var clientDocument = CreateClient( + secret: "s3cr3t!", + allowedGrantTypes: "client_credentials", + allowedScopes: "jobs:read"); + clientDocument.SenderConstraint = AuthoritySenderConstraintKinds.Mtls; + clientDocument.Properties[AuthorityClientMetadataKeys.SenderConstraint] = AuthoritySenderConstraintKinds.Mtls; + + var registry = CreateRegistry(withClientProvisioning: true, clientDescriptor: CreateDescriptor(clientDocument)); + var httpContextAccessor = new HttpContextAccessor { HttpContext = new DefaultHttpContext() }; + var validator = new AuthorityClientCertificateValidator(options, TimeProvider.System, NullLogger.Instance); + + var handler = new ValidateClientCredentialsHandler( + new TestClientStore(clientDocument), + registry, + TestActivitySource, + new TestAuthEventSink(), + new TestRateLimiterMetadataAccessor(), + TimeProvider.System, + validator, + httpContextAccessor, + NullLogger.Instance); + + var transaction = CreateTokenTransaction(clientDocument.ClientId, "s3cr3t!", scope: "jobs:read"); + var context = new OpenIddictServerEvents.ValidateTokenRequestContext(transaction); + + await handler.HandleAsync(context); + + Assert.True(context.IsRejected); + Assert.Equal(OpenIddictConstants.Errors.InvalidClient, context.Error); + } + [Fact] public async Task HandleClientCredentials_PersistsTokenAndEnrichesClaims() { @@ -124,7 +451,8 @@ public class ClientCredentialsHandlersTests secret: null, clientType: "public", allowedGrantTypes: "client_credentials", - allowedScopes: "jobs:trigger"); + allowedScopes: "jobs:trigger", + allowedAudiences: "signer"); var descriptor = CreateDescriptor(clientDocument); var registry = CreateRegistry(withClientProvisioning: true, clientDescriptor: descriptor); @@ -139,6 +467,8 @@ public class ClientCredentialsHandlersTests authSink, metadataAccessor, TimeProvider.System, + new NoopCertificateValidator(), + new HttpContextAccessor(), NullLogger.Instance); var transaction = CreateTokenTransaction(clientDocument.ClientId, secret: null, scope: "jobs:trigger"); @@ -163,6 +493,7 @@ public class ClientCredentialsHandlersTests Assert.True(context.IsRequestHandled); Assert.NotNull(context.Principal); + Assert.Contains("signer", context.Principal!.GetAudiences()); Assert.Contains(authSink.Events, record => record.EventType == "authority.client_credentials.grant" && record.Outcome == AuthEventOutcome.Success); @@ -285,6 +616,62 @@ public class TokenValidationHandlersTests Assert.Contains(principal.Claims, claim => claim.Type == "enriched" && claim.Value == "true"); } + [Fact] + public async Task ValidateAccessTokenHandler_AddsConfirmationClaim_ForMtlsToken() + { + var tokenDocument = new AuthorityTokenDocument + { + TokenId = "token-mtls", + Status = "valid", + ClientId = "mtls-client", + SenderConstraint = AuthoritySenderConstraintKinds.Mtls, + SenderKeyThumbprint = "thumb-print" + }; + + var tokenStore = new TestTokenStore + { + Inserted = tokenDocument + }; + + var clientDocument = CreateClient(); + var registry = CreateRegistry(withClientProvisioning: false, clientDescriptor: null); + var metadataAccessor = new TestRateLimiterMetadataAccessor(); + var auditSink = new TestAuthEventSink(); + var sessionAccessor = new NullMongoSessionAccessor(); + var handler = new ValidateAccessTokenHandler( + tokenStore, + sessionAccessor, + new TestClientStore(clientDocument), + registry, + metadataAccessor, + auditSink, + TimeProvider.System, + TestActivitySource, + NullLogger.Instance); + + var transaction = new OpenIddictServerTransaction + { + Options = new OpenIddictServerOptions(), + EndpointType = OpenIddictServerEndpointType.Introspection, + Request = new OpenIddictRequest() + }; + + var principal = CreatePrincipal(clientDocument.ClientId, tokenDocument.TokenId, clientDocument.Plugin); + var context = new OpenIddictServerEvents.ValidateTokenContext(transaction) + { + Principal = principal, + TokenId = tokenDocument.TokenId + }; + + await handler.HandleAsync(context); + + Assert.False(context.IsRejected); + var confirmation = context.Principal?.GetClaim(AuthorityOpenIddictConstants.ConfirmationClaimType); + Assert.False(string.IsNullOrWhiteSpace(confirmation)); + using var json = JsonDocument.Parse(confirmation!); + Assert.Equal(tokenDocument.SenderKeyThumbprint, json.RootElement.GetProperty("x5t#S256").GetString()); + } + [Fact] public async Task ValidateAccessTokenHandler_EmitsReplayAudit_WhenStoreDetectsSuspectedReplay() { @@ -358,6 +745,89 @@ public class TokenValidationHandlersTests } } +public class AuthorityClientCertificateValidatorTests +{ + [Fact] + public async Task ValidateAsync_Rejects_WhenSanTypeNotAllowed() + { + var options = new StellaOpsAuthorityOptions + { + Issuer = new Uri("https://authority.test") + }; + options.Security.SenderConstraints.Mtls.Enabled = true; + options.Security.SenderConstraints.Mtls.RequireChainValidation = false; + options.Security.SenderConstraints.Mtls.AllowedSanTypes.Clear(); + options.Security.SenderConstraints.Mtls.AllowedSanTypes.Add("uri"); + options.Signing.ActiveKeyId = "test-key"; + options.Signing.KeyPath = "/tmp/test-key.pem"; + options.Storage.ConnectionString = "mongodb://localhost/test"; + + using var rsa = RSA.Create(2048); + var request = new CertificateRequest("CN=mtls-client", rsa, HashAlgorithmName.SHA256, RSASignaturePadding.Pkcs1); + var sanBuilder = new SubjectAlternativeNameBuilder(); + sanBuilder.AddDnsName("client.mtls.test"); + request.CertificateExtensions.Add(sanBuilder.Build()); + using var certificate = request.CreateSelfSigned(DateTimeOffset.UtcNow.AddMinutes(-5), DateTimeOffset.UtcNow.AddMinutes(5)); + + var clientDocument = CreateClient(); + clientDocument.SenderConstraint = AuthoritySenderConstraintKinds.Mtls; + clientDocument.CertificateBindings.Add(new AuthorityClientCertificateBinding + { + Thumbprint = Convert.ToHexString(certificate.GetCertHash(HashAlgorithmName.SHA256)) + }); + + var httpContext = new DefaultHttpContext(); + httpContext.Connection.ClientCertificate = certificate; + + var validator = new AuthorityClientCertificateValidator(options, TimeProvider.System, NullLogger.Instance); + var result = await validator.ValidateAsync(httpContext, clientDocument, CancellationToken.None); + + Assert.False(result.Succeeded); + Assert.Equal("certificate_san_type", result.Error); + } + + [Fact] + public async Task ValidateAsync_AllowsBindingWithinRotationGrace() + { + var options = new StellaOpsAuthorityOptions + { + Issuer = new Uri("https://authority.test") + }; + options.Security.SenderConstraints.Mtls.Enabled = true; + options.Security.SenderConstraints.Mtls.RequireChainValidation = false; + options.Security.SenderConstraints.Mtls.RotationGrace = TimeSpan.FromMinutes(5); + options.Signing.ActiveKeyId = "test-key"; + options.Signing.KeyPath = "/tmp/test-key.pem"; + options.Storage.ConnectionString = "mongodb://localhost/test"; + + using var rsa = RSA.Create(2048); + var request = new CertificateRequest("CN=mtls-client", rsa, HashAlgorithmName.SHA256, RSASignaturePadding.Pkcs1); + var sanBuilder = new SubjectAlternativeNameBuilder(); + sanBuilder.AddDnsName("client.mtls.test"); + request.CertificateExtensions.Add(sanBuilder.Build()); + using var certificate = request.CreateSelfSigned(DateTimeOffset.UtcNow.AddMinutes(-5), DateTimeOffset.UtcNow.AddMinutes(10)); + + var thumbprint = Convert.ToHexString(certificate.GetCertHash(HashAlgorithmName.SHA256)); + + var clientDocument = CreateClient(); + clientDocument.SenderConstraint = AuthoritySenderConstraintKinds.Mtls; + clientDocument.CertificateBindings.Add(new AuthorityClientCertificateBinding + { + Thumbprint = thumbprint, + NotBefore = TimeProvider.System.GetUtcNow().AddMinutes(2) + }); + + var httpContext = new DefaultHttpContext(); + httpContext.Connection.ClientCertificate = certificate; + + var validator = new AuthorityClientCertificateValidator(options, TimeProvider.System, NullLogger.Instance); + var result = await validator.ValidateAsync(httpContext, clientDocument, CancellationToken.None); + + Assert.True(result.Succeeded); + Assert.Equal(thumbprint, result.HexThumbprint); + } +} + internal sealed class TestClientStore : IAuthorityClientStore { private readonly Dictionary clients = new(StringComparer.OrdinalIgnoreCase); @@ -526,6 +996,19 @@ internal sealed class TestRateLimiterMetadataAccessor : IAuthorityRateLimiterMet public void SetTag(string name, string? value) => metadata.SetTag(name, value); } +internal sealed class NoopCertificateValidator : IAuthorityClientCertificateValidator +{ + public ValueTask ValidateAsync(HttpContext httpContext, AuthorityClientDocument client, CancellationToken cancellationToken) + { + var binding = new AuthorityClientCertificateBinding + { + Thumbprint = "stub" + }; + + return ValueTask.FromResult(AuthorityClientCertificateValidationResult.Success("stub", "stub", binding)); + } +} + internal sealed class NullMongoSessionAccessor : IAuthorityMongoSessionAccessor { public ValueTask GetSessionAsync(CancellationToken cancellationToken = default) @@ -540,9 +1023,10 @@ internal static class TestHelpers string? secret = "s3cr3t!", string clientType = "confidential", string allowedGrantTypes = "client_credentials", - string allowedScopes = "jobs:read") + string allowedScopes = "jobs:read", + string allowedAudiences = "") { - return new AuthorityClientDocument + var document = new AuthorityClientDocument { ClientId = "concelier", ClientType = clientType, @@ -554,12 +1038,20 @@ internal static class TestHelpers [AuthorityClientMetadataKeys.AllowedScopes] = allowedScopes } }; + + if (!string.IsNullOrWhiteSpace(allowedAudiences)) + { + document.Properties[AuthorityClientMetadataKeys.Audiences] = allowedAudiences; + } + + return document; } public static AuthorityClientDescriptor CreateDescriptor(AuthorityClientDocument document) { var allowedGrantTypes = document.Properties.TryGetValue(AuthorityClientMetadataKeys.AllowedGrantTypes, out var grants) ? grants?.Split(' ', StringSplitOptions.RemoveEmptyEntries) : Array.Empty(); var allowedScopes = document.Properties.TryGetValue(AuthorityClientMetadataKeys.AllowedScopes, out var scopes) ? scopes?.Split(' ', StringSplitOptions.RemoveEmptyEntries) : Array.Empty(); + var allowedAudiences = document.Properties.TryGetValue(AuthorityClientMetadataKeys.Audiences, out var audiences) ? audiences?.Split(' ', StringSplitOptions.RemoveEmptyEntries) : Array.Empty(); return new AuthorityClientDescriptor( document.ClientId, @@ -567,6 +1059,7 @@ internal static class TestHelpers confidential: string.Equals(document.ClientType, "confidential", StringComparison.OrdinalIgnoreCase), allowedGrantTypes, allowedScopes, + allowedAudiences, redirectUris: Array.Empty(), postLogoutRedirectUris: Array.Empty(), properties: document.Properties); @@ -638,6 +1131,57 @@ internal static class TestHelpers }; } + public static string ConvertThumbprintToString(object thumbprint) + => thumbprint switch + { + string value => value, + byte[] bytes => Base64UrlEncoder.Encode(bytes), + _ => throw new InvalidOperationException("Unsupported thumbprint representation.") + }; + + public static string CreateDpopProof(ECDsaSecurityKey key, string method, string url, long issuedAt, string? nonce = null) + { + var jwk = JsonWebKeyConverter.ConvertFromECDsaSecurityKey(key); + jwk.KeyId ??= key.KeyId ?? Guid.NewGuid().ToString("N"); + + var signingCredentials = new SigningCredentials(key, SecurityAlgorithms.EcdsaSha256); + var header = new JwtHeader(signingCredentials) + { + ["typ"] = "dpop+jwt", + ["jwk"] = new Dictionary + { + ["kty"] = jwk.Kty, + ["crv"] = jwk.Crv, + ["x"] = jwk.X, + ["y"] = jwk.Y, + ["kid"] = jwk.Kid ?? jwk.KeyId + } + }; + + var payload = new JwtPayload + { + ["htm"] = method.ToUpperInvariant(), + ["htu"] = url, + ["iat"] = issuedAt, + ["jti"] = Guid.NewGuid().ToString("N") + }; + + if (!string.IsNullOrWhiteSpace(nonce)) + { + payload["nonce"] = nonce; + } + + var token = new JwtSecurityToken(header, payload); + return new JwtSecurityTokenHandler().WriteToken(token); + } + + public static X509Certificate2 CreateTestCertificate(string subjectName) + { + using var rsa = RSA.Create(2048); + var request = new CertificateRequest(subjectName, rsa, HashAlgorithmName.SHA256, RSASignaturePadding.Pkcs1); + return request.CreateSelfSigned(DateTimeOffset.UtcNow.AddMinutes(-5), DateTimeOffset.UtcNow.AddHours(1)); + } + public static ClaimsPrincipal CreatePrincipal(string clientId, string tokenId, string provider, string? subject = null) { var identity = new ClaimsIdentity(OpenIddictServerAspNetCoreDefaults.AuthenticationScheme); diff --git a/src/StellaOps.Authority/StellaOps.Authority.Tests/OpenIddict/TokenPersistenceIntegrationTests.cs b/src/StellaOps.Authority/StellaOps.Authority.Tests/OpenIddict/TokenPersistenceIntegrationTests.cs index fb74dd91..63722c92 100644 --- a/src/StellaOps.Authority/StellaOps.Authority.Tests/OpenIddict/TokenPersistenceIntegrationTests.cs +++ b/src/StellaOps.Authority/StellaOps.Authority.Tests/OpenIddict/TokenPersistenceIntegrationTests.cs @@ -1,6 +1,7 @@ using System.Collections.Generic; using System.Diagnostics; using System.Security.Claims; +using Microsoft.AspNetCore.Http; using Microsoft.Extensions.DependencyInjection; using Microsoft.Extensions.Logging.Abstractions; using Microsoft.Extensions.Time.Testing; @@ -20,6 +21,7 @@ using StellaOps.Authority.Storage.Mongo.Sessions; using StellaOps.Authority.Storage.Mongo.Stores; using StellaOps.Concelier.Testing; using StellaOps.Authority.RateLimiting; +using StellaOps.Authority.Security; using StellaOps.Cryptography.Audit; using Xunit; @@ -62,7 +64,7 @@ public sealed class TokenPersistenceIntegrationTests var metadataAccessor = new TestRateLimiterMetadataAccessor(); await using var scope = provider.CreateAsyncScope(); var sessionAccessor = scope.ServiceProvider.GetRequiredService(); - var validateHandler = new ValidateClientCredentialsHandler(clientStore, registry, TestActivitySource, authSink, metadataAccessor, clock, NullLogger.Instance); + var validateHandler = new ValidateClientCredentialsHandler(clientStore, registry, TestActivitySource, authSink, metadataAccessor, clock, new NoopCertificateValidator(), new HttpContextAccessor(), NullLogger.Instance); var handleHandler = new HandleClientCredentialsHandler(registry, tokenStore, sessionAccessor, clock, TestActivitySource, NullLogger.Instance); var persistHandler = new PersistTokensHandler(tokenStore, sessionAccessor, clock, TestActivitySource, NullLogger.Instance); diff --git a/src/StellaOps.Authority/StellaOps.Authority.Tests/StellaOps.Authority.Tests.csproj b/src/StellaOps.Authority/StellaOps.Authority.Tests/StellaOps.Authority.Tests.csproj index 9ed95cc7..893114ef 100644 --- a/src/StellaOps.Authority/StellaOps.Authority.Tests/StellaOps.Authority.Tests.csproj +++ b/src/StellaOps.Authority/StellaOps.Authority.Tests/StellaOps.Authority.Tests.csproj @@ -8,6 +8,7 @@ + diff --git a/src/StellaOps.Authority/StellaOps.Authority/Bootstrap/BootstrapRequests.cs b/src/StellaOps.Authority/StellaOps.Authority/Bootstrap/BootstrapRequests.cs index a6524532..d11af3aa 100644 --- a/src/StellaOps.Authority/StellaOps.Authority/Bootstrap/BootstrapRequests.cs +++ b/src/StellaOps.Authority/StellaOps.Authority/Bootstrap/BootstrapRequests.cs @@ -44,11 +44,15 @@ internal sealed record BootstrapClientRequest public IReadOnlyCollection? AllowedScopes { get; init; } + public IReadOnlyCollection? AllowedAudiences { get; init; } + public IReadOnlyCollection? RedirectUris { get; init; } public IReadOnlyCollection? PostLogoutRedirectUris { get; init; } public IReadOnlyDictionary? Properties { get; init; } + + public IReadOnlyCollection? CertificateBindings { get; init; } } internal sealed record BootstrapInviteRequest @@ -68,6 +72,25 @@ internal sealed record BootstrapInviteRequest public IReadOnlyDictionary? Metadata { get; init; } } +internal sealed record BootstrapClientCertificateBinding +{ + public string Thumbprint { get; init; } = string.Empty; + + public string? SerialNumber { get; init; } + + public string? Subject { get; init; } + + public string? Issuer { get; init; } + + public IReadOnlyCollection? SubjectAlternativeNames { get; init; } + + public DateTimeOffset? NotBefore { get; init; } + + public DateTimeOffset? NotAfter { get; init; } + + public string? Label { get; init; } +} + internal static class BootstrapInviteTypes { public const string User = "user"; diff --git a/src/StellaOps.Authority/StellaOps.Authority/OpenIddict/AuthorityOpenIddictConstants.cs b/src/StellaOps.Authority/StellaOps.Authority/OpenIddict/AuthorityOpenIddictConstants.cs index 94e39a4e..0a033030 100644 --- a/src/StellaOps.Authority/StellaOps.Authority/OpenIddict/AuthorityOpenIddictConstants.cs +++ b/src/StellaOps.Authority/StellaOps.Authority/OpenIddict/AuthorityOpenIddictConstants.cs @@ -15,4 +15,14 @@ internal static class AuthorityOpenIddictConstants internal const string AuditRequestedScopesProperty = "authority:audit_requested_scopes"; internal const string AuditGrantedScopesProperty = "authority:audit_granted_scopes"; internal const string AuditInvalidScopeProperty = "authority:audit_invalid_scope"; + internal const string ClientSenderConstraintProperty = "authority:client_sender_constraint"; + internal const string SenderConstraintProperty = "authority:sender_constraint"; + internal const string DpopKeyThumbprintProperty = "authority:dpop_thumbprint"; + internal const string DpopProofJwtIdProperty = "authority:dpop_jti"; + internal const string DpopIssuedAtProperty = "authority:dpop_iat"; + internal const string DpopConsumedNonceProperty = "authority:dpop_nonce"; + internal const string ConfirmationClaimType = "cnf"; + internal const string SenderConstraintClaimType = "authority_sender_constraint"; + internal const string MtlsCertificateThumbprintProperty = "authority:mtls_thumbprint"; + internal const string MtlsCertificateHexProperty = "authority:mtls_thumbprint_hex"; } diff --git a/src/StellaOps.Authority/StellaOps.Authority/OpenIddict/Handlers/ClientCredentialsHandlers.cs b/src/StellaOps.Authority/StellaOps.Authority/OpenIddict/Handlers/ClientCredentialsHandlers.cs index 820f68aa..b394b339 100644 --- a/src/StellaOps.Authority/StellaOps.Authority/OpenIddict/Handlers/ClientCredentialsHandlers.cs +++ b/src/StellaOps.Authority/StellaOps.Authority/OpenIddict/Handlers/ClientCredentialsHandlers.cs @@ -5,6 +5,8 @@ using System.Globalization; using System.Linq; using System.Security.Claims; using System.Security.Cryptography; +using System.Text.Json; +using Microsoft.AspNetCore.Http; using Microsoft.Extensions.Logging; using OpenIddict.Abstractions; using OpenIddict.Extensions; @@ -18,6 +20,7 @@ using StellaOps.Authority.Storage.Mongo.Documents; using StellaOps.Authority.Storage.Mongo.Sessions; using StellaOps.Authority.Storage.Mongo.Stores; using StellaOps.Authority.RateLimiting; +using StellaOps.Authority.Security; using StellaOps.Cryptography.Audit; namespace StellaOps.Authority.OpenIddict.Handlers; @@ -30,6 +33,8 @@ internal sealed class ValidateClientCredentialsHandler : IOpenIddictServerHandle private readonly IAuthEventSink auditSink; private readonly IAuthorityRateLimiterMetadataAccessor metadataAccessor; private readonly TimeProvider timeProvider; + private readonly IAuthorityClientCertificateValidator certificateValidator; + private readonly IHttpContextAccessor httpContextAccessor; private readonly ILogger logger; public ValidateClientCredentialsHandler( @@ -39,6 +44,8 @@ internal sealed class ValidateClientCredentialsHandler : IOpenIddictServerHandle IAuthEventSink auditSink, IAuthorityRateLimiterMetadataAccessor metadataAccessor, TimeProvider timeProvider, + IAuthorityClientCertificateValidator certificateValidator, + IHttpContextAccessor httpContextAccessor, ILogger logger) { this.clientStore = clientStore ?? throw new ArgumentNullException(nameof(clientStore)); @@ -47,6 +54,8 @@ internal sealed class ValidateClientCredentialsHandler : IOpenIddictServerHandle this.auditSink = auditSink ?? throw new ArgumentNullException(nameof(auditSink)); this.metadataAccessor = metadataAccessor ?? throw new ArgumentNullException(nameof(metadataAccessor)); this.timeProvider = timeProvider ?? throw new ArgumentNullException(nameof(timeProvider)); + this.certificateValidator = certificateValidator ?? throw new ArgumentNullException(nameof(certificateValidator)); + this.httpContextAccessor = httpContextAccessor ?? throw new ArgumentNullException(nameof(httpContextAccessor)); this.logger = logger ?? throw new ArgumentNullException(nameof(logger)); } @@ -111,7 +120,44 @@ internal sealed class ValidateClientCredentialsHandler : IOpenIddictServerHandle return; } - context.Transaction.Properties[AuthorityOpenIddictConstants.AuditConfidentialProperty] = string.Equals(document.ClientType, "confidential", StringComparison.OrdinalIgnoreCase); + var existingSenderConstraint = context.Transaction.Properties.TryGetValue(AuthorityOpenIddictConstants.SenderConstraintProperty, out var senderConstraintValue) && senderConstraintValue is string existingConstraint + ? existingConstraint + : null; + + var normalizedSenderConstraint = !string.IsNullOrWhiteSpace(existingSenderConstraint) + ? existingSenderConstraint + : ClientCredentialHandlerHelpers.NormalizeSenderConstraint(document); + + if (!string.IsNullOrWhiteSpace(normalizedSenderConstraint)) + { + context.Transaction.Properties[AuthorityOpenIddictConstants.ClientSenderConstraintProperty] = normalizedSenderConstraint; + } + + if (string.Equals(normalizedSenderConstraint, AuthoritySenderConstraintKinds.Mtls, StringComparison.Ordinal)) + { + var httpContext = httpContextAccessor.HttpContext; + if (httpContext is null) + { + context.Reject(OpenIddictConstants.Errors.ServerError, "HTTP context unavailable for mTLS validation."); + logger.LogWarning("Client credentials validation failed for {ClientId}: HTTP context unavailable for mTLS validation.", context.ClientId); + return; + } + + var validation = await certificateValidator.ValidateAsync(httpContext, document, context.CancellationToken).ConfigureAwait(false); + if (!validation.Succeeded) + { + context.Reject(OpenIddictConstants.Errors.InvalidClient, validation.Error ?? "Client certificate validation failed."); + logger.LogWarning("Client credentials validation failed for {ClientId}: {Reason}.", context.ClientId, validation.Error ?? "certificate_invalid"); + return; + } + + context.Transaction.Properties[AuthorityOpenIddictConstants.SenderConstraintProperty] = AuthoritySenderConstraintKinds.Mtls; + context.Transaction.Properties[AuthorityOpenIddictConstants.MtlsCertificateThumbprintProperty] = validation.ConfirmationThumbprint; + context.Transaction.Properties[AuthorityOpenIddictConstants.MtlsCertificateHexProperty] = validation.HexThumbprint; + } + + context.Transaction.Properties[AuthorityOpenIddictConstants.AuditConfidentialProperty] = + string.Equals(document.ClientType, "confidential", StringComparison.OrdinalIgnoreCase); IIdentityProviderPlugin? provider = null; if (!string.IsNullOrWhiteSpace(document.Plugin)) @@ -278,6 +324,32 @@ internal sealed class HandleClientCredentialsHandler : IOpenIddictServerHandler< return; } + var configuredAudiences = ClientCredentialHandlerHelpers.Split(document.Properties, AuthorityClientMetadataKeys.Audiences); + if (configuredAudiences.Count > 0) + { + if (context.Request.Resources is ICollection resources && configuredAudiences.Count > 0) + { + foreach (var audience in configuredAudiences) + { + if (!resources.Contains(audience)) + { + resources.Add(audience); + } + } + } + + if (context.Request.Audiences is ICollection audiencesCollection) + { + foreach (var audience in configuredAudiences) + { + if (!audiencesCollection.Contains(audience)) + { + audiencesCollection.Add(audience); + } + } + } + } + var identity = new ClaimsIdentity(OpenIddictServerAspNetCoreDefaults.AuthenticationScheme); identity.AddClaim(new Claim(OpenIddictConstants.Claims.Subject, document.ClientId)); identity.AddClaim(new Claim(OpenIddictConstants.Claims.ClientId, document.ClientId)); @@ -322,6 +394,8 @@ internal sealed class HandleClientCredentialsHandler : IOpenIddictServerHandler< activity?.SetTag("authority.identity_provider", provider.Name); } + ApplySenderConstraintClaims(context, identity, document); + var principal = new ClaimsPrincipal(identity); var grantedScopes = context.Transaction.Properties.TryGetValue(AuthorityOpenIddictConstants.ClientGrantedScopesProperty, out var scopesValue) && @@ -338,6 +412,11 @@ internal sealed class HandleClientCredentialsHandler : IOpenIddictServerHandler< principal.SetScopes(Array.Empty()); } + if (configuredAudiences.Count > 0) + { + principal.SetAudiences(configuredAudiences); + } + if (provider is not null && descriptor is not null) { var enrichmentContext = new AuthorityClaimsEnrichmentContext(provider.Context, user: null, descriptor); @@ -420,10 +499,95 @@ internal sealed class HandleClientCredentialsHandler : IOpenIddictServerHandler< ExpiresAt = expiresAt }; + if (context.Transaction.Properties.TryGetValue(AuthorityOpenIddictConstants.SenderConstraintProperty, out var constraintObj) && + constraintObj is string senderConstraint && + !string.IsNullOrWhiteSpace(senderConstraint)) + { + record.SenderConstraint = senderConstraint; + } + + string? senderThumbprint = null; + if (context.Transaction.Properties.TryGetValue(AuthorityOpenIddictConstants.DpopKeyThumbprintProperty, out var dpopThumbprintObj) && + dpopThumbprintObj is string dpopThumbprint && + !string.IsNullOrWhiteSpace(dpopThumbprint)) + { + senderThumbprint = dpopThumbprint; + } + else if (context.Transaction.Properties.TryGetValue(AuthorityOpenIddictConstants.MtlsCertificateThumbprintProperty, out var mtlsThumbprintObj) && + mtlsThumbprintObj is string mtlsThumbprint && + !string.IsNullOrWhiteSpace(mtlsThumbprint)) + { + senderThumbprint = mtlsThumbprint; + } + + if (senderThumbprint is not null) + { + record.SenderKeyThumbprint = senderThumbprint; + } + + if (context.Transaction.Properties.TryGetValue(AuthorityOpenIddictConstants.DpopConsumedNonceProperty, out var nonceObj) && + nonceObj is string nonce && + !string.IsNullOrWhiteSpace(nonce)) + { + record.SenderNonce = nonce; + } + await tokenStore.InsertAsync(record, context.CancellationToken, session).ConfigureAwait(false); context.Transaction.Properties[AuthorityOpenIddictConstants.TokenTransactionProperty] = record; activity?.SetTag("authority.token_id", tokenId); } + + private static void ApplySenderConstraintClaims( + OpenIddictServerEvents.HandleTokenRequestContext context, + ClaimsIdentity identity, + AuthorityClientDocument document) + { + _ = document; + + if (!context.Transaction.Properties.TryGetValue(AuthorityOpenIddictConstants.SenderConstraintProperty, out var constraintObj) || + constraintObj is not string senderConstraint || + string.IsNullOrWhiteSpace(senderConstraint)) + { + return; + } + + var normalized = senderConstraint.Trim().ToLowerInvariant(); + context.Transaction.Properties[AuthorityOpenIddictConstants.SenderConstraintProperty] = normalized; + identity.SetClaim(AuthorityOpenIddictConstants.SenderConstraintClaimType, normalized); + + switch (normalized) + { + case AuthoritySenderConstraintKinds.Dpop: + if (context.Transaction.Properties.TryGetValue(AuthorityOpenIddictConstants.DpopKeyThumbprintProperty, out var thumbprintObj) && + thumbprintObj is string thumbprint && + !string.IsNullOrWhiteSpace(thumbprint)) + { + var confirmation = JsonSerializer.Serialize(new Dictionary + { + ["jkt"] = thumbprint + }); + + identity.SetClaim(AuthorityOpenIddictConstants.ConfirmationClaimType, confirmation); + } + + break; + case AuthoritySenderConstraintKinds.Mtls: + if (context.Transaction.Properties.TryGetValue(AuthorityOpenIddictConstants.MtlsCertificateThumbprintProperty, out var mtlsThumbprintObj) && + mtlsThumbprintObj is string mtlsThumbprint && + !string.IsNullOrWhiteSpace(mtlsThumbprint)) + { + var confirmation = JsonSerializer.Serialize(new Dictionary + { + ["x5t#S256"] = mtlsThumbprint + }); + + identity.SetClaim(AuthorityOpenIddictConstants.ConfirmationClaimType, confirmation); + } + + break; + } + } + } internal static class ClientCredentialHandlerHelpers @@ -491,4 +655,20 @@ internal static class ClientCredentialHandlerHelpers return false; } } + + public static string? NormalizeSenderConstraint(AuthorityClientDocument document) + { + if (!string.IsNullOrWhiteSpace(document.SenderConstraint)) + { + return document.SenderConstraint.Trim().ToLowerInvariant(); + } + + if (document.Properties.TryGetValue(AuthorityClientMetadataKeys.SenderConstraint, out var value) && + !string.IsNullOrWhiteSpace(value)) + { + return value.Trim().ToLowerInvariant(); + } + + return null; + } } diff --git a/src/StellaOps.Authority/StellaOps.Authority/OpenIddict/Handlers/DpopHandlers.cs b/src/StellaOps.Authority/StellaOps.Authority/OpenIddict/Handlers/DpopHandlers.cs new file mode 100644 index 00000000..2fabcb25 --- /dev/null +++ b/src/StellaOps.Authority/StellaOps.Authority/OpenIddict/Handlers/DpopHandlers.cs @@ -0,0 +1,643 @@ +using System; +using System.Collections.Generic; +using System.Diagnostics; +using System.Globalization; +using System.Linq; +using System.Text.Json; +using Microsoft.AspNetCore.Http; +using Microsoft.AspNetCore.Http.Extensions; +using Microsoft.Extensions.Logging; +using Microsoft.Extensions.Primitives; +using OpenIddict.Abstractions; +using OpenIddict.Extensions; +using OpenIddict.Server; +using OpenIddict.Server.AspNetCore; +using StellaOps.Auth.Security.Dpop; +using StellaOps.Authority.OpenIddict; +using StellaOps.Authority.RateLimiting; +using StellaOps.Authority.Security; +using StellaOps.Authority.Storage.Mongo.Documents; +using StellaOps.Authority.Storage.Mongo.Stores; +using StellaOps.Authority.Plugins.Abstractions; +using StellaOps.Configuration; +using StellaOps.Cryptography.Audit; +using Microsoft.IdentityModel.Tokens; + +namespace StellaOps.Authority.OpenIddict.Handlers; + +internal sealed class ValidateDpopProofHandler : IOpenIddictServerHandler +{ + private readonly StellaOpsAuthorityOptions authorityOptions; + private readonly IAuthorityClientStore clientStore; + private readonly IDpopProofValidator proofValidator; + private readonly IDpopNonceStore nonceStore; + private readonly IAuthorityRateLimiterMetadataAccessor metadataAccessor; + private readonly IAuthEventSink auditSink; + private readonly TimeProvider clock; + private readonly ActivitySource activitySource; + private readonly ILogger logger; + + public ValidateDpopProofHandler( + StellaOpsAuthorityOptions authorityOptions, + IAuthorityClientStore clientStore, + IDpopProofValidator proofValidator, + IDpopNonceStore nonceStore, + IAuthorityRateLimiterMetadataAccessor metadataAccessor, + IAuthEventSink auditSink, + TimeProvider clock, + ActivitySource activitySource, + ILogger logger) + { + this.authorityOptions = authorityOptions ?? throw new ArgumentNullException(nameof(authorityOptions)); + this.clientStore = clientStore ?? throw new ArgumentNullException(nameof(clientStore)); + this.proofValidator = proofValidator ?? throw new ArgumentNullException(nameof(proofValidator)); + this.nonceStore = nonceStore ?? throw new ArgumentNullException(nameof(nonceStore)); + this.metadataAccessor = metadataAccessor ?? throw new ArgumentNullException(nameof(metadataAccessor)); + this.auditSink = auditSink ?? throw new ArgumentNullException(nameof(auditSink)); + this.clock = clock ?? throw new ArgumentNullException(nameof(clock)); + this.activitySource = activitySource ?? throw new ArgumentNullException(nameof(activitySource)); + this.logger = logger ?? throw new ArgumentNullException(nameof(logger)); + } + + public async ValueTask HandleAsync(OpenIddictServerEvents.ValidateTokenRequestContext context) + { + ArgumentNullException.ThrowIfNull(context); + + if (!context.Request.IsClientCredentialsGrantType()) + { + return; + } + + using var activity = activitySource.StartActivity("authority.token.validate_dpop", ActivityKind.Internal); + activity?.SetTag("authority.endpoint", "/token"); + activity?.SetTag("authority.grant_type", OpenIddictConstants.GrantTypes.ClientCredentials); + + var clientId = context.ClientId ?? context.Request.ClientId; + if (string.IsNullOrWhiteSpace(clientId)) + { + return; + } + + context.Transaction.Properties[AuthorityOpenIddictConstants.AuditClientIdProperty] = clientId; + + var senderConstraintOptions = authorityOptions.Security.SenderConstraints; + AuthorityClientDocument? clientDocument = await ResolveClientAsync(context, clientId, activity, cancel: context.CancellationToken).ConfigureAwait(false); + if (clientDocument is null) + { + return; + } + + var senderConstraint = NormalizeSenderConstraint(clientDocument); + context.Transaction.Properties[AuthorityOpenIddictConstants.ClientSenderConstraintProperty] = senderConstraint; + + if (!string.Equals(senderConstraint, AuthoritySenderConstraintKinds.Dpop, StringComparison.Ordinal)) + { + return; + } + + var configuredAudiences = EnsureRequestAudiences(context.Request, clientDocument); + + if (!senderConstraintOptions.Dpop.Enabled) + { + logger.LogError("Client {ClientId} requires DPoP but server-side configuration has DPoP disabled.", clientId); + context.Reject(OpenIddictConstants.Errors.ServerError, "DPoP authentication is not enabled."); + await WriteAuditAsync(context, clientDocument, AuthEventOutcome.Failure, "DPoP disabled server-side.", null, null, null, "authority.dpop.proof.disabled").ConfigureAwait(false); + return; + } + + metadataAccessor.SetTag("authority.sender_constraint", AuthoritySenderConstraintKinds.Dpop); + activity?.SetTag("authority.sender_constraint", AuthoritySenderConstraintKinds.Dpop); + + HttpRequest? httpRequest = null; + HttpResponse? httpResponse = null; + if (context.Transaction.Properties.TryGetValue(typeof(HttpContext).FullName!, out var httpContextProperty) && + httpContextProperty is HttpContext capturedContext) + { + httpRequest = capturedContext.Request; + httpResponse = capturedContext.Response; + } + if (httpRequest is null) + { + context.Reject(OpenIddictConstants.Errors.ServerError, "Unable to access HTTP context for DPoP validation."); + logger.LogError("DPoP validation aborted for {ClientId}: HTTP request not available via transaction.", clientId); + await WriteAuditAsync(context, clientDocument, AuthEventOutcome.Failure, "HTTP request unavailable for DPoP.", null, null, null, "authority.dpop.proof.error").ConfigureAwait(false); + return; + } + + if (!httpRequest.Headers.TryGetValue("DPoP", out StringValues proofHeader) || StringValues.IsNullOrEmpty(proofHeader)) + { + logger.LogWarning("Missing DPoP header for client credentials request from {ClientId}.", clientId); + await ChallengeNonceAsync( + context, + clientDocument, + audience: null, + thumbprint: null, + reasonCode: "missing_proof", + description: "DPoP proof is required.", + senderConstraintOptions, + httpResponse).ConfigureAwait(false); + return; + } + + var proof = proofHeader.ToString(); + var requestUri = BuildRequestUri(httpRequest); + + var validationResult = await proofValidator.ValidateAsync( + proof, + httpRequest.Method, + requestUri, + cancellationToken: context.CancellationToken).ConfigureAwait(false); + + if (!validationResult.IsValid) + { + var error = string.IsNullOrWhiteSpace(validationResult.ErrorDescription) + ? "DPoP proof validation failed." + : validationResult.ErrorDescription; + + logger.LogWarning("DPoP proof validation failed for client {ClientId}: {Reason}.", clientId, error); + await ChallengeNonceAsync( + context, + clientDocument, + audience: null, + thumbprint: null, + reasonCode: validationResult.ErrorCode ?? "invalid_proof", + description: error, + senderConstraintOptions, + httpResponse).ConfigureAwait(false); + return; + } + + if (validationResult.PublicKey is not Microsoft.IdentityModel.Tokens.JsonWebKey jwk) + { + logger.LogWarning("DPoP proof for {ClientId} did not expose a JSON Web Key.", clientId); + await ChallengeNonceAsync( + context, + clientDocument, + audience: null, + thumbprint: null, + reasonCode: "invalid_key", + description: "DPoP proof must embed a JSON Web Key.", + senderConstraintOptions, + httpResponse).ConfigureAwait(false); + return; + } + + object rawThumbprint = jwk.ComputeJwkThumbprint(); + string thumbprint; + if (rawThumbprint is string value && !string.IsNullOrWhiteSpace(value)) + { + thumbprint = value; + } + else if (rawThumbprint is byte[] bytes) + { + thumbprint = Base64UrlEncoder.Encode(bytes); + } + else + { + throw new InvalidOperationException("DPoP JWK thumbprint could not be computed."); + } + context.Transaction.Properties[AuthorityOpenIddictConstants.SenderConstraintProperty] = AuthoritySenderConstraintKinds.Dpop; + context.Transaction.Properties[AuthorityOpenIddictConstants.DpopKeyThumbprintProperty] = thumbprint; + if (!string.IsNullOrWhiteSpace(validationResult.JwtId)) + { + context.Transaction.Properties[AuthorityOpenIddictConstants.DpopProofJwtIdProperty] = validationResult.JwtId; + } + + if (validationResult.IssuedAt is { } issuedAt) + { + context.Transaction.Properties[AuthorityOpenIddictConstants.DpopIssuedAtProperty] = issuedAt; + } + + var nonceOptions = senderConstraintOptions.Dpop.Nonce; + var requiredAudience = ResolveNonceAudience(context.Request, nonceOptions, configuredAudiences); + + if (nonceOptions.Enabled && requiredAudience is not null) + { + activity?.SetTag("authority.dpop_nonce_audience", requiredAudience); + var suppliedNonce = validationResult.Nonce; + + if (string.IsNullOrWhiteSpace(suppliedNonce)) + { + logger.LogInformation("DPoP nonce challenge issued to {ClientId} for audience {Audience}: nonce missing.", clientId, requiredAudience); + await ChallengeNonceAsync( + context, + clientDocument, + requiredAudience, + thumbprint, + "nonce_missing", + "DPoP nonce is required for this audience.", + senderConstraintOptions, + httpResponse).ConfigureAwait(false); + return; + } + + var consumeResult = await nonceStore.TryConsumeAsync( + suppliedNonce, + requiredAudience, + clientDocument.ClientId, + thumbprint, + context.CancellationToken).ConfigureAwait(false); + + switch (consumeResult.Status) + { + case DpopNonceConsumeStatus.Success: + context.Transaction.Properties[AuthorityOpenIddictConstants.DpopConsumedNonceProperty] = suppliedNonce; + break; + case DpopNonceConsumeStatus.Expired: + logger.LogInformation("DPoP nonce expired for {ClientId} and audience {Audience}.", clientId, requiredAudience); + await ChallengeNonceAsync( + context, + clientDocument, + requiredAudience, + thumbprint, + "nonce_expired", + "DPoP nonce has expired. Retry with a fresh nonce.", + senderConstraintOptions, + httpResponse).ConfigureAwait(false); + return; + default: + logger.LogInformation("DPoP nonce invalid for {ClientId} and audience {Audience}.", clientId, requiredAudience); + await ChallengeNonceAsync( + context, + clientDocument, + requiredAudience, + thumbprint, + "nonce_invalid", + "DPoP nonce is invalid. Request a new nonce and retry.", + senderConstraintOptions, + httpResponse).ConfigureAwait(false); + return; + } + } + + await WriteAuditAsync( + context, + clientDocument, + AuthEventOutcome.Success, + "DPoP proof validated.", + thumbprint, + validationResult, + requiredAudience, + "authority.dpop.proof.valid") + .ConfigureAwait(false); + logger.LogInformation("DPoP proof validated for client {ClientId}.", clientId); + } + + private async ValueTask ResolveClientAsync( + OpenIddictServerEvents.ValidateTokenRequestContext context, + string clientId, + Activity? activity, + CancellationToken cancel) + { + if (context.Transaction.Properties.TryGetValue(AuthorityOpenIddictConstants.ClientTransactionProperty, out var value) && + value is AuthorityClientDocument cached) + { + activity?.SetTag("authority.client_id", cached.ClientId); + return cached; + } + + var document = await clientStore.FindByClientIdAsync(clientId, cancel).ConfigureAwait(false); + if (document is not null) + { + context.Transaction.Properties[AuthorityOpenIddictConstants.ClientTransactionProperty] = document; + activity?.SetTag("authority.client_id", document.ClientId); + } + + return document; + } + + private static string? NormalizeSenderConstraint(AuthorityClientDocument document) + { + if (!string.IsNullOrWhiteSpace(document.SenderConstraint)) + { + return document.SenderConstraint.Trim().ToLowerInvariant(); + } + + if (document.Properties.TryGetValue(AuthorityClientMetadataKeys.SenderConstraint, out var value) && + !string.IsNullOrWhiteSpace(value)) + { + return value.Trim().ToLowerInvariant(); + } + + return null; + } + + private static IReadOnlyList EnsureRequestAudiences(OpenIddictRequest? request, AuthorityClientDocument document) + { + if (request is null) + { + return Array.Empty(); + } + + var configuredAudiences = ClientCredentialHandlerHelpers.Split(document.Properties, AuthorityClientMetadataKeys.Audiences); + if (configuredAudiences.Count == 0) + { + return configuredAudiences; + } + + if (request.Resources is ICollection resources) + { + foreach (var audience in configuredAudiences) + { + if (!resources.Contains(audience)) + { + resources.Add(audience); + } + } + } + + if (request.Audiences is ICollection audiencesCollection) + { + foreach (var audience in configuredAudiences) + { + if (!audiencesCollection.Contains(audience)) + { + audiencesCollection.Add(audience); + } + } + } + + return configuredAudiences; + } + + private static Uri BuildRequestUri(HttpRequest request) + { + ArgumentNullException.ThrowIfNull(request); + var url = request.GetDisplayUrl(); + return new Uri(url, UriKind.Absolute); + } + +private static string? ResolveNonceAudience(OpenIddictRequest request, AuthorityDpopNonceOptions nonceOptions, IReadOnlyList configuredAudiences) + { + if (!nonceOptions.Enabled || request is null) + { + return null; + } + + if (request.Resources is not null) + { + foreach (var resource in request.Resources) + { + if (string.IsNullOrWhiteSpace(resource)) + { + continue; + } + + var normalized = resource.Trim(); + if (nonceOptions.RequiredAudiences.Contains(normalized)) + { + return normalized; + } + } + } + + if (request.Audiences is not null) + { + foreach (var audience in request.Audiences) + { + if (string.IsNullOrWhiteSpace(audience)) + { + continue; + } + + var normalized = audience.Trim(); + if (nonceOptions.RequiredAudiences.Contains(normalized)) + { + return normalized; + } + } + } + + if (configuredAudiences is { Count: > 0 }) + { + foreach (var audience in configuredAudiences) + { + if (string.IsNullOrWhiteSpace(audience)) + { + continue; + } + + var normalized = audience.Trim(); + if (nonceOptions.RequiredAudiences.Contains(normalized)) + { + return normalized; + } + } + } + + return null; + } + + private async ValueTask ChallengeNonceAsync( + OpenIddictServerEvents.ValidateTokenRequestContext context, + AuthorityClientDocument clientDocument, + string? audience, + string? thumbprint, + string reasonCode, + string description, + AuthoritySenderConstraintOptions senderConstraintOptions, + HttpResponse? httpResponse) + { + context.Reject(OpenIddictConstants.Errors.InvalidClient, description); + metadataAccessor.SetTag("authority.dpop_result", reasonCode); + + string? issuedNonce = null; + DateTimeOffset? expiresAt = null; + if (audience is not null && thumbprint is not null && senderConstraintOptions.Dpop.Nonce.Enabled) + { + var issuance = await nonceStore.IssueAsync( + audience, + clientDocument.ClientId, + thumbprint, + senderConstraintOptions.Dpop.Nonce.Ttl, + senderConstraintOptions.Dpop.Nonce.MaxIssuancePerMinute, + context.CancellationToken).ConfigureAwait(false); + + if (issuance.Status == DpopNonceIssueStatus.Success) + { + issuedNonce = issuance.Nonce; + expiresAt = issuance.ExpiresAt; + } + else + { + logger.LogWarning("Unable to issue DPoP nonce for {ClientId} (audience {Audience}): {Status}.", clientDocument.ClientId, audience, issuance.Status); + } + } + + if (httpResponse is not null) + { + httpResponse.Headers["WWW-Authenticate"] = BuildAuthenticateHeader(reasonCode, description, issuedNonce); + + if (!string.IsNullOrWhiteSpace(issuedNonce)) + { + httpResponse.Headers["DPoP-Nonce"] = issuedNonce; + } + } + + await WriteAuditAsync( + context, + clientDocument, + AuthEventOutcome.Failure, + description, + thumbprint, + validationResult: null, + audience, + "authority.dpop.proof.challenge", + reasonCode, + issuedNonce, + expiresAt) + .ConfigureAwait(false); + } + + private static string BuildAuthenticateHeader(string reasonCode, string description, string? nonce) + { + var parameters = new Dictionary + { + ["error"] = string.Equals(reasonCode, "nonce_missing", StringComparison.OrdinalIgnoreCase) + ? "use_dpop_nonce" + : "invalid_dpop_proof", + ["error_description"] = description + }; + + if (!string.IsNullOrWhiteSpace(nonce)) + { + parameters["dpop-nonce"] = nonce; + } + + var segments = new List(); + foreach (var kvp in parameters) + { + if (kvp.Value is null) + { + continue; + } + + segments.Add($"{kvp.Key}=\"{EscapeHeaderValue(kvp.Value)}\""); + } + + return segments.Count > 0 + ? $"DPoP {string.Join(", ", segments)}" + : "DPoP"; + + static string EscapeHeaderValue(string value) + => value + .Replace("\\", "\\\\", StringComparison.Ordinal) + .Replace("\"", "\\\"", StringComparison.Ordinal); + } + + private async ValueTask WriteAuditAsync( + OpenIddictServerEvents.ValidateTokenRequestContext context, + AuthorityClientDocument clientDocument, + AuthEventOutcome outcome, + string reason, + string? thumbprint, + DpopValidationResult? validationResult, + string? audience, + string eventType, + string? reasonCode = null, + string? issuedNonce = null, + DateTimeOffset? nonceExpiresAt = null) + { + var metadata = metadataAccessor.GetMetadata(); + var properties = new List + { + new() + { + Name = "sender.constraint", + Value = ClassifiedString.Public(AuthoritySenderConstraintKinds.Dpop) + } + }; + + if (!string.IsNullOrWhiteSpace(reasonCode)) + { + properties.Add(new AuthEventProperty + { + Name = "dpop.reason_code", + Value = ClassifiedString.Public(reasonCode) + }); + } + + if (!string.IsNullOrWhiteSpace(thumbprint)) + { + properties.Add(new AuthEventProperty + { + Name = "dpop.jkt", + Value = ClassifiedString.Public(thumbprint) + }); + } + + if (validationResult?.JwtId is not null) + { + properties.Add(new AuthEventProperty + { + Name = "dpop.jti", + Value = ClassifiedString.Public(validationResult.JwtId) + }); + } + + if (validationResult?.IssuedAt is { } issuedAt) + { + properties.Add(new AuthEventProperty + { + Name = "dpop.issued_at", + Value = ClassifiedString.Public(issuedAt.ToString("O", CultureInfo.InvariantCulture)) + }); + } + + if (audience is not null) + { + properties.Add(new AuthEventProperty + { + Name = "dpop.audience", + Value = ClassifiedString.Public(audience) + }); + } + + if (!string.IsNullOrWhiteSpace(validationResult?.Nonce)) + { + properties.Add(new AuthEventProperty + { + Name = "dpop.nonce.presented", + Value = ClassifiedString.Sensitive(validationResult.Nonce) + }); + } + + if (!string.IsNullOrWhiteSpace(issuedNonce)) + { + properties.Add(new AuthEventProperty + { + Name = "dpop.nonce.issued", + Value = ClassifiedString.Sensitive(issuedNonce) + }); + } + + if (nonceExpiresAt is { } expiresAt) + { + properties.Add(new AuthEventProperty + { + Name = "dpop.nonce.expires_at", + Value = ClassifiedString.Public(expiresAt.ToString("O", CultureInfo.InvariantCulture)) + }); + } + + var confidential = string.Equals(clientDocument.ClientType, "confidential", StringComparison.OrdinalIgnoreCase); + + var record = ClientCredentialsAuditHelper.CreateRecord( + clock, + context.Transaction, + metadata, + clientSecret: null, + outcome, + reason, + clientDocument.ClientId, + providerName: clientDocument.Plugin, + confidential, + requestedScopes: Array.Empty(), + grantedScopes: Array.Empty(), + invalidScope: null, + extraProperties: properties, + eventType: eventType); + + await auditSink.WriteAsync(record, context.CancellationToken).ConfigureAwait(false); + } +} diff --git a/src/StellaOps.Authority/StellaOps.Authority/OpenIddict/Handlers/TokenPersistenceHandlers.cs b/src/StellaOps.Authority/StellaOps.Authority/OpenIddict/Handlers/TokenPersistenceHandlers.cs index 258a4120..184ea162 100644 --- a/src/StellaOps.Authority/StellaOps.Authority/OpenIddict/Handlers/TokenPersistenceHandlers.cs +++ b/src/StellaOps.Authority/StellaOps.Authority/OpenIddict/Handlers/TokenPersistenceHandlers.cs @@ -4,6 +4,7 @@ using System.Diagnostics; using System.Globalization; using System.Linq; using System.Security.Claims; +using System.Text.Json; using System.Threading; using System.Threading.Tasks; using Microsoft.Extensions.Logging; @@ -92,6 +93,33 @@ internal sealed class PersistTokensHandler : IOpenIddictServerHandler claim.Type == AuthorityOpenIddictConstants.SenderConstraintClaimType)) + { + identity.SetClaim(AuthorityOpenIddictConstants.SenderConstraintClaimType, tokenDocument.SenderConstraint); + } + + if (identity.HasClaim(claim => claim.Type == AuthorityOpenIddictConstants.ConfirmationClaimType)) + { + return; + } + + if (string.IsNullOrWhiteSpace(tokenDocument.SenderConstraint) || string.IsNullOrWhiteSpace(tokenDocument.SenderKeyThumbprint)) + { + return; + } + + string confirmation = tokenDocument.SenderConstraint switch + { + AuthoritySenderConstraintKinds.Dpop => JsonSerializer.Serialize(new Dictionary + { + ["jkt"] = tokenDocument.SenderKeyThumbprint + }), + AuthoritySenderConstraintKinds.Mtls => JsonSerializer.Serialize(new Dictionary + { + ["x5t#S256"] = tokenDocument.SenderKeyThumbprint + }), + _ => string.Empty + }; + + if (!string.IsNullOrEmpty(confirmation)) + { + identity.SetClaim(AuthorityOpenIddictConstants.ConfirmationClaimType, confirmation); + } + } } diff --git a/src/StellaOps.Authority/StellaOps.Authority/Program.cs b/src/StellaOps.Authority/StellaOps.Authority/Program.cs index 0128c8b6..4d032efe 100644 --- a/src/StellaOps.Authority/StellaOps.Authority/Program.cs +++ b/src/StellaOps.Authority/StellaOps.Authority/Program.cs @@ -38,8 +38,10 @@ using StellaOps.Authority.Revocation; using StellaOps.Authority.Signing; using StellaOps.Cryptography; using StellaOps.Authority.Storage.Mongo.Documents; +using StellaOps.Authority.Security; #if STELLAOPS_AUTH_SECURITY using StellaOps.Auth.Security.Dpop; +using StackExchange.Redis; #endif var builder = WebApplication.CreateBuilder(args); @@ -98,6 +100,7 @@ builder.Services.AddHttpContextAccessor(); builder.Services.TryAddSingleton(_ => TimeProvider.System); builder.Services.TryAddSingleton(); builder.Services.TryAddSingleton(); +builder.Services.AddSingleton(); #if STELLAOPS_AUTH_SECURITY var senderConstraints = authorityOptions.Security.SenderConstraints; @@ -119,6 +122,29 @@ builder.Services.AddOptions() builder.Services.TryAddSingleton(provider => new InMemoryDpopReplayCache(provider.GetService())); builder.Services.TryAddSingleton(); +if (string.Equals(senderConstraints.Dpop.Nonce.Store, "redis", StringComparison.OrdinalIgnoreCase)) +{ + builder.Services.TryAddSingleton(_ => + ConnectionMultiplexer.Connect(senderConstraints.Dpop.Nonce.RedisConnectionString!)); + + builder.Services.TryAddSingleton(provider => + { + var multiplexer = provider.GetRequiredService(); + var timeProvider = provider.GetService(); + return new RedisDpopNonceStore(multiplexer, timeProvider); + }); +} +else +{ + builder.Services.TryAddSingleton(provider => + { + var timeProvider = provider.GetService(); + var nonceLogger = provider.GetService>(); + return new InMemoryDpopNonceStore(timeProvider, nonceLogger); + }); +} + +builder.Services.AddScoped(); #endif builder.Services.AddRateLimiter(rateLimiterOptions => @@ -219,6 +245,13 @@ builder.Services.AddOpenIddict() aspNetCoreBuilder.DisableTransportSecurityRequirement(); } +#if STELLAOPS_AUTH_SECURITY + options.AddEventHandler(descriptor => + { + descriptor.UseScopedHandler(); + }); +#endif + options.AddEventHandler(descriptor => { descriptor.UseScopedHandler(); @@ -723,6 +756,33 @@ if (authorityOptions.Bootstrap.Enabled) ? new Dictionary(StringComparer.OrdinalIgnoreCase) : new Dictionary(request.Properties, StringComparer.OrdinalIgnoreCase); + IReadOnlyCollection? certificateBindings = null; + if (request.CertificateBindings is not null) + { + var bindingRegistrations = new List(request.CertificateBindings.Count); + foreach (var binding in request.CertificateBindings) + { + if (binding is null || string.IsNullOrWhiteSpace(binding.Thumbprint)) + { + await ReleaseInviteAsync("Certificate binding thumbprint is required."); + await WriteBootstrapClientAuditAsync(AuthEventOutcome.Failure, "Certificate binding thumbprint is required.", request.ClientId, null, provider.Name, request.AllowedScopes ?? Array.Empty(), request.Confidential, inviteToken).ConfigureAwait(false); + return Results.BadRequest(new { error = "invalid_request", message = "Certificate binding thumbprint is required." }); + } + + bindingRegistrations.Add(new AuthorityClientCertificateBindingRegistration( + binding.Thumbprint, + binding.SerialNumber, + binding.Subject, + binding.Issuer, + binding.SubjectAlternativeNames, + binding.NotBefore, + binding.NotAfter, + binding.Label)); + } + + certificateBindings = bindingRegistrations; + } + var registration = new AuthorityClientRegistration( request.ClientId, request.Confidential, @@ -730,9 +790,11 @@ if (authorityOptions.Bootstrap.Enabled) request.ClientSecret, request.AllowedGrantTypes ?? Array.Empty(), request.AllowedScopes ?? Array.Empty(), + request.AllowedAudiences ?? Array.Empty(), redirectUris, postLogoutUris, - properties); + properties, + certificateBindings); var result = await provider.ClientProvisioning.CreateOrUpdateAsync(registration, cancellationToken).ConfigureAwait(false); @@ -1149,7 +1211,7 @@ static PluginHostOptions BuildPluginHostOptions(StellaOpsAuthorityOptions option { BaseDirectory = basePath, PluginsDirectory = string.IsNullOrWhiteSpace(pluginDirectory) - ? Path.Combine("PluginBinaries", "Authority") + ? "StellaOps.Authority.PluginBinaries" : pluginDirectory, PrimaryPrefix = "StellaOps.Authority" }; diff --git a/src/StellaOps.Authority/StellaOps.Authority/Security/AuthorityClientCertificateValidationResult.cs b/src/StellaOps.Authority/StellaOps.Authority/Security/AuthorityClientCertificateValidationResult.cs new file mode 100644 index 00000000..f54150a3 --- /dev/null +++ b/src/StellaOps.Authority/StellaOps.Authority/Security/AuthorityClientCertificateValidationResult.cs @@ -0,0 +1,32 @@ +using System; +using StellaOps.Authority.Storage.Mongo.Documents; + +namespace StellaOps.Authority.Security; + +internal sealed class AuthorityClientCertificateValidationResult +{ + private AuthorityClientCertificateValidationResult(bool succeeded, string? confirmationThumbprint, string? hexThumbprint, AuthorityClientCertificateBinding? binding, string? error) + { + Succeeded = succeeded; + ConfirmationThumbprint = confirmationThumbprint; + HexThumbprint = hexThumbprint; + Binding = binding; + Error = error; + } + + public bool Succeeded { get; } + + public string? ConfirmationThumbprint { get; } + + public string? HexThumbprint { get; } + + public AuthorityClientCertificateBinding? Binding { get; } + + public string? Error { get; } + + public static AuthorityClientCertificateValidationResult Success(string confirmationThumbprint, string hexThumbprint, AuthorityClientCertificateBinding binding) + => new(true, confirmationThumbprint, hexThumbprint, binding, null); + + public static AuthorityClientCertificateValidationResult Failure(string error) + => new(false, null, null, null, error); +} diff --git a/src/StellaOps.Authority/StellaOps.Authority/Security/AuthorityClientCertificateValidator.cs b/src/StellaOps.Authority/StellaOps.Authority/Security/AuthorityClientCertificateValidator.cs new file mode 100644 index 00000000..f2ce9b98 --- /dev/null +++ b/src/StellaOps.Authority/StellaOps.Authority/Security/AuthorityClientCertificateValidator.cs @@ -0,0 +1,283 @@ +using System; +using System.Linq; +using System.Security.Cryptography; +using System.Security.Cryptography.X509Certificates; +using System.Threading; +using System.Threading.Tasks; +using System.Formats.Asn1; +using System.Net; +using Microsoft.AspNetCore.Http; +using Microsoft.Extensions.Logging; +using StellaOps.Authority.Storage.Mongo.Documents; +using StellaOps.Configuration; +using Microsoft.IdentityModel.Tokens; + +namespace StellaOps.Authority.Security; + +internal sealed class AuthorityClientCertificateValidator : IAuthorityClientCertificateValidator +{ + private readonly StellaOpsAuthorityOptions authorityOptions; + private readonly TimeProvider timeProvider; + private readonly ILogger logger; + + public AuthorityClientCertificateValidator( + StellaOpsAuthorityOptions authorityOptions, + TimeProvider timeProvider, + ILogger logger) + { + this.authorityOptions = authorityOptions ?? throw new ArgumentNullException(nameof(authorityOptions)); + this.timeProvider = timeProvider ?? throw new ArgumentNullException(nameof(timeProvider)); + this.logger = logger ?? throw new ArgumentNullException(nameof(logger)); + } + + public ValueTask ValidateAsync(HttpContext httpContext, AuthorityClientDocument client, CancellationToken cancellationToken) + { + ArgumentNullException.ThrowIfNull(httpContext); + ArgumentNullException.ThrowIfNull(client); + + var certificate = httpContext.Connection.ClientCertificate; + if (certificate is null) + { + logger.LogWarning("mTLS validation failed for {ClientId}: no client certificate present.", client.ClientId); + return ValueTask.FromResult(AuthorityClientCertificateValidationResult.Failure("client_certificate_required")); + } + + var mtlsOptions = authorityOptions.Security.SenderConstraints.Mtls; + var requiresChain = mtlsOptions.RequireChainValidation || mtlsOptions.AllowedCertificateAuthorities.Count > 0; + + X509Chain? chain = null; + var chainBuilt = false; + try + { + if (requiresChain) + { + chain = CreateChain(); + chainBuilt = TryBuildChain(chain, certificate); + if (mtlsOptions.RequireChainValidation && !chainBuilt) + { + logger.LogWarning("mTLS validation failed for {ClientId}: certificate chain validation failed.", client.ClientId); + return ValueTask.FromResult(AuthorityClientCertificateValidationResult.Failure("certificate_chain_invalid")); + } + } + + var now = timeProvider.GetUtcNow(); + if (now < certificate.NotBefore || now > certificate.NotAfter) + { + logger.LogWarning("mTLS validation failed for {ClientId}: certificate outside validity window (notBefore={NotBefore:o}, notAfter={NotAfter:o}).", client.ClientId, certificate.NotBefore, certificate.NotAfter); + return ValueTask.FromResult(AuthorityClientCertificateValidationResult.Failure("certificate_expired")); + } + + if (mtlsOptions.NormalizedSubjectPatterns.Count > 0 && + !mtlsOptions.NormalizedSubjectPatterns.Any(pattern => pattern.IsMatch(certificate.Subject))) + { + logger.LogWarning("mTLS validation failed for {ClientId}: subject {Subject} did not match allowed patterns.", client.ClientId, certificate.Subject); + return ValueTask.FromResult(AuthorityClientCertificateValidationResult.Failure("certificate_subject_mismatch")); + } + + var subjectAlternativeNames = GetSubjectAlternativeNames(certificate); + if (mtlsOptions.AllowedSanTypes.Count > 0) + { + if (subjectAlternativeNames.Count == 0) + { + logger.LogWarning("mTLS validation failed for {ClientId}: certificate does not contain subject alternative names.", client.ClientId); + return ValueTask.FromResult(AuthorityClientCertificateValidationResult.Failure("certificate_san_missing")); + } + + if (subjectAlternativeNames.Any(san => !mtlsOptions.AllowedSanTypes.Contains(san.Type))) + { + logger.LogWarning("mTLS validation failed for {ClientId}: certificate SAN types [{Types}] not allowed.", client.ClientId, string.Join(",", subjectAlternativeNames.Select(san => san.Type))); + return ValueTask.FromResult(AuthorityClientCertificateValidationResult.Failure("certificate_san_type")); + } + + if (!subjectAlternativeNames.Any(san => mtlsOptions.AllowedSanTypes.Contains(san.Type))) + { + logger.LogWarning("mTLS validation failed for {ClientId}: certificate SANs did not include any of the required types.", client.ClientId); + return ValueTask.FromResult(AuthorityClientCertificateValidationResult.Failure("certificate_san_missing_required")); + } + } + + if (mtlsOptions.AllowedCertificateAuthorities.Count > 0) + { + var allowedCas = mtlsOptions.AllowedCertificateAuthorities + .Where(value => !string.IsNullOrWhiteSpace(value)) + .Select(value => value.Trim()) + .ToHashSet(StringComparer.OrdinalIgnoreCase); + + var matchedCa = false; + if (chainBuilt && chain is not null) + { + foreach (var element in chain.ChainElements.Cast().Skip(1)) + { + if (allowedCas.Contains(element.Certificate.Subject)) + { + matchedCa = true; + break; + } + } + } + + if (!matchedCa && allowedCas.Contains(certificate.Issuer)) + { + matchedCa = true; + } + + if (!matchedCa) + { + logger.LogWarning("mTLS validation failed for {ClientId}: certificate issuer {Issuer} is not allow-listed.", client.ClientId, certificate.Issuer); + return ValueTask.FromResult(AuthorityClientCertificateValidationResult.Failure("certificate_ca_untrusted")); + } + } + + if (client.CertificateBindings.Count == 0) + { + logger.LogWarning("mTLS validation failed for {ClientId}: no certificate bindings registered for client.", client.ClientId); + return ValueTask.FromResult(AuthorityClientCertificateValidationResult.Failure("certificate_binding_missing")); + } + + var certificateHash = certificate.GetCertHash(HashAlgorithmName.SHA256); + var hexThumbprint = Convert.ToHexString(certificateHash); + var base64Thumbprint = Base64UrlEncoder.Encode(certificateHash); + + var binding = client.CertificateBindings.FirstOrDefault(b => string.Equals(b.Thumbprint, hexThumbprint, StringComparison.OrdinalIgnoreCase)); + if (binding is null) + { + logger.LogWarning("mTLS validation failed for {ClientId}: certificate thumbprint {Thumbprint} not registered.", client.ClientId, hexThumbprint); + return ValueTask.FromResult(AuthorityClientCertificateValidationResult.Failure("certificate_unbound")); + } + + if (binding.NotBefore is { } bindingNotBefore) + { + var effectiveNotBefore = bindingNotBefore - mtlsOptions.RotationGrace; + if (now < effectiveNotBefore) + { + logger.LogWarning("mTLS validation failed for {ClientId}: certificate binding not active until {NotBefore:o} (grace applied).", client.ClientId, bindingNotBefore); + return ValueTask.FromResult(AuthorityClientCertificateValidationResult.Failure("certificate_binding_inactive")); + } + } + + if (binding.NotAfter is { } bindingNotAfter) + { + var effectiveNotAfter = bindingNotAfter + mtlsOptions.RotationGrace; + if (now > effectiveNotAfter) + { + logger.LogWarning("mTLS validation failed for {ClientId}: certificate binding expired at {NotAfter:o} (grace applied).", client.ClientId, bindingNotAfter); + return ValueTask.FromResult(AuthorityClientCertificateValidationResult.Failure("certificate_binding_expired")); + } + } + + return ValueTask.FromResult(AuthorityClientCertificateValidationResult.Success(base64Thumbprint, hexThumbprint, binding)); + } + finally + { + chain?.Dispose(); + } + } + + private static X509Chain CreateChain() + => new() + { + ChainPolicy = + { + RevocationMode = X509RevocationMode.NoCheck, + RevocationFlag = X509RevocationFlag.ExcludeRoot, + VerificationFlags = X509VerificationFlags.IgnoreWrongUsage + } + }; + + private bool TryBuildChain(X509Chain chain, X509Certificate2 certificate) + { + try + { + return chain.Build(certificate); + } + catch (Exception ex) + { + logger.LogWarning(ex, "mTLS chain validation threw an exception."); + return false; + } + } + + private static IReadOnlyList<(string Type, string Value)> GetSubjectAlternativeNames(X509Certificate2 certificate) + { + foreach (var extension in certificate.Extensions) + { + if (!string.Equals(extension.Oid?.Value, "2.5.29.17", StringComparison.Ordinal)) + { + continue; + } + + try + { + var reader = new AsnReader(extension.RawData, AsnEncodingRules.DER); + var sequence = reader.ReadSequence(); + var results = new List<(string, string)>(); + + while (sequence.HasData) + { + var tag = sequence.PeekTag(); + if (tag.TagClass != TagClass.ContextSpecific) + { + sequence.ReadEncodedValue(); + continue; + } + + switch (tag.TagValue) + { + case 2: + { + var dns = sequence.ReadCharacterString(UniversalTagNumber.IA5String, new Asn1Tag(TagClass.ContextSpecific, 2)); + results.Add(("dns", dns)); + break; + } + case 6: + { + var uri = sequence.ReadCharacterString(UniversalTagNumber.IA5String, new Asn1Tag(TagClass.ContextSpecific, 6)); + results.Add(("uri", uri)); + break; + } + case 7: + { + var bytes = sequence.ReadOctetString(new Asn1Tag(TagClass.ContextSpecific, 7)); + var ip = new IPAddress(bytes).ToString(); + results.Add(("ip", ip)); + break; + } + default: + sequence.ReadEncodedValue(); + break; + } + } + + return results; + } + catch + { + return Array.Empty<(string, string)>(); + } + } + + return Array.Empty<(string, string)>(); + } + private bool ValidateCertificateChain(X509Certificate2 certificate) + { + using var chain = new X509Chain + { + ChainPolicy = + { + RevocationMode = X509RevocationMode.NoCheck, + RevocationFlag = X509RevocationFlag.ExcludeRoot, + VerificationFlags = X509VerificationFlags.IgnoreWrongUsage + } + }; + + try + { + return chain.Build(certificate); + } + catch (Exception ex) + { + logger.LogWarning(ex, "mTLS chain validation threw an exception."); + return false; + } + } +} diff --git a/src/StellaOps.Authority/StellaOps.Authority/Security/AuthoritySenderConstraintKinds.cs b/src/StellaOps.Authority/StellaOps.Authority/Security/AuthoritySenderConstraintKinds.cs new file mode 100644 index 00000000..77b4a0e7 --- /dev/null +++ b/src/StellaOps.Authority/StellaOps.Authority/Security/AuthoritySenderConstraintKinds.cs @@ -0,0 +1,10 @@ +namespace StellaOps.Authority.Security; + +/// +/// Canonical string identifiers for Authority sender-constraint policies. +/// +internal static class AuthoritySenderConstraintKinds +{ + internal const string Dpop = "dpop"; + internal const string Mtls = "mtls"; +} diff --git a/src/StellaOps.Authority/StellaOps.Authority/Security/IAuthorityClientCertificateValidator.cs b/src/StellaOps.Authority/StellaOps.Authority/Security/IAuthorityClientCertificateValidator.cs new file mode 100644 index 00000000..6a2472a9 --- /dev/null +++ b/src/StellaOps.Authority/StellaOps.Authority/Security/IAuthorityClientCertificateValidator.cs @@ -0,0 +1,11 @@ +using System.Threading; +using System.Threading.Tasks; +using Microsoft.AspNetCore.Http; +using StellaOps.Authority.Storage.Mongo.Documents; + +namespace StellaOps.Authority.Security; + +internal interface IAuthorityClientCertificateValidator +{ + ValueTask ValidateAsync(HttpContext httpContext, AuthorityClientDocument client, CancellationToken cancellationToken); +} diff --git a/src/StellaOps.Authority/StellaOps.Authority/StellaOps.Authority.csproj b/src/StellaOps.Authority/StellaOps.Authority/StellaOps.Authority.csproj index d687ac4f..83d5f022 100644 --- a/src/StellaOps.Authority/StellaOps.Authority/StellaOps.Authority.csproj +++ b/src/StellaOps.Authority/StellaOps.Authority/StellaOps.Authority.csproj @@ -17,6 +17,7 @@ + diff --git a/src/StellaOps.Authority/TASKS.md b/src/StellaOps.Authority/TASKS.md index ac86d2f5..e3879c0d 100644 --- a/src/StellaOps.Authority/TASKS.md +++ b/src/StellaOps.Authority/TASKS.md @@ -20,10 +20,13 @@ | AUTHCORE-STORAGE-DEVICE-TOKENS | DONE (2025-10-14) | Authority Core, Storage Guild | AUTHCORE-BUILD-OPENIDDICT | Reintroduce `AuthorityTokenDeviceDocument` + projections removed during refactor so storage layer compiles. | ✅ Document type restored with mappings/migrations; ✅ Storage tests cover device artifacts; ✅ Authority solution build green. | | AUTHCORE-BOOTSTRAP-INVITES | DONE (2025-10-14) | Authority Core, DevOps | AUTHCORE-STORAGE-DEVICE-TOKENS | Wire bootstrap invite cleanup service against restored document schema and re-enable lifecycle tests. | ✅ `BootstrapInviteCleanupService` passes integration tests; ✅ Operator guide updated if behavior changes; ✅ Build/test matrices green. | | AUTHSTORAGE-MONGO-08-001 | DONE (2025-10-19) | Authority Core & Storage Guild | — | Harden Mongo session usage with causal consistency for mutations and follow-up reads. | • Scoped middleware/service creates `IClientSessionHandle` with causal consistency + majority read/write concerns
• Stores accept optional session parameter and reuse it for write + immediate reads
• GraphQL/HTTP pipelines updated to flow session through post-mutation queries
• Replica-set integration test exercises primary election and verifies read-your-write guarantees | -| AUTH-DPOP-11-001 | DOING (2025-10-19) | Authority Core & Security Guild | — | Implement DPoP proof validation + nonce handling for high-value audiences per architecture. | • DPoP proof validator verifies method/uri/hash, jwk thumbprint, and replay nonce per spec
• Nonce issuance endpoint integrated with audit + rate limits; high-value audiences enforce nonce requirement
• Integration tests cover success/failure paths (expired nonce, replay, invalid proof) and docs outline operator configuration | -| AUTH-MTLS-11-002 | DOING (2025-10-19) | Authority Core & Security Guild | — | Add OAuth mTLS client credential support with certificate-bound tokens and introspection updates. | • Client registration stores certificate bindings and enforces SAN/thumbprint validation during token issuance
• Token endpoint returns certificate-bound access tokens + PoP proof metadata; introspection reflects binding state
• End-to-end tests validate successful mTLS issuance, rejection of unbound certs, and docs capture configuration/rotation guidance | -> Remark (2025-10-19, AUTHSTORAGE-MONGO-08-001): Session accessor wired through Authority pipeline; stores accept optional sessions; added replica-set election regression test for read-your-write. -> Remark (2025-10-19, AUTH-DPOP-11-001): Prerequisites reviewed—none outstanding; status moved to DOING for Wave 0 kickoff. Design blueprint recorded in `docs/dev/authority-dpop-mtls-plan.md`. -> Remark (2025-10-19, AUTH-MTLS-11-002): Prerequisites reviewed—none outstanding; status moved to DOING for Wave 0 kickoff. mTLS flow design captured in `docs/dev/authority-dpop-mtls-plan.md`. +| AUTH-PLUGIN-COORD-08-002 | DOING (2025-10-19) | Authority Core, Plugin Platform Guild | PLUGIN-DI-08-001 | Coordinate scoped-service adoption for Authority plug-in registrars and background jobs ahead of PLUGIN-DI-08-002 implementation. | ✅ Workshop locked for 2025-10-20 15:00–16:00 UTC; ✅ Pre-read checklist in `docs/dev/authority-plugin-di-coordination.md`; ✅ Follow-up tasks captured in module backlogs before code changes begin. | +| AUTH-DPOP-11-001 | DOING (2025-10-19) | Authority Core & Security Guild | — | Implement DPoP proof validation + nonce handling for high-value audiences per architecture. | • Proof handler validates method/uri/hash + replay; nonce issuing/consumption implemented for in-memory + Redis stores
• Client credential path stamps `cnf.jkt` and persists sender metadata
• Remaining: finalize Redis configuration surface (docs/sample config), unskip nonce-challenge regression once HTTP pipeline emits high-value audiences, refresh operator docs | +> Remark (2025-10-19): DPoP handler now seeds request resources/audiences from client metadata; nonce challenge integration test re-enabled (still requires full suite once Concelier build restored). +| AUTH-MTLS-11-002 | DOING (2025-10-19) | Authority Core & Security Guild | — | Add OAuth mTLS client credential support with certificate-bound tokens and introspection updates. | • Certificate validator scaffold plus cnf stamping present; tokens persist sender thumbprints
• Remaining: provisioning/storage for certificate bindings, SAN/CA validation, introspection propagation, integration tests/docs before marking DONE | +> Remark (2025-10-19): Client provisioning accepts certificate bindings; validator enforces SAN types/CA allow-list with rotation grace; mtls integration tests updated (full suite still blocked by upstream build). +> Remark (2025-10-19, AUTHSTORAGE-MONGO-08-001): Prerequisites re-checked (none outstanding). Session accessor wired through Authority pipeline; stores accept optional sessions; added replica-set election regression test for read-your-write. +> Remark (2025-10-19, AUTH-DPOP-11-001): Handler, nonce store, and persistence hooks merged; Redis-backed configuration + end-to-end nonce enforcement still open. Full solution test blocked by `StellaOps.Concelier.Storage.Mongo` compile errors. +> Remark (2025-10-19, AUTH-MTLS-11-002): Certificate validator + cnf stamping delivered; binding storage, CA/SAN validation, integration suites outstanding before status can move to DONE. > Update status columns (TODO / DOING / DONE / BLOCKED) together with code changes. Always run `dotnet test src/StellaOps.Authority.sln` when touching host logic. diff --git a/src/StellaOps.Cli.Tests/Commands/CommandHandlersTests.cs b/src/StellaOps.Cli.Tests/Commands/CommandHandlersTests.cs index 3ca1351f..7f08abb9 100644 --- a/src/StellaOps.Cli.Tests/Commands/CommandHandlersTests.cs +++ b/src/StellaOps.Cli.Tests/Commands/CommandHandlersTests.cs @@ -2,6 +2,7 @@ using System; using System.Collections.Generic; using System.Collections.ObjectModel; using System.IO; +using System.Linq; using System.Net.Http; using System.Security.Cryptography; using System.Text; @@ -9,6 +10,7 @@ using System.Text.Json; using System.Text.Json.Serialization; using System.Threading; using System.Threading.Tasks; +using System.Globalization; using Microsoft.Extensions.DependencyInjection; using Microsoft.Extensions.Logging; using Microsoft.IdentityModel.Tokens; @@ -21,20 +23,22 @@ using StellaOps.Cli.Services.Models; using StellaOps.Cli.Telemetry; using StellaOps.Cli.Tests.Testing; using StellaOps.Cryptography; +using Spectre.Console; +using Spectre.Console.Testing; namespace StellaOps.Cli.Tests.Commands; - -public sealed class CommandHandlersTests -{ - [Fact] - public async Task HandleExportJobAsync_SetsExitCodeZeroOnSuccess() - { - var original = Environment.ExitCode; - try - { - var backend = new StubBackendClient(new JobTriggerResult(true, "Accepted", "/jobs/export:json/1", null)); - var provider = BuildServiceProvider(backend); - + +public sealed class CommandHandlersTests +{ + [Fact] + public async Task HandleExportJobAsync_SetsExitCodeZeroOnSuccess() + { + var original = Environment.ExitCode; + try + { + var backend = new StubBackendClient(new JobTriggerResult(true, "Accepted", "/jobs/export:json/1", null)); + var provider = BuildServiceProvider(backend); + await CommandHandlers.HandleExportJobAsync( provider, format: "json", @@ -45,36 +49,36 @@ public sealed class CommandHandlersTests includeDelta: null, verbose: false, cancellationToken: CancellationToken.None); - - Assert.Equal(0, Environment.ExitCode); - Assert.Equal("export:json", backend.LastJobKind); - } - finally - { - Environment.ExitCode = original; - } - } - - [Fact] - public async Task HandleMergeJobAsync_SetsExitCodeOnFailure() - { - var original = Environment.ExitCode; - try - { - var backend = new StubBackendClient(new JobTriggerResult(false, "Job already running", null, null)); - var provider = BuildServiceProvider(backend); - - await CommandHandlers.HandleMergeJobAsync(provider, verbose: false, CancellationToken.None); - - Assert.Equal(1, Environment.ExitCode); - Assert.Equal("merge:reconcile", backend.LastJobKind); - } - finally - { - Environment.ExitCode = original; - } - } - + + Assert.Equal(0, Environment.ExitCode); + Assert.Equal("export:json", backend.LastJobKind); + } + finally + { + Environment.ExitCode = original; + } + } + + [Fact] + public async Task HandleMergeJobAsync_SetsExitCodeOnFailure() + { + var original = Environment.ExitCode; + try + { + var backend = new StubBackendClient(new JobTriggerResult(false, "Job already running", null, null)); + var provider = BuildServiceProvider(backend); + + await CommandHandlers.HandleMergeJobAsync(provider, verbose: false, CancellationToken.None); + + Assert.Equal(1, Environment.ExitCode); + Assert.Equal("merge:reconcile", backend.LastJobKind); + } + finally + { + Environment.ExitCode = original; + } + } + [Fact] public async Task HandleScannerRunAsync_AutomaticallyUploadsResults() { @@ -83,34 +87,34 @@ public sealed class CommandHandlersTests var backend = new StubBackendClient(new JobTriggerResult(true, "Accepted", null, null)); var metadataFile = Path.Combine(tempDir.Path, "results", "scan-run.json"); var executor = new StubExecutor(new ScannerExecutionResult(0, resultsFile, metadataFile)); - var options = new StellaOpsCliOptions - { - ResultsDirectory = Path.Combine(tempDir.Path, "results") - }; - - var provider = BuildServiceProvider(backend, executor, new StubInstaller(), options); - - Directory.CreateDirectory(Path.Combine(tempDir.Path, "target")); - - var original = Environment.ExitCode; - try - { - await CommandHandlers.HandleScannerRunAsync( - provider, - runner: "docker", - entry: "scanner-image", - targetDirectory: Path.Combine(tempDir.Path, "target"), - arguments: Array.Empty(), - verbose: false, - cancellationToken: CancellationToken.None); - - Assert.Equal(0, Environment.ExitCode); + var options = new StellaOpsCliOptions + { + ResultsDirectory = Path.Combine(tempDir.Path, "results") + }; + + var provider = BuildServiceProvider(backend, executor, new StubInstaller(), options); + + Directory.CreateDirectory(Path.Combine(tempDir.Path, "target")); + + var original = Environment.ExitCode; + try + { + await CommandHandlers.HandleScannerRunAsync( + provider, + runner: "docker", + entry: "scanner-image", + targetDirectory: Path.Combine(tempDir.Path, "target"), + arguments: Array.Empty(), + verbose: false, + cancellationToken: CancellationToken.None); + + Assert.Equal(0, Environment.ExitCode); Assert.Equal(resultsFile, backend.LastUploadPath); Assert.True(File.Exists(metadataFile)); - } - finally - { - Environment.ExitCode = original; + } + finally + { + Environment.ExitCode = original; } } @@ -554,7 +558,219 @@ public sealed class CommandHandlersTests Environment.ExitCode = original; } } - + + [Fact] + public async Task HandleRuntimePolicyTestAsync_WritesInteractiveTable() + { + var originalExit = Environment.ExitCode; + var originalConsole = AnsiConsole.Console; + + var console = new TestConsole(); + console.Width(120); + console.Interactive(); + console.EmitAnsiSequences(); + + AnsiConsole.Console = console; + + var backend = new StubBackendClient(new JobTriggerResult(true, "ok", null, null)); + + var decisions = new Dictionary(StringComparer.Ordinal) + { + ["sha256:aaa"] = new RuntimePolicyImageDecision( + "allow", + true, + true, + Array.AsReadOnly(new[] { "trusted baseline" }), + new RuntimePolicyRekorReference("uuid-allow", "https://rekor.example/entries/uuid-allow", true), + new ReadOnlyDictionary(new Dictionary(StringComparer.Ordinal) + { + ["source"] = "baseline", + ["quieted"] = false, + ["confidence"] = 0.97, + ["confidenceBand"] = "high" + })), + ["sha256:bbb"] = new RuntimePolicyImageDecision( + "block", + false, + false, + Array.AsReadOnly(new[] { "missing attestation" }), + new RuntimePolicyRekorReference("uuid-block", "https://rekor.example/entries/uuid-block", false), + new ReadOnlyDictionary(new Dictionary(StringComparer.Ordinal) + { + ["source"] = "policy", + ["quieted"] = false, + ["confidence"] = 0.12, + ["confidenceBand"] = "low" + })), + ["sha256:ccc"] = new RuntimePolicyImageDecision( + "audit", + true, + false, + Array.AsReadOnly(new[] { "pending sbom sync" }), + new RuntimePolicyRekorReference(null, null, null), + new ReadOnlyDictionary(new Dictionary(StringComparer.Ordinal) + { + ["source"] = "mirror", + ["quieted"] = true, + ["quietedBy"] = "allow-temporary", + ["confidence"] = 0.42, + ["confidenceBand"] = "medium" + })) + }; + + backend.RuntimePolicyResult = new RuntimePolicyEvaluationResult( + 300, + DateTimeOffset.Parse("2025-10-19T12:00:00Z", CultureInfo.InvariantCulture), + "rev-42", + new ReadOnlyDictionary(decisions)); + + var provider = BuildServiceProvider(backend); + + try + { + await CommandHandlers.HandleRuntimePolicyTestAsync( + provider, + namespaceValue: "prod", + imageArguments: new[] { "sha256:aaa", "sha256:bbb" }, + filePath: null, + labelArguments: new[] { "app=frontend" }, + outputJson: false, + verbose: false, + cancellationToken: CancellationToken.None); + + var output = console.Output; + + Assert.Equal(0, Environment.ExitCode); + Assert.Contains("Image", output, StringComparison.Ordinal); + Assert.Contains("Verdict", output, StringComparison.Ordinal); + Assert.Contains("SBOM Ref", output, StringComparison.Ordinal); + Assert.Contains("Quieted", output, StringComparison.Ordinal); + Assert.Contains("Confidence", output, StringComparison.Ordinal); + Assert.Contains("sha256:aaa", output, StringComparison.Ordinal); + Assert.Contains("uuid-allow", output, StringComparison.Ordinal); + Assert.Contains("(verified)", output, StringComparison.Ordinal); + Assert.Contains("0.97 (high)", output, StringComparison.Ordinal); + Assert.Contains("sha256:bbb", output, StringComparison.Ordinal); + Assert.Contains("uuid-block", output, StringComparison.Ordinal); + Assert.Contains("(unverified)", output, StringComparison.Ordinal); + Assert.Contains("sha256:ccc", output, StringComparison.Ordinal); + Assert.Contains("yes", output, StringComparison.Ordinal); + Assert.Contains("allow-temporary", output, StringComparison.Ordinal); + Assert.True( + output.IndexOf("sha256:aaa", StringComparison.Ordinal) < + output.IndexOf("sha256:ccc", StringComparison.Ordinal)); + } + finally + { + Environment.ExitCode = originalExit; + AnsiConsole.Console = originalConsole; + } + } + + [Fact] + public async Task HandleRuntimePolicyTestAsync_WritesDeterministicJson() + { + var originalExit = Environment.ExitCode; + var originalOut = Console.Out; + + var backend = new StubBackendClient(new JobTriggerResult(true, "ok", null, null)); + + var decisions = new Dictionary(StringComparer.Ordinal) + { + ["sha256:json-a"] = new RuntimePolicyImageDecision( + "allow", + true, + true, + Array.AsReadOnly(new[] { "baseline allow" }), + new RuntimePolicyRekorReference("uuid-json-allow", "https://rekor.example/entries/uuid-json-allow", true), + new ReadOnlyDictionary(new Dictionary(StringComparer.Ordinal) + { + ["source"] = "baseline", + ["confidence"] = 0.66 + })), + ["sha256:json-b"] = new RuntimePolicyImageDecision( + "audit", + true, + false, + Array.AsReadOnly(Array.Empty()), + new RuntimePolicyRekorReference(null, null, null), + new ReadOnlyDictionary(new Dictionary(StringComparer.Ordinal) + { + ["source"] = "mirror", + ["quieted"] = true, + ["quietedBy"] = "risk-accepted" + })) + }; + + backend.RuntimePolicyResult = new RuntimePolicyEvaluationResult( + 600, + DateTimeOffset.Parse("2025-10-20T00:00:00Z", CultureInfo.InvariantCulture), + "rev-json-7", + new ReadOnlyDictionary(decisions)); + + var provider = BuildServiceProvider(backend); + + using var writer = new StringWriter(); + Console.SetOut(writer); + + try + { + await CommandHandlers.HandleRuntimePolicyTestAsync( + provider, + namespaceValue: "staging", + imageArguments: new[] { "sha256:json-a", "sha256:json-b" }, + filePath: null, + labelArguments: Array.Empty(), + outputJson: true, + verbose: false, + cancellationToken: CancellationToken.None); + + var output = writer.ToString().Trim(); + + Assert.Equal(0, Environment.ExitCode); + Assert.False(string.IsNullOrWhiteSpace(output)); + + using var document = JsonDocument.Parse(output); + var root = document.RootElement; + + Assert.Equal(600, root.GetProperty("ttlSeconds").GetInt32()); + Assert.Equal("rev-json-7", root.GetProperty("policyRevision").GetString()); + var expiresAt = root.GetProperty("expiresAtUtc").GetString(); + Assert.NotNull(expiresAt); + Assert.Equal( + DateTimeOffset.Parse("2025-10-20T00:00:00Z", CultureInfo.InvariantCulture, DateTimeStyles.AssumeUniversal), + DateTimeOffset.Parse(expiresAt!, CultureInfo.InvariantCulture, DateTimeStyles.AssumeUniversal)); + + var results = root.GetProperty("results"); + var keys = results.EnumerateObject().Select(p => p.Name).ToArray(); + Assert.Equal(new[] { "sha256:json-a", "sha256:json-b" }, keys); + + var first = results.GetProperty("sha256:json-a"); + Assert.Equal("allow", first.GetProperty("policyVerdict").GetString()); + Assert.True(first.GetProperty("signed").GetBoolean()); + Assert.True(first.GetProperty("hasSbomReferrers").GetBoolean()); + var rekor = first.GetProperty("rekor"); + Assert.Equal("uuid-json-allow", rekor.GetProperty("uuid").GetString()); + Assert.True(rekor.GetProperty("verified").GetBoolean()); + Assert.Equal("baseline", first.GetProperty("source").GetString()); + Assert.Equal(0.66, first.GetProperty("confidence").GetDouble(), 3); + + var second = results.GetProperty("sha256:json-b"); + Assert.Equal("audit", second.GetProperty("policyVerdict").GetString()); + Assert.True(second.GetProperty("signed").GetBoolean()); + Assert.False(second.GetProperty("hasSbomReferrers").GetBoolean()); + Assert.Equal("mirror", second.GetProperty("source").GetString()); + Assert.True(second.GetProperty("quieted").GetBoolean()); + Assert.Equal("risk-accepted", second.GetProperty("quietedBy").GetString()); + Assert.False(second.TryGetProperty("rekor", out _)); + } + finally + { + Console.SetOut(originalOut); + Environment.ExitCode = originalExit; + } + } + private static async Task WriteRevocationArtifactsAsync(TempDirectory temp, string? providerHint) { var (bundleBytes, signature, keyPem) = await BuildRevocationArtifactsAsync(providerHint); @@ -665,10 +881,17 @@ public sealed class CommandHandlersTests $"{Path.GetFileNameWithoutExtension(tempResultsFile)}-run.json"); return new StubExecutor(new ScannerExecutionResult(0, tempResultsFile, tempMetadataFile)); } - + private sealed class StubBackendClient : IBackendOperationsClient { private readonly JobTriggerResult _jobResult; + private static readonly RuntimePolicyEvaluationResult DefaultRuntimePolicyResult = + new RuntimePolicyEvaluationResult( + 0, + null, + null, + new ReadOnlyDictionary( + new Dictionary())); public StubBackendClient(JobTriggerResult result) { @@ -683,6 +906,7 @@ public sealed class CommandHandlersTests public List<(string ExportId, string DestinationPath, string? Algorithm, string? Digest)> ExportDownloads { get; } = new(); public ExcititorOperationResult? ExcititorResult { get; set; } = new ExcititorOperationResult(true, "ok", null, null); public IReadOnlyList ProviderSummaries { get; set; } = Array.Empty(); + public RuntimePolicyEvaluationResult RuntimePolicyResult { get; set; } = DefaultRuntimePolicyResult; public Task DownloadScannerAsync(string channel, string outputPath, bool overwrite, bool verbose, CancellationToken cancellationToken) => throw new NotImplementedException(); @@ -726,21 +950,18 @@ public sealed class CommandHandlersTests => Task.FromResult(ProviderSummaries); public Task EvaluateRuntimePolicyAsync(RuntimePolicyEvaluationRequest request, CancellationToken cancellationToken) - { - var empty = new ReadOnlyDictionary(new Dictionary()); - return Task.FromResult(new RuntimePolicyEvaluationResult(0, null, null, empty)); - } + => Task.FromResult(RuntimePolicyResult); } - - private sealed class StubExecutor : IScannerExecutor - { - private readonly ScannerExecutionResult _result; - - public StubExecutor(ScannerExecutionResult result) - { - _result = result; - } - + + private sealed class StubExecutor : IScannerExecutor + { + private readonly ScannerExecutionResult _result; + + public StubExecutor(ScannerExecutionResult result) + { + _result = result; + } + public Task RunAsync(string runner, string entry, string targetDirectory, string resultsDirectory, IReadOnlyList arguments, bool verbose, CancellationToken cancellationToken) { Directory.CreateDirectory(Path.GetDirectoryName(_result.ResultsPath)!); @@ -757,8 +978,8 @@ public sealed class CommandHandlersTests return Task.FromResult(_result); } - } - + } + private sealed class StubInstaller : IScannerInstaller { public Task InstallAsync(string artifactPath, bool verbose, CancellationToken cancellationToken) diff --git a/src/StellaOps.Cli.Tests/Services/BackendOperationsClientTests.cs b/src/StellaOps.Cli.Tests/Services/BackendOperationsClientTests.cs index aa1d27f8..0ed8a7ca 100644 --- a/src/StellaOps.Cli.Tests/Services/BackendOperationsClientTests.cs +++ b/src/StellaOps.Cli.Tests/Services/BackendOperationsClientTests.cs @@ -1,4 +1,4 @@ -using System; +using System; using System.Collections.ObjectModel; using System.Globalization; using System.IO; @@ -6,163 +6,163 @@ using System.Net; using System.Net.Http; using System.Net.Http.Json; using System.Security.Cryptography; -using System.Text; -using System.Text.Json; -using System.Threading; -using System.Threading.Tasks; +using System.Text; +using System.Text.Json; +using System.Threading; +using System.Threading.Tasks; using Microsoft.Extensions.Logging; using Microsoft.IdentityModel.Tokens; using StellaOps.Auth.Abstractions; using StellaOps.Auth.Client; -using StellaOps.Cli.Configuration; -using StellaOps.Cli.Services; -using StellaOps.Cli.Services.Models; -using StellaOps.Cli.Services.Models.Transport; -using StellaOps.Cli.Tests.Testing; - -namespace StellaOps.Cli.Tests.Services; - -public sealed class BackendOperationsClientTests -{ - [Fact] - public async Task DownloadScannerAsync_VerifiesDigestAndWritesMetadata() - { - using var temp = new TempDirectory(); - - var contentBytes = Encoding.UTF8.GetBytes("scanner-blob"); - var digestHex = Convert.ToHexString(SHA256.HashData(contentBytes)).ToLowerInvariant(); - - var handler = new StubHttpMessageHandler((request, _) => - { - var response = new HttpResponseMessage(HttpStatusCode.OK) - { - Content = new ByteArrayContent(contentBytes), - RequestMessage = request - }; - - response.Headers.Add("X-StellaOps-Digest", $"sha256:{digestHex}"); - response.Content.Headers.LastModified = DateTimeOffset.UtcNow; - response.Content.Headers.ContentType = new System.Net.Http.Headers.MediaTypeHeaderValue("application/octet-stream"); - return response; - }); - - var httpClient = new HttpClient(handler) - { - BaseAddress = new Uri("https://concelier.example") - }; - - var options = new StellaOpsCliOptions - { - BackendUrl = "https://concelier.example", - ScannerCacheDirectory = temp.Path, - ScannerDownloadAttempts = 1 - }; - - var loggerFactory = LoggerFactory.Create(builder => builder.SetMinimumLevel(LogLevel.Debug)); - var client = new BackendOperationsClient(httpClient, options, loggerFactory.CreateLogger()); - - var targetPath = Path.Combine(temp.Path, "scanner.tar.gz"); - var result = await client.DownloadScannerAsync("stable", targetPath, overwrite: false, verbose: true, CancellationToken.None); - - Assert.False(result.FromCache); - Assert.True(File.Exists(targetPath)); - - var metadataPath = targetPath + ".metadata.json"; - Assert.True(File.Exists(metadataPath)); - - using var document = JsonDocument.Parse(File.ReadAllText(metadataPath)); - Assert.Equal($"sha256:{digestHex}", document.RootElement.GetProperty("digest").GetString()); - Assert.Equal("stable", document.RootElement.GetProperty("channel").GetString()); - } - - [Fact] - public async Task DownloadScannerAsync_ThrowsOnDigestMismatch() - { - using var temp = new TempDirectory(); - - var contentBytes = Encoding.UTF8.GetBytes("scanner-data"); - var handler = new StubHttpMessageHandler((request, _) => - { - var response = new HttpResponseMessage(HttpStatusCode.OK) - { - Content = new ByteArrayContent(contentBytes), - RequestMessage = request - }; - response.Headers.Add("X-StellaOps-Digest", "sha256:deadbeef"); - return response; - }); - - var httpClient = new HttpClient(handler) - { - BaseAddress = new Uri("https://concelier.example") - }; - - var options = new StellaOpsCliOptions - { - BackendUrl = "https://concelier.example", - ScannerCacheDirectory = temp.Path, - ScannerDownloadAttempts = 1 - }; - - var loggerFactory = LoggerFactory.Create(builder => builder.SetMinimumLevel(LogLevel.Debug)); - var client = new BackendOperationsClient(httpClient, options, loggerFactory.CreateLogger()); - - var targetPath = Path.Combine(temp.Path, "scanner.tar.gz"); - - await Assert.ThrowsAsync(() => client.DownloadScannerAsync("stable", targetPath, overwrite: true, verbose: false, CancellationToken.None)); - Assert.False(File.Exists(targetPath)); - } - +using StellaOps.Cli.Configuration; +using StellaOps.Cli.Services; +using StellaOps.Cli.Services.Models; +using StellaOps.Cli.Services.Models.Transport; +using StellaOps.Cli.Tests.Testing; + +namespace StellaOps.Cli.Tests.Services; + +public sealed class BackendOperationsClientTests +{ + [Fact] + public async Task DownloadScannerAsync_VerifiesDigestAndWritesMetadata() + { + using var temp = new TempDirectory(); + + var contentBytes = Encoding.UTF8.GetBytes("scanner-blob"); + var digestHex = Convert.ToHexString(SHA256.HashData(contentBytes)).ToLowerInvariant(); + + var handler = new StubHttpMessageHandler((request, _) => + { + var response = new HttpResponseMessage(HttpStatusCode.OK) + { + Content = new ByteArrayContent(contentBytes), + RequestMessage = request + }; + + response.Headers.Add("X-StellaOps-Digest", $"sha256:{digestHex}"); + response.Content.Headers.LastModified = DateTimeOffset.UtcNow; + response.Content.Headers.ContentType = new System.Net.Http.Headers.MediaTypeHeaderValue("application/octet-stream"); + return response; + }); + + var httpClient = new HttpClient(handler) + { + BaseAddress = new Uri("https://concelier.example") + }; + + var options = new StellaOpsCliOptions + { + BackendUrl = "https://concelier.example", + ScannerCacheDirectory = temp.Path, + ScannerDownloadAttempts = 1 + }; + + var loggerFactory = LoggerFactory.Create(builder => builder.SetMinimumLevel(LogLevel.Debug)); + var client = new BackendOperationsClient(httpClient, options, loggerFactory.CreateLogger()); + + var targetPath = Path.Combine(temp.Path, "scanner.tar.gz"); + var result = await client.DownloadScannerAsync("stable", targetPath, overwrite: false, verbose: true, CancellationToken.None); + + Assert.False(result.FromCache); + Assert.True(File.Exists(targetPath)); + + var metadataPath = targetPath + ".metadata.json"; + Assert.True(File.Exists(metadataPath)); + + using var document = JsonDocument.Parse(File.ReadAllText(metadataPath)); + Assert.Equal($"sha256:{digestHex}", document.RootElement.GetProperty("digest").GetString()); + Assert.Equal("stable", document.RootElement.GetProperty("channel").GetString()); + } + + [Fact] + public async Task DownloadScannerAsync_ThrowsOnDigestMismatch() + { + using var temp = new TempDirectory(); + + var contentBytes = Encoding.UTF8.GetBytes("scanner-data"); + var handler = new StubHttpMessageHandler((request, _) => + { + var response = new HttpResponseMessage(HttpStatusCode.OK) + { + Content = new ByteArrayContent(contentBytes), + RequestMessage = request + }; + response.Headers.Add("X-StellaOps-Digest", "sha256:deadbeef"); + return response; + }); + + var httpClient = new HttpClient(handler) + { + BaseAddress = new Uri("https://concelier.example") + }; + + var options = new StellaOpsCliOptions + { + BackendUrl = "https://concelier.example", + ScannerCacheDirectory = temp.Path, + ScannerDownloadAttempts = 1 + }; + + var loggerFactory = LoggerFactory.Create(builder => builder.SetMinimumLevel(LogLevel.Debug)); + var client = new BackendOperationsClient(httpClient, options, loggerFactory.CreateLogger()); + + var targetPath = Path.Combine(temp.Path, "scanner.tar.gz"); + + await Assert.ThrowsAsync(() => client.DownloadScannerAsync("stable", targetPath, overwrite: true, verbose: false, CancellationToken.None)); + Assert.False(File.Exists(targetPath)); + } + [Fact] public async Task DownloadScannerAsync_RetriesOnFailure() { using var temp = new TempDirectory(); - var successBytes = Encoding.UTF8.GetBytes("success"); - var digestHex = Convert.ToHexString(SHA256.HashData(successBytes)).ToLowerInvariant(); - var attempts = 0; - - var handler = new StubHttpMessageHandler( - (request, _) => - { - attempts++; - return new HttpResponseMessage(HttpStatusCode.InternalServerError) - { - RequestMessage = request, - Content = new StringContent("error") - }; - }, - (request, _) => - { - attempts++; - var response = new HttpResponseMessage(HttpStatusCode.OK) - { - RequestMessage = request, - Content = new ByteArrayContent(successBytes) - }; - response.Headers.Add("X-StellaOps-Digest", $"sha256:{digestHex}"); - return response; - }); - - var httpClient = new HttpClient(handler) - { - BaseAddress = new Uri("https://concelier.example") - }; - - var options = new StellaOpsCliOptions - { - BackendUrl = "https://concelier.example", - ScannerCacheDirectory = temp.Path, - ScannerDownloadAttempts = 3 - }; - - var loggerFactory = LoggerFactory.Create(builder => builder.SetMinimumLevel(LogLevel.Debug)); - var client = new BackendOperationsClient(httpClient, options, loggerFactory.CreateLogger()); - - var targetPath = Path.Combine(temp.Path, "scanner.tar.gz"); - var result = await client.DownloadScannerAsync("stable", targetPath, overwrite: false, verbose: false, CancellationToken.None); - - Assert.Equal(2, attempts); + var successBytes = Encoding.UTF8.GetBytes("success"); + var digestHex = Convert.ToHexString(SHA256.HashData(successBytes)).ToLowerInvariant(); + var attempts = 0; + + var handler = new StubHttpMessageHandler( + (request, _) => + { + attempts++; + return new HttpResponseMessage(HttpStatusCode.InternalServerError) + { + RequestMessage = request, + Content = new StringContent("error") + }; + }, + (request, _) => + { + attempts++; + var response = new HttpResponseMessage(HttpStatusCode.OK) + { + RequestMessage = request, + Content = new ByteArrayContent(successBytes) + }; + response.Headers.Add("X-StellaOps-Digest", $"sha256:{digestHex}"); + return response; + }); + + var httpClient = new HttpClient(handler) + { + BaseAddress = new Uri("https://concelier.example") + }; + + var options = new StellaOpsCliOptions + { + BackendUrl = "https://concelier.example", + ScannerCacheDirectory = temp.Path, + ScannerDownloadAttempts = 3 + }; + + var loggerFactory = LoggerFactory.Create(builder => builder.SetMinimumLevel(LogLevel.Debug)); + var client = new BackendOperationsClient(httpClient, options, loggerFactory.CreateLogger()); + + var targetPath = Path.Combine(temp.Path, "scanner.tar.gz"); + var result = await client.DownloadScannerAsync("stable", targetPath, overwrite: false, verbose: false, CancellationToken.None); + + Assert.Equal(2, attempts); Assert.False(result.FromCache); Assert.True(File.Exists(targetPath)); } @@ -251,73 +251,73 @@ public sealed class BackendOperationsClientTests await Assert.ThrowsAsync(() => client.UploadScanResultsAsync(filePath, CancellationToken.None)); Assert.Equal(2, attempts); } - - [Fact] - public async Task TriggerJobAsync_ReturnsAcceptedResult() - { - var handler = new StubHttpMessageHandler((request, _) => - { - var response = new HttpResponseMessage(HttpStatusCode.Accepted) - { - RequestMessage = request, - Content = JsonContent.Create(new JobRunResponse - { - RunId = Guid.NewGuid(), - Status = "queued", - Kind = "export:json", - Trigger = "cli", - CreatedAt = DateTimeOffset.UtcNow - }) - }; - response.Headers.Location = new Uri("/jobs/export:json/runs/123", UriKind.Relative); - return response; - }); - - var httpClient = new HttpClient(handler) - { - BaseAddress = new Uri("https://concelier.example") - }; - - var options = new StellaOpsCliOptions { BackendUrl = "https://concelier.example" }; - var loggerFactory = LoggerFactory.Create(builder => builder.SetMinimumLevel(LogLevel.Debug)); - var client = new BackendOperationsClient(httpClient, options, loggerFactory.CreateLogger()); - - var result = await client.TriggerJobAsync("export:json", new Dictionary(), CancellationToken.None); - - Assert.True(result.Success); - Assert.Equal("Accepted", result.Message); - Assert.Equal("/jobs/export:json/runs/123", result.Location); - } - - [Fact] + + [Fact] + public async Task TriggerJobAsync_ReturnsAcceptedResult() + { + var handler = new StubHttpMessageHandler((request, _) => + { + var response = new HttpResponseMessage(HttpStatusCode.Accepted) + { + RequestMessage = request, + Content = JsonContent.Create(new JobRunResponse + { + RunId = Guid.NewGuid(), + Status = "queued", + Kind = "export:json", + Trigger = "cli", + CreatedAt = DateTimeOffset.UtcNow + }) + }; + response.Headers.Location = new Uri("/jobs/export:json/runs/123", UriKind.Relative); + return response; + }); + + var httpClient = new HttpClient(handler) + { + BaseAddress = new Uri("https://concelier.example") + }; + + var options = new StellaOpsCliOptions { BackendUrl = "https://concelier.example" }; + var loggerFactory = LoggerFactory.Create(builder => builder.SetMinimumLevel(LogLevel.Debug)); + var client = new BackendOperationsClient(httpClient, options, loggerFactory.CreateLogger()); + + var result = await client.TriggerJobAsync("export:json", new Dictionary(), CancellationToken.None); + + Assert.True(result.Success); + Assert.Equal("Accepted", result.Message); + Assert.Equal("/jobs/export:json/runs/123", result.Location); + } + + [Fact] public async Task TriggerJobAsync_ReturnsFailureMessage() { var handler = new StubHttpMessageHandler((request, _) => { var problem = new - { - title = "Job already running", - detail = "export job active" - }; - - var response = new HttpResponseMessage(HttpStatusCode.Conflict) - { - RequestMessage = request, - Content = JsonContent.Create(problem) - }; - return response; - }); - - var httpClient = new HttpClient(handler) - { - BaseAddress = new Uri("https://concelier.example") - }; - - var options = new StellaOpsCliOptions { BackendUrl = "https://concelier.example" }; - var loggerFactory = LoggerFactory.Create(builder => builder.SetMinimumLevel(LogLevel.Debug)); - var client = new BackendOperationsClient(httpClient, options, loggerFactory.CreateLogger()); - - var result = await client.TriggerJobAsync("export:json", new Dictionary(), CancellationToken.None); + { + title = "Job already running", + detail = "export job active" + }; + + var response = new HttpResponseMessage(HttpStatusCode.Conflict) + { + RequestMessage = request, + Content = JsonContent.Create(problem) + }; + return response; + }); + + var httpClient = new HttpClient(handler) + { + BaseAddress = new Uri("https://concelier.example") + }; + + var options = new StellaOpsCliOptions { BackendUrl = "https://concelier.example" }; + var loggerFactory = LoggerFactory.Create(builder => builder.SetMinimumLevel(LogLevel.Debug)); + var client = new BackendOperationsClient(httpClient, options, loggerFactory.CreateLogger()); + + var result = await client.TriggerJobAsync("export:json", new Dictionary(), CancellationToken.None); Assert.False(result.Success); Assert.Contains("Job already running", result.Message); @@ -403,18 +403,19 @@ public sealed class BackendOperationsClientTests ""ghcr.io/app@sha256:abc"": { ""policyVerdict"": ""pass"", ""signed"": true, - ""hasSbom"": true, + ""hasSbomReferrers"": true, ""reasons"": [], - ""rekor"": { ""uuid"": ""uuid-1"", ""url"": ""https://rekor.example/uuid-1"" }, + ""rekor"": { ""uuid"": ""uuid-1"", ""url"": ""https://rekor.example/uuid-1"", ""verified"": true }, ""confidence"": 0.87, - ""quiet"": false, + ""quieted"": false, ""metadata"": { ""note"": ""cached"" } }, ""ghcr.io/api@sha256:def"": { ""policyVerdict"": ""fail"", ""signed"": false, - ""hasSbom"": false, - ""reasons"": [""unsigned"", ""missing sbom""] + ""hasSbomReferrers"": false, + ""reasons"": [""unsigned"", ""missing sbom""], + ""quietedBy"": ""manual-override"" } } }"; @@ -458,13 +459,14 @@ public sealed class BackendOperationsClientTests var primary = result.Decisions["ghcr.io/app@sha256:abc"]; Assert.Equal("pass", primary.PolicyVerdict); Assert.True(primary.Signed); - Assert.True(primary.HasSbom); + Assert.True(primary.HasSbomReferrers); Assert.Empty(primary.Reasons); Assert.NotNull(primary.Rekor); Assert.Equal("uuid-1", primary.Rekor!.Uuid); Assert.Equal("https://rekor.example/uuid-1", primary.Rekor.Url); + Assert.True(primary.Rekor.Verified); Assert.Equal(0.87, Assert.IsType(primary.AdditionalProperties["confidence"]), 3); - Assert.False(Assert.IsType(primary.AdditionalProperties["quiet"])); + Assert.False(Assert.IsType(primary.AdditionalProperties["quieted"])); var metadataJson = Assert.IsType(primary.AdditionalProperties["metadata"]); using var metadataDocument = JsonDocument.Parse(metadataJson); Assert.Equal("cached", metadataDocument.RootElement.GetProperty("note").GetString()); @@ -472,10 +474,11 @@ public sealed class BackendOperationsClientTests var secondary = result.Decisions["ghcr.io/api@sha256:def"]; Assert.Equal("fail", secondary.PolicyVerdict); Assert.False(secondary.Signed); - Assert.False(secondary.HasSbom); + Assert.False(secondary.HasSbomReferrers); Assert.Collection(secondary.Reasons, item => Assert.Equal("unsigned", item), item => Assert.Equal("missing sbom", item)); + Assert.Equal("manual-override", Assert.IsType(secondary.AdditionalProperties["quietedBy"])); } private sealed class StubTokenClient : IStellaOpsTokenClient diff --git a/src/StellaOps.Cli.Tests/StellaOps.Cli.Tests.csproj b/src/StellaOps.Cli.Tests/StellaOps.Cli.Tests.csproj index ab4b3308..ba54eb41 100644 --- a/src/StellaOps.Cli.Tests/StellaOps.Cli.Tests.csproj +++ b/src/StellaOps.Cli.Tests/StellaOps.Cli.Tests.csproj @@ -16,13 +16,14 @@ - - - - - - - - - - + + + + + + + + + + + diff --git a/src/StellaOps.Cli/Commands/CommandFactory.cs b/src/StellaOps.Cli/Commands/CommandFactory.cs index 40868b2e..aa0230a5 100644 --- a/src/StellaOps.Cli/Commands/CommandFactory.cs +++ b/src/StellaOps.Cli/Commands/CommandFactory.cs @@ -358,6 +358,48 @@ internal static class CommandFactory return CommandHandlers.HandleExcititorExportAsync(services, format, delta, scope, since, provider, output, verbose, cancellationToken); }); + var backfill = new Command("backfill-statements", "Replay historical raw documents into Excititor statements."); + var backfillRetrievedSinceOption = new Option("--retrieved-since") + { + Description = "Only process raw documents retrieved on or after the provided ISO-8601 timestamp." + }; + var backfillForceOption = new Option("--force") + { + Description = "Reprocess documents even if statements already exist." + }; + var backfillBatchSizeOption = new Option("--batch-size") + { + Description = "Number of raw documents to fetch per batch (default 100)." + }; + var backfillMaxDocumentsOption = new Option("--max-documents") + { + Description = "Optional maximum number of raw documents to process." + }; + backfill.Add(backfillRetrievedSinceOption); + backfill.Add(backfillForceOption); + backfill.Add(backfillBatchSizeOption); + backfill.Add(backfillMaxDocumentsOption); + backfill.SetAction((parseResult, _) => + { + var retrievedSince = parseResult.GetValue(backfillRetrievedSinceOption); + var force = parseResult.GetValue(backfillForceOption); + var batchSize = parseResult.GetValue(backfillBatchSizeOption); + if (batchSize <= 0) + { + batchSize = 100; + } + var maxDocuments = parseResult.GetValue(backfillMaxDocumentsOption); + var verbose = parseResult.GetValue(verboseOption); + return CommandHandlers.HandleExcititorBackfillStatementsAsync( + services, + retrievedSince, + force, + batchSize, + maxDocuments, + verbose, + cancellationToken); + }); + var verify = new Command("verify", "Verify Excititor exports or attestations."); var exportIdOption = new Option("--export-id") { @@ -408,6 +450,7 @@ internal static class CommandFactory excititor.Add(resume); excititor.Add(list); excititor.Add(export); + excititor.Add(backfill); excititor.Add(verify); excititor.Add(reconcile); return excititor; diff --git a/src/StellaOps.Cli/Commands/CommandHandlers.cs b/src/StellaOps.Cli/Commands/CommandHandlers.cs index ee38d28d..2b06ce48 100644 --- a/src/StellaOps.Cli/Commands/CommandHandlers.cs +++ b/src/StellaOps.Cli/Commands/CommandHandlers.cs @@ -25,103 +25,103 @@ using StellaOps.Cli.Telemetry; using StellaOps.Cryptography; namespace StellaOps.Cli.Commands; - -internal static class CommandHandlers -{ - public static async Task HandleScannerDownloadAsync( - IServiceProvider services, - string channel, - string? output, - bool overwrite, - bool install, - bool verbose, - CancellationToken cancellationToken) - { - await using var scope = services.CreateAsyncScope(); - var client = scope.ServiceProvider.GetRequiredService(); - var logger = scope.ServiceProvider.GetRequiredService().CreateLogger("scanner-download"); - var verbosity = scope.ServiceProvider.GetRequiredService(); - var previousLevel = verbosity.MinimumLevel; - verbosity.MinimumLevel = verbose ? LogLevel.Debug : LogLevel.Information; - using var activity = CliActivitySource.Instance.StartActivity("cli.scanner.download", ActivityKind.Client); - activity?.SetTag("stellaops.cli.command", "scanner download"); - activity?.SetTag("stellaops.cli.channel", channel); - using var duration = CliMetrics.MeasureCommandDuration("scanner download"); - - try - { - var result = await client.DownloadScannerAsync(channel, output ?? string.Empty, overwrite, verbose, cancellationToken).ConfigureAwait(false); - - if (result.FromCache) - { - logger.LogInformation("Using cached scanner at {Path}.", result.Path); - } - else - { - logger.LogInformation("Scanner downloaded to {Path} ({Size} bytes).", result.Path, result.SizeBytes); - } - - CliMetrics.RecordScannerDownload(channel, result.FromCache); - - if (install) - { - var installer = scope.ServiceProvider.GetRequiredService(); - await installer.InstallAsync(result.Path, verbose, cancellationToken).ConfigureAwait(false); - CliMetrics.RecordScannerInstall(channel); - } - - Environment.ExitCode = 0; - } - catch (Exception ex) - { - logger.LogError(ex, "Failed to download scanner bundle."); - Environment.ExitCode = 1; - } - finally - { - verbosity.MinimumLevel = previousLevel; - } - } - - public static async Task HandleScannerRunAsync( - IServiceProvider services, - string runner, - string entry, - string targetDirectory, - IReadOnlyList arguments, - bool verbose, - CancellationToken cancellationToken) - { - await using var scope = services.CreateAsyncScope(); - var executor = scope.ServiceProvider.GetRequiredService(); - var logger = scope.ServiceProvider.GetRequiredService().CreateLogger("scanner-run"); - var verbosity = scope.ServiceProvider.GetRequiredService(); - var previousLevel = verbosity.MinimumLevel; - verbosity.MinimumLevel = verbose ? LogLevel.Debug : LogLevel.Information; - using var activity = CliActivitySource.Instance.StartActivity("cli.scan.run", ActivityKind.Internal); - activity?.SetTag("stellaops.cli.command", "scan run"); - activity?.SetTag("stellaops.cli.runner", runner); - activity?.SetTag("stellaops.cli.entry", entry); - activity?.SetTag("stellaops.cli.target", targetDirectory); - using var duration = CliMetrics.MeasureCommandDuration("scan run"); - - try - { - var options = scope.ServiceProvider.GetRequiredService(); - var resultsDirectory = options.ResultsDirectory; - - var executionResult = await executor.RunAsync( - runner, - entry, - targetDirectory, - resultsDirectory, - arguments, - verbose, - cancellationToken).ConfigureAwait(false); - - Environment.ExitCode = executionResult.ExitCode; - CliMetrics.RecordScanRun(runner, executionResult.ExitCode); - + +internal static class CommandHandlers +{ + public static async Task HandleScannerDownloadAsync( + IServiceProvider services, + string channel, + string? output, + bool overwrite, + bool install, + bool verbose, + CancellationToken cancellationToken) + { + await using var scope = services.CreateAsyncScope(); + var client = scope.ServiceProvider.GetRequiredService(); + var logger = scope.ServiceProvider.GetRequiredService().CreateLogger("scanner-download"); + var verbosity = scope.ServiceProvider.GetRequiredService(); + var previousLevel = verbosity.MinimumLevel; + verbosity.MinimumLevel = verbose ? LogLevel.Debug : LogLevel.Information; + using var activity = CliActivitySource.Instance.StartActivity("cli.scanner.download", ActivityKind.Client); + activity?.SetTag("stellaops.cli.command", "scanner download"); + activity?.SetTag("stellaops.cli.channel", channel); + using var duration = CliMetrics.MeasureCommandDuration("scanner download"); + + try + { + var result = await client.DownloadScannerAsync(channel, output ?? string.Empty, overwrite, verbose, cancellationToken).ConfigureAwait(false); + + if (result.FromCache) + { + logger.LogInformation("Using cached scanner at {Path}.", result.Path); + } + else + { + logger.LogInformation("Scanner downloaded to {Path} ({Size} bytes).", result.Path, result.SizeBytes); + } + + CliMetrics.RecordScannerDownload(channel, result.FromCache); + + if (install) + { + var installer = scope.ServiceProvider.GetRequiredService(); + await installer.InstallAsync(result.Path, verbose, cancellationToken).ConfigureAwait(false); + CliMetrics.RecordScannerInstall(channel); + } + + Environment.ExitCode = 0; + } + catch (Exception ex) + { + logger.LogError(ex, "Failed to download scanner bundle."); + Environment.ExitCode = 1; + } + finally + { + verbosity.MinimumLevel = previousLevel; + } + } + + public static async Task HandleScannerRunAsync( + IServiceProvider services, + string runner, + string entry, + string targetDirectory, + IReadOnlyList arguments, + bool verbose, + CancellationToken cancellationToken) + { + await using var scope = services.CreateAsyncScope(); + var executor = scope.ServiceProvider.GetRequiredService(); + var logger = scope.ServiceProvider.GetRequiredService().CreateLogger("scanner-run"); + var verbosity = scope.ServiceProvider.GetRequiredService(); + var previousLevel = verbosity.MinimumLevel; + verbosity.MinimumLevel = verbose ? LogLevel.Debug : LogLevel.Information; + using var activity = CliActivitySource.Instance.StartActivity("cli.scan.run", ActivityKind.Internal); + activity?.SetTag("stellaops.cli.command", "scan run"); + activity?.SetTag("stellaops.cli.runner", runner); + activity?.SetTag("stellaops.cli.entry", entry); + activity?.SetTag("stellaops.cli.target", targetDirectory); + using var duration = CliMetrics.MeasureCommandDuration("scan run"); + + try + { + var options = scope.ServiceProvider.GetRequiredService(); + var resultsDirectory = options.ResultsDirectory; + + var executionResult = await executor.RunAsync( + runner, + entry, + targetDirectory, + resultsDirectory, + arguments, + verbose, + cancellationToken).ConfigureAwait(false); + + Environment.ExitCode = executionResult.ExitCode; + CliMetrics.RecordScanRun(runner, executionResult.ExitCode); + if (executionResult.ExitCode == 0) { var backend = scope.ServiceProvider.GetRequiredService(); @@ -138,128 +138,128 @@ internal static class CommandHandlers logger.LogInformation("Run metadata written to {Path}.", executionResult.RunMetadataPath); activity?.SetTag("stellaops.cli.run_metadata", executionResult.RunMetadataPath); } - catch (Exception ex) - { - logger.LogError(ex, "Scanner execution failed."); - Environment.ExitCode = 1; - } - finally - { - verbosity.MinimumLevel = previousLevel; - } - } - - public static async Task HandleScanUploadAsync( - IServiceProvider services, - string file, - bool verbose, - CancellationToken cancellationToken) - { - await using var scope = services.CreateAsyncScope(); - var client = scope.ServiceProvider.GetRequiredService(); - var logger = scope.ServiceProvider.GetRequiredService().CreateLogger("scanner-upload"); - var verbosity = scope.ServiceProvider.GetRequiredService(); - var previousLevel = verbosity.MinimumLevel; - verbosity.MinimumLevel = verbose ? LogLevel.Debug : LogLevel.Information; - using var activity = CliActivitySource.Instance.StartActivity("cli.scan.upload", ActivityKind.Client); - activity?.SetTag("stellaops.cli.command", "scan upload"); - activity?.SetTag("stellaops.cli.file", file); - using var duration = CliMetrics.MeasureCommandDuration("scan upload"); - - try - { - var path = Path.GetFullPath(file); - await client.UploadScanResultsAsync(path, cancellationToken).ConfigureAwait(false); - logger.LogInformation("Scan results uploaded successfully."); - Environment.ExitCode = 0; - } - catch (Exception ex) - { - logger.LogError(ex, "Failed to upload scan results."); - Environment.ExitCode = 1; - } - finally - { - verbosity.MinimumLevel = previousLevel; - } - } - - public static async Task HandleConnectorJobAsync( - IServiceProvider services, - string source, - string stage, - string? mode, - bool verbose, - CancellationToken cancellationToken) - { - await using var scope = services.CreateAsyncScope(); - var client = scope.ServiceProvider.GetRequiredService(); - var logger = scope.ServiceProvider.GetRequiredService().CreateLogger("db-connector"); - var verbosity = scope.ServiceProvider.GetRequiredService(); - var previousLevel = verbosity.MinimumLevel; - verbosity.MinimumLevel = verbose ? LogLevel.Debug : LogLevel.Information; - using var activity = CliActivitySource.Instance.StartActivity("cli.db.fetch", ActivityKind.Client); - activity?.SetTag("stellaops.cli.command", "db fetch"); - activity?.SetTag("stellaops.cli.source", source); - activity?.SetTag("stellaops.cli.stage", stage); - if (!string.IsNullOrWhiteSpace(mode)) - { - activity?.SetTag("stellaops.cli.mode", mode); - } - using var duration = CliMetrics.MeasureCommandDuration("db fetch"); - - try - { - var jobKind = $"source:{source}:{stage}"; - var parameters = new Dictionary(StringComparer.Ordinal); - if (!string.IsNullOrWhiteSpace(mode)) - { - parameters["mode"] = mode; - } - - await TriggerJobAsync(client, logger, jobKind, parameters, cancellationToken).ConfigureAwait(false); - } - catch (Exception ex) - { - logger.LogError(ex, "Connector job failed."); - Environment.ExitCode = 1; - } - finally - { - verbosity.MinimumLevel = previousLevel; - } - } - - public static async Task HandleMergeJobAsync( - IServiceProvider services, - bool verbose, - CancellationToken cancellationToken) - { - await using var scope = services.CreateAsyncScope(); - var client = scope.ServiceProvider.GetRequiredService(); - var logger = scope.ServiceProvider.GetRequiredService().CreateLogger("db-merge"); - var verbosity = scope.ServiceProvider.GetRequiredService(); - var previousLevel = verbosity.MinimumLevel; - verbosity.MinimumLevel = verbose ? LogLevel.Debug : LogLevel.Information; - using var activity = CliActivitySource.Instance.StartActivity("cli.db.merge", ActivityKind.Client); - activity?.SetTag("stellaops.cli.command", "db merge"); - using var duration = CliMetrics.MeasureCommandDuration("db merge"); - - try - { - await TriggerJobAsync(client, logger, "merge:reconcile", new Dictionary(StringComparer.Ordinal), cancellationToken).ConfigureAwait(false); - } - catch (Exception ex) - { - logger.LogError(ex, "Merge job failed."); - Environment.ExitCode = 1; - } - finally - { - verbosity.MinimumLevel = previousLevel; - } - } - + catch (Exception ex) + { + logger.LogError(ex, "Scanner execution failed."); + Environment.ExitCode = 1; + } + finally + { + verbosity.MinimumLevel = previousLevel; + } + } + + public static async Task HandleScanUploadAsync( + IServiceProvider services, + string file, + bool verbose, + CancellationToken cancellationToken) + { + await using var scope = services.CreateAsyncScope(); + var client = scope.ServiceProvider.GetRequiredService(); + var logger = scope.ServiceProvider.GetRequiredService().CreateLogger("scanner-upload"); + var verbosity = scope.ServiceProvider.GetRequiredService(); + var previousLevel = verbosity.MinimumLevel; + verbosity.MinimumLevel = verbose ? LogLevel.Debug : LogLevel.Information; + using var activity = CliActivitySource.Instance.StartActivity("cli.scan.upload", ActivityKind.Client); + activity?.SetTag("stellaops.cli.command", "scan upload"); + activity?.SetTag("stellaops.cli.file", file); + using var duration = CliMetrics.MeasureCommandDuration("scan upload"); + + try + { + var path = Path.GetFullPath(file); + await client.UploadScanResultsAsync(path, cancellationToken).ConfigureAwait(false); + logger.LogInformation("Scan results uploaded successfully."); + Environment.ExitCode = 0; + } + catch (Exception ex) + { + logger.LogError(ex, "Failed to upload scan results."); + Environment.ExitCode = 1; + } + finally + { + verbosity.MinimumLevel = previousLevel; + } + } + + public static async Task HandleConnectorJobAsync( + IServiceProvider services, + string source, + string stage, + string? mode, + bool verbose, + CancellationToken cancellationToken) + { + await using var scope = services.CreateAsyncScope(); + var client = scope.ServiceProvider.GetRequiredService(); + var logger = scope.ServiceProvider.GetRequiredService().CreateLogger("db-connector"); + var verbosity = scope.ServiceProvider.GetRequiredService(); + var previousLevel = verbosity.MinimumLevel; + verbosity.MinimumLevel = verbose ? LogLevel.Debug : LogLevel.Information; + using var activity = CliActivitySource.Instance.StartActivity("cli.db.fetch", ActivityKind.Client); + activity?.SetTag("stellaops.cli.command", "db fetch"); + activity?.SetTag("stellaops.cli.source", source); + activity?.SetTag("stellaops.cli.stage", stage); + if (!string.IsNullOrWhiteSpace(mode)) + { + activity?.SetTag("stellaops.cli.mode", mode); + } + using var duration = CliMetrics.MeasureCommandDuration("db fetch"); + + try + { + var jobKind = $"source:{source}:{stage}"; + var parameters = new Dictionary(StringComparer.Ordinal); + if (!string.IsNullOrWhiteSpace(mode)) + { + parameters["mode"] = mode; + } + + await TriggerJobAsync(client, logger, jobKind, parameters, cancellationToken).ConfigureAwait(false); + } + catch (Exception ex) + { + logger.LogError(ex, "Connector job failed."); + Environment.ExitCode = 1; + } + finally + { + verbosity.MinimumLevel = previousLevel; + } + } + + public static async Task HandleMergeJobAsync( + IServiceProvider services, + bool verbose, + CancellationToken cancellationToken) + { + await using var scope = services.CreateAsyncScope(); + var client = scope.ServiceProvider.GetRequiredService(); + var logger = scope.ServiceProvider.GetRequiredService().CreateLogger("db-merge"); + var verbosity = scope.ServiceProvider.GetRequiredService(); + var previousLevel = verbosity.MinimumLevel; + verbosity.MinimumLevel = verbose ? LogLevel.Debug : LogLevel.Information; + using var activity = CliActivitySource.Instance.StartActivity("cli.db.merge", ActivityKind.Client); + activity?.SetTag("stellaops.cli.command", "db merge"); + using var duration = CliMetrics.MeasureCommandDuration("db merge"); + + try + { + await TriggerJobAsync(client, logger, "merge:reconcile", new Dictionary(StringComparer.Ordinal), cancellationToken).ConfigureAwait(false); + } + catch (Exception ex) + { + logger.LogError(ex, "Merge job failed."); + Environment.ExitCode = 1; + } + finally + { + verbosity.MinimumLevel = previousLevel; + } + } + public static async Task HandleExportJobAsync( IServiceProvider services, string format, @@ -271,16 +271,16 @@ internal static class CommandHandlers bool verbose, CancellationToken cancellationToken) { - await using var scope = services.CreateAsyncScope(); - var client = scope.ServiceProvider.GetRequiredService(); - var logger = scope.ServiceProvider.GetRequiredService().CreateLogger("db-export"); - var verbosity = scope.ServiceProvider.GetRequiredService(); - var previousLevel = verbosity.MinimumLevel; - verbosity.MinimumLevel = verbose ? LogLevel.Debug : LogLevel.Information; - using var activity = CliActivitySource.Instance.StartActivity("cli.db.export", ActivityKind.Client); - activity?.SetTag("stellaops.cli.command", "db export"); - activity?.SetTag("stellaops.cli.format", format); - activity?.SetTag("stellaops.cli.delta", delta); + await using var scope = services.CreateAsyncScope(); + var client = scope.ServiceProvider.GetRequiredService(); + var logger = scope.ServiceProvider.GetRequiredService().CreateLogger("db-export"); + var verbosity = scope.ServiceProvider.GetRequiredService(); + var previousLevel = verbosity.MinimumLevel; + verbosity.MinimumLevel = verbose ? LogLevel.Debug : LogLevel.Information; + using var activity = CliActivitySource.Instance.StartActivity("cli.db.export", ActivityKind.Client); + activity?.SetTag("stellaops.cli.command", "db export"); + activity?.SetTag("stellaops.cli.format", format); + activity?.SetTag("stellaops.cli.delta", delta); using var duration = CliMetrics.MeasureCommandDuration("db export"); activity?.SetTag("stellaops.cli.publish_full", publishFull); activity?.SetTag("stellaops.cli.publish_delta", publishDelta); @@ -330,16 +330,16 @@ internal static class CommandHandlers { parameters["includeDelta"] = includeDelta.Value; } - - await TriggerJobAsync(client, logger, jobKind, parameters, cancellationToken).ConfigureAwait(false); - } - catch (Exception ex) - { - logger.LogError(ex, "Export job failed."); - Environment.ExitCode = 1; - } - finally - { + + await TriggerJobAsync(client, logger, jobKind, parameters, cancellationToken).ConfigureAwait(false); + } + catch (Exception ex) + { + logger.LogError(ex, "Export job failed."); + Environment.ExitCode = 1; + } + finally + { verbosity.MinimumLevel = previousLevel; } } @@ -723,6 +723,62 @@ internal static class CommandHandlers } } + public static Task HandleExcititorBackfillStatementsAsync( + IServiceProvider services, + DateTimeOffset? retrievedSince, + bool force, + int batchSize, + int? maxDocuments, + bool verbose, + CancellationToken cancellationToken) + { + if (batchSize <= 0) + { + throw new ArgumentOutOfRangeException(nameof(batchSize), "Batch size must be greater than zero."); + } + + if (maxDocuments.HasValue && maxDocuments.Value <= 0) + { + throw new ArgumentOutOfRangeException(nameof(maxDocuments), "Max documents must be greater than zero when specified."); + } + + var payload = new Dictionary(StringComparer.Ordinal) + { + ["force"] = force, + ["batchSize"] = batchSize, + ["maxDocuments"] = maxDocuments + }; + + if (retrievedSince.HasValue) + { + payload["retrievedSince"] = retrievedSince.Value.ToUniversalTime().ToString("O", CultureInfo.InvariantCulture); + } + + var activityTags = new Dictionary(StringComparer.Ordinal) + { + ["stellaops.cli.force"] = force, + ["stellaops.cli.batch_size"] = batchSize, + ["stellaops.cli.max_documents"] = maxDocuments + }; + + if (retrievedSince.HasValue) + { + activityTags["stellaops.cli.retrieved_since"] = retrievedSince.Value.ToUniversalTime().ToString("O", CultureInfo.InvariantCulture); + } + + return ExecuteExcititorCommandAsync( + services, + commandName: "excititor backfill-statements", + verbose, + activityTags, + client => client.ExecuteExcititorOperationAsync( + "admin/backfill-statements", + HttpMethod.Post, + RemoveNullValues(payload), + cancellationToken), + cancellationToken); + } + public static Task HandleExcititorVerifyAsync( IServiceProvider services, string? exportId, @@ -2208,7 +2264,7 @@ internal static class CommandHandlers { ["policyVerdict"] = decision.PolicyVerdict, ["signed"] = decision.Signed, - ["hasSbom"] = decision.HasSbom + ["hasSbomReferrers"] = decision.HasSbomReferrers }; if (decision.Reasons.Count > 0) @@ -2218,11 +2274,26 @@ internal static class CommandHandlers if (decision.Rekor is not null) { - map["rekor"] = new Dictionary(StringComparer.Ordinal) + var rekorMap = new Dictionary(StringComparer.Ordinal); + if (!string.IsNullOrWhiteSpace(decision.Rekor.Uuid)) { - ["uuid"] = decision.Rekor.Uuid, - ["url"] = decision.Rekor.Url - }; + rekorMap["uuid"] = decision.Rekor.Uuid; + } + + if (!string.IsNullOrWhiteSpace(decision.Rekor.Url)) + { + rekorMap["url"] = decision.Rekor.Url; + } + + if (decision.Rekor.Verified.HasValue) + { + rekorMap["verified"] = decision.Rekor.Verified; + } + + if (rekorMap.Count > 0) + { + map["rekor"] = rekorMap; + } } foreach (var kvp in decision.AdditionalProperties) @@ -2240,7 +2311,8 @@ internal static class CommandHandlers if (AnsiConsole.Profile.Capabilities.Interactive) { - var table = new Table().Border(TableBorder.Rounded).AddColumns("Image", "Verdict", "Signed", "SBOM", "Reasons", "Attestation"); + var table = new Table().Border(TableBorder.Rounded) + .AddColumns("Image", "Verdict", "Signed", "SBOM Ref", "Quieted", "Confidence", "Reasons", "Attestation"); foreach (var image in orderedImages) { @@ -2250,9 +2322,11 @@ internal static class CommandHandlers image, decision.PolicyVerdict, FormatBoolean(decision.Signed), - FormatBoolean(decision.HasSbom), + FormatBoolean(decision.HasSbomReferrers), + FormatQuietedDisplay(decision.AdditionalProperties), + FormatConfidenceDisplay(decision.AdditionalProperties), decision.Reasons.Count > 0 ? string.Join(Environment.NewLine, decision.Reasons) : "-", - string.IsNullOrWhiteSpace(decision.Rekor?.Uuid) ? "-" : decision.Rekor!.Uuid!); + FormatAttestation(decision.Rekor)); summary[decision.PolicyVerdict] = summary.TryGetValue(decision.PolicyVerdict, out var count) ? count + 1 : 1; @@ -2264,7 +2338,7 @@ internal static class CommandHandlers } else { - table.AddRow(image, "", "-", "-", "-", "-"); + table.AddRow(image, "", "-", "-", "-", "-", "-", "-"); } } @@ -2278,12 +2352,14 @@ internal static class CommandHandlers { var reasons = decision.Reasons.Count > 0 ? string.Join(", ", decision.Reasons) : "none"; logger.LogInformation( - "{Image} -> verdict={Verdict} signed={Signed} sbom={Sbom} attestation={Attestation} reasons={Reasons}", + "{Image} -> verdict={Verdict} signed={Signed} sbomRef={Sbom} quieted={Quieted} confidence={Confidence} attestation={Attestation} reasons={Reasons}", image, decision.PolicyVerdict, FormatBoolean(decision.Signed), - FormatBoolean(decision.HasSbom), - string.IsNullOrWhiteSpace(decision.Rekor?.Uuid) ? "-" : decision.Rekor!.Uuid!, + FormatBoolean(decision.HasSbomReferrers), + FormatQuietedDisplay(decision.AdditionalProperties), + FormatConfidenceDisplay(decision.AdditionalProperties), + FormatAttestation(decision.Rekor), reasons); summary[decision.PolicyVerdict] = summary.TryGetValue(decision.PolicyVerdict, out var count) ? count + 1 : 1; @@ -2346,6 +2422,144 @@ internal static class CommandHandlers private static string FormatBoolean(bool? value) => value is null ? "unknown" : value.Value ? "yes" : "no"; + private static string FormatQuietedDisplay(IReadOnlyDictionary metadata) + { + var quieted = GetMetadataBoolean(metadata, "quieted", "quiet"); + var quietedBy = GetMetadataString(metadata, "quietedBy", "quietedReason"); + + if (quieted is true) + { + return string.IsNullOrWhiteSpace(quietedBy) ? "yes" : $"yes ({quietedBy})"; + } + + if (quieted is false) + { + return "no"; + } + + return string.IsNullOrWhiteSpace(quietedBy) ? "-" : $"? ({quietedBy})"; + } + + private static string FormatConfidenceDisplay(IReadOnlyDictionary metadata) + { + var confidence = GetMetadataDouble(metadata, "confidence"); + var confidenceBand = GetMetadataString(metadata, "confidenceBand", "confidenceTier"); + + if (confidence.HasValue && !string.IsNullOrWhiteSpace(confidenceBand)) + { + return string.Format(CultureInfo.InvariantCulture, "{0:0.###} ({1})", confidence.Value, confidenceBand); + } + + if (confidence.HasValue) + { + return confidence.Value.ToString("0.###", CultureInfo.InvariantCulture); + } + + if (!string.IsNullOrWhiteSpace(confidenceBand)) + { + return confidenceBand!; + } + + return "-"; + } + + private static string FormatAttestation(RuntimePolicyRekorReference? rekor) + { + if (rekor is null) + { + return "-"; + } + + var uuid = string.IsNullOrWhiteSpace(rekor.Uuid) ? null : rekor.Uuid; + var url = string.IsNullOrWhiteSpace(rekor.Url) ? null : rekor.Url; + var verified = rekor.Verified; + + var core = uuid ?? url; + if (!string.IsNullOrEmpty(core)) + { + if (verified.HasValue) + { + var suffix = verified.Value ? " (verified)" : " (unverified)"; + return core + suffix; + } + + return core!; + } + + if (verified.HasValue) + { + return verified.Value ? "verified" : "unverified"; + } + + return "-"; + } + + private static bool? GetMetadataBoolean(IReadOnlyDictionary metadata, params string[] keys) + { + foreach (var key in keys) + { + if (metadata.TryGetValue(key, out var value) && value is not null) + { + switch (value) + { + case bool b: + return b; + case string s when bool.TryParse(s, out var parsed): + return parsed; + } + } + } + + return null; + } + + private static string? GetMetadataString(IReadOnlyDictionary metadata, params string[] keys) + { + foreach (var key in keys) + { + if (metadata.TryGetValue(key, out var value) && value is not null) + { + if (value is string s) + { + return string.IsNullOrWhiteSpace(s) ? null : s; + } + } + } + + return null; + } + + private static double? GetMetadataDouble(IReadOnlyDictionary metadata, params string[] keys) + { + foreach (var key in keys) + { + if (metadata.TryGetValue(key, out var value) && value is not null) + { + switch (value) + { + case double d: + return d; + case float f: + return f; + case decimal m: + return (double)m; + case long l: + return l; + case int i: + return i; + case string s when double.TryParse(s, NumberStyles.Float | NumberStyles.AllowThousands, CultureInfo.InvariantCulture, out var parsed): + return parsed; + } + } + } + + return null; + } + + private static readonly IReadOnlyDictionary EmptyLabelSelectors = + new ReadOnlyDictionary(new Dictionary(0, StringComparer.OrdinalIgnoreCase)); + + private static string FormatAdditionalValue(object? value) { return value switch @@ -2359,8 +2573,6 @@ internal static class CommandHandlers }; } - private static readonly IReadOnlyDictionary EmptyLabelSelectors = - new ReadOnlyDictionary(new Dictionary(0, StringComparer.OrdinalIgnoreCase)); private static IReadOnlyList NormalizeProviders(IReadOnlyList providers) { @@ -2397,29 +2609,29 @@ internal static class CommandHandlers string jobKind, IDictionary parameters, CancellationToken cancellationToken) - { - JobTriggerResult result = await client.TriggerJobAsync(jobKind, parameters, cancellationToken).ConfigureAwait(false); - if (result.Success) - { - if (!string.IsNullOrWhiteSpace(result.Location)) - { - logger.LogInformation("Job accepted. Track status at {Location}.", result.Location); - } - else if (result.Run is not null) - { - logger.LogInformation("Job accepted. RunId: {RunId} Status: {Status}", result.Run.RunId, result.Run.Status); - } - else - { - logger.LogInformation("Job accepted."); - } - - Environment.ExitCode = 0; - } - else - { - logger.LogError("Job '{JobKind}' failed: {Message}", jobKind, result.Message); - Environment.ExitCode = 1; + { + JobTriggerResult result = await client.TriggerJobAsync(jobKind, parameters, cancellationToken).ConfigureAwait(false); + if (result.Success) + { + if (!string.IsNullOrWhiteSpace(result.Location)) + { + logger.LogInformation("Job accepted. Track status at {Location}.", result.Location); + } + else if (result.Run is not null) + { + logger.LogInformation("Job accepted. RunId: {RunId} Status: {Status}", result.Run.RunId, result.Run.Status); + } + else + { + logger.LogInformation("Job accepted."); + } + + Environment.ExitCode = 0; + } + else + { + logger.LogError("Job '{JobKind}' failed: {Message}", jobKind, result.Message); + Environment.ExitCode = 1; } } } diff --git a/src/StellaOps.Cli/Services/BackendOperationsClient.cs b/src/StellaOps.Cli/Services/BackendOperationsClient.cs index 9b69a06b..7617e568 100644 --- a/src/StellaOps.Cli/Services/BackendOperationsClient.cs +++ b/src/StellaOps.Cli/Services/BackendOperationsClient.cs @@ -1,4 +1,4 @@ -using System; +using System; using System.Collections.Generic; using System.Collections.ObjectModel; using System.IO; @@ -19,9 +19,9 @@ using StellaOps.Auth.Client; using StellaOps.Cli.Configuration; using StellaOps.Cli.Services.Models; using StellaOps.Cli.Services.Models.Transport; - -namespace StellaOps.Cli.Services; - + +namespace StellaOps.Cli.Services; + internal sealed class BackendOperationsClient : IBackendOperationsClient { private static readonly JsonSerializerOptions SerializerOptions = new(JsonSerializerDefaults.Web); @@ -48,34 +48,34 @@ internal sealed class BackendOperationsClient : IBackendOperationsClient { if (Uri.TryCreate(_options.BackendUrl, UriKind.Absolute, out var baseUri)) { - httpClient.BaseAddress = baseUri; - } - } - } - - public async Task DownloadScannerAsync(string channel, string outputPath, bool overwrite, bool verbose, CancellationToken cancellationToken) - { - EnsureBackendConfigured(); - - channel = string.IsNullOrWhiteSpace(channel) ? "stable" : channel.Trim(); - outputPath = ResolveArtifactPath(outputPath, channel); - Directory.CreateDirectory(Path.GetDirectoryName(outputPath)!); - - if (!overwrite && File.Exists(outputPath)) - { - var existing = new FileInfo(outputPath); - _logger.LogInformation("Scanner artifact already cached at {Path} ({Size} bytes).", outputPath, existing.Length); - return new ScannerArtifactResult(outputPath, existing.Length, true); - } - - var attempt = 0; - var maxAttempts = Math.Max(1, _options.ScannerDownloadAttempts); - - while (true) - { - attempt++; - try - { + httpClient.BaseAddress = baseUri; + } + } + } + + public async Task DownloadScannerAsync(string channel, string outputPath, bool overwrite, bool verbose, CancellationToken cancellationToken) + { + EnsureBackendConfigured(); + + channel = string.IsNullOrWhiteSpace(channel) ? "stable" : channel.Trim(); + outputPath = ResolveArtifactPath(outputPath, channel); + Directory.CreateDirectory(Path.GetDirectoryName(outputPath)!); + + if (!overwrite && File.Exists(outputPath)) + { + var existing = new FileInfo(outputPath); + _logger.LogInformation("Scanner artifact already cached at {Path} ({Size} bytes).", outputPath, existing.Length); + return new ScannerArtifactResult(outputPath, existing.Length, true); + } + + var attempt = 0; + var maxAttempts = Math.Max(1, _options.ScannerDownloadAttempts); + + while (true) + { + attempt++; + try + { using var request = CreateRequest(HttpMethod.Get, $"api/scanner/artifacts/{channel}"); await AuthorizeRequestAsync(request, cancellationToken).ConfigureAwait(false); using var response = await _httpClient.SendAsync(request, HttpCompletionOption.ResponseHeadersRead, cancellationToken).ConfigureAwait(false); @@ -83,55 +83,55 @@ internal sealed class BackendOperationsClient : IBackendOperationsClient { var failure = await CreateFailureMessageAsync(response, cancellationToken).ConfigureAwait(false); throw new InvalidOperationException(failure); - } - - return await ProcessScannerResponseAsync(response, outputPath, channel, verbose, cancellationToken).ConfigureAwait(false); - } - catch (Exception ex) when (attempt < maxAttempts) - { - var backoffSeconds = Math.Pow(2, attempt); - _logger.LogWarning(ex, "Scanner download attempt {Attempt}/{MaxAttempts} failed. Retrying in {Delay:F0}s...", attempt, maxAttempts, backoffSeconds); - await Task.Delay(TimeSpan.FromSeconds(backoffSeconds), cancellationToken).ConfigureAwait(false); - } - } - } - - private async Task ProcessScannerResponseAsync(HttpResponseMessage response, string outputPath, string channel, bool verbose, CancellationToken cancellationToken) - { - var tempFile = outputPath + ".tmp"; - await using (var payloadStream = await response.Content.ReadAsStreamAsync(cancellationToken).ConfigureAwait(false)) - await using (var fileStream = File.Create(tempFile)) - { - await payloadStream.CopyToAsync(fileStream, cancellationToken).ConfigureAwait(false); - } - - var expectedDigest = ExtractHeaderValue(response.Headers, "X-StellaOps-Digest"); - var signatureHeader = ExtractHeaderValue(response.Headers, "X-StellaOps-Signature"); - - var digestHex = await ValidateDigestAsync(tempFile, expectedDigest, cancellationToken).ConfigureAwait(false); - await ValidateSignatureAsync(signatureHeader, digestHex, verbose, cancellationToken).ConfigureAwait(false); - - if (verbose) - { - var signatureNote = string.IsNullOrWhiteSpace(signatureHeader) ? "no signature" : "signature validated"; - _logger.LogDebug("Scanner digest sha256:{Digest} ({SignatureNote}).", digestHex, signatureNote); - } - - if (File.Exists(outputPath)) - { - File.Delete(outputPath); - } - - File.Move(tempFile, outputPath); - - PersistMetadata(outputPath, channel, digestHex, signatureHeader, response); - - var downloaded = new FileInfo(outputPath); - _logger.LogInformation("Scanner downloaded to {Path} ({Size} bytes).", outputPath, downloaded.Length); - - return new ScannerArtifactResult(outputPath, downloaded.Length, false); - } - + } + + return await ProcessScannerResponseAsync(response, outputPath, channel, verbose, cancellationToken).ConfigureAwait(false); + } + catch (Exception ex) when (attempt < maxAttempts) + { + var backoffSeconds = Math.Pow(2, attempt); + _logger.LogWarning(ex, "Scanner download attempt {Attempt}/{MaxAttempts} failed. Retrying in {Delay:F0}s...", attempt, maxAttempts, backoffSeconds); + await Task.Delay(TimeSpan.FromSeconds(backoffSeconds), cancellationToken).ConfigureAwait(false); + } + } + } + + private async Task ProcessScannerResponseAsync(HttpResponseMessage response, string outputPath, string channel, bool verbose, CancellationToken cancellationToken) + { + var tempFile = outputPath + ".tmp"; + await using (var payloadStream = await response.Content.ReadAsStreamAsync(cancellationToken).ConfigureAwait(false)) + await using (var fileStream = File.Create(tempFile)) + { + await payloadStream.CopyToAsync(fileStream, cancellationToken).ConfigureAwait(false); + } + + var expectedDigest = ExtractHeaderValue(response.Headers, "X-StellaOps-Digest"); + var signatureHeader = ExtractHeaderValue(response.Headers, "X-StellaOps-Signature"); + + var digestHex = await ValidateDigestAsync(tempFile, expectedDigest, cancellationToken).ConfigureAwait(false); + await ValidateSignatureAsync(signatureHeader, digestHex, verbose, cancellationToken).ConfigureAwait(false); + + if (verbose) + { + var signatureNote = string.IsNullOrWhiteSpace(signatureHeader) ? "no signature" : "signature validated"; + _logger.LogDebug("Scanner digest sha256:{Digest} ({SignatureNote}).", digestHex, signatureNote); + } + + if (File.Exists(outputPath)) + { + File.Delete(outputPath); + } + + File.Move(tempFile, outputPath); + + PersistMetadata(outputPath, channel, digestHex, signatureHeader, response); + + var downloaded = new FileInfo(outputPath); + _logger.LogInformation("Scanner downloaded to {Path} ({Size} bytes).", outputPath, downloaded.Length); + + return new ScannerArtifactResult(outputPath, downloaded.Length, false); + } + public async Task UploadScanResultsAsync(string filePath, CancellationToken cancellationToken) { EnsureBackendConfigured(); @@ -194,46 +194,46 @@ internal sealed class BackendOperationsClient : IBackendOperationsClient } } } - - public async Task TriggerJobAsync(string jobKind, IDictionary parameters, CancellationToken cancellationToken) - { - EnsureBackendConfigured(); - - if (string.IsNullOrWhiteSpace(jobKind)) - { - throw new ArgumentException("Job kind must be provided.", nameof(jobKind)); - } - - var requestBody = new JobTriggerRequest - { - Trigger = "cli", - Parameters = parameters is null ? new Dictionary(StringComparer.Ordinal) : new Dictionary(parameters, StringComparer.Ordinal) - }; - + + public async Task TriggerJobAsync(string jobKind, IDictionary parameters, CancellationToken cancellationToken) + { + EnsureBackendConfigured(); + + if (string.IsNullOrWhiteSpace(jobKind)) + { + throw new ArgumentException("Job kind must be provided.", nameof(jobKind)); + } + + var requestBody = new JobTriggerRequest + { + Trigger = "cli", + Parameters = parameters is null ? new Dictionary(StringComparer.Ordinal) : new Dictionary(parameters, StringComparer.Ordinal) + }; + var request = CreateRequest(HttpMethod.Post, $"jobs/{jobKind}"); await AuthorizeRequestAsync(request, cancellationToken).ConfigureAwait(false); request.Content = JsonContent.Create(requestBody, options: SerializerOptions); - - using var response = await _httpClient.SendAsync(request, cancellationToken).ConfigureAwait(false); - if (response.StatusCode == HttpStatusCode.Accepted) - { - JobRunResponse? run = null; - if (response.Content.Headers.ContentLength is > 0) - { - try - { - run = await response.Content.ReadFromJsonAsync(SerializerOptions, cancellationToken).ConfigureAwait(false); - } - catch (JsonException ex) - { - _logger.LogWarning(ex, "Failed to deserialize job run response for job kind {Kind}.", jobKind); - } - } - - var location = response.Headers.Location?.ToString(); - return new JobTriggerResult(true, "Accepted", location, run); - } - + + using var response = await _httpClient.SendAsync(request, cancellationToken).ConfigureAwait(false); + if (response.StatusCode == HttpStatusCode.Accepted) + { + JobRunResponse? run = null; + if (response.Content.Headers.ContentLength is > 0) + { + try + { + run = await response.Content.ReadFromJsonAsync(SerializerOptions, cancellationToken).ConfigureAwait(false); + } + catch (JsonException ex) + { + _logger.LogWarning(ex, "Failed to deserialize job run response for job kind {Kind}.", jobKind); + } + } + + var location = response.Headers.Location?.ToString(); + return new JobTriggerResult(true, "Accepted", location, run); + } + var failureMessage = await CreateFailureMessageAsync(response, cancellationToken).ConfigureAwait(false); return new JobTriggerResult(false, failureMessage, null, null); } @@ -443,19 +443,24 @@ internal sealed class BackendOperationsClient : IBackendOperationsClient var reasons = ExtractReasons(decision.Reasons); var metadata = ExtractExtensionMetadata(decision.ExtensionData); + var hasSbom = decision.HasSbomReferrers ?? decision.HasSbomLegacy; + RuntimePolicyRekorReference? rekor = null; if (decision.Rekor is not null && - (!string.IsNullOrWhiteSpace(decision.Rekor.Uuid) || !string.IsNullOrWhiteSpace(decision.Rekor.Url))) + (!string.IsNullOrWhiteSpace(decision.Rekor.Uuid) || + !string.IsNullOrWhiteSpace(decision.Rekor.Url) || + decision.Rekor.Verified.HasValue)) { rekor = new RuntimePolicyRekorReference( NormalizeOptionalString(decision.Rekor.Uuid), - NormalizeOptionalString(decision.Rekor.Url)); + NormalizeOptionalString(decision.Rekor.Url), + decision.Rekor.Verified); } decisions[image] = new RuntimePolicyImageDecision( verdict, decision.Signed, - decision.HasSbom, + hasSbom, reasons, rekor, metadata); @@ -624,15 +629,15 @@ internal sealed class BackendOperationsClient : IBackendOperationsClient if (!Uri.TryCreate(relativeUri, UriKind.RelativeOrAbsolute, out var requestUri)) { throw new InvalidOperationException($"Invalid request URI '{relativeUri}'."); - } - - if (requestUri.IsAbsoluteUri) - { - // Nothing to normalize. - } - else - { - requestUri = new Uri(relativeUri.TrimStart('/'), UriKind.Relative); + } + + if (requestUri.IsAbsoluteUri) + { + // Nothing to normalize. + } + else + { + requestUri = new Uri(relativeUri.TrimStart('/'), UriKind.Relative); } return new HttpRequestMessage(method, requestUri); @@ -820,76 +825,76 @@ internal sealed class BackendOperationsClient : IBackendOperationsClient { if (_httpClient.BaseAddress is null) { - throw new InvalidOperationException("Backend URL is not configured. Provide STELLAOPS_BACKEND_URL or configure appsettings."); - } - } - - private string ResolveArtifactPath(string outputPath, string channel) - { - if (!string.IsNullOrWhiteSpace(outputPath)) - { - return Path.GetFullPath(outputPath); - } - - var directory = string.IsNullOrWhiteSpace(_options.ScannerCacheDirectory) - ? Directory.GetCurrentDirectory() - : Path.GetFullPath(_options.ScannerCacheDirectory); - - Directory.CreateDirectory(directory); - var fileName = $"stellaops-scanner-{channel}.tar.gz"; - return Path.Combine(directory, fileName); - } - - private async Task CreateFailureMessageAsync(HttpResponseMessage response, CancellationToken cancellationToken) - { - var statusCode = (int)response.StatusCode; - var builder = new StringBuilder(); - builder.Append("Backend request failed with status "); - builder.Append(statusCode); - builder.Append(' '); - builder.Append(response.ReasonPhrase ?? "Unknown"); - - if (response.Content.Headers.ContentLength is > 0) - { - try - { - var problem = await response.Content.ReadFromJsonAsync(SerializerOptions, cancellationToken).ConfigureAwait(false); - if (problem is not null) - { - if (!string.IsNullOrWhiteSpace(problem.Title)) - { - builder.AppendLine().Append(problem.Title); - } - - if (!string.IsNullOrWhiteSpace(problem.Detail)) - { - builder.AppendLine().Append(problem.Detail); - } - } - } - catch (JsonException) - { - var raw = await response.Content.ReadAsStringAsync(cancellationToken).ConfigureAwait(false); - if (!string.IsNullOrWhiteSpace(raw)) - { - builder.AppendLine().Append(raw); - } - } - } - - return builder.ToString(); - } - - private static string? ExtractHeaderValue(HttpResponseHeaders headers, string name) - { - if (headers.TryGetValues(name, out var values)) - { - return values.FirstOrDefault(); - } - - return null; - } - + throw new InvalidOperationException("Backend URL is not configured. Provide STELLAOPS_BACKEND_URL or configure appsettings."); + } + } + + private string ResolveArtifactPath(string outputPath, string channel) + { + if (!string.IsNullOrWhiteSpace(outputPath)) + { + return Path.GetFullPath(outputPath); + } + + var directory = string.IsNullOrWhiteSpace(_options.ScannerCacheDirectory) + ? Directory.GetCurrentDirectory() + : Path.GetFullPath(_options.ScannerCacheDirectory); + + Directory.CreateDirectory(directory); + var fileName = $"stellaops-scanner-{channel}.tar.gz"; + return Path.Combine(directory, fileName); + } + + private async Task CreateFailureMessageAsync(HttpResponseMessage response, CancellationToken cancellationToken) + { + var statusCode = (int)response.StatusCode; + var builder = new StringBuilder(); + builder.Append("Backend request failed with status "); + builder.Append(statusCode); + builder.Append(' '); + builder.Append(response.ReasonPhrase ?? "Unknown"); + + if (response.Content.Headers.ContentLength is > 0) + { + try + { + var problem = await response.Content.ReadFromJsonAsync(SerializerOptions, cancellationToken).ConfigureAwait(false); + if (problem is not null) + { + if (!string.IsNullOrWhiteSpace(problem.Title)) + { + builder.AppendLine().Append(problem.Title); + } + + if (!string.IsNullOrWhiteSpace(problem.Detail)) + { + builder.AppendLine().Append(problem.Detail); + } + } + } + catch (JsonException) + { + var raw = await response.Content.ReadAsStringAsync(cancellationToken).ConfigureAwait(false); + if (!string.IsNullOrWhiteSpace(raw)) + { + builder.AppendLine().Append(raw); + } + } + } + + return builder.ToString(); + } + + private static string? ExtractHeaderValue(HttpResponseHeaders headers, string name) + { + if (headers.TryGetValues(name, out var values)) + { + return values.FirstOrDefault(); + } + + return null; + } + private static string? NormalizeExpectedDigest(string? digest) { if (string.IsNullOrWhiteSpace(digest)) @@ -909,23 +914,23 @@ internal sealed class BackendOperationsClient : IBackendOperationsClient await using (var stream = File.OpenRead(filePath)) { var hash = await SHA256.HashDataAsync(stream, cancellationToken).ConfigureAwait(false); - digestHex = Convert.ToHexString(hash).ToLowerInvariant(); - } - - if (!string.IsNullOrWhiteSpace(expectedDigest)) - { - var normalized = NormalizeDigest(expectedDigest); - if (!normalized.Equals(digestHex, StringComparison.OrdinalIgnoreCase)) - { - File.Delete(filePath); - throw new InvalidOperationException($"Scanner digest mismatch. Expected sha256:{normalized}, calculated sha256:{digestHex}."); - } - } - else - { - _logger.LogWarning("Scanner download missing X-StellaOps-Digest header; relying on computed digest only."); - } - + digestHex = Convert.ToHexString(hash).ToLowerInvariant(); + } + + if (!string.IsNullOrWhiteSpace(expectedDigest)) + { + var normalized = NormalizeDigest(expectedDigest); + if (!normalized.Equals(digestHex, StringComparison.OrdinalIgnoreCase)) + { + File.Delete(filePath); + throw new InvalidOperationException($"Scanner digest mismatch. Expected sha256:{normalized}, calculated sha256:{digestHex}."); + } + } + else + { + _logger.LogWarning("Scanner download missing X-StellaOps-Digest header; relying on computed digest only."); + } + return digestHex; } @@ -945,71 +950,71 @@ internal sealed class BackendOperationsClient : IBackendOperationsClient var hash = await SHA256.HashDataAsync(stream, cancellationToken).ConfigureAwait(false); return Convert.ToHexString(hash).ToLowerInvariant(); } - - private async Task ValidateSignatureAsync(string? signatureHeader, string digestHex, bool verbose, CancellationToken cancellationToken) - { - if (string.IsNullOrWhiteSpace(_options.ScannerSignaturePublicKeyPath)) - { - if (!string.IsNullOrWhiteSpace(signatureHeader)) - { - _logger.LogDebug("Signature header present but no public key configured; skipping validation."); - } - return; - } - - if (string.IsNullOrWhiteSpace(signatureHeader)) - { - throw new InvalidOperationException("Scanner signature missing while a public key is configured."); - } - - var publicKeyPath = Path.GetFullPath(_options.ScannerSignaturePublicKeyPath); - if (!File.Exists(publicKeyPath)) - { - throw new FileNotFoundException("Scanner signature public key not found.", publicKeyPath); - } - - var signatureBytes = Convert.FromBase64String(signatureHeader); - var digestBytes = Convert.FromHexString(digestHex); - - var pem = await File.ReadAllTextAsync(publicKeyPath, cancellationToken).ConfigureAwait(false); - using var rsa = RSA.Create(); - rsa.ImportFromPem(pem); - - var valid = rsa.VerifyHash(digestBytes, signatureBytes, HashAlgorithmName.SHA256, RSASignaturePadding.Pkcs1); - if (!valid) - { - throw new InvalidOperationException("Scanner signature validation failed."); - } - - if (verbose) - { - _logger.LogDebug("Scanner signature validated using key {KeyPath}.", publicKeyPath); - } - } - + + private async Task ValidateSignatureAsync(string? signatureHeader, string digestHex, bool verbose, CancellationToken cancellationToken) + { + if (string.IsNullOrWhiteSpace(_options.ScannerSignaturePublicKeyPath)) + { + if (!string.IsNullOrWhiteSpace(signatureHeader)) + { + _logger.LogDebug("Signature header present but no public key configured; skipping validation."); + } + return; + } + + if (string.IsNullOrWhiteSpace(signatureHeader)) + { + throw new InvalidOperationException("Scanner signature missing while a public key is configured."); + } + + var publicKeyPath = Path.GetFullPath(_options.ScannerSignaturePublicKeyPath); + if (!File.Exists(publicKeyPath)) + { + throw new FileNotFoundException("Scanner signature public key not found.", publicKeyPath); + } + + var signatureBytes = Convert.FromBase64String(signatureHeader); + var digestBytes = Convert.FromHexString(digestHex); + + var pem = await File.ReadAllTextAsync(publicKeyPath, cancellationToken).ConfigureAwait(false); + using var rsa = RSA.Create(); + rsa.ImportFromPem(pem); + + var valid = rsa.VerifyHash(digestBytes, signatureBytes, HashAlgorithmName.SHA256, RSASignaturePadding.Pkcs1); + if (!valid) + { + throw new InvalidOperationException("Scanner signature validation failed."); + } + + if (verbose) + { + _logger.LogDebug("Scanner signature validated using key {KeyPath}.", publicKeyPath); + } + } + private void PersistMetadata(string outputPath, string channel, string digestHex, string? signatureHeader, HttpResponseMessage response) { var metadata = new { channel, - digest = $"sha256:{digestHex}", - signature = signatureHeader, - downloadedAt = DateTimeOffset.UtcNow, - source = response.RequestMessage?.RequestUri?.ToString(), - sizeBytes = new FileInfo(outputPath).Length, - headers = new - { - etag = response.Headers.ETag?.Tag, - lastModified = response.Content.Headers.LastModified, - contentType = response.Content.Headers.ContentType?.ToString() - } - }; - - var metadataPath = outputPath + ".metadata.json"; - var json = JsonSerializer.Serialize(metadata, new JsonSerializerOptions - { - WriteIndented = true - }); + digest = $"sha256:{digestHex}", + signature = signatureHeader, + downloadedAt = DateTimeOffset.UtcNow, + source = response.RequestMessage?.RequestUri?.ToString(), + sizeBytes = new FileInfo(outputPath).Length, + headers = new + { + etag = response.Headers.ETag?.Tag, + lastModified = response.Content.Headers.LastModified, + contentType = response.Content.Headers.ContentType?.ToString() + } + }; + + var metadataPath = outputPath + ".metadata.json"; + var json = JsonSerializer.Serialize(metadata, new JsonSerializerOptions + { + WriteIndented = true + }); File.WriteAllText(metadataPath, json); } diff --git a/src/StellaOps.Cli/Services/Models/RuntimePolicyEvaluationModels.cs b/src/StellaOps.Cli/Services/Models/RuntimePolicyEvaluationModels.cs index 92d75883..c4e08f35 100644 --- a/src/StellaOps.Cli/Services/Models/RuntimePolicyEvaluationModels.cs +++ b/src/StellaOps.Cli/Services/Models/RuntimePolicyEvaluationModels.cs @@ -17,9 +17,9 @@ internal sealed record RuntimePolicyEvaluationResult( internal sealed record RuntimePolicyImageDecision( string PolicyVerdict, bool? Signed, - bool? HasSbom, + bool? HasSbomReferrers, IReadOnlyList Reasons, RuntimePolicyRekorReference? Rekor, IReadOnlyDictionary AdditionalProperties); -internal sealed record RuntimePolicyRekorReference(string? Uuid, string? Url); +internal sealed record RuntimePolicyRekorReference(string? Uuid, string? Url, bool? Verified); diff --git a/src/StellaOps.Cli/Services/Models/Transport/RuntimePolicyEvaluationTransport.cs b/src/StellaOps.Cli/Services/Models/Transport/RuntimePolicyEvaluationTransport.cs index e84cd028..d62b7639 100644 --- a/src/StellaOps.Cli/Services/Models/Transport/RuntimePolicyEvaluationTransport.cs +++ b/src/StellaOps.Cli/Services/Models/Transport/RuntimePolicyEvaluationTransport.cs @@ -42,8 +42,12 @@ internal sealed class RuntimePolicyEvaluationImageDocument [JsonPropertyName("signed")] public bool? Signed { get; set; } + [JsonPropertyName("hasSbomReferrers")] + public bool? HasSbomReferrers { get; set; } + + // Legacy field kept for pre-contract-sync services. [JsonPropertyName("hasSbom")] - public bool? HasSbom { get; set; } + public bool? HasSbomLegacy { get; set; } [JsonPropertyName("reasons")] public List? Reasons { get; set; } @@ -62,4 +66,7 @@ internal sealed class RuntimePolicyRekorDocument [JsonPropertyName("url")] public string? Url { get; set; } + + [JsonPropertyName("verified")] + public bool? Verified { get; set; } } diff --git a/src/StellaOps.Cli/TASKS.md b/src/StellaOps.Cli/TASKS.md index 8131cdeb..fbf604b8 100644 --- a/src/StellaOps.Cli/TASKS.md +++ b/src/StellaOps.Cli/TASKS.md @@ -20,5 +20,5 @@ If you are working on this file you need to read docs/ARCHITECTURE_EXCITITOR.md |CLI-RUNTIME-13-005 – Runtime policy test verbs|DevEx/CLI|SCANNER-RUNTIME-12-302, ZASTAVA-WEBHOOK-12-102|**DONE (2025-10-19)** – Added `runtime policy test` command (stdin/file support, JSON output), backend client method + typed models, verdict table output, docs/tests updated (`dotnet test src/StellaOps.Cli.Tests`).| |CLI-OFFLINE-13-006 – Offline kit workflows|DevEx/CLI|DEVOPS-OFFLINE-14-002|TODO – Implement `offline kit pull/import/status` commands with integrity checks, resumable downloads, and doc updates.| |CLI-PLUGIN-13-007 – Plugin packaging|DevEx/CLI|CLI-RUNTIME-13-005, CLI-OFFLINE-13-006|TODO – Package non-core verbs as restart-time plug-ins (manifest + loader updates, tests ensuring no hot reload).| -|CLI-RUNTIME-13-008 – Runtime policy contract sync|DevEx/CLI, Scanner WebService Guild|SCANNER-RUNTIME-12-302|TODO – Once `/api/v1/scanner/policy/runtime` exits TODO, verify CLI output against final schema (field names, metadata) and update formatter/tests if the contract moves. Capture joint review notes in docs/09 and link Scanner task sign-off.| -|CLI-RUNTIME-13-009 – Runtime policy smoke fixture|DevEx/CLI, QA Guild|CLI-RUNTIME-13-005|TODO – Build Spectre test harness exercising `runtime policy test` against a stubbed backend to lock output shape (table + `--json`) and guard regressions. Integrate into `dotnet test` suite.| +|CLI-RUNTIME-13-008 – Runtime policy contract sync|DevEx/CLI, Scanner WebService Guild|SCANNER-RUNTIME-12-302|**DONE (2025-10-19)** – CLI runtime table/JSON now align with SCANNER-RUNTIME-12-302 (SBOM referrers, quieted provenance, confidence, verified Rekor); docs/09 updated with joint sign-off note.| +|CLI-RUNTIME-13-009 – Runtime policy smoke fixture|DevEx/CLI, QA Guild|CLI-RUNTIME-13-005|**DONE (2025-10-19)** – Spectre console harness + regression tests cover table and `--json` output paths for `runtime policy test`, using stubbed backend and integrated into `dotnet test` suite.| diff --git a/src/StellaOps.Concelier.Connector.Cccs/TASKS.md b/src/StellaOps.Concelier.Connector.Cccs/TASKS.md index 0732cc75..313f9f5c 100644 --- a/src/StellaOps.Concelier.Connector.Cccs/TASKS.md +++ b/src/StellaOps.Concelier.Connector.Cccs/TASKS.md @@ -9,3 +9,4 @@ |FEEDCONN-CCCS-02-006 Observability & documentation|DevEx|Docs|**DONE (2025-10-15)** – Added `CccsDiagnostics` meter (fetch/parse/map counters), enriched connector logs with document counts, and published `docs/ops/concelier-cccs-operations.md` covering config, telemetry, and sanitiser guidance.| |FEEDCONN-CCCS-02-007 Historical advisory harvesting plan|BE-Conn-CCCS|Research|**DONE (2025-10-15)** – Measured `/api/cccs/threats/v1/get` inventory (~5.1k rows/lang; earliest 2018-06-08), documented backfill workflow + language split strategy, and linked the runbook for Offline Kit execution.| |FEEDCONN-CCCS-02-008 Raw DOM parsing refinement|BE-Conn-CCCS|Source.Common|**DONE (2025-10-15)** – Parser now walks unsanitised DOM (heading + nested list coverage), sanitizer keeps ``/`section` nodes, and regression fixtures/tests assert EN/FR list handling + preserved HTML structure.| +|FEEDCONN-CCCS-02-009 Normalized versions rollout (Oct 2025)|BE-Conn-CCCS|Merge coordination (`FEEDMERGE-COORD-02-900`)|**TODO (due 2025-10-21)** – Implement trailing-version split helper per Merge guidance (see `../Merge/RANGE_PRIMITIVES_COORDINATION.md` “Helper snippets”) to emit `NormalizedVersions` via `SemVerRangeRuleBuilder`; refresh mapper tests/fixtures to assert provenance notes (`cccs:{serial}:{index}`) and confirm merge counters drop.| diff --git a/src/StellaOps.Concelier.Connector.CertBund/TASKS.md b/src/StellaOps.Concelier.Connector.CertBund/TASKS.md index 7b0639d0..fa8279b6 100644 --- a/src/StellaOps.Concelier.Connector.CertBund/TASKS.md +++ b/src/StellaOps.Concelier.Connector.CertBund/TASKS.md @@ -10,3 +10,4 @@ |FEEDCONN-CERTBUND-02-007 Feed history & locale assessment|BE-Conn-CERTBUND|Research|**DONE (2025-10-15)** – Measured RSS retention (~6 days/≈250 items), captured connector-driven backfill guidance in the runbook, and aligned locale guidance (preserve `language=de`, Docs glossary follow-up). **Next:** coordinate with Tools to land the state-seeding helper so scripted backfills replace manual Mongo tweaks.| |FEEDCONN-CERTBUND-02-008 Session bootstrap & cookie strategy|BE-Conn-CERTBUND|Source.Common|**DONE (2025-10-14)** – Feed client primes the portal session (cookie container via `SocketsHttpHandler`), shares cookies across detail requests, and documents bootstrap behaviour in options (`PortalBootstrapUri`).| |FEEDCONN-CERTBUND-02-009 Offline Kit export packaging|BE-Conn-CERTBUND, Docs|Offline Kit|**DONE (2025-10-17)** – Added `tools/certbund_offline_snapshot.py` to capture search/export JSON, emit deterministic manifests + SHA files, and refreshed docs (`docs/ops/concelier-certbund-operations.md`, `docs/24_OFFLINE_KIT.md`) with offline-kit instructions and manifest layout guidance. Seed data README/ignore rules cover local snapshot hygiene.| +|FEEDCONN-CERTBUND-02-010 Normalized range translator|BE-Conn-CERTBUND|Merge coordination (`FEEDMERGE-COORD-02-900`)|**TODO (due 2025-10-22)** – Translate `product.Versions` phrases (e.g., `2023.1 bis 2024.2`, `alle`) into comparator strings for `SemVerRangeRuleBuilder`, emit `NormalizedVersions` with `certbund:{advisoryId}:{vendor}` provenance, and extend tests/README with localisation notes.| diff --git a/src/StellaOps.Concelier.Connector.Ics.Cisa/TASKS.md b/src/StellaOps.Concelier.Connector.Ics.Cisa/TASKS.md index c26b4ab3..af465f7e 100644 --- a/src/StellaOps.Concelier.Connector.Ics.Cisa/TASKS.md +++ b/src/StellaOps.Concelier.Connector.Ics.Cisa/TASKS.md @@ -12,3 +12,4 @@ |FEEDCONN-ICSCISA-02-009 GovDelivery credential onboarding|Ops, BE-Conn-ICS-CISA|Ops|**DONE (2025-10-14)** – GovDelivery onboarding runbook captured in `docs/ops/concelier-icscisa-operations.md`; secret vault path and Offline Kit handling documented.| |FEEDCONN-ICSCISA-02-010 Mitigation & SemVer polish|BE-Conn-ICS-CISA|02-003, 02-004|**DONE (2025-10-16)** – Attachment + mitigation references now land as expected and SemVer primitives carry exact values; end-to-end suite green (see `HANDOVER.md`).| |FEEDCONN-ICSCISA-02-011 Docs & telemetry refresh|DevEx|02-006|**DONE (2025-10-16)** – Ops documentation refreshed (attachments, SemVer validation, proxy knobs) and telemetry notes verified.| +|FEEDCONN-ICSCISA-02-012 Normalized version decision|BE-Conn-ICS-CISA|Merge coordination (`FEEDMERGE-COORD-02-900`)|**TODO (due 2025-10-23)** – Promote existing `SemVerPrimitive` exact values into `NormalizedVersions` via `.ToNormalizedVersionRule("ics-cisa:{advisoryId}:{product}")`, add regression coverage, and open Models ticket if non-SemVer firmware requires a new scheme.| diff --git a/src/StellaOps.Concelier.Connector.Kisa/TASKS.md b/src/StellaOps.Concelier.Connector.Kisa/TASKS.md index 8b3c3e35..97083734 100644 --- a/src/StellaOps.Concelier.Connector.Kisa/TASKS.md +++ b/src/StellaOps.Concelier.Connector.Kisa/TASKS.md @@ -8,3 +8,4 @@ |FEEDCONN-KISA-02-005 Deterministic fixtures & tests|QA|Testing|**DONE (2025-10-14)** – Added `StellaOps.Concelier.Connector.Kisa.Tests` with Korean fixtures and fetch→parse→map regression; fixtures regenerate via `UPDATE_KISA_FIXTURES=1`.| |FEEDCONN-KISA-02-006 Telemetry & documentation|DevEx|Docs|**DONE (2025-10-14)** – Added diagnostics-backed telemetry, structured logs, regression coverage, and published localisation notes in `docs/dev/kisa_connector_notes.md` + fixture guidance for Docs/QA.| |FEEDCONN-KISA-02-007 RSS contract & localisation brief|BE-Conn-KISA|Research|**DONE (2025-10-11)** – Documented RSS URLs, confirmed UTF-8 payload (no additional cookies required), and drafted localisation plan (Hangul glossary + optional MT plugin). Remaining open item: capture SPA detail API contract for full-text translations.| +|FEEDCONN-KISA-02-008 Firmware scheme proposal|BE-Conn-KISA, Models|Merge coordination (`FEEDMERGE-COORD-02-900`)|**TODO (due 2025-10-24)** – Define transformation for Hangul-labelled firmware ranges (`XFU 1.0.1.0084 ~ 2.0.1.0034`), propose `kisa.build` (or equivalent) scheme to Models, implement normalized rule emission/tests once scheme approved, and update localisation notes.| diff --git a/src/StellaOps.Concelier.Connector.StellaOpsMirror.Tests/MirrorSignatureVerifierTests.cs b/src/StellaOps.Concelier.Connector.StellaOpsMirror.Tests/MirrorSignatureVerifierTests.cs new file mode 100644 index 00000000..e39595dc --- /dev/null +++ b/src/StellaOps.Concelier.Connector.StellaOpsMirror.Tests/MirrorSignatureVerifierTests.cs @@ -0,0 +1,95 @@ +using System; +using System.Collections.Generic; +using System.Security.Cryptography; +using Microsoft.Extensions.Logging.Abstractions; +using StellaOps.Concelier.Connector.StellaOpsMirror.Security; +using StellaOps.Cryptography; +using Xunit; + +namespace StellaOps.Concelier.Connector.StellaOpsMirror.Tests; + +public sealed class MirrorSignatureVerifierTests +{ + [Fact] + public async Task VerifyAsync_ValidSignaturePasses() + { + var provider = new DefaultCryptoProvider(); + var key = CreateSigningKey("mirror-key"); + provider.UpsertSigningKey(key); + + var registry = new CryptoProviderRegistry(new[] { provider }); + var verifier = new MirrorSignatureVerifier(registry, NullLogger.Instance); + + var payload = "{\"advisories\":[]}\"u8".ToUtf8Bytes(); + var (signature, _) = await CreateDetachedJwsAsync(provider, key.Reference.KeyId, payload); + + await verifier.VerifyAsync(payload, signature, CancellationToken.None); + } + + [Fact] + public async Task VerifyAsync_InvalidSignatureThrows() + { + var provider = new DefaultCryptoProvider(); + var key = CreateSigningKey("mirror-key"); + provider.UpsertSigningKey(key); + + var registry = new CryptoProviderRegistry(new[] { provider }); + var verifier = new MirrorSignatureVerifier(registry, NullLogger.Instance); + + var payload = "{\"advisories\":[]}\"u8".ToUtf8Bytes(); + var (signature, _) = await CreateDetachedJwsAsync(provider, key.Reference.KeyId, payload); + + var tampered = signature.Replace("a", "b", StringComparison.Ordinal); + + await Assert.ThrowsAsync(() => verifier.VerifyAsync(payload, tampered, CancellationToken.None)); + } + + private static CryptoSigningKey CreateSigningKey(string keyId) + { + using var ecdsa = ECDsa.Create(ECCurve.NamedCurves.nistP256); + var parameters = ecdsa.ExportParameters(includePrivateParameters: true); + return new CryptoSigningKey(new CryptoKeyReference(keyId), SignatureAlgorithms.Es256, in parameters, DateTimeOffset.UtcNow); + } + + private static async Task<(string Signature, DateTimeOffset SignedAt)> CreateDetachedJwsAsync( + DefaultCryptoProvider provider, + string keyId, + ReadOnlyMemory payload) + { + var signer = provider.GetSigner(SignatureAlgorithms.Es256, new CryptoKeyReference(keyId)); + var header = new Dictionary + { + ["alg"] = SignatureAlgorithms.Es256, + ["kid"] = keyId, + ["provider"] = provider.Name, + ["typ"] = "application/vnd.stellaops.concelier.mirror-bundle+jws", + ["b64"] = false, + ["crit"] = new[] { "b64" } + }; + + var headerJson = System.Text.Json.JsonSerializer.Serialize(header); + var protectedHeader = Microsoft.IdentityModel.Tokens.Base64UrlEncoder.Encode(headerJson); + + var signingInput = BuildSigningInput(protectedHeader, payload.Span); + var signatureBytes = await signer.SignAsync(signingInput, CancellationToken.None).ConfigureAwait(false); + var encodedSignature = Microsoft.IdentityModel.Tokens.Base64UrlEncoder.Encode(signatureBytes); + + return (string.Concat(protectedHeader, "..", encodedSignature), DateTimeOffset.UtcNow); + } + + private static ReadOnlyMemory BuildSigningInput(string encodedHeader, ReadOnlySpan payload) + { + var headerBytes = System.Text.Encoding.ASCII.GetBytes(encodedHeader); + var buffer = new byte[headerBytes.Length + 1 + payload.Length]; + headerBytes.CopyTo(buffer.AsSpan()); + buffer[headerBytes.Length] = (byte)'.'; + payload.CopyTo(buffer.AsSpan(headerBytes.Length + 1)); + return buffer; + } +} + +file static class Utf8Extensions +{ + public static ReadOnlyMemory ToUtf8Bytes(this string value) + => System.Text.Encoding.UTF8.GetBytes(value); +} diff --git a/src/StellaOps.Concelier.Connector.StellaOpsMirror.Tests/StellaOps.Concelier.Connector.StellaOpsMirror.Tests.csproj b/src/StellaOps.Concelier.Connector.StellaOpsMirror.Tests/StellaOps.Concelier.Connector.StellaOpsMirror.Tests.csproj new file mode 100644 index 00000000..77093e54 --- /dev/null +++ b/src/StellaOps.Concelier.Connector.StellaOpsMirror.Tests/StellaOps.Concelier.Connector.StellaOpsMirror.Tests.csproj @@ -0,0 +1,11 @@ + + + net10.0 + enable + enable + + + + + + diff --git a/src/StellaOps.Concelier.Connector.StellaOpsMirror.Tests/StellaOpsMirrorConnectorTests.cs b/src/StellaOps.Concelier.Connector.StellaOpsMirror.Tests/StellaOpsMirrorConnectorTests.cs new file mode 100644 index 00000000..37342f88 --- /dev/null +++ b/src/StellaOps.Concelier.Connector.StellaOpsMirror.Tests/StellaOpsMirrorConnectorTests.cs @@ -0,0 +1,319 @@ +using System; +using System.Collections.Generic; +using System.Net; +using System.Net.Http; +using System.Security.Cryptography; +using System.Text; +using System.Text.Json; +using Microsoft.Extensions.Configuration; +using Microsoft.Extensions.DependencyInjection; +using Microsoft.Extensions.Http; +using Microsoft.Extensions.Logging; +using Microsoft.Extensions.Logging.Abstractions; +using Microsoft.Extensions.Options; +using MongoDB.Bson; +using StellaOps.Concelier.Connector.Common; +using StellaOps.Concelier.Connector.Common.Testing; +using StellaOps.Concelier.Connector.StellaOpsMirror.Settings; +using StellaOps.Concelier.Storage.Mongo; +using StellaOps.Concelier.Storage.Mongo.Documents; +using StellaOps.Concelier.Storage.Mongo.SourceState; +using StellaOps.Concelier.Testing; +using StellaOps.Cryptography; +using Xunit; + +namespace StellaOps.Concelier.Connector.StellaOpsMirror.Tests; + +[Collection("mongo-fixture")] +public sealed class StellaOpsMirrorConnectorTests : IAsyncLifetime +{ + private readonly MongoIntegrationFixture _fixture; + private readonly CannedHttpMessageHandler _handler; + + public StellaOpsMirrorConnectorTests(MongoIntegrationFixture fixture) + { + _fixture = fixture; + _handler = new CannedHttpMessageHandler(); + } + + [Fact] + public async Task FetchAsync_PersistsMirrorArtifacts() + { + var manifestContent = "{\"domain\":\"primary\",\"files\":[]}"; + var bundleContent = "{\"advisories\":[{\"id\":\"CVE-2025-0001\"}]}"; + + var manifestDigest = ComputeDigest(manifestContent); + var bundleDigest = ComputeDigest(bundleContent); + + var index = BuildIndex(manifestDigest, Encoding.UTF8.GetByteCount(manifestContent), bundleDigest, Encoding.UTF8.GetByteCount(bundleContent), includeSignature: false); + + await using var provider = await BuildServiceProviderAsync(); + + SeedResponses(index, manifestContent, bundleContent, signature: null); + + var connector = provider.GetRequiredService(); + await connector.FetchAsync(provider, CancellationToken.None); + + var documentStore = provider.GetRequiredService(); + var manifestUri = "https://mirror.test/mirror/primary/manifest.json"; + var bundleUri = "https://mirror.test/mirror/primary/bundle.json"; + + var manifestDocument = await documentStore.FindBySourceAndUriAsync(StellaOpsMirrorConnector.Source, manifestUri, CancellationToken.None); + Assert.NotNull(manifestDocument); + Assert.Equal(DocumentStatuses.Mapped, manifestDocument!.Status); + Assert.Equal(NormalizeDigest(manifestDigest), manifestDocument.Sha256); + + var bundleDocument = await documentStore.FindBySourceAndUriAsync(StellaOpsMirrorConnector.Source, bundleUri, CancellationToken.None); + Assert.NotNull(bundleDocument); + Assert.Equal(DocumentStatuses.PendingParse, bundleDocument!.Status); + Assert.Equal(NormalizeDigest(bundleDigest), bundleDocument.Sha256); + + var rawStorage = provider.GetRequiredService(); + Assert.NotNull(manifestDocument.GridFsId); + Assert.NotNull(bundleDocument.GridFsId); + + var manifestBytes = await rawStorage.DownloadAsync(manifestDocument.GridFsId!.Value, CancellationToken.None); + var bundleBytes = await rawStorage.DownloadAsync(bundleDocument.GridFsId!.Value, CancellationToken.None); + Assert.Equal(manifestContent, Encoding.UTF8.GetString(manifestBytes)); + Assert.Equal(bundleContent, Encoding.UTF8.GetString(bundleBytes)); + + var stateRepository = provider.GetRequiredService(); + var state = await stateRepository.TryGetAsync(StellaOpsMirrorConnector.Source, CancellationToken.None); + Assert.NotNull(state); + + var cursorDocument = state!.Cursor ?? new BsonDocument(); + var digestValue = cursorDocument.TryGetValue("bundleDigest", out var digestBson) ? digestBson.AsString : string.Empty; + Assert.Equal(NormalizeDigest(bundleDigest), NormalizeDigest(digestValue)); + + var pendingDocumentsArray = cursorDocument.TryGetValue("pendingDocuments", out var pendingDocsBson) && pendingDocsBson is BsonArray pendingArray + ? pendingArray + : new BsonArray(); + Assert.Single(pendingDocumentsArray); + var pendingDocumentId = Guid.Parse(pendingDocumentsArray[0].AsString); + Assert.Equal(bundleDocument.Id, pendingDocumentId); + + var pendingMappingsArray = cursorDocument.TryGetValue("pendingMappings", out var pendingMappingsBson) && pendingMappingsBson is BsonArray mappingsArray + ? mappingsArray + : new BsonArray(); + Assert.Empty(pendingMappingsArray); + } + + [Fact] + public async Task FetchAsync_TamperedSignatureThrows() + { + var manifestContent = "{\"domain\":\"primary\"}"; + var bundleContent = "{\"advisories\":[{\"id\":\"CVE-2025-0002\"}]}"; + + var manifestDigest = ComputeDigest(manifestContent); + var bundleDigest = ComputeDigest(bundleContent); + var index = BuildIndex(manifestDigest, Encoding.UTF8.GetByteCount(manifestContent), bundleDigest, Encoding.UTF8.GetByteCount(bundleContent), includeSignature: true); + + await using var provider = await BuildServiceProviderAsync(options => + { + options.Signature.Enabled = true; + options.Signature.KeyId = "mirror-key"; + options.Signature.Provider = "default"; + }); + + var defaultProvider = provider.GetRequiredService(); + var signingKey = CreateSigningKey("mirror-key"); + defaultProvider.UpsertSigningKey(signingKey); + + var (signatureValue, _) = CreateDetachedJws(signingKey, bundleContent); + // Tamper with signature so verification fails. + var tamperedSignature = signatureValue.Replace('a', 'b'); + + SeedResponses(index, manifestContent, bundleContent, tamperedSignature); + + var connector = provider.GetRequiredService(); + await Assert.ThrowsAsync(() => connector.FetchAsync(provider, CancellationToken.None)); + + var stateRepository = provider.GetRequiredService(); + var state = await stateRepository.TryGetAsync(StellaOpsMirrorConnector.Source, CancellationToken.None); + Assert.NotNull(state); + Assert.True(state!.FailCount >= 1); + Assert.False(state.Cursor.TryGetValue("bundleDigest", out _)); + } + + public Task InitializeAsync() => Task.CompletedTask; + + public Task DisposeAsync() + { + _handler.Clear(); + return Task.CompletedTask; + } + + private async Task BuildServiceProviderAsync(Action? configureOptions = null) + { + await _fixture.Client.DropDatabaseAsync(_fixture.Database.DatabaseNamespace.DatabaseName); + _handler.Clear(); + + var services = new ServiceCollection(); + services.AddLogging(builder => builder.AddProvider(NullLoggerProvider.Instance)); + services.AddSingleton(_handler); + services.AddSingleton(TimeProvider.System); + + services.AddMongoStorage(options => + { + options.ConnectionString = _fixture.Runner.ConnectionString; + options.DatabaseName = _fixture.Database.DatabaseNamespace.DatabaseName; + options.CommandTimeout = TimeSpan.FromSeconds(5); + }); + + services.AddSingleton(); + services.AddSingleton(sp => sp.GetRequiredService()); + services.AddSingleton(sp => new CryptoProviderRegistry(sp.GetServices())); + + var configuration = new ConfigurationBuilder() + .AddInMemoryCollection(new Dictionary + { + ["concelier:sources:stellaopsMirror:baseAddress"] = "https://mirror.test/", + ["concelier:sources:stellaopsMirror:domainId"] = "primary", + ["concelier:sources:stellaopsMirror:indexPath"] = "/concelier/exports/index.json", + }) + .Build(); + + var routine = new StellaOpsMirrorDependencyInjectionRoutine(); + routine.Register(services, configuration); + + if (configureOptions is not null) + { + services.PostConfigure(configureOptions); + } + + services.Configure("stellaops-mirror", builder => + { + builder.HttpMessageHandlerBuilderActions.Add(options => + { + options.PrimaryHandler = _handler; + }); + }); + + var provider = services.BuildServiceProvider(); + var bootstrapper = provider.GetRequiredService(); + await bootstrapper.InitializeAsync(CancellationToken.None); + return provider; + } + + private void SeedResponses(string indexJson, string manifestContent, string bundleContent, string? signature) + { + var baseUri = new Uri("https://mirror.test"); + _handler.AddResponse(HttpMethod.Get, new Uri(baseUri, "/concelier/exports/index.json"), () => CreateJsonResponse(indexJson)); + _handler.AddResponse(HttpMethod.Get, new Uri(baseUri, "mirror/primary/manifest.json"), () => CreateJsonResponse(manifestContent)); + _handler.AddResponse(HttpMethod.Get, new Uri(baseUri, "mirror/primary/bundle.json"), () => CreateJsonResponse(bundleContent)); + + if (signature is not null) + { + _handler.AddResponse(HttpMethod.Get, new Uri(baseUri, "mirror/primary/bundle.json.jws"), () => new HttpResponseMessage(HttpStatusCode.OK) + { + Content = new StringContent(signature, Encoding.UTF8, "application/jose+json"), + }); + } + } + + private static HttpResponseMessage CreateJsonResponse(string content) + => new(HttpStatusCode.OK) + { + Content = new StringContent(content, Encoding.UTF8, "application/json"), + }; + + private static string BuildIndex(string manifestDigest, int manifestBytes, string bundleDigest, int bundleBytes, bool includeSignature) + { + var index = new + { + schemaVersion = 1, + generatedAt = new DateTimeOffset(2025, 10, 19, 12, 0, 0, TimeSpan.Zero), + targetRepository = "repo", + domains = new[] + { + new + { + domainId = "primary", + displayName = "Primary", + advisoryCount = 1, + manifest = new + { + path = "mirror/primary/manifest.json", + sizeBytes = manifestBytes, + digest = manifestDigest, + signature = (object?)null, + }, + bundle = new + { + path = "mirror/primary/bundle.json", + sizeBytes = bundleBytes, + digest = bundleDigest, + signature = includeSignature + ? new + { + path = "mirror/primary/bundle.json.jws", + algorithm = "ES256", + keyId = "mirror-key", + provider = "default", + signedAt = new DateTimeOffset(2025, 10, 19, 12, 0, 0, TimeSpan.Zero), + } + : null, + }, + sources = Array.Empty(), + } + } + }; + + return JsonSerializer.Serialize(index, new JsonSerializerOptions + { + PropertyNamingPolicy = JsonNamingPolicy.CamelCase, + WriteIndented = false, + }); + } + + private static string ComputeDigest(string content) + { + var bytes = Encoding.UTF8.GetBytes(content); + var hash = SHA256.HashData(bytes); + return "sha256:" + Convert.ToHexString(hash).ToLowerInvariant(); + } + + private static string NormalizeDigest(string digest) + => digest.StartsWith("sha256:", StringComparison.OrdinalIgnoreCase) ? digest[7..] : digest; + + private static CryptoSigningKey CreateSigningKey(string keyId) + { + using var ecdsa = ECDsa.Create(ECCurve.NamedCurves.nistP256); + var parameters = ecdsa.ExportParameters(includePrivateParameters: true); + return new CryptoSigningKey(new CryptoKeyReference(keyId), SignatureAlgorithms.Es256, in parameters, DateTimeOffset.UtcNow); + } + + private static (string Signature, DateTimeOffset SignedAt) CreateDetachedJws(CryptoSigningKey signingKey, string payload) + { + using var provider = new DefaultCryptoProvider(); + provider.UpsertSigningKey(signingKey); + var signer = provider.GetSigner(SignatureAlgorithms.Es256, signingKey.Reference); + var header = new Dictionary + { + ["alg"] = SignatureAlgorithms.Es256, + ["kid"] = signingKey.Reference.KeyId, + ["provider"] = provider.Name, + ["typ"] = "application/vnd.stellaops.concelier.mirror-bundle+jws", + ["b64"] = false, + ["crit"] = new[] { "b64" } + }; + + var headerJson = JsonSerializer.Serialize(header); + var encodedHeader = Microsoft.IdentityModel.Tokens.Base64UrlEncoder.Encode(headerJson); + var payloadBytes = Encoding.UTF8.GetBytes(payload); + var signingInput = BuildSigningInput(encodedHeader, payloadBytes); + var signatureBytes = signer.SignAsync(signingInput, CancellationToken.None).GetAwaiter().GetResult(); + var encodedSignature = Microsoft.IdentityModel.Tokens.Base64UrlEncoder.Encode(signatureBytes); + return (string.Concat(encodedHeader, "..", encodedSignature), DateTimeOffset.UtcNow); + } + + private static ReadOnlyMemory BuildSigningInput(string encodedHeader, ReadOnlySpan payload) + { + var headerBytes = Encoding.ASCII.GetBytes(encodedHeader); + var buffer = new byte[headerBytes.Length + 1 + payload.Length]; + headerBytes.CopyTo(buffer, 0); + buffer[headerBytes.Length] = (byte)'.'; + payload.CopyTo(buffer.AsSpan(headerBytes.Length + 1)); + return buffer; + } +} diff --git a/src/StellaOps.Concelier.Connector.StellaOpsMirror/Client/MirrorManifestClient.cs b/src/StellaOps.Concelier.Connector.StellaOpsMirror/Client/MirrorManifestClient.cs new file mode 100644 index 00000000..216c9662 --- /dev/null +++ b/src/StellaOps.Concelier.Connector.StellaOpsMirror/Client/MirrorManifestClient.cs @@ -0,0 +1,89 @@ +using System; +using System.Net; +using System.Net.Http.Headers; +using System.Text.Json; +using Microsoft.Extensions.Logging; +using StellaOps.Concelier.Connector.StellaOpsMirror.Internal; + +namespace StellaOps.Concelier.Connector.StellaOpsMirror.Client; + +/// +/// Lightweight HTTP client for retrieving mirror index and domain artefacts. +/// +public sealed class MirrorManifestClient +{ + private static readonly JsonSerializerOptions JsonOptions = new(JsonSerializerDefaults.Web) + { + PropertyNameCaseInsensitive = true, + ReadCommentHandling = JsonCommentHandling.Skip + }; + + private readonly HttpClient _httpClient; + private readonly ILogger _logger; + + public MirrorManifestClient(HttpClient httpClient, ILogger logger) + { + _httpClient = httpClient ?? throw new ArgumentNullException(nameof(httpClient)); + _logger = logger ?? throw new ArgumentNullException(nameof(logger)); + } + + public async Task GetIndexAsync(string indexPath, CancellationToken cancellationToken) + { + if (string.IsNullOrWhiteSpace(indexPath)) + { + throw new ArgumentException("Index path must be provided.", nameof(indexPath)); + } + + using var request = new HttpRequestMessage(HttpMethod.Get, indexPath); + using var response = await _httpClient.SendAsync(request, HttpCompletionOption.ResponseHeadersRead, cancellationToken).ConfigureAwait(false); + + await EnsureSuccessAsync(response, indexPath, cancellationToken).ConfigureAwait(false); + + await using var stream = await response.Content.ReadAsStreamAsync(cancellationToken).ConfigureAwait(false); + var document = await JsonSerializer.DeserializeAsync(stream, JsonOptions, cancellationToken).ConfigureAwait(false); + if (document is null) + { + throw new InvalidOperationException("Mirror index payload was empty."); + } + + return document; + } + + public async Task DownloadAsync(string path, CancellationToken cancellationToken) + { + if (string.IsNullOrWhiteSpace(path)) + { + throw new ArgumentException("Path must be provided.", nameof(path)); + } + + using var request = new HttpRequestMessage(HttpMethod.Get, path); + using var response = await _httpClient.SendAsync(request, HttpCompletionOption.ResponseHeadersRead, cancellationToken).ConfigureAwait(false); + + await EnsureSuccessAsync(response, path, cancellationToken).ConfigureAwait(false); + return await response.Content.ReadAsByteArrayAsync(cancellationToken).ConfigureAwait(false); + } + + private async Task EnsureSuccessAsync(HttpResponseMessage response, string path, CancellationToken cancellationToken) + { + if (response.IsSuccessStatusCode) + { + return; + } + + var status = (int)response.StatusCode; + var body = string.Empty; + + if (response.Content.Headers.ContentLength is long length && length > 0) + { + body = await response.Content.ReadAsStringAsync(cancellationToken).ConfigureAwait(false); + } + + _logger.LogWarning( + "Mirror request to {Path} failed with {StatusCode}. Body: {Body}", + path, + status, + string.IsNullOrEmpty(body) ? "" : body); + + throw new HttpRequestException($"Mirror request to '{path}' failed with status {(HttpStatusCode)status} ({status}).", null, response.StatusCode); + } +} diff --git a/src/StellaOps.Concelier.Connector.StellaOpsMirror/Internal/MirrorIndexDocument.cs b/src/StellaOps.Concelier.Connector.StellaOpsMirror/Internal/MirrorIndexDocument.cs new file mode 100644 index 00000000..bad0df9f --- /dev/null +++ b/src/StellaOps.Concelier.Connector.StellaOpsMirror/Internal/MirrorIndexDocument.cs @@ -0,0 +1,38 @@ +using System; +using System.Collections.Generic; +using System.Text.Json.Serialization; + +namespace StellaOps.Concelier.Connector.StellaOpsMirror.Internal; + +public sealed record MirrorIndexDocument( + [property: JsonPropertyName("schemaVersion")] int SchemaVersion, + [property: JsonPropertyName("generatedAt")] DateTimeOffset GeneratedAt, + [property: JsonPropertyName("targetRepository")] string? TargetRepository, + [property: JsonPropertyName("domains")] IReadOnlyList Domains); + +public sealed record MirrorIndexDomainEntry( + [property: JsonPropertyName("domainId")] string DomainId, + [property: JsonPropertyName("displayName")] string DisplayName, + [property: JsonPropertyName("advisoryCount")] int AdvisoryCount, + [property: JsonPropertyName("manifest")] MirrorFileDescriptor Manifest, + [property: JsonPropertyName("bundle")] MirrorFileDescriptor Bundle, + [property: JsonPropertyName("sources")] IReadOnlyList Sources); + +public sealed record MirrorFileDescriptor( + [property: JsonPropertyName("path")] string Path, + [property: JsonPropertyName("sizeBytes")] long SizeBytes, + [property: JsonPropertyName("digest")] string Digest, + [property: JsonPropertyName("signature")] MirrorSignatureDescriptor? Signature); + +public sealed record MirrorSignatureDescriptor( + [property: JsonPropertyName("path")] string Path, + [property: JsonPropertyName("algorithm")] string Algorithm, + [property: JsonPropertyName("keyId")] string KeyId, + [property: JsonPropertyName("provider")] string Provider, + [property: JsonPropertyName("signedAt")] DateTimeOffset SignedAt); + +public sealed record MirrorSourceSummary( + [property: JsonPropertyName("source")] string Source, + [property: JsonPropertyName("firstRecordedAt")] DateTimeOffset? FirstRecordedAt, + [property: JsonPropertyName("lastRecordedAt")] DateTimeOffset? LastRecordedAt, + [property: JsonPropertyName("advisoryCount")] int AdvisoryCount); diff --git a/src/StellaOps.Concelier.Connector.StellaOpsMirror/Internal/StellaOpsMirrorCursor.cs b/src/StellaOps.Concelier.Connector.StellaOpsMirror/Internal/StellaOpsMirrorCursor.cs new file mode 100644 index 00000000..b3500309 --- /dev/null +++ b/src/StellaOps.Concelier.Connector.StellaOpsMirror/Internal/StellaOpsMirrorCursor.cs @@ -0,0 +1,111 @@ +using System.Linq; +using MongoDB.Bson; + +namespace StellaOps.Concelier.Connector.StellaOpsMirror.Internal; + +internal sealed record StellaOpsMirrorCursor( + string? ExportId, + string? BundleDigest, + DateTimeOffset? GeneratedAt, + IReadOnlyCollection PendingDocuments, + IReadOnlyCollection PendingMappings) +{ + private static readonly IReadOnlyCollection EmptyGuids = Array.Empty(); + + public static StellaOpsMirrorCursor Empty { get; } = new( + ExportId: null, + BundleDigest: null, + GeneratedAt: null, + PendingDocuments: EmptyGuids, + PendingMappings: EmptyGuids); + + public BsonDocument ToBsonDocument() + { + var document = new BsonDocument + { + ["pendingDocuments"] = new BsonArray(PendingDocuments.Select(id => id.ToString())), + ["pendingMappings"] = new BsonArray(PendingMappings.Select(id => id.ToString())), + }; + + if (!string.IsNullOrWhiteSpace(ExportId)) + { + document["exportId"] = ExportId; + } + + if (!string.IsNullOrWhiteSpace(BundleDigest)) + { + document["bundleDigest"] = BundleDigest; + } + + if (GeneratedAt.HasValue) + { + document["generatedAt"] = GeneratedAt.Value.UtcDateTime; + } + + return document; + } + + public static StellaOpsMirrorCursor FromBson(BsonDocument? document) + { + if (document is null || document.ElementCount == 0) + { + return Empty; + } + + var exportId = document.TryGetValue("exportId", out var exportValue) && exportValue.IsString ? exportValue.AsString : null; + var digest = document.TryGetValue("bundleDigest", out var digestValue) && digestValue.IsString ? digestValue.AsString : null; + DateTimeOffset? generatedAt = null; + if (document.TryGetValue("generatedAt", out var generatedValue)) + { + generatedAt = generatedValue.BsonType switch + { + BsonType.DateTime => DateTime.SpecifyKind(generatedValue.ToUniversalTime(), DateTimeKind.Utc), + BsonType.String when DateTimeOffset.TryParse(generatedValue.AsString, out var parsed) => parsed.ToUniversalTime(), + _ => null, + }; + } + + var pendingDocuments = ReadGuidArray(document, "pendingDocuments"); + var pendingMappings = ReadGuidArray(document, "pendingMappings"); + + return new StellaOpsMirrorCursor(exportId, digest, generatedAt, pendingDocuments, pendingMappings); + } + + public StellaOpsMirrorCursor WithPendingDocuments(IEnumerable documents) + => this with { PendingDocuments = documents?.Distinct().ToArray() ?? EmptyGuids }; + + public StellaOpsMirrorCursor WithPendingMappings(IEnumerable mappings) + => this with { PendingMappings = mappings?.Distinct().ToArray() ?? EmptyGuids }; + + public StellaOpsMirrorCursor WithBundleSnapshot(string? exportId, string? digest, DateTimeOffset generatedAt) + => this with + { + ExportId = string.IsNullOrWhiteSpace(exportId) ? ExportId : exportId, + BundleDigest = digest, + GeneratedAt = generatedAt, + }; + + private static IReadOnlyCollection ReadGuidArray(BsonDocument document, string field) + { + if (!document.TryGetValue(field, out var value) || value is not BsonArray array) + { + return EmptyGuids; + } + + var results = new List(array.Count); + foreach (var element in array) + { + if (element is null) + { + continue; + } + + if (Guid.TryParse(element.ToString(), out var guid)) + { + results.Add(guid); + } + } + + return results; + } +} diff --git a/src/StellaOps.Concelier.Connector.StellaOpsMirror/Jobs.cs b/src/StellaOps.Concelier.Connector.StellaOpsMirror/Jobs.cs new file mode 100644 index 00000000..fe0f2faf --- /dev/null +++ b/src/StellaOps.Concelier.Connector.StellaOpsMirror/Jobs.cs @@ -0,0 +1,43 @@ +using StellaOps.Concelier.Core.Jobs; + +namespace StellaOps.Concelier.Connector.StellaOpsMirror; + +internal static class StellaOpsMirrorJobKinds +{ + public const string Fetch = "source:stellaops-mirror:fetch"; + public const string Parse = "source:stellaops-mirror:parse"; + public const string Map = "source:stellaops-mirror:map"; +} + +internal sealed class StellaOpsMirrorFetchJob : IJob +{ + private readonly StellaOpsMirrorConnector _connector; + + public StellaOpsMirrorFetchJob(StellaOpsMirrorConnector connector) + => _connector = connector ?? throw new ArgumentNullException(nameof(connector)); + + public Task ExecuteAsync(JobExecutionContext context, CancellationToken cancellationToken) + => _connector.FetchAsync(context.Services, cancellationToken); +} + +internal sealed class StellaOpsMirrorParseJob : IJob +{ + private readonly StellaOpsMirrorConnector _connector; + + public StellaOpsMirrorParseJob(StellaOpsMirrorConnector connector) + => _connector = connector ?? throw new ArgumentNullException(nameof(connector)); + + public Task ExecuteAsync(JobExecutionContext context, CancellationToken cancellationToken) + => _connector.ParseAsync(context.Services, cancellationToken); +} + +internal sealed class StellaOpsMirrorMapJob : IJob +{ + private readonly StellaOpsMirrorConnector _connector; + + public StellaOpsMirrorMapJob(StellaOpsMirrorConnector connector) + => _connector = connector ?? throw new ArgumentNullException(nameof(connector)); + + public Task ExecuteAsync(JobExecutionContext context, CancellationToken cancellationToken) + => _connector.MapAsync(context.Services, cancellationToken); +} diff --git a/src/StellaOps.Concelier.Connector.StellaOpsMirror/Security/MirrorSignatureVerifier.cs b/src/StellaOps.Concelier.Connector.StellaOpsMirror/Security/MirrorSignatureVerifier.cs new file mode 100644 index 00000000..c80c01c3 --- /dev/null +++ b/src/StellaOps.Concelier.Connector.StellaOpsMirror/Security/MirrorSignatureVerifier.cs @@ -0,0 +1,121 @@ +using System; +using System.Text; +using System.Text.Json; +using System.Text.Json.Serialization; +using Microsoft.Extensions.Logging; +using Microsoft.IdentityModel.Tokens; +using StellaOps.Cryptography; + +namespace StellaOps.Concelier.Connector.StellaOpsMirror.Security; + +/// +/// Validates detached JWS signatures emitted by mirror bundles. +/// +public sealed class MirrorSignatureVerifier +{ + private static readonly JsonSerializerOptions HeaderSerializerOptions = new(JsonSerializerDefaults.Web) + { + PropertyNameCaseInsensitive = true + }; + + private readonly ICryptoProviderRegistry _providerRegistry; + private readonly ILogger _logger; + + public MirrorSignatureVerifier(ICryptoProviderRegistry providerRegistry, ILogger logger) + { + _providerRegistry = providerRegistry ?? throw new ArgumentNullException(nameof(providerRegistry)); + _logger = logger ?? throw new ArgumentNullException(nameof(logger)); + } + + public async Task VerifyAsync(ReadOnlyMemory payload, string signatureValue, CancellationToken cancellationToken) + { + if (payload.IsEmpty) + { + throw new ArgumentException("Payload must not be empty.", nameof(payload)); + } + + if (string.IsNullOrWhiteSpace(signatureValue)) + { + throw new ArgumentException("Signature value must be provided.", nameof(signatureValue)); + } + + if (!TryParseDetachedJws(signatureValue, out var encodedHeader, out var encodedSignature)) + { + throw new InvalidOperationException("Detached JWS signature is malformed."); + } + + var headerJson = Encoding.UTF8.GetString(Base64UrlEncoder.DecodeBytes(encodedHeader)); + var header = JsonSerializer.Deserialize(headerJson, HeaderSerializerOptions) + ?? throw new InvalidOperationException("Detached JWS header could not be parsed."); + + if (!header.Critical.Contains("b64", StringComparer.Ordinal)) + { + throw new InvalidOperationException("Detached JWS header is missing required 'b64' critical parameter."); + } + + if (header.Base64Payload) + { + throw new InvalidOperationException("Detached JWS header sets b64=true; expected unencoded payload."); + } + + if (string.IsNullOrWhiteSpace(header.KeyId)) + { + throw new InvalidOperationException("Detached JWS header missing key identifier."); + } + + if (string.IsNullOrWhiteSpace(header.Algorithm)) + { + throw new InvalidOperationException("Detached JWS header missing algorithm identifier."); + } + + var signingInput = BuildSigningInput(encodedHeader, payload.Span); + var signatureBytes = Base64UrlEncoder.DecodeBytes(encodedSignature); + + var keyReference = new CryptoKeyReference(header.KeyId, header.Provider); + var resolution = _providerRegistry.ResolveSigner( + CryptoCapability.Verification, + header.Algorithm, + keyReference, + header.Provider); + + var verified = await resolution.Signer.VerifyAsync(signingInput, signatureBytes, cancellationToken).ConfigureAwait(false); + if (!verified) + { + _logger.LogWarning("Detached JWS verification failed for key {KeyId} via provider {Provider}.", header.KeyId, resolution.ProviderName); + throw new InvalidOperationException("Detached JWS signature verification failed."); + } + } + + private static bool TryParseDetachedJws(string value, out string encodedHeader, out string encodedSignature) + { + var parts = value.Split("..", StringSplitOptions.None); + if (parts.Length != 2 || string.IsNullOrEmpty(parts[0]) || string.IsNullOrEmpty(parts[1])) + { + encodedHeader = string.Empty; + encodedSignature = string.Empty; + return false; + } + + encodedHeader = parts[0]; + encodedSignature = parts[1]; + return true; + } + + private static ReadOnlyMemory BuildSigningInput(string encodedHeader, ReadOnlySpan payload) + { + var headerBytes = Encoding.ASCII.GetBytes(encodedHeader); + var buffer = new byte[headerBytes.Length + 1 + payload.Length]; + headerBytes.CopyTo(buffer.AsSpan()); + buffer[headerBytes.Length] = (byte)'.'; + payload.CopyTo(buffer.AsSpan(headerBytes.Length + 1)); + return buffer; + } + + private sealed record MirrorSignatureHeader( + [property: JsonPropertyName("alg")] string Algorithm, + [property: JsonPropertyName("kid")] string KeyId, + [property: JsonPropertyName("provider")] string? Provider, + [property: JsonPropertyName("typ")] string? Type, + [property: JsonPropertyName("b64")] bool Base64Payload, + [property: JsonPropertyName("crit")] string[] Critical); +} diff --git a/src/StellaOps.Concelier.Connector.StellaOpsMirror/Settings/StellaOpsMirrorConnectorOptions.cs b/src/StellaOps.Concelier.Connector.StellaOpsMirror/Settings/StellaOpsMirrorConnectorOptions.cs new file mode 100644 index 00000000..76a6aa20 --- /dev/null +++ b/src/StellaOps.Concelier.Connector.StellaOpsMirror/Settings/StellaOpsMirrorConnectorOptions.cs @@ -0,0 +1,61 @@ +using System; +using System.ComponentModel.DataAnnotations; + +namespace StellaOps.Concelier.Connector.StellaOpsMirror.Settings; + +/// +/// Configuration for the StellaOps mirror connector HTTP client. +/// +public sealed class StellaOpsMirrorConnectorOptions +{ + /// + /// Base address of the mirror distribution endpoint (e.g., https://mirror.stella-ops.org). + /// + [Required] + public Uri BaseAddress { get; set; } = new("https://mirror.stella-ops.org", UriKind.Absolute); + + /// + /// Relative path to the mirror index document. Defaults to /concelier/exports/index.json. + /// + [Required] + public string IndexPath { get; set; } = "/concelier/exports/index.json"; + + /// + /// Preferred mirror domain identifier when multiple domains are published in the index. + /// + [Required] + public string DomainId { get; set; } = "primary"; + + /// + /// Maximum duration to wait on HTTP requests. + /// + public TimeSpan HttpTimeout { get; set; } = TimeSpan.FromSeconds(30); + + /// + /// Signature verification configuration for downloaded bundles. + /// + public SignatureOptions Signature { get; set; } = new(); + + public sealed class SignatureOptions + { + /// + /// When true, downloaded bundles must include a detached JWS that validates successfully. + /// + public bool Enabled { get; set; } = false; + + /// + /// Expected signing key identifier (kid) emitted in the detached JWS header. + /// + public string KeyId { get; set; } = string.Empty; + + /// + /// Optional crypto provider hint used to resolve verification keys. + /// + public string? Provider { get; set; } + + /// + /// Optional path to a PEM-encoded EC public key used to verify signatures when registry resolution fails. + /// + public string? PublicKeyPath { get; set; } + } +} diff --git a/src/StellaOps.Concelier.Connector.StellaOpsMirror/StellaOps.Concelier.Connector.StellaOpsMirror.csproj b/src/StellaOps.Concelier.Connector.StellaOpsMirror/StellaOps.Concelier.Connector.StellaOpsMirror.csproj new file mode 100644 index 00000000..9e24755e --- /dev/null +++ b/src/StellaOps.Concelier.Connector.StellaOpsMirror/StellaOps.Concelier.Connector.StellaOpsMirror.csproj @@ -0,0 +1,16 @@ + + + + net10.0 + enable + enable + + + + + + + + + + diff --git a/src/StellaOps.Concelier.Connector.StellaOpsMirror/StellaOpsMirrorConnector.cs b/src/StellaOps.Concelier.Connector.StellaOpsMirror/StellaOpsMirrorConnector.cs new file mode 100644 index 00000000..20f3e0ab --- /dev/null +++ b/src/StellaOps.Concelier.Connector.StellaOpsMirror/StellaOpsMirrorConnector.cs @@ -0,0 +1,288 @@ +using System; +using System.Collections.Generic; +using System.Linq; +using System.Security.Cryptography; +using System.Text; +using Microsoft.Extensions.Logging; +using Microsoft.Extensions.Options; +using MongoDB.Bson; +using StellaOps.Concelier.Connector.Common.Fetch; +using StellaOps.Concelier.Connector.Common; +using StellaOps.Concelier.Connector.StellaOpsMirror.Client; +using StellaOps.Concelier.Connector.StellaOpsMirror.Internal; +using StellaOps.Concelier.Connector.StellaOpsMirror.Security; +using StellaOps.Concelier.Connector.StellaOpsMirror.Settings; +using StellaOps.Concelier.Storage.Mongo; +using StellaOps.Concelier.Storage.Mongo.Documents; +using StellaOps.Plugin; + +namespace StellaOps.Concelier.Connector.StellaOpsMirror; + +public sealed class StellaOpsMirrorConnector : IFeedConnector +{ + public const string Source = "stellaops-mirror"; + + private readonly MirrorManifestClient _client; + private readonly MirrorSignatureVerifier _signatureVerifier; + private readonly RawDocumentStorage _rawDocumentStorage; + private readonly IDocumentStore _documentStore; + private readonly ISourceStateRepository _stateRepository; + private readonly TimeProvider _timeProvider; + private readonly ILogger _logger; + private readonly StellaOpsMirrorConnectorOptions _options; + + public StellaOpsMirrorConnector( + MirrorManifestClient client, + MirrorSignatureVerifier signatureVerifier, + RawDocumentStorage rawDocumentStorage, + IDocumentStore documentStore, + ISourceStateRepository stateRepository, + IOptions options, + TimeProvider? timeProvider, + ILogger logger) + { + _client = client ?? throw new ArgumentNullException(nameof(client)); + _signatureVerifier = signatureVerifier ?? throw new ArgumentNullException(nameof(signatureVerifier)); + _rawDocumentStorage = rawDocumentStorage ?? throw new ArgumentNullException(nameof(rawDocumentStorage)); + _documentStore = documentStore ?? throw new ArgumentNullException(nameof(documentStore)); + _stateRepository = stateRepository ?? throw new ArgumentNullException(nameof(stateRepository)); + _logger = logger ?? throw new ArgumentNullException(nameof(logger)); + _timeProvider = timeProvider ?? TimeProvider.System; + _options = (options ?? throw new ArgumentNullException(nameof(options))).Value ?? throw new ArgumentNullException(nameof(options)); + ValidateOptions(_options); + } + + public string SourceName => Source; + + public async Task FetchAsync(IServiceProvider services, CancellationToken cancellationToken) + { + _ = services ?? throw new ArgumentNullException(nameof(services)); + + var now = _timeProvider.GetUtcNow(); + var cursor = await GetCursorAsync(cancellationToken).ConfigureAwait(false); + var pendingDocuments = cursor.PendingDocuments.ToHashSet(); + var pendingMappings = cursor.PendingMappings.ToHashSet(); + + MirrorIndexDocument index; + try + { + index = await _client.GetIndexAsync(_options.IndexPath, cancellationToken).ConfigureAwait(false); + } + catch (Exception ex) + { + await _stateRepository.MarkFailureAsync(Source, now, TimeSpan.FromMinutes(15), ex.Message, cancellationToken).ConfigureAwait(false); + throw; + } + + var domain = index.Domains.FirstOrDefault(entry => + string.Equals(entry.DomainId, _options.DomainId, StringComparison.OrdinalIgnoreCase)); + + if (domain is null) + { + var message = $"Mirror domain '{_options.DomainId}' not present in index."; + await _stateRepository.MarkFailureAsync(Source, now, TimeSpan.FromMinutes(30), message, cancellationToken).ConfigureAwait(false); + throw new InvalidOperationException(message); + } + + if (string.Equals(domain.Bundle.Digest, cursor.BundleDigest, StringComparison.OrdinalIgnoreCase)) + { + _logger.LogInformation("Mirror bundle digest {Digest} unchanged; skipping fetch.", domain.Bundle.Digest); + return; + } + + try + { + await ProcessDomainAsync(index, domain, pendingDocuments, cancellationToken).ConfigureAwait(false); + } + catch (Exception ex) + { + await _stateRepository.MarkFailureAsync(Source, now, TimeSpan.FromMinutes(10), ex.Message, cancellationToken).ConfigureAwait(false); + throw; + } + + var updatedCursor = cursor + .WithPendingDocuments(pendingDocuments) + .WithPendingMappings(pendingMappings) + .WithBundleSnapshot(domain.Bundle.Path, domain.Bundle.Digest, index.GeneratedAt); + + await UpdateCursorAsync(updatedCursor, cancellationToken).ConfigureAwait(false); + } + + public Task ParseAsync(IServiceProvider services, CancellationToken cancellationToken) + => Task.CompletedTask; + + public Task MapAsync(IServiceProvider services, CancellationToken cancellationToken) + => Task.CompletedTask; + + private async Task ProcessDomainAsync( + MirrorIndexDocument index, + MirrorIndexDomainEntry domain, + HashSet pendingDocuments, + CancellationToken cancellationToken) + { + var manifestBytes = await _client.DownloadAsync(domain.Manifest.Path, cancellationToken).ConfigureAwait(false); + var bundleBytes = await _client.DownloadAsync(domain.Bundle.Path, cancellationToken).ConfigureAwait(false); + + VerifyDigest(domain.Manifest.Digest, manifestBytes, domain.Manifest.Path); + VerifyDigest(domain.Bundle.Digest, bundleBytes, domain.Bundle.Path); + + if (_options.Signature.Enabled) + { + if (domain.Bundle.Signature is null) + { + throw new InvalidOperationException("Mirror bundle did not include a signature descriptor while verification is enabled."); + } + + var signatureBytes = await _client.DownloadAsync(domain.Bundle.Signature.Path, cancellationToken).ConfigureAwait(false); + var signatureValue = Encoding.UTF8.GetString(signatureBytes); + await _signatureVerifier.VerifyAsync(bundleBytes, signatureValue, cancellationToken).ConfigureAwait(false); + } + + await StoreAsync(domain, index.GeneratedAt, domain.Manifest, manifestBytes, "application/json", DocumentStatuses.Mapped, addToPending: false, pendingDocuments, cancellationToken).ConfigureAwait(false); + var bundleRecord = await StoreAsync(domain, index.GeneratedAt, domain.Bundle, bundleBytes, "application/json", DocumentStatuses.PendingParse, addToPending: true, pendingDocuments, cancellationToken).ConfigureAwait(false); + + _logger.LogInformation( + "Stored mirror bundle {Uri} as document {DocumentId} with digest {Digest}.", + bundleRecord.Uri, + bundleRecord.Id, + bundleRecord.Sha256); + } + + private async Task StoreAsync( + MirrorIndexDomainEntry domain, + DateTimeOffset generatedAt, + MirrorFileDescriptor descriptor, + byte[] payload, + string contentType, + string status, + bool addToPending, + HashSet pendingDocuments, + CancellationToken cancellationToken) + { + var absolute = ResolveAbsolutePath(descriptor.Path); + + var existing = await _documentStore.FindBySourceAndUriAsync(Source, absolute, cancellationToken).ConfigureAwait(false); + if (existing is not null && string.Equals(existing.Sha256, NormalizeDigest(descriptor.Digest), StringComparison.OrdinalIgnoreCase)) + { + if (addToPending) + { + pendingDocuments.Add(existing.Id); + } + + return existing; + } + + var gridFsId = await _rawDocumentStorage.UploadAsync(Source, absolute, payload, contentType, cancellationToken).ConfigureAwait(false); + var now = _timeProvider.GetUtcNow(); + var sha = ComputeSha256(payload); + + var metadata = new Dictionary(StringComparer.OrdinalIgnoreCase) + { + ["mirror.domainId"] = domain.DomainId, + ["mirror.displayName"] = domain.DisplayName, + ["mirror.path"] = descriptor.Path, + ["mirror.digest"] = NormalizeDigest(descriptor.Digest), + ["mirror.type"] = ReferenceEquals(descriptor, domain.Bundle) ? "bundle" : "manifest", + }; + + var record = new DocumentRecord( + existing?.Id ?? Guid.NewGuid(), + Source, + absolute, + now, + sha, + status, + contentType, + Headers: null, + Metadata: metadata, + Etag: null, + LastModified: generatedAt, + GridFsId: gridFsId, + ExpiresAt: null); + + var upserted = await _documentStore.UpsertAsync(record, cancellationToken).ConfigureAwait(false); + + if (addToPending) + { + pendingDocuments.Add(upserted.Id); + } + + return upserted; + } + + private string ResolveAbsolutePath(string path) + { + var uri = new Uri(_options.BaseAddress, path); + return uri.ToString(); + } + + private async Task GetCursorAsync(CancellationToken cancellationToken) + { + var state = await _stateRepository.TryGetAsync(Source, cancellationToken).ConfigureAwait(false); + return state is null ? StellaOpsMirrorCursor.Empty : StellaOpsMirrorCursor.FromBson(state.Cursor); + } + + private async Task UpdateCursorAsync(StellaOpsMirrorCursor cursor, CancellationToken cancellationToken) + { + var document = cursor.ToBsonDocument(); + var now = _timeProvider.GetUtcNow(); + await _stateRepository.UpdateCursorAsync(Source, document, now, cancellationToken).ConfigureAwait(false); + } + + private static void VerifyDigest(string expected, ReadOnlySpan payload, string path) + { + if (string.IsNullOrWhiteSpace(expected)) + { + return; + } + + if (!expected.StartsWith("sha256:", StringComparison.OrdinalIgnoreCase)) + { + throw new InvalidOperationException($"Unsupported digest '{expected}' for '{path}'."); + } + + var actualHash = SHA256.HashData(payload); + var actual = "sha256:" + Convert.ToHexString(actualHash).ToLowerInvariant(); + if (!string.Equals(actual, expected, StringComparison.OrdinalIgnoreCase)) + { + throw new InvalidOperationException($"Digest mismatch for '{path}'. Expected {expected}, computed {actual}."); + } + } + + private static string ComputeSha256(ReadOnlySpan payload) + { + var hash = SHA256.HashData(payload); + return Convert.ToHexString(hash).ToLowerInvariant(); + } + + private static string NormalizeDigest(string digest) + { + if (string.IsNullOrWhiteSpace(digest)) + { + return string.Empty; + } + + return digest.StartsWith("sha256:", StringComparison.OrdinalIgnoreCase) + ? digest[7..] + : digest.ToLowerInvariant(); + } + + private static void ValidateOptions(StellaOpsMirrorConnectorOptions options) + { + if (options.BaseAddress is null || !options.BaseAddress.IsAbsoluteUri) + { + throw new InvalidOperationException("Mirror connector requires an absolute baseAddress."); + } + + if (string.IsNullOrWhiteSpace(options.DomainId)) + { + throw new InvalidOperationException("Mirror connector requires domainId to be specified."); + } + } +} + +file static class UriExtensions +{ + public static Uri Combine(this Uri baseUri, string relative) + => new(baseUri, relative); +} diff --git a/src/StellaOps.Concelier.Connector.StellaOpsMirror/StellaOpsMirrorConnectorPlugin.cs b/src/StellaOps.Concelier.Connector.StellaOpsMirror/StellaOpsMirrorConnectorPlugin.cs new file mode 100644 index 00000000..651faeff --- /dev/null +++ b/src/StellaOps.Concelier.Connector.StellaOpsMirror/StellaOpsMirrorConnectorPlugin.cs @@ -0,0 +1,19 @@ +using Microsoft.Extensions.DependencyInjection; +using StellaOps.Plugin; + +namespace StellaOps.Concelier.Connector.StellaOpsMirror; + +public sealed class StellaOpsMirrorConnectorPlugin : IConnectorPlugin +{ + public const string SourceName = StellaOpsMirrorConnector.Source; + + public string Name => SourceName; + + public bool IsAvailable(IServiceProvider services) => services is not null; + + public IFeedConnector Create(IServiceProvider services) + { + ArgumentNullException.ThrowIfNull(services); + return ActivatorUtilities.CreateInstance(services); + } +} diff --git a/src/StellaOps.Concelier.Connector.StellaOpsMirror/StellaOpsMirrorDependencyInjectionRoutine.cs b/src/StellaOps.Concelier.Connector.StellaOpsMirror/StellaOpsMirrorDependencyInjectionRoutine.cs new file mode 100644 index 00000000..7d87fcee --- /dev/null +++ b/src/StellaOps.Concelier.Connector.StellaOpsMirror/StellaOpsMirrorDependencyInjectionRoutine.cs @@ -0,0 +1,79 @@ +using Microsoft.Extensions.Configuration; +using Microsoft.Extensions.DependencyInjection; +using Microsoft.Extensions.DependencyInjection.Extensions; +using Microsoft.Extensions.Options; +using StellaOps.Concelier.Connector.Common.Http; +using StellaOps.Concelier.Connector.StellaOpsMirror.Client; +using StellaOps.Concelier.Connector.StellaOpsMirror.Security; +using StellaOps.Concelier.Connector.StellaOpsMirror.Settings; +using StellaOps.Concelier.Core.Jobs; +using StellaOps.DependencyInjection; + +namespace StellaOps.Concelier.Connector.StellaOpsMirror; + +public sealed class StellaOpsMirrorDependencyInjectionRoutine : IDependencyInjectionRoutine +{ + private const string ConfigurationSection = "concelier:sources:stellaopsMirror"; + private const string HttpClientName = "stellaops-mirror"; + private static readonly TimeSpan FetchTimeout = TimeSpan.FromMinutes(5); + private static readonly TimeSpan LeaseDuration = TimeSpan.FromMinutes(4); + + public IServiceCollection Register(IServiceCollection services, IConfiguration configuration) + { + ArgumentNullException.ThrowIfNull(services); + ArgumentNullException.ThrowIfNull(configuration); + + services.AddOptions() + .Bind(configuration.GetSection(ConfigurationSection)) + .PostConfigure(options => + { + if (options.BaseAddress is null) + { + throw new InvalidOperationException("stellaopsMirror.baseAddress must be configured."); + } + }) + .ValidateOnStart(); + + services.AddSourceCommon(); + + services.AddHttpClient(HttpClientName, (sp, client) => + { + var options = sp.GetRequiredService>().Value; + client.BaseAddress = options.BaseAddress; + client.Timeout = options.HttpTimeout; + client.DefaultRequestHeaders.Accept.Clear(); + client.DefaultRequestHeaders.Accept.Add(new System.Net.Http.Headers.MediaTypeWithQualityHeaderValue("application/json")); + }); + + services.AddTransient(sp => + { + var factory = sp.GetRequiredService(); + var httpClient = factory.CreateClient(HttpClientName); + return ActivatorUtilities.CreateInstance(sp, httpClient); + }); + + services.TryAddSingleton(); + services.AddTransient(); + + var scheduler = new JobSchedulerBuilder(services); + scheduler.AddJob( + StellaOpsMirrorJobKinds.Fetch, + cronExpression: "*/15 * * * *", + timeout: FetchTimeout, + leaseDuration: LeaseDuration); + scheduler.AddJob( + StellaOpsMirrorJobKinds.Parse, + cronExpression: null, + timeout: TimeSpan.FromMinutes(5), + leaseDuration: LeaseDuration, + enabled: false); + scheduler.AddJob( + StellaOpsMirrorJobKinds.Map, + cronExpression: null, + timeout: TimeSpan.FromMinutes(5), + leaseDuration: LeaseDuration, + enabled: false); + + return services; + } +} diff --git a/src/StellaOps.Concelier.Connector.StellaOpsMirror/TASKS.md b/src/StellaOps.Concelier.Connector.StellaOpsMirror/TASKS.md index 3d8feffc..b2480fed 100644 --- a/src/StellaOps.Concelier.Connector.StellaOpsMirror/TASKS.md +++ b/src/StellaOps.Concelier.Connector.StellaOpsMirror/TASKS.md @@ -2,6 +2,6 @@ | ID | Status | Owner(s) | Depends on | Description | Exit Criteria | |----|--------|----------|------------|-------------|---------------| -| FEEDCONN-STELLA-08-001 | TODO | BE-Conn-Stella | CONCELIER-EXPORT-08-201 | Implement Concelier mirror fetcher hitting `https://.stella-ops.org/concelier/exports/index.json`, verify signatures/digests, and persist raw documents with provenance. | Fetch job downloads mirror manifest, verifies digest/signature, stores raw docs with tests covering happy-path + tampered manifest. | +| FEEDCONN-STELLA-08-001 | DOING (2025-10-19) | BE-Conn-Stella | CONCELIER-EXPORT-08-201 | Implement Concelier mirror fetcher hitting `https://.stella-ops.org/concelier/exports/index.json`, verify signatures/digests, and persist raw documents with provenance. | Fetch job downloads mirror manifest, verifies digest/signature, stores raw docs with tests covering happy-path + tampered manifest. *(In progress: HTTP client + detached JWS verifier scaffolding landed.)* | | FEEDCONN-STELLA-08-002 | TODO | BE-Conn-Stella | FEEDCONN-STELLA-08-001 | Map mirror payloads into canonical advisory DTOs with provenance referencing mirror domain + original source metadata. | Mapper produces advisories/aliases/affected with mirror provenance; fixtures assert canonical parity with upstream JSON exporters. | | FEEDCONN-STELLA-08-003 | TODO | BE-Conn-Stella | FEEDCONN-STELLA-08-002 | Add incremental cursor + resume support (per-export fingerprint) and document configuration for downstream Concelier instances. | Connector resumes from last export, handles deletion/delta cases, docs updated with config sample; integration test covers resume + new export scenario. | diff --git a/src/StellaOps.Concelier.Connector.Vndr.Cisco/TASKS.md b/src/StellaOps.Concelier.Connector.Vndr.Cisco/TASKS.md index 8f113ef5..5086d9b5 100644 --- a/src/StellaOps.Concelier.Connector.Vndr.Cisco/TASKS.md +++ b/src/StellaOps.Concelier.Connector.Vndr.Cisco/TASKS.md @@ -9,3 +9,4 @@ |FEEDCONN-CISCO-02-006 Telemetry & documentation|DevEx|Docs|**DONE (2025-10-14)** – Cisco diagnostics counters exposed and ops runbook updated with telemetry guidance (`docs/ops/concelier-cisco-operations.md`).| |FEEDCONN-CISCO-02-007 API selection decision memo|BE-Conn-Cisco|Research|**DONE (2025-10-11)** – Drafted decision matrix: openVuln (structured/delta filters, OAuth throttle) vs RSS (delayed/minimal metadata). Pending OAuth onboarding (`FEEDCONN-CISCO-02-008`) before final recommendation circulated.| |FEEDCONN-CISCO-02-008 OAuth client provisioning|Ops, BE-Conn-Cisco|Ops|**DONE (2025-10-14)** – `docs/ops/concelier-cisco-operations.md` documents OAuth provisioning/rotation, quotas, and Offline Kit distribution guidance.| +|FEEDCONN-CISCO-02-009 Normalized SemVer promotion|BE-Conn-Cisco|Merge coordination (`FEEDMERGE-COORD-02-900`)|**TODO (due 2025-10-21)** – Use helper from `../Merge/RANGE_PRIMITIVES_COORDINATION.md` to convert `SemVerPrimitive` outputs into `NormalizedVersionRule` with provenance (`cisco:{productId}`), update mapper/tests, and confirm merge normalized-rule counters drop.| diff --git a/src/StellaOps.Concelier.Core.Tests/Events/AdvisoryEventLogTests.cs b/src/StellaOps.Concelier.Core.Tests/Events/AdvisoryEventLogTests.cs new file mode 100644 index 00000000..0b1be81f --- /dev/null +++ b/src/StellaOps.Concelier.Core.Tests/Events/AdvisoryEventLogTests.cs @@ -0,0 +1,198 @@ +using System; +using System.Collections.Generic; +using System.Collections.Immutable; +using System.Linq; +using System.Text.Json; +using System.Threading; +using System.Threading.Tasks; +using StellaOps.Concelier.Core.Events; +using StellaOps.Concelier.Models; +using Xunit; + +namespace StellaOps.Concelier.Core.Tests.Events; + +public sealed class AdvisoryEventLogTests +{ + [Fact] + public async Task AppendAsync_PersistsCanonicalStatementEntries() + { + var repository = new FakeRepository(); + var timeProvider = new FixedTimeProvider(DateTimeOffset.UtcNow); + var log = new AdvisoryEventLog(repository, timeProvider); + + var advisory = new Advisory( + "adv-1", + "Test Advisory", + summary: "Summary", + language: "en", + published: DateTimeOffset.Parse("2025-10-01T00:00:00Z"), + modified: DateTimeOffset.Parse("2025-10-02T00:00:00Z"), + severity: "high", + exploitKnown: true, + aliases: new[] { "CVE-2025-0001" }, + references: Array.Empty(), + affectedPackages: Array.Empty(), + cvssMetrics: Array.Empty(), + provenance: Array.Empty()); + + var statementInput = new AdvisoryStatementInput( + VulnerabilityKey: "CVE-2025-0001", + Advisory: advisory, + AsOf: DateTimeOffset.Parse("2025-10-03T00:00:00Z"), + InputDocumentIds: new[] { Guid.Parse("11111111-1111-1111-1111-111111111111") }); + + await log.AppendAsync(new AdvisoryEventAppendRequest(new[] { statementInput }), CancellationToken.None); + + Assert.Single(repository.InsertedStatements); + var entry = repository.InsertedStatements.Single(); + Assert.Equal("cve-2025-0001", entry.VulnerabilityKey); + Assert.Equal("adv-1", entry.AdvisoryKey); + Assert.Equal(DateTimeOffset.Parse("2025-10-03T00:00:00Z"), entry.AsOf); + Assert.Contains("\"advisoryKey\":\"adv-1\"", entry.CanonicalJson); + Assert.NotEqual(ImmutableArray.Empty, entry.StatementHash); + } + + [Fact] + public async Task AppendAsync_PersistsConflictsWithCanonicalizedJson() + { + var repository = new FakeRepository(); + var timeProvider = new FixedTimeProvider(DateTimeOffset.Parse("2025-10-19T12:00:00Z")); + var log = new AdvisoryEventLog(repository, timeProvider); + + using var conflictJson = JsonDocument.Parse("{\"reason\":\"tie\",\"details\":{\"b\":2,\"a\":1}}"); + var conflictInput = new AdvisoryConflictInput( + VulnerabilityKey: "CVE-2025-0001", + Details: conflictJson, + AsOf: DateTimeOffset.Parse("2025-10-04T00:00:00Z"), + StatementIds: new[] { Guid.Parse("22222222-2222-2222-2222-222222222222") }); + + await log.AppendAsync(new AdvisoryEventAppendRequest(Array.Empty(), new[] { conflictInput }), CancellationToken.None); + + Assert.Single(repository.InsertedConflicts); + var entry = repository.InsertedConflicts.Single(); + Assert.Equal("cve-2025-0001", entry.VulnerabilityKey); + Assert.Equal("{\"details\":{\"a\":1,\"b\":2},\"reason\":\"tie\"}", entry.CanonicalJson); + Assert.NotEqual(ImmutableArray.Empty, entry.ConflictHash); + Assert.Equal(DateTimeOffset.Parse("2025-10-04T00:00:00Z"), entry.AsOf); + } + + [Fact] + public async Task ReplayAsync_ReturnsSortedSnapshots() + { + var repository = new FakeRepository(); + var timeProvider = new FixedTimeProvider(DateTimeOffset.Parse("2025-10-05T00:00:00Z")); + var log = new AdvisoryEventLog(repository, timeProvider); + + repository.StoredStatements.AddRange(new[] + { + new AdvisoryStatementEntry( + Guid.Parse("aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa"), + "cve-2025-0001", + "adv-2", + CanonicalJsonSerializer.Serialize(new Advisory( + "adv-2", + "B title", + null, + null, + null, + DateTimeOffset.Parse("2025-10-02T00:00:00Z"), + null, + false, + Array.Empty(), + Array.Empty(), + Array.Empty(), + Array.Empty(), + Array.Empty())), + ImmutableArray.Create(new byte[] { 0x01, 0x02 }), + DateTimeOffset.Parse("2025-10-04T00:00:00Z"), + DateTimeOffset.Parse("2025-10-04T01:00:00Z"), + ImmutableArray.Empty), + new AdvisoryStatementEntry( + Guid.Parse("bbbbbbbb-bbbb-bbbb-bbbb-bbbbbbbbbbbb"), + "cve-2025-0001", + "adv-1", + CanonicalJsonSerializer.Serialize(new Advisory( + "adv-1", + "A title", + null, + null, + null, + DateTimeOffset.Parse("2025-10-01T00:00:00Z"), + null, + false, + Array.Empty(), + Array.Empty(), + Array.Empty(), + Array.Empty(), + Array.Empty())), + ImmutableArray.Create(new byte[] { 0x03, 0x04 }), + DateTimeOffset.Parse("2025-10-03T00:00:00Z"), + DateTimeOffset.Parse("2025-10-04T02:00:00Z"), + ImmutableArray.Empty), + }); + + repository.StoredConflicts.Add(new AdvisoryConflictEntry( + Guid.Parse("cccccccc-cccc-cccc-cccc-cccccccccccc"), + "cve-2025-0001", + CanonicalJson: "{\"reason\":\"conflict\"}", + ConflictHash: ImmutableArray.Create(new byte[] { 0x10 }), + AsOf: DateTimeOffset.Parse("2025-10-04T00:00:00Z"), + RecordedAt: DateTimeOffset.Parse("2025-10-04T03:00:00Z"), + StatementIds: ImmutableArray.Empty)); + + var replay = await log.ReplayAsync("CVE-2025-0001", asOf: null, CancellationToken.None); + + Assert.Equal("cve-2025-0001", replay.VulnerabilityKey); + Assert.Collection( + replay.Statements, + first => Assert.Equal("adv-2", first.AdvisoryKey), + second => Assert.Equal("adv-1", second.AdvisoryKey)); + Assert.Single(replay.Conflicts); + Assert.Equal("{\"reason\":\"conflict\"}", replay.Conflicts[0].CanonicalJson); + } + + private sealed class FakeRepository : IAdvisoryEventRepository + { + public List InsertedStatements { get; } = new(); + + public List InsertedConflicts { get; } = new(); + + public List StoredStatements { get; } = new(); + + public List StoredConflicts { get; } = new(); + + public ValueTask InsertStatementsAsync(IReadOnlyCollection statements, CancellationToken cancellationToken) + { + InsertedStatements.AddRange(statements); + return ValueTask.CompletedTask; + } + + public ValueTask InsertConflictsAsync(IReadOnlyCollection conflicts, CancellationToken cancellationToken) + { + InsertedConflicts.AddRange(conflicts); + return ValueTask.CompletedTask; + } + + public ValueTask> GetStatementsAsync(string vulnerabilityKey, DateTimeOffset? asOf, CancellationToken cancellationToken) + => ValueTask.FromResult>(StoredStatements.Where(entry => + string.Equals(entry.VulnerabilityKey, vulnerabilityKey, StringComparison.Ordinal) && + (!asOf.HasValue || entry.AsOf <= asOf.Value)).ToList()); + + public ValueTask> GetConflictsAsync(string vulnerabilityKey, DateTimeOffset? asOf, CancellationToken cancellationToken) + => ValueTask.FromResult>(StoredConflicts.Where(entry => + string.Equals(entry.VulnerabilityKey, vulnerabilityKey, StringComparison.Ordinal) && + (!asOf.HasValue || entry.AsOf <= asOf.Value)).ToList()); + } + + private sealed class FixedTimeProvider : TimeProvider + { + private readonly DateTimeOffset _now; + + public FixedTimeProvider(DateTimeOffset now) + { + _now = now.ToUniversalTime(); + } + + public override DateTimeOffset GetUtcNow() => _now; + } +} diff --git a/src/StellaOps.Concelier.Core/Events/AdvisoryEventContracts.cs b/src/StellaOps.Concelier.Core/Events/AdvisoryEventContracts.cs new file mode 100644 index 00000000..5aa8895e --- /dev/null +++ b/src/StellaOps.Concelier.Core/Events/AdvisoryEventContracts.cs @@ -0,0 +1,93 @@ +using System; +using System.Collections.Immutable; +using System.Text.Json; +using StellaOps.Concelier.Models; + +namespace StellaOps.Concelier.Core.Events; + +/// +/// Input payload for appending a canonical advisory statement to the event log. +/// +public sealed record AdvisoryStatementInput( + string VulnerabilityKey, + Advisory Advisory, + DateTimeOffset AsOf, + IReadOnlyCollection InputDocumentIds, + Guid? StatementId = null, + string? AdvisoryKey = null); + +/// +/// Input payload for appending an advisory conflict entry aligned with an advisory statement snapshot. +/// +public sealed record AdvisoryConflictInput( + string VulnerabilityKey, + JsonDocument Details, + DateTimeOffset AsOf, + IReadOnlyCollection StatementIds, + Guid? ConflictId = null); + +/// +/// Append request encapsulating statement and conflict batches sharing a single persistence window. +/// +public sealed record AdvisoryEventAppendRequest( + IReadOnlyCollection Statements, + IReadOnlyCollection? Conflicts = null); + +/// +/// Replay response describing immutable statement snapshots for a vulnerability key. +/// +public sealed record AdvisoryReplay( + string VulnerabilityKey, + DateTimeOffset? AsOf, + ImmutableArray Statements, + ImmutableArray Conflicts); + +/// +/// Immutable advisory statement snapshot captured at a specific asOf time. +/// +public sealed record AdvisoryStatementSnapshot( + Guid StatementId, + string VulnerabilityKey, + string AdvisoryKey, + Advisory Advisory, + ImmutableArray StatementHash, + DateTimeOffset AsOf, + DateTimeOffset RecordedAt, + ImmutableArray InputDocumentIds); + +/// +/// Immutable advisory conflict snapshot describing divergence explanations for a vulnerability key. +/// +public sealed record AdvisoryConflictSnapshot( + Guid ConflictId, + string VulnerabilityKey, + ImmutableArray StatementIds, + ImmutableArray ConflictHash, + DateTimeOffset AsOf, + DateTimeOffset RecordedAt, + string CanonicalJson); + +/// +/// Persistence-facing representation of an advisory statement used by repositories. +/// +public sealed record AdvisoryStatementEntry( + Guid StatementId, + string VulnerabilityKey, + string AdvisoryKey, + string CanonicalJson, + ImmutableArray StatementHash, + DateTimeOffset AsOf, + DateTimeOffset RecordedAt, + ImmutableArray InputDocumentIds); + +/// +/// Persistence-facing representation of an advisory conflict used by repositories. +/// +public sealed record AdvisoryConflictEntry( + Guid ConflictId, + string VulnerabilityKey, + string CanonicalJson, + ImmutableArray ConflictHash, + DateTimeOffset AsOf, + DateTimeOffset RecordedAt, + ImmutableArray StatementIds); diff --git a/src/StellaOps.Concelier.Core/Events/AdvisoryEventLog.cs b/src/StellaOps.Concelier.Core/Events/AdvisoryEventLog.cs new file mode 100644 index 00000000..99422be5 --- /dev/null +++ b/src/StellaOps.Concelier.Core/Events/AdvisoryEventLog.cs @@ -0,0 +1,297 @@ +using System; +using System.Collections.Generic; +using System.Collections.Immutable; +using System.IO; +using System.Linq; +using System.Security.Cryptography; +using System.Text; +using System.Text.Encodings.Web; +using System.Text.Json; +using System.Threading; +using System.Threading.Tasks; +using StellaOps.Concelier.Models; + +namespace StellaOps.Concelier.Core.Events; + +/// +/// Default implementation of that coordinates statement/conflict persistence. +/// +public sealed class AdvisoryEventLog : IAdvisoryEventLog +{ + private static readonly JsonWriterOptions CanonicalWriterOptions = new() + { + Encoder = JavaScriptEncoder.UnsafeRelaxedJsonEscaping, + Indented = false, + SkipValidation = false, + }; + + private readonly IAdvisoryEventRepository _repository; + private readonly TimeProvider _timeProvider; + + public AdvisoryEventLog(IAdvisoryEventRepository repository, TimeProvider? timeProvider = null) + { + _repository = repository ?? throw new ArgumentNullException(nameof(repository)); + _timeProvider = timeProvider ?? TimeProvider.System; + } + + public async ValueTask AppendAsync(AdvisoryEventAppendRequest request, CancellationToken cancellationToken) + { + ArgumentNullException.ThrowIfNull(request); + + var statements = request.Statements ?? Array.Empty(); + var conflicts = request.Conflicts ?? Array.Empty(); + + if (statements.Count == 0 && conflicts.Count == 0) + { + return; + } + + var recordedAt = _timeProvider.GetUtcNow(); + var statementEntries = BuildStatementEntries(statements, recordedAt); + var conflictEntries = BuildConflictEntries(conflicts, recordedAt); + + if (statementEntries.Count > 0) + { + await _repository.InsertStatementsAsync(statementEntries, cancellationToken).ConfigureAwait(false); + } + + if (conflictEntries.Count > 0) + { + await _repository.InsertConflictsAsync(conflictEntries, cancellationToken).ConfigureAwait(false); + } + } + + public async ValueTask ReplayAsync(string vulnerabilityKey, DateTimeOffset? asOf, CancellationToken cancellationToken) + { + if (string.IsNullOrWhiteSpace(vulnerabilityKey)) + { + throw new ArgumentException("Vulnerability key must be provided.", nameof(vulnerabilityKey)); + } + + var normalizedKey = NormalizeKey(vulnerabilityKey, nameof(vulnerabilityKey)); + var statements = await _repository.GetStatementsAsync(normalizedKey, asOf, cancellationToken).ConfigureAwait(false); + var conflicts = await _repository.GetConflictsAsync(normalizedKey, asOf, cancellationToken).ConfigureAwait(false); + + var statementSnapshots = statements + .OrderByDescending(static entry => entry.AsOf) + .ThenByDescending(static entry => entry.RecordedAt) + .Select(ToStatementSnapshot) + .ToImmutableArray(); + + var conflictSnapshots = conflicts + .OrderByDescending(static entry => entry.AsOf) + .ThenByDescending(static entry => entry.RecordedAt) + .Select(ToConflictSnapshot) + .ToImmutableArray(); + + return new AdvisoryReplay(normalizedKey, asOf, statementSnapshots, conflictSnapshots); + } + + private static AdvisoryStatementSnapshot ToStatementSnapshot(AdvisoryStatementEntry entry) + { + ArgumentNullException.ThrowIfNull(entry); + + var advisory = CanonicalJsonSerializer.Deserialize(entry.CanonicalJson); + return new AdvisoryStatementSnapshot( + entry.StatementId, + entry.VulnerabilityKey, + entry.AdvisoryKey, + advisory, + entry.StatementHash, + entry.AsOf, + entry.RecordedAt, + entry.InputDocumentIds); + } + + private static AdvisoryConflictSnapshot ToConflictSnapshot(AdvisoryConflictEntry entry) + { + ArgumentNullException.ThrowIfNull(entry); + + return new AdvisoryConflictSnapshot( + entry.ConflictId, + entry.VulnerabilityKey, + entry.StatementIds, + entry.ConflictHash, + entry.AsOf, + entry.RecordedAt, + entry.CanonicalJson); + } + + private static IReadOnlyCollection BuildStatementEntries( + IReadOnlyCollection statements, + DateTimeOffset recordedAt) + { + if (statements.Count == 0) + { + return Array.Empty(); + } + + var entries = new List(statements.Count); + + foreach (var statement in statements) + { + ArgumentNullException.ThrowIfNull(statement); + ArgumentNullException.ThrowIfNull(statement.Advisory); + + var vulnerabilityKey = NormalizeKey(statement.VulnerabilityKey, nameof(statement.VulnerabilityKey)); + var advisory = CanonicalJsonSerializer.Normalize(statement.Advisory); + var advisoryKey = string.IsNullOrWhiteSpace(statement.AdvisoryKey) + ? advisory.AdvisoryKey + : statement.AdvisoryKey.Trim(); + + if (string.IsNullOrWhiteSpace(advisoryKey)) + { + throw new ArgumentException("Advisory key must be provided.", nameof(statement)); + } + + if (!string.Equals(advisory.AdvisoryKey, advisoryKey, StringComparison.Ordinal)) + { + throw new ArgumentException("Advisory key in payload must match provided advisory key.", nameof(statement)); + } + + var canonicalJson = CanonicalJsonSerializer.Serialize(advisory); + var hashBytes = ComputeHash(canonicalJson); + var asOf = statement.AsOf.ToUniversalTime(); + var inputDocuments = statement.InputDocumentIds?.Count > 0 + ? statement.InputDocumentIds + .Where(static id => id != Guid.Empty) + .Distinct() + .OrderBy(static id => id) + .ToImmutableArray() + : ImmutableArray.Empty; + + entries.Add(new AdvisoryStatementEntry( + statement.StatementId ?? Guid.NewGuid(), + vulnerabilityKey, + advisoryKey, + canonicalJson, + hashBytes, + asOf, + recordedAt, + inputDocuments)); + } + + return entries; + } + + private static IReadOnlyCollection BuildConflictEntries( + IReadOnlyCollection conflicts, + DateTimeOffset recordedAt) + { + if (conflicts.Count == 0) + { + return Array.Empty(); + } + + var entries = new List(conflicts.Count); + + foreach (var conflict in conflicts) + { + ArgumentNullException.ThrowIfNull(conflict); + ArgumentNullException.ThrowIfNull(conflict.Details); + + var vulnerabilityKey = NormalizeKey(conflict.VulnerabilityKey, nameof(conflict.VulnerabilityKey)); + var canonicalJson = Canonicalize(conflict.Details.RootElement); + var hashBytes = ComputeHash(canonicalJson); + var asOf = conflict.AsOf.ToUniversalTime(); + var statementIds = conflict.StatementIds?.Count > 0 + ? conflict.StatementIds + .Where(static id => id != Guid.Empty) + .Distinct() + .OrderBy(static id => id) + .ToImmutableArray() + : ImmutableArray.Empty; + + entries.Add(new AdvisoryConflictEntry( + conflict.ConflictId ?? Guid.NewGuid(), + vulnerabilityKey, + canonicalJson, + hashBytes, + asOf, + recordedAt, + statementIds)); + } + + return entries; + } + + private static string NormalizeKey(string value, string parameterName) + { + if (string.IsNullOrWhiteSpace(value)) + { + throw new ArgumentException("Value must be provided.", parameterName); + } + + return value.Trim().ToLowerInvariant(); + } + + private static ImmutableArray ComputeHash(string canonicalJson) + { + var bytes = Encoding.UTF8.GetBytes(canonicalJson); + var hash = SHA256.HashData(bytes); + return ImmutableArray.Create(hash); + } + + private static string Canonicalize(JsonElement element) + { + using var stream = new MemoryStream(); + using (var writer = new Utf8JsonWriter(stream, CanonicalWriterOptions)) + { + WriteCanonical(element, writer); + } + + return Encoding.UTF8.GetString(stream.ToArray()); + } + + private static void WriteCanonical(JsonElement element, Utf8JsonWriter writer) + { + switch (element.ValueKind) + { + case JsonValueKind.Object: + writer.WriteStartObject(); + foreach (var property in element.EnumerateObject().OrderBy(static p => p.Name, StringComparer.Ordinal)) + { + writer.WritePropertyName(property.Name); + WriteCanonical(property.Value, writer); + } + + writer.WriteEndObject(); + break; + + case JsonValueKind.Array: + writer.WriteStartArray(); + foreach (var item in element.EnumerateArray()) + { + WriteCanonical(item, writer); + } + + writer.WriteEndArray(); + break; + + case JsonValueKind.String: + writer.WriteStringValue(element.GetString()); + break; + + case JsonValueKind.Number: + writer.WriteRawValue(element.GetRawText()); + break; + + case JsonValueKind.True: + writer.WriteBooleanValue(true); + break; + + case JsonValueKind.False: + writer.WriteBooleanValue(false); + break; + + case JsonValueKind.Null: + writer.WriteNullValue(); + break; + + case JsonValueKind.Undefined: + default: + writer.WriteNullValue(); + break; + } + } +} diff --git a/src/StellaOps.Concelier.Core/Events/IAdvisoryEventLog.cs b/src/StellaOps.Concelier.Core/Events/IAdvisoryEventLog.cs new file mode 100644 index 00000000..29d999c7 --- /dev/null +++ b/src/StellaOps.Concelier.Core/Events/IAdvisoryEventLog.cs @@ -0,0 +1,15 @@ +using System; +using System.Threading; +using System.Threading.Tasks; + +namespace StellaOps.Concelier.Core.Events; + +/// +/// High-level API for recording and replaying advisory statements with deterministic as-of queries. +/// +public interface IAdvisoryEventLog +{ + ValueTask AppendAsync(AdvisoryEventAppendRequest request, CancellationToken cancellationToken); + + ValueTask ReplayAsync(string vulnerabilityKey, DateTimeOffset? asOf, CancellationToken cancellationToken); +} diff --git a/src/StellaOps.Concelier.Core/Events/IAdvisoryEventRepository.cs b/src/StellaOps.Concelier.Core/Events/IAdvisoryEventRepository.cs new file mode 100644 index 00000000..975fbaf4 --- /dev/null +++ b/src/StellaOps.Concelier.Core/Events/IAdvisoryEventRepository.cs @@ -0,0 +1,31 @@ +using System; +using System.Collections.Generic; +using System.Collections.Immutable; +using System.Threading; +using System.Threading.Tasks; + +namespace StellaOps.Concelier.Core.Events; + +/// +/// Abstraction over the persistence layer for advisory statements and conflicts. +/// +public interface IAdvisoryEventRepository +{ + ValueTask InsertStatementsAsync( + IReadOnlyCollection statements, + CancellationToken cancellationToken); + + ValueTask InsertConflictsAsync( + IReadOnlyCollection conflicts, + CancellationToken cancellationToken); + + ValueTask> GetStatementsAsync( + string vulnerabilityKey, + DateTimeOffset? asOf, + CancellationToken cancellationToken); + + ValueTask> GetConflictsAsync( + string vulnerabilityKey, + DateTimeOffset? asOf, + CancellationToken cancellationToken); +} diff --git a/src/StellaOps.Concelier.Core/Jobs/JobPluginRegistrationExtensions.cs b/src/StellaOps.Concelier.Core/Jobs/JobPluginRegistrationExtensions.cs index c4e623d6..0197ecce 100644 --- a/src/StellaOps.Concelier.Core/Jobs/JobPluginRegistrationExtensions.cs +++ b/src/StellaOps.Concelier.Core/Jobs/JobPluginRegistrationExtensions.cs @@ -3,12 +3,13 @@ using System.Collections.Generic; using System.Linq; using System.Reflection; using Microsoft.Extensions.Configuration; -using Microsoft.Extensions.DependencyInjection; -using Microsoft.Extensions.Logging; -using StellaOps.DependencyInjection; -using StellaOps.Plugin.Hosting; - -namespace StellaOps.Concelier.Core.Jobs; +using Microsoft.Extensions.DependencyInjection; +using Microsoft.Extensions.Logging; +using StellaOps.DependencyInjection; +using StellaOps.Plugin.DependencyInjection; +using StellaOps.Plugin.Hosting; + +namespace StellaOps.Concelier.Core.Jobs; public static class JobPluginRegistrationExtensions { @@ -32,12 +33,14 @@ public static class JobPluginRegistrationExtensions var currentServices = services; var seenRoutineTypes = new HashSet(StringComparer.Ordinal); - foreach (var plugin in loadResult.Plugins) - { - foreach (var routineType in GetRoutineTypes(plugin.Assembly)) - { - if (!typeof(IDependencyInjectionRoutine).IsAssignableFrom(routineType)) - { + foreach (var plugin in loadResult.Plugins) + { + PluginServiceRegistration.RegisterAssemblyMetadata(currentServices, plugin.Assembly, logger); + + foreach (var routineType in GetRoutineTypes(plugin.Assembly)) + { + if (!typeof(IDependencyInjectionRoutine).IsAssignableFrom(routineType)) + { continue; } diff --git a/src/StellaOps.Concelier.Core/TASKS.md b/src/StellaOps.Concelier.Core/TASKS.md index 8c5ce549..ef2be75d 100644 --- a/src/StellaOps.Concelier.Core/TASKS.md +++ b/src/StellaOps.Concelier.Core/TASKS.md @@ -1,21 +1,21 @@ -# TASKS -| Task | Owner(s) | Depends on | Notes | -|---|---|---|---| -|JobCoordinator implementation (create/get/mark status)|BE-Core|Storage.Mongo|DONE – `JobCoordinator` drives Mongo-backed runs.| -|Cron scheduling loop with TimeProvider|BE-Core|Core|DONE – `JobSchedulerHostedService` evaluates cron expressions.| -|Single-flight/lease semantics|BE-Core|Storage.Mongo|DONE – lease acquisition backed by `MongoLeaseStore`.| -|Trigger API contract (Result mapping)|BE-Core|WebService|DONE – `JobTriggerResult` outcomes map to HTTP statuses.| -|Run telemetry enrichment|BE-Core|Observability|DONE – `JobDiagnostics` ties activities & counters into coordinator/scheduler paths.| -|Deterministic params hashing|BE-Core|Core|DONE – `JobParametersHasher` creates SHA256 hash.| -|Golden tests for timeout/cancel|QA|Core|DONE – JobCoordinatorTests cover cancellation timeout path.| -|JobSchedulerBuilder options registry coverage|BE-Core|Core|DONE – added scheduler tests confirming cron/timeout/lease metadata persists via JobSchedulerOptions.| -|Plugin discovery + DI glue with PluginHost|BE-Core|Plugin libs|DONE – JobPluginRegistrationExtensions now loads PluginHost routines and wires connector/exporter registrations.| -|Harden lease release error handling in JobCoordinator|BE-Core|Storage.Mongo|DONE – lease release failures now logged, wrapped, and drive run failure status; fire-and-forget execution guarded. Verified with `dotnet test --no-build --filter JobCoordinator`.| +# TASKS +| Task | Owner(s) | Depends on | Notes | +|---|---|---|---| +|JobCoordinator implementation (create/get/mark status)|BE-Core|Storage.Mongo|DONE – `JobCoordinator` drives Mongo-backed runs.| +|Cron scheduling loop with TimeProvider|BE-Core|Core|DONE – `JobSchedulerHostedService` evaluates cron expressions.| +|Single-flight/lease semantics|BE-Core|Storage.Mongo|DONE – lease acquisition backed by `MongoLeaseStore`.| +|Trigger API contract (Result mapping)|BE-Core|WebService|DONE – `JobTriggerResult` outcomes map to HTTP statuses.| +|Run telemetry enrichment|BE-Core|Observability|DONE – `JobDiagnostics` ties activities & counters into coordinator/scheduler paths.| +|Deterministic params hashing|BE-Core|Core|DONE – `JobParametersHasher` creates SHA256 hash.| +|Golden tests for timeout/cancel|QA|Core|DONE – JobCoordinatorTests cover cancellation timeout path.| +|JobSchedulerBuilder options registry coverage|BE-Core|Core|DONE – added scheduler tests confirming cron/timeout/lease metadata persists via JobSchedulerOptions.| +|Plugin discovery + DI glue with PluginHost|BE-Core|Plugin libs|DONE – JobPluginRegistrationExtensions now loads PluginHost routines and wires connector/exporter registrations.| +|Harden lease release error handling in JobCoordinator|BE-Core|Storage.Mongo|DONE – lease release failures now logged, wrapped, and drive run failure status; fire-and-forget execution guarded. Verified with `dotnet test --no-build --filter JobCoordinator`.| |Validate job trigger parameters for serialization|BE-Core|WebService|DONE – trigger parameters normalized/serialized with defensive checks returning InvalidParameters on failure. Full-suite `dotnet test --no-build` currently red from live connector fixture drift (Oracle/JVN/RedHat).| |FEEDCORE-ENGINE-03-001 Canonical merger implementation|BE-Core|Merge|DONE – `CanonicalMerger` applies GHSA/NVD/OSV conflict rules with deterministic provenance and comprehensive unit coverage. **Coordination:** Connector leads must align mapper outputs with the canonical field expectations before 2025-10-18 so Merge can activate the path globally.| |FEEDCORE-ENGINE-03-002 Field precedence and tie-breaker map|BE-Core|Merge|DONE – field precedence and freshness overrides enforced via `FieldPrecedence` map with tie-breakers and analytics capture. **Reminder:** Storage/Merge owners review precedence overrides when onboarding new feeds to ensure `decisionReason` tagging stays consistent.| |Canonical merger parity for description/CWE/canonical metric|BE-Core|Models|DONE (2025-10-15) – merger now populates description/CWEs/canonical metric id with provenance and regression tests cover the new decisions.| |Reference normalization & freshness instrumentation cleanup|BE-Core, QA|Models|DONE (2025-10-15) – reference keys normalized, freshness overrides applied to union fields, and new tests assert decision logging.| -|FEEDCORE-ENGINE-07-001 – Advisory event log & asOf queries|Team Core Engine & Storage Analytics|FEEDSTORAGE-DATA-07-001|TODO – Introduce immutable advisory statement events, expose `asOf` query surface for merge/export pipelines, and document determinism guarantees for replay.| +|FEEDCORE-ENGINE-07-001 – Advisory event log & asOf queries|Team Core Engine & Storage Analytics|FEEDSTORAGE-DATA-07-001|**DONE (2025-10-19)** – Implemented `AdvisoryEventLog` service plus repository contracts, canonical hashing, and lower-cased key normalization with replay support; documented determinism guarantees. Tests: `dotnet test src/StellaOps.Concelier.Core.Tests/StellaOps.Concelier.Core.Tests.csproj`.| |FEEDCORE-ENGINE-07-002 – Noise prior computation service|Team Core Engine & Data Science|FEEDCORE-ENGINE-07-001|TODO – Build rule-based learner capturing false-positive priors per package/env, persist summaries, and expose APIs for Excititor/scan suppressors with reproducible statistics.| |FEEDCORE-ENGINE-07-003 – Unknown state ledger & confidence seeding|Team Core Engine & Storage Analytics|FEEDCORE-ENGINE-07-001|TODO – Persist `unknown_vuln_range/unknown_origin/ambiguous_fix` markers with initial confidence bands, expose query surface for Policy, and add fixtures validating canonical serialization.| diff --git a/src/StellaOps.Concelier.Exporter.Json.Tests/JsonExportSnapshotBuilderTests.cs b/src/StellaOps.Concelier.Exporter.Json.Tests/JsonExportSnapshotBuilderTests.cs index 98122474..108a4809 100644 --- a/src/StellaOps.Concelier.Exporter.Json.Tests/JsonExportSnapshotBuilderTests.cs +++ b/src/StellaOps.Concelier.Exporter.Json.Tests/JsonExportSnapshotBuilderTests.cs @@ -42,10 +42,14 @@ public sealed class JsonExportSnapshotBuilderTests : IDisposable severity: "medium"), }; - var result = await builder.WriteAsync(advisories, exportedAt, cancellationToken: CancellationToken.None); - - Assert.Equal(advisories.Length, result.AdvisoryCount); - Assert.Equal(exportedAt, result.ExportedAt); + var result = await builder.WriteAsync(advisories, exportedAt, cancellationToken: CancellationToken.None); + + Assert.Equal(advisories.Length, result.AdvisoryCount); + Assert.Equal(advisories.Length, result.Advisories.Length); + Assert.Equal( + advisories.Select(a => a.AdvisoryKey).OrderBy(key => key, StringComparer.Ordinal), + result.Advisories.Select(a => a.AdvisoryKey).OrderBy(key => key, StringComparer.Ordinal)); + Assert.Equal(exportedAt, result.ExportedAt); var expectedFiles = result.FilePaths.OrderBy(x => x, StringComparer.Ordinal).ToArray(); Assert.Contains("nvd/2024/CVE-2024-9999.json", expectedFiles); @@ -107,10 +111,11 @@ public sealed class JsonExportSnapshotBuilderTests : IDisposable }; var sequence = new SingleEnumerationAsyncSequence(advisories); - var result = await builder.WriteAsync(sequence, exportedAt, cancellationToken: CancellationToken.None); - - Assert.Equal(advisories.Length, result.AdvisoryCount); - } + var result = await builder.WriteAsync(sequence, exportedAt, cancellationToken: CancellationToken.None); + + Assert.Equal(advisories.Length, result.AdvisoryCount); + Assert.Equal(advisories.Length, result.Advisories.Length); + } private static Advisory CreateAdvisory(string advisoryKey, string[] aliases, string title, string severity) { diff --git a/src/StellaOps.Concelier.Exporter.Json.Tests/JsonExporterDependencyInjectionRoutineTests.cs b/src/StellaOps.Concelier.Exporter.Json.Tests/JsonExporterDependencyInjectionRoutineTests.cs index fc80b5bf..05a9175a 100644 --- a/src/StellaOps.Concelier.Exporter.Json.Tests/JsonExporterDependencyInjectionRoutineTests.cs +++ b/src/StellaOps.Concelier.Exporter.Json.Tests/JsonExporterDependencyInjectionRoutineTests.cs @@ -1,12 +1,14 @@ using System.Collections.Generic; using System.Runtime.CompilerServices; using System.Threading.Tasks; +using System.Collections.Immutable; using Microsoft.Extensions.Configuration; using Microsoft.Extensions.DependencyInjection; using Microsoft.Extensions.Logging; using Microsoft.Extensions.Options; using MongoDB.Driver; using StellaOps.Concelier.Core.Jobs; +using StellaOps.Concelier.Core.Events; using StellaOps.Concelier.Exporter.Json; using StellaOps.Concelier.Storage.Mongo.Advisories; using StellaOps.Concelier.Storage.Mongo.Exporting; @@ -20,10 +22,11 @@ public sealed class JsonExporterDependencyInjectionRoutineTests public void Register_AddsJobDefinitionAndServices() { var services = new ServiceCollection(); - services.AddLogging(); - services.AddSingleton(); - services.AddSingleton(); - services.AddOptions(); + services.AddLogging(); + services.AddSingleton(); + services.AddSingleton(); + services.AddSingleton(); + services.AddOptions(); var configuration = new ConfigurationBuilder() .AddInMemoryCollection(new Dictionary()) @@ -78,17 +81,34 @@ public sealed class JsonExporterDependencyInjectionRoutineTests } } - private sealed class StubExportStateStore : IExportStateStore - { - private ExportStateRecord? _record; + private sealed class StubExportStateStore : IExportStateStore + { + private ExportStateRecord? _record; - public Task FindAsync(string id, CancellationToken cancellationToken) - => Task.FromResult(_record); - - public Task UpsertAsync(ExportStateRecord record, CancellationToken cancellationToken) - { - _record = record; - return Task.FromResult(record); - } - } -} + public Task FindAsync(string id, CancellationToken cancellationToken) + { + return Task.FromResult(_record); + } + + public Task UpsertAsync(ExportStateRecord record, CancellationToken cancellationToken) + { + _record = record; + return Task.FromResult(record); + } + } + + private sealed class StubAdvisoryEventLog : IAdvisoryEventLog + { + public ValueTask AppendAsync(AdvisoryEventAppendRequest request, CancellationToken cancellationToken) + => throw new NotSupportedException(); + + public ValueTask ReplayAsync(string vulnerabilityKey, DateTimeOffset? asOf, CancellationToken cancellationToken) + { + return ValueTask.FromResult(new AdvisoryReplay( + vulnerabilityKey, + asOf, + ImmutableArray.Empty, + ImmutableArray.Empty)); + } + } +} diff --git a/src/StellaOps.Concelier.Exporter.Json.Tests/JsonFeedExporterTests.cs b/src/StellaOps.Concelier.Exporter.Json.Tests/JsonFeedExporterTests.cs index 836405a8..82ea9ce5 100644 --- a/src/StellaOps.Concelier.Exporter.Json.Tests/JsonFeedExporterTests.cs +++ b/src/StellaOps.Concelier.Exporter.Json.Tests/JsonFeedExporterTests.cs @@ -1,94 +1,101 @@ -using System; -using System.Collections.Generic; -using System.Globalization; -using System.IO; -using System.Linq; -using System.Runtime.CompilerServices; -using System.Text.Json; -using System.Threading; -using System.Threading.Tasks; -using Microsoft.Extensions.DependencyInjection; -using Microsoft.Extensions.Logging.Abstractions; +using System; +using System.Collections.Generic; +using System.Collections.Immutable; +using System.Globalization; +using System.IO; +using System.Linq; +using System.Runtime.CompilerServices; +using System.Security.Cryptography; +using System.Text; +using System.Text.Json; +using System.Threading; +using System.Threading.Tasks; +using Microsoft.Extensions.DependencyInjection; +using Microsoft.Extensions.Logging.Abstractions; using Microsoft.Extensions.Options; using MongoDB.Driver; +using StellaOps.Concelier.Core.Events; using StellaOps.Concelier.Exporter.Json; using StellaOps.Concelier.Models; using StellaOps.Concelier.Storage.Mongo.Advisories; using StellaOps.Concelier.Storage.Mongo.Exporting; - -namespace StellaOps.Concelier.Exporter.Json.Tests; - -public sealed class JsonFeedExporterTests : IDisposable -{ - private readonly string _root; - - public JsonFeedExporterTests() - { - _root = Directory.CreateTempSubdirectory("concelier-json-exporter-tests").FullName; - } - - [Fact] - public async Task ExportAsync_SkipsWhenDigestUnchanged() - { - var advisory = new Advisory( - advisoryKey: "CVE-2024-1234", - title: "Test Advisory", - summary: null, - language: "en", - published: DateTimeOffset.Parse("2024-01-01T00:00:00Z", CultureInfo.InvariantCulture), - modified: DateTimeOffset.Parse("2024-01-02T00:00:00Z", CultureInfo.InvariantCulture), - severity: "high", - exploitKnown: false, - aliases: new[] { "CVE-2024-1234" }, - references: Array.Empty(), - affectedPackages: Array.Empty(), - cvssMetrics: Array.Empty(), - provenance: Array.Empty()); - - var advisoryStore = new StubAdvisoryStore(advisory); - var options = Options.Create(new JsonExportOptions - { - OutputRoot = _root, - MaintainLatestSymlink = false, - }); - - var stateStore = new InMemoryExportStateStore(); - var timeProvider = new TestTimeProvider(DateTimeOffset.Parse("2024-07-15T12:00:00Z", CultureInfo.InvariantCulture)); - var stateManager = new ExportStateManager(stateStore, timeProvider); - var exporter = new JsonFeedExporter( - advisoryStore, - options, - new VulnListJsonExportPathResolver(), - stateManager, - NullLogger.Instance, - timeProvider); - - using var provider = new ServiceCollection().BuildServiceProvider(); - await exporter.ExportAsync(provider, CancellationToken.None); - - var record = await stateStore.FindAsync(JsonFeedExporter.ExporterId, CancellationToken.None); - Assert.NotNull(record); - var firstUpdated = record!.UpdatedAt; - Assert.Equal("20240715T120000Z", record.BaseExportId); - Assert.Equal(record.LastFullDigest, record.ExportCursor); - - var firstExportPath = Path.Combine(_root, "20240715T120000Z"); - Assert.True(Directory.Exists(firstExportPath)); - - timeProvider.Advance(TimeSpan.FromMinutes(5)); - await exporter.ExportAsync(provider, CancellationToken.None); - - record = await stateStore.FindAsync(JsonFeedExporter.ExporterId, CancellationToken.None); - Assert.NotNull(record); - Assert.Equal(firstUpdated, record!.UpdatedAt); - - var secondExportPath = Path.Combine(_root, "20240715T120500Z"); - Assert.False(Directory.Exists(secondExportPath)); - } - - [Fact] - public async Task ExportAsync_WritesManifestMetadata() - { +using StellaOps.Cryptography; + +namespace StellaOps.Concelier.Exporter.Json.Tests; + +public sealed class JsonFeedExporterTests : IDisposable +{ + private readonly string _root; + + public JsonFeedExporterTests() + { + _root = Directory.CreateTempSubdirectory("concelier-json-exporter-tests").FullName; + } + + [Fact] + public async Task ExportAsync_SkipsWhenDigestUnchanged() + { + var advisory = new Advisory( + advisoryKey: "CVE-2024-1234", + title: "Test Advisory", + summary: null, + language: "en", + published: DateTimeOffset.Parse("2024-01-01T00:00:00Z", CultureInfo.InvariantCulture), + modified: DateTimeOffset.Parse("2024-01-02T00:00:00Z", CultureInfo.InvariantCulture), + severity: "high", + exploitKnown: false, + aliases: new[] { "CVE-2024-1234" }, + references: Array.Empty(), + affectedPackages: Array.Empty(), + cvssMetrics: Array.Empty(), + provenance: Array.Empty()); + + var advisoryStore = new StubAdvisoryStore(advisory); + var options = Options.Create(new JsonExportOptions + { + OutputRoot = _root, + MaintainLatestSymlink = false, + }); + + var stateStore = new InMemoryExportStateStore(); + var timeProvider = new TestTimeProvider(DateTimeOffset.Parse("2024-07-15T12:00:00Z", CultureInfo.InvariantCulture)); + var stateManager = new ExportStateManager(stateStore, timeProvider); + var eventLog = new StubAdvisoryEventLog(new[] { advisory }, timeProvider.GetUtcNow()); + var exporter = new JsonFeedExporter( + advisoryStore, + options, + new VulnListJsonExportPathResolver(), + stateManager, + eventLog, + NullLogger.Instance, + timeProvider); + + using var provider = new ServiceCollection().BuildServiceProvider(); + await exporter.ExportAsync(provider, CancellationToken.None); + + var record = await stateStore.FindAsync(JsonFeedExporter.ExporterId, CancellationToken.None); + Assert.NotNull(record); + var firstUpdated = record!.UpdatedAt; + Assert.Equal("20240715T120000Z", record.BaseExportId); + Assert.Equal(record.LastFullDigest, record.ExportCursor); + + var firstExportPath = Path.Combine(_root, "20240715T120000Z"); + Assert.True(Directory.Exists(firstExportPath)); + + timeProvider.Advance(TimeSpan.FromMinutes(5)); + await exporter.ExportAsync(provider, CancellationToken.None); + + record = await stateStore.FindAsync(JsonFeedExporter.ExporterId, CancellationToken.None); + Assert.NotNull(record); + Assert.Equal(firstUpdated, record!.UpdatedAt); + + var secondExportPath = Path.Combine(_root, "20240715T120500Z"); + Assert.False(Directory.Exists(secondExportPath)); + } + + [Fact] + public async Task ExportAsync_WritesManifestMetadata() + { var exportedAt = DateTimeOffset.Parse("2024-08-10T00:00:00Z", CultureInfo.InvariantCulture); var recordedAt = DateTimeOffset.Parse("2024-07-02T00:00:00Z", CultureInfo.InvariantCulture); var reference = new AdvisoryReference( @@ -135,52 +142,54 @@ public sealed class JsonFeedExporterTests : IDisposable description: "Detailed description capturing remediation steps.", cwes: new[] { weakness }, canonicalMetricId: "3.1|CVSS:3.1/AV:N/AC:L/PR:N/UI:N/S:C/C:H/I:H/A:H"); - - var advisoryStore = new StubAdvisoryStore(advisory); - var optionsValue = new JsonExportOptions - { - OutputRoot = _root, - MaintainLatestSymlink = false, - }; - - var options = Options.Create(optionsValue); - var stateStore = new InMemoryExportStateStore(); - var timeProvider = new TestTimeProvider(exportedAt); - var stateManager = new ExportStateManager(stateStore, timeProvider); - var exporter = new JsonFeedExporter( - advisoryStore, - options, - new VulnListJsonExportPathResolver(), - stateManager, - NullLogger.Instance, - timeProvider); - - using var provider = new ServiceCollection().BuildServiceProvider(); - await exporter.ExportAsync(provider, CancellationToken.None); - - var exportId = exportedAt.ToString(optionsValue.DirectoryNameFormat, CultureInfo.InvariantCulture); - var exportDirectory = Path.Combine(_root, exportId); - var manifestPath = Path.Combine(exportDirectory, "manifest.json"); - - Assert.True(File.Exists(manifestPath)); - - using var document = JsonDocument.Parse(await File.ReadAllBytesAsync(manifestPath, CancellationToken.None)); - var root = document.RootElement; - - Assert.Equal(exportId, root.GetProperty("exportId").GetString()); - Assert.Equal(exportedAt.UtcDateTime, root.GetProperty("generatedAt").GetDateTime()); - Assert.Equal(1, root.GetProperty("advisoryCount").GetInt32()); - - var exportedFiles = Directory.EnumerateFiles(exportDirectory, "*.json", SearchOption.AllDirectories) - .Select(path => new - { - Absolute = path, - Relative = Path.GetRelativePath(exportDirectory, path).Replace("\\", "/", StringComparison.Ordinal), - }) - .Where(file => !string.Equals(file.Relative, "manifest.json", StringComparison.OrdinalIgnoreCase)) - .OrderBy(file => file.Relative, StringComparer.Ordinal) - .ToArray(); - + + var advisoryStore = new StubAdvisoryStore(advisory); + var optionsValue = new JsonExportOptions + { + OutputRoot = _root, + MaintainLatestSymlink = false, + }; + + var options = Options.Create(optionsValue); + var stateStore = new InMemoryExportStateStore(); + var timeProvider = new TestTimeProvider(exportedAt); + var stateManager = new ExportStateManager(stateStore, timeProvider); + var eventLog = new StubAdvisoryEventLog(new[] { advisory }, exportedAt); + var exporter = new JsonFeedExporter( + advisoryStore, + options, + new VulnListJsonExportPathResolver(), + stateManager, + eventLog, + NullLogger.Instance, + timeProvider); + + using var provider = new ServiceCollection().BuildServiceProvider(); + await exporter.ExportAsync(provider, CancellationToken.None); + + var exportId = exportedAt.ToString(optionsValue.DirectoryNameFormat, CultureInfo.InvariantCulture); + var exportDirectory = Path.Combine(_root, exportId); + var manifestPath = Path.Combine(exportDirectory, "manifest.json"); + + Assert.True(File.Exists(manifestPath)); + + using var document = JsonDocument.Parse(await File.ReadAllBytesAsync(manifestPath, CancellationToken.None)); + var root = document.RootElement; + + Assert.Equal(exportId, root.GetProperty("exportId").GetString()); + Assert.Equal(exportedAt.UtcDateTime, root.GetProperty("generatedAt").GetDateTime()); + Assert.Equal(1, root.GetProperty("advisoryCount").GetInt32()); + + var exportedFiles = Directory.EnumerateFiles(exportDirectory, "*.json", SearchOption.AllDirectories) + .Select(path => new + { + Absolute = path, + Relative = Path.GetRelativePath(exportDirectory, path).Replace("\\", "/", StringComparison.Ordinal), + }) + .Where(file => !string.Equals(file.Relative, "manifest.json", StringComparison.OrdinalIgnoreCase)) + .OrderBy(file => file.Relative, StringComparer.Ordinal) + .ToArray(); + var filesElement = root.GetProperty("files") .EnumerateArray() .Select(element => new @@ -208,58 +217,278 @@ public sealed class JsonFeedExporterTests : IDisposable } Assert.Equal(exportedFiles.Select(file => file.Relative).ToArray(), filesElement.Select(file => file.Path).ToArray()); - - long totalBytes = exportedFiles.Select(file => new FileInfo(file.Absolute).Length).Sum(); - Assert.Equal(totalBytes, root.GetProperty("totalBytes").GetInt64()); - Assert.Equal(exportedFiles.Length, root.GetProperty("fileCount").GetInt32()); - - var digest = root.GetProperty("digest").GetString(); - var digestResult = new JsonExportResult( - exportDirectory, - exportedAt, - exportedFiles.Select(file => - { - var manifestEntry = filesElement.First(f => f.Path == file.Relative); - if (manifestEntry.Digest is null) - { - throw new InvalidOperationException($"Manifest entry for {file.Relative} missing digest."); - } - - return new JsonExportFile(file.Relative, new FileInfo(file.Absolute).Length, manifestEntry.Digest); - }), - exportedFiles.Length, - totalBytes); - var expectedDigest = ExportDigestCalculator.ComputeTreeDigest(digestResult); - Assert.Equal(expectedDigest, digest); - - var exporterVersion = root.GetProperty("exporterVersion").GetString(); - Assert.Equal(ExporterVersion.GetVersion(typeof(JsonFeedExporter)), exporterVersion); - } - - public void Dispose() - { - try - { - if (Directory.Exists(_root)) - { - Directory.Delete(_root, recursive: true); - } - } - catch - { - // best effort cleanup - } - } - - private sealed class StubAdvisoryStore : IAdvisoryStore - { - private readonly IReadOnlyList _advisories; - - public StubAdvisoryStore(params Advisory[] advisories) - { - _advisories = advisories; - } - + + long totalBytes = exportedFiles.Select(file => new FileInfo(file.Absolute).Length).Sum(); + Assert.Equal(totalBytes, root.GetProperty("totalBytes").GetInt64()); + Assert.Equal(exportedFiles.Length, root.GetProperty("fileCount").GetInt32()); + + var digest = root.GetProperty("digest").GetString(); + var digestResult = new JsonExportResult( + exportDirectory, + exportedAt, + exportedFiles.Select(file => + { + var manifestEntry = filesElement.First(f => f.Path == file.Relative); + if (manifestEntry.Digest is null) + { + throw new InvalidOperationException($"Manifest entry for {file.Relative} missing digest."); + } + + return new JsonExportFile(file.Relative, new FileInfo(file.Absolute).Length, manifestEntry.Digest); + }), + exportedFiles.Length, + totalBytes); + var expectedDigest = ExportDigestCalculator.ComputeTreeDigest(digestResult); + Assert.Equal(expectedDigest, digest); + + var exporterVersion = root.GetProperty("exporterVersion").GetString(); + Assert.Equal(ExporterVersion.GetVersion(typeof(JsonFeedExporter)), exporterVersion); + } + + [Fact] + public async Task ExportAsync_WritesMirrorBundlesWithSignatures() + { + var exportedAt = DateTimeOffset.Parse("2025-01-05T00:00:00Z", CultureInfo.InvariantCulture); + var advisoryOne = new Advisory( + advisoryKey: "CVE-2025-0001", + title: "Mirror Advisory One", + summary: null, + language: "en", + published: exportedAt.AddDays(-10), + modified: exportedAt.AddDays(-9), + severity: "high", + exploitKnown: false, + aliases: new[] { "CVE-2025-0001", "GHSA-aaaa-bbbb-cccc" }, + references: Array.Empty(), + affectedPackages: Array.Empty(), + cvssMetrics: Array.Empty(), + provenance: new[] + { + new AdvisoryProvenance("ghsa", "map", "GHSA-aaaa-bbbb-cccc", exportedAt.AddDays(-9)), + new AdvisoryProvenance("nvd", "map", "CVE-2025-0001", exportedAt.AddDays(-8)), + }); + + var advisoryTwo = new Advisory( + advisoryKey: "CVE-2025-0002", + title: "Mirror Advisory Two", + summary: null, + language: "en", + published: exportedAt.AddDays(-6), + modified: exportedAt.AddDays(-5), + severity: "medium", + exploitKnown: false, + aliases: new[] { "CVE-2025-0002" }, + references: Array.Empty(), + affectedPackages: Array.Empty(), + cvssMetrics: Array.Empty(), + provenance: new[] + { + new AdvisoryProvenance("nvd", "map", "CVE-2025-0002", exportedAt.AddDays(-5)), + new AdvisoryProvenance("vendor", "map", "ADVISORY-0002", exportedAt.AddDays(-4)), + }); + + var advisoryStore = new StubAdvisoryStore(advisoryOne, advisoryTwo); + var optionsValue = new JsonExportOptions + { + OutputRoot = _root, + MaintainLatestSymlink = false, + TargetRepository = "s3://mirror/concelier" + }; + + optionsValue.Mirror.Enabled = true; + optionsValue.Mirror.DirectoryName = "mirror"; + optionsValue.Mirror.Domains.Add(new JsonExportOptions.JsonMirrorDomainOptions + { + Id = "primary", + DisplayName = "Primary" + }); + + optionsValue.Mirror.Signing.Enabled = true; + optionsValue.Mirror.Signing.KeyId = "mirror-signing-key"; + optionsValue.Mirror.Signing.Algorithm = SignatureAlgorithms.Es256; + optionsValue.Mirror.Signing.KeyPath = WriteSigningKey(_root); + + var options = Options.Create(optionsValue); + var stateStore = new InMemoryExportStateStore(); + var timeProvider = new TestTimeProvider(exportedAt); + var stateManager = new ExportStateManager(stateStore, timeProvider); + var eventLog = new StubAdvisoryEventLog(new[] { advisoryOne, advisoryTwo }, exportedAt); + var exporter = new JsonFeedExporter( + advisoryStore, + options, + new VulnListJsonExportPathResolver(), + stateManager, + eventLog, + NullLogger.Instance, + timeProvider); + + var services = new ServiceCollection(); + services.AddSingleton(); + services.AddSingleton(sp => sp.GetRequiredService()); + services.AddSingleton(sp => + { + var provider = sp.GetRequiredService(); + return new CryptoProviderRegistry(new[] { provider }); + }); + + using var provider = services.BuildServiceProvider(); + await exporter.ExportAsync(provider, CancellationToken.None); + + var exportId = exportedAt.ToString(optionsValue.DirectoryNameFormat, CultureInfo.InvariantCulture); + var exportDirectory = Path.Combine(_root, exportId); + var mirrorDirectory = Path.Combine(exportDirectory, "mirror"); + var domainDirectory = Path.Combine(mirrorDirectory, "primary"); + + Assert.True(File.Exists(Path.Combine(mirrorDirectory, "index.json"))); + Assert.True(File.Exists(Path.Combine(domainDirectory, "bundle.json"))); + Assert.True(File.Exists(Path.Combine(domainDirectory, "bundle.json.jws"))); + Assert.True(File.Exists(Path.Combine(domainDirectory, "manifest.json"))); + + var record = await stateStore.FindAsync(JsonFeedExporter.ExporterId, CancellationToken.None); + Assert.NotNull(record); + Assert.Contains(record!.Files, file => string.Equals(file.Path, "mirror/index.json", StringComparison.Ordinal)); + Assert.Contains(record.Files, file => string.Equals(file.Path, "mirror/primary/manifest.json", StringComparison.Ordinal)); + + var indexPath = Path.Combine(mirrorDirectory, "index.json"); + using (var indexDoc = JsonDocument.Parse(await File.ReadAllBytesAsync(indexPath, CancellationToken.None))) + { + var indexRoot = indexDoc.RootElement; + Assert.Equal("s3://mirror/concelier", indexRoot.GetProperty("targetRepository").GetString()); + + var domains = indexRoot.GetProperty("domains").EnumerateArray().ToArray(); + var domain = Assert.Single(domains); + Assert.Equal("primary", domain.GetProperty("domainId").GetString()); + Assert.Equal("Primary", domain.GetProperty("displayName").GetString()); + Assert.Equal(2, domain.GetProperty("advisoryCount").GetInt32()); + + var bundleDescriptor = domain.GetProperty("bundle"); + Assert.Equal("mirror/primary/bundle.json", bundleDescriptor.GetProperty("path").GetString()); + var signatureDescriptor = bundleDescriptor.GetProperty("signature"); + Assert.Equal("mirror/primary/bundle.json.jws", signatureDescriptor.GetProperty("path").GetString()); + + var manifestDescriptor = domain.GetProperty("manifest"); + Assert.Equal("mirror/primary/manifest.json", manifestDescriptor.GetProperty("path").GetString()); + } + + var bundlePathRel = "mirror/primary/bundle.json"; + var manifestPathRel = "mirror/primary/manifest.json"; + var signaturePathRel = "mirror/primary/bundle.json.jws"; + + var bundlePath = Path.Combine(exportDirectory, bundlePathRel.Replace('/', Path.DirectorySeparatorChar)); + var manifestPath = Path.Combine(exportDirectory, manifestPathRel.Replace('/', Path.DirectorySeparatorChar)); + var signaturePath = Path.Combine(exportDirectory, signaturePathRel.Replace('/', Path.DirectorySeparatorChar)); + + using (var bundleDoc = JsonDocument.Parse(await File.ReadAllBytesAsync(bundlePath, CancellationToken.None))) + { + var bundleRoot = bundleDoc.RootElement; + Assert.Equal("primary", bundleRoot.GetProperty("domainId").GetString()); + Assert.Equal(2, bundleRoot.GetProperty("advisoryCount").GetInt32()); + Assert.Equal("s3://mirror/concelier", bundleRoot.GetProperty("targetRepository").GetString()); + Assert.Equal(2, bundleRoot.GetProperty("advisories").GetArrayLength()); + + var sources = bundleRoot.GetProperty("sources").EnumerateArray().Select(element => element.GetProperty("source").GetString()).ToArray(); + Assert.Contains("ghsa", sources); + Assert.Contains("nvd", sources); + Assert.Contains("vendor", sources); + } + + using (var manifestDoc = JsonDocument.Parse(await File.ReadAllBytesAsync(manifestPath, CancellationToken.None))) + { + var manifestRoot = manifestDoc.RootElement; + Assert.Equal("primary", manifestRoot.GetProperty("domainId").GetString()); + Assert.Equal(2, manifestRoot.GetProperty("advisoryCount").GetInt32()); + Assert.Equal("mirror/primary/bundle.json", manifestRoot.GetProperty("bundle").GetProperty("path").GetString()); + } + + var bundleBytes = await File.ReadAllBytesAsync(bundlePath, CancellationToken.None); + var signatureValue = await File.ReadAllTextAsync(signaturePath, CancellationToken.None); + var signatureParts = signatureValue.Split("..", StringSplitOptions.None); + Assert.Equal(2, signatureParts.Length); + + var signingInput = BuildSigningInput(signatureParts[0], bundleBytes); + var signatureBytes = Base64UrlDecode(signatureParts[1]); + + var registry = provider.GetRequiredService(); + var verification = registry.ResolveSigner( + CryptoCapability.Signing, + optionsValue.Mirror.Signing.Algorithm, + new CryptoKeyReference(optionsValue.Mirror.Signing.KeyId, optionsValue.Mirror.Signing.Provider), + optionsValue.Mirror.Signing.Provider); + var verified = await verification.Signer.VerifyAsync(signingInput, signatureBytes, CancellationToken.None); + Assert.True(verified); + } + + public void Dispose() + { + try + { + if (Directory.Exists(_root)) + { + Directory.Delete(_root, recursive: true); + } + } + catch + { + // best effort cleanup + } + } + + private static string WriteSigningKey(string directory) + { + using var ecdsa = ECDsa.Create(ECCurve.NamedCurves.nistP256); + var pkcs8 = ecdsa.ExportPkcs8PrivateKey(); + var pem = BuildPem("PRIVATE KEY", pkcs8); + var path = Path.Combine(directory, $"mirror-key-{Guid.NewGuid():N}.pem"); + File.WriteAllText(path, pem); + return path; + } + + private static string BuildPem(string label, byte[] data) + { + var base64 = Convert.ToBase64String(data, Base64FormattingOptions.InsertLineBreaks); + return $"-----BEGIN {label}-----\n{base64}\n-----END {label}-----\n"; + } + + private static byte[] BuildSigningInput(string protectedHeader, byte[] payload) + { + var headerBytes = Encoding.ASCII.GetBytes(protectedHeader); + var buffer = new byte[headerBytes.Length + 1 + payload.Length]; + Buffer.BlockCopy(headerBytes, 0, buffer, 0, headerBytes.Length); + buffer[headerBytes.Length] = (byte)'.'; + Buffer.BlockCopy(payload, 0, buffer, headerBytes.Length + 1, payload.Length); + return buffer; + } + + private static byte[] Base64UrlDecode(string value) + { + var builder = new StringBuilder(value.Length + 3); + foreach (var ch in value) + { + builder.Append(ch switch + { + '-' => '+', + '_' => '/', + _ => ch + }); + } + + while (builder.Length % 4 != 0) + { + builder.Append('='); + } + + return Convert.FromBase64String(builder.ToString()); + } + + private sealed class StubAdvisoryStore : IAdvisoryStore + { + private readonly IReadOnlyList _advisories; + + public StubAdvisoryStore(params Advisory[] advisories) + { + _advisories = advisories; + } + public Task> GetRecentAsync(int limit, CancellationToken cancellationToken, IClientSessionHandle? session = null) { _ = session; @@ -285,38 +514,84 @@ public sealed class JsonFeedExporterTests : IDisposable async IAsyncEnumerable EnumerateAsync([EnumeratorCancellation] CancellationToken ct) { - foreach (var advisory in _advisories) - { - ct.ThrowIfCancellationRequested(); - yield return advisory; - await Task.Yield(); - } - } - } - } - - private sealed class InMemoryExportStateStore : IExportStateStore - { - private ExportStateRecord? _record; - - public Task FindAsync(string id, CancellationToken cancellationToken) - => Task.FromResult(_record); - - public Task UpsertAsync(ExportStateRecord record, CancellationToken cancellationToken) - { - _record = record; - return Task.FromResult(record); - } - } - - private sealed class TestTimeProvider : TimeProvider - { - private DateTimeOffset _now; - - public TestTimeProvider(DateTimeOffset start) => _now = start; - - public override DateTimeOffset GetUtcNow() => _now; - - public void Advance(TimeSpan delta) => _now = _now.Add(delta); - } -} + foreach (var advisory in _advisories) + { + ct.ThrowIfCancellationRequested(); + yield return advisory; + await Task.Yield(); + } + } + } + } + + private sealed class StubAdvisoryEventLog : IAdvisoryEventLog + { + private readonly Dictionary _advisories; + private readonly DateTimeOffset _recordedAt; + + public StubAdvisoryEventLog(IEnumerable advisories, DateTimeOffset recordedAt) + { + _advisories = advisories.ToDictionary(advisory => advisory.AdvisoryKey, StringComparer.OrdinalIgnoreCase); + _recordedAt = recordedAt; + } + + public ValueTask AppendAsync(AdvisoryEventAppendRequest request, CancellationToken cancellationToken) + => throw new NotSupportedException(); + + public ValueTask ReplayAsync(string vulnerabilityKey, DateTimeOffset? asOf, CancellationToken cancellationToken) + { + if (_advisories.TryGetValue(vulnerabilityKey, out var advisory)) + { + var asOfTimestamp = advisory.Modified ?? advisory.Published ?? _recordedAt; + var snapshot = new AdvisoryStatementSnapshot( + Guid.NewGuid(), + vulnerabilityKey, + advisory.AdvisoryKey, + advisory, + ImmutableArray.Empty, + asOfTimestamp, + _recordedAt, + ImmutableArray.Empty); + + return ValueTask.FromResult(new AdvisoryReplay( + vulnerabilityKey, + asOf, + ImmutableArray.Create(snapshot), + ImmutableArray.Empty)); + } + + return ValueTask.FromResult(new AdvisoryReplay( + vulnerabilityKey, + asOf, + ImmutableArray.Empty, + ImmutableArray.Empty)); + } + } + + private sealed class InMemoryExportStateStore : IExportStateStore + { + private ExportStateRecord? _record; + + public Task FindAsync(string id, CancellationToken cancellationToken) + { + return Task.FromResult(_record); + } + + public Task UpsertAsync(ExportStateRecord record, CancellationToken cancellationToken) + { + _record = record; + return Task.FromResult(record); + } + } + + private sealed class TestTimeProvider : TimeProvider + { + private DateTimeOffset _now; + + public TestTimeProvider(DateTimeOffset start) => _now = start; + + public override DateTimeOffset GetUtcNow() => _now; + + public void Advance(TimeSpan delta) => _now = _now.Add(delta); + } +} diff --git a/src/StellaOps.Concelier.Exporter.Json/JsonExportOptions.cs b/src/StellaOps.Concelier.Exporter.Json/JsonExportOptions.cs index 30fedb02..9d2a39b1 100644 --- a/src/StellaOps.Concelier.Exporter.Json/JsonExportOptions.cs +++ b/src/StellaOps.Concelier.Exporter.Json/JsonExportOptions.cs @@ -1,9 +1,11 @@ -using System.IO; - -namespace StellaOps.Concelier.Exporter.Json; - -/// -/// Configuration for JSON exporter output paths and determinism controls. +using System.Collections.Generic; +using System.IO; +using StellaOps.Cryptography; + +namespace StellaOps.Concelier.Exporter.Json; + +/// +/// Configuration for JSON exporter output paths and determinism controls. /// public sealed class JsonExportOptions { @@ -27,8 +29,87 @@ public sealed class JsonExportOptions /// public bool MaintainLatestSymlink { get; set; } = true; - /// - /// Optional repository identifier recorded alongside export state metadata. - /// - public string? TargetRepository { get; set; } -} + /// + /// Optional repository identifier recorded alongside export state metadata. + /// + public string? TargetRepository { get; set; } + + /// + /// Mirror distribution configuration producing aggregate bundles for downstream mirrors. + /// + public JsonMirrorOptions Mirror { get; set; } = new(); + + public sealed class JsonMirrorOptions + { + /// + /// Indicates whether mirror bundle generation is enabled. + /// + public bool Enabled { get; set; } + + /// + /// Directory name (relative to the export root) where mirror artefacts are written. + /// + public string DirectoryName { get; set; } = "mirror"; + + /// + /// Domains exposed to downstream mirrors. + /// + public IList Domains { get; } = new List(); + + /// + /// Signing configuration for mirror bundles. + /// + public JsonMirrorSigningOptions Signing { get; set; } = new(); + } + + public sealed class JsonMirrorDomainOptions + { + /// + /// Stable identifier for the mirror domain (used in URLs and directory names). + /// + public string Id { get; set; } = string.Empty; + + /// + /// Optional human-readable label for UI surfaces. + /// + public string? DisplayName { get; set; } + + /// + /// Optional advisory scheme filters (e.g. CVE, GHSA). Empty collection selects all schemes. + /// + public IList IncludeSchemes { get; } = new List(); + + /// + /// Optional provenance source filters (e.g. nvd, ghsa). Empty collection selects all sources. + /// + public IList IncludeSources { get; } = new List(); + } + + public sealed class JsonMirrorSigningOptions + { + /// + /// Indicates whether bundles should be signed. Defaults to disabled. + /// + public bool Enabled { get; set; } + + /// + /// Signing algorithm identifier (defaults to ES256). + /// + public string Algorithm { get; set; } = SignatureAlgorithms.Es256; + + /// + /// Active signing key identifier. + /// + public string KeyId { get; set; } = string.Empty; + + /// + /// Path to the private key (PEM) used for signing mirror bundles. + /// + public string KeyPath { get; set; } = string.Empty; + + /// + /// Optional crypto provider hint. When omitted the registry resolves an appropriate provider. + /// + public string? Provider { get; set; } + } +} diff --git a/src/StellaOps.Concelier.Exporter.Json/JsonExportResult.cs b/src/StellaOps.Concelier.Exporter.Json/JsonExportResult.cs index 8c399d18..7504d7bf 100644 --- a/src/StellaOps.Concelier.Exporter.Json/JsonExportResult.cs +++ b/src/StellaOps.Concelier.Exporter.Json/JsonExportResult.cs @@ -1,46 +1,55 @@ -using System; -using System.Collections.Generic; -using System.Collections.Immutable; -using System.Linq; - -namespace StellaOps.Concelier.Exporter.Json; - -public sealed class JsonExportResult -{ - public JsonExportResult( - string exportDirectory, - DateTimeOffset exportedAt, - IEnumerable files, - int advisoryCount, - long totalBytes) - { - if (string.IsNullOrWhiteSpace(exportDirectory)) - { - throw new ArgumentException("Export directory must be provided.", nameof(exportDirectory)); - } - - ExportDirectory = exportDirectory; - ExportedAt = exportedAt; - AdvisoryCount = advisoryCount; - TotalBytes = totalBytes; - - var list = (files ?? throw new ArgumentNullException(nameof(files))) - .Where(static file => file is not null) - .ToImmutableArray(); - - Files = list; - FilePaths = list.Select(static file => file.RelativePath).ToImmutableArray(); - } - - public string ExportDirectory { get; } - - public DateTimeOffset ExportedAt { get; } - - public ImmutableArray Files { get; } - - public ImmutableArray FilePaths { get; } - - public int AdvisoryCount { get; } - - public long TotalBytes { get; } -} +using System; +using System.Collections.Generic; +using System.Collections.Immutable; +using System.Linq; +using StellaOps.Concelier.Models; + +namespace StellaOps.Concelier.Exporter.Json; + +public sealed class JsonExportResult +{ + public JsonExportResult( + string exportDirectory, + DateTimeOffset exportedAt, + IEnumerable files, + int advisoryCount, + long totalBytes, + IEnumerable? advisories = null) + { + if (string.IsNullOrWhiteSpace(exportDirectory)) + { + throw new ArgumentException("Export directory must be provided.", nameof(exportDirectory)); + } + + var list = (files ?? throw new ArgumentNullException(nameof(files))) + .Where(static file => file is not null) + .ToImmutableArray(); + + var advisoryList = (advisories ?? Array.Empty()) + .Where(static advisory => advisory is not null) + .ToImmutableArray(); + + ExportDirectory = exportDirectory; + ExportedAt = exportedAt; + TotalBytes = totalBytes; + + Files = list; + FilePaths = list.Select(static file => file.RelativePath).ToImmutableArray(); + Advisories = advisoryList; + AdvisoryCount = advisoryList.IsDefaultOrEmpty ? advisoryCount : advisoryList.Length; + } + + public string ExportDirectory { get; } + + public DateTimeOffset ExportedAt { get; } + + public ImmutableArray Files { get; } + + public ImmutableArray FilePaths { get; } + + public ImmutableArray Advisories { get; } + + public int AdvisoryCount { get; } + + public long TotalBytes { get; } +} diff --git a/src/StellaOps.Concelier.Exporter.Json/JsonExportSnapshotBuilder.cs b/src/StellaOps.Concelier.Exporter.Json/JsonExportSnapshotBuilder.cs index 092e0831..49c531cf 100644 --- a/src/StellaOps.Concelier.Exporter.Json/JsonExportSnapshotBuilder.cs +++ b/src/StellaOps.Concelier.Exporter.Json/JsonExportSnapshotBuilder.cs @@ -67,26 +67,27 @@ public sealed class JsonExportSnapshotBuilder Directory.CreateDirectory(exportDirectory); TrySetDirectoryTimestamp(exportDirectory, exportedAt); - var seen = new HashSet(StringComparer.OrdinalIgnoreCase); - var files = new List(); - long totalBytes = 0L; - var advisoryCount = 0; - - await foreach (var advisory in advisories.WithCancellation(cancellationToken)) - { - cancellationToken.ThrowIfCancellationRequested(); - - advisoryCount++; - var entry = Resolve(advisory); - if (!seen.Add(entry.RelativePath)) - { - throw new InvalidOperationException($"Multiple advisories resolved to the same path '{entry.RelativePath}'."); - } - - var destination = Combine(exportDirectory, entry.Segments); - var destinationDirectory = Path.GetDirectoryName(destination); - if (!string.IsNullOrEmpty(destinationDirectory)) - { + var seen = new HashSet(StringComparer.OrdinalIgnoreCase); + var files = new List(); + var advisoryList = new List(); + long totalBytes = 0L; + + await foreach (var advisory in advisories.WithCancellation(cancellationToken)) + { + cancellationToken.ThrowIfCancellationRequested(); + + var entry = Resolve(advisory); + if (!seen.Add(entry.RelativePath)) + { + throw new InvalidOperationException($"Multiple advisories resolved to the same path '{entry.RelativePath}'."); + } + + advisoryList.Add(entry.Advisory); + + var destination = Combine(exportDirectory, entry.Segments); + var destinationDirectory = Path.GetDirectoryName(destination); + if (!string.IsNullOrEmpty(destinationDirectory)) + { EnsureDirectoryExists(destinationDirectory); TrySetDirectoryTimestamp(destinationDirectory, exportedAt); } @@ -97,14 +98,14 @@ public sealed class JsonExportSnapshotBuilder File.SetLastWriteTimeUtc(destination, exportedAt.UtcDateTime); var digest = ComputeDigest(bytes); - files.Add(new JsonExportFile(entry.RelativePath, bytes.LongLength, digest)); - totalBytes += bytes.LongLength; - } - - files.Sort(static (left, right) => string.CompareOrdinal(left.RelativePath, right.RelativePath)); - - return new JsonExportResult(exportDirectory, exportedAt, files, advisoryCount, totalBytes); - } + files.Add(new JsonExportFile(entry.RelativePath, bytes.LongLength, digest)); + totalBytes += bytes.LongLength; + } + + files.Sort(static (left, right) => string.CompareOrdinal(left.RelativePath, right.RelativePath)); + + return new JsonExportResult(exportDirectory, exportedAt, files, advisoryList.Count, totalBytes, advisoryList); + } private static async IAsyncEnumerable EnumerateAsync( IEnumerable advisories, @@ -168,10 +169,11 @@ public sealed class JsonExportSnapshotBuilder throw new ArgumentNullException(nameof(advisory)); } - var relativePath = _pathResolver.GetRelativePath(advisory); - var segments = NormalizeRelativePath(relativePath); - var normalized = string.Join('/', segments); - return new PathResolution(advisory, normalized, segments); + var normalized = CanonicalJsonSerializer.Normalize(advisory); + var relativePath = _pathResolver.GetRelativePath(normalized); + var segments = NormalizeRelativePath(relativePath); + var normalizedPath = string.Join('/', segments); + return new PathResolution(normalized, normalizedPath, segments); } private static string[] NormalizeRelativePath(string relativePath) diff --git a/src/StellaOps.Concelier.Exporter.Json/JsonExporterDependencyInjectionRoutine.cs b/src/StellaOps.Concelier.Exporter.Json/JsonExporterDependencyInjectionRoutine.cs index 51739214..e778eaf7 100644 --- a/src/StellaOps.Concelier.Exporter.Json/JsonExporterDependencyInjectionRoutine.cs +++ b/src/StellaOps.Concelier.Exporter.Json/JsonExporterDependencyInjectionRoutine.cs @@ -31,14 +31,19 @@ public sealed class JsonExporterDependencyInjectionRoutine : IDependencyInjectio options.OutputRoot = Path.Combine("exports", "json"); } - if (string.IsNullOrWhiteSpace(options.DirectoryNameFormat)) - { - options.DirectoryNameFormat = "yyyyMMdd'T'HHmmss'Z'"; - } - }); - - services.AddSingleton(); - services.AddTransient(); + if (string.IsNullOrWhiteSpace(options.DirectoryNameFormat)) + { + options.DirectoryNameFormat = "yyyyMMdd'T'HHmmss'Z'"; + } + + if (string.IsNullOrWhiteSpace(options.Mirror.DirectoryName)) + { + options.Mirror.DirectoryName = "mirror"; + } + }); + + services.AddSingleton(); + services.AddTransient(); services.PostConfigure(options => { diff --git a/src/StellaOps.Concelier.Exporter.Json/JsonFeedExporter.cs b/src/StellaOps.Concelier.Exporter.Json/JsonFeedExporter.cs index 65ce31bc..f516e58c 100644 --- a/src/StellaOps.Concelier.Exporter.Json/JsonFeedExporter.cs +++ b/src/StellaOps.Concelier.Exporter.Json/JsonFeedExporter.cs @@ -1,12 +1,16 @@ using System; using System.Globalization; -using System.IO; -using System.Linq; -using Microsoft.Extensions.Logging; -using Microsoft.Extensions.Options; -using StellaOps.Concelier.Storage.Mongo.Advisories; -using StellaOps.Concelier.Storage.Mongo.Exporting; -using StellaOps.Plugin; +using System.Collections.Generic; +using System.Collections.Immutable; +using System.IO; +using System.Linq; +using Microsoft.Extensions.Logging; +using Microsoft.Extensions.Options; +using StellaOps.Concelier.Core.Events; +using StellaOps.Concelier.Models; +using StellaOps.Concelier.Storage.Mongo.Advisories; +using StellaOps.Concelier.Storage.Mongo.Exporting; +using StellaOps.Plugin; namespace StellaOps.Concelier.Exporter.Json; @@ -16,29 +20,32 @@ public sealed class JsonFeedExporter : IFeedExporter public const string ExporterId = "export:json"; private readonly IAdvisoryStore _advisoryStore; - private readonly JsonExportOptions _options; - private readonly IJsonExportPathResolver _pathResolver; - private readonly ExportStateManager _stateManager; - private readonly ILogger _logger; - private readonly TimeProvider _timeProvider; - private readonly string _exporterVersion; - - public JsonFeedExporter( - IAdvisoryStore advisoryStore, - IOptions options, - IJsonExportPathResolver pathResolver, - ExportStateManager stateManager, - ILogger logger, - TimeProvider? timeProvider = null) - { - _advisoryStore = advisoryStore ?? throw new ArgumentNullException(nameof(advisoryStore)); - _options = options?.Value ?? throw new ArgumentNullException(nameof(options)); - _pathResolver = pathResolver ?? throw new ArgumentNullException(nameof(pathResolver)); - _stateManager = stateManager ?? throw new ArgumentNullException(nameof(stateManager)); - _logger = logger ?? throw new ArgumentNullException(nameof(logger)); - _timeProvider = timeProvider ?? TimeProvider.System; - _exporterVersion = ExporterVersion.GetVersion(typeof(JsonFeedExporter)); - } + private readonly JsonExportOptions _options; + private readonly IJsonExportPathResolver _pathResolver; + private readonly ExportStateManager _stateManager; + private readonly ILogger _logger; + private readonly TimeProvider _timeProvider; + private readonly string _exporterVersion; + private readonly IAdvisoryEventLog _eventLog; + + public JsonFeedExporter( + IAdvisoryStore advisoryStore, + IOptions options, + IJsonExportPathResolver pathResolver, + ExportStateManager stateManager, + IAdvisoryEventLog eventLog, + ILogger logger, + TimeProvider? timeProvider = null) + { + _advisoryStore = advisoryStore ?? throw new ArgumentNullException(nameof(advisoryStore)); + _options = options?.Value ?? throw new ArgumentNullException(nameof(options)); + _pathResolver = pathResolver ?? throw new ArgumentNullException(nameof(pathResolver)); + _stateManager = stateManager ?? throw new ArgumentNullException(nameof(stateManager)); + _eventLog = eventLog ?? throw new ArgumentNullException(nameof(eventLog)); + _logger = logger ?? throw new ArgumentNullException(nameof(logger)); + _timeProvider = timeProvider ?? TimeProvider.System; + _exporterVersion = ExporterVersion.GetVersion(typeof(JsonFeedExporter)); + } public string Name => ExporterName; @@ -52,11 +59,12 @@ public sealed class JsonFeedExporter : IFeedExporter var existingState = await _stateManager.GetAsync(ExporterId, cancellationToken).ConfigureAwait(false); - var builder = new JsonExportSnapshotBuilder(_options, _pathResolver); - var advisoryStream = _advisoryStore.StreamAsync(cancellationToken); - var result = await builder.WriteAsync(advisoryStream, exportedAt, exportId, cancellationToken).ConfigureAwait(false); - - var digest = ExportDigestCalculator.ComputeTreeDigest(result); + var builder = new JsonExportSnapshotBuilder(_options, _pathResolver); + var canonicalAdvisories = await MaterializeCanonicalAdvisoriesAsync(cancellationToken).ConfigureAwait(false); + var result = await builder.WriteAsync(canonicalAdvisories, exportedAt, exportId, cancellationToken).ConfigureAwait(false); + result = await JsonMirrorBundleWriter.WriteAsync(result, _options, services, _timeProvider, _logger, cancellationToken).ConfigureAwait(false); + + var digest = ExportDigestCalculator.ComputeTreeDigest(result); _logger.LogInformation( "JSON export {ExportId} wrote {FileCount} files ({Bytes} bytes) covering {AdvisoryCount} advisories with digest {Digest}", exportId, @@ -106,7 +114,34 @@ public sealed class JsonFeedExporter : IFeedExporter { TryUpdateLatestSymlink(exportRoot, result.ExportDirectory); } - } + } + + private async Task> MaterializeCanonicalAdvisoriesAsync(CancellationToken cancellationToken) + { + var keys = new SortedSet(StringComparer.OrdinalIgnoreCase); + + await foreach (var advisory in _advisoryStore.StreamAsync(cancellationToken)) + { + cancellationToken.ThrowIfCancellationRequested(); + if (!string.IsNullOrWhiteSpace(advisory.AdvisoryKey)) + { + keys.Add(advisory.AdvisoryKey.Trim()); + } + } + + var advisories = new List(keys.Count); + foreach (var key in keys) + { + cancellationToken.ThrowIfCancellationRequested(); + var replay = await _eventLog.ReplayAsync(key, asOf: null, cancellationToken).ConfigureAwait(false); + if (!replay.Statements.IsDefaultOrEmpty) + { + advisories.Add(replay.Statements[0].Advisory); + } + } + + return advisories; + } private void TryUpdateLatestSymlink(string exportRoot, string exportDirectory) { diff --git a/src/StellaOps.Concelier.Exporter.Json/JsonMirrorBundleWriter.cs b/src/StellaOps.Concelier.Exporter.Json/JsonMirrorBundleWriter.cs new file mode 100644 index 00000000..279dce17 --- /dev/null +++ b/src/StellaOps.Concelier.Exporter.Json/JsonMirrorBundleWriter.cs @@ -0,0 +1,622 @@ +using System; +using System.Buffers; +using System.Collections.Generic; +using System.IO; +using System.Linq; +using System.Security.Cryptography; +using System.Text; +using System.Text.Json; +using System.Text.Json.Serialization; +using System.Threading; +using System.Threading.Tasks; +using Microsoft.Extensions.DependencyInjection; +using Microsoft.Extensions.Hosting; +using Microsoft.Extensions.Logging; +using StellaOps.Concelier.Models; +using StellaOps.Cryptography; + +namespace StellaOps.Concelier.Exporter.Json; + +internal static class JsonMirrorBundleWriter +{ + private const int SchemaVersion = 1; + private const string BundleFileName = "bundle.json"; + private const string BundleSignatureFileName = "bundle.json.jws"; + private const string ManifestFileName = "manifest.json"; + private const string IndexFileName = "index.json"; + private const string SignatureMediaType = "application/vnd.stellaops.concelier.mirror-bundle+jws"; + private const string DefaultMirrorDirectoryName = "mirror"; + + private static readonly Encoding Utf8NoBom = new UTF8Encoding(encoderShouldEmitUTF8Identifier: false); + + private static readonly JsonSerializerOptions HeaderSerializerOptions = new(JsonSerializerDefaults.General) + { + PropertyNamingPolicy = null, + DefaultIgnoreCondition = JsonIgnoreCondition.WhenWritingNull, + WriteIndented = false, + }; + + public static async Task WriteAsync( + JsonExportResult result, + JsonExportOptions options, + IServiceProvider services, + TimeProvider timeProvider, + ILogger logger, + CancellationToken cancellationToken) + { + ArgumentNullException.ThrowIfNull(result); + ArgumentNullException.ThrowIfNull(options); + ArgumentNullException.ThrowIfNull(services); + ArgumentNullException.ThrowIfNull(timeProvider); + ArgumentNullException.ThrowIfNull(logger); + + var mirrorOptions = options.Mirror ?? new JsonExportOptions.JsonMirrorOptions(); + if (!mirrorOptions.Enabled || mirrorOptions.Domains.Count == 0) + { + return result; + } + + cancellationToken.ThrowIfCancellationRequested(); + + var exportedAtUtc = result.ExportedAt.UtcDateTime; + var mirrorDirectoryName = string.IsNullOrWhiteSpace(mirrorOptions.DirectoryName) + ? DefaultMirrorDirectoryName + : mirrorOptions.DirectoryName.Trim(); + + var mirrorRoot = Path.Combine(result.ExportDirectory, mirrorDirectoryName); + Directory.CreateDirectory(mirrorRoot); + TrySetDirectoryTimestamp(mirrorRoot, exportedAtUtc); + + var advisories = result.Advisories.IsDefaultOrEmpty + ? Array.Empty() + : result.Advisories + .OrderBy(static advisory => advisory.AdvisoryKey, StringComparer.Ordinal) + .ToArray(); + + var signingContext = PrepareSigningContext(mirrorOptions.Signing, services, timeProvider, logger); + var additionalFiles = new List(); + var domainEntries = new List(); + + foreach (var domainOption in mirrorOptions.Domains) + { + cancellationToken.ThrowIfCancellationRequested(); + if (domainOption is null) + { + logger.LogWarning("Encountered null mirror domain configuration; skipping."); + continue; + } + + var domainId = (domainOption.Id ?? string.Empty).Trim(); + if (domainId.Length == 0) + { + logger.LogWarning("Skipping mirror domain with empty id."); + continue; + } + + var schemeFilter = CreateFilterSet(domainOption.IncludeSchemes); + var sourceFilter = CreateFilterSet(domainOption.IncludeSources); + var domainAdvisories = advisories + .Where(advisory => MatchesFilters(advisory, schemeFilter, sourceFilter)) + .ToArray(); + + var sources = BuildSourceSummaries(domainAdvisories); + var domainDisplayName = string.IsNullOrWhiteSpace(domainOption.DisplayName) + ? domainId + : domainOption.DisplayName!.Trim(); + + var domainDirectory = Path.Combine(mirrorRoot, domainId); + Directory.CreateDirectory(domainDirectory); + TrySetDirectoryTimestamp(domainDirectory, exportedAtUtc); + + var bundleDocument = new MirrorDomainBundleDocument( + SchemaVersion, + result.ExportedAt, + options.TargetRepository, + domainId, + domainDisplayName, + domainAdvisories.Length, + domainAdvisories, + sources); + + var bundleBytes = Serialize(bundleDocument); + var bundlePath = Path.Combine(domainDirectory, BundleFileName); + await WriteFileAsync(bundlePath, bundleBytes, exportedAtUtc, cancellationToken).ConfigureAwait(false); + + var bundleRelativePath = ToRelativePath(result.ExportDirectory, bundlePath); + var bundleDigest = ComputeDigest(bundleBytes); + var bundleLength = (long)bundleBytes.LongLength; + additionalFiles.Add(new JsonExportFile(bundleRelativePath, bundleLength, bundleDigest)); + + MirrorSignatureDescriptor? signatureDescriptor = null; + if (signingContext is not null) + { + var (signatureValue, signedAt) = await CreateSignatureAsync( + signingContext, + bundleBytes, + timeProvider, + cancellationToken) + .ConfigureAwait(false); + + var signatureBytes = Utf8NoBom.GetBytes(signatureValue); + var signaturePath = Path.Combine(domainDirectory, BundleSignatureFileName); + await WriteFileAsync(signaturePath, signatureBytes, exportedAtUtc, cancellationToken).ConfigureAwait(false); + + var signatureRelativePath = ToRelativePath(result.ExportDirectory, signaturePath); + var signatureDigest = ComputeDigest(signatureBytes); + var signatureLength = (long)signatureBytes.LongLength; + additionalFiles.Add(new JsonExportFile(signatureRelativePath, signatureLength, signatureDigest)); + + signatureDescriptor = new MirrorSignatureDescriptor( + signatureRelativePath, + signingContext.Algorithm, + signingContext.KeyId, + signingContext.Provider, + signedAt); + } + + var bundleDescriptor = new MirrorFileDescriptor(bundleRelativePath, bundleLength, bundleDigest, signatureDescriptor); + + var manifestDocument = new MirrorDomainManifestDocument( + SchemaVersion, + result.ExportedAt, + domainId, + domainDisplayName, + domainAdvisories.Length, + sources, + bundleDescriptor); + + var manifestBytes = Serialize(manifestDocument); + var manifestPath = Path.Combine(domainDirectory, ManifestFileName); + await WriteFileAsync(manifestPath, manifestBytes, exportedAtUtc, cancellationToken).ConfigureAwait(false); + + var manifestRelativePath = ToRelativePath(result.ExportDirectory, manifestPath); + var manifestDigest = ComputeDigest(manifestBytes); + var manifestLength = (long)manifestBytes.LongLength; + additionalFiles.Add(new JsonExportFile(manifestRelativePath, manifestLength, manifestDigest)); + + var manifestDescriptor = new MirrorFileDescriptor(manifestRelativePath, manifestLength, manifestDigest, null); + + domainEntries.Add(new MirrorIndexDomainEntry( + domainId, + domainDisplayName, + domainAdvisories.Length, + manifestDescriptor, + bundleDescriptor, + sources)); + } + + domainEntries.Sort(static (left, right) => string.CompareOrdinal(left.DomainId, right.DomainId)); + + var indexDocument = new MirrorIndexDocument( + SchemaVersion, + result.ExportedAt, + options.TargetRepository, + domainEntries); + + var indexBytes = Serialize(indexDocument); + var indexPath = Path.Combine(mirrorRoot, IndexFileName); + await WriteFileAsync(indexPath, indexBytes, exportedAtUtc, cancellationToken).ConfigureAwait(false); + + var indexRelativePath = ToRelativePath(result.ExportDirectory, indexPath); + var indexDigest = ComputeDigest(indexBytes); + var indexLength = (long)indexBytes.LongLength; + additionalFiles.Add(new JsonExportFile(indexRelativePath, indexLength, indexDigest)); + + logger.LogInformation( + "Generated {DomainCount} Concelier mirror domain bundle(s) under {MirrorRoot}.", + domainEntries.Count, + mirrorDirectoryName); + + var combinedFiles = new List(result.Files.Length + additionalFiles.Count); + combinedFiles.AddRange(result.Files); + combinedFiles.AddRange(additionalFiles); + + var combinedTotalBytes = checked(result.TotalBytes + additionalFiles.Sum(static file => file.Length)); + + return new JsonExportResult( + result.ExportDirectory, + result.ExportedAt, + combinedFiles, + result.AdvisoryCount, + combinedTotalBytes, + result.Advisories); + } + + private static JsonMirrorSigningContext? PrepareSigningContext( + JsonExportOptions.JsonMirrorSigningOptions signingOptions, + IServiceProvider services, + TimeProvider timeProvider, + ILogger logger) + { + if (signingOptions is null || !signingOptions.Enabled) + { + return null; + } + + var algorithm = string.IsNullOrWhiteSpace(signingOptions.Algorithm) + ? SignatureAlgorithms.Es256 + : signingOptions.Algorithm.Trim(); + var keyId = (signingOptions.KeyId ?? string.Empty).Trim(); + if (keyId.Length == 0) + { + throw new InvalidOperationException("Mirror signing requires mirror.signing.keyId to be configured."); + } + + var registry = services.GetService() + ?? throw new InvalidOperationException("Mirror signing requires ICryptoProviderRegistry to be registered."); + + var providerHint = signingOptions.Provider?.Trim(); + var keyReference = new CryptoKeyReference(keyId, providerHint); + + CryptoSignerResolution resolved; + try + { + resolved = registry.ResolveSigner(CryptoCapability.Signing, algorithm, keyReference, providerHint); + } + catch (KeyNotFoundException) + { + var provider = ResolveProvider(registry, algorithm, providerHint); + var signingKey = LoadSigningKey(signingOptions, provider, services, timeProvider, algorithm); + provider.UpsertSigningKey(signingKey); + resolved = registry.ResolveSigner(CryptoCapability.Signing, algorithm, keyReference, provider.Name); + } + + logger.LogDebug( + "Mirror signing configured with key {KeyId} via provider {Provider} using {Algorithm}.", + resolved.Signer.KeyId, + resolved.ProviderName, + algorithm); + + return new JsonMirrorSigningContext(resolved.Signer, algorithm, resolved.Signer.KeyId, resolved.ProviderName); + } + + private static ICryptoProvider ResolveProvider(ICryptoProviderRegistry registry, string algorithm, string? providerHint) + { + if (!string.IsNullOrWhiteSpace(providerHint) && registry.TryResolve(providerHint, out var hinted)) + { + if (!hinted.Supports(CryptoCapability.Signing, algorithm)) + { + throw new InvalidOperationException( + $"Crypto provider '{providerHint}' does not support signing algorithm '{algorithm}'."); + } + + return hinted; + } + + return registry.ResolveOrThrow(CryptoCapability.Signing, algorithm); + } + + private static CryptoSigningKey LoadSigningKey( + JsonExportOptions.JsonMirrorSigningOptions signingOptions, + ICryptoProvider provider, + IServiceProvider services, + TimeProvider timeProvider, + string algorithm) + { + var keyPath = (signingOptions.KeyPath ?? string.Empty).Trim(); + if (keyPath.Length == 0) + { + throw new InvalidOperationException("Mirror signing requires mirror.signing.keyPath to be configured."); + } + + var environment = services.GetService(); + var basePath = environment?.ContentRootPath ?? AppContext.BaseDirectory; + var resolvedPath = Path.IsPathRooted(keyPath) + ? keyPath + : Path.GetFullPath(Path.Combine(basePath, keyPath)); + + if (!File.Exists(resolvedPath)) + { + throw new FileNotFoundException($"Mirror signing key '{signingOptions.KeyId}' not found.", resolvedPath); + } + + var pem = File.ReadAllText(resolvedPath); + using var ecdsa = ECDsa.Create(); + try + { + ecdsa.ImportFromPem(pem); + } + catch (CryptographicException ex) + { + throw new InvalidOperationException("Failed to import mirror signing key. Ensure the PEM contains an EC private key.", ex); + } + + var parameters = ecdsa.ExportParameters(includePrivateParameters: true); + return new CryptoSigningKey( + new CryptoKeyReference(signingOptions.KeyId, provider.Name), + algorithm, + in parameters, + timeProvider.GetUtcNow()); + } + + private static async Task<(string Value, DateTimeOffset SignedAt)> CreateSignatureAsync( + JsonMirrorSigningContext context, + ReadOnlyMemory payload, + TimeProvider timeProvider, + CancellationToken cancellationToken) + { + var header = new Dictionary + { + ["alg"] = context.Algorithm, + ["kid"] = context.KeyId, + ["typ"] = SignatureMediaType, + ["b64"] = false, + ["crit"] = new[] { "b64" } + }; + + if (!string.IsNullOrWhiteSpace(context.Provider)) + { + header["provider"] = context.Provider; + } + + var headerJson = JsonSerializer.Serialize(header, HeaderSerializerOptions); + var protectedHeader = Base64UrlEncode(Utf8NoBom.GetBytes(headerJson)); + var signingInputLength = protectedHeader.Length + 1 + payload.Length; + var buffer = ArrayPool.Shared.Rent(signingInputLength); + + try + { + var headerBytes = Encoding.ASCII.GetBytes(protectedHeader); + Buffer.BlockCopy(headerBytes, 0, buffer, 0, headerBytes.Length); + buffer[headerBytes.Length] = (byte)'.'; + var payloadArray = payload.ToArray(); + Buffer.BlockCopy(payloadArray, 0, buffer, headerBytes.Length + 1, payloadArray.Length); + + var signingInput = new ReadOnlyMemory(buffer, 0, signingInputLength); + var signatureBytes = await context.Signer.SignAsync(signingInput, cancellationToken).ConfigureAwait(false); + var encodedSignature = Base64UrlEncode(signatureBytes); + var signedAt = timeProvider.GetUtcNow(); + return (string.Concat(protectedHeader, "..", encodedSignature), signedAt); + } + finally + { + ArrayPool.Shared.Return(buffer); + } + } + + private static IReadOnlyList BuildSourceSummaries(IReadOnlyList advisories) + { + var builders = new Dictionary(StringComparer.OrdinalIgnoreCase); + + foreach (var advisory in advisories) + { + var counted = new HashSet(StringComparer.OrdinalIgnoreCase); + foreach (var provenance in advisory.Provenance) + { + if (string.IsNullOrWhiteSpace(provenance.Source)) + { + continue; + } + + var source = provenance.Source.Trim(); + if (!builders.TryGetValue(source, out var accumulator)) + { + accumulator = new SourceAccumulator(); + builders[source] = accumulator; + } + + accumulator.Record(provenance.RecordedAt); + if (counted.Add(source)) + { + accumulator.IncrementAdvisoryCount(); + } + } + } + + return builders + .OrderBy(static pair => pair.Key, StringComparer.Ordinal) + .Select(pair => new JsonMirrorSourceSummary( + pair.Key, + pair.Value.FirstRecordedAt, + pair.Value.LastRecordedAt, + pair.Value.AdvisoryCount)) + .ToArray(); + } + + private static HashSet? CreateFilterSet(IList? values) + { + if (values is null || values.Count == 0) + { + return null; + } + + var set = new HashSet(StringComparer.OrdinalIgnoreCase); + foreach (var value in values) + { + if (string.IsNullOrWhiteSpace(value)) + { + continue; + } + + set.Add(value.Trim()); + } + + return set.Count == 0 ? null : set; + } + + private static bool MatchesFilters(Advisory advisory, HashSet? schemeFilter, HashSet? sourceFilter) + { + if (schemeFilter is not null) + { + var scheme = ExtractScheme(advisory.AdvisoryKey); + if (!schemeFilter.Contains(scheme)) + { + return false; + } + } + + if (sourceFilter is not null) + { + var hasSource = advisory.Provenance.Any(provenance => + !string.IsNullOrWhiteSpace(provenance.Source) && + sourceFilter.Contains(provenance.Source.Trim())); + + if (!hasSource) + { + return false; + } + } + + return true; + } + + private static string ExtractScheme(string advisoryKey) + { + if (string.IsNullOrWhiteSpace(advisoryKey)) + { + return string.Empty; + } + + var trimmed = advisoryKey.Trim(); + var separatorIndex = trimmed.IndexOf(':'); + return separatorIndex <= 0 ? trimmed : trimmed[..separatorIndex]; + } + + private static byte[] Serialize(T value) + { + var json = CanonicalJsonSerializer.SerializeIndented(value); + return Utf8NoBom.GetBytes(json); + } + + private static async Task WriteFileAsync(string path, byte[] content, DateTime exportedAtUtc, CancellationToken cancellationToken) + { + await File.WriteAllBytesAsync(path, content, cancellationToken).ConfigureAwait(false); + File.SetLastWriteTimeUtc(path, exportedAtUtc); + } + + private static string ToRelativePath(string root, string fullPath) + { + var relative = Path.GetRelativePath(root, fullPath); + return relative.Replace(Path.DirectorySeparatorChar, '/'); + } + + private static string ComputeDigest(ReadOnlySpan payload) + { + var hash = SHA256.HashData(payload); + return $"sha256:{Convert.ToHexString(hash).ToLowerInvariant()}"; + } + + private static void TrySetDirectoryTimestamp(string directory, DateTime exportedAtUtc) + { + try + { + Directory.SetLastWriteTimeUtc(directory, exportedAtUtc); + } + catch (IOException) + { + } + catch (UnauthorizedAccessException) + { + } + catch (PlatformNotSupportedException) + { + } + } + + private static string Base64UrlEncode(ReadOnlySpan value) + { + var encoded = Convert.ToBase64String(value); + var builder = new StringBuilder(encoded.Length); + foreach (var ch in encoded) + { + switch (ch) + { + case '+': + builder.Append('-'); + break; + case '/': + builder.Append('_'); + break; + case '=': + break; + default: + builder.Append(ch); + break; + } + } + + return builder.ToString(); + } + + private sealed record JsonMirrorSigningContext(ICryptoSigner Signer, string Algorithm, string KeyId, string Provider); + + private sealed record MirrorIndexDocument( + int SchemaVersion, + DateTimeOffset GeneratedAt, + string? TargetRepository, + IReadOnlyList Domains); + + private sealed record MirrorIndexDomainEntry( + string DomainId, + string DisplayName, + int AdvisoryCount, + MirrorFileDescriptor Manifest, + MirrorFileDescriptor Bundle, + IReadOnlyList Sources); + + private sealed record MirrorDomainManifestDocument( + int SchemaVersion, + DateTimeOffset GeneratedAt, + string DomainId, + string DisplayName, + int AdvisoryCount, + IReadOnlyList Sources, + MirrorFileDescriptor Bundle); + + private sealed record MirrorDomainBundleDocument( + int SchemaVersion, + DateTimeOffset GeneratedAt, + string? TargetRepository, + string DomainId, + string DisplayName, + int AdvisoryCount, + IReadOnlyList Advisories, + IReadOnlyList Sources); + + private sealed record MirrorFileDescriptor( + string Path, + long SizeBytes, + string Digest, + MirrorSignatureDescriptor? Signature); + + private sealed record MirrorSignatureDescriptor( + string Path, + string Algorithm, + string KeyId, + string Provider, + DateTimeOffset SignedAt); + + private sealed record JsonMirrorSourceSummary( + string Source, + DateTimeOffset? FirstRecordedAt, + DateTimeOffset? LastRecordedAt, + int AdvisoryCount); + + private sealed class SourceAccumulator + { + public DateTimeOffset? FirstRecordedAt { get; private set; } + + public DateTimeOffset? LastRecordedAt { get; private set; } + + public int AdvisoryCount { get; private set; } + + public void Record(DateTimeOffset recordedAt) + { + var normalized = recordedAt.ToUniversalTime(); + if (FirstRecordedAt is null || normalized < FirstRecordedAt.Value) + { + FirstRecordedAt = normalized; + } + + if (LastRecordedAt is null || normalized > LastRecordedAt.Value) + { + LastRecordedAt = normalized; + } + } + + public void IncrementAdvisoryCount() + { + AdvisoryCount++; + } + } +} diff --git a/src/StellaOps.Concelier.Exporter.Json/StellaOps.Concelier.Exporter.Json.csproj b/src/StellaOps.Concelier.Exporter.Json/StellaOps.Concelier.Exporter.Json.csproj index 1c5b3932..967a1a9e 100644 --- a/src/StellaOps.Concelier.Exporter.Json/StellaOps.Concelier.Exporter.Json.csproj +++ b/src/StellaOps.Concelier.Exporter.Json/StellaOps.Concelier.Exporter.Json.csproj @@ -10,13 +10,15 @@ - - - - - - - - - - + + + + + + + + + + + + diff --git a/src/StellaOps.Concelier.Exporter.Json/TASKS.md b/src/StellaOps.Concelier.Exporter.Json/TASKS.md index b00a2741..e52740fa 100644 --- a/src/StellaOps.Concelier.Exporter.Json/TASKS.md +++ b/src/StellaOps.Concelier.Exporter.Json/TASKS.md @@ -10,4 +10,4 @@ |Stream advisories during export|BE-Export|Storage.Mongo|DONE – exporter + streaming-only test ensures single enumeration and per-file digest capture.| |Emit export manifest with digest metadata|BE-Export|Exporters|DONE – manifest now includes per-file digests/sizes alongside tree digest.| |Surface new advisory fields (description/CWEs/canonical metric)|BE-Export|Models, Core|DONE (2025-10-15) – JSON exporter validated with new fixtures ensuring description/CWEs/canonical metric are preserved in outputs; `dotnet test src/StellaOps.Concelier.Exporter.Json.Tests` run 2025-10-15 for regression coverage.| -|CONCELIER-EXPORT-08-201 – Mirror bundle + domain manifest|Team Concelier Export|FEEDCORE-ENGINE-07-001|TODO – Produce per-domain aggregate bundles (JSON + manifest) with deterministic digests, include upstream source metadata, and publish index consumed by mirror endpoints/tests.| +|CONCELIER-EXPORT-08-201 – Mirror bundle + domain manifest|Team Concelier Export|FEEDCORE-ENGINE-07-001|DONE (2025-10-19) – Mirror bundle writer emits domain aggregates + manifests with cosign-compatible JWS signatures; index/tests updated via `dotnet test src/StellaOps.Concelier.Exporter.Json.Tests/StellaOps.Concelier.Exporter.Json.Tests.csproj` (2025-10-19).| diff --git a/src/StellaOps.Concelier.Exporter.TrivyDb.Tests/TrivyDbFeedExporterTests.cs b/src/StellaOps.Concelier.Exporter.TrivyDb.Tests/TrivyDbFeedExporterTests.cs index 5b3b2337..1f986832 100644 --- a/src/StellaOps.Concelier.Exporter.TrivyDb.Tests/TrivyDbFeedExporterTests.cs +++ b/src/StellaOps.Concelier.Exporter.TrivyDb.Tests/TrivyDbFeedExporterTests.cs @@ -283,6 +283,155 @@ public sealed class TrivyDbFeedExporterTests : IDisposable Assert.Empty(orasPusher.Pushes); } + [Fact] + public async Task ExportAsync_WritesMirrorBundlesWhenConfigured() + { + var advisoryOne = CreateSampleAdvisory("CVE-2025-1001", "Mirror Advisory One"); + var advisoryTwo = CreateSampleAdvisory("CVE-2025-1002", "Mirror Advisory Two"); + var advisoryStore = new StubAdvisoryStore(advisoryOne, advisoryTwo); + + var optionsValue = new TrivyDbExportOptions + { + OutputRoot = _root, + ReferencePrefix = "example/trivy", + TargetRepository = "s3://mirror/trivy", + Json = new JsonExportOptions + { + OutputRoot = _jsonRoot, + MaintainLatestSymlink = false, + }, + KeepWorkingTree = false, + }; + + optionsValue.Mirror.Enabled = true; + optionsValue.Mirror.DirectoryName = "mirror"; + optionsValue.Mirror.Domains.Add(new TrivyDbMirrorDomainOptions + { + Id = "primary", + DisplayName = "Primary Mirror", + }); + + var options = Options.Create(optionsValue); + var packageBuilder = new TrivyDbPackageBuilder(); + var ociWriter = new TrivyDbOciWriter(); + var planner = new TrivyDbExportPlanner(); + var stateStore = new InMemoryExportStateStore(); + var exportedAt = DateTimeOffset.Parse("2024-09-18T12:00:00Z", CultureInfo.InvariantCulture); + var timeProvider = new TestTimeProvider(exportedAt); + var stateManager = new ExportStateManager(stateStore, timeProvider); + var builderMetadata = JsonSerializer.SerializeToUtf8Bytes(new + { + Version = 2, + NextUpdate = "2024-09-19T12:00:00Z", + UpdatedAt = "2024-09-18T12:00:00Z", + }); + var builder = new StubTrivyDbBuilder(_root, builderMetadata); + var orasPusher = new StubTrivyDbOrasPusher(); + var exporter = new TrivyDbFeedExporter( + advisoryStore, + new VulnListJsonExportPathResolver(), + options, + packageBuilder, + ociWriter, + stateManager, + planner, + builder, + orasPusher, + NullLogger.Instance, + timeProvider); + + using var provider = new ServiceCollection().BuildServiceProvider(); + await exporter.ExportAsync(provider, CancellationToken.None); + + var exportId = exportedAt.ToString(optionsValue.TagFormat, CultureInfo.InvariantCulture); + var layoutPath = optionsValue.GetExportRoot(exportId); + var mirrorRoot = Path.Combine(layoutPath, "mirror"); + var domainRoot = Path.Combine(mirrorRoot, "primary"); + + Assert.True(File.Exists(Path.Combine(mirrorRoot, "index.json"))); + Assert.True(File.Exists(Path.Combine(domainRoot, "manifest.json"))); + Assert.True(File.Exists(Path.Combine(domainRoot, "metadata.json"))); + Assert.True(File.Exists(Path.Combine(domainRoot, "db.tar.gz"))); + + var reference = $"{optionsValue.ReferencePrefix}:{exportId}"; + var manifestDigest = ReadManifestDigest(layoutPath); + var indexPath = Path.Combine(mirrorRoot, "index.json"); + string? indexManifestDescriptorDigest = null; + string? indexMetadataDigest = null; + string? indexDatabaseDigest = null; + + using (var indexDoc = JsonDocument.Parse(File.ReadAllBytes(indexPath))) + { + var root = indexDoc.RootElement; + Assert.Equal(1, root.GetProperty("schemaVersion").GetInt32()); + Assert.Equal(reference, root.GetProperty("reference").GetString()); + Assert.Equal(manifestDigest, root.GetProperty("manifestDigest").GetString()); + Assert.Equal("full", root.GetProperty("mode").GetString()); + Assert.Equal("s3://mirror/trivy", root.GetProperty("targetRepository").GetString()); + Assert.False(root.TryGetProperty("delta", out _)); + + var domains = root.GetProperty("domains").EnumerateArray().ToArray(); + var domain = Assert.Single(domains); + Assert.Equal("primary", domain.GetProperty("domainId").GetString()); + Assert.Equal("Primary Mirror", domain.GetProperty("displayName").GetString()); + Assert.Equal(2, domain.GetProperty("advisoryCount").GetInt32()); + + var manifestDescriptor = domain.GetProperty("manifest"); + Assert.Equal("mirror/primary/manifest.json", manifestDescriptor.GetProperty("path").GetString()); + indexManifestDescriptorDigest = manifestDescriptor.GetProperty("digest").GetString(); + + var metadataDescriptor = domain.GetProperty("metadata"); + Assert.Equal("mirror/primary/metadata.json", metadataDescriptor.GetProperty("path").GetString()); + indexMetadataDigest = metadataDescriptor.GetProperty("digest").GetString(); + + var databaseDescriptor = domain.GetProperty("database"); + Assert.Equal("mirror/primary/db.tar.gz", databaseDescriptor.GetProperty("path").GetString()); + indexDatabaseDigest = databaseDescriptor.GetProperty("digest").GetString(); + } + + var domainManifestPath = Path.Combine(domainRoot, "manifest.json"); + var rootMetadataPath = Path.Combine(layoutPath, "metadata.json"); + var domainMetadataPath = Path.Combine(domainRoot, "metadata.json"); + var domainDbPath = Path.Combine(domainRoot, "db.tar.gz"); + + var domainManifestBytes = File.ReadAllBytes(domainManifestPath); + var domainManifestDigest = "sha256:" + Convert.ToHexString(SHA256.HashData(domainManifestBytes)).ToLowerInvariant(); + var rootMetadataBytes = File.ReadAllBytes(rootMetadataPath); + var domainMetadataBytes = File.ReadAllBytes(domainMetadataPath); + Assert.Equal(rootMetadataBytes, domainMetadataBytes); + + var metadataDigest = "sha256:" + Convert.ToHexString(SHA256.HashData(domainMetadataBytes)).ToLowerInvariant(); + var databaseDigest = "sha256:" + Convert.ToHexString(SHA256.HashData(File.ReadAllBytes(domainDbPath))).ToLowerInvariant(); + Assert.Equal(domainManifestDigest, indexManifestDescriptorDigest); + Assert.Equal(metadataDigest, indexMetadataDigest); + Assert.Equal(databaseDigest, indexDatabaseDigest); + + using (var manifestDoc = JsonDocument.Parse(File.ReadAllBytes(domainManifestPath))) + { + var manifestRoot = manifestDoc.RootElement; + Assert.Equal("primary", manifestRoot.GetProperty("domainId").GetString()); + Assert.Equal("Primary Mirror", manifestRoot.GetProperty("displayName").GetString()); + Assert.Equal(reference, manifestRoot.GetProperty("reference").GetString()); + Assert.Equal(manifestDigest, manifestRoot.GetProperty("manifestDigest").GetString()); + Assert.Equal("full", manifestRoot.GetProperty("mode").GetString()); + Assert.Equal("s3://mirror/trivy", manifestRoot.GetProperty("targetRepository").GetString()); + + var metadataDescriptor = manifestRoot.GetProperty("metadata"); + Assert.Equal("mirror/primary/metadata.json", metadataDescriptor.GetProperty("path").GetString()); + Assert.Equal(metadataDigest, metadataDescriptor.GetProperty("digest").GetString()); + + var databaseDescriptor = manifestRoot.GetProperty("database"); + Assert.Equal("mirror/primary/db.tar.gz", databaseDescriptor.GetProperty("path").GetString()); + Assert.Equal(databaseDigest, databaseDescriptor.GetProperty("digest").GetString()); + + var sources = manifestRoot.GetProperty("sources").EnumerateArray().ToArray(); + Assert.NotEmpty(sources); + Assert.Contains(sources, element => string.Equals(element.GetProperty("source").GetString(), "nvd", StringComparison.OrdinalIgnoreCase)); + } + + Assert.Empty(orasPusher.Pushes); + } + [Fact] public async Task ExportAsync_SkipsOrasPushWhenDeltaPublishingDisabled() { @@ -774,7 +923,9 @@ public sealed class TrivyDbFeedExporterTests : IDisposable private ExportStateRecord? _record; public Task FindAsync(string id, CancellationToken cancellationToken) - => Task.FromResult(_record); + { + return Task.FromResult(_record); + } public Task UpsertAsync(ExportStateRecord record, CancellationToken cancellationToken) { diff --git a/src/StellaOps.Concelier.Exporter.TrivyDb/StellaOps.Concelier.Exporter.TrivyDb.csproj b/src/StellaOps.Concelier.Exporter.TrivyDb/StellaOps.Concelier.Exporter.TrivyDb.csproj index 260e48b5..370c32e8 100644 --- a/src/StellaOps.Concelier.Exporter.TrivyDb/StellaOps.Concelier.Exporter.TrivyDb.csproj +++ b/src/StellaOps.Concelier.Exporter.TrivyDb/StellaOps.Concelier.Exporter.TrivyDb.csproj @@ -15,8 +15,8 @@ - + - \ No newline at end of file + diff --git a/src/StellaOps.Concelier.Exporter.TrivyDb/TASKS.md b/src/StellaOps.Concelier.Exporter.TrivyDb/TASKS.md index a2a860b8..e7ee2ed0 100644 --- a/src/StellaOps.Concelier.Exporter.TrivyDb/TASKS.md +++ b/src/StellaOps.Concelier.Exporter.TrivyDb/TASKS.md @@ -12,4 +12,4 @@ |Streamed package building to avoid large copies|BE-Export|Exporters|DONE – metadata/config now reuse backing arrays and OCI writer streams directly without double buffering.| |Plan incremental/delta exports|BE-Export|Exporters|DONE – state captures per-file manifests, planner schedules delta vs full resets, layer reuse smoke test verifies OCI reuse, and operator guide documents the validation flow.| |Advisory schema parity export (description/CWEs/canonical metric)|BE-Export|Models, Core|DONE (2025-10-15) – exporter/test fixtures updated to handle description/CWEs/canonical metric fields during Trivy DB packaging; `dotnet test src/StellaOps.Concelier.Exporter.TrivyDb.Tests` re-run 2025-10-15 to confirm coverage.| -|CONCELIER-EXPORT-08-202 – Mirror-ready Trivy DB bundles|Team Concelier Export|CONCELIER-EXPORT-08-201|TODO – Generate domain-specific Trivy DB archives + metadata manifest, ensure deterministic digests, and document sync process for downstream Concelier nodes.| +|CONCELIER-EXPORT-08-202 – Mirror-ready Trivy DB bundles|Team Concelier Export|CONCELIER-EXPORT-08-201|**DONE (2025-10-19)** – Added mirror export options and writer emitting `mirror/index.json` plus per-domain `manifest.json`/`metadata.json`/`db.tar.gz` with deterministic SHA-256 digests; regression covered via `dotnet test src/StellaOps.Concelier.Exporter.TrivyDb.Tests/StellaOps.Concelier.Exporter.TrivyDb.Tests.csproj`.| diff --git a/src/StellaOps.Concelier.Exporter.TrivyDb/TrivyDbExportOptions.cs b/src/StellaOps.Concelier.Exporter.TrivyDb/TrivyDbExportOptions.cs index 81f8ebd4..bdbdbddc 100644 --- a/src/StellaOps.Concelier.Exporter.TrivyDb/TrivyDbExportOptions.cs +++ b/src/StellaOps.Concelier.Exporter.TrivyDb/TrivyDbExportOptions.cs @@ -24,25 +24,43 @@ public sealed class TrivyDbExportOptions OutputRoot = Path.Combine("exports", "trivy", "tree") }; - public TrivyDbBuilderOptions Builder { get; set; } = new(); - - public TrivyDbOrasOptions Oras { get; set; } = new(); - - public TrivyDbOfflineBundleOptions OfflineBundle { get; set; } = new(); - - public string GetExportRoot(string exportId) - { - ArgumentException.ThrowIfNullOrEmpty(exportId); - var root = Path.GetFullPath(OutputRoot); - return Path.Combine(root, exportId); - } -} - -public sealed class TrivyDbBuilderOptions -{ - public string ExecutablePath { get; set; } = "trivy-db"; - - public string? WorkingDirectory { get; set; } + public TrivyDbBuilderOptions Builder { get; set; } = new(); + + public TrivyDbOrasOptions Oras { get; set; } = new(); + + public TrivyDbOfflineBundleOptions OfflineBundle { get; set; } = new(); + + public TrivyDbMirrorOptions Mirror { get; set; } = new(); + + public string GetExportRoot(string exportId) + { + ArgumentException.ThrowIfNullOrEmpty(exportId); + var root = Path.GetFullPath(OutputRoot); + return Path.Combine(root, exportId); + } +} + +public sealed class TrivyDbMirrorOptions +{ + public bool Enabled { get; set; } + + public string DirectoryName { get; set; } = "mirror"; + + public IList Domains { get; } = new List(); +} + +public sealed class TrivyDbMirrorDomainOptions +{ + public string Id { get; set; } = string.Empty; + + public string? DisplayName { get; set; } +} + +public sealed class TrivyDbBuilderOptions +{ + public string ExecutablePath { get; set; } = "trivy-db"; + + public string? WorkingDirectory { get; set; } public TimeSpan UpdateInterval { get; set; } = TimeSpan.FromHours(24); diff --git a/src/StellaOps.Concelier.Exporter.TrivyDb/TrivyDbFeedExporter.cs b/src/StellaOps.Concelier.Exporter.TrivyDb/TrivyDbFeedExporter.cs index de614be9..04598dbc 100644 --- a/src/StellaOps.Concelier.Exporter.TrivyDb/TrivyDbFeedExporter.cs +++ b/src/StellaOps.Concelier.Exporter.TrivyDb/TrivyDbFeedExporter.cs @@ -118,6 +118,8 @@ public sealed class TrivyDbFeedExporter : IFeedExporter var builderResult = await _builder.BuildAsync(jsonResult, exportedAt, exportId, cancellationToken).ConfigureAwait(false); var metadataBytes = CreateMetadataJson(plan, builderResult.BuilderMetadata, treeDigest, jsonResult, exportedAt); + var metadataDigest = ComputeDigest(metadataBytes); + var metadataLength = metadataBytes.LongLength; try { @@ -137,6 +139,22 @@ public sealed class TrivyDbFeedExporter : IFeedExporter } var ociResult = await _ociWriter.WriteAsync(package, destination, reference, plan, baseLayout, cancellationToken).ConfigureAwait(false); + + await TrivyDbMirrorBundleWriter.WriteAsync( + destination, + jsonResult, + _options, + plan, + builderResult, + reference, + ociResult.ManifestDigest, + metadataBytes, + metadataDigest, + metadataLength, + _exporterVersion, + exportedAt, + _logger, + cancellationToken).ConfigureAwait(false); if (_options.Oras.Enabled && ShouldPublishToOras(plan.Mode)) { @@ -421,6 +439,13 @@ public sealed class TrivyDbFeedExporter : IFeedExporter return string.IsNullOrEmpty(normalized) ? "." : normalized; } + private static string ComputeDigest(ReadOnlySpan payload) + { + var hash = SHA256.HashData(payload); + var hex = Convert.ToHexString(hash).ToLowerInvariant(); + return $"sha256:{hex}"; + } + private bool ShouldPublishToOras(TrivyDbExportMode mode) { var overrides = TrivyDbExportOverrideScope.Current; diff --git a/src/StellaOps.Concelier.Exporter.TrivyDb/TrivyDbMirrorBundleWriter.cs b/src/StellaOps.Concelier.Exporter.TrivyDb/TrivyDbMirrorBundleWriter.cs new file mode 100644 index 00000000..55d2d998 --- /dev/null +++ b/src/StellaOps.Concelier.Exporter.TrivyDb/TrivyDbMirrorBundleWriter.cs @@ -0,0 +1,392 @@ +using System; +using System.Collections.Generic; +using System.IO; +using System.Linq; +using System.Security.Cryptography; +using System.Text.Json; +using System.Text.Json.Serialization; +using System.Threading; +using System.Threading.Tasks; +using Microsoft.Extensions.Logging; +using StellaOps.Concelier.Exporter.Json; +using StellaOps.Concelier.Models; + +namespace StellaOps.Concelier.Exporter.TrivyDb; + +internal static class TrivyDbMirrorBundleWriter +{ + private const int SchemaVersion = 1; + private const string DefaultDirectoryName = "mirror"; + private const string MetadataFileName = "metadata.json"; + private const string DatabaseFileName = "db.tar.gz"; + private const string ManifestFileName = "manifest.json"; + + private static readonly JsonSerializerOptions SerializerOptions = new() + { + PropertyNamingPolicy = JsonNamingPolicy.CamelCase, + DefaultIgnoreCondition = JsonIgnoreCondition.WhenWritingNull, + WriteIndented = false, + }; + + public static async Task WriteAsync( + string layoutRoot, + JsonExportResult jsonResult, + TrivyDbExportOptions options, + TrivyDbExportPlan plan, + TrivyDbBuilderResult builderResult, + string reference, + string manifestDigest, + ReadOnlyMemory metadataBytes, + string metadataDigest, + long metadataLength, + string exporterVersion, + DateTimeOffset exportedAt, + ILogger logger, + CancellationToken cancellationToken) + { + if (options?.Mirror is null || !options.Mirror.Enabled || options.Mirror.Domains.Count == 0) + { + return; + } + + if (string.IsNullOrWhiteSpace(layoutRoot)) + { + throw new ArgumentException("Layout root must be provided.", nameof(layoutRoot)); + } + + if (builderResult is null) + { + throw new ArgumentNullException(nameof(builderResult)); + } + + if (jsonResult is null) + { + throw new ArgumentNullException(nameof(jsonResult)); + } + + var directoryName = string.IsNullOrWhiteSpace(options.Mirror.DirectoryName) + ? DefaultDirectoryName + : options.Mirror.DirectoryName.Trim(); + + if (directoryName.Length == 0) + { + directoryName = DefaultDirectoryName; + } + + var root = Path.Combine(layoutRoot, directoryName); + Directory.CreateDirectory(root); + + var timestamp = exportedAt.UtcDateTime; + TrySetDirectoryTimestamp(root, timestamp); + + var advisories = jsonResult.Advisories.IsDefaultOrEmpty + ? Array.Empty() + : jsonResult.Advisories + .OrderBy(static advisory => advisory.AdvisoryKey, StringComparer.Ordinal) + .ToArray(); + + var domains = new List(); + + foreach (var domainOption in options.Mirror.Domains) + { + cancellationToken.ThrowIfCancellationRequested(); + + if (domainOption is null) + { + logger.LogWarning("Encountered null Trivy mirror domain configuration; skipping."); + continue; + } + + var domainId = (domainOption.Id ?? string.Empty).Trim(); + if (domainId.Length == 0) + { + logger.LogWarning("Skipping Trivy mirror domain with empty id."); + continue; + } + + var displayName = string.IsNullOrWhiteSpace(domainOption.DisplayName) + ? domainId + : domainOption.DisplayName!.Trim(); + + var domainDirectory = Path.Combine(root, domainId); + Directory.CreateDirectory(domainDirectory); + TrySetDirectoryTimestamp(domainDirectory, timestamp); + + var metadataPath = Path.Combine(domainDirectory, MetadataFileName); + await WriteFileAsync(metadataPath, metadataBytes, timestamp, cancellationToken).ConfigureAwait(false); + var metadataRelativePath = ToRelativePath(layoutRoot, metadataPath); + + var databasePath = Path.Combine(domainDirectory, DatabaseFileName); + await CopyDatabaseAsync(builderResult.ArchivePath, databasePath, timestamp, cancellationToken).ConfigureAwait(false); + var databaseRelativePath = ToRelativePath(layoutRoot, databasePath); + + var sources = BuildSourceSummaries(advisories); + + var manifestDocument = new MirrorDomainManifestDocument( + SchemaVersion, + exportedAt, + exporterVersion, + reference, + manifestDigest, + options.TargetRepository, + domainId, + displayName, + plan.Mode.ToString().ToLowerInvariant(), + plan.BaseExportId, + plan.BaseManifestDigest, + plan.ResetBaseline, + new MirrorFileDescriptor(metadataRelativePath, metadataLength, metadataDigest), + new MirrorFileDescriptor(databaseRelativePath, builderResult.ArchiveLength, builderResult.ArchiveDigest), + sources); + + var manifestBytes = JsonSerializer.SerializeToUtf8Bytes(manifestDocument, SerializerOptions); + var manifestPath = Path.Combine(domainDirectory, ManifestFileName); + await WriteFileAsync(manifestPath, manifestBytes, timestamp, cancellationToken).ConfigureAwait(false); + var manifestRelativePath = ToRelativePath(layoutRoot, manifestPath); + var manifestDigestValue = ComputeDigest(manifestBytes); + + domains.Add(new MirrorIndexDomainEntry( + domainId, + displayName, + advisories.Length, + new MirrorFileDescriptor(manifestRelativePath, manifestBytes.LongLength, manifestDigestValue), + new MirrorFileDescriptor(metadataRelativePath, metadataLength, metadataDigest), + new MirrorFileDescriptor(databaseRelativePath, builderResult.ArchiveLength, builderResult.ArchiveDigest), + sources)); + } + + if (domains.Count == 0) + { + Directory.Delete(root, recursive: true); + return; + } + + domains.Sort(static (left, right) => string.CompareOrdinal(left.DomainId, right.DomainId)); + + var delta = plan.Mode == TrivyDbExportMode.Delta + ? new MirrorDeltaMetadata( + plan.ChangedFiles.Select(static file => new MirrorDeltaFile(file.Path, file.Digest)).ToArray(), + plan.RemovedPaths.ToArray()) + : null; + + var indexDocument = new MirrorIndexDocument( + SchemaVersion, + exportedAt, + exporterVersion, + options.TargetRepository, + reference, + manifestDigest, + plan.Mode.ToString().ToLowerInvariant(), + plan.BaseExportId, + plan.BaseManifestDigest, + plan.ResetBaseline, + delta, + domains); + + var indexBytes = JsonSerializer.SerializeToUtf8Bytes(indexDocument, SerializerOptions); + var indexPath = Path.Combine(root, "index.json"); + await WriteFileAsync(indexPath, indexBytes, timestamp, cancellationToken).ConfigureAwait(false); + + logger.LogInformation( + "Generated {DomainCount} Trivy DB mirror bundle(s) under {Directory}.", + domains.Count, + directoryName); + } + + private static IReadOnlyList BuildSourceSummaries(IReadOnlyList advisories) + { + if (advisories.Count == 0) + { + return Array.Empty(); + } + + var builders = new Dictionary(StringComparer.OrdinalIgnoreCase); + + foreach (var advisory in advisories) + { + var counted = new HashSet(StringComparer.OrdinalIgnoreCase); + + foreach (var provenance in advisory.Provenance) + { + if (string.IsNullOrWhiteSpace(provenance.Source)) + { + continue; + } + + var source = provenance.Source.Trim(); + if (!builders.TryGetValue(source, out var accumulator)) + { + accumulator = new SourceAccumulator(); + builders[source] = accumulator; + } + + accumulator.Record(provenance.RecordedAt); + if (counted.Add(source)) + { + accumulator.Increment(); + } + } + } + + var entries = builders + .Select(static pair => new TrivyMirrorSourceSummary( + pair.Key, + pair.Value.FirstRecordedAt, + pair.Value.LastRecordedAt, + pair.Value.Count)) + .OrderBy(static summary => summary.Source, StringComparer.Ordinal) + .ToArray(); + + return entries; + } + + private static async Task CopyDatabaseAsync( + string sourcePath, + string destinationPath, + DateTime timestamp, + CancellationToken cancellationToken) + { + Directory.CreateDirectory(Path.GetDirectoryName(destinationPath)!); + await using var source = new FileStream( + sourcePath, + FileMode.Open, + FileAccess.Read, + FileShare.Read, + bufferSize: 81920, + options: FileOptions.Asynchronous | FileOptions.SequentialScan); + await using var destination = new FileStream( + destinationPath, + FileMode.Create, + FileAccess.Write, + FileShare.None, + bufferSize: 81920, + options: FileOptions.Asynchronous | FileOptions.SequentialScan); + await source.CopyToAsync(destination, cancellationToken).ConfigureAwait(false); + await destination.FlushAsync(cancellationToken).ConfigureAwait(false); + File.SetLastWriteTimeUtc(destinationPath, timestamp); + } + + private static async Task WriteFileAsync( + string path, + ReadOnlyMemory bytes, + DateTime timestamp, + CancellationToken cancellationToken) + { + Directory.CreateDirectory(Path.GetDirectoryName(path)!); + await using var stream = new FileStream( + path, + FileMode.Create, + FileAccess.Write, + FileShare.None, + bufferSize: 81920, + options: FileOptions.Asynchronous | FileOptions.SequentialScan); + await stream.WriteAsync(bytes, cancellationToken).ConfigureAwait(false); + await stream.FlushAsync(cancellationToken).ConfigureAwait(false); + File.SetLastWriteTimeUtc(path, timestamp); + } + + private static string ToRelativePath(string root, string fullPath) + { + var relative = Path.GetRelativePath(root, fullPath); + var normalized = relative.Replace(Path.DirectorySeparatorChar, '/'); + return string.IsNullOrEmpty(normalized) ? "." : normalized; + } + + private static string ComputeDigest(ReadOnlySpan payload) + { + var hash = SHA256.HashData(payload); + var hex = Convert.ToHexString(hash).ToLowerInvariant(); + return $"sha256:{hex}"; + } + + private static void TrySetDirectoryTimestamp(string directory, DateTime timestamp) + { + try + { + Directory.SetLastWriteTimeUtc(directory, timestamp); + } + catch + { + // Best effort – ignore failures. + } + } + + private sealed record MirrorIndexDocument( + int SchemaVersion, + DateTimeOffset GeneratedAt, + string ExporterVersion, + string? TargetRepository, + string Reference, + string ManifestDigest, + string Mode, + string? BaseExportId, + string? BaseManifestDigest, + bool ResetBaseline, + MirrorDeltaMetadata? Delta, + IReadOnlyList Domains); + + private sealed record MirrorDeltaMetadata( + IReadOnlyList ChangedFiles, + IReadOnlyList RemovedPaths); + + private sealed record MirrorDeltaFile(string Path, string Digest); + + private sealed record MirrorIndexDomainEntry( + string DomainId, + string DisplayName, + int AdvisoryCount, + MirrorFileDescriptor Manifest, + MirrorFileDescriptor Metadata, + MirrorFileDescriptor Database, + IReadOnlyList Sources); + + private sealed record MirrorDomainManifestDocument( + int SchemaVersion, + DateTimeOffset GeneratedAt, + string ExporterVersion, + string Reference, + string ManifestDigest, + string? TargetRepository, + string DomainId, + string DisplayName, + string Mode, + string? BaseExportId, + string? BaseManifestDigest, + bool ResetBaseline, + MirrorFileDescriptor Metadata, + MirrorFileDescriptor Database, + IReadOnlyList Sources); + + private sealed record MirrorFileDescriptor(string Path, long SizeBytes, string Digest); + + private sealed record TrivyMirrorSourceSummary( + string Source, + DateTimeOffset? FirstRecordedAt, + DateTimeOffset? LastRecordedAt, + int AdvisoryCount); + + private sealed class SourceAccumulator + { + public DateTimeOffset? FirstRecordedAt { get; private set; } + + public DateTimeOffset? LastRecordedAt { get; private set; } + + public int Count { get; private set; } + + public void Record(DateTimeOffset recordedAt) + { + var utc = recordedAt.ToUniversalTime(); + if (FirstRecordedAt is null || utc < FirstRecordedAt.Value) + { + FirstRecordedAt = utc; + } + + if (LastRecordedAt is null || utc > LastRecordedAt.Value) + { + LastRecordedAt = utc; + } + } + + public void Increment() => Count++; + } +} diff --git a/src/StellaOps.Concelier.Merge.Tests/AdvisoryMergeServiceTests.cs b/src/StellaOps.Concelier.Merge.Tests/AdvisoryMergeServiceTests.cs index 2833104f..fc661f5f 100644 --- a/src/StellaOps.Concelier.Merge.Tests/AdvisoryMergeServiceTests.cs +++ b/src/StellaOps.Concelier.Merge.Tests/AdvisoryMergeServiceTests.cs @@ -1,8 +1,11 @@ using System.Collections.Concurrent; +using System.Linq; +using System.Threading.Tasks; using Microsoft.Extensions.Logging.Abstractions; using Microsoft.Extensions.Time.Testing; using MongoDB.Driver; using StellaOps.Concelier.Core; +using StellaOps.Concelier.Core.Events; using StellaOps.Concelier.Merge.Services; using StellaOps.Concelier.Models; using StellaOps.Concelier.Storage.Mongo.Advisories; @@ -35,7 +38,8 @@ public sealed class AdvisoryMergeServiceTests var precedenceMerger = new AdvisoryPrecedenceMerger(new AffectedPackagePrecedenceResolver(), timeProvider); var aliasResolver = new AliasGraphResolver(aliasStore); var canonicalMerger = new CanonicalMerger(timeProvider); - var service = new AdvisoryMergeService(aliasResolver, advisoryStore, precedenceMerger, writer, canonicalMerger, NullLogger.Instance); + var eventLog = new RecordingAdvisoryEventLog(); + var service = new AdvisoryMergeService(aliasResolver, advisoryStore, precedenceMerger, writer, canonicalMerger, eventLog, timeProvider, NullLogger.Instance); var result = await service.MergeAsync("GHSA-aaaa-bbbb-cccc", CancellationToken.None); @@ -52,6 +56,11 @@ public sealed class AdvisoryMergeServiceTests var summaryDecision = Assert.Single(mergeRecord!.FieldDecisions, decision => decision.Field == "summary"); Assert.Equal("osv", summaryDecision.SelectedSource); Assert.Equal("freshness_override", summaryDecision.DecisionReason); + + var appendRequest = eventLog.LastRequest; + Assert.NotNull(appendRequest); + Assert.Contains(appendRequest!.Statements, statement => string.Equals(statement.Advisory.AdvisoryKey, "CVE-2025-4242", StringComparison.OrdinalIgnoreCase)); + Assert.True(appendRequest.Conflicts is null || appendRequest.Conflicts.Count == 0); } private static Advisory CreateGhsaAdvisory() @@ -114,6 +123,23 @@ public sealed class AdvisoryMergeServiceTests provenance: new[] { provenance }); } + + private sealed class RecordingAdvisoryEventLog : IAdvisoryEventLog + { + public AdvisoryEventAppendRequest? LastRequest { get; private set; } + + public ValueTask AppendAsync(AdvisoryEventAppendRequest request, CancellationToken cancellationToken) + { + LastRequest = request; + return ValueTask.CompletedTask; + } + + public ValueTask ReplayAsync(string vulnerabilityKey, DateTimeOffset? asOf, CancellationToken cancellationToken) + { + throw new NotSupportedException(); + } + } + private sealed class FakeAliasStore : IAliasStore { private readonly ConcurrentDictionary> _records = new(StringComparer.OrdinalIgnoreCase); @@ -130,7 +156,9 @@ public sealed class AdvisoryMergeServiceTests } public Task ReplaceAsync(string advisoryKey, IEnumerable aliases, DateTimeOffset updatedAt, CancellationToken cancellationToken) - => Task.FromResult(new AliasUpsertResult(advisoryKey, Array.Empty())); + { + return Task.FromResult(new AliasUpsertResult(advisoryKey, Array.Empty())); + } public Task> GetByAliasAsync(string scheme, string value, CancellationToken cancellationToken) { @@ -206,6 +234,8 @@ public sealed class AdvisoryMergeServiceTests } public Task> GetRecentAsync(string advisoryKey, int limit, CancellationToken cancellationToken) - => Task.FromResult>(Array.Empty()); + { + return Task.FromResult>(Array.Empty()); + } } } diff --git a/src/StellaOps.Concelier.Merge.Tests/AdvisoryPrecedenceMergerTests.cs b/src/StellaOps.Concelier.Merge.Tests/AdvisoryPrecedenceMergerTests.cs index d336b9aa..f544a4eb 100644 --- a/src/StellaOps.Concelier.Merge.Tests/AdvisoryPrecedenceMergerTests.cs +++ b/src/StellaOps.Concelier.Merge.Tests/AdvisoryPrecedenceMergerTests.cs @@ -1,70 +1,70 @@ -using System; -using System.Collections.Generic; -using System.Linq; -using Microsoft.Extensions.Logging; -using Microsoft.Extensions.Time.Testing; -using StellaOps.Concelier.Merge.Options; -using StellaOps.Concelier.Merge.Services; -using StellaOps.Concelier.Models; - -namespace StellaOps.Concelier.Merge.Tests; - -public sealed class AdvisoryPrecedenceMergerTests -{ - [Fact] - public void Merge_PrefersVendorPrecedenceOverNvd() - { - var timeProvider = new FakeTimeProvider(new DateTimeOffset(2025, 1, 1, 0, 0, 0, TimeSpan.Zero)); - var merger = new AdvisoryPrecedenceMerger(new AffectedPackagePrecedenceResolver(), timeProvider); - using var metrics = new MetricCollector("StellaOps.Concelier.Merge"); - - var (redHat, nvd) = CreateVendorAndRegistryAdvisories(); - var expectedMergeTimestamp = timeProvider.GetUtcNow(); - - var merged = merger.Merge(new[] { nvd, redHat }); - - Assert.Equal("CVE-2025-1000", merged.AdvisoryKey); - Assert.Equal("Red Hat Security Advisory", merged.Title); - Assert.Equal("Vendor-confirmed impact on RHEL 9.", merged.Summary); - Assert.Equal("high", merged.Severity); - Assert.Equal(redHat.Published, merged.Published); - Assert.Equal(redHat.Modified, merged.Modified); - Assert.Contains("RHSA-2025:0001", merged.Aliases); - Assert.Contains("CVE-2025-1000", merged.Aliases); - - var package = Assert.Single(merged.AffectedPackages); - Assert.Equal("cpe:2.3:o:redhat:enterprise_linux:9:*:*:*:*:*:*:*", package.Identifier); - Assert.Empty(package.VersionRanges); // NVD range suppressed by vendor precedence - Assert.Contains(package.Statuses, status => status.Status == "known_affected"); - Assert.Contains(package.Provenance, provenance => provenance.Source == "redhat"); - Assert.Contains(package.Provenance, provenance => provenance.Source == "nvd"); - - Assert.Contains(merged.CvssMetrics, metric => metric.Provenance.Source == "redhat"); - Assert.Contains(merged.CvssMetrics, metric => metric.Provenance.Source == "nvd"); - - var mergeProvenance = merged.Provenance.Single(p => p.Source == "merge"); - Assert.Equal("precedence", mergeProvenance.Kind); - Assert.Equal(expectedMergeTimestamp, mergeProvenance.RecordedAt); - Assert.Contains("redhat", mergeProvenance.Value, StringComparison.OrdinalIgnoreCase); - Assert.Contains("nvd", mergeProvenance.Value, StringComparison.OrdinalIgnoreCase); - - var rangeMeasurement = Assert.Single(metrics.Measurements, measurement => measurement.Name == "concelier.merge.range_overrides"); - Assert.Equal(1, rangeMeasurement.Value); - Assert.Contains(rangeMeasurement.Tags, tag => string.Equals(tag.Key, "suppressed_source", StringComparison.Ordinal) && tag.Value?.ToString()?.Contains("nvd", StringComparison.OrdinalIgnoreCase) == true); - - var severityConflict = Assert.Single(metrics.Measurements, measurement => measurement.Name == "concelier.merge.conflicts"); - Assert.Equal(1, severityConflict.Value); - Assert.Contains(severityConflict.Tags, tag => string.Equals(tag.Key, "type", StringComparison.Ordinal) && string.Equals(tag.Value?.ToString(), "severity", StringComparison.OrdinalIgnoreCase)); - } - - [Fact] - public void Merge_KevOnlyTogglesExploitKnown() - { - var timeProvider = new FakeTimeProvider(new DateTimeOffset(2025, 2, 1, 0, 0, 0, TimeSpan.Zero)); - var merger = new AdvisoryPrecedenceMerger(new AffectedPackagePrecedenceResolver(), timeProvider); - - var nvdProvenance = new AdvisoryProvenance("nvd", "document", "https://nvd", timeProvider.GetUtcNow()); - var baseAdvisory = new Advisory( +using System; +using System.Collections.Generic; +using System.Linq; +using Microsoft.Extensions.Logging; +using Microsoft.Extensions.Time.Testing; +using StellaOps.Concelier.Merge.Options; +using StellaOps.Concelier.Merge.Services; +using StellaOps.Concelier.Models; + +namespace StellaOps.Concelier.Merge.Tests; + +public sealed class AdvisoryPrecedenceMergerTests +{ + [Fact] + public void Merge_PrefersVendorPrecedenceOverNvd() + { + var timeProvider = new FakeTimeProvider(new DateTimeOffset(2025, 1, 1, 0, 0, 0, TimeSpan.Zero)); + var merger = new AdvisoryPrecedenceMerger(new AffectedPackagePrecedenceResolver(), timeProvider); + using var metrics = new MetricCollector("StellaOps.Concelier.Merge"); + + var (redHat, nvd) = CreateVendorAndRegistryAdvisories(); + var expectedMergeTimestamp = timeProvider.GetUtcNow(); + + var merged = merger.Merge(new[] { nvd, redHat }).Advisory; + + Assert.Equal("CVE-2025-1000", merged.AdvisoryKey); + Assert.Equal("Red Hat Security Advisory", merged.Title); + Assert.Equal("Vendor-confirmed impact on RHEL 9.", merged.Summary); + Assert.Equal("high", merged.Severity); + Assert.Equal(redHat.Published, merged.Published); + Assert.Equal(redHat.Modified, merged.Modified); + Assert.Contains("RHSA-2025:0001", merged.Aliases); + Assert.Contains("CVE-2025-1000", merged.Aliases); + + var package = Assert.Single(merged.AffectedPackages); + Assert.Equal("cpe:2.3:o:redhat:enterprise_linux:9:*:*:*:*:*:*:*", package.Identifier); + Assert.Empty(package.VersionRanges); // NVD range suppressed by vendor precedence + Assert.Contains(package.Statuses, status => status.Status == "known_affected"); + Assert.Contains(package.Provenance, provenance => provenance.Source == "redhat"); + Assert.Contains(package.Provenance, provenance => provenance.Source == "nvd"); + + Assert.Contains(merged.CvssMetrics, metric => metric.Provenance.Source == "redhat"); + Assert.Contains(merged.CvssMetrics, metric => metric.Provenance.Source == "nvd"); + + var mergeProvenance = merged.Provenance.Single(p => p.Source == "merge"); + Assert.Equal("precedence", mergeProvenance.Kind); + Assert.Equal(expectedMergeTimestamp, mergeProvenance.RecordedAt); + Assert.Contains("redhat", mergeProvenance.Value, StringComparison.OrdinalIgnoreCase); + Assert.Contains("nvd", mergeProvenance.Value, StringComparison.OrdinalIgnoreCase); + + var rangeMeasurement = Assert.Single(metrics.Measurements, measurement => measurement.Name == "concelier.merge.range_overrides"); + Assert.Equal(1, rangeMeasurement.Value); + Assert.Contains(rangeMeasurement.Tags, tag => string.Equals(tag.Key, "suppressed_source", StringComparison.Ordinal) && tag.Value?.ToString()?.Contains("nvd", StringComparison.OrdinalIgnoreCase) == true); + + var severityConflict = Assert.Single(metrics.Measurements, measurement => measurement.Name == "concelier.merge.conflicts"); + Assert.Equal(1, severityConflict.Value); + Assert.Contains(severityConflict.Tags, tag => string.Equals(tag.Key, "type", StringComparison.Ordinal) && string.Equals(tag.Value?.ToString(), "severity", StringComparison.OrdinalIgnoreCase)); + } + + [Fact] + public void Merge_KevOnlyTogglesExploitKnown() + { + var timeProvider = new FakeTimeProvider(new DateTimeOffset(2025, 2, 1, 0, 0, 0, TimeSpan.Zero)); + var merger = new AdvisoryPrecedenceMerger(new AffectedPackagePrecedenceResolver(), timeProvider); + + var nvdProvenance = new AdvisoryProvenance("nvd", "document", "https://nvd", timeProvider.GetUtcNow()); + var baseAdvisory = new Advisory( "CVE-2025-2000", "CVE-2025-2000", "Base registry summary", @@ -76,52 +76,52 @@ public sealed class AdvisoryPrecedenceMergerTests aliases: new[] { "CVE-2025-2000" }, credits: Array.Empty(), references: Array.Empty(), - affectedPackages: new[] - { - new AffectedPackage( - AffectedPackageTypes.Cpe, - "cpe:2.3:a:example:product:2.0:*:*:*:*:*:*:*", - null, - new[] - { - new AffectedVersionRange( - "semver", - "2.0.0", - "2.0.5", - null, - "<2.0.5", - new AdvisoryProvenance("nvd", "cpe_match", "product", timeProvider.GetUtcNow())) - }, - Array.Empty(), - new[] { nvdProvenance }) - }, - cvssMetrics: Array.Empty(), - provenance: new[] { nvdProvenance }); - - var kevProvenance = new AdvisoryProvenance("kev", "catalog", "CVE-2025-2000", timeProvider.GetUtcNow()); - var kevAdvisory = new Advisory( - "CVE-2025-2000", - "Known Exploited Vulnerability", - summary: null, - language: null, - published: null, - modified: null, + affectedPackages: new[] + { + new AffectedPackage( + AffectedPackageTypes.Cpe, + "cpe:2.3:a:example:product:2.0:*:*:*:*:*:*:*", + null, + new[] + { + new AffectedVersionRange( + "semver", + "2.0.0", + "2.0.5", + null, + "<2.0.5", + new AdvisoryProvenance("nvd", "cpe_match", "product", timeProvider.GetUtcNow())) + }, + Array.Empty(), + new[] { nvdProvenance }) + }, + cvssMetrics: Array.Empty(), + provenance: new[] { nvdProvenance }); + + var kevProvenance = new AdvisoryProvenance("kev", "catalog", "CVE-2025-2000", timeProvider.GetUtcNow()); + var kevAdvisory = new Advisory( + "CVE-2025-2000", + "Known Exploited Vulnerability", + summary: null, + language: null, + published: null, + modified: null, severity: null, exploitKnown: true, aliases: new[] { "KEV-CVE-2025-2000" }, credits: Array.Empty(), references: Array.Empty(), - affectedPackages: Array.Empty(), - cvssMetrics: Array.Empty(), - provenance: new[] { kevProvenance }); - - var merged = merger.Merge(new[] { baseAdvisory, kevAdvisory }); - - Assert.True(merged.ExploitKnown); - Assert.Equal("medium", merged.Severity); // KEV must not override severity - Assert.Equal("Base registry summary", merged.Summary); - Assert.Contains("CVE-2025-2000", merged.Aliases); - Assert.Contains("KEV-CVE-2025-2000", merged.Aliases); + affectedPackages: Array.Empty(), + cvssMetrics: Array.Empty(), + provenance: new[] { kevProvenance }); + + var merged = merger.Merge(new[] { baseAdvisory, kevAdvisory }).Advisory; + + Assert.True(merged.ExploitKnown); + Assert.Equal("medium", merged.Severity); // KEV must not override severity + Assert.Equal("Base registry summary", merged.Summary); + Assert.Contains("CVE-2025-2000", merged.Aliases); + Assert.Contains("KEV-CVE-2025-2000", merged.Aliases); Assert.Contains(merged.Provenance, provenance => provenance.Source == "kev"); Assert.Contains(merged.Provenance, provenance => provenance.Source == "merge"); } @@ -212,7 +212,7 @@ public sealed class AdvisoryPrecedenceMergerTests cvssMetrics: Array.Empty(), provenance: new[] { new AdvisoryProvenance("osv", "document", "https://osv.dev/vulnerability/CVE-2025-9000", timeProvider.GetUtcNow(), new[] { ProvenanceFieldMasks.Advisory }) }); - var merged = merger.Merge(new[] { ghsa, osv }); + var merged = merger.Merge(new[] { ghsa, osv }).Advisory; Assert.Equal("CVE-2025-9000", merged.AdvisoryKey); Assert.Contains(merged.Credits, credit => @@ -311,7 +311,7 @@ public sealed class AdvisoryPrecedenceMergerTests cvssMetrics: Array.Empty(), provenance: new[] { acscDocumentProvenance }); - var merged = merger.Merge(new[] { acsc, vendor }); + var merged = merger.Merge(new[] { acsc, vendor }).Advisory; Assert.Equal("critical", merged.Severity); // ACSC must not override vendor severity Assert.Equal("Vendor-confirmed exploit.", merged.Summary); @@ -450,7 +450,7 @@ public sealed class AdvisoryPrecedenceMergerTests new AdvisoryProvenance("nvd", "document", "https://nvd.nist.gov/vuln/detail/CVE-2025-7000", now), }); - var merged = merger.Merge(new[] { nvdAdvisory, ghsaAdvisory }); + var merged = merger.Merge(new[] { nvdAdvisory, ghsaAdvisory }).Advisory; Assert.Equal(2, merged.AffectedPackages.Length); var normalizedPackage = Assert.Single(merged.AffectedPackages, pkg => pkg.Identifier == "pkg:npm/example"); @@ -474,72 +474,72 @@ public sealed class AdvisoryPrecedenceMergerTests [Fact] public void Merge_RespectsConfiguredPrecedenceOverrides() - { - var timeProvider = new FakeTimeProvider(new DateTimeOffset(2025, 3, 1, 0, 0, 0, TimeSpan.Zero)); - var options = new AdvisoryPrecedenceOptions - { - Ranks = new Dictionary(StringComparer.OrdinalIgnoreCase) - { - ["nvd"] = 0, - ["redhat"] = 5, - } - }; - - var logger = new TestLogger(); - using var metrics = new MetricCollector("StellaOps.Concelier.Merge"); - - var merger = new AdvisoryPrecedenceMerger( - new AffectedPackagePrecedenceResolver(), - options, - timeProvider, - logger); - - var (redHat, nvd) = CreateVendorAndRegistryAdvisories(); - var merged = merger.Merge(new[] { redHat, nvd }); - - Assert.Equal("CVE-2025-1000", merged.AdvisoryKey); - Assert.Equal("CVE-2025-1000", merged.Title); // NVD preferred - Assert.Equal("NVD summary", merged.Summary); - Assert.Equal("medium", merged.Severity); - - var package = Assert.Single(merged.AffectedPackages); - Assert.NotEmpty(package.VersionRanges); // Vendor range no longer overrides - Assert.Contains(package.Provenance, provenance => provenance.Source == "nvd"); - Assert.Contains(package.Provenance, provenance => provenance.Source == "redhat"); - - var overrideMeasurement = Assert.Single(metrics.Measurements, m => m.Name == "concelier.merge.overrides"); - Assert.Equal(1, overrideMeasurement.Value); - Assert.Contains(overrideMeasurement.Tags, tag => tag.Key == "primary_source" && string.Equals(tag.Value?.ToString(), "nvd", StringComparison.OrdinalIgnoreCase)); - Assert.Contains(overrideMeasurement.Tags, tag => tag.Key == "suppressed_source" && tag.Value?.ToString()?.Contains("redhat", StringComparison.OrdinalIgnoreCase) == true); - - Assert.DoesNotContain(metrics.Measurements, measurement => measurement.Name == "concelier.merge.range_overrides"); - - var conflictMeasurement = Assert.Single(metrics.Measurements, measurement => measurement.Name == "concelier.merge.conflicts"); - Assert.Equal(1, conflictMeasurement.Value); - Assert.Contains(conflictMeasurement.Tags, tag => tag.Key == "type" && string.Equals(tag.Value?.ToString(), "severity", StringComparison.OrdinalIgnoreCase)); - Assert.Contains(conflictMeasurement.Tags, tag => tag.Key == "reason" && string.Equals(tag.Value?.ToString(), "mismatch", StringComparison.OrdinalIgnoreCase)); - - var logEntry = Assert.Single(logger.Entries, entry => entry.EventId.Name == "AdvisoryOverride"); - Assert.Equal(LogLevel.Information, logEntry.Level); - Assert.NotNull(logEntry.StructuredState); - Assert.Contains(logEntry.StructuredState!, kvp => - (string.Equals(kvp.Key, "Override", StringComparison.Ordinal) || - string.Equals(kvp.Key, "@Override", StringComparison.Ordinal)) && - kvp.Value is not null); - } - - private static (Advisory Vendor, Advisory Registry) CreateVendorAndRegistryAdvisories() - { - var redHatPublished = new DateTimeOffset(2025, 1, 10, 0, 0, 0, TimeSpan.Zero); - var redHatModified = redHatPublished.AddDays(1); - var redHatProvenance = new AdvisoryProvenance("redhat", "advisory", "RHSA-2025:0001", redHatModified); - var redHatPackage = new AffectedPackage( - AffectedPackageTypes.Cpe, - "cpe:2.3:o:redhat:enterprise_linux:9:*:*:*:*:*:*:*", - "rhel-9", - Array.Empty(), - new[] { new AffectedPackageStatus("known_affected", redHatProvenance) }, - new[] { redHatProvenance }); + { + var timeProvider = new FakeTimeProvider(new DateTimeOffset(2025, 3, 1, 0, 0, 0, TimeSpan.Zero)); + var options = new AdvisoryPrecedenceOptions + { + Ranks = new Dictionary(StringComparer.OrdinalIgnoreCase) + { + ["nvd"] = 0, + ["redhat"] = 5, + } + }; + + var logger = new TestLogger(); + using var metrics = new MetricCollector("StellaOps.Concelier.Merge"); + + var merger = new AdvisoryPrecedenceMerger( + new AffectedPackagePrecedenceResolver(), + options, + timeProvider, + logger); + + var (redHat, nvd) = CreateVendorAndRegistryAdvisories(); + var merged = merger.Merge(new[] { redHat, nvd }).Advisory; + + Assert.Equal("CVE-2025-1000", merged.AdvisoryKey); + Assert.Equal("CVE-2025-1000", merged.Title); // NVD preferred + Assert.Equal("NVD summary", merged.Summary); + Assert.Equal("medium", merged.Severity); + + var package = Assert.Single(merged.AffectedPackages); + Assert.NotEmpty(package.VersionRanges); // Vendor range no longer overrides + Assert.Contains(package.Provenance, provenance => provenance.Source == "nvd"); + Assert.Contains(package.Provenance, provenance => provenance.Source == "redhat"); + + var overrideMeasurement = Assert.Single(metrics.Measurements, m => m.Name == "concelier.merge.overrides"); + Assert.Equal(1, overrideMeasurement.Value); + Assert.Contains(overrideMeasurement.Tags, tag => tag.Key == "primary_source" && string.Equals(tag.Value?.ToString(), "nvd", StringComparison.OrdinalIgnoreCase)); + Assert.Contains(overrideMeasurement.Tags, tag => tag.Key == "suppressed_source" && tag.Value?.ToString()?.Contains("redhat", StringComparison.OrdinalIgnoreCase) == true); + + Assert.DoesNotContain(metrics.Measurements, measurement => measurement.Name == "concelier.merge.range_overrides"); + + var conflictMeasurement = Assert.Single(metrics.Measurements, measurement => measurement.Name == "concelier.merge.conflicts"); + Assert.Equal(1, conflictMeasurement.Value); + Assert.Contains(conflictMeasurement.Tags, tag => tag.Key == "type" && string.Equals(tag.Value?.ToString(), "severity", StringComparison.OrdinalIgnoreCase)); + Assert.Contains(conflictMeasurement.Tags, tag => tag.Key == "reason" && string.Equals(tag.Value?.ToString(), "mismatch", StringComparison.OrdinalIgnoreCase)); + + var logEntry = Assert.Single(logger.Entries, entry => entry.EventId.Name == "AdvisoryOverride"); + Assert.Equal(LogLevel.Information, logEntry.Level); + Assert.NotNull(logEntry.StructuredState); + Assert.Contains(logEntry.StructuredState!, kvp => + (string.Equals(kvp.Key, "Override", StringComparison.Ordinal) || + string.Equals(kvp.Key, "@Override", StringComparison.Ordinal)) && + kvp.Value is not null); + } + + private static (Advisory Vendor, Advisory Registry) CreateVendorAndRegistryAdvisories() + { + var redHatPublished = new DateTimeOffset(2025, 1, 10, 0, 0, 0, TimeSpan.Zero); + var redHatModified = redHatPublished.AddDays(1); + var redHatProvenance = new AdvisoryProvenance("redhat", "advisory", "RHSA-2025:0001", redHatModified); + var redHatPackage = new AffectedPackage( + AffectedPackageTypes.Cpe, + "cpe:2.3:o:redhat:enterprise_linux:9:*:*:*:*:*:*:*", + "rhel-9", + Array.Empty(), + new[] { new AffectedPackageStatus("known_affected", redHatProvenance) }, + new[] { redHatProvenance }); var redHat = new Advisory( "CVE-2025-1000", "Red Hat Security Advisory", @@ -554,43 +554,43 @@ public sealed class AdvisoryPrecedenceMergerTests references: new[] { new AdvisoryReference( - "https://access.redhat.com/errata/RHSA-2025:0001", - "advisory", - "redhat", - "Red Hat errata", - redHatProvenance) - }, - affectedPackages: new[] { redHatPackage }, - cvssMetrics: new[] - { - new CvssMetric( - "3.1", - "CVSS:3.1/AV:N/AC:L/PR:N/UI:N/S:U/C:H/I:H/A:H", - 9.8, - "critical", - new AdvisoryProvenance("redhat", "cvss", "RHSA-2025:0001", redHatModified)) - }, - provenance: new[] { redHatProvenance }); - - var nvdPublished = new DateTimeOffset(2025, 1, 5, 0, 0, 0, TimeSpan.Zero); - var nvdModified = nvdPublished.AddDays(2); - var nvdProvenance = new AdvisoryProvenance("nvd", "document", "https://nvd.nist.gov/vuln/detail/CVE-2025-1000", nvdModified); - var nvdPackage = new AffectedPackage( - AffectedPackageTypes.Cpe, - "cpe:2.3:o:redhat:enterprise_linux:9:*:*:*:*:*:*:*", - "rhel-9", - new[] - { - new AffectedVersionRange( - "cpe", - null, - null, - null, - "<=9.0", - new AdvisoryProvenance("nvd", "cpe_match", "RHEL", nvdModified)) - }, - Array.Empty(), - new[] { nvdProvenance }); + "https://access.redhat.com/errata/RHSA-2025:0001", + "advisory", + "redhat", + "Red Hat errata", + redHatProvenance) + }, + affectedPackages: new[] { redHatPackage }, + cvssMetrics: new[] + { + new CvssMetric( + "3.1", + "CVSS:3.1/AV:N/AC:L/PR:N/UI:N/S:U/C:H/I:H/A:H", + 9.8, + "critical", + new AdvisoryProvenance("redhat", "cvss", "RHSA-2025:0001", redHatModified)) + }, + provenance: new[] { redHatProvenance }); + + var nvdPublished = new DateTimeOffset(2025, 1, 5, 0, 0, 0, TimeSpan.Zero); + var nvdModified = nvdPublished.AddDays(2); + var nvdProvenance = new AdvisoryProvenance("nvd", "document", "https://nvd.nist.gov/vuln/detail/CVE-2025-1000", nvdModified); + var nvdPackage = new AffectedPackage( + AffectedPackageTypes.Cpe, + "cpe:2.3:o:redhat:enterprise_linux:9:*:*:*:*:*:*:*", + "rhel-9", + new[] + { + new AffectedVersionRange( + "cpe", + null, + null, + null, + "<=9.0", + new AdvisoryProvenance("nvd", "cpe_match", "RHEL", nvdModified)) + }, + Array.Empty(), + new[] { nvdProvenance }); var nvd = new Advisory( "CVE-2025-1000", "CVE-2025-1000", @@ -605,24 +605,24 @@ public sealed class AdvisoryPrecedenceMergerTests references: new[] { new AdvisoryReference( - "https://nvd.nist.gov/vuln/detail/CVE-2025-1000", - "advisory", - "nvd", - "NVD advisory", - nvdProvenance) - }, - affectedPackages: new[] { nvdPackage }, - cvssMetrics: new[] - { - new CvssMetric( - "3.1", - "CVSS:3.1/AV:N/AC:L/PR:L/UI:R/S:U/C:H/I:H/A:N", - 6.8, - "medium", - new AdvisoryProvenance("nvd", "cvss", "CVE-2025-1000", nvdModified)) - }, - provenance: new[] { nvdProvenance }); - - return (redHat, nvd); - } -} + "https://nvd.nist.gov/vuln/detail/CVE-2025-1000", + "advisory", + "nvd", + "NVD advisory", + nvdProvenance) + }, + affectedPackages: new[] { nvdPackage }, + cvssMetrics: new[] + { + new CvssMetric( + "3.1", + "CVSS:3.1/AV:N/AC:L/PR:L/UI:R/S:U/C:H/I:H/A:N", + 6.8, + "medium", + new AdvisoryProvenance("nvd", "cvss", "CVE-2025-1000", nvdModified)) + }, + provenance: new[] { nvdProvenance }); + + return (redHat, nvd); + } +} diff --git a/src/StellaOps.Concelier.Merge.Tests/MergePrecedenceIntegrationTests.cs b/src/StellaOps.Concelier.Merge.Tests/MergePrecedenceIntegrationTests.cs index 4f3e42e9..4a343cc8 100644 --- a/src/StellaOps.Concelier.Merge.Tests/MergePrecedenceIntegrationTests.cs +++ b/src/StellaOps.Concelier.Merge.Tests/MergePrecedenceIntegrationTests.cs @@ -1,231 +1,234 @@ -using System; -using System.Linq; -using System.Threading; -using System.Threading.Tasks; -using Microsoft.Extensions.Logging.Abstractions; -using Microsoft.Extensions.Time.Testing; -using MongoDB.Driver; -using StellaOps.Concelier.Merge.Services; -using StellaOps.Concelier.Models; -using StellaOps.Concelier.Storage.Mongo; -using StellaOps.Concelier.Storage.Mongo.MergeEvents; -using StellaOps.Concelier.Testing; - -namespace StellaOps.Concelier.Merge.Tests; - -[Collection("mongo-fixture")] -public sealed class MergePrecedenceIntegrationTests : IAsyncLifetime -{ - private readonly MongoIntegrationFixture _fixture; - private MergeEventStore? _mergeEventStore; - private MergeEventWriter? _mergeEventWriter; - private AdvisoryPrecedenceMerger? _merger; - private FakeTimeProvider? _timeProvider; - - public MergePrecedenceIntegrationTests(MongoIntegrationFixture fixture) - { - _fixture = fixture; - } - - [Fact] - public async Task MergePipeline_PsirtOverridesNvd_AndKevOnlyTogglesExploitKnown() - { - await EnsureInitializedAsync(); - - var merger = _merger!; - var writer = _mergeEventWriter!; - var store = _mergeEventStore!; - var timeProvider = _timeProvider!; - - var expectedTimestamp = timeProvider.GetUtcNow(); - - var nvd = CreateNvdBaseline(); - var vendor = CreateVendorOverride(); - var kev = CreateKevSignal(); - - var merged = merger.Merge(new[] { nvd, vendor, kev }); - - Assert.Equal("CVE-2025-1000", merged.AdvisoryKey); - Assert.Equal("Vendor Security Advisory", merged.Title); - Assert.Equal("Critical impact on supported platforms.", merged.Summary); - Assert.Equal("critical", merged.Severity); - Assert.True(merged.ExploitKnown); - - var affected = Assert.Single(merged.AffectedPackages); - Assert.Empty(affected.VersionRanges); - Assert.Contains(affected.Statuses, status => status.Status == "known_affected" && status.Provenance.Source == "vendor"); - - var mergeProvenance = Assert.Single(merged.Provenance, p => p.Source == "merge"); - Assert.Equal("precedence", mergeProvenance.Kind); - Assert.Equal(expectedTimestamp, mergeProvenance.RecordedAt); - Assert.Contains("vendor", mergeProvenance.Value, StringComparison.OrdinalIgnoreCase); - Assert.Contains("kev", mergeProvenance.Value, StringComparison.OrdinalIgnoreCase); - - var inputDocumentIds = new[] { Guid.NewGuid(), Guid.NewGuid(), Guid.NewGuid() }; +using System; +using System.Linq; +using System.Threading; +using System.Threading.Tasks; +using Microsoft.Extensions.Logging.Abstractions; +using Microsoft.Extensions.Time.Testing; +using MongoDB.Driver; +using StellaOps.Concelier.Merge.Services; +using StellaOps.Concelier.Models; +using StellaOps.Concelier.Storage.Mongo; +using StellaOps.Concelier.Storage.Mongo.MergeEvents; +using StellaOps.Concelier.Testing; + +namespace StellaOps.Concelier.Merge.Tests; + +[Collection("mongo-fixture")] +public sealed class MergePrecedenceIntegrationTests : IAsyncLifetime +{ + private readonly MongoIntegrationFixture _fixture; + private MergeEventStore? _mergeEventStore; + private MergeEventWriter? _mergeEventWriter; + private AdvisoryPrecedenceMerger? _merger; + private FakeTimeProvider? _timeProvider; + + public MergePrecedenceIntegrationTests(MongoIntegrationFixture fixture) + { + _fixture = fixture; + } + + [Fact] + public async Task MergePipeline_PsirtOverridesNvd_AndKevOnlyTogglesExploitKnown() + { + await EnsureInitializedAsync(); + + var merger = _merger!; + var writer = _mergeEventWriter!; + var store = _mergeEventStore!; + var timeProvider = _timeProvider!; + + var expectedTimestamp = timeProvider.GetUtcNow(); + + var nvd = CreateNvdBaseline(); + var vendor = CreateVendorOverride(); + var kev = CreateKevSignal(); + + var merged = merger.Merge(new[] { nvd, vendor, kev }).Advisory; + + Assert.Equal("CVE-2025-1000", merged.AdvisoryKey); + Assert.Equal("Vendor Security Advisory", merged.Title); + Assert.Equal("Critical impact on supported platforms.", merged.Summary); + Assert.Equal("critical", merged.Severity); + Assert.True(merged.ExploitKnown); + + var affected = Assert.Single(merged.AffectedPackages); + Assert.Empty(affected.VersionRanges); + Assert.Contains(affected.Statuses, status => status.Status == "known_affected" && status.Provenance.Source == "vendor"); + + var mergeProvenance = Assert.Single(merged.Provenance, p => p.Source == "merge"); + Assert.Equal("precedence", mergeProvenance.Kind); + Assert.Equal(expectedTimestamp, mergeProvenance.RecordedAt); + Assert.Contains("vendor", mergeProvenance.Value, StringComparison.OrdinalIgnoreCase); + Assert.Contains("kev", mergeProvenance.Value, StringComparison.OrdinalIgnoreCase); + + var inputDocumentIds = new[] { Guid.NewGuid(), Guid.NewGuid(), Guid.NewGuid() }; var record = await writer.AppendAsync(merged.AdvisoryKey, nvd, merged, inputDocumentIds, Array.Empty(), CancellationToken.None); - - Assert.Equal(expectedTimestamp, record.MergedAt); - Assert.Equal(inputDocumentIds, record.InputDocumentIds); - Assert.NotEqual(record.BeforeHash, record.AfterHash); - - var records = await store.GetRecentAsync(merged.AdvisoryKey, 5, CancellationToken.None); - var persisted = Assert.Single(records); - Assert.Equal(record.Id, persisted.Id); - Assert.Equal(merged.AdvisoryKey, persisted.AdvisoryKey); - Assert.True(persisted.AfterHash.Length > 0); - Assert.True(persisted.BeforeHash.Length > 0); - } - - [Fact] - public async Task MergePipeline_IsDeterministicAcrossRuns() - { - await EnsureInitializedAsync(); - - var merger = _merger!; - var calculator = new CanonicalHashCalculator(); - - var first = merger.Merge(new[] { CreateNvdBaseline(), CreateVendorOverride() }); - var second = merger.Merge(new[] { CreateNvdBaseline(), CreateVendorOverride() }); - - var firstHash = calculator.ComputeHash(first); - var secondHash = calculator.ComputeHash(second); - - Assert.Equal(firstHash, secondHash); - Assert.Equal(first.AdvisoryKey, second.AdvisoryKey); - Assert.Equal(first.Aliases.Length, second.Aliases.Length); - Assert.True(first.Aliases.SequenceEqual(second.Aliases)); - } - - public async Task InitializeAsync() - { - _timeProvider = new FakeTimeProvider(new DateTimeOffset(2025, 3, 1, 0, 0, 0, TimeSpan.Zero)) - { - AutoAdvanceAmount = TimeSpan.Zero, - }; - _merger = new AdvisoryPrecedenceMerger(new AffectedPackagePrecedenceResolver(), _timeProvider); - _mergeEventStore = new MergeEventStore(_fixture.Database, NullLogger.Instance); - _mergeEventWriter = new MergeEventWriter(_mergeEventStore, new CanonicalHashCalculator(), _timeProvider, NullLogger.Instance); - await DropMergeCollectionAsync(); - } - - public Task DisposeAsync() => Task.CompletedTask; - - private async Task EnsureInitializedAsync() - { - if (_mergeEventWriter is null) - { - await InitializeAsync(); - } - } - - private async Task DropMergeCollectionAsync() - { - try - { - await _fixture.Database.DropCollectionAsync(MongoStorageDefaults.Collections.MergeEvent); - } - catch (MongoCommandException ex) when (ex.CodeName == "NamespaceNotFound" || ex.Message.Contains("ns not found", StringComparison.OrdinalIgnoreCase)) - { - // Collection has not been created yet – safe to ignore. - } - } - - private static Advisory CreateNvdBaseline() - { - var provenance = new AdvisoryProvenance("nvd", "document", "https://nvd.nist.gov/vuln/detail/CVE-2025-1000", DateTimeOffset.Parse("2025-02-10T00:00:00Z")); - return new Advisory( - "CVE-2025-1000", - "CVE-2025-1000", - "Baseline description from NVD.", - "en", - DateTimeOffset.Parse("2025-02-05T00:00:00Z"), - DateTimeOffset.Parse("2025-02-10T12:00:00Z"), - "medium", - exploitKnown: false, - aliases: new[] { "CVE-2025-1000" }, - references: new[] - { - new AdvisoryReference("https://nvd.nist.gov/vuln/detail/CVE-2025-1000", "advisory", "nvd", "NVD reference", provenance), - }, - affectedPackages: new[] - { - new AffectedPackage( - AffectedPackageTypes.Cpe, - "cpe:2.3:o:vendor:product:1.0:*:*:*:*:*:*:*", - "vendor-os", - new[] - { - new AffectedVersionRange( - rangeKind: "cpe", - introducedVersion: null, - fixedVersion: null, - lastAffectedVersion: null, - rangeExpression: "<=1.0", - provenance: provenance) - }, - Array.Empty(), - new[] { provenance }) - }, - cvssMetrics: new[] - { - new CvssMetric("3.1", "CVSS:3.1/AV:N/AC:L/PR:N/UI:N/S:U/C:H/I:H/A:H", 9.8, "critical", provenance) - }, - provenance: new[] { provenance }); - } - - private static Advisory CreateVendorOverride() - { - var provenance = new AdvisoryProvenance("vendor", "psirt", "VSA-2025-1000", DateTimeOffset.Parse("2025-02-11T00:00:00Z")); - return new Advisory( - "CVE-2025-1000", - "Vendor Security Advisory", - "Critical impact on supported platforms.", - "en", - DateTimeOffset.Parse("2025-02-06T00:00:00Z"), - DateTimeOffset.Parse("2025-02-11T06:00:00Z"), - "critical", - exploitKnown: false, - aliases: new[] { "CVE-2025-1000", "VSA-2025-1000" }, - references: new[] - { - new AdvisoryReference("https://vendor.example/advisories/VSA-2025-1000", "advisory", "vendor", "Vendor advisory", provenance), - }, - affectedPackages: new[] - { - new AffectedPackage( - AffectedPackageTypes.Cpe, - "cpe:2.3:o:vendor:product:1.0:*:*:*:*:*:*:*", - "vendor-os", - Array.Empty(), - new[] - { - new AffectedPackageStatus("known_affected", provenance) - }, - new[] { provenance }) - }, - cvssMetrics: new[] - { - new CvssMetric("3.1", "CVSS:3.1/AV:N/AC:L/PR:N/UI:N/S:C/C:H/I:H/A:H", 10.0, "critical", provenance) - }, - provenance: new[] { provenance }); - } - - private static Advisory CreateKevSignal() - { - var provenance = new AdvisoryProvenance("kev", "catalog", "CVE-2025-1000", DateTimeOffset.Parse("2025-02-12T00:00:00Z")); - return new Advisory( - "CVE-2025-1000", - "Known Exploited Vulnerability", - null, - null, - published: null, - modified: null, - severity: null, - exploitKnown: true, - aliases: new[] { "KEV-CVE-2025-1000" }, - references: Array.Empty(), - affectedPackages: Array.Empty(), - cvssMetrics: Array.Empty(), - provenance: new[] { provenance }); - } -} + + Assert.Equal(expectedTimestamp, record.MergedAt); + Assert.Equal(inputDocumentIds, record.InputDocumentIds); + Assert.NotEqual(record.BeforeHash, record.AfterHash); + + var records = await store.GetRecentAsync(merged.AdvisoryKey, 5, CancellationToken.None); + var persisted = Assert.Single(records); + Assert.Equal(record.Id, persisted.Id); + Assert.Equal(merged.AdvisoryKey, persisted.AdvisoryKey); + Assert.True(persisted.AfterHash.Length > 0); + Assert.True(persisted.BeforeHash.Length > 0); + } + + [Fact] + public async Task MergePipeline_IsDeterministicAcrossRuns() + { + await EnsureInitializedAsync(); + + var merger = _merger!; + var calculator = new CanonicalHashCalculator(); + + var firstResult = merger.Merge(new[] { CreateNvdBaseline(), CreateVendorOverride() }); + var secondResult = merger.Merge(new[] { CreateNvdBaseline(), CreateVendorOverride() }); + + var first = firstResult.Advisory; + var second = secondResult.Advisory; + + var firstHash = calculator.ComputeHash(first); + var secondHash = calculator.ComputeHash(second); + + Assert.Equal(firstHash, secondHash); + Assert.Equal(first.AdvisoryKey, second.AdvisoryKey); + Assert.Equal(first.Aliases.Length, second.Aliases.Length); + Assert.True(first.Aliases.SequenceEqual(second.Aliases)); + } + + public async Task InitializeAsync() + { + _timeProvider = new FakeTimeProvider(new DateTimeOffset(2025, 3, 1, 0, 0, 0, TimeSpan.Zero)) + { + AutoAdvanceAmount = TimeSpan.Zero, + }; + _merger = new AdvisoryPrecedenceMerger(new AffectedPackagePrecedenceResolver(), _timeProvider); + _mergeEventStore = new MergeEventStore(_fixture.Database, NullLogger.Instance); + _mergeEventWriter = new MergeEventWriter(_mergeEventStore, new CanonicalHashCalculator(), _timeProvider, NullLogger.Instance); + await DropMergeCollectionAsync(); + } + + public Task DisposeAsync() => Task.CompletedTask; + + private async Task EnsureInitializedAsync() + { + if (_mergeEventWriter is null) + { + await InitializeAsync(); + } + } + + private async Task DropMergeCollectionAsync() + { + try + { + await _fixture.Database.DropCollectionAsync(MongoStorageDefaults.Collections.MergeEvent); + } + catch (MongoCommandException ex) when (ex.CodeName == "NamespaceNotFound" || ex.Message.Contains("ns not found", StringComparison.OrdinalIgnoreCase)) + { + // Collection has not been created yet – safe to ignore. + } + } + + private static Advisory CreateNvdBaseline() + { + var provenance = new AdvisoryProvenance("nvd", "document", "https://nvd.nist.gov/vuln/detail/CVE-2025-1000", DateTimeOffset.Parse("2025-02-10T00:00:00Z")); + return new Advisory( + "CVE-2025-1000", + "CVE-2025-1000", + "Baseline description from NVD.", + "en", + DateTimeOffset.Parse("2025-02-05T00:00:00Z"), + DateTimeOffset.Parse("2025-02-10T12:00:00Z"), + "medium", + exploitKnown: false, + aliases: new[] { "CVE-2025-1000" }, + references: new[] + { + new AdvisoryReference("https://nvd.nist.gov/vuln/detail/CVE-2025-1000", "advisory", "nvd", "NVD reference", provenance), + }, + affectedPackages: new[] + { + new AffectedPackage( + AffectedPackageTypes.Cpe, + "cpe:2.3:o:vendor:product:1.0:*:*:*:*:*:*:*", + "vendor-os", + new[] + { + new AffectedVersionRange( + rangeKind: "cpe", + introducedVersion: null, + fixedVersion: null, + lastAffectedVersion: null, + rangeExpression: "<=1.0", + provenance: provenance) + }, + Array.Empty(), + new[] { provenance }) + }, + cvssMetrics: new[] + { + new CvssMetric("3.1", "CVSS:3.1/AV:N/AC:L/PR:N/UI:N/S:U/C:H/I:H/A:H", 9.8, "critical", provenance) + }, + provenance: new[] { provenance }); + } + + private static Advisory CreateVendorOverride() + { + var provenance = new AdvisoryProvenance("vendor", "psirt", "VSA-2025-1000", DateTimeOffset.Parse("2025-02-11T00:00:00Z")); + return new Advisory( + "CVE-2025-1000", + "Vendor Security Advisory", + "Critical impact on supported platforms.", + "en", + DateTimeOffset.Parse("2025-02-06T00:00:00Z"), + DateTimeOffset.Parse("2025-02-11T06:00:00Z"), + "critical", + exploitKnown: false, + aliases: new[] { "CVE-2025-1000", "VSA-2025-1000" }, + references: new[] + { + new AdvisoryReference("https://vendor.example/advisories/VSA-2025-1000", "advisory", "vendor", "Vendor advisory", provenance), + }, + affectedPackages: new[] + { + new AffectedPackage( + AffectedPackageTypes.Cpe, + "cpe:2.3:o:vendor:product:1.0:*:*:*:*:*:*:*", + "vendor-os", + Array.Empty(), + new[] + { + new AffectedPackageStatus("known_affected", provenance) + }, + new[] { provenance }) + }, + cvssMetrics: new[] + { + new CvssMetric("3.1", "CVSS:3.1/AV:N/AC:L/PR:N/UI:N/S:C/C:H/I:H/A:H", 10.0, "critical", provenance) + }, + provenance: new[] { provenance }); + } + + private static Advisory CreateKevSignal() + { + var provenance = new AdvisoryProvenance("kev", "catalog", "CVE-2025-1000", DateTimeOffset.Parse("2025-02-12T00:00:00Z")); + return new Advisory( + "CVE-2025-1000", + "Known Exploited Vulnerability", + null, + null, + published: null, + modified: null, + severity: null, + exploitKnown: true, + aliases: new[] { "KEV-CVE-2025-1000" }, + references: Array.Empty(), + affectedPackages: Array.Empty(), + cvssMetrics: Array.Empty(), + provenance: new[] { provenance }); + } +} diff --git a/src/StellaOps.Concelier.Merge/Services/AdvisoryMergeService.cs b/src/StellaOps.Concelier.Merge/Services/AdvisoryMergeService.cs index e634986c..deb7104e 100644 --- a/src/StellaOps.Concelier.Merge/Services/AdvisoryMergeService.cs +++ b/src/StellaOps.Concelier.Merge/Services/AdvisoryMergeService.cs @@ -7,21 +7,23 @@ using System.Threading; using System.Threading.Tasks; using Microsoft.Extensions.Logging; using StellaOps.Concelier.Core; +using StellaOps.Concelier.Core.Events; using StellaOps.Concelier.Models; using StellaOps.Concelier.Storage.Mongo.Advisories; using StellaOps.Concelier.Storage.Mongo.Aliases; using StellaOps.Concelier.Storage.Mongo.MergeEvents; - -namespace StellaOps.Concelier.Merge.Services; - -public sealed class AdvisoryMergeService -{ - private static readonly Meter MergeMeter = new("StellaOps.Concelier.Merge"); - private static readonly Counter AliasCollisionCounter = MergeMeter.CreateCounter( - "concelier.merge.identity_conflicts", - unit: "count", - description: "Number of alias collisions detected during merge."); - +using System.Text.Json; + +namespace StellaOps.Concelier.Merge.Services; + +public sealed class AdvisoryMergeService +{ + private static readonly Meter MergeMeter = new("StellaOps.Concelier.Merge"); + private static readonly Counter AliasCollisionCounter = MergeMeter.CreateCounter( + "concelier.merge.identity_conflicts", + unit: "count", + description: "Number of alias collisions detected during merge."); + private static readonly string[] PreferredAliasSchemes = { AliasSchemes.Cve, @@ -34,6 +36,8 @@ public sealed class AdvisoryMergeService private readonly IAdvisoryStore _advisoryStore; private readonly AdvisoryPrecedenceMerger _precedenceMerger; private readonly MergeEventWriter _mergeEventWriter; + private readonly IAdvisoryEventLog _eventLog; + private readonly TimeProvider _timeProvider; private readonly CanonicalMerger _canonicalMerger; private readonly ILogger _logger; @@ -43,6 +47,8 @@ public sealed class AdvisoryMergeService AdvisoryPrecedenceMerger precedenceMerger, MergeEventWriter mergeEventWriter, CanonicalMerger canonicalMerger, + IAdvisoryEventLog eventLog, + TimeProvider timeProvider, ILogger logger) { _aliasResolver = aliasResolver ?? throw new ArgumentNullException(nameof(aliasResolver)); @@ -50,92 +56,222 @@ public sealed class AdvisoryMergeService _precedenceMerger = precedenceMerger ?? throw new ArgumentNullException(nameof(precedenceMerger)); _mergeEventWriter = mergeEventWriter ?? throw new ArgumentNullException(nameof(mergeEventWriter)); _canonicalMerger = canonicalMerger ?? throw new ArgumentNullException(nameof(canonicalMerger)); + _eventLog = eventLog ?? throw new ArgumentNullException(nameof(eventLog)); + _timeProvider = timeProvider ?? throw new ArgumentNullException(nameof(timeProvider)); _logger = logger ?? throw new ArgumentNullException(nameof(logger)); } - - public async Task MergeAsync(string seedAdvisoryKey, CancellationToken cancellationToken) - { - ArgumentException.ThrowIfNullOrWhiteSpace(seedAdvisoryKey); - - var component = await _aliasResolver.BuildComponentAsync(seedAdvisoryKey, cancellationToken).ConfigureAwait(false); - var inputs = new List(); - - foreach (var advisoryKey in component.AdvisoryKeys) - { - cancellationToken.ThrowIfCancellationRequested(); - var advisory = await _advisoryStore.FindAsync(advisoryKey, cancellationToken).ConfigureAwait(false); - if (advisory is not null) - { - inputs.Add(advisory); - } - } - - if (inputs.Count == 0) - { - _logger.LogWarning("Alias component seeded by {Seed} contains no persisted advisories", seedAdvisoryKey); - return AdvisoryMergeResult.Empty(seedAdvisoryKey, component); - } - + + public async Task MergeAsync(string seedAdvisoryKey, CancellationToken cancellationToken) + { + ArgumentException.ThrowIfNullOrWhiteSpace(seedAdvisoryKey); + + var component = await _aliasResolver.BuildComponentAsync(seedAdvisoryKey, cancellationToken).ConfigureAwait(false); + var inputs = new List(); + + foreach (var advisoryKey in component.AdvisoryKeys) + { + cancellationToken.ThrowIfCancellationRequested(); + var advisory = await _advisoryStore.FindAsync(advisoryKey, cancellationToken).ConfigureAwait(false); + if (advisory is not null) + { + inputs.Add(advisory); + } + } + + if (inputs.Count == 0) + { + _logger.LogWarning("Alias component seeded by {Seed} contains no persisted advisories", seedAdvisoryKey); + return AdvisoryMergeResult.Empty(seedAdvisoryKey, component); + } + var canonicalKey = SelectCanonicalKey(component) ?? seedAdvisoryKey; var canonicalMerge = ApplyCanonicalMergeIfNeeded(canonicalKey, inputs); var before = await _advisoryStore.FindAsync(canonicalKey, cancellationToken).ConfigureAwait(false); var normalizedInputs = NormalizeInputs(inputs, canonicalKey).ToList(); - Advisory? merged; + PrecedenceMergeResult precedenceResult; try { - merged = _precedenceMerger.Merge(normalizedInputs); - } - catch (Exception ex) - { - _logger.LogError(ex, "Failed to merge alias component seeded by {Seed}", seedAdvisoryKey); - throw; - } - - if (component.Collisions.Count > 0) - { - foreach (var collision in component.Collisions) - { - var tags = new KeyValuePair[] - { - new("scheme", collision.Scheme ?? string.Empty), - new("alias_value", collision.Value ?? string.Empty), - new("advisory_count", collision.AdvisoryKeys.Count), - }; - - AliasCollisionCounter.Add(1, tags); - - _logger.LogInformation( - "Alias collision {Scheme}:{Value} involves advisories {Advisories}", - collision.Scheme, - collision.Value, - string.Join(", ", collision.AdvisoryKeys)); - } - } - - if (merged is not null) - { - await _advisoryStore.UpsertAsync(merged, cancellationToken).ConfigureAwait(false); - await _mergeEventWriter.AppendAsync( - canonicalKey, - before, - merged, - Array.Empty(), - ConvertFieldDecisions(canonicalMerge?.Decisions), - cancellationToken).ConfigureAwait(false); + precedenceResult = _precedenceMerger.Merge(normalizedInputs); + } + catch (Exception ex) + { + _logger.LogError(ex, "Failed to merge alias component seeded by {Seed}", seedAdvisoryKey); + throw; } + var merged = precedenceResult.Advisory; + var conflictDetails = precedenceResult.Conflicts; + + if (component.Collisions.Count > 0) + { + foreach (var collision in component.Collisions) + { + var tags = new KeyValuePair[] + { + new("scheme", collision.Scheme ?? string.Empty), + new("alias_value", collision.Value ?? string.Empty), + new("advisory_count", collision.AdvisoryKeys.Count), + }; + + AliasCollisionCounter.Add(1, tags); + + _logger.LogInformation( + "Alias collision {Scheme}:{Value} involves advisories {Advisories}", + collision.Scheme, + collision.Value, + string.Join(", ", collision.AdvisoryKeys)); + } + } + + await _advisoryStore.UpsertAsync(merged, cancellationToken).ConfigureAwait(false); + await _mergeEventWriter.AppendAsync( + canonicalKey, + before, + merged, + Array.Empty(), + ConvertFieldDecisions(canonicalMerge?.Decisions), + cancellationToken).ConfigureAwait(false); + + await AppendEventLogAsync(canonicalKey, normalizedInputs, merged, conflictDetails, cancellationToken).ConfigureAwait(false); + return new AdvisoryMergeResult(seedAdvisoryKey, canonicalKey, component, inputs, before, merged); - } - - private static IEnumerable NormalizeInputs(IEnumerable advisories, string canonicalKey) - { - foreach (var advisory in advisories) - { - yield return CloneWithKey(advisory, canonicalKey); - } - } - + } + + private async Task AppendEventLogAsync( + string vulnerabilityKey, + IReadOnlyList inputs, + Advisory merged, + IReadOnlyList conflicts, + CancellationToken cancellationToken) + { + var recordedAt = _timeProvider.GetUtcNow(); + var statements = new List(inputs.Count + 1); + var statementIds = new Dictionary(ReferenceEqualityComparer.Instance); + + foreach (var advisory in inputs) + { + var statementId = Guid.NewGuid(); + statementIds[advisory] = statementId; + statements.Add(new AdvisoryStatementInput( + vulnerabilityKey, + advisory, + DetermineAsOf(advisory, recordedAt), + InputDocumentIds: Array.Empty(), + StatementId: statementId, + AdvisoryKey: advisory.AdvisoryKey)); + } + + var canonicalStatementId = Guid.NewGuid(); + statementIds[merged] = canonicalStatementId; + statements.Add(new AdvisoryStatementInput( + vulnerabilityKey, + merged, + recordedAt, + InputDocumentIds: Array.Empty(), + StatementId: canonicalStatementId, + AdvisoryKey: merged.AdvisoryKey)); + + var conflictInputs = BuildConflictInputs(conflicts, vulnerabilityKey, statementIds, canonicalStatementId, recordedAt); + + if (statements.Count == 0 && conflictInputs.Count == 0) + { + return; + } + + var request = new AdvisoryEventAppendRequest(statements, conflictInputs.Count > 0 ? conflictInputs : null); + + try + { + await _eventLog.AppendAsync(request, cancellationToken).ConfigureAwait(false); + } + finally + { + foreach (var conflict in conflictInputs) + { + conflict.Details.Dispose(); + } + } + } + + private static DateTimeOffset DetermineAsOf(Advisory advisory, DateTimeOffset fallback) + { + return (advisory.Modified ?? advisory.Published ?? fallback).ToUniversalTime(); + } + + private static List BuildConflictInputs( + IReadOnlyList conflicts, + string vulnerabilityKey, + IReadOnlyDictionary statementIds, + Guid canonicalStatementId, + DateTimeOffset recordedAt) + { + if (conflicts.Count == 0) + { + return new List(0); + } + + var inputs = new List(conflicts.Count); + + foreach (var detail in conflicts) + { + if (!statementIds.TryGetValue(detail.Suppressed, out var suppressedId)) + { + continue; + } + + var related = new List { canonicalStatementId, suppressedId }; + if (statementIds.TryGetValue(detail.Primary, out var primaryId)) + { + if (!related.Contains(primaryId)) + { + related.Add(primaryId); + } + } + + var payload = new ConflictDetailPayload( + detail.ConflictType, + detail.Reason, + detail.PrimarySources, + detail.PrimaryRank, + detail.SuppressedSources, + detail.SuppressedRank, + detail.PrimaryValue, + detail.SuppressedValue); + + var json = CanonicalJsonSerializer.Serialize(payload); + var document = JsonDocument.Parse(json); + var asOf = (detail.Primary.Modified ?? detail.Suppressed.Modified ?? recordedAt).ToUniversalTime(); + + inputs.Add(new AdvisoryConflictInput( + vulnerabilityKey, + document, + asOf, + related, + ConflictId: null)); + } + + return inputs; + } + + private sealed record ConflictDetailPayload( + string Type, + string Reason, + IReadOnlyList PrimarySources, + int PrimaryRank, + IReadOnlyList SuppressedSources, + int SuppressedRank, + string? PrimaryValue, + string? SuppressedValue); + + private static IEnumerable NormalizeInputs(IEnumerable advisories, string canonicalKey) + { + foreach (var advisory in advisories) + { + yield return CloneWithKey(advisory, canonicalKey); + } + } + private static Advisory CloneWithKey(Advisory source, string advisoryKey) => new( advisoryKey, @@ -248,47 +384,47 @@ public sealed class AdvisoryMergeService public const string Nvd = "nvd"; public const string Osv = "osv"; } - - private static string? SelectCanonicalKey(AliasComponent component) - { - foreach (var scheme in PreferredAliasSchemes) - { - var alias = component.AliasMap.Values - .SelectMany(static aliases => aliases) - .FirstOrDefault(record => string.Equals(record.Scheme, scheme, StringComparison.OrdinalIgnoreCase)); - if (!string.IsNullOrWhiteSpace(alias?.Value)) - { - return alias.Value; - } - } - - if (component.AliasMap.TryGetValue(component.SeedAdvisoryKey, out var seedAliases)) - { - var primary = seedAliases.FirstOrDefault(record => string.Equals(record.Scheme, AliasStoreConstants.PrimaryScheme, StringComparison.OrdinalIgnoreCase)); - if (!string.IsNullOrWhiteSpace(primary?.Value)) - { - return primary.Value; - } - } - - var firstAlias = component.AliasMap.Values.SelectMany(static aliases => aliases).FirstOrDefault(); - if (!string.IsNullOrWhiteSpace(firstAlias?.Value)) - { - return firstAlias.Value; - } - - return component.SeedAdvisoryKey; - } -} - -public sealed record AdvisoryMergeResult( - string SeedAdvisoryKey, - string CanonicalAdvisoryKey, - AliasComponent Component, - IReadOnlyList Inputs, - Advisory? Previous, - Advisory? Merged) -{ - public static AdvisoryMergeResult Empty(string seed, AliasComponent component) - => new(seed, seed, component, Array.Empty(), null, null); -} + + private static string? SelectCanonicalKey(AliasComponent component) + { + foreach (var scheme in PreferredAliasSchemes) + { + var alias = component.AliasMap.Values + .SelectMany(static aliases => aliases) + .FirstOrDefault(record => string.Equals(record.Scheme, scheme, StringComparison.OrdinalIgnoreCase)); + if (!string.IsNullOrWhiteSpace(alias?.Value)) + { + return alias.Value; + } + } + + if (component.AliasMap.TryGetValue(component.SeedAdvisoryKey, out var seedAliases)) + { + var primary = seedAliases.FirstOrDefault(record => string.Equals(record.Scheme, AliasStoreConstants.PrimaryScheme, StringComparison.OrdinalIgnoreCase)); + if (!string.IsNullOrWhiteSpace(primary?.Value)) + { + return primary.Value; + } + } + + var firstAlias = component.AliasMap.Values.SelectMany(static aliases => aliases).FirstOrDefault(); + if (!string.IsNullOrWhiteSpace(firstAlias?.Value)) + { + return firstAlias.Value; + } + + return component.SeedAdvisoryKey; + } +} + +public sealed record AdvisoryMergeResult( + string SeedAdvisoryKey, + string CanonicalAdvisoryKey, + AliasComponent Component, + IReadOnlyList Inputs, + Advisory? Previous, + Advisory? Merged) +{ + public static AdvisoryMergeResult Empty(string seed, AliasComponent component) + => new(seed, seed, component, Array.Empty(), null, null); +} diff --git a/src/StellaOps.Concelier.Merge/Services/AdvisoryPrecedenceMerger.cs b/src/StellaOps.Concelier.Merge/Services/AdvisoryPrecedenceMerger.cs index eab88e5a..22c72979 100644 --- a/src/StellaOps.Concelier.Merge/Services/AdvisoryPrecedenceMerger.cs +++ b/src/StellaOps.Concelier.Merge/Services/AdvisoryPrecedenceMerger.cs @@ -111,7 +111,7 @@ public sealed class AdvisoryPrecedenceMerger _logger = logger ?? NullLogger.Instance; } - public Advisory Merge(IEnumerable advisories) + public PrecedenceMergeResult Merge(IEnumerable advisories) { if (advisories is null) { @@ -193,11 +193,12 @@ public sealed class AdvisoryPrecedenceMerger var exploitKnown = ordered.Any(entry => entry.Advisory.ExploitKnown); - LogOverrides(advisoryKey, ordered); - LogPackageOverrides(advisoryKey, packageResult.Overrides); - RecordFieldConflicts(advisoryKey, ordered); - - return new Advisory( + LogOverrides(advisoryKey, ordered); + LogPackageOverrides(advisoryKey, packageResult.Overrides); + var conflicts = new List(); + RecordFieldConflicts(advisoryKey, ordered, conflicts); + + var merged = new Advisory( advisoryKey, title, summary, @@ -212,6 +213,8 @@ public sealed class AdvisoryPrecedenceMerger affectedPackages, cvssMetrics, provenance); + + return new PrecedenceMergeResult(merged, conflicts); } private static void RecordNormalizedRuleMetrics(IReadOnlyList packages) @@ -379,7 +382,7 @@ public sealed class AdvisoryPrecedenceMerger } } - private void RecordFieldConflicts(string advisoryKey, IReadOnlyList ordered) + private void RecordFieldConflicts(string advisoryKey, IReadOnlyList ordered, List conflicts) { if (ordered.Count <= 1) { @@ -396,42 +399,45 @@ public sealed class AdvisoryPrecedenceMerger if (!string.IsNullOrEmpty(candidateSeverity)) { - var reason = string.IsNullOrEmpty(primarySeverity) ? "primary_missing" : "mismatch"; - if (string.IsNullOrEmpty(primarySeverity) || !string.Equals(primarySeverity, candidateSeverity, StringComparison.OrdinalIgnoreCase)) - { - RecordConflict( - advisoryKey, - "severity", - reason, - primary, - candidate, - primarySeverity ?? "(none)", - candidateSeverity); - } - } - - if (candidate.Rank == primary.Rank) - { - RecordConflict( - advisoryKey, - "precedence_tie", - "equal_rank", - primary, - candidate, - primary.Rank.ToString(CultureInfo.InvariantCulture), - candidate.Rank.ToString(CultureInfo.InvariantCulture)); - } - } - } - - private void RecordConflict( - string advisoryKey, - string conflictType, - string reason, - AdvisoryEntry primary, - AdvisoryEntry suppressed, - string? primaryValue, - string? suppressedValue) + var reason = string.IsNullOrEmpty(primarySeverity) ? "primary_missing" : "mismatch"; + if (string.IsNullOrEmpty(primarySeverity) || !string.Equals(primarySeverity, candidateSeverity, StringComparison.OrdinalIgnoreCase)) + { + RecordConflict( + advisoryKey, + "severity", + reason, + primary, + candidate, + primarySeverity ?? "(none)", + candidateSeverity, + conflicts); + } + } + + if (candidate.Rank == primary.Rank) + { + RecordConflict( + advisoryKey, + "precedence_tie", + "equal_rank", + primary, + candidate, + primary.Rank.ToString(CultureInfo.InvariantCulture), + candidate.Rank.ToString(CultureInfo.InvariantCulture), + conflicts); + } + } + } + + private void RecordConflict( + string advisoryKey, + string conflictType, + string reason, + AdvisoryEntry primary, + AdvisoryEntry suppressed, + string? primaryValue, + string? suppressedValue, + List conflicts) { var tags = new KeyValuePair[] { @@ -445,18 +451,30 @@ public sealed class AdvisoryPrecedenceMerger ConflictCounter.Add(1, tags); - var audit = new MergeFieldConflictAudit( - advisoryKey, - conflictType, - reason, - primary.Sources, - primary.Rank, - suppressed.Sources, - suppressed.Rank, - primaryValue, - suppressedValue); - - ConflictLogged(_logger, audit, null); + var audit = new MergeFieldConflictAudit( + advisoryKey, + conflictType, + reason, + primary.Sources, + primary.Rank, + suppressed.Sources, + suppressed.Rank, + primaryValue, + suppressedValue); + + ConflictLogged(_logger, audit, null); + + conflicts.Add(new MergeConflictDetail( + primary.Advisory, + suppressed.Advisory, + conflictType, + reason, + primary.Sources.ToArray(), + primary.Rank, + suppressed.Sources.ToArray(), + suppressed.Rank, + primaryValue, + suppressedValue)); } private readonly record struct AdvisoryEntry(Advisory Advisory, int Rank) diff --git a/src/StellaOps.Concelier.Merge/Services/MergeConflictDetail.cs b/src/StellaOps.Concelier.Merge/Services/MergeConflictDetail.cs new file mode 100644 index 00000000..f9741a4b --- /dev/null +++ b/src/StellaOps.Concelier.Merge/Services/MergeConflictDetail.cs @@ -0,0 +1,17 @@ +using System; +using System.Collections.Generic; +using StellaOps.Concelier.Models; + +namespace StellaOps.Concelier.Merge.Services; + +public sealed record MergeConflictDetail( + Advisory Primary, + Advisory Suppressed, + string ConflictType, + string Reason, + IReadOnlyList PrimarySources, + int PrimaryRank, + IReadOnlyList SuppressedSources, + int SuppressedRank, + string? PrimaryValue, + string? SuppressedValue); diff --git a/src/StellaOps.Concelier.Merge/Services/PrecedenceMergeResult.cs b/src/StellaOps.Concelier.Merge/Services/PrecedenceMergeResult.cs new file mode 100644 index 00000000..87229a3c --- /dev/null +++ b/src/StellaOps.Concelier.Merge/Services/PrecedenceMergeResult.cs @@ -0,0 +1,8 @@ +using System.Collections.Generic; +using StellaOps.Concelier.Models; + +namespace StellaOps.Concelier.Merge.Services; + +public sealed record PrecedenceMergeResult( + Advisory Advisory, + IReadOnlyList Conflicts); diff --git a/src/StellaOps.Concelier.Merge/TASKS.md b/src/StellaOps.Concelier.Merge/TASKS.md index b63748ff..1a43620c 100644 --- a/src/StellaOps.Concelier.Merge/TASKS.md +++ b/src/StellaOps.Concelier.Merge/TASKS.md @@ -1,12 +1,12 @@ -# TASKS -| Task | Owner(s) | Depends on | Notes | -|---|---|---|---| -|Identity graph and alias resolver|BE-Merge|Models, Storage.Mongo|DONE – `AdvisoryIdentityResolver` builds alias-driven clusters with canonical key selection + unit coverage.| -|Precedence policy engine|BE-Merge|Architecture|**DONE** – precedence defaults enforced by `AdvisoryPrecedenceMerger`/`AdvisoryPrecedenceDefaults` with distro/PSIRT overriding registry feeds and CERT/KEV enrichers.| -|NEVRA comparer plus tests|BE-Merge (Distro WG)|Source.Distro fixtures|DONE – Added Nevra parser/comparer with tilde-aware rpm ordering and unit coverage.| -|Debian EVR comparer plus tests|BE-Merge (Distro WG)|Debian fixtures|DONE – DebianEvr comparer mirrors dpkg ordering with tilde/epoch handling and unit coverage.| -|SemVer range resolver plus tests|BE-Merge (OSS WG)|OSV/GHSA fixtures|DONE – SemanticVersionRangeResolver covers introduced/fixed/lastAffected semantics with SemVer ordering tests.| -|Canonical hash and merge_event writer|BE-Merge|Models, Storage.Mongo|DONE – Hash calculator + MergeEventWriter compute canonical SHA-256 digests and persist merge events.| +# TASKS +| Task | Owner(s) | Depends on | Notes | +|---|---|---|---| +|Identity graph and alias resolver|BE-Merge|Models, Storage.Mongo|DONE – `AdvisoryIdentityResolver` builds alias-driven clusters with canonical key selection + unit coverage.| +|Precedence policy engine|BE-Merge|Architecture|**DONE** – precedence defaults enforced by `AdvisoryPrecedenceMerger`/`AdvisoryPrecedenceDefaults` with distro/PSIRT overriding registry feeds and CERT/KEV enrichers.| +|NEVRA comparer plus tests|BE-Merge (Distro WG)|Source.Distro fixtures|DONE – Added Nevra parser/comparer with tilde-aware rpm ordering and unit coverage.| +|Debian EVR comparer plus tests|BE-Merge (Distro WG)|Debian fixtures|DONE – DebianEvr comparer mirrors dpkg ordering with tilde/epoch handling and unit coverage.| +|SemVer range resolver plus tests|BE-Merge (OSS WG)|OSV/GHSA fixtures|DONE – SemanticVersionRangeResolver covers introduced/fixed/lastAffected semantics with SemVer ordering tests.| +|Canonical hash and merge_event writer|BE-Merge|Models, Storage.Mongo|DONE – Hash calculator + MergeEventWriter compute canonical SHA-256 digests and persist merge events.| |Conflict detection and metrics|BE-Merge|Core|**DONE** – merge meters emit override/conflict counters and structured audits (`AdvisoryPrecedenceMerger`).| |FEEDMERGE-ENGINE-04-001 GHSA/NVD/OSV conflict rules|BE-Merge|Core, Storage.Mongo|DONE – `AdvisoryMergeService` applies `CanonicalMerger` output before precedence merge, replacing source advisories with the canonical transcript. **Coordination:** connector fixture owners should surface canonical deltas to Merge QA before regression sign-off.| |FEEDMERGE-ENGINE-04-002 Override metrics instrumentation|BE-Merge|Observability|DONE – merge events persist `MergeFieldDecision` records enabling analytics on precedence/freshness decisions. **Next:** hand off metrics schema to Ops for dashboard wiring.| @@ -15,7 +15,7 @@ |FEEDMERGE-QA-04-001 End-to-end conflict regression suite|QA|Merge|DONE – `AdvisoryMergeServiceTests.MergeAsync_AppliesCanonicalRulesAndPersistsDecisions` exercises GHSA/NVD/OSV conflict path and merge-event analytics. **Reminder:** QA to sync with connector teams once new fixture triples land.| |Override audit logging|BE-Merge|Observability|DONE – override audits now emit structured logs plus bounded-tag metrics suitable for prod telemetry.| |Configurable precedence table|BE-Merge|Architecture|DONE – precedence options bind via concelier:merge:precedence:ranks with docs/tests covering operator workflow.| -|Range primitives backlog|BE-Merge|Connector WGs|**DOING** – Coordinate remaining connectors (`Acsc`, `Cccs`, `CertBund`, `CertCc`, `Cve`, `Ghsa`, `Ics.Cisa`, `Kisa`, `Ru.Bdu`, `Ru.Nkcki`, `Vndr.Apple`, `Vndr.Cisco`, `Vndr.Msrc`) to emit canonical RangePrimitives with provenance tags; track progress/fixtures here.
2025-10-11: Storage alignment notes + sample normalized rule JSON now captured in `RANGE_PRIMITIVES_COORDINATION.md` (see “Storage alignment quick reference”).
2025-10-11 18:45Z: GHSA normalized rules landed; OSV connector picked up next for rollout.
2025-10-11 21:10Z: `docs/dev/merge_semver_playbook.md` Section 8 now documents the persisted Mongo projection (SemVer + NEVRA) for connector reviewers.
2025-10-11 21:30Z: Added `docs/dev/normalized_versions_rollout.md` dashboard to centralize connector status and upcoming milestones.
2025-10-11 21:55Z: Merge now emits `concelier.merge.normalized_rules*` counters and unions connector-provided normalized arrays; see new test coverage in `AdvisoryPrecedenceMergerTests.Merge_RecordsNormalizedRuleMetrics`.
2025-10-12 17:05Z: CVE + KEV normalized rule verification complete; OSV parity fixtures revalidated—downstream parity/monitoring tasks may proceed.| +|Range primitives backlog|BE-Merge|Connector WGs|**DOING** – Coordinate remaining connectors (`Acsc`, `Cccs`, `CertBund`, `CertCc`, `Cve`, `Ghsa`, `Ics.Cisa`, `Kisa`, `Ru.Bdu`, `Ru.Nkcki`, `Vndr.Apple`, `Vndr.Cisco`, `Vndr.Msrc`) to emit canonical RangePrimitives with provenance tags; track progress/fixtures here.
2025-10-11: Storage alignment notes + sample normalized rule JSON now captured in `RANGE_PRIMITIVES_COORDINATION.md` (see “Storage alignment quick reference”).
2025-10-11 18:45Z: GHSA normalized rules landed; OSV connector picked up next for rollout.
2025-10-11 21:10Z: `docs/dev/merge_semver_playbook.md` Section 8 now documents the persisted Mongo projection (SemVer + NEVRA) for connector reviewers.
2025-10-11 21:30Z: Added `docs/dev/normalized_versions_rollout.md` dashboard to centralize connector status and upcoming milestones.
2025-10-11 21:55Z: Merge now emits `concelier.merge.normalized_rules*` counters and unions connector-provided normalized arrays; see new test coverage in `AdvisoryPrecedenceMergerTests.Merge_RecordsNormalizedRuleMetrics`.
2025-10-12 17:05Z: CVE + KEV normalized rule verification complete; OSV parity fixtures revalidated—downstream parity/monitoring tasks may proceed.
2025-10-19 14:35Z: Prerequisites reviewed (none outstanding); FEEDMERGE-COORD-02-900 remains in DOING with connector follow-ups unchanged.
2025-10-19 15:25Z: Refreshed `RANGE_PRIMITIVES_COORDINATION.md` matrix + added targeted follow-ups (Cccs, CertBund, ICS-CISA, Kisa, Vndr.Cisco) with delivery dates 2025-10-21 → 2025-10-25; monitoring merge counters for regression.| |Merge pipeline parity for new advisory fields|BE-Merge|Models, Core|DONE (2025-10-15) – merge service now surfaces description/CWE/canonical metric decisions with updated metrics/tests.| |Connector coordination for new advisory fields|Connector Leads, BE-Merge|Models, Core|**DONE (2025-10-15)** – GHSA, NVD, and OSV connectors now emit advisory descriptions, CWE weaknesses, and canonical metric ids. Fixtures refreshed (GHSA connector regression suite, `conflict-nvd.canonical.json`, OSV parity snapshots) and completion recorded in coordination log.| -|FEEDMERGE-ENGINE-07-001 Conflict sets & explainers|BE-Merge|FEEDSTORAGE-DATA-07-001|TODO – Persist conflict sets referencing advisory statements, output rule/explainer payloads with replay hashes, and add integration tests covering deterministic `asOf` evaluations.| +|FEEDMERGE-ENGINE-07-001 Conflict sets & explainers|BE-Merge|FEEDSTORAGE-DATA-07-001|**DOING (2025-10-19)** – Merge now captures canonical advisory statements + prepares conflict payload scaffolding (statement hashes, deterministic JSON, tests). Next: surface conflict explainers and replay APIs for Core/WebService before marking DONE.| diff --git a/src/StellaOps.Concelier.Storage.Mongo.Tests/AdvisoryConflictStoreTests.cs b/src/StellaOps.Concelier.Storage.Mongo.Tests/AdvisoryConflictStoreTests.cs new file mode 100644 index 00000000..40ad2267 --- /dev/null +++ b/src/StellaOps.Concelier.Storage.Mongo.Tests/AdvisoryConflictStoreTests.cs @@ -0,0 +1,82 @@ +using System; +using System.Linq; +using System.Threading; +using System.Threading.Tasks; +using MongoDB.Bson; +using MongoDB.Driver; +using StellaOps.Concelier.Storage.Mongo; +using StellaOps.Concelier.Storage.Mongo.Conflicts; +using StellaOps.Concelier.Testing; +using Xunit; + +namespace StellaOps.Concelier.Storage.Mongo.Tests; + +[Collection("mongo-fixture")] +public sealed class AdvisoryConflictStoreTests +{ + private readonly IMongoDatabase _database; + + public AdvisoryConflictStoreTests(MongoIntegrationFixture fixture) + { + _database = fixture.Database ?? throw new ArgumentNullException(nameof(fixture.Database)); + } + + [Fact] + public async Task InsertAndRetrieve_PersistsConflicts() + { + var store = new AdvisoryConflictStore(_database); + var vulnerabilityKey = $"CVE-{Guid.NewGuid():N}"; + var baseTime = DateTimeOffset.UtcNow; + var statementIds = new[] { Guid.NewGuid(), Guid.NewGuid() }; + + var conflict = new AdvisoryConflictRecord( + Guid.NewGuid(), + vulnerabilityKey, + new byte[] { 0x10, 0x20 }, + baseTime, + baseTime.AddSeconds(30), + statementIds, + new BsonDocument("explanation", "first-pass")); + + await store.InsertAsync(new[] { conflict }, CancellationToken.None); + + var results = await store.GetConflictsAsync(vulnerabilityKey, null, CancellationToken.None); + + Assert.Single(results); + Assert.Equal(conflict.Id, results[0].Id); + Assert.Equal(statementIds, results[0].StatementIds); + } + + [Fact] + public async Task GetConflicts_AsOfFilters() + { + var store = new AdvisoryConflictStore(_database); + var vulnerabilityKey = $"CVE-{Guid.NewGuid():N}"; + var baseTime = DateTimeOffset.UtcNow; + + var earlyConflict = new AdvisoryConflictRecord( + Guid.NewGuid(), + vulnerabilityKey, + new byte[] { 0x01 }, + baseTime, + baseTime.AddSeconds(10), + new[] { Guid.NewGuid() }, + new BsonDocument("stage", "early")); + + var lateConflict = new AdvisoryConflictRecord( + Guid.NewGuid(), + vulnerabilityKey, + new byte[] { 0x02 }, + baseTime.AddMinutes(10), + baseTime.AddMinutes(10).AddSeconds(15), + new[] { Guid.NewGuid() }, + new BsonDocument("stage", "late")); + + await store.InsertAsync(new[] { earlyConflict, lateConflict }, CancellationToken.None); + + var results = await store.GetConflictsAsync(vulnerabilityKey, baseTime.AddMinutes(1), CancellationToken.None); + + Assert.Single(results); + Assert.Equal("early", results[0].Details["stage"].AsString); + } +} diff --git a/src/StellaOps.Concelier.Storage.Mongo.Tests/AdvisoryStatementStoreTests.cs b/src/StellaOps.Concelier.Storage.Mongo.Tests/AdvisoryStatementStoreTests.cs new file mode 100644 index 00000000..c529f4fb --- /dev/null +++ b/src/StellaOps.Concelier.Storage.Mongo.Tests/AdvisoryStatementStoreTests.cs @@ -0,0 +1,96 @@ +using System; +using System.Linq; +using System.Threading; +using System.Threading.Tasks; +using MongoDB.Bson; +using MongoDB.Driver; +using StellaOps.Concelier.Storage.Mongo; +using StellaOps.Concelier.Storage.Mongo.Statements; +using StellaOps.Concelier.Testing; +using Xunit; + +namespace StellaOps.Concelier.Storage.Mongo.Tests; + +[Collection("mongo-fixture")] +public sealed class AdvisoryStatementStoreTests +{ + private readonly IMongoDatabase _database; + + public AdvisoryStatementStoreTests(MongoIntegrationFixture fixture) + { + _database = fixture.Database ?? throw new ArgumentNullException(nameof(fixture.Database)); + } + + [Fact] + public async Task InsertAndRetrieve_WritesImmutableStatements() + { + var store = new AdvisoryStatementStore(_database); + var vulnerabilityKey = $"CVE-{Guid.NewGuid():N}"; + var baseTime = DateTimeOffset.UtcNow; + + var statements = new[] + { + new AdvisoryStatementRecord( + Guid.NewGuid(), + vulnerabilityKey, + vulnerabilityKey, + new byte[] { 0x01 }, + baseTime, + baseTime.AddSeconds(5), + new BsonDocument("version", "A"), + new[] { Guid.NewGuid() }), + new AdvisoryStatementRecord( + Guid.NewGuid(), + vulnerabilityKey, + vulnerabilityKey, + new byte[] { 0x02 }, + baseTime.AddMinutes(1), + baseTime.AddMinutes(1).AddSeconds(5), + new BsonDocument("version", "B"), + Array.Empty()), + }; + + await store.InsertAsync(statements, CancellationToken.None); + + var results = await store.GetStatementsAsync(vulnerabilityKey, null, CancellationToken.None); + + Assert.Equal(2, results.Count); + Assert.Equal(statements[1].Id, results[0].Id); // sorted by AsOf desc + Assert.True(results.All(record => record.Payload.Contains("version"))); + } + + [Fact] + public async Task GetStatements_AsOfFiltersResults() + { + var store = new AdvisoryStatementStore(_database); + var vulnerabilityKey = $"CVE-{Guid.NewGuid():N}"; + var baseTime = DateTimeOffset.UtcNow; + + var early = new AdvisoryStatementRecord( + Guid.NewGuid(), + vulnerabilityKey, + vulnerabilityKey, + new byte[] { 0xAA }, + baseTime, + baseTime.AddSeconds(10), + new BsonDocument("state", "early"), + Array.Empty()); + + var late = new AdvisoryStatementRecord( + Guid.NewGuid(), + vulnerabilityKey, + vulnerabilityKey, + new byte[] { 0xBB }, + baseTime.AddMinutes(5), + baseTime.AddMinutes(5).AddSeconds(10), + new BsonDocument("state", "late"), + Array.Empty()); + + await store.InsertAsync(new[] { early, late }, CancellationToken.None); + + var results = await store.GetStatementsAsync(vulnerabilityKey, baseTime.AddMinutes(1), CancellationToken.None); + + Assert.Single(results); + Assert.Equal("early", results[0].Payload["state"].AsString); + } +} diff --git a/src/StellaOps.Concelier.Storage.Mongo.Tests/Migrations/MongoMigrationRunnerTests.cs b/src/StellaOps.Concelier.Storage.Mongo.Tests/Migrations/MongoMigrationRunnerTests.cs index 863635f9..4ad33e26 100644 --- a/src/StellaOps.Concelier.Storage.Mongo.Tests/Migrations/MongoMigrationRunnerTests.cs +++ b/src/StellaOps.Concelier.Storage.Mongo.Tests/Migrations/MongoMigrationRunnerTests.cs @@ -215,15 +215,59 @@ public sealed class MongoMigrationRunnerTests Assert.DoesNotContain(indexList, x => x["name"].AsString == "gridfs_files_expiresAt_ttl"); } - finally - { - await _fixture.Client.DropDatabaseAsync(databaseName); - } - } - - private sealed class TestMigration : IMongoMigration - { - public int ApplyCount { get; private set; } + finally + { + await _fixture.Client.DropDatabaseAsync(databaseName); + } + } + + [Fact] + public async Task EnsureAdvisoryEventCollectionsMigration_CreatesIndexes() + { + var databaseName = $"concelier-advisory-events-{Guid.NewGuid():N}"; + var database = _fixture.Client.GetDatabase(databaseName); + await database.CreateCollectionAsync(MongoStorageDefaults.Collections.AdvisoryStatements); + await database.CreateCollectionAsync(MongoStorageDefaults.Collections.AdvisoryConflicts); + await database.CreateCollectionAsync(MongoStorageDefaults.Collections.Migrations); + + try + { + var migration = new EnsureAdvisoryEventCollectionsMigration(); + var runner = new MongoMigrationRunner( + database, + new IMongoMigration[] { migration }, + NullLogger.Instance, + TimeProvider.System); + + await runner.RunAsync(CancellationToken.None); + + var statementIndexes = await database + .GetCollection(MongoStorageDefaults.Collections.AdvisoryStatements) + .Indexes + .ListAsync(); + var statementIndexNames = (await statementIndexes.ToListAsync()).Select(x => x["name"].AsString).ToArray(); + + Assert.Contains("advisory_statements_vulnerability_asof_desc", statementIndexNames); + Assert.Contains("advisory_statements_statementHash_unique", statementIndexNames); + + var conflictIndexes = await database + .GetCollection(MongoStorageDefaults.Collections.AdvisoryConflicts) + .Indexes + .ListAsync(); + var conflictIndexNames = (await conflictIndexes.ToListAsync()).Select(x => x["name"].AsString).ToArray(); + + Assert.Contains("advisory_conflicts_vulnerability_asof_desc", conflictIndexNames); + Assert.Contains("advisory_conflicts_conflictHash_unique", conflictIndexNames); + } + finally + { + await _fixture.Client.DropDatabaseAsync(databaseName); + } + } + + private sealed class TestMigration : IMongoMigration + { + public int ApplyCount { get; private set; } public string Id => "999_test"; diff --git a/src/StellaOps.Concelier.Storage.Mongo.Tests/MongoAdvisoryEventRepositoryTests.cs b/src/StellaOps.Concelier.Storage.Mongo.Tests/MongoAdvisoryEventRepositoryTests.cs new file mode 100644 index 00000000..36bb8b0e --- /dev/null +++ b/src/StellaOps.Concelier.Storage.Mongo.Tests/MongoAdvisoryEventRepositoryTests.cs @@ -0,0 +1,107 @@ +using System; +using System.Collections.Immutable; +using System.Linq; +using System.Security.Cryptography; +using System.Text; +using System.Collections.Generic; +using System.Threading; +using System.Threading.Tasks; +using MongoDB.Driver; +using StellaOps.Concelier.Core.Events; +using StellaOps.Concelier.Models; +using StellaOps.Concelier.Storage.Mongo.Conflicts; +using StellaOps.Concelier.Storage.Mongo.Events; +using StellaOps.Concelier.Storage.Mongo.Statements; +using StellaOps.Concelier.Testing; +using Xunit; + +namespace StellaOps.Concelier.Storage.Mongo.Tests; + +[Collection("mongo-fixture")] +public sealed class MongoAdvisoryEventRepositoryTests +{ + private readonly IMongoDatabase _database; + private readonly MongoAdvisoryEventRepository _repository; + + public MongoAdvisoryEventRepositoryTests(MongoIntegrationFixture fixture) + { + _database = fixture.Database ?? throw new ArgumentNullException(nameof(fixture.Database)); + var statementStore = new AdvisoryStatementStore(_database); + var conflictStore = new AdvisoryConflictStore(_database); + _repository = new MongoAdvisoryEventRepository(statementStore, conflictStore); + } + + [Fact] + public async Task InsertAndFetchStatements_RoundTripsCanonicalPayload() + { + var advisory = CreateSampleAdvisory("CVE-2025-7777", "Sample advisory"); + var canonicalJson = CanonicalJsonSerializer.Serialize(advisory); + var hash = ImmutableArray.Create(SHA256.HashData(Encoding.UTF8.GetBytes(canonicalJson))); + + var entry = new AdvisoryStatementEntry( + Guid.NewGuid(), + "CVE-2025-7777", + "CVE-2025-7777", + canonicalJson, + hash, + DateTimeOffset.Parse("2025-10-19T14:00:00Z"), + DateTimeOffset.Parse("2025-10-19T14:05:00Z"), + ImmutableArray.Empty); + + await _repository.InsertStatementsAsync(new[] { entry }, CancellationToken.None); + + var results = await _repository.GetStatementsAsync("CVE-2025-7777", null, CancellationToken.None); + + var snapshot = Assert.Single(results); + Assert.Equal(entry.StatementId, snapshot.StatementId); + Assert.Equal(entry.CanonicalJson, snapshot.CanonicalJson); + Assert.True(entry.StatementHash.SequenceEqual(snapshot.StatementHash)); + } + + [Fact] + public async Task InsertAndFetchConflicts_PreservesDetails() + { + var detailJson = CanonicalJsonSerializer.Serialize(new ConflictPayload("severity", "mismatch")); + var hash = ImmutableArray.Create(SHA256.HashData(Encoding.UTF8.GetBytes(detailJson))); + var statementIds = ImmutableArray.Create(Guid.NewGuid(), Guid.NewGuid()); + + var entry = new AdvisoryConflictEntry( + Guid.NewGuid(), + "CVE-2025-4242", + detailJson, + hash, + DateTimeOffset.Parse("2025-10-19T15:00:00Z"), + DateTimeOffset.Parse("2025-10-19T15:05:00Z"), + statementIds); + + await _repository.InsertConflictsAsync(new[] { entry }, CancellationToken.None); + + var results = await _repository.GetConflictsAsync("CVE-2025-4242", null, CancellationToken.None); + + var conflict = Assert.Single(results); + Assert.Equal(entry.CanonicalJson, conflict.CanonicalJson); + Assert.True(entry.StatementIds.SequenceEqual(conflict.StatementIds)); + Assert.True(entry.ConflictHash.SequenceEqual(conflict.ConflictHash)); + } + + private static Advisory CreateSampleAdvisory(string key, string summary) + { + var provenance = new AdvisoryProvenance("nvd", "document", key, DateTimeOffset.Parse("2025-10-18T00:00:00Z"), new[] { ProvenanceFieldMasks.Advisory }); + return new Advisory( + key, + key, + summary, + "en", + DateTimeOffset.Parse("2025-10-17T00:00:00Z"), + DateTimeOffset.Parse("2025-10-18T00:00:00Z"), + "medium", + exploitKnown: false, + aliases: new[] { key }, + references: Array.Empty(), + affectedPackages: Array.Empty(), + cvssMetrics: Array.Empty(), + provenance: new[] { provenance }); + } + + private sealed record ConflictPayload(string Type, string Reason); +} diff --git a/src/StellaOps.Concelier.Storage.Mongo.Tests/MongoBootstrapperTests.cs b/src/StellaOps.Concelier.Storage.Mongo.Tests/MongoBootstrapperTests.cs index 0c86169f..dd885307 100644 --- a/src/StellaOps.Concelier.Storage.Mongo.Tests/MongoBootstrapperTests.cs +++ b/src/StellaOps.Concelier.Storage.Mongo.Tests/MongoBootstrapperTests.cs @@ -94,4 +94,50 @@ public sealed class MongoBootstrapperTests : IClassFixture(), + NullLogger.Instance, + TimeProvider.System); + + var bootstrapper = new MongoBootstrapper( + database, + Options.Create(new MongoStorageOptions()), + NullLogger.Instance, + runner); + + await bootstrapper.InitializeAsync(CancellationToken.None); + + var statementIndexes = await database + .GetCollection(MongoStorageDefaults.Collections.AdvisoryStatements) + .Indexes + .ListAsync(); + var statementIndexNames = (await statementIndexes.ToListAsync()).Select(x => x["name"].AsString).ToArray(); + + Assert.Contains("advisory_statements_vulnerability_asof_desc", statementIndexNames); + Assert.Contains("advisory_statements_statementHash_unique", statementIndexNames); + + var conflictIndexes = await database + .GetCollection(MongoStorageDefaults.Collections.AdvisoryConflicts) + .Indexes + .ListAsync(); + var conflictIndexNames = (await conflictIndexes.ToListAsync()).Select(x => x["name"].AsString).ToArray(); + + Assert.Contains("advisory_conflicts_vulnerability_asof_desc", conflictIndexNames); + Assert.Contains("advisory_conflicts_conflictHash_unique", conflictIndexNames); + } + finally + { + await _fixture.Client.DropDatabaseAsync(databaseName); + } + } } diff --git a/src/StellaOps.Concelier.Storage.Mongo/Conflicts/AdvisoryConflictDocument.cs b/src/StellaOps.Concelier.Storage.Mongo/Conflicts/AdvisoryConflictDocument.cs new file mode 100644 index 00000000..e5a77dc7 --- /dev/null +++ b/src/StellaOps.Concelier.Storage.Mongo/Conflicts/AdvisoryConflictDocument.cs @@ -0,0 +1,57 @@ +using System; +using System.Collections.Generic; +using System.Linq; +using MongoDB.Bson; +using MongoDB.Bson.Serialization.Attributes; + +namespace StellaOps.Concelier.Storage.Mongo.Conflicts; + +[BsonIgnoreExtraElements] +public sealed class AdvisoryConflictDocument +{ + [BsonId] + public string Id { get; set; } = Guid.Empty.ToString("N"); + + [BsonElement("vulnerabilityKey")] + public string VulnerabilityKey { get; set; } = string.Empty; + + [BsonElement("conflictHash")] + public byte[] ConflictHash { get; set; } = Array.Empty(); + + [BsonElement("asOf")] + public DateTime AsOf { get; set; } + + [BsonElement("recordedAt")] + public DateTime RecordedAt { get; set; } + + [BsonElement("statementIds")] + public List StatementIds { get; set; } = new(); + + [BsonElement("details")] + public BsonDocument Details { get; set; } = new(); +} + +internal static class AdvisoryConflictDocumentExtensions +{ + public static AdvisoryConflictDocument FromRecord(AdvisoryConflictRecord record) + => new() + { + Id = record.Id.ToString(), + VulnerabilityKey = record.VulnerabilityKey, + ConflictHash = record.ConflictHash, + AsOf = record.AsOf.UtcDateTime, + RecordedAt = record.RecordedAt.UtcDateTime, + StatementIds = record.StatementIds.Select(static id => id.ToString()).ToList(), + Details = (BsonDocument)record.Details.DeepClone(), + }; + + public static AdvisoryConflictRecord ToRecord(this AdvisoryConflictDocument document) + => new( + Guid.Parse(document.Id), + document.VulnerabilityKey, + document.ConflictHash, + DateTime.SpecifyKind(document.AsOf, DateTimeKind.Utc), + DateTime.SpecifyKind(document.RecordedAt, DateTimeKind.Utc), + document.StatementIds.Select(static value => Guid.Parse(value)).ToList(), + (BsonDocument)document.Details.DeepClone()); +} diff --git a/src/StellaOps.Concelier.Storage.Mongo/Conflicts/AdvisoryConflictRecord.cs b/src/StellaOps.Concelier.Storage.Mongo/Conflicts/AdvisoryConflictRecord.cs new file mode 100644 index 00000000..0d0c4428 --- /dev/null +++ b/src/StellaOps.Concelier.Storage.Mongo/Conflicts/AdvisoryConflictRecord.cs @@ -0,0 +1,14 @@ +using System; +using System.Collections.Generic; +using MongoDB.Bson; + +namespace StellaOps.Concelier.Storage.Mongo.Conflicts; + +public sealed record AdvisoryConflictRecord( + Guid Id, + string VulnerabilityKey, + byte[] ConflictHash, + DateTimeOffset AsOf, + DateTimeOffset RecordedAt, + IReadOnlyList StatementIds, + BsonDocument Details); diff --git a/src/StellaOps.Concelier.Storage.Mongo/Conflicts/AdvisoryConflictStore.cs b/src/StellaOps.Concelier.Storage.Mongo/Conflicts/AdvisoryConflictStore.cs new file mode 100644 index 00000000..42cbb802 --- /dev/null +++ b/src/StellaOps.Concelier.Storage.Mongo/Conflicts/AdvisoryConflictStore.cs @@ -0,0 +1,93 @@ +using System; +using System.Collections.Generic; +using System.Linq; +using System.Threading; +using System.Threading.Tasks; +using MongoDB.Driver; + +namespace StellaOps.Concelier.Storage.Mongo.Conflicts; + +public interface IAdvisoryConflictStore +{ + ValueTask InsertAsync( + IReadOnlyCollection conflicts, + CancellationToken cancellationToken, + IClientSessionHandle? session = null); + + ValueTask> GetConflictsAsync( + string vulnerabilityKey, + DateTimeOffset? asOf, + CancellationToken cancellationToken, + IClientSessionHandle? session = null); +} + +public sealed class AdvisoryConflictStore : IAdvisoryConflictStore +{ + private readonly IMongoCollection _collection; + + public AdvisoryConflictStore(IMongoDatabase database) + { + ArgumentNullException.ThrowIfNull(database); + _collection = database.GetCollection(MongoStorageDefaults.Collections.AdvisoryConflicts); + } + + public async ValueTask InsertAsync( + IReadOnlyCollection conflicts, + CancellationToken cancellationToken, + IClientSessionHandle? session = null) + { + ArgumentNullException.ThrowIfNull(conflicts); + + if (conflicts.Count == 0) + { + return; + } + + var documents = conflicts.Select(AdvisoryConflictDocumentExtensions.FromRecord).ToList(); + var options = new InsertManyOptions { IsOrdered = true }; + + try + { + if (session is null) + { + await _collection.InsertManyAsync(documents, options, cancellationToken).ConfigureAwait(false); + } + else + { + await _collection.InsertManyAsync(session, documents, options, cancellationToken).ConfigureAwait(false); + } + } + catch (MongoBulkWriteException ex) when (ex.WriteErrors.All(error => error.Category == ServerErrorCategory.DuplicateKey)) + { + // Conflicts already persisted for this state; ignore duplicates. + } + } + + public async ValueTask> GetConflictsAsync( + string vulnerabilityKey, + DateTimeOffset? asOf, + CancellationToken cancellationToken, + IClientSessionHandle? session = null) + { + ArgumentException.ThrowIfNullOrWhiteSpace(vulnerabilityKey); + + var filter = Builders.Filter.Eq(document => document.VulnerabilityKey, vulnerabilityKey); + + if (asOf.HasValue) + { + filter &= Builders.Filter.Lte(document => document.AsOf, asOf.Value.UtcDateTime); + } + + var find = session is null + ? _collection.Find(filter) + : _collection.Find(session, filter); + + var documents = await find + .SortByDescending(document => document.AsOf) + .ThenByDescending(document => document.RecordedAt) + .ToListAsync(cancellationToken) + .ConfigureAwait(false); + + return documents.Select(static document => document.ToRecord()).ToList(); + } +} diff --git a/src/StellaOps.Concelier.Storage.Mongo/Events/MongoAdvisoryEventRepository.cs b/src/StellaOps.Concelier.Storage.Mongo/Events/MongoAdvisoryEventRepository.cs new file mode 100644 index 00000000..b9fdec47 --- /dev/null +++ b/src/StellaOps.Concelier.Storage.Mongo/Events/MongoAdvisoryEventRepository.cs @@ -0,0 +1,224 @@ +using System; +using System.Collections.Generic; +using System.Collections.Immutable; +using System.IO; +using System.Linq; +using System.Text; +using System.Text.Encodings.Web; +using System.Text.Json; +using System.Threading; +using System.Threading.Tasks; +using MongoDB.Bson; +using StellaOps.Concelier.Core.Events; +using StellaOps.Concelier.Models; +using StellaOps.Concelier.Storage.Mongo.Conflicts; +using StellaOps.Concelier.Storage.Mongo.Statements; + +namespace StellaOps.Concelier.Storage.Mongo.Events; + +public sealed class MongoAdvisoryEventRepository : IAdvisoryEventRepository +{ + private readonly IAdvisoryStatementStore _statementStore; + private readonly IAdvisoryConflictStore _conflictStore; + + public MongoAdvisoryEventRepository( + IAdvisoryStatementStore statementStore, + IAdvisoryConflictStore conflictStore) + { + _statementStore = statementStore ?? throw new ArgumentNullException(nameof(statementStore)); + _conflictStore = conflictStore ?? throw new ArgumentNullException(nameof(conflictStore)); + } + + public async ValueTask InsertStatementsAsync( + IReadOnlyCollection statements, + CancellationToken cancellationToken) + { + if (statements is null) + { + throw new ArgumentNullException(nameof(statements)); + } + + if (statements.Count == 0) + { + return; + } + + var records = statements + .Select(static entry => + { + var payload = BsonDocument.Parse(entry.CanonicalJson); + return new AdvisoryStatementRecord( + entry.StatementId, + entry.VulnerabilityKey, + entry.AdvisoryKey, + entry.StatementHash.ToArray(), + entry.AsOf, + entry.RecordedAt, + payload, + entry.InputDocumentIds.ToArray()); + }) + .ToList(); + + await _statementStore.InsertAsync(records, cancellationToken).ConfigureAwait(false); + } + + public async ValueTask InsertConflictsAsync( + IReadOnlyCollection conflicts, + CancellationToken cancellationToken) + { + if (conflicts is null) + { + throw new ArgumentNullException(nameof(conflicts)); + } + + if (conflicts.Count == 0) + { + return; + } + + var records = conflicts + .Select(static entry => + { + var payload = BsonDocument.Parse(entry.CanonicalJson); + return new AdvisoryConflictRecord( + entry.ConflictId, + entry.VulnerabilityKey, + entry.ConflictHash.ToArray(), + entry.AsOf, + entry.RecordedAt, + entry.StatementIds.ToArray(), + payload); + }) + .ToList(); + + await _conflictStore.InsertAsync(records, cancellationToken).ConfigureAwait(false); + } + + public async ValueTask> GetStatementsAsync( + string vulnerabilityKey, + DateTimeOffset? asOf, + CancellationToken cancellationToken) + { + var records = await _statementStore + .GetStatementsAsync(vulnerabilityKey, asOf, cancellationToken) + .ConfigureAwait(false); + + if (records.Count == 0) + { + return Array.Empty(); + } + + var entries = records + .Select(static record => + { + var advisory = CanonicalJsonSerializer.Deserialize(record.Payload.ToJson()); + var canonicalJson = CanonicalJsonSerializer.Serialize(advisory); + + return new AdvisoryStatementEntry( + record.Id, + record.VulnerabilityKey, + record.AdvisoryKey, + canonicalJson, + record.StatementHash.ToImmutableArray(), + record.AsOf, + record.RecordedAt, + record.InputDocumentIds.ToImmutableArray()); + }) + .ToList(); + + return entries; + } + + public async ValueTask> GetConflictsAsync( + string vulnerabilityKey, + DateTimeOffset? asOf, + CancellationToken cancellationToken) + { + var records = await _conflictStore + .GetConflictsAsync(vulnerabilityKey, asOf, cancellationToken) + .ConfigureAwait(false); + + if (records.Count == 0) + { + return Array.Empty(); + } + + var entries = records + .Select(static record => + { + var canonicalJson = Canonicalize(record.Details); + return new AdvisoryConflictEntry( + record.Id, + record.VulnerabilityKey, + canonicalJson, + record.ConflictHash.ToImmutableArray(), + record.AsOf, + record.RecordedAt, + record.StatementIds.ToImmutableArray()); + }) + .ToList(); + + return entries; + } + private static readonly JsonWriterOptions CanonicalWriterOptions = new() + { + Encoder = JavaScriptEncoder.UnsafeRelaxedJsonEscaping, + Indented = false, + SkipValidation = false, + }; + + private static string Canonicalize(BsonDocument document) + { + using var json = JsonDocument.Parse(document.ToJson()); + using var stream = new MemoryStream(); + using (var writer = new Utf8JsonWriter(stream, CanonicalWriterOptions)) + { + WriteCanonical(json.RootElement, writer); + } + + return Encoding.UTF8.GetString(stream.ToArray()); + } + + private static void WriteCanonical(JsonElement element, Utf8JsonWriter writer) + { + switch (element.ValueKind) + { + case JsonValueKind.Object: + writer.WriteStartObject(); + foreach (var property in element.EnumerateObject().OrderBy(static p => p.Name, StringComparer.Ordinal)) + { + writer.WritePropertyName(property.Name); + WriteCanonical(property.Value, writer); + } + writer.WriteEndObject(); + break; + case JsonValueKind.Array: + writer.WriteStartArray(); + foreach (var item in element.EnumerateArray()) + { + WriteCanonical(item, writer); + } + writer.WriteEndArray(); + break; + case JsonValueKind.String: + writer.WriteStringValue(element.GetString()); + break; + case JsonValueKind.Number: + writer.WriteRawValue(element.GetRawText()); + break; + case JsonValueKind.True: + writer.WriteBooleanValue(true); + break; + case JsonValueKind.False: + writer.WriteBooleanValue(false); + break; + case JsonValueKind.Null: + writer.WriteNullValue(); + break; + default: + writer.WriteRawValue(element.GetRawText()); + break; + } + } + +} diff --git a/src/StellaOps.Concelier.Storage.Mongo/MIGRATIONS.md b/src/StellaOps.Concelier.Storage.Mongo/MIGRATIONS.md index 2aafce46..65bf1698 100644 --- a/src/StellaOps.Concelier.Storage.Mongo/MIGRATIONS.md +++ b/src/StellaOps.Concelier.Storage.Mongo/MIGRATIONS.md @@ -18,16 +18,18 @@ This module owns the persistent shape of Concelier's MongoDB database. Upgrades ## Current Migrations -| Id | Description | -| --- | --- | -| `20241005_document_expiry_indexes` | Ensures `document` collection uses the correct TTL/partial index depending on raw document retention settings. | -| `20241005_gridfs_expiry_indexes` | Aligns the GridFS `documents.files` TTL index with retention settings. | +| Id | Description | +| --- | --- | +| `20241005_document_expiry_indexes` | Ensures `document` collection uses the correct TTL/partial index depending on raw document retention settings. | +| `20241005_gridfs_expiry_indexes` | Aligns the GridFS `documents.files` TTL index with retention settings. | +| `20251019_advisory_event_collections` | Creates/aligns indexes for `advisory_statements` and `advisory_conflicts` collections powering the event log + conflict replay pipeline. | ## Operator Runbook -- `schema_migrations` records each applied migration (`_id`, `description`, `appliedAt`). Review this collection when auditing upgrades. -- To re-run a migration in a lab, delete the corresponding document from `schema_migrations` and restart the service. **Do not** do this in production unless the migration body is known to be idempotent and safe. -- When changing retention settings (`RawDocumentRetention`), deploy the new configuration and restart Concelier. The migration runner will adjust indexes on the next boot. +- `schema_migrations` records each applied migration (`_id`, `description`, `appliedAt`). Review this collection when auditing upgrades. +- To re-run a migration in a lab, delete the corresponding document from `schema_migrations` and restart the service. **Do not** do this in production unless the migration body is known to be idempotent and safe. +- When changing retention settings (`RawDocumentRetention`), deploy the new configuration and restart Concelier. The migration runner will adjust indexes on the next boot. +- For the event-log collections (`advisory_statements`, `advisory_conflicts`), rollback is simply `db.advisory_statements.drop()` / `db.advisory_conflicts.drop()` followed by a restart if you must revert to the pre-event-log schema (only in labs). Production rollbacks should instead gate merge features that rely on these collections. - If migrations fail, restart with `Logging__LogLevel__StellaOps.Concelier.Storage.Mongo.Migrations=Debug` to surface diagnostic output. Remediate underlying index/collection drift before retrying. ## Validating an Upgrade diff --git a/src/StellaOps.Concelier.Storage.Mongo/Migrations/EnsureAdvisoryEventCollectionsMigration.cs b/src/StellaOps.Concelier.Storage.Mongo/Migrations/EnsureAdvisoryEventCollectionsMigration.cs new file mode 100644 index 00000000..f6629579 --- /dev/null +++ b/src/StellaOps.Concelier.Storage.Mongo/Migrations/EnsureAdvisoryEventCollectionsMigration.cs @@ -0,0 +1,44 @@ +using System.Collections.Generic; +using System.Threading; +using System.Threading.Tasks; +using MongoDB.Bson; +using MongoDB.Driver; + +namespace StellaOps.Concelier.Storage.Mongo.Migrations; + +public sealed class EnsureAdvisoryEventCollectionsMigration : IMongoMigration +{ + public string Id => "20251019_advisory_event_collections"; + + public string Description => "Ensure advisory_statements and advisory_conflicts indexes exist for event log storage."; + + public async Task ApplyAsync(IMongoDatabase database, CancellationToken cancellationToken) + { + var statements = database.GetCollection(MongoStorageDefaults.Collections.AdvisoryStatements); + var conflicts = database.GetCollection(MongoStorageDefaults.Collections.AdvisoryConflicts); + + var statementIndexes = new List> + { + new( + Builders.IndexKeys.Ascending("vulnerabilityKey").Descending("asOf"), + new CreateIndexOptions { Name = "advisory_statements_vulnerability_asof_desc" }), + new( + Builders.IndexKeys.Ascending("statementHash"), + new CreateIndexOptions { Name = "advisory_statements_statementHash_unique", Unique = true }), + }; + + await statements.Indexes.CreateManyAsync(statementIndexes, cancellationToken).ConfigureAwait(false); + + var conflictIndexes = new List> + { + new( + Builders.IndexKeys.Ascending("vulnerabilityKey").Descending("asOf"), + new CreateIndexOptions { Name = "advisory_conflicts_vulnerability_asof_desc" }), + new( + Builders.IndexKeys.Ascending("conflictHash"), + new CreateIndexOptions { Name = "advisory_conflicts_conflictHash_unique", Unique = true }), + }; + + await conflicts.Indexes.CreateManyAsync(conflictIndexes, cancellationToken).ConfigureAwait(false); + } +} diff --git a/src/StellaOps.Concelier.Storage.Mongo/MongoBootstrapper.cs b/src/StellaOps.Concelier.Storage.Mongo/MongoBootstrapper.cs index 1ce51157..2e7ccba4 100644 --- a/src/StellaOps.Concelier.Storage.Mongo/MongoBootstrapper.cs +++ b/src/StellaOps.Concelier.Storage.Mongo/MongoBootstrapper.cs @@ -25,14 +25,16 @@ public sealed class MongoBootstrapper MongoStorageDefaults.Collections.KevFlag, MongoStorageDefaults.Collections.RuFlags, MongoStorageDefaults.Collections.JpFlags, - MongoStorageDefaults.Collections.PsirtFlags, - MongoStorageDefaults.Collections.MergeEvent, - MongoStorageDefaults.Collections.ExportState, - MongoStorageDefaults.Collections.ChangeHistory, - MongoStorageDefaults.Collections.Locks, - MongoStorageDefaults.Collections.Jobs, - MongoStorageDefaults.Collections.Migrations, - }; + MongoStorageDefaults.Collections.PsirtFlags, + MongoStorageDefaults.Collections.MergeEvent, + MongoStorageDefaults.Collections.ExportState, + MongoStorageDefaults.Collections.ChangeHistory, + MongoStorageDefaults.Collections.AdvisoryStatements, + MongoStorageDefaults.Collections.AdvisoryConflicts, + MongoStorageDefaults.Collections.Locks, + MongoStorageDefaults.Collections.Jobs, + MongoStorageDefaults.Collections.Migrations, + }; private readonly IMongoDatabase _database; private readonly MongoStorageOptions _options; @@ -70,15 +72,17 @@ public sealed class MongoBootstrapper EnsureAdvisoryIndexesAsync(cancellationToken), EnsureDocumentsIndexesAsync(cancellationToken), EnsureDtoIndexesAsync(cancellationToken), - EnsureAliasIndexesAsync(cancellationToken), - EnsureAffectedIndexesAsync(cancellationToken), - EnsureReferenceIndexesAsync(cancellationToken), - EnsureSourceStateIndexesAsync(cancellationToken), - EnsurePsirtFlagIndexesAsync(cancellationToken), - EnsureChangeHistoryIndexesAsync(cancellationToken), - EnsureGridFsIndexesAsync(cancellationToken)).ConfigureAwait(false); - - await _migrationRunner.RunAsync(cancellationToken).ConfigureAwait(false); + EnsureAliasIndexesAsync(cancellationToken), + EnsureAffectedIndexesAsync(cancellationToken), + EnsureReferenceIndexesAsync(cancellationToken), + EnsureSourceStateIndexesAsync(cancellationToken), + EnsurePsirtFlagIndexesAsync(cancellationToken), + EnsureAdvisoryStatementIndexesAsync(cancellationToken), + EnsureAdvisoryConflictIndexesAsync(cancellationToken), + EnsureChangeHistoryIndexesAsync(cancellationToken), + EnsureGridFsIndexesAsync(cancellationToken)).ConfigureAwait(false); + + await _migrationRunner.RunAsync(cancellationToken).ConfigureAwait(false); _logger.LogInformation("Mongo bootstrapper completed"); } @@ -238,10 +242,10 @@ public sealed class MongoBootstrapper return collection.Indexes.CreateManyAsync(indexes, cancellationToken); } - private Task EnsureReferenceIndexesAsync(CancellationToken cancellationToken) - { - var collection = _database.GetCollection(MongoStorageDefaults.Collections.Reference); - var indexes = new List> + private Task EnsureReferenceIndexesAsync(CancellationToken cancellationToken) + { + var collection = _database.GetCollection(MongoStorageDefaults.Collections.Reference); + var indexes = new List> { new( Builders.IndexKeys.Ascending("url"), @@ -250,10 +254,42 @@ public sealed class MongoBootstrapper Builders.IndexKeys.Ascending("advisoryId"), new CreateIndexOptions { Name = "reference_advisoryId" }), }; - - return collection.Indexes.CreateManyAsync(indexes, cancellationToken); - } - + + return collection.Indexes.CreateManyAsync(indexes, cancellationToken); + } + + private Task EnsureAdvisoryStatementIndexesAsync(CancellationToken cancellationToken) + { + var collection = _database.GetCollection(MongoStorageDefaults.Collections.AdvisoryStatements); + var indexes = new List> + { + new( + Builders.IndexKeys.Ascending("vulnerabilityKey").Descending("asOf"), + new CreateIndexOptions { Name = "advisory_statements_vulnerability_asof_desc" }), + new( + Builders.IndexKeys.Ascending("statementHash"), + new CreateIndexOptions { Name = "advisory_statements_statementHash_unique", Unique = true }), + }; + + return collection.Indexes.CreateManyAsync(indexes, cancellationToken); + } + + private Task EnsureAdvisoryConflictIndexesAsync(CancellationToken cancellationToken) + { + var collection = _database.GetCollection(MongoStorageDefaults.Collections.AdvisoryConflicts); + var indexes = new List> + { + new( + Builders.IndexKeys.Ascending("vulnerabilityKey").Descending("asOf"), + new CreateIndexOptions { Name = "advisory_conflicts_vulnerability_asof_desc" }), + new( + Builders.IndexKeys.Ascending("conflictHash"), + new CreateIndexOptions { Name = "advisory_conflicts_conflictHash_unique", Unique = true }), + }; + + return collection.Indexes.CreateManyAsync(indexes, cancellationToken); + } + private Task EnsureSourceStateIndexesAsync(CancellationToken cancellationToken) { var collection = _database.GetCollection(MongoStorageDefaults.Collections.SourceState); diff --git a/src/StellaOps.Concelier.Storage.Mongo/MongoStorageDefaults.cs b/src/StellaOps.Concelier.Storage.Mongo/MongoStorageDefaults.cs index afe5936b..f8f6492c 100644 --- a/src/StellaOps.Concelier.Storage.Mongo/MongoStorageDefaults.cs +++ b/src/StellaOps.Concelier.Storage.Mongo/MongoStorageDefaults.cs @@ -18,11 +18,13 @@ public static class MongoStorageDefaults public const string RuFlags = "ru_flags"; public const string JpFlags = "jp_flags"; public const string PsirtFlags = "psirt_flags"; - public const string MergeEvent = "merge_event"; - public const string ExportState = "export_state"; - public const string Locks = "locks"; - public const string Jobs = "jobs"; - public const string Migrations = "schema_migrations"; - public const string ChangeHistory = "source_change_history"; - } -} + public const string MergeEvent = "merge_event"; + public const string ExportState = "export_state"; + public const string Locks = "locks"; + public const string Jobs = "jobs"; + public const string Migrations = "schema_migrations"; + public const string ChangeHistory = "source_change_history"; + public const string AdvisoryStatements = "advisory_statements"; + public const string AdvisoryConflicts = "advisory_conflicts"; + } +} diff --git a/src/StellaOps.Concelier.Storage.Mongo/ServiceCollectionExtensions.cs b/src/StellaOps.Concelier.Storage.Mongo/ServiceCollectionExtensions.cs index 31cfbe6d..c8c56372 100644 --- a/src/StellaOps.Concelier.Storage.Mongo/ServiceCollectionExtensions.cs +++ b/src/StellaOps.Concelier.Storage.Mongo/ServiceCollectionExtensions.cs @@ -1,54 +1,57 @@ -using Microsoft.Extensions.DependencyInjection; -using Microsoft.Extensions.DependencyInjection.Extensions; -using Microsoft.Extensions.Logging; -using Microsoft.Extensions.Options; -using MongoDB.Driver; -using StellaOps.Concelier.Core.Jobs; -using StellaOps.Concelier.Storage.Mongo.Advisories; -using StellaOps.Concelier.Storage.Mongo.Aliases; -using StellaOps.Concelier.Storage.Mongo.ChangeHistory; -using StellaOps.Concelier.Storage.Mongo.Documents; -using StellaOps.Concelier.Storage.Mongo.Dtos; -using StellaOps.Concelier.Storage.Mongo.Exporting; -using StellaOps.Concelier.Storage.Mongo.JpFlags; -using StellaOps.Concelier.Storage.Mongo.MergeEvents; -using StellaOps.Concelier.Storage.Mongo.PsirtFlags; -using StellaOps.Concelier.Storage.Mongo.Migrations; - -namespace StellaOps.Concelier.Storage.Mongo; - -public static class ServiceCollectionExtensions -{ - public static IServiceCollection AddMongoStorage(this IServiceCollection services, Action configureOptions) - { - ArgumentNullException.ThrowIfNull(services); - ArgumentNullException.ThrowIfNull(configureOptions); - - services.AddOptions() - .Configure(configureOptions) - .PostConfigure(static options => options.EnsureValid()); - - services.TryAddSingleton(TimeProvider.System); - - services.AddSingleton(static sp => - { - var options = sp.GetRequiredService>().Value; - return new MongoClient(options.ConnectionString); - }); - +using Microsoft.Extensions.DependencyInjection; +using Microsoft.Extensions.DependencyInjection.Extensions; +using Microsoft.Extensions.Options; +using MongoDB.Driver; +using StellaOps.Concelier.Core.Jobs; +using StellaOps.Concelier.Storage.Mongo.Advisories; +using StellaOps.Concelier.Storage.Mongo.Aliases; +using StellaOps.Concelier.Storage.Mongo.ChangeHistory; +using StellaOps.Concelier.Storage.Mongo.Documents; +using StellaOps.Concelier.Storage.Mongo.Dtos; +using StellaOps.Concelier.Storage.Mongo.Exporting; +using StellaOps.Concelier.Storage.Mongo.JpFlags; +using StellaOps.Concelier.Storage.Mongo.MergeEvents; +using StellaOps.Concelier.Storage.Mongo.Conflicts; +using StellaOps.Concelier.Storage.Mongo.PsirtFlags; +using StellaOps.Concelier.Storage.Mongo.Statements; +using StellaOps.Concelier.Storage.Mongo.Events; +using StellaOps.Concelier.Core.Events; +using StellaOps.Concelier.Storage.Mongo.Migrations; + +namespace StellaOps.Concelier.Storage.Mongo; + +public static class ServiceCollectionExtensions +{ + public static IServiceCollection AddMongoStorage(this IServiceCollection services, Action configureOptions) + { + ArgumentNullException.ThrowIfNull(services); + ArgumentNullException.ThrowIfNull(configureOptions); + + services.AddOptions() + .Configure(configureOptions) + .PostConfigure(static options => options.EnsureValid()); + + services.TryAddSingleton(TimeProvider.System); + + services.AddSingleton(static sp => + { + var options = sp.GetRequiredService>().Value; + return new MongoClient(options.ConnectionString); + }); + services.AddSingleton(static sp => { var options = sp.GetRequiredService>().Value; var client = sp.GetRequiredService(); var settings = new MongoDatabaseSettings - { - ReadConcern = ReadConcern.Majority, - WriteConcern = WriteConcern.WMajority, - ReadPreference = ReadPreference.PrimaryPreferred, - }; - - var database = client.GetDatabase(options.GetDatabaseName(), settings); - var writeConcern = database.Settings.WriteConcern.With(wTimeout: options.CommandTimeout); + { + ReadConcern = ReadConcern.Majority, + WriteConcern = WriteConcern.WMajority, + ReadPreference = ReadPreference.PrimaryPreferred, + }; + + var database = client.GetDatabase(options.GetDatabaseName(), settings); + var writeConcern = database.Settings.WriteConcern.With(wTimeout: options.CommandTimeout); return database.WithWriteConcern(writeConcern); }); @@ -56,36 +59,41 @@ public static class ServiceCollectionExtensions services.AddSingleton(); services.AddSingleton(); - services.AddSingleton(); - services.AddSingleton(); - services.AddSingleton(); - services.AddSingleton(); - services.AddSingleton(); - services.AddSingleton(); - services.AddSingleton(); - services.AddSingleton(); - services.AddSingleton(); - services.AddSingleton(); - services.AddSingleton(); - services.TryAddSingleton(); - - services.AddSingleton>(static sp => - { - var database = sp.GetRequiredService(); - return database.GetCollection(MongoStorageDefaults.Collections.Jobs); - }); - - services.AddSingleton>(static sp => - { - var database = sp.GetRequiredService(); - return database.GetCollection(MongoStorageDefaults.Collections.Locks); - }); - - services.AddHostedService(); - + services.AddSingleton(); + services.AddSingleton(); + services.AddSingleton(); + services.AddSingleton(); + services.AddSingleton(); + services.AddSingleton(); + services.AddSingleton(); + services.AddSingleton(); + services.AddSingleton(); + services.AddSingleton(); + services.AddSingleton(); + services.AddSingleton(); + services.AddSingleton(); + services.AddSingleton(); + services.AddSingleton(); + services.TryAddSingleton(); + + services.AddSingleton>(static sp => + { + var database = sp.GetRequiredService(); + return database.GetCollection(MongoStorageDefaults.Collections.Jobs); + }); + + services.AddSingleton>(static sp => + { + var database = sp.GetRequiredService(); + return database.GetCollection(MongoStorageDefaults.Collections.Locks); + }); + + services.AddHostedService(); + services.AddSingleton(); services.AddSingleton(); services.AddSingleton(); + services.AddSingleton(); services.AddSingleton(); return services; diff --git a/src/StellaOps.Concelier.Storage.Mongo/Statements/AdvisoryStatementDocument.cs b/src/StellaOps.Concelier.Storage.Mongo/Statements/AdvisoryStatementDocument.cs new file mode 100644 index 00000000..80d591cd --- /dev/null +++ b/src/StellaOps.Concelier.Storage.Mongo/Statements/AdvisoryStatementDocument.cs @@ -0,0 +1,62 @@ +using System; +using System.Collections.Generic; +using System.Linq; +using MongoDB.Bson; +using MongoDB.Bson.Serialization.Attributes; + +namespace StellaOps.Concelier.Storage.Mongo.Statements; + +[BsonIgnoreExtraElements] +public sealed class AdvisoryStatementDocument +{ + [BsonId] + public string Id { get; set; } = string.Empty; + + [BsonElement("vulnerabilityKey")] + public string VulnerabilityKey { get; set; } = string.Empty; + + [BsonElement("advisoryKey")] + public string AdvisoryKey { get; set; } = string.Empty; + + [BsonElement("statementHash")] + public byte[] StatementHash { get; set; } = Array.Empty(); + + [BsonElement("asOf")] + public DateTime AsOf { get; set; } + + [BsonElement("recordedAt")] + public DateTime RecordedAt { get; set; } + + [BsonElement("payload")] + public BsonDocument Payload { get; set; } = new(); + + [BsonElement("inputDocuments")] + public List InputDocuments { get; set; } = new(); +} + +internal static class AdvisoryStatementDocumentExtensions +{ + public static AdvisoryStatementDocument FromRecord(AdvisoryStatementRecord record) + => new() + { + Id = record.Id.ToString(), + VulnerabilityKey = record.VulnerabilityKey, + AdvisoryKey = record.AdvisoryKey, + StatementHash = record.StatementHash, + AsOf = record.AsOf.UtcDateTime, + RecordedAt = record.RecordedAt.UtcDateTime, + Payload = (BsonDocument)record.Payload.DeepClone(), + InputDocuments = record.InputDocumentIds.Select(static id => id.ToString()).ToList(), + }; + + public static AdvisoryStatementRecord ToRecord(this AdvisoryStatementDocument document) + => new( + Guid.Parse(document.Id), + document.VulnerabilityKey, + document.AdvisoryKey, + document.StatementHash, + DateTime.SpecifyKind(document.AsOf, DateTimeKind.Utc), + DateTime.SpecifyKind(document.RecordedAt, DateTimeKind.Utc), + (BsonDocument)document.Payload.DeepClone(), + document.InputDocuments.Select(static value => Guid.Parse(value)).ToList()); +} diff --git a/src/StellaOps.Concelier.Storage.Mongo/Statements/AdvisoryStatementRecord.cs b/src/StellaOps.Concelier.Storage.Mongo/Statements/AdvisoryStatementRecord.cs new file mode 100644 index 00000000..cbdf4f82 --- /dev/null +++ b/src/StellaOps.Concelier.Storage.Mongo/Statements/AdvisoryStatementRecord.cs @@ -0,0 +1,15 @@ +using System; +using System.Collections.Generic; +using MongoDB.Bson; + +namespace StellaOps.Concelier.Storage.Mongo.Statements; + +public sealed record AdvisoryStatementRecord( + Guid Id, + string VulnerabilityKey, + string AdvisoryKey, + byte[] StatementHash, + DateTimeOffset AsOf, + DateTimeOffset RecordedAt, + BsonDocument Payload, + IReadOnlyList InputDocumentIds); diff --git a/src/StellaOps.Concelier.Storage.Mongo/Statements/AdvisoryStatementStore.cs b/src/StellaOps.Concelier.Storage.Mongo/Statements/AdvisoryStatementStore.cs new file mode 100644 index 00000000..ee75824a --- /dev/null +++ b/src/StellaOps.Concelier.Storage.Mongo/Statements/AdvisoryStatementStore.cs @@ -0,0 +1,93 @@ +using System; +using System.Collections.Generic; +using System.Linq; +using System.Threading; +using System.Threading.Tasks; +using MongoDB.Driver; + +namespace StellaOps.Concelier.Storage.Mongo.Statements; + +public interface IAdvisoryStatementStore +{ + ValueTask InsertAsync( + IReadOnlyCollection statements, + CancellationToken cancellationToken, + IClientSessionHandle? session = null); + + ValueTask> GetStatementsAsync( + string vulnerabilityKey, + DateTimeOffset? asOf, + CancellationToken cancellationToken, + IClientSessionHandle? session = null); +} + +public sealed class AdvisoryStatementStore : IAdvisoryStatementStore +{ + private readonly IMongoCollection _collection; + + public AdvisoryStatementStore(IMongoDatabase database) + { + ArgumentNullException.ThrowIfNull(database); + _collection = database.GetCollection(MongoStorageDefaults.Collections.AdvisoryStatements); + } + + public async ValueTask InsertAsync( + IReadOnlyCollection statements, + CancellationToken cancellationToken, + IClientSessionHandle? session = null) + { + ArgumentNullException.ThrowIfNull(statements); + + if (statements.Count == 0) + { + return; + } + + var documents = statements.Select(AdvisoryStatementDocumentExtensions.FromRecord).ToList(); + var options = new InsertManyOptions { IsOrdered = true }; + + try + { + if (session is null) + { + await _collection.InsertManyAsync(documents, options, cancellationToken).ConfigureAwait(false); + } + else + { + await _collection.InsertManyAsync(session, documents, options, cancellationToken).ConfigureAwait(false); + } + } + catch (MongoBulkWriteException ex) when (ex.WriteErrors.All(error => error.Category == ServerErrorCategory.DuplicateKey)) + { + // All duplicates already exist – safe to ignore for immutable statement log. + } + } + + public async ValueTask> GetStatementsAsync( + string vulnerabilityKey, + DateTimeOffset? asOf, + CancellationToken cancellationToken, + IClientSessionHandle? session = null) + { + ArgumentException.ThrowIfNullOrWhiteSpace(vulnerabilityKey); + + var filter = Builders.Filter.Eq(document => document.VulnerabilityKey, vulnerabilityKey); + + if (asOf.HasValue) + { + filter &= Builders.Filter.Lte(document => document.AsOf, asOf.Value.UtcDateTime); + } + + var find = session is null + ? _collection.Find(filter) + : _collection.Find(session, filter); + + var documents = await find + .SortByDescending(document => document.AsOf) + .ThenByDescending(document => document.RecordedAt) + .ToListAsync(cancellationToken) + .ConfigureAwait(false); + + return documents.Select(static document => document.ToRecord()).ToList(); + } +} diff --git a/src/StellaOps.Concelier.Storage.Mongo/TASKS.md b/src/StellaOps.Concelier.Storage.Mongo/TASKS.md index 2c6fb6d0..e9ddf5cc 100644 --- a/src/StellaOps.Concelier.Storage.Mongo/TASKS.md +++ b/src/StellaOps.Concelier.Storage.Mongo/TASKS.md @@ -20,5 +20,5 @@ |FEEDSTORAGE-DATA-02-002 Provenance decision persistence|BE-Storage|Models `FEEDMODELS-SCHEMA-01-002`|**DONE (2025-10-12)** – Normalized documents carry decision reasons/source/timestamps with regression coverage verifying SemVer notes + provenance fallbacks.| |FEEDSTORAGE-DATA-02-003 Normalized versions index creation|BE-Storage|Normalization, Mongo bootstrapper|**DONE (2025-10-12)** – Bootstrapper seeds `normalizedVersions.*` indexes when SemVer style is enabled; docs/tests confirm index presence.| |FEEDSTORAGE-DATA-04-001 Advisory payload parity (description/CWEs/canonical metric)|BE-Storage|Models, Core|DONE (2025-10-15) – Mongo payloads round-trip new advisory fields; serializer/tests updated, no migration required beyond optional backfill.| -|FEEDSTORAGE-MONGO-08-001 Causal-consistent session plumbing|BE-Storage|Concelier Core DI|**DONE (2025-10-19)** – Scoped session provider registered via `AddMongoStorage`, storage repositories accept optional `IClientSessionHandle`, and `MongoSessionConsistencyTests` exercises read-your-write + monotonic reads across forced primary step-down.| -|FEEDSTORAGE-DATA-07-001 Advisory statement & conflict collections|Team Normalization & Storage Backbone|FEEDMERGE-ENGINE-07-001|TODO – Create `advisory_statements` (immutable) and `advisory_conflicts` collections, define `asOf`/`vulnerabilityKey` indexes, and document migration/rollback steps for event-sourced merge.| +|FEEDSTORAGE-MONGO-08-001 Causal-consistent session plumbing|BE-Storage|Concelier Core DI|**DONE (2025-10-19)** – Scoped session provider now caches causal-consistent handles per scope, repositories accept optional sessions end-to-end, and new Mongo session consistency tests cover read-your-write + post-stepdown monotonic reads.| +|FEEDSTORAGE-DATA-07-001 Advisory statement & conflict collections|Team Normalization & Storage Backbone|FEEDMERGE-ENGINE-07-001|**DONE (2025-10-19)** – Added immutable `advisory_statements`/`advisory_conflicts` collections, bootstrapper + migration ensuring vulnerability/asOf + hash indexes, new stores (`AdvisoryStatementStore`, `AdvisoryConflictStore`), and docs outlining rollback. Tests: `dotnet test src/StellaOps.Concelier.Storage.Mongo.Tests/StellaOps.Concelier.Storage.Mongo.Tests.csproj`.| diff --git a/src/StellaOps.Concelier.WebService.Tests/PluginLoaderTests.cs b/src/StellaOps.Concelier.WebService.Tests/PluginLoaderTests.cs index db002873..66ce058e 100644 --- a/src/StellaOps.Concelier.WebService.Tests/PluginLoaderTests.cs +++ b/src/StellaOps.Concelier.WebService.Tests/PluginLoaderTests.cs @@ -13,7 +13,7 @@ public class PluginLoaderTests public void ScansConnectorPluginsDirectory() { var services = new NullServices(); - var catalog = new PluginCatalog().AddFromDirectory(Path.Combine(AppContext.BaseDirectory, "PluginBinaries")); + var catalog = new PluginCatalog().AddFromDirectory(Path.Combine(AppContext.BaseDirectory, "StellaOps.Concelier.PluginBinaries")); var plugins = catalog.GetAvailableConnectorPlugins(services); Assert.NotNull(plugins); } @@ -22,7 +22,7 @@ public class PluginLoaderTests public void ScansExporterPluginsDirectory() { var services = new NullServices(); - var catalog = new PluginCatalog().AddFromDirectory(Path.Combine(AppContext.BaseDirectory, "PluginBinaries")); + var catalog = new PluginCatalog().AddFromDirectory(Path.Combine(AppContext.BaseDirectory, "StellaOps.Concelier.PluginBinaries")); var plugins = catalog.GetAvailableExporterPlugins(services); Assert.NotNull(plugins); } diff --git a/src/StellaOps.Concelier.WebService.Tests/WebServiceEndpointsTests.cs b/src/StellaOps.Concelier.WebService.Tests/WebServiceEndpointsTests.cs index cfdd36fe..ea0dae8a 100644 --- a/src/StellaOps.Concelier.WebService.Tests/WebServiceEndpointsTests.cs +++ b/src/StellaOps.Concelier.WebService.Tests/WebServiceEndpointsTests.cs @@ -1,5 +1,7 @@ using System; -using System.Collections.Generic; +using System.Collections.Generic; +using System.Globalization; +using System.IO; using System.Linq; using System.Net; using System.Net.Http.Json; @@ -12,7 +14,9 @@ using Microsoft.Extensions.DependencyInjection; using Microsoft.Extensions.Logging; using Microsoft.Extensions.Options; using Mongo2Go; +using StellaOps.Concelier.Core.Events; using StellaOps.Concelier.Core.Jobs; +using StellaOps.Concelier.Models; using StellaOps.Concelier.WebService.Jobs; using StellaOps.Concelier.WebService.Options; using Xunit.Sdk; @@ -71,9 +75,9 @@ public sealed class WebServiceEndpointsTests : IAsyncLifetime } [Fact] - public async Task JobsEndpointsReturnExpectedStatuses() - { - using var client = _factory.CreateClient(); + public async Task JobsEndpointsReturnExpectedStatuses() + { + using var client = _factory.CreateClient(); var definitions = await client.GetAsync("/jobs/definitions"); if (!definitions.IsSuccessStatusCode) @@ -216,6 +220,167 @@ public sealed class WebServiceEndpointsTests : IAsyncLifetime } } + [Fact] + public async Task AdvisoryReplayEndpointReturnsLatestStatement() + { + var vulnerabilityKey = "CVE-2025-9000"; + var advisory = new Advisory( + advisoryKey: vulnerabilityKey, + title: "Replay Test", + summary: "Example summary", + language: "en", + published: DateTimeOffset.Parse("2025-01-01T00:00:00Z", CultureInfo.InvariantCulture), + modified: DateTimeOffset.Parse("2025-01-02T00:00:00Z", CultureInfo.InvariantCulture), + severity: "medium", + exploitKnown: false, + aliases: new[] { vulnerabilityKey }, + references: Array.Empty(), + affectedPackages: Array.Empty(), + cvssMetrics: Array.Empty(), + provenance: Array.Empty()); + + var statementId = Guid.NewGuid(); + using (var scope = _factory.Services.CreateScope()) + { + var eventLog = scope.ServiceProvider.GetRequiredService(); + var appendRequest = new AdvisoryEventAppendRequest(new[] + { + new AdvisoryStatementInput( + vulnerabilityKey, + advisory, + advisory.Modified ?? advisory.Published ?? DateTimeOffset.UtcNow, + Array.Empty(), + StatementId: statementId, + AdvisoryKey: advisory.AdvisoryKey) + }); + + await eventLog.AppendAsync(appendRequest, CancellationToken.None); + } + + using var client = _factory.CreateClient(); + var response = await client.GetAsync($"/concelier/advisories/{vulnerabilityKey}/replay"); + + Assert.Equal(HttpStatusCode.OK, response.StatusCode); + var payload = await response.Content.ReadFromJsonAsync(); + Assert.NotNull(payload); + Assert.Equal(vulnerabilityKey, payload!.VulnerabilityKey, ignoreCase: true); + var statement = Assert.Single(payload.Statements); + Assert.Equal(statementId, statement.StatementId); + Assert.Equal(advisory.AdvisoryKey, statement.Advisory.AdvisoryKey); + Assert.False(string.IsNullOrWhiteSpace(statement.StatementHash)); + Assert.True(payload.Conflicts is null || payload.Conflicts!.Count == 0); + } + + [Fact] + public async Task MirrorEndpointsServeConfiguredArtifacts() + { + using var temp = new TempDirectory(); + var exportId = "20251019T120000Z"; + var exportRoot = Path.Combine(temp.Path, exportId); + var mirrorRoot = Path.Combine(exportRoot, "mirror"); + var domainRoot = Path.Combine(mirrorRoot, "primary"); + Directory.CreateDirectory(domainRoot); + + await File.WriteAllTextAsync( + Path.Combine(mirrorRoot, "index.json"), + """{"schemaVersion":1,"domains":[]}"""); + await File.WriteAllTextAsync( + Path.Combine(domainRoot, "manifest.json"), + """{"domainId":"primary"}"""); + await File.WriteAllTextAsync( + Path.Combine(domainRoot, "bundle.json"), + """{"advisories":[]}"""); + await File.WriteAllTextAsync( + Path.Combine(domainRoot, "bundle.json.jws"), + "test-signature"); + + var environment = new Dictionary + { + ["CONCELIER_MIRROR__ENABLED"] = "true", + ["CONCELIER_MIRROR__EXPORTROOT"] = temp.Path, + ["CONCELIER_MIRROR__ACTIVEEXPORTID"] = exportId, + ["CONCELIER_MIRROR__DOMAINS__0__ID"] = "primary", + ["CONCELIER_MIRROR__DOMAINS__0__DISPLAYNAME"] = "Primary", + ["CONCELIER_MIRROR__DOMAINS__0__REQUIREAUTHENTICATION"] = "false", + ["CONCELIER_MIRROR__DOMAINS__0__MAXDOWNLOADREQUESTSPERHOUR"] = "5", + ["CONCELIER_MIRROR__MAXINDEXREQUESTSPERHOUR"] = "5" + }; + + using var factory = new ConcelierApplicationFactory(_runner.ConnectionString, environmentOverrides: environment); + using var client = factory.CreateClient(); + + var indexResponse = await client.GetAsync("/concelier/exports/index.json"); + Assert.Equal(HttpStatusCode.OK, indexResponse.StatusCode); + var indexContent = await indexResponse.Content.ReadAsStringAsync(); + Assert.Contains(@"""schemaVersion"":1", indexContent, StringComparison.Ordinal); + + var manifestResponse = await client.GetAsync("/concelier/exports/mirror/primary/manifest.json"); + Assert.Equal(HttpStatusCode.OK, manifestResponse.StatusCode); + var manifestContent = await manifestResponse.Content.ReadAsStringAsync(); + Assert.Contains(@"""domainId"":""primary""", manifestContent, StringComparison.Ordinal); + + var bundleResponse = await client.GetAsync("/concelier/exports/mirror/primary/bundle.json.jws"); + Assert.Equal(HttpStatusCode.OK, bundleResponse.StatusCode); + var signatureContent = await bundleResponse.Content.ReadAsStringAsync(); + Assert.Equal("test-signature", signatureContent); + } + + [Fact] + public async Task MirrorEndpointsEnforceAuthenticationForProtectedDomains() + { + using var temp = new TempDirectory(); + var exportId = "20251019T120000Z"; + var secureRoot = Path.Combine(temp.Path, exportId, "mirror", "secure"); + Directory.CreateDirectory(secureRoot); + + await File.WriteAllTextAsync( + Path.Combine(temp.Path, exportId, "mirror", "index.json"), + """{"schemaVersion":1,"domains":[]}"""); + await File.WriteAllTextAsync( + Path.Combine(secureRoot, "manifest.json"), + """{"domainId":"secure"}"""); + + var environment = new Dictionary + { + ["CONCELIER_MIRROR__ENABLED"] = "true", + ["CONCELIER_MIRROR__EXPORTROOT"] = temp.Path, + ["CONCELIER_MIRROR__ACTIVEEXPORTID"] = exportId, + ["CONCELIER_MIRROR__DOMAINS__0__ID"] = "secure", + ["CONCELIER_MIRROR__DOMAINS__0__REQUIREAUTHENTICATION"] = "true", + ["CONCELIER_MIRROR__DOMAINS__0__MAXDOWNLOADREQUESTSPERHOUR"] = "5", + ["CONCELIER_AUTHORITY__ENABLED"] = "true", + ["CONCELIER_AUTHORITY__ALLOWANONYMOUSFALLBACK"] = "false", + ["CONCELIER_AUTHORITY__ISSUER"] = "https://authority.example", + ["CONCELIER_AUTHORITY__REQUIREHTTPSMETADATA"] = "false", + ["CONCELIER_AUTHORITY__AUDIENCES__0"] = "api://concelier", + ["CONCELIER_AUTHORITY__REQUIREDSCOPES__0"] = StellaOpsScopes.ConcelierJobsTrigger, + ["CONCELIER_AUTHORITY__CLIENTID"] = "concelier-jobs", + ["CONCELIER_AUTHORITY__CLIENTSECRET"] = "secret", + ["CONCELIER_AUTHORITY__CLIENTSCOPES__0"] = StellaOpsScopes.ConcelierJobsTrigger + }; + + using var factory = new ConcelierApplicationFactory( + _runner.ConnectionString, + authority => + { + authority.Enabled = true; + authority.AllowAnonymousFallback = false; + authority.Issuer = "https://authority.example"; + authority.RequireHttpsMetadata = false; + authority.Audiences.Clear(); + authority.Audiences.Add("api://concelier"); + authority.RequiredScopes.Clear(); + authority.RequiredScopes.Add(StellaOpsScopes.ConcelierJobsTrigger); + authority.ClientId = "concelier-jobs"; + authority.ClientSecret = "secret"; + }, + environment); + + using var client = factory.CreateClient(); + var response = await client.GetAsync("/concelier/exports/mirror/secure/manifest.json"); + Assert.Equal(HttpStatusCode.Unauthorized, response.StatusCode); + } + [Fact] public async Task JobsEndpointsAllowBypassWhenAuthorityEnabled() { @@ -365,6 +530,31 @@ public sealed class WebServiceEndpointsTests : IAsyncLifetime Assert.Equal(TimeSpan.FromSeconds(150), options.OfflineCacheTolerance); } + private sealed record ReplayResponse( + string VulnerabilityKey, + DateTimeOffset? AsOf, + List Statements, + List? Conflicts); + + private sealed record ReplayStatement( + Guid StatementId, + string VulnerabilityKey, + string AdvisoryKey, + Advisory Advisory, + string StatementHash, + DateTimeOffset AsOf, + DateTimeOffset RecordedAt, + IReadOnlyList InputDocumentIds); + + private sealed record ReplayConflict( + Guid ConflictId, + string VulnerabilityKey, + IReadOnlyList StatementIds, + string ConflictHash, + DateTimeOffset AsOf, + DateTimeOffset RecordedAt, + string Details); + private sealed class ConcelierApplicationFactory : WebApplicationFactory { private readonly string _connectionString; @@ -417,7 +607,7 @@ public sealed class WebServiceEndpointsTests : IAsyncLifetime { var settings = new Dictionary { - ["Plugins:Directory"] = Path.Combine(context.HostingEnvironment.ContentRootPath, "PluginBinaries"), + ["Plugins:Directory"] = Path.Combine(context.HostingEnvironment.ContentRootPath, "StellaOps.Concelier.PluginBinaries"), }; configurationBuilder.AddInMemoryCollection(settings!); @@ -437,7 +627,7 @@ public sealed class WebServiceEndpointsTests : IAsyncLifetime options.Storage.Driver = "mongo"; options.Storage.Dsn = _connectionString; options.Storage.CommandTimeoutSeconds = 30; - options.Plugins.Directory ??= Path.Combine(AppContext.BaseDirectory, "PluginBinaries"); + options.Plugins.Directory ??= Path.Combine(AppContext.BaseDirectory, "StellaOps.Concelier.PluginBinaries"); options.Telemetry.Enabled = false; options.Telemetry.EnableLogging = false; options.Telemetry.EnableTracing = false; @@ -616,6 +806,32 @@ public sealed class WebServiceEndpointsTests : IAsyncLifetime } } } + + private sealed class TempDirectory : IDisposable + { + public string Path { get; } + + public TempDirectory() + { + Path = System.IO.Path.Combine(System.IO.Path.GetTempPath(), "concelier-mirror-" + Guid.NewGuid().ToString("N", CultureInfo.InvariantCulture)); + Directory.CreateDirectory(Path); + } + + public void Dispose() + { + try + { + if (Directory.Exists(Path)) + { + Directory.Delete(Path, recursive: true); + } + } + catch + { + // best effort cleanup + } + } + } private sealed record HealthPayload(string Status, DateTimeOffset StartedAt, double UptimeSeconds, StoragePayload Storage, TelemetryPayload Telemetry); diff --git a/src/StellaOps.Concelier.WebService/AGENTS.md b/src/StellaOps.Concelier.WebService/AGENTS.md index 49015f15..e09ae40c 100644 --- a/src/StellaOps.Concelier.WebService/AGENTS.md +++ b/src/StellaOps.Concelier.WebService/AGENTS.md @@ -16,7 +16,7 @@ Minimal API host wiring configuration, storage, plugin routines, and job endpoin - GET /jobs/definitions/{kind}/runs?limit= -> recent runs or 404 if kind unknown. - GET /jobs/active -> currently running. - POST /jobs/{*jobKind} with {trigger?,parameters?} -> 202 Accepted (Location:/jobs/{runId}) | 404 | 409 | 423. -- PluginHost defaults: BaseDirectory = solution root; PluginsDirectory = "PluginBinaries"; SearchPatterns += "StellaOps.Concelier.Plugin.*.dll"; EnsureDirectoryExists = true. +- PluginHost defaults: BaseDirectory = solution root; PluginsDirectory = "StellaOps.Concelier.PluginBinaries"; SearchPatterns += "StellaOps.Concelier.Plugin.*.dll"; EnsureDirectoryExists = true. ## Participants - Core job system; Storage.Mongo; Source.Common HTTP clients; Exporter and Connector plugin routines discover/register jobs. ## Interfaces & contracts diff --git a/src/StellaOps.Concelier.WebService/Extensions/MirrorEndpointExtensions.cs b/src/StellaOps.Concelier.WebService/Extensions/MirrorEndpointExtensions.cs new file mode 100644 index 00000000..2d28b069 --- /dev/null +++ b/src/StellaOps.Concelier.WebService/Extensions/MirrorEndpointExtensions.cs @@ -0,0 +1,181 @@ +using System.Globalization; +using Microsoft.AspNetCore.Http; +using Microsoft.Extensions.Options; +using StellaOps.Concelier.WebService.Options; +using StellaOps.Concelier.WebService.Services; + +namespace StellaOps.Concelier.WebService.Extensions; + +internal static class MirrorEndpointExtensions +{ + private const string IndexScope = "index"; + private const string DownloadScope = "download"; + + public static void MapConcelierMirrorEndpoints(this WebApplication app, bool authorityConfigured, bool enforceAuthority) + { + app.MapGet("/concelier/exports/index.json", async ( + MirrorFileLocator locator, + MirrorRateLimiter limiter, + IOptionsMonitor optionsMonitor, + HttpContext context, + CancellationToken cancellationToken) => + { + var mirrorOptions = optionsMonitor.CurrentValue.Mirror ?? new ConcelierOptions.MirrorOptions(); + if (!mirrorOptions.Enabled) + { + return Results.NotFound(); + } + + if (!TryAuthorize(mirrorOptions.RequireAuthentication, enforceAuthority, context, authorityConfigured, out var unauthorizedResult)) + { + return unauthorizedResult; + } + + if (!limiter.TryAcquire("__index__", IndexScope, mirrorOptions.MaxIndexRequestsPerHour, out var retryAfter)) + { + ApplyRetryAfter(context.Response, retryAfter); + return Results.StatusCode(StatusCodes.Status429TooManyRequests); + } + + if (!locator.TryResolveIndex(out var path, out _)) + { + return Results.NotFound(); + } + + return await WriteFileAsync(path, context.Response, "application/json").ConfigureAwait(false); + }); + + app.MapGet("/concelier/exports/{**relativePath}", async ( + string? relativePath, + MirrorFileLocator locator, + MirrorRateLimiter limiter, + IOptionsMonitor optionsMonitor, + HttpContext context, + CancellationToken cancellationToken) => + { + var mirrorOptions = optionsMonitor.CurrentValue.Mirror ?? new ConcelierOptions.MirrorOptions(); + if (!mirrorOptions.Enabled) + { + return Results.NotFound(); + } + + if (string.IsNullOrWhiteSpace(relativePath)) + { + return Results.NotFound(); + } + + if (!locator.TryResolveRelativePath(relativePath, out var path, out _, out var domainId)) + { + return Results.NotFound(); + } + + var domain = FindDomain(mirrorOptions, domainId); + + if (!TryAuthorize(domain?.RequireAuthentication ?? mirrorOptions.RequireAuthentication, enforceAuthority, context, authorityConfigured, out var unauthorizedResult)) + { + return unauthorizedResult; + } + + var limit = domain?.MaxDownloadRequestsPerHour ?? mirrorOptions.MaxIndexRequestsPerHour; + if (!limiter.TryAcquire(domain?.Id ?? "__mirror__", DownloadScope, limit, out var retryAfter)) + { + ApplyRetryAfter(context.Response, retryAfter); + return Results.StatusCode(StatusCodes.Status429TooManyRequests); + } + + var contentType = ResolveContentType(path); + return await WriteFileAsync(path, context.Response, contentType).ConfigureAwait(false); + }); + } + + private static ConcelierOptions.MirrorDomainOptions? FindDomain(ConcelierOptions.MirrorOptions mirrorOptions, string? domainId) + { + if (domainId is null) + { + return null; + } + + foreach (var candidate in mirrorOptions.Domains) + { + if (candidate is null) + { + continue; + } + + if (string.Equals(candidate.Id, domainId, StringComparison.OrdinalIgnoreCase)) + { + return candidate; + } + } + + return null; + } + + private static bool TryAuthorize(bool requireAuthentication, bool enforceAuthority, HttpContext context, bool authorityConfigured, out IResult result) + { + result = Results.Empty; + if (!requireAuthentication) + { + return true; + } + + if (!enforceAuthority || !authorityConfigured) + { + return true; + } + + if (context.User?.Identity?.IsAuthenticated == true) + { + return true; + } + + result = Results.StatusCode(StatusCodes.Status401Unauthorized); + return false; + } + + private static Task WriteFileAsync(string path, HttpResponse response, string contentType) + { + var fileInfo = new FileInfo(path); + if (!fileInfo.Exists) + { + return Task.FromResult(Results.NotFound()); + } + + var stream = new FileStream( + path, + FileMode.Open, + FileAccess.Read, + FileShare.Read | FileShare.Delete); + + response.Headers.CacheControl = "public, max-age=60"; + response.Headers.LastModified = fileInfo.LastWriteTimeUtc.ToString("R", CultureInfo.InvariantCulture); + response.ContentLength = fileInfo.Length; + return Task.FromResult(Results.Stream(stream, contentType)); + } + + private static string ResolveContentType(string path) + { + if (path.EndsWith(".json", StringComparison.OrdinalIgnoreCase)) + { + return "application/json"; + } + + if (path.EndsWith(".jws", StringComparison.OrdinalIgnoreCase)) + { + return "application/jose+json"; + } + + return "application/octet-stream"; + } + + private static void ApplyRetryAfter(HttpResponse response, TimeSpan? retryAfter) + { + if (retryAfter is null) + { + return; + } + + var seconds = Math.Max((int)Math.Ceiling(retryAfter.Value.TotalSeconds), 1); + response.Headers.RetryAfter = seconds.ToString(CultureInfo.InvariantCulture); + } +} diff --git a/src/StellaOps.Concelier.WebService/Options/ConcelierOptions.cs b/src/StellaOps.Concelier.WebService/Options/ConcelierOptions.cs index e23d1acf..3caec155 100644 --- a/src/StellaOps.Concelier.WebService/Options/ConcelierOptions.cs +++ b/src/StellaOps.Concelier.WebService/Options/ConcelierOptions.cs @@ -1,5 +1,6 @@ using System; using System.Collections.Generic; +using System.Text.Json.Serialization; namespace StellaOps.Concelier.WebService.Options; @@ -12,6 +13,8 @@ public sealed class ConcelierOptions public TelemetryOptions Telemetry { get; set; } = new(); public AuthorityOptions Authority { get; set; } = new(); + + public MirrorOptions Mirror { get; set; } = new(); public sealed class StorageOptions { @@ -99,4 +102,37 @@ public sealed class ConcelierOptions public TimeSpan? OfflineCacheTolerance { get; set; } } } + + public sealed class MirrorOptions + { + public bool Enabled { get; set; } + + public string ExportRoot { get; set; } = System.IO.Path.Combine("exports", "json"); + + public string? ActiveExportId { get; set; } + + public string LatestDirectoryName { get; set; } = "latest"; + + public string MirrorDirectoryName { get; set; } = "mirror"; + + public bool RequireAuthentication { get; set; } + + public int MaxIndexRequestsPerHour { get; set; } = 600; + + public IList Domains { get; } = new List(); + + [JsonIgnore] + public string ExportRootAbsolute { get; internal set; } = string.Empty; + } + + public sealed class MirrorDomainOptions + { + public string Id { get; set; } = string.Empty; + + public string? DisplayName { get; set; } + + public bool RequireAuthentication { get; set; } + + public int MaxDownloadRequestsPerHour { get; set; } = 1200; + } } diff --git a/src/StellaOps.Concelier.WebService/Options/ConcelierOptionsPostConfigure.cs b/src/StellaOps.Concelier.WebService/Options/ConcelierOptionsPostConfigure.cs index ac0959ce..3202bb5d 100644 --- a/src/StellaOps.Concelier.WebService/Options/ConcelierOptionsPostConfigure.cs +++ b/src/StellaOps.Concelier.WebService/Options/ConcelierOptionsPostConfigure.cs @@ -42,5 +42,31 @@ public static class ConcelierOptionsPostConfigure authority.ClientSecret = secret; } + + options.Mirror ??= new ConcelierOptions.MirrorOptions(); + var mirror = options.Mirror; + + if (string.IsNullOrWhiteSpace(mirror.ExportRoot)) + { + mirror.ExportRoot = Path.Combine("exports", "json"); + } + + var resolvedRoot = mirror.ExportRoot; + if (!Path.IsPathRooted(resolvedRoot)) + { + resolvedRoot = Path.Combine(contentRootPath, resolvedRoot); + } + + mirror.ExportRootAbsolute = Path.GetFullPath(resolvedRoot); + + if (string.IsNullOrWhiteSpace(mirror.LatestDirectoryName)) + { + mirror.LatestDirectoryName = "latest"; + } + + if (string.IsNullOrWhiteSpace(mirror.MirrorDirectoryName)) + { + mirror.MirrorDirectoryName = "mirror"; + } } } diff --git a/src/StellaOps.Concelier.WebService/Options/ConcelierOptionsValidator.cs b/src/StellaOps.Concelier.WebService/Options/ConcelierOptionsValidator.cs index f480b0f6..7491f86f 100644 --- a/src/StellaOps.Concelier.WebService/Options/ConcelierOptionsValidator.cs +++ b/src/StellaOps.Concelier.WebService/Options/ConcelierOptionsValidator.cs @@ -130,6 +130,9 @@ public static class ConcelierOptionsValidator throw new InvalidOperationException("Telemetry OTLP header names must be non-empty."); } } + + options.Mirror ??= new ConcelierOptions.MirrorOptions(); + ValidateMirror(options.Mirror); } private static void NormalizeList(IList values, bool toLower) @@ -186,4 +189,57 @@ public static class ConcelierOptionsValidator throw new InvalidOperationException("Authority resilience offlineCacheTolerance must be greater than or equal to zero."); } } + + private static void ValidateMirror(ConcelierOptions.MirrorOptions mirror) + { + if (mirror.MaxIndexRequestsPerHour < 0) + { + throw new InvalidOperationException("Mirror maxIndexRequestsPerHour must be greater than or equal to zero."); + } + + if (string.IsNullOrWhiteSpace(mirror.ExportRoot)) + { + throw new InvalidOperationException("Mirror exportRoot must be configured."); + } + + if (string.IsNullOrWhiteSpace(mirror.ExportRootAbsolute)) + { + throw new InvalidOperationException("Mirror export root could not be resolved."); + } + + if (string.IsNullOrWhiteSpace(mirror.LatestDirectoryName)) + { + throw new InvalidOperationException("Mirror latestDirectoryName must be provided."); + } + + if (string.IsNullOrWhiteSpace(mirror.MirrorDirectoryName)) + { + throw new InvalidOperationException("Mirror mirrorDirectoryName must be provided."); + } + + var seen = new HashSet(StringComparer.OrdinalIgnoreCase); + foreach (var domain in mirror.Domains) + { + if (string.IsNullOrWhiteSpace(domain.Id)) + { + throw new InvalidOperationException("Mirror domain id must be provided."); + } + + var normalized = domain.Id.Trim(); + if (!seen.Add(normalized)) + { + throw new InvalidOperationException($"Mirror domain id '{normalized}' is duplicated."); + } + + if (domain.MaxDownloadRequestsPerHour < 0) + { + throw new InvalidOperationException($"Mirror domain '{normalized}' maxDownloadRequestsPerHour must be greater than or equal to zero."); + } + } + + if (mirror.Enabled && mirror.Domains.Count == 0) + { + throw new InvalidOperationException("Mirror distribution requires at least one domain when enabled."); + } + } } diff --git a/src/StellaOps.Concelier.WebService/Program.cs b/src/StellaOps.Concelier.WebService/Program.cs index 05cb54ce..790dc92f 100644 --- a/src/StellaOps.Concelier.WebService/Program.cs +++ b/src/StellaOps.Concelier.WebService/Program.cs @@ -4,8 +4,9 @@ using System.Text; using Microsoft.AspNetCore.Diagnostics; using Microsoft.AspNetCore.Http; using Microsoft.AspNetCore.Mvc; -using Microsoft.Extensions.DependencyInjection; -using Microsoft.Extensions.Hosting; +using Microsoft.Extensions.DependencyInjection; +using Microsoft.Extensions.DependencyInjection.Extensions; +using Microsoft.Extensions.Hosting; using System.Diagnostics; using System.Text.Json; using System.Text.Json.Serialization; @@ -13,7 +14,8 @@ using Microsoft.Extensions.Logging; using Microsoft.Extensions.Options; using MongoDB.Bson; using MongoDB.Driver; -using StellaOps.Concelier.Core.Jobs; +using StellaOps.Concelier.Core.Events; +using StellaOps.Concelier.Core.Jobs; using StellaOps.Concelier.Storage.Mongo; using StellaOps.Concelier.WebService.Diagnostics; using Serilog; @@ -23,6 +25,7 @@ using StellaOps.Concelier.WebService.Extensions; using StellaOps.Concelier.WebService.Jobs; using StellaOps.Concelier.WebService.Options; using StellaOps.Concelier.WebService.Filters; +using StellaOps.Concelier.WebService.Services; using Serilog.Events; using StellaOps.Plugin.DependencyInjection; using StellaOps.Plugin.Hosting; @@ -62,10 +65,15 @@ builder.Services.AddOptions() .ValidateOnStart(); builder.ConfigureConcelierTelemetry(concelierOptions); - -builder.Services.AddMongoStorage(storageOptions => -{ - storageOptions.ConnectionString = concelierOptions.Storage.Dsn; + +builder.Services.TryAddSingleton(_ => TimeProvider.System); +builder.Services.AddMemoryCache(); +builder.Services.AddSingleton(); +builder.Services.AddSingleton(); + +builder.Services.AddMongoStorage(storageOptions => +{ + storageOptions.ConnectionString = concelierOptions.Storage.Dsn; storageOptions.DatabaseName = concelierOptions.Storage.Database; storageOptions.CommandTimeout = TimeSpan.FromSeconds(concelierOptions.Storage.CommandTimeoutSeconds); }); @@ -174,9 +182,58 @@ if (resolvedAuthority.Enabled && resolvedAuthority.AllowAnonymousFallback) "Authority authentication is configured but anonymous fallback remains enabled. Set authority.allowAnonymousFallback to false before 2025-12-31 to complete the rollout."); } +app.MapConcelierMirrorEndpoints(authorityConfigured, enforceAuthority); + var jsonOptions = new JsonSerializerOptions(JsonSerializerDefaults.Web); jsonOptions.Converters.Add(new JsonStringEnumConverter()); +app.MapGet("/concelier/advisories/{vulnerabilityKey}/replay", async ( + string vulnerabilityKey, + DateTimeOffset? asOf, + IAdvisoryEventLog eventLog, + CancellationToken cancellationToken) => +{ + if (string.IsNullOrWhiteSpace(vulnerabilityKey)) + { + return Results.BadRequest("vulnerabilityKey must be provided."); + } + + var replay = await eventLog.ReplayAsync(vulnerabilityKey.Trim(), asOf, cancellationToken).ConfigureAwait(false); + if (replay.Statements.Length == 0 && replay.Conflicts.Length == 0) + { + return Results.NotFound(); + } + + var response = new + { + replay.VulnerabilityKey, + replay.AsOf, + Statements = replay.Statements.Select(statement => new + { + statement.StatementId, + statement.VulnerabilityKey, + statement.AdvisoryKey, + statement.Advisory, + StatementHash = Convert.ToHexString(statement.StatementHash.ToArray()), + statement.AsOf, + statement.RecordedAt, + InputDocumentIds = statement.InputDocumentIds + }).ToArray(), + Conflicts = replay.Conflicts.Select(conflict => new + { + conflict.ConflictId, + conflict.VulnerabilityKey, + conflict.StatementIds, + ConflictHash = Convert.ToHexString(conflict.ConflictHash.ToArray()), + conflict.AsOf, + conflict.RecordedAt, + Details = conflict.CanonicalJson + }).ToArray() + }; + + return JsonResult(response); +}); + var loggingEnabled = concelierOptions.Telemetry?.EnableLogging ?? true; if (loggingEnabled) @@ -660,8 +717,9 @@ static PluginHostOptions BuildPluginOptions(ConcelierOptions options, string con { var pluginOptions = new PluginHostOptions { - BaseDirectory = options.Plugins.BaseDirectory ?? contentRoot, - PluginsDirectory = options.Plugins.Directory ?? Path.Combine(contentRoot, "PluginBinaries"), + BaseDirectory = options.Plugins.BaseDirectory ?? contentRoot, + PluginsDirectory = options.Plugins.Directory ?? Path.Combine(contentRoot, "StellaOps.Concelier.PluginBinaries"), + PrimaryPrefix = "StellaOps.Concelier", EnsureDirectoryExists = true, RecursiveSearch = false, }; diff --git a/src/StellaOps.Concelier.WebService/Services/MirrorFileLocator.cs b/src/StellaOps.Concelier.WebService/Services/MirrorFileLocator.cs new file mode 100644 index 00000000..7078e1a5 --- /dev/null +++ b/src/StellaOps.Concelier.WebService/Services/MirrorFileLocator.cs @@ -0,0 +1,184 @@ +using System; +using System.Diagnostics.CodeAnalysis; +using System.Globalization; +using System.IO; +using Microsoft.Extensions.Logging; +using Microsoft.Extensions.Options; +using StellaOps.Concelier.WebService.Options; + +namespace StellaOps.Concelier.WebService.Services; + +internal sealed class MirrorFileLocator +{ + private readonly IOptionsMonitor _options; + private readonly ILogger _logger; + + public MirrorFileLocator(IOptionsMonitor options, ILogger logger) + { + _options = options ?? throw new ArgumentNullException(nameof(options)); + _logger = logger ?? throw new ArgumentNullException(nameof(logger)); + } + + public bool TryResolveIndex([NotNullWhen(true)] out string? path, [NotNullWhen(true)] out string? exportId) + => TryResolveRelativePath("index.json", out path, out exportId, out _); + + public bool TryResolveRelativePath(string relativePath, [NotNullWhen(true)] out string? fullPath, [NotNullWhen(true)] out string? exportId, out string? domainId) + { + fullPath = null; + exportId = null; + domainId = null; + + var mirror = _options.CurrentValue.Mirror ?? new ConcelierOptions.MirrorOptions(); + if (!mirror.Enabled) + { + return false; + } + + if (!TryResolveExportDirectory(mirror, out var exportDirectory, out exportId)) + { + return false; + } + + var sanitized = SanitizeRelativePath(relativePath); + if (sanitized.Length == 0 || string.Equals(sanitized, "index.json", StringComparison.OrdinalIgnoreCase)) + { + sanitized = $"{mirror.MirrorDirectoryName}/index.json"; + } + + if (!sanitized.StartsWith($"{mirror.MirrorDirectoryName}/", StringComparison.OrdinalIgnoreCase)) + { + return false; + } + + var candidate = Combine(exportDirectory, sanitized); + if (!CandidateWithinExport(exportDirectory, candidate)) + { + _logger.LogWarning("Rejected mirror export request for path '{RelativePath}' due to traversal attempt.", relativePath); + return false; + } + + if (!File.Exists(candidate)) + { + return false; + } + + // Extract domain id from path mirror//... + var segments = sanitized.Split('/', StringSplitOptions.RemoveEmptyEntries); + if (segments.Length >= 2) + { + domainId = segments[1]; + } + + fullPath = candidate; + return true; + } + + private bool TryResolveExportDirectory(ConcelierOptions.MirrorOptions mirror, [NotNullWhen(true)] out string? exportDirectory, [NotNullWhen(true)] out string? exportId) + { + exportDirectory = null; + exportId = null; + + if (string.IsNullOrWhiteSpace(mirror.ExportRootAbsolute)) + { + _logger.LogWarning("Mirror export root is not configured; unable to serve mirror content."); + return false; + } + + var root = mirror.ExportRootAbsolute; + var candidateSegment = string.IsNullOrWhiteSpace(mirror.ActiveExportId) + ? mirror.LatestDirectoryName + : mirror.ActiveExportId!; + + if (TryResolveCandidate(root, candidateSegment, mirror.MirrorDirectoryName, out exportDirectory, out exportId)) + { + return true; + } + + if (!string.Equals(candidateSegment, mirror.LatestDirectoryName, StringComparison.OrdinalIgnoreCase) + && TryResolveCandidate(root, mirror.LatestDirectoryName, mirror.MirrorDirectoryName, out exportDirectory, out exportId)) + { + return true; + } + + try + { + var directories = Directory.Exists(root) + ? Directory.GetDirectories(root) + : Array.Empty(); + + Array.Sort(directories, StringComparer.Ordinal); + Array.Reverse(directories); + + foreach (var directory in directories) + { + if (TryResolveCandidate(root, Path.GetFileName(directory), mirror.MirrorDirectoryName, out exportDirectory, out exportId)) + { + return true; + } + } + } + catch (Exception ex) when (ex is IOException or UnauthorizedAccessException) + { + _logger.LogWarning(ex, "Failed to enumerate export directories under {Root}.", root); + } + + return false; + } + + private bool TryResolveCandidate(string root, string segment, string mirrorDirectory, [NotNullWhen(true)] out string? exportDirectory, [NotNullWhen(true)] out string? exportId) + { + exportDirectory = null; + exportId = null; + + if (string.IsNullOrWhiteSpace(segment)) + { + return false; + } + + var candidate = Path.Combine(root, segment); + if (!Directory.Exists(candidate)) + { + return false; + } + + var mirrorPath = Path.Combine(candidate, mirrorDirectory); + if (!Directory.Exists(mirrorPath)) + { + return false; + } + + exportDirectory = candidate; + exportId = segment; + return true; + } + + private static string SanitizeRelativePath(string relativePath) + { + if (string.IsNullOrWhiteSpace(relativePath)) + { + return string.Empty; + } + + var trimmed = relativePath.Replace('\\', '/').Trim().TrimStart('/'); + return trimmed; + } + + private static string Combine(string root, string relativePath) + { + var segments = relativePath.Split('/', StringSplitOptions.RemoveEmptyEntries); + if (segments.Length == 0) + { + return Path.GetFullPath(root); + } + + var combinedRelative = Path.Combine(segments); + return Path.GetFullPath(Path.Combine(root, combinedRelative)); + } + + private static bool CandidateWithinExport(string exportDirectory, string candidate) + { + var exportRoot = Path.GetFullPath(exportDirectory).TrimEnd(Path.DirectorySeparatorChar, Path.AltDirectorySeparatorChar); + var candidatePath = Path.GetFullPath(candidate); + return candidatePath.StartsWith(exportRoot, StringComparison.OrdinalIgnoreCase); + } +} diff --git a/src/StellaOps.Concelier.WebService/Services/MirrorRateLimiter.cs b/src/StellaOps.Concelier.WebService/Services/MirrorRateLimiter.cs new file mode 100644 index 00000000..472b3a6d --- /dev/null +++ b/src/StellaOps.Concelier.WebService/Services/MirrorRateLimiter.cs @@ -0,0 +1,57 @@ +using Microsoft.Extensions.Caching.Memory; + +namespace StellaOps.Concelier.WebService.Services; + +internal sealed class MirrorRateLimiter +{ + private readonly IMemoryCache _cache; + private readonly TimeProvider _timeProvider; + private static readonly TimeSpan Window = TimeSpan.FromHours(1); + + public MirrorRateLimiter(IMemoryCache cache, TimeProvider timeProvider) + { + _cache = cache ?? throw new ArgumentNullException(nameof(cache)); + _timeProvider = timeProvider ?? throw new ArgumentNullException(nameof(timeProvider)); + } + + public bool TryAcquire(string domainId, string scope, int limit, out TimeSpan? retryAfter) + { + retryAfter = null; + + if (limit <= 0 || limit == int.MaxValue) + { + return true; + } + + var key = CreateKey(domainId, scope); + var now = _timeProvider.GetUtcNow(); + + var counter = _cache.Get(key); + if (counter is null || now - counter.WindowStart >= Window) + { + counter = new Counter(now, 0); + } + + if (counter.Count >= limit) + { + var windowEnd = counter.WindowStart + Window; + retryAfter = windowEnd > now ? windowEnd - now : TimeSpan.Zero; + return false; + } + + counter = counter with { Count = counter.Count + 1 }; + var absoluteExpiration = counter.WindowStart + Window; + _cache.Set(key, counter, absoluteExpiration); + return true; + } + + private static string CreateKey(string domainId, string scope) + => string.Create(domainId.Length + scope.Length + 1, (domainId, scope), static (span, state) => + { + state.domainId.AsSpan().CopyTo(span); + span[state.domainId.Length] = '|'; + state.scope.AsSpan().CopyTo(span[(state.domainId.Length + 1)..]); + }); + + private sealed record Counter(DateTimeOffset WindowStart, int Count); +} diff --git a/src/StellaOps.Concelier.WebService/TASKS.md b/src/StellaOps.Concelier.WebService/TASKS.md index 8d4e5c6f..8e635265 100644 --- a/src/StellaOps.Concelier.WebService/TASKS.md +++ b/src/StellaOps.Concelier.WebService/TASKS.md @@ -1,18 +1,19 @@ -# TASKS -| Task | Owner(s) | Depends on | Notes | -|---|---|---|---| -|Bind & validate ConcelierOptions|BE-Base|WebService|DONE – options bound/validated with failure logging.| -|Mongo service wiring|BE-Base|Storage.Mongo|DONE – wiring delegated to `AddMongoStorage`.| -|Bootstrapper execution on start|BE-Base|Storage.Mongo|DONE – startup calls `MongoBootstrapper.InitializeAsync`.| -|Plugin host options finalization|BE-Base|Plugins|DONE – default plugin directories/search patterns configured.| -|Jobs API contract tests|QA|Core|DONE – WebServiceEndpointsTests now cover success payloads, filtering, and trigger outcome mapping.| -|Health/Ready probes|DevOps|Ops|DONE – `/health` and `/ready` endpoints implemented.| -|Serilog + OTEL integration hooks|BE-Base|Observability|DONE – `TelemetryExtensions` wires Serilog + OTEL with configurable exporters.| -|Register built-in jobs (sources/exporters)|BE-Base|Core|DONE – AddBuiltInConcelierJobs adds fallback scheduler definitions for core connectors and exporters via reflection.| -|HTTP problem details consistency|BE-Base|WebService|DONE – API errors now emit RFC7807 responses with trace identifiers and typed problem categories.| -|Request logging and metrics|BE-Base|Observability|DONE – Serilog request logging enabled with enriched context and web.jobs counters published via OpenTelemetry.| -|Endpoint smoke tests (health/ready/jobs error paths)|QA|WebService|DONE – WebServiceEndpointsTests assert success and problem responses for health, ready, and job trigger error paths.| -|Batch job definition last-run lookup|BE-Base|Core|DONE – definitions endpoint now precomputes kinds array and reuses batched last-run dictionary; manual smoke verified via local GET `/jobs/definitions`.| +# TASKS +| Task | Owner(s) | Depends on | Notes | +|---|---|---|---| +|FEEDWEB-EVENTS-07-001 Advisory event replay API|Concelier WebService Guild|FEEDCORE-ENGINE-07-001|**DONE (2025-10-19)** – Added `/concelier/advisories/{vulnerabilityKey}/replay` endpoint with optional `asOf`, hex hashes, and conflict payloads; integration covered via `dotnet test src/StellaOps.Concelier.WebService.Tests/StellaOps.Concelier.WebService.Tests.csproj`.| +|Bind & validate ConcelierOptions|BE-Base|WebService|DONE – options bound/validated with failure logging.| +|Mongo service wiring|BE-Base|Storage.Mongo|DONE – wiring delegated to `AddMongoStorage`.| +|Bootstrapper execution on start|BE-Base|Storage.Mongo|DONE – startup calls `MongoBootstrapper.InitializeAsync`.| +|Plugin host options finalization|BE-Base|Plugins|DONE – default plugin directories/search patterns configured.| +|Jobs API contract tests|QA|Core|DONE – WebServiceEndpointsTests now cover success payloads, filtering, and trigger outcome mapping.| +|Health/Ready probes|DevOps|Ops|DONE – `/health` and `/ready` endpoints implemented.| +|Serilog + OTEL integration hooks|BE-Base|Observability|DONE – `TelemetryExtensions` wires Serilog + OTEL with configurable exporters.| +|Register built-in jobs (sources/exporters)|BE-Base|Core|DONE – AddBuiltInConcelierJobs adds fallback scheduler definitions for core connectors and exporters via reflection.| +|HTTP problem details consistency|BE-Base|WebService|DONE – API errors now emit RFC7807 responses with trace identifiers and typed problem categories.| +|Request logging and metrics|BE-Base|Observability|DONE – Serilog request logging enabled with enriched context and web.jobs counters published via OpenTelemetry.| +|Endpoint smoke tests (health/ready/jobs error paths)|QA|WebService|DONE – WebServiceEndpointsTests assert success and problem responses for health, ready, and job trigger error paths.| +|Batch job definition last-run lookup|BE-Base|Core|DONE – definitions endpoint now precomputes kinds array and reuses batched last-run dictionary; manual smoke verified via local GET `/jobs/definitions`.| |Add no-cache headers to health/readiness/jobs APIs|BE-Base|WebService|DONE – helper applies Cache-Control/Pragma/Expires on all health/ready/jobs endpoints; awaiting automated probe tests once connector fixtures stabilize.| |Authority configuration parity (FSR1)|DevEx/Concelier|Authority options schema|**DONE (2025-10-10)** – Options post-config loads clientSecretFile fallback, validators normalize scopes/audiences, and sample config documents issuer/credential/bypass settings.| |Document authority toggle & scope requirements|Docs/Concelier|Authority integration|**DOING (2025-10-10)** – Quickstart updated with staging flag, client credentials, env overrides; operator guide refresh pending Docs guild review.| @@ -20,6 +21,7 @@ |Author ops guidance for resilience tuning|Docs/Concelier|Plumb Authority client resilience options|**DONE (2025-10-12)** – `docs/21_INSTALL_GUIDE.md` + `docs/ops/concelier-authority-audit-runbook.md` document resilience profiles for connected vs air-gapped installs and reference monitoring cues.| |Document authority bypass logging patterns|Docs/Concelier|FSR3 logging|**DONE (2025-10-12)** – Updated operator guides clarify `Concelier.Authorization.Audit` fields (route/status/subject/clientId/scopes/bypass/remote) and SIEM triggers.| |Update Concelier operator guide for enforcement cutoff|Docs/Concelier|FSR1 rollout|**DONE (2025-10-12)** – Installation guide emphasises disabling `allowAnonymousFallback` before 2025-12-31 UTC and connects audit signals to the rollout checklist.| -|Rename plugin drop directory to namespaced path|BE-Base|Plugins|**TODO** – Point Concelier source/exporter build outputs to `StellaOps.Concelier.PluginBinaries`, update PluginHost defaults/search patterns to match, ensure Offline Kit packaging/tests expect the new folder, and document migration guidance for operators.| +|Rename plugin drop directory to namespaced path|BE-Base|Plugins|**DONE (2025-10-19)** – Build outputs now target `StellaOps.Concelier.PluginBinaries`/`StellaOps.Authority.PluginBinaries`, plugin host defaults updated, config/docs refreshed, and `dotnet test src/StellaOps.Concelier.WebService.Tests/StellaOps.Concelier.WebService.Tests.csproj --no-restore` covers the change.| |Authority resilience adoption|Concelier WebService, Docs|Plumb Authority client resilience options|**BLOCKED (2025-10-10)** – Roll out retry/offline knobs to deployment docs and confirm CLI parity once LIB5 lands; unblock after resilience options wired and tested.| -|CONCELIER-WEB-08-201 – Mirror distribution endpoints|Concelier WebService Guild|CONCELIER-EXPORT-08-201, DEVOPS-MIRROR-08-001|TODO – Add domain-scoped mirror configuration (`*.stella-ops.org`), expose signed export index/download APIs with quota and auth, and document sync workflow for downstream Concelier instances.| +|CONCELIER-WEB-08-201 – Mirror distribution endpoints|Concelier WebService Guild|CONCELIER-EXPORT-08-201, DEVOPS-MIRROR-08-001|DOING (2025-10-19) – HTTP endpoints wired (`/concelier/exports/index.json`, `/concelier/exports/mirror/*`), mirror options bound/validated, and integration tests added; pending auth docs + smoke in ops handbook.| +|Wave 0B readiness checkpoint|Team WebService & Authority|Wave 0A completion|BLOCKED (2025-10-19) – FEEDSTORAGE-MONGO-08-001 closed, but remaining Wave 0A items (AUTH-DPOP-11-001, AUTH-MTLS-11-002, PLUGIN-DI-08-001) still open; maintain current DOING workstreams only.| diff --git a/src/StellaOps.Configuration/StellaOpsAuthorityOptions.cs b/src/StellaOps.Configuration/StellaOpsAuthorityOptions.cs index 0baec328..b828754a 100644 --- a/src/StellaOps.Configuration/StellaOpsAuthorityOptions.cs +++ b/src/StellaOps.Configuration/StellaOpsAuthorityOptions.cs @@ -2,6 +2,7 @@ using System; using System.Collections.Generic; using System.IO; using System.Linq; +using System.Text.RegularExpressions; using System.Threading.RateLimiting; using StellaOps.Authority.Plugins.Abstractions; using StellaOps.Cryptography; @@ -274,7 +275,7 @@ public sealed class AuthorityDpopOptions "ES384" }; - public bool Enabled { get; set; } = true; + public bool Enabled { get; set; } public TimeSpan ProofLifetime { get; set; } = TimeSpan.FromMinutes(2); @@ -338,6 +339,10 @@ public sealed class AuthorityDpopNonceOptions public int MaxIssuancePerMinute { get; set; } = 120; + public string Store { get; set; } = "memory"; + + public string? RedisConnectionString { get; set; } + public ISet RequiredAudiences => requiredAudiences; public IReadOnlySet NormalizedAudiences { get; private set; } = new HashSet(StringComparer.OrdinalIgnoreCase); @@ -354,6 +359,23 @@ public sealed class AuthorityDpopNonceOptions throw new InvalidOperationException("Dpop.Nonce.MaxIssuancePerMinute must be at least 1."); } + if (string.IsNullOrWhiteSpace(Store)) + { + throw new InvalidOperationException("Dpop.Nonce.Store must be specified."); + } + + Store = Store.Trim().ToLowerInvariant(); + + if (Store is not ("memory" or "redis")) + { + throw new InvalidOperationException("Dpop.Nonce.Store must be either 'memory' or 'redis'."); + } + + if (Store == "redis" && string.IsNullOrWhiteSpace(RedisConnectionString)) + { + throw new InvalidOperationException("Dpop.Nonce.RedisConnectionString must be provided when using the 'redis' store."); + } + NormalizedAudiences = requiredAudiences .Select(static aud => aud.Trim()) .Where(static aud => aud.Length > 0) @@ -373,6 +395,12 @@ public sealed class AuthorityMtlsOptions "signer" }; + private readonly HashSet allowedSanTypes = new(StringComparer.OrdinalIgnoreCase) + { + "dns", + "uri" + }; + public bool Enabled { get; set; } public bool RequireChainValidation { get; set; } = true; @@ -385,6 +413,14 @@ public sealed class AuthorityMtlsOptions public IList AllowedCertificateAuthorities { get; } = new List(); + public IList AllowedSubjectPatterns { get; } = new List(); + + public ISet AllowedSanTypes => allowedSanTypes; + + public IReadOnlyList NormalizedSubjectPatterns { get; private set; } = Array.Empty(); + + public IReadOnlySet NormalizedSanTypes { get; private set; } = new HashSet(StringComparer.OrdinalIgnoreCase); + internal void Validate() { if (RotationGrace < TimeSpan.Zero) @@ -406,6 +442,38 @@ public sealed class AuthorityMtlsOptions { throw new InvalidOperationException("Mtls.AllowedCertificateAuthorities entries must not be empty."); } + + NormalizedSanTypes = allowedSanTypes + .Select(static value => value.Trim()) + .Where(static value => value.Length > 0) + .Select(static value => value.ToLowerInvariant()) + .ToHashSet(StringComparer.OrdinalIgnoreCase); + + if (Enabled && NormalizedSanTypes.Count == 0) + { + throw new InvalidOperationException("Mtls.AllowedSanTypes must include at least one entry when enabled."); + } + + var compiledPatterns = new List(AllowedSubjectPatterns.Count); + + foreach (var pattern in AllowedSubjectPatterns) + { + if (string.IsNullOrWhiteSpace(pattern)) + { + throw new InvalidOperationException("Mtls.AllowedSubjectPatterns entries must not be empty."); + } + + try + { + compiledPatterns.Add(new Regex(pattern, RegexOptions.CultureInvariant | RegexOptions.IgnoreCase | RegexOptions.Compiled, TimeSpan.FromMilliseconds(100))); + } + catch (RegexParseException ex) + { + throw new InvalidOperationException($"Mtls.AllowedSubjectPatterns entry '{pattern}' is not a valid regular expression.", ex); + } + } + + NormalizedSubjectPatterns = compiledPatterns; } } diff --git a/src/StellaOps.DependencyInjection/ServiceBindingAttribute.cs b/src/StellaOps.DependencyInjection/ServiceBindingAttribute.cs new file mode 100644 index 00000000..1b085657 --- /dev/null +++ b/src/StellaOps.DependencyInjection/ServiceBindingAttribute.cs @@ -0,0 +1,64 @@ +using System; +using Microsoft.Extensions.DependencyInjection; + +namespace StellaOps.DependencyInjection; + +/// +/// Declares how a plug-in type should be registered with the host dependency injection container. +/// +[AttributeUsage(AttributeTargets.Class, AllowMultiple = true, Inherited = false)] +public sealed class ServiceBindingAttribute : Attribute +{ + /// + /// Creates a binding that registers the decorated type as itself with a singleton lifetime. + /// + public ServiceBindingAttribute() + : this(null, ServiceLifetime.Singleton) + { + } + + /// + /// Creates a binding that registers the decorated type as itself with the specified lifetime. + /// + public ServiceBindingAttribute(ServiceLifetime lifetime) + : this(null, lifetime) + { + } + + /// + /// Creates a binding that registers the decorated type as the specified service type with a singleton lifetime. + /// + public ServiceBindingAttribute(Type serviceType) + : this(serviceType, ServiceLifetime.Singleton) + { + } + + /// + /// Creates a binding that registers the decorated type as the specified service type. + /// + public ServiceBindingAttribute(Type? serviceType, ServiceLifetime lifetime) + { + ServiceType = serviceType; + Lifetime = lifetime; + } + + /// + /// The service contract that should resolve to the decorated implementation. When null, the implementation registers itself. + /// + public Type? ServiceType { get; } + + /// + /// The lifetime that should be used when registering the decorated implementation. + /// + public ServiceLifetime Lifetime { get; } + + /// + /// Indicates whether existing descriptors for the same service type should be removed before this binding is applied. + /// + public bool ReplaceExisting { get; init; } + + /// + /// When true, the implementation is also registered as itself even if a service type is specified. + /// + public bool RegisterAsSelf { get; init; } +} diff --git a/src/StellaOps.Excititor.Core/VexScoreEnvelope.cs b/src/StellaOps.Excititor.Core/VexScoreEnvelope.cs index 98e19253..122e4d4b 100644 --- a/src/StellaOps.Excititor.Core/VexScoreEnvelope.cs +++ b/src/StellaOps.Excititor.Core/VexScoreEnvelope.cs @@ -1,37 +1,12 @@ +using System.Collections.Generic; using System.Collections.Immutable; using System.Linq; namespace StellaOps.Excititor.Core; -public sealed record VexScoreEnvelope( - DateTimeOffset GeneratedAt, - string PolicyRevisionId, - string? PolicyDigest, - double Alpha, - double Beta, - double WeightCeiling, - ImmutableArray Entries) +public sealed record VexScoreEnvelope { public VexScoreEnvelope( - DateTimeOffset GeneratedAt, - string PolicyRevisionId, - string? PolicyDigest, - double Alpha, - double Beta, - double WeightCeiling, - IEnumerable Entries) - : this( - GeneratedAt, - PolicyRevisionId, - PolicyDigest, - Alpha, - Beta, - WeightCeiling, - NormalizeEntries(Entries)) - { - } - - private VexScoreEnvelope( DateTimeOffset generatedAt, string policyRevisionId, string? policyDigest, @@ -60,13 +35,32 @@ public sealed record VexScoreEnvelope( throw new ArgumentOutOfRangeException(nameof(weightCeiling), "Weight ceiling must be a finite number greater than zero."); } - this.GeneratedAt = generatedAt; - this.PolicyRevisionId = policyRevisionId.Trim(); - this.PolicyDigest = string.IsNullOrWhiteSpace(policyDigest) ? null : policyDigest.Trim(); - this.Alpha = alpha; - this.Beta = beta; - this.WeightCeiling = weightCeiling; - this.Entries = entries; + GeneratedAt = generatedAt; + PolicyRevisionId = policyRevisionId.Trim(); + PolicyDigest = string.IsNullOrWhiteSpace(policyDigest) ? null : policyDigest.Trim(); + Alpha = alpha; + Beta = beta; + WeightCeiling = weightCeiling; + Entries = entries; + } + + public VexScoreEnvelope( + DateTimeOffset generatedAt, + string policyRevisionId, + string? policyDigest, + double alpha, + double beta, + double weightCeiling, + IEnumerable entries) + : this( + generatedAt, + policyRevisionId, + policyDigest, + alpha, + beta, + weightCeiling, + NormalizeEntries(entries)) + { } public DateTimeOffset GeneratedAt { get; } @@ -97,32 +91,9 @@ public sealed record VexScoreEnvelope( } } -public sealed record VexScoreEntry( - string VulnerabilityId, - string ProductKey, - VexConsensusStatus Status, - DateTimeOffset CalculatedAt, - VexSignalSnapshot? Signals, - double? Score) +public sealed record VexScoreEntry { public VexScoreEntry( - string VulnerabilityId, - string ProductKey, - VexConsensusStatus Status, - DateTimeOffset CalculatedAt, - VexSignalSnapshot? Signals, - double? Score) - : this( - ValidateVulnerability(VulnerabilityId), - ValidateProduct(ProductKey), - Status, - CalculatedAt, - Signals, - ValidateScore(Score)) - { - } - - private VexScoreEntry( string vulnerabilityId, string productKey, VexConsensusStatus status, @@ -130,12 +101,12 @@ public sealed record VexScoreEntry( VexSignalSnapshot? signals, double? score) { - VulnerabilityId = vulnerabilityId; - ProductKey = productKey; + VulnerabilityId = ValidateVulnerability(vulnerabilityId); + ProductKey = ValidateProduct(productKey); Status = status; CalculatedAt = calculatedAt; Signals = signals; - Score = score; + Score = ValidateScore(score); } public string VulnerabilityId { get; } diff --git a/src/StellaOps.Excititor.Export.Tests/ExportEngineTests.cs b/src/StellaOps.Excititor.Export.Tests/ExportEngineTests.cs index 982f95a7..9fb74667 100644 --- a/src/StellaOps.Excititor.Export.Tests/ExportEngineTests.cs +++ b/src/StellaOps.Excititor.Export.Tests/ExportEngineTests.cs @@ -3,6 +3,7 @@ using System.Collections.Immutable; using System.IO; using System.Text; using Microsoft.Extensions.Logging.Abstractions; +using MongoDB.Driver; using StellaOps.Excititor.Core; using StellaOps.Excititor.Export; using StellaOps.Excititor.Policy; @@ -128,14 +129,14 @@ public sealed class ExportEngineTests public VexExportManifest? LastSavedManifest { get; private set; } - public ValueTask FindAsync(VexQuerySignature signature, VexExportFormat format, CancellationToken cancellationToken) + public ValueTask FindAsync(VexQuerySignature signature, VexExportFormat format, CancellationToken cancellationToken, IClientSessionHandle? session = null) { var key = CreateKey(signature.Value, format); _store.TryGetValue(key, out var manifest); return ValueTask.FromResult(manifest); } - public ValueTask SaveAsync(VexExportManifest manifest, CancellationToken cancellationToken) + public ValueTask SaveAsync(VexExportManifest manifest, CancellationToken cancellationToken, IClientSessionHandle? session = null) { var key = CreateKey(manifest.QuerySignature.Value, manifest.Format); _store[key] = manifest; diff --git a/src/StellaOps.Excititor.Export.Tests/VexExportCacheServiceTests.cs b/src/StellaOps.Excititor.Export.Tests/VexExportCacheServiceTests.cs index 855b52b4..e60ee52b 100644 --- a/src/StellaOps.Excititor.Export.Tests/VexExportCacheServiceTests.cs +++ b/src/StellaOps.Excititor.Export.Tests/VexExportCacheServiceTests.cs @@ -1,4 +1,5 @@ using Microsoft.Extensions.Logging.Abstractions; +using MongoDB.Driver; using StellaOps.Excititor.Core; using StellaOps.Excititor.Export; using StellaOps.Excititor.Storage.Mongo; @@ -72,10 +73,10 @@ public sealed class VexExportCacheServiceTests public int ExpiredCount { get; set; } public int DanglingCount { get; set; } - public ValueTask RemoveExpiredAsync(DateTimeOffset asOf, CancellationToken cancellationToken) + public ValueTask RemoveExpiredAsync(DateTimeOffset asOf, CancellationToken cancellationToken, IClientSessionHandle? session = null) => ValueTask.FromResult(ExpiredCount); - public ValueTask RemoveMissingManifestReferencesAsync(CancellationToken cancellationToken) + public ValueTask RemoveMissingManifestReferencesAsync(CancellationToken cancellationToken, IClientSessionHandle? session = null) => ValueTask.FromResult(DanglingCount); } } diff --git a/src/StellaOps.Excititor.Storage.Mongo.Tests/MongoVexRepositoryTests.cs b/src/StellaOps.Excititor.Storage.Mongo.Tests/MongoVexRepositoryTests.cs index 3e4abdc1..0b0711eb 100644 --- a/src/StellaOps.Excititor.Storage.Mongo.Tests/MongoVexRepositoryTests.cs +++ b/src/StellaOps.Excititor.Storage.Mongo.Tests/MongoVexRepositoryTests.cs @@ -108,7 +108,8 @@ public sealed class MongoVexRepositoryTests : IAsyncLifetime GridFsInlineThresholdBytes = 64, }); - var store = new MongoVexExportStore(_client, database, options); + var sessionProvider = new VexMongoSessionProvider(_client, options); + var store = new MongoVexExportStore(_client, database, options, sessionProvider); var signature = new VexQuerySignature("format=csaf|provider=redhat"); var manifest = new VexExportManifest( "exports/20251016/redhat", @@ -152,7 +153,8 @@ public sealed class MongoVexRepositoryTests : IAsyncLifetime GridFsInlineThresholdBytes = 64, }); - var store = new MongoVexExportStore(_client, database, options); + var sessionProvider = new VexMongoSessionProvider(_client, options); + var store = new MongoVexExportStore(_client, database, options, sessionProvider); var signature = new VexQuerySignature("format=json|provider=cisco"); var manifest = new VexExportManifest( "exports/20251016/cisco", @@ -263,7 +265,8 @@ public sealed class MongoVexRepositoryTests : IAsyncLifetime ExportCacheTtl = TimeSpan.FromHours(1), }); - return new MongoVexRawStore(_client, database, options); + var sessionProvider = new VexMongoSessionProvider(_client, options); + return new MongoVexRawStore(_client, database, options, sessionProvider); } private static string BuildExportKey(VexQuerySignature signature, VexExportFormat format) diff --git a/src/StellaOps.Excititor.Storage.Mongo.Tests/MongoVexSessionConsistencyTests.cs b/src/StellaOps.Excititor.Storage.Mongo.Tests/MongoVexSessionConsistencyTests.cs new file mode 100644 index 00000000..75b43c53 --- /dev/null +++ b/src/StellaOps.Excititor.Storage.Mongo.Tests/MongoVexSessionConsistencyTests.cs @@ -0,0 +1,184 @@ +using System; +using System.Threading; +using System.Threading.Tasks; +using Microsoft.Extensions.DependencyInjection; +using Microsoft.Extensions.Logging; +using Microsoft.Extensions.Options; +using Mongo2Go; +using MongoDB.Bson; +using MongoDB.Driver; +using StellaOps.Excititor.Core; + +namespace StellaOps.Excititor.Storage.Mongo.Tests; + +public sealed class MongoVexSessionConsistencyTests : IAsyncLifetime +{ + private readonly MongoDbRunner _runner; + + public MongoVexSessionConsistencyTests() + { + _runner = MongoDbRunner.Start(singleNodeReplSet: true); + } + + [Fact] + public async Task SessionProvidesReadYourWrites() + { + await using var provider = BuildServiceProvider(); + await using var scope = provider.CreateAsyncScope(); + + var sessionProvider = scope.ServiceProvider.GetRequiredService(); + var providerStore = scope.ServiceProvider.GetRequiredService(); + + var session = await sessionProvider.StartSessionAsync(); + var descriptor = new VexProvider("red-hat", "Red Hat", VexProviderKind.Vendor); + + await providerStore.SaveAsync(descriptor, CancellationToken.None, session); + var fetched = await providerStore.FindAsync(descriptor.Id, CancellationToken.None, session); + + Assert.NotNull(fetched); + Assert.Equal(descriptor.DisplayName, fetched!.DisplayName); + } + + [Fact] + public async Task SessionMaintainsMonotonicReadsAcrossStepDown() + { + await using var provider = BuildServiceProvider(); + await using var scope = provider.CreateAsyncScope(); + + var client = scope.ServiceProvider.GetRequiredService(); + var sessionProvider = scope.ServiceProvider.GetRequiredService(); + var providerStore = scope.ServiceProvider.GetRequiredService(); + + var session = await sessionProvider.StartSessionAsync(); + var initial = new VexProvider("cisco", "Cisco", VexProviderKind.Vendor); + + await providerStore.SaveAsync(initial, CancellationToken.None, session); + var baseline = await providerStore.FindAsync(initial.Id, CancellationToken.None, session); + Assert.Equal("Cisco", baseline!.DisplayName); + + await ForcePrimaryStepDownAsync(client, CancellationToken.None); + await WaitForPrimaryAsync(client, CancellationToken.None); + + await ExecuteWithRetryAsync(async () => + { + var updated = new VexProvider(initial.Id, "Cisco Systems", initial.Kind); + await providerStore.SaveAsync(updated, CancellationToken.None, session); + }, CancellationToken.None); + + var afterFailover = await providerStore.FindAsync(initial.Id, CancellationToken.None, session); + Assert.Equal("Cisco Systems", afterFailover!.DisplayName); + + var subsequent = await providerStore.FindAsync(initial.Id, CancellationToken.None, session); + Assert.Equal("Cisco Systems", subsequent!.DisplayName); + } + + private ServiceProvider BuildServiceProvider() + { + var services = new ServiceCollection(); + services.AddLogging(builder => builder.AddDebug()); + services.Configure(options => + { + options.ConnectionString = _runner.ConnectionString; + options.DatabaseName = $"excititor-session-tests-{Guid.NewGuid():N}"; + options.CommandTimeout = TimeSpan.FromSeconds(5); + options.RawBucketName = "vex.raw"; + }); + services.AddExcititorMongoStorage(); + return services.BuildServiceProvider(); + } + + private static async Task ExecuteWithRetryAsync(Func action, CancellationToken cancellationToken) + { + const int maxAttempts = 10; + var attempt = 0; + + while (true) + { + cancellationToken.ThrowIfCancellationRequested(); + + try + { + await action(); + return; + } + catch (MongoException ex) when (IsStepDownTransient(ex) && attempt++ < maxAttempts) + { + await Task.Delay(TimeSpan.FromMilliseconds(200), cancellationToken); + } + } + } + + private static bool IsStepDownTransient(MongoException ex) + { + if (ex is MongoConnectionException) + { + return true; + } + + if (ex is MongoCommandException command) + { + return command.Code is 7 or 89 or 91 or 10107 or 11600 + || string.Equals(command.CodeName, "NotPrimaryNoSecondaryOk", StringComparison.OrdinalIgnoreCase) + || string.Equals(command.CodeName, "NotWritablePrimary", StringComparison.OrdinalIgnoreCase) + || string.Equals(command.CodeName, "PrimarySteppedDown", StringComparison.OrdinalIgnoreCase) + || string.Equals(command.CodeName, "NotPrimary", StringComparison.OrdinalIgnoreCase); + } + + return false; + } + + private static async Task ForcePrimaryStepDownAsync(IMongoClient client, CancellationToken cancellationToken) + { + var admin = client.GetDatabase("admin"); + var command = new BsonDocument + { + { "replSetStepDown", 1 }, + { "force", true }, + }; + + try + { + await admin.RunCommandAsync(command, cancellationToken: cancellationToken); + } + catch (MongoException ex) when (IsStepDownTransient(ex)) + { + // Expected when the primary closes connections during the step-down sequence. + } + } + + private static async Task WaitForPrimaryAsync(IMongoClient client, CancellationToken cancellationToken) + { + var admin = client.GetDatabase("admin"); + var helloCommand = new BsonDocument("hello", 1); + + for (var attempt = 0; attempt < 40; attempt++) + { + cancellationToken.ThrowIfCancellationRequested(); + + try + { + var result = await admin.RunCommandAsync(helloCommand, cancellationToken: cancellationToken); + if (result.TryGetValue("isWritablePrimary", out var value) && value.IsBoolean && value.AsBoolean) + { + return; + } + } + catch (MongoException ex) when (IsStepDownTransient(ex)) + { + // Primary still recovering, retry. + } + + await Task.Delay(TimeSpan.FromMilliseconds(200), cancellationToken); + } + + throw new TimeoutException("Replica set primary did not recover in time."); + } + + public Task InitializeAsync() => Task.CompletedTask; + + public Task DisposeAsync() + { + _runner.Dispose(); + return Task.CompletedTask; + } +} diff --git a/src/StellaOps.Excititor.Storage.Mongo.Tests/MongoVexStatementBackfillServiceTests.cs b/src/StellaOps.Excititor.Storage.Mongo.Tests/MongoVexStatementBackfillServiceTests.cs new file mode 100644 index 00000000..40fa0292 --- /dev/null +++ b/src/StellaOps.Excititor.Storage.Mongo.Tests/MongoVexStatementBackfillServiceTests.cs @@ -0,0 +1,170 @@ +using System; +using System.Collections.Immutable; +using System.Threading; +using System.Threading.Tasks; +using Microsoft.Extensions.DependencyInjection; +using Microsoft.Extensions.Logging; +using Mongo2Go; +using StellaOps.Excititor.Core; + +namespace StellaOps.Excititor.Storage.Mongo.Tests; + +public sealed class MongoVexStatementBackfillServiceTests : IAsyncLifetime +{ + private readonly MongoDbRunner _runner; + + public MongoVexStatementBackfillServiceTests() + { + _runner = MongoDbRunner.Start(singleNodeReplSet: true); + } + + [Fact] + public async Task RunAsync_BackfillsStatementsFromRawDocuments() + { + await using var provider = BuildServiceProvider(); + await using var scope = provider.CreateAsyncScope(); + + var rawStore = scope.ServiceProvider.GetRequiredService(); + var claimStore = scope.ServiceProvider.GetRequiredService(); + var backfill = scope.ServiceProvider.GetRequiredService(); + + var retrievedAt = DateTimeOffset.UtcNow.AddMinutes(-15); + var metadata = ImmutableDictionary.Empty + .Add("vulnId", "CVE-2025-0001") + .Add("productKey", "pkg:test/app"); + + var document = new VexRawDocument( + "test-provider", + VexDocumentFormat.Csaf, + new Uri("https://example.test/vex.json"), + retrievedAt, + "sha256:test-doc", + ReadOnlyMemory.Empty, + metadata); + + await rawStore.StoreAsync(document, CancellationToken.None); + + var result = await backfill.RunAsync(new VexStatementBackfillRequest(), CancellationToken.None); + + Assert.Equal(1, result.DocumentsEvaluated); + Assert.Equal(1, result.DocumentsBackfilled); + Assert.Equal(1, result.ClaimsWritten); + Assert.Equal(0, result.NormalizationFailures); + + var claims = await claimStore.FindAsync("CVE-2025-0001", "pkg:test/app", since: null, CancellationToken.None); + var claim = Assert.Single(claims); + Assert.Equal(VexClaimStatus.NotAffected, claim.Status); + Assert.Equal("test-provider", claim.ProviderId); + Assert.Equal(retrievedAt.ToUnixTimeSeconds(), claim.FirstSeen.ToUnixTimeSeconds()); + Assert.NotNull(claim.Signals); + Assert.Equal(0.2, claim.Signals!.Epss); + Assert.Equal("cvss", claim.Signals!.Severity?.Scheme); + } + + [Fact] + public async Task RunAsync_SkipsExistingDocumentsUnlessForced() + { + await using var provider = BuildServiceProvider(); + await using var scope = provider.CreateAsyncScope(); + + var rawStore = scope.ServiceProvider.GetRequiredService(); + var claimStore = scope.ServiceProvider.GetRequiredService(); + var backfill = scope.ServiceProvider.GetRequiredService(); + + var metadata = ImmutableDictionary.Empty + .Add("vulnId", "CVE-2025-0002") + .Add("productKey", "pkg:test/api"); + + var document = new VexRawDocument( + "test-provider", + VexDocumentFormat.Csaf, + new Uri("https://example.test/vex-2.json"), + DateTimeOffset.UtcNow.AddMinutes(-10), + "sha256:test-doc-2", + ReadOnlyMemory.Empty, + metadata); + + await rawStore.StoreAsync(document, CancellationToken.None); + + var first = await backfill.RunAsync(new VexStatementBackfillRequest(), CancellationToken.None); + Assert.Equal(1, first.DocumentsBackfilled); + + var second = await backfill.RunAsync(new VexStatementBackfillRequest(), CancellationToken.None); + Assert.Equal(1, second.DocumentsEvaluated); + Assert.Equal(0, second.DocumentsBackfilled); + Assert.Equal(1, second.SkippedExisting); + + var forced = await backfill.RunAsync(new VexStatementBackfillRequest(Force: true), CancellationToken.None); + Assert.Equal(1, forced.DocumentsBackfilled); + + var claims = await claimStore.FindAsync("CVE-2025-0002", "pkg:test/api", since: null, CancellationToken.None); + Assert.Equal(2, claims.Count); + } + + private ServiceProvider BuildServiceProvider() + { + var services = new ServiceCollection(); + services.AddLogging(builder => builder.AddDebug()); + services.AddSingleton(TimeProvider.System); + services.Configure(options => + { + options.ConnectionString = _runner.ConnectionString; + options.DatabaseName = $"excititor-backfill-tests-{Guid.NewGuid():N}"; + options.CommandTimeout = TimeSpan.FromSeconds(5); + options.RawBucketName = "vex.raw"; + options.GridFsInlineThresholdBytes = 1024; + options.ExportCacheTtl = TimeSpan.FromHours(1); + }); + services.AddExcititorMongoStorage(); + services.AddSingleton(); + return services.BuildServiceProvider(); + } + + public Task InitializeAsync() => Task.CompletedTask; + + public Task DisposeAsync() + { + _runner.Dispose(); + return Task.CompletedTask; + } + + private sealed class TestNormalizer : IVexNormalizer + { + public string Format => "csaf"; + + public bool CanHandle(VexRawDocument document) => true; + + public ValueTask NormalizeAsync(VexRawDocument document, VexProvider provider, CancellationToken cancellationToken) + { + var productKey = document.Metadata.TryGetValue("productKey", out var value) ? value : "pkg:test/default"; + var vulnId = document.Metadata.TryGetValue("vulnId", out var vuln) ? vuln : "CVE-TEST-0000"; + + var product = new VexProduct(productKey, "Test Product"); + var claimDocument = new VexClaimDocument( + document.Format, + document.Digest, + document.SourceUri); + + var timestamp = document.RetrievedAt == default ? DateTimeOffset.UtcNow : document.RetrievedAt; + + var claim = new VexClaim( + vulnId, + provider.Id, + product, + VexClaimStatus.NotAffected, + claimDocument, + timestamp, + timestamp, + VexJustification.ComponentNotPresent, + detail: "backfill-test", + confidence: new VexConfidence("high", 0.95, "unit-test"), + signals: new VexSignalSnapshot( + new VexSeveritySignal("cvss", 5.4, "medium"), + kev: false, + epss: 0.2)); + + var claims = ImmutableArray.Create(claim); + return ValueTask.FromResult(new VexClaimBatch(document, claims, ImmutableDictionary.Empty)); + } + } +} diff --git a/src/StellaOps.Excititor.Storage.Mongo/IVexExportStore.cs b/src/StellaOps.Excititor.Storage.Mongo/IVexExportStore.cs new file mode 100644 index 00000000..cde6af4b --- /dev/null +++ b/src/StellaOps.Excititor.Storage.Mongo/IVexExportStore.cs @@ -0,0 +1,13 @@ +using System.Threading; +using System.Threading.Tasks; +using MongoDB.Driver; +using StellaOps.Excititor.Core; + +namespace StellaOps.Excititor.Storage.Mongo; + +public interface IVexExportStore +{ + ValueTask FindAsync(VexQuerySignature signature, VexExportFormat format, CancellationToken cancellationToken, IClientSessionHandle? session = null); + + ValueTask SaveAsync(VexExportManifest manifest, CancellationToken cancellationToken, IClientSessionHandle? session = null); +} diff --git a/src/StellaOps.Excititor.Storage.Mongo/IVexRawStore.cs b/src/StellaOps.Excititor.Storage.Mongo/IVexRawStore.cs index e69de29b..ba73a6c8 100644 --- a/src/StellaOps.Excititor.Storage.Mongo/IVexRawStore.cs +++ b/src/StellaOps.Excititor.Storage.Mongo/IVexRawStore.cs @@ -0,0 +1,13 @@ +using System.Threading; +using System.Threading.Tasks; +using MongoDB.Driver; +using StellaOps.Excititor.Core; + +namespace StellaOps.Excititor.Storage.Mongo; + +public interface IVexRawStore : IVexRawDocumentSink +{ + ValueTask StoreAsync(VexRawDocument document, CancellationToken cancellationToken, IClientSessionHandle? session = null); + + ValueTask FindByDigestAsync(string digest, CancellationToken cancellationToken, IClientSessionHandle? session = null); +} diff --git a/src/StellaOps.Excititor.Storage.Mongo/IVexStorageContracts.cs b/src/StellaOps.Excititor.Storage.Mongo/IVexStorageContracts.cs index ae60d3c8..93009360 100644 --- a/src/StellaOps.Excititor.Storage.Mongo/IVexStorageContracts.cs +++ b/src/StellaOps.Excititor.Storage.Mongo/IVexStorageContracts.cs @@ -3,33 +3,34 @@ using System.Collections.Generic; using System.Collections.Immutable; using System.Threading; using System.Threading.Tasks; +using MongoDB.Driver; using StellaOps.Excititor.Core; namespace StellaOps.Excititor.Storage.Mongo; public interface IVexProviderStore { - ValueTask FindAsync(string id, CancellationToken cancellationToken); + ValueTask FindAsync(string id, CancellationToken cancellationToken, IClientSessionHandle? session = null); - ValueTask> ListAsync(CancellationToken cancellationToken); + ValueTask> ListAsync(CancellationToken cancellationToken, IClientSessionHandle? session = null); - ValueTask SaveAsync(VexProvider provider, CancellationToken cancellationToken); + ValueTask SaveAsync(VexProvider provider, CancellationToken cancellationToken, IClientSessionHandle? session = null); } public interface IVexConsensusStore { - ValueTask FindAsync(string vulnerabilityId, string productKey, CancellationToken cancellationToken); + ValueTask FindAsync(string vulnerabilityId, string productKey, CancellationToken cancellationToken, IClientSessionHandle? session = null); - ValueTask> FindByVulnerabilityAsync(string vulnerabilityId, CancellationToken cancellationToken); + ValueTask> FindByVulnerabilityAsync(string vulnerabilityId, CancellationToken cancellationToken, IClientSessionHandle? session = null); - ValueTask SaveAsync(VexConsensus consensus, CancellationToken cancellationToken); + ValueTask SaveAsync(VexConsensus consensus, CancellationToken cancellationToken, IClientSessionHandle? session = null); } public interface IVexClaimStore { - ValueTask AppendAsync(IEnumerable claims, DateTimeOffset observedAt, CancellationToken cancellationToken); + ValueTask AppendAsync(IEnumerable claims, DateTimeOffset observedAt, CancellationToken cancellationToken, IClientSessionHandle? session = null); - ValueTask> FindAsync(string vulnerabilityId, string productKey, DateTimeOffset? since, CancellationToken cancellationToken); + ValueTask> FindAsync(string vulnerabilityId, string productKey, DateTimeOffset? since, CancellationToken cancellationToken, IClientSessionHandle? session = null); } public sealed record VexConnectorState( @@ -40,27 +41,44 @@ public sealed record VexConnectorState( DateTimeOffset? LastSuccessAt, int FailureCount, DateTimeOffset? NextEligibleRun, - string? LastFailureReason); + string? LastFailureReason) +{ + public VexConnectorState( + string connectorId, + DateTimeOffset? lastUpdated, + ImmutableArray documentDigests) + : this( + connectorId, + lastUpdated, + documentDigests, + ImmutableDictionary.Empty, + LastSuccessAt: null, + FailureCount: 0, + NextEligibleRun: null, + LastFailureReason: null) + { + } +} public interface IVexConnectorStateRepository { - ValueTask GetAsync(string connectorId, CancellationToken cancellationToken); + ValueTask GetAsync(string connectorId, CancellationToken cancellationToken, IClientSessionHandle? session = null); - ValueTask SaveAsync(VexConnectorState state, CancellationToken cancellationToken); + ValueTask SaveAsync(VexConnectorState state, CancellationToken cancellationToken, IClientSessionHandle? session = null); } public interface IVexCacheIndex { - ValueTask FindAsync(VexQuerySignature signature, VexExportFormat format, CancellationToken cancellationToken); + ValueTask FindAsync(VexQuerySignature signature, VexExportFormat format, CancellationToken cancellationToken, IClientSessionHandle? session = null); - ValueTask SaveAsync(VexCacheEntry entry, CancellationToken cancellationToken); + ValueTask SaveAsync(VexCacheEntry entry, CancellationToken cancellationToken, IClientSessionHandle? session = null); - ValueTask RemoveAsync(VexQuerySignature signature, VexExportFormat format, CancellationToken cancellationToken); + ValueTask RemoveAsync(VexQuerySignature signature, VexExportFormat format, CancellationToken cancellationToken, IClientSessionHandle? session = null); } public interface IVexCacheMaintenance { - ValueTask RemoveExpiredAsync(DateTimeOffset asOf, CancellationToken cancellationToken); + ValueTask RemoveExpiredAsync(DateTimeOffset asOf, CancellationToken cancellationToken, IClientSessionHandle? session = null); - ValueTask RemoveMissingManifestReferencesAsync(CancellationToken cancellationToken); + ValueTask RemoveMissingManifestReferencesAsync(CancellationToken cancellationToken, IClientSessionHandle? session = null); } diff --git a/src/StellaOps.Excititor.Storage.Mongo/MongoVexCacheIndex.cs b/src/StellaOps.Excititor.Storage.Mongo/MongoVexCacheIndex.cs index 2a937fea..4bfecc0a 100644 --- a/src/StellaOps.Excititor.Storage.Mongo/MongoVexCacheIndex.cs +++ b/src/StellaOps.Excititor.Storage.Mongo/MongoVexCacheIndex.cs @@ -18,26 +18,42 @@ public sealed class MongoVexCacheIndex : IVexCacheIndex _collection = database.GetCollection(VexMongoCollectionNames.Cache); } - public async ValueTask FindAsync(VexQuerySignature signature, VexExportFormat format, CancellationToken cancellationToken) + public async ValueTask FindAsync(VexQuerySignature signature, VexExportFormat format, CancellationToken cancellationToken, IClientSessionHandle? session = null) { ArgumentNullException.ThrowIfNull(signature); var filter = Builders.Filter.Eq(x => x.Id, VexCacheEntryRecord.CreateId(signature, format)); - var record = await _collection.Find(filter).FirstOrDefaultAsync(cancellationToken).ConfigureAwait(false); + var record = session is null + ? await _collection.Find(filter).FirstOrDefaultAsync(cancellationToken).ConfigureAwait(false) + : await _collection.Find(session, filter).FirstOrDefaultAsync(cancellationToken).ConfigureAwait(false); return record?.ToDomain(); } - public async ValueTask SaveAsync(VexCacheEntry entry, CancellationToken cancellationToken) + public async ValueTask SaveAsync(VexCacheEntry entry, CancellationToken cancellationToken, IClientSessionHandle? session = null) { ArgumentNullException.ThrowIfNull(entry); var record = VexCacheEntryRecord.FromDomain(entry); var filter = Builders.Filter.Eq(x => x.Id, record.Id); - await _collection.ReplaceOneAsync(filter, record, new ReplaceOptions { IsUpsert = true }, cancellationToken).ConfigureAwait(false); + if (session is null) + { + await _collection.ReplaceOneAsync(filter, record, new ReplaceOptions { IsUpsert = true }, cancellationToken).ConfigureAwait(false); + } + else + { + await _collection.ReplaceOneAsync(session, filter, record, new ReplaceOptions { IsUpsert = true }, cancellationToken).ConfigureAwait(false); + } } - public async ValueTask RemoveAsync(VexQuerySignature signature, VexExportFormat format, CancellationToken cancellationToken) + public async ValueTask RemoveAsync(VexQuerySignature signature, VexExportFormat format, CancellationToken cancellationToken, IClientSessionHandle? session = null) { ArgumentNullException.ThrowIfNull(signature); var filter = Builders.Filter.Eq(x => x.Id, VexCacheEntryRecord.CreateId(signature, format)); - await _collection.DeleteOneAsync(filter, cancellationToken).ConfigureAwait(false); + if (session is null) + { + await _collection.DeleteOneAsync(filter, cancellationToken).ConfigureAwait(false); + } + else + { + await _collection.DeleteOneAsync(session, filter, options: null, cancellationToken).ConfigureAwait(false); + } } } diff --git a/src/StellaOps.Excititor.Storage.Mongo/MongoVexCacheMaintenance.cs b/src/StellaOps.Excititor.Storage.Mongo/MongoVexCacheMaintenance.cs index 1ff69f8d..f171150f 100644 --- a/src/StellaOps.Excititor.Storage.Mongo/MongoVexCacheMaintenance.cs +++ b/src/StellaOps.Excititor.Storage.Mongo/MongoVexCacheMaintenance.cs @@ -24,11 +24,19 @@ internal sealed class MongoVexCacheMaintenance : IVexCacheMaintenance _logger = logger; } - public async ValueTask RemoveExpiredAsync(DateTimeOffset asOf, CancellationToken cancellationToken) + public async ValueTask RemoveExpiredAsync(DateTimeOffset asOf, CancellationToken cancellationToken, IClientSessionHandle? session = null) { var cutoff = asOf.UtcDateTime; var filter = Builders.Filter.Lt(x => x.ExpiresAt, cutoff); - var result = await _cache.DeleteManyAsync(filter, cancellationToken).ConfigureAwait(false); + DeleteResult result; + if (session is null) + { + result = await _cache.DeleteManyAsync(filter, cancellationToken).ConfigureAwait(false); + } + else + { + result = await _cache.DeleteManyAsync(session, filter, options: null, cancellationToken).ConfigureAwait(false); + } var removed = (int)result.DeletedCount; if (removed > 0) @@ -39,10 +47,12 @@ internal sealed class MongoVexCacheMaintenance : IVexCacheMaintenance return removed; } - public async ValueTask RemoveMissingManifestReferencesAsync(CancellationToken cancellationToken) + public async ValueTask RemoveMissingManifestReferencesAsync(CancellationToken cancellationToken, IClientSessionHandle? session = null) { var filter = Builders.Filter.Ne(x => x.ManifestId, null); - var cursor = await _cache.Find(filter).ToListAsync(cancellationToken).ConfigureAwait(false); + var cursor = session is null + ? await _cache.Find(filter).ToListAsync(cancellationToken).ConfigureAwait(false) + : await _cache.Find(session, filter).ToListAsync(cancellationToken).ConfigureAwait(false); if (cursor.Count == 0) { @@ -57,8 +67,11 @@ internal sealed class MongoVexCacheMaintenance : IVexCacheMaintenance continue; } - var manifestExists = await _exports - .Find(Builders.Filter.Eq(x => x.Id, entry.ManifestId)) + var manifestFilter = Builders.Filter.Eq(x => x.Id, entry.ManifestId); + var manifestQuery = session is null + ? _exports.Find(manifestFilter) + : _exports.Find(session, manifestFilter); + var manifestExists = await manifestQuery .Limit(1) .AnyAsync(cancellationToken) .ConfigureAwait(false); @@ -75,7 +88,15 @@ internal sealed class MongoVexCacheMaintenance : IVexCacheMaintenance } var danglingFilter = Builders.Filter.In(x => x.Id, danglingIds); - var result = await _cache.DeleteManyAsync(danglingFilter, cancellationToken).ConfigureAwait(false); + DeleteResult result; + if (session is null) + { + result = await _cache.DeleteManyAsync(danglingFilter, cancellationToken).ConfigureAwait(false); + } + else + { + result = await _cache.DeleteManyAsync(session, danglingFilter, options: null, cancellationToken).ConfigureAwait(false); + } var removed = (int)result.DeletedCount; _logger.LogWarning("Removed {Count} cache entries referencing missing export manifests.", removed); diff --git a/src/StellaOps.Excititor.Storage.Mongo/MongoVexClaimStore.cs b/src/StellaOps.Excititor.Storage.Mongo/MongoVexClaimStore.cs index 8d125434..6ac1c8cf 100644 --- a/src/StellaOps.Excititor.Storage.Mongo/MongoVexClaimStore.cs +++ b/src/StellaOps.Excititor.Storage.Mongo/MongoVexClaimStore.cs @@ -18,7 +18,7 @@ public sealed class MongoVexClaimStore : IVexClaimStore _collection = database.GetCollection(VexMongoCollectionNames.Statements); } - public async ValueTask AppendAsync(IEnumerable claims, DateTimeOffset observedAt, CancellationToken cancellationToken) + public async ValueTask AppendAsync(IEnumerable claims, DateTimeOffset observedAt, CancellationToken cancellationToken, IClientSessionHandle? session = null) { ArgumentNullException.ThrowIfNull(claims); var records = claims @@ -30,10 +30,17 @@ public sealed class MongoVexClaimStore : IVexClaimStore return; } - await _collection.InsertManyAsync(records, new InsertManyOptions { IsOrdered = false }, cancellationToken).ConfigureAwait(false); + if (session is null) + { + await _collection.InsertManyAsync(records, new InsertManyOptions { IsOrdered = false }, cancellationToken).ConfigureAwait(false); + } + else + { + await _collection.InsertManyAsync(session, records, new InsertManyOptions { IsOrdered = false }, cancellationToken).ConfigureAwait(false); + } } - public async ValueTask> FindAsync(string vulnerabilityId, string productKey, DateTimeOffset? since, CancellationToken cancellationToken) + public async ValueTask> FindAsync(string vulnerabilityId, string productKey, DateTimeOffset? since, CancellationToken cancellationToken, IClientSessionHandle? session = null) { ArgumentException.ThrowIfNullOrWhiteSpace(vulnerabilityId); ArgumentException.ThrowIfNullOrWhiteSpace(productKey); @@ -46,8 +53,11 @@ public sealed class MongoVexClaimStore : IVexClaimStore filter &= Builders.Filter.Gte(x => x.InsertedAt, sinceValue.UtcDateTime); } - var records = await _collection - .Find(filter) + var find = session is null + ? _collection.Find(filter) + : _collection.Find(session, filter); + + var records = await find .SortByDescending(x => x.InsertedAt) .ToListAsync(cancellationToken) .ConfigureAwait(false); diff --git a/src/StellaOps.Excititor.Storage.Mongo/MongoVexConnectorStateRepository.cs b/src/StellaOps.Excititor.Storage.Mongo/MongoVexConnectorStateRepository.cs index c45aa897..b7673153 100644 --- a/src/StellaOps.Excititor.Storage.Mongo/MongoVexConnectorStateRepository.cs +++ b/src/StellaOps.Excititor.Storage.Mongo/MongoVexConnectorStateRepository.cs @@ -18,22 +18,31 @@ public sealed class MongoVexConnectorStateRepository : IVexConnectorStateReposit _collection = database.GetCollection(VexMongoCollectionNames.ConnectorState); } - public async ValueTask GetAsync(string connectorId, CancellationToken cancellationToken) + public async ValueTask GetAsync(string connectorId, CancellationToken cancellationToken, IClientSessionHandle? session = null) { ArgumentException.ThrowIfNullOrWhiteSpace(connectorId); var filter = Builders.Filter.Eq(x => x.ConnectorId, connectorId.Trim()); - var document = await _collection.Find(filter).FirstOrDefaultAsync(cancellationToken).ConfigureAwait(false); + var document = session is null + ? await _collection.Find(filter).FirstOrDefaultAsync(cancellationToken).ConfigureAwait(false) + : await _collection.Find(session, filter).FirstOrDefaultAsync(cancellationToken).ConfigureAwait(false); return document?.ToRecord(); } - public async ValueTask SaveAsync(VexConnectorState state, CancellationToken cancellationToken) + public async ValueTask SaveAsync(VexConnectorState state, CancellationToken cancellationToken, IClientSessionHandle? session = null) { ArgumentNullException.ThrowIfNull(state); var document = VexConnectorStateDocument.FromRecord(state.WithNormalizedDigests()); var filter = Builders.Filter.Eq(x => x.ConnectorId, document.ConnectorId); - await _collection.ReplaceOneAsync(filter, document, new ReplaceOptions { IsUpsert = true }, cancellationToken).ConfigureAwait(false); + if (session is null) + { + await _collection.ReplaceOneAsync(filter, document, new ReplaceOptions { IsUpsert = true }, cancellationToken).ConfigureAwait(false); + } + else + { + await _collection.ReplaceOneAsync(session, filter, document, new ReplaceOptions { IsUpsert = true }, cancellationToken).ConfigureAwait(false); + } } } diff --git a/src/StellaOps.Excititor.Storage.Mongo/MongoVexConsensusStore.cs b/src/StellaOps.Excititor.Storage.Mongo/MongoVexConsensusStore.cs index 5a866c3c..5064c2fd 100644 --- a/src/StellaOps.Excititor.Storage.Mongo/MongoVexConsensusStore.cs +++ b/src/StellaOps.Excititor.Storage.Mongo/MongoVexConsensusStore.cs @@ -17,30 +17,42 @@ public sealed class MongoVexConsensusStore : IVexConsensusStore _collection = database.GetCollection(VexMongoCollectionNames.Consensus); } - public async ValueTask FindAsync(string vulnerabilityId, string productKey, CancellationToken cancellationToken) + public async ValueTask FindAsync(string vulnerabilityId, string productKey, CancellationToken cancellationToken, IClientSessionHandle? session = null) { ArgumentException.ThrowIfNullOrWhiteSpace(vulnerabilityId); ArgumentException.ThrowIfNullOrWhiteSpace(productKey); var id = VexConsensusRecord.CreateId(vulnerabilityId, productKey); var filter = Builders.Filter.Eq(x => x.Id, id); - var record = await _collection.Find(filter).FirstOrDefaultAsync(cancellationToken).ConfigureAwait(false); + var record = session is null + ? await _collection.Find(filter).FirstOrDefaultAsync(cancellationToken).ConfigureAwait(false) + : await _collection.Find(session, filter).FirstOrDefaultAsync(cancellationToken).ConfigureAwait(false); return record?.ToDomain(); } - public async ValueTask> FindByVulnerabilityAsync(string vulnerabilityId, CancellationToken cancellationToken) + public async ValueTask> FindByVulnerabilityAsync(string vulnerabilityId, CancellationToken cancellationToken, IClientSessionHandle? session = null) { ArgumentException.ThrowIfNullOrWhiteSpace(vulnerabilityId); var filter = Builders.Filter.Eq(x => x.VulnerabilityId, vulnerabilityId.Trim()); - var records = await _collection.Find(filter).ToListAsync(cancellationToken).ConfigureAwait(false); + var find = session is null + ? _collection.Find(filter) + : _collection.Find(session, filter); + var records = await find.ToListAsync(cancellationToken).ConfigureAwait(false); return records.ConvertAll(static record => record.ToDomain()); } - public async ValueTask SaveAsync(VexConsensus consensus, CancellationToken cancellationToken) + public async ValueTask SaveAsync(VexConsensus consensus, CancellationToken cancellationToken, IClientSessionHandle? session = null) { ArgumentNullException.ThrowIfNull(consensus); var record = VexConsensusRecord.FromDomain(consensus); var filter = Builders.Filter.Eq(x => x.Id, record.Id); - await _collection.ReplaceOneAsync(filter, record, new ReplaceOptions { IsUpsert = true }, cancellationToken).ConfigureAwait(false); + if (session is null) + { + await _collection.ReplaceOneAsync(filter, record, new ReplaceOptions { IsUpsert = true }, cancellationToken).ConfigureAwait(false); + } + else + { + await _collection.ReplaceOneAsync(session, filter, record, new ReplaceOptions { IsUpsert = true }, cancellationToken).ConfigureAwait(false); + } } } diff --git a/src/StellaOps.Excititor.Storage.Mongo/MongoVexExportStore.cs b/src/StellaOps.Excititor.Storage.Mongo/MongoVexExportStore.cs index 0ac1f210..24a13f2b 100644 --- a/src/StellaOps.Excititor.Storage.Mongo/MongoVexExportStore.cs +++ b/src/StellaOps.Excititor.Storage.Mongo/MongoVexExportStore.cs @@ -15,15 +15,18 @@ public sealed class MongoVexExportStore : IVexExportStore private readonly IMongoCollection _exports; private readonly IMongoCollection _cache; private readonly VexMongoStorageOptions _options; + private readonly IVexMongoSessionProvider _sessionProvider; public MongoVexExportStore( IMongoClient client, IMongoDatabase database, - IOptions options) + IOptions options, + IVexMongoSessionProvider sessionProvider) { _client = client ?? throw new ArgumentNullException(nameof(client)); ArgumentNullException.ThrowIfNull(database); ArgumentNullException.ThrowIfNull(options); + _sessionProvider = sessionProvider ?? throw new ArgumentNullException(nameof(sessionProvider)); _options = options.Value; Validator.ValidateObject(_options, new ValidationContext(_options), validateAllProperties: true); @@ -33,12 +36,14 @@ public sealed class MongoVexExportStore : IVexExportStore _cache = database.GetCollection(VexMongoCollectionNames.Cache); } - public async ValueTask FindAsync(VexQuerySignature signature, VexExportFormat format, CancellationToken cancellationToken) + public async ValueTask FindAsync(VexQuerySignature signature, VexExportFormat format, CancellationToken cancellationToken, IClientSessionHandle? session = null) { ArgumentNullException.ThrowIfNull(signature); var cacheId = VexCacheEntryRecord.CreateId(signature, format); var cacheFilter = Builders.Filter.Eq(x => x.Id, cacheId); - var cacheRecord = await _cache.Find(cacheFilter).FirstOrDefaultAsync(cancellationToken).ConfigureAwait(false); + var cacheRecord = session is null + ? await _cache.Find(cacheFilter).FirstOrDefaultAsync(cancellationToken).ConfigureAwait(false) + : await _cache.Find(session, cacheFilter).FirstOrDefaultAsync(cancellationToken).ConfigureAwait(false); if (cacheRecord is null) { @@ -53,37 +58,54 @@ public sealed class MongoVexExportStore : IVexExportStore var manifestId = VexExportManifestRecord.CreateId(signature, format); var manifestFilter = Builders.Filter.Eq(x => x.Id, manifestId); - var manifest = await _exports.Find(manifestFilter).FirstOrDefaultAsync(cancellationToken).ConfigureAwait(false); + var manifest = session is null + ? await _exports.Find(manifestFilter).FirstOrDefaultAsync(cancellationToken).ConfigureAwait(false) + : await _exports.Find(session, manifestFilter).FirstOrDefaultAsync(cancellationToken).ConfigureAwait(false); if (manifest is null) { - await _cache.DeleteOneAsync(cacheFilter, cancellationToken).ConfigureAwait(false); + if (session is null) + { + await _cache.DeleteOneAsync(cacheFilter, cancellationToken).ConfigureAwait(false); + } + else + { + await _cache.DeleteOneAsync(session, cacheFilter, options: null, cancellationToken).ConfigureAwait(false); + } return null; } if (!string.IsNullOrWhiteSpace(cacheRecord.ManifestId) && !string.Equals(cacheRecord.ManifestId, manifest.Id, StringComparison.Ordinal)) { - await _cache.DeleteOneAsync(cacheFilter, cancellationToken).ConfigureAwait(false); + if (session is null) + { + await _cache.DeleteOneAsync(cacheFilter, cancellationToken).ConfigureAwait(false); + } + else + { + await _cache.DeleteOneAsync(session, cacheFilter, options: null, cancellationToken).ConfigureAwait(false); + } return null; } return manifest.ToDomain(); } - public async ValueTask SaveAsync(VexExportManifest manifest, CancellationToken cancellationToken) + public async ValueTask SaveAsync(VexExportManifest manifest, CancellationToken cancellationToken, IClientSessionHandle? session = null) { ArgumentNullException.ThrowIfNull(manifest); - using var session = await _client.StartSessionAsync(cancellationToken: cancellationToken).ConfigureAwait(false); - var supportsTransactions = session.Client.Cluster.Description.Type != ClusterType.Standalone; + var sessionHandle = session ?? await _sessionProvider.StartSessionAsync(cancellationToken).ConfigureAwait(false); + var supportsTransactions = sessionHandle.Client.Cluster.Description.Type != ClusterType.Standalone + && !sessionHandle.IsInTransaction; var startedTransaction = false; if (supportsTransactions) { try { - session.StartTransaction(); + sessionHandle.StartTransaction(); startedTransaction = true; } catch (NotSupportedException) @@ -99,7 +121,7 @@ public sealed class MongoVexExportStore : IVexExportStore await _exports .ReplaceOneAsync( - session, + sessionHandle, manifestFilter, manifestRecord, new ReplaceOptions { IsUpsert = true }, @@ -112,7 +134,7 @@ public sealed class MongoVexExportStore : IVexExportStore await _cache .ReplaceOneAsync( - session, + sessionHandle, cacheFilter, cacheRecord, new ReplaceOptions { IsUpsert = true }, @@ -121,14 +143,14 @@ public sealed class MongoVexExportStore : IVexExportStore if (startedTransaction) { - await session.CommitTransactionAsync(cancellationToken).ConfigureAwait(false); + await sessionHandle.CommitTransactionAsync(cancellationToken).ConfigureAwait(false); } } catch { - if (startedTransaction && session.IsInTransaction) + if (startedTransaction && sessionHandle.IsInTransaction) { - await session.AbortTransactionAsync(cancellationToken).ConfigureAwait(false); + await sessionHandle.AbortTransactionAsync(cancellationToken).ConfigureAwait(false); } throw; } diff --git a/src/StellaOps.Excititor.Storage.Mongo/MongoVexProviderStore.cs b/src/StellaOps.Excititor.Storage.Mongo/MongoVexProviderStore.cs index ceb42488..510af05a 100644 --- a/src/StellaOps.Excititor.Storage.Mongo/MongoVexProviderStore.cs +++ b/src/StellaOps.Excititor.Storage.Mongo/MongoVexProviderStore.cs @@ -17,17 +17,22 @@ public sealed class MongoVexProviderStore : IVexProviderStore _collection = database.GetCollection(VexMongoCollectionNames.Providers); } - public async ValueTask FindAsync(string id, CancellationToken cancellationToken) + public async ValueTask FindAsync(string id, CancellationToken cancellationToken, IClientSessionHandle? session = null) { ArgumentException.ThrowIfNullOrWhiteSpace(id); var filter = Builders.Filter.Eq(x => x.Id, id.Trim()); - var record = await _collection.Find(filter).FirstOrDefaultAsync(cancellationToken).ConfigureAwait(false); + var record = session is null + ? await _collection.Find(filter).FirstOrDefaultAsync(cancellationToken).ConfigureAwait(false) + : await _collection.Find(session, filter).FirstOrDefaultAsync(cancellationToken).ConfigureAwait(false); return record?.ToDomain(); } - public async ValueTask> ListAsync(CancellationToken cancellationToken) + public async ValueTask> ListAsync(CancellationToken cancellationToken, IClientSessionHandle? session = null) { - var records = await _collection.Find(FilterDefinition.Empty) + var find = session is null + ? _collection.Find(FilterDefinition.Empty) + : _collection.Find(session, FilterDefinition.Empty); + var records = await find .Sort(Builders.Sort.Ascending(x => x.Id)) .ToListAsync(cancellationToken) .ConfigureAwait(false); @@ -35,11 +40,18 @@ public sealed class MongoVexProviderStore : IVexProviderStore return records.ConvertAll(static record => record.ToDomain()); } - public async ValueTask SaveAsync(VexProvider provider, CancellationToken cancellationToken) + public async ValueTask SaveAsync(VexProvider provider, CancellationToken cancellationToken, IClientSessionHandle? session = null) { ArgumentNullException.ThrowIfNull(provider); var record = VexProviderRecord.FromDomain(provider); var filter = Builders.Filter.Eq(x => x.Id, record.Id); - await _collection.ReplaceOneAsync(filter, record, new ReplaceOptions { IsUpsert = true }, cancellationToken).ConfigureAwait(false); + if (session is null) + { + await _collection.ReplaceOneAsync(filter, record, new ReplaceOptions { IsUpsert = true }, cancellationToken).ConfigureAwait(false); + } + else + { + await _collection.ReplaceOneAsync(session, filter, record, new ReplaceOptions { IsUpsert = true }, cancellationToken).ConfigureAwait(false); + } } } diff --git a/src/StellaOps.Excititor.Storage.Mongo/MongoVexRawStore.cs b/src/StellaOps.Excititor.Storage.Mongo/MongoVexRawStore.cs index 57dbc197..367e0b7f 100644 --- a/src/StellaOps.Excititor.Storage.Mongo/MongoVexRawStore.cs +++ b/src/StellaOps.Excititor.Storage.Mongo/MongoVexRawStore.cs @@ -18,15 +18,18 @@ public sealed class MongoVexRawStore : IVexRawStore private readonly IMongoCollection _collection; private readonly GridFSBucket _bucket; private readonly VexMongoStorageOptions _options; + private readonly IVexMongoSessionProvider _sessionProvider; public MongoVexRawStore( IMongoClient client, IMongoDatabase database, - IOptions options) + IOptions options, + IVexMongoSessionProvider sessionProvider) { _client = client ?? throw new ArgumentNullException(nameof(client)); ArgumentNullException.ThrowIfNull(database); ArgumentNullException.ThrowIfNull(options); + _sessionProvider = sessionProvider ?? throw new ArgumentNullException(nameof(sessionProvider)); _options = options.Value; Validator.ValidateObject(_options, new ValidationContext(_options), validateAllProperties: true); @@ -42,7 +45,7 @@ public sealed class MongoVexRawStore : IVexRawStore }); } - public async ValueTask StoreAsync(VexRawDocument document, CancellationToken cancellationToken) + public async ValueTask StoreAsync(VexRawDocument document, CancellationToken cancellationToken, IClientSessionHandle? session = null) { ArgumentNullException.ThrowIfNull(document); @@ -51,20 +54,22 @@ public sealed class MongoVexRawStore : IVexRawStore string? newGridId = null; string? oldGridIdToDelete = null; + var sessionHandle = session ?? await _sessionProvider.StartSessionAsync(cancellationToken).ConfigureAwait(false); + if (!useInline) { - newGridId = await UploadToGridFsAsync(document, cancellationToken).ConfigureAwait(false); + newGridId = await UploadToGridFsAsync(document, sessionHandle, cancellationToken).ConfigureAwait(false); } - using var session = await _client.StartSessionAsync(cancellationToken: cancellationToken).ConfigureAwait(false); - var supportsTransactions = session.Client.Cluster.Description.Type != ClusterType.Standalone; + var supportsTransactions = sessionHandle.Client.Cluster.Description.Type != ClusterType.Standalone + && !sessionHandle.IsInTransaction; var startedTransaction = false; if (supportsTransactions) { try { - session.StartTransaction(); + sessionHandle.StartTransaction(); startedTransaction = true; } catch (NotSupportedException) @@ -77,7 +82,7 @@ public sealed class MongoVexRawStore : IVexRawStore { var filter = Builders.Filter.Eq(x => x.Id, document.Digest); var existing = await _collection - .Find(session, filter) + .Find(sessionHandle, filter) .FirstOrDefaultAsync(cancellationToken) .ConfigureAwait(false); @@ -86,7 +91,7 @@ public sealed class MongoVexRawStore : IVexRawStore await _collection .ReplaceOneAsync( - session, + sessionHandle, filter, record, new ReplaceOptions { IsUpsert = true }, @@ -103,19 +108,19 @@ public sealed class MongoVexRawStore : IVexRawStore if (startedTransaction) { - await session.CommitTransactionAsync(cancellationToken).ConfigureAwait(false); + await sessionHandle.CommitTransactionAsync(cancellationToken).ConfigureAwait(false); } } catch { - if (startedTransaction && session.IsInTransaction) + if (startedTransaction && sessionHandle.IsInTransaction) { - await session.AbortTransactionAsync(cancellationToken).ConfigureAwait(false); + await sessionHandle.AbortTransactionAsync(cancellationToken).ConfigureAwait(false); } if (!useInline && !string.IsNullOrWhiteSpace(newGridId)) { - await DeleteFromGridFsAsync(newGridId, cancellationToken).ConfigureAwait(false); + await DeleteFromGridFsAsync(newGridId, sessionHandle, cancellationToken).ConfigureAwait(false); } throw; @@ -123,11 +128,11 @@ public sealed class MongoVexRawStore : IVexRawStore if (!string.IsNullOrWhiteSpace(oldGridIdToDelete)) { - await DeleteFromGridFsAsync(oldGridIdToDelete!, cancellationToken).ConfigureAwait(false); + await DeleteFromGridFsAsync(oldGridIdToDelete!, sessionHandle, cancellationToken).ConfigureAwait(false); } } - public async ValueTask FindByDigestAsync(string digest, CancellationToken cancellationToken) + public async ValueTask FindByDigestAsync(string digest, CancellationToken cancellationToken, IClientSessionHandle? session = null) { if (string.IsNullOrWhiteSpace(digest)) { @@ -136,7 +141,9 @@ public sealed class MongoVexRawStore : IVexRawStore var trimmed = digest.Trim(); var filter = Builders.Filter.Eq(x => x.Id, trimmed); - var record = await _collection.Find(filter).FirstOrDefaultAsync(cancellationToken).ConfigureAwait(false); + var record = session is null + ? await _collection.Find(filter).FirstOrDefaultAsync(cancellationToken).ConfigureAwait(false) + : await _collection.Find(session, filter).FirstOrDefaultAsync(cancellationToken).ConfigureAwait(false); if (record is null) { return null; @@ -144,14 +151,15 @@ public sealed class MongoVexRawStore : IVexRawStore if (!string.IsNullOrWhiteSpace(record.GridFsObjectId)) { - var bytes = await DownloadFromGridFsAsync(record.GridFsObjectId, cancellationToken).ConfigureAwait(false); + var handle = session ?? await _sessionProvider.StartSessionAsync(cancellationToken).ConfigureAwait(false); + var bytes = await DownloadFromGridFsAsync(record.GridFsObjectId, handle, cancellationToken).ConfigureAwait(false); return record.ToDomain(new ReadOnlyMemory(bytes)); } return record.ToDomain(); } - private async Task UploadToGridFsAsync(VexRawDocument document, CancellationToken cancellationToken) + private async Task UploadToGridFsAsync(VexRawDocument document, IClientSessionHandle? session, CancellationToken cancellationToken) { using var stream = new MemoryStream(document.Content.ToArray(), writable: false); var metadata = new BsonDocument @@ -170,7 +178,7 @@ public sealed class MongoVexRawStore : IVexRawStore return objectId.ToString(); } - private async Task DeleteFromGridFsAsync(string gridFsObjectId, CancellationToken cancellationToken) + private async Task DeleteFromGridFsAsync(string gridFsObjectId, IClientSessionHandle? session, CancellationToken cancellationToken) { if (!ObjectId.TryParse(gridFsObjectId, out var objectId)) { @@ -187,13 +195,16 @@ public sealed class MongoVexRawStore : IVexRawStore } } - private async Task DownloadFromGridFsAsync(string gridFsObjectId, CancellationToken cancellationToken) + private async Task DownloadFromGridFsAsync(string gridFsObjectId, IClientSessionHandle? session, CancellationToken cancellationToken) { if (!ObjectId.TryParse(gridFsObjectId, out var objectId)) { return Array.Empty(); } - return await _bucket.DownloadAsBytesAsync(objectId, cancellationToken: cancellationToken).ConfigureAwait(false); + return await _bucket.DownloadAsBytesAsync(objectId, null, cancellationToken).ConfigureAwait(false); } + + async ValueTask IVexRawDocumentSink.StoreAsync(VexRawDocument document, CancellationToken cancellationToken) + => await StoreAsync(document, cancellationToken, session: null).ConfigureAwait(false); } diff --git a/src/StellaOps.Excititor.Storage.Mongo/ServiceCollectionExtensions.cs b/src/StellaOps.Excititor.Storage.Mongo/ServiceCollectionExtensions.cs index 4700c148..e3eb8cdd 100644 --- a/src/StellaOps.Excititor.Storage.Mongo/ServiceCollectionExtensions.cs +++ b/src/StellaOps.Excititor.Storage.Mongo/ServiceCollectionExtensions.cs @@ -29,7 +29,7 @@ public static class VexMongoServiceCollectionExtensions return new MongoClient(settings); }); - services.TryAddSingleton(static provider => + services.TryAddScoped(static provider => { var options = provider.GetRequiredService>().Value; var client = provider.GetRequiredService(); @@ -46,15 +46,16 @@ public static class VexMongoServiceCollectionExtensions services.AddScoped(); - services.AddSingleton(); - services.AddSingleton(); - services.AddSingleton(); - services.AddSingleton(); - services.AddSingleton(); - services.AddSingleton(); - services.AddSingleton(); - services.AddSingleton(); - services.AddSingleton(); + services.AddScoped(); + services.AddScoped(); + services.AddScoped(); + services.AddScoped(); + services.AddScoped(); + services.AddScoped(); + services.AddScoped(); + services.AddScoped(); + services.AddScoped(); + services.AddScoped(); services.AddSingleton(); services.AddSingleton(); services.AddSingleton(); diff --git a/src/StellaOps.Excititor.Storage.Mongo/StorageBackedVexNormalizerRouter.cs b/src/StellaOps.Excititor.Storage.Mongo/StorageBackedVexNormalizerRouter.cs index d2ea123b..0fae6bf3 100644 --- a/src/StellaOps.Excititor.Storage.Mongo/StorageBackedVexNormalizerRouter.cs +++ b/src/StellaOps.Excititor.Storage.Mongo/StorageBackedVexNormalizerRouter.cs @@ -14,15 +14,18 @@ public sealed class StorageBackedVexNormalizerRouter : IVexNormalizerRouter { private readonly VexNormalizerRegistry _registry; private readonly IVexProviderStore _providerStore; + private readonly IVexMongoSessionProvider _sessionProvider; private readonly ILogger _logger; public StorageBackedVexNormalizerRouter( IEnumerable normalizers, IVexProviderStore providerStore, + IVexMongoSessionProvider sessionProvider, ILogger logger) { ArgumentNullException.ThrowIfNull(normalizers); _providerStore = providerStore ?? throw new ArgumentNullException(nameof(providerStore)); + _sessionProvider = sessionProvider ?? throw new ArgumentNullException(nameof(sessionProvider)); _logger = logger ?? throw new ArgumentNullException(nameof(logger)); _registry = new VexNormalizerRegistry(normalizers.ToImmutableArray()); @@ -42,7 +45,8 @@ public sealed class StorageBackedVexNormalizerRouter : IVexNormalizerRouter ImmutableDictionary.Empty); } - var provider = await _providerStore.FindAsync(document.ProviderId, cancellationToken).ConfigureAwait(false) + var session = await _sessionProvider.StartSessionAsync(cancellationToken).ConfigureAwait(false); + var provider = await _providerStore.FindAsync(document.ProviderId, cancellationToken, session).ConfigureAwait(false) ?? new VexProvider(document.ProviderId, document.ProviderId, VexProviderKind.Vendor); return await normalizer.NormalizeAsync(document, provider, cancellationToken).ConfigureAwait(false); diff --git a/src/StellaOps.Excititor.Storage.Mongo/TASKS.md b/src/StellaOps.Excititor.Storage.Mongo/TASKS.md index d10350ec..7871b14a 100644 --- a/src/StellaOps.Excititor.Storage.Mongo/TASKS.md +++ b/src/StellaOps.Excititor.Storage.Mongo/TASKS.md @@ -7,5 +7,5 @@ If you are working on this file you need to read docs/ARCHITECTURE_EXCITITOR.md |EXCITITOR-STORAGE-01-003 – Repository layer & transactional flows|Team Excititor Storage|EXCITITOR-STORAGE-01-001|**DONE (2025-10-16)** – Added GridFS-backed raw store with transactional upserts (including fallback for non-replicaset Mongo), export/cache repository coordination, and coverage verifying cache TTL + GridFS round-trips.| |EXCITITOR-STORAGE-01-004 – Provider/consensus/cache mappings|Team Excititor Storage|EXCITITOR-STORAGE-01-001|**DONE (2025-10-16)** – Registered MongoDB class maps for provider/consensus/cache records with forward-compatible field handling and added coverage ensuring GridFS-linked cache entries round-trip cleanly.| |EXCITITOR-STORAGE-02-001 – Statement events & scoring signals|Team Excititor Storage|EXCITITOR-CORE-02-001|DONE (2025-10-19) – Added immutable `vex.statements` collection + claim store, extended consensus persistence with severity/KEV/EPSS signals, shipped migration `20251019-consensus-signals-statements`, and updated docs. Tests: `dotnet test src/StellaOps.Excititor.Core.Tests/StellaOps.Excititor.Core.Tests.csproj` & `dotnet test src/StellaOps.Excititor.Storage.Mongo.Tests/StellaOps.Excititor.Storage.Mongo.Tests.csproj`; worker/web suites pending due to NU1903 (`Microsoft.Extensions.Caching.Memory`) advisory.| -|EXCITITOR-STORAGE-03-001 – Statement backfill tooling|Team Excititor Storage|EXCITITOR-STORAGE-02-001|DOING (2025-10-19) – Provide CLI/scripted tooling to replay historical statements into `vex.statements` (leveraging `/excititor/statements`), document operational runbook, and add smoke test verifying replayed data includes severity/KEV/EPSS signals.
2025-10-19: Prerequisite EXCITITOR-STORAGE-02-001 verified complete; Wave 0 kickoff acknowledged per EXECPLAN.| -|EXCITITOR-STORAGE-MONGO-08-001 – Session + causal consistency hardening|Team Excititor Storage|EXCITITOR-STORAGE-01-003|DOING (2025-10-19) – Register Mongo client/database with majority read/write concerns, expose scoped session helper enabling causal consistency, thread session handles through raw/export/consensus/cache stores (including GridFS reads), and extend integration tests to verify read-your-write semantics during replica-set failover.
2025-10-19: Prerequisite EXCITITOR-STORAGE-01-003 confirmed complete; prerequisites satisfied for Wave 0 hardening effort.| +|EXCITITOR-STORAGE-03-001 – Statement backfill tooling|Team Excititor Storage|EXCITITOR-STORAGE-02-001|**DONE (2025-10-19)** – Shipped Mongo-backed statement replay service + `/excititor/admin/backfill-statements`, wired CLI command `stellaops excititor backfill-statements`, added integration tests, and documented the runbook in `docs/dev/EXCITITOR_STATEMENT_BACKFILL.md`.| +|EXCITITOR-STORAGE-MONGO-08-001 – Session + causal consistency hardening|Team Excititor Storage|EXCITITOR-STORAGE-01-003|**DONE (2025-10-19)** – Completed session-aware overloads across all repositories, persisted claims/signals/connector state with new Mongo records, updated orchestrators/workers to reuse scoped sessions, and added replica-set consistency tests (`dotnet test src/StellaOps.Excititor.Storage.Mongo.Tests/StellaOps.Excititor.Storage.Mongo.Tests.csproj`). GridFS operations fall back to majority semantics due to driver limits; transactions cover metadata writes to preserve determinism.| diff --git a/src/StellaOps.Excititor.Storage.Mongo/VexMongoModels.cs b/src/StellaOps.Excititor.Storage.Mongo/VexMongoModels.cs index f6ac4ec7..f7272a92 100644 --- a/src/StellaOps.Excititor.Storage.Mongo/VexMongoModels.cs +++ b/src/StellaOps.Excititor.Storage.Mongo/VexMongoModels.cs @@ -90,6 +90,24 @@ internal sealed class VexExportManifestRecord public string? ConsensusRevision { get; set; } = null; + public string? PolicyRevisionId { get; set; } + = null; + + public string? PolicyDigest { get; set; } + = null; + + public string? ConsensusDigestAlgorithm { get; set; } + = null; + + public string? ConsensusDigestValue { get; set; } + = null; + + public string? ScoreDigestAlgorithm { get; set; } + = null; + + public string? ScoreDigestValue { get; set; } + = null; + public string? PredicateType { get; set; } = null; @@ -127,6 +145,12 @@ internal sealed class VexExportManifestRecord FromCache = manifest.FromCache, SourceProviders = manifest.SourceProviders.ToList(), ConsensusRevision = manifest.ConsensusRevision, + PolicyRevisionId = manifest.PolicyRevisionId, + PolicyDigest = manifest.PolicyDigest, + ConsensusDigestAlgorithm = manifest.ConsensusDigest?.Algorithm, + ConsensusDigestValue = manifest.ConsensusDigest?.Digest, + ScoreDigestAlgorithm = manifest.ScoreDigest?.Algorithm, + ScoreDigestValue = manifest.ScoreDigest?.Digest, PredicateType = manifest.Attestation?.PredicateType, RekorApiVersion = manifest.Attestation?.Rekor?.ApiVersion, RekorLocation = manifest.Attestation?.Rekor?.Location, @@ -157,16 +181,28 @@ internal sealed class VexExportManifestRecord EnvelopeDigest, signedAt); + var consensusDigest = ConsensusDigestAlgorithm is null || ConsensusDigestValue is null + ? null + : new VexContentAddress(ConsensusDigestAlgorithm, ConsensusDigestValue); + + var scoreDigest = ScoreDigestAlgorithm is null || ScoreDigestValue is null + ? null + : new VexContentAddress(ScoreDigestAlgorithm, ScoreDigestValue); + return new VexExportManifest( Id, new VexQuerySignature(QuerySignature), Enum.Parse(Format, ignoreCase: true), - CreatedAt, + new DateTimeOffset(DateTime.SpecifyKind(CreatedAt, DateTimeKind.Utc)), new VexContentAddress(ArtifactAlgorithm, ArtifactDigest), ClaimCount, SourceProviders, FromCache, ConsensusRevision, + PolicyRevisionId, + PolicyDigest, + consensusDigest, + scoreDigest, attestation, SizeBytes); } @@ -311,6 +347,9 @@ internal sealed class VexConsensusRecord public List Conflicts { get; set; } = new(); + public VexSignalDocument? Signals { get; set; } + = null; + public string? PolicyVersion { get; set; } = null; @@ -336,6 +375,7 @@ internal sealed class VexConsensusRecord CalculatedAt = consensus.CalculatedAt.UtcDateTime, Sources = consensus.Sources.Select(VexConsensusSourceDocument.FromDomain).ToList(), Conflicts = consensus.Conflicts.Select(VexConsensusConflictDocument.FromDomain).ToList(), + Signals = VexSignalDocument.FromDomain(consensus.Signals), PolicyVersion = consensus.PolicyVersion, PolicyRevisionId = consensus.PolicyRevisionId, PolicyDigest = consensus.PolicyDigest, @@ -350,6 +390,7 @@ internal sealed class VexConsensusRecord new DateTimeOffset(CalculatedAt, TimeSpan.Zero), Sources.Select(static source => source.ToDomain()), Conflicts.Select(static conflict => conflict.ToDomain()), + Signals?.ToDomain(), PolicyVersion, Summary, PolicyRevisionId, @@ -502,6 +543,229 @@ internal sealed class VexConfidenceDocument => new(Level, Score, Method); } +[BsonIgnoreExtraElements] +internal sealed class VexSeveritySignalDocument +{ + public string Scheme { get; set; } = default!; + + public double? Score { get; set; } + = null; + + public string? Label { get; set; } + = null; + + public string? Vector { get; set; } + = null; + + public static VexSeveritySignalDocument FromDomain(VexSeveritySignal severity) + => new() + { + Scheme = severity.Scheme, + Score = severity.Score, + Label = severity.Label, + Vector = severity.Vector, + }; + + public VexSeveritySignal ToDomain() + => new(Scheme, Score, Label, Vector); +} + +[BsonIgnoreExtraElements] +internal sealed class VexSignalDocument +{ + public VexSeveritySignalDocument? Severity { get; set; } + = null; + + public bool? Kev { get; set; } + = null; + + public double? Epss { get; set; } + = null; + + public static VexSignalDocument? FromDomain(VexSignalSnapshot? snapshot) + => snapshot is null + ? null + : new VexSignalDocument + { + Severity = snapshot.Severity is null ? null : VexSeveritySignalDocument.FromDomain(snapshot.Severity), + Kev = snapshot.Kev, + Epss = snapshot.Epss, + }; + + public VexSignalSnapshot ToDomain() + => new( + Severity?.ToDomain(), + Kev, + Epss); +} + +[BsonIgnoreExtraElements] +internal sealed class VexSignatureMetadataDocument +{ + public string Type { get; set; } = default!; + + public string? Subject { get; set; } + = null; + + public string? Issuer { get; set; } + = null; + + public string? KeyId { get; set; } + = null; + + public DateTime? VerifiedAt { get; set; } + = null; + + public string? TransparencyLogReference { get; set; } + = null; + + public static VexSignatureMetadataDocument? FromDomain(VexSignatureMetadata? signature) + => signature is null + ? null + : new VexSignatureMetadataDocument + { + Type = signature.Type, + Subject = signature.Subject, + Issuer = signature.Issuer, + KeyId = signature.KeyId, + VerifiedAt = signature.VerifiedAt?.UtcDateTime, + TransparencyLogReference = signature.TransparencyLogReference, + }; + + public VexSignatureMetadata ToDomain() + { + var verifiedAt = VerifiedAt.HasValue + ? new DateTimeOffset(DateTime.SpecifyKind(VerifiedAt.Value, DateTimeKind.Utc)) + : (DateTimeOffset?)null; + + return new VexSignatureMetadata( + Type, + Subject, + Issuer, + KeyId, + verifiedAt, + TransparencyLogReference); + } +} + +[BsonIgnoreExtraElements] +internal sealed class VexClaimDocumentRecord +{ + public string Format { get; set; } = default!; + + public string Digest { get; set; } = default!; + + public string SourceUri { get; set; } = default!; + + public string? Revision { get; set; } + = null; + + public VexSignatureMetadataDocument? Signature { get; set; } + = null; + + public static VexClaimDocumentRecord FromDomain(VexClaimDocument document) + => new() + { + Format = document.Format.ToString().ToLowerInvariant(), + Digest = document.Digest, + SourceUri = document.SourceUri.ToString(), + Revision = document.Revision, + Signature = VexSignatureMetadataDocument.FromDomain(document.Signature), + }; + + public VexClaimDocument ToDomain() + => new( + Enum.Parse(Format, ignoreCase: true), + Digest, + new Uri(SourceUri), + Revision, + Signature?.ToDomain()); +} + +[BsonIgnoreExtraElements] +internal sealed class VexStatementRecord +{ + [BsonId] + public ObjectId Id { get; set; } + = ObjectId.GenerateNewId(); + + public string VulnerabilityId { get; set; } = default!; + + public string ProviderId { get; set; } = default!; + + public VexProductDocument Product { get; set; } = default!; + + public string Status { get; set; } = default!; + + public string? Justification { get; set; } + = null; + + public string? Detail { get; set; } + = null; + + public VexClaimDocumentRecord Document { get; set; } = default!; + + public DateTime FirstSeen { get; set; } + = DateTime.SpecifyKind(DateTime.UtcNow, DateTimeKind.Utc); + + public DateTime LastSeen { get; set; } + = DateTime.SpecifyKind(DateTime.UtcNow, DateTimeKind.Utc); + + public DateTime InsertedAt { get; set; } + = DateTime.SpecifyKind(DateTime.UtcNow, DateTimeKind.Utc); + + public VexConfidenceDocument? Confidence { get; set; } + = null; + + public VexSignalDocument? Signals { get; set; } + = null; + + public Dictionary AdditionalMetadata { get; set; } = new(StringComparer.Ordinal); + + public static VexStatementRecord FromDomain(VexClaim claim, DateTimeOffset observedAt) + => new() + { + VulnerabilityId = claim.VulnerabilityId, + ProviderId = claim.ProviderId, + Product = VexProductDocument.FromDomain(claim.Product), + Status = claim.Status.ToString().ToLowerInvariant(), + Justification = claim.Justification?.ToString().ToLowerInvariant(), + Detail = claim.Detail, + Document = VexClaimDocumentRecord.FromDomain(claim.Document), + FirstSeen = claim.FirstSeen.UtcDateTime, + LastSeen = claim.LastSeen.UtcDateTime, + InsertedAt = observedAt.UtcDateTime, + Confidence = claim.Confidence is null ? null : VexConfidenceDocument.FromDomain(claim.Confidence), + Signals = VexSignalDocument.FromDomain(claim.Signals), + AdditionalMetadata = claim.AdditionalMetadata.ToDictionary(kvp => kvp.Key, kvp => kvp.Value, StringComparer.Ordinal), + }; + + public VexClaim ToDomain() + { + var firstSeen = new DateTimeOffset(DateTime.SpecifyKind(FirstSeen, DateTimeKind.Utc)); + var lastSeen = new DateTimeOffset(DateTime.SpecifyKind(LastSeen, DateTimeKind.Utc)); + var justification = string.IsNullOrWhiteSpace(Justification) + ? (VexJustification?)null + : Enum.Parse(Justification, ignoreCase: true); + var metadata = (AdditionalMetadata ?? new Dictionary(StringComparer.Ordinal)) + .ToImmutableDictionary(StringComparer.Ordinal); + + return new VexClaim( + VulnerabilityId, + ProviderId, + Product.ToDomain(), + Enum.Parse(Status, ignoreCase: true), + Document.ToDomain(), + firstSeen, + lastSeen, + justification, + Detail, + Confidence?.ToDomain(), + Signals?.ToDomain(), + metadata); + } +} + [BsonIgnoreExtraElements] internal sealed class VexCacheEntryRecord { @@ -582,12 +846,33 @@ internal sealed class VexConnectorStateDocument public List DocumentDigests { get; set; } = new(); + public Dictionary ResumeTokens { get; set; } = new(StringComparer.Ordinal); + + public DateTime? LastSuccessAt { get; set; } + = null; + + public int FailureCount { get; set; } + = 0; + + public DateTime? NextEligibleRun { get; set; } + = null; + + public string? LastFailureReason { get; set; } + = null; + public static VexConnectorStateDocument FromRecord(VexConnectorState state) => new() { ConnectorId = state.ConnectorId, LastUpdated = state.LastUpdated?.UtcDateTime, - DocumentDigests = state.DocumentDigests.ToList(), + DocumentDigests = state.DocumentDigests.IsDefault ? new List() : state.DocumentDigests.ToList(), + ResumeTokens = state.ResumeTokens.Count == 0 + ? new Dictionary(StringComparer.Ordinal) + : state.ResumeTokens.ToDictionary(kvp => kvp.Key, kvp => kvp.Value, StringComparer.Ordinal), + LastSuccessAt = state.LastSuccessAt?.UtcDateTime, + FailureCount = state.FailureCount, + NextEligibleRun = state.NextEligibleRun?.UtcDateTime, + LastFailureReason = state.LastFailureReason, }; public VexConnectorState ToRecord() @@ -595,10 +880,21 @@ internal sealed class VexConnectorStateDocument var lastUpdated = LastUpdated.HasValue ? new DateTimeOffset(DateTime.SpecifyKind(LastUpdated.Value, DateTimeKind.Utc)) : (DateTimeOffset?)null; + var lastSuccessAt = LastSuccessAt.HasValue + ? new DateTimeOffset(DateTime.SpecifyKind(LastSuccessAt.Value, DateTimeKind.Utc)) + : (DateTimeOffset?)null; + var nextEligibleRun = NextEligibleRun.HasValue + ? new DateTimeOffset(DateTime.SpecifyKind(NextEligibleRun.Value, DateTimeKind.Utc)) + : (DateTimeOffset?)null; return new VexConnectorState( ConnectorId, lastUpdated, - DocumentDigests.ToImmutableArray()); + DocumentDigests?.ToImmutableArray() ?? ImmutableArray.Empty, + (ResumeTokens ?? new Dictionary(StringComparer.Ordinal)).ToImmutableDictionary(StringComparer.Ordinal), + lastSuccessAt, + FailureCount, + nextEligibleRun, + string.IsNullOrWhiteSpace(LastFailureReason) ? null : LastFailureReason); } } diff --git a/src/StellaOps.Excititor.Storage.Mongo/VexMongoSessionProvider.cs b/src/StellaOps.Excititor.Storage.Mongo/VexMongoSessionProvider.cs index c16e06ac..887eaa92 100644 --- a/src/StellaOps.Excititor.Storage.Mongo/VexMongoSessionProvider.cs +++ b/src/StellaOps.Excititor.Storage.Mongo/VexMongoSessionProvider.cs @@ -1,36 +1,120 @@ -using MongoDB.Driver; using Microsoft.Extensions.Options; +using MongoDB.Driver; namespace StellaOps.Excititor.Storage.Mongo; -public interface IVexMongoSessionProvider +public interface IVexMongoSessionProvider : IAsyncDisposable { - Task StartSessionAsync(CancellationToken cancellationToken = default); + ValueTask StartSessionAsync(CancellationToken cancellationToken = default); } internal sealed class VexMongoSessionProvider : IVexMongoSessionProvider { private readonly IMongoClient _client; - private readonly IOptions _options; + private readonly VexMongoStorageOptions _options; + private readonly object _gate = new(); + private Task? _sessionTask; + private IClientSessionHandle? _session; + private bool _disposed; public VexMongoSessionProvider(IMongoClient client, IOptions options) { _client = client ?? throw new ArgumentNullException(nameof(client)); - _options = options ?? throw new ArgumentNullException(nameof(options)); + if (options is null) + { + throw new ArgumentNullException(nameof(options)); + } + + _options = options.Value; } - public Task StartSessionAsync(CancellationToken cancellationToken = default) + public async ValueTask StartSessionAsync(CancellationToken cancellationToken = default) + { + ObjectDisposedException.ThrowIf(_disposed, this); + + var existing = Volatile.Read(ref _session); + if (existing is not null) + { + return existing; + } + + Task startTask; + + lock (_gate) + { + if (_session is { } current) + { + return current; + } + + _sessionTask ??= StartSessionInternalAsync(cancellationToken); + startTask = _sessionTask; + } + + try + { + var handle = await startTask.WaitAsync(cancellationToken).ConfigureAwait(false); + if (_session is null) + { + lock (_gate) + { + if (_session is null) + { + _session = handle; + _sessionTask = Task.FromResult(handle); + } + } + } + + return handle; + } + catch + { + lock (_gate) + { + if (ReferenceEquals(_sessionTask, startTask)) + { + _sessionTask = null; + } + } + + throw; + } + } + + private Task StartSessionInternalAsync(CancellationToken cancellationToken) { - var options = _options.Value; var sessionOptions = new ClientSessionOptions { CausalConsistency = true, DefaultTransactionOptions = new TransactionOptions( readPreference: ReadPreference.Primary, readConcern: ReadConcern.Majority, - writeConcern: WriteConcern.WMajority.With(wTimeout: options.CommandTimeout)) + writeConcern: WriteConcern.WMajority.With(wTimeout: _options.CommandTimeout)) }; return _client.StartSessionAsync(sessionOptions, cancellationToken); } + + public ValueTask DisposeAsync() + { + if (_disposed) + { + return ValueTask.CompletedTask; + } + + _disposed = true; + + IClientSessionHandle? handle; + lock (_gate) + { + handle = _session; + _session = null; + _sessionTask = null; + } + + handle?.Dispose(); + GC.SuppressFinalize(this); + return ValueTask.CompletedTask; + } } diff --git a/src/StellaOps.Excititor.Storage.Mongo/VexStatementBackfillService.cs b/src/StellaOps.Excititor.Storage.Mongo/VexStatementBackfillService.cs new file mode 100644 index 00000000..d8ae902a --- /dev/null +++ b/src/StellaOps.Excititor.Storage.Mongo/VexStatementBackfillService.cs @@ -0,0 +1,170 @@ +using System; +using System.Collections.Generic; +using System.Linq; +using System.Threading; +using System.Threading.Tasks; +using Microsoft.Extensions.Logging; +using MongoDB.Driver; +using StellaOps.Excititor.Core; + +namespace StellaOps.Excititor.Storage.Mongo; + +public sealed record VexStatementBackfillRequest( + DateTimeOffset? RetrievedSince = null, + bool Force = false, + int BatchSize = 100, + int? MaxDocuments = null); + +public sealed record VexStatementBackfillResult( + int DocumentsEvaluated, + int DocumentsBackfilled, + int ClaimsWritten, + int SkippedExisting, + int NormalizationFailures); + +public sealed class VexStatementBackfillService +{ + private readonly IVexRawStore _rawStore; + private readonly IVexNormalizerRouter _normalizerRouter; + private readonly IVexClaimStore _claimStore; + private readonly IVexMongoSessionProvider _sessionProvider; + private readonly IMongoCollection _rawCollection; + private readonly IMongoCollection _statementCollection; + private readonly ILogger _logger; + private readonly TimeProvider _timeProvider; + + public VexStatementBackfillService( + IMongoDatabase database, + IVexRawStore rawStore, + IVexNormalizerRouter normalizerRouter, + IVexClaimStore claimStore, + IVexMongoSessionProvider sessionProvider, + TimeProvider? timeProvider, + ILogger logger) + { + ArgumentNullException.ThrowIfNull(database); + _rawStore = rawStore ?? throw new ArgumentNullException(nameof(rawStore)); + _normalizerRouter = normalizerRouter ?? throw new ArgumentNullException(nameof(normalizerRouter)); + _claimStore = claimStore ?? throw new ArgumentNullException(nameof(claimStore)); + _sessionProvider = sessionProvider ?? throw new ArgumentNullException(nameof(sessionProvider)); + _timeProvider = timeProvider ?? TimeProvider.System; + _logger = logger ?? throw new ArgumentNullException(nameof(logger)); + + VexMongoMappingRegistry.Register(); + _rawCollection = database.GetCollection(VexMongoCollectionNames.Raw); + _statementCollection = database.GetCollection(VexMongoCollectionNames.Statements); + } + + public async Task RunAsync(VexStatementBackfillRequest request, CancellationToken cancellationToken) + { + ArgumentNullException.ThrowIfNull(request); + + if (request.BatchSize < 1) + { + throw new ArgumentOutOfRangeException(nameof(request.BatchSize), "Batch size must be at least 1."); + } + + if (request.MaxDocuments is { } max && max <= 0) + { + throw new ArgumentOutOfRangeException(nameof(request.MaxDocuments), "Max documents must be positive when specified."); + } + + var evaluated = 0; + var backfilled = 0; + var claimsWritten = 0; + var skipped = 0; + var failures = 0; + + var filter = request.RetrievedSince is { } since + ? Builders.Filter.Gte(x => x.RetrievedAt, since.UtcDateTime) + : FilterDefinition.Empty; + + var findOptions = new FindOptions + { + Sort = Builders.Sort.Ascending(x => x.RetrievedAt), + BatchSize = request.BatchSize, + }; + + if (request.MaxDocuments is { } limit) + { + findOptions.Limit = limit; + } + + using var cursor = await _rawCollection.FindAsync(filter, findOptions, cancellationToken).ConfigureAwait(false); + + var session = await _sessionProvider.StartSessionAsync(cancellationToken).ConfigureAwait(false); + + while (await cursor.MoveNextAsync(cancellationToken).ConfigureAwait(false)) + { + foreach (var record in cursor.Current) + { + cancellationToken.ThrowIfCancellationRequested(); + + evaluated++; + + if (!request.Force && await StatementExistsAsync(record.Digest, session, cancellationToken).ConfigureAwait(false)) + { + skipped++; + continue; + } + + var rawDocument = await _rawStore.FindByDigestAsync(record.Digest, cancellationToken, session).ConfigureAwait(false); + if (rawDocument is null) + { + failures++; + _logger.LogWarning("Backfill skipped missing raw document {Digest}.", record.Digest); + continue; + } + + VexClaimBatch batch; + try + { + batch = await _normalizerRouter.NormalizeAsync(rawDocument, cancellationToken).ConfigureAwait(false); + } + catch (Exception ex) + { + failures++; + _logger.LogError(ex, "Failed to normalize raw document {Digest} during statement backfill.", record.Digest); + continue; + } + + if (batch.Claims.IsDefaultOrEmpty || batch.Claims.Length == 0) + { + failures++; + _logger.LogWarning("Backfill produced no claims for {Digest}; skipping.", record.Digest); + continue; + } + + var claims = batch.Claims.AsEnumerable(); + var observedAt = rawDocument.RetrievedAt == default + ? _timeProvider.GetUtcNow() + : rawDocument.RetrievedAt; + + await _claimStore.AppendAsync(claims, observedAt, cancellationToken, session).ConfigureAwait(false); + + backfilled++; + claimsWritten += batch.Claims.Length; + } + } + + var result = new VexStatementBackfillResult(evaluated, backfilled, claimsWritten, skipped, failures); + _logger.LogInformation( + "Statement backfill completed: evaluated {Evaluated} documents, backfilled {Backfilled}, wrote {Claims} claims, skipped {Skipped}, failures {Failures}.", + result.DocumentsEvaluated, + result.DocumentsBackfilled, + result.ClaimsWritten, + result.SkippedExisting, + result.NormalizationFailures); + + return result; + } + + private Task StatementExistsAsync(string digest, IClientSessionHandle session, CancellationToken cancellationToken) + { + var filter = Builders.Filter.Eq(x => x.Document.Digest, digest); + var find = session is null + ? _statementCollection.Find(filter) + : _statementCollection.Find(session, filter); + return find.Limit(1).AnyAsync(cancellationToken); + } +} diff --git a/src/StellaOps.Excititor.WebService.Tests/MirrorEndpointsTests.cs b/src/StellaOps.Excititor.WebService.Tests/MirrorEndpointsTests.cs index ccb9f58e..45f863dd 100644 --- a/src/StellaOps.Excititor.WebService.Tests/MirrorEndpointsTests.cs +++ b/src/StellaOps.Excititor.WebService.Tests/MirrorEndpointsTests.cs @@ -150,13 +150,13 @@ public sealed class MirrorEndpointsTests : IClassFixture FindAsync(VexQuerySignature signature, VexExportFormat format, CancellationToken cancellationToken) + public ValueTask FindAsync(VexQuerySignature signature, VexExportFormat format, CancellationToken cancellationToken, IClientSessionHandle? session = null) { _manifests.TryGetValue((signature.Value, format), out var manifest); return ValueTask.FromResult(manifest); } - public ValueTask SaveAsync(VexExportManifest manifest, CancellationToken cancellationToken) + public ValueTask SaveAsync(VexExportManifest manifest, CancellationToken cancellationToken, IClientSessionHandle? session = null) => ValueTask.CompletedTask; } diff --git a/src/StellaOps.Excititor.WebService/Program.cs b/src/StellaOps.Excititor.WebService/Program.cs index f16a57ff..2ddb7cdb 100644 --- a/src/StellaOps.Excititor.WebService/Program.cs +++ b/src/StellaOps.Excititor.WebService/Program.cs @@ -32,7 +32,7 @@ services.AddCsafNormalizer(); services.AddCycloneDxNormalizer(); services.AddOpenVexNormalizer(); services.AddSingleton(); -services.AddSingleton(); +services.AddScoped(); services.AddVexExportEngine(); services.AddVexExportCacheServices(); services.AddVexAttestation(); @@ -140,6 +140,23 @@ app.MapGet("/excititor/statements/{vulnerabilityId}/{productKey}", async ( return Results.Ok(claims); }); +app.MapPost("/excititor/admin/backfill-statements", async ( + VexStatementBackfillRequest? request, + VexStatementBackfillService backfillService, + CancellationToken cancellationToken) => +{ + request ??= new VexStatementBackfillRequest(); + var result = await backfillService.RunAsync(request, cancellationToken).ConfigureAwait(false); + var message = FormattableString.Invariant( + $"Backfill completed: evaluated {result.DocumentsEvaluated}, backfilled {result.DocumentsBackfilled}, claims written {result.ClaimsWritten}, skipped {result.SkippedExisting}, failures {result.NormalizationFailures}."); + + return Results.Ok(new + { + message, + summary = result + }); +}); + IngestEndpoints.MapIngestEndpoints(app); ResolveEndpoint.MapResolveEndpoint(app); MirrorEndpoints.MapMirrorEndpoints(app); diff --git a/src/StellaOps.Excititor.WebService/Services/VexIngestOrchestrator.cs b/src/StellaOps.Excititor.WebService/Services/VexIngestOrchestrator.cs index 722806b2..6f4b5df4 100644 --- a/src/StellaOps.Excititor.WebService/Services/VexIngestOrchestrator.cs +++ b/src/StellaOps.Excititor.WebService/Services/VexIngestOrchestrator.cs @@ -3,6 +3,7 @@ using System.Diagnostics; using System.Globalization; using System.Linq; using Microsoft.Extensions.Logging; +using MongoDB.Driver; using StellaOps.Excititor.Connectors.Abstractions; using StellaOps.Excititor.Core; using StellaOps.Excititor.Storage.Mongo; @@ -30,6 +31,7 @@ internal sealed class VexIngestOrchestrator : IVexIngestOrchestrator private readonly IVexConnectorStateRepository _stateRepository; private readonly IVexNormalizerRouter _normalizerRouter; private readonly IVexSignatureVerifier _signatureVerifier; + private readonly IVexMongoSessionProvider _sessionProvider; private readonly TimeProvider _timeProvider; private readonly ILogger _logger; @@ -42,6 +44,7 @@ internal sealed class VexIngestOrchestrator : IVexIngestOrchestrator IVexConnectorStateRepository stateRepository, IVexNormalizerRouter normalizerRouter, IVexSignatureVerifier signatureVerifier, + IVexMongoSessionProvider sessionProvider, TimeProvider timeProvider, ILogger logger) { @@ -52,6 +55,7 @@ internal sealed class VexIngestOrchestrator : IVexIngestOrchestrator _stateRepository = stateRepository ?? throw new ArgumentNullException(nameof(stateRepository)); _normalizerRouter = normalizerRouter ?? throw new ArgumentNullException(nameof(normalizerRouter)); _signatureVerifier = signatureVerifier ?? throw new ArgumentNullException(nameof(signatureVerifier)); + _sessionProvider = sessionProvider ?? throw new ArgumentNullException(nameof(sessionProvider)); _timeProvider = timeProvider ?? throw new ArgumentNullException(nameof(timeProvider)); _logger = logger ?? throw new ArgumentNullException(nameof(logger)); @@ -73,6 +77,8 @@ internal sealed class VexIngestOrchestrator : IVexIngestOrchestrator var startedAt = _timeProvider.GetUtcNow(); var results = ImmutableArray.CreateBuilder(); + var session = await _sessionProvider.StartSessionAsync(cancellationToken).ConfigureAwait(false); + var (handles, missing) = ResolveConnectors(options.Providers); foreach (var providerId in missing) { @@ -85,7 +91,7 @@ internal sealed class VexIngestOrchestrator : IVexIngestOrchestrator try { await ValidateConnectorAsync(handle, cancellationToken).ConfigureAwait(false); - await EnsureProviderRegistrationAsync(handle.Descriptor, cancellationToken).ConfigureAwait(false); + await EnsureProviderRegistrationAsync(handle.Descriptor, session, cancellationToken).ConfigureAwait(false); stopwatch.Stop(); results.Add(new InitProviderResult( @@ -133,6 +139,7 @@ internal sealed class VexIngestOrchestrator : IVexIngestOrchestrator var startedAt = _timeProvider.GetUtcNow(); var since = ResolveSince(options.Since, options.Window, startedAt); var results = ImmutableArray.CreateBuilder(); + var session = await _sessionProvider.StartSessionAsync(cancellationToken).ConfigureAwait(false); var (handles, missing) = ResolveConnectors(options.Providers); foreach (var providerId in missing) @@ -142,7 +149,7 @@ internal sealed class VexIngestOrchestrator : IVexIngestOrchestrator foreach (var handle in handles) { - var result = await ExecuteRunAsync(handle, since, options.Force, cancellationToken).ConfigureAwait(false); + var result = await ExecuteRunAsync(handle, since, options.Force, session, cancellationToken).ConfigureAwait(false); results.Add(result); } @@ -157,6 +164,7 @@ internal sealed class VexIngestOrchestrator : IVexIngestOrchestrator var runId = Guid.NewGuid(); var startedAt = _timeProvider.GetUtcNow(); var results = ImmutableArray.CreateBuilder(); + var session = await _sessionProvider.StartSessionAsync(cancellationToken).ConfigureAwait(false); var (handles, missing) = ResolveConnectors(options.Providers); foreach (var providerId in missing) @@ -166,8 +174,8 @@ internal sealed class VexIngestOrchestrator : IVexIngestOrchestrator foreach (var handle in handles) { - var since = await ResolveResumeSinceAsync(handle.Descriptor.Id, options.Checkpoint, cancellationToken).ConfigureAwait(false); - var result = await ExecuteRunAsync(handle, since, force: false, cancellationToken).ConfigureAwait(false); + var since = await ResolveResumeSinceAsync(handle.Descriptor.Id, options.Checkpoint, session, cancellationToken).ConfigureAwait(false); + var result = await ExecuteRunAsync(handle, since, force: false, session, cancellationToken).ConfigureAwait(false); results.Add(result); } @@ -183,6 +191,7 @@ internal sealed class VexIngestOrchestrator : IVexIngestOrchestrator var startedAt = _timeProvider.GetUtcNow(); var threshold = options.MaxAge is null ? (DateTimeOffset?)null : startedAt - options.MaxAge.Value; var results = ImmutableArray.CreateBuilder(); + var session = await _sessionProvider.StartSessionAsync(cancellationToken).ConfigureAwait(false); var (handles, missing) = ResolveConnectors(options.Providers); foreach (var providerId in missing) @@ -194,14 +203,14 @@ internal sealed class VexIngestOrchestrator : IVexIngestOrchestrator { try { - var state = await _stateRepository.GetAsync(handle.Descriptor.Id, cancellationToken).ConfigureAwait(false); + var state = await _stateRepository.GetAsync(handle.Descriptor.Id, cancellationToken, session).ConfigureAwait(false); var lastUpdated = state?.LastUpdated; var stale = threshold.HasValue && (lastUpdated is null || lastUpdated < threshold.Value); if (stale || state is null) { var since = stale ? threshold : lastUpdated; - var result = await ExecuteRunAsync(handle, since, force: false, cancellationToken).ConfigureAwait(false); + var result = await ExecuteRunAsync(handle, since, force: false, session, cancellationToken).ConfigureAwait(false); results.Add(new ReconcileProviderResult( handle.Descriptor.Id, result.Status, @@ -262,22 +271,23 @@ internal sealed class VexIngestOrchestrator : IVexIngestOrchestrator await handle.Connector.ValidateAsync(VexConnectorSettings.Empty, cancellationToken).ConfigureAwait(false); } - private async Task EnsureProviderRegistrationAsync(VexConnectorDescriptor descriptor, CancellationToken cancellationToken) + private async Task EnsureProviderRegistrationAsync(VexConnectorDescriptor descriptor, IClientSessionHandle session, CancellationToken cancellationToken) { - var existing = await _providerStore.FindAsync(descriptor.Id, cancellationToken).ConfigureAwait(false); + var existing = await _providerStore.FindAsync(descriptor.Id, cancellationToken, session).ConfigureAwait(false); if (existing is not null) { return; } var provider = new VexProvider(descriptor.Id, descriptor.DisplayName, descriptor.Kind); - await _providerStore.SaveAsync(provider, cancellationToken).ConfigureAwait(false); + await _providerStore.SaveAsync(provider, cancellationToken, session).ConfigureAwait(false); } private async Task ExecuteRunAsync( ConnectorHandle handle, DateTimeOffset? since, bool force, + IClientSessionHandle session, CancellationToken cancellationToken) { var providerId = handle.Descriptor.Id; @@ -287,12 +297,12 @@ internal sealed class VexIngestOrchestrator : IVexIngestOrchestrator try { await ValidateConnectorAsync(handle, cancellationToken).ConfigureAwait(false); - await EnsureProviderRegistrationAsync(handle.Descriptor, cancellationToken).ConfigureAwait(false); + await EnsureProviderRegistrationAsync(handle.Descriptor, session, cancellationToken).ConfigureAwait(false); if (force) { var resetState = new VexConnectorState(providerId, null, ImmutableArray.Empty); - await _stateRepository.SaveAsync(resetState, cancellationToken).ConfigureAwait(false); + await _stateRepository.SaveAsync(resetState, cancellationToken, session).ConfigureAwait(false); } var context = new VexConnectorContext( @@ -316,13 +326,13 @@ internal sealed class VexIngestOrchestrator : IVexIngestOrchestrator if (!batch.Claims.IsDefaultOrEmpty && batch.Claims.Length > 0) { claims += batch.Claims.Length; - await _claimStore.AppendAsync(batch.Claims, _timeProvider.GetUtcNow(), cancellationToken).ConfigureAwait(false); + await _claimStore.AppendAsync(batch.Claims, _timeProvider.GetUtcNow(), cancellationToken, session).ConfigureAwait(false); } } stopwatch.Stop(); var completedAt = _timeProvider.GetUtcNow(); - var state = await _stateRepository.GetAsync(providerId, cancellationToken).ConfigureAwait(false); + var state = await _stateRepository.GetAsync(providerId, cancellationToken, session).ConfigureAwait(false); var checkpoint = state?.DocumentDigests.IsDefaultOrEmpty == false ? state.DocumentDigests[^1] @@ -392,7 +402,7 @@ internal sealed class VexIngestOrchestrator : IVexIngestOrchestrator } } - private async Task ResolveResumeSinceAsync(string providerId, string? checkpoint, CancellationToken cancellationToken) + private async Task ResolveResumeSinceAsync(string providerId, string? checkpoint, IClientSessionHandle session, CancellationToken cancellationToken) { if (!string.IsNullOrWhiteSpace(checkpoint)) { @@ -406,14 +416,14 @@ internal sealed class VexIngestOrchestrator : IVexIngestOrchestrator } var digest = checkpoint.Trim(); - var document = await _rawStore.FindByDigestAsync(digest, cancellationToken).ConfigureAwait(false); + var document = await _rawStore.FindByDigestAsync(digest, cancellationToken, session).ConfigureAwait(false); if (document is not null) { return document.RetrievedAt; } } - var state = await _stateRepository.GetAsync(providerId, cancellationToken).ConfigureAwait(false); + var state = await _stateRepository.GetAsync(providerId, cancellationToken, session).ConfigureAwait(false); return state?.LastUpdated; } diff --git a/src/StellaOps.Excititor.Worker/Scheduling/DefaultVexProviderRunner.cs b/src/StellaOps.Excititor.Worker/Scheduling/DefaultVexProviderRunner.cs index a867e96f..cd02184c 100644 --- a/src/StellaOps.Excititor.Worker/Scheduling/DefaultVexProviderRunner.cs +++ b/src/StellaOps.Excititor.Worker/Scheduling/DefaultVexProviderRunner.cs @@ -68,6 +68,8 @@ internal sealed class DefaultVexProviderRunner : IVexProviderRunner var providerStore = scopeProvider.GetRequiredService(); var normalizerRouter = scopeProvider.GetRequiredService(); var signatureVerifier = scopeProvider.GetRequiredService(); + var sessionProvider = scopeProvider.GetRequiredService(); + var session = await sessionProvider.StartSessionAsync(cancellationToken).ConfigureAwait(false); var descriptor = connector switch { @@ -75,10 +77,10 @@ internal sealed class DefaultVexProviderRunner : IVexProviderRunner _ => new VexConnectorDescriptor(connector.Id, VexProviderKind.Vendor, connector.Id) }; - var provider = await providerStore.FindAsync(descriptor.Id, cancellationToken).ConfigureAwait(false) + var provider = await providerStore.FindAsync(descriptor.Id, cancellationToken, session).ConfigureAwait(false) ?? new VexProvider(descriptor.Id, descriptor.DisplayName, descriptor.Kind); - await providerStore.SaveAsync(provider, cancellationToken).ConfigureAwait(false); + await providerStore.SaveAsync(provider, cancellationToken, session).ConfigureAwait(false); await connector.ValidateAsync(VexConnectorSettings.Empty, cancellationToken).ConfigureAwait(false); @@ -101,7 +103,7 @@ internal sealed class DefaultVexProviderRunner : IVexProviderRunner if (!batch.Claims.IsDefaultOrEmpty && batch.Claims.Length > 0) { claimCount += batch.Claims.Length; - await claimStore.AppendAsync(batch.Claims, _timeProvider.GetUtcNow(), cancellationToken).ConfigureAwait(false); + await claimStore.AppendAsync(batch.Claims, _timeProvider.GetUtcNow(), cancellationToken, session).ConfigureAwait(false); } } diff --git a/src/StellaOps.Notify.Connectors.Email/EmailChannelTestProvider.cs b/src/StellaOps.Notify.Connectors.Email/EmailChannelTestProvider.cs new file mode 100644 index 00000000..14c7d1f7 --- /dev/null +++ b/src/StellaOps.Notify.Connectors.Email/EmailChannelTestProvider.cs @@ -0,0 +1,47 @@ +using System; +using System.Collections.Generic; +using System.Threading; +using System.Threading.Tasks; +using Microsoft.Extensions.DependencyInjection; +using StellaOps.DependencyInjection; +using StellaOps.Notify.Engine; +using StellaOps.Notify.Models; + +namespace StellaOps.Notify.Connectors.Email; + +[ServiceBinding(typeof(INotifyChannelTestProvider), ServiceLifetime.Singleton)] +public sealed class EmailChannelTestProvider : INotifyChannelTestProvider +{ + public NotifyChannelType ChannelType => NotifyChannelType.Email; + + public Task BuildPreviewAsync(ChannelTestPreviewContext context, CancellationToken cancellationToken) + { + cancellationToken.ThrowIfCancellationRequested(); + + var subject = context.Request.Title ?? "Stella Ops Notify Preview"; + var summary = context.Request.Summary ?? $"Preview generated at {context.Timestamp:O}."; + var htmlBody = !string.IsNullOrWhiteSpace(context.Request.Body) + ? context.Request.Body! + : $"

{summary}

Trace: {context.TraceId}

"; + var textBody = context.Request.TextBody ?? $"{summary}{Environment.NewLine}Trace: {context.TraceId}"; + + var preview = NotifyDeliveryRendered.Create( + NotifyChannelType.Email, + NotifyDeliveryFormat.Email, + context.Target, + subject, + htmlBody, + summary, + textBody, + context.Request.Locale, + ChannelTestPreviewUtilities.ComputeBodyHash(htmlBody), + context.Request.Attachments); + + var metadata = new Dictionary(StringComparer.Ordinal) + { + ["email.to"] = context.Target + }; + + return Task.FromResult(new ChannelTestPreviewResult(preview, metadata)); + } +} diff --git a/src/StellaOps.Notify.Connectors.Email/StellaOps.Notify.Connectors.Email.csproj b/src/StellaOps.Notify.Connectors.Email/StellaOps.Notify.Connectors.Email.csproj index 6c3a8871..c103ad31 100644 --- a/src/StellaOps.Notify.Connectors.Email/StellaOps.Notify.Connectors.Email.csproj +++ b/src/StellaOps.Notify.Connectors.Email/StellaOps.Notify.Connectors.Email.csproj @@ -4,4 +4,10 @@ enable enable + + + + + + diff --git a/src/StellaOps.Notify.Connectors.Email/TASKS.md b/src/StellaOps.Notify.Connectors.Email/TASKS.md index 012d4f94..92ad11a9 100644 --- a/src/StellaOps.Notify.Connectors.Email/TASKS.md +++ b/src/StellaOps.Notify.Connectors.Email/TASKS.md @@ -3,5 +3,5 @@ | ID | Status | Owner(s) | Depends on | Description | Exit Criteria | |----|--------|----------|------------|-------------|---------------| | NOTIFY-CONN-EMAIL-15-701 | TODO | Notify Connectors Guild | NOTIFY-ENGINE-15-303 | Implement SMTP connector with STARTTLS/implicit TLS support, HTML+text rendering, attachment policy enforcement. | Integration tests with SMTP stub pass; TLS enforced; attachments blocked per policy. | -| NOTIFY-CONN-EMAIL-15-702 | TODO | Notify Connectors Guild | NOTIFY-CONN-EMAIL-15-701 | Add DKIM signing optional support and health/test-send flows. | DKIM optional config verified; test-send passes; secrets handled securely. | +| NOTIFY-CONN-EMAIL-15-702 | DOING (2025-10-19) | Notify Connectors Guild | NOTIFY-CONN-EMAIL-15-701 | Add DKIM signing optional support and health/test-send flows. | DKIM optional config verified; test-send passes; secrets handled securely. | | NOTIFY-CONN-EMAIL-15-703 | TODO | Notify Connectors Guild | NOTIFY-CONN-EMAIL-15-702 | Package Email connector as restart-time plug-in (manifest + host registration). | Plugin manifest added; host loads connector from `plugins/notify/email/`; restart validation passes. | diff --git a/src/StellaOps.Notify.Connectors.Slack/SlackChannelTestProvider.cs b/src/StellaOps.Notify.Connectors.Slack/SlackChannelTestProvider.cs new file mode 100644 index 00000000..bcedbf91 --- /dev/null +++ b/src/StellaOps.Notify.Connectors.Slack/SlackChannelTestProvider.cs @@ -0,0 +1,70 @@ +using System; +using System.Collections.Generic; +using System.Text.Json; +using System.Threading; +using System.Threading.Tasks; +using Microsoft.Extensions.DependencyInjection; +using StellaOps.DependencyInjection; +using StellaOps.Notify.Engine; +using StellaOps.Notify.Models; + +namespace StellaOps.Notify.Connectors.Slack; + +[ServiceBinding(typeof(INotifyChannelTestProvider), ServiceLifetime.Singleton)] +public sealed class SlackChannelTestProvider : INotifyChannelTestProvider +{ + private static readonly JsonSerializerOptions JsonOptions = new(JsonSerializerDefaults.Web); + + public NotifyChannelType ChannelType => NotifyChannelType.Slack; + + public Task BuildPreviewAsync(ChannelTestPreviewContext context, CancellationToken cancellationToken) + { + cancellationToken.ThrowIfCancellationRequested(); + + var title = context.Request.Title ?? $"Stella Ops Notify Preview"; + var summary = context.Request.Summary ?? $"Preview generated for Slack destination at {context.Timestamp:O}."; + var bodyText = context.Request.Body ?? summary; + + var payload = new + { + text = $"{title}\n{bodyText}", + blocks = new object[] + { + new + { + type = "section", + text = new { type = "mrkdwn", text = $"*{title}*\n{bodyText}" } + }, + new + { + type = "context", + elements = new object[] + { + new { type = "mrkdwn", text = $"Preview generated {context.Timestamp:O} · Trace `{context.TraceId}`" } + } + } + } + }; + + var body = JsonSerializer.Serialize(payload, JsonOptions); + + var preview = NotifyDeliveryRendered.Create( + NotifyChannelType.Slack, + NotifyDeliveryFormat.Slack, + context.Target, + title, + body, + summary, + context.Request.TextBody ?? bodyText, + context.Request.Locale, + ChannelTestPreviewUtilities.ComputeBodyHash(body), + context.Request.Attachments); + + var metadata = new Dictionary(StringComparer.Ordinal) + { + ["slack.channel"] = context.Target + }; + + return Task.FromResult(new ChannelTestPreviewResult(preview, metadata)); + } +} diff --git a/src/StellaOps.Notify.Connectors.Slack/StellaOps.Notify.Connectors.Slack.csproj b/src/StellaOps.Notify.Connectors.Slack/StellaOps.Notify.Connectors.Slack.csproj index 6c3a8871..c103ad31 100644 --- a/src/StellaOps.Notify.Connectors.Slack/StellaOps.Notify.Connectors.Slack.csproj +++ b/src/StellaOps.Notify.Connectors.Slack/StellaOps.Notify.Connectors.Slack.csproj @@ -4,4 +4,10 @@ enable enable + + + + + + diff --git a/src/StellaOps.Notify.Connectors.Slack/TASKS.md b/src/StellaOps.Notify.Connectors.Slack/TASKS.md index 08753de5..67f5b976 100644 --- a/src/StellaOps.Notify.Connectors.Slack/TASKS.md +++ b/src/StellaOps.Notify.Connectors.Slack/TASKS.md @@ -3,5 +3,5 @@ | ID | Status | Owner(s) | Depends on | Description | Exit Criteria | |----|--------|----------|------------|-------------|---------------| | NOTIFY-CONN-SLACK-15-501 | TODO | Notify Connectors Guild | NOTIFY-ENGINE-15-303 | Implement Slack connector with bot token auth, message rendering (blocks), rate limit handling, retries/backoff. | Integration tests stub Slack API; retries/jitter validated; 429 handling documented. | -| NOTIFY-CONN-SLACK-15-502 | TODO | Notify Connectors Guild | NOTIFY-CONN-SLACK-15-501 | Health check & test-send support with minimal scopes and redacted tokens. | `/channels/{id}/test` hitting Slack stub passes; secrets never logged; health endpoint returns diagnostics. | +| NOTIFY-CONN-SLACK-15-502 | DOING (2025-10-19) | Notify Connectors Guild | NOTIFY-CONN-SLACK-15-501 | Health check & test-send support with minimal scopes and redacted tokens. | `/channels/{id}/test` hitting Slack stub passes; secrets never logged; health endpoint returns diagnostics. | | NOTIFY-CONN-SLACK-15-503 | TODO | Notify Connectors Guild | NOTIFY-CONN-SLACK-15-502 | Package Slack connector as restart-time plug-in (manifest + host registration). | Plugin manifest added; host loads connector from `plugins/notify/slack/`; restart validation passes. | diff --git a/src/StellaOps.Notify.Connectors.Teams/StellaOps.Notify.Connectors.Teams.csproj b/src/StellaOps.Notify.Connectors.Teams/StellaOps.Notify.Connectors.Teams.csproj index 6c3a8871..c103ad31 100644 --- a/src/StellaOps.Notify.Connectors.Teams/StellaOps.Notify.Connectors.Teams.csproj +++ b/src/StellaOps.Notify.Connectors.Teams/StellaOps.Notify.Connectors.Teams.csproj @@ -4,4 +4,10 @@ enable enable + + + + + + diff --git a/src/StellaOps.Notify.Connectors.Teams/TASKS.md b/src/StellaOps.Notify.Connectors.Teams/TASKS.md index 5acb4d8d..462d06c8 100644 --- a/src/StellaOps.Notify.Connectors.Teams/TASKS.md +++ b/src/StellaOps.Notify.Connectors.Teams/TASKS.md @@ -3,5 +3,5 @@ | ID | Status | Owner(s) | Depends on | Description | Exit Criteria | |----|--------|----------|------------|-------------|---------------| | NOTIFY-CONN-TEAMS-15-601 | TODO | Notify Connectors Guild | NOTIFY-ENGINE-15-303 | Implement Teams connector using Adaptive Cards 1.5, handle webhook auth, size limits, retries. | Adaptive card payloads validated; 413/429 handling implemented; integration tests cover success/fail. | -| NOTIFY-CONN-TEAMS-15-602 | TODO | Notify Connectors Guild | NOTIFY-CONN-TEAMS-15-601 | Provide health/test-send support with fallback text for legacy clients. | Test-send returns card preview; fallback text logged; docs updated. | +| NOTIFY-CONN-TEAMS-15-602 | DOING (2025-10-19) | Notify Connectors Guild | NOTIFY-CONN-TEAMS-15-601 | Provide health/test-send support with fallback text for legacy clients. | Test-send returns card preview; fallback text logged; docs updated. | | NOTIFY-CONN-TEAMS-15-603 | TODO | Notify Connectors Guild | NOTIFY-CONN-TEAMS-15-602 | Package Teams connector as restart-time plug-in (manifest + host registration). | Plugin manifest added; host loads connector from `plugins/notify/teams/`; restart validation passes. | diff --git a/src/StellaOps.Notify.Connectors.Teams/TeamsChannelTestProvider.cs b/src/StellaOps.Notify.Connectors.Teams/TeamsChannelTestProvider.cs new file mode 100644 index 00000000..b9a778af --- /dev/null +++ b/src/StellaOps.Notify.Connectors.Teams/TeamsChannelTestProvider.cs @@ -0,0 +1,74 @@ +using System; +using System.Collections.Generic; +using System.Text.Json; +using System.Threading; +using System.Threading.Tasks; +using Microsoft.Extensions.DependencyInjection; +using StellaOps.DependencyInjection; +using StellaOps.Notify.Engine; +using StellaOps.Notify.Models; + +namespace StellaOps.Notify.Connectors.Teams; + +[ServiceBinding(typeof(INotifyChannelTestProvider), ServiceLifetime.Singleton)] +public sealed class TeamsChannelTestProvider : INotifyChannelTestProvider +{ + private static readonly JsonSerializerOptions JsonOptions = new(JsonSerializerDefaults.Web); + + public NotifyChannelType ChannelType => NotifyChannelType.Teams; + + public Task BuildPreviewAsync(ChannelTestPreviewContext context, CancellationToken cancellationToken) + { + cancellationToken.ThrowIfCancellationRequested(); + + var title = context.Request.Title ?? "Stella Ops Notify Preview"; + var summary = context.Request.Summary ?? $"Preview generated at {context.Timestamp:O}."; + var bodyContent = context.Request.Body ?? summary; + + var card = new + { + type = "AdaptiveCard", + version = "1.5", + body = new object[] + { + new { type = "TextBlock", weight = "Bolder", text = title, wrap = true }, + new { type = "TextBlock", text = bodyContent, wrap = true }, + new { type = "TextBlock", spacing = "None", isSubtle = true, text = $"Trace: {context.TraceId}", wrap = true } + } + }; + + var payload = new + { + type = "message", + attachments = new object[] + { + new + { + contentType = "application/vnd.microsoft.card.adaptive", + content = card + } + } + }; + + var body = JsonSerializer.Serialize(payload, JsonOptions); + + var preview = NotifyDeliveryRendered.Create( + NotifyChannelType.Teams, + NotifyDeliveryFormat.Teams, + context.Target, + title, + body, + summary, + context.Request.TextBody ?? bodyContent, + context.Request.Locale, + ChannelTestPreviewUtilities.ComputeBodyHash(body), + context.Request.Attachments); + + var metadata = new Dictionary(StringComparer.Ordinal) + { + ["teams.webhook"] = context.Target + }; + + return Task.FromResult(new ChannelTestPreviewResult(preview, metadata)); + } +} diff --git a/src/StellaOps.Notify.Connectors.Webhook/StellaOps.Notify.Connectors.Webhook.csproj b/src/StellaOps.Notify.Connectors.Webhook/StellaOps.Notify.Connectors.Webhook.csproj index 6c3a8871..c103ad31 100644 --- a/src/StellaOps.Notify.Connectors.Webhook/StellaOps.Notify.Connectors.Webhook.csproj +++ b/src/StellaOps.Notify.Connectors.Webhook/StellaOps.Notify.Connectors.Webhook.csproj @@ -4,4 +4,10 @@ enable enable + + + + + + diff --git a/src/StellaOps.Notify.Connectors.Webhook/TASKS.md b/src/StellaOps.Notify.Connectors.Webhook/TASKS.md index a0cc45b5..f5ecd305 100644 --- a/src/StellaOps.Notify.Connectors.Webhook/TASKS.md +++ b/src/StellaOps.Notify.Connectors.Webhook/TASKS.md @@ -3,5 +3,5 @@ | ID | Status | Owner(s) | Depends on | Description | Exit Criteria | |----|--------|----------|------------|-------------|---------------| | NOTIFY-CONN-WEBHOOK-15-801 | TODO | Notify Connectors Guild | NOTIFY-ENGINE-15-303 | Implement webhook connector: JSON payload, signature (HMAC/Ed25519), retries/backoff, status code handling. | Integration tests with webhook stub validate signatures, retries, error handling; payload schema documented. | -| NOTIFY-CONN-WEBHOOK-15-802 | TODO | Notify Connectors Guild | NOTIFY-CONN-WEBHOOK-15-801 | Health/test-send support with signature validation hints and secret management. | Test-send returns success with sample payload; docs include verification guide; secrets never logged. | +| NOTIFY-CONN-WEBHOOK-15-802 | DOING (2025-10-19) | Notify Connectors Guild | NOTIFY-CONN-WEBHOOK-15-801 | Health/test-send support with signature validation hints and secret management. | Test-send returns success with sample payload; docs include verification guide; secrets never logged. | | NOTIFY-CONN-WEBHOOK-15-803 | TODO | Notify Connectors Guild | NOTIFY-CONN-WEBHOOK-15-802 | Package Webhook connector as restart-time plug-in (manifest + host registration). | Plugin manifest added; host loads connector from `plugins/notify/webhook/`; restart validation passes. | diff --git a/src/StellaOps.Notify.Connectors.Webhook/WebhookChannelTestProvider.cs b/src/StellaOps.Notify.Connectors.Webhook/WebhookChannelTestProvider.cs new file mode 100644 index 00000000..e12b380e --- /dev/null +++ b/src/StellaOps.Notify.Connectors.Webhook/WebhookChannelTestProvider.cs @@ -0,0 +1,58 @@ +using System; +using System.Collections.Generic; +using System.Text.Json; +using System.Threading; +using System.Threading.Tasks; +using Microsoft.Extensions.DependencyInjection; +using StellaOps.DependencyInjection; +using StellaOps.Notify.Engine; +using StellaOps.Notify.Models; + +namespace StellaOps.Notify.Connectors.Webhook; + +[ServiceBinding(typeof(INotifyChannelTestProvider), ServiceLifetime.Singleton)] +public sealed class WebhookChannelTestProvider : INotifyChannelTestProvider +{ + private static readonly JsonSerializerOptions JsonOptions = new(JsonSerializerDefaults.Web); + + public NotifyChannelType ChannelType => NotifyChannelType.Webhook; + + public Task BuildPreviewAsync(ChannelTestPreviewContext context, CancellationToken cancellationToken) + { + cancellationToken.ThrowIfCancellationRequested(); + + var title = context.Request.Title ?? "Stella Ops Notify Preview"; + var summary = context.Request.Summary ?? $"Preview generated at {context.Timestamp:O}."; + + var payload = new + { + title, + summary, + traceId = context.TraceId, + timestamp = context.Timestamp, + body = context.Request.Body, + metadata = context.Request.Metadata + }; + + var body = JsonSerializer.Serialize(payload, JsonOptions); + + var preview = NotifyDeliveryRendered.Create( + NotifyChannelType.Webhook, + NotifyDeliveryFormat.Webhook, + context.Target, + title, + body, + summary, + context.Request.TextBody ?? summary, + context.Request.Locale, + ChannelTestPreviewUtilities.ComputeBodyHash(body), + context.Request.Attachments); + + var metadata = new Dictionary(StringComparer.Ordinal) + { + ["webhook.endpoint"] = context.Target + }; + + return Task.FromResult(new ChannelTestPreviewResult(preview, metadata)); + } +} diff --git a/src/StellaOps.Notify.Engine/ChannelTestPreviewContracts.cs b/src/StellaOps.Notify.Engine/ChannelTestPreviewContracts.cs new file mode 100644 index 00000000..7a1e6d8f --- /dev/null +++ b/src/StellaOps.Notify.Engine/ChannelTestPreviewContracts.cs @@ -0,0 +1,84 @@ +using System; +using System.Collections.Generic; +using System.Security.Cryptography; +using System.Text; +using System.Threading; +using System.Threading.Tasks; +using StellaOps.Notify.Models; + +namespace StellaOps.Notify.Engine; + +/// +/// Contract implemented by Notify channel plug-ins to generate channel-specific test preview payloads. +/// +public interface INotifyChannelTestProvider +{ + /// + /// Channel type supported by the provider. + /// + NotifyChannelType ChannelType { get; } + + /// + /// Builds a channel-specific preview for a test-send request. + /// + Task BuildPreviewAsync(ChannelTestPreviewContext context, CancellationToken cancellationToken); +} + +/// +/// Sanitised request payload passed to channel plug-ins when building a preview. +/// +public sealed record ChannelTestPreviewRequest( + string? TargetOverride, + string? TemplateId, + string? Title, + string? Summary, + string? Body, + string? TextBody, + string? Locale, + IReadOnlyDictionary Metadata, + IReadOnlyList Attachments); + +/// +/// Immutable context describing the channel and request for a test preview. +/// +public sealed record ChannelTestPreviewContext( + string TenantId, + NotifyChannel Channel, + string Target, + ChannelTestPreviewRequest Request, + DateTimeOffset Timestamp, + string TraceId); + +/// +/// Result returned by channel plug-ins for test preview generation. +/// +public sealed record ChannelTestPreviewResult( + NotifyDeliveryRendered Preview, + IReadOnlyDictionary? Metadata); + +/// +/// Exception thrown by plug-ins when preview input is invalid. +/// +public sealed class ChannelTestPreviewException : Exception +{ + public ChannelTestPreviewException(string message) + : base(message) + { + } +} + +/// +/// Shared helpers for channel preview generation. +/// +public static class ChannelTestPreviewUtilities +{ + /// + /// Computes a lowercase hex SHA-256 body hash for preview payloads. + /// + public static string ComputeBodyHash(string body) + { + var bytes = Encoding.UTF8.GetBytes(body); + var hash = SHA256.HashData(bytes); + return Convert.ToHexString(hash).ToLowerInvariant(); + } +} diff --git a/src/StellaOps.Notify.Engine/StellaOps.Notify.Engine.csproj b/src/StellaOps.Notify.Engine/StellaOps.Notify.Engine.csproj index 6c3a8871..77de164c 100644 --- a/src/StellaOps.Notify.Engine/StellaOps.Notify.Engine.csproj +++ b/src/StellaOps.Notify.Engine/StellaOps.Notify.Engine.csproj @@ -4,4 +4,8 @@ enable enable + + + + diff --git a/src/StellaOps.Notify.WebService.Tests/CrudEndpointsTests.cs b/src/StellaOps.Notify.WebService.Tests/CrudEndpointsTests.cs index 79019435..61f3a835 100644 --- a/src/StellaOps.Notify.WebService.Tests/CrudEndpointsTests.cs +++ b/src/StellaOps.Notify.WebService.Tests/CrudEndpointsTests.cs @@ -1,13 +1,17 @@ +using System.Collections.Generic; using System.IdentityModel.Tokens.Jwt; using System.IO; using System.Linq; using System.Net; using System.Net.Http; using System.Net.Http.Headers; +using System.Security.Cryptography; using System.Text; using System.Text.Json.Nodes; using Microsoft.AspNetCore.Mvc.Testing; using Microsoft.IdentityModel.Tokens; +using Microsoft.Extensions.DependencyInjection; +using StellaOps.Notify.Engine; using StellaOps.Notify.Models; namespace StellaOps.Notify.WebService.Tests; @@ -34,6 +38,12 @@ public sealed class CrudEndpointsTests : IClassFixture()); + Assert.Null(deliveriesEnvelope? ["continuationToken"]?.GetValue()); + var deliveries = deliveriesEnvelope? ["items"] as JsonArray; Assert.NotNull(deliveries); Assert.NotEmpty(deliveries!.OfType()); @@ -160,6 +174,163 @@ public sealed class CrudEndpointsTests : IClassFixture()); } + [Fact] + public async Task ChannelTestSendReturnsPreview() + { + var client = _factory.CreateClient(); + + var channelPayload = LoadSample("notify-channel@1.sample.json"); + channelPayload["channelId"] = "channel-test"; + channelPayload["tenantId"] = "tenant-web"; + channelPayload["config"]! ["target"] = "#ops-alerts"; + await PostAsync(client, "/api/v1/notify/channels", channelPayload); + + var payload = JsonNode.Parse(""" + { + "target": "#ops-alerts", + "title": "Smoke test", + "body": "Sample body" + } + """)!; + + var response = await PostAsync(client, "/api/v1/notify/channels/channel-test/test", payload); + Assert.Equal(HttpStatusCode.Accepted, response.StatusCode); + + var json = JsonNode.Parse(await response.Content.ReadAsStringAsync())!.AsObject(); + Assert.Equal("tenant-web", json["tenantId"]?.GetValue()); + Assert.Equal("channel-test", json["channelId"]?.GetValue()); + Assert.NotNull(json["queuedAt"]); + Assert.NotNull(json["traceId"]); + + var preview = json["preview"]?.AsObject(); + Assert.NotNull(preview); + Assert.Equal("#ops-alerts", preview? ["target"]?.GetValue()); + Assert.Equal("Smoke test", preview? ["title"]?.GetValue()); + Assert.Equal("Sample body", preview? ["body"]?.GetValue()); + + var expectedHash = Convert.ToHexString(SHA256.HashData(Encoding.UTF8.GetBytes("Sample body"))).ToLowerInvariant(); + Assert.Equal(expectedHash, preview? ["bodyHash"]?.GetValue()); + + var metadata = json["metadata"] as JsonObject; + Assert.NotNull(metadata); + Assert.Equal("#ops-alerts", metadata?["target"]?.GetValue()); + Assert.Equal("slack", metadata?["channelType"]?.GetValue()); + Assert.Equal("fallback", metadata?["previewProvider"]?.GetValue()); + Assert.Equal(json["traceId"]?.GetValue(), metadata?["traceId"]?.GetValue()); + } + + [Fact] + public async Task ChannelTestSendHonoursRateLimit() + { + using var limitedFactory = _factory.WithWebHostBuilder(builder => + { + builder.UseSetting("notify:api:rateLimits:testSend:tokenLimit", "1"); + builder.UseSetting("notify:api:rateLimits:testSend:tokensPerPeriod", "1"); + builder.UseSetting("notify:api:rateLimits:testSend:queueLimit", "0"); + }); + + var client = limitedFactory.CreateClient(); + + var channelPayload = LoadSample("notify-channel@1.sample.json"); + channelPayload["channelId"] = "channel-rate-limit"; + channelPayload["tenantId"] = "tenant-web"; + channelPayload["config"]! ["target"] = "#ops-alerts"; + await PostAsync(client, "/api/v1/notify/channels", channelPayload); + + var payload = JsonNode.Parse(""" + { + "body": "First" + } + """)!; + + var first = await PostAsync(client, "/api/v1/notify/channels/channel-rate-limit/test", payload); + Assert.Equal(HttpStatusCode.Accepted, first.StatusCode); + + var secondRequest = new HttpRequestMessage(HttpMethod.Post, "/api/v1/notify/channels/channel-rate-limit/test") + { + Content = new StringContent(payload.ToJsonString(), Encoding.UTF8, "application/json") + }; + + var second = await SendAsync(client, secondRequest); + Assert.Equal(HttpStatusCode.TooManyRequests, second.StatusCode); + Assert.NotNull(second.Headers.RetryAfter); + } + + [Fact] + public async Task ChannelTestSendUsesRegisteredProvider() + { + var providerName = typeof(FakeSlackTestProvider).FullName!; + + using var providerFactory = _factory.WithWebHostBuilder(builder => + { + builder.ConfigureServices(services => + { + services.AddSingleton(); + }); + }); + + var client = providerFactory.CreateClient(); + + var channelPayload = LoadSample("notify-channel@1.sample.json"); + channelPayload["channelId"] = "channel-provider"; + channelPayload["tenantId"] = "tenant-web"; + channelPayload["config"]! ["target"] = "#ops-alerts"; + await PostAsync(client, "/api/v1/notify/channels", channelPayload); + + var payload = JsonNode.Parse(""" + { + "target": "#ops-alerts", + "title": "Provider Title", + "summary": "Provider Summary" + } + """)!; + + var response = await PostAsync(client, "/api/v1/notify/channels/channel-provider/test", payload); + Assert.Equal(HttpStatusCode.Accepted, response.StatusCode); + + var json = JsonNode.Parse(await response.Content.ReadAsStringAsync())!.AsObject(); + var preview = json["preview"]?.AsObject(); + Assert.NotNull(preview); + Assert.Equal("#ops-alerts", preview?["target"]?.GetValue()); + Assert.Equal("Provider Title", preview?["title"]?.GetValue()); + Assert.Equal("{\"provider\":\"fake\"}", preview?["body"]?.GetValue()); + + var metadata = json["metadata"]?.AsObject(); + Assert.NotNull(metadata); + Assert.Equal(providerName, metadata?["previewProvider"]?.GetValue()); + Assert.Equal("fake-provider", metadata?["provider.name"]?.GetValue()); + } + + private sealed class FakeSlackTestProvider : INotifyChannelTestProvider + { + public NotifyChannelType ChannelType => NotifyChannelType.Slack; + + public Task BuildPreviewAsync(ChannelTestPreviewContext context, CancellationToken cancellationToken) + { + cancellationToken.ThrowIfCancellationRequested(); + + var body = "{\"provider\":\"fake\"}"; + var preview = NotifyDeliveryRendered.Create( + NotifyChannelType.Slack, + NotifyDeliveryFormat.Slack, + context.Target, + context.Request.Title ?? "Provider Title", + body, + context.Request.Summary ?? "Provider Summary", + context.Request.TextBody, + context.Request.Locale, + ChannelTestPreviewUtilities.ComputeBodyHash(body), + context.Request.Attachments); + + var metadata = new Dictionary(StringComparer.Ordinal) + { + ["provider.name"] = "fake-provider" + }; + + return Task.FromResult(new ChannelTestPreviewResult(preview, metadata)); + } + } + private static JsonNode LoadSample(string fileName) { var path = Path.Combine(AppContext.BaseDirectory, fileName); diff --git a/src/StellaOps.Notify.WebService/Contracts/ChannelTestSendRequest.cs b/src/StellaOps.Notify.WebService/Contracts/ChannelTestSendRequest.cs new file mode 100644 index 00000000..06ae3d92 --- /dev/null +++ b/src/StellaOps.Notify.WebService/Contracts/ChannelTestSendRequest.cs @@ -0,0 +1,56 @@ +using System.Collections.Generic; +using System.Text.Json.Serialization; + +namespace StellaOps.Notify.WebService.Contracts; + +/// +/// Payload for Notify channel test send requests. +/// +public sealed record ChannelTestSendRequest +{ + /// + /// Optional override for the default channel destination (email address, webhook URL, Slack channel, etc.). + /// + public string? Target { get; init; } + + /// + /// Optional template identifier to drive future rendering hooks. + /// + public string? TemplateId { get; init; } + + /// + /// Preview title (fallback supplied when omitted). + /// + public string? Title { get; init; } + + /// + /// Optional short summary to show in UI cards. + /// + public string? Summary { get; init; } + + /// + /// Primary body payload rendered for the connector. + /// + public string? Body { get; init; } + + /// + /// Optional text-only representation (used by email/plaintext destinations). + /// + public string? TextBody { get; init; } + + /// + /// Optional locale hint (RFC 5646). + /// + public string? Locale { get; init; } + + /// + /// Optional metadata for future expansion (headers, context labels, etc.). + /// + public IDictionary? Metadata { get; init; } + + /// + /// Optional attachment references emitted with the preview. + /// + [JsonPropertyName("attachments")] + public IList? AttachmentRefs { get; init; } +} diff --git a/src/StellaOps.Notify.WebService/Contracts/ChannelTestSendResponse.cs b/src/StellaOps.Notify.WebService/Contracts/ChannelTestSendResponse.cs new file mode 100644 index 00000000..1144060d --- /dev/null +++ b/src/StellaOps.Notify.WebService/Contracts/ChannelTestSendResponse.cs @@ -0,0 +1,16 @@ +using System; +using System.Collections.Generic; +using StellaOps.Notify.Models; + +namespace StellaOps.Notify.WebService.Contracts; + +/// +/// Response payload summarising a Notify channel test send preview. +/// +public sealed record ChannelTestSendResponse( + string TenantId, + string ChannelId, + NotifyDeliveryRendered Preview, + DateTimeOffset QueuedAt, + string TraceId, + IReadOnlyDictionary Metadata); diff --git a/src/StellaOps.Notify.WebService/Options/NotifyWebServiceOptions.cs b/src/StellaOps.Notify.WebService/Options/NotifyWebServiceOptions.cs index c6b69d0d..95a80394 100644 --- a/src/StellaOps.Notify.WebService/Options/NotifyWebServiceOptions.cs +++ b/src/StellaOps.Notify.WebService/Options/NotifyWebServiceOptions.cs @@ -96,6 +96,47 @@ public sealed class NotifyWebServiceOptions public string InternalBasePath { get; set; } = "/internal/notify"; public string TenantHeader { get; set; } = "X-StellaOps-Tenant"; + + public RateLimitOptions RateLimits { get; set; } = new(); + } + + public sealed class RateLimitOptions + { + public RateLimitPolicyOptions DeliveryHistory { get; set; } = RateLimitPolicyOptions.CreateDefault( + tokenLimit: 60, + tokensPerPeriod: 30, + replenishmentPeriodSeconds: 60, + queueLimit: 20); + + public RateLimitPolicyOptions TestSend { get; set; } = RateLimitPolicyOptions.CreateDefault( + tokenLimit: 5, + tokensPerPeriod: 5, + replenishmentPeriodSeconds: 60, + queueLimit: 2); + } + + public sealed class RateLimitPolicyOptions + { + public bool Enabled { get; set; } = true; + + public int TokenLimit { get; set; } = 10; + + public int TokensPerPeriod { get; set; } = 10; + + public int ReplenishmentPeriodSeconds { get; set; } = 60; + + public int QueueLimit { get; set; } = 0; + + public static RateLimitPolicyOptions CreateDefault(int tokenLimit, int tokensPerPeriod, int replenishmentPeriodSeconds, int queueLimit) + { + return new RateLimitPolicyOptions + { + TokenLimit = tokenLimit, + TokensPerPeriod = tokensPerPeriod, + ReplenishmentPeriodSeconds = replenishmentPeriodSeconds, + QueueLimit = queueLimit + }; + } } public sealed class TelemetryOptions diff --git a/src/StellaOps.Notify.WebService/Options/NotifyWebServiceOptionsValidator.cs b/src/StellaOps.Notify.WebService/Options/NotifyWebServiceOptionsValidator.cs index a4c4c9d0..d78f145d 100644 --- a/src/StellaOps.Notify.WebService/Options/NotifyWebServiceOptionsValidator.cs +++ b/src/StellaOps.Notify.WebService/Options/NotifyWebServiceOptionsValidator.cs @@ -92,5 +92,45 @@ internal static class NotifyWebServiceOptionsValidator { throw new InvalidOperationException("notify:api:tenantHeader must be provided."); } + + ValidateRateLimits(api.RateLimits); + } + + private static void ValidateRateLimits(NotifyWebServiceOptions.RateLimitOptions rateLimits) + { + ArgumentNullException.ThrowIfNull(rateLimits); + + ValidatePolicy(rateLimits.DeliveryHistory, "notify:api:rateLimits:deliveryHistory"); + ValidatePolicy(rateLimits.TestSend, "notify:api:rateLimits:testSend"); + + static void ValidatePolicy(NotifyWebServiceOptions.RateLimitPolicyOptions options, string prefix) + { + ArgumentNullException.ThrowIfNull(options); + + if (!options.Enabled) + { + return; + } + + if (options.TokenLimit <= 0) + { + throw new InvalidOperationException($"{prefix}:tokenLimit must be positive when enabled."); + } + + if (options.TokensPerPeriod <= 0) + { + throw new InvalidOperationException($"{prefix}:tokensPerPeriod must be positive when enabled."); + } + + if (options.ReplenishmentPeriodSeconds <= 0) + { + throw new InvalidOperationException($"{prefix}:replenishmentPeriodSeconds must be positive when enabled."); + } + + if (options.QueueLimit < 0) + { + throw new InvalidOperationException($"{prefix}:queueLimit cannot be negative."); + } + } } } diff --git a/src/StellaOps.Notify.WebService/Program.cs b/src/StellaOps.Notify.WebService/Program.cs index 6721c257..6efb8056 100644 --- a/src/StellaOps.Notify.WebService/Program.cs +++ b/src/StellaOps.Notify.WebService/Program.cs @@ -7,13 +7,16 @@ using System.Text; using System.Text.Json; using System.Text.Json.Nodes; using System.Threading; +using System.Threading.RateLimiting; using Microsoft.AspNetCore.Authentication.JwtBearer; using Microsoft.AspNetCore.Authorization; using Microsoft.AspNetCore.Http; using Microsoft.AspNetCore.Mvc; +using Microsoft.AspNetCore.RateLimiting; using Microsoft.Extensions.Options; using Microsoft.IdentityModel.Tokens; using Microsoft.Extensions.Logging; +using Microsoft.Extensions.Primitives; using Serilog; using Serilog.Events; using StellaOps.Auth.ServerIntegration; @@ -94,8 +97,10 @@ var pluginHostOptions = NotifyPluginHostFactory.Build(bootstrapOptions, contentR builder.Services.AddSingleton(pluginHostOptions); builder.Services.RegisterPluginRoutines(builder.Configuration, pluginHostOptions); builder.Services.AddSingleton(); +builder.Services.AddSingleton(); ConfigureAuthentication(builder, bootstrapOptions); +ConfigureRateLimiting(builder, bootstrapOptions); builder.Services.AddEndpointsApiExplorer(); @@ -178,6 +183,68 @@ static void ConfigureAuthentication(WebApplicationBuilder builder, NotifyWebServ } } +static void ConfigureRateLimiting(WebApplicationBuilder builder, NotifyWebServiceOptions options) +{ + ArgumentNullException.ThrowIfNull(options); + var tenantHeader = options.Api.TenantHeader; + var limits = options.Api.RateLimits; + + builder.Services.AddRateLimiter(rateLimiterOptions => + { + rateLimiterOptions.RejectionStatusCode = StatusCodes.Status429TooManyRequests; + rateLimiterOptions.OnRejected = static (context, _) => + { + context.HttpContext.Response.Headers.TryAdd("Retry-After", "1"); + return ValueTask.CompletedTask; + }; + + ConfigurePolicy(rateLimiterOptions, NotifyRateLimitPolicies.DeliveryHistory, limits.DeliveryHistory, tenantHeader, "deliveries"); + ConfigurePolicy(rateLimiterOptions, NotifyRateLimitPolicies.TestSend, limits.TestSend, tenantHeader, "channel-test"); + }); + + static void ConfigurePolicy( + RateLimiterOptions rateLimiterOptions, + string policyName, + NotifyWebServiceOptions.RateLimitPolicyOptions policy, + string tenantHeader, + string prefix) + { + rateLimiterOptions.AddPolicy(policyName, httpContext => + { + if (policy is null || !policy.Enabled) + { + return RateLimitPartition.GetNoLimiter("notify-disabled"); + } + + var identity = ResolveIdentity(httpContext, tenantHeader, prefix); + + return RateLimitPartition.GetTokenBucketLimiter(identity, _ => new TokenBucketRateLimiterOptions + { + TokenLimit = policy.TokenLimit, + TokensPerPeriod = policy.TokensPerPeriod, + ReplenishmentPeriod = TimeSpan.FromSeconds(policy.ReplenishmentPeriodSeconds), + QueueLimit = policy.QueueLimit, + QueueProcessingOrder = QueueProcessingOrder.OldestFirst, + AutoReplenishment = true + }); + }); + } + + static string ResolveIdentity(HttpContext httpContext, string tenantHeader, string prefix) + { + var tenant = httpContext.Request.Headers.TryGetValue(tenantHeader, out var header) && !StringValues.IsNullOrEmpty(header) + ? header.ToString().Trim() + : "anonymous"; + + var subject = httpContext.User.FindFirst("sub")?.Value + ?? httpContext.User.Identity?.Name + ?? httpContext.Connection.RemoteIpAddress?.ToString() + ?? "anonymous"; + + return string.Concat(prefix, ':', tenant, ':', subject); + } +} + static async Task InitialiseAsync(IServiceProvider services, ServiceStatus status, Microsoft.Extensions.Logging.ILogger logger, NotifyWebServiceOptions options) { var stopwatch = Stopwatch.StartNew(); @@ -244,6 +311,7 @@ static void ConfigureRequestPipeline(WebApplication app, NotifyWebServiceOptions } app.UseAuthentication(); + app.UseRateLimiter(); app.UseAuthorization(); } @@ -383,10 +451,41 @@ static void ConfigureEndpoints(WebApplication app) return Results.BadRequest(new { error = "Tenant mismatch between header and payload." }); } - await repository.UpsertAsync(channel, cancellationToken); - return CreatedJson(BuildResourceLocation(apiBasePath, "channels", channel.ChannelId), channel); + await repository.UpsertAsync(channel, cancellationToken); + return CreatedJson(BuildResourceLocation(apiBasePath, "channels", channel.ChannelId), channel); +}) +.RequireAuthorization(NotifyPolicies.Admin); + + apiGroup.MapPost("/channels/{channelId}/test", async (string channelId, [FromBody] ChannelTestSendRequest? request, INotifyChannelRepository repository, INotifyChannelTestService testService, HttpContext context, CancellationToken cancellationToken) => + { + if (!TryResolveTenant(context, tenantHeader, out var tenant, out var error)) + { + return error!; + } + + if (request is null) + { + return Results.BadRequest(new { error = "Request body is required." }); + } + + var channel = await repository.GetAsync(tenant, channelId, cancellationToken); + if (channel is null) + { + return Results.NotFound(); + } + + try + { + var response = await testService.SendAsync(tenant, channel, request, context.TraceIdentifier, cancellationToken).ConfigureAwait(false); + return JsonResponse(response, StatusCodes.Status202Accepted); + } + catch (ChannelTestSendValidationException ex) + { + return Results.BadRequest(new { error = ex.Message }); + } }) - .RequireAuthorization(NotifyPolicies.Admin); + .RequireAuthorization(NotifyPolicies.Admin) + .RequireRateLimiting(NotifyRateLimitPolicies.TestSend); apiGroup.MapDelete("/channels/{channelId}", async (string channelId, INotifyChannelRepository repository, HttpContext context, CancellationToken cancellationToken) => { @@ -470,17 +569,26 @@ static void ConfigureEndpoints(WebApplication app) }) .RequireAuthorization(NotifyPolicies.Admin); - apiGroup.MapGet("/deliveries", async ([FromServices] INotifyDeliveryRepository repository, HttpContext context, [FromQuery] DateTimeOffset? since, [FromQuery] string? status, [FromQuery] int? limit, CancellationToken cancellationToken) => + apiGroup.MapGet("/deliveries", async ([FromServices] INotifyDeliveryRepository repository, HttpContext context, [FromQuery] DateTimeOffset? since, [FromQuery] string? status, [FromQuery] int? limit, [FromQuery] string? continuationToken, CancellationToken cancellationToken) => { if (!TryResolveTenant(context, tenantHeader, out var tenant, out var error)) { return error!; } - var deliveries = await repository.QueryAsync(tenant, since, status, limit, continuationToken: null, cancellationToken); - return JsonResponse(deliveries.Items); + var effectiveLimit = NormalizeLimit(limit); + var result = await repository.QueryAsync(tenant, since, status, effectiveLimit, continuationToken, cancellationToken).ConfigureAwait(false); + var payload = new + { + items = result.Items, + continuationToken = result.ContinuationToken, + count = result.Items.Count + }; + + return JsonResponse(payload); }) - .RequireAuthorization(NotifyPolicies.Read); + .RequireAuthorization(NotifyPolicies.Read) + .RequireRateLimiting(NotifyRateLimitPolicies.DeliveryHistory); apiGroup.MapGet("/deliveries/{deliveryId}", async (string deliveryId, INotifyDeliveryRepository repository, HttpContext context, CancellationToken cancellationToken) => { @@ -492,7 +600,8 @@ static void ConfigureEndpoints(WebApplication app) var delivery = await repository.GetAsync(tenant, deliveryId, cancellationToken); return delivery is null ? Results.NotFound() : JsonResponse(delivery); }) - .RequireAuthorization(NotifyPolicies.Read); + .RequireAuthorization(NotifyPolicies.Read) + .RequireRateLimiting(NotifyRateLimitPolicies.DeliveryHistory); apiGroup.MapPost("/digests", async ([FromBody] NotifyDigestDocument payload, INotifyDigestRepository repository, HttpContext context, CancellationToken cancellationToken) => { @@ -621,6 +730,16 @@ static void ConfigureEndpoints(WebApplication app) .RequireAuthorization(NotifyPolicies.Read); } +static int NormalizeLimit(int? value) +{ + if (value is null || value <= 0) + { + return 50; + } + + return Math.Min(value.Value, 200); +} + static bool TryResolveTenant(HttpContext context, string tenantHeader, out string tenant, out IResult? error) { if (!context.Request.Headers.TryGetValue(tenantHeader, out var header) || string.IsNullOrWhiteSpace(header)) diff --git a/src/StellaOps.Notify.WebService/Security/NotifyRateLimitPolicies.cs b/src/StellaOps.Notify.WebService/Security/NotifyRateLimitPolicies.cs new file mode 100644 index 00000000..846de9bb --- /dev/null +++ b/src/StellaOps.Notify.WebService/Security/NotifyRateLimitPolicies.cs @@ -0,0 +1,8 @@ +namespace StellaOps.Notify.WebService.Security; + +internal static class NotifyRateLimitPolicies +{ + public const string DeliveryHistory = "notify-deliveries"; + + public const string TestSend = "notify-test-send"; +} diff --git a/src/StellaOps.Notify.WebService/Services/NotifyChannelTestService.cs b/src/StellaOps.Notify.WebService/Services/NotifyChannelTestService.cs new file mode 100644 index 00000000..e56a16ae --- /dev/null +++ b/src/StellaOps.Notify.WebService/Services/NotifyChannelTestService.cs @@ -0,0 +1,313 @@ +using System; +using System.Collections.Generic; +using System.Threading; +using Microsoft.Extensions.Logging; +using StellaOps.Notify.Engine; +using StellaOps.Notify.Models; +using StellaOps.Notify.WebService.Contracts; + +namespace StellaOps.Notify.WebService.Services; + +internal interface INotifyChannelTestService +{ + Task SendAsync( + string tenantId, + NotifyChannel channel, + ChannelTestSendRequest request, + string traceId, + CancellationToken cancellationToken); +} + +internal sealed class NotifyChannelTestService : INotifyChannelTestService +{ + private readonly TimeProvider _timeProvider; + private readonly ILogger _logger; + private readonly IReadOnlyDictionary _providers; + + public NotifyChannelTestService( + TimeProvider timeProvider, + ILogger logger, + IEnumerable providers) + { + _timeProvider = timeProvider ?? throw new ArgumentNullException(nameof(timeProvider)); + _logger = logger ?? throw new ArgumentNullException(nameof(logger)); + _providers = BuildProviderMap(providers ?? Array.Empty(), _logger); + } + + public async Task SendAsync( + string tenantId, + NotifyChannel channel, + ChannelTestSendRequest request, + string traceId, + CancellationToken cancellationToken) + { + ArgumentNullException.ThrowIfNull(channel); + ArgumentNullException.ThrowIfNull(request); + + cancellationToken.ThrowIfCancellationRequested(); + + if (!channel.Enabled) + { + throw new ChannelTestSendValidationException("Channel is disabled. Enable it before issuing test sends."); + } + + var target = ResolveTarget(channel, request.Target); + var timestamp = _timeProvider.GetUtcNow(); + var previewRequest = BuildPreviewRequest(request); + var context = new ChannelTestPreviewContext( + tenantId, + channel, + target, + previewRequest, + timestamp, + traceId); + + ChannelTestPreviewResult? providerResult = null; + var providerName = "fallback"; + + if (_providers.TryGetValue(channel.Type, out var provider)) + { + try + { + providerResult = await provider.BuildPreviewAsync(context, cancellationToken).ConfigureAwait(false); + providerName = provider.GetType().FullName ?? provider.GetType().Name; + } + catch (ChannelTestPreviewException ex) + { + throw new ChannelTestSendValidationException(ex.Message); + } + } + + var rendered = providerResult is not null + ? EnsureBodyHash(providerResult.Preview) + : CreateFallbackPreview(context); + + var metadata = MergeMetadata( + context, + providerName, + providerResult?.Metadata); + + var response = new ChannelTestSendResponse( + tenantId, + channel.ChannelId, + rendered, + timestamp, + traceId, + metadata); + + _logger.LogInformation( + "Notify test send preview generated for tenant {TenantId}, channel {ChannelId} ({ChannelType}) using provider {Provider}.", + tenantId, + channel.ChannelId, + channel.Type, + providerName); + + return response; + } + + private static IReadOnlyDictionary BuildProviderMap( + IEnumerable providers, + ILogger logger) + { + var map = new Dictionary(); + foreach (var provider in providers) + { + if (provider is null) + { + continue; + } + + if (map.TryGetValue(provider.ChannelType, out var existing)) + { + logger?.LogWarning( + "Multiple Notify channel test providers registered for {ChannelType}. Keeping {ExistingProvider} and ignoring {NewProvider}.", + provider.ChannelType, + existing.GetType().FullName, + provider.GetType().FullName); + continue; + } + + map[provider.ChannelType] = provider; + } + + return map; + } + + private static ChannelTestPreviewRequest BuildPreviewRequest(ChannelTestSendRequest request) + { + return new ChannelTestPreviewRequest( + TrimToNull(request.Target), + TrimToNull(request.TemplateId), + TrimToNull(request.Title), + TrimToNull(request.Summary), + request.Body, + TrimToNull(request.TextBody), + TrimToLowerInvariant(request.Locale), + NormalizeInputMetadata(request.Metadata), + NormalizeAttachments(request.AttachmentRefs)); + } + + private static string ResolveTarget(NotifyChannel channel, string? overrideTarget) + { + var target = string.IsNullOrWhiteSpace(overrideTarget) + ? channel.Config.Target ?? channel.Config.Endpoint + : overrideTarget.Trim(); + + if (string.IsNullOrWhiteSpace(target)) + { + throw new ChannelTestSendValidationException("Channel target is required. Provide 'target' or configure channel.config.target/endpoint."); + } + + return target; + } + + private static NotifyDeliveryRendered CreateFallbackPreview(ChannelTestPreviewContext context) + { + var format = MapFormat(context.Channel.Type); + var title = context.Request.Title ?? $"Stella Ops Notify Test ({context.Channel.Name})"; + var body = context.Request.Body ?? BuildDefaultBody(context.Channel, context.Timestamp); + var summary = context.Request.Summary ?? $"Preview generated for {context.Channel.Type} destination."; + + return NotifyDeliveryRendered.Create( + context.Channel.Type, + format, + context.Target, + title, + body, + summary, + context.Request.TextBody, + context.Request.Locale, + ChannelTestPreviewUtilities.ComputeBodyHash(body), + context.Request.Attachments); + } + + private static NotifyDeliveryRendered EnsureBodyHash(NotifyDeliveryRendered preview) + { + if (!string.IsNullOrWhiteSpace(preview.BodyHash)) + { + return preview; + } + + var hash = ChannelTestPreviewUtilities.ComputeBodyHash(preview.Body); + return NotifyDeliveryRendered.Create( + preview.ChannelType, + preview.Format, + preview.Target, + preview.Title, + preview.Body, + preview.Summary, + preview.TextBody, + preview.Locale, + hash, + preview.Attachments); + } + + private static IReadOnlyDictionary MergeMetadata( + ChannelTestPreviewContext context, + string providerName, + IReadOnlyDictionary? providerMetadata) + { + var metadata = new Dictionary(StringComparer.Ordinal) + { + ["channelType"] = context.Channel.Type.ToString().ToLowerInvariant(), + ["target"] = context.Target, + ["previewProvider"] = providerName, + ["traceId"] = context.TraceId + }; + + foreach (var pair in context.Request.Metadata) + { + metadata[pair.Key] = pair.Value; + } + + if (providerMetadata is not null) + { + foreach (var pair in providerMetadata) + { + if (string.IsNullOrWhiteSpace(pair.Key) || pair.Value is null) + { + continue; + } + + metadata[pair.Key.Trim()] = pair.Value; + } + } + + return metadata; + } + + private static NotifyDeliveryFormat MapFormat(NotifyChannelType type) + => type switch + { + NotifyChannelType.Slack => NotifyDeliveryFormat.Slack, + NotifyChannelType.Teams => NotifyDeliveryFormat.Teams, + NotifyChannelType.Email => NotifyDeliveryFormat.Email, + NotifyChannelType.Webhook => NotifyDeliveryFormat.Webhook, + _ => NotifyDeliveryFormat.Json + }; + + private static string BuildDefaultBody(NotifyChannel channel, DateTimeOffset timestamp) + { + return $"This is a Stella Ops Notify test message for channel '{channel.Name}' " + + $"({channel.ChannelId}, type {channel.Type}). Generated at {timestamp:O}."; + } + + private static IReadOnlyDictionary NormalizeInputMetadata(IDictionary? source) + { + if (source is null || source.Count == 0) + { + return new Dictionary(StringComparer.Ordinal); + } + + var result = new Dictionary(source.Count, StringComparer.Ordinal); + foreach (var pair in source) + { + if (string.IsNullOrWhiteSpace(pair.Key) || string.IsNullOrWhiteSpace(pair.Value)) + { + continue; + } + + result[pair.Key.Trim()] = pair.Value.Trim(); + } + + return result; + } + + private static IReadOnlyList NormalizeAttachments(IList? attachments) + { + if (attachments is null || attachments.Count == 0) + { + return Array.Empty(); + } + + var list = new List(attachments.Count); + foreach (var attachment in attachments) + { + if (string.IsNullOrWhiteSpace(attachment)) + { + continue; + } + + list.Add(attachment.Trim()); + } + + return list.Count == 0 ? Array.Empty() : list; + } + + private static string? TrimToNull(string? value) + => string.IsNullOrWhiteSpace(value) ? null : value.Trim(); + + private static string? TrimToLowerInvariant(string? value) + { + var trimmed = TrimToNull(value); + return trimmed?.ToLowerInvariant(); + } +} + +internal sealed class ChannelTestSendValidationException : Exception +{ + public ChannelTestSendValidationException(string message) + : base(message) + { + } +} diff --git a/src/StellaOps.Notify.WebService/StellaOps.Notify.WebService.csproj b/src/StellaOps.Notify.WebService/StellaOps.Notify.WebService.csproj index 351ed673..561f8fca 100644 --- a/src/StellaOps.Notify.WebService/StellaOps.Notify.WebService.csproj +++ b/src/StellaOps.Notify.WebService/StellaOps.Notify.WebService.csproj @@ -18,6 +18,7 @@ + diff --git a/src/StellaOps.Notify.WebService/TASKS.md b/src/StellaOps.Notify.WebService/TASKS.md index e4c3103c..cc3e3b50 100644 --- a/src/StellaOps.Notify.WebService/TASKS.md +++ b/src/StellaOps.Notify.WebService/TASKS.md @@ -4,5 +4,5 @@ |----|--------|----------|------------|-------------|---------------| | NOTIFY-WEB-15-101 | DONE (2025-10-19) | Notify WebService Guild | NOTIFY-MODELS-15-101 | Bootstrap minimal API host with Authority auth, health endpoints, and plug-in discovery per architecture. | Service starts with config validation, `/healthz`/`/readyz` pass, plug-ins loaded at restart. | | NOTIFY-WEB-15-102 | DONE (2025-10-19) | Notify WebService Guild | NOTIFY-WEB-15-101 | Rules/channel/template CRUD endpoints with tenant scoping, validation, audit logging. | CRUD endpoints tested; invalid inputs rejected; audit entries persisted. | -| NOTIFY-WEB-15-103 | TODO | Notify WebService Guild | NOTIFY-WEB-15-102 | Delivery history + test-send endpoints with rate limits. | `/deliveries` and `/channels/{id}/test` tested; rate limits enforced. | +| NOTIFY-WEB-15-103 | DONE (2025-10-19) | Notify WebService Guild | NOTIFY-WEB-15-102 | Delivery history + test-send endpoints with rate limits. | `/deliveries` and `/channels/{id}/test` tested; rate limits enforced. | | NOTIFY-WEB-15-104 | TODO | Notify WebService Guild | NOTIFY-STORAGE-15-201, NOTIFY-QUEUE-15-401 | Configuration binding for Mongo/queue/secrets; startup diagnostics. | Misconfiguration fails fast; diagnostics logged; integration tests cover env overrides. | diff --git a/src/StellaOps.Plugin.Tests/DependencyInjection/PluginServiceRegistrationTests.cs b/src/StellaOps.Plugin.Tests/DependencyInjection/PluginServiceRegistrationTests.cs new file mode 100644 index 00000000..022a879d --- /dev/null +++ b/src/StellaOps.Plugin.Tests/DependencyInjection/PluginServiceRegistrationTests.cs @@ -0,0 +1,112 @@ +using System.Linq; +using Microsoft.Extensions.DependencyInjection; +using Microsoft.Extensions.Logging.Abstractions; +using StellaOps.DependencyInjection; +using StellaOps.Plugin.DependencyInjection; +using Xunit; + +namespace StellaOps.Plugin.Tests.DependencyInjection; + +public sealed class PluginServiceRegistrationTests +{ + [Fact] + public void RegisterAssemblyMetadata_RegistersScopedDescriptor() + { + var services = new ServiceCollection(); + + PluginServiceRegistration.RegisterAssemblyMetadata( + services, + typeof(ScopedTestService).Assembly, + NullLogger.Instance); + + var descriptor = Assert.Single(services, static d => d.ServiceType == typeof(IScopedService)); + Assert.Equal(ServiceLifetime.Scoped, descriptor.Lifetime); + Assert.Equal(typeof(ScopedTestService), descriptor.ImplementationType); + } + + [Fact] + public void RegisterAssemblyMetadata_HonoursRegisterAsSelf() + { + var services = new ServiceCollection(); + + PluginServiceRegistration.RegisterAssemblyMetadata( + services, + typeof(SelfRegisteringService).Assembly, + NullLogger.Instance); + + Assert.Contains(services, static d => + d.ServiceType == typeof(SelfRegisteringService) && + d.ImplementationType == typeof(SelfRegisteringService)); + } + + [Fact] + public void RegisterAssemblyMetadata_ReplacesExistingDescriptorsWhenRequested() + { + var services = new ServiceCollection(); + services.AddSingleton(); + + PluginServiceRegistration.RegisterAssemblyMetadata( + services, + typeof(ReplacementService).Assembly, + NullLogger.Instance); + + var descriptor = Assert.Single( + services, + static d => d.ServiceType == typeof(IReplacementService) && + d.ImplementationType == typeof(ReplacementService)); + Assert.Equal(ServiceLifetime.Transient, descriptor.Lifetime); + } + + [Fact] + public void RegisterAssemblyMetadata_SkipsInvalidAssignments() + { + var services = new ServiceCollection(); + + PluginServiceRegistration.RegisterAssemblyMetadata( + services, + typeof(InvalidServiceBinding).Assembly, + NullLogger.Instance); + + Assert.DoesNotContain(services, static d => d.ServiceType == typeof(IAnotherService)); + } + + private interface IScopedService + { + } + + private interface ISelfContract + { + } + + private interface IReplacementService + { + } + + private interface IAnotherService + { + } + + private sealed class ExistingReplacementService : IReplacementService + { + } + + [ServiceBinding(typeof(IScopedService), ServiceLifetime.Scoped)] + private sealed class ScopedTestService : IScopedService + { + } + + [ServiceBinding(typeof(ISelfContract), ServiceLifetime.Singleton, RegisterAsSelf = true)] + private sealed class SelfRegisteringService : ISelfContract + { + } + + [ServiceBinding(typeof(IReplacementService), ServiceLifetime.Transient, ReplaceExisting = true)] + private sealed class ReplacementService : IReplacementService + { + } + + [ServiceBinding(typeof(IAnotherService), ServiceLifetime.Singleton)] + private sealed class InvalidServiceBinding + { + } +} diff --git a/src/StellaOps.Plugin.Tests/StellaOps.Plugin.Tests.csproj b/src/StellaOps.Plugin.Tests/StellaOps.Plugin.Tests.csproj new file mode 100644 index 00000000..51ff787e --- /dev/null +++ b/src/StellaOps.Plugin.Tests/StellaOps.Plugin.Tests.csproj @@ -0,0 +1,18 @@ + + + net10.0 + enable + enable + false + + + + + + + + + + + + diff --git a/src/StellaOps.Plugin/DependencyInjection/PluginDependencyInjectionExtensions.cs b/src/StellaOps.Plugin/DependencyInjection/PluginDependencyInjectionExtensions.cs index 40d72992..24de33ec 100644 --- a/src/StellaOps.Plugin/DependencyInjection/PluginDependencyInjectionExtensions.cs +++ b/src/StellaOps.Plugin/DependencyInjection/PluginDependencyInjectionExtensions.cs @@ -33,15 +33,17 @@ public static class PluginDependencyInjectionExtensions throw new ArgumentNullException(nameof(options)); } - var loadResult = PluginHost.LoadPlugins(options, logger); - - foreach (var plugin in loadResult.Plugins) - { - foreach (var routine in CreateRoutines(plugin.Assembly)) - { - logger?.LogDebug( - "Registering DI routine '{RoutineType}' from plugin '{PluginAssembly}'.", - routine.GetType().FullName, + var loadResult = PluginHost.LoadPlugins(options, logger); + + foreach (var plugin in loadResult.Plugins) + { + PluginServiceRegistration.RegisterAssemblyMetadata(services, plugin.Assembly, logger); + + foreach (var routine in CreateRoutines(plugin.Assembly)) + { + logger?.LogDebug( + "Registering DI routine '{RoutineType}' from plugin '{PluginAssembly}'.", + routine.GetType().FullName, plugin.Assembly.FullName); routine.Register(services, configuration); @@ -88,4 +90,4 @@ public static class PluginDependencyInjectionExtensions } } } -} \ No newline at end of file +} diff --git a/src/StellaOps.Plugin/DependencyInjection/PluginServiceRegistration.cs b/src/StellaOps.Plugin/DependencyInjection/PluginServiceRegistration.cs new file mode 100644 index 00000000..6085bce2 --- /dev/null +++ b/src/StellaOps.Plugin/DependencyInjection/PluginServiceRegistration.cs @@ -0,0 +1,169 @@ +using System; +using System.Linq; +using System.Reflection; +using Microsoft.Extensions.DependencyInjection; +using Microsoft.Extensions.Logging; +using StellaOps.DependencyInjection; +using StellaOps.Plugin.Internal; + +namespace StellaOps.Plugin.DependencyInjection; + +public static class PluginServiceRegistration +{ + public static void RegisterAssemblyMetadata(IServiceCollection services, Assembly assembly, ILogger? logger) + { + ArgumentNullException.ThrowIfNull(services); + ArgumentNullException.ThrowIfNull(assembly); + + foreach (var implementationType in assembly.GetLoadableTypes()) + { + if (implementationType is null || !implementationType.IsClass || implementationType.IsAbstract) + { + continue; + } + + var attributes = implementationType.GetCustomAttributes(inherit: false); + if (!attributes.Any()) + { + continue; + } + + foreach (var attribute in attributes) + { + try + { + ApplyBinding(services, implementationType, attribute, logger); + } + catch (Exception ex) + { + logger?.LogWarning( + ex, + "Failed to register service binding for implementation '{Implementation}' declared in assembly '{Assembly}'.", + implementationType.FullName ?? implementationType.Name, + assembly.FullName ?? assembly.GetName().Name); + } + } + } + } + + private static void ApplyBinding( + IServiceCollection services, + Type implementationType, + ServiceBindingAttribute attribute, + ILogger? logger) + { + var serviceType = attribute.ServiceType ?? implementationType; + + if (!IsValidBinding(serviceType, implementationType)) + { + logger?.LogWarning( + "Service binding metadata ignored: implementation '{Implementation}' is not assignable to service '{Service}'.", + implementationType.FullName ?? implementationType.Name, + serviceType.FullName ?? serviceType.Name); + return; + } + + if (attribute.ReplaceExisting) + { + RemoveExistingDescriptors(services, serviceType); + } + + AddDescriptorIfMissing(services, serviceType, implementationType, attribute.Lifetime, logger); + + if (attribute.RegisterAsSelf && serviceType != implementationType) + { + AddDescriptorIfMissing(services, implementationType, implementationType, attribute.Lifetime, logger); + } + } + + private static bool IsValidBinding(Type serviceType, Type implementationType) + { + if (serviceType.IsGenericTypeDefinition) + { + return implementationType.IsGenericTypeDefinition + && implementationType.IsClass + && implementationType.IsAssignableToGenericTypeDefinition(serviceType); + } + + return serviceType.IsAssignableFrom(implementationType); + } + + private static void AddDescriptorIfMissing( + IServiceCollection services, + Type serviceType, + Type implementationType, + ServiceLifetime lifetime, + ILogger? logger) + { + if (services.Any(descriptor => + descriptor.ServiceType == serviceType && + descriptor.ImplementationType == implementationType)) + { + logger?.LogDebug( + "Skipping duplicate service binding for {ServiceType} -> {ImplementationType}.", + serviceType.FullName ?? serviceType.Name, + implementationType.FullName ?? implementationType.Name); + return; + } + + ServiceDescriptor descriptor; + if (serviceType.IsGenericTypeDefinition || implementationType.IsGenericTypeDefinition) + { + descriptor = ServiceDescriptor.Describe(serviceType, implementationType, lifetime); + } + else + { + descriptor = new ServiceDescriptor(serviceType, implementationType, lifetime); + } + + services.Add(descriptor); + logger?.LogDebug( + "Registered service binding {ServiceType} -> {ImplementationType} with {Lifetime} lifetime.", + serviceType.FullName ?? serviceType.Name, + implementationType.FullName ?? implementationType.Name, + lifetime); + } + + private static void RemoveExistingDescriptors(IServiceCollection services, Type serviceType) + { + for (var i = services.Count - 1; i >= 0; i--) + { + if (services[i].ServiceType == serviceType) + { + services.RemoveAt(i); + } + } + } + + private static bool IsAssignableToGenericTypeDefinition(this Type implementationType, Type serviceTypeDefinition) + { + if (!serviceTypeDefinition.IsGenericTypeDefinition) + { + return false; + } + + if (implementationType == serviceTypeDefinition) + { + return true; + } + + if (implementationType.IsGenericType && implementationType.GetGenericTypeDefinition() == serviceTypeDefinition) + { + return true; + } + + var interfaces = implementationType.GetInterfaces(); + foreach (var iface in interfaces) + { + if (iface.IsGenericType && iface.GetGenericTypeDefinition() == serviceTypeDefinition) + { + return true; + } + } + + var baseType = implementationType.BaseType; + return baseType is not null && baseType.IsGenericTypeDefinition + ? baseType.GetGenericTypeDefinition() == serviceTypeDefinition + : baseType is not null && baseType.IsAssignableToGenericTypeDefinition(serviceTypeDefinition); + } +} diff --git a/src/StellaOps.Plugin/Hosting/PluginHost.cs b/src/StellaOps.Plugin/Hosting/PluginHost.cs index b37a1c03..3b1b843b 100644 --- a/src/StellaOps.Plugin/Hosting/PluginHost.cs +++ b/src/StellaOps.Plugin/Hosting/PluginHost.cs @@ -71,10 +71,13 @@ public static class PluginHost private static string ResolvePluginDirectory(PluginHostOptions options, string baseDirectory) { - if (string.IsNullOrWhiteSpace(options.PluginsDirectory)) - { - return Path.Combine(baseDirectory, "PluginBinaries"); - } + if (string.IsNullOrWhiteSpace(options.PluginsDirectory)) + { + var defaultDirectory = !string.IsNullOrWhiteSpace(options.PrimaryPrefix) + ? $"{options.PrimaryPrefix}.PluginBinaries" + : "PluginBinaries"; + return Path.Combine(baseDirectory, defaultDirectory); + } if (Path.IsPathRooted(options.PluginsDirectory)) { @@ -213,4 +216,4 @@ public static class PluginHost return false; } -} \ No newline at end of file +} diff --git a/src/StellaOps.Plugin/Hosting/PluginHostOptions.cs b/src/StellaOps.Plugin/Hosting/PluginHostOptions.cs index 634a2f2e..71146697 100644 --- a/src/StellaOps.Plugin/Hosting/PluginHostOptions.cs +++ b/src/StellaOps.Plugin/Hosting/PluginHostOptions.cs @@ -16,8 +16,8 @@ public sealed class PluginHostOptions public string? BaseDirectory { get; set; } /// - /// Directory that contains plugin assemblies. Relative values are resolved against . - /// Defaults to PluginBinaries under the base directory. + /// Directory that contains plugin assemblies. Relative values are resolved against . + /// Defaults to {PrimaryPrefix}.PluginBinaries when a primary prefix is provided, otherwise PluginBinaries. /// public string? PluginsDirectory { get; set; } @@ -56,4 +56,4 @@ public sealed class PluginHostOptions => string.IsNullOrWhiteSpace(BaseDirectory) ? AppContext.BaseDirectory : Path.GetFullPath(BaseDirectory); -} \ No newline at end of file +} diff --git a/src/StellaOps.Plugin/Properties/AssemblyInfo.cs b/src/StellaOps.Plugin/Properties/AssemblyInfo.cs new file mode 100644 index 00000000..e7f6540f --- /dev/null +++ b/src/StellaOps.Plugin/Properties/AssemblyInfo.cs @@ -0,0 +1,3 @@ +using System.Runtime.CompilerServices; + +[assembly: InternalsVisibleTo("StellaOps.Plugin.Tests")] diff --git a/src/StellaOps.Plugin/TASKS.md b/src/StellaOps.Plugin/TASKS.md index f07fe8d2..7f5af613 100644 --- a/src/StellaOps.Plugin/TASKS.md +++ b/src/StellaOps.Plugin/TASKS.md @@ -1,5 +1,6 @@ # TASKS | Task | Owner(s) | Depends on | Notes | |---|---|---|---| -|PLUGIN-DI-08-001 Scoped service support in plugin bootstrap|Plugin Platform Guild (DOING 2025-10-19)|StellaOps.DependencyInjection|Introduce plugin metadata capable of registering scoped services; update plugin loader to instantiate DI routines once per plugin while honouring lifetime hints; document registration contract.| -|PLUGIN-DI-08-002 Authority plugin integration updates|Plugin Platform Guild, Authority Core|PLUGIN-DI-08-001|Update Authority identity-provider plugin registrar/registry to resolve scoped services correctly; adjust bootstrap flows and background services to create scopes when needed; add regression tests.| +|PLUGIN-DI-08-001 Scoped service support in plugin bootstrap|Plugin Platform Guild (DONE 2025-10-19)|StellaOps.DependencyInjection|Introduced `ServiceBindingAttribute` metadata for scoped DI, taught plugin/job loaders to consume it with duplicate-safe registration, added coverage, and refreshed the plug-in SDK guide.| +|PLUGIN-DI-08-002.COORD Authority scoped-service handshake|Plugin Platform Guild, Authority Core (DOING 2025-10-19)|PLUGIN-DI-08-001|Workshop scheduled for 2025-10-20 15:00–16:00 UTC; agenda + attendee list tracked in `docs/dev/authority-plugin-di-coordination.md`; pending pre-read contributions prior to session.| +|PLUGIN-DI-08-002 Authority plugin integration updates|Plugin Platform Guild, Authority Core|PLUGIN-DI-08-001, PLUGIN-DI-08-002.COORD|Update Authority identity-provider plugin registrar/registry to resolve scoped services correctly; adjust bootstrap flows and background services to create scopes when needed; add regression tests.| diff --git a/src/StellaOps.UI/TASKS.md b/src/StellaOps.UI/TASKS.md index 1f2cfbd6..c807e47d 100644 --- a/src/StellaOps.UI/TASKS.md +++ b/src/StellaOps.UI/TASKS.md @@ -8,5 +8,5 @@ | UI-ADMIN-13-004 | TODO | UI Guild | AUTH-MTLS-11-002 | Deliver admin area (tenants/clients/quotas/licensing) with RBAC + audit hooks. | Admin e2e tests pass; unauthorized access blocked; telemetry wired. | | UI-ATTEST-11-005 | TODO | UI Guild | SIGNER-API-11-101, ATTESTOR-API-11-201 | Attestation visibility (Rekor id, status) on Scan Detail. | UI shows Rekor UUID/status; mock attestation fixtures displayed; tests cover success/failure. | | UI-SCHED-13-005 | TODO | UI Guild | SCHED-WEB-16-101 | Scheduler panel: schedules CRUD, run history, dry-run preview using API/mocks. | Panel functional with mocked endpoints; UX signoff; integration tests added. | -| UI-NOTIFY-13-006 | TODO | UI Guild | NOTIFY-WEB-15-101 | Notify panel: channels/rules CRUD, deliveries view, test send integration. | Panel interacts with mocked Notify API; tests cover rule lifecycle; docs updated. | +| UI-NOTIFY-13-006 | DOING (2025-10-19) | UI Guild | NOTIFY-WEB-15-101 | Notify panel: channels/rules CRUD, deliveries view, test send integration. | Panel interacts with mocked Notify API; tests cover rule lifecycle; docs updated. | | UI-POLICY-13-007 | TODO | UI Guild | POLICY-CORE-09-006, SCANNER-WEB-09-103 | Surface policy confidence metadata (band, age, quiet provenance) on preview and report views. | UI renders new columns/tooltips, accessibility and responsive checks pass, Cypress regression updated with confidence fixtures. | diff --git a/src/StellaOps.sln b/src/StellaOps.sln index dea89c57..4323b558 100644 --- a/src/StellaOps.sln +++ b/src/StellaOps.sln @@ -23,10 +23,12 @@ Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "StellaOps.Authority.Plugins EndProject Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "StellaOps.Authority.Plugin.Standard", "StellaOps.Authority\StellaOps.Authority.Plugin.Standard\StellaOps.Authority.Plugin.Standard.csproj", "{93DB06DC-B254-48A9-8F2C-6130A5658F27}" EndProject -Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "StellaOps.Plugin", "StellaOps.Plugin\StellaOps.Plugin.csproj", "{03CA315C-8AA1-4CEA-A28B-5EB35C586F4A}" -EndProject -Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "StellaOps.Cli", "StellaOps.Cli\StellaOps.Cli.csproj", "{40094279-250C-42AE-992A-856718FEFBAC}" -EndProject +Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "StellaOps.Plugin", "StellaOps.Plugin\StellaOps.Plugin.csproj", "{03CA315C-8AA1-4CEA-A28B-5EB35C586F4A}" +EndProject +Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "StellaOps.Plugin.Tests", "StellaOps.Plugin.Tests\StellaOps.Plugin.Tests.csproj", "{C6DC3C29-C2AD-4015-8872-42E95A0FE63F}" +EndProject +Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "StellaOps.Cli", "StellaOps.Cli\StellaOps.Cli.csproj", "{40094279-250C-42AE-992A-856718FEFBAC}" +EndProject Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "StellaOps.Cli.Tests", "StellaOps.Cli.Tests\StellaOps.Cli.Tests.csproj", "{B2967228-F8F7-4931-B257-1C63CB58CE1D}" EndProject Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "StellaOps.Concelier.Testing", "StellaOps.Concelier.Testing\StellaOps.Concelier.Testing.csproj", "{6D52EC2B-0A1A-4693-A8EE-5AB32A4A3ED9}" @@ -438,9 +440,21 @@ Global {03CA315C-8AA1-4CEA-A28B-5EB35C586F4A}.Release|Any CPU.ActiveCfg = Release|Any CPU {03CA315C-8AA1-4CEA-A28B-5EB35C586F4A}.Release|Any CPU.Build.0 = Release|Any CPU {03CA315C-8AA1-4CEA-A28B-5EB35C586F4A}.Release|x64.ActiveCfg = Release|Any CPU - {03CA315C-8AA1-4CEA-A28B-5EB35C586F4A}.Release|x64.Build.0 = Release|Any CPU - {03CA315C-8AA1-4CEA-A28B-5EB35C586F4A}.Release|x86.ActiveCfg = Release|Any CPU - {03CA315C-8AA1-4CEA-A28B-5EB35C586F4A}.Release|x86.Build.0 = Release|Any CPU + {03CA315C-8AA1-4CEA-A28B-5EB35C586F4A}.Release|x64.Build.0 = Release|Any CPU + {03CA315C-8AA1-4CEA-A28B-5EB35C586F4A}.Release|x86.ActiveCfg = Release|Any CPU + {03CA315C-8AA1-4CEA-A28B-5EB35C586F4A}.Release|x86.Build.0 = Release|Any CPU + {C6DC3C29-C2AD-4015-8872-42E95A0FE63F}.Debug|Any CPU.ActiveCfg = Debug|Any CPU + {C6DC3C29-C2AD-4015-8872-42E95A0FE63F}.Debug|Any CPU.Build.0 = Debug|Any CPU + {C6DC3C29-C2AD-4015-8872-42E95A0FE63F}.Debug|x64.ActiveCfg = Debug|Any CPU + {C6DC3C29-C2AD-4015-8872-42E95A0FE63F}.Debug|x64.Build.0 = Debug|Any CPU + {C6DC3C29-C2AD-4015-8872-42E95A0FE63F}.Debug|x86.ActiveCfg = Debug|Any CPU + {C6DC3C29-C2AD-4015-8872-42E95A0FE63F}.Debug|x86.Build.0 = Debug|Any CPU + {C6DC3C29-C2AD-4015-8872-42E95A0FE63F}.Release|Any CPU.ActiveCfg = Release|Any CPU + {C6DC3C29-C2AD-4015-8872-42E95A0FE63F}.Release|Any CPU.Build.0 = Release|Any CPU + {C6DC3C29-C2AD-4015-8872-42E95A0FE63F}.Release|x64.ActiveCfg = Release|Any CPU + {C6DC3C29-C2AD-4015-8872-42E95A0FE63F}.Release|x64.Build.0 = Release|Any CPU + {C6DC3C29-C2AD-4015-8872-42E95A0FE63F}.Release|x86.ActiveCfg = Release|Any CPU + {C6DC3C29-C2AD-4015-8872-42E95A0FE63F}.Release|x86.Build.0 = Release|Any CPU {40094279-250C-42AE-992A-856718FEFBAC}.Debug|Any CPU.ActiveCfg = Debug|Any CPU {40094279-250C-42AE-992A-856718FEFBAC}.Debug|Any CPU.Build.0 = Debug|Any CPU {40094279-250C-42AE-992A-856718FEFBAC}.Debug|x64.ActiveCfg = Debug|Any CPU @@ -2207,8 +2221,9 @@ Global {85AB3BB7-C493-4387-B39A-EB299AC37312} = {361838C4-72E2-1C48-5D76-CA6D1A861242} {5C5E91CA-3F98-4E9A-922B-F6415EABD1A3} = {361838C4-72E2-1C48-5D76-CA6D1A861242} {93DB06DC-B254-48A9-8F2C-6130A5658F27} = {361838C4-72E2-1C48-5D76-CA6D1A861242} - {03CA315C-8AA1-4CEA-A28B-5EB35C586F4A} = {827E0CD3-B72D-47B6-A68D-7590B98EB39B} - {40094279-250C-42AE-992A-856718FEFBAC} = {827E0CD3-B72D-47B6-A68D-7590B98EB39B} + {03CA315C-8AA1-4CEA-A28B-5EB35C586F4A} = {827E0CD3-B72D-47B6-A68D-7590B98EB39B} + {C6DC3C29-C2AD-4015-8872-42E95A0FE63F} = {827E0CD3-B72D-47B6-A68D-7590B98EB39B} + {40094279-250C-42AE-992A-856718FEFBAC} = {827E0CD3-B72D-47B6-A68D-7590B98EB39B} {B2967228-F8F7-4931-B257-1C63CB58CE1D} = {827E0CD3-B72D-47B6-A68D-7590B98EB39B} {6D52EC2B-0A1A-4693-A8EE-5AB32A4A3ED9} = {827E0CD3-B72D-47B6-A68D-7590B98EB39B} {37F203A3-624E-4794-9C99-16CAC22C17DF} = {827E0CD3-B72D-47B6-A68D-7590B98EB39B}