diff --git a/docs/implplan/SPRINT_0112_0001_0001_concelier_i.md b/docs/implplan/SPRINT_0112_0001_0001_concelier_i.md index e2fb3ac9b..cefb6f04e 100644 --- a/docs/implplan/SPRINT_0112_0001_0001_concelier_i.md +++ b/docs/implplan/SPRINT_0112_0001_0001_concelier_i.md @@ -27,7 +27,7 @@ | 1 | CONCELIER-LNM-21-001 | DONE (2025-11-22) | Await Cartographer schema. | Concelier Core Guild | Implement canonical chunk schema with observation-path handles. | | 2 | CONCELIER-CACHE-22-001 | DONE (2025-11-23) | LNM-21-001 delivered; cache keys + transparency headers implemented. | Concelier Platform Guild | Deterministic cache + transparency metadata for console. | | 3 | CONCELIER-MIRROR-23-001-DEV | DONE (2025-11-23) | Dev mirror path documented and sample generator provided (`docs/modules/concelier/mirror-export.md`); uses existing endpoints with unsigned dev bundle layout. | Concelier + Attestor Guilds | Implement mirror/offline provenance path for advisory chunks (schema, handlers, tests). | -| 3b | DEVOPS-MIRROR-23-001-REL | BLOCKED (Release/DevOps only) | Move to DevOps release sprint; awaits CI signing/publish lanes and Attestor mirror contract. Not a development blocker. | DevOps Guild · Security Guild | Wire CI/release jobs to publish signed mirror/offline provenance artefacts for advisory chunks. | +| 3b | DEVOPS-MIRROR-23-001-REL | BLOCKED (Release/DevOps only) | DEPLOY-MIRROR-23-001 (SPRINT_501_ops_deployment_i) — awaits CI signing/publish lanes + Attestor mirror contract; not a development blocker. | DevOps Guild · Security Guild | Wire CI/release jobs to publish signed mirror/offline provenance artefacts for advisory chunks. | ## Action Tracker | Focus | Action | Owner(s) | Due | Status | @@ -49,6 +49,7 @@ | 2025-11-23 | Implemented deterministic chunk cache transparency headers (key hash, hit, ttl) in WebService; CONCELIER-CACHE-22-001 set to DONE. | Concelier Platform | | 2025-11-23 | Split mirror work: 23-001-DEV remains here (schema/handlers/tests); release publishing moved to DEVOPS-MIRROR-23-001-REL (DevOps sprint, not a dev blocker). | Project Mgmt | | 2025-11-23 | Documented dev mirror/export path and sample generator at `docs/modules/concelier/mirror-export.md`; CONCELIER-MIRROR-23-001-DEV marked DONE. | Implementer | +| 2025-11-23 | Routed release publishing to ops sprint: DEVOPS-MIRROR-23-001-REL now depends on DEPLOY-MIRROR-23-001 (SPRINT_501_ops_deployment_i); dev sprint stays unblocked. | Project Mgmt | ## Decisions & Risks - Keep Concelier aggregation-only; no consensus merges. diff --git a/docs/implplan/SPRINT_0113_0001_0002_concelier_ii.md b/docs/implplan/SPRINT_0113_0001_0002_concelier_ii.md index 8a4a2b9e8..9101777ca 100644 --- a/docs/implplan/SPRINT_0113_0001_0002_concelier_ii.md +++ b/docs/implplan/SPRINT_0113_0001_0002_concelier_ii.md @@ -26,19 +26,19 @@ | P2 | PREP-CONCELIER-LNM-21-002-WAITING-ON-FINALIZE | DONE (2025-11-20) | Due 2025-11-21 · Accountable: Concelier Core Guild · Data Science Guild (`src/Concelier/__Libraries/StellaOps.Concelier.Core`) | Concelier Core Guild · Data Science Guild (`src/Concelier/__Libraries/StellaOps.Concelier.Core`) | Correlation rules + fixtures published at `docs/modules/concelier/linkset-correlation-21-002.md` with samples under `docs/samples/lnm/`. Downstream linkset builder can proceed. | | 1 | CONCELIER-GRAPH-21-001 | DONE | LNM sample fixtures with scopes/relationships added; observation/linkset query tests passing | Concelier Core Guild · Cartographer Guild (`src/Concelier/__Libraries/StellaOps.Concelier.Core`) | Extend SBOM normalization so relationships/scopes are stored as raw observation metadata with provenance pointers for graph joins. | | 2 | CONCELIER-GRAPH-21-002 | DONE (2025-11-22) | PREP-CONCELIER-GRAPH-21-002-PLATFORM-EVENTS-S | Concelier Core Guild · Scheduler Guild (`src/Concelier/__Libraries/StellaOps.Concelier.Core`) | Publish `sbom.observation.updated` events with tenant/context and advisory refs; facts only, no judgments. | -| 3 | CONCELIER-GRAPH-24-101 | BLOCKED (CI runner required) | Implementation and tests pending due to local vstest build hang; needs CI/clean runner to compile WebService.Tests and run `AdvisorySummary` contract tests. | Concelier WebService Guild (`src/Concelier/StellaOps.Concelier.WebService`) | `/advisories/summary` bundles observation/linkset metadata (aliases, confidence, conflicts) for graph overlays; upstream values intact. | -| 4 | CONCELIER-GRAPH-28-102 | BLOCKED (blocked on 24-101 + CI runner) | Awaiting 24-101 completion and CI to execute batch evidence endpoint tests. | Concelier WebService Guild (`src/Concelier/StellaOps.Concelier.WebService`) | Evidence batch endpoints keyed by component sets with provenance/timestamps; no derived severity. | +| 3 | CONCELIER-GRAPH-24-101 | BLOCKED (CI runner required) | DEVOPS-CONCELIER-CI-24-101 (SPRINT_503_ops_devops_i) — needs CI/clean runner + vstest harness to compile WebService.Tests and run `AdvisorySummary` contract tests. | Concelier WebService Guild (`src/Concelier/StellaOps.Concelier.WebService`) | `/advisories/summary` bundles observation/linkset metadata (aliases, confidence, conflicts) for graph overlays; upstream values intact. | +| 4 | CONCELIER-GRAPH-28-102 | BLOCKED (blocked on 24-101 + CI runner) | Depends on 24-101 and DEVOPS-CONCELIER-CI-24-101 (SPRINT_503_ops_devops_i) to execute batch evidence endpoint tests in CI. | Concelier WebService Guild (`src/Concelier/StellaOps.Concelier.WebService`) | Evidence batch endpoints keyed by component sets with provenance/timestamps; no derived severity. | | 5 | CONCELIER-LNM-21-001 | DONE | Start of Link-Not-Merge chain | Concelier Core Guild (`src/Concelier/__Libraries/StellaOps.Concelier.Core`) | Define immutable `advisory_observations` model (per-source fields, version ranges, severity text, provenance metadata, tenant guards). | | 6 | CONCELIER-LNM-21-002 | DONE (2025-11-22) | PREP-CONCELIER-LNM-21-002-WAITING-ON-FINALIZE | Concelier Core Guild · Data Science Guild (`src/Concelier/__Libraries/StellaOps.Concelier.Core`) | Correlation pipelines output linksets with confidence + conflict markers, avoiding value collapse. | | 7 | CONCELIER-LNM-21-003 | DONE (2025-11-22) | Depends on 21-002 | Concelier Core Guild (`src/Concelier/__Libraries/StellaOps.Concelier.Core`) | Record disagreements (severity, CVSS, references) as structured conflict entries. | -| 8 | CONCELIER-LNM-21-004 | BLOCKED (CI runner required) | Depends on 21-003; local test harness blocked (`invalid test source`). Needs CI/clean runner to remove legacy merge logic and run guardrail tests. | Concelier Core Guild (`src/Concelier/__Libraries/StellaOps.Concelier.Core`) | Remove legacy merge/dedup logic; add guardrails/tests to keep ingestion append-only; document linkset supersession. | -| 9 | CONCELIER-LNM-21-005 | BLOCKED (blocked on 21-004 + CI runner) | Awaiting 21-004 completion and CI to run event emission tests. | Concelier Core Guild · Platform Events Guild (`src/Concelier/__Libraries/StellaOps.Concelier.Core`) | Emit `advisory.linkset.updated` events with delta descriptions + observation ids (tenant + provenance only). | -| 10 | CONCELIER-LNM-21-101-DEV | BLOCKED (blocked on 21-005 + CI runner) | Needs CI/clean runner to build Storage.Mongo and validate shard/index migrations. | Concelier Storage Guild (`src/Concelier/__Libraries/StellaOps.Concelier.Storage.Mongo`) | Provision Mongo collections (`advisory_observations`, `advisory_linksets`) with hashed shard keys, tenant indexes, TTL for ingest metadata. | -| 11 | CONCELIER-LNM-21-102-DEV | BLOCKED (blocked on 21-101-DEV + CI runner) | Backfill/rollback tooling needs CI to validate migrations and Offline Kit assets. | Concelier Storage Guild · DevOps Guild (`src/Concelier/__Libraries/StellaOps.Concelier.Storage.Mongo`) | Backfill legacy merged advisories; seed tombstones; provide rollback tooling for Offline Kit. | -| 12 | CONCELIER-LNM-21-103-DEV | BLOCKED (blocked on 21-102-DEV + CI runner) | Object store move requires CI to validate bootstrapper/offline seeds. | Concelier Storage Guild (`src/Concelier/__Libraries/StellaOps.Concelier.Storage.Mongo`) | Move large raw payloads to object storage with deterministic pointers; update bootstrapper/offline seeds; preserve provenance metadata. | -| 13 | CONCELIER-LNM-21-201 | BLOCKED (blocked on 21-103 + CI runner) | WebService tests need CI to compile/run; awaiting storage/object-store completion. | Concelier WebService Guild · BE-Base Platform Guild (`src/Concelier/StellaOps.Concelier.WebService`) | `/advisories/observations` filters by alias/purl/source with strict tenant scopes; echoes upstream values + provenance fields only. | -| 14 | CONCELIER-LNM-21-202 | BLOCKED (blocked on 21-201 + CI runner) | Await upstream and CI to run `/advisories/linksets` export tests. | Concelier WebService Guild (`src/Concelier/StellaOps.Concelier.WebService`) | `/advisories/linksets`/`export`/`evidence` endpoints surface correlation + conflict payloads and `ERR_AGG_*` mapping; no synthesis/merge. | -| 15 | CONCELIER-LNM-21-203 | BLOCKED (blocked on 21-202 + CI runner) | Event publishing tests need CI transport harness. | Concelier WebService Guild · Platform Events Guild (`src/Concelier/StellaOps.Concelier.WebService`) | Publish idempotent NATS/Redis events for new observations/linksets with documented schemas; include tenant + provenance references only. | +| 8 | CONCELIER-LNM-21-004 | BLOCKED (CI runner required) | Depends on 21-003; waiting on DEVOPS-CONCELIER-CI-24-101 (SPRINT_503_ops_devops_i) for CI/clean runner + vstest harness to execute guardrail tests. | Concelier Core Guild (`src/Concelier/__Libraries/StellaOps.Concelier.Core`) | Remove legacy merge/dedup logic; add guardrails/tests to keep ingestion append-only; document linkset supersession. | +| 9 | CONCELIER-LNM-21-005 | BLOCKED (blocked on 21-004 + CI runner) | Awaiting 21-004 completion and DEVOPS-CONCELIER-CI-24-101 to run event emission tests in CI. | Concelier Core Guild · Platform Events Guild (`src/Concelier/__Libraries/StellaOps.Concelier.Core`) | Emit `advisory.linkset.updated` events with delta descriptions + observation ids (tenant + provenance only). | +| 10 | CONCELIER-LNM-21-101-DEV | BLOCKED (blocked on 21-005 + CI runner) | Needs DEVOPS-CONCELIER-CI-24-101 to provide CI/clean runner for Storage.Mongo build + shard/index migration validation. | Concelier Storage Guild (`src/Concelier/__Libraries/StellaOps.Concelier.Storage.Mongo`) | Provision Mongo collections (`advisory_observations`, `advisory_linksets`) with hashed shard keys, tenant indexes, TTL for ingest metadata. | +| 11 | CONCELIER-LNM-21-102-DEV | BLOCKED (blocked on 21-101-DEV + CI runner) | Backfill/rollback tooling waits on DEVOPS-CONCELIER-CI-24-101 CI runner to validate migrations and Offline Kit assets. | Concelier Storage Guild · DevOps Guild (`src/Concelier/__Libraries/StellaOps.Concelier.Storage.Mongo`) | Backfill legacy merged advisories; seed tombstones; provide rollback tooling for Offline Kit. | +| 12 | CONCELIER-LNM-21-103-DEV | BLOCKED (blocked on 21-102-DEV + CI runner) | Requires DEVOPS-CONCELIER-CI-24-101 CI runner to validate object-store bootstrapper/offline seeds. | Concelier Storage Guild (`src/Concelier/__Libraries/StellaOps.Concelier.Storage.Mongo`) | Move large raw payloads to object storage with deterministic pointers; update bootstrapper/offline seeds; preserve provenance metadata. | +| 13 | CONCELIER-LNM-21-201 | BLOCKED (blocked on 21-103 + CI runner) | WebService tests await DEVOPS-CONCELIER-CI-24-101 CI runner after storage/object-store completion. | Concelier WebService Guild · BE-Base Platform Guild (`src/Concelier/StellaOps.Concelier.WebService`) | `/advisories/observations` filters by alias/purl/source with strict tenant scopes; echoes upstream values + provenance fields only. | +| 14 | CONCELIER-LNM-21-202 | BLOCKED (blocked on 21-201 + CI runner) | Await upstream and DEVOPS-CONCELIER-CI-24-101 to run `/advisories/linksets` export tests in CI. | Concelier WebService Guild (`src/Concelier/StellaOps.Concelier.WebService`) | `/advisories/linksets`/`export`/`evidence` endpoints surface correlation + conflict payloads and `ERR_AGG_*` mapping; no synthesis/merge. | +| 15 | CONCELIER-LNM-21-203 | BLOCKED (blocked on 21-202 + CI runner) | Event publishing tests need CI transport harness from DEVOPS-CONCELIER-CI-24-101. | Concelier WebService Guild · Platform Events Guild (`src/Concelier/StellaOps.Concelier.WebService`) | Publish idempotent NATS/Redis events for new observations/linksets with documented schemas; include tenant + provenance references only. | | 16 | CONCELIER-AIRGAP-56-001..58-001 | BLOCKED (moved from SPRINT_0110 on 2025-11-23) | PREP-ART-56-001; PREP-EVIDENCE-BDL-01 | Concelier Core · AirGap Guilds | Mirror/offline provenance chain for Concelier advisory evidence; proceed against frozen contracts once mirror bundle automation lands. | | 17 | CONCELIER-CONSOLE-23-001..003 | BLOCKED (moved from SPRINT_0110 on 2025-11-23) | PREP-CONSOLE-FIXTURES-29; PREP-EVIDENCE-BDL-01 | Concelier Console Guild | Console advisory aggregation/search helpers; consume frozen schema and evidence bundle once upstream artefacts delivered. | | 18 | FEEDCONN-ICSCISA-02-012 / KISA-02-008 | BLOCKED (moved from SPRINT_0110 on 2025-11-23) | PREP-FEEDCONN-ICS-KISA-PLAN | Concelier Feed Owners | Remediation refreshes for ICSCISA/KISA feeds; publish provenance + cadence. | @@ -49,6 +49,7 @@ | 2025-11-23 | Local build of `StellaOps.Concelier.WebService.Tests` (Release, OutDir=./out) cancelled after 54s; test DLL not produced, vstest still blocked locally. Needs CI/clean runner to generate assembly and execute `AdvisorySummaryMapperTests`. | Concelier Core | | 2025-11-23 | Retried WebService.Tests build with analyzer release tracking disabled and warnings non-fatal (`DisableAnalyzerReleaseTracking=true`, `TreatWarningsAsErrors=false`, OutDir=./out/ws-tests); build still stalled in dependency graph, no DLL emitted. CI runner still required to produce test assembly. | Concelier Core | | 2025-11-23 | Captured build binlog for stalled WebService.Tests attempt at `out/ws-tests.binlog` for CI triage. | Concelier Core | +| 2025-11-23 | Split CI runner blocker into DEVOPS-CONCELIER-CI-24-101 (SPRINT_503_ops_devops_i); all CI/vstest-related blocks now point to that ops task. | Project Mgmt | | 2025-11-23 | Marked downstream tasks (GRAPH-24-101/28-102, LNM-21-004..203) BLOCKED pending CI/clean runner; local harness cannot compile or run tests (`invalid test source` / hang). Development awaiting CI resources. Split storage/backfill/object-store tasks into DEV (here) vs DEVOPS release items (10b/11b/12b) to avoid dev blockage. | Project Mgmt | | 2025-11-23 | Imported CONCELIER-AIRGAP-56-001..58-001, CONCELIER-CONSOLE-23-001..003, FEEDCONN-ICSCISA-02-012/KISA-02-008 from SPRINT_0110; statuses remain BLOCKED pending mirror/console/feed artefacts. | Project Mgmt | | 2025-11-20 | Wired optional NATS transport for `advisory.observation.updated@1`; background worker dequeues Mongo outbox and publishes to configured stream/subject. | Implementer | diff --git a/docs/implplan/SPRINT_0114_0001_0003_concelier_iii.md b/docs/implplan/SPRINT_0114_0001_0003_concelier_iii.md index 8e6d7536d..b5308ec9b 100644 --- a/docs/implplan/SPRINT_0114_0001_0003_concelier_iii.md +++ b/docs/implplan/SPRINT_0114_0001_0003_concelier_iii.md @@ -34,10 +34,10 @@ | P7 | PREP-CONCELIER-OBS-53-001-DEPENDS-ON-52-001-B | DONE (2025-11-22) | Due 2025-11-21 · Accountable: Concelier Core Guild · Evidence Locker Guild (`src/Concelier/__Libraries/StellaOps.Concelier.Core`) | Concelier Core Guild · Evidence Locker Guild (`src/Concelier/__Libraries/StellaOps.Concelier.Core`) | Evidence bundle/timeline linkage requirements documented; unblock evidence locker integration. | | P8 | PREP-CONCELIER-OBS-54-001-DEPENDS-ON-OBS-TIME | DONE (2025-11-22) | Due 2025-11-21 · Accountable: Concelier Core Guild · Provenance Guild (`src/Concelier/__Libraries/StellaOps.Concelier.Core`) | Concelier Core Guild · Provenance Guild (`src/Concelier/__Libraries/StellaOps.Concelier.Core`) | Attestation timeline enrichment + DSSE envelope fields recorded in prep note. | | P9 | PREP-CONCELIER-OBS-55-001-DEPENDS-ON-54-001-I | DONE (2025-11-22) | Due 2025-11-21 · Accountable: Concelier Core Guild · DevOps Guild (`src/Concelier/__Libraries/StellaOps.Concelier.Core`) | Concelier Core Guild · DevOps Guild (`src/Concelier/__Libraries/StellaOps.Concelier.Core`) | Incident-mode hooks and sealed-mode redaction guidance captured; see prep note. | -| 10 | CONCELIER-ORCH-32-001 | BLOCKED (2025-11-22) | Build/restore failures on local runner (missing packages, nullable warnings); awaiting CI/clean runner to validate registry wiring. | Concelier Core Guild (`src/Concelier/__Libraries/StellaOps.Concelier.Core`) | Register every advisory connector with orchestrator (metadata, auth scopes, rate policies) for transparent, reproducible scheduling. | -| 11 | CONCELIER-ORCH-32-002 | BLOCKED (2025-11-22) | Blocked on 32-001 build validation; needs CI runner. | Concelier Core Guild (`src/Concelier/__Libraries/StellaOps.Concelier.Core`) | Adopt orchestrator worker SDK in ingestion loops; emit heartbeats/progress/artifact hashes for deterministic replays. | -| 12 | CONCELIER-ORCH-33-001 | BLOCKED (2025-11-22) | Blocked on 32-001/002 build validation; needs CI runner. | Concelier Core Guild (`src/Concelier/__Libraries/StellaOps.Concelier.Core`) | Honor orchestrator pause/throttle/retry controls with structured errors and persisted checkpoints. | -| 13 | CONCELIER-ORCH-34-001 | BLOCKED (2025-11-22) | Blocked on 32-001/002 build validation; needs CI runner. | Concelier Core Guild (`src/Concelier/__Libraries/StellaOps.Concelier.Core`) | Execute orchestrator-driven backfills reusing artifact hashes/signatures, logging provenance, and pushing run metadata to ledger. | +| 10 | CONCELIER-ORCH-32-001 | BLOCKED (2025-11-22) | DEVOPS-CONCELIER-CI-24-101 (SPRINT_503_ops_devops_i) — build/restore fails locally; needs CI/clean runner to validate registry wiring. | Concelier Core Guild (`src/Concelier/__Libraries/StellaOps.Concelier.Core`) | Register every advisory connector with orchestrator (metadata, auth scopes, rate policies) for transparent, reproducible scheduling. | +| 11 | CONCELIER-ORCH-32-002 | BLOCKED (2025-11-22) | Blocked on 32-001 build validation; depends on DEVOPS-CONCELIER-CI-24-101 CI runner. | Concelier Core Guild (`src/Concelier/__Libraries/StellaOps.Concelier.Core`) | Adopt orchestrator worker SDK in ingestion loops; emit heartbeats/progress/artifact hashes for deterministic replays. | +| 12 | CONCELIER-ORCH-33-001 | BLOCKED (2025-11-22) | Blocked on 32-001/002 build validation; needs DEVOPS-CONCELIER-CI-24-101 CI runner. | Concelier Core Guild (`src/Concelier/__Libraries/StellaOps.Concelier.Core`) | Honor orchestrator pause/throttle/retry controls with structured errors and persisted checkpoints. | +| 13 | CONCELIER-ORCH-34-001 | BLOCKED (2025-11-22) | Blocked on 32-001/002 build validation; needs DEVOPS-CONCELIER-CI-24-101 CI runner. | Concelier Core Guild (`src/Concelier/__Libraries/StellaOps.Concelier.Core`) | Execute orchestrator-driven backfills reusing artifact hashes/signatures, logging provenance, and pushing run metadata to ledger. | | 14 | CONCELIER-POLICY-20-001 | DOING (2025-11-23) | OpenAPI source drafted at `src/Concelier/StellaOps.Concelier.WebService/openapi/concelier-lnm.yaml` (published copy: `docs/api/concelier/concelier-lnm.yaml`); list/search/get endpoints exposed, field coverage still partial (no severity/timeline). | Concelier WebService Guild (`src/Concelier/StellaOps.Concelier.WebService`) | Provide batch advisory lookup APIs for Policy Engine (purl/advisory filters, tenant scopes, explain metadata) so policy joins raw evidence without inferred outcomes. | ## Execution Log @@ -74,6 +74,7 @@ | 2025-11-22 | Marked ORCH-32/33/34 BLOCKED pending CI/clean runner build + restore (local runner stuck on missing packages/nullability). | Concelier Core | | 2025-11-22 | Retried `dotnet restore concelier-webservice.slnf -v minimal` with timeout guard; cancelled at ~25s with `NuGet.targets` reporting "Restore canceled!". No packages downloaded; ORCH-32/33/34 remain blocked until CI/warm cache is available. | Concelier Implementer | | 2025-11-22 | Ran `dotnet restore concelier-webservice.slnf -v diag` (60s timeout); aborted after prolonged spinner, no packages fetched, no new diagnostic log produced. Orchestrator tasks stay blocked pending CI/runner with warm cache. | Concelier Implementer | +| 2025-11-23 | Routed ORCH-32/33/34 CI dependency to DEVOPS-CONCELIER-CI-24-101 (SPRINT_503_ops_devops_i); dev sprint waits on ops runner deliverable. | Project Mgmt | ## Decisions & Risks - Link-Not-Merge and OpenAPI alignment must precede SDK/examples; otherwise downstream clients will drift from canonical facts. diff --git a/docs/implplan/SPRINT_0119_0001_0001_excititor_i.md b/docs/implplan/SPRINT_0119_0001_0001_excititor_i.md index 8b056d945..972a43e18 100644 --- a/docs/implplan/SPRINT_0119_0001_0001_excititor_i.md +++ b/docs/implplan/SPRINT_0119_0001_0001_excititor_i.md @@ -91,6 +91,8 @@ | 2025-11-23 | Enforced air-gap import idempotency with unique indexes on `Id` and `(bundleId,mirrorGeneration)`; duplicate imports now return 409 `AIRGAP_IMPORT_DUPLICATE`. Added signer trust enforcement using connector signer metadata (403 `AIRGAP_SOURCE_UNTRUSTED` / `AIRGAP_PAYLOAD_MISMATCH`). Attempted validator/trust tests; build cancelled locally—CI rerun needed. | Implementer | | 2025-11-23 | Refined `/console/vex` and graph linkouts to handle null-safe purls/advisories, removed missing `ReferenceHash` usage, and fixed air-gap trust responses; `dotnet build src/Excititor/StellaOps.Excititor.WebService -c Release` now succeeds. | Implementer | | 2025-11-23 | Ran `dotnet test -c Release --filter AirgapImportEndpointTests --logger trx`; both air-gap endpoint tests now PASS (TRX at `src/Excititor/__Tests/StellaOps.Excititor.WebService.Tests/TestResults/airgap.trx`). Marked EXCITITOR-AIRGAP-56-001 DONE. | Implementer | +| 2025-11-23 | Ran Core unit test `VexEvidenceChunkServiceTests` (`dotnet test -c Release --filter FullyQualifiedName~VexEvidenceChunkServiceTests --logger trx`); PASS (TRX at `src/Excititor/__Tests/StellaOps.Excititor.Core.UnitTests/TestResults/chunks.trx`). | Implementer | +| 2025-11-23 | Ran full Core UnitTests (`dotnet test -c Release --results-directory TestResults --logger trx`); 3 tests executed, all PASS (TRX at `src/Excititor/__Tests/StellaOps.Excititor.Core.UnitTests/TestResults/core-all.trx`). | Implementer | ## Decisions & Risks - **Decisions** diff --git a/docs/implplan/SPRINT_0120_0000_0001_policy_reasoning.md b/docs/implplan/SPRINT_0120_0000_0001_policy_reasoning.md index a23061e27..12fa78ca6 100644 --- a/docs/implplan/SPRINT_0120_0000_0001_policy_reasoning.md +++ b/docs/implplan/SPRINT_0120_0000_0001_policy_reasoning.md @@ -44,7 +44,7 @@ | P3 | PREP-LEDGER-AIRGAP-56-001-MIRROR-BUNDLE-SCHEM | DONE (2025-11-22) | Due 2025-11-21 · Accountable: Findings Ledger Guild / `src/Findings/StellaOps.Findings.Ledger` | Findings Ledger Guild / `src/Findings/StellaOps.Findings.Ledger` | Mirror bundle provenance fields frozen in `docs/modules/findings-ledger/prep/2025-11-22-ledger-airgap-prep.md`; staleness/anchor rules defined. | | 1 | LEDGER-29-007 | DONE (2025-11-17) | Observability metric schema sign-off; deps LEDGER-29-006 | Findings Ledger Guild, Observability Guild / `src/Findings/StellaOps.Findings.Ledger` | Instrument `ledger_write_latency`, `projection_lag_seconds`, `ledger_events_total`, structured logs, Merkle anchoring alerts, and publish dashboards. | | 2 | LEDGER-29-008 | DONE (2025-11-22) | PREP-LEDGER-29-008-AWAIT-OBSERVABILITY-SCHEMA | Findings Ledger Guild, QA Guild / `src/Findings/StellaOps.Findings.Ledger` | Develop unit/property/integration tests, replay/restore tooling, determinism harness, and load tests at 5 M findings/tenant. | -| 3 | LEDGER-29-009-DEV | BLOCKED | Waiting on DevOps to assign target paths for Helm/Compose/offline-kit assets; backup/restore runbook review pending | Findings Ledger Guild / `src/Findings/StellaOps.Findings.Ledger` | Provide Helm/Compose manifests, backup/restore guidance, optional Merkle anchor externalization, and offline kit instructions (dev/staging artifacts). | +| 3 | LEDGER-29-009-DEV | BLOCKED | DEPLOY-LEDGER-29-009 (SPRINT_501_ops_deployment_i) — waiting on DevOps to assign target paths for Helm/Compose/offline-kit assets; backup/restore runbook review pending | Findings Ledger Guild / `src/Findings/StellaOps.Findings.Ledger` | Provide Helm/Compose manifests, backup/restore guidance, optional Merkle anchor externalization, and offline kit instructions (dev/staging artifacts). | | 4 | LEDGER-34-101 | DONE (2025-11-22) | PREP-LEDGER-34-101-ORCHESTRATOR-LEDGER-EXPORT | Findings Ledger Guild / `src/Findings/StellaOps.Findings.Ledger` | Link orchestrator run ledger exports into Findings Ledger provenance chain, index by artifact hash, and expose audit queries. | | 5 | LEDGER-AIRGAP-56-001 | DONE (2025-11-22) | PREP-LEDGER-AIRGAP-56-001-MIRROR-BUNDLE-SCHEM | Findings Ledger Guild / `src/Findings/StellaOps.Findings.Ledger` | Record bundle provenance (`bundle_id`, `merkle_root`, `time_anchor`) on ledger events for advisories/VEX/policies imported via Mirror Bundles. | | 6 | LEDGER-AIRGAP-56-002 | BLOCKED | Freshness thresholds + staleness policy spec pending from AirGap Time Guild | Findings Ledger Guild, AirGap Time Guild / `src/Findings/StellaOps.Findings.Ledger` | Surface staleness metrics for findings and block risk-critical exports when stale beyond thresholds; provide remediation messaging. | @@ -61,6 +61,7 @@ | 2025-11-22 | LEDGER-29-009 remains BLOCKED: DevOps/Offline kit overlays live outside module working dir; awaiting approved path for Helm/Compose assets and backup runbooks. | Findings Ledger Guild | | 2025-11-22 | Marked AIRGAP-56-002 BLOCKED pending freshness threshold spec; downstream AIRGAP-57/58 remain blocked accordingly. | Findings Ledger Guild | | 2025-11-22 | Added backup/restore and restore-replay guidance to `docs/modules/findings-ledger/deployment.md`; noted placeholder until DevOps assigns manifest paths. | Findings Ledger Guild | +| 2025-11-23 | Routed deployment assets to DEPLOY-LEDGER-29-009 (SPRINT_501_ops_deployment_i); LEDGER-29-009-DEV remains blocked until ops task delivers target paths. | Project Mgmt | | 2025-11-22 | Switched LEDGER-29-008 to DOING; created `src/Findings/StellaOps.Findings.Ledger/TASKS.md` mirror for status tracking. | Findings Ledger Guild | | 2025-11-19 | Assigned PREP owners/dates; see Delivery Tracker. | Planning | | 2025-11-19 | Marked PREP tasks P1–P3 BLOCKED: observability schema, orchestrator ledger export contract, and mirror bundle schema are still missing, keeping LEDGER-29-008/34-101/AIRGAP-56-* blocked. | Project Mgmt | diff --git a/docs/implplan/SPRINT_0125_0001_0001_policy_reasoning.md b/docs/implplan/SPRINT_0125_0001_0001_policy_reasoning.md index cce0782ec..a6939a066 100644 --- a/docs/implplan/SPRINT_0125_0001_0001_policy_reasoning.md +++ b/docs/implplan/SPRINT_0125_0001_0001_policy_reasoning.md @@ -33,7 +33,7 @@ | P13 | PREP-POLICY-ENGINE-40-001-DEPENDS-ON-38-201 | DONE (2025-11-22) | Due 2025-11-22 · Accountable: Policy · Concelier Guild / `src/Policy/StellaOps.Policy.Engine` | Policy · Concelier Guild / `src/Policy/StellaOps.Policy.Engine` | Depends on 38-201.

Document artefact/deliverable for POLICY-ENGINE-40-001 and publish location so downstream tasks can proceed. | | P14 | PREP-POLICY-ENGINE-40-002-DEPENDS-ON-40-001 | DONE (2025-11-22) | Due 2025-11-22 · Accountable: Policy · Excititor Guild / `src/Policy/StellaOps.Policy.Engine` | Policy · Excititor Guild / `src/Policy/StellaOps.Policy.Engine` | Depends on 40-001.

Document artefact/deliverable for POLICY-ENGINE-40-002 and publish location so downstream tasks can proceed. | | 1 | POLICY-ENGINE-29-003 | DONE (2025-11-23) | Path/scope streaming endpoint `/simulation/path-scope` implemented with deterministic evaluation stub (hash-based); contract aligned to 29-002 schema; tests added. | Policy · SBOM Service Guild / `src/Policy/StellaOps.Policy.Engine` | Path/scope aware evaluation. | -| 2 | POLICY-ENGINE-29-004 | TODO | PREP-POLICY-ENGINE-29-004-DEPENDS-ON-29-003 | Policy · Observability Guild / `src/Policy/StellaOps.Policy.Engine` | Metrics/logging for path-aware eval. | +| 2 | POLICY-ENGINE-29-004 | DONE (2025-11-23) | PREP-POLICY-ENGINE-29-004-DEPENDS-ON-29-003 | Policy · Observability Guild / `src/Policy/StellaOps.Policy.Engine` | Metrics/logging for path-aware eval. | | 3 | POLICY-ENGINE-30-001 | TODO | PREP-POLICY-ENGINE-30-001-NEEDS-29-004-OUTPUT | Policy · Cartographer Guild / `src/Policy/StellaOps.Policy.Engine` | Overlay projection contract. | | 4 | POLICY-ENGINE-30-002 | TODO | PREP-POLICY-ENGINE-30-002-DEPENDS-ON-30-001 | Policy · Cartographer Guild / `src/Policy/StellaOps.Policy.Engine` | Simulation bridge. | | 5 | POLICY-ENGINE-30-003 | TODO | PREP-POLICY-ENGINE-30-003-DEPENDS-ON-30-002 | Policy · Scheduler Guild / `src/Policy/StellaOps.Policy.Engine` | Change events. | @@ -58,6 +58,7 @@ | 2025-11-23 | POLICY-ENGINE-29-002 streaming simulation contract finalized at `docs/modules/policy/contracts/29-002-streaming-simulation.md`; shifted POLICY-ENGINE-29-003..40-002 from BLOCKED to TODO. | Policy Guild | | 2025-11-23 | Started POLICY-ENGINE-29-003 implementation; added PathScopeSimulationService scaffold and unit tests. | Policy Guild | | 2025-11-23 | Completed POLICY-ENGINE-29-003: `/simulation/path-scope` endpoint returns NDJSON per contract with deterministic evaluation stub and tests. | Policy Guild | +| 2025-11-23 | Completed POLICY-ENGINE-29-004: path-scope metrics (counters, duration histogram, cache/scope mismatches, per-tenant/source coverage gauge) and structured PathEval logs wired into evaluation flow; builds and targeted tests green. | Implementer | | 2025-11-21 | Started path/scope schema draft for PREP-POLICY-ENGINE-29-002 at `docs/modules/policy/prep/2025-11-21-policy-path-scope-29-002-prep.md`; waiting on SBOM Service coordinate mapping rules. | Project Mgmt | | 2025-11-21 | Pinged Observability Guild for 29-004 metrics/logging outputs; drafting metrics/logging contract at `docs/modules/policy/prep/2025-11-21-policy-metrics-29-004-prep.md` while awaiting path/scope payloads from 29-003. | Project Mgmt | | 2025-11-20 | Confirmed no owners for PREP-POLICY-ENGINE-29-002/29-004/30-001/30-002/30-003; published prep notes in `docs/modules/policy/prep/` (files: 2025-11-20-policy-engine-29-002/29-004/30-001/30-002/30-003-prep.md); set P0–P4 DONE. | Implementer | @@ -72,6 +73,7 @@ ## Decisions & Risks - Downstream implementations must conform to `docs/modules/policy/contracts/29-002-streaming-simulation.md`; any schemaVersion change must be logged here and in affected sprints. +- Path-scope metrics/logs implemented; future overlays should reuse the same metric names/tags and log fields to avoid cardinality drift. ## Next Checkpoints - Kick off POLICY-ENGINE-29-003 implementation using frozen path/scope schema and metrics contracts (week of 2025-11-21). diff --git a/docs/implplan/SPRINT_0131_0001_0001_scanner_surface.md b/docs/implplan/SPRINT_0131_0001_0001_scanner_surface.md index 9ef952e5f..6c6e816dd 100644 --- a/docs/implplan/SPRINT_0131_0001_0001_scanner_surface.md +++ b/docs/implplan/SPRINT_0131_0001_0001_scanner_surface.md @@ -27,15 +27,15 @@ | 1 | SCANNER-ANALYZERS-DENO-26-009 | DOING (2025-11-22) | Implement runtime trace shim execution + NDJSON/AnalysisStore alignment; pending CI runner for end-to-end trace. | Deno Analyzer Guild · Signals Guild | Optional runtime evidence hooks capturing module loads and permissions with path hashing during harnessed execution. | | 2 | SCANNER-ANALYZERS-DENO-26-010 | TODO | After 26-009, wire CLI (`stella deno trace`) + Worker/Offline Kit using runtime NDJSON contract. | Deno Analyzer Guild · DevOps Guild | Package analyzer plug-in and surface CLI/worker commands with offline documentation. | | 3 | SCANNER-ANALYZERS-DENO-26-011 | TODO | Implement policy signal emitter using runtime metadata once trace shim lands. | Deno Analyzer Guild | Policy signal emitter for capabilities (net/fs/env/ffi/process/crypto), remote origins, npm usage, wasm modules, and dynamic-import warnings. | -| 4 | SCANNER-ANALYZERS-JAVA-21-005 | BLOCKED (2025-11-17) | PREP-SCANNER-ANALYZERS-JAVA-21-005-TESTS-BLOC | Java Analyzer Guild | Framework config extraction: Spring Boot imports, spring.factories, application properties/yaml, Jakarta web.xml/fragments, JAX-RS/JPA/CDI/JAXB configs, logging files, Graal native-image configs. | +| 4 | SCANNER-ANALYZERS-JAVA-21-005 | BLOCKED (2025-11-17) | PREP-SCANNER-ANALYZERS-JAVA-21-005-TESTS-BLOC; DEVOPS-SCANNER-CI-11-001 (SPRINT_503_ops_devops_i) for CI runner/binlogs. | Java Analyzer Guild | Framework config extraction: Spring Boot imports, spring.factories, application properties/yaml, Jakarta web.xml/fragments, JAX-RS/JPA/CDI/JAXB configs, logging files, Graal native-image configs. | | 5 | SCANNER-ANALYZERS-JAVA-21-006 | TODO | Needs outputs from 21-005. | Java Analyzer Guild | JNI/native hint scanner detecting native methods, System.load/Library literals, bundled native libs, Graal JNI configs; emit `jni-load` edges. | | 6 | SCANNER-ANALYZERS-JAVA-21-007 | TODO | After 21-006; align manifest parsing with resolver. | Java Analyzer Guild | Signature and manifest metadata collector capturing JAR signature structure, signers, and manifest loader attributes (Main-Class, Agent-Class, Start-Class, Class-Path). | -| 7 | SCANNER-ANALYZERS-JAVA-21-008 | BLOCKED (2025-10-27) | PREP-SCANNER-ANALYZERS-JAVA-21-008-WAITING-ON | Java Analyzer Guild | Implement resolver + AOC writer emitting entrypoints, components, and edges (jpms, cp, spi, reflect, jni) with reason codes and confidence. | +| 7 | SCANNER-ANALYZERS-JAVA-21-008 | BLOCKED (2025-10-27) | PREP-SCANNER-ANALYZERS-JAVA-21-008-WAITING-ON; DEVOPS-SCANNER-CI-11-001 for CI runner/restore logs. | Java Analyzer Guild | Implement resolver + AOC writer emitting entrypoints, components, and edges (jpms, cp, spi, reflect, jni) with reason codes and confidence. | | 8 | SCANNER-ANALYZERS-JAVA-21-009 | TODO | Unblock when 21-008 lands; prepare fixtures in parallel where safe. | Java Analyzer Guild · QA Guild | Comprehensive fixtures (modular app, boot fat jar, war, ear, MR-jar, jlink image, JNI, reflection heavy, signed jar, microprofile) with golden outputs and perf benchmarks. | | 9 | SCANNER-ANALYZERS-JAVA-21-010 | TODO | After 21-009; requires runtime capture design. | Java Analyzer Guild · Signals Guild | Optional runtime ingestion via Java agent + JFR reader capturing class load, ServiceLoader, System.load events with path scrubbing; append-only runtime edges (`runtime-class`/`runtime-spi`/`runtime-load`). | | 10 | SCANNER-ANALYZERS-JAVA-21-011 | TODO | Depends on 21-010; finalize DI/manifest registration and docs. | Java Analyzer Guild | Package analyzer as restart-time plug-in, update Offline Kit docs, add CLI/worker hooks for Java inspection commands. | | 10b | DEVOPS-SCANNER-JAVA-21-011-REL | BLOCKED (DevOps release-only) | Depends on 10 dev; add CI/release packaging/signing for Java analyzer plug-in + Offline Kit docs. | DevOps Guild | Package/sign Java analyzer plug-in, publish to Offline Kit/CLI release pipelines. | -| 11 | SCANNER-ANALYZERS-LANG-11-001 | BLOCKED (2025-11-17) | PREP-SCANNER-ANALYZERS-LANG-11-001-DOTNET-TES | StellaOps.Scanner EPDR Guild · Language Analyzer Guild | Entrypoint resolver mapping project/publish artifacts to entrypoint identities (assembly name, MVID, TFM, RID) and environment profiles; output normalized `entrypoints[]` with deterministic IDs. | +| 11 | SCANNER-ANALYZERS-LANG-11-001 | BLOCKED (2025-11-17) | PREP-SCANNER-ANALYZERS-LANG-11-001-DOTNET-TES; DEVOPS-SCANNER-CI-11-001 for clean runner + binlogs/TRX. | StellaOps.Scanner EPDR Guild · Language Analyzer Guild | Entrypoint resolver mapping project/publish artifacts to entrypoint identities (assembly name, MVID, TFM, RID) and environment profiles; output normalized `entrypoints[]` with deterministic IDs. | ## Execution Log | Date (UTC) | Update | Owner | @@ -54,6 +54,7 @@ | 2025-11-17 | Reviewed Deno analyzer scope; runtime evidence hook contract and policy-signal keys not defined in docs or code. Marked DENO-26-009/010/011 as BLOCKED pending approved trace/signal schema shared with Surface/Signals. | Implementer | | 2025-11-17 | SCANNER-ANALYZERS-JAVA-21-005: Added JNI/native hint scanning (native libs, Graal jni-config, System.load/Library strings) with component metadata + evidence; targeted tests added. Test run aborted ~80s in due to concurrent repo-wide builds; rerun on clean runner. | Java Analyzer Guild | | 2025-11-17 | Authored `docs/modules/scanner/design/deno-runtime-signals.md` defining NDJSON runtime trace + policy signal keys; unblocked DENO-26-009/010/011 back to TODO. | Implementer | +| 2025-11-23 | Pointed Java/Lang analyzer blocks to DEVOPS-SCANNER-CI-11-001 (SPRINT_503_ops_devops_i) to obtain CI runner/binlogs for restore/test hangs. | Project Mgmt | | 2025-11-17 | Implemented Deno runtime NDJSON serializer + metadata (module/permission counts, remote origins, npm/wasm/dynamic import counts) with deterministic ordering and hash; added regression tests for serializer, path hashing, recorder ordering, and policy signal emission. Loader/require shim still pending. | Implementer | | 2025-11-17 | Deno runtime tests passing: `dotnet test src/Scanner/__Tests/StellaOps.Scanner.Analyzers.Lang.Deno.Tests/StellaOps.Scanner.Analyzers.Lang.Deno.Tests.csproj --no-restore`. | Implementer | | 2025-11-17 | DenoLanguageAnalyzer now ingests `deno-runtime.ndjson` if present, computes metadata/hash, stores runtime payload in AnalysisStore, and emits policy signals; added runtime probe parser + tests. Loader/require shim that generates the trace remains to be built. | Implementer | diff --git a/docs/implplan/SPRINT_0142_0001_0001_sbomservice.md b/docs/implplan/SPRINT_0142_0001_0001_sbomservice.md index c278cb464..d8da7b1d2 100644 --- a/docs/implplan/SPRINT_0142_0001_0001_sbomservice.md +++ b/docs/implplan/SPRINT_0142_0001_0001_sbomservice.md @@ -24,14 +24,14 @@ | P3 | PREP-BUILD-INFRA-SBOM-SERVICE-GUILD-BLOCKED-M | DONE (2025-11-22) | Due 2025-11-22 · Accountable: Planning | Planning | BLOCKED (multiple restore attempts still hang/fail; need vetted feed/cache).

Document artefact/deliverable for Build/Infra · SBOM Service Guild and publish location so downstream tasks can proceed. Prep artefact: `docs/modules/sbomservice/prep/2025-11-20-build-infra-prep.md`. | | 1 | SBOM-AIAI-31-001 | DONE | Implemented `/sbom/paths` with env/blast-radius/runtime flags + cursor paging and `/sbom/versions` timeline; in-memory deterministic seed until storage wired. | SBOM Service Guild (src/SbomService/StellaOps.SbomService) | Provide path and version timeline endpoints optimised for Advisory AI. | | 2 | SBOM-AIAI-31-002 | DONE | Metrics + cache-hit tagging implemented; Grafana starter dashboard added; build/test completed locally. | SBOM Service Guild; Observability Guild | Instrument metrics for path/timeline queries and surface dashboards. | -| 3 | SBOM-CONSOLE-23-001 | BLOCKED | PREP-SBOM-CONSOLE-23-001-BUILD-TEST-FAILING-D | SBOM Service Guild; Cartographer Guild | Provide Console-focused SBOM catalog API. | -| 4 | SBOM-CONSOLE-23-002 | BLOCKED | Stub implemented; blocked on storage wiring and console schema approval. | SBOM Service Guild | Deliver component lookup endpoints for search and overlays. | +| 3 | SBOM-CONSOLE-23-001 | BLOCKED | DEVOPS-SBOM-23-001 (SPRINT_503_ops_devops_i) — needs vetted offline feed + CI proof to run restore/tests. | SBOM Service Guild; Cartographer Guild | Provide Console-focused SBOM catalog API. | +| 4 | SBOM-CONSOLE-23-002 | BLOCKED | Stub implemented; awaiting DEVOPS-SBOM-23-001 feed + console schema approval before storage wiring. | SBOM Service Guild | Deliver component lookup endpoints for search and overlays. | | 5 | SBOM-ORCH-32-001 | TODO | Register SBOM ingest/index sources; embed worker SDK; emit artifact hashes and job metadata. | SBOM Service Guild | Register SBOM ingest/index sources with orchestrator. | | 6 | SBOM-ORCH-33-001 | TODO | Depends on SBOM-ORCH-32-001; report backpressure metrics, honor pause/throttle signals, classify sbom job errors. | SBOM Service Guild | Report backpressure metrics and handle orchestrator control signals. | | 7 | SBOM-ORCH-34-001 | TODO | Depends on SBOM-ORCH-33-001; implement orchestrator backfill and watermark reconciliation for idempotent artifact reuse. | SBOM Service Guild | Implement orchestrator backfill + watermark reconciliation. | | 8 | SBOM-SERVICE-21-001 | DONE (2025-11-23) | WAF aligned; projection tests pass with fixture-backed in-memory repo; duplicate test PackageReferences removed. | SBOM Service Guild; Cartographer Guild | Projection read API (`/sboms/{snapshotId}/projection`) validated with hash output; ready to proceed to storage-backed wiring/events. | -| 9 | SBOM-SERVICE-21-002 | TODO | Depends on SBOM-SERVICE-21-001; emit `sbom.version.created` change events and add replay/backfill tooling. | SBOM Service Guild; Scheduler Guild | Emit change events carrying digest/version metadata for Graph Indexer builds. | -| 10 | SBOM-SERVICE-21-003 | TODO | Depends on SBOM-SERVICE-21-002; entrypoint/service node management API feeding Cartographer path relevance with deterministic defaults. | SBOM Service Guild | Provide entrypoint/service node management API. | +| 9 | SBOM-SERVICE-21-002 | DONE (2025-11-23) | Emits `sbom.version.created` change events via in-memory publisher; internal `/internal/sbom/events` + backfill endpoint wired; component lookup cursor fixed. | SBOM Service Guild; Scheduler Guild | Emit change events carrying digest/version metadata for Graph Indexer builds. | +| 10 | SBOM-SERVICE-21-003 | DONE (2025-11-23) | Depends on SBOM-SERVICE-21-002; entrypoint/service node API delivered (`GET/POST /entrypoints` with tenant guard, deterministic ordering, in-memory seed). | SBOM Service Guild | Provide entrypoint/service node management API. | | 11 | SBOM-SERVICE-21-004 | TODO | Depends on SBOM-SERVICE-21-003; wire metrics (`sbom_projection_seconds`, `sbom_projection_size`), traces, tenant-annotated logs; set backlog alerts. | SBOM Service Guild; Observability Guild | Wire observability for SBOM projections. | | 12 | SBOM-SERVICE-23-001 | TODO | Depends on SBOM-SERVICE-21-004; extend projections with asset metadata (criticality, owner, environment, exposure flags); update schema docs. | SBOM Service Guild; Policy Guild | Extend projections to include asset metadata. | | 13 | SBOM-SERVICE-23-002 | TODO | Depends on SBOM-SERVICE-23-001; emit `sbom.asset.updated` events with idempotent payloads; document envelopes. | SBOM Service Guild; Platform Events Guild | Emit asset metadata change events. | @@ -51,6 +51,9 @@ ## Execution Log | Date (UTC) | Update | Owner | | --- | --- | --- | +| 2025-11-23 | Implemented `sbom.version.created` events (in-memory publisher + `/internal/sbom/events` + backfill); fixed component lookup pagination cursor; SbomService tests now passing (SbomEvent/Sbom/Projection suites). SBOM-SERVICE-21-002 marked DONE. | SBOM Service | +| 2025-11-23 | Delivered entrypoint/service node API (`GET/POST /entrypoints` with tenant guard, deterministic ordering, in-memory seed). SBOM-SERVICE-21-003 marked DONE. | SBOM Service | +| 2025-11-23 | Split build/feed blocker into DEVOPS-SBOM-23-001 (SPRINT_503_ops_devops_i); SBOM-CONSOLE-23-001/002 remain BLOCKED pending ops feed + CI proof. | Project Mgmt | | 2025-11-23 | ProjectionEndpointTests now pass (400/200 responses); WAF configured with fixture path + in-memory component repo; duplicate test PackageReferences removed. SBOM-SERVICE-21-001 marked DONE. | SBOM Service | | 2025-11-23 | Added Mongo fallback to in-memory component lookup to keep tests/offline runs alive; WebApplicationFactory still returns HTTP 500 for projection endpoints (manual curl against `dotnet run` returns 400/200). Investigation pending; SBOM-SERVICE-21-001 remains DOING. | SBOM Service | | 2025-11-23 | Fixed test package references (`FluentAssertions`, `Microsoft.AspNetCore.Mvc.Testing`, xUnit) and attempted `dotnet test --filter ProjectionEndpointTests`; build runs but projection endpoint responses returned HTTP 500 instead of expected 400/200, leaving SBOM-SERVICE-21-001 in DOING pending investigation. | SBOM Service | @@ -96,6 +99,8 @@ ## Decisions & Risks - LNM v1 fixtures staged (2025-11-22) and approved; hash recorded in `docs/modules/sbomservice/fixtures/lnm-v1/SHA256SUMS`. SBOM-SERVICE-21-001 DONE (2025-11-23); 21-002..004 remain TODO and now unblocked. - Projection endpoint validated (400 without tenant, 200 with fixture data) via WebApplicationFactory; WAF configured with fixture path + in-memory component repo fallback. +- `sbom.version.created` now emitted via in-memory publisher with `/internal/sbom/events` + backfill endpoint; production outbox/queue wiring still required before release. +- Component lookup pagination now returns deterministic `nextCursor` for seeded data (fixed null cursor bug). - Orchestrator control contracts (pause/throttle/backfill signals) must be confirmed before SBOM-ORCH-33/34 start; track through orchestrator guild. - Keep `docs/modules/sbomservice/architecture.md` aligned with schema/event decisions made during implementation. - Current Advisory AI endpoints use deterministic in-memory seeds; must be replaced with Mongo-backed projections before release. diff --git a/docs/implplan/SPRINT_0400_0001_0001_reachability_runtime_static_union.md b/docs/implplan/SPRINT_0400_0001_0001_reachability_runtime_static_union.md index 3a581b09a..29e4a6683 100644 --- a/docs/implplan/SPRINT_0400_0001_0001_reachability_runtime_static_union.md +++ b/docs/implplan/SPRINT_0400_0001_0001_reachability_runtime_static_union.md @@ -21,10 +21,10 @@ | # | Task ID | Status | Key dependency / next step | Owners | Task Definition | | --- | --- | --- | --- | --- | --- | | 1 | ZASTAVA-REACH-201-001 | TODO | Need runtime symbol sampling design; align with GAP-ZAS-002 | Zastava Observer Guild | Implement runtime symbol sampling in `StellaOps.Zastava.Observer` (EntryTrace-aware shell AST + build-id capture) and stream ND-JSON batches to Signals `/runtime-facts`, including CAS pointers for traces. Update runbook + config references. | -| 2 | SCAN-REACH-201-002 | BLOCKED | Await runtime/static union schema (SymbolID + CAS layout) | Scanner Worker Guild | Ship language-aware static lifters (JVM, .NET/Roslyn+IL, Go SSA, Node/Deno TS AST, Rust MIR, Swift SIL, shell/binary analyzers) in Scanner Worker; emit canonical SymbolIDs, CAS-stored graphs, and attach reachability tags to SBOM components. | -| 3 | SIGNALS-REACH-201-003 | BLOCKED | Runtime/static union schema not published | Signals Guild | Extend Signals ingestion to accept the new multi-language graphs + runtime facts, normalize into `reachability_graphs` CAS layout, and expose retrieval APIs for Policy/CLI. | -| 4 | SIGNALS-REACH-201-004 | BLOCKED | Depends on 201-003 schema | Signals Guild · Policy Guild | Build the reachability scoring engine (state/score/confidence), wire Redis caches + `signals.fact.updated` events, and integrate reachability weights defined in `docs/11_DATA_SCHEMAS.md`. | -| 5 | REPLAY-REACH-201-005 | BLOCKED | Needs finalized graph payload shape | BE-Base Platform Guild | Update `StellaOps.Replay.Core` manifest schema + bundle writer so replay packs capture reachability graphs, runtime traces, analyzer versions, and evidence hashes; document new CAS namespace. | +| 2 | SCAN-REACH-201-002 | TODO | Schema published: `docs/reachability/runtime-static-union-schema.md` (v0.1). Implement emitters against CAS layout. | Scanner Worker Guild | Ship language-aware static lifters (JVM, .NET/Roslyn+IL, Go SSA, Node/Deno TS AST, Rust MIR, Swift SIL, shell/binary analyzers) in Scanner Worker; emit canonical SymbolIDs, CAS-stored graphs, and attach reachability tags to SBOM components. | +| 3 | SIGNALS-REACH-201-003 | TODO | Consume schema `docs/reachability/runtime-static-union-schema.md`; wire ingestion + CAS storage. | Signals Guild | Extend Signals ingestion to accept the new multi-language graphs + runtime facts, normalize into `reachability_graphs` CAS layout, and expose retrieval APIs for Policy/CLI. | +| 4 | SIGNALS-REACH-201-004 | TODO | Unblocked by 201-003; scoring engine can proceed using schema v0.1. | Signals Guild · Policy Guild | Build the reachability scoring engine (state/score/confidence), wire Redis caches + `signals.fact.updated` events, and integrate reachability weights defined in `docs/11_DATA_SCHEMAS.md`. | +| 5 | REPLAY-REACH-201-005 | TODO | Schema v0.1 available; update replay manifest/bundle to include CAS namespace + hashes per spec. | BE-Base Platform Guild | Update `StellaOps.Replay.Core` manifest schema + bundle writer so replay packs capture reachability graphs, runtime traces, analyzer versions, and evidence hashes; document new CAS namespace. | | 6 | DOCS-REACH-201-006 | TODO | Requires outputs from 1–5 | Docs Guild | Author the reachability doc set (`docs/signals/reachability.md`, `callgraph-formats.md`, `runtime-facts.md`, CLI/UI appendices) plus update Zastava + Replay guides with the new evidence and operator workflows. | | 7 | QA-REACH-201-007 | TODO | Move fixtures + create evaluator harness | QA Guild | Integrate `reachbench-2025-expanded` fixture pack under `tests/reachability/fixtures/`, add evaluator harness tests that validate reachable vs unreachable cases, and wire CI guidance for deterministic runs. | | 8 | GAP-SCAN-001 | TODO | Align with task 2; binary symbolizers | Scanner Worker Guild | Implement binary/language symbolizers that emit `richgraph-v1` payloads with canonical SymbolIDs and `code_id` anchors, persist graphs to CAS via `StellaOps.Scanner.Reachability`, and refresh analyzer docs/fixtures. | @@ -36,11 +36,11 @@ | Date (UTC) | Update | Owner | | --- | --- | --- | | 2025-11-18 | Normalised sprint to standard template; renamed from SPRINT_400_runtime_facts_static_callgraph_union.md. | Docs | -| 2025-11-19 | Marked tasks 201-002..201-005 BLOCKED pending runtime/static union schema (SymbolID+CAS layout); no implementation until schema is published. | Implementer | +| 2025-11-23 | Published runtime/static union schema v0.1 at `docs/reachability/runtime-static-union-schema.md`; moved 201-002..201-005 to TODO. | Project Mgmt | | 2025-11-20 | Added tasks 201-008 (Unknowns Registry) and 201-009 (purl + symbol-digest edge merge); awaiting schema freeze. | Planning | ## Decisions & Risks -- Runtime/static schema alignment pending (SymbolID, CAS layout, overlay tags); blocks ingestion and scoring finalization. +- Schema v0.1 published at `docs/reachability/runtime-static-union-schema.md` (2025-11-23); treat as add-only. Breaking changes require version bump and mirrored updates in Signals/Replay. - reachbench fixtures not yet relocated into tests tree; QA task 201-007 must complete before CI enablement. - Offline posture: ensure reachability pipelines avoid external downloads; rely on sealed/mock bundles. - Unknowns Registry schema and API must align with Signals scoring before 201-008 can start; derive `unknowns_pressure` math from policy team. diff --git a/docs/implplan/SPRINT_501_ops_deployment_i.md b/docs/implplan/SPRINT_501_ops_deployment_i.md index 50ad766da..2935de8a8 100644 --- a/docs/implplan/SPRINT_501_ops_deployment_i.md +++ b/docs/implplan/SPRINT_501_ops_deployment_i.md @@ -4,21 +4,50 @@ Active items only. Completed/historic work now resides in docs/implplan/archived [Ops & Offline] 190.A) Ops Deployment.I Depends on: Sprint 100.A - Attestor, Sprint 110.A - AdvisoryAI, Sprint 120.A - AirGap, Sprint 130.A - Scanner, Sprint 140.A - Graph, Sprint 150.A - Orchestrator, Sprint 160.A - EvidenceLocker, Sprint 170.A - Notifier, Sprint 180.A - Cli -Summary: Ops & Offline focus on Ops Deployment (phase I). -Task ID | State | Task description | Owners (Source) ---- | --- | --- | --- -COMPOSE-44-001 | TODO | Author `docker-compose.yml`, `.env.example`, and `quickstart.sh` with all core services + dependencies (postgres, redis, object-store, queue, otel). | Deployment Guild, DevEx Guild (ops/deployment) -COMPOSE-44-002 | TODO | Implement `backup.sh` and `reset.sh` scripts with safety prompts and documentation. Dependencies: COMPOSE-44-001. | Deployment Guild (ops/deployment) -COMPOSE-44-003 | TODO | Package seed data container and onboarding wizard toggle (`QUICKSTART_MODE`), ensuring default creds randomized on first run. Dependencies: COMPOSE-44-002. | Deployment Guild, Docs Guild (ops/deployment) -DEPLOY-AIAI-31-001 | TODO | Provide Helm/Compose manifests, GPU toggle, scaling/runbook, and offline kit instructions for Advisory AI service + inference container. | Deployment Guild, Advisory AI Guild (ops/deployment) -DEPLOY-AIRGAP-46-001 | TODO | Provide instructions and scripts (`load.sh`) for importing air-gap bundle into private registry; update Offline Kit guide. | Deployment Guild, Offline Kit Guild (ops/deployment) -DEPLOY-CLI-41-001 | TODO | Package CLI release artifacts (tarballs per OS/arch, checksums, signatures, completions, container image) and publish distribution docs. | Deployment Guild, DevEx/CLI Guild (ops/deployment) -DEPLOY-COMPOSE-44-001 | TODO | Finalize Quickstart scripts (`quickstart.sh`, `backup.sh`, `reset.sh`), seed data container, and publish README with imposed rule reminder. | Deployment Guild (ops/deployment) -DEPLOY-EXPORT-35-001 | BLOCKED (2025-10-29) | Package exporter service/worker Helm overlays (download-only), document rollout/rollback, and integrate signing KMS secrets. | Deployment Guild, Exporter Service Guild (ops/deployment) -DEPLOY-EXPORT-36-001 | TODO | Document OCI/object storage distribution workflows, registry credential automation, and monitoring hooks for exports. Dependencies: DEPLOY-EXPORT-35-001. | Deployment Guild, Exporter Service Guild (ops/deployment) -DEPLOY-HELM-45-001 | TODO | Publish Helm install guide and sample values for prod/airgap; integrate with docs site build. | Deployment Guild (ops/deployment) -DEPLOY-NOTIFY-38-001 | BLOCKED (2025-10-29) | Package notifier API/worker Helm overlays (email/chat/webhook), secrets templates, rollout guide. | Deployment Guild, DevOps Guild (ops/deployment) -DEPLOY-ORCH-34-001 | TODO | Provide orchestrator Helm/Compose manifests, scaling defaults, secret templates, offline kit instructions, and GA rollout/rollback playbook. | Deployment Guild, Orchestrator Service Guild (ops/deployment) -DEPLOY-PACKS-42-001 | TODO | Provide deployment manifests for packs-registry and task-runner services, including Helm/Compose overlays, scaling defaults, and secret templates. | Deployment Guild, Packs Registry Guild (ops/deployment) -DEPLOY-PACKS-43-001 | TODO | Ship remote Task Runner worker profiles, object storage bootstrap, approval workflow integration, and Offline Kit packaging instructions. Dependencies: DEPLOY-PACKS-42-001. | Deployment Guild, Task Runner Guild (ops/deployment) -DEPLOY-POLICY-27-001 | TODO | Produce Helm/Compose overlays for Policy Registry + simulation workers, including Mongo migrations, object storage buckets, signing key secrets, and tenancy defaults. | Deployment Guild, Policy Registry Guild (ops/deployment) \ No newline at end of file + +## Topic & Scope +- Ship deployable artefacts (Helm/Compose/offline kits) across modules without leaving deployment work inside dev sprints. +- Provide signed mirror/export bundles and backup/restore guidance for regulated environments. + +## Dependencies & Concurrency +- Upstream module artefacts must exist before packaging; see task-level dependencies (e.g., MIRROR-KEY-56-002-CI, LEDGER-29-009-DEV). +- Can run in parallel to module development; outputs live under `ops/deployment`. + +## Documentation Prerequisites +- docs/modules/devops/architecture.md +- docs/modules/ci/architecture.md +- docs/airgap/** (for mirror/import tasks) + +## Delivery Tracker +| Task ID | State | Task description | Owners (Source) | +| --- | --- | --- | --- | +| COMPOSE-44-001 | TODO | Author `docker-compose.yml`, `.env.example`, and `quickstart.sh` with all core services + dependencies (postgres, redis, object-store, queue, otel). | Deployment Guild, DevEx Guild (ops/deployment) | +| COMPOSE-44-002 | TODO | Implement `backup.sh` and `reset.sh` scripts with safety prompts and documentation. Dependencies: COMPOSE-44-001. | Deployment Guild (ops/deployment) | +| COMPOSE-44-003 | TODO | Package seed data container and onboarding wizard toggle (`QUICKSTART_MODE`), ensuring default creds randomized on first run. Dependencies: COMPOSE-44-002. | Deployment Guild, Docs Guild (ops/deployment) | +| DEPLOY-AIAI-31-001 | TODO | Provide Helm/Compose manifests, GPU toggle, scaling/runbook, and offline kit instructions for Advisory AI service + inference container. | Deployment Guild, Advisory AI Guild (ops/deployment) | +| DEPLOY-AIRGAP-46-001 | TODO | Provide instructions and scripts (`load.sh`) for importing air-gap bundle into private registry; update Offline Kit guide. | Deployment Guild, Offline Kit Guild (ops/deployment) | +| DEPLOY-CLI-41-001 | TODO | Package CLI release artifacts (tarballs per OS/arch, checksums, signatures, completions, container image) and publish distribution docs. | Deployment Guild, DevEx/CLI Guild (ops/deployment) | +| DEPLOY-COMPOSE-44-001 | TODO | Finalize Quickstart scripts (`quickstart.sh`, `backup.sh`, `reset.sh`), seed data container, and publish README with imposed rule reminder. | Deployment Guild (ops/deployment) | +| DEPLOY-EXPORT-35-001 | BLOCKED (2025-10-29) | Package exporter service/worker Helm overlays (download-only), document rollout/rollback, and integrate signing KMS secrets. | Deployment Guild, Exporter Service Guild (ops/deployment) | +| DEPLOY-EXPORT-36-001 | TODO | Document OCI/object storage distribution workflows, registry credential automation, and monitoring hooks for exports. Dependencies: DEPLOY-EXPORT-35-001. | Deployment Guild, Exporter Service Guild (ops/deployment) | +| DEPLOY-HELM-45-001 | TODO | Publish Helm install guide and sample values for prod/airgap; integrate with docs site build. | Deployment Guild (ops/deployment) | +| DEPLOY-NOTIFY-38-001 | BLOCKED (2025-10-29) | Package notifier API/worker Helm overlays (email/chat/webhook), secrets templates, rollout guide. | Deployment Guild, DevOps Guild (ops/deployment) | +| DEPLOY-ORCH-34-001 | TODO | Provide orchestrator Helm/Compose manifests, scaling defaults, secret templates, offline kit instructions, and GA rollout/rollback playbook. | Deployment Guild, Orchestrator Service Guild (ops/deployment) | +| DEPLOY-PACKS-42-001 | TODO | Provide deployment manifests for packs-registry and task-runner services, including Helm/Compose overlays, scaling defaults, and secret templates. | Deployment Guild, Packs Registry Guild (ops/deployment) | +| DEPLOY-PACKS-43-001 | TODO | Ship remote Task Runner worker profiles, object storage bootstrap, approval workflow integration, and Offline Kit packaging instructions. Dependencies: DEPLOY-PACKS-42-001. | Deployment Guild, Task Runner Guild (ops/deployment) | +| DEPLOY-POLICY-27-001 | TODO | Produce Helm/Compose overlays for Policy Registry + simulation workers, including Mongo migrations, object storage buckets, signing key secrets, and tenancy defaults. | Deployment Guild, Policy Registry Guild (ops/deployment) | +| DEPLOY-MIRROR-23-001 | BLOCKED (2025-11-23) | Publish signed mirror/offline artefacts; needs `MIRROR_SIGN_KEY_B64` wired in CI (from MIRROR-KEY-56-002-CI) and Attestor mirror contract. | Deployment Guild, Security Guild (ops/deployment) | +| DEPLOY-LEDGER-29-009 | BLOCKED (2025-11-23) | Provide Helm/Compose/offline-kit manifests + backup/restore runbook paths for Findings Ledger; waits on DevOps-approved target directories before committing artefacts. | Deployment Guild, Findings Ledger Guild, DevOps Guild (ops/deployment) | + +## Execution Log +| Date (UTC) | Update | Owner | +| --- | --- | --- | +| 2025-11-23 | Added DEPLOY-MIRROR-23-001 and DEPLOY-LEDGER-29-009; normalised sprint with template sections. | Project Mgmt | + +## Decisions & Risks +- Mirror signing secret (`MIRROR_SIGN_KEY_B64`) and Attestor contract are outstanding; DEPLOY-MIRROR-23-001 remains blocked until provided. +- Findings Ledger deployment assets cannot be committed until DevOps assigns target directories to keep module boundaries clean. + +## Next Checkpoints +- 2025-11-25: Review mirror signing secret readiness with Security/DevOps. +- 2025-11-26: Findings Ledger deployment path/backup runbook review with DevOps Guild. diff --git a/docs/implplan/SPRINT_503_ops_devops_i.md b/docs/implplan/SPRINT_503_ops_devops_i.md index f98bf3306..389351b38 100644 --- a/docs/implplan/SPRINT_503_ops_devops_i.md +++ b/docs/implplan/SPRINT_503_ops_devops_i.md @@ -4,29 +4,57 @@ Active items only. Completed/historic work now resides in docs/implplan/archived [Ops & Offline] 190.B) Ops Devops.I Depends on: Sprint 100.A - Attestor, Sprint 110.A - AdvisoryAI, Sprint 120.A - AirGap, Sprint 130.A - Scanner, Sprint 140.A - Graph, Sprint 150.A - Orchestrator, Sprint 160.A - EvidenceLocker, Sprint 170.A - Notifier, Sprint 180.A - Cli -Summary: Ops & Offline focus on Ops Devops (phase I). -Task ID | State | Task description | Owners (Source) ---- | --- | --- | --- -DEVOPS-AIAI-31-001 | TODO | Stand up CI pipelines, inference monitoring, privacy logging review, and perf dashboards for Advisory AI (summaries/conflicts/remediation). | DevOps Guild, Advisory AI Guild (ops/devops) -DEVOPS-AIRGAP-56-001 | TODO | Ship deny-all egress policies for Kubernetes (NetworkPolicy/eBPF) and docker-compose firewall rules; provide verification script for sealed mode. | DevOps Guild (ops/devops) -DEVOPS-AIRGAP-56-002 | TODO | Provide import tooling for bundle staging: checksum validation, offline object-store loader scripts, removable media guidance. Dependencies: DEVOPS-AIRGAP-56-001. | DevOps Guild, AirGap Importer Guild (ops/devops) -DEVOPS-AIRGAP-56-003 | TODO | Build Bootstrap Pack pipeline bundling images/charts, generating checksums, and publishing manifest for offline transfer. Dependencies: DEVOPS-AIRGAP-56-002. | DevOps Guild, Container Distribution Guild (ops/devops) -DEVOPS-AIRGAP-57-001 | TODO | Automate Mirror Bundle creation jobs with dual-control approvals, artifact signing, and checksum publication. Dependencies: DEVOPS-AIRGAP-56-003. | DevOps Guild, Mirror Creator Guild (ops/devops) -DEVOPS-AIRGAP-57-002 | BLOCKED (2025-11-18) | Waiting on upstream DEVOPS-AIRGAP-57-001 (mirror bundle automation) to provide artifacts/endpoints for sealed-mode CI; no sealed fixtures available to exercise tests. | DevOps Guild, Authority Guild (ops/devops) -> 2025-11-07: Harness scaffolded at `ops/devops/sealed-mode-ci/*` (README + runner script); integrate into CI to unblock AUTH-AIRGAP-57-001. -> 2025-11-08: `sealed-mode-compose.yml`, `run-sealed-ci.sh`, and `egress_probe.py` committed plus a `sealed-mode-ci` workflow stage that uploads `artifacts/sealed-mode-ci//authority-sealed-ci.json`; Authority can now read the sealed evidence feed. -> 2025-11-18: DEVOPS-AIRGAP-57-002 set to BLOCKED; mirror bundle automation (57-001) not delivered, so no sealed fixtures/artifacts exist to exercise egress checks. -DEVOPS-AIRGAP-58-001 | TODO | Provide local SMTP/syslog container templates and health checks for sealed environments; integrate into Bootstrap Pack. Dependencies: DEVOPS-AIRGAP-57-002. | DevOps Guild, Notifications Guild (ops/devops) -DEVOPS-AIRGAP-58-002 | TODO | Ship sealed-mode observability stack (Prometheus/Grafana/Tempo/Loki) pre-configured with offline dashboards and no remote exporters. Dependencies: DEVOPS-AIRGAP-58-001. | DevOps Guild, Observability Guild (ops/devops) -DEVOPS-AOC-19-001 | BLOCKED (2025-10-26) | Integrate the AOC Roslyn analyzer and guard tests into CI, failing builds when ingestion projects attempt banned writes. | DevOps Guild, Platform Guild (ops/devops) -DEVOPS-AOC-19-002 | BLOCKED (2025-10-26) | Add pipeline stage executing `stella aoc verify --since` against seeded Mongo snapshots for Concelier + Excititor, publishing violation report artefacts. Dependencies: DEVOPS-AOC-19-001. | DevOps Guild (ops/devops) -DEVOPS-AOC-19-003 | BLOCKED (2025-10-26) | Enforce unit test coverage thresholds for AOC guard suites and ensure coverage exported to dashboards. Dependencies: DEVOPS-AOC-19-002. | DevOps Guild, QA Guild (ops/devops) -DEVOPS-AOC-19-101 | TODO (2025-10-28) | Draft supersedes backfill rollout (freeze window, dry-run steps, rollback) once advisory_raw idempotency index passes staging verification. Dependencies: DEVOPS-AOC-19-003. | DevOps Guild, Concelier Storage Guild (ops/devops) -DEVOPS-ATTEST-73-001 | TODO | Provision CI pipelines for attestor service (lint/test/security scan, seed data) and manage secrets for KMS drivers. | DevOps Guild, Attestor Service Guild (ops/devops) -DEVOPS-ATTEST-73-002 | TODO | Establish secure storage for signing keys (vault integration, rotation schedule) and audit logging. Dependencies: DEVOPS-ATTEST-73-001. | DevOps Guild, KMS Guild (ops/devops) -DEVOPS-ATTEST-74-001 | TODO | Deploy transparency log witness infrastructure and monitoring. Dependencies: DEVOPS-ATTEST-73-002. | DevOps Guild, Transparency Guild (ops/devops) -DEVOPS-GRAPH-INDEX-28-010-REL | TODO | Publish signed Helm/Compose/offline bundles for Graph Indexer; depends on GRAPH-INDEX-28-010 dev artefacts. | DevOps Guild, Graph Indexer Guild (ops/devops) -DEVOPS-LNM-21-101-REL | TODO | Run/apply shard/index migrations (Concelier LNM) in release pipelines; capture artefacts and rollback scripts. | DevOps Guild, Concelier Storage Guild (ops/devops) -DEVOPS-LNM-21-102-REL | TODO | Package/publish LNM backfill/rollback bundles for release/offline kit; depends on 21-102 dev outputs. | DevOps Guild, Concelier Storage Guild (ops/devops) -DEVOPS-LNM-21-103-REL | TODO | Publish/rotate object-store seeds and offline bootstraps with provenance hashes; depends on 21-103 dev outputs. | DevOps Guild, Concelier Storage Guild (ops/devops) -DEVOPS-STORE-AOC-19-005-REL | BLOCKED | Release/offline-kit packaging for Concelier backfill; waiting on dataset hash + dev rehearsal. | DevOps Guild, Concelier Storage Guild (ops/devops) + +## Topic & Scope +- Stand up CI, signing, and offline pipelines that unblock module sprints without embedding DevOps work in dev backlogs. +- Provide sealed/airgap bootstrap artefacts and mirrors required by downstream airgap/attestation tasks. +- Ensure AOC/guard rails are enforced in CI across ingestion-heavy modules. + +## Dependencies & Concurrency +- Upstream artefacts: mirror bundle automation (DEVOPS-AIRGAP-57-001), AOC analyzers, module-specific prep notes referenced per task. +- Runs in parallel with module sprints; deliverables are CI/pipeline assets, not code changes inside module working dirs. + +## Documentation Prerequisites +- docs/modules/devops/architecture.md +- docs/modules/ci/architecture.md +- docs/airgap/** (for sealed-mode tasks) + +## Delivery Tracker +| Task ID | State | Task description | Owners (Source) | +| --- | --- | --- | --- | +| DEVOPS-AIAI-31-001 | TODO | Stand up CI pipelines, inference monitoring, privacy logging review, and perf dashboards for Advisory AI (summaries/conflicts/remediation). | DevOps Guild, Advisory AI Guild (ops/devops) | +| DEVOPS-AIRGAP-56-001 | TODO | Ship deny-all egress policies for Kubernetes (NetworkPolicy/eBPF) and docker-compose firewall rules; provide verification script for sealed mode. | DevOps Guild (ops/devops) | +| DEVOPS-AIRGAP-56-002 | TODO | Provide import tooling for bundle staging: checksum validation, offline object-store loader scripts, removable media guidance. Dependencies: DEVOPS-AIRGAP-56-001. | DevOps Guild, AirGap Importer Guild (ops/devops) | +| DEVOPS-AIRGAP-56-003 | TODO | Build Bootstrap Pack pipeline bundling images/charts, generating checksums, and publishing manifest for offline transfer. Dependencies: DEVOPS-AIRGAP-56-002. | DevOps Guild, Container Distribution Guild (ops/devops) | +| DEVOPS-AIRGAP-57-001 | TODO | Automate Mirror Bundle creation jobs with dual-control approvals, artifact signing, and checksum publication. Dependencies: DEVOPS-AIRGAP-56-003. | DevOps Guild, Mirror Creator Guild (ops/devops) | +| DEVOPS-AIRGAP-57-002 | BLOCKED (2025-11-18) | Waiting on upstream DEVOPS-AIRGAP-57-001 (mirror bundle automation) to provide artifacts/endpoints for sealed-mode CI; no sealed fixtures available to exercise tests. | DevOps Guild, Authority Guild (ops/devops) | +| DEVOPS-AIRGAP-58-001 | TODO | Provide local SMTP/syslog container templates and health checks for sealed environments; integrate into Bootstrap Pack. Dependencies: DEVOPS-AIRGAP-57-002. | DevOps Guild, Notifications Guild (ops/devops) | +| DEVOPS-AIRGAP-58-002 | TODO | Ship sealed-mode observability stack (Prometheus/Grafana/Tempo/Loki) pre-configured with offline dashboards and no remote exporters. Dependencies: DEVOPS-AIRGAP-58-001. | DevOps Guild, Observability Guild (ops/devops) | +| DEVOPS-AOC-19-001 | BLOCKED (2025-10-26) | Integrate the AOC Roslyn analyzer and guard tests into CI, failing builds when ingestion projects attempt banned writes. | DevOps Guild, Platform Guild (ops/devops) | +| DEVOPS-AOC-19-002 | BLOCKED (2025-10-26) | Add pipeline stage executing `stella aoc verify --since` against seeded Mongo snapshots for Concelier + Excititor, publishing violation report artefacts. Dependencies: DEVOPS-AOC-19-001. | DevOps Guild (ops/devops) | +| DEVOPS-AOC-19-003 | BLOCKED (2025-10-26) | Enforce unit test coverage thresholds for AOC guard suites and ensure coverage exported to dashboards. Dependencies: DEVOPS-AOC-19-002. | DevOps Guild, QA Guild (ops/devops) | +| DEVOPS-AOC-19-101 | TODO (2025-10-28) | Draft supersedes backfill rollout (freeze window, dry-run steps, rollback) once advisory_raw idempotency index passes staging verification. Dependencies: DEVOPS-AOC-19-003. | DevOps Guild, Concelier Storage Guild (ops/devops) | +| DEVOPS-ATTEST-73-001 | TODO | Provision CI pipelines for attestor service (lint/test/security scan, seed data) and manage secrets for KMS drivers. | DevOps Guild, Attestor Service Guild (ops/devops) | +| DEVOPS-ATTEST-73-002 | TODO | Establish secure storage for signing keys (vault integration, rotation schedule) and audit logging. Dependencies: DEVOPS-ATTEST-73-001. | DevOps Guild, KMS Guild (ops/devops) | +| DEVOPS-ATTEST-74-001 | TODO | Deploy transparency log witness infrastructure and monitoring. Dependencies: DEVOPS-ATTEST-73-002. | DevOps Guild, Transparency Guild (ops/devops) | +| DEVOPS-GRAPH-INDEX-28-010-REL | TODO | Publish signed Helm/Compose/offline bundles for Graph Indexer; depends on GRAPH-INDEX-28-010 dev artefacts. | DevOps Guild, Graph Indexer Guild (ops/devops) | +| DEVOPS-LNM-21-101-REL | TODO | Run/apply shard/index migrations (Concelier LNM) in release pipelines; capture artefacts and rollback scripts. | DevOps Guild, Concelier Storage Guild (ops/devops) | +| DEVOPS-LNM-21-102-REL | TODO | Package/publish LNM backfill/rollback bundles for release/offline kit; depends on 21-102 dev outputs. | DevOps Guild, Concelier Storage Guild (ops/devops) | +| DEVOPS-LNM-21-103-REL | TODO | Publish/rotate object-store seeds and offline bootstraps with provenance hashes; depends on 21-103 dev outputs. | DevOps Guild, Concelier Storage Guild (ops/devops) | +| DEVOPS-STORE-AOC-19-005-REL | BLOCKED | Release/offline-kit packaging for Concelier backfill; waiting on dataset hash + dev rehearsal. | DevOps Guild, Concelier Storage Guild (ops/devops) | +| DEVOPS-CONCELIER-CI-24-101 | TODO | Provide clean CI runner + warmed NuGet cache + vstest harness for Concelier WebService & Storage; deliver TRX/binlogs and unblock CONCELIER-GRAPH-24-101/28-102 and LNM-21-004..203. | DevOps Guild, Concelier Core Guild (ops/devops) | +| DEVOPS-SCANNER-CI-11-001 | TODO | Supply warmed cache/diag runner for Scanner analyzers (LANG-11-001, JAVA 21-005/008) with binlogs + TRX; unblock restore/test hangs. | DevOps Guild, Scanner EPDR Guild (ops/devops) | +| DEVOPS-SBOM-23-001 | TODO | Publish vetted offline NuGet feed + CI recipe for SbomService; prove with `dotnet test` run and share cache hashes; unblock SBOM-CONSOLE-23-001/002. | DevOps Guild, SBOM Service Guild (ops/devops) | + +## Execution Log +| Date (UTC) | Update | Owner | +| --- | --- | --- | +| 2025-11-23 | Normalised sprint toward template (sections added); added DEVOPS-CONCELIER-CI-24-101, DEVOPS-SCANNER-CI-11-001, DEVOPS-SBOM-23-001 to absorb CI/restore blockers from module sprints. | Project Mgmt | + +## Decisions & Risks +- Mirror bundle automation (DEVOPS-AIRGAP-57-001) and AOC guardrails remain gating risks; several downstream tasks inherit these. +- New CI-runner tasks must produce reproducible binlogs/TRX and cache hashes to keep offline posture intact. + +## Next Checkpoints +- 2025-11-25: CI runner provisioning check for Concelier/Scanner/SBOM cache jobs. +- 2025-11-27: Sealed-mode fixture availability review (DEVOPS-AIRGAP-57-002). diff --git a/docs/modules/sbomservice/architecture.md b/docs/modules/sbomservice/architecture.md index 0f0707254..8302dd595 100644 --- a/docs/modules/sbomservice/architecture.md +++ b/docs/modules/sbomservice/architecture.md @@ -44,6 +44,8 @@ Operational rules: - `GET /console/sboms` — Console catalog with filters (artifact, license, scope, asset tags), cursor pagination, evaluation metadata, immutable JSON projection for drawer views. - `GET /components/lookup?purl=...` — component neighborhood for global search/Graph overlays; returns caches hints + tenant enforcement. - `POST /entrypoints` / `GET /entrypoints` — manage entrypoint/service node overrides feeding Cartographer relevance; deterministic defaults when unset. +- `GET /internal/sbom/events` — internal diagnostics endpoint returning the in-memory event outbox for validation. +- `POST /internal/sbom/events/backfill` — replays existing projections into the event stream; deterministic ordering, clock abstraction for tests. ## 4) Ingestion & orchestrator integration - Ingest sources: Scanner pipeline (preferred) or uploaded SPDX 3.0.1/CycloneDX 1.6 bundles. @@ -54,6 +56,8 @@ Operational rules: - `sbom.version.created` — emitted per new SBOM snapshot; payload: tenant, artifact digest, sbomVersion, projection hash, source bundle hash, import provenance; replay/backfill via outbox with watermark. - `sbom.asset.updated` — emitted when asset metadata changes; idempotent payload keyed by `(tenant, assetId, version)`. - Inventory/resolver feeds — queue/topic delivering `(artifact, purl, version, paths, runtime_flag, scope, nearest_safe_version)` for Vuln Explorer/Findings Ledger. + - Current implementation uses an in-memory event store/publisher (with clock abstraction) plus `/internal/sbom/events` + `/internal/sbom/events/backfill` to validate envelopes until the Mongo-backed outbox is wired. + - Entrypoint/service node overrides are exposed via `/entrypoints` (tenant-scoped) and should be mirrored into Cartographer relevance jobs when the outbox lands. ## 6) Determinism & offline posture - Stable ordering for projections and paths; timestamps in UTC ISO-8601; hash inputs canonicalised. diff --git a/docs/reachability/runtime-static-union-schema.md b/docs/reachability/runtime-static-union-schema.md new file mode 100644 index 000000000..3318e2f3b --- /dev/null +++ b/docs/reachability/runtime-static-union-schema.md @@ -0,0 +1,129 @@ +# Runtime + Static Reachability Union Schema (v0.1, 2025-11-23) + +## Goals +- Provide a single, deterministic graph shape that merges static lifter output and runtime traces across languages. +- Keep SymbolID stable across hosts (path/location independent) so CAS lookups are reproducible and cacheable. +- Make outputs offline-friendly: line-delimited JSON, UTF-8, sorted, with explicit content hashes. + +## File layout (CAS) +- Namespace root: `reachability_graphs//` (analysis_id is caller-supplied UUID or hash). +- Files (all NDJSON, UTF-8, newline terminated, sorted as noted): + - `nodes.ndjson` (sorted by `symbol_id`) + - `edges.ndjson` (sorted by `from` then `to` then `edge_type`) + - `facts_runtime.ndjson` (sorted by `symbol_id`, optional) + - `meta.json` (single JSON object; schema version, produced_by, timestamps, tool versions, hashes) +- Hashing: SHA-256 of each file recorded in `meta.json` under `files[]` with `path`, `sha256`, `records`. +- Compression/packaging is left to the CAS store; files must be valid uncompressed NDJSON first. + +## SymbolID (language-agnostic envelope) +``` +symbol_id = "sym:" + + ":" + +``` +- `lang`: `java|dotnet|go|node|deno|rust|swift|shell|binary` +- `stable-fragment`: SHA-256(base64url-no-pad) of the canonical tuple per language: + - **java**: (`package`, `class`, `method`, `descriptor`) lowercased, descriptor in JVM format. + - **dotnet**: (`assembly_name`, `namespace`, `type`, `member_signature`) using ECMA-335 signature string. + - **node/deno**: (`pkg_name_or_path`, `export_path`, `kind`) where `export_path` is slash-joined ESM/CJS path; `pkg_name_or_path` uses npm name or normalized absolute path with drive stripped. + - **go**: (`module_path`, `package_path`, `receiver`, `func`), with receiver empty for functions. + - **rust**: (`crate`, `module_path`, `item_name`, `mangled`) + - **swift**: (`module`, `type`, `member`, `swift-mangled`) + - **shell**: (`script_relpath`, `function_or_cmd`) + - **binary**: (`binary_build_id`, `section`, `symbol_name`) + +## nodes.ndjson +Each line: +``` +{ + "symbol_id": "sym:lang:...", + "lang": "dotnet", + "kind": "function|method|type|module|package|binary", + "display": "Human readable name", + "source": { + "file": "relative/or/pkg/path", + "line": 123, + "col": 1, + "digest": "sha256:" + }, + "attributes": { + "visibility": "public|internal|private", + "async": true, + "static": false, + "generic_arity": 2 + } +} +``` +Fields are optional when not applicable; omit rather than null. Additional language-specific fields allowed inside `attributes` (e.g., `jvm_descriptor`, `dotnet_signature`). + +## edges.ndjson +Each line (static or runtime-derived; see `source`): +``` +{ + "from": "sym:...", + "to": "sym:...", + "edge_type": "call|import|inherits|loads|dynamic|reflects|dlopen|ffi|wasm|spawn", + "confidence": "certain|high|medium|low", + "source": { + "origin": "static|runtime", + "provenance": "jvm-bytecode|il|ts-ast|ssa|ebpf|etw|jfr|hook", + "evidence": "file:path:line" + } +} +``` +- Ordering: primary `from`, secondary `to`, tertiary `edge_type`. +- Duplicate edges with different provenance are allowed; consumers deduplicate by (`from`,`to`,`edge_type`,`provenance`). + +## facts_runtime.ndjson (optional) +Runtime-only observations attached to symbols: +``` +{ + "symbol_id": "sym:...", + "samples": { + "call_count": 14, + "first_seen_utc": "2025-11-22T18:21:12Z", + "last_seen_utc": "2025-11-22T18:23:01Z" + }, + "env": { + "pid": 1234, + "image": "sha256:...", + "entrypoint": "main", + "tags": ["sealed","offline"] + } +} +``` +Sorting by `symbol_id`. Time fields must be UTC ISO-8601 with `Z`. + +## meta.json +``` +{ + "schema": "reachability-union@0.1", + "generated_at": "2025-11-23T00:00:00Z", + "produced_by": { + "tool": "StellaOps.Scanner.Worker", + "version": "0.1.0", + "analyzers": ["dotnet-11.1.0","jvm-8.0.0","node-6.2.0"] + }, + "files": [ + {"path":"nodes.ndjson","sha256":"...","records":1234}, + {"path":"edges.ndjson","sha256":"...","records":4567}, + {"path":"facts_runtime.ndjson","sha256":"...","records":89} + ], + "options": { + "dedupe_edges": false, + "include_runtime": true + } +} +``` + +## Determinism rules +- Sort order as noted; no nulls; omit empty objects/arrays. +- All strings UTF-8 NFC; booleans lower-case; edge_type enumerated list above. +- Hash inputs use exact serialized bytes (no trailing spaces, newline `\n` only). + +## Validation +- JSON Schema draft 2020-12 available at `docs/reachability/runtime-static-union-schema.json` (to be generated from this spec; allowable values match enumerations above). +- Minimal required fields: `symbol_id`, `lang`, `kind` (nodes); `from`, `to`, `edge_type`, `source.origin` (edges). + +## Integration guidance +- Static lifters must emit SymbolIDs using the language rules; runtime probes must map call targets to the same SymbolID space (via demangled names + package/module resolution). +- CAS writers store each file under the namespace path and return the root manifest path for downstream consumers (Signals, Replay, Policy). +- Consumers should treat runtime edges as additive; when both origins exist, prefer `origin=runtime` for exploitability scoring but keep static edges for coverage. diff --git a/src/Policy/StellaOps.Policy.Engine/Endpoints/PathScopeSimulationEndpoint.cs b/src/Policy/StellaOps.Policy.Engine/Endpoints/PathScopeSimulationEndpoint.cs index 622871e41..2cd1f9a4a 100644 --- a/src/Policy/StellaOps.Policy.Engine/Endpoints/PathScopeSimulationEndpoint.cs +++ b/src/Policy/StellaOps.Policy.Engine/Endpoints/PathScopeSimulationEndpoint.cs @@ -1,4 +1,5 @@ using System.Text; +using System.Text.Json; using Microsoft.AspNetCore.Http.HttpResults; using Microsoft.AspNetCore.Mvc; using StellaOps.Policy.Engine.Streaming; @@ -10,8 +11,7 @@ public static class PathScopeSimulationEndpoint public static IEndpointRouteBuilder MapPathScopeSimulation(this IEndpointRouteBuilder routes) { routes.MapPost("/simulation/path-scope", HandleAsync) - .WithName("PolicyEngine.PathScopeSimulation") - .WithOpenApi(); + .WithName("PolicyEngine.PathScopeSimulation"); return routes; } diff --git a/src/Policy/StellaOps.Policy.Engine/Program.cs b/src/Policy/StellaOps.Policy.Engine/Program.cs index 73b2714a9..c809a2dd1 100644 --- a/src/Policy/StellaOps.Policy.Engine/Program.cs +++ b/src/Policy/StellaOps.Policy.Engine/Program.cs @@ -104,13 +104,14 @@ builder.Services.AddOptions() builder.Services.AddSingleton(sp => sp.GetRequiredService>().Value); builder.Services.AddSingleton(TimeProvider.System); -builder.Services.AddSingleton(); -builder.Services.AddHostedService(); +builder.Services.AddSingleton(); +builder.Services.AddHostedService(); builder.Services.AddSingleton(); builder.Services.AddSingleton(); +builder.Services.AddSingleton(); builder.Services.AddSingleton(); builder.Services.AddSingleton(); -builder.Services.AddSingleton(); +builder.Services.AddSingleton(); builder.Services.AddHttpContextAccessor(); builder.Services.AddRouting(options => options.LowercaseUrls = true); diff --git a/src/Policy/StellaOps.Policy.Engine/Services/PathScopeMetrics.cs b/src/Policy/StellaOps.Policy.Engine/Services/PathScopeMetrics.cs new file mode 100644 index 000000000..b87e799a4 --- /dev/null +++ b/src/Policy/StellaOps.Policy.Engine/Services/PathScopeMetrics.cs @@ -0,0 +1,185 @@ +using System.Collections.Concurrent; +using System.Diagnostics.Metrics; + +namespace StellaOps.Policy.Engine.Services; + +/// +/// Metrics sink for path/scope-aware policy evaluation (POLICY-ENGINE-29-004). +/// Mirrors the prep contract at docs/modules/policy/prep/2025-11-20-policy-engine-29-004-prep.md. +/// +internal sealed class PathScopeMetrics : IDisposable +{ + private readonly Meter _meter; + private readonly Counter _evaluations; + private readonly Histogram _evaluationDurationMs; + private readonly Counter _cacheHit; + private readonly Counter _scopeMismatch; + private readonly ConcurrentDictionary<(string Tenant, string Source), CoverageState> _coverage = new(); + private readonly ObservableGauge _coverageGauge; + + public PathScopeMetrics() + { + _meter = new Meter("StellaOps.Policy.Engine", "1.0.0"); + + _evaluations = _meter.CreateCounter( + name: "policy.path.eval.total", + unit: "count", + description: "Total path/scope-aware evaluations processed."); + + _evaluationDurationMs = _meter.CreateHistogram( + name: "policy.path.eval.duration.ms", + unit: "ms", + description: "Latency distribution for path/scope-aware evaluations."); + + _cacheHit = _meter.CreateCounter( + name: "policy.path.eval.cache.hit", + unit: "count", + description: "Cache hit/miss counts for path/scope rule lookups."); + + _scopeMismatch = _meter.CreateCounter( + name: "policy.path.eval.scope.mismatch", + unit: "count", + description: "Counts of scope mismatches (depth/confidence/coverage)."); + + Func>> observe = ObserveCoverage; + _coverageGauge = _meter.CreateObservableGauge( + name: "policy.path.eval.coverage", + observeValues: observe, + unit: "percent", + description: "Share of observations with matching scope."); + } + + public void RecordEvaluation( + string tenant, + string subject, + string ruleId, + string pathMatch, + string result, + double durationMs, + bool scopeMatched = true) + { + var evalTags = new[] + { + new KeyValuePair("tenant", NormalizeTenant(tenant)), + new KeyValuePair("subject", NormalizeSubject(subject)), + new KeyValuePair("result", NormalizeResult(result)), + new KeyValuePair("ruleId", TruncateRule(ruleId)), + new KeyValuePair("pathMatch", NormalizePathMatch(pathMatch)) + }; + + _evaluations.Add(1, evalTags); + + var durationTags = new[] + { + new KeyValuePair("tenant", NormalizeTenant(tenant)), + new KeyValuePair("subject", NormalizeSubject(subject)), + new KeyValuePair("ruleId", TruncateRule(ruleId)) + }; + + _evaluationDurationMs.Record(durationMs, durationTags); + RecordCoverage(tenant, "path-scope", scopeMatched); + } + + public void RecordCacheHit(string tenant, string cache, bool hit) + { + var tags = new[] + { + new KeyValuePair("tenant", NormalizeTenant(tenant)), + new KeyValuePair("cache", NormalizeCache(cache)), + new KeyValuePair("hit", hit ? "true" : "false") + }; + + _cacheHit.Add(1, tags); + } + + public void RecordScopeMismatch(string tenant, string reason) + { + var tags = new[] + { + new KeyValuePair("tenant", NormalizeTenant(tenant)), + new KeyValuePair("reason", NormalizeScopeReason(reason)) + }; + + _scopeMismatch.Add(1, tags); + RecordCoverage(tenant, "path-scope", matched: false); + } + + private void RecordCoverage(string tenant, string source, bool matched) + { + var key = (NormalizeTenant(tenant), NormalizeSource(source)); + + _coverage.AddOrUpdate( + key, + _ => matched ? new CoverageState(1, 1) : new CoverageState(0, 1), + (_, state) => matched + ? new CoverageState(state.Matched + 1, state.Total + 1) + : new CoverageState(state.Matched, state.Total + 1)); + } + + private IEnumerable> ObserveCoverage() + { + foreach (var kvp in _coverage) + { + var state = kvp.Value; + if (state.Total == 0) + { + continue; + } + + var percentage = state.Matched * 100.0 / state.Total; + yield return new Measurement( + percentage, + new[] + { + new KeyValuePair("tenant", kvp.Key.Tenant), + new KeyValuePair("source", kvp.Key.Source) + }); + } + } + + private static string NormalizeTenant(string tenant) => + string.IsNullOrWhiteSpace(tenant) ? "unknown" : tenant; + + private static string NormalizeSubject(string subject) => + string.IsNullOrWhiteSpace(subject) ? "unknown" : subject; + + private static string NormalizeResult(string result) => + result switch + { + "allow" or "deny" or "error" => result, + _ => "deny" + }; + + private static string NormalizePathMatch(string pathMatch) => + pathMatch switch + { + "exact" or "prefix" or "glob" => pathMatch, + _ => "unknown" + }; + + private static string NormalizeCache(string cache) => + string.IsNullOrWhiteSpace(cache) ? "decision" : cache; + + private static string NormalizeScopeReason(string reason) => + string.IsNullOrWhiteSpace(reason) ? "no-scope" : reason; + + private static string NormalizeSource(string source) => + string.IsNullOrWhiteSpace(source) ? "path-scope" : source; + + private static string TruncateRule(string ruleId) + { + if (string.IsNullOrWhiteSpace(ruleId)) + { + return "unspecified"; + } + + return ruleId.Length <= 32 ? ruleId : ruleId[..32]; + } + + public void Dispose() + { + _meter.Dispose(); + } + + private readonly record struct CoverageState(long Matched, long Total); +} diff --git a/src/Policy/StellaOps.Policy.Engine/Services/PolicyEvaluationService.PathScope.cs b/src/Policy/StellaOps.Policy.Engine/Services/PolicyEvaluationService.PathScope.cs index 398eddfbc..d9c35f4f2 100644 --- a/src/Policy/StellaOps.Policy.Engine/Services/PolicyEvaluationService.PathScope.cs +++ b/src/Policy/StellaOps.Policy.Engine/Services/PolicyEvaluationService.PathScope.cs @@ -1,10 +1,23 @@ +using System.Diagnostics; +using System.Globalization; +using System.Linq; using System.Security.Cryptography; +using System.Text; +using System.Text.Json; +using System.Text.Json.Nodes; +using System.Text.Json.Serialization; using StellaOps.Policy.Engine.Streaming; namespace StellaOps.Policy.Engine.Services; -public sealed partial class PolicyEvaluationService +internal sealed partial class PolicyEvaluationService { + private const string StubRuleId = "policy.rules.path-scope.stub"; + private static readonly JsonSerializerOptions SerializerOptions = new(JsonSerializerDefaults.Web) + { + DefaultIgnoreCondition = JsonIgnoreCondition.WhenWritingNull + }; + public Task EvaluatePathScopeAsync( PathScopeSimulationRequest request, PathScopeTarget target, @@ -12,13 +25,16 @@ public sealed partial class PolicyEvaluationService { ct.ThrowIfCancellationRequested(); + var start = Stopwatch.GetTimestamp(); var stableKey = string.Create(CultureInfo.InvariantCulture, $"{request.BasePolicyRef}|{request.CandidatePolicyRef}|{target.FilePath}|{target.Pattern}"); var verdictDelta = ComputeDelta(stableKey); + var result = NormalizeResult(verdictDelta.candidateVerdict); + var correlationId = ComputeCorrelationId(stableKey); var finding = new JsonObject { ["id"] = target.EvidenceHash ?? "stub-ghsa", - ["ruleId"] = "policy.rules.path-scope.stub", + ["ruleId"] = StubRuleId, ["severity"] = "info", ["verdict"] = new JsonObject { @@ -67,6 +83,34 @@ public sealed partial class PolicyEvaluationService } }; + var durationMs = ElapsedMilliseconds(start); + ((JsonObject)envelope["metrics"]!)[@"durationMs"] = Math.Round(durationMs, 3, MidpointRounding.ToZero); + + _pathMetrics.RecordEvaluation( + tenant: request.Tenant, + subject: SimplifySubject(request.Subject), + ruleId: StubRuleId, + pathMatch: target.PathMatch, + result: result, + durationMs: durationMs, + scopeMatched: true); + + _pathMetrics.RecordCacheHit(request.Tenant, cache: "rule", hit: false); + + _logger.LogInformation( + "Policy.PathEval {@Tenant} {@RuleId} {@Subject} {@FilePath} {@PathMatch} {@Pattern} {@Confidence} {@Decision} {@DurationMs} {@EvidenceHash} {@CorrelationId}", + request.Tenant, + StubRuleId, + SimplifySubject(request.Subject), + target.FilePath, + target.PathMatch, + target.Pattern, + target.Confidence, + verdictDelta.candidateVerdict, + durationMs, + target.EvidenceHash ?? string.Empty, + correlationId); + return Task.FromResult(envelope); } @@ -83,4 +127,56 @@ public sealed partial class PolicyEvaluationService var delta = baseVerdict == candidateVerdict ? "unchanged" : "softened"; return (baseVerdict, candidateVerdict, delta); } + + private static string NormalizeResult(string candidateVerdict) => + string.Equals(candidateVerdict, "deny", StringComparison.OrdinalIgnoreCase) ? "deny" : "allow"; + + private static double ElapsedMilliseconds(long startTimestamp) + { + var elapsedTicks = Stopwatch.GetTimestamp() - startTimestamp; + return elapsedTicks * 1000.0 / Stopwatch.Frequency; + } + + private static string ComputeCorrelationId(string stableKey) + { + Span hashBytes = stackalloc byte[16]; + SHA256.HashData(Encoding.UTF8.GetBytes(stableKey), hashBytes); + return Convert.ToHexString(hashBytes); + } + + private static string SimplifySubject(PathScopeSubject subject) + { + if (subject is null) + { + return "unknown"; + } + + if (!string.IsNullOrWhiteSpace(subject.Purl)) + { + var purl = subject.Purl; + var trimmed = purl.StartsWith("pkg:", StringComparison.OrdinalIgnoreCase) ? purl[4..] : purl; + var slashIndex = trimmed.IndexOf('/', StringComparison.Ordinal); + if (slashIndex >= 0 && slashIndex + 1 < trimmed.Length) + { + var remainder = trimmed[(slashIndex + 1)..]; + var atIndex = remainder.IndexOf('@'); + var withoutVersion = atIndex >= 0 ? remainder[..atIndex] : remainder; + var lastSlash = withoutVersion.LastIndexOf('/'); + return lastSlash >= 0 ? withoutVersion[(lastSlash + 1)..] : withoutVersion; + } + } + + if (!string.IsNullOrWhiteSpace(subject.Cpe)) + { + var parts = subject.Cpe.Split(':', StringSplitOptions.RemoveEmptyEntries); + if (parts.Length > 4) + { + return parts[4]; + } + + return parts.LastOrDefault() ?? "unknown"; + } + + return "unknown"; + } } diff --git a/src/Policy/StellaOps.Policy.Engine/Services/PolicyEvaluationService.cs b/src/Policy/StellaOps.Policy.Engine/Services/PolicyEvaluationService.cs index 3ba566cc7..4ec2e5ff0 100644 --- a/src/Policy/StellaOps.Policy.Engine/Services/PolicyEvaluationService.cs +++ b/src/Policy/StellaOps.Policy.Engine/Services/PolicyEvaluationService.cs @@ -1,24 +1,29 @@ using System.Collections.Immutable; +using Microsoft.Extensions.Logging; +using Microsoft.Extensions.Logging.Abstractions; using StellaOps.Policy.Engine.Compilation; using StellaOps.Policy.Engine.Evaluation; namespace StellaOps.Policy.Engine.Services; -internal sealed class PolicyEvaluationService +internal sealed partial class PolicyEvaluationService { private readonly PolicyEvaluator evaluator = new(); private readonly PathScopeMetrics _pathMetrics; + private readonly ILogger _logger; - public PolicyEvaluationService() : this(new PathScopeMetrics()) + public PolicyEvaluationService() + : this(new PathScopeMetrics(), NullLogger.Instance) { } - public PolicyEvaluationService(PathScopeMetrics pathMetrics) + public PolicyEvaluationService(PathScopeMetrics pathMetrics, ILogger logger) { _pathMetrics = pathMetrics ?? throw new ArgumentNullException(nameof(pathMetrics)); + _logger = logger ?? throw new ArgumentNullException(nameof(logger)); } - public PolicyEvaluationResult Evaluate(PolicyIrDocument document, PolicyEvaluationContext context) + internal PolicyEvaluationResult Evaluate(PolicyIrDocument document, PolicyEvaluationContext context) { if (document is null) { diff --git a/src/Policy/StellaOps.Policy.Engine/Streaming/PathScopeSimulationService.cs b/src/Policy/StellaOps.Policy.Engine/Streaming/PathScopeSimulationService.cs index f7423aaeb..287f9e306 100644 --- a/src/Policy/StellaOps.Policy.Engine/Streaming/PathScopeSimulationService.cs +++ b/src/Policy/StellaOps.Policy.Engine/Streaming/PathScopeSimulationService.cs @@ -1,6 +1,7 @@ using System.Text.Json; using System.Text.Json.Nodes; using System.Text.Json.Serialization; +using System.Runtime.CompilerServices; using StellaOps.Policy.Engine.Services; namespace StellaOps.Policy.Engine.Streaming; @@ -10,7 +11,7 @@ namespace StellaOps.Policy.Engine.Streaming; /// Current behaviour emits no findings but enforces request validation, canonical ordering, /// and NDJSON framing so downstream consumers can integrate without schema drift. /// -public sealed class PathScopeSimulationService +internal sealed class PathScopeSimulationService { private static readonly JsonSerializerOptions SerializerOptions = new(JsonSerializerDefaults.Web) { @@ -19,6 +20,10 @@ public sealed class PathScopeSimulationService private readonly PolicyEvaluationService _evaluationService; + public PathScopeSimulationService() : this(new PolicyEvaluationService()) + { + } + public PathScopeSimulationService(PolicyEvaluationService evaluationService) { _evaluationService = evaluationService ?? throw new ArgumentNullException(nameof(evaluationService)); diff --git a/src/Policy/__Tests/StellaOps.Policy.Engine.Tests/PathScopeSimulationServiceTests.cs b/src/Policy/__Tests/StellaOps.Policy.Engine.Tests/PathScopeSimulationServiceTests.cs index 597b583e3..3db1f5fab 100644 --- a/src/Policy/__Tests/StellaOps.Policy.Engine.Tests/PathScopeSimulationServiceTests.cs +++ b/src/Policy/__Tests/StellaOps.Policy.Engine.Tests/PathScopeSimulationServiceTests.cs @@ -27,8 +27,8 @@ public sealed class PathScopeSimulationServiceTests var lines = await service.StreamAsync(request).ToListAsync(); Assert.Equal(2, lines.Count); - Assert.Contains(lines[0], s => s.Contains("\"filePath\":\"a/file.js\"")); - Assert.Contains(lines[1], s => s.Contains("\"filePath\":\"b/file.js\"")); + Assert.Contains("\"filePath\":\"a/file.js\"", lines[0]); + Assert.Contains("\"filePath\":\"b/file.js\"", lines[1]); } [Fact] @@ -44,6 +44,7 @@ public sealed class PathScopeSimulationServiceTests Targets: Array.Empty(), Options: new SimulationOptions("path,finding,verdict", 100, IncludeTrace: true, Deterministic: true)); - await Assert.ThrowsAsync(() => service.StreamAsync(request).ToListAsync()); + await Assert.ThrowsAsync(async () => + await service.StreamAsync(request).ToListAsync()); } } diff --git a/src/SbomService/StellaOps.SbomService.Tests/EntrypointEndpointsTests.cs b/src/SbomService/StellaOps.SbomService.Tests/EntrypointEndpointsTests.cs new file mode 100644 index 000000000..c5ba5cddd --- /dev/null +++ b/src/SbomService/StellaOps.SbomService.Tests/EntrypointEndpointsTests.cs @@ -0,0 +1,61 @@ +using System.Net; +using System.Net.Http.Json; +using StellaOps.SbomService.Models; + +namespace StellaOps.SbomService.Tests; + +public class EntrypointEndpointsTests : IClassFixture +{ + private readonly SbomServiceWebApplicationFactory _factory; + + public EntrypointEndpointsTests(SbomServiceWebApplicationFactory factory) + { + _factory = factory; + } + + [Fact] + public async Task Get_entrypoints_requires_tenant() + { + var client = _factory.CreateClient(); + + var response = await client.GetAsync("/entrypoints"); + response.StatusCode.Should().Be(HttpStatusCode.BadRequest); + } + + [Fact] + public async Task Get_entrypoints_returns_seeded_list() + { + var client = _factory.CreateClient(); + + var response = await client.GetAsync("/entrypoints?tenant=tenant-a"); + response.EnsureSuccessStatusCode(); + + var payload = await response.Content.ReadFromJsonAsync(); + payload.Should().NotBeNull(); + payload!.Tenant.Should().Be("tenant-a"); + payload.Items.Should().NotBeEmpty(); + payload.Items.Select(e => e.Artifact).Should().Contain("ghcr.io/stellaops/sample-api"); + } + + [Fact] + public async Task Post_entrypoints_upserts_and_returns_ordered_list() + { + var client = _factory.CreateClient(); + + var upsert = new EntrypointUpsertRequest( + Tenant: "tenant-a", + Artifact: "ghcr.io/stellaops/sample-api", + Service: "web", + Path: "/api/v2", + Scope: "runtime", + RuntimeFlag: true); + + var post = await client.PostAsJsonAsync("/entrypoints", upsert); + post.EnsureSuccessStatusCode(); + + var payload = await post.Content.ReadFromJsonAsync(); + payload.Should().NotBeNull(); + payload!.Items.First(e => e.Service == "web").Path.Should().Be("/api/v2"); + payload.Items.Should().BeInAscendingOrder(e => e.Artifact); + } +} diff --git a/src/SbomService/StellaOps.SbomService.Tests/ProjectionEndpointTests.cs b/src/SbomService/StellaOps.SbomService.Tests/ProjectionEndpointTests.cs index 8850cebb8..ece0a3957 100644 --- a/src/SbomService/StellaOps.SbomService.Tests/ProjectionEndpointTests.cs +++ b/src/SbomService/StellaOps.SbomService.Tests/ProjectionEndpointTests.cs @@ -3,6 +3,7 @@ using System.Net.Http.Json; using System.Reflection; using FluentAssertions; using Microsoft.AspNetCore.Mvc.Testing; +using Microsoft.AspNetCore.Hosting; using Microsoft.Extensions.Configuration; using Microsoft.Extensions.DependencyInjection; using Microsoft.Extensions.DependencyInjection.Extensions; diff --git a/src/SbomService/StellaOps.SbomService.Tests/SbomEndpointsTests.cs b/src/SbomService/StellaOps.SbomService.Tests/SbomEndpointsTests.cs index 32e1d7474..698622160 100644 --- a/src/SbomService/StellaOps.SbomService.Tests/SbomEndpointsTests.cs +++ b/src/SbomService/StellaOps.SbomService.Tests/SbomEndpointsTests.cs @@ -23,7 +23,8 @@ public class SbomEndpointsTests : IClassFixture> var response = await client.GetAsync("/sbom/paths"); - response.StatusCode.Should().Be(HttpStatusCode.BadRequest); + var body = await response.Content.ReadAsStringAsync(); + response.StatusCode.Should().Be(HttpStatusCode.BadRequest, body); } [Fact] @@ -47,7 +48,7 @@ public class SbomEndpointsTests : IClassFixture> var client = _factory.CreateClient(); var response = await client.GetAsync("/sbom/versions?artifact=ghcr.io/stellaops/sample-api"); - response.EnsureSuccessStatusCode(); + response.StatusCode.Should().Be(HttpStatusCode.OK, await response.Content.ReadAsStringAsync()); var payload = await response.Content.ReadFromJsonAsync(); payload.Should().NotBeNull(); diff --git a/src/SbomService/StellaOps.SbomService.Tests/SbomEventEndpointsTests.cs b/src/SbomService/StellaOps.SbomService.Tests/SbomEventEndpointsTests.cs index 84eb29d6b..94a1fdabf 100644 --- a/src/SbomService/StellaOps.SbomService.Tests/SbomEventEndpointsTests.cs +++ b/src/SbomService/StellaOps.SbomService.Tests/SbomEventEndpointsTests.cs @@ -31,9 +31,10 @@ public class SbomEventEndpointsTests : IClassFixture>("/internal/sbom/events"); events.Should().NotBeNull(); - events!.Should().HaveCount(1); - events[0].SnapshotId.Should().Be("snap-001"); - events[0].TenantId.Should().Be("tenant-a"); + var nonNullEvents = events!; + nonNullEvents.Should().HaveCount(1); + nonNullEvents[0].SnapshotId.Should().Be("snap-001"); + nonNullEvents[0].TenantId.Should().Be("tenant-a"); // Requesting the projection should not duplicate events. var projectionResponse = await client.GetAsync("/sboms/snap-001/projection?tenant=tenant-a"); diff --git a/src/SbomService/StellaOps.SbomService/Models/EntrypointModels.cs b/src/SbomService/StellaOps.SbomService/Models/EntrypointModels.cs new file mode 100644 index 000000000..36904fb40 --- /dev/null +++ b/src/SbomService/StellaOps.SbomService/Models/EntrypointModels.cs @@ -0,0 +1,20 @@ +namespace StellaOps.SbomService.Models; + +public sealed record Entrypoint( + string Artifact, + string Service, + string Path, + string Scope, + bool RuntimeFlag); + +public sealed record EntrypointUpsertRequest( + string Tenant, + string Artifact, + string Service, + string Path, + string Scope, + bool RuntimeFlag); + +public sealed record EntrypointListResponse( + string Tenant, + IReadOnlyList Items); diff --git a/src/SbomService/StellaOps.SbomService/Observability/SbomMetrics.cs b/src/SbomService/StellaOps.SbomService/Observability/SbomMetrics.cs index 7e1c865bf..8964820c6 100644 --- a/src/SbomService/StellaOps.SbomService/Observability/SbomMetrics.cs +++ b/src/SbomService/StellaOps.SbomService/Observability/SbomMetrics.cs @@ -21,4 +21,19 @@ internal static class SbomMetrics public static readonly Counter TimelineQueryTotal = Meter.CreateCounter("sbom_timeline_queries_total", description: "Total SBOM timeline queries"); -} + + public static readonly Histogram ProjectionLatencySeconds = + Meter.CreateHistogram("sbom_projection_seconds", unit: "s", + description: "Latency for SBOM projection reads"); + + public static readonly Histogram ProjectionSizeBytes = + Meter.CreateHistogram("sbom_projection_size_bytes", unit: "By", + description: "Payload size of SBOM projections returned"); + + public static readonly Counter ProjectionQueryTotal = + Meter.CreateCounter("sbom_projection_queries_total", + description: "Total SBOM projection queries"); + + public static readonly Histogram EventBacklogSize = + Meter.CreateHistogram("sbom_events_backlog", unit: "events", + description: "Observed size of the SBOM event outbox (in-memory) \ No newline at end of file diff --git a/src/SbomService/StellaOps.SbomService/Observability/SbomTracing.cs b/src/SbomService/StellaOps.SbomService/Observability/SbomTracing.cs new file mode 100644 index 000000000..6f9ba9eb9 --- /dev/null +++ b/src/SbomService/StellaOps.SbomService/Observability/SbomTracing.cs @@ -0,0 +1,9 @@ +using System.Diagnostics; + +namespace StellaOps.SbomService.Observability; + +internal static class SbomTracing +{ + public const string SourceName = "StellaOps.SbomService"; + public static readonly ActivitySource Source = new(SourceName); +} diff --git a/src/SbomService/StellaOps.SbomService/Program.cs b/src/SbomService/StellaOps.SbomService/Program.cs index 0d9ac4e17..9b6884b34 100644 --- a/src/SbomService/StellaOps.SbomService/Program.cs +++ b/src/SbomService/StellaOps.SbomService/Program.cs @@ -7,22 +7,24 @@ using StellaOps.SbomService.Services; using StellaOps.SbomService.Observability; using StellaOps.SbomService.Repositories; using System.Text.Json; +using System.Diagnostics; + +var builder = WebApplication.CreateBuilder(args); + +builder.Configuration + .AddJsonFile("appsettings.json", optional: true, reloadOnChange: true) + .AddEnvironmentVariables("SBOM_"); -var builder = WebApplication.CreateBuilder(args); - -builder.Configuration - .AddJsonFile("appsettings.json", optional: true, reloadOnChange: true) - .AddEnvironmentVariables("SBOM_"); - -builder.Services.AddOptions(); -builder.Services.AddLogging(); - +builder.Services.AddOptions(); +builder.Services.AddLogging(); + // Register SBOM query services (InMemory seed; replace with Mongo-backed repository later). builder.Services.AddSingleton(_ => new InMemoryComponentLookupRepository()); builder.Services.AddSingleton(); builder.Services.AddSingleton(); builder.Services.AddSingleton(sp => sp.GetRequiredService()); builder.Services.AddSingleton(); +builder.Services.AddSingleton(); builder.Services.AddSingleton(sp => { @@ -54,16 +56,85 @@ builder.Services.AddSingleton(sp => return new FileProjectionRepository(string.Empty); }); + +var app = builder.Build(); + +if (app.Environment.IsDevelopment()) +{ + app.Use(async (context, next) => + { + try + { + await next(); + } + catch (Exception ex) + { + Console.WriteLine($"[dev-exception] {ex}"); + throw; + } + }); + app.UseDeveloperExceptionPage(); +} + +app.MapGet("/healthz", () => Results.Ok(new { status = "ok" })); +app.MapGet("/readyz", () => Results.Ok(new { status = "warming" })); + +app.MapGet("/entrypoints", async Task ( + [FromServices] IEntrypointRepository repo, + [FromQuery] string? tenant, + CancellationToken cancellationToken) => +{ + if (string.IsNullOrWhiteSpace(tenant)) + { + return Results.BadRequest(new { error = "tenant is required" }); + } + + var tenantId = tenant.Trim(); + using var activity = SbomTracing.Source.StartActivity("entrypoints.list", ActivityKind.Server); + activity?.SetTag("tenant", tenantId); + + var items = await repo.ListAsync(tenantId, cancellationToken); + return Results.Ok(new EntrypointListResponse(tenantId, items)); +}); + +app.MapPost("/entrypoints", async Task ( + [FromServices] IEntrypointRepository repo, + [FromBody] EntrypointUpsertRequest request, + CancellationToken cancellationToken) => +{ + if (string.IsNullOrWhiteSpace(request.Tenant)) + { + return Results.BadRequest(new { error = "tenant is required" }); + } + + if (string.IsNullOrWhiteSpace(request.Artifact) || string.IsNullOrWhiteSpace(request.Service) || string.IsNullOrWhiteSpace(request.Path)) + { + return Results.BadRequest(new { error = "artifact, service, and path are required" }); + } + + var entrypoint = new Entrypoint( + request.Artifact.Trim(), + request.Service.Trim(), + request.Path.Trim(), + string.IsNullOrWhiteSpace(request.Scope) ? "runtime" : request.Scope.Trim(), + request.RuntimeFlag); + + var tenantId = request.Tenant.Trim(); + using var activity = SbomTracing.Source.StartActivity("entrypoints.upsert", ActivityKind.Server); + activity?.SetTag("tenant", tenantId); + activity?.SetTag("artifact", entrypoint.Artifact); + activity?.SetTag("service", entrypoint.Service); + + await repo.UpsertAsync(tenantId, entrypoint, cancellationToken); + + var items = await repo.ListAsync(tenantId, cancellationToken); + return Results.Ok(new EntrypointListResponse(tenantId, items)); +}); -var app = builder.Build(); - -app.MapGet("/healthz", () => Results.Ok(new { status = "ok" })); -app.MapGet("/readyz", () => Results.Ok(new { status = "warming" })); - -app.MapGet("/console/sboms", async Task ( - [FromServices] ISbomQueryService service, - [FromQuery] string? artifact, - [FromQuery] string? license, +app.MapGet("/console/sboms", async Task ( + [FromServices] ISbomQueryService service, + [FromQuery] string? artifact, + [FromQuery] string? license, [FromQuery] string? scope, [FromQuery(Name = "assetTag")] string? assetTag, [FromQuery] string? cursor, @@ -80,15 +151,17 @@ app.MapGet("/console/sboms", async Task ( return Results.BadRequest(new { error = "cursor must be an integer offset" }); } - var offset = cursor is null ? 0 : int.Parse(cursor, CultureInfo.InvariantCulture); - var pageSize = limit ?? 50; - - var start = Stopwatch.GetTimestamp(); - var result = await service.GetConsoleCatalogAsync( - new SbomCatalogQuery(artifact?.Trim(), license?.Trim(), scope?.Trim(), assetTag?.Trim(), pageSize, offset), - cancellationToken); - - var elapsedSeconds = Stopwatch.GetElapsedTime(start).TotalSeconds; + var offset = cursor is null ? 0 : int.Parse(cursor, CultureInfo.InvariantCulture); + var pageSize = limit ?? 50; + + using var activity = SbomTracing.Source.StartActivity("console.sboms", ActivityKind.Server); + activity?.SetTag("artifact", artifact); + var start = Stopwatch.GetTimestamp(); + var result = await service.GetConsoleCatalogAsync( + new SbomCatalogQuery(artifact?.Trim(), license?.Trim(), scope?.Trim(), assetTag?.Trim(), pageSize, offset), + cancellationToken); + + var elapsedSeconds = Stopwatch.GetElapsedTime(start).TotalSeconds; SbomMetrics.PathsLatencySeconds.Record(elapsedSeconds, new TagList { { "scope", scope ?? string.Empty }, @@ -103,10 +176,10 @@ app.MapGet("/console/sboms", async Task ( return Results.Ok(result.Result); }); -app.MapGet("/components/lookup", async Task ( - [FromServices] ISbomQueryService service, - [FromQuery] string? purl, - [FromQuery] string? artifact, +app.MapGet("/components/lookup", async Task ( + [FromServices] ISbomQueryService service, + [FromQuery] string? purl, + [FromQuery] string? artifact, [FromQuery] string? cursor, [FromQuery] int? limit, CancellationToken cancellationToken) => @@ -126,13 +199,16 @@ app.MapGet("/components/lookup", async Task ( return Results.BadRequest(new { error = "cursor must be an integer offset" }); } - var offset = cursor is null ? 0 : int.Parse(cursor, CultureInfo.InvariantCulture); - var pageSize = limit ?? 50; - - var start = Stopwatch.GetTimestamp(); - var result = await service.GetComponentLookupAsync( - new ComponentLookupQuery(purl.Trim(), artifact?.Trim(), pageSize, offset), - cancellationToken); + var offset = cursor is null ? 0 : int.Parse(cursor, CultureInfo.InvariantCulture); + var pageSize = limit ?? 50; + + using var activity = SbomTracing.Source.StartActivity("components.lookup", ActivityKind.Server); + activity?.SetTag("purl", purl); + activity?.SetTag("artifact", artifact); + var start = Stopwatch.GetTimestamp(); + var result = await service.GetComponentLookupAsync( + new ComponentLookupQuery(purl.Trim(), artifact?.Trim(), pageSize, offset), + cancellationToken); var elapsedSeconds = Stopwatch.GetElapsedTime(start).TotalSeconds; SbomMetrics.PathsLatencySeconds.Record(elapsedSeconds, new TagList @@ -149,13 +225,13 @@ app.MapGet("/components/lookup", async Task ( return Results.Ok(result.Result); }); -app.MapGet("/sbom/paths", async Task ( - [FromServices] ISbomQueryService service, - [FromQuery] string? purl, - [FromQuery] string? artifact, - [FromQuery] string? scope, - [FromQuery(Name = "env")] string? environment, - [FromQuery] string? cursor, +app.MapGet("/sbom/paths", async Task ( + [FromServices] IServiceProvider services, + [FromQuery] string? purl, + [FromQuery] string? artifact, + [FromQuery] string? scope, + [FromQuery(Name = "env")] string? environment, + [FromQuery] string? cursor, [FromQuery] int? limit, CancellationToken cancellationToken) => { @@ -172,15 +248,16 @@ app.MapGet("/sbom/paths", async Task ( if (cursor is { Length: > 0 } && !int.TryParse(cursor, NumberStyles.Integer, CultureInfo.InvariantCulture, out _)) { return Results.BadRequest(new { error = "cursor must be an integer offset" }); - } - - var offset = cursor is null ? 0 : int.Parse(cursor, CultureInfo.InvariantCulture); - var pageSize = limit ?? 50; - - var start = Stopwatch.GetTimestamp(); - var result = await service.GetPathsAsync( - new SbomPathQuery(purl.Trim(), artifact?.Trim(), scope?.Trim(), environment?.Trim(), pageSize, offset), - cancellationToken); + } + + var offset = cursor is null ? 0 : int.Parse(cursor, CultureInfo.InvariantCulture); + var pageSize = limit ?? 50; + + var service = services.GetRequiredService(); + var start = Stopwatch.GetTimestamp(); + var result = await service.GetPathsAsync( + new SbomPathQuery(purl.Trim(), artifact?.Trim(), scope?.Trim(), environment?.Trim(), pageSize, offset), + cancellationToken); var elapsedSeconds = Stopwatch.GetElapsedTime(start).TotalSeconds; SbomMetrics.PathsLatencySeconds.Record(elapsedSeconds, new TagList @@ -250,20 +327,37 @@ app.MapGet("/sboms/{snapshotId}/projection", async Task ( return Results.BadRequest(new { error = "tenant is required" }); } + var start = Stopwatch.GetTimestamp(); var projection = await service.GetProjectionAsync(snapshotId.Trim(), tenantId.Trim(), cancellationToken); if (projection is null) { return Results.NotFound(new { error = "projection not found" }); } - return Results.Ok(new + using var activity = SbomTracing.Source.StartActivity("sbom.projection", ActivityKind.Server); + activity?.SetTag("tenant", projection.TenantId); + activity?.SetTag("snapshotId", projection.SnapshotId); + activity?.SetTag("schema", projection.SchemaVersion); + + var payload = new { snapshotId = projection.SnapshotId, tenantId = projection.TenantId, schemaVersion = projection.SchemaVersion, hash = projection.ProjectionHash, projection = projection.Projection - }); + }; + + var json = JsonSerializer.Serialize(payload); + var sizeBytes = System.Text.Encoding.UTF8.GetByteCount(json); + SbomMetrics.ProjectionSizeBytes.Record(sizeBytes, new TagList { { "tenant", projection.TenantId } }); + SbomMetrics.ProjectionLatencySeconds.Record(Stopwatch.GetElapsedTime(start).TotalSeconds, + new TagList { { "tenant", projection.TenantId } }); + SbomMetrics.ProjectionQueryTotal.Add(1, new TagList { { "tenant", projection.TenantId } }); + + app.Logger.LogInformation("projection_returned tenant={Tenant} snapshot={Snapshot} size={SizeBytes}", projection.TenantId, projection.SnapshotId, sizeBytes); + + return Results.Ok(payload); }); app.MapGet("/internal/sbom/events", async Task ( diff --git a/src/SbomService/StellaOps.SbomService/Repositories/IComponentLookupRepository.cs b/src/SbomService/StellaOps.SbomService/Repositories/IComponentLookupRepository.cs index 22926fdc1..d8548bff4 100644 --- a/src/SbomService/StellaOps.SbomService/Repositories/IComponentLookupRepository.cs +++ b/src/SbomService/StellaOps.SbomService/Repositories/IComponentLookupRepository.cs @@ -4,5 +4,9 @@ namespace StellaOps.SbomService.Repositories; public interface IComponentLookupRepository { - Task> QueryAsync(ComponentLookupQuery query, CancellationToken cancellationToken); + /// + /// Returns a page of component neighbors along with the total count that match the query filters. + /// The total is required for deterministic pagination cursors. + /// + Task<(IReadOnlyList Items, int Total)> QueryAsync(ComponentLookupQuery query, CancellationToken cancellationToken); } diff --git a/src/SbomService/StellaOps.SbomService/Repositories/IEntrypointRepository.cs b/src/SbomService/StellaOps.SbomService/Repositories/IEntrypointRepository.cs new file mode 100644 index 000000000..6155a4d41 --- /dev/null +++ b/src/SbomService/StellaOps.SbomService/Repositories/IEntrypointRepository.cs @@ -0,0 +1,9 @@ +using StellaOps.SbomService.Models; + +namespace StellaOps.SbomService.Repositories; + +public interface IEntrypointRepository +{ + Task> ListAsync(string tenantId, CancellationToken cancellationToken); + Task UpsertAsync(string tenantId, Entrypoint entrypoint, CancellationToken cancellationToken); +} diff --git a/src/SbomService/StellaOps.SbomService/Repositories/InMemoryComponentLookupRepository.cs b/src/SbomService/StellaOps.SbomService/Repositories/InMemoryComponentLookupRepository.cs index 23ed0bf49..37ad5eaa8 100644 --- a/src/SbomService/StellaOps.SbomService/Repositories/InMemoryComponentLookupRepository.cs +++ b/src/SbomService/StellaOps.SbomService/Repositories/InMemoryComponentLookupRepository.cs @@ -6,7 +6,7 @@ public sealed class InMemoryComponentLookupRepository : IComponentLookupReposito { private static readonly IReadOnlyList Components = Seed(); - public Task> QueryAsync(ComponentLookupQuery query, CancellationToken cancellationToken) + public Task<(IReadOnlyList Items, int Total)> QueryAsync(ComponentLookupQuery query, CancellationToken cancellationToken) { var filtered = Components .Where(c => c.Purl.Equals(query.Purl, StringComparison.OrdinalIgnoreCase)) @@ -20,7 +20,7 @@ public sealed class InMemoryComponentLookupRepository : IComponentLookupReposito .Take(query.Limit) .ToList(); - return Task.FromResult>(page); + return Task.FromResult<(IReadOnlyList, int)>((page, filtered.Count)); } private static IReadOnlyList Seed() diff --git a/src/SbomService/StellaOps.SbomService/Repositories/InMemoryEntrypointRepository.cs b/src/SbomService/StellaOps.SbomService/Repositories/InMemoryEntrypointRepository.cs new file mode 100644 index 000000000..1714a3c1d --- /dev/null +++ b/src/SbomService/StellaOps.SbomService/Repositories/InMemoryEntrypointRepository.cs @@ -0,0 +1,56 @@ +using System.Collections.Concurrent; +using StellaOps.SbomService.Models; + +namespace StellaOps.SbomService.Repositories; + +public sealed class InMemoryEntrypointRepository : IEntrypointRepository +{ + // tenant -> list of entrypoints + private readonly ConcurrentDictionary> _store = new(StringComparer.OrdinalIgnoreCase); + + public InMemoryEntrypointRepository() + { + _store["tenant-a"] = new List + { + new("ghcr.io/stellaops/sample-api", "web", "/api", "runtime", true), + new("ghcr.io/stellaops/sample-worker", "worker", "queue:jobs", "runtime", true) + }; + + _store["tenant-b"] = new List + { + new("ghcr.io/stellaops/console", "ui", "/", "runtime", true) + }; + } + + public Task> ListAsync(string tenantId, CancellationToken cancellationToken) + { + var items = _store.TryGetValue(tenantId, out var list) + ? list.OrderBy(e => e.Artifact, StringComparer.OrdinalIgnoreCase) + .ThenBy(e => e.Service, StringComparer.OrdinalIgnoreCase) + .ThenBy(e => e.Path, StringComparer.Ordinal) + .ToList() + : new List(); + + return Task.FromResult>(items); + } + + public Task UpsertAsync(string tenantId, Entrypoint entrypoint, CancellationToken cancellationToken) + { + var list = _store.GetOrAdd(tenantId, _ => new List()); + + var existingIndex = list.FindIndex(e => + e.Artifact.Equals(entrypoint.Artifact, StringComparison.OrdinalIgnoreCase) && + e.Service.Equals(entrypoint.Service, StringComparison.OrdinalIgnoreCase)); + + if (existingIndex >= 0) + { + list[existingIndex] = entrypoint; + } + else + { + list.Add(entrypoint); + } + + return Task.CompletedTask; + } +} diff --git a/src/SbomService/StellaOps.SbomService/Repositories/MongoComponentLookupRepository.cs b/src/SbomService/StellaOps.SbomService/Repositories/MongoComponentLookupRepository.cs deleted file mode 100644 index 4ec2fda89..000000000 --- a/src/SbomService/StellaOps.SbomService/Repositories/MongoComponentLookupRepository.cs +++ /dev/null @@ -1,33 +0,0 @@ -using MongoDB.Driver; -using StellaOps.SbomService.Models; - -namespace StellaOps.SbomService.Repositories; - -internal sealed class MongoComponentLookupRepository : IComponentLookupRepository -{ - private readonly IMongoCollection _collection; - - public MongoComponentLookupRepository(IMongoDatabase database) - { - _collection = database.GetCollection("sbom_components"); - } - - public async Task> QueryAsync(ComponentLookupQuery query, CancellationToken cancellationToken) - { - var filter = Builders.Filter.Eq(c => c.Purl, query.Purl); - - if (!string.IsNullOrWhiteSpace(query.Artifact)) - { - filter &= Builders.Filter.Eq(c => c.Artifact, query.Artifact); - } - - var results = await _collection - .Find(filter) - .Skip(query.Offset) - .Limit(query.Limit) - .Sort(Builders.Sort.Ascending(c => c.Artifact).Ascending(c => c.NeighborPurl)) - .ToListAsync(cancellationToken); - - return results; - } -} diff --git a/src/SbomService/StellaOps.SbomService/Services/InMemorySbomQueryService.cs b/src/SbomService/StellaOps.SbomService/Services/InMemorySbomQueryService.cs index bc3ab0534..38a97f9eb 100644 --- a/src/SbomService/StellaOps.SbomService/Services/InMemorySbomQueryService.cs +++ b/src/SbomService/StellaOps.SbomService/Services/InMemorySbomQueryService.cs @@ -152,15 +152,15 @@ internal sealed class InMemorySbomQueryService : ISbomQueryService return new QueryResult(cachedResult, true); } - var page = await _componentLookupRepository.QueryAsync(query, cancellationToken); - - string? nextCursor = query.Offset + query.Limit < page.Count - ? (query.Offset + query.Limit).ToString(CultureInfo.InvariantCulture) - : null; - - var neighbors = page - .Select(c => new ComponentNeighbor(c.NeighborPurl, c.Relationship, c.License, c.Scope, c.RuntimeFlag)) - .ToList(); + var (items, total) = await _componentLookupRepository.QueryAsync(query, cancellationToken); + + string? nextCursor = query.Offset + query.Limit < total + ? (query.Offset + query.Limit).ToString(CultureInfo.InvariantCulture) + : null; + + var neighbors = items + .Select(c => new ComponentNeighbor(c.NeighborPurl, c.Relationship, c.License, c.Scope, c.RuntimeFlag)) + .ToList(); var result = new ComponentLookupResult(query.Purl, query.Artifact, neighbors, nextCursor, CacheHint: "seeded"); _cache[cacheKey] = result; diff --git a/src/SbomService/StellaOps.SbomService/StellaOps.SbomService.csproj b/src/SbomService/StellaOps.SbomService/StellaOps.SbomService.csproj index 3c72514e9..317d0c560 100644 --- a/src/SbomService/StellaOps.SbomService/StellaOps.SbomService.csproj +++ b/src/SbomService/StellaOps.SbomService/StellaOps.SbomService.csproj @@ -15,6 +15,5 @@ - diff --git a/src/SbomService/TASKS.md b/src/SbomService/TASKS.md index 1db3f6155..621eda8d7 100644 --- a/src/SbomService/TASKS.md +++ b/src/SbomService/TASKS.md @@ -3,3 +3,5 @@ | Task ID | Status | Notes | Updated (UTC) | | --- | --- | --- | --- | | PREP-SBOM-CONSOLE-23-001-BUILD-TEST-FAILING-D | DONE | Offline feed cache + script added; see `docs/modules/sbomservice/offline-feed-plan.md`. | 2025-11-20 | +| SBOM-SERVICE-21-002 | DONE | `sbom.version.created` events emitted via in-memory publisher; `/internal/sbom/events` + backfill wired; component lookup pagination cursor fixed; tests pass. | 2025-11-23 | +| SBOM-SERVICE-21-003 | DONE | Entrypoint/service node API (`GET/POST /entrypoints`) with tenant guard, deterministic ordering, seeded data; tests added. | 2025-11-23 |