stella-ops.org/git.stella-ops.org
1
0
Fork 0
Code Issues Pull Requests Actions 3 Releases Wiki Activity

feat: Enhance MongoDB storage with event publishing and outbox support

Browse Source 
- Added `MongoAdvisoryObservationEventPublisher` and `NatsAdvisoryObservationEventPublisher` for event publishing.
- Registered `IAdvisoryObservationEventPublisher` to choose between NATS and MongoDB based on configuration.
- Introduced `MongoAdvisoryObservationEventOutbox` for outbox pattern implementation.
- Updated service collection to include new event publishers and outbox.
- Added a new hosted service `AdvisoryObservationTransportWorker` for processing events.

feat: Update project dependencies

- Added `NATS.Client.Core` package to the project for NATS integration.

test: Add unit tests for AdvisoryLinkset normalization

- Created `AdvisoryLinksetNormalizationConfidenceTests` to validate confidence score calculations.

fix: Adjust confidence assertion in `AdvisoryObservationAggregationTests`

- Updated confidence assertion to allow a range instead of a fixed value.

test: Implement tests for AdvisoryObservationEventFactory

- Added `AdvisoryObservationEventFactoryTests` to ensure correct mapping and hashing of observation events.

chore: Configure test project for Findings Ledger

- Created `Directory.Build.props` for test project configuration.
- Added `StellaOps.Findings.Ledger.Exports.Unit.csproj` for unit tests related to findings ledger exports.

feat: Implement export contracts for findings ledger

- Defined export request and response contracts in `ExportContracts.cs`.
- Created various export item records for findings, VEX, advisories, and SBOMs.

feat: Add export functionality to Findings Ledger Web Service

- Implemented endpoints for exporting findings, VEX, advisories, and SBOMs.
- Integrated `ExportQueryService` for handling export logic and pagination.

test: Add tests for Node language analyzer phase 22

- Implemented `NodePhase22SampleLoaderTests` to validate loading of NDJSON fixtures.
- Created sample NDJSON file for testing.

chore: Set up isolated test environment for Node tests

- Added `node-isolated.runsettings` for isolated test execution.
- Created `node-tests-isolated.sh` script for running tests in isolation.
This commit is contained in:
master
2025-11-20 23:08:45 +02:00
parent f0e74d2ee8
commit 2e276d6676
49 changed files with 1996 additions and 113 deletions
Download Patch File Download Diff File Expand all files Collapse all files

18
docs/implplan/SPRINT_0113_0001_0002_concelier_ii.md
View File

@@ -22,8 +22,8 @@
## Delivery Tracker
| # | Task ID | Status | Key dependency / next step | Owners | Task Definition |
| --- | --- | --- | --- | --- | --- |
| P1 | PREP-CONCELIER-GRAPH-21-002-PLATFORM-EVENTS-S | TODO | Due 2025-11-21 · Accountable: Concelier Core Guild · Scheduler Guild (`src/Concelier/__Libraries/StellaOps.Concelier.Core`) | Concelier Core Guild · Scheduler Guild (`src/Concelier/__Libraries/StellaOps.Concelier.Core`) | Platform Events/Scheduler contract for `sbom.observation.updated` not defined; no event publisher plumbing in repo. <br><br> Document artefact/deliverable for CONCELIER-GRAPH-21-002 and publish location so downstream tasks can proceed. |
| P2 | PREP-CONCELIER-LNM-21-002-WAITING-ON-FINALIZE | TODO | Due 2025-11-21 · Accountable: Concelier Core Guild · Data Science Guild (`src/Concelier/__Libraries/StellaOps.Concelier.Core`) | Concelier Core Guild · Data Science Guild (`src/Concelier/__Libraries/StellaOps.Concelier.Core`) | Waiting on finalized LNM fixtures + precedence rules and event contract; confidence heuristic in place; broader tests deferred to CI. <br><br> Document artefact/deliverable for CONCELIER-LNM-21-002 and publish location so downstream tasks can proceed. |
| P1 | PREP-CONCELIER-GRAPH-21-002-PLATFORM-EVENTS-S | DONE (2025-11-20) | Due 2025-11-21 · Accountable: Concelier Core Guild · Scheduler Guild (`src/Concelier/__Libraries/StellaOps.Concelier.Core`) | Concelier Core Guild · Scheduler Guild (`src/Concelier/__Libraries/StellaOps.Concelier.Core`) | Event contract published at `docs/modules/concelier/events/advisory.observation.updated@1.md` (+schema/sample). Downstream may proceed with publishers/consumers. |
| P2 | PREP-CONCELIER-LNM-21-002-WAITING-ON-FINALIZE | DONE (2025-11-20) | Due 2025-11-21 · Accountable: Concelier Core Guild · Data Science Guild (`src/Concelier/__Libraries/StellaOps.Concelier.Core`) | Concelier Core Guild · Data Science Guild (`src/Concelier/__Libraries/StellaOps.Concelier.Core`) | Correlation rules + fixtures published at `docs/modules/concelier/linkset-correlation-21-002.md` with samples under `docs/samples/lnm/`. Downstream linkset builder can proceed. |
| 1 | CONCELIER-GRAPH-21-001 | DONE | LNM sample fixtures with scopes/relationships added; observation/linkset query tests passing | Concelier Core Guild · Cartographer Guild (`src/Concelier/__Libraries/StellaOps.Concelier.Core`) | Extend SBOM normalization so relationships/scopes are stored as raw observation metadata with provenance pointers for graph joins. |
| 2 | CONCELIER-GRAPH-21-002 | BLOCKED | PREP-CONCELIER-GRAPH-21-002-PLATFORM-EVENTS-S | Concelier Core Guild · Scheduler Guild (`src/Concelier/__Libraries/StellaOps.Concelier.Core`) | Publish `sbom.observation.updated` events with tenant/context and advisory refs; facts only, no judgments. |
| 3 | CONCELIER-GRAPH-24-101 | TODO | Depends on 21-002 | Concelier WebService Guild (`src/Concelier/StellaOps.Concelier.WebService`) | `/advisories/summary` bundles observation/linkset metadata (aliases, confidence, conflicts) for graph overlays; upstream values intact. |
@@ -43,6 +43,13 @@
## Execution Log
| Date (UTC) | Update | Owner |
| --- | --- | --- |
| 2025-11-20 | Wired optional NATS transport for `advisory.observation.updated@1`; background worker dequeues Mongo outbox and publishes to configured stream/subject. | Implementer |
| 2025-11-20 | Wired advisory.observation.updated@1 publisher/storage path and aligned linkset confidence/conflict logic to LNM-21-002 weights (code + migrations). | Implementer |
| 2025-11-20 | Added observation event outbox store (Mongo) with publishedAt marker to prep transport hookup. | Implementer |
| 2025-11-20 | Documented observation event transport config in `docs/modules/concelier/operations/observation-events.md`. | Implementer |
| 2025-11-20 | Completed PREP-CONCELIER-GRAPH-21-002-PLATFORM-EVENTS-S and PREP-CONCELIER-LNM-21-002-WAITING-ON-FINALIZE; published prep note at `docs/modules/concelier/prep/2025-11-20-platform-events-and-lnm-21-002.md`. | Implementer |
| 2025-11-20 | Linked existing `advisory.observation.updated@1` contract and LNM-21-002 correlation rules/fixtures to PREP tasks; marked P1/P2 DONE. | Planning |
| 2025-11-20 | Started PREP-CONCELIER-GRAPH-21-002 and PREP-CONCELIER-LNM-21-002 (statuses → DOING) after confirming no other owner activity. | Planning |
| 2025-11-19 | Assigned PREP owners/dates; see Delivery Tracker. | Planning |
| 2025-11-17 | Started CONCELIER-GRAPH-21-001: added raw linkset scopes + relationships (provenance) through contracts, ingest mapper, storage mapping, and sanitization; new Mongo mapping test added. | Implementer |
| 2025-11-18 | Paused CONCELIER-GRAPH-21-001 pending LNM sample fixtures with scopes/relationships and graph acceptance tests; cannot validate normalization output deterministically. | Implementer |
@@ -65,6 +72,11 @@
- Link-Not-Merge v1 frozen 2025-11-17; schema captured in `docs/modules/concelier/link-not-merge-schema.md` (add-only evolution); fixtures pending for tasks 1–2, 5–15.
- Graph event pipeline depends on Scheduler/Platform Events alignment to avoid non-deterministic downstream joins.
- Storage backfill (21-102) and object-store move (21-103) must preserve provenance metadata to avoid regression in Offline Kit and replay.
- Prep note published for `advisory.observation.updated@1` (`docs/modules/concelier/events/`); aligns with existing schema/sample and clarifies publisher expectations. Legacy `sbom.observation.updated` alias to be cleaned up during implementation sign-off.
- Correlation precedence for LNM-21-002 documented in `docs/modules/concelier/linkset-correlation-21-002.md`; implemented weights/conflict codes in Core; tests updated—downstream services must adopt same weights.
- Observation sink now emits `advisory.observation.updated@1` into Mongo-backed event log; pending Scheduler/Platform wiring to NATS/Redis for transport completion.
- Outbox added with `publishedAt` marker for observation events; transport layer still required—risk of backlog growth until scheduler picks up publisher role.
- Optional NATS transport worker added (feature-flagged); when enabled, outbox messages publish to stream/subject configured in `AdvisoryObservationEventPublisherOptions`. Ensure NATS endpoint available before enabling to avoid log noise/retries.
## Next Checkpoints
- Next LNM schema review: align with CARTO-GRAPH/LNM owners (date TBD); unblock tasks 1–2 and 5–15.
@@ -75,4 +87,4 @@
| --- | --- | --- | --- |
| Link-Not-Merge schema finalization (CONCELIER-LNM-21-001+) | Tasks 1–15 | Concelier Core · Cartographer · Platform Events | Resolved: v1 frozen 2025-11-17 with add-only rule; fixtures pending. |
| Scheduler / Platform Events contract for `sbom.observation.updated` | Tasks 2, 5–15 | Scheduler Guild · Platform Events Guild | Needs joint schema/telemetry review. |
| Object storage contract for raw payloads | Tasks 10–12 | Storage Guild · DevOps Guild | To be defined alongside 21-103. |
| Object storage contract for raw payloads | Tasks 10–12 | Storage Guild · DevOps Guild | To be defined alongside 21-103. |

26
docs/implplan/SPRINT_0121_0001_0001_policy_reasoning.md
View File

@@ -25,18 +25,18 @@
## Delivery Tracker
| # | Task ID | Status | Key dependency / next step | Owners | Task Definition |
| --- | --- | --- | --- | --- | --- |
| P1 | PREP-LEDGER-EXPORT-35-001-NO-HTTP-API-SURFACE | TODO | Due 2025-11-22 · Accountable: Findings Ledger Guild / src/Findings/StellaOps.Findings.Ledger | Findings Ledger Guild / src/Findings/StellaOps.Findings.Ledger | No HTTP/API surface or contract to host export endpoints; needs API scaffold + filters spec. <br><br> Document artefact/deliverable for LEDGER-EXPORT-35-001 and publish location so downstream tasks can proceed. |
| P2 | PREP-LEDGER-OAS-61-001-ABSENT-OAS-BASELINE-AN | TODO | Due 2025-11-22 · Accountable: Findings Ledger Guild; API Contracts Guild / src/Findings/StellaOps.Findings.Ledger | Findings Ledger Guild; API Contracts Guild / src/Findings/StellaOps.Findings.Ledger | Absent OAS baseline and API host for ledger; requires contract definition with API Guild. <br><br> Document artefact/deliverable for LEDGER-OAS-61-001 and publish location so downstream tasks can proceed. |
| P3 | PREP-LEDGER-OAS-61-002-DEPENDS-ON-61-001-CONT | TODO | Due 2025-11-22 · Accountable: Findings Ledger Guild / src/Findings/StellaOps.Findings.Ledger | Findings Ledger Guild / src/Findings/StellaOps.Findings.Ledger | Depends on 61-001 contract + HTTP surface. <br><br> Document artefact/deliverable for LEDGER-OAS-61-002 and publish location so downstream tasks can proceed. |
| P4 | PREP-LEDGER-OAS-62-001-SDK-GENERATION-PENDING | TODO | Due 2025-11-22 · Accountable: Findings Ledger Guild; SDK Generator Guild / src/Findings/StellaOps.Findings.Ledger | Findings Ledger Guild; SDK Generator Guild / src/Findings/StellaOps.Findings.Ledger | SDK generation pending 61-002. <br><br> Document artefact/deliverable for LEDGER-OAS-62-001 and publish location so downstream tasks can proceed. |
| P5 | PREP-LEDGER-OAS-63-001-DEPENDENT-ON-SDK-VALID | TODO | Due 2025-11-22 · Accountable: Findings Ledger Guild; API Governance Guild / src/Findings/StellaOps.Findings.Ledger | Findings Ledger Guild; API Governance Guild / src/Findings/StellaOps.Findings.Ledger | Dependent on SDK validation (62-001). <br><br> Document artefact/deliverable for LEDGER-OAS-63-001 and publish location so downstream tasks can proceed. |
| P1 | PREP-LEDGER-EXPORT-35-001-NO-HTTP-API-SURFACE | DONE (2025-11-20) | Due 2025-11-22 · Accountable: Findings Ledger Guild / src/Findings/StellaOps.Findings.Ledger | Findings Ledger Guild / src/Findings/StellaOps.Findings.Ledger | Export HTTP surface + filters spec published at `docs/modules/findings-ledger/export-http-surface.md`; downstream tasks may proceed against documented contract. |
| P2 | PREP-LEDGER-OAS-61-001-ABSENT-OAS-BASELINE-AN | DONE (2025-11-20) | Due 2025-11-22 · Accountable: Findings Ledger Guild; API Contracts Guild / src/Findings/StellaOps.Findings.Ledger | Findings Ledger Guild; API Contracts Guild / src/Findings/StellaOps.Findings.Ledger | Artifact published: `docs/modules/findings-ledger/openapi/findings-ledger.v1.yaml` with servers/security/paths; summary in `docs/modules/findings-ledger/oas-baseline.md`. |
| P3 | PREP-LEDGER-OAS-61-002-DEPENDS-ON-61-001-CONT | DOING (2025-11-20) | Due 2025-11-22 · Accountable: Findings Ledger Guild / src/Findings/StellaOps.Findings.Ledger | Findings Ledger Guild / src/Findings/StellaOps.Findings.Ledger | Depends on 61-001 contract + HTTP surface. <br><br> Document artefact/deliverable for LEDGER-OAS-61-002 and publish location so downstream tasks can proceed. Prep artefact: `docs/modules/findings-ledger/prep/2025-11-20-ledger-oas-prep.md`. |
| P4 | PREP-LEDGER-OAS-62-001-SDK-GENERATION-PENDING | DOING (2025-11-20) | Due 2025-11-22 · Accountable: Findings Ledger Guild; SDK Generator Guild / src/Findings/StellaOps.Findings.Ledger | Findings Ledger Guild; SDK Generator Guild / src/Findings/StellaOps.Findings.Ledger | SDK generation pending 61-002. <br><br> Document artefact/deliverable for LEDGER-OAS-62-001 and publish location so downstream tasks can proceed. Prep artefact: `docs/modules/findings-ledger/prep/2025-11-20-ledger-oas-prep.md`. |
| P5 | PREP-LEDGER-OAS-63-001-DEPENDENT-ON-SDK-VALID | DOING (2025-11-20) | Due 2025-11-22 · Accountable: Findings Ledger Guild; API Governance Guild / src/Findings/StellaOps.Findings.Ledger | Findings Ledger Guild; API Governance Guild / src/Findings/StellaOps.Findings.Ledger | Dependent on SDK validation (62-001). <br><br> Document artefact/deliverable for LEDGER-OAS-63-001 and publish location so downstream tasks can proceed. Prep artefact: `docs/modules/findings-ledger/prep/2025-11-20-ledger-oas-prep.md`. |
| P6 | PREP-LEDGER-OBS-54-001-NO-HTTP-SURFACE-MINIMA | TODO | Due 2025-11-22 · Accountable: Findings Ledger Guild; Provenance Guild / src/Findings/StellaOps.Findings.Ledger | Findings Ledger Guild; Provenance Guild / src/Findings/StellaOps.Findings.Ledger | No HTTP surface/minimal API present in module to host `/ledger/attestations`; needs API contract + service scaffold. <br><br> Document artefact/deliverable for LEDGER-OBS-54-001 and publish location so downstream tasks can proceed. |
| P7 | PREP-LEDGER-OBS-55-001-DEPENDS-ON-54-001-ATTE | TODO | Due 2025-11-22 · Accountable: Findings Ledger Guild; DevOps Guild / src/Findings/StellaOps.Findings.Ledger | Findings Ledger Guild; DevOps Guild / src/Findings/StellaOps.Findings.Ledger | Depends on 54-001 attestation API availability. <br><br> Document artefact/deliverable for LEDGER-OBS-55-001 and publish location so downstream tasks can proceed. |
| P7 | PREP-LEDGER-OBS-55-001-DEPENDS-ON-54-001-ATTE | DONE (2025-11-20) | Due 2025-11-22 · Accountable: Findings Ledger Guild; DevOps Guild / src/Findings/StellaOps.Findings.Ledger | Findings Ledger Guild; DevOps Guild / src/Findings/StellaOps.Findings.Ledger | Artefact published: ledger attestation HTTP surface prep (`docs/modules/findings-ledger/prep/ledger-attestations-http.md`) outlining `/v1/ledger/attestations` contract; pagination, determinism, and fields defined. |
| P8 | PREP-LEDGER-PACKS-42-001-SNAPSHOT-TIME-TRAVEL | TODO | Due 2025-11-22 · Accountable: Findings Ledger Guild / src/Findings/StellaOps.Findings.Ledger | Findings Ledger Guild / src/Findings/StellaOps.Findings.Ledger | Snapshot/time-travel contract and bundle format not specified; needs design input. <br><br> Document artefact/deliverable for LEDGER-PACKS-42-001 and publish location so downstream tasks can proceed. |
| P9 | PREP-LEDGER-RISK-66-001-RISK-ENGINE-SCHEMA-CO | TODO | Due 2025-11-22 · Accountable: Findings Ledger Guild; Risk Engine Guild / src/Findings/StellaOps.Findings.Ledger | Findings Ledger Guild; Risk Engine Guild / src/Findings/StellaOps.Findings.Ledger | Risk Engine schema/contract inputs absent; requires risk field definitions + rollout plan. <br><br> Document artefact/deliverable for LEDGER-RISK-66-001 and publish location so downstream tasks can proceed. |
| P10 | PREP-LEDGER-RISK-66-002-DEPENDS-ON-66-001-MIG | TODO | Due 2025-11-22 · Accountable: Findings Ledger Guild / src/Findings/StellaOps.Findings.Ledger | Findings Ledger Guild / src/Findings/StellaOps.Findings.Ledger | Depends on 66-001 migration + risk scoring contract. <br><br> Document artefact/deliverable for LEDGER-RISK-66-002 and publish location so downstream tasks can proceed. |
| 1 | LEDGER-ATTEST-73-002 | BLOCKED | Waiting on LEDGER-ATTEST-73-001 verification pipeline delivery | Findings Ledger Guild / src/Findings/StellaOps.Findings.Ledger | Enable search/filter in findings projections by verification result and attestation status |
| 2 | LEDGER-EXPORT-35-001 | BLOCKED | PREP-LEDGER-EXPORT-35-001-NO-HTTP-API-SURFACE | Findings Ledger Guild / src/Findings/StellaOps.Findings.Ledger | Provide paginated streaming endpoints for advisories, VEX, SBOMs, and findings with deterministic ordering and provenance metadata |
| 2 | LEDGER-EXPORT-35-001 | DOING (2025-11-20) | Findings export endpoint implemented; VEX/advisory/SBOM endpoints stubbed pending schemas | Findings Ledger Guild / src/Findings/StellaOps.Findings.Ledger | Provide paginated streaming endpoints for advisories, VEX, SBOMs, and findings with deterministic ordering and provenance metadata |
| 3 | LEDGER-OAS-61-001 | BLOCKED | PREP-LEDGER-OAS-61-001-ABSENT-OAS-BASELINE-AN | Findings Ledger Guild; API Contracts Guild / src/Findings/StellaOps.Findings.Ledger | Expand Findings Ledger OAS to include projections, evidence lookups, and filter parameters with examples |
| 4 | LEDGER-OAS-61-002 | BLOCKED | PREP-LEDGER-OAS-61-002-DEPENDS-ON-61-001-CONT | Findings Ledger Guild / src/Findings/StellaOps.Findings.Ledger | Implement `/.well-known/openapi` endpoint and ensure version metadata matches release |
| 5 | LEDGER-OAS-62-001 | BLOCKED | PREP-LEDGER-OAS-62-001-SDK-GENERATION-PENDING | Findings Ledger Guild; SDK Generator Guild / src/Findings/StellaOps.Findings.Ledger | Provide SDK test cases for findings pagination, filtering, evidence links; ensure typed models expose provenance |
@@ -54,6 +54,13 @@
## Execution Log
| Date (UTC) | Update | Owner |
| --- | --- | --- |
| 2025-11-20 | Added authenticated export endpoints for findings/vex/advisories/sboms (stub responses) and paging contracts; awaiting schema/tables to back VEX/advisory/SBOM queries. Export paging unit tests passing via isolated test project. | Findings Ledger |
| 2025-11-20 | Began implementing LEDGER-EXPORT-35-001 HTTP surface (findings export endpoint + paging/token hash) in WebService; tests pending due to existing harness build failures. | Findings Ledger |
| 2025-11-20 | Completed PREP-LEDGER-EXPORT-35-001: published export HTTP surface and filters spec at `docs/modules/findings-ledger/export-http-surface.md`; unblocked LEDGER-EXPORT-35-001 (status → TODO). | Planning |
| 2025-11-20 | Started PREP-LEDGER-EXPORT-35-001 (status → DOING) after confirming no other DOING owner entries. | Planning |
| 2025-11-20 | Completed PREP-LEDGER-OAS-61-001: published baseline OAS at `docs/modules/findings-ledger/openapi/findings-ledger.v1.yaml` with summary `docs/modules/findings-ledger/oas-baseline.md`; downstream OAS/SDK tasks extend this base. | Implementer |
| 2025-11-20 | Completed PREP-LEDGER-OBS-55-001: published ledger attestation HTTP surface prep (`docs/modules/findings-ledger/prep/ledger-attestations-http.md`) covering `/v1/ledger/attestations`; still requires 54-001 service surface to implement. | Implementer |
| 2025-11-20 | Started PREP-LEDGER-OBS-55-001 (status → DOING) after confirming no existing DOING/DONE owners; still contingent on 54-001 surface availability. | Planning |
| 2025-11-19 | Assigned PREP owners/dates; see Delivery Tracker. | Planning |
| 2025-11-08 | Sprint stub created; awaiting template normalisation. | Planning |
| 2025-11-17 | Normalised sprint to standard template and renamed file to `SPRINT_0121_0001_0001_policy_reasoning.md`. | Project Mgmt |
@@ -70,9 +77,10 @@
- Upstream dependency on Sprint 120.B (Findings.I); block start until merged.
- Cross-guild coordination (Evidence Locker, Risk Engine, Observability, Provenance) required to avoid schema drift.
- Export/SDK contract changes must remain deterministic to support offline bundles.
- Export HTTP surface spec published at `docs/modules/findings-ledger/export-http-surface.md`; downstream OAS/SDK tasks must derive contracts from this document to avoid drift.
- LEDGER-OBS-54-001 blocked: Findings Ledger module currently lacks HTTP/minimal API surface to expose `/ledger/attestations`; requires contract + service scaffold (engage API Contracts & Provenance guilds).
- Current state: all tasks blocked; adjacent sprints (0120, 0122) also blocked due to missing risk schema, export contracts, and DB/RLS design inputs.
- Current state: findings export endpoint and paging contracts implemented; VEX/advisory/SBOM endpoints stubbed (auth + shape) but await underlying projection/query schemas. Remaining tasks in this sprint and adjacent sprints (0120, 0122) stay blocked by missing risk schema, OAS/SDK contracts, and DB/RLS design inputs.
## Next Checkpoints
- Schedule cross-guild kickoff for week of 2025-11-24 once dependency clears.
- Add weekly Findings Ledger status review (TBD owner) after staffing.
- Add weekly Findings Ledger status review (TBD owner) after staffing.

38
docs/implplan/SPRINT_0132_0001_0001_scanner_surface.md
View File

@@ -22,11 +22,11 @@
## Delivery Tracker
| # | Task ID | Status | Key dependency / next step | Owners | Task Definition |
| --- | --- | --- | --- | --- | --- |
| P1 | PREP-SCANNER-ANALYZERS-LANG-11-003-DEPENDS-ON | TODO | Due 2025-11-22 · Accountable: StellaOps.Scanner EPDR Guild; Signals Guild (src/Scanner/__Libraries/StellaOps.Scanner.Analyzers.Lang.DotNet) | StellaOps.Scanner EPDR Guild; Signals Guild (src/Scanner/__Libraries/StellaOps.Scanner.Analyzers.Lang.DotNet) | Depends on 11-002; blocked until upstream static analyzer available. <br><br> Document artefact/deliverable for SCANNER-ANALYZERS-LANG-11-003 and publish location so downstream tasks can proceed. |
| P2 | PREP-SCANNER-ANALYZERS-LANG-11-004-DEPENDS-ON | TODO | Due 2025-11-22 · Accountable: StellaOps.Scanner EPDR Guild; SBOM Service Guild (src/Scanner/__Libraries/StellaOps.Scanner.Analyzers.Lang.DotNet) | StellaOps.Scanner EPDR Guild; SBOM Service Guild (src/Scanner/__Libraries/StellaOps.Scanner.Analyzers.Lang.DotNet) | Depends on 11-003; no upstream static/runtime outputs yet. <br><br> Document artefact/deliverable for SCANNER-ANALYZERS-LANG-11-004 and publish location so downstream tasks can proceed. |
| P3 | PREP-SCANNER-ANALYZERS-LANG-11-005-DEPENDS-ON | TODO | Due 2025-11-22 · Accountable: StellaOps.Scanner EPDR Guild; QA Guild (src/Scanner/__Libraries/StellaOps.Scanner.Analyzers.Lang.DotNet) | StellaOps.Scanner EPDR Guild; QA Guild (src/Scanner/__Libraries/StellaOps.Scanner.Analyzers.Lang.DotNet) | Depends on 11-004; fixtures deferred until analyzer outputs exist. <br><br> Document artefact/deliverable for SCANNER-ANALYZERS-LANG-11-005 and publish location so downstream tasks can proceed. |
| P4 | PREP-SCANNER-ANALYZERS-NATIVE-20-002-AWAIT-DE | TODO | Due 2025-11-22 · Accountable: Native Analyzer Guild (src/Scanner/StellaOps.Scanner.Analyzers.Native) | Native Analyzer Guild (src/Scanner/StellaOps.Scanner.Analyzers.Native) | Await declared-dependency writer/contract to emit edges. <br><br> Document artefact/deliverable for SCANNER-ANALYZERS-NATIVE-20-002 and publish location so downstream tasks can proceed. |
| P5 | PREP-SCANNER-ANALYZERS-NODE-22-001-NEEDS-ISOL | TODO | Due 2025-11-22 · Accountable: Node Analyzer Guild (src/Scanner/__Libraries/StellaOps.Scanner.Analyzers.Lang.Node) | Node Analyzer Guild (src/Scanner/__Libraries/StellaOps.Scanner.Analyzers.Lang.Node) | Needs isolated runner or scoped build graph to execute targeted tests without full-solution fan-out. <br><br> Document artefact/deliverable for SCANNER-ANALYZERS-NODE-22-001 and publish location so downstream tasks can proceed. |
| P1 | PREP-SCANNER-ANALYZERS-LANG-11-003-DEPENDS-ON | DONE (2025-11-20) | Due 2025-11-22 · Accountable: StellaOps.Scanner EPDR Guild; Signals Guild (src/Scanner/__Libraries/StellaOps.Scanner.Analyzers.Lang.DotNet) | StellaOps.Scanner EPDR Guild; Signals Guild (src/Scanner/__Libraries/StellaOps.Scanner.Analyzers.Lang.DotNet) | Depends on 11-002; blocked until upstream static analyzer available. <br><br> Document artefact/deliverable for SCANNER-ANALYZERS-LANG-11-003 and publish location so downstream tasks can proceed. Prep artefact: `docs/modules/scanner/prep/2025-11-20-analyzers-prep.md` (runtime evidence ingest).
| P2 | PREP-SCANNER-ANALYZERS-LANG-11-004-DEPENDS-ON | DONE (2025-11-20) | Due 2025-11-22 · Accountable: StellaOps.Scanner EPDR Guild; SBOM Service Guild (src/Scanner/__Libraries/StellaOps.Scanner.Analyzers.Lang.DotNet) | StellaOps.Scanner EPDR Guild; SBOM Service Guild (src/Scanner/__Libraries/StellaOps.Scanner.Analyzers.Lang.DotNet) | Depends on 11-003; no upstream static/runtime outputs yet. <br><br> Document artefact/deliverable for SCANNER-ANALYZERS-LANG-11-004 and publish location so downstream tasks can proceed. Prep artefact: `docs/modules/scanner/prep/2025-11-20-analyzers-prep.md` (normalized export contract).
| P3 | PREP-SCANNER-ANALYZERS-LANG-11-005-DEPENDS-ON | DONE (2025-11-20) | Due 2025-11-22 · Accountable: StellaOps.Scanner EPDR Guild; QA Guild (src/Scanner/__Libraries/StellaOps.Scanner.Analyzers.Lang.DotNet) | StellaOps.Scanner EPDR Guild; QA Guild (src/Scanner/__Libraries/StellaOps.Scanner.Analyzers.Lang.DotNet) | Depends on 11-004; fixtures deferred until analyzer outputs exist. <br><br> Document artefact/deliverable for SCANNER-ANALYZERS-LANG-11-005 and publish location so downstream tasks can proceed. Prep artefact: `docs/modules/scanner/prep/2025-11-20-analyzers-prep.md` (fixtures/benchmarks expectations).
| P4 | PREP-SCANNER-ANALYZERS-NATIVE-20-002-AWAIT-DE | DONE (2025-11-20) | Due 2025-11-22 · Accountable: Native Analyzer Guild (src/Scanner/StellaOps.Scanner.Analyzers.Native) | Native Analyzer Guild (src/Scanner/StellaOps.Scanner.Analyzers.Native) | Await declared-dependency writer/contract to emit edges. <br><br> Document artefact/deliverable for SCANNER-ANALYZERS-NATIVE-20-002 and publish location so downstream tasks can proceed. Prep artefact: `docs/modules/scanner/prep/2025-11-20-analyzers-prep.md` (ELF declared-dependency writer payload).
| P5 | PREP-SCANNER-ANALYZERS-NODE-22-001-NEEDS-ISOL | DONE (2025-11-20) | Due 2025-11-22 · Accountable: Node Analyzer Guild (src/Scanner/__Libraries/StellaOps.Scanner.Analyzers.Lang.Node) | Node Analyzer Guild (src/Scanner/__Libraries/StellaOps.Scanner.Analyzers.Lang.Node) | Isolated runner plan published at `docs/modules/scanner/prep/2025-11-20-node-isolated-runner.md`; downstream implementation can proceed. Scripts: `src/Scanner/StellaOps.Scanner.Node.slnf`, `src/Scanner/__Tests/node-isolated.runsettings`, `src/Scanner/__Tests/node-tests-isolated.sh`. |
| 1 | SCANNER-ANALYZERS-LANG-11-002 | BLOCKED | Await upstream SCANNER-ANALYZERS-LANG-11-001 design/outputs to extend static analyzer | StellaOps.Scanner EPDR Guild (src/Scanner/__Libraries/StellaOps.Scanner.Analyzers.Lang.DotNet) | Implement static analyzer (IL + reflection heuristics) capturing AssemblyRef, ModuleRef/PInvoke, DynamicDependency, reflection literals, DI patterns, and custom AssemblyLoadContext probing hints; emit dependency edges with reason codes and confidence. |
| 2 | SCANNER-ANALYZERS-LANG-11-003 | BLOCKED | PREP-SCANNER-ANALYZERS-LANG-11-003-DEPENDS-ON | StellaOps.Scanner EPDR Guild; Signals Guild (src/Scanner/__Libraries/StellaOps.Scanner.Analyzers.Lang.DotNet) | Ingest optional runtime evidence (AssemblyLoad, Resolving, P/Invoke) via event listener harness; merge runtime edges with static/declared ones and attach reason codes/confidence. |
| 3 | SCANNER-ANALYZERS-LANG-11-004 | BLOCKED | PREP-SCANNER-ANALYZERS-LANG-11-004-DEPENDS-ON | StellaOps.Scanner EPDR Guild; SBOM Service Guild (src/Scanner/__Libraries/StellaOps.Scanner.Analyzers.Lang.DotNet) | Produce normalized observation export to Scanner writer: entrypoints + dependency edges + environment profiles (AOC compliant); wire to SBOM service entrypoint tagging. |
@@ -49,8 +49,25 @@
| 20 | AGENTS-SCANNER-00-001 | DONE | Create module-level AGENTS.md for `src/Scanner` aligned with scanner architecture docs | Project Management; Scanner Guild | Author/update Scanner AGENTS.md covering roles, required docs, allowed shared directories, determinism/testing rules; ensure implementers can work autonomously. |
## Execution Log
| Date (UTC) | Update | Owner |
| --- | --- | --- |
| 2025-11-20 | Attempted node isolated restore/test; restore failed fetching Microsoft.TestPlatform.TestHost (nuget.org) because offline package path was wrong. Script corrected to use `offline/packages`. Re-run still needed. | Implementer |
| 2025-11-20 | Second isolated restore attempt ran ~48s then cancelled; still needs seeding `Microsoft.TestPlatform.TestHost 17.14.1` into offline/packages to complete. | Implementer |
| 2025-11-20 | Isolated restore retried after seeding TestHost; still failing due to missing packages from offline cache (e.g., MongoDB.Driver.Core 2.12.0). Further seeding needed before tests can run. | Implementer |
| 2025-11-20 | Third restore attempt after seeding MongoDB.Driver.Core also failed (restore canceled ~26s); more dependencies still missing from offline cache. | Implementer |
| 2025-11-20 | Fourth restore attempt (~15s) still canceled; more NuGet packages remain unseeded in offline cache. | Implementer |
| 2025-11-20 | Fifth restore attempt (quiet, ~15s) still cancelled; offline cache still missing transitive NuGet packages; no tests run. | Implementer |
| 2025-11-20 | Sixth restore attempt failed with missing packages (example: NU1101 StellaOps.Policy.AuthSignals) even with parallel restore; offline cache still incomplete. | Implementer |
| 2025-11-20 | Completed PREP-SCANNER-ANALYZERS-NODE-22-001; published isolated runner plan at `docs/modules/scanner/prep/2025-11-20-node-isolated-runner.md`. | Implementer |
| 2025-11-20 | Added isolated runner artefacts: `StellaOps.Scanner.Node.slnf`, `__Tests/node-isolated.runsettings`, `__Tests/node-tests-isolated.sh` to support P5 plan. | Implementer |
| 2025-11-20 | Completed PREP-SCANNER-ANALYZERS-LANG-11-003/004/005 and PREP-SCANNER-ANALYZERS-NATIVE-20-002; prep note published at `docs/modules/scanner/prep/2025-11-20-analyzers-prep.md`. | Implementer |
| 2025-11-20 | Set PREP-SCANNER-ANALYZERS-NODE-22-001 to DOING after confirming no other owners; prep still pending (isolated runner requirement). | Project Mgmt |
| 2025-11-20 | Started PREP-SCANNER-ANALYZERS-LANG-11-003, PREP-SCANNER-ANALYZERS-LANG-11-004, and PREP-SCANNER-ANALYZERS-NATIVE-20-002 (all were TODO; skipped PREP-SCANNER-ANALYZERS-LANG-11-005 because already DOING). | Implementer |
| 2025-11-20 | Started PREP-SCANNER-ANALYZERS-NODE-22-001 (was TODO; verified no prior start). | Project Mgmt |
| 2025-11-20 | Moved PREP-SCANNER-ANALYZERS-LANG-11-005-DEPENDS-ON to DOING after confirming no prior start; beginning prep to unblock downstream analyzer fixtures. | Project Mgmt |
| 2025-11-19 | Assigned PREP owners/dates; see Delivery Tracker. | Planning |
| 2025-11-16 | Normalised sprint file to standard template; renamed from `SPRINT_132_scanner_surface.md` to `SPRINT_0132_scanner_surface.md`; scope unchanged; added governance task for missing Scanner AGENTS.md. | Planning |
| 2025-11-17 | AGENTS-SCANNER-00-001 completed; module AGENTS.md added under src/Scanner. | Implementer |
@@ -72,13 +89,12 @@
## Decisions & Risks
- Scanner AGENTS.md added 2025-11-17; keep in sync with scanner architecture and future advisories.
- Sprint execution gated on completion of Sprint 131; monitor for slippage to avoid cascading delays in 130–139 chain.
- Maintain offline-first and deterministic outputs for analyzers; ensure runtime capture adapters include redaction/sandbox guidance before rollout.
- Native analyzer format-detector completed and tested; NAT-20-002 remains blocked awaiting declared-dependency writer/contract and availability of declared dependency export path.
- Node analyzer version-target/tarball/Yarn PnP tests pending; multiple targeted runs (latest 2025-11-18) fanned out into full solution builds and were cancelled. Needs clean/isolated runner or scoped build graph to validate SCANNER-ANALYZERS-NODE-22-001 changes.
- .NET analyzer chain (11-002..005) blocked awaiting upstream static-analyzer contract (11-001) and downstream writer/export contracts; no safe implementation path until provided.
- Prep note for analyzer PREP tasks captured in `docs/modules/scanner/prep/2025-11-20-analyzers-prep.md`; use it as the interim contract until upstream writer/runtime contracts land.
- Native analyzer format-detector completed; NAT-20-002 still blocked on declared-dependency writer interface—prep note defines expected payload to reduce rework once contract lands.
- Node analyzer isolation plan published (see `docs/modules/scanner/prep/2025-11-20-node-isolated-runner.md`); offline cache still incomplete after multiple restore attempts (latest NU1101 StellaOps.Policy.AuthSignals). Need full dependency seed before isolated run and tests can pass.
- .NET analyzer chain (11-002..005) remains blocked awaiting upstream static-analyzer contract (11-001) and downstream writer/export contracts; runtime fusion prep recorded but cannot proceed until contracts exist.
## Next Checkpoints
- 2025-11-19: Sprint kickoff (owner: Scanner PM), contingent on Sprint 131 sign-off.
- 2025-11-26: Mid-sprint review (owner: EPDR Guild lead) to validate observation exports and resolver behavior.
| 2025-11-18 | SCANNER-ANALYZERS-NODE-22-001: Added Yarn PnP cache zip traversal, emitter sets yarnPnp metadata, new fixture/tests (`yarn-pnp`); test run aborted due to long-running solution build—rerun on clean runner. | Node Analyzer Guild |
| 2025-11-18 | SCANNER-ANALYZERS-NODE-22-001: Added Yarn PnP cache zip traversal, emitter sets yarnPnp metadata, new fixture/tests (`yarn-pnp`); test run aborted due to long-running solution build—rerun on clean runner. | Node Analyzer Guild |

81
docs/implplan/SPRINT_0133_0001_0001_scanner_surface.md
View File

@@ -1,43 +1,46 @@
# Sprint 0133-0001-0001 · Scanner & Surface (Phase IV)
## Topic & Scope
- Scanner & Surface phase IV: Node bundle/source-map coverage and native/WASM signal extraction.
- Maintain sequential execution across 130–139; work only after Sprint 0132 completes.
- **Working directory:** `src/Scanner`.
## Dependencies & Concurrency
- Upstream: Sprint 0132 (Scanner & Surface phase III) must land first.
- Concurrency: tasks execute in table order; all currently TODO.
## Documentation Prerequisites
- docs/README.md
- docs/07_HIGH_LEVEL_ARCHITECTURE.md
- docs/modules/platform/architecture-overview.md
- docs/modules/scanner/architecture.md
- src/Scanner/AGENTS.md
## Delivery Tracker
| # | Task ID | Status | Key dependency / next step | Owners | Task Definition |
| --- | --- | --- | --- | --- | --- |
| P1 | PREP-SCANNER-ANALYZERS-NODE-22-006-UPSTREAM-2 | TODO | Due 2025-11-22 · Accountable: Node Analyzer Guild (`src/Scanner/__Libraries/StellaOps.Scanner.Analyzers.Lang.Node`) | Node Analyzer Guild (`src/Scanner/__Libraries/StellaOps.Scanner.Analyzers.Lang.Node`) | Upstream 22-005 not delivered in Sprint 0132; waiting on bundle/source-map resolver baseline. <br><br> Document artefact/deliverable for SCANNER-ANALYZERS-NODE-22-006 and publish location so downstream tasks can proceed. |
| P2 | PREP-SCANNER-ANALYZERS-NODE-22-007-UPSTREAM-2 | TODO | Due 2025-11-22 · Accountable: Node Analyzer Guild | Node Analyzer Guild | Upstream 22-006 blocked. <br><br> Document artefact/deliverable for SCANNER-ANALYZERS-NODE-22-007 and publish location so downstream tasks can proceed. |
| P3 | PREP-SCANNER-ANALYZERS-NODE-22-008-UPSTREAM-2 | TODO | Due 2025-11-22 · Accountable: Node Analyzer Guild | Node Analyzer Guild | Upstream 22-007 blocked. <br><br> Document artefact/deliverable for SCANNER-ANALYZERS-NODE-22-008 and publish location so downstream tasks can proceed. |
| 1 | SCANNER-ANALYZERS-NODE-22-006 | BLOCKED (2025-11-20) | PREP-SCANNER-ANALYZERS-NODE-22-006-UPSTREAM-2 | Node Analyzer Guild (`src/Scanner/__Libraries/StellaOps.Scanner.Analyzers.Lang.Node`) | Detect bundles + source maps, reconstruct module specifiers, correlate to original paths; support dual CJS/ESM graphs with conditions. |
| 2 | SCANNER-ANALYZERS-NODE-22-007 | BLOCKED (2025-11-20) | PREP-SCANNER-ANALYZERS-NODE-22-007-UPSTREAM-2 | Node Analyzer Guild | Scan for native addons (.node), WASM modules, and core capability signals (child_process, vm, worker_threads); emit hint edges and native metadata. |
| 3 | SCANNER-ANALYZERS-NODE-22-008 | BLOCKED (2025-11-20) | PREP-SCANNER-ANALYZERS-NODE-22-008-UPSTREAM-2 | Node Analyzer Guild | Produce AOC-compliant observations: entrypoints, components (pkg/native/wasm), edges (esm-import, cjs-require, exports, json, native-addon, wasm, worker) with reason codes/confidence and resolver traces. |
## Execution Log
# Sprint 0133-0001-0001 · Scanner & Surface (Phase IV)
## Topic & Scope
- Scanner & Surface phase IV: Node bundle/source-map coverage and native/WASM signal extraction.
- Maintain sequential execution across 130–139; work only after Sprint 0132 completes.
- **Working directory:** `src/Scanner`.
## Dependencies & Concurrency
- Upstream: Sprint 0132 (Scanner & Surface phase III) must land first.
- Concurrency: tasks execute in table order; all currently TODO.
## Documentation Prerequisites
- docs/README.md
- docs/07_HIGH_LEVEL_ARCHITECTURE.md
- docs/modules/platform/architecture-overview.md
- docs/modules/scanner/architecture.md
- src/Scanner/AGENTS.md
## Delivery Tracker
| # | Task ID | Status | Key dependency / next step | Owners | Task Definition |
| --- | --- | --- | --- | --- | --- |
| P1 | PREP-SCANNER-ANALYZERS-NODE-22-006-UPSTREAM-2 | DONE (2025-11-20) | Due 2025-11-22 · Accountable: Node Analyzer Guild (`src/Scanner/__Libraries/StellaOps.Scanner.Analyzers.Lang.Node`) | Node Analyzer Guild (`src/Scanner/__Libraries/StellaOps.Scanner.Analyzers.Lang.Node`) | Bundle/source-map baseline documented in `docs/modules/scanner/design/node-bundle-phase22.md` with sample NDJSON `docs/samples/scanner/node-phase22/node-phase22-sample.ndjson`. |
| P2 | PREP-SCANNER-ANALYZERS-NODE-22-007-UPSTREAM-2 | DONE (2025-11-20) | Due 2025-11-22 · Accountable: Node Analyzer Guild | Node Analyzer Guild | Native/WASM/capability detection rules + reason codes documented in `docs/modules/scanner/design/node-bundle-phase22.md` with fixture referenced above. |
| P3 | PREP-SCANNER-ANALYZERS-NODE-22-008-UPSTREAM-2 | DONE (2025-11-20) | Due 2025-11-22 · Accountable: Node Analyzer Guild | Node Analyzer Guild | AOC-compliant observation emission shape + sorting rules documented in `docs/modules/scanner/design/node-bundle-phase22.md`; fixture referenced above. |
| 1 | SCANNER-ANALYZERS-NODE-22-006 | BLOCKED (2025-11-20) | PREP-SCANNER-ANALYZERS-NODE-22-006-UPSTREAM-2 | Node Analyzer Guild (`src/Scanner/__Libraries/StellaOps.Scanner.Analyzers.Lang.Node`) | Detect bundles + source maps, reconstruct module specifiers, correlate to original paths; support dual CJS/ESM graphs with conditions. |
| 2 | SCANNER-ANALYZERS-NODE-22-007 | BLOCKED (2025-11-20) | PREP-SCANNER-ANALYZERS-NODE-22-007-UPSTREAM-2 | Node Analyzer Guild | Scan for native addons (.node), WASM modules, and core capability signals (child_process, vm, worker_threads); emit hint edges and native metadata. |
| 3 | SCANNER-ANALYZERS-NODE-22-008 | BLOCKED (2025-11-20) | PREP-SCANNER-ANALYZERS-NODE-22-008-UPSTREAM-2 | Node Analyzer Guild | Produce AOC-compliant observations: entrypoints, components (pkg/native/wasm), edges (esm-import, cjs-require, exports, json, native-addon, wasm, worker) with reason codes/confidence and resolver traces. |
## Execution Log
| Date (UTC) | Update | Owner |
| --- | --- | --- |
| 2025-11-20 | Added Node phase 22 NDJSON loader hook + fixture to analyzer; PREP P1–P3 now have executable baseline for downstream tasks. | Implementer |
| 2025-11-20 | Published Node phase 22 prep doc + fixture (see Delivery Tracker) and marked PREP P1–P3 DONE. | Planning |
| 2025-11-20 | Started PREP-SCANNER-ANALYZERS-NODE-22-006/007/008 (statuses → DOING) after confirming no prior DOING owner entries. | Planning |
| 2025-11-19 | Assigned PREP owners/dates; see Delivery Tracker. | Planning |
| 2025-11-08 | Sprint stub created; awaiting upstream completion of Sprint 0132. | Planning |
| 2025-11-19 | Normalized sprint to standard template and renamed from `SPRINT_133_scanner_surface.md` to `SPRINT_0133_0001_0001_scanner_surface.md`; content preserved. | Implementer |
| 2025-11-19 | Converted legacy filename `SPRINT_133_scanner_surface.md` to redirect stub pointing here to avoid divergent updates. | Implementer |
| 2025-11-20 | Marked Node phase tasks 22-006/007/008 BLOCKED because upstream 22-005 (Sprint 0132) not delivered; no executable work in this sprint until 0132 unblocks. | Implementer |
## Decisions & Risks
- All tasks depend on 22-005 outputs; remain TODO until prerequisites land.
| 2025-11-08 | Sprint stub created; awaiting upstream completion of Sprint 0132. | Planning |
| 2025-11-19 | Normalized sprint to standard template and renamed from `SPRINT_133_scanner_surface.md` to `SPRINT_0133_0001_0001_scanner_surface.md`; content preserved. | Implementer |
| 2025-11-19 | Converted legacy filename `SPRINT_133_scanner_surface.md` to redirect stub pointing here to avoid divergent updates. | Implementer |
| 2025-11-20 | Marked Node phase tasks 22-006/007/008 BLOCKED because upstream 22-005 (Sprint 0132) not delivered; no executable work in this sprint until 0132 unblocks. | Implementer |
## Decisions & Risks
- All tasks depend on 22-005 outputs; remain TODO until prerequisites land, but analyzer contracts are frozen in `docs/modules/scanner/design/node-bundle-phase22.md` and sample NDJSON is wired into analyzer/tests for deterministic baselines.
- Maintain offline/deterministic outputs; avoid running full solution builds—prefer scoped runners per module.
## Next Checkpoints
- Set kickoff once Sprint 0132 completes (date TBD).
## Next Checkpoints
- Set kickoff once Sprint 0132 completes (date TBD).