feat: Enhance MongoDB storage with event publishing and outbox support

- Added `MongoAdvisoryObservationEventPublisher` and `NatsAdvisoryObservationEventPublisher` for event publishing.
- Registered `IAdvisoryObservationEventPublisher` to choose between NATS and MongoDB based on configuration.
- Introduced `MongoAdvisoryObservationEventOutbox` for outbox pattern implementation.
- Updated service collection to include new event publishers and outbox.
- Added a new hosted service `AdvisoryObservationTransportWorker` for processing events.

feat: Update project dependencies

- Added `NATS.Client.Core` package to the project for NATS integration.

test: Add unit tests for AdvisoryLinkset normalization

- Created `AdvisoryLinksetNormalizationConfidenceTests` to validate confidence score calculations.

fix: Adjust confidence assertion in `AdvisoryObservationAggregationTests`

- Updated confidence assertion to allow a range instead of a fixed value.

test: Implement tests for AdvisoryObservationEventFactory

- Added `AdvisoryObservationEventFactoryTests` to ensure correct mapping and hashing of observation events.

chore: Configure test project for Findings Ledger

- Created `Directory.Build.props` for test project configuration.
- Added `StellaOps.Findings.Ledger.Exports.Unit.csproj` for unit tests related to findings ledger exports.

feat: Implement export contracts for findings ledger

- Defined export request and response contracts in `ExportContracts.cs`.
- Created various export item records for findings, VEX, advisories, and SBOMs.

feat: Add export functionality to Findings Ledger Web Service

- Implemented endpoints for exporting findings, VEX, advisories, and SBOMs.
- Integrated `ExportQueryService` for handling export logic and pagination.

test: Add tests for Node language analyzer phase 22

- Implemented `NodePhase22SampleLoaderTests` to validate loading of NDJSON fixtures.
- Created sample NDJSON file for testing.

chore: Set up isolated test environment for Node tests

- Added `node-isolated.runsettings` for isolated test execution.
- Created `node-tests-isolated.sh` script for running tests in isolation.
This commit is contained in:
master
2025-11-20 23:08:45 +02:00
parent f0e74d2ee8
commit 2e276d6676
49 changed files with 1996 additions and 113 deletions

View File

@@ -0,0 +1,50 @@
# Scanner Node Phase 22 (22-006/007/008) · Prep deliverable
Purpose: unblock PREP tasks by freezing analyzer inputs/outputs, resolver traces, and fixtures for Node bundle/source-map coverage, native/WASM detection, and AOC-compliant observation emission.
## Output artefacts
- Sample NDJSON: `docs/samples/scanner/node-phase22/node-phase22-sample.ndjson` (covers 22-006/007/008 in one run).
- Resolver trace spec and reason codes (below) are binding for workers and tests.
## 22-006 · Bundle + source-map reconstruction
- Detect bundles by `sourceMappingURL` trailers and common bundle signatures (webpack runtime, rollup intro, esbuild banners).
- Load `.map` file (inline/base64 or adjacent file); reject maps >50 MB or with missing `sourcesContent`.
- Resolver trace must include: `["bundle:<path>", "map:<path|inline>", "source:<original-path>"]`.
- Recovered module specifier shape: `{ "type": "component", "componentType": "pkg", "path": "<relative path>", "format": "esm|cjs", "fromBundle": true, "confidence": 0.8+ }`.
- Normalize paths to POSIX, strip inline `webpack://` prefixes, collapse duplicated `..` segments, and dedupe.
## 22-007 · Native addon / WASM / capability signals
- Native addons: detect `.node` files (ELF/PE/Mach-O) and `process.dlopen` calls. Emit `componentType:"native"` with `arch`, `platform`, and optional `soname`.
- WASM: detect `.wasm` files and dynamic imports (`WebAssembly.instantiate*`). Emit `componentType:"wasm"` with `exports` (function names if discoverable).
- Capability signals: AST scan for `child_process`, `vm`, `worker_threads`, `process.binding`, `fs.promises` `openFileHandle`.
- Reason codes (stable strings):
- `native-dlopen-string`, `native-dlopen-template`, `native-addon-file`,
- `wasm-import`, `wasm-file`,
- `capability-child-process`, `capability-vm`, `capability-worker`, `capability-binding`, `capability-fs-promises`.
- Hint edges: `{ "type":"edge", "edgeType":"native-addon|wasm|capability", "from":"<module>", "to":"<artifact>", "reason":"<code>", "confidence": 0.60.9 }`.
## 22-008 · AOC-compliant observation emission
- Emit NDJSON records grouped by `entrypoints`, `components`, `edges`, `resolverTrace` per record.
- Required fields: `type`, `from`, `to` (for edges), `componentType`, `format`, `reason`, `confidence`, `resolverTrace` (array, stable ordered). Optional `scopes` (`runtime|dev|optional`) when derived from package.json sections.
- Determinism:
- Sort output by `type` then `path/from/to` strings; stable sort within edges by `edgeType`.
- Timestamps forbidden; no filesystem mtime or random IDs.
- All paths POSIX, absolute paths stripped to container root.
## Validation gates
- Reject payloads if resolverTrace missing, empty, or unsorted.
- Reject entries when `confidence < 0.4` to avoid noise; keep suppressed entries in debug logs only.
- Large maps: emit `ERR_NODE_BUNDLE_MAP_TOO_LARGE` and skip map (still report bundle presence with `confidence:0.51`).
## Fixtures
- `docs/samples/scanner/node-phase22/node-phase22-sample.ndjson` contains:
1) webpack bundle w/ source map mapping to `/src/app.js` (22-006)
2) native addon load via `process.dlopen('./native/addon.node')` (22-007)
3) WASM module import via `WebAssembly.instantiateStreaming(fetch('./pkg.wasm'))` (22-007)
4) capability signal for `child_process.execFile` (22-007)
5) consolidated edges + components emitted per AOC rules (22-008)
## Implementation notes
- Keep analyser pure: no execution of JS; rely on parse + static string matching. Use cached sourcemap parser in `StellaOps.Scanner.Analyzers.Lang.Node`.
- Resolver trace must be included verbatim in tests; any change requires fixture update and sprint note.
- Workers must mark node phase 22 outputs as experimental behind feature flag `scanner:node:phase22=true` defaulting true for CI only until stabilized.