stella-ops.org/git.stella-ops.org
1
0
Fork 0
Code Issues Pull Requests Actions 16 Releases Wiki Activity

feat: Enhance MongoDB storage with event publishing and outbox support

Browse Source 
- Added `MongoAdvisoryObservationEventPublisher` and `NatsAdvisoryObservationEventPublisher` for event publishing.
- Registered `IAdvisoryObservationEventPublisher` to choose between NATS and MongoDB based on configuration.
- Introduced `MongoAdvisoryObservationEventOutbox` for outbox pattern implementation.
- Updated service collection to include new event publishers and outbox.
- Added a new hosted service `AdvisoryObservationTransportWorker` for processing events.

feat: Update project dependencies

- Added `NATS.Client.Core` package to the project for NATS integration.

test: Add unit tests for AdvisoryLinkset normalization

- Created `AdvisoryLinksetNormalizationConfidenceTests` to validate confidence score calculations.

fix: Adjust confidence assertion in `AdvisoryObservationAggregationTests`

- Updated confidence assertion to allow a range instead of a fixed value.

test: Implement tests for AdvisoryObservationEventFactory

- Added `AdvisoryObservationEventFactoryTests` to ensure correct mapping and hashing of observation events.

chore: Configure test project for Findings Ledger

- Created `Directory.Build.props` for test project configuration.
- Added `StellaOps.Findings.Ledger.Exports.Unit.csproj` for unit tests related to findings ledger exports.

feat: Implement export contracts for findings ledger

- Defined export request and response contracts in `ExportContracts.cs`.
- Created various export item records for findings, VEX, advisories, and SBOMs.

feat: Add export functionality to Findings Ledger Web Service

- Implemented endpoints for exporting findings, VEX, advisories, and SBOMs.
- Integrated `ExportQueryService` for handling export logic and pagination.

test: Add tests for Node language analyzer phase 22

- Implemented `NodePhase22SampleLoaderTests` to validate loading of NDJSON fixtures.
- Created sample NDJSON file for testing.

chore: Set up isolated test environment for Node tests

- Added `node-isolated.runsettings` for isolated test execution.
- Created `node-tests-isolated.sh` script for running tests in isolation.
This commit is contained in:
master
2025-11-20 23:08:45 +02:00
parent f0e74d2ee8
commit 2e276d6676
49 changed files with 1996 additions and 113 deletions
Download Patch File Download Diff File Expand all files Collapse all files

50
docs/modules/scanner/design/node-bundle-phase22.md Normal file
View File

@@ -0,0 +1,50 @@
# Scanner Node Phase 22 (22-006/007/008) · Prep deliverable
Purpose: unblock PREP tasks by freezing analyzer inputs/outputs, resolver traces, and fixtures for Node bundle/source-map coverage, native/WASM detection, and AOC-compliant observation emission.
## Output artefacts
- Sample NDJSON: `docs/samples/scanner/node-phase22/node-phase22-sample.ndjson` (covers 22-006/007/008 in one run).
- Resolver trace spec and reason codes (below) are binding for workers and tests.
## 22-006 · Bundle + source-map reconstruction
- Detect bundles by `sourceMappingURL` trailers and common bundle signatures (webpack runtime, rollup intro, esbuild banners).
- Load `.map` file (inline/base64 or adjacent file); reject maps >50 MB or with missing `sourcesContent`.
- Resolver trace must include: `["bundle:<path>", "map:<path|inline>", "source:<original-path>"]`.
- Recovered module specifier shape: `{ "type": "component", "componentType": "pkg", "path": "<relative path>", "format": "esm|cjs", "fromBundle": true, "confidence": 0.8+ }`.
- Normalize paths to POSIX, strip inline `webpack://` prefixes, collapse duplicated `..` segments, and dedupe.
## 22-007 · Native addon / WASM / capability signals
- Native addons: detect `.node` files (ELF/PE/Mach-O) and `process.dlopen` calls. Emit `componentType:"native"` with `arch`, `platform`, and optional `soname`.
- WASM: detect `.wasm` files and dynamic imports (`WebAssembly.instantiate*`). Emit `componentType:"wasm"` with `exports` (function names if discoverable).
- Capability signals: AST scan for `child_process`, `vm`, `worker_threads`, `process.binding`, `fs.promises` `openFileHandle`.
- Reason codes (stable strings):
- `native-dlopen-string`, `native-dlopen-template`, `native-addon-file`,
- `wasm-import`, `wasm-file`,
- `capability-child-process`, `capability-vm`, `capability-worker`, `capability-binding`, `capability-fs-promises`.
- Hint edges: `{ "type":"edge", "edgeType":"native-addon|wasm|capability", "from":"<module>", "to":"<artifact>", "reason":"<code>", "confidence": 0.6–0.9 }`.
## 22-008 · AOC-compliant observation emission
- Emit NDJSON records grouped by `entrypoints`, `components`, `edges`, `resolverTrace` per record.
- Required fields: `type`, `from`, `to` (for edges), `componentType`, `format`, `reason`, `confidence`, `resolverTrace` (array, stable ordered). Optional `scopes` (`runtime|dev|optional`) when derived from package.json sections.
- Determinism:
- Sort output by `type` then `path/from/to` strings; stable sort within edges by `edgeType`.
- Timestamps forbidden; no filesystem mtime or random IDs.
- All paths POSIX, absolute paths stripped to container root.
## Validation gates
- Reject payloads if resolverTrace missing, empty, or unsorted.
- Reject entries when `confidence < 0.4` to avoid noise; keep suppressed entries in debug logs only.
- Large maps: emit `ERR_NODE_BUNDLE_MAP_TOO_LARGE` and skip map (still report bundle presence with `confidence:0.51`).
## Fixtures
- `docs/samples/scanner/node-phase22/node-phase22-sample.ndjson` contains:
1) webpack bundle w/ source map mapping to `/src/app.js` (22-006)
2) native addon load via `process.dlopen('./native/addon.node')` (22-007)
3) WASM module import via `WebAssembly.instantiateStreaming(fetch('./pkg.wasm'))` (22-007)
4) capability signal for `child_process.execFile` (22-007)
5) consolidated edges + components emitted per AOC rules (22-008)
## Implementation notes
- Keep analyser pure: no execution of JS; rely on parse + static string matching. Use cached sourcemap parser in `StellaOps.Scanner.Analyzers.Lang.Node`.
- Resolver trace must be included verbatim in tests; any change requires fixture update and sprint note.
- Workers must mark node phase 22 outputs as experimental behind feature flag `scanner:node:phase22=true` defaulting true for CI only until stabilized.

32
docs/modules/scanner/prep/2025-11-20-node-isolated-runner.md Normal file
View File

@@ -0,0 +1,32 @@
# Scanner PREP — Node Analyzer Isolated Runner (22-001)
Date: 2025-11-20
Owner: Node Analyzer Guild
Scope: Requirements and plan to provide an isolated/scoped runner so targeted Node analyzer tests complete without whole-solution fan‑out.
## Goals
- Enable `StellaOps.Scanner.Analyzers.Lang.Node.Tests` to run deterministically without restoring/building the entire solution.
- Reduce CI/local runtime (<5 min) and contention during restores.
- Keep offline/air-gap posture: rely only on `local-nugets/` + repo fixtures; no external fetches.
## Proposed approach
1) **Scoped solution file**
- Create `src/Scanner/StellaOps.Scanner.Analyzers.Node.slnf` including only Node analyzer projects + their direct deps (`StellaOps.Scanner.Analyzers.Lang.Node`, tests, shared test utilities).
- CI job uses `dotnet test --solution St...Node.slnf --no-restore --no-build` after targeted restore.
2) **Isolated restore cache**
- Pre-populate `local-nugets/` via existing offline feed; add msbuild property `RestorePackagesPath=$(RepoRoot)/offline/packages` to avoid global cache churn.
3) **Test shim**
- Add runsettings to disable collectors that trigger solution-wide discovery; set `RunConfiguration.DisableAppDomain=true` for determinism.
4) **Tarball/pnpm/Yarn PnP fixtures**
- Move heavy fixtures under `src/Scanner/__Tests/Fixtures/node/` and reference via deterministic VFS layer; no temp extraction outside repo.
5) **Entry point**
- New `scripts/scanner/node-tests-isolated.sh` wrapper: restore scoped solution, then run `dotnet test` with `/m:1` and explicit test filters as needed.
## Deliverables for implementation task
- Add `.slnf`, runsettings, and wrapper script as above.
- Update CI pipeline (Scanner) to include isolated target; keep full solution tests separate.
- Document usage in `src/Scanner/__Tests/README.md`.
## Blocking items
- None identified; all inputs are local to the repo/offline feeds.
This note satisfies PREP-SCANNER-ANALYZERS-NODE-22-001-NEEDS-ISOL by defining the isolated runner plan and artefact locations.