Add new features and tests for AirGap and Time modules

- Introduced `SbomService` tasks documentation. - Updated `StellaOps.sln` to include new projects: `StellaOps.AirGap.Time` and `StellaOps.AirGap.Importer`. - Added unit tests for `BundleImportPlanner`, `DsseVerifier`, `ImportValidator`, and other components in the `StellaOps.AirGap.Importer.Tests` namespace. - Implemented `InMemoryBundleRepositories` for testing bundle catalog and item repositories. - Created `MerkleRootCalculator`, `RootRotationPolicy`, and `TufMetadataValidator` tests. - Developed `StalenessCalculator` and `TimeAnchorLoader` tests in the `StellaOps.AirGap.Time.Tests` namespace. - Added `fetch-sbomservice-deps.sh` script for offline dependency fetching.
2025-11-20 23:29:54 +02:00
parent 65b1599229
commit 79b8e53441
182 changed files with 6660 additions and 1242 deletions
--- a/docs/modules/scanner/design/analyzer-prep-0132.md
+++ b/docs/modules/scanner/design/analyzer-prep-0132.md
@@ -0,0 +1,118 @@
+# Scanner Analyzer Prep · Sprint 0132
+
+This note captures the unblockers promised in PREP tasks for Sprint 0132. Each subsection gives the artifact location, assumption set, and the handoff needed by downstream implementation tasks.
+
+## SCANNER-ANALYZERS-LANG-11-003 (runtime fusion)
+- **Objective:** Define the runtime evidence ingest contract to merge AssemblyLoad/Resolving/PInvoke signals with static edges from 11-002.
+- **Inputs required:**
+  - Static edge export format from 11-002 (AssemblyRef/ModuleRef/PInvoke with reason codes).
+  - Event listener tap points: `AssemblyLoadContext.Resolving`, `AssemblyLoad`, `NativeLibrary.SetDllImportResolver`, `DynamicDependency` attributes, and optional ETW provider `Microsoft-Windows-DotNETRuntime` (keyword 0x8, task AssemblyLoad).
+- **Runtime evidence envelope (AOC-aligned):**
+  ```json
+  {
+    "runtime_observation_id": "uuid",
+    "assembly_name": "System.Text.Json",
+    "kind": "assembly-load|p-invoke|dynamic-dependency",
+    "source": "Resolving|AssemblyLoad|NativeLibrary|ETW",
+    "details": {
+      "requested_name": "System.Text.Json",
+      "resolved_path": "<normalized absolute path>",
+      "assembly_version": "8.0.0.0",
+      "culture": "neutral",
+      "package_purl": "pkg:nuget/system.text.json@8.0.0",
+      "confidence": 0.72,
+      "reason_code": "runtime-resolve"
+    },
+    "timestamp_utc": "2025-11-20T00:00:00Z"
+  }
+  ```
+- **Merge rules for downstream 11-003 implementation:**
+  - De-dup edges by (assembly_name, resolved_path, kind).
+  - Prefer static edge confidence when present; runtime adds `confidence_bonus = +0.1` but never exceeds 1.0.
+  - Keep provenance: `edge.provenance = { "static": bool, "runtime": bool }`.
+- **Publication:** This doc section is the frozen location for the runtime ingest contract; downstream tasks should reference this path.
+
+## SCANNER-ANALYZERS-LANG-11-004 (observation export → writer/SBOM)
+- **Objective:** Define the observation payload emitted to Scanner writer and SBOM entrypoint tagging.
+- **Export envelope (AOC-compliant):**
+  ```json
+  {
+    "entrypoints": [
+      {
+        "label": "app",
+        "rids": ["win-x64","linux-x64"],
+        "tfms": ["net8.0","net8.0-windows"],
+        "command": "dotnet ./bin/app.dll",
+        "sources": ["src/App/Program.cs"],
+        "rank": 1
+      }
+    ],
+    "dependency_edges": [
+      {
+        "from": "app",
+        "to": "pkg:nuget/system.text.json@8.0.0",
+        "reason_code": "assembly-ref",
+        "confidence": 0.86,
+        "provenance": {"static": true, "runtime": false}
+      }
+    ],
+    "environment_profiles": {
+      "tfm": "net8.0",
+      "rid": "linux-x64",
+      "host_policy": "portable",
+      "features": ["singlefile:false","trimmed:false","nativeaot:false"]
+    }
+  }
+  ```
+- **Writer handoff:**
+  - Serialize as deterministic JSON (sorted keys) to the Scanner writer contract `writer/observations/lang/dotnet`.
+  - Attach `sbom_entrypoint_tags` derived from entrypoint labels to feed SBOM Service tagging.
+- **Publication:** Payload shape and field meanings fixed here for Sprint 0132 downstream work.
+
+## SCANNER-ANALYZERS-LANG-11-005 (fixtures & benchmarks)
+- **Objective:** Provide fixture plan so QA can start without waiting on further design.
+- **Fixture matrix:**
+  - Framework-dependent: `net8.0`, `net9.0-preview` sample apps (console + web minimal API).
+  - Self-contained: `linux-x64` trimmed vs non-trimmed.
+  - Single-file: `win-x64` single-file publish, include native hosting bundle.
+  - NativeAOT: `linux-x64` HelloWorld + P/Invoke stub.
+  - Multi-RID: RID graph `linux-x64`, `linux-arm64`, `win-x64` with RID fallback expectations.
+- **Locations:** place fixtures under `src/Scanner/__Tests/Fixtures/DotNet/11-005/*`; store expected observation JSON in `__Tests/Fixtures/DotNet/11-005/expected/*.json` with sorted keys.
+- **Bench envelopes:**
+  - Target <150 ms p95 per project scan on dev laptop, <25 MB heap delta; capture via BenchmarkDotNet and report to `__Benchmarks/11-005.md`.
+- **Determinism:** lock timestamps to `1970-01-01T00:00:00Z` in serialized outputs; stable ordering by (entrypoint label, dependency to PURL, reason_code).
+
+## SCANNER-ANALYZERS-NATIVE-20-002 (ELF declared-dependency writer contract)
+- **Objective:** Unblock writer schema so native analyzer can emit DT_NEEDED/DT_RPATH/DT_RUNPATH data.
+- **Edge record (per ELF binary):**
+  ```json
+  {
+    "image": "libssl.so.3",
+    "build_id": "cafef00d",
+    "rpath": ["$ORIGIN/lib","/usr/lib"],
+    "runpath": ["$ORIGIN","/opt/openssl"],
+    "needed": [
+      {"name": "libcrypto.so.3", "slot": 0, "version": "OPENSSL_3.0", "reason_code": "elf-dtneeded"},
+      {"name": "libpthread.so.0", "slot": 1, "version": null, "reason_code": "elf-dtneeded"}
+    ],
+    "interpreter": "/lib64/ld-linux-x86-64.so.2",
+    "origin": "virtual-fs",
+    "confidence": 0.82
+  }
+  ```
+- **Writer path:** `writer/observations/native/elf-declared-deps` (append-only NDJSON; sorted by image name then slot).
+- **Redaction:** no host absolute paths; resolve `$ORIGIN` using virtual image root only.
+- **Publication:** schema above is the agreed baseline for downstream tasks; time-boxed to Sprint 0132.
+
+## SCANNER-ANALYZERS-NODE-22-001 (isolated runner / scoped build graph)
+- **Objective:** Provide a deterministic way to run Node analyzer tests without fanning out the whole solution.
+- **Approach:**
+  - Add target solution filter: `src/Scanner/StellaOps.Scanner.Analyzers.Lang.Node.slnf` including only Node projects + shared test utilities.
+  - Introduce `Directory.Build.props` override for `Lang.Node` tests to disable cross-solution restore (`DisableTransitiveProjectReferences=true`).
+  - Test command for CI + local: `dotnet test src/Scanner/StellaOps.Scanner.Analyzers.Lang.Node.Tests/StellaOps.Scanner.Analyzers.Lang.Node.Tests.csproj /p:DisableTransitiveProjectReferences=true --no-restore --logger:"console;verbosity=minimal"`.
+  - Cache seeds: copy pnpm/Yarn fixtures into `obj/fixtures-cache` during test init; deterministic zip timestamps set to `1980-01-01`.
+- **Publication:** This runbook unblocks execution while broader solution build contention is resolved; downstream tasks should adopt this invocation until Sprint 131 completes.
+
+---
+**Owners:** Scanner EPDR Guild (DotNet), SBOM Service Guild, Native Analyzer Guild, Node Analyzer Guild.  
+**Status:** All PREP artifacts published 2025-11-20.
--- a/docs/modules/scanner/prep/2025-11-20-analyzers-prep.md
+++ b/docs/modules/scanner/prep/2025-11-20-analyzers-prep.md
@@ -0,0 +1,33 @@
+# Scanner Analyzers PREP Notes — 2025-11-20
+
+Owner: Scanner EPDR Guild · Signals Guild · SBOM Service Guild · Native Analyzer Guild
+Scope: Prep deliverables for PREP-SCANNER-ANALYZERS-LANG-11-003-DEPENDS-ON, PREP-SCANNER-ANALYZERS-LANG-11-004-DEPENDS-ON, and PREP-SCANNER-ANALYZERS-NATIVE-20-002-AWAIT-DE.
+
+## 1) LANG-11-003 runtime evidence ingest (dependent on 11-002)
+- Required upstream: static analyzer outputs from 11-002 (AssemblyRef/ModuleRef/PInvoke/reflection edges with reason codes/confidence).
+- Runtime harness spec (draft):
+  - Listener surfaces `AssemblyLoad`, `Resolving`, `FirstChanceException` events; optional OS-specific providers gated by feature flag `scanner:analyzers:dotnet:runtime-capture` (default off).
+  - Output shape fed into merger: `runtime_edges[]` with fields `{from_entrypoint, to_identity, reason_code, confidence, evidence}`; `evidence` captures event type, payload fragment, timestamp (UTC ISO-8601), and source provider.
+  - Merge rules: prefer static edges; runtime edges only add new edges or upgrade confidence; no removals.
+- Test plan stub: targeted harness tests under `StellaOps.Scanner.Analyzers.Lang.DotNet.Tests` using recorded ETW/EventListener traces; determinism enforced via sorted edges and normalized timestamps.
+
+## 2) LANG-11-004 normalized observation export (dependent on 11-003)
+- Export contract (AOC compliant) to Scanner writer:
+  - `entrypoints[]` with `{id, assembly_name, mvid, tfm, rid, kind}`.
+  - `dependency_edges[]` with `{from_entrypoint, to_component, reason_code, confidence, source={static|runtime|declared}, evidence_ref}`.
+  - `environment_profiles[]` capturing loader configuration (search paths, probing settings) without host-specific absolute paths; deterministic ordering.
+- Integration expectations:
+  - Writer API endpoint path: `scanner/writer/analyzers/dotnet/runtime-static-fusion` (to be aligned with writer team).
+  - SBOM tagging: entrypoints annotated with export IDs; avoid adding derived severity.
+- Testing hook: golden JSON exports under `src/Scanner/__Tests/Fixtures/lang11/export/*.json`, referenced by `ObservationExportTests` once 11-003 is ready.
+
+## 3) NATIVE-20-002 declared-dependency writer (await declared-dependency contract)
+- Scope: emit declared dependencies from ELF dynamic sections with `reason_code=elf-dtneeded`, include `rpath/runpath` and symbol version needs when present.
+- Contract expectations:
+  - Writer record fields: `{binary_id, needed_soname, search_path_hint[], runpath[], build_id, interpreter, version_need[]}`; all ordered deterministically.
+  - Input parser must normalize duplicate `DT_NEEDED` entries and preserve order of appearance.
+- Dependencies: needs finalized declared-dependency writer interface from Scanner writer team; block remains until contract lands, but this prep defines expected payload shape and ordering.
+- Test stub guidance: place fixtures under `src/Scanner/__Tests/Fixtures/native/elf-dtneeded/*` with baseline YAML/JSON; benchmark target <25ms per binary on baseline fixtures.
+
+## Handoff
+- This document is the published prep artefact requested by the above PREP tasks. Implementation tasks should cite this file until upstream contracts arrive.
--- a/docs/modules/scanner/prep/2025-11-20-java-21-005-prep.md
+++ b/docs/modules/scanner/prep/2025-11-20-java-21-005-prep.md
@@ -0,0 +1,26 @@
+# Java Analyzer Prep — SCANNER-ANALYZERS-JAVA-21-005 (PREP)
+
+Status: Draft (2025-11-20)
+Owners: Java Analyzer Guild
+Scope: Capture prerequisites and fixture expectations to unblock SCANNER-ANALYZERS-JAVA-21-005 once upstream build issues clear.
+
+## Blocking issues observed
+- Repository build fails in Concelier due to missing `CoreLinksets` Mongo interfaces; prevents running targeted Java analyzer tests.
+- Targeted `dotnet test` stalls during restore/build on shared runner; needs clean CI slot or scoped solution.
+
+## Required upstream artifacts
+- Concelier/CoreLinksets packages or mocks to let Java analyzer tests restore/build.
+- CI job or local script to run Java analyzer tests in isolation (`StellaOps.Scanner.Analyzers.Lang.Java.Tests.csproj`) without full solution restore.
+
+## Expected outputs once unblocked
+- Framework config extraction evidence covering Spring Boot imports, app/bootstrap configs, web.xml/fragments, JPA/CDI/JAXB configs, logging files, Graal native-image configs.
+- JNI/native hint detection: System.load/Library literals, bundled native libs, Graal JNI configs with component metadata.
+- Deterministic hashing of config evidence (SHA-256) with stable ordering.
+
+## Test/fixture plan
+- Fixtures under `src/Scanner/__Tests/Fixtures/java/21-005/*` capturing the above configs.
+- Regression tests in `StellaOps.Scanner.Analyzers.Lang.Java.Tests/FrameworkConfigTests.cs` asserting evidence presence and hashes.
+- Add CI note: prefer `dotnet test ...Java.Tests.csproj --filter Category=FrameworkConfig` once solutions restore cleanly.
+
+## Handoff
+Use this document as the published prep artefact for PREP-SCANNER-ANALYZERS-JAVA-21-005-TESTS-BLOC. Update once Concelier/CoreLinksets dependency is resolved or CI isolation is available.
--- a/docs/modules/scanner/prep/2025-11-20-java-21-008-prep.md
+++ b/docs/modules/scanner/prep/2025-11-20-java-21-008-prep.md
@@ -0,0 +1,26 @@
+# Java Analyzer Prep — SCANNER-ANALYZERS-JAVA-21-008
+
+Status: Draft (2025-11-20)
+Owners: Java Analyzer Guild
+Scope: Resolver + AOC writer emitting entrypoints/components/edges with reason codes/confidence; depends on 21-007 outputs.
+
+## Dependencies
+- 21-007 manifest metadata collector outputs (signers, manifest attributes) required to seed resolver inputs.
+- CoreLinksets/Concelier build health to allow Java analyzer test runs.
+
+## Proposed resolver outputs
+- `entrypoints[]`: `{id, path, manifest_main_class?, agent_class?, start_class?, module}`.
+- `components[]`: modules/JARs with `{purl?, sha256, module_name?, signed?, signer_ids[]}`.
+- `edges[]`: `{from_entrypoint, to_component, reason_code (jpms|cp|spi|reflect|jni|runtime), confidence, evidence}`.
+- Deterministic ordering: sort edges by (from_entrypoint, to_component, reason_code).
+
+## Tests/fixtures
+- Place fixtures under `src/Scanner/__Tests/Fixtures/java/21-008/*` covering jpms, classpath, SPI, reflection, JNI cases.
+- Regression tests: `ResolverOutputs_AreDeterministic` and `EdgesIncludeReasonAndConfidence`.
+
+## Open decisions
+- Exact confidence scale (0–1 vs categorical) to align with downstream Surface.
+- Whether to emit runtime edges in this task vs deferring to 21-010.
+
+## Handoff
+Use this doc as the PREP artefact for 21-008; update once 21-007 outputs and confidence scale are finalized.
--- a/docs/modules/scanner/prep/2025-11-20-lang-11-001-prep.md
+++ b/docs/modules/scanner/prep/2025-11-20-lang-11-001-prep.md
@@ -0,0 +1,28 @@
+# .NET Lang Analyzer Prep — SCANNER-ANALYZERS-LANG-11-001
+
+Status: Draft (2025-11-20)
+Owners: Scanner EPDR Guild · Language Analyzer Guild
+Scope: Entrypoint resolver mapping project/publish artifacts to deterministic entrypoint identities; PREP covers test isolation and hang debugging.
+
+## Blocking issues
+- `dotnet test` hangs/returns empty output on shared runner; high restore/build fan-out.
+- Concelier/CoreLinksets build errors encountered during prior attempts.
+
+## Proposed mitigation
+- Add CI job to run `dotnet test src/Scanner/__Tests/StellaOps.Scanner.Analyzers.Lang.DotNet.Tests/StellaOps.Scanner.Analyzers.Lang.DotNet.Tests.csproj --filter Category=Entrypoints --blame-hang-timeout 45s` on clean agent.
+- Provide scoped solution `StellaOps.Scanner.Analyzers.Lang.DotNet.slnf` to limit restore.
+
+## Expected outputs
+- Entrypoints array: `{id, assembly_name, mvid, tfm, rid, kind}` with deterministic ID hashing (`sha256` over path+tfm+rid).
+- Environment profiles: loader/probing settings minus host absolute paths.
+
+## Test/fixtures
+- Fixtures under `src/Scanner/__Tests/Fixtures/dotnet/entrypoints/*` for framework-dependent, self-contained, NativeAOT, multi-RID, single-file, trimmed builds.
+- Golden outputs sorted by entrypoint id; verify hash stability across OS.
+
+## Open decisions
+- Hash input shape for `id` (include project GUID?): to confirm with Signals guild.
+- Whether to allow RID inference from runtimeconfig vs RID graph; requires Signals sign-off.
+
+## Handoff
+Treat this as the PREP artefact for PREP-SCANNER-ANALYZERS-LANG-11-001; update once CI isolation and hash rules are confirmed.