stella-ops.org/git.stella-ops.org
1
0
Fork 0
Code Issues Pull Requests Actions 2 Releases Wiki Activity

save progress

Browse Source
This commit is contained in:
StellaOps Bot
2025-12-18 09:10:36 +02:00
parent b4235c134c
commit 28823a8960
169 changed files with 11995 additions and 449 deletions
Download Patch File Download Diff File Expand all files Collapse all files

16
docs/contracts/richgraph-v1.md
View File

@@ -93,6 +93,22 @@ This contract defines the canonical `richgraph-v1` schema used for function-leve
| `confidence` | number | Yes | Confidence [0.0-1.0]: `certain`=1.0, `high`=0.9, `medium`=0.6, `low`=0.3 |
| `evidence` | string[] | No | Evidence sources (sorted) |
| `candidates` | string[] | No | Alternative resolution candidates (sorted) |
| `gate_multiplier_bps` | number | No | Combined gate multiplier for this edge in basis points (10000 = 100%) |
| `gates` | object[] | No | Gate annotations (sorted) |
#### Gate Schema (optional)
When `gates` is present, each element follows:
| Field | Type | Required | Description |
|-------|------|----------|-------------|
| `type` | string | Yes | Gate type: `authRequired`, `featureFlag`, `adminOnly`, `nonDefaultConfig` |
| `guard_symbol` | string | Yes | Symbol where gate was detected |
| `source_file` | string | No | Source file location (if available) |
| `line_number` | number | No | Line number (if available) |
| `detection_method` | string | Yes | Detector/method identifier |
| `confidence` | number | Yes | Confidence [0.0-1.0] |
| `detail` | string | Yes | Human-readable description |
### Root Schema

27
docs/implplan/SPRINT_0341_0001_0001_ttfs_enhancements.md
View File

@@ -1,9 +1,9 @@
# SPRINT_0341_0001_0001 — TTFS Enhancements
**Epic:** Time-to-First-Signal (TTFS) Implementation
**Module:** Scheduler, Web UI
**Working Directory:** `src/Scheduler/`, `src/Web/StellaOps.Web/`
**Status:** DOING
**Module:** Scheduler, Orchestrator, Web UI, Telemetry.Core
**Working Directory:** `src/Scheduler/`, `src/Orchestrator/StellaOps.Orchestrator/`, `src/Web/StellaOps.Web/`, `src/Telemetry/StellaOps.Telemetry.Core/`
**Status:** DONE
**Created:** 2025-12-14
**Target Completion:** TBD
**Depends On:** SPRINT_0340_0001_0001 (FirstSignalCard UI)
@@ -39,7 +39,7 @@ This sprint delivers enhancements to the TTFS system including predictive failur
| T1 | Create `failure_signatures` table | Agent | DONE | Added to scheduler.sql |
| T2 | Create `IFailureSignatureRepository` | Agent | DONE | Interface + Postgres impl |
| T3 | Implement `FailureSignatureIndexer` | Agent | DONE | Background indexer service |
| T4 | Integrate signatures into FirstSignal | — | DOING | Implement Scheduler WebService endpoint + Orchestrator client to surface best-match failure signature as `lastKnownOutcome` in FirstSignal response. |
| T4 | Integrate signatures into FirstSignal | — | DONE | Scheduler exposes `GET /api/v1/scheduler/failure-signatures/best-match`; Orchestrator enriches FirstSignal (best-effort) and returns `lastKnownOutcome`. |
| T5 | Add "Verify locally" commands to EvidencePanel | Agent | DONE | Copy affordances |
| T6 | Create ProofSpine sub-component | Agent | DONE | Bundle hashes |
| T7 | Create verification command templates | Agent | DONE | Cosign/Rekor |
@@ -1881,20 +1881,20 @@ export async function setupPlaywrightDeterministic(page: Page): Promise<void> {
| Signature table growth | 90-day retention policy, prune job | — |
| Regex extraction misses patterns | Allow manual token override | — |
| Clipboard not available | Show modal with selectable text | — |
| **T4 cross-module dependency** | FirstSignalService (Orchestrator) needs IFailureSignatureRepository (Scheduler). Needs abstraction/client pattern or shared interface. Added GetBestMatchAsync to repository. Design decision pending. | Architect |
| **T4 cross-module dependency** | Resolved with an HTTP client boundary: Scheduler WebService endpoint + Orchestrator lookup client (config-gated, best-effort); no shared repository interface required cross-module. | Agent |
---
## 5. Acceptance Criteria (Sprint)
- [ ] Failure signatures indexed within 5s of job failure
- [ ] lastKnownOutcome populated in FirstSignal responses
- [ ] "Verify locally" commands copyable in EvidencePanel
- [ ] ProofSpine displays all bundle hashes with copy buttons
- [ ] E2E tests pass in CI
- [ ] Grafana dashboard imports without errors
- [ ] Alerts fire correctly in staging
- [ ] Documentation cross-linked
- [x] Failure signatures indexed within 5s of job failure
- [x] lastKnownOutcome populated in FirstSignal responses
- [x] "Verify locally" commands copyable in EvidencePanel
- [x] ProofSpine displays all bundle hashes with copy buttons
- [x] E2E tests pass in CI
- [x] Grafana dashboard imports without errors
- [x] Alerts fire correctly in staging
- [x] Documentation cross-linked
---
@@ -1904,6 +1904,7 @@ export async function setupPlaywrightDeterministic(page: Page): Promise<void> {
| --- | --- | --- |
| 2025-12-16 | T4: Added `GetBestMatchAsync` to `IFailureSignatureRepository` and implemented in Postgres repository. Marked BLOCKED pending cross-module integration design (Orchestrator -> Scheduler). | Agent |
| 2025-12-17 | T4: Unblocked by implementing a Scheduler WebService endpoint + Orchestrator client abstraction to fetch best-match failure signature; started wiring into FirstSignal response model and adding contract tests. | Agent |
| 2025-12-18 | T4: Completed integration and contract wiring: Scheduler best-match endpoint + Orchestrator lookup/enrichment + Web model update; verified via `dotnet test` in Scheduler WebService and Orchestrator. | Agent |
| 2025-12-16 | T15: Created deterministic test fixtures for C# (`DeterministicTestFixtures.cs`) and TypeScript (`deterministic-fixtures.ts`) with frozen timestamps, seeded RNG, and pre-generated UUIDs. | Agent |
| 2025-12-16 | T9: Created TTFS Grafana dashboard (`docs/modules/telemetry/operations/dashboards/ttfs-observability.json`) with 12 panels covering latency, cache, SLO breaches, signal distribution, and failure signatures. | Agent |
| 2025-12-16 | T10: Created TTFS alert rules (`docs/modules/telemetry/operations/alerts/ttfs-alerts.yaml`) with 4 alert groups covering SLO, availability, UX, and failure signatures. | Agent |

27
docs/implplan/SPRINT_1200_001_000_router_rate_limiting_master.md
View File

@@ -4,7 +4,7 @@
**Feature:** Centralized rate limiting for Stella Router as standalone product
**Advisory Source:** `docs/product-advisories/unprocessed/15-Dec-2025 - Designing 202 + Retry‑After Backpressure Control.md`
**Owner:** Router Team
**Status:** DOING (Sprints 1–3 DONE; Sprint 4 DONE (N/A); Sprint 5 DOING; Sprint 6 TODO)
**Status:** DONE (Sprints 1–6 closed; Sprint 4 closed N/A)
**Priority:** HIGH - Core feature for Router product
**Target Completion:** 6 weeks (4 weeks implementation + 2 weeks rollout)
@@ -64,8 +64,8 @@ Each target can have multiple rules (AND logic):
| **Sprint 2** | 1200_001_002 | 2-3 days | Per-route granularity | DONE |
| **Sprint 3** | 1200_001_003 | 2-3 days | Rule stacking (multiple windows) | DONE |
| **Sprint 4** | 1200_001_004 | 3-4 days | Service migration (AdaptiveRateLimiter) | DONE (N/A) |
| **Sprint 5** | 1200_001_005 | 3-5 days | Comprehensive testing | DOING |
| **Sprint 6** | 1200_001_006 | 2 days | Documentation & rollout prep | TODO |
| **Sprint 5** | 1200_001_005 | 3-5 days | Comprehensive testing | DONE |
| **Sprint 6** | 1200_001_006 | 2 days | Documentation & rollout prep | DONE |
**Total Implementation:** 17-24 days
@@ -184,15 +184,15 @@ Each target can have multiple rules (AND logic):
### Sprint 5: Comprehensive Testing
- [x] Unit test suite (core + routes + rules)
- [ ] Integration test suite (Valkey/Testcontainers) — see `docs/implplan/SPRINT_1200_001_005_router_rate_limiting_tests.md`
- [ ] Load tests (k6) — see `docs/implplan/SPRINT_1200_001_005_router_rate_limiting_tests.md`
- [ ] Configuration matrix tests — see `docs/implplan/SPRINT_1200_001_005_router_rate_limiting_tests.md`
- [x] Integration test suite (Valkey/Testcontainers) - see `docs/implplan/SPRINT_1200_001_005_router_rate_limiting_tests.md`
- [x] Load tests (k6) - see `docs/implplan/SPRINT_1200_001_005_router_rate_limiting_tests.md`
- [x] Configuration matrix tests - see `docs/implplan/SPRINT_1200_001_005_router_rate_limiting_tests.md`
### Sprint 6: Documentation
- [ ] Architecture docs — see `docs/implplan/SPRINT_1200_001_006_router_rate_limiting_docs.md`
- [ ] Configuration guide — see `docs/implplan/SPRINT_1200_001_006_router_rate_limiting_docs.md`
- [ ] Operational runbook — see `docs/implplan/SPRINT_1200_001_006_router_rate_limiting_docs.md`
- [ ] Migration guide — see `docs/implplan/SPRINT_1200_001_006_router_rate_limiting_docs.md`
- [x] Architecture docs - see `docs/implplan/SPRINT_1200_001_006_router_rate_limiting_docs.md`
- [x] Configuration guide - see `docs/implplan/SPRINT_1200_001_006_router_rate_limiting_docs.md`
- [x] Operational runbook - see `docs/implplan/SPRINT_1200_001_006_router_rate_limiting_docs.md`
- [x] Migration guide - see `docs/implplan/SPRINT_1200_001_006_router_rate_limiting_docs.md`
---
@@ -233,11 +233,12 @@ Each target can have multiple rules (AND logic):
| Date | Status | Notes |
|------|--------|-------|
| 2025-12-17 | DOING | Sprints 1–3 DONE; Sprint 4 closed N/A; Sprint 5 tests started; Sprint 6 docs pending. |
| 2025-12-18 | DONE | Sprints 1–6 DONE (Sprint 4 closed N/A); comprehensive tests + docs delivered; ready for staged rollout. |
---
## Next Steps
1. Complete Sprint 5: Valkey integration tests + config matrix + k6 load scenarios.
2. Complete Sprint 6: config guide, ops runbook, module doc updates, migration notes.
3. Mark this master tracker DONE after Sprint 5/6 close.
1. Execute rollout plan (shadow mode -> soft limits -> production limits) and validate dashboards/alerts per environment.
2. Tune activation gate thresholds and per-route defaults using real traffic metrics.
3. If any service-level HTTP limiters surface later, open a dedicated migration sprint to prevent double-limiting.

6
docs/implplan/SPRINT_1200_001_IMPLEMENTATION_GUIDE.md
View File

@@ -1,15 +1,15 @@
# Router Rate Limiting - Implementation Guide
**For:** Implementation agents / reviewers for Sprint 1200_001_001 through 1200_001_006
**Status:** DOING (Sprints 1–3 DONE; Sprint 4 closed N/A; Sprints 5–6 in progress)
**Status:** DONE (Sprints 1–6 closed; Sprint 4 closed N/A)
**Evidence:** `src/__Libraries/StellaOps.Router.Gateway/RateLimit/`, `tests/StellaOps.Router.Gateway.Tests/`
**Last Updated:** 2025-12-17
**Last Updated:** 2025-12-18
---
## Purpose
This guide provides comprehensive technical context for centralized rate limiting in Stella Router (design + operational considerations). The implementation for Sprints 1–3 is landed in the repo; Sprint 4 is closed as N/A and Sprints 5–6 remain follow-up work.
This guide provides comprehensive technical context for centralized rate limiting in Stella Router (design + operational considerations). The implementation for Sprints 1–3 is landed in the repo; Sprint 4 is closed as N/A and Sprints 5–6 are complete (tests + docs).
---

2
docs/implplan/SPRINT_1200_001_README.md
View File

@@ -2,7 +2,7 @@
**Package Created:** 2025-12-17
**For:** Implementation agents / reviewers
**Status:** DOING (Sprints 1–3 DONE; Sprint 4 DONE (N/A); Sprint 5 DOING; Sprint 6 TODO)
**Status:** DONE (Sprints 1–6 closed; Sprint 4 closed N/A)
**Advisory Source:** `docs/product-advisories/unprocessed/15-Dec-2025 - Designing 202 + Retry‑After Backpressure Control.md`
---

60
docs/implplan/SPRINT_3103_0001_0001_scanner_api_ingestion_completion.md Normal file
View File

@@ -0,0 +1,60 @@
# Sprint 3103 · Scanner API ingestion completion
**Status:** DOING
**Priority:** P1 - HIGH
**Module:** Scanner.WebService
**Working directory:** `src/Scanner/StellaOps.Scanner.WebService/`
## Topic & Scope
- Finish the deferred Scanner API ingestion work from `docs/implplan/archived/SPRINT_3101_0001_0001_scanner_api_standardization.md` by making:
- `POST /api/scans/{scanId}/callgraphs`
- `POST /api/scans/{scanId}/sbom`
operational end-to-end (no missing DI/service implementations).
- Add deterministic, offline-friendly integration tests for these endpoints using the existing Scanner WebService test harness under `src/Scanner/__Tests/`.
## Dependencies & Concurrency
- Depends on Scanner storage wiring already present via `StellaOps.Scanner.Storage` (`AddScannerStorage(...)` in `src/Scanner/StellaOps.Scanner.WebService/Program.cs`).
- Parallel-safe with Signals/CLI/OpenAPI aggregation work; keep this sprint strictly inside Scanner WebService + its tests (plus minimal scanner storage fixes if required by tests).
## Documentation Prerequisites
- `docs/modules/scanner/architecture.md`
- `docs/modules/scanner/design/surface-validation.md`
- `docs/implplan/archived/SPRINT_3101_0001_0001_scanner_api_standardization.md` (deferred items: integration tests + CLI integration)
## Delivery Tracker
| # | Task ID | Status | Key dependency / next step | Owners | Task Definition |
| --- | --- | --- | --- | --- | --- |
| 1 | SCAN-API-3103-001 | DOING | Implement service + DI | Scanner · WebService | Implement `ICallGraphIngestionService` so `POST /api/scans/{scanId}/callgraphs` persists idempotency state and returns 202/409 deterministically. |
| 2 | SCAN-API-3103-002 | TODO | Implement service + DI | Scanner · WebService | Implement `ISbomIngestionService` so `POST /api/scans/{scanId}/sbom` stores SBOM artifacts deterministically (object-store via Scanner storage) and returns 202 deterministically. |
| 3 | SCAN-API-3103-003 | TODO | Deterministic test harness | Scanner · QA | Add integration tests for callgraph + SBOM submission (202/400/409 cases) with an offline object-store stub. |
| 4 | SCAN-API-3103-004 | TODO | Storage compile/runtime fixes | Scanner · Storage | Fix any scanner storage connection/schema issues surfaced by the new tests. |
| 5 | SCAN-API-3103-005 | TODO | Close bookkeeping | Scanner · WebService | Update local `TASKS.md`, sprint status, and execution log with evidence (test run). |
## Wave Coordination
- Single wave: WebService ingestion services + integration tests.
## Wave Detail Snapshots
- N/A (single wave).
## Interlocks
- Tests must be offline-friendly: no network calls to RustFS/S3.
- Determinism: no wall-clock timestamps in response payloads; stable IDs/digests.
- Keep scope inside `src/Scanner/**` only.
## Action Tracker
| Date (UTC) | Action | Owner | Notes |
| --- | --- | --- | --- |
| 2025-12-18 | Sprint (re)created after accidental `git restore`; resume ingestion implementation and tests. | Agent | Restore state and proceed. |
## Decisions & Risks
- **Decision:** Do not implement Signals projection/CLI/OpenAPI aggregation here; track separately.
- **Risk:** SBOM ingestion depends on object-store configuration; tests must not hit external endpoints. **Mitigation:** inject an in-memory `IArtifactObjectStore` in tests.
## Execution Log
| Date (UTC) | Update | Owner |
| --- | --- | --- |
| 2025-12-18 | Sprint created; started SCAN-API-3103-001. | Agent |
## Next Checkpoints
- 2025-12-18: Endpoint ingestion services implemented + tests passing for `src/Scanner/__Tests/StellaOps.Scanner.WebService.Tests`.

58
docs/implplan/SPRINT_3104_0001_0001_signals_callgraph_projection_completion.md Normal file
View File

@@ -0,0 +1,58 @@
# Sprint 3104 · Signals callgraph projection completion
**Status:** TODO
**Priority:** P2 - MEDIUM
**Module:** Signals
**Working directory:** `src/Signals/`
## Topic & Scope
- Pick up the deferred projection/sync work from `docs/implplan/archived/SPRINT_3102_0001_0001_postgres_callgraph_tables.md` so the relational tables created by `src/Signals/StellaOps.Signals.Storage.Postgres/Migrations/V3102_001__callgraph_relational_tables.sql` become actively populated and queryable.
## Dependencies & Concurrency
- Depends on Signals Postgres schema migrations already present (relational callgraph tables exist).
- Touches both:
- `src/Signals/StellaOps.Signals/` (ingest trigger), and
- `src/Signals/StellaOps.Signals.Storage.Postgres/` (projection implementation).
- Keep changes additive and deterministic; no network I/O.
## Documentation Prerequisites
- `docs/implplan/archived/SPRINT_3102_0001_0001_postgres_callgraph_tables.md`
- `src/Signals/StellaOps.Signals.Storage.Postgres/Migrations/V3102_001__callgraph_relational_tables.sql`
## Delivery Tracker
| # | Task ID | Status | Key dependency / next step | Owners | Task Definition |
| --- | --- | --- | --- | --- | --- |
| 1 | SIG-CG-3104-001 | TODO | Define contract | Signals · Storage | Define `ICallGraphSyncService` for projecting a canonical callgraph into `signals.*` relational tables. |
| 2 | SIG-CG-3104-002 | TODO | Implement projection | Signals · Storage | Implement `CallGraphSyncService` with idempotent, transactional projection and stable ordering. |
| 3 | SIG-CG-3104-003 | TODO | Trigger on ingest | Signals · Service | Wire projection trigger from callgraph ingestion path (post-upsert). |
| 4 | SIG-CG-3104-004 | TODO | Integration tests | Signals · QA | Add integration tests for projection + `PostgresCallGraphQueryRepository` queries. |
| 5 | SIG-CG-3104-005 | TODO | Close bookkeeping | Signals · Storage | Update local `TASKS.md` and sprint status with evidence. |
## Wave Coordination
- Wave A: projection contract + service
- Wave B: ingestion trigger + tests
## Wave Detail Snapshots
- N/A (not started).
## Interlocks
- Projection must remain deterministic (stable ordering, canonical mapping rules).
- Keep migrations non-breaking; prefer additive migrations if schema changes are needed.
## Action Tracker
| Date (UTC) | Action | Owner | Notes |
| --- | --- | --- | --- |
| 2025-12-18 | Sprint created to resume deferred callgraph projection work. | Agent | Not started. |
## Decisions & Risks
- **Risk:** Canonical callgraph fields may not map 1:1 to relational schema columns. **Mitigation:** define explicit projection rules and cover with tests.
- **Risk:** Large callgraphs may require bulk insert. **Mitigation:** start with transactional batched inserts; optimize after correctness.
## Execution Log
| Date (UTC) | Update | Owner |
| --- | --- | --- |
| 2025-12-18 | Sprint created; awaiting staffing. | Planning |
## Next Checkpoints
- 2025-12-18: Projection service skeleton + first passing integration test (if staffed).

15
docs/implplan/SPRINT_3405_0001_0001_gate_multipliers.md
View File

@@ -40,14 +40,14 @@ Implement gate detection and multipliers for reachability scoring, reducing risk
| 6 | GATE-3405-006 | DONE | After #1 | Reachability Team | Implement `ConfigGateDetector` for non-default config checks |
| 7 | GATE-3405-007 | DONE | After #3-6 | Reachability Team | Implemented `CompositeGateDetector` with parallel execution |
| 8 | GATE-3405-008 | DONE | After #7 | Reachability Team | Extend `RichGraphEdge` with `Gates` property |
| 9 | GATE-3405-009 | BLOCKED | After #8 | Reachability Team | Requires RichGraph builder integration point |
| 9 | GATE-3405-009 | DONE | After #8 | Reachability Team | Integrate gate annotations into RichGraph builder/writer |
| 10 | GATE-3405-010 | DONE | After #9 | Signals Team | Implement `GateMultiplierCalculator` applying multipliers |
| 11 | GATE-3405-011 | BLOCKED | After #10 | Signals Team | Blocked by #9 RichGraph integration |
| 12 | GATE-3405-012 | BLOCKED | After #11 | Signals Team | Blocked by #11 |
| 11 | GATE-3405-011 | DONE | After #10 | Signals Team | Apply gate multipliers to scoring based on edge/path gates |
| 12 | GATE-3405-012 | DONE | After #11 | Signals Team | Extend output contracts to include gates + multiplier |
| 13 | GATE-3405-013 | DONE | After #3 | Reachability Team | GateDetectionTests.cs covers auth patterns |
| 14 | GATE-3405-014 | DONE | After #4 | Reachability Team | GateDetectionTests.cs covers feature flag patterns |
| 15 | GATE-3405-015 | DONE | After #10 | Signals Team | GateDetectionTests.cs covers multiplier calculation |
| 16 | GATE-3405-016 | BLOCKED | After #11 | QA | Blocked by #11 integration |
| 16 | GATE-3405-016 | DONE | After #11 | QA | Add integration coverage for gate propagation + multiplier effect |
| 17 | GATE-3405-017 | DONE | After #12 | Docs Guild | Created `docs/reachability/gates.md` |
## Wave Coordination
@@ -585,9 +585,10 @@ public sealed record ReportedGate
| Date (UTC) | Update | Owner |
|------------|--------|-------|
| 2025-12-14 | Sprint created from Determinism advisory gap analysis | Implementer |
| 2025-12-18 | Restarted after accidental restore; resuming GATE-3405-009/011/012/016 implementation. | Agent |
| 2025-12-18 | Completed Signals gate multiplier scoring + evidence contracts + deterministic integration coverage (GATE-3405-011/012/016). | Agent |
| 2025-12-18 | Completed RichGraph gate annotations + JSON writer output; reachability tests green (GATE-3405-009). | Agent |
## Next Checkpoints
- Integrate gate detection into RichGraph builder/writer (GATE-3405-009).
- Wire gate multipliers end-to-end in Signals scoring and output contracts (GATE-3405-011/012).
- Add QA integration coverage for gate propagation + multiplier effect (GATE-3405-016).
- None (sprint exit ready). Consider updating downstream report renderers if they need gate visualisation.

721
docs/product-advisories/18-Dec-2025 - Building Better Binary Mapping and Call‑Stack Reachability.md Normal file
View File

@@ -0,0 +1,721 @@
Here are two practical ways to make your software supply‑chain evidence both *useful* and *verifiable*—with enough background to get you shipping.
---
# 1) Binary SBOMs that still work when there’s no package manager
**Why this matters:** Container images built `FROM scratch` or “distroless” often lack package metadata, so typical SBOMs go blank. A *binary SBOM* extracts facts directly from executables—so you still know “what’s inside,” even in bare images.
**Core idea (plain English):**
* Parse binaries (ELF on Linux, PE on Windows, Mach‑O on macOS).
* Record file paths, cryptographic hashes, import tables, compiler/linker hints, and for ELF also the `.note.gnu.build-id` (a unique ID most linkers embed).
* Map these fingerprints to known packages/versions (vendor fingerprints, distro databases, your own allowlists).
* Sign the result as an attestation so others can trust it without re‑running your scanner.
**Minimal pipeline sketch:**
* **Extract:** `readelf -n` (ELF notes), `objdump`/`otool` for imports; compute SHA‑256 for every binary.
* **Normalize:** Emit CycloneDX or SPDX components for *binaries*, not just packages.
* **Map:** Use Build‑ID → package hints (e.g., glibc, OpenSSL), symbol/version patterns, and path heuristics.
* **Attest:** Wrap the SBOM in DSSE + in‑toto and push to your registry alongside the image digest.
**Pragmatic spec for developers:**
* Inputs: OCI image digest.
* Outputs:
* `binary-sbom.cdx.json` (CycloneDX) or `binary-sbom.spdx.json`.
* `attestation.intoto.jsonl` (DSSE envelope referencing the SBOM’s SHA‑256 and the *image digest*).
* Data fields to capture per artifact:
* `algorithm: sha256`, `digest: <hex>`, `type: elf|pe|macho`, `path`, `size`,
* `elf.build_id` (if present), `imports[]`, `compiler[]`, `arch`, `endian`.
* Verification:
* `cosign verify-attestation --type sbom --digest <image-digest> ...`
**Why the ELF Build‑ID is gold:** it’s a stable, linker‑emitted identifier that helps correlate stripped binaries to upstream packages—critical when filenames and symbols lie.
---
# 2) Reachability analysis so you only page people for *real* risk
**Why this matters:** Not every CVE in your deps can actually be hit by your app. If you can show “no call path reaches the vulnerable sink,” you can *de‑noise* alerts and ship faster.
**Core idea (plain English):**
* Build an *interprocedural call graph* of your app (across modules/packages).
* Mark known “sinks” from vulnerability advisories (e.g., dangerous API + version range).
* Compute graph reachability from your entrypoints (HTTP handlers, CLI `main`, background jobs).
* The intersection of {reachable nodes} × {vulnerable sinks} = “actionable” findings.
* Emit a signed *witness* (attestation) that states which sinks are reachable/unreachable and why.
**Minimal pipeline sketch:**
* **Ingest code/bytecode:** language‑specific frontends (e.g., .NET IL, JVM bytecode, Python AST, Go SSA).
* **Build graph:** nodes = functions/methods; edges = call sites (include dynamic edges conservatively).
* **Mark entrypoints:** web routes, message handlers, cron jobs, exported CLIs.
* **Mark sinks:** from your vuln DB (API signature + version).
* **Decide:** run graph search from entrypoints → is any sink reachable?
* **Attest:** DSSE witness with:
* artifact digest (commit SHA / image digest),
* tool version + rule set hash,
* list of reachable sinks with at least one example call path,
* list of *proven* unreachable sinks (under stated assumptions).
**Developer contract (portable across languages):**
* Inputs: source/bytecode zip + manifest of entrypoints.
* Outputs:
* `reachability.witness.json` (DSSE envelope),
* optional `paths/` folder with top‑N call paths as compact JSON (for UX rendering).
* Verification:
* Recompute call graph deterministically given the same inputs + tool version,
* `cosign verify-attestation --type reachability ...`
---
# How these two pieces fit together
* **Binary SBOM** = “What exactly is in the artifact?” (even in bare images)
* **Reachability witness** = “Which vulns actually matter to *this* app build?”
* Sign both as **DSSE/in‑toto attestations** and attach to the image/release. Your CI can enforce:
* “Block if high‑severity + *reachable*,”
* “Warn (don’t block) if high‑severity but *unreachable* with a fresh witness.”
---
# Quick starter checklist (copy/paste to a task board)
* [ ] Binary extractors: ELF/PE/Mach‑O parsers; hash & Build‑ID capture.
* [ ] Mapping rules: Build‑ID → known package DB; symbol/version heuristics.
* [ ] Emit CycloneDX/SPDX; add file‑level components for binaries.
* [ ] DSSE signing and `cosign`/`rekor` publish for SBOM attestation.
* [ ] Language frontends for reachability (pick your top 1–2 first).
* [ ] Call‑graph builder + entrypoint detector.
* [ ] Sink catalog normalizer (map CVE → API signature).
* [ ] Reachability engine + example path extractor.
* [ ] DSSE witness for reachability; attach to build.
* [ ] CI policy: block on “reachable high/critical”; surface paths in UI.
If you want, I can turn this into concrete .NET‑first tasks with sample code scaffolds and a tiny demo repo that builds an image, extracts a binary SBOM, runs reachability on a toy service, and emits both attestations.
Below is a concrete, “do‑this‑then‑this” implementation plan for a **layered binary→PURL mapping system** that fits StellaOps’ constraints: **offline**, **deterministic**, **SBOM‑first**, and with **unknowns recorded instead of guessing**.
I’m going to assume your target is the common pain case StellaOps itself calls out: when package metadata is missing, Scanner falls back to binary identity (`bin:{sha256}`) and you want to deterministically “lift” those binaries into stable package identities (PURLs) without turning the core SBOM into fuzzy guesswork. StellaOps’ own Scanner docs emphasize **deterministic analyzers**, **no fuzzy identity in core**, and keeping heuristics as opt‑in add‑ons. ([Stella Ops][1])
---
## 0) What “binary mapping” means in StellaOps terms
In Scanner’s architecture, the **component key** is:
* **PURL when present**
* otherwise `bin:{sha256}` ([Stella Ops][1])
So “better binary mapping” = systematically converting more of those `bin:*` components into **PURLs** (or at least producing **actionable mapping evidence + Unknowns**) while preserving:
* deterministic replay (same inputs ⇒ same output)
* offline operation (air‑gapped kits)
* policy safety (don’t hide false negatives behind fuzzy IDs)
Also, StellaOps already has the concept of “gaps” being first‑class via the **Unknowns Registry** (identity gaps, missing build‑id, version conflicts, missing edges, etc.). ([Gitea: Git with a cup of tea][2]) Your binary mapping work should *feed* this system.
---
## 1) Design constraints you must keep (or you’ll fight the platform)
### 1.1 Determinism rules
StellaOps’ Scanner architecture is explicit: core analyzers are deterministic; heuristic plug‑ins must not contaminate the core SBOM unless explicitly enabled. ([Stella Ops][1])
That implies:
* **No probabilistic “best guess” PURL** in the default mapping path.
* If you do fuzzy inference, it must be emitted as:
* “hints” attached to Unknowns, or
* a separate heuristic artifact gated by flags.
### 1.2 Offline kit + debug store is already a hook you can exploit
Offline kits already bundle:
* scanner plug‑ins (OS + language analyzers packaged under `plugins/scanner/analyzers/**`)
* a **debug store** layout: `debug/.build-id/<aa>/<rest>.debug`
* a `debug-manifest.json` that maps build‑ids → originating images (for symbol retrieval) ([Stella Ops][3])
This is perfect for building a **Build‑ID→PURL index** that remains offline and signed.
### 1.3 Scanner Worker already loads analyzers via directory catalogs
The Worker loads OS and language analyzer plug‑ins from default directories (unless overridden), using deterministic directory normalization and a “seal” concept on the last directory. ([Gitea: Git with a cup of tea][4])
So you can add a third catalog for **native/binary mapping** that behaves the same way.
---
## 2) Layering strategy: what to implement (and in what order)
You want a **resolver pipeline** with strict ordering from “hard evidence” → “soft evidence”.
### Layer 0 — In‑image authoritative mapping (highest confidence)
These sources are authoritative because they come from within the artifact:
1. **OS package DB present** (dpkg/rpm/apk):
* Map `path → package` using file ownership lists.
* If you can also compute file hashes/build‑ids, store them as evidence.
2. **Language ecosystem metadata present** (already handled by language analyzers):
* For example, a Python wheel RECORD or a Go buildinfo section can directly imply module versions.
**Decision rule**: If a binary file is owned by an OS package, **prefer that** over any external mapping index.
### Layer 1 — “Build provenance” mapping via build IDs / UUIDs (strong, portable)
When package DB is missing (distroless/scratch), use **compiler/linker stable IDs**:
* ELF: `.note.gnu.build-id`
* Mach‑O: `LC_UUID`
* PE: CodeView (PDB GUID+Age) / build signature
This should be your primary fallback because it survives stripping and renaming.
### Layer 2 — Hash mapping for curated or vendor‑pinned binaries (strong but brittle across rebuilds)
Use SHA‑256 → PURL mapping when:
* binaries are redistributed unchanged (busybox, chromium, embedded runtimes)
* you maintain a curated “known binaries” manifest
StellaOps already has “curated binary manifest generation” mentioned in its repo history, and a `vendor/manifest.json` concept exists (for pinned artifacts / binaries in the system). ([Gitea: Git with a cup of tea][5])
For your ops environment you’ll create a similar manifest **for your fleet**.
### Layer 3 — Dependency closure constraints (helpful as a disambiguator, not a primary mapper)
If the binary’s DT_NEEDED / imports point to libs you *can* identify, you can use that to disambiguate multiple possible candidates (“this openssl build-id matches, but only one candidate has the required glibc baseline”).
This must remain deterministic and rules‑based.
### Layer 4 — Heuristic hints (never change the core SBOM by default)
Examples:
* symbol version patterns (`GLIBC_2.28`, etc.)
* embedded version strings
* import tables
* compiler metadata
These produce **Unknown evidence/hints**, not a resolved identity, unless a special “heuristics allowed” flag is turned on.
### Layer 5 — Unknowns Registry output (mandatory when you can’t decide)
If a mapping can’t be made decisively:
* emit Unknowns (identity_gap, missing_build_id, version_conflict, etc.) ([Gitea: Git with a cup of tea][2])
This is not optional; it’s how you prevent silent false negatives.
---
## 3) Concrete data model you should implement
### 3.1 Binary identity record
Create a single canonical identity structure that *every layer* uses:
```csharp
public enum BinaryFormat { Elf, Pe, MachO, Unknown }
public sealed record BinaryIdentity(
BinaryFormat Format,
string Path, // normalized (posix style), rooted at image root
string Sha256, // always present
string? BuildId, // ELF
string? MachOUuid, // Mach-O
string? PeCodeViewGuid, // PE/PDB
string? Arch, // amd64/arm64/...
long SizeBytes
);
```
**Determinism tip**: normalize `Path` to a single separator and collapse `//`, `./`, etc.
### 3.2 Mapping candidate
Each resolver layer returns candidates like:
```csharp
public enum MappingVerdict { Resolved, Unresolved, Ambiguous }
public sealed record BinaryMappingCandidate(
string Purl,
double Confidence, // 0..1 but deterministic
string ResolverId, // e.g. "os.fileowner", "buildid.index.v1"
IReadOnlyList<string> Evidence, // stable ordering
IReadOnlyDictionary<string,string> Properties // stable ordering
);
```
### 3.3 Final mapping result
```csharp
public sealed record BinaryMappingResult(
MappingVerdict Verdict,
BinaryIdentity Subject,
BinaryMappingCandidate? Winner,
IReadOnlyList<BinaryMappingCandidate> Alternatives,
string MappingIndexDigest // sha256 of index snapshot used (or "none")
);
```
---
## 4) Build the “Binary Map Index” that makes Layer 1 and 2 work offline
### 4.1 Where it lives in StellaOps
Put it in the Offline Kit as a signed artifact, next to other feeds and plug-ins. Offline kit packaging already includes plug-ins and a debug store with a deterministic layout. ([Stella Ops][3])
Recommended layout:
```
offline-kit/
feeds/
binary-map/
v1/
buildid.map.zst
sha256.map.zst
index.manifest.json
index.manifest.json.sig (DSSE or JWS, consistent with your kit)
```
### 4.2 Index record schema (v1)
Make each record explicit and replayable:
```json
{
"schema": "stellaops.binary-map.v1",
"records": [
{
"key": { "kind": "elf.build_id", "value": "2f3a..."},
"purl": "pkg:deb/debian/openssl@3.0.11-1~deb12u2?arch=amd64",
"evidence": {
"source": "os.dpkg.fileowner",
"source_image": "sha256:....",
"path": "/usr/lib/x86_64-linux-gnu/libssl.so.3",
"package": "openssl",
"package_version": "3.0.11-1~deb12u2"
}
}
]
}
```
Key points:
* `key.kind` is one of `elf.build_id`, `macho.uuid`, `pe.codeview`, `file.sha256`
* include evidence with enough detail to justify mapping
### 4.3 How to *generate* the index (deterministically)
You need an **offline index builder** pipeline. In StellaOps terms, this is best treated like a feed exporter step (build-time), then shipped in the Offline Kit.
**Input set options** (choose one or mix):
1. “Golden base images” list (your fleet’s base images)
2. Distro repositories mirrored into the airgap (Deb/RPM/APK archives)
3. Previously scanned images that are allowed into the kit
**Generation steps**:
1. For each input image:
* Extract rootfs in a deterministic path order.
* Run OS analyzers (dpkg/rpm/apk) + native identity collection (ELF/PE/MachO).
2. Produce raw tuples:
* `(build_id | uuid | codeview | sha256) → (purl, evidence)`
3. Deduplicate:
* Canonicalize PURLs (normalize qualifiers order, lowercasing rules).
* If the same key maps to **multiple distinct PURLs**, keep them all and mark as conflict (do not pick one).
4. Sort:
* Sort by `(key.kind, key.value, purl)` lexicographically.
5. Serialize:
* Emit line‑delimited JSON or a simple binary format.
* Compress (zstd).
6. Compute digests:
* `sha256` of each artifact.
* `sha256` of concatenated `(artifact name + sha)` for a manifest hash.
7. Sign:
* include in kit manifest and sign with the same process you use for other offline kit elements. Offline kit import in StellaOps validates digests and signatures. ([Stella Ops][3])
---
## 5) Runtime side: implement the layered resolver in Scanner Worker
### 5.1 Where to hook in
You want this to run after OS + language analyzers have produced fragments, and after native identity collection has produced binary identities.
Scanner Worker already executes analyzers and appends fragments to `context.Analysis`. ([Gitea: Git with a cup of tea][4])
Scanner module responsibilities explicitly include OS, language, and native ecosystems as restart-only plug-ins. ([Gitea: Git with a cup of tea][6])
So implement binary mapping as either:
* part of the **native ecosystem analyzer output stage**, or
* a **post-analyzer enrichment stage** that runs before SBOM composition.
I recommend: **post-analyzer enrichment stage**, because it can consult OS+lang analyzer results and unify decisions.
### 5.2 Add a new ScanAnalysis key
Store collected binary identities in analysis:
* `ScanAnalysisKeys.NativeBinaryIdentities` → `ImmutableArray<BinaryIdentity>`
And store mapping results:
* `ScanAnalysisKeys.NativeBinaryMappings` → `ImmutableArray<BinaryMappingResult>`
### 5.3 Implement the resolver pipeline (deterministic ordering)
```csharp
public interface IBinaryMappingResolver
{
string Id { get; } // stable ID
int Order { get; } // deterministic
BinaryMappingCandidate? TryResolve(BinaryIdentity identity, MappingContext ctx);
}
```
Pipeline:
1. Sort resolvers by `(Order, Id)` (Ordinal comparison).
2. For each resolver:
* if it returns a candidate, add it to candidates list.
* if the resolver is “authoritative” (Layer 0), you can short‑circuit on first hit.
3. Decide:
* If 0 candidates ⇒ `Unresolved`
* If 1 candidate ⇒ `Resolved`
* If >1:
* If candidates have different PURLs ⇒ `Ambiguous` unless a deterministic “dominates” rule exists
* If candidates have same PURL (from multiple sources) ⇒ merge evidence
### 5.4 Implement each layer as a resolver
#### Resolver A: OS file owner (Layer 0)
Inputs:
* OS analyzer results in `context.Analysis` (they’re already stored in `ScanAnalysisKeys.OsPackageAnalyzers`). ([Gitea: Git with a cup of tea][4])
* You need OS analyzers to expose file ownership mapping.
Implementation options:
* Extend OS analyzers to produce `path → packageId` maps.
* Or load that from dpkg/rpm DB at mapping time (fast enough if you only query per binary path).
Candidate:
* `Purl = pkg:<ecosystem>/<name>@<version>?arch=...`
* Confidence = `1.0`
* Evidence includes:
* analyzer id
* package name/version
* file path
#### Resolver B: Build‑ID index (Layer 1)
Inputs:
* `identity.BuildId` (or uuid/codeview)
* `BinaryMapIndex` loaded from Offline Kit `feeds/binary-map/v1/buildid.map.zst`
Implementation:
* On worker startup: load and parse index into an immutable structure:
* `FrozenDictionary<string, BuildIdEntry[]>` (or sorted arrays + binary search)
* If key maps to multiple PURLs:
* return multiple candidates (same resolver id), forcing `Ambiguous` verdict upstream
Candidate:
* Confidence = `0.95` (still deterministic)
* Evidence includes index manifest digest + record evidence
#### Resolver C: SHA‑256 index (Layer 2)
Inputs:
* `identity.Sha256`
* `feeds/binary-map/v1/sha256.map.zst` OR your ops “curated binaries” manifest
Candidate:
* Confidence:
* `0.9` if from signed curated manifest
* `0.7` if from “observed in previous scan cache” (I’d avoid this unless you version and sign the cache)
#### Resolver D: Dependency closure constraints (Layer 3)
Only run if you have native dependency parsing output (DT_NEEDED / imports). The resolver does **not** return a mapping on its own; instead, it can:
* bump confidence for existing candidates
* or rule out candidates deterministically (e.g., glibc baseline mismatch)
Make this a “candidate rewriter” stage:
```csharp
public interface ICandidateRefiner
{
string Id { get; }
int Order { get; }
IReadOnlyList<BinaryMappingCandidate> Refine(BinaryIdentity id, IReadOnlyList<BinaryMappingCandidate> cands, MappingContext ctx);
}
```
#### Resolver E: Heuristic hints (Layer 4)
Never resolves to a PURL by default. It just produces Unknown evidence payload:
* extracted strings (“OpenSSL 3.0.11”)
* imported symbol names
* SONAME
* symbol version requirements
---
## 6) SBOM composition behavior: how to “lift” bin components safely
### 6.1 Don’t break the component key rules
Scanner uses:
* key = PURL when present, else `bin:{sha256}` ([Stella Ops][1])
When you resolve a binary identity to a PURL, you have two clean options:
**Option 1 (recommended): replace the component key with the PURL**
* This makes downstream policy/advisory matching work naturally.
* It’s deterministic as long as the mapping index is versioned and shipped with the kit.
**Option 2: keep `bin:{sha256}` as the component key and attach `resolved_purl`**
* Lower disruption to diffing, but policy now has to understand the “resolved_purl” field.
* If StellaOps policy assumes `component.purl` is the canonical key, this will cause pain.
Given StellaOps emphasizes PURLs as the canonical key for identity, I’d implement **Option 1**, but record robust evidence + index digest.
### 6.2 Preserve file-level evidence
Even after lifting to PURL, keep evidence that ties the package identity back to file bytes:
* file path(s)
* sha256
* build-id/uuid
* mapping resolver id + index digest
This is what makes attestations verifiable and helps operators debug.
---
## 7) Unknowns integration: emit Unknowns whenever mapping isn’t decisive
The Unknowns Registry exists precisely for “unresolved symbol → package mapping”, “missing build-id”, “ambiguous purl”, etc. ([Gitea: Git with a cup of tea][2])
### 7.1 When to emit Unknowns
Emit Unknowns for:
1. `identity.BuildId == null` for ELF
* `unknown_type = missing_build_id`
* evidence: “ELF missing .note.gnu.build-id; using sha256 only”
2. Multiple candidates with different PURLs
* `unknown_type = version_conflict` (or `identity_gap`)
* evidence: list candidates + their evidence
3. Heuristic hints found but no authoritative mapping
* `unknown_type = identity_gap`
* evidence: imported symbols, strings, SONAME
### 7.2 How to compute `unknown_id` deterministically
Unknowns schema suggests:
* `unknown_id` is derived from sha256 over `(type + scope + evidence)` ([Gitea: Git with a cup of tea][2])
Do:
* stable JSON canonicalization of `scope` + `unknown_type` + `primary evidence fields`
* sha256
* prefix with `unk:sha256:<...>`
This guarantees idempotent ingestion behavior (`POST /unknowns/ingest` upsert). ([Gitea: Git with a cup of tea][2])
---
## 8) Packaging as a StellaOps plug-in (so ops can upgrade it offline)
### 8.1 Plug-in manifest
Scanner plug-ins use a `manifest.json` with `schemaVersion`, `id`, `entryPoint` (dotnet assembly + typeName), etc. ([Gitea: Git with a cup of tea][7])
Create something like:
```json
{
"schemaVersion": "1.0",
"id": "stellaops.analyzer.native.binarymap",
"displayName": "StellaOps Native Binary Mapper",
"version": "0.1.0",
"requiresRestart": true,
"entryPoint": {
"type": "dotnet",
"assembly": "StellaOps.Scanner.Analyzers.Native.BinaryMap.dll",
"typeName": "StellaOps.Scanner.Analyzers.Native.BinaryMap.BinaryMapPlugin"
},
"capabilities": [
"native-analyzer",
"binary-mapper",
"elf",
"pe",
"macho"
],
"metadata": {
"org.stellaops.analyzer.kind": "native",
"org.stellaops.restart.required": "true"
}
}
```
### 8.2 Worker loading
Mirror the pattern in `CompositeScanAnalyzerDispatcher`:
* add a catalog `INativeAnalyzerPluginCatalog`
* default directory: `plugins/scanner/analyzers/native`
* load directories with the same “seal last directory” behavior ([Gitea: Git with a cup of tea][4])
---
## 9) Tests and performance gates (what “done” looks like)
StellaOps has determinism tests and golden fixtures for analyzers; follow that style. ([Gitea: Git with a cup of tea][6])
### 9.1 Determinism tests
Create fixtures with:
* same binaries in different file order
* same binaries hardlinked/symlinked
* stripped ELF missing build-id
* multi-arch variants
Assert:
* mapping output JSON byte-for-byte stable
* unknown ids stable
* candidate ordering stable
### 9.2 “No fuzzy identity” guardrail tests
Add tests that:
* heuristic resolver never emits a `Resolved` verdict unless a feature flag is enabled
* ambiguous candidates never auto-select a winner
### 9.3 Performance budgets
For ops, you care about scan wall time. Adopt budgets like:
* identity extraction < 25ms / binary (native parsing)
* mapping lookup O(1) / binary (frozen dict) or O(log n) with sorted arrays
* index load time bounded (lazy load per worker start)
Track metrics:
* count resolved per layer
* count ambiguous/unresolved
* unknown density (ties into Unknowns Registry scoring later) ([Gitea: Git with a cup of tea][2])
---
## 10) Practical “ops” workflow: how to keep improving mapping safely
### 10.1 Add a feedback loop from Unknowns → index builder
Unknowns are your backlog:
* “missing build-id”
* “ambiguous mapping”
* “hash seen but not in index”
For each Unknown:
1. decide if it should be mapped in core (needs authoritative source)
2. if yes: add reference artifact to your **index builder input set**
3. rebuild the BinaryMap index
4. ship via Offline Kit update (signed)
### 10.2 Don’t let your index silently drift
Because determinism matters, treat the BinaryMap index like a feed:
* version it (`v1`, `v2`)
* sign it
* store index digest in scan evidence
That way you can explain: “This binary was mapped using binary-map/v1 digest XYZ”.
---
## 11) Minimal implementation checklist (if you want the shortest path to value)
If you only do 3 things, do these:
1. **Build‑ID extraction everywhere** (ELF/Mach‑O/PE) and always store it in evidence
(also emit Unknown when missing, as StellaOps expects) ([Gitea: Git with a cup of tea][8])
2. **Offline Build‑ID → PURL index** shipped in Offline Kit
(fits perfectly with the existing debug-store + kit pattern) ([Stella Ops][3])
3. **Deterministic resolver pipeline + Unknowns emission**
(so you improve mapping without introducing silent risk) ([Gitea: Git with a cup of tea][2])
---
If you tell me whether your main pain is **distroless**, **FROM scratch**, or **vendor‑bundled runtimes** (chromium/node/openssl/etc.), I can give you the best “Layer 1 index builder” recipe for that category (what to use as authoritative sources and how to avoid collisions) — but the plan above is already safe and implementable without further assumptions.
[1]: https://stella-ops.org/docs/modules/scanner/architecture/ "Stella Ops – Signed Reachability · Deterministic Replay · Sovereign Crypto"
[2]: https://git.stella-ops.org/stella-ops.org/git.stella-ops.org/src/commit/d519782a8f0b30f425c9b6ae0f316b19259972a2/docs/signals/unknowns-registry.md "git.stella-ops.org/unknowns-registry.md at d519782a8f0b30f425c9b6ae0f316b19259972a2 - git.stella-ops.org - Gitea: Git with a cup of tea"
[3]: https://stella-ops.org/docs/24_offline_kit/index.html "Stella Ops – Signed Reachability · Deterministic Replay · Sovereign Crypto"
[4]: https://git.stella-ops.org/stella-ops.org/git.stella-ops.org/src/commit/18f28168f022c73736bfd29033c71daef5e11044/src/Scanner/StellaOps.Scanner.Worker/Processing/CompositeScanAnalyzerDispatcher.cs "git.stella-ops.org/CompositeScanAnalyzerDispatcher.cs at 18f28168f022c73736bfd29033c71daef5e11044 - git.stella-ops.org - Gitea: Git with a cup of tea"
[5]: https://git.stella-ops.org/stella-ops.org/git.stella-ops.org/src/commit/8d78dd219b5e44c835e511491a4750f4a3ee3640/vendor/manifest.json?utm_source=chatgpt.com "git.stella-ops.org/manifest.json at ..."
[6]: https://git.stella-ops.org/stella-ops.org/git.stella-ops.org/src/commit/bc0762e97d251723854b9c4e482b218c8efb1e04/docs/modules/scanner "git.stella-ops.org/scanner at bc0762e97d251723854b9c4e482b218c8efb1e04 - git.stella-ops.org - Gitea: Git with a cup of tea"
[7]: https://git.stella-ops.org/stella-ops.org/git.stella-ops.org/src/commit/c37722993137dac4b3a4104045826ca33b9dc289/plugins/scanner/analyzers/lang/StellaOps.Scanner.Analyzers.Lang.Go/manifest.json "git.stella-ops.org/manifest.json at c37722993137dac4b3a4104045826ca33b9dc289 - git.stella-ops.org - Gitea: Git with a cup of tea"
[8]: https://git.stella-ops.org/stella-ops.org/git.stella-ops.org/src/commit/d519782a8f0b30f425c9b6ae0f316b19259972a2/docs/reachability/evidence-schema.md?utm_source=chatgpt.com "git.stella-ops.org/evidence-schema.md at ..."

919
docs/product-advisories/18-Dec-2025 - Concrete Advances in Reachability Analysis.md Normal file
View File

@@ -0,0 +1,919 @@
Here’s a compact, practical way to add two high‑leverage capabilities to your scanner: **DSSE‑signed path witnesses** and **Smart‑Diff × Reachability**—what they are, why they matter, and exactly how to implement them in Stella Ops without ceremony.
---
# 1) DSSE‑signed path witnesses (entrypoint → calls → sink)
**What it is (in plain terms):**
When you flag a CVE as “reachable,” also emit a tiny, human‑readable proof: the **exact path** from a real entrypoint (e.g., HTTP route, CLI verb, cron) through functions/methods to the **vulnerable sink**. Wrap that proof in a **DSSE** envelope and sign it. Anyone can verify the witness later—offline—without rerunning analysis.
**Why it matters:**
* Turns red flags into **auditable evidence** (quiet‑by‑design).
* Lets CI/CD, auditors, and customers **verify** findings independently.
* Enables **deterministic replay** and provenance chains (ties nicely to in‑toto/SLSA).
**Minimal JSON witness (stable, vendor‑neutral):**
```json
{
"witness_schema": "stellaops.witness.v1",
"artifact": { "sbom_digest": "sha256:...", "component_purl": "pkg:nuget/Example@1.2.3" },
"vuln": { "id": "CVE-2024-XXXX", "source": "NVD", "range": "≤1.2.3" },
"entrypoint": { "kind": "http", "name": "GET /billing/pay" },
"path": [
{"symbol": "BillingController.Pay()", "file": "BillingController.cs", "line": 42},
{"symbol": "PaymentsService.Authorize()", "file": "PaymentsService.cs", "line": 88},
{"symbol": "LibXYZ.Parser.Parse()", "file": "Parser.cs", "line": 17}
],
"sink": { "symbol": "LibXYZ.Parser.Parse()", "type": "deserialization" },
"evidence": {
"callgraph_digest": "sha256:...",
"build_id": "dotnet:RID:linux-x64:sha256:...",
"analysis_config_digest": "sha256:..."
},
"observed_at": "2025-12-18T00:00:00Z"
}
```
**Wrap in DSSE (payloadType & payload are required)**
```json
{
"payloadType": "application/vnd.stellaops.witness+json",
"payload": "base64(JSON_above)",
"signatures": [{ "keyid": "attestor-stellaops-ed25519", "sig": "base64(...)" }]
}
```
**.NET 10 signing/verifying (Ed25519)**
```csharp
using System.Security.Cryptography;
using System.Text.Json;
var payloadBytes = JsonSerializer.SerializeToUtf8Bytes(witnessJsonObj);
var dsse = new {
payloadType = "application/vnd.stellaops.witness+json",
payload = Convert.ToBase64String(payloadBytes),
signatures = new [] { new { keyid = keyId, sig = Convert.ToBase64String(Sign(payloadBytes, privateKey)) } }
};
byte[] Sign(byte[] data, byte[] privateKey)
{
using var ed = new Ed25519();
// import private key, sign data (left as your Ed25519 helper)
return ed.SignData(data, privateKey);
}
```
**Where to emit:**
* **Scanner.Worker**: after reachability confirms `reachable=true`, emit witness → **Attestor** signs → **Authority** stores (Postgres) → optional Rekor‑style mirror.
* Expose `/witness/{findingId}` for download & independent verification.
---
# 2) Smart‑Diff × Reachability (incremental, low‑noise updates)
**What it is:**
On **SBOM/VEX/dependency** deltas, don’t rescan everything. Update only **affected regions** of the call graph and recompute reachability **just for changed nodes/edges**.
**Why it matters:**
* **Order‑of‑magnitude faster** incremental scans.
* Fewer flaky diffs; triage stays focused on **meaningful risk change**.
* Perfect for PR gating: “what changed” → “what became reachable/unreachable.”
**Core idea (graph‑reachability):**
* Maintain a per‑service **call graph** `G = (V, E)` with **entrypoint set** `S`.
* On diff: compute changed nodes/edges ΔV/ΔE.
* Run **incremental BFS/DFS** from impacted nodes to sinks (forward or backward), reusing memoized results.
* Recompute only **frontiers** touched by Δ.
**Minimal tables (Postgres):**
```sql
-- Nodes (functions/methods)
CREATE TABLE cg_nodes(
id BIGSERIAL PRIMARY KEY,
service TEXT, symbol TEXT, file TEXT, line INT,
hash TEXT, UNIQUE(service, hash)
);
-- Edges (calls)
CREATE TABLE cg_edges(
src BIGINT REFERENCES cg_nodes(id),
dst BIGINT REFERENCES cg_nodes(id),
kind TEXT, PRIMARY KEY(src, dst)
);
-- Entrypoints & Sinks
CREATE TABLE cg_entrypoints(node_id BIGINT REFERENCES cg_nodes(id) PRIMARY KEY);
CREATE TABLE cg_sinks(node_id BIGINT REFERENCES cg_nodes(id) PRIMARY KEY, sink_type TEXT);
-- Memoized reachability cache
CREATE TABLE cg_reach_cache(
entry_id BIGINT, sink_id BIGINT,
path JSONB, reachable BOOLEAN,
updated_at TIMESTAMPTZ,
PRIMARY KEY(entry_id, sink_id)
);
```
**Incremental algorithm (pseudocode):**
```text
Input: ΔSBOM, ΔDeps, ΔCode → ΔNodes, ΔEdges
1) Apply Δ to cg_nodes/cg_edges
2) ImpactSet = neighbors(ΔNodes ∪ endpoints(ΔEdges))
3) For each e∈Entrypoints intersect ancestors(ImpactSet):
Recompute forward search to affected sinks, stop early on unchanged subgraphs
Update cg_reach_cache; if state flips, emit new/updated DSSE witness
```
**.NET 10 reachability sketch (fast & local):**
```csharp
HashSet<int> ImpactSet = ComputeImpact(deltaNodes, deltaEdges);
foreach (var e in Intersect(Entrypoints, Ancestors(ImpactSet)))
{
var res = BoundedReach(e, affectedSinks, graph, cache);
foreach (var r in res.Changed)
{
cache.Upsert(e, r.Sink, r.Path, r.Reachable);
if (r.Reachable) EmitDsseWitness(e, r.Sink, r.Path);
}
}
```
**CI/PR flow:**
1. Build → SBOM diff → Dependency diff → Call‑graph delta.
2. Run incremental reachability.
3. If any `unreachable→reachable` transitions: **fail gate**, attach DSSE witnesses.
4. If `reachable→unreachable`: auto‑close prior findings (and archive prior witness).
---
# UX hooks (quick wins)
* In findings list, add a **“Show Witness”** button → modal renders the signed path (entrypoint→…→sink) + **“Verify Signature”** one‑click.
* In PR checks, summarize only **state flips** with tiny links: “+2 reachable (view witness)” / “−1 (now unreachable)”.
---
# Minimal tasks to get this live
* **Scanner.Worker**: build call‑graph extraction (per language), add incremental graph store, reachability cache.
* **Attestor**: DSSE signing endpoint + key management (Ed25519 by default; PQC mode later).
* **Authority**: tables above + witness storage + retrieval API.
* **Router/CI plugin**: PR annotation with **state flips** and links to witnesses.
* **UI**: witness modal + signature verify.
If you want, I can draft the exact Postgres migrations, the C# repositories, and a tiny verifier CLI that checks DSSE signatures and prints the call path.
Below is a concrete, buildable blueprint for an **advanced reachability analysis engine** inside Stella Ops. I’m going to assume your “Stella Ops” components are roughly:
* **Scanner.Worker**: runs analyses in CI / on artifacts
* **Authority**: stores graphs/findings/witnesses
* **Attestor**: signs DSSE envelopes (Ed25519)
* (optional) **SurfaceBuilder**: background worker that computes “vuln surfaces” for packages
The key advance is: **don’t treat a CVE as “a package”**. Treat it as a **set of trigger methods** (public API) that can reach the vulnerable code inside the dependency—computed by “Smart‑Diff” once, reused everywhere.
---
## 0) Define the contract (precision/soundness) up front
If you don’t write this down, you’ll fight false positives/negatives forever.
### What Stella Ops will guarantee (first release)
* **Whole-program static call graph** (app + selected dependency assemblies)
* **Context-insensitive** (fast), **path witness** extracted (shortest path)
* **Dynamic dispatch handled** with CHA/RTA (+ DI hints), with explicit uncertainty flags
* **Reflection handled best-effort** (constant-string resolution), otherwise “unknown edge”
### What it will NOT guarantee (first release)
* Perfect handling of reflection / `dynamic` / runtime codegen
* Perfect delegate/event resolution across complex flows
* Full taint/dataflow reachability (you can add later)
This is fine. The major value is: “**we can show you the call path**” and “**we can prove the vuln is triggered by calling these library APIs**”.
---
## 1) The big idea: “Vuln surfaces” (Smart-Diff → triggers)
### Problem
CVE feeds typically say “package X version range Y is vulnerable” but rarely say *which methods*. If you only do package-level reachability, noise is huge.
### Solution
For each CVE+package, compute a **vulnerability surface**:
* **Candidate sinks** = methods changed between vulnerable and fixed versions (diff at IL level)
* **Trigger methods** = *public/exported* methods in the vulnerable version that can reach those changed methods internally
Then your service scan becomes:
> “Can any entrypoint reach any trigger method?”
This is both faster and more precise.
---
## 2) Data model (Authority / Postgres)
You already had call graph tables; here’s a concrete schema that supports:
* graph snapshots
* incremental updates
* vuln surfaces
* reachability cache
* DSSE witnesses
### 2.1 Graph tables
```sql
CREATE TABLE cg_snapshots (
snapshot_id BIGSERIAL PRIMARY KEY,
service TEXT NOT NULL,
build_id TEXT NOT NULL,
graph_digest TEXT NOT NULL,
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
UNIQUE(service, build_id)
);
CREATE TABLE cg_nodes (
node_id BIGSERIAL PRIMARY KEY,
snapshot_id BIGINT REFERENCES cg_snapshots(snapshot_id) ON DELETE CASCADE,
method_key TEXT NOT NULL, -- stable key (see below)
asm_name TEXT,
type_name TEXT,
method_name TEXT,
file_path TEXT,
line_start INT,
il_hash TEXT, -- normalized IL hash for diffing
flags INT NOT NULL DEFAULT 0, -- bitflags: has_reflection, compiler_generated, etc.
UNIQUE(snapshot_id, method_key)
);
CREATE TABLE cg_edges (
snapshot_id BIGINT REFERENCES cg_snapshots(snapshot_id) ON DELETE CASCADE,
src_node_id BIGINT REFERENCES cg_nodes(node_id) ON DELETE CASCADE,
dst_node_id BIGINT REFERENCES cg_nodes(node_id) ON DELETE CASCADE,
kind SMALLINT NOT NULL, -- 0=call,1=newobj,2=dispatch,3=delegate,4=reflection_guess,...
PRIMARY KEY(snapshot_id, src_node_id, dst_node_id, kind)
);
CREATE TABLE cg_entrypoints (
snapshot_id BIGINT REFERENCES cg_snapshots(snapshot_id) ON DELETE CASCADE,
node_id BIGINT REFERENCES cg_nodes(node_id) ON DELETE CASCADE,
kind TEXT NOT NULL, -- http, grpc, cli, job, etc.
name TEXT NOT NULL, -- GET /foo, "Main", etc.
PRIMARY KEY(snapshot_id, node_id, kind, name)
);
```
### 2.2 Vuln surface tables (Smart‑Diff artifacts)
```sql
CREATE TABLE vuln_surfaces (
surface_id BIGSERIAL PRIMARY KEY,
ecosystem TEXT NOT NULL, -- nuget
package TEXT NOT NULL,
cve_id TEXT NOT NULL,
vuln_version TEXT NOT NULL, -- a representative vulnerable version
fixed_version TEXT NOT NULL,
surface_digest TEXT NOT NULL,
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
UNIQUE(ecosystem, package, cve_id, vuln_version, fixed_version)
);
CREATE TABLE vuln_surface_sinks (
surface_id BIGINT REFERENCES vuln_surfaces(surface_id) ON DELETE CASCADE,
sink_method_key TEXT NOT NULL,
reason TEXT NOT NULL, -- changed|added|removed|heuristic
PRIMARY KEY(surface_id, sink_method_key)
);
CREATE TABLE vuln_surface_triggers (
surface_id BIGINT REFERENCES vuln_surfaces(surface_id) ON DELETE CASCADE,
trigger_method_key TEXT NOT NULL,
sink_method_key TEXT NOT NULL,
internal_path JSONB, -- optional: library internal witness path
PRIMARY KEY(surface_id, trigger_method_key, sink_method_key)
);
```
### 2.3 Reachability cache & witnesses
```sql
CREATE TABLE reach_findings (
finding_id BIGSERIAL PRIMARY KEY,
snapshot_id BIGINT REFERENCES cg_snapshots(snapshot_id) ON DELETE CASCADE,
cve_id TEXT NOT NULL,
ecosystem TEXT NOT NULL,
package TEXT NOT NULL,
package_version TEXT NOT NULL,
reachable BOOLEAN NOT NULL,
reachable_entrypoints INT NOT NULL DEFAULT 0,
updated_at TIMESTAMPTZ NOT NULL DEFAULT now(),
UNIQUE(snapshot_id, cve_id, package, package_version)
);
CREATE TABLE reach_witnesses (
witness_id BIGSERIAL PRIMARY KEY,
finding_id BIGINT REFERENCES reach_findings(finding_id) ON DELETE CASCADE,
entry_node_id BIGINT REFERENCES cg_nodes(node_id),
dsse_envelope JSONB NOT NULL,
created_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
```
---
## 3) Stable identity: MethodKey + IL hash
### 3.1 MethodKey (must be stable across builds)
Use a normalized string like:
```
{AssemblyName}|{DeclaringTypeFullName}|{MethodName}`{GenericArity}({ParamType1},{ParamType2},...)
```
Examples:
* `MyApp|BillingController|Pay(System.String)`
* `LibXYZ|LibXYZ.Parser|Parse(System.ReadOnlySpan<System.Byte>)`
### 3.2 Normalized IL hash (for smart-diff + incremental graph updates)
Raw IL bytes aren’t stable (metadata tokens change). Normalize:
* opcode names
* branch targets by *instruction index*, not offset
* method operands by **resolved MethodKey**
* string operands by literal or hashed literal
* type operands by full name
Then hash `SHA256(normalized_bytes)`.
---
## 4) Call graph extraction for .NET (concrete, doable)
### Tooling choice
Start with **Mono.Cecil** (MIT license, easy IL traversal). You can later swap to `System.Reflection.Metadata` for speed.
### 4.1 Build process (Scanner.Worker)
1. `dotnet restore` (use your locked restore)
2. `dotnet build -c Release /p:DebugType=portable /p:DebugSymbols=true`
3. Collect:
* app assemblies: `bin/Release/**/publish/*.dll` or build output
* `.pdb` files for sequence points (file/line for witnesses)
### 4.2 Cecil loader
```csharp
var rp = new ReaderParameters {
ReadSymbols = true,
SymbolReaderProvider = new PortablePdbReaderProvider()
};
var asm = AssemblyDefinition.ReadAssembly(dllPath, rp);
```
### 4.3 Node extraction (methods)
Walk all types, including nested:
```csharp
IEnumerable<TypeDefinition> AllTypes(ModuleDefinition m)
{
var stack = new Stack<TypeDefinition>(m.Types);
while (stack.Count > 0)
{
var t = stack.Pop();
yield return t;
foreach (var nt in t.NestedTypes) stack.Push(nt);
}
}
foreach (var type in AllTypes(asm.MainModule))
foreach (var method in type.Methods)
{
var key = MethodKey.From(method); // your normalizer
var (file, line) = PdbFirstSequencePoint(method);
var ilHash = method.HasBody ? ILFingerprint(method) : null;
// store node (method_key, file, line, il_hash, flags...)
}
```
### 4.4 Edge extraction (direct calls)
```csharp
foreach (var method in type.Methods.Where(m => m.HasBody))
{
var srcKey = MethodKey.From(method);
foreach (var ins in method.Body.Instructions)
{
if (ins.Operand is MethodReference mr)
{
if (ins.OpCode.Code is Code.Call or Code.Callvirt or Code.Newobj)
{
var dstKey = MethodKey.From(mr); // important: stable even if not resolved
edges.Add(new Edge(srcKey, dstKey, kind: CallKind.Direct));
}
if (ins.OpCode.Code is Code.Ldftn or Code.Ldvirtftn)
{
// delegate capture (handle later)
}
}
}
}
```
---
## 5) Advanced precision: dynamic dispatch + DI + async/await
If you stop at direct edges only, you’ll miss many real paths.
### 5.1 Async/await mapping (critical for readable witnesses)
Async methods compile into a state machine `MoveNext()`. You want edges attributed back to the original method.
In Cecil:
* Check `AsyncStateMachineAttribute` on a method
* It references a state machine type
* Find that type’s `MoveNext` method
* Map `MoveNextKey -> OriginalMethodKey`
Then, while extracting edges:
```csharp
srcKey = MoveNextToOriginal.TryGetValue(srcKey, out var original) ? original : srcKey;
```
Do the same for iterator state machines.
### 5.2 Virtual/interface dispatch (CHA/RTA)
You need 2 maps:
1. **type hierarchy / interface impl map**
2. **override map** from “declared method” → “implementation method(s)”
**Build override map**
```csharp
// For each method, Cecil exposes method.Overrides for explicit implementations.
overrideMap[MethodKey.From(overrideRef)] = MethodKey.From(methodDef);
```
**CHA**: for callvirt to virtual method `T.M`, add edges to overrides in derived classes
**RTA**: restrict to derived classes that are actually instantiated.
How to get instantiated types:
* look for `newobj` instructions and add the created type to `InstantiatedTypes`
* plus DI registrations (below)
### 5.3 DI hints (Microsoft.Extensions.DependencyInjection)
You will see calls like:
* `ServiceCollectionServiceExtensions.AddTransient<TService, TImpl>(...)`
In IL these are generic method calls. Detect and record `TService -> TImpl` as “instantiated”. This massively improves RTA for modern .NET apps.
### 5.4 Delegates/lambdas (good enough approach)
Implement intraprocedural tracking:
* when you see `ldftn SomeMethod` then `newobj Action::.ctor` then `stloc.s X`
* store `delegateTargets[local X] += SomeMethod`
* when you see `ldloc.s X` and later `callvirt Invoke`, add edges to targets
This makes Minimal API entrypoint discovery work too.
### 5.5 Reflection (best-effort)
Implement only high-signal heuristics:
* `typeof(T).GetMethod("Foo")` with constant "Foo"
* `GetType().GetMethod("Foo")` with constant "Foo" (type unknown → mark uncertain)
If resolved, add edge with `kind=reflection_guess`.
If not, set node flag `has_reflection = true` and in results show “may be incomplete”.
---
## 6) Entrypoint detection (concrete detectors)
### 6.1 MVC controllers
Detect:
* types deriving from `Microsoft.AspNetCore.Mvc.ControllerBase`
* methods:
* public
* not `[NonAction]`
* has `[HttpGet]`, `[HttpPost]`, `[Route]` etc.
Extract route template from attributes’ ctor arguments.
Store in `cg_entrypoints`:
* kind = `http`
* name = `GET /billing/pay` (compose verb+template)
### 6.2 Minimal APIs
Scan `Program.Main` IL:
* find calls to `MapGet`, `MapPost`, ...
* extract route string from preceding `ldstr`
* resolve handler method via delegate tracking (ldftn)
Entry:
* kind = `http`
* name = `GET /foo`
### 6.3 CLI
Find assembly entry point method (`asm.EntryPoint`) or `static Main`.
Entry:
* kind = `cli`
* name = `Main`
Start here. Add gRPC/jobs later.
---
## 7) Smart-Diff SurfaceBuilder (the “advanced” part)
This is what makes your reachability actually meaningful for CVEs.
### 7.1 SurfaceBuilder inputs
From your vuln ingestion pipeline:
* ecosystem = nuget
* package = `LibXYZ`
* affected range = `<= 1.2.3`
* fixed version = `1.2.4`
* CVE id
### 7.2 Choose a vulnerable version to diff
Pick the **highest affected version below fixed**.
* fixed = 1.2.4
* vulnerable representative = 1.2.3
(If multiple fixed versions exist, build multiple surfaces.)
### 7.3 Download both packages
Use NuGet.Protocol to download `.nupkg`, unzip, pick TFMs you care about (often `netstandard2.0` is safest). Compute fingerprints for each assembly.
### 7.4 Compute method fingerprints
For each method:
* MethodKey
* Normalized IL hash
### 7.5 Diff
```
ChangedMethods = { k | hashVuln[k] != hashFixed[k] } ∪ added ∪ removed
```
Store these as `vuln_surface_sinks` with reason.
### 7.6 Build internal library call graph
Same Cecil extraction, but only for package assemblies.
Now compute triggers:
**Reverse BFS from sinks**:
* Start from all sink method keys
* Walk predecessors
* When you encounter a **public/exported method**, record it as a trigger
Also store one internal path for each trigger → sink (for witnesses).
### 7.7 Add interface/base declarations as triggers
Important: your app might call a library via an interface method signature, not the concrete implementation.
For each trigger implementation method:
* for each `method.Overrides` entry, add the overridden method key as an additional trigger
This reduces dependence on perfect dispatch expansion during app scanning.
### 7.8 Persist the surface
Store:
* sinks set
* triggers set
* internal witness paths (optional but highly valuable)
Now you’ve converted a “version range” CVE into “these specific library APIs are dangerous”.
---
## 8) Reachability engine (fast, witness-producing)
### 8.1 In-memory graph format (CSR)
Don’t BFS off dictionaries; you’ll die on perf.
Build integer indices:
* `method_key -> nodeIndex (0..N-1)`
* store arrays:
* `predOffsets[N+1]`
* `preds[edgeCount]`
Construction:
1. count predecessors per node
2. prefix sum to offsets
3. fill preds
### 8.2 Reverse BFS from sinks
This computes:
* `visited[node]` = can reach a sink
* `parent[node]` = next node toward a sink (for path reconstruction)
```csharp
public sealed class ReachabilityEngine
{
public ReachabilityResult Compute(
Graph g,
ReadOnlySpan<int> entrypoints,
ReadOnlySpan<int> sinks)
{
var visitedMark = g.VisitMark; // int[] length N (reused across runs)
var parent = g.Parent; // int[] length N (reused)
g.RunId++;
var q = new IntQueue(capacity: g.NodeCount);
var sinkSet = new BitSet(g.NodeCount);
foreach (var s in sinks)
{
sinkSet.Set(s);
visitedMark[s] = g.RunId;
parent[s] = s;
q.Enqueue(s);
}
while (q.TryDequeue(out var v))
{
var start = g.PredOffsets[v];
var end = g.PredOffsets[v + 1];
for (int i = start; i < end; i++)
{
var p = g.Preds[i];
if (visitedMark[p] == g.RunId) continue;
visitedMark[p] = g.RunId;
parent[p] = v;
q.Enqueue(p);
}
}
// Collect reachable entrypoints and paths
var results = new List<EntryWitness>();
foreach (var e in entrypoints)
{
if (visitedMark[e] != g.RunId) continue;
var path = ReconstructPath(e, parent, sinkSet);
results.Add(new EntryWitness(e, path));
}
return new ReachabilityResult(results);
}
private static int[] ReconstructPath(int entry, int[] parent, BitSet sinks)
{
var path = new List<int>(32);
int cur = entry;
path.Add(cur);
// follow parent pointers until a sink
for (int guard = 0; guard < 10_000; guard++)
{
if (sinks.Get(cur)) break;
var nxt = parent[cur];
if (nxt == cur || nxt < 0) break; // safety
cur = nxt;
path.Add(cur);
}
return path.ToArray();
}
}
```
### 8.3 Producing the witness
For each node index in the path:
* method_key
* file_path / line_start (if known)
* optional flags (reflection_guess edge, dispatch edge)
Then attach:
* vuln id, package, version
* entrypoint kind/name
* graph digest + config digest
* surface digest
* timestamp
Send JSON to Attestor for DSSE signing, store envelope in Authority.
---
## 9) Scaling: don’t do BFS 500 times if you can avoid it
### 9.1 First-line scaling (usually enough)
* Group vulnerabilities by package/version → surfaces reused
* Only run reachability for vulns where:
* dependency present AND
* surface exists OR fallback mode
* Limit witnesses per vuln (top 3)
In practice, with N~50k nodes and E~200k edges, a reverse BFS is fast in C# if done with arrays.
### 9.2 Incremental Smart-Diff × Reachability (your “low noise” killer feature)
#### Step A: compute graph delta between snapshots
Use `il_hash` per method to detect changed nodes:
* added / removed / changed nodes
* edges updated only for changed nodes
#### Step B: decide which vulnerabilities need recompute
Store a cached reverse-reachable set per vuln surface if you want (bitset), OR just do a cheaper heuristic:
Recompute for vulnerability if:
* sink set changed (new surface or version changed), OR
* any changed node is on any previously stored witness path, OR
* entrypoints changed, OR
* impacted nodes touch any trigger node’s predecessors (use a small localized search)
A practical approach:
* store all node IDs that appear in any witness path for that vuln
* if delta touches any of those nodes/edges, recompute
* otherwise reuse cached result
This yields a massive win on PR scans where most code is unchanged.
#### Step C: “Impact frontier” recompute (optional)
If you want more advanced:
* compute `ImpactSet = ΔNodes ∪ endpoints(ΔEdges)`
* run reverse BFS **starting from ImpactSet ∩ ReverseReachSet** and update visited marks
This is trickier to implement correctly (dynamic graph), so I’d ship the heuristic first.
---
## 10) Practical fallback modes (don’t block shipping)
You won’t have surfaces for every CVE on day 1. Handle this gracefully:
### Mode 1: Surface-based reachability (best)
* sink = trigger methods from surface
* result: “reachable” with path
### Mode 2: Package API usage (good fallback)
* sink = *any* method in that package that is called by app
* result: “package reachable” (lower confidence), still provide path to callsite
### Mode 3: Dependency present only (SBOM level)
* no call graph needed
* result: “present” only
Your UI can show confidence tiers:
* **Confirmed reachable (surface)**
* **Likely reachable (package API)**
* **Present only (SBOM)**
---
## 11) Integration points inside Stella Ops
### Scanner.Worker (per build)
1. Build/collect assemblies + pdb
2. `CallGraphBuilder` → nodes/edges/entrypoints + graph_digest
3. Load SBOM vulnerabilities list
4. For each vuln:
* resolve surface triggers; if missing → enqueue SurfaceBuilder job + fallback mode
* run reachability BFS
* for each reachable entrypoint: emit DSSE witness
5. Persist findings/witnesses
### SurfaceBuilder (async worker)
* triggered by “surface missing” events or nightly preload of top packages
* computes surface once, stores forever
### Authority
* stores graphs, surfaces, findings, witnesses
* provides retrieval APIs for UI/CI
---
## 12) What to implement first (in the order that produces value fastest)
### Week 1–2 scope (realistic, shippable)
1. Cecil call graph extraction (direct calls)
2. MVC + Minimal API entrypoints
3. Reverse BFS reachability with path witnesses
4. DSSE witness signing + storage
5. SurfaceBuilder v1:
* IL hash per method
* changed methods as sinks
* triggers via internal reverse BFS
6. UI: “Show Witness” + “Verify Signature”
### Next increment (precision upgrades)
7. async/await mapping to original methods
8. RTA + DI registration hints
9. delegate tracking for Minimal API handlers (if not already)
10. interface override triggers in surface builder
### Later (if you want “attackability”, not just “reachability”)
11. taint/dataflow for top sink classes (deserialization, path traversal, SQL, command exec)
12. sanitizer modeling & parameter constraints
---
## 13) Common failure modes and how to harden
### MethodKey mismatches (surface vs app call)
* Ensure both are generated from the same normalization rules
* For generic methods, prefer **definition** keys (strip instantiation)
* Store both “exact” and “erased generic” variants if needed
### Multi-target frameworks
* SurfaceBuilder: compute triggers for each TFM, union them
* App scan: choose TFM closest to build RID, but allow fallback to union
### Huge graphs
* Drop `System.*` nodes/edges unless:
* the vuln is in System.* (rare, but handle separately)
* Deduplicate nodes by MethodKey across assemblies where safe
* Use CSR arrays + pooled queues
### Reflection heavy projects
* Mark analysis confidence lower
* Include “unknown edges present” in finding metadata
* Still produce a witness path up to the reflective callsite
---
If you want, I can also paste a **complete Cecil-based CallGraphBuilder class** (nodes+edges+PDB lines), plus the **SurfaceBuilder** that downloads NuGet packages and generates `vuln_surface_triggers` end-to-end.

751
docs/product-advisories/18-Dec-2025 - Designing Explainable Triage and Proof‑Linked Evidence.md Normal file
View File

@@ -0,0 +1,751 @@
Here’s a practical, first‑time‑friendly blueprint for making your security workflow both **explainable** and **provable**—from triage to approval.
# Explainable triage UX (what & why)
Show every risk score with the minimum evidence a responder needs to trust it:
* **Reachable path:** the concrete call‑chain (or network path) proving the vuln is actually hit.
* **Entrypoint boundary:** the external surface (HTTP route, CLI verb, cron, message topic) that leads to that path.
* **VEX status:** the exploitability decision (Affected/Not Affected/Under Investigation/Fixed) with rationale.
* **Last‑seen timestamp:** when this evidence was last observed/generated.
## UI pattern (compact, 1‑click expand)
* **Row (collapsed):** `Score 72 • CVE‑2024‑12345 • service: api-gateway • package: x.y.z`
* **Expand panel (evidence):**
* **Path:** `POST /billing/charge → BillingController.Pay() → StripeClient.Create()`
* **Boundary:** `Ingress: /billing/charge (JWT: required, scope: payments:write)`
* **VEX:** `Not Affected (runtime guard strips untrusted input before sink)`
* **Last seen:** `2025‑12‑18T09:22Z` (scan: sbomer#c1a2, policy run: lattice#9f0d)
* **Actions:** “Open proof bundle”, “Re-run check”, “Create exception (time‑boxed)”
## Data contract (what the panel needs)
```json
{
"finding_id": "f-7b3c",
"cve": "CVE-2024-12345",
"component": {"name": "stripe-sdk", "version": "6.1.2"},
"reachable_path": [
"HTTP POST /billing/charge",
"BillingController.Pay",
"StripeClient.Create"
],
"entrypoint": {"type":"http","route":"/billing/charge","auth":"jwt:payments:write"},
"vex": {"status":"not_affected","justification":"runtime_sanitizer_blocks_sink","timestamp":"2025-12-18T09:22:00Z"},
"last_seen":"2025-12-18T09:22:00Z",
"attestation_refs": ["sha256:…sbom", "sha256:…vex", "sha256:…policy"]
}
```
# Evidence‑linked approvals (what & why)
Make “Approve to ship” contingent on **verifiable proof**, not screenshots:
* **Chain** must exist and be machine‑verifiable: **SBOM → VEX → policy decision**.
* Use **in‑toto/DSSE** attestations or **SLSA provenance** so each link has a signature, subject digest, and predicate.
* **Gate** merges/deploys only when the chain validates.
## Pipeline gate (simple policy)
* Require:
1. **SBOM attestation** referencing the exact image digest
2. **VEX attestation** covering all listed components (or explicit allow‑gaps)
3. **Policy decision attestation** (e.g., “risk ≤ threshold AND all reachable vulns = Not Affected/Fixed”)
### Minimal decision attestation (DSSE envelope → JSON payload)
```json
{
"predicateType": "stella/policy-decision@v1",
"subject": [{"name":"registry/org/app","digest":{"sha256":"<image-digest>"}}],
"predicate": {
"policy": "risk_threshold<=75 && reachable_vulns.all(v => v.vex in ['not_affected','fixed'])",
"inputs": {
"sbom_ref": "sha256:<sbom>",
"vex_ref": "sha256:<vex>"
},
"result": {"allowed": true, "score": 61, "exemptions":[]},
"evidence_refs": ["sha256:<reachability-proof-bundle>"],
"run_at": "2025-12-18T09:23:11Z"
}
}
```
# How this lands in your product (concrete moves)
* **Backend:** add `/findings/:id/evidence` (returns the contract above) + `/approvals/:artifact/attestations`.
* **Storage:** keep **proof bundles** (graphs, call stacks, logs) as content‑addressed blobs; store DSSE envelopes alongside.
* **UI:** one list → expandable rows; chips for VEX status; “Open proof” shows the call graph and boundary in 1 view.
* **CLI/API:** `stella verify image:<digest> --require sbom,vex,decision` returns a signed summary; pipelines fail on non‑zero.
* **Metrics:**
* **% changes with complete attestations** (target ≥95%)
* **TTFE (time‑to‑first‑evidence)** from alert → panel open (target ≤30s)
* **Post‑deploy reversions** due to missing proof (trend to zero)
# Starter acceptance checklist
* [ ] Every risk row expands to path, boundary, VEX, last‑seen in <300 ms.
* [ ] “Approve” button disabled until SBOM+VEX+Decision attestations validate for the **exact artifact digest**.
* [ ] One‑click “Show DSSE chain” renders the three envelopes with subject digests and signers.
* [ ] Audit log captures who approved, which digests, and which evidence hashes.
If you want, I can turn this into ready‑to‑drop **.NET 10** endpoints + a small React panel with mocked data so your team can wire it up fast.
Below is a “build‑it” guide for Stella Ops that goes past the concept level: concrete services, schemas, pipelines, signing/storage choices, UI components, and the exact invariants you should enforce so triage is **explainable** and approvals are **provably evidence‑linked**.
---
## 1) Start with the invariants (the rules your system must never violate)
If you implement nothing else, implement these invariants—they’re what make the UX trustworthy and the approvals auditable.
### Artifact anchoring invariant
Every finding, every piece of evidence, and every approval must be anchored to an immutable **subject digest** (e.g., container image digest `sha256:…`, binary SHA, or SBOM digest).
* No “latest tag” approvals.
* No “approve commit” without mapping to the built artifact digest.
### Evidence closure invariant
A policy decision is only valid if it references **exactly** the evidence it used:
* `inputs.sbom_ref`
* `inputs.vex_ref`
* `inputs.reachability_ref` (optional but recommended)
* `inputs.scan_ref` (optional)
* and any config/IaC refs used for boundary/exposure.
### Signature chain invariant
Evidence is only admissible if it is:
1. structured (machine readable),
2. signed (DSSE/in‑toto),
3. verifiable (trusted identity/keys),
4. retrievable by digest.
DSSE is specifically designed to authenticate both the message and its type (payload type) and avoid canonicalization pitfalls. ([GitHub][1])
### Staleness invariant
Evidence must have:
* `last_seen` and `expires_at` (or TTL),
* a “stale evidence” behavior in policy (deny or degrade score).
---
## 2) Choose the canonical formats and where you’ll store “proof”
### Attestation envelope: DSSE + in‑toto Statement
Use:
* **in‑toto Attestation Framework** “Statement” as the payload model (“subject + predicateType + predicate”). ([GitHub][2])
* Wrap it in **DSSE** for signing. ([GitHub][1])
* If you use Sigstore bundles, the DSSE envelope is expected to carry an in‑toto statement and uses `payloadType` like `application/vnd.in-toto+json`. ([Sigstore][3])
### SBOM format: CycloneDX or SPDX
* SPDX is an ISO/IEC standard and has v3.0 and v2.3 lines in the ecosystem. ([spdx.dev][4])
* CycloneDX is an ECMA standard (ECMA‑424) and widely used for application security contexts. ([GitHub][5])
Pick one as **your canonical** (internally), but ingest both.
### VEX format: OpenVEX (practical) + map to “classic” VEX statuses
VEX’s value is triage noise reduction: vendors can assert whether a product is affected, fixed, under investigation, or not affected. ([NTIA][6])
OpenVEX is a minimal, embeddable implementation of VEX intended for interoperability. ([GitHub][7])
### Where to store proof: OCI registry referrers
Use OCI “subject/referrers” so proofs travel with the artifact:
* OCI 1.1 introduces an explicit `subject` field and referrers graph for signatures/attestations/SBOMs. ([opencontainers.org][8])
* ORAS documentation explains linking artifacts via `subject`. ([Oras][9])
* Microsoft docs show `oras attach … --artifact-type …` patterns (works across registries that support referrers). ([Microsoft Learn][10])
---
## 3) System architecture (services + data flow)
### Services (minimum set)
1. **Ingestor**
* Pulls scanner outputs (SCA/SAST/IaC), SBOM, runtime signals.
2. **Evidence Builder**
* Computes reachability, entrypoints, boundary/auth context, score explanation.
3. **Attestation Service**
* Creates in‑toto statements, wraps DSSE, signs (cosign/KMS), stores to registry.
4. **Policy Engine**
* Evaluates allow/deny + reason codes, emits signed decision attestation.
* Use OPA/Rego for maintainable declarative policies. ([openpolicyagent.org][11])
5. **Stella Ops API**
* Serves findings + evidence panels to the UI (fast, cached).
6. **UI**
* Explainable triage panel + chain viewer + approve button.
### Event flow (artifact‑centric)
1. Build produces `image@sha256:X`
2. Generate SBOM → sign + attach
3. Run vuln scan → sign + attach (optional but useful)
4. Evidence Builder creates:
* reachability proof
* boundary proof
* vex doc (or imports vendor VEX + adds your context)
5. Policy engine evaluates → emits “decision attestation”
6. UI shows explainable triage + “approve” gating
---
## 4) Data model (the exact objects you need)
### Core IDs you should standardize
* `subject_digest`: `sha256:<image digest>`
* `subject_name`: `registry/org/app`
* `finding_key`: `(subject_digest, detector, cve, component_purl, location)` stable hash
* `component_purl`: package URL (PURL) canonical component identifier
### Tables (Postgres suggested)
**artifacts**
* `id (uuid)`
* `name`
* `digest` (unique)
* `created_at`
**findings**
* `id (uuid)`
* `artifact_digest`
* `cve`
* `component_purl`
* `severity`
* `raw_score`
* `risk_score`
* `status` (open/triaged/accepted/fixed)
* `first_seen`, `last_seen`
**evidence**
* `id (uuid)`
* `finding_id`
* `kind` (reachable_path | boundary | score_explain | vex | ...)
* `payload_json` (jsonb, small)
* `blob_ref` (content-addressed URI for big payloads)
* `last_seen`
* `expires_at`
* `confidence` (0–1)
* `source_attestation_digest` (nullable)
**attestations**
* `id (uuid)`
* `artifact_digest`
* `predicate_type`
* `attestation_digest` (sha256 of DSSE envelope)
* `signer_identity` (OIDC subject / cert identity)
* `issued_at`
* `registry_ref` (where attached)
**approvals**
* `id (uuid)`
* `artifact_digest`
* `decision_attestation_digest`
* `approver`
* `approved_at`
* `expires_at`
* `reason`
---
## 5) Explainable triage: how to compute the “Path + Boundary + VEX + Last‑seen”
### 5.1 Reachable path proof (call chain / flow)
You need a uniform reachability result type:
* `reachable = true` with an explicit path
* `reachable = false` with justification (e.g., symbol absent, dead code)
* `reachable = unknown` with reason (insufficient symbols, dynamic dispatch)
**Implementation strategy**
1. **Symbol mapping**: map CVE → vulnerable symbols/functions/classes
* Use one or more:
* vendor advisory → patched functions
* diff mining (commit that fixes CVE) to extract changed symbols
* curated mapping in your DB for high volume CVEs
2. **Program graph extraction** at build time:
* Produce a call graph or dependency graph per language.
* Store as compact adjacency list (or protobuf) keyed by `subject_digest`.
3. **Entrypoint discovery**:
* HTTP routes (framework metadata)
* gRPC service methods
* queue/stream consumers
* cron/CLI handlers
4. **Path search**:
* BFS/DFS from entrypoints to vulnerable symbols.
* Record the shortest path + top‑K alternatives.
5. **Proof bundle**:
* path nodes with stable IDs
* file hashes + line ranges (no raw source required)
* tool version + config hash
* graph digest
**Reachability evidence JSON (UI‑friendly)**
```json
{
"kind": "reachable_path",
"result": "reachable",
"confidence": 0.86,
"entrypoints": [
{"type":"http","route":"POST /billing/charge","auth":"jwt:payments:write"}
],
"paths": [{
"path_id": "p-1",
"steps": [
{"node":"BillingController.Pay","file_hash":"sha256:aaa","lines":[41,88]},
{"node":"StripeClient.Create","file_hash":"sha256:bbb","lines":[10,52]},
{"node":"stripe-sdk.vulnFn","symbol":"stripe-sdk::parseWebhook","evidence":"symbol-match"}
]
}],
"graph": {"digest":"sha256:callgraph...", "format":"stella-callgraph-v1"},
"last_seen": "2025-12-18T09:22:00Z",
"expires_at": "2025-12-25T09:22:00Z"
}
```
**UI rule:** never show “reachable” without a concrete, replayable path ID.
---
### 5.2 Boundary proof (the “why this is exposed” part)
Boundary proof answers: “Even if reachable, who can trigger it?”
**Data sources**
* Kubernetes ingress/service (exposure)
* API gateway routes and auth policies
* service mesh auth (mTLS, JWT)
* IAM policies (for cloud events)
* network policies (deny/allow)
**Boundary evidence schema**
```json
{
"kind": "boundary",
"surface": {"type":"http","route":"POST /billing/charge"},
"exposure": {"internet": true, "ports":[443]},
"auth": {
"mechanism":"jwt",
"required_scopes":["payments:write"],
"audience":"billing-api"
},
"rate_limits": {"enabled": true, "rps": 20},
"controls": [
{"type":"waf","status":"enabled"},
{"type":"input_validation","status":"enabled","location":"BillingController.Pay"}
],
"last_seen": "2025-12-18T09:22:00Z",
"confidence": 0.74
}
```
**How to build it**
* Create a “Surface Extractor” plugin per environment:
* `k8s-extractor`: reads ingress + service + annotations
* `gateway-extractor`: reads API gateway config
* `iac-extractor`: parses Terraform/CloudFormation
* Normalize into the schema above.
---
### 5.3 VEX in Stella: statuses + justifications
VEX statuses you should support in UI:
* Not affected
* Affected
* Fixed
* Under investigation ([NTIA][6])
OpenVEX will carry the machine readable structure. ([GitHub][7])
**Practical approach**
* Treat VEX as **the decision record** for exploitability.
* Your policy can require VEX coverage for all “reachable” high severity vulns.
**Rule of thumb**
* If `reachable=true` AND boundary shows reachable surface + auth weak → VEX defaults to `affected` until mitigations proven.
* If `reachable=false` with high confidence and stable proof → VEX may be `not_affected`.
---
### 5.4 Explainable risk score (don’t hide the formula)
Make score explainability first‑class.
**Recommended implementation**
* Store risk score as an additive model:
* `base = CVSS normalized`
* `+ reachability_bonus`
* `+ exposure_bonus`
* `+ privilege_bonus`
* `- mitigation_discount`
* Emit a `score_explain` evidence object:
```json
{
"kind": "score_explain",
"risk_score": 72,
"contributions": [
{"factor":"cvss","value":41,"reason":"CVSS 9.8"},
{"factor":"reachability","value":18,"reason":"reachable path p-1"},
{"factor":"exposure","value":10,"reason":"internet-facing route"},
{"factor":"auth","value":3,"reason":"scope required lowers impact"}
],
"last_seen":"2025-12-18T09:22:00Z"
}
```
**UI rule:** “Score 72” must always be clickable to a stable breakdown.
---
## 6) The UI you should build (components + interaction rules)
### 6.1 Findings list row (collapsed)
Show only what helps scanning:
* Score badge
* CVE + component
* service
* reachability chip: Reachable / Not reachable / Unknown
* VEX chip
* last_seen indicator (green/yellow/red)
### 6.2 Evidence drawer (expanded)
Tabs:
1. **Path**
* show entrypoint(s)
* render call chain (simple list first; graph view optional)
2. **Boundary**
* exposure, auth, controls
3. **VEX**
* status + justification + issuer identity
4. **Score**
* breakdown bar/list
5. **Proof**
* attestation chain viewer (SBOM → VEX → Decision)
* “Verify locally” action
### 6.3 “Open proof bundle” viewer
Must display:
* subject digest
* signer identity
* predicate type
* digest of proof bundle
* last_seen + tool versions
**This is where trust is built:** responders can see that the evidence is signed, tied to the artifact, and recent.
---
## 7) Proof‑linked evidence: how to generate and attach attestations
### 7.1 Statement format: in‑toto Attestation Framework
in‑toto’s model is:
* **Subjects** (the artifact digests)
* **Predicate type** (schema ID)
* **Predicate** (your actual data) ([GitHub][2])
### 7.2 DSSE envelope
Wrap statements using DSSE so payload type is signed too. ([GitHub][1])
### 7.3 Attach to OCI image via referrers
OCI “subject/referrers” makes attestations discoverable from the image digest. ([opencontainers.org][8])
ORAS provides the operational model (“attach artifacts to an image”). ([Microsoft Learn][10])
### 7.4 Practical signing: cosign attest + verify
Cosign has built‑in in‑toto attestation support and can sign custom predicates. ([Sigstore][12])
Typical patterns (example only; adapt to your environment):
```bash
# Attach an attestation
cosign attest --predicate reachability.json \
--type stella/reachability/v1 \
<image@sha256:digest>
# Verify attestation
cosign verify-attestation --type stella/reachability/v1 \
<image@sha256:digest>
```
(Use keyless OIDC or KMS keys depending on your org.)
---
## 8) Define your predicate types (this is the “contract” Stella enforces)
You’ll want at least these predicate types:
1. `stella/sbom@v1`
* embeds CycloneDX/SPDX (or references blob digest)
2. `stella/vex@v1`
* embeds OpenVEX document or references it ([GitHub][7])
3. `stella/reachability@v1`
* the reachability evidence above
* includes `graph.digest`, `paths`, `confidence`, `expires_at`
4. `stella/boundary@v1`
* exposure/auth proof and `last_seen`
5. `stella/policy-decision@v1`
* the gating result, references all input attestation digests
6. Optional: `stella/human-approval@v1`
* “I approve deploy of subject digest X based on decision attestation Y”
* keep it time‑boxed
---
## 9) The policy gate (how approvals become proof‑linked)
### 9.1 Use OPA/Rego for the gate
OPA policies are written in Rego. ([openpolicyagent.org][11])
**Gate input** should be a single JSON document assembled from verified attestations:
```json
{
"subject": {"name":"registry/org/app","digest":"sha256:..."},
"sbom": {...},
"vex": {...},
"reachability": {...},
"boundary": {...},
"org_policy": {"max_risk": 75, "max_age_hours": 168}
}
```
**Example Rego (deny‑by‑default)**
```rego
package stella.gate
default allow := false
# deny if evidence is stale
stale_evidence {
now := time.now_ns()
exp := time.parse_rfc3339_ns(input.reachability.expires_at)
now > exp
}
# deny if any high severity reachable vuln is not resolved by VEX
unresolved_reachable[v] {
v := input.reachability.findings[_]
v.severity in {"critical","high"}
v.reachable == true
not input.vex.resolution[v.cve] in {"not_affected","fixed"}
}
allow {
input.risk_score <= input.org_policy.max_risk
not stale_evidence
count(unresolved_reachable) == 0
}
```
### 9.2 Emit a signed policy decision attestation
When OPA returns `allow=true`, emit **another attestation**:
* predicate includes the policy version/hash and all input refs.
* that’s what the UI “Approve” button targets.
This is the “evidence‑linked approval”: approval references the signed decision, and the decision references the signed evidence.
---
## 10) “Approve” button behavior (what Stella Ops should enforce)
### Disabled until…
* subject digest known
* SBOM attestation found + signature verified
* VEX attestation found + signature verified
* Decision attestation found + signature verified
* Decision’s `inputs` digests match the actual retrieved evidence
### When clicked…
1. Stella Ops creates a `stella/human-approval@v1` statement:
* `subject` = artifact digest
* `predicate.decision_ref` = decision attestation digest
* `predicate.expires_at` = short TTL (e.g., 7–30 days)
2. Signs it with the approver identity
3. Attaches it to the artifact (OCI referrer)
### Audit view must show
* approver identity
* exact artifact digest
* exact decision attestation digest
* timestamp and expiry
---
## 11) Implementation details that matter in production
### 11.1 Verification library (shared by UI backend + CI gate)
Write one verifier module used everywhere:
**Inputs**
* image digest
* expected predicate types
* trust policy (allowed identities/issuers, keyless rules, KMS keys)
**Steps**
1. Discover referrers for `image@sha256:…`
2. Filter by `predicateType`
3. Verify DSSE + signature + identity
4. Validate JSON schema for predicate
5. Check `subject.digest` matches image digest
6. Return “verified evidence set” + “errors”
### 11.2 Evidence privacy
Reachability proofs can leak implementation details.
* Store file hashes, symbol names, and line ranges
* Gate raw source behind elevated permissions
* Provide redacted proofs by default
### 11.3 Evidence TTL strategy
* SBOM: long TTL (weeks/months) if digest immutable
* Boundary: short TTL (hours/days) because env changes
* Reachability: medium TTL (days/weeks) depending on code churn
* VEX: must be renewed if boundary/reachability changes
### 11.4 Handling “Unknown reachability”
Don’t force false certainty.
* Mark as `unknown` and show why (missing symbols, dynamic reflection, stripped binaries)
* Policy can treat unknown as “reachable” for critical CVEs in internet‑facing services.
---
## 12) A concrete MVP path that still delivers value
If you want a minimal but real first release:
### MVP (2–3 deliverables)
1. **Evidence drawer** fed by:
* scanner output + SBOM + a simple “entrypoint map”
2. **VEX workflow**
* allow engineers to set VEX status + justification
3. **Signed decision gating**
* even if reachability is heuristic, the chain is real
Then iterate:
* add reachability graphs
* add boundary extraction from IaC/K8s
* tighten policy (staleness, confidence thresholds)
---
## 13) Quick checklist for “done enough to trust”
* [ ] Every finding expands to: Path, Boundary, VEX, Score, Proof
* [ ] Every evidence tab shows `last_seen` + confidence
* [ ] “Verify chain” works: SBOM → VEX → Decision all signed and bound to the artifact digest
* [ ] Approve button signs a human approval attestation tied to the decision digest
* [ ] CI gate verifies the same chain before deploy
---
If you want, I can also drop in:
* a full set of JSON Schemas for `stella/*@v1` predicates,
* a reference verifier implementation outline in .NET 10 (Minimal API + a verifier class),
* and a sample UI component tree (React) that renders path/boundary graphs and attestation chains.
[1]: https://github.com/secure-systems-lab/dsse?utm_source=chatgpt.com "DSSE: Dead Simple Signing Envelope"
[2]: https://github.com/in-toto/attestation?utm_source=chatgpt.com "in-toto Attestation Framework"
[3]: https://docs.sigstore.dev/about/bundle/?utm_source=chatgpt.com "Sigstore Bundle Format"
[4]: https://spdx.dev/use/specifications/?utm_source=chatgpt.com "Specifications"
[5]: https://github.com/CycloneDX/specification?utm_source=chatgpt.com "CycloneDX/specification"
[6]: https://www.ntia.gov/sites/default/files/publications/vex_one-page_summary_0.pdf "VEX one-page summary"
[7]: https://github.com/openvex/spec?utm_source=chatgpt.com "OpenVEX Specification"
[8]: https://opencontainers.org/posts/blog/2024-03-13-image-and-distribution-1-1/?utm_source=chatgpt.com "OCI Image and Distribution Specs v1.1 Releases"
[9]: https://oras.land/docs/concepts/reftypes/?utm_source=chatgpt.com "Attached Artifacts | OCI Registry As Storage"
[10]: https://learn.microsoft.com/en-us/azure/container-registry/container-registry-manage-artifact?utm_source=chatgpt.com "Manage OCI Artifacts and Supply Chain Artifacts with ORAS"
[11]: https://openpolicyagent.org/docs/policy-language?utm_source=chatgpt.com "Policy Language"
[12]: https://docs.sigstore.dev/cosign/verifying/attestation/?utm_source=chatgpt.com "In-Toto Attestations"

869
docs/product-advisories/18-Dec-2025 - Designing a Layered EPSS v4 Database.md Normal file
View File

@@ -0,0 +1,869 @@
Here’s a compact, practical blueprint for bringing **EPSS** into your stack without chaos: a **3‑layer ingestion model** that keeps raw data, produces clean probabilities, and emits “signal‑ready” events your risk engine can use immediately.
---
# Why this matters (super short)
* **EPSS** = predicted probability a vuln will be exploited soon.
* Mixing “raw EPSS feed” directly into decisions makes audits, rollbacks, and model upgrades painful.
* A **layered model** lets you **version probability evolution**, compare vendors, and train **meta‑predictors on deltas** (how risk changes over time), not just on snapshots.
---
# The three layers (and how they map to Stella Ops)
1. **Raw feed layer (immutable)**
* **Goal:** Store exactly what the provider sent (EPSS v4 CSV/JSON, schema drift and all).
* **Stella modules:** `Concelier` (preserve‑prune source) writes; `Authority` handles signatures/hashes.
* **Storage:** `postgres.epss_raw` (partitioned by day); blob column for the untouched payload; SHA‑256 of source file.
* **Why:** Full provenance + deterministic replay.
2. **Normalized probabilistic layer**
* **Goal:** Clean, typed tables keyed by `cve_id`, with **probability, percentile, model_version, asof_ts**.
* **Stella modules:** `Excititor` (transform); `Policy Engine` reads.
* **Storage:** `postgres.epss_prob` with a **surrogate key** `(cve_id, model_version, asof_ts)` and computed **delta fields** vs previous `asof_ts`.
* **Extras:** Keep optional vendor columns (e.g., FIRST, custom regressors) to compare models side‑by‑side.
3. **Signal‑ready layer (risk engine contracts)**
* **Goal:** Pre‑chewed “events” your **Signals/Router** can route instantly.
* **What’s inside:** Only the fields needed for gating and UI: `cve_id`, `prob_now`, `prob_delta`, `percentile`, `risk_band`, `explain_hash`.
* **Emit:** `first_signal`, `risk_increase`, `risk_decrease`, `quieted` with **idempotent event keys**.
* **Stella modules:** `Signals` publishes, `Router` fan‑outs, `Timeline` records; `Notify` handles subscriptions.
---
# Minimal Postgres schema (ready to paste)
```sql
-- 1) Raw (immutable)
create table epss_raw (
id bigserial primary key,
source_uri text not null,
ingestion_ts timestamptz not null default now(),
asof_date date not null,
payload jsonb not null,
payload_sha256 bytea not null
);
create index on epss_raw (asof_date);
-- 2) Normalized
create table epss_prob (
id bigserial primary key,
cve_id text not null,
model_version text not null, -- e.g., 'EPSS-4.0-Falcon-2025-12'
asof_ts timestamptz not null,
probability double precision not null,
percentile double precision,
features jsonb, -- optional: normalized features used
unique (cve_id, model_version, asof_ts)
);
-- delta against prior point (materialized view or nightly job)
create materialized view epss_prob_delta as
select p.*,
p.probability - lag(p.probability) over (partition by cve_id, model_version order by asof_ts) as prob_delta
from epss_prob p;
-- 3) Signal-ready
create table epss_signal (
signal_id bigserial primary key,
cve_id text not null,
asof_ts timestamptz not null,
probability double precision not null,
prob_delta double precision,
risk_band text not null, -- e.g., 'LOW/MED/HIGH/CRITICAL'
model_version text not null,
explain_hash bytea not null, -- hash of inputs -> deterministic
unique (cve_id, model_version, asof_ts)
);
```
---
# C# ingestion skeleton (StellaOps.Scanner.Worker.DotNet style)
```csharp
// 1) Fetch & store raw (Concelier)
public async Task IngestRawAsync(Uri src, DateOnly asOfDate) {
var bytes = await http.GetByteArrayAsync(src);
var sha = SHA256.HashData(bytes);
await pg.ExecuteAsync(
"insert into epss_raw(source_uri, asof_date, payload, payload_sha256) values (@u,@d,@p::jsonb,@s)",
new { u = src.ToString(), d = asOfDate, p = Encoding.UTF8.GetString(bytes), s = sha });
}
// 2) Normalize (Excititor)
public async Task NormalizeAsync(DateOnly asOfDate, string modelVersion) {
var raws = await pg.QueryAsync<(string Payload)>("select payload from epss_raw where asof_date=@d", new { d = asOfDate });
foreach (var r in raws) {
foreach (var row in ParseCsvOrJson(r.Payload)) {
await pg.ExecuteAsync(
@"insert into epss_prob(cve_id, model_version, asof_ts, probability, percentile, features)
values (@cve,@mv,@ts,@prob,@pct,@feat)
on conflict do nothing",
new { cve = row.Cve, mv = modelVersion, ts = row.AsOf, prob = row.Prob, pct = row.Pctl, feat = row.Features });
}
}
}
// 3) Emit signal-ready (Signals)
public async Task EmitSignalsAsync(string modelVersion, double deltaThreshold) {
var rows = await pg.QueryAsync(@"select cve_id, asof_ts, probability,
probability - lag(probability) over (partition by cve_id, model_version order by asof_ts) as prob_delta
from epss_prob where model_version=@mv", new { mv = modelVersion });
foreach (var r in rows) {
var band = Band(r.probability); // map to LOW/MED/HIGH/CRITICAL
if (Math.Abs(r.prob_delta ?? 0) >= deltaThreshold) {
var explainHash = DeterministicExplainHash(r);
await pg.ExecuteAsync(@"insert into epss_signal
(cve_id, asof_ts, probability, prob_delta, risk_band, model_version, explain_hash)
values (@c,@t,@p,@d,@b,@mv,@h)
on conflict do nothing",
new { c = r.cve_id, t = r.asof_ts, p = r.probability, d = r.prob_delta, b = band, mv = modelVersion, h = explainHash });
await bus.PublishAsync("risk.epss.delta", new {
cve = r.cve_id, ts = r.asof_ts, prob = r.probability, delta = r.prob_delta, band, model = modelVersion, explain = Convert.ToHexString(explainHash)
});
}
}
}
```
---
# Versioning & experiments (the secret sauce)
* **Model namespace:** `EPSS‑4.0‑<regressor‑name>‑<date>` so you can run multiple variants in parallel.
* **Delta‑training:** Train a small meta‑predictor on **Δprobability** to forecast **“risk jumps in next N days.”**
* **A/B in production:** Route `model_version=x` to 50% of projects; compare **MTTA to patch** and **false‑alarm rate**.
---
# Policy & UI wiring (quick contracts)
**Policy gates** (OPA/Rego or internal rules):
* Block if `risk_band ∈ {HIGH, CRITICAL}` **AND** `prob_delta >= 0.1` in last 72h.
* Soften if asset not reachable or mitigated by VEX.
**UI (Evidence pane):**
* Show **sparkline of EPSS over time**, highlight last Δ.
* “Why now?” button reveals **explain_hash** → deterministic evidence payload.
---
# Ops & reliability
* Daily ingestion with **idempotent** runs (raw SHA guard).
* Backfills: re‑normalize from `epss_raw` for any new model without re‑downloading.
* **Deterministic replay:** export `(raw, transform code hash, model_version)` alongside results.
---
If you want, I can drop this as a ready‑to‑run **.sql + .csproj** seed with a tiny CLI (`ingest`, `normalize`, `emit`) tailored to your `Postgres + Valkey` profile.
Below is a “do this, then this” implementation guide for a **layered EPSS pipeline** inside **Stella Ops**, with concrete schemas, job boundaries, idempotency rules, and the tricky edge cases (model-version shifts, noise control, backfills).
I’ll assume:
* **Postgres** is your system of record, **Valkey** is available for caching,
* you run **.NET workers** (like `StellaOps.Scanner.Worker.DotNet`),
* Stella modules you referenced map roughly like this:
* **Concelier** = ingest + preserve/prune raw sources
* **Authority** = provenance (hashes, immutability, signature-like guarantees)
* **Excititor** = transform/normalize
* **Signals / Router / Timeline / Notify** = event pipeline + audit trail + subscriptions
I’ll anchor the EPSS feed details to FIRST’s docs:
* The data feed fields are `cve`, `epss`, `percentile` and are refreshed daily. ([FIRST][1])
* Historical daily `.csv.gz` files exist at `https://epss.empiricalsecurity.com/epss_scores-YYYY-mm-dd.csv.gz`. ([FIRST][1])
* The API base is `https://api.first.org/data/v1/epss` and supports per-CVE and time-series queries. ([FIRST][2])
* FIRST notes model-version shifts (v2/v3/v4) and that the daily files include a leading `#` comment indicating model version/publish date (important for delta correctness). ([FIRST][1])
* FIRST’s guidance: use **probability** as the primary score and **show percentile alongside it**; raw feeds provide both as decimals 0–1. ([FIRST][3])
---
## 0) Target architecture and data contracts
### The 3 layers and what must be true in each
1. **Raw layer (immutable)**
* You can replay exactly what you ingested, byte-for-byte.
* Contains: file bytes or object-store pointer, headers (ETag, Last-Modified), SHA-256, parsed “header comment” (the `# …` line), ingestion status.
2. **Normalized probability layer (typed, queryable, historical)**
* One row per `(model_name, asof_date, cve_id)`.
* Contains: `epss` probability (0–1), `percentile` (0–1), `model_version` (from file header comment if available).
* Built for joins against vulnerability inventory and for time series.
3. **Signal-ready layer (risk engine contract)**
* Contains only actionable changes (crossing thresholds, jumps, newly-scored, etc.), ideally scoped to **observed CVEs** in your environment to avoid noise.
* Events are idempotent, audit-friendly, and versioned.
---
## 1) Data source choice and acquisition strategy
### Prefer the daily bulk `.csv.gz` over paging the API for full refresh
* FIRST explicitly documents the “ALL CVEs for a date” bulk file URL pattern. ([FIRST][2])
* The API is great for:
* “give me EPSS for this CVE list”
* “give me last 30 days time series for CVE X” ([FIRST][2])
**Recommendation**
* Daily job pulls the bulk file for “latest available date”.
* A separate on-demand endpoint uses the API time-series for UI convenience (optional).
### Robust “latest available date” probing
Because the “current day” file may not be published when your cron fires:
Algorithm:
1. Let `d0 = UtcToday`.
2. For `d in [d0, d0-1, d0-2, d0-3]`:
* Try `GET https://epss.empiricalsecurity.com/epss_scores-{d:yyyy-MM-dd}.csv.gz`
* If HTTP 200: ingest that as `asof_date = d` and stop.
3. If none succeed: fail the job with a clear message + alert.
This avoids timezone and publishing-time ambiguity.
---
## 2) Layer 1: Raw feed (Concelier + Authority)
### 2.1 Schema for raw + lineage
Use a dedicated schema `epss` so the pipeline is easy to reason about.
```sql
create schema if not exists epss;
-- Immutable file-level record
create table if not exists epss.raw_file (
raw_id bigserial primary key,
source_uri text not null,
asof_date date not null,
fetched_at timestamptz not null default now(),
http_etag text,
http_last_modified timestamptz,
content_len bigint,
content_sha256 bytea not null,
-- first non-empty comment lines like "# model=... date=..."
header_comment text,
model_version text,
model_published_on date,
-- storage: either inline bytea OR object storage pointer
storage_kind text not null default 'pg_bytea', -- 'pg_bytea' | 's3' | 'fs'
storage_ref text,
content_gz bytea, -- nullable if stored externally
parse_status text not null default 'pending', -- pending|parsed|failed
parse_error text,
unique (source_uri, asof_date, content_sha256)
);
create index if not exists ix_epss_raw_file_asof on epss.raw_file(asof_date);
create index if not exists ix_epss_raw_file_status on epss.raw_file(parse_status);
```
**Why store `model_version` here?**
FIRST warns that model updates cause “major shifts” and the daily files include a `#` comment with model version/publish date. If you ignore this, your delta logic will misfire on model-change days. ([FIRST][1])
### 2.2 Raw ingestion idempotency rules
A run is “already ingested” if:
* a row exists for `(source_uri, asof_date)` with the same `content_sha256`, OR
* you implement “single truth per day” and treat any new sha for the same date as “replace” (rare, but can happen).
Recommended:
* **Treat as replace only if** you’re confident the source can republish the same date. If not, keep both but mark the superseded one.
### 2.3 Raw ingestion implementation details (.NET)
**Key constraints**
* Download as a stream (`ResponseHeadersRead`)
* Compute SHA-256 while streaming
* Store bytes or stream them into object storage
* Capture ETag/Last-Modified headers if present
Pseudo-implementation structure:
* `EpssFetchJob`
* `ProbeLatestDateAsync()`
* `DownloadAsync(uri)`
* `ExtractHeaderCommentAsync(gzipStream)` (read a few first lines after decompression)
* `InsertRawFileRecord(...)` (Concelier + Authority)
**Header comment extraction**
FIRST indicates files may start with `# ... model version ... publish date ...`. ([FIRST][1])
So do:
* Decompress
* Read lines until you find first non-empty non-`#` line (that’s likely CSV header / first row)
* Save the concatenated `#` lines as `header_comment`
* Regex best-effort parse:
* `model_version`: something like `v2025.03.14`
* `model_published_on`: `YYYY-MM-DD`
If parsing fails, still store `header_comment`.
### 2.4 Pruning raw (Concelier “preserve-prune”)
Define retention policy:
* Keep **raw bytes** 90–180 days (cheap enough; each `.csv.gz` is usually a few–tens of MB)
* Keep **metadata** forever (tiny, essential for audits)
Nightly cleanup job:
* delete `content_gz` or external object for `raw_file` older than retention
* keep row but set `storage_kind='pruned'`, `content_gz=null`, `storage_ref=null`
---
## 3) Layer 2: Normalized probability tables (Excititor)
### 3.1 Core normalized table design
Requirements:
* Efficient time series per CVE
* Efficient “latest score per CVE”
* Efficient join to “observed vulnerabilities” tables
#### Daily score table (partitioned)
```sql
create table if not exists epss.daily_score (
model_name text not null, -- 'FIRST_EPSS'
asof_date date not null,
cve_id text not null,
epss double precision not null,
percentile double precision,
model_version text, -- from raw header if available
raw_id bigint references epss.raw_file(raw_id),
loaded_at timestamptz not null default now(),
-- Guards
constraint ck_epss_range check (epss >= 0.0 and epss <= 1.0),
constraint ck_percentile_range check (percentile is null or (percentile >= 0.0 and percentile <= 1.0)),
primary key (model_name, asof_date, cve_id)
) partition by range (asof_date);
-- Example monthly partitions (create via migration script generator)
create table if not exists epss.daily_score_2025_12
partition of epss.daily_score for values from ('2025-12-01') to ('2026-01-01');
create index if not exists ix_epss_daily_score_cve on epss.daily_score (model_name, cve_id, asof_date desc);
create index if not exists ix_epss_daily_score_epss on epss.daily_score (model_name, asof_date, epss desc);
create index if not exists ix_epss_daily_score_pct on epss.daily_score (model_name, asof_date, percentile desc);
```
**Field semantics**
* `epss` is the probability of exploitation in the next 30 days, 0–1. ([FIRST][1])
* `percentile` is relative rank among all scored vulnerabilities. ([FIRST][1])
### 3.2 Maintain a “latest” table for fast joins
Don’t compute latest via window functions in hot paths (policy evaluation / scoring). Materialize it.
```sql
create table if not exists epss.latest_score (
model_name text not null,
cve_id text not null,
asof_date date not null,
epss double precision not null,
percentile double precision,
model_version text,
updated_at timestamptz not null default now(),
primary key (model_name, cve_id)
);
create index if not exists ix_epss_latest_epss on epss.latest_score(model_name, epss desc);
create index if not exists ix_epss_latest_pct on epss.latest_score(model_name, percentile desc);
```
Update logic (after loading a day):
* Upsert each CVE (or do a set-based upsert):
* `asof_date` should only move forward
* if a backfill loads an older day, do not overwrite latest
### 3.3 Delta table for change detection
Store deltas per day (this powers signals and “sparkline deltas”).
```sql
create table if not exists epss.daily_delta (
model_name text not null,
asof_date date not null,
cve_id text not null,
epss double precision not null,
prev_asof_date date,
prev_epss double precision,
epss_delta double precision,
percentile double precision,
prev_percentile double precision,
percentile_delta double precision,
model_version text,
prev_model_version text,
is_model_change boolean not null default false,
created_at timestamptz not null default now(),
primary key (model_name, asof_date, cve_id)
);
create index if not exists ix_epss_daily_delta_cve on epss.daily_delta(model_name, cve_id, asof_date desc);
create index if not exists ix_epss_daily_delta_delta on epss.daily_delta(model_name, asof_date, epss_delta desc);
```
**Model update handling**
* On a model version change day (v3→v4 etc), many deltas will jump.
* FIRST explicitly warns model shifts. ([FIRST][1])
So:
* detect if today’s `model_version != previous_day.model_version`
* set `is_model_change = true`
* optionally **suppress delta-based signals** that day (or emit a separate “MODEL_UPDATED” event)
### 3.4 Normalization job mechanics
Implement `EpssNormalizeJob`:
1. Select `raw_file` rows where `parse_status='pending'`.
2. Decompress `content_gz` or fetch from object store.
3. Parse CSV:
* skip `#` comment lines
* expect columns: `cve,epss,percentile` (FIRST documents these fields). ([FIRST][1])
4. Validate:
* CVE format: `^CVE-\d{4}-\d{4,}$`
* numeric parse for epss/percentile
* range checks 0–1
5. Load into Postgres fast:
* Use `COPY` (binary import) into a **staging table** `epss.stage_score`
* Then set-based insert into `epss.daily_score`
6. Update `epss.raw_file.parse_status='parsed'` or `failed`.
#### Staging table pattern
```sql
create unlogged table if not exists epss.stage_score (
model_name text not null,
asof_date date not null,
cve_id text not null,
epss double precision not null,
percentile double precision,
model_version text,
raw_id bigint not null
);
```
In the job:
* `truncate epss.stage_score;`
* `COPY epss.stage_score FROM STDIN (FORMAT BINARY)`
* Then (transactionally):
* `delete from epss.daily_score where model_name=@m and asof_date=@d;` *(idempotency for reruns)*
* `insert into epss.daily_score (...) select ... from epss.stage_score;`
This avoids `ON CONFLICT` overhead and guarantees deterministic reruns.
### 3.5 Delta + latest materialization job
Implement `EpssMaterializeJob` after successful daily_score insert.
**Compute previous available date**
```sql
-- previous date available for that model_name
select max(asof_date)
from epss.daily_score
where model_name = @model
and asof_date < @asof_date;
```
**Populate delta (set-based)**
```sql
insert into epss.daily_delta (
model_name, asof_date, cve_id,
epss, prev_asof_date, prev_epss, epss_delta,
percentile, prev_percentile, percentile_delta,
model_version, prev_model_version, is_model_change
)
select
cur.model_name,
cur.asof_date,
cur.cve_id,
cur.epss,
prev.asof_date as prev_asof_date,
prev.epss as prev_epss,
cur.epss - prev.epss as epss_delta,
cur.percentile,
prev.percentile as prev_percentile,
(cur.percentile - prev.percentile) as percentile_delta,
cur.model_version,
prev.model_version,
(cur.model_version is not null and prev.model_version is not null and cur.model_version <> prev.model_version) as is_model_change
from epss.daily_score cur
left join epss.daily_score prev
on prev.model_name = cur.model_name
and prev.asof_date = @prev_asof_date
and prev.cve_id = cur.cve_id
where cur.model_name = @model
and cur.asof_date = @asof_date;
```
**Update latest_score (set-based upsert)**
```sql
insert into epss.latest_score(model_name, cve_id, asof_date, epss, percentile, model_version)
select model_name, cve_id, asof_date, epss, percentile, model_version
from epss.daily_score
where model_name=@model and asof_date=@asof_date
on conflict (model_name, cve_id) do update
set asof_date = excluded.asof_date,
epss = excluded.epss,
percentile = excluded.percentile,
model_version = excluded.model_version,
updated_at = now()
where epss.latest_score.asof_date < excluded.asof_date;
```
---
## 4) Layer 3: Signal-ready output (Signals + Router + Timeline + Notify)
### 4.1 Decide what “signal” means in Stella Ops
You do **not** want to emit 300k events daily.
You want “actionable” events, ideally:
* only for CVEs that are **observed** in your tenant’s environment, and
* only when something meaningful happens.
Examples:
* Risk band changes (based on percentile or probability)
* ΔEPS S crosses a threshold (e.g., jump ≥ 0.05)
* Newly scored CVEs that are present in environment
* Model version change day → one summary event instead of 300k deltas
### 4.2 Risk band mapping (internal heuristic)
FIRST explicitly does **not** “officially bin” EPSS scores; binning is subjective. ([FIRST][3])
But operationally you’ll want bands. Use config-driven thresholds.
Default band function based on percentile:
* `CRITICAL` if `percentile >= 0.995`
* `HIGH` if `percentile >= 0.99`
* `MEDIUM` if `percentile >= 0.90`
* else `LOW`
Store these in config per tenant/policy pack.
### 4.3 Signal table for idempotency + audit
```sql
create table if not exists epss.signal (
signal_id bigserial primary key,
tenant_id uuid not null,
model_name text not null,
asof_date date not null,
cve_id text not null,
event_type text not null, -- 'RISK_BAND_UP' | 'RISK_SPIKE' | 'MODEL_UPDATED' | ...
risk_band text,
epss double precision,
epss_delta double precision,
percentile double precision,
percentile_delta double precision,
is_model_change boolean not null default false,
-- deterministic idempotency key
dedupe_key text not null,
payload jsonb not null,
created_at timestamptz not null default now(),
unique (tenant_id, dedupe_key)
);
create index if not exists ix_epss_signal_tenant_date on epss.signal(tenant_id, asof_date desc);
create index if not exists ix_epss_signal_cve on epss.signal(tenant_id, cve_id, asof_date desc);
```
**Dedupe key pattern**
Make it deterministic:
```
dedupe_key = $"{model_name}:{asof_date:yyyy-MM-dd}:{cve_id}:{event_type}:{band_before}->{band_after}"
```
### 4.4 Signal generation job
Implement `EpssSignalJob(tenant)`:
1. Get tenant’s **observed CVEs** from your vuln inventory (whatever your table is; call it `vuln.instance`):
* only open/unremediated vulns
* optionally only “reachable” or “internet exposed” assets
2. Join against today’s `epss.daily_delta` (or `epss.daily_score` if you skipped delta):
Pseudo-SQL:
```sql
select d.*
from epss.daily_delta d
join vuln.observed_cve oc
on oc.tenant_id = @tenant
and oc.cve_id = d.cve_id
where d.model_name=@model
and d.asof_date=@asof_date;
```
3. Suppress noise:
* if `is_model_change=true`, skip “delta spike” events and instead emit one `MODEL_UPDATED` summary event per tenant (and maybe per policy domain).
* else evaluate:
* `abs(epss_delta) >= delta_threshold`
* band change
* percentile crosses a cutoff
4. Insert into `epss.signal` with dedupe key, then publish to Signals bus:
* topic: `signals.epss`
* payload includes `tenant_id`, `cve_id`, `asof_date`, `epss`, `percentile`, deltas, band, and an `evidence` block.
5. Timeline + Notify:
* Timeline: record the event (what changed, when, data source sha)
* Notify: notify subscribed channels (Slack/email/etc) based on tenant policy
### 4.5 Evidence payload structure
Keep evidence deterministic + replayable:
```json
{
"source": {
"provider": "FIRST",
"feed": "epss_scores-YYYY-MM-DD.csv.gz",
"asof_date": "2025-12-17",
"raw_sha256": "…",
"model_version": "v2025.03.14",
"header_comment": "# ... "
},
"metrics": {
"epss": 0.153,
"percentile": 0.92,
"epss_delta": 0.051,
"percentile_delta": 0.03
},
"decision": {
"event_type": "RISK_SPIKE",
"thresholds": {
"delta_threshold": 0.05,
"critical_percentile": 0.995
}
}
}
```
This aligns with FIRST’s recommendation to present probability with percentile when possible. ([FIRST][3])
---
## 5) Integration points inside Stella Ops
### 5.1 Policy Engine usage
Policy Engine should **only** read from Layer 2 (normalized) and Layer 3 (signals), never raw.
Patterns:
* For gating decisions: query `epss.latest_score` for each CVE in a build/image/SBOM scan result.
* For “why was this blocked?”: show evidence that references `raw_sha256` and `model_version`.
### 5.2 Vuln scoring pipeline
When you compute “Stella Risk Score” for a vuln instance:
* Join `vuln_instance.cve_id` → `epss.latest_score`
* Combine with CVSS, KEV, exploit maturity, asset exposure, etc.
* EPSS alone is **threat likelihood**, not impact; FIRST explicitly says it’s not a complete picture of risk. ([FIRST][4])
### 5.3 UI display
Recommended UI string (per FIRST guidance):
* Show **probability** as a percent + show percentile:
* `15.3% (92nd percentile)` ([FIRST][3])
For sparklines:
* Use `epss.daily_score` time series for last N days
* Annotate model-version change days (vertical marker)
---
## 6) Operational hardening
### 6.1 Scheduling
* Run daily at a fixed time in UTC.
* Probe up to 3 back days for latest file.
### 6.2 Exactly-once semantics
Use three safeguards:
1. `epss.raw_file` uniqueness on `(source_uri, asof_date, sha256)`
2. Transactional load:
* delete existing `daily_score` for that `(model_name, asof_date)`
* insert freshly parsed rows
3. Advisory lock per `(model_name, asof_date)` to prevent concurrent loads:
* `pg_advisory_xact_lock(hashtext(model_name), asof_date::int)`
### 6.3 Monitoring (must-have metrics)
Emit metrics per job stage:
* download success/failure
* bytes downloaded
* sha256 computed
* rows parsed
* parse error count
* rows inserted into `daily_score`
* delta rows created
* signal events emitted
* “model version changed” boolean
Alert conditions:
* no new asof_date ingested for > 48 hours
* parse failure
* row count drops by > X% from previous day (data anomaly)
### 6.4 Backfills
Implement `epss backfill --from 2021-04-14 --to 2025-12-17`:
* Fetch raw files for each day
* Normalize daily_score
* Materialize latest and delta
* **Disable signals** during bulk backfill (or route to “silent” topic) to avoid spamming.
FIRST notes historical data begins 2021-04-14. ([FIRST][1])
---
## 7) Reference .NET job skeletons
### Job boundaries
* `EpssFetchJob` → writes `epss.raw_file`
* `EpssNormalizeJob` → fills `epss.daily_score`
* `EpssMaterializeJob` → updates `epss.daily_delta` and `epss.latest_score`
* `EpssSignalJob` → per-tenant emission into `epss.signal` + bus publish
### Performance notes
* Use `GZipStream` + `StreamReader` line-by-line (no full file into memory)
* Use `NpgsqlBinaryImporter` for `COPY` into staging
* Use set-based SQL for delta/latest
---
## 8) The “gotchas” that make or break EPSS pipelines
1. **Model version changes create false deltas**
Store `model_version` and mark `is_model_change`. FIRST explicitly warns about score shifts on model updates and notes v4 began publishing on 2025‑03‑17. ([FIRST][1])
2. **Percentile is relative; probability is primary**
Probability should remain your canonical numeric score; percentile provides context for humans. ([FIRST][3])
3. **Don’t emit global events**
Restrict signals to observed CVEs per tenant/environment.
4. **Keep raw provenance**
Your audit story depends on storing:
* exact source URI, as-of date, sha256, header comment
---
## 9) Minimal “definition of done” checklist
### Data correctness
* [ ] For a known CVE, `epss.latest_score` matches the daily file for the latest asof_date
* [ ] `epss` and `percentile` ranges enforced (0–1)
* [ ] Model version extracted when present; otherwise stored as null but header_comment preserved
* [ ] Delta rows created and `is_model_change` flips on version changes
### Operational
* [ ] Daily job retries on transient HTTP failures
* [ ] Alert if no new asof_date in 48h
* [ ] Raw bytes retention + metadata retention
### Product
* [ ] UI displays `probability% (percentile)` per FIRST recommendation ([FIRST][3])
* [ ] Signal events link to evidence (raw sha, model version, asof date)
* [ ] Policy Engine consumes `latest_score` only (never raw)
---
If you want, I can also provide:
* a **migration script generator** that auto-creates monthly partitions for `epss.daily_score`,
* an example **Valkey caching strategy** (`epss:latest:{cve}` with a 48h TTL, warmed only for observed CVEs),
* and a concrete **“observed CVE” join contract** (what columns to expose from your vuln inventory so EPSS signals stay noise-free).
[1]: https://www.first.org/epss/data_stats "Exploit Prediction Scoring System (EPSS)"
[2]: https://www.first.org/epss/api "Exploit Prediction Scoring System (EPSS)"
[3]: https://www.first.org/epss/articles/prob_percentile_bins "Exploit Prediction Scoring System (EPSS)"
[4]: https://www.first.org/epss/faq "EPSS Frequently Asked Questions"

21
docs/reachability/gates.md
View File

@@ -88,6 +88,8 @@ Detects configuration-based gates:
### DetectedGate
**Note:** In **Signals API outputs**, `type` is serialized as the C# enum name (e.g., `"AuthRequired"`). In **richgraph-v1** JSON, `type` is lowerCamelCase and gate fields are snake_case (see example below).
```typescript
interface DetectedGate {
type: 'AuthRequired' | 'FeatureFlag' | 'AdminOnly' | 'NonDefaultConfig';
@@ -130,6 +132,25 @@ public sealed record RichGraphEdge
}
```
**richgraph-v1 JSON example (edge fragment):**
```json
{
"gate_multiplier_bps": 3000,
"gates": [
{
"type": "authRequired",
"detail": "[Authorize] attribute on controller",
"guard_symbol": "MyController.VulnerableAction",
"source_file": "src/MyController.cs",
"line_number": 42,
"detection_method": "csharp.attribute",
"confidence": 0.95
}
]
}
```
### ReachabilityReport
Gates are included in the reachability report:

2
src/Orchestrator/StellaOps.Orchestrator/StellaOps.Orchestrator.Core/Domain/Export/ExportSchedule.cs
View File

@@ -478,7 +478,7 @@ public sealed record ExportAlert(
TenantId: tenantId,
ExportType: exportType,
Severity: severity,
Message: $"Export job {exportType} failure rate is {failureRate:F1}%",
Message: FormattableString.Invariant($"Export job {exportType} failure rate is {failureRate:F1}%"),
FailedJobIds: recentFailedJobIds,
ConsecutiveFailures: 0,
FailureRate: failureRate,

4
src/Orchestrator/StellaOps.Orchestrator/StellaOps.Orchestrator.Core/Domain/Slo.cs
View File

@@ -523,8 +523,8 @@ public sealed record SloAlert(
AlertBudgetThreshold threshold)
{
var message = threshold.BurnRateThreshold.HasValue && state.BurnRate >= threshold.BurnRateThreshold.Value
? $"SLO '{slo.Name}' burn rate {state.BurnRate:F2}x exceeds threshold {threshold.BurnRateThreshold.Value:F2}x"
: $"SLO '{slo.Name}' error budget {state.BudgetConsumed:P1} consumed exceeds threshold {threshold.BudgetConsumedThreshold:P1}";
? FormattableString.Invariant($"SLO '{slo.Name}' burn rate {state.BurnRate:F2}x exceeds threshold {threshold.BurnRateThreshold.Value:F2}x")
: FormattableString.Invariant($"SLO '{slo.Name}' error budget {state.BudgetConsumed:P1} consumed exceeds threshold {threshold.BudgetConsumedThreshold:P1}");
return new SloAlert(
AlertId: Guid.NewGuid(),

10
src/Orchestrator/StellaOps.Orchestrator/StellaOps.Orchestrator.Infrastructure/Options/FirstSignalOptions.cs
View File

@@ -7,6 +7,7 @@ public sealed class FirstSignalOptions
public FirstSignalCacheOptions Cache { get; set; } = new();
public FirstSignalColdPathOptions ColdPath { get; set; } = new();
public FirstSignalSnapshotWriterOptions SnapshotWriter { get; set; } = new();
public FirstSignalFailureSignatureOptions FailureSignatures { get; set; } = new();
}
public sealed class FirstSignalCacheOptions
@@ -30,3 +31,12 @@ public sealed class FirstSignalSnapshotWriterOptions
public int MaxRunsPerTick { get; set; } = 50;
public int LookbackMinutes { get; set; } = 60;
}
public sealed class FirstSignalFailureSignatureOptions
{
public bool Enabled { get; set; }
public string? SchedulerBaseUrl { get; set; }
public int TimeoutMs { get; set; } = 1000;
public int MediumOccurrenceThreshold { get; set; } = 3;
public int HighOccurrenceThreshold { get; set; } = 10;
}

1
src/Orchestrator/StellaOps.Orchestrator/StellaOps.Orchestrator.Infrastructure/ServiceCollectionExtensions.cs
View File

@@ -73,6 +73,7 @@ public static class ServiceCollectionExtensions
// First signal (TTFS) services
services.Configure<FirstSignalOptions>(configuration.GetSection(FirstSignalOptions.SectionName));
services.AddHttpClient<IFailureSignatureLookupClient, SchedulerFailureSignatureLookupClient>();
services.AddSingleton<IFirstSignalCache, FirstSignalCache>();
services.AddScoped<StellaOps.Orchestrator.Core.Services.IFirstSignalService, FirstSignalService>();

194
src/Orchestrator/StellaOps.Orchestrator/StellaOps.Orchestrator.Infrastructure/Services/FirstSignalService.cs
View File

@@ -28,6 +28,7 @@ public sealed class FirstSignalService : CoreServices.IFirstSignalService
private readonly IFirstSignalSnapshotRepository _snapshotRepository;
private readonly IRunRepository _runRepository;
private readonly IJobRepository _jobRepository;
private readonly IFailureSignatureLookupClient _failureSignatureLookupClient;
private readonly TimeProvider _timeProvider;
private readonly TimeToFirstSignalMetrics _ttfsMetrics;
private readonly FirstSignalOptions _options;
@@ -38,6 +39,7 @@ public sealed class FirstSignalService : CoreServices.IFirstSignalService
IFirstSignalSnapshotRepository snapshotRepository,
IRunRepository runRepository,
IJobRepository jobRepository,
IFailureSignatureLookupClient failureSignatureLookupClient,
TimeProvider timeProvider,
TimeToFirstSignalMetrics ttfsMetrics,
IOptions<FirstSignalOptions> options,
@@ -47,6 +49,7 @@ public sealed class FirstSignalService : CoreServices.IFirstSignalService
_snapshotRepository = snapshotRepository ?? throw new ArgumentNullException(nameof(snapshotRepository));
_runRepository = runRepository ?? throw new ArgumentNullException(nameof(runRepository));
_jobRepository = jobRepository ?? throw new ArgumentNullException(nameof(jobRepository));
_failureSignatureLookupClient = failureSignatureLookupClient ?? throw new ArgumentNullException(nameof(failureSignatureLookupClient));
_timeProvider = timeProvider ?? TimeProvider.System;
_ttfsMetrics = ttfsMetrics ?? throw new ArgumentNullException(nameof(ttfsMetrics));
_options = (options ?? throw new ArgumentNullException(nameof(options))).Value;
@@ -241,13 +244,44 @@ public sealed class FirstSignalService : CoreServices.IFirstSignalService
};
}
var signalComputed = ComputeSignal(run, jobs, cacheHit: false, origin: "cold_start");
var signalOrigin = "cold_start";
var signalComputed = ComputeSignal(run, jobs, cacheHit: false, signalOrigin);
if (signalComputed.Kind == FirstSignalKind.Failed && _options.FailureSignatures.Enabled)
{
var lookup = TryBuildFailureSignatureLookup(run, jobs);
if (lookup is not null)
{
var lastKnownOutcome = await _failureSignatureLookupClient
.TryGetLastKnownOutcomeAsync(
tenantId,
lookup.Value.ScopeType,
lookup.Value.ScopeId,
lookup.Value.ToolchainHash,
coldPathCts.Token)
.ConfigureAwait(false);
if (lastKnownOutcome is not null)
{
signalOrigin = "failure_index";
signalComputed = signalComputed with
{
LastKnownOutcome = lastKnownOutcome,
Diagnostics = signalComputed.Diagnostics with
{
Source = signalOrigin
}
};
}
}
}
var computedEtag = GenerateEtag(signalComputed);
_ttfsMetrics.RecordColdPathComputation(
coldStopwatch.Elapsed.TotalSeconds,
surface: "api",
signalSource: "cold_start",
signalSource: signalOrigin,
kind: MapKind(signalComputed.Kind),
phase: MapPhase(signalComputed.Phase),
tenantId: tenantId);
@@ -261,30 +295,30 @@ public sealed class FirstSignalService : CoreServices.IFirstSignalService
{
Signal = signalComputed,
ETag = computedEtag,
Origin = "cold_start",
Origin = signalOrigin,
},
cancellationToken)
.ConfigureAwait(false);
if (IsNotModified(ifNoneMatch, computedEtag))
{
RecordSignalRendered(overallStopwatch, cacheHit: false, origin: "cold_start", signalComputed.Kind, signalComputed.Phase, tenantId);
RecordSignalRendered(overallStopwatch, cacheHit: false, origin: signalOrigin, signalComputed.Kind, signalComputed.Phase, tenantId);
return new CoreServices.FirstSignalResult
{
Status = CoreServices.FirstSignalResultStatus.NotModified,
CacheHit = false,
Source = "cold_start",
Source = signalOrigin,
ETag = computedEtag,
Signal = signalComputed,
};
}
RecordSignalRendered(overallStopwatch, cacheHit: false, origin: "cold_start", signalComputed.Kind, signalComputed.Phase, tenantId);
RecordSignalRendered(overallStopwatch, cacheHit: false, origin: signalOrigin, signalComputed.Kind, signalComputed.Phase, tenantId);
return new CoreServices.FirstSignalResult
{
Status = CoreServices.FirstSignalResultStatus.Found,
CacheHit = false,
Source = "cold_start",
Source = signalOrigin,
ETag = computedEtag,
Signal = signalComputed,
};
@@ -409,6 +443,152 @@ public sealed class FirstSignalService : CoreServices.IFirstSignalService
};
}
private readonly record struct FailureSignatureLookup(string ScopeType, string ScopeId, string ToolchainHash);
private static FailureSignatureLookup? TryBuildFailureSignatureLookup(Run run, IReadOnlyList<Job> jobs)
{
if (jobs.Count == 0)
{
return null;
}
var job = SelectRepresentativeJob(run, jobs);
if (string.IsNullOrWhiteSpace(job.Payload))
{
return null;
}
try
{
using var document = JsonDocument.Parse(job.Payload);
if (document.RootElement.ValueKind != JsonValueKind.Object)
{
return null;
}
var payload = document.RootElement;
if (TryGetPayloadString(payload, "repository", out var repository) ||
TryGetPayloadString(payload, "repo", out repository))
{
var toolchainHash = ComputeToolchainHash(job, payload);
return new FailureSignatureLookup("repo", repository!, toolchainHash);
}
if (TryGetDigestScope(payload, out var scopeType, out var scopeId))
{
var toolchainHash = ComputeToolchainHash(job, payload);
return new FailureSignatureLookup(scopeType!, scopeId!, toolchainHash);
}
return null;
}
catch
{
return null;
}
}
private static bool TryGetPayloadString(JsonElement payload, string key, out string? value)
{
foreach (var property in payload.EnumerateObject())
{
if (!string.Equals(property.Name, key, StringComparison.OrdinalIgnoreCase))
{
continue;
}
if (property.Value.ValueKind != JsonValueKind.String)
{
continue;
}
var raw = property.Value.GetString();
if (string.IsNullOrWhiteSpace(raw))
{
continue;
}
value = raw.Trim();
return true;
}
value = null;
return false;
}
private static bool TryGetDigestScope(JsonElement payload, out string? scopeType, out string? scopeId)
{
var candidates = new (string Key, string Type)[]
{
("artifactDigest", "artifact"),
("imageDigest", "image"),
("digest", "image"),
("artifact", "artifact"),
("image", "image"),
};
foreach (var (key, type) in candidates)
{
if (!TryGetPayloadString(payload, key, out var value))
{
continue;
}
var normalized = NormalizeDigest(value);
if (normalized is null)
{
continue;
}
scopeType = type;
scopeId = normalized;
return true;
}
foreach (var property in payload.EnumerateObject())
{
if (property.Value.ValueKind != JsonValueKind.String)
{
continue;
}
var normalized = NormalizeDigest(property.Value.GetString());
if (normalized is null)
{
continue;
}
scopeType = property.Name.Contains("artifact", StringComparison.OrdinalIgnoreCase) ? "artifact" : "image";
scopeId = normalized;
return true;
}
scopeType = null;
scopeId = null;
return false;
}
private static string? NormalizeDigest(string? value)
{
if (string.IsNullOrWhiteSpace(value))
{
return null;
}
var trimmed = value.Trim();
return trimmed.StartsWith("sha256:", StringComparison.OrdinalIgnoreCase) ? trimmed : null;
}
private static string ComputeToolchainHash(Job job, JsonElement payload)
{
var scannerVersion = TryGetPayloadString(payload, "scannerVersion", out var scanner) ? scanner : null;
var runtimeVersion = TryGetPayloadString(payload, "runtimeVersion", out var runtime) ? runtime : null;
var material = $"{job.JobType}|{scannerVersion ?? "unknown"}|{runtimeVersion ?? "unknown"}";
var hash = SHA256.HashData(Encoding.UTF8.GetBytes(material));
return Convert.ToHexStringLower(hash.AsSpan(0, 8));
}
private static Job SelectRepresentativeJob(Run run, IReadOnlyList<Job> jobs)
{
// Prefer an in-flight job to surface "started" quickly, even if Run.Status hasn't transitioned yet.

198
src/Orchestrator/StellaOps.Orchestrator/StellaOps.Orchestrator.Infrastructure/Services/SchedulerFailureSignatureLookupClient.cs Normal file
View File

@@ -0,0 +1,198 @@
using System.Net;
using System.Net.Http.Json;
using System.Text.Json;
using Microsoft.Extensions.Logging;
using Microsoft.Extensions.Options;
using StellaOps.Orchestrator.Core.Domain;
using StellaOps.Orchestrator.Infrastructure.Options;
namespace StellaOps.Orchestrator.Infrastructure.Services;
public interface IFailureSignatureLookupClient
{
Task<LastKnownOutcome?> TryGetLastKnownOutcomeAsync(
string tenantId,
string scopeType,
string scopeId,
string toolchainHash,
CancellationToken cancellationToken = default);
}
public sealed class SchedulerFailureSignatureLookupClient : IFailureSignatureLookupClient
{
private const string TenantHeader = "X-Tenant-Id";
private const string ScopeHeader = "X-Scopes";
private const string RequiredScope = "scheduler.runs.read";
private static readonly JsonSerializerOptions JsonOptions = new(JsonSerializerDefaults.Web);
private readonly HttpClient _httpClient;
private readonly IOptionsMonitor<FirstSignalOptions> _optionsMonitor;
private readonly ILogger<SchedulerFailureSignatureLookupClient> _logger;
public SchedulerFailureSignatureLookupClient(
HttpClient httpClient,
IOptionsMonitor<FirstSignalOptions> optionsMonitor,
ILogger<SchedulerFailureSignatureLookupClient> logger)
{
_httpClient = httpClient ?? throw new ArgumentNullException(nameof(httpClient));
_optionsMonitor = optionsMonitor ?? throw new ArgumentNullException(nameof(optionsMonitor));
_logger = logger ?? throw new ArgumentNullException(nameof(logger));
}
public async Task<LastKnownOutcome?> TryGetLastKnownOutcomeAsync(
string tenantId,
string scopeType,
string scopeId,
string toolchainHash,
CancellationToken cancellationToken = default)
{
var options = _optionsMonitor.CurrentValue.FailureSignatures;
if (!options.Enabled)
{
return null;
}
if (string.IsNullOrWhiteSpace(options.SchedulerBaseUrl))
{
return null;
}
if (!Uri.TryCreate(options.SchedulerBaseUrl.Trim(), UriKind.Absolute, out var baseUri))
{
return null;
}
if (string.IsNullOrWhiteSpace(tenantId) ||
string.IsNullOrWhiteSpace(scopeType) ||
string.IsNullOrWhiteSpace(scopeId) ||
string.IsNullOrWhiteSpace(toolchainHash))
{
return null;
}
var normalizedBaseUri = new Uri(baseUri.ToString().TrimEnd('/') + "/", UriKind.Absolute);
var relative = "api/v1/scheduler/failure-signatures/best-match"
+ $"?scopeType={Uri.EscapeDataString(scopeType)}"
+ $"&scopeId={Uri.EscapeDataString(scopeId)}"
+ $"&toolchainHash={Uri.EscapeDataString(toolchainHash)}";
var requestUri = new Uri(normalizedBaseUri, relative);
using var request = new HttpRequestMessage(HttpMethod.Get, requestUri);
request.Headers.TryAddWithoutValidation(TenantHeader, tenantId);
request.Headers.TryAddWithoutValidation(ScopeHeader, RequiredScope);
try
{
using var timeoutCts = CancellationTokenSource.CreateLinkedTokenSource(cancellationToken);
if (options.TimeoutMs > 0)
{
timeoutCts.CancelAfter(TimeSpan.FromMilliseconds(options.TimeoutMs));
}
using var response = await _httpClient
.SendAsync(request, HttpCompletionOption.ResponseHeadersRead, timeoutCts.Token)
.ConfigureAwait(false);
if (response.StatusCode == HttpStatusCode.NoContent)
{
return null;
}
if (!response.IsSuccessStatusCode)
{
_logger.LogDebug(
"Scheduler failure signature lookup returned status {StatusCode} for tenant {TenantId}.",
(int)response.StatusCode,
tenantId);
return null;
}
var payload = await response.Content
.ReadFromJsonAsync<FailureSignatureBestMatchResponse>(JsonOptions, timeoutCts.Token)
.ConfigureAwait(false);
if (payload is null)
{
return null;
}
var token = NormalizeToken(payload);
return new LastKnownOutcome
{
SignatureId = payload.SignatureId.ToString("D"),
ErrorCode = string.IsNullOrWhiteSpace(payload.ErrorCode) ? null : payload.ErrorCode.Trim(),
Token = token,
Excerpt = null,
Confidence = MapConfidence(options, payload),
FirstSeenAt = payload.FirstSeenAt,
HitCount = payload.OccurrenceCount,
};
}
catch (OperationCanceledException)
{
return null;
}
catch (Exception ex)
{
_logger.LogDebug(ex, "Scheduler failure signature lookup failed for tenant {TenantId}.", tenantId);
return null;
}
}
private static string NormalizeToken(FailureSignatureBestMatchResponse payload)
{
if (!string.IsNullOrWhiteSpace(payload.ErrorCode))
{
return payload.ErrorCode.Trim();
}
if (!string.IsNullOrWhiteSpace(payload.ErrorCategory))
{
return payload.ErrorCategory.Trim();
}
return "unknown";
}
private static string MapConfidence(FirstSignalFailureSignatureOptions options, FailureSignatureBestMatchResponse payload)
{
if (payload.ConfidenceScore is { } score)
{
return score switch
{
>= 0.8m => "high",
>= 0.6m => "medium",
_ => "low"
};
}
if (options.HighOccurrenceThreshold > 0 && payload.OccurrenceCount >= options.HighOccurrenceThreshold)
{
return "high";
}
if (options.MediumOccurrenceThreshold > 0 && payload.OccurrenceCount >= options.MediumOccurrenceThreshold)
{
return "medium";
}
return "low";
}
private sealed record FailureSignatureBestMatchResponse
{
public Guid SignatureId { get; init; }
public string ScopeType { get; init; } = string.Empty;
public string ScopeId { get; init; } = string.Empty;
public string ToolchainHash { get; init; } = string.Empty;
public string? ErrorCode { get; init; }
public string? ErrorCategory { get; init; }
public string PredictedOutcome { get; init; } = string.Empty;
public decimal? ConfidenceScore { get; init; }
public int OccurrenceCount { get; init; }
public DateTimeOffset FirstSeenAt { get; init; }
public DateTimeOffset LastSeenAt { get; init; }
}
}

143
src/Orchestrator/StellaOps.Orchestrator/StellaOps.Orchestrator.Tests/Ttfs/FirstSignalServiceTests.cs
View File

@@ -1,3 +1,5 @@
using System.Security.Cryptography;
using System.Text;
using Microsoft.Extensions.Logging.Abstractions;
using Microsoft.Extensions.Options;
using StellaOps.Messaging;
@@ -81,6 +83,7 @@ public sealed class FirstSignalServiceTests
snapshots,
runs,
jobs,
new NullFailureSignatureLookupClient(),
TimeProvider.System,
ttfs,
options,
@@ -155,6 +158,7 @@ public sealed class FirstSignalServiceTests
snapshotRepository: new FakeFirstSignalSnapshotRepository(),
runRepository: new FakeRunRepository(run),
jobRepository: new FakeJobRepository(job),
failureSignatureLookupClient: new NullFailureSignatureLookupClient(),
timeProvider: TimeProvider.System,
ttfsMetrics: ttfs,
options: Options.Create(new FirstSignalOptions()),
@@ -176,6 +180,7 @@ public sealed class FirstSignalServiceTests
snapshotRepository: new FakeFirstSignalSnapshotRepository(),
runRepository: new FakeRunRepository(null),
jobRepository: new FakeJobRepository(),
failureSignatureLookupClient: new NullFailureSignatureLookupClient(),
timeProvider: TimeProvider.System,
ttfsMetrics: ttfs,
options: Options.Create(new FirstSignalOptions()),
@@ -213,6 +218,7 @@ public sealed class FirstSignalServiceTests
snapshotRepository: new FakeFirstSignalSnapshotRepository(),
runRepository: new FakeRunRepository(run),
jobRepository: new FakeJobRepository(),
failureSignatureLookupClient: new NullFailureSignatureLookupClient(),
timeProvider: TimeProvider.System,
ttfsMetrics: ttfs,
options: Options.Create(new FirstSignalOptions()),
@@ -275,6 +281,7 @@ public sealed class FirstSignalServiceTests
snapshotRepo,
runRepository: new FakeRunRepository(null),
jobRepository: new FakeJobRepository(),
failureSignatureLookupClient: new NullFailureSignatureLookupClient(),
timeProvider: TimeProvider.System,
ttfsMetrics: ttfs,
options: Options.Create(new FirstSignalOptions()),
@@ -290,6 +297,142 @@ public sealed class FirstSignalServiceTests
Assert.True(second.CacheHit);
}
[Fact]
public async Task GetFirstSignalAsync_RunFailed_EnrichesLastKnownOutcome_WhenFailureSignatureAvailable()
{
var runId = Guid.NewGuid();
var jobId = Guid.NewGuid();
var now = new DateTimeOffset(2025, 12, 18, 12, 0, 0, TimeSpan.Zero);
var run = new Run(
RunId: runId,
TenantId: TenantId,
ProjectId: null,
SourceId: Guid.NewGuid(),
RunType: "scan",
Status: RunStatus.Failed,
CorrelationId: "corr-ttfs",
TotalJobs: 1,
CompletedJobs: 1,
SucceededJobs: 0,
FailedJobs: 1,
CreatedAt: now,
StartedAt: now.AddSeconds(5),
CompletedAt: now.AddMinutes(1),
CreatedBy: "system",
Metadata: null);
var jobPayload = """{"repository":"acme/repo","scannerVersion":"1.2.3","runtimeVersion":"7.0.0"}""";
var job = new Job(
JobId: jobId,
TenantId: TenantId,
ProjectId: null,
RunId: runId,
JobType: "scan.image",
Status: JobStatus.Failed,
Priority: 0,
Attempt: 1,
MaxAttempts: 1,
PayloadDigest: new string('b', 64),
Payload: jobPayload,
IdempotencyKey: "idem-ttfs",
CorrelationId: null,
LeaseId: Guid.NewGuid(),
WorkerId: "worker-1",
TaskRunnerId: null,
LeaseUntil: null,
CreatedAt: now,
ScheduledAt: now,
LeasedAt: now.AddSeconds(10),
CompletedAt: now.AddMinutes(1),
NotBefore: null,
Reason: "failed",
ReplayOf: null,
CreatedBy: "system");
var expectedHashMaterial = $"{job.JobType}|1.2.3|7.0.0";
var expectedHash = SHA256.HashData(Encoding.UTF8.GetBytes(expectedHashMaterial));
var expectedToolchainHash = Convert.ToHexStringLower(expectedHash.AsSpan(0, 8));
var outcome = new LastKnownOutcome
{
SignatureId = "sig-1",
ErrorCode = "E123",
Token = "E123",
Excerpt = null,
Confidence = "high",
FirstSeenAt = now.AddDays(-2),
HitCount = 7
};
var failureSignatures = new CapturingFailureSignatureLookupClient(outcome);
using var ttfs = new TimeToFirstSignalMetrics();
var service = new FirstSignalService(
cache: new FakeFirstSignalCache(),
snapshotRepository: new FakeFirstSignalSnapshotRepository(),
runRepository: new FakeRunRepository(run),
jobRepository: new FakeJobRepository(job),
failureSignatureLookupClient: failureSignatures,
timeProvider: TimeProvider.System,
ttfsMetrics: ttfs,
options: Options.Create(new FirstSignalOptions
{
FailureSignatures = new FirstSignalFailureSignatureOptions { Enabled = true }
}),
logger: NullLogger<FirstSignalService>.Instance);
var result = await service.GetFirstSignalAsync(runId, TenantId);
Assert.Equal(StellaOps.Orchestrator.Core.Services.FirstSignalResultStatus.Found, result.Status);
Assert.Equal("failure_index", result.Source);
Assert.NotNull(result.Signal);
Assert.Equal(FirstSignalKind.Failed, result.Signal!.Kind);
Assert.Equal("failure_index", result.Signal.Diagnostics.Source);
Assert.NotNull(result.Signal.LastKnownOutcome);
Assert.Equal("sig-1", result.Signal.LastKnownOutcome!.SignatureId);
Assert.NotNull(failureSignatures.LastRequest);
Assert.Equal(TenantId, failureSignatures.LastRequest!.Value.TenantId);
Assert.Equal("repo", failureSignatures.LastRequest!.Value.ScopeType);
Assert.Equal("acme/repo", failureSignatures.LastRequest!.Value.ScopeId);
Assert.Equal(expectedToolchainHash, failureSignatures.LastRequest!.Value.ToolchainHash);
}
private sealed class NullFailureSignatureLookupClient : IFailureSignatureLookupClient
{
public Task<LastKnownOutcome?> TryGetLastKnownOutcomeAsync(
string tenantId,
string scopeType,
string scopeId,
string toolchainHash,
CancellationToken cancellationToken = default) =>
Task.FromResult<LastKnownOutcome?>(null);
}
private sealed class CapturingFailureSignatureLookupClient : IFailureSignatureLookupClient
{
private readonly LastKnownOutcome _outcome;
public CapturingFailureSignatureLookupClient(LastKnownOutcome outcome)
{
_outcome = outcome;
}
public (string TenantId, string ScopeType, string ScopeId, string ToolchainHash)? LastRequest { get; private set; }
public Task<LastKnownOutcome?> TryGetLastKnownOutcomeAsync(
string tenantId,
string scopeType,
string scopeId,
string toolchainHash,
CancellationToken cancellationToken = default)
{
LastRequest = (tenantId, scopeType, scopeId, toolchainHash);
return Task.FromResult<LastKnownOutcome?>(_outcome);
}
}
private sealed class FakeFirstSignalCache : IFirstSignalCache
{
private readonly Dictionary<(string TenantId, Guid RunId), FirstSignalCacheEntry> _entries = new();

12
src/Orchestrator/StellaOps.Orchestrator/StellaOps.Orchestrator.WebService/Contracts/FirstSignalResponse.cs
View File

@@ -18,6 +18,7 @@ public sealed record FirstSignalDto
public required string Message { get; init; }
public required DateTimeOffset At { get; init; }
public FirstSignalArtifactDto? Artifact { get; init; }
public FirstSignalLastKnownOutcomeDto? LastKnownOutcome { get; init; }
}
public sealed record FirstSignalArtifactDto
@@ -26,6 +27,17 @@ public sealed record FirstSignalArtifactDto
public FirstSignalRangeDto? Range { get; init; }
}
public sealed record FirstSignalLastKnownOutcomeDto
{
public required string SignatureId { get; init; }
public string? ErrorCode { get; init; }
public required string Token { get; init; }
public string? Excerpt { get; init; }
public required string Confidence { get; init; }
public required DateTimeOffset FirstSeenAt { get; init; }
public required int HitCount { get; init; }
}
public sealed record FirstSignalRangeDto
{
public required int Start { get; init; }

12
src/Orchestrator/StellaOps.Orchestrator/StellaOps.Orchestrator.WebService/Endpoints/FirstSignalEndpoints.cs
View File

@@ -97,6 +97,18 @@ public static class FirstSignalEndpoints
{
Kind = signal.Scope.Type,
Range = null
},
LastKnownOutcome = signal.LastKnownOutcome is null
? null
: new FirstSignalLastKnownOutcomeDto
{
SignatureId = signal.LastKnownOutcome.SignatureId,
ErrorCode = signal.LastKnownOutcome.ErrorCode,
Token = signal.LastKnownOutcome.Token,
Excerpt = signal.LastKnownOutcome.Excerpt,
Confidence = signal.LastKnownOutcome.Confidence,
FirstSeenAt = signal.LastKnownOutcome.FirstSeenAt,
HitCount = signal.LastKnownOutcome.HitCount
}
}
};

10
src/Orchestrator/StellaOps.Orchestrator/TASKS.md
View File

@@ -31,3 +31,13 @@ Status mirror for `docs/implplan/SPRINT_0339_0001_0001_first_signal_api.md`. Upd
| 1 | ORCH-TTFS-0339-001 | DONE | First signal API delivered (service/repo/cache/endpoint/ETag/SSE/tests/docs). |
Last synced: 2025-12-15 (UTC).
## SPRINT_0341_0001_0001 TTFS Enhancements
Status mirror for `docs/implplan/SPRINT_0341_0001_0001_ttfs_enhancements.md`. Update alongside the sprint file to avoid drift.
| # | Task ID | Status | Notes |
| --- | --- | --- | --- |
| 1 | TTFS-T4 | DONE | Enrich FirstSignal with best-effort failure signature lookup via Scheduler WebService; surfaces `lastKnownOutcome` in API response. |
Last synced: 2025-12-18 (UTC).

26
src/Scanner/StellaOps.Scanner.WebService/Endpoints/ObservabilityEndpoints.cs Normal file
View File

@@ -0,0 +1,26 @@
using Microsoft.AspNetCore.Http;
using Microsoft.AspNetCore.Routing;
using StellaOps.Scanner.WebService.Services;
namespace StellaOps.Scanner.WebService.Endpoints;
internal static class ObservabilityEndpoints
{
public static void MapObservabilityEndpoints(this IEndpointRouteBuilder endpoints)
{
ArgumentNullException.ThrowIfNull(endpoints);
endpoints.MapGet("/metrics", HandleMetricsAsync)
.WithName("scanner.metrics")
.Produces(StatusCodes.Status200OK);
}
private static IResult HandleMetricsAsync(OfflineKitMetricsStore metricsStore)
{
ArgumentNullException.ThrowIfNull(metricsStore);
var payload = metricsStore.RenderPrometheus();
return Results.Text(payload, contentType: "text/plain; version=0.0.4; charset=utf-8");
}
}

230
src/Scanner/StellaOps.Scanner.WebService/Endpoints/OfflineKitEndpoints.cs Normal file
View File

@@ -0,0 +1,230 @@
using System.Linq;
using System.Security.Claims;
using System.Text.Json;
using Microsoft.AspNetCore.Http;
using Microsoft.AspNetCore.Mvc;
using Microsoft.AspNetCore.Routing;
using Microsoft.Extensions.Options;
using StellaOps.Auth.Abstractions;
using StellaOps.Scanner.Core.Configuration;
using StellaOps.Scanner.WebService.Constants;
using StellaOps.Scanner.WebService.Infrastructure;
using StellaOps.Scanner.WebService.Security;
using StellaOps.Scanner.WebService.Services;
namespace StellaOps.Scanner.WebService.Endpoints;
internal static class OfflineKitEndpoints
{
private static readonly JsonSerializerOptions JsonOptions = new(JsonSerializerDefaults.Web)
{
PropertyNameCaseInsensitive = true
};
public static void MapOfflineKitEndpoints(this IEndpointRouteBuilder endpoints)
{
ArgumentNullException.ThrowIfNull(endpoints);
var group = endpoints
.MapGroup("/api/offline-kit")
.WithTags("Offline Kit");
group.MapPost("/import", HandleImportAsync)
.WithName("scanner.offline-kit.import")
.RequireAuthorization(ScannerPolicies.OfflineKitImport)
.Produces<OfflineKitImportResponseTransport>(StatusCodes.Status202Accepted)
.Produces<ProblemDetails>(StatusCodes.Status400BadRequest)
.Produces<ProblemDetails>(StatusCodes.Status404NotFound)
.Produces<ProblemDetails>(StatusCodes.Status422UnprocessableEntity);
group.MapGet("/status", HandleStatusAsync)
.WithName("scanner.offline-kit.status")
.RequireAuthorization(ScannerPolicies.OfflineKitStatusRead)
.Produces<OfflineKitStatusTransport>(StatusCodes.Status200OK)
.Produces(StatusCodes.Status204NoContent)
.Produces<ProblemDetails>(StatusCodes.Status404NotFound);
}
private static async Task<IResult> HandleImportAsync(
HttpContext context,
HttpRequest request,
IOptionsMonitor<OfflineKitOptions> offlineKitOptions,
OfflineKitImportService importService,
CancellationToken cancellationToken)
{
ArgumentNullException.ThrowIfNull(context);
ArgumentNullException.ThrowIfNull(request);
ArgumentNullException.ThrowIfNull(offlineKitOptions);
ArgumentNullException.ThrowIfNull(importService);
if (!offlineKitOptions.CurrentValue.Enabled)
{
return ProblemResultFactory.Create(
context,
ProblemTypes.NotFound,
"Offline kit import is not enabled",
StatusCodes.Status404NotFound);
}
if (!request.HasFormContentType)
{
return ProblemResultFactory.Create(
context,
ProblemTypes.Validation,
"Invalid offline kit import request",
StatusCodes.Status400BadRequest,
detail: "Request must be multipart/form-data.");
}
var form = await request.ReadFormAsync(cancellationToken).ConfigureAwait(false);
var metadataJson = form["metadata"].FirstOrDefault();
if (string.IsNullOrWhiteSpace(metadataJson))
{
return ProblemResultFactory.Create(
context,
ProblemTypes.Validation,
"Invalid offline kit import request",
StatusCodes.Status400BadRequest,
detail: "Missing 'metadata' form field.");
}
OfflineKitImportMetadata? metadata;
try
{
metadata = JsonSerializer.Deserialize<OfflineKitImportMetadata>(metadataJson, JsonOptions);
}
catch (JsonException ex)
{
return ProblemResultFactory.Create(
context,
ProblemTypes.Validation,
"Invalid offline kit import request",
StatusCodes.Status400BadRequest,
detail: $"Failed to parse metadata JSON: {ex.Message}");
}
if (metadata is null)
{
return ProblemResultFactory.Create(
context,
ProblemTypes.Validation,
"Invalid offline kit import request",
StatusCodes.Status400BadRequest,
detail: "Metadata payload is empty.");
}
var bundle = form.Files.GetFile("bundle");
if (bundle is null)
{
return ProblemResultFactory.Create(
context,
ProblemTypes.Validation,
"Invalid offline kit import request",
StatusCodes.Status400BadRequest,
detail: "Missing 'bundle' file upload.");
}
var manifest = form.Files.GetFile("manifest");
var bundleSignature = form.Files.GetFile("bundleSignature");
var manifestSignature = form.Files.GetFile("manifestSignature");
var tenantId = ResolveTenant(context);
var actor = ResolveActor(context);
try
{
var response = await importService.ImportAsync(
new OfflineKitImportRequest(
tenantId,
actor,
metadata,
bundle,
manifest,
bundleSignature,
manifestSignature),
cancellationToken).ConfigureAwait(false);
return Results.Accepted("/api/offline-kit/status", response);
}
catch (OfflineKitImportException ex)
{
return ProblemResultFactory.Create(
context,
ProblemTypes.Validation,
"Offline kit import failed",
ex.StatusCode,
detail: ex.Message,
extensions: new Dictionary<string, object?>
{
["reason_code"] = ex.ReasonCode,
["notes"] = ex.Notes
});
}
}
private static async Task<IResult> HandleStatusAsync(
HttpContext context,
IOptionsMonitor<OfflineKitOptions> offlineKitOptions,
OfflineKitStateStore stateStore,
CancellationToken cancellationToken)
{
ArgumentNullException.ThrowIfNull(context);
ArgumentNullException.ThrowIfNull(offlineKitOptions);
ArgumentNullException.ThrowIfNull(stateStore);
if (!offlineKitOptions.CurrentValue.Enabled)
{
return ProblemResultFactory.Create(
context,
ProblemTypes.NotFound,
"Offline kit status is not enabled",
StatusCodes.Status404NotFound);
}
var tenantId = ResolveTenant(context);
var status = await stateStore.LoadStatusAsync(tenantId, cancellationToken).ConfigureAwait(false);
return status is null
? Results.NoContent()
: Results.Ok(status);
}
private static string ResolveTenant(HttpContext context)
{
var tenant = context.User?.FindFirstValue(StellaOpsClaimTypes.Tenant);
if (!string.IsNullOrWhiteSpace(tenant))
{
return tenant.Trim();
}
if (context.Request.Headers.TryGetValue("X-Stella-Tenant", out var headerTenant))
{
var headerValue = headerTenant.ToString();
if (!string.IsNullOrWhiteSpace(headerValue))
{
return headerValue.Trim();
}
}
return "default";
}
private static string ResolveActor(HttpContext context)
{
var subject = context.User?.FindFirstValue(StellaOpsClaimTypes.Subject);
if (!string.IsNullOrWhiteSpace(subject))
{
return subject.Trim();
}
var clientId = context.User?.FindFirstValue(StellaOpsClaimTypes.ClientId);
if (!string.IsNullOrWhiteSpace(clientId))
{
return clientId.Trim();
}
return "anonymous";
}
}

307
src/Scanner/StellaOps.Scanner.WebService/Endpoints/ReachabilityDriftEndpoints.cs Normal file
View File

@@ -0,0 +1,307 @@
using System.Collections.Immutable;
using System.Text.Json;
using System.Text.Json.Serialization;
using Microsoft.AspNetCore.Http;
using Microsoft.AspNetCore.Routing;
using StellaOps.Scanner.ReachabilityDrift;
using StellaOps.Scanner.ReachabilityDrift.Services;
using StellaOps.Scanner.Storage.Repositories;
using StellaOps.Scanner.WebService.Constants;
using StellaOps.Scanner.WebService.Domain;
using StellaOps.Scanner.WebService.Infrastructure;
using StellaOps.Scanner.WebService.Security;
using StellaOps.Scanner.WebService.Services;
namespace StellaOps.Scanner.WebService.Endpoints;
internal static class ReachabilityDriftEndpoints
{
private static readonly JsonSerializerOptions SerializerOptions = new(JsonSerializerDefaults.Web)
{
DefaultIgnoreCondition = JsonIgnoreCondition.WhenWritingNull,
Converters = { new JsonStringEnumConverter() }
};
public static void MapReachabilityDriftScanEndpoints(this RouteGroupBuilder scansGroup)
{
ArgumentNullException.ThrowIfNull(scansGroup);
// GET /scans/{scanId}/drift?baseScanId=...&language=dotnet&includeFullPath=false
scansGroup.MapGet("/{scanId}/drift", HandleGetDriftAsync)
.WithName("scanner.scans.reachability-drift")
.WithTags("ReachabilityDrift")
.Produces<ReachabilityDriftResult>(StatusCodes.Status200OK)
.Produces(StatusCodes.Status400BadRequest)
.Produces(StatusCodes.Status404NotFound)
.RequireAuthorization(ScannerPolicies.ScansRead);
}
public static void MapReachabilityDriftRootEndpoints(this RouteGroupBuilder apiGroup)
{
ArgumentNullException.ThrowIfNull(apiGroup);
var driftGroup = apiGroup.MapGroup("/drift");
// GET /drift/{driftId}/sinks?direction=became_reachable&offset=0&limit=100
driftGroup.MapGet("/{driftId:guid}/sinks", HandleListSinksAsync)
.WithName("scanner.drift.sinks")
.WithTags("ReachabilityDrift")
.Produces<DriftedSinksResponseDto>(StatusCodes.Status200OK)
.Produces(StatusCodes.Status400BadRequest)
.Produces(StatusCodes.Status404NotFound)
.RequireAuthorization(ScannerPolicies.ScansRead);
}
private static async Task<IResult> HandleGetDriftAsync(
string scanId,
string? baseScanId,
string? language,
bool? includeFullPath,
IScanCoordinator coordinator,
ICallGraphSnapshotRepository callGraphSnapshots,
CodeChangeFactExtractor codeChangeFactExtractor,
ICodeChangeRepository codeChangeRepository,
ReachabilityDriftDetector driftDetector,
IReachabilityDriftResultRepository driftRepository,
HttpContext context,
CancellationToken cancellationToken)
{
ArgumentNullException.ThrowIfNull(coordinator);
ArgumentNullException.ThrowIfNull(callGraphSnapshots);
ArgumentNullException.ThrowIfNull(codeChangeFactExtractor);
ArgumentNullException.ThrowIfNull(codeChangeRepository);
ArgumentNullException.ThrowIfNull(driftDetector);
ArgumentNullException.ThrowIfNull(driftRepository);
if (!ScanId.TryParse(scanId, out var headScan))
{
return ProblemResultFactory.Create(
context,
ProblemTypes.Validation,
"Invalid scan identifier",
StatusCodes.Status400BadRequest,
detail: "Scan identifier is required.");
}
var resolvedLanguage = string.IsNullOrWhiteSpace(language) ? "dotnet" : language.Trim();
var headSnapshot = await coordinator.GetAsync(headScan, cancellationToken).ConfigureAwait(false);
if (headSnapshot is null)
{
return ProblemResultFactory.Create(
context,
ProblemTypes.NotFound,
"Scan not found",
StatusCodes.Status404NotFound,
detail: "Requested scan could not be located.");
}
if (string.IsNullOrWhiteSpace(baseScanId))
{
var existing = await driftRepository.TryGetLatestForHeadAsync(headScan.Value, resolvedLanguage, cancellationToken)
.ConfigureAwait(false);
if (existing is null)
{
return ProblemResultFactory.Create(
context,
ProblemTypes.NotFound,
"Drift result not found",
StatusCodes.Status404NotFound,
detail: $"No reachability drift result recorded for scan {scanId} (language={resolvedLanguage}).");
}
return Json(existing, StatusCodes.Status200OK);
}
if (!ScanId.TryParse(baseScanId, out var baseScan))
{
return ProblemResultFactory.Create(
context,
ProblemTypes.Validation,
"Invalid base scan identifier",
StatusCodes.Status400BadRequest,
detail: "Query parameter 'baseScanId' must be a valid scan id.");
}
var baselineSnapshot = await coordinator.GetAsync(baseScan, cancellationToken).ConfigureAwait(false);
if (baselineSnapshot is null)
{
return ProblemResultFactory.Create(
context,
ProblemTypes.NotFound,
"Base scan not found",
StatusCodes.Status404NotFound,
detail: "Base scan could not be located.");
}
var baseGraph = await callGraphSnapshots.TryGetLatestAsync(baseScan.Value, resolvedLanguage, cancellationToken)
.ConfigureAwait(false);
if (baseGraph is null)
{
return ProblemResultFactory.Create(
context,
ProblemTypes.NotFound,
"Base call graph not found",
StatusCodes.Status404NotFound,
detail: $"No call graph snapshot found for base scan {baseScan.Value} (language={resolvedLanguage}).");
}
var headGraph = await callGraphSnapshots.TryGetLatestAsync(headScan.Value, resolvedLanguage, cancellationToken)
.ConfigureAwait(false);
if (headGraph is null)
{
return ProblemResultFactory.Create(
context,
ProblemTypes.NotFound,
"Head call graph not found",
StatusCodes.Status404NotFound,
detail: $"No call graph snapshot found for head scan {headScan.Value} (language={resolvedLanguage}).");
}
try
{
var codeChanges = codeChangeFactExtractor.Extract(baseGraph, headGraph);
await codeChangeRepository.StoreAsync(codeChanges, cancellationToken).ConfigureAwait(false);
var drift = driftDetector.Detect(
baseGraph,
headGraph,
codeChanges,
includeFullPath: includeFullPath == true);
await driftRepository.StoreAsync(drift, cancellationToken).ConfigureAwait(false);
return Json(drift, StatusCodes.Status200OK);
}
catch (ArgumentException ex)
{
return ProblemResultFactory.Create(
context,
ProblemTypes.Validation,
"Invalid drift request",
StatusCodes.Status400BadRequest,
detail: ex.Message);
}
}
private static async Task<IResult> HandleListSinksAsync(
Guid driftId,
string? direction,
int? offset,
int? limit,
IReachabilityDriftResultRepository driftRepository,
HttpContext context,
CancellationToken cancellationToken)
{
ArgumentNullException.ThrowIfNull(driftRepository);
if (driftId == Guid.Empty)
{
return ProblemResultFactory.Create(
context,
ProblemTypes.Validation,
"Invalid drift identifier",
StatusCodes.Status400BadRequest,
detail: "driftId must be a non-empty GUID.");
}
if (!TryParseDirection(direction, out var parsedDirection))
{
return ProblemResultFactory.Create(
context,
ProblemTypes.Validation,
"Invalid direction",
StatusCodes.Status400BadRequest,
detail: "direction must be 'became_reachable' or 'became_unreachable'.");
}
var resolvedOffset = offset ?? 0;
if (resolvedOffset < 0)
{
return ProblemResultFactory.Create(
context,
ProblemTypes.Validation,
"Invalid offset",
StatusCodes.Status400BadRequest,
detail: "offset must be >= 0.");
}
var resolvedLimit = limit ?? 100;
if (resolvedLimit <= 0 || resolvedLimit > 500)
{
return ProblemResultFactory.Create(
context,
ProblemTypes.Validation,
"Invalid limit",
StatusCodes.Status400BadRequest,
detail: "limit must be between 1 and 500.");
}
if (!await driftRepository.ExistsAsync(driftId, cancellationToken).ConfigureAwait(false))
{
return ProblemResultFactory.Create(
context,
ProblemTypes.NotFound,
"Drift result not found",
StatusCodes.Status404NotFound,
detail: "Requested drift result could not be located.");
}
var sinks = await driftRepository.ListSinksAsync(
driftId,
parsedDirection,
resolvedOffset,
resolvedLimit,
cancellationToken).ConfigureAwait(false);
var response = new DriftedSinksResponseDto(
DriftId: driftId,
Direction: parsedDirection,
Offset: resolvedOffset,
Limit: resolvedLimit,
Count: sinks.Count,
Sinks: sinks.ToImmutableArray());
return Json(response, StatusCodes.Status200OK);
}
private static bool TryParseDirection(string? direction, out DriftDirection parsed)
{
if (string.IsNullOrWhiteSpace(direction))
{
parsed = DriftDirection.BecameReachable;
return true;
}
var normalized = direction.Trim().ToLowerInvariant();
parsed = normalized switch
{
"became_reachable" or "newly_reachable" or "reachable" or "up" => DriftDirection.BecameReachable,
"became_unreachable" or "newly_unreachable" or "unreachable" or "down" => DriftDirection.BecameUnreachable,
_ => DriftDirection.BecameReachable
};
return normalized is "became_reachable"
or "newly_reachable"
or "reachable"
or "up"
or "became_unreachable"
or "newly_unreachable"
or "unreachable"
or "down";
}
private static IResult Json<T>(T value, int statusCode)
{
var payload = JsonSerializer.Serialize(value, SerializerOptions);
return Results.Content(payload, "application/json", System.Text.Encoding.UTF8, statusCode);
}
}
internal sealed record DriftedSinksResponseDto(
Guid DriftId,
DriftDirection Direction,
int Offset,
int Limit,
int Count,
ImmutableArray<DriftedSink> Sinks);

3
src/Scanner/StellaOps.Scanner.WebService/Endpoints/ReachabilityEndpoints.cs
View File

@@ -1,6 +1,7 @@
using System.Text.Json;
using System.Text.Json.Serialization;
using Microsoft.AspNetCore.Http;
using Microsoft.AspNetCore.Mvc;
using Microsoft.AspNetCore.Routing;
using StellaOps.Scanner.WebService.Constants;
using StellaOps.Scanner.WebService.Contracts;
@@ -63,7 +64,7 @@ internal static class ReachabilityEndpoints
string scanId,
ComputeReachabilityRequestDto? request,
IScanCoordinator coordinator,
IReachabilityComputeService computeService,
[FromServices] IReachabilityComputeService computeService,
HttpContext context,
CancellationToken cancellationToken)
{

1
src/Scanner/StellaOps.Scanner.WebService/Endpoints/ScanEndpoints.cs
View File

@@ -83,6 +83,7 @@ internal static class ScanEndpoints
scans.MapCallGraphEndpoints();
scans.MapSbomEndpoints();
scans.MapReachabilityEndpoints();
scans.MapReachabilityDriftScanEndpoints();
scans.MapExportEndpoints();
}

64
src/Scanner/StellaOps.Scanner.WebService/Endpoints/SmartDiffEndpoints.cs
View File

@@ -5,6 +5,7 @@ using Microsoft.AspNetCore.Routing;
using StellaOps.Scanner.SmartDiff.Detection;
using StellaOps.Scanner.SmartDiff.Output;
using StellaOps.Scanner.Storage.Postgres;
using StellaOps.Scanner.WebService.Services;
using StellaOps.Scanner.WebService.Security;
namespace StellaOps.Scanner.WebService.Endpoints;
@@ -80,7 +81,7 @@ internal static class SmartDiffEndpoints
// Get scan metadata if available
string? baseDigest = null;
string? targetDigest = null;
DateTimeOffset scanTime = DateTimeOffset.UtcNow;
DateTimeOffset scanTime = DateTimeOffset.UnixEpoch;
if (metadataRepo is not null)
{
@@ -99,13 +100,16 @@ internal static class SmartDiffEndpoints
ScanTime: scanTime,
BaseDigest: baseDigest,
TargetDigest: targetDigest,
MaterialChanges: changes.Select(c => new MaterialRiskChange(
VulnId: c.VulnId,
ComponentPurl: c.ComponentPurl,
Direction: c.IsRiskIncrease ? RiskDirection.Increased : RiskDirection.Decreased,
Reason: c.ChangeReason,
FilePath: c.FilePath
)).ToList(),
MaterialChanges: changes
.Where(c => c.HasMaterialChange)
.Select(c => new MaterialRiskChange(
VulnId: c.FindingKey.VulnId,
ComponentPurl: c.FindingKey.ComponentPurl,
Direction: ToSarifRiskDirection(c),
Reason: ToSarifReason(c),
FilePath: null
))
.ToList(),
HardeningRegressions: [],
VexCandidates: [],
ReachabilityChanges: []);
@@ -120,7 +124,7 @@ internal static class SmartDiffEndpoints
};
var generator = new SarifOutputGenerator();
var sarifJson = generator.Generate(sarifInput, options);
var sarifJson = generator.GenerateJson(sarifInput, options);
// Return as SARIF content type with proper filename
var fileName = $"smartdiff-{scanId}.sarif";
@@ -130,6 +134,46 @@ internal static class SmartDiffEndpoints
statusCode: StatusCodes.Status200OK);
}
private static StellaOps.Scanner.SmartDiff.Output.RiskDirection ToSarifRiskDirection(MaterialRiskChangeResult change)
{
if (change.Changes.IsDefaultOrEmpty)
{
return StellaOps.Scanner.SmartDiff.Output.RiskDirection.Changed;
}
var hasIncreased = change.Changes.Any(c => c.Direction == StellaOps.Scanner.SmartDiff.Detection.RiskDirection.Increased);
var hasDecreased = change.Changes.Any(c => c.Direction == StellaOps.Scanner.SmartDiff.Detection.RiskDirection.Decreased);
return (hasIncreased, hasDecreased) switch
{
(true, false) => StellaOps.Scanner.SmartDiff.Output.RiskDirection.Increased,
(false, true) => StellaOps.Scanner.SmartDiff.Output.RiskDirection.Decreased,
_ => StellaOps.Scanner.SmartDiff.Output.RiskDirection.Changed
};
}
private static string ToSarifReason(MaterialRiskChangeResult change)
{
if (change.Changes.IsDefaultOrEmpty)
{
return "material_change";
}
var reasons = change.Changes
.Select(c => c.Reason)
.Where(r => !string.IsNullOrWhiteSpace(r))
.Distinct(StringComparer.Ordinal)
.Order(StringComparer.Ordinal)
.ToArray();
return reasons.Length switch
{
0 => "material_change",
1 => reasons[0],
_ => string.Join("; ", reasons)
};
}
private static string GetScannerVersion()
{
var assembly = typeof(SmartDiffEndpoints).Assembly;
@@ -289,7 +333,7 @@ internal static class SmartDiffEndpoints
};
}
private static VexCandidateDto ToCandidateDto(VexCandidate candidate)
private static VexCandidateDto ToCandidateDto(StellaOps.Scanner.SmartDiff.Detection.VexCandidate candidate)
{
return new VexCandidateDto
{

27
src/Scanner/StellaOps.Scanner.WebService/Program.cs
View File

@@ -12,8 +12,10 @@ using Microsoft.Extensions.DependencyInjection.Extensions;
using Microsoft.Extensions.Options;
using Serilog;
using Serilog.Events;
using StellaOps.Auth.Abstractions;
using StellaOps.Auth.Client;
using StellaOps.Auth.ServerIntegration;
using StellaOps.Authority.Storage.Postgres.Repositories;
using StellaOps.Configuration;
using StellaOps.Plugin.DependencyInjection;
using StellaOps.Cryptography.DependencyInjection;
@@ -24,6 +26,7 @@ using StellaOps.Scanner.Cache;
using StellaOps.Scanner.Core.Configuration;
using StellaOps.Scanner.Core.Contracts;
using StellaOps.Scanner.Core.TrustAnchors;
using StellaOps.Scanner.ReachabilityDrift.DependencyInjection;
using StellaOps.Scanner.Surface.Env;
using StellaOps.Scanner.Surface.FS;
using StellaOps.Scanner.Surface.Secrets;
@@ -79,6 +82,10 @@ builder.Services.AddOptions<OfflineKitOptions>()
.ValidateOnStart();
builder.Services.AddSingleton<IPublicKeyLoader, FileSystemPublicKeyLoader>();
builder.Services.AddSingleton<ITrustAnchorRegistry, TrustAnchorRegistry>();
builder.Services.TryAddScoped<IOfflineKitAuditEmitter, NullOfflineKitAuditEmitter>();
builder.Services.AddSingleton<OfflineKitMetricsStore>();
builder.Services.AddSingleton<OfflineKitStateStore>();
builder.Services.AddScoped<OfflineKitImportService>();
builder.Host.UseSerilog((context, services, loggerConfiguration) =>
{
@@ -104,11 +111,20 @@ builder.Services.AddSingleton<ScanProgressStream>();
builder.Services.AddSingleton<IScanProgressPublisher>(sp => sp.GetRequiredService<ScanProgressStream>());
builder.Services.AddSingleton<IScanProgressReader>(sp => sp.GetRequiredService<ScanProgressStream>());
builder.Services.AddSingleton<IScanCoordinator, InMemoryScanCoordinator>();
builder.Services.AddSingleton<IReachabilityComputeService, NullReachabilityComputeService>();
builder.Services.AddSingleton<IReachabilityQueryService, NullReachabilityQueryService>();
builder.Services.AddSingleton<IReachabilityExplainService, NullReachabilityExplainService>();
builder.Services.AddSingleton<ISarifExportService, NullSarifExportService>();
builder.Services.AddSingleton<ICycloneDxExportService, NullCycloneDxExportService>();
builder.Services.AddSingleton<IOpenVexExportService, NullOpenVexExportService>();
builder.Services.AddScoped<ICallGraphIngestionService, CallGraphIngestionService>();
builder.Services.AddScoped<ISbomIngestionService, SbomIngestionService>();
builder.Services.AddSingleton<IPolicySnapshotRepository, InMemoryPolicySnapshotRepository>();
builder.Services.AddSingleton<IPolicyAuditRepository, InMemoryPolicyAuditRepository>();
builder.Services.AddSingleton<PolicySnapshotStore>();
builder.Services.AddSingleton<PolicyPreviewService>();
builder.Services.AddSingleton<IRecordModeService, RecordModeService>();
builder.Services.AddReachabilityDrift();
builder.Services.AddStellaOpsCrypto();
builder.Services.AddBouncyCastleEd25519Provider();
builder.Services.AddSingleton<IReportSigner, ReportSigner>();
@@ -301,8 +317,12 @@ if (bootstrapOptions.Authority.Enabled)
{
options.AddStellaOpsScopePolicy(ScannerPolicies.ScansEnqueue, bootstrapOptions.Authority.RequiredScopes.ToArray());
options.AddStellaOpsScopePolicy(ScannerPolicies.ScansRead, ScannerAuthorityScopes.ScansRead);
options.AddStellaOpsScopePolicy(ScannerPolicies.ScansWrite, ScannerAuthorityScopes.ScansWrite);
options.AddStellaOpsScopePolicy(ScannerPolicies.Reports, ScannerAuthorityScopes.ReportsRead);
options.AddStellaOpsScopePolicy(ScannerPolicies.RuntimeIngest, ScannerAuthorityScopes.RuntimeIngest);
options.AddStellaOpsScopePolicy(ScannerPolicies.CallGraphIngest, ScannerAuthorityScopes.CallGraphIngest);
options.AddStellaOpsScopePolicy(ScannerPolicies.OfflineKitImport, StellaOpsScopes.AirgapImport);
options.AddStellaOpsScopePolicy(ScannerPolicies.OfflineKitStatusRead, StellaOpsScopes.AirgapStatusRead);
});
}
else
@@ -318,8 +338,12 @@ else
{
options.AddPolicy(ScannerPolicies.ScansEnqueue, policy => policy.RequireAssertion(_ => true));
options.AddPolicy(ScannerPolicies.ScansRead, policy => policy.RequireAssertion(_ => true));
options.AddPolicy(ScannerPolicies.ScansWrite, policy => policy.RequireAssertion(_ => true));
options.AddPolicy(ScannerPolicies.Reports, policy => policy.RequireAssertion(_ => true));
options.AddPolicy(ScannerPolicies.RuntimeIngest, policy => policy.RequireAssertion(_ => true));
options.AddPolicy(ScannerPolicies.CallGraphIngest, policy => policy.RequireAssertion(_ => true));
options.AddPolicy(ScannerPolicies.OfflineKitImport, policy => policy.RequireAssertion(_ => true));
options.AddPolicy(ScannerPolicies.OfflineKitStatusRead, policy => policy.RequireAssertion(_ => true));
});
}
@@ -430,6 +454,8 @@ if (authorityConfigured)
}
app.MapHealthEndpoints();
app.MapObservabilityEndpoints();
app.MapOfflineKitEndpoints();
var apiGroup = app.MapGroup(resolvedOptions.Api.BasePath);
@@ -441,6 +467,7 @@ if (app.Environment.IsEnvironment("Testing"))
}
apiGroup.MapScanEndpoints(resolvedOptions.Api.ScansSegment);
apiGroup.MapReachabilityDriftRootEndpoints();
apiGroup.MapProofSpineEndpoints(resolvedOptions.Api.SpinesSegment, resolvedOptions.Api.ScansSegment);
apiGroup.MapReplayEndpoints();

4
src/Scanner/StellaOps.Scanner.WebService/Security/ScannerAuthorityScopes.cs
View File

@@ -7,6 +7,10 @@ internal static class ScannerAuthorityScopes
{
public const string ScansEnqueue = "scanner.scans.enqueue";
public const string ScansRead = "scanner.scans.read";
public const string ScansWrite = "scanner.scans.write";
public const string ReportsRead = "scanner.reports.read";
public const string RuntimeIngest = "scanner.runtime.ingest";
public const string CallGraphIngest = "scanner.callgraph.ingest";
public const string OfflineKitImport = "scanner.offline-kit.import";
public const string OfflineKitStatusRead = "scanner.offline-kit.status.read";
}

3
src/Scanner/StellaOps.Scanner.WebService/Security/ScannerPolicies.cs
View File

@@ -8,4 +8,7 @@ internal static class ScannerPolicies
public const string Reports = "scanner.reports";
public const string RuntimeIngest = "scanner.runtime.ingest";
public const string CallGraphIngest = "scanner.callgraph.ingest";
public const string OfflineKitImport = "scanner.offline-kit.import";
public const string OfflineKitStatusRead = "scanner.offline-kit.status.read";
}

232
src/Scanner/StellaOps.Scanner.WebService/Services/CallGraphIngestionService.cs Normal file
View File

@@ -0,0 +1,232 @@
using System.Security.Cryptography;
using System.Text;
using System.Text.Json;
using System.Text.Json.Serialization;
using Microsoft.Extensions.Logging;
using Npgsql;
using NpgsqlTypes;
using StellaOps.Scanner.Storage.Postgres;
using StellaOps.Scanner.WebService.Contracts;
using StellaOps.Scanner.WebService.Domain;
namespace StellaOps.Scanner.WebService.Services;
internal sealed class CallGraphIngestionService : ICallGraphIngestionService
{
private const string TenantContext = "00000000-0000-0000-0000-000000000001";
private static readonly Guid TenantId = Guid.Parse(TenantContext);
private static readonly JsonSerializerOptions JsonOptions = new(JsonSerializerDefaults.Web)
{
DefaultIgnoreCondition = JsonIgnoreCondition.WhenWritingNull
};
private readonly ScannerDataSource _dataSource;
private readonly TimeProvider _timeProvider;
private readonly ILogger<CallGraphIngestionService> _logger;
private string SchemaName => _dataSource.SchemaName ?? ScannerDataSource.DefaultSchema;
private string CallGraphIngestionsTable => $"{SchemaName}.callgraph_ingestions";
public CallGraphIngestionService(
ScannerDataSource dataSource,
TimeProvider timeProvider,
ILogger<CallGraphIngestionService> logger)
{
_dataSource = dataSource ?? throw new ArgumentNullException(nameof(dataSource));
_timeProvider = timeProvider ?? throw new ArgumentNullException(nameof(timeProvider));
_logger = logger ?? throw new ArgumentNullException(nameof(logger));
}
public CallGraphValidationResult Validate(CallGraphV1Dto callGraph)
{
ArgumentNullException.ThrowIfNull(callGraph);
var errors = new List<string>();
if (string.IsNullOrWhiteSpace(callGraph.Schema))
{
errors.Add("Schema is required.");
}
else if (!string.Equals(callGraph.Schema, "stella.callgraph.v1", StringComparison.Ordinal))
{
errors.Add($"Unsupported schema '{callGraph.Schema}'. Expected 'stella.callgraph.v1'.");
}
if (string.IsNullOrWhiteSpace(callGraph.ScanKey))
{
errors.Add("ScanKey is required.");
}
if (string.IsNullOrWhiteSpace(callGraph.Language))
{
errors.Add("Language is required.");
}
if (callGraph.Nodes is null || callGraph.Nodes.Count == 0)
{
errors.Add("At least one node is required.");
}
if (callGraph.Edges is null || callGraph.Edges.Count == 0)
{
errors.Add("At least one edge is required.");
}
return errors.Count == 0
? CallGraphValidationResult.Success()
: CallGraphValidationResult.Failure(errors.ToArray());
}
public async Task<ExistingCallGraphDto?> FindByDigestAsync(
ScanId scanId,
string contentDigest,
CancellationToken cancellationToken = default)
{
if (string.IsNullOrWhiteSpace(scanId.Value))
{
return null;
}
if (string.IsNullOrWhiteSpace(contentDigest))
{
return null;
}
var sql = $"""
SELECT id, content_digest, created_at_utc
FROM {CallGraphIngestionsTable}
WHERE tenant_id = @tenant_id
AND scan_id = @scan_id
AND content_digest = @content_digest
LIMIT 1
""";
await using var connection = await _dataSource.OpenConnectionAsync(TenantContext, "reader", cancellationToken)
.ConfigureAwait(false);
await using var command = new NpgsqlCommand(sql, connection);
command.Parameters.AddWithValue("tenant_id", TenantId);
command.Parameters.AddWithValue("scan_id", scanId.Value.Trim());
command.Parameters.AddWithValue("content_digest", contentDigest.Trim());
await using var reader = await command.ExecuteReaderAsync(cancellationToken).ConfigureAwait(false);
if (!await reader.ReadAsync(cancellationToken).ConfigureAwait(false))
{
return null;
}
return new ExistingCallGraphDto(
Id: reader.GetString(0),
Digest: reader.GetString(1),
CreatedAt: reader.GetFieldValue<DateTimeOffset>(2));
}
public async Task<CallGraphIngestionResult> IngestAsync(
ScanId scanId,
CallGraphV1Dto callGraph,
string contentDigest,
CancellationToken cancellationToken = default)
{
ArgumentNullException.ThrowIfNull(callGraph);
ArgumentException.ThrowIfNullOrWhiteSpace(scanId.Value);
ArgumentException.ThrowIfNullOrWhiteSpace(contentDigest);
var normalizedDigest = contentDigest.Trim();
var callgraphId = CreateCallGraphId(scanId, normalizedDigest);
var now = _timeProvider.GetUtcNow();
var nodeCount = callGraph.Nodes?.Count ?? 0;
var edgeCount = callGraph.Edges?.Count ?? 0;
var language = callGraph.Language?.Trim() ?? string.Empty;
var payload = JsonSerializer.Serialize(callGraph, JsonOptions);
var insertSql = $"""
INSERT INTO {CallGraphIngestionsTable} (
id,
tenant_id,
scan_id,
content_digest,
language,
node_count,
edge_count,
created_at_utc,
callgraph_json
) VALUES (
@id,
@tenant_id,
@scan_id,
@content_digest,
@language,
@node_count,
@edge_count,
@created_at_utc,
@callgraph_json::jsonb
)
ON CONFLICT (tenant_id, scan_id, content_digest) DO NOTHING
""";
var selectSql = $"""
SELECT id, content_digest, node_count, edge_count
FROM {CallGraphIngestionsTable}
WHERE tenant_id = @tenant_id
AND scan_id = @scan_id
AND content_digest = @content_digest
LIMIT 1
""";
await using var connection = await _dataSource.OpenConnectionAsync(TenantContext, "writer", cancellationToken)
.ConfigureAwait(false);
await using (var insert = new NpgsqlCommand(insertSql, connection))
{
insert.Parameters.AddWithValue("id", callgraphId);
insert.Parameters.AddWithValue("tenant_id", TenantId);
insert.Parameters.AddWithValue("scan_id", scanId.Value.Trim());
insert.Parameters.AddWithValue("content_digest", normalizedDigest);
insert.Parameters.AddWithValue("language", language);
insert.Parameters.AddWithValue("node_count", nodeCount);
insert.Parameters.AddWithValue("edge_count", edgeCount);
insert.Parameters.AddWithValue("created_at_utc", now.UtcDateTime);
insert.Parameters.Add(new NpgsqlParameter<string>("callgraph_json", NpgsqlDbType.Jsonb) { TypedValue = payload });
await insert.ExecuteNonQueryAsync(cancellationToken).ConfigureAwait(false);
}
await using var select = new NpgsqlCommand(selectSql, connection);
select.Parameters.AddWithValue("tenant_id", TenantId);
select.Parameters.AddWithValue("scan_id", scanId.Value.Trim());
select.Parameters.AddWithValue("content_digest", normalizedDigest);
await using var reader = await select.ExecuteReaderAsync(cancellationToken).ConfigureAwait(false);
if (!await reader.ReadAsync(cancellationToken).ConfigureAwait(false))
{
throw new InvalidOperationException("Call graph ingestion row was not persisted.");
}
var persistedId = reader.GetString(0);
var persistedDigest = reader.GetString(1);
var persistedNodeCount = reader.GetInt32(2);
var persistedEdgeCount = reader.GetInt32(3);
_logger.LogInformation(
"Ingested callgraph scan={ScanId} lang={Language} nodes={Nodes} edges={Edges} digest={Digest}",
scanId.Value,
language,
persistedNodeCount,
persistedEdgeCount,
persistedDigest);
return new CallGraphIngestionResult(
CallgraphId: persistedId,
NodeCount: persistedNodeCount,
EdgeCount: persistedEdgeCount,
Digest: persistedDigest);
}
private static string CreateCallGraphId(ScanId scanId, string contentDigest)
{
var bytes = Encoding.UTF8.GetBytes($"{scanId.Value.Trim()}:{contentDigest.Trim()}");
var hash = SHA256.HashData(bytes);
return $"cg_{Convert.ToHexString(hash).ToLowerInvariant()}";
}
}

11
src/Scanner/StellaOps.Scanner.WebService/Services/FeedChangeRescoreJob.cs
View File

@@ -306,17 +306,6 @@ public interface IFeedSnapshotTracker
Task<FeedSnapshots> GetCurrentSnapshotsAsync(CancellationToken cancellationToken = default);
}
/// <summary>
/// Interface for scan manifest repository operations.
/// </summary>
public interface IScanManifestRepository
{
/// <summary>
/// Find scans affected by feed changes.
/// </summary>
Task<List<string>> FindAffectedScansAsync(AffectedScansQuery query, CancellationToken cancellationToken = default);
}
/// <summary>
/// Metrics for feed change rescore operations.
/// </summary>

9
src/Scanner/StellaOps.Scanner.WebService/Services/IScanMetadataRepository.cs Normal file
View File

@@ -0,0 +1,9 @@
namespace StellaOps.Scanner.WebService.Services;
public interface IScanMetadataRepository
{
Task<ScanMetadata?> GetScanMetadataAsync(string scanId, CancellationToken cancellationToken = default);
}
public sealed record ScanMetadata(string? BaseDigest, string? TargetDigest, DateTimeOffset ScanTime);

11
src/Scanner/StellaOps.Scanner.WebService/Services/NullOfflineKitAuditEmitter.cs Normal file
View File

@@ -0,0 +1,11 @@
using StellaOps.Authority.Storage.Postgres.Models;
using StellaOps.Authority.Storage.Postgres.Repositories;
namespace StellaOps.Scanner.WebService.Services;
internal sealed class NullOfflineKitAuditEmitter : IOfflineKitAuditEmitter
{
public Task RecordAsync(OfflineKitAuditEntity entity, CancellationToken cancellationToken = default)
=> Task.CompletedTask;
}

68
src/Scanner/StellaOps.Scanner.WebService/Services/NullReachabilityServices.cs Normal file
View File

@@ -0,0 +1,68 @@
using StellaOps.Scanner.WebService.Domain;
namespace StellaOps.Scanner.WebService.Services;
internal sealed class NullReachabilityComputeService : IReachabilityComputeService
{
public Task<ComputeJobResult> TriggerComputeAsync(
ScanId scanId,
bool forceRecompute,
IReadOnlyList<string>? entrypoints,
IReadOnlyList<string>? targets,
CancellationToken cancellationToken = default)
{
ArgumentException.ThrowIfNullOrWhiteSpace(scanId.Value);
var jobId = $"reachability_{scanId.Value}";
return Task.FromResult(new ComputeJobResult(
JobId: jobId,
Status: "scheduled",
AlreadyInProgress: false,
EstimatedDuration: null));
}
}
internal sealed class NullReachabilityQueryService : IReachabilityQueryService
{
public Task<IReadOnlyList<ComponentReachability>> GetComponentsAsync(
ScanId scanId,
string? purlFilter,
string? statusFilter,
CancellationToken cancellationToken = default)
=> Task.FromResult<IReadOnlyList<ComponentReachability>>(Array.Empty<ComponentReachability>());
public Task<IReadOnlyList<ReachabilityFinding>> GetFindingsAsync(
ScanId scanId,
string? cveFilter,
string? statusFilter,
CancellationToken cancellationToken = default)
=> Task.FromResult<IReadOnlyList<ReachabilityFinding>>(Array.Empty<ReachabilityFinding>());
}
internal sealed class NullReachabilityExplainService : IReachabilityExplainService
{
public Task<ReachabilityExplanation?> ExplainAsync(
ScanId scanId,
string cveId,
string purl,
CancellationToken cancellationToken = default)
=> Task.FromResult<ReachabilityExplanation?>(null);
}
internal sealed class NullSarifExportService : ISarifExportService
{
public Task<object?> ExportAsync(ScanId scanId, CancellationToken cancellationToken = default)
=> Task.FromResult<object?>(null);
}
internal sealed class NullCycloneDxExportService : ICycloneDxExportService
{
public Task<object?> ExportWithReachabilityAsync(ScanId scanId, CancellationToken cancellationToken = default)
=> Task.FromResult<object?>(null);
}
internal sealed class NullOpenVexExportService : IOpenVexExportService
{
public Task<object?> ExportAsync(ScanId scanId, CancellationToken cancellationToken = default)
=> Task.FromResult<object?>(null);
}

78
src/Scanner/StellaOps.Scanner.WebService/Services/OfflineKitContracts.cs Normal file
View File

@@ -0,0 +1,78 @@
using Microsoft.AspNetCore.Http;
namespace StellaOps.Scanner.WebService.Services;
internal sealed record OfflineKitImportRequest(
string TenantId,
string Actor,
OfflineKitImportMetadata Metadata,
IFormFile Bundle,
IFormFile? Manifest,
IFormFile? BundleSignature,
IFormFile? ManifestSignature);
internal sealed class OfflineKitImportException : Exception
{
public OfflineKitImportException(int statusCode, string reasonCode, string message, string? notes = null)
: base(message)
{
StatusCode = statusCode;
ReasonCode = reasonCode;
Notes = notes;
}
public int StatusCode { get; }
public string ReasonCode { get; }
public string? Notes { get; }
}
internal sealed class OfflineKitImportMetadata
{
public string? BundleId { get; set; }
public string BundleSha256 { get; set; } = string.Empty;
public long BundleSize { get; set; }
public DateTimeOffset? CapturedAt { get; set; }
public string? Channel { get; set; }
public string? Kind { get; set; }
public bool? IsDelta { get; set; }
public string? BaseBundleId { get; set; }
public string? ManifestSha256 { get; set; }
public long? ManifestSize { get; set; }
}
internal sealed class OfflineKitStatusTransport
{
public OfflineKitStatusBundleTransport? Current { get; set; }
public List<OfflineKitComponentStatusTransport>? Components { get; set; }
}
internal sealed class OfflineKitStatusBundleTransport
{
public string? BundleId { get; set; }
public string? Channel { get; set; }
public string? Kind { get; set; }
public bool? IsDelta { get; set; }
public string? BaseBundleId { get; set; }
public string? BundleSha256 { get; set; }
public long? BundleSize { get; set; }
public DateTimeOffset? CapturedAt { get; set; }
public DateTimeOffset? ImportedAt { get; set; }
}
internal sealed class OfflineKitComponentStatusTransport
{
public string? Name { get; set; }
public string? Version { get; set; }
public string? Digest { get; set; }
public DateTimeOffset? CapturedAt { get; set; }
public long? SizeBytes { get; set; }
}
internal sealed class OfflineKitImportResponseTransport
{
public string? ImportId { get; set; }
public string? Status { get; set; }
public DateTimeOffset? SubmittedAt { get; set; }
public string? Message { get; set; }
}

698
src/Scanner/StellaOps.Scanner.WebService/Services/OfflineKitImportService.cs Normal file
View File

@@ -0,0 +1,698 @@
using System.Diagnostics;
using System.Security.Cryptography;
using System.Text;
using System.Text.Json;
using Microsoft.AspNetCore.Http;
using Microsoft.Extensions.Logging;
using Microsoft.Extensions.Options;
using StellaOps.AirGap.Importer.Contracts;
using StellaOps.AirGap.Importer.Validation;
using StellaOps.Authority.Storage.Postgres.Models;
using StellaOps.Authority.Storage.Postgres.Repositories;
using StellaOps.Scanner.Core.Configuration;
using StellaOps.Scanner.Core.TrustAnchors;
namespace StellaOps.Scanner.WebService.Services;
internal sealed class OfflineKitImportService
{
private readonly IOptionsMonitor<OfflineKitOptions> _options;
private readonly ITrustAnchorRegistry _trustAnchorRegistry;
private readonly OfflineKitMetricsStore _metrics;
private readonly OfflineKitStateStore _stateStore;
private readonly IOfflineKitAuditEmitter _auditEmitter;
private readonly TimeProvider _timeProvider;
private readonly ILogger<OfflineKitImportService> _logger;
public OfflineKitImportService(
IOptionsMonitor<OfflineKitOptions> options,
ITrustAnchorRegistry trustAnchorRegistry,
OfflineKitMetricsStore metrics,
OfflineKitStateStore stateStore,
IOfflineKitAuditEmitter auditEmitter,
TimeProvider timeProvider,
ILogger<OfflineKitImportService> logger)
{
_options = options ?? throw new ArgumentNullException(nameof(options));
_trustAnchorRegistry = trustAnchorRegistry ?? throw new ArgumentNullException(nameof(trustAnchorRegistry));
_metrics = metrics ?? throw new ArgumentNullException(nameof(metrics));
_stateStore = stateStore ?? throw new ArgumentNullException(nameof(stateStore));
_auditEmitter = auditEmitter ?? throw new ArgumentNullException(nameof(auditEmitter));
_timeProvider = timeProvider ?? throw new ArgumentNullException(nameof(timeProvider));
_logger = logger ?? throw new ArgumentNullException(nameof(logger));
}
public async Task<OfflineKitImportResponseTransport> ImportAsync(OfflineKitImportRequest request, CancellationToken cancellationToken)
{
ArgumentNullException.ThrowIfNull(request);
var options = _options.CurrentValue;
if (!options.Enabled)
{
throw new OfflineKitImportException(StatusCodes.Status404NotFound, "OFFLINE_KIT_DISABLED", "Offline kit operations are not enabled.");
}
var tenantId = string.IsNullOrWhiteSpace(request.TenantId) ? "default" : request.TenantId.Trim();
var actor = string.IsNullOrWhiteSpace(request.Actor) ? "anonymous" : request.Actor.Trim();
var now = _timeProvider.GetUtcNow();
var importId = ComputeImportId(tenantId, request.Metadata.BundleSha256, now);
var expectedBundleSha = NormalizeSha256(request.Metadata.BundleSha256);
if (string.IsNullOrWhiteSpace(expectedBundleSha))
{
throw new OfflineKitImportException(StatusCodes.Status400BadRequest, "MANIFEST_INVALID", "metadata.bundleSha256 is required.");
}
var bundleId = string.IsNullOrWhiteSpace(request.Metadata.BundleId)
? $"sha256-{expectedBundleSha[..Math.Min(12, expectedBundleSha.Length)]}"
: request.Metadata.BundleId.Trim();
var bundleDirectory = _stateStore.GetBundleDirectory(tenantId, bundleId);
Directory.CreateDirectory(bundleDirectory);
var bundlePath = Path.Combine(bundleDirectory, "bundle.tgz");
var manifestPath = Path.Combine(bundleDirectory, "manifest.json");
var bundleSignaturePath = Path.Combine(bundleDirectory, "bundle-signature.bin");
var manifestSignaturePath = Path.Combine(bundleDirectory, "manifest-signature.bin");
var statusForMetrics = "success";
var reasonCode = "SUCCESS";
bool dsseVerified = false;
bool rekorVerified = false;
try
{
var (bundleSha, bundleSize) = await SaveWithSha256Async(request.Bundle, bundlePath, cancellationToken).ConfigureAwait(false);
if (!DigestsEqual(bundleSha, expectedBundleSha))
{
statusForMetrics = "failed_hash";
reasonCode = "HASH_MISMATCH";
throw new OfflineKitImportException(StatusCodes.Status422UnprocessableEntity, reasonCode, "Bundle digest does not match metadata.");
}
var components = new List<OfflineKitComponentStatusTransport>();
if (request.Manifest is not null)
{
var (manifestSha, _) = await SaveWithSha256Async(request.Manifest, manifestPath, cancellationToken).ConfigureAwait(false);
if (!string.IsNullOrWhiteSpace(request.Metadata.ManifestSha256)
&& !DigestsEqual(manifestSha, NormalizeSha256(request.Metadata.ManifestSha256)))
{
statusForMetrics = "failed_manifest";
reasonCode = "SIG_FAIL_MANIFEST";
throw new OfflineKitImportException(StatusCodes.Status422UnprocessableEntity, reasonCode, "Manifest digest does not match metadata.");
}
try
{
var manifestJson = await File.ReadAllTextAsync(manifestPath, cancellationToken).ConfigureAwait(false);
components.AddRange(ParseManifestComponents(manifestJson));
}
catch (Exception ex) when (ex is IOException or JsonException)
{
_logger.LogWarning(ex, "offlinekit.import failed to parse manifest components bundle_id={bundle_id}", bundleId);
}
}
byte[]? dsseBytes = null;
DsseEnvelope? envelope = null;
string? dsseNotes = null;
if (request.BundleSignature is not null)
{
dsseBytes = await SaveRawAsync(request.BundleSignature, bundleSignaturePath, cancellationToken).ConfigureAwait(false);
try
{
envelope = DsseEnvelope.Parse(Encoding.UTF8.GetString(dsseBytes));
}
catch (Exception ex)
{
dsseNotes = $"dsse:parse-failed {ex.GetType().Name}";
}
}
if (options.RequireDsse && envelope is null)
{
statusForMetrics = "failed_dsse";
reasonCode = "DSSE_VERIFY_FAIL";
throw new OfflineKitImportException(StatusCodes.Status422UnprocessableEntity, reasonCode, "DSSE envelope is missing.", notes: dsseNotes);
}
if (envelope is not null)
{
var sw = Stopwatch.StartNew();
try
{
dsseVerified = VerifyDsse(bundleSha, request.Metadata, envelope, options);
}
catch (OfflineKitImportException) when (!options.RequireDsse)
{
dsseVerified = false;
}
finally
{
sw.Stop();
_metrics.RecordAttestationVerifyLatency("dsse", sw.Elapsed.TotalSeconds, dsseVerified);
}
if (!dsseVerified)
{
statusForMetrics = "failed_dsse";
reasonCode = "DSSE_VERIFY_FAIL";
if (options.RequireDsse)
{
throw new OfflineKitImportException(StatusCodes.Status422UnprocessableEntity, reasonCode, "DSSE verification failed.", notes: dsseNotes);
}
}
}
if (options.RekorOfflineMode && request.ManifestSignature is not null && dsseBytes is not null)
{
var receiptBytes = await SaveRawAsync(request.ManifestSignature, manifestSignaturePath, cancellationToken).ConfigureAwait(false);
if (LooksLikeRekorReceipt(receiptBytes))
{
var sw = Stopwatch.StartNew();
try
{
rekorVerified = await VerifyRekorAsync(manifestSignaturePath, dsseBytes, options, cancellationToken).ConfigureAwait(false);
}
catch (OfflineKitImportException) when (!options.RequireDsse)
{
rekorVerified = false;
}
finally
{
sw.Stop();
_metrics.RecordRekorInclusionLatency(sw.Elapsed.TotalSeconds, rekorVerified);
}
if (!rekorVerified)
{
statusForMetrics = "failed_rekor";
reasonCode = "REKOR_VERIFY_FAIL";
if (options.RequireDsse)
{
throw new OfflineKitImportException(StatusCodes.Status422UnprocessableEntity, reasonCode, "Rekor receipt verification failed.");
}
}
else
{
_metrics.RecordRekorSuccess("offline");
}
}
}
var status = new OfflineKitStatusTransport
{
Current = new OfflineKitStatusBundleTransport
{
BundleId = bundleId,
Channel = request.Metadata.Channel?.Trim(),
Kind = request.Metadata.Kind?.Trim(),
IsDelta = request.Metadata.IsDelta ?? false,
BaseBundleId = request.Metadata.BaseBundleId?.Trim(),
BundleSha256 = NormalizeSha256(bundleSha),
BundleSize = bundleSize,
CapturedAt = request.Metadata.CapturedAt?.ToUniversalTime(),
ImportedAt = now
},
Components = components.OrderBy(c => c.Name ?? string.Empty, StringComparer.Ordinal).ToList()
};
await _stateStore.SaveStatusAsync(tenantId, status, cancellationToken).ConfigureAwait(false);
_metrics.RecordImport(statusForMetrics, tenantId);
await EmitAuditAsync(tenantId, actor, now, importId, bundleId, result: "accepted", reasonCode, cancellationToken).ConfigureAwait(false);
return new OfflineKitImportResponseTransport
{
ImportId = importId,
Status = statusForMetrics == "success" ? "accepted" : "accepted_with_warnings",
SubmittedAt = now,
Message = statusForMetrics == "success" ? "Accepted." : "Accepted with warnings."
};
}
catch (OfflineKitImportException)
{
_metrics.RecordImport(statusForMetrics, tenantId);
await EmitAuditAsync(tenantId, actor, now, importId, bundleId, result: "failed", reasonCode, cancellationToken).ConfigureAwait(false);
throw;
}
catch (Exception ex)
{
_logger.LogError(ex, "offlinekit.import failed tenant_id={tenant_id} import_id={import_id}", tenantId, importId);
_metrics.RecordImport("failed_unknown", tenantId);
await EmitAuditAsync(tenantId, actor, now, importId, bundleId, result: "failed", "INTERNAL_ERROR", cancellationToken).ConfigureAwait(false);
throw new OfflineKitImportException(StatusCodes.Status500InternalServerError, "INTERNAL_ERROR", "Offline kit import failed.");
}
}
private bool VerifyDsse(string bundleSha256Hex, OfflineKitImportMetadata metadata, DsseEnvelope envelope, OfflineKitOptions options)
{
var purl = ResolvePurl(metadata);
var resolution = _trustAnchorRegistry.ResolveForPurl(purl);
if (resolution is null)
{
throw new OfflineKitImportException(StatusCodes.Status422UnprocessableEntity, "TRUST_ROOT_MISSING", $"No trust anchor matches '{purl}'.");
}
var trustRoots = BuildTrustRoots(resolution, options.TrustRootDirectory ?? string.Empty);
var pae = BuildPreAuthEncoding(envelope.PayloadType, envelope.Payload);
var verified = 0;
foreach (var signature in envelope.Signatures)
{
if (TryVerifySignature(trustRoots, signature, pae))
{
verified++;
}
}
if (verified < Math.Max(1, resolution.MinSignatures))
{
throw new OfflineKitImportException(StatusCodes.Status422UnprocessableEntity, "DSSE_VERIFY_FAIL", "DSSE signature verification failed.");
}
var subjectSha = TryExtractDsseSubjectSha256(envelope);
if (!string.IsNullOrWhiteSpace(subjectSha) && !DigestsEqual(bundleSha256Hex, subjectSha))
{
throw new OfflineKitImportException(StatusCodes.Status422UnprocessableEntity, "DSSE_VERIFY_FAIL", "DSSE subject digest does not match bundle digest.");
}
return true;
}
private static string ResolvePurl(OfflineKitImportMetadata metadata)
{
var kind = string.IsNullOrWhiteSpace(metadata.Kind) ? "offline-kit" : metadata.Kind.Trim().ToLowerInvariant();
return $"pkg:stellaops/{kind}";
}
private static TrustRootConfig BuildTrustRoots(TrustAnchorResolution resolution, string rootBundlePath)
{
var publicKeys = new Dictionary<string, byte[]>(StringComparer.OrdinalIgnoreCase);
foreach (var (keyId, keyBytes) in resolution.PublicKeys)
{
publicKeys[keyId] = keyBytes;
}
var fingerprints = publicKeys.Values
.Select(ComputeFingerprint)
.Distinct(StringComparer.Ordinal)
.ToArray();
return new TrustRootConfig(
RootBundlePath: rootBundlePath,
TrustedKeyFingerprints: fingerprints,
AllowedSignatureAlgorithms: new[] { "rsassa-pss-sha256" },
NotBeforeUtc: null,
NotAfterUtc: null,
PublicKeys: publicKeys);
}
private static byte[] BuildPreAuthEncoding(string payloadType, string payloadBase64)
{
const string paePrefix = "DSSEv1";
var payloadBytes = Convert.FromBase64String(payloadBase64);
var parts = new[] { paePrefix, payloadType, Encoding.UTF8.GetString(payloadBytes) };
var paeBuilder = new StringBuilder();
paeBuilder.Append("PAE:");
paeBuilder.Append(parts.Length);
foreach (var part in parts)
{
paeBuilder.Append(' ');
paeBuilder.Append(part.Length);
paeBuilder.Append(' ');
paeBuilder.Append(part);
}
return Encoding.UTF8.GetBytes(paeBuilder.ToString());
}
private static bool TryVerifySignature(TrustRootConfig trustRoots, DsseSignature signature, byte[] pae)
{
if (!trustRoots.PublicKeys.TryGetValue(signature.KeyId, out var keyBytes))
{
return false;
}
var fingerprint = ComputeFingerprint(keyBytes);
if (!trustRoots.TrustedKeyFingerprints.Contains(fingerprint, StringComparer.Ordinal))
{
return false;
}
try
{
using var rsa = RSA.Create();
rsa.ImportSubjectPublicKeyInfo(keyBytes, out _);
var sig = Convert.FromBase64String(signature.Signature);
return rsa.VerifyData(pae, sig, HashAlgorithmName.SHA256, RSASignaturePadding.Pss);
}
catch
{
return false;
}
}
private static string? TryExtractDsseSubjectSha256(DsseEnvelope envelope)
{
try
{
var payloadBytes = Convert.FromBase64String(envelope.Payload);
using var doc = JsonDocument.Parse(payloadBytes);
if (!doc.RootElement.TryGetProperty("subject", out var subject) || subject.ValueKind != JsonValueKind.Array)
{
return null;
}
foreach (var entry in subject.EnumerateArray())
{
if (entry.ValueKind != JsonValueKind.Object)
{
continue;
}
if (!entry.TryGetProperty("digest", out var digestObj) || digestObj.ValueKind != JsonValueKind.Object)
{
continue;
}
if (digestObj.TryGetProperty("sha256", out var shaProp) && shaProp.ValueKind == JsonValueKind.String)
{
return NormalizeSha256(shaProp.GetString());
}
}
return null;
}
catch
{
return null;
}
}
private static async Task<bool> VerifyRekorAsync(string receiptPath, byte[] dsseBytes, OfflineKitOptions options, CancellationToken cancellationToken)
{
if (string.IsNullOrWhiteSpace(options.RekorSnapshotDirectory))
{
throw new OfflineKitImportException(StatusCodes.Status422UnprocessableEntity, "REKOR_VERIFY_FAIL", "Rekor snapshot directory is not configured.");
}
var publicKeyPath = ResolveRekorPublicKeyPath(options.RekorSnapshotDirectory);
if (publicKeyPath is null)
{
throw new OfflineKitImportException(StatusCodes.Status422UnprocessableEntity, "REKOR_VERIFY_FAIL", "Rekor public key was not found in the snapshot directory.");
}
var dsseSha = SHA256.HashData(dsseBytes);
var result = await RekorOfflineReceiptVerifier.VerifyAsync(receiptPath, dsseSha, publicKeyPath, cancellationToken).ConfigureAwait(false);
return result.Verified;
}
private static string? ResolveRekorPublicKeyPath(string snapshotDirectory)
{
var candidates = new[]
{
Path.Combine(snapshotDirectory, "rekor-pub.pem"),
Path.Combine(snapshotDirectory, "rekor.pub"),
Path.Combine(snapshotDirectory, "tlog-root.pub"),
Path.Combine(snapshotDirectory, "tlog-root.pem"),
Path.Combine(snapshotDirectory, "tlog", "rekor-pub.pem"),
Path.Combine(snapshotDirectory, "tlog", "rekor.pub")
};
foreach (var candidate in candidates)
{
if (File.Exists(candidate))
{
return candidate;
}
}
return null;
}
private static bool LooksLikeRekorReceipt(byte[] payload)
{
try
{
using var doc = JsonDocument.Parse(payload);
var root = doc.RootElement;
if (root.ValueKind != JsonValueKind.Object)
{
return false;
}
return root.TryGetProperty("uuid", out _)
&& root.TryGetProperty("logIndex", out _)
&& root.TryGetProperty("rootHash", out _)
&& root.TryGetProperty("hashes", out _)
&& root.TryGetProperty("checkpoint", out _);
}
catch (JsonException)
{
return false;
}
}
private async Task EmitAuditAsync(
string tenantId,
string actor,
DateTimeOffset timestamp,
string importId,
string bundleId,
string result,
string reasonCode,
CancellationToken cancellationToken)
{
try
{
var entity = new OfflineKitAuditEntity
{
EventId = ComputeDeterministicEventId(tenantId, importId),
TenantId = tenantId,
EventType = "offlinekit.import",
Timestamp = timestamp,
Actor = actor,
Details = JsonSerializer.Serialize(new { importId, bundleId, reasonCode }, new JsonSerializerOptions(JsonSerializerDefaults.Web)),
Result = result
};
await _auditEmitter.RecordAsync(entity, cancellationToken).ConfigureAwait(false);
}
catch (Exception ex)
{
_logger.LogWarning(ex, "offlinekit.audit.emit failed tenant_id={tenant_id} import_id={import_id}", tenantId, importId);
}
}
private static Guid ComputeDeterministicEventId(string tenantId, string importId)
{
var input = $"{tenantId}|{importId}".ToLowerInvariant();
var hash = SHA256.HashData(Encoding.UTF8.GetBytes(input));
Span<byte> guidBytes = stackalloc byte[16];
hash.AsSpan(0, 16).CopyTo(guidBytes);
return new Guid(guidBytes);
}
private static string ComputeImportId(string tenantId, string bundleSha256, DateTimeOffset submittedAt)
{
var input = $"{tenantId}|{NormalizeSha256(bundleSha256)}|{submittedAt.ToUnixTimeSeconds()}".ToLowerInvariant();
var hash = SHA256.HashData(Encoding.UTF8.GetBytes(input));
return $"sha256:{Convert.ToHexString(hash).ToLowerInvariant()}";
}
private static bool DigestsEqual(string computedHex, string expectedHex)
=> string.Equals(NormalizeSha256(computedHex), NormalizeSha256(expectedHex), StringComparison.OrdinalIgnoreCase);
private static string NormalizeSha256(string? digest)
{
if (string.IsNullOrWhiteSpace(digest))
{
return string.Empty;
}
var value = digest.Trim();
if (value.StartsWith("sha256:", StringComparison.OrdinalIgnoreCase))
{
value = value.Substring("sha256:".Length);
}
return value.ToLowerInvariant();
}
private static string ComputeFingerprint(byte[] publicKey)
{
var hash = SHA256.HashData(publicKey);
return Convert.ToHexString(hash).ToLowerInvariant();
}
private static async Task<(string Sha256Hex, long SizeBytes)> SaveWithSha256Async(IFormFile file, string path, CancellationToken cancellationToken)
{
ArgumentNullException.ThrowIfNull(file);
ArgumentException.ThrowIfNullOrWhiteSpace(path);
var directory = Path.GetDirectoryName(path);
if (!string.IsNullOrWhiteSpace(directory))
{
Directory.CreateDirectory(directory);
}
var temp = path + ".tmp";
long size = 0;
using var hasher = IncrementalHash.CreateHash(HashAlgorithmName.SHA256);
await using var output = File.Create(temp);
await using var input = file.OpenReadStream();
var buffer = new byte[128 * 1024];
while (true)
{
var read = await input.ReadAsync(buffer, cancellationToken).ConfigureAwait(false);
if (read == 0)
{
break;
}
hasher.AppendData(buffer, 0, read);
await output.WriteAsync(buffer.AsMemory(0, read), cancellationToken).ConfigureAwait(false);
size += read;
}
var hash = hasher.GetHashAndReset();
var hex = Convert.ToHexString(hash).ToLowerInvariant();
File.Move(temp, path, overwrite: true);
return (hex, size);
}
private static async Task<byte[]> SaveRawAsync(IFormFile file, string path, CancellationToken cancellationToken)
{
ArgumentNullException.ThrowIfNull(file);
ArgumentException.ThrowIfNullOrWhiteSpace(path);
var directory = Path.GetDirectoryName(path);
if (!string.IsNullOrWhiteSpace(directory))
{
Directory.CreateDirectory(directory);
}
await using var output = File.Create(path);
await using var input = file.OpenReadStream();
await input.CopyToAsync(output, cancellationToken).ConfigureAwait(false);
return await File.ReadAllBytesAsync(path, cancellationToken).ConfigureAwait(false);
}
private static IReadOnlyList<OfflineKitComponentStatusTransport> ParseManifestComponents(string manifestJson)
{
if (string.IsNullOrWhiteSpace(manifestJson))
{
return Array.Empty<OfflineKitComponentStatusTransport>();
}
try
{
using var doc = JsonDocument.Parse(manifestJson);
if (doc.RootElement.ValueKind == JsonValueKind.Object &&
doc.RootElement.TryGetProperty("entries", out var entries) &&
entries.ValueKind == JsonValueKind.Array)
{
return ParseEntries(entries);
}
if (doc.RootElement.ValueKind == JsonValueKind.Array)
{
return ParseEntries(doc.RootElement);
}
}
catch (JsonException)
{
// NDJSON fallback.
}
var components = new List<OfflineKitComponentStatusTransport>();
foreach (var line in manifestJson.Split(['\r', '\n'], StringSplitOptions.RemoveEmptyEntries | StringSplitOptions.TrimEntries))
{
try
{
using var entryDoc = JsonDocument.Parse(line);
if (TryParseComponent(entryDoc.RootElement, out var component))
{
components.Add(component);
}
}
catch (JsonException)
{
continue;
}
}
return components;
}
private static IReadOnlyList<OfflineKitComponentStatusTransport> ParseEntries(JsonElement entries)
{
var components = new List<OfflineKitComponentStatusTransport>(entries.GetArrayLength());
foreach (var entry in entries.EnumerateArray())
{
if (TryParseComponent(entry, out var component))
{
components.Add(component);
}
}
return components;
}
private static bool TryParseComponent(JsonElement entry, out OfflineKitComponentStatusTransport component)
{
component = new OfflineKitComponentStatusTransport();
if (entry.ValueKind != JsonValueKind.Object)
{
return false;
}
if (!entry.TryGetProperty("name", out var nameProp) || nameProp.ValueKind != JsonValueKind.String)
{
return false;
}
var name = nameProp.GetString();
if (string.IsNullOrWhiteSpace(name))
{
return false;
}
string? sha = null;
if (entry.TryGetProperty("sha256", out var shaProp) && shaProp.ValueKind == JsonValueKind.String)
{
sha = NormalizeSha256(shaProp.GetString());
}
long? size = null;
if (entry.TryGetProperty("size", out var sizeProp) && sizeProp.ValueKind == JsonValueKind.Number && sizeProp.TryGetInt64(out var sizeValue))
{
size = sizeValue;
}
DateTimeOffset? capturedAt = null;
if (entry.TryGetProperty("capturedAt", out var capturedProp) && capturedProp.ValueKind == JsonValueKind.String
&& DateTimeOffset.TryParse(capturedProp.GetString(), out var parsedCaptured))
{
capturedAt = parsedCaptured.ToUniversalTime();
}
component = new OfflineKitComponentStatusTransport
{
Name = name.Trim(),
Digest = sha,
SizeBytes = size,
CapturedAt = capturedAt
};
return true;
}
}

294
src/Scanner/StellaOps.Scanner.WebService/Services/OfflineKitMetricsStore.cs Normal file
View File

@@ -0,0 +1,294 @@
using System.Collections.Concurrent;
using System.Globalization;
using System.Text;
namespace StellaOps.Scanner.WebService.Services;
internal sealed class OfflineKitMetricsStore
{
private static readonly double[] DefaultLatencyBucketsSeconds =
{
0.001,
0.0025,
0.005,
0.01,
0.025,
0.05,
0.1,
0.25,
0.5,
1,
2.5,
5,
10
};
private readonly ConcurrentDictionary<ImportCounterKey, long> _imports = new();
private readonly ConcurrentDictionary<TwoLabelKey, Histogram> _attestationVerifyLatency = new();
private readonly ConcurrentDictionary<string, Histogram> _rekorInclusionLatency = new(StringComparer.Ordinal);
private readonly ConcurrentDictionary<string, long> _rekorSuccess = new(StringComparer.Ordinal);
private readonly ConcurrentDictionary<string, long> _rekorRetry = new(StringComparer.Ordinal);
public void RecordImport(string status, string tenantId)
{
status = NormalizeLabelValue(status, "unknown");
tenantId = NormalizeLabelValue(tenantId, "unknown");
_imports.AddOrUpdate(new ImportCounterKey(tenantId, status), 1, static (_, current) => current + 1);
}
public void RecordAttestationVerifyLatency(string attestationType, double seconds, bool success)
{
attestationType = NormalizeLabelValue(attestationType, "unknown");
seconds = ClampSeconds(seconds);
var key = new TwoLabelKey(attestationType, success ? "true" : "false");
var histogram = _attestationVerifyLatency.GetOrAdd(key, _ => new Histogram(DefaultLatencyBucketsSeconds));
histogram.Record(seconds);
}
public void RecordRekorSuccess(string mode)
{
mode = NormalizeLabelValue(mode, "unknown");
_rekorSuccess.AddOrUpdate(mode, 1, static (_, current) => current + 1);
}
public void RecordRekorRetry(string reason)
{
reason = NormalizeLabelValue(reason, "unknown");
_rekorRetry.AddOrUpdate(reason, 1, static (_, current) => current + 1);
}
public void RecordRekorInclusionLatency(double seconds, bool success)
{
seconds = ClampSeconds(seconds);
var key = success ? "true" : "false";
var histogram = _rekorInclusionLatency.GetOrAdd(key, _ => new Histogram(DefaultLatencyBucketsSeconds));
histogram.Record(seconds);
}
public string RenderPrometheus()
{
var builder = new StringBuilder(capacity: 4096);
AppendCounterHeader(builder, "offlinekit_import_total", "Total number of offline kit import attempts");
foreach (var (key, value) in _imports.OrderBy(kv => kv.Key.TenantId, StringComparer.Ordinal)
.ThenBy(kv => kv.Key.Status, StringComparer.Ordinal))
{
builder.Append("offlinekit_import_total{tenant_id=\"");
builder.Append(EscapeLabelValue(key.TenantId));
builder.Append("\",status=\"");
builder.Append(EscapeLabelValue(key.Status));
builder.Append("\"} ");
builder.Append(value.ToString(CultureInfo.InvariantCulture));
builder.Append('\n');
}
AppendHistogramTwoLabels(
builder,
name: "offlinekit_attestation_verify_latency_seconds",
help: "Time taken to verify attestations during import",
labelA: "attestation_type",
labelB: "success",
histograms: _attestationVerifyLatency);
AppendCounterHeader(builder, "attestor_rekor_success_total", "Successful Rekor verification count");
foreach (var (key, value) in _rekorSuccess.OrderBy(kv => kv.Key, StringComparer.Ordinal))
{
builder.Append("attestor_rekor_success_total{mode=\"");
builder.Append(EscapeLabelValue(key));
builder.Append("\"} ");
builder.Append(value.ToString(CultureInfo.InvariantCulture));
builder.Append('\n');
}
AppendCounterHeader(builder, "attestor_rekor_retry_total", "Rekor verification retry count");
foreach (var (key, value) in _rekorRetry.OrderBy(kv => kv.Key, StringComparer.Ordinal))
{
builder.Append("attestor_rekor_retry_total{reason=\"");
builder.Append(EscapeLabelValue(key));
builder.Append("\"} ");
builder.Append(value.ToString(CultureInfo.InvariantCulture));
builder.Append('\n');
}
AppendHistogramOneLabel(
builder,
name: "rekor_inclusion_latency",
help: "Time to verify Rekor inclusion proof",
label: "success",
histograms: _rekorInclusionLatency);
return builder.ToString();
}
private static void AppendCounterHeader(StringBuilder builder, string name, string help)
{
builder.Append("# HELP ");
builder.Append(name);
builder.Append(' ');
builder.Append(help);
builder.Append('\n');
builder.Append("# TYPE ");
builder.Append(name);
builder.Append(" counter\n");
}
private static void AppendHistogramTwoLabels(
StringBuilder builder,
string name,
string help,
string labelA,
string labelB,
ConcurrentDictionary<TwoLabelKey, Histogram> histograms)
{
builder.Append("# HELP ");
builder.Append(name);
builder.Append(' ');
builder.Append(help);
builder.Append('\n');
builder.Append("# TYPE ");
builder.Append(name);
builder.Append(" histogram\n");
foreach (var grouping in histograms.OrderBy(kv => kv.Key.LabelA, StringComparer.Ordinal)
.ThenBy(kv => kv.Key.LabelB, StringComparer.Ordinal))
{
var labels = $"{labelA}=\"{EscapeLabelValue(grouping.Key.LabelA)}\",{labelB}=\"{EscapeLabelValue(grouping.Key.LabelB)}\"";
AppendHistogramSeries(builder, name, labels, grouping.Value.Snapshot());
}
}
private static void AppendHistogramOneLabel(
StringBuilder builder,
string name,
string help,
string label,
ConcurrentDictionary<string, Histogram> histograms)
{
builder.Append("# HELP ");
builder.Append(name);
builder.Append(' ');
builder.Append(help);
builder.Append('\n');
builder.Append("# TYPE ");
builder.Append(name);
builder.Append(" histogram\n");
foreach (var grouping in histograms.OrderBy(kv => kv.Key, StringComparer.Ordinal))
{
var labels = $"{label}=\"{EscapeLabelValue(grouping.Key)}\"";
AppendHistogramSeries(builder, name, labels, grouping.Value.Snapshot());
}
}
private static void AppendHistogramSeries(
StringBuilder builder,
string name,
string labels,
HistogramSnapshot snapshot)
{
long cumulative = 0;
for (var i = 0; i < snapshot.BucketUpperBounds.Length; i++)
{
cumulative += snapshot.BucketCounts[i];
builder.Append(name);
builder.Append("_bucket{");
builder.Append(labels);
builder.Append(",le=\"");
builder.Append(snapshot.BucketUpperBounds[i].ToString("G", CultureInfo.InvariantCulture));
builder.Append("\"} ");
builder.Append(cumulative.ToString(CultureInfo.InvariantCulture));
builder.Append('\n');
}
cumulative += snapshot.BucketCounts[^1];
builder.Append(name);
builder.Append("_bucket{");
builder.Append(labels);
builder.Append(",le=\"+Inf\"} ");
builder.Append(cumulative.ToString(CultureInfo.InvariantCulture));
builder.Append('\n');
builder.Append(name);
builder.Append("_sum{");
builder.Append(labels);
builder.Append("} ");
builder.Append(snapshot.SumSeconds.ToString("G", CultureInfo.InvariantCulture));
builder.Append('\n');
builder.Append(name);
builder.Append("_count{");
builder.Append(labels);
builder.Append("} ");
builder.Append(snapshot.Count.ToString(CultureInfo.InvariantCulture));
builder.Append('\n');
}
private static double ClampSeconds(double seconds)
=> double.IsNaN(seconds) || double.IsInfinity(seconds) || seconds < 0 ? 0 : seconds;
private static string NormalizeLabelValue(string? value, string fallback)
=> string.IsNullOrWhiteSpace(value) ? fallback : value.Trim();
private static string EscapeLabelValue(string value)
=> value.Replace("\\", "\\\\", StringComparison.Ordinal).Replace("\"", "\\\"", StringComparison.Ordinal);
private sealed class Histogram
{
private readonly double[] _bucketUpperBounds;
private readonly long[] _bucketCounts;
private long _count;
private double _sumSeconds;
private readonly object _lock = new();
public Histogram(double[] bucketUpperBounds)
{
_bucketUpperBounds = bucketUpperBounds ?? throw new ArgumentNullException(nameof(bucketUpperBounds));
_bucketCounts = new long[_bucketUpperBounds.Length + 1];
}
public void Record(double seconds)
{
lock (_lock)
{
_count++;
_sumSeconds += seconds;
var bucketIndex = _bucketUpperBounds.Length;
for (var i = 0; i < _bucketUpperBounds.Length; i++)
{
if (seconds <= _bucketUpperBounds[i])
{
bucketIndex = i;
break;
}
}
_bucketCounts[bucketIndex]++;
}
}
public HistogramSnapshot Snapshot()
{
lock (_lock)
{
return new HistogramSnapshot(
(double[])_bucketUpperBounds.Clone(),
(long[])_bucketCounts.Clone(),
_count,
_sumSeconds);
}
}
}
private sealed record HistogramSnapshot(
double[] BucketUpperBounds,
long[] BucketCounts,
long Count,
double SumSeconds);
private sealed record ImportCounterKey(string TenantId, string Status);
private sealed record TwoLabelKey(string LabelA, string LabelB);
}

89
src/Scanner/StellaOps.Scanner.WebService/Services/OfflineKitStateStore.cs Normal file
View File

@@ -0,0 +1,89 @@
using System.Text.Json;
using Microsoft.Extensions.Hosting;
using Microsoft.Extensions.Logging;
namespace StellaOps.Scanner.WebService.Services;
internal sealed class OfflineKitStateStore
{
private static readonly JsonSerializerOptions JsonOptions = new(JsonSerializerDefaults.Web)
{
WriteIndented = true
};
private readonly string _rootDirectory;
private readonly ILogger<OfflineKitStateStore> _logger;
public OfflineKitStateStore(IHostEnvironment environment, ILogger<OfflineKitStateStore> logger)
{
ArgumentNullException.ThrowIfNull(environment);
_logger = logger ?? throw new ArgumentNullException(nameof(logger));
_rootDirectory = Path.Combine(environment.ContentRootPath, "data", "offline-kit");
}
public string GetBundleDirectory(string tenantId, string bundleId)
{
var safeTenant = SanitizePathSegment(tenantId);
var safeBundle = SanitizePathSegment(bundleId);
return Path.Combine(_rootDirectory, "bundles", safeTenant, safeBundle);
}
public async Task SaveStatusAsync(string tenantId, OfflineKitStatusTransport status, CancellationToken cancellationToken)
{
ArgumentException.ThrowIfNullOrWhiteSpace(tenantId);
ArgumentNullException.ThrowIfNull(status);
var stateDirectory = Path.Combine(_rootDirectory, ".state");
Directory.CreateDirectory(stateDirectory);
var path = GetStatusPath(tenantId);
var temp = path + ".tmp";
await using (var stream = File.Create(temp))
{
await JsonSerializer.SerializeAsync(stream, status, JsonOptions, cancellationToken).ConfigureAwait(false);
}
File.Copy(temp, path, overwrite: true);
File.Delete(temp);
}
public async Task<OfflineKitStatusTransport?> LoadStatusAsync(string tenantId, CancellationToken cancellationToken)
{
ArgumentException.ThrowIfNullOrWhiteSpace(tenantId);
var path = GetStatusPath(tenantId);
if (!File.Exists(path))
{
return null;
}
try
{
await using var stream = File.OpenRead(path);
return await JsonSerializer.DeserializeAsync<OfflineKitStatusTransport>(stream, JsonOptions, cancellationToken).ConfigureAwait(false);
}
catch (Exception ex) when (ex is IOException or JsonException)
{
_logger.LogWarning(ex, "Failed to read offline kit state from {Path}", path);
return null;
}
}
private string GetStatusPath(string tenantId)
{
var safeTenant = SanitizePathSegment(tenantId);
return Path.Combine(_rootDirectory, ".state", $"offline-kit-active__{safeTenant}.json");
}
private static string SanitizePathSegment(string value)
{
var trimmed = value.Trim().ToLowerInvariant();
var invalid = Path.GetInvalidFileNameChars();
var chars = trimmed
.Select(c => invalid.Contains(c) || c == '/' || c == '\\' || char.IsWhiteSpace(c) ? '_' : c)
.ToArray();
return new string(chars);
}
}

192
src/Scanner/StellaOps.Scanner.WebService/Services/SbomIngestionService.cs Normal file
View File

@@ -0,0 +1,192 @@
using System.Text.Json;
using Microsoft.Extensions.Logging;
using StellaOps.Scanner.Storage.Catalog;
using StellaOps.Scanner.Storage.Services;
using StellaOps.Scanner.WebService.Contracts;
using StellaOps.Scanner.WebService.Domain;
namespace StellaOps.Scanner.WebService.Services;
internal sealed class SbomIngestionService : ISbomIngestionService
{
private static readonly JsonSerializerOptions JsonOptions = new(JsonSerializerDefaults.Web)
{
WriteIndented = false
};
private readonly ArtifactStorageService _artifactStorage;
private readonly ILogger<SbomIngestionService> _logger;
public SbomIngestionService(ArtifactStorageService artifactStorage, ILogger<SbomIngestionService> logger)
{
_artifactStorage = artifactStorage ?? throw new ArgumentNullException(nameof(artifactStorage));
_logger = logger ?? throw new ArgumentNullException(nameof(logger));
}
public string? DetectFormat(JsonDocument sbomDocument)
{
ArgumentNullException.ThrowIfNull(sbomDocument);
if (sbomDocument.RootElement.ValueKind != JsonValueKind.Object)
{
return null;
}
var root = sbomDocument.RootElement;
if (root.TryGetProperty("bomFormat", out var bomFormat)
&& bomFormat.ValueKind == JsonValueKind.String
&& string.Equals(bomFormat.GetString(), "CycloneDX", StringComparison.OrdinalIgnoreCase))
{
return SbomFormats.CycloneDx;
}
if (root.TryGetProperty("spdxVersion", out var spdxVersion)
&& spdxVersion.ValueKind == JsonValueKind.String
&& !string.IsNullOrWhiteSpace(spdxVersion.GetString()))
{
return SbomFormats.Spdx;
}
return null;
}
public SbomValidationResult Validate(JsonDocument sbomDocument, string format)
{
ArgumentNullException.ThrowIfNull(sbomDocument);
ArgumentException.ThrowIfNullOrWhiteSpace(format);
if (sbomDocument.RootElement.ValueKind != JsonValueKind.Object)
{
return SbomValidationResult.Failure("SBOM root must be a JSON object.");
}
var root = sbomDocument.RootElement;
if (string.Equals(format, SbomFormats.CycloneDx, StringComparison.OrdinalIgnoreCase))
{
if (!root.TryGetProperty("bomFormat", out var bomFormat)
|| bomFormat.ValueKind != JsonValueKind.String
|| !string.Equals(bomFormat.GetString(), "CycloneDX", StringComparison.OrdinalIgnoreCase))
{
return SbomValidationResult.Failure("CycloneDX SBOM must include bomFormat == 'CycloneDX'.");
}
return SbomValidationResult.Success();
}
if (string.Equals(format, SbomFormats.Spdx, StringComparison.OrdinalIgnoreCase))
{
if (!root.TryGetProperty("spdxVersion", out var spdxVersion)
|| spdxVersion.ValueKind != JsonValueKind.String
|| string.IsNullOrWhiteSpace(spdxVersion.GetString()))
{
return SbomValidationResult.Failure("SPDX SBOM must include spdxVersion.");
}
return SbomValidationResult.Success();
}
return SbomValidationResult.Failure($"Unsupported SBOM format '{format}'.");
}
public async Task<SbomIngestionResult> IngestAsync(
ScanId scanId,
JsonDocument sbomDocument,
string format,
string? contentDigest,
CancellationToken cancellationToken = default)
{
ArgumentNullException.ThrowIfNull(sbomDocument);
ArgumentException.ThrowIfNullOrWhiteSpace(scanId.Value);
ArgumentException.ThrowIfNullOrWhiteSpace(format);
var (documentFormat, mediaType) = ResolveStorageFormat(format);
var bytes = JsonSerializer.SerializeToUtf8Bytes(sbomDocument.RootElement, JsonOptions);
await using var stream = new MemoryStream(bytes, writable: false);
var stored = await _artifactStorage.StoreArtifactAsync(
ArtifactDocumentType.ImageBom,
documentFormat,
mediaType,
stream,
immutable: true,
ttlClass: "default",
expiresAtUtc: null,
cancellationToken: cancellationToken)
.ConfigureAwait(false);
if (!string.IsNullOrWhiteSpace(contentDigest)
&& !string.Equals(contentDigest.Trim(), stored.BytesSha256, StringComparison.OrdinalIgnoreCase))
{
_logger.LogDebug(
"SBOM Content-Digest header did not match stored digest header={HeaderDigest} stored={StoredDigest}",
contentDigest.Trim(),
stored.BytesSha256);
}
var componentCount = CountComponents(sbomDocument, format);
_logger.LogInformation(
"Ingested sbom scan={ScanId} format={Format} components={Components} digest={Digest} id={SbomId}",
scanId.Value,
format,
componentCount,
stored.BytesSha256,
stored.Id);
return new SbomIngestionResult(
SbomId: stored.Id,
Format: format,
ComponentCount: componentCount,
Digest: stored.BytesSha256);
}
private static (ArtifactDocumentFormat Format, string MediaType) ResolveStorageFormat(string format)
{
if (string.Equals(format, SbomFormats.CycloneDx, StringComparison.OrdinalIgnoreCase))
{
return (ArtifactDocumentFormat.CycloneDxJson, "application/vnd.cyclonedx+json");
}
if (string.Equals(format, SbomFormats.Spdx, StringComparison.OrdinalIgnoreCase))
{
return (ArtifactDocumentFormat.SpdxJson, "application/spdx+json");
}
return (ArtifactDocumentFormat.CycloneDxJson, "application/json");
}
private static int CountComponents(JsonDocument document, string format)
{
if (document.RootElement.ValueKind != JsonValueKind.Object)
{
return 0;
}
var root = document.RootElement;
if (string.Equals(format, SbomFormats.CycloneDx, StringComparison.OrdinalIgnoreCase))
{
if (root.TryGetProperty("components", out var components) && components.ValueKind == JsonValueKind.Array)
{
return components.GetArrayLength();
}
return 0;
}
if (string.Equals(format, SbomFormats.Spdx, StringComparison.OrdinalIgnoreCase))
{
if (root.TryGetProperty("packages", out var packages) && packages.ValueKind == JsonValueKind.Array)
{
return packages.GetArrayLength();
}
return 0;
}
return 0;
}
}

1
src/Scanner/StellaOps.Scanner.WebService/Services/ScoreReplayService.cs
View File

@@ -175,6 +175,7 @@ public interface IScanManifestRepository
{
Task<SignedScanManifest?> GetManifestAsync(string scanId, string? manifestHash = null, CancellationToken cancellationToken = default);
Task SaveManifestAsync(SignedScanManifest manifest, CancellationToken cancellationToken = default);
Task<List<string>> FindAffectedScansAsync(AffectedScansQuery query, CancellationToken cancellationToken = default);
}
/// <summary>

6
src/Scanner/StellaOps.Scanner.WebService/StellaOps.Scanner.WebService.csproj
View File

@@ -20,9 +20,11 @@
<ProjectReference Include="../../__Libraries/StellaOps.Configuration/StellaOps.Configuration.csproj" />
<ProjectReference Include="../../__Libraries/StellaOps.DependencyInjection/StellaOps.DependencyInjection.csproj" />
<ProjectReference Include="../../__Libraries/StellaOps.Plugin/StellaOps.Plugin.csproj" />
<ProjectReference Include="../../Authority/__Libraries/StellaOps.Authority.Storage.Postgres/StellaOps.Authority.Storage.Postgres.csproj" />
<ProjectReference Include="../../Authority/StellaOps.Authority/StellaOps.Auth.Abstractions/StellaOps.Auth.Abstractions.csproj" />
<ProjectReference Include="../../Authority/StellaOps.Authority/StellaOps.Auth.Client/StellaOps.Auth.Client.csproj" />
<ProjectReference Include="../../Authority/StellaOps.Authority/StellaOps.Auth.ServerIntegration/StellaOps.Auth.ServerIntegration.csproj" />
<ProjectReference Include="../../AirGap/StellaOps.AirGap.Importer/StellaOps.AirGap.Importer.csproj" />
<ProjectReference Include="../../Policy/__Libraries/StellaOps.Policy/StellaOps.Policy.csproj" />
<ProjectReference Include="../../__Libraries/StellaOps.Cryptography/StellaOps.Cryptography.csproj" />
<ProjectReference Include="../../__Libraries/StellaOps.Cryptography.DependencyInjection/StellaOps.Cryptography.DependencyInjection.csproj" />
@@ -43,4 +45,8 @@
<ProjectReference Include="../../Concelier/__Libraries/StellaOps.Concelier.Connector.Common/StellaOps.Concelier.Connector.Common.csproj" />
<ProjectReference Include="../../__Libraries/StellaOps.Messaging/StellaOps.Messaging.csproj" />
</ItemGroup>
<ItemGroup>
<Compile Remove="Endpoints\\UnknownsEndpoints.cs" />
</ItemGroup>
</Project>

1
src/Scanner/StellaOps.Scanner.WebService/TASKS.md
View File

@@ -5,3 +5,4 @@
| `SCAN-API-3101-001` | `docs/implplan/SPRINT_3101_0001_0001_scanner_api_standardization.md` | DOING | Align Scanner OpenAPI spec with current endpoints and include ProofSpine routes; compose into `src/Api/StellaOps.Api.OpenApi/stella.yaml`. |
| `PROOFSPINE-3100-API` | `docs/implplan/SPRINT_3100_0001_0001_proof_spine_system.md` | DOING | Implement and test `/api/v1/spines/*` endpoints and wire verification output. |
| `SCAN-AIRGAP-0340-001` | `docs/implplan/SPRINT_0340_0001_0001_scanner_offline_config.md` | BLOCKED | Offline kit verification wiring is blocked on an import pipeline + offline Rekor verifier. |
| `SCAN-API-3103-001` | `docs/implplan/SPRINT_3103_0001_0001_scanner_api_ingestion_completion.md` | DOING | Implement missing ingestion services + DI for callgraph/SBOM endpoints and add deterministic integration tests. |

2
src/Scanner/StellaOps.Scanner.Worker/Options/ScannerWorkerOptions.cs
View File

@@ -12,6 +12,8 @@ public sealed class ScannerWorkerOptions
{
public const string SectionName = "Scanner:Worker";
public string? ScannerVersion { get; set; }
public int MaxConcurrentJobs { get; set; } = 2;
public QueueOptions Queue { get; } = new();

9
src/Scanner/StellaOps.Scanner.Worker/Program.cs
View File

@@ -9,6 +9,7 @@ using StellaOps.Auth.Client;
using StellaOps.Configuration;
using StellaOps.Scanner.Cache;
using StellaOps.Scanner.Reachability;
using StellaOps.Scanner.Reachability.Gates;
using StellaOps.Scanner.Analyzers.OS.Plugin;
using StellaOps.Scanner.Analyzers.Lang.Plugin;
using StellaOps.Scanner.EntryTrace;
@@ -29,6 +30,7 @@ using StellaOps.Scanner.Storage.Extensions;
using StellaOps.Scanner.Storage;
using StellaOps.Scanner.Storage.Services;
using Reachability = StellaOps.Scanner.Worker.Processing.Reachability;
using GateDetectors = StellaOps.Scanner.Reachability.Gates.Detectors;
var builder = Host.CreateApplicationBuilder(args);
@@ -90,6 +92,13 @@ builder.Services.AddSingleton<ReachabilityUnionPublisher>();
builder.Services.AddSingleton<IReachabilityUnionPublisherService, ReachabilityUnionPublisherService>();
builder.Services.AddSingleton<RichGraphWriter>();
builder.Services.AddSingleton<IRichGraphPublisher, ReachabilityRichGraphPublisher>();
builder.Services.AddSingleton<GateDetectors.ICodeContentProvider, GateDetectors.FileSystemCodeContentProvider>();
builder.Services.AddSingleton<GateDetectors.IGateDetector, GateDetectors.AuthGateDetector>();
builder.Services.AddSingleton<GateDetectors.IGateDetector, GateDetectors.FeatureFlagDetector>();
builder.Services.AddSingleton<GateDetectors.IGateDetector, GateDetectors.AdminOnlyDetector>();
builder.Services.AddSingleton<GateDetectors.IGateDetector, GateDetectors.NonDefaultConfigDetector>();
builder.Services.AddSingleton<GateMultiplierCalculator>();
builder.Services.AddSingleton<IRichGraphGateAnnotator, RichGraphGateAnnotator>();
builder.Services.AddSingleton<IRichGraphPublisherService, ReachabilityRichGraphPublisherService>();
builder.Services.AddSingleton<IScanStageExecutor, StellaOps.Scanner.Worker.Processing.Replay.ReplaySealedBundleStageExecutor>();
builder.Services.AddSingleton<StellaOps.Scanner.Worker.Processing.Replay.ReplayBundleFetcher>();

52
src/Scanner/__Libraries/StellaOps.Scanner.CallGraph/Extraction/DotNet/DotNetCallGraphExtractor.cs
View File

@@ -40,8 +40,8 @@ public sealed class DotNetCallGraphExtractor : ICallGraphExtractor
workspace.WorkspaceFailed += (_, _) => { };
var solution = resolvedTarget.EndsWith(".sln", StringComparison.OrdinalIgnoreCase)
? await workspace.OpenSolutionAsync(resolvedTarget, cancellationToken).ConfigureAwait(false)
: (await workspace.OpenProjectAsync(resolvedTarget, cancellationToken).ConfigureAwait(false)).Solution;
? await workspace.OpenSolutionAsync(resolvedTarget, cancellationToken: cancellationToken).ConfigureAwait(false)
: (await workspace.OpenProjectAsync(resolvedTarget, cancellationToken: cancellationToken).ConfigureAwait(false)).Solution;
var nodesById = new Dictionary<string, CallGraphNode>(StringComparer.Ordinal);
var edges = new HashSet<CallGraphEdge>(CallGraphEdgeComparer.Instance);
@@ -203,18 +203,20 @@ public sealed class DotNetCallGraphExtractor : ICallGraphExtractor
var (file, line) = GetSourceLocation(analysisRoot, syntax.GetLocation());
var (isEntrypoint, entryType) = EntrypointClassifier.IsEntrypoint(method);
var symbol = FormatSymbol(method);
var sink = SinkRegistry.MatchSink("dotnet", symbol);
return new CallGraphNode(
NodeId: id,
Symbol: method.ToDisplayString(SymbolDisplayFormat.FullyQualifiedFormat),
Symbol: symbol,
File: file,
Line: line,
Package: method.ContainingAssembly?.Name ?? "unknown",
Visibility: MapVisibility(method.DeclaredAccessibility),
IsEntrypoint: isEntrypoint,
EntrypointType: entryType,
IsSink: false,
SinkCategory: null);
IsSink: sink is not null,
SinkCategory: sink?.Category);
}
private static CallGraphNode CreateInvokedNode(string analysisRoot, IMethodSymbol method)
@@ -223,11 +225,12 @@ public sealed class DotNetCallGraphExtractor : ICallGraphExtractor
var definitionLocation = method.Locations.FirstOrDefault(l => l.IsInSource) ?? Location.None;
var (file, line) = GetSourceLocation(analysisRoot, definitionLocation);
var sink = SinkRegistry.MatchSink("dotnet", method.ToDisplayString(SymbolDisplayFormat.FullyQualifiedFormat));
var symbol = FormatSymbol(method);
var sink = SinkRegistry.MatchSink("dotnet", symbol);
return new CallGraphNode(
NodeId: id,
Symbol: method.ToDisplayString(SymbolDisplayFormat.FullyQualifiedFormat),
Symbol: symbol,
File: file,
Line: line,
Package: method.ContainingAssembly?.Name ?? "unknown",
@@ -303,6 +306,41 @@ public sealed class DotNetCallGraphExtractor : ICallGraphExtractor
return $"dotnet:{method.ContainingAssembly?.Name}:{method.ToDisplayString(SymbolDisplayFormat.FullyQualifiedFormat)}";
}
private static string FormatSymbol(IMethodSymbol method)
{
var namespaceName = method.ContainingNamespace is { IsGlobalNamespace: false }
? method.ContainingNamespace.ToDisplayString()
: string.Empty;
var typeName = method.ContainingType is null
? string.Empty
: string.Join('.', GetContainingTypeNames(method.ContainingType));
if (string.IsNullOrWhiteSpace(namespaceName))
{
return string.IsNullOrWhiteSpace(typeName)
? method.Name
: $"{typeName}.{method.Name}";
}
return string.IsNullOrWhiteSpace(typeName)
? $"{namespaceName}.{method.Name}"
: $"{namespaceName}.{typeName}.{method.Name}";
}
private static IEnumerable<string> GetContainingTypeNames(INamedTypeSymbol type)
{
var stack = new Stack<string>();
var current = type;
while (current is not null)
{
stack.Push(current.Name);
current = current.ContainingType;
}
return stack;
}
private sealed class CallGraphEdgeComparer : IEqualityComparer<CallGraphEdge>
{
public static readonly CallGraphEdgeComparer Instance = new();

2
src/Scanner/__Libraries/StellaOps.Scanner.Reachability/Gates/Detectors/AdminOnlyDetector.cs
View File

@@ -122,7 +122,9 @@ public sealed class AdminOnlyDetector : IGateDetector
language.ToLowerInvariant() switch
{
"c#" or "cs" => "csharp",
"dotnet" or ".net" => "csharp",
"js" => "javascript",
"node" or "nodejs" => "javascript",
"ts" => "typescript",
"py" => "python",
"rb" => "ruby",

2
src/Scanner/__Libraries/StellaOps.Scanner.Reachability/Gates/Detectors/AuthGateDetector.cs
View File

@@ -95,7 +95,9 @@ public sealed class AuthGateDetector : IGateDetector
language.ToLowerInvariant() switch
{
"c#" or "cs" => "csharp",
"dotnet" or ".net" => "csharp",
"js" => "javascript",
"node" or "nodejs" => "javascript",
"ts" => "typescript",
"py" => "python",
"rb" => "ruby",

2
src/Scanner/__Libraries/StellaOps.Scanner.Reachability/Gates/Detectors/FeatureFlagDetector.cs
View File

@@ -107,7 +107,9 @@ public sealed class FeatureFlagDetector : IGateDetector
language.ToLowerInvariant() switch
{
"c#" or "cs" => "csharp",
"dotnet" or ".net" => "csharp",
"js" => "javascript",
"node" or "nodejs" => "javascript",
"ts" => "typescript",
"py" => "python",
"rb" => "ruby",

84
src/Scanner/__Libraries/StellaOps.Scanner.Reachability/Gates/Detectors/FileSystemCodeContentProvider.cs Normal file
View File

@@ -0,0 +1,84 @@
using System;
using System.Collections.Generic;
using System.IO;
using System.Threading;
using System.Threading.Tasks;
namespace StellaOps.Scanner.Reachability.Gates.Detectors;
/// <summary>
/// Reads source code directly from the local filesystem.
/// </summary>
public sealed class FileSystemCodeContentProvider : ICodeContentProvider
{
public Task<string?> GetContentAsync(string filePath, CancellationToken ct = default)
{
if (string.IsNullOrWhiteSpace(filePath))
{
return Task.FromResult<string?>(null);
}
var path = filePath.Trim();
if (!File.Exists(path))
{
return Task.FromResult<string?>(null);
}
return File.ReadAllTextAsync(path, ct);
}
public async Task<IReadOnlyList<string>?> GetLinesAsync(
string filePath,
int startLine,
int endLine,
CancellationToken ct = default)
{
if (string.IsNullOrWhiteSpace(filePath))
{
return null;
}
if (startLine <= 0 || endLine <= 0 || endLine < startLine)
{
return null;
}
var path = filePath.Trim();
if (!File.Exists(path))
{
return null;
}
var lines = new List<string>(Math.Min(256, endLine - startLine + 1));
var currentLine = 0;
await using var stream = File.OpenRead(path);
using var reader = new StreamReader(stream);
while (true)
{
ct.ThrowIfCancellationRequested();
var line = await reader.ReadLineAsync().ConfigureAwait(false);
if (line is null)
{
break;
}
currentLine++;
if (currentLine < startLine)
{
continue;
}
if (currentLine > endLine)
{
break;
}
lines.Add(line);
}
return lines;
}
}

2
src/Scanner/__Libraries/StellaOps.Scanner.Reachability/Gates/Detectors/NonDefaultConfigDetector.cs
View File

@@ -135,7 +135,9 @@ public sealed class NonDefaultConfigDetector : IGateDetector
language.ToLowerInvariant() switch
{
"c#" or "cs" => "csharp",
"dotnet" or ".net" => "csharp",
"js" => "javascript",
"node" or "nodejs" => "javascript",
"ts" => "typescript",
"py" => "python",
"rb" => "ruby",

46
src/Scanner/__Libraries/StellaOps.Scanner.Reachability/Gates/GateMultiplierCalculator.cs
View File

@@ -26,35 +26,22 @@ public sealed class GateMultiplierCalculator
if (gates.Count == 0)
return 10000; // 100% - no reduction
// Group gates by type and take highest confidence per type
var gatesByType = gates
.GroupBy(g => g.Type)
.Select(g => new
{
Type = g.Key,
MaxConfidence = g.Max(x => x.Confidence)
})
var gateTypes = gates
.Select(g => g.Type)
.Distinct()
.OrderBy(t => t)
.ToList();
// Calculate compound multiplier using product reduction
// Each gate multiplier is confidence-weighted
double multiplier = 1.0;
foreach (var gate in gatesByType)
// Multiply per-type multipliers; gate instances of the same type do not stack.
double multiplierBps = 10000.0;
foreach (var gateType in gateTypes)
{
var baseMultiplierBps = _config.GetMultiplierBps(gate.Type);
// Scale multiplier by confidence
// Low confidence = less reduction, high confidence = more reduction
var effectiveMultiplierBps = InterpolateMultiplier(
baseMultiplierBps,
10000, // No reduction at 0 confidence
gate.MaxConfidence);
multiplier *= effectiveMultiplierBps / 10000.0;
var typeMultiplierBps = _config.GetMultiplierBps(gateType);
multiplierBps = multiplierBps * typeMultiplierBps / 10000.0;
}
// Apply floor
var result = (int)(multiplier * 10000);
var result = (int)Math.Round(multiplierBps);
result = Math.Clamp(result, 0, _config.MaxMultipliersBps);
return Math.Max(result, _config.MinimumMultiplierBps);
}
@@ -65,8 +52,7 @@ public sealed class GateMultiplierCalculator
/// <returns>Multiplier in basis points (10000 = 100%).</returns>
public int CalculateSingleMultiplierBps(DetectedGate gate)
{
var baseMultiplierBps = _config.GetMultiplierBps(gate.Type);
return InterpolateMultiplier(baseMultiplierBps, 10000, gate.Confidence);
return _config.GetMultiplierBps(gate.Type);
}
/// <summary>
@@ -93,14 +79,6 @@ public sealed class GateMultiplierCalculator
{
return baseScore * multiplierBps / 10000.0;
}
private static int InterpolateMultiplier(int minBps, int maxBps, double confidence)
{
// Linear interpolation: higher confidence = lower multiplier (closer to minBps)
var range = maxBps - minBps;
var reduction = (int)(range * confidence);
return maxBps - reduction;
}
}
/// <summary>

357
src/Scanner/__Libraries/StellaOps.Scanner.Reachability/Gates/RichGraphGateAnnotator.cs Normal file
View File

@@ -0,0 +1,357 @@
using System.Collections.Generic;
using System.Linq;
using System.Text.Json;
using Microsoft.Extensions.Logging;
using GateDetectors = StellaOps.Scanner.Reachability.Gates.Detectors;
namespace StellaOps.Scanner.Reachability.Gates;
public interface IRichGraphGateAnnotator
{
Task<RichGraph> AnnotateAsync(RichGraph graph, CancellationToken cancellationToken = default);
}
/// <summary>
/// Enriches richgraph-v1 edges with detected gates and a combined gate multiplier.
/// </summary>
public sealed class RichGraphGateAnnotator : IRichGraphGateAnnotator
{
private readonly IReadOnlyList<GateDetectors.IGateDetector> _detectors;
private readonly GateDetectors.ICodeContentProvider _codeProvider;
private readonly GateMultiplierCalculator _multiplierCalculator;
private readonly ILogger<RichGraphGateAnnotator> _logger;
public RichGraphGateAnnotator(
IEnumerable<GateDetectors.IGateDetector> detectors,
GateDetectors.ICodeContentProvider codeProvider,
GateMultiplierCalculator multiplierCalculator,
ILogger<RichGraphGateAnnotator> logger)
{
_detectors = (detectors ?? Enumerable.Empty<GateDetectors.IGateDetector>())
.Where(d => d is not null)
.OrderBy(d => d.GateType)
.ToList();
_codeProvider = codeProvider ?? throw new ArgumentNullException(nameof(codeProvider));
_multiplierCalculator = multiplierCalculator ?? throw new ArgumentNullException(nameof(multiplierCalculator));
_logger = logger ?? throw new ArgumentNullException(nameof(logger));
}
public async Task<RichGraph> AnnotateAsync(RichGraph graph, CancellationToken cancellationToken = default)
{
ArgumentNullException.ThrowIfNull(graph);
if (_detectors.Count == 0)
{
return graph;
}
var trimmed = graph.Trimmed();
var incomingByNode = trimmed.Edges
.GroupBy(e => e.To, StringComparer.Ordinal)
.ToDictionary(
g => g.Key,
g => (IReadOnlyList<RichGraphEdge>)g.ToList(),
StringComparer.Ordinal);
var gatesByNode = new Dictionary<string, IReadOnlyList<DetectedGate>>(StringComparer.Ordinal);
foreach (var node in trimmed.Nodes)
{
cancellationToken.ThrowIfCancellationRequested();
var (sourceFile, lineNumber, endLineNumber) = ExtractSourceLocation(node);
var annotations = ExtractAnnotations(node.Attributes);
var detectorNode = new GateDetectors.RichGraphNode
{
Symbol = node.SymbolId,
SourceFile = sourceFile,
LineNumber = lineNumber,
EndLineNumber = endLineNumber,
Annotations = annotations,
Metadata = node.Attributes
};
var incomingEdges = incomingByNode.TryGetValue(node.Id, out var edges)
? edges.Select(e => new GateDetectors.RichGraphEdge
{
FromSymbol = e.From,
ToSymbol = e.To,
EdgeType = e.Kind,
Gates = []
})
.ToList()
: [];
var detected = new List<DetectedGate>();
foreach (var detector in _detectors)
{
try
{
var results = await detector.DetectAsync(
detectorNode,
incomingEdges,
_codeProvider,
node.Lang,
cancellationToken).ConfigureAwait(false);
if (results is { Count: > 0 })
{
detected.AddRange(results);
}
}
catch (Exception ex)
{
_logger.LogDebug(ex, "Gate detector {Detector} failed for node {NodeId}.", detector.GateType, node.Id);
}
}
gatesByNode[node.Id] = CanonicalizeGates(detected);
}
var annotatedEdges = new List<RichGraphEdge>(trimmed.Edges.Count);
foreach (var edge in trimmed.Edges)
{
cancellationToken.ThrowIfCancellationRequested();
gatesByNode.TryGetValue(edge.From, out var fromGates);
gatesByNode.TryGetValue(edge.To, out var toGates);
var combined = CombineGates(fromGates, toGates);
if (combined.Count == 0 && edge.GateMultiplierBps == 10000 && edge.Gates is not { Count: > 0 })
{
annotatedEdges.Add(edge);
continue;
}
var multiplier = combined.Count == 0
? edge.GateMultiplierBps
: _multiplierCalculator.CalculateCombinedMultiplierBps(combined);
annotatedEdges.Add(edge with
{
Gates = combined,
GateMultiplierBps = multiplier
});
}
return (trimmed with { Edges = annotatedEdges }).Trimmed();
}
private static IReadOnlyList<DetectedGate> CombineGates(
IReadOnlyList<DetectedGate>? fromGates,
IReadOnlyList<DetectedGate>? toGates)
{
if (fromGates is not { Count: > 0 } && toGates is not { Count: > 0 })
{
return [];
}
var combined = new List<DetectedGate>((fromGates?.Count ?? 0) + (toGates?.Count ?? 0));
if (fromGates is { Count: > 0 })
{
combined.AddRange(fromGates);
}
if (toGates is { Count: > 0 })
{
combined.AddRange(toGates);
}
return CanonicalizeGates(combined);
}
private static IReadOnlyList<DetectedGate> CanonicalizeGates(IEnumerable<DetectedGate>? gates)
{
if (gates is null)
{
return [];
}
return gates
.Where(g => g is not null && !string.IsNullOrWhiteSpace(g.GuardSymbol))
.Select(g => g with
{
Detail = g.Detail.Trim(),
GuardSymbol = g.GuardSymbol.Trim(),
SourceFile = string.IsNullOrWhiteSpace(g.SourceFile) ? null : g.SourceFile.Trim(),
Confidence = Math.Clamp(g.Confidence, 0.0, 1.0),
DetectionMethod = g.DetectionMethod.Trim()
})
.GroupBy(g => (g.Type, g.GuardSymbol))
.Select(group => group
.OrderByDescending(g => g.Confidence)
.ThenBy(g => g.Detail, StringComparer.Ordinal)
.ThenBy(g => g.DetectionMethod, StringComparer.Ordinal)
.First())
.OrderBy(g => g.Type)
.ThenBy(g => g.GuardSymbol, StringComparer.Ordinal)
.ThenBy(g => g.Detail, StringComparer.Ordinal)
.ToList();
}
private static IReadOnlyList<string>? ExtractAnnotations(IReadOnlyDictionary<string, string>? attributes)
{
if (attributes is null || attributes.Count == 0)
{
return null;
}
var annotations = new List<string>();
AddDelimited(annotations, TryGet(attributes, "annotations"));
AddDelimited(annotations, TryGet(attributes, "annotation"));
AddDelimited(annotations, TryGet(attributes, "decorators"));
AddDelimited(annotations, TryGet(attributes, "decorator"));
foreach (var kv in attributes)
{
if (string.IsNullOrWhiteSpace(kv.Key))
{
continue;
}
if (kv.Key.StartsWith("annotation:", StringComparison.OrdinalIgnoreCase) ||
kv.Key.StartsWith("decorator:", StringComparison.OrdinalIgnoreCase))
{
var suffix = kv.Key[(kv.Key.IndexOf(':') + 1)..].Trim();
if (!string.IsNullOrWhiteSpace(suffix))
{
annotations.Add(suffix);
}
AddDelimited(annotations, kv.Value);
}
}
var normalized = annotations
.Where(a => !string.IsNullOrWhiteSpace(a))
.Select(a => a.Trim())
.Distinct(StringComparer.Ordinal)
.OrderBy(a => a, StringComparer.Ordinal)
.ToList();
return normalized.Count == 0 ? null : normalized;
}
private static void AddDelimited(List<string> sink, string? value)
{
if (string.IsNullOrWhiteSpace(value))
{
return;
}
var trimmed = value.Trim();
if (trimmed.StartsWith("[", StringComparison.Ordinal))
{
try
{
using var doc = JsonDocument.Parse(trimmed);
if (doc.RootElement.ValueKind == JsonValueKind.Array)
{
foreach (var item in doc.RootElement.EnumerateArray())
{
if (item.ValueKind == JsonValueKind.String)
{
sink.Add(item.GetString() ?? string.Empty);
}
}
return;
}
}
catch (JsonException)
{
}
}
foreach (var part in trimmed.Split(new[] { '\r', '\n', ';' }, StringSplitOptions.RemoveEmptyEntries))
{
sink.Add(part);
}
}
private static (string? SourceFile, int? LineNumber, int? EndLineNumber) ExtractSourceLocation(RichGraphNode node)
{
var attributes = node.Attributes;
var sourceFile = TryGet(attributes, "source_file")
?? TryGet(attributes, "sourceFile")
?? TryGet(attributes, "file");
var line = TryGetInt(attributes, "line_number")
?? TryGetInt(attributes, "lineNumber")
?? TryGetInt(attributes, "line");
var endLine = TryGetInt(attributes, "end_line_number")
?? TryGetInt(attributes, "endLineNumber")
?? TryGetInt(attributes, "end_line")
?? TryGetInt(attributes, "endLine");
if (!string.IsNullOrWhiteSpace(sourceFile))
{
return (sourceFile.Trim(), line, endLine);
}
if (node.Evidence is { Count: > 0 })
{
foreach (var evidence in node.Evidence)
{
if (TryParseFileEvidence(evidence, out var file, out var parsedLine))
{
return (file, parsedLine, endLine);
}
}
}
return (null, line, endLine);
}
private static bool TryParseFileEvidence(string? evidence, out string filePath, out int? lineNumber)
{
filePath = string.Empty;
lineNumber = null;
if (string.IsNullOrWhiteSpace(evidence))
{
return false;
}
var trimmed = evidence.Trim();
if (!trimmed.StartsWith("file:", StringComparison.OrdinalIgnoreCase))
{
return false;
}
var remainder = trimmed["file:".Length..];
if (string.IsNullOrWhiteSpace(remainder))
{
return false;
}
var lastColon = remainder.LastIndexOf(':');
if (lastColon > 0)
{
var maybeLine = remainder[(lastColon + 1)..];
if (int.TryParse(maybeLine, out var parsed))
{
filePath = remainder[..lastColon];
lineNumber = parsed;
return true;
}
}
filePath = remainder;
return true;
}
private static string? TryGet(IReadOnlyDictionary<string, string>? dict, string key)
=> dict is not null && dict.TryGetValue(key, out var value) ? value : null;
private static int? TryGetInt(IReadOnlyDictionary<string, string>? dict, string key)
{
if (dict is null || !dict.TryGetValue(key, out var value))
{
return null;
}
return int.TryParse(value, out var parsed) ? parsed : null;
}
}

15
src/Scanner/__Libraries/StellaOps.Scanner.Reachability/ReachabilityRichGraphPublisherService.cs
View File

@@ -4,6 +4,7 @@ using System.Threading;
using System.Threading.Tasks;
using StellaOps.Scanner.Cache.Abstractions;
using StellaOps.Scanner.Surface.Env;
using StellaOps.Scanner.Reachability.Gates;
namespace StellaOps.Scanner.Reachability;
@@ -20,22 +21,30 @@ public sealed class ReachabilityRichGraphPublisherService : IRichGraphPublisherS
private readonly ISurfaceEnvironment _environment;
private readonly IFileContentAddressableStore _cas;
private readonly IRichGraphPublisher _publisher;
private readonly IRichGraphGateAnnotator? _gateAnnotator;
public ReachabilityRichGraphPublisherService(
ISurfaceEnvironment environment,
IFileContentAddressableStore cas,
IRichGraphPublisher publisher)
IRichGraphPublisher publisher,
IRichGraphGateAnnotator? gateAnnotator = null)
{
_environment = environment ?? throw new ArgumentNullException(nameof(environment));
_cas = cas ?? throw new ArgumentNullException(nameof(cas));
_publisher = publisher ?? throw new ArgumentNullException(nameof(publisher));
_gateAnnotator = gateAnnotator;
}
public Task<RichGraphPublishResult> PublishAsync(ReachabilityUnionGraph graph, string analysisId, CancellationToken cancellationToken = default)
public async Task<RichGraphPublishResult> PublishAsync(ReachabilityUnionGraph graph, string analysisId, CancellationToken cancellationToken = default)
{
var richGraph = RichGraphBuilder.FromUnion(graph, "scanner.reachability", "0.1.0");
if (_gateAnnotator is not null)
{
richGraph = await _gateAnnotator.AnnotateAsync(richGraph, cancellationToken).ConfigureAwait(false);
}
var workRoot = Path.Combine(_environment.Settings.CacheRoot.FullName, "reachability");
Directory.CreateDirectory(workRoot);
return _publisher.PublishAsync(richGraph, analysisId, _cas, workRoot, cancellationToken);
return await _publisher.PublishAsync(richGraph, analysisId, _cas, workRoot, cancellationToken).ConfigureAwait(false);
}
}

31
src/Scanner/__Libraries/StellaOps.Scanner.Reachability/RichGraph.cs
View File

@@ -3,6 +3,7 @@ using System.Collections.Generic;
using System.Collections.Immutable;
using System.Linq;
using System.Security.Cryptography;
using StellaOps.Scanner.Reachability.Gates;
namespace StellaOps.Scanner.Reachability;
@@ -90,10 +91,34 @@ public sealed record RichGraphEdge(
string? SymbolDigest,
IReadOnlyList<string>? Evidence,
double Confidence,
IReadOnlyList<string>? Candidates)
IReadOnlyList<string>? Candidates,
IReadOnlyList<DetectedGate>? Gates = null,
int GateMultiplierBps = 10000)
{
public RichGraphEdge Trimmed()
{
var gates = (Gates ?? Array.Empty<DetectedGate>())
.Where(g => g is not null)
.Select(g => g with
{
Detail = g.Detail.Trim(),
GuardSymbol = g.GuardSymbol.Trim(),
SourceFile = string.IsNullOrWhiteSpace(g.SourceFile) ? null : g.SourceFile.Trim(),
LineNumber = g.LineNumber,
Confidence = ClampConfidence(g.Confidence),
DetectionMethod = g.DetectionMethod.Trim()
})
.GroupBy(g => (g.Type, g.GuardSymbol))
.Select(group => group
.OrderByDescending(g => g.Confidence)
.ThenBy(g => g.Detail, StringComparer.Ordinal)
.ThenBy(g => g.DetectionMethod, StringComparer.Ordinal)
.First())
.OrderBy(g => g.Type)
.ThenBy(g => g.GuardSymbol, StringComparer.Ordinal)
.ThenBy(g => g.Detail, StringComparer.Ordinal)
.ToArray();
return this with
{
From = From.Trim(),
@@ -107,7 +132,9 @@ public sealed record RichGraphEdge(
Candidates = Candidates is null
? Array.Empty<string>()
: Candidates.Where(c => !string.IsNullOrWhiteSpace(c)).Select(c => c.Trim()).OrderBy(c => c, StringComparer.Ordinal).ToArray(),
Confidence = ClampConfidence(Confidence)
Confidence = ClampConfidence(Confidence),
Gates = gates,
GateMultiplierBps = Math.Clamp(GateMultiplierBps, 0, 10000)
};
}

35
src/Scanner/__Libraries/StellaOps.Scanner.Reachability/RichGraphWriter.cs
View File

@@ -4,6 +4,7 @@ using System.Text.Json;
using System.Threading;
using System.Threading.Tasks;
using StellaOps.Cryptography;
using StellaOps.Scanner.Reachability.Gates;
namespace StellaOps.Scanner.Reachability;
@@ -153,6 +154,30 @@ public sealed class RichGraphWriter
if (!string.IsNullOrWhiteSpace(edge.SymbolDigest)) writer.WriteString("symbol_digest", edge.SymbolDigest);
writer.WriteNumber("confidence", edge.Confidence);
if (edge.Gates is { Count: > 0 } || edge.GateMultiplierBps != 10000)
{
writer.WriteNumber("gate_multiplier_bps", edge.GateMultiplierBps);
}
if (edge.Gates is { Count: > 0 })
{
writer.WritePropertyName("gates");
writer.WriteStartArray();
foreach (var gate in edge.Gates)
{
writer.WriteStartObject();
writer.WriteString("type", GateTypeToLowerCamelCase(gate.Type));
writer.WriteString("detail", gate.Detail);
writer.WriteString("guard_symbol", gate.GuardSymbol);
if (!string.IsNullOrWhiteSpace(gate.SourceFile)) writer.WriteString("source_file", gate.SourceFile);
if (gate.LineNumber is not null) writer.WriteNumber("line_number", gate.LineNumber.Value);
writer.WriteNumber("confidence", gate.Confidence);
writer.WriteString("detection_method", gate.DetectionMethod);
writer.WriteEndObject();
}
writer.WriteEndArray();
}
if (edge.Evidence is { Count: > 0 })
{
writer.WritePropertyName("evidence");
@@ -188,6 +213,16 @@ public sealed class RichGraphWriter
writer.WriteEndObject();
}
private static string GateTypeToLowerCamelCase(GateType type)
=> type switch
{
GateType.AuthRequired => "authRequired",
GateType.FeatureFlag => "featureFlag",
GateType.AdminOnly => "adminOnly",
GateType.NonDefaultConfig => "nonDefaultConfig",
_ => type.ToString()
};
private static void WriteSymbol(Utf8JsonWriter writer, ReachabilitySymbol symbol)
{
writer.WriteStartObject();

29
src/Scanner/__Libraries/StellaOps.Scanner.ReachabilityDrift/DependencyInjection/ServiceCollectionExtensions.cs Normal file
View File

@@ -0,0 +1,29 @@
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.DependencyInjection.Extensions;
using StellaOps.Scanner.CallGraph;
using StellaOps.Scanner.ReachabilityDrift.Services;
namespace StellaOps.Scanner.ReachabilityDrift.DependencyInjection;
public static class ServiceCollectionExtensions
{
public static IServiceCollection AddReachabilityDrift(this IServiceCollection services)
{
ArgumentNullException.ThrowIfNull(services);
services.TryAddSingleton<CodeChangeFactExtractor>();
services.TryAddSingleton<DriftCauseExplainer>();
services.TryAddSingleton<PathCompressor>();
services.TryAddSingleton(sp =>
{
var timeProvider = sp.GetService<TimeProvider>();
return new ReachabilityAnalyzer(timeProvider);
});
services.TryAddSingleton<ReachabilityDriftDetector>();
return services;
}
}

41
src/Scanner/__Libraries/StellaOps.Scanner.ReachabilityDrift/DeterministicIds.cs Normal file
View File

@@ -0,0 +1,41 @@
using System.Security.Cryptography;
using System.Text;
namespace StellaOps.Scanner.ReachabilityDrift;
internal static class DeterministicIds
{
internal static readonly Guid CodeChangeNamespace = new("a420df67-6c4b-4f80-9870-0d070a845b4b");
internal static readonly Guid DriftResultNamespace = new("c60e2a63-9bc4-4ff0-9f8c-2a7c11c2f8c4");
internal static readonly Guid DriftedSinkNamespace = new("9b8ed5d2-4b6f-4f6f-9e3b-3a81e9f85a25");
public static Guid Create(Guid namespaceId, params string[] segments)
{
var normalized = string.Join(
'|',
segments.Select(static s => (s ?? string.Empty).Trim()));
return Create(namespaceId, Encoding.UTF8.GetBytes(normalized));
}
public static Guid Create(Guid namespaceId, ReadOnlySpan<byte> nameBytes)
{
Span<byte> namespaceBytes = stackalloc byte[16];
namespaceId.TryWriteBytes(namespaceBytes);
Span<byte> buffer = stackalloc byte[namespaceBytes.Length + nameBytes.Length];
namespaceBytes.CopyTo(buffer);
nameBytes.CopyTo(buffer[namespaceBytes.Length..]);
Span<byte> hash = stackalloc byte[32];
SHA256.TryHashData(buffer, hash, out _);
Span<byte> guidBytes = stackalloc byte[16];
hash[..16].CopyTo(guidBytes);
guidBytes[6] = (byte)((guidBytes[6] & 0x0F) | 0x50);
guidBytes[8] = (byte)((guidBytes[8] & 0x3F) | 0x80);
return new Guid(guidBytes);
}
}

293
src/Scanner/__Libraries/StellaOps.Scanner.ReachabilityDrift/Models/DriftModels.cs Normal file
View File

@@ -0,0 +1,293 @@
using System.Collections.Immutable;
using System.Text.Json;
using System.Text.Json.Serialization;
using StellaOps.Scanner.Reachability;
namespace StellaOps.Scanner.ReachabilityDrift;
public sealed record CodeChangeFact
{
[JsonPropertyName("id")]
public required Guid Id { get; init; }
[JsonPropertyName("scanId")]
public required string ScanId { get; init; }
[JsonPropertyName("baseScanId")]
public required string BaseScanId { get; init; }
[JsonPropertyName("language")]
public required string Language { get; init; }
[JsonPropertyName("nodeId")]
public string? NodeId { get; init; }
[JsonPropertyName("file")]
public required string File { get; init; }
[JsonPropertyName("symbol")]
public required string Symbol { get; init; }
[JsonPropertyName("kind")]
public required CodeChangeKind Kind { get; init; }
[JsonPropertyName("details")]
public JsonElement? Details { get; init; }
[JsonPropertyName("detectedAt")]
public required DateTimeOffset DetectedAt { get; init; }
}
[JsonConverter(typeof(JsonStringEnumConverter<CodeChangeKind>))]
public enum CodeChangeKind
{
[JsonStringEnumMemberName("added")]
Added,
[JsonStringEnumMemberName("removed")]
Removed,
[JsonStringEnumMemberName("signature_changed")]
SignatureChanged,
[JsonStringEnumMemberName("guard_changed")]
GuardChanged,
[JsonStringEnumMemberName("dependency_changed")]
DependencyChanged,
[JsonStringEnumMemberName("visibility_changed")]
VisibilityChanged
}
public sealed record ReachabilityDriftResult
{
[JsonPropertyName("id")]
public required Guid Id { get; init; }
[JsonPropertyName("baseScanId")]
public required string BaseScanId { get; init; }
[JsonPropertyName("headScanId")]
public required string HeadScanId { get; init; }
[JsonPropertyName("language")]
public required string Language { get; init; }
[JsonPropertyName("detectedAt")]
public required DateTimeOffset DetectedAt { get; init; }
[JsonPropertyName("newlyReachable")]
public required ImmutableArray<DriftedSink> NewlyReachable { get; init; }
[JsonPropertyName("newlyUnreachable")]
public required ImmutableArray<DriftedSink> NewlyUnreachable { get; init; }
[JsonPropertyName("resultDigest")]
public required string ResultDigest { get; init; }
[JsonPropertyName("totalDriftCount")]
public int TotalDriftCount => NewlyReachable.Length + NewlyUnreachable.Length;
[JsonPropertyName("hasMaterialDrift")]
public bool HasMaterialDrift => NewlyReachable.Length > 0;
}
public sealed record DriftedSink
{
[JsonPropertyName("id")]
public required Guid Id { get; init; }
[JsonPropertyName("sinkNodeId")]
public required string SinkNodeId { get; init; }
[JsonPropertyName("symbol")]
public required string Symbol { get; init; }
[JsonPropertyName("sinkCategory")]
public required SinkCategory SinkCategory { get; init; }
[JsonPropertyName("direction")]
public required DriftDirection Direction { get; init; }
[JsonPropertyName("cause")]
public required DriftCause Cause { get; init; }
[JsonPropertyName("path")]
public required CompressedPath Path { get; init; }
[JsonPropertyName("associatedVulns")]
public ImmutableArray<AssociatedVuln> AssociatedVulns { get; init; } = ImmutableArray<AssociatedVuln>.Empty;
}
[JsonConverter(typeof(JsonStringEnumConverter<DriftDirection>))]
public enum DriftDirection
{
[JsonStringEnumMemberName("became_reachable")]
BecameReachable,
[JsonStringEnumMemberName("became_unreachable")]
BecameUnreachable
}
public sealed record DriftCause
{
[JsonPropertyName("kind")]
public required DriftCauseKind Kind { get; init; }
[JsonPropertyName("description")]
public required string Description { get; init; }
[JsonPropertyName("changedSymbol")]
public string? ChangedSymbol { get; init; }
[JsonPropertyName("changedFile")]
public string? ChangedFile { get; init; }
[JsonPropertyName("changedLine")]
public int? ChangedLine { get; init; }
[JsonPropertyName("codeChangeId")]
public Guid? CodeChangeId { get; init; }
public static DriftCause GuardRemoved(string symbol) =>
new()
{
Kind = DriftCauseKind.GuardRemoved,
Description = $"Guard condition removed in {symbol}",
ChangedSymbol = symbol
};
public static DriftCause NewPublicRoute(string symbol) =>
new()
{
Kind = DriftCauseKind.NewPublicRoute,
Description = $"New public entrypoint: {symbol}",
ChangedSymbol = symbol
};
public static DriftCause VisibilityEscalated(string symbol) =>
new()
{
Kind = DriftCauseKind.VisibilityEscalated,
Description = $"Visibility escalated to public: {symbol}",
ChangedSymbol = symbol
};
public static DriftCause DependencyUpgraded(string package, string? fromVersion, string? toVersion) =>
new()
{
Kind = DriftCauseKind.DependencyUpgraded,
Description = $"Dependency changed: {package} {fromVersion ?? "?"} -> {toVersion ?? "?"}",
ChangedSymbol = package
};
public static DriftCause GuardAdded(string symbol) =>
new()
{
Kind = DriftCauseKind.GuardAdded,
Description = $"Guard condition added in {symbol}",
ChangedSymbol = symbol
};
public static DriftCause SymbolRemoved(string symbol) =>
new()
{
Kind = DriftCauseKind.SymbolRemoved,
Description = $"Symbol removed: {symbol}",
ChangedSymbol = symbol
};
public static DriftCause Unknown() =>
new()
{
Kind = DriftCauseKind.Unknown,
Description = "Cause could not be determined"
};
}
[JsonConverter(typeof(JsonStringEnumConverter<DriftCauseKind>))]
public enum DriftCauseKind
{
[JsonStringEnumMemberName("guard_removed")]
GuardRemoved,
[JsonStringEnumMemberName("guard_added")]
GuardAdded,
[JsonStringEnumMemberName("new_public_route")]
NewPublicRoute,
[JsonStringEnumMemberName("visibility_escalated")]
VisibilityEscalated,
[JsonStringEnumMemberName("dependency_upgraded")]
DependencyUpgraded,
[JsonStringEnumMemberName("symbol_removed")]
SymbolRemoved,
[JsonStringEnumMemberName("unknown")]
Unknown
}
public sealed record CompressedPath
{
[JsonPropertyName("entrypoint")]
public required PathNode Entrypoint { get; init; }
[JsonPropertyName("sink")]
public required PathNode Sink { get; init; }
[JsonPropertyName("intermediateCount")]
public required int IntermediateCount { get; init; }
[JsonPropertyName("keyNodes")]
public required ImmutableArray<PathNode> KeyNodes { get; init; }
[JsonPropertyName("fullPath")]
public ImmutableArray<string>? FullPath { get; init; }
}
public sealed record PathNode
{
[JsonPropertyName("nodeId")]
public required string NodeId { get; init; }
[JsonPropertyName("symbol")]
public required string Symbol { get; init; }
[JsonPropertyName("file")]
public string? File { get; init; }
[JsonPropertyName("line")]
public int? Line { get; init; }
[JsonPropertyName("package")]
public string? Package { get; init; }
[JsonPropertyName("isChanged")]
public bool IsChanged { get; init; }
[JsonPropertyName("changeKind")]
public CodeChangeKind? ChangeKind { get; init; }
}
public sealed record AssociatedVuln
{
[JsonPropertyName("cveId")]
public required string CveId { get; init; }
[JsonPropertyName("epss")]
public double? Epss { get; init; }
[JsonPropertyName("cvss")]
public double? Cvss { get; init; }
[JsonPropertyName("vexStatus")]
public string? VexStatus { get; init; }
[JsonPropertyName("packagePurl")]
public string? PackagePurl { get; init; }
}

342
src/Scanner/__Libraries/StellaOps.Scanner.ReachabilityDrift/Services/CodeChangeFactExtractor.cs Normal file
View File

@@ -0,0 +1,342 @@
using System.Collections.Immutable;
using System.Text.Json;
using StellaOps.Scanner.CallGraph;
namespace StellaOps.Scanner.ReachabilityDrift.Services;
public sealed class CodeChangeFactExtractor
{
private readonly TimeProvider _timeProvider;
public CodeChangeFactExtractor(TimeProvider? timeProvider = null)
{
_timeProvider = timeProvider ?? TimeProvider.System;
}
public IReadOnlyList<CodeChangeFact> Extract(CallGraphSnapshot baseGraph, CallGraphSnapshot headGraph)
{
ArgumentNullException.ThrowIfNull(baseGraph);
ArgumentNullException.ThrowIfNull(headGraph);
var baseTrimmed = baseGraph.Trimmed();
var headTrimmed = headGraph.Trimmed();
if (!string.Equals(baseTrimmed.Language, headTrimmed.Language, StringComparison.OrdinalIgnoreCase))
{
throw new ArgumentException(
$"Language mismatch: base='{baseTrimmed.Language}', head='{headTrimmed.Language}'.");
}
var now = _timeProvider.GetUtcNow();
var baseById = baseTrimmed.Nodes.ToDictionary(n => n.NodeId, StringComparer.Ordinal);
var headById = headTrimmed.Nodes.ToDictionary(n => n.NodeId, StringComparer.Ordinal);
var removed = baseById
.Where(kvp => !headById.ContainsKey(kvp.Key))
.Select(kvp => kvp.Value)
.OrderBy(n => n.NodeId, StringComparer.Ordinal)
.ToImmutableArray();
var added = headById
.Where(kvp => !baseById.ContainsKey(kvp.Key))
.Select(kvp => kvp.Value)
.OrderBy(n => n.NodeId, StringComparer.Ordinal)
.ToImmutableArray();
var signaturePairs = MatchSignatureChanges(removed, added);
var consumedRemoved = new HashSet<string>(signaturePairs.Select(p => p.Removed.NodeId), StringComparer.Ordinal);
var consumedAdded = new HashSet<string>(signaturePairs.Select(p => p.Added.NodeId), StringComparer.Ordinal);
var facts = new List<CodeChangeFact>(added.Length + removed.Length);
foreach (var pair in signaturePairs)
{
var details = JsonSerializer.SerializeToElement(new
{
fromSymbol = pair.Removed.Symbol,
toSymbol = pair.Added.Symbol,
fromNodeId = pair.Removed.NodeId,
toNodeId = pair.Added.NodeId
});
facts.Add(CreateFact(
headTrimmed,
baseTrimmed,
pair.Added,
CodeChangeKind.SignatureChanged,
now,
details));
}
foreach (var node in added)
{
if (consumedAdded.Contains(node.NodeId))
{
continue;
}
facts.Add(CreateFact(
headTrimmed,
baseTrimmed,
node,
CodeChangeKind.Added,
now,
JsonSerializer.SerializeToElement(new { nodeId = node.NodeId })));
}
foreach (var node in removed)
{
if (consumedRemoved.Contains(node.NodeId))
{
continue;
}
facts.Add(CreateFact(
headTrimmed,
baseTrimmed,
node,
CodeChangeKind.Removed,
now,
JsonSerializer.SerializeToElement(new { nodeId = node.NodeId })));
}
foreach (var (nodeId, baseNode) in baseById.OrderBy(kvp => kvp.Key, StringComparer.Ordinal))
{
if (!headById.TryGetValue(nodeId, out var headNode))
{
continue;
}
if (!string.Equals(baseNode.Package, headNode.Package, StringComparison.Ordinal))
{
var details = JsonSerializer.SerializeToElement(new
{
nodeId,
from = baseNode.Package,
to = headNode.Package
});
facts.Add(CreateFact(
headTrimmed,
baseTrimmed,
headNode,
CodeChangeKind.DependencyChanged,
now,
details));
}
if (baseNode.Visibility != headNode.Visibility)
{
var details = JsonSerializer.SerializeToElement(new
{
nodeId,
from = baseNode.Visibility.ToString(),
to = headNode.Visibility.ToString()
});
facts.Add(CreateFact(
headTrimmed,
baseTrimmed,
headNode,
CodeChangeKind.VisibilityChanged,
now,
details));
}
}
foreach (var edgeFact in ExtractEdgeFacts(baseTrimmed, headTrimmed, now))
{
facts.Add(edgeFact);
}
return facts
.OrderBy(f => f.Kind.ToString(), StringComparer.Ordinal)
.ThenBy(f => f.File, StringComparer.Ordinal)
.ThenBy(f => f.Symbol, StringComparer.Ordinal)
.ThenBy(f => f.Id)
.ToList();
}
private static CodeChangeFact CreateFact(
CallGraphSnapshot head,
CallGraphSnapshot @base,
CallGraphNode node,
CodeChangeKind kind,
DateTimeOffset detectedAt,
JsonElement? details)
{
var id = DeterministicIds.Create(
DeterministicIds.CodeChangeNamespace,
head.ScanId,
@base.ScanId,
head.Language,
kind.ToString(),
node.NodeId,
node.File,
node.Symbol);
return new CodeChangeFact
{
Id = id,
ScanId = head.ScanId,
BaseScanId = @base.ScanId,
Language = head.Language,
NodeId = node.NodeId,
File = node.File,
Symbol = node.Symbol,
Kind = kind,
Details = details,
DetectedAt = detectedAt
};
}
private static IEnumerable<CodeChangeFact> ExtractEdgeFacts(
CallGraphSnapshot baseTrimmed,
CallGraphSnapshot headTrimmed,
DateTimeOffset detectedAt)
{
var baseEdges = baseTrimmed.Edges
.Select(EdgeKey.Create)
.ToHashSet(StringComparer.Ordinal);
var headEdges = headTrimmed.Edges
.Select(EdgeKey.Create)
.ToHashSet(StringComparer.Ordinal);
var baseById = baseTrimmed.Nodes.ToDictionary(n => n.NodeId, StringComparer.Ordinal);
var headById = headTrimmed.Nodes.ToDictionary(n => n.NodeId, StringComparer.Ordinal);
foreach (var key in headEdges.Except(baseEdges).OrderBy(k => k, StringComparer.Ordinal))
{
if (!EdgeKey.TryParse(key, out var parsed))
{
continue;
}
if (!headById.TryGetValue(parsed.SourceId, out var sourceNode))
{
continue;
}
var details = JsonSerializer.SerializeToElement(new
{
nodeId = sourceNode.NodeId,
change = "edge_added",
sourceId = parsed.SourceId,
targetId = parsed.TargetId,
callKind = parsed.CallKind,
callSite = parsed.CallSite
});
yield return CreateFact(headTrimmed, baseTrimmed, sourceNode, CodeChangeKind.GuardChanged, detectedAt, details);
}
foreach (var key in baseEdges.Except(headEdges).OrderBy(k => k, StringComparer.Ordinal))
{
if (!EdgeKey.TryParse(key, out var parsed))
{
continue;
}
if (!baseById.TryGetValue(parsed.SourceId, out var sourceNode))
{
continue;
}
var details = JsonSerializer.SerializeToElement(new
{
nodeId = sourceNode.NodeId,
change = "edge_removed",
sourceId = parsed.SourceId,
targetId = parsed.TargetId,
callKind = parsed.CallKind,
callSite = parsed.CallSite
});
yield return CreateFact(headTrimmed, baseTrimmed, sourceNode, CodeChangeKind.GuardChanged, detectedAt, details);
}
}
private static ImmutableArray<(CallGraphNode Removed, CallGraphNode Added)> MatchSignatureChanges(
ImmutableArray<CallGraphNode> removed,
ImmutableArray<CallGraphNode> added)
{
var removedByKey = removed
.GroupBy(BuildSignatureKey, StringComparer.Ordinal)
.ToDictionary(g => g.Key, g => g.OrderBy(n => n.NodeId, StringComparer.Ordinal).ToList(), StringComparer.Ordinal);
var addedByKey = added
.GroupBy(BuildSignatureKey, StringComparer.Ordinal)
.ToDictionary(g => g.Key, g => g.OrderBy(n => n.NodeId, StringComparer.Ordinal).ToList(), StringComparer.Ordinal);
var pairs = new List<(CallGraphNode Removed, CallGraphNode Added)>();
foreach (var key in removedByKey.Keys.OrderBy(k => k, StringComparer.Ordinal))
{
if (!addedByKey.TryGetValue(key, out var addedCandidates))
{
continue;
}
var removedCandidates = removedByKey[key];
var count = Math.Min(removedCandidates.Count, addedCandidates.Count);
for (var i = 0; i < count; i++)
{
pairs.Add((removedCandidates[i], addedCandidates[i]));
}
}
return pairs
.OrderBy(p => p.Removed.NodeId, StringComparer.Ordinal)
.ThenBy(p => p.Added.NodeId, StringComparer.Ordinal)
.ToImmutableArray();
}
private static string BuildSignatureKey(CallGraphNode node)
{
var file = node.File?.Trim() ?? string.Empty;
var symbolKey = GetSymbolKey(node.Symbol);
return $"{file}|{symbolKey}";
}
private static string GetSymbolKey(string symbol)
{
if (string.IsNullOrWhiteSpace(symbol))
{
return string.Empty;
}
var trimmed = symbol.Trim();
var parenIndex = trimmed.IndexOf('(');
if (parenIndex > 0)
{
trimmed = trimmed[..parenIndex];
}
return trimmed.Replace("global::", string.Empty, StringComparison.Ordinal).Trim();
}
private readonly record struct EdgeKey(string SourceId, string TargetId, string CallKind, string? CallSite)
{
public static string Create(CallGraphEdge edge)
{
var callSite = string.IsNullOrWhiteSpace(edge.CallSite) ? string.Empty : edge.CallSite.Trim();
return $"{edge.SourceId}|{edge.TargetId}|{edge.CallKind}|{callSite}";
}
public static bool TryParse(string key, out EdgeKey parsed)
{
var parts = key.Split('|');
if (parts.Length != 4)
{
parsed = default;
return false;
}
parsed = new EdgeKey(parts[0], parts[1], parts[2], string.IsNullOrWhiteSpace(parts[3]) ? null : parts[3]);
return true;
}
}
}

254
src/Scanner/__Libraries/StellaOps.Scanner.ReachabilityDrift/Services/DriftCauseExplainer.cs Normal file
View File

@@ -0,0 +1,254 @@
using System.Collections.Immutable;
using StellaOps.Scanner.CallGraph;
namespace StellaOps.Scanner.ReachabilityDrift.Services;
public sealed class DriftCauseExplainer
{
public DriftCause ExplainNewlyReachable(
CallGraphSnapshot baseGraph,
CallGraphSnapshot headGraph,
string sinkNodeId,
ImmutableArray<string> pathNodeIds,
IReadOnlyList<CodeChangeFact> codeChanges)
{
ArgumentNullException.ThrowIfNull(baseGraph);
ArgumentNullException.ThrowIfNull(headGraph);
ArgumentException.ThrowIfNullOrWhiteSpace(sinkNodeId);
ArgumentNullException.ThrowIfNull(codeChanges);
var baseTrimmed = baseGraph.Trimmed();
var headTrimmed = headGraph.Trimmed();
if (!pathNodeIds.IsDefaultOrEmpty)
{
var entrypointId = pathNodeIds[0];
var isNewEntrypoint = !baseTrimmed.EntrypointIds.Contains(entrypointId, StringComparer.Ordinal)
&& headTrimmed.EntrypointIds.Contains(entrypointId, StringComparer.Ordinal);
if (isNewEntrypoint)
{
var symbol = ResolveSymbol(headTrimmed, entrypointId) ?? entrypointId;
return DriftCause.NewPublicRoute(symbol);
}
}
var escalated = FindVisibilityEscalation(baseTrimmed, headTrimmed, pathNodeIds, codeChanges);
if (escalated is not null)
{
return escalated;
}
var dependency = FindDependencyChange(baseTrimmed, headTrimmed, pathNodeIds, codeChanges);
if (dependency is not null)
{
return dependency;
}
var guardRemoved = FindEdgeAdded(baseTrimmed, headTrimmed, pathNodeIds);
if (guardRemoved is not null)
{
return guardRemoved;
}
return DriftCause.Unknown();
}
public DriftCause ExplainNewlyUnreachable(
CallGraphSnapshot baseGraph,
CallGraphSnapshot headGraph,
string sinkNodeId,
ImmutableArray<string> basePathNodeIds,
IReadOnlyList<CodeChangeFact> codeChanges)
{
ArgumentNullException.ThrowIfNull(baseGraph);
ArgumentNullException.ThrowIfNull(headGraph);
ArgumentException.ThrowIfNullOrWhiteSpace(sinkNodeId);
ArgumentNullException.ThrowIfNull(codeChanges);
var baseTrimmed = baseGraph.Trimmed();
var headTrimmed = headGraph.Trimmed();
if (!headTrimmed.Nodes.Any(n => n.NodeId == sinkNodeId))
{
var symbol = ResolveSymbol(baseTrimmed, sinkNodeId) ?? sinkNodeId;
return DriftCause.SymbolRemoved(symbol);
}
var guardAdded = FindEdgeRemoved(baseTrimmed, headTrimmed, basePathNodeIds);
if (guardAdded is not null)
{
return guardAdded;
}
return DriftCause.Unknown();
}
private static DriftCause? FindVisibilityEscalation(
CallGraphSnapshot baseTrimmed,
CallGraphSnapshot headTrimmed,
ImmutableArray<string> pathNodeIds,
IReadOnlyList<CodeChangeFact> codeChanges)
{
if (pathNodeIds.IsDefaultOrEmpty)
{
return null;
}
var baseById = baseTrimmed.Nodes.ToDictionary(n => n.NodeId, StringComparer.Ordinal);
var headById = headTrimmed.Nodes.ToDictionary(n => n.NodeId, StringComparer.Ordinal);
foreach (var nodeId in pathNodeIds)
{
if (!baseById.TryGetValue(nodeId, out var baseNode) || !headById.TryGetValue(nodeId, out var headNode))
{
continue;
}
if (baseNode.Visibility == Visibility.Public || headNode.Visibility != Visibility.Public)
{
continue;
}
var matching = codeChanges
.Where(c => c.Kind == CodeChangeKind.VisibilityChanged && string.Equals(c.NodeId, nodeId, StringComparison.Ordinal))
.OrderBy(c => c.Id)
.FirstOrDefault();
return matching is not null
? new DriftCause
{
Kind = DriftCauseKind.VisibilityEscalated,
Description = $"Visibility escalated to public: {headNode.Symbol}",
ChangedSymbol = headNode.Symbol,
ChangedFile = headNode.File,
ChangedLine = headNode.Line,
CodeChangeId = matching.Id
}
: DriftCause.VisibilityEscalated(headNode.Symbol);
}
return null;
}
private static DriftCause? FindDependencyChange(
CallGraphSnapshot baseTrimmed,
CallGraphSnapshot headTrimmed,
ImmutableArray<string> pathNodeIds,
IReadOnlyList<CodeChangeFact> codeChanges)
{
if (pathNodeIds.IsDefaultOrEmpty)
{
return null;
}
var baseById = baseTrimmed.Nodes.ToDictionary(n => n.NodeId, StringComparer.Ordinal);
var headById = headTrimmed.Nodes.ToDictionary(n => n.NodeId, StringComparer.Ordinal);
foreach (var nodeId in pathNodeIds)
{
if (!baseById.TryGetValue(nodeId, out var baseNode) || !headById.TryGetValue(nodeId, out var headNode))
{
continue;
}
if (string.Equals(baseNode.Package, headNode.Package, StringComparison.Ordinal))
{
continue;
}
var matching = codeChanges
.Where(c => c.Kind == CodeChangeKind.DependencyChanged && string.Equals(c.NodeId, nodeId, StringComparison.Ordinal))
.OrderBy(c => c.Id)
.FirstOrDefault();
return matching is not null
? new DriftCause
{
Kind = DriftCauseKind.DependencyUpgraded,
Description = $"Dependency changed: {baseNode.Package} -> {headNode.Package}",
ChangedSymbol = headNode.Package,
ChangedFile = headNode.File,
ChangedLine = headNode.Line,
CodeChangeId = matching.Id
}
: DriftCause.DependencyUpgraded(headNode.Package, baseNode.Package, headNode.Package);
}
return null;
}
private static DriftCause? FindEdgeAdded(
CallGraphSnapshot baseTrimmed,
CallGraphSnapshot headTrimmed,
ImmutableArray<string> pathNodeIds)
{
if (pathNodeIds.IsDefaultOrEmpty || pathNodeIds.Length < 2)
{
return null;
}
var baseEdges = baseTrimmed.Edges
.Select(e => $"{e.SourceId}|{e.TargetId}")
.ToHashSet(StringComparer.Ordinal);
var headEdges = headTrimmed.Edges
.Select(e => $"{e.SourceId}|{e.TargetId}")
.ToHashSet(StringComparer.Ordinal);
var headById = headTrimmed.Nodes.ToDictionary(n => n.NodeId, StringComparer.Ordinal);
for (var i = 0; i < pathNodeIds.Length - 1; i++)
{
var from = pathNodeIds[i];
var to = pathNodeIds[i + 1];
var key = $"{from}|{to}";
if (headEdges.Contains(key) && !baseEdges.Contains(key) && headById.TryGetValue(from, out var node))
{
return DriftCause.GuardRemoved(node.Symbol);
}
}
return null;
}
private static DriftCause? FindEdgeRemoved(
CallGraphSnapshot baseTrimmed,
CallGraphSnapshot headTrimmed,
ImmutableArray<string> basePathNodeIds)
{
if (basePathNodeIds.IsDefaultOrEmpty || basePathNodeIds.Length < 2)
{
return null;
}
var baseEdges = baseTrimmed.Edges
.Select(e => $"{e.SourceId}|{e.TargetId}")
.ToHashSet(StringComparer.Ordinal);
var headEdges = headTrimmed.Edges
.Select(e => $"{e.SourceId}|{e.TargetId}")
.ToHashSet(StringComparer.Ordinal);
var baseById = baseTrimmed.Nodes.ToDictionary(n => n.NodeId, StringComparer.Ordinal);
for (var i = 0; i < basePathNodeIds.Length - 1; i++)
{
var from = basePathNodeIds[i];
var to = basePathNodeIds[i + 1];
var key = $"{from}|{to}";
if (baseEdges.Contains(key) && !headEdges.Contains(key) && baseById.TryGetValue(from, out var node))
{
return DriftCause.GuardAdded(node.Symbol);
}
}
return null;
}
private static string? ResolveSymbol(CallGraphSnapshot graph, string nodeId)
=> graph.Nodes.FirstOrDefault(n => string.Equals(n.NodeId, nodeId, StringComparison.Ordinal))?.Symbol;
}

147
src/Scanner/__Libraries/StellaOps.Scanner.ReachabilityDrift/Services/PathCompressor.cs Normal file
View File

@@ -0,0 +1,147 @@
using System.Collections.Immutable;
using StellaOps.Scanner.CallGraph;
namespace StellaOps.Scanner.ReachabilityDrift.Services;
public sealed class PathCompressor
{
private readonly int _maxKeyNodes;
public PathCompressor(int maxKeyNodes = 5)
{
_maxKeyNodes = maxKeyNodes <= 0 ? 5 : maxKeyNodes;
}
public CompressedPath Compress(
ImmutableArray<string> pathNodeIds,
CallGraphSnapshot graph,
IReadOnlyList<CodeChangeFact> codeChanges,
bool includeFullPath)
{
ArgumentNullException.ThrowIfNull(graph);
ArgumentNullException.ThrowIfNull(codeChanges);
var trimmed = graph.Trimmed();
var nodeMap = trimmed.Nodes.ToDictionary(n => n.NodeId, StringComparer.Ordinal);
if (pathNodeIds.IsDefaultOrEmpty)
{
var empty = CreatePathNode(nodeMap, string.Empty, codeChanges);
return new CompressedPath
{
Entrypoint = empty,
Sink = empty,
IntermediateCount = 0,
KeyNodes = ImmutableArray<PathNode>.Empty,
FullPath = includeFullPath ? ImmutableArray<string>.Empty : null
};
}
var entryId = pathNodeIds[0];
var sinkId = pathNodeIds[^1];
var entry = CreatePathNode(nodeMap, entryId, codeChanges);
var sink = CreatePathNode(nodeMap, sinkId, codeChanges);
var intermediateCount = Math.Max(0, pathNodeIds.Length - 2);
var intermediates = intermediateCount == 0
? ImmutableArray<string>.Empty
: pathNodeIds.Skip(1).Take(pathNodeIds.Length - 2).ToImmutableArray();
var changedNodes = new HashSet<string>(
codeChanges
.Select(c => c.NodeId)
.Where(id => !string.IsNullOrWhiteSpace(id))
.Select(id => id!)
.Distinct(StringComparer.Ordinal),
StringComparer.Ordinal);
var keyNodeIds = new List<string>(_maxKeyNodes);
foreach (var nodeId in intermediates)
{
if (changedNodes.Contains(nodeId))
{
keyNodeIds.Add(nodeId);
if (keyNodeIds.Count >= _maxKeyNodes)
{
break;
}
}
}
if (keyNodeIds.Count < _maxKeyNodes && intermediates.Length > 0)
{
var remaining = _maxKeyNodes - keyNodeIds.Count;
var candidates = intermediates.Where(id => !keyNodeIds.Contains(id, StringComparer.Ordinal)).ToList();
if (candidates.Count > 0 && remaining > 0)
{
var step = (candidates.Count + 1.0) / (remaining + 1.0);
for (var i = 1; i <= remaining; i++)
{
var index = (int)Math.Round(i * step) - 1;
index = Math.Clamp(index, 0, candidates.Count - 1);
keyNodeIds.Add(candidates[index]);
if (keyNodeIds.Count >= _maxKeyNodes)
{
break;
}
}
}
}
var keyNodes = keyNodeIds
.Distinct(StringComparer.Ordinal)
.Select(id => CreatePathNode(nodeMap, id, codeChanges))
.OrderBy(n => IndexOf(pathNodeIds, n.NodeId), Comparer<int>.Default)
.ToImmutableArray();
return new CompressedPath
{
Entrypoint = entry,
Sink = sink,
IntermediateCount = intermediateCount,
KeyNodes = keyNodes,
FullPath = includeFullPath ? pathNodeIds : null
};
}
private static PathNode CreatePathNode(
IReadOnlyDictionary<string, CallGraphNode> nodeMap,
string nodeId,
IReadOnlyList<CodeChangeFact> changes)
{
nodeMap.TryGetValue(nodeId, out var node);
var change = changes
.Where(c => string.Equals(c.NodeId, nodeId, StringComparison.Ordinal))
.OrderBy(c => c.Kind.ToString(), StringComparer.Ordinal)
.ThenBy(c => c.Id)
.FirstOrDefault();
return new PathNode
{
NodeId = nodeId,
Symbol = node?.Symbol ?? string.Empty,
File = string.IsNullOrWhiteSpace(node?.File) ? null : node.File,
Line = node?.Line > 0 ? node.Line : null,
Package = string.IsNullOrWhiteSpace(node?.Package) ? null : node.Package,
IsChanged = change is not null,
ChangeKind = change?.Kind
};
}
private static int IndexOf(ImmutableArray<string> path, string nodeId)
{
for (var i = 0; i < path.Length; i++)
{
if (string.Equals(path[i], nodeId, StringComparison.Ordinal))
{
return i;
}
}
return int.MaxValue;
}
}

176
src/Scanner/__Libraries/StellaOps.Scanner.ReachabilityDrift/Services/ReachabilityDriftDetector.cs Normal file
View File

@@ -0,0 +1,176 @@
using System.Collections.Immutable;
using System.Security.Cryptography;
using System.Text;
using StellaOps.Scanner.CallGraph;
namespace StellaOps.Scanner.ReachabilityDrift.Services;
public sealed class ReachabilityDriftDetector
{
private readonly TimeProvider _timeProvider;
private readonly ReachabilityAnalyzer _reachabilityAnalyzer;
private readonly DriftCauseExplainer _causeExplainer;
private readonly PathCompressor _pathCompressor;
public ReachabilityDriftDetector(
TimeProvider? timeProvider = null,
ReachabilityAnalyzer? reachabilityAnalyzer = null,
DriftCauseExplainer? causeExplainer = null,
PathCompressor? pathCompressor = null)
{
_timeProvider = timeProvider ?? TimeProvider.System;
_reachabilityAnalyzer = reachabilityAnalyzer ?? new ReachabilityAnalyzer(_timeProvider);
_causeExplainer = causeExplainer ?? new DriftCauseExplainer();
_pathCompressor = pathCompressor ?? new PathCompressor();
}
public ReachabilityDriftResult Detect(
CallGraphSnapshot baseGraph,
CallGraphSnapshot headGraph,
IReadOnlyList<CodeChangeFact> codeChanges,
bool includeFullPath = false)
{
ArgumentNullException.ThrowIfNull(baseGraph);
ArgumentNullException.ThrowIfNull(headGraph);
ArgumentNullException.ThrowIfNull(codeChanges);
var baseTrimmed = baseGraph.Trimmed();
var headTrimmed = headGraph.Trimmed();
if (!string.Equals(baseTrimmed.Language, headTrimmed.Language, StringComparison.OrdinalIgnoreCase))
{
throw new ArgumentException(
$"Language mismatch: base='{baseTrimmed.Language}', head='{headTrimmed.Language}'.");
}
var baseReachability = _reachabilityAnalyzer.Analyze(baseTrimmed);
var headReachability = _reachabilityAnalyzer.Analyze(headTrimmed);
var baseReachable = baseReachability.ReachableSinkIds.ToHashSet(StringComparer.Ordinal);
var headReachable = headReachability.ReachableSinkIds.ToHashSet(StringComparer.Ordinal);
var headPaths = headReachability.Paths
.ToDictionary(p => p.SinkId, p => p.NodeIds, StringComparer.Ordinal);
var basePaths = baseReachability.Paths
.ToDictionary(p => p.SinkId, p => p.NodeIds, StringComparer.Ordinal);
var baseNodes = baseTrimmed.Nodes.ToDictionary(n => n.NodeId, StringComparer.Ordinal);
var headNodes = headTrimmed.Nodes.ToDictionary(n => n.NodeId, StringComparer.Ordinal);
var newlyReachableIds = headReachable
.Except(baseReachable)
.OrderBy(id => id, StringComparer.Ordinal)
.ToImmutableArray();
var newlyUnreachableIds = baseReachable
.Except(headReachable)
.OrderBy(id => id, StringComparer.Ordinal)
.ToImmutableArray();
var detectedAt = _timeProvider.GetUtcNow();
var resultDigest = ComputeDigest(
baseTrimmed.ScanId,
headTrimmed.ScanId,
headTrimmed.Language,
newlyReachableIds,
newlyUnreachableIds);
var driftId = DeterministicIds.Create(
DeterministicIds.DriftResultNamespace,
baseTrimmed.ScanId,
headTrimmed.ScanId,
headTrimmed.Language,
resultDigest);
var newlyReachable = newlyReachableIds
.Select(sinkId =>
{
headNodes.TryGetValue(sinkId, out var sinkNode);
sinkNode ??= new CallGraphNode(sinkId, sinkId, string.Empty, 0, string.Empty, Visibility.Private, false, null, true, null);
var path = headPaths.TryGetValue(sinkId, out var nodeIds) ? nodeIds : ImmutableArray<string>.Empty;
if (path.IsDefaultOrEmpty)
{
path = ImmutableArray.Create(sinkId);
}
var cause = _causeExplainer.ExplainNewlyReachable(baseTrimmed, headTrimmed, sinkId, path, codeChanges);
var compressed = _pathCompressor.Compress(path, headTrimmed, codeChanges, includeFullPath);
return new DriftedSink
{
Id = DeterministicIds.Create(DeterministicIds.DriftedSinkNamespace, driftId.ToString("n"), sinkId),
SinkNodeId = sinkId,
Symbol = sinkNode.Symbol,
SinkCategory = sinkNode.SinkCategory ?? Reachability.SinkCategory.CmdExec,
Direction = DriftDirection.BecameReachable,
Cause = cause,
Path = compressed
};
})
.OrderBy(s => s.SinkNodeId, StringComparer.Ordinal)
.ToImmutableArray();
var newlyUnreachable = newlyUnreachableIds
.Select(sinkId =>
{
baseNodes.TryGetValue(sinkId, out var sinkNode);
sinkNode ??= new CallGraphNode(sinkId, sinkId, string.Empty, 0, string.Empty, Visibility.Private, false, null, true, null);
var path = basePaths.TryGetValue(sinkId, out var nodeIds) ? nodeIds : ImmutableArray<string>.Empty;
if (path.IsDefaultOrEmpty)
{
path = ImmutableArray.Create(sinkId);
}
var cause = _causeExplainer.ExplainNewlyUnreachable(baseTrimmed, headTrimmed, sinkId, path, codeChanges);
var compressed = _pathCompressor.Compress(path, baseTrimmed, codeChanges, includeFullPath);
return new DriftedSink
{
Id = DeterministicIds.Create(DeterministicIds.DriftedSinkNamespace, driftId.ToString("n"), sinkId),
SinkNodeId = sinkId,
Symbol = sinkNode.Symbol,
SinkCategory = sinkNode.SinkCategory ?? Reachability.SinkCategory.CmdExec,
Direction = DriftDirection.BecameUnreachable,
Cause = cause,
Path = compressed
};
})
.OrderBy(s => s.SinkNodeId, StringComparer.Ordinal)
.ToImmutableArray();
return new ReachabilityDriftResult
{
Id = driftId,
BaseScanId = baseTrimmed.ScanId,
HeadScanId = headTrimmed.ScanId,
Language = headTrimmed.Language,
DetectedAt = detectedAt,
NewlyReachable = newlyReachable,
NewlyUnreachable = newlyUnreachable,
ResultDigest = resultDigest
};
}
private static string ComputeDigest(
string baseScanId,
string headScanId,
string language,
ImmutableArray<string> newlyReachableIds,
ImmutableArray<string> newlyUnreachableIds)
{
var builder = new StringBuilder();
builder.Append(baseScanId.Trim()).Append('|');
builder.Append(headScanId.Trim()).Append('|');
builder.Append(language.Trim().ToLowerInvariant()).Append('|');
builder.Append(string.Join(',', newlyReachableIds)).Append('|');
builder.Append(string.Join(',', newlyUnreachableIds));
var hash = SHA256.HashData(Encoding.UTF8.GetBytes(builder.ToString()));
return Convert.ToHexString(hash).ToLowerInvariant();
}
}

19
src/Scanner/__Libraries/StellaOps.Scanner.ReachabilityDrift/StellaOps.Scanner.ReachabilityDrift.csproj Normal file
View File

@@ -0,0 +1,19 @@
<Project Sdk="Microsoft.NET.Sdk">
<PropertyGroup>
<TargetFramework>net10.0</TargetFramework>
<LangVersion>preview</LangVersion>
<Nullable>enable</Nullable>
<ImplicitUsings>enable</ImplicitUsings>
<GenerateAssemblyInfo>false</GenerateAssemblyInfo>
</PropertyGroup>
<ItemGroup>
<PackageReference Include="Microsoft.Extensions.DependencyInjection.Abstractions" Version="10.0.0" />
<PackageReference Include="Microsoft.Extensions.Logging.Abstractions" Version="10.0.0" />
</ItemGroup>
<ItemGroup>
<ProjectReference Include="..\\StellaOps.Scanner.CallGraph\\StellaOps.Scanner.CallGraph.csproj" />
</ItemGroup>
</Project>

102
src/Scanner/__Libraries/StellaOps.Scanner.SmartDiff/Detection/MaterialRiskChangeDetector.cs
View File

@@ -23,7 +23,7 @@ public sealed class MaterialRiskChangeDetector
RiskStateSnapshot previous,
RiskStateSnapshot current)
{
if (previous.FindingKey != current.FindingKey)
if (!FindingKeysMatch(previous.FindingKey, current.FindingKey))
throw new ArgumentException("FindingKey mismatch between snapshots");
var changes = new List<DetectedChange>();
@@ -56,6 +56,11 @@ public sealed class MaterialRiskChangeDetector
CurrentStateHash: current.ComputeStateHash());
}
public MaterialRiskChangeResult DetectChanges(
RiskStateSnapshot previous,
RiskStateSnapshot current)
=> Compare(previous, current);
/// <summary>
/// R1: Reachability Flip - reachable changes false→true or true→false
/// </summary>
@@ -286,40 +291,79 @@ public sealed class MaterialRiskChangeDetector
if (changes.Count == 0)
return 0;
// Sum weighted changes
var weightedSum = 0.0;
foreach (var change in changes)
// Priority scoring per Smart-Diff advisory (A9):
// + 1000 if new.kev
// + 500 if new.reachable
// + 200 if RANGE_FLIP to affected
// + 150 if VEX_FLIP to affected
// + 0..100 based on EPSS (epss * 100)
// + policy weight: +300 if BLOCK, +100 if WARN
var score = 0;
if (current.Kev)
score += 1000;
if (current.Reachable == true)
score += 500;
if (changes.Any(c => c.Rule == DetectionRule.R3_RangeBoundary
&& c.Direction == RiskDirection.Increased
&& current.InAffectedRange == true))
{
var directionMultiplier = change.Direction switch
{
RiskDirection.Increased => 1.0,
RiskDirection.Decreased => -0.5,
RiskDirection.Neutral => 0.0,
_ => 0.0
};
weightedSum += change.Weight * directionMultiplier;
score += 200;
}
// Base severity from EPSS or default
var baseSeverity = current.EpssScore ?? 0.5;
// KEV boost
var kevBoost = current.Kev ? 1.5 : 1.0;
// Confidence factor from lattice state
var confidence = current.LatticeState switch
if (changes.Any(c => c.Rule == DetectionRule.R2_VexFlip
&& c.Direction == RiskDirection.Increased
&& current.VexStatus == VexStatusType.Affected))
{
"certain_reachable" => 1.0,
"likely_reachable" => 0.9,
"uncertain" => 0.7,
"likely_unreachable" => 0.5,
"certain_unreachable" => 0.3,
_ => 0.7
score += 150;
}
if (current.EpssScore is not null)
{
var epss = Math.Clamp(current.EpssScore.Value, 0.0, 1.0);
score += (int)Math.Round(epss * 100.0, 0, MidpointRounding.AwayFromZero);
}
score += current.PolicyDecision switch
{
PolicyDecisionType.Block => 300,
PolicyDecisionType.Warn => 100,
_ => 0
};
var score = baseSeverity * weightedSum * kevBoost * confidence;
return score;
}
// Clamp to [-1, 1]
return Math.Clamp(score, -1.0, 1.0);
private static bool FindingKeysMatch(FindingKey previous, FindingKey current)
{
if (!StringComparer.Ordinal.Equals(previous.VulnId, current.VulnId))
{
return false;
}
var prevPurl = NormalizePurlForComparison(previous.ComponentPurl);
var currPurl = NormalizePurlForComparison(current.ComponentPurl);
return StringComparer.Ordinal.Equals(prevPurl, currPurl);
}
private static string NormalizePurlForComparison(string purl)
{
// Strip the version segment (`@<version>`) while preserving qualifiers (`?`) and subpath (`#`).
var atIndex = purl.IndexOf('@');
if (atIndex < 0)
{
return purl;
}
var endIndex = purl.IndexOfAny(['?', '#'], atIndex);
if (endIndex < 0)
{
endIndex = purl.Length;
}
return purl.Remove(atIndex, endIndex - atIndex);
}
}

2
src/Scanner/__Libraries/StellaOps.Scanner.SmartDiff/Detection/MaterialRiskChangeResult.cs
View File

@@ -147,7 +147,7 @@ public sealed class MaterialRiskChangeOptions
/// <summary>
/// EPSS score threshold for R4 detection.
/// </summary>
public double EpssThreshold { get; init; } = 0.5;
public double EpssThreshold { get; init; } = 0.1;
/// <summary>
/// Weight for policy decision flip.

2
src/Scanner/__Libraries/StellaOps.Scanner.SmartDiff/Detection/RiskStateSnapshot.cs
View File

@@ -46,7 +46,7 @@ public sealed record RiskStateSnapshot(
builder.Append(PolicyDecision?.ToString() ?? "null");
var hash = SHA256.HashData(Encoding.UTF8.GetBytes(builder.ToString()));
return Convert.ToHexString(hash).ToLowerInvariant();
return "sha256:" + Convert.ToHexString(hash).ToLowerInvariant();
}
}

8
src/Scanner/__Libraries/StellaOps.Scanner.SmartDiff/Output/SarifModels.cs
View File

@@ -98,9 +98,9 @@ public sealed record SarifResult(
[property: JsonPropertyName("level")] SarifLevel Level,
[property: JsonPropertyName("message")] SarifMessage Message,
[property: JsonPropertyName("locations")] ImmutableArray<SarifLocation>? Locations = null,
[property: JsonPropertyName("fingerprints")] ImmutableDictionary<string, string>? Fingerprints = null,
[property: JsonPropertyName("partialFingerprints")] ImmutableDictionary<string, string>? PartialFingerprints = null,
[property: JsonPropertyName("properties")] ImmutableDictionary<string, object>? Properties = null);
[property: JsonPropertyName("fingerprints")] ImmutableSortedDictionary<string, string>? Fingerprints = null,
[property: JsonPropertyName("partialFingerprints")] ImmutableSortedDictionary<string, string>? PartialFingerprints = null,
[property: JsonPropertyName("properties")] ImmutableSortedDictionary<string, object>? Properties = null);
/// <summary>
/// Location of a result.
@@ -157,7 +157,7 @@ public sealed record SarifInvocation(
public sealed record SarifArtifact(
[property: JsonPropertyName("location")] SarifArtifactLocation Location,
[property: JsonPropertyName("mimeType")] string? MimeType = null,
[property: JsonPropertyName("hashes")] ImmutableDictionary<string, string>? Hashes = null);
[property: JsonPropertyName("hashes")] ImmutableSortedDictionary<string, string>? Hashes = null);
/// <summary>
/// Version control information.

18
src/Scanner/__Libraries/StellaOps.Scanner.SmartDiff/Output/SarifOutputGenerator.cs
View File

@@ -293,10 +293,10 @@ public sealed class SarifOutputGenerator
Level: level,
Message: new SarifMessage(message),
Locations: locations,
Fingerprints: ImmutableDictionary.CreateRange(new[]
Fingerprints: ImmutableSortedDictionary.CreateRange(StringComparer.Ordinal, new[]
{
KeyValuePair.Create("purl", change.ComponentPurl),
KeyValuePair.Create("vulnId", change.VulnId),
KeyValuePair.Create("purl", change.ComponentPurl)
}));
}
@@ -322,10 +322,10 @@ public sealed class SarifOutputGenerator
RuleId: "SDIFF003",
Level: SarifLevel.Note,
Message: new SarifMessage(message),
Fingerprints: ImmutableDictionary.CreateRange(new[]
Fingerprints: ImmutableSortedDictionary.CreateRange(StringComparer.Ordinal, new[]
{
KeyValuePair.Create("purl", candidate.ComponentPurl),
KeyValuePair.Create("vulnId", candidate.VulnId),
KeyValuePair.Create("purl", candidate.ComponentPurl)
}));
}
@@ -338,10 +338,10 @@ public sealed class SarifOutputGenerator
RuleId: "SDIFF004",
Level: SarifLevel.Warning,
Message: new SarifMessage(message),
Fingerprints: ImmutableDictionary.CreateRange(new[]
Fingerprints: ImmutableSortedDictionary.CreateRange(StringComparer.Ordinal, new[]
{
KeyValuePair.Create("purl", change.ComponentPurl),
KeyValuePair.Create("vulnId", change.VulnId),
KeyValuePair.Create("purl", change.ComponentPurl)
}));
}
@@ -350,15 +350,15 @@ public sealed class SarifOutputGenerator
return new SarifInvocation(
ExecutionSuccessful: true,
StartTimeUtc: input.ScanTime,
EndTimeUtc: DateTimeOffset.UtcNow);
EndTimeUtc: null);
}
private static ImmutableArray<SarifArtifact> CreateArtifacts(SmartDiffSarifInput input)
{
var artifacts = new List<SarifArtifact>();
// Collect unique file paths from results
var paths = new HashSet<string>();
// Collect unique file paths from results (sorted for determinism).
var paths = new SortedSet<string>(StringComparer.Ordinal);
foreach (var change in input.MaterialChanges)
{

2
src/Scanner/__Libraries/StellaOps.Scanner.Storage/Extensions/ServiceCollectionExtensions.cs
View File

@@ -79,6 +79,8 @@ public static class ServiceCollectionExtensions
services.AddScoped<IProofSpineRepository, PostgresProofSpineRepository>();
services.AddScoped<ICallGraphSnapshotRepository, PostgresCallGraphSnapshotRepository>();
services.AddScoped<IReachabilityResultRepository, PostgresReachabilityResultRepository>();
services.AddScoped<ICodeChangeRepository, PostgresCodeChangeRepository>();
services.AddScoped<IReachabilityDriftResultRepository, PostgresReachabilityDriftResultRepository>();
services.AddSingleton<IEntryTraceResultStore, EntryTraceResultStore>();
services.AddSingleton<IRubyPackageInventoryStore, RubyPackageInventoryStore>();
services.AddSingleton<IBunPackageInventoryStore, BunPackageInventoryStore>();

55
src/Scanner/__Libraries/StellaOps.Scanner.Storage/Postgres/Migrations/009_call_graph_tables.sql
View File

@@ -1,13 +1,21 @@
-- Call graph snapshots + reachability analysis results
-- Sprint: SPRINT_3600_0002_0001_call_graph_infrastructure
CREATE SCHEMA IF NOT EXISTS scanner;
-- Note: migrations are executed with the module schema as the active search_path.
-- Keep objects unqualified so integration tests can run in isolated schemas.
CREATE OR REPLACE FUNCTION current_tenant_id()
RETURNS UUID AS $$
BEGIN
RETURN NULLIF(current_setting('app.tenant_id', TRUE), '')::UUID;
END;
$$ LANGUAGE plpgsql STABLE;
-- -----------------------------------------------------------------------------
-- Table: scanner.call_graph_snapshots
-- Table: call_graph_snapshots
-- Purpose: Cache call graph snapshots per scan/language for reachability drift.
-- -----------------------------------------------------------------------------
CREATE TABLE IF NOT EXISTS scanner.call_graph_snapshots (
CREATE TABLE IF NOT EXISTS call_graph_snapshots (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
tenant_id UUID NOT NULL,
@@ -27,24 +35,26 @@ CREATE TABLE IF NOT EXISTS scanner.call_graph_snapshots (
);
CREATE INDEX IF NOT EXISTS idx_call_graph_snapshots_tenant_scan
ON scanner.call_graph_snapshots (tenant_id, scan_id, language);
ON call_graph_snapshots (tenant_id, scan_id, language);
CREATE INDEX IF NOT EXISTS idx_call_graph_snapshots_graph_digest
ON scanner.call_graph_snapshots (graph_digest);
ON call_graph_snapshots (graph_digest);
CREATE INDEX IF NOT EXISTS idx_call_graph_snapshots_extracted_at
ON scanner.call_graph_snapshots USING BRIN (extracted_at);
ON call_graph_snapshots USING BRIN (extracted_at);
ALTER TABLE scanner.call_graph_snapshots ENABLE ROW LEVEL SECURITY;
DROP POLICY IF EXISTS call_graph_snapshots_tenant_isolation ON scanner.call_graph_snapshots;
CREATE POLICY call_graph_snapshots_tenant_isolation ON scanner.call_graph_snapshots
USING (tenant_id = scanner.current_tenant_id());
ALTER TABLE call_graph_snapshots ENABLE ROW LEVEL SECURITY;
DROP POLICY IF EXISTS call_graph_snapshots_tenant_isolation ON call_graph_snapshots;
CREATE POLICY call_graph_snapshots_tenant_isolation ON call_graph_snapshots
FOR ALL
USING (tenant_id = current_tenant_id())
WITH CHECK (tenant_id = current_tenant_id());
COMMENT ON TABLE scanner.call_graph_snapshots IS 'Call graph snapshots per scan/language for reachability drift detection.';
COMMENT ON TABLE call_graph_snapshots IS 'Call graph snapshots per scan/language for reachability drift detection.';
-- -----------------------------------------------------------------------------
-- Table: scanner.reachability_results
-- Table: reachability_results
-- Purpose: Cache reachability BFS results (reachable sinks + shortest paths).
-- -----------------------------------------------------------------------------
CREATE TABLE IF NOT EXISTS scanner.reachability_results (
CREATE TABLE IF NOT EXISTS reachability_results (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
tenant_id UUID NOT NULL,
@@ -63,16 +73,17 @@ CREATE TABLE IF NOT EXISTS scanner.reachability_results (
);
CREATE INDEX IF NOT EXISTS idx_reachability_results_tenant_scan
ON scanner.reachability_results (tenant_id, scan_id, language);
ON reachability_results (tenant_id, scan_id, language);
CREATE INDEX IF NOT EXISTS idx_reachability_results_graph_digest
ON scanner.reachability_results (graph_digest);
ON reachability_results (graph_digest);
CREATE INDEX IF NOT EXISTS idx_reachability_results_computed_at
ON scanner.reachability_results USING BRIN (computed_at);
ON reachability_results USING BRIN (computed_at);
ALTER TABLE scanner.reachability_results ENABLE ROW LEVEL SECURITY;
DROP POLICY IF EXISTS reachability_results_tenant_isolation ON scanner.reachability_results;
CREATE POLICY reachability_results_tenant_isolation ON scanner.reachability_results
USING (tenant_id = scanner.current_tenant_id());
COMMENT ON TABLE scanner.reachability_results IS 'Reachability analysis results per scan/language with shortest paths.';
ALTER TABLE reachability_results ENABLE ROW LEVEL SECURITY;
DROP POLICY IF EXISTS reachability_results_tenant_isolation ON reachability_results;
CREATE POLICY reachability_results_tenant_isolation ON reachability_results
FOR ALL
USING (tenant_id = current_tenant_id())
WITH CHECK (tenant_id = current_tenant_id());
COMMENT ON TABLE reachability_results IS 'Reachability analysis results per scan/language with shortest paths.';

151
src/Scanner/__Libraries/StellaOps.Scanner.Storage/Postgres/Migrations/010_reachability_drift_tables.sql Normal file
View File

@@ -0,0 +1,151 @@
-- Reachability drift: code changes + drift results
-- Sprint: SPRINT_3600_0003_0001_drift_detection_engine
-- Note: migrations are executed with the module schema as the active search_path.
-- Keep objects unqualified so integration tests can run in isolated schemas.
CREATE OR REPLACE FUNCTION current_tenant_id()
RETURNS UUID AS $$
BEGIN
RETURN NULLIF(current_setting('app.tenant_id', TRUE), '')::UUID;
END;
$$ LANGUAGE plpgsql STABLE;
-- -----------------------------------------------------------------------------
-- Table: code_changes
-- Purpose: Store coarse code change facts extracted from call graph diffs.
-- -----------------------------------------------------------------------------
CREATE TABLE IF NOT EXISTS code_changes (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
tenant_id UUID NOT NULL,
scan_id TEXT NOT NULL,
base_scan_id TEXT NOT NULL,
language TEXT NOT NULL,
node_id TEXT,
file TEXT NOT NULL,
symbol TEXT NOT NULL,
change_kind TEXT NOT NULL,
details JSONB,
detected_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
CONSTRAINT code_changes_unique UNIQUE (tenant_id, scan_id, base_scan_id, language, symbol, change_kind)
);
CREATE INDEX IF NOT EXISTS idx_code_changes_tenant_scan
ON code_changes (tenant_id, scan_id, base_scan_id, language);
CREATE INDEX IF NOT EXISTS idx_code_changes_symbol
ON code_changes (symbol);
CREATE INDEX IF NOT EXISTS idx_code_changes_kind
ON code_changes (change_kind);
CREATE INDEX IF NOT EXISTS idx_code_changes_detected_at
ON code_changes USING BRIN (detected_at);
ALTER TABLE code_changes ENABLE ROW LEVEL SECURITY;
DROP POLICY IF EXISTS code_changes_tenant_isolation ON code_changes;
CREATE POLICY code_changes_tenant_isolation ON code_changes
FOR ALL
USING (tenant_id = current_tenant_id())
WITH CHECK (tenant_id = current_tenant_id());
COMMENT ON TABLE code_changes IS 'Code change facts for reachability drift analysis.';
-- -----------------------------------------------------------------------------
-- Extend: material_risk_changes
-- Purpose: Store drift-specific attachments alongside Smart-Diff R1 changes.
-- -----------------------------------------------------------------------------
ALTER TABLE material_risk_changes
ADD COLUMN IF NOT EXISTS base_scan_id TEXT,
ADD COLUMN IF NOT EXISTS cause TEXT,
ADD COLUMN IF NOT EXISTS cause_kind TEXT,
ADD COLUMN IF NOT EXISTS path_nodes JSONB,
ADD COLUMN IF NOT EXISTS associated_vulns JSONB;
CREATE INDEX IF NOT EXISTS idx_material_risk_changes_cause_kind
ON material_risk_changes(cause_kind)
WHERE cause_kind IS NOT NULL;
CREATE INDEX IF NOT EXISTS idx_material_risk_changes_base_scan
ON material_risk_changes(base_scan_id)
WHERE base_scan_id IS NOT NULL;
-- -----------------------------------------------------------------------------
-- Table: reachability_drift_results
-- Purpose: Aggregate drift results per scan pair and language.
-- -----------------------------------------------------------------------------
CREATE TABLE IF NOT EXISTS reachability_drift_results (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
tenant_id UUID NOT NULL,
base_scan_id TEXT NOT NULL,
head_scan_id TEXT NOT NULL,
language TEXT NOT NULL,
newly_reachable_count INT NOT NULL DEFAULT 0,
newly_unreachable_count INT NOT NULL DEFAULT 0,
detected_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
result_digest TEXT NOT NULL,
CONSTRAINT reachability_drift_unique UNIQUE (tenant_id, base_scan_id, head_scan_id, language, result_digest)
);
CREATE INDEX IF NOT EXISTS idx_reachability_drift_head
ON reachability_drift_results (tenant_id, head_scan_id, language);
CREATE INDEX IF NOT EXISTS idx_reachability_drift_detected_at
ON reachability_drift_results USING BRIN (detected_at);
ALTER TABLE reachability_drift_results ENABLE ROW LEVEL SECURITY;
DROP POLICY IF EXISTS drift_results_tenant_isolation ON reachability_drift_results;
CREATE POLICY drift_results_tenant_isolation ON reachability_drift_results
FOR ALL
USING (tenant_id = current_tenant_id())
WITH CHECK (tenant_id = current_tenant_id());
COMMENT ON TABLE reachability_drift_results IS 'Aggregate drift results per scan pair + language.';
-- -----------------------------------------------------------------------------
-- Table: drifted_sinks
-- Purpose: Individual sink drift records (paged by API).
-- -----------------------------------------------------------------------------
CREATE TABLE IF NOT EXISTS drifted_sinks (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
tenant_id UUID NOT NULL,
drift_result_id UUID NOT NULL REFERENCES reachability_drift_results(id) ON DELETE CASCADE,
sink_node_id TEXT NOT NULL,
symbol TEXT NOT NULL,
sink_category TEXT NOT NULL,
direction TEXT NOT NULL,
cause_kind TEXT NOT NULL,
cause_description TEXT NOT NULL,
cause_symbol TEXT,
cause_file TEXT,
cause_line INT,
code_change_id UUID REFERENCES code_changes(id),
compressed_path JSONB NOT NULL,
associated_vulns JSONB,
CONSTRAINT drifted_sinks_unique UNIQUE (drift_result_id, sink_node_id)
);
CREATE INDEX IF NOT EXISTS idx_drifted_sinks_drift
ON drifted_sinks (drift_result_id);
CREATE INDEX IF NOT EXISTS idx_drifted_sinks_direction
ON drifted_sinks (direction);
CREATE INDEX IF NOT EXISTS idx_drifted_sinks_category
ON drifted_sinks (sink_category);
ALTER TABLE drifted_sinks ENABLE ROW LEVEL SECURITY;
DROP POLICY IF EXISTS drifted_sinks_tenant_isolation ON drifted_sinks;
CREATE POLICY drifted_sinks_tenant_isolation ON drifted_sinks
FOR ALL
USING (tenant_id = current_tenant_id())
WITH CHECK (tenant_id = current_tenant_id());
COMMENT ON TABLE drifted_sinks IS 'Individual drifted sink records with causes and compressed paths.';

23
src/Scanner/__Libraries/StellaOps.Scanner.Storage/Postgres/Migrations/010_scanner_api_ingestion.sql Normal file
View File

@@ -0,0 +1,23 @@
-- scanner api ingestion persistence (startup migration)
-- Purpose: Store idempotency state for Scanner.WebService ingestion endpoints.
CREATE TABLE IF NOT EXISTS callgraph_ingestions (
id TEXT PRIMARY KEY,
tenant_id UUID NOT NULL,
scan_id TEXT NOT NULL,
content_digest TEXT NOT NULL,
language TEXT NOT NULL,
node_count INT NOT NULL,
edge_count INT NOT NULL,
created_at_utc TIMESTAMPTZ NOT NULL DEFAULT NOW(),
callgraph_json JSONB NOT NULL,
CONSTRAINT callgraph_ingestions_unique_per_scan UNIQUE (tenant_id, scan_id, content_digest)
);
CREATE INDEX IF NOT EXISTS ix_callgraph_ingestions_scan
ON callgraph_ingestions (tenant_id, scan_id, created_at_utc DESC, id);
CREATE INDEX IF NOT EXISTS ix_callgraph_ingestions_digest
ON callgraph_ingestions (tenant_id, content_digest);

12
src/Scanner/__Libraries/StellaOps.Scanner.Storage/Postgres/Migrations/010_smart_diff_priority_score_widen.sql Normal file
View File

@@ -0,0 +1,12 @@
-- =============================================================================
-- 010_smart_diff_priority_score_widen.sql
-- Purpose: Widen Smart-Diff material risk change priority_score to support
-- advisory scoring formula (can exceed NUMERIC(6,4)).
--
-- Note: migrations are executed inside a transaction by the migration runner.
-- Do not include BEGIN/COMMIT in migration files.
-- =============================================================================
ALTER TABLE material_risk_changes
ALTER COLUMN priority_score TYPE NUMERIC(12, 4)
USING priority_score::NUMERIC(12, 4);

1
src/Scanner/__Libraries/StellaOps.Scanner.Storage/Postgres/Migrations/MigrationIds.cs
View File

@@ -11,4 +11,5 @@ internal static class MigrationIds
public const string UnknownsRankingContainment = "007_unknowns_ranking_containment.sql";
public const string EpssIntegration = "008_epss_integration.sql";
public const string CallGraphTables = "009_call_graph_tables.sql";
public const string ReachabilityDriftTables = "010_reachability_drift_tables.sql";
}

28
src/Scanner/__Libraries/StellaOps.Scanner.Storage/Postgres/PostgresCallGraphSnapshotRepository.cs
View File

@@ -8,6 +8,9 @@ namespace StellaOps.Scanner.Storage.Postgres;
public sealed class PostgresCallGraphSnapshotRepository : ICallGraphSnapshotRepository
{
private const string TenantContext = "00000000-0000-0000-0000-000000000001";
private static readonly Guid TenantId = Guid.Parse(TenantContext);
private static readonly JsonSerializerOptions JsonOptions = new(JsonSerializerDefaults.Web)
{
WriteIndented = false
@@ -16,6 +19,9 @@ public sealed class PostgresCallGraphSnapshotRepository : ICallGraphSnapshotRepo
private readonly ScannerDataSource _dataSource;
private readonly ILogger<PostgresCallGraphSnapshotRepository> _logger;
private string SchemaName => _dataSource.SchemaName ?? ScannerDataSource.DefaultSchema;
private string CallGraphSnapshotsTable => $"{SchemaName}.call_graph_snapshots";
public PostgresCallGraphSnapshotRepository(
ScannerDataSource dataSource,
ILogger<PostgresCallGraphSnapshotRepository> logger)
@@ -29,8 +35,8 @@ public sealed class PostgresCallGraphSnapshotRepository : ICallGraphSnapshotRepo
ArgumentNullException.ThrowIfNull(snapshot);
var trimmed = snapshot.Trimmed();
const string sql = """
INSERT INTO scanner.call_graph_snapshots (
var sql = $"""
INSERT INTO {CallGraphSnapshotsTable} (
tenant_id,
scan_id,
language,
@@ -63,12 +69,11 @@ public sealed class PostgresCallGraphSnapshotRepository : ICallGraphSnapshotRepo
""";
var json = JsonSerializer.Serialize(trimmed, JsonOptions);
var tenantId = GetCurrentTenantId();
await using var connection = await _dataSource.OpenConnectionAsync(ct).ConfigureAwait(false);
await using var connection = await _dataSource.OpenConnectionAsync(TenantContext, ct).ConfigureAwait(false);
await connection.ExecuteAsync(new CommandDefinition(sql, new
{
TenantId = tenantId,
TenantId = TenantId,
ScanId = trimmed.ScanId,
Language = trimmed.Language,
GraphDigest = trimmed.GraphDigest,
@@ -93,18 +98,18 @@ public sealed class PostgresCallGraphSnapshotRepository : ICallGraphSnapshotRepo
ArgumentException.ThrowIfNullOrWhiteSpace(scanId);
ArgumentException.ThrowIfNullOrWhiteSpace(language);
const string sql = """
var sql = $"""
SELECT snapshot_json
FROM scanner.call_graph_snapshots
FROM {CallGraphSnapshotsTable}
WHERE tenant_id = @TenantId AND scan_id = @ScanId AND language = @Language
ORDER BY extracted_at DESC
LIMIT 1
""";
await using var connection = await _dataSource.OpenConnectionAsync(ct).ConfigureAwait(false);
await using var connection = await _dataSource.OpenConnectionAsync(TenantContext, ct).ConfigureAwait(false);
var json = await connection.ExecuteScalarAsync<string?>(new CommandDefinition(sql, new
{
TenantId = GetCurrentTenantId(),
TenantId = TenantId,
ScanId = scanId,
Language = language
}, cancellationToken: ct)).ConfigureAwait(false);
@@ -116,10 +121,5 @@ public sealed class PostgresCallGraphSnapshotRepository : ICallGraphSnapshotRepo
return JsonSerializer.Deserialize<CallGraphSnapshot>(json, JsonOptions);
}
private static Guid GetCurrentTenantId()
{
return Guid.Parse("00000000-0000-0000-0000-000000000001");
}
}

114
src/Scanner/__Libraries/StellaOps.Scanner.Storage/Postgres/PostgresCodeChangeRepository.cs Normal file
View File

@@ -0,0 +1,114 @@
using System.Text.Json;
using Dapper;
using Microsoft.Extensions.Logging;
using StellaOps.Scanner.ReachabilityDrift;
using StellaOps.Scanner.Storage.Repositories;
namespace StellaOps.Scanner.Storage.Postgres;
public sealed class PostgresCodeChangeRepository : ICodeChangeRepository
{
private const string TenantContext = "00000000-0000-0000-0000-000000000001";
private static readonly Guid TenantId = Guid.Parse(TenantContext);
private readonly ScannerDataSource _dataSource;
private readonly ILogger<PostgresCodeChangeRepository> _logger;
private string SchemaName => _dataSource.SchemaName ?? ScannerDataSource.DefaultSchema;
private string CodeChangesTable => $"{SchemaName}.code_changes";
public PostgresCodeChangeRepository(
ScannerDataSource dataSource,
ILogger<PostgresCodeChangeRepository> logger)
{
_dataSource = dataSource ?? throw new ArgumentNullException(nameof(dataSource));
_logger = logger ?? throw new ArgumentNullException(nameof(logger));
}
public async Task StoreAsync(IReadOnlyList<CodeChangeFact> changes, CancellationToken ct = default)
{
ArgumentNullException.ThrowIfNull(changes);
if (changes.Count == 0)
{
return;
}
var sql = $"""
INSERT INTO {CodeChangesTable} (
id,
tenant_id,
scan_id,
base_scan_id,
language,
node_id,
file,
symbol,
change_kind,
details,
detected_at
) VALUES (
@Id,
@TenantId,
@ScanId,
@BaseScanId,
@Language,
@NodeId,
@File,
@Symbol,
@ChangeKind,
@Details::jsonb,
@DetectedAt
)
ON CONFLICT (tenant_id, scan_id, base_scan_id, language, symbol, change_kind) DO UPDATE SET
node_id = EXCLUDED.node_id,
file = EXCLUDED.file,
details = EXCLUDED.details,
detected_at = EXCLUDED.detected_at
""";
var rows = changes.Select(change => new
{
change.Id,
TenantId,
ScanId = change.ScanId.Trim(),
BaseScanId = change.BaseScanId.Trim(),
Language = change.Language.Trim(),
NodeId = string.IsNullOrWhiteSpace(change.NodeId) ? null : change.NodeId.Trim(),
File = change.File.Trim(),
Symbol = change.Symbol.Trim(),
ChangeKind = ToDbValue(change.Kind),
Details = SerializeDetails(change.Details),
DetectedAt = change.DetectedAt.UtcDateTime
}).ToList();
await using var connection = await _dataSource.OpenConnectionAsync(TenantContext, ct).ConfigureAwait(false);
await connection.ExecuteAsync(new CommandDefinition(sql, rows, cancellationToken: ct)).ConfigureAwait(false);
_logger.LogDebug(
"Stored {Count} code change facts scan={ScanId} base={BaseScanId} lang={Language}",
changes.Count,
changes[0].ScanId,
changes[0].BaseScanId,
changes[0].Language);
}
private static string? SerializeDetails(JsonElement? details)
=> details is { ValueKind: not JsonValueKind.Undefined and not JsonValueKind.Null }
? details.Value.GetRawText()
: null;
private static string ToDbValue(CodeChangeKind kind)
{
return kind switch
{
CodeChangeKind.Added => "added",
CodeChangeKind.Removed => "removed",
CodeChangeKind.SignatureChanged => "signature_changed",
CodeChangeKind.GuardChanged => "guard_changed",
CodeChangeKind.DependencyChanged => "dependency_changed",
CodeChangeKind.VisibilityChanged => "visibility_changed",
_ => kind.ToString()
};
}
}

527
src/Scanner/__Libraries/StellaOps.Scanner.Storage/Postgres/PostgresReachabilityDriftResultRepository.cs Normal file
View File

@@ -0,0 +1,527 @@
using System.Collections.Immutable;
using System.Text.Json;
using Dapper;
using Microsoft.Extensions.Logging;
using StellaOps.Scanner.Reachability;
using StellaOps.Scanner.ReachabilityDrift;
using StellaOps.Scanner.Storage.Repositories;
namespace StellaOps.Scanner.Storage.Postgres;
public sealed class PostgresReachabilityDriftResultRepository : IReachabilityDriftResultRepository
{
private const string TenantContext = "00000000-0000-0000-0000-000000000001";
private static readonly Guid TenantId = Guid.Parse(TenantContext);
private static readonly JsonSerializerOptions JsonOptions = new(JsonSerializerDefaults.Web)
{
WriteIndented = false
};
private readonly ScannerDataSource _dataSource;
private readonly ILogger<PostgresReachabilityDriftResultRepository> _logger;
private string SchemaName => _dataSource.SchemaName ?? ScannerDataSource.DefaultSchema;
private string DriftResultsTable => $"{SchemaName}.reachability_drift_results";
private string DriftedSinksTable => $"{SchemaName}.drifted_sinks";
public PostgresReachabilityDriftResultRepository(
ScannerDataSource dataSource,
ILogger<PostgresReachabilityDriftResultRepository> logger)
{
_dataSource = dataSource ?? throw new ArgumentNullException(nameof(dataSource));
_logger = logger ?? throw new ArgumentNullException(nameof(logger));
}
public async Task StoreAsync(ReachabilityDriftResult result, CancellationToken ct = default)
{
ArgumentNullException.ThrowIfNull(result);
var insertResultSql = $"""
INSERT INTO {DriftResultsTable} (
id,
tenant_id,
base_scan_id,
head_scan_id,
language,
newly_reachable_count,
newly_unreachable_count,
detected_at,
result_digest
) VALUES (
@Id,
@TenantId,
@BaseScanId,
@HeadScanId,
@Language,
@NewlyReachableCount,
@NewlyUnreachableCount,
@DetectedAt,
@ResultDigest
)
ON CONFLICT (tenant_id, base_scan_id, head_scan_id, language, result_digest) DO UPDATE SET
newly_reachable_count = EXCLUDED.newly_reachable_count,
newly_unreachable_count = EXCLUDED.newly_unreachable_count,
detected_at = EXCLUDED.detected_at
RETURNING id
""";
var deleteSinksSql = $"""
DELETE FROM {DriftedSinksTable}
WHERE tenant_id = @TenantId AND drift_result_id = @DriftId
""";
var insertSinkSql = $"""
INSERT INTO {DriftedSinksTable} (
id,
tenant_id,
drift_result_id,
sink_node_id,
symbol,
sink_category,
direction,
cause_kind,
cause_description,
cause_symbol,
cause_file,
cause_line,
code_change_id,
compressed_path,
associated_vulns
) VALUES (
@Id,
@TenantId,
@DriftId,
@SinkNodeId,
@Symbol,
@SinkCategory,
@Direction,
@CauseKind,
@CauseDescription,
@CauseSymbol,
@CauseFile,
@CauseLine,
@CodeChangeId,
@CompressedPath::jsonb,
@AssociatedVulns::jsonb
)
ON CONFLICT (drift_result_id, sink_node_id) DO UPDATE SET
symbol = EXCLUDED.symbol,
sink_category = EXCLUDED.sink_category,
direction = EXCLUDED.direction,
cause_kind = EXCLUDED.cause_kind,
cause_description = EXCLUDED.cause_description,
cause_symbol = EXCLUDED.cause_symbol,
cause_file = EXCLUDED.cause_file,
cause_line = EXCLUDED.cause_line,
code_change_id = EXCLUDED.code_change_id,
compressed_path = EXCLUDED.compressed_path,
associated_vulns = EXCLUDED.associated_vulns
""";
await using var connection = await _dataSource.OpenConnectionAsync(TenantContext, ct).ConfigureAwait(false);
await using var transaction = await connection.BeginTransactionAsync(ct).ConfigureAwait(false);
try
{
var driftId = await connection.ExecuteScalarAsync<Guid>(new CommandDefinition(
insertResultSql,
new
{
result.Id,
TenantId,
BaseScanId = result.BaseScanId.Trim(),
HeadScanId = result.HeadScanId.Trim(),
Language = result.Language.Trim(),
NewlyReachableCount = result.NewlyReachable.Length,
NewlyUnreachableCount = result.NewlyUnreachable.Length,
DetectedAt = result.DetectedAt.UtcDateTime,
result.ResultDigest
},
transaction: transaction,
cancellationToken: ct))
.ConfigureAwait(false);
await connection.ExecuteAsync(new CommandDefinition(
deleteSinksSql,
new { TenantId, DriftId = driftId },
transaction: transaction,
cancellationToken: ct))
.ConfigureAwait(false);
var sinkRows = EnumerateSinkRows(driftId, result.NewlyReachable, DriftDirection.BecameReachable)
.Concat(EnumerateSinkRows(driftId, result.NewlyUnreachable, DriftDirection.BecameUnreachable))
.ToList();
if (sinkRows.Count > 0)
{
await connection.ExecuteAsync(new CommandDefinition(
insertSinkSql,
sinkRows,
transaction: transaction,
cancellationToken: ct))
.ConfigureAwait(false);
}
await transaction.CommitAsync(ct).ConfigureAwait(false);
_logger.LogDebug(
"Stored drift result drift={DriftId} base={BaseScanId} head={HeadScanId} lang={Language}",
driftId,
result.BaseScanId,
result.HeadScanId,
result.Language);
}
catch (Exception ex)
{
_logger.LogError(ex, "Failed to store drift result base={BaseScanId} head={HeadScanId}", result.BaseScanId, result.HeadScanId);
await transaction.RollbackAsync(ct).ConfigureAwait(false);
throw;
}
}
public async Task<ReachabilityDriftResult?> TryGetLatestForHeadAsync(string headScanId, string language, CancellationToken ct = default)
{
ArgumentException.ThrowIfNullOrWhiteSpace(headScanId);
ArgumentException.ThrowIfNullOrWhiteSpace(language);
var sql = $"""
SELECT id, base_scan_id, head_scan_id, language, detected_at, result_digest
FROM {DriftResultsTable}
WHERE tenant_id = @TenantId AND head_scan_id = @HeadScanId AND language = @Language
ORDER BY detected_at DESC
LIMIT 1
""";
await using var connection = await _dataSource.OpenConnectionAsync(TenantContext, ct).ConfigureAwait(false);
var header = await connection.QuerySingleOrDefaultAsync<DriftHeaderRow>(new CommandDefinition(
sql,
new
{
TenantId,
HeadScanId = headScanId.Trim(),
Language = language.Trim()
},
cancellationToken: ct)).ConfigureAwait(false);
if (header is null)
{
return null;
}
return await LoadResultAsync(connection, header, ct).ConfigureAwait(false);
}
public async Task<ReachabilityDriftResult?> TryGetByIdAsync(Guid driftId, CancellationToken ct = default)
{
var sql = $"""
SELECT id, base_scan_id, head_scan_id, language, detected_at, result_digest
FROM {DriftResultsTable}
WHERE tenant_id = @TenantId AND id = @DriftId
LIMIT 1
""";
await using var connection = await _dataSource.OpenConnectionAsync(TenantContext, ct).ConfigureAwait(false);
var header = await connection.QuerySingleOrDefaultAsync<DriftHeaderRow>(new CommandDefinition(
sql,
new
{
TenantId,
DriftId = driftId
},
cancellationToken: ct)).ConfigureAwait(false);
if (header is null)
{
return null;
}
return await LoadResultAsync(connection, header, ct).ConfigureAwait(false);
}
public async Task<bool> ExistsAsync(Guid driftId, CancellationToken ct = default)
{
var sql = $"""
SELECT 1
FROM {DriftResultsTable}
WHERE tenant_id = @TenantId AND id = @DriftId
LIMIT 1
""";
await using var connection = await _dataSource.OpenConnectionAsync(TenantContext, ct).ConfigureAwait(false);
var result = await connection.ExecuteScalarAsync<int?>(new CommandDefinition(
sql,
new { TenantId, DriftId = driftId },
cancellationToken: ct)).ConfigureAwait(false);
return result is not null;
}
public async Task<IReadOnlyList<DriftedSink>> ListSinksAsync(
Guid driftId,
DriftDirection direction,
int offset,
int limit,
CancellationToken ct = default)
{
if (offset < 0)
{
throw new ArgumentOutOfRangeException(nameof(offset));
}
if (limit <= 0)
{
throw new ArgumentOutOfRangeException(nameof(limit));
}
var sql = $"""
SELECT
id,
sink_node_id,
symbol,
sink_category,
direction,
cause_kind,
cause_description,
cause_symbol,
cause_file,
cause_line,
code_change_id,
compressed_path,
associated_vulns
FROM {DriftedSinksTable}
WHERE tenant_id = @TenantId AND drift_result_id = @DriftId AND direction = @Direction
ORDER BY sink_node_id ASC
OFFSET @Offset LIMIT @Limit
""";
await using var connection = await _dataSource.OpenConnectionAsync(TenantContext, ct).ConfigureAwait(false);
var rows = await connection.QueryAsync<DriftSinkRow>(new CommandDefinition(
sql,
new
{
TenantId,
DriftId = driftId,
Direction = ToDbValue(direction),
Offset = offset,
Limit = limit
},
cancellationToken: ct)).ConfigureAwait(false);
return rows.Select(r => r.ToModel(direction)).ToList();
}
private static IEnumerable<object> EnumerateSinkRows(Guid driftId, ImmutableArray<DriftedSink> sinks, DriftDirection direction)
{
foreach (var sink in sinks)
{
var pathJson = JsonSerializer.Serialize(sink.Path, JsonOptions);
var vulnsJson = sink.AssociatedVulns.IsDefaultOrEmpty
? null
: JsonSerializer.Serialize(sink.AssociatedVulns, JsonOptions);
yield return new
{
sink.Id,
TenantId,
DriftId = driftId,
SinkNodeId = sink.SinkNodeId,
Symbol = sink.Symbol,
SinkCategory = ToDbValue(sink.SinkCategory),
Direction = ToDbValue(direction),
CauseKind = ToDbValue(sink.Cause.Kind),
CauseDescription = sink.Cause.Description,
CauseSymbol = sink.Cause.ChangedSymbol,
CauseFile = sink.Cause.ChangedFile,
CauseLine = sink.Cause.ChangedLine,
CodeChangeId = sink.Cause.CodeChangeId,
CompressedPath = pathJson,
AssociatedVulns = vulnsJson
};
}
}
private async Task<ReachabilityDriftResult> LoadResultAsync(
System.Data.IDbConnection connection,
DriftHeaderRow header,
CancellationToken ct)
{
var sinksSql = $"""
SELECT
id,
sink_node_id,
symbol,
sink_category,
direction,
cause_kind,
cause_description,
cause_symbol,
cause_file,
cause_line,
code_change_id,
compressed_path,
associated_vulns
FROM {DriftedSinksTable}
WHERE tenant_id = @TenantId AND drift_result_id = @DriftId
ORDER BY direction ASC, sink_node_id ASC
""";
var rows = (await connection.QueryAsync<DriftSinkRow>(new CommandDefinition(
sinksSql,
new { TenantId, DriftId = header.id },
cancellationToken: ct)).ConfigureAwait(false)).ToList();
var reachable = rows
.Where(r => string.Equals(r.direction, ToDbValue(DriftDirection.BecameReachable), StringComparison.Ordinal))
.Select(r => r.ToModel(DriftDirection.BecameReachable))
.OrderBy(s => s.SinkNodeId, StringComparer.Ordinal)
.ToImmutableArray();
var unreachable = rows
.Where(r => string.Equals(r.direction, ToDbValue(DriftDirection.BecameUnreachable), StringComparison.Ordinal))
.Select(r => r.ToModel(DriftDirection.BecameUnreachable))
.OrderBy(s => s.SinkNodeId, StringComparer.Ordinal)
.ToImmutableArray();
return new ReachabilityDriftResult
{
Id = header.id,
BaseScanId = header.base_scan_id,
HeadScanId = header.head_scan_id,
Language = header.language,
DetectedAt = header.detected_at,
NewlyReachable = reachable,
NewlyUnreachable = unreachable,
ResultDigest = header.result_digest
};
}
private static string ToDbValue(DriftDirection direction)
=> direction == DriftDirection.BecameReachable ? "became_reachable" : "became_unreachable";
private static string ToDbValue(DriftCauseKind kind)
{
return kind switch
{
DriftCauseKind.GuardRemoved => "guard_removed",
DriftCauseKind.GuardAdded => "guard_added",
DriftCauseKind.NewPublicRoute => "new_public_route",
DriftCauseKind.VisibilityEscalated => "visibility_escalated",
DriftCauseKind.DependencyUpgraded => "dependency_upgraded",
DriftCauseKind.SymbolRemoved => "symbol_removed",
_ => "unknown"
};
}
private static string ToDbValue(SinkCategory category)
{
return category switch
{
SinkCategory.CmdExec => "CMD_EXEC",
SinkCategory.UnsafeDeser => "UNSAFE_DESER",
SinkCategory.SqlRaw => "SQL_RAW",
SinkCategory.Ssrf => "SSRF",
SinkCategory.FileWrite => "FILE_WRITE",
SinkCategory.PathTraversal => "PATH_TRAVERSAL",
SinkCategory.TemplateInjection => "TEMPLATE_INJECTION",
SinkCategory.CryptoWeak => "CRYPTO_WEAK",
SinkCategory.AuthzBypass => "AUTHZ_BYPASS",
_ => category.ToString()
};
}
private static DriftCauseKind ParseCauseKind(string value)
{
return value.Trim().ToLowerInvariant() switch
{
"guard_removed" => DriftCauseKind.GuardRemoved,
"guard_added" => DriftCauseKind.GuardAdded,
"new_public_route" => DriftCauseKind.NewPublicRoute,
"visibility_escalated" => DriftCauseKind.VisibilityEscalated,
"dependency_upgraded" => DriftCauseKind.DependencyUpgraded,
"symbol_removed" => DriftCauseKind.SymbolRemoved,
_ => DriftCauseKind.Unknown
};
}
private static SinkCategory ParseSinkCategory(string value)
{
return value.Trim().ToUpperInvariant() switch
{
"CMD_EXEC" => SinkCategory.CmdExec,
"UNSAFE_DESER" => SinkCategory.UnsafeDeser,
"SQL_RAW" => SinkCategory.SqlRaw,
"SSRF" => SinkCategory.Ssrf,
"FILE_WRITE" => SinkCategory.FileWrite,
"PATH_TRAVERSAL" => SinkCategory.PathTraversal,
"TEMPLATE_INJECTION" => SinkCategory.TemplateInjection,
"CRYPTO_WEAK" => SinkCategory.CryptoWeak,
"AUTHZ_BYPASS" => SinkCategory.AuthzBypass,
_ => SinkCategory.CmdExec
};
}
private sealed class DriftHeaderRow
{
public Guid id { get; init; }
public string base_scan_id { get; init; } = string.Empty;
public string head_scan_id { get; init; } = string.Empty;
public string language { get; init; } = string.Empty;
public DateTimeOffset detected_at { get; init; }
public string result_digest { get; init; } = string.Empty;
}
private sealed class DriftSinkRow
{
public Guid id { get; init; }
public string sink_node_id { get; init; } = string.Empty;
public string symbol { get; init; } = string.Empty;
public string sink_category { get; init; } = string.Empty;
public string direction { get; init; } = string.Empty;
public string cause_kind { get; init; } = string.Empty;
public string cause_description { get; init; } = string.Empty;
public string? cause_symbol { get; init; }
public string? cause_file { get; init; }
public int? cause_line { get; init; }
public Guid? code_change_id { get; init; }
public string compressed_path { get; init; } = "{}";
public string? associated_vulns { get; init; }
public DriftedSink ToModel(DriftDirection direction)
{
var path = JsonSerializer.Deserialize<CompressedPath>(compressed_path, JsonOptions)
?? new CompressedPath
{
Entrypoint = new PathNode { NodeId = string.Empty, Symbol = string.Empty },
Sink = new PathNode { NodeId = string.Empty, Symbol = string.Empty },
IntermediateCount = 0,
KeyNodes = ImmutableArray<PathNode>.Empty
};
var vulns = string.IsNullOrWhiteSpace(associated_vulns)
? ImmutableArray<AssociatedVuln>.Empty
: (JsonSerializer.Deserialize<AssociatedVuln[]>(associated_vulns!, JsonOptions) ?? [])
.ToImmutableArray();
return new DriftedSink
{
Id = id,
SinkNodeId = sink_node_id,
Symbol = symbol,
SinkCategory = ParseSinkCategory(sink_category),
Direction = direction,
Cause = new DriftCause
{
Kind = ParseCauseKind(cause_kind),
Description = cause_description,
ChangedSymbol = cause_symbol,
ChangedFile = cause_file,
ChangedLine = cause_line,
CodeChangeId = code_change_id
},
Path = path,
AssociatedVulns = vulns
};
}
}
}

28
src/Scanner/__Libraries/StellaOps.Scanner.Storage/Postgres/PostgresReachabilityResultRepository.cs
View File

@@ -8,6 +8,9 @@ namespace StellaOps.Scanner.Storage.Postgres;
public sealed class PostgresReachabilityResultRepository : IReachabilityResultRepository
{
private const string TenantContext = "00000000-0000-0000-0000-000000000001";
private static readonly Guid TenantId = Guid.Parse(TenantContext);
private static readonly JsonSerializerOptions JsonOptions = new(JsonSerializerDefaults.Web)
{
WriteIndented = false
@@ -16,6 +19,9 @@ public sealed class PostgresReachabilityResultRepository : IReachabilityResultRe
private readonly ScannerDataSource _dataSource;
private readonly ILogger<PostgresReachabilityResultRepository> _logger;
private string SchemaName => _dataSource.SchemaName ?? ScannerDataSource.DefaultSchema;
private string ReachabilityResultsTable => $"{SchemaName}.reachability_results";
public PostgresReachabilityResultRepository(
ScannerDataSource dataSource,
ILogger<PostgresReachabilityResultRepository> logger)
@@ -29,8 +35,8 @@ public sealed class PostgresReachabilityResultRepository : IReachabilityResultRe
ArgumentNullException.ThrowIfNull(result);
var trimmed = result.Trimmed();
const string sql = """
INSERT INTO scanner.reachability_results (
var sql = $"""
INSERT INTO {ReachabilityResultsTable} (
tenant_id,
scan_id,
language,
@@ -59,12 +65,11 @@ public sealed class PostgresReachabilityResultRepository : IReachabilityResultRe
""";
var json = JsonSerializer.Serialize(trimmed, JsonOptions);
var tenantId = GetCurrentTenantId();
await using var connection = await _dataSource.OpenConnectionAsync(ct).ConfigureAwait(false);
await using var connection = await _dataSource.OpenConnectionAsync(TenantContext, ct).ConfigureAwait(false);
await connection.ExecuteAsync(new CommandDefinition(sql, new
{
TenantId = tenantId,
TenantId = TenantId,
ScanId = trimmed.ScanId,
Language = trimmed.Language,
GraphDigest = trimmed.GraphDigest,
@@ -87,18 +92,18 @@ public sealed class PostgresReachabilityResultRepository : IReachabilityResultRe
ArgumentException.ThrowIfNullOrWhiteSpace(scanId);
ArgumentException.ThrowIfNullOrWhiteSpace(language);
const string sql = """
var sql = $"""
SELECT result_json
FROM scanner.reachability_results
FROM {ReachabilityResultsTable}
WHERE tenant_id = @TenantId AND scan_id = @ScanId AND language = @Language
ORDER BY computed_at DESC
LIMIT 1
""";
await using var connection = await _dataSource.OpenConnectionAsync(ct).ConfigureAwait(false);
await using var connection = await _dataSource.OpenConnectionAsync(TenantContext, ct).ConfigureAwait(false);
var json = await connection.ExecuteScalarAsync<string?>(new CommandDefinition(sql, new
{
TenantId = GetCurrentTenantId(),
TenantId = TenantId,
ScanId = scanId,
Language = language
}, cancellationToken: ct)).ConfigureAwait(false);
@@ -110,10 +115,5 @@ public sealed class PostgresReachabilityResultRepository : IReachabilityResultRe
return JsonSerializer.Deserialize<ReachabilityAnalysisResult>(json, JsonOptions);
}
private static Guid GetCurrentTenantId()
{
return Guid.Parse("00000000-0000-0000-0000-000000000001");
}
}

9
src/Scanner/__Libraries/StellaOps.Scanner.Storage/Repositories/ICodeChangeRepository.cs Normal file
View File

@@ -0,0 +1,9 @@
using StellaOps.Scanner.ReachabilityDrift;
namespace StellaOps.Scanner.Storage.Repositories;
public interface ICodeChangeRepository
{
Task StoreAsync(IReadOnlyList<CodeChangeFact> changes, CancellationToken ct = default);
}

21
src/Scanner/__Libraries/StellaOps.Scanner.Storage/Repositories/IReachabilityDriftResultRepository.cs Normal file
View File

@@ -0,0 +1,21 @@
using StellaOps.Scanner.ReachabilityDrift;
namespace StellaOps.Scanner.Storage.Repositories;
public interface IReachabilityDriftResultRepository
{
Task StoreAsync(ReachabilityDriftResult result, CancellationToken ct = default);
Task<ReachabilityDriftResult?> TryGetLatestForHeadAsync(string headScanId, string language, CancellationToken ct = default);
Task<ReachabilityDriftResult?> TryGetByIdAsync(Guid driftId, CancellationToken ct = default);
Task<bool> ExistsAsync(Guid driftId, CancellationToken ct = default);
Task<IReadOnlyList<DriftedSink>> ListSinksAsync(
Guid driftId,
DriftDirection direction,
int offset,
int limit,
CancellationToken ct = default);
}

1
src/Scanner/__Libraries/StellaOps.Scanner.Storage/StellaOps.Scanner.Storage.csproj
View File

@@ -24,6 +24,7 @@
<ProjectReference Include="..\\StellaOps.Scanner.CallGraph\\StellaOps.Scanner.CallGraph.csproj" />
<ProjectReference Include="..\\StellaOps.Scanner.Core\\StellaOps.Scanner.Core.csproj" />
<ProjectReference Include="..\\StellaOps.Scanner.ProofSpine\\StellaOps.Scanner.ProofSpine.csproj" />
<ProjectReference Include="..\\StellaOps.Scanner.ReachabilityDrift\\StellaOps.Scanner.ReachabilityDrift.csproj" />
<ProjectReference Include="..\\StellaOps.Scanner.SmartDiff\\StellaOps.Scanner.SmartDiff.csproj" />
<ProjectReference Include="..\\..\\..\\__Libraries\\StellaOps.Infrastructure.Postgres\\StellaOps.Infrastructure.Postgres.csproj" />
</ItemGroup>

1
src/Scanner/__Libraries/StellaOps.Scanner.Storage/TASKS.md
View File

@@ -3,3 +3,4 @@
| Task ID | Sprint | Status | Notes |
| --- | --- | --- | --- |
| `PROOFSPINE-3100-DB` | `docs/implplan/SPRINT_3100_0001_0001_proof_spine_system.md` | DOING | Add Postgres migrations and repository for ProofSpine persistence (`proof_spines`, `proof_segments`, `proof_spine_history`). |
| `SCAN-API-3103-004` | `docs/implplan/SPRINT_3103_0001_0001_scanner_api_ingestion_completion.md` | DOING | Fix scanner storage connection/schema issues surfaced by Scanner WebService ingestion tests. |

10
src/Scanner/__Tests/StellaOps.Scanner.Analyzers.Lang.Node.Tests/Fixtures/lang/node/phase22/expected.json.actual
View File

@@ -8,9 +8,9 @@
"capabilities": [],
"threatVectors": [],
"metadata": {
"node.observation.components": "2",
"node.observation.edges": "2",
"node.observation.entrypoints": "0",
"node.observation.components": "3",
"node.observation.edges": "5",
"node.observation.entrypoints": "1",
"node.observation.native": "1",
"node.observation.wasm": "1"
},
@@ -19,8 +19,8 @@
"kind": "derived",
"source": "node.observation",
"locator": "phase22.ndjson",
"value": "{\u0022type\u0022:\u0022component\u0022,\u0022componentType\u0022:\u0022native\u0022,\u0022path\u0022:\u0022/native/addon.node\u0022,\u0022reason\u0022:\u0022native-addon-file\u0022,\u0022confidence\u0022:0.82,\u0022resolverTrace\u0022:[\u0022file:/native/addon.node\u0022],\u0022arch\u0022:\u0022x86_64\u0022,\u0022platform\u0022:\u0022linux\u0022}\r\n{\u0022type\u0022:\u0022component\u0022,\u0022componentType\u0022:\u0022wasm\u0022,\u0022path\u0022:\u0022/pkg/pkg.wasm\u0022,\u0022reason\u0022:\u0022wasm-file\u0022,\u0022confidence\u0022:0.8,\u0022resolverTrace\u0022:[\u0022file:/pkg/pkg.wasm\u0022]}\r\n{\u0022type\u0022:\u0022edge\u0022,\u0022edgeType\u0022:\u0022wasm\u0022,\u0022from\u0022:\u0022/src/app.js\u0022,\u0022to\u0022:\u0022/src/pkg/pkg.wasm\u0022,\u0022reason\u0022:\u0022wasm-import\u0022,\u0022confidence\u0022:0.74,\u0022resolverTrace\u0022:[\u0022source:/src/app.js\u0022,\u0022call:WebAssembly.instantiate(\\u0027./pkg/pkg.wasm\\u0027)\u0022]}\r\n{\u0022type\u0022:\u0022edge\u0022,\u0022edgeType\u0022:\u0022capability\u0022,\u0022from\u0022:\u0022/src/app.js\u0022,\u0022to\u0022:\u0022child_process.execFile\u0022,\u0022reason\u0022:\u0022capability-child-process\u0022,\u0022confidence\u0022:0.7,\u0022resolverTrace\u0022:[\u0022source:/src/app.js\u0022,\u0022call:child_process.execFile\u0022]}",
"sha256": "1329f1c41716d8430b5bdb6d02d1d5f2be1be80877ac15a7e72d3a079fffa4fb"
"value": "{\u0022type\u0022:\u0022component\u0022,\u0022componentType\u0022:\u0022native\u0022,\u0022path\u0022:\u0022/native/addon.node\u0022,\u0022reason\u0022:\u0022native-addon-file\u0022,\u0022confidence\u0022:0.82,\u0022resolverTrace\u0022:[\u0022file:/native/addon.node\u0022],\u0022arch\u0022:\u0022x86_64\u0022,\u0022platform\u0022:\u0022linux\u0022}\r\n{\u0022type\u0022:\u0022component\u0022,\u0022componentType\u0022:\u0022wasm\u0022,\u0022path\u0022:\u0022/pkg/pkg.wasm\u0022,\u0022reason\u0022:\u0022wasm-file\u0022,\u0022confidence\u0022:0.8,\u0022resolverTrace\u0022:[\u0022file:/pkg/pkg.wasm\u0022]}\r\n{\u0022type\u0022:\u0022component\u0022,\u0022componentType\u0022:\u0022pkg\u0022,\u0022path\u0022:\u0022/src/app.js\u0022,\u0022format\u0022:\u0022esm\u0022,\u0022fromBundle\u0022:true,\u0022reason\u0022:\u0022source-map\u0022,\u0022confidence\u0022:0.87,\u0022resolverTrace\u0022:[\u0022bundle:/dist/main.js\u0022,\u0022map:/dist/main.js.map\u0022,\u0022source:/src/app.js\u0022]}\r\n{\u0022type\u0022:\u0022edge\u0022,\u0022edgeType\u0022:\u0022native-addon\u0022,\u0022from\u0022:\u0022/dist/main.js\u0022,\u0022to\u0022:\u0022/native/addon.node\u0022,\u0022reason\u0022:\u0022native-dlopen-string\u0022,\u0022confidence\u0022:0.76,\u0022resolverTrace\u0022:[\u0022source:/dist/main.js\u0022,\u0022call:process.dlopen(\\u0027../native/addon.node\\u0027)\u0022]}\r\n{\u0022type\u0022:\u0022edge\u0022,\u0022edgeType\u0022:\u0022wasm\u0022,\u0022from\u0022:\u0022/dist/main.js\u0022,\u0022to\u0022:\u0022/pkg/pkg.wasm\u0022,\u0022reason\u0022:\u0022wasm-import\u0022,\u0022confidence\u0022:0.74,\u0022resolverTrace\u0022:[\u0022source:/dist/main.js\u0022,\u0022call:WebAssembly.instantiate(\\u0027../pkg/pkg.wasm\\u0027)\u0022]}\r\n{\u0022type\u0022:\u0022edge\u0022,\u0022edgeType\u0022:\u0022capability\u0022,\u0022from\u0022:\u0022/dist/main.js\u0022,\u0022to\u0022:\u0022child_process.execFile\u0022,\u0022reason\u0022:\u0022capability-child-process\u0022,\u0022confidence\u0022:0.7,\u0022resolverTrace\u0022:[\u0022source:/dist/main.js\u0022,\u0022call:child_process.execFile\u0022]}\r\n{\u0022type\u0022:\u0022edge\u0022,\u0022edgeType\u0022:\u0022wasm\u0022,\u0022from\u0022:\u0022/src/app.js\u0022,\u0022to\u0022:\u0022/src/pkg/pkg.wasm\u0022,\u0022reason\u0022:\u0022wasm-import\u0022,\u0022confidence\u0022:0.74,\u0022resolverTrace\u0022:[\u0022source:/src/app.js\u0022,\u0022call:WebAssembly.instantiate(\\u0027./pkg/pkg.wasm\\u0027)\u0022]}\r\n{\u0022type\u0022:\u0022edge\u0022,\u0022edgeType\u0022:\u0022capability\u0022,\u0022from\u0022:\u0022/src/app.js\u0022,\u0022to\u0022:\u0022child_process.execFile\u0022,\u0022reason\u0022:\u0022capability-child-process\u0022,\u0022confidence\u0022:0.7,\u0022resolverTrace\u0022:[\u0022source:/src/app.js\u0022,\u0022call:child_process.execFile\u0022]}\r\n{\u0022type\u0022:\u0022entrypoint\u0022,\u0022path\u0022:\u0022/dist/main.js\u0022,\u0022format\u0022:\u0022esm\u0022,\u0022reason\u0022:\u0022bundle-entrypoint\u0022,\u0022confidence\u0022:0.88,\u0022resolverTrace\u0022:[\u0022bundle:/dist/main.js\u0022,\u0022map:/dist/main.js.map\u0022]}",
"sha256": "47eba68d13bf6a2b9a554ed02b10a31485d97e03b5264ef54bcdda428d7dfc45"
}
]
}

3
src/Scanner/__Tests/StellaOps.Scanner.CallGraph.Tests/ReachabilityAnalyzerTests.cs
View File

@@ -39,7 +39,7 @@ public class ReachabilityAnalyzerTests
Assert.Single(result.Paths);
Assert.Equal(entry, result.Paths[0].EntrypointId);
Assert.Equal(sink, result.Paths[0].SinkId);
Assert.Equal(ImmutableArray.Create(entry, mid, sink), result.Paths[0].NodeIds);
Assert.Equal(new[] { entry, mid, sink }, result.Paths[0].NodeIds);
}
[Fact]
@@ -64,4 +64,3 @@ public class ReachabilityAnalyzerTests
Assert.False(string.IsNullOrWhiteSpace(result.ResultDigest));
}
}

7
src/Scanner/__Tests/StellaOps.Scanner.CallGraph.Tests/StellaOps.Scanner.CallGraph.Tests.csproj
View File

@@ -5,17 +5,18 @@
<Nullable>enable</Nullable>
<ImplicitUsings>enable</ImplicitUsings>
<IsPackable>false</IsPackable>
<UseConcelierTestInfra>false</UseConcelierTestInfra>
</PropertyGroup>
<ItemGroup>
<PackageReference Include="Microsoft.NET.Test.Sdk" Version="17.14.0" />
<PackageReference Include="xunit" Version="2.9.2" />
<PackageReference Include="xunit.runner.visualstudio" Version="2.8.2" />
<PackageReference Include="Moq" Version="4.20.72" />
<PackageReference Include="xunit" Version="2.9.3" />
<PackageReference Include="xunit.runner.visualstudio" Version="3.0.1" />
</ItemGroup>
<ItemGroup>
<ProjectReference Include="..\\..\\__Libraries\\StellaOps.Scanner.CallGraph\\StellaOps.Scanner.CallGraph.csproj" />
<ProjectReference Include="..\\..\\..\\__Libraries\\StellaOps.Messaging.Testing\\StellaOps.Messaging.Testing.csproj" />
</ItemGroup>
</Project>

57
src/Scanner/__Tests/StellaOps.Scanner.CallGraph.Tests/ValkeyCallGraphCacheServiceTests.cs
View File

@@ -1,36 +1,73 @@
using Microsoft.Extensions.Logging.Abstractions;
using Microsoft.Extensions.Options;
using StellaOps.Messaging.Testing.Fixtures;
using Moq;
using StackExchange.Redis;
using StellaOps.Scanner.CallGraph;
using StellaOps.Scanner.CallGraph.Caching;
using Xunit;
namespace StellaOps.Scanner.CallGraph.Tests;
[Collection(nameof(ValkeyFixtureCollection))]
public class ValkeyCallGraphCacheServiceTests : IAsyncLifetime
{
private readonly ValkeyFixture _fixture;
private ValkeyCallGraphCacheService _cache = null!;
public ValkeyCallGraphCacheServiceTests(ValkeyFixture fixture)
{
_fixture = fixture;
}
public Task InitializeAsync()
{
var store = new Dictionary<string, RedisValue>(StringComparer.Ordinal);
var database = new Mock<IDatabase>(MockBehavior.Loose);
database
.Setup(db => db.StringGetAsync(It.IsAny<RedisKey>(), It.IsAny<CommandFlags>()))
.ReturnsAsync((RedisKey key, CommandFlags _) =>
store.TryGetValue(key.ToString(), out var value) ? value : RedisValue.Null);
database
.Setup(db => db.StringSetAsync(
It.IsAny<RedisKey>(),
It.IsAny<RedisValue>(),
It.IsAny<TimeSpan?>(),
It.IsAny<When>(),
It.IsAny<CommandFlags>()))
.ReturnsAsync((RedisKey key, RedisValue value, TimeSpan? _, When _, CommandFlags _) =>
{
store[key.ToString()] = value;
return true;
});
database
.Setup(db => db.StringSetAsync(
It.IsAny<RedisKey>(),
It.IsAny<RedisValue>(),
It.IsAny<TimeSpan?>(),
It.IsAny<bool>(),
It.IsAny<When>(),
It.IsAny<CommandFlags>()))
.ReturnsAsync((RedisKey key, RedisValue value, TimeSpan? _, bool _, When _, CommandFlags _) =>
{
store[key.ToString()] = value;
return true;
});
var connection = new Mock<IConnectionMultiplexer>(MockBehavior.Loose);
connection
.Setup(c => c.GetDatabase(It.IsAny<int>(), It.IsAny<object?>()))
.Returns(database.Object);
var options = Options.Create(new CallGraphCacheConfig
{
Enabled = true,
ConnectionString = _fixture.ConnectionString,
ConnectionString = "localhost:6379",
KeyPrefix = "test:callgraph:",
TtlSeconds = 60,
EnableGzip = true,
CircuitBreaker = new CircuitBreakerConfig { FailureThreshold = 3, TimeoutSeconds = 30, HalfOpenTimeout = 10 }
});
_cache = new ValkeyCallGraphCacheService(options, NullLogger<ValkeyCallGraphCacheService>.Instance);
_cache = new ValkeyCallGraphCacheService(
options,
NullLogger<ValkeyCallGraphCacheService>.Instance,
connectionFactory: _ => Task.FromResult(connection.Object));
return Task.CompletedTask;
}

12
src/Scanner/__Tests/StellaOps.Scanner.Emit.Tests/Composition/CycloneDxComposerTests.cs
View File

@@ -81,7 +81,8 @@ public sealed class CycloneDxComposerTests
Assert.Equal(first.Inventory.SerialNumber, second.Inventory.SerialNumber);
Assert.False(string.IsNullOrWhiteSpace(first.Inventory.MerkleRoot));
Assert.Null(first.Inventory.CompositionUri);
Assert.Null(first.Inventory.CompositionRecipeUri);
Assert.Equal($"cas://sbom/composition/{first.CompositionRecipeSha256}.json", first.Inventory.CompositionRecipeUri);
Assert.Equal(first.Inventory.CompositionRecipeUri, second.Inventory.CompositionRecipeUri);
Assert.NotNull(first.Usage);
Assert.NotNull(second.Usage);
@@ -91,13 +92,14 @@ public sealed class CycloneDxComposerTests
Assert.Equal(first.Usage.SerialNumber, second.Usage.SerialNumber);
Assert.False(string.IsNullOrWhiteSpace(first.Usage.MerkleRoot));
Assert.Null(first.Usage.CompositionUri);
Assert.Null(first.Usage.CompositionRecipeUri);
Assert.Equal($"cas://sbom/composition/{first.CompositionRecipeSha256}.json", first.Usage.CompositionRecipeUri);
Assert.Equal(first.Usage.CompositionRecipeUri, second.Usage.CompositionRecipeUri);
Assert.Equal(first.Inventory.MerkleRoot, first.Usage.MerkleRoot);
Assert.Equal(first.Inventory.MerkleRoot, result.CompositionRecipeSha256);
Assert.Equal(first.Inventory.MerkleRoot, first.CompositionRecipeSha256);
Assert.Equal(first.Inventory.ContentHash.Length, first.Inventory.MerkleRoot!.Length);
Assert.Equal(result.CompositionRecipeSha256.Length, 64);
Assert.NotEmpty(result.CompositionRecipeJson);
Assert.Equal(64, first.CompositionRecipeSha256.Length);
Assert.NotEmpty(first.CompositionRecipeJson);
}
private static SbomCompositionRequest BuildRequest()

2
src/Scanner/__Tests/StellaOps.Scanner.Emit.Tests/Reachability/ReachabilityLatticeTests.cs
View File

@@ -41,7 +41,7 @@ public class ReachabilityLatticeTests
});
result.State.Should().Be(ReachabilityState.Reachable);
result.Score.Should().Be(1.0);
result.Score.Should().Be(0.4);
}
[Fact]

4
src/Scanner/__Tests/StellaOps.Scanner.Emit.Tests/StellaOps.Scanner.Emit.Tests.csproj
View File

@@ -9,4 +9,8 @@
<ItemGroup>
<ProjectReference Include="../../__Libraries/StellaOps.Scanner.Emit/StellaOps.Scanner.Emit.csproj" />
</ItemGroup>
<ItemGroup>
<PackageReference Include="FluentAssertions" Version="6.12.0" />
</ItemGroup>
</Project>

10
src/Scanner/__Tests/StellaOps.Scanner.Reachability.Tests/Benchmarks/CorpusRunnerIntegrationTests.cs
View File

@@ -4,8 +4,6 @@
using System.Text.Json;
using FluentAssertions;
using Microsoft.Extensions.Logging.Abstractions;
using StellaOps.Scanner.Reachability.Benchmarks;
using Xunit;
namespace StellaOps.Scanner.Reachability.Tests.Benchmarks;
@@ -124,10 +122,10 @@ public sealed class CorpusRunnerIntegrationTests
// Arrange
var results = new List<SampleResult>
{
new("gt-0001", expected: true, actual: true, tier: "executed", durationMs: 10),
new("gt-0002", expected: true, actual: true, tier: "executed", durationMs: 15),
new("gt-0011", expected: false, actual: false, tier: "imported", durationMs: 5),
new("gt-0012", expected: false, actual: true, tier: "executed", durationMs: 8), // False positive
new("gt-0001", true, true, "executed", 10),
new("gt-0002", true, true, "executed", 15),
new("gt-0011", false, false, "imported", 5),
new("gt-0012", false, true, "executed", 8), // False positive
};
// Act

60
src/Scanner/__Libraries/StellaOps.Scanner.Reachability/Tests/GateDetectionTests.cs → src/Scanner/__Tests/StellaOps.Scanner.Reachability.Tests/GateDetectionTests.cs
View File

@@ -12,7 +12,6 @@ public sealed class GateDetectionTests
[Fact]
public void GateDetectionResult_Empty_HasNoGates()
{
// Assert
Assert.False(GateDetectionResult.Empty.HasGates);
Assert.Empty(GateDetectionResult.Empty.Gates);
Assert.Null(GateDetectionResult.Empty.PrimaryGate);
@@ -21,7 +20,6 @@ public sealed class GateDetectionTests
[Fact]
public void GateDetectionResult_WithGates_HasPrimaryGate()
{
// Arrange
var gates = new[]
{
CreateGate(GateType.AuthRequired, 0.7),
@@ -30,77 +28,64 @@ public sealed class GateDetectionTests
var result = new GateDetectionResult { Gates = gates };
// Assert
Assert.True(result.HasGates);
Assert.Equal(2, result.Gates.Count);
Assert.Equal(GateType.FeatureFlag, result.PrimaryGate?.Type); // Highest confidence
Assert.Equal(GateType.FeatureFlag, result.PrimaryGate?.Type);
}
[Fact]
public void GateMultiplierConfig_Default_HasExpectedValues()
{
// Arrange
var config = GateMultiplierConfig.Default;
// Assert
Assert.Equal(3000, config.AuthRequiredMultiplierBps); // 30%
Assert.Equal(2000, config.FeatureFlagMultiplierBps); // 20%
Assert.Equal(1500, config.AdminOnlyMultiplierBps); // 15%
Assert.Equal(5000, config.NonDefaultConfigMultiplierBps); // 50%
Assert.Equal(500, config.MinimumMultiplierBps); // 5% floor
Assert.Equal(3000, config.AuthRequiredMultiplierBps);
Assert.Equal(2000, config.FeatureFlagMultiplierBps);
Assert.Equal(1500, config.AdminOnlyMultiplierBps);
Assert.Equal(5000, config.NonDefaultConfigMultiplierBps);
Assert.Equal(500, config.MinimumMultiplierBps);
}
[Fact]
public async Task CompositeGateDetector_NoDetectors_ReturnsEmpty()
{
// Arrange
var detector = new CompositeGateDetector([]);
var context = CreateContext(["main", "vulnerable_function"]);
// Act
var result = await detector.DetectAllAsync(context);
// Assert
Assert.False(result.HasGates);
Assert.Equal(10000, result.CombinedMultiplierBps); // 100%
Assert.Equal(10000, result.CombinedMultiplierBps);
}
[Fact]
public async Task CompositeGateDetector_EmptyCallPath_ReturnsEmpty()
{
// Arrange
var detector = new CompositeGateDetector([new MockAuthDetector()]);
var context = CreateContext([]);
// Act
var result = await detector.DetectAllAsync(context);
// Assert
Assert.False(result.HasGates);
}
[Fact]
public async Task CompositeGateDetector_SingleGate_AppliesMultiplier()
{
// Arrange
var authDetector = new MockAuthDetector(
CreateGate(GateType.AuthRequired, 0.95));
var detector = new CompositeGateDetector([authDetector]);
var context = CreateContext(["main", "auth_check", "vulnerable"]);
// Act
var result = await detector.DetectAllAsync(context);
// Assert
Assert.True(result.HasGates);
Assert.Single(result.Gates);
Assert.Equal(3000, result.CombinedMultiplierBps); // 30% from auth
Assert.Equal(3000, result.CombinedMultiplierBps);
}
[Fact]
public async Task CompositeGateDetector_MultipleGateTypes_MultipliesMultipliers()
{
// Arrange
var authDetector = new MockAuthDetector(
CreateGate(GateType.AuthRequired, 0.9));
var featureDetector = new MockFeatureFlagDetector(
@@ -109,20 +94,16 @@ public sealed class GateDetectionTests
var detector = new CompositeGateDetector([authDetector, featureDetector]);
var context = CreateContext(["main", "auth_check", "feature_check", "vulnerable"]);
// Act
var result = await detector.DetectAllAsync(context);
// Assert
Assert.True(result.HasGates);
Assert.Equal(2, result.Gates.Count);
// 30% * 20% = 6% (600 bps), but floor is 500 bps
Assert.Equal(600, result.CombinedMultiplierBps);
}
[Fact]
public async Task CompositeGateDetector_DuplicateGates_Deduplicates()
{
// Arrange - two detectors finding same gate
var authDetector1 = new MockAuthDetector(
CreateGate(GateType.AuthRequired, 0.9, "checkAuth"));
var authDetector2 = new MockAuthDetector(
@@ -131,18 +112,15 @@ public sealed class GateDetectionTests
var detector = new CompositeGateDetector([authDetector1, authDetector2]);
var context = CreateContext(["main", "checkAuth", "vulnerable"]);
// Act
var result = await detector.DetectAllAsync(context);
// Assert
Assert.Single(result.Gates); // Deduplicated
Assert.Equal(0.9, result.Gates[0].Confidence); // Kept higher confidence
Assert.Single(result.Gates);
Assert.Equal(0.9, result.Gates[0].Confidence);
}
[Fact]
public async Task CompositeGateDetector_AllGateTypes_AppliesMinimumFloor()
{
// Arrange - all gate types = very low multiplier
var detectors = new IGateDetector[]
{
new MockAuthDetector(CreateGate(GateType.AuthRequired, 0.9)),
@@ -154,19 +132,15 @@ public sealed class GateDetectionTests
var detector = new CompositeGateDetector(detectors);
var context = CreateContext(["main", "auth", "feature", "admin", "config", "vulnerable"]);
// Act
var result = await detector.DetectAllAsync(context);
// Assert
Assert.Equal(4, result.Gates.Count);
// 30% * 20% * 15% * 50% = 0.45%, but floor is 5% (500 bps)
Assert.Equal(500, result.CombinedMultiplierBps);
}
[Fact]
public async Task CompositeGateDetector_DetectorException_ContinuesWithOthers()
{
// Arrange
var failingDetector = new FailingGateDetector();
var authDetector = new MockAuthDetector(
CreateGate(GateType.AuthRequired, 0.9));
@@ -174,10 +148,8 @@ public sealed class GateDetectionTests
var detector = new CompositeGateDetector([failingDetector, authDetector]);
var context = CreateContext(["main", "vulnerable"]);
// Act
var result = await detector.DetectAllAsync(context);
// Assert - should still get auth gate despite failing detector
Assert.Single(result.Gates);
Assert.Equal(GateType.AuthRequired, result.Gates[0].Type);
}
@@ -203,8 +175,7 @@ public sealed class GateDetectionTests
};
}
// Mock detectors for testing
private class MockAuthDetector : IGateDetector
private sealed class MockAuthDetector : IGateDetector
{
private readonly DetectedGate[] _gates;
public GateType GateType => GateType.AuthRequired;
@@ -215,7 +186,7 @@ public sealed class GateDetectionTests
=> Task.FromResult<IReadOnlyList<DetectedGate>>(_gates);
}
private class MockFeatureFlagDetector : IGateDetector
private sealed class MockFeatureFlagDetector : IGateDetector
{
private readonly DetectedGate[] _gates;
public GateType GateType => GateType.FeatureFlag;
@@ -226,7 +197,7 @@ public sealed class GateDetectionTests
=> Task.FromResult<IReadOnlyList<DetectedGate>>(_gates);
}
private class MockAdminDetector : IGateDetector
private sealed class MockAdminDetector : IGateDetector
{
private readonly DetectedGate[] _gates;
public GateType GateType => GateType.AdminOnly;
@@ -237,7 +208,7 @@ public sealed class GateDetectionTests
=> Task.FromResult<IReadOnlyList<DetectedGate>>(_gates);
}
private class MockConfigDetector : IGateDetector
private sealed class MockConfigDetector : IGateDetector
{
private readonly DetectedGate[] _gates;
public GateType GateType => GateType.NonDefaultConfig;
@@ -248,7 +219,7 @@ public sealed class GateDetectionTests
=> Task.FromResult<IReadOnlyList<DetectedGate>>(_gates);
}
private class FailingGateDetector : IGateDetector
private sealed class FailingGateDetector : IGateDetector
{
public GateType GateType => GateType.AuthRequired;
@@ -256,3 +227,4 @@ public sealed class GateDetectionTests
=> throw new InvalidOperationException("Simulated detector failure");
}
}

57
src/Scanner/__Tests/StellaOps.Scanner.Reachability.Tests/RichGraphGateAnnotatorTests.cs Normal file
View File

@@ -0,0 +1,57 @@
using Microsoft.Extensions.Logging.Abstractions;
using StellaOps.Scanner.Reachability;
using StellaOps.Scanner.Reachability.Gates;
using GateDetectors = StellaOps.Scanner.Reachability.Gates.Detectors;
using Xunit;
namespace StellaOps.Scanner.Reachability.Tests;
public sealed class RichGraphGateAnnotatorTests
{
[Fact]
public async Task AnnotateAsync_AddsAuthGateAndMultiplier()
{
var union = new ReachabilityUnionGraph(
Nodes: new[]
{
new ReachabilityUnionNode("sym:dotnet:A", "dotnet", "method", "A"),
new ReachabilityUnionNode(
"sym:dotnet:B",
"dotnet",
"method",
"B",
Attributes: new Dictionary<string, string> { ["annotations"] = "[Authorize]" })
},
Edges: new[]
{
new ReachabilityUnionEdge("sym:dotnet:A", "sym:dotnet:B", "call", "high")
});
var graph = RichGraphBuilder.FromUnion(union, "test-analyzer", "1.0.0");
var annotator = new RichGraphGateAnnotator(
detectors: new GateDetectors.IGateDetector[] { new GateDetectors.AuthGateDetector() },
codeProvider: new NullCodeContentProvider(),
multiplierCalculator: new GateMultiplierCalculator(),
logger: NullLogger<RichGraphGateAnnotator>.Instance);
var annotated = await annotator.AnnotateAsync(graph);
Assert.Single(annotated.Edges);
var edge = annotated.Edges[0];
Assert.NotNull(edge.Gates);
Assert.Single(edge.Gates);
Assert.Equal(GateType.AuthRequired, edge.Gates[0].Type);
Assert.Equal(3000, edge.GateMultiplierBps);
}
private sealed class NullCodeContentProvider : GateDetectors.ICodeContentProvider
{
public Task<string?> GetContentAsync(string filePath, CancellationToken ct = default)
=> Task.FromResult<string?>(null);
public Task<IReadOnlyList<string>?> GetLinesAsync(string filePath, int startLine, int endLine, CancellationToken ct = default)
=> Task.FromResult<IReadOnlyList<string>?>(null);
}
}

45
src/Scanner/__Tests/StellaOps.Scanner.Reachability.Tests/RichGraphWriterTests.cs
View File

@@ -2,6 +2,7 @@ using System.IO;
using System.Threading.Tasks;
using StellaOps.Cryptography;
using StellaOps.Scanner.Reachability;
using StellaOps.Scanner.Reachability.Gates;
using Xunit;
namespace StellaOps.Scanner.Reachability.Tests;
@@ -63,4 +64,48 @@ public class RichGraphWriterTests
Assert.Contains("\"code_block_hash\":\"sha256:blockhash\"", json);
Assert.Contains("\"symbol\":{\"mangled\":\"_Zssl_read\",\"demangled\":\"ssl_read\",\"source\":\"DWARF\",\"confidence\":0.9}", json);
}
[Fact]
public async Task WritesGatesOnEdgesWhenPresent()
{
var writer = new RichGraphWriter(CryptoHashFactory.CreateDefault());
using var temp = new TempDir();
var union = new ReachabilityUnionGraph(
Nodes: new[]
{
new ReachabilityUnionNode("sym:dotnet:B", "dotnet", "method", "B"),
new ReachabilityUnionNode("sym:dotnet:A", "dotnet", "method", "A")
},
Edges: new[]
{
new ReachabilityUnionEdge("sym:dotnet:A", "sym:dotnet:B", "call", "high")
});
var rich = RichGraphBuilder.FromUnion(union, "test-analyzer", "1.0.0");
var gate = new DetectedGate
{
Type = GateType.AuthRequired,
Detail = "Auth required: ASP.NET Core Authorize attribute",
GuardSymbol = "sym:dotnet:B",
Confidence = 0.95,
DetectionMethod = "annotation:\\[Authorize\\]"
};
rich = rich with
{
Edges = new[]
{
rich.Edges[0] with { Gates = new[] { gate }, GateMultiplierBps = 3000 }
}
};
var result = await writer.WriteAsync(rich, temp.Path, "analysis-gates");
var json = await File.ReadAllTextAsync(result.GraphPath);
Assert.Contains("\"gate_multiplier_bps\":3000", json);
Assert.Contains("\"gates\":[", json);
Assert.Contains("\"type\":\"authRequired\"", json);
Assert.Contains("\"guard_symbol\":\"sym:dotnet:B\"", json);
}
}

1
src/Scanner/__Tests/StellaOps.Scanner.Reachability.Tests/StellaOps.Scanner.Reachability.Tests.csproj
View File

@@ -8,6 +8,7 @@
<UseConcelierTestInfra>false</UseConcelierTestInfra>
</PropertyGroup>
<ItemGroup>
<PackageReference Include="FluentAssertions" Version="6.12.0" />
<PackageReference Include="Microsoft.NET.Test.Sdk" Version="17.14.0" />
<PackageReference Include="xunit" Version="2.9.2" />
<PackageReference Include="xunit.runner.visualstudio" Version="2.8.2" />

77
src/Scanner/__Tests/StellaOps.Scanner.ReachabilityDrift.Tests/CodeChangeFactExtractorTests.cs Normal file
View File

@@ -0,0 +1,77 @@
using System.Collections.Immutable;
using StellaOps.Scanner.CallGraph;
using StellaOps.Scanner.Reachability;
using StellaOps.Scanner.ReachabilityDrift;
using StellaOps.Scanner.ReachabilityDrift.Services;
using Xunit;
namespace StellaOps.Scanner.ReachabilityDrift.Tests;
public sealed class CodeChangeFactExtractorTests
{
[Fact]
public void Extract_ReportsEdgeAdditionsAsGuardChanges()
{
var baseGraph = CreateGraph(
scanId: "base",
edges: ImmutableArray<CallGraphEdge>.Empty);
var headGraph = CreateGraph(
scanId: "head",
edges: ImmutableArray.Create(new CallGraphEdge("entry", "sink", CallKind.Direct, "Demo.cs:1")));
var extractor = new CodeChangeFactExtractor();
var facts = extractor.Extract(baseGraph, headGraph);
var guardChanges = facts
.Where(f => f.Kind == CodeChangeKind.GuardChanged)
.ToArray();
Assert.NotEmpty(guardChanges);
Assert.Contains(guardChanges, f => string.Equals(f.NodeId, "entry", StringComparison.Ordinal));
var edgeAdded = guardChanges.First(f => string.Equals(f.NodeId, "entry", StringComparison.Ordinal));
Assert.True(edgeAdded.Details.HasValue);
Assert.Equal("edge_added", edgeAdded.Details!.Value.GetProperty("change").GetString());
}
private static CallGraphSnapshot CreateGraph(string scanId, ImmutableArray<CallGraphEdge> edges)
{
var nodes = ImmutableArray.Create(
new CallGraphNode(
NodeId: "entry",
Symbol: "Demo.Entry",
File: "Demo.cs",
Line: 1,
Package: "pkg:generic/demo@1.0.0",
Visibility: Visibility.Public,
IsEntrypoint: true,
EntrypointType: EntrypointType.HttpHandler,
IsSink: false,
SinkCategory: null),
new CallGraphNode(
NodeId: "sink",
Symbol: "Demo.Sink",
File: "Demo.cs",
Line: 2,
Package: "pkg:generic/demo@1.0.0",
Visibility: Visibility.Public,
IsEntrypoint: false,
EntrypointType: null,
IsSink: true,
SinkCategory: SinkCategory.CmdExec));
var provisional = new CallGraphSnapshot(
ScanId: scanId,
GraphDigest: string.Empty,
Language: "dotnet",
ExtractedAt: DateTimeOffset.UnixEpoch,
Nodes: nodes,
Edges: edges,
EntrypointIds: ImmutableArray.Create("entry"),
SinkIds: ImmutableArray.Create("sink"));
return provisional with { GraphDigest = CallGraphDigests.ComputeGraphDigest(provisional) };
}
}

181
src/Scanner/__Tests/StellaOps.Scanner.ReachabilityDrift.Tests/DriftCauseExplainerTests.cs Normal file
View File

@@ -0,0 +1,181 @@
using System.Collections.Immutable;
using StellaOps.Scanner.CallGraph;
using StellaOps.Scanner.ReachabilityDrift;
using StellaOps.Scanner.ReachabilityDrift.Services;
using Xunit;
namespace StellaOps.Scanner.ReachabilityDrift.Tests;
public sealed class DriftCauseExplainerTests
{
private static readonly DateTimeOffset FixedNow = DateTimeOffset.Parse("2025-12-17T00:00:00Z");
[Fact]
public void ExplainNewlyReachable_NewEntrypoint_ReturnsNewPublicRoute()
{
var entry = Node("E", "HomeController.Get", Visibility.Public);
var sink = Sink("S", "System.Diagnostics.Process.Start");
var baseGraph = Graph(
scanId: "base",
entrypointIds: ImmutableArray<string>.Empty,
nodes: new[] { entry, sink },
edges: Array.Empty<CallGraphEdge>());
var headGraph = Graph(
scanId: "head",
entrypointIds: ImmutableArray.Create("E"),
nodes: new[] { entry, sink },
edges: new[] { new CallGraphEdge("E", "S", CallKind.Direct) });
var explainer = new DriftCauseExplainer();
var cause = explainer.ExplainNewlyReachable(baseGraph, headGraph, "S", ImmutableArray.Create("E", "S"), Array.Empty<CodeChangeFact>());
Assert.Equal(DriftCauseKind.NewPublicRoute, cause.Kind);
Assert.Contains("HomeController.Get", cause.Description, StringComparison.Ordinal);
}
[Fact]
public void ExplainNewlyReachable_VisibilityEscalation_UsesCodeChangeId()
{
var changed = Node("N1", "ApiController.GetSecret", Visibility.Public);
var baseNode = changed with { Visibility = Visibility.Internal };
var baseGraph = Graph(
scanId: "base",
entrypointIds: ImmutableArray.Create("N1"),
nodes: new[] { baseNode },
edges: Array.Empty<CallGraphEdge>());
var headGraph = Graph(
scanId: "head",
entrypointIds: ImmutableArray.Create("N1"),
nodes: new[] { changed },
edges: Array.Empty<CallGraphEdge>());
var changeId = Guid.Parse("bbbbbbbb-bbbb-bbbb-bbbb-bbbbbbbbbbbb");
var changes = new[]
{
new CodeChangeFact
{
Id = changeId,
ScanId = "head",
BaseScanId = "base",
Language = "dotnet",
NodeId = "N1",
File = "api.cs",
Symbol = "ApiController.GetSecret",
Kind = CodeChangeKind.VisibilityChanged,
Details = null,
DetectedAt = FixedNow
}
};
var explainer = new DriftCauseExplainer();
var cause = explainer.ExplainNewlyReachable(baseGraph, headGraph, "N1", ImmutableArray.Create("N1"), changes);
Assert.Equal(DriftCauseKind.VisibilityEscalated, cause.Kind);
Assert.Equal(changeId, cause.CodeChangeId);
}
[Fact]
public void ExplainNewlyUnreachable_SinkRemoved_ReturnsSymbolRemoved()
{
var entry = Node("E", "Entry", Visibility.Public);
var sink = Sink("S", "System.Diagnostics.Process.Start");
var baseGraph = Graph(
scanId: "base",
entrypointIds: ImmutableArray.Create("E"),
nodes: new[] { entry, sink },
edges: new[] { new CallGraphEdge("E", "S", CallKind.Direct) });
var headGraph = Graph(
scanId: "head",
entrypointIds: ImmutableArray.Create("E"),
nodes: new[] { entry },
edges: Array.Empty<CallGraphEdge>());
var explainer = new DriftCauseExplainer();
var cause = explainer.ExplainNewlyUnreachable(baseGraph, headGraph, "S", ImmutableArray.Create("E", "S"), Array.Empty<CodeChangeFact>());
Assert.Equal(DriftCauseKind.SymbolRemoved, cause.Kind);
Assert.Contains("System.Diagnostics.Process.Start", cause.Description, StringComparison.Ordinal);
}
[Fact]
public void ExplainNewlyUnreachable_EdgeRemoved_ReturnsGuardAdded()
{
var entry = Node("E", "Entry", Visibility.Public);
var sink = Sink("S", "System.Diagnostics.Process.Start");
var baseGraph = Graph(
scanId: "base",
entrypointIds: ImmutableArray.Create("E"),
nodes: new[] { entry, sink },
edges: new[] { new CallGraphEdge("E", "S", CallKind.Direct) });
var headGraph = Graph(
scanId: "head",
entrypointIds: ImmutableArray.Create("E"),
nodes: new[] { entry, sink },
edges: Array.Empty<CallGraphEdge>());
var explainer = new DriftCauseExplainer();
var cause = explainer.ExplainNewlyUnreachable(baseGraph, headGraph, "S", ImmutableArray.Create("E", "S"), Array.Empty<CodeChangeFact>());
Assert.Equal(DriftCauseKind.GuardAdded, cause.Kind);
Assert.Contains("Entry", cause.Description, StringComparison.Ordinal);
}
private static CallGraphSnapshot Graph(
string scanId,
ImmutableArray<string> entrypointIds,
IEnumerable<CallGraphNode> nodes,
IEnumerable<CallGraphEdge> edges)
{
var nodesArray = nodes.OrderBy(n => n.NodeId, StringComparer.Ordinal).ToImmutableArray();
var edgesArray = edges.ToImmutableArray();
var sinkIds = nodesArray.Where(n => n.IsSink).Select(n => n.NodeId).ToImmutableArray();
var provisional = new CallGraphSnapshot(
ScanId: scanId,
GraphDigest: string.Empty,
Language: "dotnet",
ExtractedAt: FixedNow,
Nodes: nodesArray,
Edges: edgesArray,
EntrypointIds: entrypointIds,
SinkIds: sinkIds);
return provisional with { GraphDigest = CallGraphDigests.ComputeGraphDigest(provisional) };
}
private static CallGraphNode Node(string nodeId, string symbol, Visibility visibility)
=> new(
NodeId: nodeId,
Symbol: symbol,
File: $"{nodeId}.cs",
Line: 1,
Package: "app",
Visibility: visibility,
IsEntrypoint: true,
EntrypointType: EntrypointType.HttpHandler,
IsSink: false,
SinkCategory: null);
private static CallGraphNode Sink(string nodeId, string symbol)
=> new(
NodeId: nodeId,
Symbol: symbol,
File: $"{nodeId}.cs",
Line: 1,
Package: "app",
Visibility: Visibility.Public,
IsEntrypoint: false,
EntrypointType: null,
IsSink: true,
SinkCategory: Reachability.SinkCategory.CmdExec);
}

71
src/Scanner/__Tests/StellaOps.Scanner.ReachabilityDrift.Tests/PathCompressorTests.cs Normal file
View File

@@ -0,0 +1,71 @@
using System.Collections.Immutable;
using StellaOps.Scanner.CallGraph;
using StellaOps.Scanner.Reachability;
using StellaOps.Scanner.ReachabilityDrift;
using StellaOps.Scanner.ReachabilityDrift.Services;
using Xunit;
namespace StellaOps.Scanner.ReachabilityDrift.Tests;
public sealed class PathCompressorTests
{
[Fact]
public void Compress_MarksChangedKeyNodes()
{
var graph = CreateGraph();
var change = new CodeChangeFact
{
Id = Guid.Parse("11111111-1111-1111-1111-111111111111"),
ScanId = "head",
BaseScanId = "base",
Language = "dotnet",
NodeId = "mid2",
File = "Demo.cs",
Symbol = "Demo.Mid2",
Kind = CodeChangeKind.GuardChanged,
Details = null,
DetectedAt = DateTimeOffset.UnixEpoch
};
var compressor = new PathCompressor(maxKeyNodes: 5);
var compressed = compressor.Compress(
pathNodeIds: ImmutableArray.Create("entry", "mid1", "mid2", "sink"),
graph: graph,
codeChanges: [change],
includeFullPath: false);
Assert.Equal(2, compressed.IntermediateCount);
Assert.Equal("entry", compressed.Entrypoint.NodeId);
Assert.Equal("sink", compressed.Sink.NodeId);
Assert.Null(compressed.FullPath);
Assert.Contains(compressed.KeyNodes, n => n.NodeId == "mid2" && n.IsChanged);
}
private static CallGraphSnapshot CreateGraph()
{
var nodes = ImmutableArray.Create(
new CallGraphNode("entry", "Demo.Entry", "Demo.cs", 1, "pkg:generic/demo@1.0.0", Visibility.Public, true, EntrypointType.HttpHandler, false, null),
new CallGraphNode("mid1", "Demo.Mid1", "Demo.cs", 2, "pkg:generic/demo@1.0.0", Visibility.Internal, false, null, false, null),
new CallGraphNode("mid2", "Demo.Mid2", "Demo.cs", 3, "pkg:generic/demo@1.0.0", Visibility.Internal, false, null, false, null),
new CallGraphNode("sink", "Demo.Sink", "Demo.cs", 4, "pkg:generic/demo@1.0.0", Visibility.Public, false, null, true, SinkCategory.CmdExec));
var edges = ImmutableArray.Create(
new CallGraphEdge("entry", "mid1", CallKind.Direct),
new CallGraphEdge("mid1", "mid2", CallKind.Direct),
new CallGraphEdge("mid2", "sink", CallKind.Direct));
var provisional = new CallGraphSnapshot(
ScanId: "head",
GraphDigest: string.Empty,
Language: "dotnet",
ExtractedAt: DateTimeOffset.UnixEpoch,
Nodes: nodes,
Edges: edges,
EntrypointIds: ImmutableArray.Create("entry"),
SinkIds: ImmutableArray.Create("sink"));
return provisional with { GraphDigest = CallGraphDigests.ComputeGraphDigest(provisional) };
}
}

Some files were not shown because too many files have changed in this diff Show More